Qualtrics production system is comprised of many micro-services. Visualizing the network connections among them can be a daunting task, so I decided to write an internal tool to help myself with it. What if I could list out the micro-services and their dependencies in a table, and then slice and dice the data by filtering and sorting by the columns to interact with the data? The amount of data is small, in the order of thousands of relationships, so a lightweight tool would suffice.

Well, this sounds like a perfect job for the DataTables jQuery plug-in. Out of the box, it lets you sort by multiple columns by holding down the SHIFT key, provides paginated UI, and has a nice search box for filtering. I integrated it with AngularJS, served the data from Django REST Framework, and put them inside a docker container.

This article is a mini-tutorial on how to put these open source components together. Just to illustrate what I am talking about, here is a screenshot of the internal tool.

Figure 1. Screenshot of the Node Group Dependency Tab (with fake data)

The tool finds network connections within production docker containers as well as bare metal machines, aggregates them by the source and destination micro-services, and displays connection stats between them. As you can see in the screenshot, I filtered the list to show only the dc1 datacenter and sample service only. Then, sorted the table by Source Group first, and then Short-lived Connections in descending order. This UI makes it a snap to find my dependencies, and see how many network connections I am creating from a given micro-service tier to another.

Serving Static Assets From Django

Before we get on our way to write some JavaScript code, let’s set up a lightweight web server that can serve them. I used Django for that. Getting up and running with Django is straight-forward, as described in the official Getting Started Guide.

Setting up static assets may require a few tweaks, though. You can follow the official documentation about managing static files, and this will get your static assets up and running just fine, as long as you have DEBUG = True in your settings.py file. However, before you release your application to production, you should set DEBUG = False, which enables the cache busting feature. Unless you do this, the user’s browser will cache your static assets (like JavaScript files), and may not pick up the changes when you release new versions of your application.

If you are using a web server like nginx to serve static assets, you can simply point its configuration to the static files directory that you set up. However, my internal tool is very simple, and I prefer to serve the static assets from Django itself, so that I can ship a single docker container that contains everything. The trouble is that Django will not serve the static assets directly if you set DEBUG = False. It expects an external web server (like nginx) to serve the static assets on its behalf in production mode.

You can introduce the White Noise module into your code to solve this problem. First, import it in your wsgi.py file, by adding the two lines annotated below.

import os
from django.core.wsgi import get_wsgi_application
from whitenoise.django import DjangoWhiteNoise   # <-- LINE 1
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.settings")
application = get_wsgi_application()
application = DjangoWhiteNoise(application)      # <-- LINE 2

Second, make sure that all your static files are referenced using Django’s static file mark up in your HTML template files. They should look something like:

{% load staticfiles %}
<script type="text/javascript"
        src="{% static 'static/datatables/media/js/jquery.dataTables.js' %}"></script>
<script type="text/javascript"
        src="{% static 'javascript/angular.js' %}"></script> ...

Finally, add the following to your settings.py file.

MIDDLEWARE_CLASSES = [
    'django.middleware.security.SecurityMiddleware',
    'whitenoise.middleware.WhiteNoiseMiddleware', # Add this just below security middleware
    ...
]
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
STATIC_URL = '/static/'
STATICFILES_DIRS = [ os.path.join(BASE_DIR, "static") ]
STATIC_ROOT = '/var/www'
STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.ManifestStaticFilesStorage'

Once you set DEBUG = False, you need to make sure that the static files are installed into the target directory, in this case /var/www. You can do that by running:

./manage.py collectstatic --noinput

This installs all your static assets with unique names, and updates references to them in your HTML. This way, you don’t have to worry about browser cache eclipsing the newer versions of JavaScript or CSS files. Now that static assets are taken care of, let’s move on to the fun part: integrating DataTables and AngularJS!

Integrating DataTables with AngularJS

We will be using DataTables version 1.10.12, AngularJS version 1.5.7, and angular-datatables, which is a nice angular directive that allows easy integration of the two components. Incidentally, the newest documentation of angular-datatables uses Angular 2 and TypeScript to illustrate example usages. If you are still using an earlier version of AngularJS and JavaScript, you can view the archived documentation, or this blog will help you get started easily.

In case you are new to AngularJS itself, I would like to show the main entry point into our AngularJS app. Here is my main app.js file. My application is randomly named “novascotia.” I omitted some lines to fit the code into a blog format.

(function () {

  app = angular.module("novascotiaApp", [
                       "bookmarksModule",
                       "servicesModule",
                       "portsModule",
                       "nodesModule",
                       "nodeGroupsModule",
                       "exportGraphModule",
                       "nodeGroupDepsModule", // <---- we will look at this module
                       "dependenciesModule",
                       "teamsModule",
                       "mapModule"
                       ],
      function($locationProvider) {
          $locationProvider.html5Mode({
              enabled: true,
              requireBase: false
          });
      });

  app.controller("novascotiaMain",
                 ["$scope", "$currentTab", "$window", novascotiaMainController])
     .controller("navControl",
                 ["$scope", "$currentTab", "$window", "$location", navController])
     .factory("$currentTab", currentTabFactory)
     .factory("$helper", helperFactory);

  function novascotiaMainController($scope, $currentTab, $window) {
      // The application has a "tabbed" UI, and currentTab specifies
      // which tab you should land on. For space reasons, I omitted that code.
      $scope.currentTab = $currentTab;
  }

  // ... Abbreviated for space (e.g. $helper and $currentTab) ...

})();

As you can see above, it doesn’t take much to create an AngularJS app. All the modules listed in angular.module call above should be loaded by individual <script> tags, as follows:

<!DOCTYPE html>
<html>
    <head>
        <title>My Internal Tool</title>
        <meta charset="UTF-8"/>
        {% load staticfiles %}
        <link rel="stylesheet" type="text/css"
         href="{% static 'vendor/angular-datatables/dist/css/angular-datatables.css' %}"></link>
        <link rel="stylesheet" type="text/css"
         href="{% static 'vendor/datatables/media/css/jquery.dataTables.css' %}"></link>
        <script type="text/javascript"
         src="{% static 'javascript/app.js' %}"></script>
        <script type="text/javascript"
         src="{% static 'javascript/module/bookmarks-tab.js' %}"></script>
        <script type="text/javascript"
         src="{% static 'javascript/module/nodes-tab.js' %}"></script>
        <script type="text/javascript"
         src="{% static 'javascript/module/node-groups-tab.js' %}"></script>
        <script type="text/javascript"
         src="{% static 'javascript/module/node-group-deps-tab.js' %}"></script>
        <!-- ... Abbreviated ... -->
    </head>
    <body ng-app="novascotiaApp" ng-controller="novascotiaMain">
        <!-- ... Abbreviated ... -->
    </body>
</html>

Here node-group-deps.js is the file that will contain our code that uses angular-datatables to load DataTables with AngularJS. Before we look at that, here is the HTML template that we will be referring to in our AngularJS directive:

<div class="tab-body nodes">
  <p>
      <table id="results" datatable=""
          dt-options="page.dtOptions"
          dt-columns="page.dtColumns"
          dt-instance="page.dtInstance"
          class="row-border hover compact">
      </table>
  </p>
</div>

This snippet of HTML is the template used by our AngularJS directive defined. It provides the <table> element that a data table will be attached to. The key attribute here is datatable, which links the DOM element to angular-datatables module. As you can see, this element is quite empty. We add the desired columns and data rows using code, as shown below:

angular.module("nodeGroupDepsModule", ["datatables", "ng"])
.directive("nodeGroupDepsTab", nodeGroupDepsTabDirective)
.controller("nodeGroupDepsTabController",
    ["$scope", "DTOptionsBuilder", "DTColumnBuilder", "$q", "$http", "$compile", nodeGroupDepsTabController]);

function nodeGroupDepsTabDirective() {
    return {
        templateUrl: "/static/html/node-group-deps-tab.html",
        controller: "nodeGroupDepsTabController",
        controllerAs: "page"
    };
}

function nodeGroupDepsTabController($scope, DTOptionsBuilder, DTColumnBuilder, $q, $http, $compile) {

    var vm = this;

    function c_count(i) {
      return '<span class=count-color>(' + i + ')</span>';
    }

    function queryAPI() {
        var deferred = $q.defer();
        $http.get('/api/v1/nodegroupdeps/').then(function(result) {
            var nodeGroupDeps = [];
            jQuery.each(result.data.results, function(_idx, item) {
                var ports = [];
                var longlived = 0;
                var shortlived = 0;
                jQuery.each(JSON.parse(item.connections), function(port, stat) {
                    var lines = ['Port ' + port + ':'];
                    jQuery.each(stat, function(state, count) {
                        lines.push(state + ': ' + count);
                        if (state === 'ESTABLISHED') {
                            longlived += count;
                        } else {
                            shortlived += count;
                        }
                    });
                    ports.push(lines.join(' '));
                });
                item.connections = ports.sort().join('\n');
                item.longlived = longlived;
                item.shortlived = shortlived;
                nodeGroupDeps.push(item);
            });
            deferred.resolve(nodeGroupDeps);
        });
        return deferred.promise;
    }

    vm.dtOptions = DTOptionsBuilder
        .fromFnPromise(queryAPI)
        .withDisplayLength(400)
        .withOption('lengthMenu', [20, 50, 100, 400, 800])
        .withOption('stateSave', true)
        .withPaginationType('full_numbers');

    vm.dtColumns = [
        DTColumnBuilder.newColumn('id').withTitle('ID'),
        DTColumnBuilder.newColumn('src_group_name').withTitle('Source Group'),
        DTColumnBuilder.newColumn('src_datacenter').withTitle('Source Domain'),
        DTColumnBuilder.newColumn('dst_group_name').withTitle('Destination Group'),
        DTColumnBuilder.newColumn('dst_datacenter').withTitle('Destination Domain'),
        DTColumnBuilder.newColumn('longlived').withTitle('Long-lived Connections').withClass('numeric'),
        DTColumnBuilder.newColumn('shortlived').withTitle('Short-lived Connections').withClass('numeric'),
        DTColumnBuilder.newColumn('connections').withTitle('Connection Details').withClass('text-pre')
        ];

    vm.dtInstance = {};
}

At the very top of the file, you can see that I am loading the “datatables” module, which corresponds to angular-datatables module. Below that, we are defining the nodeGroupDepsTab directive, and the corresponding nodeGroupDepsTabController that defines the behavior. By AngularJS convention, the string nodeGroupDepsTab in .directive() call will define the new directive called node-group-deps-tab. So, I can simply add <node-group-deps-tab> tag anywhere in my AngularJS app to display the data table defined by this code.

This code defines the three attributes used by angular-datatable to configure our data table (refer to the template that we listed earlier): dtOptions, dtColumns, and dtInstance. They are instance variables of the controller object, and are accessible via the controllerAs variable named page in the template.

The DTOptionBuilder is used to define dtOptions. One nice feature here is that you can use a function that returns a promise object (in this case the queryAPI function) to populate the contents of the data table. As you can see in the code, queryAPI function uses the $http angular service to perform a GET against an API endpoint, then processes the result to produce a format suitable to render in an HTML UI. For each record in the result, we are counting the long-lived and short-lived connections, so that we can populate the columns the Long-lived Connections and Short-lived Connections as seen in the screenshot. Here is a sample of what gets returned from the API call:

{
    "count": 7869,
    "next": null,
    "previous": null,
    "results": [
        {
            "connections": "{\"8800\": {\"ESTABLISHED\": 16}}",
            "created": "2016-11-15T18:55:17.258502Z",
            "dst_datacenter": "dc1",
            "dst_group_name": "example-svc",
            "id": 1598,
            "src_datacenter": "dc1",
            "src_group_name": "sample-svc",
            "updated": "2016-11-15T18:55:17.258821Z"
        },... ]
}

The nested jQuery.each loops in the code process each item in the API response, and builds an array of “row data” objects that will be used to render the data table. The shapes of the columns are defined in the dtColumn variable. Besides .fromFnPromise, you can control other data table options such as display length or the menu of the available display lengths. Note that you can use .withOption method to set arbitrary options available on the original jQuery DataTable object. For example, the stateSave is a useful DataTables option that preserves the current sort order, pagination and filter state across site navigation.

The dtColumns is an array of column definitions defined by DTColumnBuilder objects. The code example shows how you can map the “row data” objects prepared inside the jQuery.each loops earlier to actual data table columns. You just need to pass the correct string to .newColumn() call to map the column to the desired JavaScript property. The .withTitle method allows you to set the column header, and .withClass lets you control the CSS class you want to apply to the column.

Finally, the .dtInstance gets set to an object that wraps the original jQuery data table object, so that you can perform some granular control. This code example does not need to interact with this object, but since you have a reference to the raw data table object, you could write code like the following (the following snippet is unrelated to what we have been looking at):

vm.updateRow = function(fix) {
  vm.dtInstance.DataTable.cell(
    /* row selector */ function(idx, data, node) { return (fix.id === data.id); },
    /* column-selector index */ 10).data(fix.value);
};

The example here illustrates the very basic implementation that involves angular-datatables to use jQuery DataTables in AngularJS code base. There is a rich set of API provided by angular-datatables and the underlying DataTables that enables much more advanced functionality. I hope that the coding pattern presented here can provide a good starting point for you.

Django REST Framework

Django is one of the popular web application frameworks that is also lightweight and user friendly. Django REST Framework is an extension that makes Django even easier to use when building REST API endpoints. Essentially, once you align the following components, you get the REST API endpoints very easily.

• Django model class: In Django, you can define your database schema using model classes. In my case, there is the NodeGroupDep class in api/models.py.
• Serializer class: This is required by the Django REST Framework. It defines how the Django model class will be transformed when serialized for or deserialized from API payload.
• ModelViewSet class: if you want to create a standard CRUD API against a model class, Django REST Framework makes it easy. You can define a corresponding “view set” which automatically generates a set of CRUD handlers.
• Routes: Once the three classes above are defined, you just need to hook the auto-generated handlers up with the HTTP request router, usually defined in your application’s urls.py.

Here are the sample code snippets from my application. In api/models.py, I have the NodeGroupDep model class as follows:

class NodeGroupDep(models.Model):
  src_group_name = models.CharField(max_length=255)
  src_datacenter = models.CharField(max_length=255)
  dst_group_name = models.CharField(max_length=255)
  dst_datacenter = models.CharField(max_length=255)
  connections    = models.TextField(null=True)
  updated     = models.DateTimeField(auto_now=True, db_index=True)
  created     = models.DateTimeField(default=timezone.now)

In api/serializers.py, I have NodeGroupDepSerializer as follows:

Import api.models as m
class NodeGroupDepSerializer(ser.ModelSerializer):
    class Meta:
        model = m.NodeGroupDep
        fields = '__all__'

In a serializer class, you can choose which fields from the table will be exposed via API. The __all__ string here is a special string that indicates that we will include all fields from the model in our API payloads.

In api/views.py, I have the corresponding view set called NodeGroupDepViewSet.

class NodeGroupDepViewSet(viewsets.ModelViewSet):
    queryset = m.NodeGroupDep.objects.all()
    serializer_class = s.NodeGroupDepSerializer
    filter_backends = (filters.DjangoFilterBackend,)
    filter_fields = ('src_datacenter', 'dst_datacenter',)

This is the class that creates the standard GET, PUT, POST and DELETE API endpoints for the given resource. Note that the filter_fields defines the query string parameters that you can use to filter the result set. In this case, you could add ?src_datacenter=dc1&dst_datacenter=dc2 to the GET API URL to only retrieve the cross-datacenter dependencies from dc1 to dc2.

In novascotia/urls.py, I define a router object that forwards HTTP requests to the right handlers defined by the view set object, as follows:

from rest_framework.routers import DefaultRouter
import api.views as rest
router = DefaultRouter()
router.register(r'nodegroupdeps', rest.NodeGroupDepViewSet)
# ... abbreviated ...
urlpatterns = [
    url(r'^admin/', admin.site.urls),
    url(r'^api/v1/', include(router.urls)), # <-- include the routes from DefaultRouter
    # ... abbreviated ...
]

That’s all you need to do. With the ~20 lines of code above you created GET, PUT, POST and DELETE REST API endpoints that allows you to read/write to a database table corresponding to the model class. As an added bonus, Django REST Framework gives you a self-documenting UI that allows you to interact with the API using HTML forms, as shown below.

Figure 2. Django REST Framework API UI

Conclusion

I found using DataTables to visualize and interact with a thousands of records to be very effective, and angular-datatables makes it easy to leverage the rich feature set of DataTables inside an AngularJS app (e.g. sorting by multiple columns). On the server-side, I opted to use Django REST framework, which gives you the REST API based on the Django Object-relational Mapping (ORM) layer almost for free. Once you package all these components together in a docker container, your software will be ready to deploy in production. I hope this combination can prove to be useful for your next project, or at least some information presented here could be used when working with one of the listed technologies.