Building Custom MapD Viz: Tweetmap Example

It's nearing the end of my summer as an intern at MapD, and looking back, I've learned quite a few essential engineering skills and completed some impactful work. At the top of my list of projects is a redesign of the Tweetmap Demo. Tweetmap is a living visualization of Twitter data built on top of MapD database technology. There are a lot of gems hidden in this dataset, including the path of the 2017 solar eclipse.

The new Tweetmap is open source and serves as a guide for building your own custom visualizations powered by MapD. Luckily, with our open source front end libraries, it's a piece of cake. Building your own visualization means that you can make it look exactly the way you want, and incorporate custom charts not supported by MapD Immerse. Let's take a dive into the Tweetmap code base, and how you can build your own visualization.

How does Tweetmap work?

While you can build your visualization however you want, Tweetmap uses React and Webpack. The repo is organized by domain which means all the functionality behind each component sits in their own directory:

tweetmap/  
├── src/
│   ├── components/
│   │   ├── Legend/
│   │   │   ├── Legend.js
│   │   │   ├── actions.js
│   │   │   ├── reducer.js
│   │   │   ├── LegendItem/
│   │   ├── TweetSidebar/
│   │   │   ├── TweetSidebar.js
│   │   │   ├── actions.js
│   │   │   ├── reducer.js
...

You'll most likely be interested the actions.js file for each component. Chart setup and chart interactions live here and are written as thunks. The actions.js files are where Tweetmap interfaces with the open source MapD front-end libraries:

• MapD Connector
• MapD CrossFilter
• MapD Charting

MapD Connector

Connector is our library to facilitate connecting to your MapD server. You will need to configure the host, port, username and password on your connector instance before calling the connect() method. Tweetmap does this here. You can use the connection instance to directly make SQL queries to the database. The CrossFilter library also uses the connection instance.

MapD CrossFilter

CrossFilter enables the unique chart behavior you see in MapD Immerse. When you apply a filter to one chart in the dashboard, you instantly see the other charts update to reflect the new filter. The API for our CrossFilter is based on Square's Crossfilter. You can create a MapD CrossFilter instance by calling the crossfilter() method on the module, passing in your Connector instance and the name of the database table you would like to use. Tweetmap does this here. CrossFilter is your channel for slicing and dicing up your data. We do this by creating dimensions and groups. Think about dimensions as the table columns you want to work with and groups as what you would put in the GROUP BY section of an SQL query. For example, to get the data necessary for the legend in Tweetmap, you might write an SQL query like:

SELECT lang, COUNT(*) AS count FROM tweet_table GROUP BY lang ORDER BY count

In this case, lang is both the dimension and the group:

const dimension = crossfilter.dimension("lang")  
const group = dimension.group()  

Actions like aggregating and filtering are exposed as methods on these objects.

MapD Charting

Charting gives you the charts. This API is based on dc.js. For your charts to participate in the crossfilter behavior, their data needs to come from a dimension and group created by MapD CrossFilter. From there you'll need to configure some basic properties like the chart size and the HTML element the chart will be injected into. The best way to learn how to configure a chart from MapD Charting and integrate it into your visualization is by looking at the examples in the Charting repo.

Custom React Rendered Chart

In Tweetmap, the legend, top hashtags list and recent Tweets list are all custom charts that don't come from MapD Charting, but still participate in the crossfilter behavior. To achieve this in Tweetmap, we create a "dummy" chart with MapD Charting, and replace hooks for receiving data and rendering, with code that exposes the resulting data in a way React can take over. Let's look at the setup for the legend as an example:

dummyChart = dc.baseMixin({})  
dummyChart.dimension(dimension)  
dummyChart.group(group)  
dummyChart.minWidth(0)  
dummyChart.minHeight(0)  
dummyChart._doRender = dummyChart._doRedraw = () => {}

dummyChart.setDataAsync((_group, callback) => {  
 group.reduceCount("*").topAsync(LIMIT).then(
   results => {

     // results is the list of data points to render !

     callback()
   }
 )
})

dummyChart.anchor("#legendDummy")  

MapD Charting charts use a mixin architecture. Our strategy here is to create the most minimal chart possible with dc.baseMixin. We'll need to pass in the dimension and group created by CrossFilter. Since the legend is rendered with React, we don't want this "dummy" MapD Charting object to inject anything into HTML. We set the width and height to zero, set the render and redraw methods to no-ops and choose a dummy HTML element as the chart parent. It's important to use the dummy element because anchor() will register the chart with the dc chart registry, which is necessary for filters to apply across charts. We're also interested in setDataAsync(), which sets the handler for changes in filters. Here we can write a handler that completes our query for top languages and gets access to the resulting data.

Conclusion

We've covered the idea behind using each of MapD's open source libraries. Take a deeper look with examples using MapD Charting or by exploring the Tweetmap repo. Now you can go on to making your own visualizations using MapD technology!