New Vega Charts in OmniSci Immerse

New chart architecture

If you have seen the OmniSci platform in action, you already know that OmniSciDB is fast. Crazy fast. In the Immerse development team, we are leveraging that speed to provide very fast filtering and visualization.

As a user, that speed comes in handy - you have questions and you don't usually have hours to wait for each step on your way to an answer. You need to quickly narrow down on your region of interest. Let's see how your assets are doing for the last quarter. What's this spike? It seems to happen in that region. Maybe it's caused by that factor. This process is so common: you need to explore your data at the speed of thought, and that means selecting data, filtering it and visualizing it fast. Crazy fast.

We have a very special trick to make our maps super fast. A description of your visualization is sent to a custom GPU rendering engine using the Vega schema definition. That's where the magic happens - your data layer comes back so you can interact with it on a map.

Our new chart architecture also uses this Vega schema. 🎉 I will explain later the reasons for this decision. But first, let's see what features this unlocks.

Improved cartesian charts

The first goal of using Vega in our new chart architecture was to improve stability, consistency and speed of delivering new features. And the first step to that end was to integrate all of our existing bar and line chart types into the new Vega chart. You will still be able to select a stacked bar, line, area, combo and all the chart types you use and love. (Everyone has a chart they ❤️️ right?) So same chart goodness, but under the hood, they now use a common Vega architecture.

The most immediate benefit of this integration was to share all features across all of the chart types. For example, we added cumulative and percentage mode to our bar chart, and now it's instantly available to line charts and other chart types as well. Same for stability improvements and performance tuning - improving one improves all.

The first milestone was to upgrade all of the cartesian charts. To validate that this new Vega architecture also helped us deliver new features faster, we sneaked in a few goodies: horizontal mode, better combinations of chart types, chart specific filters, better group by management, better custom SQL column editing, a more robust crossfiltering system, etc. Okay, we sneaked in more than a few ...

The history

Vega is an open source schema definition and rendering engine that quickly became a standard. It's a natural choice for us for multiple reasons.

Our core platform (OmniSciDB) as well as our libraries mapd-charting, crossfilter, and connector, are all open source. We know that the best way to build a product is to be part of an ecosystem. Vega was developed under the guidance of Jeffrey Heer who's also behind Protovis, D3, Altair, etc. We are proud to have him as an advisor and to share his vision of the power of communities, both users and developers.

There are also more technical reasons to choose Vega. When Immerse got started many years ago, one of the great chart libraries (also in the D3 ecosystem like Vega) was Dc.js. This was a great choice at the time because it was integrated with a nice cross-filtering engine aptly named Crossfilter.js. The first prototype was a simple JQuery file but it already validated our core value proposition: highly interactive filtering of big data.

Later we started using React, building custom charts, and expanding Immerse to tackle more complex use cases. After a few years, it became clear that we needed a more versatile interface that was more decoupled and more declarative, to match React's undirectional dataflow, and other patterns used by modern apps. Making the move to Vega fit those goals to a tee.

The future

Immerse 5.5 is our way to prove that we can still improve our charts further. But we're not done. Our next milestone is to convert more chart types, and to develop new features that will help you explore your data and make decisions even faster.

This new chart architecture will unlock some other very interesting projects. For example, users are discovering how fast we are even on commodity CPUs. So the same chart that we would traditionally generate on the GPU could be simply switched to CPU mode when a GPU is not available. Vega also comes with some interesting additional chart types out of the box. What if we can accelerate them all? And going beyond Immerse, we're excited to expand our integration with Jupyter Notebooks for Machine Learning and custom data flows as well.

Vega is a central piece of our new chart architecture, but it also enables us to build a better filtering and cross-filtering system, more consistent UI components, and to focus on delivering greater value to our users. Everyone loves us when we fix their favorite bugs, but our goal is bolder. We want people to stop caring about bugs, about architecture, even about data. We want our tools to feel so natural that these other concerns disappear. So forget about everything I just said. Just try Immerse! Upgrade, use the new charts, and start answering questions.