What's New in HEAVY.AI 5.8?

We are incredibly excited to announce HEAVY.AI Version 5.8, a release that ushers in foundational changes to HEAVY.AI's rendering engine, introduces new database functionality, geospatial operators, and system administration tools.

Some of our long-time users may be asking themselves, "Didn't HEAVY.AI just deliver Version 5.7 ?" To that, the answer is a resounding "Yes!" 

In fact, under the leadership of Pey Silvester, our talented engineering team has committed to a tighter, more consistent release schedule that our customers, partners, and internal staff are already benefiting from.

The remainder of this post will highlight key features of the HEAVY.AI Version 5.8 release and provide some examples of how you can leverage them.
‍

Rendering

In HEAVY.AI Version 5.7, we launched the official beta of the new Vulkan-backed renderer to replace the current OpenGL renderer. Vulkan offers advantages over its predecessor in the form of:

• lower overhead
• more direct control over the GPU
• lower CPU usage
• cross-platform support 
• better support for multithreaded processing
  

Version 5.8 officially enables the new Vulkan backend renderer by default, replacing OpenGL, which is now the fallback. Still, OpenGL will be deprecated and removed in subsequent releases.

In addition to Vulkan, we've improved the memory footprint and performance of the multi-GPU compositor. Simply put, the more GPUs on a node, the more significant potential gains.
‍

Database

Query Engine

Version 5.7 introduced parallel executors into public beta, allowing queries to proceed asynchronously in parallel and improve overall concurrency. Parallel executors are now on by default using the --num-executors=2 flag in your configuration file. For example:
‍

Experienced SQL developers noticed, in previous versions, that our window functions could only be applied to certain tables and required a partition clause. Now they operate over tables with multiple fragments and shards and no longer need a partition clause to be specified.

Lastly, we've enabled support for the WIDTH_BUCKET operator to make numeric binning easier and enhance histogram performance in Immerse.
‍

Storage I/O

HEAVY.AI Version 5.8 includes file path regex globbing, filtering, and sorting when running the COPY FROM command, a significant update for those that commonly load data with SQL.

For example, let's assume you have the following folder and file structure:

You can glob local parquet or CSV files by specifying either a pathname with a wildcard or a folder name. Globbing a folder recursively returns all files under the specified folder, while globbing with a wildcard returns any file paths matching the expanded file path.
‍

File filtering extends your ability to remove unwanted files that have been globbed. To use filtering, specify the REGEX_PATH_FILTER option and remember files that do not match this pattern are omitted.
‍

Sorting rounds out our new file path regex capabilities by allowing you to specify the order in which files are imported with options like:

• pathname (default)
• date_modified
• regex *
• regex_date *
• regex_number *
  ‍

Lastly, we've added more robust handling of long decimal strings during import and support for date types in ArrowCSVForeignStorage.
‍

Geospatial

HEAVY.AI now natively supports ST_Transform to/from all UTM zones and EPSG:4326 (Lon/Lat) and EPSG:900913 (Web Mercator). These improvements make importing, manipulating, and analyzing geospatial data stored in niche or regional coordinate systems easier.

Version 5.8 continues our commitment to bolstering our supported geospatial operators with:

• ST_BUFFER
• ST_DIFFERENCE
• ST_UNION
• ST_Equals
  

Finally, Spatial joins between point types using ST_Distance are now up to 100x faster, and ST_DWithin runs with GEOGRAPHY inputs for geodesic distance calculations. 
‍

System Administration

Understanding what queries are running and interrupting unwanted jobs can be handy when testing and optimizing large queries in HEAVY.AI. 

In previous versions, the ability to view and interrupt queries was only available to superusers. Now non-superusers can do the same with:
‍

At Version 5.8, superusers can clear CPU, GPU, or Render memory using the new ALTER SYSTEM CLEAR commands. The server handles memory management, and you typically don't need to use this command. 

But if superusers have unexpected memory issues, they can try clearing the memory to see if performance improves with:
‍

You can download the latest release of HEAVY.AI today!

As always, we'd like to thank our phenomenal engineering team for producing a great release. If you wish to review the release notes in their entirety, visit https://docs.omnisci.com/overview/release-notes. 

For those planning to upgrade to HEAVY.AI Version 5.8, we recommend you backup your data directories and follow the upgrading OmniSci documentation. Enterprise customers can reach out to HEAVY.AI Support for assistance if necessary. 

Please ask any of your remaining questions or share your experiences with us on  LinkedIn,  Twitter, or our Community Forums!