April 19, 2024
  • We’ve partnered with Voltron Data and the Arrow neighborhood to align and converge Apache Arrow with Velox, Meta’s open supply execution engine.
  • Apache Arrow 15 contains three new format layouts developed by this partnership: StringView, ListView, and Run-Finish-Encoding (REE).
  • This new convergence helps Meta and the bigger neighborhood construct knowledge administration programs which might be unified, extra environment friendly, and composable.

Meta’s Knowledge Infrastructure groups have been rethinking how data management systems are designed. We need to make our data management systems more composable – which means that as a substitute of individually growing programs as monoliths we determine frequent elements, issue them out as reusable libraries, and leverage frequent APIs and requirements to extend the interoperability between them. 

As we decompose our massive, monolithic programs right into a extra modular stack of reusable elements, open requirements, resembling Apache Arrow, play an essential function for interoperability of those elements. To additional our efforts in making a extra unified knowledge panorama for our programs in addition to these within the bigger neighborhood, we’ve partnered with Voltron Knowledge and the Arrow neighborhood to converge Apache Arrow’s open supply columnar layouts with Velox, Meta’s open supply execution engine.

The consequence combines the effectivity and agility provided by Velox with the widely-used Apache customary.  

Why we’d like a composable knowledge administration system

Meta’s knowledge engines assist large-scale workloads that embrace processing massive datasets offline (ETL), interactive dashboard era, advert hoc knowledge exploration, and stream processing. Extra not too long ago, a wide range of characteristic engineering, knowledge preprocessing, and coaching programs have been constructed to assist our quickly increasing AI/ML infrastructure. To make sure our engineering groups can effectively preserve and improve these engines as our merchandise evolve, Meta has began a collection of tasks aimed toward growing our engineering effectivity by minimizing the duplication of labor, enhancing the expertise of inner knowledge customers by extra constant semantics throughout these engines, and, finally, accelerating the tempo of innovation in knowledge administration. 

An introduction to Velox

Velox is the primary challenge in our composable knowledge administration system program. It’s a unified execution engine, carried out as a C++ library, aimed toward changing the very processing core of many of those knowledge administration programs – their execution engine.

Velox improves the effectivity of those programs by offering a unified, state-of-the-art implementation of options and optimizations that have been beforehand solely accessible in particular person engines. It additionally improves the engineering effectivity of our group since these options can now be written as soon as, in a single library, and be (re-)used in every single place.

Velox is at the moment in several levels of integration in additional than 10 of Meta’s knowledge programs. Now we have noticed 3-10x efficiency improvements in integrations with well-known programs within the business like Apache Spark and Presto. 

We open-sourced Velox in 2022. As we speak, it’s developed in collaboration with greater than 200 particular person contributors world wide from greater than 20 corporations. 

Open requirements and Apache Arrow

To be able to allow interoperability with different elements, a composable knowledge administration system has to grasp frequent storage (file) codecs, community serialization protocols, desk APIs, and have a unified manner of expressing computation. Oftentimes these elements should instantly share in-memory datasets with one another, for instance, when transferring knowledge throughout language boundaries (C++ to Java or Python) for environment friendly UDF assist.

Our focus is to make use of open requirements in these APIs as usually as doable. Apache Arrow is an open supply in-memory structure customary for columnar knowledge that has been extensively adopted within the business. In a manner, Arrow may be seen because the layer beneath Velox: Arrow describes how columnar knowledge is represented in reminiscence; Velox supplies a collection of execution and useful resource administration primitives to course of this knowledge.

Though the Arrow format predates Velox, we made a acutely aware design determination whereas creating Velox to increase and deviate from the Arrow format, making a structure we name Velox Vectors. The aim was to speed up the info processing operations generally present in our workloads in ways in which weren’t doable utilizing Arrow. Velox Vectors supplied the effectivity and agility we have to transfer quick, however in return created a fragmented area with restricted element interoperability. 

To bridge this hole and create a extra unified knowledge panorama for our programs and the neighborhood, we partnered with Voltron Knowledge and the Arrow neighborhood to align and converge these two codecs. After a 12 months of labor, the brand new Apache Arrow launch, Apache Arrow 15.0.0, contains three new format layouts impressed by Velox Vectors: StringView, ListView, and Run-Finish-Encoding (REE).

Arrow 15 not solely permits environment friendly (zero-copy) in-memory communication throughout elements utilizing Velox and Arrow, but additionally will increase Arrow’s applicability in fashionable execution engines, unlocking a wide range of use circumstances throughout the business. 

Particulars of the Arrow and Velox structure

Each Arrow and Velox Vectors are columnar layouts whose objective is to characterize batches of information in reminiscence. A column is often composed of a sequential buffer the place row values are saved contiguously and an optionally available bitmask to characterize the nullability/validity of every worth: 

(a) Logical and (b) bodily illustration of an instance dataset.

The Arrow and Velox Vectors codecs already had suitable structure representations for scalar fixed-size knowledge varieties (resembling integers, floats, and booleans) and dictionary-encoded knowledge. Nonetheless, there have been incompatibilities in string illustration and container varieties resembling arrays and maps, and an absence of assist for fixed and run-length-encoded (RLE) knowledge.

StringView – strings

Arrow’s typical string illustration makes use of the variable-sized element layout, which consists of 1 contiguous buffer containing the string contents (the info), and one buffer marking the place every string begins (the offsets). The dimensions of a string i may be obtained by subtracting offsets[i+1] by offsets[i]. That is equal to representing strings as an array of characters: 

Arrow unique string illustration.

Whereas Arrow’s illustration stands out in simplicity, we discovered by a collection of experiments that the next alternate string illustration (which is now known as StringView) supplies compelling properties which might be essential for environment friendly string processing: 

New StringView illustration in Arrow 15.

Within the new representation, the primary 4 bytes of the view object all the time include the string measurement. If the string is brief (as much as 12 characters), the contents are saved inline within the view construction. In any other case, a prefix of the string is saved within the subsequent 4 bytes, adopted by the buffer ID (StringViews can include a number of knowledge buffers) and the offset in that knowledge buffer.

The advantages of this structure are:

  1. Small strings of as much as 12 bytes are absolutely inlined throughout the views buffer and may be learn with out dereferencing the info buffer. This will increase reminiscence locality as the standard cache miss of accessing the info buffer is prevented, growing efficiency.
  2. Since StringViews retailer a small (4 bytes) prefix with the view object, string comparisons can fail-fast and, in lots of circumstances, keep away from accessing the info buffer. This property hastens frequent operations resembling extremely selective filters and sorting.
  3. StringView provides builders extra flexibility on how string knowledge is specified by reminiscence. For instance, it permits for sure frequent string operations, resembling 𝑡𝑟𝑖𝑚() and 𝑠𝑢𝑏𝑠𝑡𝑟(), to be executed zero-copy by solely updating the view object.
  4. Since StringView’s view object has a set measurement (16 bytes), StringViews may be written out of order (e.g., first writing StringView at place 2, then 0 and 1). 

Moreover these properties, now we have discovered that different fashionable processing engines and libraries like Umbra and DuckDB observe an identical string illustration method, and, consequently, additionally used to deviate from Arrow. In Arrow 15, StringView has been added as a supported structure and may now be used to effectively switch string batches throughout these programs.

ListView – variable-sized containers

Variable-size containers like arrays and maps are represented in Arrow utilizing one buffer containing the flattened components from all rows, and one offsets buffer marking the place the container on every row begins, just like the unique string illustration. The variety of components a container on row i shops may be obtained by subtracting offsets[i+1] by offsets[i]: 

Arrow unique record illustration.

To effectively assist execution of vectorized conditionals (e.g., IF and SWITCH operations), the Velox Vectors structure has to permit builders to write down columns out of order. Which means that builders can, for instance, first write all even row information then all odd row information with out having to reorganize components which have already been written.

Primitive varieties can all the time be written out of order for the reason that aspect measurement is fixed and identified beforehand. Likewise, strings may also be written out of order utilizing StringView as a result of the string metadata objects have a continuing measurement (16 bytes), and string contents don’t should be written contiguously. To extend flexibility and assist out-of-order writes for the remaining variable-sized varieties in Velox, we determined to maintain each lengths and offsets buffers:

New ListView illustration in Arrow 15.

To bridge the hole, a brand new format referred to as ListView has been added to Arrow 15. It permits the illustration of variable-sized components which have each lengths and offsets buffers.

Past permitting for environment friendly execution of conditionals, ListView provides builders extra flexibility to slice and rearrange containers (e.g., operations like slice() and trim_array() may be carried out zero-copy), aside from permitting for containers with overlapping ranges of components.

REE – extra encodings

Now we have additionally added two extra encoding codecs generally present in knowledge warehouse workloads into Velox: fixed encoding, to characterize that each one values in a column are the identical, sometimes used to characterize literals and partition keys; and RLE, to compactly characterize consecutive runs of the identical aspect.

Upon dialogue with the neighborhood, it was determined so as to add the REE format to Arrow. The REE format is a slight variation of RLE that, as a substitute of storing the lengths of every run, shops the offset wherein every run ends, offering higher random-access assist. With REEs additionally it is doable to characterize fixed encoded values by encoding them as a single run whose measurement is all the batch.

Composability is the way forward for knowledge administration

Converging Arrow and Velox’s reminiscence structure is a crucial step in the direction of making knowledge administration programs extra composable. It permits programs to mix the facility of Velox’s state-of-the-art execution with the widespread business adoption of Arrow’s customary, leading to a extra environment friendly and seamless cooperation. The brand new extensions are already seeing adoption in libraries like PyArrow and Polars and inside Meta. Sooner or later, it would permit extra environment friendly interaction between tasks like Apache Gluten (which makes use of Velox internally) and PySpark (which consumes Arrow), for instance.

We envision that fragmentation and duplication of labor may be lowered by decomposing knowledge programs into reusable elements that are open supply and constructed based mostly on open requirements and APIs. In the end, we hope this work will assist present the inspiration required to speed up the tempo of innovation in knowledge administration.


This format alignment was solely doable as a result of a broad collaboration throughout completely different teams. A particular thanks to Masha Basmanova, Orri Erling, Xiaoxuan Meng, Krishna Pai, Jimmy Lu, Kevin Wilfong, Laith Sakka, Wei He, Bikramjeet Vig, and Sridhar Anumandla from the Velox crew at Meta; Felipe Carvalho, Ben Kietzman, Jacob Wujciak-Jens, Srikanth Nadukudy, Wes McKinney, and Keith Kraus from Voltron Knowledge; and all the Apache Arrow neighborhood for the insightful discussions, suggestions, and receptivity to new concepts.