April 19, 2024

David J. Berg, Romain Cledat, Kayla Seeley, Shashank Srikanth, Chaoying Wang, Darin Yu

Netflix makes use of information science and machine studying throughout all sides of the corporate, powering a variety of enterprise functions from our inner infrastructure and content material demand modeling to media understanding. The Machine Studying Platform (MLP) workforce at Netflix offers a complete ecosystem of instruments round Metaflow, an open supply machine studying infrastructure framework we began, to empower information scientists and machine studying practitioners to construct and handle a wide range of ML programs.

Since its inception, Metaflow has been designed to supply a human-friendly API for constructing information and ML (and immediately AI) functions and deploying them in our manufacturing infrastructure frictionlessly. Whereas human-friendly APIs are pleasant, it’s actually the integrations to our manufacturing programs that give Metaflow its superpowers. With out these integrations, initiatives can be caught on the prototyping stage, or they must be maintained as outliers exterior the programs maintained by our engineering groups, incurring unsustainable operational overhead.

Given the very various set of ML and AI use circumstances we assist — immediately we’ve got tons of of Metaflow initiatives deployed internally — we don’t count on all initiatives to observe the identical path from prototype to manufacturing. As a substitute, we offer a strong foundational layer with integrations to our company-wide information, compute, and orchestration platform, in addition to numerous paths to deploy functions to manufacturing easily. On prime of this, groups have constructed their very own domain-specific libraries to assist their particular use circumstances and wishes.

On this article, we cowl just a few key integrations that we offer for numerous layers of the Metaflow stack at Netflix, as illustrated above. We can even showcase real-life ML initiatives that depend on them, to offer an concept of the breadth of initiatives we assist. Notice that every one initiatives leverage a number of integrations, however we spotlight them within the context of the combination that they use most prominently. Importantly, all of the use circumstances have been engineered by practitioners themselves.

These integrations are applied by Metaflow’s extension mechanism which is publicly obtainable however topic to vary, and therefore not part of Metaflow’s secure API but. In case you are interested in implementing your personal extensions, get in contact with us on the Metaflow community Slack.

Let’s go over the stack layer by layer, beginning with probably the most foundational integrations.

Our essential information lake is hosted on S3, organized as Apache Iceberg tables. For ETL and different heavy lifting of information, we primarily depend on Apache Spark. Along with Spark, we need to assist last-mile information processing in Python, addressing use circumstances reminiscent of characteristic transformations, batch inference, and coaching. Sometimes, these use circumstances contain terabytes of information, so we’ve got to concentrate to efficiency.

To allow quick, scalable, and sturdy entry to the Netflix information warehouse, we’ve got developed a Quick Knowledge library for Metaflow, which leverages high-performance elements from the Python information ecosystem:

As depicted within the diagram, the Quick Knowledge library consists of two essential interfaces:

  • The Desk object is liable for interacting with the Netflix information warehouse which incorporates parsing Iceberg (or legacy Hive) desk metadata, resolving partitions and Parquet information for studying. Lately, we added assist for the write path, so tables might be up to date as effectively utilizing the library.
  • As soon as we’ve got found the Parquet information to be processed, MetaflowDataFrame takes over: it downloads information utilizing Metaflow’s high-throughput S3 shopper on to the method’ reminiscence, which often outperforms reading of local files.

We use Apache Arrow to decode Parquet and to host an in-memory illustration of information. The consumer can select probably the most appropriate device for manipulating information, reminiscent of Pandas or Polars to make use of a dataframe API, or one in all our inner C++ libraries for numerous high-performance operations. Because of Arrow, information might be accessed by these libraries in a zero-copy vogue.

We additionally take note of dependency points: (Py)Arrow is a dependency of many ML and information libraries, so we don’t need our customized C++ extensions to rely on a particular model of Arrow, which might simply result in unresolvable dependency graphs. As a substitute, within the type of nanoarrow, our Quick Knowledge library solely depends on the stable Arrow C data interface, producing a hermetically sealed library with no exterior dependencies.

Instance use case: Content material Information Graph

Our data graph of the leisure world encodes relationships between titles, actors and different attributes of a movie or collection, supporting all facets of enterprise at Netflix.

A key problem in making a data graph is entity decision. There could also be many alternative representations of barely totally different or conflicting details about a title which have to be resolved. That is usually finished by a pairwise matching process for every entity which turns into non-trivial to do at scale.

This challenge leverages Quick Knowledge and horizontal scaling with Metaflow’s foreach construct to load massive quantities of title data — roughly a billion pairs — saved within the Netflix Knowledge Warehouse, so the pairs might be matched in parallel throughout many Metaflow duties.

We use metaflow.Desk to resolve all enter shards that are distributed to Metaflow duties that are liable for processing terabytes of information collectively. Every process hundreds the information utilizing metaflow.MetaflowDataFrame, performs matching utilizing Pandas, and populates a corresponding shard in an output Desk. Lastly, when all matching is finished and information is written the brand new desk is dedicated so it may be learn by different jobs.

Whereas open-source customers of Metaflow depend on AWS Batch or Kubernetes as the compute backend, we depend on our centralized compute-platform, Titus. Underneath the hood, Titus is powered by Kubernetes, nevertheless it offers a thick layer of enhancements over off-the-shelf Kubernetes, to make it extra observable, safe, scalable, and cost-efficient.

By focusing on @titus, Metaflow duties profit from these battle-hardened options out of the field, with no in-depth technical data or engineering required from the ML engineers or information scientist finish. Nonetheless, to be able to profit from scalable compute, we have to assist the developer to package deal and rehydrate the entire execution atmosphere of a challenge in a distant pod in a reproducible method (ideally shortly). Particularly, we don’t need to ask builders to handle Docker pictures of their very own manually, which shortly ends in extra issues than it solves.

This is the reason Metaflow provides support for dependency management out of the field. Initially, we supported solely @conda, however based mostly on our work on Portable Execution Environments, open-source Metaflow gained support for @pypi just a few months in the past as effectively.

Instance use case: Constructing mannequin explainers

Right here’s an enchanting instance of the usefulness of transportable execution environments. For a lot of of our functions, mannequin explainability issues. Stakeholders like to know why fashions produce a sure output and why their conduct adjustments over time.

There are a number of methods to supply explainability to fashions however a method is to coach an explainer mannequin based mostly on every skilled mannequin. With out going into the main points of how that is finished precisely, suffice to say that Netflix trains numerous fashions, so we have to practice numerous explainers too.

Because of Metaflow, we are able to permit every utility to decide on one of the best modeling method for his or her use circumstances. Correspondingly, every utility brings its personal bespoke set of dependencies. Coaching an explainer mannequin subsequently requires:

  1. Entry to the unique mannequin and its coaching atmosphere, and
  2. Dependencies particular to constructing the explainer mannequin.

This poses an fascinating problem in dependency administration: we’d like a higher-order coaching system, “Explainer circulation” within the determine under, which is ready to take a full execution atmosphere of one other coaching system as an enter and produce a mannequin based mostly on it.

Explainer circulation is event-triggered by an upstream circulation, such Mannequin A, B, C flows within the illustration. The build_environment step makes use of the metaflow atmosphere command offered by our portable environments, to construct an atmosphere that features each the necessities of the enter mannequin in addition to these wanted to construct the explainer mannequin itself.

The constructed atmosphere is given a singular identify that is determined by the run identifier (to supply uniqueness) in addition to the mannequin kind. Given this atmosphere, the train_explainer step is then capable of discuss with this uniquely named atmosphere and function in an atmosphere that may each entry the enter mannequin in addition to practice the explainer mannequin. Notice that, not like in typical flows utilizing vanilla @conda or @pypi, the transportable environments extension permits customers to additionally fetch these environments immediately at execution time versus at deploy time which subsequently permits customers to, as on this case, resolve the atmosphere proper earlier than utilizing it within the subsequent step.

If information is the gas of ML and the compute layer is the muscle, then the nerves have to be the orchestration layer. Now we have talked concerning the significance of a production-grade workflow orchestrator within the context of Metaflow once we launched assist for AWS Step Capabilities years in the past. Since then, open-source Metaflow has gained assist for Argo Workflows, a Kubernetes-native orchestrator, in addition to support for Airflow which remains to be extensively utilized by information engineering groups.

Internally, we use a manufacturing workflow orchestrator known as Maestro. The Maestro submit shares particulars about how the system helps scalability, high-availability, and usefulness, which offer the spine for all of our Metaflow initiatives in manufacturing.

A massively vital element that usually goes missed is event-triggering: it permits a workforce to combine their Metaflow flows to surrounding programs upstream (e.g. ETL workflows), in addition to downstream (e.g. flows managed by different groups), utilizing a protocol shared by the entire group, as exemplified by the instance use case under.

Instance use case: Content material determination making

One of the vital business-critical programs working on Metaflow helps our content material determination making, that’s, the query of what content material Netflix ought to deliver to the service. We assist an enormous scale of over 260M subscribers spanning over 190 nations representing massively various cultures and tastes, all of whom we need to delight with our content material slate. Reflecting the breadth and depth of the problem, the programs and fashions specializing in the query have grown to be very subtle.

We method the query from a number of angles however we’ve got a core set of information pipelines and fashions that present a basis for determination making. As an example the complexity of simply the core elements, take into account this high-level diagram:

On this diagram, grey packing containers characterize integrations to accomplice groups downstream and upstream, inexperienced packing containers are numerous ETL pipelines, and blue packing containers are Metaflow flows. These packing containers encapsulate tons of of superior fashions and complex enterprise logic, dealing with large quantities of information every day.

Regardless of its complexity, the system is managed by a comparatively small workforce of engineers and information scientists autonomously. That is made potential by just a few key options of Metaflow:

The workforce has additionally developed their very own domain-specific libraries and configuration administration instruments, which assist them enhance and function the system.

To provide enterprise worth, all our Metaflow initiatives are deployed to work with different manufacturing programs. In lots of circumstances, the combination is likely to be through shared tables in our information warehouse. In different circumstances, it’s extra handy to share the outcomes through a low-latency API.

Notably, not all API-based deployments require real-time analysis, which we cowl within the part under. Now we have numerous business-critical functions the place some or all predictions might be precomputed, guaranteeing the bottom potential latency and operationally easy excessive availability on the international scale.

Now we have developed an formally supported sample to cowl such use circumstances. Whereas the system depends on our inner caching infrastructure, you could possibly observe the identical sample utilizing companies like Amazon ElasticCache or DynamoDB.

Instance use case: Content material efficiency visualization

The historic efficiency of titles is utilized by determination makers to know and enhance the movie and collection catalog. Efficiency metrics might be advanced and are sometimes finest understood by people with visualizations that break down the metrics throughout parameters of curiosity interactively. Content material determination makers are geared up with self-serve visualizations by a real-time net utility constructed with metaflow.Cache, which is accessed by an API supplied with metaflow.Internet hosting.

A every day scheduled Metaflow job computes mixture portions of curiosity in parallel. The job writes a big quantity of outcomes to an internet key-value retailer utilizing metaflow.Cache. A Streamlit app homes the visualization software program and information aggregation logic. Customers can dynamically change parameters of the visualization utility and in real-time a message is distributed to a easy Metaflow hosting service which seems up values within the cache, performs computation, and returns the outcomes as a JSON blob to the Streamlit utility.

For deployments that require an API and real-time analysis, we offer an built-in mannequin internet hosting service, Metaflow Internet hosting. Though particulars have developed rather a lot, this old talk still gives a good overview of the service.

Metaflow Internet hosting is particularly geared in direction of internet hosting artifacts or fashions produced in Metaflow. This offers a straightforward to make use of interface on prime of Netflix’s present microservice infrastructure, permitting information scientists to shortly transfer their work from experimentation to a manufacturing grade net service that may be consumed over a HTTP REST API with minimal overhead.

Its key advantages embrace:

  • Easy decorator syntax to create RESTFull endpoints.
  • The back-end auto-scales the variety of cases used to again your service based mostly on site visitors.
  • The back-end will scale-to-zero if no requests are made to it after a specified period of time thereby saving price notably in case your service requires GPUs to successfully produce a response.
  • Request logging, alerts, monitoring and tracing hooks to Netflix infrastructure

Take into account the service much like managed mannequin internet hosting companies like AWS Sagemaker Model Hosting, however tightly built-in with our microservice infrastructure.

Instance use case: Media

Now we have a protracted historical past of utilizing machine studying to course of media belongings, as an example, to personalize paintings and to assist our creatives create promotional content material effectively. Processing massive quantities of media belongings is technically non-trivial and computationally costly, so through the years, we’ve got developed loads of specialised infrastructure devoted for this objective normally, and infrastructure supporting media ML use circumstances specifically.

To exhibit the advantages of Metaflow Internet hosting that gives a general-purpose API layer supporting each synchronous and asynchronous queries, take into account this use case involving Amber, our characteristic retailer for media.

Whereas Amber is a characteristic retailer, precomputing and storing all media options prematurely can be infeasible. As a substitute, we compute and cache options in an on-demand foundation, as depicted under:

When a service requests a characteristic from Amber, it computes the characteristic dependency graph after which sends a number of asynchronous requests to Metaflow Internet hosting, which locations the requests in a queue, ultimately triggering characteristic computations when compute sources turn into obtainable. Metaflow Internet hosting caches the response, so Amber can fetch it after some time. We might have constructed a devoted microservice only for this use case, however due to the pliability of Metaflow Internet hosting, we have been capable of ship the characteristic quicker with no further operational burden.

Our urge for food to use ML in various use circumstances is simply rising, so our Metaflow platform will hold increasing its footprint correspondingly and proceed to supply pleasant integrations to programs constructed by different groups at Netlfix. As an example, we’ve got plans to work on enhancements within the versioning layer, which wasn’t coated by this text, by giving extra choices for artifact and mannequin administration.

We additionally plan on constructing extra integrations with different programs which are being developed by sister groups at Netflix. For instance, Metaflow Internet hosting fashions are at present not effectively built-in into mannequin logging amenities — we plan on engaged on bettering this to make fashions developed with Metaflow extra built-in with the suggestions loop essential in coaching new fashions. We hope to do that in a pluggable method that will permit different customers to combine with their very own logging programs.

Moreover we need to provide extra methods Metaflow artifacts and fashions might be built-in into non-Metaflow environments and functions, e.g. JVM based mostly edge service, in order that Python-based information scientists can contribute to non-Python engineering programs simply. This may permit us to raised bridge the hole between the fast iteration that Metaflow offers (in Python) with the necessities and constraints imposed by the infrastructure serving Netflix member going through requests.

In case you are constructing business-critical ML or AI programs in your group, join the Metaflow Slack community! We’re blissful to share experiences, reply any questions, and welcome you to contribute to Metaflow.


Because of Wenbing Bai, Jan Florjanczyk, Michael Li, Aliki Mavromoustaki, and Sejal Rai for assist with use circumstances and figures. Because of our OSS contributors for making Metaflow a greater product.