February 25, 2024
  1. Video preprocessing, which encompasses any transformation utilized to the high-quality supply video previous to encoding. Video downscaling is essentially the most pertinent instance herein, which tailors our encoding to display resolutions of various gadgets and optimizes image high quality below various community situations. With video downscaling, a number of resolutions of a supply video are produced. For instance, a 4K supply video will likely be downscaled to 1080p, 720p, 540p and so forth. That is sometimes achieved by a standard resampling filter, like Lanczos.
  2. Video encoding utilizing a standard video codec, like AV1. Encoding drastically reduces the quantity of video information that must be streamed to your system, by leveraging spatial and temporal redundancies that exist in a video.
  • A realized method for downscaling can enhance video high quality and be tailor-made to Netflix content material.
  • It may be built-in as a drop-in resolution, i.e., we don’t want some other modifications on the Netflix encoding facet or the shopper system facet. Tens of millions of gadgets that assist Netflix streaming routinely profit from this resolution.
  • A definite, NN-based, video processing block can evolve independently, be used past video downscaling and be mixed with totally different codecs.
Structure of the deep downscaler mannequin, consisting of a preprocessing block adopted by a resizing block.
Left: Lanczos downscaling; proper: deep downscaler. Each movies are encoded with VP9 on the identical bitrate and have been upscaled to FHD decision (1920×1080). It’s possible you’ll have to zoom in to see the visible distinction.
  • The neural community structure was designed to be computationally environment friendly and likewise keep away from any unfavourable visible high quality impression. For instance, we discovered that only a few neural community layers have been enough for our wants. To scale back the enter channels even additional, we solely apply NN-based scaling on luma and scale chroma with a regular Lanczos filter.
  • We carried out the deep downscaler as an FFmpeg-based filter that runs along with different video transformations, like pixel format conversions. Our filter can run on each CPU and GPU. On a CPU, we leveraged oneDnn to additional scale back latency.
A top-down view of integrating the deep downscaler into Cosmos.