Professional video hardware and software usually uses 10 bits per component for color representation. Consumer products are usually 8-bit, and video distribution to consumers is universally 8 bits per component. Up until recently, GStreamer‘s support for 10- and 16-bit formats has been minimal, mostly enough to convert 10-bit input into an 8-bit format for further processing.
One of my side projects recently has been to improve much of the basic video infrastructure to handle 10-bit and also 16-bit color components. There are a few interesting 10-bit packing structures, mainly v210 and UYVP, which are YCbCr 4:2:2 formats used by QuickTime and SDI capture cards, and also r210, which is a 10-bit RGB format. Packed structures of this form are difficult to work with in SIMD code, so they tend to be inefficient for most processing. They are more interesting for interfacing with hardware and reading/writing to disk.
For more computational efficiency, and also to promote using a lot more precision instead of a little more precision, I’m encouraging people to use 4:4:4:4 16-bit AYUV or ARGB as an intermediate format for high color depth work instead of any of the 10 or 12 bit formats. Many GStreamer plugins already support 8-bit AYUV, so adding support for 16-bit AYUV is usually a pretty simple task.
Orc has made the task easier as well. I would not have started this task if it had involved writing a bunch of SIMD code directly, since deep-color processing is a specialized interest and I only have a finite amount of spare time.
Format negotiation among elements is similar to how it works with any other format. The 10 and 16 bit formats are simply extensions of how the existing GStreamer capabilities (caps) system works.
Some elements supporting 10 and 16-bit formats:
- videotestsrc – The video test signal generator. Of course, this is the first place to start for implementing any new formats. The patterns are all still fundamentally 8-bit, unfortunately.
- videoscale – The video scaling element.
- colorspace – This is the new format conversion and RGB/YCbCr matrixing element in gst-plugins-bad, which is based on Orc. At some point, it will be moved to -base and be used instead of ffmpegcolorspace.
- decklinksrc/sink and linsyssdisrc/sink – These SDI capture and playback elements handle 10-bit video in hardware, so it makes sense to deliver this to the GStreamer pipeline.
- schroenc/schrodec – Schrödinger has handled 10-bit video for several years, but only when handling VC-2 video (i.e., the intra subset of Dirac).
- qtmux/qtdemux – Handles a few 10 and 16 bit formats in the QuickTime container.
- pngdec – Decodes 16-bit PNG images.
The project to make Schrödinger handle high-precision video has been moving along as well. Although not ready for end users, it is possible to decode up to 26-bit component depth video. (26 happens to be a natural limitation. 10 and 16 work fine, too. I just said 26 to brag a little.)
Most of this work is done simply to have a high-precision pipeline available for test purposes, such as converting the 16-bit RGB Sintel PNGs into a clean high-bit-depth master for encoder testing. Hopefully soon, enough elements will have been updated to allow entire applications in 16-bit, for example, PiTiVi rendering.
