<title>Sub-device Interface</title> | |
<note> | |
<title>Experimental</title> | |
<para>This is an <link linkend="experimental">experimental</link> | |
interface and may change in the future.</para> | |
</note> | |
<para>The complex nature of V4L2 devices, where hardware is often made of | |
several integrated circuits that need to interact with each other in a | |
controlled way, leads to complex V4L2 drivers. The drivers usually reflect | |
the hardware model in software, and model the different hardware components | |
as software blocks called sub-devices.</para> | |
<para>V4L2 sub-devices are usually kernel-only objects. If the V4L2 driver | |
implements the media device API, they will automatically inherit from media | |
entities. Applications will be able to enumerate the sub-devices and discover | |
the hardware topology using the media entities, pads and links enumeration | |
API.</para> | |
<para>In addition to make sub-devices discoverable, drivers can also choose | |
to make them directly configurable by applications. When both the sub-device | |
driver and the V4L2 device driver support this, sub-devices will feature a | |
character device node on which ioctls can be called to | |
<itemizedlist> | |
<listitem><para>query, read and write sub-devices controls</para></listitem> | |
<listitem><para>subscribe and unsubscribe to events and retrieve them</para></listitem> | |
<listitem><para>negotiate image formats on individual pads</para></listitem> | |
</itemizedlist> | |
</para> | |
<para>Sub-device character device nodes, conventionally named | |
<filename>/dev/v4l-subdev*</filename>, use major number 81.</para> | |
<section> | |
<title>Controls</title> | |
<para>Most V4L2 controls are implemented by sub-device hardware. Drivers | |
usually merge all controls and expose them through video device nodes. | |
Applications can control all sub-devices through a single interface.</para> | |
<para>Complex devices sometimes implement the same control in different | |
pieces of hardware. This situation is common in embedded platforms, where | |
both sensors and image processing hardware implement identical functions, | |
such as contrast adjustment, white balance or faulty pixels correction. As | |
the V4L2 controls API doesn't support several identical controls in a single | |
device, all but one of the identical controls are hidden.</para> | |
<para>Applications can access those hidden controls through the sub-device | |
node with the V4L2 control API described in <xref linkend="control" />. The | |
ioctls behave identically as when issued on V4L2 device nodes, with the | |
exception that they deal only with controls implemented in the sub-device. | |
</para> | |
<para>Depending on the driver, those controls might also be exposed through | |
one (or several) V4L2 device nodes.</para> | |
</section> | |
<section> | |
<title>Events</title> | |
<para>V4L2 sub-devices can notify applications of events as described in | |
<xref linkend="event" />. The API behaves identically as when used on V4L2 | |
device nodes, with the exception that it only deals with events generated by | |
the sub-device. Depending on the driver, those events might also be reported | |
on one (or several) V4L2 device nodes.</para> | |
</section> | |
<section id="pad-level-formats"> | |
<title>Pad-level Formats</title> | |
<warning><para>Pad-level formats are only applicable to very complex device that | |
need to expose low-level format configuration to user space. Generic V4L2 | |
applications do <emphasis>not</emphasis> need to use the API described in | |
this section.</para></warning> | |
<note><para>For the purpose of this section, the term | |
<wordasword>format</wordasword> means the combination of media bus data | |
format, frame width and frame height.</para></note> | |
<para>Image formats are typically negotiated on video capture and output | |
devices using the <link linkend="crop">cropping and scaling</link> ioctls. | |
The driver is responsible for configuring every block in the video pipeline | |
according to the requested format at the pipeline input and/or | |
output.</para> | |
<para>For complex devices, such as often found in embedded systems, | |
identical image sizes at the output of a pipeline can be achieved using | |
different hardware configurations. One such example is shown on | |
<xref linkend="pipeline-scaling" />, where | |
image scaling can be performed on both the video sensor and the host image | |
processing hardware.</para> | |
<figure id="pipeline-scaling"> | |
<title>Image Format Negotiation on Pipelines</title> | |
<mediaobject> | |
<imageobject> | |
<imagedata fileref="pipeline.pdf" format="PS" /> | |
</imageobject> | |
<imageobject> | |
<imagedata fileref="pipeline.png" format="PNG" /> | |
</imageobject> | |
<textobject> | |
<phrase>High quality and high speed pipeline configuration</phrase> | |
</textobject> | |
</mediaobject> | |
</figure> | |
<para>The sensor scaler is usually of less quality than the host scaler, but | |
scaling on the sensor is required to achieve higher frame rates. Depending | |
on the use case (quality vs. speed), the pipeline must be configured | |
differently. Applications need to configure the formats at every point in | |
the pipeline explicitly.</para> | |
<para>Drivers that implement the <link linkend="media-controller-intro">media | |
API</link> can expose pad-level image format configuration to applications. | |
When they do, applications can use the &VIDIOC-SUBDEV-G-FMT; and | |
&VIDIOC-SUBDEV-S-FMT; ioctls. to negotiate formats on a per-pad basis.</para> | |
<para>Applications are responsible for configuring coherent parameters on | |
the whole pipeline and making sure that connected pads have compatible | |
formats. The pipeline is checked for formats mismatch at &VIDIOC-STREAMON; | |
time, and an &EPIPE; is then returned if the configuration is | |
invalid.</para> | |
<para>Pad-level image format configuration support can be tested by calling | |
the &VIDIOC-SUBDEV-G-FMT; ioctl on pad 0. If the driver returns an &EINVAL; | |
pad-level format configuration is not supported by the sub-device.</para> | |
<section> | |
<title>Format Negotiation</title> | |
<para>Acceptable formats on pads can (and usually do) depend on a number | |
of external parameters, such as formats on other pads, active links, or | |
even controls. Finding a combination of formats on all pads in a video | |
pipeline, acceptable to both application and driver, can't rely on formats | |
enumeration only. A format negotiation mechanism is required.</para> | |
<para>Central to the format negotiation mechanism are the get/set format | |
operations. When called with the <structfield>which</structfield> argument | |
set to <constant>V4L2_SUBDEV_FORMAT_TRY</constant>, the | |
&VIDIOC-SUBDEV-G-FMT; and &VIDIOC-SUBDEV-S-FMT; ioctls operate on a set of | |
formats parameters that are not connected to the hardware configuration. | |
Modifying those 'try' formats leaves the device state untouched (this | |
applies to both the software state stored in the driver and the hardware | |
state stored in the device itself).</para> | |
<para>While not kept as part of the device state, try formats are stored | |
in the sub-device file handles. A &VIDIOC-SUBDEV-G-FMT; call will return | |
the last try format set <emphasis>on the same sub-device file | |
handle</emphasis>. Several applications querying the same sub-device at | |
the same time will thus not interact with each other.</para> | |
<para>To find out whether a particular format is supported by the device, | |
applications use the &VIDIOC-SUBDEV-S-FMT; ioctl. Drivers verify and, if | |
needed, change the requested <structfield>format</structfield> based on | |
device requirements and return the possibly modified value. Applications | |
can then choose to try a different format or accept the returned value and | |
continue.</para> | |
<para>Formats returned by the driver during a negotiation iteration are | |
guaranteed to be supported by the device. In particular, drivers guarantee | |
that a returned format will not be further changed if passed to an | |
&VIDIOC-SUBDEV-S-FMT; call as-is (as long as external parameters, such as | |
formats on other pads or links' configuration are not changed).</para> | |
<para>Drivers automatically propagate formats inside sub-devices. When a | |
try or active format is set on a pad, corresponding formats on other pads | |
of the same sub-device can be modified by the driver. Drivers are free to | |
modify formats as required by the device. However, they should comply with | |
the following rules when possible: | |
<itemizedlist> | |
<listitem><para>Formats should be propagated from sink pads to source pads. | |
Modifying a format on a source pad should not modify the format on any | |
sink pad.</para></listitem> | |
<listitem><para>Sub-devices that scale frames using variable scaling factors | |
should reset the scale factors to default values when sink pads formats | |
are modified. If the 1:1 scaling ratio is supported, this means that | |
source pads formats should be reset to the sink pads formats.</para></listitem> | |
</itemizedlist> | |
</para> | |
<para>Formats are not propagated across links, as that would involve | |
propagating them from one sub-device file handle to another. Applications | |
must then take care to configure both ends of every link explicitly with | |
compatible formats. Identical formats on the two ends of a link are | |
guaranteed to be compatible. Drivers are free to accept different formats | |
matching device requirements as being compatible.</para> | |
<para><xref linkend="sample-pipeline-config" /> | |
shows a sample configuration sequence for the pipeline described in | |
<xref linkend="pipeline-scaling" /> (table | |
columns list entity names and pad numbers).</para> | |
<table pgwide="0" frame="none" id="sample-pipeline-config"> | |
<title>Sample Pipeline Configuration</title> | |
<tgroup cols="3"> | |
<colspec colname="what"/> | |
<colspec colname="sensor-0" /> | |
<colspec colname="frontend-0" /> | |
<colspec colname="frontend-1" /> | |
<colspec colname="scaler-0" /> | |
<colspec colname="scaler-1" /> | |
<thead> | |
<row> | |
<entry></entry> | |
<entry>Sensor/0</entry> | |
<entry>Frontend/0</entry> | |
<entry>Frontend/1</entry> | |
<entry>Scaler/0</entry> | |
<entry>Scaler/1</entry> | |
</row> | |
</thead> | |
<tbody valign="top"> | |
<row> | |
<entry>Initial state</entry> | |
<entry>2048x1536</entry> | |
<entry>-</entry> | |
<entry>-</entry> | |
<entry>-</entry> | |
<entry>-</entry> | |
</row> | |
<row> | |
<entry>Configure frontend input</entry> | |
<entry>2048x1536</entry> | |
<entry><emphasis>2048x1536</emphasis></entry> | |
<entry><emphasis>2046x1534</emphasis></entry> | |
<entry>-</entry> | |
<entry>-</entry> | |
</row> | |
<row> | |
<entry>Configure scaler input</entry> | |
<entry>2048x1536</entry> | |
<entry>2048x1536</entry> | |
<entry>2046x1534</entry> | |
<entry><emphasis>2046x1534</emphasis></entry> | |
<entry><emphasis>2046x1534</emphasis></entry> | |
</row> | |
<row> | |
<entry>Configure scaler output</entry> | |
<entry>2048x1536</entry> | |
<entry>2048x1536</entry> | |
<entry>2046x1534</entry> | |
<entry>2046x1534</entry> | |
<entry><emphasis>1280x960</emphasis></entry> | |
</row> | |
</tbody> | |
</tgroup> | |
</table> | |
<para> | |
<orderedlist> | |
<listitem><para>Initial state. The sensor output is set to its native 3MP | |
resolution. Resolutions on the host frontend and scaler input and output | |
pads are undefined.</para></listitem> | |
<listitem><para>The application configures the frontend input pad resolution to | |
2048x1536. The driver propagates the format to the frontend output pad. | |
Note that the propagated output format can be different, as in this case, | |
than the input format, as the hardware might need to crop pixels (for | |
instance when converting a Bayer filter pattern to RGB or YUV).</para></listitem> | |
<listitem><para>The application configures the scaler input pad resolution to | |
2046x1534 to match the frontend output resolution. The driver propagates | |
the format to the scaler output pad.</para></listitem> | |
<listitem><para>The application configures the scaler output pad resolution to | |
1280x960.</para></listitem> | |
</orderedlist> | |
</para> | |
<para>When satisfied with the try results, applications can set the active | |
formats by setting the <structfield>which</structfield> argument to | |
<constant>V4L2_SUBDEV_FORMAT_TRY</constant>. Active formats are changed | |
exactly as try formats by drivers. To avoid modifying the hardware state | |
during format negotiation, applications should negotiate try formats first | |
and then modify the active settings using the try formats returned during | |
the last negotiation iteration. This guarantees that the active format | |
will be applied as-is by the driver without being modified. | |
</para> | |
</section> | |
<section> | |
<title>Cropping and scaling</title> | |
<para>Many sub-devices support cropping frames on their input or output | |
pads (or possible even on both). Cropping is used to select the area of | |
interest in an image, typically on a video sensor or video decoder. It can | |
also be used as part of digital zoom implementations to select the area of | |
the image that will be scaled up.</para> | |
<para>Crop settings are defined by a crop rectangle and represented in a | |
&v4l2-rect; by the coordinates of the top left corner and the rectangle | |
size. Both the coordinates and sizes are expressed in pixels.</para> | |
<para>The crop rectangle is retrieved and set using the | |
&VIDIOC-SUBDEV-G-CROP; and &VIDIOC-SUBDEV-S-CROP; ioctls. Like for pad | |
formats, drivers store try and active crop rectangles. The format | |
negotiation mechanism applies to crop settings as well.</para> | |
<para>On input pads, cropping is applied relatively to the current pad | |
format. The pad format represents the image size as received by the | |
sub-device from the previous block in the pipeline, and the crop rectangle | |
represents the sub-image that will be transmitted further inside the | |
sub-device for processing. The crop rectangle be entirely containted | |
inside the input image size.</para> | |
<para>Input crop rectangle are reset to their default value when the input | |
image format is modified. Drivers should use the input image size as the | |
crop rectangle default value, but hardware requirements may prevent this. | |
</para> | |
<para>Cropping behaviour on output pads is not defined.</para> | |
</section> | |
</section> | |
&sub-subdev-formats; |