This document supplements the material in the "IRIS Media Libraries Programming Guide" (hereinafter, "the Guide"). It adds information about the VL that I think is missing or wrong in the Guide. It doesn't attempt to answer all remaining questions about the VL, or to be a complete reference. This is merely the information that I figured out while making InPerson work on VINO, EV1, Galileo 1.5, and Sirius video for IRIX 5.3 and 6.2. This information applies only to those VL video devices, and not to any other present or future VL video devices.
This document does not represent SGI, and is not an official SGI publication. The entire contents of this document are my personal observations, and/or opinions, and all errors and opinions in it are mine alone. SGI makes no guarantees about anything in this document. Neither do I. You use it at your own risk. Additional disclaimers below.
Some topics NOT discussed in this page include:
The VL_SIZE control on the video source node is a read-only control. The x and y values returned by this control are affected by the setting of the VL_TIMING control on the video source node. The x and y values of this control are not, in general, affected by the settings of any controls in the memory drain node, including VL_ZOOM, VL_SIZE, and VL_CAP_TYPE.
The x component value of this control reveals the width, in pixels, of
the unzoomed, unclipped video input images (fields or frames).
The meaning of the y component value of the video source node's VL_SIZE
control depends on the video device.
On Sirius, the y value is the number of pixel rows in each FIELD,
and includes the count of rows of pixel samples taken from the field's
Vertical Retrace Interval.
On EV1 and VINO, the y value is the number of pixel rows in each FRAME
(pair of fields), and does not include any pixel rows from the Vertical
Retrace Interval.
On all VL video devices except Sirius, VL_FORMAT is not applicable to memory drain nodes, and VL_PACKING is used to select the color space as well as the pattern by which the components are packed into memory buffers. Packings that imply RGB or RGBA color spaces select those spaces. Packings that imply Y, or YUV or YCrCb color spaces select one of those spaces.
A complete description of each of the available color spaces is beyond the scope of this web page. However some discussion of the YUV and YCrCB spaces is necessary. The YUV and YCrCb color spaces are extremely similar to each other, and primarily differ in the ranges of acceptable values for the three components when represented as digital integers.
There is one set of packings that is used with both YUV and YCrCb color spaces. These packings are typically referred to using the letters Y, U, and V, such as "YVYU_422" or "YUV422" (which are common names for the same packing). These names refer to the packing only. The use of a packing named "YUV" or "YVYU" does not imply that the data packed is YUV data, as opposed to YCrCb data. When YCrCb data is being packed with a YUV packing, the Cr component is packed as U, and the Cb component is packed as V. The VL_PACKING_YVYU_422_8 packing is the only packing that is natively supported in hardware (requiring no software conversion) on all VL video devices.
The "422" designation in the packing name means that the pixels are packed so that each horizontally-adjacent pair of pixels share one common set of chrominance (e.g., "UV", or alternately, "CrCb") data. Each pixel has its own value of luminance or "Y" data. So data is packed in pairs of two pixels, two Y values, and one U and one V (or alternately, one Cr and one Cb) value pair, in each pixel pair. In such a pixel packing, the number of pixels in each row will always be even.
The set of VL packings presently defined does not enable the application to choose between the YUV and CCIR-601 YCrCb color spaces. When an app selects VL_PACKING_YVYU_422_8, it will get YUV or YCrCb, depending on the device and the source node from which the data is coming. The Analog-to-Digital converter chip used in VINO and EV1 produces YUV data. Most external digital sources produce YCrCb data. There is no way to tell, from the VL_PACKING control, which of those two spaces (YUV or YCrCb) will be used.
Each of the different VL video devices has a different set of color spaces and packings implemented in hardware. Any other color spaces and/or packings are implemented via software conversion, if they are implemented at all! The table below shows which color-space and packing combinations are implemented in hardware, or software, or not at all, for each device.
Color Spaces and Packings
VL_PACKING_ | Pixels/ 32-bit word | Format MSB -- LSB |
---|---|---|
_RGB_332_P | BBGGGRRR BBGGGRRR BBGGGRRR BBGGGRRR | |
_RGBA_8 | AAAAAAAA BBBBBBBB GGGGGGGG RRRRRRRR | |
_RGB_8 | xxxxxxxx BBBBBBBB GGGGGGGG RRRRRRRR | |
_YVYU_422_8 | UUUUUUUU YYYYYYYY VVVVVVVV YYYYYYYY | |
_Y_8_P | YYYYYYYY YYYYYYYY YYYYYYYY YYYYYYYY |
Implementations of Color Spaces and Packings
VL_PACKING_ | Color Space | VINO | EV1 | Galileo 1.5 | SIRIUS |
---|---|---|---|---|---|
_RGB_332_P | RGB | H/W | N/A | N/A | N/A |
_RGBA_8 | RGBA | H/W | S/W | S/W | H/W |
_RGB_8 | RGB | H/W | S/W | S/W | H/W |
_YVYU_422_8 | YUV, YCrCb | H/W | H/W | H/W | H/W |
_Y_8_P | Y | H/W | ??? | ??? | N/A |
The chipset used in VINO and EV1 to convert analog input to digital pixels produces YUV output, not YCrCb output. That is, the values of Y, U and V are in the range 0..255 (the SMPTE YUV ranges), not the smaller 16..235/240 range specified for CCIR601 YCrCb. IndyCam produces CCIR601-compliant YCrCb. For some devices that can't convert color space in hardware, e.g. EV1, the VL converts from YUV to RGBX/RGBA in software. In IRIX 6.2, the VL routines used for this purpose assume the input is CCIR601 YCrCb, not YUV, regardless of what the hardware actually produces. The moral of this story is that if the harware doesn't support the color space you want, and you want the color space conversion to be really accurate, then ask for the pixels in a color space supported by the hardware, and do the color space conversion in your own software, rather than depending on the VL's software colorspace conversion.
With Sirius Video, color space and packing are independent. Color space is chosen by the settings of the VL_FORMAT on the memory drain node, according to table below, and any packing can be applied to any color space, whether it makes sense or not. Color space conversion occurs when the VL_FORMAT of the video source node and the VL_FORMAT of the memory drain node imply different color spaces.
Sirius Color Spaces
Color Space | VL_FORMAT_ |
---|---|
RGB, RGBA | _RGB, _COMPOSITE, _SVIDEO |
YUV | _BETACAM, _MII, _SMPTE_YUV |
CCIR601 YCrCb | _FORMAT_DIGITAL_COMPONENT_* |
With Sirius Video, this control is read-only, and is permanently set to
VL_CAPTURE_NONINTERLEAVED. Each captured buffer contains exactly one
field, unclipped, unzoomed, with N leading pixel rows of samples from the
vertical retrace interval.
EV1 implements VL_CAPTURE_NONINTERLEAVED differently from all other
VL video devices. On all VL video devices except EV1, when VL_CAP_TYPE
is set to VL_CAPTURE_NONINTERLEAVED, each image buffer that the VL gives to
the application contains one field, either "even" or "odd", and VL_RATE
(the rate at which these buffers are returned) is in fields per second, not
frames per second. But on EV1 video devices, when VL_CAP_TYPE is set to
VL_CAPTURE_NONINTERLEAVED, each image buffer contains two non-interleaved
fields, and VL_RATE is in frames per second.
VINO has a bug that affects capturing with VL_CAPTURE_INTERLEAVED.
The first buffer returned after the call to vlStartCapture
may have only one field in it.
The other field may contain all zero values. For continuous transfers,
this may go unnoticed, but for VL_TRANSFER_MODE_DISCRETE captures
(e.g. of a single image), this will often result in an image in which
every other line is black, or green, depending on the color space in use.
A missing field can be detected by
examining the first pixel or two in several successive pixel rows. The
workaround for this problem is to examine the image for the presence of one
black field, and one non-black field, and recapturing another buffer if one
field is black.
Another solution worth exploring is to capture two buffers, instead of one,
and then throw the first one away. This might suffice.
There is no single VL_CAP_TYPE that is available, and implemented in the
same way, on all VL video devices. VL_CAPTURE_NONINTERLEAVED is available
on all devices, but has different meanings on different platforms.
VL_CAPTURE_INTERLEAVED, VL_CAPTURE_EVEN_FIELDS, and VL_CAPTURE_ODD_FIELDS
are available and common to all VL video devices except Sirius.
On all VL video devices except Sirius,
this control sets the target rate (upper bound) of image buffers per second
that will be captured and returned to the application. That is, the VL
will not deliver more buffers per second than the rate you specify, but it
might deliver less than you specify, especially on EV1.
The contents of each image buffer will be either a frame or a field, as
determined by the VL_CAP_TYPE control. Accordingly, VL_RATE is in units of
fields per second or frames per second, as determined by the VL_CAP_TYPE
control.
VL_RATE is a fraction. Both the numerator and denominator must be
specified. The usual value for the denominator is 1. Some devices,
e.g. EV1, convert the fraction to a whole integer (truncating, not
rounding) of images per second, so using values that are equivalent
to integer values is the safest thing to do.
Because VL_RATE is a fraction, vlGetControlInfo() cannot be used to obtain the
minimum or maximum values for VL_RATE.
Acceptable values are determined from the following table for non-Sirius
devices:
VL_TIMING
On all VL video devices except Sirius, the VL_TIMING control applies only
to the video (not memory) nodes. The timing on the video source node is
usually set through the Video Control Panel.
With Sirius Video, the VL_TIMING control must be set properly on the memory
drain node.
An easy way to set the VL_TIMING value for the memory node is to read
the value of the VL_TIMING control from the video source node, and then
set that value into the VL_TIMING control for the memory node.
VL_CAP_TYPE
On all VL video devices except Sirius,
this control is settable by the application.
Its setting determines whether the images in the buffers returned by the
VL are individual fields, or interleaved frames, or pairs of
non-interleaved fields. VL_RATE
In the section entitled "Setting Drain Node Controls for Data Transfer",
the Guide states, incorrectly, that the VL_RATE control is "not applicable"
to memory drain nodes. For vid-to-mem applications, the memory drain node
is precisely where video rate is controlled.
VL_CAPTURE_NONINTERLEAVED (all devices except EV1 and Sirius) | NTSC | All multiples of 10 and 12 between 10 and 60 |
PAL | All multiples of 10 between 10 and 50 | |
VL_CAPTURE_NONINTERLEAVED (EV1) | NTSC | All multiples of 5 and 6 between 5 and 30 |
PAL | All multiples of 5 between 5 and 25 | |
VL_CAPTURE_INTERLEAVED, VL_CAPTURE_EVEN_FIELDS, and VL_CAPTURE_ODD_FIELDS | NTSC | All multiples of 5 and 6 between 5 and 30 |
PAL | All multiples of 5 between 5 and 25 |
With Sirius Video, this control is read-only. Its value is determined by
the setting of the VL_TIMING control on the memory node.
VL_CAPTURE_NONINTERLEAVED | NTSC | 60 fields per second |
PAL | 50 fields per second |
VINO's VL_RATE cannot be set to a value less than 5/1.
1/1, 1/2, 1/3 | Implemented in hardware, looks OK. |
1/4, 1/6 | Implemented partially in hardware, partially in software. Looks OK, but slower and uses 10% of an R4600 CPU. |
1/5, 1/7, 1/8 | implemented in hardware. Looks bad. Green shift. |
EV1:
1/1, 1/2, 1/4, 1/8 | Works OK for vid-to-mem |
1/3, 1/5, 1/7 | Works only for vid-to-screen, not vid-to-mem, and only with VL_CAPTURE_INTERLEAVED. |
2/1, 4/1 | Works only for vid-to-screen, not vid-to-mem. |
Sirius and Galileo 1.5:
1/1 | Sirius and Galileo 1.5 don't zoom. |
The VL_SIZE control on the memory drain node determines the number of rows
of pixels, and the number of pixels in each row, in each image buffer
(field or frame) that the VL returns to the application.
If zooming (decimation) is being done, the VL_SIZE control on the memory
drain node specifies the size of the image after it has been decimated.
The VL_SIZE control on the memory drain node can be used to "clip" a region
out of an image by setting the X and/or Y components to values that are
smaller than the size of the captured (and decimated, if applicable) image.
When the (possibly decimated) image is being clipped, the clipped region
does not have to come from the upper left hand corner of the (possibly
decimated) source image. The VL_OFFSET control on the memory drain node
determines the number of top pixel rows to skip and the number of leading
pixels to skip in each row to find the first pixel in the (possibly
decimated) image to place in the image buffer, the first pixel of the
clipping region.
The Guide says, incorrectly, that "VL_OFFSET operates on the unzoomed image;
it does not change if the zoom factor is changed.". The truth is that
when zooming (decimation) is being used, VL_OFFSET is always in coordinates
of the zoomed image. It is as if the entire source image is decimated
down, and then the clipping function is applied to the decimated image.
The hardware rarely works that way, and usually clips before decimating,
but the VL API always specifies the VL_OFFSET in the coordinates of the
decimated (virtual) image.
On all VL devices except Sirius, the vertical (Y) component of VL_OFFSET
may be specified with a negative value. This causes the clipping region
to include row of samples taken before the top of the image, e.g. rows
from the Vertical retrace interval. This feature is usually used with
VL_ZOOM of 1/1, since the information in the Vertical Retrace Interval
isn't an image and doesn't make sense to decimate or average, at least
not in the vertical direction.
The VL imposes these requirements on the values of VL_OFFSET and VL_SIZE:
VL_OFFSET and VL_SIZE cannot be both set in one atomic operation.
A change in either component of either control could violate
one of the rules above (or below), especially after VL_ZOOM is set to a
smaller fraction. It may be necessary to alternately and repeatedly set
VL_OFFSET and VL_SIZE until no VLValueOutOfRange errors are reported.
Follow this link to see a code sample that does this.
Every VL video device places additional limitations on the
range of acceptable values of VL_SIZE and VL_OFFSET.
Each device has different limitations.
Sirius Video doesn't support preemption. EV1 Video allows preemption
in some "sync modes" but not in others. When preemption is not allowed,
the VL behaves as if the path was in mode VL_LOCK instead of VL_SHARE;
preemptions simply don't happen.
When preemption occurs, your program will receive a VlStreamPreempted
event. This means the following things:
The VLStreamAvailable events create a race between the preempted programs.
The outcome of the race depends on the actions taken by the programs, and
also on the device's implementation. Generally, as long as all the
programs are attempting shared use of the path, the last caller of
vlSetupPaths() wins.
When a program receives the VLStreamAvailable event, to regain the usage
of the path's data stream, it should call vlSetupPaths(..., VL_SHARE,
VL_SHARE) again. This call to vlSetupPaths() to regain access to the
path's data stream can succeed or it can fail. The VLStreamAvailable event
does not guarantee that it will succeed. Let's look at these cases
separately.
If the call to vlSetupPaths() succeeds, the application should set all
the controls for the path to the values they had before it was preempted.
Your application, that is now regaining the use of the stream,
should not assume that the controls are as they were before the
preemption.
The preemption caused the usage of the path's controls to become VL_SHARE,
so the preempting application could have changed any or all of those
controls.
Since preemption does not deregister or destroy the ring
buffer used by your path, you may resume usage of that same buffer,
provided that all the path's controls are returned to the same values
they had before the preemption (especially VL_SIZE and VL_PACKING).
However, you might want to be doubly sure that the ring buffer is empty
before restarting the transfer with a call to vlBeginTransfer().
If the call to vlSetupPaths() to regain acceess to the path's data stream
fails (returns a negative value), the path should remain in the same
mode in which it was before the failed call to vlSetupPaths(). If your
program hasn't changed the path since it was preempted, and if the call to
vlSetupPaths() failed with vl error VLPathInUse, the path should
still be in the mode in which it was placed by the original preemption,
namely as if vlSetupPaths(..., VL_SHARE, VL_READ_ONLY) had been done on
this path. The path should continue to behave exactly as when it was
preempted, and another VLStreamAvailable should be generated, if and when
the path becomes available again.
A VL application that wishes to use the default input source on its device
calls vlGetNode(vlServer, VL_SRC, VL_VIDEO, VL_ANY).
Each device's default input source node may be changed via a control.
The Video Control Panel supports this, and is usually used to make this
choice. When the default input source on any VL video device is changed,
(e.g. between a digital input source node and an analog input source node),
the VL sends a VLDefaultSource event to all programs that have selected
this event, regardless of whether they are using the device whose default
input changed, or another VL device. A change in input source between
jacks connected to the same input node usually results in a
VlControlChanged event, not a VLDefaultSource event.
No node changes are forced upon any VL transfers that are in progress.
A program that wishes to switch to the new video input source node must
take steps to make the switch itself.
When a program receives the VLDefaultSource and wishes to switch to the new
default input source, it must take these steps:
While the path's stream usage is in VL_READ_ONLY mode, a VLStreamAvailable
event should be generated when the data stream for the selected path
becomes available. That is, a program should be able to put itself into
a "voluntarily preempted" mode by putting the path's stream usage is in
VL_READ_ONLY mode. Whether this actually works or not depends on the
device, and on the OS release. It has worked on EV1 at one time.
Sirius does not support putting a path into VL_READ_ONLY mode.
Sirius also does not support preemption.
An application should be able to transition the stream usage of a path
from any one to any other of the four modes (VL_SHARE, VL_READ_ONLY,
VL_LOCK, VL_DONE_USING) as many times as it wishes. VL_DONE_USING is
not the "mode of no return". The principal difference between VL_READ_ONLY
and VL_DONE_USING seems to be the receipt of VLStreamAvailable events in
the VL_READ_ONLY mode.
Since Sirius doesn't support VL_READ_ONLY, an application cannot use
VL_READ_ONLY to put itself into a mode where it will be notified when
another device has finished with the path. A Sirius app has no choice
but to go to VL_DONE_USING mode when relenquishing the path, and to use
some external or manual means to know when to attempt to reaquire the path.
VL_SIZE and VL_OFFSET
There are two separate VL_SIZE controls that a vid-to-mem app must deal
with. One is a control on the video node, the other is on the memory
node. They have different meanings, different units, and do different
things. The most common problem in discussing the VL_SIZE control is
people forget to specify which one they're talking about, or are confused
about that. This section discusses the VL_SIZE control and the VL_OFFSET
control on the memory drain node.
When an attempt to set either one of these controls
would violate either of the rules above,
the call to vlSetControl() fails with the vlErrno VLValueOutOfRange,
and the offending component (horizontal or vertical) is set to the largest
non-negative value that does not violate the rule, or to zero if no such
non-negative value exists.
VINO imposes the following additional clipping requirements:
Other clipping BUGS:
Hints about VL Events
VLStreamPreempted
If your application has setup a VL path with "shared" usage of controls
and the data stream (e.g. vlSetupPaths(..., VL_SHARE, VL_SHARE), then
whenever another application attempts to setup the same path as yours,
or a path that shares nodes with your path, your program will be preempted.
To handle preemption properly, your program must select VLStreamPreempted
and VLStreamAvailable events in its call to vlSelectEvents().
No immediate action is required on the part of a program that receives
a VlStreamPreempted event.
The transfer has been ended, so you don't need to call vlEndTransfer().
The ring buffer in use by your path at the time of its preemption is
neither deregistered nor destroyed by the preemption.
There may still be one or more image buffers available.
Calls to vlGetNextValid() or vlGetLatestValid() may continue to return
previously unprocessed images. Non-NULL values returned from
vlGetNextValid() or vlGetLatestValid() should not be construed to mean
that the transfer is still in progress.
It is probably a good idea to process and free all image buffers that
can be obtained from calls to vlGetNextValid() or vlGetLatestValid(). VLStreamAvailable
When all the nodes of a path that is being held in VL_READ_ONLY mode
are released, all programs holding the path in this mode will receive a
VLStreamAvailable event. This will happen when the program that was
last using the path called vlDestroyPath(), or released the path by setting
the stream usage to VL_READ_ONLY or VL_DONE_USING in a call to
vlSetupPaths(). VLDefaultSource
Each Video device (e.g. VINO, EV1) may have multiple input sources (jacks).
The VL supports the concept of a "default" input source for each device.
That is, each device has its own default input source. There is no
system-wide default input source, nor is there a default input device.
An application on a system with multiple video devices must choose one of
the devices according to its own methods.
Other Miscellany
More About vlSetupPaths()
If a first attempt to gain access to a path via
vlSetupPaths(..., VL_SHARE, VLSHARE) fails,
and vlGetErrno() returns VLPathInUse, then it should be possible to call
vlSetupPaths(..., VL_SHARE, VL_READ_ONLY)
to put the path in the same state as it would be if it had been preempted.
Likewise, a program that wishes to voluntarily release the path, and then
later regain it, as if it had been preempted should be able to put the path
info VL_READ_ONLY mode. Premption vs. Cooperative Management
Since pre-emption and VL_READ_ONLY modes don't work on all platforms,
two or more programs that wish to cooperatively share the use of a VL
device might be better off to use their own protocol to pass ownership
of the device between them. The programs in the InPerson product used
this approach succesfully. IndyCam on EV1 Video Devices
IndyCam works on many of the boards in the EV1 family. The images produced
by IndyCam on EV1 tend to be very noisy in low light, much more so than
the images produced by IndyCam on VINO. This noise is seen as vertical
and horizontal lines of gray in the image. Since the output of IndyCam is
digital (CCIR-601), this noise is probably induced via the power supplied
to the IndyCam by the video board.
$Id: vid2mem.html,v 1.1 1996/08/02 00:11:50 dave Exp $
Additional Disclaimers:
This product is intended for educational purposes only.
Not responsible for direct, indirect, incidental or consequential damages
resulting from any defect, error, or failure to perform.
NO OTHER WARRANTY EXPRESSED OR IMPLIED. Reproduction strictly prohibited.
The trademarks mentioned in this product appear for identification
purposes only. Your mileage may vary. Not for resale.
Any resemblance to real persons, living or dead, is purely coincidental.
Void where prohibited. This is not an offer to sell securities.
Some assembly required. Batteries not included. Not to be taken internally.
Contents may settle during shipment. Freshest if eaten before date on carton.
Use only as directed. Avoid contact with skin. Apply only to affected areas.
Use only in well-ventilated area. Keep away from fire or flame.
Do not use while operating a motor vehicle or heavy equipment.
If condition persists, contact your physician.
Contains a substantial amount of non-tobacco ingredients.
Not recommended for children under five. May be too intense for some viewers.
No passes accepted for this engagement. All models over 18 years of age.
Times approximate. Simulated picture. Edited for television.
Processed at location stamped in code at top of carton.
No user-serviceable parts inside. Subject to change without notice.
For recreational use only. For off-road use only. Subject to approval.
One size fits all. Colors may fade. See label for washing instructions.
For office use only. Post office will not deliver without postage.
Penalty for private use. Substantial penalty for early withdrawl.
Nor for removal except by the consumer. List was current at time of printing.
Lost ticket pays maximum rate. At participating locations only.
Your canceled check is your receipt. Slightly higher west of the Mississippi.
Employees and their families not eligible. You must be present to win.
Contestants have been briefed on some questions before the show.
Limited time offer. No purchase necessary. Price does not include sales tax.
Slippery when wet. Breaking seal constitutes acceptance of agreement.
Some equipment shown is optional. Replace with same type.
No anchovies unless otherwise specified. Driver does not carry cash.