Dmu Utility Library - ALPHA

Created by Ed Allard and Tim Davison Oct 1996

Table of Contents

Overview
List of Objects
Object model diagram
Sample programs

Overview


Motivation

While we were writting demos for the O2 we spent tons of time rewriting the same code in a slightly different way. In one demo we would need to use the O2 camera and the next would need to use the s-video input. The code is virtually identical, but depending on where the input is going, the code could change significantly. This library is an effort to make writing digital media demos easier. Another problem we ran into with writing demos was constanly mixing different libraries together. Making these different libraries such as the video library (VL) and the image conversion library (dmic) work together requires allocating dmbuffers.


General Overview

The dmu library consists of a group of c++ objects that wrap up the digital media libraries available on the O2. There are many different libraries that you can use to manipulate images on the O2. A couple of examples are: the video libarary (VL), the movie library (MV), the image conversion library (DMIC), and openGL. All of these libraries have different apis and deal with data differently. DMbuffers and DMBufferPools are data structures that act as the glue between all of these libraries. If one libaray can generate its output into a DMbuffer, another library can probably read that data and process.

For the DMbuffers to be properly allocated and configured, they must be constrained by both the library feeding data into it and the libaray taking data from the DMbuffer. Making sure that all of the constraints match is a real trick. With this utility library we hope to hide the creation, constraining, and allocation of the DMbuffer DMbufferPools from the user. The library also encasulates a number of commonly used features of the various digital media libraies.


Architecture

Since digital media usually means working with a flow of images (the current version of the library does not deal with audio), we chose to create a set of objects that know how to take image data (in the form of a DMBuffer) in, process it, and pass the processed data to multiple children. This is the basic function of any object in the library.

To use these objects, you create a tree of dmu objects. The root of a libdmu tree (sometimes referred to as a chain) is the source for all image data - for example, a live video source can act as a tree root. From this root, image data is propagated down to the leaves of the tree, being processed by each object it encounters along the way.

The O2 Unified Memory Architecture helps this scheme run efficiently. Since all of the subsystems on the O2 are connected to a single, shared memory system, we can process data by simply passing around references to it. In other words, when passing the data down the tree, we never have to copy it, we only have to pass pointers (in the form of DMBuffer handles). Since we never copy the bits of data, the performance of this utility library is very good. Passing full size full frame rate video to texture map works beautifully.


Functionality - Using the library

So far we haven't mentioned what types of objects the dmu library provides. See the object list, or the diagram of the object hierarchy for a complete list. Examples of the objects that you can work with include: video input, video output, texture map sources, compression encoders and decoders, and movie input objects. This list will grow as the library grows (some of the objects that are planned are listed on the object list page). This section outlines the basic steps you need to take to use these objects effectively in a programe.We have provided a number of sample programs that will show you how to use the library in more detail.

There are a couple of steps needed to make an application using this library. The example code used here is taken from the compressionTest sample program. Note that we are only using a subset of the total code for these examples. We have also removed the error checking code here to make it easier to read the code.

Construct:

The first stop is to create the objects you want to use in your application. For example if you want to take video in and compress it, you would need to construct both a video object and an encoder object:

  // Fix this with the new encoder test later
  // create a video in object
  video = new dmuVideoIn();
  video->setName("video");

  // create an jpeg encoder object
  encoder = new dmuICjencoder();
  encoder->setName("encoder");

Wire:

After all of the objects you want are created, you need to wire them together to build the tree. Each object can have any number of children but no object is allowed to have multiple parents. This is not enforced explicitly in the error checking of the library, but the behavior will probably not be what you'd expect.

In our example, we want the encoder to be a child or the video object.

  // add the encoder as a child or the video object
  video->addObject(encoder);

In the compressionTest sample code, you will notice that the video object is wired also to a display object. You can add as many objects as children as you want. In a real application you would probably want to wire the encode to something also.

Setup:

Once all of the objects have been created and wired, you need to set them up so that they can talk to each other. What you need to do in the setup stage is different for each class, so look at the individual objects for more details.

  video->setup(VL_ANY);
  // set the pixel packing 
  video->setPacking(VL_PACKING_YVYU_422_8);
  video->setCaptureType(VL_CAPTURE_NONINTERLEAVED);
  video->getSize(&sizeX,&sizeY);

  encoder->setImgWidth(sizeX);
  encoder->setImgHeight(sizeY);
  encoder->setQuality(90);
  encoder->setup();
Initialize:

This step is very simple on the surface, but it is the heart of the whole system. Within the initialization step, we create the DMBuffer pools for objects to dump their data into. For this to work, the entire chain must be wired together, and the setup method must be called on each object that requires it.

These requirements are neccessary because the DMBuffer Pools that are created in this step have dependencies both the parent and the children objects. To perform initialization on a tree, you need only call the init method on the root object. The initialization propagates through the chain automatically.

  // Call that creates the dmbuffers throughout the system
  // This call will progagate down the chain automatically
  video->init();

If this step completes with no errors you are most of the way there to having a working application.

Start:

Next we want the data to start flowing. Start() is another method that will propagate down the tree, so you need only call it on the root of the tree. Some objects may not do anything upon startup, but all objects will at least propagate the method down the tree. To start the video transfer in our test, we call the start method on the root of the tree.

  // Start the video transfer.  This call will progagate to all 
  // of the other objects.  
  video->start();
Handle Events:

At this point we need to set up the event loop. The library is designed to be thread safe, so it's OK for this loop to happen in a sproc'd thread. The event loop is used to coordinate the passage of data between objects which perform some sort of asynchronous, or multi-threaded processing. This event loop is analogus to the vlMainLoop in a typical VL application, or an X event loop in a Motif application.

Most of the objects have file descriptors assosiated with them to tell the main program that they have new data which should be passed along to its children. This isn't appropriate for all objects - see the individual object declarations (and the sample code) for details on events for a given object.

A typical event loop looks something like this:

 // event loop. loop forever 
  while (1) {
    // zero out the fd set
    FD_ZERO(&readFDSet);

    // this call will propagate down the tree and set the fd for each
    // object
    video->setFD(&readFDSet, &maxFD, 1);

    // block until one of the fd's has some data on it
    select (maxFD+1, &readFDSet, NULL, NULL, NULL);

    // let the objects handle their events.  each object checks to see
    // if their own fd is set in the FDset and if so, they act on it.  
    encoder->handleEvents(readFDSet);
    video->handleEvents(readFDSet);
  }

The first step in the event loop is to set up a file descriptor set using the setFD method. This allows us to block our main process until one of the objects in the tree has data to be dealt with.

We then use the select system call to wait for activity on the descriptor set. This call blocks untill one of the file descriptors is set, informing us that there is something to be done.

Next, we call the handleEvents method on each object in the chain which has a file descriptor associated with it. If the object is not the one which caused the select to fall through, this call returns immediately and returns the value (DMU_NOT_MY_EVENT). Otherwise the event is handled, and the data is propagated down the tree as appropriate.

 Each object has a method called handleEvents which checks the fdSet to see if that object's file descriptor was set. If the file descriptor was set, the object goes and propagates the data down to the next object in the chain. 
Oooh and Ahh..

Now you can sit back and Oooh and Ahh about how cool your demo looks.


Overview    Object List     Object Hierarchy       Sample Apps