Using Video Mixing Renderer 9 with Multiple Filter Graphs

1. What Is New

VMR7 contains all the necessary functionality to manage several filter graphs; this task can be accomplished by providing VMR7 with custom allocator-presenter (as shown in sample VMRMulti, see Samples\cpp\DirectShow\VMR\VMRMulti). However, introduction of VMR9 essentially simplifies this task:

• In VMR7, developers had bad times with surface allocation, while VMR9 introduces helper function IVMRSurfaceAllocatorNotify9::AllocateSurfaceHelper, so that allocated surface is compatible both with VMR9 and client application preferences
• Since VMR9 is based on DirectX 9, there is nothing special to be done to put video in the DirectDraw exclusive mode: client application just gives VMR9 properly initialized Direct3D9 device.

2. Using Custom Allocator-Presenter to Manage Several Filter Graphs

2.1. Developing Custom Allocator-Presenter for VMR9

Development of custom (user provided) Allocator-Presenter for the renderer has not changed much since VMR7, but there are several differences. Below is the “how to” scenario, it is implemented in the sample VMR9Allocator (See Samples\cpp\DirectShow\VMR9\VMR9Allocator). First, developer of the client application has to implement custom allocator-presenter (A/P), an object derived from IVMRSurfaceAllocator9 and IVMRImagePresenter9. See DirectX 9.0 SDK help for more information on custom allocator-presenter (see “Supplying a Custom Allocator-Presenter for VMR9”). Figure 1 shows relationship between VMR9 filter and its Allocator-Presenter.

• Please note that this is developer’s responsibility to make custom allocator-presenter object thread safe.
• IVMRSurfaceAllocator9::AdviseNotify. Client application calls this method to inform A/P about allocator notifier. Custom implementation should just save the pointer to IVMRSurfaceAllocatorNotify9. It will be used in IVMRSurfaceAllocator9::InitializeDevice.
• IVMRSurfaceAllocator9::InitializeDevice. VMR9 calls this method to initialize Direct3D9 device and allocate the chain of Direct3D9 surfaces. When implementing this method, client application can modify flags of VMR9AllocationInfo structure. Usually this method performs the following tasks:
• Create Direct3D9 object (or acquire it from some other part of the application).
• Create Direct3D9 device (or acquire it from some other part of the application).
• Allocate Direct3D9 surfaces according to VMR9AllocationInfo structure. In most cases, it is advised to use helper function IVMRSurfaceAllocatorNotify9::AllocateSurfaceHelper rather than directly allocate surfaces with methods of IDirect3Ddevice9.
• Create all the other Direct3D9 objects associated with this custom A/P (if applicable)
• IVMRSurfaceAllocator9::GetSurface. VMR9 calls this method every time it needs Direct3D surface. Custom implementation of this method returns a pointer to one of the surfaces allocated in IVMRSurfaceAllocator9::InitializeDevice.
• IVMRSurfaceAllocator9::TerminateDevice. Custom implementation of this method releases all the resources allocated in IVMRSurfaceAllocator9::InitializeDevice.
• IVMRImagePresenter9::StartPresenting. VMR9 calls this method to indicate that presenting is about to start. Custom implementation just returns S_OK in most cases.
• IVMRImagePresenter9::StopPresenting. VMR9 calls this method to indicate that presenting is about to stop. Custom implementation just returns S_OK in most cases.
• IVMRImagePresenter9::PresentImage. VMR9 calls this method to indicate that sample is ready to be rendered. Custom implementation can either perform actual rendering at this point (as in VMR9Allocator sample), or save the buffer with image (when rendering is asynchronous from VMR9 calls). If surfaces were not allocated as render targets, it is developer’s responsibility to correctly transorm formats to perform rendering. For example, if the video source is in YUV format, developer might use IDirect3Ddevice9::StretchRect function, etc.

2.2. Using Custom Allocator-Presenter

When client application supplies a custom A/P for VMR9, there are several rules to be followed: custom A/P must be thread-safe; when advising custom A/P, filter graph must be stopped and VMR9 filter must be in the rendeless mode; and all its inputs pins must be disconnected. Below is the typical scenario of using custom A/P.

1. Create instance of the filter graph (CLSID_FilterGraph).

2. Create instance of Video Mixing Renderer 9 (CLSID_VideoMixingRender9) and add it to the filter graph.

3. Set VMR9 to the Renderless mode (see IVMRFilterConfig9::SetRenderingMode) .

4. Query VMR9 filter for interface IVMRSurfaceAllocatorNotify9.

5. Create instance of custom allocator-presenter.

6. Associate your filter graph with some cookie (any DWORD_PTR value that uniquely identifies the filter graph).

7. Advice custom A/P to the VMR9 (see IVMRSurfaceAllocatorNotify9::AdviseSurfaceAllocator).

8. Inform custom A/P about the allocator notifier (see IVMRSurfaceAllocator9::AdviseNotify).

9. Build the rest of the graph, render, and run.

10. When destroying the filter graph, stop the graph, disconnect all the pins of VMR9 filter and advise it with NULL surface allocator

2.3. Using Custom Allocator-Presenter in Multi-Graph Environment

To provide multi-graph functionality to the scenario above, few modifications are needed. Custom A/P should contain references to all the connected filter graphs. Each filter graph is uniquely identified with a cookie (parameter DWORD_PTR dwUserID in methods of IVMRSurfaceAllocator9 and IVMRImagePresenter9). Since several filter graphs are asynchronous to each other, it is advised to perform rendering in a separate thread with some fixed user-defined frame rate. In this case, method IVMRImagePresenter9::PresentImage just copies images from VMR9 filters to some private textures. This causes some performance drop, but leaves filter graphs independent on each other. It is advised to set the frame rate to the highest frame rate of all connected video streams to prevent frame dropping. At last, it would be logical if such custom allocator-presenter have methods Attach and Detach, to connect and disconnect involved filter graphs. Figure 2 illustrates relationship between connecting filter graphs and custom A/P.

3. MultiVMR9.dll Helper Library

DirectX 9.0 SDK samples include sources for MultiVMR9.dll (see Samples\cpp\DirectShow\VMR9\MultiVMR9). This helper library demonstrates how to implement a user-provided (customized) allocator-presenter for the Video Mixing Renderer 9 in a multigraph environment. It seemed logical to extract several abstraction layers that could be overriden by client applications:

· Wizard, an object that implements actual allocation-presenting and contains references to all connected filter graphs. The wizard is responsible for interaction with client application as well as connecting and disconnecting graphs. Object is implemented by IMultiVMR9Wizard interface.

· Rendering Engine, an object that contains all Direct3D data and performs actual rendering asynchronously from calls to IVMRImagePresenter9::PresentImage. Object is implemented by IMultiVMR9RenderEngine interface.

· Mixer Control, an object that performs scene composition and defines geometry of Direct3D objects. By default, this object acts as IVMRMixerControl9 in a single graph (see sample MultiPlayer, Samples\cpp\DirectShow\VMR9\MultiVMR9\MultiPlayer), but can be overriden to perform any sort of 3D transformation (as in the sample GamePlayer, Samples\cpp\DirectShow\VMR9\MultiVMR9\GamePlayer). Object is implemented by IMultiVMR9MixerControl interface.

· User Interface Layer, an object that is responsible for interaction with user in video playback window. User Interface Layer is always drawn over the scene rendered by mixer control. Message processing function of the video window sends all the appropriate messages to this object. Object is implemented by IMultiVMR9UILayer interface. By default, MultiVMR9.dll does not use this object, so if client application needs user interaction, it must provide IMultiVMR9UILayer object.

Figure 3 illustrates structure of MultiVMR9.dll.

3.1. Interfaces

See Samples\cpp\DirectShow\VMR9\MultiVMR9\DLL\MultiVMR9.idl

IMultiVMR9Wizard

Objects exposing this interface are responsible for managing several VMRs with single rendering environment. Object that implements this interface must also implement IVMRSurfaceAllocator9 and IVMRImagePresenter9. Otherwise method Initialize should return E_NOTIMPL. CLSID_MultiVMR9Wizard is the default implementation of this interface.

Initialize	Configure and initialize internal structures. This method must be called before using the wizard; otherwise consequent calls to the wizard will return VFW_E_WRONG_STATE. Also, use this method to supply customized Rendering Engine.
Terminate	Releases all internal resources of the wizard; disconnects all connected filter graphs. This method must be called before destroying the wizard.
Attach	Attaches filter graph to the wizard
Detach	Detaches filter graph from the wizard
VerifyID	Verifies if there is any attached filter graph with specified “UserID” cookie
GetGraph	Returns pointer to IFilterGraph of the connected filter graph identified by “UserID” cookie
GetRenderEngine	Returns pointer to IMultiVMR9RenderEngine actively used by the wizard
GetMixerControl	Returns pointer to IMultiVMR9MixerControl actively used by the wizard
GetTexture	Returns pointer to IDirect3DTexture9 that contains “ready-to-render” image from the filter graph identified by “UserID” cookie
GetVideoSize	Returns native image size of the video stream from the filter graph identified by “UserID” cookie

IMultiVMR9RenderEngine

Objects exposing this interface perfom actual rendering of video streams to a single Direct3D device. While IMultiVMR9Wizard provides surface management requested by VMR9, IMultiVMR9RenderEngine performs actual rendering of the image to the screen, asynchronously from VMR9 calls. Rendering engine manages two objects: MultiVMR9MixerControl and MultiVMR9UILayer. By default, UI layer is not implemented, and the default mixer control acts similar to IVMRMixerControl9 in a single graph. CLSID_MultiVMR9RenderEngine is the default implementation of this interface.

Initialize	This method must be called before connecting “this” rendering engine to the wizard. Client application has to call this method only if it supplies customized rendering engine (in IMultiVMR9Wizard::Initialize). Otherwise, wizard creates and initializes default instance of this object. Also, this method can be used to supply customized Mixer Control and User Interface Layer to the rendering engine.
Terminate	Releases all internal resources of the rendering engine. Client application must call this method only if it supplies customized rendering engine. Client applications must call this method after destroying the wizard.
Render	A call-back function invoked from a separate thread with frequency specified by IMultiVMR9RenderEngine::SetFrameRate. Method performs actual rendering of the scene in several steps: scene initialization (such as IDirect3Ddevice9::BeginScene); composition of the mixer objects (IMultiVMR9MixerControl::Compose); rendering of the mixer objects (IMultiVMR9MixerControl::Render); composition of the UI layer if it is present (IMultiVMR9UILayer::Compose); rendering of the UI Layer (IMultiVMR9UILayer::Render), and scene completion (IDirect3Ddevice9::EndScene and IDirect3Ddevice9::Present). It also updates actual frame rate.
GetUILayer	Returns pointer to IMultiVMR9UILayer actively used by the render engine
SetFrameRate	Sets desired frames-per-second rate, multiplied by 100
GetFrameRate	Returns actual frame rate (multiplied by 100) from the last rendering
GetFrameRateAvg	Returns actual frame rate (multiplied by 100) averaged over the lifetime of the rendering object
GetMixingPrefs	Returns mixing preferences flags originally set in the method Initialize. By default, mixing preferences flags are ignored
SetWizardOwner	Wizard calls this method to inform redering engine that it is the owner of the object. Also, when wizard is terminating, it sets the owner of its child rendering engine to NULL.
GetWizardOwner	Returns pointer to IMultiVMR9Wizard, the parent wizard object that currently owns this rendering engine
GetMixerControl	Returns pointer to IMultiVMR9MixerControl actively used by the render engine
Get3DDevice	Returns pointer to IDirect3DDevice9 actively used for rendering
GetVideoWindow	Returns handle to the video window.

IMultiVMR9MixerControl

Object that implements this interface, performs scene composition and defines geometry of Direct3D primitives. By default, this interface is similar to IVMRMixerControl9 interface of a single graph. Provide custom implementation of this interface to add animation, special effects, or to inject video into 3D environent. CLSID_MultiVMR9MixerControl is the default implementation of this interface.

Initialize	This method must be called before connecting “this” mixer control to the rendering engine. Client application has to call this method only if it supplies customized mixer control (in IMultiVMR9RenderEngine::Initialize). Otherwise, rendering engine creates and initializes default instance of this object.
SetRenderEngineOwner	Rendering engine calls this method to inform mixer control that it is the owner of the object. Also, when rendering engine is terminating, it sets the owner of its child mixer control to NULL.
GetRenderEngineOwner	Returns pointer to IMultiVMR9RenderEngine, the parent rendering engine object that currently owns this mixer control
Compose	This method is called from IMultiVMR9RenderEngine::Render. Method performs calculation of primitives’ coordinates that provides animation, special effects, etc.

Render	This method is called from IMultiVMR9RenderEngine::Render. Method performs actual rendering of primitives. Please note that this method must not call IDirect3Ddevice9::BeginScene or IDirect3Ddevice9::EndScene, because it is done inside rendering engine.
AddVideoSource	Wizard calls this method when it attaches a new filter graph. Upon receiving this call, mixer control must allocate objects (primitives) associated with new video source.
DeleteVideoSource	Wizard calls this method when it detaches some filter graph. Upon receiving this call, mixer control must release objects (primitives) associated with this video source.
GetIdealOutputRect	Returns “ideal” location of the video image, that is, normalized rectangle that covers maximum space of the render target and maintains original aspect ratio of the image. Please note that normalized rectangle is calculated with respect to the render target size (rather than the client size of the video window). Use IMultiVMR9RenderEngine::Get3DDevice and IDirect3Ddevice9::GetRenderTarger to transform the “render target” space into the “client area of the window” space.

The following methods are implemented in the default instance of IMultiVMR9MixerControl and basically repeat functionality of IVMRMixerControl9. The only difference is that cookie “UserID” is used instead of the number of input pin. When these methods do not make sense in the application-specific implementation of IMultiVMR9MixerControl, just make them returning S_OK.

GetOutputRect

SetOutputRect

GetZOrder

SetZOrder

GetAlpha

SetAlpha

GetBackgroundColor

SetBackgroundColor

IMultiVMR9UILayer

Object that exposes this interface, represents UI- related Direct3D primitives (such as buttons, menus, sliders etc.) as well as user input to the video window (Win32 messages). User Interface layer is always drawn over the scene rendered by IMultiVMR9MixerControl. By default, rendering engine does not use this object, so if client application implies UI interaction from the video window, it should provide implementation of IMultiVMR9UILayer and send it to IMultiVMR9RenderEngine::Initialize.

Initialize	This method can be called to initialize internal objects before connecting “this” UI layer to the rendering engine.
ProcessMessage	This method is called by the video window message processing thread. Method can modify state of the UI Layer and send commands to client application, rendering engine, or the wizard.
Render	This method is called from IMultiVMR9RenderEngine::Render. Method performs actual rendering of the primitives of UI layer. Please note that this method must not call IDirect3Ddevice9::BeginScene or IDirect3Ddevice9::EndScene, because it is done inside rendering engine
SetRenderEngineOwner	Rendering engine calls this method to inform UI Layer that it is the owner of the object. Also, when rendering engine is terminating, it sets the owner of its child UI Layer to NULL.
GetRenderEngineOwner	Returns pointer to IMultiVMR9RenderEngine, the parent rendering engine object that currently owns this UI Layer

3.2. Samples

MultiSample

Samples\cpp\DirectShow\VMR9\MultiVMR9\MultiPlayer.

This MFC-based sample implements default functionality of MultiVMR9.dll, so make sure that MultiVMR9.dll is built and registered with regsvr32.exe. MultiPlayer looks much like the property page of the VMR9 in the mixing mode; the only evident difference is that this application supports several filter graphs at once. Application starts with configuration dialog.

1. Controls related to the application:

• To add a new source, hit '+' (‘plus’) button and select some media file. Application has not been tested on mpeg2 streams.
• Use 'FPS' slider to set desired frames per second rate. Default value is 30; possible range is [5; 100]
• To exit application, hit 'Close'
• To change the color of the background, hit 'brush' button and select the color

2. Controls related to the selected item of the combo box:

• To delete the source, hit '-' (‘minus’)
• Use sliders to change x,y positions and vertical and horizontal sizes
• Use slider to change z-order of the source (0 is foremost)
• Use slider to change alpha-level of the source
• Use 'Timeline' slider and buttons 'Play' and 'Pause' to control state and seek the stream.
• Use 'rulers' button to restore aspect ratio of selected source

Notes: Application does NOT support multiple monitors. Desktop must be set to the 32 bit mode.

For other details, see the source code.

GameSample

Samples\cpp\DirectShow\VMR9\MultiVMR9\GamePlayer.

This sample illustrates how to integrate video powered by VMR9 into (some imaginary 3D game) Direct3D environment. Application uses the same MultiVMR9.dll, but overrides IMultiVMR9MixerControl and IMultiVMR9UILayer. Before building this sample, make sure that MultiVMR9.dll is built and registered with regsvr32.exe. Please note that slow machines and older video cards can demonstrate significant performance drop. Machines like Pentium-III 500MHz and up with videocards no older that 2 years would be enough.

Usage

1. Run GamePlayer

2. Select a few (3-5) mpeg1 or AVI movies and hit "start".

3. New window should appear; a girl walking along the wall.

4. In a few seconds, girl starts passing videos on the wall.

5. To focus on the highlighted movie, hit the yellow button in the lower right corner.

6. To stop the presentation, go back to initial dialog and hit "stop" and "exit".

For other details, see the source code.