Using Video Mixing Renderer 9 with Multiple Filter Graphs

 

1. What Is New

VMR7 contains all the necessary functionality to manage several filter graphs; this task can be accomplished by providing VMR7 with custom allocator-presenter (as shown in sample VMRMulti, see Samples\cpp\DirectShow\VMR\VMRMulti). However, introduction of VMR9 essentially simplifies this task:

2. Using Custom Allocator-Presenter to Manage Several Filter Graphs

2.1. Developing Custom Allocator-Presenter for VMR9

 

Development of custom (user provided) Allocator-Presenter for the renderer has not changed much since VMR7, but there are several differences. Below is the “how to” scenario, it is implemented in the sample VMR9Allocator (See Samples\cpp\DirectShow\VMR9\VMR9Allocator). First, developer of the client application has to implement custom allocator-presenter (A/P), an object derived from IVMRSurfaceAllocator9 and IVMRImagePresenter9. See DirectX 9.0 SDK help for more information on custom allocator-presenter (see “Supplying a Custom Allocator-Presenter for VMR9”). Figure 1 shows relationship between VMR9 filter and its Allocator-Presenter.

2.2. Using Custom Allocator-Presenter

 

When client application supplies a custom A/P for VMR9, there are several rules to be followed: custom A/P must be thread-safe; when advising custom A/P, filter graph must be stopped and VMR9 filter must be in the rendeless mode; and all its inputs pins must be disconnected. Below is the typical scenario of using custom A/P.

 

1.      Create instance of the filter graph (CLSID_FilterGraph).

2.      Create instance of Video Mixing Renderer 9 (CLSID_VideoMixingRender9) and add it to the filter graph.

3.      Set VMR9 to the Renderless mode (see IVMRFilterConfig9::SetRenderingMode) .

4.      Query VMR9 filter for interface IVMRSurfaceAllocatorNotify9.

5.      Create instance of custom allocator-presenter.

6.      Associate your filter graph with some cookie (any DWORD_PTR value that uniquely identifies the filter graph).

7.      Advice custom A/P to the VMR9 (see IVMRSurfaceAllocatorNotify9::AdviseSurfaceAllocator).

8.      Inform custom A/P about the allocator notifier (see IVMRSurfaceAllocator9::AdviseNotify).

9.      Build the rest of the graph, render, and run.

10.  When destroying the filter graph, stop the graph, disconnect all the pins of VMR9 filter and advise it with NULL surface allocator

2.3. Using Custom Allocator-Presenter in Multi-Graph Environment

 

To provide multi-graph functionality to the scenario above, few modifications are needed. Custom A/P should contain references to all the connected filter graphs. Each filter graph is uniquely identified with a cookie (parameter DWORD_PTR dwUserID in methods of IVMRSurfaceAllocator9 and IVMRImagePresenter9). Since several filter graphs are asynchronous to each other, it is advised to perform rendering in a separate thread with some fixed user-defined frame rate. In this case, method IVMRImagePresenter9::PresentImage just copies images from VMR9 filters to some private textures. This causes some performance drop, but leaves filter graphs independent on each other. It is advised to set the frame rate to the highest frame rate of all connected video streams to prevent frame dropping. At last, it would be logical if such custom allocator-presenter have methods Attach and Detach, to connect and disconnect involved filter graphs. Figure 2 illustrates relationship between connecting filter graphs and custom A/P.

 

3. MultiVMR9.dll Helper Library

 

DirectX 9.0 SDK samples include sources for MultiVMR9.dll (see Samples\cpp\DirectShow\VMR9\MultiVMR9). This helper library demonstrates how to implement a user-provided (customized) allocator-presenter for the Video Mixing Renderer 9 in a multigraph environment. It seemed logical to extract several abstraction layers that could be overriden by client applications:

·         Wizard, an object that implements actual allocation-presenting and contains references to all connected filter graphs. The wizard is responsible for interaction with client application as well as connecting and disconnecting graphs. Object is implemented by IMultiVMR9Wizard interface.

·         Rendering Engine, an object that contains all Direct3D data and performs actual rendering asynchronously from calls to IVMRImagePresenter9::PresentImage. Object is implemented by IMultiVMR9RenderEngine interface.

·         Mixer Control, an object that performs scene composition and defines geometry of Direct3D objects. By default, this object acts as IVMRMixerControl9 in a single graph (see sample MultiPlayer, Samples\cpp\DirectShow\VMR9\MultiVMR9\MultiPlayer), but can be overriden to perform any sort of 3D transformation (as in the sample GamePlayer, Samples\cpp\DirectShow\VMR9\MultiVMR9\GamePlayer). Object is implemented by IMultiVMR9MixerControl interface.

·         User Interface Layer, an object that is responsible for interaction with user in video playback window. User Interface Layer is always drawn over the scene rendered by mixer control. Message processing function of the video window sends all the appropriate messages to this object. Object is implemented by IMultiVMR9UILayer interface. By default, MultiVMR9.dll does not use this object, so if client application needs user interaction, it must provide IMultiVMR9UILayer object.

 

Figure 3 illustrates structure of MultiVMR9.dll.

3.1. Interfaces

 

See Samples\cpp\DirectShow\VMR9\MultiVMR9\DLL\MultiVMR9.idl

IMultiVMR9Wizard

 

Objects exposing this interface are responsible for managing several VMRs with single rendering environment. Object that implements this interface must also implement IVMRSurfaceAllocator9 and IVMRImagePresenter9. Otherwise method Initialize should return E_NOTIMPL. CLSID_MultiVMR9Wizard is the default implementation of this interface.

 

Initialize

Configure and initialize internal structures. This method must be called before using the wizard; otherwise consequent calls to the wizard will return VFW_E_WRONG_STATE. Also, use this method to supply customized Rendering Engine.

Terminate

Releases all internal resources of the wizard; disconnects all connected filter graphs. This method must be called before destroying the wizard.

Attach

Attaches filter graph to the wizard

Detach

Detaches filter graph from the wizard

VerifyID

Verifies if there is any attached filter graph with specified “UserID” cookie

GetGraph

Returns pointer to IFilterGraph of the connected filter graph identified by “UserID” cookie

GetRenderEngine

Returns pointer to IMultiVMR9RenderEngine actively used by the wizard

GetMixerControl

Returns pointer to IMultiVMR9MixerControl actively used by the wizard

GetTexture

Returns pointer to IDirect3DTexture9 that contains “ready-to-render” image from the filter graph identified by “UserID” cookie

GetVideoSize

Returns native image size of the video stream from the filter graph identified by “UserID” cookie

 

IMultiVMR9RenderEngine

 

Objects exposing this interface perfom actual rendering of video streams to a single Direct3D device. While IMultiVMR9Wizard provides surface management requested by VMR9, IMultiVMR9RenderEngine performs actual rendering of the image to the screen, asynchronously from VMR9 calls. Rendering engine manages two objects:  MultiVMR9MixerControl and MultiVMR9UILayer. By default, UI layer is not implemented, and the default mixer control acts similar to IVMRMixerControl9 in a single graph. CLSID_MultiVMR9RenderEngine is the default implementation of this interface.

 

Initialize

This method must be called before connecting “this” rendering engine to the wizard. Client application has to call this method only if it supplies customized rendering engine (in IMultiVMR9Wizard::Initialize). Otherwise, wizard creates and initializes default instance of this object. Also, this method can be used to supply customized Mixer Control and User Interface Layer to the rendering engine.

Terminate

Releases all internal resources of the rendering engine. Client application must call this method only if it supplies customized rendering engine. Client applications must call this method after destroying the wizard.

Render

A call-back function invoked from a separate thread with frequency specified by IMultiVMR9RenderEngine::SetFrameRate. Method performs actual rendering of the scene in several steps: scene initialization (such as IDirect3Ddevice9::BeginScene); composition of the mixer objects (IMultiVMR9MixerControl::Compose); rendering of the mixer objects (IMultiVMR9MixerControl::Render); composition of the UI layer if it is present (IMultiVMR9UILayer::Compose); rendering of the UI Layer (IMultiVMR9UILayer::Render), and scene completion (IDirect3Ddevice9::EndScene and IDirect3Ddevice9::Present). It also updates actual frame rate.

GetUILayer

Returns pointer to IMultiVMR9UILayer actively used by the render engine

SetFrameRate

Sets desired frames-per-second rate, multiplied by 100

GetFrameRate

Returns actual frame rate (multiplied by 100) from the last rendering

GetFrameRateAvg

Returns actual frame rate (multiplied by 100) averaged over the lifetime of the rendering object

GetMixingPrefs

Returns mixing preferences flags originally set in the method Initialize. By default, mixing preferences flags are ignored

SetWizardOwner

Wizard calls this method to inform redering engine that it is the owner of the object. Also, when wizard is terminating, it sets the owner of its child rendering engine to NULL.

GetWizardOwner

Returns pointer to IMultiVMR9Wizard, the parent wizard object that currently owns this rendering engine

GetMixerControl

Returns pointer to IMultiVMR9MixerControl actively used by the render engine

Get3DDevice

Returns pointer to IDirect3DDevice9 actively used for rendering

GetVideoWindow

Returns handle to the video window.

 

IMultiVMR9MixerControl

 

Object that implements this interface, performs scene composition and defines geometry of Direct3D primitives. By default, this interface is similar to IVMRMixerControl9 interface of a single graph. Provide custom implementation of this interface to add animation, special effects, or to inject video into 3D environent. CLSID_MultiVMR9MixerControl is the default implementation of this interface.

 

Initialize

This method must be called before connecting “this” mixer control to the rendering engine. Client application has to call this method only if it supplies customized mixer control (in IMultiVMR9RenderEngine::Initialize). Otherwise, rendering engine creates and initializes default instance of this object.

SetRenderEngineOwner

Rendering engine calls this method to inform mixer control that it is the owner of the object. Also, when rendering engine is terminating, it sets the owner of its child mixer control to NULL.

GetRenderEngineOwner

Returns pointer to IMultiVMR9RenderEngine, the parent rendering engine object that currently owns this mixer control

Compose

This method is called from IMultiVMR9RenderEngine::Render. Method performs calculation of primitives’ coordinates that provides animation, special effects, etc.


 

Render

This method is called from IMultiVMR9RenderEngine::Render. Method performs actual rendering of primitives. Please note that this method must not call IDirect3Ddevice9::BeginScene or IDirect3Ddevice9::EndScene, because it is done inside rendering engine.

AddVideoSource

Wizard calls this method when it attaches a new filter graph. Upon receiving this call, mixer control must allocate objects (primitives) associated with new video source.

DeleteVideoSource

Wizard calls this method when it detaches some filter graph. Upon receiving this call, mixer control must release objects (primitives) associated with this video source.

GetIdealOutputRect

Returns “ideal” location of the video image, that is, normalized rectangle that covers maximum space of the render target and maintains original aspect ratio of the image. Please note that normalized rectangle is calculated with respect to the render target size (rather than the client size of the video window). Use IMultiVMR9RenderEngine::Get3DDevice and IDirect3Ddevice9::GetRenderTarger to transform the “render target” space into the “client area of the window” space.

 

The following methods are implemented in the default instance of IMultiVMR9MixerControl and basically repeat functionality of IVMRMixerControl9. The only difference is that cookie “UserID” is used instead of the number of input pin. When these methods do not make sense in the application-specific implementation of IMultiVMR9MixerControl, just make them returning S_OK.

 

GetOutputRect

SetOutputRect

GetZOrder

SetZOrder

GetAlpha

SetAlpha

GetBackgroundColor

SetBackgroundColor

 

IMultiVMR9UILayer

 

Object that exposes this interface, represents UI- related Direct3D primitives (such as buttons, menus, sliders etc.) as well as user input to the video window (Win32 messages). User Interface layer is always drawn over the scene rendered by IMultiVMR9MixerControl. By default, rendering engine does not use this object, so if client application implies UI interaction from the video window, it should provide implementation of IMultiVMR9UILayer and send it to IMultiVMR9RenderEngine::Initialize.

 

Initialize

This method can be called to initialize internal objects before connecting “this” UI layer to the rendering engine.

ProcessMessage

This method is called by the video window message processing thread. Method can modify state of the UI Layer and send commands to client application, rendering engine, or the wizard.

Render

This method is called from IMultiVMR9RenderEngine::Render. Method performs actual rendering of the primitives of UI layer. Please note that this method must not call IDirect3Ddevice9::BeginScene or IDirect3Ddevice9::EndScene, because it is done inside rendering engine

SetRenderEngineOwner

Rendering engine calls this method to inform UI Layer that it is the owner of the object. Also, when rendering engine is terminating, it sets the owner of its child UI Layer to NULL.

GetRenderEngineOwner

Returns pointer to IMultiVMR9RenderEngine, the parent rendering engine object that currently owns this UI Layer

 

3.2. Samples

MultiSample

Samples\cpp\DirectShow\VMR9\MultiVMR9\MultiPlayer.

 

This MFC-based sample implements default functionality of MultiVMR9.dll, so make sure that MultiVMR9.dll is built and registered with regsvr32.exe. MultiPlayer looks much like the property page of the VMR9 in the mixing mode; the only evident difference is that this application supports several filter graphs at once. Application starts with configuration dialog.

 

1. Controls related to the application:

 

 

2. Controls related to the selected item of the combo box:

 

 

Notes: Application does NOT support multiple monitors. Desktop must be set to the 32 bit mode.

For other details, see the source code.

 


GameSample

Samples\cpp\DirectShow\VMR9\MultiVMR9\GamePlayer.

 

This sample illustrates how to integrate video powered by VMR9 into (some imaginary 3D game) Direct3D environment. Application uses the same MultiVMR9.dll, but overrides IMultiVMR9MixerControl and IMultiVMR9UILayer. Before building this sample, make sure that MultiVMR9.dll is built and registered with regsvr32.exe. Please note that slow machines and older video cards can demonstrate significant performance drop. Machines like Pentium-III 500MHz and up with videocards no older that 2 years would be enough.

 

Usage

 

1. Run GamePlayer

2. Select a few (3-5) mpeg1 or AVI movies and hit "start".

3. New window should appear; a girl walking along the wall.

4. In a few seconds, girl starts passing videos on the wall.

5. To focus on the highlighted movie, hit the yellow button in the lower right corner.

6. To stop the presentation, go back to initial dialog and hit "stop" and "exit".

 

For other details, see the source code.