QuickTime GWorlds
Volume Number: 11
Issue Number: 9
Column Tag: Apple Technology
QuickTime and Offscreen GWorlds 
Manipulate QuickTime video frame by frame.
By Kent Sandvik, Apple Developer Technical Support
Note: Source code files accompanying article are located on MacTech CD-ROM orsource code disks.
This article explains how to write QuickTime video frames (samples) to offscreen
GWorlds, and blit these frames back to the main screen, while avoiding flickering and
frame drops if possible. The techniques shown should provide you with the
information needed to implement various applications that manipulate QuickTime video
frames offscreen.
For instance, you may want to draw something on top of each QuickTime frame
displayed, as quickly as possible. Or you want to modify a movie’s pixmaps before they
are displayed. In all cases, we don’t want any flickering due to redrawing.
You may want to step through each video frame one at a time, modifying the
frame’s pixmap, and you are not worried about frame drops or other time critical
conditions. In other words, it is fine for you to browse and modify frames at your own
pace because the QuickTime movie is not playing.
Maybe you want to be informed every time QuickTime has drawn a frame, and
perform additional operations unrelated to the frame itself. In other words, you are
not interested in the actual GWorld of the movie.
We will cover these three cases: how to draw to an offscreen pixmap (remember,
QuickTime works only with 32 bit Color Quickdraw), modify the contents and blit this
back to the main screen as quickly as possible; how to display and change frame
information one frame at at time without playing the movie; and how to be notified
when QuickTime has drawn a frame. The sample code also has functions that measure
the slowdown that additional drawing imposes on playback, and measures how long the
drawing operations take.
We assume that the reader has basic knowledge about GWorld and CopyBits
techniques.
Fast Video Frame Drawing
When a QuickTime movie plays, it is of utmost importance that as little as possible is
happening beside displaying movie frames on the screen. Time spent doing other
activities will provide fewer CPU cycles for QuickTime to read, arrange and display
media samples. If the application spends time doing additional activities, it will
directly impact the QuickTime movie’s playback performance.
So, whatever we do when we annotate the video frames, we’d better do it quickly.
The less we do, the better.
QuickTime will always use only one GWorld for drawing. Most importantly, it
uses this single GWorld for decompression purposes, building pixmaps based on
sample information in the video track. For instance, if we are dealing with a
decompressor that understands temporal compression (where only the changes from
frame to frame are recorded), the pixmaps are built using temporal information, and
QuickTime will not redraw every pixel in the GWorld. This means that we can’t draw
directly on top of existing QuickTime video frame information. If we do, our drawing
information will be retained in the GWorld, not erased at all, and we get the dreaded
“smearing” effect.
In order for us to control the drawing, or have access to the specific GWorld that
QuickTime is using, we need to detour QuickTime to use a known GWorld. We can think
in terms of layered drawing environments. We also want to be informed when the
frame is drawn into this GWorld. When we’re informed by QuickTime, we can
manipulate the GWorld contents, or blit the contents back to the main window port and
then do our drawing operations on top of the copied image (this is a layer oriented
approach).
To avoid flickering, we need to copy the original video track’s GWorld contents to
another GWorld, do our drawing in this second GWorld, and then finally blit this
GWorld pixmap to the main screen. This sounds like a lot of work, but you can run
tests to measure if performance is suffering from two CopyBits calls.
QuickTime 1.6 introduced the a method to specifying which GWorld a track will
be drawn into. The function SetTrackGWorld takes a specified GWorld we have created
and tells the QuickTime toolbox to draw a specified video track into a specified GWorld.
In addition, with SetTrackGWorld, we can specify that a specific callback function is
called when a frame is transferred from a track into this GWorld. When this track
transfer callback function is triggered, we can do additional graphics operations on the
frame pixmap (or anything else).
As long as we don’t spend too much time inside a track transfer callback, copying
from a specified GWorld back to the main screen is reasonably fast. However, we need
to always worry about speed. The more work we do before the operation, the better.
For instance, we should preallocate any buffers, handles and anything else that does not
change dynamically during a track transfer callback. Memory allocation may take
more time than expected, especially if the Memory Manager needs to compact the heaps
in order to find memory blocks of required size.
Similarly, the less work we do inside the callback, the better. As we will use
CopyBits at the end of the track transfer callback, it is important to know how to
optimize the CopyBits function.
Preallocation, Precalculation
Before the track transfer callback we create the needed GWorlds and other buffers, and
also try to precalculate any values that we know won’t change while we are inside the
track transfer callback. To some degree smart compilers will also optimize away
unnecessary operations, but it makes sense to do this ourselves as well.
For example, we want to calculate how far we are into the movie based on how
many frames have been drawn. We need the total duration of the movie for the
calculations; we can’t avoid the fact that we need to calculate the value inside the track
transfer callback. However, we could store away the duration value of the movie
before the track transfer callback is operating.
CopyBits Optimization
CopyBits optimization is very well documented in the Tech Note “QD21 Of Time and
Space and _CopyBits”. Here’s a short summary, a comparison of the techniques
covering our example of QuickTime video frame blitting:.
We should try to copy the smallest possible area; the less bytes CopyBits need to
move, the better. Now, in the case of QuickTime movies with defined movie box
dimensions we can’t avoid this, we need to copy the whole area from the GWorld to the
final destination. However, if we know we want to copy parts of the movie, we could
cheat and just copy the rectangular area we are interested in.
In most cases we use the srcCopy mode when copying video frames from the
offscreen to the main port, and this is the fastest transfer mode. Using other modes,
such as transparent or blend mode, will slow down the copy process considerably. Note
that adding the dither mode will really slow down CopyBits. Various codecs, such as
Cinepak, will dither anyway (from 24 bit mode down to 8 or 16 bit depth of the
monitor GWorld).
It makes sense to specify the background and the foreground colors for the
current port, so that the background color is white and the foreground color is black.
This will provide a hint to CopyBits so that it won’t do colorization operations while
copying information. We also assume that the GDevice pixel map’s color table has
white in the first position and black in the last position (this is the normal setup of a
Macintosh color table).
The alignment of pixels in the source pixel map is also very important. If
CopyBits needs to realign the pixels it will slow down the copy process considerably.
For instance, 32-bit CPU environments are at their fastest when they can transfer
long words aligned on long-word boundaries in memory. This is the reason why
AlignWindow is a very important QuickTime call; AlignWindow will move the specified
window to the nearest optimal alignment position on the screen (or screen video
memory position). This means that if this is done with the destination GWorld, then
CopyBits will operate very fast. QuickTime movies are mostly also aligned for
four-byte values (movie rectangle dimension are divided by four) so this will help as
well. If the movie does not have this property (check this with for instance
MoviePlayer), then it’s time to re-crop the movie so that we have such sizes.
Sometimes cropping of the original movie might cause odd sized movie rectangle
values.
We should also, if possible, use the smallest pixel depth. If we want to operate in
256 color mode we should stick to 8-bit pixel depths. Note that various QuickTime
codecs are optimized for various pixel depths. For instance, Cinepak is optimized for
24-bit color mode, and it will dither to 16 and 8-bit screen depths. One issue is if we
should stick to the original image depth of the movie and build a GWorld based on this
pixel depth, and then later copy from this GWorld to the monitor bit depth GWorld. Or
if we should create the offscreen GWorld with the same bit depth as the monitor bit
depth. The sample code has both approaches so you could do more thorough tests to
narrow down what case is the fastest one in your application.
We should also stick to the same source and destination rectangles so that
CopyBits does not need to scale the copied image. In most cases we want to use the
movie box rectangle sizes, as we want to blit the whole movie back to the monitor
window.
Color QuickDraw expects a color table attached to every indexed pixel map (8 bit
mode). Color tables specify what color each pixel value in the pixel map represents.
Color QuickDraw often uses the ctSeed field of the color table data structure as a fast
check for color table equality. If two color tables have the same ctSeed, then Color
QuickDraw often assumes that their color table contents are equal. We could make use
of this feature and copy the ctSeed value (long word) from the original color table to
the destination color table, and this way CopyBits assumes that the two color tables are
identical and will copy the pixels directly without any color mapping. This will
improve CopyBits performance in 8-bit mode. We will copy the existing color table
from the GDevice, and use it to create the two offscreen GWorlds.
It also makes sense to make an estimate about the slowest platform the
application is aimed for. This way we will know if our offscreen operations will work
properly for a targeted entry level system or not. One cardinal sin is to implement
most of the application using a high performance PowerPC development system, and
then do the system testing using a low performance CPU system. If possible test out
the graphics drawing ideas with low end machines before you complete the project.
Setting Up the Track Transfer Proc
The basic code we use for setting up the test environment creates a window, gets a
QuickTime movie from a file, adjusts the movie box (rectangle) values, and makes
sure that our movie GWorld points to the window port, or:
SetMovieGWorld(gMovie, (GWorldPtr)gWindow, NULL);
After this we fetch the duration of the movie and find the first suitable video
track. Most QuickTime movies have just one video track, so we assume that this is the
case in this test application. We also fetch the pixel depth of the movie using a specific
call from the QuickTime utilities function library (more about that at the end of this
article). Or we could get the monitor bit depth from the monitor GDevice structure.
Note that in to speed things up, we have left out the error testing code in the example:
gMovieDuration = GetMovieDuration(gMovie);
firstVideoTrack = GetMovieIndTrackType(gMovie, 1, VideoMediaType,
movieTrackMediaType);
firstVideoMedia = GetTrackMedia(firstVideoTrack);
mediaPixelDepth = QTUGetVideoMediaPixelDepth(firstVideoMedia, 1);
monitorPixelDepth = (**(**aSavedGDevice).gdPMap).pixelSize;
Then we create our GWorlds we will use for the offscreen handling (one for the