Cybermato Consulting - Video Analysis with Avidemux

Professor Grünbaum of the University of Washington's Oceanography Department does a lot of video analysis of critters (ranging from microorganisms to regular fish) swimming around in tanks, to gather data for use in models of individual and group-level behavior.  Much of the work that I've done relevant to this is to modify Avidemux, an open-source video-editing program, to add video analysis capability and other enhancements needed by (or helpful to) Professor Grünbaum's research.  A list of the major changes appears a little further down on this page, but in general, he needed the software to be able to identify where each critter was in each frame, and output a list of locations for each frame.  Then his software (Tracker3D) reads that list and computes each critter's path through 3D-space.  Pretty cool stuff!

This page has some information about Professor Grünbaum's research, but it predates our work with Avidemux, and refers to the very slow and painful process that was required before he started using Avidemux:

http://peduncleii.ocean.washington.edu/~t3d/Tracker3D/

This page refers to an older version of Avidemux - I first modified it in 2005, and then did considerably more work on it in 2007 (see below).  So this is for the 2005 version, in which some things were different:

http://peduncleii.ocean.washington.edu/~t3d/VideoAnalysis/avidemux_docs/
http://peduncleii.ocean.washington.edu/~t3d/VideoAnalysis/avidemux_docs/avidemux_particle_detection.pdf

Most of my changes to Avidemux (listed below) have already been merged into the main svn tree, and many of the changes were included in the recent 2.4 release (which included binaries for Mac and Windows as well as Linux).  However, the latest version that incorporates all of my changes is available via this page:

Download Avidemux modified for Video Analysis

All binaries on the page above are for 32-bit x86 Linux.  (I'll provide some 64-bit builds some time soon.)  The FC7 ones are reported to work just fine on FC6 and Ubuntu.  For FC5 and earlier, you'll need an FC5 build.  The latest versions weren't built for FC5 because no one cared any more - if you need an FC5 build, contact me via the contact form (link at the bottom of the page).  It is not possible to build it for FC4, as Avidemux (2.4 and beyond, on which this work is based) requires newer libraries than are available for FC4.

The changes made to Avidemux as part of this project include:
 As most of the operations being performed in the filters are per-pixel, per-frame, over thousands of video frames, performance was a concern.  I used various techniques to improve performance, including some fairly sophisticated use of C++ templates in the Swiss Army Knife filter.  The general idea there was to write classes (had to be classes, not functions - you don't get the desired inlining behavior and thus the desired performance benefit if you use functions) for each operation (pixel minus input, input divided by pixel, etc.) and a template function for each input type (rolling average, pre-computed average, convolution, etc.) which took as an argument an object of a template parameter type, which would be passed an object of one of the operation classes.  The net effect was to inline every operation into the function for every input type - effectively instantiating 10 copies of each of the 4 input types (and actually it's worse even than that, because of the optional histogram and some other options such as lookahead on the rolling average).  This has the unfortunate effect of inflating the code size quite a bit, but again, we're talking about a loop that iterates over EVERY PIXEL in EVERY FRAME.  The alternative was to either inline the code by hand (which would be a maintenance nightmare and have the same effect on code size) or to incur a function call via a function pointer inside the loop - ouch!  Fortunately this is an already-large desktop application and so code size was not a concern in comparison to the performance benefit.  The only other downside is the giant switch statement that invokes the appropriate combinations of operation & input - no way to avoid that, either, without losing the performance benefit.  If performance wasn't such an issue (or if we weren't talking about 720x480 = 345600 iterations per frame), we could have instantiated an operation object in one switch and then passed it to the appropriate input function in a separate, subsequent switch, which would be much cleaner-looking.

(Sorry, I know the preceding paragraph is hard to follow.  I'll rewrite this with example code and such when I'm awake and have more time.)

I used the same template trick to support an optional histogram (where a histogram of incoming and outgoing pixel values is output every N frames, so you can tune the parameters to avoid pixel value clipping due to saturation at the high or low end) without impacting performance when the histogram option is not selected.

Here's the file I'm talking about:  ADM_vidSwissArmyKnife.cpp

Another interesting bit was the problem of detecting "shape" and "orientation", particularly for particles that were not very large (5-20 pixels).  I'll add a writeup of that here soon, too.

I will try to post a complete demo kit here (with input files and instructions) in the next few days.  Send me a note (see contact info at the bottom of the page) if you're interested and it's not here yet.

If you are a researcher (or anyone else) who would like to fund additional work on Avidemux for video analysis or other purposes, please get in touch with me via the contact form (link below).

You can also see more information about Chris MacGregor and other Cybermato Consulting projects,
or send me a note via the Cybermato Consulting contact form.