Cybermato Consulting - Video Analysis with Avidemux
Professor Grünbaum of the University
of Washington's Oceanography Department does a lot
of video analysis of critters (ranging from microorganisms to regular
fish) swimming around in tanks, to gather data for use in models of
individual and group-level behavior. Much of the work that
I've done relevant to this is to modify Avidemux,
an open-source video-editing program, to add video analysis capability
and other enhancements needed by (or helpful to) Professor
A list of the major changes appears a little further down on
this page, but in general, he needed the software to be able to
identify where each critter was in each frame, and output a list of
locations for each frame. Then his software (Tracker3D) reads that list and computes each critter's path through 3D-space. Pretty cool stuff!
This page has some information about Professor Grünbaum's research, but
it predates our work with Avidemux, and refers to the very slow and
painful process that was required before he started using Avidemux:
This page refers to an older version of Avidemux - I first modified it
in 2005, and then did considerably more work on it in 2007 (see below).
So this is for the 2005 version, in which some things were different:
Most of my changes to Avidemux (listed below) have already been merged
into the main svn tree, and many of the changes were included in the
recent 2.4 release (which included binaries for Mac and Windows as well
as Linux). However, the latest version that incorporates all of
my changes is available via this page:
Download Avidemux modified for Video Analysis
binaries on the page above are for 32-bit x86 Linux. (I'll
provide some 64-bit builds some time soon.) The FC7 ones are
reported to work just fine on FC6 and Ubuntu. For FC5 and
earlier, you'll need an FC5 build. The latest versions weren't
built for FC5 because no one cared any more - if you need an FC5 build,
contact me via the contact form (link at the bottom of the page).
It is not possible to build it for FC4, as Avidemux (2.4 and
beyond, on which this work is based) requires newer libraries than are
available for FC4.
The changes made to Avidemux as part of this project include:
As most of the operations being performed in the filters are per-pixel,
per-frame, over thousands of video frames, performance was a concern.
I used various techniques to improve performance, including some
fairly sophisticated use of C++ templates in the Swiss Army Knife
filter. The general idea there was to write classes (had to
be classes, not functions - you don't get the desired inlining
behavior and thus the desired performance benefit if you use functions)
for each operation (pixel minus input, input divided by pixel, etc.)
and a template function for each input type (rolling average,
pre-computed average, convolution, etc.) which took as an argument an
object of a template parameter type, which would be passed an object of
one of the operation classes. The net effect was to inline every
operation into the function for every input type - effectively
instantiating 10 copies of each of the 4 input types (and actually it's
worse even than that, because of the optional histogram and some other
options such as lookahead on the rolling average). This has the
unfortunate effect of inflating the code size quite a bit, but again,
we're talking about a loop that iterates over EVERY PIXEL in
EVERY FRAME. The alternative was to either inline the code
by hand (which would be a maintenance nightmare and have the same
effect on code size) or to incur a function call via a function pointer
inside the loop - ouch! Fortunately this is an already-large
desktop application and so code size was not a concern in comparison to
the performance benefit. The only other downside is the giant
switch statement that invokes the appropriate combinations of operation
& input - no way to avoid that, either, without losing the
performance benefit. If performance wasn't such an issue (or if
we weren't talking about 720x480 = 345600 iterations per frame), we
could have instantiated an operation object in one switch and then
passed it to the appropriate input function in a separate, subsequent
switch, which would be much cleaner-looking.
- Adding several new "filters" to modify video frames to eliminate "noise" and everything unrelated to the swimming critters:
filter to compute the average value of each pixel across an entire
video and write the averages to a file (used by the "Swiss Army Knife"
- A filter to allow "erasing" (forcing to a
specified value) of arbitrary areas of a frame, with a
"paint-program"-like user interface for ease of use,and with the ability to erase different areas in different ranges of frames (i.e., different parts of the video).
filter affectionately known as the "Swiss Army Knife", because it can
apply any of a list of operations (pixel minus input, pixel divided by
input, input minus pixel, etc.) to any of a list of inputs (a rolling
average of each pixel's value with user-specified parameters, or a
pre-computed average across the whole video (output by the filter noted
above), or a constant floating-point value, or a convolution using a
kernel read from a user-specified file), with saturation of the results
at the ends of the pixel value range (0-255) and arbitrary scaling and
biasing of the results.
- A "threshold" filter to force all
pixels whose values are inside a specified range to black (or white, as
selected by the user), and the other pixels to the opposite color
(i.e., white or black).
- Adding an output step that
identifies each critter (referred to as a "particle", as in many cases
they are microorganisms), traces the outer boundary of it, computes the
size, centroid location, shape, bounding box, and apparent orientation,
and outputs that information.
- Adding the ability to preview the
output of any filter in the filter list while configuring any filter -
previously, the preview was only of the output of the filter being
configured, and so it was much harder to adjust parameters since you
had to guess the effect on filters "down the line", as you couldn't
actually see it.
- Implementing support for hardware jog/shuttle controllers (e.g., Contour ShuttlePRO);
if you have a controller that isn't supported, contact me (see form
below) regarding adding support. Donating a example of the
controller you want supported to Professor Grünbaum's lab is sufficient payment, and gives me access to it for testing.
various bugs, such as issues with video whose resolution is large
enough that it needs to be scaled down in configuration dialog windows
with video preview enabled).
(Sorry, I know the
preceding paragraph is hard to follow. I'll rewrite this with
example code and such when I'm awake and have more time.)
the same template trick to support an optional histogram (where a
histogram of incoming and outgoing pixel values is output every N
frames, so you can tune the parameters to avoid pixel value clipping
due to saturation at the high or low end) without impacting performance
when the histogram option is not selected.
Here's the file I'm talking about: ADM_vidSwissArmyKnife.cpp
interesting bit was the problem of detecting "shape" and "orientation",
particularly for particles that were not very large (5-20 pixels).
I'll add a writeup of that here soon, too.
I will try to
post a complete demo kit here (with input files and instructions) in
the next few days. Send me a note (see contact info at the bottom
of the page) if you're interested and it's not here yet.
If you are a researcher (or anyone else) who would like to fund
additional work on Avidemux for video analysis or other purposes,
please get in touch with me via the contact form (link below).
You can also see more information about Chris
MacGregor and other Cybermato
or send me a note via the Cybermato
Consulting contact form.