Pulldown Removal with 32-bit videoIO on Windows

by Gerald Dalley
April 2008

This document describes how to recover the original progressive 24fps frames from certain types of 60 field-per-second videos using the 32-bit videoIO toolbox for Matlab on Windows.  

Background

We assume that the reader is already familiar with the the basic concepts of interlacing and pulldown.
As of early 2008, most consumer-grade high-definition camcorders use a mode called "24PF", "PF24", "24F", "24PsF", or a few other names. These modes exist for some legacy reasons and are not really the same thing as a true 24p recording (despite what the marketing materials suggest). For this discussion, we will focus on the 24pf mode of the Canon HG10 camcorder.

For this camcorder, (a) whole 1920x0180 progressive video frames are captured at 24 frames per second, then they are (b) scaled down to 1440x1080, (c) broken up into fields, (d) processed with 2-3 pulldown, and (e) encoded using an AVCHD codec. This means the video is stored as if it were 60 fields-per-second, interlaced (60i) using a somewhat new and very complex encoding.

If one were to simply decode this video and use it as-is, 3 out of every 5 frames would have interlacing artifacts in them and the comb artifacts for these frames will be roughly twice as bad as they would be had the video been captured as true 60i. When processing videos in Matlab, one often would much prefer recovering the original 24p frames.

To recover the original progressive frames, one must perform something called "pulldown removal", "reverse pulldown", "inverse pulldown", "inverse telecine", or "ivtc".  Unfortunately, this is not a trivial task.

In this document we describe the easiest robust solution we have found for pulldown removal.  The specific details may need to be adjusted for the user's videos that are not obtained from a Canon HG10 or similar camcorder.  This method will only work on Windows with 32-bit applications (unless you can find or build 64-bit ports of AviSynth, DeComb, and FDecimate). Feel free to experiment and to tweak the instructions to try using ffdshow for decoding instead of CoreAVC.

Required Software

Basic Filter Configuration

We need to make sure that the filters we have just installed are the defaults. Launch the DirectShow Filter Manager:

Haali Media Splitter Merit

Find the "Haali Media Splitter". Double-click on it to launch the filter properties.  If the "Filter merit" is below 0x00800001, then change it to that value and click the "Set new merit" button.

Close the dialog.

The Haali splitter is the only one we know that handles transport streams correctly (ArcSoft, CyberLink, and most others incorrectly report the AVCHD video stream as MPEG2).

CoreAVC Configuration

Open the filter properties of the "CoreAVC Video Decoder".  Press the "Filter settings" button.  Place YUY2 as the preferred output format (move that one to the top of the list).  The DeComb AviSynth filter requires either YUY2 or YV12 and YUY2 is higher-quality.  Change the deinterlacing setting to "None (Weave)".  If the deinterlacer is not disabled, it will destroy some of the information we wish to recover.  Check "Crop 1088 to 1080".  AVCHD videos must have a height that is evenly divisible by 16, so 1080 videos are padded to 1088.  This checkbox reverses the padding.  Check "Preferred decoder".

When you dismiss the settings dialog, verify that the filter merit is greater than 0x00800000

These changes will make CoreAVC the default and have it output non-deinterlaced frames.

ffdshow Video Decoder Merit

Change the "ffdshow Video Decoder"'s merit to something below 0x00800000 (0x00600000 is a good value).  We need to make sure that the CoreAVC decoder gets chosen instead of ffdshow.  If you want to test ffdshow, just set its merit to be larger than CoreAVC's.

ffdshow raw video filter Merit

Change the "ffdshow raw video filter"'s merit (not the "ffdshow Video Decoder"'s) to a value larger than 0x00800001.  We used 0xFFFF0000.

This will cause the ffdshow filter to be automatically inserted whenever a "Video Renderer" is used, such as when troubleshooting in GraphStudio.

If you are using ffdshow for decoding instead of CoreAVC, make sure the raw filter's merit is 0 so it will never be automatically instantiated. The ffdshow decoder will already perform the filtering and we don't want to filter the video twice.

Reboot

Sometimes the filter merit settings do not take effect without a reboot.

ffdshow Configuration

We will now configure the "ffdshow raw video filter" to do the pulldown removal for Canon HG10 videos.  If you are using the "ffdshow Video Decoder" for decoding instead of CoreAVC, make the changes to that filter instead of the raw video filter.

Creating a Profile

In the DirectShow Filter Manager, open the filter settings dialog for the "ffdshow raw video filter".  We will be creating a new preset for these videos so that other videos that use ffdshow will not be affected.

  1. Select the "Profiles / Preset settings" in the leftmost listbox.
  2. On the right, press the "New" button to create a new profile.
  3. Press the "Rename" button and choose the name "hg10pf24coreavc" or some other descriptive name.
  4. Make sure "default" is still bold. If it isn't, double click on "default" to make it really be the default, then click on "hg10pf24coreavc" again so the remaining modifications will apply to our new profile.
  5. Check the "Automatic preset loading" checkbox.
  6. Check the "Try to load preset from file first" checkbox.

Autoload Configuration

Press the "Preset autoload conditions..." button.  Here we will set some heuristics to let the filter know on which videos it should apply the pulldown removal.  Adjust these as appropriate for your videos.

  1. Select the "On all conditions match (AND)" radio button.
  2. Check the "on movie dimensions match" checkbox.
  3. Put 1440 as the minimum and maximum width.
  4. Put 1080 as the minimum height.
  5. Put 1088 as the maximum height.
  6. Check the "on a DirectShow filter presence" checkbox and add "Core AVC Video Decoder" to its textbox.  This is a trick to avoid processing 30pf MPEG2 streams that do not need pulldown removal (because 30pf decodes the same as true 30p).
  7. Check the "on frame rate match" checkbox and add "fps>=29.9 AND fps<=30" to its textbox. 24pf will be reported as 30p.  If you have AVCHD 30p or 30pf content, you may need to figure out some additional heuristics.

Resize & aspect

The combination of CoreAVC and ffdshow often result in the video being rescaled and letterboxed if care is not taken.  We can prevent the resizing from happening by forcing the size to be what we want (1440x1080). 

On the right pane in the top left corner, make sure that "hg10pf24coreavc" is selected (instead of "default" or some other profile). 

  1. On the left treeview, check the "Resize & aspect" entry.   On the right again,
    1. check the "Resize"  checkbox,
    2. select the "Specify horizontal and vertical size" radio button,
    3. type "1440" x "1080" in the "New Size" area, and
    4. choose "4":"3" as the aspect ratio.  
  2. On the rightmost inner panel,
    1. choose the "Resize always" radio button,
    2. check the "Process pixel aspect ratio internally" checkbox, and
    3. enter "4" x "3" as the "PAR of output device". 
  3. In the lower panel, choose "No aspect ratio correction". 

The properties box should look like this:

Our white treeview on the left of the dialog is likely simpler than yours since we removed most of the filters using the "Show / hide filters" option.  This is purely a personal preference (unless you wish to use those filters).

Pulldown Removal via AviSynth

We are now finally ready to configure the pulldown removal.  We assume that the AviSynth and the DeComb and FDecimate filters have been installed as previously described

  1. On the right pane in the top left corner, make sure that "hg10pf24coreavc" is selected (instead of "default" or some other profile). 
  2. On the left treeview, check the "AviSynth" entry. 
  3. On the right again, check all of the checkboxes that are not already checked:
    1. "AviSynth",
    2. "Add ffdshow video source",
    3. all of the input colorspaces (or at least YUY2), and
    4. "Buffer back/ahead". 
  4. Set the buffer values to "8" and "8". 
  5. The large textbox is a space for AviSynth scripts.  Enter the following script (it's okay to remove the comments):
# Tell AviSynth it's a 30fps source.  It doesn't matter if it's 
# really an NTSC 29.97fps source.  The rates are only used 
# internally.
AssumeFPS(30) 
 
# Assume that the top field is first (bottom first is very rare)
AssumeTFF()
 
# Use the DeComb filter without extra postprocessing in 2-3
# pulldown reversal mode.  This will reconstruct 30fps progressive
# frames.  For clean sources, every 5th frame will be a duplicate.
Telecide(guide=1,post=0)
 
# Drop the duplicate frames.  For clean sources, every 5th frame
# will be dropped because 30/(30-24)=5.
FDecimate(rate=24,show=false)

The dialog should now look like this (shown with the script comments removed):

Feel free to explore the rest of the ffdshow options as desired.  Note that turning on the onscreen display ("OSD" in the left white treeview) can sometimes confuse for the DeComb filter.

GraphStudio Testing

Before testing videoIO, it is useful to test the configuration with GraphStudio.  Launch GraphStudio and load the tests/hg10-pf24-clip2.mts clip.

If you are using CoreAVC, make sure that that filter is instantiated as well as the "ffdshow raw video filter". 

If instead you are using the "ffdshow Video Filter", make sure that the raw filter is not also instantiated. 

Press the play and/or single step buttons to view the results.  You should see a clean video without any comb artifacts like those in the image at the very top of this article.

videoIO Testing

Launch Matlab and try running the following code:

% The file of interest is in the tests/ directory
cd(fullfile(videoIODir, 'tests'));
file     = 'hg10-pf24-clip2.mts';
% Here's an early frame that shows combing 
frameNum = 125;
 
% Original video without pulldown removal
vrOrig   = videoReader(file, 'preciseFrames',-1);
% With pulldown removal 
vrIvtc   = videoReader(file, 'preciseFrames',-1, ...
                       'dfilters','ffdshow raw video filter');
 
% show the frame of interest in the original video
seek(vrOrig, frameNum)
figure(1); imshow(getframe(vrOrig)); title('with combing');
 
% show the equivalent frame after removal (w/ framerate conversion)
seek(vrIvtc, frameNum*4/5)
figure(2); imshow(getframe(vrIvtc)); title('pulldown removed');
 
% cleanup
vrOrig = close(vrOrig);
vrIvtc = close(vrIvtc);

There should be a very obvious difference between the two figures when looking at the gray car. 

Note that with version 1.6.2.5 of CoreAVC, 'preciseFrames',-1 must be used in the videoReader constructor because seeking is partially broken.  The CoreAVC team is working on fixing this issue, so future versions are likely to work.

Extra Reading

For more information on reverse pulldown, see: