Difference between revisions of "Stabilo video stabilizer"

From Noah.org
Jump to navigationJump to search
m
 
(10 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
[[Category:Engineering]]
 
[[Category:Engineering]]
 
[[Category:Free_Software]]
 
[[Category:Free_Software]]
 +
[[Category: Imaging]]
 +
 +
= Notes =
 +
 +
It appears that this has all been written. Which is fine. It was a fun exercise, but I am no mathematician.
 +
 +
Go here to see a nice matrix of what software is available (the best appears to be [http://auricle.dyndns.org/ALE/ ALE for Anti-Lamenessing Engine, supposedly]): [http://chdk.wikia.com/wiki/Stacking_Software Wikia list of Stacking Software]
 +
 +
There is a tool called [http://www.luxal.eu/resources/hdr/hdrprep/ hdprep] which performs very similar registration transformations that my script does. Hdprep uses the [http://auricle.dyndns.org/ALE/ ALE] tool.
 +
 +
I appears that what I am trying to do (crudely and slowly) is perform two operations: Image Registration and Motion Vector Calculation.
 +
 +
# Image registration. Image registration gives the transformation operations necessary to maximize the correlation between image A and image B. The operations considered are linear transformations (rotation, scaling, shear) and translation (or shift). In my case, I consider only rotation and X and Y translation. I believe that my mathematician friends call this an "Affine transformation". Note that an image from a camera lens can also transform an image on other ways that I do not consider here. For example, I do not consider "scale". This would happen if you are walking towards a subject (or zooming) while filming. For extremely fast rates of zooming it may not be possible to register the images.
 +
# Motion vector calculation: The X, Y shift and rotation angle give a motion vector. This describes how to convert image A into image B so that the new transformed image has a maximal correlation. This basically undoes any motion of the camera between image A and image B.
 +
-- Noah 2010-10-21
  
 
== stabilo -- video stabilization filter ==
 
== stabilo -- video stabilization filter ==
Line 8: Line 23:
 
This is a working alpha project at the moment.
 
This is a working alpha project at the moment.
  
This requires [PIL_patches | a patch to PIL Imaging-1.1.6] to add an RMS function, difference_rms(), to the ImageChops module. This function will calculate the RMS difference between two images. I could have implemented this in pure Python, but writing the function in C yielded a 10X speed increase. This video filter processing is slow to begin with, so any whole factors of speed increase is a great help.
+
This requires [[PIL_patches | a patch to PIL Imaging-1.1.6]] to add an RMS function, difference_rms(), to the ImageChops module. This function will calculate the RMS difference between two images. I could have implemented this in pure Python, but writing the function in C yielded a 10X speed increase.
 +
 
 +
== samples ==
 +
 
 +
This unprocessed video was shot from a paraglider. This shows a bad, shaky video -- I was a thousand feet up, holding the camera with one hand and steering the glider away from the rocks with the other hand.
 +
 
 +
  [http://www.noah.org/pictures/stabilo_input.avi before]
 +
 
 +
This processed video shows how the algorithm smooths the video. In this example, offset compensation is unrestricted so you can see how the video will wrap around as the algorithm tries to chase a moving video.
 +
 
 +
  [http://www.noah.org/pictures/stabilo_output_unrestricted.avi after unrestricted]
  
 
== the source ==
 
== the source ==
Line 14: Line 39:
 
The code is fairly simple. Currently the algorithm does not check if rotation of the image will improve smoothness. Rotations are very slow and don't help much. I have code stubbed out to add this feature as a final enhancement once a good translation offset is found.
 
The code is fairly simple. Currently the algorithm does not check if rotation of the image will improve smoothness. Rotations are very slow and don't help much. I have code stubbed out to add this feature as a final enhancement once a good translation offset is found.
  
The algorithm is fairly stupid -- it's a O(N^2) search. I have a few ideas for improving this. One way would be by sampling different offsets and ignoring regions that hurt smoothness and then returning to regions that improved smoothness.
+
The algorithm is fairly stupid -- it's a O(N^2) search. I have a few ideas for improving this. One way would be by sampling different offsets and ignoring regions that hurt smoothness and then returning to regions that improved smoothness... Apparently, using the FFT to convert to the frequency domain makes registration a much simpler problem; however, the FFT operation is relatively slow and would have to be done on every frame. Perhaps registering just a small sample space in the middle of the image would work well enough. Once registration is found for a small subsection you then apply the same transform to the entire image.
  
 +
Click to download: [http://www.noah.org/downloadsvn.php?src=file:///home/svn/src/python/stabilo/stabilo.py stabilo.py]
 
<include svncat src="file:///home/svn/src/python/stabilo/stabilo.py" highlight="python" />
 
<include svncat src="file:///home/svn/src/python/stabilo/stabilo.py" highlight="python" />

Latest revision as of 00:44, 16 April 2014


Notes

It appears that this has all been written. Which is fine. It was a fun exercise, but I am no mathematician.

Go here to see a nice matrix of what software is available (the best appears to be ALE for Anti-Lamenessing Engine, supposedly): Wikia list of Stacking Software

There is a tool called hdprep which performs very similar registration transformations that my script does. Hdprep uses the ALE tool.

I appears that what I am trying to do (crudely and slowly) is perform two operations: Image Registration and Motion Vector Calculation.

  1. Image registration. Image registration gives the transformation operations necessary to maximize the correlation between image A and image B. The operations considered are linear transformations (rotation, scaling, shear) and translation (or shift). In my case, I consider only rotation and X and Y translation. I believe that my mathematician friends call this an "Affine transformation". Note that an image from a camera lens can also transform an image on other ways that I do not consider here. For example, I do not consider "scale". This would happen if you are walking towards a subject (or zooming) while filming. For extremely fast rates of zooming it may not be possible to register the images.
  2. Motion vector calculation: The X, Y shift and rotation angle give a motion vector. This describes how to convert image A into image B so that the new transformed image has a maximal correlation. This basically undoes any motion of the camera between image A and image B.

-- Noah 2010-10-21

stabilo -- video stabilization filter

This script processes successive frames of a video stream and finds the offset between each frame. It then adjust each frame of the output to minimize the offset. The result is smoother video.

This is a working alpha project at the moment.

This requires a patch to PIL Imaging-1.1.6 to add an RMS function, difference_rms(), to the ImageChops module. This function will calculate the RMS difference between two images. I could have implemented this in pure Python, but writing the function in C yielded a 10X speed increase.

samples

This unprocessed video was shot from a paraglider. This shows a bad, shaky video -- I was a thousand feet up, holding the camera with one hand and steering the glider away from the rocks with the other hand.

 before

This processed video shows how the algorithm smooths the video. In this example, offset compensation is unrestricted so you can see how the video will wrap around as the algorithm tries to chase a moving video.

 after unrestricted

the source

The code is fairly simple. Currently the algorithm does not check if rotation of the image will improve smoothness. Rotations are very slow and don't help much. I have code stubbed out to add this feature as a final enhancement once a good translation offset is found.

The algorithm is fairly stupid -- it's a O(N^2) search. I have a few ideas for improving this. One way would be by sampling different offsets and ignoring regions that hurt smoothness and then returning to regions that improved smoothness... Apparently, using the FFT to convert to the frequency domain makes registration a much simpler problem; however, the FFT operation is relatively slow and would have to be done on every frame. Perhaps registering just a small sample space in the middle of the image would work well enough. Once registration is found for a small subsection you then apply the same transform to the entire image.

Click to download: stabilo.py <include svncat src="file:///home/svn/src/python/stabilo/stabilo.py" highlight="python" />