Image Mosaicing

Introduction

What is image mosaicing?
What is image mosaicing for?
What is this project about?
How we should shoot the pictures?

Program Implementation Issues

Image Input
Correspondence Point Picking
Building Homography from Correspondence Points(Manual Registration)
Improving the Homography by Harry Shum's error minimization method(Automatic Improving Registration)
Warp the image and composite with another image

Results

Example of Mosaic with images that had rotation in all the x,y,z axis.
An example of obvious automatic improvement

Download

Download the program

Introduction

What is image mosaicing?

Imaging mosaicing is being doen in this project such that images taken by normal camera can be used to create a larger field of view using a image mosaicing program. The program is basically a tools which help you to find out the corresponding camera angles that you used to take the image.

What is image mosaicing for?

Image mosaicing not only allow you to create a large field of view using normal camera, the result image can also be used for texture mapping of a 3D environment such that users can view the surrounding scene with real images.

What is this project about?

This project is to implement a tools for helping the user the create image mosaic. The user can take images using a hand held camera and then take images at different angles with some overlapping area, then the user can use the program to load the image in and then make the mosaic.

How we should shoot the pictures?

Before we take the pictures, we should know that there are some limitation and some important points about image taking for image moasic.
Since all the formula and all the assumption for this project is the pin hole camera model, hence any camera which behave very different from the pin hole camera model will create undesirable result.
Another, important thing is that when taking the pictures, we must try to take the images without translation, only rotation along the x,y,z axis is allowed.

Implementation Issues

Image Input

The program will take in image file of BMP format, the user can view the image, move the image around in the image working area and can also zoom in and zoom out within the image working area. The current 2 images is highlighted by "RED" and BLUE" rectangle, where the "BLUE" is the first first selected and the "RED" one is the second selected image, doubling clicking the left button on an image make it the "BLUE" one and doubling clicking the right button make it the "RED" one.

Correspondence Point Picking

The correspondence point picking is done by first selecting the 2 current working set of image by doubling clicking the left mouse button for the first image and the right button for the second image. After choosing the current 2 images. The user can add correspondence point by clicking on the Add Corr. Point button. Then the user can move the correspondence cursor to the 2 points, the user wanted to associate them.
The user could also save the correspondence points sets to a file as well as loading correspondence point from file, hence the user can modify the correspondence easily.

Building Homography from Correspondence Points(Manual Registration)

The homography for the warping of the image can be done by clicking on the build homography from coresspondence points button. This function can only be successful, if the current 2 images has been chosen and there are at least 4 correspondence points added between the 2 images. The homography is estimated by the solving a set of 2n equations, where n is at least 4.
This process is actually using 2n equations for solving the 8 parameters unknown for the homography matrix and since it is an over determined equation. We need an equation solver for this purpose. I have used a least square equation solver which is implemented by David Smith (drs) at Carnegie-Mellon University (1980).
The homography estimated is in fact a transformation matrix which transform the "RED" image to the "BLUE", and can be used for warping and composite for later purpose.
The homography computed, can be saved, loaded and the user can also invert the homography such that the warping can be done in the other way around.

Improving the Homography by Harry Shum's error minimization method(Automatic Improving Registration)

I have also implemented Harry Shum's idea for error minimization method for registering the images automatically in an semi-automatic sequence. In fact, because I didn't optimize the algorithm I have implemented this automatic registion function runs a bit slow. The user can choose to use this function for running N iteration for updating the D matrix incrementally. Or the user can update the D matrix once at a time and see how the D is changed. For the single step method, the user could see the result after each of the update and can desire when to stop.
However, because the algorithm only consider image color intensity for minizing the intensity difference, I find that it will not be quite accurate if the intensity of the 2 images is not very exact. In case when the 2 image's intensity is not quite exact, we might have a visible error in the composite result.
After the updates, the user could choose to keep the update or choose not to keep it.

Warp the image and composite with another image

After the homography is computed, the user can warp the image and composite them together. The warping will be done by warping the "RED" image to the "BLUE" one. Since forward warping will create holes for the composite, the warping is done by sampling points from the "RED" images using the inverse of the homography.
The program will first compute the bounding rectangle of the image and then for each of the pixels it uses the inverse homography matrix to find out which pixel to take from the "RED" images. After scanning all the pixel the image will be warped to the new projection.
After the image is warped, we will composite the 2 images together. During the composite, we need to interpolate color intensity for blending the images. This is done by first computing the center of gravity for each of the images.
Then the color of those overlapping pixel is computed as follow: I=I0*d1(d0+d1) + I1*d0(d0+d1)
where
I0 is the color intensity of image 0
I1 is the color intensity of image 1
d0 is the distance between the center of gravity of image 0 to the pixel
d1 is the distance between the center of gravity of image 1 to the pixel

Results (Hall Room View)

Input Image 0


Image 0

Input Image 1


Image 1

Output Image 10


Image 1 warped on Image 0 -> Image 10

Input Image 2


Image 2

Output Image 210


Image 2 warped on Image 10 -> Image 210

Input Image 3


Image 3

Output Image 3210


Image 3 warped on Image 210 -> Image 3210

Input Image 4


Image 4

Output Image 43210


Image 4 warped on Image 3210 -> Image 43210

Input Image 5


Image 5

Output Image 543210


Image 5 warped on Image 43210 -> Image 543210

Input Image 6


Image 6

Output Image 6543210


Image 6 warped on Image 543210 -> Image 6543210

Input Image 7


Image 7

Output Image 76543210


Image 7 warped on Image 6543210 -> Image 76543210

Results (Auto Improve by Harry's method)

Input Image 0

Input Image 1

Output Image 10 before automatic improvement registration

Output Image 10 after automatic improvement registration


This automatic update has been run for <30 times for this case, and the result is end with a convergent because no further update will change the pictures.

Download

The program
Please note that the program has only be tested on Win95, Win98 and WinNT 4.0
The program is written in VC++ Version 5.0, it has not been tested on VC6.
Please press the "start" button before going on for other function.