View Morphing

Introduction

What is view morphing?
What is this project about?
How we should obtain the pictures?

Program Implementation Issues

Image Input
Correspondence Point Picking
Estimation of Projection Matrix
Complete Correspondence estimation
Prewarp, Morph, Postwarp
Hole Filling

Example of Set 0
Example of Set 1

Introduction

What is view morphing

View morphing is a technique used to generate images at different view point along the 2 optical center of the given 2 images. In differ from traditional morphing in that traditional morphing only handle 2D transformation and hence distortion will be produced when changes in viewpoint along 3D space is not parallel.
View morphing is a technique to take in the consideration of 3D space transformation in terms of 2D prewarping and postwarping and thereby reducing the possible amount of distortion. In fact, the prewarping step is to make the 2 images plane parallel to each other such that position as well as intensity could reduce back to linear relationship, thus we could interpolate the position and color intensity just like normal morphing.
Therefore, view morphing is in fact a sequence of steps which could help reducing the 3D distortion from 2D morphing, by first prewarping the images such that the images are on the same plane or parallel plane then normal morphing technique is applied. The output images are then postwarp back to the plane it should be and the result will give much more 3D sense than the normal morphing as well as a possible reductation of 3D transformation distort to the 2D images.

What is this project about?

This project is to implement a tools for helping the user the create view morphing The user can load up 2 image to the program and then do the correspondence picking for the 2 images. The program will then help the user to create view morphing images, whereas the user will be asked the 2 cameras orientation information and the program will use these information to estimate the projection matrix for the 2 images.
Once the program got the projectiopn matrix, it will prewarp the images and then perform image morphing each image will be postwarped back to it's projection plane and the images will be outputed to the user.

How we should shoot the pictures?

The images could be obtained from cameras and/or web pictures. Since this project is best for human face morphing. We suggest that the image be obtained on the web. In fact, for easier and better result the images about the person should not be too complicated.
For this project purpose we are mainly focus on morphing from one image, i.e. to obtain the mirror images of the input image and then morph it.
To do mirror images morphing, we need to make sure that the mirrored image will not be too different from the original one, to ensure the best result the person's face feature should best be symmetrical and hence the features will not appear in one side and not the other side, and thus creating some kind of contracting effects.

Implementation Issues

Image Input

The program will take in image file of BMP format, the user can view the image, move the image around in the image working area and can also zoom in and zoom out within the image working area. The current 2 images is highlighted by "RED" and BLUE" rectangle, where the "BLUE" is the first first selected and the "RED" one is the second selected image, doubling clicking the left button on an image make it the "BLUE" one and doubling clicking the right button make it the "RED" one.

Correspondence Point Picking

The correspondence point picking is done by first selecting the 2 current working set of image by doubling clicking the left mouse button for the first image and the right button for the second image. After choosing the current 2 images. The user can add correspondence point by clicking on the Add Corr. Point button. Then the user can move the correspondence cursor to the 2 points, the user wanted to associate them.
The user could also save the correspondence points sets to a file as well as loading correspondence point from file, hence the user can modify the correspondence easily.

Estimation of Projection Matrix

When the user have finished picking the correspondence between the 2 images. He/she could start the morphing by giving the camera, orientation information, which include the x,y,z coordinate of 2 camera and the rotation angles. All these information should be given in terms of the world coordinate and the world coordinate should be chosen such that the 2 images plane will be prewarp to the x,y plane of the world coordinate. Thus, we need to ensure that the prewarping resulting from that projection will minimize the distortion generated. The details of how to choose the world coordinate could be obtained from the following paper:
View Morphing(by Steven M.Seitz, CharlesR. Dyer)

Complete Correspondence Estimation

The complete correspondence is estimated base on the sparse set of points given by the user as image correspondence.
Since we need to have complete correspondence for morphing, we need to perform complete correspondence estimation. This is done by the using a slab window to find out the closest left and right points within the slab windows and then interpolate the corresponding x coordinate using the x coordinate distance ratio between the left and right point. The corresponding y coordinate is done using similar way, we use a slab width to find out the closest top and bottom points and then interpolate the corresponding y coordinate using the y coordinate distance ratio between them.

Hence we have the following:

P=Point in which no direct correspondent point is found.
PL=Left Closest Point within slab window
PR=Right Closest Point within slab window
PT=Top Closest Point within slab window
PB=Bottom Closest Point within slab window
DL=horizontal distance between P and PL
DR=horizontal distance between P and PR
DT=vertical distance between P and PT
DB=vertical distance between P and PB
C()=correspondent point function, given P given the corresponding point in the other image.

C(P).x=C(PL).x * (DR/(DL+DR) + C(PR).x * (DL/(DL+DR)
C(P).y=C(PT).y * (DB/(DT+DB) + C(PB).x * (DT/(DT+DB)

We need to have a bounding box for the images such that when no correspondent point is found, the bounding edge will be used to interpolate the correspondent point.

Prewarp, Morph, Postwarp

After the projection matrix is estimated, the program will use it to prewarp the images to the x,y plane and then perform morphing using linear interpolation of positioning and color intensity. The morphed images is then postwarp back to the image plane of where the morphed image should be.
Hole Filling

Hole Filling

Since forward mapping is use for this the view morphing. Hence there are some holes after the steps. We perform a hole filling technique to fill back the holes generated. I define a Hole Filling window size and for each point with hole. All the points with the window will be used to help compute the color intensity of the hole point.

Here is the method:
I(P) = Intensity of the P
D(P) = distance between the point P and the hole point
I(Hole Point)= ( I(P0)*(1/D(P0)) + I(P1)*(1/D(P1)) ....... ) /(1/D(P0) + 1/ D(P1) + .....)
where P0, P1..... are the points within the window and is not a hole.

Results (Lisa)

Camera 0 position:
X=-100
Y=0
Z=100
Rotation in Y Axis = 15
Camera 1 position:
X=-100
Y=0
Z=-100
Rotation in Y Axis = -15
Focal Length = 300
I want to thank Tsz Ming Wong for his dataset, I used his correspondence points for this test sample.
Input Images  Movie Results (Ryder)

Camera 0 position:
X=-100
Y=0
Z=100
Rotation in Y Axis = 15
Camera 1 position:
X=-100
Y=0
Z=-100
Rotation in Y Axis = -15
Focal Length = 450
Input Images  Movie 