Transcript Document

ECE 738 Paper presentation
Paper: Active Appearance Models
Author: T.F.Cootes, G.J. Edwards and C.J.Taylor
Student: Zhaozheng Yin
Instructor: Dr. Yuhen Hu
Date: Feb. 14 2005
Note: some slides copyrighted by the original authors
1
Papers
• T.F.Cootes, G.J. Edwards and C.J.Taylor. "Active
Appearance Models", IEEE PAMI, Vol.23, No.6,
pp.681-685, 2001
• T.F.Cootes, G.J. Edwards and C.J.Taylor. "Active
Appearance Models", in Proc. European
Conference on Computer Vision 1998 Vol. 2, pp.
484-498, Springer, 1998. (Best paper prize)
2
Flexible models
•
•
•
•
Statistical Shape Models
Active Shape Models (ASM)
Combined Appearance Models
Active Appearance Models (AAM)
Shape model
ASM
AAM
3
Flexible models
• Shape
Shape is the geometric
information invariant to a
particular class of
transformations
(translation+rotation+scaling)
• Appearance
4
Applications
•
Flexible models can be used to:
–
–
–
•
Locate examples of structures in new images
Classify objects found in images
Filter images to pick out interesting features
Practical problems:
Face recognition, industrial inspection and
medical image analysis
5
Flexible models
•
•
•
•
Statistical Shape Models
Active Shape Models (ASM)
Combined Appearance Models
Active Appearance Models (AAM)
Shape model
ASM
AAM
6
Statistical Shape Models
•Given sets of training images
build a statistical
shape model
•Each shape in the training set is represented by a set of n
labeled landmark points, which must be consistent from
one shape to the next.
Ex. The outline of a hand is represented by 72 labeled
points
6
5
4
3
2
1
7
Statistical Shape Models
•Each shape is represented by a 2n*1 vector
X  ( x1,...,xn, y1,..., yn )
•Using Principal Component Analysis (PCA) or eigen
analysis, the shape model is
X  X  Pb
where P is a 2n*t matrix whose columns are unit vectors
along principle axes or basis vector
b is a t*1 vector of shape parameters or weight
Ex. Vary the first three
parameters of the shape
vector, b, one at a time
8
Aligning Two Shapes
• Procrustes analysis:
– Find transformation which minimizes
| x1  T (x 2 ) |
2
– Resulting shapes have
• approximately the same scale and orientation
9
Aligning a Set of Shapes
• Generalized Procrustes Analysis
– Find the transformations Ti which minimise
2
|
m

T
(
x
)
|

i
i
– Where
1
m   Ti (x i )
n
– Under the constraint that
| m | 1
10
Dimensionality Reduction
b1
p1
x
x
x  x  p1b1
11
Dimensionality Reduction
• Data lies in subspace of reduced dim.
x  x  p1b1    p nbn
• However, for some t,
i
b j  0 if j  t
(Variance of b j is  j )
t
i
12
Statistical Shape Models
•Another example
Shape of the facial structures with 68 points
13
Flexible models
•
•
•
•
Statistical Shape Models
Active Shape Models (ASM)
Combined Appearance Models
Active Appearance Models (AAM)
14
Active Shape Models
• Suppose we have a statistical shape model
– Trained from sets of examples
• How do we use it to interpret new images?
• Use an “Active Shape Model”
• Iterative method of matching model to
image
15
Active Shape Models (ASM)
• Assume we have an initial estimate for the pose
and shape parameters (eg the mean shape).
X  X  Pb
16
Active Shape Models (ASM)
• Iterative algorithm
– Look along normals through each model point to find the
best local match for the model of the image appearance at
that point (eg strongest nearby edge)
– Update the pose and shape parameters to best fit the model
instance to the found points
– Repeat until convergence
Initial pos
5th iterations
convergence
17
ASM Search Overview
• Local optimization
• Initialize near target
– Search along profiles for best match,X’
– Update parameters to match to X’.
( X i , Yi )
18
Active Shape Models (ASM)
• Performance improvement
(Multi-resolution implementation/coarse-tofine approach)
we start searching on a coarse level of a
Gaussian image pyramid, and progressively
refine. This leads to much faster, more
accurate and more robust search.
19
Flexible models
•
•
•
•
Statistical Shape Models
Active Shape Models (ASM)
Combined Appearance Models
Active Appearance Models (AAM)
20
Combined Appearance Models
• Idea:
Statistical Shape Model models the shape
change of an object
construct a
similar statistical model to represented the
intensity variation across a region
(Think: skeleton and muscle)
21
Combined Appearance Models
• Method:
Given a set of training images, labeled with land
mark points, we can use image warping to deform
each image so that the object has the mean shape,
then build a statistical model of the grey-levels
across the object.
Ex. The central image is the
mean
22
Building Appearance Models
• For each example extract shape vector
Shape, x = (x1,y1, … , xn, yn)T
• Build statistical shape model, x  x  Psb s
23
Building Appearance Models
• For each example, extract texture vector
Shape, x = (x1,y1, … , xn, yn)T
Warp to
mean
shape
Texture, g
24
Warping texture
•
Problem:
–
•
Given corresponding points in two images,
how do we warp one into the other?
Two common solutions
1. Piece-wise linear using triangle mesh
2. Thin-plate spline interpolation
25
Interpolation using Triangles
Controlpoints: ( xi , yi )
Warpedpoints: ( xi ' , yi ' )
Region of interest enclosed by triangles.
Moving nodes changes each triangle
Just need to map regions between two triangles
26
Barycentric Co-ordinates
c
c'
x
a
a'
b
x  a  b  c
x'
b'
x'  a' b'c'
    1
x is inside the triangleif
0  α  1 and 0  β  1
27
Building Texture Models
• For each example, extract texture vector
Warp to
mean
shape
Texture, g
• Normalise vectors (as for eigenfaces)
• Build eigen-model g  g  Pg b g
28
Combined Models
• Shape and texture often correlated
– When smile, shadows change (texture) and
shape changes
• Learning this correlation leads to more
compact (and specific) model
29
Combined Appearance Models
x  x  Psb s
In this paper:
g  g  Pg b g
x  x  Qsc
g  g  Qgc
Varying c changes both
shape and texture
30
Flexible models
•
•
•
•
Statistical Shape Models
Active Shape Models (ASM)
Combined Appearance Models
Active Appearance Models (AAM)
31
Active Appearance Models
• Suppose we have a statistical appearance
model
– Trained from sets of examples
• How do we use it to interpret new images?
• Use an “Active Appearance Model”
• Iterative method of matching model to
image
32
Interpreting Images
Place model
in image
Measure
Difference
Update Model
Iterate
Active Appearance Models (AAM)
• AAM vs. ASM
The Active Appearance Model (AAM) is a
generalization of the widely used Active
Shape Model approach, but uses all the
information in the image region covered by
the target object, rather than just that near
modeled edges.
34
Quality of Match
• Residual difference: r(p)  I m (p)  I im (p)
• p : all parameters, eg p  (c, X c , Yc , s, )
• Ideally find and optimize p(p|r)
p(r | p) pT(p)
Bayes rule : p(p | r )E(p)  r(p) r(p)
p(r )
• Cannot usually know p(r)
35
Quality of Match
• Usually attempt to maximize
p(r | p) p(p)
(1)
• This is equivalent to maximizing
log p(r(p) | p)  log p(p)
(2)
• Which is equivalent to minimizing
E (p)   log p(r(p))  log p(p) (3)
36
Quality of Match
• Assuming independent Gaussian noise:
  r (p)T r (p) 

p(r (p))  exp 
2
2 r


(1)
 r (p)T r (p)
log p(r (p)) 
 const
2
2 r
(2)
| r (p) |2
E (p) 
 log p(p)  const
2
2 r
(3)
37
Quality of Match
• If we assume all parameters equally likely
(within certain limits)
p(p)  const
| r (p) |2
E (p) 
 const
2
2 r
(1)
• Thus we need to find the parameters which
minimize the sum of squares of residuals,
(2)
E (p) | r(p) |2
38
Learning the Relationship
• For each of a training set
– find best fit given landmarks, p
– randomly perturb p by p and measure
r(p  p)  I m (p)  I im (p  p)
(in model frame)
39
More Analytic Approach
r
δp
Taylor expansion: r (p  p)  r (p) 
p
To minimize E(p  p),
δp  Rr(p)
Final result in the paper:
1
 r r  r T

R  
 p

p

p


T
where
or E  r Tr
40
AAM Algorithm
• Initial estimate Im(p)
• Start at coarse resolution
• At each resolution
– Measure residual error, r(p)
– predict correction p = -Rr
– p  p - p
– repeat to convergence
41
Active Appearance Models (AAM)
• Example
A face model built from 400 images. The figure
below shows frames from an AAM search for a
new face, each starting with the mean model
displaced from the true face centre.
Figure: Multi-Resolution search from displaced position
42
Problems
• Automatic Model Building
– Require correspondences across a set
– Hard to achieve reliably
• Reliable measure of quality of fit
– Necessary for good matching
– Essential for detection
• Model initialization
– Getting good initial estimate can be hard
–  10% percent of the image size and scale
43
AAM Summery
• Parameters
An AAM contains a statistical model of the shape and
grey-level appearance of the object of interest.
• Goals
Matching to an image involves finding model parameters
which minimize the difference between the image and a
synthesized model example, projected into the image. The
potentially large number of parameters makes this a
difficult problem.
44
AAM Summery
• Iterations
We observe that displacing each model parameter
from the correct value induces a particular pattern
in the residuals. In a training phase, the AAM
learns a linear model of the relationship between
parameter displacements and the induced
residuals. During search it measures the residuals
and uses this model to correct the current
parameters, leading to a better fit.
45
Some other group’s AAM research
•
Simon Baker and Iain Matthews at CMU are doing some wonderful work
on analyzing and improving the AAM update algorithm. They have gone
on to develop fast tracking, model building and 3D reconstruction
algorithms. Awesome.
46