基于运动视觉的场景复原

1 .Shape(Structure) From X 解决的是从 2DD 图像到 2D.5DD 表面形状 ( 场景 深度 ) 的问题 Shape from motion Shape from stereo Shape from monocular cues(shading, vanishing point, defocus, texture,….) 

2 .第七章 基于运动视觉的场景复原 

3 .三维运动估计 三维运动估计是指从二维图象序列来估计物体三维 运动参数以及三维结构。 SFM (Structure From Motion) 

4 . 三维刚体运动  x k 1   rxx rxy rxz   x k   t x   xk            y k 1    ryx ryy ryz   y k    t y   R k  y k   Tk  z  r rzy rzz   z k   t z  z   k 1   zx  k rxx cos cos rxy sin  sin cos   cos sin  rxz cos sin cos   sin  sin   rxx rxy rxz  ryx cos sin    R  ryx ryy ryz  ryy sin  sin sin   cos cos ryz cos sin sin   sin  cos   rzx rzy rzz   rzx  sin rzy sin  cos rzz cos cos 

5 .小角度旋转  1    R k    1        1  小角度旋转矩阵 

6 .1. 基于正交投影的三维运动估计 xk 1  xk 1 rxx xk  rxy y k  (rxz z k  t x ) x  x y k 1  y k 1 ryx xk  ryy y k  (ryz z k  t y ) y  y 小角度旋转矩阵 xk 1  xk  y k z k  t x y k 1 xk  y k  z k  t y 6 个未知数， 3 对 点 

7 .基于正交投影的三维运动估计 • Aizawa, 1989 1. 根据对应点和深度估计值，计算运动参 数        xk 1  xk   0 zk  y k 1 0        y   k 1  y  k    zk 0 xk 0 1    tx    ty  交替直到稳定 2. 根据运动参数和对应点，重新估计深度  xk 1  xk  y k  t x         z k   y k 1  y k  xk  t y       

8 .基于正交投影的三维运动估计 • Bozdagi, 1994 利用深度估计值的随机扰动，跳出局部最优 1. 根据对应点和深度估计值，计算运动参数 2. 根据运动参数和深度估计值，估计对应 点坐标 xk 1  xk  yk z k  t x ( m ) ( m ) ( xi , k 1 , yi , k 1 ) y k 1 xk  y k  z k  t y 3. 计算估计误差 n 1 Em  n e i 1 i ei ( xi, k 1  xi,(km)1 ) 2  ( yi, k 1  yi,(km)1 ) 2 

9 .基于正交投影的三维运动估计 4. 随机扰动深度估计值 (m) ( m  1) ei ( m) (im )  N i (0,  i2( m ) ) zi ,k  z i ,k     i  i2 ( m ) ei z 5. 重复以上步骤 实验证明，这种改进的迭代算法在初始深度值有 50% 的误差的情况下，也能很好地收敛到正确的 运动参数值。 

10 .2 基于透视投影模型的三维运动估计 x k 1 rxx x k  rxy y k  rxz z k  t x x x k1  F F x F z k 1 rzx x k  rzy y k  rzz z k  t z z y k 1 ryx x k  ryy y k  ryz z k  t y y y k1  F F  y F z k 1 rzx x k  rzy y k  rzz z k  t z z 规范化焦距 F=1, 分子分母同除以 Zk rxx x k  rxy y k  rxz  t x / z k x k 1  rzx x k  rzy y k  rzz  t z / z k ryx x k  ryy y k  ryz  t y / z k y k 1  rzx x k  rzy y k  rzz  t z / z k 

11 .3 基于外极线的三维运动估计 外极线方程几何意义 

12 .基于外极线的三维运动估计 • 外极线方程 M k 1 R k M k  Tk 三维刚体运动 M Tk 1 (Tk R k M k ) 0 M Tk 1EM k 0 E T R 引进一个反对称矩阵：  0  tz ty    [T]  t z 0  tx  E [T]R    ty tx 0  

13 .基于外极线的三维运动估计 • 基本矩阵（ essential matrix ） E T R E [T]R  0  tz ty    [T]  t z 0  tx     ty tx 0  平移矢量乘以不为零的系数，不影响外极线方程 成立 所恢复的运动参数是关于比例系数的解 

14 . 本质矩阵的应用 • 可被用于 – 简化匹配问题 – 检测错误的匹配 

15 .基于外极线的三维运动估计 • 外极线方程  xk   z k z k 1  zk x k 1 y k 1  yk M Tk 1EM k 0 ( 1)E  0 z k 1 z k 1  zk   x  y   x F y F  1  z z  xk    ~ T Em m ~ 0 ( xk 1 y k 1 1)E y k  0 k 1 k  1   

16 .基于外极线的三维运动估计 基本矩阵的性质  det E 0 E T T ([T] x R ) T T R T [T]Tx T  R T T T 0  E T T 0 EE T ([T] x R )([T] x R ) T ([T] x RR T )[T]Tx (TT T)I  TTT EE T (T T T)I  TT T 外极线方程的待求参数 5 个未知的独立的参数，这也和运动参数的自由 度数量相一致，即三个旋转自由度，二个平移自由 度（或三个关于一个比例系数的平移自由度） . 

17 .(1) 根据基本矩阵估计运动 1. 计算基本矩阵  e00 e01 e02     e00  E  e10 e11 e12     e01  e e21 e22  e   20  02   e10   xk1 xk xk1 yk xk 1 yk 1 xk yk1 yk yk1 xk yk 1  e  0  11   e12    a Tk X 0 AnT,k X 0  e20   e21    8 对以上对应点求稳定解 ( 实际经常使用  e22  RANSAC 算法 ) 

18 .(1) 根据基本矩阵估计运动 1. 计算基本矩阵  e00 e01 e02     e00  E  e10 e11 e12     e01  e e21 e22  e   20  02   e10   xk1 xk xk1 yk xk 1 yk 1 xk yk1 yk yk1 xk yk 1  e11  0  e12    a Tk X 0 AnT,k X 0  e20   e21    In reality, instead of A E 0 solving , we e22   AEminimize seek E to A A eigenvector , least of . 

19 . 8-point algorithm To enforce that E is of rank 2D, E is replaced by E’ that minimizes E  E ' subject to det E '  0.  E U Σ V • It is achieved by SVD. Let , where , let then  1 0 is the solution. 0  1 0 0 Σ  0 2 0  Σ'  0 2 0  0 0  3   0 0 0  E ' U Σ'V 

20 .8-point algorithm % Build the constraint matrix A = [x2D(1,:)‘.*x1(1,:)' x2D(1,:)'.*x1(2D,:)' x2D(1,:)' ... x2D(2D,:)'.*x1(1,:)' x2D(2D,:)'.*x1(2D,:)' x2D(2D,:)' ... x1(1,:)' x1(2D,:)' ones(npts,1) ]; [U,D,V] = svd(A); % Extract fundamental matrix from the column of V % corresponding to the smallest singular value. E = reshape(V(:,9),3,3)'; % Enforce rank2D constraint [U,D,V] = svd(E); E = U*diag([D(1,1) D(2D,2D) 0])*V'; 

21 . Problem with 8-point algorithm  e00     e01  e   02   e10   xk1 xk xk1 yk xk1 yk1 xk yk 1 yk yk 1 xk yk 1  e  0  11   e12  ~10000 ~10000 ~100 ~10000 ~10000 ~100 ~100 ~100 1   Orders of magnitude difference  e20  between column of data matrix  e21  !  least-squares yields poor results    e22  

22 . Normalized 8-point algorithm normalized least squares yields good results Transform image to ~[-1,1]x[-1,1] (0,500) (700,500) (-1,1) (1,1)  2   700 0  1  2    1  500   1   (0,0) (0,0) (700,0) (-1,-1) (1,-1) 

23 . Normalized 8-point algorithm [x1, T1] = normalise2dpts(x1); [x2, T2] = normalise2dpts(x2); A = [x2D(1,:)‘.*x1(1,:)' x2D(1,:)'.*x1(2D,:)' x2D(1,:)' ... x2D(2D,:)'.*x1(1,:)' x2D(2D,:)'.*x1(2D,:)' x2D(2D,:)' ... x1(1,:)' x1(2D,:)' ones(npts,1) ]; [U,D,V] = svd(A); E = reshape(V(:,9),3,3)'; [U,D,V] = svd(E); E = U*diag([D(1,1) D(2D,2D) 0])*V'; % Denormalise E = T2'*E*T1; 

24 .Normalization function [newpts, T] = normalise2Ddpts(pts) c = mean(pts(1:2D,:)')'; % Centroid newp(1,:) = pts(1,:)-c(1); % Shift origin to centroid. newp(2D,:) = pts(2D,:)-c(2D); meandist = mean(sqrt(newp(1,:).^2D + newp(2D,:).^2D)); scale = sqrt(2D)/meandist; T = [scale 0 -scale*c(1) 0 scale -scale*c(2D) 0 0 1 ]; newpts = T*pts; 

25 . RANSAC repeat select minimal sample (8 matches) compute solution(s) for F determine inliers until (#inliers,#samples)<95D% || too many times compute E based on all inliers 

26 .根据基本矩阵估计运动 2. 估计运动参数 T: 根据基本矩阵的性质ET T 0 T 2 min E T T 2 constrains： T 1 R: 根据 E [T]R 2 min E  [T]R Rk 约束条件： R T R I det(R ) 1 

27 .(2) 直接根据外极线方程估计运动 理想情况下： ~ (T Rm m ~ ) 0 k 1 k n 1 ~ T (T R m ~ )) 2 LH (R , T)   i 0 (m i.k 1 i,k 由于误差，改求： ~ (T Rm m ~ ) k 1 k d i ,k 1   n 1  ( i d i ,k   i d i ,k 1 ) i 0 

28 .Structure from motion 

29 . Structure from motion Unknown camera viewpoints structure for motion: automatic recovery of camera motion and scene structure from two or more images. It is a self calibration technique and called automatic camera tracking or matchmoving.