Depth Estimation from Videos

Ever wondered how to make your 2D videos pop with a 3D feel? This classic research paper from 2009, Depth from Video Using Learned Priors and Sparse User Interactions (Zhang et al., 2009)tackled this exact problem, offering a way to estimate depth information from regular video footage.

Overview

Think about it: a flat image can represent many different 3D arrangements. This paper addresses this by combining intelligent algorithms with a little bit of help from the user. This post breaks down the process.

Structure from Motion

The first step uses classical structure from motion techniques to estimate the camera’s motion and intrinsic parameters. This information is essential as it sets the stage for subsequent depth estimation by providing the necessary geometric context.

Disparity initialization

For each frame, the method applies loopy belief propagation to minimize an energy function defined in the paper (referred to as Equation (5)). This process produces an initial disparity estimate by finding the most consistent disparity assignment for each pixel.

While the paper provides a detailed formulation, the energy function can be conceptually summarized as:

$$E(D)=\sum _p\varphi (D_p)+\lambda \sum _{(p,q)\in \mathcal{N}}\psi (D_p,D_q)$$

Here,

• $D$ represents the disparity map,
• $\varphi (D_p)$ is the data term measuring how well a disparity $D_p$ at pixel $p$ fits the observed data,
• $\psi (D_p,D_q)$ is a smoothness term encouraging similar disparities for neighboring pixels $p$ and $q$.

(The exact form and parameters of these functions are detailed in the paper.)

Figure 1: Disparity maps

Figure 2: Markov Random Field for disparity map denoising

Bundle optimization

This optimization proceeds sequentially through the video frames (from 1 to $n$). For each frame $t$, while keeping the disparities in other frames fixed, the disparity map $D_t$ is refined by minimizing an energy function given as Equation (1) in the paper.

The optimization for each frame t is formulated as:

$$\underset{D_t}{\min }E(D_t)$$

This represents the refinement of the disparity map for the current frame, where E(D_t) encapsulates both the data fidelity and the smoothness constraints.

Space-time fusion

In the final stage, the algorithm performs a space-time fusion. This step integrates disparity information not just spatially within each frame but also temporally across frames. The fusion helps in smoothing out noise and inconsistencies, producing a coherent depth map for the entire video sequence.

This structured approach ensures that the depth maps generated are both accurate and temporally consistent, making it a significant contribution in the field of real-time video depth estimation.

For the details of the implementation, check out my note on Assignment 2 at .

References

Zhang, G., Jia, J., Wong, T.-T., & Bao, H. (2009). Consistent depth maps recovery from a video sequence. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(6), 974–988.