Spatial and Angular SuperResolution in a 4D Light Field
back to main page
Publications
Refereed Articles and Book Chapters
Variational Light Field Analysis for Disparity Estimation and SuperResolution In IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013 (to appear). [bib] [pdf] 
Refereed Conference Papers
Spatial and Angular Variational SuperResolution of 4D Light Fields In European Conference on Computer Vision (ECCV), 2012. [bib] [pdf] [poster] 
Contributions

Simultaneous spatial and angular superresolution for 4D light fields
\(\rightarrow\) data can e.g. be obtained by recent plenoptic cameras

Subpixelaccurate correspondence information required
\(\rightarrow\) from our depth map algorithm tailored for Lumigraphs

A single convex inverse problem is solved to arrive at the result
\(\rightarrow\) first time view synthesis is formulated in this way
A 4D light field or Lumigraph
Image Formation Model
Input:
 Images \(v_i:\Omega_i \to \mathbb{R}\) of a scene with depth maps \(d_i:\Omega_i \to \mathbb{R}\) and camera projections \(\pi_i:\mathbb{R}^3 \to \Omega_i\).
 Each pixel integrates intensities from a collection of rays, corresponding PSF modelled by a blur kernel \(b\).
 Synthesized view \(u:\Gamma \to \mathbb{R}\) of the light field from a novel view point, represented by a camera projection \(\pi: \mathbb{R}^3 \to \Gamma\), where \(\Gamma\) is the image plane of the novel view.
 This projection together with the scene information induces a transfer map \(\tau_i:\Omega_i\to\Gamma\) from each input view to the novel view, together with a binary visibility mask $m_i$ denoting unoccluded points.
Not all points \(x \in \Omega_i\) are visible in \(\Gamma\) due to occlusion,
which is described by the binary mask \(m_i\) on \(\Omega_i\).
Above, \(m_i\left( x \right) = 1\), while \(m_i\left( x \prime \right) = 0\).
Superresolved View Synthesis as a Variational Inverse Problem
In the ideal, noisefree case, the image formation model implies that the superresolved novel view $u$ is related to each input view by $b*(u\circ\tau_i)=v_i$. With realworld input, this is never satisfied exactly, so we instead minimize the variational energy corresponding to a MAP estimate \begin{equation} \label{eq:energy} E(u) = \sigma^2 \int_\Gamma \Du\ + \sum_{i=1}^n \underbrace{\frac{1}{2}\int_{\Omega_i} m_i(b* (u\circ \tau_i)  v_i) ^2 dx}_{=: {E^i}_{data}\left(u\right)}. \end{equation} where the total variation on $u$ acts as a regularizer. This is a convex model, which can be minimized globally and efficiently.
Illustration of the terms in the superresolution energy.
The figure shows the ground truth depth map for a single input view and the resulting mappings for forward and backward warps
as well as the visibility mask $m_i$. White pixels in the mask denote points in $\Omega_i$ which are visible in $\Gamma$ as well.
Discretization and Optimization
The functional derivative for the inverse problem above is required in order to find solutions. It is wellknown in principle, but made slightly more complicated by the different domains of the integrals. Transforming all integrals to the domain $\Gamma$, we obtain \begin{equation} \label{eq:energy_gradient_transformed} dE^i_\text{data} \left( u \right) = \left( \tilde m_i \;\bar b * (b * (u \circ \tau_i)  v_i ) \right) \circ \beta_i \end{equation} with $\tilde m_i := m_i \ \det \left( D \tau_i \right) \^{1}$. We can then apply standard convex optimization techniques to minimize the energy.
Superresolution algorithm for minimization of the energy.
The above method is a specialization of FISTA, where the inner loop computes a proximation for the total variation using the BermudezMoreno algorithm.
The operator \(\Pi_{\sigma^2 \mathbb{E}}\) denotes a pointwise projection onto the ball of radius \(\sigma^2\).
Results on Synthetic Light Fields
Results on Light Fields from a Plenoptic Camera
Superresolved epipolar plane image
$5\times 5$ input views
superresolved to $9\times 9$
superresolved to $17\times 17$
Reconstruction quality
Method  Scene Demo  Scene Motor 
Original resolution  36.91  35.36 
$3\times 3$ superresolution  30.82  31.72 
$3\times 3$ bilinear interpolation  23.89  22.84 
The proposed superresolution framework leads to significantly superior results.
Depth maps
Demo  Motor 
Last update: 28.05.2013, 22:32 