Constructing 3D scenes from RGB-D images : How ?

This week, I stumbled upon an interesting publication that transforms 2D images into 3D photography using a technique called Context-aware Layered Depth Inpainting.

So, I played a little with the demo code using some pictures, and these are the results :

This was so fascinating to me, that I started looking up the logic behind this.

What is an RGB-D image ?

An RGB-D image is a combination between a standard RGB image (colored image) and a Depth image.

A depth image sounds vague at first, but it’s simply a 2D representation of the distance between the image plane and the objects in the RGB image.

You can compute the depth image by using specific hardware such as Microsoft Kinect or by estimating it from different images of the same scene taken from multiple cameras.

Why is this useful ?

RGB-D images are very useful. In fact, they are used in several fields such as Object Recognition, 3D Reconstruction, Augmented Reality, Image Processing, Robotic, and Interaction.

One of the many applications is 3D mapping. It’s constructing a 3D map using information provided by RGB-D images. You can find more here.

What is Context-aware Layered Depth Inpainting ?

The technique used in this publication takes a single RGB-D image as an input and generates a Layered Depth Image with inpainted color and depth.

There are a lot of concepts in this so, let’s break it down :

• The task of image inpainting aims to fill missing regions in images with plausible content.
• An LDI is similar to a regular 4-connected image, except at every position in the pixel lattice it can hold any number of pixels, from zero to many. Each LDI pixel stores a color and a depth value. (source)

After creating an LDI, they cut across discontinuities to separate the foreground and the background. Then, they synthesize new color and depth pixels to fill in missing values. Finally, they form the 3D textured mesh by integrating all the inpainted depth and color values back into the original LDI.

This is the most interesting step to me. Of course, this is not all it takes to construct a 3D image. For more details, visit their website and read the paper.