Robust Omnimatte with 3D Background Modeling

ICCV 2023
Geng Lin1Chen Gao2Jia-Bin Huang1,2Changil Kim2Yipeng Wang2Matthias Zwicker1Ayush Saraf2
1University of Maryland, College Park2Meta

We present OmnimatteRF, which creates mattes with associated effects like shadows from in-the-wild videos with coarse masks.


Video matting has broad applications, from adding interesting effects to casually captured movies to assisting video production professionals. Matting with associated effects like shadows and reflections has also attracted increasing research activity, and methods like Omnimatte have been proposed to separate foreground objects of interest into their own layers. However, prior works represent video backgrounds as 2D image layers, limiting their capacity to express more complicated scenes, thus hindering application to real-world videos. In this paper, we propose a novel video matting method, F2B3, that combines 2D foreground layers and a 3D background model. The 2D layers preserve the details of the subjects, while the 3D background robustly reconstructs scenes in real-world videos. Extensive experiments demonstrate that our method reconstructs with better quality on various videos.


OmnimatteRF extends Omnimatting to a larger variety of real-world videos with a combination of 2D foreground layers and a background radiance field.

Data & Results

Choose a dataset and video to view input with coarse masks.
Input & Coarse Masks
Choose an experiment to view results. To control all videos:
Foreground 1
Foreground 2
Background RGB
Background Depth

Download our Movies and Wild datasets here: Google Drive

The Movies dataset contains 5 sequences from 3 Blender movies. They come with ground truth camera poses, object masks, and clean background videos.

Vidoes in Wild are captured by us and come with reconstructed camera poses and coarse masks.

Other videos used in the paper are obtained from their authors: DAVISKubricdogwalk

Presentation Video

Coming soon!


  author    = {Geng Lin and Chen Gao and Jia-Bin Huang and Changil Kim and Yipeng Wang and Matthias Zwicker and Ayush Saraf},
  title     = {OmnimatteRF: Robust Omnimatte with 3D Background Modeling},
  booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  month     = {October},
  year      = {2023}