in ,

Few-shot Video-to-Video Synthesis, Hacker News

Few-shot Video-to-Video Synthesis, Hacker News


  Ting-Chun Wang  Ming-Yu Liu     Andrew Tao  Guilin Liu  Jan Kautz  Bryan Catanzaro

  

NVIDIA Corporation

[Paper][arXiv][Video][Code]

Abstract

Video-to-video synthesis (vid2vid) aims at converting an input semantic video, such as videos of human poses or segmentation masks, to an output photorealistic video. While the state-of-the-art of vid2vid has advanced significantly, existing approaches share two major limitations. First, they are data-hungry. Numerous images of a target human subject or a scene are required for training. Second, a learned model has limited generalization capability. A pose-to-human vid2vid model can only synthesize poses of the single person in the training set. It does not generalize to other humans that are not in the training set. To address the limitations, we propose a few-shot vid2vid framework, which learns to synthesize videos of previously unseen subjects or scenes by leveraging few example images of the target at test time. Our model achieves this few-shot generalization capability via a novel network weight generation module utilizing an attention mechanism. We conduct extensive experimental validations with comparisons to strong baselines using several large-scale video datasets including human-dancing videos, talking-head videos, and street-scene videos. The experimental results verify the effectiveness of the proposed framework in addressing the two limitations of existing vid2vid approaches.

paper thumbnail

Paper

arXiv, 2019.

Citation

Ting-Chun Wang, Ming-Yu Liu, Andrew Tao, Guilin Liu, Jan Kautz, and Bryan Catanzaro . “Few-shot Video-to-Video Synthesis”, in NeurIPS, 2019.Bibtex

Code:Pytorch

Acknowledgement

We thank Karan Sapra for generating the segmentation maps for us.

Citation

If you find this useful for your research, please use the following.
@inproceedings {wang 2019 fewshotvid2vid,
  title={Few-shot Video-to-Video Synthesis},
  author={Ting-Chun Wang and Ming-Yu Liu and Andrew Tao and Guilin Liu and Jan Kautz and Bryan Catanzaro},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
  year={2019}
}

Brave Browser
Read More
Payeer

What do you think?

Leave a Reply

Your email address will not be published.

GIPHY App Key not set. Please check settings

General election LIVE – Latest news as Sir Lindsay Hoyle replaces John Bercow as Commons speaker – The Sun, Thesun.co.uk

General election LIVE – Latest news as Sir Lindsay Hoyle replaces John Bercow as Commons speaker – The Sun, Thesun.co.uk

McDonald's HR Chief Exits After CEO Ousted for ‘Consensual’ Relationship With Employee – News18, News18.com

McDonald's HR Chief Exits After CEO Ousted for ‘Consensual’ Relationship With Employee – News18, News18.com