Dynamic Texture Editing

Summary

Editing Dynamic Textures by Changing Model Parameters

The operation that by processing video data allows producing new video data in Computer Graphics is known as Video-Based Rendering (VBR). Developing new VBR techniques is important because they allow to quickly synthesize new photo-realistic scenes (because of the origin of the data) without having to develop and simulate a synthetic model of the scene. The difficulty is in developing editing techniques that, once applied to the original data, can produce the desired perceptual effect. (Doretto & Soatto, 2002) and (Doretto & Soatto, 2003) represent a VBR approach that allows learning dynamic texture models from video, and then simulating them for synthesizing/rendering new unseen videos. Modeling the spatial stationarity enables the synthesis not only in time, but also in space (so the frame size can grow) (Doretto et al., 2004). It is shown how dynamic texture model parameters can be edited (changed) online, and mapped to meaningful perceptual changes, such as the spatial frequency content, the speed, the time axis, or the intensity of the visual process. This means that from a video sequence of sea waves one could, for instance, produce a new video with a rougher or smoother sea movement, according to the desire.

Plenoptic Modeling for Viewpoint Editing

(Doretto & Soatto, 2003) further extends the previous VBR approach by introducing a model of the spatio-temporal statistics of a collection of images of dynamic scenes as seen from a moving camera. The joint modeling of the moving vantage point together with the statistics of the scene motion is obtained by introducing a time-variant linear dynamical system. The resulting algorithms could be useful for video editing where the motion of a virtual camera can be controlled interactively, as well as for performing stabilized synthetic generation of video sequences.

References

1. ECCV

   Spatially homogeneous dynamic textures Doretto, G., Jones, E., and Soatto, S. In Proceedings of European Conference on Computer Vision, 2004. Oral abstract bibTeX pdf

   We address the problem of modeling the spatial and temporal second-order statistics of video sequences that exhibit both spatial and temporal regularity, intended in a statistical sense. We model such sequences as dynamic multiscale autoregressive models, and introduce an efficient algorithm to learn the model parameters. We then show how the model can be used to synthesize novel sequences that extend the original ones in both space and time, and illustrate the power, and limitations, of the models we propose with a number of real image sequences.

   @inproceedings{dorettoJS04eccv,
     abbr = {ECCV},
     author = {Doretto, G. and Jones, E. and Soatto, S.},
     title = {Spatially homogeneous dynamic textures},
     booktitle = {Proceedings of European Conference on Computer Vision},
     year = {2004},
     volume = {2},
     pages = {591--602},
     address = {Prague, Czech Republic},
     month = may,
     bib2html_pubtype = {Conferences},
     bib2html_rescat = {Dynamic Textures, Visual Motion Analysis},
     file = {dorettoJS04eccv.pdf:doretto/conference/dorettoJS04eccv.pdf:PDF},
     owner = {doretto},
     timestamp = {2013.10.18},
     wwwnote = {Oral}
   }
   

2. CVPR

   Editable dynamic textures Doretto, G., and Soatto, S. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. abstract bibTeX pdf

   We present a simple and efficient algorithm for modifying the temporal behavior of “dynamic textures,” i.e. sequences of images that exhibit some form of temporal regularity, such as flowing water, steam, smoke, flames, foliage of trees in wind.

   @inproceedings{dorettoS03cvpr,
     abbr = {CVPR},
     author = {Doretto, G. and Soatto, S.},
     title = {Editable dynamic textures},
     booktitle = {Proceedings of the IEEE Computer Society Conference on Computer Vision
     	and Pattern Recognition},
     year = {2003},
     volume = {2},
     pages = {137--142},
     address = {Madison, Wisconsin, USA},
     month = jun,
     bib2html_pubtype = {Conferences},
     bib2html_rescat = {Dynamic Textures, Image Based Rendering},
     file = {dorettoS03cvpr.pdf:doretto\\conference\\dorettoS03cvpr.pdf:PDF;dorettoS03cvpr.pdf:doretto\\conference\\dorettoS03cvpr.pdf:PDF}
   }
   

3. Towards plenoptic dynamic textures Doretto, G., and Soatto, S. In Proceedings of the 3rd International Workshop on Texture Analysis and Synthesis, 2003. abstract bibTeX pdf

   We present a technique to infer a model of the spatio-temporal statistics of a collection of images of dynamic scenes seen from a moving camera. We use a time-variant linear dynamical system to jointly model the statistics of the video signal and the moving vantage point. We propose three approaches to inference, the first based on the plenoptic function, the second based on interpolating linear dynamical models, the third based on approximating the scene as piecewise planar. For the last two approaches, we also illustrate the potential of the proposed techniques with a number of experiments. The resulting algorithms could be useful for video editing where the motion of the vantage point can be controlled interactively, as well as to perform stabilized synthetic generation of video sequences.

   @inproceedings{dorettoS03texture,
     author = {Doretto, G. and Soatto, S.},
     title = {Towards plenoptic dynamic textures},
     booktitle = {Proceedings of the 3rd International Workshop on Texture Analysis
     	and Synthesis},
     year = {2003},
     pages = {25--30},
     address = {Nice, France},
     month = oct,
     bib2html_pubtype = {Conferences},
     bib2html_rescat = {Dynamic Textures, Image Based Rendering, Registration from Dynamic
     	Textures},
     file = {dorettoS03texture.pdf:doretto\\conference\\dorettoS03texture.pdf:PDF;dorettoS03texture.pdf:doretto\\conference\\dorettoS03texture.pdf:PDF},
     owner = {doretto}
   }
   

4. Editable dynamic textures Doretto, G., and Soatto, S. In Conference Abstracts and Applications of SIGGRAPH ’02, 2002. Oral bibTeX pdf

   @inproceedings{dorettoS02siggraph,
     author = {Doretto, G. and Soatto, S.},
     title = {Editable dynamic textures},
     booktitle = {Conference Abstracts and Applications of SIGGRAPH '02},
     year = {2002},
     pages = {177},
     address = {San Antonio, Texas, USA},
     month = jul,
     bib2html_pubtype = {Conferences},
     bib2html_rescat = {Dynamic Textures, Image Based Rendering},
     file = {dorettoS02siggraph.pdf:doretto\\conference\\dorettoS02siggraph.pdf:PDF;dorettoS02siggraph.pdf:doretto\\conference\\dorettoS02siggraph.pdf:PDF},
     owner = {doretto},
     timestamp = {2007.01.19},
     wwwnote = {Oral}
   }