Segmenting the image plane of video sequences is often one of the first steps towards the analysis of video. A lot of effort has been spent on developing image segmentation techniques based on cues such as color, or texture. Similarly, there are several methods for segmenting image motion based on optical flow, or motion features. On the other hand, there might be cases where segmentation based on photometry, or motion (dynamics) alone might be insufficient, because the motion of the object, segmented based on photometry, is incorrect, or because the color of the object, segmented based on motion, is wrong. (Doretto et al., 2003) addresses, for the first time, the problem of segmenting video based on the joint photometry and dynamics of the scene. The approach aims at solving a variational optimization problem that looks for the region boundaries and the dynamic texture models that can optimally represent the video data inside each region. The result is an algorithm that can group regions with the same spatio-temporal statistics. Extensions of this approach have been successfully used for applications such as traffic monitoring, and medical image analysis for segmenting organs in motion.


  1. ICCV
    Dynamic texture segmentation Doretto, G., Cremers, D., Favaro, P., and Soatto, S. In Proceedings of IEEE International Conference on Computer Vision, 2003. abstract bibTeX pdf