Video Tracking

Summary

Video tracking is the process of localizing an object in video at every time instant. It is a challenging problem especially when there are multiple objects to follow, and they may look similar and overlap as they move or deform over time. A particular scenario is given by people tracking across a network of multiple cameras. People need to first be detected (Perera et al., 2006), potentially against a dynamic background (Kim et al., 2009), and then segmented while accounting for occlusions (Tu et al., 2008). If the camera system is fully calibrated, the detections from multiple cameras can be backprojected to a planar location and integrated for occlusion resolution and temporal filtering (Krahnstoever et al., 2009) (Tu et al., 2007). Tracking information can then be linked across cameras and temporal gaps with person re-identification (Tu et al., 2011) (Doretto et al., 2011).

Related Projects

People Detection

3 minute read

People detection and tracking in video are fundamental Computer Vision capabilities that still constitute a research challenge. Important difficulties are du...

Video Surveillance

7 minute read

Increasingly, large networks of surveillance cameras are employed to monitor public and private facilities. This continuous collection of imagery has the pot...

Video Understanding

9 minute read

Video understanding is concerned with the parsing of the image data flow for the semantic understanding of the objects in the scene, but also their actions a...

References

1. JAIHC

   Appearance-based person reidentification in camera networks: problem overview and current approaches Doretto, G., Sebastian, T., Tu, P., and Rittscher, J. Journal of Ambient Intelligence and Humanized Computing, 2011. abstract bibTeX pdf html

   Recent advances in visual tracking methods allow following a given object or individual in presence of significant clutter or partial occlusions in a single or a set of overlapping camera views. The question of when person detections in different views or at different time instants can be linked to the same individual is of fundamental importance to the video analysis in large-scale network of cameras. This is the person reidentification problem. The paper focuses on algorithms that use the overall appearance of an individual as opposed to passive biometrics such as face and gait. Methods that effectively address the challenges associated with changes in illumination, pose, and clothing appearance variation are discussed. More specifically, the development of a set of models that capture the overall appearance of an individual and can effectively be used for information retrieval are reviewed. Some of them provide a holistic description of a person, and some others require an intermediate step where specific body parts need to be identified. Some are designed to extract appearance features over time, and some others can operate reliably also on single images. The paper discusses algorithms for speeding up the computation of signatures. In particular it describes very fast procedures for computing co-occurrence matrices by leveraging a generalization of the integral representation of images. The algorithms are deployed and tested in a camera network comprising of three cameras with non-overlapping field of views, where a multi-camera multi-target tracker links the tracks in different cameras by reidentifying the same people appearing in different views.

   @article{dorettoSTR11jaihc,
     abbr = {JAIHC},
     author = {Doretto, G. and Sebastian, T. and Tu, P. and Rittscher, J.},
     title = {Appearance-based person reidentification in camera networks: problem
     	overview and current approaches},
     journal = {Journal of Ambient Intelligence and Humanized Computing},
     year = {2011},
     volume = {2},
     pages = {127-151},
     affiliation = {West Virginia University, P.O. Box 6901, Morgantown, WV 26506, USA},
     bib2html_pubtype = {Journals},
     bib2html_rescat = {Human Reidentification, Identity Management, Video Analysis, Appearance
     	Modeling, Shape and Appearance Modeling, Integral Image Computations,
     	Track Matching},
     file = {dorettoSTR11jaihc.pdf:doretto\\journal\\dorettoSTR11jaihc.pdf:PDF},
     issn = {1868-5137},
     issue = {2},
     keyword = {Engineering},
     owner = {doretto},
     publisher = {Springer Berlin / Heidelberg},
     timestamp = {2010.10.17},
     url = {http://dx.doi.org/10.1007/s12652-010-0034-y}
   }
   

2. Video Analytics for Force Protection Tu, P.H., Brooksby, G.W., Doretto, G., Hamilton, D.W., Krahnstoever, N., Laflan, J.B., Liu, X., Patwardhan, K.A., Sebastian, T., Tong, Y., Tu, J., Wheeler, F.W., Wynnyk, C. M., Yao, Y., and Yu, T. In Distributed Video Sensor Networks, 2011. abstract bibTeX pdf html

   For troop and military installation protection, modern computer vision methods must be harnessed to enable a comprehensive approach to contextual awareness. In this chapter we present a collection of intelligent video technologies currently under development at the General Electric Global Research Center, which can be applied to this challenging problem. These technologies include: aerial analysis for object detection and tracking, site-wide tracking from networks of fixed video cameras, person detection from moving platforms, biometrics at a distance and facial analysis for the purposes of inferring intent. We hypothesize that a robust approach to troop protection will require the synthesis of all of these technologies into a comprehensive system of systems.

   @incollection{tuBDHKLLPSTTWWYY11chapter,
     author = {Tu, P.H. and Brooksby, G.W. and Doretto, G. and Hamilton, D.W. and Krahnstoever, N. and Laflan, J.B. and Liu, X. and Patwardhan, K.A. and Sebastian, T. and Tong, Y. and Tu, J. and Wheeler, F.W. and Wynnyk, C. M. and Yao, Y. and Yu, T.},
     title = {Video Analytics for Force Protection},
     booktitle = {Distributed Video Sensor Networks},
     publisher = {Springer London},
     year = {2011},
     editor = {Bhanu, B. and Ravishankar, C.V. and Roy-Chowdhury, A.K. and Aghajan, H. and Terzopoulos, D.},
     pages = {405--425},
     bib2html_pubtype = {Book Chapters},
     bib2html_rescat = {Video Analysis},
     file = {tuBDHKLLPSTTWWYY11chapter.pdf:doretto\\chapter\\tuBDHKLLPSTTWWYY11chapter.pdf:PDF},
     owner = {doretto},
     timestamp = {2010.01.10},
     url = {http://dx.doi.org/10.1007/978-0-85729-127-1_27}
   }
   

3. AVSS

   A model change detection approach to dynamic scene modeling Kim, S. J., Doretto, G., Rittscher, J., Tu, P., Krahnstoever, N., and Pollefeys, M. In IEEE International Conference on Advanced Video and Signal Based Surveillance, 2009. Oral abstract bibTeX pdf

   In this work we propose a dynamic scene model to provide information about the presence of salient motion in the scene, and that could be used for focusing the attention of a pan/tilt/zoom camera, or for background modeling purposes. Rather than proposing a set of saliency detectors, we define what we mean by salient motion, and propose a precise model for it. Detecting salient motion becomes equivalent to detecting a model change. We derive optimal online procedures to solve this problem, which enable a very fast implementation. Promising results show that our model can effectively detect salient motion even in severely cluttered scenes, and while a camera is panning and tilting.

   @inproceedings{kimDRTKP09avss,
     abbr = {AVSS},
     author = {Kim, S. J. and Doretto, G. and Rittscher, J. and Tu, P. and Krahnstoever, N. and Pollefeys, M.},
     title = {A model change detection approach to dynamic scene modeling},
     booktitle = {IEEE International Conference on Advanced Video and Signal Based
     	Surveillance},
     year = {2009},
     pages = {490--495},
     month = sep,
     bib2html_pubtype = {Conferences},
     bib2html_rescat = {Video Analysis, Dynamic Textures, Visual Motion Detection},
     file = {kimDRTKP09avss.pdf:doretto\\conference\\kimDRTKP09avss.pdf:PDF},
     keywords = {dynamic scene modeling, video analysis, PTZ camera, focus-ofattention, background
     	modeling, sequential generalized likelihood ratio, model change detection,
     	linear dynamical systems, dynamic textures},
     owner = {doretto},
     timestamp = {2009.09.28},
     wwwnote = {Oral}
   }
   

4. AVSS

   Intelligent video for protecting crowded sports venues Krahnstoever, N., Tu, P., Yu, T., Patwardhan, K., Hamilton, D., Yu, B., Greco, C., and Doretto, G. In IEEE International Conference on Advanced Video and Signal Based Surveillance, 2009. (Invited paper) Oral bibTeX pdf

   @inproceedings{krahnstoeverTYPHYGD09avss,
     abbr = {AVSS},
     author = {Krahnstoever, N. and Tu, P. and Yu, T. and Patwardhan, K. and Hamilton, D. and Yu, B. and Greco, C. and Doretto, G.},
     title = {Intelligent video for protecting crowded sports venues},
     booktitle = {IEEE International Conference on Advanced Video and Signal Based
     	Surveillance},
     year = {2009},
     pages = {116--121},
     month = sep,
     note = {(Invited paper)},
     bib2html_pubtype = {Conferences},
     bib2html_rescat = {Video Analysis},
     file = {krahnstoeverTYPHYGD09avss.pdf:doretto\\conference\\krahnstoeverTYPHYGD09avss.pdf:PDF},
     owner = {doretto},
     timestamp = {2009.09.28},
     wwwnote = {Oral}
   }
   

5. ECCV

   Unified crowd segmentation Tu, P., Sebastian, T., Doretto, G., Krahnstoever, N., Rittscher, J., and Yu, T. In Proceedings of European Conference on Computer Vision, 2008. abstract bibTeX pdf

   This paper presents a unified approach to crowd segmentation. A global solution is generated using an Expectation Maximization framework. Initially, a head and shoulder detector is used to nominate an exhaustive set of person locations and these form the person hypotheses. The image is then partitioned into a grid of small patches which are each assigned to one of the person hypotheses. A key idea of this paper is that while whole body monolithic person detectors can fail due to occlusion, a partial response to such a detector can be used to evaluate the likelihood of a single patch being assigned to a hypothesis. This captures local appearance information without having to learn specific appearance models. The likelihood of a pair of patches being assigned to a person hypothesis is evaluated based on low level image features such as uniform motion fields and color constancy. During the E-step, the single and pairwise likelihoods are used to compute a globally optimal set of assignments of patches to hypotheses. In the M-step, parameters which enforce global consistency of assignments are estimated. This can be viewed as a form of occlusion reasoning. The final assignment of patches to hypotheses constitutes a segmentation of the crowd. The resulting system provides a global solution that does not require background modeling and is robust with respect to clutter and partial occlusion.

   @inproceedings{tuSDKRY08eccv,
     abbr = {ECCV},
     author = {Tu, P. and Sebastian, T. and Doretto, G. and Krahnstoever, N. and Rittscher, J. and Yu, T.},
     title = {Unified crowd segmentation},
     booktitle = {Proceedings of European Conference on Computer Vision},
     year = {2008},
     pages = {691--704},
     bib2html_pubtype = {Conferences},
     bib2html_rescat = {Video Analysis, People Detection, Integral Image Computations,
     	People Tracking},
     file = {tuSDKRY08eccv.pdf:doretto\\conference\\tuSDKRY08eccv.pdf:PDF},
     owner = {doretto},
     timestamp = {2008.01.16}
   }
   

6. SPIE DSS

   An intelligent video framework for homeland protection Tu, P. H., Doretto, G., Krahnstoever, N. O., Perera, A. A. G., Wheeler, F. W., Liu, X., Rittscher, J., Sebastian, T. B., Yu, T., and Harding, K. G. In Proceedings of SPIE Defence and Security Symposium - Unattended Ground, Sea, and Air Sensor Technologies and Applications IX, 2007. (Invited paper) abstract bibTeX pdf

   This paper presents an overview of Intelligent Video work currently under development at the GE Global Research Center and other research institutes. The image formation process is discussed in terms of illumination, methods for automatic camera calibration and lessons learned from machine vision. A variety of approaches for person detection are presented. Crowd segmentation methods enabling the tracking of individuals through dense environments such as retail and mass transit sites are discussed. It is shown how signature generation based on gross appearance can be used to reacquire targets as they leave and enter disjoint fields of view. Camera calibration information is used to further constrain the detection of people and to synthesize a top-view, which fuses all camera views into a composite representation. It is shown how site-wide tracking can be performed in this unified framework. Human faces are an important feature as both a biometric identifier and as a method for determining the focus of attention via head pose estimation. It is shown how automatic pantilt-zoom control; active shape/appearance models and super-resolution methods can be used to enhance the face capture and analysis problem. A discussion of additional features that can be used for inferring intent is given. These include body-part motion cues and physiological phenomena such as thermal images of the face.

   @conference{tuDKPWLRSYH07SPIEdss,
     abbr = {SPIE DSS},
     author = {Tu, P. H. and Doretto, G. and Krahnstoever, N. O. and Perera, A. A. G. and Wheeler, F. W. and Liu, X. and Rittscher, J. and Sebastian, T. B. and Yu, T. and Harding, K. G.},
     title = {An intelligent video framework for homeland protection},
     booktitle = {Proceedings of SPIE Defence and Security Symposium - Unattended Ground,
     	Sea, and Air Sensor Technologies and Applications IX},
     year = {2007},
     editor = {Carapezza, E. M.},
     volume = {6562},
     address = {Orlando, FL, USA},
     month = apr,
     note = {(Invited paper)},
     bib2html_pubtype = {Conferences},
     bib2html_rescat = {Video Analysis},
     file = {tuDKPWLRSYH07SPIEdss.pdf:doretto\\conference\\tuDKPWLRSYH07SPIEdss.pdf:PDF;tuDKPWLRSYH07SPIEdss.pdf:doretto\\conference\\tuDKPWLRSYH07SPIEdss.pdf:PDF},
     keywords = {intelligent video, surveillance, camera calibration, person detection,
     	crowd segmentation, site-wide tracking, person reidentification,
     	face modeling, face super-resolution, deception detection},
     owner = {doretto},
     timestamp = {2007.01.20}
   }
   

7. CVPRW

   Moving object segmentation using scene understanding Perera, A. G. A., Brooksby, G., Hoogs, A., and Doretto, G. In Proceedings of IEEE Computer Society Workshop on Perceptual Organization in Computer Vision, 2006. abstract bibTeX pdf doi

   We present a novel approach to moving object detection in video taken from a translating, rotating and zooming sensor, with a focus on detecting very small objects in as few frames as possible. The primary innovation is to incorporate automatically computed scene understanding of the video directly into the motion segmentation process. Scene understanding provides spatial and semantic context that is used to improve frame-to-frame homography computation, as well as direct reduction of false alarms. The method can be applied to virtually any motion segmentation algorithm, and we explore its utility for three: frame differencing, tensor voting, and generalized PCA. The approach is especially effective on sequences with large scene depth and much parallax, as often occurs when the sensor is close to the scene. In one difficult sequence, our results show an 8-fold reduction of false positives on average, with essentially no impact on the true positive rate. We also show how scene understanding can be used to increase the accuracy of frame-to-frame homography estimates.

   @inproceedings{pereraBHD06pocv,
     abbr = {CVPRW},
     author = {Perera, A. G. A. and Brooksby, G. and Hoogs, A. and Doretto, G.},
     title = {Moving object segmentation using scene understanding},
     booktitle = {Proceedings of IEEE Computer Society Workshop on Perceptual Organization
     	in Computer Vision},
     year = {2006},
     pages = {201--208},
     address = {New York City, NY, USA},
     month = jun,
     bib2html_pubtype = {Conferences},
     bib2html_rescat = {Video Analysis, Visual Motion Segmentation},
     doi = {10.1109/CVPRW.2006.132},
     file = {pereraBHD06pocv.pdf:doretto\\conference\\pereraBHD06pocv.pdf:PDF;pereraBHD06pocv.pdf:doretto\\conference\\pereraBHD06pocv.pdf:PDF},
     owner = {doretto},
     timestamp = {2006.11.29}
   }