Publications

ECCVW
Online Stochastic Optimization for Data with Temporal Dependencies Patel, S., Zaveri, R. J., Chambers, S., Randhawa, Z., and Doretto, G. In European Conference on Computer Vision Workshop on Out of Distribution Generalization, 2024. [In press] abstract bibTeX
Agents operating in the open-world sense data streams that are non-stationary and temporally correlated, from which they need to quickly learn new knowledge to adapt and be resilient to the changing environment they explore. Current learning-based vision approaches are not designed for agents operating in real-time in the open-world. They are based on stochastic optimizations performed offline with data samples independent and identically distributed according to a stationary probability distribution. Data streams processed by open-world agents have temporal dependencies, and are typically processed with a single-pass. In such conditions, vanilla SGD leads to slow convergence rates and biased estimations, and current approaches make simplifying assumptions, like having prior knowledge of the statistical properties of the stream. We overcome these limitations by introducing an online stochastic optimization approach that does not make any prior assumption and self-adapts to the current time dependency properties of the stream. We demonstrate the validity of the approach on synthetic data as well as on video datasets where we show very promising results for learning self-supervised representations and for classification downstream tasks.
```
@inproceedings{patelZCRD24eccvw,
  author = {Patel, S. and Zaveri, R. J. and Chambers, S. and Randhawa, Z. and Doretto, G.},
  booktitle = {European Conference on Computer Vision Workshop on Out of Distribution Generalization},
  title = {{Online Stochastic Optimization for Data with Temporal Dependencies}},
  year = {2024},
  address = {MIlan, Italy},
  month = sep,
  note = {[In press]},
  publisher = {Springer},
  bib2html_pubtype = {Conferences},
  abbr = {ECCVW}
}
```
ICMLA
Multi-label Classification using Self-Supervised Learning: Addressing Class Inter-Dependency and Data Imbalance Mirzaee, G., Doretto, G., and Adjeroh, D. A. In IEEE International Conference on Machine Learning and Applications, 2024. [In press] abstract bibTeX
Diagnosing thoracic diseases using chest X-rays is demanding. It requires expert-level annotation and is challenged by class imbalance and diverse disease presentations. To tackle these problems, we propose a novel multi-label classification model that combines self-supervised learning (SSL), transfer learning, and domain knowledge. Our system leverages an effective backbone network, EfficientNetB0, which is pre-trained on 80% portion of the Chest X-ray14 dataset and is fine-tuned on the remaining portion of the dataset using a contrastive SSL method. We introduce a new data augmentation strategy called Adaptive Class Balance Augmentation (ACBA), which is designed to specifically address the class imbalance problem that is prevalent in this domain while maintaining the high quality and diversity of our synthetic images. We have developed a Squeeze-and-Excitation block for channel-wise attention and a feature fusion module into our model. This model uses 14 binary classifiers, which work with a custom Focal Weighted Cross-Entropy loss function that emphasizes hard-to-classify examples. The predictions made by the binary classifiers are then integrated by a correlation module that integrates statistical and biological disease correlations to refine predictions. Finally, a meta-learner works with a Gated Recurrent Unit (GRU) and multi-head attention mechanism to capture the complex relationships between diseases that are not necessarily also known to occur together. Experiments on the Chest X-ray14 dataset showed that our method significantly improved performance across 14 different diseases of the thorax.
```
@inproceedings{mirzaeeDA24icmla,
  author = {Mirzaee, G. and Doretto, G. and Adjeroh, D. A.},
  booktitle = {IEEE International Conference on Machine Learning and Applications},
  title = {Multi-label Classification using Self-Supervised Learning: Addressing Class Inter-Dependency and Data Imbalance},
  year = {2024},
  address = {Miami, FL},
  month = dec,
  note = {[In press]},
  publisher = {IEEE},
  bib2html_pubtype = {Conferences},
  abbr = {ICMLA}
}
```
ACCV
FG-CXR: A Radiologist-Aligned Gaze Dataset for Enhancing Interpretability in Chest X-Ray Report Generation Pham, T. T., Ho, N.-V., Bui, N.-T., Phan, T., Patel, B., Adjeroh, D., Doretto, G., Nguyen, A., Wu, C. C., Nguyen, H., and Le, N. H. In Asian Conference on Computer Vision (ACCV), 2024. [In press] abstract bibTeX pdf
Developing an interpretable system for generating reports in chest X-ray (CXR) analysis is becoming increasingly crucial in Computer-aided Diagnosis (CAD) systems, enabling radiologists to comprehend the decisions made by these systems. Despite the growth of diverse datasets and methods focusing on report generation, there remains a notable gap in how closely these models’ generated reports align with the interpretations of real radiologists. In this study, we tackle this challenge by initially introducing Fine-Grained CXR (FG-CXR) dataset, which provides fine-grained paired information between the captions generated by radiologists and the corresponding gaze attention heatmaps for each anatomy. Unlike existing datasets that include a raw sequence of gaze alongside a report, with significant misalignment between gaze location and report content, our FG-CXR dataset offers a more grained alignment between gaze attention and diagnosis transcript. Furthermore, our analysis reveals that simply applying black-box image captioning methods to generate reports cannot adequately explain which information in CXR is utilized and how long it needs to attend to accurately generate reports. Consequently, we propose a novel explainable radiologist’s attention generator network (Gen-XAI) that mimics the diagnosis process of radiologists, explicitly constraining its output to closely align with both radiologist’s gaze attention and transcript. Finally, we perform extensive experiments to illustrate the effectiveness of our method.
```
@inproceedings{phamHBPPADNWNL24accv,
  author = {Pham, T. T. and Ho, N.-V. and Bui, N.-T. and Phan, T. and Patel, B. and Adjeroh, D. and Doretto, G. and Nguyen, A. and Wu, C. C. and Nguyen, H. and Le, N. H.},
  booktitle = {Asian Conference on Computer Vision (ACCV)},
  title = {{FG-CXR: A Radiologist-Aligned Gaze Dataset for Enhancing Interpretability in Chest X-Ray Report Generation}},
  year = {2024},
  address = {Hanoi, Vietnam},
  month = dec,
  note = {[In press]},
  publisher = {Springer},
  bib2html_pubtype = {Conferences},
  abbr = {ACCV}
}
```
TAI
Retain and Adapt: Online Sequential EEG Classification with Subject Shift Duan, T., Wang, Z., Shen, L., Doretto, G., Adjeroh, D. A., Li, F., and Tao, C. IEEE Transactions on Artificial Intelligence, 2024. abstract bibTeX pdf doi
Large variance exists in Electroencephalogram (EEG) signals with its pattern differing significantly across subjects. It is a challenging problem to perform online sequential decoding of EEG signals across different subjects, where a sequence of subjects arrive in temporal order and no signal data is jointly available beforehand. The challenges include the following two aspects: 1) the knowledge learned from previous subjects does not readily fit to future subjects, and fast adaptation is needed in the process; and 2) the EEG classifier could drastically erase information of learnt subjects as learning progresses, namely catastrophic forgetting. Most existing EEG decoding explorations use sizable data for pretraining purposes, and to the best of our knowledge we are the first to tackle this challenging online sequential decoding setting. In this work, we propose a unified bi-level meta-learning framework that enables the EEG decoder to simultaneously perform fast adaptation on future subjects and retain knowledge of previous subjects. In addition, we extend to the more general subject-agnostic scenario and propose a subject shift detection algorithm for situations that subject identity and the occurrence of subject shifts are unknown. We conducted experiments on three public EEG datasets for both subject-aware and subject-agnostic scenarios. The proposed method demonstrates its effectiveness in most of the ablation settings, e.g. an improvement of 5.73% for forgetting mitigation and 3.50% for forward adaptation on SEED dataset for subject agnostic scenarios.
```
@article{duanWLTDA24tai,
  author = {Duan, T. and Wang, Z. and Shen, L. and Doretto, G. and Adjeroh, D. A. and Li, F. and Tao, C.},
  journal = {IEEE Transactions on Artificial Intelligence},
  title = {{Retain and Adapt: Online Sequential EEG Classification with Subject Shift}},
  year = {2024},
  issn = {2691-4581},
  month = apr,
  number = {9},
  pages = {4479--4492},
  volume = {5},
  doi = {10.1109/TAI.2024.3385390},
  publisher = {IEEE},
  bib2html_pubtype = {Journals},
  abbr = {TAI}
}
```
NAACL
Z-GMOT: Zero-shot Generic Multiple Object Tracking Tran, K., Le Dinh, A. D., Nguyen, T.-P., Phan, T., Nguyen, P., Luu, K., Adjeroh, D., Doretto, G., and Le, N. H. In Findings of the Association for Computational Linguistics (NAACL), 2024. abstract bibTeX pdf html code data doi
Despite recent significant progress, Multi-Object Tracking (MOT) faces limitations such as reliance on prior knowledge and predefined categories and struggles with unseen objects. To address these issues, Generic Multiple Object Tracking (GMOT) has emerged as an alternative approach, requiring less prior information. However, current GMOT methods often rely on initial bounding boxes and struggle to handle variations in factors such as viewpoint, lighting, occlusion, and scale, among others. Our contributions commence with the introduction of the Referring GMOT dataset a collection of videos, each accompanied by detailed textual descriptions of their attributes. Subsequently, we propose Z-GMOT, a cutting-edge tracking solution capable of tracking objects from never-seen categories without the need of initial bounding boxes or predefined categories. Within our Z-GMOT framework, we introduce two novel components: (i) iGLIP, an improved Grounded language-image pretraining, for accurately detecting unseen objects with specific characteristics. (ii) MA-SORT, a novel object association approach that adeptly integrates motion and appearance-based matching strategies to tackle the complex task of tracking objects with high similarity. Our contributions are benchmarked through extensive experiments conducted on the Referring GMOT dataset for GMOT task. Additionally, to assess the generalizability of the proposed Z-GMOT, we conduct ablation studies on the DanceTrack and MOT20 datasets for the MOT task.
```
@inproceedings{tranLNPNLADL24naacl,
  author = {Tran, K. and Le Dinh, A. D. and Nguyen, T.-P. and Phan, T. and Nguyen, P. and Luu, K. and Adjeroh, D. and Doretto, G. and Le, N. H.},
  booktitle = {Findings of the Association for Computational Linguistics (NAACL)},
  title = {{Z}-{GMOT}: Zero-shot Generic Multiple Object Tracking},
  year = {2024},
  address = {Mexico City, Mexico},
  editor = {Duh, K. and Gomez, H. and Bethard, S.},
  month = jun,
  pages = {3468--3479},
  publisher = {Association for Computational Linguistics},
  doi = {10.18653/v1/2024.findings-naacl.220},
  url = {https://aclanthology.org/2024.findings-naacl.220},
  code = {https://fsoft-aic.github.io/Z-GMOT},
  data = {https://fsoft-aic.github.io/Z-GMOT},
  bib2html_pubtype = {Conferences},
  abbr = {NAACL}
}
```
NN
Online continual decoding of streaming EEG signal with a balanced and informative memory buffer Duan, T., Wang, Z., Li, F., Doretto, G., Adjeroh, D. A., Yin, Y., and Tao, C. Neural Networks, 2024. abstract bibTeX pdf doi
Electroencephalography (EEG) based Brain Computer Interface (BCI) systems play a significant role in facilitating how individuals with neurological impairments effectively interact with their environment. In real world applications of BCI system for clinical assistance and rehabilitation training, the EEG classifier often needs to learn on sequentially arriving subjects in an online manner. As patterns of EEG signals can be significantly different for different subjects, the EEG classifier can easily erase knowledge of learnt subjects after learning on later ones as it performs decoding in online streaming scenario, namely catastrophic forgetting. In this work, we tackle this problem with a memory-based approach, which considers the following conditions: (1) subjects arrive sequentially in an online manner, with no large scale dataset available for joint training beforehand, (2) data volume from the different subjects could be imbalanced, (3) decoding difficulty of the sequential streaming signal vary, (4) continual classification for a long time is required. This online sequential EEG decoding problem is more challenging than classic cross subject EEG decoding as there is no large-scale training data from the different subjects available beforehand. The proposed model keeps a small balanced memory buffer during sequential learning, with memory data dynamically selected based on joint consideration of data volume and informativeness. Furthermore, for the more general scenarios where subject identity is unknown to the EEG decoder, aka. subject agnostic scenario, we propose a kernel based subject shift detection method that identifies underlying subject changes on the fly in a computationally efficient manner. We develop challenging benchmarks of streaming EEG data from sequentially arriving subjects with both balanced and imbalanced data volumes, and performed extensive experiments with a detailed ablation study on the proposed model. The results show the effectiveness of our proposed approach, enabling the decoder to maintain performance on all previously seen subjects over a long period of sequential decoding. The model demonstrates the potential for real-world applications.
```
@article{duanWLDAYT24nn,
  author = {Duan, T. and Wang, Z. and Li, F. and Doretto, G. and Adjeroh, D. A. and Yin, Y. and Tao, C.},
  journal = {Neural Networks},
  title = {Online continual decoding of streaming EEG signal with a balanced and informative memory buffer},
  year = {2024},
  issn = {0893-6080},
  month = aug,
  pages = {106338},
  volume = {176},
  publisher = {Elsevier},
  doi = {10.1016/j.neunet.2024.106338},
  keywords = {EEG decoding, Continual learning, Online Learning},
  bib2html_pubtype = {Journals},
  abbr = {NN}
}
```
ICPR
TabSeq: A Framework for Deep Learning on Tabular Data via Sequential Ordering Habib, A. Z. S. B., Wang, K., Hartley, M.-A., Doretto, G., and Adjeroh, D. A. In International Conference on Pattern Recognition (ICPR), 2024. [In press] abstract bibTeX pdf
Effective analysis of tabular data still poses a significant problem in deep learning, mainly because features in tabular datasets are often heterogeneous and have different levels of relevance. This work introduces TabSeq, a novel framework for the sequential ordering of features, addressing the vital necessity to optimize the learning process. Features are only sometimes equally informative, and for certain deep learning models, their random arrangement can hinder the model’s learning capacity. Finding the optimum sequence order for such features could improve the deep learning models’ learning process. The novel feature ordering technique, which we provide in this work, is based on clustering and incorporates both local ordering and global ordering. It is designed to be used with a multi-head attention mechanism in a denoising autoen- coder network. Our framework uses clustering to align comparable fea- tures and improve data organization. Multi-head attention focuses on es- sential characteristics, whereas denoising autoencoder highlights important aspects by rebuilding from distorted inputs. This method improves the capability to learn from tabular data while lowering redundancy. Our research demonstrating improved performance through appropriate feature sequence rearrangement utilizing raw antibody microarray and two other real-world biomedical datasets validates the impact of feature ordering. These results demonstrate that feature ordering can be a viable approach to improved deep learning of tabular data.
```
@inproceedings{habibWHDA2024icpr,
  author = {Habib, A. Z. S. B. and Wang, K. and Hartley, M.-A. and Doretto, G. and Adjeroh, D. A.},
  booktitle = {International Conference on Pattern Recognition (ICPR)},
  title = {TabSeq: A Framework for Deep Learning on Tabular Data via Sequential Ordering},
  year = {2024},
  month = dec,
  note = {[In press]},
  publisher = {Springer},
  bib2html_pubtype = {Conferences},
  abbr = {ICPR}
}
```
ICPR
Efficient Classification of Histopathology Images using Highly Imbalanced Data Nouyed, M. I., Hartley, M.-A., Doretto, G., and Adjeroh, D. A. In International Conference on Pattern Recognition (ICPR), 2024. [In press] abstract bibTeX arXiv pdf
This work addresses how to efficiently classify challenging histopathology images, such as gigapixel whole-slide images for cancer diagnostics with image-level annotation. We use images with annotated tumor regions to identify a set of tumor patches and a set of benign patches in a cancerous slide. Due to the variable nature of region of interest the tumor positive regions may refer to an extreme minority of the pixels. This creates an important problem during patch-level classification, where the majority of patches from an image labeled as ’cancerous’ are actually tumor-free. This problem is different from semantic segmentation which associates a label to every pixel in an image, because after patch extraction we are only dealing with patch-level this http URL existing approaches address the data imbalance issue by mitigating the data shortage in minority classes in order to prevent the model from being dominated by the majority classes. These methods include data re-sampling, loss re-weighting, margin modification, and data augmentation. In this work, we mitigate the patch-level class imbalance problem by taking a divide-and-conquer approach. First, we partition the data into sub-groups, and define three separate classification sub-problems based on these data partitions. Then, using an information-theoretic cluster-based sampling of deep image patch features, we sample discriminative patches from the sub-groups. Using these sampled patches, we build corresponding deep models to solve the new classification sub-problems. Finally, we integrate information learned from the respective models to make a final decision on the patches. Our result shows that the proposed approach can perform competitively using a very low percentage of the available patches in a given whole-slide image.
```
@inproceedings{nouyedHDA2024icpr,
  abbr = {ICPR},
  author = {Nouyed, M. I. and Hartley, M.-A. and Doretto, G. and Adjeroh, D. A.},
  booktitle = {International Conference on Pattern Recognition (ICPR)},
  title = {Efficient Classification of Histopathology Images using Highly Imbalanced Data},
  year = {2024},
  month = dec,
  note = {[In press]},
  publisher = {Springer},
  arxiv = {2409.13720},
  bib2html_pubtype = {Conferences}
}
```
ICRA
Open-Fusion: Real-time Open-Vocabulary 3D Mapping and Queryable Scene Representation Yamazaki, K., Hanyu, T., Vo, K., Pham, T., Tran, M., Doretto, G., Nguyen, A., and Le, N. In IEEE International Conference on Robotics and Automation (ICRA), 2024. Oral abstract bibTeX arXiv pdf webpage code doi
Precise 3D environmental mapping is pivotal in robotics. Existing methods often rely on predefined concepts during training or are time-intensive when generating semantic maps. This paper presents Open-Fusion, a groundbreaking approach for real-time open-vocabulary 3D mapping and queryable scene representation using RGB-D data. Open-Fusion harnesses the power of a pre-trained vision-language foundation model (VLFM) for open-set semantic comprehension and employs the Truncated Signed Distance Function (TSDF) for swift 3D scene reconstruction. By leveraging the VLFM, we extract region-based embeddings and their associated confidence maps. These are then integrated with 3D knowledge from TSDF using an enhanced Hungarian-based feature-matching mechanism. Notably, Open-Fusion delivers outstanding annotation-free 3D segmentation for open-vocabulary without necessitating additional 3D training. Benchmark tests on the ScanNet dataset against leading zero-shot methods highlight Open-Fusion’s superiority. Furthermore, it seamlessly combines the strengths of region-based VLFM and TSDF, facilitating realtime 3D scene comprehension that includes object concepts and open-world semantics. We encourage the readers to view the demos on our project page: https://uark-aicv.github.io/OpenFusion.
```
@inproceedings{yamazakiHVPTDNL24icra,
  abbr = {ICRA},
  author = {Yamazaki, K. and Hanyu, T. and Vo, K. and Pham, T. and Tran, M. and Doretto, G. and Nguyen, A. and Le, N.},
  booktitle = {IEEE International Conference on Robotics and Automation (ICRA)},
  title = {Open-Fusion: Real-time Open-Vocabulary 3D Mapping and Queryable Scene Representation},
  year = {2024},
  address = {Yokohama, Japan},
  pages = {9411--9417},
  month = may,
  publisher = {IEEE},
  code = {https://github.com/UARK-AICV/OpenFusion},
  doi = {10.1109/ICRA57147.2024.10610193},
  arxiv = {2310.03923},
  webpage = {{https://uark-aicv.github.io/OpenFusion/}},
  wwwnote = {Oral},
  bib2html_pubtype = {Conferences}
}
```
ISBI
SAM3D: Segment Anything Model in Volumetric Medical Images Bui, N. T., Hoang, D. H., Tran, M. T., Doretto, G., Adjeroh, D. A., Patel, B., Choudhary, A., and Le, N. In IEEE International Symposium on Biomedical Imaging (ISBI), 2024. abstract bibTeX pdf code doi
Image segmentation remains a pivotal component in medical image analysis, aiding in the extraction of critical information for precise diagnostic practices. With the advent of deep learning, automated image segmentation methods have risen to prominence, showcasing exceptional proficiency in processing medical imagery. Motivated by the Segment Anything Model (SAM)—a foundational model renowned for its remarkable precision and robust generalization capabilities in segmenting 2D natural images—we introduce SAM3D, an innovative adaptation tailored for 3D volumetric medical image analysis. Unlike current SAM-based methods that segment volumetric data by converting the volume into separate 2D slices for individual analysis, our SAM3D model processes the entire 3D volume image in a unified approach. Extensive experiments are conducted on multiple medical image datasets to demonstrate that our network attains competitive results compared with other state-of-the-art methods in 3D medical segmentation tasks while being significantly efficient in terms of parameters. Code and checkpoints are available at https://github.com/UARK-AICV/SAM3D.
```
@inproceedings{buiHTDAPCL24isbi,
  abbr = {ISBI},
  author = {Bui, N. T. and Hoang, D. H. and Tran, M. T. and Doretto, G. and Adjeroh, D. A. and Patel, B. and Choudhary, A. and Le, N.},
  booktitle = {IEEE International Symposium on Biomedical Imaging (ISBI)},
  title = {{SAM3D: Segment Anything Model in Volumetric Medical Images}},
  year = {2024},
  month = may,
  publisher = {IEEE},
  pages = {1--4},
  doi = {10.1109/ISBI56570.2024.10635844},
  code = {https://github.com/UARK-AICV/SAM3D},
  bib2html_pubtype = {Conferences}
}
```
ISBI
Few-shot adaptation for morphology-independent cell instance segmentation Zaveri, R., Brume, V., and Doretto, G. In IEEE International Symposium on Biomedical Imaging (ISBI), 2024. abstract bibTeX arXiv pdf doi
Microscopy data collections are becoming larger and more frequent. Accurate and precise quantitative analysis tools like cell instance segmentation are necessary to benefit from them. This is challenging due to the variability in the data, which requires retraining the segmentation model to maintain high accuracy on new collections. This is needed especially for segmenting cells with elongated and non-convex morphology like bacteria. We propose to reduce the amount of annotation and computing power needed for retraining the model by introducing a few-shot domain adaptation approach that requires annotating only one to five cells of the new data to process and that quickly adapts the model to maintain high accuracy. Our results show a significant boost in accuracy after adaptation to very challenging bacteria datasets.
```
@inproceedings{zaveriB24isbi,
  abbr = {ISBI},
  author = {Zaveri, R. and Brume, V. and Doretto, G.},
  booktitle = {IEEE International Symposium on Biomedical Imaging (ISBI)},
  title = {Few-shot adaptation for morphology-independent cell instance segmentation},
  year = {2024},
  month = may,
  publisher = {IEEE},
  arxiv = {2402.17165},
  pages = {1--5},
  doi = {10.1109/ISBI56570.2024.10635320},
  bib2html_pubtype = {Conferences}
}
```
WACV
ZEETAD: Adapting Pretrained Vision-Language Model for Zero-Shot End-to-End Temporal Action Detection Phan, T., Vo, K., Le Dinh, A. D., Doretto, G., Adjeroh, D., and Le, N. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2024. abstract bibTeX arXiv pdf code doi
Temporal action detection (TAD) involves the localization and classification of action instances within untrimmed videos. While standard TAD follows fully supervised learning with closed-set setting on large training data, recent zero-shot TAD methods showcase the promising openset setting by leveraging large-scale contrastive visuallanguage (ViL) pretrained models. However, existing zeroshot TAD methods have limitations on how to properly construct the strong relationship between two interdependent tasks of localization and classification and adapt ViL model to video understanding. In this work, we present ZEETAD, featuring two modules: dual-localization and zeroshot proposal classification. The former is a Transformerbased module that detects action events while selectively collecting crucial semantic embeddings for later recognition. The latter one, CLIP-based module, generates semantic embeddings from text and frame inputs for each temporal unit. Additionally, we enhance discriminative capability on unseen classes by minimally updating the frozen CLIP encoder with lightweight adapters. Extensive experiments on THUMOS14 and ActivityNet-1.3 datasets demonstrate our approach’s superior performance in zero-shot TAD and effective knowledge transfer from ViL models to unseen action categories. Code is available at https://github.com/UARK-AICV/ZEETAD.
```
@inproceedings{phanVDDAL24wacv,
  abbr = {WACV},
  author = {Phan, T. and Vo, K. and Le Dinh, A. D. and Doretto, G. and Adjeroh, D. and Le, N.},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  title = {{ZEETAD: Adapting Pretrained Vision-Language Model for Zero-Shot End-to-End Temporal Action Detection}},
  year = {2024},
  month = jan,
  publisher = {IEEE},
  address = {Waikoloa, HI, USA},
  pages = {7031--7040},
  doi = {10.1109/WACV57701.2024.00689},
  arxiv = {2311.00729},
  code = {https://github.com/UARK-AICV/ZEETAD},
  bib2html_pubtype = {Conferences}
}
```

BIBM
Replay with Stochastic Neural Transformation for Online Continual EEG Classification Duan, T., Wang, Z., Doretto, G., Li, F., Tao, C., and Adjeroh, D. In IEEE International Conference on Bioinformatics and Biomedicine (BIBM), 2023. Oral abstract bibTeX pdf
Brain computer interface (BCI) systems used for clinical assistance purposes such as wheelchair control require decoding of streaming brain signals i.e. electroencephalography (EEG) signals over a long period of time with subject shift in the middle. Numerous challenges arise during this online continual brain signal decoding process: 1) the EEG decoder needs to deal with streaming EEG signals from sequentially arriving subjects, with no data available beforehand for large-scale pretraining; 2) the EEG decoder should avoid catastrophic forgetting on previous subjects after learning on a new subject; 3) the EEG decoder should perform well on noisy signals with high variance across subjects. We proposed a principled replay-based approach for this general decoding scenario, forming a bi-level optimization framework with stochastic neural transformation for dynamic memory evolution, making them representative in feature space and encouraging the model to generalize well. The evolved signal segments are stored and replayed during later decoding stages to achieve optimal model performance on all previous subjects. The stochastic neural transformation performed in inner sup of bi-level optimization significantly enhances the diversity of stored signal segments and improves model robustness during online continual decoding. We perform detailed theoretical analysis on model’s generalization ability in addition to the empirical evaluations. We construct multiple new benchmarks to mimic real-world online sequential EEG decoding scenarios with underlying subject shifts. The extensive evaluation of the proposed approach shows it outperforms related strong baselines by a large margin.
```
@inproceedings{duanWDLTA23bibm,
  abbr = {BIBM},
  author = {Duan, T. and Wang, Z. and Doretto, G. and Li, F. and Tao, C. and Adjeroh, D.},
  booktitle = {IEEE International Conference on Bioinformatics and Biomedicine (BIBM)},
  title = {{Replay with Stochastic Neural Transformation for Online Continual EEG Classification}},
  year = {2023},
  month = dec,
  publisher = {IEEE},
  bib2html_pubtype = {Conferences},
  wwwnote = {Oral}
}
```
Current Topological and Machine Learning Applications for Bias Detection in Text Farrelly, Y., Hathaway, Q. A., Carlsson, G., Choudhary, A., Paul, R., Doretto, G., Himeur, Y., Atalls, S., and Mansoor, W. In International Conference on Signal Processing and Information Security (ICSPIS), 2023. Oral abstract bibTeX pdf
Institutional bias can impact patient outcomes, educational attainment, and legal system navigation. Written records often reflect bias, and once bias is identified; it is possible to refer individuals for training to reduce bias. Many machine learning tools exist to explore text data and create predictive models that can search written records to identify real-time bias. However, few previous studies investigate large language model embeddings and geometric models of biased text data to understand geometry’s impact on bias modeling accuracy. To overcome this issue, this study utilizes the RedditBias database to analyze textual biases. Four transformer models, including BERT and RoBERTa variants, were explored. Post-embedding, t-SNE allowed two-dimensional visualization of data. KNN classifiers differentiated bias types, with lower k-values proving more effective. Findings suggest BERT, particularly mini BERT, excels in bias classification, while multilingual models lag. The recommendation emphasizes refining monolingual models and exploring domain-specific biases.
```
@inproceedings{farrellySHCCPDHAM23icspis,
  author = {Farrelly, C. amd Singh, Y. and Hathaway, Q. A. and Carlsson, G. and Choudhary, A. and Paul, R. and Doretto, G. and Himeur, Y. and Atalls, S. and Mansoor, W.},
  booktitle = {International Conference on Signal Processing and Information Security (ICSPIS)},
  title = {Current Topological and Machine Learning Applications for Bias Detection in Text},
  year = {2023},
  month = nov,
  pages = {77--82},
  publisher = {IEEE},
  bib2html_pubtype = {Conferences},
  wwwnote = {Oral}
}
```
ICIP
More Synergy, Less Redundancy: Exploiting Joint Mutual Information for Self-Supervised Learning Mohamadi, S., Doretto, G., and Adjeroh, D. In IEEE International Conference on Image Processing (ICIP), 2023. abstract bibTeX arXiv pdf doi
Self-supervised learning (SSL) is now a serious competitor for supervised learning, even though it does not require data annotation. Several baselines have attempted to make SSL models exploit information about data distribution, and less dependent on the augmentation effect. However, there is no clear consensus on whether maximizing or minimizing the mutual information between representations of augmentation views practically contribute to improvement or degradation in performance of SSL models. This paper is a fundamental work where, we investigate the role of mutual information in SSL, and reformulate the problem of SSL in the context of a new perspective on mutual information. To this end, we consider joint mutual information from the perspective of partial information decomposition (PID) as a key step in reliable multivariate information measurement. PID enables us to decompose joint mutual information into three important components, namely, unique information, redundant information and synergistic information. Our framework aims for minimizing the redundant information between views and the desired target representation while maximizing the synergistic information at the same time. Our experiments lead to a re-calibration of two redundancy reduction baselines, and a proposal for a new SSL training protocol. Experimental results on multiple datasets and two downstream tasks show the effectiveness of this framework.
```
@inproceedings{mohamadiDA23icip,
  abbr = {ICIP},
  author = {Mohamadi, S. and Doretto, G. and Adjeroh, D.},
  booktitle = {{IEEE International Conference on Image Processing (ICIP)}},
  title = {{More Synergy, Less Redundancy: Exploiting Joint Mutual Information for Self-Supervised Learning}},
  year = {2023},
  address = {Kuala Lumpur, Malaysia},
  month = oct,
  pages = {1390--1394},
  publisher = {IEEE},
  doi = {10.1109/ICIP49359.2023.10222547},
  arxiv = {2307.00651},
  bib2html_pubtype = {Conferences}
}
```
ECAI
Distributionally Robust Cross Subject EEG Decoding Duan, T., Wang, Z., Doretto, G., Li, F., Tao, C., and Adjeroh, D. In European Conference on Artificial Intelligence (ECAI), 2023. abstract bibTeX arXiv pdf doi
Recently, deep learning has shown to be effective for Electroencephalography (EEG) decoding tasks. Yet, its performance can be negatively influenced by two key factors: 1) the high variance and different types of corruption that are inherent in the signal, 2) the EEG datasets are usually relatively small given the acquisition cost, annotation cost and amount of effort needed. Data augmentation approaches for alleviation of this problem have been empirically studied, with augmentation operations on spatial domain, time domain or frequency domain handcrafted based on expertise of domain knowledge. In this work, we propose a principled approach to perform dynamic evolution on the data for improvement of decoding robustness. The approach is based on distributionally robust optimization and achieves robustness by optimizing on a family of evolved data distributions instead of the single training data distribution. We derived a general data evolution framework based on Wasserstein gradient flow (WGF) and provides two different forms of evolution within the framework. Intuitively, the evolution process helps the EEG decoder to learn more robust and diverse features. It is worth mentioning that the proposed approach can be readily integrated with other data augmentation approaches for further improvements. We performed extensive experiments on the proposed approach and tested its performance on different types of corrupted EEG signals. The model significantly outperforms competitive baselines on challenging decoding scenarios.
```
@inproceedings{duanWDLTA23ecai,
  abbr = {ECAI},
  author = {Duan, T. and Wang, Z. and Doretto, G. and Li, F. and Tao, C. and Adjeroh, D.},
  booktitle = {European Conference on Artificial Intelligence (ECAI)},
  title = {Distributionally Robust Cross Subject {EEG} Decoding},
  year = {2023},
  address = {Krak{\'o}w, Poland},
  month = sep,
  pages = {575--582},
  publisher = {IOS Press},
  series = {Frontiers in Artificial Intelligence and Applications},
  volume = {372},
  doi = {10.3233/FAIA230318},
  arxiv = {2308.11651},
  bib2html_pubtype = {Conferences}
}
```
IJCB
AG-ReID 2023: Aerial-Ground Person Re-identification Challenge Results Nguyen, K., Fookes, C., Sridharan, S., Liu, F., Liu, X., Ross, A., Michalski, D., Nguyen, H., Deb, D., Kothari, M., Saini, M., Du, D., McCloskey, S., Bertocco, G., Boult, T., Andaló, F., Rocha, A., Zhu, H., Zheng, Z., Nevatia, R., Randhawa, Z. A., Sabri, S., and Doretto, G. In IEEE International Joint Conference on Biometrics , 2023. abstract bibTeX pdf
Person re-identification (Re-ID) on aerial-ground platforms has emerged as an intriguing topic within computer vision, presenting a plethora of unique challenges. High- flying altitudes of aerial cameras make persons appear dif- ferently in terms of viewpoints, poses, and resolution com- pared to the images of the same person viewed from ground cameras. Despite its potential, few algorithms have been developed for person re-identification on aerial-ground data, mainly due to the absence of comprehensive datasets. In response, we have collected a large-scale dataset and or- ganized the Aerial-Ground person Re-IDentification Chal- lenge (AG-ReID2023) to foster advancements in the field. The dataset comprises 100, 502 images with 1, 615 unique identities, including 51, 530 training images featuring 807 identities. The test set is divided into two subsets: Aerial to Ground (808 ids, 4,348 query images, 19,259 gallery images) and Ground to Aerial (808 ids, 4, 151 query im- ages, 21, 214 gallery images). In addition, we manually annotate individuals with their matching IDs across cam- eras and provide 15 soft attribute labels. The AG-ReID2023 Challenge in conjunction with the 7th IEEE International Joint Conference on Biometrics (IJCB) has garnered inter- est from numerous institutes, resulting in the submission of five distinct algorithms. We provide an in-depth examina- tion of the evaluation outcomes and present our findings from the contest. For additional details, kindly refer to the official website.
```
@inproceedings{nguyen23ijcb,
  abbr = {IJCB},
  author = {Nguyen, K. and Fookes, C. and Sridharan, S. and Liu, F. and Liu, X. and Ross, A. and Michalski, D. and Nguyen, H. and Deb, D. and Kothari, M. and Saini, M. and Du, D. and McCloskey, S. and Bertocco, G. and Boult, T. and Andal{\'o}, F. and Rocha, A. and Zhu, H. and Zheng, Z. and Nevatia, R. and Randhawa, Z. A. and Sabri, S. and Doretto, G.},
  booktitle = {{IEEE International Joint Conference on Biometrics }},
  title = {{AG-ReID 2023: Aerial-Ground Person Re-identification Challenge Results}},
  year = {2023},
  address = {Ljubljana, Slovenia},
  month = sep,
  publisher = {IEEE},
  bib2html_pubtype = {Conferences}
}
```
IGARSS
A Framework for Token-based Scene Classification of Remote Sensing Images Nouyed, M. I., Dretto, G., and Adjeroh, A. In IEEE International Geoscience and Remote Sensing Symposium (IGARSS), 2023. abstract bibTeX doi
Remote sensing images often come in very high resolutions. For instance, large images of resolution 1000x1000 are quite common in remote sensing applications. Thus, one key challenge is how to effectively analysis such very high resolution images using low computational resources, while maintain a reasonable performance as appropriate to the specific problem domain, such as scene classification, object segmentation, or object detection. In this work, we propose a framework for addressing this challenge, with focus on the specific problem of scene-classification for remote-sensed aerial imagery. Various approaches have been proposed for scene-based classification of remote sensing imagery [1] , [2] , [3] . For a survey of approaches and challenges, see [4] . See also [5] .
```
@inproceedings{nouyedDA23igrass,
  abbr = {IGARSS},
  author = {Nouyed, M. I. and Dretto, G. and Adjeroh, A.},
  booktitle = {IEEE International Geoscience and Remote Sensing Symposium (IGARSS)},
  title = {A Framework for Token-based Scene Classification of Remote Sensing Images},
  year = {2023},
  month = jul,
  pages = {6239--6242},
  publisher = {IEEE},
  doi = {10.1109/IGARSS52108.2023.10282874},
  bib2html_pubtype = {Conferences}
}
```
CVPRW
A Robust Likelihood Model for Novelty Detection Almohsen, R., Patel, S., Adjeroh, D. A., and Doretto, G. In IEEE/CVF CVPR Workshops on Computer Vision in The Wild (CVPRW), 2023. abstract bibTeX arXiv pdf html
Current approaches to novelty or anomaly detection are based on deep neural networks. Despite their effectiveness, neural networks are also vulnerable to imperceptible defor- mations of the input data. This is a serious issue in critical applications, or when data alterations are generated by an adversarial attack. While this is a known problem that has been studied in recent years for the case of supervised learn- ing, the case of novelty detection has received very limited attention. Indeed, in this latter setting the learning is typ- ically unsupervised because outlier data is not available during training, and new approaches for this case need to be investigated. We propose a new prior that aims at learning a robust likelihood for the novelty test, as a defense against attacks. We also integrate the same prior with a state-of-the- art novelty detection approach. Because of the geometric properties of that approach, the resulting robust training is computationally very efficient. An initial evaluation of the method indicates that it is effective at improving perfor- mance with respect to the standard models in the absence and presence of attacks.
```
@inproceedings{almohsenPAD23cvprw,
  abbr = {CVPRW},
  author = {Almohsen, R. and Patel, S. and Adjeroh, D. A. and Doretto, G.},
  booktitle = {IEEE/CVF CVPR Workshops on Computer Vision in The Wild (CVPRW)},
  title = {A Robust Likelihood Model for Novelty Detection},
  year = {2023},
  month = jun,
  bib2html_pubtype = {Conferences},
  arxiv = {2306.03331},
  url = {https://computer-vision-in-the-wild.github.io/cvpr-2023/}
}
```
CVPRW
Self-supervised Interest Point Detection and Description for Fisheye and Perspective Images Mera-Trujillo, M., Patel, S., Gu, Y., and Doretto, G. In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2023. abstract bibTeX arXiv pdf html doi
Keypoint detection and matching is a fundamental task in many computer vision problems, from shape reconstruction, to structure from motion, to AR/VR applications and robotics. It is a well-studied problem with remarkable successes such as SIFT, and more recent deep learning approaches. While great robustness is exhibited by these techniques with respect to noise, illumination variation, and rigid motion transformations, less attention has been placed on image distortion sensitivity. In this work, we focus on the case when this is caused by the geometry of the cameras used for image acquisition, and consider the keypoint detection and matching problem between the hybrid scenario of a fisheye and a projective image. We build on a state-of-the-art approach and derive a self-supervised procedure that enables training an interest point detector and descriptor network. We also collected two new datasets for additional training and testing in this unexplored scenario, and we demonstrate that current approaches are suboptimal because they are designed to work in traditional projective conditions, while the proposed approach turns out to be the most effective.
```
@inproceedings{mera-trujilloPD23cvprw,
  abbr = {CVPRW},
  author = {Mera-Trujillo, M. and Patel, S. and Gu, Y. and Doretto, G.},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)},
  title = {Self-supervised Interest Point Detection and Description for Fisheye and Perspective Images},
  year = {2023},
  address = {Los Alamitos, CA, USA},
  month = jun,
  pages = {6498--6507},
  publisher = {IEEE Computer Society},
  doi = {10.1109/CVPRW59228.2023.00691},
  bib2html_pubtype = {Conferences},
  arxiv = {2306.01938},
  keywords = {training; structure from motion; sensitivity; shape;detectors; self-supervised learning; cameras},
  url = {https://doi.ieeecomputersociety.org/10.1109/CVPRW59228.2023.00691}
}
```
WACV
CellTranspose: Few-shot Domain Adaptation for Cellular Instance Segmentation Keaton, M. R., Zaveri, R. J., and Doretto, G. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2023. abstract bibTeX arXiv pdf
Automated cellular instance segmentation is a process utilized for accelerating biological research for the past two decades, and recent advancements have produced higher quality results with less effort from the biologist. Most current endeavors focus on completely cutting the researcher out of the picture by generating highly generalized models. However, these models invariably fail when faced with novel data, distributed differently than the ones used for training. Rather than approaching the problem with methods that presume the availability of large amounts of target data and computing power for retraining, in this work we address the even greater challenge of designing an approach that requires minimal amounts of new annotated data as well as training time. We do so by designing specialized contrastive losses that leverage the few annotated samples very efficiently. A large set of results show that 3 to 5 annotations lead to models with accuracy that: 1) significantly mitigate the covariate shift effects; 2) matches or surpasses other adaptation methods; 3) even approaches methods that have been fully retrained on the target distribution. The adaptation training is only a few minutes, paving a path towards a balance between model performance, computing requirements and expert-level annotation needs.
```
@inproceedings{keatonZD23wacv,
  abbr = {WACV},
  author = {Keaton, M. R. and Zaveri, R. J. and Doretto, G.},
  booktitle = {{Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision}},
  title = {{CellTranspose: Few-shot Domain Adaptation for Cellular Instance Segmentation}},
  year = {2023},
  month = jan,
  pages = {455--466},
  publisher = {IEEE},
  bib2html_pubtype = {Conferences},
  arxiv = {2212.14121}
}
```
WACV
FUSSL: Fuzzy Uncertain Self Supervised Learning Mohamadi, S., Doretto, G., and Adjeroh, D. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2023. abstract bibTeX arXiv pdf
Self supervised learning (SSL) has become a very successful technique to harness the power of unlabeled data, with no annotation effort. A number of developed approaches are evolving with the goal of outperforming supervised alternatives, which have been relatively successful. Similar to some other disciplines in deep representation learning, one main issue in SSL is robustness of the approaches under different settings. In this paper, for the first time, we recognize the fundamental limits of SSL coming from the use of a single-supervisory signal. To address this limitation, we leverage the power of uncertainty representation to devise a robust and general standard hierarchical learning/training protocol for any SSL baseline, regard- less of their assumptions and approaches. Essentially, using the information bottleneck principle, we decompose feature learning into a two-stage training procedure, each with a distinct supervision signal. This double supervision approach is captured in two key steps: 1) invariance enforcement to data augmentation, and 2) fuzzy pseudo labeling (both hard and soft annotation). This simple, yet, effective protocol which enables cross-class/cluster feature learning, is instantiated via an initial training of an ensemble of models through invariance enforcement to data augmentation as first training phase, and then assigning fuzzy labels to the original samples for the second training phase. We consider multiple alternative scenarios with double supervision and evaluate the effectiveness of our approach on recent baselines, covering four different SSL paradigms, including geometrical, contrastive, non-contrastive, and hard/soft whitening (redundancy reduction) baselines. We performed extensive experiments under multiple settings to show that the proposed training protocol consistently improves the performance of the former baselines, independent of their respective underlying principles.
```
@inproceedings{mohamadiDA23wacv,
  abbr = {WACV},
  author = {Mohamadi, S. and Doretto, G. and Adjeroh, D.},
  booktitle = {{Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision}},
  title = {{FUSSL: Fuzzy Uncertain Self Supervised Learning}},
  year = {2023},
  month = jan,
  pages = {2799--2808},
  publisher = {IEEE},
  bib2html_pubtype = {Conferences},
  arxiv = {2210.15818}
}
```

BIBM
Efficient Classification of Very High Resolution Histopathological Images Nouyed, M. I., Doretto, G., and Adjeroh, D. A. In Proceedings of the IEEE International Conference on Bioinformatics and Biomedicine, 2022. Oral abstract bibTeX pdf
Over the years, deep learning approaches have shown significant improvement in various image understanding tasks. However, analysis of high resolution images still remains a major challenge. Apart from the huge computational resources required for such images, the large image sizes make it difficult to extract effective contextual information needed for important tasks, such as classification, segmentation, or clustering of such images. In this work, we address the challenge of high resolution image classification u sing a new discriminative patch selection approach. We embed our patch selection approach inside a novel classification framework, supporting potential use of different pre-trained learning models. We show results on a high resolution image dataset, namely, gigapixel whole slide tissue images for cancer tumors. We demonstrate the performance of the proposed approaches using comparative analysis with state-of-the art methods on this dataset.
```
@inproceedings{nouyedDA2022bibm,
  abbr = {BIBM},
  author = {Nouyed, M. I. and Doretto, G. and Adjeroh, D. A.},
  booktitle = {Proceedings of the {IEEE} International Conference on Bioinformatics and Biomedicine},
  title = {Efficient Classification of Very High Resolution Histopathological Images},
  year = {2022},
  month = dec,
  wwwnote = {Oral},
  pages = {3114--3121},
  publisher = {{IEEE}},
  bib2html_pubtype = {Conferences}
}
```
ACCV
Deep Active Ensemble Sampling For Image Classification Mohamadi, S., Doretto, G., and Adjeroh, D. A. In Proceedings of The Asian Conference on Computer Vision, 2022. abstract bibTeX arXiv pdf
Conventional active learning (AL) frameworks aim to reduce the cost of data annotation by actively requesting the labeling for the most informative data points. However, introducing AL to data hungry deep learning algorithms has been a challenge. Some proposed approaches include uncertainty-based techniques, geometric methods, implicit combination of uncertainty-based and geometric approaches, and more recently, frameworks based on semi/self supervised techniques. In this paper, we address two specific problems in this area. The first is the need for efficient exploitation/exploration trade-off in sample selection in AL. For this, we present an innovative integration of recent progress in both uncertainty-based and geometric frameworks to enable an efficient exploration/exploitation trade-off in sample selection strategy. To this end, we build on a computationally efficient approximate of Thompson sampling with key changes as a posterior estimator for uncertainty representation. Our framework provides two advantages: (1) accurate posterior estimation, and (2) tune-able trade-off between computational overhead and higher accuracy. The second problem is the need for improved training protocols in deep AL. For this, we use ideas from semi/self supervised learning to propose a general approach that is independent of the specific AL technique being used. Taken these together, our framework shows a significant improvement over the state-of-the-art, with results that are comparable to the performance of supervised-learning under the same setting. We show empirical results of our framework, and comparative performance with the state-of-the-art on four datasets, namely, MNIST, CIFAR10, CIFAR100 and ImageNet to establish a new baseline in two different settings.
```
@inproceedings{mohamadiDA22accv,
  abbr = {ACCV},
  author = {Mohamadi, S. and Doretto, G. and Adjeroh, D. A.},
  booktitle = {{Proceedings of The Asian Conference on Computer Vision}},
  title = {Deep Active Ensemble Sampling For Image Classification},
  year = {2022},
  month = dec,
  pages = {4531-4547},
  publisher = {Springer Berlin Heidelberg},
  arxiv = {2210.05770},
  bib2html_pubtype = {Conferences},
  owner = {doretto}
}
```
ISVC
Learning Representations for Masked Facial Recovery Randhawa, Z. A., Patel, S., Adjeroh, D. A., and Doretto, G. In Advances in Visual Computing, 2022. Oral abstract bibTeX arXiv pdf doi
The pandemic of these very recent years has led to a dramatic increase in people wearing protective masks in public venues. This poses obvious challenges to the pervasive use of face recognition technology that now is suffering a decline in performance. One way to address the problem is to revert to face recovery methods as a preprocessing step. Current approaches to face reconstruction and manipulation leverage the ability to model the face manifold, but tend to be generic. We introduce a method that is specific for the recovery of the face image from an image of the same individual wearing a mask. We do so by designing a specialized GAN inversion method, based on an appropriate set of losses for learning an unmasking encoder. With extensive experiments, we show that the approach is effective at unmasking face images. In addition, we also show that the identity information is preserved sufficiently well to improve face verification performance based on several face recognition benchmark datasets.
```
@inproceedings{randhawaPAD22isvc,
  abbr = {ISVC},
  author = {Randhawa, Z. A. and Patel, S. and Adjeroh, D. A. and Doretto, G.},
  booktitle = {Advances in Visual Computing},
  title = {Learning Representations for Masked Facial Recovery},
  year = {2022},
  address = {Cham},
  editor = {Bebis, G. and Li, B. and Yao, A. and Liu, Y. and Duan, Y. and Lau, M. and Khadka, R. and Crisan, A. and Chang, R.},
  pages = {22--35},
  month = oct,
  publisher = {Springer International Publishing},
  bib2html_pubtype = {Conferences},
  doi = {10.1007/978-3-031-20713-6_2},
  arxiv = {2212.14110},
  wwwnote = {Oral}
}
```
ISVC
Joint Discriminative and Metric Embedding Learning for Person Re-Identification Sabri, S. I., Randhawa, Z. A., and Doretto, G. In Advances in Visual Computing, 2022. Oral abstract bibTeX arXiv pdf doi
Person re-identification is a challenging task because of the high intra-class variance induced by the unrestricted nuisance factors of variations such as pose, illumination, viewpoint, background, and sensor noise. Recent approaches postulate that powerful architectures have the capacity to learn feature representations invariant to nuisance factors, by training them with losses that minimize intra-class variance and maximize inter-class separation, without modeling nuisance factors explicitly. The dominant approaches use either a discriminative loss with margin, like the softmax loss with the additive angular margin, or a metric learning loss, like the triplet loss with batch hard mining of triplets. Since the softmax imposes feature normalization, it limits the gradient flow supervising the feature embedding. We address this by joining the losses and leveraging the triplet loss as a proxy for the missing gradients. We further improve invariance to nuisance factors by adding the discriminative task of predicting attributes. Our extensive evaluation highlights that when only a holistic representation is learned, we consistently outperform the state-of-the-art on the three most challenging datasets. Such representations are easier to deploy in practical systems. Finally, we found that joining the losses removes the requirement for having a margin in the softmax loss while increasing performance.
```
@inproceedings{sabriRD22isvc,
  abbr = {ISVC},
  author = {Sabri, S. I. and Randhawa, Z. A. and Doretto, G.},
  booktitle = {Advances in Visual Computing},
  title = {Joint Discriminative and Metric Embedding Learning for Person Re-Identification},
  year = {2022},
  address = {Cham},
  editor = {Bebis, G. and Li, B. and Yao, A. and Liu, Y. and Duan, Y. and Lau, M. and Khadka, R. and Crisan, A. and Chang, R.},
  pages = {165--178},
  month = oct,
  publisher = {Springer International Publishing},
  bib2html_pubtype = {Conferences},
  doi = {10.1007/978-3-031-20716-7_13},
  arxiv = {2212.14107},
  wwwnote = {Oral}
}
```
CVPRW
Generative Probabilistic Novelty Detection With Isometric Adversarial Autoencoders Almohsen, R., Keaton, M. R., Adjeroh, D. A., and Doretto, G. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2022. Oral abstract bibTeX pdf doi
Learning the manifold of a complex distribution is a fundamental challenge for novelty or anomaly detection. We introduce a revised learning and inference procedure that takes into account a key underlying assumption made by the framework of generative probabilistic novelty detection. The traditional framework implies the ability to not only learn the manifold of the generative distribution of inliers but also to compute non-linear orthogonal projections onto this manifold from the ambient space. We augment the original training with priors that endow the model with this property, and prove that inference becomes easier and computationally more efficient. We show experimentally that the new framework leads to improved and more stable results.
```
@inproceedings{almohsenKAD22cvprw,
  abbr = {CVPRW},
  author = {Almohsen, R. and Keaton, M. R. and Adjeroh, D. A. and Doretto, G.},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops},
  title = {Generative Probabilistic Novelty Detection With Isometric Adversarial Autoencoders},
  year = {2022},
  month = jun,
  pages = {2002--2012},
  publisher = {IEEE},
  bib2html_pubtype = {Conferences},
  doi = {10.1109/cvprw56347.2022.00218},
  wwwnote = {Oral}
}
```
Emergency Medical Services Shock Index is the Most Accurate Predictor of Patient Outcomes After Blunt Torso Trauma Bardes, J. M., Price, B. S., Adjeroh, D. A., Doretto, G., and Wilson, A. Journal of Trauma and Acute Care Surgery, 2022. abstract bibTeX html doi
INTRODUCTION: Shock index (SI) and delta shock index (∆SI) predict mortality and blood transfusion in trauma patients. This study aimed to evaluate the predictive ability of SI and ∆SI in a rural environment with prolonged transport times and transfers from critical access hospitals or level IV trauma centers. METHODS: We completed a retrospective database review at an American College of Surgeons verified level 1 trauma center for 2 years. Adult subjects analyzed sustained torso trauma. Subjects with missing data or severe head trauma were excluded. For analysis, poisson regression and binomial logistic regression were used to study the effect of time in transport and SI/∆SI on resource utilization and outcomes. p < 0.05 was considered significant. RESULTS: Complete data were available on 549 scene patients and 127 transfers. Mean Injury Severity Score was 11 (interquartile range, 9.0) for scene and 13 (interquartile range, 6.5) for transfers. Initial emergency medical services SI was the most significant predictor for blood transfusion and intensive care unit care in both scene and transferred patients (p < 0.0001) compared with trauma center arrival SI or transferring center SI. A negative ∆SI was significantly associated with the need for transfusion and the number of units transfused. Longer transport time also had a significant relationship with increasing intensive care unit length of stay. Cohorts were analyzed separately. CONCLUSION: Providers must maintain a high level of clinical suspicion for patients who had an initially elevated SI. Emergency medical services SI was the greatest predictor of injury and need for resources. Enroute SI and ∆SI were less predictive as time from injury increased. This highlights the improvements in en route care but does not eliminate the need for high-level trauma intervention. LEVEL OF EVIDENCE: Therapeutic/care management, level IV.
```
@article{bardesPADW2021,
  author = {Bardes, J. M. and Price, B. S. and Adjeroh, D. A. and Doretto, G. and Wilson, A.},
  journal = {Journal of Trauma and Acute Care Surgery},
  title = {Emergency Medical Services Shock Index is the Most Accurate Predictor of Patient Outcomes After Blunt Torso Trauma},
  year = {2022},
  month = mar,
  number = {3},
  pages = {499--503},
  volume = {92},
  doi = {10.1097/ta.0000000000003483},
  publisher = {Ovid Technologies (Wolters Kluwer Health)},
  url = {https://doi.org/10.1097/ta.0000000000003483},
  bib2html_pubtype = {Journals}
}
```

BIBM
Detecting Drug-Drug Interactions using Protein Sequence-Structure Similarity Networks Islam, S., Abbasi, A., Agarwal, N., Zheng, W., Doretto, G., and Adjeroh, D. A. In Proceedings of the IEEE International Conference on Bioinformatics and Biomedicine, 2021. Oral abstract bibTeX pdf
Adverse drug events represent a key challenge in public health, especially with respect to drug safety profiling and drug surveillance. Drug-drug interactions represent one of the most popular types of adverse drug events. Most computational approaches to this problem have used different types of data, such as drug chemical structure, information about protein targets, side effects, pathways, etc to predict potential interactions between drugs. In this work, we study the question of whether using just genetic information about the drugs can provide significant information about the potential safety profile for a given drug. We propose a novel neural network model to predict adverse drug events using only data about the protein sequence and protein structure associated with the drug targets. We compare the results with those from the state-of-the-art methods on this problem. Our results show that the proposed method is quite competitive, at times outperforming the state-of-the-art.
```
@inproceedings{mohamadiDNAbibm,
  abbr = {BIBM},
  author = {Islam, S. and Abbasi, A. and Agarwal, N. and Zheng, W. and Doretto, G. and Adjeroh, D. A.},
  title = {{Detecting Drug-Drug Interactions using Protein Sequence-Structure Similarity Networks}},
  booktitle = {Proceedings of the IEEE International Conference on Bioinformatics and Biomedicine},
  year = {2021},
  bib2html_pubtype = {Conferences},
  wwwnote = {Oral}
}
```
BIBM
Human Age Estimation from Gene Expression Data using Artificial Neural Networks Mohamadi, S., Doretto, G., Nasrabadi, N. M., and Adjeroh, D. A. In Proceedings of the IEEE International Conference on Bioinformatics and Biomedicine, 2021. Oral abstract bibTeX arXiv
The study of signatures of aging in terms of genomic biomarkers can be uniquely helpful in understanding the mechanisms of aging and developing models to accurately predict the age. Prior studies have employed gene expression and DNA methylation data aiming at accurate prediction of age. In this line, we propose a new framework for human age estimation using information from human dermal fibroblast gene expression data. First, we propose a new spatial representation as well as a data augmentation approach for gene expression data. Next in order to predict the age, we design an architecture of neural network and apply it to this new representation of the original and augmented data, as an ensemble classification approach. Our experimental results suggest the superiority of the proposed framework over state-of-the-art age estimation methods using DNA methylation and gene expression data.
```
@inproceedings{mohamadiDNAbibn,
  abbr = {BIBM},
  author = {Mohamadi, S. and Doretto, G. and Nasrabadi, N. M. and Adjeroh, D. A.},
  title = {{Human Age Estimation from Gene Expression Data using Artificial Neural Networks}},
  booktitle = {Proceedings of the IEEE International Conference on Bioinformatics and Biomedicine},
  year = {2021},
  wwwnote = {Oral},
  arxiv = {2111.02692},
  bib2html_pubtype = {Conferences}
}
```
CVPRW
Fine-Grained Visual Classification of Plant Species In The Wild: Object Detection as A Reinforced Means of Attention Keaton, M. R., Zaveri, R. J., Kovur, M., Henderson, C., Adjeroh, D. A., and Doretto, G. In Proceedings of the IEEE CVPR Workshop on Fine-Grained Visual Categorization, 2021. abstract bibTeX arXiv pdf data
Plant species identification in the wild is a difficult problem in part due to the high variability of the input data, but also because of complications induced by the long-tail effects of the datasets distribution. Inspired by the most recent fine-grained visual classification approaches which are based on attention to mitigate the effects of data variability, we explore the idea of using object detection as a form of attention. We introduce a bottom-up approach based on detecting plant organs and fusing the predictions of a variable number of organ-based species classifiers. We also curate a new dataset with a long-tail distribution for evaluating plant organ detection and organ-based species identification, which is publicly available.
```
@inproceedings{keatonZKHAD21cvprw,
  abbr = {CVPRW},
  author = {Keaton, M. R. and Zaveri, R. J. and Kovur, M. and Henderson, C. and Adjeroh, D. A. and Doretto, G.},
  title = {{Fine-Grained Visual Classification of Plant Species In The Wild: Object Detection as A Reinforced Means of Attention}},
  booktitle = {Proceedings of the IEEE CVPR Workshop on Fine-Grained Visual Categorization},
  year = {2021},
  arxiv = {2106.02141},
  data = {https://github.com/wvuvl/DARMA},
  bib2html_pubtype = {Conferences}
}
```
Coronary artery disease phenotype detection in an academic hospital system setting Joseph, A., Mullett, C., Lilly, C., Armistead, M., Cox, H. J., Denney, M., Varma, M., Rich, D., Adjeroh, D. A., Doretto, G., Neal, W., and Pyles, L. A. Applied Clinical Informatics, 2021. abstract bibTeX doi
Background - The United States, and especially West Virginia, have a tremendous burden of coronary artery disease (CAD). Undiagnosed familial hypercholesterolemia (FH) is an important factor for CAD in the U.S. Identification of a CAD phenotype is an initial step to find families with FH. Objective - We hypothesized that a CAD phenotype detection algorithm that uses discrete data elements from electronic health records (EHRs) can be validated from EHR information housed in a data repository. Methods - We developed an algorithm to detect a CAD phenotype which searched through discrete data elements, such as diagnosis, problem lists, medical history, billing, and procedure (International Classification of Diseases [ICD]-9/10 and Current Procedural Terminology [CPT]) codes. The algorithm was applied to two cohorts of 500 patients, each with varying characteristics. The second (younger) cohort consisted of parents from a school child screening program. We then determined which patients had CAD by systematic, blinded review of EHRs. Following this, we revised the algorithm by refining the acceptable diagnoses and procedures. We ran the second algorithm on the same cohorts and determined the accuracy of the modification. Results - CAD phenotype Algorithm I was 89.6% accurate, 94.6% sensitive, and 85.6% specific for group 1. After revising the algorithm (denoted CAD Algorithm II) and applying it to the same groups 1 and 2, sensitivity 98.2%, specificity 87.8%, and accuracy 92.4; accuracy 93% for group 2. Group 1 F1 score was 92.4%. Specific ICD-10 and CPT codes such as “coronary angiography through a vein graft” were more useful than generic terms. Conclusion - We have created an algorithm, CAD Algorithm II, that detects CAD on a large scale with high accuracy and sensitivity (recall). It has proven useful among varied patient populations. Use of this algorithm can extend to monitor a registry of patients in an EHR and/or to identify a group such as those with likely FH.
```
@article{josephMLACDVRADNP2021aci,
  title = {Coronary artery disease phenotype detection in an academic hospital system setting},
  author = {Joseph, A. and Mullett, C. and Lilly, C. and Armistead, M. and Cox, H. J. and Denney, M. and Varma, M. and Rich, D. and Adjeroh, D. A. and Doretto, G. and Neal, W. and Pyles, L. A.},
  journal = {Applied Clinical Informatics},
  year = {2021},
  number = {1},
  volume = {12},
  pages = {10--16},
  doi = {10.1055/s-0040-1721012},
  bib2html_pubtype = {Journals}
}
```

BB
Deep learning for biological age estimation Ashiqur Rahman, S., Giacobbi, P., Pyles, L., Mullett, C., Doretto, G., and Adjeroh, D. A. Briefings in Bioinformatics, 2020. abstract bibTeX doi
Modern machine learning techniques (such as deep learning) offer immense opportunities in the field of human biological aging research. Aging is a complex process, experienced by all living organisms. While traditional machine learning and data mining approaches are still popular in aging research, they typically need feature engineering or feature extraction for robust performance. Explicit feature engineering represents a major challenge, as it requires significant domain knowledge. The latest advances in deep learning provide a paradigm shift in eliciting meaningful knowledge from complex data without performing explicit feature engineering. In this article, we review the recent literature on applying deep learning in biological age estimation. We consider the current data modalities that have been used to study aging and the deep learning architectures that have been applied. We identify four broad classes of measures to quantify the performance of algorithms for biological age estimation and based on these evaluate the current approaches. The paper concludes with a brief discussion on possible future directions in biological aging research using deep learning. This study has significant potentials for improving our understanding of the health status of individuals, for instance, based on their physical activities, blood samples and body shapes. Thus, the results of the study could have implications in different health care settings, from palliative care to public health.
```
@article{ashiqurrahmanGPMDA2020bb,
  abbr = {BB},
  title = {Deep learning for biological age estimation},
  author = {Ashiqur Rahman, S. and Giacobbi, P. and Pyles, L. and Mullett, C. and Doretto, G. and Adjeroh, D. A.},
  journal = {Briefings in Bioinformatics},
  month = may,
  year = {2020},
  issn = {1477-4054},
  doi = {10.1093/bib/bbaa021},
  bib2html_pubtype = {Journals}
}
```
CVPR
Adversarial Latent Autoencoders Pidhorskyi, S., Adjeroh, D. A., and Doretto, G. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2020. abstract bibTeX arXiv pdf code
Autoencoder networks are unsupervised approaches aiming at combining generative and representational properties by learning simultaneously an encoder-generator map. Although studied extensively, the issues of whether they have the same generative power of GANs, or learn disentangled representations, have not been fully addressed. We introduce an autoencoder that tackles these issues jointly, which we call Adversarial Latent Autoencoder (ALAE). It is a general architecture that can leverage recent improvements on GAN training procedures. We designed two autoencoders: one based on a MLP encoder, and another based on a StyleGAN generator, which we call StyleALAE. We verify the disentanglement properties of both architectures. We show that StyleALAE can not only generate 1024x1024 face images with comparable quality of StyleGAN, but at the same resolution can also produce face reconstructions and manipulations based on real images. This makes ALAE the first autoencoder able to compare with, and go beyond the capabilities of a generatoronly type of architecture.
```
@inproceedings{pidhorskyiAD2020cvpr,
  abbr = {CVPR},
  author = {Pidhorskyi, S. and Adjeroh, D. A. and Doretto, G.},
  booktitle = {Proceedings of the IEEE Computer Society Conference on Computer Vision
  	and Pattern Recognition},
  title = {Adversarial Latent Autoencoders},
  pages = {14104--14113},
  year = {2020},
  arxiv = {2004.04467},
  code = {https://github.com/podgorskiy/ALAE},
  bib2html_pubtype = {Conferences}
}
```

CVPRW
Unsupervised Learning of Paired Style Statistics for Unpaired Image Translation Motiian, S., Jones, Q., Pidhorskyi, S., and Doretto, G. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2019. abstract bibTeX pdf
Image-to-image translation has the goal of learning how to transform an input image from one domain as if it was from another domain, while preserving semantic and global information from the input. We present an image-to-image translation method that can be trained with unpaired images from source and target domains. However, we introduce a regularization that allows the model to specifically translate the local spatial statistic from one domain to another in an effort to leave unchanged gross structures and discourage translations of the semantic content. We do so by learning to generate paired images mapping the local statistic from one domain to the other. In turn, such images are used to improve the training of the translation networks, which become more focused on translating only the“style” of images while preserving the semantic content. Experiments on domain translation as well as domain adaptation highlight the effectiveness of our approach in comparison with the state-of-the-art.
```
@inproceedings{motiianJPD19cvprw,
  abbr = {CVPRW},
  author = {Motiian, S. and Jones, Q. and Pidhorskyi, S. and Doretto, G.},
  title = {{Unsupervised Learning of Paired Style Statistics for Unpaired Image Translation}},
  booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops},
  year = {2019},
  pages = {112--121},
  bib2html_pubtype = {Conferences}
}
```

arXiv
syGlass: Interactive Exploration of Multidimensional Images Using Virtual Reality Head-mounted Displays Pidhorskyi, S., Morehead, M., Jones, Q., Spirou, G., and Doretto, G. arXiv.org:1804.08197, 2018. abstract bibTeX arXiv pdf
The quest for deeper understanding of biological systems has driven the acquisition of increasingly larger multidimensional image datasets. Inspecting and manipulating data of this complexity is very challenging in traditional visualization systems. We developed syGlass, a software package capable of visualizing large scale volumetric data with inexpensive virtual reality head-mounted display technology. This allows leveraging stereoscopic vision to significantly improve perception of complex 3D structures, and provides immersive interaction with data directly in 3D. We accomplished this by developing highly optimized data flow and volume rendering pipelines, tested on datasets up to 16TB in size, as well as tools available in a virtual reality GUI to support advanced data exploration, annotation, and cataloguing.
```
@article{pidhorskyiMJSD18tr,
  abbr = {arXiv},
  author = {Pidhorskyi, S. and Morehead, M. and Jones, Q. and Spirou, G. and Doretto, G.},
  title = {{syGlass: Interactive Exploration of Multidimensional Images Using Virtual Reality Head-mounted Displays}},
  journal = {arXiv.org:1804.08197},
  arxiv = {1804.08197},
  month = apr,
  year = {2018},
  bib2html_pubtype = {Tech Reports}
}
```
NeurIPS
Generative probabilistic novelty detection with adversarial autoencoders Pidhorskyi, S., Almohsen, R., Adjeroh, D., and Doretto, G. In Neural Information Processing Systems, 2018. abstract bibTeX arXiv pdf code
Novelty detection is the problem of identifying whether a new data point is considered to be an inlier or an outlier. We assume that training data is available to describe only the inlier distribution. Recent approaches primarily leverage deep encoder-decoder network architectures to compute a reconstruction error that is used to either compute a novelty score or to train a one-class classifier. While we too leverage a novel network of that kind, we take a probabilistic approach and effectively compute how likely it is that a sample was generated by the inlier distribution. We achieve this with two main contributions. First, we make the computation of the novelty probability feasible because we linearize the parameterized manifold capturing the underlying structure of the inlier distribution, and show how the probability factorizes and can be computed with respect to local coordinates of the manifold tangent space. Second, we improve the training of the autoencoder network. An extensive set of results show that the approach achieves state-of-the-art performance on several benchmark datasets.
```
@inproceedings{pidhorskyiAAD18neurips,
  abbr = {NeurIPS},
  author = {Pidhorskyi, S. and Almohsen, R. and Adjeroh, D. and Doretto, G.},
  title = {Generative probabilistic novelty detection with adversarial autoencoders},
  booktitle = {Neural Information Processing Systems},
  year = {2018},
  pages = {6823--6834},
  arxiv = {1807.02588},
  code = {https://github.com/podgorskiy/GPND},
  bib2html_pubtype = {Conferences}
}
```
ACCV
Deep supervised hashing with spherical embedding Pidhorskyi, S., Jones, Q., Motiian, S., Adjeroh, D., and Doretto, G. In Proceedings of The Asian Conference on Computer Vision, 2018. abstract bibTeX pdf
Deep hashing approaches are widely applied to approximate nearest neighbor search for large-scale image retrieval. We propose Spherical Deep Supervised Hashing (SDSH), a new supervised deep hashing approach to learn compact binary codes. The goal of SDSH is to go beyond learning similarity preserving codes, by encouraging them to also be balanced and to maximize the mean average precision. This is enabled by advocating the use of a different relaxation method, allowing the learning of a spherical embedding, which overcomes the challenge of maintaining the learning problem well-posed without the need to add extra binarizing priors. This allows the formulation of a general triplet loss framework, with the introduction of the spring loss for learning balanced codes, and of the ability to learn an embedding quantization that maximizes the mean average precision. Extensive experiments demonstrate that the approach compares favorably with the state-of-the-art while providing significant performance increase at more compact code sizes.
```
@inproceedings{pidhorskyiJMAD18accv,
  abbr = {ACCV},
  author = {Pidhorskyi, S. and Jones, Q. and Motiian, S. and Adjeroh, D. and Doretto, G.},
  title = {Deep supervised hashing with spherical embedding},
  booktitle = {Proceedings of The Asian Conference on Computer Vision},
  year = {2018},
  pages = {417--434},
  bib2html_pubtype = {Conferences}
}
```

NIPS
Few-shot adversarial domain adaptation Motiian, S., Jones, Q., Iranmanesh, S. M., and Doretto, G. In Advances in Neural Information Processing Systems, 2017. abstract bibTeX arXiv pdf
This work provides a framework for addressing the problem of supervised domain adaptation with deep models. The main idea is to exploit adversarial learning to learn an embedded subspace that simultaneously maximizes the confusion between two domains while semantically aligning their embedding. The supervised setting becomes attractive especially when there are only a few target data samples that need to be labeled. In this few-shot learning scenario, alignment and separation of semantic probability distributions is difficult because of the lack of data. We found that by carefully designing a training scheme whereby the typical binary adversarial discriminator is augmented to distinguish between four different classes, it is possible to effectively address the supervised adaptation problem. In addition, the approach has a high “speed” of adaptation, i.e. it requires an extremely low number of labeled target training samples, even one per category can be effective. We then extensively compare this approach to the state of the art in domain adaptation in two experiments: one using datasets for handwritten digit recognition, and one using datasets for visual object recognition.
```
@inproceedings{motiianJID17nips,
  abbr = {NIPS},
  author = {Motiian, S. and Jones, Q. and Iranmanesh, S. M. and Doretto, G.},
  title = {Few-shot adversarial domain adaptation},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2017},
  pages = {6652--6662},
  arxiv = {1711.02536},
  bib2html_pubtype = {Conferences}
}
```

ICCV

Unified deep supervised domain adaptation andgeneralization Motiian, S., Piccirilli, M., and Adjeroh, G. In Proceedings of IEEE International Conference on Computer Vision, 2017. bibTeX arXiv pdf webpage code

@inproceedings{motiianPAD17iccv,
  abbr = {ICCV},
  author = {Motiian, S. and Piccirilli, M. and Adjeroh, D. adn Doretto, G.},
  title = {Unified deep supervised domain adaptation andgeneralization},
  booktitle = {Proceedings of IEEE International Conference on Computer Vision},
  year = {2017},
  pages = {5715--5725},
  arxiv = {1709.10190},
  code = {https://github.com/samotiian/CCSA},
  bib2html_pubtype = {Conferences},
  webpage = {{https://vision.csee.wvu.edu/~motiian/Details/CCSA.html}}
}

TCSVT
Online Human Interaction Detection and Recognition with Multiple Cameras Motiian, S., Siyahjani, F., Almohsen, R., and Doretto, G. IEEE Transactions on Circuits and Systems for Video Technology, 2017. abstract bibTeX pdf
We address the problem of detecting and recognizing online the occurrence of human interactions as seen by a network of multiple cameras. We represent interactions by forming temporal trajectories, coupling together the body motion of each individual and their proximity relationships with others, and also sound whenever available. Such trajectories are modeled with kernel state-space (KSS) models. Their advantage is being suitable for the online interaction detection, recognition, and also for fusing information from multiple cameras, while enabling a fast implementation based on online recursive updates. For recognition, in order to compare interaction trajectories in the space of KSS models, we design so-called pairwise kernels with a special symmetry. For detection, we exploit the geometry of linear operators in Hilbert space, and extend to KSS models the concept of parity space, originally defined for linear models. For fusion, we combine KSS models with kernel construction and multiview learning techniques. We extensively evaluate the approach on four single view publicly available data sets, and we also introduce, and will make public, a new challenging human interactions data set that we have collected using a network of three cameras. The results show that the approach holds promise to become an effective building block for the analysis of real-time human behavior from multiple cameras.
```
@article{motiianSAD2015tcsvt,
  abbr = {TCSVT},
  author = {Motiian, S. and Siyahjani, F. and Almohsen, R. and Doretto, G.},
  title = {{Online Human Interaction Detection and Recognition with Multiple Cameras}},
  journal = {IEEE Transactions on Circuits and Systems for Video Technology},
  year = {2017},
  volume = {27},
  number = {3},
  pages = {649--663},
  owner = {doretto},
  timestamp = {2015.09.16},
  bib2html_pubtype = {Journals}
}
```
PlorOne
A framework for analyzing the whole-body surface area from a single view Piccirilli, M., Doretto, G., and Adjeroh, D. Plos One, 2017. abstract bibTeX pdf
We present a virtual reality (VR) framework for the analysis of whole human body surface area. Usual methods for determining the whole body surface area (WBSA) are based on well known formulae, characterized by large errors when the subject is obese, or belongs to certain subgroups. For these situations, we believe that a computer vision approach can overcome these problems and provide a better estimate of this important body indicator. Unfortunately, using machine learning techniques to design a computer vision system able to provide a new body indicator that goes beyond the use of only body weight and height, entails a long and expensive data acquisition process. A more viable solution is to use a dataset composed of virtual subjects. Generating a virtual dataset allowed us to build a population with different characteristics (obese, underweight, age, gender). However, synthetic data might differ from a real scenario, typical of the physician’s clinic. For this reason we develop a new virtual environment to facilitate the analysis of human subjects in 3D. This framework can simulate the acquisition process of a real camera, making it easy to analyze and to create training data for machine learning algorithms. With this virtual environment, we can easily simulate the real setup of a clinic, where a subject is standing in front of a camera, or may assume a different pose with respect to the camera. We use this newly designated environment to analyze the whole body surface area (WBSA). In particular, we show that we can obtain accurate WBSA estimations with just one view, virtually enabling the possibility to use inexpensive depth sensors (e.g., the Kinect) for large scale quantification of the WBSA from a single view 3D map.
```
@article{piccirilliDA2015pone,
  abbr = {PlorOne},
  author = {Piccirilli, M. and Doretto, G. and Adjeroh, D.},
  title = {A framework for analyzing the whole-body surface area from a single view},
  journal = {Plos One},
  year = {2017},
  volume = {12},
  number = {1},
  pages = {e0166749},
  owner = {doretto},
  timestamp = {2015.09.16},
  bib2html_pubtype = {Journals}
}
```

Rapid and semiautomatedextraction of neuronal cell bodies and nuclei from electron microscopy image stacks Holcomb, P. S., Morehead, M., Doretto, G., Chen, P., Berg, S., Plaza, S., and Spirou, G. Methods in Molecular Biology, 2016. abstract bibTeX pdf
Connectomics—the study of how neurons wire together in the brain—is at the forefront of modern neuroscience research. However, many connectomics studies are limited by the time and precision needed to correctly segment large volumes of electron microscopy (EM) image data. We present here a semi-automated segmentation pipeline using freely available software that can significantly decrease segmentation time for extracting both nuclei and cell bodies from EM image volumes.
```
@article{holcombMDCBPS16methods,
  author = {Holcomb, P. S. and Morehead, M. and Doretto, G. and Chen, P. and Berg, S. and Plaza, S. and Spirou, G.},
  title = {Rapid and semiautomatedextraction of neuronal cell bodies and nuclei from electron microscopy image stacks},
  journal = {Methods in Molecular Biology},
  year = {2016},
  volume = {1427},
  pages = {277--290},
  bib2html_pubtype = {Journals}
}
```
ECCV
Information bottleneck domain adaptation with privileged information forvisual recognition Motiian, S., and Doretto, G. In Proceedings of European Conference on Computer Vision, 2016. abstract bibTeX pdf
We address the unsupervised domain adaptation problem for visual recognition when an auxiliary data view is available during training. This is im- portant because it allows improving the training of visual classifiers on a new tar- get visual domain when paired additional source data is cheaply available. This is the case when we learn from a source of RGB plus depth data, for then test on a new RGB domain. The problem is challenging because of the intrinsic asymmetry caused by the missing auxiliary view during testing and from which discrimina- tive information should be carried over to the new domain. We jointly account for the auxiliary view during training and for the domain shift by extending the information bottleneck method, and by combining it with risk minimization. In this way, we establish an information theoretic principle for learning any type of visual classifier under this particular settings. We use this principle to design a multi-class large-margin classifier with an efficient optimization in the primal space. We extensively compare our method with the state-of-the-art on several datasets, by effectively learning from RGB plus depth data to recognize objects and gender from a new RGB domain.
```
@inproceedings{motiianD16eccv,
  abbr = {ECCV},
  author = {Motiian, S. and Doretto, G.},
  title = {Information bottleneck domain adaptation with privileged information forvisual recognition},
  booktitle = {Proceedings of European Conference on Computer Vision},
  year = {2016},
  pages = {630--647},
  bib2html_pubtype = {Conferences}
}
```
CVPR
Information bottleneck learning using privileged information for visual recognition Motiian, S., Piccirilli, M., Adjeroh, D., and Doretto, G. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016. abstract bibTeX pdf
We explore the visual recognition problem from a main data view when an auxiliary data view is available during training. This is important because it allows improving the training of visual classifiers when paired additional data is cheaply available, and it improves the recognition from multi-view data when there is a missing view at testing time. The problem is challenging because of the intrinsic asym- metry caused by the missing auxiliary view during testing. We account for such view during training by extending the information bottleneck method, and by combining it with risk minimization. In this way, we establish an information theoretic principle for leaning any type of visual classifier under this particular setting. We use this principle to design a large-margin classifier with an efficient optimization in the primal space. We extensively compare our method with the state-of-the-art on different visual recognition datasets, and with different types of auxiliary data, and show that the proposed framework has a very promising potential.
```
@inproceedings{motiianPAD16cvpr,
  abbr = {CVPR},
  author = {Motiian, S. and Piccirilli, M. and Adjeroh, D. and Doretto, G.},
  title = {Information bottleneck learning using privileged information for visual recognition},
  booktitle = {Proceedings of the IEEE Computer Society Conference on Computer Vision
  	and Pattern Recognition},
  year = {2016},
  pages = {1496--1505},
  bib2html_pubtype = {Conferences}
}
```
Automated classification of vowel category and speaker type in the high-frequency spectrum Donai, J. J., Motiian, S., and Doretto, G. Audiology Research, 2016. abstract bibTeX pdf
The high-frequency region of vowel signals (above the third formant or F3) has received little research attention. Recent evidence, however, has documented the perceptual utility of high-frequency information in the speech signal above the traditional frequency bandwidth known to contain important cues for speech and speaker recognition. The purpose of this study was to determine if high-pass filtered vowels could be separated by vowel category and speaker type in a supervised learning framework. Mel frequency cepstral coefficients (MFCCs) were extracted from productions of six vowel categories produced by two male, two female, and two child speakers. Results revealed that the filtered vowels were well separated by vowel category and speaker type using MFCCs from the high-frequency spectrum. This demonstrates the presence of useful information for auto- mated classification from the high-frequency region and is the first study to report findings of this nature in a supervised learning framework.
```
@article{donaiMD2015ar,
  author = {Donai, J. J. and Motiian, S. and Doretto, G.},
  title = {Automated classification of vowel category and speaker type in the high-frequency spectrum},
  journal = {Audiology Research},
  year = {2016},
  volume = {6},
  number = {1},
  pages = {137},
  owner = {doretto},
  timestamp = {2015.09.16},
  bib2html_pubtype = {Journals}
}
```
IEEESJ
A Mobile Structured Light System for 3D Face Acquisition Piccirilli, M., Doretto, G., Ross, A., and Adjeroh, D. IEEE Sensors Journal, 2016. abstract bibTeX pdf
A mobile sensor based on fringe projection techniques is developed with the goal of acquiring face 3D and color with a smartphone device. The system consists of a portable pico-projector and an Android-based smartphone. The data acquisition, pattern generation. and reconstruction of the final 3D point cloud are all driven by the smartphone. We present results on the root-mean-square error (RMSE) of the sensor and on 3D face matching.
```
@article{piccirilliDRA2015sj,
  abbr = {IEEESJ},
  author = {Piccirilli, M. and Doretto, G. and Ross, A. and Adjeroh, D.},
  title = {{A Mobile Structured Light System for 3D Face Acquisition}},
  journal = {IEEE Sensors Journal},
  year = {2016},
  volume = {16},
  number = {7},
  pages = {1854--1855},
  owner = {doretto},
  timestamp = {2015.09.16},
  bib2html_pubtype = {Journals}
}
```

ICCV
A Supervised Low-rank Method for Learning Invariant Subspaces Siyahjani, F., Almohsen, R., Sabri, S., and Doretto, G. In Proceedings of IEEE International Conference on Computer Vision, 2015. abstract bibTeX pdf
Sparse representation and low-rank matrix decomposi- tion approaches have been successfully applied to several computer vision problems. They build a generative repre- sentation of the data, which often requires complex training as well as testing to be robust against data variations in- duced by nuisance factors. We introduce the invariant com- ponents, a discriminative representation invariant to nui- sance factors, because it spans subspaces orthogonal to the space where nuisance factors are defined. This allows de- veloping a framework based on geometry that ensures a uni- form inter-class separation, and a very efficient and robust classification based on simple nearest neighbor. In addi- tion, we show how the approach is equivalent to a local metric learning, where the local metrics (one for each class) are learned jointly, rather than independently, thus avoiding the risk of overfitting without the need for additional regu- larization. We evaluated the approach for face recognition with highly corrupted training and testing data, obtaining very promising results.
```
@inproceedings{siyahjaniASD15iccv,
  abbr = {ICCV},
  author = {Siyahjani, F. and Almohsen, R. and Sabri, S. and Doretto, G.},
  title = {{A Supervised Low-rank Method for Learning Invariant Subspaces}},
  booktitle = {Proceedings of IEEE International Conference on Computer Vision},
  year = {2015},
  pages = {4220--4228},
  owner = {doretto},
  timestamp = {2015.09.16},
  bib2html_pubtype = {Conferences}
}
```
BMEoL
Automated Extraction and Validation of Children’s Gait Parameters with the Kinect Motiian, S., Pergami, P., Guffey, K., Mancinelli, C. A., and Doretto, G. BioMedical Engineering OnLine, 2015. abstract bibTeX pdf doi
Background: Gait analysis for therapy regimen prescription and monitoring requires patients to physically access clinics with specialized equipment. The timely availability of such infrastructure at the right frequency is especially important for small children. Besides being very costly, this is a challenge for many children living in rural areas. This is why this work develops a low-cost, portable, and automated approach for in-home gait analysis, based on the Microsoft Kinect. Methods: A robust and efficient method for extracting gait parameters is intro- duced, which copes with the high variability of noisy Kinect skeleton tracking data experienced across the population of young children. This is achieved by temporally segmenting the data with an approach based on coupling a probabilistic matching of stride template models, learned offline, with the estimation of their global and local temporal scaling. A preliminary study conducted on healthy children between 2 and 4 years of age is performed to analyze the accuracy, precision, repeatability, and con- current validity of the proposed method against the GAITRite when measuring several spatial and temporal children’s gait parameters. Results: The method has excellent accuracy and good precision, with segmenting temporal sequences of body joint locations into stride and step cycles. Also, the spatial and temporal gait parameters, estimated automatically, exhibit good concurrent valid- ity with those provided by the GAITRite, as well as very good repeatability. In particular, on a range of nine gait parameters, the relative and absolute agreements were found to be good and excellent, and the overall agreements were found to be good and moderate. Conclusion: This work enables and validates the automated use of the Kinect for chil- dren’s gait analysis in healthy subjects. In particular, the approach makes a step forward towards developing a low-cost, portable, parent-operated in-home tool for clinicians assisting young children.
```
@article{motiianPGMD15bmeol,
  abbr = {BMEoL},
  author = {Motiian, S. and Pergami, P. and Guffey, K. and Mancinelli, C. A. and Doretto, G.},
  title = {{Automated Extraction and Validation of Children's Gait Parameters with the Kinect}},
  journal = {{BioMedical Engineering OnLine}},
  year = {2015},
  volume = {14},
  pages = {112},
  doi = {https://doi.org/10.1186/s12938-015-0102-9},
  owner = {doretto},
  timestamp = {2015.09.16},
  bib2html_pubtype = {Journals}
}
```

Dynamical Systems in Video Analysis Doretto, G., Ravichandran, A., Vidal, R., and Soatto, S. In Academic Press Library in Signal Processing, 2014. bibTeX

@incollection{dorettoRVS13chapter,
  author = {Doretto, G. and Ravichandran, A. and Vidal, R. and Soatto, S.},
  title = {Dynamical Systems in Video Analysis},
  booktitle = {Academic Press Library in Signal Processing},
  publisher = {Elsevier Ltd.},
  year = {2014},
  editor = {Chellapa, R. and Theodoridis, S.},
  volume = {4},
  chapter = {16},
  edition = {1},
  month = oct,
  bib2html_pubtype = {Book Chapters},
  bib2html_rescat = {Dynamic Textures, Visual Motion Analysis, Visual Motion Recognition},
  owner = {doretto},
  timestamp = {2013.05.25}
}

3DUI
BrainTrek: An Immersive Environment for Investigating Neuronal Tissue Morehead, M., Jones, Q., Blatt, J., Holcomb, P., Schultz, J., DeFanti, T., Ellisman, M., Doretto, G., and Spirou, G. A. In Proceedings of the IEEE Symposium on 3D User Interfaces, 2014. abstract bibTeX pdf
The high degree of complexity in cellular and circuit structure of the brain poses challenges for understanding tissue organization, extrapolated from large serial sections electron microscopy (ssEM) image data. We advocate the use of 3D immersive virtual reality (IVR) to facilitate the human analysis of such data. We have developed and evaluated the BrainTrek system: a CAVE-based IVR environment with a dedicated and intuitive user interface tailored to the investigation of neural tissue by scientists and educators.
```
@inproceedings{moreheadJBHSFEDS143dui,
  abbr = {3DUI},
  author = {Morehead, M. and Jones, Q. and Blatt, J. and Holcomb, P. and Schultz, J. and DeFanti, T. and Ellisman, M. and Doretto, G. and Spirou, G. A.},
  title = {Brain{T}rek: {A}n Immersive Environment for Investigating Neuronal
  	Tissue},
  booktitle = {Proceedings of the IEEE Symposium on 3D User Interfaces},
  year = {2014},
  pages = {157--158},
  address = {Minneapolis, MN},
  month = mar,
  bib2html_pubtype = {Conferences},
  file = {moreheadJBHSFEDS143dui.pdf:doretto/conference/moreheadJBHSFEDS143dui.pdf:PDF}
}
```
ICME
Online Geometric Human Interaction Segmentation and Recognition Siyahjani, F., Motiian, S., Bharthavarapu, H., Sharlemin, S., and Doretto, G. In Proceedings of IEEE International Conference on Multimedia and Expo, 2014. abstract bibTeX pdf
We address the problem of online temporal segmentation and recognition of human interactions in video sequences. The complexity of the high-dimensional data variability representing interactions is handled by combining kernel methods with linear models, giving rise to kernel regression and kernel state space models. By exploiting the geometry of linear operators in Hilbert space, we show how the concept of parity space, defined for linear models, generalizes to the kernellized extensions. This provides a powerful and flexible framework for online temporal segmentation and recognition. We extensively evaluate the approach on a publicly available dataset, and on a new challenging human interactions dataset that we have collected. The results show that the approach holds the promise to become an effective building block for the analysis in real-time of human behavior.
```
@inproceedings{siyahjaniMBSD14icme,
  abbr = {ICME},
  author = {Siyahjani, F. and Motiian, S. and Bharthavarapu, H. and Sharlemin, S. and Doretto, G.},
  title = {Online Geometric Human Interaction Segmentation and Recognition},
  booktitle = {Proceedings of IEEE International Conference on Multimedia and Expo},
  year = {2014},
  month = jul,
  bib2html_pubtype = {Conferences},
  bib2html_rescat = {Interaction Recognition, Video Analysis},
  file = {siyahjaniMBSD14icme.pdf:doretto/conference/siyahjaniMBSD14icme.pdf:PDF},
  owner = {doretto},
  timestamp = {2014.05.21}
}
```
System and method to manage delivery of healthcare to a patient Johnson, C., Akbay, K., Day, A., Doretto, G., Tu, P. H., Dulgeroglu, O., Peterson, M., and Toledano, D. U.S. Patent 8,799,008 B2, Issued aug 5, 2014. abstract bibTeX html
A system and method to manage delivery of healthcare via a plurality of resources to a patient is provided. The system and method track and output a signal representative of a location of at least one of a series of resources relative to a control volume associated with the patient, acquire at least one signal representative of detecting ingress or egress of at least one of the plurality of resources relative to the control volume; and output a first signal representative of one of a series of milestones as defined in a predetermined protocol in response to detecting ingress or egress of at least one of the resources relative to the control volume.
```
@patent{johnsonADDTDPTpatent,
  title = {System and method to manage delivery of healthcare to a patient},
  nationality = {U.S.},
  number = {8,799,008 B2},
  year = {2014},
  author = {Johnson, C. and Akbay, K. and Day, A. and Doretto, G. and Tu, P. H. and Dulgeroglu, O. and Peterson, M. and Toledano, D.},
  day = {5},
  month = aug,
  url = {https://patents.google.com/patent/US8799008B2/en},
  bib2html_dl_html = {https://patents.google.com/patent/US8799008B2/en},
  bib2html_pubtype = {Patents},
  owner = {doretto},
  timestamp = {2009.09.28}
}
```

ISVC
Pairwise Kernels for Human Interaction Recognition Motiian, S., Feng, K., Bharthavarapu, H., Sharlemin, S., and Doretto, G. In Advances in Visual Computing, 2013. Oral abstract bibTeX pdf html doi
In this paper we model binary people interactions by forming tempo- ral interaction trajectories, under the form of a time series, coupling together the body motion of each individual as well as their proximity relationships. Such tra- jectories are modeled with a non-linear dynamical system (NLDS). We develop a framework that entails the use of so-called pairwise kernels, able to compare interaction trajectories in the space of NLDS. To do so we address the problem of modeling the Riemannian structure of the trajectory space, and we also prove that kernels have to satisfy certain symmetry properties, which are peculiar of this interaction modeling framework. Experiment results show that this approach is quite promising, as it is able to match and improve state-of-the-art classification and retrieval accuracies on two human interaction datasets.
```
@incollection{motiianFBSD13isvc,
  abbr = {ISVC},
  author = {Motiian, S. and Feng, K. and Bharthavarapu, H. and Sharlemin, S. and Doretto, G.},
  title = {Pairwise Kernels for Human Interaction Recognition},
  booktitle = {Advances in Visual Computing},
  publisher = {Springer Berlin Heidelberg},
  year = {2013},
  volume = {8034},
  series = {Lecture Notes in Computer Science},
  pages = {210-221},
  bib2html_pubtype = {Conferences},
  bib2html_rescat = {Interaction Recognition, Video Analysis},
  doi = {10.1007/978-3-642-41939-3_21},
  file = {motiianFBSD13isvc.pdf:doretto/conference/motiianFBSD13isvc.pdf:PDF},
  isbn = {978-3-642-41938-6},
  owner = {doretto},
  timestamp = {2013.10.18},
  url = {http://dx.doi.org/10.1007/978-3-642-41939-3_21},
  wwwnote = {Oral}
}
```
Virtual simulator for testing a vision based pose estimation system for autonomous capture of satellites with interface rings Velasquez, A. F., Marani, G., Evans, T., Napolitano, M. R., Christian, J. A., and Doretto, G. In 21st Mediterranean Conference on Control and Automation (MED), 2013. Oral abstract bibTeX pdf doi
This paper describes the design and the performance of a virtual simulation environment to evaluate a machine vision based pose estimation system used for the general problem of satellite servicing. The vision system features a wide angle monocular camera to track the interface ring of a non-cooperative satellite using ellipse extraction. The effects of the camera parameters and of the relative camera-satellite position on the system accuracy are evaluated. Different parameters such as resolution, field of view angle, distortion, occlusions and errors in the intrinsic parameters are considered. The study reveals the importance of the availability of such a visual simulation environment for the purpose of mission planning.
```
@inproceedings{velasquezMENCD13med,
  author = {Velasquez, A. F. and Marani, G. and Evans, T. and Napolitano, M. R. and Christian, J. A. and Doretto, G.},
  title = {Virtual simulator for testing a vision based pose estimation system
  	for autonomous capture of satellites with interface rings},
  booktitle = {21st Mediterranean Conference on Control and Automation (MED)},
  year = {2013},
  pages = {1597-1602},
  bib2html_pubtype = {Conferences},
  bib2html_rescat = {Vision for Robotics},
  doi = {10.1109/MED.2013.6608935},
  file = {velasquezMENCD13med.pdf:doretto/conference/velasquezMENCD13med.pdf:PDF},
  keywords = {Accuracy;Calibration;Cameras;Estimation;Noise;Satellites;Testing},
  owner = {doretto},
  timestamp = {2013.10.18},
  wwwnote = {Oral}
}
```
Identifying descriptor for person or object in an image Doretto, G., Wang, X., Sebastian, T., Rittscher, J., and Tu, P. H. U.S. Patent 8,452,096 B2, Issued may 28, 2013. abstract bibTeX html
A device and method for processing an image to create appearance and shape labeled images of a person or object captured within the image. The appearance and shape labeled images are unique properties of the person or object and can be used to re-identify the person or object in subsequent images. The appearance labeled image is an aggregate of pre-stored appearance labels that are assigned to image segments of the image based on calculated appearance attributes of each image segment. The shape labeled image is an aggregate of pre-stored shape labels that are assigned to image segments of the image based on calculated shape attributes of each image segment. An identifying descriptor of the person or object can be computed based on both the appearance labeled image and the shape labeled image. The descriptor can be compared with other descriptors of later captured images to re-identify a person or object.
```
@patent{dorettoWSRTpatentB,
  title = {Identifying descriptor for person or object in an image},
  nationality = {U.S.},
  number = {8,452,096 B2},
  year = {2013},
  author = {Doretto, G. and Wang, X. and Sebastian, T. and Rittscher, J. and Tu, P. H.},
  day = {28},
  month = may,
  url = {https://patents.google.com/patent/US8452096B2/en},
  bib2html_dl_html = {https://patents.google.com/patent/US8452096B2/en},
  bib2html_pubtype = {Patents},
  bib2html_rescat = {Video Analysis},
  owner = {doretto},
  timestamp = {2007.01.20}
}
```
Identifying descriptor for person and object in an image Doretto, G., Wang, X., Sebastian, T., Rittscher, J., and Tu, P. H. U.S. Patent 8,457,406 B2, Issued jun 4, 2013. abstract bibTeX html
A device and method for processing an image to create appearance and shape labeled images of a person or object captured within the image. The appearance and shape labeled images are unique properties of the person or object and can be used to re-identify the person or object in subsequent images. The appearance labeled image is an aggregate of pre-stored appearance labels that are assigned to image segments of the image based on calculated appearance attributes of each image segment. The shape labeled image is an aggregate of pre-stored shape labels that are assigned to image segments of the image based on calculated shape attributes of each image segment. An identifying descriptor of the person or object can be computed based on both the appearance labeled image and the shape labeled image. The descriptor can be compared with other descriptors of later captured images to re-identify a person or object.
```
@patent{dorettoWSRTpatentC,
  title = {Identifying descriptor for person and object in an image},
  nationality = {U.S.},
  number = {8,457,406 B2},
  year = {2013},
  author = {Doretto, G. and Wang, X. and Sebastian, T. and Rittscher, J. and Tu, P. H.},
  day = {4},
  month = jun,
  url = {https://patents.google.com/patent/US8457406B2/en},
  bib2html_dl_html = {https://patents.google.com/patent/US8457406B2/en},
  bib2html_pubtype = {Patents},
  bib2html_rescat = {Video Analysis},
  owner = {doretto},
  timestamp = {2007.01.20}
}
```
Method and system for crowd segmentation Tu, P. H., Turner, K. R., Rittscher, J., Sebastian, T. B., Krahnstoever, N. O., Doretto, G., and Yu, T. U.S. Patent 8,355,576 B2, Issued jan 15, 2013. abstract bibTeX html
Aspects of the disclosure provide a method for crowd segmentation that can globally optimize crowd segmentation of an input image based on local information of the input image. The method can include receiving an input image of a site, initializing a plurality of hypothesis based on the input image, dividing the input image into a plurality of patches, calculating an affinity measure of one or more patches to a hypothesis based on a partial response of the patches to a whole body classifier of the hypothesis that includes a combination of weak classifiers, and optimizing assignments of the plurality of patches to the plurality of hypothesis based on the affinity measures of the plurality of patches to the plurality of hypothesis.
```
@patent{tuTRSKDYpatent,
  title = {Method and system for crowd segmentation},
  nationality = {U.S.},
  number = {8,355,576 B2},
  year = {2013},
  author = {Tu, P. H. and Turner, K. R. and Rittscher, J. and Sebastian, T. B. and Krahnstoever, N. O. and Doretto, G. and Yu, T.},
  day = {15},
  month = jan,
  url = {https://patents.google.com/patent/US8355576B2/en},
  bib2html_dl_html = {https://patents.google.com/patent/US8355576B2/en},
  bib2html_pubtype = {Patents},
  bib2html_rescat = {Video Analysis},
  howpublished = {US 8355576},
  owner = {doretto},
  timestamp = {2007.01.20}
}
```

PRL
M-VIVIE: A multi-thread video indexer via identity extraction De Marsico, M., Doretto, G., and Riccio, D. Pattern Recognition Letters, 2012. abstract bibTeX pdf html doi
In this work we present M-VIVIE, a system for video sequence indexing based on the identity of appearing subjects, which exploits a multithread architecture for fast processing. The system is composed of more conceptual component modules, each performing a specific kind of processing. Each module can possibly be substituted with a different one performing the same task. Special attention is devoted to classification and clustering activities. M-VIVIE presents a number of peculiarities, that distinguish it from existing systems with similar goals. Among these, the account for the concomitant appearance of two identities in the same clip, and the use of such information in the process of identity mapping. M-VIVIE was tested on different kinds of real video clips to assess both its efficacy and efficiency.
```
@article{demarsicoDR12prl,
  abbr = {PRL},
  author = {De Marsico, M. and Doretto, G. and Riccio, D.},
  title = {{M-VIVIE}: {A} multi-thread video indexer via identity extraction},
  journal = {Pattern Recognition Letters},
  year = {2012},
  volume = {33},
  pages = {1882--1890},
  number = {14},
  bib2html_pubtype = {Journals},
  bib2html_rescat = {Video Indexing, Human Reidentification, Identity Management, Video
  	Surveillance},
  doi = {10.1016/j.patrec.2012.03.005},
  file = {demarsicoDR12prl.pdf:doretto/journal/demarsicoDR12prl.pdf:PDF},
  issn = {0167-8655},
  keywords = {Video indexing},
  owner = {doretto},
  timestamp = {2012.12.03},
  url = {http://www.sciencedirect.com/science/article/pii/S0167865512000803}
}
```
ACCV
Learning a Context Aware Dictionary for Sparse Representation Siyahjani, F., and Doretto, G. In Proceedings of The Asian Conference on Computer Vision, 2012. Oral abstract bibTeX pdf
Recent successes in the use of sparse coding for many com- puter vision applications have triggered the attention towards the prob- lem of how an over-complete dictionary should be learned from data. This is because the quality of a dictionary greatly affects performance in many respects, including computational. While so far the focus has been on learning compact, reconstructive, and discriminative dictionar- ies, in this work we propose to retain the previous qualities, and further enhance them by learning a dictionary that is able to predict the con- textual information surrounding a sparsely coded signal. The proposed framework leverages the K-SVD for learning, fully inheriting its benefits of simplicity and efficiency. A model of structured prediction is designed around this approach, which leverages contextual information to improve the combined recognition and localization of multiple objects from multi- ple classes within one image. Results on the PASCAL VOC 2007 dataset are in line with the state-of-the-art, and clearly indicate that this is a viable approach for learning a context aware dictionary for sparse repre- sentation.
```
@inproceedings{siyahjaniD12accv,
  abbr = {ACCV},
  author = {Siyahjani, F. and Doretto, G.},
  title = {Learning a Context Aware Dictionary for Sparse Representation},
  booktitle = {Proceedings of The Asian Conference on Computer Vision},
  year = {2012},
  pages = {1--14},
  bib2html_pubtype = {Conferences},
  bib2html_rescat = {Dictionary Learning, Sparse Coding, Object Detection, Context, Object
  	Classification},
  file = {siyahjaniD12accv.pdf:doretto/conference/siyahjaniD12accv.pdf:PDF},
  owner = {doretto},
  timestamp = {2012.12.03},
  wwwnote = {Oral}
}
```
Device and method for fast computation of region based image features Doretto, G., Wang, X., Sebastian, T., Rittscher, J., and Tu, P. H. U.S. Patent 8,184,915 B2, Issued may 22, 2012. abstract bibTeX html
A device and method for efficient computation of statistical information, such as a mean, co-variance, or histogram of the image pixels, over discrete image regions. The computation employs integral computations to determine the statistical information over image regions of arbitrary shape, including irregular polygonal shaped regions. The integral computations are simplified by categorizing corner points of boundaries of image regions. The computation can be applied to calculate descriptors or signatures of persons or objects within an image. The computation also has a low computational cost enabling fast calculation of image statistics.
```
@patent{dorettoWSRTpatent,
  title = {Device and method for fast computation of region based image features},
  nationality = {U.S.},
  number = {8,184,915 B2},
  year = {2012},
  author = {Doretto, G. and Wang, X. and Sebastian, T. and Rittscher, J. and Tu, P. H.},
  day = {22},
  month = may,
  url = {https://patents.google.com/patent/US8184915B2/en},
  bib2html_dl_html = {https://patents.google.com/patent/US8184915B2/en},
  bib2html_pubtype = {Patents},
  bib2html_rescat = {Video Analysis},
  owner = {doretto},
  timestamp = {2008.01.17}
}
```
Identifying descriptor for person or object in an image Doretto, G., Wang, X., Sebastian, T., Rittscher, J., and Tu, P. H. U.S. Patent 8,165,397 B2, Issued apr 24, 2012. abstract bibTeX html
A device and method for processing an image to create appearance and shape labeled images of a person or object captured within the image. The appearance and shape labeled images are unique properties of the person or object and can be used to re-identify the person or object in subsequent images. The appearance labeled image is an aggregate of pre-stored appearance labels that are assigned to image segments of the image based on calculated appearance attributes of each image segment. The shape labeled image is an aggregate of pre-stored shape labels that are assigned to image segments of the image based on calculated shape attributes of each image segment. An identifying descriptor of the person or object can be computed based on both the appearance labeled image and the shape labeled image. The descriptor can be compared with other descriptors of later captured images to re-identify a person or object.
```
@patent{dorettoWSRTpatentA,
  title = {Identifying descriptor for person or object in an image},
  nationality = {U.S.},
  number = {8,165,397 B2},
  year = {2012},
  author = {Doretto, G. and Wang, X. and Sebastian, T. and Rittscher, J. and Tu, P. H.},
  day = {24},
  month = apr,
  url = {https://patents.google.com/patent/US8165397B2/en},
  bib2html_dl_html = {https://patents.google.com/patent/US8165397B2/en},
  bib2html_pubtype = {Patents},
  bib2html_rescat = {Video Analysis},
  owner = {doretto},
  timestamp = {2007.01.20}
}
```
Device and method for detecting targets in images based on user-defined classifiers Tu, P. H., Krahnstoever, N., Rittscher, J., and Doretto, G. U.S. Patent 8,295,543 B2, Issued oct 23, 2012. bibTeX html
```
@patent{tuKRDpatent,
  title = {Device and method for detecting targets in images based on user-defined classifiers},
  nationality = {U.S.},
  number = {8,295,543 B2},
  year = {2012},
  author = {Tu, P. H. and Krahnstoever, N. and Rittscher, J. and Doretto, G.},
  day = {23},
  month = oct,
  url = {https://patents.google.com/patent/US8295543B2/en},
  bib2html_dl_html = {https://patents.google.com/patent/US8295543B2/en},
  bib2html_pubtype = {Patents},
  bib2html_rescat = {Video Analysis},
  owner = {doretto},
  timestamp = {2012.12.07}
}
```

Detecting and tracking targets in images based on estimated target geometry Krahnstoever, N., Tu, P H.., Doretto, G., and Perera, A. G. A. U.S. Patent 8,320,613 B2, Issued nov 27, 2012. bibTeX html

@patent{krahnstoeverTDPpatent,
  title = {Detecting and tracking targets in images based on estimated target geometry},
  nationality = {U.S.},
  number = {8,320,613 B2},
  year = {2012},
  author = {Krahnstoever, N. and Tu, P H.. and Doretto, G. and Perera, A. G. A.},
  day = {27},
  month = nov,
  url = {https://patents.google.com/patent/US8320613B2/en},
  bib2html_dl_html = {https://patents.google.com/patent/US8320613B2/en},
  bib2html_pubtype = {Patents},
  bib2html_rescat = {Video Analysis},
  owner = {doretto},
  timestamp = {2008.01.17}
}

JAIHC
Appearance-based person reidentification in camera networks: problem overview and current approaches Doretto, G., Sebastian, T., Tu, P., and Rittscher, J. Journal of Ambient Intelligence and Humanized Computing, 2011. abstract bibTeX pdf html
Recent advances in visual tracking methods allow following a given object or individual in presence of significant clutter or partial occlusions in a single or a set of overlapping camera views. The question of when person detections in different views or at different time instants can be linked to the same individual is of fundamental importance to the video analysis in large-scale network of cameras. This is the person reidentification problem. The paper focuses on algorithms that use the overall appearance of an individual as opposed to passive biometrics such as face and gait. Methods that effectively address the challenges associated with changes in illumination, pose, and clothing appearance variation are discussed. More specifically, the development of a set of models that capture the overall appearance of an individual and can effectively be used for information retrieval are reviewed. Some of them provide a holistic description of a person, and some others require an intermediate step where specific body parts need to be identified. Some are designed to extract appearance features over time, and some others can operate reliably also on single images. The paper discusses algorithms for speeding up the computation of signatures. In particular it describes very fast procedures for computing co-occurrence matrices by leveraging a generalization of the integral representation of images. The algorithms are deployed and tested in a camera network comprising of three cameras with non-overlapping field of views, where a multi-camera multi-target tracker links the tracks in different cameras by reidentifying the same people appearing in different views.
```
@article{dorettoSTR11jaihc,
  abbr = {JAIHC},
  author = {Doretto, G. and Sebastian, T. and Tu, P. and Rittscher, J.},
  title = {Appearance-based person reidentification in camera networks: problem
  	overview and current approaches},
  journal = {Journal of Ambient Intelligence and Humanized Computing},
  year = {2011},
  volume = {2},
  pages = {127-151},
  affiliation = {West Virginia University, P.O. Box 6901, Morgantown, WV 26506, USA},
  bib2html_pubtype = {Journals},
  bib2html_rescat = {Human Reidentification, Identity Management, Video Analysis, Appearance
  	Modeling, Shape and Appearance Modeling, Integral Image Computations,
  	Track Matching},
  file = {dorettoSTR11jaihc.pdf:doretto\\journal\\dorettoSTR11jaihc.pdf:PDF},
  issn = {1868-5137},
  issue = {2},
  keyword = {Engineering},
  owner = {doretto},
  publisher = {Springer Berlin / Heidelberg},
  timestamp = {2010.10.17},
  url = {http://dx.doi.org/10.1007/s12652-010-0034-y}
}
```
ISVC
Multi-class Object Layout with Unsupervised Image Classification and Object Localization Lim, S., Doretto, G., and Rittscher, J. In International Symposium on Visual Computing, 2011. Oral abstract bibTeX pdf
Recognizingthepresenceofobjectclassesinanimage,orimageclas- sification, has become an increasingly important topic of interest. Equally impor- tant, however, is also the capability to locate these object classes in the image. We consider in this paper an approach to these two related problems with the primary goal of minimizing the training requirements so as to allow for ease of adding new object classes, as opposed to approaches that favor training a suite of object-specific classifiers. To this end, we provide the analysis of an exemplar- based approach that leverages unsupervised clustering for classification purpose, and sliding window matching for localization. While such exemplar based ap- proach by itself is brittle towards intraclass and viewpoint variations, we achieve robustness by introducing a novel Conditional Random Field model that facili- tates a straightforward accept/reject decision of the localized object classes. Per- formance of our approach on the PASCAL Visual Object Challenge 2007 dataset demonstrates its efficacy.
```
@inproceedings{limDR11isvc,
  abbr = {ISVC},
  author = {Lim, S. and Doretto, G. and Rittscher, J.},
  title = {Multi-class Object Layout with Unsupervised Image Classification
  	and Object Localization},
  booktitle = {International Symposium on Visual Computing},
  year = {2011},
  pages = {577--589},
  bib2html_pubtype = {Conferences},
  bib2html_rescat = {Object Detection, Context, Object Classification},
  file = {limDR11isvc.pdf:doretto/conference/limDR11isvc.pdf:PDF},
  owner = {doretto},
  timestamp = {2010.12.20},
  wwwnote = {Oral}
}
```
Video Analytics for Force Protection Tu, P.H., Brooksby, G.W., Doretto, G., Hamilton, D.W., Krahnstoever, N., Laflan, J.B., Liu, X., Patwardhan, K.A., Sebastian, T., Tong, Y., Tu, J., Wheeler, F.W., Wynnyk, C. M., Yao, Y., and Yu, T. In Distributed Video Sensor Networks, 2011. abstract bibTeX pdf html
For troop and military installation protection, modern computer vision methods must be harnessed to enable a comprehensive approach to contextual awareness. In this chapter we present a collection of intelligent video technologies currently under development at the General Electric Global Research Center, which can be applied to this challenging problem. These technologies include: aerial analysis for object detection and tracking, site-wide tracking from networks of fixed video cameras, person detection from moving platforms, biometrics at a distance and facial analysis for the purposes of inferring intent. We hypothesize that a robust approach to troop protection will require the synthesis of all of these technologies into a comprehensive system of systems.
```
@incollection{tuBDHKLLPSTTWWYY11chapter,
  author = {Tu, P.H. and Brooksby, G.W. and Doretto, G. and Hamilton, D.W. and Krahnstoever, N. and Laflan, J.B. and Liu, X. and Patwardhan, K.A. and Sebastian, T. and Tong, Y. and Tu, J. and Wheeler, F.W. and Wynnyk, C. M. and Yao, Y. and Yu, T.},
  title = {Video Analytics for Force Protection},
  booktitle = {Distributed Video Sensor Networks},
  publisher = {Springer London},
  year = {2011},
  editor = {Bhanu, B. and Ravishankar, C.V. and Roy-Chowdhury, A.K. and Aghajan, H. and Terzopoulos, D.},
  pages = {405--425},
  bib2html_pubtype = {Book Chapters},
  bib2html_rescat = {Video Analysis},
  file = {tuBDHKLLPSTTWWYY11chapter.pdf:doretto\\chapter\\tuBDHKLLPSTTWWYY11chapter.pdf:PDF},
  owner = {doretto},
  timestamp = {2010.01.10},
  url = {http://dx.doi.org/10.1007/978-0-85729-127-1_27}
}
```

System, Method And Program Product For Camera-Based Object Analysis Doretto, G., and Yi, Y. U.S. Patent 7,953,245 B2, Issued may 31, 2011. bibTeX html

@patent{dorettoYpatent,
  title = {System, Method And Program Product For Camera-Based Object Analysis},
  nationality = {U.S.},
  number = {7,953,245 B2},
  year = {2011},
  author = {Doretto, G. and Yi, Y.},
  day = {31},
  month = may,
  url = {https://patents.google.com/patent/US7953245B2/en},
  bib2html_dl_html = {https://patents.google.com/patent/US7953245B2/en},
  bib2html_pubtype = {Patents},
  bib2html_rescat = {Video Analysis},
  owner = {doretto},
  timestamp = {2011.10.17}
}

Standoff detection systems and methods Tu, P. H., Thomenius, K. E., Krahnstoever, N. O., Rittscher, J., Hill, J. C., Doretto, G., Sebastian, T. B., and Ashe, J. M. U.S. Patent 7,885,429 B2, Issued feb 8, 2011. bibTeX html
```
@patent{tuTKRHDSApatent,
  title = {Standoff detection systems and methods},
  nationality = {U.S.},
  number = {7,885,429 B2},
  year = {2011},
  author = {Tu, P. H. and Thomenius, K. E. and Krahnstoever, N. O. and Rittscher, J. and Hill, J. C. and Doretto, G. and Sebastian, T. B. and Ashe, J. M.},
  day = {8},
  month = feb,
  url = {https://patents.google.com/patent/US7885429B2/en},
  bib2html_dl_html = {https://patents.google.com/patent/US7885429B2/en},
  bib2html_pubtype = {Patents},
  bib2html_rescat = {Video Analysis},
  owner = {doretto},
  timestamp = {2007.08.07}
}
```

CVPR
Region Moments: Fast invariant descriptors for detecting small image structures Doretto, G., and Yao, Y. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2010. abstract bibTeX pdf
This paper presents region moments, a class of appear- ance descriptors based on image moments applied to a pool of image features. A careful design of the moments and the image features, makes the descriptors scale and rotation in- variant, and therefore suitable for vehicle detection from aerial video, where targets appear at different scales and orientations. Region moments are linearly related to the image features. Thus, comparing descriptors by comput- ing costly geodesic distances and non-linear classifiers can be avoided, because Euclidean geometry and linear classi- fiers are still effective. The descriptor computation is made efficient by designing a fast procedure based on the inte- gral representation. An extensive comparison between re- gion moments and the region covariance descriptors, re- ports theoretical, qualitative, and quantitative differences among them, with a clear advantage of the region moments, when used for detecting small image structures, such as ve- hicles in aerial video. The proposed descriptors hold the promise to become an effective building block in other ap- plications.
```
@inproceedings{dorettoY10cvpr,
  abbr = {CVPR},
  author = {Doretto, G. and Yao, Y.},
  title = {Region {M}oments: Fast invariant descriptors for detecting small
  	image structures},
  booktitle = {Proceedings of the IEEE Computer Society Conference on Computer Vision
  	and Pattern Recognition},
  year = {2010},
  bib2html_pubtype = {Conferences},
  bib2html_rescat = {Video Analysis, Appearance Modeling, Integral Image Computations},
  file = {dorettoY10cvpr.pdf:doretto\\conference\\dorettoY10cvpr.pdf:PDF;wangDSRT07iccv.pdf:doretto\\conference\\wangDSRT07iccv.pdf:PDF},
  owner = {doretto},
  timestamp = {2008.01.19}
}
```
ECCVW
Object Constellations: Scalable, simultaneous detection and recognition of multiple specific objects Lim, S., Doretto, G., and Rittscher, J. In Proceedings of the ECCV Workshop on Vision for Cognitive Tasks, 2010. Oral abstract bibTeX pdf
Given a library of objects specified by example images, we describe a probabilistic framework for detecting multiple such objects in the scene, as well as estimating their positions and sizes. Detection of such an object constellation is faced with several major challenges. The approach has to be scalable to a large number of objects in the library while being robust towards outliers and noise. We propose to overcome these challenges by generating object and geometry hypotheses as priors for estimating the constellation. Generating object hypotheses avoids the need for evaluating the presence of all the objects in the image, which allows us to achieve scalability. Generating geometry hypotheses local- izes the region of interests (ROIs) in which detection can be conducted. Object constellations are then estimated through a feature matching procedure, where we make recognition decision based on the number of feature matches for each object hypothesis, as well as the quality of the matches. We demonstrate the efficacy of our approach with both public datasets and TV broadcast image sequences.
```
@inproceedings{limDR10workshop,
  abbr = {ECCVW},
  author = {Lim, S. and Doretto, G. and Rittscher, J.},
  title = {{O}bject {C}onstellations: {S}calable, simultaneous detection and
  	recognition of multiple specific objects},
  booktitle = {Proceedings of the ECCV Workshop on Vision for Cognitive Tasks},
  year = {2010},
  bib2html_pubtype = {Conferences},
  bib2html_rescat = {Object Detection, Context, Object Classification},
  file = {limDR10workshop.pdf:doretto\\conference\\limDR10workshop.pdf:PDF},
  owner = {doretto},
  timestamp = {2010.10.17},
  wwwnote = {Oral}
}
```
CVPR
Boosting for transfer learning with multiple sources Yao, Y., and Doretto, G. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2010. abstract bibTeX pdf
Transfer learning allows leveraging the knowledge of source domains, available a priori, to help training a classi- fier for a target domain, where the available data is scarce. The effectiveness of the transfer is affected by the relation- ship between source and target. Rather than improving the learning, brute force leveraging of a source poorly related to the target may decrease the classifier performance. One strategy to reduce this negative transfer is to import knowl- edge from multiple sources to increase the chance of find- ing one source closely related to the target. This work ex- tends the boosting framework for transferring knowledge from multiple sources. Two new algorithms, MultiSource- TrAdaBoost, and TaskTrAdaBoost, are introduced, analyzed, and applied for object category recognition and specific ob- ject detection. The experiments demonstrate their improved performance by greatly reducing the negative transfer as the number of sources increases. TaskTrAdaBoost is a fast algorithm enabling rapid retraining over new targets.
```
@inproceedings{yaoD10cvpr,
  abbr = {CVPR},
  author = {Yao, Y. and Doretto, G.},
  title = {Boosting for transfer learning with multiple sources},
  booktitle = {Proceedings of the IEEE Computer Society Conference on Computer Vision
  	and Pattern Recognition},
  year = {2010},
  bib2html_pubtype = {Conferences},
  bib2html_rescat = {Object Classification},
  file = {yaoD10cvpr.pdf:doretto\\conference\\yaoD10cvpr.pdf:PDF;wangDSRT07iccv.pdf:doretto\\conference\\wangDSRT07iccv.pdf:PDF},
  owner = {doretto},
  timestamp = {2008.01.19}
}
```
Method and system for performing image re-identification Tu, P. H., Sebastian, T. B., Rittscher, J., Krahnstoever, N. O., Doretto, G., and Liu, X. U.S. Patent 7,711,146 B2, Issued may 4, 2010. abstract bibTeX html
The invention allows for reidentitication of people using video cameras. Based on the overall appearance of a person, we are able to reidentify them based on what they are wearing and their general appearance.
```
@patent{tuSRKDLpatent,
  title = {Method and system for performing image re-identification},
  nationality = {U.S.},
  number = {7,711,146 B2},
  year = {2010},
  author = {Tu, P. H. and Sebastian, T. B. and Rittscher, J. and Krahnstoever, N. O. and Doretto, G. and Liu, X.},
  day = {4},
  month = may,
  url = {https://patents.google.com/patent/US7711146B2/en},
  bib2html_dl_html = {https://patents.google.com/patent/US7711146B2/en},
  bib2html_pubtype = {Patents},
  bib2html_rescat = {Video Analysis},
  owner = {doretto},
  timestamp = {2007.01.20}
}
```

Fast invariant region descriptors based on moments of image features Doretto, G., and Yao, Y. Technical Report 2009GRC355, GE Global Research, 2009. Visualization and Computer Vision Laboratory bibTeX

@techreport{dorettoY09tr,
  author = {Doretto, G. and Yao, Y.},
  title = {Fast invariant region descriptors based on moments of image features},
  institution = {GE Global Research},
  year = {2009},
  number = {2009GRC355},
  address = {Niskayuna, NY, USA},
  month = mar,
  note = {Visualization and Computer Vision Laboratory},
  bib2html_pubtype = {Tech Reports},
  bib2html_rescat = {Video Analysis, Appearance Modeling, Integral Image Computations},
  file = {dorettoY09tr.pdf:doretto\\report\\dorettoY09tr.pdf:PDF},
  owner = {doretto},
  timestamp = {2009.09.28}
}

AVSS
A model change detection approach to dynamic scene modeling Kim, S. J., Doretto, G., Rittscher, J., Tu, P., Krahnstoever, N., and Pollefeys, M. In IEEE International Conference on Advanced Video and Signal Based Surveillance, 2009. Oral abstract bibTeX pdf
In this work we propose a dynamic scene model to provide information about the presence of salient motion in the scene, and that could be used for focusing the attention of a pan/tilt/zoom camera, or for background modeling purposes. Rather than proposing a set of saliency detectors, we define what we mean by salient motion, and propose a precise model for it. Detecting salient motion becomes equivalent to detecting a model change. We derive optimal online procedures to solve this problem, which enable a very fast implementation. Promising results show that our model can effectively detect salient motion even in severely cluttered scenes, and while a camera is panning and tilting.
```
@inproceedings{kimDRTKP09avss,
  abbr = {AVSS},
  author = {Kim, S. J. and Doretto, G. and Rittscher, J. and Tu, P. and Krahnstoever, N. and Pollefeys, M.},
  title = {A model change detection approach to dynamic scene modeling},
  booktitle = {IEEE International Conference on Advanced Video and Signal Based
  	Surveillance},
  year = {2009},
  pages = {490--495},
  month = sep,
  bib2html_pubtype = {Conferences},
  bib2html_rescat = {Video Analysis, Dynamic Textures, Visual Motion Detection},
  file = {kimDRTKP09avss.pdf:doretto\\conference\\kimDRTKP09avss.pdf:PDF},
  keywords = {dynamic scene modeling, video analysis, PTZ camera, focus-ofattention, background
  	modeling, sequential generalized likelihood ratio, model change detection,
  	linear dynamical systems, dynamic textures},
  owner = {doretto},
  timestamp = {2009.09.28},
  wwwnote = {Oral}
}
```
AVSS
Intelligent video for protecting crowded sports venues Krahnstoever, N., Tu, P., Yu, T., Patwardhan, K., Hamilton, D., Yu, B., Greco, C., and Doretto, G. In IEEE International Conference on Advanced Video and Signal Based Surveillance, 2009. (Invited paper) Oral bibTeX pdf
```
@inproceedings{krahnstoeverTYPHYGD09avss,
  abbr = {AVSS},
  author = {Krahnstoever, N. and Tu, P. and Yu, T. and Patwardhan, K. and Hamilton, D. and Yu, B. and Greco, C. and Doretto, G.},
  title = {Intelligent video for protecting crowded sports venues},
  booktitle = {IEEE International Conference on Advanced Video and Signal Based
  	Surveillance},
  year = {2009},
  pages = {116--121},
  month = sep,
  note = {(Invited paper)},
  bib2html_pubtype = {Conferences},
  bib2html_rescat = {Video Analysis},
  file = {krahnstoeverTYPHYGD09avss.pdf:doretto\\conference\\krahnstoeverTYPHYGD09avss.pdf:PDF},
  owner = {doretto},
  timestamp = {2009.09.28},
  wwwnote = {Oral}
}
```

Boosting for transfer learning with multiple sources Yao, Y., and Doretto, G. Technical Report 2009GRC1080, GE Global research, 2009. Visualization and Computer Vision Laboratory bibTeX

@techreport{yaoD09tr,
  author = {Yao, Y. and Doretto, G.},
  title = {Boosting for transfer learning with multiple sources},
  institution = {GE Global research},
  year = {2009},
  number = {2009GRC1080},
  address = {Niskayuna, NY, USA},
  month = dec,
  note = {Visualization and Computer Vision Laboratory},
  bib2html_pubtype = {Tech Reports},
  bib2html_rescat = {Object Classification},
  file = {yaoD09tr.pdf:doretto\\report\\yaoD09tr.pdf:PDF;wuLD08tr.pdf:doretto\\report\\wuLD08tr.pdf:PDF},
  owner = {doretto},
  timestamp = {2006.11.29}
}

Adaptive Dynamic Background Modeling Doretto, G. In Abstract Collection – Statistical and Geometrical Approaches to Visual Motion Analysis, 2008. (Abstract) Oral bibTeX html

@conference{doretto08dagstuhl,
  author = {Doretto, G.},
  title = {Adaptive Dynamic Background Modeling},
  booktitle = {Abstract Collection -- Statistical and Geometrical Approaches to
  	Visual Motion Analysis},
  year = {2008},
  editor = {Cremers, D. and Rosenhahn, B. and Yuille, A. L.},
  number = {08291},
  series = {Dagstuhl Seminar Proceedings},
  address = {Dagstuhl, Germany},
  publisher = {Schloss Dagstuhl - Leibniz-Zentrum fuer Informatik, Germany},
  note = {(Abstract)},
  bib2html_pubtype = {Other},
  bib2html_rescat = {Dynamic Textures, Visual Motion Analysis, Video Analysis},
  file = {doretto08dagstuhl.pdf:doretto\\abstract\\doretto08dagstuhl.pdf:PDF},
  issn = {1862-4405},
  keywords = {Motion Segmentation, Statistical Methods, Computational Geometry},
  owner = {doretto},
  timestamp = {2008.10.23},
  url = {http://drops.dagstuhl.de/opus/volltexte/2008/1620},
  wwwnote = {Oral}
}

From dynamic texture to dynamic shape and appearance models: An overview Doretto, G., and Soatto, S. In Handbook of texture analysis, 2008. abstract bibTeX pdf
In modeling complex visual phenomena one can employ rich models that characterize the global statistics of images, or choose simple classes of models to represent the local statistics of a spatiotemporal �segment,� together with the partition of the data into such segments. Each segment could be characterized by certain statistical regularity properties in space and/or time. The former approach is often pursued in Computer Graphics, where a global model is necessary to capture effects such as mutual illumination or cast shadows. However, such models cannot be uniquely inferred as they are far more complex than the data, and one has to revert to a much simpler representation that, for instance, models the visual complexity of single segments in terms of statistical variability from a nominal model. In this chapter we do so by modeling the image variability of dynamic scenes through the joint temporal variation of shape and appearance. We describe how this framework can be specialized to Dynamic Texture models for both static and moving cameras. The characterization poses the problems of modeling, learning, and synthesis of video sequences that exhibit certain temporal regularity properties (such as sea-waves, smoke, foliage, talking faces, flags in wind, etc.), using tools from time series analysis, system identification theory, and finite element methods.
```
@incollection{dorettoS08chapter,
  author = {Doretto, G. and Soatto, S.},
  title = {From dynamic texture to dynamic shape and appearance models: {A}n
  	overview},
  booktitle = {Handbook of texture analysis},
  publisher = {Imperial {C}ollege {P}ress},
  year = {2008},
  editor = {Mirmehdi, M. and Xie, X. and Suri, J.},
  chapter = {9},
  bib2html_pubtype = {Book Chapters},
  bib2html_rescat = {Dynamic Textures, Visual Motion Analysis, Shape and Appearance Modeling},
  file = {dorettoS08chapter.pdf:doretto\\chapter\\dorettoS08chapter.pdf:PDF;dorettoS07chapter.pdf:doretto\\chapter\\dorettoS07chapter.pdf:PDF},
  owner = {doretto},
  timestamp = {2007.01.19}
}
```
ECCV
Unified crowd segmentation Tu, P., Sebastian, T., Doretto, G., Krahnstoever, N., Rittscher, J., and Yu, T. In Proceedings of European Conference on Computer Vision, 2008. abstract bibTeX pdf
This paper presents a unified approach to crowd segmentation. A global solution is generated using an Expectation Maximization framework. Initially, a head and shoulder detector is used to nominate an exhaustive set of person locations and these form the person hypotheses. The image is then partitioned into a grid of small patches which are each assigned to one of the person hypotheses. A key idea of this paper is that while whole body monolithic person detectors can fail due to occlusion, a partial response to such a detector can be used to evaluate the likelihood of a single patch being assigned to a hypothesis. This captures local appearance information without having to learn specific appearance models. The likelihood of a pair of patches being assigned to a person hypothesis is evaluated based on low level image features such as uniform motion fields and color constancy. During the E-step, the single and pairwise likelihoods are used to compute a globally optimal set of assignments of patches to hypotheses. In the M-step, parameters which enforce global consistency of assignments are estimated. This can be viewed as a form of occlusion reasoning. The final assignment of patches to hypotheses constitutes a segmentation of the crowd. The resulting system provides a global solution that does not require background modeling and is robust with respect to clutter and partial occlusion.
```
@inproceedings{tuSDKRY08eccv,
  abbr = {ECCV},
  author = {Tu, P. and Sebastian, T. and Doretto, G. and Krahnstoever, N. and Rittscher, J. and Yu, T.},
  title = {Unified crowd segmentation},
  booktitle = {Proceedings of European Conference on Computer Vision},
  year = {2008},
  pages = {691--704},
  bib2html_pubtype = {Conferences},
  bib2html_rescat = {Video Analysis, People Detection, Integral Image Computations,
  	People Tracking},
  file = {tuSDKRY08eccv.pdf:doretto\\conference\\tuSDKRY08eccv.pdf:PDF},
  owner = {doretto},
  timestamp = {2008.01.16}
}
```
Unified crowd segementation Tu, P., Yu, T., Rittscher, J., Doretto, G., Krahnstoever, N., and Sebastian, T. Technical Report 2008GRC253, GE Global research, 2008. Visualization and Computer Vision Laboratory bibTeX
```
@techreport{tuYRDKS08tr,
  author = {Tu, P. and Yu, T. and Rittscher, J. and Doretto, G. and Krahnstoever, N. and Sebastian, T.},
  title = {Unified crowd segementation},
  institution = {GE Global research},
  year = {2008},
  number = {2008GRC253},
  address = {Niskayuna, NY, USA},
  month = mar,
  note = {Visualization and Computer Vision Laboratory},
  bib2html_pubtype = {Tech Reports},
  bib2html_rescat = {Video Analysis, People Detection, People Tracking, Appearance
  	Modeling, Shape and Appearance Modeling, Integral Image Computations},
  file = {tuYRDKS08tr.pdf:doretto\\report\\tuYRDKS08tr.pdf:PDF;tuYRDKS08tr.pdf:doretto\\report\\tuYRDKS08tr.pdf:PDF},
  owner = {doretto},
  timestamp = {2008.05.19}
}
```
CVPR
Face alignment using boosted ranking models Wu, H., Liu, X., and Doretto, G. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2008. Oral abstract bibTeX pdf
Face alignment seeks to deform a face model to match it with the features of the image of a face by optimizing an appropriate cost function. We propose a new face model that is aligned by maximizing a score function, which we learn from training data, and that we impose to be concave. We show that this problem can be reduced to learning a classifier that is able to say whether or not by switching from one alignment to a new one, the model is approaching the correct fitting. This relates to the ranking problem where a number of instances need to be ordered. For training the model, we propose to extend GentleBoost [23] to ranklearning. Extensive experimentation shows the superiority of this approach to other learning paradigms, and demonstrates that this model exceeds the alignment performance of the state-of-the-art.
```
@inproceedings{wuLD08cvpr,
  abbr = {CVPR},
  author = {Wu, H. and Liu, X. and Doretto, G.},
  title = {Face alignment using boosted ranking models},
  booktitle = {Proceedings of the IEEE Computer Society Conference on Computer Vision
  	and Pattern Recognition},
  year = {2008},
  pages = {1--8},
  bib2html_pubtype = {Conferences},
  bib2html_rescat = {Video Analysis, Appearance Modeling, Shape and Appearance Modeling,
  	Integral Image Computations, Face Tracking, Face Modeling},
  file = {wuLD08cvpr.pdf:doretto\\conference\\wuLD08cvpr.pdf:PDF;wuLD08cvpr.pdf:doretto\\conference\\wuLD08cvpr.pdf:PDF},
  owner = {doretto},
  timestamp = {2008.01.16},
  wwwnote = {Oral}
}
```
Face alignment via boosted ranking models Wu, H., Liu, X., and Doretto, G. Technical Report 2008GRC239, GE Global research, 2008. Visualization and Computer Vision Laboratory abstract bibTeX
Face alignment seeks to deform a face model to match it with the features of the image of a face by optimizing an appropriate cost function. We propose a new face model that is aligned by maximizing a score function, which we learn from training data, and that we impose to be concave. We show that this problem can be reduced to learning a classifier that is able to say whether or not by switching from one alignment to a new one, the model is approaching the correct fitting. This relates to the ranking problem where a number of instances need to be ordered. For training the model, we propose to extend GentleBoost [22] to ranklearning. Extensive experimentation shows the superiority of this approach to other learning paradigms, and demonstrates that this model exceeds the alignment performance of the state-of-the-art.
```
@techreport{wuLD08tr,
  author = {Wu, H. and Liu, X. and Doretto, G.},
  title = {Face alignment via boosted ranking models},
  institution = {GE Global research},
  year = {2008},
  number = {2008GRC239},
  address = {Niskayuna, NY, USA},
  month = apr,
  note = {Visualization and Computer Vision Laboratory},
  bib2html_pubtype = {Tech Reports},
  bib2html_rescat = {Video Analysis, Appearance Modeling, Shape and Appearance Modeling,
  	Integral Image Computations, Face Tracking, Face Modeling},
  file = {wuLD08tr.pdf:doretto\\report\\wuLD08tr.pdf:PDF;wuLD08tr.pdf:doretto\\report\\wuLD08tr.pdf:PDF},
  owner = {doretto},
  timestamp = {2006.11.29}
}
```

Integral computations: A framework to compute fast region based features Doretto, G., and Wang, X. Technical Report 2007GRC593, GE Global Research, 2007. Visualization and Computer Vision Laboratory abstract bibTeX
The integral image and the integral histogram are very popular image representations in Computer Vision. After a pre-computation step (an O(N^2) operation for an NxN image), they allow the computation of image statistics such as mean, covariance, and histogram of the image pixels of given rectangular image regions in constant time, regardless of the dimension of such regions. In this report, while focusing on the case of the integral image, we show that it is possible to establish very simple rules that allow extending the computational benefits of the integral representations to image regions of any shape. Based on this framework, it is possible to derive a very efficient algorithm for computing the co-occurrence of image values, or labels, lowering the computational complexity of such an operation to O(N^2). Several aspects of the implementation of the proposed algorithms are also reported.
```
@techreport{dorettoW07tr,
  author = {Doretto, G. and Wang, X.},
  title = {Integral computations: {A} framework to compute fast region based
  	features},
  institution = {GE Global Research},
  year = {2007},
  number = {2007GRC593},
  address = {Niskayuna, NY, USA},
  month = jun,
  note = {Visualization and Computer Vision Laboratory},
  bib2html_pubtype = {Tech Reports},
  bib2html_rescat = {Video Analysis, Integral Image Computations},
  file = {dorettoW07tr.pdf:doretto\\report\\dorettoW07tr.pdf:PDF;dorettoW07tr.pdf:doretto\\report\\dorettoW07tr.pdf:PDF},
  keywords = {integral image, integral histogram, integral computation, co-occurrence
  	matrix, correlogram},
  owner = {doretto},
  timestamp = {2007.04.30}
}
```
Shape and appearance context modeling: A fast framework for matching the appearance of people Doretto, G., Wang, X., Sebastian, T. B., Rittscher, J., and Tu, P. Technical Report 2007GRC594, GE Global Research, 2007. Visualization and Computer Vision Laboratory abstract bibTeX
In this work we develop appearance models for computing the similarity between image regions containing deformable objects of a given class in realtime. We introduce the concept of shape and appearance context. The main idea is to model the spatial distribution of the appearance relative to each of the object parts. Estimating the model entails computing occurrence matrices. We introduce a generalization of the integral image and integral histogram frameworks, and prove that it can be used to dramatically speed up occurrence computation. We demonstrate the ability of this framework to recognize an individual walking across a network of cameras. Finally, we show that the proposed approach outperforms several other methods.
```
@techreport{dorettoWSRT07tr,
  author = {Doretto, G. and Wang, X. and Sebastian, T. B. and Rittscher, J. and Tu, P.},
  title = {Shape and appearance context modeling: {A} fast framework for matching
  	the appearance of people},
  institution = {GE Global Research},
  year = {2007},
  number = {2007GRC594},
  address = {Niskayuna, NY, USA},
  month = jun,
  note = {Visualization and Computer Vision Laboratory},
  bib2html_pubtype = {Tech Reports},
  bib2html_rescat = {Video Analysis, Integral Image Computations, Appearance Modeling,
  	Shape and Appearance Modeling, Track Matching},
  file = {dorettoWSRT07tr.pdf:doretto\\report\\dorettoWSRT07tr.pdf:PDF;dorettoWSRT07tr.pdf:doretto\\report\\dorettoWSRT07tr.pdf:PDF},
  keywords = {integral image, integral histogram, integral computation, co-occurrence
  	matrix, correlogram, appearance modeling, bag-of-features, histogram
  	of oriented gradients, occurrence, appearance context, shape context,
  	shape and appearance context, person reacquisition, person reidentification,
  	track linking},
  owner = {doretto},
  timestamp = {2007.01.20}
}
```
An intelligent video framework for homeland protection Tu, P., Doretto, G., Krahnstoever, N., Perera, A. G. A., Wheeler, F., Liu, X., Rittscher, J., Sebastian, T., Yu, T., and Harding, K. Technical Report 2007GRC326, GE Global Research, 2007. Visualization and Computer Vision Laboratory abstract bibTeX
This paper presents an overview of Intelligent Video work currently under development at the GE Global Research Center and other research institutes. The image formation process is discussed in terms of illumination, methods for automatic camera calibration and lessons learned from machine vision. A variety of approaches for person detection are presented. Crowd segmentation methods enabling the tracking of individuals through dense environments such as retail and mass transit sites are discussed. It is shown how signature generation based on gross appearance can be used to reacquire targets as they leave and enter disjoint fields of view. Camera calibration information is used to further constrain the detection of people and to synthesize a top-view, which fuses all camera views into a composite representation. It is shown how site-wide tracking can be performed in this unified framework. Human faces are an important feature as both a biometric identifier and as a method for determining the focus of attention via head pose estimation. It is shown how automatic pantiltzoom control; active shape/appearance models and super-resolution methods can be used to enhance the face capture and analysis problem. A discussion of additional features that can be used for inferring intent is given. These include body-part motion cues and physiological phenomena such as thermal images of the face.
```
@techreport{tuDKPWLRSYH07tr,
  author = {Tu, P. and Doretto, G. and Krahnstoever, N. and Perera, A. G. A. and Wheeler, F. and Liu, X. and Rittscher, J. and Sebastian, T. and Yu, T. and Harding, K.},
  title = {An intelligent video framework for homeland protection},
  institution = {GE Global Research},
  year = {2007},
  number = {2007GRC326},
  address = {Niskayuna, NY, USA},
  month = apr,
  note = {Visualization and Computer Vision Laboratory},
  bib2html_pubtype = {Tech Reports},
  bib2html_rescat = {Video Analysis},
  file = {tuDKPWLRSYH07tr.pdf:doretto\\report\\tuDKPWLRSYH07tr.pdf:PDF;tuDKPWLRSYH07tr.pdf:doretto\\report\\tuDKPWLRSYH07tr.pdf:PDF},
  keywords = {intelligent video; surveillance; camera calibration; person detection;
  	crowd segmentation; site-wide tracking; person reidentification;
  	face modeling; face super-resolution; deception detection},
  owner = {doretto},
  timestamp = {2007.04.30}
}
```
Surveillance video analytics for large camera networks Tu, P., Wheeler, F., Krahnstoever, N., Sebastian, T., Rittscher, J., Liu, X., Perera, A., and Doretto, G. In SPIE Newsroom, 2007. bibTeX pdf html doi
```
@misc{tuWKSRLPD07spie,
  author = {Tu, P. and Wheeler, F. and Krahnstoever, N. and Sebastian, T. and Rittscher, J. and Liu, X. and Perera, A. and Doretto, G.},
  title = {Surveillance video analytics for large camera networks},
  howpublished = {SPIE Newsroom},
  year = {2007},
  bib2html_pubtype = {Other},
  bib2html_rescat = {Video Analysis},
  doi = {10.1117/2.1200706.0712},
  file = {tuWKSRLPD07spie.pdf:doretto\\press\\tuWKSRLPD07spie.pdf:PDF;tuWKSRLPD07spie.pdf:doretto\\press\\tuWKSRLPD07spie.pdf:PDF},
  owner = {doretto},
  timestamp = {2007.07.17},
  url = {https://doi.org/10.1117/2.1200706.0712}
}
```
SPIE DSS
An intelligent video framework for homeland protection Tu, P. H., Doretto, G., Krahnstoever, N. O., Perera, A. A. G., Wheeler, F. W., Liu, X., Rittscher, J., Sebastian, T. B., Yu, T., and Harding, K. G. In Proceedings of SPIE Defence and Security Symposium - Unattended Ground, Sea, and Air Sensor Technologies and Applications IX, 2007. (Invited paper) abstract bibTeX pdf
This paper presents an overview of Intelligent Video work currently under development at the GE Global Research Center and other research institutes. The image formation process is discussed in terms of illumination, methods for automatic camera calibration and lessons learned from machine vision. A variety of approaches for person detection are presented. Crowd segmentation methods enabling the tracking of individuals through dense environments such as retail and mass transit sites are discussed. It is shown how signature generation based on gross appearance can be used to reacquire targets as they leave and enter disjoint fields of view. Camera calibration information is used to further constrain the detection of people and to synthesize a top-view, which fuses all camera views into a composite representation. It is shown how site-wide tracking can be performed in this unified framework. Human faces are an important feature as both a biometric identifier and as a method for determining the focus of attention via head pose estimation. It is shown how automatic pantilt-zoom control; active shape/appearance models and super-resolution methods can be used to enhance the face capture and analysis problem. A discussion of additional features that can be used for inferring intent is given. These include body-part motion cues and physiological phenomena such as thermal images of the face.
```
@conference{tuDKPWLRSYH07SPIEdss,
  abbr = {SPIE DSS},
  author = {Tu, P. H. and Doretto, G. and Krahnstoever, N. O. and Perera, A. A. G. and Wheeler, F. W. and Liu, X. and Rittscher, J. and Sebastian, T. B. and Yu, T. and Harding, K. G.},
  title = {An intelligent video framework for homeland protection},
  booktitle = {Proceedings of SPIE Defence and Security Symposium - Unattended Ground,
  	Sea, and Air Sensor Technologies and Applications IX},
  year = {2007},
  editor = {Carapezza, E. M.},
  volume = {6562},
  address = {Orlando, FL, USA},
  month = apr,
  note = {(Invited paper)},
  bib2html_pubtype = {Conferences},
  bib2html_rescat = {Video Analysis},
  file = {tuDKPWLRSYH07SPIEdss.pdf:doretto\\conference\\tuDKPWLRSYH07SPIEdss.pdf:PDF;tuDKPWLRSYH07SPIEdss.pdf:doretto\\conference\\tuDKPWLRSYH07SPIEdss.pdf:PDF},
  keywords = {intelligent video, surveillance, camera calibration, person detection,
  	crowd segmentation, site-wide tracking, person reidentification,
  	face modeling, face super-resolution, deception detection},
  owner = {doretto},
  timestamp = {2007.01.20}
}
```
ICCV
Shape and appearance context modeling Wang, X., Doretto, G., Sebastian, T. B., Rittscher, J., and Tu, P. H. In Proceedings of IEEE International Conference on Computer Vision, 2007. abstract bibTeX pdf
In this work we develop appearance models for computing the similarity between image regions containing deformable objects of a given class in realtime. We introduce the concept of shape and appearance context. The main idea is to model the spatial distribution of the appearance relative to each of the object parts. Estimating the model entails computing occurrence matrices. We introduce a generalization of the integral image and integral histogram frameworks, and prove that it can be used to dramatically speed up occurrence computation. We demonstrate the abiity of this framework to recognize an individual walking across a network of cameras. Finally, we show that the proposed approach outperforms several other methods.
```
@inproceedings{wangDSRT07iccv,
  abbr = {ICCV},
  author = {Wang, X. and Doretto, G. and Sebastian, T. B. and Rittscher, J. and Tu, P. H.},
  title = {Shape and appearance context modeling},
  booktitle = {Proceedings of IEEE International Conference on Computer Vision},
  year = {2007},
  pages = {1--8},
  bib2html_pubtype = {Conferences},
  bib2html_rescat = {Human Reidentification, Video Analysis, Appearance Modeling, Shape
  	and Appearance Modeling, Integral Image Computations, Track Matching},
  file = {wangDSRT07iccv.pdf:doretto\\conference\\wangDSRT07iccv.pdf:PDF;wangDSRT07iccv.pdf:doretto\\conference\\wangDSRT07iccv.pdf:PDF},
  owner = {doretto},
  timestamp = {2007.01.19}
}
```

Event recognition with fragmented object tracks Chan, M., Hoogs, A., Sun, Z., Schmiederer, J., Bhotika, R., and Doretto, G. Technical Report 2006GRC038, GE Global research, 2006. Visualization and Computer Vision Laboratory abstract bibTeX
Complete and accurate video tracking is very difficult to achieve in practice due to occlusions, traffic, shadows and appearance changes. In this paper, we study the feasibility of event recognition when object tracks are fragmented at various levels. By changing the lock score threshold controlling track termination, different levels of track fragmentation are generated. In addition, the data contains lengthy occlusions of objects involved in the events. The effect on event recognition is revealed by examining the event model match score as a function of lock score threshold. Using a Dynamic Bayes network to model events, it is shown that event recognition improves with greater track fragmentation, assuming fragmented tracks for the same object are linked together. This is counter-intuitive, as it implies that event recognition works better as tracking degrades. The experiments also show that the model is capable of handling long tracking gaps. The study is conducted on busy scenes of airplane servicing activities, with occlusions lasting hundreds of frames, object appearance changes and significant clutter traffic.
```
@techreport{chanHSSBD06tr,
  author = {Chan, M. and Hoogs, A. and Sun, Z. and Schmiederer, J. and Bhotika, R. and Doretto, G.},
  title = {Event recognition with fragmented object tracks},
  institution = {GE Global research},
  year = {2006},
  number = {2006GRC038},
  address = {Niskayuna, NY, USA},
  month = jan,
  note = {Visualization and Computer Vision Laboratory},
  bib2html_pubtype = {Tech Reports},
  bib2html_rescat = {Video Analysis, Event Recognition},
  file = {chanHSSBD06tr.pdf:doretto\\report\\chanHSSBD06tr.pdf:PDF;chanHSSBD06tr.pdf:doretto\\report\\chanHSSBD06tr.pdf:PDF},
  keywords = {Video Analysis, event recognition, semantics, dynamic bayesian
  	network, video analysis},
  owner = {doretto},
  timestamp = {2006.11.29}
}
```
CVPR
Joint recognition of complex events and track matching Chan, M. T., Hoogs, A., Bhotika, R., Perera, A., Schmiederer, J., and Doretto, G. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2006. abstract bibTeX pdf doi
We present a novel method for jointly performing recognition of complex events and linking fragmented tracks into coherent, long-duration tracks. Many event recognition methods require highly accurate tracking, and may fail when tracks corresponding to event actors are fragmented or partially missing. However, these conditions occur frequently from occlusions, traffic and tracking errors. Recently, methods have been proposed for linking track fragments from multiple objects under these difficult conditions. Here, we develop a method for solving these two problems jointly. A hypothesized event model, represented as a Dynamic Bayes Net, supplies data-driven constraints on the likelihood of proposed track fragment matches. These event-guided constraints are combined with appearance and kinematic constraints used in the previous track linking formulation. The result is the most likely track linking solution given the event model, and the highest event score given all of the track fragments. The event model with the highest score is determined to have occurred, if the score exceeds a threshold. Results demonstrated on a busy scene of airplane servicing activities, where many non-event movers and long fragmented tracks are present, show the promise of the approach to solving the joint problem.
```
@inproceedings{chanHBPSD06cvpr,
  abbr = {CVPR},
  author = {Chan, M. T. and Hoogs, A. and Bhotika, R. and Perera, A. and Schmiederer, J. and Doretto, G.},
  title = {Joint recognition of complex events and track matching},
  booktitle = {Proceedings of the IEEE Computer Society Conference on Computer Vision
  	and Pattern Recognition},
  year = {2006},
  volume = {2},
  pages = {1615--1622},
  address = {New York City, NY, USA},
  month = jun,
  bib2html_pubtype = {Conferences},
  bib2html_rescat = {Video Analysis, Event Recognition, Track Matching},
  doi = {10.1109/CVPR.2006.160},
  file = {chanHBPSD06cvpr.pdf:doretto\\conference\\chanHBPSD06cvpr.pdf:PDF;chanHBPSD06cvpr.pdf:doretto\\conference\\chanHBPSD06cvpr.pdf:PDF},
  issn = {1063-6919},
  owner = {doretto},
  timestamp = {2006.11.29}
}
```
ICPR
Event recognition with fragmented object tracks Chan, M. T., Hoogs, A., Sun, Z., Schmiederer, J., Bhotika, R., and Doretto, G. In Proceedings of the International Conference on Pattern Recognition, 2006. abstract bibTeX pdf doi
Complete and accurate video tracking is very difficult to achieve in practice due to long occlusions, traffic clutter, shadows and appearance changes. In this paper, we study the feasibility of event recognition when object tracks are fragmented. By changing the lock score threshold controlling track termination, different levels of track fragmentation are generated. The effect on event recognition is revealed by examining the event model match score as a function of lock score threshold. Using a Dynamic Bayesian Network to model events, it is shown that event recognition actually improves with greater track fragmentation, assuming fragmented tracks for the same object are linked together. The improvement continues up to a point when it is more likely to be offset by other errors such as those caused by frequent object reinitialization. The study is conducted on busy scenes of airplane servicing activities where long tracking gaps occur intermittently.
```
@inproceedings{chanHSSBD06icpr,
  abbr = {ICPR},
  author = {Chan, M. T. and Hoogs, A. and Sun, Z. and Schmiederer, J. and Bhotika, R. and Doretto, G.},
  title = {Event recognition with fragmented object tracks},
  booktitle = {Proceedings of the International Conference on Pattern Recognition},
  year = {2006},
  volume = {1},
  pages = {412--416},
  month = aug,
  bib2html_pubtype = {Conferences},
  bib2html_rescat = {Video Analysis, Event Recognition},
  doi = {10.1109/ICPR.2006.513},
  file = {chanHSSBD06icpr.pdf:doretto\\conference\\chanHSSBD06icpr.pdf:PDF;chanHSSBD06icpr.pdf:doretto\\conference\\chanHSSBD06icpr.pdf:PDF},
  issn = {1051-4651},
  owner = {doretto},
  timestamp = {2006.11.29}
}
```
Modeling dynamic scenes: An overview of dynamic textures Doretto, G., and Soatto, S. In Handbook of mathematical models in computer vision, 2006. abstract bibTeX pdf
Dynamic scenes with arbitrary radiometry and geometry present a challenge in that a physical model of their motion, shape, and reflectance cannot be inferred. Therefore, the issue of representation becomes crucial, and while there is no right or wrong representation, the task at hand should guide the modeling process.For instance, if the task is three-dimensional reconstruction, one can make assumptions on reflectance and illumination in order to recover shape and motion. If the task is synthesis, or reprojection, the correct shape is unimportant, as long as the model supports the generation of a valid view of the scene. If the task is detection or recognition, a physical model is not necessary as long as one can infer a statistical model that can be used to perform classification. We concentrate our attention on the two latter cases, and describe a modeling framework for dynamic scenes for the purpose of synthesis, detection and recognition. In particular, we restrict our attention to sequences of images of moving scenes that exhibit certain statistical stationarity properties, which have been called Dynamic Textures. They include sea-waves, smoke, foliage, whirlwind etc. In this chapter we describe a characterization of dynamic textures and pose the problems of modeling, learning, recognition and segmentation of dynamic textures using tools from time series analysis, and system identification theory.
```
@incollection{dorettoS06chapter,
  author = {Doretto, G. and Soatto, S.},
  title = {Modeling dynamic scenes: {A}n overview of dynamic textures},
  booktitle = {Handbook of mathematical models in computer vision},
  publisher = {Springer-Verlag},
  year = {2006},
  editor = {Paragios, N. and Chen, Y. and Faugeras, O.},
  chapter = {21},
  pages = {343--357},
  bib2html_pubtype = {Book Chapters},
  bib2html_rescat = {Dynamic Textures, Visual Motion Analysis, Shape and Appearance Modeling},
  file = {dorettoS06chapter.pdf:doretto\\chapter\\dorettoS06chapter.pdf:PDF;dorettoS05chapter.pdf:doretto\\chapter\\dorettoS05chapter.pdf:PDF},
  owner = {doretto},
  timestamp = {2007.01.19}
}
```

TPAMI

Dynamic shape and appearance models Doretto, G., and Soatto, S. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006. abstract bibTeX pdf doi

We propose a model of the joint variation of shape and appearance of portions of an image sequence. The model is conditionally linear, and can be thought of as an extension of active appearance models to exploit the temporal correlation of adjacent image frames. Inference of the model parameters can be performed efficiently using established numerical optimization techniques borrowed from finite-element analysis and system identification techniques.

@article{dorettoS06IEEEtpami,
  abbr = {TPAMI},
  author = {Doretto, G. and Soatto, S.},
  title = {Dynamic shape and appearance models},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2006},
  volume = {28},
  pages = {2006--2019},
  number = {12},
  address = {Los Alamitos, CA, USA},
  bib2html_pubtype = {Journals},
  bib2html_rescat = {Dynamic Textures, Visual Motion Analysis, Shape and Appearance Modeling,
  	Image Based Rendering, Registration from Dynamic Textures},
  doi = {http://doi.ieeecomputersociety.org/10.1109/TPAMI.2006.243},
  file = {dorettoS06IEEEtpami.pdf:doretto\\journal\\dorettoS06IEEEtpami.pdf:PDF;dorettoS06IEEEtpami.pdf:doretto\\journal\\dorettoS06IEEEtpami.pdf:PDF},
  issn = {0162-8828},
  owner = {doretto},
  publisher = {IEEE Computer Society},
  timestamp = {2006.11.24}
}

Intelligent video based detection and tracking of suicide bombers at standoff distances Krahnstoever, N., Sebastian, T., Doretto, G., Tu, P., and Mahony, M. In TSWG Explosives Detection Conference, 2006. (Abstract and poster) abstract bibTeX
Millimeter wave (MMW) imagers can be utilized very effectively for concealed weapons detection at standoff distances. However, from an operational perspective, the MMW imager only partially solves the problem of suicide bomber detection; significant operator interaction is currently required to track subjects, aim the imager and analyze images. Hence, we present a GE-developed auxiliary video-based system that introduces advanced targeting, detection and tracking capabilities that greatly improves the effectiveness of MMW threat detection systems. Our system has the following capabilities: - Since the MMW imager has to be accurately aimed at targets due to their narrow field of view, a video-based person detection system is used to automatically aim the MMW imager. - Person tracking algorithms are used to track individuals before and after threat detection, even in busy and crowded environments. - The captured MMW images are automatically analyzed with weapons detection algorithms. All of these video-based processes occur without human operator interaction. The talk will present a GE prototype system that is currently in operation at a GE test facility.
```
@conference{krahnstoeverSDTM06TSWGabstract,
  author = {Krahnstoever, N. and Sebastian, T. and Doretto, G. and Tu, P. and Mahony, M.},
  title = {Intelligent video based detection and tracking of suicide bombers
  	at standoff distances},
  booktitle = {TSWG Explosives Detection Conference},
  year = {2006},
  address = {Miami, FL, USA},
  month = jun,
  organization = {Combating Terrorism Technology Support Office, Technical Support
  	Working Group},
  note = {(Abstract and poster)},
  bib2html_pubtype = {Other},
  bib2html_rescat = {Video Analysis},
  file = {krahnstoeverSDTM06TSWGabstract.pdf:doretto\\poster\\krahnstoeverSDTM06TSWGabstract.pdf:PDF;krahnstoeverSDTM06TSWGabstract.pdf:doretto\\poster\\krahnstoeverSDTM06TSWGabstract.pdf:PDF},
  owner = {doretto},
  timestamp = {2006.11.29}
}
```
CVPRW
Moving object segmentation using scene understanding Perera, A. G. A., Brooksby, G., Hoogs, A., and Doretto, G. In Proceedings of IEEE Computer Society Workshop on Perceptual Organization in Computer Vision, 2006. abstract bibTeX pdf doi
We present a novel approach to moving object detection in video taken from a translating, rotating and zooming sensor, with a focus on detecting very small objects in as few frames as possible. The primary innovation is to incorporate automatically computed scene understanding of the video directly into the motion segmentation process. Scene understanding provides spatial and semantic context that is used to improve frame-to-frame homography computation, as well as direct reduction of false alarms. The method can be applied to virtually any motion segmentation algorithm, and we explore its utility for three: frame differencing, tensor voting, and generalized PCA. The approach is especially effective on sequences with large scene depth and much parallax, as often occurs when the sensor is close to the scene. In one difficult sequence, our results show an 8-fold reduction of false positives on average, with essentially no impact on the true positive rate. We also show how scene understanding can be used to increase the accuracy of frame-to-frame homography estimates.
```
@inproceedings{pereraBHD06pocv,
  abbr = {CVPRW},
  author = {Perera, A. G. A. and Brooksby, G. and Hoogs, A. and Doretto, G.},
  title = {Moving object segmentation using scene understanding},
  booktitle = {Proceedings of IEEE Computer Society Workshop on Perceptual Organization
  	in Computer Vision},
  year = {2006},
  pages = {201--208},
  address = {New York City, NY, USA},
  month = jun,
  bib2html_pubtype = {Conferences},
  bib2html_rescat = {Video Analysis, Visual Motion Segmentation},
  doi = {10.1109/CVPRW.2006.132},
  file = {pereraBHD06pocv.pdf:doretto\\conference\\pereraBHD06pocv.pdf:PDF;pereraBHD06pocv.pdf:doretto\\conference\\pereraBHD06pocv.pdf:PDF},
  owner = {doretto},
  timestamp = {2006.11.29}
}
```

CVPR
Modeling dynamic scenes with active appearance Doretto, G. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2005. Oral abstract bibTeX pdf
In this work we propose a model for video scenes that contain temporal variability in shape and appearance. We propose a conditionally linear model akin to a dynamic extension of active appearance models. We formulate the problem variationally, and propose a framework where a model complexity cost dictates the �modeling responsibility� of each of the factors: appearance, shape and motion. We render the learning problem well-posed by reverting to a physical and a dynamic prior, and use the finite element method to compute a numerical solution. We illustrate our model to learn and simulate the shape, appearance, and motion of scenes that exhibit some form of temporal regularity, intended in a statistical sense.
```
@inproceedings{doretto05cvpr,
  abbr = {CVPR},
  author = {Doretto, G.},
  title = {Modeling dynamic scenes with active appearance},
  booktitle = {Proceedings of the IEEE Computer Society Conference on Computer Vision
  	and Pattern Recognition},
  year = {2005},
  volume = {1},
  pages = {66--73},
  address = {San Diego, CA, USA},
  month = jun,
  bib2html_pubtype = {Conferences},
  bib2html_rescat = {Dynamic Textures, Visual Motion Analysis, Shape and Appearance Modeling},
  file = {doretto05cvpr.pdf:doretto\\conference\\doretto05cvpr.pdf:PDF;doretto05cvpr.pdf:doretto\\conference\\doretto05cvpr.pdf:PDF},
  owner = {doretto},
  wwwnote = {Oral}
}
```
DYNAMIC TEXTURES: modeling, learning, synthesis, animation, segmentation, and recognition Doretto, G. Ph.D. Thesis, University of California, Los Angeles, CA, 2005. Committee: Adnan Darwiche, Petros Faloutsos, Demetri Terzopoulos, Ying Nian Wu, Stefano Soatto (Chair) abstract bibTeX pdf
Dynamic textures are sequences of images of dynamic scenes that exhibit some temporal regularity properties, intended in a statistical sense; these include, for example, ocean waves, smoke, whirlwind, fire, foliage, but also moving objects with a “defined shape,” for instance flowers, or flags in wind etc. This work presents a characterization of this class of video sequences, and poses the problems of modeling, learning, synthesis, animation, recognition, and segmentation of dynamic textures. Since, in absence of any additional prior knowledge, the visual reconstruction problem from images alone is ill-posed, in this work we give up trying to infer the physical model that generated the images, and analyze sequences of images solely as visual signals. We do so by building a statistical framework, and draw on disciplines like time series analysis, system, control, and identification theory. We derive three generative models, the simplest possible, that are able to capture, respectively, the temporal second-order statistics, the spatio-temporal second-order statistics, and the higher-order temporal statistics of dynamic textures. We propose to learn model parameters in the maximum-likelihood sense, or minimum prediction error variance. We derive efficient closed-form inference procedures for learning the second-order statistics, and revert to nonlinear optimization techniques for the higher-order ones. After learning a model, it can be used to extrapolate, or predict new image data both in the temporal and spatial domain. We analyze the meaning of the parameters of a model, and show how they can be manipulated to control, or animate the simulation. Using the geometry of subspaces, and statistical pattern recognition theory we derive a technique to discriminate between models, and assess the potential for building a recognition system. Finally, by combining these results with a variational framework, we design a region-based segmentation system able to partition a video sequence into regions characterized by different spatio-temporal statistics.
```
@phdthesis{doretto05dissertation,
  author = {Doretto, G.},
  title = {{DYNAMIC TEXTURES}: modeling, learning, synthesis, animation, segmentation,
  	and recognition},
  school = {University of California},
  year = {2005},
  address = {Los Angeles, CA},
  month = mar,
  note = {{C}ommittee: {A}dnan {D}arwiche, {P}etros {F}aloutsos, {D}emetri
  	{T}erzopoulos, {Y}ing {N}ian {W}u, {S}tefano {S}oatto ({C}hair)},
  bib2html_pubtype = {Theses},
  bib2html_rescat = {Dynamic Textures, Visual Motion Analysis, Visual Motion Segmentation,
  	Visual Motion Recognition, Shape and Appearance Modeling, Image Based
  	Rendering},
  file = {doretto05dissertation.pdf:doretto\\thesis\\doretto05dissertation.pdf:PDF;doretto05dissertation.pdf:doretto\\thesis\\doretto05dissertation.pdf:PDF},
  owner = {doretto}
}
```
A saliency model for active camera control Kim, S. J., Doretto, G., Rittscher, J., Tu, P., and Pollefeys, M. Technical Report 2005GRC591, GE Global Research, 2005. Visualization and Computer Vision Laboratory abstract bibTeX
The goal of this work is to provide information for the automatic control of an active camera. One way to achieve this is to run a set of saliency detectors in the scene which is tuned to a specific class of objects. As opposed to this, we propose a formal definition for what we mean for salient motion by means of the concept of stationarity of stochastic processes. We propose a hierarchy of generative models for salient and nuisance motion, along with recursive learning procedures for on-line processing. By doing so, we turn the problem of detecting salient motion into the problem of model change detection, which we solve optimally online using the sequential generalized likelihood ratio test. Our system is designed for realtime applications, and is able to detect salient motion even in severely cluttered scenes.
```
@techreport{kimDRTP05tr,
  author = {Kim, S. J. and Doretto, G. and Rittscher, J. and Tu, P. and Pollefeys, M.},
  title = {A saliency model for active camera control},
  institution = {GE Global Research},
  year = {2005},
  number = {2005GRC591},
  address = {Niskayuna, NY, USA},
  month = nov,
  note = {Visualization and Computer Vision Laboratory},
  bib2html_pubtype = {Tech Reports},
  bib2html_rescat = {Video Analysis, Dynamic Textures, Visual Motion Detection},
  file = {kimDRTP05tr.pdf:doretto\\report\\kimDRTP05tr.pdf:PDF;kimDRTP05tr.pdf:doretto\\report\\kimDRTP05tr.pdf:PDF},
  keywords = {salient activity detection, linear dynamic systems, motion detection,
  	pan tilt zoom, camera, foreground/background segmentation, activity/novelty
  	detection, dynamic textures, sequential likelihood ratio},
  owner = {doretto},
  timestamp = {2006.11.29}
}
```
Moving object segmentation using scene understanding Perera, A. G. A., Brooksby, G., Hoogs, A., Doretto, G., and Kaufhold, J. Technical Report 2005GRC182, GE Global Research, 2005. Visualization and Computer Vision Laboratory abstract bibTeX
We present a novel approach to improving moving object detection algorithms for aerial video by incorporating scene understanding. We argue that scene understanding is a valuable input feature for moving object segmentation, and show how it can be used to reduce the false positive rate of any moving object detection algorithm. The approach is especially effective on sequences with large scene depth and much parallax, as often occurs with low, fast-moving sensors. In one difficult sequence, our results show an 8-fold reduction of false positives on average, with essentially no impact on the true positive rate, even when scene understanding is only moderately accurate. We show how scene understanding can be used to increase the accuracy of frame-to-frame homography estimates, and also present a novel approach that uses motion information from the previous frame for outlier rejection at the current frame to further increase accuracy.
```
@techreport{pereraBHDK05tr,
  author = {Perera, A. G. A. and Brooksby, G. and Hoogs, A. and Doretto, G. and Kaufhold, J.},
  title = {Moving object segmentation using scene understanding},
  institution = {GE Global Research},
  year = {2005},
  number = {2005GRC182},
  address = {Niskayuna, NY, USA},
  month = mar,
  note = {Visualization and Computer Vision Laboratory},
  bib2html_pubtype = {Tech Reports},
  bib2html_rescat = {Video Analysis, Visual Motion Segmentation},
  file = {pereraBHDK05tr.pdf:doretto\\report\\pereraBHDK05tr.pdf:PDF;pereraBHDK05tr.pdf:doretto\\report\\pereraBHDK05tr.pdf:PDF},
  keywords = {motion segmentation, moving object detection, scene understanding,
  	scene content, recognition},
  owner = {doretto},
  timestamp = {2006.11.29}
}
```

ECCV
Spatially homogeneous dynamic textures Doretto, G., Jones, E., and Soatto, S. In Proceedings of European Conference on Computer Vision, 2004. Oral abstract bibTeX pdf
We address the problem of modeling the spatial and temporal second-order statistics of video sequences that exhibit both spatial and temporal regularity, intended in a statistical sense. We model such sequences as dynamic multiscale autoregressive models, and introduce an efficient algorithm to learn the model parameters. We then show how the model can be used to synthesize novel sequences that extend the original ones in both space and time, and illustrate the power, and limitations, of the models we propose with a number of real image sequences.
```
@inproceedings{dorettoJS04eccv,
  abbr = {ECCV},
  author = {Doretto, G. and Jones, E. and Soatto, S.},
  title = {Spatially homogeneous dynamic textures},
  booktitle = {Proceedings of European Conference on Computer Vision},
  year = {2004},
  volume = {2},
  pages = {591--602},
  address = {Prague, Czech Republic},
  month = may,
  bib2html_pubtype = {Conferences},
  bib2html_rescat = {Dynamic Textures, Visual Motion Analysis},
  file = {dorettoJS04eccv.pdf:doretto/conference/dorettoJS04eccv.pdf:PDF},
  owner = {doretto},
  timestamp = {2013.10.18},
  wwwnote = {Oral}
}
```
Modeling dynamic scenes with active appearance Doretto, G., and Soatto, S. Technical Report TR040053, UCLA Computer Science Department, 2004. abstract bibTeX pdf
In this work we propose a model for video scenes that contain temporal variability in shape and appearance. We propose a conditionally linear model akin to a dynamic extension of active appearance models. We formulate the problem variationally, and use finite-element methods to compute a numerical solution. We illustrate our model to learn and simulate the shape, appearance, and motion of scenes that exhibit some form of temporal regularity, intended in a statistical sense.
```
@techreport{UCLA-CSD-TR040053,
  author = {Doretto, G. and Soatto, S.},
  title = {Modeling dynamic scenes with active appearance},
  institution = {UCLA Computer Science Department},
  year = {2004},
  number = {TR040053},
  address = {Los Angeles, CA, USA},
  month = dec,
  bib2html_pubtype = {Tech Reports},
  bib2html_rescat = {Dynamic Textures, Shape and Appearance Modeling, Visual Motion Analysis,
  	Image Based Rendering},
  file = {UCLA-CSD-TR040053.pdf:doretto\\report\\UCLA-CSD-TR040053.pdf:PDF;UCLA-CSD-TR040053.pdf:doretto\\report\\UCLA-CSD-TR040053.pdf:PDF},
  owner = {doretto}
}
```

IJCV
Dynamic textures Doretto, G., Chiuso, A., Wu, Y. N., and Soatto, S. International Journal of Computer Vision, 2003. abstract bibTeX pdf
Dynamic textures are sequences of images of moving scenes that exhibit certain stationarity properties in time; these include sea-waves, smoke, foliage, whirlwind etc. We present a characterization of dynamic textures that poses the problems of modeling, learning, recognizing and synthesizing dynamic textures on a firm analytical footing. We borrow tools from system identification to capture the “essence” of dynamic textures; we do so by learning (i.e. identifying) models that are optimal in the sense of maximum likelihood or minimum prediction error variance. For the special case of second-order stationary processes, we identify the model sub-optimally in closed-form. Once learned, a model has predictive power and can be used for extrapolating synthetic sequences to infinite length with negligible computational cost. We present experimental evidence that, within our framework, even low-dimensional models can capture very complex visual phenomena.
```
@article{dorettoCWS03ijcv,
  abbr = {IJCV},
  author = {Doretto, G. and Chiuso, A. and Wu, Y. N. and Soatto, S.},
  title = {Dynamic textures},
  journal = {International Journal of Computer Vision},
  year = {2003},
  volume = {51},
  pages = {91--109},
  number = {2},
  bib2html_pubtype = {Journals},
  bib2html_rescat = {Dynamic Textures, Visual Motion Analysis, Visual Motion Recognition},
  file = {dorettoCWS03ijcv.pdf:doretto\\journal\\dorettoCWS03ijcv.pdf:PDF;dorettoCWS03ijcv.pdf:doretto\\journal\\dorettoCWS03ijcv.pdf:PDF}
}
```
ICCV
Dynamic texture segmentation Doretto, G., Cremers, D., Favaro, P., and Soatto, S. In Proceedings of IEEE International Conference on Computer Vision, 2003. abstract bibTeX pdf
We address the problem of segmenting a sequence of images of natural scenes into disjoint regions that are characterized by constant spatio-temporal statistics. We model the spatio-temporal dynamics in each region by Gauss-Markov models, and infer the model parameters as well as the boundary of the regions in a variational optimization framework. Numerical results demonstrate that, in contrast to purely texture-based segmentation schemes, our method is effective in segmenting regions that differ in their dynamics even when spatial statistics are identical.
```
@inproceedings{dorettoCFS03iccv,
  abbr = {ICCV},
  author = {Doretto, G. and Cremers, D. and Favaro, P. and Soatto, S.},
  title = {Dynamic texture segmentation},
  booktitle = {Proceedings of IEEE International Conference on Computer Vision},
  year = {2003},
  volume = {2},
  pages = {1236--1242},
  address = {Nice, France},
  month = oct,
  bib2html_pubtype = {Conferences},
  bib2html_rescat = {Dynamic Textures, Visual Motion Segmentation},
  file = {dorettoCFS03iccv.pdf:doretto\\conference\\dorettoCFS03iccv.pdf:PDF;dorettoCFS03iccv.pdf:doretto\\conference\\dorettoCFS03iccv.pdf:PDF},
  owner = {doretto}
}
```

CVPR

Editable dynamic textures Doretto, G., and Soatto, S. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. abstract bibTeX pdf

@inproceedings{dorettoS03cvpr,
  abbr = {CVPR},
  author = {Doretto, G. and Soatto, S.},
  title = {Editable dynamic textures},
  booktitle = {Proceedings of the IEEE Computer Society Conference on Computer Vision
  	and Pattern Recognition},
  year = {2003},
  volume = {2},
  pages = {137--142},
  address = {Madison, Wisconsin, USA},
  month = jun,
  bib2html_pubtype = {Conferences},
  bib2html_rescat = {Dynamic Textures, Image Based Rendering},
  file = {dorettoS03cvpr.pdf:doretto\\conference\\dorettoS03cvpr.pdf:PDF;dorettoS03cvpr.pdf:doretto\\conference\\dorettoS03cvpr.pdf:PDF}
}

Towards plenoptic dynamic textures Doretto, G., and Soatto, S. In Proceedings of the 3rd International Workshop on Texture Analysis and Synthesis, 2003. abstract bibTeX pdf
We present a technique to infer a model of the spatio-temporal statistics of a collection of images of dynamic scenes seen from a moving camera. We use a time-variant linear dynamical system to jointly model the statistics of the video signal and the moving vantage point. We propose three approaches to inference, the first based on the plenoptic function, the second based on interpolating linear dynamical models, the third based on approximating the scene as piecewise planar. For the last two approaches, we also illustrate the potential of the proposed techniques with a number of experiments. The resulting algorithms could be useful for video editing where the motion of the vantage point can be controlled interactively, as well as to perform stabilized synthetic generation of video sequences.
```
@inproceedings{dorettoS03texture,
  author = {Doretto, G. and Soatto, S.},
  title = {Towards plenoptic dynamic textures},
  booktitle = {Proceedings of the 3rd International Workshop on Texture Analysis
  	and Synthesis},
  year = {2003},
  pages = {25--30},
  address = {Nice, France},
  month = oct,
  bib2html_pubtype = {Conferences},
  bib2html_rescat = {Dynamic Textures, Image Based Rendering, Registration from Dynamic
  	Textures},
  file = {dorettoS03texture.pdf:doretto\\conference\\dorettoS03texture.pdf:PDF;dorettoS03texture.pdf:doretto\\conference\\dorettoS03texture.pdf:PDF},
  owner = {doretto}
}
```

Dynamic texture modeling Doretto, G. Master's Thesis, University of California, Los Angeles, CA, 2002. Committee: Adnan Darwiche, Michael Dyer, Stefano Soatto (Chair). abstract bibTeX pdf
Dynamic textures are sequences of images of moving scenes that exhibit certain stationarity properties in time; these include, for example, sea-waves, smoke, foliage, whirlwind etc. This work presents a novel characterization of dynamic textures that poses the problems of modeling, learning, recognizing, classifying, synthesizing, and editing dynamic textures on a firm analytical footing. By means of system identification tools it is possible to capture the �essence� of dynamic textures; this is done by learning (i.e. identifying) models that are optimal in the sense of maximum-likelihood or minimum prediction error variance. For the special case of second-order stationary processes, a model can be identified sub-optimally in closed-form. Once learned, a model has predictive power and can be used for extrapolating, and editing (i.e. modifying the temporal and spatial behavior of) synthetic sequences. It is presented experimental evidence that, within this framework, even low-dimensional models can capture very complex visual phenomena. Furthermore, it is shown the possibility to map the manipulation of model parameters into sensible changes of visual appearance in extrapolated sequences. The uniqueness of the model allows to pose the problem of recognition and classification in the space of models. Since the space is non-linear, a distance between models must be defined. This work examines three different distances in the space of autoregressive models and assess their power.
```
@mastersthesis{doretto02thesis,
  author = {Doretto, G.},
  title = {Dynamic texture modeling},
  school = {University of California},
  year = {2002},
  address = {Los Angeles, CA},
  month = jun,
  note = {{C}ommittee: {A}dnan {D}arwiche, {M}ichael {D}yer, {S}tefano {S}oatto
  	({C}hair).},
  bib2html_pubtype = {Theses},
  bib2html_rescat = {Dynamic Textures, Visual Motion Analysis, Image Based Rendering},
  file = {doretto02thesis.pdf:doretto\\thesis\\doretto02thesis.pdf:PDF;doretto02thesis.pdf:doretto\\thesis\\doretto02thesis.pdf:PDF},
  owner = {doretto},
  timestamp = {2007.01.19}
}
```

Editable dynamic textures Doretto, G., and Soatto, S. In Conference Abstracts and Applications of SIGGRAPH ’02, 2002. Oral bibTeX pdf

@inproceedings{dorettoS02siggraph,
  author = {Doretto, G. and Soatto, S.},
  title = {Editable dynamic textures},
  booktitle = {Conference Abstracts and Applications of SIGGRAPH '02},
  year = {2002},
  pages = {177},
  address = {San Antonio, Texas, USA},
  month = jul,
  bib2html_pubtype = {Conferences},
  bib2html_rescat = {Dynamic Textures, Image Based Rendering},
  file = {dorettoS02siggraph.pdf:doretto\\conference\\dorettoS02siggraph.pdf:PDF;dorettoS02siggraph.pdf:doretto\\conference\\dorettoS02siggraph.pdf:PDF},
  owner = {doretto},
  timestamp = {2007.01.19},
  wwwnote = {Oral}
}

Editable dynamic textures Doretto, G., and Soatto, S. Technical Report TR020001, UCLA Computer Science Department, 2002. abstract bibTeX pdf

@techreport{UCLA-CSD-TR020001,
  author = {Doretto, G. and Soatto, S.},
  title = {Editable dynamic textures},
  institution = {UCLA Computer Science Department},
  year = {2002},
  number = {TR020001},
  address = {Los Angeles, CA, USA},
  month = jan,
  bib2html_pubtype = {Tech Reports},
  bib2html_rescat = {Dynamic Textures, Image Based Rendering},
  file = {UCLA-CSD-TR020001.pdf:doretto\\report\\UCLA-CSD-TR020001.pdf:PDF;UCLA-CSD-TR020001.pdf:doretto\\report\\UCLA-CSD-TR020001.pdf:PDF},
  keywords = {Animation, Texture Synthesis, Video, Computer Vision, Image Processing,
  	Compression},
  owner = {doretto}
}

Towards plenoptic dynamic textures Doretto, G., and Soatto, S. Technical Report TR020043, UCLA Computer Science Department, 2002. abstract bibTeX pdf

We present a technique to infer a model of the spatio-temporal statistics of a collection of images of dynamic scenes seen from a moving camera. We propose three inference algorithms that can be used to stabilize and register images of dynamic scenes, as well as to generate synthetic sequences where the motion of the vantage point can be controlled interactively.

@techreport{UCLA-CSD-TR020043,
  author = {Doretto, G. and Soatto, S.},
  title = {Towards plenoptic dynamic textures},
  institution = {UCLA Computer Science Department},
  year = {2002},
  number = {TR020043},
  address = {Los Angeles, CA, USA},
  month = nov,
  bib2html_pubtype = {Tech Reports},
  bib2html_rescat = {Dynamic Textures, Image Based Rendering, Registration from Dynamic
  	Textures},
  file = {UCLA-CSD-TR020043.pdf:doretto\\report\\UCLA-CSD-TR020043.pdf:PDF;UCLA-CSD-TR020043.pdf:doretto\\report\\UCLA-CSD-TR020043.pdf:PDF},
  keywords = {stereo and structure from motion, image-based modeling, multiview
  	stereo, 3D reconstruction},
  owner = {doretto}
}

TPAMI
A frequency domain technique for range data registration Lucchese, L., Doretto, G., and Cortelazzo, G. M. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002. abstract bibTeX pdf doi
This work introduces an original method for registering pairs of 3D views consisting of range data sets which operates in the frequency domain. The Fourier transform allows the decoupling of the estimate of the rotation parameters from the estimate of the translation parameters, our algorithm exploits this well- known property by suggesting a three-step procedure. The rotation parameters are estimated by the first two steps through convenient representations and projections of the Fourier transforms’ magnitudes and the translational displacement is recovered by the third step by means of a standard phase correlation technique after compensating one of the two views for rotation. The performance of the algorithm, which is well-suited for unsupervised registration, is clearly assessed through extensive testing with several objects and shows that good and robust estimates of 3D rigid motion are achievable. Our algorithm can be used as a prealignment tool for more accurate space-domain registration techniques, like the ICP algorithm.
```
@article{luccheseDC02IEEEtpami,
  abbr = {TPAMI},
  author = {Lucchese, L. and Doretto, G. and Cortelazzo, G. M.},
  title = {A frequency domain technique for range data registration},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2002},
  volume = {24},
  pages = {1468--1484},
  number = {11},
  month = nov,
  bib2html_pubtype = {Journals},
  bib2html_rescat = {3D Object modeling},
  doi = {10.1109/TPAMI.2002.1046160},
  file = {luccheseDC02IEEEtpami.pdf:doretto\\journal\\luccheseDC02IEEEtpami.pdf:PDF;luccheseDC02IEEEtpami.pdf:doretto\\journal\\luccheseDC02IEEEtpami.pdf:PDF},
  issn = {0162-8828},
  keywords = {Fourier transforms, frequency-domain analysis, image motion analysis,
  	image registration, parameter estimation},
  owner = {doretto},
  timestamp = {2007.01.19}
}
```

CVPR
Dynamic texture recognition Saisan, P., Doretto, G., Wu, Y. N., and Soatto, S. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2001. abstract bibTeX pdf
Dynamic textures are sequences of images that exhibit some form of temporal stationarity, such as waves, steam, and foliage. We pose the problem of recognizing and classifying dynamic textures in the space of dynamical systems where each dynamic texture is uniquely represented. Since the space is non-linear, a distance between models must be defined. We examine three different distances in the space of autoregressive models and assess their power.
```
@inproceedings{saisanDWS01cvpr,
  abbr = {CVPR},
  author = {Saisan, P. and Doretto, G. and Wu, Y. N. and Soatto, S.},
  title = {Dynamic texture recognition},
  booktitle = {Proceedings of the IEEE Computer Society Conference on Computer Vision
  	and Pattern Recognition},
  year = {2001},
  volume = {2},
  pages = {58--63},
  address = {Kauai, Hawaii, USA},
  month = dec,
  bib2html_pubtype = {Conferences},
  bib2html_rescat = {Dynamic Textures, Visual Motion Recognition},
  file = {saisanDWS01cvpr.pdf:doretto\\conference\\saisanDWS01cvpr.pdf:PDF;saisanDWS01cvpr.pdf:doretto\\conference\\saisanDWS01cvpr.pdf:PDF},
  owner = {doretto}
}
```
ICCV
Dynamic textures Soatto, S., Doretto, G., and Wu, Y. N. In Proceedings of IEEE International Conference on Computer Vision, 2001. Oral abstract bibTeX pdf
Dynamic textures are sequences of images of moving scenes that exhibit certain stationarity properties in time; these include sea-waves, smoke, foliage, whirlwind but also talking faces, traffic scenes etc. We present a novel characterization of dynamic textures that poses the problems of modelling, learning, recognizing and synthesizing dynamic textures on a firm analytical footing. We borrow tools from system identification to capture the �essence� of dynamic textures; we do so by learning (i.e. identifying) models that are optimal in the sense of maximum likelihood or minimum prediction error variance. For the special case of secondorder stationary processes we identify the model in closed form. Once learned, a model has predictive power and can be used for extrapolating synthetic sequences to infinite length with negligible computational cost. We present experimental evidence that, within our framework, even low dimensional models can capture very complex visual phenomena.
```
@inproceedings{soattoDW01iccv,
  abbr = {ICCV},
  author = {Soatto, S. and Doretto, G. and Wu, Y. N.},
  title = {Dynamic textures},
  booktitle = {Proceedings of IEEE International Conference on Computer Vision},
  year = {2001},
  volume = {2},
  pages = {439--446},
  address = {Vancouver, BC, Canada},
  month = jul,
  bib2html_pubtype = {Conferences},
  bib2html_rescat = {Dynamic Textures, Visual Motion Analysis},
  file = {soattoDW01iccv.pdf:doretto\\conference\\soattoDW01iccv.pdf:PDF;soattoDW01iccv.pdf:doretto\\conference\\soattoDW01iccv.pdf:PDF},
  wwwnote = {Oral}
}
```

Closed-form learning and synthesis of dynamic textures Soatto, S., Doretto, G., and Wu, Y. N. In Workshop on the Convergence of Graphics, Vision, and Video, 2001. (Abstract and poster) bibTeX

@conference{soattoDW01workshop,
  author = {Soatto, S. and Doretto, G. and Wu, Y. N.},
  title = {Closed-form learning and synthesis of dynamic textures},
  booktitle = {Workshop on the Convergence of Graphics, Vision, and Video},
  year = {2001},
  address = {Berkeley, CA, USA},
  month = mar,
  note = {(Abstract and poster)},
  bib2html_pubtype = {Other},
  bib2html_rescat = {Dynamic Textures, Visual Motion Analysis},
  owner = {doretto},
  timestamp = {2007.01.21}
}

Dynamic textures Doretto, G., Pundir, P., Wu, Y. N., and Soatto, S. Technical Report TR200032, UCLA Computer Science Department, 2000. abstract bibTeX pdf
Dynamic textures are sequences of images of moving scenes that exhibit certain stationariety properties in space and time; these include sea-waves, smoke, foliage, whirlwinds etc. We present a novel characterization of dynamic texture that poses the problems of modeling, learning, recognizing and synthesizing dynamic textures on a firm analytical footing. We borrow tools from stochastic realization theory to capture the �essence� of dynamic textures; we do so by learning models that are optimal in the sense of maximum likelihood or minimum prediction error variance. Once learned, therefore, a model has predictive power and can be used for extrapolating synthetic sequences to infinite length with negligible computational cost. We present experimental evidence that, within our framework, even low dimensional models can capture very complex visual phenomena.
```
@techreport{UCLA-CSD-TR200032,
  author = {Doretto, G. and Pundir, P. and Wu, Y. N. and Soatto, S.},
  title = {Dynamic textures},
  institution = {UCLA Computer Science Department},
  year = {2000},
  number = {TR200032},
  address = {Los Angeles, CA, USA},
  month = dec,
  bib2html_pubtype = {Tech Reports},
  bib2html_rescat = {Dynamic Textures, Visual Motion Analysis, Visual Motion Recognition},
  file = {UCLA-CSD-TR200032.pdf:doretto\\report\\UCLA-CSD-TR200032.pdf:PDF;UCLA-CSD-TR200032.pdf:doretto\\report\\UCLA-CSD-TR200032.pdf:PDF},
  keywords = {textures, dynamic scene analysis, 3D textures, minimum description
  	length, image priors, image compression, generative model, prediction
  	error methods, ARMA regression, system identification, learning,
  	E-M algorithm},
  owner = {doretto}
}
```

ICIP
Free-form textured surfaces registration by a frequency domaintechnique Cortelazzo, G. M., Doretto, G., and Lucchese, L. In Proceedings of IEEE International Conference on Image Processing, 1998. abstract bibTeX pdf doi
Free-form 3-D surfaces registration is a fundamental problem in 3-D imaging, typically approached by extensions or variations of the ICP algorithm. This work presents a new frequency domain technique for 3-D view registration, totally different form any other techniques for 3-D motion estimation also, based on the Fourier transform. The proposed method can give a non-feature- based method for unsupervised registration of 3-D views. The obtained results are useful “per se” in applications targeted to visual quality or can serve as good starting point for the ICP algorithm when a higher precision is needed
```
@inproceedings{cortelazzoDL98icip,
  abbr = {ICIP},
  author = {Cortelazzo, G. M. and Doretto, G. and Lucchese, L.},
  title = {Free-form textured surfaces registration by a frequency domaintechnique},
  booktitle = {Proceedings of IEEE International Conference on Image Processing},
  year = {1998},
  volume = {1},
  pages = {813--817},
  address = {Chicago, IL, USA},
  month = oct,
  bib2html_pubtype = {Conferences},
  bib2html_rescat = {3D Object Modeling},
  doi = {10.1109/ICIP.1998.723634},
  file = {cortelazzoDL98icip.pdf:doretto\\conference\\cortelazzoDL98icip.pdf:PDF;cortelazzoDL98icip.pdf:doretto\\conference\\cortelazzoDL98icip.pdf:PDF},
  keywords = {Fourier transforms, frequency-domain analysis, image registration,
  	image texture, motion estimation},
  owner = {doretto},
  timestamp = {2007.01.19}
}
```
EUSIPCO
Three dimensional view registration by a frequency domain technique Cortelazzo, G. M., Doretto, G., Lucchese, L., and Totaro, S. In Proceedings of the European Signal Processing Conference, 1998. abstract bibTeX pdf
This work presents the algorithmic variations of a 3D motion estimation method which make it suitable to 3D view registration. Important characteristics are that the method is suitable for unsupervised 3D registration and that texture information can be incorporated with beneficial effects on the overall 3D registration performance. A peculiarity of the method is that it operates in the frequency domain.
```
@inproceedings{cortelazzoDLT98eusipco,
  abbr = {EUSIPCO},
  author = {Cortelazzo, G. M. and Doretto, G. and Lucchese, L. and Totaro, S.},
  title = {Three dimensional view registration by a frequency domain technique},
  booktitle = {Proceedings of the European Signal Processing Conference},
  year = {1998},
  pages = {1289--1292},
  address = {Island of Rhodes, Greece},
  month = sep,
  bib2html_pubtype = {Conferences},
  bib2html_rescat = {3D Object Modeling},
  file = {cortelazzoDLT98eusipco.pdf:doretto\\conference\\cortelazzoDLT98eusipco.pdf:PDF;cortelazzoDLT98eusipco.pdf:doretto\\conference\\cortelazzoDLT98eusipco.pdf:PDF},
  owner = {doretto},
  timestamp = {2007.01.19}
}
```
ISCAS
A frequency domain method for registration of range data Cortelazzo, G. M., Doretto, G., Lucchese, L., and Totaro, S. In Proceedings of the IEEE International Symposium on Circuits and Systems, 1998. abstract bibTeX pdf doi
Free-form 3-D surfaces registration is a fundamental problem in 3-D imaging, typically approached by extensions or variations of the ICP algorithm. This work presents an alternative method for 3-D motion estimation based on the Fourier transform of the range data. The frequency domain techniques for estimating motion parameters are nonfeature-based methods suitable for unsupervised registration of 3-D views. The proposed method can give unsupervised registration of range data within 1� of accuracy of the angular parameters, useful �per se� in applications where this kind of precision is adequate or which can serve as good starting point for the ICP algorithm when a higher precision is needed
```
@inproceedings{cortelazzoDLT98iscas,
  abbr = {ISCAS},
  author = {Cortelazzo, G. M. and Doretto, G. and Lucchese, L. and Totaro, S.},
  title = {A frequency domain method for registration of range data},
  booktitle = {Proceedings of the IEEE International Symposium on Circuits and Systems},
  year = {1998},
  volume = {5},
  pages = {518--521},
  address = {Monterey, CA, USA},
  month = may,
  bib2html_pubtype = {Conferences},
  bib2html_rescat = {3D Object Modeling},
  doi = {10.1109/ISCAS.1998.694546},
  file = {cortelazzoDLT98iscas.pdf:doretto\\conference\\cortelazzoDLT98iscas.pdf:PDF;cortelazzoDLT98iscas.pdf:doretto\\conference\\cortelazzoDLT98iscas.pdf:PDF},
  keywords = {Fourier transforms, frequency-domain analysis, motion estimation},
  owner = {doretto},
  timestamp = {2007.01.19}
}
```

Frequency domain estimation of the 3D motion parameters Doretto, G. Master's Thesis, University of Padua, Padua, Italy, 1998. (In Italian) Telecom Italia Prize bibTeX pdf

@mastersthesis{doretto98thesis,
  author = {Doretto, G.},
  title = {Frequency domain estimation of the 3{D} motion parameters},
  school = {University of {P}adua},
  year = {1998},
  address = {Padua, Italy},
  month = oct,
  note = {(In {I}talian)},
  bib2html_pubtype = {Theses},
  bib2html_rescat = {3D Object Modeling},
  file = {doretto98thesis.pdf:doretto\\thesis\\doretto98thesis.pdf:PDF;doretto98thesis.pdf:doretto\\thesis\\doretto98thesis.pdf:PDF},
  owner = {doretto},
  timestamp = {2007.01.19},
  wwwnote = {Telecom Italia Prize}
}

3DIM
Frequency domain estimation of 3D rigid motion based on range and intensity data Lucchese, L., Doretto, G., and Cortelazzo, G. M. In Proceedings of the International Conference on 3D Digital Imaging and Modeling, 1997. Oral abstract bibTeX pdf
Videorate registered range and intensity data are at reach of current sensor technology. This wealth of data can be profitably exploited in order to estimate rigid motion parameters as the approaches to 3D motion estimation based on the opticalflow of both types of data indicate. This work introduces an alternative for 3D motion estimation based on the Fourier transform of the 3D intensity function implicitly described by the registered timesequences of range and intensity data. The proposed procedure can lead to an unsupervised method for 3D rigid motion estimation. This method has several advantages related to the fact that it uses the total available information and not sets of features. With respect to memory occupancy the use of a time sequence of a 3D intensity function represents a considerable data reduction with respect to a pair of time sequences of 3D functions. The proposed technique which extends to the 3D case previous frequency domain estimation algorithms developed for the planar case retain their robustness.
```
@inproceedings{luccheseDC97threedim,
  abbr = {3DIM},
  author = {Lucchese, L. and Doretto, G. and Cortelazzo, G. M.},
  title = {Frequency domain estimation of 3{D} rigid motion based on range and
  	intensity data},
  booktitle = {Proceedings of the International Conference on 3D Digital Imaging
  	and Modeling},
  year = {1997},
  pages = {107--112},
  address = {Ottawa, Ontario, Canada},
  month = may,
  bib2html_pubtype = {Conferences},
  bib2html_rescat = {3D Object Modeling},
  file = {luccheseDC97threedim.pdf:doretto\\conference\\luccheseDC97threedim.pdf:PDF;luccheseDC97threedim.pdf:doretto\\conference\\luccheseDC97threedim.pdf:PDF},
  owner = {doretto},
  timestamp = {2007.01.19},
  wwwnote = {Oral}
}
```

Publications

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1998

1997