Readspeaker menu

Active Learning

Which of your millions of unlabeled images are actually worth being labeled?

Alexander Freytag, Erik Rodner, Paul Bodesheim, Christoph Käding, and Joachim Denzler


Motivation

Although labeled data lies at the very core of most computer vision systems, obtaining labeled data that is useful and reliable is commonly a crucial problem. To reduce the amount of manual labeling, active learning techniques aim at explicitely picking samples that are actually worth being labeled with respect to the problem on hand. In this area of research, we are interested in modeling the "worthyness" of an unlabeled sample and to apply our algorithms to human-in-the-loop recognition systems.


Expected Model Output Changes as a proxy for estimated error reduction.


[Kaeding16_ACE]

Christoph Käding and Alexander Freytag and Erik Rodner and Joachim Denzler. Large-scale Active Learning with Approximated Expected Model Output Changes. NIPS Workshop on Continual Learning and Deep Networks (NIPS-WS). 2016. [pdf] [bib]

Abstract.
The demands on visual recognition systems do not end with the complexity offered by current large-scale image datasets, such as ImageNet. In consequence, we need curious and continuously learning algorithms that actively acquire knowledge about semantic concepts which are present in available unlabeled data. As a step towards this goal, we show how to perform continuous active learning and exploration, where an algorithm actively selects relevant batches of unlabeled examples for annotation. These examples could either belong to already known or to yet undiscovered classes. Our algorithm is based on a new generalization of the Expected Model Output Change principle for deep architectures and is especially tailored to deep neural networks. Furthermore, we show easy-to-implement approximations that yield efficient techniques for active selection. Empirical experiments show that our method outperforms currently used heuristics.


[Kaeding16_LAA]

Christoph Käding and Alexander Freytag and Erik Rodner and Joachim Denzler. Large-scale Active Learning with Approximated Expected Model Output Changes. German Conference on Pattern Recognition (GCPR). 2016. [pdf] [bib]

Abstract.
Current visual recognition algorithms are ``hungry'' for data but massive annotation is extremely costly. Therefore, active learning algorithms are required that reduce labeling efforts to a minimum by selecting examples that are most valuable for labeling. In active learning, all categories occurring in collected data are usually assumed to be known in advance and experts should be able to label every requested instance. But do these assumptions really hold in practice? Could you name all categories in every image? Existing algorithms completely ignore the fact that there are certain examples where an oracle can not provide an answer or which even do not belong to the current problem domain. Ideally, active learning techniques should be able to discover new classes and at the same time cope with queries an expert is not able or willing to label. To meet these observations, we present a variant of the expected model output change principle for active learning and discovery in the presence of unnameable instances. Our experiments show that in these realistic scenarios, our approach substantially outperforms previous active learning methods, which are often not even able to improve with respect to the baseline of random query selection.


[Kaeding16_WALI]

Christoph Käding and Erik Rodner and Alexander Freytag and Joachim Denzler. Watch, Ask, Learn, and Improve: A Lifelong Learning Cycle for Visual Recognition. European Symposium on Artificial Neural Networks (ESANN). 2016. [pdf] [bib]

Abstract.
We present WALI, a prototypical system that learns object categories over time by continuously watching online videos. WALI actively asks questions to a human annotator about the visual content of observed video frames. Thereby, WALI is able to receive information about new categories and to simultaneously improve its generalization abilities. The functionality of WALI is driven by scalable active learning, efficient incremental learning, as well as state-of-the-art visual descriptors. In our experiments, we show qualitative and quantitative statistics about WALI's learning process. WALI runs continuously and regularly asks questions.

Image region selected with EMOC for annotation

[Kaeding15_ALD]

Christoph Käding and Alexander Freytag and Erik Rodner and Paul Bodesheim and Joachim Denzler. Active Learning and Discovery of Object Categories in the Presence of Unnameable Instances. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2015. [pdf] [bib]

Abstract.
Current visual recognition algorithms are ``hungry'' for data but massive annotation is extremely costly. Therefore, active learning algorithms are required that reduce labeling efforts to a minimum by selecting examples that are most valuable for labeling. In active learning, all categories occurring in collected data are usually assumed to be known in advance and experts should be able to label every requested instance. But do these assumptions really hold in practice? Could you name all categories in every image? Existing algorithms completely ignore the fact that there are certain examples where an oracle can not provide an answer or which even do not belong to the current problem domain. Ideally, active learning techniques should be able to discover new classes and at the same time cope with queries an expert is not able or willing to label. To meet these observations, we present a variant of the expected model output change principle for active learning and discovery in the presence of unnameable instances. Our experiments show that in these realistic scenarios, our approach substantially outperforms previous active learning methods, which are often not even able to improve with respect to the baseline of random query selection.

active learning with EMOC

[Freytag14_SIE]

Alexander Freytag and Erik Rodner and Joachim Denzler. Selecting Influential Examples: Active Learning with Expected Model Output Changes. European Conference on Computer Vision (ECCV). 2014. 562--577. [pdf] [bib]

Abstract.
In this paper, we introduce a new general strategy for active learning. The key idea of our approach is to measure the expected change of model outputs, a concept that generalizes previous methods based on expected model change and incorporates the underlying data distribution. For each example of an unlabeled set, the expected change of model predictions is calculated and marginalized over the unknown label. This results in a score for each unlabeled example that can be used for active learning with a broad range of models and learning algorithms. In particular, we show how to derive very efficient active learning methods for Gaussian process regression, which implement this general strategy, and link them to previous methods. We analyze our algorithms and compare them to a broad range of previous active learning strategies in experiments showing that they outperform state-of-the-art on well-established benchmark datasets in the area of visual object recognition.

Implicitely balancing the exploration-exploitation tradeoff using Gaussian Processes

pitfallsInAL

[Freytag13_LET]

Alexander Freytag and Erik Rodner and Paul Bodesheim and Joachim Denzler: Labeling Examples that matter: Relevance-Based Active Learning with Gaussian Processes. German Conference on Computer Vision (GCPR). 2013. 282 -- 291. (Oral) [pdf] [bib]

Abstract.
Active learning is an essential tool to reduce manual annotation costs in the presence of large amounts of unsupervised data. In this paper, we introduce new active learning methods based on measuring the impact of a new example on the current model. This is done by deriving model changes of Gaussian process models in closed form. Furthermore, we study typical pitfalls in active learning and show that our methods automatically balance between the exploitation and the exploration trade-off. Experiments are performed with established benchmark datasets for visual object recognition and show that our new active learning techniques are able to outperform state-of-the-art methods.

Code available!

We released the software (Matlab) for our GCPR 13 paper on active learning query strategies based on model impact.
dl2 Code for active learning with Gaussian process based query strategies (GCPR 13) ... latest version is available on the >>Project page on GitHub<<

lgpl-logo

Active Learning with Gaussian Processes

  GP-HIK_AL_ACCV

[Freytag12_RUC]

Alexander Freytag and Erik Rodner and Paul Bodesheim and Joachim Denzler. Rapid Uncertainty Computation with Gaussian Processes and Histogram Intersection Kernels. Asian Conference on Computer Vision (ACCV). 2012. (Oral). [pdf] [bib] [poster] Best Paper Honorable Mention

Short summary.
The main focus of this paper is to speed up computations of the GP posterior variance as well as the model update when a new sample is included. However, these techniques can successfully be used to speed up computing active learning scores as well. We therefor applied the GP active learning methods introduced by Kapoor et al. to active learning scenarios. In addition, we introduced a novel combination of explorative (variance-based) and exploitative (mean-based) strategies which gave an additional boost compared to passive learning.

Publications

[Kaeding16_ACE]

Christoph Käding and Alexander Freytag and Erik Rodner and Joachim Denzler. Large-scale Active Learning with Approximated Expected Model Output Changes. NIPS Workshop on Continual Learning and Deep Networks (NIPS-WS). 2016. [pdf][bib]

[Kaeding16_LAA]

Christoph Käding and Alexander Freytag and Erik Rodner and Joachim Denzler. Large-scale Active Learning with Approximated Expected Model Output Changes. German Conference on Pattern Recognition (GCPR). 2016. [pdf][bib]

[Kaeding16_WALI]

Christoph Käding and Erik Rodner and Alexander Freytag and Joachim Denzler. Watch, Ask, Learn, and Improve: A Lifelong Learning Cycle for Visual Recognition. European Symposium on Artificial Neural Networks (ESANN). 2016. [pdf] [bib]

[Kaeding15_ALD]

Christoph Käding and Alexander Freytag and Erik Rodner and Paul Bodesheim and Joachim Denzler. Active Learning and Discovery of Object Categories in the Presence of Unnameable Instances. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2015. [pdf] [bib]

[Freytag14_SIE]

Alexander Freytag and Erik Rodner and Joachim Denzler. Selecting Influential Examples: Active Learning with Expected Model Output Changes. European Conference on Computer Vision (ECCV). 2014. 562--577. [pdf] [bib]

[Freytag13_LET]

Alexander Freytag and Erik Rodner and Paul Bodesheim and Joachim Denzler: Labeling Examples that matter: Relevance-Based Active Learning with Gaussian Processes. German Conference on Computer Vision (GCPR). 2013. 282 -- 291. (Oral) [pdf] [bib]

[Freytag12_RUC]

Alexander Freytag and Erik Rodner and Paul Bodesheim and Joachim Denzler. Rapid Uncertainty Computation with Gaussian Processes and Histogram Intersection Kernels. Asian Conference on Computer Vision (ACCV). 2012. (Oral). [pdf] [bib] [poster] Best Paper Honorable Mention