Skip to content
Jaehyun Jun
한국어
Back to publications

International Conferences

Overcoming Catastrophic Forgetting by Incremental Moment Matching

A continual learning method that trains tasks independently and merges their networks by matching weight-posterior moments, reducing catastrophic forgetting without constraining new-task training.

Published
2017
Venue
NeurIPS 2017★ Spotlight
Authors
Sang-Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Jung-Woo Ha, Byoung-Tak Zhang
  • Continual Learning
  • Catastrophic Forgetting
Averaged accuracies on disjoint MNIST (two sequential tasks) and shuffled MNIST (three sequential tasks), from paper Table 1. Untuned uses the most natural hyperparameter in each algorithm's formulation; tuned uses hand-tuned values.
Averaged accuracies on disjoint MNIST (two sequential tasks) and shuffled MNIST (three sequential tasks), from paper Table 1. Untuned uses the most natural hyperparameter in each algorithm's formulation; tuned uses hand-tuned values.

Overview

Incremental Moment Matching (IMM) addresses catastrophic forgetting in continual learning. Instead of constraining how a model learns each new task, IMM trains tasks independently and then merges the resulting networks by matching the moments of their parameter distributions.

The work was published as a Spotlight paper at NeurIPS 2017.

My Contributions

  • Co-authored the research that reframed continual learning as a network-merging problem based on posterior moment matching.
  • Implemented the IMM methodology and its experimental codebase.
  • Designed and conducted MNIST-based transfer-learning experiments to validate the IMM methodology.

Technical Approach

  • Independent task training — Learns each task without directly restricting the optimization path of subsequent tasks.
  • mean-IMM — Merges independently trained networks by averaging their parameters.
  • mode-IMM — Uses task-specific Fisher information to find the mode of a Gaussian mixture and weight parameters by their importance to each task.
  • Transfer techniques — Weight transfer, L2-transfer, and drop-transfer smooth the parameter space so that interpolation produces a useful shared solution.

Results

  • Achieved 94.12% average accuracy on disjoint MNIST, compared with 52.72% for EWC and 71.32% for standard SGD.
  • Reached 98.30% on shuffled MNIST with mode-IMM and L2-transfer.
  • Demonstrated that the approach generalizes beyond MNIST to image-recognition and Lifelog datasets.
  • Selected for a Spotlight presentation at NeurIPS 2017.

Why It Matters

IMM offered a different perspective on continual learning: previously learned knowledge can be retained by combining independently optimized models rather than only by restricting future updates. This separation makes it possible to learn new tasks freely and consolidate knowledge afterward.