Auto Seed Vl2 [ Premium — 2024 ]

[6] von Oswald, J., et al. (2020). Continual learning with hypernetworks. ICLR.

. A seed is a tuple ( s = (v, w) ), where ( v \in \mathbbR^d ) is a visual prototype and ( w \in \mathbbR^d ) is a textual prototype, such that for any example ( (x, y) ) from a past task, ( |f_I(x) - v| ) and ( |f_T(y) - w| ) are small, and ( \textsim(v, w) ) is high.

Auto-Seed VL2 maintains a set of auto-generated seeds ( \mathcalS ) that grows slowly over tasks. Auto-Seed VL2 operates in three phases per task: (1) Seed replay, (2) Online adaptation, (3) Seed update. 4.1 Overall Architecture auto seed vl2

The consistency loss and gradient-conditioned generation are crucial. Seed pruning is memory-efficient without hurting accuracy. We measure FWT: performance on task ( t ) after training on tasks ( 1..t-1 ). Auto-Seed VL2 achieves positive forward transfer (FWT = +4.1%) on VL-CL, meaning seeds from earlier tasks help learn new tasks. ER-VLM shows near-zero FWT; generative replay shows negative transfer due to noisy synthetic images. 7. Analysis and Discussion What do generated seeds encode? We project seeds into CLIP space and compare to real class means. The cosine similarity is 0.89 ± 0.05, indicating faithful representation. However, seeds are more “regularized” – they have lower variance along task-irrelevant directions.

[2] Shin, H., et al. (2017). Continual learning with deep generative replay. NIPS. [6] von Oswald, J

| Configuration | Avg Acc | Drop | |----------------------------------------|---------|------| | Full Auto-Seed VL2 | 82.2 | — | | w/o consistency loss (( \mathcalL \textconsist )) | 75.4 | -6.8 | | w/o gradient-conditioned generation (random seeds) | 68.9 | -13.3 | | w/o meta-update of ( G \phi ) | 74.1 | -8.1 | | w/o seed pruning (full memory) | 82.0 | -0.2 (ns) |

[4] Thengane, V., et al. (2023). Continual-CLIP: Fine-tuning CLIP for continual learning. CVPR Workshop. Auto-Seed VL2 maintains a set of auto-generated seeds

By generating seeds in embedding space rather than pixel space, we avoid the compounding errors of full image generation. The hypernetwork’s meta-learning objective ensures that seeds are discriminative for the original task and compatible with the continually updated VLM.