The Information-Theoretic Benefit of Shared Representations under Orthogonality Constraints

Thomas Dittrich, Oliver Potocki, Philipp Grohs

Modern deep learning architectures are increasingly multi-task and multi-modal, using a pretrained foundation model combined with task-specific, fine-tuned models. Empirically, exp…