Hyperbolic Deep Learning for Foundation Models: A Survey

Abstract

Foundation models pre-trained on massive datasets, including large language models (LLMs), vision-language models (VLMs), and large multimodal models, have demonstrated remarkable success in diverse downstream tasks. However, recent studies have shown fundamental limitations of these models: (1) limited representational capacity, (2) lower adaptability, and (3) diminishing scalability. These shortcomings raise a critical question: is Euclidean geometry truly the optimal inductive bias for all foundation models, or could incorporating alternative geometric spaces enable models to better align with the intrinsic structure of real-world data and improve reasoning processes? Hyperbolic spaces, a class of non-Euclidean manifolds characterized by exponential volume growth with respect to distance, offer a mathematically grounded solution. These spaces enable low-distortion embeddings of hierarchical structures (e.g., trees, taxonomies) and power-law distributions with substantially fewer dimensions compared to Euclidean counterparts. Recent advances have leveraged these properties to enhance foundation models, including improving LLMs' complex reasoning ability, VLMs' zero-shot generalization, and cross-modal semantic alignment, while maintaining parameter efficiency. This paper provides a comprehensive review of hyperbolic neural networks and their recent development for foundation models. We further outline key challenges and research directions to advance the field.

11 Pages, SIGKDD 2025

References in corpus (19)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• An Introduction to Variational Autoencoders
• A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
• Hyperbolic Geometry of Complex Networks
• Simple and Deep Graph Convolutional Networks
• Hyperbolic Graph Convolutional Neural Networks
• Representation Tradeoffs for Hyperbolic Embeddings
• Greedy Forwarding in Dynamic Scale-Free Networks Embedded in Hyperbolic Metric Spaces
• Poincaré Embeddings for Learning Hierarchical Representations
• Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry
• HICF: Hyperbolic Informative Collaborative Filtering
• A Wrapped Normal Distribution on Hyperbolic Space for Gradient-Based Learning
• Hyperbolic Entailment Cones for Learning Hierarchical Embeddings
• Graph Geometry Interaction Learning
• Modeling Heterogeneous Hierarchies with Relation-specific Hyperbolic Cones
• Understanding and Mitigating Hyperbolic Dimensional Collapse in Graph Contrastive Learning
• Hyperbolic Contrastive Learning for Visual Representations beyond Objects
• Hyperbolic Image-Text Representations
• Hyperbolic Vision Transformers: Combining Improvements in Metric Learning