8 papers
SJEPA: Learning Elegant Latent Dynamics with Hybrid Symbolic-Neural Predictors
Yongchao Huang
Joint-embedding predictive architectures learn abstract states by predicting target embeddings from context embeddings, but their transition models are typically opaque neural maps…
Gaussian Mixture Attention: Linear-Time Sequence Mixing via Probabilistic Latent Routing
Yongchao Huang, Hassan Raza
The dense token-to-token interaction pattern of standard dot-product attention remains a central bottleneck in scaling Transformer architectures to long contexts. We introduce \tex…
Knowledge, Rules and Their Embeddings: Two Paths towards Neuro-Symbolic JEPA
Yongchao Huang, Hassan Raza
Modern self-supervised predictive architectures excel at capturing complex statistical correlations from high-dimensional data but lack mechanisms to internalize verifiable human l…
VJEPA: Variational Joint Embedding Predictive Architectures as Probabilistic World Models
Yongchao Huang
Joint Embedding Predictive Architectures (JEPA) offer a scalable paradigm for self-supervised learning by predicting latent representations rather than reconstructing high-entropy…
Sampling via Gaussian Mixture Approximations
Yongchao Huang
We present a family of \textit{Gaussian Mixture Approximation} (GMA) samplers for sampling unnormalised target densities, encompassing \textit{weights-only GMA} (W-GMA), \textit{La…
Semantic Fusion with Fuzzy-Membership Features for Controllable Language Modelling
Yongchao Huang, Hassan Raza
We propose semantic fusion, a lightweight scheme that augments a Transformer language model (LM) with a parallel, fuzzy-membership feature channel that encodes token-level semantic…