4 papers
Kairos: Toward a Byte-Level Foundation Model for Every Modality
Fabien Furfaro
Small language models trained on a single local machine leave little room for architectural trial and error. Kairos combines bidirectional gated-DeltaNet linear attention and slidi…
TPTT: Transforming Pretrained Transformers into Titans
Fabien Furfaro
Transformer-based large language models (LLMs) have achieved strong performance across many natural language processing tasks. Nonetheless, their quadratic computational and memory…
PixelBytes: Catching Unified Embedding for Multimodal Generation
Fabien Furfaro
This report introduces PixelBytes Embedding, a novel approach for unified multimodal representation learning. Our method captures diverse inputs in a single, cohesive representatio…
PixelBytes: Catching Unified Representation for Multimodal Generation
Fabien Furfaro
This report presents PixelBytes, an approach for unified multimodal representation learning. Drawing inspiration from sequence models like Image Transformers, PixelCNN, and Mamba-B…