5 papers
Sign Lock-In: Randomly Initialized Weight Signs Persist and Bottleneck Sub-Bit Model Compression
Akira Sakai, Yuma Ichikawa
Sub-bit model compression targets storage below one bit per weight; as magnitudes are aggressively compressed, the sign bit becomes a fixed-cost bottleneck. Across Transformers, CN…
PHOTON: Hierarchical Autoregressive Modeling for Lightspeed and Memory-Efficient Language Generation
Yuma Ichikawa, Naoya Takagi, Takumi Nakagawa +2
Transformers operate as horizontal token-by-token scanners; at each generation step, attending to an ever-growing sequence of token-level states. This access pattern increases pref…
More Than Bits: Multi-Envelope Double Binary Factorization for Extreme Quantization
Yuma Ichikawa, Yoshihiko Fujisawa, Yudai Fujimoto +2
For extreme low-bit quantization of large language models (LLMs), Double Binary Factorization (DBF) is attractive as it enables efficient inference without sacrificing accuracy. Ho…
LPCD: Unified Framework from Layer-Wise to Submodule Quantization
Yuma Ichikawa, Yudai Fujimoto, Akira Sakai
Post-training quantization (PTQ) aims to preserve model-level behavior; however, most methods focus on individual linear layers. Even recent extensions, such as QEP and LoaQ, which…
Bermuda Triangles: GNNs Fail to Detect Simple Topological Structures
Arseny Tolmachev, Akira Sakai, Masaru Todoriki +1
Most graph neural network architectures work by message-passing node vector embeddings over the adjacency matrix, and it is assumed that they capture graph topology by doing that.…