6 citations · 10 across the 17 of their papers we have counts for
6 papers · 1 filter
Soft-NBCE: Entropy-Weighted Chunk Fusion for Long-Context
Shihao Ji, Mingyu Li, Zihui Song
The quadratic complexity of self-attention remains a bottleneck for Large Language Models (LLMs) processing ultra-long contexts. The Naive Bayes Cognitive Engine (NBCE) parallelize…
L-MoE: End-to-End Training of a Lightweight Mixture of Low-Rank Adaptation Experts
Shihao Ji, Zihui Song
The Mixture of Experts (MoE) architecture enables the scaling of Large Language Models (LLMs) to trillions of parameters by activating a sparse subset of weights for each input, ma…
MyGO: Memory Yielding Generative Offline-consolidation for Lifelong Learning Systems
Shihao Ji, Zihui Song
Continual or Lifelong Learning aims to develop models capable of acquiring new knowledge from a sequence of tasks without catastrophically forgetting what has been learned before.…
OpenGrok: Enhancing SNS Data Processing with Distilled Knowledge and Mask-like Mechanisms
Lumen AI, Zaozhuang No. 28 Middle School, Shihao Ji +6
This report details Lumen Labs' novel approach to processing Social Networking Service (SNS) data. We leverage knowledge distillation, specifically a simple distillation method ins…
Chinese Stock Prediction Based on a Multi-Modal Transformer Framework: Macro-Micro Information Fusion
Lumen AI, Tengzhou No. 1 Middle School, Shihao Ji +6
This paper proposes an innovative Multi-Modal Transformer framework (MMF-Trans) designed to significantly improve the prediction accuracy of the Chinese stock market by integrating…
Transformer^-1: Input-Adaptive Computation for Resource-Constrained Deployment
Lumen AI, Tengzhou No. 1 Middle School, Shihao Ji +6
Addressing the resource waste caused by fixed computation paradigms in deep learning models under dynamic scenarios, this paper proposes a Transformer architecture based on…