20 citations · 82 across the 18 of their papers we have counts for
18 papers
Gemma 4 Technical Report
Gemma Team, Sherif El Abd, Vaibhav Aggarwal +320
We introduce Gemma 4, a new generation of open-weight, natively multimodal language models in the Gemma model family. Designed to advance compute efficiency and reasoning, the Gemm…
Boosting LLM via Learning from Data Iteratively and Selectively
Qi Jia, Siyu Ren, Ziheng Qin +3
Datasets nowadays are generally constructed from multiple sources and using different synthetic techniques, making data de-noising and de-duplication crucial before being used for…
MixEval-X: Any-to-Any Evaluations from Real-World Data Mixtures
Jinjie Ni, Yifan Song, Deepanway Ghosal +10
Perceiving and generating diverse modalities are crucial for AI models to effectively learn from and engage with real-world signals, necessitating reliable evaluations for their de…
MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures
Jinjie Ni, Fuzhao Xue, Xiang Yue +5
Evaluating large language models (LLMs) is challenging. Traditional ground-truth-based benchmarks fail to capture the comprehensiveness and nuance of real-world queries, while LLM-…
OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models
Fuzhao Xue, Zian Zheng, Yao Fu +4
To help the open-source community have a better understanding of Mixture-of-Experts (MoE) based large language models (LLMs), we train and release OpenMoE, a series of fully open-s…
Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline
Zangwei Zheng, Xiaozhe Ren, Fuzhao Xue +3
Large language models (LLMs) have revolutionized the field of AI, demonstrating unprecedented capacity across various tasks. However, the inference process for LLMs comes with sign…
To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis
Fuzhao Xue, Yao Fu, Wangchunshu Zhou +2
Recent research has highlighted the importance of dataset size in scaling language models. However, large language models (LLMs) are notoriously token-hungry during pre-training, a…
Hierarchical Dialogue Understanding with Special Tokens and Turn-level Attention
Xiao Liu, Jian Zhang, Heng Zhang +2
Compared with standard text, understanding dialogue is more challenging for machines as the dynamic and unexpected semantic changes in each turn. To model such inconsistent semanti…
Adaptive Computation with Elastic Input Sequence
Fuzhao Xue, Valerii Likhosherstov, Anurag Arnab +3
Humans have the ability to adapt the type of information they use, the procedure they employ, and the amount of time they spend when solving problems. However, most standard neural…
A Study on Transformer Configuration and Training Objective
Fuzhao Xue, Jianghai Chen, Aixin Sun +6
Transformer-based models have delivered impressive results on many tasks, particularly vision and language tasks. In many model training situations, conventional configurations are…
Modeling Motion with Multi-Modal Features for Text-Based Video Segmentation
Wangbo Zhao, Kai Wang, Xiangxiang Chu +3
Text-based video segmentation aims to segment the target object in a video based on a describing sentence. Incorporating motion information from optical flow maps with appearance a…
CowClip: Reducing CTR Prediction Model Training Time from 12 hours to 10 minutes on 1 GPU
Zangwei Zheng, Pengtai Xu, Xuan Zou +12
The click-through rate (CTR) prediction task is to predict whether a user will click on the recommended item. As mind-boggling amounts of data are produced online daily, accelerati…
One Student Knows All Experts Know: From Sparse to Dense
Fuzhao Xue, Xiaoxin He, Xiaozhe Ren +2
Human education system trains one student by multiple experts. Mixture-of-experts (MoE) is a powerful sparse architecture including multiple experts. However, sparse MoE model is e…
Large-Scale Deep Learning Optimizations: A Comprehensive Survey
Xiaoxin He, Fuzhao Xue, Xiaozhe Ren +1
Deep learning have achieved promising results on a wide spectrum of AI applications. Larger datasets and models consistently yield better performance. However, we generally spend l…
Cross-token Modeling with Conditional Computation
Yuxuan Lou, Fuzhao Xue, Zangwei Zheng +1
Mixture-of-Experts (MoE), a conditional computation architecture, achieved promising performance by scaling local module (i.e. feed-forward network) of transformer. However, scalin…
Automated Audio Captioning using Transfer Learning and Reconstruction Latent Space Similarity Regularization
Andrew Koh, Fuzhao Xue, Eng Siong Chng
In this paper, we examine the use of Transfer Learning using Pretrained Audio Neural Networks (PANNs), and propose an architecture that is able to better leverage the acoustic feat…
Go Wider Instead of Deeper
Fuzhao Xue, Ziji Shi, Futao Wei +3
More transformer blocks with residual connections have recently achieved impressive results on various tasks. To achieve better performance with fewer trainable parameters, recent…
Sequence Parallelism: Long Sequence Training from System Perspective
Shenggui Li, Fuzhao Xue, Chaitanya Baranwal +2
Transformer achieves promising results on various tasks. However, self-attention suffers from quadratic memory requirements with respect to the sequence length. Existing work focus…