773 citations · 1.2k across the 18 of their papers we have counts for
18 papers
oHC: Orthogonal Hyper-Connections on SO(4) via Quaternions
Haoqiang Guo, Xuyi Chen, Bo Ke +5
Hyper-Connections (HC) replace the single residual stream of a Transformer with parallel ones, mixing them at every layer with a learned residual matrix. Leaving t…
Eureka-Audio: Triggering Audio Intelligence in Compact Language Models
Dan Zhang, Yishu Lei, Jing Hu +10
We present Eureka-Audio, a compact yet high-performance audio language model that achieves competitive performance against models that are 4 to 18 times larger across a broad range…
ERNIE 5.0 Technical Report
Haifeng Wang, Hua Wu, Tian Wu +432
In this report, we introduce ERNIE 5.0, a natively autoregressive foundation model desinged for unified multimodal understanding and generation across text, image, video, and audio…
CORD: Bridging the Audio-Text Reasoning Gap via Weighted On-policy Cross-modal Distillation
Jing Hu, Danxiang Zhu, Xianlong Luo +9
Large Audio Language Models (LALMs) have garnered significant research interest. Despite being built upon text-based large language models (LLMs), LALMs frequently exhibit a degrad…
MoE Adapter for Large Audio Language Models: Sparsity, Disentanglement, and Gradient-Conflict-Free
Yishu Lei, Shuwei He, Jing Hu +9
Extending the input modality of Large Language Models~(LLMs) to the audio domain is essential for achieving comprehensive multimodal perception. However, it is well-known that acou…
MatryoshkaThinking: Recursive Test-Time Scaling Enables Efficient Reasoning
Hongwei Chen, Yishu Lei, Dan Zhang +10
Test-time scaling has emerged as a promising paradigm in language modeling, wherein additional computational resources are allocated during inference to enhance model performance.…
WellDunn: On the Robustness and Explainability of Language Models and Large Language Models in Identifying Wellness Dimensions
Seyedali Mohammadi, Edward Raff, Jinendra Malekar +3
Language Models (LMs) are being proposed for mental health applications where the heightened risk of adverse outcomes means predictive performance may not be a sufficient litmus te…
Layout-aware Webpage Quality Assessment
Anfeng Cheng, Yiding Liu, Weibin Li +6
Identifying high-quality webpages is fundamental for real-world search engines, which can fulfil users' information need with the less cognitive burden. Early studies of \emph{webp…
ERNIE-GeoL: A Geography-and-Language Pre-trained Model and its Applications in Baidu Maps
Jizhou Huang, Haifeng Wang, Yibo Sun +4
Pre-trained models (PTMs) have become a fundamental backbone for downstream tasks in natural language processing and computer vision. Despite initial gains that were obtained by ap…
Graph4Rec: A Universal Toolkit with Graph Neural Networks for Recommender Systems
Weibin Li, Mingkai He, Zhengjie Huang +4
In recent years, owing to the outstanding performance in graph representation learning, graph neural network (GNN) techniques have gained considerable interests in many real-world…
ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
Yu Sun, Shuohuan Wang, Shikun Feng +19
Pre-trained models have achieved state-of-the-art results in various Natural Language Processing (NLP) tasks. Recent works such as T5 and GPT-3 have shown that scaling up pre-train…
NOTE: Solution for KDD-CUP 2021 WikiKG90M-LSC
Weiyue Su, Zeyang Fang, Hui Zhong +6
WikiKG90M in KDD Cup 2021 is a large encyclopedic knowledge graph, which could benefit various downstream applications such as question answering and recommender systems. Participa…
kk2018 at SemEval-2020 Task 9: Adversarial Training for Code-Mixing Sentiment Classification
Jiaxiang Liu, Xuyi Chen, Shikun Feng +5
Code switching is a linguistic phenomenon that may occur within a multilingual setting where speakers share more than one language. With the increasing communication between groups…
ERNIE at SemEval-2020 Task 10: Learning Word Emphasis Selection by Pre-trained Language Model
Zhengjie Huang, Shikun Feng, Weiyue Su +5
This paper describes the system designed by ERNIE Team which achieved the first place in SemEval-2020 Task 10: Emphasis Selection For Written Text in Visual Media. Given a sentence…
Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classification
Yunsheng Shi, Zhengjie Huang, Shikun Feng +3
Graph neural network (GNN) and label propagation algorithm (LPA) are both message passing algorithms, which have achieved superior performance in semi-supervised classification. GN…
ERNIE 2.0: A Continual Pre-training Framework for Language Understanding
Yu Sun, Shuohuan Wang, Yukun Li +4
Recently, pre-trained models have achieved state-of-the-art results in various language understanding tasks, which indicates that pre-training on large-scale corpora may play a cru…
ERNIE: Enhanced Representation through Knowledge Integration
Yu Sun, Shuohuan Wang, Yukun Li +7
We present a novel language representation model enhanced by knowledge called ERNIE (Enhanced Representation through kNowledge IntEgration). Inspired by the masking strategy of BER…
Functional Hashing for Compressing Neural Networks
Lei Shi, Shikun Feng, Zhifan Zhu
As the complexity of deep neural networks (DNNs) trend to grow to absorb the increasing sizes of data, memory and energy consumption has been receiving more and more attentions for…