32 citations · 97 across the 27 of their papers we have counts for
27 papers
MergeDNA: Context-aware Genome Modeling with Dynamic Tokenization through Token Merging
Siyuan Li, Kai Yu, Anna Wang +7
Modeling genomic sequences faces two unsolved challenges: the information density varies widely across different regions, while there is no clearly defined minimum vocabulary unit.…
TrinityDNA: A Bio-Inspired Foundational Model for Efficient Long-Sequence DNA Modeling
Qirong Yang, Yucheng Guo, Zicheng Liu +7
The modeling of genomic sequences presents unique challenges due to their length and structural complexity. Traditional sequence models struggle to capture long-range dependencies…
Taming LLMs by Scaling Learning Rates with Gradient Grouping
Siyuan Li, Juanxi Tian, Zedong Wang +4
Training large language models (LLMs) poses challenges due to their massive scale and heterogeneous architectures. While adaptive optimizers like AdamW help address gradient variat…
MergeVQ: A Unified Framework for Visual Generation and Representation with Disentangled Token Merging and Quantization
Siyuan Li, Luyuan Zhang, Zedong Wang +8
Masked Image Modeling (MIM) with Vector Quantization (VQ) has achieved great success in both self-supervised pre-training and image generation. However, most existing methods strug…
Life-Code: Central Dogma Modeling with Multi-Omics Sequence Unification
Zicheng Liu, Siyuan Li, Zhiyuan Chen +6
The interactions between DNA, RNA, and proteins are fundamental to biological processes, as illustrated by the central dogma of molecular biology. Although modern biological pre-tr…
Artificial Intelligence for Central Dogma-Centric Multi-Omics: Challenges and Breakthroughs
Lei Xin, Caiyun Huang, Hao Li +8
With the rapid development of high-throughput sequencing platforms, an increasing number of omics technologies, such as genomics, metabolomics, and transcriptomics, are being appli…
Unveiling the Backbone-Optimizer Coupling Bias in Visual Representation Learning
Siyuan Li, Juanxi Tian, Zedong Wang +6
This paper delves into the interplay between vision backbones and optimizers, unvealing an inter-dependent phenomenon termed \textit{\textbf{b}ackbone-\textbf{o}ptimizer \textbf{c}…
A Survey on Mixup Augmentations and Beyond
Xin Jin, Hongyu Zhu, Siyuan Li +6
As Deep Neural Networks have achieved thrilling breakthroughs in the past decade, data augmentations have garnered increasing attention as regularization techniques when massive la…
Short-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequences
Zicheng Liu, Siyuan Li, Li Wang +3
To mitigate the computational complexity in the self-attention mechanism on long sequences, linear attention utilizes computation tricks to achieve linear complexity, while state s…
Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactions
Cheng Tan, Dongxin Lyu, Siyuan Li +5
Large Language Models (LLMs) have demonstrated wide-ranging applications across various fields and have shown significant potential in the academic peer-review process. However, ex…
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation Models
Zicheng Liu, Jiahui Li, Siyuan Li +5
The Genomic Foundation Model (GFM) paradigm is expected to facilitate the extraction of generalizable representations from massive genomic data, thereby enabling their application…
VQDNA: Unleashing the Power of Vector Quantization for Multi-Species Genomic Sequence Modeling
Siyuan Li, Zedong Wang, Zicheng Liu +5
Similar to natural language models, pre-trained genome language models are proposed to capture the underlying intricacies within genomes with unsupervised sequence modeling. They h…
CBGBench: Fill in the Blank of Protein-Molecule Complex Binding Graph
Haitao Lin, Guojiang Zhao, Odin Zhang +7
Structure-based drug design (SBDD) aims to generate potential drugs that can bind to a target protein and is greatly expedited by the aid of AI techniques in generative models. How…
PPFlow: Target-aware Peptide Design with Torsional Flow Matching
Haitao Lin, Odin Zhang, Huifeng Zhao +5
Therapeutic peptides have proven to have great pharmaceutical value and potential in recent decades. However, methods of AI-assisted peptide drug discovery are not fully explored.…
LongVQ: Long Sequence Modeling with Vector Quantization on Structured Memory
Zicheng Liu, Li Wang, Siyuan Li +3
Transformer models have been successful in various sequence processing tasks, but the self-attention mechanism's computational cost limits its practicality for long sequences. Alth…
Advances of Deep Learning in Protein Science: A Comprehensive Survey
Bozhen Hu, Cheng Tan, Lirong Wu +7
Protein representation learning plays a crucial role in understanding the structure and function of proteins, which are essential biomolecules involved in various biological proces…
PSC-CPI: Multi-Scale Protein Sequence-Structure Contrasting for Efficient and Generalizable Compound-Protein Interaction Prediction
Lirong Wu, Yufei Huang, Cheng Tan +5
Compound-Protein Interaction (CPI) prediction aims to predict the pattern and strength of compound-protein interactions for rational drug discovery. Existing deep learning-based me…
Switch EMA: A Free Lunch for Better Flatness and Sharpness
Siyuan Li, Zicheng Liu, Juanxi Tian +9
Exponential Moving Average (EMA) is a widely used weight averaging (WA) regularization to learn flat optima for better generalizations without extra cost in deep neural network (DN…
Masked Modeling for Self-supervised Representation Learning on Vision and Beyond
Siyuan Li, Luyuan Zhang, Zedong Wang +8
As the deep learning revolution marches on, self-supervised learning has garnered increasing attention in recent years thanks to its remarkable representation learning ability and…
SemiReward: A General Reward Model for Semi-supervised Learning
Siyuan Li, Weiyang Jin, Zedong Wang +4
Semi-supervised learning (SSL) has witnessed great progress with various improvements in the self-training framework with pseudo labeling. The main challenge is how to distinguish…
OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive Learning
Cheng Tan, Siyuan Li, Zhangyang Gao +5
Spatio-temporal predictive learning is a learning paradigm that enables models to learn spatial and temporal patterns by predicting future frames from given past frames in an unsup…
Towards Reasonable Budget Allocation in Untargeted Graph Structure Attacks via Gradient Debias
Zihan Liu, Yun Luo, Lirong Wu +2
It has become cognitive inertia to employ cross-entropy loss function in classification related tasks. In the untargeted attacks on graph structure, the gradients derived from the…
RDesign: Hierarchical Data-efficient Representation Learning for Tertiary Structure-based RNA Design
Cheng Tan, Yijie Zhang, Zhangyang Gao +4
While artificial intelligence has made remarkable strides in revealing the relationship between biological macromolecules' primary sequence and tertiary structure, designing RNA se…
Are Gradients on Graph Structure Reliable in Gray-box Attacks?
Zihan Liu, Yun Luo, Lirong Wu +3
Graph edge perturbations are dedicated to damaging the prediction of graph neural networks by modifying the graph structure. Previous gray-box attackers employ gradients from the s…