LLaMA: Open and Efficient Foundation Language Models

Abstract

We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B. We release all our models to the research community.

Cited by in corpus (269)

• A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
• Unifying Large Language Models and Knowledge Graphs: A Roadmap
• DINOv2: Learning Robust Visual Features without Supervision
• A Survey on Multimodal Large Language Models
• Graph of Thoughts: Solving Elaborate Problems with Large Language Models
• Large language models in medicine: the potentials and pitfalls
• Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineering
• Worldwide AI Ethics: a review of 200 guidelines and recommendations for AI governance
• BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks
• Large AI Models in Health Informatics: Applications, Challenges, and the Future
• Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review
• 14 Examples of How LLMs Can Transform Materials Science and Chemistry: A Reflection on a Large Language Model Hackathon
• Bad Actor, Good Advisor: Exploring the Role of Large Language Models in Fake News Detection
• AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
• Factuality Challenges in the Era of Large Language Models
• Automated data processing and feature engineering for deep learning and big data applications: a survey
• ChatEDA: A Large Language Model Powered Autonomous Agent for EDA
• Large Language Models as Zero-Shot Conversational Recommenders
• Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Review
• Taking the Next Step with Generative Artificial Intelligence: The Transformative Role of Multimodal Large Language Models in Science Education
• Towards Interpretable Mental Health Analysis with Large Language Models
• Leveraging Large Language Models for Sequential Recommendation
• OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models
• Machine Culture
• NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencing
• Generative AI for Synthetic Data Across Multiple Medical Modalities: A Systematic Review of Recent Developments and Challenges
• MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots
• Opening up ChatGPT: Tracking openness, transparency, and accountability in instruction-tuned text generators
• ChartGPT: Leveraging LLMs to Generate Charts from Abstract Natural Language
• Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!
• RiNALMo: General-Purpose RNA Language Models Can Generalize Well on Structure Prediction Tasks
• Fish Tracking, Counting, and Behaviour Analysis in Digital Aquaculture: A Comprehensive Survey
• A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine
• Enhancing the LLM-Based Robot Manipulation Through Human-Robot Collaboration
• From task structures to world models: What do LLMs know?
• General Purpose Artificial Intelligence Systems (GPAIS): Properties, Definition, Taxonomy, Societal Implications and Responsible Governance
• TimeGPT in Load Forecasting: A Large Time Series Model Perspective
• Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inference
• UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobility
• LLM-based policy generation for intent-based management of applications
• Using LLMs in Software Requirements Specifications: An Empirical Evaluation
• Enhancing Phenotype Recognition in Clinical Notes Using Large Language Models: PhenoBCBERT and PhenoGPT
• ChatCAD+: Towards a Universal and Reliable Interactive CAD using LLMs
• Learning from models beyond fine-tuning
• Exploring ChatGPT and its Impact on Society
• Towards a Holistic Framework for Multimodal Large Language Models in Three-dimensional Brain CT Report Generation
• Refactoring Programs Using Large Language Models with Few-Shot Examples
• LLM4PLC: Harnessing Large Language Models for Verifiable Programming of PLCs in Industrial Control Systems
• Empowering Molecule Discovery for Molecule-Caption Translation with Large Language Models: A ChatGPT Perspective
• Generative AI and Process Systems Engineering: The Next Frontier
• Neural Natural Language Processing for Long Texts: A Survey on Classification and Summarization
• Emotion Detection for Misinformation: A Review
• Unreflected Acceptance -- Investigating the Negative Consequences of ChatGPT-Assisted Problem Solving in Physics Education
• AppPoet: Large Language Model based Android malware detection via multi-view prompt engineering
• Measure-Observe-Remeasure: An Interactive Paradigm for Differentially-Private Exploratory Analysis
• HPC-GPT: Integrating Large Language Model for High-Performance Computing
• How Good is ChatGPT at Face Biometrics? A First Look into Recognition, Soft Biometrics, and Explainability
• Evaluation of LLM Chatbots for OSINT-based Cyber Threat Awareness
• When Geoscience Meets Generative AI and Large Language Models: Foundations, Trends, and Future Challenges
• Development of Skip Connection in Deep Neural Networks for Computer Vision and Medical Image Analysis: A Survey
• LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain
• Multi-task Paired Masking with Alignment Modeling for Medical Vision-Language Pre-training
• Embedding Large Language Models into Extended Reality: Opportunities and Challenges for Inclusion, Engagement, and Privacy
• LLM-FP4: 4-Bit Floating-Point Quantized Transformers
• Contextual Feature Extraction Hierarchies Converge in Large Language Models and the Brain
• LLM4SecHW: Leveraging Domain Specific Large Language Model for Hardware Debugging
• Generative Artificial Intelligence Meets Synthetic Aperture Radar: A Survey
• Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Models
• A Survey on Symbolic Knowledge Distillation of Large Language Models
• A Survey on Occupancy Perception for Autonomous Driving: The Information Fusion Perspective
• FoodSAM: Any Food Segmentation
• Leveraging Large Language Models for Patient Engagement: The Power of Conversational AI in Digital Health
• Quo Vadis ChatGPT? From Large Language Models to Large Knowledge Models
• FuzzLLM: A Novel and Universal Fuzzing Framework for Proactively Discovering Jailbreak Vulnerabilities in Large Language Models
• Achieving Peak Performance for Large Language Models: A Systematic Review
• Hallucination Detection in Foundation Models for Decision-Making: A Flexible Definition and Review of the State of the Art
• Developing ChemDFM as a large language foundation model for chemistry
• Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark
• Examining Inter-Consistency of Large Language Models Collaboration: An In-depth Analysis via Debate
• Detecting The Corruption Of Online Questionnaires By Artificial Intelligence
• Incremental Learning of Humanoid Robot Behavior from Natural Interaction and Large Language Models
• From Screens to Scenes: A Survey of Embodied AI in Healthcare
• Flexible, Model-Agnostic Method for Materials Data Extraction from Text Using General Purpose Language Models
• Large language models as oracles for instantiating ontologies with domain-specific knowledge
• An Evaluation on Large Language Model Outputs: Discourse and Memorization
• WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia
• Trajectory Prediction for Autonomous Driving: Progress, Limitations, and Future Directions
• Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition
• Large-Scale Text Analysis Using Generative Language Models: A Case Study in Discovering Public Value Expressions in AI Patents
• IQAGPT: Image Quality Assessment with Vision-language and ChatGPT Models
• Clinical Insights: A Comprehensive Review of Language Models in Medicine
• Still No Lie Detector for Language Models: Probing Empirical and Conceptual Roadblocks
• Automatic Personalized Impression Generation for PET Reports Using Large Language Models
• Potential Societal Biases of ChatGPT in Higher Education: A Scoping Review
• Applications of Generative AI in Healthcare: algorithmic, ethical, legal and societal considerations
• Higher education assessment practice in the era of generative AI tools
• Two Directions for Clinical Data Generation with Large Language Models: Data-to-Label and Label-to-Data
• Opportunities for Large Language Models and Discourse in Engineering Design
• Large language models for automated scholarly paper review: A survey
• MCI-GRU: Stock Prediction Model Based on Multi-Head Cross-Attention and Improved GRU
• Building Privacy-Preserving and Secure Geospatial Artificial Intelligence Foundation Models
• On Explaining Recommendations with Large Language Models: A Review
• Computational Politeness in Natural Language Processing: A Survey
• PromptMTopic: Unsupervised Multimodal Topic Modeling of Memes using Large Language Models
• An Unified Search and Recommendation Foundation Model for Cold-Start Scenario
• Think Fast, Think Slow, Think Critical: Designing an Automated Propaganda Detection Tool
• OpenECAD: An Efficient Visual Language Model for Editable 3D-CAD Design
• The VoxCeleb Speaker Recognition Challenge: A Retrospective
• A Comprehensive Review of State-of-The-Art Methods for Java Code Generation from Natural Language Text
• A Reliable Knowledge Processing Framework for Combustion Science using Foundation Models
• Will Code Remain a Relevant User Interface for End-User Programming with Generative AI Models?
• Token Prediction as Implicit Classification to Identify LLM-Generated Text
• Recent Advances in Named Entity Recognition: A Comprehensive Survey and Comparative Study
• Transition Role of Entangled Data in Quantum Machine Learning
• Joint Estimation and Prediction of City-wide Delivery Demand: A Large Language Model Empowered Graph-based Learning Approach
• Transfer Learning with Foundational Models for Time Series Forecasting using Low-Rank Adaptations
• Unsupervised Domain Adaption Harnessing Vision-Language Pre-training
• FabGPT: An Efficient Large Multimodal Model for Complex Wafer Defect Knowledge Queries
• Homogeneous Tokenizer Matters: Homogeneous Visual Tokenizer for Remote Sensing Image Understanding
• Multi-Meta-RAG: Improving RAG for Multi-Hop Queries using Database Filtering with LLM-Extracted Metadata
• Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modeling
• CoVR-2: Automatic Data Construction for Composed Video Retrieval
• Stock Movement Prediction with Multimodal Stable Fusion via Gated Cross-Attention Mechanism
• Breaking reCAPTCHAv2
• LLaVA-Chef: A Multi-modal Generative Model for Food Recipes
• Collaborative Cross-modal Fusion with Large Language Model for Recommendation
• Task-Oriented Semantic Communication in Large Multimodal Models-based Vehicle Networks
• Computational Argumentation-based Chatbots: a Survey
• Prompting Disentangled Embeddings for Knowledge Graph Completion with Pre-trained Language Model
• Multimodal machine learning with large language embedding model for polymer property prediction
• Isolating Compiler Bugs by Generating Effective Witness Programs with Large Language Models
• RSTeller: Scaling Up Visual Language Modeling in Remote Sensing with Rich Linguistic Semantics from Openly Available Data and Large Language Models
• MMSR: Symbolic Regression is a Multi-Modal Information Fusion Task
• Cross-lingual Aspect-Based Sentiment Analysis: A Survey on Tasks, Approaches, and Challenges
• Power-LLaVA: Large Language and Vision Assistant for Power Transmission Line Inspection
• Transforming the Bootstrap: Using Transformers to Compute Scattering Amplitudes in Planar N = 4 Super Yang-Mills Theory
• A Survey on Moral Foundation Theory and Pre-Trained Language Models: Current Advances and Challenges
• Supersonic: Learning to Generate Source Code Optimizations in C/C++
• Health Text Simplification: An Annotated Corpus for Digestive Cancer Education and Novel Strategies for Reinforcement Learning
• Guideline Learning for In-context Information Extraction
• Automating Traffic Model Enhancement with AI Research Agent
• Engineering Artificial Intelligence: Framework, Challenges, and Future Direction
• EvoPath: Evolutionary Meta-path Discovery with Large Language Models for Complex Heterogeneous Information Networks
• Large Language Models for Depression Recognition in Spoken Language Integrating Psychological Knowledge
• A Triumvirate of AI Driven Theoretical Discovery
• GPTSee: Enhancing Moment Retrieval and Highlight Detection via Description-Based Similarity Features
• Zero-shot Bilingual App Reviews Mining with Large Language Models
• Tucano: Advancing Neural Text Generation for Portuguese
• Towards More Human-like AI Communication: A Review of Emergent Communication Research
• Stylometry recognizes human and LLM-generated texts in short samples
• Silent Guardian: Protecting Text from Malicious Exploitation by Large Language Models
• Entity Recognition from Colloquial Text
• Leveraging Large Language Models for Knowledge-free Weak Supervision in Clinical Natural Language Processing
• Reward Guidance for Reinforcement Learning Tasks Based on Large Language Models: The LMGT Framework
• Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models
• Large Language Models to the Rescue: Reducing the Complexity in Scientific Workflow Development Using ChatGPT
• From Handcrafted Features to LLMs: A Brief Survey for Machine Translation Quality Estimation
• Text-to-OverpassQL: A Natural Language Interface for Complex Geodata Querying of OpenStreetMap
• Towards an automated workflow in materials science for combining multi-modal simulative and experimental information using data mining and large language models
• LAMBDA: A Large Model Based Data Agent
• Deanthropomorphising NLP: Can a Language Model Be Conscious?
• Unveiling the Vulnerability of Private Fine-Tuning in Split-Based Frameworks for Large Language Models: A Bidirectionally Enhanced Attack
• A Survey of Large Language Models for Arabic Language and its Dialects
• SC4ANM: Identifying Optimal Section Combinations for Automated Novelty Prediction in Academic Papers
• Exploring the Reversal Curse and Other Deductive Logical Reasoning in BERT and GPT-Based Large Language Models
• Survey and Experiments on Mental Disorder Detection via Social Media: From Large Language Models and RAG to Agents
• Democratizing LLMs: An Exploration of Cost-Performance Trade-offs in Self-Refined Open-Source Models
• Improved particle-flow event reconstruction with scalable neural networks for current and future particle detectors
• Black-box Prompt Tuning with Subspace Learning
• Gender Bias Detection in Court Decisions: A Brazilian Case Study
• Prompting Video-Language Foundation Models with Domain-specific Fine-grained Heuristics for Video Question Answering
• Symbolic Equation Solving via Reinforcement Learning
• Revisiting Block-based Quantisation: What is Important for Sub-8-bit LLM Inference?
• A Multimodal Approach to Device-Directed Speech Detection with Large Language Models
• Large Language Models for Combinatorial Optimization: A Systematic Review
• On the Potential and Limitations of Few-Shot In-Context Learning to Generate Metamorphic Specifications for Tax Preparation Software
• Reviewing Clinical Knowledge in Medical Large Language Models: Training and Beyond
• A Hybrid Intelligence Method for Argument Mining
• Recommender systems and reinforcement learning for human-building interaction and context-aware support: A text mining-driven review of scientific literature
• Paraphrase Types for Generation and Detection
• LLM-Assisted Visual Analytics: Opportunities and Challenges
• A Survey of Spatio-Temporal EEG data Analysis: from Models to Applications
• Enhancing Soft Skills in Network Management Education: A Study on the Impact of GenAI-based Virtual Assistants
• Fully Autonomous Programming using Iterative Multi-Agent Debugging with Large Language Models
• Sentiment Analysis Using Averaged Weighted Word Vector Features
• Structural Entities Extraction and Patient Indications Incorporation for Chest X-ray Report Generation
• Learning the Simplicity of Scattering Amplitudes
• Context-Enhanced Language Models for Generating Multi-Paper Citations
• Vox Populi, Vox AI? Using Language Models to Estimate German Public Opinion
• Large language models in finance : what is financial sentiment?
• Large Language Models as Autonomous Spacecraft Operators in Kerbal Space Program
• Non-autoregressive Personalized Bundle Generation
• TDML -- A Trustworthy Distributed Machine Learning Framework
• MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity
• Mosaic: Composite Projection Pruning for Resource-efficient LLMs
• Efficient Tuning of Large Language Models for Knowledge-Grounded Dialogue Generation
• Developmental Scaffolding with Large Language Models
• PreCog: Exploring the Relation between Memorization and Performance in Pre-trained Language Models
• FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models
• OPT-R: Exploring the Role of Explanations in Finetuning and Prompting for Reasoning Skills of Large Language Models
• TrICy: Trigger-guided Data-to-text Generation with Intent aware Attention-Copy
• Blacks is to Anger as Whites is to Joy? Understanding Latent Affective Bias in Large Pre-trained Neural Language Models
• Flows: Building Blocks of Reasoning and Collaborating AI
• MULTI: Multimodal Understanding Leaderboard with Text and Images
• Exploring the Impact of Corpus Diversity on Financial Pretrained Language Models
• A Canary in the AI Coal Mine: American Jews May Be Disproportionately Harmed by Intellectual Property Dispossession in Large Language Model Training
• HELIOT: LLM-Based CDSS for Adverse Drug Reaction Management
• The Strong Pull of Prior Knowledge in Large Language Models and Its Impact on Emotion Recognition
• CARIn: Constraint-Aware and Responsive Inference on Heterogeneous Devices for Single- and Multi-DNN Workloads
• Molecular Dynamics and Machine Learning Unlock Possibilities in Beauty Design -- A Perspective
• AdditiveLLM: Large Language Models Predict Defects in Additive Manufacturing
• Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Model
• Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talk
• Speaker attribution in German parliamentary debates with QLoRA-adapted large language models
• A Cost-Effective LLM-based Approach to Identify Wildlife Trafficking in Online Marketplaces
• Explicitly Representing Syntax Improves Sentence-to-layout Prediction of Unexpected Situations
• Unsupervised Text Embedding Space Generation Using Generative Adversarial Networks for Text Synthesis
• Target-driven Attack for Large Language Models
• Enhancing Financial Domain Adaptation of Language Models via Model Augmentation
• Integrating Artificial Open Generative Artificial Intelligence into Software Supply Chain Security
• FlanEC: Exploring Flan-T5 for Post-ASR Error Correction
• Harmful Terms and Where to Find Them: Measuring and Modeling Unfavorable Financial Terms and Conditions in Shopping Websites at Scale
• Fine-tuning Strategies for Domain Specific Question Answering under Low Annotation Budget Constraints
• Classification of integers based on residue classes via modern deep learning algorithms
• Deep Insights into Cognitive Decline: A Survey of Leveraging Non-Intrusive Modalities with Deep Learning Techniques
• Perovskite-R1: a domain-specialized large language model for intelligent discovery of precursor additives and experimental design
• Lightweight and Direct Document Relevance Optimization for Generative Information Retrieval
• Revisiting Instruction Fine-tuned Model Evaluation to Guide Industrial Applications
• Are NLP Models Good at Tracing Thoughts: An Overview of Narrative Understanding
• Vision Language Models as Values Detectors
• Fabricator: An Open Source Toolkit for Generating Labeled Training Data with Teacher LLMs
• Generative Meta-Learning for Zero-Shot Relation Triplet Extraction
• Bridging Text and Crystal Structures: Literature-driven Contrastive Learning for Materials Science
• Generative AI Takes a Statistics Exam: A Comparison of Performance between ChatGPT3.5, ChatGPT4, and ChatGPT4o-mini
• BALI: Enhancing Biomedical Language Representations through Knowledge Graph and Language Model Alignment
• Multi-word Tokenization for Sequence Compression
• Vision Language Models for Dynamic Human Activity Recognition in Healthcare Settings
• Bringing Multi-Modal Multi-Task Federated Foundation Models to Education Domain: Prospects and Challenges
• On Bilingual Lexicon Induction with Large Language Models
• MAP-Elites with Transverse Assessment for Multimodal Problems in Creative Domains
• Text Embedding Inversion Security for Multilingual Language Models
• DISCIE -- Discriminative Closed Information Extraction
• Exploring news intent and its application: A theory-driven approach
• Generating clickbait spoilers with an ensemble of large language models
• Argus: Leveraging Multiview Images for Improved 3-D Scene Understanding With Large Language Models
• Wrong Answers Can Also Be Useful: PlausibleQA -- A Large-Scale QA Dataset with Answer Plausibility Scores
• DemoBias: An Empirical Study to Trace Demographic Biases in Vision Foundation Models
• An Overview on Generative AI at Scale with Edge-Cloud Computing
• A Glitch in the Matrix? Locating and Detecting Language Model Grounding with Fakepedia
• Scale-free Characteristics of Multilingual Legal Texts and the Limitations of LLMs
• gym-saturation: Gymnasium environments for saturation provers (System description)
• Transductive Learning for Textual Few-Shot Classification in API-based Embedding Models
• Searching Personal Collections
• Are Compressed Language Models Less Subgroup Robust?
• Benchmarking quantized LLaMa-based models on the Brazilian Secondary School Exam
• A Hierarchical Neural Framework for Classification and its Explanation in Large Unstructured Legal Documents
• CompanyKG: A Large-Scale Heterogeneous Graph for Company Similarity Quantification
• AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving System
• Hallucinations in LLMs: A Lifecycle-Based Survey of Causes, Detection, Mitigation, and Prevention
• Vectorizing string entries for data processing on tables: when are larger language models better?
• Humanoid Artificial Consciousness Designed with Large Language Model Based on Psychoanalysis and Personality Theory
• It Ain't That Bad: Understanding the Mysterious Performance Drop in OOD Generalization for Generative Transformer Models
• Simple and Effective Input Reformulations for Translation
• epiGPTope: A machine learning-based epitope generator and classifier
• ModuLoRA: Finetuning 2-Bit LLMs on Consumer GPUs by Integrating with Modular Quantizers
• Tensorization of neural networks for improved privacy and interpretability
• Tailored-LLaMA: Optimizing Few-Shot Learning in Pruned LLaMA Models with Task-Specific Prompts
• Emergent inabilities? Inverse scaling over the course of pretraining
• BoundingDocs: a Unified Dataset for Document Question Answering with Spatial Annotations