Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Review

Abstract

This paper provides a comprehensive review of the literature concerning the utilization of Natural Language Processing (NLP) techniques, with a particular focus on transformer-based large language models (LLMs) trained using Big Code, within the domain of AI-assisted programming tasks. LLMs, augmented with software naturalness, have played a crucial role in facilitating AI-assisted programming applications, including code generation, code completion, code translation, code refinement, code summarization, defect detection, and clone detection. Notable examples of such applications include the GitHub Copilot powered by OpenAI's Codex and DeepMind AlphaCode. This paper presents an overview of the major LLMs and their applications in downstream tasks related to AI-assisted programming. Furthermore, it explores the challenges and opportunities associated with incorporating NLP techniques with software naturalness in these applications, with a discussion on extending AI-assisted programming capabilities to Apple's Xcode for mobile software development. This paper also presents the challenges of and opportunities for incorporating NLP techniques with software naturalness, empowering developers with advanced coding assistance and streamlining the software development process.

References in corpus (22)

• Training language models to follow instructions with human feedback
• LLaMA: Open and Efficient Foundation Language Models
• Evaluating Large Language Models Trained on Code
• VulDeePecker: A Deep Learning-Based System for Vulnerability Detection
• Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
• TBar: Revisiting Template-based Automated Program Repair
• The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development
• Elixir: Effective object-oriented program repair
• A Deep Reinforcement Learning Chatbot
• Pythia: AI-assisted Code Completion System
• CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning
• SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representation
• CodeT: Code Generation with Generated Tests
• Exploring Software Naturalness through Neural Language Models
• SantaCoder: don't reach for the stars!
• Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping
• DeepDebug: Fixing Python Bugs Using Stack Traces, Backtranslation, and Code Skeletons
• MetaShift: A Dataset of Datasets for Evaluating Contextual Distribution Shifts and Training Conflicts
• SPT-Code: Sequence-to-Sequence Pre-Training for Learning Source Code Representations
• Deep Learning Meets Software Engineering: A Survey on Pre-Trained Models of Source Code

Cited by in corpus (4)

• Materials science in the era of large language models: a perspective
• The Impact of Generative AI Coding Assistants on Developers Who Are Visually Impaired
• DeepSeek-V3, GPT-4, Phi-4, and LLaMA-3.3 generate correct code for LoRaWAN-related engineering tasks
• Measuring how changes in code readability attributes affect code quality evaluation by Large Language Models