Music Transformer

Abstract

Music relies heavily on repetition to build structure and meaning. Self-reference occurs on multiple timescales, from motifs to phrases to reusing of entire sections of music, such as in pieces with ABA structure. The Transformer (Vaswani et al., 2017), a sequence model based on self-attention, has achieved compelling results in many generation tasks that require maintaining long-range coherence. This suggests that self-attention might also be well-suited to modeling music. In musical composition and performance, however, relative timing is critically important. Existing approaches for representing relative positional information in the Transformer modulate attention based on pairwise distance (Shaw et al., 2018). This is impractical for long sequences such as musical compositions since their memory complexity for intermediate relative information is quadratic in the sequence length. We propose an algorithm that reduces their intermediate memory requirement to linear in the sequence length. This enables us to demonstrate that a Transformer with our modified relative attention mechanism can generate minute-long compositions (thousands of steps, four times the length modeled in Oore et al., 2018) with compelling structure, generate continuations that coherently elaborate on a given motif, and in a seq2seq setup generate accompaniments conditioned on melodies. We evaluate the Transformer with our relative attention mechanism on two datasets, JSB Chorales and Piano-e-Competition, and obtain state-of-the-art results on the latter.

arXiv commentImproved skewing section and accompanying figures. Previous titles are "An Improved Relative Self-Attention Mechanism for Transformer with Application to Music Generation" and "Music Transformer"

Cited by in corpus (12)

• Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting
• The AI Ghostwriter Effect: When Users Do Not Perceive Ownership of AI-Generated Text But Self-Declare as Authors
• Copyright in Generative Deep Learning
• Creativity and Machine Learning: A Survey
• The Ethical Implications of Generative Audio Models: A Systematic Literature Review
• Representation learning for neural population activity with Neural Data Transformers
• Learning to Traverse Latent Spaces for Musical Score Inpainting
• Score and Lyrics-Free Singing Voice Generation
• Dual Learning Music Composition and Dance Choreography
• Distinguishing a planetary transit from false positives: a Transformer-based classification for planetary transit signals
• The Beauty of Repetition in Machine Composition Scenarios
• SMYRF: Efficient Attention using Asymmetric Clustering