13 Music Generation with MuseGAN

This chapter shows how to generate multi-track music by treating a piece as a multi-dimensional object and applying GAN techniques similar to those used for images. Music is represented as piano-roll tensors with the shape (4, 2, 16, 84): four tracks (e.g., SATB), two bars per track, 16 time steps per bar, and 84 possible pitches encoded in one-hot form centered at zero. Using chorales by Johann Sebastian Bach as training data, the chapter explains the essentials of digital music (notes, octaves, pitch numbers, MIDI, and piano rolls) and demonstrates how to convert between MIDI and the tensor representation so that model inputs and outputs can be listened to.

The core MuseGAN design introduces four independent latent codes—chords, style, melody, and groove—to give structured control over harmony, timbre/character, track-specific content, and rhythmic feel. A temporal network expands selected codes across bars so that chords vary by bar and melody varies by track and bar, while style remains global and groove remains track-specific. The generator builds music one bar per track via a BarGenerator that uses transposed convolutions, then combines eight bar/track outputs into a full piece, whereas the critic uses 3D convolutions to score entire (4, 2, 16, 84) tensors with a real-valued rating.

Training follows the WGAN with gradient penalty framework for stability. The critic is optimized to assign high scores to real samples and low scores to generated ones, with a gradient penalty computed on interpolations; the generator is optimized to maximize the critic’s score on its outputs. Training alternates multiple critic steps with one generator step using Adam. After convergence, the critic is discarded and the trained generator produces Bach-like multi-track pieces from sampled latent codes. Multiple generated tensors can be concatenated in time and converted back to MIDI, enabling longer, controllable compositions where chords, style, melody, and groove can be manipulated independently.

The relationship between musical notes, octaves, and pitches (also known as note numbers). The first column displays the eleven octaves (ranging from -1 to 9), representing different levels of musical sound. Each octave is subdivided into 12 semitones, which are listed in the top row: C, C#, D, D#, ..., B. Within each octave, each note is assigned a specific pitch number, ranging from 0 to 127, as indicated in the figure.

Staff notation for a piece of music in JSB Chorales dataset. The music has four tracks, representing the four voices in a chorale: soprano, alto, tenor, and bass. The notation is structured into two bars for each track, with the left and right halves representing the first and second bars, respectively. Each bar consists of 16 steps, aligning with the 4/4 time signature where a bar contains four beats, each subdivided into four sixteenth notes. A total of 84 different pitches are possible, and each note is represented as a one-hot vector with 84 values.

The piano roll for a piece of music. The piano roll is a graphical representation of a musical piece, depicted as a grid with time progressing horizontally from left to right, and pitch represented vertically from bottom to top. Each row on the grid corresponds to a distinct musical note, arranged in a manner akin to the keyboard of a piano, with higher notes positioned at the top and lower notes at the bottom. This specific piece of music comprises two bars, resulting in two distinct sections visible in the graph. The vertical placement of a note block signifies its pitch, while its horizontal location indicates when the note is played in the piece. Additionally, the length of the note block reflects the duration for which the note is sustained.

How to represent a piece of music using a four-dimensional object. In our training data, each piece of music is represented by a four-dimensional object in the shape of (4, 2, 16, 84). The first dimension represents the four music tracks, which are the four voices in the music (soprano, alto, tenor, and bass). Each music track is divided into two bars. There are four beats in each bar and each beat has four notes; we therefore have 16 notes in a bar. Finally, each note is represented by a one-hot variable with 84 values, with -1 everywhere and 1 in one place.

Music generation using chords, style, melody, and groove. Each music composition consists of four tracks and spans two bars. We will extract four noise vectors from the latent space for this purpose. The first vector, representing chords, has a dimension of (1, 32). This vector will be processed through a temporal network to expand the chords into two (1, 32) vectors, corresponding to the two bars, with identical values across all tracks. The second vector, denoting style, also has a dimension of (1, 32) and remains constant across all tracks and bars. The third vector, melody, is shaped as (4, 32). It will be stretched through a temporal network into two (4, 32) vectors, resulting in eight (1, 32) vectors, each representing a unique track and bar combination. Lastly, the fourth vector, groove, with a dimension of (4, 32), will be applied to the four tracks, maintaining the same values for both bars.

A diagram of the steps involved in training MuseGAN to generate music. The generator produces a fake music piece by drawing 4 random noise vectors from the latent space (top left) and presents it to the critic (middle). The critic evaluates the piece and assigns a rating. A high rating suggests that the piece is likely from the training dataset, while a lower rating indicates that the piece is likely fake (generated by the generator). Additionally, an interpolated music piece created from a mix of real and fake samples (top left) is presented to the critic. The training process incorporates a gradient penalty based on the critic's rating of this interpolated piece, which is added to the total loss. The ratings are then compared to the ground truth, allowing both the critic and the generator to learn from these evaluations. After numerous training iterations, the generator becomes proficient at producing music pieces that are virtually indistinguishable from real samples.

• MuseGAN treats a piece of music as a multi-dimensional object akin to an image. The generator produces a piece of music and submits it, along with real music pieces from the training set, to the critic for evaluation. The generator then modifies the music based on the critic's feedback until it closely resembles real music from the training dataset.
• Musical notes, octaves, and pitch are fundamental concepts in music theory. Octaves represent different levels of musical sound. Each octave is subdivided into 12 semitones: C, C#, D, D#, E, F, F#, G, G#, A, A#, B. Within an octave, a note is assigned a specific pitch number.
• In electronic music production, a track typically refers to an individual layer or component of the music. Each track contains multiple bars (or measures). A bar is further divided into multiple steps.
• To represent a piece of music as a multi-dimensional object, we structure it with a (4, 2, 16, 84) shape: four music tracks, with each track consisting of two bars, each bar containing 16 steps, and each step capable of playing one of the 84 different notes.
• In music creation, incorporating more detailed inputs is essential for achieving greater control and variety. Instead of using a single noise vector from the latent space for generating shapes, numbers, and images as in previous chapters, we employ four distinct noise vectors in the music generation process. Given that each music piece consists of four tracks and two bars, we use these four vectors to effectively manage this structure. One vector controls all tracks and bars collectively, another controls each bar across all tracks, a third oversees all tracks across bars, and the fourth manages each individual bar in each track.

FAQ