KTH ▸ Stockholm ▸ 2024

Tradi-fusion

Riffusion fine-tuned on Irish traditional music

Tradi-fusion is my master's thesis at KTH, and Bob L. T. Sturm was my supervisor. Riffusion is a version of the Stable Diffusion image model that its authors fine-tuned to make spectrograms. A spectrogram is an image of a sound that shows how loud each pitch is over time. I fine-tuned Riffusion on about 9,000 spectrograms from recordings of Irish traditional music, and then I used it to make jigs, reels and airs.

Click any spectrogram to hear it.

▸ 00A generated set

▸ W02 7 clips, seed 7, 0:36 Jig → reel → air → hornpipe

The set uses four prompts, one for each kind of tune, and the clip between each pair of prompts mixes the two prompts.

0:00 / 0:36

▸ 01Text to audio

The text prompt changes the music only a little, because every training image had the same caption, An Irish Traditional Tune. Each clip is five seconds long and comes from one prompt and one random seed, a number that sets the random noise. The model starts each clip from a real spectrogram in the dataset, and the starting spectrogram has a bigger effect on how the clip sounds.

▸ T01 seed 42 An Irish traditional tune
▸ T02 seed 7 A lively Irish traditional jig with a prominent fiddle lead
▸ T03 seed 11 A celebration of Irish traditional music with upbeat fiddle tunes
▸ T04 seed 21 A slow Irish air on uilleann pipes
▸ T05 seed 33 An Irish reel on tin whistle
▸ T06 seed 5 An Irish hornpipe on button accordion
▸ T07 seed 99 A pub session with fiddle, wooden flute and bodhrán
▸ T08 seed 13 A mournful violin piece
▸ T09 seed 17 A polka on concertina
▸ T10 seed 23 A gentle slow air on Irish harp
▸ T11 seed 29 Tenor banjo playing a fast Irish reel
▸ T12 seed 31 A slip jig on wooden flute

▸ 02Between two prompts

Each strip starts with one prompt and ends with another one, and the clips in between mix the two prompts. The seed and the starting spectrogram stay the same in every clip. The number α under each clip shows how much of the second prompt is in the mix, from 0 to 1. More steps make the change smoother, but the music also repeats more.

▸ I01 10 steps, seed 42, 0:51 An Irish traditional tune → An Irish tune with acoustic fiddle lead

0:00 / 0:51

▸ I02 5 steps, seed 21, 0:26 A slow Irish air on uilleann pipes → A lively Irish reel on tin whistle

0:00 / 0:26

▸ I03 5 steps, seed 13, 0:26 An Irish traditional tune → A mournful violin piece

0:00 / 0:26

▸ I04 5 steps, seed 64, 0:26 An Irish jig on fiddle → A Swedish folk polska on nyckelharpa

0:00 / 0:26

▸ I05 5 steps, seed 5, 0:26 An Irish traditional tune → Lively tin whistle and hand drum rhythm

0:00 / 0:26

▸ I06 5 steps, seed 8, 0:26 An Irish traditional tune → An Irish traditional tune featuring a solo uilleann pipes melody

0:00 / 0:26

▸ 03Changing the seed

The prompt stays the same in every clip, and only the random seed changes, in small steps between three seeds. Riffusion uses the same method to make music that plays without stopping.

▸ W01 5 clips, 0:26 An Irish traditional tune

0:00 / 0:26

▸ 04What the settings do

Every clip in the section uses the same prompt and the same seed, and each row changes one setting. The other sections on the page use a denoising value of 0.75 and a guidance value of 7.

Denoising: how much the model changes the starting spectrogram

The starting spectrogram, made from a real recording in the dataset
▸ START from the datasetThe starting spectrogram
▸ S030 denoising 0.30 An Irish traditional tune
▸ S050 denoising 0.50 An Irish traditional tune
▸ S075 denoising 0.75 An Irish traditional tune
▸ S095 denoising 0.95 An Irish traditional tune

Guidance: how closely the model follows the prompt

▸ G01 guidance 1 An Irish traditional tune
▸ G03 guidance 3 An Irish traditional tune
▸ G07 guidance 7 An Irish traditional tune
▸ G12 guidance 12 An Irish traditional tune

Starting spectrogram: four recordings from the dataset

▸ M1 start A An Irish traditional tune
▸ M2 start B An Irish traditional tune
▸ M3 start C An Irish traditional tune
▸ M4 start D An Irish traditional tune

▸ 05How it works

  1. S1

    Cut the recordings

    I cut about 1,700 recordings from Bob L. T. Sturm's collection into pieces that are five seconds long.

  2. S2

    Make spectrograms

    I turned each piece into a mel spectrogram of 512 by 512 pixels, which is a spectrogram with its pitch scale matched to human hearing.

  3. S3

    Add captions

    I gave every spectrogram the same caption, An Irish Traditional Tune.

  4. S4

    Fine-tune the model

    I fine-tuned Riffusion on the pairs of spectrograms and captions for about four days on two GPUs.

  5. S5

    Generate new clips

    To make a new clip, I add noise to a starting spectrogram from the dataset, and the model removes the noise step by step with the prompt as a guide.

  6. S6

    Turn spectrograms into sound

    The Griffin-Lim algorithm turns each spectrogram back into sound, and I join the five-second clips end to end.

Training pairs
About 9,000
Clip
512 × 512 pixels, 5.12 s
Training
About 122,000 steps, 4 days

▸ 06What it can't do

  • The model doesn't make whole tunes. A reel or a jig has an A part and a B part, and each part is played twice. The model only makes five-second clips, so a longer piece on the page is several clips joined end to end.
  • The sound is a little metallic. A spectrogram doesn't store the phase of the sound waves, so the Griffin-Lim algorithm has to estimate it. A neural vocoder such as HiFi-GAN, which is a network trained to turn spectrograms into sound, would give cleaner audio.
  • The prompt has a small effect. All 9,000 training images had the same caption, so the model never learned to link words such as banjo or harp to a sound.
  • You can hear where the clips are joined. Each clip fades in and fades out before it's joined to the next clip.

▸ ¶Cite

@mastersthesis{parmar2024tradifusion,
  author  = {Parmar, Harshdeepsinh},
  title   = {Tradi-fusion Refined: Evaluating and Fine-tuning
             the Riffusion Model for Irish Traditional Music},
  school  = {KTH Royal Institute of Technology},
  address = {Stockholm, Sweden},
  year    = {2024}
}