Tradi-fusion
Riffusion fine-tuned on Irish traditional music
Tradi-fusion is my master's thesis at KTH, and Bob L. T. Sturm was my supervisor. Riffusion is a version of the Stable Diffusion image model that its authors fine-tuned to make spectrograms. A spectrogram is an image of a sound that shows how loud each pitch is over time. I fine-tuned Riffusion on about 9,000 spectrograms from recordings of Irish traditional music, and then I used it to make jigs, reels and airs.
Click any spectrogram to hear it.
▸ 00A generated set
The set uses four prompts, one for each kind of tune, and the clip between each pair of prompts mixes the two prompts.
0:00 / 0:36
▸ 01Text to audio
The text prompt changes the music only a little, because every training image had the same
caption, An Irish Traditional Tune
. Each clip is five seconds long and comes from one prompt and one
random seed, a number that sets the random noise. The model starts each clip from a real spectrogram in the
dataset, and the starting spectrogram has a bigger effect on how the clip sounds.
▸ 02Between two prompts
Each strip starts with one prompt and ends with another one, and the clips in between mix the two prompts. The seed and the starting spectrogram stay the same in every clip. The number α under each clip shows how much of the second prompt is in the mix, from 0 to 1. More steps make the change smoother, but the music also repeats more.
0:00 / 0:51
0:00 / 0:26
0:00 / 0:26
0:00 / 0:26
0:00 / 0:26
0:00 / 0:26
▸ 03Changing the seed
The prompt stays the same in every clip, and only the random seed changes, in small steps between three seeds. Riffusion uses the same method to make music that plays without stopping.
0:00 / 0:26
▸ 04What the settings do
Every clip in the section uses the same prompt and the same seed, and each row changes one setting. The other sections on the page use a denoising value of 0.75 and a guidance value of 7.
Denoising: how much the model changes the starting spectrogram

Guidance: how closely the model follows the prompt
Starting spectrogram: four recordings from the dataset
▸ 05How it works
- S1
Cut the recordings
I cut about 1,700 recordings from Bob L. T. Sturm's collection into pieces that are five seconds long.
- S2
Make spectrograms
I turned each piece into a mel spectrogram of 512 by 512 pixels, which is a spectrogram with its pitch scale matched to human hearing.
- S3
Add captions
I gave every spectrogram the same caption,
An Irish Traditional Tune
. - S4
Fine-tune the model
I fine-tuned Riffusion on the pairs of spectrograms and captions for about four days on two GPUs.
- S5
Generate new clips
To make a new clip, I add noise to a starting spectrogram from the dataset, and the model removes the noise step by step with the prompt as a guide.
- S6
Turn spectrograms into sound
The Griffin-Lim algorithm turns each spectrogram back into sound, and I join the five-second clips end to end.
- Training pairs
- About 9,000
- Clip
- 512 × 512 pixels, 5.12 s
- Training
- About 122,000 steps, 4 days
▸ 06What it can't do
- The model doesn't make whole tunes. A reel or a jig has an A part and a B part, and each part is played twice. The model only makes five-second clips, so a longer piece on the page is several clips joined end to end.
- The sound is a little metallic. A spectrogram doesn't store the phase of the sound waves, so the Griffin-Lim algorithm has to estimate it. A neural vocoder such as HiFi-GAN, which is a network trained to turn spectrograms into sound, would give cleaner audio.
- The prompt has a small effect. All 9,000 training images had the same caption, so the model never learned to link words such as
banjo
orharp
to a sound. - You can hear where the clips are joined. Each clip fades in and fades out before it's joined to the next clip.
▸ ¶Cite
@mastersthesis{parmar2024tradifusion,
author = {Parmar, Harshdeepsinh},
title = {Tradi-fusion Refined: Evaluating and Fine-tuning
the Riffusion Model for Irish Traditional Music},
school = {KTH Royal Institute of Technology},
address = {Stockholm, Sweden},
year = {2024}
}