mugen/demos/audiogen_demo.ipynb

AudioGen

Welcome to AudioGen's demo jupyter notebook. Here you will find a series of self-contained examples of how to use AudioGen in different settings.

First, we start by initializing AudioGen. For now, we provide only a medium sized model for AudioGen: facebook/audiogen-medium - 1.5B transformer decoder.

Important note: This variant is different from the original AudioGen model presented at "AudioGen: Textually-guided audio generation" as the model architecture is similar to MusicGen with a smaller frame rate and multiple streams of tokens, allowing to reduce generation time.

In [ ]:

from audiocraft.models import AudioGen

model = AudioGen.get_pretrained('facebook/audiogen-medium')

Next, let us configure the generation parameters. Specifically, you can control the following:

• use_sampling (bool, optional): use sampling if True, else do argmax decoding. Defaults to True.
• top_k (int, optional): top_k used for sampling. Defaults to 250.
• top_p (float, optional): top_p used for sampling, when set to 0 top_k is used. Defaults to 0.0.
• temperature (float, optional): softmax temperature parameter. Defaults to 1.0.
• duration (float, optional): duration of the generated waveform. Defaults to 10.0.
• cfg_coef (float, optional): coefficient used for classifier free guidance. Defaults to 3.0.

When left unchanged, AudioGen will revert to its default parameters.

In [ ]:

model.set_generation_params(
    use_sampling=True,
    top_k=250,
    duration=5
)

Next, we can go ahead and start generating sound using one of the following modes:

• Audio continuation using model.generate_continuation
• Text-conditional samples using model.generate

Audio Continuation

In [ ]:

import math
import torchaudio
import torch
from audiocraft.utils.notebook import display_audio

def get_bip_bip(bip_duration=0.125, frequency=440,
                duration=0.5, sample_rate=16000, device="cuda"):
    """Generates a series of bip bip at the given frequency."""
    t = torch.arange(
        int(duration * sample_rate), device="cuda", dtype=torch.float) / sample_rate
    wav = torch.cos(2 * math.pi * frequency * t)[None]
    tp = (t % (2 * bip_duration)) / (2 * bip_duration)
    envelope = (tp >= 0.5).float()
    return wav * envelope

In [ ]:

# Here we use a synthetic signal to prompt the generated audio.
res = model.generate_continuation(
    get_bip_bip(0.125).expand(2, -1, -1), 
    16000, ['Whistling with wind blowing', 
            'Typing on a typewriter'], 
    progress=True)
display_audio(res, 16000)

In [ ]:

# You can also use any audio from a file. Make sure to trim the file if it is too long!
prompt_waveform, prompt_sr = torchaudio.load("../assets/sirens_and_a_humming_engine_approach_and_pass.mp3")
prompt_duration = 2
prompt_waveform = prompt_waveform[..., :int(prompt_duration * prompt_sr)]
output = model.generate_continuation(prompt_waveform, prompt_sample_rate=prompt_sr, progress=True)
display_audio(output, sample_rate=16000)

Text-conditional Generation

In [ ]:

from audiocraft.utils.notebook import display_audio

output = model.generate(
    descriptions=[
        'Subway train blowing its horn',
        'A cat meowing',
    ],
    progress=True
)
display_audio(output, sample_rate=16000)

In [ ]: