papers/zen-musician.tex

\documentclass[11pt,a4paper]{article}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{amsmath,amsfonts,amssymb}
\usepackage{graphicx}
\usepackage{hyperref}
\usepackage{listings}
\usepackage[dvipsnames]{xcolor}
\usepackage{booktabs}
\usepackage{float}
\usepackage{geometry}
\geometry{margin=1in}

\definecolor{zenblue}{RGB}{41,121,255}
\definecolor{zengreen}{RGB}{52,199,89}
\definecolor{codegray}{RGB}{245,245,245}

\hypersetup{colorlinks=true,linkcolor=zenblue,urlcolor=zenblue,citecolor=zenblue}

\lstset{
    backgroundcolor=\color{codegray},
    basicstyle=\ttfamily\small,
    breaklines=true,
    captionpos=b,
    frame=single,
    numbers=left,
    numberstyle=\tiny\color{gray}
}

\title{
    \vspace{-2cm}
    \Large \textbf{Zen AI Model Family} \\
    \vspace{0.5cm}
    \Huge \textbf{Zen-Musician} \\
    \vspace{0.3cm}
    \large AI Music Composition, MIDI Generation, and Audio Synthesis \\
    \vspace{0.5cm}
    \normalsize Technical Report v2025.01
}

\author{
    Hanzo AI Research Team\thanks{[email protected]} \and
    Zoo Labs Foundation\thanks{[email protected]}
}

\date{January 2025}

\begin{document}

\maketitle

\begin{abstract}
We present \textbf{Zen-Musician}, a 3-billion parameter generative music model capable of
multi-genre composition, MIDI generation, audio synthesis, and real-time accompaniment.
Zen-Musician unifies symbolic music representation (MIDI) and acoustic audio generation
within a single model, enabling coherent compositions that span from high-level structural
planning (chord progressions, song sections, orchestration) to low-level acoustic rendering
(instrument timbre, expression, dynamics). Trained on 180,000 hours of licensed music spanning
40 genres, the model achieves a Frechet Audio Distance (FAD) score of 2.3 (compared to 1.8
for human composers in identical conditions), earns a Music Opinion Score (MOS) of 4.1/5.0,
and generates 30-second compositions in under 4 seconds on an A10G GPU. A novel
\textbf{Section-Aware Transformer (SAT)} architecture enforces long-range musical structure
(verse/chorus/bridge coherence) that pure autoregressive models lack. Zen-Musician supports
real-time accompaniment generation, adaptive score following, and style transfer across genres.
\end{abstract}

\tableofcontents
\newpage

\section{Introduction}

Music generation has seen rapid progress, yet existing models suffer from two persistent failure
modes. \textbf{Structural incoherence}: autoregressive models generate locally plausible music
that lacks the large-scale structure (repetition, contrast, development) that defines
professional composition. \textbf{Modality gap}: separate symbolic and acoustic models require
clumsy pipelines (MIDI $\to$ synthesis) that break acoustic coherence.

Zen-Musician resolves both through:

\begin{enumerate}
    \item \textbf{Section-Aware Transformer (SAT)}: A hierarchical attention architecture that
    maintains a global structural plan (verse, chorus, bridge, solo) and conditions local
    generation on the current section's role in the overall form.
    \item \textbf{Unified token space}: MIDI events and audio codec tokens share a single
    vocabulary, enabling the model to interleave symbolic and acoustic generation.
    \item \textbf{Multi-genre training}: 180,000 hours of licensed music spanning 40 genres
    with rich metadata (key, tempo, mood, instrumentation).
\end{enumerate}

\subsection{Model Overview}

\begin{table}[H]
\centering
\begin{tabular}{ll}
\toprule
\textbf{Property} & \textbf{Value} \\
\midrule
Parameters & 3B \\
Architecture & Section-Aware Transformer (SAT) \\
Token Vocabulary & 32,768 (MIDI events + Encodec audio tokens) \\
Context Length & 30K tokens ($\approx$2 min of music) \\
Genres & 40 (classical, jazz, pop, EDM, folk, metal, + 34 more) \\
Max Generation Length & 10 minutes continuous \\
Training Data & 180,000 hours licensed music \\
\bottomrule
\end{tabular}
\caption{Zen-Musician Model Specifications}
\end{table}

\section{Architecture}

\subsection{Unified Token Space}

Zen-Musician's key architectural decision is a unified vocabulary covering both symbolic music
and acoustic audio:

\textbf{MIDI tokens} (4,096 tokens): Representing note-on, note-off, velocity, program change,
tempo, and time shift events at 5ms resolution.

\textbf{Audio tokens} (28,672 tokens): Encodec \cite{defossez2022encodec} codes at 24kHz with
4 codebooks at 75Hz frame rate, providing compact acoustic representation.

\textbf{Structural tokens} (128 tokens): Section markers (VERSE, CHORUS, BRIDGE, INTRO,
OUTRO, SOLO, BREAKDOWN) and metadata (KEY, TEMPO, GENRE, INSTRUMENT).

The model learns to transition fluidly between MIDI and audio modalities, using MIDI for
structural planning and audio tokens for acoustic detail within each section.

\subsection{Section-Aware Transformer}

The SAT adds a \textbf{structural level} atop the token-level transformer:

\begin{itemize}
    \item A \textbf{Structure Encoder} (4 transformer layers) processes a sequence of section
    descriptors $(s_1, s_2, \ldots, s_K)$ representing the planned song form.
    \item A \textbf{Section Embedding} is computed for each section and injected into token-level
    generation via cross-attention at every fourth transformer layer.
    \item During generation, the model can dynamically revise the structural plan based on
    what has been generated, enabling coherent improvisation.
\end{itemize}

Formally, at each generation step $t$ within section $k$:

\begin{equation}
    h_t = \text{TransformerLayer}(h_{t-1}, \text{CrossAttn}(h_{t-1}, E_k))
\end{equation}

where $E_k = \text{StructureEncoder}(s_k)$ is the section embedding.

\subsection{Accompaniment Mode}

In accompaniment mode, the model conditions on an input melody (provided as MIDI or audio)
and generates harmonically and rhythmically coherent accompaniment. The melody is encoded
via a lightweight 2-layer transformer and injected as an additional conditioning signal
alongside the section embedding. Accompaniment latency is 120ms (suitable for real-time
performance with modest lookahead).

\section{Training}

\subsection{Dataset}

\begin{table}[H]
\centering
\begin{tabular}{lrrl}
\toprule
\textbf{Source} & \textbf{Hours} & \textbf{Proportion} & \textbf{Licensing} \\
\midrule
Licensed studio recordings & 80,000 & 44.4\% & Commercial license \\
Classical MIDI corpus & 12,000 & 6.7\% & Public domain \\
Jazz Real Book corpus & 8,000 & 4.4\% & Licensed \\
Folk music archives & 15,000 & 8.3\% & Creative Commons \\
Electronic music library & 25,000 & 13.9\% & Commercial license \\
Synthetic augmentation & 40,000 & 22.2\% & Generated \\
\midrule
\textbf{Total} & \textbf{180,000} & 100\% & \\
\bottomrule
\end{tabular}
\caption{Zen-Musician Training Data}
\end{table}

\subsection{Training Protocol}

\textbf{Stage 1 -- Audio codec training} (50K steps): Encodec-based audio codec is trained
on the full audio corpus. Codebook utilization target $>$95\%.

\textbf{Stage 2 -- Symbolic pretraining} (100K steps): Token-level transformer is pretrained
on MIDI token sequences with masked token prediction.

\textbf{Stage 3 -- Joint pretraining} (150K steps): Training on the full unified token space
(MIDI + audio) with next-token prediction.

\textbf{Stage 4 -- Structural fine-tuning} (50K steps): SAT structural components are trained
with section-boundary supervision from annotated music datasets.

\textbf{Stage 5 -- RLHF from musicians} (20K steps): Reward model trained on 30,000 pairwise
comparisons from professional musicians, optimized via PPO.

\begin{table}[H]
\centering
\begin{tabular}{lllll}
\toprule
\textbf{Stage} & \textbf{Steps} & \textbf{Batch} & \textbf{LR} & \textbf{Hardware} \\
\midrule
Codec & 50K & 256 & 1e-4 & 8$\times$A100 \\
Symbolic & 100K & 128 & 3e-4 & 16$\times$A100 \\
Joint & 150K & 64 & 2e-4 & 32$\times$A100 \\
Structural & 50K & 64 & 5e-5 & 16$\times$A100 \\
RLHF & 20K & 32 & 2e-5 & 8$\times$A100 \\
\bottomrule
\end{tabular}
\caption{Zen-Musician Training Configuration}
\end{table}

\section{Evaluation}

\subsection{Frechet Audio Distance (FAD)}

FAD measures distribution distance between generated and real music in VGGish embedding space.
Lower is better; human composers in identical prompt conditions achieve 1.8.

\begin{table}[H]
\centering
\begin{tabular}{lcc}
\toprule
\textbf{Model} & \textbf{FAD} $\downarrow$ & \textbf{Parameters} \\
\midrule
MusicLM & 4.0 & N/A (proprietary) \\
AudioCraft / MusicGen-Large & 3.8 & 3.3B \\
Stable Audio Open & 4.4 & 1.1B \\
JEN-1 & 2.7 & 0.3B \\
Human composers (reference) & 1.8 & -- \\
\textbf{Zen-Musician} & \textbf{2.3} & 3B \\
\bottomrule
\end{tabular}
\caption{Frechet Audio Distance on MusicCaps evaluation set}
\end{table}

\subsection{Music Opinion Score (MOS)}

Twenty professional musicians rated 200 generated compositions on five criteria.

\begin{table}[H]
\centering
\begin{tabular}{lc}
\toprule
\textbf{Criterion} & \textbf{Mean Score (1--5)} \\
\midrule
Harmonic coherence & 4.2 \\
Rhythmic consistency & 4.4 \\
Structural form (verse/chorus logic) & 3.9 \\
Genre authenticity & 4.1 \\
Emotional expressiveness & 4.0 \\
\midrule
\textbf{Overall MOS} & \textbf{4.1} \\
\bottomrule
\end{tabular}
\caption{Music Opinion Score (N=20 professional musicians)}
\end{table}

\subsection{Genre Consistency}

We evaluate genre consistency using a fine-tuned genre classifier on 500 generated samples
per genre, measuring what fraction are correctly classified:

\begin{table}[H]
\centering
\begin{tabular}{lcc}
\toprule
\textbf{Genre} & \textbf{Zen-Musician} & \textbf{MusicGen-Large} \\
\midrule
Classical & 91.4\% & 83.2\% \\
Jazz & 87.2\% & 74.1\% \\
Pop & 93.8\% & 88.6\% \\
Electronic/EDM & 95.1\% & 91.3\% \\
Folk & 82.3\% & 69.4\% \\
Metal & 88.7\% & 72.8\% \\
\textbf{Average} & \textbf{89.8\%} & \textbf{79.9\%} \\
\bottomrule
\end{tabular}
\caption{Genre Classification Accuracy of Generated Music}
\end{table}

\subsection{Structural Analysis}

To evaluate large-scale structure, we measure the presence and proper placement of structural
sections (intro, verse, chorus, bridge, outro) in generated 3-minute compositions, as
detected by a structure segmentation model:

\begin{table}[H]
\centering
\begin{tabular}{lcc}
\toprule
\textbf{Model} & \textbf{Section F1} $\uparrow$ & \textbf{Repetition Consistency} \\
\midrule
MusicGen-Large & 0.52 & 61.3\% \\
AudioCraft XL & 0.58 & 67.4\% \\
\textbf{Zen-Musician (SAT)} & \textbf{0.79} & \textbf{84.2\%} \\
\bottomrule
\end{tabular}
\caption{Musical Structure Quality Metrics}
\end{table}

\subsection{Accompaniment Quality}

\begin{table}[H]
\centering
\begin{tabular}{lcc}
\toprule
\textbf{Metric} & \textbf{Value} \\
\midrule
Harmonic correctness (% in-key) & 94.7\% \\
Beat alignment error (ms) & 22.3 \\
Musician preference vs. random & 91.4\% \\
Real-time latency & 120ms \\
\bottomrule
\end{tabular}
\caption{Accompaniment Mode Evaluation}
\end{table}

\section{Applications}

\subsection{Content Creation}

Zen-Musician is integrated into Hanzo's content creation platform, enabling video creators to
generate custom background music from natural language descriptions (``upbeat jazz, 90 seconds,
suitable for a cooking tutorial''). Time-to-music is under 8 seconds for 90-second clips.

\subsection{Real-Time Performance}

Musicians use Zen-Musician in live performance via the Zen-Musician VST plugin, which provides
real-time accompaniment that adapts to the performer's tempo, key, and dynamics.

\subsection{Film Scoring}

The model generates adaptive film scores conditioned on video (via integration with Zen-Foley's
video encoder) that automatically adjust mood and intensity to match scene content.

\section{Integration}

\begin{lstlisting}[language=Python, caption=Zen-Musician Composition Generation]
from zen import ZenMusician

model = ZenMusician.from_pretrained("zenlm/zen-musician-3b")

# Text-to-music generation
audio = model.compose(
    prompt="An introspective jazz ballad in F minor, "
           "featuring solo piano with brushed drums, "
           "verse-chorus-bridge structure, 3 minutes",
    bpm=72,
    duration_sec=180,
    format="wav"
)
audio.save("composition.wav")

# Accompaniment generation
from zen import MidiStream
melody = MidiStream.from_file("melody.mid")
accompaniment = model.accompany(
    melody=melody,
    style="bossa nova",
    instruments=["guitar", "bass", "drums"]
)
\end{lstlisting}

\section{Conclusion}

Zen-Musician's Section-Aware Transformer architecture and unified MIDI-audio token space produce
music with measurably better large-scale structure (SAT section F1: 0.79 vs. 0.58 for best
baseline) and competitive perceptual quality (FAD 2.3, MOS 4.1/5). The model's real-time
accompaniment capability opens new applications in live performance while its large-scale
generation pipeline serves content creation at production scale.

\begin{thebibliography}{10}
\bibitem{defossez2022encodec} A. Defossez et al., ``High Fidelity Neural Audio Compression,'' arXiv:2210.13438, 2022.
\bibitem{copet2023musicgen} J. Copet et al., ``Simple and Controllable Music Generation,'' NeurIPS, 2023.
\bibitem{agostinelli2023musiclm} A. Agostinelli et al., ``MusicLM: Generating Music From Text,'' arXiv:2301.11325, 2023.
\end{thebibliography}

\end{document}