FlowFusion AI DJ

What This Is

An AI DJ that learns to mix songs detecting optimal transition points, matching tempos, and blending audio seamlessly. But the deeper question is architectural: can State-Space Models (Mamba) replace Transformers for sequential audio modelling?

This is an open research question. Mamba has shown strong results on language and genomics, but audio is different it has multi-scale temporal structure, strict causality requirements for real-time mixing, and a very different token density than text.

The Research Question

Transformers process audio well but their quadratic attention cost makes long-context audio expensive. SSMs like Mamba have linear complexity and near-perfect recall over long sequences in theory but does this translate to musical structure understanding?

We run a controlled head-to-head:

• Architecture A: Standard Transformer encoder on audio tokens (EnCodec / DAC tokenisation)
• Architecture B: Mamba SSM on the same token stream
• Task: Predict DJ-quality transition points that are tempo-coherent, harmonically smooth, and energy-matched

The MedleyDB dataset provides professional multi-track stems with ground-truth transition metadata.

Current Stack

• PyTorch for both architectures
• DAC (Descript Audio Codec) for audio tokenisation 44 kHz at ~80 tokens/second
• Mamba (state-space-models library) for the SSM branch
• HuggingFace Transformers for the attention branch
• Librosa for tempo, beat, and key extraction as auxiliary features

What We're Learning

• How well SSMs handle the periodic structure of music (beats, bars, phrases)
• Whether Mamba's selectivity mechanism naturally attends to musical boundaries
• The effect of tokenisation quality on downstream mixing decisions
• How to frame "good mix" as a learnable reward signal

What's Next

• Publish quantitative comparison (MOS score, transition quality metrics)
• Extend to real-time streaming mixing with a Mamba-based causal model
• Explore hybrid architectures (local attention + Mamba for global context)