Music Analyser & Recommender

What This Is

A music analysis and recommendation engine built around self-supervised audio representations learning what makes two songs similar from the audio signal itself, without manual labels, then using those embeddings for fast retrieval.

The mainstream approach (Spotify, Apple Music) uses a combination of collaborative filtering and audio features extracted by classical DSP. We're exploring whether purely learned representations from a contrastive objective can produce better recommendations, especially for niche music that collaborative filtering can't handle due to sparse interaction data.

Self-Supervised Audio Learning

We use a contrastive learning objective (similar to CLMR Contrastive Learning for Music Representations): given two augmented views of the same audio clip, pull their embeddings together; push embeddings of different clips apart. Augmentations include:

• Time-domain stretching ±10%
• Pitch shifting ±2 semitones
• Random frequency masking
• Polarity inversion

The encoder learns to be invariant to these transformations, preserving only musically meaningful information (genre, mood, instrumentation, energy).

Retrieval

Once encoded, recommendations are retrieved via approximate nearest-neighbour search (FAISS). For a query track, we find the K closest embeddings in the database those are the recommendations. No collaborative filtering, no popularity bias.

YouTube and Playlist Tooling

We built around the constraint that most people's music lives on YouTube: a downloader pipeline handles YouTube → WAV → analysis → embedding → recommendation in one flow. OAuth integration allows reading streaming service playlists directly.

What We're Learning

• Whether contrastive representations capture genre and mood better than hand-crafted features (tempo, MFCC, spectral centroid)
• How embedding quality degrades for lower-bitrate audio (128kbps vs. lossless)
• The effect of augmentation choices on downstream recommendation quality

What's Next

• Compare against MusicGen embeddings (Meta's audio foundation model)
• Build a listening feedback loop: update embeddings based on skip/repeat signals
• Real-time recommendation for DJ sets