Talk to Notes

What This Is

A voice-first personal knowledge system. You speak; the system transcribes, restructures, categorises, and stores making spoken thought instantly searchable and actionable. The target user is anyone whose ideas come faster than their typing.

The product insight: the bottleneck in note-taking is usually not capture (voice is faster than typing) but retrieval. Most voice notes are recorded and never found again. This system is designed so that everything you say is immediately usable.

The Pipeline

Step 1: Transcription via Whisper OpenAI Whisper (large-v3) at 96% accuracy on clean speech and ~88% on challenging audio. We use the faster-whisper implementation (CTranslate2 backend) for ~4× speed improvement with negligible quality loss. Real-time streaming transcription is enabled via the whisper.cpp server-sent events API.

Step 2: LLM Restructuring Raw transcription is messy filler words, non-linear thinking, repeated phrases. The first LLM pass (GPT-4o or Claude Sonnet) restructures the content: removes filler, infers paragraph breaks from semantic shifts, and formats lists where appropriate. The key constraint: no hallucination the output must only contain what was said, restructured for clarity.

Step 3: LLM Categorisation The second pass categorises the note into the user's existing taxonomy (or suggests a new category if none fit). This uses a retrieval-augmented classifier: embed the note, find similar existing notes, infer the category from their labels. New categories are surfaced for user confirmation, not auto-created.

Step 4: Storage and Indexing Notes are stored in PostgreSQL with full-text search and vector embeddings (pgvector). Retrieval supports both keyword search (fast, for known content) and semantic search (embedding similarity, for "what did I say about X?").

What Makes This Hard

Semantic chunking: a 10-minute voice recording shouldn't be one note. We detect topic shifts using embedding drift over sliding windows and split into sub-notes accordingly. The right granularity is "one idea", not "one recording".

Hallucination prevention in restructuring: LLMs have a tendency to add detail that sounds plausible but wasn't said. We use a factuality check: the restructured note is compared against the original transcript using a critic LLM that flags insertions.

What's Next

• Mobile app for one-tap recording
• Proactive retrieval: surface related past notes when you start recording a new one
• Integration with calendar: auto-link notes to meetings they were recorded during