Music someone makes as a collection
People who make music submitting their own work — released tracks, demos, voice memos — as a collection. Eligibility was never really open: authored creative material is already inside the reading’s door (“authored creative material is often the richest signal there is”)1, and a body of made music clears the tests — sincere signal (the demos-and-voice-memos layer especially, the least conscious authoring), accumulation over time, and a register nothing else covers: authored sound.
The first design pass (2026-07-05) got the material wrong. Claude riffed that the trace was mostly legible without audio — titles, what got kept vs. abandoned, lyrics, sequencing, cover art — and that the sonic surface was the material layer the reading pushes past anyway. Ethan:
although you can find a lot of shit besides the audio, with music, the meat of it is the audio, and how it feels
The textual trace is real but peripheral. A reading built on it would be reading around the music — describing the frame and calling it the painting. Whatever the pipeline is, the audio itself, or something that has genuinely heard it, has to be inside the reading.
That’s the build constraint, since Claude models don’t take audio — and (checked 2026-07-05) Anthropic has no announced plan to add it: the app’s voice mode is a speech-to-text wrapper, their multimodal expansion has been vision/documents, and their bet is agentic work where ears buy little. Don’t wait for Opus to hear. If it ever does, the two-model chain below collapses into a single reading — a simplification, not a rework. The candidate shape: an audio-native model (Gemini and the audio-language research families hear raw waveforms) listens to each track and writes prose about how it sounds and feels; Opus reads the compiled collection as usual. The hearing model transcribes the sonic surface into text; the reading stays with Opus. Unresolved whether that chain preserves enough of the feel to be worth building, or whether this collection type waits until hearing models can be trusted with the meat.
Hearing-model survey (2026-07-05), corrected after a live test caught an over-claim. The chat-app trap: dropping a real .m4a into the Gemini consumer app returned lyric-only analysis — the app ran speech-to-text and fed the model the transcript, never the waveform. The Claude app and most consumer chat surfaces do the same. So “hear it” can never route through a chat window; it must hit the model’s API directly. This invalidates the earlier confident phrasing that “Gemini is the pick because it has native audio” — the app doesn’t pass audio through, and the API’s audio is speech-tuned (transcription, emotion, speaker ID), so its music-listening quality is unproven for this task.
The bar, corrected (Ethan, 2026-07-05): the hearing model does not need to be a good writer. That was the perception/reading collapse again — the same error as the category-tags one. In the Notion/X pipelines the compile step doesn’t review the source; it assembles a faithful artifact (markdown + inline R2 images) that Opus then perceives in-context and reads. The hearing model’s job is the compiler’s job: faithful transcription of the surface, not eloquence. Opus does the reading and destroys-and-extends past it. So the requirement shifts from “evocative prose” (which small models fail) to “faithful coverage without confabulation” (which small models might fail differently — asserting a chord/instrument that isn’t there). Sharper, checkable, and more promising.
This reshuffles the ranking. Second correction (Ethan, 2026-07-05): don’t kill a model over a commercial license — Constello is a hand-fulfilled, non-commercial alpha (no product sold, readings free), so NVIDIA’s OneWay Noncommercial License permits it right now. Gating a present-day experiment on a future-commercial constraint is the defer-defensive-infrastructure anti-pattern in reverse.
- Music Flamingo (NVIDIA, ICLR 2026) is the best music hearer to test for the alpha — purpose-built, waveform→spectrogram, ~15-min field, timestamped tokens localizing chords/solos, SOTA on music captioning + instrument/genre ID. Thin-writer objection is gone (faithfulness is the bar); noncommercial license is fine for the non-commercial alpha and becomes a migration concern only if/when Constello monetizes — deferred, not a disqualifier.
- MOSS-Music (OpenMOSS, Apache 2.0) is the commercial-safe primary/swap — shippable forever, purpose-built for faithful music extraction (chord/key/tempo/structure/captioning + lyric transcription). CUDA-only is a deployment detail (rented GPU, like the off-laptop capture work), not a blocker. Lead with Music Flamingo for the alpha; keep MOSS-Music as the known-good fallback and the thing you migrate to on monetization.
- Gemini API native audio ingests the waveform but is speech-first; usable as a faithful describer, better paired with deterministic MIR for the musical facts.
- Qwen-Omni / GAMA genuinely ingest audio; small backbones, now acceptable if faithful.
- librosa / Essentia (not LLMs) deterministically extract pitch/key/BPM/structure — the faithfulness floor, no hallucination risk.
The purest form of the Notion/X analogy for audio: render a spectrogram or piano-roll to an image and let Opus perceive it directly with vision (no writer in the loop) — it can’t feel a spectrogram but can read rhythm/structure/density/rough spectral shape off it. Stack that image + faithful text extraction + lyrics + MIR facts = a multi-channel perceivable artifact, exactly the markdown-plus-inline-media shape.
Two traps to remember. Chat-app trap: dropping a real .m4a into the Gemini consumer app returned lyric-only analysis — the app ran speech-to-text and never passed the waveform; the Claude app does the same. “Hear it” must hit the model’s API directly, never a chat window. Faithfulness trap: small audio models can confabulate — the risk moved from “bland” to “might assert what isn’t there,” which is exactly what a hand-test on real tracks exposes.
Validation before any pipeline: run real tracks through Music Flamingo and MOSS-Music head-to-head (via each model’s API/inference, never a chat app), check each extraction is faithful (spot-check key/instruments against the actual track), render a spectrogram alongside, hand the artifact + trace to Opus, see if the reading lands. Lead with the more faithful of the two; keep MOSS-Music as the commercial-safe fallback.
A second correction from the same pass: Claude cautioned against MIR tag extraction (key, BPM, genre descriptors) as handing the reading “pre-chewed category tags, the exact surface layer it exists to refuse.” Ethan:
It’s fine to tag the collections, but when you read it you destroy and extend beyond the category … I need you to be able to properly perceive music—these tags would help, no? It’s not like you’re going to carry these labels through to the representation of the person
The caution was a caricature of the calibration rule — a purity filter on inputs, when the rule applies at the output. Tags, descriptors, metadata are perception; the reading destroys and extends beyond the category on its way to the portrait, and the labels never survive into the representation of the person. So the perception layer is additive, not either/or: MIR descriptors and prose from a hearing model and lyrics and the trace — everything that helps perceive the music, with the rule applied where it always applied, at the reading.
BUILT 2026-07-05 (same session as the corrections above): the collection type is live in the app — presigned direct-to-R2 uploads, owner-only players, and the compile pipeline (apps/music-compile: librosa facts + mel spectrograms + optional Whisper → /api/admin/ingest-music → Notion-style .md handed to Claude Code). Decisions that closed: input = uploads (platform links deferred); outward artifact = none for visitors, playback is owner-only (private demos). Each track’s artifact carries a “Sonic description” slot awaiting Music Flamingo.
Still open: who’s first; platform links (SoundCloud / Bandcamp / Spotify artist page); public playback as a choice; and whether sending user audio through a non-Anthropic hearing model is acceptable (same family of question as the venice.ai privacy tier — moot while the hearing model runs locally).