Multi-Instrument Articulation Classifier

a cnn that takes raw audio and says how it was played, not what played it. three articulations — normal, legato, staccato — across five virtual instruments (guitar, muted guitar, flamenco guitar, trumpet, piano), labelled a phrase at a time. the applications are practice feedback and expressive transcription, and both of them need to know how a note was produced. most music information retrieval stops at the instrument.

that turns out to be a much harder question than the one instrument-id networks answer. instrument identity sits in a stable spectral envelope, which is why cnns handle it routinely on datasets like irmas. articulation is a property of how energy changes inside a note, and it has to survive timbres that differ from each other far more than the articulations do. attack sharpness, damping, sustain texture — fixed acoustic thresholds separate those badly, which is exactly the case where a network earns its keep.

no dataset existed for it, so i built one. procedural midi phrases, 30 seconds each, varying in pitch set, note rate, rhythm, key and velocity range, rendered in ableton live 12 through multi-sampled vst instruments, one mono wav per take. every render is named instrument__articulation__kind__takeNN, and the take is the unit of independence — that naming convention is the thing that makes the evaluation honest later. 120 phrase takes plus 45 isolated-note takes, the latter held out entirely as a separate test domain. all five instruments play all three articulations, which is what makes an unseen-instrument test possible.

preprocessing is deliberately boring so it stays reproducible from the raw renders. load at 22.05 khz mono through librosa, cut into 1 second windows at 50% overlap, drop anything under an rms floor, normalise each window to a fixed rms, then mel: 128 bins, n_fft 2048, hop 512, 44 frames, db against a fixed reference with absolute clipping. the first and second temporal differences get stacked on as extra channels, so each window is 3×128×44. that comes to 9,286 windows — 7,019 from phrases, 2,267 from isolated notes — split by whole take, 5 train / 2 validation / 1 test per class.

the network is small on purpose. three 3×3 conv–batchnorm–relu–maxpool stages at 16, 32 and 64 channels, then the feature map is averaged over both time and frequency, which collapses the parameter count. dropout at 0.3, a 64→128 fully connected layer, and two output heads: one over five instruments, one over three articulations. 33,160 parameters in total. splitting the heads lets the instrument head carry the timbre so the articulation head can specialise on temporal cues, and it is what makes the unseen-instrument test possible at all — you can score articulation on a timbre the model has never been trained on.

training is cross entropy on both heads summed, adam at 1e-3 with 1e-4 weight decay, batch size 32, up to 60 epochs, keeping the weights from the best validation epoch (51). augmentation is applied to the spectrogram rather than the waveform: random gain, low-level gaussian jitter, random time shifts, and specaugment. that matters more here than it usually would, because generated audio has almost no natural variation, and no variation is precisely the condition under which a network memorises instead of learning.

the baseline is an rbf-kernel svm over 21 standardised features — mean and standard deviation of rms, spectral centroid and bandwidth, mean rolloff and zero-crossing rate, and 13 mfcc means — on the same windows and the same splits. since articulation is defined by change over time and those features mostly aren't, the gap between the two models is a direct measurement of what the temporal information is worth: 0.913 against 0.803.

the first version of this reported 100% test accuracy, and it was wrong. clips were being split randomly out of continuous ableton renders, so a test clip and its nearest training clip shared instrument patch, room, reverb tail and note grid — mean cosine similarity in mel space of 0.9986. computing spectrograms with ref=np.max compounded it, since every window gets rescaled to its own peak, so a neighbouring note 4× louder drags the whole spectrogram down by about 12 db relative to an isolated one. splitting strictly by take and pinning the reference dropped the number to 0.913 and moved the learned features from grouping by source render to grouping by articulation. that was the real lesson of the project: the model will take the shortcut every single time you leave one lying around.

so the evaluation is built as tiers that get progressively further from the training distribution, and the shape of the decline is the result, not any one number. held-out takes: 0.913. averaged into one label per phrase, which is how you would actually use it: 15 of 15 correct, instrument and articulation both. isolated notes, after training on phrases only: 0.491, where the svm beats the cnn at 0.551. an entire instrument held out and scored on articulation alone: 0.640 against 0.33 chance. background noise added at test time: 0.933 at 40 db snr, 0.600 by 30 db, and chance at 10 db — partly because augmentation acted on spectrograms rather than waveforms so the model never saw acoustic noise, and partly because it leans on the near-silence between notes, which broadband noise fills in.

the isolated-note collapse is the most interesting failure and it is very specific. staccato recall falls from 0.98 on phrases to 0.05 on single notes — 41 correct out of 754. trained on continuous phrases, the model learned staccato as note density and rhythmic silence rather than as an attack shape, so take the neighbouring notes away and there is nothing left for it to look at. legato is the least confident class throughout, with the lowest recall at 0.86, which fits acoustically: legato differs from normal playing only by note overlap, a smaller difference than the silence that marks staccato.

worth stating the limits plainly. everything is generated from professional multi-sampled libraries, so the model has only ever heard pristine, studio-grade equipment, and accuracy dropping to 0.600 at 30 db snr is an early sign of what that costs — a system like this would give better feedback to whoever already owns better gear, and it has never been tested against a real player at all. the next steps are real recordings and a lot more single-note data, so it has to learn attack transients instead of gaps.