Separating the Inseparable

Two people, one microphone, talking over each other. To a listener it is a mildly annoying recording. To a model it is a single waveform in which two voices have already been summed into one number per sample, and no amount of filtering will unsum them. Recovering the two tracks is not a cleanup job but a reconstruction, and it takes more than one model to do it.

DUETA exposes one endpoint for this: POST /v1/jobs/separation. You send one recording of two speakers and you get back two mono tracks, one per speaker, each carrying a separation score. Tracks come back at 48 kHz when super-resolution runs, and at 16 kHz when you turn it off. Inside that single job, up to five models run in a fixed order. This post is about why that order exists.

Why one model is not enough

A speaker separator is trained on a specific kind of input: a mixture of voices. Real recordings are not that. They have room tone, air conditioning, a laptop fan, background music from the café, and a codec that already threw away everything above 8 kHz. Every one of those is energy the separator has to account for, and none of it belongs to either speaker.

So the separator is not the first stage. It sits in the middle of a chain whose job is to hand it something close to what it was trained on, and then to repair the damage that separation itself leaves behind.

upload
  │
  ├─ vocal_separation    Mel-Band RoFormer    →  _vocals   16 kHz
  ├─ separation          TF-Locoformer        →  _spk1 _spk2
  ├─ enhancement         FRCRN                →  _enh      16 kHz
  ├─ super_resolution    AP-BWE               →  _48k      48 kHz
  └─ si_sdr              ScoreEstimator       →  one scalar
                                                  │
                                        two tracks + one score

Stage one: take the room away

The first pass is a music source separation model, Mel-Band RoFormer, used for something slightly off-label: we keep only its vocal stem. Anything the model considers non-vocal (music, hum, broadband room noise, the mechanical parts of the recording) is discarded along with the other stems.

It is a blunt instrument and that is the point. A speaker separator is trained on mixtures of voices, and music is structured and voice-like in exactly the ways that confuse it. A music separator does not struggle with music. Running it first means the separator that follows only has to model voices.

Stage two: the actual split

This is the stage the product is named for. A TF-Locoformer model (the in-house checkpoint, trained for this task) takes the single vocal track and emits two: _spk1 and _spk2. From here on there are two files where there was one, and every later stage runs on both.

Two things about this stage are easy to get wrong. First, the assignment of speakers to outputs is arbitrary and not stable across jobs: _spk1 is not "the first person to talk", it is just one of the two lanes the model settled on. Second, and more consequential:

Two speakers are assumed to be present, and two tracks always come back. Given a monologue the pipeline force-splits the single voice into two overlapping tracks rather than detecting it, so send it audio you expect to contain two people.

From the input requirements in the docs

There is no speaker-count detector in front of the separator. If you cannot guarantee two speakers in the audio you send, that check belongs in your code, not in ours, and we would rather say so than let you discover it on a batch of solo recordings.

Stage three: enhancement, one track at a time

The third pass is FRCRN, a speech enhancement model, run at 16 kHz on each separated track. Where stage one removed things that were never speech, this stage works on what the split left behind: residual noise, reverberation smearing, and the artifacts the earlier stages introduced.

The order matters more than it looks. FRCRN is a single-speaker enhancer: it decides what is speech and suppresses the rest. Run it on the still-mixed recording and the overlapped second voice reads as noise: it partially erases exactly the signal the separator needs. Run it after the split, once per track, and each pass sees the one voice it was built for; on clean input it is close to a no-op, and on noisy input it earns its keep.

Stage four: putting the bandwidth back

Everything up to this point ran at 16 kHz, which is where these models are trained and where they are good. It is also a ceiling: a 16 kHz sample rate carries nothing above 8 kHz, and the sibilance and air that make a voice sound like a person in a room live above that line.

So the fourth stage is AP-BWE, a bandwidth extension model, which reconstructs high-frequency content that is not present in its input. The pipeline writes its output at 48 kHz, and that is the sample rate you receive whenever this stage runs; skip it and the tracks come back at the 16 kHz the stages before it work in.

This stage is one of the cheapest of the five, not the most expensive. Measured as a real-time factor (processing seconds per second of audio), the model stages sit at roughly 0.10 for vocal separation, 0.134 for speaker separation, and 0.036 for enhancement across both stems. Super-resolution is about 0.004 — more than an order of magnitude below any of the others — so reconstructing detail turns out to cost far less than removing it. Speaker separation, at 0.134, is where the pipeline's time actually goes.

Stage five: the check

The last stage produces no audio at all. It takes the two finished tracks, runs them through a no-reference scoring model in a single forward pass, and returns one number: an estimated SI-SDR in decibels for the pair.

One number for the pair, not one per track. Both stems in the response carry the same figure, because the question the estimator answers is "how cleanly are these two separated from each other", which is a property of the pair and not of either track alone. Roughly, above 10 dB is normally usable as-is; between 5 and 10 dB there is some bleed left; below 5 dB you can usually hear it. How that estimator works is its own post.

The intermediates you never see

Five stages produce five sets of files, and the names accumulate as the chain advances: a stem that has been through all four audio stages is called something like mix-00_vocals_spk1_enh_48k. Only the last pair is ever uploaded off the GPU box.

The other four are your audio too, and nothing downloads them. Each stage's working directory is deleted as soon as the next stage has produced its own output, so a long recording never sits on the machine five times over, and the scratch tree is removed in a finally block whether the job succeeded, failed, or was canceled. There is no shared filesystem between the API and the worker; the GPU box holds no user data at rest.

That is the whole path from one waveform to two. If you want the operational side (why these five models cannot share a Python process, and how five stages become one progress bar), that is the next post. If you want to send something through it, the docs start with a curl command.