A user tapped Play, said nothing, and the rehearsal ended by itself. The speech engine had just finalized a sentence spoken twelve seconds earlier, and those words happened to match the next line.
Scriptmate is a line rehearsal app for actors. It speaks the other characters’ lines, listens for the actor’s cue, and fires the next line the instant the actor finishes. That is a narrower job than transcription and a harder one: the app has to decide, in real time, that a specific sentence has been said well enough. On iOS 26 it does that with SpeechAnalyzer and SpeechTranscriber. The analyzer has been in production since late June. What follows is what broke, what fixed it, and what is still open.
Why I left SFSpeechRecognizer
The iOS 16 to 25 path still uses SFSpeechRecognizer, forced on device, with the current line’s words passed as contextual strings. It works, and it has four limits that hurt a rehearsal loop specifically:
- A cold start per line. Each line needs a new recognition request and task, which costs 200 to 400 ms before the first partial arrives. Actors notice.
- Frozen vocabulary.
contextualStringsis fixed when the request is created. To bias the recognizer toward the next line you have to tear down and recreate the task. - A task timeout of roughly one minute. Long monologues need a restart mid line, and a restart means a cold start.
- A weak on-device German model. On room audio, the on-device German model transcribed the first two or three seconds of each line and dropped the rest, missing five or six cues out of six.
SpeechTranscribertranscribed the same microphone capture nearly verbatim. That single finding moved every supported language to the new engine.
SpeechAnalyzer fixes three of the four. The vocabulary one is a story of its own, below.
The setup that works
The whole pipeline lives in one class behind a VoiceRecognizer protocol, so the rest of the app cannot tell which engine it is talking to. A factory picks the analyzer on iOS 26 when SpeechTranscriber.isAvailable is true and the script language has a locale in its map, and falls back to the old engine otherwise.
The transcriber is built from the progressiveTranscription preset with two attributes added:
let preset = SpeechTranscriber.Preset.progressiveTranscription
let transcriber = SpeechTranscriber(
locale: locale,
transcriptionOptions: preset.transcriptionOptions,
reportingOptions: preset.reportingOptions,
attributeOptions: preset.attributeOptions.union([.transcriptionConfidence, .audioTimeRange])
)
let analyzer = SpeechAnalyzer(modules: [transcriber])
let (inputSequence, inputBuilder) = AsyncStream.makeStream(of: AnalyzerInput.self)progressiveTranscription gives you volatile partials as the model refines, plus a smaller context window that lowers latency on short utterances. .transcriptionConfidence stamps every word with a confidence. .audioTimeRange stamps every word with where it sits on the audio timeline. Ask for both on day one; everything below depends on them.
Notice what is not there: a SpeechDetector module. Its voice activity gate swallowed short lines spoken right after the app resumed listening. Two lines from a live run, “Everything?” and “That’s not healthy!”, were never transcribed in four runs with the detector attached and transcribed every time without it. Nothing in the app even consumed its results; stall detection already lived in my own state machine as an eight second silence timer. The detector was removed, and the same scene runs twenty seconds shorter.
Assets install lazily and take about thirty seconds on Wi-Fi the first time, so the warm-up runs in a detached task as soon as the rehearsal screen is composed, well before the actor reaches line one:
Task.detached { [analyzer, transcriber] in
if let request = try await AssetInventory.assetInstallationRequest(supporting: [transcriber]) {
try await request.downloadAndInstall()
}
let format = await SpeechAnalyzer.bestAvailableAudioFormat(compatibleWith: [transcriber])
try await analyzer.prepareToAnalyze(in: format)
}Two tasks then live for the lifetime of the recognizer: one iterates transcriber.results, the other awaits analyzer.analyzeSequence(inputSequence). The input stream stays open across every line change and is only finished when the rehearsal stops.
Three definitions the rest of this post leans on. A volatile result is a partial the model may still revise; it carries one time range for its whole text, starting at the last point the model committed and reaching back through any silence. A final result is committed, which in practice happens after a real pause, and it carries a time range per word. An utterance is open from the first volatile until the final that closes it.
Feed it 16 kHz mono int16, and stamp every buffer
The hardware delivers 32-bit float at 48 kHz, often stereo. I feed the analyzer 16 kHz mono 16-bit integer PCM regardless of what bestAvailableAudioFormat returns, because it is the canonical speech format on Apple platforms and the converter cost is negligible. One AVAudioConverter handles the whole conversion inside the input tap:
let target = AVAudioFormat(commonFormat: .pcmFormatInt16, sampleRate: 16000, channels: 1, interleaved: true)!
let converter = AVAudioConverter(from: hardwareFormat, to: target)!
audioEngine.inputNode.installTap(onBus: 0, bufferSize: 1024, format: hardwareFormat) { [weak self, converter] buffer, _ in
guard let self, self.isFeedingAnalyzer else { return }
guard let converted = Self.convert(buffer, using: converter), converted.frameLength > 0 else { return }
let start = self.audioClock.stamp(frameCount: converted.frameLength)
self.inputBuilder.yield(AnalyzerInput(buffer: converted, bufferStartTime: start))
}The audioClock is a tiny lock-protected frame counter that returns a CMTime for each buffer and advances by its length. AnalyzerInput accepts a bufferStartTime, and if you supply it, every result you get back is positioned on the same timeline. Both bugs below were fixed with that timeline.
The isFeedingAnalyzer flag is how the app pauses listening while the reader speaks, without stopping the engine. The audio session’s interruption notification does not fire for your own text to speech, so the view model calls pauseForReaderPlayback() before each reader line, and that method drops the flag.
One more thing about the engine: after a stop, the recognizer swaps in a fresh AVAudioEngine. A stopped engine’s input node keeps reporting a stale hardware format, and installing a tap with that format throws a fatal exception the next time the route has changed. A new engine costs nothing and never lies about its format.
Swap vocabulary per line: the API is there, the hints are not
This was supposed to be the feature that made the migration worth it. When the actor’s line changes, the app builds an AnalysisContext from the words of the new line plus a few script-wide names, caps it at a hundred entries, and hands it to the running analyzer:
func start(line text: String, cue: String, contextualStrings: [String]) {
cueStartTime = audioClock.now
awaitingPreviousFinal = hasOpenUtterance
isFeedingAnalyzer = true
Task {
let context = AnalysisContext()
var hints = Array(Set(text.components(separatedBy: .whitespaces).filter { !$0.isEmpty } + contextualStrings))
if hints.count > 100 { hints = Array(hints.prefix(100)) }
context.contextualStrings[.general] = hints
try? await analyzer.setContext(context)
}
}No stop, no restart, no cold start. And on every iOS 26 build I have tested, every call to setContext fails with SFSpeechErrorDomain Code=6 "Invalid JIT profile". The try? swallows it, which is why it took a device log to notice. The hints have never applied. The accuracy gains over the old engine came from the model, not from the vocabulary. It is filed in my harness doc as open, and I would like to hear from anyone who has seen that error go away.
The two lines above the task are the ones that matter: cueStartTime = audioClock.now and awaitingPreviousFinal = hasOpenUtterance. They exist because of the first bug.
The stale final that fired cues early
On September 5 two versions of the same failure arrived within hours of each other.
The field report first. On the last line of a scene, “Wait, I need some help!”, the rehearsal ended as if Stop had been tapped, before the actor said a word. The previous line was “Hey, I need some help here.” The two share four of five words. SpeechAnalyzer had finalized the first line only after the reader’s line in between, because no audio is fed while the reader speaks, and that late final satisfied the new line’s cue. No string matcher can tell those two lines apart, so the fix had to live upstream.
The same day the device harness, described below, caught a purer form. After a reader line, the previous utterance was finalized one to two seconds into the next actor line, and three incidental words from it, “it”, “the” and “of”, were enough to fire a nine word cue. The matcher walks the expected words in order, and each transcribed word may skip up to two expected words to find its match, so a handful of common words can carry the cursor most of the way along a line without really matching it.
Two fixes. The matcher now requires that at least sixty percent of the cue’s words genuinely matched, so alignment position alone cannot advance. That closes the second case. For the first, the recognizer drops or holds anything that belongs to the previous line, using the time ranges it asked for at setup:
private func handleTranscriberResult(_ result: SpeechTranscriber.Result) async {
hasOpenUtterance = !result.isFinal
// A result is stale only when its audio ENDS before the line began.
guard result.range.end > cueStartTime else { return }
let text: String
if awaitingPreviousFinal {
if result.isFinal {
awaitingPreviousFinal = false
let filtered = Self.wordsSpoken(startingAfter: cueStartTime, in: result.text)
guard !filtered.text.isEmpty else { return }
text = filtered.text
} else if result.range.end > cueStartTime + Self.awaitFinalLimit { // 3 s safety valve
awaitingPreviousFinal = false
text = String(result.text.characters)
} else {
return // hold volatiles until the previous utterance is finalized
}
} else {
text = String(result.text.characters)
}
await cueSession?.handlePartial(text, isFinal: result.isFinal, tailConfidence: Self.tailConfidence(from: result.text))
}The staleness rule is the subtle part. A result is stale only when its audio ends before the current line started. Gating on range start instead throws away real volatiles, because a volatile’s range starts at the last finalization point, which can be long before the actor spoke. When a final straddles the line boundary, the app keeps only the words whose own range begins after the cue start:
private static func wordsSpoken(startingAfter start: CMTime, in text: AttributedString) -> (text: String, droppedWords: Int) {
var kept = "", dropped = 0
for run in text.runs {
if let range = run[AttributeScopes.SpeechAttributes.TimeRangeAttribute.self], range.start < start {
dropped += 1
continue
}
kept += String(text[run.range].characters)
}
return (kept.trimmingCharacters(in: .whitespaces), dropped)
}The three second valve exists because the engine does not promise a final for every utterance. Without it, one missing final would freeze recognition for the rest of the scene.
The obvious fix, forcing the old utterance closed with SpeechAnalyzer.finalize(through:) when the line changes, does not work. The engine logs “call on finished analyzer; ignoring” and produces nothing for the rest of the session. The app then appears to do nothing at all.
The utterance that survived stop()
Six days later, a second report: a line advanced by itself about a second into silence, after a recording had been deleted from the line editor. Eight silent variants of the report passed on the device. It reproduced only once real speech entered the room.
SpeechAnalyzer keeps an utterance open across stop() and a full pipeline restart when the audio feed was paused for a reader line, because it never heard trailing silence. On the next Play, as soon as audio flows again, it re-emits that utterance as a volatile whose range runs from before the stop to now. The range end passes the stale guard. And because stop() had cleared hasOpenUtterance, the hold for a previous utterance was off, so the whole old line matched the new cue. The genuine final arrived right after and was correctly dropped as stale, which is what made the log so confusing.
The fix is one deliberate omission in two places. stop() clears the session state but leaves hasOpenUtterance alone, and the pipeline restart leaves both that flag and awaitingPreviousFinal alone. The next start() then arms the hold, the re-emitted volatile waits for the old final, and the final is filtered by word time exactly as for a line change mid rehearsal. The comment on that line in the source is longer than the code around it, on purpose.
When to fire
The state machine that decides whether a cue fired uses the volatile and final distinction in three ways. A final always advances if the cue matched. A volatile advances immediately only if the mean confidence of its last three words is at least 0.4; otherwise two consecutive matching volatiles are required. And when the cue was reached only through the tracker’s trailing slack, meaning the last one or two words are still unmatched, the advance is held until a final arrives, because the actor is probably still saying them. That last rule is on for SpeechAnalyzer and off for the old engine, whose isFinal on device is too unreliable to wait for.
Per-word confidence is read straight off the attributed string:
private static func tailConfidence(from text: AttributedString) -> Float {
var values: [Float] = []
for run in text.runs {
if let c = run[AttributeScopes.SpeechAttributes.ConfidenceAttribute.self] { values.append(Float(c)) }
}
let tail = values.suffix(3)
return tail.isEmpty ? 0.5 : tail.reduce(0, +) / Float(tail.count)
}You cannot test any of this in the simulator
The simulator has no on-device speech assets for either engine, so no cue can ever fire there. The unit suite, roughly 770 tests with about 90 on the cue recognition core, covers the alignment, the coverage rules, the confidence gating and the state machine. The engines themselves are covered by a live harness on a physical phone.
The harness composes the real view model with no fakes, calls Play, and plays the actor’s lines as WAV files out loud so the sound crosses the air into the real microphone. It fails on a missed cue, a ghost advance before the actor spoke, a premature advance more than half a second before the words ended, and a double advance. Both bugs above were reproduced by it, dated, and turned into unit fixtures.
Typical healthy numbers from it: the cue fires 0.1 to 1.0 seconds after the actor’s audio ends in English, 0.4 to 1.4 seconds for the other languages, and a twelve line scene takes about fifty seconds.
The most transferable lesson from building it is about the fixtures, not the code. The stale final bug landed inside the reader’s line on three consecutive runs and the harness saw nothing, until the fixture generator appended a short tail of extra speech after the cue words so the final would land where a real person’s does. One run is not proof. Give a new failure mode a few runs before concluding.
One measurement worth knowing: iOS suppresses the phone’s own speaker output from its microphone, and what reaches the recognizer keeps only the part above about 1 kHz. SpeechAnalyzer decodes that residue. The old engine hears one word in ten, so the probes that exercise it play the actor through the Mac’s speaker instead, over a tiny socket server. When the phone’s local network prompt got in the way, the CoreDevice tunnel that Xcode already opens over the cable turned out to route to the Mac with no permission at all.
Still open
- Vocabulary hints never apply on iOS 26. Every
setContextfails with “Invalid JIT profile”. - Korean: the first utterance of a recognizer session is never recognized. Six rehearsals, six fresh recognizers, the first line the microphone delivers produces nothing or only its tail, and every later line is recognized within a second. The same clips fed from a file are never lost. Priming with a synthesized phrase passed once and then failed twice.
- The first non-English rehearsal on a device without that keyboard language fails once while the assets install, then works.
What transfers
None of this is specific to a rehearsal app. If you feed SpeechAnalyzer from a live microphone, stamp every buffer with a start time and treat results as positioned on that timeline, not as a sequence you can trust to be in order. If your product reacts to speech rather than just displaying it, keep the reaction logic in a pure, injectable state machine with its own tests, and build the device harness before the first ghost report, not after the third.
Frequently asked questions
Does SpeechAnalyzer work in the iOS simulator? No. The simulator has no on-device speech assets, so SpeechTranscriber.isAvailable is false and nothing is transcribed. Test on a device.
What audio format does it want? 16 kHz mono 16-bit integer PCM is the safe choice. Convert with AVAudioConverter inside the input tap and pass bufferStartTime on every AnalyzerInput.
Can I change the contextual strings without restarting? The API exists: build an AnalysisContext, set contextualStrings[.general], and call setContext on the running analyzer. In my testing every call fails with SFSpeechErrorDomain Code=6 "Invalid JIT profile" on iOS 26, and the error is easy to miss if you use try?.
Does the user need anything installed? SpeechAnalyzer downloads its own assets through AssetInventory on first use, about thirty seconds on Wi-Fi. The legacy on-device SFSpeechRecognizer instead depends on the locale’s dictation assets, which arrive only when that keyboard language is added in Settings; a locale without them never returns a result.
Scriptmate is on the App Store, built solo, and the full engineering story including the audio routing and the script parsing pipeline is in the case study.