Speech Framework I've been checking for SFSpeechRecognitionMetadata to determine the end of a sentence when using Voice Recognition. Yet it doesn't detect small pauses but only large ones, so that I've transcribed basically an entire paragraph before going onto the next one. Besides implementing your own timer, are there any other ways to have more natural pauses to detect the end of sentences, similar to the browser's Web Speech recognition? Since it's in Safari, I assume there should be some similar feature that can be equivalent in MacOS.

I'm making a Safari extension for learning languages. I need speech synthesis for any language the user chooses to learn. I initially tried to make this work within JavaScript, but Safari 18 doesn't reliably list voices for all languages on the web SpeechSynthesis API as described here: https://stackoverflow.com/questions/79179072/how-do-you-use-a-japanese-voice-with-speechsynthesis-in-safari-ios-18 As a workaround, I've had to use AVSpeechSynthesizer in SafariWebExtensionHandler (NSExtensionRequestHandling implementation for the extension). This works in the simulator but not on a real device. I've found this note from Apple in a StackOverflow reply: "Safari extensions are very short-lived, hence not fit for audio playback or speech synthesis. Not being able to validate an app extension in Xcode with a manually-added plist entry for background audio is the designed behavior. The general recommendation is to synthesize speech using JavaScript in conjunction with the Web Speech API." Unfortunately, the suggestion to use the Web Speech API is unsuitable as I just explained. Is there a way to set up a background process in the host app that can do speech synthesis? The app extension would need a way to communicate with this process, and start it if it's not running. Is that possible?

Hi, I'd like to develop an app which runs speech recognition even after going into background. I know I can accomplish this using audio background mode and the process the audio but I am not sure if this workaround would get accepted into App Store because of the processing limitations while in the background. How can I accomplish this while still being compliant with Apples privacy policy and other restrictions? Thanks, Marek

Hello all, I'm working on a project that involves listening to a person speak off of a script and I want to stop then restart the recognitionTask between sections so I don't run afoul of keeping the recognitionTask running for longer than it needs to. Also, I'd like to be able to flush the current input between sections so the input from the previous section doesn't roll over into the next one. This is based on the sample code for SFSpeechRecognizer so there's a chance I might be misunderstanding something. private func restartRecording() { let inputNode = audioEngine.inputNode audioEngine.stop() inputNode.removeTap(onBus: 0) recognitionRequest?.endAudio() recordingStarted = false recognitionTask?.cancel() do { try startRecording() } catch { print("Oopsie.") } } Here's my code. When I run it, the recognition task doesn't restart. Any ideas?

Description: I have encountered an issue with SFSpeechRecognizer on iOS 18.0. During live dictation, if a natural pause (e.g., 1-2 seconds) is introduced, the previously transcribed text is cleared, and the transcription starts over. This behavior makes it difficult to use the API for real-time speech recognition scenarios where pauses are expected. Steps to Reproduce: Open Apple's demo app "SpokenWord". Start the dictation process using SFSpeechRecognizer. Speak a few words, pause for 1-2 seconds, and then continue speaking. Observe that the previously transcribed text is truncated, and the transcription starts anew. Expected Behavior: The transcription should continue appending new results to the previous ones after a natural pause, maintaining a seamless user experience. Observed Behavior: After a pause, the transcription resets, clearing previously transcribed text. Impact: This behavior makes the SFSpeechRecognizer API unreliable for scenarios requiring continuous speech recognition with intermittent pauses. Additional Information: iOS Version: 18.0 Device: [Specify your device, e.g., iPhone 13 Pro] Speech Recognizer Locale: [Specify locale, e.g., en-US] App Behavior: Issue persists in both Apple's demo app ('SpokenWord') and custom implementations.

I see this error in the debugger: #FactoryInstall Unable to query results, error: 5 IPCAUClient.cpp:129 IPCAUClient: bundle display name is nil Error in destroying pipe Error Domain=NSCocoaErrorDomain Code=4099 "The connection from pid 5476 on anonymousListener or serviceListener was invalidated from this process." UserInfo={NSDebugDescription=The connection from pid 5476 on anonymousListener or serviceListener was invalidated from this process.} on this function: func speakItem() { let utterance = AVSpeechUtterance(string: item.toString()) utterance.voice = AVSpeechSynthesisVoice(language: "en-GB") try? AVAudioSession.sharedInstance().setCategory(.playback) utterance.rate = 0.3 let synthesizer = AVSpeechSynthesizer() synthesizer.speak(utterance) } When running without the debugger, it will (usually) speak once, then it won't speak unless I tap the button that calls this function many times. I know AVSpeech has problems that Apple is long aware of, but I'm wondering if anyone has a work around. I was thinking there might be a way to call the destructor for AVSpeechUtterance and generate a new object each time speech is needed, but utterance.deinit() shows: "Deinitializers cannot be accessed"

I was testing SFSpeechRecognition on my real device running ios 18.2 beta, and found that the result's "final" field is true, the result itself does not contain entire conversation's transcription. I came across some blog posts saying it's fixed in a 18.1 beta, is this not the case for 18.2 beta? Example code: recognitionTask = recognizer.recognitionTask(with: request) { [weak self] result, error in guard let self = self else { return } if let error = error { DispatchQueue.main.async { self.errorMessage = "Transcription failed: \(error.localizedDescription)" self.isTranscribing = false } } else if let result = result, result.isFinal { // HERE! } }

Hi, Apple's engineer. Hoping that you can reply to this one. We're developing a Text-to-Speak app. Everything went well until the IOS got upgraded to 18. AVSpeechSynthesisVoice(language: "zh-CN") is running well under IOS 16 AND IOS 17. It speaks Mandarin correctly. In IOS 18, we noticed that Siri's Language setting interrupted the performance of AVSpeechSynthesisVoice. It plays Cantonese instead of Mandarin. Buggy language setting in Siri that affects the AVSpeechSynthesisVoice : Chinese (Cantonese - China mainland) Chinese (Cantonese -Hong Kong)

I am attempting to do batch Transcription of audio files exported from Voice Memos, and I am running into an interesting issue. If I only transcribe a single file it works every time, but if I try to batch it, only the last one works, and the others fail with No speech detected. I assumed it must be something about concurrency, so I implemented what I think should remove any chance of transcriptions running in parallel. And with a mocked up unit of work, everything looked good. So I added the transcription back in, and 1: It still fails on all but the last file. This happens if I am processing 10 files or just 2. 2: It no longer processes in order, any file can be the last one that succeeds. And it seems to not be related to file size. I have had paragraph sized notes finish last, but also a single short sentence that finishes last. I left the mocked processFiles() for reference. Any insights would be greatly appreciated. import Speech import SwiftUI struct ContentView: View { @State private var processing: Bool = false @State private var fileNumber: String? @State private var fileName: String? @State private var files: [URL] = [] let locale = Locale(identifier: "en-US") let recognizer: SFSpeechRecognizer? init() { self.recognizer = SFSpeechRecognizer(locale: self.locale) } var body: some View { VStack { if files.count > 0 { ZStack { ProgressView() Text(fileNumber ?? "-") .bold() } Text(fileName ?? "-") } else { Image(systemName: "folder.badge.minus") Text("No audio files found") } } .onAppear { files = getFiles() Task { await processFiles() } } } private func getFiles() -> [URL] { do { let documentsURL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first! let path = documentsURL.appendingPathComponent("Voice Memos").absoluteURL let contents = try FileManager.default.contentsOfDirectory(at: path, includingPropertiesForKeys: nil, options: []) let files = (contents.filter {$0.pathExtension == "m4a"}).sorted { url1, url2 in url1.path < url2.path } return files } catch { print(error.localizedDescription) return [] } } private func processFiles() async { var fileCount = files.count for file in files { fileNumber = String(fileCount) fileName = file.lastPathComponent await processFile(file) fileCount -= 1 } } // private func processFile(_ url: URL) async { // let seconds = Double.random(in: 2.0...10.0) // await withCheckedContinuation { continuation in // DispatchQueue.main.asyncAfter(deadline: .now() + seconds) { // continuation.resume() // print("\(url.lastPathComponent) \(seconds)") // } // } // } private func processFile(_ url: URL) async { let recognitionRequest = SFSpeechURLRecognitionRequest(url: url) recognitionRequest.requiresOnDeviceRecognition = false recognitionRequest.shouldReportPartialResults = false await withCheckedContinuation { continuation in recognizer?.recognitionTask(with: recognitionRequest) { (transcriptionResult, error) in guard transcriptionResult != nil else { print("\(url.lastPathComponent.uppercased())") print(error?.localizedDescription ?? "") return } if ((transcriptionResult?.isFinal) == true) { if let finalText: String = transcriptionResult?.bestTranscription.formattedString { print("\(url.lastPathComponent.uppercased())") print(finalText) } } } continuation.resume() } } }

While running Swift's SpeechRecognition capabilities I get the error below. However, the app successfully transcribes the audio file. So am not sure how worried I have to be, as well, would like to know that if when that error occurred, did that mean that the app went to the internet to transcribe that file? Yes, requiresOnDeviceRecognition is set to false. Would like to know what that error meant, and how much I need to worry about it? Received an error while accessing com.apple.speech.localspeechrecognition service: Error Domain=kAFAssistantErrorDomain Code=1101 "(null)"

VoiceOver does not support the plist property CFBundleSpokenName. This is wrong and should be fixed. Ultimately the issue I am dealing with is that our app name is UWCU, and instead of VoiceOver pronouncing each letter, it tries to read this as a word and horribly butchers our organization's/app's name. Alternatives such as using U.W.C.U. and U W C U are not acceptable. @Apple, I know you're first response is going to be "no, it is working perfectly," but quite frankly you are wrong. I know you feel strongly about this, given your response in posts like this: https://forums.developer.apple.com/forums/thread/734545?answerId=760084022 HOWEVER, with iOS 18, your argument for "VoiceOver should read what's on the screen" doesn't hold water anymore. With iOS 18, you Apple have added a new feature that lets users customize their home screens and completely remove the name of apps. Here's your own guide: https://support.apple.com/guide/iphone/customize-apps-and-widgets-on-the-home-screen-iph385473442/ios Quoted from your guide: Make the icons bigger: Tap Large. (In large size, the names of the apps disappear.) With large icons + VoiceOver turned on, VoiceOver still reads the app name even though it has disappeared from the screen. So, your own argument "VoiceOver should read the text as it appears on the screen" is invalid, because there is NO text on the screen. If you can't tell, I'm pretty peeved about all this. There's a reason why screen readers support aria attributes to help deliver the right accessible experience. It's a simple ask for VoiceOver to do the same thing.

Can anyone please guide me on how to use SFCustomLanguageModelData.CustomPronunciation? I am following the below example from WWDC23 https://wwdcnotes.com/documentation/wwdcnotes/wwdc23-10101-customize-ondevice-speech-recognition/ While using this kind of custom pronunciations we need X-SAMPA string of the specific word. There are tools available on the web to do the same Word to IPA: https://openl.io/ IPA to X-SAMPA: https://tools.lgm.cl/xsampa.html But these tools does not seem to produce the same kind of X-SAMPA strings used in demo, example - "Winawer" is converted to "w I n aU @r". While using any online tools it gives - "/wI"nA:w@r/".

So, I'm trying to create my own text-to-speech setup. Problem I'm having is whenever I do a test run, the speech gets a bit choppy at the start kind of skipping over maybe a word or a few characters. A few details: I've essentially built a separate class for handling the speech events. AVSpeechSynthesizer is set up as a private variable for the class so I don't expect deallocation to be the issue. Especially since it's a problem at the start. I've got a queue set up for what it's worth so that shouldn't be a problem. I'd appreciate any advice.

Hello. I can't find anything about the SSML that is used in Apple's speech synthesis. SSML from Google, Amazon and W3C either don't work or work incorrectly. Where is Apple's documentation for their implementation of SSML?

Have there been any hints that Apple may offer speech diarization services (speech recongnition which recognizes multiple speakers) in the near future?

Here is the demo from Apple's site This issues is specific to iOS 18. When running this demo, we are getting new text when we have a gap in speaking, the recognitionTask(with:resultHandler:) provides new text which is only spoken after the gap and not the concatenation of old text and the new spoken text.

Hello, I noticed that SFSpeechRecognizer is broken on iOS 18. During a recognition task, it keeps dropping the recognized text on every pause. For example, if you say "how are you fine", it will drop the "how are you" part and only give you "fine" as the result. Say "how are you <pause> fine" // iOS 17 ✅ (perfect final result) How How are How are you How are you. How are you. Fine. // iOS 18 ❌ How How are How are you How are you Fine (the text before the pause is dropped, and fail to recognize the punctuations.) Reproducing the issue: Download the official sample project. Run it on an iOS 18 device or simulator. Say "how are you fine" Only "fine" will be displayed.

let debugString = "<speak><emphasis level=\"reduced\">Hello</emphasis></speak>" let utterance = AVSpeechUtterance(ssmlRepresentation: debugString)! // <--- Freezes I encountered this bug in iOS 18 beta I sent a feedback through Feedback app.

It looks like Apple has added some new API(s) to SFSpeechRecognition My app, which is currently listed on App Store does feature speech recognition. Yet, trying to use it under iOS 18.0 throws errors: -[SFSpeechRecognitionTask localSpeechRecognitionClient:speechRecordingDidFail:]_block_invoke Ignoring subsequent local speech recording error: Error Domain=kAFAssistantErrorDomain Code=1101 "(null)" What happens is that after several words are transcribed and displayed, the next sentence results in previous words disappearance. That's probably what that portion of the error text - "Ignoring subsequent local speech recording error: Error Domain=kAFAssistantErrorDomain Code=1101 "(null)" means. The problem occurs ONLY when the app is running under iOS 18.0 Even when it's compiled in Xcode 16.0 using iOS 17.5 everything works fine. Any suggestions?

I'm writing an app that uses on-device voice to text for recognising scientific terms. It works fine on my phone but now in beta my first tester cannot make it work. All the permission requests are working: p&s Mic and Speech Recognition are both now enabled on the target device where the user granted the app permission. Is there something else I'm missing? Incidentally, both my phone, the target phone and my XCode are fully up to date. Thanks.