Introducing AnySpeechRecognizer and AnyDecisionModel: Open Models for Speech and Decisions on Apple Platforms

Update on GitHub

Upvote 17

Language models can do a lot. They write, summarize, translate, and call tools. But they're only a small part of what machine learning can do.

Say a customer calls your support line. First, you want to know what they said (speech recognition). Then, you want to decide where to send them (decision). You could hand both to a language model, but there are models built for each that are better, faster, and less expensive.

In our AnyLanguageModel 1.0 recap, we looked at a year of local and hosted models in Swift. With AnyLanguageModel and swift-huggingface now on their 1.x releases, we're extending the same approach to speech and decisions: familiar Swift APIs with a choice of model.

Today we're announcing two packages:

• AnySpeechRecognizer implements Apple's new speech-to-text API with a choice of recognition engine, including NVIDIA's Parakeet and Alibaba's Qwen3-ASR running locally on Core ML, and adds speaker diarization to track who speaks when.
• AnyDecisionModel answers yes-or-no, multiple-choice, and rating-scale questions with typed results and probabilities. It derives each answer from the model's probabilities instead of generating text, using a model on your device with MLX, Core AI, or llama.cpp, or a decision model from Jev or Ollama.

Both packages can download open models from the Hugging Face Hub and run them on device.

AnySpeechRecognizer

Computers have been transcribing speech for decades, and in the past few years, they've gotten very good at it. Apple platforms already have an API for it. macOS 26 and iOS 26 introduced

SpeechAnalyzer

, Apple's new speech-to-text API. You add modules such as

SpeechTranscriber

and

SpeechDetector

to an analyzer, give it a file or a stream of audio buffers, and read results from async sequences as they arrive.

AssetInventory

installs the models, so your app decides when downloads happen.

It's a well-designed API, but the only engine behind it is Apple's. AnySpeechRecognizer keeps the API and lets you choose the engine:

- import Speech
+ import AnySpeechRecognizer

The package mirrors

SpeechAnalyzer

,

SpeechTranscriber

,

DictationTranscriber

,

SpeechDetector

, and

AssetInventory

, with Apple's Speech framework as the default engine. If your code uses that shared API, changing the import is enough. To use Parakeet instead, change only how you create the transcriber:

let transcriber = SpeechTranscriber(
    model: ParakeetSpeechModel(variant: .v3),
    transcriptionOptions: [],
    reportingOptions: [.volatileResults],
    attributeOptions: [.audioTimeRange]
)

The steps for installing assets, running the analyzer, and reading results stay the same. The README has a complete example and lists the differences from Apple's framework.

Meet the models

Parakeet v3 is NVIDIA's Parakeet TDT 0.6B v3. It's a polyglot: it recognizes 25 European languages and works out which one it's hearing, so you don't pass a locale.

Parakeet Ultra is Moondream's post-trained take on v3. It was more accurate in most of our measurements.

Parakeet Unified is NVIDIA's English-only Parakeet.

All three Parakeets run on the package's own Core ML runtime, which supports file and live input, volatile results, word timing, and word confidence.

Qwen3-ASR is Alibaba's speech model, in 0.6B and 1.7B sizes. It recognizes 30 languages and 22 Chinese dialects, including Mandarin, Cantonese, Japanese, and Korean, which Parakeet doesn't cover. It detects the language automatically, or you can set one:

let transcriber = SpeechTranscriber(
    model: QwenSpeechModel(variant: .large, language: Locale.Language(identifier: "yue")),
    transcriptionOptions: [],
    reportingOptions: [],
    attributeOptions: []
)

Qwen3-ASR gives one final result for each segment of up to 40 seconds, without volatile results, word timing, or confidence. It runs on the GPU and is slower than Parakeet, so it's better suited to transcribing files than live input.

To add another engine, implement

SpeechRecognitionModel

for its configuration and

SpeechRecognitionExecutor

for one module's session. The README includes a small provider that reports placeholder text to help you get started.

Core ML models on the Hub

To run these models in apps, we converted them to Core ML and published them on the Hub:

Model	Core ML conversion	Original checkpoint	License
Parakeet TDT 0.6B v3	mattt/parakeet-tdt-0.6b-v3-coreml	nvidia/parakeet-tdt-0.6b-v3	CC BY 4.0
Parakeet Ultra	mattt/parakeet-ultra-coreml	moondream/parakeet-ultra	CC BY 4.0
Parakeet Unified EN 0.6B	mattt/parakeet-unified-en-0.6b-coreml	nvidia/parakeet-unified-en-0.6b	NVIDIA Open Model License
Qwen3-ASR 0.6B	mattt/qwen3-asr-0.6b-coreml	Qwen/Qwen3-ASR-0.6B-hf	Apache 2.0
Qwen3-ASR 1.7B	mattt/qwen3-asr-1.7b-coreml	Qwen/Qwen3-ASR-1.7B-hf	Apache 2.0
Streaming Sortformer 4spk v2	mattt/diar-streaming-sortformer-4spk-v2-coreml	nvidia/diar_streaming_sortformer_4spk-v2	CC BY 4.0
Streaming Sortformer 4spk v2.1	mattt/diar-streaming-sortformer-4spk-v2.1-coreml	nvidia/diar_streaming_sortformer_4spk-v2.1	NVIDIA Open Model License

Each repository records its source checkpoint and a checksum for every file, and the package pins it to a fixed revision. The Parakeet repositories add an 8-second encoder for live input, which cuts live analysis time by about a third at the cost of a somewhat higher word error rate. The repository also has the conversion workflows, most of which follow FluidInference's mobius scripts, and checks each conversion against the original model.

Downloads happen only when your app asks for them: about 600 MB for each Parakeet, plus about 590 MB for the live encoder, 0.9 to 2.4 GB for Qwen3-ASR, and about 250 MB for Sortformer. Files go in the standard Hugging Face cache, so a Mac app outside the sandbox shares them with Python and other Hugging Face tools.

Speaker diarization

When you transcribe a conversation, you often want to know who said what. That's speaker diarization, and Apple's Speech framework doesn't have a module for it, so AnySpeechRecognizer adds one.

SpeakerDiarizer

runs NVIDIA's Streaming Sortformer on Core ML alongside an Apple or Parakeet transcriber, on the same timeline:

let diarizer = SpeakerDiarizer(model: SortformerDiarizationModel())
if let request = try await AssetInventory.assetInstallationRequest(
    supporting: [transcriber, diarizer]
) {
    try await request.downloadAndInstall()
}
let analyzer = SpeechAnalyzer(modules: [transcriber, diarizer])

The diarizer tracks up to four speakers, even when they talk over each other. A

SpeakerTimeline

collects its results and attributes each timed word to a speaker, so you can print a transcript by speaker turn. Like image support in AnyLanguageModel, this goes beyond what Apple's API offers today.

The diarizer uses Sortformer v2 by default. You can also choose v2.1, which NVIDIA trained with more meeting data and which was more accurate on meetings in our measurements, too.

How well does it work?

We measured the package with its

hear

command-line tool on an M4 Max Mac Studio with macOS 27.0.1. Here are the word error rates on LibriSpeech files of 3 to 8 minutes:

Transcriber	test-clean	test-other
Parakeet v3, file	1.55%	2.78%
Parakeet v3, live	4.73%	4.29%
Parakeet Ultra, file	1.66%	2.55%
Parakeet Ultra, live	2.65%	3.28%
Parakeet Unified, file	1.43%	2.19%
Parakeet Unified, live	1.45%	2.42%
Apple's transcriber, file	2.12%	3.97%

Parakeet transcribes an hour of audio in about 20 to 40 seconds from a file, or about 1.5 to 2.5 minutes live, with about 100 MB of memory once Core ML has compiled the model. Qwen3-ASR is much slower and needs a few gigabytes of memory. Meetings are hard for everyone: on AMI recordings, where people talk over each other, every engine we tested had a word error rate above 23%.

These numbers come from one machine and a few datasets, so measure with your own hardware and audio.

hear evaluate

works on your own files, and the repository's

natural-speech-eval.py

tool reproduces these measurements. So far, we've tested only on macOS 27, not on an iPhone.

Next, we'd like to add more engines, including Core AI on macOS 27 and iOS 27.

AnyDecisionModel

AnyDecisionModel is a different story. There's no Apple framework to map onto, and decision models aren't new, exactly.

It's only a decision model if it comes from the Décision region of France; otherwise, it's sparkling zero-shot classification.

What's new is the name, a lot of hype, and a crop of models trained to make decisions. We think that's where things are going, so we wanted a Swift API that's ready for them.

Apps often ask language models to make decisions. Is this message urgent? Which team should handle this ticket? How frustrated is this customer? The usual approach is to prompt a language model and parse the text that it generates. That gives you one answer, but doesn't show the probabilities it assigned to the alternatives.

Here's what a decision looks like in Swift:

import AnyDecisionModel

enum Department: String, Choosable {
    case billing, technical, sales
}

let ticket = "I was charged twice for my order, and now the app logs me out when I try to ask for a refund."

let model = MLXDecisionModel(modelID: "mlx-community/Qwen3-4B-Instruct-2507-4bit")
let session = DecisionSession(model: model, state: .text(ticket))

let urgent = try await session.probability(of: "Does this convey urgency?")
let team = try await session.choice("Which team?", from: Department.self)
let anger = try await session.score(
    "How frustrated?",
    levels: ["Calm", "Frustrated", "Very angry"]
)

urgent

is a

Double

.

team

is a

Choice<Department>

containing the selected case, a probability for each option, and a confidence value.

anger

is a

Score

containing the expected level, a probability for each level, and a confidence value. Because

Department

conforms to

Choosable

,

team.value

is a

Department

that you can use in an exhaustive

switch

.

If you've used Foundation Models or AnyLanguageModel, the model-and-session pattern will look familiar. A

DecisionSession

holds a single state, which can be text or JSON, and every question is about that state. Questions are independent: an answer never becomes input to a later question. To ask several questions at once, pass them to

decide(_:)