Modulate AI

Pros and cons

About Modulate

Modulate is a speech-to-text platform specifically designed to address the limitations of legacy models, tackling even the hardest audio while maintaining exceptional accuracy.

Uniquely, they skip testing against clean benchmarks like LibriSpeech or CommonVoice, which tend to produce extremely, and even unreasonably, low WERs. Instead, they have challenged their model against 500+ million hours of only the most difficult, real-world benchmarks.

It is because of the additional challenges that Modulate reports that Velma-2, their flagship STT model, achieves an average 7.8% WER.

Against the AMI Meeting Corpus, known for overlapping, noisy audio with multiple speakers, Velma achieves a WER of 14.9%. Against Earnings-22, Velma achieves a WER of 7.5%, and 8% against VoxPopuli.

It achieves such accuracy via its conversation-understanding engine and voice-native architecture, which retains vital acoustic and behavioral signals, such as prosody and tone, that are often lost with legacy or repurposed STT models.

Despite such accuracy on both live streaming and pre-recorded audio, a full, high-level narrative might not be immediately available until the call is processed in batch mode. This is because the most computationally challenging forensic analysis occurs asynchronously.

Most of the features that businesses, particularly enterprises, need in modern production environments, such as speaker diarization and emotion detection, are offered within Modulate's base rate.

Additional features like PII redaction are offered at a rate of $0.02/hour on top of the base transcription rate.

Note, however, that if you're moving from a monolithic model to an ensemble model such as Modulate's Velma, you might need a different technical management approach. The system is designed for flexibility, and Modulate offers continuous support to handle such migration.

Below, we dive deeper into Modulate's Velma model and its accuracy, features, pricing, and more.

Modulate's Velma STT model

Modulate's flagship STT model is Velma-2, designed to handle volumes of complex and technical audio with a high degree of accuracy. Because of Velma's unique architecture, the model captures over 150 behavioral signals such as harassment, burnout signals, even gratitude or price sensitivity that tie directly to business outcomes or potential losses.

• Velma-2 Transcribe

  Flagship

  Ensemble speech-to-text built for chaotic, overlapping, real-world audio. Analyzes 150+ behavioral cues alongside the transcript.

  • $0.03/hr batch
  • $0.06/hr streaming
  • 57 languages
  • Real-time diarization
  • Emotion, accent, and stress detection included
  • Structured JSON with word-level timestamps

• Velma-2 English Fast

  Batch

  Throughput-optimized English-only variant for high-volume pre-recorded audio where cost per hour matters most.

  • $0.025/hr
  • Up to 200x realtime factor
  • English only

• Velma PII/PHI Redact

  Add-on

  Dedicated redaction model that returns a compliant transcript plus a redacted audio file, for regulated call environments.

  • $0.05/hr batch
  • $0.08/hr streaming
  • Redacted audio output
  • Transcription + PII/PHI tagging

• Domain sub-models

  Ensemble

  Narrow expert models (e.g. healthcare, deepfake detection) orchestrated by Velma's shared layer, which swaps models dynamically mid-conversation when it detects specialized jargon.

  • Mid-call model swapping
  • Custom behavior packs

This positions Modulate heavily in the deepfake detection and fraud prevention space, in addition to speech-to-text. Users can also use the shared orchestration layer, which allows for dynamic model swapping, even mid-conversation. For instance, if the system detects medical terminology during a call, it will automatically switch from a generalist model to a healthcare model.

Because of the company's early stages, there are limited reviews on how this accuracy and fraud detection translates to business results.

Modulate's Velma-2 features and capabilities

While Velma isn't the most feature-rich platform on the market, it includes many of the most instrumental features in its base rate, with the exception of PII redaction, which is offered at $0.02/hr.

Core Velma-2 features

• Emotion detectionMonitors for signals like stress, anger, and other signs of churn or escalation.
• Accent detectionDetects accent spoofing and improves transcription accuracy.
• Speaker diarizationIdentifies and labels each speaker in a call.
• 57 dialectsLanguage support across 57 dialects for greater accuracy in multilingual calls.
• PII/PHI redactionProtects callers and your organization while remaining compliant.

Modulate also offers music detection. Fraudsters may add music to the background of their calls to throw off other detectors. Modulate's music detector is trained to catch AI-generated music and identify the deepfake without false positives.

This feature is a separate add-on that further adds to Modulate's positioning as a fraud detection tool wrapped in everyday STT.

Features included in Velma inherently

Velma utilizes a 7-layer capture system to analyze audio dynamically, meaning it runs tone and emotion detection alongside prosody analysis, deception and stress cues, and acoustic authenticity. Its modular nature allows you to configure these features:

• Cross-talk and overlap resolutionHandles natural interruptions, colloquial interjections, and cross-talk.
• Acoustic filteringStrips out keyboard typing, ambient room noise, and music without separate pre-processing tools.
• Automated formattingStructured JSON payloads with word- and segment-level timestamps, capitalization, and readable punctuation.
• Deception indicatorsFlags patterns associated with vocal stress, lying, coercion, or social engineering.
• Sarcasm and intent parsingDifferentiates between identical phrases with opposite meanings based on tone.
• Customer retention analyticsSignals attrition and dissatisfaction risks so managers can intervene before a call ends.
• Compliance risk monitoringScreens for policy violations, disclosure failures, script deviations, or inappropriate content.
• Vulnerable customer protectionIdentifies extreme distress or confusion in callers, and monitors agents for burnout or abusive callers.
• Hallucination and policy detectionMonitors automated voice agents in real time for inaccurate claims or policy violations.
• Caller response trackingMeasures how callers react emotionally to an AI agent's pacing, tone, and answers.
• Tailored behavioral promptingDescribe a conversational pattern in natural language and Velma surfaces that behavior across your streams.
• Accent and dialect detectionRecognizes regional and global accents so behavioral analysis stays accurate across populations.

The architecture of Modulate's Velma STT model

Modulate's Velma STT model is constructed on an Ensemble Listening Model.

The model operates on several specialized sub “expert” models with narrow mandates, such as detecting emotional states or deepfakes. Because of the ensemble, Velma can detect over 150 behavioral signals, making it an option for fraud detection, training, and compliance.

It also uses a shared, or orchestration, layer that reasons over overlapping signals, even swapping models when it makes sense, such as when there is heavy jargon.

Because Velma is a modular model, you can customize it with your SOPs or new behavior packs and sub-models without retraining the entire system.

Processing modes

Live-streaming

Immediate alerts on live audio

Batch

Asynchronous deep forensic analysis

Velma-2 deployment and infrastructure

Modulate's Velma, along with its analytics engine (ToxMod), supports high-security, low-latency, and sovereign data requirements with these deployment options:

• Fully managed cloud: live stream audio via WebSockets or upload batch audio via HTTPS REST to Modulate's hosted endpoints.
• Virtual Private Cloud (VPC): Modulate spins up a single-tenant instance of their Ensemble Listening Model within a dedicated AWS VPC. Data is isolated within that environment and retention is fully configurable.
• IP whitelisting and secure VPN tunnels: for non-AWS infrastructure, lock down communication via site-to-site VPN tunnels or strict IP address restrictions.

Modulate's Velma performance and accuracy metrics

Modulate reports topping the Hugging Face ASR leaderboard against legacy systems and some of the most challenging public benchmarks. The company then tests and retests its models against 500 million hours of real-world audio.

They have published their own benchmark reporting, along with comparisons against other models, both on their website and across various other sources. Those metrics show how Velma-2 fares against benchmarks such as Earnings-22 and AMI Meeting Corpus.

The AMI Meeting Corpus tends to push even prominent legacy systems well into higher WERs, yet Velma maintains one of the lower published scores for that benchmark.

Because Modulate concentrates its testing against the hardest benchmarks rather than clean ones, you might notice WERs that are higher than some competitors' published figures.

Modulate pricing for Velma-2

Modulate keeps their pricing simple, offering a self-serve and enterprise option for their transcription model, Velma-2.

They offer their model at $0.03/hour for pre-recorded audio and $0.06/hour for live-streaming, and many of their features are automatically included in the base rate. This often grants businesses a lower cost of ownership as they scale.

Additionally, Modulate provides new users with 400 hours in free credits via their self-serve option.

Additional features such as PII redaction run $0.05/hour for batch and $0.08/hour for streaming as dedicated redaction models.

Modulate customer support

While there aren't a lot of details from users about Modulate's customer service, Modulate positions itself as a collaborative partner, particularly for its enterprise customers.

They offer integration support as well as continuous model calibration, which is helpful considering the dynamic ensemble model may require additional developer support.

For enterprise customers

• Call stack integration: Modulate works directly with enterprise teams to fit its voice-native engine into existing application stacks via APIs and webhooks.
• Privacy and compliance calibration: data minimization, custom retention policies, and regional data residency constraints.
• Configured insights: support in configuring behavioral insights and “behavior packs” tailored to business needs.
• Custom feature development priority: enterprise clients on custom contracts can influence the product roadmap with priority engineering resources.

For developers

• Emergency on-call escalation: critical production failures or severe streaming latency spikes route directly to on-call engineers.
• Comprehensive data diagnostics: their engineering support pipeline expects structured request data (HTTP/WebSocket close codes, request IDs, audio formatting data, payload durations) to resolve infrastructure errors quickly.
• Reproducible benchmarking, a public demo, and 400 hours of free credits to support evaluation.

For self-service customers

• Velma Preview browser tool: a web-based sandbox where developers can upload audio or test microphone inputs to see how the models capture transcription text, emotion, accents, and deepfakes before writing code.
• LLM-friendly documentation: an organized developer portal alongside an optimized /llms.txt index file for ingesting API specifications into LLM-driven coding tools.

FAQs