FFmpeg’s First AI Feature: Whisper Audio Transcription Filter

FFmpeg breaks new ground by integrating its first AI-driven capability: the Whisper-based af_whisper audio filter, arriving as part of the upcoming FFmpeg 8.0 release .

This filter, powered by OpenAI’s Whisper via the whisper.cpp runtime, enables automatic speech recognition (ASR) directly within FFmpeg’s command-line toolchain. Users can transcribe audio files or live streams into plain text, SRT subtitle files, or JSON metadata, simplifying workflows that previously required separate transcription tools or services

Key features include:

• Local processing, eliminating the need to send data to cloud services .

• GPU acceleration support, enhancing transcription speed for capable hardware.

• Voice Activity Detection (VAD) and model tuning options, allowing users to balance transcription accuracy and performance

• Flexible output formats (text, SRT, JSON), with streamlined conversion workflows ideal for content creators and streamers

The filter strategically positions FFmpeg as an intelligent media processing toolkit—transforming it from a traditional transcoding suite into a tool capable of generating transcripts and captions seamlessly. This milestone paves the way for future AI-powered filters while preserving FFmpeg’s core strengths in speed and flexibility