Audio Speaker Tools

Tools for speaker separation, voice comparison, and audio processing using Demucs, pyannote, and Resemblyzer.

Overview

This skill provides three main workflows:

1. 1. Speaker separation - Extract per-speaker audio from multi-speaker recordings
2. Voice comparison - Measure speaker similarity between two audio files
3. Audio processing - Segment extraction and voice isolation

Prerequisites

Setup Virtual Environment

Run once to create the venv and install dependencies:

CODEBLOCK0

Default venv location: INLINECODE0

Requirements:

• - Python 3.9+
• ffmpeg (brew install ffmpeg)
• HuggingFace token (set as env var HF_TOKEN)

Scripts

1. Speaker Separation: diarize_and_slice_mps.py

Separate speakers from multi-speaker audio:

CODEBLOCK1

Process:

1. 1. Converts input to 16kHz mono WAV
2. Runs Demucs vocal/background separation (optional, for cleaner input)
3. Runs pyannote speaker diarization (MPS-accelerated)
4. Extracts concatenated per-speaker WAV files

Output:

• - <prefix>_speaker1.wav, <prefix>_speaker2.wav, etc. (one per detected speaker)
• INLINECODE6 (time-stamped speaker segments)
• INLINECODE7 (JSON segments metadata)
• INLINECODE8 (pipeline info and speaker index)

Important:

• - Always pass HF token via HF_TOKEN env var, never as CLI arg
• MPS first, CPU fallback - Script prefers Metal GPU, falls back to CPU if unavailable
• Default output: INLINECODE10

2. Voice Comparison: compare_voices.py

Measure similarity between two voice samples using Resemblyzer:

CODEBLOCK2

Scores:

• - < 0.75 = Different speakers
• INLINECODE13 = Likely same speaker
• INLINECODE14 = Excellent match (ideal for voice cloning validation)

Use cases:

• - Voice clone quality assessment (compare clone vs. original)
• Speaker verification (authenticate speaker identity)
• Validate speaker separation (confirm separated speakers are distinct)

See: references/scoring-guide.md for detailed interpretation

3. Audio Trimming

Use ffmpeg directly for segment extraction:

CODEBLOCK3

Workflows

Workflow 1: Extract Clean Voice Sample for Cloning

Goal: Get a clean, single-speaker sample for ElevenLabs voice cloning

CODEBLOCK4

See: references/elevenlabs-cloning.md for best practices

Workflow 2: Validate Voice Clone Quality

Goal: Measure how well a cloned voice matches the original

CODEBLOCK5

See: references/scoring-guide.md for troubleshooting low scores

Workflow 3: Multi-Speaker Conversation Analysis

Goal: Separate and identify speakers in a conversation

CODEBLOCK6

Technical Notes

Device Acceleration

• - pyannote diarization: MPS (Metal) by default, CPU fallback
• Resemblyzer: CPU only (no GPU acceleration)
• Demucs: MPS by default when available

To force CPU for diarization: INLINECODE19

Audio Formats

• - Input: Any format supported by ffmpeg (wav, mp3, flac, m4a, etc.)
• Processing: Internally converted to 16kHz mono WAV for diarization
• Output: WAV format (44.1kHz stereo preserved from source)

HuggingFace Token

• - Required for: pyannote speaker diarization
• Access: Must accept gated repo pyannote/speaker-diarization-3.1 on HF
• Storage: Any secure secrets manager
• Usage: Always pass via HF_TOKEN env var, never CLI arg

Sample Quality Tips

• - Shorter is better: 5-30s clean samples often score higher than 60+ second samples
• Clean audio: Remove background noise with Demucs INLINECODE22
• Single speaker: Ensure isolated voice, not mixed conversation
• Good recording: Studio mic > phone mic for voice comparison accuracy

References

• - elevenlabs-cloning.md - Best practices for ElevenLabs instant voice cloning (model settings, sample selection, proven configurations)
• scoring-guide.md - How to interpret Resemblyzer similarity scores (thresholds, use cases, troubleshooting)

Common Issues

"Missing HF token" error

• - Export token before running: INLINECODE23
• Or pass inline: INLINECODE24

Low voice comparison scores for same speaker

• - Try shorter, cleaner samples (5-30s)
• Use Demucs to isolate vocals: INLINECODE25
• Ensure consistent recording quality (same mic, environment)
• See references/scoring-guide.md troubleshooting section

Diarization not detecting all speakers

• - Adjust --min-speakers and --max-speakers flags
• Check audio quality (clear speech, minimal overlap)
• Try longer audio (30+ seconds) for better speaker modeling

MPS/Metal acceleration not working

• - Ensure PyTorch with MPS support: INLINECODE29
• Fallback to CPU: INLINECODE30
• Re-run setup_venv.sh to reinstall PyTorch