Corpora Blog
Engineering guides on YouTube video and audio datasets, yt-dlp at scale, proxies and bandwidth planning for multimodal AI training data.
-
Dataset Documentation: Datasheets, Cards and Provenance
Practical dataset documentation for ML training data: datasheets, data cards, Croissant, per-sample provenance, manifests and the EU AI Act summary.
-
Video Data Loader: Fast Video Decoding for Model Training
Speed up your video data loader: keyframe and GOP seek cost, TorchCodec vs PyAV vs DALI vs decord, short-GOP re-encoding, and CPU/GPU decode math.
-
URL Frontier Design: Dedup, Politeness and Recrawls at Scale
Design a URL frontier for a large crawler: RFC 3986 canonicalization, Bloom filter math, Mercator per-host queues, sitemaps, 304 recrawls and trap limits.
-
Data Decontamination: Removing Benchmarks from Training Data
Data decontamination for LLM and multimodal training sets: n-gram overlap, Bloom filters, embedding checks, canary strings, thresholds and audit logs.
-
Speaker Diarization with pyannote for Speech Datasets
Speaker diarization with pyannote.audio 4.x for speech datasets: RTTM output, community-1, DER, overlap, Whisper word alignment and single-speaker TTS clips.
-
Whisper Transcription at Scale for Speech Datasets
Whisper transcription at scale: large-v3 vs turbo, faster-whisper batching, WhisperX alignment, VAD against hallucinations, throughput math and WER checks.
-
Residential vs Datacenter Proxies vs ISP and Mobile for Scraping
Residential vs datacenter proxies, plus ISP and mobile: how each works, per-IP vs per-GB vs per-Gbps billing math, sticky sessions, and how to test a proxy.
-
Build an Image-Text Dataset: img2dataset + CLIP Score Filtering
Build an image-text dataset for CLIP/VLM training: LAION and DataComp sources, img2dataset flags, CLIP score thresholds, dedup, safety and recaptioning.
-
Web Scraping Rate Limiting: Retries, Backoff and 429 Handling
Web scraping rate limiting done right: per-host token buckets, 429/503 Retry-After handling, jittered backoff, circuit breakers and Scrapy AutoThrottle.
-
Headless Browser Scraping with Playwright: Running It at Scale
Headless browser scraping with Playwright: when to skip the browser, blocking images, reliable waits, per-context proxies, concurrency and crash recovery.