Skip to content

ADR-0008: Serialised Single-Worker Downloads

Status

Accepted

Date

2026-08-12

Context

A sync cycle can produce many new tracks, each needing a YouTube download. Two constraints shape how downloads are run:

  1. YouTube rate limits. Downloading many videos concurrently is exactly the behaviour that gets an IP throttled or banned. The download rate must be bounded and predictable. The service is passive — nobody is waiting on a batch — so download throughput is not a design goal.
  2. yt-dlp is blocking. It extracts, downloads, and post-processes synchronously and cannot run on the asyncio event loop. It must run off the loop, in a thread. A thread pool would allow parallel downloads but multiply YouTube load.

Cancellation is a third constraint. yt-dlp exposes no "stop this download now" API; the only reliable injection point is its progress hook, which yt-dlp calls periodically while a download is running. The old code tracked a global cancellation flag and a single yt-dlp instance; the rewrite makes the mechanism explicit and per-download.

An aborted download also leaves partial .part files behind, which must be cleaned up so the download directory stays free of junk that the full scan (ADR-0005) might otherwise mistake for real files.

Decision

  • Downloads are serialised: one at a time, in a single worker thread, via anyio.to_thread.run_sync. A per-downloader anyio.Lock guarantees nothing else can start a download concurrently.
  • Cancellation is cooperative and mid-download: a CancellationToken (backed by a threading.Event) is polled inside yt-dlp's progress hook. When the token fires, the hook raises yt_dlp.utils.DownloadCancelled, aborting the in-progress download; the downloader then removes partial .part files. Cancellation is checked both before a download starts and inside a running download.

The single worker thread is the first step toward an intended design where the worker persists and consumes from a real job queue that the main thread appends to.

Consequences

Positive

  • Rate-limit friendly: a bounded, predictable one-request-at-a-time rate.
  • Simple concurrency: no parallel-download bugs, no throttling knobs to tune.
  • Responsive event loop: blocking yt-dlp work never stalls async code.
  • Clean cancellation: abort works mid-download and leaves no partial files.

Negative

  • Throughput is one track at a time; a large batch takes a while (acceptable for a passive service).
  • A single stalled download blocks the rest of the batch until it finishes or is cancelled.

Neutral

  • yt-dlp runs off the event loop in a worker thread; the rest of the application is async.

Alternatives Considered

Multiple worker threads / download queue (the old approach)

Rejected: parallel downloads multiply YouTube load, complicate rate limiting and cancellation, and the old global-instance cancellation was fragile.

Concurrent downloads capped by a semaphore

Rejected for now: adds throughput at the cost of throttle tuning and a more complex failure model. Revisitable if batch speed ever matters.

Reimplement downloads with async streaming

Rejected: would mean reimplementing extraction, format selection and post-processing that yt-dlp already provides.