ADR-0003: Hybrid Filesystem-Based State Management

Status

Accepted

Supersedes

ADR-0002: Internal State File for Download Tracking

Context

The original approach used a JSON state file (state.json) to track:

Which tracks have been downloaded for each playlist
The last snapshot/etag for each playlist
The filesystem path where each track was downloaded

This approach had several problems:

State drift: If files are deleted, the state file still thinks they exist
Migration issues: When the program was initially deployed, it only fetched the first 100 tracks per playlist. The state file recorded these as "downloaded". When pagination was later added, the etag hadn't changed, so the program assumed all tracks were already downloaded.
Fragile recovery: If a previous run failed partway through, incomplete downloads were not retried

An alternative is to use the filesystem as the source of truth for track-level state, keeping only playlist-level state in the JSON file.

Decision

Use a hybrid approach:

Playlist-level state in state.json: etag, output path, sync metadata
Track-level state in file metadata: track ID embedded in downloaded files

Implementation

state.json Structure

{
  "spotify:playlist:xxx": {
    "etag": "abc123",
    "output_path": "/srv/music/my-playlist",
    "last_synced": "2024-01-01T00:00:00Z"
  },
  "ytm:playlist:yyy": {
    "etag": "def456",
    "output_path": "/srv/music/ytm-playlist",
    "last_synced": "2024-01-01T00:00:00Z"
  },
  "needs_full_scan": true
}

File Metadata

Track identity is stored in the ID3 UFID (Unique File Identifier) frame:

UFID:passive-music-dl → spotify:track:4uLU6hMCjMI75M1A2tKUQC
                        or
                        ytm:video:dQw4w9WgXcQ

Existing metadata (artist, title, album art) is already embedded and remains in place.

Sync Algorithm

On process start:
  needs_full_scan = state.json.needs_full_scan OR true

For each playlist in config:
  if needs_full_scan:
    # First sync after boot - always scan
    do_full_sync(playlist)
  else:
    # Check etag first
    if playlist.etag != state[playlist.id].etag:
      do_full_sync(playlist)
    else:
      skip (no changes)

  After successful sync:
    state[playlist.id].etag = playlist.etag
    state[playlist.id].last_synced = now
    state.needs_full_scan = false

state.json.save()

Full Sync Algorithm

do_full_sync(playlist):
  1. Fetch full playlist tracks from API (with pagination)
  2. Scan playlist.output_path directory
  3. For each file in directory:
       read UFID metadata
       add to scanned_ids set
  4. For each track in playlist:
       if track.id not in scanned_ids:
         download(track, embed UFID:track.id in metadata)

Rationale

Why Keep Playlist-Level State?

Etag for change detection: Avoids unnecessary filesystem scans when playlist hasn't changed
Output path mapping: Associates playlist IDs with their configured output directories
Sync metadata: Useful for debugging and rate limiting

Why Move Track-Level State to Files?

No state drift: Filesystem is the source of truth for what exists
Automatic failure recovery: If a previous run failed, next boot does full scan
Migration-safe: Pagination fix doesn't cause missed tracks
Simpler model: "If file exists, it's downloaded" is intuitive
Resilient to file operations: Moving/renaming files doesn't break tracking (metadata stays)

Why `needs_full_scan` Flag?

Failure recovery: If the process crashes during a sync, some tracks may not have been downloaded. Without this flag, subsequent runs would skip the playlist (etag unchanged).
Single flag for all playlists: Simpler than tracking per-playlist. If any sync fails, all playlists get rescanned on next boot.

Why UFID Tag?

The UFID (Unique File Identifier) frame in ID3 is designed for exactly this purpose - storing unique identifiers from external databases. Using a namespace like passive-music-dl avoids conflicts with other tools that might use UFID.

Consequences

Positive

No state drift between state.json and filesystem
Automatic recovery from failed downloads
Migration-safe across program updates
Simple, intuitive mental model
Playlist etag still optimizes for unchanged playlists

Negative

Requires reading metadata from files (slightly slower than JSON lookup)
Existing files without UFID will be re-downloaded (acceptable for migration)
UFID tag is non-standard (though UFID frame itself is standard)

Migration

Existing users with state.json files:

First run after this change will do full scans (flag is set on boot)
Tracks with existing metadata but no UFID will be re-downloaded
After migration, behavior is as described above

Alternatives Considered

Filesystem-Only (No state.json)

Rejected because:

Loses etag-based change detection
Loses playlist path mapping in state
More filesystem I/O on every sync

Per-Playlist Full Scan Flag

Instead of single needs_full_scan, track per-playlist:

{
  "spotify:playlist:xxx": { "needs_full_scan": true, ... }
}