ADR-0002: Internal State File for Download Tracking
Status
Proposed
Context
The current implementation uses a JSON state file (state.json) to track:
- Which tracks have been downloaded for each playlist
- The last snapshot/etag for each playlist (for change detection)
- The filesystem path where each track was downloaded
An alternative approach would be to derive all state from the filesystem:
- Scan output directory for existing files
- Compare playlist tracks against filenames
- Only download tracks not present as files
Decision
Use an internal state file rather than filesystem-only state detection.
Rationale
Track Identity
Playlist tracks are identified by provider-specific IDs (Spotify track ID, YouTube video ID), not filenames. Matching by filename is fragile because:
- yt-dlp generates filenames from search results, which may differ from track metadata
- Filenames may contain special characters that get sanitized differently over time
- A track can be re-downloaded with a different filename, creating duplicates
- We have no reliable way to map a filename back to its source track ID
The state file stores track IDs directly, providing a reliable mapping between playlist entries and downloaded files.
Change Detection
Spotify and YouTube Music playlists change over time:
- Tracks are added
- Tracks are removed
- Tracks are reordered
Without state, if a track is removed from a playlist and later re-added:
- A filesystem-only approach would re-download it (wasteful)
- We would have no record that it was previously in the playlist
The state file stores playlist snapshots/etags, allowing us to:
- Detect when a playlist has changed
- Calculate which tracks are new (present in playlist, absent from state)
- Optionally handle removed tracks (present in state, absent from playlist)
Playlist Attribution
A track may appear in multiple playlists. The state file tracks which playlist each track came from, allowing:
- Separate output directories per playlist
- Understanding of download attribution
- Handling of tracks that move between playlists
History and Auditability
The state file provides an audit trail:
- Which tracks were downloaded and when
- Playlist state at each sync
- Download history (useful for debugging)
Performance
Scanning a large music library on every sync cycle is expensive. The state file allows O(1) lookup of known tracks instead of O(n) filesystem scans.
Consequences
Positive
- Reliable track identity tracking
- Accurate change detection
- Playlist history
- Fast sync cycles
Negative
- State can drift from filesystem reality
- Deleted files appear as "already downloaded"
- State file is a single point of failure
- Complexity in state management
Alternatives Considered
Filesystem-Only Approach
Derive all state from the filesystem. Problems:
- No reliable track-to-file mapping
- Cannot handle removed/re-added tracks
- Filename matching is fragile
Provider API + Filesystem
Query provider API for track IDs, compare against scanned filenames. Problems:
- Requires stable filename generation
- Still doesn't track playlist history
- More complex than it appears
Hybrid: Rebuild State from Filesystem
On startup, scan filesystem and rebuild state. Could work, but:
- Still needs reliable filename-to-ID mapping
- Loses playlist history
- Adds complexity for marginal benefit
Re-Evaluation
The user has raised valid concerns about state drift. Possible improvements:
- Rescan Option: Add a flag to resync and rebuild state from playlist
- Filesystem Verification: Periodically verify that state files exist
- Conflict Resolution: If a track is in state but file is missing, offer to re-download
- Trust Filesystem: Option to trust filesystem over state, rebuilding state from files
The core question is: What is the source of truth?
Current answer: The state file is the source of truth for "what should be downloaded" Alternative answer: The filesystem is the source of truth for "what has been downloaded"
A hybrid approach might serve both needs.