Running my Reddit archive as a real service
Two containers on my Linux server, a two-day production check, and 64 commits to get my Reddit archive running 24/7.
In part 1 I covered getting the saved posts out of Reddit and into the vault, and in part 2 the refresh engine that keeps it current. This part is about how it runs day to day on my Linux server, and what it took to get it to the point where I’d call it production.
Running it as a real service in my homelab has been a great success. I’m able to monitor the service and keep things running and syncing 24/7 with low overhead.
It runs as two containers built from one image on my Linux server, one handles ingest every 2 hours and the other runs the refresh sweep every 24 hours. I generally let a new service run for two days before I’d call it “production”. Thats what I did here. Almost 6000 posts refreshed in that window, score and comment-count history piling up, deletion markers landing with timestamps, new saves getting tagged roughly 20 seconds after they came in. I dropped tagging requests to the LLM endpoint to 3 a minute and after that, zero rate-limit errors. If something fails a refresh it gets picked up on the next run (17 in the first full run).
It took 64 commits to get there, the image size dropped from 10.1GB to 535MB after some optimizations. I tagged a rollback image before every risky change, and I kept a cutover/rollback runbook fresh so switching versions back and forth is documented step by step.
I may switch from saving posts to a system where I just send the post or comment URL (Apple Shortcut, hotkeys on macOS, etc) to an API and it handles the scraping, almost like a Raindrop type of thing. This would also allow me to send any URL or YouTube video to it and have the stash API figure out how and where to save it (and how and when to refresh it in perpetuity), but this is a little ways out. It has been a really fun project, that’s for sure!