---
title: "The refresh engine that keeps my Reddit archive alive"
description: "The refresh engine pulls updates on a schedule that tapers with post age, stores every change in JSON sidecars, and recovers deleted comments."
url: "https://michael.zazon.us/notes/stash-2-the-refresh-engine-that-keeps-my-reddit-archive-alive"
published: "2026-04-23T12:00:00-04:00"
kind: "build-log"
subject: "tools"
tags: ["backups"]
series: "Stash"
part: 2
---
My favorite feature of this has to be the refresh feature. The ability to keep pulling updates and adding them to the JSON sidecar allows me to build on top of this and keep the corpus "alive" with fresh data as time goes on. I have leveraged this body of content in my research, hobbies, even my work.

In [part 1](https://michael.zazon.us/notes/stash-automating-reddit-saved-content-into-my-personal-knowledge-base) I covered getting my saved posts out of Reddit and into the vault. In this post I'll cover what happens to the posts once they are in the vault and why the refresh engine has been the most valuable feature.

## JSON sidecars next to the markdown

I store Reddit posts in a sidecar JSON format (along with the original markdown) which allows me to query them on disk, append easily on changes to posts, keep records of deleted or edited posts, and all kinds of useful metadata about posts as soon as they are saved. This gives me a lot of flexibility in searching and recall activities and ensures I have the full history of the post and comments.

Trimmed example of a sidecar for one saved post (values made generic):

```json
{
  "schema_version": 1,
  "id": "abc123",
  "type": "saved_post",
  "post": {
    "title": "...",
    "posted_at": "2025-10-11T19:50:37Z",
    "body_original": "...",
    "body_current": "...",
    "body_versions": [ ... ],
    "score_history": [ ... ],
    "deleted_at": null,
    "deleted_by": null
  },
  "comments": [
    {
      "id": "...",
      "depth": 0,
      "first_observed_at": "2025-10-12T14:02:11Z",
      "last_observed_alive_at": "2026-04-20T03:15:40Z",
      "body_original": "...",
      "body_current": "...",
      "body_versions": [],
      "score_history": [ ... ],
      "deleted_at": null
    }
  ],
  "capture": {
    "saved_at": "2025-10-12T14:02:11Z",
    "last_refreshed_at": "2026-04-20T03:15:40Z",
    "refresh_count": 3,
    "refresh_tier": "archived"
  },
  "recovery": {
    "attempts": [],
    "successful_provider": null
  },
  "derived": {
    "changed_since_prior_refresh": {
      "any_change": true,
      "post_body_edited": false,
      "new_comments": 0,
      "edited_comments": 0,
      "deleted_comments": 0,
      "removed_comments": 2
    },
    "latest_score": 31,
    "latest_num_comments": 91,
    "comment_stats": { "total": 92 }
  }
}
```

The original body is never replaced (`body_original` stays put, `body_current` moves, and all edits in between land in `body_versions`). Comments get a `first_observed_at` and a `last_observed_alive_at`, so if one disappears I know when it went. The `derived` block is the quick answer to "did anything change since last time?" without diffing the whole file (in this case, two comments were dropped since the prior refresh). I spent a lot of time working this out to work the way I wanted. I’m super happy with how it came out because it lets me leverage and visualize the data in really neat ways for my needs.

## The refresh schedule

Merges are additive only, I don't overwrite and store all deltas in the data type on disk. I do more frequent refreshes when the post is newer and taper that off to weekly, monthly, quarterly, and yearly as the post ages since activity is generally lower and we've already captured a lot of the changes during the post beginning. This is for performance and API respectfulness.

The tiers as I run them today:

| Tier | Post age | Refresh every |
|---|---|---|
| fresh | first day or so | 24 hours |
| active | after that | 7 days |
| trailing | older | 30 days |
| archived | older still | 90 days |
| cold | oldest | 365 days |

The refreshes are all done through the Reddit API with [PRAW](https://praw.readthedocs.io/) and only the posts that are on their tier schedule are fetched, so the daily sweep is very small even though I have thousands of posts on disk.

I also have some logic that checks the [Wayback Machine](https://web.archive.org/), [PullPush](https://pullpush.io/), [Reveddit](https://www.reveddit.com/) and Reddit previews for any comments that were deleted (on older posts that I may save for whatever reason). These are logged in the sidecar `recovery` block as well, along with which provider was successful in getting the comment (if any).

[Diagram: Flow of a saved post: saved on Reddit, written as a JSON sidecar and markdown note, refreshed when its tier is due (tapering from daily to yearly), merged additively with deltas appended and nothing overwritten, sent to recovery (Wayback, PullPush, Reveddit, previews) if something was deleted, then the note is re-rendered and waits for its next refresh](https://michael.zazon.us/notes/stash-2-the-refresh-engine-that-keeps-my-reddit-archive-alive)

## Browsing the archive in Obsidian

Since I have them stored in my knowledge base vault, I'm able to search and browse interactively. The AI summary is available in markdown and this is useful for agents querying the dataset, but I'm able to search across the raw posts and comments using some queries on the JSON sidecars. I'm building an interface that lets me browse this on my phone similar to the Narwhal experience, but that is a little further away.