Overview

Something I have noticed when developing software, is that I have a MASSIVE sprawling set of notes on my machines. Sometimes they are checked into version control, sometimes not. They might be on my PC, my laptop, or rarely, my server. The notes might be simple thoughts, full writethroughs of proposed ideas, or analysis / reflections of past work.

For some reason, I have not given documenting my ideas the same level of care that I have my code. For Annona, I have a fairly extensive set of CI stages that help keep the bar for the code I write high. Why can't I do something similar with documentation? This post is just me implementing that, with some thoughts / opinions sprawled throughout.

Does it really matter?

Sometimes? Many teams roll without any docs, and some of them are successful. It probably depends on how many people you work with, what you're working on, the kind of people you are working with.

While I can't prove it, I am inclined to believe that teams waste a ton of time on communicating ideas redundantly, and even more time fixing / updating code that was based on incorrect assumptions. I get that "agile" development teaches you to be ready to adapt for change, but we should still strive to get it as right as we can the first time around. Docs help tremendously with that. I maintained a BookStack deployment at my previous company that contained high level docs about our IoT platform. I shoved everything in there. PLC communication, networking, platform architecture, ideas we purposely didn't implement, etc. For the rest of the team, it meant accessible information. For me, it was a high level reference for how I set things up and why.

So I argue yes. Docs are important for others, and your future self.

Requirements

Documenting ideas is not new. Wikis, Obsidian, Notion, Google Docs, Sharepoint, etc. For software specifically, we have the RFC and RFD flows, the ADR flow, and of course the trusted README. Each approach has its own set of tradeoffs, which for the sake of brevity I won't explore. Instead, I'll just list my expectations for the solution I come up with.

It needs to be simple
It needs to use the same workflows I am already used to (VCS specifically)
It needs to be easily browsable for most people
It needs to support discussions
It needs to live alongside code

Solutioning

BookStack was the first thing I considered, I had already used it, and its data structure was familiar enough to me that I could get it up and running somewhat quickly. It is also easy to host, and supports comments, so the discussion piece is taken care of. The one thing I don't like is that it doesn't live with the code itself, and also doesn't use the same workflows I am already used to. Despite that it scores pretty well.

zk is the second idea I explored a little. I already use zk for notes in school, and its linking system being integrated with my LSP is pretty nifty. I'd argue it's as simple as you can get above just plaintext notes (which might be a little too simple), integrates very well with my workflow, and lives alongside my code. However, it doesn't support discussions (sane ones at least), and isn't super accessible for most people. Browsing through files on git is not the frictionless approach I have in mind.

Clearly, the solution lies somewhere in the middle of this. Markdown driven, that somehow supports discussions and is easily browsable.

mdBook is what the Rust book and a bunch of other Rust resources use, and I think it seems like it should fit pretty well assuming we have somewhere to host it on our internal network. It's easily browsable, built from docs that sit alongside your code, simple, and uses my existing workflow. Nice. The one thing it doesn't do is support discussions, but does it need to?

What if the browsable high level docs could just reference the discussions elsewhere? What if I just used the standard RFC / RFD flow, and just linked into them from the browsable source? The discussion happens on tooling we are already familiar with, and we still have a clean browsable set of high level docs. Let's get to solutioning that.

Implementation

We clearly need a few things:

First, we need the actual location where the service independent docs are held, alongside the RFC / RFD entries. We'll create a separate repo for this, which just pulls in docs from other repos as needed. Here is the structure:

annona_docs  ❯ tree
.
├── assemble.rs
├── book.toml
├── Dockerfile
├── flake.lock
├── flake.nix
├── justfile
├── README.md
├── src
│   ├── README.md
│   ├── rfc
│   │   └── README.md
│   ├── rfd
│   │   └── README.md
│   ├── architecture
│   │   ├── data-flow.md
│   │   └── overview.md
│   ├── getting-started
│   │   ├── glossary.md
│   │   ├── overview.md
│   │   ├── repo-tour.md
│   │   └── setup.md
│   ├── intro.md
│   └── SUMMARY.md
└── _typos.toml

Some notes:

flake.nix is for installing things I need and running the full pipeline:

# flake.nix
{
  description = "annona_docs dev environment";
 
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    rust-overlay.url = "github:oxalica/rust-overlay";
    flake-utils.url = "github:numtide/flake-utils";
  };
 
  outputs = { self, nixpkgs, rust-overlay, flake-utils, ... }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs {
          inherit system;
          overlays = [ rust-overlay.overlays.default ];
        };
 
        # pinned to match .github/workflows/docs.yaml (needed for -Zscript)
        rustToolchain = pkgs.rust-bin.nightly."2026-07-01".default;
      in
      {
        devShells.default = pkgs.mkShell {
          buildInputs = [
            rustToolchain
            pkgs.mdbook
            pkgs.lychee
            pkgs.typos
            pkgs.just
          ];
        };
      });
}

The justfile is just a simple runner for testing everything locally, and cleaning of course:

clean:
    rm -r build book
 
# mimics .github/workflows/docs.yaml build-deploy job, minus the image push step
ci:
    typos
    cargo -Zscript assemble.rs
    mdbook build
    mdbook test
    lychee --offline --no-progress 'build/**/*.md'
    docker build -t annona_docs .

_typos.toml is just the config for typos, pretty much straight from the docs plus some Annona specific jargon.

book.toml is mdBook specific config, also very very simple, just used to override the source directory mdBook reads from (normally src) to be the build dir that assemble.rs generates.

The src dir contains all of our service independent documentation, things like the problem we are solving, current architecture, how to get started, etc. Additionally, this is where we store our RFC and RFD entries, more on adding to this later.

Finally, we have the assemble.rs, which handles the actual cloning and moving things around to provide the final book source:

#!/usr/bin/env -S cargo +nightly -Zscript
 
---cargo
[dependencies]
anyhow = "1.0.104"
---
 
use anyhow::{bail, Context, Result};
use std::fs;
use std::path::Path;
 
const BUILD_DIR: &str = "build";
const CLONE_PATH: &str = "/tmp/annona_backend_rust";
const REPO_URL: &str = "https://github.com/RushFlour/annona_backend_rust";
 
fn main() -> Result<()> {
    for dir in [BUILD_DIR, CLONE_PATH] {
        if Path::new(dir).exists() {
            println!("cleaning {dir}");
            fs::remove_dir_all(dir).with_context(|| format!("clean {dir}"))?;
        }
    }
 
    copy_dir("src", BUILD_DIR).context("copy source to build dir")?;
 
    let url = match std::env::var("GH_TOKEN") {
        Ok(token) => {
            format!("https://x-access-token:{token}@github.com/RushFlour/annona_backend_rust")
        }
        // still works locally, will error out in CI and that's ok
        Err(_) => REPO_URL.to_string(),
    };
 
    let status = std::process::Command::new("git")
        .args([
            "clone", "--depth", "1", "--branch", "master", &url, CLONE_PATH,
        ])
        .status()
        .context("run git clone")?;
    if !status.success() {
        bail!("git clone failed with {status}");
    }
 
    // discover services that have docs, sorted for the summary order is always the same
    //
    // NOTE: we ignore any service that has no docs, rather than provide a warning or error
    let services_root = Path::new(CLONE_PATH).join("services");
    let mut services: Vec<String> = fs::read_dir(&services_root)
        .context("read services/")?
        .filter_map(|e| e.ok())
        .filter(|e| e.path().join("docs").is_dir())
        .map(|e| e.file_name().to_string_lossy().into_owned())
        .collect();
    services.sort();
 
    // entries captures our services section for the SUMMARY.md
    let mut entries = String::new();
 
    // create the services dir
    let services_dir = Path::new(BUILD_DIR).join("services");
    fs::create_dir_all(&services_dir).context("create services dir")?;
 
    for name in &services {
        let src_docs = services_root.join(name).join("docs");
        copy_dir(
            &src_docs.as_path().to_str().context("get path")?,
            services_dir
                .clone()
                .join(name)
                .as_path()
                .to_str()
                .context("get path")?,
        )
        .with_context(|| format!("copy docs for {name}"))?;
 
        entries.push_str(&format!("- [{name}](services/{name}/README.md)\n"));
 
        // since the docs for each service can itself be a valid mdbook, we convert its summary into
        // a useful legend
        let fragment_path = src_docs.join("SUMMARY.md");
        if fragment_path.exists() {
            let fragment = fs::read_to_string(&fragment_path)
                .with_context(|| format!("read SUMMARY fragment for {name}"))?;
            entries.push_str(&splice_fragment(&fragment, name));
        }
    }
 
    // discover RFDs and RFCs, each living in its own numbered directory with a README.md
    let rfd_entries =
        collect_numbered_entries(&Path::new(BUILD_DIR).join("rfd"), "rfd").context("collect RFDs")?;
    let rfc_entries =
        collect_numbered_entries(&Path::new(BUILD_DIR).join("rfc"), "rfc").context("collect RFCs")?;
 
    // update the services/RFDs/RFCs sections of the SUMMARY.md
    let summary_path = Path::new(BUILD_DIR).join("SUMMARY.md");
    let master = fs::read_to_string(&summary_path).context("read master SUMMARY")?;
    for marker in ["<!-- services -->", "<!-- rfds -->", "<!-- rfcs -->"] {
        if !master.contains(marker) {
            bail!("master SUMMARY.md is missing the {marker} marker");
        }
    }
    let spliced = master
        .replace("<!-- services -->", entries.trim_end())
        .replace("<!-- rfds -->", rfd_entries.trim_end())
        .replace("<!-- rfcs -->", rfc_entries.trim_end());
    fs::write(&summary_path, spliced).context("write spliced SUMMARY")?;
 
    println!("assembled {} services into {BUILD_DIR}/", services.len());
    Ok(())
}
 
/// Discover numbered RFC/RFD subdirectories
fn collect_numbered_entries(dir: &Path, section: &str) -> Result<String> {
    let mut names: Vec<String> = fs::read_dir(dir)
        .with_context(|| format!("read {}", dir.display()))?
        .filter_map(|e| e.ok())
        .filter(|e| e.path().is_dir() && e.path().join("README.md").is_file())
        .map(|e| e.file_name().to_string_lossy().into_owned())
        .collect();
    names.sort();
 
    let mut entries = String::new();
    for name in &names {
        let readme_path = dir.join(name).join("README.md");
        let readme = fs::read_to_string(&readme_path)
            .with_context(|| format!("read {}", readme_path.display()))?;
        let title = readme
            .lines()
            .find_map(|l| l.strip_prefix("# "))
            .with_context(|| format!("{} is missing a top-level heading", readme_path.display()))?;
        entries.push_str(&format!("    - [{title}]({section}/{name}/README.md)\n"));
    }
    Ok(entries)
}
 
/// copy dir recursively
fn copy_dir(src: &str, dest: &str) -> Result<()> {
    let status = std::process::Command::new("cp")
        .args(["-r", src, dest])
        .status()
        .context("run cp")?;
    if !status.success() {
        bail!("cp -r {src} {dest} failed with {status}");
    }
    Ok(())
}
 
/// Indent a SUMMARY fragment one level and prefix its links with the
/// service's path, so `[API](api.md)` becomes `[API](services/<name>/api.md)`.
fn splice_fragment(fragment: &str, name: &str) -> String {
    fragment
        .lines()
        .filter(|l| !l.trim().is_empty())
        .map(|line| {
            let line = line.replace("](", &format!("](services/{name}/"));
            format!("  {line}\n")
        })
        .collect()
}

This is a script you can run just like a bash script! Hopefully at some point LSPs work with scripts, rather than needing hacky workarounds.

This pipeline is fully driven by 2 different GitHub Actions. The first is present in the docs repo itself, and just rebuilds when something changes:

# .github/workflows/docs.yaml
name: Build
on:
  # needed for status checks before merging
  pull_request:
 
  # book actually gets built
  push:
    branches: [master]
    paths:
      - "src/**"
      - "book.toml"
      - "assemble.rs"
      - ".github/workflows/docs.yaml"
 
  # anytime the docs change in the annona backend, it'll fire a repository dispatch event
  repository_dispatch:
    types: [docs-updated]
 
  # can update manually too
  workflow_dispatch:
 
concurrency:
  group: docs-${{ github.ref }}
  cancel-in-progress: true
 
permissions:
  contents: read
  packages: write
 
jobs:
  build-deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
 
      - name: Check for typos
        uses: crate-ci/typos@v1
 
      - name: Install Rust nightly (pinned, -Zscript is unstable)
        uses: dtolnay/rust-toolchain@master
        with:
          toolchain: nightly-2026-07-01
 
      - name: Install mdBook & lychee
        uses: taiki-e/install-action@v2
        with:
          tool: [email protected],lychee
 
      - uses: actions/create-github-app-token@v1
        id: app
        with:
          app-id: ${{ secrets.DOCS_APP_ID }}
          private-key: ${{ secrets.DOCS_APP_KEY }}
          repositories: annona_backend_rust
 
      - name: Assemble book source
        env:
          GH_TOKEN: ${{ steps.app.outputs.token }}
        run: cargo +nightly-2026-07-01 -Zscript assemble.rs
 
      - name: Build book
        run: mdbook build
 
      - name: Test book code blocks
        run: mdbook test
 
      - name: Check internal links
        run: lychee --offline --no-progress 'build/**/*.md'
 
      # where you would deploy to cloudflare pages, or github pages, or
      # whatever. because I am already hosting annona, I just build a small
      # docker image I can run on my server
 
      - name: Set up Docker Buildx
        if: github.event_name != 'pull_request'
        uses: docker/setup-buildx-action@v4
 
      - name: Log in to GitHub Container Registry
        if: github.event_name != 'pull_request'
        uses: docker/login-action@v4
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
 
      - name: Build and push annona_docs image
        if: github.event_name != 'pull_request'
        uses: docker/build-push-action@v7
        with:
          context: .
          push: true
          tags: |
            ghcr.io/rushflour/annona_docs:latest
            ghcr.io/rushflour/annona_docs:${{ github.sha }}
 
  # RFC/RFD PRs must be titled 'RFC NNNN: ...' or 'RFD NNNN: ...'
  pr-title:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: read
    steps:
      - uses: actions/checkout@v4
      - uses: dorny/paths-filter@v3
        id: filter
        with:
          filters: |
            rfd:
              - 'src/rfc/**'
              - 'src/rfd/**'
 
      - name: Check RFC/RFD PR title
        if: steps.filter.outputs.rfd == 'true'
        env:
          TITLE: ${{ github.event.pull_request.title }}
        run: |
          echo "$TITLE" | grep -qE '^RF[CD] [0-9]{4}: .+' || {
            echo "::error::PRs touching src/rfc/ or src/rfd/ must be titled 'RFC NNNN: <title>' or 'RFD NNNN: <title>'"
            echo "Got: '$TITLE'"
            exit 1
          }

So anytime anything we care about changes, the book gets rebuilt and the docker image updated.

The second is in any repository we are pulling docs from:

name: notify-docs-hub
 
on:
  push:
    branches: [master]
    paths:
      - "services/*/docs/**"
 
jobs:
  dispatch:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/create-github-app-token@v1
        id: app
        with:
          app-id: ${{ secrets.DOCS_APP_ID }}
          private-key: ${{ secrets.DOCS_APP_KEY }}
          repositories: annona_docs
      - uses: peter-evans/repository-dispatch@v3
        with:
          token: ${{ steps.app.outputs.token }}
          repository: RushFlour/annona_docs
          event-type: docs-updated

Whenever docs change, we tell the docs repo to rebuild. Pretty simple.

RFC / RFD Flow

Updating the service independent docs is pretty straightforward, as is the service docs in the Annona backend, but creating RFC/RFDs is a little more involved. I'll focus on RFC, but the RFD is identical.

First, we need a way to generate a unique PR number, which should monotonically increase. Since we don't have 100s of developers making RFCs, it suffices to just keep track of it by looking at the current max RFC in the docs repository. A simple just command helps find the next version according to your local tree:

# create a new RFC: just rfc create "Title here"
rfc action title: (_new-doc "rfc" action title)
 
# create a new RFD: just rfd create "Title here"
rfd action title: (_new-doc "rfd" action title)
 
_new-doc section action title:
    #!/usr/bin/env bash
    set -euo pipefail
    section="$1"
    action="$2"
    title="$3"
 
    if [ "$action" != "create" ]; then
        echo "unknown action '$action', only 'create' is supported" >&2
        exit 1
    fi
 
    dir="src/$section"
    max=$(find "$dir" -maxdepth 1 -type d -name '[0-9]*' -exec basename {} \; | cut -d- -f1 | sort -n | tail -1)
    next=$(printf '%04d' $((10#${max:-0} + 1)))
    slug=$(echo "$title" | tr '[:upper:]' '[:lower:]' | sed -E 's/[^a-z0-9]+/-/g; s/^-+|-+$//g')
    name=$(git config user.name || echo "TODO")
 
    newdir="$dir/$next-$slug"
    mkdir -p "$newdir"
    printf '# %s: %s\n\n- Name: %s\n- Upstream: TODO\n- Status: Draft\n' "$next" "$title" "$name" > "$newdir/README.md"
    echo "created $newdir/README.md"

If two people end up making an RFC with the same number, then we can just manually fix that. A complex solution to force linearizability is not worth it at this time.

So when I have an idea I want to discuss, I checkout a new branch off of origin/master, and run:

just rfc create "Normalize units"

Then I commit, push, and open a PR. Once the PR is actually made, I just go back into the created markdown file for the RFC and update the upstream URL, commit and push.

Once that's done, you have a location to discuss your ideas that follow formal conventions, and as soon as it gets merged into master, it's included in the docs. I am following the Rust model here. If you want to include unmerged RFCs in your docs, see Oxide. Here's a snapshot of the resulting docs, accessible to anyone on my tailnet:

New RFC entry in the mdBook summary

Result

Annona now has a system where documentation is easy to browse, easy to contribute to, built on the workflows we already had, and simple enough to explain in less than 5 minutes. I consider that a win. It obviously isn't perfect, and has scaling issues, but for keeping everyone on the same page, it is good enough for now.

Where Ideas Lie