devops

devops

Apps and scripts built as part of the DevOps pipeline for the monoexp.com server.

The main piece is the deploy pipeline: when a pull request is merged into main in Gogs, the new tip of main is built, deployed, tagged deploy/<target>/<UTC time> and logged.

It deploys origin/main rather than the webhook's merge_commit_sha, because Gogs fills that field with the head branch's commit, not the merge commit. Deploying the branch commit would drop any other PRs merged since the branch was made. Releases run one at a time and each fetches first, so the tip of main is always what's merged.

 PR merged in Gogs
   │  webhook (HMAC-signed), 127.0.0.1:9000/deploy
   ▼
 deployd  ── checks signature, merged into main ──▶  release <repo> origin/main
                                                     │
              clean checkout of that commit ◀────────┤
              go build  /  rsync static files ◀──────┤
              rsync content/ (if configured) ◀───────┤
              sudo systemctl restart goapp@<name> ◀──┤
              tag deploy/<target>/<time>, push ◀─────┤
              line in /var/log/deploys.log ◀─────────┘

Layout

Path	Installed as	What it is
deployd/	/usr/local/bin/deployd	Webhook receiver (Go, standard library only)
release/release	/usr/local/bin/release	Builds, deploys, tags and logs one commit
release/test.sh	not installed	Runs release end to end in a sandbox
install.sh	not installed	Runs install steps 1–3 below; safe to re-run
sites.json	/etc/deployd/sites.json	Which repo deploys to which site
systemd/deployd.service	/etc/systemd/system/	Runs deployd as the deploy user
sudoers/60-deploy	/etc/sudoers.d/60-deploy	Lets deploy restart goapp@<name> only
certs/issue-cert	/usr/local/sbin/issue-cert	Certificate wrapper (Phase 10 of the server plan)

Everything is installed by the admin, owned by root. The claude user can propose changes here through pull requests but can't install anything.

sites.json

{
  "bajdev/posts": { "kind": "go", "name": "posts", "content": "content" }
}

Key	Kinds	Meaning
kind	all	go (a goapp@<name> app) or static (files served by nginx)
name	go	App name: binary at /opt/apps/<name>/<name>, unit goapp@<name>
content	go	Optional folder in the repo synced to /opt/apps/<name>/content/
host	static	Site under /srv/www/vhosts/<host>/
dir	static	Folder in the repo to publish (default public)

Repos not listed are refused.

Install (once, as the admin)

Steps 1–4 match Phase 7 of the server plan. Steps 1–3 are one script:

git clone https://git.monoexp.com/bajdev/devops.git ~/devops && cd ~/devops
./install.sh --dry-run    # optional: print every command, change nothing
./install.sh

It's safe to re-run, for example to upgrade after merging changes here. It keeps the deploy user's SSH key, the webhook secret and the deploy log, reinstalls the binaries and config, and restarts deployd. At the end it prints the deploy user's public key for Gogs. The steps it runs are below, for reference or to do by hand; then do step 4.

1. The deploy user, its log, and its sudo rule:

sudo useradd -r -m -d /var/lib/deploy -s /bin/bash -G webops deploy
sudo install -m 664 -o deploy -g webops /dev/null /var/log/deploys.log
sudo -iu deploy sh -c 'mkdir -p -m 700 ~/.ssh && ssh-keygen -t ed25519 -N "" -f ~/.ssh/id_ed25519 -C deploy@vps'
sudo -iu deploy git config --global user.name Deploy
sudo -iu deploy git config --global user.email deploy@monoexp.com
sudo cat /var/lib/deploy/.ssh/id_ed25519.pub        # for the Gogs deploy account

In Gogs, create a deploy user, add that key to it, and add it as a Write collaborator on each repo it releases. It needs Write to push tags; branch protection still stops it pushing to main. Then trust Gogs' host key:

sudo -iu deploy ssh -o StrictHostKeyChecking=accept-new git@git.monoexp.com true

2. Build and install from a checkout of this repo:

git clone https://git.monoexp.com/bajdev/devops.git ~/devops && cd ~/devops
(cd deployd && go test ./... && CGO_ENABLED=0 go build -trimpath -o deployd .)
sudo install -o root -g root -m 755 deployd/deployd /usr/local/bin/deployd
sudo install -o root -g root -m 755 release/release /usr/local/bin/release
sudo install -d -o root -g root -m 755 /etc/deployd
sudo install -o root -g root -m 644 sites.json /etc/deployd/sites.json
sudo install -o root -g root -m 644 systemd/deployd.service /etc/systemd/system/deployd.service
# sudo rule: install under a name sudo ignores, check it, then move it into place
sudo install -o root -g root -m 440 sudoers/60-deploy /etc/sudoers.d/60-deploy.new
sudo visudo -cf /etc/sudoers.d/60-deploy.new && sudo mv /etc/sudoers.d/60-deploy.new /etc/sudoers.d/60-deploy

3. Secret and service:

printf 'WEBHOOK_SECRET=%s\n' "$(openssl rand -hex 32)" | sudo tee /etc/deployd/secret.env >/dev/null
sudo chmod 600 /etc/deployd/secret.env
sudo systemctl daemon-reload && sudo systemctl enable --now deployd
sudo cat /etc/deployd/secret.env        # the secret for the Gogs webhook

4. Let Gogs call it. Since 0.14.3 Gogs refuses webhooks to local addresses unless allowed. In /home/git/gogs/custom/conf/app.ini, then sudo systemctl restart gogs:

[security]
LOCAL_NETWORK_ALLOWLIST = 127.0.0.1

Moving a site onto the pipeline

1. Add it to sites.json (through a PR here) and reinstall that file.

2. Hand its deployed files to deploy, so release can update them. For a Go app:

   sudo chown -R deploy /opt/apps/<name>/<name> /opt/apps/<name>/content
   

Leave data/ alone. After this, deploy the site only through the pipeline; deploy-go is for a site's first deploy.

1. In the repo's Gogs settings: Webhooks → Add Webhook → Gogs. Payload URL http://127.0.0.1:9000/deploy, content type application/json, secret from /etc/deployd/secret.env, events "Let me choose what I need" → only Pull Request.
2. Merge a test PR and check the result (below).

Checking it works

• A merged PR produces a deploy/<target>/... tag in Gogs and a line in /var/log/deploys.log.
• A PR closed without merging deploys nothing.
• journalctl -u deployd shows each release's output.
• In Gogs, the webhook's Recent Deliveries shows each request and deployd's reply: 202 started a release, 204 was ignored, 401 was a bad signature.

Rollback

Redeploy any earlier release by its tag, as the admin:

sudo -iu deploy env DEPLOY_BY="rollback by bajdev" release bajdev/posts deploy/posts/20261026T120000Z

That checks out the tag, redeploys it, and records the rollback as a new tag and log line.

Development

(cd deployd && go vet ./... && go test -race ./...)
release/test.sh ~/src/posts      # sandboxed end-to-end run of release

release/test.sh uses a local bare git remote and stand-in sudo and systemctl, so nothing on the server is touched. It covers a Go release with content sync, a rollback, a static site, refused input, and a service that fails to start (no tag, no log line).