# devops

Apps and scripts built as part of the DevOps pipeline for the monoexp.com
server.

The main piece is the **deploy pipeline**: when a pull request is merged into
`main` in Gogs, the new tip of `main` is built, deployed, tagged
`deploy/<target>/<UTC time>` and logged.

It deploys `origin/main` rather than the webhook's `merge_commit_sha`,
because Gogs fills that field with the head branch's commit, not the merge
commit. Deploying the branch commit would drop any other PRs merged since the
branch was made. Releases run one at a time and each fetches first, so the
tip of `main` is always what's merged.

```
 PR merged in Gogs
   │  webhook (HMAC-signed), 127.0.0.1:9000/deploy
   ▼
 deployd  ── checks signature, merged into main ──▶  release <repo> origin/main
                                                     │
              clean checkout of that commit ◀────────┤
              go build  /  rsync static files ◀──────┤
              rsync content/ (if configured) ◀───────┤
              sudo systemctl restart goapp@<name> ◀──┤
              tag deploy/<target>/<time>, push ◀─────┤
              line in /var/log/deploys.log ◀─────────┘
```

## Layout

| Path | Installed as | What it is |
|---|---|---|
| `deployd/` | `/usr/local/bin/deployd` | Webhook receiver (Go, standard library only) |
| `release/release` | `/usr/local/bin/release` | Builds, deploys, tags and logs one commit |
| `release/test.sh` | not installed | Runs `release` end to end in a sandbox |
| `install.sh` | not installed | Runs install steps 1–3 below; safe to re-run |
| `sites.json` | `/etc/deployd/sites.json` | Which repo deploys to which site |
| `systemd/deployd.service` | `/etc/systemd/system/` | Runs `deployd` as the `deploy` user |
| `sudoers/60-deploy` | `/etc/sudoers.d/60-deploy` | Lets `deploy` restart `goapp@<name>` only |
| `certs/issue-cert` | `/usr/local/sbin/issue-cert` | Certificate wrapper (Phase 10 of the server plan) |

Everything is installed by the admin, owned by root. The `claude` user can
propose changes here through pull requests but can't install anything.

## sites.json

```json
{
  "bajdev/posts": { "kind": "go", "name": "posts", "content": "content" }
}
```

| Key | Kinds | Meaning |
|---|---|---|
| `kind` | all | `go` (a `goapp@<name>` app) or `static` (files served by nginx) |
| `name` | go | App name: binary at `/opt/apps/<name>/<name>`, unit `goapp@<name>` |
| `content` | go | Optional folder in the repo synced to `/opt/apps/<name>/content/` |
| `host` | static | Site under `/srv/www/vhosts/<host>/` |
| `dir` | static | Folder in the repo to publish (default `public`) |

Repos not listed are refused.

## Install (once, as the admin)

Steps 1–4 match Phase 7 of the server plan. **Steps 1–3 are one script:**

```bash
git clone https://git.monoexp.com/bajdev/devops.git ~/devops && cd ~/devops
./install.sh --dry-run    # optional: print every command, change nothing
./install.sh
```

It's safe to re-run, for example to upgrade after merging changes here. It
keeps the deploy user's SSH key, the webhook secret and the deploy log,
reinstalls the binaries and config, and restarts `deployd`. At the end it
prints the deploy user's public key for Gogs. The steps it runs are below,
for reference or to do by hand; then do step 4.

**1. The `deploy` user**, its log, and its sudo rule:

```bash
sudo useradd -r -m -d /var/lib/deploy -s /bin/bash -G webops deploy
sudo install -m 664 -o deploy -g webops /dev/null /var/log/deploys.log
sudo -iu deploy sh -c 'mkdir -p -m 700 ~/.ssh && ssh-keygen -t ed25519 -N "" -f ~/.ssh/id_ed25519 -C deploy@vps'
sudo -iu deploy git config --global user.name Deploy
sudo -iu deploy git config --global user.email deploy@monoexp.com
sudo cat /var/lib/deploy/.ssh/id_ed25519.pub        # for the Gogs deploy account
```

In Gogs, create a `deploy` user, add that key to it, and add it as a **Write**
collaborator on each repo it releases. It needs Write to push tags; branch
protection still stops it pushing to `main`. Then trust Gogs' host key:

```bash
sudo -iu deploy ssh -o StrictHostKeyChecking=accept-new git@git.monoexp.com true
```

**2. Build and install** from a checkout of this repo:

```bash
git clone https://git.monoexp.com/bajdev/devops.git ~/devops && cd ~/devops
(cd deployd && go test ./... && CGO_ENABLED=0 go build -trimpath -o deployd .)
sudo install -o root -g root -m 755 deployd/deployd /usr/local/bin/deployd
sudo install -o root -g root -m 755 release/release /usr/local/bin/release
sudo install -d -o root -g root -m 755 /etc/deployd
sudo install -o root -g root -m 644 sites.json /etc/deployd/sites.json
sudo install -o root -g root -m 644 systemd/deployd.service /etc/systemd/system/deployd.service
# sudo rule: install under a name sudo ignores, check it, then move it into place
sudo install -o root -g root -m 440 sudoers/60-deploy /etc/sudoers.d/60-deploy.new
sudo visudo -cf /etc/sudoers.d/60-deploy.new && sudo mv /etc/sudoers.d/60-deploy.new /etc/sudoers.d/60-deploy
```

**3. Secret and service:**

```bash
printf 'WEBHOOK_SECRET=%s\n' "$(openssl rand -hex 32)" | sudo tee /etc/deployd/secret.env >/dev/null
sudo chmod 600 /etc/deployd/secret.env
sudo systemctl daemon-reload && sudo systemctl enable --now deployd
sudo cat /etc/deployd/secret.env        # the secret for the Gogs webhook
```

**4. Let Gogs call it.** Since 0.14.3 Gogs refuses webhooks to local
addresses unless allowed. In `/home/git/gogs/custom/conf/app.ini`, then
`sudo systemctl restart gogs`:

```ini
[security]
LOCAL_NETWORK_ALLOWLIST = 127.0.0.1
```

## Moving a site onto the pipeline

1. Add it to `sites.json` (through a PR here) and reinstall that file.
2. Hand its deployed files to `deploy`, so `release` can update them. For a
   Go app:

   ```bash
   sudo chown -R deploy /opt/apps/<name>/<name> /opt/apps/<name>/content
   ```

   Leave `data/` alone. After this, deploy the site only through the
   pipeline; `deploy-go` is for a site's first deploy.
3. In the repo's Gogs settings: **Webhooks → Add Webhook → Gogs**. Payload URL
   `http://127.0.0.1:9000/deploy`, content type `application/json`, secret
   from `/etc/deployd/secret.env`, events "Let me choose what I need" → only
   **Pull Request**.
4. Merge a test PR and check the result (below).

## Checking it works

- A merged PR produces a `deploy/<target>/...` tag in Gogs and a line in
  `/var/log/deploys.log`.
- A PR closed without merging deploys nothing.
- `journalctl -u deployd` shows each release's output.
- In Gogs, the webhook's **Recent Deliveries** shows each request and
  deployd's reply: `202` started a release, `204` was ignored, `401` was a
  bad signature.

## Rollback

Redeploy any earlier release by its tag, as the admin:

```bash
sudo -iu deploy env DEPLOY_BY="rollback by bajdev" release bajdev/posts deploy/posts/20261026T120000Z
```

That checks out the tag, redeploys it, and records the rollback as a new tag
and log line.

## Development

```bash
(cd deployd && go vet ./... && go test -race ./...)
release/test.sh ~/src/posts      # sandboxed end-to-end run of release
```

`release/test.sh` uses a local bare git remote and stand-in `sudo` and
`systemctl`, so nothing on the server is touched. It covers a Go release
with content sync, a rollback, a static site, refused input, and a service
that fails to start (no tag, no log line).