shoggoth

NixOS infrastructure managed with Clan, Tailscale, OpenTofu, and Podman.

Documentation

• Getting Started - Initial setup and prerequisites
• Hetzner Provisioning - Provisioning cloud VMs
• Mini PC Provisioning - Provisioning physical machines
• Adding Services - How to add new applications
• Secrets Management - Managing secrets with sops-nix
• Tailscale Setup - Tailscale configuration details
• Backup and Restore - Restic backup and Forgejo restore runbook
• Hermes Audit Access - Restricted support/audit access over Tailscale SSH

Current topology

Internet
  │
  ├─ HTTPS :443 / HTTP :80
  │
  └─ Forgejo Git SSH :222
      │
      ▼
Hetzner bastion (`bastion`)
  - Caddy terminates HTTPS for forgejo.fairlabs.dev
  - HAProxy accepts public Forgejo Git SSH on :222 and adds PROXY protocol
  - rathole server exposes app tunnels locally/publicly as needed
  - rathole control :2333 is reachable only over Tailscale
      │
      │ rathole over Tailscale MagicDNS (`bastion:2333`)
      ▼
Hetzner app host (`app-server-1`)
  - Forgejo container bound to 127.0.0.1:3000
  - Forgejo SSH container port bound to 127.0.0.1:222
  - PostgreSQL container on private Podman network
  - Restic backs up Forgejo data plus logical PostgreSQL dump

Tailscale is the management network after Clan/NixOS install. The default Hetzner cloud-init template is intentionally SSH-only for bootstrap, because Clan install can disrupt an initial Tailscale connection.

tsnsrv remains available for future/internal per-service Tailscale exposure, but Forgejo's public path is currently bastion + rathole + Caddy/HAProxy.

Quick start

Prerequisites

• Nix with flakes enabled
• Tailscale account
• Hetzner Cloud account for cloud VMs

Setup

1. Enter the development shell:

   nix develop
   # Or with direnv: direnv allow
   

2. Configure Tailscale:

   • Create a server OAuth client at https://login.tailscale.com/admin/settings/oauth with tag:server.
   • Create a separate tsnsrv service OAuth client with tag:service.
   • Set up ACL tags for servers and services (tag:server, tag:service).

3. Configure Hetzner Cloud:

   cd infra
   cp terraform.tfvars.example terraform.tfvars
   # Edit terraform.tfvars with your credentials and server definitions.
   

Workflows

Provision Hetzner VMs

cd infra
tofu init
tofu plan
tofu apply

Wait 2-3 minutes for cloud-init to complete. The default template prepares temporary key-only SSH; Tailscale starts after Clan installs NixOS.

Install NixOS on a server

During initial bootstrap, use the server public IP. Temporarily set servers.<name>.allow_public_ssh = true if public SSH is needed, then set it back to false after Tailscale works.

# Generate hardware configuration
clan machines init-hardware-config app-server-1 --target-host root@<public-ip>

# Install NixOS (this wipes the disk)
clan machines install app-server-1 --target-host root@<public-ip>

After install, manage over Tailscale:

ssh root@app-server-1 true
clan machines update app-server-1

Deploy the current machines

clan machines update bastion
clan machines update app-server-1

Useful runtime checks:

ssh root@bastion systemctl status rathole caddy haproxy fail2ban --no-pager
ssh root@app-server-1 systemctl status rathole podman-forgejo restic-backups-forgejo.timer --no-pager
ssh root@app-server-1 podman ps

Provision a Mini PC manually

For physical machines like Intel NUCs or mini PCs:

1. Boot any Linux live USB.
2. Install Tailscale manually:

   curl -fsSL https://tailscale.com/install.sh | sh
   tailscale up --ssh --hostname=my-minipc
   

3. From your workstation, create/adapt a machine config, generate hardware config, then install:

   clan machines init-hardware-config my-minipc --target-host root@my-minipc
   clan machines install my-minipc --target-host root@my-minipc
   

Repository layout

shoggoth/
├── flake.nix              # Flake outputs, Clan/NixOS configs, formatter, dev shell
├── clan.nix               # Clan inventory and centralized admin SSH keys
├── infra/                 # OpenTofu for Hetzner Cloud
├── machines/              # Per-machine NixOS configs and committed facter.json
│   ├── bastion/
│   └── app-server-1/
├── modules/               # Shared NixOS modules
│   ├── base.nix
│   ├── rathole.nix
│   ├── restic.nix
│   ├── support-audit.nix
│   └── tsnsrv-service.nix
├── services/
│   └── forgejo/           # Forgejo + PostgreSQL + Restic integration
└── docs/

Services

Forgejo

Forgejo is the current production app service:

• module: services/forgejo/default.nix
• enabled on: machines/app-server-1/configuration.nix
• public HTTPS: https://forgejo.fairlabs.dev through bastion Caddy
• public Git SSH: ssh://git@forgejo.fairlabs.dev:222/... through bastion HAProxy/rathole
• backups: shoggoth.restic.backups.forgejo

Future/internal services

For services that should only live inside Tailscale, use the self-registration pattern in docs/04-adding-services.md with shoggoth.tsnsrv.services.<name>.

Secrets management

Secrets are managed through Clan vars and sops-nix. Prefer module-owned clan.core.vars.generators over manual secret writes.

Common commands:

clan vars generate app-server-1
clan vars list app-server-1
sops sops/secrets/<name>.yaml

Validation

Run inside the dev shell:

nix develop
tofu -chdir=infra validate
nix build .#nixosConfigurations.bastion.config.system.build.toplevel --no-link
nix build .#nixosConfigurations.app-server-1.config.system.build.toplevel --no-link

Troubleshooting

Server not appearing in Tailscale

This is expected before Clan/NixOS install with the default Hetzner template.

1. SSH via public IP during bootstrap: ssh root@<public-ip>
2. Check cloud-init: cat /var/log/cloud-init-output.log
3. After Clan install, check Tailscale: tailscale status

NixOS installation fails

1. Ensure the target has at least 1.5GB RAM.
2. Check SSH connectivity to the bootstrap target or, after install, the Tailscale hostname.
3. Run with debug: clan machines install <machine> --debug.

tsnsrv service not starting

1. Check logs: journalctl -u tsnsrv-<service>
2. Verify the service OAuth client has only tag:service.
3. Ensure ACL tags match the service config.