glass beadian
9365b561b2
Cloudflare's tunnel edge-pool assignment can blackhole the site for whole regions (2026-10-10: tunnel hostnames served from an unreachable anycast range for hours). This adds an independent path and the automation that keeps occupymedia.cc reachable when either path is sick: - nginx edge mirroring the tunnel ingress (app/api/tube/chat/ping), LE wildcard via dns-01, twice-daily renewal - DOCKER-USER firewall: 80/443 restricted to Cloudflare ranges (CF mode) or OPEN for grey-cloud mode via firewall.mode - edge-watcher (60s systemd timer): dynamic public-IP sync, dual-path health checks, DNS failover TUNNEL<->ORIGIN/GREY with hysteresis, gated by EDGE_ENABLED - ping.occupymedia.cc sacrificial probe host for activation - deployment runbook in README.md; credential files shipped as .example
76 lines
3.7 KiB
Markdown
76 lines
3.7 KiB
Markdown
# Edge — second ingress path + DNS failover
|
|
|
|
The primary ingress is the Cloudflare Tunnel (cloudflared, token-managed —
|
|
ingress lives in the Zero Trust dashboard). Its edge-IP assignment can go bad
|
|
for whole regions (2026-10-10: the tunnel hostnames were served from
|
|
188.114.96.0/24, blackholed on several networks for hours). This directory is
|
|
the independent second path and the automation that keeps the site reachable
|
|
when either path is sick.
|
|
|
|
## The three paths
|
|
|
|
| Mode | DNS | Path | Tradeoff |
|
|
|---|---|---|---|
|
|
| TUNNEL (default) | proxied CNAME → `<tunnel>.cfargotunnel.com` | CF edge → tunnel → apps | no inbound ports; edge-pool assignment out of our control |
|
|
| ORIGIN | proxied A → current public IP | CF edge → :443 → nginx edge → apps | needs router forward 443; edge pool may still land badly (pool is per-hostname, opaque) |
|
|
| GREY | **unproxied** A → current public IP | visitors → :443 → nginx edge → apps | deterministic reachability; origin IP public, no CF WAF |
|
|
|
|
nginx edge mirrors the tunnel ingress by host:
|
|
|
|
- `app.occupymedia.cc` → `web:3001` (occ-app compose network)
|
|
- `api.occupymedia.cc` → `api:3000`
|
|
- `tube.occupymedia.cc` → `100.92.231.98:9000` (occ-tube via Tailscale)
|
|
- `chat.occupymedia.cc` → `100.117.210.52:6167` (occ-chat conduwuit)
|
|
- `ping.occupymedia.cc` → 200 "edge-ok" (activation probe target)
|
|
- unknown SNI → `ssl_reject_handshake`
|
|
|
|
TLS: Let's Encrypt wildcard `*.occupymedia.cc` via dns-01 (no port 80 needed).
|
|
Renewal: `occ-edge-certbot.timer`, twice daily, reloads nginx.
|
|
|
|
## Files
|
|
|
|
| File | Role |
|
|
|---|---|
|
|
| `nginx.conf` | edge reverse proxy (deployed to CT 150) |
|
|
| `docker-compose.edge.yml` | occ-edge-1 on the occ-app_default network |
|
|
| `edge-firewall.sh` | DOCKER-USER rules: 80/443 from CF ranges only, or OPEN (grey) via `firewall.mode` |
|
|
| `edge-watcher.sh` | 60s systemd timer: public-IP sync (the IP is dynamic), dual-path health checks, DNS failover with hysteresis |
|
|
| `watcher.env.example` | CF token + `EDGE_ENABLED` + `TARGET_MODE` (secrets stay on the CT) |
|
|
| `cloudflare.ini.example` | certbot dns-01 credentials (secrets stay on the CT) |
|
|
| `occ-edge-*.service/.timer` | systemd units for firewall, watcher, certbot |
|
|
|
|
## Install (CT 150)
|
|
|
|
```bash
|
|
cd /opt/anarchoposter/infra/occupymedia/edge
|
|
cp cloudflare.ini.example cloudflare.ini # + real token, chmod 600
|
|
cp watcher.env.example watcher.env # + real token
|
|
docker compose -f docker-compose.edge.yml up -d
|
|
sudo cp occ-edge-*.service occ-edge-*.timer /etc/systemd/system/
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl enable --now occ-edge-firewall.service occ-edge-watcher.timer occ-edge-certbot.timer
|
|
```
|
|
|
|
Router: forward TCP 443 (80 optional) → the CT's LAN IP.
|
|
|
|
## Activation
|
|
|
|
The workstation watcher (`edge-activate.sh`, not shipped here) probes
|
|
`ping.occupymedia.cc` through a reachable edge IP. Two consecutive 200s after
|
|
the port-forward goes live → it sets `EDGE_ENABLED=true`, flips the four
|
|
hostnames, verifies the public path for 6 minutes, and **reverts to TUNNEL
|
|
automatically** if the new pool assignment is broken too. The CT watcher then
|
|
owns ongoing failover (2 tunnel failures → ORIGIN/GREY; 3 origin failures →
|
|
TUNNEL).
|
|
|
|
Target mode: `TARGET_MODE=ORIGIN` (default) or `TARGET_MODE=GREY` in
|
|
`watcher.env` / the activation environment.
|
|
|
|
## Troubleshooting
|
|
|
|
- `edge.state` — current mode + failure counters. `edge-watcher.log` — every
|
|
decision. Both in this directory on the CT.
|
|
- `EDGE_ENABLED=false` gates ALL DNS writes — observation only.
|
|
- `docker logs occ-edge-1` — nginx. `docker logs occ-app-cloudflared-1` — tunnel.
|
|
- Rollback to tunnel only: `./edge-firewall.sh` stays; set mode TUNNEL via the
|
|
watcher's `set_mode TUNNEL <ip>` logic or restore the four CNAMEs by hand.
|