glass beadian
161f06f1e9
One-shot watcher for the workstation: probes ping.occupymedia.cc through a reachable edge IP, flips the four hostnames after the port-forward verifies, checks the public path for 6 minutes and auto-reverts to TUNNEL if the new pool assignment is broken too. Retires itself if the tunnel heals first. README updated to match.
83 lines
4 KiB
Markdown
83 lines
4 KiB
Markdown
# Edge — second ingress path + DNS failover
|
|
|
|
The primary ingress is the Cloudflare Tunnel (cloudflared, token-managed —
|
|
ingress lives in the Zero Trust dashboard). Its edge-IP assignment can go bad
|
|
for whole regions (2026-10-10: the tunnel hostnames were served from
|
|
188.114.96.0/24, blackholed on several networks for hours). This directory is
|
|
the independent second path and the automation that keeps the site reachable
|
|
when either path is sick.
|
|
|
|
## The three paths
|
|
|
|
| Mode | DNS | Path | Tradeoff |
|
|
|---|---|---|---|
|
|
| TUNNEL (default) | proxied CNAME → `<tunnel>.cfargotunnel.com` | CF edge → tunnel → apps | no inbound ports; edge-pool assignment out of our control |
|
|
| ORIGIN | proxied A → current public IP | CF edge → :443 → nginx edge → apps | needs router forward 443; edge pool may still land badly (pool is per-hostname, opaque) |
|
|
| GREY | **unproxied** A → current public IP | visitors → :443 → nginx edge → apps | deterministic reachability; origin IP public, no CF WAF |
|
|
|
|
nginx edge mirrors the tunnel ingress by host:
|
|
|
|
- `app.occupymedia.cc` → `web:3001` (occ-app compose network)
|
|
- `api.occupymedia.cc` → `api:3000`
|
|
- `tube.occupymedia.cc` → `100.92.231.98:9000` (occ-tube via Tailscale)
|
|
- `chat.occupymedia.cc` → `100.117.210.52:6167` (occ-chat conduwuit)
|
|
- `ping.occupymedia.cc` → 200 "edge-ok" (activation probe target)
|
|
- unknown SNI → `ssl_reject_handshake`
|
|
|
|
TLS: Let's Encrypt wildcard `*.occupymedia.cc` via dns-01 (no port 80 needed).
|
|
Renewal: `occ-edge-certbot.timer`, twice daily, reloads nginx.
|
|
|
|
## Files
|
|
|
|
| File | Role |
|
|
|---|---|
|
|
| `nginx.conf` | edge reverse proxy (deployed to CT 150) |
|
|
| `docker-compose.edge.yml` | occ-edge-1 on the occ-app_default network |
|
|
| `edge-firewall.sh` | DOCKER-USER rules: 80/443 from CF ranges only, or OPEN (grey) via `firewall.mode` |
|
|
| `edge-watcher.sh` | 60s systemd timer: public-IP sync (the IP is dynamic), dual-path health checks, DNS failover with hysteresis |
|
|
| `watcher.env.example` | CF token + `EDGE_ENABLED` + `TARGET_MODE` (secrets stay on the CT) |
|
|
| `cloudflare.ini.example` | certbot dns-01 credentials (secrets stay on the CT) |
|
|
| `occ-edge-*.service/.timer` | systemd units for firewall, watcher, certbot |
|
|
|
|
## Install (CT 150)
|
|
|
|
```bash
|
|
cd /opt/anarchoposter/infra/occupymedia/edge
|
|
cp cloudflare.ini.example cloudflare.ini # + real token, chmod 600
|
|
cp watcher.env.example watcher.env # + real token
|
|
docker compose -f docker-compose.edge.yml up -d
|
|
sudo cp occ-edge-*.service occ-edge-*.timer /etc/systemd/system/
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl enable --now occ-edge-firewall.service occ-edge-watcher.timer occ-edge-certbot.timer
|
|
```
|
|
|
|
Router: forward TCP 443 (80 optional) → the CT's LAN IP.
|
|
|
|
## Activation
|
|
|
|
The workstation watcher (`edge-activate.sh`) probes `ping.occupymedia.cc`
|
|
through a reachable edge IP. Two consecutive 200s after the port-forward goes
|
|
live → it sets `EDGE_ENABLED=true`, flips the four hostnames, verifies the
|
|
public path for 6 minutes, and **reverts to TUNNEL automatically** if the new
|
|
pool assignment is broken too. The CT watcher then owns ongoing failover (2
|
|
tunnel failures → ORIGIN/GREY; 3 origin failures → TUNNEL).
|
|
|
|
Run it from a machine that (a) has the repo `.env` with the Cloudflare token,
|
|
(b) can SSH to the CT, and (c) is OUTSIDE the affected region if you need to
|
|
distinguish "edge pool broken" from "origin unreachable":
|
|
|
|
```bash
|
|
TARGET_MODE=ORIGIN ./edge-activate.sh # or GREY for direct DNS
|
|
```
|
|
|
|
Target mode: `TARGET_MODE=ORIGIN` (default) or `TARGET_MODE=GREY` in
|
|
`watcher.env` / the activation environment.
|
|
|
|
## Troubleshooting
|
|
|
|
- `edge.state` — current mode + failure counters. `edge-watcher.log` — every
|
|
decision. Both in this directory on the CT.
|
|
- `EDGE_ENABLED=false` gates ALL DNS writes — observation only.
|
|
- `docker logs occ-edge-1` — nginx. `docker logs occ-app-cloudflared-1` — tunnel.
|
|
- Rollback to tunnel only: `./edge-firewall.sh` stays; set mode TUNNEL via the
|
|
watcher's `set_mode TUNNEL <ip>` logic or restore the four CNAMEs by hand.
|