Add edge stack: second ingress path + dual-path DNS failover

Cloudflare's tunnel edge-pool assignment can blackhole the site for whole regions (2026-10-10: tunnel hostnames served from an unreachable anycast range for hours). This adds an independent path and the automation that keeps occupymedia.cc reachable when either path is sick: - nginx edge mirroring the tunnel ingress (app/api/tube/chat/ping), LE wildcard via dns-01, twice-daily renewal - DOCKER-USER firewall: 80/443 restricted to Cloudflare ranges (CF mode) or OPEN for grey-cloud mode via firewall.mode - edge-watcher (60s systemd timer): dynamic public-IP sync, dual-path health checks, DNS failover TUNNEL<->ORIGIN/GREY with hysteresis, gated by EDGE_ENABLED - ping.occupymedia.cc sacrificial probe host for activation - deployment runbook in README.md; credential files shipped as .example
2026-10-10 21:22:15 +00:00 · 2026-10-10 21:22:15 +00:00 · 9365b561b2
commit 9365b561b2
parent aa5e262f3a
12 changed files with 438 additions and 0 deletions
--- a/infra/occupymedia/edge/README.md
+++ b/infra/occupymedia/edge/README.md
@ -0,0 +1,76 @@
+# Edge — second ingress path + DNS failover
+
+The primary ingress is the Cloudflare Tunnel (cloudflared, token-managed —
+ingress lives in the Zero Trust dashboard). Its edge-IP assignment can go bad
+for whole regions (2026-10-10: the tunnel hostnames were served from
+188.114.96.0/24, blackholed on several networks for hours). This directory is
+the independent second path and the automation that keeps the site reachable
+when either path is sick.
+
+## The three paths
+
+| Mode | DNS | Path | Tradeoff |
+|---|---|---|---|
+| TUNNEL (default) | proxied CNAME → `<tunnel>.cfargotunnel.com` | CF edge → tunnel → apps | no inbound ports; edge-pool assignment out of our control |
+| ORIGIN | proxied A → current public IP | CF edge → :443 → nginx edge → apps | needs router forward 443; edge pool may still land badly (pool is per-hostname, opaque) |
+| GREY | **unproxied** A → current public IP | visitors → :443 → nginx edge → apps | deterministic reachability; origin IP public, no CF WAF |
+
+nginx edge mirrors the tunnel ingress by host:
+
+- `app.occupymedia.cc` → `web:3001` (occ-app compose network)
+- `api.occupymedia.cc` → `api:3000`
+- `tube.occupymedia.cc` → `100.92.231.98:9000` (occ-tube via Tailscale)
+- `chat.occupymedia.cc` → `100.117.210.52:6167` (occ-chat conduwuit)
+- `ping.occupymedia.cc` → 200 "edge-ok" (activation probe target)
+- unknown SNI → `ssl_reject_handshake`
+
+TLS: Let's Encrypt wildcard `*.occupymedia.cc` via dns-01 (no port 80 needed).
+Renewal: `occ-edge-certbot.timer`, twice daily, reloads nginx.
+
+## Files
+
+| File | Role |
+|---|---|
+| `nginx.conf` | edge reverse proxy (deployed to CT 150) |
+| `docker-compose.edge.yml` | occ-edge-1 on the occ-app_default network |
+| `edge-firewall.sh` | DOCKER-USER rules: 80/443 from CF ranges only, or OPEN (grey) via `firewall.mode` |
+| `edge-watcher.sh` | 60s systemd timer: public-IP sync (the IP is dynamic), dual-path health checks, DNS failover with hysteresis |
+| `watcher.env.example` | CF token + `EDGE_ENABLED` + `TARGET_MODE` (secrets stay on the CT) |
+| `cloudflare.ini.example` | certbot dns-01 credentials (secrets stay on the CT) |
+| `occ-edge-*.service/.timer` | systemd units for firewall, watcher, certbot |
+
+## Install (CT 150)
+
+```bash
+cd /opt/anarchoposter/infra/occupymedia/edge
+cp cloudflare.ini.example cloudflare.ini   # + real token, chmod 600
+cp watcher.env.example watcher.env         # + real token
+docker compose -f docker-compose.edge.yml up -d
+sudo cp occ-edge-*.service occ-edge-*.timer /etc/systemd/system/
+sudo systemctl daemon-reload
+sudo systemctl enable --now occ-edge-firewall.service occ-edge-watcher.timer occ-edge-certbot.timer
+```
+
+Router: forward TCP 443 (80 optional) → the CT's LAN IP.
+
+## Activation
+
+The workstation watcher (`edge-activate.sh`, not shipped here) probes
+`ping.occupymedia.cc` through a reachable edge IP. Two consecutive 200s after
+the port-forward goes live → it sets `EDGE_ENABLED=true`, flips the four
+hostnames, verifies the public path for 6 minutes, and **reverts to TUNNEL
+automatically** if the new pool assignment is broken too. The CT watcher then
+owns ongoing failover (2 tunnel failures → ORIGIN/GREY; 3 origin failures →
+TUNNEL).
+
+Target mode: `TARGET_MODE=ORIGIN` (default) or `TARGET_MODE=GREY` in
+`watcher.env` / the activation environment.
+
+## Troubleshooting
+
+- `edge.state` — current mode + failure counters. `edge-watcher.log` — every
+  decision. Both in this directory on the CT.
+- `EDGE_ENABLED=false` gates ALL DNS writes — observation only.
+- `docker logs occ-edge-1` — nginx. `docker logs occ-app-cloudflared-1` — tunnel.
+- Rollback to tunnel only: `./edge-firewall.sh` stays; set mode TUNNEL via the
+  watcher's `set_mode TUNNEL <ip>` logic or restore the four CNAMEs by hand.
--- a/infra/occupymedia/edge/cloudflare.ini.example
+++ b/infra/occupymedia/edge/cloudflare.ini.example
@ -0,0 +1 @@
+dns_cloudflare_api_token = <CLOUDFLARE_API_TOKEN>
--- a/infra/occupymedia/edge/docker-compose.edge.yml
+++ b/infra/occupymedia/edge/docker-compose.edge.yml
@ -0,0 +1,16 @@
+services:
+  edge:
+    image: nginx:1.27-alpine
+    container_name: occ-edge-1
+    restart: unless-stopped
+    networks: [occ-app_default]
+    ports:
+      - "80:80"
+      - "443:443"
+    volumes:
+      - ./nginx.conf:/etc/nginx/nginx.conf:ro
+      - ./certs:/etc/letsencrypt:ro
+
+networks:
+  occ-app_default:
+    external: true
--- a/infra/occupymedia/edge/edge-firewall.sh
+++ b/infra/occupymedia/edge/edge-firewall.sh
@ -0,0 +1,40 @@
+#!/bin/bash
+# edge-firewall.sh — restrict the edge ports to Cloudflare's ranges only.
+# Mode file ./firewall.mode: "CF" (default, CF ranges only) or "OPEN" (anyone —
+# grey mode: visitors connect directly, nginx rejects unknown SNI).
+# Idempotent; safe to re-run (rebuilds the DOCKER-USER rules from scratch).
+set -euo pipefail
+cd "$(dirname "$0")"
+
+MODE="CF"
+[[ -f firewall.mode ]] && MODE=$(cat firewall.mode)
+CF4_URL="https://www.cloudflare.com/ips-v4"
+CF6_URL="https://www.cloudflare.com/ips-v6"
+
+# Reset our rules, keep anything else in DOCKER-USER.
+iptables -F DOCKER-USER
+ip6tables -F DOCKER-USER 2>/dev/null || true
+
+if [[ $MODE == OPEN ]]; then
+  echo "edge firewall: OPEN (any source allowed on 80/443; nginx SNI gate applies)"
+  exit 0
+fi
+
+mapfile -t CF4 < <(curl -s -m 15 "$CF4_URL" | grep -v '^$')
+mapfile -t CF6 < <(curl -s -m 15 "$CF6_URL" | grep -v '^$')
+
+((${#CF4[@]} > 5)) || { echo "cf ips-v4 fetch failed" >&2; exit 1; }
+
+for net in "${CF4[@]}"; do
+  iptables -A DOCKER-USER -p tcp -m multiport --dports 80,443 -s "$net" -j RETURN
+done
+iptables -A DOCKER-USER -p tcp -m multiport --dports 80,443 -j DROP
+
+if ((${#CF6[@]} > 0)); then
+  for net in "${CF6[@]}"; do
+    ip6tables -A DOCKER-USER -p tcp -m multiport --dports 80,443 -s "$net" -j RETURN 2>/dev/null || true
+  done
+  ip6tables -A DOCKER-USER -p tcp -m multiport --dports 80,443 -j DROP 2>/dev/null || true
+fi
+
+echo "edge firewall: ${#CF4[@]} v4 + ${#CF6[@]} v6 CF ranges allowed on 80/443"
--- a/infra/occupymedia/edge/edge-watcher.sh
+++ b/infra/occupymedia/edge/edge-watcher.sh
@ -0,0 +1,120 @@
+#!/bin/bash
+# edge-watcher.sh — keeps occupymedia.cc reachable regardless of which path is sick.
+#
+# Two ingress paths behind Cloudflare's proxy:
+#   TUNNEL  proxied CNAMEs -> <tunnel>.cfargotunnel.com   (edge pool can go bad per-region)
+#   ORIGIN  proxied A records -> current public IP -> nginx edge on CT 150 (443 CF-only)
+#
+# Every run:
+#   1. sync ORIGIN A records to the current public IP (the IP is dynamic)
+#   2. health-check both paths (tunnel = what visitors see via default DNS)
+#   3. fail over with hysteresis, log every decision
+#
+# Gated by EDGE_ENABLED=true in watcher.env — no DNS writes until the port
+# forward is verified. State: edge.state. Log: edge-watcher.log (rotated by size).
+set -uo pipefail
+
+cd "$(dirname "$0")"
+# shellcheck disable=SC1091
+source ./watcher.env
+
+ZONE="4bd0ac2e13a735d2e0068ce3feea2996"
+TUNNEL_NAME="e5695d8b-94eb-4e2b-9600-ed3cbf20e541.cfargotunnel.com"
+HOSTS=(app api tube chat)
+STATE_FILE="./edge.state"
+LOG="./edge-watcher.log"
+MAXLOG=1048576
+
+log() { echo "$(date -u '+%Y-%m-%dT%H:%M:%SZ') $*" >> "$LOG"; }
+# keep the log bounded
+if [[ -f $LOG && $(stat -c%s "$LOG") -gt $MAXLOG ]]; then
+  tail -n 2000 "$LOG" > "$LOG.tmp" && mv "$LOG.tmp" "$LOG"
+fi
+
+api() { curl -s -m 15 -H "Authorization: Bearer $CF_API_TOKEN" "$@"; }
+
+record_id() { # host type -> id or empty
+  api "https://api.cloudflare.com/client/v4/zones/$ZONE/dns_records?type=$2&name=$1.occupymedia.cc" \
+    | python3 -c "import json,sys; r=json.load(sys.stdin)['result']; print(r[0]['id'] if r else '')" 2>/dev/null
+}
+
+public_ip() { curl -s -m 10 ifconfig.me || curl -s -m 10 api.ipify.org; }
+
+set_mode() { # TUNNEL|ORIGIN|GREY
+  local mode=$1 ip=$2
+  local proxied=true
+  [[ $mode == GREY ]] && proxied=false
+  for h in "${HOSTS[@]}"; do
+    if [[ $mode == ORIGIN || $mode == GREY ]]; then
+      local cid; cid=$(record_id "$h" CNAME)
+      [[ -n $cid ]] && api -X DELETE "https://api.cloudflare.com/client/v4/zones/$ZONE/dns_records/$cid" > /dev/null
+      local aid; aid=$(record_id "$h" A)
+      if [[ -n $aid ]]; then
+        api -X PATCH -H "Content-Type: application/json" \
+          -d "{\"content\":\"$ip\",\"proxied\":$proxied}" \
+          "https://api.cloudflare.com/client/v4/zones/$ZONE/dns_records/$aid" > /dev/null
+      else
+        api -X POST -H "Content-Type: application/json" \
+          -d "{\"type\":\"A\",\"name\":\"$h.occupymedia.cc\",\"content\":\"$ip\",\"proxied\":$proxied}" \
+          "https://api.cloudflare.com/client/v4/zones/$ZONE/dns_records" > /dev/null
+      fi
+    else
+      local aid; aid=$(record_id "$h" A)
+      [[ -n $aid ]] && api -X DELETE "https://api.cloudflare.com/client/v4/zones/$ZONE/dns_records/$aid" > /dev/null
+      local cid; cid=$(record_id "$h" CNAME)
+      if [[ -z $cid ]]; then
+        api -X POST -H "Content-Type: application/json" \
+          -d "{\"type\":\"CNAME\",\"name\":\"$h.occupymedia.cc\",\"content\":\"$TUNNEL_NAME\",\"proxied\":true}" \
+          "https://api.cloudflare.com/client/v4/zones/$ZONE/dns_records" > /dev/null
+      fi
+    fi
+  done
+  echo "mode=$mode failures=0 tunnel_ok=0 origin_ok=0 ip=$ip" > "$STATE_FILE"
+}
+
+# ---- state ----
+if [[ -f $STATE_FILE ]]; then source "$STATE_FILE"; else mode=TUNNEL; fi
+: "${mode:=TUNNEL}"; : "${failures:=0}"; : "${tunnel_ok:=0}"; : "${origin_ok:=0}"; : "${ip:=}"
+: "${TARGET_MODE:=ORIGIN}"
+
+now_ip=$(public_ip)
+[[ $now_ip =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]] || { log "ERROR public ip lookup failed ('$now_ip')"; exit 0; }
+
+# ---- health checks ----
+tunnel_code=$(curl -s -o /dev/null -w '%{http_code}' -m 12 "https://api.occupymedia.cc/healthz" 2>/dev/null)
+[[ $tunnel_code == 200 ]] && tunnel_healthy=1 || tunnel_healthy=0
+
+origin_code=$(curl -s -o /dev/null -w '%{http_code}' -m 12 --resolve api.occupymedia.cc:443:127.0.0.1 \
+  --cacert ./certs/live/occupymedia.cc/chain.pem 2>/dev/null "https://api.occupymedia.cc/healthz")
+[[ $origin_code == 200 ]] && origin_healthy=1 || origin_healthy=0
+
+# ---- IP drift (only meaningful in ORIGIN mode; harmless otherwise) ----
+if [[ $mode == ORIGIN && -n ${EDGE_ENABLED:-} && $ip != "$now_ip" ]]; then
+  log "public ip changed $ip -> $now_ip; updating A records"
+  set_mode ORIGIN "$now_ip"; mode=ORIGIN
+fi
+
+# ---- decisions (gated) ----
+if [[ ${EDGE_ENABLED:-} != true ]]; then
+  echo "mode=$mode failures=$failures tunnel_ok=$tunnel_ok origin_ok=$origin_ok ip=$now_ip" > "$STATE_FILE"
+  log "EDGE_ENABLED!=true — observation only (tunnel=$tunnel_code origin=$origin_code)"
+  exit 0
+fi
+
+if [[ $mode == TUNNEL ]]; then
+  if [[ $tunnel_healthy == 1 ]]; then failures=0; else failures=$((failures+1)); fi
+  if [[ $failures -ge 2 && $origin_healthy == 1 ]]; then
+    log "tunnel failed $failuresx, origin healthy — FAILING OVER to $TARGET_MODE ($now_ip)"
+    set_mode "$TARGET_MODE" "$now_ip"
+  else
+    echo "mode=TUNNEL failures=$failures tunnel_ok=$tunnel_ok origin_ok=$origin_ok ip=$now_ip" > "$STATE_FILE"
+  fi
+else
+  if [[ $origin_healthy == 1 ]]; then failures=0; else failures=$((failures+1)); fi
+  if [[ $failures -ge 3 ]]; then
+    log "origin path failing ($failuresx) — FAILING BACK to TUNNEL"
+    set_mode TUNNEL "$now_ip"
+  else
+    echo "mode=$mode failures=$failures tunnel_ok=$tunnel_ok origin_ok=$origin_ok ip=$now_ip" > "$STATE_FILE"
+  fi
+fi
--- a/infra/occupymedia/edge/nginx.conf
+++ b/infra/occupymedia/edge/nginx.conf
@ -0,0 +1,134 @@
+# occupymedia.cc edge — Cloudflare origin-pull terminator.
+# Mirrors the tunnel ingress: host-routed reverse proxy in front of the apps.
+# TLS: Let's Encrypt wildcard (dns-01). HTTP: redirect only.
+
+worker_processes auto;
+events { worker_connections 1024; }
+
+http {
+  include /etc/nginx/mime.types;
+  default_type application/octet-stream;
+  sendfile on;
+  server_tokens off;
+
+  map $http_upgrade $connection_upgrade {
+    default upgrade;
+    ""      close;
+  }
+
+  # HTTP: redirect everything to HTTPS.
+  server {
+    listen 80;
+    listen [::]:80;
+    return 301 https://$host$request_uri;
+  }
+
+  # Public apps. Origins mirror the cloudflared ingress:
+  #   app  -> occ-app web:3001   api -> occ-app api:3000
+  #   tube -> occ-tube 9000     chat -> occ-chat 6167
+  server {
+    listen 443 ssl;
+    listen [::]:443 ssl;
+    http2 on;
+    server_name app.occupymedia.cc;
+
+    ssl_certificate     /etc/letsencrypt/live/occupymedia.cc/fullchain.pem;
+    ssl_certificate_key /etc/letsencrypt/live/occupymedia.cc/privkey.pem;
+
+    client_max_body_size 512m;
+    location / {
+      proxy_pass http://web:3001;
+      proxy_http_version 1.1;
+      proxy_set_header Host $host;
+      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+      proxy_set_header X-Forwarded-Proto https;
+      proxy_set_header Upgrade $http_upgrade;
+      proxy_set_header Connection $connection_upgrade;
+      proxy_read_timeout 3600s;
+      proxy_send_timeout 3600s;
+    }
+  }
+
+  server {
+    listen 443 ssl;
+    listen [::]:443 ssl;
+    http2 on;
+    server_name api.occupymedia.cc;
+
+    ssl_certificate     /etc/letsencrypt/live/occupymedia.cc/fullchain.pem;
+    ssl_certificate_key /etc/letsencrypt/live/occupymedia.cc/privkey.pem;
+
+    client_max_body_size 512m;
+    location / {
+      proxy_pass http://api:3000;
+      proxy_http_version 1.1;
+      proxy_set_header Host $host;
+      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+      proxy_set_header X-Forwarded-Proto https;
+      proxy_set_header Upgrade $http_upgrade;
+      proxy_set_header Connection $connection_upgrade;
+      proxy_read_timeout 3600s;
+      proxy_send_timeout 3600s;
+    }
+  }
+
+  server {
+    listen 443 ssl;
+    listen [::]:443 ssl;
+    server_name tube.occupymedia.cc;
+
+    ssl_certificate     /etc/letsencrypt/live/occupymedia.cc/fullchain.pem;
+    ssl_certificate_key /etc/letsencrypt/live/occupymedia.cc/privkey.pem;
+
+    client_max_body_size 0;
+    location / {
+      proxy_pass http://100.92.231.98:9000;
+      proxy_http_version 1.1;
+      proxy_set_header Host $host;
+      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+      proxy_set_header X-Forwarded-Proto https;
+      proxy_set_header Upgrade $http_upgrade;
+      proxy_set_header Connection $connection_upgrade;
+      proxy_read_timeout 3600s;
+      proxy_send_timeout 3600s;
+      proxy_buffering off;
+    }
+  }
+
+  server {
+    listen 443 ssl;
+    listen [::]:443 ssl;
+    server_name chat.occupymedia.cc;
+
+    ssl_certificate     /etc/letsencrypt/live/occupymedia.cc/fullchain.pem;
+    ssl_certificate_key /etc/letsencrypt/live/occupymedia.cc/privkey.pem;
+
+    location / {
+      proxy_pass http://100.117.210.52:6167;
+      proxy_http_version 1.1;
+      proxy_set_header Host $host;
+      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+      proxy_set_header X-Forwarded-Proto https;
+      proxy_set_header Upgrade $http_upgrade;
+      proxy_set_header Connection $connection_upgrade;
+      proxy_read_timeout 3600s;
+      proxy_send_timeout 3600s;
+    }
+  }
+
+  server {
+    listen 443 ssl;
+    listen [::]:443 ssl;
+    server_name ping.occupymedia.cc;
+    ssl_certificate     /etc/letsencrypt/live/occupymedia.cc/fullchain.pem;
+    ssl_certificate_key /etc/letsencrypt/live/occupymedia.cc/privkey.pem;
+    return 200 "edge-ok";
+  }
+
+  # Everything else on this IP: drop the connection.
+  server {
+    listen 443 ssl default_server;
+    listen [::]:443 ssl default_server;
+    ssl_reject_handshake on;
+  }
+}
--- a/infra/occupymedia/edge/occ-edge-certbot.service
+++ b/infra/occupymedia/edge/occ-edge-certbot.service
@ -0,0 +1,7 @@
+[Unit]
+Description=Renew occupymedia edge Let's Encrypt cert (dns-01)
+
+[Service]
+Type=oneshot
+ExecStart=docker run --rm -v /opt/anarchoposter/infra/occupymedia/edge/certs:/etc/letsencrypt -v /opt/anarchoposter/infra/occupymedia/edge/cloudflare.ini:/etc/cloudflare.ini:ro certbot/dns-cloudflare:latest renew --dns-cloudflare --dns-cloudflare-credentials=/etc/cloudflare.ini
+ExecStartPost=docker exec occ-edge-1 nginx -s reload
--- a/infra/occupymedia/edge/occ-edge-certbot.timer
+++ b/infra/occupymedia/edge/occ-edge-certbot.timer
@ -0,0 +1,9 @@
+[Unit]
+Description=Twice-daily cert renewal check
+
+[Timer]
+OnCalendar=*-*-* 03,15:14:00
+RandomizedDelaySec=20m
+
+[Install]
+WantedBy=timers.target
--- a/infra/occupymedia/edge/occ-edge-firewall.service
+++ b/infra/occupymedia/edge/occ-edge-firewall.service
@ -0,0 +1,12 @@
+[Unit]
+Description=occupymedia edge firewall (Cloudflare-only 80/443)
+After=docker.service network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+ExecStart=/opt/anarchoposter/infra/occupymedia/edge/edge-firewall.sh
+RemainAfterExit=yes
+
+[Install]
+WantedBy=multi-user.target
--- a/infra/occupymedia/edge/occ-edge-watcher.service
+++ b/infra/occupymedia/edge/occ-edge-watcher.service
@ -0,0 +1,9 @@
+[Unit]
+Description=occupymedia edge watcher — dual-path DNS failover
+After=network-online.target docker.service
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+WorkingDirectory=/opt/anarchoposter/infra/occupymedia/edge
+ExecStart=/opt/anarchoposter/infra/occupymedia/edge/edge-watcher.sh
--- a/infra/occupymedia/edge/occ-edge-watcher.timer
+++ b/infra/occupymedia/edge/occ-edge-watcher.timer
@ -0,0 +1,9 @@
+[Unit]
+Description=Run occupymedia edge watcher every minute
+
+[Timer]
+OnBootSec=90
+OnUnitActiveSec=60
+
+[Install]
+WantedBy=timers.target
--- a/infra/occupymedia/edge/watcher.env.example
+++ b/infra/occupymedia/edge/watcher.env.example
@ -0,0 +1,5 @@
+# Cloudflare API token (Zone.DNS edit) — root-only file
+CF_API_TOKEN=<CLOUDFLARE_API_TOKEN>
+# Set to true ONLY after the router port-forward 443->192.168.13.221 is verified.
+EDGE_ENABLED=false
+TARGET_MODE=ORIGIN
				`@ -0,0 +1 @@`
				`dns_cloudflare_api_token = <CLOUDFLARE_API_TOKEN>`