Tactical Media Broadcast Hub — initial release
Self-hosted multi-platform broadcast newsroom for activist collectives: publishing pipeline with group approval and DST-proof scheduling, TweetDeck-style column wall with hostile-content-safe rendering, live control room with Auto Director, bridged chat governance with polls and quorum, distributed sandboxed transcode workers, encrypted content-addressed archive, monthly analytics. MIT licensed. See README.md and docs/.
This commit is contained in:
commit
aa5e262f3a
880 changed files with 111042 additions and 0 deletions
142
docs/occupymedia-storage.md
Normal file
142
docs/occupymedia-storage.md
Normal file
|
|
@ -0,0 +1,142 @@
|
|||
# OccupyMedia shared files, documents and pads — phase one
|
||||
|
||||
**Implemented: contracts, config validation and independent compose scaffolding.**
|
||||
No services, Matrix upgrade, archive bot, index API, database migration or UI have
|
||||
been deployed/implemented by this phase. Existing API/media-worker storage paths
|
||||
and deployment edits are preserved. [Deployment steps and resources](../infra/occupymedia/files/README.md)
|
||||
are the next operator runbook. Design input: `runs/wp4/fileshare-research.md`.
|
||||
|
||||
## Storage ownership
|
||||
|
||||
| Payload bucket | Exclusive owner | Status |
|
||||
|---|---|---|
|
||||
| `occupymedia-matrix-media` | Future Tuwunel native S3 media | Existing bucket; live conduwuit still local |
|
||||
| `occupymedia-files-blobs` | oCIS S3NG, working files/docs and pad exports | Proposed new bucket; inventory/provision first |
|
||||
| `occupymedia-media-encrypted` | Tactical Media Broadcast Hub media/library, existing envelope format | Existing configuration; preserve KEK |
|
||||
| `occupymedia-peertube-videos` | PeerTube video objects | Existing dedicated bucket |
|
||||
| `occupymedia-peertube-playlists` | PeerTube playlists | Existing dedicated bucket |
|
||||
|
||||
Verify actual inventory; names here are ownership contracts, not proof of
|
||||
provisioning. Do not let the bot write directly into oCIS's opaque bucket. It must
|
||||
upload via authorized WebDAV/CS3 so metadata and ACLs remain coherent. Each
|
||||
product uses its own scoped credentials; bucket prefixes are not isolation.
|
||||
|
||||
The target is **R2-only durable file/media payloads, minimal local metadata/config
|
||||
allowed, no local payload cache by design**. POSIX oCIS metadata, Matrix
|
||||
RocksDB/state, Hub metadata rows, identity/config and keys remain local and need
|
||||
backups. Source files, document bodies, thumbnails, extracted full text, pad
|
||||
revisions, attachments and staging/cache copies are payloads. Calling pad text
|
||||
or full-text extraction a database column does not turn it into metadata.
|
||||
Bounded transient RAM processing is allowed; disk/swap/core-dump payload copies
|
||||
are not. This phase does not make existing local Matrix bytes disappear.
|
||||
|
||||
## Telegram-like group file contract
|
||||
|
||||
The shared TypeScript boundary is `packages/core/src/shared-files.ts`; future
|
||||
HTTP/DB adapters must validate these shapes at runtime. Use stable
|
||||
`workspaceId + groupId` (existing Hub groups) and immutable `roomId` to map to an
|
||||
oCIS `spaceId + folderId`. Start with one restricted Space per group; allow
|
||||
multiple rooms to map explicitly to that group. Never infer identity/ACL from
|
||||
room names, aliases or bridge membership. Membership changes must reconcile
|
||||
both Matrix and file grants; mismatches suspend sharing/mirroring until corrected.
|
||||
|
||||
A `GroupFile` records room/group/workspace, sender MXID (not a display name),
|
||||
Matrix event ID, origin (`matrix`, provider-described `bridge`, direct `upload`,
|
||||
or `pad-export`), original name, MIME, verified byte size, SHA-256, UTC ISO
|
||||
`createdAt` and `indexedAt`, stable library resource/version, share link,
|
||||
retention and availability. For bridged sources preserve both Matrix sender
|
||||
and provider/remote event attribution when supplied by the trusted bridge;
|
||||
do not invent attribution. Original `createdAt` comes from event time or accepted
|
||||
upload time, not the replay/index time. Direct uploads and exports use null event
|
||||
IDs until a room announcement supplies one. Hash actual bytes while streaming;
|
||||
S3 ETags, claimed Matrix MIME/size and filenames are not proof. Normalize a safe
|
||||
display filename and sniff MIME; reject over-limit transfers, including when
|
||||
Content-Length is missing or lies.
|
||||
|
||||
Default links require current group membership; downloads/edit launches must
|
||||
recheck ACLs and retention. Public links require an explicit group grant and
|
||||
expiry; revoke them on removal. Persist library links/share IDs and expiry, not
|
||||
presigned object URLs, WOPI tokens or passwords. Keep retention policy separate
|
||||
from link expiry: `keep` is the phase-one default with no automated purge;
|
||||
`until` requires an explicit timestamp, `legalHold` prevents deletion.
|
||||
Future policy must account for source Matrix media and library copies separately;
|
||||
one message redaction does not authorize deleting a blob used by another record.
|
||||
|
||||
## Metadata index and hub-archive seam
|
||||
|
||||
Future `chat_media_archive` persistence is metadata-only. Required constraints:
|
||||
unique `(workspace_id, room_id, event_id)` for attachment events, stable file
|
||||
identity/version, indexes on `(workspace_id, group_id, created_at, id)`,
|
||||
sender, MIME class, name and checksum. List/search return only available,
|
||||
unexpired rows visible through current group/file ACLs. Use a `(createdAt,id)`
|
||||
cursor scoped to workspace/group. Filename/MIME/sender/date/origin queries and
|
||||
oCIS metadata search are the initial searchable/retrievable media experience.
|
||||
Full-text/OCR/Tika is deferred: neither a local text index nor an extra JVM is
|
||||
silently included. Archive rows never contain file bodies or encrypted-file keys.
|
||||
|
||||
`HubArchivePort` defines future archive/revoke operations. A plain sync-loop bot
|
||||
(not an appservice) uses a narrowly scoped Matrix account and a scoped oCIS
|
||||
service identity. Required processing contract:
|
||||
|
||||
1. Load an explicitly enabled group mapping and current room encryption state.
|
||||
Exclude own bot/reply events to prevent loops. Abort on unknown state, lost
|
||||
membership/ACL, redaction, encrypted events or `content.file` attachments.
|
||||
2. Accept only `m.room.message` attachments (`m.file`, `m.image`, `m.audio`,
|
||||
`m.video`) with a valid `mxc://` URI. Use authenticated Matrix media download
|
||||
endpoints (with version negotiation), never arbitrary user-supplied HTTP URLs.
|
||||
Validate any redirect to the configured R2 endpoint and drop Matrix credentials
|
||||
before following; never forward bearer auth to the object store.
|
||||
3. Claim `attachmentArchiveKey(workspace,room,event)` durably. Use a deterministic
|
||||
upload destination/resource lookup and reconcile partial retries to avoid
|
||||
duplicate files across crashes. Stream bounded bytes/hash to WebDAV/CS3;
|
||||
**no temp files on local disk**. Record the committed library reference,
|
||||
verified metadata and a durable reply/outbox entry before advancing the
|
||||
stored Matrix sync cursor. Advancing past a failed event requires a durable
|
||||
retry record; R2/ACL failures must not lose attachments.
|
||||
4. Reply once with an authenticated file link using a stable Matrix transaction
|
||||
ID derived from the replay key. Search is an index query; retrieve by library
|
||||
resource through current ACLs. An encrypted Tactical Media Broadcast Hub library copy, if
|
||||
needed for publishing, goes through the existing encryption path into its
|
||||
own bucket; no cross-product object-key reuse.
|
||||
5. On redaction/retention or membership removal, revoke links and mark the index
|
||||
unavailable; stop serving immediately. A later audited deletion workflow
|
||||
handles library versions/trash, references, source media and legal holds.
|
||||
Backfill, retries and live sync obey the same policy and idempotency rules.
|
||||
|
||||
No implementation of this loop or SQL table is present in phase one; config's
|
||||
`HUB_ARCHIVE_*` variables reserve its interface. `canArchiveAttachment` is a
|
||||
tested pure eligibility check, not complete authorization or byte validation.
|
||||
|
||||
## Encryption policy
|
||||
|
||||
**Encrypted Matrix rooms cannot be server-side mirrored by this architecture.**
|
||||
No bot encryption keys, decryption engine or silent downgrade is added. Group
|
||||
admins designate a separate **unencrypted archive room**, tell members its
|
||||
attachments enter the shared library, and enable its immutable room mapping
|
||||
explicitly. Users may deliberately upload/share a file there; encrypted rooms
|
||||
remain untouched. Treat bridged rooms the same: verify encryption state instead
|
||||
of assuming the bridge made them plaintext. If an archive room becomes encrypted,
|
||||
or its state cannot be established, mirroring stops. R2 can retain encrypted
|
||||
Matrix media as ciphertext in a future Tuwunel deployment, but that does not
|
||||
make it server-searchable or give the bot access to plaintext.
|
||||
|
||||
## Collaborative ODT and pad boundaries
|
||||
|
||||
ODT editing is **oCIS file -> collaboration WOPI host -> Collabora -> save/version
|
||||
back through oCIS -> R2**. Matrix posts links and notifications; it does not edit
|
||||
ODT. The editor gets short-lived, permission-scoped launch tokens and file locks.
|
||||
Test simultaneous edits, reconnects, saves, versions and grant revocation; keep
|
||||
Collabora's working files in transient RAM and block launch if that cannot be
|
||||
guaranteed for the selected image.
|
||||
|
||||
`SharedPad` / `PadExportPort` allow Etherpad- or HedgeDoc-compatible group pad
|
||||
links and revision exports. Pad content, revisions and attachments must be
|
||||
durable R2 payloads; local storage may retain only identifiers, ACLs and session
|
||||
metadata. Standard pad deployments typically persist document text/revisions
|
||||
in their database: they need a separately validated R2 persistence adapter or
|
||||
another compliant service. R2 attachment support or periodic exports alone is
|
||||
insufficient. Consequently there is **no running pad service** in this scaffold,
|
||||
and `PADS_ENABLED=false`. Approved exports become ordinary versioned oCIS
|
||||
resources with checksums and group retention; live ODT collaboration remains
|
||||
Collabora's responsibility. This preserves the lightweight pad option without
|
||||
introducing an incompatible local payload database.
|
||||
Loading…
Add table
Add a link
Reference in a new issue