T-HUB/docs/occupymedia-storage.md

OccupyMedia shared files, documents and pads — phase one

Implemented: contracts, config validation and independent compose scaffolding. No services, Matrix upgrade, archive bot, index API, database migration or UI have been deployed/implemented by this phase. Existing API/media-worker storage paths and deployment edits are preserved. Deployment steps and resources are the next operator runbook. Design input: runs/wp4/fileshare-research.md.

Storage ownership

Payload bucket	Exclusive owner	Status
occupymedia-matrix-media	Future Tuwunel native S3 media	Existing bucket; live conduwuit still local
occupymedia-files-blobs	oCIS S3NG, working files/docs and pad exports	Proposed new bucket; inventory/provision first
occupymedia-media-encrypted	Tactical Media Broadcast Hub media/library, existing envelope format	Existing configuration; preserve KEK
occupymedia-peertube-videos	PeerTube video objects	Existing dedicated bucket
occupymedia-peertube-playlists	PeerTube playlists	Existing dedicated bucket

Verify actual inventory; names here are ownership contracts, not proof of provisioning. Do not let the bot write directly into oCIS's opaque bucket. It must upload via authorized WebDAV/CS3 so metadata and ACLs remain coherent. Each product uses its own scoped credentials; bucket prefixes are not isolation.

The target is R2-only durable file/media payloads, minimal local metadata/config allowed, no local payload cache by design. POSIX oCIS metadata, Matrix RocksDB/state, Hub metadata rows, identity/config and keys remain local and need backups. Source files, document bodies, thumbnails, extracted full text, pad revisions, attachments and staging/cache copies are payloads. Calling pad text or full-text extraction a database column does not turn it into metadata. Bounded transient RAM processing is allowed; disk/swap/core-dump payload copies are not. This phase does not make existing local Matrix bytes disappear.

Telegram-like group file contract

The shared TypeScript boundary is packages/core/src/shared-files.ts; future HTTP/DB adapters must validate these shapes at runtime. Use stable workspaceId + groupId (existing Hub groups) and immutable roomId to map to an oCIS spaceId + folderId. Start with one restricted Space per group; allow multiple rooms to map explicitly to that group. Never infer identity/ACL from room names, aliases or bridge membership. Membership changes must reconcile both Matrix and file grants; mismatches suspend sharing/mirroring until corrected.

A GroupFile records room/group/workspace, sender MXID (not a display name), Matrix event ID, origin (matrix, provider-described bridge, direct upload, or pad-export), original name, MIME, verified byte size, SHA-256, UTC ISO createdAt and indexedAt, stable library resource/version, share link, retention and availability. For bridged sources preserve both Matrix sender and provider/remote event attribution when supplied by the trusted bridge; do not invent attribution. Original createdAt comes from event time or accepted upload time, not the replay/index time. Direct uploads and exports use null event IDs until a room announcement supplies one. Hash actual bytes while streaming; S3 ETags, claimed Matrix MIME/size and filenames are not proof. Normalize a safe display filename and sniff MIME; reject over-limit transfers, including when Content-Length is missing or lies.

Default links require current group membership; downloads/edit launches must recheck ACLs and retention. Public links require an explicit group grant and expiry; revoke them on removal. Persist library links/share IDs and expiry, not presigned object URLs, WOPI tokens or passwords. Keep retention policy separate from link expiry: keep is the phase-one default with no automated purge; until requires an explicit timestamp, legalHold prevents deletion. Future policy must account for source Matrix media and library copies separately; one message redaction does not authorize deleting a blob used by another record.

Metadata index and hub-archive seam

Future chat_media_archive persistence is metadata-only. Required constraints: unique (workspace_id, room_id, event_id) for attachment events, stable file identity/version, indexes on (workspace_id, group_id, created_at, id), sender, MIME class, name and checksum. List/search return only available, unexpired rows visible through current group/file ACLs. Use a (createdAt,id) cursor scoped to workspace/group. Filename/MIME/sender/date/origin queries and oCIS metadata search are the initial searchable/retrievable media experience. Full-text/OCR/Tika is deferred: neither a local text index nor an extra JVM is silently included. Archive rows never contain file bodies or encrypted-file keys.

HubArchivePort defines future archive/revoke operations. A plain sync-loop bot (not an appservice) uses a narrowly scoped Matrix account and a scoped oCIS service identity. Required processing contract:

1. Load an explicitly enabled group mapping and current room encryption state. Exclude own bot/reply events to prevent loops. Abort on unknown state, lost membership/ACL, redaction, encrypted events or content.file attachments.
2. Accept only m.room.message attachments (m.file, m.image, m.audio, m.video) with a valid mxc:// URI. Use authenticated Matrix media download endpoints (with version negotiation), never arbitrary user-supplied HTTP URLs. Validate any redirect to the configured R2 endpoint and drop Matrix credentials before following; never forward bearer auth to the object store.
3. Claim attachmentArchiveKey(workspace,room,event) durably. Use a deterministic upload destination/resource lookup and reconcile partial retries to avoid duplicate files across crashes. Stream bounded bytes/hash to WebDAV/CS3; no temp files on local disk. Record the committed library reference, verified metadata and a durable reply/outbox entry before advancing the stored Matrix sync cursor. Advancing past a failed event requires a durable retry record; R2/ACL failures must not lose attachments.
4. Reply once with an authenticated file link using a stable Matrix transaction ID derived from the replay key. Search is an index query; retrieve by library resource through current ACLs. An encrypted Tactical Media Broadcast Hub library copy, if needed for publishing, goes through the existing encryption path into its own bucket; no cross-product object-key reuse.
5. On redaction/retention or membership removal, revoke links and mark the index unavailable; stop serving immediately. A later audited deletion workflow handles library versions/trash, references, source media and legal holds. Backfill, retries and live sync obey the same policy and idempotency rules.

No implementation of this loop or SQL table is present in phase one; config's HUB_ARCHIVE_* variables reserve its interface. canArchiveAttachment is a tested pure eligibility check, not complete authorization or byte validation.

Encryption policy

Encrypted Matrix rooms cannot be server-side mirrored by this architecture. No bot encryption keys, decryption engine or silent downgrade is added. Group admins designate a separate unencrypted archive room, tell members its attachments enter the shared library, and enable its immutable room mapping explicitly. Users may deliberately upload/share a file there; encrypted rooms remain untouched. Treat bridged rooms the same: verify encryption state instead of assuming the bridge made them plaintext. If an archive room becomes encrypted, or its state cannot be established, mirroring stops. R2 can retain encrypted Matrix media as ciphertext in a future Tuwunel deployment, but that does not make it server-searchable or give the bot access to plaintext.

Collaborative ODT and pad boundaries

ODT editing is oCIS file -> collaboration WOPI host -> Collabora -> save/version back through oCIS -> R2. Matrix posts links and notifications; it does not edit ODT. The editor gets short-lived, permission-scoped launch tokens and file locks. Test simultaneous edits, reconnects, saves, versions and grant revocation; keep Collabora's working files in transient RAM and block launch if that cannot be guaranteed for the selected image.

SharedPad / PadExportPort allow Etherpad- or HedgeDoc-compatible group pad links and revision exports. Pad content, revisions and attachments must be durable R2 payloads; local storage may retain only identifiers, ACLs and session metadata. Standard pad deployments typically persist document text/revisions in their database: they need a separately validated R2 persistence adapter or another compliant service. R2 attachment support or periodic exports alone is insufficient. Consequently there is no running pad service in this scaffold, and PADS_ENABLED=false. Approved exports become ordinary versioned oCIS resources with checksums and group retention; live ODT collaboration remains Collabora's responsibility. This preserves the lightweight pad option without introducing an incompatible local payload database.