Realtime backend spec (NestJS, TypeScript 6)

Owner

Backend

Responsibilities

WebSocket gateway for Yjs CRDT sync, Redis-backed presence service, JWT validation at handshake, horizontal scaling via Redis. See ADR 0002.

Protocol

• YjsGateway (src/yjs/) speaks the standard Yjs sync sub-protocol (y-protocols/sync) over a raw WebSocket at a fixed path, /yjs. The board id is not a path segment — @nestjs/platform-ws's WsAdapter routes an upgrade to a gateway by exact pathname match, with no wildcard/pattern support, so a dynamic per-board path isn't possible without a custom adapter. Instead, the board id travels as a query parameter: ws://<host>/yjs?board=<board-id>.
• One Y.Doc per board id, held in memory (YjsRoomRegistry) and persisted through the business backend (see Persistence); document updates are relayed between instances through Valkey (see Document relay).
• WS handshake auth: every connection needs a WS token for its board (see Authentication below).

Authentication to the business backend

Every call to the document API carries Authorization: Bearer <service token> (ADR 0017): InternalTokenSigner signs a JWT locally (HS256 with INTERNAL_API_SECRET, issuer elysion-realtime, audience elysion-backend-internal, valid for 60 seconds, a new one for each call), so there is no call to anybody to get it and persistence does not depend on Keycloak. The service does not start without INTERNAL_API_SECRET (at least 32 characters, and not the value of WS_TOKEN_SECRET: one leaked secret must not forge both kinds of token). A 401 from the backend (a different secret on the two sides) is a failed call like any other: the board is not loaded, saves are retried, and the logs say Document store answered 401.

Authentication

?board=<id>&token=<ws token>: the token is the board-scoped credential the BFF issues (identity.md, POST /api/realtime/token). src/auth/WsTokenVerifier checks it locally: HS256 with WS_TOKEN_SECRET (shared with the BFF), issuer elysion-bff, audience elysion-realtime, expiry (5 s of clock tolerance) and the claims sub, boardId and role (owner, editor, viewer), typed by @elysion/shared-types. There is no call to the BFF or Keycloak, so a handshake never depends on either.

• Admission (YjsGateway.admit) happens before the board is loaded: a token that is missing, malformed, expired, signed with another key or without valid claims closes the socket with 4401; a valid token whose boardId is not the board of the URL closes it with 4403 (otherwise one valid token would open every board); a failure of the check itself closes with 1011, never lets the connection in. A missing board is still 1008. These are the contract's codes (WS_CLOSE_UNAUTHORIZED, WS_CLOSE_FORBIDDEN in @elysion/shared-types), in the private 4000 range instead of the generic 1008. Messages that arrive while the token is checked and the board loads are handled in order afterwards.
• Bound to the connection: the verified sub and role are kept per socket in YjsRoom.memberBySocket (removed when the socket goes away), for read-only viewers (#303) and presence identity. Presence and the document relay are unchanged for admitted connections.
• Viewers are read-only, except for comments (#303, #733): on a connection whose token has role: viewer, sync step 1 (asking for the board's state) is always handled; its sync step 2 and update messages are applied and relayed only when they change nothing but the comment threads, the map session.comments and the types inside it (src/yjs/viewer-writes.ts, viewerMayApply). Anything else is dropped, never applied and never relayed: a change to an element, to the timer or any other key of session, to another root type, replacing or deleting the comments map itself, or an update that mixes a comment with such a change (the whole update goes). A viewer may create comments when the board has none; once it exists only its content can change, so the editors' canvas creates the map when it is missing. The check applies the update to a scratch copy of the room's document and reads what the transaction changed (the structs of an update name their parents by reference, not by path, so this is the only way to know). An update that cannot be applied completely (it refers to structs the board does not have, or deletes ones it does not) is refused, not held, since a held struct could change something else once it resolves; so is anything that does not decode. The copy costs about one encoding of the document, so only a viewer's writes pay it, and only after the size limits (MAX_UPDATE_BYTES, the document limit). Limits of this rule: the server cannot tell whose comment a viewer edits or deletes inside comments (authorship is a rule of the interface), and a viewer can fill the comments up to the document limit like an editor can fill the board. Awareness (cursors) still works. The client is told only if it keeps sending refused writes: after 20 the socket is closed with 4403 (Viewers cannot change the board); a viewer that just looks sends one (the reply to the server's first sync step, which changes nothing and is accepted) and is not affected. Owners and editors are not restricted by role here.
• A token is checked once, at the handshake: an open connection outlives it. The client asks the BFF for a new token on every connect.
• Fail closed: the service does not start without WS_TOKEN_SECRET (at least 32 characters; src/config/). For local runs copy apps/realtime/.env.example to apps/realtime/.env (loaded at startup).

Persistence

Board documents survive a restart (ADR 0011).

• The business backend stores one full Yjs snapshot per board (table BoardDocuments, bytea, with a version) behind GET/PUT/DELETE /internal/boards/{id}/document (minimal API endpoints in Endpoints/BoardDocumentEndpoints.cs). PUT needs If-Match: "<version>", or If-None-Match: * for the first save; a stale version answers 409 with the stored state, no precondition 428, and a UUID board id with no board record 404 (#778: a room that still has a deleted board open cannot bring its document back; ids that are no UUID, such as default, are not checked). The route is not exposed at the edge. Deleting a board (DELETE /boards/{id}) deletes its document; the document table has no foreign key to Boards, because ids such as default have no board row.
• Realtime reaches it through the DocumentStore interface (src/persistence/: load, save, delete); HttpDocumentStore is the implementation, configured with BUSINESS_BACKEND_URL (default http://localhost:5174).
• YjsRoomRegistry.getOrLoad loads a room's stored state before the first sync step is answered; connections arriving during the load are queued, and concurrent first connections share one load. If the load fails the connection is closed with code 1011 (Board could not be loaded): an empty board is never served in place of an unloaded one, because the next save would overwrite the real content.
• A changed room is saved 2 s after its last change (at the latest 10 s after the first unsaved change), when its last client leaves, and when the process stops (enableShutdownHooks). A save names the version it is based on. If another instance saved first (409), realtime merges the stored state in with Y.applyUpdate and saves the union. A failed save is retried after 1 s, doubling up to 30 s, and logged as an error; the room stays in memory meanwhile. An unchanged room is never saved.
• Unloading idle rooms: when the last client of a room leaves, the room is saved at once and unloaded 30 s later (ROOM_EVICT_AFTER_MS changes the grace period): its Y.Doc and Awareness are destroyed and its presence channel is unsubscribed. A client that connects during the grace period keeps the room; one that connects later gets the stored state. If the final save keeps failing the room stays in memory (its content is the only copy) and unloading is retried every grace period.
• Instances do not rely on the store to see each other's edits: the document relay (below) does that within milliseconds, the store only matters across restarts and as the conflict check.
• Sizes: the state is about the size of the live elements' JSON (a typical board is a few hundred KB, a large one a few MB) and grows by about 1 KB per two seconds of dragging because of Yjs tombstones; see the ADR for the measurements.
• A save that can never work (#774). When the backend answers a save with 404 or 410 (the board was deleted) or 400, 413 or 422 (the state is refused), the room gives up at once: the unsaved state is dropped, the connected clients are closed with 4403 (their reconnect asks the BFF for a token and is told what is true) and the room is unloaded. 401 and 403 are not among them (a secret set right later makes them work), and other failures are retried with the growing delay for as long as somebody is connected; a room nobody has open is given up after maxFailuresWithoutClients (30) failed saves in a row, so a dead backend cannot keep rooms in memory for ever.

Who is still a member

The role in the WS token is the one the person had when it was issued. MembershipWatcher (src/membership/) keeps open sockets honest: every MEMBERSHIP_RECHECK_MS (default 15000; 0 switches it off) it asks the business backend's GET /internal/boards/{id}/access?sub=<subject> for every person who has a socket on a board of this instance, once per person and board. A 404 (removed, board deleted, never a member) closes their sockets with 4403 ("No longer a member of this board"); another role replaces the one in YjsRoom.memberBySocket, so a demoted editor's writes are dropped from the next message on; an error or timeout changes nothing. Every instance checks its own sockets. Logs: WebSocket connection closed: the person has no role on the board any more, Role of an open connection changed. Tokens are single-use through Valkey (elysion:wsjti:), see identity.

Board settings in the document

The document's map meta holds settings of the board (not of a person), outside the Excalidraw binding: grid is { show, snap, size, guides } (#754, #755; size is 10, 20 or 40; guides is snapping to other elements; a board without the key has { show: true, snap: true, size: 20, guides: true }, a missing guides is true), generation is compaction's (ADR 0026). The realtime service does not read them: it stores and relays them like any other part of the document, and a copy of a board keeps them. Editors write them, a viewer's canvas never does; the canvas treats a value that is not valid as the default.

Limits and compaction (ADR 0026)

The document. A board is one Y.Map of element ids to element JSON plus a few small maps (meta, session, files). Every write leaves a small struct behind; Yjs already drops the content of an overwritten value, so re-encoding wins nothing (measured in the ADR, apps/realtime/scripts/measure-compaction.mjs). What shrinks a board is rebuilding it from its live elements: 866 KB to 262 KB after 20 minutes of five people dragging, 4.5 MB to 1.3 MB on a large board.

Compaction (YjsRoomRegistry.compact, src/yjs/compaction.ts). When an idle room is unloaded (30 s after its last client left, ROOM_EVICT_AFTER_MS) and its state is larger than COMPACT_THRESHOLD_BYTES (1 MiB, 0 is off), the document is rebuilt: every root map is copied entry by entry into a fresh Y.Doc (new client id, new clocks), and meta.generation is set to a new random id. It is saved only if it is at most half as large, and only if nobody on any instance has the board open (the presence snapshot in Valkey is empty; a failure of that check means no rebuild). The room is dropped afterwards, the next load starts from the small document, and a reset message on the relay channel makes other instances drop an idle copy that has nothing unsaved. Whoever asks for the room during the rebuild waits for it. It never happens while anybody is connected, so no cursor, selection or undo history is touched. Metric elysion_realtime_document_compactions_total, log line Rebuilt board …: A bytes became B.

The generation. meta.generation exists only after the first rebuild (a board that was never rebuilt has none, and writing one at load would make every unchanged board save). A client reads it after its first sync (sync step 2) and sends it as generation on every reconnect: the generation, or an empty string when its copy had none. No generation parameter is a first connection. The gateway compares it with the document's ('' for none): different closes the socket with 4409 (WS_CLOSE_STALE_COPY). The canvas then throws its Y.Doc away, connects again with a new one and, once the server's state has arrived, writes what its scene holds into it by the usual per-element rule (an element goes in when the document lacks it or has an older version), so what the tab changed while away is kept and what others changed is not overwritten.

Limits, refused and never dropped silently:

Limit	Where	What happens
MAX_UPDATE_BYTES, 2 MiB per Yjs update	the gateway, before applying	the connection is closed with 1009; the canvas says "That change is too large to send." (the WebSocket's maxPayload stays 16 MiB for sync step 2 of a large board)
MAX_DOCUMENT_BYTES, 8 MiB encoded state	the gateway: YjsRoomRegistry.fits (an estimate: the size at load and at each save plus the updates since, measured for real only when the estimate says no)	the update is not applied and the client gets message type 4 (board-full, no payload); the canvas says "This board is full."; reads and presence go on; a rebuild when the room empties may bring it under the limit
20,000 live elements	the canvas, before it writes (MAX_ELEMENTS in excalidraw-binding.ts; the element's max-elements attribute lowers it, for tests)	the new element is not written and is taken off the scene, the canvas says "This board is full: 20,000 elements." (the notice event)
32 MiB	the business backend (MaxStateBytes)	413 instead of 500: the realtime service gives such a board up (see "A save that can never work")

Settings: MAX_UPDATE_BYTES, MAX_DOCUMENT_BYTES, COMPACT_THRESHOLD_BYTES of the realtime service (compose, .env.example, the chart's limits.*). Images are references to the object store and are not counted here.

Document relay

Two realtime instances serving one board show the same content: DocumentRelay (src/document/) publishes every Yjs update that a client sent to an instance on the Valkey channel elysion:doc:<boardId> (the elysion: prefix keeps the shared Valkey tidy, ADR 0006), and every other instance that has the room open applies it. It has the same shape as PresenceRelay: each envelope carries the sender's instance id, so an instance ignores its own messages, and one subscription shares the two Valkey connections.

• Updates that arrive from the relay are applied with their own transaction origin: they are sent to the receiving instance's clients, but not published again (no ping-pong) and not saved by the receiver. The instance whose client made the change saves it (see Persistence); if that instance dies inside the save window the client still has the change and resends it when it reconnects.
• A lost update must not leave instances diverged, so the relay heals itself. Whenever an instance starts following a board (a room is loaded) and whenever Valkey reconnects after an outage, it publishes hello with its Yjs state vector. Every peer answers with the update the sender lacks and with its own state vector (hello-ack); the sender answers that with the update the peer lacks. An exchange ends after one round, and it works in both directions, so changes made on either side during an outage are exchanged.
• Valkey being unreachable never stops editing: publishing failures are logged and the room keeps working and saving on its own; the hello after the reconnect catches up.
• An unloaded room (see Persistence) unsubscribes from the channel; when someone returns it loads from the store and says hello.

Voting notifications (ADR 0029)

The votes of a board live in the business backend, not in the document. After every change (start, vote, retract, close, clear) the backend calls POST /internal/boards/{boardId}/voting on this service with { "event": "start|vote|retract|close|clear", "voting": VotingBroadcast | null } (voting is null only for clear) and the service tells every client of the board. The route is not routed at the edge, and 204 is the answer; 400 for a board id that is no UUID or a body that does not match the shape (voting-broadcast.ts checks every field, passes on only the known ones and caps lengths and counts; the body limit is Express's 100 KiB), 401 without a valid service token.

Service token (InternalCallVerifier): Authorization: Bearer <jwt>, HS256 with INTERNAL_API_SECRET (the key of ADR 0017, in the other direction), iss elysion-business-backend, aud elysion-realtime (a token made for the backend's own internal API does not work here), exp and iat required, at most 60 s of life (a token that claims more is refused), 5 s clock tolerance.

Relay. The instance that got the call sends the message to its own clients of the board and publishes it on the board's document channel (elysion:doc:<boardId>) as relay type voting (the data is the UTF-8 JSON); every other instance that holds the room sends it to its clients. An instance with no client of the board has no subscription, and needs none.

Wire format to clients: message type 5 (voting). Binary WebSocket frame: varUint(5) followed by varString(json) (lib0 writeVarString: a varUint byte length, then UTF-8). It goes to every socket of the board, viewers included. The JSON is { "event": ..., "voting": VotingBroadcast | null } with VotingBroadcast = { id, name, votesPerPerson, status: "open"|"closed", named, startedBy: {id, name}, startedAt, closedAt: string|null, placed?: { [userId]: number }, result?: [{ elementId, count, voters?: [{id, name}] }] }: placed while the voting is open (how many votes each person has placed, never on what), result once it is closed (voters only when it is named). A client never sends type 5; the gateway ignores it. A failed notification is not retried: clients fetch the voting from the API when they connect.

Presence

Cursor/avatar presence (Yjs awareness, message type 1) is broadcast cross-instance via Redis pub/sub (PresenceRelay, src/presence/), not just to other clients on the same realtime process:

• YjsRoomRegistry creates one y-protocols/awareness Awareness object per room (alongside its Y.Doc) and applies every incoming awareness update to it via applyAwarenessUpdate.
• YjsGateway relays each incoming awareness message to local clients in-memory (as before) and publishes it to a Redis channel keyed by board id (PresenceRelay.publish); every instance holding that room subscribes to the same channel and relays what it receives to its own local clients. Each publish is tagged with a per-process instance id so an instance ignores its own message coming back off the subscription.
• A client that joins is caught up without a gap: YjsGateway first subscribes to the board's presence channel and waits until the subscription is active (PresenceRelay.subscribe resolves for every caller only then, also for a connection that arrives while the first one is still subscribing), and only then reads the snapshot. An announcement is recorded in the hash before it is published, so one published before the subscription is in the snapshot and one published after reaches the subscription.
• YjsRoomRegistry's Awareness.on('update', ...) listener persists every changed client's state to a Redis hash keyed by board id (PresenceRelay.recordState/removeState), independent of pub/sub — this is what makes a snapshot possible: a newly-connecting client (on any instance, including one that has never seen this board before) is sent every other client's current state (PresenceRelay.snapshot → YjsGateway.sendPresenceSnapshot), mirroring the sync-step-1-on-connect pattern already used for doc state.
• Valkey is Elysion's own (a valkey service of docker-compose.yml, ADR 0023) and has a password (VALKEY_PASSWORD, a development default in the compose file; REDIS_URL=redis://:<password>@host:6379; #775): the instances trust what arrives on a relay channel, so who can publish there writes into any board. A message that is not a well-formed envelope ({ from, type?, data } as JSON, a known type) is dropped with a warning at most every 10 s, never thrown from the subscriber's callback (src/redis/envelope.ts); channels and keys are still namespaced: elysion:presence:<boardId> (pub/sub) and elysion:presence:state:<boardId> (hash).
• All Redis calls from the gateway/registry are fire-and-forget with a logged warning on failure — a Redis outage degrades presence to same-instance-only, it doesn't drop WS connections or doc sync.
• A client's presence ends when its socket does, whatever the reason (clean close, network drop, killed browser): YjsRoomRegistry remembers which awareness client ids each socket announced, and YjsGateway.handleDisconnect removes them from the room's Awareness, which deletes them from Valkey, then sends the removal to the local clients and publishes it so clients on other instances drop the collaborator too.
• Entries left behind by an instance that crashed expire: each hash entry is stored as <written-at-ms>:<base64 update>, a client's every announcement (about every 15 s) rewrites it, and snapshot deletes and skips entries older than 30 s (PRESENCE_ENTRY_TTL_MS, the Awareness outdatedTimeout) as well as entries without a timestamp (the format before this rule). The whole hash also expires after 24 hours without a write, so keys of tests and deleted boards do not pile up in the shared Valkey.

Logging

ADR 0025. nestjs-pino with the options of packages/node-logging (@elysion/node-logging, shared with the BFF so that both write the same line); src/logging/logging.module.ts, switched on in main.ts (bufferLogs, useLogger).

• Format. One JSON object per line on stdout (LOG_FORMAT=json, the default whenever stdout is not a terminal); readable text through pino-pretty in a terminal (LOG_FORMAT=text). LOG_LEVEL is trace, debug, info (default), warn, error or fatal; loadConfig validates both at start, like the secrets. OTEL_EXPORTER_OTLP_LOGS_ENDPOINT (and OTEL_EXPORTER_OTLP_LOGS_HEADERS) also sends every line to an OTLP/HTTP logs endpoint, the dev stack's viewer (below); unset, nothing is sent.
• Fields (the same in the BFF and the business backend): timestamp (ISO 8601, UTC), level, service (elysion-realtime), requestId, message, userId (the WS token's sub and nothing else about the person), err on an exception, context (the Nest class that logged), and the fields of the event (boardId, role, reason, closeCode). An HTTP request (/health, /metrics) ends with the one request line: http with method, route, status, durationMs (at debug for those two paths).
• A connection has a request id. YjsGateway.handleConnection takes the X-Request-Id of the upgrade request when it matches ^[A-Za-z0-9._-]{8,64}$ and creates a UUID otherwise (a browser cannot set the header on a WebSocket, so in practice the service creates it). The id and, once the WS token is verified, the user are kept on a ConnectionContext that every callback of the socket enters again (runInConnection), so every line the connection causes carries requestId and userId: WebSocket connection admitted and WebSocket connection closed (boardId, role), a viewer's dropped writes, a failed load.
• The internal document API gets the same id. HttpDocumentStore sends X-Request-Id on every call, so the business backend's lines for a load or a save carry the connection's id. A save runs from a timer, outside any connection: the registry remembers the connection whose change made the room dirty and runs the save as that connection, so a failed save can be followed to the user who caused it.
• Refused connections log WebSocket connection refused at warn with reason (missing, expired, wrong_issuer, wrong_audience, wrong_algorithm, invalid_signature, malformed, missing_claim, invalid, or wrong_board for a valid token for another board) and closeCode (4401 or 4403), never the token. The upgrade URL carries the token in its query, so no line contains a URL: the request line has the route, the connection lines have the board id.
• Never logged: Authorization, Cookie, query strings (the WS token), and secrets. test/logging.e2e-spec.ts connects with markers in each and asserts that none reaches the output.

Open questions

None at the moment. The WS-token issuance question is settled: the BFF issues it (identity.md).