confluence_web/backend/README.md

Confluence Web Backend Service

A FastAPI backend service providing isolated, secure Confluence research orchestration using rootless Docker containers, authoritative tool history, and private artifact retention.

Architecture

backend/
  app.py                 # FastAPI application factory, session cookies, security headers, endpoints
  settings.py            # Validated deployment settings and URL validation/canonicalization
  runner.py              # QueryRunner orchestrating deadlines, gate, disconnect cancellation, cleanup
  admission.py           # In-memory admission queue (FIFO tickets, reservation, estimate)
  transport.py           # Bounded NDJSON bridge reader/writer and protocol state machine (Rev 1)
  containers.py          # Rootless Docker container lifecycle, tmpfs mounts, kill/remove, reconciliation
  confluence.py          # Request-scoped Confluence client, streaming bounds, and tool dispatcher
  model.py               # Neutral model contract adapter (OpenAI-compatible Chat Completions / Fake)
  history.py             # Authoritative in-memory tool execution history and page auditing
  artifacts.py           # Private temporary artifact staging, validation, session binding, downloads
  dev/
    fake_peer.py         # Scripted container peer implementing NDJSON contract
    fake_entrypoint.py   # Container entrypoint for fake peer
    Dockerfile.fake      # Test container image definition
  README.md              # Configuration & documentation

Environment Variables

Variable	Default	Description
CONFLUENCE_WEB_APPROVED_ORIGINS	https://approved.example.com	Comma-separated list of approved Confluence base origins and context paths.
CONFLUENCE_WEB_CORPORATE_CA_PATH	None	Path to corporate CA bundle for TLS verification if needed.
CONFLUENCE_WEB_MODEL_PROVIDER	fake	Model provider adapter (fake, openai).
CONFLUENCE_WEB_MODEL_NAME	fake-model	Model name (e.g. gpt-4o).
CONFLUENCE_WEB_MODEL_API_KEY	None	Backend-held API key for the model provider.
CONFLUENCE_WEB_MODEL_ENDPOINT	None	Custom model provider endpoint URL.
CONFLUENCE_WEB_MODEL_CONTEXT_WINDOW_TOKENS	128000	Model context window token limit.
CONFLUENCE_WEB_MODEL_MAX_OUTPUT_TOKENS	4096	Model max output tokens limit.
CONFLUENCE_WEB_RUNTIME_IMAGE	confluence-agent:latest	Pinned Docker container image for the pi agent runtime.
CONFLUENCE_WEB_DOCKER_HOST	None	Custom Docker host / unix socket path.
CONFLUENCE_WEB_CONTAINER_LABEL_KEY	com.confluence_web.app	Docker container label namespace key.
CONFLUENCE_WEB_CONTAINER_LABEL_VALUE	query-runner	Set a unique value for each deployment; reconciliation uses this value.
CONFLUENCE_WEB_CONTAINER_INSTANCE_ID	generated UUID	Non-secret instance metadata label; orphan discovery also includes previous instances.
CONFLUENCE_WEB_ARTIFACT_DIR	/tmp/confluence_web_artifacts	Private backend-managed directory for artifact staging and downloads.
CONFLUENCE_WEB_BIND_HOST	127.0.0.1	Bind address for FastAPI service.
CONFLUENCE_WEB_BIND_PORT	8000	Port for FastAPI service.
CONFLUENCE_WEB_FRONTEND_DIST_DIR	None	Directory containing built frontend static files.
CONFLUENCE_WEB_DEV_MODE	false	Explicit opt-in development mode with fake test dependencies.
CONFLUENCE_WEB_QUERY_TIMEOUT_SECONDS	180.0	Total query execution deadline; must not exceed CONFLUENCE_WEB_MAX_DEADLINE_SECONDS.
CONFLUENCE_WEB_MAX_DEADLINE_SECONDS	900.0	Protocol maximum for one query (60–3600). Passed into the agent container so the supervisor and bridge enforce the same bound.
CONFLUENCE_WEB_CLEANUP_TIMEOUT_SECONDS	10.0	Dedicated cleanup timeout.
CONFLUENCE_WEB_MAX_CONFLUENCE_CALLS	100	Confluence tool calls allowed per query (1–1000); cache hits are free on the backend side. Passed into the agent container, whose bridge enforces the same count. The history keeps at most 100 tool entries, so budgets above 100 lose later entries.
CONFLUENCE_WEB_MAX_MODEL_CALLS	50	Model requests allowed per query (1–1000). Passed into the agent container, whose bridge enforces the same count.
CONFLUENCE_WEB_QUEUE_RESERVATION_SECONDS	45	Admission queue reservation window after promotion; allowed 30-60. See QUEUE_SPECIFICATION.md.
CONFLUENCE_WEB_QUEUE_HEARTBEAT_SECONDS	15	Admission queue heartbeat timeout for queued (not yet reserved) tickets; allowed 5-60.
CONFLUENCE_WEB_QUEUE_MAX_LENGTH	20	Maximum queued tickets, excluding the reserved and running sessions; allowed 1-100.

Running the Service

Use the existing virtual environment (.venv/bin/python -m pip install -r requirements.txt if dependencies need installing).

Version 1 requires one worker process. Multiple workers have separate query gates, sessions and artifact indexes and violate the service assumptions. Do not run concurrent service instances with the same artifact directory or container label value.

Production configuration must set CONFLUENCE_WEB_MODEL_PROVIDER=openai, the provider key/model, approved Confluence origins, runtime image, and a deployment-specific container label value. Unknown providers and missing/blank real-provider keys fail configuration; Docker/provider failures never select a fake automatically. The fake provider is an explicit test configuration.

.venv/bin/uvicorn backend.app:create_app --factory --workers 1 --host 127.0.0.1 --port 8000

Startup requires rootless Docker and cgroup v2 with memory, CPU quota/period and PID enforcement reported by the daemon. It purges prior-run artifacts and confirms removal of all residual containers in the deployment label namespace before accepting work. Maintenance runs every 60 seconds while the query gate is idle. Uncertain cleanup leaves query admission closed until reconciliation succeeds.

For network-free development, all three dependencies are fake unless explicitly injected: an in-process NDJSON peer, a fake model, and an HTTPX Confluence substitute. No Docker or remote provider is required. Use URL https://approved.example.com and PAT dev-pat in the browser or API client; other PATs fail verification.

CONFLUENCE_WEB_DEV_MODE=true .venv/bin/uvicorn backend.app:create_app --factory --workers 1 --host 127.0.0.1 --port 8000

Browser/manual POST clients must send an Origin matching the backend scheme, host and effective port. Bootstrap GET / issues the server-owned session cookie. Downloads require that cookie; query UUIDs are not ownership credentials.

Running Tests

Build the independent fake image before real Docker tests:

docker build -t confluence-fake-agent:test -f backend/dev/Dockerfile.fake .
.venv/bin/python -m pytest tests/backend

Docker checks run against a responsive rootless daemon and the built image; a missing/unresponsive daemon skips those checks, rather than claiming success. To run only deterministic tests:

.venv/bin/python -m pytest tests/backend --ignore=tests/backend/test_docker_live.py

Bare pytest deselects the existing live crawler marker. The crawler configuration tests still require their own environment variables. A completely offline full-suite invocation uses synthetic values:

CONFLUENCE_PAT=backend-synthetic-token-no-network CONFLUENCE_URL=https://approved.example.com .venv/bin/python -m pytest

Resource contracts remain 16 MiB decoded prompt, 128 MiB decoded answer, 128 KiB verify body and 6 * 16 MiB + 64 KiB query body. Request bytes are counted while reading, independent of Content-Length. History is at most 100 entries / 128 MiB, with 64 KiB reserved metadata per entry. Artifact limits are fixed contract values: 20 files, 10 MiB/file, 50 MiB/query, 500 MiB global; default TTL is 900 seconds. Call totals default to 100 Confluence / 50 model per query and are set with CONFLUENCE_WEB_MAX_CONFLUENCE_CALLS / CONFLUENCE_WEB_MAX_MODEL_CALLS; retention limits have Python Settings defaults but no environment switches.