Artur Mukhamadiev
6743c358f5
Every internal failure was mapped to a fixed sanitized code before anything recorded the cause, so an intermittent execution_failed was undebuggable: an exhausted model-call budget, a model turn with no text, and a genuine crash all looked identical. The bridge now writes one bounded line to container stderr on failure with the code, the internal reason, the state and both call counters, and the failure sites pass a reason (budget exhausted, empty final answer with its content block types, token counts against the limits). The wire error is unchanged. The backend logs the agent's terminal code together with its own call counters, and the sanitized tail of container stderr rather than only its byte count. deploy/logging.json gives every logger a timestamp (uvicorn's default config leaves non-uvicorn loggers on logging's fallback handler); override with CONFLUENCE_WEB_LOG_CONFIG.
112 lines
8.8 KiB
Markdown
112 lines
8.8 KiB
Markdown
# Confluence Web Backend Service
|
||
|
||
A FastAPI backend service providing isolated, secure Confluence research orchestration using rootless Docker containers, authoritative tool history, and private artifact retention.
|
||
|
||
## Architecture
|
||
|
||
```text
|
||
backend/
|
||
app.py # FastAPI application factory, session cookies, security headers, endpoints
|
||
settings.py # Validated deployment settings and URL validation/canonicalization
|
||
runner.py # QueryRunner orchestrating deadlines, gate, disconnect cancellation, cleanup
|
||
admission.py # In-memory admission queue (FIFO tickets, reservation, estimate)
|
||
transport.py # Bounded NDJSON bridge reader/writer and protocol state machine (Rev 1)
|
||
containers.py # Rootless Docker container lifecycle, tmpfs mounts, kill/remove, reconciliation
|
||
confluence.py # Request-scoped Confluence client, streaming bounds, and tool dispatcher
|
||
model.py # Neutral model contract adapter (OpenAI-compatible Chat Completions / Fake)
|
||
history.py # Authoritative in-memory tool execution history and page auditing
|
||
artifacts.py # Private temporary artifact staging, validation, session binding, downloads
|
||
dev/
|
||
fake_peer.py # Scripted container peer implementing NDJSON contract
|
||
fake_entrypoint.py # Container entrypoint for fake peer
|
||
Dockerfile.fake # Test container image definition
|
||
README.md # Configuration & documentation
|
||
```
|
||
|
||
## Environment Variables
|
||
|
||
| Variable | Default | Description |
|
||
|---|---|---|
|
||
| `CONFLUENCE_WEB_APPROVED_ORIGINS` | `https://approved.example.com` | Comma-separated list of approved Confluence base origins and context paths. Exposed in canonical form by `GET /api/v1/config`, which the UI uses to offer the origin as a fixed choice. |
|
||
| `CONFLUENCE_WEB_CORPORATE_CA_PATH` | `None` | Path to corporate CA bundle for TLS verification if needed. |
|
||
| `CONFLUENCE_WEB_MODEL_PROVIDER` | `fake` | Model provider adapter (`fake`, `openai`). |
|
||
| `CONFLUENCE_WEB_MODEL_NAME` | `fake-model` | Model name (e.g. `gpt-4o`). |
|
||
| `CONFLUENCE_WEB_MODEL_API_KEY` | `None` | Backend-held API key for the model provider. |
|
||
| `CONFLUENCE_WEB_MODEL_ENDPOINT` | `None` | Custom model provider endpoint URL. |
|
||
| `CONFLUENCE_WEB_MODEL_CONTEXT_WINDOW_TOKENS` | `128000` | Model context window token limit. |
|
||
| `CONFLUENCE_WEB_MODEL_MAX_OUTPUT_TOKENS` | `4096` | Model max output tokens limit. |
|
||
| `CONFLUENCE_WEB_RUNTIME_IMAGE` | `confluence-agent:latest` | Pinned Docker container image for the pi agent runtime. |
|
||
| `CONFLUENCE_WEB_DOCKER_HOST` | `None` | Custom Docker host / unix socket path. |
|
||
| `CONFLUENCE_WEB_CONTAINER_LABEL_KEY` | `com.confluence_web.app` | Docker container label namespace key. |
|
||
| `CONFLUENCE_WEB_CONTAINER_LABEL_VALUE` | `query-runner` | Set a unique value for each deployment; reconciliation uses this value. |
|
||
| `CONFLUENCE_WEB_CONTAINER_INSTANCE_ID` | generated UUID | Non-secret instance metadata label; orphan discovery also includes previous instances. |
|
||
| `CONFLUENCE_WEB_ARTIFACT_DIR` | `/tmp/confluence_web_artifacts` | Private backend-managed directory for artifact staging and downloads. |
|
||
| `CONFLUENCE_WEB_BIND_HOST` | `127.0.0.1` | Bind address for FastAPI service. |
|
||
| `CONFLUENCE_WEB_BIND_PORT` | `8000` | Port for FastAPI service. |
|
||
| `CONFLUENCE_WEB_FRONTEND_DIST_DIR` | `None` | Directory containing built frontend static files. |
|
||
| `CONFLUENCE_WEB_DEV_MODE` | `false` | Explicit opt-in development mode with fake test dependencies. |
|
||
| `CONFLUENCE_WEB_QUERY_TIMEOUT_SECONDS` | `180.0` | Total query execution deadline; must not exceed `CONFLUENCE_WEB_MAX_DEADLINE_SECONDS`. |
|
||
| `CONFLUENCE_WEB_MAX_DEADLINE_SECONDS` | `900.0` | Protocol maximum for one query (60–3600). Passed into the agent container so the supervisor and bridge enforce the same bound. |
|
||
| `CONFLUENCE_WEB_CLEANUP_TIMEOUT_SECONDS` | `10.0` | Dedicated cleanup timeout. |
|
||
| `CONFLUENCE_WEB_MAX_CONFLUENCE_CALLS` | `100` | Confluence tool calls allowed per query (1–1000); cache hits are free on the backend side. Passed into the agent container, whose bridge enforces the same count. The history keeps at most 100 tool entries, so budgets above 100 lose later entries. |
|
||
| `CONFLUENCE_WEB_MAX_MODEL_CALLS` | `50` | Model requests allowed per query (1–1000). Passed into the agent container, whose bridge enforces the same count. |
|
||
| `CONFLUENCE_WEB_QUEUE_RESERVATION_SECONDS` | `45` | Admission queue reservation window after promotion; allowed 30-60. See [QUEUE_SPECIFICATION.md](../docs/QUEUE_SPECIFICATION.md). |
|
||
| `CONFLUENCE_WEB_QUEUE_HEARTBEAT_SECONDS` | `15` | Admission queue heartbeat timeout for queued (not yet reserved) tickets; allowed 5-60. |
|
||
| `CONFLUENCE_WEB_QUEUE_MAX_LENGTH` | `20` | Maximum queued tickets, excluding the reserved and running sessions; allowed 1-100. |
|
||
|
||
## Running the Service
|
||
|
||
Use the existing virtual environment (`.venv/bin/python -m pip install -r requirements.txt` if dependencies need installing).
|
||
|
||
Version 1 requires **one worker process**. Multiple workers have separate query gates, sessions and artifact indexes and violate the service assumptions. Do not run concurrent service instances with the same artifact directory or container label value.
|
||
|
||
Production configuration must set `CONFLUENCE_WEB_MODEL_PROVIDER=openai`, the provider key/model, approved Confluence origins, runtime image, and a deployment-specific container label value. Unknown providers and missing/blank real-provider keys fail configuration; Docker/provider failures never select a fake automatically. The `fake` provider is an explicit test configuration.
|
||
|
||
```bash
|
||
.venv/bin/uvicorn backend.app:create_app --factory --workers 1 --host 127.0.0.1 --port 8000
|
||
```
|
||
|
||
Startup requires rootless Docker and cgroup v2 with memory, CPU quota/period and PID enforcement reported by the daemon. It purges prior-run artifacts and confirms removal of all residual containers in the deployment label namespace before accepting work. Maintenance runs every 60 seconds while the query gate is idle. Uncertain cleanup leaves query admission closed until reconciliation succeeds.
|
||
|
||
For network-free development, all three dependencies are fake unless explicitly injected: an in-process NDJSON peer, a fake model, and an HTTPX Confluence substitute. No Docker or remote provider is required. Use URL `https://approved.example.com` and PAT `dev-pat` in the browser or API client; other PATs fail verification.
|
||
|
||
```bash
|
||
CONFLUENCE_WEB_DEV_MODE=true .venv/bin/uvicorn backend.app:create_app --factory --workers 1 --host 127.0.0.1 --port 8000
|
||
```
|
||
|
||
Browser/manual POST clients must send an Origin matching the backend scheme, host and effective port. Bootstrap `GET /` issues the server-owned session cookie. Downloads require that cookie; query UUIDs are not ownership credentials.
|
||
|
||
## Running Tests
|
||
|
||
Build the independent fake image before real Docker tests:
|
||
|
||
```bash
|
||
docker build -t confluence-fake-agent:test -f backend/dev/Dockerfile.fake .
|
||
.venv/bin/python -m pytest tests/backend
|
||
```
|
||
|
||
Docker checks run against a responsive rootless daemon and the built image; a missing/unresponsive daemon skips those checks, rather than claiming success. To run only deterministic tests:
|
||
|
||
```bash
|
||
.venv/bin/python -m pytest tests/backend --ignore=tests/backend/test_docker_live.py
|
||
```
|
||
|
||
Bare pytest deselects the existing `live` crawler marker. The crawler configuration tests still require their own environment variables. A completely offline full-suite invocation uses synthetic values:
|
||
|
||
```bash
|
||
CONFLUENCE_PAT=backend-synthetic-token-no-network CONFLUENCE_URL=https://approved.example.com .venv/bin/python -m pytest
|
||
```
|
||
|
||
### Debugging a failed run
|
||
|
||
The HTTP error is a fixed sanitized code, so the reason lives in the backend log
|
||
(timestamped via `deploy/logging.json`; override with `CONFLUENCE_WEB_LOG_CONFIG`).
|
||
A failed query logs two lines: the agent's terminal code with the backend's own
|
||
model/Confluence call counters, and the tail of the container's stderr, where the
|
||
agent writes `[agent] run failed code=... reason=... model_calls=n/m tool_calls=n/m`.
|
||
An exhausted model-call budget and a model that ends its turn with no text both
|
||
surface as `execution_failed`; the counters and the reason tell them apart. Raising
|
||
`CONFLUENCE_WEB_MAX_MODEL_CALLS` helps only the first.
|
||
|
||
Resource contracts remain 16 MiB decoded prompt, 128 MiB decoded answer, 128 KiB verify body and `6 * 16 MiB + 64 KiB` query body. Request bytes are counted while reading, independent of Content-Length. History is at most 100 entries / 128 MiB, with 64 KiB reserved metadata per entry. Artifact limits are fixed contract values: 20 files, 10 MiB/file, 50 MiB/query, 500 MiB global; default TTL is 900 seconds. Call totals default to 100 Confluence / 50 model per query and are set with `CONFLUENCE_WEB_MAX_CONFLUENCE_CALLS` / `CONFLUENCE_WEB_MAX_MODEL_CALLS`; retention limits have Python Settings defaults but no environment switches.
|