Artur Mukhamadiev
6743c358f5
Every internal failure was mapped to a fixed sanitized code before anything recorded the cause, so an intermittent execution_failed was undebuggable: an exhausted model-call budget, a model turn with no text, and a genuine crash all looked identical. The bridge now writes one bounded line to container stderr on failure with the code, the internal reason, the state and both call counters, and the failure sites pass a reason (budget exhausted, empty final answer with its content block types, token counts against the limits). The wire error is unchanged. The backend logs the agent's terminal code together with its own call counters, and the sanitized tail of container stderr rather than only its byte count. deploy/logging.json gives every logger a timestamp (uvicorn's default config leaves non-uvicorn loggers on logging's fallback handler); override with CONFLUENCE_WEB_LOG_CONFIG.
165 lines
9.1 KiB
Markdown
165 lines
9.1 KiB
Markdown
# Confluence Research Web UI
|
|
|
|
A minimalist web UI that researches a Confluence Data Center instance with a
|
|
[pi](https://github.com/earendil-works/pi) agent running in a fresh, network-less
|
|
rootless Docker container per query, and returns a cited Markdown answer plus
|
|
optional files the agent produced. Each component documents its own interface
|
|
in `agent/README.md`, `backend/README.md` and `frontend/README.md`.
|
|
|
|
The repository also contains the original standalone
|
|
[Confluence crawler](#standalone-crawler), which is unchanged.
|
|
|
|
## Components
|
|
|
|
| Directory | Role | Developer guide |
|
|
| --- | --- | --- |
|
|
| `backend/` | FastAPI service: credentials in memory, Confluence/model upstreams, authoritative tool history, container lifecycle, artifact downloads, serves the frontend | [backend/README.md](backend/README.md) |
|
|
| `agent/` | Container image: pinned pi SDK bridge, Python supervisor, local tools, artifact exporter | [agent/README.md](agent/README.md) |
|
|
| `frontend/` | Static UI (vendored marked, DOMPurify and thinking-orbs engine), same-origin mock server for UI development | [frontend/README.md](frontend/README.md) |
|
|
| `tests/integration/` | Integration-stage tests connecting the real components | this file |
|
|
| `deploy/`, `scripts/`, `Makefile` | Deployment configuration and entry points | this file |
|
|
|
|
## Requirements
|
|
|
|
- Linux with a **rootless** Docker daemon (cgroup v2, memory/CPU/PID limits enforced). `docker info` must list `rootless` under Security Options.
|
|
- Python 3.12+ in `.venv` (verified with 3.14.7), Node 22+ (verified with 26.7) and npm.
|
|
- An OpenAI-compatible Chat Completions endpoint the backend can reach (key held by the backend only).
|
|
- Network access from the backend host to the approved Confluence origin, optionally through a SOCKS5/HTTP proxy.
|
|
- For browser checks: Google Chrome with `--remote-debugging-port=9444`.
|
|
|
|
## Setup
|
|
|
|
```bash
|
|
python3 -m venv .venv && .venv/bin/python -m pip install -r requirements.txt
|
|
make install # also installs agent/ and frontend/ dev dependencies
|
|
make build-image # builds confluence-pi-agent:rev1 and prints its image ID
|
|
cp deploy/confluence-web.env.example deploy/confluence-web.env # then edit
|
|
```
|
|
|
|
`deploy/confluence-web.env` is git-ignored. Every variable is documented in the
|
|
example file and in [backend/README.md](backend/README.md). Pin
|
|
`CONFLUENCE_WEB_RUNTIME_IMAGE` to the image ID printed by `make build-image` for
|
|
an exact runtime, set a deployment-unique `CONFLUENCE_WEB_CONTAINER_LABEL_VALUE`,
|
|
and list only approved Confluence origins.
|
|
|
|
## Run
|
|
|
|
```bash
|
|
make run # scripts/run-backend.sh deploy/confluence-web.env
|
|
```
|
|
|
|
The backend binds `127.0.0.1:8000` by default and serves the UI at
|
|
`http://127.0.0.1:8000/`. Open it in a browser, click the key icon, pick the
|
|
Confluence origin (the list comes from `CONFLUENCE_WEB_APPROVED_ORIGINS`, each
|
|
entry including its context path, for example `https://collab.lge.com/main`),
|
|
enter your personal access token (PAT), test the connection, then ask a question. On
|
|
instances that allow anonymous REST reads, "Test connection" proves the
|
|
destination is reachable but cannot prove the PAT is valid; a wrong token then
|
|
shows up as denied or missing pages. Credentials live only in browser memory and in the backend
|
|
for the duration of a request; a page reload clears them.
|
|
|
|
Version 1 is a small-team deployment: one backend worker, one query at a time
|
|
(a second query gets `busy`), loopback by default. The application has no login
|
|
of its own; anyone who can reach the port can run queries with their own PAT.
|
|
Never bind it to a network interface over plain HTTP, because every request
|
|
carries the user's PAT. If you put a reverse proxy in front, it must accept
|
|
`6*16 MiB + 64 KiB` request bodies, pass client aborts through promptly, avoid
|
|
buffering bodies to disk, and allow at least 200 s per request.
|
|
|
|
### Serving colleagues on an internal network (IP only, TLS)
|
|
|
|
Uvicorn terminates TLS itself; no reverse proxy or DNS name is needed. The
|
|
certificate comes from a private CA that is **name-constrained** to the listed
|
|
IP addresses, so colleagues who import it are trusting it for those addresses
|
|
only. A certificate issued by the same CA for any hostname or any other address
|
|
is rejected by browsers and by OpenSSL (`permitted subtree violation`).
|
|
|
|
```bash
|
|
make tls IP=172.26.10.20 # creates deploy/tls/{ca,server}.{key,crt}; git-ignored
|
|
```
|
|
|
|
Then in `deploy/confluence-web.env`:
|
|
|
|
```
|
|
CONFLUENCE_WEB_BIND_HOST=172.26.10.20
|
|
CONFLUENCE_WEB_TLS_CERT=./deploy/tls/server.crt
|
|
CONFLUENCE_WEB_TLS_KEY=./deploy/tls/server.key
|
|
```
|
|
|
|
Restart with `make run` and open `https://172.26.10.20:8000/`. Under HTTPS the
|
|
session cookie gains the `Secure` flag automatically and the same-origin check
|
|
expects `https://` origins.
|
|
|
|
Hand colleagues only `deploy/tls/ca.crt`, imported once as a trusted root
|
|
(Windows: double-click, "Trusted Root Certification Authorities"; macOS:
|
|
Keychain Access; Chrome on Linux: chrome://settings/certificates; Firefox:
|
|
Settings, Certificates, Authorities). They can read the constraint themselves in
|
|
the certificate details ("Name Constraints: Permitted IP ..."). Keep `ca.key`
|
|
and `server.key` on the server host. The server certificate is valid for 825
|
|
days (the browser maximum); rotate it with the same CA by deleting
|
|
`deploy/tls/server.*` and re-running `make tls` with the CA files kept, or
|
|
rotate everything by deleting the directory.
|
|
|
|
Network-free UI development against explicit fakes (no Docker, PAT `dev-pat`,
|
|
URL `https://approved.example.com`):
|
|
|
|
```bash
|
|
make run-dev
|
|
```
|
|
|
|
## Credentials, expiry and limits
|
|
|
|
- **Confluence PAT**: entered in the browser per session, sent only to the backend over the same origin, never stored, logged, or passed to the container. The token inherits your Confluence permissions.
|
|
- **Model key**: backend environment only (`CONFLUENCE_WEB_MODEL_API_KEY`). For a local llama.cpp server any placeholder works.
|
|
- **Session cookie** `cw_session`: HttpOnly, SameSite=Strict, marks artifact ownership only. It is not authentication.
|
|
- **Artifacts**: at most 20 files, 10 MiB each, 50 MiB per query, 500 MiB total; downloadable for 15 minutes after a successful query, then deleted. Failed or cancelled queries keep nothing.
|
|
- **Query**: 180 s total deadline by default (`CONFLUENCE_WEB_QUERY_TIMEOUT_SECONDS`, up to `CONFLUENCE_WEB_MAX_DEADLINE_SECONDS`, default 900 s) plus 10 s cleanup, prompt up to 16 MiB, answer up to 128 MiB, 100 Confluence calls and 50 model calls per query by default (`CONFLUENCE_WEB_MAX_CONFLUENCE_CALLS`, `CONFLUENCE_WEB_MAX_MODEL_CALLS`), container limited to 1 GiB RAM, 1 CPU, 128 processes, no network.
|
|
- **Model tokens**: `CONFLUENCE_WEB_MODEL_CONTEXT_WINDOW_TOKENS` and `..._MAX_OUTPUT_TOKENS` describe the provider; they are independent of the byte limits above. Large pages or answers can exceed the model's context before the application limits; the UI then shows `model_context_exceeded` or `model_output_limit`.
|
|
|
|
## Debugging a failed query
|
|
|
|
`execution_failed` in the browser is deliberately vague; the backend log has the
|
|
reason. Run `make run` in a terminal you can read (or redirect it to a file) and
|
|
look for the two lines a failed query emits: the agent's terminal code with the
|
|
call counters, and `[agent] run failed code=... reason=...` from the container's
|
|
stderr. Common causes: the per-query model call budget running out on a long
|
|
multi-step research prompt (`CONFLUENCE_WEB_MAX_MODEL_CALLS`), and a model ending
|
|
its final turn with no text. Neither is a timeout, so raising the deadline does
|
|
not help either one.
|
|
|
|
## Checks
|
|
|
|
| Command | What it runs | Needs |
|
|
| --- | --- | --- |
|
|
| `make check` | Crawler + backend + integration (no Docker) pytest, agent host tests, frontend unit/contract tests | nothing external |
|
|
| `make check-docker` | Backend real-Docker checks, backend + real runtime image + scripted peers, real provider adapter + runtime, runtime image/isolation/boundary checks | rootless Docker, built images |
|
|
| `make check-browser` | Real Chrome against the real backend (scripted runtime), frontend's own e2e suite | Chrome on `127.0.0.1:9444`, Node |
|
|
|
|
Each subsystem's own commands are documented in its guide and can be run independently:
|
|
|
|
```bash
|
|
CONFLUENCE_PAT=synthetic-token-no-network CONFLUENCE_URL=https://approved.example.com .venv/bin/python -m pytest tests/backend
|
|
npm --prefix agent test && npm --prefix agent run proof
|
|
npm --prefix frontend test && npm --prefix frontend run test:e2e
|
|
```
|
|
|
|
Fake modes (`CONFLUENCE_WEB_DEV_MODE`, the backend fake image, the frontend mock
|
|
server, the runtime's fake backend peer) are explicit opt-ins and never activate
|
|
on a production failure.
|
|
|
|
## Standalone crawler
|
|
|
|
Crawls Confluence Data Center pages into Markdown files using a PAT from `.env`
|
|
(`CONFLUENCE_URL`, `CONFLUENCE_PAT`; see `.env.example`). PATs require
|
|
Confluence Data Center / Server 7.9+.
|
|
|
|
```bash
|
|
python main.py --space KEY --out output/KEY # one space
|
|
python main.py --all --out output # all visible spaces
|
|
pytest -m live # live crawler tests (uses .env)
|
|
```
|
|
|
|
Each page becomes `output/<space>/<page_id>_<slug>.md` with YAML front matter.
|
|
The web backend imports only the crawler's pure conversion functions; it never
|
|
reads `.env`.
|