confluence_web/README.md

# Confluence Research Web UI

A minimalist web UI that researches a Confluence Data Center instance with a
[pi](https://github.com/earendil-works/pi) agent running in a fresh, network-less
rootless Docker container per query, and returns a cited Markdown answer plus
optional files the agent produced. Each component documents its own interface
in `agent/README.md`, `backend/README.md` and `frontend/README.md`.

The repository also contains the original standalone
[Confluence crawler](#standalone-crawler), which is unchanged.

## Components

| Directory | Role | Developer guide |
| --- | --- | --- |
| `backend/` | FastAPI service: credentials in memory, Confluence/model upstreams, authoritative tool history, container lifecycle, artifact downloads, serves the frontend | [backend/README.md](backend/README.md) |
| `agent/` | Container image: pinned pi SDK bridge, Python supervisor, local tools, artifact exporter | [agent/README.md](agent/README.md) |
| `frontend/` | Static UI (vendored marked, DOMPurify and thinking-orbs engine), same-origin mock server for UI development | [frontend/README.md](frontend/README.md) |
| `tests/integration/` | Integration-stage tests connecting the real components | this file |
| `deploy/`, `scripts/`, `Makefile` | Deployment configuration and entry points | this file |

## Requirements

- Linux with a **rootless** Docker daemon (cgroup v2, memory/CPU/PID limits enforced). `docker info` must list `rootless` under Security Options.
- Python 3.12+ in `.venv` (verified with 3.14.7), Node 22+ (verified with 26.7) and npm.
- An OpenAI-compatible Chat Completions endpoint the backend can reach (key held by the backend only).
- Network access from the backend host to the approved Confluence origin, optionally through a SOCKS5/HTTP proxy.
- For browser checks: Google Chrome with `--remote-debugging-port=9444`.

## Setup

```bash
python3 -m venv .venv && .venv/bin/python -m pip install -r requirements.txt
make install                 # also installs agent/ and frontend/ dev dependencies
make build-image             # builds confluence-pi-agent:rev1 and prints its image ID
cp deploy/confluence-web.env.example deploy/confluence-web.env   # then edit
```

`deploy/confluence-web.env` is git-ignored. Every variable is documented in the
example file and in [backend/README.md](backend/README.md). Pin
`CONFLUENCE_WEB_RUNTIME_IMAGE` to the image ID printed by `make build-image` for
an exact runtime, set a deployment-unique `CONFLUENCE_WEB_CONTAINER_LABEL_VALUE`,
and list only approved Confluence origins.

## Run

```bash
make run                     # scripts/run-backend.sh deploy/confluence-web.env
```

The backend binds `127.0.0.1:8000` by default and serves the UI at
`http://127.0.0.1:8000/`. Open it in a browser, click the key icon, enter the
Confluence base URL **including its context path** (for example
`https://collab.lge.com/main`, which must match an approved origin), your
personal access token (PAT), test the connection, then ask a question. On
instances that allow anonymous REST reads, "Test connection" proves the
destination is reachable but cannot prove the PAT is valid; a wrong token then
shows up as denied or missing pages. Credentials live only in browser memory and in the backend
for the duration of a request; a page reload clears them.

Version 1 is a small-team deployment: one backend worker, one query at a time
(a second query gets `busy`), loopback by default. The application has no login
of its own; anyone who can reach the port can run queries with their own PAT.
Never bind it to a network interface over plain HTTP, because every request
carries the user's PAT. If you put a reverse proxy in front, it must accept
`6*16 MiB + 64 KiB` request bodies, pass client aborts through promptly, avoid
buffering bodies to disk, and allow at least 200 s per request.

### Serving colleagues on an internal network (IP only, TLS)

Uvicorn terminates TLS itself; no reverse proxy or DNS name is needed. The
certificate comes from a private CA that is **name-constrained** to the listed
IP addresses, so colleagues who import it are trusting it for those addresses
only. A certificate issued by the same CA for any hostname or any other address
is rejected by browsers and by OpenSSL (`permitted subtree violation`).

```bash
make tls IP=172.26.10.20          # creates deploy/tls/{ca,server}.{key,crt}; git-ignored
```

Then in `deploy/confluence-web.env`:

```
CONFLUENCE_WEB_BIND_HOST=172.26.10.20
CONFLUENCE_WEB_TLS_CERT=./deploy/tls/server.crt
CONFLUENCE_WEB_TLS_KEY=./deploy/tls/server.key
```

Restart with `make run` and open `https://172.26.10.20:8000/`. Under HTTPS the
session cookie gains the `Secure` flag automatically and the same-origin check
expects `https://` origins.

Hand colleagues only `deploy/tls/ca.crt`, imported once as a trusted root
(Windows: double-click, "Trusted Root Certification Authorities"; macOS:
Keychain Access; Chrome on Linux: chrome://settings/certificates; Firefox:
Settings, Certificates, Authorities). They can read the constraint themselves in
the certificate details ("Name Constraints: Permitted IP ..."). Keep `ca.key`
and `server.key` on the server host. The server certificate is valid for 825
days (the browser maximum); rotate it with the same CA by deleting
`deploy/tls/server.*` and re-running `make tls` with the CA files kept, or
rotate everything by deleting the directory.

Network-free UI development against explicit fakes (no Docker, PAT `dev-pat`,
URL `https://approved.example.com`):

```bash
make run-dev
```

## Credentials, expiry and limits

- **Confluence PAT**: entered in the browser per session, sent only to the backend over the same origin, never stored, logged, or passed to the container. The token inherits your Confluence permissions.
- **Model key**: backend environment only (`CONFLUENCE_WEB_MODEL_API_KEY`). For a local llama.cpp server any placeholder works.
- **Session cookie** `cw_session`: HttpOnly, SameSite=Strict, marks artifact ownership only. It is not authentication.
- **Artifacts**: at most 20 files, 10 MiB each, 50 MiB per query, 500 MiB total; downloadable for 15 minutes after a successful query, then deleted. Failed or cancelled queries keep nothing.
- **Query**: 180 s total deadline plus 10 s cleanup, prompt up to 16 MiB, answer up to 128 MiB, 100 Confluence calls and 50 model calls per query, container limited to 1 GiB RAM, 1 CPU, 128 processes, no network.
- **Model tokens**: `CONFLUENCE_WEB_MODEL_CONTEXT_WINDOW_TOKENS` and `..._MAX_OUTPUT_TOKENS` describe the provider; they are independent of the byte limits above. Large pages or answers can exceed the model's context before the application limits; the UI then shows `model_context_exceeded` or `model_output_limit`.

## Checks

| Command | What it runs | Needs |
| --- | --- | --- |
| `make check` | Crawler + backend + integration (no Docker) pytest, agent host tests, frontend unit/contract tests | nothing external |
| `make check-docker` | Backend real-Docker checks, backend + real runtime image + scripted peers, real provider adapter + runtime, runtime image/isolation/boundary checks | rootless Docker, built images |
| `make check-browser` | Real Chrome against the real backend (scripted runtime), frontend's own e2e suite | Chrome on `127.0.0.1:9444`, Node |

Each subsystem's own commands are documented in its guide and can be run independently:

```bash
CONFLUENCE_PAT=synthetic-token-no-network CONFLUENCE_URL=https://approved.example.com .venv/bin/python -m pytest tests/backend
npm --prefix agent test && npm --prefix agent run proof
npm --prefix frontend test && npm --prefix frontend run test:e2e
```

Fake modes (`CONFLUENCE_WEB_DEV_MODE`, the backend fake image, the frontend mock
server, the runtime's fake backend peer) are explicit opt-ins and never activate
on a production failure.

## Standalone crawler

Crawls Confluence Data Center pages into Markdown files using a PAT from `.env`
(`CONFLUENCE_URL`, `CONFLUENCE_PAT`; see `.env.example`). PATs require
Confluence Data Center / Server 7.9+.

```bash
python main.py --space KEY --out output/KEY   # one space
python main.py --all --out output             # all visible spaces
pytest -m live                                # live crawler tests (uses .env)
```

Each page becomes `output/<space>/<page_id>_<slug>.md` with YAML front matter.
The web backend imports only the crawler's pure conversion functions; it never
reads `.env`.