score-back/API.md

# JSON-RPC API Documentation

The Cloud Point RPC server implements the **JSON-RPC 2.0** protocol over TCP.

## General Format

All requests and responses are JSON objects.

### Request
```json
{
  "jsonrpc": "2.0",
  "method": "<method_name>",
  "params": {},
  "id": <integer|string>
}
```
*`params` is currently ignored by all handlers but is valid per JSON-RPC 2.0.*

### Response (Success)
```json
{
  "jsonrpc": "2.0",
  "result": <method_specific_result>,
  "id": <matching_request_id>
}
```

### Response (Error)
```json
{
  "jsonrpc": "2.0",
  "error": {
    "code": <integer>,
    "message": "<string>"
  },
  "id": <matching_request_id>
}
```

---

## Conventions

- **Matrices:** Row-major storage. A 3×3 matrix `M` with rows `[r0, r1, r2]` serialises as a flat 9-element JSON array `[r0[0], r0[1], r0[2], r1[0], ...]`.
- **Units:** Translation in **metres**. No pixel units unless stated.
- **Extrinsics convention (OpenCV):** `x_right = R · x_left + T`. For a parallel rig with baseline `b`, `R = I` and `T = [-b, 0, 0]` (e.g. `[-0.06, 0, 0]` for a 6 cm baseline).
- **Image layout:** Top-left origin, row-major, packed channels (BGR order unless otherwise noted). Unity must flip GPU readback vertically before encoding.
- **Image data encoding:** Raw pixel bytes encoded as **Base64** (standard alphabet, no line breaks). The `type` field indicates channel layout.
- **Calibration arrays:** Plain JSON double arrays — **not** base64. Only image pixel data uses base64.

---

## Methods Served by the Unity Side

These methods are implemented server-side (in Unity, or in the C++ test mock in `src/server_main.cpp`).

---

### `get-available-methods`

Returns the list of method names registered on this server instance.

**Request:**
```json
{ "jsonrpc": "2.0", "method": "get-available-methods", "id": 0 }
```

**Response `result`:** `["method-name-1", "method-name-2", ...]`

---

### `get-stereo-calibration`

Returns full stereo rig calibration: intrinsics for both cameras, stereo rotation and translation.

**Request:**
```json
{ "jsonrpc": "2.0", "method": "get-stereo-calibration", "id": 1 }
```

**Response `result`:**
```json
{
  "left": {
    "camera_matrix": [fx, 0, cx,  0, fy, cy,  0, 0, 1],
    "dist_coeffs":   [k1, k2, p1, p2, k3]
  },
  "right": {
    "camera_matrix": [fx, 0, cx,  0, fy, cy,  0, 0, 1],
    "dist_coeffs":   [k1, k2, p1, p2, k3]
  },
  "rotation":    [r00, r01, r02,  r10, r11, r12,  r20, r21, r22],
  "translation": [tx, ty, tz],
  "image_size":  { "width": 640, "height": 480 }
}
```

Field details:

| Field | Type | Size | Description |
|-------|------|------|-------------|
| `camera_matrix` | `double[]` | 9 | Row-major 3×3 intrinsic matrix: `[fx, 0, cx, 0, fy, cy, 0, 0, 1]` |
| `dist_coeffs` | `double[]` | 5 | Radial/tangential coefficients `[k1, k2, p1, p2, k3]` |
| `rotation` | `double[]` | 9 | Row-major rotation matrix R (left-to-right frame, OpenCV convention) |
| `translation` | `double[]` | 3 | Translation vector in metres |
| `image_size` | object | — | Sensor resolution before any rectification |

**Test mock defaults:** fx=fy=800, cx=320, cy=240, zero distortion, R=identity, T=[-0.06, 0, 0], 640×480.

---

### `get-image-pair`

Returns a synchronised stereo frame as two base64-encoded images.

**Request:**
```json
{ "jsonrpc": "2.0", "method": "get-image-pair", "id": 2 }
```

**Response `result`:**
```json
{
  "frame": 42,
  "left":  { "width": 640, "height": 480, "type": "BGR", "data": "<base64>" },
  "right": { "width": 640, "height": 480, "type": "BGR", "data": "<base64>" }
}
```

Field details:

| Field | Description |
|-------|-------------|
| `frame` | Monotonically increasing counter per server instance; wraps at `uint64_t` max. |
| `type` | Channel layout: `"BGR"` (3 ch), `"RGBA"` (4 ch), or `"DEPTH"` (1 ch float32). |
| `data` | Base64-encoded raw pixel bytes. Size = `width × height × channels`. |

Unity must supply images in top-left-origin row-major order; flip GPU readback vertically before encoding.

---

### `get-intrinsic-params` *(legacy)*

Returns left-camera intrinsic matrix as a flat 9-element double array.

**Request:**
```json
{ "jsonrpc": "2.0", "method": "get-intrinsic-params", "id": 3 }
```

**Response `result`:** `[fx, 0, cx, 0, fy, cy, 0, 0, 1]`

9 plain JSON doubles, row-major 3×3. **Not base64.**

---

### `get-extrinsic-params` *(legacy)*

Returns a flat 16-element double array (4×4 row-major extrinsic matrix).

**Request:**
```json
{ "jsonrpc": "2.0", "method": "get-extrinsic-params", "id": 4 }
```

**Response `result`:** `[r00, r01, r02, tx, r10, r11, r12, ty, r20, r21, r22, tz, 0, 0, 0, 1]`

16 plain JSON doubles. **Not base64.**

---

## Client-Side Outputs

These are computed locally by the C++ client (`RpcClient`) from data received via the methods above. They are **not** JSON-RPC methods callable on the server.

---

### `get-cloud-point` *(computed by client — spec in progress)*

Reconstructs a dense 3-D point cloud from the rectified stereo pair. The `RpcClient::get_cloud_point()` method will call `get-stereo-calibration` and `get-image-pair`, run stereo rectification and disparity computation locally (OpenCV SGBM or CUDA stereo), and reproject to 3-D.

**Expected future result shape** (subject to change in Phase 2):
```json
{
  "width":  <int>,
  "height": <int>,
  "data":   "<base64-encoded float32 XYZ triplets, row-major>"
}
```

`data` will encode `width × height × 3` little-endian `float32` values (x, y, z in metres per pixel, `NaN` for invalid/occluded depth). Full specification and encoding details are deferred to Phase 2.