# Cloud Point RPC

Communication JSON RPC protocol and implementation with Unity Scene.

## Project Structure

- `include/`: Header files for the RPC server, TCP server, and C-API.
- `src/`: Implementation of the RPC logic, networking, and C-API.
- `src/cloud_point/`: OpenCV-based image processing and rectification logic.
- `docs/`: Documentation diagrams and models.
- `subprojects/`: Dependencies managed by Meson.

## Status

- [x] Server implementation with C-API for Unity
- [x] OpenCV stereo client (StereoRectifier, PointCloudBuilder, CloudPointClient facade)
- [ ] Unity-side C# implementation per [docs/unity-integration.md](docs/unity-integration.md)

## API Documentation

See [API.md](API.md) for detailed request/response formats.

## Pipeline

Unity acts as a data source: it serves stereo image pairs (`get-image-pair`) and full stereo calibration (`get-stereo-calibration`) over JSON-RPC 2.0. The C++ `CloudPointClient` calls `connect()` once to fetch calibration, then on each `compute_cloud()` call it fetches a synchronised image pair, runs stereo rectification (`StereoRectifier`, cv::stereoRectify + remap), computes disparity with SGBM (16× scaling), reprojects to 3-D with `cv::reprojectImageTo3D` (`PointCloudBuilder`), filters NaN/invalid points, and returns a `PointCloud`. An optional `write_ply()` helper serialises the result to disk.

See [API.md](API.md) for wire schemas and [docs/unity-integration.md](docs/unity-integration.md) for the Unity C# design spec.

## Development

The project uses **Meson** build system and **C++23**.

### Dependencies

- Meson (>= 1.1.0), Ninja
- GCC/Clang (C++23 support)
- Git (for subprojects)
- OpenCV 4 (optional; required for stereo point cloud compute)

The following dependencies are managed via Meson subprojects:
- [ASIO](https://think-async.com/Asio/) (Networking)
- [nlohmann/json](https://github.com/nlohmann/json) (JSON serialization)
- [yaml-cpp](https://github.com/jbeder/yaml-cpp) (Configuration loading)
- [glog](https://github.com/google/glog) (Logging)
- [jsonrpccxx](https://github.com/uS-S/jsonrpccxx) (JSON-RPC 2.0 implementation)

### Build & Run

```bash
meson setup build
meson compile -C build
./build/src/cloud_point_rpc_server config.yaml
```

*Note: You need a `config.yaml` file. See `config.yaml.example` for the required format.*

Run the interactive CLI client:

```bash
./build/src/cloud_point_rpc_cli config.yaml
```

CLI menu options (OpenCV options are hidden when built without opencv4):

| Option | Action |
|--------|--------|
| 1 | List available RPC methods |
| 2 | Get intrinsic params (legacy) |
| 3 | Get extrinsic params (legacy) |
| 4 | Compute point cloud — prints point count and bounding box |
| 5 | Compute point cloud and save to `output.ply` |
| 0 | Exit |

#### Build on windows

It's assumed that you have `GCC` and `make`/`ninja` installed on your system (and available in `PATH`)

```powershell
## FIRST OF ALL!
git submodule init
git submodule update
# Next python:
python3 -m venv .\venv
.\venv\Scripts\Activate.ps1
# or
.\venv\bin\Activate.ps1
pip install meson cmake
meson setup -Ddefault_library=static build
meson compile -C build
# To correctly get dlls:
meson devenv -C build
## .\build\tests\unit_tests < for dummy test
## .\build\src\.. < produced execs and libs
```

### Testing

```bash
meson test -C build -v
```

## Docker

You can build and run the cli using `Docker`.

### 1. Build Image

```bash
docker build -t cloud-point-rpc .
```

### 2. Run Container

The cli will try to connect to a **running server** on ip and port defined in config.yml file. (defined in `config.yaml` inside the image).
For simplicity, it's better to use a host network, so you will not have any headache with accessability.

> _Server is not configured to run through container, if you need, contact me_

You also can mount your own `config.yaml` to override the default settings:

```bash
docker run --network=host -it -v $(pwd)/my_config.yaml:/app/config.yaml cloud-point-rpc
```

## Validation with SCARED Dataset

The `scared_dataset_server` executable lets you validate the stereo point-cloud
pipeline against real endoscopic images from the
[SCARED dataset](https://huggingface.co/datasets/maxhallan7/scared).

### Obtaining the data

1. Download `test_dataset_8.zip` from
   <https://huggingface.co/datasets/maxhallan7/scared>.
2. Extract so that `keyframe_0/` through `keyframe_4/` exist under
   `test_dataset_8/`.

Each keyframe directory contains:
- `Left_Image.png`, `Right_Image.png` — 1280×1024 unrectified RGBA images.
- `endoscope_calibration.yaml` — OpenCV FileStorage with `M1`, `D1`, `M2`,
  `D2`, `R`, `T` nodes.

**Note:** `T` is stored in **millimetres** in the YAML file (baseline ≈ −4.35 mm).
`scared_dataset_server` divides `T` by 1000 before placing it on the wire
(the wire protocol uses metres).

### Running the server

```bash
./build/src/cloud_point/scared_dataset_server \
    /path/to/test_dataset_8/keyframe_0 8080
```

If port 8080 is already taken on your machine (Docker's `rootlesskit`
commonly holds it) pass another port and update `server.port` in the CLI
config accordingly. Connecting the CLI to a foreign service on 8080 shows up
as `invalid JSON response from server` / `std::bad_alloc` errors.

### Connecting with the CLI

In a second terminal run the interactive CLI with the SCARED-tuned config:

```bash
./build/src/cloud_point_rpc_cli config.scared.yml
# Option 4 — compute point cloud and print valid point count + bounding box
# Option 5 — compute point cloud and save a triangulated PLY mesh
```

`config.scared.yml` sets the optional `cloud_point` section that options 4/5
honour:

```yaml
cloud_point:
  algorithm: cpu        # "gpu" falls back to CPU when CUDA is unavailable
  num_disparities: 160  # fx ~1024 px, baseline ~4.35 mm -> up to ~160 px
  min_depth_m: 0.02     # endoscopic working range: 20 mm .. 300 mm
  max_depth_m: 0.30
```

Depth limits are a physical bound on the scene: with this rig depth is
roughly `4.45 m / disparity_px`, so any mismatch with a disparity below
~15 px reprojects metres away. Without the section the CLI falls back to the
generic defaults (GPU, 128 disparities, 0.01–10 m). The SGBM matcher itself
is configured with OpenCV's recommended smoothness penalties (P1 = 8·bs²,
P2 = 32·bs²), a 5×5 block, uniqueness ratio 10, speckle filtering and a
left-right consistency check; see `CpuStereoMatcher::Params`.

### Checking the result

Option 4 should report a bounding box with z inside roughly
`[0.03, 0.16]` m for `test_dataset_8` keyframes. Option 5 writes a binary
little-endian PLY containing every valid point **and** a mesh triangulated
from the pixel grid (triangles are dropped where the longest edge exceeds 5 % of the local
depth, so the mesh breaks at occlusions). The mesh is what makes web viewers
usable: viewers such as Meshy's online PLY viewer fabricate a triangle from
every three consecutive vertices of a vertex-only PLY, which draws long
slivers across the surface and makes a correct cloud look like a fan of
rays. Pass `PlyOptions{false, 0.0f, false}` to `write_ply` for a points-only ASCII
file.

When the keyframe contains `point_cloud.obj`, run the benchmark to compare the
reconstruction with its pixel-aligned XYZ ground truth:

```bash
./build/src/cloud_point/scared_dataset_benchmark \
    /path/to/test_dataset_8/keyframe_0 160
```

The benchmark reports coverage, component-wise and 3-D errors, threshold
accuracy, and matching/reconstruction timings as JSON. OBJ coordinates are
converted from millimetres to metres and rectified into the same left-camera
frame as the reconstructed cloud before evaluation. Reference numbers for
`dataset_1/keyframe_1` with the tuned SGBM configuration: coverage ≈ 0.85,
MAE₃D ≈ 0.8 mm, RMSE₃D ≈ 1.4 mm, 80 % of points within 2 mm (the previous
unregularised SGBM gave MAE₃D ≈ 44 mm and RMSE₃D ≈ 289 mm).

The E2E test (`tests/test_scared_dataset.cpp`) exercises the same pipeline
with `num_disparities = 160` and asserts >50 000 valid points and a median
depth in `[0.02, 0.20]` m.

## Communication model

![Communicatoin model plantuml diagram](docs/cm.png)