score-back/README.md

# Cloud Point RPC

Communication JSON RPC protocol and implementation with Unity Scene.

## Project Structure

- `include/`: Header files for the RPC server, TCP server, and C-API.
- `src/`: Implementation of the RPC logic, networking, and C-API.
- `src/cloud_point/`: OpenCV-based image processing and rectification logic.
- `docs/`: Documentation diagrams and models.
- `subprojects/`: Dependencies managed by Meson.

## Status

- [x] Server implementation with C-API for Unity
- [x] OpenCV stereo client (StereoRectifier, PointCloudBuilder, CloudPointClient facade)
- [ ] Unity-side C# implementation per [docs/unity-integration.md](docs/unity-integration.md)

## API Documentation

See [API.md](API.md) for detailed request/response formats.

## Pipeline

Unity acts as a data source: it serves stereo image pairs (`get-image-pair`) and full stereo calibration (`get-stereo-calibration`) over JSON-RPC 2.0. The C++ `CloudPointClient` calls `connect()` once to fetch calibration, then on each `compute_cloud()` call it fetches a synchronised image pair, runs stereo rectification (`StereoRectifier`, cv::stereoRectify + remap), computes disparity with SGBM (16× scaling), reprojects to 3-D with `cv::reprojectImageTo3D` (`PointCloudBuilder`), filters NaN/invalid points, and returns a `PointCloud`. An optional `write_ply()` helper serialises the result to disk.

See [API.md](API.md) for wire schemas and [docs/unity-integration.md](docs/unity-integration.md) for the Unity C# design spec.

## Development

The project uses **Meson** build system and **C++23**.

### Dependencies

- Meson (>= 1.1.0), Ninja
- GCC/Clang (C++23 support)
- Git (for subprojects)
- OpenCV 4 (optional; required for stereo point cloud compute)

The following dependencies are managed via Meson subprojects:
- [ASIO](https://think-async.com/Asio/) (Networking)
- [nlohmann/json](https://github.com/nlohmann/json) (JSON serialization)
- [yaml-cpp](https://github.com/jbeder/yaml-cpp) (Configuration loading)
- [glog](https://github.com/google/glog) (Logging)
- [jsonrpccxx](https://github.com/uS-S/jsonrpccxx) (JSON-RPC 2.0 implementation)

### Build & Run

```bash
meson setup build
meson compile -C build
./build/src/cloud_point_rpc_server config.yaml
```

*Note: You need a `config.yaml` file. See `config.yaml.example` for the required format.*

Run the interactive CLI client:

```bash
./build/src/cloud_point_rpc_cli config.yaml
```

CLI menu options (OpenCV options are hidden when built without opencv4):

| Option | Action |
|--------|--------|
| 1 | List available RPC methods |
| 2 | Get intrinsic params (legacy) |
| 3 | Get extrinsic params (legacy) |
| 4 | Compute point cloud — prints point count and bounding box |
| 5 | Compute point cloud and save to `output.ply` |
| 0 | Exit |

#### Build on windows

It's assumed that you have `GCC` and `make`/`ninja` installed on your system (and available in `PATH`)

```powershell
## FIRST OF ALL!
git submodule init
git submodule update
# Next python:
python3 -m venv .\venv
.\venv\Scripts\Activate.ps1
# or
.\venv\bin\Activate.ps1
pip install meson cmake
meson setup -Ddefault_library=static build
meson compile -C build
# To correctly get dlls:
meson devenv -C build
## .\build\tests\unit_tests < for dummy test
## .\build\src\.. < produced execs and libs
```

### Testing

```bash
meson test -C build -v
```

## Docker

You can build and run the cli using `Docker`.

### 1. Build Image

```bash
docker build -t cloud-point-rpc .
```

### 2. Run Container

The cli will try to connect to a **running server** on ip and port defined in config.yml file. (defined in `config.yaml` inside the image).
For simplicity, it's better to use a host network, so you will not have any headache with accessability.

> _Server is not configured to run through container, if you need, contact me_

You also can mount your own `config.yaml` to override the default settings:

```bash
docker run --network=host -it -v $(pwd)/my_config.yaml:/app/config.yaml cloud-point-rpc
```

## Validation with SCARED Dataset

The `scared_dataset_server` executable lets you validate the stereo point-cloud
pipeline against real endoscopic images from the
[SCARED dataset](https://huggingface.co/datasets/maxhallan7/scared).

### Obtaining the data

1. Download `test_dataset_8.zip` from
   <https://huggingface.co/datasets/maxhallan7/scared>.
2. Extract so that `keyframe_0/` through `keyframe_4/` exist under
   `test_dataset_8/`.

Each keyframe directory contains:
- `Left_Image.png`, `Right_Image.png` — 1280×1024 unrectified RGBA images.
- `endoscope_calibration.yaml` — OpenCV FileStorage with `M1`, `D1`, `M2`,
  `D2`, `R`, `T` nodes.

**Note:** `T` is stored in **millimetres** in the YAML file (baseline ≈ −4.35 mm).
`scared_dataset_server` divides `T` by 1000 before placing it on the wire
(the wire protocol uses metres).

### Running the server

```bash
./build/src/cloud_point/scared_dataset_server \
    /path/to/test_dataset_8/keyframe_0 8080
```

### Connecting with the CLI

In a second terminal run the interactive CLI against the same host and port:

```bash
# Adjust ip/port in config.yaml if needed, then:
./build/src/cloud_point_rpc_cli config.yaml
# Option 4 — compute point cloud and print valid point count
# Option 5 — compute point cloud and save to output.ply (inspect in MeshLab)
```

The SCARED test set contains no ground-truth depth, so validation is
qualitative (inspect the PLY in MeshLab or similar).

**Disparity range caveat:** the CLI constructs `CloudPointClient` with the
default of 128 disparity levels, while this rig (fx ≈ 1024 px, baseline
≈ 4.35 mm) produces disparities above 128 px for tissue nearer than
~35 mm — those pixels silently drop out of the cloud. The E2E test
(`tests/test_scared_dataset.cpp`) passes `num_disparities = 160` for full
coverage; programmatic consumers should do the same via the
`CloudPointClient` constructor. For the CLI's qualitative check the default
is fine (observed median depth ≈ 115 mm is well within range).

## Communication model

![Communicatoin model plantuml diagram](docs/cm.png)