Cloud Point RPC

Communication JSON RPC protocol and implementation with Unity Scene.

Project Structure

• include/: Header files for the RPC server, TCP server, and C-API.
• src/: Implementation of the RPC logic, networking, and C-API.
• src/cloud_point/: OpenCV-based image processing and rectification logic.
• docs/: Documentation diagrams and models.
• subprojects/: Dependencies managed by Meson.

Status

• Server implementation with C-API for Unity
• OpenCV stereo client (StereoRectifier, PointCloudBuilder, CloudPointClient facade)
• Unity-side C# implementation per docs/unity-integration.md

API Documentation

See API.md for detailed request/response formats.

Pipeline

Unity acts as a data source: it serves stereo image pairs (get-image-pair) and full stereo calibration (get-stereo-calibration) over JSON-RPC 2.0. The C++ CloudPointClient calls connect() once to fetch calibration, then on each compute_cloud() call it fetches a synchronised image pair, runs stereo rectification (StereoRectifier, cv::stereoRectify + remap), computes disparity with SGBM (16× scaling), reprojects to 3-D with cv::reprojectImageTo3D (PointCloudBuilder), filters NaN/invalid points, and returns a PointCloud. An optional write_ply() helper serialises the result to disk.

See API.md for wire schemas and docs/unity-integration.md for the Unity C# design spec.

Development

The project uses Meson build system and C++23.

Dependencies

• Meson (>= 1.1.0), Ninja
• GCC/Clang (C++23 support)
• Git (for subprojects)
• OpenCV 4 (optional; required for stereo point cloud compute)

The following dependencies are managed via Meson subprojects:

• ASIO (Networking)
• nlohmann/json (JSON serialization)
• yaml-cpp (Configuration loading)
• glog (Logging)
• jsonrpccxx (JSON-RPC 2.0 implementation)

Build & Run

meson setup build
meson compile -C build
./build/src/cloud_point_rpc_server config.yaml

Note: You need a config.yaml file. See config.yaml.example for the required format.

Run the interactive CLI client:

./build/src/cloud_point_rpc_cli config.yaml

CLI menu options (OpenCV options are hidden when built without opencv4):

Option	Action
1	List available RPC methods
2	Get intrinsic params (legacy)
3	Get extrinsic params (legacy)
4	Compute point cloud — prints point count and bounding box
5	Compute point cloud and save to output.ply
0	Exit

Build on windows

It's assumed that you have GCC and make/ninja installed on your system (and available in PATH)

## FIRST OF ALL!
git submodule init
git submodule update
# Next python:
python3 -m venv .\venv
.\venv\Scripts\Activate.ps1
# or
.\venv\bin\Activate.ps1
pip install meson cmake
meson setup -Ddefault_library=static build
meson compile -C build
# To correctly get dlls:
meson devenv -C build
## .\build\tests\unit_tests < for dummy test
## .\build\src\.. < produced execs and libs

Testing

meson test -C build -v

Docker

You can build and run the cli using Docker.

1. Build Image

docker build -t cloud-point-rpc .

2. Run Container

The cli will try to connect to a running server on ip and port defined in config.yml file. (defined in config.yaml inside the image). For simplicity, it's better to use a host network, so you will not have any headache with accessability.

Server is not configured to run through container, if you need, contact me

You also can mount your own config.yaml to override the default settings:

docker run --network=host -it -v $(pwd)/my_config.yaml:/app/config.yaml cloud-point-rpc

Validation with SCARED Dataset

The scared_dataset_server executable lets you validate the stereo point-cloud pipeline against real endoscopic images from the SCARED dataset.

Obtaining the data

1. Download test_dataset_8.zip from https://huggingface.co/datasets/maxhallan7/scared.
2. Extract so that keyframe_0/ through keyframe_4/ exist under test_dataset_8/.

Each keyframe directory contains:

• Left_Image.png, Right_Image.png — 1280×1024 unrectified RGBA images.
• endoscope_calibration.yaml — OpenCV FileStorage with M1, D1, M2, D2, R, T nodes.

Note: T is stored in millimetres in the YAML file (baseline ≈ −4.35 mm). scared_dataset_server divides T by 1000 before placing it on the wire (the wire protocol uses metres).

Running the server

./build/src/cloud_point/scared_dataset_server \
    /path/to/test_dataset_8/keyframe_0 8080

Connecting with the CLI

In a second terminal run the interactive CLI against the same host and port:

# Adjust ip/port in config.yaml if needed, then:
./build/src/cloud_point_rpc_cli config.yaml
# Option 4 — compute point cloud and print valid point count
# Option 5 — compute point cloud and save to output.ply (inspect in MeshLab)

The SCARED test set contains no ground-truth depth, so validation is qualitative (inspect the PLY in MeshLab or similar).

Disparity range caveat: the CLI constructs CloudPointClient with the default of 128 disparity levels, while this rig (fx ≈ 1024 px, baseline ≈ 4.35 mm) produces disparities above 128 px for tissue nearer than ~35 mm — those pixels silently drop out of the cloud. The E2E test (tests/test_scared_dataset.cpp) passes num_disparities = 160 for full coverage; programmatic consumers should do the same via the CloudPointClient constructor. For the CLI's qualitative check the default is fine (observed median depth ≈ 115 mm is well within range).

Communication model