Artur Mukhamadiev
14ae0a901f
All checks were successful
Verification / Is-Buildable (push) Successful in 2m43s
- report valid coverage and depth percentiles even when XYZ ground truth is absent - accept depth bounds and optional color-mapped PNG output for visual inspection - retain accuracy metrics when point_cloud.obj is available - add a batch helper that renders Markdown tables and optional JSON and PNG artifacts - document benchmark inputs, dataset limits, and visualization tradeoffs
258 lines
10 KiB
Markdown
258 lines
10 KiB
Markdown
# Cloud Point RPC
|
||
|
||
Communication JSON RPC protocol and implementation with Unity Scene.
|
||
|
||
## Project Structure
|
||
|
||
- `include/`: Header files for the RPC server, TCP server, and C-API.
|
||
- `src/`: Implementation of the RPC logic, networking, and C-API.
|
||
- `src/cloud_point/`: OpenCV-based image processing and rectification logic.
|
||
- `docs/`: Documentation diagrams and models.
|
||
- `subprojects/`: Dependencies managed by Meson.
|
||
|
||
## Status
|
||
|
||
- [x] Server implementation with C-API for Unity
|
||
- [x] OpenCV stereo client (StereoRectifier, PointCloudBuilder, CloudPointClient facade)
|
||
- [ ] Unity-side C# implementation per [docs/unity-integration.md](docs/unity-integration.md)
|
||
|
||
## API Documentation
|
||
|
||
See [API.md](API.md) for detailed request/response formats.
|
||
|
||
## Pipeline
|
||
|
||
Unity acts as a data source: it serves stereo image pairs (`get-image-pair`) and full stereo calibration (`get-stereo-calibration`) over JSON-RPC 2.0. The C++ `CloudPointClient` calls `connect()` once to fetch calibration, then on each `compute_cloud()` call it fetches a synchronised image pair, runs stereo rectification (`StereoRectifier`, cv::stereoRectify + remap), computes disparity with SGBM (16× scaling), reprojects to 3-D with `cv::reprojectImageTo3D` (`PointCloudBuilder`), filters NaN/invalid points, and returns a `PointCloud`. An optional `write_ply()` helper serialises the result to disk.
|
||
|
||
See [API.md](API.md) for wire schemas and [docs/unity-integration.md](docs/unity-integration.md) for the Unity C# design spec.
|
||
|
||
## Development
|
||
|
||
The project uses **Meson** build system and **C++23**.
|
||
|
||
### Dependencies
|
||
|
||
- Meson (>= 1.1.0), Ninja
|
||
- GCC/Clang (C++23 support)
|
||
- Git (for subprojects)
|
||
- OpenCV 4 (optional; required for stereo point cloud compute)
|
||
|
||
The following dependencies are managed via Meson subprojects:
|
||
- [ASIO](https://think-async.com/Asio/) (Networking)
|
||
- [nlohmann/json](https://github.com/nlohmann/json) (JSON serialization)
|
||
- [yaml-cpp](https://github.com/jbeder/yaml-cpp) (Configuration loading)
|
||
- [glog](https://github.com/google/glog) (Logging)
|
||
- [jsonrpccxx](https://github.com/uS-S/jsonrpccxx) (JSON-RPC 2.0 implementation)
|
||
|
||
### Build & Run
|
||
|
||
```bash
|
||
meson setup build
|
||
meson compile -C build
|
||
./build/src/cloud_point_rpc_server config.yaml
|
||
```
|
||
|
||
*Note: You need a `config.yaml` file. See `config.yaml.example` for the required format.*
|
||
|
||
Run the interactive CLI client:
|
||
|
||
```bash
|
||
./build/src/cloud_point_rpc_cli config.yaml
|
||
```
|
||
|
||
CLI menu options (OpenCV options are hidden when built without opencv4):
|
||
|
||
| Option | Action |
|
||
|--------|--------|
|
||
| 1 | List available RPC methods |
|
||
| 2 | Get intrinsic params (legacy) |
|
||
| 3 | Get extrinsic params (legacy) |
|
||
| 4 | Compute point cloud — prints point count and bounding box |
|
||
| 5 | Compute point cloud and save to `output.ply` |
|
||
| 0 | Exit |
|
||
|
||
#### Build on windows
|
||
|
||
It's assumed that you have `GCC` and `make`/`ninja` installed on your system (and available in `PATH`)
|
||
|
||
```powershell
|
||
## FIRST OF ALL!
|
||
git submodule init
|
||
git submodule update
|
||
# Next python:
|
||
python3 -m venv .\venv
|
||
.\venv\Scripts\Activate.ps1
|
||
# or
|
||
.\venv\bin\Activate.ps1
|
||
pip install meson cmake
|
||
meson setup -Ddefault_library=static build
|
||
meson compile -C build
|
||
# To correctly get dlls:
|
||
meson devenv -C build
|
||
## .\build\tests\unit_tests < for dummy test
|
||
## .\build\src\.. < produced execs and libs
|
||
```
|
||
|
||
### Testing
|
||
|
||
```bash
|
||
meson test -C build -v
|
||
```
|
||
|
||
## Docker
|
||
|
||
You can build and run the cli using `Docker`.
|
||
|
||
### 1. Build Image
|
||
|
||
```bash
|
||
docker build -t cloud-point-rpc .
|
||
```
|
||
|
||
### 2. Run Container
|
||
|
||
The cli will try to connect to a **running server** on ip and port defined in config.yml file. (defined in `config.yaml` inside the image).
|
||
For simplicity, it's better to use a host network, so you will not have any headache with accessability.
|
||
|
||
> _Server is not configured to run through container, if you need, contact me_
|
||
|
||
You also can mount your own `config.yaml` to override the default settings:
|
||
|
||
```bash
|
||
docker run --network=host -it -v $(pwd)/my_config.yaml:/app/config.yaml cloud-point-rpc
|
||
```
|
||
|
||
## Validation with SCARED Dataset
|
||
|
||
The `scared_dataset_server` executable lets you validate the stereo point-cloud
|
||
pipeline against real endoscopic images from the
|
||
[SCARED dataset](https://huggingface.co/datasets/maxhallan7/scared).
|
||
|
||
### Obtaining the data
|
||
|
||
1. Download `test_dataset_8.zip` from
|
||
<https://huggingface.co/datasets/maxhallan7/scared>.
|
||
2. Extract so that `keyframe_0/` through `keyframe_4/` exist under
|
||
`test_dataset_8/`.
|
||
|
||
Each keyframe directory contains:
|
||
- `Left_Image.png`, `Right_Image.png` — 1280×1024 unrectified RGBA images.
|
||
- `endoscope_calibration.yaml` — OpenCV FileStorage with `M1`, `D1`, `M2`,
|
||
`D2`, `R`, `T` nodes.
|
||
|
||
**Note:** `T` is stored in **millimetres** in the YAML file (baseline ≈ −4.35 mm).
|
||
`scared_dataset_server` divides `T` by 1000 before placing it on the wire
|
||
(the wire protocol uses metres).
|
||
|
||
### Running the server
|
||
|
||
```bash
|
||
./build/src/cloud_point/scared_dataset_server \
|
||
/path/to/test_dataset_8/keyframe_0 8080
|
||
```
|
||
|
||
If port 8080 is already taken on your machine (Docker's `rootlesskit`
|
||
commonly holds it) pass another port and update `server.port` in the CLI
|
||
config accordingly. Connecting the CLI to a foreign service on 8080 shows up
|
||
as `invalid JSON response from server` / `std::bad_alloc` errors.
|
||
|
||
### Connecting with the CLI
|
||
|
||
In a second terminal run the interactive CLI with the SCARED-tuned config:
|
||
|
||
```bash
|
||
./build/src/cloud_point_rpc_cli config.scared.yml
|
||
# Option 4 — compute point cloud and print valid point count + bounding box
|
||
# Option 5 — compute point cloud and save a triangulated PLY mesh
|
||
```
|
||
|
||
`config.scared.yml` sets the optional `cloud_point` section that options 4/5
|
||
honour:
|
||
|
||
```yaml
|
||
cloud_point:
|
||
algorithm: cpu # "gpu" falls back to CPU when CUDA is unavailable
|
||
num_disparities: 160 # fx ~1024 px, baseline ~4.35 mm -> up to ~160 px
|
||
min_depth_m: 0.02 # endoscopic working range: 20 mm .. 300 mm
|
||
max_depth_m: 0.30
|
||
ply_stride: 1 # option 5: 4 = 16x smaller, block-averaged mesh
|
||
wls_filter: false # true = smoother mesh for viewers, less accurate
|
||
```
|
||
|
||
Depth limits are a physical bound on the scene: with this rig depth is
|
||
roughly `4.45 m / disparity_px`, so any mismatch with a disparity below
|
||
~15 px reprojects metres away. Without the section the CLI falls back to the
|
||
generic defaults (GPU, 128 disparities, 0.01–10 m). The SGBM matcher itself
|
||
is configured with OpenCV's recommended smoothness penalties (P1 = 8·bs²,
|
||
P2 = 32·bs²), a 5×5 block, uniqueness ratio 10, speckle filtering and a
|
||
left-right consistency check; see `CpuStereoMatcher::Params`.
|
||
|
||
### Checking the result
|
||
|
||
Option 4 should report a bounding box with z inside roughly
|
||
`[0.03, 0.16]` m for `test_dataset_8` keyframes. Option 5 writes a binary
|
||
little-endian PLY containing every valid point **and** a mesh triangulated
|
||
from the pixel grid (triangles are dropped where the longest edge exceeds 5 % of the local
|
||
depth, so the mesh breaks at occlusions). The mesh is what makes web viewers
|
||
usable: viewers such as Meshy's online PLY viewer fabricate a triangle from
|
||
every three consecutive vertices of a vertex-only PLY, which draws long
|
||
slivers across the surface and makes a correct cloud look like a fan of
|
||
rays. Pass `PlyOptions{false, 0.0f, false}` to `write_ply` for a points-only ASCII
|
||
file.
|
||
|
||
**Surface roughness vs accuracy.** SGBM's sub-pixel disparity noise
|
||
(~0.2 px, correlated over several pixels) is ~1 mm of depth on this rig,
|
||
far more than the 0.07 mm lateral pixel pitch, so a mesh built from the raw
|
||
cloud is "hairy": face normals sit a median 45° off the camera axis on
|
||
tissue that faces the camera. Colour depth maps hide this; shaded mesh
|
||
viewers show it as fuzz. `wls_filter: true` applies OpenCV's edge-aware
|
||
WLS disparity filter (needs `opencv_ximgproc`, doubles matching time) and
|
||
brings the median normal to ~23° with neighbour depth jumps down from 0.15 mm
|
||
to 0.06 mm, but it also costs accuracy on SCARED (keyframe 1: MAE 0.84 →
|
||
0.92 mm, within 2 mm 80 → 78 %; dataset_3: MAE 1.75 → 2.20 mm). It is
|
||
therefore off by default: use it for pictures, not for measurements. SGBM's
|
||
own holes are never filled by the filter.
|
||
|
||
When the keyframe contains `point_cloud.obj`, run the benchmark to compare the
|
||
reconstruction with its pixel-aligned XYZ ground truth:
|
||
|
||
```bash
|
||
./build/src/cloud_point/scared_dataset_benchmark \
|
||
/path/to/dataset_1/keyframe_1 160 [depth.png|-] [min_depth_m max_depth_m]
|
||
```
|
||
|
||
The benchmark always reports the valid-point fraction, depth percentiles
|
||
and matching/reconstruction timings as JSON, and optionally writes a
|
||
colour-mapped depth image (third argument, `-` to skip) for visual
|
||
inspection. The optional depth range applies the same filter as the CLI's
|
||
`cloud_point` section; without it the generic 0.01–10 m defaults are used,
|
||
which lets a few residual mismatches at metres of depth inflate RMSE. When
|
||
`point_cloud.obj` is present it also reports coverage, component-wise and
|
||
3-D errors and threshold accuracy. Note that the `test_dataset_*` archives
|
||
ship without `point_cloud.obj`; ground truth is only in the full
|
||
`dataset_N.zip` archives (13–40 GB each). The zips are served with HTTP
|
||
range support, so single keyframes can be extracted remotely with Python's
|
||
`zipfile` over a seekable HTTP file object instead of downloading the whole
|
||
archive.
|
||
|
||
`scripts/scared_overview.py` runs the benchmark over many keyframes and
|
||
prints a Markdown table:
|
||
|
||
```bash
|
||
scripts/scared_overview.py --png-dir out/depth --depth-range 0.02 0.30 \
|
||
datasets/scared/dataset_1/keyframe_* datasets/scared/test_dataset_8/keyframe_*
|
||
``` OBJ coordinates are
|
||
converted from millimetres to metres and rectified into the same left-camera
|
||
frame as the reconstructed cloud before evaluation. Reference numbers for
|
||
`dataset_1/keyframe_1` with the tuned SGBM configuration: coverage ≈ 0.85,
|
||
MAE₃D ≈ 0.8 mm, RMSE₃D ≈ 1.4 mm, 80 % of points within 2 mm (the previous
|
||
unregularised SGBM gave MAE₃D ≈ 44 mm and RMSE₃D ≈ 289 mm).
|
||
|
||
The E2E test (`tests/test_scared_dataset.cpp`) exercises the same pipeline
|
||
with `num_disparities = 160` and asserts >50 000 valid points and a median
|
||
depth in `[0.02, 0.20]` m.
|
||
|
||
## Communication model
|
||
|
||
|