From 1740f9d6fc9484f52b50d5ddbe43a9f425cee8a9 Mon Sep 17 00:00:00 2001 From: Artur Mukhamadiev Date: Mon, 14 Sep 2026 11:31:55 +0300 Subject: [PATCH] docs(readme): document stereo pipeline stages, results table and roadmap --- README.md | 57 +++++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 49 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index bb5200a..dd025e7 100644 --- a/README.md +++ b/README.md @@ -12,9 +12,25 @@ Communication JSON RPC protocol and implementation with Unity Scene. ## Status +Done: + - [x] Server implementation with C-API for Unity -- [x] OpenCV stereo client (StereoRectifier, PointCloudBuilder, CloudPointClient facade) +- [x] OpenCV stereo client (StereoRectifier, CPU/GPU matchers, PointCloudBuilder, CloudPointClient facade) +- [x] Tuned SGBM for the SCARED rig (P1/P2 penalties, uniqueness, speckle, LRC) and depth-range clipping +- [x] Binary PLY export with grid triangulation and `ply_stride` decimation +- [x] Optional WLS / median disparity post-filters (`wls_filter`, `CpuStereoMatcher::Params`) +- [x] SCARED benchmark with ground-truth metrics, depth PNGs and `scripts/scared_overview.py` + +To do: + - [ ] Unity-side C# implementation per [docs/unity-integration.md](docs/unity-integration.md) +- [ ] Remove sliver triangles at the border of the valid region (long spikes are still visible in + shaded viewers at `ply_stride: 4` because 5 % of local depth allows 3 mm edges) +- [ ] Visually compare raw vs WLS meshes in a shaded web viewer and decide the visualisation default +- [ ] Expose the remaining `CpuStereoMatcher::Params` (block size, uniqueness, speckle, median kernel, + WLS lambda/sigma) in the YAML `cloud_point` section +- [ ] Validate the CUDA `StereoSGM` path on a GPU machine (currently only exercised via CPU fallback) +- [ ] Reduce sub-pixel SGBM noise without the WLS accuracy loss (e.g. bilateral or guided filter on depth) ## API Documentation @@ -22,7 +38,17 @@ See [API.md](API.md) for detailed request/response formats. ## Pipeline -Unity acts as a data source: it serves stereo image pairs (`get-image-pair`) and full stereo calibration (`get-stereo-calibration`) over JSON-RPC 2.0. The C++ `CloudPointClient` calls `connect()` once to fetch calibration, then on each `compute_cloud()` call it fetches a synchronised image pair, runs stereo rectification (`StereoRectifier`, cv::stereoRectify + remap), computes disparity with SGBM (16× scaling), reprojects to 3-D with `cv::reprojectImageTo3D` (`PointCloudBuilder`), filters NaN/invalid points, and returns a `PointCloud`. An optional `write_ply()` helper serialises the result to disk. +Unity acts as a data source: it serves stereo image pairs (`get-image-pair`) and full stereo calibration (`get-stereo-calibration`) over JSON-RPC 2.0. The C++ `CloudPointClient` calls `connect()` once to fetch calibration, then on each `compute_cloud()` call it fetches a synchronised image pair and runs the stages below. + +| Stage | Class | Algorithm | +|-------|-------|-----------| +| Rectification | `StereoRectifier` | `cv::stereoRectify` + `initUndistortRectifyMap`/`remap`; also yields the Q reprojection matrix | +| Disparity (CPU) | `CpuStereoMatcher` | `cv::StereoSGBM` (semi-global block matching, `MODE_SGBM`, 16× fixed point) with optional median blur or `cv::ximgproc` WLS post-filter | +| Disparity (GPU) | `GpuStereoMatcher` | `cv::cuda::StereoSGM` (`MODE_HH4`, 64/128/256 disparity levels); default, falls back to CPU without CUDA | +| Reprojection | `PointCloudBuilder` | `cv::reprojectImageTo3D` with Q, then rejects disparity ≤ 0, OpenCV sentinels and depth outside `[min_depth_m, max_depth_m]` | +| Export | `write_ply` | Binary little-endian PLY with grid triangulation (edges ≤ 5 % of local depth) and optional `stride` block averaging | + +`StereoMatcherFactory` picks the matcher from `CloudPointConfig::algorithm` and forwards `CpuStereoMatcher::Params`. Depth follows `z = fx·B / disparity`; on the SCARED rig that is about `4.45 m / disparity_px`, which is why the disparity range and depth limits in `config.scared.yml` matter. See [API.md](API.md) for wire schemas and [docs/unity-integration.md](docs/unity-integration.md) for the Unity C# design spec. @@ -241,12 +267,27 @@ prints a Markdown table: ```bash scripts/scared_overview.py --png-dir out/depth --depth-range 0.02 0.30 \ datasets/scared/dataset_1/keyframe_* datasets/scared/test_dataset_8/keyframe_* -``` OBJ coordinates are -converted from millimetres to metres and rectified into the same left-camera -frame as the reconstructed cloud before evaluation. Reference numbers for -`dataset_1/keyframe_1` with the tuned SGBM configuration: coverage ≈ 0.85, -MAE₃D ≈ 0.8 mm, RMSE₃D ≈ 1.4 mm, 80 % of points within 2 mm (the previous -unregularised SGBM gave MAE₃D ≈ 44 mm and RMSE₃D ≈ 289 mm). +``` + +Ground-truth OBJ coordinates are converted from millimetres to metres and +rectified into the same left-camera frame as the reconstructed cloud before +evaluation. Reference numbers with the tuned SGBM configuration and the +0.02–0.30 m depth range (raw disparity, no WLS): + +| Keyframe | Valid points | MAE₃D | RMSE₃D | Within 2 mm | +|----------|--------------|-------|--------|-------------| +| dataset_1 kf1 | 84.6 % | 0.84 mm | 1.43 mm | 80 % | +| dataset_1 kf2 | 86.3 % | 1.15 mm | 2.05 mm | 76 % | +| dataset_1 kf3 | 86.9 % | 1.09 mm | 8.15 mm | 76 % | +| dataset_1 kf4 | 84.7 % | 0.66 mm | 1.26 mm | 85 % | +| dataset_1 kf5 | 84.6 % | 0.79 mm | 1.51 mm | 77 % | +| dataset_2 kf1 | 72 % | 1.10 mm | 3.76 mm | — (near tissue at the disparity limit) | +| dataset_3 kf1 | 83.6 % | 1.75 mm | 4.06 mm | 67 % | +| test_dataset_8 kf0–4 | 79–86 % | no ground truth; median depth 56–115 mm | | | + +The previous unregularised SGBM gave MAE₃D ≈ 44 mm and RMSE₃D ≈ 289 mm on +`dataset_1/keyframe_1`. Matching takes ~325 ms per 1280×1024 frame on the +CPU (~650 ms with WLS). The E2E test (`tests/test_scared_dataset.cpp`) exercises the same pipeline with `num_disparities = 160` and asserts >50 000 valid points and a median