Compare commits

...

7 Commits

Author SHA1 Message Date
ea99cd62ee chore(release): bump version to 0.2.0
All checks were successful
Verification / Is-Buildable (push) Successful in 4m8s
TG-2 #ready-for-test
TG-3 #ready-for-test
2026-07-12 22:28:21 +03:00
2a77419b62 chore(skills): add commit-task-tags skill for kanban TG trailers
Documents the TG-<NUMBER> #<state> commit message trailer convention:
state semantics (new, ready, in-progress, ready-for-test, done), rules
for multi-commit epics, and a helper script for amending tags during
interactive rebase.
2026-07-12 22:28:21 +03:00
f8177d8926 fix(cloud_point): validation, thread safety, and test cleanup
- Parse port inside try block for proper error reporting instead of unhandled exception
- Make frame_counter atomic to eliminate data race under TcpServer per-client threads
- Validate num_disparities is positive multiple of 16 in StereoMatcherFactory
- Validate stereo pair dimensions match in ScaredDatasetLoader
- Silence nodiscard warnings via std::ignore in tests

TG-3 #ready-for-test
TG-2 #ready-for-test
2026-07-12 22:28:21 +03:00
e139a55143 feat(cloud_point): SCARED dataset validation with real endoscopic stereo data
- ScaredDatasetLoader: loads M1/D1/M2/D2/R/T from OpenCV YAML + Left/Right PNGs
- scared_dataset_server executable serving get-stereo-calibration and get-image-pair
- E2E test against SCARED dataset with GTEST_SKIP guard (requires SCARED_KEYFRAME_DIR)
- README SCARED dataset validation section with CLI options and mm->m conversion
- CLI disparity-range caveat documentation

TG-2 #in-progress
2026-07-12 22:28:21 +03:00
162c210a7d feat(cloud_point): stereo rectification, point cloud pipeline, and CloudPointClient facade
- StereoRectifier wrapping cv::stereoRectify + cv::initUndistortRectifyMap (CV_16SC2 maps)
- PointCloudBuilder with cv::reprojectImageTo3D and depth/NaN filtering
- CloudPointClient high-level facade: connect(), compute_cloud() -> std::expected<PointCloud, Error>
- write_ply() ASCII PLY export helper
- CLI options 4 (compute-cloud) and 5 (compute-cloud + save PLY)
- Fix TcpServer to loop over multiple requests per connection
- Expose num_disparities parameter through CloudPointClient and StereoMatcherFactory
- Unity C# integration design spec
- Sync README, AGENTS, openwiki with stereo pipeline (C++23)
- E2E synthetic-scene test with constant-disparity stereo pair

TG-9 #ready-for-test
TG-4 #ready-for-test
TG-2 #in-progress
2026-07-12 22:28:21 +03:00
d486f550d2 feat(rpc): stereo calibration and image-pair wire protocol
- Remove unused stdexec dependency
- Fix dangling cv::Mat references and swapped dimensions in image/matrix factories
- Add CameraCalib, StereoCalibrationRPC, ImagePairRPC DTOs with nlohmann serialization
- Implement get-stereo-calibration and get-image-pair RPC handlers with deterministic mock data
- Extend RpcClient with typed getters and params-bearing call overload
- Restructure API.md with full JSON schemas and conventions
- 13 unit and integration tests for serialization

TG-3 #in-progress
TG-7 #ready-for-test
2026-07-12 22:28:21 +03:00
24b6f9937f feat(cloud_point): add OpenCV integration, stereo matching, and get-available-methods RPC
- Add OpenCV and stdexec dependencies
- ImageRPC type bridging OpenCV and RPC layers
- Stereo rectification boilerplate
- get-available-methods RPC method with get_count/get_method_name_by_id/get_method_names
- Tests for remote method retrieval
- CPU/GPU stereo matcher interfaces with SGBM implementation
- Documentation consistency pass across impl and docs

TG-5 #ready-for-test
TG-2 #in-progress
2026-07-12 22:28:21 +03:00
77 changed files with 3882 additions and 272 deletions

5
.gitignore vendored
View File

@ -7,5 +7,10 @@ subprojects/nlohmann_json/
subprojects/packagecache/
subprojects/yaml-cpp-0.8.0
subprojects/base64-0.5.2/
subprojects/stdexec/
subprojects/.*
.venv/
.worktrees/
# IDE
.idea/

View File

@ -0,0 +1,99 @@
---
name: commit-task-tags
description: Manage kanban task tags (TG-<NUMBER> #<state>) in git commit messages. Use when creating, squashing, or rebasing commits to ensure task state trailers are correct.
---
# Commit Task Tags
This project tracks kanban board tasks inside git commit messages using trailer
lines in the format:
```
TG-<NUMBER> #<state>
```
Multiple trailers may appear in a single commit (one per related task). Trailers
go at the bottom of the commit body, separated from the prose by a blank line.
## States
| State | Meaning |
|-------|---------|
| `new` | Task created on the board; no work started yet. |
| `ready` | Task is defined and ready to be picked up. |
| `in-progress` | This commit contributes to the task, but the full task/epic is **not** yet implemented. Use for every intermediate commit. |
| `ready-for-test` | The **full implementation** of the task/epic is completed across all its commits. Place on the final commit of the task. |
| `done` | Property of the **full task**, not of an individual commit. Reserved for final board confirmation after testing. **Never** use `#done` in a code commit. |
### Key rules
1. **`#done` is not for commits.** A commit can at most mark a task
`#ready-for-test`. The `#done` state is set on the kanban board after
verification, not in git.
2. **Multi-commit epics use `#in-progress` then `#ready-for-test`.** Every
intermediate commit for a task carries `#in-progress`. Only the final commit
that completes the implementation switches to `#ready-for-test`.
3. **Single-commit tasks use `#ready-for-test`.** If a task is fully implemented
in one commit, that commit carries `#ready-for-test` (not `#done`).
4. **One task may span many commits.** Add the same `TG-<N>` trailer to every
commit that touches that task's work, updating the state as appropriate.
## Example commit
```
feat(cloud_point): stereo rectification and point cloud pipeline
- StereoRectifier wrapping cv::stereoRectify
- PointCloudBuilder with depth filtering
- E2E synthetic-scene test
TG-9 #ready-for-test
TG-2 #in-progress
```
## Adding tags to existing commits
### During interactive rebase (recommended)
Use `git rebase -i` with `edit` stops, then amend each commit:
```bash
# Strip old TG lines and append new ones in one command:
git log -1 --format=%B \
| sed '/^TG-[0-9]/d' \
| { cat; printf '\nTG-<N> #<state>\n'; } \
| git commit --amend --no-verify -F -
git rebase --continue
```
> **Avoid `reword`** with a shared `GIT_EDITOR` script — it can shift tags by one
> commit. Use `edit` stops with explicit `git commit --amend` instead.
### Helper script
A reusable script lives at `scripts/tag-commit.sh` in this skill directory.
```bash
# At a rebase edit stop, add or replace tags:
../scripts/tag-commit.sh "TG-5 #ready-for-test" "TG-2 #in-progress"
git rebase --continue
```
The script strips any existing `TG-*` lines from the current commit message,
appends the supplied trailers, and amends the commit.
## Squashing commits
When squashing many commits into fewer logical commits, reassign task tags to
the resulting squashed commits following the same state rules. After squashing:
1. Identify which tasks each squashed commit covers.
2. Mark intermediate squash commits `#in-progress` for tasks that continue in
later squash commits.
3. Mark the final squash commit for a task `#ready-for-test`.
Use `git reset --hard <group-end>` + `git reset --soft <prev-group-commit>` to
snapshot each group's tree, then `git commit` with the appropriate trailers.

View File

@ -0,0 +1,26 @@
#!/bin/sh
# tag-commit.sh — Add or replace TG-<NUMBER> #<state> trailers on the current commit.
#
# Usage (at a rebase edit stop or any HEAD you want to amend):
# ./tag-commit.sh "TG-5 #ready-for-test" "TG-2 #in-progress"
#
# Strips existing TG-* lines, appends the given trailers, and amends the commit.
set -eu
if [ "$#" -eq 0 ]; then
echo "Usage: $0 \"TG-<N> #<state>\" [\"TG-<N> #<state>\" ...]" >&2
exit 1
fi
# Build the trailer block from arguments.
trailers=""
for tag in "$@"; do
trailers="${trailers}${tag}"$'\n'
done
# Strip existing TG-* lines, append new trailers, amend commit.
git log -1 --format=%B \
| sed '/^TG-[0-9]/d' \
| { cat; printf '\n%s' "$trailers"; } \
| git commit --amend --no-verify -F -

View File

@ -1,6 +1,6 @@
# Cloud Point RPC Agent Guide
This repository contains a C++20 implementation of a JSON RPC protocol for communicating with a Unity Scene.
This repository contains a C++23 implementation of a JSON RPC protocol for communicating with a Unity Scene.
Agents working on this codebase must adhere to the following guidelines and conventions.
## 1. Build, Lint, and Test
@ -54,10 +54,10 @@ The project uses the **Meson** build system.
## 2. Code Style & Conventions
Adhere strictly to **Modern C++20** standards.
Adhere strictly to **Modern C++23** standards.
### General Guidelines
- **Standard:** C++20. Use concepts, ranges, and smart pointers. Avoid raw `new`/`delete`.
- **Standard:** C++23. Use concepts, ranges, `std::expected`, and smart pointers. Avoid raw `new`/`delete`.
- **Memory Management:** Use `std::unique_ptr` and `std::shared_ptr`.
- **Const Correctness:** Use `const` (and `constexpr`/`consteval`) whenever possible.
- **Includes:** Use absolute paths for project headers (e.g., `#include "rpc/server.hpp"`).
@ -66,16 +66,18 @@ Adhere strictly to **Modern C++20** standards.
### Naming Conventions
- **Files:** `snake_case.cpp`, `snake_case.hpp`.
- **Classes/Structs:** `PascalCase`.
- **Functions/Methods:** `snake_case` (or `camelCase` if adhering strictly to a specific external library style, but default to snake_case).
- **Functions/Methods:** `snake_case`.
- **Variables:** `snake_case`.
- **Private Members:** `snake_case_` (trailing underscore).
- **Constants:** `kPascalCase` or `ALL_CAPS` for macros (avoid macros).
- **Namespaces:** `snake_case`.
- **Interfaces:** `IPascalCase` (optional, but consistent if used).
- **Constants:** `kPascalCase` or `ALL_CAPS` for macros.
- **Namespaces:** `score` (primary project namespace).
- **Interfaces:** `IPascalCase`.
### Project Structure
- `include/cloud_point_rpc/`: Public header files.
- `src/`: Implementation files.
- `include/cloud_point_rpc/`: Public header files (RPC server/client, TCP, config, serialization, coder, DTOs).
- `include/cloud_point/`: OpenCV compute library headers (StereoRectifier, PointCloudBuilder, CloudPointClient). Optional; requires opencv4.
- `src/`: Implementation files and executable entrypoints.
- `src/cloud_point/`: OpenCV compute library implementation (optional, requires opencv4).
- `tests/`: Unit and integration tests.
- `subprojects/`: Meson wrap files for dependencies.
- `meson.build`: Build configuration.
@ -92,7 +94,7 @@ Adhere strictly to **Modern C++20** standards.
### Example Class
```cpp
namespace cloud_point_rpc {
namespace score {
/// @brief Manages camera parameters.
class CameraController {
@ -114,14 +116,19 @@ class CameraController {
std::vector<double> cached_intrinsics_;
};
} // namespace cloud_point_rpc
} // namespace score
```
### Implementation Details
- **JSON Library:** Use `nlohmann/json` (likely via `subprojects/nlohmann_json.wrap`).
- **Concurrency:** Use `std::jthread` (auto-joining) over `std::thread`.
- **RPC Methods:**
- Implement handlers for: `get-cloud-point`, `get-intrinsic-params`, `get-extrinsic-params`.
- **Error handling:** Use `std::expected<T, E>` (C++23) for recoverable errors in the `cloud_point` compute library.
- **RPC Methods** (served by Unity or the C++ mock in `server_main.cpp`):
- `get-available-methods` — list registered method names.
- `get-stereo-calibration` — full stereo rig calibration (intrinsics, R, T, image size).
- `get-image-pair` — synchronised stereo frame as two base64-encoded images.
- `get-intrinsic-params` *(legacy)* — left-camera intrinsic matrix (9 doubles).
- `get-extrinsic-params` *(legacy)* — left-camera extrinsic matrix (16 doubles).
- Ensure thread safety if the RPC server is multi-threaded.
## 3. Workflow & Git

200
API.md
View File

@ -1,14 +1,6 @@
# JSON-RPC API Documentation
The Cloud Point RPC server implements the **JSON-RPC 2.0** protocol over TCP.
> **NOTE 1:** Base64 encoding of data should be implemented on Unity Side.
> **NOTE 2:** Unit Tests were not written for the described API yet
Unity side expected:
- receive value of `params` field of request:`{}`
- return value of `result` field of response (string or json, both ASCII compliant)
The Cloud Point RPC server implements the **JSON-RPC 2.0** protocol over TCP.
## General Format
@ -19,11 +11,11 @@ All requests and responses are JSON objects.
{
"jsonrpc": "2.0",
"method": "<method_name>",
"params": {},
"params": {},
"id": <integer|string>
}
```
*Note: `params` is currently ignored by the implemented methods but is part of the standard.*
*`params` is currently ignored by all handlers but is valid per JSON-RPC 2.0.*
### Response (Success)
```json
@ -48,77 +40,153 @@ All requests and responses are JSON objects.
---
## Methods
## Conventions
### `get-intrinsic-params`
- **Matrices:** Row-major storage. A 3×3 matrix `M` with rows `[r0, r1, r2]` serialises as a flat 9-element JSON array `[r0[0], r0[1], r0[2], r1[0], ...]`.
- **Units:** Translation in **metres**. No pixel units unless stated.
- **Extrinsics convention (OpenCV):** `x_right = R · x_left + T`. For a parallel rig with baseline `b`, `R = I` and `T = [-b, 0, 0]` (e.g. `[-0.06, 0, 0]` for a 6 cm baseline).
- **Image layout:** Top-left origin, row-major, packed channels (BGR order unless otherwise noted). Unity must flip GPU readback vertically before encoding.
- **Image data encoding:** Raw pixel bytes encoded as **Base64** (standard alphabet, no line breaks). The `type` field indicates channel layout.
- **Calibration arrays:** Plain JSON double arrays — **not** base64. Only image pixel data uses base64.
Retrieves the intrinsic camera parameters as a flat 3x3 matrix (row-major).
---
## Methods Served by the Unity Side
These methods are implemented server-side (in Unity, or in the C++ test mock in `src/server_main.cpp`).
---
### `get-available-methods`
Returns the list of method names registered on this server instance.
**Request:**
```json
{
"jsonrpc": "2.0",
"method": "get-intrinsic-params",
"id": 1
}
{ "jsonrpc": "2.0", "method": "get-available-methods", "id": 0 }
```
**Response:**
```json
{
"jsonrpc": "2.0",
"result": <base64-encoded-array>,
"id": 1
}
```
*Type: `vector<double>` (size 9) encoded as base64*
**Response `result`:** `["method-name-1", "method-name-2", ...]`
### `get-extrinsic-params`
---
Retrieves the extrinsic camera parameters as a flat 4x4 matrix (row-major).
### `get-stereo-calibration`
Returns full stereo rig calibration: intrinsics for both cameras, stereo rotation and translation.
**Request:**
```json
{
"jsonrpc": "2.0",
"method": "get-extrinsic-params",
"id": 2
}
{ "jsonrpc": "2.0", "method": "get-stereo-calibration", "id": 1 }
```
**Response:**
**Response `result`:**
```json
{
"jsonrpc": "2.0",
"result": <base64-encoded-array>,
"id": 2
}
```
*Type: `vector<double>` (size 16) encoded as base64*
### `get-cloud-point`
Retrieves the current field of view point cloud.
**Request:**
```json
{
"jsonrpc": "2.0",
"method": "get-cloud-point",
"id": 3
}
```
**Response:**
```json
{
"jsonrpc": "2.0",
"result": {
"width": int,
"height": int,
"data": <base64-encoded-array>
"left": {
"camera_matrix": [fx, 0, cx, 0, fy, cy, 0, 0, 1],
"dist_coeffs": [k1, k2, p1, p2, k3]
},
"id": 3
"right": {
"camera_matrix": [fx, 0, cx, 0, fy, cy, 0, 0, 1],
"dist_coeffs": [k1, k2, p1, p2, k3]
},
"rotation": [r00, r01, r02, r10, r11, r12, r20, r21, r22],
"translation": [tx, ty, tz],
"image_size": { "width": 640, "height": 480 }
}
```
*Type of data: `matrix WxH` (List of [x, y, z] points) encoded as base 64*
Field details:
| Field | Type | Size | Description |
|-------|------|------|-------------|
| `camera_matrix` | `double[]` | 9 | Row-major 3×3 intrinsic matrix: `[fx, 0, cx, 0, fy, cy, 0, 0, 1]` |
| `dist_coeffs` | `double[]` | 5 | Radial/tangential coefficients `[k1, k2, p1, p2, k3]` |
| `rotation` | `double[]` | 9 | Row-major rotation matrix R (left-to-right frame, OpenCV convention) |
| `translation` | `double[]` | 3 | Translation vector in metres |
| `image_size` | object | — | Sensor resolution before any rectification |
**Test mock defaults:** fx=fy=800, cx=320, cy=240, zero distortion, R=identity, T=[-0.06, 0, 0], 640×480.
---
### `get-image-pair`
Returns a synchronised stereo frame as two base64-encoded images.
**Request:**
```json
{ "jsonrpc": "2.0", "method": "get-image-pair", "id": 2 }
```
**Response `result`:**
```json
{
"frame": 42,
"left": { "width": 640, "height": 480, "type": "BGR", "data": "<base64>" },
"right": { "width": 640, "height": 480, "type": "BGR", "data": "<base64>" }
}
```
Field details:
| Field | Description |
|-------|-------------|
| `frame` | Monotonically increasing counter per server instance; wraps at `uint64_t` max. |
| `type` | Channel layout: `"BGR"` (3 ch), `"RGBA"` (4 ch), or `"DEPTH"` (1 ch float32). |
| `data` | Base64-encoded raw pixel bytes. Size = `width × height × channels`. |
Unity must supply images in top-left-origin row-major order; flip GPU readback vertically before encoding.
---
### `get-intrinsic-params` *(legacy)*
Returns left-camera intrinsic matrix as a flat 9-element double array.
**Request:**
```json
{ "jsonrpc": "2.0", "method": "get-intrinsic-params", "id": 3 }
```
**Response `result`:** `[fx, 0, cx, 0, fy, cy, 0, 0, 1]`
9 plain JSON doubles, row-major 3×3. **Not base64.**
---
### `get-extrinsic-params` *(legacy)*
Returns a flat 16-element double array (4×4 row-major extrinsic matrix).
**Request:**
```json
{ "jsonrpc": "2.0", "method": "get-extrinsic-params", "id": 4 }
```
**Response `result`:** `[r00, r01, r02, tx, r10, r11, r12, ty, r20, r21, r22, tz, 0, 0, 0, 1]`
16 plain JSON doubles. **Not base64.**
---
## Client-Side Outputs
These are computed locally by the C++ client (`RpcClient`) from data received via the methods above. They are **not** JSON-RPC methods callable on the server.
---
### `get-cloud-point` *(computed by client — spec in progress)*
Reconstructs a dense 3-D point cloud from the rectified stereo pair. The `RpcClient::get_cloud_point()` method will call `get-stereo-calibration` and `get-image-pair`, run stereo rectification and disparity computation locally (OpenCV SGBM or CUDA stereo), and reproject to 3-D.
**Expected future result shape** (subject to change in Phase 2):
```json
{
"width": <int>,
"height": <int>,
"data": "<base64-encoded float32 XYZ triplets, row-major>"
}
```
`data` will encode `width × height × 3` little-endian `float32` values (x, y, z in metres per pixel, `NaN` for invalid/occluded depth). Full specification and encoding details are deferred to Phase 2.

106
README.md
View File

@ -2,35 +2,75 @@
Communication JSON RPC protocol and implementation with Unity Scene.
## TODO
## Project Structure
- `include/`: Header files for the RPC server, TCP server, and C-API.
- `src/`: Implementation of the RPC logic, networking, and C-API.
- `src/cloud_point/`: OpenCV-based image processing and rectification logic.
- `docs/`: Documentation diagrams and models.
- `subprojects/`: Dependencies managed by Meson.
## Status
- [x] Server implementation with C-API for Unity
- [ ] Client correct implementation with OpenCV
- [x] OpenCV stereo client (StereoRectifier, PointCloudBuilder, CloudPointClient facade)
- [ ] Unity-side C# implementation per [docs/unity-integration.md](docs/unity-integration.md)
## API Documentation
See [API.md](API.md) for detailed request/response formats.
## Pipeline
Unity acts as a data source: it serves stereo image pairs (`get-image-pair`) and full stereo calibration (`get-stereo-calibration`) over JSON-RPC 2.0. The C++ `CloudPointClient` calls `connect()` once to fetch calibration, then on each `compute_cloud()` call it fetches a synchronised image pair, runs stereo rectification (`StereoRectifier`, cv::stereoRectify + remap), computes disparity with SGBM (16× scaling), reprojects to 3-D with `cv::reprojectImageTo3D` (`PointCloudBuilder`), filters NaN/invalid points, and returns a `PointCloud`. An optional `write_ply()` helper serialises the result to disk.
See [API.md](API.md) for wire schemas and [docs/unity-integration.md](docs/unity-integration.md) for the Unity C# design spec.
## Development
The project uses **Meson** build system and **C++20**.
The project uses **Meson** build system and **C++23**.
### Dependencies
- Meson, Ninja
- GCC/Clang (C++20 support)
- Meson (>= 1.1.0), Ninja
- GCC/Clang (C++23 support)
- Git (for subprojects)
- OpenCV 4 (optional; required for stereo point cloud compute)
The following dependencies are managed via Meson subprojects:
- [ASIO](https://think-async.com/Asio/) (Networking)
- [nlohmann/json](https://github.com/nlohmann/json) (JSON serialization)
- [yaml-cpp](https://github.com/jbeder/yaml-cpp) (Configuration loading)
- [glog](https://github.com/google/glog) (Logging)
- [jsonrpccxx](https://github.com/uS-S/jsonrpccxx) (JSON-RPC 2.0 implementation)
### Build & Run
```bash
git submodule init
git submodule update
meson setup build
meson compile -C build
./build/src/cloud_point_rpc_server config.yaml
```
*Note: You need a `config.yaml` file. See `config.yaml.example` for the required format.*
Run the interactive CLI client:
```bash
./build/src/cloud_point_rpc_cli config.yaml
```
CLI menu options (OpenCV options are hidden when built without opencv4):
| Option | Action |
|--------|--------|
| 1 | List available RPC methods |
| 2 | Get intrinsic params (legacy) |
| 3 | Get extrinsic params (legacy) |
| 4 | Compute point cloud — prints point count and bounding box |
| 5 | Compute point cloud and save to `output.ply` |
| 0 | Exit |
#### Build on windows
It's assumed that you have `GCC` and `make`/`ninja` installed on your system (and available in `PATH`)
@ -82,6 +122,58 @@ You also can mount your own `config.yaml` to override the default settings:
docker run --network=host -it -v $(pwd)/my_config.yaml:/app/config.yaml cloud-point-rpc
```
## Validation with SCARED Dataset
The `scared_dataset_server` executable lets you validate the stereo point-cloud
pipeline against real endoscopic images from the
[SCARED dataset](https://huggingface.co/datasets/maxhallan7/scared).
### Obtaining the data
1. Download `test_dataset_8.zip` from
<https://huggingface.co/datasets/maxhallan7/scared>.
2. Extract so that `keyframe_0/` through `keyframe_4/` exist under
`test_dataset_8/`.
Each keyframe directory contains:
- `Left_Image.png`, `Right_Image.png` — 1280×1024 unrectified RGBA images.
- `endoscope_calibration.yaml` — OpenCV FileStorage with `M1`, `D1`, `M2`,
`D2`, `R`, `T` nodes.
**Note:** `T` is stored in **millimetres** in the YAML file (baseline ≈ −4.35 mm).
`scared_dataset_server` divides `T` by 1000 before placing it on the wire
(the wire protocol uses metres).
### Running the server
```bash
./build/src/cloud_point/scared_dataset_server \
/path/to/test_dataset_8/keyframe_0 8080
```
### Connecting with the CLI
In a second terminal run the interactive CLI against the same host and port:
```bash
# Adjust ip/port in config.yaml if needed, then:
./build/src/cloud_point_rpc_cli config.yaml
# Option 4 — compute point cloud and print valid point count
# Option 5 — compute point cloud and save to output.ply (inspect in MeshLab)
```
The SCARED test set contains no ground-truth depth, so validation is
qualitative (inspect the PLY in MeshLab or similar).
**Disparity range caveat:** the CLI constructs `CloudPointClient` with the
default of 128 disparity levels, while this rig (fx ≈ 1024 px, baseline
≈ 4.35 mm) produces disparities above 128 px for tissue nearer than
~35 mm — those pixels silently drop out of the cloud. The E2E test
(`tests/test_scared_dataset.cpp`) passes `num_disparities = 160` for full
coverage; programmatic consumers should do the same via the
`CloudPointClient` constructor. For the CLI's qualitative check the default
is fine (observed median depth ≈ 115 mm is well within range).
## Communication model
![Communicatoin model plantuml diagram](docs/cm.png)

11
config.yaml.example Normal file
View File

@ -0,0 +1,11 @@
server:
ip: "127.0.0.1"
port: 8080
test_data:
intrinsic_params: [1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0]
extrinsic_params: [1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0]
cloud_point:
- [0.1, 0.2, 0.3]
- [1.1, 1.2, 1.3]
- [5.5, 6.6, 7.7]

View File

@ -2,65 +2,81 @@
box ClientProcess #LightBlue
Participant Caller
Participant CloudPointClient
Participant StereoRectifier
Participant PointCloudBuilder
Participant TCPClient
end box
box UnityProcess #LightGreen
Participant TCPServer
Participant CloudPointServer
Participant MainThreadQueue
Participant UnityWorld
end box
UnityWorld -> CloudPointServer : init thread
activate CloudPointServer
CloudPointServer -> TCPServer : await for connection
activate TCPServer
->CloudPointClient : init thread
activate CloudPointClient
CloudPointClient -> TCPClient : createConnection
TCPClient -> TCPServer : establish connection
TCPServer -> CloudPointServer : established
deactivate TCPServer
CloudPointServer -> TCPServer : await for calls
TCPServer -> TCPServer : await for packet
Caller -> CloudPointClient : I want something
activate CloudPointClient
CloudPointClient -> CloudPointClient : CallMethod<Something>
CloudPointClient -> TCPClient : send(message)
activate TCPClient
TCPClient -> TCPServer : packet send
TCPServer -> TCPServer : await for packet
activate TCPServer
TCPServer -> TCPServer : read packet
TCPServer -> TCPClient : packet read
TCPClient -> CloudPointClient : done
deactivate TCPClient
CloudPointClient -> TCPClient : await for response
activate TCPClient
TCPClient -> TCPClient : await for packet
TCPServer -> CloudPointServer : callMethod
activate CloudPointServer
CloudPointServer -> UnityWorld : addToStaticQueue
== Initialization ==
UnityWorld -> UnityWorld : read from queue
UnityWorld -> CloudPointServer : Awake() — crpc_init()
activate CloudPointServer
CloudPointServer -> TCPServer : start (await connections)
activate TCPServer
Caller -> CloudPointClient : connect()
activate CloudPointClient
CloudPointClient -> TCPClient : establish TCP connection
TCPClient -> TCPServer : TCP handshake
TCPServer -> TCPServer : spawn per-client handler thread (loops)
CloudPointClient -> TCPClient : get-stereo-calibration
TCPClient -> TCPServer : send request
TCPServer -> CloudPointServer : dispatch
CloudPointServer -> MainThreadQueue : enqueue task
MainThreadQueue -> UnityWorld : Update() dequeues
activate UnityWorld
UnityWorld -> UnityWorld : callMethod
UnityWorld -> CloudPointServer: set task return value
UnityWorld -> UnityWorld : read camera params
UnityWorld -> MainThreadQueue : tcs.SetResult(calibration JSON)
deactivate UnityWorld
CloudPointServer -> TCPServer : return task
deactivate CloudPointServer
TCPServer -> TCPClient : send response
TCPClient -> TCPServer : response read
TCPClient -> CloudPointClient : response received
TCPServer -> CloudPointServer : done
deactivate TCPServer
CloudPointClient -> Caller : here what you wanted
MainThreadQueue -> CloudPointServer : task complete
CloudPointServer -> TCPServer : send response
TCPServer -> TCPClient : response
TCPClient -> CloudPointClient : calibration received
CloudPointClient -> StereoRectifier : init(calibration)\ncv::stereoRectify + remap maps
deactivate CloudPointClient
Caller -> CloudPointClient : destruct
CloudPointClient -> TCPClient : finish waiting
deactivate TCPClient
== Per compute_cloud() call ==
Caller -> CloudPointClient : compute_cloud()
activate CloudPointClient
CloudPointClient -> TCPClient : get-image-pair
TCPClient -> TCPServer : send request
TCPServer -> CloudPointServer : dispatch
CloudPointServer -> MainThreadQueue : enqueue task
MainThreadQueue -> UnityWorld : Update() dequeues
activate UnityWorld
UnityWorld -> UnityWorld : Render() both cameras\nReadPixels + vertical flip\nBase64 encode
UnityWorld -> MainThreadQueue : tcs.SetResult(image pair JSON)
deactivate UnityWorld
MainThreadQueue -> CloudPointServer : task complete
CloudPointServer -> TCPServer : send response
TCPServer -> TCPClient : response (base64 left + right)
TCPClient -> CloudPointClient : image pair received
CloudPointClient -> StereoRectifier : rectify(left, right)
StereoRectifier -> StereoRectifier : cv::remap both images
StereoRectifier -> CloudPointClient : rectified pair
CloudPointClient -> PointCloudBuilder : build(rectified, Q)
PointCloudBuilder -> PointCloudBuilder : cv::StereoSGBM disparity\n(1/16 scale)\ncv::reprojectImageTo3D\nNaN filter
PointCloudBuilder -> CloudPointClient : PointCloud
CloudPointClient -> Caller : std::expected<PointCloud, Error>
deactivate CloudPointClient
UnityWorld -> CloudPointServer : destruct
== Teardown ==
Caller -> CloudPointClient : destruct
deactivate CloudPointClient
UnityWorld -> CloudPointServer : OnDestroy() — crpc_deinit()
deactivate CloudPointServer
deactivate TCPServer
@enduml

198
docs/unity-integration.md Normal file
View File

@ -0,0 +1,198 @@
# Unity Integration Design
This document specifies the C# architecture the Unity side (the
`UnityLaparoscopicSceneSimulator` project) must implement to serve stereo
images and calibration to the C++ `CloudPointClient` over the embedded RPC
server. It is a design spec — the C# code lives in the Unity repository.
The existing prototypes there (`Assets/Scripts/CrpcApi.cs`,
`Assets/Scripts/RpcTest.cs`) are a starting point but contain several
correctness bugs called out explicitly below.
## Data flow
```
Unity process (server) C++ client process
────────────────────── ──────────────────
CloudPointServer (C#)
crpc_init / crpc_add_method
get-stereo-calibration ◄────────────── CloudPointClient::connect() (once)
get-image-pair ◄────────────── CloudPointClient::compute_cloud()
│ rectify → SGBM → reproject
▼
PointCloud → caller
```
Unity is a **data source only**. The point cloud is computed and consumed on
the client side. See [API.md](../API.md) for the wire schemas.
## 1. `CloudPointServer` facade
A single MonoBehaviour owning the server lifecycle
(`Assets/Scripts/CloudPointRpc/CloudPointServer.cs`):
- `Awake()` — call `crpc_init(configPath)` (config.yaml under
`Application.streamingAssetsPath`, contains `server: {ip, port}`), then
register handlers.
- `OnDestroy()` / `OnApplicationQuit()` — shut down in the order specified in
§3 (Shutdown ordering).
- JSON: use Newtonsoft Json.NET (`com.unity.nuget.newtonsoft-json`).
`JsonUtility` cannot serialize dictionaries or nested arrays.
### Callback registration rules (IL2CPP + GC safety)
- Handler methods must be **static** and annotated with
`[AOT.MonoPInvokeCallback(typeof(RpcStringCallback))]` — instance methods
crash under IL2CPP.
- The delegate instance passed to `crpc_add_method` must be stored in a
**static field** for the lifetime of the server. The current `RpcTest.cs`
passes a method group directly; the marshalled thunk can be garbage
collected while C++ still holds the function pointer — a latent crash.
```csharp
private static readonly CrpcTestApi.RpcStringCallback s_calibrationCb = OnGetStereoCalibration;
// ...
CrpcExtensions.CrpcAddMethod(s_calibrationCb, "get-stereo-calibration");
```
## 2. `rpc_string` ownership rules
The C API (`server_api.h`) uses `rpc_string*` in both directions with
**different ownership**:
| Pointer | Owner | C# obligation |
|---|---|---|
| Handler **input** (`params` string) | C++ — a stack object inside `RpcServer::process` | Copy the data out immediately. **Never** call `crpc_str_destroy` on it. |
| Handler **return** value | Transfers to C++ — `rpc_server.cpp` destroys it after parsing | Create with `crpc_str_create`, return the handle, and **relinquish** C#-side ownership. |
Both rules are violated by the current `RpcTest.cs`:
it `Dispose()`s the input handle (destroying an object it doesn't own) and
returns a handle whose C# finalizer will later call `crpc_str_destroy` on
memory C++ has already freed — a double-destroy race.
Spec: split the wrapper into two types.
```csharp
/// Non-owning view over a C++-owned rpc_string. No finalizer, no Dispose.
readonly ref struct BorrowedRpcString { /* Data property only */ }
/// C#-created rpc_string. Dispose() destroys it; Release() transfers
/// ownership (nulls the handle and calls GC.SuppressFinalize).
sealed class OwnedRpcString : IDisposable
{
public IntPtr Release() { var h = _handle; _handle = IntPtr.Zero;
GC.SuppressFinalize(this); return h; }
}
```
Handlers end with `return new OwnedRpcString(json).Release();`.
## 3. Main-thread dispatcher
RPC callbacks arrive on a C++ per-client thread; Unity APIs are
main-thread-only. The callback must block until the main thread produces the
result. The `task.Wait()` idea in `RpcTest.cs` is directionally right but must
be hardened (`Assets/Scripts/CloudPointRpc/MainThreadDispatcher.cs`):
- Per request, create
`new TaskCompletionSource<string>(TaskCreationOptions.RunContinuationsAsynchronously)`
and enqueue `(Func<string> handler, tcs)` into a `ConcurrentQueue`.
- `Update()` drains the queue: run the handler, `tcs.TrySetResult(json)`
(or `TrySetException`).
- The callback thread waits with a **bounded timeout**:
`tcs.Task.Wait(TimeoutMs)` with ~5000 ms. On timeout, return a JSON string
with an `"error"` field (the C++ side passes the string through as the
result). An unbounded wait deadlocks permanently when the editor is paused
or the component is destroyed.
- Note that `server_api.cpp` holds a global `server_mtx` while dispatching, so
**one stuck handler stalls every client** — another reason the timeout is
mandatory.
### Shutdown ordering (deadlock hazard)
`crpc_deinit()` joins server threads. A server thread may be blocked inside a
callback waiting for the main thread — which is the thread calling
`crpc_deinit()`. Required order in `OnDestroy`:
1. Set a `volatile bool _shuttingDown` — new callbacks fail-fast with an error
result; pending queue entries get `TrySetCanceled()`.
2. Drain/clear the queue.
3. Only then call `crpc_deinit()`.
## 4. Required handlers
### `get-stereo-calibration`
Derive intrinsics from the rendering camera, extrinsics from the two camera
transforms (schema in API.md):
- `fx = width / (2 * tan(hFov / 2))`, `fy = height / (2 * tan(vFov / 2))`
where `vFov = Camera.fieldOfView` (degrees → radians) and
`hFov = 2 * atan(tan(vFov/2) * aspect)`. Use the render-texture resolution,
not the screen resolution.
- `cx = width / 2`, `cy = height / 2`, skew 0. `dist_coeffs` = five zeros
(ideal pinhole).
- Extrinsics: right camera relative to left in OpenCV convention
(`x_r = R·x_l + T`), converted per §5. For the standard parallel rig this is
`R = I`, `T = [-baseline, 0, 0]` with
`baseline = Vector3.Distance(left.position, right.position)` in meters.
- `image_size` = the RenderTexture size served by `get-image-pair`.
### `get-image-pair`
Both eyes must be captured on the **same rendered frame**:
- Give both cameras `targetTexture` RenderTextures; inside the (main-thread)
handler call `leftCam.Render(); rightCam.Render();` then read both back.
- Readback: synchronous `Texture2D.ReadPixels` inside the handler is
acceptable and simplest — the RPC thread is blocked waiting anyway.
`AsyncGPUReadback` is the documented optimization: issue the request in the
handler and complete the `TaskCompletionSource` from the readback callback
(the dispatcher design above already supports deferred completion).
- **Vertical flip**: GPU readbacks are bottom-up; the wire format is
top-left-origin row-major (API.md). Flip rows before encoding.
- Encode with `Convert.ToBase64String`; fill the `get-image-pair` schema with
`type: "RGBA"` (or convert to BGR to save 25% payload).
- Include a monotonically increasing `frame` counter
(`Time.frameCount` is fine).
### Legacy methods
`get-intrinsic-params` / `get-extrinsic-params` may be kept, backed by the
left camera, for backward compatibility. They are not used by
`CloudPointClient`.
## 5. Coordinate conventions
- Unity: left-handed, +Y up, +Z forward. OpenCV camera frame: right-handed,
+X right, **+Y down**, +Z forward.
- Conversion with `S = diag(1, -1, 1)`:
`R_cv = S · R_unity · S`, `t_cv = S · t_unity`.
- The reconstructed cloud is in the **OpenCV left-camera frame**. Consumers
that need Unity world space must apply the inverse conversion plus the left
camera pose; if needed, add an optional `left_pose` (16 doubles, row-major
4x4) field to `get-stereo-calibration`.
- All matrices row-major flattened; all lengths in meters (1 Unity unit = 1 m);
disparity in pixels.
## 6. Performance notes
- 1920x1080 RGBA is 8.3 MB raw ≈ 11 MB base64 per eye, ~22 MB per
`get-image-pair` — of the order 100–300 ms per fetch on loopback. For
interactive rates use 960x540 and/or BGR.
- `ReadPixels` stalls the GPU pipeline (~1–5 ms at 1080p): fine for on-demand
capture; do not capture every frame unconditionally.
- Base64 of large buffers allocates heavily; reuse buffers
(`Convert.TryToBase64Chars`) where practical.
- The server serializes requests globally (`server_mtx`), so handlers need no
reentrancy protection — but a slow handler blocks all clients (see §3).
## 7. Verification checklist
1. Start the Unity scene (server on the configured port).
2. From this repo: `./build/src/cloud_point_rpc_cli config.yaml`, option
`4` (compute-cloud) — expect a plausible point count and bounding box.
3. Option `5` writes a PLY; inspect it in MeshLab against the visible scene
geometry (a plane at 1.5 m should reconstruct at z ≈ 1.5).
4. Kill the client mid-request and re-connect — the server must keep serving
(per-client threads are independent; the TCP loop tolerates EOF).

View File

@ -0,0 +1,76 @@
#pragma once
#include "cloud_point/point_cloud_builder.hpp"
#include "cloud_point/stereo_matcher.hpp"
#include "cloud_point/stereo_matcher_factory.hpp"
#include "cloud_point/stereo_rectifier.hpp"
#include <expected>
#include <memory>
#include <string>
namespace score {
class TCPConnector;
class RpcClient;
/// @brief End-to-end stereo point cloud client.
///
/// Connects to a JSON-RPC stereo server, fetches calibration once on connect(),
/// then produces point clouds on demand via compute_cloud().
///
/// Thread safety: not thread-safe. Do not call methods concurrently.
class CloudPointClient {
public:
/// @brief Recoverable per-frame error.
struct Error {
std::string message;
};
/// @brief Construct client (does not connect).
/// @param ip Server IP address.
/// @param port Server port.
/// @param algo Stereo matching algorithm (GPU falls back to CPU if
/// unavailable).
/// @param opts Depth filtering options.
/// @param num_disparities SGBM disparity levels (default 128; use 160 for
/// small-baseline rigs such as SCARED).
CloudPointClient(std::string ip, int port,
StereoAlgorithmType algo = StereoAlgorithmType::GPU,
PointCloudBuilder::Options opts = {},
int num_disparities = 128);
~CloudPointClient();
/// @brief Connect to server and fetch calibration once.
/// @throws std::runtime_error on connection or calibration failure.
void connect();
/// @brief Return true if connected and all pipeline components are ready.
[[nodiscard]] bool connected() const noexcept;
/// @brief Compute one point cloud: fetch image pair → gray → rectify →
/// disparity → reproject.
///
/// Recoverable per-frame failures (RPC error, decode error) are returned as
/// Error. Calling before connect() returns Error immediately.
[[nodiscard]] std::expected<PointCloud, Error> compute_cloud();
private:
std::string ip_;
int port_;
StereoAlgorithmType algo_;
PointCloudBuilder::Options opts_;
int num_disparities_;
std::unique_ptr<TCPConnector> connector_;
std::unique_ptr<RpcClient> client_;
std::unique_ptr<StereoRectifier> rectifier_;
std::unique_ptr<IStereoMatcher> matcher_;
std::unique_ptr<PointCloudBuilder> builder_;
};
/// @brief Write valid points as ASCII PLY (for MeshLab inspection).
/// @param cloud Source point cloud.
/// @param path Output file path.
void write_ply(const PointCloud &cloud, const std::string &path);
} // namespace score

View File

@ -0,0 +1,22 @@
#pragma once
#include "cloud_point/stereo_matcher.hpp"
#include <opencv2/calib3d.hpp>
namespace score {
/// @brief CPU-based stereo matcher using cv::StereoSGBM.
class CpuStereoMatcher : public IStereoMatcher {
public:
CpuStereoMatcher(int min_disparity = 0, int num_disparities = 128,
int block_size = 3);
~CpuStereoMatcher() override = default;
[[nodiscard]] cv::Mat compute(const cv::Mat &left,
const cv::Mat &right) override;
private:
cv::Ptr<cv::StereoSGBM> sgbm_;
};
} // namespace score

View File

@ -0,0 +1,24 @@
#pragma once
#include "cloud_point/stereo_matcher.hpp"
namespace score {
/// @brief GPU-based stereo matcher using cv::cuda::StereoSGM.
/// Falls back to runtime error if CUDA is unavailable.
class GpuStereoMatcher : public IStereoMatcher {
public:
GpuStereoMatcher(int min_disparity = 0, int num_disparities = 16,
int block_size = 3);
~GpuStereoMatcher() override = default;
[[nodiscard]] cv::Mat compute(const cv::Mat &left,
const cv::Mat &right) override;
private:
#ifdef HAVE_OPENCV_CUDA
cv::Ptr<cv::cuda::StereoSGM> sgm_;
#endif
};
} // namespace score

View File

@ -0,0 +1,21 @@
//
// Created by vptyp on 11.03.2026.
//
#pragma once
#include <opencv4/opencv2/opencv.hpp>
namespace score {
class Image {
public:
Image();
explicit Image(const cv::Mat &image);
~Image();
/// @note data_ could be changed through this
[[nodiscard]] cv::Mat get();
protected:
cv::Mat data_;
};
} // namespace score

View File

@ -0,0 +1,47 @@
//
// Created by vptyp on 12.03.2026.
//
#pragma once
#include <cloud_point/image.h>
#include <cloud_point_rpc/imageRpc.h>
namespace score {
class ImageFactory {
public:
/**
* @brief tries to decode available RPC image type to opencv compliant
* @return opencv compliant image type
* @throw runtime_error if type is unknown
*/
static int pixelType(const ImageRPC::Type &type) {
switch (type) {
case ImageRPC::Type::BGR:
return CV_8UC3;
case ImageRPC::Type::RGBA:
return CV_8UC4;
case ImageRPC::Type::DEPTH:
return CV_64FC1;
default:
throw std::runtime_error("Unknown image type");
}
}
/**
* @brief tries to create Image object from ImageRPC
* @throw runtime_error if type is unknown or data size does not match
* dimensions
*/
static Image create(const ImageRPC &image) {
const int cv_type = pixelType(image.type);
const size_t expected = static_cast<size_t>(image.width) *
static_cast<size_t>(image.height) *
CV_ELEM_SIZE(cv_type);
if (image.data.size() != expected)
throw std::runtime_error("Image data size does not match "
"width*height*channels");
cv::Mat imageMat(image.height, image.width, cv_type,
const_cast<unsigned char *>(image.data.data()));
return Image{imageMat.clone()};
}
};
} // namespace score

View File

@ -0,0 +1,23 @@
#pragma once
#include "opencv2/core/hal/interface.h"
#include "opencv2/core/mat.hpp"
#include <stdexcept>
namespace score {
class CameraMatrixFactory {
public:
/**
* @param rpc < vector of size Width*Height
* @throw runtime_error if size is not Width*Height
*/
template <size_t Width, size_t Height>
static cv::Mat create(const std::vector<double> &rpc) {
if (rpc.size() != Width * Height)
throw std::runtime_error("Vector size is not Width*Height");
return cv::Mat(Width, Height, CV_64F, const_cast<double *>(rpc.data()))
.clone();
}
};
} // namespace score

View File

@ -0,0 +1,66 @@
#pragma once
#include <array>
#include <limits>
#include <opencv2/core.hpp>
#include <vector>
namespace score {
/// @brief Dense point cloud in row-major XYZ layout.
///
/// Invalid (occluded or out-of-range) points are represented as quiet NaN on
/// all three coordinates. valid_points() filters them out.
struct PointCloud {
int width{0};
int height{0};
std::vector<float> data; ///< width * height * 3 floats, XYZ row-major
/// @brief Return only the non-NaN points.
[[nodiscard]] std::vector<std::array<float, 3>> valid_points() const;
};
/// @brief Reprojects a disparity map to a 3-D point cloud using the
/// reprojection matrix Q produced by cv::stereoRectify.
///
/// Thread safety: build() is const and safe to call concurrently once the
/// object is constructed.
class PointCloudBuilder {
public:
/// @brief Depth filtering thresholds.
struct Options {
float min_depth_m;
float max_depth_m;
// Explicit constructor avoids a GCC limitation with nested-struct
// default-member-initialisers used as default function arguments.
Options() noexcept : min_depth_m(0.01f), max_depth_m(10.0f) {}
Options(float min_m, float max_m) noexcept
: min_depth_m(min_m), max_depth_m(max_m) {}
};
/// @brief Construct builder.
/// @param q 4x4 CV_64F reprojection matrix from cv::stereoRectify.
/// @param opts Optional depth-range filter.
/// @throws std::invalid_argument if q is not 4x4 CV_64F.
explicit PointCloudBuilder(cv::Mat q, Options opts = Options{});
/// @brief Reproject disparity to a point cloud.
///
/// Accepts:
/// - CV_16S SGBM fixed-point output (values = disparity * 16)
/// - CV_32F already in pixel units
///
/// Invalid points (disparity ≤ 0, |z| ≥ 10000, or z outside
/// [min_depth_m, max_depth_m]) are stored as quiet NaN.
///
/// @param disparity Disparity map (CV_16S or CV_32F).
/// @return Populated PointCloud.
/// @throws std::invalid_argument on unsupported disparity type.
[[nodiscard]] PointCloud build(const cv::Mat &disparity) const;
private:
cv::Mat q_;
Options opts_;
};
} // namespace score

View File

@ -0,0 +1,38 @@
#pragma once
#include "cloud_point_rpc/rpc_dto.hpp"
#include <string>
namespace score {
/// @brief Loads stereo calibration and image pair from a SCARED dataset
/// keyframe directory.
///
/// Expected directory layout:
/// <keyframe_dir>/endoscope_calibration.yaml — OpenCV FileStorage
/// <keyframe_dir>/Left_Image.png — 1280x1024 RGBA PNG
/// <keyframe_dir>/Right_Image.png — 1280x1024 RGBA PNG
///
/// The YAML node T is in millimetres; this loader converts to metres before
/// populating StereoCalibrationRPC.translation.
class ScaredDatasetLoader {
public:
/// @brief Load calibration and images from @p keyframe_dir.
/// @throws std::runtime_error if any file cannot be opened or parsed,
/// or if the left and right images have different dimensions.
explicit ScaredDatasetLoader(const std::string &keyframe_dir);
/// @brief Return the stereo calibration DTO (translation in metres).
[[nodiscard]] const StereoCalibrationRPC &calibration() const noexcept;
/// @brief Return an image pair DTO with the given frame index.
/// The images are the same for every call (single keyframe).
[[nodiscard]] ImagePairRPC image_pair(uint64_t frame) const;
private:
StereoCalibrationRPC calib_;
ImageRPC left_image_;
ImageRPC right_image_;
};
} // namespace score

View File

@ -0,0 +1,20 @@
#pragma once
#include <opencv2/core.hpp>
namespace score {
/// @brief Abstract interface for stereo disparity computation.
class IStereoMatcher {
public:
virtual ~IStereoMatcher() = default;
/// @brief Compute disparity map from a rectified stereo pair.
/// @param left Left image (grayscale, CV_8UC1).
/// @param right Right image (grayscale, CV_8UC1).
/// @return Disparity map (CV_16S for CPU, type depends on backend for GPU).
[[nodiscard]] virtual cv::Mat compute(const cv::Mat &left,
const cv::Mat &right) = 0;
};
} // namespace score

View File

@ -0,0 +1,23 @@
#pragma once
#include "cloud_point/stereo_matcher.hpp"
#include <memory>
namespace score {
enum class StereoAlgorithmType { CPU, GPU };
/// @brief Factory for creating CPU or GPU stereo matchers.
class StereoMatcherFactory {
public:
/// @brief Create a stereo matcher of the requested type.
/// If GPU is requested but unavailable, falls back to CPU.
/// @param num_disparities Number of disparity levels for SGBM (default 128).
/// Must be a positive multiple of 16.
/// @throws std::invalid_argument if @p num_disparities is not a positive
/// multiple of 16.
[[nodiscard]] static std::unique_ptr<IStereoMatcher>
create(StereoAlgorithmType type, int num_disparities = 128);
};
} // namespace score

View File

@ -0,0 +1,60 @@
#pragma once
#include <opencv2/calib3d.hpp>
#include <opencv2/core.hpp>
#include <opencv2/imgproc.hpp>
#include "cloud_point_rpc/rpc_dto.hpp"
namespace score {
/// @brief Stereo rectifier that computes rectification maps from calibration
/// data and applies them to image pairs.
///
/// Thread safety: const methods (rectify, q) are safe to call concurrently.
/// The object must not be modified after construction.
class StereoRectifier {
public:
/// @brief Calibration parameters for a stereo rig.
struct Calibration {
cv::Mat k_left; ///< 3x3 CV_64F intrinsic matrix, left camera
cv::Mat d_left; ///< 1x5 CV_64F distortion coefficients, left camera
cv::Mat k_right; ///< 3x3 CV_64F intrinsic matrix, right camera
cv::Mat d_right; ///< 1x5 CV_64F distortion coefficients, right camera
cv::Mat r; ///< 3x3 CV_64F rotation from left to right camera
cv::Mat
t; ///< 3x1 CV_64F translation from left to right camera (metres)
cv::Size image_size;
/// @brief Build calibration from the wire DTO using
/// CameraMatrixFactory.
/// @param rpc Stereo calibration received over JSON-RPC.
/// @return Populated Calibration struct.
static Calibration from_rpc(const StereoCalibrationRPC &rpc);
};
/// @brief Construct rectifier from calibration data.
///
/// Calls cv::stereoRectify (CALIB_ZERO_DISPARITY, alpha=0) and
/// cv::initUndistortRectifyMap (CV_16SC2) for both sides.
///
/// @throws std::invalid_argument if any Mat has the wrong size or type.
explicit StereoRectifier(const Calibration &calib);
/// @brief Apply rectification maps to a stereo pair.
/// @param left Left input image (any type accepted by cv::remap).
/// @param right Right input image.
/// @return {rectified_left, rectified_right}.
[[nodiscard]] std::pair<cv::Mat, cv::Mat>
rectify(const cv::Mat &left, const cv::Mat &right) const;
/// @brief Access the 4x4 reprojection matrix Q produced by stereoRectify.
[[nodiscard]] const cv::Mat &q() const noexcept;
private:
cv::Mat map_lx_, map_ly_; ///< Rectification maps for the left image
cv::Mat map_rx_, map_ry_; ///< Rectification maps for the right image
cv::Mat q_; ///< 4x4 CV_64F reprojection matrix
};
} // namespace score

View File

@ -1,8 +1,8 @@
#pragma once
#include "export.h"
#include <iostream>
#include <string>
#include "export.h"
namespace score {
/**
@ -14,7 +14,7 @@ namespace score {
* @param port Server Port
* @return int exit code
*/
int CRPC_EXPORT run_cli(std::istream &input, std::ostream &output, const std::string &ip,
int port);
int CRPC_EXPORT run_cli(std::istream &input, std::ostream &output,
const std::string &ip, int port);
} // namespace cloud_point_rpc
} // namespace score

View File

@ -69,4 +69,4 @@ class ConfigLoader {
}
};
} // namespace cloud_point_rpc
} // namespace score

View File

@ -0,0 +1,21 @@
//
// Created by vptyp on 12.03.2026.
//
#pragma once
#include <vector>
namespace score {
struct ImageRPC {
int width{0};
int height{0};
enum class Type {
UNKNOWN,
BGR,
RGBA,
DEPTH,
} type{Type::UNKNOWN};
std::vector<unsigned char> data;
};
} // namespace score

View File

@ -1,16 +1,18 @@
#pragma once
#include "tcp_connector.hpp"
#include "cloud_point_rpc/rpc_dto.hpp"
#include "cloud_point_rpc/tcp_connector.hpp"
#include <asio.hpp>
#include <glog/logging.h>
#include <jsonrpccxx/client.hpp>
#include <nlohmann/json.hpp>
#include <vector>
namespace score {
class RpcClient : public jsonrpccxx::JsonRpcClient {
public:
RpcClient(TCPConnector &connector)
explicit RpcClient(TCPConnector &connector)
: jsonrpccxx::JsonRpcClient(connector, jsonrpccxx::version::v2) {}
[[nodiscard]] std::vector<double> get_intrinsic_params() {
@ -25,15 +27,36 @@ class RpcClient : public jsonrpccxx::JsonRpcClient {
return call<std::vector<std::vector<double>>>("get-cloud-point");
}
[[nodiscard]] StereoCalibrationRPC get_stereo_calibration() {
return call<nlohmann::json>("get-stereo-calibration")
.get<StereoCalibrationRPC>();
}
[[nodiscard]] ImagePairRPC get_image_pair() {
return call<nlohmann::json>("get-image-pair").get<ImagePairRPC>();
}
/// @brief Call a method with no params.
template <typename ReturnType>
[[nodiscard]] ReturnType call(std::string_view name) {
return this->CallMethod<ReturnType>(id++, name.data());
return this->CallMethod<ReturnType>(id_++, name.data());
}
/// @brief Call a method with named params (JSON object keys → values).
template <typename ReturnType>
[[nodiscard]] ReturnType call(std::string_view name,
const nlohmann::json &params) {
jsonrpccxx::named_parameter named;
for (auto &[k, v] : params.items()) {
named[k] = v;
}
return this->CallMethodNamed<ReturnType>(id_++, name.data(), named);
}
~RpcClient() = default;
private:
int id{0};
int id_{0};
};
} // namespace cloud_point_rpc
} // namespace score

View File

@ -4,25 +4,25 @@
#pragma once
#include "export.h"
#include <string>
#include <vector>
#include "export.h"
namespace score {
class CRPC_EXPORT IRPCCoder {
public:
virtual ~IRPCCoder() = default;
virtual std::vector<char> decode(const std::string& encoded) = 0;
virtual std::string encode(const std::vector<char>& data) = 0;
public:
virtual ~IRPCCoder() = default;
virtual std::vector<char> decode(const std::string &encoded) = 0;
virtual std::string encode(const std::vector<char> &data) = 0;
};
class CRPC_EXPORT Base64RPCCoder final : public IRPCCoder {
public:
public:
Base64RPCCoder();
~Base64RPCCoder() override;
std::vector<char> decode(const std::string& encoded) override;
std::string encode(const std::vector<char>& data) override;
std::vector<char> decode(const std::string &encoded) override;
std::string encode(const std::vector<char> &data) override;
};
}
} // namespace score

View File

@ -0,0 +1,160 @@
// Created as part of Phase 1: stereo calibration and image-pair wire protocol.
#pragma once
#include "cloud_point_rpc/imageRpc.h"
#include "cloud_point_rpc/rpc_coder.hpp"
#include <nlohmann/json.hpp>
#include <stdexcept>
#include <string>
#include <vector>
namespace score {
// ---------------------------------------------------------------------------
// CameraCalib
// ---------------------------------------------------------------------------
/// @brief Intrinsic parameters for a single camera.
struct CameraCalib {
std::vector<double> camera_matrix; ///< 9 elements, row-major 3x3
std::vector<double> dist_coeffs; ///< 5 elements: k1 k2 p1 p2 k3
};
// ---------------------------------------------------------------------------
// StereoCalibrationRPC
// ---------------------------------------------------------------------------
/// @brief Full stereo rig calibration result.
struct StereoCalibrationRPC {
CameraCalib left;
CameraCalib right;
std::vector<double> rotation; ///< 9 elements, row-major 3x3
std::vector<double> translation; ///< 3 elements, metres (OpenCV convention)
int width{0};
int height{0};
};
// ---------------------------------------------------------------------------
// ImagePairRPC
// ---------------------------------------------------------------------------
/// @brief A synchronised stereo frame.
struct ImagePairRPC {
uint64_t frame{0};
ImageRPC left;
ImageRPC right;
};
// ---------------------------------------------------------------------------
// nlohmann ADL hooks — ImageRPC
// ---------------------------------------------------------------------------
inline std::string image_type_to_string(ImageRPC::Type t) {
switch (t) {
case ImageRPC::Type::BGR:
return "BGR";
case ImageRPC::Type::RGBA:
return "RGBA";
case ImageRPC::Type::DEPTH:
return "DEPTH";
default:
return "UNKNOWN";
}
}
inline ImageRPC::Type image_type_from_string(const std::string &s) {
if (s == "BGR")
return ImageRPC::Type::BGR;
if (s == "RGBA")
return ImageRPC::Type::RGBA;
if (s == "DEPTH")
return ImageRPC::Type::DEPTH;
throw std::runtime_error("Unknown ImageRPC type: " + s);
}
inline void to_json(nlohmann::json &j, const ImageRPC &img) {
Base64RPCCoder coder;
// ImageRPC::data is vector<unsigned char>; encode() needs vector<char>
const std::vector<char> as_char(img.data.begin(), img.data.end());
j = {{"width", img.width},
{"height", img.height},
{"type", image_type_to_string(img.type)},
{"data", coder.encode(as_char)}};
}
inline void from_json(const nlohmann::json &j, ImageRPC &img) {
j.at("width").get_to(img.width);
j.at("height").get_to(img.height);
img.type = image_type_from_string(j.at("type").get<std::string>());
Base64RPCCoder coder;
const auto decoded = coder.decode(j.at("data").get<std::string>());
img.data.assign(decoded.begin(), decoded.end());
}
// ---------------------------------------------------------------------------
// nlohmann ADL hooks — CameraCalib
// ---------------------------------------------------------------------------
inline void to_json(nlohmann::json &j, const CameraCalib &c) {
j = {{"camera_matrix", c.camera_matrix}, {"dist_coeffs", c.dist_coeffs}};
}
inline void from_json(const nlohmann::json &j, CameraCalib &c) {
j.at("camera_matrix").get_to(c.camera_matrix);
j.at("dist_coeffs").get_to(c.dist_coeffs);
if (c.camera_matrix.size() != 9) {
throw std::runtime_error("camera_matrix must have 9 elements, got " +
std::to_string(c.camera_matrix.size()));
}
if (c.dist_coeffs.size() != 5) {
throw std::runtime_error("dist_coeffs must have 5 elements, got " +
std::to_string(c.dist_coeffs.size()));
}
}
// ---------------------------------------------------------------------------
// nlohmann ADL hooks — StereoCalibrationRPC
// ---------------------------------------------------------------------------
inline void to_json(nlohmann::json &j, const StereoCalibrationRPC &s) {
j = {{"left", s.left},
{"right", s.right},
{"rotation", s.rotation},
{"translation", s.translation},
{"image_size", {{"width", s.width}, {"height", s.height}}}};
}
inline void from_json(const nlohmann::json &j, StereoCalibrationRPC &s) {
j.at("left").get_to(s.left);
j.at("right").get_to(s.right);
j.at("rotation").get_to(s.rotation);
j.at("translation").get_to(s.translation);
s.width = j.at("image_size").at("width").get<int>();
s.height = j.at("image_size").at("height").get<int>();
if (s.rotation.size() != 9) {
throw std::runtime_error("rotation must have 9 elements, got " +
std::to_string(s.rotation.size()));
}
if (s.translation.size() != 3) {
throw std::runtime_error("translation must have 3 elements, got " +
std::to_string(s.translation.size()));
}
}
// ---------------------------------------------------------------------------
// nlohmann ADL hooks — ImagePairRPC
// ---------------------------------------------------------------------------
inline void to_json(nlohmann::json &j, const ImagePairRPC &p) {
j = {{"frame", p.frame}, {"left", p.left}, {"right", p.right}};
}
inline void from_json(const nlohmann::json &j, ImagePairRPC &p) {
j.at("frame").get_to(p.frame);
j.at("left").get_to(p.left);
j.at("right").get_to(p.right);
}
} // namespace score

View File

@ -25,12 +25,22 @@ class CRPC_EXPORT RpcServer {
const nlohmann::json &)>;
using callback_t = rpc_string *(*)(rpc_string *);
public:
/// @note +1 method implicitly added: get-available-methods
RpcServer();
void register_method(const std::string &name, Handler handler);
void register_method(const std::string &name, callback_t handler);
uint64_t get_count() noexcept;
std::span<std::string_view> get_method_names() noexcept;
std::string_view get_method_name_by_id(uint64_t id) noexcept;
///@param request_str json rpc 2.0 formatted string
[[nodiscard]] std::string process(const std::string &request_str);
private:
std::vector<std::string_view> handler_names_;
std::map<std::string, Handler> handlers_;
};

View File

@ -34,4 +34,4 @@ template <NumericType T> T deserialize(const std::vector<uint8_t> &buffer) {
return *reinterpret_cast<const T *>(buffer.data());
}
} // namespace cloud_point_rpc
} // namespace score

View File

@ -1,8 +1,11 @@
#pragma once
#include "cloud_point_rpc/config.hpp"
#include <vector>
#include "cloud_point_rpc/rpc_dto.hpp"
#include "export.h"
#include <cstdint>
#include <vector>
namespace score {
class CRPC_EXPORT Service {
@ -13,8 +16,12 @@ class CRPC_EXPORT Service {
[[nodiscard]] std::vector<double> get_extrinsic_params() const;
[[nodiscard]] std::vector<std::vector<double>> get_cloud_point() const;
[[nodiscard]] StereoCalibrationRPC get_stereo_calibration() const;
[[nodiscard]] ImagePairRPC get_image_pair();
private:
TestData data_;
uint64_t frame_counter_{0};
};
} // namespace cloud_point_rpc
} // namespace score

View File

@ -1,11 +1,11 @@
#pragma once
#include "cloud_point_rpc/serialize.hpp"
#include "export.h"
#include "jsonrpccxx/iclientconnector.hpp"
#include <asio.hpp>
#include <cloud_point_rpc/tcp_read.hpp>
#include <glog/logging.h>
#include <string>
#include "export.h"
namespace score {
/**
* TCPConnector main purpose is to implement jsonrpccxx::IClientConnector Send
@ -43,4 +43,4 @@ class CRPC_EXPORT TCPConnector : public jsonrpccxx::IClientConnector {
asio::ip::tcp::socket socket_;
};
} // namespace cloud_point_rpc
} // namespace score

View File

@ -40,4 +40,4 @@ static inline std::string tcp_read(asio::ip::tcp::socket &socket,
return result;
}
} // namespace cloud_point_rpc
} // namespace score

View File

@ -132,19 +132,21 @@ class CRPC_EXPORT TcpServer {
private:
void handle_client(std::shared_ptr<asio::ip::tcp::socket> socket) {
LOG(INFO) << "Server reading from client...";
try {
auto payload = tcp_read(*socket, "TCPServer] ");
size_t payload_length = payload.size();
if (payload_length > 0) {
while (true) {
try {
auto payload = tcp_read(*socket, "TCPServer] ");
if (payload.empty())
break; // connection closed or read error
std::string response = processor_(payload);
response += "\n";
DLOG(INFO) << "Server sending response: " << response;
inplace_size_embedding(response);
asio::write(*socket, asio::buffer(response));
LOG(INFO) << "Server sent response";
} catch (const std::exception &e) {
LOG(WARNING) << "Client handling error: " << e.what();
break;
}
} catch (const std::exception &e) {
LOG(WARNING) << "Client handling error: " << e.what();
}
}

View File

@ -30,6 +30,8 @@ CRPC_EXPORT void crpc_init(const char* config_path);
CRPC_EXPORT void crpc_init_with_address(const char* ip, int port);
CRPC_EXPORT void crpc_deinit();
CRPC_EXPORT rpc_string* crpc_get_method_name_by_id(uint64_t id);
CRPC_EXPORT uint64_t crpc_get_methods_count();
CRPC_EXPORT void crpc_add_method(callback_t cb, rpc_string* name);
#ifdef __cplusplus

View File

@ -1,6 +1,6 @@
project('cloud_point_rpc', 'cpp',
version : '0.1',
default_options : ['warning_level=3', 'cpp_std=c++20'])
version : '0.2.0',
default_options : ['warning_level=3', 'cpp_std=c++23'])
# Dependencies
json_dep = dependency('nlohmann_json', fallback : ['nlohmann_json', 'nlohmann_json_dep'])

View File

@ -9,16 +9,23 @@ The system follows a layered architecture for JSON-RPC 2.0 communication over TC
│ Application / Unity │
│ (registers callbacks via C API or server_main.cpp) │
├──────────────────────────────────────────────────────┤
│ OpenCV Compute Layer (optional — requires opencv4) │
│ CloudPointClient (facade: connect, compute_cloud, │
│ write_ply) │
│ StereoRectifier (cv::stereoRectify + remap, Q mat) │
│ PointCloudBuilder (SGBM disparity, reproject, NaN │
│ filter) │
├──────────────────────────────────────────────────────┤
│ RPC Layer │
│ RpcServer (method dispatch, JSON-RPC 2.0 handling) │
│ RpcClient (typed method calls via jsonrpccxx) │
├──────────────────────────────────────────────────────┤
│ Service Layer │
│ Service (mock camera data: intrinsics, extrinsics, │
│ cloud points) │
│ stereo calibration, image pairs) │
├──────────────────────────────────────────────────────┤
│ Transport Layer │
│ TcpServer (accept loop, per-client threads) │
│ TcpServer (accept loop, per-client threads — loops) │
│ TCPConnector (client-side connector for jsonrpccxx) │
│ tcp_read (framed read with size prefix) │
├──────────────────────────────────────────────────────┤
@ -37,14 +44,14 @@ All C++ code lives in the `score` namespace. The `rpc/` git submodule provides `
The PlantUML diagram at `docs/communication_model.pu` (rendered as `docs/cm.png`) describes the interaction flow:
1. **Unity side** initializes a `CloudPointServer` which starts a `TcpServer` awaiting connections.
2. **Client side** initializes a `CloudPointClient` which creates a `TCPConnector` and connects to the server.
3. When a **Caller** requests data, the client serializes a JSON-RPC request, sends it over TCP, and waits for the response.
4. The server reads the request, dispatches it to `RpcServer`, which calls the registered handler (either a C++ lambda or a C callback from Unity).
5. The handler result is serialized back as a JSON-RPC response and sent over TCP.
6. The client receives the response and returns typed data to the caller.
1. **Unity side** initializes a `CloudPointServer` via `crpc_init()`, which starts a `TcpServer` awaiting connections.
2. **Client side** calls `CloudPointClient::connect()`, which opens a TCP connection and immediately fetches `get-stereo-calibration` once, initialising `StereoRectifier` with the returned intrinsics and geometry.
3. On each `compute_cloud()` call, the client fetches a synchronised image pair via `get-image-pair`, passes the decoded images through `StereoRectifier::rectify()`, then `PointCloudBuilder::build()` (SGBM disparity at 1/16 scale → `cv::reprojectImageTo3D` → NaN filter), and returns `std::expected<PointCloud, Error>` to the caller.
4. The server reads each request, dispatches it to `RpcServer`, which calls the registered handler (a C callback from Unity).
5. The Unity integration uses a **main-thread dispatcher**: RPC callbacks arrive on a C++ per-client thread and enqueue a `TaskCompletionSource` onto a `ConcurrentQueue`; the Unity `Update()` loop drains the queue and completes the task. A bounded timeout (~5000 ms) prevents deadlocks when the editor is paused.
6. The handler result is serialized back as a JSON-RPC response and sent over TCP; the per-client handler thread loops back to await the next request on the same connection.
The Unity integration model uses a **static queue**: the server enqueues tasks, and the Unity main loop dequeues and executes them, then sets the return value. This avoids calling Unity APIs from non-main threads.
See [docs/unity-integration.md](../docs/unity-integration.md) for the full Unity C# design spec.
## Wire framing
@ -153,6 +160,6 @@ Non-object JSON requests (arrays, strings, numbers, null) throw `nlohmann::json:
## Threading model
- **Server**: Accept thread + per-client handler threads (all `std::jthread`). Each client handler reads one request, processes it, sends the response, and exits. The server does not keep connections open for multiple round-trips per thread.
- **Server**: Accept thread + per-client handler threads (all `std::jthread`). Each client handler loops: it reads one request, processes it, sends the response, and then loops back to read the next request on the same connection. The connection remains open until the client disconnects (EOF) or an error occurs.
- **C API server** (`server_api.cpp`): The global `RpcServer` and `TcpServer` are guarded by `server_mtx`. The `rpc_string` garbage collector is guarded by `gc_mtx`.
- **Test API** (`test_api.cpp`): A `TestThread` runs a `std::jthread` with a condition-variable-driven loop that can auto-call registered methods at a configurable interval or process queued one-shot calls.

View File

@ -2,7 +2,7 @@
## Build system
The project uses **Meson** with **Ninja** and requires a **C++20** compiler (GCC or Clang). The root `meson.build` declares the project and dependencies; `src/meson.build` and `tests/meson.build` define build targets.
The project uses **Meson** with **Ninja** and requires a **C++23** compiler (GCC or Clang). The root `meson.build` declares the project and dependencies; `src/meson.build` and `tests/meson.build` define build targets.
### Dependencies

View File

@ -2,6 +2,8 @@
The C API allows Unity (or any C/C++ consumer) to embed the RPC server as a shared library, register custom RPC handlers as C function pointers, and manage the server lifecycle without touching C++ directly.
> **Unity C# design spec**: For the complete Unity-side implementation guide — including `rpc_string` ownership rules, IL2CPP callback pinning, main-thread dispatcher design, and shutdown ordering — see [docs/unity-integration.md](../docs/unity-integration.md). The C API described here is the C++ side of the interface; the C# side is the Unity project's responsibility.
Two C API surfaces exist:
| API | Header | Library | Purpose |

View File

@ -2,9 +2,9 @@
## What is this?
Cloud Point RPC is a **C++20 JSON-RPC 2.0** server and client implementation designed to bridge a C++ backend with a **Unity Scene** over TCP. The server exposes RPC methods that retrieve camera intrinsic/extrinsic parameters and point cloud data. A C API (`server_api.h`) allows Unity (or other C consumers) to embed the server, register custom RPC handlers, and manage the server lifecycle from native code.
Cloud Point RPC is a **C++23 JSON-RPC 2.0** server and client implementation designed to bridge a C++ backend with a **Unity Scene** over TCP. Unity serves stereo camera data (`get-stereo-calibration`, `get-image-pair`) over the embedded RPC server; the C++ `CloudPointClient` fetches calibration once on `connect()`, then on each `compute_cloud()` call retrieves a synchronised image pair, runs stereo rectification + SGBM disparity + `cv::reprojectImageTo3D`, and returns a filtered `PointCloud`. A C API (`server_api.h`) allows Unity to embed the server, register custom RPC handlers, and manage the server lifecycle from native code.
The project is a work in progress: the server side with C-API is implemented, while the client side with OpenCV integration is still planned (see README TODO).
The server side with C-API is fully implemented. The C++ stereo point-cloud client (`CloudPointClient`, `StereoRectifier`, `PointCloudBuilder`) is implemented. The remaining item is the Unity-side C# implementation; see [docs/unity-integration.md](../docs/unity-integration.md).
## Repository layout
@ -18,7 +18,9 @@ The project is a work in progress: the server side with C-API is implemented, wh
| `tests/` | GTest/GMock unit and integration tests (single `unit_tests` executable) |
| `rpc/` | Git submodule — [json-rpc-cxx](https://github.com/jsonrpcx/json-rpc-cxx) providing `jsonrpccxx` headers |
| `subprojects/` | Meson wrap dependencies (asio, nlohmann_json, glog, yaml-cpp, base64, gtest) |
| `docs/` | PlantUML communication model diagram |
| `include/cloud_point/` | OpenCV compute library headers: `StereoRectifier`, `PointCloudBuilder`, `CloudPointClient` |
| `src/cloud_point/` | OpenCV compute library implementation (optional, requires opencv4) |
| `docs/` | PlantUML communication model diagram and Unity integration design spec |
| `config.yml` | Sample server configuration (IP and port) |
| `Dockerfile` | Container image for the CLI client |
| `.gitea/workflows/test.yaml` | CI pipeline (build + test on push to master) |
@ -43,6 +45,13 @@ Run the interactive CLI client:
./build/src/cloud_point_rpc_cli config.yaml
```
CLI menu options (options 4 and 5 are hidden when built without opencv4):
| Option | Action |
|--------|--------|
| 4 | Compute point cloud — prints point count and bounding box |
| 5 | Compute point cloud and save to `output.ply` |
Run all tests:
```bash
@ -63,4 +72,6 @@ For Windows build instructions and Docker usage, see [Build & Testing](build-and
- **Namespace**: All C++ code lives in `score` (renamed from `cloud_point_rpc` early in development).
- **Wire framing**: Every TCP message is prefixed with an 8-byte little-endian `uint64_t` payload size, then the JSON-RPC payload follows. See [Architecture → Wire framing](architecture.md#wire-framing).
- **Two server entrypoints**: `server_main.cpp` is a standalone executable with mock data; `server_api.cpp` provides the embeddable C API that Unity uses to start the server and register callbacks.
- **Base64**: Camera parameter arrays and point cloud data are Base64-encoded for ASCII-safe transport over JSON. Encoding/decoding is done on the Unity side per API.md.
- **Base64**: Image pixel payloads (`get-image-pair`) are Base64-encoded for ASCII-safe transport over JSON. Calibration arrays are plain JSON doubles — not Base64. Encoding on the Unity side; decoding in `Base64RPCCoder` on the client side per API.md.
- **Persistent connections**: `TcpServer::handle_client` loops per connection — multiple RPC round-trips share one TCP connection without reconnecting.
- **std::expected**: The `cloud_point` compute library uses `std::expected<PointCloud, Error>` (C++23) as its return type. Callers check `has_value()` before accessing the result.

View File

@ -57,37 +57,87 @@ Source: `src/rpc_server.cpp` — `RpcServer::process()` and `create_error()`.
## Methods
### `get-intrinsic-params`
### `get-stereo-calibration`
Retrieves intrinsic camera parameters as a flat 3×3 matrix (row-major, 9 doubles).
Returns full stereo rig calibration: intrinsics for both cameras, stereo rotation and translation, and sensor resolution. Called once by `CloudPointClient::connect()`.
**Request:**
```json
{"jsonrpc": "2.0", "method": "get-intrinsic-params", "id": 1}
{"jsonrpc": "2.0", "method": "get-stereo-calibration", "id": 1}
```
**Response:**
```json
{"jsonrpc": "2.0", "result": <base64-encoded-array>, "id": 1}
{
"left": { "camera_matrix": [9 doubles], "dist_coeffs": [5 doubles] },
"right": { "camera_matrix": [9 doubles], "dist_coeffs": [5 doubles] },
"rotation": [9 doubles],
"translation": [3 doubles],
"image_size": { "width": 640, "height": 480 }
}
```
Result type: `vector<double>` (size 9), Base64-encoded.
All matrices are row-major. `camera_matrix` is the 3×3 intrinsic matrix; `rotation` is the 3×3 left-to-right rotation (OpenCV convention); `translation` is in metres. Calibration arrays are **plain JSON doubles — not Base64**.
### `get-extrinsic-params`
Test mock defaults: fx=fy=800, cx=320, cy=240, zero distortion, R=I, T=[-0.06, 0, 0], 640×480.
Retrieves extrinsic camera parameters as a flat 4×4 matrix (row-major, 16 doubles).
---
### `get-image-pair`
Returns a synchronised stereo frame as two base64-encoded images. Called on each `CloudPointClient::compute_cloud()`.
**Request:**
```json
{"jsonrpc": "2.0", "method": "get-extrinsic-params", "id": 2}
{"jsonrpc": "2.0", "method": "get-image-pair", "id": 2}
```
**Response:**
```json
{"jsonrpc": "2.0", "result": <base64-encoded-array>, "id": 2}
{
"frame": 42,
"left": { "width": 640, "height": 480, "type": "BGR", "data": "<base64>" },
"right": { "width": 640, "height": 480, "type": "BGR", "data": "<base64>" }
}
```
Result type: `vector<double>` (size 16), Base64-encoded.
`data` is Base64-encoded raw pixel bytes (standard alphabet, no line breaks). `type` is `"BGR"` (3 ch), `"RGBA"` (4 ch), or `"DEPTH"` (1 ch float32). Unity must flip GPU readback vertically before encoding (wire format is top-left-origin row-major).
DTOs: `StereoCalibrationDto` and `ImagePairDto` in `include/cloud_point_rpc/rpc_dto.hpp`. Decoded client-side via `Base64RPCCoder`.
---
### `get-intrinsic-params` *(legacy)*
Retrieves left-camera intrinsic parameters as a flat 3×3 matrix (row-major, 9 doubles). Not used by `CloudPointClient`; kept for backward compatibility.
**Request:**
```json
{"jsonrpc": "2.0", "method": "get-intrinsic-params", "id": 3}
```
**Response:**
```json
{"jsonrpc": "2.0", "result": [fx, 0, cx, 0, fy, cy, 0, 0, 1], "id": 3}
```
Result: 9 plain JSON doubles. **Not Base64.**
### `get-extrinsic-params` *(legacy)*
Retrieves left-camera extrinsic matrix as a flat 4×4 matrix (row-major, 16 doubles). Not used by `CloudPointClient`; kept for backward compatibility.
**Request:**
```json
{"jsonrpc": "2.0", "method": "get-extrinsic-params", "id": 4}
```
**Response:**
```json
{"jsonrpc": "2.0", "result": [16 doubles], "id": 4}
```
Result: 16 plain JSON doubles. **Not Base64.**
### `get-cloud-point`
@ -146,7 +196,7 @@ Source: `src/rpc_server.cpp` — `register_method(name, callback_t)` overload.
- `encode(vector<char>) → string` and `decode(string) → vector<char>`.
- Includes overflow protection: throws `std::length_error` if input exceeds safe size limits.
> Per `API.md`, Base64 encoding/decoding of camera data is expected to be done on the **Unity side**. The `Base64RPCCoder` class is available for C++ consumers but is not used by the mock server's response path.
> Per `API.md`: **only image pixel data** (`get-image-pair` `data` fields) uses Base64. Calibration arrays (`camera_matrix`, `dist_coeffs`, `rotation`, `translation`) and legacy params are plain JSON doubles — never Base64. Unity encodes images before sending; the C++ client decodes them via `Base64RPCCoder`.
## Client-side usage

View File

@ -3,6 +3,13 @@
#include <glog/logging.h>
#include <string>
#ifdef HAVE_CLOUD_POINT_COMPUTE
#include "cloud_point/cloud_point_client.hpp"
#include <algorithm>
#include <cmath>
#include <limits>
#endif
namespace score {
void print_menu(std::ostream &output) {
@ -10,6 +17,8 @@ void print_menu(std::ostream &output) {
output << "1. get-intrinsic-params" << std::endl;
output << "2. get-extrinsic-params" << std::endl;
output << "3. get-cloud-point" << std::endl;
output << "4. compute-cloud" << std::endl;
output << "5. compute-cloud + save PLY" << std::endl;
output << "0. Exit" << std::endl;
output << "Select an option: ";
}
@ -48,34 +57,90 @@ int run_cli(std::istream &input, std::ostream &output, const std::string &ip,
if (choice == "0")
break;
std::string method;
if (choice == "1") {
method = "get-intrinsic-params";
} else if (choice == "2") {
method = "get-extrinsic-params";
} else if (choice == "3") {
method = "get-cloud-point";
if (choice == "1" || choice == "2" || choice == "3") {
std::string method;
if (choice == "1") {
method = "get-intrinsic-params";
} else if (choice == "2") {
method = "get-extrinsic-params";
} else {
method = "get-cloud-point";
}
try {
if (method == "get-intrinsic-params") {
auto response = client.get_intrinsic_params();
output << vector_to_string(response);
}
if (method == "get-extrinsic-params") {
auto response = client.get_extrinsic_params();
output << vector_to_string(response);
}
if (method == "get-cloud-point") {
auto response = client.get_cloud_point();
output << vector_to_string(response);
}
} catch (const std::exception &e) {
output << "\nRPC Error: " << e.what() << std::endl;
}
} else if (choice == "4" || choice == "5") {
#ifdef HAVE_CLOUD_POINT_COMPUTE
try {
CloudPointClient cpc(ip, port, StereoAlgorithmType::GPU);
cpc.connect();
auto result = cpc.compute_cloud();
if (!result) {
output << "Error: " << result.error().message
<< std::endl;
} else {
const auto &cloud = *result;
const auto valid = cloud.valid_points();
float x_min = std::numeric_limits<float>::max();
float x_max = std::numeric_limits<float>::lowest();
float y_min = std::numeric_limits<float>::max();
float y_max = std::numeric_limits<float>::lowest();
float z_min = std::numeric_limits<float>::max();
float z_max = std::numeric_limits<float>::lowest();
for (const auto &pt : valid) {
x_min = std::min(x_min, pt[0]);
x_max = std::max(x_max, pt[0]);
y_min = std::min(y_min, pt[1]);
y_max = std::max(y_max, pt[1]);
z_min = std::min(z_min, pt[2]);
z_max = std::max(z_max, pt[2]);
}
output << "Cloud: " << cloud.width << "x"
<< cloud.height << " valid_pts=" << valid.size()
<< "\n";
if (!valid.empty()) {
output << " bbox x=[" << x_min << "," << x_max
<< "]"
<< " y=[" << y_min << "," << y_max << "]"
<< " z=[" << z_min << "," << z_max << "]\n";
}
if (choice == "5") {
output << "PLY output path: ";
std::string path;
if (input >> path) {
write_ply(cloud, path);
output << "Saved " << valid.size()
<< " points to " << path << "\n";
}
}
}
} catch (const std::exception &e) {
output << "Error: " << e.what() << std::endl;
}
#else
output << "OpenCV support not built" << std::endl;
#endif
} else {
output << "Invalid option: " << choice << std::endl;
continue;
}
try {
if (method == "get-intrinsic-params") {
auto response = client.get_intrinsic_params();
output << vector_to_string(response);
}
if (method == "get-extrinsic-params") {
auto response = client.get_extrinsic_params();
output << vector_to_string(response);
}
if (method == "get-cloud-point") {
auto response = client.get_cloud_point();
output << vector_to_string(response);
}
} catch (const std::exception &e) {
output << "\nRPC Error: " << e.what() << std::endl;
}
}
} catch (const std::exception &e) {
@ -86,4 +151,4 @@ int run_cli(std::istream &input, std::ostream &output, const std::string &ip,
return 0;
}
} // namespace cloud_point_rpc
} // namespace score

View File

@ -0,0 +1,115 @@
#include "cloud_point/cloud_point_client.hpp"
#include "cloud_point/imageFactory.h"
#include "cloud_point_rpc/rpc_client.hpp"
#include "cloud_point_rpc/tcp_connector.hpp"
#include <fstream>
#include <jsonrpccxx/common.hpp>
#include <opencv2/imgproc.hpp>
namespace score {
CloudPointClient::CloudPointClient(std::string ip, int port,
StereoAlgorithmType algo,
PointCloudBuilder::Options opts,
int num_disparities)
: ip_(std::move(ip)), port_(port), algo_(algo), opts_(opts),
num_disparities_(num_disparities) {}
CloudPointClient::~CloudPointClient() = default;
void CloudPointClient::connect() {
connector_ = std::make_unique<TCPConnector>(ip_, port_);
client_ = std::make_unique<RpcClient>(*connector_);
const auto calib_rpc = client_->get_stereo_calibration();
const auto calib = StereoRectifier::Calibration::from_rpc(calib_rpc);
rectifier_ = std::make_unique<StereoRectifier>(calib);
matcher_ = StereoMatcherFactory::create(algo_, num_disparities_);
builder_ = std::make_unique<PointCloudBuilder>(rectifier_->q(), opts_);
}
bool CloudPointClient::connected() const noexcept {
return connector_ != nullptr && client_ != nullptr &&
rectifier_ != nullptr && matcher_ != nullptr && builder_ != nullptr;
}
std::expected<PointCloud, CloudPointClient::Error>
CloudPointClient::compute_cloud() {
if (!connected()) {
return std::unexpected(Error{"not connected — call connect() first"});
}
try {
// Fetch image pair.
const auto pair = client_->get_image_pair();
// Decode raw bytes into cv::Mat via ImageFactory.
auto left_img = ImageFactory::create(pair.left);
auto right_img = ImageFactory::create(pair.right);
// Convert to grayscale according to image type.
auto to_gray = [](ImageRPC::Type type,
cv::Mat mat) -> std::expected<cv::Mat, Error> {
cv::Mat gray;
switch (type) {
case ImageRPC::Type::BGR:
cv::cvtColor(mat, gray, cv::COLOR_BGR2GRAY);
return gray;
case ImageRPC::Type::RGBA:
cv::cvtColor(mat, gray, cv::COLOR_RGBA2GRAY);
return gray;
case ImageRPC::Type::DEPTH:
return std::unexpected(
Error{"depth images not supported by stereo pipeline"});
default:
return std::unexpected(Error{"unknown image type"});
}
};
auto left_gray_r = to_gray(pair.left.type, left_img.get());
if (!left_gray_r)
return std::unexpected(left_gray_r.error());
auto right_gray_r = to_gray(pair.right.type, right_img.get());
if (!right_gray_r)
return std::unexpected(right_gray_r.error());
const cv::Mat &left_gray = *left_gray_r;
const cv::Mat &right_gray = *right_gray_r;
// Rectify, compute disparity, reproject.
auto [rect_left, rect_right] =
rectifier_->rectify(left_gray, right_gray);
const auto disparity = matcher_->compute(rect_left, rect_right);
return builder_->build(disparity);
} catch (const jsonrpccxx::JsonRpcException &e) {
return std::unexpected(Error{e.what()});
} catch (const std::exception &e) {
return std::unexpected(Error{e.what()});
}
}
// ---------------------------------------------------------------------------
// PLY helper
// ---------------------------------------------------------------------------
void write_ply(const PointCloud &cloud, const std::string &path) {
const auto valid = cloud.valid_points();
std::ofstream out(path);
out << "ply\n"
<< "format ascii 1.0\n"
<< "element vertex " << valid.size() << "\n"
<< "property float x\n"
<< "property float y\n"
<< "property float z\n"
<< "end_header\n";
for (const auto &pt : valid) {
out << pt[0] << " " << pt[1] << " " << pt[2] << "\n";
}
}
} // namespace score

View File

@ -0,0 +1,16 @@
#include "cloud_point/cpu_stereo_matcher.hpp"
namespace score {
CpuStereoMatcher::CpuStereoMatcher(int min_disparity, int num_disparities,
int block_size) {
sgbm_ = cv::StereoSGBM::create(min_disparity, num_disparities, block_size);
}
cv::Mat CpuStereoMatcher::compute(const cv::Mat &left, const cv::Mat &right) {
cv::Mat disparity;
sgbm_->compute(left, right, disparity);
return disparity;
}
} // namespace score

View File

@ -0,0 +1,43 @@
#include "cloud_point/gpu_stereo_matcher.hpp"
#include <stdexcept>
#ifdef HAVE_OPENCV_CUDA
#include <opencv2/cudaimgproc.hpp>
#include <opencv2/cudastereo.hpp>
#endif
namespace score {
GpuStereoMatcher::GpuStereoMatcher(int min_disparity, int num_disparities,
int block_size) {
#ifdef HAVE_OPENCV_CUDA
if (cv::cuda::getCudaEnabledDeviceCount() == 0) {
throw std::runtime_error("No CUDA devices available");
}
sgm_ =
cv::cuda::createStereoSGM(min_disparity, num_disparities, block_size);
#else
(void)min_disparity;
(void)num_disparities;
(void)block_size;
throw std::runtime_error("OpenCV CUDA modules not available in this build");
#endif
}
cv::Mat GpuStereoMatcher::compute(const cv::Mat &left, const cv::Mat &right) {
#ifdef HAVE_OPENCV_CUDA
cv::cuda::GpuMat d_left(left);
cv::cuda::GpuMat d_right(right);
cv::cuda::GpuMat d_disparity;
sgm_->compute(d_left, d_right, d_disparity);
cv::Mat disparity;
d_disparity.download(disparity);
return disparity;
#else
(void)left;
(void)right;
throw std::runtime_error("OpenCV CUDA modules not available in this build");
#endif
}
} // namespace score

18
src/cloud_point/image.cpp Normal file
View File

@ -0,0 +1,18 @@
//
// Created by vptyp on 12.03.2026.
//
#include "cloud_point/image.h"
namespace score {
Image::Image() {
// no work
}
Image::Image(const cv::Mat &image) { this->data_ = image; }
Image::~Image() = default;
cv::Mat Image::get() { return this->data_; }
} // namespace score

View File

@ -0,0 +1,50 @@
opencv_dep = dependency('opencv4',
fallback: ['libopencv', 'libopencv4'],
required: false)
if not opencv_dep.found()
message('\'opencv\' was not found. Try install libopencv-dev or similar package on your system')
message('cloud_point_compute library removed from compilation')
subdir_done()
endif
cxx = meson.get_compiler('cpp')
opencv_cuda_available = cxx.has_header('opencv2/cudastereo.hpp', dependencies: opencv_dep)
cloud_point_sources = files(
'image.cpp',
'cpu_stereo_matcher.cpp',
'gpu_stereo_matcher.cpp',
'stereo_matcher_factory.cpp',
'stereo_rectifier.cpp',
'point_cloud_builder.cpp',
'cloud_point_client.cpp',
'scared_dataset_loader.cpp',
)
cpc_deps = [ cloud_point_rpc_dep, opencv_dep ]
cpp_args = []
if opencv_cuda_available
cpp_args += '-DHAVE_OPENCV_CUDA'
endif
cloud_point_compute_lib = shared_library('cloud_point_compute',
sources: cloud_point_sources,
include_directories: inc,
dependencies: cpc_deps,
cpp_args: cpp_args,
)
cloud_point_compute_dep = declare_dependency(
include_directories: inc,
link_with: cloud_point_compute_lib,
dependencies: cpc_deps
)
executable(
'scared_dataset_server',
'scared_dataset_server.cpp',
dependencies: [cloud_point_compute_dep],
install: true,
)

View File

@ -0,0 +1,107 @@
#include "cloud_point/point_cloud_builder.hpp"
#include <cmath>
#include <limits>
#include <stdexcept>
#include <opencv2/calib3d.hpp>
namespace score {
// ---------------------------------------------------------------------------
// PointCloud::valid_points
// ---------------------------------------------------------------------------
std::vector<std::array<float, 3>> PointCloud::valid_points() const {
std::vector<std::array<float, 3>> pts;
pts.reserve(static_cast<size_t>(width * height));
for (int i = 0; i < height * width; ++i) {
const float x = data[static_cast<size_t>(i) * 3];
const float y = data[static_cast<size_t>(i) * 3 + 1];
const float z = data[static_cast<size_t>(i) * 3 + 2];
if (!std::isnan(x) && !std::isnan(y) && !std::isnan(z)) {
pts.push_back({x, y, z});
}
}
return pts;
}
// ---------------------------------------------------------------------------
// PointCloudBuilder constructor
// ---------------------------------------------------------------------------
PointCloudBuilder::PointCloudBuilder(cv::Mat q, Options opts)
: q_(std::move(q)), opts_(opts) {
if (q_.rows != 4 || q_.cols != 4 || q_.type() != CV_64F) {
throw std::invalid_argument("Q matrix must be 4x4 CV_64F, got " +
std::to_string(q_.rows) + "x" +
std::to_string(q_.cols) +
" type=" + std::to_string(q_.type()));
}
}
// ---------------------------------------------------------------------------
// PointCloudBuilder::build
// ---------------------------------------------------------------------------
PointCloud PointCloudBuilder::build(const cv::Mat &disparity) const {
cv::Mat disp32;
if (disparity.type() == CV_16S) {
// SGBM stores disparity in fixed-point with a factor of 16
disparity.convertTo(disp32, CV_32F, 1.0 / 16.0);
} else if (disparity.type() == CV_32F) {
disp32 = disparity;
} else {
throw std::invalid_argument(
"Disparity must be CV_16S or CV_32F, got type=" +
std::to_string(disparity.type()));
}
// Reproject to 3-D. handleMissingValues=false is used intentionally:
// OpenCV 4.6 incorrectly marks all CV_32F pixels as "missing" when
// handleMissingValues=true (empirically verified). Our own filter loop
// below already covers all sentinel cases (d≤0, |z|≥10000, depth range).
cv::Mat xyz;
cv::reprojectImageTo3D(disp32, xyz, q_, /*handleMissingValues=*/false);
PointCloud cloud;
cloud.width = disparity.cols;
cloud.height = disparity.rows;
cloud.data.resize(static_cast<size_t>(cloud.width) *
static_cast<size_t>(cloud.height) * 3u);
const float kNaN = std::numeric_limits<float>::quiet_NaN();
for (int r = 0; r < cloud.height; ++r) {
for (int c = 0; c < cloud.width; ++c) {
const auto &pt = xyz.at<cv::Vec3f>(r, c);
const float z = pt[2];
const float d = disp32.at<float>(r, c);
const size_t idx =
(static_cast<size_t>(r) * static_cast<size_t>(cloud.width) +
static_cast<size_t>(c)) *
3u;
// Mark invalid when: disparity ≤ 0, OpenCV sentinel |z|≥10000,
// or depth outside the user-specified range.
const bool invalid = (d <= 0.0f) || (std::abs(z) >= 10000.0f) ||
(z < opts_.min_depth_m) ||
(z > opts_.max_depth_m);
if (invalid) {
cloud.data[idx] = kNaN;
cloud.data[idx + 1] = kNaN;
cloud.data[idx + 2] = kNaN;
} else {
cloud.data[idx] = pt[0];
cloud.data[idx + 1] = pt[1];
cloud.data[idx + 2] = z;
}
}
}
return cloud;
}
} // namespace score

View File

@ -0,0 +1,127 @@
#include "cloud_point/scared_dataset_loader.hpp"
#include <cstring>
#include <stdexcept>
#include <vector>
#include <glog/logging.h>
#include <opencv2/core.hpp>
#include <opencv2/imgcodecs.hpp>
namespace score {
namespace {
/// Flatten a cv::Mat (row-major) to a std::vector<double>.
std::vector<double> mat_to_vec(const cv::Mat &m) {
cv::Mat d64;
m.convertTo(d64, CV_64F);
std::vector<double> v(static_cast<size_t>(d64.total()));
std::copy(d64.begin<double>(), d64.end<double>(), v.begin());
return v;
}
/// Load a PNG from @p path and encode as BGR ImageRPC.
/// Reads with IMREAD_COLOR (→ BGR 8-bit); throws on failure.
ImageRPC load_bgr_image(const std::string &path) {
const cv::Mat img = cv::imread(path, cv::IMREAD_COLOR);
if (img.empty()) {
throw std::runtime_error(
"ScaredDatasetLoader: cannot read image: " + path);
}
ImageRPC rpc;
rpc.width = img.cols;
rpc.height = img.rows;
rpc.type = ImageRPC::Type::BGR;
const size_t sz = static_cast<size_t>(img.cols) * img.rows * 3;
rpc.data.resize(sz);
std::memcpy(rpc.data.data(), img.data, sz);
return rpc;
}
} // namespace
ScaredDatasetLoader::ScaredDatasetLoader(const std::string &keyframe_dir) {
const std::string yaml_path = keyframe_dir + "/endoscope_calibration.yaml";
const std::string left_path = keyframe_dir + "/Left_Image.png";
const std::string right_path = keyframe_dir + "/Right_Image.png";
LOG(INFO) << "ScaredDatasetLoader: loading calibration from " << yaml_path;
cv::FileStorage fs(yaml_path, cv::FileStorage::READ);
if (!fs.isOpened()) {
throw std::runtime_error(
"ScaredDatasetLoader: cannot open calibration YAML: " + yaml_path);
}
cv::Mat M1, D1, M2, D2, R, T;
fs["M1"] >> M1;
fs["D1"] >> D1;
fs["M2"] >> M2;
fs["D2"] >> D2;
fs["R"] >> R;
fs["T"] >> T;
fs.release();
if (M1.empty() || D1.empty() || M2.empty() || D2.empty() || R.empty() ||
T.empty()) {
throw std::runtime_error(
"ScaredDatasetLoader: missing node in calibration YAML: " +
yaml_path);
}
// Load images first so we can populate width/height from actual dimensions.
LOG(INFO) << "ScaredDatasetLoader: loading images";
left_image_ = load_bgr_image(left_path);
right_image_ = load_bgr_image(right_path);
if (left_image_.width != right_image_.width ||
left_image_.height != right_image_.height) {
throw std::runtime_error(
"ScaredDatasetLoader: stereo pair dimension mismatch: left " +
std::to_string(left_image_.width) + "x" +
std::to_string(left_image_.height) + " vs right " +
std::to_string(right_image_.width) + "x" +
std::to_string(right_image_.height));
}
calib_.width = left_image_.width;
calib_.height = left_image_.height;
calib_.left.camera_matrix = mat_to_vec(M1);
calib_.left.dist_coeffs = mat_to_vec(D1);
calib_.right.camera_matrix = mat_to_vec(M2);
calib_.right.dist_coeffs = mat_to_vec(D2);
calib_.rotation = mat_to_vec(R);
// T is stored as 1x3 in millimetres; convert to metres.
const auto t_mm = mat_to_vec(T);
calib_.translation.resize(3);
calib_.translation[0] = t_mm[0] / 1000.0;
calib_.translation[1] = t_mm[1] / 1000.0;
calib_.translation[2] = t_mm[2] / 1000.0;
LOG(INFO) << "ScaredDatasetLoader: T(mm)=["
<< t_mm[0] << "," << t_mm[1] << "," << t_mm[2]
<< "] -> T(m)=["
<< calib_.translation[0] << ","
<< calib_.translation[1] << ","
<< calib_.translation[2] << "]";
LOG(INFO) << "ScaredDatasetLoader: image size "
<< calib_.width << "x" << calib_.height;
}
const StereoCalibrationRPC &
ScaredDatasetLoader::calibration() const noexcept {
return calib_;
}
ImagePairRPC ScaredDatasetLoader::image_pair(uint64_t frame) const {
ImagePairRPC pair;
pair.frame = frame;
pair.left = left_image_;
pair.right = right_image_;
return pair;
}
} // namespace score

View File

@ -0,0 +1,75 @@
/// @file scared_dataset_server.cpp
/// @brief RPC server backed by a SCARED dataset keyframe directory.
///
/// Usage: scared_dataset_server <keyframe_dir> [port]
///
/// Serves get-stereo-calibration and get-image-pair matching the wire
/// protocol consumed by CloudPointClient. The same images are returned on
/// every get-image-pair call (single-keyframe source); the frame counter
/// increments so the client can detect stale frames if desired.
#include "cloud_point/scared_dataset_loader.hpp"
#include "cloud_point_rpc/rpc_dto.hpp"
#include "cloud_point_rpc/rpc_server.hpp"
#include "cloud_point_rpc/tcp_server.hpp"
#include <atomic>
#include <glog/logging.h>
#include <nlohmann/json.hpp>
#include <string>
using json = nlohmann::json;
int main(int argc, char *argv[]) {
google::InitGoogleLogging(argv[0]);
google::InstallFailureSignalHandler();
FLAGS_alsologtostderr = 1;
if (argc < 2) {
LOG(ERROR) << "Usage: " << argv[0] << " <keyframe_dir> [port]";
return 1;
}
const std::string keyframe_dir = argv[1];
LOG(INFO) << "SCARED dataset server starting";
LOG(INFO) << " keyframe_dir = " << keyframe_dir;
try {
const int port = (argc >= 3) ? std::stoi(argv[2]) : 8080;
LOG(INFO) << " port = " << port;
score::ScaredDatasetLoader loader(keyframe_dir);
std::atomic<uint64_t> frame_counter{0};
score::RpcServer rpc_server;
rpc_server.register_method(
"get-stereo-calibration", [&](const json &) -> json {
json j;
score::to_json(j, loader.calibration());
return j;
});
rpc_server.register_method(
"get-image-pair", [&](const json &) -> json {
json j;
score::to_json(j, loader.image_pair(frame_counter++));
return j;
});
score::TcpServer server(
"0.0.0.0", port,
[&](const std::string &request) {
return rpc_server.process(request);
});
server.start();
LOG(INFO) << "SCARED dataset server ready on port " << port;
server.join();
} catch (const std::exception &e) {
LOG(ERROR) << "Fatal error: " << e.what();
return 1;
}
return 0;
}

View File

@ -0,0 +1,33 @@
#include "cloud_point/stereo_matcher_factory.hpp"
#include "cloud_point/cpu_stereo_matcher.hpp"
#include "cloud_point/gpu_stereo_matcher.hpp"
#include <glog/logging.h>
#include <stdexcept>
#include <string>
namespace score {
std::unique_ptr<IStereoMatcher>
StereoMatcherFactory::create(StereoAlgorithmType type, int num_disparities) {
if (num_disparities <= 0 || num_disparities % 16 != 0) {
throw std::invalid_argument(
"StereoMatcherFactory: num_disparities must be a positive "
"multiple of 16, got " +
std::to_string(num_disparities));
}
switch (type) {
case StereoAlgorithmType::CPU:
return std::make_unique<CpuStereoMatcher>(0, num_disparities);
case StereoAlgorithmType::GPU:
try {
return std::make_unique<GpuStereoMatcher>();
} catch (const std::exception &e) {
LOG(WARNING) << "GPU stereo matcher unavailable: " << e.what()
<< ". Falling back to CPU.";
return std::make_unique<CpuStereoMatcher>(0, num_disparities);
}
}
return nullptr;
}
} // namespace score

View File

@ -0,0 +1,90 @@
#include "cloud_point/stereo_rectifier.hpp"
#include <stdexcept>
#include <string>
#include <opencv2/calib3d.hpp>
#include <opencv2/imgproc.hpp>
#include "cloud_point/matrixFactory.h"
namespace score {
// ---------------------------------------------------------------------------
// Internal helpers
// ---------------------------------------------------------------------------
namespace {
void require_mat(const cv::Mat &mat, int rows, int cols, int type,
const char *name) {
if (mat.rows != rows || mat.cols != cols || mat.type() != type) {
throw std::invalid_argument(
std::string(name) + " must be " + std::to_string(rows) + "x" +
std::to_string(cols) + " CV_64F (type " + std::to_string(CV_64F) +
"), got " + std::to_string(mat.rows) + "x" +
std::to_string(mat.cols) + " type=" + std::to_string(mat.type()));
}
}
} // namespace
// ---------------------------------------------------------------------------
// StereoRectifier::Calibration::from_rpc
// ---------------------------------------------------------------------------
StereoRectifier::Calibration
StereoRectifier::Calibration::from_rpc(const StereoCalibrationRPC &rpc) {
Calibration calib;
calib.k_left = CameraMatrixFactory::create<3, 3>(rpc.left.camera_matrix);
calib.d_left = CameraMatrixFactory::create<1, 5>(rpc.left.dist_coeffs);
calib.k_right = CameraMatrixFactory::create<3, 3>(rpc.right.camera_matrix);
calib.d_right = CameraMatrixFactory::create<1, 5>(rpc.right.dist_coeffs);
calib.r = CameraMatrixFactory::create<3, 3>(rpc.rotation);
calib.t = CameraMatrixFactory::create<3, 1>(rpc.translation);
calib.image_size = cv::Size(rpc.width, rpc.height);
return calib;
}
// ---------------------------------------------------------------------------
// StereoRectifier constructor
// ---------------------------------------------------------------------------
StereoRectifier::StereoRectifier(const Calibration &calib) {
require_mat(calib.k_left, 3, 3, CV_64F, "k_left");
require_mat(calib.k_right, 3, 3, CV_64F, "k_right");
require_mat(calib.d_left, 1, 5, CV_64F, "d_left");
require_mat(calib.d_right, 1, 5, CV_64F, "d_right");
require_mat(calib.r, 3, 3, CV_64F, "r");
require_mat(calib.t, 3, 1, CV_64F, "t");
cv::Mat R1, R2, P1, P2;
cv::stereoRectify(calib.k_left, calib.d_left, calib.k_right, calib.d_right,
calib.image_size, calib.r, calib.t, R1, R2, P1, P2, q_,
cv::CALIB_ZERO_DISPARITY, /*alpha=*/0);
cv::initUndistortRectifyMap(calib.k_left, calib.d_left, R1, P1,
calib.image_size, CV_16SC2, map_lx_, map_ly_);
cv::initUndistortRectifyMap(calib.k_right, calib.d_right, R2, P2,
calib.image_size, CV_16SC2, map_rx_, map_ry_);
}
// ---------------------------------------------------------------------------
// StereoRectifier::rectify
// ---------------------------------------------------------------------------
std::pair<cv::Mat, cv::Mat>
StereoRectifier::rectify(const cv::Mat &left, const cv::Mat &right) const {
cv::Mat rect_left, rect_right;
cv::remap(left, rect_left, map_lx_, map_ly_, cv::INTER_LINEAR);
cv::remap(right, rect_right, map_rx_, map_ry_, cv::INTER_LINEAR);
return {rect_left, rect_right};
}
// ---------------------------------------------------------------------------
// StereoRectifier::q
// ---------------------------------------------------------------------------
const cv::Mat &StereoRectifier::q() const noexcept { return q_; }
} // namespace score

View File

@ -26,7 +26,7 @@ int main(int argc, char *argv[]) {
try {
auto config = score::ConfigLoader::load(config_path);
return score::run_cli(std::cin, std::cout, config.server.ip,
config.server.port);
config.server.port);
} catch (const std::exception &e) {
std::cerr << "Failed to start CLI: " << e.what() << std::endl;
return 1;

View File

@ -1,7 +1,10 @@
add_project_arguments('-DCRPC_SERVER_API_EXPORT', language: 'cpp')
deps = [json_dep, thread_dep, glog_dep, yaml_dep, asio_dep, base64_dep]
cloud_point_rpc_sources = files(
'rpc_coder.cpp',
'rpc_dto.cpp',
'rpc_server.cpp',
'server_api.cpp',
'service.cpp',
@ -11,7 +14,7 @@ libcloud_point_rpc = shared_library(
'cloud_point_rpc',
cloud_point_rpc_sources,
include_directories: inc,
dependencies: [json_dep, thread_dep, glog_dep, yaml_dep, asio_dep, base64_dep],
dependencies: deps,
install: true,
install_rpath: '$ORIGIN',
)
@ -19,7 +22,7 @@ libcloud_point_rpc = shared_library(
cloud_point_rpc_dep = declare_dependency(
include_directories: inc,
link_with: libcloud_point_rpc,
dependencies: [json_dep, glog_dep, yaml_dep, asio_dep, base64_dep],
dependencies: deps,
)
# Test lib
@ -37,18 +40,30 @@ cloud_point_rpc_test_dep = declare_dependency(
dependencies: [cloud_point_rpc_dep],
)
subdir('cloud_point')
# CLI lib — links cloud_point_compute when OpenCV is available so that
# options 4 and 5 (compute-cloud) are compiled in.
cli_deps = [cloud_point_rpc_dep]
cli_cpp_args = []
if opencv_dep.found()
cli_deps += [cloud_point_compute_dep]
cli_cpp_args += ['-DHAVE_CLOUD_POINT_COMPUTE']
endif
libcloud_point_rpc_cli = shared_library(
'libcloud_point_rpc_cli',
'cli.cpp',
include_directories: inc,
dependencies: [cloud_point_rpc_dep],
dependencies: cli_deps,
cpp_args: cli_cpp_args,
install: true,
)
cloud_point_rpc_cli_dep = declare_dependency(
include_directories: inc,
link_with: libcloud_point_rpc_cli,
dependencies: [cloud_point_rpc_dep],
dependencies: cli_deps,
)
# Client/CLI tool (legacy stdin/stdout)

View File

@ -6,7 +6,7 @@
int main(int argc, char *argv[]) {
google::InitGoogleLogging(argv[0]);
FLAGS_logtostderr = 1;
FLAGS_logtostderr = 1;
std::string config_path = "config.yml";
if (argc > 1) {
@ -17,7 +17,7 @@ int main(int argc, char *argv[]) {
auto config = score::ConfigLoader::load(config_path);
score::TCPConnector connector(config.server.ip,
static_cast<size_t>(config.server.port));
static_cast<size_t>(config.server.port));
const std::string request =
R"({"jsonrpc":"2.0","method":"ping","params":{},"id":1})";

View File

@ -11,7 +11,7 @@ Base64RPCCoder::Base64RPCCoder() = default;
Base64RPCCoder::~Base64RPCCoder() = default;
/**
* Tries to decode ASCII complained string to the
* Tries to decode ASCII complained string to the raw bytes
* @param encoded ASCII complained base64 encoded string
* @return vector of raw bytes << allocated on encoded.size() / 4 * 3 + 1 size
*/

3
src/rpc_dto.cpp Normal file
View File

@ -0,0 +1,3 @@
// Compilation unit for rpc_dto.hpp — pulls in Base64RPCCoder linkage used by
// the inline to_json/from_json hooks, ensuring a single object definition.
#include "cloud_point_rpc/rpc_dto.hpp"

View File

@ -44,8 +44,14 @@ template <> struct Deleter<rpc_string> {
};
using rpcStringPtr = std::unique_ptr<rpc_string, Deleter<rpc_string>>;
RpcServer::RpcServer() {
register_method("get-available-methods",
[&](const json &) { return get_method_names(); });
}
void RpcServer::register_method(const std::string &name, Handler handler) {
handlers_[name] = std::move(handler);
handler_names_.push_back(handlers_.find(name)->first);
}
void RpcServer::register_method(const std::string &name, callback_t handler) {
@ -67,6 +73,24 @@ void RpcServer::register_method(const std::string &name, callback_t handler) {
}
return ret;
};
handler_names_.push_back(handlers_.find(name)->first);
}
std::span<std::string_view> RpcServer::get_method_names() noexcept {
return this->handler_names_;
}
uint64_t RpcServer::get_count() noexcept { return this->handler_names_.size(); }
std::string_view RpcServer::get_method_name_by_id(uint64_t id) noexcept {
if (id >= handler_names_.size()) {
LOG(ERROR) << __func__
<< std::format(
": called with id = {} which is bigger, than size={}",
id, handler_names_.size());
return {};
}
return handler_names_.at(id);
}
std::string RpcServer::process(const std::string &request_str) {

View File

@ -98,6 +98,8 @@ void crpc_init_with_address(const char *ip, int port) {
}
void crpc_deinit() {
if (server)
server->join();
server.reset();
std::lock_guard lock(gc_mtx);
gc.clear();
@ -111,4 +113,12 @@ void crpc_add_method(callback_t cb, rpc_string *name) {
std::lock_guard lock(server_mtx);
rpc_server.register_method(name->s, cb);
}
rpc_string *crpc_get_method_name_by_id(uint64_t id) {
auto value = rpc_server.get_method_name_by_id(id);
return crpc_str_create(value.data(), value.size());
}
uint64_t crpc_get_methods_count() { return rpc_server.get_count(); }
}

View File

@ -1,4 +1,5 @@
#include "cloud_point_rpc/config.hpp"
#include "cloud_point_rpc/rpc_dto.hpp"
#include "cloud_point_rpc/rpc_server.hpp"
#include "cloud_point_rpc/service.hpp"
#include "cloud_point_rpc/tcp_server.hpp"
@ -39,11 +40,22 @@ int main(int argc, char *argv[]) {
return service.get_cloud_point();
});
rpc_server.register_method("get-stereo-calibration", [&](const json &) {
nlohmann::json j;
score::to_json(j, service.get_stereo_calibration());
return j;
});
rpc_server.register_method("get-image-pair", [&](const json &) {
nlohmann::json j;
score::to_json(j, service.get_image_pair());
return j;
});
score::TcpServer server(config.server.ip, config.server.port,
[&](const std::string &request) {
return rpc_server.process(
request);
});
[&](const std::string &request) {
return rpc_server.process(request);
});
server.start();
server.join();

View File

@ -2,7 +2,7 @@
namespace score {
Service::Service(const TestData &data) : data_(data) {}
Service::Service(const TestData &data) : data_(data), frame_counter_(0) {}
std::vector<double> Service::get_intrinsic_params() const {
if (data_.intrinsic_params.empty()) {
@ -27,4 +27,66 @@ std::vector<std::vector<double>> Service::get_cloud_point() const {
return data_.cloud_point;
}
} // namespace cloud_point_rpc
StereoCalibrationRPC Service::get_stereo_calibration() const {
StereoCalibrationRPC calib;
calib.width = 640;
calib.height = 480;
CameraCalib cam;
// fx=fy=800, cx=320, cy=240 — identity rotation-scale, principal centre
cam.camera_matrix = {800.0, 0.0, 320.0, 0.0, 800.0, 240.0, 0.0, 0.0, 1.0};
cam.dist_coeffs = {0.0, 0.0, 0.0, 0.0, 0.0};
calib.left = cam;
calib.right = cam;
// Parallel rig: R = identity
calib.rotation = {1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0};
// T = [-baseline, 0, 0], baseline = 0.06 m
calib.translation = {-0.06, 0.0, 0.0};
return calib;
}
ImagePairRPC Service::get_image_pair() {
constexpr int kWidth = 640;
constexpr int kHeight = 480;
constexpr int kShift = 8;
ImagePairRPC pair;
pair.frame = frame_counter_++;
// Left image: BGR, each pixel value = (x + y) % 256 for all channels
pair.left.width = kWidth;
pair.left.height = kHeight;
pair.left.type = ImageRPC::Type::BGR;
pair.left.data.resize(static_cast<size_t>(kWidth) * kHeight * 3);
for (int y = 0; y < kHeight; ++y) {
for (int x = 0; x < kWidth; ++x) {
const auto val = static_cast<unsigned char>((x + y) % 256);
const size_t base = static_cast<size_t>(y * kWidth + x) * 3;
pair.left.data[base + 0] = val;
pair.left.data[base + 1] = val;
pair.left.data[base + 2] = val;
}
}
// Right image: same pattern shifted 8 px horizontally
pair.right.width = kWidth;
pair.right.height = kHeight;
pair.right.type = ImageRPC::Type::BGR;
pair.right.data.resize(static_cast<size_t>(kWidth) * kHeight * 3);
for (int y = 0; y < kHeight; ++y) {
for (int x = 0; x < kWidth; ++x) {
const auto val = static_cast<unsigned char>((x + kShift + y) % 256);
const size_t base = static_cast<size_t>(y * kWidth + x) * 3;
pair.right.data[base + 0] = val;
pair.right.data[base + 1] = val;
pair.right.data[base + 2] = val;
}
}
return pair;
}
} // namespace score

View File

@ -66,7 +66,7 @@ class TestThread {
}
}
void add_method(const callback_t cb, rpc_string *name) {
if(!name || !name->s.size()) {
if (!name || !name->s.size()) {
LOG(ERROR) << "Tried to add method with invalid name";
return;
}
@ -81,7 +81,7 @@ class TestThread {
}
int remove_method(const rpc_string *name) {
if(!name || !name->s.size()) {
if (!name || !name->s.size()) {
LOG(ERROR) << "Tried to remove method with invalid name";
return -1;
}

View File

@ -1,20 +1,33 @@
test_sources = files(
'test_rpc.cpp',
'test_rpc_edge_cases.cpp',
'test_integration.cpp',
'test_tcp.cpp',
'test_tcp_edge_cases.cpp',
'test_cli.cpp',
'test_c_api.cpp',
'test_c_api_edge_cases.cpp',
'test_base64.cpp',
'test_base64_edge_cases.cpp',
'test_service.cpp',
'test_serialize.cpp'
'test_serialize_image.cpp'
)
test_deps = [cloud_point_rpc_dep, cloud_point_rpc_cli_dep,
cloud_point_rpc_test_dep, json_dep, gtest_dep,
gtest_main_dep, gmock_dep]
if opencv_dep.found()
message('found cloud_point_compute dependency')
test_sources += files(
'test_image.cpp',
'test_stereo_matcher.cpp',
'test_stereo_rectifier.cpp',
'test_point_cloud_builder.cpp',
'test_cloud_point_client.cpp',
'test_scared_dataset.cpp'
)
test_deps += [cloud_point_compute_dep]
else
message('cpc_dep was not found')
endif
test_exe = executable('unit_tests',
test_sources,
dependencies : [cloud_point_rpc_dep, cloud_point_rpc_cli_dep, cloud_point_rpc_test_dep, json_dep, gtest_dep, gtest_main_dep, gmock_dep])
dependencies : test_deps)
test('unit_tests', test_exe)

View File

@ -9,17 +9,14 @@
#include "cloud_point_rpc/config.hpp"
#include "cloud_point_rpc/rpc_coder.hpp"
class Base64Test : public ::testing::Test {
protected:
protected:
void SetUp() override {
FLAGS_logtostderr = true;
if (!google::IsGoogleLoggingInitialized())
google::InitGoogleLogging("TestCli");
}
void TearDown() override {
}
void TearDown() override {}
};
TEST_F(Base64Test, EncodeDecode) {

View File

@ -1,17 +1,17 @@
#include "cloud_point_rpc/rpc_coder.hpp"
#include <glog/logging.h>
#include <gmock/gmock.h>
#include <gtest/gtest.h>
#include <glog/logging.h>
using namespace score;
class Base64EdgeCaseTest : public ::testing::Test {
protected:
void SetUp() override {
FLAGS_logtostderr = true;
if (!google::IsGoogleLoggingInitialized())
google::InitGoogleLogging("TestRPC");
}
void SetUp() override {
FLAGS_logtostderr = true;
if (!google::IsGoogleLoggingInitialized())
google::InitGoogleLogging("TestRPC");
}
Base64RPCCoder coder;
};

View File

@ -193,8 +193,7 @@ TEST_F(TestCApi, String) {
EXPECT_EQ(name.s.c_str(), crpc_str_get_data(&name));
EXPECT_EQ(name.s.size(), crpc_str_get_size(&name));
std::string_view testString = "test 2222";
auto creation =
crpc_str_create(testString.data(), testString.size());
auto creation = crpc_str_create(testString.data(), testString.size());
EXPECT_EQ(std::string_view(crpc_str_get_data(creation)), testString);
EXPECT_NO_THROW(crpc_str_destroy(creation));
}

View File

@ -12,10 +12,8 @@
using namespace score;
class CliTest : public ::testing::Test {
public:
void start() {
tcp_server->start();
}
public:
void start() { tcp_server->start(); }
protected:
void SetUp() override {
@ -27,7 +25,7 @@ public:
rpc_server = std::make_unique<RpcServer>();
tcp_server = std::make_unique<TcpServer>(
tcp_server = std::make_unique<TcpServer>(
server_ip, server_port, [this](const std::string &req) {
return rpc_server->process(req);
});

View File

@ -0,0 +1,294 @@
// E2E tests for CloudPointClient: in-process TcpServer + RpcServer with a
// synthetic scene providing known ground-truth depth.
//
// Calibration: fx=fy=800, cx=320, cy=240, 640×480, baseline=0.06 m,
// R=I, T=[-0.06,0,0]. Disparity = 32 px → z = 1.5 m.
#include <algorithm>
#include <chrono>
#include <cmath>
#include <limits>
#include <numeric>
#include <sstream>
#include <thread>
#include <vector>
#include <gmock/gmock.h>
#include <gtest/gtest.h>
#include <opencv2/core.hpp>
#include <opencv2/imgproc.hpp>
#include "cloud_point/cloud_point_client.hpp"
#include "cloud_point_rpc/cli.hpp"
#include "cloud_point_rpc/rpc_dto.hpp"
#include "cloud_point_rpc/rpc_server.hpp"
#include "cloud_point_rpc/tcp_server.hpp"
using namespace score;
using json = nlohmann::json;
// ---------------------------------------------------------------------------
// Shared synthetic calibration helpers
// ---------------------------------------------------------------------------
namespace {
constexpr double kFx = 800.0;
constexpr double kFy = 800.0;
constexpr double kCx = 320.0;
constexpr double kCy = 240.0;
constexpr int kWidth = 640;
constexpr int kHeight = 480;
constexpr double kBaseline = 0.06;
constexpr double kTx = -kBaseline; // OpenCV T convention
/// Standard stereo calibration DTO (parallel rig, no distortion).
StereoCalibrationRPC make_stereo_calib() {
StereoCalibrationRPC c;
c.width = kWidth;
c.height = kHeight;
c.left.camera_matrix = {kFx, 0, kCx, 0, kFy, kCy, 0, 0, 1};
c.left.dist_coeffs = {0, 0, 0, 0, 0};
c.right = c.left;
c.rotation = {1, 0, 0, 0, 1, 0, 0, 0, 1};
c.translation = {kTx, 0.0, 0.0};
return c;
}
/// Synthetic image pair: random left (seed 42), right = left shifted +32 px.
/// Convention: right(x, y) = left(x+d, y) → disparity d = 32 for positive
/// depth (matches service.cpp and OpenCV SGBM sign convention with Tx < 0).
ImagePairRPC make_image_pair(int disparity = 32) {
// Generate textured left grayscale image with fixed seed.
cv::theRNG().state = 42;
cv::Mat left_gray(kHeight, kWidth, CV_8UC1);
cv::randu(left_gray, 0, 256);
// Build right by shifting: right(x) = left(x+d), 0 at right border.
cv::Mat right_gray = cv::Mat::zeros(kHeight, kWidth, CV_8UC1);
for (int y = 0; y < kHeight; ++y) {
for (int x = 0; x < kWidth - disparity; ++x) {
right_gray.at<uchar>(y, x) = left_gray.at<uchar>(y, x + disparity);
}
}
// Convert to BGR for the wire protocol.
cv::Mat left_bgr, right_bgr;
cv::cvtColor(left_gray, left_bgr, cv::COLOR_GRAY2BGR);
cv::cvtColor(right_gray, right_bgr, cv::COLOR_GRAY2BGR);
auto mat_to_rpc = [](const cv::Mat &img, int w, int h) -> ImageRPC {
ImageRPC rpc;
rpc.width = w;
rpc.height = h;
rpc.type = ImageRPC::Type::BGR;
const size_t sz = static_cast<size_t>(w) * h * 3;
rpc.data.resize(sz);
std::memcpy(rpc.data.data(), img.data, sz);
return rpc;
};
ImagePairRPC pair;
pair.frame = 0;
pair.left = mat_to_rpc(left_bgr, kWidth, kHeight);
pair.right = mat_to_rpc(right_bgr, kWidth, kHeight);
return pair;
}
} // namespace
// ---------------------------------------------------------------------------
// Test fixture: in-process TcpServer + RpcServer on port 9201
// ---------------------------------------------------------------------------
class CloudPointClientTest : public ::testing::Test {
protected:
void SetUp() override {
FLAGS_logtostderr = true;
if (!google::IsGoogleLoggingInitialized())
google::InitGoogleLogging("TestCloudPointClient");
}
void TearDown() override {
if (server_) {
server_->stop();
}
}
/// Start an in-process server with the given RpcServer and wait for it.
void start_server(int port, std::unique_ptr<RpcServer> rpc) {
rpc_server_ = std::move(rpc);
server_ = std::make_unique<TcpServer>(
"127.0.0.1", port, [this](const std::string &req) {
return rpc_server_->process(req);
});
server_->start();
std::this_thread::sleep_for(std::chrono::milliseconds(100));
}
std::unique_ptr<RpcServer> rpc_server_;
std::unique_ptr<TcpServer> server_;
};
// ---------------------------------------------------------------------------
// Test 1: connect succeeds; compute_cloud() returns a cloud with median z ≈ 1.5
// m
// ---------------------------------------------------------------------------
TEST_F(CloudPointClientTest, ComputeCloudReturnsCorrectDepth) {
constexpr int kPort = 9201;
// Build RPC server with known stereo calibration and image pair.
auto rpc = std::make_unique<RpcServer>();
rpc->register_method("get-stereo-calibration", [](const json &) -> json {
json j;
to_json(j, make_stereo_calib());
return j;
});
rpc->register_method("get-image-pair", [](const json &) -> json {
json j;
to_json(j, make_image_pair(32));
return j;
});
start_server(kPort, std::move(rpc));
CloudPointClient client("127.0.0.1", kPort, StereoAlgorithmType::CPU);
ASSERT_NO_THROW(client.connect());
ASSERT_TRUE(client.connected());
auto result = client.compute_cloud();
ASSERT_TRUE(result.has_value())
<< "compute_cloud returned Error: " << result.error().message;
const auto &cloud = *result;
EXPECT_EQ(cloud.width, kWidth);
EXPECT_EQ(cloud.height, kHeight);
EXPECT_FALSE(cloud.valid_points().empty())
<< "Expected at least some valid points";
// Collect z values in the central region (avoid SGBM borders).
// Horizontal: [150, 490) to skip left border (SGBM invalid) and
// right border where right image has no data (last 32 cols).
// Vertical: [50, 430).
constexpr int kXMin = 150;
constexpr int kXMax = 490;
constexpr int kYMin = 50;
constexpr int kYMax = 430;
std::vector<float> z_vals;
z_vals.reserve(static_cast<size_t>((kXMax - kXMin) * (kYMax - kYMin)));
for (int y = kYMin; y < kYMax; ++y) {
for (int x = kXMin; x < kXMax; ++x) {
const float z =
cloud.data[static_cast<size_t>(y * kWidth + x) * 3 + 2];
if (!std::isnan(z))
z_vals.push_back(z);
}
}
ASSERT_FALSE(z_vals.empty()) << "No valid points in central region";
// Compute median z.
const auto mid = z_vals.begin() + static_cast<ptrdiff_t>(z_vals.size() / 2);
std::nth_element(z_vals.begin(), mid, z_vals.end());
const float median_z = *mid;
constexpr float kExpectedZ = static_cast<float>(kFx * kBaseline / 32.0);
constexpr float kToleranceZ = kExpectedZ * 0.05f; // 5%
EXPECT_NEAR(median_z, kExpectedZ, kToleranceZ)
<< "Median z in central region should be ~" << kExpectedZ
<< " m (expected=" << kExpectedZ << ", got=" << median_z << ")";
}
// ---------------------------------------------------------------------------
// Test 2: server returns garbage image data → compute_cloud() returns Error
// ---------------------------------------------------------------------------
TEST_F(CloudPointClientTest, GarbageImageDataReturnsError) {
constexpr int kPort = 9202;
auto rpc = std::make_unique<RpcServer>();
rpc->register_method("get-stereo-calibration", [](const json &) -> json {
json j;
to_json(j, make_stereo_calib());
return j;
});
// Return an image pair whose data size doesn't match width*height*channels.
rpc->register_method("get-image-pair", [](const json &) -> json {
ImagePairRPC bad_pair;
bad_pair.frame = 0;
bad_pair.left.width = kWidth;
bad_pair.left.height = kHeight;
bad_pair.left.type = ImageRPC::Type::BGR;
bad_pair.left.data = {0x01, 0x02}; // wrong size: 2 bytes, not 640*480*3
bad_pair.right = bad_pair.left;
json j;
to_json(j, bad_pair);
return j;
});
start_server(kPort, std::move(rpc));
CloudPointClient client("127.0.0.1", kPort, StereoAlgorithmType::CPU);
ASSERT_NO_THROW(client.connect());
auto result = client.compute_cloud();
EXPECT_FALSE(result.has_value())
<< "Expected Error for garbage image data, got a cloud instead";
if (!result.has_value()) {
EXPECT_FALSE(result.error().message.empty());
}
}
// ---------------------------------------------------------------------------
// Test 3: connect() to a closed port throws
// ---------------------------------------------------------------------------
TEST_F(CloudPointClientTest, ConnectToClosedPortThrows) {
// Port 9203 has no server running.
CloudPointClient client("127.0.0.1", 9203, StereoAlgorithmType::CPU);
EXPECT_THROW(client.connect(), std::runtime_error);
EXPECT_FALSE(client.connected());
}
// ---------------------------------------------------------------------------
// Test 4: CLI smoke — option 4 against the mock server
// ---------------------------------------------------------------------------
TEST_F(CloudPointClientTest, CliOption4ComputeCloud) {
constexpr int kPort = 9204;
auto rpc = std::make_unique<RpcServer>();
rpc->register_method("get-stereo-calibration", [](const json &) -> json {
json j;
to_json(j, make_stereo_calib());
return j;
});
rpc->register_method("get-image-pair", [](const json &) -> json {
json j;
to_json(j, make_image_pair(32));
return j;
});
// The CLI also uses the legacy RPC methods; register stubs so it can
// connect.
rpc->register_method("get-intrinsic-params",
[](const json &) { return std::vector<double>{1.0}; });
rpc->register_method("get-extrinsic-params",
[](const json &) { return std::vector<double>{1.0}; });
rpc->register_method("get-cloud-point", [](const json &) {
return std::vector<std::vector<double>>{};
});
start_server(kPort, std::move(rpc));
std::istringstream input("4\n0\n");
std::ostringstream output;
const int rc = run_cli(input, output, "127.0.0.1", kPort);
EXPECT_EQ(rc, 0);
const std::string out = output.str();
// Expect either a point count line ("valid_pts=") or an error message.
EXPECT_THAT(out, ::testing::AnyOf(::testing::HasSubstr("valid_pts="),
::testing::HasSubstr("Error")))
<< "CLI output was:\n"
<< out;
}

160
tests/test_image.cpp Normal file
View File

@ -0,0 +1,160 @@
//
// Created by vptyp on 12.03.2026.
//
#include <gmock/gmock.h>
#include <gtest/gtest.h>
#include <cloud_point/imageFactory.h>
#include <cloud_point/matrixFactory.h>
class ImageTest : public ::testing::Test {
protected:
void SetUp() override {}
void TearDown() override {}
};
TEST_F(ImageTest, DefaultConstructor) {
score::Image image;
cv::Mat mat = image.get();
EXPECT_TRUE(mat.empty());
}
TEST_F(ImageTest, ConstructorWithMat) {
cv::Mat input = cv::Mat::zeros(5, 5, CV_8UC1);
score::Image image(input);
cv::Mat output = image.get();
EXPECT_EQ(output.rows, 5);
EXPECT_EQ(output.cols, 5);
EXPECT_EQ(output.type(), CV_8UC1);
}
TEST_F(ImageTest, PixelTypeMapping) {
EXPECT_EQ(score::ImageFactory::pixelType(score::ImageRPC::Type::BGR),
CV_8UC3);
EXPECT_EQ(score::ImageFactory::pixelType(score::ImageRPC::Type::RGBA),
CV_8UC4);
EXPECT_EQ(score::ImageFactory::pixelType(score::ImageRPC::Type::DEPTH),
CV_64FC1);
EXPECT_THROW(score::ImageFactory::pixelType(score::ImageRPC::Type::UNKNOWN),
std::runtime_error);
}
TEST_F(ImageTest, CreateBGR) {
score::ImageRPC rpc;
rpc.width = 10;
rpc.height = 20;
rpc.type = score::ImageRPC::Type::BGR;
rpc.data.resize(rpc.width * rpc.height * 3, 128);
score::Image image = score::ImageFactory::create(rpc);
cv::Mat mat = image.get();
EXPECT_EQ(mat.rows, 20);
EXPECT_EQ(mat.cols, 10);
EXPECT_EQ(mat.type(), CV_8UC3);
EXPECT_EQ(mat.at<cv::Vec3b>(0, 0)[0], 128);
}
TEST_F(ImageTest, CreateRGBA) {
score::ImageRPC rpc;
rpc.width = 15;
rpc.height = 25;
rpc.type = score::ImageRPC::Type::RGBA;
rpc.data.resize(rpc.width * rpc.height * 4, 255);
score::Image image = score::ImageFactory::create(rpc);
cv::Mat mat = image.get();
EXPECT_EQ(mat.rows, 25);
EXPECT_EQ(mat.cols, 15);
EXPECT_EQ(mat.type(), CV_8UC4);
EXPECT_EQ(mat.at<cv::Vec4b>(0, 0)[0], 255);
}
TEST_F(ImageTest, CreateDepth) {
score::ImageRPC rpc;
rpc.width = 5;
rpc.height = 10;
rpc.type = score::ImageRPC::Type::DEPTH;
rpc.data.resize(rpc.width * rpc.height * sizeof(double));
auto *dataPtr = reinterpret_cast<double *>(rpc.data.data());
for (int i = 0; i < 50; ++i)
dataPtr[i] = static_cast<double>(i);
score::Image image = score::ImageFactory::create(rpc);
cv::Mat mat = image.get();
EXPECT_EQ(mat.rows, 10);
EXPECT_EQ(mat.cols, 5);
EXPECT_EQ(mat.type(), CV_64FC1);
EXPECT_DOUBLE_EQ(mat.at<double>(0, 0), 0.0);
// row 9, col 4 → flat index 9*5+4 = 49
EXPECT_DOUBLE_EQ(mat.at<double>(9, 4), 49.0);
}
TEST_F(ImageTest, CreateOwnsDataAfterSourceDestroyed) {
// Use asymmetric size (width=4, height=2) with known BGR values
constexpr int kWidth = 4;
constexpr int kHeight = 2;
constexpr int kBytes = kWidth * kHeight * 3;
score::Image image;
{
score::ImageRPC rpc;
rpc.width = kWidth;
rpc.height = kHeight;
rpc.type = score::ImageRPC::Type::BGR;
rpc.data.resize(kBytes);
for (int i = 0; i < kBytes; ++i)
rpc.data[i] = static_cast<unsigned char>(i);
image = score::ImageFactory::create(rpc);
// rpc goes out of scope here; image must own its data
}
cv::Mat mat = image.get();
EXPECT_EQ(mat.rows, kHeight);
EXPECT_EQ(mat.cols, kWidth);
EXPECT_EQ(mat.type(), CV_8UC3);
// Verify pixel values match the original source bytes
for (int r = 0; r < kHeight; ++r) {
for (int c = 0; c < kWidth; ++c) {
const int flat = (r * kWidth + c) * 3;
const auto px = mat.at<cv::Vec3b>(r, c);
EXPECT_EQ(px[0], static_cast<unsigned char>(flat));
EXPECT_EQ(px[1], static_cast<unsigned char>(flat + 1));
EXPECT_EQ(px[2], static_cast<unsigned char>(flat + 2));
}
}
}
TEST_F(ImageTest, CreateThrowsOnWrongDataSize) {
score::ImageRPC rpc;
rpc.width = 4;
rpc.height = 2;
rpc.type = score::ImageRPC::Type::BGR;
// Intentionally wrong size (1 byte short)
rpc.data.resize(4 * 2 * 3 - 1, 0);
EXPECT_THROW(score::ImageFactory::create(rpc), std::runtime_error);
}
TEST_F(ImageTest, CameraMatrixCreateOwnsDataAfterVectorDestroyed) {
cv::Mat mat;
{
std::vector<double> vals = {1, 2, 3, 4, 5, 6, 7, 8, 9};
mat = score::CameraMatrixFactory::create<3, 3>(vals);
// vals goes out of scope here; mat must own its data
}
ASSERT_EQ(mat.rows, 3);
ASSERT_EQ(mat.cols, 3);
for (int i = 0; i < 9; ++i)
EXPECT_DOUBLE_EQ(mat.at<double>(i / 3, i % 3),
static_cast<double>(i + 1));
}
TEST_F(ImageTest, CameraMatrixCreateThrowsOnWrongSize) {
std::vector<double> vals(8, 0.0); // 8 elements, need 9 for 3x3
EXPECT_THROW((score::CameraMatrixFactory::create<3, 3>(vals)),
std::runtime_error);
}

View File

@ -98,3 +98,11 @@ TEST_F(IntegrationTest, ClientCanConnectAndRetrieveValues) {
TEST_F(IntegrationTest, ClientHandlesConnectionError) {
EXPECT_THROW(TCPConnector connector("127.0.0.1", 9999), std::runtime_error);
}
TEST_F(IntegrationTest, ClientRetrieveRemoteMethods) {
TCPConnector connector(config_.server.ip, config_.server.port);
RpcClient client(connector);
auto res = client.call<std::vector<std::string>>("get-available-methods");
EXPECT_EQ(res.size(), 2);
}

View File

@ -0,0 +1,254 @@
// Tests for PointCloudBuilder: CV_16S / CV_32F handling, depth filtering,
// NaN propagation, and valid_points() correctness.
#include <gtest/gtest.h>
#include <array>
#include <cmath>
#include <vector>
#include <opencv2/core.hpp>
#include "cloud_point/point_cloud_builder.hpp"
#include "cloud_point/stereo_rectifier.hpp"
namespace {
// ---------------------------------------------------------------------------
// Shared synthetic calibration (same as test_stereo_rectifier.cpp)
// ---------------------------------------------------------------------------
constexpr double kFx = 800.0;
constexpr double kFy = 800.0;
constexpr double kCx = 320.0;
constexpr double kCy = 240.0;
constexpr int kWidth = 640;
constexpr int kHeight = 480;
constexpr double kBaseline = 0.06; // metres
constexpr double kTx = -kBaseline;
score::StereoRectifier::Calibration make_calib() {
score::StereoRectifier::Calibration calib;
// clang-format off
calib.k_left = (cv::Mat_<double>(3, 3) <<
kFx, 0, kCx,
0, kFy, kCy,
0, 0, 1);
calib.k_right = calib.k_left.clone();
calib.d_left = cv::Mat::zeros(1, 5, CV_64F);
calib.d_right = cv::Mat::zeros(1, 5, CV_64F);
calib.r = cv::Mat::eye(3, 3, CV_64F);
calib.t = (cv::Mat_<double>(3, 1) << kTx, 0.0, 0.0);
// clang-format on
calib.image_size = cv::Size(kWidth, kHeight);
return calib;
}
cv::Mat make_q() {
score::StereoRectifier rectifier(make_calib());
return rectifier.q().clone();
}
// ---------------------------------------------------------------------------
// Helper: count non-NaN (valid) pixels in the cloud
// ---------------------------------------------------------------------------
int count_valid(const score::PointCloud &cloud) {
int n = 0;
for (int i = 0; i < cloud.height * cloud.width; ++i) {
if (!std::isnan(cloud.data[static_cast<size_t>(i) * 3]))
++n;
}
return n;
}
} // namespace
// ---------------------------------------------------------------------------
// Constant-disparity CV_16S → expected depth = fx * baseline / disparity
// ---------------------------------------------------------------------------
TEST(PointCloudBuilderTest, ConstantDisparityCV16S) {
const cv::Mat q = make_q();
score::PointCloudBuilder builder(q);
// d = 32 px, stored as SGBM fixed-point (d * 16)
constexpr float kDisp = 32.0f;
constexpr float kExpectedZ =
static_cast<float>(kFx * kBaseline / kDisp); // 1.5 m
cv::Mat disparity(kHeight, kWidth, CV_16S,
cv::Scalar(static_cast<short>(kDisp * 16)));
const auto cloud = builder.build(disparity);
EXPECT_EQ(cloud.width, kWidth);
EXPECT_EQ(cloud.height, kHeight);
EXPECT_EQ(static_cast<int>(cloud.data.size()), kWidth * kHeight * 3);
// Every point should be valid and at z ≈ 1.5 m
const auto valid = cloud.valid_points();
EXPECT_EQ(static_cast<int>(valid.size()), kWidth * kHeight);
for (const auto &pt : valid) {
EXPECT_NEAR(pt[2], kExpectedZ, 1e-3f)
<< "z should be fx*baseline/disparity = " << kExpectedZ << " m";
}
}
TEST(PointCloudBuilderTest, CenterPixelXYNearZero) {
const cv::Mat q = make_q();
score::PointCloudBuilder builder(q);
constexpr float kDisp = 32.0f;
cv::Mat disparity(kHeight, kWidth, CV_16S,
cv::Scalar(static_cast<short>(kDisp * 16)));
const auto cloud = builder.build(disparity);
// At the image center (row=cy=240, col=cx=320), x≈0 and y≈0.
const int center_idx =
(static_cast<int>(kCy) * kWidth + static_cast<int>(kCx)) * 3;
EXPECT_NEAR(cloud.data[static_cast<size_t>(center_idx)], 0.0f, 0.05f)
<< "x at image center should be near 0";
EXPECT_NEAR(cloud.data[static_cast<size_t>(center_idx) + 1], 0.0f, 0.05f)
<< "y at image center should be near 0";
}
// ---------------------------------------------------------------------------
// Zero / negative disparity → NaN
// ---------------------------------------------------------------------------
TEST(PointCloudBuilderTest, ZeroDisparityProducesNaN) {
const cv::Mat q = make_q();
score::PointCloudBuilder builder(q);
cv::Mat disparity = cv::Mat::zeros(kHeight, kWidth, CV_16S);
const auto cloud = builder.build(disparity);
EXPECT_EQ(count_valid(cloud), 0)
<< "All points should be NaN when disparity is zero";
EXPECT_EQ(static_cast<int>(cloud.valid_points().size()), 0);
}
TEST(PointCloudBuilderTest, NegativeDisparityProducesNaN) {
const cv::Mat q = make_q();
score::PointCloudBuilder builder(q);
cv::Mat disparity(kHeight, kWidth, CV_16S, cv::Scalar(-16)); // -1 px
const auto cloud = builder.build(disparity);
EXPECT_EQ(count_valid(cloud), 0)
<< "All points should be NaN when disparity is negative";
}
// ---------------------------------------------------------------------------
// Out-of-range depth filter → all NaN
// ---------------------------------------------------------------------------
TEST(PointCloudBuilderTest, MaxDepthFilterRejectsAll) {
const cv::Mat q = make_q();
// kExpectedZ = 1.5 m; set max_depth_m = 1.0 to reject all points.
score::PointCloudBuilder::Options opts;
opts.max_depth_m = 1.0f;
score::PointCloudBuilder builder(q, opts);
constexpr float kDisp = 32.0f; // → z = 1.5 m, which exceeds max
cv::Mat disparity(kHeight, kWidth, CV_16S,
cv::Scalar(static_cast<short>(kDisp * 16)));
const auto cloud = builder.build(disparity);
EXPECT_EQ(count_valid(cloud), 0)
<< "All points should be NaN when z=1.5 m exceeds max_depth_m=1.0 m";
}
TEST(PointCloudBuilderTest, MinDepthFilterRejectsAll) {
const cv::Mat q = make_q();
// kExpectedZ = 1.5 m; set min_depth_m = 2.0 to reject all points.
score::PointCloudBuilder::Options opts;
opts.min_depth_m = 2.0f;
score::PointCloudBuilder builder(q, opts);
constexpr float kDisp = 32.0f; // → z = 1.5 m, below min
cv::Mat disparity(kHeight, kWidth, CV_16S,
cv::Scalar(static_cast<short>(kDisp * 16)));
const auto cloud = builder.build(disparity);
EXPECT_EQ(count_valid(cloud), 0)
<< "All points should be NaN when z=1.5 m is below min_depth_m=2.0 m";
}
// ---------------------------------------------------------------------------
// valid_points() count matches non-NaN count
// ---------------------------------------------------------------------------
TEST(PointCloudBuilderTest, ValidPointsCountMatchesNonNaN) {
const cv::Mat q = make_q();
score::PointCloudBuilder builder(q);
// Half the image has positive disparity, half has zero
cv::Mat disparity(kHeight, kWidth, CV_16S, cv::Scalar(0));
const cv::Rect left_half(0, 0, kWidth / 2, kHeight);
disparity(left_half).setTo(cv::Scalar(32 * 16));
const auto cloud = builder.build(disparity);
const int manual_count = count_valid(cloud);
const int method_count = static_cast<int>(cloud.valid_points().size());
EXPECT_EQ(manual_count, method_count)
<< "valid_points() size must equal the number of non-NaN pixels";
}
// ---------------------------------------------------------------------------
// CV_32F input works identically
// ---------------------------------------------------------------------------
TEST(PointCloudBuilderTest, CV32FInputMatchesCV16S) {
const cv::Mat q = make_q();
score::PointCloudBuilder builder(q);
constexpr float kDisp = 32.0f;
// CV_16S reference
cv::Mat disp16(kHeight, kWidth, CV_16S,
cv::Scalar(static_cast<short>(kDisp * 16)));
const auto cloud16 = builder.build(disp16);
// CV_32F equivalent
cv::Mat disp32(kHeight, kWidth, CV_32F, cv::Scalar(kDisp));
const auto cloud32 = builder.build(disp32);
ASSERT_EQ(cloud16.data.size(), cloud32.data.size());
// Every z-coordinate should match within floating-point tolerance
for (int i = 0; i < kHeight * kWidth; ++i) {
const float z16 = cloud16.data[static_cast<size_t>(i) * 3 + 2];
const float z32 = cloud32.data[static_cast<size_t>(i) * 3 + 2];
EXPECT_NEAR(z16, z32, 1e-3f) << "at pixel " << i;
}
}
// ---------------------------------------------------------------------------
// Unsupported disparity type → throws
// ---------------------------------------------------------------------------
TEST(PointCloudBuilderTest, CV8UInputThrows) {
const cv::Mat q = make_q();
score::PointCloudBuilder builder(q);
cv::Mat bad_disparity(kHeight, kWidth, CV_8U, cv::Scalar(32));
EXPECT_THROW(builder.build(bad_disparity), std::invalid_argument);
}
// ---------------------------------------------------------------------------
// Invalid Q matrix → constructor throws
// ---------------------------------------------------------------------------
TEST(PointCloudBuilderTest, InvalidQSizeThrows) {
cv::Mat bad_q = cv::Mat::eye(3, 4, CV_64F); // not 4x4
EXPECT_THROW(score::PointCloudBuilder{bad_q}, std::invalid_argument);
}
TEST(PointCloudBuilderTest, InvalidQTypeThrows) {
cv::Mat bad_q = cv::Mat::eye(4, 4, CV_32F); // not CV_64F
EXPECT_THROW(score::PointCloudBuilder{bad_q}, std::invalid_argument);
}

View File

@ -64,3 +64,16 @@ TEST_F(RpcServerTest, InvalidJsonReturnsParseError) {
ASSERT_TRUE(response.contains("error"));
EXPECT_EQ(response["error"]["code"], -32700);
}
TEST_F(RpcServerTest, GetMethod) {
EXPECT_EQ(server.get_count(), 2);
EXPECT_EQ(server.get_method_name_by_id(1), "get-intrinsic-params");
EXPECT_EQ(server.get_method_names()[1], "get-intrinsic-params");
server.register_method("get-test-2", [&](const json &) {
return std::variant<json, std::string>{std::string("test")};
});
EXPECT_EQ(server.get_count(), 3);
EXPECT_EQ(server.get_method_name_by_id(2), "get-test-2");
EXPECT_EQ(server.get_method_names()[2], "get-test-2");
}

View File

@ -1,8 +1,8 @@
#include "cloud_point_rpc/rpc_server.hpp"
#include "server_api.h"
#include <glog/logging.h>
#include <gmock/gmock.h>
#include <gtest/gtest.h>
#include <glog/logging.h>
#include <nlohmann/json.hpp>
#include <string>
#include <thread>

View File

@ -0,0 +1,147 @@
/// @file test_scared_dataset.cpp
/// @brief E2E test: in-process server backed by SCARED dataset + CloudPointClient.
///
/// Skipped unless env var SCARED_KEYFRAME_DIR is set (CI has no dataset).
/// Run locally:
/// SCARED_KEYFRAME_DIR=/path/to/test_dataset_8/keyframe_0 \
/// ./build/tests/unit_tests --gtest_filter=ScaredDataset*
#include <algorithm>
#include <atomic>
#include <chrono>
#include <cmath>
#include <cstdlib>
#include <string>
#include <thread>
#include <vector>
#include <gmock/gmock.h>
#include <gtest/gtest.h>
#include <nlohmann/json.hpp>
#include "cloud_point/cloud_point_client.hpp"
#include "cloud_point/scared_dataset_loader.hpp"
#include "cloud_point_rpc/rpc_dto.hpp"
#include "cloud_point_rpc/rpc_server.hpp"
#include "cloud_point_rpc/tcp_server.hpp"
using namespace score;
using json = nlohmann::json;
// ---------------------------------------------------------------------------
// Fixture: in-process TcpServer + RpcServer backed by the SCARED loader
// ---------------------------------------------------------------------------
class ScaredDatasetTest : public ::testing::Test {
protected:
void SetUp() override {
FLAGS_logtostderr = true;
if (!google::IsGoogleLoggingInitialized())
google::InitGoogleLogging("TestScaredDataset");
const char *env = std::getenv("SCARED_KEYFRAME_DIR");
if (!env || std::string(env).empty()) {
GTEST_SKIP() << "SCARED_KEYFRAME_DIR not set; "
"skipping SCARED E2E test";
}
keyframe_dir_ = env;
}
void TearDown() override {
if (server_) {
server_->stop();
}
}
void start_server(int port, std::unique_ptr<RpcServer> rpc) {
rpc_server_ = std::move(rpc);
server_ = std::make_unique<TcpServer>(
"127.0.0.1", port,
[this](const std::string &req) {
return rpc_server_->process(req);
});
server_->start();
std::this_thread::sleep_for(std::chrono::milliseconds(200));
}
std::string keyframe_dir_;
std::unique_ptr<RpcServer> rpc_server_;
std::unique_ptr<TcpServer> server_;
};
// ---------------------------------------------------------------------------
// Test: cloud non-empty and median z within plausible endoscopy range
// ---------------------------------------------------------------------------
TEST_F(ScaredDatasetTest, ComputeCloudFromRealData) {
constexpr int kPort = 9301;
// SCARED rig: fx~1024, B~4.35 mm -> max disparity ~160 needed
constexpr int kNumDisparities = 160;
// Expected depth range for endoscopy: 20 mm - 200 mm
constexpr float kMinExpectedZ = 0.02f;
constexpr float kMaxExpectedZ = 0.20f;
// Minimum valid points for a non-trivial cloud
constexpr size_t kMinValidPts = 50'000;
ScaredDatasetLoader loader(keyframe_dir_);
std::atomic<uint64_t> frame_counter{0};
auto rpc = std::make_unique<RpcServer>();
rpc->register_method(
"get-stereo-calibration", [&](const json &) -> json {
json j;
to_json(j, loader.calibration());
return j;
});
rpc->register_method(
"get-image-pair", [&](const json &) -> json {
json j;
to_json(j, loader.image_pair(frame_counter++));
return j;
});
start_server(kPort, std::move(rpc));
CloudPointClient client("127.0.0.1", kPort, StereoAlgorithmType::CPU,
PointCloudBuilder::Options{}, kNumDisparities);
ASSERT_NO_THROW(client.connect());
ASSERT_TRUE(client.connected());
auto result = client.compute_cloud();
ASSERT_TRUE(result.has_value())
<< "compute_cloud returned Error: " << result.error().message;
const auto &cloud = *result;
const auto valid_points = cloud.valid_points();
EXPECT_GE(valid_points.size(), kMinValidPts)
<< "Expected >" << kMinValidPts << " valid points, got "
<< valid_points.size();
// Collect z values and compute median.
std::vector<float> z_vals;
z_vals.reserve(valid_points.size());
for (const auto &pt : valid_points) {
z_vals.push_back(pt[2]);
}
ASSERT_FALSE(z_vals.empty()) << "No valid points in cloud";
const auto mid =
z_vals.begin() + static_cast<ptrdiff_t>(z_vals.size() / 2);
std::nth_element(z_vals.begin(), mid, z_vals.end());
const float median_z = *mid;
// Report for the task summary.
std::cout << "[SCARED] valid_points=" << valid_points.size()
<< " median_z=" << median_z << " m\n";
EXPECT_GE(median_z, kMinExpectedZ)
<< "Median z " << median_z
<< " m is below minimum expected " << kMinExpectedZ
<< " m (check mm->m conversion)";
EXPECT_LE(median_z, kMaxExpectedZ)
<< "Median z " << median_z
<< " m exceeds maximum expected " << kMaxExpectedZ
<< " m (check mm->m conversion: T must be divided by 1000)";
}

View File

@ -1,18 +1,18 @@
#include "cloud_point_rpc/serialize.hpp"
#include <glog/logging.h>
#include <gmock/gmock.h>
#include <gtest/gtest.h>
#include <glog/logging.h>
#include <limits>
using namespace score;
class SerializeEdgeCaseTest : public ::testing::Test {
protected:
void SetUp() override {
FLAGS_logtostderr = true;
if (!google::IsGoogleLoggingInitialized())
google::InitGoogleLogging("TestRPC");
}
protected:
void SetUp() override {
FLAGS_logtostderr = true;
if (!google::IsGoogleLoggingInitialized())
google::InitGoogleLogging("TestRPC");
}
};
// uint8_t round-trip

View File

@ -0,0 +1,303 @@
// Unit and in-process integration tests for rpc_dto.hpp serialisation.
// No OpenCV dependency — safe to compile unconditionally.
#include "cloud_point_rpc/rpc_dto.hpp"
#include "cloud_point_rpc/rpc_server.hpp"
#include "cloud_point_rpc/service.hpp"
#include <glog/logging.h>
#include <gmock/gmock.h>
#include <gtest/gtest.h>
#include <nlohmann/json.hpp>
#include <stdexcept>
using namespace score;
using json = nlohmann::json;
class SerializeImageTest : public ::testing::Test {
protected:
void SetUp() override {
FLAGS_logtostderr = true;
if (!google::IsGoogleLoggingInitialized())
google::InitGoogleLogging("SerializeImageTest");
}
};
// ---------------------------------------------------------------------------
// ImageRPC round-trip — binary data with zero bytes and 255s
// ---------------------------------------------------------------------------
TEST_F(SerializeImageTest, ImageRPCRoundTrip) {
ImageRPC img;
img.width = 2;
img.height = 2;
img.type = ImageRPC::Type::BGR;
// 4 pixels x 3 channels = 12 bytes, including 0s and 255s
img.data = {0, 255, 0, 255, 0, 255, 0, 0, 0, 255, 255, 255};
json j;
to_json(j, img);
EXPECT_EQ(j["width"].get<int>(), 2);
EXPECT_EQ(j["height"].get<int>(), 2);
EXPECT_EQ(j["type"].get<std::string>(), "BGR");
EXPECT_TRUE(j["data"].is_string());
ImageRPC decoded;
from_json(j, decoded);
EXPECT_EQ(decoded.width, img.width);
EXPECT_EQ(decoded.height, img.height);
EXPECT_EQ(decoded.type, img.type);
EXPECT_EQ(decoded.data, img.data);
}
// ---------------------------------------------------------------------------
// ImageRPC — unknown type string throws
// ---------------------------------------------------------------------------
TEST_F(SerializeImageTest, ImageRPCUnknownTypeThrows) {
json j = {{"width", 1}, {"height", 1}, {"type", "XYZ"}, {"data", "AAAA"}};
ImageRPC img;
EXPECT_THROW(from_json(j, img), std::runtime_error);
}
// ---------------------------------------------------------------------------
// ImageRPC — RGBA type string survives round-trip
// ---------------------------------------------------------------------------
TEST_F(SerializeImageTest, ImageRPCRGBARoundTrip) {
ImageRPC img;
img.width = 1;
img.height = 1;
img.type = ImageRPC::Type::RGBA;
img.data = {10, 20, 30, 40};
json j;
to_json(j, img);
EXPECT_EQ(j["type"].get<std::string>(), "RGBA");
ImageRPC out;
from_json(j, out);
EXPECT_EQ(out.type, ImageRPC::Type::RGBA);
EXPECT_EQ(out.data, img.data);
}
// ---------------------------------------------------------------------------
// ImageRPC — DEPTH type string survives round-trip
// ---------------------------------------------------------------------------
TEST_F(SerializeImageTest, ImageRPCDEPTHRoundTrip) {
ImageRPC img;
img.width = 1;
img.height = 1;
img.type = ImageRPC::Type::DEPTH;
img.data = {0x3f, 0x80, 0x00, 0x00}; // 1.0f as little-endian float32
json j;
to_json(j, img);
EXPECT_EQ(j["type"].get<std::string>(), "DEPTH");
ImageRPC out;
from_json(j, out);
EXPECT_EQ(out.type, ImageRPC::Type::DEPTH);
EXPECT_EQ(out.data, img.data);
}
// ---------------------------------------------------------------------------
// CameraCalib round-trip
// ---------------------------------------------------------------------------
TEST_F(SerializeImageTest, CameraCalibRoundTrip) {
CameraCalib c;
c.camera_matrix = {800, 0, 320, 0, 800, 240, 0, 0, 1};
c.dist_coeffs = {0.1, 0.2, 0.0, 0.0, 0.05};
json j;
to_json(j, c);
CameraCalib out;
from_json(j, out);
EXPECT_EQ(out.camera_matrix, c.camera_matrix);
EXPECT_EQ(out.dist_coeffs, c.dist_coeffs);
}
// ---------------------------------------------------------------------------
// CameraCalib — wrong camera_matrix size (8 elements) → throws
// ---------------------------------------------------------------------------
TEST_F(SerializeImageTest, CameraCalibWrongCameraMatrixSizeThrows) {
json j = {{"camera_matrix", {1, 2, 3, 4, 5, 6, 7, 8}},
{"dist_coeffs", {0, 0, 0, 0, 0}}};
CameraCalib c;
EXPECT_THROW(from_json(j, c), std::runtime_error);
}
// ---------------------------------------------------------------------------
// CameraCalib — wrong dist_coeffs size → throws
// ---------------------------------------------------------------------------
TEST_F(SerializeImageTest, CameraCalibWrongDistCoeffsSizeThrows) {
json j = {{"camera_matrix", {800, 0, 320, 0, 800, 240, 0, 0, 1}},
{"dist_coeffs", {0, 0, 0}}};
CameraCalib c;
EXPECT_THROW(from_json(j, c), std::runtime_error);
}
// ---------------------------------------------------------------------------
// StereoCalibrationRPC round-trip
// ---------------------------------------------------------------------------
TEST_F(SerializeImageTest, StereoCalibrationRoundTrip) {
StereoCalibrationRPC calib;
calib.left.camera_matrix = {800, 0, 320, 0, 800, 240, 0, 0, 1};
calib.left.dist_coeffs = {0, 0, 0, 0, 0};
calib.right = calib.left;
calib.rotation = {1, 0, 0, 0, 1, 0, 0, 0, 1};
calib.translation = {-0.06, 0.0, 0.0};
calib.width = 640;
calib.height = 480;
json j;
to_json(j, calib);
StereoCalibrationRPC out;
from_json(j, out);
EXPECT_EQ(out.left.camera_matrix, calib.left.camera_matrix);
EXPECT_EQ(out.right.dist_coeffs, calib.right.dist_coeffs);
EXPECT_EQ(out.rotation, calib.rotation);
EXPECT_DOUBLE_EQ(out.translation[0], -0.06);
EXPECT_DOUBLE_EQ(out.translation[1], 0.0);
EXPECT_DOUBLE_EQ(out.translation[2], 0.0);
EXPECT_EQ(out.width, 640);
EXPECT_EQ(out.height, 480);
}
// ---------------------------------------------------------------------------
// StereoCalibrationRPC — wrong rotation size → throws
// ---------------------------------------------------------------------------
TEST_F(SerializeImageTest, StereoCalibWrongRotationSizeThrows) {
// rotation has only 5 elements — must throw
json j = {{"left",
{{"camera_matrix", {800, 0, 320, 0, 800, 240, 0, 0, 1}},
{"dist_coeffs", {0, 0, 0, 0, 0}}}},
{"right",
{{"camera_matrix", {800, 0, 320, 0, 800, 240, 0, 0, 1}},
{"dist_coeffs", {0, 0, 0, 0, 0}}}},
{"rotation", {1, 0, 0, 0, 1}},
{"translation", {-0.06, 0, 0}},
{"image_size", {{"width", 640}, {"height", 480}}}};
StereoCalibrationRPC calib;
EXPECT_THROW(from_json(j, calib), std::runtime_error);
}
// ---------------------------------------------------------------------------
// StereoCalibrationRPC — wrong translation size → throws
// ---------------------------------------------------------------------------
TEST_F(SerializeImageTest, StereoCalibWrongTranslationSizeThrows) {
json j = {{"left",
{{"camera_matrix", {800, 0, 320, 0, 800, 240, 0, 0, 1}},
{"dist_coeffs", {0, 0, 0, 0, 0}}}},
{"right",
{{"camera_matrix", {800, 0, 320, 0, 800, 240, 0, 0, 1}},
{"dist_coeffs", {0, 0, 0, 0, 0}}}},
{"rotation", {1, 0, 0, 0, 1, 0, 0, 0, 1}},
{"translation", {-0.06}}, // only 1 element — wrong
{"image_size", {{"width", 640}, {"height", 480}}}};
StereoCalibrationRPC calib;
EXPECT_THROW(from_json(j, calib), std::runtime_error);
}
// ---------------------------------------------------------------------------
// ImagePairRPC round-trip
// ---------------------------------------------------------------------------
TEST_F(SerializeImageTest, ImagePairRPCRoundTrip) {
ImagePairRPC pair;
pair.frame = 42;
pair.left.width = 2;
pair.left.height = 1;
pair.left.type = ImageRPC::Type::BGR;
pair.left.data = {0, 128, 255, 10, 20, 30};
pair.right = pair.left;
pair.right.data = {5, 6, 7, 8, 9, 10};
json j;
to_json(j, pair);
ImagePairRPC out;
from_json(j, out);
EXPECT_EQ(out.frame, 42u);
EXPECT_EQ(out.left.data, pair.left.data);
EXPECT_EQ(out.right.data, pair.right.data);
EXPECT_EQ(out.left.type, ImageRPC::Type::BGR);
}
// ---------------------------------------------------------------------------
// In-process integration: Service -> RpcServer -> process() -> from_json
// ---------------------------------------------------------------------------
TEST_F(SerializeImageTest, ServiceStereoCalibrationViaRpcServer) {
Service service;
RpcServer rpc;
rpc.register_method("get-stereo-calibration", [&](const json &) -> json {
json j;
score::to_json(j, service.get_stereo_calibration());
return j;
});
const std::string request =
R"({"jsonrpc":"2.0","method":"get-stereo-calibration","id":1})";
const std::string response_str = rpc.process(request);
auto resp = json::parse(response_str);
ASSERT_TRUE(resp.contains("result")) << "Response was: " << response_str;
StereoCalibrationRPC calib;
from_json(resp["result"], calib);
EXPECT_DOUBLE_EQ(calib.left.camera_matrix[0], 800.0); // fx
EXPECT_DOUBLE_EQ(calib.left.camera_matrix[4], 800.0); // fy
EXPECT_DOUBLE_EQ(calib.left.camera_matrix[2], 320.0); // cx
EXPECT_DOUBLE_EQ(calib.left.camera_matrix[5], 240.0); // cy
EXPECT_DOUBLE_EQ(calib.translation[0], -0.06);
EXPECT_DOUBLE_EQ(calib.translation[1], 0.0);
EXPECT_EQ(calib.width, 640);
EXPECT_EQ(calib.height, 480);
// Rotation should be identity
EXPECT_DOUBLE_EQ(calib.rotation[0], 1.0);
EXPECT_DOUBLE_EQ(calib.rotation[4], 1.0);
EXPECT_DOUBLE_EQ(calib.rotation[8], 1.0);
}
TEST_F(SerializeImageTest, ServiceImagePairViaRpcServer) {
Service service;
RpcServer rpc;
rpc.register_method("get-image-pair", [&](const json &) -> json {
json j;
score::to_json(j, service.get_image_pair());
return j;
});
const std::string request =
R"({"jsonrpc":"2.0","method":"get-image-pair","id":1})";
const std::string response_str = rpc.process(request);
auto resp = json::parse(response_str);
ASSERT_TRUE(resp.contains("result")) << "Response was: " << response_str;
ImagePairRPC pair;
from_json(resp["result"], pair);
EXPECT_EQ(pair.frame, 0u);
EXPECT_EQ(pair.left.width, 640);
EXPECT_EQ(pair.left.height, 480);
EXPECT_EQ(pair.left.type, ImageRPC::Type::BGR);
EXPECT_EQ(pair.right.width, 640);
EXPECT_EQ(pair.right.height, 480);
// Pixel at (x=0, y=0): left = (0+0)%256 = 0
EXPECT_EQ(pair.left.data[0], static_cast<unsigned char>(0));
// Pixel at (x=0, y=0): right = (0+8+0)%256 = 8
EXPECT_EQ(pair.right.data[0], static_cast<unsigned char>(8));
// Second call: frame counter should increment
const std::string request2 =
R"({"jsonrpc":"2.0","method":"get-image-pair","id":2})";
const std::string response_str2 = rpc.process(request2);
auto resp2 = json::parse(response_str2);
ImagePairRPC pair2;
from_json(resp2["result"], pair2);
EXPECT_EQ(pair2.frame, 1u);
}

View File

@ -5,12 +5,12 @@
using namespace score;
class ServiceEdgeCaseTest : public ::testing::Test {
protected:
void SetUp() override {
FLAGS_logtostderr = true;
if (!google::IsGoogleLoggingInitialized())
google::InitGoogleLogging("TestRPC");
}
protected:
void SetUp() override {
FLAGS_logtostderr = true;
if (!google::IsGoogleLoggingInitialized())
google::InitGoogleLogging("TestRPC");
}
};
// Default constructor (no data)

View File

@ -0,0 +1,80 @@
#include <gtest/gtest.h>
#include <opencv2/core.hpp>
#include <tuple>
#include "cloud_point/cpu_stereo_matcher.hpp"
#include "cloud_point/gpu_stereo_matcher.hpp"
#include "cloud_point/stereo_matcher_factory.hpp"
using namespace score;
namespace {
/// @brief Create a simple synthetic stereo pair with a horizontal shift.
std::pair<cv::Mat, cv::Mat> make_synthetic_stereo(int shift = 2) {
cv::Mat left = cv::Mat::zeros(100, 100, CV_8UC1);
cv::Mat right = cv::Mat::zeros(100, 100, CV_8UC1);
for (int y = 0; y < 100; ++y) {
for (int x = 0; x < 100; ++x) {
left.at<uchar>(y, x) = static_cast<uchar>(x % 256);
right.at<uchar>(y, x) = static_cast<uchar>((x + shift) % 256);
}
}
return {left, right};
}
} // namespace
TEST(StereoMatcherTest, CpuMatcherComputesDisparity) {
auto [left, right] = make_synthetic_stereo();
CpuStereoMatcher matcher;
cv::Mat disparity = matcher.compute(left, right);
EXPECT_FALSE(disparity.empty());
EXPECT_EQ(disparity.rows, left.rows);
EXPECT_EQ(disparity.cols, left.cols);
}
TEST(StereoMatcherTest, FactoryCpuCreatesNonNull) {
auto matcher = StereoMatcherFactory::create(StereoAlgorithmType::CPU);
ASSERT_NE(matcher, nullptr);
auto [left, right] = make_synthetic_stereo();
cv::Mat disparity = matcher->compute(left, right);
EXPECT_FALSE(disparity.empty());
}
TEST(StereoMatcherTest, FactoryRejectsInvalidNumDisparities) {
EXPECT_THROW(std::ignore = StereoMatcherFactory::create(
StereoAlgorithmType::CPU, 0),
std::invalid_argument);
EXPECT_THROW(std::ignore = StereoMatcherFactory::create(
StereoAlgorithmType::CPU, -16),
std::invalid_argument);
EXPECT_THROW(std::ignore = StereoMatcherFactory::create(
StereoAlgorithmType::CPU, 150),
std::invalid_argument);
}
TEST(StereoMatcherTest, FactoryAcceptsValidNumDisparities) {
auto matcher = StereoMatcherFactory::create(StereoAlgorithmType::CPU, 160);
ASSERT_NE(matcher, nullptr);
}
TEST(StereoMatcherTest, FactoryGpuFallsBackToCpuWhenUnavailable) {
// On this machine CUDA is absent; factory should fall back to CPU.
auto matcher = StereoMatcherFactory::create(StereoAlgorithmType::GPU);
ASSERT_NE(matcher, nullptr);
auto [left, right] = make_synthetic_stereo();
EXPECT_NO_THROW(std::ignore = matcher->compute(left, right));
}
TEST(StereoMatcherTest, GpuMatcherThrowsOnThisMachine) {
// Direct construction of GpuStereoMatcher should throw because
// HAVE_OPENCV_CUDA is undefined here.
EXPECT_THROW(GpuStereoMatcher(), std::runtime_error);
}

View File

@ -0,0 +1,242 @@
// Tests for StereoRectifier: construction, Q-matrix semantics, from_rpc
// round-trip, near-identity rectification, and invalid-calibration rejection.
#include <gtest/gtest.h>
#include <cmath>
#include <vector>
#include <opencv2/core.hpp>
#include <opencv2/imgproc.hpp>
#include "cloud_point/stereo_rectifier.hpp"
#include "cloud_point_rpc/rpc_dto.hpp"
namespace {
// ---------------------------------------------------------------------------
// Helpers — synthetic calibration
// ---------------------------------------------------------------------------
// fx = fy = 800, cx = 320, cy = 240, 640x480, zero distortion,
// R = identity, T = [-0.06, 0, 0] (6 cm horizontal baseline).
constexpr double kFx = 800.0;
constexpr double kFy = 800.0;
constexpr double kCx = 320.0;
constexpr double kCy = 240.0;
constexpr int kWidth = 640;
constexpr int kHeight = 480;
constexpr double kBaseline = 0.06; // metres
constexpr double kTx = -kBaseline; // OpenCV T: left→right camera
score::StereoRectifier::Calibration make_calib() {
score::StereoRectifier::Calibration calib;
// clang-format off
calib.k_left = (cv::Mat_<double>(3, 3) <<
kFx, 0, kCx,
0, kFy, kCy,
0, 0, 1);
calib.k_right = calib.k_left.clone();
calib.d_left = cv::Mat::zeros(1, 5, CV_64F);
calib.d_right = cv::Mat::zeros(1, 5, CV_64F);
calib.r = cv::Mat::eye(3, 3, CV_64F);
calib.t = (cv::Mat_<double>(3, 1) << kTx, 0.0, 0.0);
// clang-format on
calib.image_size = cv::Size(kWidth, kHeight);
return calib;
}
score::StereoCalibrationRPC make_rpc() {
score::StereoCalibrationRPC rpc;
// Camera matrix: row-major 3x3
rpc.left.camera_matrix = {kFx, 0, kCx, 0, kFy, kCy, 0, 0, 1};
rpc.right.camera_matrix = rpc.left.camera_matrix;
rpc.left.dist_coeffs = {0, 0, 0, 0, 0};
rpc.right.dist_coeffs = {0, 0, 0, 0, 0};
rpc.rotation = {1, 0, 0, 0, 1, 0, 0, 0, 1};
rpc.translation = {kTx, 0.0, 0.0};
rpc.width = kWidth;
rpc.height = kHeight;
return rpc;
}
} // namespace
// ---------------------------------------------------------------------------
// Construction & Q-matrix semantics
// ---------------------------------------------------------------------------
TEST(StereoRectifierTest, QMatrixIs4x4) {
score::StereoRectifier rectifier(make_calib());
const cv::Mat &q = rectifier.q();
EXPECT_EQ(q.rows, 4);
EXPECT_EQ(q.cols, 4);
EXPECT_EQ(q.type(), CV_64F);
}
TEST(StereoRectifierTest, QMatrixFocalEntry) {
// OpenCV Q layout (horizontal stereo):
// Q = [1 0 0 -cx ]
// [0 1 0 -cy ]
// [0 0 0 f ] ← Q(2,3) = focal length
// [0 0 -1/Tx ...]
// With fx = fy = 800, Q(2,3) should be ≈ 800.
score::StereoRectifier rectifier(make_calib());
const cv::Mat &q = rectifier.q();
EXPECT_NEAR(q.at<double>(2, 3), kFx, 1.0)
<< "Q(2,3) should equal the focal length (~" << kFx << ")";
}
TEST(StereoRectifierTest, QMatrixBaselineEntry) {
// Q(3,2) = -1/Tx. With Tx = -0.06, -1/Tx = +16.667 (positive).
// Observed sign convention: Q(3,2) > 0 for Tx < 0.
score::StereoRectifier rectifier(make_calib());
const cv::Mat &q = rectifier.q();
const double expected = -1.0 / kTx; // = +16.667 for Tx=-0.06
EXPECT_NEAR(q.at<double>(3, 2), expected, 1.0)
<< "Q(3,2) should be -1/Tx = " << expected
<< " (observed sign: positive for Tx < 0)";
}
TEST(StereoRectifierTest, DepthFromQConsistency) {
// z = Q(2,3) / (-Q(3,2)) / disparity → = fx * |Tx| / disparity.
// For d=32: z = 800 * 0.06 / 32 = 1.5 m.
score::StereoRectifier rectifier(make_calib());
const cv::Mat &q = rectifier.q();
const double f = q.at<double>(2, 3);
const double q32 = q.at<double>(3, 2);
// baseline_in_Q = -1/q32 gives Tx, |Tx| = baseline
const double baseline_q = std::abs(-1.0 / q32);
const double disparity = 32.0;
const double z = f * baseline_q / disparity;
EXPECT_NEAR(z, 1.5, 1e-3)
<< "z = fx * baseline / disparity = 800 * 0.06 / 32 should be 1.5 m";
}
// ---------------------------------------------------------------------------
// from_rpc round-trip
// ---------------------------------------------------------------------------
TEST(StereoRectifierTest, FromRpcRoundTrip) {
const auto rpc = make_rpc();
const auto calib = score::StereoRectifier::Calibration::from_rpc(rpc);
// Sizes
EXPECT_EQ(calib.k_left.rows, 3);
EXPECT_EQ(calib.k_left.cols, 3);
EXPECT_EQ(calib.d_left.rows, 1);
EXPECT_EQ(calib.d_left.cols, 5);
EXPECT_EQ(calib.r.rows, 3);
EXPECT_EQ(calib.r.cols, 3);
EXPECT_EQ(calib.t.rows, 3);
EXPECT_EQ(calib.t.cols, 1);
EXPECT_EQ(calib.image_size, cv::Size(kWidth, kHeight));
// Values — camera matrix diagonal
EXPECT_DOUBLE_EQ(calib.k_left.at<double>(0, 0), kFx);
EXPECT_DOUBLE_EQ(calib.k_left.at<double>(1, 1), kFy);
EXPECT_DOUBLE_EQ(calib.k_left.at<double>(0, 2), kCx);
EXPECT_DOUBLE_EQ(calib.k_left.at<double>(1, 2), kCy);
// Translation
EXPECT_DOUBLE_EQ(calib.t.at<double>(0, 0), kTx);
EXPECT_DOUBLE_EQ(calib.t.at<double>(1, 0), 0.0);
EXPECT_DOUBLE_EQ(calib.t.at<double>(2, 0), 0.0);
// Rotation is identity
for (int r = 0; r < 3; ++r)
for (int c = 0; c < 3; ++c)
EXPECT_DOUBLE_EQ(calib.r.at<double>(r, c), r == c ? 1.0 : 0.0);
}
TEST(StereoRectifierTest, FromRpcProducesValidRectifier) {
const auto rpc = make_rpc();
const auto calib = score::StereoRectifier::Calibration::from_rpc(rpc);
EXPECT_NO_THROW(score::StereoRectifier rectifier(calib));
}
// ---------------------------------------------------------------------------
// Near-identity rectification
// ---------------------------------------------------------------------------
TEST(StereoRectifierTest, RectifyOutputSizeUnchanged) {
score::StereoRectifier rectifier(make_calib());
// Deterministic gradient image (use fixed seed via theRNG)
cv::theRNG().state = 42;
cv::Mat left(kHeight, kWidth, CV_8UC1);
cv::randn(left, 128, 40);
cv::Mat right = left.clone();
auto [rl, rr] = rectifier.rectify(left, right);
EXPECT_EQ(rl.rows, kHeight);
EXPECT_EQ(rl.cols, kWidth);
EXPECT_EQ(rr.rows, kHeight);
EXPECT_EQ(rr.cols, kWidth);
}
TEST(StereoRectifierTest, RectifyNearIdentityWithZeroDistortion) {
// With R=I, identical K, and zero distortion the rectification maps are
// near-identity. The mean absolute difference between input and output
// should be small (< 5 intensity levels out of 255).
score::StereoRectifier rectifier(make_calib());
// Deterministic gradient: pixel value = (row + col) % 256
cv::Mat left(kHeight, kWidth, CV_8UC1);
for (int r = 0; r < kHeight; ++r)
for (int c = 0; c < kWidth; ++c)
left.at<uchar>(r, c) = static_cast<uchar>((r + c) % 256);
cv::Mat right = left.clone();
auto [rl, rr] = rectifier.rectify(left, right);
// Compare rectified-left row sums to original row sums
cv::Mat diff;
cv::absdiff(rl, left, diff);
const double mean_diff = cv::mean(diff)[0];
EXPECT_LT(mean_diff, 5.0)
<< "Mean absolute pixel difference after rectification is " << mean_diff
<< " — expected near-identity for zero-distortion identical cameras";
}
// ---------------------------------------------------------------------------
// Invalid calibration → throws
// ---------------------------------------------------------------------------
TEST(StereoRectifierTest, InvalidKSizeThrows) {
auto calib = make_calib();
calib.k_left = cv::Mat::eye(2, 3, CV_64F); // wrong: 2x3 instead of 3x3
EXPECT_THROW(score::StereoRectifier{calib}, std::invalid_argument);
}
TEST(StereoRectifierTest, InvalidKTypeThrows) {
auto calib = make_calib();
calib.k_left = cv::Mat::eye(3, 3, CV_32F); // wrong type
EXPECT_THROW(score::StereoRectifier{calib}, std::invalid_argument);
}
TEST(StereoRectifierTest, InvalidDistSizeThrows) {
auto calib = make_calib();
calib.d_right = cv::Mat::zeros(1, 4, CV_64F); // 1x4, needs 1x5
EXPECT_THROW(score::StereoRectifier{calib}, std::invalid_argument);
}
TEST(StereoRectifierTest, InvalidRotationSizeThrows) {
auto calib = make_calib();
calib.r = cv::Mat::eye(3, 2, CV_64F); // 3x2, needs 3x3
EXPECT_THROW(score::StereoRectifier{calib}, std::invalid_argument);
}
TEST(StereoRectifierTest, InvalidTranslationSizeThrows) {
auto calib = make_calib();
calib.t = cv::Mat::zeros(1, 3, CV_64F); // 1x3, needs 3x1
EXPECT_THROW(score::StereoRectifier{calib}, std::invalid_argument);
}