c51e6d1369
:Release Notes: - Agents files for opencode and claude - Skills for opencode and claude - 3d models with medical organs - Some textures :Detailed Notes: - :Testing Performed: - :QA Notes: - Looks like shit :) :Issues Addressed: TG-1
61 lines
3.3 KiB
Markdown
61 lines
3.3 KiB
Markdown
# Unity Perception Package — Notes for This Project
|
|
|
|
## What it is
|
|
|
|
[Unity Perception](https://docs.unity3d.com/Packages/com.unity.perception@1.0/manual/index.html) (`com.unity.perception`) is Unity Technologies' official package for synthetic dataset generation. It provides:
|
|
|
|
- **Labelers** (ground-truth exporters): depth, instance segmentation, semantic segmentation, 2D/3D bounding boxes, keypoints, normals
|
|
- **Randomizers**: domain randomization for lights, textures, camera pose, object placement
|
|
- **Perception Camera**: a `Camera` component wrapper that drives per-frame capture
|
|
- **Solo format**: standardised JSON + PNG/EXR output, one folder per sequence
|
|
|
|
## The Critical Incompatibility
|
|
|
|
> **Perception only supports HDRP. It does not work with URP or Built-in RP.**
|
|
|
|
This project uses **URP 14.0.11**. Dropping `com.unity.perception` into this project would either fail silently or require switching the render pipeline, which would break the existing depth pipeline, materials, and Volume configuration.
|
|
|
|
This incompatibility is confirmed in the package docs:
|
|
> "For URP projects, you can add a Ground Truth Renderer Feature to the ForwardRenderer" — but this is an older partial workaround (pre-1.0), not the full labeler system.
|
|
|
|
## What to Do Instead
|
|
|
|
The project already has the right foundation for a custom URP ground-truth exporter:
|
|
|
|
| Needed output | Existing foundation | What's missing |
|
|
|---|---|---|
|
|
| Depth (metric, `.exr`/`.npy`) | `DepthRenderPassFeature` + `DepthGetter` | Write-to-disk from `RenderTexture` |
|
|
| Surface normals | URP DepthNormals prepass | Custom `ScriptableRenderPass` to capture & save |
|
|
| RGB left/right | `StreamCamera` pair | `ReadPixels` / `AsyncGPUReadback` to PNG |
|
|
| Segmentation masks | Not yet present | Replacement shader + stencil pass per semantic class |
|
|
| Camera intrinsics/extrinsics | `CrpcApi` → `get-intrinsic-params` / `get-extrinsic-params` | Wire output to JSON writer |
|
|
| Scene parameters | Not yet present | C# `SceneParameterExporter` serialising scene state |
|
|
|
|
The write-to-disk layer is the main gap. A single `GroundTruthExporter.cs` script (attached to the stereo rig, not touching off-limits scripts) can:
|
|
|
|
1. Subscribe to `Camera.onPostRender` or use `AsyncGPUReadback`
|
|
2. Read depth `RenderTexture` → save as `.exr` (left + right)
|
|
3. Read RGB `RenderTexture` → save as `.png`
|
|
4. Call `CrpcApi` for intrinsics/extrinsics → serialise to JSON
|
|
5. Write `scene_parameters.json` from a `SceneConfig` ScriptableObject
|
|
|
|
## Dataset Format Target (from `workflow.md §15`)
|
|
|
|
```
|
|
sequence_0001/
|
|
├── left/000000.png ...
|
|
├── right/000000.png ...
|
|
├── depth_left/000000.npy ...
|
|
├── disparity_left/000000.npy ...
|
|
├── masks/{tissue,instrument,specularity,occlusion,smoke}/
|
|
├── normals/000000.npy ...
|
|
└── calibration/{intrinsics_left,intrinsics_right,extrinsics,stereo_calibration}.json
|
|
metadata/{scene_parameters.json, random_seed.txt, config.yaml}
|
|
```
|
|
|
|
## Links
|
|
|
|
- [Perception 1.0 docs](https://docs.unity3d.com/Packages/com.unity.perception@1.0/manual/index.html)
|
|
- [Custom render pass workflow in URP 14](https://docs.unity3d.com/Packages/com.unity.render-pipelines.universal@14.0/manual/renderer-features/custom-rendering-pass-workflow-in-urp.html)
|
|
- [Unity Perception GitHub](https://github.com/Unity-Technologies/com.unity.perception)
|