UnityLaparoscopicSceneSimulator/docs/notes-unity-perception.md

Unity Perception Package — Notes for This Project

What it is

Unity Perception (com.unity.perception) is Unity Technologies' official package for synthetic dataset generation. It provides:

• Labelers (ground-truth exporters): depth, instance segmentation, semantic segmentation, 2D/3D bounding boxes, keypoints, normals
• Randomizers: domain randomization for lights, textures, camera pose, object placement
• Perception Camera: a Camera component wrapper that drives per-frame capture
• Solo format: standardised JSON + PNG/EXR output, one folder per sequence

The Critical Incompatibility

Perception only supports HDRP. It does not work with URP or Built-in RP.

This project uses URP 14.0.11. Dropping com.unity.perception into this project would either fail silently or require switching the render pipeline, which would break the existing depth pipeline, materials, and Volume configuration.

This incompatibility is confirmed in the package docs:

"For URP projects, you can add a Ground Truth Renderer Feature to the ForwardRenderer" — but this is an older partial workaround (pre-1.0), not the full labeler system.

What to Do Instead

The project already has the right foundation for a custom URP ground-truth exporter:

Needed output	Existing foundation	What's missing
Depth (metric, .exr/.npy)	DepthRenderPassFeature + DepthGetter	Write-to-disk from RenderTexture
Surface normals	URP DepthNormals prepass	Custom ScriptableRenderPass to capture & save
RGB left/right	StreamCamera pair	ReadPixels / AsyncGPUReadback to PNG
Segmentation masks	Not yet present	Replacement shader + stencil pass per semantic class
Camera intrinsics/extrinsics	CrpcApi → get-intrinsic-params / get-extrinsic-params	Wire output to JSON writer
Scene parameters	Not yet present	C# SceneParameterExporter serialising scene state

The write-to-disk layer is the main gap. A single GroundTruthExporter.cs script (attached to the stereo rig, not touching off-limits scripts) can:

1. Subscribe to Camera.onPostRender or use AsyncGPUReadback
2. Read depth RenderTexture → save as .exr (left + right)
3. Read RGB RenderTexture → save as .png
4. Call CrpcApi for intrinsics/extrinsics → serialise to JSON
5. Write scene_parameters.json from a SceneConfig ScriptableObject

Dataset Format Target (from workflow.md §15)

sequence_0001/
├── left/000000.png ...
├── right/000000.png ...
├── depth_left/000000.npy ...
├── disparity_left/000000.npy ...
├── masks/{tissue,instrument,specularity,occlusion,smoke}/
├── normals/000000.npy ...
└── calibration/{intrinsics_left,intrinsics_right,extrinsics,stereo_calibration}.json
    metadata/{scene_parameters.json, random_seed.txt, config.yaml}

Links

• Perception 1.0 docs
• Custom render pass workflow in URP 14
• Unity Perception GitHub