For VJs, installation artists, and anyone chasing the intersection of computer vision, GPU compute, and physical light, a new open-source project has landed that feels purpose-built for our world.
GPU Laser Vision by Joshua Batty is a real-time Rust pipeline that takes 720p video (or live sources in future iterations) and converts it into segmented projector output and coloured laser geometry ready for an Ether Dream DAC. It runs two parallel vision paths side-by-side—classic custom CUDA edge detection and modern YOLO11 instance segmentation—then extracts contours, recovers original colours, and packs scanner-safe paths.
The result sits comfortably in the nannou creative-coding ecosystem (Batty is a co-founder of nannou) and already supports fullscreen secondary-display projector isolation plus laser streaming.
Why This Matters for VJs and Live Visual Artists
Most of us already push pixels, particles, and shaders. Few of us have a clean, open, high-performance bridge from camera/video → semantic understanding → actual laser scanners or isolated projector silhouettes without heavy custom engineering or proprietary tools.
GPU Laser Vision gives you:
- Two complementary perceptions: dense classical edges (fast, detailed, fragmented) versus clean person silhouettes from YOLO11n-seg.
- Colour recovery so the laser or projector inherits the source palette instead of flat white or arbitrary tints.
- Live comparison dashboard (nannou + egui) showing intermediate stages, thresholds, performance numbers, and source selection.
- Physical outputs ready to go: person isolation on a secondary display, and Ether Dream streaming at up to 30k points/sec with scanner-aware packing (different strategies for dense CUDA edges vs coherent YOLO contours).
Performance on an RTX 5090 is already production-relevant for many setups. With a 1280×720 @ 25 fps source the processing callback sits in the 14.8–15.6 ms range (roughly 64–68 fps of compute headroom). The custom CUDA edge pipeline alone is ~1.8–1.9 ms; YOLO11n segmentation is 9.1–10.1 ms. The media itself is currently the limiting factor.
Architecture at a Glance
Custom CUDA path
RGBA → grayscale → Scharr magnitude + gradients → live threshold + colour sampling along the gradient normal → compact host copy → path tracing into open paths, closed loops, and points. Everything runs as a captured CUDA graph with persistent buffers.
Neural path
Upload/resize/normalize/letterbox on CUDA → fixed 640×640 YOLO11n-seg TorchScript (LibTorch) → strongest person mask restored to 720p → one-pixel contour + green-spill replacement → same path tracer.
Both routes feed the same geometry, colour, display, and point-count surfaces so the comparison stays honest. Projector output isolates the detected person against black. Laser output is explicitly gated and disabled by default—important for safety.
Getting It Running
Requirements are Linux or WSL2 + NVIDIA GPU, Nix with flakes, and a 720p demo clip. CUDA, nightly Rust, LibTorch/PyTorch, FFmpeg, etc. are pinned by the flake—no system CUDA toolkit needed.
git clone https://github.com/JoshuaBatty/gpu-laser-vision.git
cd gpu-laser-vision
nix develop --command uv run scripts/export_yolo.py # YOLO11n-seg → TorchScript
# place your 1280×720 clip at assets/jcvd_green_screen_720p.mp4
nix develop --command cargo oxide run
Optional Ether Dream: set ETHER_DREAM_IP if discovery fails (common under WSL). Secondary display automatically enables the projector isolation window.
The interface lets you tweak Scharr thresholds live, switch CUDA vs YOLO geometry for the laser, inspect grayscale/Scharr/mask stages, and monitor FPS + stage timings.
Current Scope and Roadmap
The full chain—source → GPU vision → projector / Ether Dream—is already implemented. Next work targets production installation realities: live camera capture, projector/scanner calibration, venue safety integration, and documenting finished light performances.
Safety note (please read it twice): Laser hardware needs proper scan-rate limits, blanking, interlocks, and venue-specific controls. On-screen paths alone are not a complete laser-safety system.
Why You Should Care Right Now
This is the kind of project that lowers the barrier between “I can track a person in Resolume/TouchDesigner/Isadora” and “I can drive a real laser scanner or isolated projection with the same contours and original colours at interactive rates.” It is written in Rust with careful CUDA graphs and Nsight-profiled kernels, licensed AGPL-3.0, and sits in the same creative-coding lineage as nannou.
If you work with lasers, projection mapping, interactive installations, or simply want a high-performance vision playground that already speaks laser geometry, clone it, throw a green-screen clip at it, and start playing.
Repo: https://github.com/JoshuaBatty/gpu-laser-vision
Have you already experimented with computer-vision-driven lasers or person isolation in a live context? Drop your experiences, hardware setups, or questions below—let’s talk about what production-ready versions of tools like this need next.
Top comments (0)