Pili Hu Stream Thumbnail
#2 · by Pili Hu

A Small Step for HumanUse

Cite / Share: https://hupili.net/stream/world-agent/2/

When people imagine giving AI a physical body, the immediate picture is usually a robot: metal limbs, servo motors, and expensive hardware built to mimic a human.

At yesterday's Rokid Hackathon, we tried the opposite question: what if the AI is the brain, and the human is the body?

The Setup

We paired a pair of Rokid AR glasses directly with Antigravity and Gemini 3.8 Flash over Wi-Fi.

  • Rokid is the Eye: The front-facing camera captures what is ahead; the optical waveguide HUD projects simple directional cues into the wearer's line of sight.
  • Gemini is the Brain: Antigravity handles the loop, running spatial perception and deciding navigation paths.
  • The Human is the Feet: The wearer provides locomotion, walking through physical space guided by HUD arrows and voice prompts.

No bulky robot chassis. No physical tether. Just an agent orchestrating perception and movement through off-the-shelf AR glasses.

Physical Actuator (Wearer with AR Glasses) Expand
(a) Physical Actuator (Wearer with AR Glasses)
Waveguide Optical HUD (Line of Sight) Expand
(b) Waveguide Optical HUD (Line of Sight)
Supervisor Cockpit & Telemetry (:8089) Expand
(c) Supervisor Cockpit & Telemetry (:8089)
The HumanUse Closed Loop: Pairing an untethered human locomotive body with waveguide directional vectors and remote AI supervisor telemetry.

Making It Real

Getting this to work smoothly in the physical world required solving gritty, practical friction:

  1. The Minimal Wired POC: We started with the baseline: paired the glasses in the Hi Rokid App, enabled Developer Mode, and plugged into the Mac via USB-C. Using Antigravity with the rokid-aiui-agent-skill and official samples from rokid-aiui-demos, we verified wired ADB connectivity, optical display controls, and basic sensor triggers before attempting mobility.
  2. Untethered Pipeline (Reverse ADB over Wi-Fi): You cannot walk around a room following navigation arrows while tethered by a 1-meter cable. Leveraging Rokid-Wireless-ADB pairing techniques, we moved the connection to an 802.11 Wi-Fi hotspot (172.20.10.3:5555) with reverse port forwarding (tcp:8088 -> tcp:8088), letting the glasses stream frames directly into a local Node.js relay bridge without a physical leash.
  3. Zero-Wait Perception (Decoupled Capture): Calling Rokid's JS camera API (capture.takePhoto returning WebP frames) takes 3.8s to 4.5s. If the human stops and waits for every shutter, movement feels broken. We ran rolling background capture cycles so Gemini 3.8 Flash always reasons against a fresh in-memory frame (<2.0s old), cutting decision wait times from ~9s down to ~1.5s.
  4. Cutting the Noise (Visual Delta Gating): Language models love to talk like verbose museum guides. In physical navigation, excessive chatter is exhausting. We gated speech using normalized visual difference ($\Delta \ge 0.14$) on a $64\times 64$ grayscale buffer: when stationary, the system stays dead silent; it speaks only when physical movement shifts the scene.
  5. Waveguide HUD Guidance (Green Vector Badges): Instead of cluttered AR graphics, the AI flashed minimal directional cues (Turn Left 90° to Glass Wall) directly onto the 480×352 green monochrome optical waveguide display, mirrored in real time to the supervisor dashboard via Server-Sent Events (SSE).
  6. Architectural Deduction over Brittle Markers: The agent didn't rely on pre-scanned point clouds or fiducial tags. Using Gemini 3.8 Flash, it deduced the room layout from architectural features—identifying whiteboard walls versus glass partitions—to locate exits dynamically.
  7. Supervisor Cockpit & Safety Telemetry: We built a local web supervisor cockpit (:8089) mirroring the live camera stream and waveguide display in real time. It gave the observer manual override controls (Left 45°, Walk 5, Right 45°, speech intercom, and an emergency stop button) to intervene and transmit directional hints straight to the wearer's glasses.
  8. Closing the Causal Loop: When the AI directs a turn and the person steps forward, the next incoming camera frame reflects that action. If the wearer turns forty degrees instead of forty-five, the AI sees the difference, re-localizes, and adapts.

The Loop

The intelligence wasn't just in the model, and it wasn't just in the person. It was in the loop itself: perceive the room, give a hint, take a step, and adapt.

Yesterday was a small step for Rokid and Antigravity, but a big step for HumanUse.

Developer Stack & References

World AgentHumanUseRokidAntigravityClosed LoopSpatial Intelligence