NVIDIA logo

Skill

cloudxr-js-sdk

build WebXR streaming clients with CloudXR

Published by NVIDIA Updated Jul 14
Covers VR Immersive AR WebXR NVIDIA

Description

Build WebXR streaming clients with NVIDIA CloudXR.js SDK. Use when developing WebXR apps, integrating CloudXR streaming, creating VR/AR web clients, working with @nvidia/cloudxr, or building React Three Fiber XR applications.

SKILL.md

NVIDIA CloudXR.js SDK

CloudXR.js streams XR content from a GPU server to a web browser. The server renders; the client displays video and sends back head/hand tracking. Think "Netflix for VR."

Quick Start

Install

From NGC tarball:

npm install nvidia-cloudxr-<version>.tgz

Peer dependencies (app must provide): gl-matrix, long.

Minimal Integration (5 steps)

import { createSession, SessionState } from '@nvidia/cloudxr';

// 1. Get WebXR session + WebGL context
const xrSession = await navigator.xr.requestSession('immersive-vr', {
  optionalFeatures: ['local-floor'],
});
const gl = xrSession.renderState.baseLayer.context;
const referenceSpace = await xrSession.requestReferenceSpace('local-floor');

// 2. Create CloudXR session
const session = createSession(
  {
    serverAddress: '192.168.1.100',
    serverPort: 49100,
    useSecureConnection: false,
    gl,
    perEyeWidth: 2048,
    perEyeHeight: 1792,
    referenceSpace,
  },
  {
    onStreamStarted: () => console.log('Streaming'),
    onStreamStopped: err => err && console.error(err.message),
  }
);

// 3. Connect
session.connect();

// 4. Render loop
function onFrame(time, frame) {
  session.sendTrackingStateToServer(time, frame);
  session.render(time, frame, xrSession.renderState.baseLayer);
  xrSession.requestAnimationFrame(onFrame);
}
xrSession.requestAnimationFrame(onFrame);

// 5. Disconnect when done
session.disconnect();

Core API

ExportPurpose
createSession(options, delegates?)Create a session. Returns Session.
SessionInterface: state, connect(), disconnect(), sendTrackingStateToServer(), render(), availableMessageChannels
SessionOptionsConfig: server address/port, WebGL context, resolution, codec, bitrate, ICE servers, etc.
SessionDelegatesCallbacks: onStreamStarted, onStreamStopped, onWebGLStateChangeBegin/End, onMetrics, onLog
SessionStateEnum: Initialized → Connecting → Connected → Disconnecting → Disconnected → Error
MessageChannelBinary data channel for custom app messages (preferred over deprecated sendServerMessage)

Required SessionOptions

FieldTypeNotes
serverAddressstringIP or hostname of CloudXR Runtime
serverPortnumberDefault 49100
useSecureConnectionbooleantrue for WSS (production), false for WS (dev)
glWebGL2RenderingContextFrom canvas or XR layer
perEyeWidthnumberMust be multiple of 16, ≥ 256
perEyeHeightnumberMust be multiple of 64, ≥ 256
referenceSpaceXRReferenceSpaceFrom WebXR session

Key Optional SessionOptions

FieldDefaultNotes
codec'h264'Also 'av1'
deviceFrameRatedevice defaulte.g. 72, 90
maxStreamingBitrateKbpsSDK defaulte.g. 150000
enablePoseSmoothingSmooth head tracking
posePredictionFactorPrediction multiplier
mediaAddress / mediaPortNAT traversal
signalingResourcePathProxy routing path
signalingQueryParametersAuth tokens, etc.
iceServersSTUN/TURN for enterprise NAT
glBindingXRWebGLBinding for R3F
telemetry{ enabled, appInfo: { version, product } }

Resolution Rules

  • perEyeWidth must be a multiple of 16 and ≥ 256
  • perEyeHeight must be a multiple of 64 and ≥ 256
  • Stream resolution: width = perEyeWidth * 2, height = perEyeHeight * 9 / 4
  • Use validatePerEyeResolution() and getResolutionValidationError() before calling createSession

Two Integration Patterns

Pattern 1: Vanilla WebGL + WebXR

See the Simple WebGL sample.

Key flow: check capabilities → create WebGL context → request XR session → createSession() → XR RAF loop with sendTrackingStateToServer() + render().

WebGL state: CloudXR modifies GL state during render. If you share the context, save/restore via onWebGLStateChangeBegin / onWebGLStateChangeEnd delegates.

Pattern 2: React Three Fiber (R3F)

See the React sample.

import * as CloudXR from '@nvidia/cloudxr';
import { useThree, useFrame } from '@react-three/fiber';
import { useRef } from 'react';
import { bindGL } from '@helpers/WebGLStateBinding';

function CloudXRComponent({ config }) {
  const gl = useThree().gl.getContext() as WebGL2RenderingContext;
  const trackedGL = bindGL(gl);
  const sessionRef = useRef<CloudXR.Session | null>(null);

  // On XR session start → createSession + connect
  // On XR session end → disconnect

  useFrame(state => {
    const xrFrame = state.gl.xr.getFrame();
    if (sessionRef.current?.state === CloudXR.SessionState.Connected && xrFrame) {
      const timestamp = state.clock.elapsedTime * 1000;
      const layer = state.gl.xr.getBaseLayer() as XRWebGLLayer;
      sessionRef.current.sendTrackingStateToServer(timestamp, xrFrame);
      sessionRef.current.render(timestamp, xrFrame, layer);
    }
  }, -1000); // negative priority = run before Three.js render
}

Critical R3F detail: set threeRenderer.autoClear = false so Three.js doesn't clear the framebuffer after CloudXR renders. Use bindGL() from helpers to save/restore WebGL state around CloudXR calls.

Error Handling

// connect() throws synchronously if it can't initiate
try {
  session.connect();
} catch (e) {
  /* handle */
}

// sendTrackingStateToServer returns false if not connected
if (!session.sendTrackingStateToServer(time, frame)) {
  /* not ready */
}

// Streaming errors arrive via delegate
onStreamStopped: (error?: StreamingError) => {
  if (error) {
    console.error(error.message);
    if (error.code) console.error(`0x${error.code.toString(16).toUpperCase()}`);
  }
};

Message Channels (Custom App Data)

const channels = session.availableMessageChannels;
for (const ch of channels) {
  if (ch.status === MessageChannelStatus.Ready) {
    ch.sendServerMessage(new Uint8Array([...]));
    const msg = await ch.receiveMessage();
  }
}

Proxy / HTTPS Setup

For headsets requiring HTTPS (Pico 4 Ultra) or production:

createSession({
  serverAddress: 'proxy.example.com',
  serverPort: 443,
  useSecureConnection: true,
  signalingResourcePath: '/192.168.1.100', // target server behind proxy
  // ...
});

See the WSS proxy sample for a Docker-based HAProxy setup.

Desktop Development with IWER

On desktop Chrome (no headset), both samples auto-load IWER (Immersive Web Emulation Runtime) via loadIWERIfNeeded(). IWER emulates a Meta Quest 3 so requestSession('immersive-vr') succeeds and sendTrackingStateToServer() sends valid poses. The device is exposed as window.xrDevice for programmatic access.

IWER's XRDevice API supports programmatic control — set window.xrDevice.controlMode = 'programmatic' to take over from DevUI, then set headset/controller position, orientation, button values, and thumbstick axes from code. Call notifyStateChange() after changes to keep DevUI in sync. See agent-api-reference.md for the full IWER XRDevice API and AGENTS.md for operational rules on IWER automation.

Samples Overview

SampleStackLocation
Simple WebGLVanilla WebGL2 + WebXRsimple/
ReactR3F + @react-three/xr + uikitreact/
WSS ProxyDocker + HAProxyproxy/

Run any sample: npm install && npm run dev in its directory. For Isaac Teleop, see github.com/NVIDIA/IsaacTeleop.

Common Pitfalls

  1. Resolution validation failsperEyeWidth must be multiple of 16, perEyeHeight multiple of 64, both ≥ 256
  2. Black screen in R3F — forgot autoClear = false or missing WebGL state save/restore
  3. Connection fails on headset — check HTTPS requirement (Pico needs WSS), firewall ports: 49100 (WS signaling), 47998 (WebRTC media), 48322 (WSS signaling TLS proxy), or 443 with external proxy
  4. Choppy tracking — enable enablePoseSmoothing, tune posePredictionFactor
  5. Shared WebGL context corruption — always use onWebGLStateChangeBegin/End delegates

Additional Resources

© 2026 YourAI.tools. Every skill from an identity-verified publisher.

Independent catalog. Not affiliated with, endorsed by, or sponsored by Anthropic or any listed publisher. All trademarks belong to their respective owners.