Recipes/Live 3D (URDF rendering)

Recipe: Live 3D — rendering a robot from its URDF

Three traps this recipe exists to keep closed:

prepareUrdfScene used to match only the literal package://… string, while URDFLoader’s own default packages config rewrites that to a root-relative path first — so every mesh was silently fetched from the app’s own origin instead of through the SDK. Fixed in the SDK (the URL modifier registers both forms), not worked around here. It still matters if you configure URDFLoader.packages yourself; see step 3 and “Common mistakes”.
An unmapped reference used to fall through the URL modifier to the original string. For an ordinary unsynced package:// URI that string is unfetchable and harmless — but a hostile <mesh filename="http://attacker.example/x.stl"> (or an off-origin <texture>) was fetched by the browser exactly as written, no credential attached: a network beacon fired by every viewer who renders that robot’s URDF. Now refused — the fallback is a shared, empty, page-local blob: URL, which cannot resolve off-page by construction.
A .dae’s internal reference written as ./textures/skin.png or ../textures/skin.png — ordinary exporter output, not an edge case — used to miss the direct name lookup because asset.name stores the normalized form. It is now tried a second time through the same normalization the naming rule uses.

All three are prepareUrdfScene internals; this recipe adds no code for them.

What this is, and what it deliberately is not

“Live-3D” is a documented recipe, not a Fleetless feature. There is no 3D viewer anywhere in the platform, and there will not be one — the asset store gives you a URDF, its meshes, and a joint-state datapoint; rendering them is your app’s job, using ordinary web 3D tooling (three.js + urdf-loader). This page is that recipe, written once so every app does not reinvent it slightly differently.

Textures are covered, not only geometry. A <material>'s own <texture>, and an image a .dae references internally, both go through the same authenticated path as a mesh, and an unauthenticated fetch of either is refused.

Why a plain `<img>` or the default three.js/urdf-loader setup will not work

Every asset byte on Fleetless is served behind the same rule as everything else on the platform: the Authorization header, and nothing else. Not a signed URL, not a cookie, not a token in the query string — one mechanism, so assets inherit token renewal, role checks, audit and quota from what already exists instead of growing a second credential with its own lifetime.

The consequence is mechanical, not a design opinion: neither an <img src="..."> tag nor urdf-loader’s default mesh/texture loading can set a header. They issue a plain, unauthenticated GET, which the cloud correctly refuses. This is why the SDK carries this glue itself — it is not a convenience, it is the only way this ever works. Writing your own fetch-with-header code is possible; the SDK ships it so every app does not write it, and each one slightly differently.

Two mechanisms exist, and they answer two different questions — do not install both on one load.

assets.prepareUrdfScene — one call that intercepts every load a LoadingManager issues: meshes, a <material>'s own <texture>, and an image a mesh file (e.g. a .dae) references internally. This is the one to reach for by default, and the one this recipe now uses.
assets.createMeshLoader — the original, mesh-only, renderer-agnostic callback (any loadMeshCb-shaped consumer, not only urdf-loader/three.js). Still supported, unchanged, for apps that render meshes without materials or textures at all.

They consume different URDF texts (raw vs. cloud-rewritten, see step 2) — pick one per scene. Combining them does not double-fetch, which is the guess to resist: createMeshLoader as loadMeshCb intercepts mesh loading entirely, bypassing the LoadingManager (and therefore prepareUrdfScene’s URL modifier) for meshes but not for textures — so the combination gets one half right and one half wrong, not everything fetched twice. A developer told to expect double-fetching debugs the wrong symptom when a mesh or a texture simply comes back empty instead.

The four steps

1. Sync the URDF — a developer action, once, in the console

A connected bridge with /robot_description reports availability only — nothing transfers automatically. Your app never triggers a sync itself: POST /api/robots/:id/assets/sync requires a developer session at Owner tier, because a sync spends the org’s asset quota. Before your app can render anything, an Owner opens the robot’s Assets tab in the console and clicks Sync.

The tab shows one of three states, and they are genuinely different facts:

URDF available, not synced — a bridge is connected and sees a URDF, but nothing has moved yet.
N meshes missing — synced, but the URDF references package:// paths the bridge could not resolve in its workspace. The console names each one; matching them against your robot’s workspace is a one-time fix, not something your app can work around.
Synced — everything the URDF references is in the store.

Your app only ever sees the last state as “ready to render.” Treat the first two as “nothing to render yet,” and surface missing, if you read it, as a setup problem for the robot’s owner — not a bug in your viewer.

2. Read the completeness report

const completeness = await client.assets.list(robotId)
// completeness.urdf: { present: boolean, mesh_count: number, missing: { uri: string, element: 'mesh' | 'texture' }[] }
// completeness.urdf_available: boolean | null — a connected bridge's own
//   report, distinct from `present`. `null` means no bridge is online to ask.

if (!completeness.urdf.present) {
  // Nothing synced yet — show your app's own "not ready" state, do not
  // treat this as an error.
  return
}

The URDF text itself is fetched in step 3, as part of prepareUrdfScene — not here. There are two ways to get URDF text, and they are not interchangeable: client.assets.urdf(robotId) returns it with every package:// mesh/texture URI already rewritten to an absolute Fleetless asset URL (a full https://... URL, not relative to your app’s own origin — assume otherwise and you’ll get 404s from your own web server, not Fleetless); prepareUrdfScene (step 3) hands you the raw text instead, package:// URIs intact, because that is what its URL modifier needs to resolve a .dae’s internal references correctly (see step 3). Reach for assets.urdf() directly only if you’re rendering with createMeshLoader instead, or displaying the XML for a reason that isn’t rendering.

Do not paste a rewritten mesh URI into a browser address bar to check it. It will look broken — a plain GET from the address bar carries no Authorization header, so you’ll see the same refusal an <img> tag gets. This is the first thing anyone on this recipe tries when a mesh doesn’t render, and the result looks exactly like a dead link. It isn’t one; verify with an authenticated request (your app’s own fetch, or curl -H "Authorization: Bearer ..."), never a browser tab.

3. Render it: `urdf-loader` + `prepareUrdfScene`

Use urdf-loader@^0.13.0 or later. loadMeshCb/defaultMeshLoader gained a material parameter in 0.13; ^0.12.x has an older three-argument shape (path, manager, done, no material) and will not line up; verified against both versions’ published source, not assumed from the changelog. urdf-loader also ships no TypeScript types and there is no @types/urdf-loader — write a small local .d.ts shim. This much is enough for everything below:

declare module 'urdf-loader' {
  import type { LoadingManager, Object3D } from 'three'
  type Done = (obj: unknown | null, err?: Error) => void
  export interface URDFRobot extends Object3D {
    setJointValue(joint: string, value: number): boolean
  }
  export default class URDFLoader {
    constructor(manager?: LoadingManager)
    packages: string | Record<string, string> | ((pkg: string) => string)
    loadMeshCb: (path: string, manager: unknown, material: unknown, done: Done) => void
    parse(content: string, workingPath?: string): URDFRobot
  }
}

import URDFLoader from 'urdf-loader'
import { LoadingManager } from 'three'

const manager = new LoadingManager()

// `prepareUrdfScene` installs a `setURLModifier` on `manager` that answers
// every load the loader below issues — a mesh, a `<material>`'s own
// `<texture>`, or an image a `.dae` references internally via `<init_from>`
// — with a pre-fetched, `Authorization`-headed blob URL, never a live
// network request. It returns the URDF **raw**: `urdfText` is what step 2's
// `assets.urdf()` would give you before rewriting, because three.js
// computes a `.dae`'s own base directory from the URL it was *originally*
// asked to load, before any URL modifier runs — feed it the rewritten
// (opaque asset-URL) form instead and that resolution breaks. `missing`
// carries the same entries `completeness.urdf.missing` does — `{ uri,
// element }`, not bare strings: show `uri`, and branch on
// `element` if you want to tell a missing mesh from a missing texture.
// `dispose()`
// revokes every blob URL and resets the modifier to identity — call it when
// the scene is torn down, or your app leaks one object URL per asset for as
// long as the tab lives.
const { urdfText, missing, dispose } = await client.assets.prepareUrdfScene(robotId, manager)

// No `loadMeshCb` assignment needed, and no `loader.packages` configuration
// either: the URL modifier above registers both the literal `package://…`
// name and `URDFLoader`'s own default root-relative rewrite
// (`package://pkg/rel` → `/pkg/rel`, which `resolvePath` applies *before*
// `loadMeshCb`/`manager.resolveURL` ever run — see "Common mistakes" below),
// so a plain `new URDFLoader(manager)` resolves correctly out of the box.
// That also covers `defaultMeshLoader`'s internal dispatch to three.js's
// `ColladaLoader` for `.dae` meshes and that loader's own texture fetches.
// Nothing here is format-specific — your app never sees raw bytes.
const loader = new URDFLoader(manager)

// `parse` is synchronous — it returns the assembled robot directly. Easy to
// get wrong once: `urdf-loader`'s own `load(url, onLoad)` (fetches a URL,
// callback-style) looks similar enough to invite passing a callback here
// too, but you already have the raw XML text above, so `parse` is the one
// to call, and it does not take one.
const robot = loader.parse(urdfText)
scene.add(robot)

// ... when the scene is torn down:
dispose()

missing is not fatal — a partial robot is still worth rendering; surface it as a setup problem for the robot’s owner (same advice as step 1’s completeness.urdf.missing), not a bug in your viewer.

Mesh-only, no textures, any renderer: use createMeshLoader instead — see the SDK reference for the exact shape. It wraps urdf-loader’s defaultMeshLoader the same way prepareUrdfScene wraps the whole LoadingManager, but only for meshes, and it consumes the rewritten URDF from assets.urdf(), not the raw text above. Don’t mix the two on one load — see the note above step 1.

Whichever mechanism you use, do not swap in the default loadMeshCb unwrapped, and do not point urdf-loader at raw asset URLs directly (e.g. via its packages option) — both paths end in an unauthenticated fetch that the cloud refuses.

4. Drive it from `/joint_states`

Joint state is an ordinary datapoint — typically a whole-topic subscription on /joint_states, nothing asset-specific about it:

client.datapoints.subscribe(robotId, 'joint_states', {
  onEvent(event) {
    const { name, position } = event.value as { name: string[]; position: number[] }
    name.forEach((jointName, i) => {
      robot.setJointValue(jointName, position[i])
    })
  },
  onError(error) {
    console.error(error.code, error.message)
  },
})

robot.setJointValue is urdf-loader’s own API on the object loader.parse handed you — nothing Fleetless-specific here either. This is the whole point of treating Live-3D as a recipe: steps 3 and 4 are exactly what you would write against any URDF from any source, once prepareUrdfScene (or createMeshLoader) has solved the one part that is Fleetless-specific.

“Historical pose” — the same recipe, a different datapoint source

“Historical pose” is not a second feature either: replay /joint_states from client.datapoints.history(...) instead of subscribe, and drive setJointValue from the replay timeline instead of live events. Steps 1–3 above are unchanged.

const { samples, truncated, truncated_by } = await client.datapoints.history(
  robotId,
  'joint_states',
  { from: 'now-30s', to: 'now' }, // no `aggregate` — see below
)

for (const sample of samples) {
  const { name, position } = sample.value as { name: string[]; position: number[] }
  // schedule this against sample.timestamp_ms — real bridge capture times
  //, not an assumed fixed frame interval; see the rate note below.
}

Three things worth stating plainly, because a recipe that glosses over them promises more than retention delivers:

Recording is opt-in, per datapoint, and off by default. If /joint_states was never switched to “record” in the config editor, history() throws not_recorded — not an empty array. Don’t build a “no history in this window” UI from that error; it means recording was never turned on, and the fix is in the config, not the query.
Use raw samples, not aggregate, for pose. window/agg averages a single numeric field (min/max/avg) — meaningless for a whole joint array, and the route answers not_aggregatable if you try. That is the correct refusal, not a bug to work around: pose replay wants every recorded joint-state message intact, which is exactly what leaving aggregate out gives you.
truncated/truncated_by are not optional to check. A long window or a many-joint robot can hit the row limit ('limit', default 1000, cap 10 000) or the byte limit ('bytes') before covering the range you asked for — and the response says so honestly rather than silently handing back a shorter replay. A recipe that ignores this plays a truncated window as if it were the whole one. Handle it: page through smaller from/to windows, or tell the person watching that the replay only covers part of the range.

There is no promise of a smooth playback rate, and none is needed: timestamp_ms on each sample is real bridge capture time, so a replay steps through actual recorded instants — via those timestamps, not a fixed frame interval — the same way any keyframed animation would.

One more grant to have, separate from assets: /joint_states’s history (like its live subscription) is gated by the ordinary role × service matrix on that specific slug, checked independently of the assets capability that gates the URDF and meshes. A role can have one without the other — granting assets does not imply the robot’s other datapoints are readable, and vice versa.

Common mistakes, named up front

Meshes silently 404 against your own app’s origin if you set a custom URDFLoader.packages mapping. prepareUrdfScene registers both the literal package://… name and URDFLoader’s own default root-relative rewrite, so the default packages config (i.e. not setting it at all) resolves correctly out of the box. It is still a trap if you configure packages yourself (a string, object map, or function producing anything other than those two forms): whatever path your mapping produces has to be one prepareUrdfScene actually registered, or the same silent-404 failure reappears for your custom form.
A shared cache serving a stale or wrong-robot mesh. Assets answer Cache-Control: private, immutable and never public on purpose — this is not a CDN, and nothing in front of the browser should be caching these responses. If you put your own reverse proxy or CDN in front of your app and it touches these URLs, you have reintroduced the exact failure mode the header exists to prevent.
Assuming a 403 on an asset means it does not exist. Same rule as everywhere else on the platform: a role without the assets capability gets the same refusal as a request for something that is not there. Don’t build a “this robot has no URDF” UI state from a 403 alone — check assets.list’s urdf.present first.
Re-fetching an identical mesh per robot. The store deduplicates by content hash (sha256 on Asset) across robots in the same org, but that is a storage optimization, not a caching one — your app still issues one authenticated request per mesh URL it is handed. If you render a fleet of identical robots, consider caching parsed geometry client-side, keyed by sha256, rather than assuming the platform does it for you.