Reference/fleetless.yaml/Live Video

📦 Live Video

The bridge sends live video as RTP packets. This section says how large one may be and what encodes the video.

It holds for the whole robot: the packet size is a property of the transport and the encoder is part of the computer, and neither belongs to one camera. A key here overrides the bridge’s ROS parameter of the same name; an absent key leaves the parameter alone.

yaml
live:
  max_packet_bytes: 1200
  encoder: software

max_packet_bytes is bridge 6.3.0 and up, encoder 6.4.0 and up. An older bridge ignores what it does not know: before 6.3.0 it sends packets of up to 1326 bytes, which a path with an MTU of 1280 does not carry, and before 6.4.0 it encodes on the CPU whatever this section says.

The keys

Key Default Bounds Meaning
max_packet_bytes 1200 integer, 576 to 1350 The largest RTP packet the bridge sends.
encoder software software, auto, nvenc, vaapi or v4l2 What encodes live video.

The packet size

The number counts the packet before encryption: fixed header, header extensions and payload. Retransmissions are included.

It is not an MTU. The MTU belongs to the path to each viewer, and the robot does not know that path. On the wire a packet is up to about 80 bytes longer than the setting: the header extensions the media server may add, the SRTP tag, UDP, IP and, through a relay, TURN.

A change holds from the next frame, for running streams too. Nobody reconnects.

Why 1200

It is what browsers send. With those 80 bytes on top it fits a path of 1280, the smallest MTU IPv6 guarantees and the one NetBird and Tailscale set.

A packet that does not fit is not trimmed on the way. It is dropped, and the viewer gets a track that never shows a picture: the short last packet of each frame arrives, every full one is lost.

When to lower it

Lower it when a viewer’s path carries less than 1280 bytes, which is what a tunnel inside a tunnel leaves: each one takes its own header. Take the smallest MTU on the path and subtract 80.

Smaller packets mean more of them, each with its own headers, so go as low as the path needs and no lower.

The ceiling is 1350 because the bridge’s SRTP library protects no packet above 1356 bytes. A larger setting would not be a larger packet. It would be no video.

Trying a value on the robot

The key exists as a ROS parameter on the bridge node, so a value can be tried without a publish:

sh
ros2 param set /fleetless_bridge live.max_packet_bytes 1000

A value outside the bounds is refused by the set, with the rule, and the parameter keeps what it had. In this file the editor marks it as you type and publish refuses it, so it never reaches the robot.

The key in this section wins for as long as it is there. Publish a configuration without it and the parameter is in force again; without either, the bridge sends 1200.

The encoder

By default the bridge encodes live video on the CPU. A robot with an encoder chip can use that instead.

encoder What encodes GStreamer element, tried in this order
software the CPU: VP8 or H.264, whichever the media server picks none
nvenc NVIDIA: a Jetson, then a desktop card nvv4l2h264enc, then nvh264enc
vaapi Intel and AMD graphics vah264enc, then vah264lpenc, then vaapih264enc
v4l2 a V4L2 encoder, as on a Raspberry Pi 4 v4l2h264enc
auto the first of nvenc, vaapi and v4l2 that starts as above

software is what the bridge did before the setting existed, so an upgrade changes no robot. Every other value sends H.264 and nothing else, so a viewer’s browser has to decode H.264. Google Chrome does, and is the browser this was shown in; a Chromium built without proprietary codecs does not, and shows no picture.

vah264lpenc is the low-power encoder of the same driver. Intel Arc and Meteor Lake encode H.264 only there, so on them the first chain finds no element and the second one does.

A Raspberry Pi 5 and a Jetson Orin Nano have no hardware encoder. With any value but software they end on software, as described below.

The setting applies at the next live start. A stream that is running keeps the encoder it has. A camera whose source already delivers H.264 and is passed through is not encoded at all, so this key does not reach it.

Even sizes only

With any value but software, every camera’s width and height must be even. H.264 takes no odd size, in a hardware encoder or in software, and a camera with one would get no picture. That holds for a passed-through camera too: it is not encoded, but its stream is H.264 of the configured size.

So it is refused before it gets that far:

  • In this file. A configuration that names such an encoder here and has a camera of odd size is refused, with a finding at the camera’s width or height. The editor shows it among its findings, and publish refuses it.
  • On the robot. When the encoder comes from the bridge’s ROS parameter instead, the cloud cannot see it. The bridge then refuses the live start of a camera it encodes with live_unavailable and a message naming the setting and the size. It logs that once per camera for a setting and a size, and again after either changes. Its snapshots go on, and the robot’s other cameras start as before. A passed-through camera meets its own check instead: its stream cannot have an odd size, so it never matches the configured one.

Either way the fix is an even size, or encoder: software. A camera that becomes even in a new configuration starts at the next attempt; nothing has to be restarted.

What it needs on the robot

The hardware encoders are reached through GStreamer, and the bridge package does not depend on it: a robot that stays on software installs nothing. Every other value needs these packages:

sh
sudo apt install python3-gi gir1.2-gstreamer-1.0 gir1.2-gst-plugins-base-1.0 \
    gstreamer1.0-plugins-base gstreamer1.0-plugins-bad

and the plugin that holds the encoder element:

Element Comes with
nvv4l2h264enc L4T, on a Jetson
nvh264enc, vah264enc, vah264lpenc gstreamer1.0-plugins-bad
vaapih264enc gstreamer1.0-vaapi
v4l2h264enc gstreamer1.0-plugins-good

GStreamer is loaded with the first hardware session after the bridge starts. When its plugin registry is missing or stale, GStreamer then runs its own plugin scanner, gst-plugin-scanner, as a child process, and starts it again for the rest of the scan if a plugin crashes it. That delays that one picture: by 0.2 s for 250 plugins on our development machine.

GStreamer writes the registry to the bridge user’s cache directory: $XDG_CACHE_HOME/gstreamer-1.0/, which is ~/.cache/gstreamer-1.0/ unless that variable is set.

Its libraries may write short-lived files of their own besides. One is known: GStreamer’s orc library creates orcexec.*, maps it and deletes it at once, in $XDG_RUNTIME_DIR, else in $HOME, else in $TMPDIR or /tmp. The bridge’s own process does so at the first session whose chain converts the picture in videoconvert, as every vaapi chain does, on every supported ROS distribution; with liborc 0.4.42 the scanner creates one as well.

The scan happens at the first hardware start, again after an update of a GStreamer package, and at every start of a bridge whose user cannot write the cache directory, unless GST_REGISTRY names a file it can write. Nothing the cloud sends reaches the scanner’s command line: the binary and its arguments are GStreamer’s. A robot on software loads no GStreamer and starts no such process.

When the encoder does not start

A hardware encoder that fails costs no stream. If GStreamer is missing, the element is not there, the device cannot be opened, or the encoder returns nothing within a second for a frame, that live session is encoded in software H.264 instead. Nothing is renegotiated, and the stream goes on.

The session’s first picture waits for the attempt: up to a second for an encoder that returns nothing, up to five for a device that does not open.

The bridge logs one warning with the setting, the element chain it tried and the reason:

live.encoder=nvenc: no hardware encoder could be used (nvenc/jetson cannot be
built: GStreamer has no element 'nvvidconv'; nvenc/desktop cannot be built:
GStreamer has no element 'nvh264enc'). This session is encoded in software
H.264. Until live.encoder changes or the bridge restarts, a chain that cannot
be built is not tried again, and one that did not encode a picture is not
tried again for pictures of that size.

As the last sentence says, the bridge remembers two things, and they are not the same finding.

A chain that cannot be built is about the robot: a package, a driver or the chip is missing. Install what is missing and restart the bridge: GStreamer reads its plugins once, when the bridge first loads it. Where nothing had to be installed, changing the key to another value and back makes the bridge try again; publishing the value it already has changes nothing.

A chain that was built and did not encode a camera’s first picture is remembered for pictures of that size. That camera stays on software, and so does any other of the same size. A camera of another size is still tried on the chip.

An encoder whose driver stops answering costs one late frame, and the session goes on in software. A pipeline that does not shut down within two seconds is left behind, and the bridge says so in a warning of its own. That chain is not built again until the key changes or the bridge restarts.

The fallback catches an encoder that fails, not one that draws a wrong picture. Look at the stream once after choosing an encoder.

Which encoders are proven

None on a robot yet. Every chain is written from the documentation of its GStreamer element.

Chain Proven on hardware
nvenc on a Jetson (nvv4l2h264enc) not yet: the first robot, a Jetson Orin NX, is next
nvenc on a desktop card (nvh264enc) no
vaapi (vah264enc, vah264lpenc, vaapih264enc) no
v4l2 (v4l2h264enc) no

What is proven is everything around the encoder element. The bridge’s test suite runs the whole pipeline, the choice of encoder and the fallback over a software stand-in, on all three ROS distributions.

Treat an encoder as untested until this table says otherwise, and prove it on your machine first:

sh
python3 tools/encoder-proof/encoder_proof.py --backend nvenc

The tool is in the bridge repository, under tools/encoder-proof/. It runs the encoder with a moving pattern and checks what the bridge needs from it: one picture out for one picture in, a keyframe when asked, a bitrate that follows, the picture that went in coming out, and its colours right by the label the stream carries.

Its last line is the verdict. Send its output along if you report an encoder that works, or one that does not.

Bitrate

With a hardware encoder the bridge hands the encoder a camera’s bitrate_kbps over its whole range. The software encoders clamp it: VP8 to 250 to 1500 kbps, H.264 to 500 to 3000 kbps. A session that fell back from a hardware encoder keeps the whole range.

Trying an encoder on the robot

As for the packet size, the key exists as a ROS parameter:

sh
ros2 param set /fleetless_bridge live.encoder nvenc

A name that is not one of the five is refused by the set. Start a live stream, and read the bridge’s log: it says which chain encodes, or why none does.