Mistral unveils AI model that navigates robots using single camera
What's the story
French AI start-up Mistral has unveiled its first-ever model for embodied navigation, dubbed Robostral Navigate. The 8B model uses RGB images and plain-language instructions to guide a robot through complex environments. Unlike other models that rely on multiple sensors or cameras, Robostral Navigate only requires a single RGB camera to operate. The new tech has been designed to let robots navigate through complicated spaces like offices, homes, commercial buildings, and outdoor areas without human intervention.
Performance
Mistral's AI outperforms other models in unseen environments
Robostral Navigate has a success rate of 76.6% on the unseen Room-to-Room in Continuous Environments (R2R-CE) benchmark, which tests instruction-following capabilities in environments not seen during training. This beats the best single-camera approach by 9.7 points and even outperforms systems using depth or multiple cameras by 4.5 points, despite not using any of them. The model was built entirely in-house with simulated data and token-efficient techniques, making it adaptable to real-world obstacles that weren't seen during training.
Versatility
Can be used in various fields and on different robots
The new AI model can be used in various fields like manufacturing, delivery, logistics, and hospitality. It can work on wheeled, legged, and flying robots of different sizes. The model is also robust to differences in camera intrinsics and uses prefix-caching for token-efficient training. After the supervised training stage, Robostral Navigate's performance is further improved using CISPO (an online reinforcement learning algorithm), enabling it to learn from trial-and-error experiences effectively.
Innovation
Uses 'pointing' for navigation
Robostral Navigate uses a unique method called "pointing" for navigation. Given a task and observation history, it predicts where the robot should move next by inferring image coordinates of the target location in the current camera view. However, if the target location is outside its current field of view, it uses displacements in its local coordinate frame as a fallback.
Development
Trained using an efficient prefix-caching algorithm
The model was trained using an efficient algorithm based on prefix-caching. This method compresses an entire episode into a single sequence, allowing training on all time steps in one forward pass while preventing information leakage between time steps. Compared to training with one sample per time step, this approach reduces the number of tokens by 22 times while preserving all learning signals.