Summary：NVIDIA and UC Berkeley have introduced T-Rex, a tactile foundation model that uses a decoupled encoding architecture to resolve performance degradation in VLA systems when integrating touch sensing. Built on a dataset of 7,700+ real robotic interactions and 200+ fine manipulation tasks, the model improves operational stability in handling soft and fragile objects by separating low-frequency visual planning from high-frequency tactile feedback, boosting success rates by around 30% and enabling deployment in humanoid and industrial robotics.
Recently, NVIDIA and the University of California, Berkeley jointly released a new tactile-responsive dexterous manipulation foundation model called T-Rex. The model addresses a key industry challenge in traditional Vision-Language-Action (VLA) robotics systems, where integrating tactile feedback often leads to a significant drop in overall performance. To solve this, the team introduced an innovative architecture featuring a decoupled tactile encoding channel.
In existing general-purpose robotic VLA models, mismatches in sensor frequencies between vision and touch, as well as cross-modal feature interference, mean that simply adding tactile sensor data can substantially reduce task success rates. T-Rex instead adopts a hybrid Transformer Mixture-of-Experts asynchronous parallel structure, separating a low-frequency visual planning branch from a dedicated high-frequency tactile expert branch. It also incorporates a spatiotemporal tactile VQ-VAE encoder to independently process millisecond-level force, slip, and contact surface deformation signals. This design ensures that visual understanding and real-time tactile correction operate without interference, while fully preserving the model’s original visual-language capabilities.
The research team constructed a multimodal tactile dataset containing over 7,700 real-world robotic interaction trajectories, covering more than 200 fine-grained manipulation tasks such as wrapping, twisting, and sliding, and including hundreds of everyday objects with varying properties such as soft, rigid, fragile, and deformable materials. Experimental results show that in challenging tasks such as soft-object grasping, precision assembly, and anti-slip manipulation, the model achieves an average improvement of 30% in task success rate compared to mainstream VLA baselines.
The framework is compatible with widely used commercial tactile sensors, including GelSight and miniature six-axis force sensors. It can be directly deployed on NVIDIA’s Isaac humanoid robotics development platform, enabling a general-purpose dexterous manipulation solution for humanoid service robots and precision industrial manipulators through a “vision perception + tactile closed-loop control” system. It also provides a standardized data and encoding foundation for scalable training of future tactile foundation models.
