NVIDIA Highlights How Frontier AI Models and Omniverse Libraries Accelerate Simulation Development

Transforming a conceptual simulation idea into a fully working, interactive software application has traditionally required a painstaking choreography of assembling assets, bridging physics engines with advanced rendering pipelines, and meticulously verifying that virtual scenes behave according to physical laws. Today, developers are drastically cutting down development time by combining frontier artificial intelligence models with NVIDIA Omniverse libraries. This synergy enables engineers to build sophisticated applications capable of exploring complex scenarios, investigating mechanical failures, and systematically improving physical designs through natural language instructions.

The methodology relies on developers directing advanced AI agents using conversational instructions, reviewing the generated output, and guiding iterative refinements. Meanwhile, underlying NVIDIA Omniverse libraries provide the heavy lifting, delivering GPU-accelerated physics, real-time rendering, and high-fidelity sensor simulation capabilities. By streamlining these technical workflows, NVIDIA teams and external ecosystem developers are demonstrating how artificial intelligence can rapidly prototype environments that previously demanded weeks or months of manual coding and integration work.

Build a Humanoid Simulator for a Warehouse Environment

Before automating complex tasks within industrial warehouses, developers require robust, interactive simulation environments to observe task behavior and evaluate operational workflows. Frank DeLise, an Omniverse product manager at NVIDIA, utilized the Astra AI model to transform a pre-existing SimReady warehouse environment and a humanoid robot model into a fully interactive simulator equipped with both first- and third-person viewing modes.

To achieve this, DeLise directed the AI agent to connect various NVIDIA Omniverse libraries, including ovphysx for physics calculations, ovstage for dynamic scene updates, ovrtx for advanced rendering, and ovui for managing the user interface. By combining Astra with the simready-foundation toolset, the system successfully generated the physical scene in simulation. Astra subsequently produced the required animation and application code to bind these diverse capabilities into a cohesive, gamified, and physics-based control environment for humanoid robotics.

Connect an Autonomous-Driving Testing Workflow

Modifying an individual scene element, adjusting a sensor configuration, or swapping out a driving model can introduce massive ripple effects in autonomous-vehicle simulations. To address this complexity, Doyub Kim, a manager on the simulation technology team at NVIDIA, tasked Astra with constructing a reusable simulation environment modeled after a familiar urban landscape: San Francisco’s Market Street, known as the Zero to Alpamayo project.

Kim instructed Astra to map out the entire workflow and incrementally connect asset creation, traffic generation, Omniverse RTX sensor simulation, and Alpamayo driving models in distinct stages, verifying each integration along the way. The resulting prototype serves as a comprehensive testing ground for comparing different driving models and tracing how alterations to the surrounding scene or onboard sensors directly influence downstream vehicle behavior. In a complementary Cosmos3-Nano experiment, researchers varied weather and lighting conditions within recorded simulation videos, allowing Kim to rigorously evaluate how the driving model responds to identical scenarios under vastly different environmental circumstances.

Use Sensor Differences to Create and Improve Digital Twins

Evaluating the reliability of robots and autonomous vehicles depends heavily on understanding how closely simulated sensors match their real-world counterparts. Ashley Reid, who works on RTX sensor validation at NVIDIA, directed Astra alongside Claude Fable 5 agents to compare ovrtx camera and raw LiDAR outputs directly against recorded real-world data. Through this collaborative effort, the AI agents successfully built two digital twins entirely from scratch and substantially improved two existing ones.

Over the course of a three-day period, Reid guided an iterative development cycle where the agents continuously measured discrepancies, generated or modified OpenUSD scenes, and verified the updated outputs. The adjustments targeted missing objects, complex geometry, and material properties, with acceptance criteria determined strictly by camera and LiDAR performance metrics. This approach provides developers with a structured pathway to use measured physical discrepancies as a guiding mechanism for refining digital environments.

Robo Olympics: Test Robot Skills With Simulation

Teaching advanced robots new physical movements requires rigorous validation to ensure those actions remain viable under real-world physical constraints. Tae Kim, who leads NVIDIA Omniverse engineering and product, leveraged sports video footage and natural-language instructions to guide Astra in developing Robo Olympics, an experimental project designed to test simulated Unitree G1 humanoid robots performing specialized athletic maneuvers.

Operating under Kim’s direct supervision, Astra constructed specialized controllers and refined them through continuous physics trials. The system relied on the Newton Physics Engine to simulate behavioral mechanics, the open-source NVIDIA Warp framework to accelerate heavy calculations, and ovrtx to render scenes and virtual camera feeds. In one notable trial, the humanoid robot successfully cleared a single hurdle in 64 out of 100 simulation runs, providing Kim with valuable performance feedback necessary to fine-tune the robot’s movement timing and motion control policies.

Test Robotic Disassembly With Computer-Aided Design and Simulation

Before deploying a robot to dismantle manufactured products, developers must confirm whether the robot’s mechanical tools can effectively reach and remove individual components. Jens Jebens, a senior product manager for OpenUSD at NVIDIA, directed Astra to model a vehicle suspension system originally designed in PTC Onshape and configure it within NVIDIA Isaac Sim.

Through this integration, Jebens explored how computer-aided design and tooling revisions could be directly informed by physics simulation. The AI agent measured the restricted physical space surrounding the assembly and intelligently designed a custom wrench that the robot could successfully maneuver to reach the suspension’s stubborn bolts. Jebens subsequently reported the successful simulated removal of the suspension component, effectively bridging the gap between engineering design decisions and robotic disassembly outcomes, while establishing a firm foundation for subsequent robot policy training.

Bring the International Space Station Into the Browser

Translating complex 3D models into a functional web application demands seamless integration between static assets, live telemetry data, and an intuitive user interface. Nic Johns, an engineering director at NVIDIA, prompted Astra to assemble raw NASA assets into an OpenUSD-based model of the International Space Station complete with operational telemetry. Johns constructed the entire application using a single initial prompt, then utilized a follow-up prompt to rotate the scene so that the daylight side of Earth became clearly visible in the background.

The underlying technical workflow utilized Blender for initial asset preparation alongside Omniverse libraries for real-time rendering via ovrtx, scene runtime management through ovstage, and video streaming via ovstream. This application successfully brings intricate 3D models and live operational data directly into standard web browsers, with Johns steering the development process entirely through conversational prompts and targeted corrections.

Turn Captured Rooms Into Testing Environments

Digitally reconstructed rooms require fully editable objects and accurate physical behavior before developers can safely test robot interactions within them. Chirag Majithia, from the Isaac engineering applications team at NVIDIA, directed Astra to transform raw stereo camera captures into a fully editable OpenUSD studio environment.

The reconstruction workflow integrated PyCuSFM, FoundationStereo, and nvblox to process the spatial data, while human review guided the selection and placement of individual objects within the scene. Astra subsequently assembled a mixture of generated and Blender-authored assets, employing USD Content Agents to configure how various objects move and interact within the simulation. Isaac Sim tests then drove necessary collision and contact revisions for articulating elements like doors and drawers. By connecting captured real-world geometry directly to interactive testing pipelines, developers can more easily inspect environmental gaps and evaluate realistic object behaviors.