Transforming a conceptual simulation idea into a fully working, robust application has traditionally been a complex and labor-intensive engineering challenge. Developers typically must meticulously assemble disparate assets, bridge the gap between physics engines and rendering pipelines, and rigorously test whether a virtual scene behaves as intended under various physical constraints. Today, however, that paradigm is undergoing a massive shift. Engineers are actively combining frontier artificial intelligence models with NVIDIA Omniverse libraries to streamline this demanding workflow, building sophisticated applications capable of exploring intricate scenarios, investigating mechanical failures, and refining real-world designs with unprecedented speed.
Rather than manually coding every parameter, developers are increasingly directing AI agents through natural-language instructions, carefully reviewing the generated results, and guiding iterative changes in real time. NVIDIA Omniverse libraries provide the underlying powerhouse for this workflow, supplying GPU-accelerated physics, advanced rendering, and high-fidelity sensor simulation capabilities. By leveraging these tools, teams across NVIDIA and the broader ecosystem are producing innovative simulation projects that showcase the practical power of frontier AI models like GPT-6 Astra and Claude Fable 5, paving the way for the next generation of virtual prototyping and automated testing environments.
Build a Humanoid Simulator for a Warehouse Environment
Before organizations can safely deploy automated systems to handle complex warehouse tasks, developers require interactive simulation environments to explore task behavior and thoroughly evaluate how the work gets executed. To address this need, Frank DeLise, an Omniverse product manager at NVIDIA, utilized the Astra AI agent to transform a pre-built SimReady warehouse environment and a humanoid robot into an interactive simulator featuring both first- and third-person views for gamified, physics-based control.
DeLise directed the AI agent to connect various NVIDIA Omniverse libraries, integrating modules for physics via ovphysx, scene updates through ovstage, rendering handled by ovrtx, and the user interface powered by ovui. By combining Astra with the simready-foundation asset library, the system successfully generated the physical scene within the simulation. Astra then automatically generated the necessary animation and application code to unify these disparate capabilities, demonstrating how developers can rapidly spin up complex robotic training grounds using conversational prompts.
Connect an Autonomous-Driving Testing Workflow
When developing self-driving vehicles, altering a single scene asset, modifying a sensor configuration, or updating a driving model can drastically and unpredictably affect autonomous-vehicle simulations. To study and mitigate these cascading impacts, Doyub Kim, a manager on the simulation technology team at NVIDIA, tasked Astra with building an environment known as Zero to Alpamayo, which serves as a reusable simulation testing ground modeled after San Francisco’s bustling Market Street.
Kim directed the AI agent to map out the overarching workflow step by step, subsequently connecting asset creation, simulated traffic patterns, Omniverse RTX sensor simulation, and Alpamayo driving models in distinct stages while verifying each integration. The resulting prototype delivered a powerful testing environment for comparing different driving models and tracing precisely how scene or sensor modifications impact downstream vehicle behavior. In a complementary experiment, a Cosmos3-Nano setup varied dynamic weather and lighting conditions within recorded simulation video feeds, enabling Kim to closely compare how the driving model responded to identical physical scenarios under vastly different environmental circumstances.
Use Sensor Differences to Create and Improve Digital Twins
To effectively test robots and autonomous vehicles in virtual spaces, developers must know with high fidelity how closely simulated sensors match their real-world counterparts. Addressing this challenge, Ashley Reid, who works on RTX sensor validation at NVIDIA, directed Astra and Claude Fable 5 agents to systematically compare ovrtx camera and raw LiDAR outputs against actual recorded data. Through this collaborative effort, the AI agents successfully created two digital twins entirely from scratch while substantially improving two existing ones.
Over a period of approximately three days, Reid guided an iterative workflow where the agents measured discrepancies between the real and virtual data, created or modified OpenUSD scenes, and verified the updated results. The implemented changes directly resolved missing objects, corrected flawed geometry, and adjusted material properties, with final acceptance hinged strictly on meeting rigorous camera and LiDAR metrics. This methodology provides developers with a structured, metric-driven approach to leverage measured real-world discrepancies for guiding scene creation and ongoing digital twin enhancement.
Robo Olympics: Test Robot Skills With Simulation
Teaching advanced robots new physical movements requires verifying whether those dynamic actions can successfully execute under strict physical constraints. Tae Kim, who leads NVIDIA Omniverse engineering and product, leveraged sports reference videos alongside natural-language instructions to guide Astra in building Robo Olympics, an experimental project designed to test simulated Unitree G1 humanoids as they perform various sports movements.
Under Kim’s direct supervision, Astra built specialized controllers and continuously refined them through repeated physics trials. The system relied on the Newton Physics Engine to simulate complex physical behavior, the open-source NVIDIA Warp framework to accelerate heavy mathematical calculations, and ovrtx to render the surrounding scenes and virtual-camera perspectives. In one notable experiment, the humanoid robot successfully cleared a single hurdle in 64 out of 100 simulation trials, providing Kim with valuable performance feedback necessary to fine-tune the robot’s timing and motion control policies.
Test Robotic Disassembly With Computer-Aided Design and Simulation
Before a robot can be deployed to dismantle a product or piece of machinery, developers must verify whether its mechanical tools can successfully reach, grasp, and remove internal components. Jens Jebens, a senior product manager for OpenUSD at NVIDIA, directed Astra to model a car suspension system originally designed in PTC Onshape and correctly configure it within NVIDIA Isaac Sim.
Working alongside Astra, Jebens explored computer-aided design and tooling revisions for robots informed entirely by simulation insights. The AI agent measured the available physical space and designed a custom wrench that the robot could effectively maneuver to reach the suspension system’s tightly packed bolts. Jebens subsequently reported the successful removal of a suspension component entirely within simulation. This approach successfully connects upfront engineering design and tooling decisions directly to physical disassembly outcomes, providing a solid foundation for subsequent robot policy training.
Bring the International Space Station Into the Browser
Translating complex 3D engineering models into a fully accessible web application requires seamlessly connecting static assets, live data feeds, and an intuitive user interface. Nic Johns, an engineering director at NVIDIA, prompted Astra to assemble raw NASA assets into a comprehensive OpenUSD model of the International Space Station equipped with live operational telemetry. Johns built the functional application using a single prompt, and then followed up with a secondary instruction to shift the virtual scene to Earth’s daytime side so that the planet remained clearly visible in the background.
The underlying workflow utilized Blender for initial asset preparation alongside core Omniverse libraries handling advanced rendering via ovstage, scene runtime management, and live streaming capabilities via ovstream. The resulting application successfully brings intricate 3D models and real-time operational data directly into standard web browsers, with Johns steering the developmental trajectory entirely through conversational prompts and targeted corrections.
Turn Captured Rooms Into Testing Environments
Before developers can effectively test robotic interactions within a digitally reconstructed room, the virtual space must feature fully editable objects and accurate physical behavior. Chirag Majithia, a member of the Isaac engineering applications team at NVIDIA, directed Astra to convert real-world stereo camera captures into a fully editable OpenUSD studio environment.
The reconstruction workflow combined PyCuSFM, FoundationStereo, and nvblox for initial spatial capture, relying on human review to guide object selection and placement. Astra then intelligently assembled both generated assets and Blender-authored content, utilizing specialized USD Content Agents to configure how objects move, collide, and interact within the simulation. Subsequent Isaac Sim tests guided necessary collision and contact revisions for interactive elements like doors and drawers. This studio environment bridges captured physical geometry with rigorous interaction testing, making spatial gaps and object behavior significantly easier to inspect and refine.
No comments yet. Be the first to join the conversation!