Build instruments, interactive worlds, reactive visuals, games, and live audiovisual systems with AI agents inside PatchWorld.
PatchXR is opening beta access to agentic patching, a new AI layer that operates directly inside live PatchWorld systems. This framework allows creators to analyze, build, wire, inspect, and debug interactive environments across both virtual reality and desktop.
PatchWorld has always been about making virtual things in a tangible way. In our world, patching means building by connecting modular blocks together. Instead of writing code, creators grab, arrange, and connect blocks that control sound, visuals, logic, physics, motion, and interaction. You can use these blocks to shape everything from individual instruments and electronic devices to entire games and interactive worlds.
Until now, those blocks had to be patched entirely by human hands. Now, AI agents can start patching too.
The agent is not an autonomous director. It is a creative collaborator and assistant operating directly inside the creative loop. Right now, the beta introduces three distinct entry points for working with the AI: external assistants through the Model Context Protocol (MCP), a typed in-app Agent, and a real-time Live Voice Agent inside the headset.
Beyond the Chatbot: Building Living Systems
This is far from a typical chatbot sitting on a screen. The agent operates on the same live world while creators and collaborators watch its changes appear in real time. It can look at what is already in your space, spawn blocks, wire them together, alter parameters, inspect data flows, and help you repair or extend your creation.
PatchWorld already includes Genie Box, a feature that lets creators generate assets like skyboxes, images, sounds, audio samples, and 3D meshes directly in VR using written prompts or voice. Agentic patching works at a completely different level. While Genie Box generates isolated assets, agentic patching connects, configures, inspects, and modifies the underlying systems that determine how a world behaves.
Because the AI works with the actual live building blocks of the experience, the resulting creations remain fully transparent, editable, inspectable, and remixable. It shifts the focus from generating static output to building open systems that you can continue to play, expand, and share.
Watch this quick preview of the agent in action, building a wobbly bass patch from a simple conversation:
This workflow extends far beyond basic music creation. To show what is possible when you treat a world as an open canvas, we asked the agent to build a functional, data-driven model of the solar system from scratch.
Rather than simply importing a static 3D model, the agent spawned individual planetary bodies and arranged them according to their relative positions. It used astronomical reference values to shape their relative scale, distance, orbital movement, and axial rotation.
The build phase does not stop at the initial prompt. From there, the conversation continues naturally. A creator can ask the agent to add major moons, refine the layout of the scene, or reorganize the space. Because PatchWorld is a unified environment, these planetary values can then be connected directly to sound, visuals, modulation, or rhythm. Through this iterative loop, a visual world becomes a system, the system becomes an instrument, and the instrument becomes a live audiovisual experience.
Powering the Desktop to VR Pipeline
Agentic patching introduces a massive workflow improvement for the desktop version of PatchWorld. Previously, our deepest creation tools were centered almost entirely on VR. The desktop version was highly useful for exploring, streaming, and multiplayer sessions, but building complex 3D patches on a flat screen manually was not always fluid.
The agent changes this dynamic, turning desktop into a powerful creative entry point.
The ideal workflow follows a clear hybrid sequence: describe and structure on desktop, step inside and experience it in VR, and continue directing the agent through voice. On a desktop monitor, you can type long, precise prompts with a physical keyboard, input complex reference data, and watch the agent assemble the initial architecture on screen. Once the foundation is set, you can put on your headset to step inside the result, understand the physical scale, test the embodied interactions, and use your voice to keep refining the system.
Three Ways to Collaborate
The underlying tool layer supports three distinct methods for interacting with the AI, letting you choose the right interface for your workflow.
1. External Assistants via MCP
PatchWorld now exposes a Model Context Protocol (MCP) server. This open standard allows compatible external AI clients, such as Claude Desktop, Cursor, or custom development models, to connect directly to a live PatchWorld session with your explicit authorization. The external AI manages the conversation and reasoning loop, while PatchWorld exposes the creative tools and executes the actions in the world. Any compatible MCP-capable agent can potentially connect to the platform, opening PatchWorld up to the broader AI development community.
2. The Typed In-App Agent
Accessible without configuring any external clients, you can type a request directly into the in-world console. You can watch the agent map out its execution logic and see the visual block nodes connect themselves on screen. Because it reads the current scene and understands exactly what is selected, watching it work is an incredibly fast way to learn spatial programming logic.
3. The Live Voice Agent
In an embodied environment like VR, voice is the path of least resistance. The Live Voice Agent allows you to speak naturally to the system while keeping your hands entirely on your controller, instrument, or patch array.
For example, while working on a sound system, you can say, “Make the bassline more wobbly.” The agent identifies the correct modules, modifies the existing patch, and provides immediate feedback while you stay entirely inside the creative flow state of the headset.
An In-Headset Mentor and Creative Sounding Board
Let’s be honest: patching is not simple. It has a real learning curve. Experienced creators love the depth it offers, but diving into complex sound synthesis or spatial logic without a formal background can be intimidating.
In a boundless modular spatial sandbox, it is impossible to write a tutorial for every creative choice a user might make. The agentic layer can become a bridge: a way to make patching more accessible without flattening what makes it powerful.
Instead of forcing you to break focus, leave the application, and search for answers elsewhere, the agent can act as a private tutor inside the engine. If you are unsure whether a specific logic block is right for your project, or if you are stuck trying to build an intricate interaction, you can brainstorm alternatives with the agent.
It does not only execute commands. It can explain its choices, compare options, and talk through the problem with you, helping you learn the ropes while staying inside the creative flow.
A Creator Tried It, and Days Became Minutes
This framework can turn days of repetitive patching into a working prototype within minutes. Shortly after deploying our first internal beta builds, Hanumanatee, an early beta tester, musician, and visual artist within the community, used the agent to tackle a notoriously difficult acoustic challenge:
“I gave the MCP connection a try through Codex and it was worth getting over my initial skepticism. Very interesting and strange to give a prompt in the app and then put on the headset and watch it build it in real time.
I published a Shepard tone simulator sometime last year that I was never very satisfied with. Feeding it a basic instruction like ‘build a Shepard-Risset glissando device’ resulted in pretty much the same construction with the same weaknesses that my original had, while possibly sounding a little better.
Took about ten minutes vs. a couple of days for the original. This could get interesting.”
This example highlights the real utility of the platform. The agent does not replace human refinement, and the AI result still had structural weaknesses that required attention. However, by bypassing the blank page and handling the repetitive construction work, the agent leaves the creator with more time to test, tune, improve, and make meaningful artistic decisions.
One Playground for All Creators
PatchWorld operates on the principle that audio engineering, visual art, interaction design, physics, and worldbuilding should not exist as separate silos. They are nodes on the same creative graph. While music remains at our core, this update provides a powerful sandbox for a massive ecosystem of creators:
World builders and visual artists designing reactive environments and VJ sets.
Game and interaction designers prototyping playable systems and physics loops without code.
Performers and installation artists crafting adaptive live experiences.
Educators and students using the agent as a patient, self-documenting laboratory mentor.
Here is another small example: the agent creates a particle burst and connects it to a bass patch, turning sound into a reactive visual behavior.
Safety, Control, and Shaping the Future
Because external control is completely opt-in, users must explicitly enable agent access. All actions can be monitored, and built-in snapshot and undo systems are in place to protect your active projects.
This is an early beta release, meaning certain structural boundaries are intentionally constrained. The agent is fully capable of making logic errors, creating awkward layouts, connecting the wrong blocks, or breaking an existing signal path. It requires a human creator to steer its outputs and decide what is actually compelling. But this is just the beginning. The beta already points toward something much bigger: agents that become more spatially aware, more conversational, and more useful inside live creative sessions. We expect things to get much wilder from here.
That human-in-the-loop relationship is exactly why we are launching this public beta. We want you to push the model, stress-test its reasoning limits, and share your broken experiments alongside your successes. The future of AI-native creation should be shaped by the people who actually build, perform, teach, and play here.
The beta is officially live. Welcome to agentic patching.
If you want to join the beta, join the PatchWorld Discord and tell us you would like to test the agentic patching tools:
