PatchWorld now has an AI agent that builds modular instruments and spatial audio with you, in VR or on desktop. No code, no headset required to start.
For a while, I was genuinely worried about the future of patching.
Not PatchWorld specifically, patching as a way of making things. It’s a deep, weird, rewarding way to build sound and image, in the lineage of Pure Data, Max/MSP, TouchDesigner, no code required. But it has a real learning curve, and for a while it felt like the culture was moving the opposite direction: why sit down and learn a wiring-and-blocks system when generative tools can hand you a finished song or image with zero effort and zero friction?
Then the last few months happened. Cyberdecks went properly viral: people hand-building strange, deeply personal little computers, decorated with pearls and fake moss, captioned things like “gatekeep this from AI and megacorp.” TouchDesigner show-and-tell videos are everywhere right now. There’s a real, named backlash to AI slop running through music, media, design, all of it. It’s not that people stopped wanting complexity, craft, or deep tools. If anything they want it more, on purpose, as a kind of resistance. What they’re rejecting is flat. Generic. No fingerprints. Output that looks like everyone else’s output.
At almost exactly that moment, we were heads down building our own patching agent inside PatchWorld, and realized it sat right in that gap. It doesn’t generate a finished thing for you. It gets you into the deep tool, makes the entry easier without making what’s inside any less deep.
What Agentic Patching Actually Is
Patching in PatchWorld means grabbing blocks (sound, visuals, logic, physics, motion) and wiring them together in 3D. It’s how every instrument, device, world, and system in PatchWorld gets built. Until this beta, a human had to do all of that wiring by hand.
Now an AI agent can do it too: spawn blocks, connect them, set values, inspect what’s actually happening on a live signal path, fix what’s broken, and keep iterating, the same actions a human patcher performs, but driven by a model with real tool access to the live scene. It can step into a subpatch to work inside an instrument, not just at the top level of a world. It can take a photo of the current space to reason about layout and clutter before deciding what to build next. It’s not a chatbot answering questions next to your project. It’s operating inside it.
This is different from Genie Box, our existing in-VR generator for assets like skyboxes, sounds, and 3D meshes from a prompt. Genie Box hands you a finished object. Agentic patching connects, configures, and modifies the systems that decide how a world behaves: it builds and rewires the relationships between things, not just the things themselves.
What It Felt Like Watching It For The First Time
The first time I saw it, I met with Boris, our lead developer, in his VR world. He was just talking to the agent. “Build an arpeggiator in A flat minor pentatonic.” “Add a bassline in the same key.” “Make it more surround, since we’re in VR.” “Add some reverb.” The agent kept building and checking in after each step, and at one point Boris told it the patch had turned into a blob of blocks and asked if it could spread things out and label each section, and it did, right there in the world. The conversational part of it is what hit me. Watching a build happen in the space around me while someone talked through it, like working with a collaborator who can move things around as fast as the idea comes out of your mouth.
That’s when I realized how powerful this could be for education and learning.
Then there’s the solar system demo. Visually, it’s not that impressive when you first see it. What got me is what’s underneath it: the agent went and found the actual data (real relative size, real distance, real rotation and orbital speed for each planet) and built a small working model out of that, accurately, blocks and all, in a few minutes. Then those same values got wired into a real-time spatial audio system, turning planetary motion into an atmospheric 3D audio piece. That’s the thing that’s hard to explain to someone who hasn’t tried it. You’re not describing a picture, you’re asking for a system with real relationships inside it, and it just builds the thing.
Same idea, smaller scale: a particle burst wired to a bass patch, so the visuals actually move with the sound.
Finally, I Could Patch
I’m an artist, not a developer, and not a musician, and I’m not particularly good with logic and math. I tried to learn Max/MSP during my master’s at CalArts, in a music class about Alain Badiou and Max/MSP together, which was fascinating, but I didn’t have the patience to push through that learning curve. So I never patched. Not because I didn’t want to, because the entry was too high for me.
That’s the first thing agentic patching fixes. It lowers the entry point enough that a complete beginner like me could actually start building something instead of staring at a blank patch. And from there, actually start learning, instead of bouncing off it the way I did.
The second thing it fixes is one of our oldest internal headaches: PatchWorld is a deep, modular system. There’s a huge amount you can do with it, which is exactly why it’s nearly impossible to write tutorials that cover it. You can’t document every creative choice someone might want to make in an open spatial sandbox. With an agent inside the engine, you get something closer to a personal mentor: it can explain why a block does what it does, suggest what to use for the thing you’re trying to build, or just talk through an idea with you while you’re stuck. That’s not a feature we could have shipped any other way.
Three Ways To Work With It
If you want to try it right now, here’s how to get in. The beta ships three entry points into the same underlying agent.
External agents over MCP. PatchWorld exposes a Model Context Protocol server, so an external AI client can connect directly to a live PatchWorld session once you’ve explicitly turned on agent access. Right now that’s documented for Claude Code, OpenAI Codex, Google Antigravity, and ChatGPT web, with any other MCP-compatible client able to connect the same way. Your agent does the reasoning, PatchWorld exposes the tools and executes the actions in the world. Full setup guide here.
The typed in-app agent. No external setup needed. Type a request into the in-world console and watch the agent map out its plan and wire the blocks on screen. Because it reads the current scene and what’s selected, it’s also a genuinely fast way to learn spatial patching logic by watching it happen.
The Live Voice Agent. Speak to it directly in VR, hands still on your controller or instrument. Say something like “make the bassline more wobbly,” and it finds the right modules, edits the existing patch, and you stay in flow without breaking out of the headset.
A Power User’s Honest Take
Tom Vigal, known in the community as Hanumanatee, is the artist behind some of the most genuinely impressive real-time visual work in PatchWorld, including his Visual Synth. He tried the MCP connection through Codex early on. He’d built a Shepard-Risset glissando device a while back and was never quite happy with it. He asked the agent to build one from scratch.
“I gave the MCP connection a try through Codex and it was worth getting over my initial skepticism. Very interesting and strange to give a prompt in the app and then put on the headset and watch it build it in real time.
I published a Shepard tone simulator sometime last year that I was never very satisfied with. Feeding it a basic instruction like ‘build a Shepard-Risset glissando device’ resulted in pretty much the same construction with the same weaknesses that my original had, while possibly sounding a little better.
Took about ten minutes vs. a couple of days for the original. This could get interesting.”
And after more testing, he added:
“I’ve come to think of it more as an amplifier than a synthesizer: ‘build me the ultimate sequencer’ isn’t going to work too well.
It isn’t an oracle, more like a well educated young lab assistant with tons of book learning but very little understanding of the practicalities of patching, while eager to learn. You have to watch it, since it might struggle with some mundane mathematical conversions when all it needs is to be interrupted and told to put a map block on it. That said, I’ve picked up some interesting new ways of doing things and uses of blocks that I hadn’t thought of before.
It’s a little eerie when it’s figuring out some sub-part of a larger project, and I tell it don’t mind me, I’ll just be cleaning up the patching arrangement over here while you’re busy over there, keeping one corner of an eye on its progress. It’s not either/or on the patching vs. agentry.”
No Headset Required To Start Building
We’ve had a desktop version of PatchWorld for a while. Genuinely useful for filming, streaming, managing assets, joining events. But if you actually wanted to build a patch, it never quite worked. Patching is spatial, and a mouse and keyboard fight it. So desktop was always the easier way in, except for the one thing PatchWorld is actually about.
The agent changes that. On desktop you can type out a long, precise instruction with a real keyboard, hand it reference data, and watch the architecture get built on screen. No headset needed to start. That’s a much bigger door than VR alone, since most people who’d love this don’t own a headset and never will, but plenty of them already have a laptop and an hour of curiosity. Once something exists, you can put the headset on, step into what got built, feel the actual scale of it, and keep directing the agent by voice from inside VR. Describe and build on desktop, inhabit and refine in VR.
Where This Goes From Here
Patching has always asked a lot of a newcomer, and for a while that felt like a losing proposition against tools built for instant, zero-effort results. Agentic patching doesn’t remove that depth, it changes who can reach it, while keeping every build editable, inspectable, and yours to take apart. It also opens the door wider for people with limited mobility or cognitive differences, who’ve rarely had a creative interface built around them instead of around an assumed body and an assumed mind. We think this is where collaborative creative tools are headed generally, not just where PatchWorld is headed.
The beta is live now, in front of real users, with real builds already coming out of it. If you want to try it yourself, join the PatchWorld Discord and ask for access.
