“I love the options and freedom VR allows me to experience any instrument I want. It gives me opportunities I otherwise would not have.”
— Mary Louise
What if musical instruments didn’t begin with assumptions about hands, fingers, or how a body should move and sense? What if they were designed around each person’s unique abilities, so different gestures, voluntary or involuntary, and ways of sensing could become expressive strengths instead of limitations?
For musician Mary Louise and researcher Damian Mills, those questions have become a journey of collaboration, technological experimentation, and co-design. Together, they are exploring how virtual and mixed reality can open entirely new possibilities for musical expression by inventing instruments around the people who will play them.
When Traditional Instruments Become Barriers
Mary-Louise McCord has cerebral palsy, a congenital neuromuscular impairment that affects movement. Because it affects her hands and arms, she finds it difficult to hold and play traditional instruments. Yet her relationship to music has never been defined by these limitations. Instead, her use of digital musical instruments has created opportunities to reimagine what a musical instrument can be.
These instruments have, in recent years, included VR technologies that she has co-designed and performed with at festivals, most notably “Zoom Time, Over & Out” in October 2023, representing Drake Music Northern Ireland as part of research group Performance without Barriers, alongside inclusive ensemble Acoustronic and the Ulster Orchestra, at the Belfast International Arts Festival at Ulster University.
Eight Years Of Collaborative Research
Damian Mills is a researcher associated with the Performance without Barriers group at Queen’s University Belfast. His doctoral research explores inclusive methods and affordances in VR musical spaces, and he is also the founder of Musica11y.net, an initiative exploring accessible XR music facilitation.
The work goes back to 2018, when Damian and Mary Louise began designing accessible VR musical instruments using an HTC Vive with a programme called EXA: The Infinite Instrument. By attaching the handheld controller to Mary Louise using an atypical grip, she could trigger MIDI sounds placed around her, however she chose to move. A key principle emerged from the start: what Damian calls centered control. Mary Louise operates from a static position, so everything in the virtual space has to come to her, not the other way around. That constraint shaped every design decision that followed. That early research led to a showcase performance called “Embrace” later that year, where Mary Louise and Damian co-designed a VR stage and performed through a sixteen-channel sound system at the Sonic Arts Research Centre at Queen’s University Belfast. The work then travelled to SOUNDFORM in Hamburg and ZUKUNFTSMUSIK in Bern, in 2019, where Tabula Musica, a Swiss accessible music group, picked up VR for their own workshops.
Wearing a headset became part of that practice: at first Mary Louise could only use it for short periods, but over time she grew comfortable staying in VR for longer creative sessions, comfortable enough to keep building on this work for the better part of a decade.
Damian first trialled PatchWorld in 2022, while it was still gaining popularity. He found real potential in it, including something unusual: using avatars as a way to onboard newcomers into XR music-making. The learning curve was steep at first. What changed it was the community: weekly workshops, and new blocks the PatchWorld team kept adding, opened up more targeted, accessible ways to interact. A ray-cast device, added by request, and sliders that responded to hand presence instead of requiring a pinch, turned out to be the two that mattered most.
For Damian, usability is often the first access barrier to inclusive music making.
Over the years, his toolkit has included EyeHarp, Soundbeam, switches, iPads, drum machines, Ableton, and many experimental interfaces. But XR offered something unique.
“VR musical instruments were literally lightweight, made of light, and gesture-led design was a good place to start in moulding the technology for musical and interface, and environmental agency that is not possible in the physical world.”
In this conversation inside PatchWorld, Damian walks through the actual instrument system he and Mary Louise have built, the design decisions behind it, and what makes this kind of co-design genuinely different from retrofitting an existing interface.
From Painting To Playing
One of the breakthrough moments did not come from the technology, but from seeing some of the paintings Mary Louise had created. Damian noticed that she had painted with remarkable precision using a brush attached to her head, guided by her neck muscles. This sparked an idea.
A raycast is traditionally something you hold in your hand, like a laser pointer. Seeing how precisely Mary Louise could paint just by moving her head made Damian think to attach one there instead. Head-controlled raycasting grew directly out of that, turning a painting technique she already trusted into a way to reach, select, and play inside PatchWorld.
The point was to let her existing precision lead the instrument design, instead of asking her body to fit an existing interface. Damian and Mary Louise call the technique HOST, head-orientated selection technique. One of the instruments built this way is a VR saxophone, played entirely through raycast: she calls it into position, then selects each note by letter using her head.
One of the central design challenges is what Damian calls the Midas touch problem. A head-mounted raycast is always on, which means everywhere you look, it’s pointing at something. Turn to face another musician mid-performance and the raycast hits them. Glance away and you might accidentally trigger a control. The original mythological Midas turned everything he touched into gold, including things he didn’t mean to, and the VR equivalent is just as hard to live with. Getting the instrument to respond only when the movement is deliberate is one of the hardest problems in this kind of design.
There’s also the question of what’s already hard to reach beyond VR. Mary Louise is currently learning Ableton Live, and the small twisty knobs and dials of traditional music software aren’t accessible to her. Head-mounted raycasting could reach those parameters too.
Interdependence, Not Independence
For Damian, accessibility isn’t about retrofitting an existing instrument. It’s a chance to invent something new, and the process usually starts with what Mary Louise wants to play, with the interface built around that, not the other way around. Over eight years, that’s become a real back-and-forth: Damian brings technical options, Mary Louise decides what’s worth keeping, and the instrument changes based on what she actually wants it to do.
Accessibility often raises questions about assistance, independence, and agency, but for Damian, we are all entangled in our music making: musicians depend on sound engineers, instrument makers, venues, audiences, and fellow performers. Disabled musicians are no exception. He compares it to directing a film: the act of directing the design of an instrument is itself a form of artistic control.
The result, in practice, is that Mary Louise stays in charge of her own choices, in VR and out of it. As she puts it:
“I should always have control of my choices. Whether it’s by the VR or by directing Damian.”
In this sense, assistance can belong to artistic agency. It is part of the ecology of making music. Disability advocates have been making this case about VR specifically for years. A 2017 survey by Alice Wong, Heather Gillis, and Bryan Peck, published through the Disability Visibility Project, gathered direct input from disabled people on what VR access actually requires, work that helped set the terms this kind of research still builds on.
During ensemble performances, this collaboration takes a very concrete form. At “Embrace” in 2018, Mary Louise retained artistic control over the music while Damian managed the surrounding sound field through a sixteen-channel system at the Sonic Arts Research Centre. It was the first time they tried a two-part setup: performer inside VR, facilitator working from a separate interface. Asked years later whether that kind of split system was actually worth it, Damian didn’t hesitate:
“In performance, Mary Louise controlled the virtual instrument, myself selecting sounds and panning them around the speaker array. So yes, the method is useful!”
Five years later, at “Zoom Time, Over & Out” in 2023, the method shifted toward Mary Louise directing more of the instrument herself, through head-orientated ray-cast control, with Damian supporting the conditions that let those choices be heard.
Collaboration does not diminish authorship. It expands it. Damian and Mary Louise’s work suggests something closer to interdependence than independence: not the absence of help, but agency that is negotiated and supported by people, technologies, and communities working together. The same idea runs through community music and disability studies more broadly, where creativity is understood as something that emerges through relationships, not in spite of them.
Why PatchWorld, And Where This Goes Next
PatchWorld turns out to be especially well suited to this co-design work. Damian describes it as a multiplayer musical instrument construction playground: a blank canvas of modular blocks that can be assembled visually, letting creators rapidly prototype ideas, share them with others, and evolve them together. The community itself becomes part of the creative process, what Damian calls a community of “patchers who are engineers with dreams.”
PatchWorld’s AI agent, currently in beta, can already build and rewire instruments and interfaces inside the world in real time, directed by voice. That opens a real path toward ability-aware instruments, and eventually ability-aware worlds, environments that adapt themselves to a person rather than asking the person to adapt to them.
Right now, ability-aware design happens the way it always has: through co-design, trust, and a lot of iteration between a musician and a facilitator who knows their gestures and their range of movement. What changes with AI is the possibility of that knowledge becoming part of the system itself. An agent that can observe gesture ranges in the virtual space, recognise resting positions, understand the difference between an intentional movement and an accidental one, and use all of that to co-locate instruments where they’re actually reachable, that’s ability-aware design at a different scale. Eight years of knowledge that currently lives in Damian’s head could start traveling with Mary Louise instead, instrument to instrument, without having to be rebuilt by hand each time.
This is early, and the people leading it are researchers and musicians like Damian and Mary Louise, not PatchXR alone. What they’re building is specific to music for now, but the underlying idea is one we think about for PatchWorld more broadly too: inclusive by design.
Mary Louise put it simply:
“Inclusive design to me would mean I could completely independently choose, play, compose and save music in VR.”
That’s what all of this has been in service of.
Further watching: Zoom Time, Over & Out on YouTube, starting at 9:43.
Damian shares more of this work on his YouTube channel: @DrDamianMills.
Explore his initiatives: Musica11y — Support Musica11y on Patreon — Performance without Barriers.
