Designing Music Around Abilities

Private preview for feedback. This article is not ready for publication.

How Mary Louise and researcher Damian Mills are co-designing new musical futures through embodied creativity and XR.

“I love the options and freedom VR allows me to experience any instrument I want. It gives me opportunities I otherwise would not have.”

Mary Louise

[Image placeholder: Mary Louise, Damian Mills, or an accessible XR music setup inside PatchWorld.]

Caption placeholder.

What if musical instruments did not begin with assumptions about hands, fingers, or physical limitations?

What if they were designed around abilities instead?

For musician Mary Louise and researcher Damian Mills, that question has become a journey of collaboration, experimentation, and co-design. Together, they are exploring how virtual and mixed reality can open entirely new possibilities for musical expression, not by adapting existing instruments, but by inventing new ones.

When Traditional Instruments Become Barriers

Mary Louise lives with cerebral palsy, which affects her hands and arms. Traditional instruments present significant barriers.

Because cerebral palsy affects her hands and arms, she finds it difficult to hold and use traditional instruments.

Yet her relationship to music has never been defined by these limitations. Instead, new technologies have created opportunities to reimagine what a musical instrument can be.

Through projects with the Ulster Orchestra and Accoustronic, Mary Louise has already performed using VR technologies as part of the Belfast International Arts Festival and at Ulster University.

Eight Years Of Collaborative Research

Damian Mills is a researcher associated with the Performance without Barriers group at Queen’s University Belfast. His doctoral research explores inclusive methods and affordances in VR musical spaces, and he is also the founder of Musica11y.net, an initiative exploring accessible XR music facilitation.

His journey into immersive music began through community music practice and collaborations with disabled musicians. For Damian, the first access barrier to inclusive music making is not disability itself, but usability.

Over the years, his toolkit has included EyeHarp, Soundbeam, switches, iPads, drum machines, Ableton, and many experimental interfaces. But XR offered something unique.

“VR musical instruments were literally lightweight, made of light, and gesture-led design was a good place to start in moulding the technology for musical and interface, and environmental agency that is not possible in the physical world.”

BBC feature placeholder caption.

From Painting To Playing

One of the most inspiring moments came not from technology, but from observing Mary Louise paint.

Damian noticed that she painted with remarkable precision using a brush attached to her head, guided by her neck muscles. That observation sparked an idea.

What if a raycast, a laser-like pointer often held in the hand, could instead be mounted on the head?

This insight led to experiments with head-controlled raycasting, allowing Mary Louise to interact with virtual instruments using the same precision she already used in painting. Her existing skill became a design starting point: an accurate method that opened more choices for interaction design and complexity.

Rather than forcing people to adapt to instruments, the instruments themselves can adapt to the performer.

[Image placeholder: Mary Louise painting, head-mounted raycast interaction, or a close-up of the interface concept.]

Caption placeholder.

Co-Design Rather Than Adaptation

For Damian, accessibility is not simply about making existing instruments usable. It is about creating entirely new forms of musical expression.

He explains that instrument design usually begins with what his collaborator wants to play, and the virtual interface is then adapted to their abilities.

Over eight years of collaborations, this process has become deeply reciprocal. Damian has come to see co-design as a social process, one that changes what exists into what people want to exist.

This approach places disabled musicians at the center of the design process, ensuring agency remains part of artistic creation itself.

As Mary Louise puts it:

“I should always have control of my choices. Whether it’s by the VR or by directing Damian.”

A Community Of Patchers And Dreamers

PatchWorld provides an unusual environment for these explorations.

Damian describes PatchWorld as a multiplayer musical instrument construction playground: a blank canvas of modular building blocks where people can create, test, and share knowledge together.

Because instruments can be assembled visually from modular blocks, creators can rapidly prototype ideas, share them with others, and evolve them collaboratively.

The community itself becomes part of the creative process. He describes PatchWorld as a community of “patchers who are engineers with dreams.”

[Image placeholder: a PatchWorld instrument patch, shared workshop scene, or accessible instrument prototype.]

Caption placeholder.

Presence Beyond Distance

Beyond accessibility, XR offers something else that is often overlooked: presence.

Musicians, facilitators, and researchers can meet, rehearse, and create together regardless of geography. In a field where expertise and communities are often dispersed, shared virtual spaces create new forms of collaboration and knowledge exchange.

Remote participation does more than remove physical barriers. It allows communities of practice to emerge across distances, connecting musicians, researchers, and facilitators who might otherwise never meet.

Creativity becomes something that can be shared, iterated, and transmitted together.

Inclusive ensemble performance placeholder caption.

Interdependence And Agency

Accessibility often raises questions about assistance and independence. But Damian sees agency differently.

As he puts it, we are all entangled in our music making. Musicians depend on sound engineers, instrument makers, venues, audiences, and fellow performers. Disabled musicians are no exception.

In this sense, assistance is not the opposite of artistic agency. It is part of the ecology of making music.

During orchestral performances, this collaboration takes a very concrete form. Mary Louise retains artistic control through eye-gaze interaction and her musical decisions, while Damian acts as facilitator, managing the surrounding sound field and spatialisation through a sixteen-speaker ambisonic system.

Collaboration does not diminish authorship. It expands it.

This philosophy echoes broader ideas found in community music and disability studies, where creativity is understood not as an isolated act, but as something emerging through relationships, trust, and shared practices. Agency is not lost through collaboration. It is negotiated and supported by people, technologies, and communities working together.

Rather than striving for complete independence, Damian and Mary Louise’s work suggests another possibility: interdependence. Not the absence of help, but the presence of meaningful participation, choice, and authorship.

Towards Ability-Aware Musical Worlds

For Damian, XR is not a future technology. It is already transforming musical performance right now.

Mary Louise continues to explore the possibilities offered by XR. She can use eye gaze technologies, sometimes her hands, and enjoys the freedom to experience instruments that would otherwise remain inaccessible.

Her dream is simple and profound:

“Inclusive design to me would mean I could completely independently choose, play, compose and save music in VR.”

For Damian, these developments extend far beyond accessibility.

They point toward ability-aware musical worlds: instruments, interfaces, and environments designed around a performer’s gestures, energy, range of movement, preferences, and creative choices.

Today, this happens through co-design, through trust, iteration, and experimentation between musicians, facilitators, researchers, and communities. With new AI-assisted creation tools, another possibility begins to emerge: musical environments that can understand, remember, and adapt, not to replace human collaboration, but to make it more fluid and responsive.

Perhaps the most exciting aspect is not the technology itself.

It is the possibility that new forms of culture, collaboration, and creativity emerge when instruments are designed around abilities rather than limitations.