My Music Mosaic

An opportunity came to work on a project with the All Greater Good Foundation. A team was formed of 3 designers and a developer to help make an application that would take music to create art. The target audience was teenagers and children who have had traumatic backgrounds and it would be used as a therapy tool. We had a short deadline to find a solution because the first class was scheduled to start in approximately 1 month, so we designed and developed in parallel.

Because we had a lot of technology constraints we started by investigating the equipment the students would be using. It was a musical keyboard that would be attached to a Windows laptop.

The developer told us what information came from the keyboard that could be used to translate into art. Things like force of the key pressed, length of the note, pitch and time are things we integrated into ways to manipulate paint strokes of the artwork. We came up with different kinds of strokes that changed depending on the type of music chosen on the keyboard.

When the application is started the students can choose a color palette and background before entering into the main program and beginning their “painting”. After completing a piece, the artwork can be saved as a TIF and printed for the student to keep. This was important because it allowed the students to take artwork to put up on their walls and give pieces to others.

After completing the application we had an opportunity to talk with and observe the students using the program. From those observations we learned a lot of new information, like how students mostly filled the canvas with brush strokes and they put their laptops in front of their keyboards. Also, some students would take their finished painting and put it into the Windows Paint program to add text and drawings on top of the artwork, really making it their own. These were all powerful new observations and we took them with us when we came together to work on improving the application.

When we started working on creating another version of the program we started by creating empathy maps to help us empathize with the students we had met. Then we did an investigation into different kinds of strokes of paint. We reworked the interface to be more accommodating for the small laptop screen and designed new features that allowed for more customization. Unfortunately we lost our developer and weren't able to improve the program further. But, the current application is still having an impact on children and teens as part of the program at the San Diego Center for Children.

During this process my team wrote a paper that we submitted to the Human Computer Interaction Conference in 2013 where we represented it. The paper can be found here.