Can you Really Paint With All the Colours of the Wind? : Interview with Michelle ‘Misha’ Oraa Ali, creator of MuSyC
Synesthesia, from the Greek words syn (union) and aesthesis (sensation), affects roughly 1 in 2,000 people. Michelle (Misha) Oraali happens to be one of those people graciously bestowed with the ability to make connections from sound to colour. She has been working on her software app, MuSyC (Music, Synesthesia, Color), which is a music to colour synesthesia simulator that takes auditory stimulus input and presents real time visual feedback.
In itself, the project “MuSyC” is tremendously exciting. Yet, in speaking with her, learning about her past motivations and future intentions, the world of MuSyC has become even more colourful than I had envisioned when I first heard of it. As a senior at Mount Holyoke, Misha is working on MuSyC at the same time that she tackles her senior Honors Thesis as a Neuroscience major. In talking about her thesis, we reach some insightful parallels between the two projects. She says both her thesis and her Digital Humanities project are the same in spirit, but not in operation. In this interview (more like conversation) we delve into both the spirit and the operations of Misha’s MuSyC
Misha: My thesis? Well, it’s a combination of linguistics and neuroscience. I’ve been doing EEG ERP experiments the whole year, and it’s been two years in the making.Broadly, its about imagination. Specifically, it’s about auditory imagination. What’s interesting but often overlooked is that imagery as a term just means “a mental representation” except it is often conflated with the visual realm, so much so that when you say imagery, people automatically just think “pictures!” and other visuals. So, there hasn’t been that much empirical research done in auditory imagery.
Sheila: So auditory like music or noises or sounds?
Misha: Auditory even like when we are talking with each other and we use different stress patterns/intonations. We phrase things so that if we say something like “The Panda eats shoots and leaves” vs “The Panda eats, shoots, and leaves” we are aware of two different sentiments. One phrase refers to a Panda that has a gun, and after consuming something, the panda shoots something, then gets up and leaves (laughs). But then the other sentence means it consumes “leaves and shoots”. And that difference is pretty easy to understand verbally; there has been a lot of research done about that in an explicit prosody context. Prosody is just a fancy term for for all these things on intonation, pronunciation and stress on certain parts of words versus other parts of words. Most of the time, we only notice “explicit” prosody because the words/phrases have actually been verbalized— you’re listening to the speech and you’re hearing the rhythms and patterns. Whereas, in silent reading (for instance); you still have that prosody, except that it’s implicit. There hasn’t been that much research done on that because silent reading is an implicit format and for that kind of observation, you have to wait for technologies to be developed. But for example, let’s take the word “present”: Present could mean multiple things but it depends on how you read it and what the context is— it could be “present” like a gift or “PRE-sent” like I’m presenting my thesis to you
Sheila: But isn’t that based on stress? For instance if we look back at your earlier example, “The Panda eats shoots and leaves,” would you say the distinction is based on how the entire phrase is said, or where the phrase is placed in a context of other things around it?
Misha: Well, when you say that phrase out loud, the guiding point would be the comma, right? So, the comma will tell you where you put a pause, which means apart from stress, there is also phrasing cuz that’s basically what a pause does; it tells you where the silence is.
Sheila: I see. So back to implicit prosody?
Misha: Yes, so it’s implicit because it’s non-verbalized. And in my thesis, i’m looking at the rhythm for Implicit Prosody. Im using poetry for that and specifically, I’m using Limericks because they have a very strict rhyming pattern
Sheila: Oh I love Limericks!
Misha: Yeah me too! I’m always looking for new examples of Limericks. Because of their strict pattern of where rhymes and stress are supposed to be, it gives me a very good framework and infrastructure to then manipulate certain forms of that. People are usually familiar with Limericks. Every one has read or heard at least one Limerick, at least from school and so that makes it a really good experimental tool.
Sheila: What is your experiment for the Implicit Prosody like?
Misha: Basically, I place people in an EEG cap- those fancy caps with electrodes.
Sheila: Oh, amazing! They always look like great fun
Misha: Right? Well, I put them in the EEG hat, and I ask them to read a series of couplets which are taken from Limericks. I have 320 of them.
Sheila: How did you compile them?
Misha: Me and my professor wrote about half of them, the rest were from books. We had to use these very specific words for stress alternating noun verb homographs (which is just a fancy way of describing a word like “present” which could be a noun or a verb).
Sheila: This reminds me of a conversation I was having about tonal cultures. I was explaining to a professor of mine that my local language-Igbo- is a tonal language. And this professor said that, in comparison, American English (or English in general) was not tonal. That English is more text based where they try to have a different word for a different meaning. But now, here in this conversation, it seems like that might not be very true because a word like “present” is one word but it means different things, which change according to the speaker’s tone and stress.
Misha: Yes, yes. Take for example, the word “To”…
Sheila: Yes! Exactly
Misha: …as a word, “to” can mean so many different things. Even the word “break” can mean seventeen different things. You have “spring break”, or “a break in the paragraph” etc. And so, it does depend on context because language is ambiguous
Sheila: Yes. Now this makes me feel like my language may not be that special for being tonal because it seems like English has it. I’m kind of pissed off (laughs). It seems everyone has it but where others call it tone, English calls it stress—yes? no? But then again, based on the fact that there is a group of languages called tonal languages, would that be a misunderstanding of the fact that every language actually has tone?
Misha: Hmmm, I don’t think so. For instance, I know Mandarin is a tonal language. And in Mandarin, you have the word 馬 (Ma), which is said as, “mā,” “má,” “mǎ,” “mà.” They all mean different things. Whereas, English has that; that thing where the same arrangement of letters convey different sentiments (ref: present), but then it has to be in the correct context. Like, if it was your birthday party and I were to say “I’m giving you a birthday present” but then I said “I’m giving you a PRE-sent,” you would know that it’s wrong. What I’m trying to say is that I think, in English, the semantic context, really dictates how you would pronounce the word. Whereas, in Mandarin for example, the way you say the word determines the semantic context.
Sheila: Ah! I think I understand you. It makes me remember the part of Chinua Achebe’s Things Fall Apart where he wrote about a white priest that would try and add some parts of the local Igbo language into their sermon. And so this priest would say the word “Ike,” which is pronounced in two different tones to produce two different meanings. One of the tones conveys “bum” and the other conveys “strength.” And he (the priest) would constantly refer to the “God’s bum” when trying to say “God’s strength” and the congregation would be so scandalized
Misha: (Laughs) Yes language is very interesting. Last semester I took a course on language and thought that was amazing.
Sheila: What drew you to studying language?
Misha: So, I’m half Indian and half Filipinx. Growing up, we moved a lot. Especially within South East Asia where there are so many languages and universal words across most of them. Which means you might find that one word in one language is also present in another language, just slightly mutilated. But then there are also words that stay the same in different languages but assume very different meanings across the geographical spaces. For example, the word “Susu”: In Tagalog, “susu” means “breast.” Meanwhile, in Malay, “susu” means milk. In these two contexts, you can see the relationship right? Breast? Milk? That’s easy to see. Whereas, in Hindi (which is what I would speak to my paternal family in India) I would say “I want to drink susu” and they would be so scandalized because there, “susu” means “pee.”
Misha: It was those cross cultural linguistic quirks that first made me interested in language. How do these languages evolve to become the latest versions of themselves?
Sheila: Actually, language and imagery are quite connected to think about. Somehow, i’m remembering this one time when my mother jolted up from sleep and said something in Igbo. And I couldn’t believe that the first thing that came out of her mind -from her sleep- was Igbo. So I asked her “Were you dreaming in Igbo?” and she said “Yes of course.” And all of a sudden, it clicked; Igbo is not a thing she puts on or translates into, it’s actually the language that shapes her world, it’s what she translates from. I can’t imagine speaking a different language in my dreams. It made me, for the first time, acknowledge this kind of deep perceptual/conceptual distance between my mother and I
Misha: And language as a lens to look at human cognition is very interesting because does language really affect the way you think or does the way you think affect your language? For example, there’s this tribe in South America called the Pirahã, and they only have two distinct words for one/small quantity (hói) and two/large quantity (hoí). So they have like 1and then, Many. English as a language has 44 phonemes, on the other hand, the Piraha have about 12 phonemes (phonemes are the individual units of sound e.g fire is one syllable but it is comprised of 3 phonemes “Fa” “ay” “ra”) . And it’s not like they are incapable of learning to make other sounds, it is just that those are the sounds that they make as a language. So does that necessarily limit their vocabulary? For example, because they don’t have a word for four or ten, does that mean that they are less capable- compared to an English speaker- of perceiving plurals or being able to quantify those plurals? It’s fascinating! and now, there are people doing experiments around that.
Sheila: That’s a really interesting way to bring in the conversation about how education is skewed to favour certain people who speak certain languages in a way that most of the capital for educational development happens in certain countries. Which makes it seem that their education is the real education; like they have the best method of thinking about things. Even with English! Because sometimes, the words people fight for here, are not the only ways of thinking about a certain subject. But now, because I have gone to school here (in the U.S) I have learnt that certain words are the appropriate revolutionary terms. For instance, a word like “Trans” which is a word that is very American/English in origin even though the concept is not. It has existed since the beginning of time, for multiple communities. But then, now, people are exporting “Trans” like its the single way to communicate the idea. But maybe “Trans” is not my language’s way of understanding that complexity and hybridity within gender… Which is hard in terms of activism because…
Misha: You don’t want to prescribe these terms to people in different communities
Misha: For instance, there are two-spirit people in certain Native American tribes. The concept has always existed, so it wouldn’t make sense to import the term of transness and use it to refer to them.
Sheila: Exactly. So, does your thesis work on language and prosody have any relation with your Digital Humanities project?
Misha: Operationally, no. In spirit, yes! The reason I applied to college in the US, and something I always remember wanting, was to pursue things in an interdisciplinary manner. So this project for me, was an opportunity for me to integrate neuroscience (one of my big loves) and, poetry. Why not? (laughs) My digital humanities project is not only an opportunity to combine neuroscience with computation, but also, to integrate those into an artistic format…being able to integrate music. And that’s where the inspiration for the project comes from. In spirit my thesis and my fellowship project are very similar: My digital humanities project is very focused on creating something new whereas, my thesis is about doing experiments to verify certain facts
Sheila: But aren’t those experiments new? I guess nothing is ever that new in academia-
Misha: Well, the thesis question I’m using to define the scope is actually quite new
Sheila: And if you could define your thesis question in one sentence, what would that be?
Misha: Essentially, i’m trying to investigate the role of rhythm in the context of Implicit prosody during silent reading and what implications it has for auditory imagery
Sheila: That was very clear and wonderful. It seems like something you’ve said to a couple of people over time
Misha: No, it’s just that I wrote that last night (laughs)
Sheila: Great summary of your whole 70 pages. So, in practice, operationally, what is your Digital Humanities Fellowship project?
Misha: It is to be able to create a device that takes auditory input of any kind, and gives you real time visual feedback as an attempt to simulate music to colour synesthesia which is otherwise called chromesthesia but I just like to say music to colour (smiles). I don’t like to say synesthesia is a “condition” or “disorder” because that medicalizes it. And, it doesn’t ever really negatively affect my life so I don’t call it a “condition.” But anyway, apart from this software/device being an applied neuro-technology tool, it could also be used in a pedagogical format for music teachers (for example); If a kid is coming in for their first week of music training, and the kid cant tell the difference between “F” and “F sharp” because she has no musical training, using colour to make distinctions between those two notes might trigger other ways of remembering. So, if i heard that “F” is green and “F sharp” is dark green, that could help me associate those two tones with those colours and provide a visual aid that could help me in my musical journey.
Sheila: So, now is a good time to say, I tested the software app that you sent me, and I used the Willow smith song, and then I also used the Chopin song. And as you can guess, both were very different experiences. What was even more intriguing, was that I used the “Max Pitch” to experience it… it was intense (to say the least)
Misha: Yeah, that’s very intense. I wouldn’t demo that to people because it is so intense
Sheila: (Laughs) But I must say, it was interesting to have options. It’s always great to have options for that kind of thing because I’m already in an exploratory mode when I come into that space, so anything I can try, I will try. Which reminds me, one of the things I learned through that experimentation was that I wasn’t hearing everything I was seeing. Sometimes the colour would pop, but I didn’t actually hear any notes in my ear. And it made me wonder, are there notes that I dont pick up in music that I regularly listen to?
Misha: Yeah, it picks up notes that might be a little bit beyond the hearing range of human beings. And also, everyone has their own frequency ranges that they can hear. So, you and I will have different frequency ranges which means that maybe you can hear more extended range in the higher levels and I can only hear extended ranges in the lower levels. That’s more a function of genetics.
Sheila: I see. It really made me question my experience of music “Have I not been hearing music the right way?” Your software visualized just how much really goes on in one song…
Misha: Yes, and especially with compositions. In the beginning, I tested it out with just pure tones. So you would have one sound producing one bubble. And then, using the same tool to test a musical score really makes you appreciate how complex all these songs are, and how much work goes into constructing any one of these songs. Even pop music; people chastise it for being like 3 chords played over and over again, but actually there’s effort in that!
Sheila:Also, it’s interesting having an idea of the volue, or the length…and even the vigour, with which something is played versus the calmness. That was great
Misha: …even seeing the visual distinctions of the chorus versus the verse
Sheila: Exactly! And it really is wonderful seeing the chaos. Honestly, it made me think of how I may be ill-equipped to read visual representations of non-visual imagery. Because if some people saw the algorithms your software creates as a visual only painting/picture, they might respond negatively like “this is chaos, I can’t make out anything” But if you hear the auditory version of that visual picture, it might affect you deeply as a song or piece of music, it might even be your favorite song. And this led to the question, is there a way in which the auditory imagery, is only good in an auditory version? Or, is it just not understood in the visual imagery? For instance, if you put a really good/beautiful picture into an app that translates visual pictures into auditory waves, would it be possibly boring? Even though it is appealing visually, is it possible that any translation of the visual image into an auditory representation doesn’t replicate those feelings of beauty?
Misha: Hmmm so you’re saying for instance, you may have a beautiful classical, anatomically correct figure, which might be conventionally beautiful but then, if you were to translate that in a different format, would it still be interesting?
Misha: I see what you mean. Well, when I took some Beatles’s songs—and as you know, many people consider the Beatles to have made the best music in the 60s and 70s— and I put them through the app, they weren’t very interesting for me to look at. Whereas, when I took something more like Finnish Death Metal, and put that in the algorithm, it looked pretty nice. Ordinarily, you would not really make that association or assumption
Sheila: Very true. And I believe this kind of conversation is very important for conversations about rematerialization. By rematerialization, I mean the idea of taking something born in one context, and translating it to survive and be viable in another context – like making stimuli auditory even though it was first created visually or vice versa. And I don’t think that translation should mean replication, but really, what fascinates me is seeing that such translations are possible. Perhaps you’re not trying to make sure the translated output is as beautiful or wonderful because it’s primary state was wonderful and beautiful
Misha: Yeah! And that’s a big part of why I wanted to create an algorithm which did that. Initially, my plan was to just animate these things. But then I thought, wouldn’t my biases and tendencies make me choose to animate only songs I liked? And also wouldn’t it also mean that if I liked a song more, I would put more effort into animating it beautifully, to be more aesthetically pleasing? On the other hand, if someone were to tell me a song that they liked and I dont like it at all, I’d probably put less or maybe no, effort into it all. Wouldnt that be biased? Whereas, an algorithm is more unbiased: It depends on the variables you put in, which are controlled by the song that you choose
Sheila: I like that. I’m glad you rethought that (laughs) It is really interesting for me to have a visual representation of absences, like pauses and silences and even reductions of volumes or pitches. Out of all the current and potential applications of this tool, what is the most exciting possibility for you?
Misha: For me, the most exciting application would be for it to be developed as an accessibility tool. The thing that happens when I tell some people (mostly people my age) about this project, they usually say something like “Oh that would be so cool in a party.” And to be honest, that’s not really the motivating factor. For me, it would be really cool as an accessibility tool especially because part of the inspiration for me to pursue this project was in speaking with one of my friends who loves house music/dubstep and happens to be deaf. I mean, at first it was really weird for me to learn that. I didnt want to be insensitive, but I really wondered how she could listen to the music. But I found out, it’s because she can feel the vibrations of the beat. So if you have an app like mine, it gives a new wholistic approach to experiencing music for people who don’t have access to being able to hear those notes. And then, anyone can refer to the red part of the song, which could maybe be the high part of the song or the chorus. Right now, the way engineering works in concerts is that there is someone who works with the artists, and asks the artists what lighting they want during their set but then that makes it very choreographed. And that’s not to say there isnt a certain art to that choreography- there is. There is an effort to that. But then again, it all becomes too standardized without taking enough into account. But now, we could use an app like this to translate across various scores of music – Beethoven, Chopin, Bach , for instance, and then that would create a new alternate way of sensing those choices-
Sheila: Almost like a vocabulary?
Misha: Exactly! A visual colour vocabulary for music.
Sheila: Wonderful. Sometimes, when I listen to music, I can pick out the violin and maybe the piano. But in some songs, I cant really pick out anything separately- it all merges into this wonderful thing. And I’m always pleasantly thrilled to just feel that wonderful amorphous thing that I feel. Thinking about this idea of a vocabulary makes me question: if I knew exactly what was playing each time, would that diminish the feeling of the “wonderful thing”? And is that a fear in terms of standardizing vocabulary for auditory sound?
Misha: Well, honestly, for me, my approach to music is based on timbre not really pitch. At this stage of my prototype, pitch is much easier to show which is why I consider that. But I think that there is definitely space for the app to grow in terms of being customizable by the conductor, or even the orchestra as a whole. Since they are the ones in charge of creating the vibe of the concert, they can also be in charge of setting the vibe for the colour.
Sheila: Wouldn’t it be wonderful if you had an orchestra of people, maybe even just deaf people, who were playing the most beautiful music just based on colours?
Misha: Yeah, exactly
Sheila: They would be playing based on colours in the same way that people played based on notes. Honestly, I sing but I cannot read music. I feel like colour would have been so helpful for me as I moved from singing in Nigeria to singing in America
Misha: Exactly. I mean, people usually say that the language of music is a “universal language that anyone can read” But then I say, it’s basically the same as English as a language because those notes uses the western scale. There are also seven notes in the Indian scale of music just that instead of “Do Re Mi” it is “Sa Re Ga Ma” And they might not really have a written way like that, but you can teach a person to sing a song in Do Re Mi or Sa Re Ga Ma, and it would still sound the same. And now, if you have colour, which is consistent, then that just makes it easier!
Sheila: And I think one other thing that’s wonderful about colour is that it is a non-textual way of communicating. And that’s great because it always seems so hard to get away from English or anything that’s English based. Which then brings me back to what you asked earlier: “Is it the language that creates the thought or is it the thought that creates the language?” If Im so used to Do Re Mi, and the examples I’m getting are songs like “Mary had a little lamb”, my Mary is going to have a little lamb even though I might have never seen a lamb before. For instance, in some places in the world, people have throat singing which may be considered bad singing here but it is a type of singing elsewhere. And people communicate different specific things with that specific “bad throat singing”
Misha: Which is very true. And if you look at the history of singing, people have different gradations of the music they produce. There might be a low Do, and a higher Re: one can make those divisions and call them whatever terms they want. But people can also make those associations themselves, with their own terms , or maybe now, their own colours
Sheila: And which is why when you say accessibility, I almost think, to a certain extent, I’m not really able to “access” the music as written with a western musical notation. In my first year in college actually, when I wanted to get access to a voice class, I was told that I needed to learn how to read sheet music first. Which was baffling to me, because all of a sudden, I felt like I wasn’t able to sing. But it wasn’t about singing- it was about reading the song. And the piano has always seemed so inaccessible to me because it seemed like the language of the piano is the language of the notes. But now we both see that is not necessarily true, because it is just an instrument that doesn’t speak a certain textual language
Sheila: Wow, that’s sad. Where was your app when I was young and desperate for music education?
Sheila: Really though, this app really gives an alternative to standards of teaching that enforce these ways of reading notes because they seem like the only ones available.
Misha: And for me this also relates to my upbringing which I spoke about earlier. For instance, in India, they have a certain scale that they use. And in the Phillipines, where I’m from, they either use the western scale or they have a local indigenous Fillipino scale. And then in Malaysia, where I also lived, there is something completely different. Back then, if I had a standardization which I could carry around myself, (like in this app form), then maybe I would have been able to learn an instrument long term. But because there were just so many different systems, I was just never able to continue learning and become proficient in guitar, or piano or whatever. And that’s also part of my selfish, third culture kid reason for making this app.
Sheila:I don’t think it’s selfish- at all! Now, I’m in my head, thinking about how different colour is from other forms of notation. Is colour really that universal?
Misha: Yesss, that’s another notion. (Laughs) “Is colour really universal?” I mean, that is my assumption coming into this but it could also not be true at all.
Sheila: At the moment, I find no reason to doubt that. Maybe, apart from people who are colour blind….which then leads me to think, “Are people colour blind in the same way universally?” Would an American person who is colour blind have the same realities as a Nigerian person who is colour blind? Now im thinking about the white and gold dress debate (laughs) But even in that heated moment, the world was on two sides of the debate. So it was either white & gold or blue & black, there wasn’t someone from Lagos saying they think it’s maybe Orange. So could it be that colour is one of those truths or facts, that has a universality to it?
Misha: In many ways there are those minor differences, like if you show someone a Teal, will they think it is a green or a blue? But those are the “in-betweens.” If you stick with primary colours, I guess that may not arise. Suppose you just want to stick with seven colours like Blue, yellow red, orange green, which are 5 that many people can distinguish from across the board. But then you might add pink and purple which some people might think are the same colour. Yet, there is hot pink and lavender which are very different in my perception. But also, brown and orange, same thing! So those are things that do need to be investigated and worked out.
Sheila: That’s totally true. Because people would name colours different things. Just the other day, I asked a question about what came first; “orange the fruit, or orange the colour?” And it turns out the fruit came first, and the reference to its colour followed.
Misha: Thats from Sanskrit actually
Sheila: Oh really?
Misha: Yeah, nāranga
Sheila: See! that’s to say that people name colours based on the things that are around them. So in a way that someone would name something “maroon”, I could have named it “Dust” because maroon refers to the colour of dust in the village where my father was born. In that context, maybe naming the colours could be something we revisit.
Misha: Yet again, English colonialism has prescribed itself!
Sheila: Right! (Laughs)
Misha: For example, there is no word for “Purple” in Hindi (at least that I can think of). I mean, there’s a word which means “like a red grape” which people use to refer to purple stuff. But there’s no word as “purple”. And I’ve certainly seen purple things back home so it’s not like the colour doesn’t exist in the cultural mind space. It’s just that, it is never articulated.
Sheila: Well, maybe someone will have to take up a study naming colours. Honestly, it would be pretty interesting if I went around and renamed colours based on the things around-
Misha: Right. So then, Red would become “Apple”
Sheila: Yeah exactly
Misha: But then there are so many apples. So which apple?
Sheila: Red apple? But then, there’s red again! So I guess there is no way to escape it then. We are all trapped! (laughs) But really though, I see an opportunity for those “in between” colours to be renamed without denying that those colours have a universality. Now I know what I want to do with the rest of my life!
Sheila: (Laughs) In all seriousness, this would make a great project, especially for people like kids. Because kids are usually very excited by “Why Not?” questions. And kids especially, call things based on what they see. For them, the word is the thing itself. So they would say ” a squeegy bobby thing” and you would get what they mean; you get that they are referring to a texture through the sound of the word itself. But then you reply and say “oh, you mean Sponge?”, because you’re so used to a name.
Misha: For example you could go out and try to rename the word blue and you could name it “Sky” But then, there is the sky at sunset, which is closer to orange.
Sheila: Well, we can have “Evening Sky” and “Morning Sky.” It’s all reminding me of some collection of untranslatable words. There is also this great article on The New Yorker written by this woman who is learning to speak French because her husband is French. And in one of the paragraphs, she talks about these certain untranslatable words- like a Swedish word “Knullrufs” which describes the way a person’s hair looks after sex. It is very specific to that particular thing. It’s a phrase in English but one word for the Swedish. Or the Portuguese word, “Saudade” which translates to ” a deep emotional state of nostalgic or profound melancholic longing for an absent something or someone that one loves.” And in Portuguese speaking places like Brazil, people use that one word in response to a question like “How are you?” Whereas, in English we seem to have just “fine.” So we could definitely have “Morning sky” or “Evening sky” to refer to blue of the morning or pinkish-orange of the sunset. But then again, some people may not have the same colour of Sky.
Misha: You’re right, it depends on where you are. For example, we are in the Pioneer valley where there is significantly less pollution and therefore, we get these really gorgeous sunsets. But do you see with equal regularity, in New York for example- sunsets like this?
Sheila: Still, there is something so great in that opportunity to recognize what unique things communities lack and love. And that in itself can create a song that based off of the specificities of your view/world/life. In that you think more of translating that imagery into sounds; For example, if I know a song is made up of Dust, Evening Sky, Morning Sky; that’s such a unique expression. Almost like having a smell of food that makes you recognize home.
Misha: Yes…isn’t it much more salient that way? And much more unique to the individual or the community that the individual is from
Sheila: In terms of neurological discoveries, there was something I was reading on Wired that was written about this woman who had lost her sight. And then this Dr came up with this tool that enables him translate physical imagery into auditory imagery. And the author wrote about how this woman was given this tool, and through auditory waves, she could “see” the fence. Like she could see the density- and the entire sensation was so whelming for her that she broke down in tears…
Misha: So interesting! I saw this TED talk once by this guy who has achromatopsia which is a really rare condition where you cannot see any colours at all except black and white. And he collaborated with a neurosurgeon to make a tool: So your occipital lobe is where your vision lies in your brain, is at the back of your brain. So…they installed a sensor, connected to a camera that hovered around the top-middle of his head. And the sensor had a camera attached to an earpiece. And this camera captured everything he looked at; and so, for every colour that the camera caught, it would translate that colour into an auditory sound that would ring in his head. The camera ended up picking about 256 sounds for colours. He had to learn all 256 sounds; which, essentially, means that he kinda learned synesthesia. And now, he is an artist that makes these wildly colourful pieces, using the sounds to paint colour. And that way, he learned to see colour
[A Piece by Neil Harbisson who “Listens to Colour”]
Sheila: Just the fact that you can learn something like colour, is fascinating news to me. It’s almost a disservice to think so simply about what we call “facts.” Like, I would usually say you either see colour or you don’t, and then maybe, I’d make some room for being colour blind. But that’s such an unfortunate way of closing that conversation because obviously, colour doesn’t have to be perceived in one visual way; that’s not the only way colour can add to a person’s life, you know? That’s why these conversations are important and not just interesting. For instance, if you go to a place like where I’m from, and you talk about the idea of the digital world, and some people would still say stuff like “well, I dont know if that’s necessary when we are still hungry. Wait till we eat before we start to code” and etc. But if someone who only saw the world in black and white, didnt have access to digital technology like Neil Harbisson, there is a way in which that difference could be dismissed and even demonized. Basically, it’s interesting to see how the digital space opens up possibilities for the quality of someone’s life which can change by recognizing you do have the power to experience a coloured life, even if you don’t see it in the same way others do. And that’s something that can change your outlook on life. Yet, in some communities people think you have to be stuck with the conditions you were born in/with. Especially in environments where people are very stuck with the idea of a God who knows all and who plans all, so that the idea is that God creates you a certain way, with a certain destiny in mind, so the idea is that you shouldnt try and change/alter yourself because that’s changing God’s plan, or saying God made a mistake
Misha: Right! They might say, “You should be grateful” instead
Misha: I definitely get that. It reminds me of what happened when I was at a conference in Vienna. There were all sorts of people there; artists, graphic designers, art historians. A really wide variety of people from different disciplines. So the neuroscientists understood my perspective. But then, there were a bunch of people who just wouldn’t get it; they would say stuff like “But why should we see music as colour, or why should the deaf people see music in colour. They are deaf, thats what they were born with” And I’m like, this is just trying to make a tool, I’m not to make them undeaf, especially because it’s not as deafness is a thing that always needs to be cured
Sheila: (Laughs) Why do you think they were so stuck in their ways?
Misha: I don’t even know
Sheila: I think what happens in some cases like that is that sometimes, instead of having conversations with you, some people would have conversations with themselves about the use and then they come to conclusions, and then they say it out loud with a question mark at the end
Sheila: Well, apart from those type of people, what interesting, thrilled people have you met in the course of trying to fulfill your project?
Misha: What I really like about this project is that it has given me the chance to talk to musicians about how they want their own music to be represented. And it’s given me the opportunity to talk to other synesthetes, and I hadn’t really talked to any other synesthetes before I started this project. I didnt know that they were out there (laughs) and then most times I find out when I describe my project and some people say “Oh, I have that too!” Apart from talking to other synesthetes, I’ve also met some really interesting people at conferences. I went to a Hacking Arts Tech Conference at MIT, and part of it was remixing senses. And I’m there with my tiny little prototype box, next to these huge companies/startups with millions of dollars of grant money. And Im like “Hi, I had a five college Digital Humanities Microgrant, and I made these…”
Misha: (laughs) …With my little ukulele. And right next to me is Red bull with their giant LED screens (laughs) But really, this fellowship has given me the opportunity to stand in those spaces, even with those type of companies/startups. The opportunity to learn about some high tech engineering skills that I never would have bothered with. And because I’m not doing this for academic credit at all—this is just self-motivated learning— it has given me those skills, opportunity and vocabulary, to talk to engineering experts about what I need. And also, it has allowed me to learn from them. Apart from going to the hackathons, it has also given me the confidence to talk to venue organizers: So, not only in an orchestral context, but also talk to people who are in charge of venue spaces like Flywheel in Easthampton. So, I would ask them what they thought about my ideas, if it was possible…and it’s given me the opportunity to learn how to format those kind of questions/conversations
Sheila: That’s such an important skill
Misha: Yeah! And also, if I hadnt had something to talk to them about, I wouldnt have spoken to these people at all. And being able to talk about Kandisnky for instance…
Misha: So Wassily Kandinsky is the guy who primarily motivated the project because he is an artist who had synesthesia: He would go to concerts in the evening and paint what he saw at those concerts. Kandinsky is arguably the father of modern abstract art. And that’s another thing, this project has also enabled me to talk to art historians and tell them certain things with confidence; especially if they consider abstract art their field of expertise and then I tell them Kandinsky had synesthesia, it’s usually a learning moment for some of them. They know his paintings were influenced by music, but most don’t actually know the extent of that “influence.” And so apart from learning from these art historians, I also come with an opportunity to tell them something new.
Wassily Kandinsky. Murnau Street With Women, 1908