Growing up, Melissa Malzkuhn cherished story time, squeezing on a couch or bed with her two siblings while their mom read them their favorite books.
“My mom would just open up a book and just start signing the book,” Malzkuhn said.
Malzhuhn’s mom did the same with TV, translating captions in real time as the kids watched cartoons or movies. The exposure paid off. Before long, Malzkuhn was able to read on her own.
Malzkuhn’s experience and early exposure to signing, and therefore reading, is unfortunately far from the norm. As the third generation in her family born without the ability to hear, she grew up learning American Sign Language. But more than 90 percent of deaf children are born to hearing parents. Many of those children lack early access to ASL, a language used by hundreds of thousands of members of the deaf community that is celebrated every April 15 on National ASL Day. As a result, many deaf children experience a language deficit early on that can set them behind on learning and literacy.
“Language deprivation really means that someone is set on a path of continued struggle, they’re always catching up,” Malzkuhn, who signs, said through a translator. “They’re missing a lot.”
Malzkuhn doesn’t think that should be the case. The 37-year-old advocate, artist and digital strategist has spent the last decade-plus leveraging her passion for art, technology and creative storytelling to increase exposure to ASL and, hopefully, close that gap.
These days, that work is happening in Motion Light Lab, a research center Malzkuhn runs at Gallaudet University in Washington, D.C. Core to its mission is using the power of visuals to improve learning and language for the deaf community — “The lion’s share of information comes in through the eyes. How does that impact cognitive processes?” she said of the question driving the center’s work — and even the building that houses the lab reflects that. It’s airy, with wide hallways and open spaces, and the hallway leading to the lab is lined with colorful posters. Those posters depict the cover art for a series of digital children’s books Malzkuhn built for iPads so families can follow along as actors sign the words. Every aspect of the digital books, from the technology to the illustrations and storylines, was specifically designed with a deaf audience in mind.
“Our whole approach was very organic,” she said. “How do you create the reading experience for deaf children? How do you create an experience that’s bilingual and seamless visually? And then we also needed to develop the actual story.”
The books, which have been translated into multiple languages, have been a hit among both kids and their parents.
“A lot of parents will struggle with feeling like they don’t sign well enough. And we have to tell them, like, actually it’s okay. It’s okay even if you don’t sign perfectly,” she said. “This has turned out to be an amazing way for parents to feel a little bit more confident about signing with their kid. They’re able to point things out and still engage, and there’s that reading experience.”
Malzkuhn is now building on the success of that concept with projects like an app featuring nursery rhymes signed by cartoon avatars and new experiments involving artificial intelligence and early learning. The work to make those ideas a reality happens inside the one-room lab, which looks like a cross between a classroom and a commercial studio. Chairs and tables are mixed in with photo lights and giant monitors. A black track suit covered in motion sensors, which Malzkuhn and colleagues use to record the movements they’ll turn into signing avatars, hangs on a mannequin. No fewer than 16 cameras are mounted around the room to catch the gesturing from every angle.
“Our goal is to create signing characters, and we need to figure out a technology that can do that, and do that in 3-D,” she said. “We need to see what that looks like and the complexity of signing, like how much detail do you need in the fingers? How much do you need to use the right facial expressions?”
While promoting early learning and literacy is a central focus of Malzkuhn’s work, her products aren’t just for kids. Over the years, she’s launched an award-winning iPhone ASL app that teaches users how to use ASL to communicate common phrases (“You’re cute,” “That’s a cool shirt”); she created GIFs that translate key Washington, D.C., words (“The Capitol,” “The monument”); and she brought ASL to group chats everywhere with a series of Apple iOs stickers signed by America’s Top Model alum Nyle DiMarco. The stickers include slang like “Omg” and “Lol.” High demand has led Malzkuhn and her team to add even more vocabulary categories to the ASL app since its initial launch.
“We found that our audience ended up being so much wider than we had ever expected,” she said.
Malzkuhn’s groundbreaking initiatives have already snagged her time on a Ted Talk stage and a spot in the Obama Foundation’s inaugural two-year fellowship class. But her work is far from over. The lab is now experimenting with 3-D avatars, artificial intelligence and more cutting-edge technology to further its mission. She just launched a new edition in her children’s book series. And a recent grant from Mitsubishi Electric America Foundation will allow her to spend the next three years training members of the deaf community to make storybook apps of their own.
In recent years, Malzkuhn’s commitment to early exposure to language and literacy for deaf children extends well beyond work hours. After her own son was born deaf, she found herself bringing her experiments into his life. He loves the apps and books his mom created. And, of course, she’s continuing the tradition of interpreting shows and story time at home.
