Getting college kids together for a weekend with laptops and whiteboards to dream up new apps has been at the core of every “hackathon” held at MIT’s Computer Science and Artificial Intelligence Laboratory. Last month, about 100 mostly undergrad students came for that experience from as close as MIT and Olin College and as far away as UC Berkeley and UT Austin.
The twist at the hackathon, organized by the Perkins School for the Blind, was the supporting cast of two dozen experts and mentors from the blind and visually disabled community.
The students divided into 10 teams, named after colors, and picked one of eight challenges Perkins had crafted, such as assisting a blind person to navigate an indoor space or to pick up nonverbal cues in video conference conversations. But before writing a line of code, the teams met with people with a disability relevant to the challenge they had selected.
The orange team, which chose to tackle indoor navigation, spent all morning talking with mentors who were blind or had low vision. They heard about the difficulty of navigating an office hallway, for example, despite braille descriptions on the walls outside every room.
“Some feedback that we got was that most blind or visually impaired individuals are pretty much able to navigate to an extent but have trouble with orienting in new spaces,” Ayush Chakraborty, a junior at Olin, said.
Chakraborty, along with three other Olin students, came up with the idea for a mobile app that could “speak” room titles, airport gates, or other descriptive information as the user passed by low-cost Bluetooth beacons that could be placed around a building, mall, or airport. They estimated it would take about 250 such beacons, priced at $3 each, to mark all of the gates and other major features at Logan Airport.
Perkins organized the hackathon, with Amazon as a sponsor, to get students thinking about using artificial intelligence and other technologies to solve the real-life challenges faced by people with disabilities, Sandy K. Lacey, executive director of the Howe Innovation Center at the Perkins School, explained. The innovation center, established last year, also tracks tech accessibility efforts and resources around the world.
“We wanted ‘Hack Disability’ to amplify the lived experience of people with disabilities and to center the innovation process in a human-centered design approach,” Lacey said.
The pink team, with students from all over the country, focused on the challenge of emotional interpretation. A blind person who cannot not see facial expressions on a Zoom call might miss out on a speaker’s nonverbal cues, like a smile or a grimace. The students worked on an AI-based app dubbed “Hear the Room” to evaluate speakers’ nonverbal cues and emit specific sounds, such as a rising tone to signify excitement, while the person is talking.
After two days of hacking in various corners of MIT’s sprawling Stata Center, fueled on pizza and soda, the teams presented their work to a panel of judges from Perkins, Amazon, and MIT.
Olin junior Krishna Suresh presented for the orange team, talking quickly with just two minutes to explain how the system could help a blind person at the airport get audio cues from the smart beacons as they passed gates, a restroom, and a lounge.
The judges probed the estimate for 250 beacons needed to cover Logan Airport but seemed satisfied by the team’s rationale, which was based on the airport having about 100 gates.
Some teams got tougher pushback. The green team worked on an app called “VizWhiz” to help a blind or visually impaired person design charts and graphics for presentations using AI. But the judging panel was concerned that major commercial developers could add similar features to their apps.
After the presentations, the four judges stepped into a private conference room to compare notes.
The panel selected as winners the orange team’s smart beacons and the pink team’s emotional assessment app. The third winner was the grey team, addressing the challenge of apps that can interpret sign language and “speak” what is being signed. The team developed an app called “My Own Voice” allowing for personalized computer voices when speaking sign language interpreted from a deaf person. Winners got $2,500 to continue their work, as well as being featured at evening programs at the Museum of Science this summer.
Josh Miele, one of the mentors and principal accessibility researcher at Amazon’s Lab 126, the company’s R&D hub, said he had “super fun” interacting with the students and giving feedback on their design concepts.
“What will last is not the projects, probably, but the learning,” Miele, who is blind, said. “We want these folks to be building a lifelong understanding of what meaningful disability technology really looks like and how it has to be customer-centered and user-centered.”
Aaron Pressman can be reached at aaron.pressman@globe.com. Follow him @ampressman.