Cornell Tech doctoral student Tobias Weinberg (left) and adviser Thijs Roumen (right), assistant professor at Cornell Tech, are working to transform assistive communication technologies. Credit: Alexandra Bayer/Cornell University

At 15, when a neurological condition took Tobias Weinberg’s ability to speak, aspects of his personality became more difficult to express.

Typing to communicate, he struggled to keep up in conversations, especially to make the jokes or sarcastic comments that had been his norm. And his first text-to-voice device was monotone, with Mexican or Spanish accents but not his native Argentinian.

“The monotone voices, the timing of interjections and conveying my personality through this new way of communication was definitely frustrating,” wrote Weinberg, now a doctoral student and Siegel PiTech Fellow at Cornell Tech. As part of the Matter of Tech Lab, he is exploring how artificial intelligence (AI) can enhance the technologies that he and more than two million Americans with speech disabilities use to communicate.

Through a standing partnership between Cornell Tech and YAI—a nonprofit that supports more than 20,000 people with intellectual and developmental disabilities in New York, New Jersey and California—Weinberg spent a year working with a group of Augmentative and Alternative Communication (AAC) users who live in group homes in Tarrytown, New York to better understand needs and behaviors and to improve prototypes.

The resulting research and lines of inquiry, which incorporate Weinberg’s own experience, could transform assistive technology design.

The field is taking notice. Weinberg’s first paper—“Why so serious?”—won best paper honorable mention and jury best demo awards at the prestigious Conference on Human Factors in Computing Systems (CHI). Those are highly coveted commendations according to his advisor, Thijs Roumen, assistant professor at Cornell Tech.

“Tobi really is a trailblazer,” said Roumen, who has a joint appointment in the Cornell Ann S. Bowers College of Computing and Information Science. “He’s been developing technology while also using the technology, which changes the way it’s shaped and the way we reflect on it. In the process, he’s hitting on a richness that is going to make the future of AAC technology much much better, and he’s also inspiring a whole generation of researchers.”

Judith Bailey-Hung, supervisor of the YAI Center for Innovation and Engagement, said he’s also inspiring the AAC users involved in his studies.

“For the people we support, it was very powerful to see that this person’s working on their Ph.D., they’re interested in improving how you communicate, and they want to involve you in that process,” said Bailey-Hung, who has supervised three Cornell Tech interns as part of a larger partnership. “It gives them a voice and a way to advocate for themselves.”

Humor, backchanneling and AI villains

Heather Klippel, who has cerebral palsy and lives in a YAI group home, has similar frustrations with AAC devices to Weinberg’s—she gets overwhelmed when too many people are speaking and struggles to convey tone and humor.

“Those things are very hard to express as a nonverbal person,” Klippel wrote.

In the first of two studies, Weinberg interviewed Klippel and six others and designed an interface that could help users write jokes or humorous comments they can then interject in real time.

“There is an inherent tradeoff between agency and efficiency when designing AI tools that support communication,” Weinberg wrote. “While an AI auto-complete will enable making humorous comments faster, there is a risk that it diminishes the user’s sense of agency by making jokes for users instead of with the user.”

Weinberg designed interfaces that explored this tradeoff—in one, users selected keywords they wanted the AI to use in crafting a joke; in another, they were able to edit and modify AI-written jokes; and in another, they could simply choose a joke that the AI provided.

“What we found is in time-pressured scenarios, like making a humorous comment, AAC users were willing to give up some agency to deliver the comment faster,” Weinberg wrote. “This challenged the existing research that said AAC users care most about maximum agency, which is true in general but not always.”

At 15, Weinberg lost the ability to speak and found it harder to communicate certain aspects of his personality, like humor. Now, he’s working to make assistive communication technologies more expressive. Credit: Alexandra Bayer/Cornell University

That led Weinberg and his collaborators to think about the purpose of humor. Often, he said, the joke itself is less important than participation and engagement in the conversation. The team started to consider other types of “backchanneling,” or ways we communicate engagement, alongside the primary conversation, like saying “uh-huh” or nodding.

In a second study with the AAC users—resulting in a paper, “One does not simply ’Mm-hmm’” presented at the ASSETS’ Conference on Computers and Accessibility in October—Weinberg and his team found that the participants formed their own micro-culture of backchanneling, such as tapping their armrests to indicate agreement or raising eyebrows. The interviews and observations led him and his team to recommend a design approach that amplifies and incorporates what users are already doing, rather than imposing mainstream behaviors.

“There can be this tendency to just want to build an app and solve a problem,” Roumen said. “But by asking ourselves these fundamental questions and driving the curiosity that Tobi brings as a researcher to really understand what’s happening, we can now start to understand how we can be really impactful in this space.”

Those fundamental questions are often also ethical ones. For a third paper currently in submission, Weinberg developed an app that collected everything he’d typed over a period of seven months and used the text to train a large language model that could help facilitate and speed his communication.

While the resulting “AI-twin” captured a verbal identity, incorporating characteristic phrases and Argentinian slang, it failed in practice to suggest or provide that language in appropriate contexts and risked exposing private information at the wrong times. Weinberg also felt the app dampened control over his own self-presentation.

“AI is a very wonderful but dangerous technology, especially if it mediates everything we say as AAC users,” Weinberg wrote. “So, my work serves both sides, providing design guidelines for future developers and also playing the villain, warning of the socio-technical implications of AI in the lives of AAC users like myself.”

Building community, inspiring others

Weinberg disassembled his first computer at age 2 and at age 7 told his parents he wanted to invent things that would help people. But when he arrived at Cornell Tech for a summer internship in 2022, he didn’t know what a Ph.D. was and did not see it in his future.

Wendy Ju, associate professor at Cornell Tech, encouraged Weinberg to apply for the doctoral program after completing his bachelor’s in mechanical engineering at the Technion—Israel Institute of Technology. In 2023, he joined Roumen’s lab, intending to work on digital fabrication. But Roumen encouraged Weinberg, as he does all students, to find a project he really cared about.

“I told Thijs, there was this other thing I really care about, but neither of us has any experience with it,” Weinberg wrote. “He was on board to give it a try, and here we are.”

Weinberg and Roumen teamed with Stephanie Valencia at the University of Maryland, who specializes in AI and agency in AAC use. After overcoming steep learning curves—embarking on what Roumen calls “a journey” for them both—Weinberg is now inspiring others.

“It amazes me that somebody with an AAC device was going for his doctorate,” Klippel wrote. “I know that people with disabilities can achieve such high degrees in education, but it was quite an honor to actually meet somebody like this.”

The studies have also built community. Klippel said she became closer to another AAC user during the course of the studies and continued the friendship.

For Weinberg, seeing that connection form was one of the most rewarding parts of the research. “It didn’t feel like a workshop, it felt like a couple of friends hanging out and sharing anecdotes about our AAC hurdles and use, not only for me but also for them,” he wrote.

The other reward was seeing the participants use the systems to express themselves in new ways. Weinberg often replays a video from the humor study, of an AAC user working with the platform to write a joke and bursting into laughter at what she had created.

“That made all the hard work worth it,” he wrote.

Looking ahead, Weinberg hopes to reframe AAC—not as a workaround for missing speech but as a medium of expression. “This vision represents a step toward the broader goal of enabling AAC users to fully participate in spoken communication and to flourish in society,” he wrote.

More information: Tobias M Weinberg et al, Why So Serious? Exploring Timely Humorous Comments in AAC Through AI-Powered Interfaces, Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (2025). DOI: 10.1145/3706598.3714102

Tobias M Weinberg et al, One Does Not Simply ‘Mm-hmm’: Exploring Backchanneling in the AAC Micro-Culture, Proceedings of the 27th International ACM SIGACCESS Conference on Computers and Accessibility (2025). DOI: 10.1145/3663547.3746381

Citation: Student draws on experience to transform assistive communication (2025, November 4) retrieved 4 November 2025 from https://techxplore.com/news/2025-11-student-communication.html

This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.