Swiss software engineer and cloud architecture specialist Marco Gerber enjoys getting hands-on with tech in his free time as well as at work. His expertise in machine learning led him to recognise the potential for a handheld AI device that could describe the surrounding environment and that could be particularly useful as an accessibility tool for people with hearing or visual impairments. His idea was to develop a “point and shoot” device that would provide an audio description of the user’s surroundings. The result was the Raspberry Pi Zero 2 W-based Hear The World device, which offers both text and spoken descriptions alongside haptic feedbac…
Swiss software engineer and cloud architecture specialist Marco Gerber enjoys getting hands-on with tech in his free time as well as at work. His expertise in machine learning led him to recognise the potential for a handheld AI device that could describe the surrounding environment and that could be particularly useful as an accessibility tool for people with hearing or visual impairments. His idea was to develop a “point and shoot” device that would provide an audio description of the user’s surroundings. The result was the Raspberry Pi Zero 2 W-based Hear The World device, which offers both text and spoken descriptions alongside haptic feedback.
Instant inspiration
Zurich-based Marco is a Microsoft MVP with a focus on AI and Azure, so cloud-based projects are second nature. He enjoys trying out new technologies, “Raspberry Pi and various sensors being some of my favourite ways to combine hands-on tinkering with learning”. His other passions are travel and photography. With a keen visual eye, Marco found himself experimenting with early multimodal AI models capable of analysing images. This led on to interpreting individual images from a webcam, sequences of frames and, eventually, a form of video analysis in which the AI processes information about its surroundings. “Later, I also explored audio models, creating small pipelines where an image could be transformed into text, and that text into audio – essentially image-to-text-to-audio.”
Having got this far, it struck Marco that Raspberry Pi could serve as the perfect platform for an image to text to audio project. “With its camera port and compact form factor, it opened the possibility of building a handheld device that could describe the surrounding environment – something particularly useful as an accessibility tool for people with hearing or visual impairments,” he says. Having a tangible use case in mind motivated him to push the idea further.
New territory
Marco was already an enthusiast of both Python and Raspberry Pi thanks to his AI work and “the countless great libraries” that he could use in his project. The compact dimensions of Raspberry Pi Zero 2 W seemed ideal for his portable project. However, most aspects of the project were new territory for him, “so it was very much a process of learning by doing”.
Getting the speaker working with Raspberry Pi required a DAC (digital-to-analogue converter), but also an amplifier to boost the resulting file enough to be audible. “Each component had its own quirks – different libraries, different formats, and unique requirements – which added complexity but also made the project exciting.”
Overall, Marco is pleased with his foray into assistive technology, completed as and when over a couple of months in his free time. “It could be adapted to transcribe spoken words and display them for hearing-impaired people, or even serve as an interpreter to translate sign language into spoken language,” he suggests.
Read more articles like this in Raspberry Pi Official Magazine #159
You can grab this issue from Tesco, Sainsbury’s, Asda, WHSmith, and other newsagents, including the Raspberry Pi Store in Cambridge. It’s also available from our online store, which ships around the world. And you can get a digital version via our app on Android or iOS.
You can also subscribe to the print version of our magazine. Not only do we deliver worldwide, but people who sign up to the six- or twelve-month print subscription get a FREE Raspberry Pi Pico 2 W!