Swiss software engineer and cloud architecture specialist Marco Gerber enjoys getting hands-on with tech in his free time as well as at work. His expertise in machine learning led him to recognise the potential for a handheld AI device that could describe the surrounding environment, which would be particularly useful as an accessibility tool for people with hearing or visual impairments. His idea was to develop a ‘point-and-shoot’ device that would provide an audio description of the user’s surroundings. The result was the Raspberry Pi Zero 2 W–based Hear The World device, which offers both text and spoken descriptions alongside haptic feedback...
Swiss software engineer and cloud architecture specialist Marco Gerber enjoys getting hands-on with tech in his free time as well as at work. His expertise in machine learning led him to recognise the potential for a handheld AI device that could describe the surrounding environment, which would be particularly useful as an accessibility tool for people with hearing or visual impairments. His idea was to develop a ‘point-and-shoot’ device that would provide an audio description of the user’s surroundings. The result was the Raspberry Pi Zero 2 W–based Hear The World device, which offers both text and spoken descriptions alongside haptic feedback.
Azure OpenAI Service (GPT-4) provides audio- and text-based descriptions of what it sees, helping visually impaired users
Instant inspiration
Zurich-based Marco is a Microsoft MVP with a focus on AI and Azure, so cloud-based projects are second nature. He enjoys trying out new technologies, “Raspberry Pi and various sensors being some of my favourite ways to combine hands-on tinkering with learning”. His other passions are travel and photography. With a keen visual eye, Marco found himself experimenting with early multimodal AI models capable of analysing images. This led on to interpreting individual images from a webcam, sequences of frames and, eventually, a form of video analysis in which the AI processes information about its surroundings. “Later, I also explored audio models, creating small pipelines where an image could be transformed into text, and that text into audio — essentially, image to text to audio.”
Marco’s Azure AI image recognition device began with a breadboard, testing each component in turn
Having got this far, it struck Marco that Raspberry Pi could serve as the perfect platform for an image-to-text-to-audio project. “With its camera port and compact form factor, it opened the possibility of building a handheld device that could describe the surrounding environment — something particularly useful as an accessibility tool for people with hearing or visual impairments,” he says. Having a tangible use case in mind motivated him to push the idea further.
New territory
Marco was already an enthusiast of both Python and Raspberry Pi thanks to his AI work and “the countless great libraries” that he could use in his project. The compact dimensions of Raspberry Pi Zero 2 W seemed ideal for his portable device. However, most aspects of the project were new territory for him, “so it was very much a process of learning by doing”.
Pressing a large button on the Hear The World device results in a wide-angle photo being taken by the Adafruit Zero Spy Camera and analysed by Azure GPT 4.0
Getting the speaker working with Raspberry Pi required a digital-to-analogue converter (DAC), as well as an amplifier to boost the resulting file enough to be audible. “Each component had its own quirks — different libraries, different formats, and unique requirements — which added complexity but also made the project exciting.”
Overall, Marco is pleased with his foray into assistive technology, completed as and when during his free time over the course of a couple of months. “It could be adapted to transcribe spoken words and display them for hearing-impaired people, or even serve as an interpreter to translate sign language into spoken language,” he suggests.
Adafruit’s mini oval speaker is loud enough to provide clear details of what the camera sees
Read more articles like this in Raspberry Pi Official Magazine #159
You can grab this issue from Tesco, Sainsbury’s, Asda, WHSmith, and other newsagents, including the Raspberry Pi Store in Cambridge. It’s also available from our online store, which ships around the world. And you can get a digital version via our app on Android or iOS.
You can also subscribe to the print version of our magazine. Not only do we deliver worldwide, but people who sign up to the six- or twelve-month print subscription get a FREE Raspberry Pi Pico 2 W!