Whispering Hat

Whispering Hat is a wearable assistive AI system designed to help blind and visually impaired people move through the world with more awareness and confidence.

Instead of describing everything in view, the hat focuses on information that can help in the moment: an obstacle in the walking path, something getting closer, or a person the wearer may recognize. It turns those observations into short spoken alerts, helping the wearer notice important changes without being overwhelmed by constant narration.

Image recognition can also help the wearer know who they are interacting with by identifying familiar people locally. Facial-expression recognition adds another layer of social context by describing visible cues such as a smile, surprise, or concern, helping the wearer take part in conversations and understand the emotional tone of an interaction. These are estimates of what is visible, not claims about what someone is truly feeling.

By bringing this awareness directly to the wearer through local audio, the project can support more independent movement and make nearby surroundings easier to understand. Voice controls also let the wearer pause or resume detection when they choose.

Watch the wearable system detect nearby obstructions and communicate relevant alerts to the wearer.

Watch the demonstration on YouTube ↗

A normal object detector can identify dozens of things in a camera frame, but announcing every detection would overwhelm a blind user. Whispering Hat therefore evaluates context before deciding whether a spoken alert is useful.

• What the object is, where it is, and whether it is inside the walking path
• How far away it is, whether distance is shrinking, and whether its bounding box is growing
• Whether a collision may be approaching or the hazard level has increased
• Whether the system already warned the wearer
• Whether a person is known and what visible facial expression is estimated

Another important direction for the project is reading diagrams. A future mode could describe the layout of a chart, map, science figure, or classroom diagram in words, helping a visually impaired student access visual information without needing someone beside them.

All awareness and decision-making runs inside the Raspberry Pi 5 local processing boundary. Camera capture, depth sensing, inference, risk logic, speech, and voice control are coordinated as separate workloads rather than one blocking loop.

Whispering Hat does not simply speak the name of every object YOLO detects. Its rules classify situations as SAFE / PASSIVE, NOTICE, CAUTION, or STOP, then modify those risks using the walking corridor and the object's category.

• People receive context-aware notices or cautions rather than constant STOP warnings at ordinary conversation distance.
• Vehicles are treated more aggressively when close or approaching.
• Animals become more important with distance and approach behavior.
• Chairs, benches, bags, and suitcases matter more when low in the image and inside the path.
• Traffic lights and stop signs are normally passive visual context.
• Real ToF time-to-collision information can further escalate an in-path result.

The risk score is built from engineered rules and sensor information, not copied from YOLO confidence. A trapezoidal walking corridor makes objects in the path more important than objects at the side, while camera/depth fusion adds real distance and time-to-collision signals. Short tracking and alert histories smooth approach signals and prevent the same warning from being repeated on every frame.

Whispering Hat is designed so its core awareness system does not depend on Wi-Fi or cloud services.

YOLO11n, ByteTrack, VL53L5CX processing, risk logic, HOG face detection, cached face matching, FER2013 TFLite inference, Vosk recognition, and eSpeak speech output operate locally on the Raspberry Pi. Once the models and software are installed, no cloud inference is required.

No Wi-Fi required for core operation.

Vosk accepts only “detection on” and “detection off.” Turning detection off pauses awareness processing while the camera continues running, making voice control deliberately simple rather than a general assistant.

The difficult part was not running one AI model. It was getting everything to work together quickly enough to be useful. The camera may produce frames faster than YOLO can process them, while the depth sensor produces a separate 8x8 coordinate system that must be aligned with the camera view.

Background workers, latest-frame capture, scheduled inference, tracking and depth memory, face memory, expression smoothing, alert memory, priority queues, cooldowns, and risk escalation keep one slow operation from freezing the wearable system. Face recognition, emotion inference, voice recognition, and speech therefore run on their own schedules or workers.

The hardest part of Whispering Hat was not getting YOLO, a distance sensor, or a face model to work individually. It was deciding how all of them should work together.

• An object detector can tell me that a chair exists, but that alone does not tell a blind wearer whether the chair matters.
• Walking-corridor logic made location important; the VL53L5CX made real distance available; tracking made it possible to follow individual objects over time.
• Distance history, bounding-box growth, and time-to-collision added multiple ways to recognize danger.
• Face matching can turn a generic person detection into the name of someone the wearer knows, while expression estimation adds optional social context.
• Alert memory and prioritized speech made the system more selective about what it actually says.
• The biggest lesson was that assistive AI is not about announcing everything a model can recognize. It is about deciding which information deserves the user's attention.