AR Subtitles
Andre Ahuna
AR Subtitles is a project to help people communicate across language barriers and hearing disabilities.
The aim is to develop an app for Meta Quest 3 which shows the user's surroundings with the device's onboard passthrough cameras and composites speech-to-text subtitles onto it. The user has the ability to pick a language the processed text will translated to. All in-app menus will support hand tracking.
A possible additional feature is locating speaker(s). With this, subtitles could be positioned more helpfully and individual speaker audios may be possible to isolate from background chatter. These features are still in R&D.
Platform: Meta Quest 3
Technology: Unity
Milestone 1 (07.10)
- Feature 1 - OpenXR Unity Scene setup
- Feature 2 - Hand tracking setup + a mock menu
Milestone 2 (21.10)
- Feature 1 - Implement cloud-based speech-to-text. Required supported languages: English, Estonian, German, French, Spanish.
Milestone 3 (04.11)
- Feature 1 - Implement cloud-based translation. Required supported languages: English, Estonian, German, French, Spanish.
Milestone 4 (18.11)
- Feature 1 - Integrate everything. VR passthrough + speech-to-text + translation.
- Bonus feature - Hand-tracked menus for selecting languages. Gesture-based menu summoning.
Milestone 5 (25.11)
- Feature 1 - Gesture-based menu summoning.
- Feature 2 - Always-on, threshold based decaying voice detection.
- Feature 3 - Polish: UI coat of paint, non-interrupting language change.