Bisindo Sign Language CoreML & Foundation Models App
On-Device Sign Language Recognition & Contextual Translation using CoreML & Foundation Models
Kumo
Link to LinkedIn Post
Storytelling: Bisindo Sign Language App
For the past few weeks, my team and I have been diving deep into Apple’s ecosystem to expand our technical understanding. The initial concept was simple: explore Apple frameworks and hardware by experimenting with iPhones, iPads, and Apple Watches.
We explored CoreML, AVFoundation, and Foundation Models to build a sign language app designed to bridge communication between teman tuli (deaf friends) and non-signers. Our goal was to leverage Foundation Models and CoreML to translate and summarize contextual conversations between Sign Language (Bisindo) and spoken text.
Eventually, we scaled the project down to its core — a streamlined, highly performant sign language learning and recognition app.
🤖 Deep Dive: CoreML & Foundation Models
- CoreML & CreateML: Built custom machine learning vision models using CreateML and optimized them through the ML Pipeline to keep on-device inference lightweight and responsive.
- Foundation Models: Designed system prompts to contextualize dialogue between teman tuli and non-signers, ensuring natural conversation flow.
- Showcase Prototype: Demonstrated a live two-way translation prototype during the academy showcase to industry experts.
📸 Gallery & Project Photos
🤟 Bisindo Sign Language app interface & gesture recognition
🧠 CoreML & CreateML dataset training & pipeline optimization
💬 Foundation Models system prompt design for contextual translation
📷 AVFoundation real-time camera video stream processing
✨ Live prototype demonstration for showcase visitors & industry experts
🎉 Team celebration after successful macro challenge showcase
👥 Team & Mentors
A huge thank you to our macro challenge team and mentors:
🌟 Mentors
- 🎯 Dimas Ackyl Mohamad — Design Mentor @ Apple Developer Academy
- 🎯 Daniel Aditya Istyana — Technical Mentor
- 🌿 Kamilia L. — Technical Guidance