ios-keyboard-limitations

# iOS Keyboard Extension Limitations When building iOS custom keyboards with voice/audio features, these are the hard limitations discovered through the PolyVoice project. ## 🔴 Hard Limitations (Cannot be worked around) ### 1. Microphone Access — DISALLOWED **Keyboard extensions cannot access the microphone.** - `AVAudioRecorder` will fail with permission error - `SFSpeechRecognizer` is unavailable - No Siri integration from keyboard context **Why:** Apple security model — keyboards run in sandbox and could keylog audio. ### 2. Open Other Apps — BLOCKED **Keyboards cannot programmatically open the main app or any other app.** - `UIApplication.shared.open()` returns false - URL schemes don't work (`myapp://`) - ` ExtensionContext.open()` not available **Why:** Prevents malicious keyboards from launching apps without user consent. ### 3. Memory Limit — ~50MB **Keyboard extensions have strict memory limits (~30-60MB).** - App terminated silently if exceeded - No crash log, just disappears - Heavy audio processing = instant death **Mitigation:** - Record at 16kHz mono (not 44.1kHz) - Use 32kbps bitrate max - Immediate file cleanup after processing - 60-second max recording hard limit ### 4. No Persistent Storage **UserDefaults unavailable, only App Groups.** - Standard `UserDefaults` doesn't persist - Must use `UserDefaults(suiteName: "group.com.company.app")` - Requires App Group capability in both targets ### 5. Network Requires "Full Access" **API calls fail without user enabling "Allow Full Access" in Settings.** - User must explicitly enable: Settings → General → Keyboard → [Keyboard Name] → Allow Full Access - Most users won't do this - Cannot prompt or explain from keyboard UI effectively ## 🟡 Partial Workarounds (User friction) ### The "Open App" Workaround **Goal:** Let user tap a button to open main app for recording. **Attempt:** ```swift // This does NOT work extensionContext?.open(URL(string: "myapp://record")!) ``` **Reality:** Must use `UIApplication.shared.open()` outside extension context, but keyboards can't call this. ### The Manual Switch Pattern **What actually works (with friction):** 1. **User taps button in keyboard** → Shows alert: "Open PolyVoice to record?" 2. **User manually switches to main app** (Home button, swipe, etc.) 3. **Main app detects active session** (via App Groups / shared state) 4. **Main app auto-records** on appear 5. **Auto-stops on silence** (2 seconds) 6. **Auto-copies** to clipboard 7. **User manually switches back** to target app 8. **Keyboard auto-pastes** on reappear **User flow:** ``` Keyboard → Tap mic → [Manual: Switch to app] → App auto-records → [Manual: Switch back] → Keyboard auto-pastes ``` **Friction points:** - Two manual app switches - Context switching breaks flow - Users forget to return - Clipboard may be overwritten ## 🟢 Alternative Architectures ### Option 1: Share Extension (Better for Audio) **Use Share Sheet instead of keyboard.** - Full app capabilities - Can record audio - Can process and return text **Limitation:** Not a keyboard — user must open share sheet per text field. ### Option 2: Full App Mode **Don't use keyboard extension — use main app only.** - User opens app - Records dictation - Copies result - Switches to target app - Pastes manually **Benefit:** No memory limits, full mic access, reliable. **Cost:** More friction than keyboard. ### Option 3: Siri Shortcuts Integration **Provide Siri Shortcuts for voice-to-text.** - "Hey Siri, dictate with PolyVoice" - Returns text to current app - Fully supported by Apple **Limitation:** Not instant, requires Siri setup. ## 📊 Decision Matrix | Approach | Mic Access | Memory | User Friction | Apple Approved | |----------|-----------|---------|---------------|----------------| | Keyboard extension | ❌ No | ⚠️ 50MB | Low (if no audio) | ✅ Yes | | Keyboard + audio workaround | ❌ No | ⚠️ 50MB | 🔴 High | ✅ Yes | | Share extension | ✅ Yes | ✅ Full | 🟡 Medium | ✅ Yes | | Full app only | ✅ Yes | ✅ Full | 🟡 Medium | ✅ Yes | | Siri Shortcuts | ✅ Yes | ✅ Full | 🟡 Medium | ✅ Yes | ## 🎯 Recommendation **For voice dictation/AI transcription:** 1. **Don't build a keyboard extension** — the limitations make it frustrating 2. **Use Share Extension** — Apple-supported, full capabilities 3. **Or full app** — simplest to build, most reliable 4. **Add Shortcuts** — for power users who want speed **For non-audio keyboards (emoji, translation, etc.):** Keyboard extension works great. Just avoid audio features. ## 📚 References - Apple's official docs: https://developer.apple.com/documentation/uikit/keyboards_and_input/creating_a_custom_keyboard - Custom Keyboard Programming Guide (WWDC sessions) - PolyVoice project learnings (~/Projects/polyvoice-keyboard/)