Case study
Livoic
A Chrome extension and realtime backend that captures live Twitch audio, transcribes and translates speech, and renders Netflix-style subtitles over the player.
Overview
Livoic (Live Translator) is an independent product: a Chrome Manifest V3 extension plus a Fastify realtime backend that turns live Twitch audio into translated player subtitles while preserving the original stream audio.
Context
I built Livoic solo to explore realtime media pipelines, extension architecture, and game-aware machine translation for live streaming — without microphone access or provider secrets in the extension.
My role
I own product direction, extension UX, realtime protocol, speech/translation providers, and iteration as creator and developer.
Ownership
End-to-end ownership across the pnpm monorepo: extension (popup, service worker, offscreen, overlay), Fastify WS server, shared protocol/schemas, game profiles, and correction/routing pipelines.
Problem
Viewers who do not speak a streamer’s language miss live context. Generic translators ignore gaming terminology, and naive subtitle UX flashes unfinished English fragments over the player.
Contribution
- Shipped tab-only audio capture → PCM streaming → STT → translation → overlay pipeline.
- Designed typed client/server events with Zod and short-lived HMAC realtime tokens.
- Added game detection from Twitch category metadata and terminology profiles (e.g. Path of Exile).
- Built low-confidence transcript correction with an audio ring buffer and selective re-transcription.
- Added topic-aware routing and sentence assembly so incomplete STT fragments merge into natural subtitle lines.
Technical decisions
- pnpm workspace monorepo with shared protocol and UI packages.
- Provider interfaces so mock STT/MT work locally and Deepgram/OpenAI plug in for production.
- Shadow DOM overlay isolation to survive Twitch player CSS and fullscreen modes.
- No API keys in the extension — secrets stay on the Fastify backend.
Challenges
- Reliable Chrome tab capture and offscreen audio without capturing the microphone.
- Avoiding partial-transcript “English flash” in the subtitle overlay.
- Preserving official skill/item names while producing natural target-language speech.
- Keeping correction and routing latency acceptable for live viewing.
Outcome
- A working independent realtime translation stack for Twitch with game-aware terminology and local mock providers.
- Demonstrates extension + websocket backend ownership beyond a UI-only Chrome project.
Visuals
Technologies
- TypeScript
- Chrome Extension APIs
- Vite
- React
- Fastify
- WebSockets
- Zod
- Deepgram
- OpenAI
- pnpm workspaces
- Docker