🔍 LensTranslate

Paste a screenshot or a TikTok link, get it translated in place.

Drop in a screenshot of a foreign comment section and LensTranslate OCRs it, translates every text block, and draws the translation back over the original image, Google-Lens-style, in the browser. Paste a TikTok link instead and it pulls the subtitles and translates those. No account, no install.

▶️ Try the live demo: free-tier hosting, so the first request after idle takes a second or two to cold-start. YouTube and Instagram links are blocked by those platforms (the app explains why in under a second). Daily API quotas are capped; if a demo button ever errors, that's the cap.

Spanish comment section, one click: translated in the card, then flipped in place over the original image.

flowchart TD
    A[🖼️ Screenshot<br/>pick, paste or drop] --> C[🔍 OCR<br/>Google Vision REST with per-block boxes]
    B[🔗 Video link<br/>TikTok URL] --> D[📝 Subtitles<br/>yt-dlp VTT parsing]
    C --> E[🌐 Translate<br/>Google Translate v2, one batched call]
    D --> E
    E --> F[🎯 In-place overlay<br/>translation drawn over the image]
    E --> G[📋 Translation card<br/>original plus translated text]

Loading

---

✨ Features

• 🎯 In-place overlay translation: translated text rendered over each source block, with backdrop colors sampled from the screenshot and a show-original toggle.
• 📋 Paste or drag-drop anywhere: Win+Shift+S a foreign tweet, Ctrl+V into the page, done.
• 🌐 Six target languages: English, Español, Français, Deutsch, 한국어, 日本語.
• 🇰🇷 One-click examples: bundled Korean/Spanish/German screenshots that run the real pipeline, not canned responses.
• 🎬 Video link translation: TikTok URL in, translated subtitle transcript out.
• ⚡ Honest fast-fails: YouTube (datacenter-IP bot checks) and Instagram (login wall) fail in under a second with the real reason, not a 20-second retry loop.
• 🧱 Dense-image guard: text walls (40+ blocks) skip the overlay gracefully instead of rendering a box mosaic.
• 📱 Same codebase runs on mobile: the React Native app works in Expo Go against a local backend.

---

🧰 Tech stack

Area	Tooling
Frontend	React Native + Expo (react-native-web for the browser)
Routing	expo-router
Styling	NativeWind v4 (Tailwind for React Native)
Backend	FastAPI as a Vercel Python serverless function
OCR	Google Cloud Vision (plain REST, no SDK)
Translation	Google Cloud Translate v2 (plain REST, batched per block)
Video subtitles	yt-dlp with curl_cffi impersonation
Image processing	Pillow
Hosting	Vercel (Expo static export + Python function)

---

🛠️ How I built it (the process)

I kept running into TikToks and comment sections I couldn't read, and I wanted translating them to feel like nothing: share to an app, see the translation. The catch is that an iOS share extension needs a Mac and Xcode, and I have neither. So I pivoted to the web, where paste does the share sheet's job: Ctrl+V a screenshot or paste a link, same zero-friction idea, no App Store required.

I built the pipeline first as a React Native app in Expo Go talking to a local FastAPI server (screenshot to Google Vision for OCR, text to Google Translate, links through yt-dlp for subtitles). Getting it actually online was its own project: I refactored the app to run in the browser with react-native-web, wrapped the backend as a Vercel serverless function, and swapped Google's heavy gRPC SDKs for plain REST calls so everything fit in a serverless bundle. The overlay came last, and it was the fun part: Vision already returns a bounding box for every text block, so I draw each translated block back onto the image where the original text was.

---

📚 What I learned

• Constraints are design inputs. Losing the Mac killed the share extension, not the idea. Paste and drag-drop on web deliver the same zero-friction flow, and refactoring the Expo Go app into react-native-web + serverless taught me that "runs on my machine" and "deployed" are two different projects.
• Platform walls are real engineering. YouTube blocks datacenter IPs (works locally, fails from Vercel); Instagram walls everything behind a login (fails everywhere). Root-causing which wall you're hitting, then shipping an honest sub-second error instead of a hanging retry loop, is still shipping.
• Kill your own features when the evidence says so. I built a Gemini-powered "slang-aware" translation layer, then A/B-tested it against plain Google Translate on Korean internet slang. Google won: more accurate, more specific, free. I cut the feature the same day. The A/B cost an hour; shipping the worse version would have cost the demo.

---

🚀 How it could be improved

• Caption-less videos: audio transcription via Whisper needs ffmpeg and more memory than a serverless function offers; a small dedicated host would unlock it.
• YouTube in production: works locally, blocked from datacenter IPs; a residential proxy would fix it at the cost of the $0 budget.
• Overlay text fitting: current version uses per-block boxes with sampled solid backgrounds; per-line boxes and real inpainting would reach Lens-level polish.
• PWA + Android share target: would complete the original share-sheet vision on the platform that allows it.
• Translation history: localStorage persistence of past results.

---

▶️ How to run the project

1. Install deps

# Backend (from repo root)
python -m venv backend/venv
backend\venv\Scripts\activate   # Windows (source backend/venv/bin/activate on mac/linux)
pip install -r requirements.txt

# Frontend
npm install

2. Run it

# Terminal 1: backend
cd backend
venv\Scripts\activate
uvicorn main:app --reload --port 8000

# Terminal 2: frontend (press w for web)
npx expo start

Opens at http://localhost:8081; the web app talks to the backend on :8000.

3. Use it

1. Click a Try an example button, or paste (Ctrl+V) / drop any screenshot with foreign text.
2. Pick a target language chip if you don't want English.
3. Read the translation in the card, and in place on the image below.
4. Or paste a TikTok link and hit Translate video.

Google Cloud setup (required for OCR + translation)

1. Create a Google Cloud project and enable the Cloud Vision API and Cloud Translation API.
2. Create a service account, download its JSON key.
3. Create backend/.env:

GOOGLE_APPLICATION_CREDENTIALS=C:\path\to\service-account.json

Recommended: set daily quota caps on both APIs in the console; this app is built to live inside the free tier.

Deploying to Vercel

The repo deploys as-is with vercel --prod: vercel.json builds the Expo static export to dist/ and mounts api/index.py (a wrapper around backend/main.py) as a serverless function. Set one env var in Vercel: GOOGLE_SERVICE_ACCOUNT_JSON = the full JSON key pasted as a string.

Mobile (Expo Go)

The same app runs in Expo Go as a proof of concept: start the backend on your LAN, set your machine's LAN IP in app.config.js (extra.apiBaseUrl), run npx expo start, scan the QR code.

Tests

cd backend
venv\Scripts\python test_links.py

Self-check for the VTT subtitle parser (prints ok).

_{Built on: a Windows laptop, a $0 budget, and no Mac. That last constraint shaped the whole architecture.}