From a personal pain point to a fully-featured web app. A deep dive into building an AI-powered fitness coach using Gemini’s multimodal capabilities, structured JSON, and a privacy-first frontend.
The Hardest Part is Starting
Getting into fitness is intimidating. The gym feels like a stage, and every exercise comes with a nagging question: “Am I even doing this right?”
I was lucky—my cousins were experienced lifters who guided me through those first uncertain steps. They showed me what to do, how to do it, and made the beginning feel far less overwhelming. That experience sparked an idea: what if I could build an app that offered that same guidance? An AI partner that could help anyone start their fitness journey with confidence instead of confusion.
That idea became **Cloud…
From a personal pain point to a fully-featured web app. A deep dive into building an AI-powered fitness coach using Gemini’s multimodal capabilities, structured JSON, and a privacy-first frontend.
The Hardest Part is Starting
Getting into fitness is intimidating. The gym feels like a stage, and every exercise comes with a nagging question: “Am I even doing this right?”
I was lucky—my cousins were experienced lifters who guided me through those first uncertain steps. They showed me what to do, how to do it, and made the beginning feel far less overwhelming. That experience sparked an idea: what if I could build an app that offered that same guidance? An AI partner that could help anyone start their fitness journey with confidence instead of confusion.
That idea became CloudFitness, an AI-powered fitness coach built to run entirely in the browser. Here’s how I built it.
What it Does: Meet Your AI Fitness Partner
CloudFitness is designed to guide you through every stage of your journey with four core tools.
AI Form Coach
Upload a video of an exercise, and Gemini analyzes it frame-by-frame. It doesn’t just list mistakes; it identifies the single most important fix you need and gives you a simple, actionable correction plan, complete with encouragement.
Personalized Starter Plan
New to the gym? Answer a few questions about your goals, body, and lifestyle, and the AI generates a customized 4-week starter plan. It includes a full workout split and a basic nutrition guide to get you moving in the right direction.
Interactive Progress Book
Log your daily photos, weight, and notes. The app transforms your entries into a dynamic, satisfying flip-book that visually captures your transformation over time.
Activity Dashboard
The homepage keeps you motivated with a GitHub-style consistency graph and gives you quick access to every tool, making your progress easy to track and understand at a glance.
How I Built It: The Tech Stack
CloudFitness is built on a modern, privacy-focused tech stack designed for speed and reliability.
- Frontend: React and TypeScript form the foundation, providing a clean, maintainable, and type-safe interface. Tailwind CSS makes styling a breeze, keeping the design responsive and dark-mode-ready right out of the box.
- The AI Brain: The Google Gemini API is the star of the show. We use its multimodal capabilities to send a sequence of video frames for analysis. But the real game-changer is its support for structured JSON output. We can force the AI’s response to fit a specific schema, which makes the data predictable and incredibly easy to work with on the frontend.
- Privacy-First Data: All user data—reports, progress photos, and notes—stays on the device. We use IndexedDB (via the simple
idblibrary) to ensure complete privacy and full offline access. Nothing is ever sent to a server.
The User Experience: To make the app feel polished, I built a few custom UI features, including a smooth video trimmer with draggable handles, animated report visuals using SVG, and the react-pageflip library for that tactile progress book experience.
The Toughest Rep: Refining the AI Coach’s Eye
The biggest challenge was teaching the AI to be a nuanced coach. Early on, the feedback was too generic. It took dozens of prompt iterations to get Gemini to behave like a real kinesiologist—one who understands that “perfect” form varies by person and that feedback should be encouraging, not demoralizing.
The breakthrough came from instructing the AI to focus on the single most important correction. Instead of overwhelming the user with a list of ten minor issues, it hones in on the one thing that will make the biggest difference in their safety and performance.
A Personal Best: What I’m Proud Of
I’m incredibly proud of the depth and quality of the AI’s analysis. It goes beyond just spotting errors. It delivers constructive, safe, and genuinely encouraging feedback that feels like it’s coming from a real coach who wants you to succeed. It praises what you’re doing well before offering a clear, step-by-step plan to improve. That positive reinforcement loop is something I think is missing from a lot of automated tools.
My Biggest Takeaway: Gemini’s Secret Weapon
If there’s one thing I learned, it’s that Gemini’s structured JSON output is a superpower for building reliable AI apps.
Before, you’d have to pray the model’s text response was parseable. Now, you can define a schema and guarantee the output will be a clean, usable JSON object every single time. This feature alone saved me countless hours of development and debugging.
The Next Set: What’s Next for CloudFitness
This is just the beginning. The next major feature is to introduce real-time feedback using the Gemini Live API. Imagine getting instant, audio-based form corrections through your device’s camera while you’re exercising.
Alongside that, I’m expanding the exercise library to cover more movements—from yoga and HIIT to advanced weightlifting—so the AI can coach an even broader range of workouts with precision.
Thanks for reading! I hope this inspires you to build something amazing with AI.