We switched simulator streaming to H.264 and it felt worse. Here's how we fixed the latency. (opens in new tab)

In an earlier post I described how tapflow streams iOS simulators to the browser: pull frames off the simulator's IOSurface, JPEG-encode them on the Mac, push them over WebSocket at ~30fps. JPEG has one great property for interactive streaming: every frame is independent and decodes instantly. There's no buffer, no inter-frame dependency. On localhost it feels like you're touching the simulator directly. It also has one terrible property: size. A full-frame JPEG of a scrolling screen is ~590K...