From SSH to REST: A Security-Driven Modernization of Slack's EMR Data Pipelines (15 minute read) (opens in new tab)

Slack modernized its data pipelines by migrating over 700 SSH-based operators on AWS EMR to a secure REST-based architecture with zero downtime across 8 regions. Its team replaced direct SSH access with Quarry, their internal REST job submission gateway, and used YARN's Distributed Shell to run arbitrary commands for proper resource management, reliable tracking, clean cancellation, and server-side lifecycle handling.