This is a submission for the DEV's Worldwide Show and Tell Challenge Presented by Mux
What I Built
I built pardoX, a high-performance ETL engine written in Rust designed specifically for the "Mid Data" gap. While modern tools excel at gigabytes in the cloud, they often choke on local hardware. PardoX allows a standard corporate laptop (like an i5 with 16GB RAM) to process massive datasets (640M+ rows) with the efficiency of a high-end server, converting raw CSVs to optimized Parquet files at speeds exceeding 80 MB/s.
My Pitch Video
Demo
...This is a submission for the DEV's Worldwide Show and Tell Challenge Presented by Mux
What I Built
I built pardoX, a high-performance ETL engine written in Rust designed specifically for the "Mid Data" gap. While modern tools excel at gigabytes in the cloud, they often choke on local hardware. PardoX allows a standard corporate laptop (like an i5 with 16GB RAM) to process massive datasets (640M+ rows) with the efficiency of a high-end server, converting raw CSVs to optimized Parquet files at speeds exceeding 80 MB/s.
My Pitch Video
Demo
PardoX is currently in Private Beta as we refine the final engine.
Beta Launch: January 19, 2026.
Project Updates & Benchmarks: Read the full story on Medium.
Early Access: If you are a judge or a developer dealing with "Memory Errors" on your local machine, please contact me at iam@albertocardenas.com for a pre-release binary and testing instructions.
The Story Behind It
I built this because of the "Forgotten Sector." I was tired of being at the office at 7 PM, watching my laptop freeze while trying to load a 3GB CSV into Pandas. Most engineers don't have $5,000 workstations; we have corporate laptops. I wanted to build a tool that respects the user's time and hardware. PardoX is my "love letter" to data engineers working in the trenches who need industrial-scale power without the cloud's price tag or the JVM's overhead.
Technical Highlights
- Rust-Powered: Leverages Rust's memory safety and performance without a Garbage Collector.
- Zero-Copy Architecture: Data is streamed and processed with minimal allocations, preventing OOM (Out of Memory) errors.
- Hardware-Awareness: The engine "reads" your CPU and RAM to surgically allocate thread pools for reading and compression, preventing system contention.
- Smart Sharding: Automatically fragments massive outputs to prevent OS file-system choking, ensuring a "Zero Friction" data flow.
- Performance: In recent tests, PardoX processed 640 million rows in 206 seconds, outperforming both DuckDB and Polars on the same local hardware.
Use of Mux (Additional Prize Category Participants Only)
I utilized Mux to host and stream the pitch video for this challenge. The integration was seamless, providing a high-performance video delivery that matches the "speed and efficiency" philosophy of my own project.