Hackathons are the perfect playground for experimentation - three days of chaos, coffee, and the kind of coding that leaves your brain buzzing.
My latest adventure was diving headfirst into Rust bindings for Python and Node.JS.
My goal was simple: build a Rust-based wrapper for S3 operations and see if it could outperform the usual suspects like Boto3 or AWS SDK for JavaScript in a AWS Lambda Function.
It wasn't about proving it could be done - that’s already clear - it was about exploring what it's like to actually build it.
It was also less about Rust’s raw speed and more about understanding its strengths and limits.
Spoiler alert: the journey was anything but straightforward.
Rust 101: The Steep Climb Begins
Rust is like a grumpy teacher - it's strict, but it makes you better. I've always admired its reputation for speed and memory safety, so I was excited to dive in. But the learning curve? Let's just say I didn't walk away unscathed. Writing Python bindings using PyO3 was thankfully well-documented, but there were plenty of “aha” and “oh no” moments as I got to grips with Rust's ownership model and its infamous borrow checker.
Initially, I questioned why I'd signed up for this level of complexity. After all, boto3 works just fine for most cases, right? But something about Rust's promise of efficiency and control kept me going. That, and the idea of doing something new.
It also taught me a thing or two about debugging and design. The need to manually propagate and handle them effectively forced me to think carefully about how my code communicates across two languages.
Performance Reality Check
With my first Rust wrapper for S3 up and running (ListObjects, Put, Delete), I ran some performance tests, eager to see Rust blow Boto3 out of the water. What I saw instead was... disappointing. Boto3 was faster and aioboto fas way fast (obviously).
Left with a mix of frustration and curiosity, I dug into the issue and quickly realized my mistake: I hadn't accounted for connection pooling (and parallelism at this point). Once I added connection pooling and asyncio support to my wrapper, Rust finally started to shine. The results? Faster than Boto3 and aioboto. Am I the best at tweaking Boto3? No, so better spike your use-case yourself that before you go all in.
The lesson here? Performance isn't just about the language - it's about how you use it. Rust gave me the tools, but I had to learn how to use them. And it also depends on you access patterns.
The Serverless Sweet Spot
Here's where I wanted to use it: serverless computing. By compiling Rust for AWS Lambda's target architecture, I hoped for a lightweight, dependency-free package that would fit seamlessly with the function. No bloated SDKs, no unnecessary dependencies - just clean, optimized code. For performance-critical tasks like high-volume S3 operations.
After fumbling with building and packaging everything, I deployed my first function and ran the same tests inside my new lambda function.
It just worked! How awesome is that!
After some massaging, everything was fitting in a normal serverless-first project, like we use it at Intenics. Testing, packaging, deployment and automation, it all fits well together in a manageable way. So no frankenstein monorepo madness.
Besides the initial hype: would I recommend Rust bindings for every use case? Probably not. There is still some complexity added to the project and the bus factor for rust developers may be too low for your project. But if milliseconds matter or your Lambda size needs to stay lean, Rust could give you an edge where it counts.
Where are the numbers?
Don't worry - I'm not here to share my messy hackathon code, but i have some timings for my test setup:
Create 300 objects with the content "hello world".
| test | time in seconds |
|---|---|
| boto3 put_object | 16,94 |
| aioboto3 put_object | 2.44 |
| rust-wrapper with connection pooling and synchronous call | 10.00 |
| rust-wrapper with connection pooling and asyncio.gather | 1.02 |
For clean performance analysis and deeper insights, check out Joshua Robinson's post, which dives into Python S3 performance enhancements using Rust. He has done an awesome job, go check it out!
My journey was more about experimenting with possibilities and brainstorming future use cases.
Beyond S3 operations, Rust in Lambdas opens doors to exciting possibilities, such as:
- Fast Lambda Extensions for Circuit Breakers
- OpenTelemetry Tracing (extensions?)
- Caching Layers
- ...
What About Node.js Bindings?
While PyO3 made working with Python bindings relatively smooth, exploring Node.js bindings with Rust brought its own set of lessons. Using libraries like neon or napi-rs I played around building Node.js wrappers for the Rust-based S3 SDK. Node's async nature pairs well with Rust, but the process wasn't without challenges. Interfacing between the two languages involved careful attention to async/await compatibility and error propagation.
The result? It can also be an edge case alternative for the AWS SDK for JavaScript. Rust bindings in Node.js can unlock performance gains while keeping the convenience of Typescript.
Final Thoughts
Rust isn't the easiest tool, but its potential is undeniable. This hackathon showed me that - while tools like Boto3 or AWS SDK for JS work well for most cases - Rust bindings can indeed increase performance with reasonable added complexity to the project. Whether it's faster S3 interactions or advanced Lambda extensions, there's a lot more to explore.
And it fits well with our serverless-first approach.
Rust is worth the ride. Just bring a lot of patience, some band-aid, and coffee. 🚀
