How Much Faster is Ruby 2.5.0 Preview 1?
/Ruby 2.5 is coming! Preview 1 was released. There are a bunch of new features. I'm looking forward to delete_prefix and delete_suffix, myself. There are more articles coming.
And of course, as always, there are performance improvements.
I spend a lot of time benchmarking Ruby. I'm here answering the question, "but how much faster does this make my Rails app?" Clearly it's time for some Ruby 2.5 benchmarking.
What Are We Measuring?
My benchmark Rails Ruby Bench sets up Discourse under a pretty heavy concurrent load of user requests. It determines how fast it can handle them all as it saturates a large, dedicated EC2 instance with requests that need to be handled by Rails (e.g. no static assets.)
Ruby 2.3 and 2.4 were very similar in Rails performance. Ruby 2.5 is very similar to 2.4.1. So when you look at the graphs below, you'll likely have to squint. Also, as always, feel free to ask me for my JSON data from the test runs, and the code is open-source. Very soon it should be automatically running on RubyBench.org, too.
Those bars on the right are all very slightly smaller. That's obvious at first glance, right?
Ruby 2.5.0 is just slightly faster at every request percentile shown above and nearly every percentile, but only very slightly faster. The 100th percentile is literally a single request which, in my tests, just happened to be 4% slower than the equivalent for Ruby 2.4.1 - you can probably ignore it as an outlier.
Here are the same numbers as a table, to three significant digits:
| Percentile | Ruby 2.4.1 | Ruby 2.5.0 | % Faster |
|---|---|---|---|
| 0% | 0.00592 | 0.00572 | 3.5% |
| 1% | 0.0124 | 0.121 | 2.5% |
| 5% | 0.0205 | 0.0198 | 3.5% |
| 10% | 0.0279 | 0.0270 | 3.3% |
| 50% | 0.140 | 0.135 | 3.7% |
| 90% | 0.377 | 0.372 | 1.3% |
| 95% | 0.440 | 0.436 | 0.9% |
| 99% | 0.578 | 0.569 | 1.6% |
| 100% | 1.17 | 1.22 | -4% |
And now for some numbers that are too small to really see on graphs... Ruby 2.5.0 has about 1.5% higher throughput overall. That makes sense - a throughput is effectively a mean, and means are easily dominated by a few larger entries, like the higher-percentile table rows above. So you see a throughput that is faster by about the same amount as the 90th percentile, not similar to the median request.
I've run enough trials that I'm convinced it holds up and isn't just statistical noise, but... Yeah. It's very, very similar in speed.
Conclusions
As we move toward Ruby 3x3, it's important to keep watching Ruby's overall speed, and speed specifically when running Rails. Overall, Ruby 2.4.1 is about 150% faster than Ruby 2.0.0 (slides). Not too shabby! But it's not 300% yet, either.
Ruby 2.5.0 preview 1 is another 3% faster on top of the 150%, which helps - they multiply, so you're seeing more like another 4.5% speedup based on the Ruby 2.0.0 baseline. But it's clear that Ruby has squeezed out a lot of the performance gains it can easily get - we're starting to see diminishing returns. Getting another 50% faster is going to be difficult this way, let alone getting to 300%. For that, we're going to need MJIT (Just-In-Time compilation for Ruby) or something like it.
PostScript, added on Dec 4th: it appears that head-of-master Ruby has added another change after preview 1, which adds around 6% speed. So Ruby 2.5 will have around three times the speedup shown in this post. We'll look at that in another post soon. That's around a 10% speedup over Ruby 2.4.1. Not bad at all, but I stand by my conclusion -- it'll take a lot of those to get to 300%. The Ruby 2.5.0 speedup will then be from around 150% of Ruby 2.0.0's speed to around 165% of it. See the future article for more details.