Rails and Discourse Startup Times

I've spent a lot of time benchmarking how fast Discourse handles HTTP requests with various Ruby versions, to see how much new Ruby fixes help Rails speed. But I haven't looked yet at startup time, which can be very important for Rails apps. Specifically, I'm looking at time to handle first request. It's not the only definition of "startup time," but I think it's a very useful one.

Using a "real world" benchmark with Discourse, a production Rails app, makes for a few challenges. Specifically, Ruby, Rails and Discourse are all independently changing. It's not a synthetic benchmark app, it's a real app with a real user base and it only works with certain specific Ruby versions (and a single Rails version) at any given time.

There are only a few Discourse versions with compatibility from Ruby 2.0.X through 2.3.X. I'm using v1.5.0. Then we'll look at Discourse v1.8.0 (basically up-to-date) for Ruby 2.3.4 and 2.4.0, since current Discourse only supports very recent Ruby.

Older Ruby, Older Discourse

Using Discourse v1.5.0, we see nice clean numbers for startup time - very low variance, very consistent, and speeding up from older Ruby to newer Ruby. As always, feel free to request my JSON data for my results, and the benchmark code is all open.

Discourse startup times get better by increasing Ruby version, and by Discourse version.

Overall, Discourse 1.5.0 time-to-first request drops from 9.9 seconds with Ruby 2.0.0 to 7.7 seconds with Ruby 2.3.4. That's a 23.3% improvement. Then switching from Discourse 1.5.0 to 1.8.0 on Ruby 2.3.4 improves startup time to 6.5 seconds, or about a 15% speedup from Discourse improvements. And then Ruby 2.4.1 drops startup time to 5.85 seconds, or another 10% speedup.

So overall, Discourse delivers a 15% startup-time speedup from 1.5.0 to 1.8.0, and Ruby delivers a 30% startup-time speedup from 2.0.0 to 2.4.1. Not too shabby!

Future Goodness

None of this takes Bootsnap into account, which apparently drops Discourse startup time by half. Bootsnap is a complex beast, but mostly it works through a combination of caching filesystem checks in the require path and pre-parsing your Ruby code and caching the result.

Bootsnap is scheduled to be standard in every Rails app Gemfile for all new Rails apps, starting some time around Rails 5.0.

So for those following along at home, we expect the 30% Ruby-based startup-time improvement and the 15% Discourse-based startup improvement to be joined by a 50% startup-time improvement in Rails itself. Stay tuned.

Methodology and Picky Details

With only minor tweaks and improvements, this is the same benchmark I've been using for most of my Ruby benchmarking.

The tested Discourse v1.8.0beta13 AMI is public: "ami-f678218d".

The tested Discourse v1.5.0 AMI is also public: "ami-554a4543".

Discourse v1.8.0beta13 was chosen for a combination of Ruby compatibility and benchmark compatibility - there are some changes to Discourse which haven't yet been reflected in Rails Ruby Bench which currently prevent testing with the latest versions. I believe that will be fixed soon, but in the mean time I'm testing with v1.8.0beta13. I have no current reason to believe that this makes a significant difference in speed, or especially in the difference in speed between Ruby 2.3 and 2.4. Should I find problems in the methodology, you should expect them to be published on this blog.

"Time to first request" is measured by dispatching HTTP requests in a tight loop with a short sleep in between. There are a number of interesting variables which could be changed - startup currently is done with many workers and threads, for instance, and it could certainly be faster with fewer of them, though I think it would be less typical of real-world usage. RRB specifically and explicitly aims to be a "production, real-world" benchmark, and so startup time is measured with many threads and workers. The sleep in the loop will eventually be a problem for precision if Rails startup time improves by around 3x-4x. Should that happen, I'll decrease the duration of the sleep or remove it. For now, the sleep is short enough in duration not to cause a precision problem, but long enough to allow the Rails app to handle end-of-request work seamlessly and keep latency low. You can see the measurement logic in start.rb.

Rails' startup time for a large application like Discourse with complex startup logic is not necessarily the same as for a "vanilla" freshly-created Rails app with minimal startup complexity. Indeed, I have seen specific cases where an optimization will improve the "vanilla" startup time while making Discourse startup time worse. My intent is to measure both of those separately so that we can see the impact by Rails version on both. This article only covers the Discourse/complex/production case, not the plain-Rails/vanilla/simple case.