Thomas Cannon

This gives me "checking the blinking lights on my Linksys router back when they still had the blue front face" nostalgia, thank you

I'm trying my best to make systems understandable. Years ago I read about how macOS (and software writ large) got rid of status bars, syncing/activity windows, and it stuck with me. So for Little CRM, I’m adding status lights to show if JS appears to be working, and if the Web Sockets connection was successful.

tl;dr: The best combination of Hetzner's ARM servers and Crunchy Bridge's regions are:

• Hetzner: nbg1-dc3
• Crunchy Bridge:  AWS eu-central-1
• Benchmarks at the end of the post

I’ve been working on getting the first Practical Computer app out of beta (a dispatcher app for service companies). It, and the rest of our smaller apps, have the following constraints:

• Extremely shoestring budget. $30/month operating budget. Not much to work with!
• Since it’s production data, there’s responsibility for the data to be redundant, safe, and secure.

Given those, I ended up choosing Hetzner’s ARM servers, given the sheer power + cost combination (even with them being in the EU; and this is a US-based project).

I’ve had great experiences with the Crunchy Bridge team, and 18+ years of experience have taught me that I really want someone else to make sure the database is secure for me. So even though it’s 10% of my operating budget here, it’s worth it (because the database is the most crucial part of the app!).

The problem is that the app server & database are on two separate networks; I don’t get the benefit of an internal network, so I had to benchmark what combination of regions would work best.

The previous iteration was completely misconfigured: Hetzner’s Helsinki region for the app, AWS eu-north–1 for Crunchy Bridge. This turned out to be awful for performance. After some benchmarking and experimenting, I found that the following combination gets pretty dang close to good performance given the constraints:

• Hetzner: nbg1-dc3
• Crunchy Bridge: AWS eu-central–1

(I’ve got benchmarks at the end of the blog post, don’t worry!)

Lessons learned

I wanted to write this not only to help other folks find the best combination easily; but to talk about the mistakes I made and lessons I learned.

• Mistake: I should have benchmarked different options & done more research. My first Crunchy Bridge cluster was actually in eu-central–1 , with the Hetzner server in Helsinki; and if I’d benchmarked & explored my options I would have avoided an unnecessary database migration.
• Win: When spooling up the Hetzner server, I did my best to use Ansible to provision the server, and use Dokku for a PaaS-like experience. This made it so that spooling up a new server in nbg1-dc3 was significantly easier. This is why I advocate for “automate as you go”, especially for infrastructure.
• Win: choosing the right vendors. Hetzner allows me to quickly & cheaply spool up a small project, and having Crunchy Bridge’s support & their tooling made spooling up the replicas & promoting them dead-simple.

Alternative solves

To be clear, I had a few other options available to me, some of which I could still choose, and have implemented:

• Application-level caching: This is done, and basically a given for any production app. I ended up using Redis here.
• Making a read replica on the Hetzner instance; use that for reads (a heroically misguided, but technically feasible idea!)
• Move from Hetzner to AWS. This would only make sense for an app that’s actually generating the profit to justify the switch. It’s hard to bear Hetzner’s price & power combo here.
• Move the primary DB back to Hetzner (not ideal. Again, 18+ years of experience have taught me that I don’t want to manage a bunch of databases)

Benchmarks, as promised

# Benchmark script
# x.report("crunchy-eu-central") { ApplicationRecord.connected_to(role: :crunchy-eu-central){ Organization.first.onsites }}
# x.report("crunchy-eu-north-1") { ApplicationRecord.connected_to(role: :crunchy-eu-north-1){ Organization.first.onsites } }
hetzner-helsinki:~$ dokku run dispatcher
rails@aaaaaa:/rails$ ruby script/benchmarks/database_performance.rb
…
ruby 3.3.1 (2024-04-23 revision c56cd86388) +YJIT [aarch64-linux]
Warming up ————————————–
crunchy-eu-central     1.000 i/100ms
crunchy-eu-north-1     1.000 i/100ms
Calculating ————————————-
crunchy-eu-central      9.788 (±10.2%) i/s -     49.000 in   5.031111s
crunchy-eu-north-1     14.124 (± 7.1%) i/s -     71.000 in   5.045164s
Comparison:
crunchy-eu-north-1:       14.1 i/s
crunchy-eu-central:        9.8 i/s - 1.44x  slower
====================================================================================================================
hetzner-nuremberg:~$ dokku run dispatcher
rails@5e9d1df43591:/rails$ ruby script/benchmarks/database_performance.rb
…
ruby 3.3.1 (2024-04-23 revision c56cd86388) +YJIT [aarch64-linux]
Warming up ————————————–
crunchy-eu-central     1.000 i/100ms
crunchy-eu-north-1     7.000 i/100ms
Calculating ————————————-
crunchy-eu-central     20.452 (± 0.0%) i/s -    103.000 in   5.037650s
crunchy-eu-north-1     96.340 (± 2.1%) i/s -    483.000 in   5.016063s
Comparison:
crunchy-eu-north-1:       96.3 i/s
crunchy-eu-central:       20.5 i/s - 4.71x  slower
====================================================================================================================
/// Spot-check of query latency
hetzner-helsinki:~$ dokku run dispatcher
rails@aaaaaa:/rails$ bin/rails c
Loading production environment (Rails 7.1.3.2)
irb(main):001> 10.times { p Benchmark.ms{ ApplicationRecord.connected_to(role: :crunchy-eu-central){ Organization.first.onsites } } }
1637.3784099705517
103.8463581353426
106.55084392055869
105.82964192144573
102.70471591502428
102.40731597878039
107.70620591938496
112.95577604323626
111.48629407398403
105.25572090409696
=> 10
irb(main):002> 10.times { p Benchmark.ms{ ApplicationRecord.connected_to(role: :crunchy-eu-north-1){ Organization.first.onsites } } }
1040.9399520140141
72.33657804317772
73.68614105507731
77.82750786282122
75.32874401658773
73.2035799883306
77.43134815245867
72.01353716664016
71.68629695661366
74.11726214922965
=> 10
====================================================================================================================
hetzner-nuremberg:~$ dokku run dispatcher
rails@aaaaaa:/rails$ bin/rails c
Loading production environment (Rails 7.1.3.2)
irb(main):001> 10.times { p Benchmark.ms{ ApplicationRecord.connected_to(role: :crunchy-eu-central){ Organization.first.onsites } } }
846.7416610001237
54.84945299758692
56.269697000971064
55.53825500101084
54.511651000211714
54.32877099883626
57.52018099883571
59.374027001467766
63.36915899737505
55.06629299998167
=> 10
irb(main):002> 10.times { p Benchmark.ms{ ApplicationRecord.connected_to(role: :crunchy-eu-north-1){ Organization.first.onsites } } }
168.30048900010297
11.614236002060352
12.266477999219205
15.882210002018837
15.326369000831619
11.819797000498511
14.219884997146437
10.650393000105396
11.742956001398852
13.63104199845111
=> 10

This post has been sitting in my drafts for months, so I'm shoving it out into the world!

Garrett Dimon and I have been chatting back & forth about Rails configurations. This is an extension/riff on his idea for Unified Configuration in Rails

While Garrett's approach hooks nicely into Rails and has multiple metaprogramming niceties; my approach is generalized & direct (but verbose). Essentially: create a Configuration class that is a PORO; I named mine AppSettings. You can see a gist of it here!

This is born out of a convention in my iOS apps, which use a Swift struct called Configuration, that pulls the configuration from various sources into a strongly-typed, consistent location that is easy to remember.

My primary focus is to explicitly declare the configuration values and standardize how they’re accessed with a straightforward API, rather than try to standardize how they’re stored. This is because there are so many cases where either:

• “It depends” on how they should be stored.
• There’s an external dependency that makes you use a particular storage mechanism.
• The configuration is so entrenched as part of the environment that it ends up in the environment’s configuration.

Using a PORO with clearly defined methods gives you:

• Clarity in how the value is retrieved.
• The flexibility for different value types. Some are single keys, some are nested objects, etc.
• The same API for dynamically generated values; such as subdomained URIs
• An easy object to mock out for tests as needed.

I've been using this approach for an internal app I've been commissioned to make; it's worked out very well so far! I'd definitely recommend moving towards this approach in your projects.

Inspired by Lucian’s post, I finally setup code quality for the first Practical Computer app.

This whole process was definitely borderline “do it right, or do it twice.” I wish I’d solved this a bit sooner. I knew it was necessary, but had kept pushing it back because the app isn’t even close to being live yet. But this line from Lucian’s post changed my opinion:

Side projects developed while having a full-time job have a unique characteristic worth noting. The time dedicated to working on the side project is not continuous. For instance, you may work on it for 1-2 hours on Saturday, and the next opportunity to work on it may only arise a week later.

It is then essential to make the code quality built-in and use as much automation as possible.

That’s a very strong argument, and one I hadn’t heard yet. Of course, since I delayed, it caused the past 3 work sessions to be solely about fixing up the repo. But the upside is now I have all the code ready for the next project. And speaking of that, here’s a gist of my customized set of Rubocop & CircleCI configuration. I hope it helps!

I made a few technical differences than Lucian:

• I trimmed down the Rubocops used. This gives me a balance of expressiveness & the benefits of a linter
• I chose CircleCI because it’s what I’ve been using for years and it’s Fine™️. Plus it has the distinct advantage of SSH access to debug jobs, and job reporters
• I’m using Bun, so relying on Dependabot for my JS dependencies
• I’m using Code Climate for maintainability monitoring

Posted a quick update about the Ruby Passkeys organization in the passkeys issue for Devise

The gist is:

• There are very early alpha implementations for Devise and Warden, including a template repo
• Once again, I am asking for desperately needed maintainers for this

Getting puma to use self-signed certificates in Test

Install the localhost gem, which allows Puma to use self-signed certificates in your test environment: https://github.com/puma/puma/#self-signed-ssl-certificates-via-the-localhost-gem-for-development-use

group(:test) do
  gem 'localhost'
end

It’s not documented anywhere in Capybara, but Capybara’s built-in Puma handler allows you to issue a custom bind : https://github.com/teamcapybara/capybara/pull/2028

Capybara.run_server = true

Capybara.server = :puma, { Host: "ssl://#{Capybara.server_host}"}

I found that you do need to explicitly pass the Capybara.run_server = true argument in your system tests

You also need to have Selenium Webdriver explicitly accept insecure hosts: https://developer.mozilla.org/en-US/docs/Web/WebDriver/Capabilities/acceptInsecureCerts

For example:

Capybara.register_driver :chrome do |app|
  options = Selenium::WebDriver::Chrome::Options.new
  options.accept_insecure_certs = true
  Capybara::Selenium::Driver.new(app, browser: :chrome, options: options)
end

With that, Capybara runs with self-signed certificates in development, and the WebDriver will accept them.

Running subdomained tests

Because of how macOS does not handle wildcard DNS resolution for localhost by default, if you need to run your tests using subdomains, you’ll need to use a loopback DNS service like nip.io
For example:

Capybara.app_host = "https://myapp.127.0.0.1.nip.io"

Depending on how your existing tests are setup, you might also need to setup a custom initializer block in config/environments/test.rb to override the tld_length

#config/environments/test.rb
Rails.application.configure do
  #...
config.hosts << “.127.0.0.1.nip.io”
if ENV[“PREP_SYSTEM_TESTS”].present?
config.action_dispatch.tld_length = 5 # 127.0.0.1.nip.io = 5 top-level host details to throw out
Rails.application.routes.default_url_options[:host] = “127.0.0.1.nip.io” # Necessary for routes to play nicely
# …
end
end

Working on a brand-new feature for Noko, and it’s progressing incredibly quickly. I can’t wait for folks to get their hands on it.

Related: Maintaining software is a really tricky thing, but pays off. The theme of this year has to be behind-the-scenes rewrites; we’ve touched a lot of Noko’s core internals, because they needed the maintenance desperately. Since we approached it strategically and tried to double-dip as much as possible; this new feature is coming together pretty seamlessly.

14+ years of a product being around means that stuff needs fixed! Frameworks change, heck, the execution environment is wildly different now with browsers. The best thing you can do for yourself + your team is to build with maintainability in mind as much as possible. Which is different than make everything generic, but that’s a whole other can of worms.