This is a quick post, but wanted to get it out there!

Over the past week, I’ve been having to detangle some serious test flakiness. Below are a few findings that I wanted to put out there for other folks to hopefully find useful!

Sync up your randomizer seeds (especially for fake data generators) with the test randomizer seed

For example, I use Faker, which allows you to customize the seed it uses. Using the same seed that your test suite is randomized with makes it so that re-running a test with the same seed always generates the same data:

# test/minitest/faker_random_seed_plugin.rb
module Minitest
	def self.plugin_faker_random_seed_init(options)
		Faker::Config.random = options[:seed]
	end
end

Always print the number of parallel workers the suite is run with

Sometimes tests fail under certain randomization & parallelization combinations, such as a subset of tests failing for:

• Seed 1234
• 4 parallel workers

In order to help with test reproduction, it’s good to always print the number of parallel workers to replicate the failure

# test/test_helper.rb
puts "MINITEST_WORKER_COUNT: #{Minitest.parallel_executor.instance_variable_get(:@worker_count)}”
puts "PARALLEL_WORKERS: #{ENV["PARALLEL_WORKERS"]}”

Use deterministic data sources whenever possible

I know I just mentioned Faker above, but there are some data sources that are deterministic & should always be stable (like billing plans!) This is where Oaken shines, and why I use it. It gives you a hook to automatically create synced, deterministic data that is cross-environment, while still giving you the right balance of:

• Dynamic field values for data like names & emails
• A cohesive “story” for your data’s shape
• Cleanly defined flows for per-case datasets, like pagination rules