Today we had to upgrade Devise on the Government's Enterprise Finance Guarantee management application due to a recent security announcement.
On running the usual
bundle update to grab the latest version of Devise and then running the test suite, one of our request specs started failing.
F Failures: 1) User creation and first login Failure/Error: page.should have_content('Invalid username or password') expected there to be content "Invalid username or password" in "Enterprise Finance Guarantee\n\n \n \n \n Enterprise Finance Guarantee\n\n \n \n \n\n \n\n Invalid email or password.\n\n \n Sign In\n\n\n\n Username\n Password\n\n \n\n Forgot your password?\n \n\n \n\n" # ./spec/requests/user_creation_and_first_login_spec.rb:38:in `block (2 levels) in
'Finished in 1.09 seconds 1 example, 1 failure Failed examples: rspec ./spec/requests/user_creation_and_first_login_spec.rb:11 # User creation and first login
How could I narrow down what had changed to cause the failure? If this were my own code I'd use git bisect, but we can also use it for gem dependencies using bundler's
path option. Here's how we solved the problem:
Checkout the Devise source code.
cd ~/Source git clone https://github.com/plataformatec/devise.git
Configure your application's Gemfile to use the local checkout of the gem.
gem 'devise', path: '~/Source/devise'
git bisect. If you're unfamiliar with this tool this article explains it really well.
cd devise git bisect start
Mark our previous version as good and the one we're upgrading to as bad.
git bisect good v2.1.0 git bisect bad v2.2.3
From the application directory run the
bundle and your test. Then from the devise directory mark it as either good or bad depending on whether the test passed or failed.
bundle && bundle exec rspec spec/requests/user_creation_and_first_login_spec.rb:11
Eventually, after a number of test runs, we get the result: This commit:
cde2229e598d0b91e75c039ead4311fa6a89a2e9 is the first bad commit commit cde2229e598d0b91e75c039ead4311fa6a89a2e9 Author: Gabe Martin-Dempesy
Date: Sun Nov 18 15:08:57 2012 -0800 Support alternate sign in error message when email record does not exist By default, the nonexistent error is still identical to the :invalid message, and must be customized by the developer to implement. :040000 040000 f3d61b1f4def429b532b251601b0c6b00b7f3b0b 20f98c3ef29dde4cbe03ba7389a5fbcf34a19642 M config :040000 040000 5c2448231976fbe98589192ce23b9c8afeb7be27 2db6b1091741eddf4d3b30158c0d689acb17a5de M lib
Once we knew the cause the fix was simple. Voila, passing tests.
. Finished in 1.43 seconds 1 example, 0 failures
A more thorough read of Devise's well documented CHANGELOG might have got the answer quicker, but this approach made the computer do the work!