We added queue classic to our Rails 4.1 + PostgreSQL application running on the AWS EB free tier. Then we hit problems. Later deployments failed (10 min time-out) yet the website continued to work okay.
The cause could have been the QC worker rake task, additional gems or maybe a combination of the two. So we tried restarting the app server => failed (10 min time-out). Then we tried rebuilding the whole EB environment => failed to remove a security group due to dependencies. On removing the dependency the rebuild wouldn’t resolve itself. Now the environment was stuck in a grey and unusable state.
So we created a new environment using the eb command line tool again. The AMI was a newer image [64bit Amazon Linux 2014.03 v1.0.2 running Ruby 1.9.3]. After creation we were unable to add the existing pg RDS instance - which we had also recreated because the earlier rebuild took the DB instance down with it. So we created a 3rd new DB instance from within the new EB environment; restored the DB from backup point and deployed the latest Rails app version to the new green EB environment.
On openingthe EB environment URL in a browser we got a semi familiar looking 500 html page. On inspecting EB log snapshots we saw two errors:
2014-05-20 23:48:53,779 directory-hooks-executor.log: rake assets:precompile => Gem::LoadError: You have already activated rake 10.3.2, but your Gemfile requires rake 10.2.2
and
2014-05-20 23:49:00.8120 passenger.log: “cannot load such file -- pg_ext (LoadError)”
The pg_ext error we have seen before, and installed the pg gem prerequisites:
SSH to the server:
sudo yum install postgresql postgresql-server postgresql-devel postgresql-contrib postgresql-docs postgresql-libs
However both issues persisted. Several hours of research later, the answer appears to be to include this file:
.ebextensions / ruby.config
Now we are back in business. Thanks to gkop for https://github.com/gkop/elastic-beanstalk-ruby
We didnt envisage having to go to these lengths to get this working ( and we are still touching the 10 min timeout on deploying new versions ) but it was probably a useful if painful experience.