Adventures in creating a new AWS EB environment

May 21, 2014


We added queue classic to our Rails 4.1 + PostgreSQL application running on the AWS EB free tier. Then we hit problems. Later deployments failed (10 min time-out) yet the website continued to work okay.

The cause could have been the QC worker rake task, additional gems or maybe a combination of the two. So we tried restarting the app server => failed (10 min time-out). Then we tried rebuilding the whole EB environment => failed to remove a security group due to dependencies. On removing the dependency the rebuild wouldn’t resolve itself. Now the environment was stuck in a grey and unusable state.

So we created a new environment using the eb command line tool again. The AMI was a newer image [64bit Amazon Linux 2014.03 v1.0.2 running Ruby 1.9.3]. After creation we were unable to add the existing pg RDS instance - which we had also recreated because the earlier rebuild took the DB instance down with it. So we created a 3rd new DB instance from within the new EB environment; restored the DB from backup point and deployed the latest Rails app version to the new green EB environment.

On openingthe EB environment URL in a browser we got a semi familiar looking 500 html page. On inspecting EB log snapshots we saw two errors:

2014-05-20 23:48:53,779  directory-hooks-executor.log: rake assets:precompile => Gem::LoadError: You have already activated rake 10.3.2, but your Gemfile requires rake 10.2.2

and

2014-05-20 23:49:00.8120 passenger.log: “cannot load such file -- pg_ext (LoadError)”

The pg_ext error we have seen before, and installed the pg gem prerequisites:

SSH to the server:

sudo yum install postgresql postgresql-server postgresql-devel postgresql-contrib postgresql-docs postgresql-libs

However both issues persisted. Several hours of research later, the answer appears to be to include this file:

.ebextensions / ruby.config

Now we are back in business. Thanks to gkop for https://github.com/gkop/elastic-beanstalk-ruby

We didnt envisage having to go to these lengths to get this working ( and we are still touching the 10 min timeout on deploying new versions ) but it was probably a useful if painful experience.

© 2018 Keith P | Follow on Twitter | Git