Run Docker-Selenium as a Kamal Accessory

The task:

To scrape a javascript-heavy web page you will need a web browser to load the dynamic page into. To automate the browser, you will need something like Puppeteer or Selenium.

We opted for the selenium-webdriver gem with Google-chrome running in headless mode.

It was useful to install the chrome dev package locally for developing and testing. However running this way in production wont cut it. The Rails App server will grind to a halt after some hours when it eventually runs out of memory.

The solution:

Connect selenium-webdriver in your Rails app to a separate ‘remote’ server running Selenium Grid + Chrome (or Firefox).

Brilliantly, our friends at docker-selenium publish these ready-to-use images to the Docker Hub registry. From this collection we selected a ‘Selenium Grid in Standalone mode with Chrome’ image.

We deploy this image as a Kamal accessory which are long-lived services that the app depends on. We already use Kamal to deploy the Rails app, so all that was needed was to add this accessory to our deploy.yml file. In just 6 lines, you can specify the image, host and port, and Kamal does the rest. (wow!):

## myrailsapp/config/deploy.yml:
service: myrailsapp
...
servers:
  web:
...
accessories:
  selenium:
    image: selenium/standalone-chrome:4.34.0-20250717
    host: xxx.xxx.xxx.x
    port: 4444
    options:
      shm-size: 2g
...

Boot the new accessory service on the host:

$ kamal accessory boot selenium

Now the selenium accessory is running in its own container on your app server, or wherever.

To connect to the accessory, the URL is in the form:

"http://{service name}-{accessory name}:4444/" # eg. http://myrailsapp-selenium:4444/

Finally back in our Rails app, the code that connects selenium-webdriver to the accessory service looks like this:

## myrailsapp/app/models/scraper.rb: 
...
grid_url = ENV["REMOTE_SELENIUM_URL"] # eg. http://myrailsapp-selenium:4444/
options = Selenium::WebDriver::Chrome::Options.new(args: [ "--headless", "--no-sandbox", "--disable-setuid-sandbox" ])
driver = Selenium::WebDriver.for :remote, url: grid_url, options: options
...

That is all there is to it. The task is achieved in these few simple steps.