To scrape a javascript-heavy web page you will need a web browser to load the dynamic page into. To automate the browser, you will need something like Puppeteer or Selenium. We opted for selenium-webdriver with Google-chrome running in headless mode. It was useful to install the chrome dev package locally for developing and testing. However running this way in production didn’t work, the Rails App server will grind to a halt after some hours when it eventually runs out of memory. ... Read more

How we successfully used Kamal to leave AWS Beanstalk and deploy anywhere. We were new to containers, so this was part of the learning curve as well. In Part 1 We simply wanted to replicate the example by DHH and use Kamal to deploy to a single server on Hetzner. Includes some errors we overcame. In Part 2 We added the pieces in preparation for deploying the product. The Gotchas are some errors we saw and how to fix them. ... Read more