Page tree
Skip to end of metadata
Go to start of metadata

As hard as we try, sometimes we'll push out a change that breaks things. The three main apps, deployed with mina, work by copying a complete codebase to a numbered release directory and creating a symlink to the current release. This means that old releases can be made current again by changing the symlink. This page tells you how to do that.

1 - Turn off automatic deploys

All three apps automatically deploy to staging and production/training every night (in case of server reboots and to manage memory usage). You'll want to turn these off so that the broken version isn't redeployed while you fix the bug.

  1. Open the gitlab repo for the project in question
  2. From the left-hand menu, select CI/CD → Schedules
  3. For affected system, open the appropriate schedule.
  4. Untick "Active", and save. The summary page should show "Inactive" for the next run time.

2 - Log on to the server and become the app user

  1. Connect to the server from your development machine, using ssh digilin-production or ssh digilin-staging as appropriate
  2. Switch user to the affected app, using (for instance) sudo su cms-editor and providing your password
  3. Move to the app's home directory, using cd ~

3 - Find a known-good release

Because of nightly deploys, if the bug has been shipped for more than a day, the most recent previous release may also be bugged.

  1. By date or by code, determine some way to identify a known-good release.
  2. List the available releases on the server, using ls -l releases
  3. Find a known-good release from that list, and note its number
    1. If you can't, this process won't work for you. You'll need to revert the code through gitlab and deploy those changes.

4 - Stop the running app

Make sure you're set up to use mina to control the app on the target server, from your development VM. If necessary, follow the instructions at Controlling Rails applications via mina commands first.

  1. Log on to your development VM, using (for instance) ssh digital-dev-crh54
  2. On your VM, move to the broken app's code directory, using (for instance) cd /opt/www/apps/cms-editor
  3. Use mina to stop the app on the target server, using (for instance) bundle exec mina unicorn:stop to=digilin_production (note that the server name is underscored here)

5 - Switch the current release

The "current" release is simply a symlink from current to a numbered release directory. Changing releases is simply a matter of repointing it.

  1. Back on the target server, in the app's home directory, remove the current current link, using rm current
    1. Be very sure not to have a trailing slash. current is the link, and removing it just removes the indirection. current/ is the linked-to directory. It would be... detrimental to remove that.
  2. Now recreate the link, pointing to the release number you found in stage 3, using ln -s releases/123 current (for whatever release number is appropriate)
    1. Aside: no-one ever remembers which way round the arguments to ln go. Here's a trick - ln has a single-argument version ln -s /full/path/somewhere, which creates a link with the same name in the working directory. Thus, the thing you're pointing at must come first.

6 - Restart the old app

  1. On your VM, in the app's code directory, use mina to start the app, using (for instance) bundle exec mina unicorn:start to=digilin_production
  • No labels