How to: Recover EC2 Instance Failing To Start

  1. Create a new machine with the same instance family you want to recover.
  2. Before starting the new machine, detach the root volume from the machine you want to recover and attach it to the new machine. Rename it in case you have the /dev/sda1 in its Name tag (in AWS Console) to make sure AWS won’t attach it as a root volume to the new machine as well.
  3. Start the new machine
  4. If it is not started (due to the same root volume conflict probably), try to snapshot the old root volume, then create a new volume out of it, and then attach the new one. Sometimes volumes get messed up and the snapshot trick will fix it.
  5. lsblk to see the name of the attached volume
  6. mkdir /mnt/recovered
  7. mount -o nouuid /dev/nvme1n1p1 /mnt/recovered
    The -o here is to overcome the issue (mount: unknown filesystem type '(null)') of root volumes having the same UUID
  8. cd /mnt/recovered/etc
  9. vi fstab
  10. Add ,nofail after the defaults . This will allow the old instance to start and for you to do the right thing and change your /dev/nvme2n1 with a proper UUID to prevent it to change after a restart (to get the UUID use blkid).
  11. Shutdown the new machine. Detach the recovered volume and attach it back to the original machine (make sure to name it sda1 when prompted to be able to start it, otherwise you’ll get the “no root volume found” error). Start it. Done. You’ve saved the day :)
  12. Remember to terminate the recovery machine $$



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Regev Golan

Regev Golan

R&D Group Manager @ Check Point | Software Engineer