Need help increasing root volume size on the production instance

Technical Support
, , , ,
1

Hello all,
I recently encountered an issue after increasing the volume size on my production instance, where all the containers are currently down. I am seeking assistance to increase volume and ensure their proper functioning. The root volume on my production instance is running out of space, causing system instability and impacting my application’s functionality and the other external systems that depend on it. I need to increase the size of the root volume to accommodate the growing storage requirements.

Here are the steps I have already taken in an attempt to recover the containers:

  1. I have already tried using the “resize2fs” command to resize the file system on the existing volume, but it did not allow me to expand the partition due to its current size limitations.
  2. Executed the “docker ps” command to check the status of the containers and confirmed that they are all in a stopped state.

Relevant Information:
• The current size of the root volume is 30GB.
• The file system type is ext4.
• I have SSH access to the instance and can execute commands as root.
• I have already taken a backup of the important data on the instance.

Troubleshooting Efforts:
I have attempted to restart the containers using the “docker restart $(docker ps -q)” command, but they do not start successfully. There may be some underlying issue related to the increased volume size that is affecting the container startup.

Environment Details:

  • Operating System: Ubuntu 20.04
  • Docker version: 20.10.7
  • AWS Instance Type: t2.medium
  • Volume Type: gp2
  • Filesystem Type: ext4

Specific Questions:
1. What are the alternative approaches I can take to increase the size of the root volume?
2. Is it possible to resize the root volume without detaching it from the instance?
3. Are there any risks or precautions I should consider before attempting to resize the volume?

Any assistance or guidance you can provide to help me resolve this issue would be greatly appreciated. Thank you in advance for your time and expertise.

2

Hi @gk0harris ! Thanks for all the details in your issue.

  1. What are the alternative approaches I can take to increase the size of the root volume?

Did you modify the volume size from AWS Console or aws cli?
We’ll want to modify the volume size, once that is complete, you can ssh into the box and run:

  1. Is it possible to resize the root volume without detaching it from the instance?

It is possible, but sometimes you are unable to run any commands due to root disk being full (even resize2fs won’t work, etc), as well as if you selected an older generation ec2 instance. If that’s the case, you want to stop the ec2 instance, detach the root drive and attach that root disk to a brand new ec2 as the secondary storage drive. You will then be able to run growpart, resize2fs and delete files for space.

  1. Are there any risks or precautions I should consider before attempting to resize the volume?

Nope, as long as you have a backup, your process is correct. Please make sure you are also backing up the data drive (if that is different than the root disk)

2 Likes
3

Thank you, @hareet , for your helpful suggestion! Following your advice, I resized the root volume using the ‘growpart’ and ‘resize2fs’ commands at the optimal time during volume optimization. It worked perfectly, and I now have restored functionality across all systems. Your guidance was much appreciated!

1 Like