Recently we faced an issue while executing user_data on my EC2 instance. This instance was created using Ubuntu 20.04 AMI. The user_data failed to execute the commands without any specific error message.
We tried different solutions available online like below,
- Added / Removed #!/usr/bin/bash -x
- Added / Removed #!/usr/bin/bash -v -e
- Changed the interpreter of bash from #!/usr/bin/bash to #!/bin/bash
None of the above fixes worked for us, so we troubleshot it further by analyzing the user_data logs on the server.
The logs of user_data execution on an Ubuntu server can be found at, /var/log/cloud-init-output.log file.
The error recorded in the file /var/log/cloud-init-output.log was as below,
Cloud-init v. 22.2-0ubuntu1~22.04.3 running 'modules:final' at Fri, 30 Sep 2022 04:56:41 +0000. Up 24.86 seconds.
2022-09-30 04:56:41,635 - cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
2022-09-30 04:56:41,635 - util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_scripts_user.py'>) failed
Cloud-init v. 22.2-0ubuntu1~22.04.3 finished at Fri, 30 Sep 2022 04:56:41 +0000. Datasource DataSourceEc2Local. Up 25.26 seconds
Based on the errors recorded in the cloud-init-output.log file, we inspected the user_data recorded on the server. The user_data details on the host server can be located at,
- The cloud-config file is located at /var/lib/cloud/instance/cloud-config.txt
- The user data can be found at, /var/lib/cloud/instance/scripts/
- On certain instances, you may see symlinks with the instance id at the above script location. The user data are stored in files name part_001 etc.
The inspection of user data captured on the host server showed spaces on the first line near the bash interpreter #!/bin/bash
As we can see in the below image, these spaces were the culprits in our case.
To fix the issue, we removed those spaces, re-published the user_data, and launched a new instance. The new instance executed the user_data without any failures.
The new user_data captured/recorded on the newly launched EC2 Instance does not have any spaces as we can see in the below image,
