LXD Cloud Init
Use cloud-init to build LXD/LXC containers
Over the last few months I've been reading and writing a lot about containers using podman on this site. I even went so far as to move this site onto the podman container platform, though I've recently de-containerized this site. Managing each container image was getting exhausting and in the long run I really didn't see the point in all the extra work, so I carefully backed out my container changes, and my workload on the web server has gone way down. I can hopefully focus more on writting now, rather than just constantly feeding the machine.
Even though I couldn't keep up with the overhead created by containers on this site, I do still occasionally use containers for a lot of things, mostly programing projects and scripting tests. However, I don't generally use podman or Docker – instead I usually end up building containers with LXD.
I like LXD containers for a lot of reasons. Primarily, I tend to think LXD is the easiest container platform to use and that is enough of a reason by itself for me to use it over other container platforms. They make it easy to spin up and manage containers from a multitude of distributions and you don't have to worry about complex volume mapping, or setting up new service unit files. LXD just takes care of it all, which means you can get to work just a little quicker. Or I can anyway – a lot of people prefer the application style containers, I'm just not one of them.
Here is how I rapidly build Ubuntu LXD containers using cloud-init when I need to create a clean environment to work on a new project.
Credit to Simos Xenitellis for doing a great write up on this a couple years ago https://blog.simos.info/how-to-preconfigure-lxd-containers-with-cloud-init/ I use quite a bit from his blog in this post while adding a few extra pieces that I felt were missing after quite a bit of trial and error on my part.
What is cloud-init?
According to the official cloud-init documentation site.
> Cloud-init is the industry standard multi-distribution method for cross-platform cloud instance initialization. It is supported across all major public cloud providers, provisioning systems for private cloud infrastructure, and bare-metal installations.
Cloud-init is a technology that allows you to define a desired initial state of whatever system you are building. In this context it's generally refering to cloud server instances, or bare-metal.
In this instance we'll see how it can be applied to LXD containers. All images (that I'm aware of) in the standard LXD repositories come with cloud-init out of the box. So you don't need to do anything special to get your images ready for cloud-init. Unless of you are making your own instances, in which case – why are you reading this?
What is LXD?
LXD is a next generation system container manager. It offers a user experience similar to virtual machines but using Linux containers instead.
https://linuxcontainers.org/lxd/introduction/
LXD is container platform that allows you to use Linux containers in a way that resembles a virtual machine. In my humble opinion LXD is the best of both worlds it has all the advantages of containers, from infrastructure as code design, to increased density per node when compared to a vm. While also providing a more familiar mechanism to manange each instance since you can easily treat each container as if it were a virtual machine.
You can read more about the feature set in LXD here: https://linuxcontainers.org/lxd/introduction/#features
Install LXD
Installing LXD on Ubntu is simple.
snap install lxd
Just run that command and follow the prompts. You can get a bit more details from the linuxcontainers web site
LXD profiles
Instead of jumping into creating a new LXD profile I like to copy the default profile into a new one.
You can list the current profiles like this:
lxc profile list
If you have just installed LXD or have never messed with the profiles that command should output something like this:
+---------+---------+
| NAME | USED BY |
+---------+---------+
| default | 0 |
+---------+---------+
After the initial installation LXD comes with a default profile that any new container will inherit. You can see whats is in that profile like this:
lxc profile show default
config: {}
description: Default LXD profile
devices:
eth0:
name: eth0
network: lxdbr0
type: nic
root:
path: /
pool: default
type: disk
name: default
used_by: []
Copy the default profile
I like to keep my profiles with cloud-init in my home directory so that I can version control them with git.
mkdir -p ~/lxd-profiles
cd ~/lxd-profiles
From here lets copy the default profile and cat it to a file.
lxc profile copy default dev-test
lxc profile list
+----------+---------+
| NAME | USED BY |
+----------+---------+
| default | 0 |
+----------+---------+
| dev-test | 0 |
+----------+---------+
If you've been following along you should now have a new profile called dev-test that was copied from the default profile.
Now lets save that profile to a file so we can start making some changes to it.
lxc profile show dev-test >> dev-test.profile
Now take a look at our new dev-test.profile file
cat dev-test.profile
config: {}
description: Default LXD profile
devices:
eth0:
name: eth0
network: lxdbr0
type: nic
root:
path: /
pool: default
type: disk
name: dev-test
used_by: []
If you compare this to your default profile they should be the same at this point. We are going to use the default profile as a template to insert our cloud-init data and customize the default container images automatically after they are built.
Using your favorite text editor open the file dev-test.profile and add the lines highlighted below:
{{< highlight bash “hllines=2-15” >}}
config:
user.user-data:
#cloud-config
packageupdate: true
packageupgrade: true
packagerebootifrequired: true
packages:
– git
– build-essential
– awscli
– python3-pip
runcmd:
– [ pip3, install, ansible ]
– [ pip3, install, pyvmomi ]
description: developement profile
devices:
eth0:
name: eth0
network: lxdbr0
type: nic
root:
path: /
pool: default
type: disk
name: dev-test
used_by: []
{{< /highlight >}}
I want to point out a couple things here.
1) The line that starts with #cloud-config is required – cloud-init won't work without it.
2) Spacing is important in yaml syntax, watch your spaces. Generally each sub-section is indented with 2 spaces.
Here is what our changes are doing:
1) user.user-data: | —> This tells LXD to pipe the next set of instructions to cloud-init as user-data.
2) cloud-config —> Cloud-init requires this line just do it.
3) package_update, package_upgrade, and package_reboot_if_required. —> Do a full update and reboot if necessary.
4) packages —> install the following packages.
1) Note that each package in the list is indented with 2 spaces and and starts with a dash -.
5) runcmd —> tells cloud-init to run a command.
1) Note that each argument to the command is separated by a comma “,”.
2) Read more about here runcmd
Push changes into the profile
Now that we have a simple profile with cloud-init we need to load this new profile into LXD.
cat dev-test.profile | lxc profile edit dev-test
Verify the changes took effect.
lxc profile show dev-test
Launch a LXC container using the new profile
lxc launch -p dev-test ubuntu:lts test1
If this is the first time you've run a LXD container then it will take a minute for the image to be downloaded and start up, otherwise the launch command should execute within just a few seconds.
Next you will want to exec into the container and check to see if our cloud-init worked.
lxc exec test1 bash
Once inside the container execute the following command:
cloud-init status
You should see one of three outputs: running, done, or error.
If it say's running you'll need to wait a bit longer for cloud-init to complete it's work. This profile is pretty simple so it should complete fairly quickly.
If you want to wait for the configuration to finish, you can execute:
cloud-init status --wait
Once the status changes from “running to done” lets verify we got a some of our packages:
ansible --version
ansible 2.10.2
config file = None
configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python3.8/dist-packages/ansible
executable location = /usr/local/bin/ansible
python version = 3.8.5 (default, Jul 28 2020, 12:59:40) [GCC 9.3.0]
If everything went well you should now have a clean working environment to test ansible playbooks, write some python scripts, or build stuff in aws. This template should be enough to get you started customizing your LXD containers in anyway you want.
A note on cloud-init errors
Sometimes cloud-init can be a little finicky. Occasionally you might check cloud-init status and see “done”, but when you check for your packages or other configuration you will find that it's all missing. This is sometimes a symptom of your cloud-init having syntax issues; i.e. incorrect spacing, no cloud-config decleration, or a missing dependency when running a command.
Check the logs
The log files for cloud-init are kept on the container at /var/log/cloud-init-output.log.
If you find yourself with a problem, either cloud-init reporting fail, or reporting done but you are not seeing the expected results, make sure to check the output log first.
Look for log entries that look something like this:
2020-10-22 21:08:00,969 - util.py[WARNING]: Failed loading yaml blob. Invalid format at line 10
column 5: "while parsing a flow sequence
in "<unicode string>", line 10, column 5:
Cloud-init has surprisingly good error messages. An error message like that tells you which line and which column you need to look at... well, probably close to that line anyway. If you are using the runcmd declaration make sure you are separating each command argument with a comma “,”.
After you make any needed corrections to the dev-test.profile file, make sure to reload it into LXD the same way we did earlier.
Enough to get started
This is just a sample build file that demonstrates a small subset of the declarations that cloud-init will understand and execute for you.
Check the cloud-init docs for some great examples of what you can do.
I use cloud-init to create containers to modify hugo themes like the one I'm using now. hugo theme introduction. I also keep a profile to build containers that act as web servers to test the full site. But that is only scratching the surface.
You could also use cloud-init to build complete systems in a matter of minutes that have everything from a standard set of users and groups with complex file system structures, and install applications with complete configuration pre-loaded.
There is a lot of power in combining cloud-init with LXD containers. I hope this article helped give you a better picture of how they can be combined to customize system containers.