My goal is to programmatically provision a VM. I want to use a script and upstream image as input, and produce a customised, reproducible VM as output with zero manual intervention.

I’ve written before on setting up quick and simple VMs with qemu. That article covered manually setting up VMs, which then need to be further customised via serial console or SSH.

Automating provisioning of a VM via serial console or SSH is non-trivial. I’d need a mechanism on the host to determine when the VM is ready to receive commands, but there’s no obvious way to do so, other than parsing the output of the serial port, waiting for a login: prompt, and then feeding the command. This approach is brittle and error-prone.

tiny-cloud

[permalink]

I opted for using images with tiny-cloud. tiny-cloud is a tool for auto-configuring VMs, and it’s intended for hosting providers. These VM images start up, and automatically seeks a configuration file on how to provision itself. There’s essentially two approaches to providing a configuration mechanism: via a well-known endpoint (a static URL at [fd00:ec2::254] or 169.254.169.254) or a CD image with label cidata.

Surprisingly, the CD image approach is much easier for a development/testing setup with static configuration. I use the official upstream images, the nocloud variant being the one that properly auto-detects a CD. I opted for the variant with properties Release=3.22.2, Arch=x86_64, Firmware=UEFI, Bootstrap=“Tiny Cloud” and Machine=Virtual.

Preparing the metadata is quite straightforward.

First I prepared a meta-data file with a machine identification profile:

instance-id: dev-vm-001
hostname: alpine-dev.whynothugo.nl

And a user-data file with user configuration data (for a VPS provider, this is data that varies depending on the tenant’s requested settings):

#cloud-config
users:
- default
- name: hugo
ssh_authorized_keys:
- ssh-ed25519
AAAAC3NzaC1lZDI1NTE5AAAAIJiHg+cQwtqA7Ay+Fsimh9O5QnALlt6561TEu/Z67mU1
hugo@hyperion.whynothugo.nl
groups: wheel
shell: /bin/sh

packages:
- git

runcmd:
- echo "Setup complete"

Most of this should be quite self-explanatory, and runcmd can be used for further programmatic configuration of the guest.

Creating the VM

[permalink]

I prepared a script which creates the VM itself:

# Fetch the image once, manually.
# curl -LO https://dl-cdn.alpinelinux.org/alpine/v3.22/releases/cloud/nocloud_alpine-3.22.2-x86_64-uefi-tiny-r0.qcow2

# Keep the original image clean.
cp nocloud_alpine-3.22.2-x86_64-uefi-tiny-r0.qcow2 alpine.qcow2

cloud-localds seed.iso user-data meta-data

# tiny-cloud auto-grows the partition, but the disk itself needs to be larger.
qemu-img resize alpine.qcow2 +5G
qemu-system-x86_64 \
-drive file=alpine.qcow2,format=qcow2 \
-drive file=seed.iso,format=raw \
-drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_CODE.fd \
-drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_VARS.fd \
-nic user,hostfwd=tcp:127.0.0.1:2222-:22 \
-m 2G \
-smp cores=2 \
-monitor unix:monitor.sock,server,nowait \
-nographic

A brief explanation of the above:

• I copy the original image since the bootstrap process mutates it; using a copy allows me to quickly start over.

• cloud-localds prepares the previously mentioned ISO image with filename seed.iso.

• I resize (grow) the image because the default is full. I’ll need extra space to install my own packages.

• -drive if=pflash arguments configure the UEFI firmware; the VM won’t start without those.

• -nic user sets up an interface using user-space networking (this allows running the VM without needing to elevate privileges for any network setup.

• hostfwd=tcp:127.0.0.1:2222-:22 forwards port localhost:2222 from the host to port 22 on the VM. This allows me to SSH into the VM. When running other services, I’ll forward the appropriate port as necessary.

• -monitor unix:monitor.sock,server,nowait sets up a monitor socket at ./monitor.sock. I can use this one to send commands to qemu, instructing it to shut down the VM, put it to sleep, resume, etc.

First run

[permalink]

During start-up, the VM logs:

* Tiny Cloud - boot phase ...
++ expand_root: starting
GPT PMBR size mismatch (278527 != 10764287) will be corrected by write.
The backup GPT table is not on the end of the device. This problem will be corrected by write.
Re-reading the partition table failed.: Resource busy
resize2fs 1.47.2 (1-Jan-2025)
Filesystem at /dev/sda2 is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 21
The filesystem on /dev/sda2 is now 5381100 (1k) blocks long.

++ expand_root: done
++ set_ephemeral_network: starting
++ set_ephemeral_network: done
++ set_network_interfaces: starting
++ set_network_interfaces: done
++ enable_sshd: starting

It also auto-configures the network interface (via RA and DHCP).

Near the end of the start-up sequence, it logs:

* Tiny Cloud - early phase ...
++ save_userdata: starting
++ save_userdata: done
[ ok ]
* Tiny Cloud - main phase ...
++ userdata_user: starting
++ userdata_user: done
++ create_default_user: starting
create_default_user: already exists
++ create_default_user: done
++ set_hostname: starting
++ set_hostname: done
++ userdata_bootcmd: starting
++ userdata_bootcmd: done
++ userdata_groups: starting
++ userdata_groups: done
++ userdata_users: starting
chpasswd: password for 'hugo' changed
++ userdata_users: done
++ userdata_write_files: starting
++ userdata_write_files: done
++ userdata_ntp: starting
++ userdata_ntp: done
++ userdata_package_update: starting
++ userdata_package_update: done
++ userdata_package_upgrade: starting
++ userdata_package_upgrade: done
++ userdata_packages: starting
fetch https://dl-cdn.alpinelinux.org/alpine/v3.22/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.22/community/x86_64/APKINDEX.tar.gz
(1/9) Installing brotli-libs (1.1.0-r2)
(2/9) Installing c-ares (1.34.5-r0)
(3/9) Installing nghttp2-libs (1.65.0-r0)
(4/9) Installing libpsl (0.21.5-r3)
(5/9) Installing libcurl (8.14.1-r2)
(6/9) Installing libexpat (2.7.3-r0)
(7/9) Installing pcre2 (10.46-r0)
(8/9) Installing git (2.49.1-r0)
(9/9) Installing git-init-template (2.49.1-r0)
Executing busybox-1.37.0-r19.trigger
OK: 99 MiB in 95 packages
++ userdata_packages: done
++ set_ssh_keys: starting
set_ssh_keys: no ssh key found
++ set_ssh_keys: done
++ ssh_authorized_keys: starting
++ ssh_authorized_keys: done
[ ok ]
ssh-keygen: generating new host keys: RSA ECDSA ED25519
* Starting sshd ... [ ok ]
* Tiny Cloud - final phase ...
++ userdata_runcmd: starting
Setup complete
++ userdata_runcmd: done
++ bootstrap_complete: starting
++ bootstrap_complete: done
[ ok ]

I can now log into the VM via ssh -p 2222 hugo@localhost.

Controlling the VM

[permalink]

Commands can be sent over monitor.sock using echo "$COMMAND_HERE" | socat - UNIX-CONNECT:monitor.sock. An “interactive shell” can be obtained with socat stdio UNIX-CONNECT:monitor.sock. Some example commands:

• info status: show a brief summary of the VM status
• system_powerdown: send ACPI shutdown signal (graceful shutdown)
• quit: immediately stop the VM
• stop: pause/freeze VM execution
• cont: continue VM execution
• system_wakeup: wake up from sleep
• help: show all available commands

Further steps

[permalink]

This is the basis for programmatically providing VMs using an upstream image as a base. With further customisation of user-data, I can configure VMs for specific purposes, and use them in scripted environments.