The Wombelix Post - Proxmox

Customized openSUSE Images, to run on Proxmox VE out of the box, build on OBS

2022-04-26T00:00:00+02:00

I'm using openSUSE whenever possible but had some trouble to use the official images on Proxmox. There is mostly a kvm-and-xen image available, but that doesn't contain cloud-init and / or qemu-guest-agent, but for my use case, automated IPv6-only deployments, both are mandatory.

Proxmox supports cloud-init to configure basics like search domain, nameserver, user, password and ssh key. If the qemu-guest-agent is installed, the IPv6 address configured via SLAAC is available through the API as soon the VM was booted.

In regards to openSUSE MicroOS the situation was even worse, only the OpenStack-Cloud image includes cloud-init but relies on the package cloud-init-config-MicroOS which installs a /etc/cloud/cloud.cfg without a default_user section and enabled root user.

That way configuring a User account with sudo permissions via cloud-init will just not work.

So after some playing around with different approaches and especially due to the fact that I use a lot of different openSUSE Images, I decided to branch the relevant projects / packages on https://build.opensuse.org and let OBS do the heavy lifting for me to build and publish the different images.

Leap 15.3 and 15.4 required also an adjustment of the OBS Project Config, most important for automation was staticlinks to ensure the same link will work even after a new build.

%if "%_repository" == "images"
Type: kiwi
Repotype: staticlinks
Patterntype: none
%endif

# slsaprovenance - as described at https://slsa.dev/provenance/v0.2
BuildFlags: slsaprovenance

After some further testing, let's see which of my changes might be possible to get included upstream.

I'm also working on Proxmox VE "optimized" images for other Distributions like Rocky Linux or Debian, I will probably pick a different approach to customize them, but that's something for another Post some day.

openSUSE MicroOS
openSUSE Tubmleweed JeOS
openSUSE Leap 15.3 JeOS
openSUSE Leap 15.4 JeOS

openSUSE MicroOS

Open Build Service (OBS) Project

https://build.opensuse.org/package/show/home:wombelix:branches:devel:kubic:images/openSUSE-MicroOS

Download

openSUSE Mirror (Automatic Build / Sync): https://download.opensuse.org/repositories/home:/wombelix:/branches:/devel:/kubic:/images/openSUSE_Tumbleweed/

Customizations

Please be aware: The below diff should give you an idea about the changes, it will not be updated, please check the OBS Project for the latest revision

Index: _multibuild
===================================================================
--- _multibuild (revision 105)
+++ _multibuild (revision 8)
@@ -1,8 +1,15 @@
 <multibuild>
+        <flavor>pve</flavor>
+        <flavor>ContainerHost-pve</flavor>
+        <flavor>Kubic-kubeadm-pve</flavor>
+        <flavor>k3s-pve</flavor>
+
         <!-- Special case: Pi2, VBox and onie only get plain MicroOS -->
+        <!--
         <flavor>RaspberryPi2</flavor>
         <flavor>VirtualBox</flavor>
         <flavor>onie</flavor>
+        -->
         <!--
                 for a in kvm-and-xen VMware MS-HyperV OpenStack-Cloud Pine64 Rock64 RaspberryPi Vagrant hardware; do
                         for f in "" "ContainerHost-" "Kubic-kubeadm-"; do
@@ -10,6 +17,7 @@
                         done
                 done
         -->
+        <!--
         <flavor>kvm-and-xen</flavor>
         <flavor>ContainerHost-kvm-and-xen</flavor>
         <flavor>Kubic-kubeadm-kvm-and-xen</flavor>
@@ -48,4 +56,5 @@
         <flavor>k3s-Vagrant</flavor>
         <flavor>k3s-SelfInstall</flavor>
         <flavor>k3s-DigitalOcean-Cloud</flavor>
+        -->
 </multibuild>


Index: config.sh
===================================================================
--- config.sh (revision 105)
+++ config.sh (revision 8)
@@ -157,6 +157,7 @@

 ignition_platform='metal'
 case "${kiwi_profiles}" in
+     *pve*) ignition_platform='qemu' ;;
      *kvm*) ignition_platform='qemu' ;;
      *DigitalOcean*) ignition_platform='digitalocean' ;;
      *VMware*) ignition_platform='vmware' ;;
@@ -281,3 +282,21 @@
         chmod 0600 /home/vagrant/.ssh/authorized_keys
         chown -R vagrant /home/vagrant
 fi
+
+#======================================
+# Configure PVE specifics
+#--------------------------------------
+if [[ "$kiwi_profiles" == *"pve"* ]]; then
+     cat > /etc/cloud/cloud.cfg.d/default_user.cfg <<-EOF
+             users:
+               - default
+             disable_root: true
+             system_info:
+               default_user:
+                 name: geeko
+                 lock_passwd: True
+                 groups: [cdrom, users]
+                 sudo: ["ALL=(ALL) NOPASSWD:ALL"]
+                 shell: /bin/bash
+     EOF
+fi


Index: openSUSE-MicroOS.kiwi
===================================================================
--- openSUSE-MicroOS.kiwi (revision 105)
+++ openSUSE-MicroOS.kiwi (revision 8)
@@ -13,6 +13,7 @@
         <profile name="k3s" description="MicroOS with k3s"/>
         <profile name="Kubic-kubeadm" description="MicroOS with CRI-O and kubeadm"/>
         <!-- Platforms -->
+        <profile name="pve" description="Proxmox VE (pve) for x86_64" arch="x86_64"/>
         <profile name="kvm-and-xen" description="kvm-and-xen" arch="x86_64,aarch64"/>
         <profile name="VMware" description="VMware for x86_64" arch="x86_64"/>
         <profile name="MS-HyperV" description="Hyper-V for x86_64" arch="x86_64"/>
@@ -27,6 +28,10 @@
         <profile name="SelfInstall" description="Self Installing Image" arch="x86_64,aarch64"/>
         <profile name="onie" description="ONIE Installer Image" arch="x86_64"/>
         <!-- Images (flavor + platform) -->
+        <profile name="ContainerHost-pve" description="MicroOS with Podman for Proxmox VE (pve)" arch="x86_64">
+            <requires profile="pve"/>
+            <requires profile="ContainerHost"/>
+        </profile>
         <profile name="ContainerHost-kvm-and-xen" description="MicroOS with Podman for KVM and HVM Xen" arch="x86_64,aarch64">
             <requires profile="kvm-and-xen"/>
             <requires profile="ContainerHost"/>
@@ -59,6 +64,10 @@
             <requires profile="Vagrant"/>
             <requires profile="ContainerHost"/>
         </profile>
+        <profile name="Kubic-kubeadm-pve" description="kubeadm with CRI-O for Proxmox VE (pve)" arch="x86_64">
+            <requires profile="pve"/>
+            <requires profile="Kubic-kubeadm"/>
+        </profile>
         <profile name="Kubic-kubeadm-kvm-and-xen" description="kubeadm with CRI-O for KVM and HVM Xen" arch="x86_64,aarch64">
             <requires profile="kvm-and-xen"/>
             <requires profile="Kubic-kubeadm"/>
@@ -103,6 +112,10 @@
             <requires profile="SelfInstall"/>
             <requires profile="Kubic-kubeadm"/>
         </profile>
+        <profile name="k3s-pve" description="MicroOS with k3s for Proxmox VE (pve)" arch="x86_64">
+            <requires profile="pve"/>
+            <requires profile="k3s"/>
+        </profile>
         <profile name="k3s-kvm-and-xen" description="MicroOS with k3s for KVM and HVM Xen" arch="x86_64,aarch64">
             <requires profile="kvm-and-xen"/>
             <requires profile="k3s"/>
@@ -144,6 +157,39 @@
             <requires profile="k3s"/>
         </profile>
     </profiles>
+    <preferences profiles="pve" arch="x86_64">
+        <version>16.0.0</version>
+        <packagemanager>zypper</packagemanager>
+        <bootloader-theme>openSUSE</bootloader-theme>
+        <rpm-excludedocs>true</rpm-excludedocs>
+        <locale>en_US</locale>
+        <type
+            image="vmx"
+            filesystem="btrfs"
+            format="qcow2"
+            firmware="uefi"
+            bootpartition="false"
+            bootkernel="custom"
+            devicepersistency="by-uuid"
+            btrfs_root_is_snapshot="true"
+            btrfs_root_is_readonly_snapshot="true"
+            btrfs_quota_groups="true"
+            spare_part="5G" spare_part_mountpoint="/var" spare_part_fs="btrfs" spare_part_is_last="true" spare_part_fs_attributes="no-copy-on-write"
+        >
+            <bootloader name="grub2" console="gfxterm" />
+            <systemdisk>
+                <volume name="home"/>
+                <volume name="root"/>
+                <volume name="opt"/>
+                <volume name="srv"/>
+                <volume name="boot/grub2/i386-pc"/>
+                <volume name="boot/grub2/x86_64-efi" mountpoint="boot/grub2/x86_64-efi"/>
+                <volume name="boot/writable"/>
+                <volume name="usr/local"/>
+            </systemdisk>
+            <size unit="G">20</size>
+        </type>
+    </preferences>
     <preferences profiles="kvm-and-xen" arch="x86_64">
         <version>16.0.0</version>
         <packagemanager>zypper</packagemanager>
@@ -720,6 +766,10 @@
         <package name="patterns-containers-kubic_loadbalancer"/>
         <package name="patterns-containers-kubic_worker"/>
     </packages>
+    <packages type="image" profiles="pve">
+        <package name="qemu-guest-agent"/>
+        <package name="patterns-microos-cloud"/>
+    </packages>
     <packages type="image" profiles="kvm-and-xen">
         <!-- KVM and Xen specific packages -->
         <package name="xen-tools-domU" arch="x86_64"/>

openSUSE Tubmleweed JeOS

Open Build Service (OBS) Project

https://build.opensuse.org/package/show/home:wombelix:branches:Virtualization:Appliances:Images:openSUSE-Tumbleweed/kiwi-templates-JeOS

Download

openSUSE Mirror (Automatic Build / Sync): https://download.opensuse.org/repositories/home:/wombelix:/branches:/Virtualization:/Appliances:/Images:/openSUSE-Tumbleweed/images/

Customizations

Please be aware: The below diff should give you an idea about the changes, it will not be updated, please check the OBS Project for the latest revision

Index: JeOS.kiwi
===================================================================
--- JeOS.kiwi (revision 70)
+++ JeOS.kiwi (revision 2)
@@ -9,6 +9,7 @@
         <specification>openSUSE Tumbleweed JeOS</specification>
     </description>
     <profiles>
+        <profile name="pve" description="JeOS for Proxmox VE (pve)" arch="x86_64"/>
         <!-- Those two are only used as deps -->
         <profile name="kvm-and-xen" description="JeOS for KVM and Xen" arch="aarch64,x86_64"/>
         <profile name="VMware" description="JeOS for VMware" arch="x86_64"/>
@@ -17,6 +18,39 @@
         <profile name="OpenStack-Cloud" description="JeOS for OpenStack Cloud" arch="x86_64"/>
         <profile name="RaspberryPi" description="JeOS for the Raspberry Pi" arch="aarch64"/>
     </profiles>
+    <preferences profiles="pve" arch="x86_64">
+        <version>15.1.0</version>
+        <packagemanager>zypper</packagemanager>
+        <bootloader-theme>openSUSE</bootloader-theme>
+        <rpm-excludedocs>true</rpm-excludedocs>
+        <type
+            image="vmx"
+            filesystem="btrfs"
+            format="qcow2"
+            bootloader="grub2"
+            firmware="uefi"
+            efipartsize="33"
+            kernelcmdline="rw quiet systemd.show_status=1 console=ttyS0,115200 console=tty0"
+            bootpartition="false"
+            bootkernel="custom"
+            devicepersistency="by-uuid"
+            btrfs_quota_groups="true"
+            btrfs_root_is_snapshot="true"
+        >
+            <systemdisk>
+                <volume name="home"/>
+                <volume name="root"/>
+                <volume name="opt"/>
+                <volume name="srv"/>
+                <volume name="boot/grub2/i386-pc"/>
+                <volume name="boot/grub2/x86_64-efi" mountpoint="boot/grub2/x86_64-efi"/>
+                <volume name="usr/local"/>
+                <volume name="var" copy_on_write="false"/>
+            </systemdisk>
+            <size unit="G">24</size>
+        </type>
+    </preferences>
+
     <preferences profiles="kvm-and-xen" arch="x86_64">
         <version>15.1.0</version>
         <packagemanager>zypper</packagemanager>
@@ -284,6 +318,15 @@
         <package name="zypper-needs-restarting"/> <!-- Some deployment script use this (bsc#1173548) -->
     </packages>

+    <packages type="image" profiles="pve">
+        <package name="btrfsprogs"/>
+        <package name="btrfsmaintenance"/>
+        <package name="grub2-snapper-plugin"/>
+        <package name="snapper-zypp-plugin"/>
+        <package name="snapper"/>
+        <package name="firewalld"/>
+    </packages>
+
     <!-- Not needed in OpenStack as it uses XFS and cloud-init -->
     <packages type="image" profiles="kvm-and-xen,VMware,MS-HyperV,RaspberryPi">
         <!-- Only for btrfs -->
@@ -305,7 +348,7 @@
         <package name="firewalld"/>
     </packages>

-    <packages type="image" profiles="kvm-and-xen,VMware,MS-HyperV,OpenStack-Cloud">
+    <packages type="image" profiles="kvm-and-xen,VMware,MS-HyperV,OpenStack-Cloud,pve">
         <!-- Shim for secure boot everywhere except for RPi -->
         <package name="shim" arch="aarch64,x86_64"/>
     </packages>
@@ -315,14 +358,14 @@
         <package name="xen-libs"/>
         <package name="kernel-default-base"/>
     </packages>
-    <packages type="image" profiles="kvm-and-xen,OpenStack-Cloud">
+    <packages type="image" profiles="kvm-and-xen,OpenStack-Cloud,pve">
         <package name="qemu-guest-agent"/>
     </packages>
     <packages type="image" profiles="VMware">
         <package name="open-vm-tools" arch="x86_64"/>
         <package name="kernel-default-base"/>
     </packages>
-    <packages type="image" profiles="OpenStack-Cloud">
+    <packages type="image" profiles="OpenStack-Cloud,pve">
         <package name="cloud-init"/>
         <package name="cloud-init-config-suse" />
         <package name="xfsprogs"/>
@@ -360,7 +403,7 @@
         <package name="gzip"/>
     </packages>

-    <packages type="bootstrap" profiles="kvm-and-xen">
+    <packages type="bootstrap" profiles="kvm-and-xen,pve">
         <package name="openSUSE-release-appliance-kvm"/>
     </packages>
     <packages type="bootstrap" profiles="OpenStack-Cloud">


Index: _multibuild
===================================================================
--- _multibuild (revision 70)
+++ _multibuild (revision 2)
@@ -1,7 +1,10 @@
 <multibuild>
+  <flavor>pve</flavor>
+  <!--
   <flavor>kvm-and-xen</flavor>
   <flavor>VMware</flavor>
   <flavor>MS-HyperV</flavor>
   <flavor>OpenStack-Cloud</flavor>
   <flavor>RaspberryPi</flavor>
+  -->
 </multibuild>

openSUSE Leap 15.3 JeOS

Open Build Service (OBS) Project

https://build.opensuse.org/package/show/home:wombelix:branches:Virtualization:Appliances:Images:openSUSE-Leap-15.3/kiwi-templates-JeOS

Download

openSUSE Mirror (Automatic Build / Sync): https://download.opensuse.org/repositories/home:/wombelix:/branches:/Virtualization:/Appliances:/Images:/openSUSE-Leap-15.3/images/

Customizations

Please be aware: The below diff should give you an idea about the changes, it will not be updated, please check the OBS Project for the latest revision

Index: JeOS.kiwi
===================================================================
--- JeOS.kiwi (revision 13)
+++ JeOS.kiwi (revision 3)
@@ -9,6 +9,7 @@
         <specification>openSUSE Leap 15.3 JeOS</specification>
     </description>
     <profiles>
+        <profile name="pve" description="JeOS for Proxmox VE (pve)" arch="x86_64"/>
         <profile name="kvm-and-xen" description="JeOS for KVM and Xen" arch="x86_64"/>
         <profile name="kvm" description="JeOS for KVM" arch="aarch64"/>
         <profile name="VMware" description="JeOS for VMware" arch="x86_64"/>
@@ -16,6 +17,47 @@
         <profile name="OpenStack-Cloud" description="JeOS for OpenStack Cloud" arch="x86_64"/>
         <profile name="RaspberryPi" description="JeOS for the Raspberry Pi" arch="aarch64"/>
     </profiles>
+    <preferences profiles="pve">
+        <version>15.3</version>
+        <packagemanager>zypper</packagemanager>
+        <bootsplash-theme>openSUSE</bootsplash-theme>
+        <bootloader-theme>openSUSE</bootloader-theme>
+    <!-- those settings are applied by suseConfig in config.sh
+        <locale>en_US</locale>
+        <keytable>us.map.gz</keytable>
+        <timezone>Europe/Berlin</timezone>
+        <hwclock>utc</hwclock>
+    -->
+        <rpm-excludedocs>true</rpm-excludedocs>
+        <type
+            image="vmx"
+            filesystem="btrfs"
+            format="qcow2"
+            bootloader="grub2"
+            firmware="uefi"
+            efipartsize="33"
+            kernelcmdline="systemd.show_status=1 console=ttyS0,115200 console=tty0 quiet"
+            bootpartition="false"
+            bootkernel="custom"
+            devicepersistency="by-uuid"
+            btrfs_quota_groups="true"
+            btrfs_root_is_snapshot="true"
+        >
+            <systemdisk>
+                <volume name="home"/>
+                <volume name="root"/>
+                <volume name="tmp"/>
+                <volume name="opt"/>
+                <volume name="srv"/>
+                <volume name="boot/grub2/i386-pc"/>
+                <volume name="boot/grub2/x86_64-efi" mountpoint="boot/grub2/x86_64-efi"/>
+                <volume name="usr/local"/>
+                <volume name="var" copy_on_write="false"/>
+            </systemdisk>
+            <size unit="G">24</size>
+        </type>
+    </preferences>
+
     <preferences profiles="kvm-and-xen">
         <version>15.3</version>
         <packagemanager>zypper</packagemanager>
@@ -296,10 +338,19 @@
     </packages>

     <!-- Shim for secure boot everywhere except for RPi -->
-    <packages type="image" profiles="kvm-and-xen,kvm,VMware,MS-HyperV,OpenStack-Cloud">
+    <packages type="image" profiles="kvm-and-xen,kvm,VMware,MS-HyperV,OpenStack-Cloud,pve">
         <package name="shim" arch="aarch64,x86_64"/>
     </packages>

+    <packages type="image" profiles="pve">
+        <package name="btrfsprogs"/>
+        <package name="btrfsmaintenance"/>
+        <package name="grub2-snapper-plugin"/>
+        <package name="snapper-zypp-plugin"/>
+        <package name="snapper"/>
+        <package name="firewalld"/>
+    </packages>
+
     <!-- Not needed in OpenStack as it uses XFS and cloud-init -->
     <packages type="image" profiles="kvm-and-xen,kvm,VMware,MS-HyperV,RaspberryPi">
         <!-- Only for btrfs -->
@@ -322,17 +373,17 @@
         <package name="xen-tools-domU" arch="x86_64"/>
         <package name="xen-libs" arch="x86_64"/>
     </packages>
-    <packages type="image" profiles="kvm-and-xen,kvm">
+    <packages type="image" profiles="kvm-and-xen,kvm,pve">
         <package name="kernel-default-base"/>
     </packages>
-    <packages type="image" profiles="kvm-and-xen,kvm,OpenStack-Cloud">
+    <packages type="image" profiles="kvm-and-xen,kvm,OpenStack-Cloud,pve">
         <package name="qemu-guest-agent"/>
     </packages>
     <packages type="image" profiles="VMware">
         <package name="open-vm-tools" arch="x86_64"/>
         <package name="kernel-default-base"/>
     </packages>
-    <packages type="image" profiles="OpenStack-Cloud">
+    <packages type="image" profiles="OpenStack-Cloud,pve">
         <package name="cloud-init" />
         <package name="cloud-init-config-suse" />
         <package name="xfsprogs" />
@@ -363,7 +414,7 @@
         <package name="openSUSE-release"/>
     </packages>

-    <packages type="bootstrap" profiles="kvm,kvm-and-xen">
+    <packages type="bootstrap" profiles="kvm,kvm-and-xen,pve">
         <package name="openSUSE-release-appliance-kvm"/>
     </packages>
     <packages type="bootstrap" profiles="OpenStack-Cloud">


Index: _multibuild
===================================================================
--- _multibuild (revision 13)
+++ _multibuild (revision 3)
@@ -1,8 +1,11 @@
 <multibuild>
+  <flavor>pve</flavor>
+  <!--
   <flavor>kvm-and-xen</flavor>
   <flavor>kvm</flavor>
   <flavor>VMware</flavor>
   <flavor>MS-HyperV</flavor>
   <flavor>OpenStack-Cloud</flavor>
   <flavor>RaspberryPi</flavor>
+  -->
 </multibuild>

openSUSE Leap 15.4 JeOS

Open Build Service (OBS) Project

https://build.opensuse.org/package/show/home:wombelix:branches:Virtualization:Appliances:Images:openSUSE-Leap-15.4/kiwi-templates-JeOS

Download

openSUSE Mirror (Automatic Build / Sync): https://download.opensuse.org/repositories/home:/wombelix:/branches:/Virtualization:/Appliances:/Images:/openSUSE-Leap-15.4/images/

Customizations

Please be aware: The below diff should give you an idea about the changes, it will not be updated, please check the OBS Project for the latest revision

Index: JeOS.kiwi
===================================================================
--- JeOS.kiwi (revision 2)
+++ JeOS.kiwi (revision 3)
@@ -9,6 +9,7 @@
         <specification>openSUSE Leap 15.4 JeOS</specification>
     </description>
     <profiles>
+        <profile name="pve" description="JeOS for Proxmox VE (pve)" arch="x86_64"/>
         <profile name="kvm-and-xen" description="JeOS for KVM and Xen" arch="x86_64"/>
         <profile name="kvm" description="JeOS for KVM" arch="aarch64"/>
         <profile name="VMware" description="JeOS for VMware" arch="x86_64"/>
@@ -16,6 +17,47 @@
         <profile name="OpenStack-Cloud" description="JeOS for OpenStack Cloud" arch="x86_64"/>
         <profile name="RaspberryPi" description="JeOS for the Raspberry Pi" arch="aarch64"/>
     </profiles>
+    <preferences profiles="pve">
+        <version>15.4</version>
+        <packagemanager>zypper</packagemanager>
+        <bootsplash-theme>openSUSE</bootsplash-theme>
+        <bootloader-theme>openSUSE</bootloader-theme>
+    <!-- those settings are applied by suseConfig in config.sh
+        <locale>en_US</locale>
+        <keytable>us.map.gz</keytable>
+        <timezone>Europe/Berlin</timezone>
+        <hwclock>utc</hwclock>
+    -->
+        <rpm-excludedocs>true</rpm-excludedocs>
+        <type
+            image="vmx"
+            filesystem="btrfs"
+            format="qcow2"
+            bootloader="grub2"
+            firmware="uefi"
+            efipartsize="33"
+            kernelcmdline="rw systemd.show_status=1 console=ttyS0,115200 console=tty0 quiet"
+            bootpartition="false"
+            bootkernel="custom"
+            devicepersistency="by-uuid"
+            btrfs_quota_groups="true"
+            btrfs_root_is_snapshot="true"
+        >
+            <systemdisk>
+                <volume name="home"/>
+                <volume name="root"/>
+                <volume name="tmp"/>
+                <volume name="opt"/>
+                <volume name="srv"/>
+                <volume name="boot/grub2/i386-pc"/>
+                <volume name="boot/grub2/x86_64-efi" mountpoint="boot/grub2/x86_64-efi"/>
+                <volume name="usr/local"/>
+                <volume name="var" copy_on_write="false"/>
+            </systemdisk>
+            <size unit="G">24</size>
+        </type>
+    </preferences>
+
     <preferences profiles="kvm-and-xen">
         <version>15.4</version>
         <packagemanager>zypper</packagemanager>
@@ -298,10 +340,19 @@
     </packages>

     <!-- Shim for secure boot everywhere except for RPi -->
-    <packages type="image" profiles="kvm-and-xen,kvm,VMware,MS-HyperV,OpenStack-Cloud">
+    <packages type="image" profiles="kvm-and-xen,kvm,VMware,MS-HyperV,OpenStack-Cloud,pve">
         <package name="shim" arch="aarch64,x86_64"/>
     </packages>

+    <packages type="image" profiles="pve">
+        <package name="btrfsprogs"/>
+        <package name="btrfsmaintenance"/>
+        <package name="grub2-snapper-plugin"/>
+        <package name="snapper-zypp-plugin"/>
+        <package name="snapper"/>
+        <package name="firewalld"/>
+    </packages>
+
     <!-- Not needed in OpenStack as it uses XFS and cloud-init -->
     <packages type="image" profiles="kvm-and-xen,kvm,VMware,MS-HyperV,RaspberryPi">
         <!-- Only for btrfs -->
@@ -324,17 +375,17 @@
         <package name="xen-tools-domU" arch="x86_64"/>
         <package name="xen-libs" arch="x86_64"/>
     </packages>
-    <packages type="image" profiles="kvm-and-xen,kvm">
+    <packages type="image" profiles="kvm-and-xen,kvm,pve">
         <package name="kernel-default-base"/>
     </packages>
-    <packages type="image" profiles="kvm-and-xen,kvm,OpenStack-Cloud">
+    <packages type="image" profiles="kvm-and-xen,kvm,OpenStack-Cloud,pve">
         <package name="qemu-guest-agent"/>
     </packages>
     <packages type="image" profiles="VMware">
         <package name="open-vm-tools" arch="x86_64"/>
         <package name="kernel-default-base"/>
     </packages>
-    <packages type="image" profiles="OpenStack-Cloud">
+    <packages type="image" profiles="OpenStack-Cloud,pve">
         <package name="cloud-init" />
         <package name="cloud-init-config-suse" />
         <package name="xfsprogs" />
@@ -365,7 +416,7 @@
         <package name="openSUSE-release"/>
     </packages>

-    <packages type="bootstrap" profiles="kvm,kvm-and-xen">
+    <packages type="bootstrap" profiles="kvm,kvm-and-xen,pve">
         <package name="openSUSE-release-appliance-kvm"/>
     </packages>
     <packages type="bootstrap" profiles="OpenStack-Cloud">


Index: _multibuild
===================================================================
--- _multibuild (revision 2)
+++ _multibuild (revision 3)
@@ -1,8 +1,11 @@
 <multibuild>
+  <flavor>pve</flavor>
+  <!--
   <flavor>kvm-and-xen</flavor>
   <flavor>kvm</flavor>
   <flavor>VMware</flavor>
   <flavor>MS-HyperV</flavor>
   <flavor>OpenStack-Cloud</flavor>
   <flavor>RaspberryPi</flavor>
+  -->
 </multibuild>

Getting started with OBS took a while but then it's really amazing how easy it is to branch existing packages, doesn't matter if it's classic rpm, vm or container images, apply changes and get results.

Import 'protective MBR' GPT disk images like SUSE or openSUSE to LVM-Thin with disabled zeroing cause corruption

2022-04-14T00:00:00+02:00

Any available SUSE and openSUSE KVM Image I tried, ended in a VM that stuck on boot with the message "a start job is running for /dev/disk/by-uuid" or dropped into the dracut emergency shell after a timeout.

No partitions could be found, only "/dev/sda" was visible from inside the VM, gdisk dropped a lot of warnings and errors as well:

Caution! After loading partitions, the CRC doesn't check out!
Warning! Main and backup partition tables differ! Use the 'c' and 'e' options
on the recovery & transformation menu to examine the two tables.

Warning! One or more CRCs don't match. You should repair the disk!
Main header: OK
Backup header: OK
Main partition table: ERROR
Backup partition table: ERROR

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: damaged

****************************************************************************
Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk
verification and recovery are STRONGLY recommended.
****************************************************************************

Warning! Secondary partition table overlaps the last partition by
18315034227491254276 blocks!
You will need to delete this partition or resize it in another utility.

I tried it on a Proxmox VE Host with a LVM-Thin Datastore, a closer look at the LVM-Thin Device of one of the affected VMs confirmed the missing partitions:

fdisk -l /dev/mapper/pve-vm--9181--disk--0
Disk /dev/mapper/pve-vm--9181--disk--0: 24 GiB, 25769803776 bytes, 50331648 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 524288 bytes / 524288 bytes
Disklabel type: dos
Disk identifier: 0x406b6f28

Device                                  Boot Start      End  Sectors Size Id Type
/dev/mapper/pve-vm--9181--disk--0-part1          1 50331647 50331647  24G ee GPT

Partition 1 does not start on physical sector boundary.

I chased that issue down for weeks and almost gave up because I didn't found any clue, it just felt like no one ever faced a similar issue before. After lot of different tests on multiple systems and configurations, my LVM-Thin Storage seemed to cause the issue and by accident I finally found a hint in the Proxmox Forum (Archive: [1], [2]).

Indeed, no Idea why but I disabled zeroing on the LV during conversion to Thin:

lvconvert --type thin-pool pve/lv_test -Zn -y

The official Proxmox Documentation (Archive: [1], [2]) doesn't mention the Parameter -Zn, obviously because of very good reasons.

Tests with a new LVM-Thin with enabled zeroing (default) confirmed that all issues I faced earlier seem to be caused by that. A new VM based on the same qcow2 image as before is now starting without any issue, delay or timeout, just as expected, also the partition table looks way better:

fdisk -l /dev/mapper/pve-vm--9181--disk--0
Disk /dev/mapper/pve-vm--9181--disk--0: 24 GiB, 25769803776 bytes, 50331648 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disklabel type: gpt
Disk identifier: 0D13903E-0408-4963-9C0D-A455B37C6062

Device                                  Start      End  Sectors Size Type
/dev/mapper/pve-vm--9181--disk--0-part1  2048     6143     4096   2M BIOS boot
/dev/mapper/pve-vm--9181--disk--0-part2  6144    73727    67584  33M EFI System
/dev/mapper/pve-vm--9181--disk--0-part3 73728 50331614 50257887  24G Linux filesystem

I installed some VMs "normally" by booting a .iso files, also importing qcow2 images with a "flat" / "oldschool" Partition Layout, like the Rocky Linux Generic Cloud Image, which comes with a single XFS formatted MBR Partition, was working without the above problems.

So it looks like that's something specific to disk images with 'protective MBR' GPT and EFI Partitions. So far only SUSE and openSUSE Images seem to use such a Layout by default.

To be perfectly honest, I didn't have a fully technical explanation why disabled zeroing on LVM-Thin has such an impact when importing PMBR disks. I can only assume that during the LV creation and qcow2 import as well as conversion to raw there is some messing around with the first few sectors.

Fact is: I enabled zeroing on the LVM-Thin LV, everything got back to normal and the pre-build images behave now as they are supposed to.

EMail Notification to Unlock LUKS encrypted disk via Dropbear SSH

2022-03-16T00:00:00+01:00

I'm running Proxmox VE Hosts with full encrypted disks and Dropbear SSH to unlock them remotely. I wanted to get notified when I have to login and enter the key, e.g. in case of an unexpected reboot.

My setup is based on the mailjet Free Tier (6000 Mails / Month, 200 Mails / Day) and some initramfs customizing to include curl, a few other required libraries and a simple script to send an API request, tested on PVE 7.1 / Debian 11 (bullseye)

I will enclose placeholder in < >, please replace them with appropriate values based on your Environment.

The hook script /etc/initramfs-tools/hooks/curl take care that curl, libnss_dns and a resolv.conf are included in the initramfs:

#!/bin/sh

PREREQ=""

prereqs()
{
        echo "$PREREQ"
}

case $1 in
prereqs)
        prereqs
        exit 0
        ;;
esac

. /usr/share/initramfs-tools/hook-functions

copy_exec /usr/bin/curl /bin

# Fix DNS resolver
cp -a /usr/lib/x86_64-linux-gnu/libnss_dns* $DESTDIR/usr/lib/x86_64-linux-gnu/
printf "nameserver <dns_server1_ipv4_address>\nnameserver <dns_server2_ipv4_address>\n" > $DESTDIR/etc/resolv.conf

The actual notification will be send by /etc/initramfs-tools/scripts/init-premount/notification through the mailjet API. You shouldn't have a problem to adjust it to use any other Service as long you can trigger a API via curl :)

#!/bin/sh

PREREQ=""

prereqs()
{
        echo "$PREREQ"
}

case $1 in
prereqs)
        prereqs
        exit 0
        ;;
esac

. /scripts/functions

configure_networking

/bin/curl --insecure -s \
-X POST \
--user "<mailjet_api_user>:<mailjet_api_pass>" \
https://api.mailjet.com/v3.1/send \
-H 'Content-Type: application/json' \
-d '{
  "Messages":[
    {
      "From": {
        "Email": "<sender_email>",
        "Name": "<sender_name>"
      },
      "To": [
        {
          "Email": "<recipient_mail>",
          "Name": "<recipient_name>"
        },
        {
          "Email": "<recipient2_mail>",
          "Name": "Pushover"
        }
      ],
      "Subject": "Action required: Unlock <fqdn>!",
      "TextPart": "Server <fqdn> was restarted and need to be unlocked to proceed boot sequence.",
      "CustomID": "DropbearUnlockRequest"
    }
  ]
}'

I'm also using Pushover to receive notifications on my mobile, they offer mail2push, so I just added my personal pushover address as second recipient to get notified by Mail and Pushover.

Make both scripts executable:

chmod +x /etc/initramfs-tools/hooks/curl`
chmod +x /etc/initramfs-tools/scripts/init-premount/notification

Run update-initramfs -u and you are good to go, during the next reboot you should receive an Email Notification to enter your LUKS Key and unlock your disk.

Proxmox and NAT64 - jool seems incompatible with PVE Firewall, Tayga to rescue

2022-03-03T00:00:00+01:00

I'm using IPv6-only whenever possible, but some sites like github.com are still not reachable via IPv6. In such a case NAT64 is the way to go and to simplify the setup as well as to avoid wasting further IPv4 addresses, I wantend to configure it directly on my Proxmox PVE Hosts instead a separate Router VM or similar.

After some research I decided to give Jool a try, it's running as Kernel module, what's promising from a performance perspective and is still under some sort of active development.

Compared to Tayga, which is running in the userland and had the last release in June 2011.

Spoiler: Due to issues in combination with the PVE Firewall, I had to use Tayga at the end, keep reading for the details.

I just wanted to use stateful NAT64, so the setup on my Proxmox Host, which is running Debian 11, was quite easy but it's necessary to install a lot of devel packages and a compiler to build the kernel module, also the module will be rebuild during every kernel upgrade, that might not in all Environments acceptable.

apt install -y pve-headers-$(uname -r)
apt install -y jool-dkms jool-tools

mkdir /etc/jool

/etc/jool/jool.conf
```
{
      "instance": "default",
      "framework": "netfilter",
      "global": {
              "pool6": "64:ff9b::/96"
      }
}
```

systemctl enable --now jool

That's it, was working as a charme and I thought this will be my new standard setup, until I enabled the Proxmox Firewall, which immediately dropped any NAT64 traffic. But nothing was blocked by the Firewall also nothing showed up in the logs.

There wasn't much I could find online, doesn't look like a widely used or even evaluated setup, jool has 0 search results in the Proxmox Forum for example.

After a few hours, I was able to track it down and from my understanding about the implementation of jool as well as the Proxmox PVE Firewall, both just seem to be incompatible.

Debug output of a successful routed package with deactivated firewall:

Jool NAT64/8fee3fc0/default: Packet: 140.82.121.4->78.46.88.247
Jool NAT64/8fee3fc0/default: ICMPv4 type:0 code:0 id:63596
Jool NAT64/8fee3fc0/default: Step 1: Determining the Incoming Tuple
Jool NAT64/8fee3fc0/default: Tuple: 140.82.121.4#63596 -> 78.46.88.247#63596 (ICMP)
Jool NAT64/8fee3fc0/default: Done step 1.
Jool NAT64/8fee3fc0/default: Step 2: Filtering and Updating
Jool NAT64/8fee3fc0/default: BIB entry: 2a01:4f8:120:2700:7864:c5ff:fead:4d8c#1547 - 78.46.88.247#63596 (ICMP)
Jool NAT64/8fee3fc0/default: Session entry: 2a01:4f8:120:2700:7864:c5ff:fead:4d8c#1547 - 64:ff9b::8c52:7904#1547 | 78.46.88.247#63596 - 140.82.121.4#63596 (ICMP)
Jool NAT64/8fee3fc0/default: Done: Step 2.
Jool NAT64/8fee3fc0/default: Step 3: Computing the Outgoing Tuple
Jool NAT64/8fee3fc0/default: Tuple: 64:ff9b::8c52:7904#1547 -> 2a01:4f8:120:2700:7864:c5ff:fead:4d8c#1547 (ICMP)
Jool NAT64/8fee3fc0/default: Done step 3.
Jool NAT64/8fee3fc0/default: Step 4: Translating the Packet
Jool NAT64/8fee3fc0/default: Routing: 64:ff9b::8c52:7904->2a01:4f8:120:2700:7864:c5ff:fead:4d8c
Jool NAT64/8fee3fc0/default: Packet routed via device 'vlan10'.
Jool NAT64/8fee3fc0/default: Done step 4.
Jool NAT64/8fee3fc0/default: Sending packet.
Jool NAT64/8fee3fc0/default: Success.

Debug output of a dropped package, error Hairpinning loop, with enabled firewall:

Jool NAT64/8fee3fc0/default: ===============================================
Jool NAT64/8fee3fc0/default: Packet: 64:ff9b::8c52:7904->2a01:4f8:120:2700:7864:c5ff:fead:4d8c
Jool NAT64/8fee3fc0/default: ICMPv6 type:129 code:0 id:1551
Jool NAT64/8fee3fc0/default: Step 1: Determining the Incoming Tuple
Jool NAT64/8fee3fc0/default: Tuple: 64:ff9b::8c52:7904#1551 -> 2a01:4f8:120:2700:7864:c5ff:fead:4d8c#1551 (ICMP)
Jool NAT64/8fee3fc0/default: Done step 1.
Jool NAT64/8fee3fc0/default: Step 2: Filtering and Updating
Jool NAT64/8fee3fc0/default: Hairpinning loop. Dropping...

First things first, why was it dropped? Because Jool doesn't allow packages, where the source is in the same IPv6 range as the configured pool. Even though the problem is different, the behavior regarding the source IP was explained quite well in this Post on the Jool-list (Archive: [1], [2]).

Well, but why does packages with source IPs from the Jool Pool come up at all, when the Firewall on the NIC of a Virtual Machine is enabled? I learned that three additional interfaces will be created per NIC and PVE VM - Source (Archive: [1], [2]):

fwbr<VMID>i<network interface X> is the firewall bridge where the filtering happens.
fwpr<VMID>i<network interface X> is the bridge in device.
fwln<VMID>i<network interface X> is the bridge out device.

So the traffic flow changes, packages passing further bridges and devices and packages with NAT64 pool addresses as source start to show up, but are dropped by the Jool Kernel Module before they are further routed or filtered by the PVE Firewall.

I understand why it's happening but not how to solve it, my Google Fu wasn't strong enough and I just couldn't find any useful hints how to tackel this problem.

So I decided to replace Jool, even though it feels like the right solution for NAT64, with Tayga, which is using a own TAP device to route the traffic and perform the actual 6<>4 translation. The Setup was, to be fair, as simple as Jool, with the benefit that I didn't had to install devel and compiler packages at all.

apt install -y tayga

/etc/tayga.conf
```
tun-device nat64
ipv4-addr 192.168.255.1
ipv6-addr 2a01:4f8:120:225e::64
prefix 64:ff9b::/96
dynamic-pool 192.168.255.0/24
data-dir /var/spool/tayga
```

systemctl enable --now tayga

VMs with enabled Firewall could reach targets via NAT64 right away after starting the tayga service. Also all rules where applied as expected and for my scenario, the performance was more than sufficient.

At the end I have to admit that Tayga does a great job and is the better fit in a scenario like running on a Hypervisor which also provides Firewalling. This just seem to be something where it can get quite complicated, frustrating and maybe even more or less impossible to use Jool.

I'm still a little concerned about the fact that Tayga doesn't seem to be maintained anymore and might end as abandonware. But as long the package is available in the regular Debian Repository and no critical security issue comes up, I will stick with Tayga for NAT64.