The Wombelix Post - openSUSE

SUSE Hack Week 2024 - Day 5

2024-11-22T00:00:00+01:00

The problem with a failing test was driving me nuts yesterday. And I made a classic Engineer mistake, I didn't take a step back to clear my head and recharge. I continued to work on the same problem till 1 AM, of course without a result, before I gave up. After some sleep and getting back to work with a structured approach, it took me less than 15 minutes today to validate my theory and figure out what the problem was... Damn, I'm happy I found it but all the time I wasted last night... Whatever, chasing the white rabbit and getting a bit lost happens to the best sometimes.

So, what was the issue? I want to know if a method was called with a specific set of args by using assert_called_once_with. Visually, the expected and actual values match. But I pass an object of model.Project and Mock checks if it's the same instance of the object, not if the content matches. I create a project object in my tests, and pagure.lib.tasks_services.trigger_ci_build() does the same. Even though the keys and values look identical, there are two different objects and the assertion fails.

To solve it, I have to Mock pagure.lib.query.get_authorized_project() and inject the project object that's created in the test. Then the whole thing works as I want it. Next step: Refactoring to use model.PullRequest object with access to all pull request related information instead of pull_request_id and pull_request_uid.

That worked pretty well, writing the tests before the actual code feels weird at first but today that was a productivity booster. It's much easier to focus on the next feature or refactoring, while I don't have to worry that silently breaks something else. As long the other tests continue to pass, I'm still on track. I tried Test-Driven-Development before, but I guess bad habits came back and I never went all-in with it for a longer session.

This Pull Request contains the current state and the outcome of my Hack Week project. I finished around 2/3 of what was planned, I hope to get the rest done in the next week or two.

I wrote a lot Python code before, including tests, but a couple days with an intensive focus helped a lot to brush up my knowledge. Good chances that I not waste time with such a stupid problem next time ;) This week re-affirmed that my passion is solving problems and writing code. I hope I can do that more regularly in future and have the opportunity to free up time and attend SUSE Hack Week 25 as well.

SUSE Hack Week 2024 - Day 4

2024-11-21T00:00:00+01:00

Today was the focus on writing the pagure ci plugin and fine-tune my design and architecture assumptions on the way. Basically transferring the outcome of day 3 into some code. I decided to use a Test-Driven approach and to keep a close eye on existing pagure ci related tests while I have to refactor some existing code too.

Before I started to write tests, I wanted to validate a couple of assumptions on my pagure dev instance. Unfortunately I had to fix a couple errors over there (file permissions, pagure_worker runs as apache and not git user, db scheme forgotten to upgrade...) before I finally could start. It's always a good opportunity, you learn most when things are broken. But I lost quite some coding time that way...

Speaking of coding, the main challenge was that some information I want/need in my generic plugin are currently not passed.

What we currently have doesn't leaves much room for the plugin to get additional data:

def trigger_build(
    project_path,
    url,
    job,
    token,
    branch,
    branch_to,
    cause,
    ci_username=None,
    ci_password=None,
):

What I want in future to make things more flexible:

def trigger_build(
        project: model.Project,
        ci_url: str,
        ci_job: str,
        ci_token: str,
        branch: str,
        branch_to: str,
        is_pull_request: bool = False,
        is_commit: bool = False,
        commit_hash: str = None,
        pull_request_id: str = None,
        pull_request_uid: str = None,
        ci_username = None,
        ci_password = None,
) -> None:

This could be further improved by passing model.PullRequest instead of pull_request_id and pull_request_uid. Then it's up to the pagure ci plugin to decide what data are needed, similar as with project. When I'm done with the generic plugin then the existing jenkins need some refactoring to work with the new parameters.

I started to write the tests and iterate through the different plugin functionality. All that went well until I hit a problem that still drives me crazy. It shouldn't be hard to verify that a method was called with the expected args by using assert_called_once_with. But it throws a nice AssertionError: expected call not found. even though Expected and Actual match:

E           AssertionError: expected call not found.
E           Expected: trigger_build(project=Project(1, name:test, namespace:None, url:None, is_fork:False, parent_id:None), ci_url='https://ci.example.com/', ci_job='pagure', ci_token='random_token', branch='feature', branch_to='main', is_pull_request=True, is_commit=False, commit_hash=None, pull_request_id='1', pull_request_uid='720a0568c1274e74966e54b433b2003e', ci_username=None, ci_password=None)
E           Actual:   trigger_build(project=Project(1, name:test, namespace:None, url:None, is_fork:False, parent_id:None), ci_url='https://ci.example.com/', ci_job='pagure', ci_token='random_token', branch='feature', branch_to='main', is_pull_request=True, is_commit=False, commit_hash=None, pull_request_id='1', pull_request_uid='720a0568c1274e74966e54b433b2003e', ci_username=None, ci_password=None)

I take a break and try it again tomorrow after some sleep. Maybe I'm tired and missing something obvious. But for me both look the same and there shouldn't be an assertion error.

Let's hope I can figure out tomorrow what's going wrong here...

SUSE Hack Week 2024 - Day 3

2024-11-20T00:00:00+01:00

I continued where I left off on the second day, I was thinking about my architecture last night. AWS CodePipeline requires a bit of a custom implementation within the AWS Account, for example with AWS Lambda, to handle third party repos that are not GitHub, Gitlab or Bitbucket. Maybe the pagure ci plugin should then a generic webhook style implementation. It defines the payload that is send and the expected answer to update the PR status. But leaves the actual implementation up to the target system. That way, the plugin isn't limited to one single service. I can use it to interact with AWS CodePipeline, or directly AWS CodeBuild, or any other system were I can implement the server side logic myself.

This shifts the scope of my project a little, but the outcome stays the same. I probably need two AWS Lambda functions, one to receive the webhook and trigger the build. Another to receive status updates from CodeBuild phases and send them back to pagure. The AWS credentials for pagure need permission to trigger the Lambda function for incoming requests. The function has permissions to interact with S3 and CodePipeline / Codebuild. The outgoing Lambda uses the pagure ci token and gets it from a Env Var or AWS Secrets Manager. Every component follows the least privilege principle.

The pagure ci jenkins plugin routes expect a POST request but the token is part of the URL. My plugin will use routes like /ci/webhook/<repo> instead of /ci/jenkins/<repo>/<pagure_ci_token>/build-finished. The token has to be submitted as part of the request. This keeps the token away from plain text logs, it's still a secret and should be treated like that.

With the idea of a generic plugin in mind, Authentication becomes a challenge. Calling an AWS Lambda on its function URL works best with a SIGv4 authentication request. For any other, basic authentication seems the most common and reasonable way. I have to figure out how to handle this properly. I don't want to add more fields to pagure ci and the Web UI. First thing in mind: URI Scheme. https means plain request with basic auth based on ci_username and ci_password, if set. https+sigv4 = AWS SIGv4 authentication request, uses ci_username (Access Key) and ci_password (Secret Key).

I'll add the Lambda code and a IaC example (AWS CDK, AWS CloudFormation or OpenTofu/Terraform) to the pagure repo. That way the end-to-end integration becomes a first-class citizen in pagure, not just the ci plugin.

The time I could invest today was pretty limited and focused on further Architecture details. I started with the pagure ci plugin, it is in an early stage but I'm confident that there is more to demonstrate tomorrow.

SUSE Hack Week 2024 - Day 2

2024-11-19T00:00:00+01:00

Second day, time to continue the work on my project after a successful first day. The focus was on research, tests and design decisions for the AWS CodePipeline pagure ci plugin. I want to make the set up on the AWS side as easy and maintenance free as possible. But keep the code and customization logic in pagure lightweight too.

The Buildspec that CodeBuild uses to run a build should be in the git repository and managed independently. The current design and workflow I have in mind looks like this:

User performs Git push to repository on pagure

pagure ci triggers AWS Lambda through function URL

On commits and / or on pull-requests

Payload contains: cause, repo, branch, branch_to

AWS Lambda gets buildspec from default branch and performs Git clone

Clone branch depends on received payload

Buildspec is used from default branch to reduce risk of untrusted Pull Requests

AWS Lambda inject payload data into buildspec, generates zip archive and uploads it to S3

AWS CodePipeline detects changed file on S3

AWS CodePipeline retrieves archive and triggers AWS CodeBuild

AWS CodeBuild gets pagure ci token from AWS Secrets Manager

Injected as Environment Variable

AWS CodeBuild executes phases and reports status back to pagure ci

I still have to work out some details but I can start the implementation tomorrow.

Another thing I stumbled across, interesting: The pagure ci code contains a ci_username and ci_password field since 5 years. But for whatever reason, they were never added to the fields that are rendered in the Web UI. Also, the WTForms field type for ci_password was StringField, it should be PasswordField instead.

diff --git a/pagure/hooks/pagure_ci.py b/pagure/hooks/pagure_ci.py
index 819b2a1b..38702a72 100644
--- a/pagure/hooks/pagure_ci.py
+++ b/pagure/hooks/pagure_ci.py
@@ -121,7 +121,7 @@ class PagureCiForm(FlaskForm):
        [wtforms.validators.Optional(), wtforms.validators.Length(max=255)],
    )

-    ci_password = wtforms.StringField(
+    ci_password = wtforms.PasswordField(
        "Password to authenticate with if needed",
        [wtforms.validators.Optional(), wtforms.validators.Length(max=255)],
    )
@@ -174,7 +174,7 @@ class PagureCi(BaseHook):
    form = PagureCiForm
    db_object = PagureCITable
    backref = "ci_hook"
-    form_fields = ["ci_type", "ci_url", "ci_job", "active_commit", "active_pr"]
+    form_fields = ["ci_type", "ci_url", "ci_job", "ci_username", "ci_password", "active_commit", "active_pr"]
    runner = PagureCIRunner

    @classmethod

Last but not least, when I refactored the pagure ci code 5 months ago one problem stayed unnoticed: To add routes to a Flask Blueprint, the modules need to be imported before the Blueprint is registered with the app. This was the case before the refactoring, but now the import to add the routes didn't happen and at the time the pagure ci plugin is activated, it's too late. I doubt that I came up with the most elegant solution, but at least a functional ;)

diff --git a/pagure/flask_app.py b/pagure/flask_app.py
index e9baa507..c22ea72c 100644
--- a/pagure/flask_app.py
+++ b/pagure/flask_app.py
@@ -11,6 +11,8 @@
from __future__ import absolute_import, unicode_literals

import datetime
+import importlib
+
import gc
import logging
import os
@@ -135,6 +137,14 @@ def create_app(config=None):

    from pagure.api import API  # noqa: E402

+    # Load all configured pagure ci plugins once to initiate routes
+    # Required before Blueprint registration
+    ci_plugins = {}
+    for ci_type in pagure_config["PAGURE_CI_SERVICES"]:
+        ci_plugins[ci_type] = importlib.import_module(
+            "pagure.api.ci." + ci_type
+        )
+
    app.register_blueprint(API)

    from pagure.ui import UI_NS  # noqa: E402

I iterate over all configured pagure ci plugins and import them immediately before the API Blueprint is registered. That way the routes, required to receive status updates, are added, variable ci_plugins is kind of a dummy and never used.

I was hoping for a bit more progress today, troubleshooting the route import bug and planning took quite a while. Three days left, I'm still on track and confident that I have something usable at the end of the week.

SUSE Hack Week 2024 - Day 1

2024-11-18T00:00:00+01:00

I started the work on my project AWS CodePipeline CI plugin for pagure on code.opensuse.org with my nemesis, the mysterious Unable to find object Exception that I couldn't solve in the past. This problem blocked my PR to Bump test containers to F40, bump pip version pinning to align with rpm versions, address some tech debts in dependencies.

One observation I've made last time was that the Database Session is rolled back and new created in the middle of the failing test. I started my troubleshooting there to understand which session was created and re-created during the test. And which one is actually used by the celery task that throws the exception.

Before the rollback (line 171 in test_pagure_flask_dump_load_ticket.py) self.session and self.db_session are not identical, afterwards they are. That was somehow suspicious, but after taking a closer look and playing around with the session and db_session value, it doesn't seem to make any difference and is a dead end.

Then I checked the error message and stack trace once more, the mock patch that injects a lambda function as override for _maybe_wait seem to cause the issues, a detail I didn't notice before:

tests/test_pagure_flask_dump_load_ticket.py:203:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pagure/lib/git.py:549: in update_ticket_from_git
    issue = pagure.lib.query.new_issue(
pagure/lib/query.py:1757: in new_issue
    pagure.lib.git.update_git(issue, repo=repo)
pagure/lib/git.py:165: in update_git
    _maybe_wait(queued)
/usr/lib/python3.12/site-packages/mock/mock.py:1100: in __call__
    return _mock_self._mock_call(*args, **kwargs)
/usr/lib/python3.12/site-packages/mock/mock.py:1104: in _mock_call
    return _mock_self._execute_mock_call(*args, **kwargs)
/usr/lib/python3.12/site-packages/mock/mock.py:1167: in _execute_mock_call
    result = effect(*args, **kwargs)
tests/test_pagure_flask_dump_load_ticket.py:43: in <lambda>
    mw.side_effect = lambda result: result.get()
/usr/lib/python3.12/site-packages/celery/result.py:1026: in get
    raise self.result if isinstance(
/usr/lib/python3.12/site-packages/celery/app/trace.py:477: in trace_task
    R = retval = fun(*args, **kwargs)
pagure/lib/tasks_utils.py:36: in decorated_function
    return function(self, session, *args, **kwargs)

I checked older logs, they confirm that this trace came up earlier already...

So what does _maybe_wait do? well ...

def _maybe_wait(result):
    """Function to patch if one wants to wait for finish.

    This function should only ever be overridden by a few tests that depend
    on counting and very precise timing."""
    pass

Seem to cover very few edge cases then?! How is it used by other tests?

tests/__init__.py:def create_maybe_waiter(method, getter):
tests/__init__.py:        self.app.get = create_maybe_waiter(self.app.get, self.app.get)
tests/__init__.py:        self.app.post = create_maybe_waiter(self.app.post, self.app.get)
tests/__init__.py:    """Helper function for definitely waiting in _maybe_wait."""
tests/test_pagure_flask_dump_load_ticket.py:    @patch("pagure.lib.git._maybe_wait")
tests/test_pagure_lib_drop_issue.py:    @patch("pagure.lib.git._maybe_wait", tests.definitely_wait)
tests/test_pagure_lib_drop_issue.py:    @patch("pagure.lib.git._maybe_wait", tests.definitely_wait)
tests/test_pagure_lib_git.py:        with patch("pagure.lib.git._maybe_wait", tests.definitely_wait):

Twice in tests/test_pagure_lib_drop_issue.py, once in tests/test_pagure_lib_git.py. Always with tests.definitely_wait, never in the way it's used in tests/test_pagure_flask_dump_load_ticket.py. A test with @patch("pagure.lib.git._maybe_wait", tests.definitely_wait) ended up in the same Unable to find object exception. After dropping the whole patch / mock, tests/test_pagure_flask_dump_load_ticket.py::PagureFlaskDumpLoadTicketTests::test_dumping_reloading_ticket is happy after all and passes.

I have to admit, I still don't fully understand why the test was implemented like that. My only explanation is, that some Celery behavior must have changed over time in a release. And after removing / bumping the version pinning and using Fedora 40 it came up as a problem.

Such tests are there to rely on, and if that one now passes without mocking, I'm fine with that.

After finally fixing this annoying problem, one of four Goals that I have for the Hack Week is completed.

Project Goals:

Resolve issues with outdated python dependencies to ensure pagure runs on Python 3.11 and current package versions. At least Fedora 40 python and python package versions. Fedora 41 would be even better. This will also satisfy openSUSE Tumbleweed and Leap 15.6 package versions used on code.opensuse.org

Validate if https://bugzilla.opensuse.org/show_bug.cgi?id=1229570 is fixed for code.opensuse.org and that all required packages / backports landed in openSUSE:infrastructure:pagure

Develop AWS CodePipeline pagure CI plugin and create an upstream pull request

Backport Plugin into openSUSE pagure package on OBS till new upstream release

While I was waiting for my pagure PR to be reviewed and merged, I worked on Goal 2. Based on the Bugzilla Ticket, the problem with the unsatisfied runtime dependency on cffi is solved for openSUSE:infrastructure:pagure. And still pending for openSUSE Leap 15.6 / SLES 15 SP6 in general. But for know I only care about pagure on code.opensuse.org. The OBS project had two broken packages that I fixed. Nothing else to do, Goal 2 completed.

In preparation for Day 2, I updated my pagure dev system to the latest master branch version, including the Patch from my pending PR. This allows me to develop the actual pagure CI plugin for AWS CodePipeline on the latest build. Avoids running into issues because of old code and refactoring to get it included into the pagure code base.

I'm pretty happy with the progress of the first Day, can't wait how things going tomorrow :)

Recordings of my sessions at openSUSE Conference 2024 are online

2024-07-07T00:00:00+02:00

I had the great opportunity to speak at openSUSE Conference 24 in Nuremberg about Pagure CI integration with AWS CodePipeline and NeuVector Integration into AWS CodePipeline CI/CD workflow, the recordings are now online.

Session 1: Pagure CI integration with AWS CodePipeline

Pagure is a lightweight git centered forge based on Python with a long usage history in the Fedora project. It's well known in the openSUSE project too with an instance on code.opensuse.org. I demonstrate in this talk the current state of the Pagure Plugin system. How I implemented AWS CodePipeline as additional CI type. And how this can serve as blueprint to optimize the Plugin system and add other CI types. I will share the current status of my work to run Pagure on Kubernetes. And some good-first-issues you can tackle if you want to contribute to Pagure.

Session 2: NeuVector Integration into AWS CodePipeline CI/CD workflow

NeuVector is a open source container security platform. Key strengths are vulnerability and runtime scanning. I demonstrate in this talk how you ensure that only container images without a detected vulnerability move to the next stage in your Pipeline. How you define the baseline of allowed activities of your application. And how you can block the deployment into production if an unexpected behavior at runtime was detected in your testing stage. I'll use AWS CodePipeline, AWS CodeDeploy and AWS CloudFormation. The procedure is applicable to other toolset and Hybrid environments as well.

Recordings of my sessions at openSUSE Conference 2023 are online

2023-07-13T00:00:00+02:00

I had the great opportunity to speak at openSUSE Conference 23 in Nuremberg about Rancher integration with AWS services: possibilities, challenges, outlook and openSUSE ALP prototype on AWS, experimental, but fun!, the recordings are now online.

openSUSE ALP prototype on AWS, experimental, but fun!

How does the latest prototype of the Adaptable Linux Platform behave on AWS? What is required to build a EC2 compatible image via OBS? How does the workflow look like to go from OBS to an available and deployable AMI? What is the state of integration with AWS compared to openSUSE Leap? And what has SUSE NeuVector todo with all that? Let me take you on my journey and share my experiences, thoughts and results with you.

Rancher integration with AWS services: possibilities, challenges, outlook.

Rancher can deploy and manage your Kubernetes clusters on AWS EKS and EC2. But what about things like Authentication, Logging, Monitoring or Backup? I will give an overview of AWS services for these four pillars and talk about what’s already possible, which challenges some integrations might have and an outlook what’s planned. Learn more about how the integrations are working under the hood and which technologies and open-sources solutions are involved.

openSUSE Tumbleweed - Python 3.10 - No package metadata was found for vorta

2022-07-17T00:00:00+02:00

I'm using Vorta, a Desktop Client for BorgBackup, on my Notebook with openSUSE Tumbleweed, which stopped working after upgrading Tumbleweed to a Version that started using Python 3.10 as default.

Error message when starting vorta from the command line:

vorta

Traceback (most recent call last):
  File "/usr/bin/vorta", line 33, in <module>
    sys.exit(load_entry_point('vorta==0.8.2', 'gui_scripts', 'vorta')())
  File "/usr/bin/vorta", line 22, in importlib_load_entry_point
    for entry_point in distribution(dist_name).entry_points
  File "/usr/lib64/python3.10/importlib/metadata/__init__.py", line 957, in distribution
    return Distribution.from_name(distribution_name)
  File "/usr/lib64/python3.10/importlib/metadata/__init__.py", line 548, in from_name
    raise PackageNotFoundError(name)
importlib.metadata.PackageNotFoundError: No package metadata was found for vorta

I use the openSUSE Tumbleweed build from https://copr.fedorainfracloud.org/coprs/luminoso/vorta/, first tried to update it, but that dropped a dependency error:

sudo zypper up vorta

Loading repository data...
Reading installed packages...
Resolving package dependencies...

Problem: nothing provides 'python3-secretstorage' needed by the to be installed vorta-0.8.7-2.suse.tw.x86_64
 Solution 1: do not install vorta-0.8.7-2.suse.tw.x86_64
 Solution 2: break vorta-0.8.7-2.suse.tw.x86_64 by ignoring some of its dependencies

Well, sure that there is no secretstorage package for python installed? Let's see:

sudo zypper se secretstorage

Loading repository data...
Reading installed packages...

S | Name                    | Summary                                              | Type
--+-------------------------+------------------------------------------------------+--------
i | python38-SecretStorage  | Python bindings to FreeDesktoporg Secret Service API | package
  | python39-SecretStorage  | Python bindings to FreeDesktoporg Secret Service API | package
i | python310-SecretStorage | Python bindings to FreeDesktoporg Secret Service API | package

Looks good but the vorta package expects "python3-secretstorage" and not "python310-SecretStorage", let's ignore the zypper warning about missing dependencies and just install the latest version of vorta:

sudo zypper up vorta

Loading repository data...
Reading installed packages...
Resolving package dependencies...

Problem: nothing provides 'python3-secretstorage' needed by the to be installed vorta-0.8.7-2.suse.tw.x86_64
 Solution 1: do not install vorta-0.8.7-2.suse.tw.x86_64
 Solution 2: break vorta-0.8.7-2.suse.tw.x86_64 by ignoring some of its dependencies

Choose from above solutions by number or cancel [1/2/c/d/?] (c): 2
Resolving dependencies...
Resolving package dependencies...

The following package is going to be upgraded:
  vorta

1 package to upgrade.
Overall download size: 492.3 KiB. Already cached: 0 B. After the operation, additional 215.5 KiB will be used.
Continue? [y/n/v/...? shows all options] (y): y
Retrieving package vorta-0.8.7-2.suse.tw.x86_64                          (1/1), 492.3 KiB (  1.7 MiB unpacked)
Retrieving: vorta-0.8.7-2.suse.tw.x86_64.rpm ...........................................................[done]

Checking for file conflicts: ...........................................................................[done]
(1/1) Installing: vorta-0.8.7-2.suse.tw.x86_64 .........................................................[done]

Great, started vorta again from the CLI, but now "PyQt5" is missing:

vorta

Traceback (most recent call last):
  File "/usr/bin/vorta", line 33, in <module>
    sys.exit(load_entry_point('vorta==0.8.7', 'gui_scripts', 'vorta')())
  File "/usr/bin/vorta", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/lib64/python3.10/importlib/metadata/__init__.py", line 171, in load
    module = import_module(match.group('module'))
  File "/usr/lib64/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/lib/python3.10/site-packages/vorta/__main__.py", line 7, in <module>
    from vorta.i18n import trans_late, translate
  File "/usr/lib/python3.10/site-packages/vorta/i18n/__init__.py", line 7, in <module>
    from PyQt5.QtCore import QLocale, QTranslator
ModuleNotFoundError: No module named 'PyQt5'

There is no rpm package so I installed it globally via pip:

sudo pip3 install PyQt5

Collecting PyQt5
  Downloading PyQt5-5.15.7-cp37-abi3-manylinux1_x86_64.whl (8.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.4/8.4 MB 29.4 MB/s eta 0:00:00
Collecting PyQt5-Qt5>=5.15.0
  Downloading PyQt5_Qt5-5.15.2-py3-none-manylinux2014_x86_64.whl (59.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 59.9/59.9 MB 26.1 MB/s eta 0:00:00
Collecting PyQt5-sip<13,>=12.11
  Downloading PyQt5_sip-12.11.0-cp310-cp310-manylinux1_x86_64.whl (359 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 359.7/359.7 KB 21.9 MB/s eta 0:00:00
Installing collected packages: PyQt5-Qt5, PyQt5-sip, PyQt5
Successfully installed PyQt5-5.15.7 PyQt5-Qt5-5.15.2 PyQt5-sip-12.11.0

Afterwards vorta could be started without issues, immediatly created a backup and uploaded the files to BorgBase

Overall not a big deal, the fact that the vorta rpm is expecting a dependency that can't be found on openSUSE Tumbleweed, even though it's installed but with a different name, is more of a cosmetic issue in my opinion, writing this blog post took probably longer than fixing the actual problem ;)

Customized openSUSE Images, to run on Proxmox VE out of the box, build on OBS

2022-04-26T00:00:00+02:00

I'm using openSUSE whenever possible but had some trouble to use the official images on Proxmox. There is mostly a kvm-and-xen image available, but that doesn't contain cloud-init and / or qemu-guest-agent, but for my use case, automated IPv6-only deployments, both are mandatory.

Proxmox supports cloud-init to configure basics like search domain, nameserver, user, password and ssh key. If the qemu-guest-agent is installed, the IPv6 address configured via SLAAC is available through the API as soon the VM was booted.

In regards to openSUSE MicroOS the situation was even worse, only the OpenStack-Cloud image includes cloud-init but relies on the package cloud-init-config-MicroOS which installs a /etc/cloud/cloud.cfg without a default_user section and enabled root user.

That way configuring a User account with sudo permissions via cloud-init will just not work.

So after some playing around with different approaches and especially due to the fact that I use a lot of different openSUSE Images, I decided to branch the relevant projects / packages on https://build.opensuse.org and let OBS do the heavy lifting for me to build and publish the different images.

Leap 15.3 and 15.4 required also an adjustment of the OBS Project Config, most important for automation was staticlinks to ensure the same link will work even after a new build.

%if "%_repository" == "images"
Type: kiwi
Repotype: staticlinks
Patterntype: none
%endif

# slsaprovenance - as described at https://slsa.dev/provenance/v0.2
BuildFlags: slsaprovenance

After some further testing, let's see which of my changes might be possible to get included upstream.

I'm also working on Proxmox VE "optimized" images for other Distributions like Rocky Linux or Debian, I will probably pick a different approach to customize them, but that's something for another Post some day.

openSUSE MicroOS
openSUSE Tubmleweed JeOS
openSUSE Leap 15.3 JeOS
openSUSE Leap 15.4 JeOS

openSUSE MicroOS

Open Build Service (OBS) Project

https://build.opensuse.org/package/show/home:wombelix:branches:devel:kubic:images/openSUSE-MicroOS

Download

openSUSE Mirror (Automatic Build / Sync): https://download.opensuse.org/repositories/home:/wombelix:/branches:/devel:/kubic:/images/openSUSE_Tumbleweed/

Customizations

Please be aware: The below diff should give you an idea about the changes, it will not be updated, please check the OBS Project for the latest revision

Index: _multibuild
===================================================================
--- _multibuild (revision 105)
+++ _multibuild (revision 8)
@@ -1,8 +1,15 @@
 <multibuild>
+        <flavor>pve</flavor>
+        <flavor>ContainerHost-pve</flavor>
+        <flavor>Kubic-kubeadm-pve</flavor>
+        <flavor>k3s-pve</flavor>
+
         <!-- Special case: Pi2, VBox and onie only get plain MicroOS -->
+        <!--
         <flavor>RaspberryPi2</flavor>
         <flavor>VirtualBox</flavor>
         <flavor>onie</flavor>
+        -->
         <!--
                 for a in kvm-and-xen VMware MS-HyperV OpenStack-Cloud Pine64 Rock64 RaspberryPi Vagrant hardware; do
                         for f in "" "ContainerHost-" "Kubic-kubeadm-"; do
@@ -10,6 +17,7 @@
                         done
                 done
         -->
+        <!--
         <flavor>kvm-and-xen</flavor>
         <flavor>ContainerHost-kvm-and-xen</flavor>
         <flavor>Kubic-kubeadm-kvm-and-xen</flavor>
@@ -48,4 +56,5 @@
         <flavor>k3s-Vagrant</flavor>
         <flavor>k3s-SelfInstall</flavor>
         <flavor>k3s-DigitalOcean-Cloud</flavor>
+        -->
 </multibuild>


Index: config.sh
===================================================================
--- config.sh (revision 105)
+++ config.sh (revision 8)
@@ -157,6 +157,7 @@

 ignition_platform='metal'
 case "${kiwi_profiles}" in
+     *pve*) ignition_platform='qemu' ;;
      *kvm*) ignition_platform='qemu' ;;
      *DigitalOcean*) ignition_platform='digitalocean' ;;
      *VMware*) ignition_platform='vmware' ;;
@@ -281,3 +282,21 @@
         chmod 0600 /home/vagrant/.ssh/authorized_keys
         chown -R vagrant /home/vagrant
 fi
+
+#======================================
+# Configure PVE specifics
+#--------------------------------------
+if [[ "$kiwi_profiles" == *"pve"* ]]; then
+     cat > /etc/cloud/cloud.cfg.d/default_user.cfg <<-EOF
+             users:
+               - default
+             disable_root: true
+             system_info:
+               default_user:
+                 name: geeko
+                 lock_passwd: True
+                 groups: [cdrom, users]
+                 sudo: ["ALL=(ALL) NOPASSWD:ALL"]
+                 shell: /bin/bash
+     EOF
+fi


Index: openSUSE-MicroOS.kiwi
===================================================================
--- openSUSE-MicroOS.kiwi (revision 105)
+++ openSUSE-MicroOS.kiwi (revision 8)
@@ -13,6 +13,7 @@
         <profile name="k3s" description="MicroOS with k3s"/>
         <profile name="Kubic-kubeadm" description="MicroOS with CRI-O and kubeadm"/>
         <!-- Platforms -->
+        <profile name="pve" description="Proxmox VE (pve) for x86_64" arch="x86_64"/>
         <profile name="kvm-and-xen" description="kvm-and-xen" arch="x86_64,aarch64"/>
         <profile name="VMware" description="VMware for x86_64" arch="x86_64"/>
         <profile name="MS-HyperV" description="Hyper-V for x86_64" arch="x86_64"/>
@@ -27,6 +28,10 @@
         <profile name="SelfInstall" description="Self Installing Image" arch="x86_64,aarch64"/>
         <profile name="onie" description="ONIE Installer Image" arch="x86_64"/>
         <!-- Images (flavor + platform) -->
+        <profile name="ContainerHost-pve" description="MicroOS with Podman for Proxmox VE (pve)" arch="x86_64">
+            <requires profile="pve"/>
+            <requires profile="ContainerHost"/>
+        </profile>
         <profile name="ContainerHost-kvm-and-xen" description="MicroOS with Podman for KVM and HVM Xen" arch="x86_64,aarch64">
             <requires profile="kvm-and-xen"/>
             <requires profile="ContainerHost"/>
@@ -59,6 +64,10 @@
             <requires profile="Vagrant"/>
             <requires profile="ContainerHost"/>
         </profile>
+        <profile name="Kubic-kubeadm-pve" description="kubeadm with CRI-O for Proxmox VE (pve)" arch="x86_64">
+            <requires profile="pve"/>
+            <requires profile="Kubic-kubeadm"/>
+        </profile>
         <profile name="Kubic-kubeadm-kvm-and-xen" description="kubeadm with CRI-O for KVM and HVM Xen" arch="x86_64,aarch64">
             <requires profile="kvm-and-xen"/>
             <requires profile="Kubic-kubeadm"/>
@@ -103,6 +112,10 @@
             <requires profile="SelfInstall"/>
             <requires profile="Kubic-kubeadm"/>
         </profile>
+        <profile name="k3s-pve" description="MicroOS with k3s for Proxmox VE (pve)" arch="x86_64">
+            <requires profile="pve"/>
+            <requires profile="k3s"/>
+        </profile>
         <profile name="k3s-kvm-and-xen" description="MicroOS with k3s for KVM and HVM Xen" arch="x86_64,aarch64">
             <requires profile="kvm-and-xen"/>
             <requires profile="k3s"/>
@@ -144,6 +157,39 @@
             <requires profile="k3s"/>
         </profile>
     </profiles>
+    <preferences profiles="pve" arch="x86_64">
+        <version>16.0.0</version>
+        <packagemanager>zypper</packagemanager>
+        <bootloader-theme>openSUSE</bootloader-theme>
+        <rpm-excludedocs>true</rpm-excludedocs>
+        <locale>en_US</locale>
+        <type
+            image="vmx"
+            filesystem="btrfs"
+            format="qcow2"
+            firmware="uefi"
+            bootpartition="false"
+            bootkernel="custom"
+            devicepersistency="by-uuid"
+            btrfs_root_is_snapshot="true"
+            btrfs_root_is_readonly_snapshot="true"
+            btrfs_quota_groups="true"
+            spare_part="5G" spare_part_mountpoint="/var" spare_part_fs="btrfs" spare_part_is_last="true" spare_part_fs_attributes="no-copy-on-write"
+        >
+            <bootloader name="grub2" console="gfxterm" />
+            <systemdisk>
+                <volume name="home"/>
+                <volume name="root"/>
+                <volume name="opt"/>
+                <volume name="srv"/>
+                <volume name="boot/grub2/i386-pc"/>
+                <volume name="boot/grub2/x86_64-efi" mountpoint="boot/grub2/x86_64-efi"/>
+                <volume name="boot/writable"/>
+                <volume name="usr/local"/>
+            </systemdisk>
+            <size unit="G">20</size>
+        </type>
+    </preferences>
     <preferences profiles="kvm-and-xen" arch="x86_64">
         <version>16.0.0</version>
         <packagemanager>zypper</packagemanager>
@@ -720,6 +766,10 @@
         <package name="patterns-containers-kubic_loadbalancer"/>
         <package name="patterns-containers-kubic_worker"/>
     </packages>
+    <packages type="image" profiles="pve">
+        <package name="qemu-guest-agent"/>
+        <package name="patterns-microos-cloud"/>
+    </packages>
     <packages type="image" profiles="kvm-and-xen">
         <!-- KVM and Xen specific packages -->
         <package name="xen-tools-domU" arch="x86_64"/>

openSUSE Tubmleweed JeOS

Open Build Service (OBS) Project

https://build.opensuse.org/package/show/home:wombelix:branches:Virtualization:Appliances:Images:openSUSE-Tumbleweed/kiwi-templates-JeOS

Download

openSUSE Mirror (Automatic Build / Sync): https://download.opensuse.org/repositories/home:/wombelix:/branches:/Virtualization:/Appliances:/Images:/openSUSE-Tumbleweed/images/

Customizations

Please be aware: The below diff should give you an idea about the changes, it will not be updated, please check the OBS Project for the latest revision

Index: JeOS.kiwi
===================================================================
--- JeOS.kiwi (revision 70)
+++ JeOS.kiwi (revision 2)
@@ -9,6 +9,7 @@
         <specification>openSUSE Tumbleweed JeOS</specification>
     </description>
     <profiles>
+        <profile name="pve" description="JeOS for Proxmox VE (pve)" arch="x86_64"/>
         <!-- Those two are only used as deps -->
         <profile name="kvm-and-xen" description="JeOS for KVM and Xen" arch="aarch64,x86_64"/>
         <profile name="VMware" description="JeOS for VMware" arch="x86_64"/>
@@ -17,6 +18,39 @@
         <profile name="OpenStack-Cloud" description="JeOS for OpenStack Cloud" arch="x86_64"/>
         <profile name="RaspberryPi" description="JeOS for the Raspberry Pi" arch="aarch64"/>
     </profiles>
+    <preferences profiles="pve" arch="x86_64">
+        <version>15.1.0</version>
+        <packagemanager>zypper</packagemanager>
+        <bootloader-theme>openSUSE</bootloader-theme>
+        <rpm-excludedocs>true</rpm-excludedocs>
+        <type
+            image="vmx"
+            filesystem="btrfs"
+            format="qcow2"
+            bootloader="grub2"
+            firmware="uefi"
+            efipartsize="33"
+            kernelcmdline="rw quiet systemd.show_status=1 console=ttyS0,115200 console=tty0"
+            bootpartition="false"
+            bootkernel="custom"
+            devicepersistency="by-uuid"
+            btrfs_quota_groups="true"
+            btrfs_root_is_snapshot="true"
+        >
+            <systemdisk>
+                <volume name="home"/>
+                <volume name="root"/>
+                <volume name="opt"/>
+                <volume name="srv"/>
+                <volume name="boot/grub2/i386-pc"/>
+                <volume name="boot/grub2/x86_64-efi" mountpoint="boot/grub2/x86_64-efi"/>
+                <volume name="usr/local"/>
+                <volume name="var" copy_on_write="false"/>
+            </systemdisk>
+            <size unit="G">24</size>
+        </type>
+    </preferences>
+
     <preferences profiles="kvm-and-xen" arch="x86_64">
         <version>15.1.0</version>
         <packagemanager>zypper</packagemanager>
@@ -284,6 +318,15 @@
         <package name="zypper-needs-restarting"/> <!-- Some deployment script use this (bsc#1173548) -->
     </packages>

+    <packages type="image" profiles="pve">
+        <package name="btrfsprogs"/>
+        <package name="btrfsmaintenance"/>
+        <package name="grub2-snapper-plugin"/>
+        <package name="snapper-zypp-plugin"/>
+        <package name="snapper"/>
+        <package name="firewalld"/>
+    </packages>
+
     <!-- Not needed in OpenStack as it uses XFS and cloud-init -->
     <packages type="image" profiles="kvm-and-xen,VMware,MS-HyperV,RaspberryPi">
         <!-- Only for btrfs -->
@@ -305,7 +348,7 @@
         <package name="firewalld"/>
     </packages>

-    <packages type="image" profiles="kvm-and-xen,VMware,MS-HyperV,OpenStack-Cloud">
+    <packages type="image" profiles="kvm-and-xen,VMware,MS-HyperV,OpenStack-Cloud,pve">
         <!-- Shim for secure boot everywhere except for RPi -->
         <package name="shim" arch="aarch64,x86_64"/>
     </packages>
@@ -315,14 +358,14 @@
         <package name="xen-libs"/>
         <package name="kernel-default-base"/>
     </packages>
-    <packages type="image" profiles="kvm-and-xen,OpenStack-Cloud">
+    <packages type="image" profiles="kvm-and-xen,OpenStack-Cloud,pve">
         <package name="qemu-guest-agent"/>
     </packages>
     <packages type="image" profiles="VMware">
         <package name="open-vm-tools" arch="x86_64"/>
         <package name="kernel-default-base"/>
     </packages>
-    <packages type="image" profiles="OpenStack-Cloud">
+    <packages type="image" profiles="OpenStack-Cloud,pve">
         <package name="cloud-init"/>
         <package name="cloud-init-config-suse" />
         <package name="xfsprogs"/>
@@ -360,7 +403,7 @@
         <package name="gzip"/>
     </packages>

-    <packages type="bootstrap" profiles="kvm-and-xen">
+    <packages type="bootstrap" profiles="kvm-and-xen,pve">
         <package name="openSUSE-release-appliance-kvm"/>
     </packages>
     <packages type="bootstrap" profiles="OpenStack-Cloud">


Index: _multibuild
===================================================================
--- _multibuild (revision 70)
+++ _multibuild (revision 2)
@@ -1,7 +1,10 @@
 <multibuild>
+  <flavor>pve</flavor>
+  <!--
   <flavor>kvm-and-xen</flavor>
   <flavor>VMware</flavor>
   <flavor>MS-HyperV</flavor>
   <flavor>OpenStack-Cloud</flavor>
   <flavor>RaspberryPi</flavor>
+  -->
 </multibuild>

openSUSE Leap 15.3 JeOS

Open Build Service (OBS) Project

https://build.opensuse.org/package/show/home:wombelix:branches:Virtualization:Appliances:Images:openSUSE-Leap-15.3/kiwi-templates-JeOS

Download

openSUSE Mirror (Automatic Build / Sync): https://download.opensuse.org/repositories/home:/wombelix:/branches:/Virtualization:/Appliances:/Images:/openSUSE-Leap-15.3/images/

Customizations

Please be aware: The below diff should give you an idea about the changes, it will not be updated, please check the OBS Project for the latest revision

Index: JeOS.kiwi
===================================================================
--- JeOS.kiwi (revision 13)
+++ JeOS.kiwi (revision 3)
@@ -9,6 +9,7 @@
         <specification>openSUSE Leap 15.3 JeOS</specification>
     </description>
     <profiles>
+        <profile name="pve" description="JeOS for Proxmox VE (pve)" arch="x86_64"/>
         <profile name="kvm-and-xen" description="JeOS for KVM and Xen" arch="x86_64"/>
         <profile name="kvm" description="JeOS for KVM" arch="aarch64"/>
         <profile name="VMware" description="JeOS for VMware" arch="x86_64"/>
@@ -16,6 +17,47 @@
         <profile name="OpenStack-Cloud" description="JeOS for OpenStack Cloud" arch="x86_64"/>
         <profile name="RaspberryPi" description="JeOS for the Raspberry Pi" arch="aarch64"/>
     </profiles>
+    <preferences profiles="pve">
+        <version>15.3</version>
+        <packagemanager>zypper</packagemanager>
+        <bootsplash-theme>openSUSE</bootsplash-theme>
+        <bootloader-theme>openSUSE</bootloader-theme>
+    <!-- those settings are applied by suseConfig in config.sh
+        <locale>en_US</locale>
+        <keytable>us.map.gz</keytable>
+        <timezone>Europe/Berlin</timezone>
+        <hwclock>utc</hwclock>
+    -->
+        <rpm-excludedocs>true</rpm-excludedocs>
+        <type
+            image="vmx"
+            filesystem="btrfs"
+            format="qcow2"
+            bootloader="grub2"
+            firmware="uefi"
+            efipartsize="33"
+            kernelcmdline="systemd.show_status=1 console=ttyS0,115200 console=tty0 quiet"
+            bootpartition="false"
+            bootkernel="custom"
+            devicepersistency="by-uuid"
+            btrfs_quota_groups="true"
+            btrfs_root_is_snapshot="true"
+        >
+            <systemdisk>
+                <volume name="home"/>
+                <volume name="root"/>
+                <volume name="tmp"/>
+                <volume name="opt"/>
+                <volume name="srv"/>
+                <volume name="boot/grub2/i386-pc"/>
+                <volume name="boot/grub2/x86_64-efi" mountpoint="boot/grub2/x86_64-efi"/>
+                <volume name="usr/local"/>
+                <volume name="var" copy_on_write="false"/>
+            </systemdisk>
+            <size unit="G">24</size>
+        </type>
+    </preferences>
+
     <preferences profiles="kvm-and-xen">
         <version>15.3</version>
         <packagemanager>zypper</packagemanager>
@@ -296,10 +338,19 @@
     </packages>

     <!-- Shim for secure boot everywhere except for RPi -->
-    <packages type="image" profiles="kvm-and-xen,kvm,VMware,MS-HyperV,OpenStack-Cloud">
+    <packages type="image" profiles="kvm-and-xen,kvm,VMware,MS-HyperV,OpenStack-Cloud,pve">
         <package name="shim" arch="aarch64,x86_64"/>
     </packages>

+    <packages type="image" profiles="pve">
+        <package name="btrfsprogs"/>
+        <package name="btrfsmaintenance"/>
+        <package name="grub2-snapper-plugin"/>
+        <package name="snapper-zypp-plugin"/>
+        <package name="snapper"/>
+        <package name="firewalld"/>
+    </packages>
+
     <!-- Not needed in OpenStack as it uses XFS and cloud-init -->
     <packages type="image" profiles="kvm-and-xen,kvm,VMware,MS-HyperV,RaspberryPi">
         <!-- Only for btrfs -->
@@ -322,17 +373,17 @@
         <package name="xen-tools-domU" arch="x86_64"/>
         <package name="xen-libs" arch="x86_64"/>
     </packages>
-    <packages type="image" profiles="kvm-and-xen,kvm">
+    <packages type="image" profiles="kvm-and-xen,kvm,pve">
         <package name="kernel-default-base"/>
     </packages>
-    <packages type="image" profiles="kvm-and-xen,kvm,OpenStack-Cloud">
+    <packages type="image" profiles="kvm-and-xen,kvm,OpenStack-Cloud,pve">
         <package name="qemu-guest-agent"/>
     </packages>
     <packages type="image" profiles="VMware">
         <package name="open-vm-tools" arch="x86_64"/>
         <package name="kernel-default-base"/>
     </packages>
-    <packages type="image" profiles="OpenStack-Cloud">
+    <packages type="image" profiles="OpenStack-Cloud,pve">
         <package name="cloud-init" />
         <package name="cloud-init-config-suse" />
         <package name="xfsprogs" />
@@ -363,7 +414,7 @@
         <package name="openSUSE-release"/>
     </packages>

-    <packages type="bootstrap" profiles="kvm,kvm-and-xen">
+    <packages type="bootstrap" profiles="kvm,kvm-and-xen,pve">
         <package name="openSUSE-release-appliance-kvm"/>
     </packages>
     <packages type="bootstrap" profiles="OpenStack-Cloud">


Index: _multibuild
===================================================================
--- _multibuild (revision 13)
+++ _multibuild (revision 3)
@@ -1,8 +1,11 @@
 <multibuild>
+  <flavor>pve</flavor>
+  <!--
   <flavor>kvm-and-xen</flavor>
   <flavor>kvm</flavor>
   <flavor>VMware</flavor>
   <flavor>MS-HyperV</flavor>
   <flavor>OpenStack-Cloud</flavor>
   <flavor>RaspberryPi</flavor>
+  -->
 </multibuild>

openSUSE Leap 15.4 JeOS

Open Build Service (OBS) Project

https://build.opensuse.org/package/show/home:wombelix:branches:Virtualization:Appliances:Images:openSUSE-Leap-15.4/kiwi-templates-JeOS

Download

openSUSE Mirror (Automatic Build / Sync): https://download.opensuse.org/repositories/home:/wombelix:/branches:/Virtualization:/Appliances:/Images:/openSUSE-Leap-15.4/images/

Customizations

Please be aware: The below diff should give you an idea about the changes, it will not be updated, please check the OBS Project for the latest revision

Index: JeOS.kiwi
===================================================================
--- JeOS.kiwi (revision 2)
+++ JeOS.kiwi (revision 3)
@@ -9,6 +9,7 @@
         <specification>openSUSE Leap 15.4 JeOS</specification>
     </description>
     <profiles>
+        <profile name="pve" description="JeOS for Proxmox VE (pve)" arch="x86_64"/>
         <profile name="kvm-and-xen" description="JeOS for KVM and Xen" arch="x86_64"/>
         <profile name="kvm" description="JeOS for KVM" arch="aarch64"/>
         <profile name="VMware" description="JeOS for VMware" arch="x86_64"/>
@@ -16,6 +17,47 @@
         <profile name="OpenStack-Cloud" description="JeOS for OpenStack Cloud" arch="x86_64"/>
         <profile name="RaspberryPi" description="JeOS for the Raspberry Pi" arch="aarch64"/>
     </profiles>
+    <preferences profiles="pve">
+        <version>15.4</version>
+        <packagemanager>zypper</packagemanager>
+        <bootsplash-theme>openSUSE</bootsplash-theme>
+        <bootloader-theme>openSUSE</bootloader-theme>
+    <!-- those settings are applied by suseConfig in config.sh
+        <locale>en_US</locale>
+        <keytable>us.map.gz</keytable>
+        <timezone>Europe/Berlin</timezone>
+        <hwclock>utc</hwclock>
+    -->
+        <rpm-excludedocs>true</rpm-excludedocs>
+        <type
+            image="vmx"
+            filesystem="btrfs"
+            format="qcow2"
+            bootloader="grub2"
+            firmware="uefi"
+            efipartsize="33"
+            kernelcmdline="rw systemd.show_status=1 console=ttyS0,115200 console=tty0 quiet"
+            bootpartition="false"
+            bootkernel="custom"
+            devicepersistency="by-uuid"
+            btrfs_quota_groups="true"
+            btrfs_root_is_snapshot="true"
+        >
+            <systemdisk>
+                <volume name="home"/>
+                <volume name="root"/>
+                <volume name="tmp"/>
+                <volume name="opt"/>
+                <volume name="srv"/>
+                <volume name="boot/grub2/i386-pc"/>
+                <volume name="boot/grub2/x86_64-efi" mountpoint="boot/grub2/x86_64-efi"/>
+                <volume name="usr/local"/>
+                <volume name="var" copy_on_write="false"/>
+            </systemdisk>
+            <size unit="G">24</size>
+        </type>
+    </preferences>
+
     <preferences profiles="kvm-and-xen">
         <version>15.4</version>
         <packagemanager>zypper</packagemanager>
@@ -298,10 +340,19 @@
     </packages>

     <!-- Shim for secure boot everywhere except for RPi -->
-    <packages type="image" profiles="kvm-and-xen,kvm,VMware,MS-HyperV,OpenStack-Cloud">
+    <packages type="image" profiles="kvm-and-xen,kvm,VMware,MS-HyperV,OpenStack-Cloud,pve">
         <package name="shim" arch="aarch64,x86_64"/>
     </packages>

+    <packages type="image" profiles="pve">
+        <package name="btrfsprogs"/>
+        <package name="btrfsmaintenance"/>
+        <package name="grub2-snapper-plugin"/>
+        <package name="snapper-zypp-plugin"/>
+        <package name="snapper"/>
+        <package name="firewalld"/>
+    </packages>
+
     <!-- Not needed in OpenStack as it uses XFS and cloud-init -->
     <packages type="image" profiles="kvm-and-xen,kvm,VMware,MS-HyperV,RaspberryPi">
         <!-- Only for btrfs -->
@@ -324,17 +375,17 @@
         <package name="xen-tools-domU" arch="x86_64"/>
         <package name="xen-libs" arch="x86_64"/>
     </packages>
-    <packages type="image" profiles="kvm-and-xen,kvm">
+    <packages type="image" profiles="kvm-and-xen,kvm,pve">
         <package name="kernel-default-base"/>
     </packages>
-    <packages type="image" profiles="kvm-and-xen,kvm,OpenStack-Cloud">
+    <packages type="image" profiles="kvm-and-xen,kvm,OpenStack-Cloud,pve">
         <package name="qemu-guest-agent"/>
     </packages>
     <packages type="image" profiles="VMware">
         <package name="open-vm-tools" arch="x86_64"/>
         <package name="kernel-default-base"/>
     </packages>
-    <packages type="image" profiles="OpenStack-Cloud">
+    <packages type="image" profiles="OpenStack-Cloud,pve">
         <package name="cloud-init" />
         <package name="cloud-init-config-suse" />
         <package name="xfsprogs" />
@@ -365,7 +416,7 @@
         <package name="openSUSE-release"/>
     </packages>

-    <packages type="bootstrap" profiles="kvm,kvm-and-xen">
+    <packages type="bootstrap" profiles="kvm,kvm-and-xen,pve">
         <package name="openSUSE-release-appliance-kvm"/>
     </packages>
     <packages type="bootstrap" profiles="OpenStack-Cloud">


Index: _multibuild
===================================================================
--- _multibuild (revision 2)
+++ _multibuild (revision 3)
@@ -1,8 +1,11 @@
 <multibuild>
+  <flavor>pve</flavor>
+  <!--
   <flavor>kvm-and-xen</flavor>
   <flavor>kvm</flavor>
   <flavor>VMware</flavor>
   <flavor>MS-HyperV</flavor>
   <flavor>OpenStack-Cloud</flavor>
   <flavor>RaspberryPi</flavor>
+  -->
 </multibuild>

Getting started with OBS took a while but then it's really amazing how easy it is to branch existing packages, doesn't matter if it's classic rpm, vm or container images, apply changes and get results.

Import 'protective MBR' GPT disk images like SUSE or openSUSE to LVM-Thin with disabled zeroing cause corruption

2022-04-14T00:00:00+02:00

Any available SUSE and openSUSE KVM Image I tried, ended in a VM that stuck on boot with the message "a start job is running for /dev/disk/by-uuid" or dropped into the dracut emergency shell after a timeout.

No partitions could be found, only "/dev/sda" was visible from inside the VM, gdisk dropped a lot of warnings and errors as well:

Caution! After loading partitions, the CRC doesn't check out!
Warning! Main and backup partition tables differ! Use the 'c' and 'e' options
on the recovery & transformation menu to examine the two tables.

Warning! One or more CRCs don't match. You should repair the disk!
Main header: OK
Backup header: OK
Main partition table: ERROR
Backup partition table: ERROR

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: damaged

****************************************************************************
Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk
verification and recovery are STRONGLY recommended.
****************************************************************************

Warning! Secondary partition table overlaps the last partition by
18315034227491254276 blocks!
You will need to delete this partition or resize it in another utility.

I tried it on a Proxmox VE Host with a LVM-Thin Datastore, a closer look at the LVM-Thin Device of one of the affected VMs confirmed the missing partitions:

fdisk -l /dev/mapper/pve-vm--9181--disk--0
Disk /dev/mapper/pve-vm--9181--disk--0: 24 GiB, 25769803776 bytes, 50331648 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 524288 bytes / 524288 bytes
Disklabel type: dos
Disk identifier: 0x406b6f28

Device                                  Boot Start      End  Sectors Size Id Type
/dev/mapper/pve-vm--9181--disk--0-part1          1 50331647 50331647  24G ee GPT

Partition 1 does not start on physical sector boundary.

I chased that issue down for weeks and almost gave up because I didn't found any clue, it just felt like no one ever faced a similar issue before. After lot of different tests on multiple systems and configurations, my LVM-Thin Storage seemed to cause the issue and by accident I finally found a hint in the Proxmox Forum (Archive: [1], [2]).

Indeed, no Idea why but I disabled zeroing on the LV during conversion to Thin:

lvconvert --type thin-pool pve/lv_test -Zn -y

The official Proxmox Documentation (Archive: [1], [2]) doesn't mention the Parameter -Zn, obviously because of very good reasons.

Tests with a new LVM-Thin with enabled zeroing (default) confirmed that all issues I faced earlier seem to be caused by that. A new VM based on the same qcow2 image as before is now starting without any issue, delay or timeout, just as expected, also the partition table looks way better:

fdisk -l /dev/mapper/pve-vm--9181--disk--0
Disk /dev/mapper/pve-vm--9181--disk--0: 24 GiB, 25769803776 bytes, 50331648 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disklabel type: gpt
Disk identifier: 0D13903E-0408-4963-9C0D-A455B37C6062

Device                                  Start      End  Sectors Size Type
/dev/mapper/pve-vm--9181--disk--0-part1  2048     6143     4096   2M BIOS boot
/dev/mapper/pve-vm--9181--disk--0-part2  6144    73727    67584  33M EFI System
/dev/mapper/pve-vm--9181--disk--0-part3 73728 50331614 50257887  24G Linux filesystem

I installed some VMs "normally" by booting a .iso files, also importing qcow2 images with a "flat" / "oldschool" Partition Layout, like the Rocky Linux Generic Cloud Image, which comes with a single XFS formatted MBR Partition, was working without the above problems.

So it looks like that's something specific to disk images with 'protective MBR' GPT and EFI Partitions. So far only SUSE and openSUSE Images seem to use such a Layout by default.

To be perfectly honest, I didn't have a fully technical explanation why disabled zeroing on LVM-Thin has such an impact when importing PMBR disks. I can only assume that during the LV creation and qcow2 import as well as conversion to raw there is some messing around with the first few sectors.

Fact is: I enabled zeroing on the LVM-Thin LV, everything got back to normal and the pre-build images behave now as they are supposed to.

openSUSE Leap 15.3 on VMware vSphere 6.7: Get Guest Customization working

2022-03-10T00:00:00+01:00

Technically openSUSE Leap 15.3 is equal to SUSE Linux Enterprise Server 15 SP3, both share the same code and binary packages, the only difference is community vs. commercial support. It's even possible to quickly migrate from Leap 15.3 to SLES 15 SP3 (Archive: [1], [2]) if you decide vendor support is preferred at a later point.

VMWare still doesn't officially support Leap 15.3 as VM on vSphere neither allows Guest Customization, but for sure, when you run SLES 15 SP3 it's not a problem and fully supported?!

As explained here (Archive: [1], [2]) in regards to Leap 15.2, one option is to replace to content of /etc/issue with SUSE Linux Enterprise Server 15.

Keep in mind that /etc/issue is a symlink to /run/issue, a volatile and temporary file, managed by systemd service issue-generator. So you have to remove the symlink and create a new /etc/issue file, otherwise your changes will be lost after the next reboot.

sudo rm /etc/issue && echo "SUSE Linux Enterprise Server 15" > /etc/issue

In addition I had to change the OS Type in the VM Settings on vSphere to "SUSE Linux Enterprise Server 15" as well, otherwise the Guest Customization couldn't even be started without an error message.

I like the generated issue file and the additional information, so after the new deployed VM is up and running, let's just revert the change:

sudo rm /etc/issue && ln -s /run/issue /etc/issue

Little more complicated, but also possible, is to replace /etc/os-release with the content from a SLES 15 SP3 System. It has the same effect as changing /etc/issue, the Guest Customization will think it's a SUSE Linux Enterprise Server and apply all settings.

NAME="SLES"
VERSION="15-SP3"
VERSION_ID="15.3"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP3"
ID="sles"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:15:sp3"

Uyuni Test Environment with sumaform on local libvirt host (openSUSE Tumbleweed)

2022-03-08T00:00:00+01:00

A convenient way to quickly setup a local Uyuni Instance inclusive Clients, either for Development or Testing, is sumaform. It supports different backends and provides a ton of options, check it out: https://github.com/uyuni-project/sumaform

I just want to run a Uyuni Server and a few Clients (openSUSE, EL, Debian) on my Laptop so I can verify reported bugs, try some configs as well as deploy and test changes during development, without investing much time.

Following a overview of the, in my case, relevant steps, examples of my config and some Ansible snippets.

Setup

sudo zypper addrepo http://download.opensuse.org/repositories/systemsmanagement:/sumaform/openSUSE_Tumbleweed/systemsmanagement:sumaform.repo
sudo zypper install git-core
sudo zypper install --from systemsmanagement_sumaform terraform terraform-provider-libvirt
git clone https://github.com/uyuni-project/sumaform.git

Follow the First-time configuration (libvirt) instructions in the README, but use the uyuni example file instead: copy main.tf.uyuni.example to main.tf

Config & Deployment

vim main.tf

terraform init
terraform validate
terraform apply

My terraform main.cf based on the uyuni example file

terraform {
 required_version = "1.0.10"
 required_providers {
   libvirt = {
     version = "0.6.3"
   }
 }
}
provider "libvirt" {
}
module "base" {
  cc_username = ""
  cc_password = ""
  images = ["centos7", "opensuse153o", "ubuntu1804"]
}
module "server" {
  base_configuration = module.base.configuration
  product_version = "uyuni-master"
  name = "server"
  image = "opensuse153o"
  use_os_released_updates = true
  create_first_user = false
  auto_accept = false
  skip_changelog_import = false
  mgr_sync_autologin = false
  create_sample_channel = false
  create_sample_activation_key = false
  create_sample_bootstrap_script = false
  publish_private_ssl_key = false
  java_debugging = true
  provider_settings = {
    memory = 8192
    vcpu = 4
  }
}
module "redhat-minion" {
  base_configuration = module.base.configuration
  product_version = "uyuni-master"
  name = "minion-centos7"
  image = "centos7"
  server_configuration = module.server.configuration
  provider_settings = {
    memory = 1024
  }
  auto_connect_to_master = true
}
module "debian-minion" {
  base_configuration = module.base.configuration
  product_version = "uyuni-master"
  name = "minion-ubuntu1804"
  image = "ubuntu1804"
  server_configuration = module.server.configuration
  provider_settings = {
    memory = 1024
  }
  auto_connect_to_master = true

}

Ansible Snippets for the Setup steps

- name: Import systemsmanagement:/sumaform RPM Key
  rpm_key:
    key: https://download.opensuse.org/repositories/systemsmanagement:/sumaform/openSUSE_Tumbleweed/repodata/repomd.xml.key
    state: present

- name: Add systemsmanagement:/sumaform RPM Repository
  community.general.zypper_repository:
    name: systemsmanagement_sumaform
    description: Using Terraform to create a SUSE Manager test environment (openSUSE_Tumbleweed)
    repo: https://download.opensuse.org/repositories/systemsmanagement:/sumaform/openSUSE_Tumbleweed/
    priority: 90
    state: present
  ignore_errors: true

- name: Package Installation (Uyuni Development - sumaform)
  community.general.zypper:
    name:
      - terraform
      - terraform-provider-libvirt
      - git-core
    allow_vendor_change: true
    force_resolution: true
    force: true
    state: latest

Release: MoinMoin Wiki Container, powered by PyPy2

2021-10-09T00:00:00+02:00

Today I published the first version of moinmoin-pypy2-container, a ready to use image to run MoinMoin Wiki, based on openSUSE Leap.

It contains some enhancements regarding the intial wiki setup to make it easier to get started and already ships with a pre-configured nginx.

Latest stable release of MoinMoin was 8th November 2020 and requires Python 2, development of the successor moin2 is ongoing but wasn't released yet.

moin2 supports Python 3, but there is still a lot to do and the Project could use some help, what about a good first issue?

It might be that User still want to keep using the latest stable release and wait with the migration for a while.

Besides minor enhancements, the main goal of moinmoin-pypy2-container is to provide a easy to use alternative to keep using MoinMoin.

The official Python 2 Runtime is End-of-Life. But PyPy will support Python 2 "forever" and still provide updates in future.

Please check the README in the below linked Source code Repository for further details about the features and usage.

The repository is available on Codeberg (Primary), Gitlab (Mirror), Github (Mirror).

Pre-build Container Images can be downloaded from quay.io.

openSUSE Tumbleweed: Issues on resume from hibernate / suspend to disk

2021-10-03T00:00:00+02:00

So far I always used sleep when closing the lid of my Notebook, was ok even though you risk that your battery runs empty and your system shutdown. Wanted to activate hibernate which then would also support hybrid sleep, what seem to be compareable with what Apple does to achieve a massive standby and short wakeup time on the MacBook.

During the Setup I already created a Swap Partition equal to the size of my Memory, so I thought this should all work out of the box, but didn't.

Found a Post on reddit (Archive: [1]) and a related Bug Report (Archive: [1], [2]), both contained contrary information. Based on Reddit the resume module would be missing in initrd and should be included. The Bug makes clear that systemd-hibernate-resume is used and _not_ the resume module that could be added by dracut.

I trust the Bug Report more, but adding resume=/dev/mapper/system-swap (Has to be the path to your Swap Partition and might differ) to /etc/default/grub didn't solved the issue. So I generated a new initramfs by running dracut -f, without modifying any config, afterwards hibernate and hybrid sleep were working.

grep resume /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="splash=silent mitigations=auto quiet resume=/dev/mapper/system-swap"

sudo grub2-mkconfig -o /boot/efi/EFI/opensuse/grub.cfg

sudo dracut -f

No Idea why I had to rebuild the initramfs, should already be done (automatically) during the last Kernel Upgrade, anyway, I can now use hybrid sleep which looks actually very good so far. Roughtly 30 - 60 sec until sleep after closing the lid, resume just a few seconds, not really different to sleep as far I can see.

ScummVM did the trick!

2020-04-03T15:43:00+02:00

After playing around with DOSBox a while ago i gave ScummVM a try and actually it couldn't be easier. I picked Discworld II and after adding it i could just start it without changing any settings at all, impressive.

I'm quite sure it will work very well with other old Games of my collection, we will see.

Accessing SMB Share on old NAS failed from Linux

2019-12-24T13:33:00+01:00

Sometimes it can be challenging to use some old Hardware. In my case I found an old Buffalo LinkStation LS-WTGL NAS from around 2008.

Connecting from an up-to-date openSUSE System surprisingly failed with the Error Message:

Unable to initialize messaging context. protocol negotiation failed: NT_STATUS_CONNECTION_DISCONNECTED

But what's the issue? I was able to connect to my QNAP TS-253A NAS without problems. So i played around with different parameters of the smbclient cli tool, no success.

After a while the Hint in a Ask Fedora! (Archive: [1], [2]) Thread from eeijlar put me into the right Direction. The workaround, also mentioned in Red Hat Bug 1768117 (Archive: [1], [2]), solved the issue and i was able to mount my SMB Shares from the old Buffalo NAS.

eeijlar:

> Update /etc/samba/smb.conf with:
> [global]
> client min protocol = NT1

For security reasons, the SMB1 / NT1 protocol is disabled by default in Samba 4.11 and higher. But some old NAS only support these outdated protocol.

I highly suggest that you only enable it temporary and migrate your files.

No Sound in DOSBox on Linux

2019-12-06T18:53:00+01:00

I installed DOSBox to play some old DOS Games like Discworld on Linux (openSUSE Tumbleweed) but didn't had sound prior figure out the correct MIDI Port and adjusting the dosbox config accordingly.

Due to the fact that i installed Unreal Tournament from 1999 already (successful!) on my Linux Notebook, it's obvious that i had to do the same with some even older Games too :D

I thought executing "pmidi -l" would show the correct MIDI Port, but at least in my case, further steps were necessary.

The troubleshooting approach described in the AskUbuntu Forum (Archive: [1], [2]) is my favorite and did the trick.

You have to Download a test MIDI File, start a MIDI Server, start the playback and if you have audio, put the correct Port into your dosbox config.

zypper install timidity
zypper install pmidi

cd /tmp
wget http://www.angelfire.com/fl/herky/images/teddybear.mid

pmidi -l
#    Port     Client name                       Port name
#    14:0     Midi Through                      Midi Through Port-0

timidity -iA
#    Requested buffer size 32768, fragment size 8192
#    ALSA pcm 'default' set buffer size 32768, period size 8192 bytes
#    TiMidity starting in ALSA server mode
#    Opening sequencer port: 128:0 128:1 128:2 128:3

pmidi -p 128:0 teddybear.mid

In my case, pmidi identified Port "14:0" but based on the test with timidity i have to use "128:0" instead.

So i changed the corresponding parameter in my dosbox config:

[midi]
midiconfig=128:0

When you start dosbox from the console, you should see something like the following an hear the nicest MIDI sound when starting an old Game ;)

ALSA:Client initialised [128:0]
MIDI:Opened device:alsa

I will play around with ScummVM as well within the next days, so i'm quite sure further Posts regarding some sort of gaming on Linux will follow soon!

Unreal Tournament (UT99) on openSUSE Tumbleweed Linux

2019-12-05T13:38:00+01:00

To be honest, i didn't played Unreal Tournament from 1999 for years but after i found the Game CD, i just had to install and play it on my Linux Notebook :D

TL;DR

Summary how to install Unreal Tournament (Classic / 1999) on openSUSE Tumbleweed Linux:

Add games Repository and install package gtk

either via YaST or through the openSUSE Software repository

zypper addrepo https://download.opensuse.org/repositories/games/openSUSE_Tumbleweed/games.repo
zypper refresh
zypper install gtk

Install missing 32bit libraries

zypper install libX11-6-32bit
zypper install libXext6-32bit
zypper install libglvnd-32bit
zypper install Mesa-libGL1-32bit
zypper install pulseaudio-utils-32bit

Download and run the UT99 installer

Update 2021-02-26: The address used by wget below - pointing to liflg.org - isn't working anymore and redirect to Github, you can use archive.org #2 instead.

# wget https://liflg.org/?what=dl&catid=6&gameid=51&filename=unreal.tournament_436-multilanguage.run

wget https://web.archive.org/web/20180614001819if_/https://liflg.reto-schneider.ch/files/final/unreal.tournament_436-multilanguage.run
chmod +x unreal.tournament_436-multilanguage.run
./unreal.tournament_436-multilanguage.run

Enjoy UT99

padsp <InstallPath>/ut

The whole journey

I own a TUXEDO InfinityBook Pro 14 v5 (Archive: [1]) with an Intel UHD Graphics 620 Chip, running openSUSE Tumbleweed Linux, should hopefully be sufficient to play an 20 year old Game ;)

Linux Installer

The Installer can be downloaded from Linux Installers for Linux Games, i picked unreal.tournament_436-multilanguage.run and unreal.tournament.official.bonus.pack.collection.run - the other one available is for the Game of the Year (goty) Edition.

Update 2021-02-26: The above link to Linux Installers for Linux Games isn't working anymore and redirect to Github, you can use the following Backup Links instead:
- liflg.org Page: archive.org #1
- UT99 Installer: archive.org #2
- UT99 BonusPack: archive.org #3

Troubleshooting

GTK in Version 1 is required by both installer, after some research i found the package in the openSUSE Software repository.

After installing the GTK package and inserting the Game CD into my external Drive, i just started the installer, adjusted the Install Path and started the installation.

Seems just too easy!?

Yep, after starting UT, i received an error message that libX11.so.6 is missing. Looks like the 32bit version is not installed by default, zypper install libX11-6-32bit solved that problem. Same with the next missing library libXext.so.6, zypper install libXext6-32bit.

Well, next Error:

Signal: SIGIOT [iot trap]
Aborting.

Let's take a look into the log file by running cat ~/.loki/ut/System/UnrealTournament.log:

Log: binding libGL.so.1
Critical: appError called:
Critical: Could not load OpenGL library

Ok, different kind of error message but after some research same as above, there is a 32bit version of libGL.so.1 but, for sure, i only had the 64bit one installed by default. Running zypper install libglvnd-32bit solved that issue too.

Hmm but again the SIGIOT error occurred, let's take a look into the log files:

Log: OpenGL
Critical: appError called:
Critical: Couldn't set video mode: Couldn't find matching GLX visual

Lesson learned, to get a 20 year old game running on a bleeding edge Distribution in 2019, you have to install a bunch of 32bit libraries ;) This time Mesa-libGL1-32bit was missing and need to be installed.

And voila, it's starting! without sound :O ... Quite sure it's a missing 32bit lib again ... I found some hints regarding PulseAudio in the Debian Wiki (Archive: [1], [2]). So i installed pulseaudio-utils-32bit and after starting the game with padsp ut instead just ut i had sound :D

Audio Issues

To get rid of delays in the Audio Output, it was necessary to adjust one line in the config file ~/.loki/ut/System/UnrealTournament.ini as well:

# Change this:
AudioDevice=ALAudio.ALAudioSubsystem

# To that:
AudioDevice=Audio.GenericAudioSubsystem

Play Online

It looks like that lot of audio sample rates which are required to play Online are not supported using the GenericAudioSubsystem. Connection to most of the UT Server fail and you find an Error about unsupported Rate in your UnrealTournament.log file. The workaround so far was to use the ALAudioSubsystem with a delay in the Audio Output :(

Also lot Public Server require an installed NPLoader. Lot of different Version can be found at: http://utgl.unrealadmin.org/NPLoader/

You have to Download either the Linux.tar.gz or the .dll, .so as well .u file and copy all of them in your System folder of your UT Installation.

Well done

Even if i had to look around for some time to locate and fix most of the issues, install a bunch of 32bit libraries, adjust something in the config as well start the game with an extra parameter, it's worth it! Didn't know the exact FPS count but look and feel is like 20 years ago and i love it!

Btw. installing the Bonus Pack was very straight forward with the unreal.tournament.official.bonus.pack.collection.run script.

Enjoy Unreal Tournament from 1999 on your Linux Machine!

Merge PDF Files on Linux

2019-10-28T15:38:00+01:00

I had to use Cups-PDF and "Print" bunch of Pages to Single PDF Files due to the fact that no Download as PDF was possible. But how to Merge tons of files named as Pagetitle-job_id<Number> on Linux via command line in a fast and easy way?

After some testing, pdfunite - included in Poppler and already installed on openSUSE Tumbleweed - did the job, but the result with pdftk - included in Ghostscript - was nearly identical and worked the same way.

pdfunite $(ls -cr *.pdf) output.pdf

With $(), ls will executed and the results are added to the pdfunite command.

All PDF Files in the same Folder will be merged to a single output.pdf File.

The Parameter -cr sorts by creation time in reverse order.

That was necessary for me because the Job IDs were from 36 till 223, all other parameter mixed up the order and the merged PDF wasn't as expected.

Apple AirPods on Linux?

2019-10-26T12:55:00+02:00

I was wondering if my Apple AirPods would work on openSUSE Linux out of the Box. Just enable pairing mode, click on connect and all fine? Not exactly, it failed with the default settings and some additional Steps were necessary.

Set ControllerMode = dual in config File /etc/bluetooth/main.conf

On openSUSE Tumbleweed (Patch Level 10/2019) the config file doesn't seem exist by default. I copied the example of the bluez Package located in /usr/share/doc/packages/bluez/

sudo cp /usr/share/doc/packages/bluez/main.conf /etc/bluetooth/main.conf
sudo vim /etc/bluetooth/main.conf

# Restricts all controllers to the specified transport. Default value
# is "dual", i.e. both BR/EDR and LE enabled (when supported by the HW).
# Possible values: "dual", "bredr", "le"
ControllerMode = dual

sudo systemctl restart bluetooth

Add Parameter --noplugin=avrcp at the end of ExecStart in bluetooth.service, otherwise the AirPods running with poor Sound Quality and very low volume

Source: https://wiki.archlinux.org/index.php/Bluetooth_headset#Apple_Airpods_have_low_volume (Archive: [1], [2])

sudo systemctl edit --full bluetooth.service

[...]
ExecStart=/usr/lib/bluetooth/bluetoothd --noplugin=avrcp
[...]

sudo systemctl restart bluetooth

After those few adjustments they actually working quite good as Headphone. The Microphone is not working yet, maybe i will take a look into that later.