The Wombelix Post - Fedora

Giving up on Pagure - Almost 3 years of trying to keep it alive

2025-06-15T00:00:00+02:00

After almost 3 years of investing countless hours of my free time in the attempt to keep Pagure alive, I give up. I can't motivate myself anymore to continue and I simply don't have the time anymore between my job, family life and all the other responsibilities people have.

It all started in January 2023 when I wrote to the pagure-devel mailing list "Putting the Band back together - Road to Pagure v6.0" (Archive: [1], [2]). Pagure had been essentially abandoned since release 5.13.3 in 2021, with over 600 open issues and 20 pull requests piling up. My attempt to re-activate a community around it was unheard and failed, there wasn't much of a user base outside Fedora left already.

I believed strongly that we needed Pagure as a true Open Source git forge with a lightweight, extensible and well-designed codebase. So I tried to bring all Pagure enthusiasts back together, clean up the backlog, and work toward a major v6.0 release.

Over the following years, I fixed bugs, updated dependencies, got rid of some tech debt, addressed three critical CVEs that emerged, worked on the gitolite backend removal, and so much more I can't even remember anymore... Eventually I managed to cut a new release in May 2024 (Archive: [1], [2]).

The work was sometimes really fun and educational. Going deep down the rabbit hole to troubleshoot and fix problems, learning how everything fit together, reverse engineering how things were implemented back then. But it also became extremely frustrating at some point and the opposite of fun and enjoyable.

The harsh truth is that Pagure's codebase is years old and honestly, even though I've tried, it is basically all duct tape and chicken wire after years of accumulated technical debt. Being most of the time on my own, I started to question why I'm doing all this.

With the Fedora project investing since years in moving away from Pagure instead of supporting it, the writing was on the wall. The git forge evaluation in April 2024 (Archive: [1], [2]) made it clear that Pagure was no longer considered viable for Fedora's future. The decision came in December 2024 when Fedora chose Forgejo (Archive: [1], [2]).

Since May 2025 (Archive: [1], [2]), even the creation of new projects on pagure.io is disabled and it is just in keep the lights on mode without any future.

With Fedora's clear direction toward Forgejo and the platform essentially in maintenance-only mode, I just couldn't motivate myself anymore. The combination of a legacy codebase, lack of community engagement, Fedora's move away from the platform, the enormous time investment required, and three critical security vulnerabilities in the past year made it clear that continuing would be unproductive.

Despite all the frustrations, I don't regret the journey. It was sometimes educational and I learned a lot about maintaining legacy open source projects and the importance of community. I'm grateful for the experience and the few community members who did engage and support the effort.

Sometimes projects have their time, and that time comes to an end. I guess the future of git forges lies with more modern solutions like Forgejo.

Thank you to everyone who supported Pagure over the years. It was a good run, but all good things must come to an end.

I'm curious what Open Source project is going to draw my attention next.

Fedora / EPEL updates for aws-c-io and aws-c-http, new package aws-c-s3

2024-12-30T00:00:00+01:00

As mentioned in my last Fedora / EPEL packaging post, updats for aws-c-io and aws-c-http are coming. They now arrived in Rawhide and all Stable Fedora branches as well as EPEL8 and EPEL9.

Since today, the new package aws-c-s3 is available in Fedora and EPEL too.

I plan to package aws-c-iot, aws-crt-ffi and aws-lc for Fedora and EPEL in the next couple of months.

In addition, I will build all current EPEL9 packages I maintain soon for EPEL10 as well, stay tuned.

Fedora and EPEL updates for various AWS C lib packages

2024-11-25T00:00:00+01:00

Among the Fedora and EPEL packages I maintain are 11 AWS C lib packages. I wrote some time ago about there function and challenge to package them. Today 9 of those packages received updates to the latest available release. 2 are pending because of dependencies, but I should get them through the door within the next 2-3 weeks as well.

The majority went smoothly, no breaking changes, minor bug fixes, sometimes a patch that had to be updated. But aws-c-cal required a bit more work. Whenever tests are available, I run them as part of the package build. And they started to fail when I wanted to update the package:

98% tests passed, 3 tests failed out of 137
Total Test time (real) =   3.58 sec
The following tests FAILED:
    66 - rsa_signing_roundtrip_pkcs1_sha1_from_user (Failed)
    71 - rsa_verify_signing_pkcs1_sha1 (Failed)
    77 - rsa_signing_mismatch_pkcs1_sha1 (Failed)
Errors while running CTest

The sha1 in the test name lead me pretty fast to the problem. In version 0.8.1 there was some SHA1 related code and tests added:

https://github.com/awslabs/aws-c-cal/releases/tag/v0.8.1

https://github.com/awslabs/aws-c-cal/pull/201

I can only assume it's for backward compatibility reasons. The thing is, SHA1 is distrusted in Fedora 41 (Archive: [1], [2]) and RHEL 9 (Archive: [1], [2]).

The code provides additional functionality and will not be called on systems that don't use SHA1 anymore. So the fastest way forward was to patch out the three failing tests and call it a day.

diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt
index 346e38a..e3966cb 100644
--- a/tests/CMakeLists.txt
+++ b/tests/CMakeLists.txt
@@ -77,18 +77,15 @@ add_test_case(rsa_encryption_roundtrip_oaep_sha256_from_user)
add_test_case(rsa_encryption_roundtrip_oaep_sha512_from_user)
add_test_case(rsa_signing_roundtrip_pkcs1_sha256_from_user)
add_test_case(rsa_signing_roundtrip_pss_sha256_from_user)
-add_test_case(rsa_signing_roundtrip_pkcs1_sha1_from_user)
add_test_case(rsa_getters)
add_test_case(rsa_private_pkcs1_der_parsing)
add_test_case(rsa_public_pkcs1_der_parsing)
add_test_case(rsa_verify_signing_pkcs1_sha256)
-add_test_case(rsa_verify_signing_pkcs1_sha1)
add_test_case(rsa_verify_signing_pss_sha256)
add_test_case(rsa_decrypt_pkcs1)
add_test_case(rsa_decrypt_oaep256)
add_test_case(rsa_decrypt_oaep512)
add_test_case(rsa_signing_mismatch_pkcs1_sha256)
-add_test_case(rsa_signing_mismatch_pkcs1_sha1)

add_test_case(aes_cbc_NIST_CBCGFSbox256_case_1)
add_test_case(aes_cbc_NIST_CBCVarKey256_case_254)

An Overview of all updates today, available in Rawhide (F42) latest tomorrow. Around one week till they arrive in all stable Fedora and EPEL branches.

aws-c-cal, 0.7.4 to 0.8.1

aws-c-mqtt, 0.10.6 to 0.11.0

aws-checksums, 0.1.20 to 0.2.2

aws-c-auth, 0.7.31 to 0.8.0

s2n-tls, 1.5.3 to 1.5.9

aws-c-sdkutils 0.1.19 to 0.2.1

aws-c-event-stream 0.4.3 to 0.5.0

aws-c-compression 0.2.19 to 0.3.0

aws-c-common 0.9.28 to 0.10.3

Pending are the updates for aws-c-io and aws-c-http. For Rawhide that's a matter of a couple days. Stable Fedora and EPEL branches going to take a bit longer, probably 2-3 weeks.

AWS C libraries, the unknown heroes behind AWS tools and the adventure to package them

2024-07-10T00:00:00+02:00

It all started with an orphaned Fedora package called aws-php-sdk3. I was interested to adopt and update it. I learned quickly that I have to take a look in the different dependencies too. I opened a can of worms, there were a few obvious dependencies and then some more hidden.

Like aws-php-crt, which isn't packaged for Fedora yet and upstream builds it in a way that it bundles a couple of AWS C libraries. As I learned in the meantime, bundling libraries or modules has to be avoided whenever possible based on the Fedora and EPEL packaging guidelines. You might get an exception but you need a good reason for it.

So I dig deeper and learned that there were some activities two years ago but except one of them, nothing went through and made it into Fedora. I spend a couple of days to understand what dependencies each library has. In which order do I need to build and package them to make them work. What was the state two years ago, what can be re-used.

After this exercise I had a full plan written down and started to create the packages in a copr project. That way I can satisfy the dependencies during development. Even if the packages are not in Fedora yet.

I learned that architecture s390x isn't supported by upstream (Archive: [1], [2]) and there is also no plan to change that. So this means every package needs an Arch Exception. In addition to that some packages had unit tests that require an internet connection. That's not given when Fedora packages are build, so I had to create patches so disable this kind of tests. Even if this means that the test coverage drops slightly. It's still better to run the majority of tests during package build to discover issues, instead of deactivating all of them.

With s2n-tls I hit an interesting problem that the tests were failing only on RHEL 9. Building on different Fedora versions and RHEL 8 was working without a problem. I reported the bug upstream (Archive: [1], [2]) and was at first unsure what makes RHEL 9 unique and causes the problems. After a bit of thinking and some feedback from the maintainers, I remembered that Red Hat announced a while back to deprecate SHA1 in their openSSL package (Archive: [1], [2]). So I did some research and came pretty close but couldn't track it down in the s2n-tls source. A maintainer provided a patch to verify my assumption. And indeed, the problem is that they use SHA1 certs in the unit tests. I have to admit it made me proud to help to identify this problem. I think they going to provide a fix in an upcoming release and then I can continue to package it for EPEL 9 :)

Right now I'm at roughtly 1/3, I packaged:

and they are at least available in Fedora rawhide. Most of them are already in all stable Fedora and EPEL branches.

Publishing happens with some delay because a new package needs ~7-8 days to make it into the stable repository. Which means, you start with one package were all other depend on. You go through the package review process, submit it, wait for a week. If you have other packages ready now. You do the same, you submit and wait. And so on and so on. Which means it takes overall around two to three months, depending on how fast the package reviews went through, till all AWS C libs are available.

Yes, there is the concept of Karma, but let's assume you not always find people that can invest the time. So the above assumption is based on the "worst case".

As you see, I was dragged away from my initial goal, packaging and keeping aws-php-sdk3 alive, into a massive amount of groundwork. But as soon all AWS C libs are packaged and available, it opens the door for a lot of other AWS tools to be properly packaged without bundled libraries.

I'm really looking forward to that!

Becoming a Fedora Packager

2024-05-31T00:00:00+02:00

I recently decided to finally become an active package maintainer. Quickly I had to realize that there are different ways to become a Fedora Packager. All have in common that, what a surprise, you maintain or co-maintain a package.

But, from all options, the most "classic" way seem to:

Create a package
Submit a Review Request
When it's approved ask for a Sponsor
Get sponsored into the packager group
Publish your first package

The Fedora project has tons of documentation. I found it on the one hand great as a greenhorn to find all these information. And at the same time it was overwhelming and sometimes unclear which is current.

So I put together a short list with resources that helped me most. They are not necessary in a logical order. They will also not cover each and every question you might have. But you can use them as a starting point.

What if you still have question?

Don't be shy, write to the devel mailing list and or join the Fedora Devel Matrix room!

After many many years in the industry, a lot of touch points and contributions to open source projects, I finally made the jump to become a package maintainer.

I learned so much in the last month and went from zero to ~15 Fedora and EPEL packages I became responsible for. This feels like a great success and a valuable way to give something back to the community.

My first approved Fedora Package, yippie! ec2-instance-connect

2024-05-19T00:00:00+02:00

A while ago I was asked if I want to package ec2-instance-connect for Fedora and eventually EPEL. More specific "with Packit", which did send me down a weird path as I explain in more detail in Fedoda dist-git packit onboarding <{filename}/posts/2024/fedora-dist-git-packit-onboarding_en.rst>. After learning what Packit can and can't do for me, I started to make good progress ;)

I spend quite some time to learn about Fedora Packaging. The does and don't when writing spec files. How package testing, reviewing and publishing works. More about this in a later Blog, now I focus on my first approved package.

I was lucky that my Reviewer was Neal Gompa. People describe Neal with "he is just everywhere". And that's true in the most positive way. It's nearly impossible to be active in the open source world without crossing paths :) He has a lot of experience and is a great mentor. Receiving feedback from him is always a great opportunity to learn.

First I thought ec2-instance-connect will be an easy package, great for the first one. Technically it's just a handful of files and a systemd unit. How hard can that be? Spoiler: Very hard.

The challenge is the way how ec2-instance-connect works. It adjusts the authcommand from sshd so that, by default, all authentication attempts go through it. This is fine for brand new systems but becomes a problem when you deploy on existing systems with a already customized config. Or if someone wants to apply a custom config after the initial deployment and ec2-instance-connect installation.

So there are a lot of ways to break ssh login to the system which is discussed on GitHub. But keeping this fact aside, there is obviously demand for a Fedora and EPEL package. So I was encouraged to improve the user experience and make it available :)

Upstream has a generic spec file so this became my starting point. But I had to learn quickly that shell snippets and nested if/else statements are not what is expected from a high quality spec file in Fedora. So I had to find a way to replace the pretty unique logic that was implemented with rpm macros and in a way that aligns with Fedora packaging guidelines. The result is a good compromise, not perfect but it gives users flexibility and reduces the risk of problems.

So after a couple iterations and very valuable feedback from Neal, he approved my request and I was good to bring my first package into Fedora :) If you are interested in the details, feel free to take a look at the Fedora Review Request ticket.

In the meantime the package is available in all Fedora and EPEL repositories. It is also on it's way to be pre-installed in Fedora Cloud images in future :D

It was an awesome experience and I can't wait to work on more packages!

Fedora dist-git Packit onboarding

2024-05-05T00:00:00+02:00

Packit, oh my god, that's a tool and service that gave me a pretty hard time to understand how it works. Not necessarily because it's a complicated tool. But it expects a good portion of background knowledge how things work in Fedora and the specific wording. If you are new and make your first steps, it becomes very challenging and frustrating to get started.

So first, what's Packit? Very simplified: Packit get triggered when a new version of a software is released. It can then trigger package builds. Or create PRs in Fedora packages to update them. The main goal is to reduce the work maintainer have to keep Fedora packages up to date.

My first false assumption was that Packit always requires to have a configuration in the repository of the upstream project. Another mistake was that Packit will help me to test and build packages during development. You can probably imagine that those two misunderstandings lead to a good portion of confusion and wasted time.

So how to benefit from Packit if you don't control the upstream project and they are not interested to onboard their repository to the Packit Service? Fedora runs Release Monitoring a service that checks for new releases in configured projects on a regular basis. If there is one, it will publish a message into Fedora Messaging. A RabbitMQ based messaging service that allows other services to react on events. Packit will pick up such an event and check if there is a packit configuration file in the dist-git package repo. If that's the case, Packit will execute the configured actions.

So for example, Packit creates a PR in the package repo. The PR bumps the package version and adds a changelog entry. Package maintainer can add additional changes to the PR if necessary and upload the new source file into the lookaside cache.

Then your job boils down to merge the PR into the rawhide branch. If correctly configured, Packit will pick up this event, remember everything in the Fedora world becomes an event in the Fedora Messaging bus, and trigger the build. If that build is successful the update is triggered and the package becomes available in rawhide. Next, if you have other branches you want to make the package available, you merge it and then again Packit picks up the event, starts a build and the update.

This is a massive time saver already, even if you still have a couple of steps to do. That's the workflow I prefer, but you can of course adjust it and let Packit do even more in an automated way. The decision is up to you :)

What I explained here is just one part of what Packit can do. I encourage you to explore the project and documentation to dive deeper.

For me it was a massive "AHA" moment when I had my first Packit config working and saw the magic happen :)

Following the .packit.yaml file I prepared for the first Fedora package I'm going to release:

# See the documentation for more information:
# https://packit.dev/docs/configuration/

upstream_project_url: https://github.com/aws/aws-ec2-instance-connect-config
upstream_package_name: aws-ec2-instance-connect-config
downstream_package_name: ec2-instance-connect

jobs:
- job: pull_from_upstream
    trigger: release
    # Keeping dist-git branches non-divergent
    # Requirs manual local merge from rawhide to stable release branches
    # https://packit.dev/docs/fedora-releases-guide#keeping-dist-git-branches-non-divergent
    dist_git_branches:
    - fedora-rawhide

- job: koji_build
    trigger: commit
    allowed_pr_authors:
    - packit
    - all_admins
    - all_committers
    - '@cloud-sig' # string with @ needs quotes to be valid yaml
    allowed_committers:
    - all_admins
    - all_committers
    - '@cloud-sig' # string with @ needs quotes to be valid yaml
    dist_git_branches:
    - fedora-all
    - epel-all

- job: bodhi_update
    trigger: commit
    allowed_builders:
    - packit
    - all_admins
    - all_committers
    - '@cloud-sig' # string with @ needs quotes to be valid yaml
    dist_git_branches:
    - fedora-branched # rawhide updates are created automatically
    - epel-all

Adopting orphaned Fedora packages or how one thing leads to another

2024-05-01T00:00:00+02:00

Recently I started to become an active Fedora packager, which is something I really enjoy. It happens, for various reasons, that a package gets orphaned by its maintainer. Lack of time, changes in upstream that make it hard to package a new version, or incompatibilities and no updates. In such a case, packages can be adopted by other packagers.

I stumbled across php-aws-sdk3 and even though I'm not actively using the SDK right now, I found it appealing to maintain this package. Might be a good opportunity to understand PHP packaging for Fedora better and brush up my old PHP knowledge a bit. And then I started to realize how fast one package can become many ;)

The magic word: Dependencies. Of course, php-aws-sdk3 depends on other packages. Some of them are orphaned to. Some need and update to be compatible. Others don't exist yet in Fedora. I ended up adopting three other packages and with ~14 new packages on my list :O

So it will take me some time to reach my goal, to keep php-aws-sdk3 alive. But I will learn a lot on that journey and build some cool and valuable packages.