The Wombelix Post - Packages

Fedora / EPEL updates for aws-c-io and aws-c-http, new package aws-c-s3

2024-12-30T00:00:00+01:00

As mentioned in my last Fedora / EPEL packaging post, updats for aws-c-io and aws-c-http are coming. They now arrived in Rawhide and all Stable Fedora branches as well as EPEL8 and EPEL9.

Since today, the new package aws-c-s3 is available in Fedora and EPEL too.

I plan to package aws-c-iot, aws-crt-ffi and aws-lc for Fedora and EPEL in the next couple of months.

In addition, I will build all current EPEL9 packages I maintain soon for EPEL10 as well, stay tuned.

Fedora and EPEL updates for various AWS C lib packages

2024-11-25T00:00:00+01:00

Among the Fedora and EPEL packages I maintain are 11 AWS C lib packages. I wrote some time ago about there function and challenge to package them. Today 9 of those packages received updates to the latest available release. 2 are pending because of dependencies, but I should get them through the door within the next 2-3 weeks as well.

The majority went smoothly, no breaking changes, minor bug fixes, sometimes a patch that had to be updated. But aws-c-cal required a bit more work. Whenever tests are available, I run them as part of the package build. And they started to fail when I wanted to update the package:

98% tests passed, 3 tests failed out of 137
Total Test time (real) =   3.58 sec
The following tests FAILED:
    66 - rsa_signing_roundtrip_pkcs1_sha1_from_user (Failed)
    71 - rsa_verify_signing_pkcs1_sha1 (Failed)
    77 - rsa_signing_mismatch_pkcs1_sha1 (Failed)
Errors while running CTest

The sha1 in the test name lead me pretty fast to the problem. In version 0.8.1 there was some SHA1 related code and tests added:

https://github.com/awslabs/aws-c-cal/releases/tag/v0.8.1

https://github.com/awslabs/aws-c-cal/pull/201

I can only assume it's for backward compatibility reasons. The thing is, SHA1 is distrusted in Fedora 41 (Archive: [1], [2]) and RHEL 9 (Archive: [1], [2]).

The code provides additional functionality and will not be called on systems that don't use SHA1 anymore. So the fastest way forward was to patch out the three failing tests and call it a day.

diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt
index 346e38a..e3966cb 100644
--- a/tests/CMakeLists.txt
+++ b/tests/CMakeLists.txt
@@ -77,18 +77,15 @@ add_test_case(rsa_encryption_roundtrip_oaep_sha256_from_user)
add_test_case(rsa_encryption_roundtrip_oaep_sha512_from_user)
add_test_case(rsa_signing_roundtrip_pkcs1_sha256_from_user)
add_test_case(rsa_signing_roundtrip_pss_sha256_from_user)
-add_test_case(rsa_signing_roundtrip_pkcs1_sha1_from_user)
add_test_case(rsa_getters)
add_test_case(rsa_private_pkcs1_der_parsing)
add_test_case(rsa_public_pkcs1_der_parsing)
add_test_case(rsa_verify_signing_pkcs1_sha256)
-add_test_case(rsa_verify_signing_pkcs1_sha1)
add_test_case(rsa_verify_signing_pss_sha256)
add_test_case(rsa_decrypt_pkcs1)
add_test_case(rsa_decrypt_oaep256)
add_test_case(rsa_decrypt_oaep512)
add_test_case(rsa_signing_mismatch_pkcs1_sha256)
-add_test_case(rsa_signing_mismatch_pkcs1_sha1)

add_test_case(aes_cbc_NIST_CBCGFSbox256_case_1)
add_test_case(aes_cbc_NIST_CBCVarKey256_case_254)

An Overview of all updates today, available in Rawhide (F42) latest tomorrow. Around one week till they arrive in all stable Fedora and EPEL branches.

aws-c-cal, 0.7.4 to 0.8.1

aws-c-mqtt, 0.10.6 to 0.11.0

aws-checksums, 0.1.20 to 0.2.2

aws-c-auth, 0.7.31 to 0.8.0

s2n-tls, 1.5.3 to 1.5.9

aws-c-sdkutils 0.1.19 to 0.2.1

aws-c-event-stream 0.4.3 to 0.5.0

aws-c-compression 0.2.19 to 0.3.0

aws-c-common 0.9.28 to 0.10.3

Pending are the updates for aws-c-io and aws-c-http. For Rawhide that's a matter of a couple days. Stable Fedora and EPEL branches going to take a bit longer, probably 2-3 weeks.

AWS C libraries, the unknown heroes behind AWS tools and the adventure to package them

2024-07-10T00:00:00+02:00

It all started with an orphaned Fedora package called aws-php-sdk3. I was interested to adopt and update it. I learned quickly that I have to take a look in the different dependencies too. I opened a can of worms, there were a few obvious dependencies and then some more hidden.

Like aws-php-crt, which isn't packaged for Fedora yet and upstream builds it in a way that it bundles a couple of AWS C libraries. As I learned in the meantime, bundling libraries or modules has to be avoided whenever possible based on the Fedora and EPEL packaging guidelines. You might get an exception but you need a good reason for it.

So I dig deeper and learned that there were some activities two years ago but except one of them, nothing went through and made it into Fedora. I spend a couple of days to understand what dependencies each library has. In which order do I need to build and package them to make them work. What was the state two years ago, what can be re-used.

After this exercise I had a full plan written down and started to create the packages in a copr project. That way I can satisfy the dependencies during development. Even if the packages are not in Fedora yet.

I learned that architecture s390x isn't supported by upstream (Archive: [1], [2]) and there is also no plan to change that. So this means every package needs an Arch Exception. In addition to that some packages had unit tests that require an internet connection. That's not given when Fedora packages are build, so I had to create patches so disable this kind of tests. Even if this means that the test coverage drops slightly. It's still better to run the majority of tests during package build to discover issues, instead of deactivating all of them.

With s2n-tls I hit an interesting problem that the tests were failing only on RHEL 9. Building on different Fedora versions and RHEL 8 was working without a problem. I reported the bug upstream (Archive: [1], [2]) and was at first unsure what makes RHEL 9 unique and causes the problems. After a bit of thinking and some feedback from the maintainers, I remembered that Red Hat announced a while back to deprecate SHA1 in their openSSL package (Archive: [1], [2]). So I did some research and came pretty close but couldn't track it down in the s2n-tls source. A maintainer provided a patch to verify my assumption. And indeed, the problem is that they use SHA1 certs in the unit tests. I have to admit it made me proud to help to identify this problem. I think they going to provide a fix in an upcoming release and then I can continue to package it for EPEL 9 :)

Right now I'm at roughtly 1/3, I packaged:

and they are at least available in Fedora rawhide. Most of them are already in all stable Fedora and EPEL branches.

Publishing happens with some delay because a new package needs ~7-8 days to make it into the stable repository. Which means, you start with one package were all other depend on. You go through the package review process, submit it, wait for a week. If you have other packages ready now. You do the same, you submit and wait. And so on and so on. Which means it takes overall around two to three months, depending on how fast the package reviews went through, till all AWS C libs are available.

Yes, there is the concept of Karma, but let's assume you not always find people that can invest the time. So the above assumption is based on the "worst case".

As you see, I was dragged away from my initial goal, packaging and keeping aws-php-sdk3 alive, into a massive amount of groundwork. But as soon all AWS C libs are packaged and available, it opens the door for a lot of other AWS tools to be properly packaged without bundled libraries.

I'm really looking forward to that!

My first approved Fedora Package, yippie! ec2-instance-connect

2024-05-19T00:00:00+02:00

A while ago I was asked if I want to package ec2-instance-connect for Fedora and eventually EPEL. More specific "with Packit", which did send me down a weird path as I explain in more detail in Fedoda dist-git packit onboarding <{filename}/posts/2024/fedora-dist-git-packit-onboarding_en.rst>. After learning what Packit can and can't do for me, I started to make good progress ;)

I spend quite some time to learn about Fedora Packaging. The does and don't when writing spec files. How package testing, reviewing and publishing works. More about this in a later Blog, now I focus on my first approved package.

I was lucky that my Reviewer was Neal Gompa. People describe Neal with "he is just everywhere". And that's true in the most positive way. It's nearly impossible to be active in the open source world without crossing paths :) He has a lot of experience and is a great mentor. Receiving feedback from him is always a great opportunity to learn.

First I thought ec2-instance-connect will be an easy package, great for the first one. Technically it's just a handful of files and a systemd unit. How hard can that be? Spoiler: Very hard.

The challenge is the way how ec2-instance-connect works. It adjusts the authcommand from sshd so that, by default, all authentication attempts go through it. This is fine for brand new systems but becomes a problem when you deploy on existing systems with a already customized config. Or if someone wants to apply a custom config after the initial deployment and ec2-instance-connect installation.

So there are a lot of ways to break ssh login to the system which is discussed on GitHub. But keeping this fact aside, there is obviously demand for a Fedora and EPEL package. So I was encouraged to improve the user experience and make it available :)

Upstream has a generic spec file so this became my starting point. But I had to learn quickly that shell snippets and nested if/else statements are not what is expected from a high quality spec file in Fedora. So I had to find a way to replace the pretty unique logic that was implemented with rpm macros and in a way that aligns with Fedora packaging guidelines. The result is a good compromise, not perfect but it gives users flexibility and reduces the risk of problems.

So after a couple iterations and very valuable feedback from Neal, he approved my request and I was good to bring my first package into Fedora :) If you are interested in the details, feel free to take a look at the Fedora Review Request ticket.

In the meantime the package is available in all Fedora and EPEL repositories. It is also on it's way to be pre-installed in Fedora Cloud images in future :D

It was an awesome experience and I can't wait to work on more packages!

Fedora dist-git Packit onboarding

2024-05-05T00:00:00+02:00

Packit, oh my god, that's a tool and service that gave me a pretty hard time to understand how it works. Not necessarily because it's a complicated tool. But it expects a good portion of background knowledge how things work in Fedora and the specific wording. If you are new and make your first steps, it becomes very challenging and frustrating to get started.

So first, what's Packit? Very simplified: Packit get triggered when a new version of a software is released. It can then trigger package builds. Or create PRs in Fedora packages to update them. The main goal is to reduce the work maintainer have to keep Fedora packages up to date.

My first false assumption was that Packit always requires to have a configuration in the repository of the upstream project. Another mistake was that Packit will help me to test and build packages during development. You can probably imagine that those two misunderstandings lead to a good portion of confusion and wasted time.

So how to benefit from Packit if you don't control the upstream project and they are not interested to onboard their repository to the Packit Service? Fedora runs Release Monitoring a service that checks for new releases in configured projects on a regular basis. If there is one, it will publish a message into Fedora Messaging. A RabbitMQ based messaging service that allows other services to react on events. Packit will pick up such an event and check if there is a packit configuration file in the dist-git package repo. If that's the case, Packit will execute the configured actions.

So for example, Packit creates a PR in the package repo. The PR bumps the package version and adds a changelog entry. Package maintainer can add additional changes to the PR if necessary and upload the new source file into the lookaside cache.

Then your job boils down to merge the PR into the rawhide branch. If correctly configured, Packit will pick up this event, remember everything in the Fedora world becomes an event in the Fedora Messaging bus, and trigger the build. If that build is successful the update is triggered and the package becomes available in rawhide. Next, if you have other branches you want to make the package available, you merge it and then again Packit picks up the event, starts a build and the update.

This is a massive time saver already, even if you still have a couple of steps to do. That's the workflow I prefer, but you can of course adjust it and let Packit do even more in an automated way. The decision is up to you :)

What I explained here is just one part of what Packit can do. I encourage you to explore the project and documentation to dive deeper.

For me it was a massive "AHA" moment when I had my first Packit config working and saw the magic happen :)

Following the .packit.yaml file I prepared for the first Fedora package I'm going to release:

# See the documentation for more information:
# https://packit.dev/docs/configuration/

upstream_project_url: https://github.com/aws/aws-ec2-instance-connect-config
upstream_package_name: aws-ec2-instance-connect-config
downstream_package_name: ec2-instance-connect

jobs:
- job: pull_from_upstream
    trigger: release
    # Keeping dist-git branches non-divergent
    # Requirs manual local merge from rawhide to stable release branches
    # https://packit.dev/docs/fedora-releases-guide#keeping-dist-git-branches-non-divergent
    dist_git_branches:
    - fedora-rawhide

- job: koji_build
    trigger: commit
    allowed_pr_authors:
    - packit
    - all_admins
    - all_committers
    - '@cloud-sig' # string with @ needs quotes to be valid yaml
    allowed_committers:
    - all_admins
    - all_committers
    - '@cloud-sig' # string with @ needs quotes to be valid yaml
    dist_git_branches:
    - fedora-all
    - epel-all

- job: bodhi_update
    trigger: commit
    allowed_builders:
    - packit
    - all_admins
    - all_committers
    - '@cloud-sig' # string with @ needs quotes to be valid yaml
    dist_git_branches:
    - fedora-branched # rawhide updates are created automatically
    - epel-all