Ansible Hackathon 2021 Contribution: Improved idempotency - zypper_repository

2021-10-01T00:00:00+02:00

Parallel to the Ansible Contributor Summit 2021.09 and the Ansible Fest 2021 there was a Hackathon with focus on first time Contributor. Even though I use Ansible since years, but mainly from a User perspective (Playbooks, Roles, Collections), I wasn't officially Contributing to a Plugin or Module yet.

I did some smaller bugfixing or improvements in the past, but mainly for myself or a very unique use case at work. So far I never worked on one of the GitHub Issues or created a PR for any of the community modules, something I always wanted to change.

As already mentioned a few days ago, I started putting all my configs in Ansible Playbook(s). I'm using openSUSE Tumbleweed with some additional repositories and wanted to managed them also via Ansible. I prefer .repo files, they are very convinient, you don't have to manage anything manually. There is a Module available (community.general.zypper_repository) but there was a bug regarding the idempotency when using .repo files.

You could add a new repository by using a .repo file but then every future run will fail, also removing a repository the same way wasn't working either.

So I decided to work on this Issue during the Hackathon and get some Hands-On Experience contributing to Ansible Collections.

Due to the fact that you need at least a name and baseurl to identify if something need to be added, changed or removed, I decided to download the .repo file, parse it and work with the actual values. The previous Implementation was calling zypper more or less directly with the URL to the file as Parameter, which caused the reported issues.

Now the community.general.zypper_repository module supports .repo files via URL or from a local path, behave as expected in regards to idempotency and is fully covered by integration tests.

There were always Ansible Community.General Maintainer available via Matrix to answer questions, pointing into the right direction and assisting in case of any problems.

It was a great experience for me! I can just recommend such Events, I've learned a lot during these two days and finished my Pull Request, which also fixes a second issue.

All CI tests are finally passed, feedback from the first Reviewer is positive, fingers crossed everything will be fine and the PR merged soon :)

I will write a little more about the Summit, my expiriences and things learned or found useful, to get started contributing, within the next days in a separate post.

Even though you can find all details in my Pull Request on GitHub, I don't want to force navigating to a different site, so let me share the diff (at time of writing) here as well:

diff --git a/changelogs/fragments/3474-zypper_repository_improve_repo_file_idempotency.yml b/changelogs/fragments/3474-zypper_repository_improve_repo_file_idempotency.yml
new file mode 100644
index 00000000000..4f3b56329cb
--- /dev/null
+++ b/changelogs/fragments/3474-zypper_repository_improve_repo_file_idempotency.yml
@@ -0,0 +1,7 @@
+bugfixes:
+  - zypper_repository - when an URL to a .repo file was provided in option
+    ``repo=`` and ``state=present`` only the first run was successful,
+    future runs failed due to missing checks prior starting zypper.
+    Usage of ``state=absent`` in combination with a .repo file was not
+    working either (https://github.com/ansible-collections/community.general/issues/1791,
+    https://github.com/ansible-collections/community.general/issues/3466).

diff --git a/plugins/modules/packaging/os/zypper_repository.py b/plugins/modules/packaging/os/zypper_repository.py
index 38aeab618e8..a29650a17ef 100644
--- a/plugins/modules/packaging/os/zypper_repository.py
+++ b/plugins/modules/packaging/os/zypper_repository.py
@@ -137,6 +137,10 @@

 from ansible.module_utils.basic import AnsibleModule, missing_required_lib

+from ansible.module_utils.urls import fetch_url
+from ansible.module_utils.common.text.converters import to_text
+from ansible.module_utils.six.moves import configparser, StringIO
+from io import open

 REPO_OPTS = ['alias', 'name', 'priority', 'enabled', 'autorefresh', 'gpgcheck']

@@ -382,12 +386,62 @@ def exit_unchanged():
         if not alias and state == "present":
             module.fail_json(msg='Name required when adding non-repo files.')

+    # Download / Open and parse .repo file to ensure idempotency
+    if repo and repo.endswith('.repo'):
+        if repo.startswith(('http://', 'https://')):
+            response, info = fetch_url(module=module, url=repo, force=True)
+            if not response or info['status'] != 200:
+                module.fail_json(msg='Error downloading .repo file from provided URL')
+            repofile_text = to_text(response.read(), errors='surrogate_or_strict')
+        else:
+            try:
+                with open(repo, encoding='utf-8') as file:
+                    repofile_text = file.read()
+            except IOError:
+                module.fail_json(msg='Error opening .repo file from provided path')
+
+        repofile = configparser.ConfigParser()
+        try:
+            repofile.readfp(StringIO(repofile_text))
+        except configparser.Error:
+            module.fail_json(msg='Invalid format, .repo file could not be parsed')
+
+        # No support for .repo file with zero or more than one repository
+        if len(repofile.sections()) != 1:
+            err = "Invalid format, .repo file contains %s repositories, expected 1" % len(repofile.sections())
+            module.fail_json(msg=err)
+
+        section = repofile.sections()[0]
+        repofile_items = dict(repofile.items(section))
+        # Only proceed if at least baseurl is available
+        if 'baseurl' not in repofile_items:
+            module.fail_json(msg='No baseurl found in .repo file')
+
+        # Set alias (name) and url based on values from .repo file
+        alias = section
+        repodata['alias'] = section
+        repodata['url'] = repofile_items['baseurl']
+
+        # If gpgkey is part of the .repo file, auto import key
+        if 'gpgkey' in repofile_items:
+            auto_import_keys = True
+
+        # Map additional values, if available
+        if 'name' in repofile_items:
+            repodata['name'] = repofile_items['name']
+        if 'enabled' in repofile_items:
+            repodata['enabled'] = repofile_items['enabled']
+        if 'autorefresh' in repofile_items:
+            repodata['autorefresh'] = repofile_items['autorefresh']
+        if 'gpgcheck' in repofile_items:
+            repodata['gpgcheck'] = repofile_items['gpgcheck']
+
     exists, mod, old_repos = repo_exists(module, repodata, overwrite_multiple)

-    if repo:
-        shortname = repo
-    else:
+    if alias:
         shortname = alias
+    else:
+        shortname = repo

     if state == 'present':
         if exists and not mod:

diff --git a/tests/integration/targets/zypper_repository/files/systemsmanagement_Uyuni_Utils.repo b/tests/integration/targets/zypper_repository/files/systemsmanagement_Uyuni_Utils.repo
new file mode 100644
index 00000000000..1df76802a70
--- /dev/null
+++ b/tests/integration/targets/zypper_repository/files/systemsmanagement_Uyuni_Utils.repo
@@ -0,0 +1,7 @@
+[systemsmanagement_Uyuni_Utils]
+name=Several utilities to develop, build or release Uyuni (openSUSE_Leap_15.3)
+type=rpm-md
+baseurl=https://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Utils/openSUSE_Leap_15.3/
+gpgcheck=1
+gpgkey=https://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Utils/openSUSE_Leap_15.3/repodata/repomd.xml.key
+enabled=1

diff --git a/tests/integration/targets/zypper_repository/meta/main.yml b/tests/integration/targets/zypper_repository/meta/main.yml
new file mode 100644
index 00000000000..1810d4bec98
--- /dev/null
+++ b/tests/integration/targets/zypper_repository/meta/main.yml
@@ -0,0 +1,2 @@
+dependencies:
+  - setup_remote_tmp_dir

diff --git a/tests/integration/targets/zypper_repository/tasks/test.yml b/tests/integration/targets/zypper_repository/tasks/test.yml
index e2b2f8473b6..1033ee1e7d5 100644
--- a/tests/integration/targets/zypper_repository/tasks/test.yml
+++ b/tests/integration/targets/zypper_repository/tasks/test.yml
@@ -19,6 +19,8 @@
         - testrefresh
         - testprio
         - Apache_PHP_Modules
+        - systemsmanagement_Uyuni_Stable
+        - systemsmanagement_Uyuni_Utils

     - name: collect repo configuration after test
       shell: "grep . /etc/zypp/repos.d/*"

diff --git a/tests/integration/targets/zypper_repository/tasks/zypper_repository.yml b/tests/integration/targets/zypper_repository/tasks/zypper_repository.yml
index 4490ddca7db..dbd9bb0064b 100644
--- a/tests/integration/targets/zypper_repository/tasks/zypper_repository.yml
+++ b/tests/integration/targets/zypper_repository/tasks/zypper_repository.yml
@@ -4,6 +4,11 @@
     state: absent
   register: zypper_result

+- name: verify no change on test repo deletion
+  assert:
+    that:
+      - "not zypper_result.changed"
+
 - name: Add test repo
   community.general.zypper_repository:
     name: test
@@ -51,7 +56,8 @@
   command: zypper -x lr testrefresh
   register: zypper_result

-- assert:
+- name: verify autorefresh option set properly
+  assert:
     that:
       - '"autorefresh=\"0\"" in zypper_result.stdout'

@@ -66,7 +72,8 @@
   command: zypper -x lr testprio
   register: zypper_result

-- assert:
+- name: verify priority option set properly
+  assert:
     that:
       - '"priority=\"55\"" in zypper_result.stdout'

@@ -88,7 +95,8 @@
   command: zypper lr chrome2
   register: zypper_result2

-- assert:
+- name: ensure same url cause update of existing repo even if name differ
+  assert:
     that:
       - "zypper_result1.rc != 0"
       - "'not found' in zypper_result1.stderr"
@@ -108,7 +116,8 @@
   command: zypper lr samename
   register: zypper_result

-- assert:
+- name: ensure url get updated on repo with same name
+  assert:
     that:
       - "'/science/' not in zypper_result.stdout"
       - "'/devel:/languages:/ruby/' in zypper_result.stdout"
@@ -140,7 +149,8 @@
     state: present
   register: add_repo_again

-- assert:
+- name: no update in case of $releasever usage in url
+  assert:
     that:
       - add_repo is changed
       - add_repo_again is not changed
@@ -151,10 +161,21 @@
     state: absent
   register: remove_repo

-- assert:
+- name: verify repo was removed
+  assert:
     that:
       - remove_repo is changed

+- name: get list of files in /etc/zypp/repos.d/
+  command: ls /etc/zypp/repos.d/
+  changed_when: false
+  register: releaseverrepo_etc_zypp_reposd
+
+- name: verify removal of file releaseverrepo.repo in /etc/zypp/repos.d/
+  assert:
+    that:
+      - "'releaseverrepo' not in releaseverrepo_etc_zypp_reposd.stdout"
+
 - name: add a repo by basearch
   community.general.zypper_repository:
     name: basearchrepo
@@ -169,7 +190,8 @@
     state: present
   register: add_repo_again

-- assert:
+- name: no update in case of $basearch usage in url
+  assert:
     that:
       - add_repo is changed
       - add_repo_again is not changed
@@ -180,6 +202,74 @@
     state: absent
   register: remove_repo

-- assert:
+- name: verify repo was removed
+  assert:
     that:
       - remove_repo is changed
+
+- name: add new repository via url to .repo file
+  community.general.zypper_repository:
+    repo: http://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable/openSUSE_Leap_{{ ansible_distribution_version }}/systemsmanagement:Uyuni:Stable.repo
+    state: present
+  register: added_by_repo_file
+
+- name: get repository details from zypper
+  command: zypper lr systemsmanagement_Uyuni_Stable
+  register: get_repository_details_from_zypper
+
+- name: verify adding via .repo file was successful
+  assert:
+    that:
+      - "added_by_repo_file is changed"
+      - "get_repository_details_from_zypper.rc == 0"
+      - "'/systemsmanagement:/Uyuni:/Stable/' in get_repository_details_from_zypper.stdout"
+
+- name: add same repository via url to .repo file again to verify idempotency
+  community.general.zypper_repository:
+    repo: http://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable/openSUSE_Leap_{{ ansible_distribution_version }}/systemsmanagement:Uyuni:Stable.repo
+    state: present
+  register: added_again_by_repo_file
+
+- name: verify nothing was changed adding a repo with the same .repo file
+  assert:
+    that:
+      - added_again_by_repo_file is not changed
+
+- name: remove repository via url to .repo file
+  community.general.zypper_repository:
+    repo: http://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable/openSUSE_Leap_{{ ansible_distribution_version }}/systemsmanagement:Uyuni:Stable.repo
+    state: absent
+  register: removed_by_repo_file
+
+- name: get list of files in /etc/zypp/repos.d/
+  command: ls /etc/zypp/repos.d/
+  changed_when: false
+  register: etc_zypp_reposd
+
+- name: verify removal via .repo file was successful, including cleanup of local .repo file in /etc/zypp/repos.d/
+  assert:
+    that:
+      - "removed_by_repo_file"
+      - "'/systemsmanagement:/Uyuni:/Stable/' not in etc_zypp_reposd.stdout"
+
+- name: Copy test .repo file
+  copy:
+    src: 'files/systemsmanagement_Uyuni_Utils.repo'
+    dest: '{{ remote_tmp_dir }}'
+
+- name: add new repository via local path to .repo file
+  community.general.zypper_repository:
+    repo: "{{ remote_tmp_dir }}/systemsmanagement_Uyuni_Utils.repo"
+    state: present
+  register: added_by_repo_local_file
+
+- name: get repository details for systemsmanagement_Uyuni_Utils from zypper
+  command: zypper lr systemsmanagement_Uyuni_Utils
+  register: get_repository_details_from_zypper_for_systemsmanagement_Uyuni_Utils
+
+- name: verify adding repository via local .repo file was successful
+  assert:
+    that:
+      - "added_by_repo_local_file is changed"
+      - "get_repository_details_from_zypper_for_systemsmanagement_Uyuni_Utils.rc == 0"
+      - "'/systemsmanagement:/Uyuni:/Utils/' in get_repository_details_from_zypper_for_systemsmanagement_Uyuni_Utils.stdout"

Red Hat Certified Specialist in Advanced Automation: Ansible Best Practices - Ansible 2.8, Tower 3.5 & Enterprise Linux 8

2021-06-01T00:00:00+02:00

After passing the RHCSA and RHCE Exam already a few weeks ago, I also completed the Red Hat Exam EX447 today and earned the Red Hat Certified Specialist in Advanced Automation: Ansible Best Practices Certification.

It was the next logical Step to finish the Ansible Track before moving on to other Specialist Exams on my way to become a Red Hat Certified Architect (RHCA) one day ;).

Compared to the RHCE with a more or less 100% Ansible focus, EX447 was roughly 50% Ansible in more depth and 50% Ansible Tower.

I'm using AWX, the Tower Upstream Project, for quite some time already, I think that expirience helped a lot.

Summary: A Red Hat Certified Specialist in Advanced Automation: Ansible Best Practices is able to automate management of large or complex networks of machines. He demonstrate mastery of Ansible, knowledge of Ansible Tower and an understanding of Ansible best practices.

Source & Copyright: https://www.redhat.com

Skills

Understand and use Git to manage collections of playbooks
Manage inventory variables
Manage and Administer Ansible task execution
Transform data with filters and plugins
Delegate tasks
Install Ansible Tower
Manage access for Ansible Tower
Manage inventories, advanced inventories and credentials
Manage projects
Manage job workflows
Work with the Ansible Tower API
Backup and Restore Ansible Tower

Source & Copyright: https://www.redhat.com

Certificate

Downloads
- Certificate (PDF, 856.2K)
- Badge (PNG, 59.4K)
Links
- Verify Credly Badge (Archive: [1], [2])
- Verify Red Hat Badge (Archive: [1], [2])
- Verify Certificate

Red Hat Certified Engineer - Ansible 2.8 & Enterprise Linux 8

2021-05-25T00:00:00+02:00

Today I earned the Red Hat Certified Engineer Certification by passing the Red Hat Exam EX294.

It's the Version 8 RHCE Exam which quite differ from the "old" one by focussing on Ansible.

So during the performance-based Exam you have to create a lot of Ansible Playbooks and Roles to solve the Tasks. It's a tough Exam but fair and preparing for it improved my Ansible Skills a lot, even though I was using Ansible quite often within the last years.

Summary: A Red Hat® Certified Engineer (RHCE®) is a Red Hat Certified System Administrator (RHCSA) who is ready to automate Red Hat® Enterprise Linux® tasks, integrate Red Hat emerging technologies, and apply learned automation skills for efficiency and innovation.

Source & Copyright: https://www.redhat.com

Skills

Be able to perform all tasks expected of a Red Hat Certified System Administrator
Understand core components of Ansible
Install and configure an Ansible control node
Configure Ansible managed nodes
Script administration tasks
Create Ansible plays and playbooks
Use Ansible modules for system administration tasks
Work with roles
Use advanced Ansible features

Source & Copyright: https://www.redhat.com

Certificate

Downloads
- Certificate (PDF, 855.9K)
- Badge (PNG, 51.8K)
Links
- Verify Credly Badge (Archive: [1], [2])
- Verify Red Hat Badge (Archive: [1], [2])
- Verify Certificate

The Wombelix Post - Ansible

Ansible Hackathon 2021 Contribution: Improved idempotency - zypper_repository

Red Hat Certified Specialist in Advanced Automation: Ansible Best Practices - Ansible 2.8, Tower 3.5 & Enterprise Linux 8

Skills

Certificate

Red Hat Certified Engineer - Ansible 2.8 & Enterprise Linux 8

Skills

Certificate