The Wombelix Post - SSH

Access to SUSE Live Lab Environment through SSH reverse Tunnel and RDP

2022-09-10T00:00:00+02:00

SUSE provides some great Training Courses with a lot of Exercises and Online Labs are also available for most of them. But, at least in the Environment available to SUSE Partners, the Labs can only be accessed through a Browser, with a fixed size of 1080x800, no support for Resize, Full-screen or Copy/Paste.

Update 2024-10-06: There is a new Open in new window button available which allows higher resolutions than 1080x800 because it uses the full browser window. But all other limitations are the same since I wrote about it two years ago. Therefore I still prefer to reverse tunnel into the Lab machine and use an RDP client as explained in this post.

The Tech Stack used to provide those access: hastexo XBlock (Archive: [1], [2]), Apache Guacamole (Archive: [1], [2]), and Open edX (Archive: [1], [2]).

As much I love the provided Trainings, that's just not really comfortable in my opinion. It's fine for a few quick Exercises, but if you have some complex ones or do multiple hours of hands on Training, a more direct way to connect, with support for Full-screen and Copy/Paste, would be way better.

The alternative is to setup the Lab Environment locally, but that requires additional time and resources.

SUSE Labs are based on a main server, which runs all other virtual machines, relevant for the lab, via KVM. You always access this main server and from there you can either use SSH or the Virtual Machine Manager Console to connect to the other systems.

The good thing, this main server has full Internet Access via NAT (IPv4), we can establish outgoing connections without restrictions and also have full root permissions. So by establishing a SSH Reverse Tunnel to a public server, as rendezvous point, it's possible to access the Lab Server quite easy via RDP, which is enabled by default, because of the used Tech Stack to provide Browser access.

-----------------------                                           ------------------------
| Local Workstation   |          -----------------------          | SUSE Live Lab Server |   ------------------------------
|                     | tcp/22 > | VPS (Public Server) | < tcp/22 |                      | > | Other Lab Virtual Machines |
| SSH Tunnel to VPS   |          -----------------------          | SSH Tunnel to VPS    |   ------------------------------
| Outgoing Connection |                                           | Outgoing Connection  |
-----------------------                                           ------------------------

    localhost:3389         >            SSH Tunnel            >        localhost:3389

Steps:

Create VPS

Any cheap Virtual Machine, for example Hetzner Cloud (Referral Link) with Public IPv4 address does the trick, you can also re-use any Server you might already have.

Create additional User Account on the VPS used for SSH Tunneling

You have to type the User and Password into the SUSE Live Lab Server (no copy/paste), so I picked something simple and didn't used my regular User for the tunneling, but it's up to you.

Establish a SSH Reverse Tunnel from the Lab Server to the VPS and also from your local Workstation to the VPS

The commands used below are based on the Article SSH tunnel between two servers behind firewalls (Archive: [1], [2])

Connect via RDP

I'm running openSUSE Tumbleweed and use Remmina, any other Client with RDP support will work as well.

It's a connection on your local Workstation to localhost:3389, which then goes through the established SSH Tunnel and hit Port 3389 (RDP) on the Lab Server.

Commands:

# VPS (Hetzner Cloud, CX11, Falkenstein Germany, 0.0071 € / hour, Rocky Linux 9)
useradd -c "SSH Tunnel" -m -U sshtunnel
passwd sshtunnel

# SUSE Live Lab Server
ssh -o TCPKeepAlive=no -o ServerAliveInterval=15 -nNT -R 3389:localhost:3389 sshtunnel@<vps-public-ip>

# Local Workstation
ssh -o TCPKeepAlive=no -o ServerAliveInterval=15 -nNT -L 3389:localhost:3389 sshtunnel@<vps-public-ip>

# Lab Server
# The password for user "tux" is unknown so you have to reset it
sudo su -
passwd tux

# Local Workstation
# Open an RDP Client, connect to localhost as user "tux" with the password you set earlier on the Lab Server

Important:

1) Leave the Web Browser with the Live Lab HTML Viewer open, if you close it the VM will be suspended, keep an eye on it, move the mouse from time to time when the windows is active or click the confirmation that you still want to keep the VM running when you are inactive for a while.

2) This hack will affect the access to the lab server through the Browser, you will see a login failed for display :0 error message, click ok and on the XRDP prompt, select xorg, user tux and the password you set earlier.

This is just a quick & dirty way to interact a little more comfortable with the Lab Environment, you can tunnel through SSH whatever you want, SSH Sessions or X Forwarding, whatever you prefer.

Especially the fact that your VM get suspended when you leave the website, makes it not really reliable, but for me it worked quite well to prepare for some Exams.

Happy hacking and enjoy your next SUSE Training :)

FreeBSD 13 Base System OpenSSH Server Hardening

2022-03-17T00:00:00+01:00

Default sshd configs tend to focus more on compatibility instead security. Therefore hardening should be one of the first things after setup a new system.

I'm using the OpenSSH Daemon which comes with the FreeBSD Base System. When you want to use the openssh-portable port instead, skip Step 3, the rest should be identical.

Step 1) Delete existing host keys, generate new rsa and ed25519 key:

rm /etc/ssh/ssh_host_*
ssh-keygen -q -t rsa -b 4096 -f ssh_host_rsa_key -N ""
ssh-keygen -q -t ed25519 -f ssh_host_ed25519_key -N ""

Step 2) Create new Diffie-Hellman groups and avoid small moduli

ssh-keygen -G moduli-3072.candidates -b 3072
ssh-keygen -T moduli-3072 -f moduli-3072.candidates
mv moduli-3072 /etc/ssh/moduli
rm moduli-3072.candidates

Step 3) Disable DSA and ECDSA host keys, only use RSA and ED25519

sysrc sshd_dsa_enable="NO"
sysrc sshd_ecdsa_enable="NO"
sysrc sshd_ed25519_enable="YES"
sysrc sshd_rsa_enable="YES"

Step 4) Optimize /etc/ssh/sshd_config, improve security, restrict allowed key exchange, cipher and MAC algorithms

# Hardening
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha256
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
MACs hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,umac-128-etm@openssh.com
HostKeyAlgorithms ssh-ed25519,ssh-ed25519-cert-v01@openssh.com

# Security
PermitRootLogin no
AuthenticationMethods publickey
ChallengeResponseAuthentication no
UsePAM no
VersionAddendum none
X11Forwarding no
AuthorizedKeysFile .ssh/authorized_keys
Subsystem sftp /usr/libexec/sftp-server

Run service sshd restart to apply the new settings, to verify the results of your hardening, you can use the CLI Tool ssh-audit which is also available as Online Version.

Following a Custom Policy for ssh-audit based on the above recommendations:

name = "Custom Policy - FreeBSD 13 Base System OpenSSH Daemon (2022/03/17)"
version = 1
dh_modulus_size_diffie-hellman-group-exchange-sha256 = 2048
host keys = ssh-ed25519
key exchanges = curve25519-sha256, curve25519-sha256@libssh.org, diffie-hellman-group16-sha512, diffie-hellman-group18-sha512, diffie-hellman-group-exchange-sha256
ciphers = chacha20-poly1305@openssh.com, aes256-gcm@openssh.com, aes128-gcm@openssh.com, aes256-ctr, aes192-ctr, aes128-ctr
macs = hmac-sha2-256-etm@openssh.com, hmac-sha2-512-etm@openssh.com, umac-128-etm@openssh.com

Copy the policy into a file and run ssh-audit -P=<policy.txt> <servername>. The result should be similar to following example without warnings or errors:

# general
(gen) banner: SSH-2.0-OpenSSH_8.8
(gen) software: OpenSSH 8.8
(gen) compatibility: OpenSSH 7.4+, Dropbear SSH 2018.76+
(gen) compression: enabled (zlib@openssh.com)

# key exchange algorithms
(kex) curve25519-sha256                     -- [info] available since OpenSSH 7.4, Dropbear SSH 2018.76
(kex) curve25519-sha256@libssh.org          -- [info] available since OpenSSH 6.5, Dropbear SSH 2013.62
(kex) diffie-hellman-group16-sha512         -- [info] available since OpenSSH 7.3, Dropbear SSH 2016.73
(kex) diffie-hellman-group18-sha512         -- [info] available since OpenSSH 7.3
(kex) diffie-hellman-group-exchange-sha256 (2048-bit) -- [info] available since OpenSSH 4.4

# host-key algorithms
(key) ssh-ed25519                           -- [info] available since OpenSSH 6.5

# encryption algorithms (ciphers)
(enc) chacha20-poly1305@openssh.com         -- [info] available since OpenSSH 6.5
                                            `- [info] default cipher since OpenSSH 6.9.
(enc) aes256-gcm@openssh.com                -- [info] available since OpenSSH 6.2
(enc) aes128-gcm@openssh.com                -- [info] available since OpenSSH 6.2
(enc) aes256-ctr                            -- [info] available since OpenSSH 3.7, Dropbear SSH 0.52
(enc) aes192-ctr                            -- [info] available since OpenSSH 3.7
(enc) aes128-ctr                            -- [info] available since OpenSSH 3.7, Dropbear SSH 0.52

# message authentication code algorithms
(mac) hmac-sha2-256-etm@openssh.com         -- [info] available since OpenSSH 6.2
(mac) hmac-sha2-512-etm@openssh.com         -- [info] available since OpenSSH 6.2
(mac) umac-128-etm@openssh.com              -- [info] available since OpenSSH 6.2

# fingerprints
(fin) ssh-ed25519: SHA256:uN9Oton+VmLL793KirVFB+ilD3Bndra4I/3yFntgX8k

# algorithm recommendations (for OpenSSH 8.8)
(rec) +diffie-hellman-group14-sha256        -- kex algorithm to append
(rec) +rsa-sha2-256                         -- key algorithm to append
(rec) +rsa-sha2-512                         -- key algorithm to append

Sources:

https://ozgurkazancci.com/ssh-server-security-audit-hardening-freebsd (Archive: [1], [2])
https://gist.github.com/koobs/e01cf8869484a095605404cd0051eb11 (Archive: [1], [2])
https://www.ssh-audit.com/hardening_guides.html (Archive: [1], [2])

EMail Notification to Unlock LUKS encrypted disk via Dropbear SSH

2022-03-16T00:00:00+01:00

I'm running Proxmox VE Hosts with full encrypted disks and Dropbear SSH to unlock them remotely. I wanted to get notified when I have to login and enter the key, e.g. in case of an unexpected reboot.

My setup is based on the mailjet Free Tier (6000 Mails / Month, 200 Mails / Day) and some initramfs customizing to include curl, a few other required libraries and a simple script to send an API request, tested on PVE 7.1 / Debian 11 (bullseye)

I will enclose placeholder in < >, please replace them with appropriate values based on your Environment.

The hook script /etc/initramfs-tools/hooks/curl take care that curl, libnss_dns and a resolv.conf are included in the initramfs:

#!/bin/sh

PREREQ=""

prereqs()
{
        echo "$PREREQ"
}

case $1 in
prereqs)
        prereqs
        exit 0
        ;;
esac

. /usr/share/initramfs-tools/hook-functions

copy_exec /usr/bin/curl /bin

# Fix DNS resolver
cp -a /usr/lib/x86_64-linux-gnu/libnss_dns* $DESTDIR/usr/lib/x86_64-linux-gnu/
printf "nameserver <dns_server1_ipv4_address>\nnameserver <dns_server2_ipv4_address>\n" > $DESTDIR/etc/resolv.conf

The actual notification will be send by /etc/initramfs-tools/scripts/init-premount/notification through the mailjet API. You shouldn't have a problem to adjust it to use any other Service as long you can trigger a API via curl :)

#!/bin/sh

PREREQ=""

prereqs()
{
        echo "$PREREQ"
}

case $1 in
prereqs)
        prereqs
        exit 0
        ;;
esac

. /scripts/functions

configure_networking

/bin/curl --insecure -s \
-X POST \
--user "<mailjet_api_user>:<mailjet_api_pass>" \
https://api.mailjet.com/v3.1/send \
-H 'Content-Type: application/json' \
-d '{
  "Messages":[
    {
      "From": {
        "Email": "<sender_email>",
        "Name": "<sender_name>"
      },
      "To": [
        {
          "Email": "<recipient_mail>",
          "Name": "<recipient_name>"
        },
        {
          "Email": "<recipient2_mail>",
          "Name": "Pushover"
        }
      ],
      "Subject": "Action required: Unlock <fqdn>!",
      "TextPart": "Server <fqdn> was restarted and need to be unlocked to proceed boot sequence.",
      "CustomID": "DropbearUnlockRequest"
    }
  ]
}'

I'm also using Pushover to receive notifications on my mobile, they offer mail2push, so I just added my personal pushover address as second recipient to get notified by Mail and Pushover.

Make both scripts executable:

chmod +x /etc/initramfs-tools/hooks/curl`
chmod +x /etc/initramfs-tools/scripts/init-premount/notification

Run update-initramfs -u and you are good to go, during the next reboot you should receive an Email Notification to enter your LUKS Key and unlock your disk.

SUSE Manager / Uyuni - Salt SSH Push: Thin package corrupted

2021-03-02T00:00:00+01:00

At work we had issues with almost all Salt-SSH / SSH-Push connected Clients for months due to an unhandled exception ImportError: No module named 'salt.exceptions' that occurred regularly. It took me a lot of time to track the issue down together with SUSE so I want to share my Experience.

Error

[...]
"stderr": "Traceback (most recent call last):\n File
\"/var/tmp/.user_a567dc_salt/salt-call\", line 26, in \n from
salt.scripts import salt_call\n File
\"/var/tmp/.user_a567dc_salt/pyall/salt/scripts.py\", line 21, in \n
from salt.exceptions import SaltSystemExit, SaltClientError,
SaltReqTimeoutError\nImportError: No module named 'salt.exceptions'"
[...]

Background

Normally Salt is using an Agent (Salt-Minion) but also supports an Agentless approach called Salt-SSH. A package ("thin.tgz"), which contains all python code and files that Salt need to run Jobs on a server, will be uploaded and extracted to /var/tmp/.<user>_<id>_salt/ on the target Server. The "id" is related to the used Salt Version, the "user" is the one configured on the Salt Master - in our Case SUSE Manager - which has full sudo permissions and is allowed to login via SSH on the target.

Everytime Salt connect to run a Job, some checks will performed to verify that state of the thin package:

Does the Folder for the specified user with the matching id exist in /var/tmp/
Is the file "code-checksum" available and does the containing checksum matches with the thin.tgz package on the Salt Master

As long both checks are successful, the Salt Master will _not_ upload a new thin.tgz - that's important to understand.

Troubleshooting

First it was totally unclear were the issue was coming from, it could occur with every job type, on different server at a different time. The thin package was always corrupted at some point what caused the unhandled exception. The Workaround was to manually delete the whole folder of the Salt thin package on the target Server so the Salt Master uploaded a fresh copy next time.

SUSE helped to go through tons of logs and to visualize a timeline were I could see that the issue seem to occur every 7 days.

Further investigation confirmed, in this case the problem was self-made. Someone from the Team configured systemd to automatically cleanup the tmp folder on a lot of server and neither tested nor documented it well. Removing the thin package folder wouldn't be a problem, it would be just newly uploaded, as long it's done right but it wasn't.

/etc/tmpfiles.d/clean_tmp.conf

D /tmp 1777 root root 7d
D /var/tmp 1777 root root 7d

The tmpfiles config removed only files and folder owned by user root, but a few files, like "code-checksum", of the thin package folder are owned by salt. So every 7 days almost all files were removed and therefore corrupted Salt thin, but the Salt Master found the folder and the "code-checksum" file with the correct hash and thought the thin state is healthy.

Solution

At the end I just removed the custom tmpfiles config and once again the thin folder manually on all affected server.

For us there wasn't much benefit in deleting the thin package weekly and generate high load and consume bandwith to re-upload it. In case you prefer the regular cleanup of your tmp folders, just ensure that the whole thin folder and all files are covered and either removed ore kept untouched by your ruleset.

Even though it was a human error and not directly related to SUSE Manager / Uyuni or Salt it demonstrated how fragile such components can be. From a technical point of view I understand why there are just a few checks and validation of the hash in a file. Doing that for the whole folder content on every job run would horrible slow and increase the load. But as we can see, it comes with a price and can be affected by something you wouldn't expect.

SUSE Manager / Uyuni - Valid PTR Record required for SSH-Push Clients

2021-02-23T00:00:00+01:00

I'm responsible for a SUSE Manager Installation, the commercial product based on Uyuni, with almost 900 connected Server.

The recommend connection method is using a Salt Minion, but SSH-Push - based on Salt-SSH - is the agentless alternative. In our Environment, something around 200 Server using this connection type, because there is already a Salt Minion running, attached to a different Salt Master.

Problem

During the last weeks we faced some issues were the salt-api service is going to crash due to an unhandled exception from time to time.

Investigation together with SUSE is still ongoing but so far it seems like, that under rare conditions, a Server with connection method SSH-Push and without valid PTR DNS Record can indeed raise an Exception in the Salt Codebase.

Background

Function ip_to_host located in /usr/lib/python3.6/site-packages/salt/utils/network.py is passing the FQDN to Python built-in Function socket.gethostbyaddr. This will raise an Exception, handled by ip_to_host which then return None (Normally that should be a Hostname).

Based on the return value, a new Minion Job for Server None will be started and end up in an unhandled Exception that cause salt-api to crash as shown in the following error message (Debug Mode need to be enabled for Salt as well Taskomatic):

/var/log/salt/api_error

Traceback (most recent call last):
        File "/usr/lib/python3.6/site-packages/salt/netapi/rest_cherrypy/app.py", line 860, in hypermedia_handler
                ret = cherrypy.serving.request._hypermedia_inner_handler(*args, **kwargs)
        File "/usr/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__
                return self.callable(*self.args, **self.kwargs)
        File "/usr/lib/python3.6/site-packages/salt/netapi/rest_cherrypy/app.py", line 2140, in POST
                'return': list(self.exec_lowstate()),
        File "/usr/lib/python3.6/site-packages/salt/netapi/rest_cherrypy/app.py", line 1187, in exec_lowstate
                ret = self.api.run(chunk)
        File "/usr/lib/python3.6/site-packages/salt/netapi/__init__.py", line 152, in run
                return l_fun(*f_call.get('args', ()), **f_call.get('kwargs', {}))
        File "/usr/lib/python3.6/site-packages/salt/netapi/__init__.py", line 218, in ssh
                return ssh_client.cmd_sync(kwargs)
        File "/usr/lib/python3.6/site-packages/salt/client/ssh/client.py", line 159, in cmd_sync
                **kwargs)
        File "/usr/lib/python3.6/site-packages/salt/client/ssh/client.py", line 123, in cmd
                for ret in ssh.run_iter(jid=kwargs.get('jid', None)):
        File "/usr/lib/python3.6/site-packages/salt/client/ssh/__init__.py", line 696, in run_iter
                self.cache_job(jid, host, ret[host], fun)
        File "/usr/lib/python3.6/site-packages/salt/client/ssh/__init__.py", line 720, in cache_job
                'fun': fun})
        File "/usr/lib/python3.6/site-packages/salt/returners/local_cache.py", line 147, in returner
                hn_dir = os.path.join(jid_dir, load['id'])
        File "/usr/lib64/python3.6/posixpath.py", line 94, in join
                genericpath._check_arg_types('join', a, *p)
        File "/usr/lib64/python3.6/genericpath.py", line 149, in _check_arg_types
                (funcname, s.__class__.__name__)) from None
TypeError: join() argument must be str or bytes, not 'NoneType'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
        File "/usr/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 638, in respond
                self._do_respond(path_info)
        File "/usr/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 697, in _do_respond
                response.body = self.handler()
        File "/usr/lib/python3.6/site-packages/cherrypy/lib/encoding.py", line 219, in __call__
                self.body = self.oldhandler(*args, **kwargs)
        File "/usr/lib/python3.6/site-packages/salt/netapi/rest_cherrypy/app.py", line 894, in hypermedia_handler
                if cherrypy.config['debug']
        File "/usr/lib64/python3.6/traceback.py", line 167, in format_exc
                return "".join(format_exception(*sys.exc_info(), limit=limit, chain=chain))
        File "/usr/lib64/python3.6/traceback.py", line 121, in format_exception
                type(value), value, tb, limit=limit).format(chain=chain))
        File "/usr/lib64/python3.6/traceback.py", line 498, in __init__
                _seen=_seen)
        File "/usr/lib64/python3.6/traceback.py", line 509, in __init__
                capture_locals=capture_locals)
        File "/usr/lib64/python3.6/traceback.py", line 338, in extract
                if limit >= 0:

TypeError: '>=' not supported between instances of 'TypeError' and 'int'

/var/log/salt/api

[...]
2021-02-08 01:08:01,229 [salt.utils.network:239 ][DEBUG ][38989] salt.utils.network.ip_to_host('server.example.com') failed: [Errno 0] Resolver Error 0 (no error)
[...]
2021-02-08 01:08:02,801 [salt.loaded.int.returner.local_cache:252 ][DEBUG ][38989] Adding minions for job 20210208000802780236: [None]
[...]
2021-02-08 01:08:02,839 [salt.utils.process:767 ][ERROR ][5340] An un-handled exception from the multiprocessing process 'Process-1:335' was caught:
[...]
2021-02-08 01:08:02,916 [salt.client.ssh :644 ][ERROR ][38989] Target 'None' did not return any data, probably due to an error.
2021-02-08 01:08:02,917 [salt.loaded.int.netapi.rest_cherrypy.app:887 ][DEBUG ][38989] Error while processing request for: /run

/var/log/rhn/rhn_taskomatic_daemon.log

2021-02-08 01:08:02,976 [Thread-70313] ERROR com.redhat.rhn.taskomatic.task.SSHPush - com.suse.salt.netapi.
exception.SaltException: Response code: 500
java.lang.RuntimeException: com.suse.salt.netapi.exception.SaltException: Response code: 500
        at com.suse.manager.webui.services.impl.SaltService.callSync(SaltService.java:235)
        at com.suse.manager.webui.services.impl.SaltService.ping(SaltService.java:244)
        at com.redhat.rhn.taskomatic.task.sshpush.SSHPushWorkerSalt.performCheckin(SSHPushWorkerSalt.java:337)
        at com.redhat.rhn.taskomatic.task.sshpush.SSHPushWorkerSalt.lambda$run$0(SSHPushWorkerSalt.java:122)
        at java.base/java.util.Optional.ifPresent(Optional.java:183)
        at com.redhat.rhn.taskomatic.task.sshpush.SSHPushWorkerSalt.run(SSHPushWorkerSalt.java:110)
        at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:732)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: com.suse.salt.netapi.exception.SaltException: Response code: 500
        at com.suse.salt.netapi.client.impl.HttpAsyncClientImpl.createSaltException(HttpAsyncClientImpl.java:
        145)
        at com.suse.salt.netapi.client.impl.HttpAsyncClientImpl.access$000(HttpAsyncClientImpl.java:27)
        at com.suse.salt.netapi.client.impl.HttpAsyncClientImpl$1.completed(HttpAsyncClientImpl.java:121)
        at com.suse.salt.netapi.client.impl.HttpAsyncClientImpl$1.completed(HttpAsyncClientImpl.java:101)
        at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:123)
        at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted
(DefaultClientExchangeHandlerImpl.java:181)
        at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:
        442)
        at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:332)
        at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:
        265)
        at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
        at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
        at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run
(AbstractMultiworkerIOReactor.java:588)
... 1 more

You can quickly check, with two Python based commands, if the reverse lookup is working for a given FQDN. In this Example we assume the FQDN is server.example.com and the assigned IP Address is 10.11.12.13:

python3 -c 'import socket; print(socket.gethostbyaddr("server.example.com"))'
python3 -c 'import socket; print(socket.getaddrinfo("server.example.com", 0, 0, 0, 0))'

In case everything is fine, both should return something similar as the following:

('server.example.com', [], ['10.11.12.13'])
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('10.11.12.13', 0)),
        (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_DGRAM: 2>, 17, '', ('10.11.12.13', 0)),
        (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_RAW: 3>, 0, '', ('10.11.12.13', 0))]

Further reading regarding gethostbyaddr: https://docs.python.org/3/library/socket.html#socket.gethostbyaddr

Solution

I expect that SUSE is going to include a Patch in future SUSE Manager and Uyuni releases were a Server will probably skipped if there is no valid PTR Record. So you should double check in your Environment, that all Systems connected via SSH-Push / Salt-SSH have the necessary DNS Records and can be resolved by their FQDN as well IP Address to avoid potential issues.