2026-03-21T14:44:22  <acidsys> lkocman I forwarded you some npm problem with quiz build
2026-03-21T17:52:07  <cboltz> I'm just updating narwal7 to Leap 16
2026-03-21T17:52:42  <cboltz> zypper dup is still running, but terribly slow since kanidm-unixd-clients was updated
2026-03-21T17:53:58  <cboltz> a new ssh connection I opened after that package update greets me with
2026-03-21T17:54:01  <cboltz>       I have no name!@narwal7 (static.o.o):~ >
2026-03-21T17:54:25  <cboltz> which sounds funny - but also indicates that name resolution is somehow broken currently
2026-03-21T17:56:41  <acidsys> try if removing it from /etc/nsswitch.conf speeds it up
2026-03-21T17:58:17  <cboltz> indeed, that helps :-)
2026-03-21T18:05:44  <cboltz> note to myself: if you revert this change before rebooting, you don't need to do a highstate via the saltmaster to be able to login
2026-03-21T18:06:03  <cboltz> (you may laugh nau ;-)
2026-03-21T18:06:38  <cboltz> anyway - update done, everything seems to work
2026-03-21T18:08:11  <acidsys> we did not have this on the machine we did last time
2026-03-21T18:11:55  <cboltz> actually I'm not completely sure - IIRC we also had problems to open an additional ssh connection. This time it took $timeout (probably 30) seconds to get one, maybe we just weren't patient enough last time
2026-03-21T18:13:58  <cboltz> I'll upgrade narwal8 now, let's see if it shows similar symptoms
2026-03-21T18:19:28  <cboltz> if you have a minute, can you have a look at the just-rebased MR 2558 please?
2026-03-21T18:20:39  <acidsys> lgtm
2026-03-21T18:21:13  <cboltz> :-)
2026-03-21T18:21:27  <acidsys> I also updated https://progress.opensuse.org/issues/189810 now, you can track there if you want
2026-03-21T18:26:16  <cboltz> I'm not sure how useful it is to add "$host upgraded" comments - they are not too useful, and a   salt \* grains.get osrelease   is probably easier to read ;-)  (and always up-to-date)
2026-03-21T18:28:33  <cboltz> I'd only add "surprises" to the ticket
2026-03-21T18:29:10  <cboltz> speaking about surprises - on both narwals, zypper dup gives me (for several, but not all packages)
2026-03-21T18:29:13  <cboltz>     error: Key 70af9e8139db7c82 (SuSE Package Signing Key <build@suse.de>) expired on 2024-09-20 08:21:47
2026-03-21T18:29:52  <cboltz> I did a highstate before, so the salt-managed keys should be up to date
2026-03-21T18:36:58  <cboltz> zypper dup is now also terribly slow on narwal8. I didn't watch exactly when this started, but the  kanidm  package looks like a good candidate
2026-03-21T18:38:06  <cboltz> restarting kanidm-unixd helps :-)
2026-03-21T18:39:57  <acidsys> are you actually updating kanidm, because last time leap 16 had an older version
2026-03-21T18:41:43  <cboltz> well, actually it's a downgrade (1.9.1 -> 1.7.1 according to /v/l/zypp/history)
2026-03-21T18:43:52  <acidsys> right
2026-03-21T18:45:39  <cboltz> according to zypp history, one of the last packages that was quickly installed was   kanidm-unixd-clients
2026-03-21T18:57:22  <cboltz> narwal8 completed, now over to narwal5 aka static_master
2026-03-21T19:15:58  <cboltz> narwal5 done, with the same kanidm-unixd slowdown
2026-03-21T19:21:06  <acidsys> if needed we can add the workaround to upgrade-os
2026-03-21T19:21:53  <cboltz> the problem is that ideally you need to restart kanidm-unixd in the middle of zypper dup
2026-03-21T19:22:04  <cboltz> doing it after zypper dup is possible, but wastes lots of time
2026-03-21T19:22:18  <acidsys> I mean to just patch nsswitch temporarily
2026-03-21T19:23:08  <cboltz> sadly that temporarily breaks ssh logins and sudo
2026-03-21T19:23:20  <acidsys> which you don't need for the duration of dup
2026-03-21T19:24:16  <cboltz> indeed
2026-03-21T19:25:00  <cboltz> if you update the script, please also extend the "after reboot" section with a mention of    salt-call saltutil.refresh_grains
2026-03-21T21:01:12  <cboltz> all jekyll pages are now also served by narwal[5-8] :-)
2026-03-21T21:07:30  <acidsys> nice
2026-03-21T21:07:43  <acidsys> now we can upgrade jekyll without downtime :)
2026-03-21T21:07:59  <cboltz> right
2026-03-21T21:15:24  <acidsys> https://get.opensuse.org/tumbleweed/ is 404
2026-03-21T21:15:35  <acidsys> andhttps://get.opensuse.org/ returns static.o.o
2026-03-21T21:19:07  <cboltz> Mar 21 21:17:58 narwal5 nginx[7853]: nginx: [emerg] a duplicate default server for [::]:80 in /etc/nginx/vhosts.d/static.opensuse.org.conf:18
2026-03-21T21:19:08  <cboltz> nice[tm]
2026-03-21T21:20:04  <acidsys> ah because both web_static and web_jekyll have default_server
2026-03-21T21:20:16  <acidsys> I suggest removing that default from both
2026-03-21T21:27:19  <cboltz> good idea, MR 2737 created
2026-03-21T21:45:24  <cboltz> for the records: we have default_server in several other roles (calendar, discourse, mailman etc.), but it's unlikely that we'll deploy two of them on one server