2025-12-06T01:23:59  *** teepee_ is now known as teepee
2025-12-06T05:30:47  *** teepee_ is now known as teepee
2025-12-06T18:54:57  *** knurpht_ is now known as knurpht
2025-12-06T21:01:11  <cboltz> just talked to darix - the locks for discourse are still needed since the update is ... difficult
2025-12-06T21:01:48  <cboltz> I even added a few more locks so that zypper dup is happy again (which also means the automated updates should work again)
2025-12-06T21:02:32  <cboltz> since the dup brought a new kernel, I'll reboot the VM now - expect a short downtime
2025-12-06T21:18:29  <matrix-o-o> <hendersj> Sounds good, looks like we're still offline?
2025-12-06T21:18:51  <cboltz> yes, I know, and am still looking for the reason
2025-12-06T21:19:08  <matrix-o-o> <hendersj> Sounds good, thanks
2025-12-06T21:21:01  <cboltz> it would be even better if I'd know what the problem is
2025-12-06T21:21:51  <matrix-o-o> <hendersj> I have a little time and could take a look if you wanted.  I didn't want to interfere with ongoing investigations - too many cooks and all that
2025-12-06T21:21:58  <cboltz> curl '[::1]'  on discourse01 gives me the HTML of the forums page
2025-12-06T21:22:23  <cboltz> but from atlas1, it can't be reached
2025-12-06T21:22:24  <matrix-o-o> <hendersj> Hmmm, so it's up and running, but the reverse proxy isn't allowing connections?
2025-12-06T21:23:10  <cboltz> Dec 06 21:13:59 atlas1 haproxy[7724]: [WARNING]  (7724) : Health check for server forums/discourse01 failed, reason: Layer4 connection problem, info: "Permission denied", check duration: 0ms, status: 0/2 DOWN.
2025-12-06T21:23:12  <cboltz> Dec 06 21:13:59 atlas1 haproxy[7724]: [WARNING]  (7724) : Server forums/discourse01 is DOWN. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2025-12-06T21:23:13  <cboltz> Dec 06 21:13:59 atlas1 haproxy[7724]: [ALERT]    (7724) : backend 'forums' has no server available!
2025-12-06T21:23:34  <cboltz> note that I collected these log lines right now, so it seems haproxy did last check 10 minutes ago
2025-12-06T21:24:35  <cboltz> but curl from atlas1 also fails, so it's probably not haproxy's fault
2025-12-06T21:25:08  <cboltz> ah, found it
2025-12-06T21:25:11  <matrix-o-o> <hendersj> Sounds like it's not listening on the expected interface for some reason (obvious, I know .... thinking out loud)
2025-12-06T21:25:28  <cboltz> firewalld on discourse blocked it - as a hotfix, I just stopped it
2025-12-06T21:25:52  <matrix-o-o> <hendersj> That works, confirmed from here I can see it externally
2025-12-06T21:26:32  <matrix-o-o> <hendersj> Weird that the firewalld config changed with the updates that were applied
2025-12-06T21:27:30  <cboltz>      8203      0 lrwxrwxrwx   1 root     root           41 Dec  6 20:52 /etc/systemd/system/multi-user.target.wants/firewalld.service -> /usr/lib/systemd/system/firewalld.service
2025-12-06T21:27:44  <cboltz> so my guess is that firewalld was _enabled_ by the update
2025-12-06T21:27:58  <matrix-o-o> <hendersj> Ah, and it was probably previously disabled
2025-12-06T21:28:08  <cboltz> most likely
2025-12-06T21:28:22  <cboltz> and there's no config in /etc/firewalld that would allow anything
2025-12-06T21:28:36  <cboltz> so I just systemctl disable'd it again
2025-12-06T21:29:04  <matrix-o-o> <hendersj> Sounds good.  We probably should investigate configuring it so it can be enabled, but back to status quo for now
2025-12-06T21:29:32  <matrix-o-o> <hendersj> I can confirm that on my sandbox system here it is also currently disabled by default.
2025-12-06T21:29:43  <matrix-o-o> <hendersj> And that system was set up using discourse01 as a template (but hasn't had the recent updates)
2025-12-06T21:30:43  <cboltz> firewalld can't hurt, but as long as we don't have a service running that listens on the (internally) public interface, but shouldn't be reachable, the firewall won't change much
2025-12-06T21:31:06  <matrix-o-o> <hendersj> That makes sense as well
2025-12-06T21:38:49  <matrix-o-o> <hendersj> Doing the update on my sandbox, and I saw this in the output of "zypper dup":
2025-12-06T21:46:58  <cboltz> nice, looks like man.o.o also got the firewall enabled today, and is now down
2025-12-06T21:47:14  <cboltz> disabled again, and man.o.o is back
2025-12-06T21:47:33  <matrix-o-o> <hendersj> Looks like the change came in from "systemd-presets-branding-openSUSE-12.2-27.1.noarch"
2025-12-06T21:48:05  <cboltz> doesn't sound too surprising (without having looked at the changelog)
2025-12-06T21:48:18  <matrix-o-o> <hendersj> Looks like it's enabling cupsd and a bunch of other stuff too.
2025-12-06T21:49:47  <cboltz> latest changelog only mentions firewalld (including a non-public bug reference :-/ )
2025-12-06T21:50:01  <cboltz> (and that's the only change since July)
2025-12-06T21:50:47  <cboltz> DimStar: since you wrote that changelog - see above
2025-12-06T21:50:51  <matrix-o-o> <hendersj> Yep, I see that.  I really wish we'd take the position of default changes for new installs only, rather than making changes to running systems when they're being updated.
2025-12-06T21:51:14  <matrix-o-o> <hendersj> Because changes like that cause this kind of breakage for no really good reason IMO
2025-12-06T21:51:46  <cboltz> I _guess_ that was the plan, but without access to bsc#1237923 it's really just a guess
2025-12-06T21:56:06  <cboltz> Reported as https://bugzilla.opensuse.org/show_bug.cgi?id=1254536
2025-12-06T22:31:12  <acidsys> we could remove firewalld from machines without firewalld:enabled pillar, but yes not a cool change
2025-12-06T22:33:15  <cboltz> I checked via salt, and discourse01 and man were the only VMs with a new firewalld symlink in /etc/systemd/
2025-12-06T22:33:42  <cboltz> the other VMs either have firewalld enabled since a longer time (which means it's intentional), or don't have it enabled
2025-12-06T22:34:08  <cboltz> so for now, I'd say we can just wait what happens in the bugreport
2025-12-06T22:34:39  <acidsys> or are running leap
2025-12-06T22:45:35  <cboltz> right, that also makes things easier ;-)