2023-11-26T02:59:19  *** teepee_ is now known as teepee
2023-11-26T11:18:00  <pjessen> any issues with stage.o.o ?
2023-11-26T11:19:02  <pjessen> ignore
2023-11-26T15:43:27  <acidsys> my admin-auto@ subscription was suspended again :(
2023-11-26T16:01:01  <cboltz> ssh (actually rsync over ssh) from narwal5 to narwal4 gets blocked - firewall issue?
2023-11-26T16:01:19  <acidsys> s/issue/feature/
2023-11-26T16:02:04  <acidsys> works fine for me
2023-11-26T16:02:54  <cboltz> the mails on admin-auto seem to disagree
2023-11-26T16:03:09  <cboltz> the last mail is from 14:00 - did you change something in the last hours?
2023-11-26T16:03:43  <acidsys> crameleon@narwal5:/home/crameleon> echo > /dev/tcp/narwal4/22 ; echo $?
2023-11-26T16:03:45  <acidsys> 0
2023-11-26T16:03:55  <acidsys> (also ssh itself returns)
2023-11-26T16:04:05  <acidsys> I didn't besides getting some sleep
2023-11-26T16:05:07  <cboltz> sounds like a good change :-)
2023-11-26T16:05:17  <acidsys> :)
2023-11-26T16:05:46  <cboltz> ssh indeed works now, so whatever you did while sleeping was good ;-)
2023-11-26T16:06:00  <acidsys> maybe you were ..
2023-11-26T16:06:02  <acidsys> dreaming ..
2023-11-26T16:08:03  <cboltz> no idea, but as long as it works now... ;-)
2023-11-26T16:08:13  <acidsys> ^^
2023-11-26T16:08:28  <acidsys> pjessen: https://gitlab.infra.opensuse.org/infra/salt/-/merge_requests/1120
2023-11-26T16:31:35  <cboltz> any objections against merging this _now_ ? It blocks applying the fix for the member aliases script
2023-11-26T16:32:46  <pjessen> those rate settings are optional anyway
2023-11-26T16:33:05  <cboltz> (the main.cf settings might also benefit from some sorting (even if in some cases the grouping by topic makes more sense and should be kept), but that's something for another MR)
2023-11-26T16:36:16  <acidsys> the mailserver state for main.cf needs refactoring too
2023-11-26T16:36:29  <acidsys> the regex replacement is not a good way to keep the file compliant with salt
2023-11-26T16:36:36  <acidsys> there are other lines which get missed
2023-11-26T16:42:20  <cboltz> file.keyvalue might be slightly better, but still won't handle multiline values
2023-11-26T16:43:05  <cboltz> there's also https://docs.saltproject.io/en/latest/ref/modules/all/salt.modules.postfix.html#salt.modules.postfix.set_main - but unfortunately it's a module, and there doesn't seem to be a postfix _state_
2023-11-26T16:48:56  <acidsys> imo there is no need for Salt to manage multiline values in the rendered file
2023-11-26T16:49:14  <acidsys> the pillar should use lists so it'll be easy to read there
2023-11-26T16:49:59  <acidsys> set_main is cool but it too doesn't enforce the compliance of the whole file
2023-11-26T16:51:14  <cboltz> multiline values might exist "for historical reasons" (= hand-edited main.cf) - but I wonder if it wouldn't be better to manage the whole main.cf instead of changing some values
2023-11-26T16:51:53  <acidsys> that's what I mean, the whole file should be managed
2023-11-26T16:54:17  <cboltz> does file.keyvalue support dropping unknown/unmanaged keys, or do we need to write some boring jinja ourself?
2023-11-26T16:55:28  <acidsys> it does
2023-11-26T16:57:10  <acidsys> upon checking the documentation though, I don't find what it was
2023-11-26T17:03:12  <acidsys> boring solution, file.managed on main.cf which only writes "include main.cf.salt" (or similar) ^^
2023-11-26T17:05:19  <cboltz> postfix and "include"? dream on ;-)
2023-11-26T17:05:51  <cboltz> we'll have to write the whole file, but that shouldn't be too difficult
2023-11-26T17:06:12  <cboltz> (basically loop over the pillar)
2023-11-26T17:07:14  <cboltz> we could even do that directly in profile/postfix, no need for a main.cf.jinja
2023-11-26T17:08:45  <acidsys> oh, sure that works as well
2023-11-26T17:26:25  <acidsys> I checked both our and my mail server logs and I did not find any bounces for admin-auto@
2023-11-26T17:26:51  <acidsys> so I'm not sure what the issue is
2023-11-26T17:32:07  <acidsys> does admin-auto@ have any different restrictions/settings than our other mailing lists?
2023-11-26T17:42:17  <acidsys> upon checking mailman it's also interesting how my username in the database is just "Georg" whereas other people seem to be stored using their IDP username
2023-11-26T17:44:28  <acidsys> I was now able to give my user the staff flag with which I was able to set my delivery settings for admin-auto@ from "disabled by bounces" to "enabled"
2023-11-26T17:46:51  <acidsys> is it expected that https://lists.opensuse.org/archives/list/admin-auto@lists.opensuse.org/ is empty?
2023-11-26T17:47:41  <cboltz> I think so - it wouldn't make much sense to restrict subscriptions (because admin-auto might receive stuff that shouldn't become public), and OTOH have a public archive
2023-11-26T17:47:56  <cboltz> besides that - who cares about yesterday's failure mails?
2023-11-26T17:51:06  <acidsys> not public makes sense but I figured upon logging in
2023-11-26T17:51:41  <acidsys> not a problem for me; I rarely delete emails from my inbox
2023-11-26T17:52:14  <acidsys> it would have just been cool to at least be able to browse using the web GUI if delivery is broken again
2023-11-26T18:16:49  <acidsys> regarding get_member_aliases, "Error: Could not connect to host monitor.infra.opensuse.org on port 5667" - there is nothing listening on monitor at 56667
2023-11-26T18:17:07  <acidsys> there is only nrpe at 5666
2023-11-26T18:20:07  <acidsys> also monitor update complains "Nov 26 02:17:30 monitor os-update[24592]: File './x86_64/chromium-119.0.6045.159-bp155.2.58.1.x86_64.rpm' not found on medium 'http://download-prg.infra.opensuse.org/update/leap/15.5/backports/'" - yes, I excluded web browsers from the mirror on purpose, how could I know we really have a server having a web browser installed ...
2023-11-26T18:21:13  <acidsys> it seems "icingaweb2-module-idoreports icingaweb2-module-pdfexport icingaweb2-module-reporting" depend on chromium .. I'm out of words
2023-11-26T18:21:39  <cboltz> hmm, that's... interesting[tm]
2023-11-26T18:26:42  <cboltz> for the monitor.i.o.o port - do you think sending to 5666 would work, or do we need something[tm] listening at 5667?
2023-11-26T18:28:21  <acidsys> well I don't know what it "expected" at 5667
2023-11-26T18:28:42  <acidsys> I guess you can try the command to 5666 and see if it returns?
2023-11-26T18:29:09  <cboltz> ok, I'll manually change the script on mx1
2023-11-26T18:29:18  <acidsys> but you can't test from mx* because I did not allow it in the firewall. so best test either locally from monitor or from another machine in os-internal
2023-11-26T18:29:27  <acidsys> if it works I can allow it :)
2023-11-26T18:29:33  <cboltz> ah, ok
2023-11-26T18:36:26  <cboltz> tested directly on monitor.i.o.o, and the only result I get is   Error:
2023-11-26T18:36:36  <cboltz> (yes, that's a very useful error message...)
2023-11-26T18:38:22  <cboltz> ah,   systemctl status nsca   explains why nothing is listening
2023-11-26T18:38:28  <cboltz> Nov 22 00:59:48 monitor nsca[1169]: Server address 192.168.47.7 port 5667: Address family for hostname not supported
2023-11-26T18:40:11  <cboltz> ... and once this is fixed in the config, send_nsca works :-)
2023-11-26T18:41:24  <cboltz> oh, funny - https://monitor.opensuse.org/icingaweb2/ says "Access denied". Without asking me to login...
2023-11-26T18:41:42  <acidsys> icinga2 is down
2023-11-26T18:42:05  <acidsys> nsca thanks, repairing it now. it's funny because I did check `rcnsca status` and it returned green. I did not consider there was an ignored error further up inthe journal
2023-11-26T18:42:28  <acidsys> ohh, you did just repair it
2023-11-26T18:45:26  <cboltz> yes, it only needed the IP updated
2023-11-26T18:48:54  <cboltz> so - can you please open 5667 in the firewall?
2023-11-26T18:50:22  <acidsys> done
2023-11-26T18:50:46  <acidsys> (all mx's, ip6 saddr @host6_mx ip6 daddr $host6_monitor tcp dport 5667 accept)
2023-11-26T18:50:57  <cboltz> thanks!
2023-11-26T19:04:25  <cboltz> FYI: !1037 is down to 16 failures - which also means it can test 49 roles successfully
2023-11-26T19:04:52  <darix> you nerds.
2023-11-26T19:05:20  <cboltz> now the question is - do we want to improve it to handle these 16 failures correctly, or should we (for now) whitelist them so that the other 49 roles get tested?
2023-11-26T19:08:15  <acidsys> I would prefer to deal with all of them first
2023-11-26T19:08:45  <acidsys> otherwise it's one of those hardcoded "TODO"Äs
2023-11-26T19:09:27  <cboltz> I get your point, but OTOH that's still better to test most roles than testing none of them ;-)
2023-11-26T19:10:09  <cboltz> also, the remaining error messages are not too helpful (and/or hard to fix or workaround), so if you have ideas how to fix them, please tell me
2023-11-26T19:10:28  <darix> cboltz: fix them with ||: :D
2023-11-26T19:10:31  <darix> _runs_
2023-11-26T19:12:29  <acidsys> I will check the output
2023-11-26T19:12:43  * acidsys locks darix into a while true loop
2023-11-26T19:13:10  <darix> i will just keep allocating memory until your cell runs ouf of space and bursts :p
2023-11-26T19:17:50  <acidsys> !
2023-11-26T19:17:52  * acidsys swaps
2023-11-26T19:19:52  <acidsys> bmwiedemann: I tried Munin but the distribution packaging seems incomplete. For example the provided CGI scripts want Perl CGI::Fast but that's only in some devel projects.
2023-11-26T19:37:29  <darix> acidsys: the move-nue-prg etherpad can be updated for MX no?
2023-11-26T19:44:39  <acidsys> yep!
2023-11-26T19:45:16  * acidsys still fights with distribution munin .. wrong permissions in /var/log, missing files and directories :(
2023-11-26T19:46:50  <acidsys> woo .. http://monitor.infra.opensuse.org:8024/
2023-11-26T19:54:35  <acidsys> unrelated, does anyone else have this problem where after some hours in a SSH session over the VPN, it breaks with "client_loop: send disconnect: Broken pipe" ?
2023-11-26T20:04:12  <acidsys> next problem on monitor, grafana login uses hardcoded freeipa ip address
2023-11-26T20:12:41  <acidsys> grafana repaired, the icinga2 on the other hand, it's hardcoded IP addresses seemingly for every single host, in addition to some broken dependency with provo-gate for Provo hosts
2023-11-26T20:15:17  <acidsys> now it's started after removing all provo hosts
2023-11-26T20:16:07  <acidsys> funnily moving /etc/icinga2/hosts to /etc/icinga2/hosts_disabled did not work, I had to move it outside of /etc/icinga2 .. seems some very wild wildcard include
2023-11-26T20:35:15  <cboltz> icinga2.conf:include_recursive "conf.d"
2023-11-26T20:35:34  <cboltz> which makes it less surprising that conf.d/hosts_disabled still gets included
2023-11-26T20:41:47  <acidsys> oh recursive even
2023-11-26T22:14:08  *** teepee_ is now known as teepee