2023-11-06T17:35:54 hi robin_listas, I (we) haven't, thanks for pointing it out, now it runs again 2023-11-06T17:37:46 Thanks. 2023-11-06T17:38:09 I don't know if it has been mentioned here, but it seems that mail lists are down 2023-11-06T17:40:02 Las mail I see was yesterday at 15 hours 2023-11-06T17:52:18 lists.o.o migration is not completed unfortunately 2023-11-06T17:52:31 also see https://status.opensuse.org 2023-11-06T17:59:25 hi cboltz, any chance you could the haproxy parts in my !936 .. the idea is to have a frontend which currently only serves ssh/http to pagure01 without "revealing" anything else, but allowing other miscellaneous services to be added to it in the future if needed 2023-11-06T18:07:07 Ah, ok. I thought it was just the /usual/ problem with the list server >;-) 2023-11-06T18:14:34 heh not this time :) 2023-11-06T18:51:21 acidsys: 936 looks good, at least i didn't find obvious errors 2023-11-06T18:54:59 neat, thanks for confirming 2023-11-06T19:03:51 rebased deadservices once more, I'll merge before you cause another conflict ;-) 2023-11-06T19:06:41 you better be quick :-) 2023-11-06T19:07:45 I am - it's already merged 2023-11-06T19:09:38 I try to bring lists.o.o back up, I deployed the certificate like the multiple other successful certificates I deployed today, but somehow HAProxy continues to serve our internal one 2023-11-06T19:09:42 https://paste.opensuse.org/pastes/26c7bcf78d89 any ideas? 2023-11-06T19:13:39 I noticed that you should also create a certificate for lists.uyuni-project.org - but that's probably not too related to lists.o.o 2023-11-06T19:14:12 I wondered as well but we don't have uyunu-project.org pointing to our nameservers .. 2023-11-06T19:14:27 uyuni* 2023-11-06T19:15:07 (and their current A record does not point to our NUE servers either) 2023-11-06T19:16:02 indeed, they host their own page on lists.uyuni-project.org (with links to lists.o.o) - so no cert needed on our side (but also no haproxy entry IMHO) 2023-11-06T19:17:09 right .. 2023-11-06T19:18:06 ah .. I think it is not reloading because Starting proxy ssh-pagure01: cannot bind socket (Address already in use) [2a07:de40:b27e:1204::13:22] 2023-11-06T19:18:15 I need to adjust sshd to only listen on 11/12 2023-11-06T19:18:23 *** kraih_ is now known as kraih 2023-11-06T19:19:53 yep, that did it 2023-11-06T19:20:27 now I get a securely transmitted error page. success 2023-11-06T19:23:37 curl -v mailman3.infra.opensuse.org (on atlas) says Connection refused for the v6 IP. v4 times out 2023-11-06T19:24:19 thx, applying my v6 patch atm 2023-11-06T19:25:54 can you create some certificates for the deadservices? See https://gitlab.infra.opensuse.org/infra/salt/-/merge_requests/918/diffs (atlas/services.sls) for the domain list 2023-11-06T19:28:55 sure, or do you prefer me to show you how 2023-11-06T19:29:17 show me ;-) 2023-11-06T19:29:28 ok, ssh to acme.infra.opensuse.org 2023-11-06T19:29:38 ah, lists.o.o is back :-) 2023-11-06T19:30:09 ok, I'm in 2023-11-06T19:30:24 edit /etc/dehydrated/our-domains.txt - either find an existing certificate to extend or make a new line for a new one. the first column is the CN and first SAN, the others are subsequent SANs 2023-11-06T19:30:43 in your case I suggest making something like "deadservice.opensuse.org $service1.opensuse.org $service2.opensuse.org etc" 2023-11-06T19:31:18 except that deadservice.o.o doesn't exist in DNS, which might be a problem 2023-11-06T19:31:28 that's not a problem 2023-11-06T19:31:41 we use DNS based ACME challenge so a domain does not need to exist in order for it to be eligible for a certificate 2023-11-06T19:31:53 ah, ok 2023-11-06T19:32:19 once your line looks good do `systemctl start --no-block dehydrated ; journalctl -f` and just confirm nothing ugly happens 2023-11-06T19:32:44 if all goes well it should issue the certificate and deploy it to atlas all in one go 2023-11-06T19:33:41 also, nice that lists.o.o is back, the theme looks like it got an update 2023-11-06T19:33:53 login still fails, checking 2023-11-06T19:35:36 ok, login repaired 2023-11-06T19:36:20 indeed, the header looks strange (and nearly unreadable) with white on light gray :-/ 2023-11-06T19:36:55 firefox says 200 for all requests - everything else would have been too easy 2023-11-06T19:36:56 now postorius complains about mailman API being down .. which correlates to the mailman service failing 2023-11-06T19:37:20 do you think the header looks strange? I think it looks new but not too shabby 2023-11-06T19:38:04 compare with news.o.o - the text should be black, not white 2023-11-06T19:38:43 certificate generation worked :-) 2023-11-06T19:39:06 for me the different size of the top bar is more distressing :p 2023-11-06T19:39:08 nice! 2023-11-06T19:40:27 right, full width vs. content width 2023-11-06T19:41:16 darix: when you have time .. I'm not sure it's another packaging problem but the mailman.service fails with lots of odd Alembic output such as " alembic.script.revision.ResolutionError: No such revision or branch '2156fc3f6f7d'" 2023-11-06T19:44:03 I run dup and changed some packages it suggested to o:i:mailman3 2023-11-06T19:44:23 but for example mailman3 itself is now from factory 2023-11-06T19:44:55 unrelated to this, I notice mx1 tries to chat to mailman3 on port 25 over the internet .. not sure what kinda mails it sends out via plaintext that it would have previously sent internally .. 2023-11-06T19:45:45 we should migrate mx* soon anyways, pjessen sounded like he's happy to help. but would be cool to know where I need to adjust this in the meanwhile 2023-11-06T19:49:09 salt/profile/mailserver/files/transport:lists.opensuse.org smtp:[mailman3.infra.opensuse.org] 2023-11-06T19:49:21 but note that only one mx is salted, and even that has a diff 2023-11-06T19:50:03 thanks .. 2023-11-06T19:50:11 * acidsys sighs and procees to hotpatch 2023-11-06T19:50:15 hmm, mx1 has this transport entry, so in theory it should use the internal way 2023-11-06T19:50:54 oh the behavior is somewhat expected, it resolves the AAAA record, which has a route, but not an internal one, only the one through the internet 2023-11-06T19:51:08 ah, ok 2023-11-06T19:51:21 then the hotpatch might be to use smtp:[$IP] 2023-11-06T19:51:21 because internally to/from NUE1 we only have IPv4 .. wouldn't have made sense to implement for the short tie m 2023-11-06T19:51:51 (+ a comment saying it's a temporary workaround to avoid that someone puts it in salt) 2023-11-06T19:51:54 I'll add a compat DNS record similar to what I did for MySQL 2023-11-06T19:56:59 not sure if that worked - mailq has lots of 2023-11-06T19:57:02 (delivery temporarily suspended: connect to legacy-ip.mailman3.infra.opensuse.org[172.16.164.159]:25: Connection refused) 2023-11-06T20:01:10 only just finished implementing .. now it should 2023-11-06T20:02:43 also did the same for the other entries in the transport file 2023-11-06T20:03:57 I restarted postfix on mx1, still same error in the log 2023-11-06T20:05:12 :^) 2023-11-06T20:06:00 when I do `nmap -p25 legacy-ip.mailman3.infra.opensuse.org` on mx1 I get "closed", but I also get " 2a07:de40:b27e:64::c0a8:2f5f.59544 > 2a07:de40:b27e:1203::b46.25: Flags [S], seq 2805422195, win 1024, options [mss 1460], length 0" on mailman3 2023-11-06T20:06:03 so the packet arrives 2023-11-06T20:06:28 maybe postfix on mailman3 needs some tuning to accept it? 2023-11-06T20:06:52 ooh I see .. it is listening on single stack socket 2023-11-06T20:07:15 "inet_protocols = ipv4" doh! 2023-11-06T20:07:44 changed to ipv6, now it shows as open correctly 2023-11-06T20:08:07 I have to offer some more fun: 2023-11-06T20:08:11 2023-11-06T20:10:00.884779+00:00 mailman3 postfix/smtp[8252]: 50871110: to=, relay=none, delay=1904, delays=1904/0.01/0.01/0, dsn=4.4.1, status=deferred (connect to mx1.infra.opensuse.org[2a07:de40:b27e:64::c0a8:2f5f]:25: Permission denied) 2023-11-06T20:08:20 2023-11-06T20:10:00.889288+00:00 mailman3 postfix/smtp[8234]: 7A67B81B: to=, orig_to=, relay=none, delay=4042, delays=4042/0.01/0.01/0, dsn=5.4.6, status=bounced (mail for mailman3.infra.opensuse.org loops back to myself) 2023-11-06T20:08:49 oh I see, it needs to chat in the other direction too, will fix 2023-11-06T20:09:50 :-) 2023-11-06T20:10:01 the bounce for that /dev/null mail end up at 2023-11-06T20:10:04 2023-11-06T20:10:00.907441+00:00 mailman3 postfix/smtp[8234]: D93821473: to=, relay=none, delay=0.02, delays=0/0/0.01/0, dsn=5.4.6, status=bounced (mail for mailman3.infra.opensuse.org loops back to myself) 2023-11-06T20:10:23 also not nice :-/ 2023-11-06T20:10:31 that /dev/null sounds like someone did it on purpose 2023-11-06T20:10:37 reverse traffic allowed as well now 2023-11-06T20:11:03 yes, but - _mail_ to /dev/null@...? 2023-11-06T20:11:13 I mean, &>/dev/null exists 2023-11-06T20:11:32 I think slash inside an email address violates some RFC 2023-11-06T20:12:43 right, but that's my smallest problem with it ;-) 2023-11-06T20:13:02 I stand corrected, according to wikipedia's summary of the RFC it is actually a valid email address 2023-11-06T20:13:42 * cboltz wonders what the RFC authors smoked 2023-11-06T20:14:05 well you can get some good ideas if you inhale /dev/null 2023-11-06T20:14:17 lol 2023-11-06T20:14:36 no more jitsi? :p 2023-11-06T20:14:56 anyway, progress - now we are at 2023-11-06T20:15:00 darix: coming 2023-11-06T20:15:00 2023-11-06T20:16:53.523554+00:00 mailman3 postfix/error[8539]: 7CAD22613: to=, relay=none, delay=0.01, delays=0.01/0/0/0, dsn=4.4.1, status=deferred (delivery temporarily suspended: connect to localhost[::1]:8024: Connection refused) 2023-11-06T20:16:10 I guess mailman should listen there, but it doesn't 2023-11-06T20:16:41 this is probably the failed mailman service, checking with darix now 2023-11-06T20:17:13 that one time you think oh cboltz could fix it and then nothing :p 2023-11-06T20:17:15 tststs 2023-11-06T20:17:36 ;-) 2023-11-06T20:17:52 I'll add two more bugreports ;-) 2023-11-06T20:18:02 mailman also wants to talk to mx2 -> firewall 2023-11-06T20:18:24 2023-11-06T20:20:00.472932+00:00 mailman3 postfix/smtp[8660]: 1CACD5AE: to=, relay=mx1.infra.opensuse.org[2a07:de40:b27e:64::c0a8:2f5f]:25, delay=1050, delays=1050/0.01/0.05/0.01, dsn=4.2.0, status=deferred (host mx1.infra.opensuse.org[2a07:de40:b27e:64::c0a8:2f5f] said: 450 4.2.0 : Client host rejected: Service temporarily unavailable, please retry later (in reply to RCPT TO command)) 2023-11-06T20:19:03 so mx1 doesn't like mailman3's IP (or misses a reverse DNS entry) 2023-11-06T20:19:12 oversight on my end, allowed now 2023-11-06T20:24:49 much better :-) 2023-11-06T21:07:33 darix: 2a07:de40:b27e::/48 2023-11-06T21:07:59 darix: 2a07:de40:b27e:1203::/64 (os-internal) 2023-11-06T21:13:10 4.0.2.1.e.7.2.b.0.4.e.d.7.0.a.2.ip6.arpa 2023-11-06T21:13:13 3.0.2.1.e.7.2.b.0.4.e.d.7.0.a.2.ip6.arpa 2023-11-06T21:21:08 16.172.in-addr.arpa 2023-11-06T21:41:45 perl-IO-Socket-INET6 perl-Socket6 2023-11-06T21:45:56 Compressing... Error parsing template socialaccount/signup.html: socialaccount/base.html 2023-11-06T21:56:23 hey, some email from lists.opensuse.org just arrived, first in over 18 hours 2023-11-06T22:00:49 indeed, we also got the first spam to admin@ again ;-) 2023-11-06T22:56:13 yep, lists.o.o is now fully back 2023-11-06T23:15:12 *** teepee_ is now known as teepee