2023-11-23T03:47:19 *** teepee_ is now known as teepee 2023-11-23T08:59:07 acidsys: are the forums directly using my package or via some intermediate project? 2023-11-23T09:06:57 hi darix, theoretically all from home:darix:apps 2023-11-23T09:07:16 practically I see they are still locked and (system packages) 2023-11-23T09:08:30 well yeah 2023-11-23T09:09:05 can you "zypper al discourse\* \*rubygem\*" ... working on updating the package and i want to prevent os-update from breaking the instance 2023-11-23T09:09:07 TIA 2023-11-23T09:10:08 ah ok, is already I think: https://paste.opensuse.org/pastes/a9846c603dcf but I can add the rubygem one 2023-11-23T09:10:48 https://www.heise.de/news/500-000-Sterne-Weltraumteleskop-JWT-zeigt-Zentrum-der-Milchstrasse-9537132.html :o 2023-11-23T09:14:54 hi pjessen and darix. yesterday did not work out :( can we work on mx today? 2023-11-23T09:15:06 i could migrate one of them now 2023-11-23T10:07:45 ok, mx1 shut now 2023-11-23T10:09:52 + mx-test 2023-11-23T10:34:02 i mean we could have prepared the config while it was still in nbg 2023-11-23T10:35:17 or did you merge the anna/elsa stuff into mx[12] mx-test already? 2023-11-23T10:45:27 I was preparing configs two days ago and there were no further requests after me stating it should be ready 2023-11-23T10:46:26 there is no special mail configuration for anna/elsa in salt. if there is manual configuration which needs to be transplanted I need to be told which parts 2023-11-23T13:07:19 *** teepee_ is now known as teepee 2023-11-23T13:10:52 mx-test has lots of localhost ipv4 ... 2023-11-23T13:13:07 maybe a question for pjessen? i was never involved with that machine. 2023-11-23T13:15:59 after a lot of replacing of custom /etc/hosts entries with generic {ipv6-,}localhost postfix now came up 2023-11-23T13:16:53 next up it tries to chat with mailman but I'm not sure I should "release" it into the wild yet, will wait for Per 2023-11-23T13:18:31 darix: with PROXY do we terminate TLS at HAProxy or at the Postfix backend ? 2023-11-23T13:21:40 also how will we have multiple mx by one haproxy? one frontend for each? or do you want us to do single-MX and "hidden" balancing? 2023-11-23T13:31:40 also it seems we only had port 25 open in the past, no 465 for SMTPS 2023-11-23T13:39:56 at postfix 2023-11-23T13:40:11 smtps aka submission is not really used for mail server to mail server traffic 2023-11-23T13:40:19 they use 25 + starttls 2023-11-23T13:40:25 which also answers your first question 2023-11-23T13:46:30 ok what about the balancing? will we still have public mx1 and mx2 ? 2023-11-23T13:47:56 either works 2023-11-23T13:48:32 having one MX IP on each proxy node which then proxies to both MX VMs 2023-11-23T13:48:49 has the advantage that if one proxy node has problems the other still works 2023-11-23T13:52:53 hm okey so you mean instead of using the VIP we add one individual address mapped to public mx1 and mx2 respectively 2023-11-23T13:53:09 but then to still have both mx servers as backends on each proxy 2023-11-23T13:54:08 guess it sounds reasonable just needs a bit more shuffling in salt as we currently always assume both proxies to have the same config 2023-11-23T14:09:59 i would do the same too 2023-11-23T14:10:19 if you have the sysctl set for allow non local bind 2023-11-23T14:10:29 then it doesnt matter if both proxies listen to ip1 and ip2 2023-11-23T14:10:39 as long as each IP is only bound to one server 2023-11-23T14:10:54 haproxy will only handle connections via the locally bound IP 2023-11-23T14:13:28 right .. but I can't apply the same for the network interface config 2023-11-23T14:16:05 https://gitlab.infra.opensuse.org/infra/salt/-/merge_requests/1074 2023-11-23T14:16:12 * acidsys summons cboltz 2023-11-23T14:27:35 correct 2023-11-23T14:27:42 but you dont need it for the network config 2023-11-23T14:35:56 I do, because I cannot have two devices with the same IP address in my network 2023-11-23T14:36:20 nobody said you should do that 2023-11-23T14:36:30 i said you should have _bind_ lines in haproxy for both IPs 2023-11-23T14:36:43 that does not imply that you also assign the 2 IPs on both nodes 2023-11-23T14:36:53 hence the sysctl setting for non local binds 2023-11-23T14:42:03 network config != haproxy config 2023-11-23T14:42:05 anyways 2023-11-23T17:03:48 looks like something broke... 2023-11-23T17:04:51 * malcolmlewis go makes another coffee... 2023-11-23T17:33:31 no forums again 2023-11-23T17:38:54 ah I see what happened 2023-11-23T17:39:03 a-865k, not just the forums.... 2023-11-23T17:40:41 back 2023-11-23T17:40:45 ? 2023-11-23T17:41:09 acidsys, yup 2023-11-23T17:41:18 ok thx for confirming .. sigh 2023-11-23T17:41:55 acidsys, there were a few others, progress, matrix etc 2023-11-23T17:42:05 I see they are back 2023-11-23T18:53:19 can't figure this out, google search is dead end: mx1 postfix/postscreen[9315]: fatal: service mx1.infra.opensuse.org:smtp requires a process limit of 1 2023-11-23T18:58:32 master.cf 2023-11-23T18:58:58 8 # ========================================================================== 2023-11-23T18:58:58 9 # service type private unpriv chroot wakeup maxproc command + args 2023-11-23T18:58:58 10 # (yes) (yes) (no) (never) (100) 2023-11-23T18:58:58 11 # ========================================================================== 2023-11-23T18:59:05 that's the columns in master.cf 2023-11-23T18:59:18 "smtp inet n - n - 1 postscreen" 2023-11-23T18:59:26 if you want to use postscreen 2023-11-23T18:59:31 see the 2nd to last column? 2023-11-23T19:00:00 acidsys: does that help? 2023-11-23T19:00:38 perfect, thank you! 2023-11-23T19:02:12 postscreen_upstream_proxy_protocol = 2023-11-23T19:02:12 smtpd_upstream_proxy_protocol = 2023-11-23T19:02:15 btw 2023-11-23T19:02:19 you do not have to use postscreen 2023-11-23T19:02:24 you can also use smtpd 2023-11-23T19:02:31 both support proxy protocol 2023-11-23T19:02:32 HTH 2023-11-23T19:03:21 oh ok I read online that it is required to use postscreen for PROXY support 2023-11-23T19:03:38 any benefit from using postscreen if both work? 2023-11-23T19:04:10 postscreen can do some of those "spam" checks that are built into postfix a bit more efficient 2023-11-23T19:04:27 but if you just want to switch to "works behind haproxy" 2023-11-23T19:04:37 then all you need is smtpd_upstream_proxy_protocol 2023-11-23T19:04:53 oh okay thanks then let me try regular smtpd for now 2023-11-23T19:08:16 any idea why smtp.mail.yahoo.com returns SERVFAIL against hel1? I also tried without dnssec with no luck :( 2023-11-23T19:08:34 against other public dns servers it resolves fine 2023-11-23T19:14:01 I think the issue might be when it tries to reach upstream NS which only return an A record 2023-11-23T19:14:36 https://dnssec-analyzer.verisignlabs.com/smtp.mail.yahoo.com 2023-11-23T19:15:42 tbh their dnssec looks fucked up 2023-11-23T19:20:16 hmm 2023-11-23T19:20:18 host smtp.mail.yahoo.com 2023-11-23T19:20:19 smtp.mail.yahoo.com is an alias for smtp.mail.global.gm0.yahoodns.net. 2023-11-23T19:20:19 smtp.mail.global.gm0.yahoodns.net has address 87.248.97.36 2023-11-23T19:20:27 my pdns-recursor can resolve it 2023-11-23T19:20:37 can hel connect via ipv4 to the internet? 2023-11-23T19:22:57 not quite that's what I think is the issue 2023-11-23T19:23:02 because it needs to do a lookup against itself to determine whether to do DNS64 which is a catch 22 if it needs that info to look it up 2023-11-23T19:23:28 so I think I need a IPv4 capable recursor for such queries 2023-11-23T19:25:28 maybe those pdns-recursor would be better on asgard where they have native v4? 2023-11-23T19:25:37 would give you a lot less trouble? 2023-11-23T19:27:44 atlas or some other machine in the public segment could work 2023-11-23T19:37:11 next issue is lots of this https://paste.opensuse.org/pastes/0b30643a1f7a ... that is without smtpchk. if I enable smtpchk on the HAProxy backends it thinks postfix is down (curl mx1:25 works) 2023-11-23T19:41:31 can you paste me the whole proxy block for postfix? 2023-11-23T19:42:18 https://paste.opensuse.org/pastes/885f2a5296a2 2023-11-23T19:43:37 I followed https://www.haproxy.com/blog/efficient-smtp-relay-infrastructure-with-postfix-and-load-balancers#configuration (just adapted postscreen to smtpd after your input) 2023-11-23T19:47:41 youc an make that into a listen block 2023-11-23T19:47:56 then you can kill 6 to 9 2023-11-23T19:48:15 in your server line 2023-11-23T19:48:20 it lacks "send-proxy" 2023-11-23T19:48:26 or send-proxy-v2 2023-11-23T19:48:34 depending if postfix supports v2 2023-11-23T19:49:41 acidsys: like this https://gist.github.com/darix/228f20b79f113d49766f242bb8631f36 2023-11-23T19:51:22 send-proxy-v2 2023-11-23T19:51:26 postfix supports this 2023-11-23T19:51:37 https://www.postfix.org/postconf.5.html#smtpd_upstream_proxy_protocol 2023-11-23T19:53:32 ok thanks let me try 2023-11-23T20:09:06 https://paste.opensuse.org/pastes/02f69d7fca74 2023-11-23T20:09:28 still says mx1 is down :( 2023-11-23T20:15:30 in main.cf I put "smtpd_upstream_proxy_protocol = haproxy" and master.cf is https://github.com/openSUSE/heroes-salt/blob/production/salt/profile/mailserver/files/master.cf 2023-11-23T20:25:44 ok, at least it seems to work when disabling the health check 2023-11-23T20:26:18 even though there's still the occasional "warning: haproxy read: timeout error" .. not sure that's normal 2023-11-23T20:31:43 *** teepee_ is now known as teepee 2023-11-23T21:02:51 now there's also lots of: warning: proxy ipv6-localhost:10024 rejected "connection request": "421 4.3.0 mx1.opensuse.org Server local error" .. on [::1]:10024 is the spampd 2023-11-23T21:15:35 so there's various warnings but it seems to send out at least some of the mail. I also allowed it to chat with mailman3 now but not sure how to tell whether mailman is successfully using mx1 or going via mx2 2023-11-23T21:17:50 I give up for now 2023-11-23T23:08:58 FTR, el_comrado on matrix mentioned https://survey.opensuse.org is *very* slow