2023-11-25T01:47:44  *** teepee_ is now known as teepee
2023-11-25T09:24:35  <pjessen> fwiw, I sent an email to my opensuse address, it went thru, but is now stuck on mx1 (cannot deliver)
2023-11-25T09:25:13  <pjessen> connect to katharina.spamchek.net[185.85.248.19]:25: Connection timed out
2023-11-25T14:18:49  <acidsys> hi pjessen, repaired, better?
2023-11-25T14:20:50  <acidsys> relay=katharina.spamchek.net[185.85.248.24]:25, delay=92126, delays=92125/0.03/0.45/0.15, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as D3D20404DD) looks good
2023-11-25T14:23:59  <darix> acidsys: what was it?
2023-11-25T14:24:10  <darix> acidsys: are we now in a state that we can move mx2?
2023-11-25T14:26:15  <acidsys> outbound smtp via v4
2023-11-25T14:26:31  <acidsys> I guess? but would be good to confirm
2023-11-25T14:26:50  <acidsys> so we can enable internet traffic on mx1 before shutting mx2
2023-11-25T14:27:33  <darix> uhm
2023-11-25T14:27:41  <darix> but it couldnt reach tengu.nordisch.org via v6either?
2023-11-25T14:27:44  <darix> is that working now?
2023-11-25T14:28:45  <acidsys> https://paste.opensuse.org/pastes/8a9f201a080f
2023-11-25T14:30:07  <acidsys> my server works now, not sure about yours, Per explained that it only allows sending to @opensuse, your address hence expectedly returns relay access denied
2023-11-25T14:30:26  <darix> well try to swaks me something and we shall see
2023-11-25T14:31:00  <acidsys> I just said it returns relay access denied
2023-11-25T14:31:16  <acidsys> https://paste.opensuse.org/pastes/b806c192fd0f
2023-11-25T14:39:10  <darix> well i would run it locally on mx1 :)
2023-11-25T14:40:38  <acidsys> localhost is the same
2023-11-25T14:40:53  <acidsys> but works with --server tengu.nordisch.org if that helps
2023-11-25T14:48:53  <robin_listas> Hi, I think mail list is down.
2023-11-25T14:58:59  <acidsys> hi robin_listas, checking ..
2023-11-25T14:59:37  <robin_listas> At least my mail is not appearing.
2023-11-25T15:00:35  <acidsys> I see the problem, it tries to chat with mx1.i.o.o (which is now "responsive") directly instead of to mx1.o.o (i.e. not going via the proxy on a listener which only does proxy protocol)
2023-11-25T15:00:58  <darix> you could setup something like mx.infra.o.o on hel?
2023-11-25T15:00:59  <acidsys> not sure why it does not fall back to mx2 but let me try to fix the mx1 issue first
2023-11-25T15:01:03  <darix> and make all machines go through that?
2023-11-25T15:01:33  <acidsys> right I was thinking to ask you if it wouldn't make sense for sending mail from the LAN to make another proxy on Hel
2023-11-25T15:01:51  <darix> it does
2023-11-25T15:02:16  <darix> and then you could limit lan internal port 25 access to only allow hel
2023-11-25T15:02:48  <acidsys> cool I like that, let me set it up
2023-11-25T15:02:57  <acidsys> also as for why mailman does not fall back to mx2
2023-11-25T15:03:05  <acidsys> transport map: opensuse.org    smtpint:[mx1.infra.opensuse.org]
2023-11-25T15:03:20  <acidsys> so if we have "mx.infra.opensuse.org" HA will also be solved  :-)
2023-11-25T15:03:52  <acidsys> let me set that up
2023-11-25T15:04:13  <darix> [] means no MX lookup IIRc
2023-11-25T15:04:53  <acidsys> yes it does
2023-11-25T15:08:59  <acidsys> for Hel, can I use the same listening addresses in the SMTP listens block as I already use in the http one?
2023-11-25T15:10:05  <darix> sure
2023-11-25T15:10:35  <acidsys> cool
2023-11-25T15:11:35  <darix> internally it doesnt matter
2023-11-25T15:14:12  <darix> when this is done acidsys will be an haproxy addict
2023-11-25T15:15:31  <darix> did you know that haproxy is on the distro because we showcased on the original alice setup for opensuse how useful it is?
2023-11-25T15:15:47  <acidsys> :) ye I think you told me
2023-11-25T15:16:07  <acidsys> it's cool it even made it to the SLE HA extension
2023-11-25T15:17:56  <acidsys> btw! we can just update the relay.infra.opensuse.org dns record to be a cname to hel.infra.opensuse.org and we do not even need any client side changes for "cron mail"
2023-11-25T15:18:41  <darix> yeah
2023-11-25T15:19:11  <darix> haproxy does the magic for us then ^^
2023-11-25T15:24:09  <acidsys> change is deployed, mx1 is up on hel. updated relay.i.o.o CNAME + i.o.o MX
2023-11-25T15:24:28  <acidsys> now let me try if it works for mailman too
2023-11-25T15:31:43  <acidsys> there's lots of "Relay access denied; from=<factory-bounces@lists.opensuse.org> to=<various user email addresses>"
2023-11-25T15:32:30  <acidsys> "swaks --to georg@syscid.com --from crameleon@opensuse.org --server localhost" from mailman3 works
2023-11-25T15:32:43  <acidsys> but I get a mailer daemon with relay access denied
2023-11-25T15:33:09  <acidsys> I think I need to add mailman3.i.o.o to mx{1,2} mynetworks
2023-11-25T15:33:17  <acidsys> ?
2023-11-25T15:37:32  <darix> probably
2023-11-25T15:37:39  <darix> or relay networks
2023-11-25T15:37:42  <darix> were code.o.o is alread
2023-11-25T15:44:30  <darix> acidsys: did relay domains work?
2023-11-25T15:44:42  <darix> and does that mean we can now do mx2?
2023-11-25T15:44:43  <darix> also
2023-11-25T15:45:01  <darix> we should have a quick chat how this whole daffy and id.o.o move should be handled
2023-11-25T15:54:55  <acidsys> it does not work, I'm not sure how it works with spampd. because our "mynetworks" option is under the localhost:10025 listener and I figure it makes more sense underneath the external listener
2023-11-25T15:55:12  <acidsys> but then again it works from outside/atlas as well this way
2023-11-25T15:56:17  <acidsys> for now I stopped postfix on mailman3 as to hopefully not flood users with bounce messages :|
2023-11-25T15:59:38  <acidsys> hm putting "-o mynetworks=[2a07:de40:b27e:1204::]/64,[2a07:de40:b27e:1203::]/64" under the external listener does not yield success with sending through hel either
2023-11-25T16:01:29  <darix> uhm
2023-11-25T16:01:41  <darix> did you have mynetworks settings in master.cf already?
2023-11-25T16:01:51  <darix> or just in main.cf
2023-11-25T16:01:58  <darix> i think you can have those in main.cf
2023-11-25T16:02:02  <acidsys> we only have it in master.cf
2023-11-25T16:02:08  <acidsys> so far only under the 10025 listener
2023-11-25T16:02:09  <darix> uhm
2023-11-25T16:02:11  <acidsys> which is the input from spampd
2023-11-25T16:02:17  <darix> that is something else
2023-11-25T16:02:24  <darix> that only needs localhost
2023-11-25T16:02:33  <acidsys> ok
2023-11-25T16:02:36  <darix> infra?
2023-11-25T16:02:41  <acidsys> ok
2023-11-25T16:08:32  *** teepee_ is now known as teepee
2023-11-25T16:24:01  <darix> pjessen cboltz: on anna/elsa as well on mx1 there is a lot of extra smtp services defined in master.cf ... could you explain what they do?
2023-11-25T16:27:25  <darix> also could you merge the relevant settings from your anna/elsa setup into the mx1 config? in doubt via -o options in the :26 block in master.cf
2023-11-25T16:37:51  <pjessen> darix: I see two extra - smtpslow and smtcox. they are rate-limited deliveries
2023-11-25T16:38:15  <pjessen> that was for mx1, I expect I have set up similar on anna/elsa
2023-11-25T16:38:25  <darix> well
2023-11-25T16:38:37  <darix> can you do us the favor and port the relevant settings from anna/elsa over?
2023-11-25T16:38:48  <darix> so acidsys and me do not have to reverse engineer things?
2023-11-25T16:38:52  <pjessen> sure
2023-11-25T16:40:07  <pjessen> umm, port to atlas12 ?
2023-11-25T16:40:22  <darix> to mx1 port master.cf :26 entry
2023-11-25T16:41:48  <pjessen> okay, let me check what we have on anna,
2023-11-25T16:42:06  <pjessen> ahhh, I see.
2023-11-25T16:42:41  <darix> is it on purpose that you scan outgoing mails from mailman?
2023-11-25T16:45:01  <pjessen> when they're coming back in?
2023-11-25T16:45:05  <darix> yeah
2023-11-25T16:45:19  <darix> this could be a bug due to the removal of the internet interface
2023-11-25T16:45:28  <pjessen> no, not intentional, just never saw a reason to skip
2023-11-25T16:45:36  <darix> well you double scan
2023-11-25T16:45:48  <darix> once when it goes from internet to mailman and then again when it leaves
2023-11-25T16:46:04  <pjessen> only for deliveries to opensuse.org though
2023-11-25T16:46:22  <pjessen> besides, mails can also be sent via the web interface
2023-11-25T16:46:41  <pjessen> those are a nuisance in fact
2023-11-25T16:48:00  <darix> ok
2023-11-25T17:08:18  <acidsys> just for clarification for pjessen, for relay.infra.opensuse.org we will now use a proxy on our internal HAProxy pair (hel.i.o.o), similar to how we have a proxy on atlas.i.o.o for mx{1,2}.o.o
2023-11-25T17:08:53  <acidsys> currently the proxy on hel goes to the same :25 listener on mx1 which atlas uses as well
2023-11-25T17:10:24  <acidsys> if the double spam filter concern from darix is a non-issue we can keep it this way and don't need the additional :26 listener
2023-11-25T17:18:18  <darix> acidsys: the better question is if you also want this for all the mails that formerly went to relay.infra.o.o
2023-11-25T17:20:06  <pjessen> acidsys: tnx.
2023-11-25T17:20:15  <a-865k> https://status.opensuse.org/ doesn't suggest why I've seen no mailing list posts in over 12 hours: none from support, user, factory, mirror, project, buildservice, kernel, announce, heroes, etc.
2023-11-25T17:26:42  <darix> acidsys: and then it might be easier to have an relay. config on hel:25 ... and a hel:26 for mailman
2023-11-25T17:27:10  <acidsys> in the current configuration mail from all internal machines (going hel:25 -> mx{1,2}:25) will go through the spamfilter. this is good, no?
2023-11-25T17:28:19  <darix> if all services behind it expect such behavior sure
2023-11-25T17:28:37  <darix> i just want to make sure that this is really what the heroes team once. as it is a change compared to the previous setup
2023-11-25T17:30:55  <pjessen> a-865k: because we're working on the mail setup
2023-11-25T17:31:39  <pjessen> wrt spam - anything that goes to an opensuse.org address is scanned.
2023-11-25T17:32:08  <acidsys> what about me sending mail to my non-opensuse.org address from an arbitrary internal machines?
2023-11-25T17:36:28  <pjessen> that is not scanned, we don't scan anything on outbound
2023-11-25T17:36:45  <acidsys> okay
2023-11-25T17:37:12  <pjessen> I've made some adjustments on mx1, main.cf, master.cf and the ratelimit table.
2023-11-25T17:37:24  <darix> short nap time
2023-11-25T17:37:34  <pjessen> alwaysa good idea.
2023-11-25T17:37:36  * acidsys puts darix to bed in the server room
2023-11-25T17:37:58  <acidsys> pjessen: okay, let me check
2023-11-25T17:38:34  <acidsys> do we still want the additional :26 listener now / shall I set the hel.i.o.o proxy to go there instead of to the :25 one?
2023-11-25T17:38:48  <acidsys> i.e. for all email going via relay.i.o.o to not pass the spam filter including mailman
2023-11-25T17:38:59  <pjessen> sounds good to me. both.
2023-11-25T17:39:15  <acidsys> ok great, patching that
2023-11-25T17:42:08  <acidsys> can I set "mx1.infra.opensuse.org:26        inet" to only listen on the IPv6 socket without hardcoding the IP address?
2023-11-25T17:42:14  <acidsys> because hel does not need the v4 listener
2023-11-25T17:45:29  <pjessen> yes, you can add inet_protocol=ipv6  (see example on mx1)
2023-11-25T17:45:40  <pjessen> yes, you can add "-o inet_protocol=ipv6"  (see example on mx1)
2023-11-25T17:46:29  <darix> Or set all and it does both
2023-11-25T17:46:33  <darix> Bin
2023-11-25T17:50:30  <acidsys> hm does not seem to budge, but not a problem for now, does not impact functionality
2023-11-25T17:50:45  <acidsys> proxy is configured to go to mx1:26 now
2023-11-25T17:51:03  <acidsys> swaks from mailman3 "--to hel" works
2023-11-25T17:51:36  <acidsys> as for mailman3 "--to localhost", can you check if /etc/postfix/transport is correct pjessen ?
2023-11-25T17:53:33  <pjessen> acidsys: looks good
2023-11-25T17:55:01  <acidsys> okay, then let's start postfix on mailman3 again ..
2023-11-25T17:55:05  * acidsys braces
2023-11-25T17:55:34  <acidsys> cool, swaks to localhost works as well
2023-11-25T17:55:49  <acidsys> mx1 postfix/smtp[19050]: connect to discourse01.infra.opensuse.org[2a07:de40:b27e:1203::b47]:25: Permission denied
2023-11-25T17:56:00  <acidsys> does mx1 need to talk "back" to discourse?
2023-11-25T17:56:43  <pjessen> sometimes it needs to deliver mails, I think. bounces maybe?
2023-11-25T17:57:18  <pjessen> maybe people can also participate using email?
2023-11-25T17:58:34  <pjessen> forums.o.o is listed as a relay_domain
2023-11-25T17:59:11  <acidsys> oh okay, so then I should allow mx{1,2}.i.o.o -> port 25 on all hosts related to domains listed in relay_domains
2023-11-25T18:06:06  <pjessen> that sounds about right
2023-11-25T18:10:47  <acidsys> okay, done
2023-11-25T18:11:45  <acidsys> the "mynetworks" we added to mx1's main.cf + the addition of "permit_mynetworks" to smtpd_recipient_restrictions goes together with your changes?
2023-11-25T18:13:29  <pjessen> this looks a little limited: mynetworks = [2a07:de40:b27e:1204::]/64,[2a07:de40:b27e:1203::]/64
2023-11-25T18:15:34  <pjessen> I think at least it ought to include localhost
2023-11-25T18:16:19  <acidsys> a right
2023-11-25T18:17:12  <acidsys> okay
2023-11-25T18:17:35  <acidsys> seems quite ok with me, what about allowing inbound smtp from the wild internet now?
2023-11-25T18:19:24  <acidsys> cboltz, where are you, I'm accumulating quite a MR queue :P
2023-11-25T18:21:22  <pjessen> my test mails worked fine, after you fixed something for the outbound delivery.
2023-11-25T18:21:31  <acidsys> I did?
2023-11-25T18:21:42  <acidsys> sorry, I mean, yes, right, totally I fixed all the things
2023-11-25T18:21:49  <pjessen> you didn't?
2023-11-25T18:22:00  <pjessen> haha,
2023-11-25T18:22:13  <acidsys> ah maybe earlier when I allowed outbound SMTP via IPv4
2023-11-25T18:22:16  <acidsys> for access to spamhaus
2023-11-25T18:22:27  <pjessen> and to me .....
2023-11-25T18:23:15  <pjessen> I think we're ready for the big wild internet
2023-11-25T18:23:26  <acidsys> great! let me open the walls
2023-11-25T18:23:50  <pjessen> mx1 had ~500 mails queued before, not it is only 25.
2023-11-25T18:24:08  <pjessen> now it is only 25.  I am taking a typing course next week
2023-11-25T18:24:42  <acidsys> :) by the way, I also fixed one problem with DNS64 (mx{1,2} have native IPv4 hence should not receive fake AAAA records)
2023-11-25T18:25:00  <acidsys> ok, internet is open, and the postfix journal seems to confirm that
2023-11-25T18:25:18  <pjessen> Kruzifix!  you opened the floodgates allright
2023-11-25T18:25:29  * acidsys hands pjessen an inflatable boat
2023-11-25T18:25:47  <pjessen> acidsys: no pump????
2023-11-25T18:26:10  <acidsys> pump already washed away
2023-11-25T18:27:10  <pjessen> i'm following the mail log, it looks good, but just a gut feeling
2023-11-25T18:27:21  <acidsys> mx1 postfix/smtpd[19397]: warning: hostname smtp-out1.suse.de does not resolve to address 2001:67c:2178:6::1c
2023-11-25T18:27:24  <acidsys> is this a SUSE problem?
2023-11-25T18:28:15  <pjessen> hmm, wait
2023-11-25T18:28:43  <pjessen> are we getting an traffic from 2001:67c:2178:6::1c?
2023-11-25T18:29:15  <pjessen> hmm, we are.
2023-11-25T18:29:46  <acidsys> domain name pointer smtp-out1.suse.de.
2023-11-25T18:29:46  <pjessen> well, 2001:67c:2178:6::1c = smtp-out1.suse.de, but smtp-out1.suse.de != 2001:67c:2178:6::1c
2023-11-25T18:30:12  <acidsys> maybe that's their (ehrm, "our", with my corp hat on) pre-migration address
2023-11-25T18:30:37  <pjessen> yeah
2023-11-25T18:30:56  <acidsys> ok at least not on our openSUSE end, then I can ask my colleagues about it next week
2023-11-25T18:30:56  <pjessen> I don't think it is a problem.
2023-11-25T18:31:18  <acidsys> you have some more minutes to validate your gut feelings while transplant your additions to Salt. then I can shut/migrate mx2 and apply the same configuration there
2023-11-25T18:31:39  <pjessen> muchos gracias
2023-11-25T18:34:30  <pjessen> it is looking good to me.  atlas12 are connecting far too often and spamming the logs, but we can get rid of the noise later.
2023-11-25T18:37:23  <pjessen> I see some gmail bounces due to DKIM, but that may not be our problem.
2023-11-25T18:37:41  <robin_listas> Just now I got an email from Bugzilla "New", in quatriplicate. Date says Nov 23, but my ISP says today, 19:27 or 19:28.
2023-11-25T18:38:43  <acidsys> re: atlas1,2 yes that's the backend health check; by default it checks every second. I already have a patch in the pipeline which reduces the frequency a bit
2023-11-25T18:46:12  <pjessen> maybe we can just suppress the logging.
2023-11-25T18:49:39  <acidsys> that'd be great, because I don't want to set the interval too high so HAProxy can detect when a backend is offline reasonably fast
2023-11-25T18:49:50  <acidsys> as to switch to the second one
2023-11-25T18:50:22  <pjessen> makes sense.  I'm sure syslog-something can suppress the noise
2023-11-25T18:50:28  <acidsys> here's the import of the changes I found, if you want to check https://gitlab.infra.opensuse.org/infra/salt/-/merge_requests/1109/diffs
2023-11-25T18:51:37  <pjessen> tnx, looks good.
2023-11-25T18:51:53  <acidsys> do you want to click approve ;)
2023-11-25T18:54:49  <pjessen> omg
2023-11-25T18:55:59  <pjessen> i clicked something
2023-11-25T18:55:59  <acidsys> thank you!
2023-11-25T18:56:13  <acidsys> on your way to becoming a salt expert
2023-11-25T18:57:26  <acidsys> shutting mx2 now
2023-11-25T19:01:16  <acidsys> regarding robin_listas's question, are we sending mail for bugzilla?
2023-11-25T19:01:27  <acidsys> I think bugzilla uses SUSE mail servers
2023-11-25T19:03:32  <robin_listas> Yes, it passes mx1.opensuse.org. Do you want full headers somewhere?
2023-11-25T19:04:26  <robin_listas> Maybe because the address I registered is the o.o alias.
2023-11-25T19:06:43  <acidsys> ah okey that makes more sense. I assume it's just fallout from migration but per should be able to judge better
2023-11-25T19:07:23  <robin_listas> The curious thing, to me, is not the delay, but that I got 4 copies.
2023-11-25T19:14:54  <pjessen> I think that is a redmine issue.
2023-11-25T19:24:23  <acidsys> "postlog   unix-dgram n  -       n       -       1       postlogd was in Salt but not on mx1". should it be deployed again or can I remove it from Salt?
2023-11-25T19:24:33  <acidsys> also how does redmine interface with mails to/from bugzilla
2023-11-25T19:25:40  <acidsys> re: postlog, I notice now it's actually just a duplicate, need to dedupe
2023-11-25T19:28:21  <pjessen> cool. gotta leave you, dinner is coming up.
2023-11-25T19:29:08  <acidsys> enjoy, thank you for the help, great progress
2023-11-25T19:56:36  <acidsys> mx2 is now live from Prague as well
2023-11-25T19:56:49  <acidsys> does that mean we finished the mail server migration? \o/
2023-11-25T20:08:11  <acidsys> same configuration applied on mx-test as well
2023-11-25T21:54:47  <acidsys> hi, my admin-auto@ subscription was canceled because "it has received a number of bounces indicating that there may be a problem delivering messages to georg@syscid.com."
2023-11-25T21:54:57  <acidsys> I now subscribed again, someone approve it please :)
2023-11-25T21:55:22  <acidsys> I'm not sure what the issue was, I am subscribed to various other mailing lists with this address, also openSUSE ones
2023-11-25T21:56:18  <cboltz> admin-auto probably has more traffic which also means more possible bounces
2023-11-25T21:57:03  <acidsys> oh okay will need to see which ones bounce because I'm not "missing" any other mailing list emails
2023-11-25T21:57:38  <cboltz> BTW: looks like there was some mail duplication - i got up to 5 copies of some gitlab.i.o.o mails (to my @o.o address) today
2023-11-25T21:58:04  <cboltz> I know you waited for my review, but one copy of each mail would still have been a quite full inbox ;-)
2023-11-25T21:59:09  <acidsys> that sounds similar to what robin_listas reported above with their mails from bugzilla
2023-11-25T21:59:20  <acidsys> where pjessen said it was some redmine problem? which does not sound quite right
2023-11-25T21:59:47  <acidsys> I counted about 20 MR's from me today, so yes :P
2023-11-25T22:00:45  <cboltz> for the minion groups - what's the syntax to target for example all narwals?
2023-11-25T22:03:01  <acidsys> for sample `salt -N narwal state....`
2023-11-25T22:04:48  <acidsys> JFYI short for `state.highstate test=True` is `state.test`  :)
2023-11-25T22:05:14  <cboltz> nice, that saves some typing :-)
2023-11-25T22:05:27  <acidsys> also from pressing enter too early :P
2023-11-25T23:08:12  <cboltz> FYI: (static) mirrors.o.o deployed, but I'm too tired to do some follow-up work to get it really working (will do tomorrow if nobody is faster)
2023-11-25T23:08:31  <cboltz> - change the mirrors.o.o DNS entry to CNAME proxy-prg2
2023-11-25T23:08:55  <cboltz> - while on it, drop mirrors-static.o.o from DNS (its lifetime was too short to be worth a redirect or deprecation page)
2023-11-25T23:08:58  <cboltz> - update status.o.o
2023-11-25T23:09:37  <cboltz> - drop traces of mirrors-static from narwal* (in /srv/www/vhosts/ and /home/web_static/git/)
2023-11-25T23:10:22  <cboltz> with that said- good night!
2023-11-25T23:10:27  <acidsys> DNS ok, waiting a bit with the status.o.o update for the TTL to expire
2023-11-25T23:10:29  <acidsys> good night