2021-03-28T14:42:11  <malcolmlewis> a-865k, new snapshot released, and there is the 4.8.26 package of mc available.
2021-03-28T17:11:46  <robin_listas> <offtopic@lists.opensuse.org>: unable to look up host     mailman3.infra.opensuse.org: No address associated with hostname
2021-03-28T17:11:57  <robin_listas> Just now.
2021-03-28T17:20:46  <robin_listas> Even the admin address mail bounces.
2021-03-28T17:29:42  <openSUSEBot> <LCP> that sounds like a broken ns
2021-03-28T17:30:09  <openSUSEBot> <LCP> that would indeed be very bad™
2021-03-28T17:38:29  <lcp> no, that works just fine
2021-03-28T17:38:51  <lcp> let's see, mailman3 machine is up
2021-03-28T17:40:36  <lcp> mailman itself is also up
2021-03-28T17:51:01  <lcp> and delivery works
2021-03-28T17:51:09  <lcp> uh
2021-03-28T17:55:51  <robin_listas> I emailed the two bounces to Per. Maybe it affects only emails sent from gmx.es, not apparently gmail.com
2021-03-28T18:01:04  <robin_listas> Mail to admin submitted using o.o alias, which is sent via gmail.
2021-03-28T18:03:57  <lcp> ah
2021-03-28T18:04:36  <robin_listas> I just sent a test email to test list, it went through.
2021-03-28T18:04:40  <cboltz> I can see the problem in the log (12 times on mx1, 16 times on mx2), seems to happen randomly (while most mails get delivered) and with various senders
2021-03-28T18:05:02  <robin_listas> Ah, random. I hate random problems.
2021-03-28T18:05:45  <cboltz> this one is rare and random which makes debugging even more interesting[tm]
2021-03-28T18:07:13  <cboltz> (850 successful deliveries to mailman3 vs. 28 failures)
2021-03-28T18:08:15  <robin_listas> The other day I had random problems when loading https://connect.opensuse.org/pg/members/all/.
2021-03-28T18:09:33  <robin_listas> Yes, got hit now. "Welcome to Elgg. / Elgg couldn't connect to the database using the given credentials."
2021-03-28T18:09:53  <robin_listas> Random.
2021-03-28T18:11:26  <robin_listas> Not 50%
2021-03-28T18:11:58  <robin_listas> Sorry, I have to leave.
2021-03-28T18:51:32  <pjessen> cboltz: that DNS issue only started on 26 March, at 1705 UTC.  Something was changed, somewhere. Friday 1800 CET ?
2021-03-28T18:56:15  <a-865k> malcolmlewis: soic :)
2021-03-28T18:58:50  <cboltz> the DNS setup changed in the last days (using powerdns on chip.i.o.o as master, obsoleting DNS management in FreeIPA)
2021-03-28T18:59:00  <cboltz> mx* use anna and elsa as resolver
2021-03-28T18:59:43  <cboltz> and the dnsmasq config on anna (and probably also elsa) lists two nameservers for infra.o.o - freeipa and chip
2021-03-28T19:01:14  <cboltz> freeipa still answers DNS queries, but it will become outdated sooner or later
2021-03-28T19:01:25  <pjessen> well, whoever changed the DNS setup will surely fix it :-)
2021-03-28T19:01:46  <cboltz> I guess we should drop freeipa from the dnsmasq config on anna/elsa (and whereever else it is used)
2021-03-28T19:02:11  <cboltz> actually I will do it on anna/elsa now ;-)
2021-03-28T19:02:31  <cboltz> (I somewhat doubt that it's related to the errors we see, but we have to drop it nevertheless ;-)
2021-03-28T19:02:31  <pjessen> freeipa was not changed, so I would not expect to find the problem there
2021-03-28T19:02:53  <pjessen> cboltz: agree
2021-03-28T19:02:58  <cboltz> right, it will "just" become outdated
2021-03-28T19:03:22  <pjessen> iow, the problem must be caused by powerfdns on chip
2021-03-28T19:03:41  <pjessen> maybe change anna/elsa not to use chip, just to check
2021-03-28T19:04:22  <pjessen> should not be an issue for a day or two.
2021-03-28T19:04:51  <cboltz> powerdns on chip exists since years (got fed by freeipa, and acted as hidden master for the public nameservers)
2021-03-28T19:05:06  <cboltz> therefore I'd be surprised if it would be the problem
2021-03-28T19:05:13  <cboltz> but then, freeipa also exists since years
2021-03-28T19:05:16  <cboltz> hmm...
2021-03-28T19:05:26  <pjessen> I have no access to chip?
2021-03-28T19:05:47  <cboltz> you should at least be able to login as user, not sure if you can sudo
2021-03-28T19:06:47  <pjessen> cboltz: could not login, asked for a password.
2021-03-28T19:06:55  <cboltz> hmm, strange
2021-03-28T19:07:31  <cboltz> confirmed :-/
2021-03-28T19:08:00  <cboltz> give me a minute to remove freeipa from dnsmasq on anna/elsa, then I'll have a look at chip
2021-03-28T19:08:33  <pjessen> cboltz: careful. might really be better to leaana/elsa as is
2021-03-28T19:08:38  <pjessen> to leave
2021-03-28T19:09:14  <cboltz> I'll be online in the next 3 or 4 hours - in worst case I'll revert the change
2021-03-28T19:09:39  <pjessen> ok
2021-03-28T19:11:19  <pjessen> chip.i.o.o does not resolve on anna nor on pontifex ??
2021-03-28T19:12:31  <cboltz> confirmed, I reverted the dnsmasq change
2021-03-28T19:13:12  <cboltz> pontifex uses anna/elsa, see resolv.conf
2021-03-28T19:13:52  <pjessen> cboltz: yup.
2021-03-28T19:14:23  <cboltz> for chip, at least the salt backdoor works, and /v/l/messages says
2021-03-28T19:14:26  <cboltz>     2021-03-28T19:13:40.154124+00:00 chip sshd[17016]: userauth_pubkey: key type ssh-dss not in PubkeyAcceptedKeyTypes [preauth]
2021-03-28T19:14:58  <cboltz> looks like a funny side effect of the update to 15.3 beta...
2021-03-28T19:15:14  <pjessen> aha ......
2021-03-28T19:19:08  <pjessen> I worry about the DNS issue causing permanent bounces - DNS issues should only cause 45x bounces.
2021-03-28T19:19:34  <cboltz> yes, that's strange[tm]
2021-03-28T19:19:58  <cboltz> maybe enable soft_bounce on anna/elsa temporarily?
2021-03-28T19:21:24  <pjessen> do they have the problem too?
2021-03-28T19:21:58  <cboltz> sorry, adjusting dnsmasq on anna/elsa confused me ;-) - but I'll check nevertheless
2021-03-28T19:22:06  <cboltz> soft_bounce on mx* is probably more important
2021-03-28T19:22:49  <pjessen> i'll adjust mx12
2021-03-28T19:23:14  <cboltz> the log on anna is more interesting[tm] - I see about 15 lines like
2021-03-28T19:23:19  <cboltz> 2021-03-28T17:55:32.502090+00:00 anna postfix/smtpd[25704]: warning: hostname mailman3.infra.opensuse.org does not resolve to address 192.168.47.80: No address associated with hostname
2021-03-28T19:24:32  <pjessen> weird
2021-03-28T19:24:33  <cboltz> so "only" the reverse lookup had problems, and that's just a warning
2021-03-28T19:26:04  <cboltz> regarding chip - I moved my ~/.ssh/id_dsa{,.pub} away and now can login again
2021-03-28T19:26:26  <cboltz> so sending the "wrong" key first seems to abort key auth, and falls back to password
2021-03-28T19:26:26  <pjessen> i only have rsa and ed25519
2021-03-28T19:26:42  <cboltz> try moving your rsa key away
2021-03-28T19:27:10  <cboltz> (I guess I'll report a bug for 15.3 - that looks like a regression in ssh)
2021-03-28T19:27:20  <pjessen> no change
2021-03-28T19:28:33  <cboltz> Invalid user per -> try pjessen ;-)
2021-03-28T19:28:43  <pjessen> aaaaaaah
2021-03-28T19:29:32  <pjessen> nah, same thing.
2021-03-28T19:30:41  <pjessen> oh well, not important for now.
2021-03-28T19:31:15  <cboltz> now it says   Invalid user pjessen
2021-03-28T19:31:46  <cboltz> so maybe the ldap connection is also broken with 15.3 :-/
2021-03-28T19:31:50  <pjessen> so we have gone from bad to worse :-)
2021-03-28T19:32:22  <pjessen> I have to ask - why are we testing 15.3 on something as important as DNS ?
2021-03-28T19:33:43  <cboltz> not my bug^Wupdate ;-) - I guess someone had some hackweek time left
2021-03-28T19:34:47  <pjessen> Someone should have had a hackweek beer instead ..... oh well, I'm going to go and watch telly.
2021-03-28T19:34:57  <pjessen> see ya
2021-03-28T19:35:16  <cboltz> enjoy the evening ;-)
2021-03-28T19:35:41  <cboltz> #
2021-03-28T19:35:42  <cboltz> salt chip.infra.opensuse.org cmd.run '/usr/local/bin/fetch_freeipa_ldap_sshpubkey.sh cboltz'
2021-03-28T19:35:44  <cboltz> chip.infra.opensuse.org:
2021-03-28T19:35:46  <cboltz>     ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)
2021-03-28T19:35:47  <cboltz> nice[tm]
2021-03-28T19:42:01  <cboltz> hmm, doesn't seem to be a general problem with 15.3 - some other machines are also on 15.3 and I can still login
2021-03-28T20:15:14  <cboltz> https://progress.opensuse.org/issues/90455 is the summary of the DNS issues