2023-11-22T03:15:12  <bmwiedemann> etherpad is down, but VPN fails for me, so I cannot ssh in to fix.
2023-11-22T08:34:30  <luna> the etherpad is down
2023-11-22T09:02:56  <pjessen> the vpn seems to be down too
2023-11-22T09:05:14  <andrii> AUTH_FAILED from vpn as well
2023-11-22T09:13:06  *** Guest517 is now known as danidoni
2023-11-22T09:50:54  <pjessen> yep, AUTH_FAILED
2023-11-22T09:53:30  <acidsys> can manage ssh to qsc and prv but whilst I can ping some hosts in prg I don't have any with ssh open from other sites
2023-11-22T09:53:38  <acidsys> asked someone to check squanchy on site
2023-11-22T10:30:35  <luna> also problems to login to Weblate fyi
2023-11-22T10:41:47  <henne> anything with database...
2023-11-22T10:58:45  <kingt_of_the_lig> Hi acidsys, I have received confirmation from Doug. Thanks for your help.
2023-11-22T11:17:00  <acidsys> checking with bmwiedemann now
2023-11-22T11:33:01  <acidsys> the freeipa issue hit back
2023-11-22T11:43:27  <pjessen> works for me now, tnx
2023-11-22T11:53:57  <acidsys> vpn works again
2023-11-22T12:01:28  <darix> good thing freeipa will be killed soon
2023-11-22T12:04:14  <acidsys> seems maintenance update of libvirt caused restart of virtlogd which then sent signal to stop all vm's
2023-11-22T12:04:34  <acidsys> pacemaker then reacted and started the vms back up but due to broken freeipa network init there the network did not reconfigure automatically
2023-11-22T12:05:19  <acidsys> now galera is probably broken, still checking w/ bernhard
2023-11-22T12:18:40  <cboltz> s/probably//
2023-11-22T12:21:11  <acidsys> galera3 is safe to bootstrap and comes up. trying to then add galera2
2023-11-22T12:21:13  <acidsys> 2023-11-22 12:19:59 0 [Warning] WSREP: 0.0 (galera3.infra.opensuse.org): State transfer to 1.0 (galera2.infra.opensuse.org) failed: -22 (Invalid argument)
2023-11-22T12:25:29  <acidsys> maybe this is relevant 2023-11-22 12:19:59 341 [Warning] Access denied for user 'sst_user'@'localhost' (using password: YES) and there is no "sst_user" in "SELECT user FROM mysql.user;"
2023-11-22T12:28:27  <darix> acidsys: why in hell would a restart of virtlogd trigger pacemaker to kill all VMs o.O
2023-11-22T12:29:43  <acidsys> I did not write that.
2023-11-22T12:30:03  <darix> acidsys: we restart virtlogd regularly
2023-11-22T12:30:08  <darix> and it does not kill VMs here
2023-11-22T12:30:14  <acidsys> good
2023-11-22T12:30:39  <acidsys> ioo it is a bug with this update
2023-11-22T12:30:45  <acidsys> libvirt update and service restarts should be safe
2023-11-22T13:23:04  <acidsys> https://etherpad.opensuse.org/p/20231122-postmortem
2023-11-22T13:49:30  <okurz> acidsys: thanks for sharing that. It seems "tags" on progress.opensuse.org are now missing. Ticket queries like https://progress.opensuse.org/issues?query_id=757 broke therefore
2023-11-22T13:55:38  <acidsys> hi okurz, we are still waiting for Galera state transfer to the other two nodes to finish, so services using MySQL are currently pointing to a single node ..
2023-11-22T13:56:00  <acidsys> I'm not sure why that would cause Redmine to report "ERROR -- : [406c44a1-151a-4d4e-a311-0cf61c5d317b] Query::StatementInvalid: Mysql2::Error: Unknown column 'issues.issue_t ags' in 'where clause'" but I'll check one MySQL is fully healthy
2023-11-22T13:56:14  <okurz> ok. Do you have an ETA so that I know when to check again?
2023-11-22T13:57:48  <acidsys> transfer should finish in ~5 minutes but not sure yet if anything explodes afterwards
2023-11-22T14:00:33  <acidsys> 56/64GB transfer just started again from 0 :(
2023-11-22T14:06:19  <darix> acidsys: remember how we did the galera resync during the migration?
2023-11-22T14:06:25  <darix> run it outside the service file
2023-11-22T14:24:39  <acidsys> darix: it does not help, the process is not running into the typical timeout, but rather gets killed using a SIGABRT
2023-11-22T14:24:50  <acidsys> if I run it manually, it goes up to ~56GB, then the shell prints "Aborted"
2023-11-22T14:26:57  <acidsys> if I run it in the service, it similarly goes up to ~56GB, then systemd prints "killed, status=6/ABRT"
2023-11-22T15:11:42  <acidsys> okurz: are you sure this error only occured today ?
2023-11-22T15:35:20  <darix> acidsys: the sigabrt should have an explanation in the mysqld.log
2023-11-22T15:35:44  <acidsys> you don't say
2023-11-22T15:53:06  <okurz> acidsys: absolutely sure because we use those queries every day
2023-11-22T15:53:18  <okurz> but works now \o/
2023-11-22T16:06:38  <acidsys> thanks for confirming
2023-11-22T16:34:36  <acidsys> hi cboltz, andri asked us to point mirrors.o.o to mirrors-static.o.o. is that something you could do?
2023-11-22T16:36:23  <cboltz> it will need a bit of haproxy magic because it lives on narwal* as mirrors-static.o.o
2023-11-22T16:37:11  <cboltz> so - in general yes, I'm just not sure if I have time for it today (have to leave for an offline event later)
2023-11-22T16:45:09  <acidsys> ok, no rush :)
2023-11-22T17:28:56  *** teepee_ is now known as teepee