2020-07-02T02:13:24  -heroes-bot- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 63 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-07-02T02:53:24  -heroes-bot- RECOVERY: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=39 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-07-02T02:55:56  *** okurz_ is now known as okurz
2020-07-02T03:00:32  <malcolmlewis> pjessen, when you get a minute some re-indexing needs to be done after I merged and old account with a new one....
2020-07-02T04:03:24  -heroes-bot- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 50 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-07-02T04:13:24  -heroes-bot- RECOVERY: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=43 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-07-02T04:45:14  -heroes-bot- PROBLEM: PSQL locks on mirrordb2.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 305 * total waiting locks: 148 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb2.infra.opensuse.org&service=PSQL%20locks
2020-07-02T04:55:22  -heroes-bot- PROBLEM: HTTP download on pontifex2.infra.opensuse.org - CRITICAL - Socket timeout after 10 seconds ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=pontifex2.infra.opensuse.org&service=HTTP%20download
2020-07-02T05:05:12  -heroes-bot- RECOVERY: HTTP download on pontifex2.infra.opensuse.org - HTTP OK: HTTP/1.1 200 OK - 6638 bytes in 0.006 second response time ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=pontifex2.infra.opensuse.org&service=HTTP%20download
2020-07-02T05:05:15  -heroes-bot- RECOVERY: PSQL locks on mirrordb2.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=2 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb2.infra.opensuse.org&service=PSQL%20locks
2020-07-02T05:34:25  * klein rebooted gitlab to apply updates
2020-07-02T05:41:25  <klein> gitlab is back with version 13.1.2 (2e7a6f64b07)
2020-07-02T06:11:18  <pjessen> malcolmlewis: re-indexing ?
2020-07-02T06:23:37  <kl_eisbaer> FYI: the database of forum.opensuse.org is now running on the galera cluster
2020-07-02T06:23:55  <kl_eisbaer> I switched off mysql on forumsoo_temp by intention
2020-07-02T06:28:16  <pjessen> kl_eisbaer: cool.
2020-07-02T06:29:21  <kl_eisbaer> ...and soon the first two machines are running 15.2 :-)
2020-07-02T06:40:10  <kl_eisbaer> pjessen: can you test if everything looks ok with the forums ?  - for me it looks ok, but I might have missed something
2020-07-02T06:44:36  <pjessen> kl_eisbaer: I'll probably miss the same :-)
2020-07-02T06:46:45  <pjessen> kl_eisbaer: looks okay to me. but a bit slow.
2020-07-02T08:30:40  <kl_eisbaer1> FYI: https://monitor.opensuse.org/grafana/d/s_k9wGNiz/galera-cluster-overview?orgId=1  - forums migrated around 08:00
2020-07-02T09:41:16  <a-865k> "The database has encountered a problem." trying to access forums.
2020-07-02T10:01:48  <kl_eisbaer1> fixed
2020-07-02T10:02:20  <kl_eisbaer1> but I need to debug some error messages in the logs. Looks like there is some tuning option that is not used...?
2020-07-02T10:03:11  <kl_eisbaer1> looks like vbulleting does not properly close connections :-(
2020-07-02T10:34:00  <okurz> does progress.o.o work for you?
2020-07-02T10:36:09  <guillaume_g> wiki and openqa are unreachable
2020-07-02T10:36:35  <guillaume_g> bad gateway for the wiki
2020-07-02T10:36:42  <guillaume_g> nothing for openQA
2020-07-02T10:39:14  <lkocman> Reason: DNS lookup failure for: proxy-opensuse-ha.login.infra.opensuse.org
2020-07-02T10:41:38  <kl_eisbaer1> hm. let me have a look ...
2020-07-02T10:44:44  *** Martchus_ is now known as Martchus
2020-07-02T10:45:17  <lkocman> so both wiki and progress.opensuse.org are affected
2020-07-02T10:45:21  <kl_eisbaer1> looks like pdns-recursor is broken
2020-07-02T10:50:29  <okurz> kl_eisbaer1: did I get this right that you are upgrading openSUSE infrastructure to 15.2 on the actual release day of 15.2? ;)
2020-07-02T10:50:45  <lkocman> okurz I think that not heroes
2020-07-02T10:50:57  <lkocman> obs was updating kernel in obs ... that was clarified
2020-07-02T10:51:04  <lkocman> *obs team
2020-07-02T10:51:16  <lkocman> but rest of the things is bit worrying
2020-07-02T10:53:32  <lkocman> wiki is back online
2020-07-02T10:53:38  <lkocman> Thank you kl_eisbaer1 !
2020-07-02T10:54:00  <kl_eisbaer1> replaced pdns-recursor with dnsmasq on anna/elsa
2020-07-02T11:01:52  <darix> that vbulletin has DB problems is known?
2020-07-02T11:20:28  <kl_eisbaer1> darix: that was during the morning and should hopefully be fixed around 12:00
2020-07-02T11:20:53  <kl_eisbaer1> darix: if you still get Emails after 12:00, I need to investigate further
2020-07-02T11:22:45  <darix> well it is still fetching mails
2020-07-02T11:22:49  <darix> i can check after that
2020-07-02T11:23:54  <henne> the etherpad.o.o routing is also borked
2020-07-02T11:27:40  <kl_eisbaer1> henne: I'm working on that right now
2020-07-02T11:27:43  <kl_eisbaer1> gimme 10min
2020-07-02T11:30:32  <henne> you have 5! </kirk>
2020-07-02T11:40:11  <kl_eisbaer1> done :-/
2020-07-02T11:40:41  <kl_eisbaer1> I know, I know. Took too long to fix https://etherpad.opensuse.org/ - but I will try to do it better in the future!
2020-07-02T11:45:54  <kl_eisbaer1> looks to me like the forum is speeding up. Maybe because the cache needs to get filled first
2020-07-02T11:46:19  <kl_eisbaer1> I would leave it in the current state untili tomorrow and look if the speed is still too slow
2020-07-02T12:36:14  <bmwiedemann1> pjessen: do you know how to get mirrorbrain to create .torrent files in http://download.opensuse.org/distribution/leap/15.2/iso/openSUSE-Leap-15.2-NET-x86_64.iso.mirrorlist ?
2020-07-02T12:38:48  <darix> bmwiedemann1: we normally create those for the isos manually
2020-07-02T12:39:16  <darix> bmwiedemann1: the automatically created torrents depend on the mb makehashes cronjob
2020-07-02T12:40:12  <darix> and they need to be registered with the torrent server
2020-07-02T12:40:17  <darix> so it would even allow serving them
2020-07-02T12:40:37  <darix> i am not even sure the automated torrents work
2020-07-02T12:42:35  <bmwiedemann1> they worked for 15.1
2020-07-02T12:44:32  <darix> for 15.1 we manually created them?
2020-07-02T12:44:47  <darix> and then put in the right directory so the torrent master
2020-07-02T12:45:30  <bmwiedemann1> no, the 15.1 ones I seed say created by18:MirrorBrain/2.18.113
2020-07-02T12:46:42  <darix> aha
2020-07-02T12:46:48  <darix> well then you might have to wait for the make hashes
2020-07-02T12:47:00  <darix> though I would be curious if the herfurd actually accepts them
2020-07-02T12:47:10  <bmwiedemann1> curl -s http://tracker.opensuse.org:6969/file/openSUSE-Leap-15.1-DVD-x86_64.iso.torrent?info_hash=f8da57958058d8bf7384e4919e6d966ccc1a1931 | strings|head -1
2020-07-02T12:47:13  <darix> or if it is just based on the p2p hash table
2020-07-02T12:47:30  <bmwiedemann1> and I dont see a cron job on pontifex2
2020-07-02T12:47:42  <darix> pontifex2 (download.o.o):/var/lib/hefur # l
2020-07-02T12:47:48  <darix> go there. do that.
2020-07-02T12:48:04  <darix> so it is manually created torrets
2020-07-02T12:48:35  <darix> 15.2 is there too
2020-07-02T12:48:39  <darix> our tracker is not open
2020-07-02T12:49:48  <bmwiedemann1> yes, I know, but the 15.2 one is a manually created .torrent from Lubos
2020-07-02T13:03:24  <darix> probably because both have the same hash
2020-07-02T13:03:26  <darix> it works then
2020-07-02T13:31:34  <malcolmlewis> pjessen, the search index AFAIK by running php searchindex.php
2020-07-02T13:34:06  <bmwiedemann1> torrents for 15.2 isos work now. Left out live ones because there was no interest in 15.1 ones
2020-07-02T15:13:57  <lkocman> Hello team
2020-07-02T15:14:13  <lkocman> darix, I just sent a request to admin and heroes to fix the sha256 signature
2020-07-02T15:17:54  <lkocman> weird I did send email to admin@opensuse.org but it's still not in progress
2020-07-02T15:18:01  <lkocman> I mean progress-o-o
2020-07-02T15:18:21  <bmwiedemann1> maybe because new tickets are private by default?
2020-07-02T15:18:27  <lkocman> bmwiedemann1, ah didn't know that
2020-07-02T15:18:31  <lkocman> but still I'm the reporter ;-)
2020-07-02T15:18:47  <bmwiedemann1> there was also some backlog on relay2 mail processing from some DNS hickup
2020-07-02T15:18:56  <lkocman> bmwiedemann1, oky
2020-07-02T15:19:13  <lkocman> kl_eisbaer1, darix ^ I suppose you're the only ones who can fix it
2020-07-02T15:21:49  <darix> lkocman: no there is nothing we can fix besides waiting
2020-07-02T15:23:22  <lkocman> darix, not that, the signature
2020-07-02T15:23:34  <lkocman> darix, sorry for not specifying it
2020-07-02T15:23:34  <darix> lkocman: see rocketchat?
2020-07-02T15:23:38  <lkocman> darix, checking
2020-07-02T15:24:00  <lkocman> darix, you're rockstar! thank you
2020-07-02T15:40:47  <bmwiedemann1> 4400 left in the mail queue, going down at 2-3 per second.
2020-07-02T15:42:51  <bmwiedemann1> 4200
2020-07-02T15:44:33  <bmwiedemann1> 4000
2020-07-02T15:45:45  <bmwiedemann1> 3800
2020-07-02T15:50:49  <darix> lcp: matrix-synapse 1.15.2
2020-07-02T16:06:22  <fcrozat> Hi. Quick question: I'm trying to have my ISP to list openSUSE as pre-installed VM on their  router (I did the work at the last hackweek), starting with 15.2. ATM, they are official mirror for openSUSE (ftp.free.fr) but it seems mirrorbrain hasn't catched-up with 15.2 being available on their mirror.
2020-07-02T16:07:28  <fcrozat> Is there anything possible to do to improve the situation. I noticed they didn't advertise their ftp access (despite being present) and they even have rsync possible (but not setup for opensuse).
2020-07-02T16:09:34  <darix> you mean that download.o.o redirects to their mirror?
2020-07-02T16:10:01  <fcrozat> it doesn't yet for 15.2 (I think it didn't notice files are there)
2020-07-02T16:10:40  <fcrozat> https://download.opensuse.org/ports/aarch64/distribution/leap/15.2/appliances/openSUSE-Leap-15.2-ARM-JeOS-efi.aarch64.qcow2.mirrorlist
2020-07-02T16:11:06  <darix> well
2020-07-02T16:11:15  <fcrozat> (same issue is also for the regular 15.2 iso ;)
2020-07-02T16:11:15  <darix> do they have the files on their mirror?
2020-07-02T16:11:19  <darix> then it is just a scan
2020-07-02T16:11:20  <fcrozat> yes :)
2020-07-02T16:11:50  <fcrozat> http://opensuse.mirrors.proxad.net/opensuse/ports/aarch64/distribution/leap/15.2/appliances/
2020-07-02T16:11:50  <darix> scanning
2020-07-02T16:12:05  <darix> well free.fr is also listed in mirrorbrain
2020-07-02T16:12:08  <darix> scanning that for now
2020-07-02T16:12:26  <fcrozat> sorry, this is the same mirror : free.fr = proxad.net
2020-07-02T16:12:48  <darix> on mirror at the time
2020-07-02T16:12:55  <darix> http://opensuse.mirrors.proxad.net/opensuse/distribution/leap/15.2/iso/?F=1: error "403 Forbidden" fetching index, skipped.
2020-07-02T16:13:02  <darix> they didnt sync the permission change yet
2020-07-02T16:14:26  <fcrozat> ok, so this should be just a matter of time
2020-07-02T16:14:54  <darix> do you have contact to the admins?
2020-07-02T16:15:21  <fcrozat> no, I have contact with the guy who did the VM work on the set-top-box
2020-07-02T16:27:45  <fcrozat> I think I found their mail in my archive. I'll ping them tomorrow if it is not fixed
2020-07-02T16:34:21  <darix> fcrozat: scan done
2020-07-02T16:43:50  <lkocman> Thank you folks. I'm now leaving for a dinner "Happy Release day!"
2020-07-02T16:43:59  <okurz> lkocman: enjoy!
2020-07-02T16:44:29  <lkocman> I'll most likely spend evening by sending "ovations" to everyone who participated!
2020-07-02T17:45:49  *** lurchi_ is now known as lurchi__
2020-07-02T19:06:54  *** lurchi__ is now known as lurchi_
2020-07-02T19:46:18  <bmwiedemann1> mail queue on relay2 is now empty, so from there should no more be delays atm
2020-07-02T19:48:47  *** lurchi_ is now known as lurchi__
2020-07-02T20:17:23  <J_L_P> Hi, I have not been able to log into the openSUSE forums since the change
2020-07-02T20:19:55  <cboltz> can you login to other services, for example bugzilla or the wikis?
2020-07-02T20:46:15  *** lurchi__ is now known as lurchi_
2020-07-02T20:56:02  <J_L_P> cboltz - yes I can log into everything but the forums
2020-07-02T20:57:14  <cboltz> ok, so the problem is probably specific to the forums - and I'm afraid I don't know enough about the forums to help you instantly
2020-07-02T20:58:24  <J_L_P> cboltz - been locked out of the forums since the log in system change
2020-07-02T20:58:32  <cboltz> please send a mail to admin@opensuse.org with a description of your problem (and stating that only forums login is broken) so that our forums admins can check what's going on
2020-07-02T20:58:45  <cboltz> (also include your username ;-)
2020-07-02T21:25:32  <J_L_P> cboltz - ok, thanks!
2020-07-02T21:29:36  *** lurchi_ is now known as lurchi__
2020-07-02T22:05:34  *** lurchi__ is now known as lurchi_
2020-07-02T23:16:15  *** lurchi_ is now known as lurchi__