2022-04-19T01:34:14  <malcolmlewis> LCP0968[m], cboltz I'm getting random database errors when trying to read threads, new posts etc
2022-04-19T07:37:43  <pjessen> progress.o.o seems a bit slow this morning?
2022-04-19T08:24:12  <anikitin> pjessen, do you still see duplicates in https://mirrorcache.opensuse.org/update/leap/15.3/sle/repodata/970feef70cbd786e4312437d7b2b59286370e6cb58ea88f3d56639c8727a7cc5-primary.xml.gz.mirrorlist ?
2022-04-19T08:25:22  <anikitin> I did see duplicates in past, but iirc they gone relatively quickly. I have an idea how to fix it, but since it is mostly cosmetic problem - I give it low priority
2022-04-19T08:31:13  *** Papabears[m] is now known as Papabears9969[m]
2022-04-19T08:41:33  <pjessen> anikitin: let me check
2022-04-19T08:42:17  <pjessen> anikitin: yes, it looks like every entry is duplicated :-) 
2022-04-19T08:43:46  <anikitin> not for me :) , what if you append ?COUNTRY=de in url ? In any case - I have the bug, I will have a look thx
2022-04-19T08:44:34  <pjessen> I'll try with country - what is also "funny" is that I don't see any Swiss mirrors, none. My own mirror is being scanned though.
2022-04-19T08:45:29  <pjessen> with COUNTRY=de, I see no duplicates
2022-04-19T08:46:37  <pjessen> with COUNTRY=it or da, I have duplicates 
2022-04-19T08:52:41  <pjessen> COUNTRY=fr - no dupes, but otherwise be/nl/no/se/fi etc
2022-04-19T09:38:01  <henne> the wiki has problems connecting to the DB
2022-04-19T09:38:02  <henne> https://en.opensuse.org/
2022-04-19T09:38:36  <henne> I also got reports of "slowness" for events.o.o
2022-04-19T09:38:46  <henne> anything up with the cluster?
2022-04-19T09:40:21  <pjessen> henne: progress.,o.o has also been slow all morning
2022-04-19T09:42:52  * henne uploaded an image: (14KiB) < https://libera.ems.host/_matrix/media/r0/download/matrix.org/gwotAtFMxqqlVoBZRPFpvfwv/Screenshot%20from%202022-04-19%2011-41-18.png >
2022-04-19T09:42:57  <henne> this is events.o.o
2022-04-19T09:45:04  <pjessen> cboltz was having some issues yesterday, but he is not here yet
2022-04-19T09:45:32  <pjessen> he was working on the wiki, some update
2022-04-19T10:36:50  <henne> so is anyone looking into this?
2022-04-19T10:39:01  * henne twists some arms
2022-04-19T11:40:02  <malcolmlewis> likewise, forum (well for me) still having database issues
2022-04-19T11:46:09  <henne> cboltz: darix found out it's the wiki eating all connection slots
2022-04-19T11:46:29  <henne> cboltz: sounds like fallout from the upgrade?
2022-04-19T11:48:15  <darix-> any heroes available? :D
2022-04-19T11:48:25  <darix-> *stabs cboltz a bit*
2022-04-19T11:49:11  <darix-> in case you wonder why barely anything works that uses mysql: the wiki eats up all the connections 
2022-04-19T11:49:54  <darix-> so if someone could rcapache2 stop on the VM(s) that would be awesome
2022-04-19T11:50:16  <lkocman> Hello team, progress-o-o is down  https://progress.opensuse.org/projects/opensuse-leap-15-4/issues/gantt
2022-04-19T11:50:23  <lkocman> I suppose it's related to ^
2022-04-19T11:50:35  <darix-> it is
2022-04-19T11:53:17  <pjessen> progress has been slow all morning, but it's working
2022-04-19T11:54:44  <darix-> pjessen: then you were lucky and wiki gave you some connections :D
2022-04-19T11:55:24  <darix-> basically all galera nodes have 100% cpu load and the wiki seems to trigger it
2022-04-19T11:56:01  <darix-> mysqladmin processlist | grep -c wiki_ 
2022-04-19T11:56:01  <darix-> 583
2022-04-19T11:57:06  <darix-> holy shit
2022-04-19T11:57:13  <darix-> why is there sql traffic on more than one node
2022-04-19T11:58:12  <pjessen> 583 - I'm sure we don't have that many apache workers
2022-04-19T11:58:37  <darix-> so the wiki has connections stuck at least on node 1 and 2 ... that also explains why i saw cluster resume events from earlier in the day 
2022-04-19T11:58:51  <darix-> pjessen: can we turn off the wikis for a moment?
2022-04-19T11:59:10  <pjessen> yeah, just logged in
2022-04-19T11:59:34  <darix-> because i can tell you it clogs up the whole galera cluster right now
2022-04-19T11:59:51  <pjessen> ]have stopped apache on riesling
2022-04-19T12:00:24  <cboltz> hmm, interesting[tm]
2022-04-19T12:00:35  <cboltz> the most likely reason is the updated mediawiki on en.o.o
2022-04-19T12:00:50  <darix-> all those connections are stuck in "sending data" 
2022-04-19T12:01:23  <pjessen> I guess they might time out, eventually
2022-04-19T12:02:45  <cboltz> darix-: the wikis connect only to one node (via haproxy), therefore "more than one node" might mean it _writes_ to the database (which then gets synced to all nodes)
2022-04-19T12:03:36  <darix-> purged all wiki connections
2022-04-19T12:03:50  <darix-> all the queries were select
2022-04-19T12:04:21  <darix-> tons of '(SELECT /* SpecialRecentChangesLinked::doMainQuery...)'
2022-04-19T12:04:46  <darix-> only one "SELECT /* MediaWiki\Revision\RevisionStore::getAuthorsBetween" on galera2
2022-04-19T12:05:14  <darix-> galera 1 only had that SpecialRecentChangesLinked::doMainQuery
2022-04-19T12:05:27  <darix-> for galera3 i didnt grab the list before purging connections 
2022-04-19T12:06:31  <cboltz> can you paste the full SpecialRecentChangesLinked query?
2022-04-19T12:06:40  <pjessen> its a bit weird for SELECTs to be getting hung up - any I/O issues?
2022-04-19T12:06:49  <darix-> "(SELECT /* SpecialRecentChangesLinked::doMainQuery  */  rc_id,rc_timestamp,rc_namespace,rc_title,rc_"
2022-04-19T12:07:00  <darix-> that is all i see in mysqladmin processlist
2022-04-19T12:07:10  <cboltz> looks cut off :-(
2022-04-19T12:07:32  <darix-> pjessen: well one reason could be ... that table runs over a large table and e.g. indexes are missing our outdated
2022-04-19T12:07:40  <darix-> so each query does a full table scan 
2022-04-19T12:07:58  <darix-> s/that table/that query/
2022-04-19T12:07:59  <cboltz> right, would be an explanation
2022-04-19T12:08:08  <darix-> e.g. any migrations which might not have been run?
2022-04-19T12:08:20  <darix-> which would have created the index
2022-04-19T12:08:23  <cboltz> the output of the update script looked good
2022-04-19T12:08:37  <cboltz> let me start apache again, and then "show full processlist" so that we can find out what causes the load
2022-04-19T12:09:08  <darix-> mysqladmin kill $(mysqladmin processlist | awk '/wiki_/ {print $2}' | tr '\n' ',')
2022-04-19T12:09:11  <darix-> just in case :)
2022-04-19T12:09:52  <cboltz> hehe ;-)
2022-04-19T12:10:59  <cboltz> no hanging queries for wiki_en so far...
2022-04-19T12:11:26  <cboltz> ah, managed to catch it, and it's quite long
2022-04-19T12:12:04  <cboltz> https://paste.opensuse.org/80160298
2022-04-19T12:12:37  <darix-> with tons of sub queries :D
2022-04-19T12:12:40  <darix-> good luck!
2022-04-19T12:12:49  <pjessen> wow, that is long .... 
2022-04-19T12:19:24  <pjessen> essentially 3 xc the sdame query, with different where
2022-04-19T12:21:28  <pjessen> main tables involved, 'change_tags' and  'recent_changes'
2022-04-19T12:25:43  <cboltz> explain (separate for each of the 3 UNIONed queries): https://paste.opensuse.org/view/raw/22966656
2022-04-19T12:26:08  <cboltz> looks like the first part misses an index
2022-04-19T12:28:32  <pjessen> how big are those tables?
2022-04-19T12:29:17  <LCP0968[m]> doug notified me tsp is broken, I should probably have a look (it's unrelated to galera issues, it's running on pg)
2022-04-19T12:30:54  <cboltz> recentchanges has about 1000 rows, comment (renamed to comment_rc_comment in the query) about 57000
2022-04-19T12:32:50  <cboltz> interestingly, comment.comment_id is   PRIMARY KEY (`comment_id`),   so that key should get used in the JOIN - but explain says that despite PRIMARY would be a possible key, no key gets used
2022-04-19T12:34:12  <pjessen> its not a lot of rows
2022-04-19T12:35:06  <cboltz> right, but I still wonder why the primary key doesn't get used - especially because that results in "Using temporary; Using filesort"
2022-04-19T12:35:51  <pjessen> any issue with space on galera?
2022-04-19T12:36:55  <pjessen> 583 hanging queries mush have been waiting for something.
2022-04-19T12:37:29  <cboltz>  /var/lib/mysql-files on galera2 only has 1.6G free
2022-04-19T12:38:09  <pjessen> hmm, not a lot, but sounds okay 
2022-04-19T12:40:35  <henne> no one will ever need more than 640k disk space!
2022-04-19T12:41:14  <cboltz> no idea what caused the queue, and "of course" now that I look at it, there is _no_ hanging/waiting apache process and _no_ hanging query
2022-04-19T12:46:50  <henne> heisenbug
2022-04-19T13:00:28  <henne> at least my P95s are down to 2s
2022-04-19T13:03:04  <pjessen> progress.o.o is working too :-)
2022-04-19T13:05:36  <cboltz> I have to leave
2022-04-19T13:05:47  <cboltz> if the problem appears again, please save the output of   ps Zaux   before killing apache so that we can see if it's really the english wiki or another one
2022-04-19T13:06:52  <malcolmlewis> forums are better too, thanks all
2022-04-19T13:11:25  <darix-> you can tell him all connections were to wiki_en 
2022-04-19T13:11:36  <darix-> (in the mysqladmin processlist output)
2022-04-19T13:16:35  <lethliel> Hi.. Who can add some announcements to status.opensuse.org? 
2022-04-19T13:16:49  <darix-> lethliel: pjessen: might be able to
2022-04-19T13:29:36  <lethliel> Thanks
2022-04-19T13:42:45  <LCP0968[m]> Conan Kudo (ニール・ゴンパ) you tried fixing tsp, what did you actually find out?
2022-04-19T13:43:02  <Eighth_Doctor> I did?
2022-04-19T13:43:08  <Eighth_Doctor> no I tried to use it and it broke all over me
2022-04-19T13:43:11  <LCP0968[m]> that's what doug told me
2022-04-19T13:43:14  <LCP0968[m]> oh
2022-04-19T13:43:18  <LCP0968[m]> that's fun
2022-04-19T13:43:34  <LCP0968[m]> let's debug this then, i have no idea how to trigger the error
2022-04-19T13:47:12  <LCP0968[m]> I would love to actually debug this but my vpn connection seems to freeze for long periods of time
2022-04-19T14:01:46  <henne> that's your mandatory burnout protection 😉
2022-04-19T14:02:30  <LCP0968[m]> I feel like it will cause me burnout faster than fix anything
2022-04-19T14:11:53  <ConanKudo7057[m]> I can't access reimbursements, Doug can't set the right settings so that I can actually add my bank info to the reimbursement filing, etc.
2022-04-19T14:11:53  <ConanKudo7057[m]> it throws 500 errors a lot
2022-04-19T14:12:00  <LCP0968[m]> hhhh
2022-04-19T14:12:15  <LCP0968[m]> I really have no sympathy for leap staying on ruby 2.5
2022-04-19T14:12:26  <LCP0968[m]> updating dependencies is straight up impossible
2022-04-19T14:12:35  <darix-> LCP0968[m]: you can happily run every ruby version in parallel
2022-04-19T14:12:36  <darix-> just fine 
2022-04-19T14:12:50  <darix-> and we can easily build you all ruby extensions for it as well
2022-04-19T14:12:51  <LCP0968[m]> well, yes
2022-04-19T14:12:57  <LCP0968[m]> I do have to install them
2022-04-19T14:13:08  <darix-> guess how henne runs OBS on ruby 3.1 
2022-04-19T14:13:09  <LCP0968[m]> which means I need to get the repo
2022-04-19T14:13:45  <LCP0968[m]> I like that the first result on build.o.o for ruby is crystal project
2022-04-19T14:13:46  <darix-> i would accept complains about ruby if it werent so easy to run multiple versions in parallel
2022-04-19T14:14:04  <darix-> even packaging rails apps is super easy 
2022-04-19T14:14:09  <darix-> (if anyone wants to learn it)
2022-04-19T14:14:12  <LCP0968[m]> well, I'm complaining about leap
2022-04-19T14:14:13  <LCP0968[m]> I love ruby, it's great
2022-04-19T14:14:18  <LCP0968[m]> leap kinda blows though
2022-04-19T14:14:28  <darix-> well you can run your services on TW if you want
2022-04-19T14:14:33  <darix-> i do that for all my servers
2022-04-19T14:14:47  <darix-> including the stuff we host under https://pixls.us
2022-04-19T14:14:58  <LCP0968[m]> we don't really have tw in our infra
2022-04-19T14:15:09  <darix-> who is stopping you from doing it? 
2022-04-19T14:15:11  <LCP0968[m]> we should probably run all of our rails servers with tw
2022-04-19T14:15:18  <henne> LCP#0968: I'll add TSP to my list...
2022-04-19T14:15:47  <darix-> henne did the software-o-o upgrade in super fast time 
2022-04-19T14:15:50  <darix-> if i recall correctly
2022-04-19T14:16:05  <LCP0968[m]> there is a starting point for that, though idk how valuable that is anymore https://github.com/openSUSE/travel-support-program/pull/121
2022-04-19T14:16:16  <darix-> LCP0968[m]: and the problem for running different ruby versions which is the non default ruby is the same on leap as it is on TW
2022-04-19T14:16:18  <LCP0968[m]> since we are on rails 7
2022-04-19T14:16:39  <LCP0968[m]> well, for tumbleweed I can just use the default version
2022-04-19T14:16:43  <darix-> sure 
2022-04-19T14:17:20  <darix-> in any case ... https://build.opensuse.org/package/view_file/home:darix:apps/discourse/discourse.spec?expand=1
2022-04-19T14:17:30  <darix-> if you are curious about rails app packaging 
2022-04-19T14:17:41  <darix-> (the gitlab package is pretty much the same)
2022-04-19T14:18:01  <LCP0968[m]> yeah
2022-04-19T14:18:08  <darix-> runtime deps are autogenerated based on Gemfile.lock
2022-04-19T14:18:11  <LCP0968[m]> we don't package tsp in obs yet
2022-04-19T14:18:26  <darix-> with hard equal requires 
2022-04-19T14:18:28  <LCP0968[m]> mostly because it was a quick migration from connect server
2022-04-19T14:19:04  <darix-> in any case ... getting ruby stuff fixed seems easier than having leap ship newer python modules :)
2022-04-19T14:19:13  <darix-> your matrix server also wants TW 
2022-04-19T14:19:23  <LCP0968[m]> oh it really does
2022-04-19T14:19:34  <LCP0968[m]> I did try backporting, that didn't go well at all
2022-04-19T14:19:36  <darix-> no more py 3.6 support 
2022-04-19T14:19:47  <LCP0968[m]> yeah
2022-04-19T14:37:55  <darix-> henne: couldnt tsp just be merged into osem?
2022-04-19T14:39:04  <henne> OSEM already suffers from tons of unmaintained features...
2022-04-19T14:39:16  <henne> I had some TSP replacement
2022-04-19T14:40:38  <henne> I think I showed Gerald...
2022-04-19T14:43:59  <henne> https://cobudget.com/
2022-04-19T14:45:02  <henne> they seem to be pretty far
2022-04-19T14:45:34  <henne> I guess any other tool for participatory processes could be used too
2022-04-19T14:46:41  <henne> like https://meta.decidim.org/processes or https://www.loomio.com/about