2020-03-22T03:10:52 *** okurz_ is now known as okurz 2020-03-22T04:40:04 -heroes-bot- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total waiting locks: 9 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks 2020-03-22T04:50:03 -heroes-bot- RECOVERY: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=26 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks 2020-03-22T08:34:03 -heroes-bot- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 68 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks 2020-03-22T09:50:35 -heroes-bot- PROBLEM: MySQL WSREP recv on galera1.infra.opensuse.org - CRIT wsrep_local_recv_queue_avg = 28.401735 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=galera1.infra.opensuse.org&service=MySQL%20WSREP%20recv 2020-03-22T09:54:04 -heroes-bot- RECOVERY: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=44 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks 2020-03-22T11:50:50 lcp: there are two 404s left on lizards: 2020-03-22T11:50:52 https://lizards.opensuse.org/wp-content/plugins/anti-captcha/anti-captcha-0.3.js.php?ver=898dd88cca7b2f65461bc491dacb9b25 2020-03-22T11:50:57 https://lizards.opensuse.org/wp-includes/js/wp-emoji-release.min.js?ver=4.7.5 2020-03-22T11:51:21 (both included in https://lizards.opensuse.org/page/3/ and probably other pages) 2020-03-22T11:51:30 can you please have a look at them? 2020-03-22T11:54:26 sure 2020-03-22T12:05:00 cboltz: it is fixed, and live 2020-03-22T12:05:58 confirmed, no more 404s :-) 2020-03-22T12:06:00 thanks! 2020-03-22T12:25:30 cboltz: out of curiosity, do you know what is the status on the progress update? 2020-03-22T12:30:42 I haven't heard anything for a while 2020-03-22T12:30:56 last time I tested, the web frontend worked without problems 2020-03-22T12:31:16 but creating tickets by mail (basically admin@) was broken 2020-03-22T12:31:54 tuanpembual: are there any news on this, and/or do you need help? 2020-03-22T12:32:49 hi cboltz 2020-03-22T12:32:59 sorry for not sent update 2020-03-22T12:33:14 i not yet touch last config 2020-03-22T12:34:06 my bad, I will do somethings this weeks. 2020-03-22T12:35:20 no need to apologize - we are all volunteers, and often don't have time to do everything quickly ;-) (I won't exclude me from that - I've also been too busy in the last months) 2020-03-22T12:36:01 if you need a helping hand, feel free to ask ;-) 2020-03-22T12:48:14 I will take care of the theming for now then 2020-03-22T12:51:18 ack, what version do we have deployed (and is it gonna change) 2020-03-22T13:13:22 https://progress-test.opensuse.org/admin/info says Redmine 3.4.12.stable 2020-03-22T13:17:48 alright 2020-03-22T13:38:21 https://progress.opensuse.org/issues/64722 says 2020-03-22T13:38:23 http://opensuse.org + https://opensuse.org results in a "403 Forbidden? 2020-03-22T13:38:35 but it works for me (redirects to www.o.o) 2020-03-22T13:39:11 am I the only lucky one, or is the reporter of this ticket the only unlucky one? 2020-03-22T13:39:16 please test and report back ;-) 2020-03-22T13:45:43 yeah, I redirected them to post a ticket to admin@o.o, it works for me too 2020-03-22T13:46:12 but I did have an issue with forums today which first loaded to 403 and only worked on the second reload 2020-03-22T13:46:22 so it might be something(?) 2020-03-22T14:29:18 Per also confirmed the 403 in the ticket, so something is indeed wrong 2020-03-22T14:30:15 no idea what, and since the o.o redirect is still hosted in Provo, getting a fast answer and/or fix is not too likely :-( 2020-03-22T14:31:14 the thing really holding back moving that redirect is the account system tho >:D 2020-03-22T14:31:31 (at least the openid part of it) 2020-03-22T14:32:44 just wondering - is it? https://opensuse.org/openid/ gets redirected to https://www.opensuse.org/openid/ so maybe moving the o.o redirect to Nuremberg wouldn't be a problem? 2020-03-22T14:32:59 (moving www.o.o is of course more difficult because of openid) 2020-03-22T14:48:06 hm, I don't actually know if they also don't have rewrite rules due to openid though 2020-03-22T14:49:36 look at how that looks in case of fedora's ipsilon server for example https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/ipsilon/templates/ipsilon-httpd.conf.j2#n4 2020-03-22T14:50:13 of course, in their case it is way easier, because they aren't dumb enough to host the openid server on the domain and/or www of the domain 2020-03-22T15:29:55 right, things would be somewhat boring if we'd have openid on a separate subdomain 2020-03-22T15:31:41 I kinda wish we could redirect to provo proxy www.o.o to another subdomain and move over the openid stuff to that subdomain 2020-03-22T15:31:49 no idea if there are openid rewrite rules on o.o - given that the whole domain is a redirect, I'd _guess_ no 2020-03-22T15:32:21 at the same time, I don't think we know what uses our openid stuff 2020-03-22T15:32:46 right, that's the next problem 2020-03-22T15:32:56 I know that paste.o.o allows openid logins 2020-03-22T15:33:16 and I wouldn't be surprised if some people use it to login on random external websites 2020-03-22T15:34:12 eh, the mistakes of the past 2020-03-22T15:34:39 indeed, and now we "enjoy" fixing it ;-) 2020-03-22T15:36:02 interestingly, now I also get the forbidden for curl -6 http://opensuse.org (worked before) 2020-03-22T15:36:50 is someone here who can open a ticket in Provo? 2020-03-22T15:40:36 yup, it's broken 2020-03-22T15:41:27 just wondering: what's worse - broken http://o.o or broken openid? 2020-03-22T15:41:41 handling the redirect in Nuremberg would be a quick fix for http://o.o 2020-03-22T15:42:04 but if there is really a special config for openid which we don't know, it will break 2020-03-22T15:42:10 which poison would you pick? 2020-03-22T15:45:31 I would take a broken openid with a "moved" notice on o.o/openid 2020-03-22T15:46:15 I'd guess/hope that a redirect to www.o.o also works for /openid 2020-03-22T15:46:42 however, I don't even know how openid is configured, so I will try to figure that one out 2020-03-22T15:58:08 -heroes-bot- PROBLEM: MySQL WSREP recv on galera2.infra.opensuse.org - CRIT wsrep_local_recv_queue_avg = 4.136601 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=galera2.infra.opensuse.org&service=MySQL%20WSREP%20recv 2020-03-22T15:58:38 cboltz: ah, alright, I see, if we could redirect just whatever is after /openid/ in www.o.o, that's enough to keep openid working 2020-03-22T15:59:07 yeah, I also hope so 2020-03-22T15:59:18 I just tested it :P 2020-03-22T15:59:54 and given that much more people use http(s)://o.o than openid, even if it breaks openid, moving the o.o redirect to Nuremberg is the smaller problem 2020-03-22T16:00:25 give me some minutes to do that ;-) 2020-03-22T16:01:28 * lcp uploaded an image: Screenshot from 2020-03-22 17-01-01.png (15KB) < https://matrix.org/_matrix/media/r0/download/matrix.org/EfyTgbxjrBbrqzJIpTHROomX > 2020-03-22T16:02:08 basically all of those files are hosted under /openid/, which this screenshot doesn't actually show because I didn't catch the important parts >:T 2020-03-22T16:03:12 ;-) 2020-03-22T16:03:34 I noticed a funny thing - the haproxy in Nuremberg already has the redirect for o.o -> www.o.o configured 2020-03-22T16:03:43 so I only need to change the DNS entry 2020-03-22T16:05:20 Conan Kudo: actually, something very ironic which you might find a little funny for a novell/suse bit of history https://www.opensuse.org/openid/secureprovider 2020-03-22T16:05:43 oh? 2020-03-22T16:06:07 jboss, hosted on suse apache server 2020-03-22T16:06:15 by novell 2020-03-22T16:10:31 DNS updated (to the IPs of redirector.o.o because CNAME doesn't allow other entries, and we need MX entries) 2020-03-22T16:19:22 cboltz: while you are doing dns stuff, maybe https://progress.opensuse.org/issues/63463 ;) 2020-03-22T16:21:53 the only issue with the matrix server now is that it doesn't really work anymore, because highstate run and broke stuff that can't be merged because of the gitlab pipeline >:T 2020-03-22T16:22:40 so basically you want a SRV record? 2020-03-22T16:23:11 yes 2020-03-22T16:23:40 (frankly the haproxy thing is important too, but I will take anything I can get) 2020-03-22T16:24:40 cboltz: https://progress.opensuse.org/issues/64722 2020-03-22T16:24:50 You were faster than me :-) 2020-03-22T16:25:01 ;-) 2020-03-22T16:25:39 and even more courageous - I would have started just with the IPv6 address :-) 2020-03-22T16:26:06 well, when I first tested, both v4 and v6 worked for me 2020-03-22T16:26:15 in the second test v6 was broken 2020-03-22T16:26:31 so I just took that risk before seeing broken v4 ;-) 2020-03-22T16:26:46 as long as it works: congratulations :-) 2020-03-22T16:26:59 IMHO the biggest risk is /openid/ - do you know if this change could affect /openid/ ? (lcp thinks it shouldn't, and I also hope so) 2020-03-22T16:27:08 this means we can even think about pointing www.opensuse.org to the narwal machines ;-) 2020-03-22T16:27:27 not really, there is www.o.o/openid/ which isn't static 2020-03-22T16:27:36 as long as you keep the openid part in provo, it should work 2020-03-22T16:27:41 You can still have a rule in haproxy that moves /openid/ requests to Provo 2020-03-22T16:28:41 something like: acl is_openid path_beg     /openid/ 2020-03-22T16:28:49 and than use a combination of 2020-03-22T16:29:19 actually, if the requests now go through the local dns, maybe monitor what stuff contains openid that people are trying to access >:D 2020-03-22T16:29:21 use_backend openid if is_openid is_plain_opensuse 2020-03-22T16:29:50 lcp: so far I tried to avoid logging DNS queries 2020-03-22T16:30:08 lcp: but we *can* enable them, if needed. 2020-03-22T16:30:24 for openid, I'd guess we'll need to log the haproxy requests, not dns 2020-03-22T16:30:45 cboltz: ah, right - as it is a subpath 2020-03-22T16:31:19 as a sidenote: bugzilla.o.o is CNAME www.o.o - we'll need to change this before changing www.o.o 2020-03-22T16:31:56 same for commons.o.o, but I'm not sure if/for what that is used (https://commons.opensuse.org/ goes 404) 2020-03-22T16:32:08 ha: and if I remember correctly, this was the main problem the last time: bugzilla access got broken 2020-03-22T16:32:42 > Apache/2.2.34 (Linux/SUSE) 2020-03-22T16:32:48 well, that's very old 2020-03-22T16:33:00 SLE12? 2020-03-22T16:33:16 and has 3 CVEs ;) 2020-03-22T16:33:49 IF it is SLE12, those CVEs should be patched (I hope) 2020-03-22T16:33:54 yikes the CVEs 2020-03-22T16:34:10 I hope so too, because those do not look good 2020-03-22T16:34:44 A classical problem with all these scanners: they don't know if an application is affected, as they don't know anything about the patch level... :-/ 2020-03-22T16:36:27 BTW: should we log haproxy messages directly to graylog ? 2020-03-22T16:37:08 I'll probably need a quick introduction to graylog before I can answer that ;-) 2020-03-22T16:37:46 cboltz: https://graylog.opensuse.org/search?q=source%3Aanna&rangetype=relative&relative=300 2020-03-22T16:38:59 https://docs.graylog.org/en/3.1/pages/queries.html 2020-03-22T16:39:31 sorry: https://docs.graylog.org/en/3.2/pages/queries.html 2020-03-22T16:39:50 so basically it's a frontend to view and search the syslog 2020-03-22T16:39:58 cboltz: exactly 2020-03-22T16:40:08 I always use grep for that ;-) 2020-03-22T16:40:48 cboltz: well: the problem is that you need to be root on the syslog server for that 2020-03-22T16:41:18 with graylog or elk, you can use a WebUI and complex searches 2020-03-22T16:41:29 chmod o+r (and maybe some logrotate config changes) could fix that ;-) 2020-03-22T16:41:30 and even generate some graphs out of your searches 2020-03-22T16:41:46 cboltz: ok, in that case I will shut down graylog again 2020-03-22T16:42:07 well, it's just my personal opinion/way - if you or others prefer graylog, I won't stop you ;-) 2020-03-22T16:42:25 Well: I have to maintain 3 different services for this. 2020-03-22T16:42:36 If nobody wants to make use of it, it's just wasted time 2020-03-22T16:43:26 I know that it needs some time to get familar with it - but for me personally, it helped me already to find things I never ever would have been seen with simple "grep"s in my logs 2020-03-22T16:44:12 commons just doesn't seem to exist anywhere on the internet, was it ever actually used or did it always return 404 2020-03-22T16:44:33 It's just way easier to have something graphical, where you can check at any time without VPN or local console. and being able to save your search and re-use it is something even the bash_history is somethings lacking ;-) 2020-03-22T16:45:02 kl_eisbaer: well, you mentioned graylog in your short (haha ;-) status update on the mailinglist some days ago - I'd say keep it running for some weeks, and then check if/how many people use it before deciding about its future ;-) 2020-03-22T16:45:19 (and the fact that it already was useful for you sounds like a good argument to have it) 2020-03-22T16:45:59 cboltz: sorry for the longer status report. I will try to get it shorter. 2020-03-22T16:46:11 no need to apologize, I enjoyed reading it 2020-03-22T16:46:13 At least good to know that at least you read it ... 2020-03-22T16:46:20 and I enjoyed the "short" in the subject even more ;-) 2020-03-22T16:47:21 nevertheless, as an idea - maybe send separate mails for each item, whenever you finish something? 2020-03-22T16:47:44 cboltz: I did not want to spam the ML ... 2020-03-22T16:48:21 it will take *lots of* status reports to flood heroes@ ;-) 2020-03-22T16:48:51 some more mails won't hurt 2020-03-22T16:49:08 okey, dokey 2020-03-22T16:50:51 lcp: regarding commons.opensuse.org, I also wonder if it was ever used 2020-03-22T16:51:00 it isn't opensuse-project@ over the last few weeks, ngl 2020-03-22T16:51:01 searching for site:commons.opensuse.org returns nothing 2020-03-22T16:51:10 yup, did the same 2020-03-22T16:51:28 also searched for literal "commons.opensuse.org", also nothing 2020-03-22T16:52:37 sounds like we can simply delete the DNS entry ;-) 2020-03-22T16:52:41 kl_eisbaer: any objections? 2020-03-22T16:53:34 cboltz: feel free. Worst case you need to restore it, if we get reports ... ;-) 2020-03-22T16:53:46 ok, I'll delete it 2020-03-22T16:53:53 for the archive: commons.o.o in CNAME www.o.o 2020-03-22T16:54:16 ah yeah, that reminds me, malcolmlewis noticed we have some machines with open ssh ports to the outside world, might be worth checking if that's fine for all of them 2020-03-22T16:54:26 cboltz: JFYI: there is a AFXR dump stored on the FreeIPA server every night as backup. 2020-03-22T16:54:37 ah, nice 2020-03-22T16:55:49 just as an idea - maybe push that backup to a repo on gitlab? Would be easier than manually diffing backup files 2020-03-22T16:56:10 cboltz: good idea. 2020-03-22T16:56:54 speaking about useless subdomains - http://wwwstage.opensuse.org/ times out 2020-03-22T16:57:05 cboltz: feel free 2020-03-22T16:57:31 cboltz: JFYI : the $lang-new.opensuse.org CNAMEs that point to amazon domains: please leave them for now. 2020-03-22T16:57:49 cboltz: they are requested by SUSE-IT for "migration tests". 2020-03-22T16:58:19 cboltz: I don't now much more atm - but there is a meeting tomorrow 2020-03-22T16:58:46 no problem, and I'll only delete entries I somewhat know and understand anyway ;-) 2020-03-22T16:58:54 for the archive: wwwstage.opensuse.org has address 130.57.8.104 - and is gone now 2020-03-22T16:58:56 * kl_eisbaer things they want to use them to test the new authentication solution - but this is just rumor until tomorrow 2020-03-22T17:01:08 wwwstage had an entry in salt repo, so that might require deleting 2020-03-22T17:02:09 hmm, where exactly? grepping for stage doesn't show it 2020-03-22T17:03:05 maybe you mean webtest.o.o (on community.o.o)? (should probably be moved to narwal*/static, but - openid is a different beast) 2020-03-22T17:03:47 ah, I might have be mistaken, I swear I saw it on the list of machines recently 2020-03-22T17:04:03 * ah, I have to be mistaken, I swear I saw it on the list of machines recently 2020-03-22T17:07:28 yeah, looking through history I had to have seen it somewhere else 2020-03-22T17:14:13 cboltz: https://graylog.opensuse.org/dashboards/5e6ebeb2eaa503d8f7d5ec17 2020-03-22T17:14:42 ^^ this is just an example why you might want to use dashboards 2020-03-22T17:15:21 message peaks or hosts that suddenly sent out a lot of Emails or see a lot of SSH log ins are not so easy to detect with a simple "grep" 2020-03-22T17:16:06 but a chart shows such anomalies and you might notice possible problems earlier 2020-03-22T17:16:55 indeed, nice examples 2020-03-22T17:17:10 I've just started and the queries used to produce the charts might need some fine tuning. 2020-03-22T17:17:17 But I hope you get the idea behind this. 2020-03-22T17:17:42 The good thing: graylog even allows to generate alerts, if we like... 2020-03-22T17:18:15 yes, I get the idea :-) 2020-03-22T17:18:51 ...and if you want to shoot the bull on a conference, you can hit the "play" button and show live how much stuff happens in openSUSE ;-) 2020-03-22T17:19:03 I even start to see why it can be better than grep - but it will probably need a while before I switch over ;-) 2020-03-22T17:19:20 (simply because I'm used to use grep) 2020-03-22T17:19:57 cboltz: thanks. ..and don't worry: I also just started with it. But I think this is something we (as heroes) and show our community later 2020-03-22T17:20:36 -heroes-bot- RECOVERY: MySQL WSREP recv on galera1.infra.opensuse.org - OK wsrep_local_recv_queue_avg = 0.438298 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=galera1.infra.opensuse.org&service=MySQL%20WSREP%20recv 2020-03-22T17:20:40 https://monitor.opensuse.org/grafana/d/YyV2BduWk/base-metrics?orgId=1&refresh=30s&from=1584876032152&to=1584897632153&var-hostname=anna.infra.opensuse.org&var-command=hostalive&var-service=&var-disk=&var-windisk=&var-metric=pl 2020-03-22T17:20:52 is another example, I'm currently working on 2020-03-22T17:21:19 Rudi (aka bugfinder) was so nice and already added more hosts to this 2020-03-22T17:22:05 right, that's another thing from your short ;-) status report I should probably get more familiar with 2020-03-22T17:22:06 it still needs some love - but this is already integrated in the new icinga2 monitoring and we can beautify the graphs over the time :-) 2020-03-22T17:22:44 * kl_eisbaer isout for a short break 2020-03-22T17:49:30 -heroes-bot- PROBLEM: MySQL WSREP recv on galera3.infra.opensuse.org - CRIT wsrep_local_recv_queue_avg = 1.894823 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=galera3.infra.opensuse.org&service=MySQL%20WSREP%20recv 2020-03-22T18:28:08 -heroes-bot- PROBLEM: MySQL WSREP recv on galera2.infra.opensuse.org - CRIT wsrep_local_recv_queue_avg = 1637.749778 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=galera2.infra.opensuse.org&service=MySQL%20WSREP%20recv 2020-03-22T19:41:34 kl_eisbaer: I did use the old matomo tracking id from the old search for the new search, so it's already regaining some popularity 2020-03-22T19:42:21 I also switched over the default homepage in Firefox to search-o-o 2020-03-22T19:43:16 (as a sidenote, I need to setup translations again, because we have very few languages there right now, so I will have to contact the weblate guys) 2020-03-22T19:43:57 lcp: good to know, thanks! 2020-03-22T19:53:13 just noticed Per's notice that IPv6 opensuse.org still gives a 403 from nginx 2020-03-22T19:53:44 one possible reason is that the haproxy config had two rules for "opensuse.org" - one pointing to is_static (narwal*), one pointing to is_mainpage (redirect) 2020-03-22T19:54:05 but even after fixing this, I see the nginx 403 for IPv6 opensuse.org :-( 2020-03-22T19:54:42 kl_eisbaer: maybe I'm blind ;-) - can you please have a look? 2020-03-22T19:55:35 < HTTP/2 301 2020-03-22T19:55:35 < content-length: 0 2020-03-22T19:55:35 < location: https://www.opensuse.org/ 2020-03-22T19:56:56 for me, everything looks good. Maybe some caching? 2020-03-22T19:57:53 I see no nginx answering my request for https://www.opensuse.org/ but an apache 2020-03-22T19:58:31 I'm only talking about the https://opensuse.org redirect, not about www.o.o 2020-03-22T19:59:15 curl -v --ipv6 https://opensuse.org 2020-03-22T19:59:24 HTTP/2 301 2020-03-22T19:59:24 < content-length: 0 2020-03-22T19:59:24 < location: https://www.opensuse.org/ 2020-03-22T19:59:40 this is what I get... 2020-03-22T19:59:47 caching could indeed be a problem - host gives me the correct IP, but curl -6 still uses the old one :-/ (as I noticed with -v in the "trying $IP" line) 2020-03-22T20:00:08 *   Trying 2620:113:80c0:8::16... 2020-03-22T20:00:13 this is what I get here.. 2020-03-22T20:00:23 yes, that's the right one 2020-03-22T20:00:58 but for some strange reason I still got (after "host opensuse.org" gave me 2620:113:80c0:8::16) Trying 2620:113:80c0:8::19... 2020-03-22T20:01:25 sudo nscd --invalidate hosts maybe ? 2020-03-22T20:01:41 I used rcnscd restart - and it helped :-) 2020-03-22T20:01:54 tss: lazy admins everywhere 2020-03-22T20:01:56 ;-) 2020-03-22T20:03:04 -heroes-bot- PROBLEM: HAProxy on elsa.infra.opensuse.org - HAPROXY CRITICAL - Active service galera2 is DOWN on galera proxy ! ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=elsa.infra.opensuse.org&service=HAProxy 2020-03-22T20:03:07 -heroes-bot- PROBLEM: HAProxy on anna.infra.opensuse.org - HAPROXY CRITICAL - Active service galera2 is DOWN on galera proxy ! ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=anna.infra.opensuse.org&service=HAProxy 2020-03-22T20:03:29 WTF 2020-03-22T20:05:29 ARG! THANK YOU, packagers, for starting to remove all the service files for SuSEfirewall2 :-( 2020-03-22T20:05:57 yeah, great idea :-/ 2020-03-22T20:09:10 ok, fixed 2020-03-22T20:13:42 -heroes-bot- RECOVERY: HAProxy on anna.infra.opensuse.org - HAPROXY OK - lists (Active: 1/1) kernel-git-in (Active: 0/1) etherpad (Active: 1/1) duke (Active: 1/1) redmine_test (Active: 1/1) elections (Active: 1/1) redmine (Active: 1/1) download (Active: 1/1) community2 (Active: 1/1) kubic (Active: 1/1) mirrorlist (Active: 1/1) galera (Active: 1/1 , Backup: 2/2) forums (Active: 1/1) smt (Active: 1/1) jenkins (Active: 1/1) freeipa (Active: 1/1) dale (Active: 1/1) static (Active: 3/3) svn (Active: 1/1) connect (Active: 1/1) monitor (Active: 1/1) rpmlint (Active: 1/1) html5test (Active: 3/3) jekyll (Active: 1/1) community (Active: 1/1) riesling (Active: 1/1) studioexpress (Active: 3/3) nuka (Active: 1/1) gccstats (Active: 1/1) github_101 (Active: 0/1) mickey (Active: 1/1) metrics (Active: 1/1) download-private (Active: 1/1) osccollab (Active: 1/1) status (Active: 1/1) galera-slave (Active: 1/1 , Backup: 2/2) pinot (Active: 1/1) lnt (Active: 1/1) monitor_grafana (Active: 1/1) shop (Active: 3/3) obsreview (Active: 1/1) rsync-community2 (Active: 0/1) elections-test (Active ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=anna.infra.opensuse.org&service=HAProxy 2020-03-22T20:13:44 -heroes-bot- RECOVERY: HAProxy on elsa.infra.opensuse.org - HAPROXY OK - rpmlint (Active: 1/1) monitor_graylog (Active: 1/1) duke (Active: 1/1) community (Active: 1/1) lnt (Active: 1/1) rsync-community2 (Active: 0/1) dale (Active: 1/1) gccstats (Active: 1/1) kernel-git-in (Active: 0/1) mickey (Active: 1/1) smt (Active: 1/1) static (Active: 3/3) galera (Active: 1/1 , Backup: 2/2) github_101 (Active: 0/1) download-private (Active: 1/1) community2 (Active: 1/1) etherpad (Active: 1/1) obsreview (Active: 1/1) jekyll (Active: 1/1) mirrorlist (Active: 1/1) elections (Active: 1/1) galera-slave (Active: 1/1 , Backup: 2/2) monitor_grafana (Active: 1/1) jenkins (Active: 1/1) osccollab (Active: 1/1) monitor (Active: 1/1) kubic (Active: 1/1) connect (Active: 1/1) freeipa (Active: 1/1) lists (Active: 1/1) status (Active: 1/1) redmine (Active: 1/1) metrics (Active: 1/1) redmine_test (Active: 1/1) forums (Active: 1/1) shop (Active: 3/3) studioexpress (Active: 3/3) svn (Active: 1/1) download (Active: 1/1) elections-test (Active: 1/1) riesling (Active: 1/1) pinot (Active: 1/1) nuka ( ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=elsa.infra.opensuse.org&service=HAProxy 2020-03-22T20:14:44 ok: this means that galera2 is now "a bit" behind the other two nodes, but this should get sorted out automatically during the next hours 2020-03-22T20:32:29 cboltz: and now you have single status reports :-) 2020-03-22T20:32:47 cboltz: especially the last one about the galera cluster contains some important informations for you ... 2020-03-22T20:37:12 nice, thanks! 2020-03-22T20:38:11 regarding the strange things in the wiki tables - no idea what caused this, especially since you wrote that this wasn't the case when you migrated the databases some months ago 2020-03-22T20:38:40 cboltz: I'm also not sure - the funny thing is that it does not affect all wikis, only some. 2020-03-22T20:38:43 sadly I didn't have time for upgrading mediawiki yet, and IIRC didn't do anything that could change the database scheme 2020-03-22T20:39:04 the "only same" is easily explained: 2020-03-22T20:39:17 cboltz: at least it looks like no major problem. At least I noticed no problems on the en wiki 2020-03-22T20:39:30 the wikis were created at different dates (depending on when each wiki was created) 2020-03-22T20:39:51 and while the mediawiki upgrade scripts work very well, it seems they don't care about database engines and things like that 2020-03-22T20:39:53 I will try to keep this sorted out during the next days. I just want to wait for the backup run tonight in case I break something... 2020-03-22T20:40:27 I hope some alter table commands will solve most of the issues 2020-03-22T20:40:38 yes, probably ;-) 2020-03-22T20:41:29 regarding the missing indexes - are the wiki tables without index small or big ones? 2020-03-22T20:42:37 (also, if you tell me the affected tables, maybe I can do educated guesses for possible indexes/primary keys) 2020-03-22T20:42:54 wiki_en.change_tag 2020-03-22T20:42:54 wiki_en.hit_counter_extension 2020-03-22T20:42:54 wiki_en.l10n_cache 2020-03-22T20:42:54 wiki_en.oldimage 2020-03-22T20:42:54 wiki_en.querycache 2020-03-22T20:43:32 haha: kicked because of excess flood ;-) 2020-03-22T20:43:49 the last table is wiki_en.user_properties 2020-03-22T20:44:10 give me a minute to look at these tables 2020-03-22T20:44:47 cboltz: relax 2020-03-22T20:44:52 I need to go to bed anyway. 2020-03-22T20:45:03 ok ;-) 2020-03-22T20:45:17 and good night! 2020-03-22T20:45:19 The last two days were a bit "stressful": my family decided to play some table games ... :-) 2020-03-22T20:45:28 until 03:00 in the morning.... 2020-03-22T20:45:42 and my son woke me up at 07:00 again 2020-03-22T20:45:42 sounds like you had some fun ;-) 2020-03-22T20:45:46 on both days 2020-03-22T20:45:55 jip: definitively a lot of fun :-) 2020-03-22T20:46:02 ;-) 2020-03-22T20:46:29 "CU" 2020-03-22T20:47:13 * kl_eisbaer dreams about a jitsi or nextcloud server (incl. talk) for openSUSE ... 2020-03-22T20:48:06 yeah, video conferences are great 2020-03-22T20:48:19 but even without an own server, I'm sure we'll find a way ;-) 2020-03-22T21:24:27 cboltz: well, we do have jangouts >:D 2020-03-22T21:24:41 out of principle kinda 2020-03-22T21:25:08 I have to admit that I never used it ;-) 2020-03-22T21:25:42 me neither 2020-03-22T21:26:01 it is only used by yast team really, as they are the ones that developed it 2020-03-22T21:26:12 and I don't hear good things outside of that circle sooo