2019-07-12T00:58:40 *** knurpht has quit IRC (Ping timeout: 246 seconds) 2019-07-12T02:41:18 *** okurz has quit IRC (Ping timeout: 245 seconds) 2019-07-12T02:42:06 *** okurz has joined #opensuse-admin 2019-07-12T03:33:48 *** srinidhi has quit IRC (Ping timeout: 245 seconds) 2019-07-12T04:27:20 *** srinidhi has joined #opensuse-admin 2019-07-12T05:51:30 *** marxin has quit IRC (Quit: Leaving) 2019-07-12T06:06:15 *** srinidhi has quit IRC (Quit: Leaving.) 2019-07-12T06:09:55 *** srinidhi has joined #opensuse-admin 2019-07-12T06:22:55 *** jadamek has joined #opensuse-admin 2019-07-12T06:27:47 *** ldevulder_ has joined #opensuse-admin 2019-07-12T06:31:46 *** moozaad has joined #opensuse-admin 2019-07-12T06:31:52 *** ldevulder has quit IRC (Ping timeout: 272 seconds) 2019-07-12T06:34:31 *** ldevulder_ is now known as ldevulder 2019-07-12T06:48:30 *** marxin has joined #opensuse-admin 2019-07-12T06:50:44 *** lkocman has joined #opensuse-admin 2019-07-12T06:50:52 *** lkocman has quit IRC (Remote host closed the connection) 2019-07-12T06:52:47 *** matthias_bgg has joined #opensuse-admin 2019-07-12T06:55:19 *** malcolmlewis has quit IRC (Ping timeout: 246 seconds) 2019-07-12T06:57:11 *** malcolmlewis has joined #opensuse-admin 2019-07-12T07:48:42 *** antlarr has joined #opensuse-admin 2019-07-12T07:49:46 *** antlarr2 has quit IRC (Ping timeout: 272 seconds) 2019-07-12T08:59:59 *** antlarr2 has joined #opensuse-admin 2019-07-12T09:00:28 *** antlarr has quit IRC (Ping timeout: 245 seconds) 2019-07-12T09:40:09 *** cboltz has joined #opensuse-admin 2019-07-12T10:20:17 *** jadamek2 has joined #opensuse-admin 2019-07-12T10:23:23 *** jadamek has quit IRC (Ping timeout: 245 seconds) 2019-07-12T10:50:03 *** lcp has joined #opensuse-admin 2019-07-12T11:27:14 *** knurpht has joined #opensuse-admin 2019-07-12T12:48:52 *** marxin has quit IRC (Quit: Leaving) 2019-07-12T12:50:35 *** matthias_bgg has quit IRC (Quit: Leaving) 2019-07-12T12:57:53 *** jadamek2 is now known as jadamek 2019-07-12T13:00:37 *** Pharaoh_Atem has quit IRC (Excess Flood) 2019-07-12T13:01:05 *** cboltz has quit IRC () 2019-07-12T13:01:06 *** Pharaoh_Atem has joined #opensuse-admin 2019-07-12T13:26:20 *** lcp has quit IRC (Remote host closed the connection) 2019-07-12T13:26:51 *** lcp has joined #opensuse-admin 2019-07-12T13:31:10 *** srinidhi has quit IRC (Ping timeout: 258 seconds) 2019-07-12T13:48:46 *** lcp has quit IRC (Quit: lcp) 2019-07-12T13:49:37 *** lcp has joined #opensuse-admin 2019-07-12T13:52:19 *** lcp has quit IRC (Remote host closed the connection) 2019-07-12T13:52:53 *** lcp has joined #opensuse-admin 2019-07-12T14:41:30 *** boombatower has joined #opensuse-admin 2019-07-12T14:50:05 *** lcp has quit IRC (Remote host closed the connection) 2019-07-12T15:03:19 *** agraul has joined #opensuse-admin 2019-07-12T16:04:57 *** antlarr has joined #opensuse-admin 2019-07-12T16:05:03 *** antlarr2 has quit IRC (Ping timeout: 245 seconds) 2019-07-12T16:16:29 *** cboltz has joined #opensuse-admin 2019-07-12T16:18:16 *** antlarr2 has joined #opensuse-admin 2019-07-12T16:19:12 *** antlarr has quit IRC (Ping timeout: 245 seconds) 2019-07-12T16:19:43 please have look at mirrorbrain / rsync.opensuse.org / download.o.o 2019-07-12T16:20:06 *** moozaad has quit IRC (Remote host closed the connection) 2019-07-12T16:20:28 *** moozaad has joined #opensuse-admin 2019-07-12T16:35:28 *** thomic has quit IRC (Ping timeout: 245 seconds) 2019-07-12T16:38:28 *** matthias_bgg has joined #opensuse-admin 2019-07-12T16:43:29 *** srinidhi has joined #opensuse-admin 2019-07-12T16:44:52 *** lkocman has joined #opensuse-admin 2019-07-12T16:45:15 Hello team, I did open a ticket with suse-infra as we have again very bad line to downloads.opensuse.org https://infra.nue.suse.com/SelfService/Display.html?id=142086 2019-07-12T16:45:18 I'll keep you posted 2019-07-12T16:45:23 (at infra.nue.suse.com) 2019-07-12T18:04:50 *** thomic has joined #opensuse-admin 2019-07-12T18:05:31 Jul 12 17:52:50 mirrordb1 su[21298]: pam_unix(su:session): session opened for user root by cboltz(uid=0) 2019-07-12T18:05:36 pls dont destroy it :D cboltz 2019-07-12T18:05:56 yes, I'm looking at it - but will only try things that (hopefully) don't break it ;-) 2019-07-12T18:06:07 already restarted apache on pontifex - no change 2019-07-12T18:06:17 do you know what the problem is? 2019-07-12T18:06:22 or are you just kicking tires? 2019-07-12T18:06:57 the apache error log had (and still has) lots of "[mod_mirrorbrain] Could not retrieve row from database for ..." 2019-07-12T18:07:45 monitoring doesn't help much, except that it got a timeout for measuring the pgsql bloat 2019-07-12T18:08:02 did you restart the db? 2019-07-12T18:08:10 i see this in the log 2019-07-12T18:08:13 it also reports a high number of pgsql threads on mirrordb2 2019-07-12T18:08:19 I restarted it on mirrordb1 2019-07-12T18:08:24 hrm 2019-07-12T18:08:38 given the high number of threads, maybe I should also do it on mirrordb2? 2019-07-12T18:09:02 pls dont 2019-07-12T18:09:07 i guess darix will look into it 2019-07-12T18:09:21 ok 2019-07-12T18:09:36 at least only if you know how to fix it :D 2019-07-12T18:09:36 I'll happily hand over to someone with more postgresql knownledge ;-) 2019-07-12T18:09:42 correct 2019-07-12T18:09:48 thats why i "dont just restart" 2019-07-12T18:09:56 if its really broken, thats even worse for the DB 2019-07-12T18:11:37 mirrordb2 (postgreSQL cluster):~ # su - postgres 2019-07-12T18:11:37 su: failed to execute /bin/bash: Resource temporarily unavailable 2019-07-12T18:11:44 this is maybe nothing good 2019-07-12T18:11:51 *** jadamek2 has joined #opensuse-admin 2019-07-12T18:12:42 indeed, also fails for me 2019-07-12T18:12:57 *** jadamek has quit IRC (Ping timeout: 245 seconds) 2019-07-12T18:13:04 sudo -u postgres bash works 2019-07-12T18:14:05 oh, there's one interesting thing in the monitoring - 17466 pgsql locks 2019-07-12T18:14:12 that might explain a lot... 2019-07-12T18:14:19 postgres=# SELECT pg_is_in_recovery(); 2019-07-12T18:14:19 pg_is_in_recovery 2019-07-12T18:14:19 ------------------- 2019-07-12T18:14:19 t 2019-07-12T18:14:38 *** knurpht has quit IRC (Ping timeout: 245 seconds) 2019-07-12T18:14:42 well the locks exist because nobody did the vacuum stuff last weeks, but darix will also work on that cboltz 2019-07-12T18:14:48 i'm just checking standard stuff 2019-07-12T18:14:52 to see how bad it is 2019-07-12T18:14:57 i'm not changing anything 2019-07-12T18:15:55 I have a feeling that we should make the vacuum a cronjob ;-) (even if darix doesn't like it, but better have two minutes slowness each day during vacuum than some hours of downtime every two weeks) 2019-07-12T18:16:45 cboltz: darix is thinking about that 2019-07-12T18:16:52 there are some things to keep in mind 2019-07-12T18:16:57 not regarding downtime 2019-07-12T18:17:06 but actually things like recovery-mode and stuff like that 2019-07-12T18:17:54 indeed, that makes it more interesting 2019-07-12T18:18:33 as a crazy idea - IIRC postgres does auto-vacuum when it's bored 2019-07-12T18:19:07 could we setup two slaves instead of one, and let mirrorbrain switch servers once per day so that the bored server can cleanup? 2019-07-12T18:19:33 i will not discuss this with the amount of knowledge i have about this setup 2019-07-12T18:19:37 i leave that up to darix 2019-07-12T18:19:40 who is maintaining this 2019-07-12T18:19:42 and taking care 2019-07-12T18:20:22 ok, I'll tell darix about this crazy idea when the current issue is fixed ;-) 2019-07-12T18:20:55 correct 2019-07-12T18:28:03 looks like we are back to normal speed :-) 2019-07-12T18:28:08 ? 2019-07-12T18:28:46 download.o.o works with usual speed, and monitoring no longer shows all the locks 2019-07-12T18:28:54 hrm 2019-07-12T18:28:58 maybe darix did something 2019-07-12T18:29:10 that's also my guess ;-) 2019-07-12T18:30:27 we still have 40 GB bloat, but - that's another problem ;-) 2019-07-12T18:31:37 *** darix has joined #opensuse-admin 2019-07-12T18:31:45 restart is never a solution until you understand the problem. 2019-07-12T18:31:50 in doubt you make things worse 2019-07-12T18:32:17 hi darix 2019-07-12T18:32:27 whatever you did to fix this - thanks! 2019-07-12T18:35:37 cboltz: can you update status.o.o? 2019-07-12T18:35:40 thanks 2019-07-12T18:35:53 yes, of course 2019-07-12T18:36:32 do you have some details, or should I just write "fixed again"? 2019-07-12T18:40:05 * cboltz takes the silence as "update it without details" 2019-07-12T18:40:54 vacuum running 2019-07-12T18:41:14 wait until vacuum is done 2019-07-12T18:42:29 too late, I already updated status.o.o before you wrote this 2019-07-12T18:42:51 on the positive side, download.o.o works at normal speed for me, so the impact of the vacuum seems to be limited 2019-07-12T18:48:31 yay cboltz sorry 2019-07-12T18:48:32 correct 2019-07-12T18:48:34 without details 2019-07-12T18:48:38 nobody reads it anyways 2019-07-12T18:48:39 :) 2019-07-12T18:48:56 I'm not sure about that 2019-07-12T18:49:13 for example, _I_ read it, and I'm always interested in details (aka "learning from errors") 2019-07-12T18:50:10 INFO: "filearr": found 496050 removable, 28146246 nonremovable row versions in 5901452 pages 2019-07-12T18:50:46 cboltz: but it was no error, just a minor relaxing time. even servers have fridays 2019-07-12T18:51:09 lol 2019-07-12T18:52:55 darix: monitoring complains about "unsupported packages" (which basically means package not from an openSUSE:* repo, in this case from the postgres repo) 2019-07-12T18:53:00 *** jadamek2 has quit IRC (Quit: Leaving) 2019-07-12T18:53:03 does the list on https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb2.infra.opensuse.org&service=Updates look correct/expected? 2019-07-12T18:53:11 (at monitor.opensuse.org) 2019-07-12T18:53:19 if yes, I can whitelist these packages in the monitoring to get rid of that warning 2019-07-12T18:54:46 cboltz: actually the check should tell you why it is warning 2019-07-12T18:54:52 and even sends you to the right file 2019-07-12T18:54:57 if you execute it on the host :) 2019-07-12T18:55:00 afail 2019-07-12T18:55:03 afaik 2019-07-12T18:55:41 I know how to check the package source (even if it's good to know that the check can help with that) 2019-07-12T18:56:03 but it can't tell if these unsupported packages are intentionally from another repo or not ;-) 2019-07-12T18:56:12 i cant see the list so i cant tell 2019-07-12T18:56:30 unsupported Packages: libpq5 postgresql postgresql-contrib postgresql-llvmjit postgresql-server postgresql11 postgresql11-contrib postgresql11-ip4r postgresql11-llvmjit postgresql11-repmgr postgresql11-server repmgr 2019-07-12T18:58:00 ah 2019-07-12T18:58:10 yes they are... as we have there brand-new stuff 2019-07-12T18:58:12 :D 2019-07-12T18:58:30 ok, expect a MR to whitelist them in some minutes ;-) 2019-07-12T19:09:11 zypper ref for 32bit here in FL was creepy slow, and the mozilla repos were not available at all 2019-07-12T19:09:30 well a-865k should be fixed by now? 2019-07-12T19:09:33 a-865k: that shold be fixed? 2019-07-12T19:09:48 well no 2019-07-12T19:09:48 the standards came through, but not the mozillas 2019-07-12T19:09:55 check if they have mirrors 2019-07-12T19:23:24 https://gitlab.infra.opensuse.org/infra/salt/pipelines/1246 has the whitelisting 2019-07-12T19:23:33 (at gitlab.infra.opensuse.org) 2019-07-12T19:23:42 sorry, better link: https://gitlab.infra.opensuse.org/infra/salt/merge_requests/250 2019-07-12T19:23:50 (at gitlab.infra.opensuse.org) 2019-07-12T19:25:30 i'm out for today 2019-07-12T19:25:35 see ya people 2019-07-12T19:25:39 have a nice weekend 2019-07-12T19:25:53 enjoy the evening and the weekend ;-) 2019-07-12T19:43:21 http://download.opensuse.org/repositories/mozilla/openSUSE_Tumbleweed/x86_64/ loaded right up in browser, and another zypper ref went very fast 2019-07-12T19:43:30 (at download.opensuse.org) 2019-07-12T19:46:01 *** matthias_bgg has quit IRC (Ping timeout: 246 seconds) 2019-07-12T21:02:13 *** jadamek has joined #opensuse-admin 2019-07-12T21:04:19 *** moozaad has quit IRC (Quit: Konversation terminated!) 2019-07-12T21:54:42 *** cboltz has quit IRC () 2019-07-12T22:22:29 *** jadamek has quit IRC (Quit: Leaving) 2019-07-12T22:46:28 *** boombatower has quit IRC (Quit: Konversation terminated!) 2019-07-12T23:40:44 *** a-865k has quit IRC (Quit: using sirc version 2.211+KSIRC/1.3.12)