2018-06-26T01:55:53 *** plinnell has joined #opensuse-admin 2018-06-26T01:55:54 *** plinnell has joined #opensuse-admin 2018-06-26T02:09:45 *** okurz has quit IRC 2018-06-26T02:11:19 *** okurz has joined #opensuse-admin 2018-06-26T02:32:33 *** Son_Goku has joined #opensuse-admin 2018-06-26T03:02:49 *** Son_Goku has quit IRC 2018-06-26T03:25:14 *** plinnell has quit IRC 2018-06-26T03:33:28 *** plinnell has joined #opensuse-admin 2018-06-26T03:56:37 *** plinnell has quit IRC 2018-06-26T04:04:20 *** plinnell has joined #opensuse-admin 2018-06-26T04:11:39 PROBLEM: HTTP rsync.o.o on widehat.opensuse.org - HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 714 bytes in 0.004 second response time ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=widehat.opensuse.org&service=HTTP%20rsync.o.o 2018-06-26T04:40:30 *** dddh is now known as dddh__ 2018-06-26T05:03:46 *** dddh_ is now known as dddh 2018-06-26T05:40:21 *** sven15 has joined #opensuse-admin 2018-06-26T06:07:33 *** sven15 has quit IRC 2018-06-26T06:40:21 *** plinnell has quit IRC 2018-06-26T06:40:27 *** nicolasbock has joined #opensuse-admin 2018-06-26T06:47:42 *** duncanmv has joined #opensuse-admin 2018-06-26T06:51:21 PROBLEM: NRPE on widehat.opensuse.org - connect to address 62.146.92.203 port 5666: Connection refused ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=widehat.opensuse.org&service=NRPE 2018-06-26T06:52:29 PROBLEM: rsync on widehat.opensuse.org - rsync: failed to connect to rsync.opensuse.org (62.146.92.202): Connection refused (111) ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=widehat.opensuse.org&service=rsync 2018-06-26T07:04:54 widehat: i have stopped rsync and nginx. 2018-06-26T07:44:32 *** ldevulder_ is now known as ldevulder 2018-06-26T07:49:12 *** enavarro has joined #opensuse-admin 2018-06-26T07:51:21 RECOVERY: NRPE on widehat.opensuse.org - NRPE v3.2.1 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=widehat.opensuse.org&service=NRPE 2018-06-26T07:51:38 RECOVERY: HTTP rsync.o.o on widehat.opensuse.org - HTTP OK: HTTP/1.1 200 OK - 1217 bytes in 0.264 second response time ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=widehat.opensuse.org&service=HTTP%20rsync.o.o 2018-06-26T07:51:43 *** sven15 has joined #opensuse-admin 2018-06-26T07:52:29 RECOVERY: rsync on widehat.opensuse.org - OK: Rsync is up ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=widehat.opensuse.org&service=rsync 2018-06-26T07:56:50 *** sven15 has quit IRC 2018-06-26T08:06:21 widehat: ports/ and repositories/ are taking up the space. can anyone think of a good reason why I should not exclude openSUSE_Leap_42.1 directories from all repos? 2018-06-26T08:10:17 *** sven15 has joined #opensuse-admin 2018-06-26T08:15:26 *** matthias_bgg has joined #opensuse-admin 2018-06-26T08:30:15 *** dddh__ is now known as dddh_ 2018-06-26T08:33:13 *** asmorodskyi has quit IRC 2018-06-26T08:38:03 *** asmorodskyi has joined #opensuse-admin 2018-06-26T09:08:06 *** enavarro has quit IRC 2018-06-26T09:12:14 *** enavarro has joined #opensuse-admin 2018-06-26T09:33:35 *** sven15 has quit IRC 2018-06-26T09:50:37 tampakrap: this would be great https://hackweek.suse.com/17/projects/microservices-and-serverless-for-the-opensuse-dot-org-infrastructure 2018-06-26T09:52:14 ChrisBr: yep, there's already an instance running on caasp-admin.i.o.o 2018-06-26T09:52:32 and the trello board I linked there has tasks to be figured out :) 2018-06-26T09:52:34 I would love to use this for osem 2018-06-26T09:52:53 PROBLEM: PerfData on monitor.infra.opensuse.org - CRITICAL: Found files older than 2880 minutes /var/lib/pnp4nagios/perfdata//proc_Rsyslog_Process.xml was last modified on Thu Jun 21 09:42:54 2018 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=monitor.infra.opensuse.org&service=PerfData 2018-06-26T09:54:12 caasp-admin.i.o.o is running in heroes network I guess? 2018-06-26T09:55:55 yes, everything behind i.o.o is 2018-06-26T09:56:23 ok 2018-06-26T09:57:18 hi guys, would it be ok if I add maxlin to https://progress.opensuse.org/projects/opensuse-admin ? 2018-06-26T09:57:45 okurz: sure 2018-06-26T09:59:27 maxlin: I added "maxlin_factory". please try to access the wiki link I gave you 2018-06-26T09:59:58 ok, thanks :) 2018-06-26T10:11:01 *** Son_Goku has joined #opensuse-admin 2018-06-26T10:17:53 maxlin: 50 bucks 2018-06-26T10:19:52 *** matthias_bgg has quit IRC 2018-06-26T10:22:16 tampakrap: haha, or I can buy you a couple of beers next time we meet :) 2018-06-26T10:41:36 *** cboltz has joined #opensuse-admin 2018-06-26T10:41:36 *** cboltz has joined #opensuse-admin 2018-06-26T10:42:25 PROBLEM: SSH on mickey.infra.opensuse.org - SSH CRITICAL - OpenSSH_7.6 (protocol 2.0) version mismatch, expected OpenSSH_7.2 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mickey.infra.opensuse.org&service=SSH 2018-06-26T10:51:17 *** dddh_ has quit IRC 2018-06-26T10:53:21 nice - syncing the update repos to provo-mirror worked :-) 2018-06-26T10:53:35 now I'll install the ~90 updates... 2018-06-26T10:58:31 FYI: one of the updates was a kernel update, so I'll reboot provo-mirror now 2018-06-26T10:59:04 when I did that last time, the reboot took several minutes, so don't panic ;-) 2018-06-26T11:04:37 *** matthias_bgg has joined #opensuse-admin 2018-06-26T11:26:59 just checked - provo-mirror works as expected :-) 2018-06-26T11:27:22 (but still has to rsync a lot) 2018-06-26T11:28:14 gitlab.i.o.o upgraded to leap15 2018-06-26T11:28:23 :-) 2018-06-26T11:32:26 RECOVERY: SSH on mickey.infra.opensuse.org - SSH OK - OpenSSH_7.6 (protocol 2.0) ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mickey.infra.opensuse.org&service=SSH 2018-06-26T13:09:54 *** megamaced has joined #opensuse-admin 2018-06-26T13:24:12 *** Son_Goku has quit IRC 2018-06-26T13:27:14 *** paka has quit IRC 2018-06-26T13:30:54 *** paka has joined #opensuse-admin 2018-06-26T13:30:54 *** paka has joined #opensuse-admin 2018-06-26T13:40:25 *** paka has quit IRC 2018-06-26T13:41:21 *** paka has joined #opensuse-admin 2018-06-26T14:02:37 *** duncanmv has quit IRC 2018-06-26T14:12:35 forgot to send an update - widehat is alive and doing well. 2018-06-26T14:54:02 *** megamaced has quit IRC 2018-06-26T15:03:20 *** cboltz has quit IRC 2018-06-26T15:23:53 *** duncanmv has joined #opensuse-admin 2018-06-26T15:35:45 *** matthias_bgg has quit IRC 2018-06-26T15:52:24 *** ldevulder_ has joined #opensuse-admin 2018-06-26T15:55:27 *** ldevulder has quit IRC 2018-06-26T15:57:14 *** Son_Goku has joined #opensuse-admin 2018-06-26T16:11:49 *** mcaj has quit IRC 2018-06-26T16:37:45 *** Son_Goku has quit IRC 2018-06-26T16:41:37 *** Son_Goku has joined #opensuse-admin 2018-06-26T16:58:22 *** dakechi has quit IRC 2018-06-26T17:15:20 *** enavarro has quit IRC 2018-06-26T17:58:11 *** Son_Goku has quit IRC 2018-06-26T18:01:10 *** Son_Goku has joined #opensuse-admin 2018-06-26T18:44:57 *** dddh has quit IRC 2018-06-26T19:00:08 *** dddh has joined #opensuse-admin 2018-06-26T19:00:45 *** cboltz has joined #opensuse-admin 2018-06-26T19:00:45 *** cboltz has joined #opensuse-admin 2018-06-26T19:12:49 *** Sir_Gallantmon has joined #opensuse-admin 2018-06-26T19:13:57 *** Son_Goku has quit IRC 2018-06-26T19:19:49 PROBLEM: HTTP rsync.o.o on widehat.opensuse.org - CRITICAL - Socket timeout after 10 seconds ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=widehat.opensuse.org&service=HTTP%20rsync.o.o 2018-06-26T19:29:39 RECOVERY: HTTP rsync.o.o on widehat.opensuse.org - HTTP OK: HTTP/1.1 200 OK - 1217 bytes in 0.735 second response time ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=widehat.opensuse.org&service=HTTP%20rsync.o.o 2018-06-26T19:29:41 *** IonutVan_ has quit IRC 2018-06-26T19:31:58 *** IonutVan_ has joined #opensuse-admin 2018-06-26T19:36:17 cboltz: any idea what are the latest mails we get from provo-mirror in admin-auto? 2018-06-26T19:43:46 they are probably the result to fix the postfix config - before, they were rejected because they were From: mirror@localhost 2018-06-26T19:44:07 so it isn't new breakage, "just" new notifications about the breakage 2018-06-26T19:44:43 ah cool 2018-06-26T19:45:27 so the breakage is fixed already? 2018-06-26T19:45:44 no, otherwise you wouldn't see those mails ;-) 2018-06-26T19:49:00 what's the issue then? 2018-06-26T19:50:46 one of the failing commands is 2018-06-26T19:50:51 /usr/bin/rsync -aqr0H --no-motd '--exclude=/repodata/*' stage.opensuse.org::opensuse-full-really-everything/opensuse/update/leap/42.2/ /srv/ftp/pub/opensuse/update/leap/42.2/ 2018-06-26T19:51:20 which says that 42.2 doesn't exist 2018-06-26T19:51:34 I confirmed this with rsync stage.opensuse.org::opensuse-full-really-everything/opensuse/update/leap/ 2018-06-26T19:52:05 it shows 15.0, 15.0-test, 42.3, 42.3-test, 42.2-test and 42.1 - but not 42.2 or 42.1-test 2018-06-26T19:53:17 for source/update/leap, only 42.x and 15.0 exist, but none of the *-test 2018-06-26T19:54:03 it exists on pontifex 2018-06-26T19:54:38 [opensuse-full-really-everything] 2018-06-26T19:54:40 path = /srv/ftp-stage/pub 2018-06-26T19:55:03 pontifex2 (download.o.o):~ # l -d /srv/ftp-stage/pub/opensuse/update/leap/42.2* 2018-06-26T19:55:05 drwxr-xr-x 4 mirror stage 30 Feb 15 2017 /srv/ftp-stage/pub/opensuse/update/leap/42.2/ 2018-06-26T19:55:07 drwxr-xr-x 7 mirror stage 116 Jan 26 17:15 /srv/ftp-stage/pub/opensuse/update/leap/42.2-test/ 2018-06-26T19:56:00 and the exclude line of that rsync module is 2018-06-26T19:56:02 exclude = /opensuse/repositories drpmsync/*** 10.3-test/*** 11.0-test/*** 11.1-test/*** 11.2-test/*** /opensuse/distribution/10.2 2018-06-26T19:56:09 so it should be shown 2018-06-26T19:56:15 am I wrong? 2018-06-26T19:56:43 update/leap/42.1-test really doesn't exist on pontifex 2018-06-26T19:57:05 but update/leap/42.2 exists, so it's surprising that it causes errors 2018-06-26T20:00:04 PROBLEM: HAProxy on elsa.infra.opensuse.org - HAPROXY CRITICAL - Active service gccstats is DOWN on gccstats proxy ! Active service dale is DOWN on dale proxy ! ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=elsa.infra.opensuse.org&service=HAProxy 2018-06-26T20:06:04 an interesting detail is that the size of update (249 GB) and tumbleweed (85 GB) is correct - but maybe one of the other rsync cronjobs helps to fix that 2018-06-26T20:08:25 actually I wonder if it reallly makes sense to let the sync_update.sh script check for every version separately, or if it should just sync all of /update/ in one big step 2018-06-26T20:08:43 (please have a look at /home/mirror/bin/sync_update.sh on provo-mirror before answering ;-) 2018-06-26T20:09:42 *** Sir_Gallantmon has quit IRC 2018-06-26T20:11:35 on a somewhat unrelated note - I just removed the -v option from sync_tumbleweed.sh to avoid superfluous "look what I just synced" mails 2018-06-26T20:13:35 checking 2018-06-26T20:15:40 oh, nice - sync_tumbleweed.sh had a nice crontab timing: * */2 * * * which means it did run every minute, but only during even hours ;-) 2018-06-26T20:16:35 *** Son_Goku has joined #opensuse-admin 2018-06-26T20:16:35 how do I login to provo-mirror? 2018-06-26T20:16:39 port 2222? 2018-06-26T20:16:39 I just changed it to 15 */2 * * * so that it runs every two hours 2018-06-26T20:17:02 via status2.o.o, and yes, port 2222 2018-06-26T20:17:13 I'll log in through mufasa then 2018-06-26T20:18:32 actually it is even easier, you can log in through any nuremberg vm 2018-06-26T20:18:47 because the suse prg/nue/prv ips are all whitelisted 2018-06-26T20:22:11 inside the for loop, the first three rsync commands are identical? 2018-06-26T20:22:15 or I'm missing something? 2018-06-26T20:22:50 they are slightly different 2018-06-26T20:23:02 the first has --exclude=/repodata/* 2018-06-26T20:23:11 the second has --include=/repodata/* 2018-06-26T20:23:18 ah yes 2018-06-26T20:23:19 and the third has --delte 2018-06-26T20:23:22 why? 2018-06-26T20:23:45 no idea, I didn't write those scripts ;-) 2018-06-26T20:23:54 I'd guess Lars wrote them 2018-06-26T20:24:11 there should be a reason 2018-06-26T20:24:20 and we're missing it 2018-06-26T20:24:27 my guess is: 2018-06-26T20:24:35 - first get the new (additional) packages 2018-06-26T20:24:43 - then sync the repodata so that the new packages get used 2018-06-26T20:24:53 - finally delete packages that are gone 2018-06-26T20:25:08 makes sence 2018-06-26T20:28:36 I don't know man 2018-06-26T20:28:41 it looks fine to me, I have no idea why it fails 2018-06-26T20:29:20 the rsync command that you gave earlier 2018-06-26T20:29:27 you gave it to the provo-mirror? 2018-06-26T20:31:52 yes 2018-06-26T20:32:55 you removed 42.1-test already from the script right? 2018-06-26T20:33:41 no, it wasn't there 2018-06-26T20:33:54 (my changes all have a [cboltz] comment) 2018-06-26T20:35:24 * tampakrap is clueless 2018-06-26T20:37:17 maybe pjessen has an idea? 2018-06-26T20:40:02 PROBLEM: HAProxy on anna.infra.opensuse.org - HAPROXY CRITICAL - Active service dale is DOWN on dale proxy ! Active service gccstats is DOWN on gccstats proxy ! ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=anna.infra.opensuse.org&service=HAProxy 2018-06-26T20:42:01 another interesting question - disk usage 2018-06-26T20:42:13 http://paste.opensuse.org/7c9dfa47 is a comparison of provo-mirror and pontifex 2018-06-26T20:42:42 if I read the sync scripts right, pontifex /srv/ftp-stage gets synced to provo-mirror /srv/ftp 2018-06-26T20:42:44 but 2018-06-26T20:43:05 a) provo-mirror has much more data in some of the /srv/ftp/ directories 2018-06-26T20:44:01 b) provo-mirror also has /srv/ftp-stage - no idea if it gets used or synced 2018-06-26T20:48:55 *** Son_Goku has quit IRC 2018-06-26T20:56:26 maybe it was used only during the blackout? 2018-06-26T20:57:14 maybe 2018-06-26T20:57:35 /etc/rsyncd.conf on provo-mirror refers to /srv/ftp-stage, so your guess doesn't sound too wrong 2018-06-26T20:57:58 *** Son_Goku has joined #opensuse-admin 2018-06-26T20:59:01 hmm, actually it's more boring (and I should have checked this before) 2018-06-26T20:59:15 ftp-stage is a symlink to ftp 2018-06-26T20:59:39 (and the size differences between ftp and ftp-stage might be caused by a sync happening in the meantime) 2018-06-26T21:00:25 so one mystery solved - but this still doesn't explain why for example "distribution" is much bigger on provo-mirror than on pontifex 2018-06-26T21:06:28 *** duncanmv has quit IRC 2018-06-26T21:08:17 *** Son_Goku has quit IRC 2018-06-26T21:42:03 *** Son_Goku has joined #opensuse-admin 2018-06-26T22:24:16 *** cboltz has quit IRC 2018-06-26T22:35:15 *** Fraser_Bell has joined #opensuse-admin 2018-06-26T22:35:15 *** Fraser_Bell has joined #opensuse-admin 2018-06-26T22:44:32 *** Son_Goku has quit IRC 2018-06-26T22:55:49 PROBLEM: HTTP rsync.o.o on widehat.opensuse.org - CRITICAL - Socket timeout after 10 seconds ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=widehat.opensuse.org&service=HTTP%20rsync.o.o 2018-06-26T23:06:52 *** nicolasbock has quit IRC 2018-06-26T23:21:31 *** Fraser_Bell has quit IRC 2018-06-26T23:25:38 RECOVERY: HTTP rsync.o.o on widehat.opensuse.org - HTTP OK: HTTP/1.1 200 OK - 1217 bytes in 0.466 second response time ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=widehat.opensuse.org&service=HTTP%20rsync.o.o 2018-06-26T23:42:14 *** Son_Goku has joined #opensuse-admin