2018-01-02T01:35:21 *** fvogt has quit IRC 2018-01-02T01:52:00 *** Son_Goku has quit IRC 2018-01-02T02:25:06 *** Son_Goku has joined #opensuse-admin 2018-01-02T03:42:31 *** okurz has quit IRC 2018-01-02T03:43:10 *** okurz has joined #opensuse-admin 2018-01-02T03:46:18 *** Son_Goku has quit IRC 2018-01-02T03:52:55 *** plinnell has quit IRC 2018-01-02T04:08:42 *** plinnell has joined #opensuse-admin 2018-01-02T04:38:03 *** nicolasbock has quit IRC 2018-01-02T04:45:43 *** plinnell has quit IRC 2018-01-02T04:51:48 *** plinnell has joined #opensuse-admin 2018-01-02T04:51:48 *** plinnell has joined #opensuse-admin 2018-01-02T05:25:21 *** tampakrap has quit IRC 2018-01-02T08:22:57 *** tigerfoot has quit IRC 2018-01-02T08:33:48 *** tigerfoot has joined #opensuse-admin 2018-01-02T08:39:38 *** marxin has joined #opensuse-admin 2018-01-02T08:41:58 *** default__ is now known as ldevulder 2018-01-02T08:50:15 *** asmorodskyi has joined #opensuse-admin 2018-01-02T09:27:33 *** mcaj has joined #opensuse-admin 2018-01-02T09:36:34 *** Son_Goku has joined #opensuse-admin 2018-01-02T10:09:59 *** sven15 has joined #opensuse-admin 2018-01-02T10:10:24 *** tampakrap has joined #opensuse-admin 2018-01-02T10:55:42 *** kollex has joined #opensuse-admin 2018-01-02T11:09:01 *** cboltz has joined #opensuse-admin 2018-01-02T11:09:01 *** cboltz has joined #opensuse-admin 2018-01-02T11:10:42 asmorodskyi: I put a (late) christmas present in your ~/bin/ on monitor.infra.o.o 2018-01-02T11:25:20 good day cboltz 2018-01-02T11:39:14 *** ldevulder has quit IRC 2018-01-02T11:39:25 *** ldevulder has joined #opensuse-admin 2018-01-02T11:44:30 cboltz: thanks ! I will take a look later today :) 2018-01-02T11:44:49 Happy New Year - Heroes :) 2018-01-02T11:51:45 hi asmorodskyi 2018-01-02T11:52:21 Hi 2018-01-02T11:53:31 happy new year to you too :) 2018-01-02T11:58:08 *** nicolasbock has joined #opensuse-admin 2018-01-02T12:03:36 *** fvogt has joined #opensuse-admin 2018-01-02T12:57:58 *** cboltz has quit IRC 2018-01-02T14:00:29 *** ldevulder has quit IRC 2018-01-02T14:18:50 sysrich: https://progress.opensuse.org/issues/27196 not sure if you got a notification about this already 2018-01-02T14:19:11 sysrich: should I move it to the board redmine project? 2018-01-02T14:54:47 *** kollex has left #opensuse-admin 2018-01-02T15:06:27 *** ldevulder has joined #opensuse-admin 2018-01-02T15:10:11 *** asmorodskyi has quit IRC 2018-01-02T15:45:32 *** cboltz has joined #opensuse-admin 2018-01-02T15:57:23 tampakrap: thanks for your work on the open tickets 2018-01-02T15:57:27 two notes: 2018-01-02T15:57:53 - the monitoring is still or again down :-( - syslog.infra.o.o/icinga/ shows an error 2018-01-02T15:58:03 - ticket 29918 is funny[tm] ;-) 2018-01-02T15:59:31 *** mcaj has quit IRC 2018-01-02T16:01:03 29918 is the definition of communication failure 2018-01-02T16:01:14 exactly 2018-01-02T16:01:17 can you check the monitoring please? I'm quite packed with stuff 2018-01-02T16:01:38 maybe you mentioned progress.o.o too often, and www.o.o not often enough ;-) 2018-01-02T16:01:56 or maybe he read the summary only 2018-01-02T16:02:01 and not the content 2018-01-02T16:02:43 just tried to restart icinga - still Error: Failed to load module '/usr/lib/check_mk/livestatus.o'. 2018-01-02T16:05:01 *** megamaced has joined #opensuse-admin 2018-01-02T16:07:03 yep, can you fix it? 2018-01-02T16:07:30 I'm afraid I know nothing about icinga 2018-01-02T16:07:41 and googling for the error message was less useful than usual 2018-01-02T16:07:43 just try to downgrade the package 2018-01-02T16:09:17 the check_mk, monitoring-plugins-*, nrpe and pnp4nagios packages are from server:monitoring, I doubt the old version still exists there 2018-01-02T16:11:33 they are in openSUSE:infrastructure, you can try to linkpac previous version 2018-01-02T16:11:38 do you know how to do that? 2018-01-02T16:13:57 need to go to catch a shop before it closes, I'll be back in an hour plus 2018-01-02T16:17:42 yes, I know zypper in --from $repo ;-) 2018-01-02T16:23:58 I replaced the monitoring packages with those from openSUSE:infrastructure - still the same error message :-( 2018-01-02T16:39:32 found it - zypper in strace 2018-01-02T16:39:41 bind(8, {sa_family=AF_LOCAL, sun_path="/var/spool/check_mk/livestatus.cmd"}, 36) = -1 EACCES (Permission denied) 2018-01-02T16:40:13 write(3, "[1514911191] livestatus: Unable to bind adress /var/spool/check_mk/livestatus.cmd to UNIX socket: Permission denied\n", 116) = 116 2018-01-02T16:42:15 workaround (!): chmod 777 /var/spool/check_mk 2018-01-02T16:42:27 /var/spool/check_mk is owned by nagios:nagcmd, but the socket gets created as icinga:icinga 2018-01-02T16:45:13 *** megamaced has quit IRC 2018-01-02T16:45:55 now the monitoring is up again, but completely red 2018-01-02T16:46:17 check_icmp: Failed to obtain ICMP socket: Operation not permitted for 45 hosts 2018-01-02T16:46:36 sounds like another permission problem... 2018-01-02T16:50:11 ah, there's a /etc/permissions.d/check_mk 2018-01-02T16:50:33 one chkstat --system --set later, things look better :-) 2018-01-02T16:54:07 *** tigerfoot has quit IRC 2018-01-02T16:54:47 *** tigerfoot has joined #opensuse-admin 2018-01-02T17:08:02 *** tigerfoot has quit IRC 2018-01-02T17:08:31 *** tigerfoot has joined #opensuse-admin 2018-01-02T17:08:49 PROBLEM: memcached data stored on riesling.infra.opensuse.org - MEMCACHE CRITICAL: bytes is 246478114 200000000 - memcached 1.4.33 on 127.0.0.1:11211, up 13 days 21 hours ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=riesling.infra.opensuse.org&service=memcached%20data%20stored 2018-01-02T17:11:17 *** sven15 has quit IRC 2018-01-02T17:42:55 *** asmorodskyi has joined #opensuse-admin 2018-01-02T17:49:30 cboltz: I meant to downgrade the opensuse:infrastructure package 2018-01-02T17:50:28 the updated packages in question (according to the /v/l/zypp/histoy were from the monitoring repo, so switching them to the frozen version in the infra repo made more sense for me ;-) 2018-01-02T17:50:54 I have no problem with switching them back to the monitoring repo if you think that's the better choice 2018-01-02T17:52:01 last time I checked (during the christmas vacation days) the opensuse:infrastructure and the monitoring repo packages had same versions 2018-01-02T17:52:20 and I am pretty sure I checked also the opensuse:infrastructure packages if they were locked and they weren't 2018-01-02T17:52:45 maybe there was a rebuild in monitoring which triggered the package installations/updates on Dec 22 2018-01-02T17:53:11 after all the debugging I did, I'm quite sure packages from both repos work 2018-01-02T17:53:22 as long as we fix the permissions with chkstat :-/ 2018-01-02T17:53:56 a socket has wrong perms? 2018-01-02T17:54:46 several directories have wrong perms, one of them (the first one I found with strace) is meant to store a socket 2018-01-02T17:56:29 so packaging issue then? 2018-01-02T17:56:54 /etc/permissions.d/check_mk fixes them all, but it seems the package update didn't trigger chkstat (or didn't trigger it in the right way) 2018-01-02T17:57:08 so yes, probably a packaging issue 2018-01-02T17:57:16 *** plinnell has quit IRC 2018-01-02T17:58:05 and that's where the fun starts 2018-01-02T17:58:20 /etc/permissions.d/check_mk is owned by the check_mk package 2018-01-02T17:58:44 and the postinstall script of check_mk calls /usr/bin/chkstat -n --set --system /etc/permissions.d/check_mk 2018-01-02T18:00:20 so in theory it should work... 2018-01-02T18:00:30 *** plinnell has joined #opensuse-admin 2018-01-02T18:00:31 *** plinnell has joined #opensuse-admin 2018-01-02T18:03:39 okay 2018-01-02T18:03:57 let's see what happens on the next update then? 2018-01-02T18:04:24 maybe I start to see the reason - it could depend on the install order 2018-01-02T18:04:51 /var/spool/check_mk is packaged in mk-livestatus as nagios:nagcmd 2018-01-02T18:05:39 %post of this package calls /usr/bin/chkstat -n --set --system /etc/permissions.d/mk-livestatus but this file doesn't exist, so it probably does nothing 2018-01-02T18:06:13 so if mk-livestatus gets installed after check_mk, that explains the wrong permissions at least for this directory 2018-01-02T18:09:40 the permissions (adding the suid bit) for /usr/lib/nagios/plugins/check_icmp get done in another permissions file, which is not owned by any package 2018-01-02T18:13:10 for check_mk, the .changes is funny: 2018-01-02T18:13:12 - remove obsolete permissions.mk-livestatus 2018-01-02T18:13:30 that's from 2015 (!) so I'm surprised this didn't bite earlier 2018-01-02T18:19:38 created request id 561178 2018-01-02T18:23:49 tampakrap: should I switch back the monitoring packages to server:monitoring? 2018-01-02T18:24:19 (we still have ~30 packages from there which don't exist in the infrastructure repo) 2018-01-02T18:29:32 I don't know, I'm confused now 2018-01-02T18:55:56 *** plinnell has quit IRC 2018-01-02T19:03:15 *** plinnell has joined #opensuse-admin 2018-01-02T19:03:15 *** plinnell has joined #opensuse-admin 2018-01-02T19:24:13 *** asmorodskyi has quit IRC 2018-01-02T19:35:24 tampakrap: originally (before I used zypper in --from) all monitoring-related packages were from server:monitoring 2018-01-02T19:36:47 then I "zypper in --from infra"d all packages that also exist in the infra repo to that repo - but there are still ~30 packages from server:monitoring installed because those packages don't exist in the infra repo 2018-01-02T19:37:20 so at the moment we have monitoring packages from infra _and_ from server:monitoring installed 2018-01-02T19:37:31 does that solve the confusion? 2018-01-02T19:40:41 that I understood already, but I'm not sure which package is better 2018-01-02T19:41:26 I'd say the current situation (with a mix of frozen packages from infra and non-frozen packages from monitoring) is the worst choice 2018-01-02T19:41:40 so either we switch everything back to server:monitoring 2018-01-02T19:41:59 or we linkpac the ~30 packages to infra and get rid of the monitoring repo - but that sounds like superfluous work to me 2018-01-02T19:42:20 (assuming we need those packages only on monitor.infra.o.o) 2018-01-02T19:51:18 subproject then 2018-01-02T19:51:51 well, yes and no 2018-01-02T19:52:13 I'm quite sure lots of the packages we already have in infra are used on various hosts 2018-01-02T19:52:50 so if we create a subproject for "the other monitoring packages", we might have to move them to the main infra repo later 2018-01-02T19:54:30 we can figure out which packages are only for the monitoring host then 2018-01-02T19:54:41 and if they are more than 5, then subproject 2018-01-02T19:56:22 have a look at http://paste.opensuse.org/25bc8015 - these are all packages that are not from the oss repo 2018-01-02T20:02:49 okay 2018-01-02T20:03:00 now to figure out which of those are needed in other hosts as well 2018-01-02T20:03:03 PROBLEM: HAProxy on mufasa.infra.opensuse.org - HAPROXY CRITICAL - Active service narwal4 is DOWN on static proxy ! Active service riesling is DOWN on riesling proxy ! ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mufasa.infra.opensuse.org&service=HAProxy 2018-01-02T20:16:23 salt \* cmd.run 'zypper lr' ;-) - that reduces the number of hosts you have to check to those with additional repos 2018-01-02T20:17:23 in general, I'd say all monitoring-plugins-* should be in the main infra repo next to the ~20 monitoring-plugins-* that are already there 2018-01-02T20:19:45 sure 2018-01-02T20:20:34 anyway, enough for today 2018-01-02T22:45:45 *** dddh has quit IRC