2017-10-06T01:26:41 *** ChrisBr has quit IRC 2017-10-06T01:27:05 *** ChrisBr has joined #opensuse-admin 2017-10-06T02:06:41 *** skiarxon has quit IRC 2017-10-06T02:06:41 *** differentreality has quit IRC 2017-10-06T02:06:54 *** skiarxon has joined #opensuse-admin 2017-10-06T02:08:24 *** differentreality has joined #opensuse-admin 2017-10-06T02:08:24 *** differentreality has joined #opensuse-admin 2017-10-06T02:08:32 *** tampakrap has quit IRC 2017-10-06T02:09:54 *** dirk_ has joined #opensuse-admin 2017-10-06T02:11:01 *** tampakrap has joined #opensuse-admin 2017-10-06T02:11:19 *** awafaa_ has joined #opensuse-admin 2017-10-06T02:12:38 *** awafaa has quit IRC 2017-10-06T02:12:38 *** dirk has quit IRC 2017-10-06T02:12:41 *** plusky has quit IRC 2017-10-06T02:12:42 *** aboe[m] has quit IRC 2017-10-06T02:12:45 *** dirk_ is now known as dirk 2017-10-06T02:12:45 *** awafaa_ is now known as awafaa 2017-10-06T02:13:13 *** plusky has joined #opensuse-admin 2017-10-06T02:15:07 PROBLEM: Fail2Ban on status1.opensuse.org - CHECK FAIL2BAN ACTIVITY - CRITICAL - 1 detected jails with 2 current banned IP(s) ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=status1.opensuse.org&service=Fail2Ban 2017-10-06T02:16:02 *** IonutVan_ has quit IRC 2017-10-06T02:16:12 *** aboe[m] has joined #opensuse-admin 2017-10-06T02:20:41 *** IonutVan_ has joined #opensuse-admin 2017-10-06T02:35:59 *** IonutVan_ has quit IRC 2017-10-06T02:41:16 *** okurz has quit IRC 2017-10-06T02:41:26 *** IonutVan_ has joined #opensuse-admin 2017-10-06T02:41:42 *** okurz_ has joined #opensuse-admin 2017-10-06T02:41:50 *** okurz_ is now known as okurz 2017-10-06T03:00:17 *** nicolasbock_ has quit IRC 2017-10-06T03:01:02 *** nicolasbock has quit IRC 2017-10-06T03:25:07 PROBLEM: Fail2Ban on status1.opensuse.org - CHECK FAIL2BAN ACTIVITY - CRITICAL - 1 detected jails with 2 current banned IP(s) ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=status1.opensuse.org&service=Fail2Ban 2017-10-06T05:11:16 *** Son_Goku has joined #opensuse-admin 2017-10-06T05:32:29 *** Son_Goku has quit IRC 2017-10-06T07:19:53 *** asmorodskyi has joined #opensuse-admin 2017-10-06T07:25:06 PROBLEM: Fail2Ban on status1.opensuse.org - CHECK FAIL2BAN ACTIVITY - CRITICAL - 1 detected jails with 2 current banned IP(s) ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=status1.opensuse.org&service=Fail2Ban 2017-10-06T07:36:55 *** kl_eisbaer has joined #opensuse-admin 2017-10-06T07:36:55 *** kl_eisbaer has joined #opensuse-admin 2017-10-06T08:30:13 *** asmorodskyi_ has joined #opensuse-admin 2017-10-06T08:32:08 *** asmorodskyi has quit IRC 2017-10-06T08:37:28 *** asmorodskyi_ has quit IRC 2017-10-06T08:37:47 *** asmorodskyi_ has joined #opensuse-admin 2017-10-06T08:53:12 *** asmorodskyi has joined #opensuse-admin 2017-10-06T08:56:08 *** asmorodskyi_ has quit IRC 2017-10-06T08:56:37 *** matthias_bgg has joined #opensuse-admin 2017-10-06T09:01:49 *** asmorodskyi_ has joined #opensuse-admin 2017-10-06T09:05:10 *** asmorodskyi has quit IRC 2017-10-06T09:08:34 *** asmorodskyi has joined #opensuse-admin 2017-10-06T09:11:11 *** asmorodskyi_ has quit IRC 2017-10-06T10:04:17 *** kollex has joined #opensuse-admin 2017-10-06T10:11:36 *** Son_Goku has joined #opensuse-admin 2017-10-06T10:22:56 *** Son_Goku has quit IRC 2017-10-06T11:10:28 *** nicolasbock has joined #opensuse-admin 2017-10-06T11:10:49 *** nicolasbock_ has joined #opensuse-admin 2017-10-06T11:14:28 *** cboltz has joined #opensuse-admin 2017-10-06T11:14:28 *** cboltz has joined #opensuse-admin 2017-10-06T11:15:26 *** nicolasbock_ has quit IRC 2017-10-06T11:15:26 *** Son_Goku has joined #opensuse-admin 2017-10-06T11:15:35 *** nicolasbock has quit IRC 2017-10-06T11:25:06 PROBLEM: Fail2Ban on status1.opensuse.org - CHECK FAIL2BAN ACTIVITY - CRITICAL - 1 detected jails with 2 current banned IP(s) ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=status1.opensuse.org&service=Fail2Ban 2017-10-06T11:27:26 *** nicolasbock has joined #opensuse-admin 2017-10-06T11:27:37 *** nicolasbock_ has joined #opensuse-admin 2017-10-06T11:29:16 *** nicolasbock has quit IRC 2017-10-06T11:35:29 *** Son_Goku has quit IRC 2017-10-06T11:43:16 *** Son_Goku has joined #opensuse-admin 2017-10-06T12:10:12 cboltz: can you put a header on the wikis with a link to the downtime announcement please? 2017-10-06T12:10:28 also, does anybody know who is an admin in formus that could do the same? 2017-10-06T12:14:32 I'll add the wiki headers tonight 2017-10-06T12:20:40 *** cboltz has quit IRC 2017-10-06T12:22:04 *** nicolasbock has joined #opensuse-admin 2017-10-06T12:31:20 *** solevi|2 has joined #opensuse-admin 2017-10-06T12:38:27 *** kl_eisbaer has quit IRC 2017-10-06T12:39:58 *** nicolasbock has quit IRC 2017-10-06T12:51:01 *** nicolasbock has joined #opensuse-admin 2017-10-06T13:05:21 *** nicolasbock has quit IRC 2017-10-06T13:05:48 *** nicolasbock has joined #opensuse-admin 2017-10-06T13:34:52 *** kollex has left #opensuse-admin 2017-10-06T14:10:58 *** Son_Goku has quit IRC 2017-10-06T14:17:03 *** kl_eisbaer has joined #opensuse-admin 2017-10-06T14:52:01 *** matthias_bgg has quit IRC 2017-10-06T15:10:42 *** Son_Goku has joined #opensuse-admin 2017-10-06T15:25:06 /notify RECOVERY: Fail2Ban on status1.opensuse.org - CHECK FAIL2BAN ACTIVITY - OK - 1 detected jails with 0 current banned IP(s) ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=status1.opensuse.org&service=Fail2Ban 2017-10-06T15:37:48 *** cboltz has joined #opensuse-admin 2017-10-06T15:37:48 *** cboltz has joined #opensuse-admin 2017-10-06T15:51:04 *** mcaj has quit IRC 2017-10-06T16:22:32 kl_eisbaer: I have a question about the external monitoring: 2017-10-06T16:22:51 if we want the results fed into monitor.o.o - what would an external host need to run? 2017-10-06T16:23:29 *** Son_Goku has quit IRC 2017-10-06T16:38:12 *** asmorodskyi has quit IRC 2017-10-06T17:31:50 *** matthias_bgg has joined #opensuse-admin 2017-10-06T17:35:32 *** fvogt has joined #opensuse-admin 2017-10-06T18:02:42 tampakrap: https://languages.opensuse.org/Main_Page - does the message about the planned downtime look good and correct? 2017-10-06T18:03:14 (hmm, BTW - which timezone?) 2017-10-06T18:05:30 ah, your opensuse-announce mail at least answers the timezone - news.o.o doesn't ;-) 2017-10-06T18:06:24 perfect thanks 2017-10-06T18:07:00 ok, I'll salt it to all wikis 2017-10-06T18:08:08 any idea who could add it on the forums? 2017-10-06T18:11:50 https://gitlab.opensuse.org/infra/salt/merge_requests/63 2017-10-06T18:15:08 cboltz: you could also set a variable with that message 2017-10-06T18:15:16 instead of duplicating 2017-10-06T18:15:18 that would be boring ;-) 2017-10-06T18:15:44 actually I considered to introduce a site_message_all pillar (that would be used if no wiki-specific site_message is set) 2017-10-06T18:16:00 correct 2017-10-06T18:16:05 merged btw 2017-10-06T18:16:06 but well, sometimes duplication is the fastest way to do things ;-) 2017-10-06T18:17:01 yep agree 2017-10-06T18:19:46 our saltmaster is lagging - highstate doesn't see the merge yet ;-) 2017-10-06T18:21:22 salt-call event.fire_master update salt/fileserver/gitfs/updat 2017-10-06T18:21:30 +e 2017-10-06T18:21:35 this will give it a kick 2017-10-06T18:23:40 did it help? 2017-10-06T18:25:39 unfortunately not :-( 2017-10-06T18:25:57 okay let's see the master 2017-10-06T18:29:13 I suspect an issue with the new gitlab and the deploy key 2017-10-06T18:29:15 checking logs 2017-10-06T18:30:13 nope it gets the code fine 2017-10-06T18:31:18 3f5750eeddd0973ca880475439d9fc2e65794113 that's on the master 2017-10-06T18:31:40 3f5750eeddd0973ca880475439d9fc2e65794113 that's latest commit 2017-10-06T18:31:43 they are the same 2017-10-06T18:36:09 cboltz: if we want the results fed into monitor.o.o - what would an external host need to run? => depends on what exactly you want to monitor. In the easiest case a simple curl + ssh would be enough. But you can also have a full blown cucumber or Sakuli stack up and running there 2017-10-06T18:37:57 *** plinnell has quit IRC 2017-10-06T18:37:59 I was thinking of something lightweight (and might even have a bored server that could help out), so curl + ssh doesn't sound bad ;-) 2017-10-06T18:39:29 cboltz: the interesting question is how to feed it back to the monitoring 2017-10-06T18:40:09 cboltz: here you can a) offer a http page with the results, that can be queried by the monitoring server at any time (question here: how to check if the page is presenting current information?) 2017-10-06T18:40:22 b) let the monitoring server itself push the check 2017-10-06T18:41:21 in case of b, you can provide for example a restricted ssh access to run a specific command - or a nrpe daemon, or whatever you like that can be used to trigger something and get the results back 2017-10-06T18:45:12 for a), including a timestamp in the page would help ;-) 2017-10-06T18:46:08 that said - I wouldn't be surprised if b) would be easier to implement (well, except for me, my icinga knownledge is terribly small) 2017-10-06T18:46:22 cboltz: fixed, git_pillar was not getting updated (probably something changed in 2017.x.x, need to check), I gave it a kick with "salt-run git_pillar.update" on the master and now I'm going to google it a bit more 2017-10-06T18:47:02 confirmed, it worked now and we have a nice downtime message on all wikis :-) 2017-10-06T18:52:24 tampakrap: https://en.opensuse.org/openSUSE:Forums_team has some names 2017-10-06T18:53:47 thank you! 2017-10-06T19:08:42 *** plusky has quit IRC 2017-10-06T19:19:12 *** plusky has joined #opensuse-admin 2017-10-06T19:39:06 /notify PROBLEM: Fail2Ban on status1.opensuse.org - CHECK FAIL2BAN ACTIVITY - CRITICAL - 1 detected jails with 2 current banned IP(s) ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=status1.opensuse.org&service=Fail2Ban 2017-10-06T19:42:27 kl_eisbaer: I'd guess Fail2Ban means ssh (?) brute force attacks? is this important enough for CRITICAL? 2017-10-06T19:43:17 * cboltz would raise an alert if there is _no_ ssh brute force attemp within 2 hours - that would give less alerts ;-) 2017-10-06T19:50:14 cboltz: fine tuning is a normal job 2017-10-06T19:50:35 cboltz: the fun part is that they brute force port 2222 ;-) 2017-10-06T19:55:33 /notify PROBLEM: PostgreSQL on mirrordb3.infra.opensuse.org - CRITICAL - no connection to mb_opensuse (could not connect to server: No such file or directory ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb3.infra.opensuse.org&service=PostgreSQL 2017-10-06T19:56:02 /notify PROBLEM: HTTP gitlab on mickey.infra.opensuse.org - HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 500 Internal Server Error ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mickey.infra.opensuse.org&service=HTTP%20gitlab 2017-10-06T19:56:03 ^^ sorry, my fault. Needed to add the new IPs in pg_hba.conf 2017-10-06T19:56:34 looks like the monitoring was faster than me 2017-10-06T19:58:32 *** kl_eisbaer has left #opensuse-admin 2017-10-06T20:03:03 /notify PROBLEM: Updates on rpmlint.infra.opensuse.org - Updates CRITICAL : At least one of your Repositories might be out of date. Please run zypper refresh as root to update it. ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=rpmlint.infra.opensuse.org&service=Updates 2017-10-06T20:05:34 /notify RECOVERY: PostgreSQL on mirrordb3.infra.opensuse.org - OK - database mb_opensuse (0.003296 sec.) ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb3.infra.opensuse.org&service=PostgreSQL 2017-10-06T20:06:03 /notify RECOVERY: HTTP gitlab on mickey.infra.opensuse.org - HTTP OK: Status line output matched 302 - 534 bytes in 0.034 second response time ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mickey.infra.opensuse.org&service=HTTP%20gitlab 2017-10-06T20:38:34 *** plinnell has joined #opensuse-admin 2017-10-06T20:38:34 *** plinnell has joined #opensuse-admin 2017-10-06T21:20:58 *** matthias_bgg has quit IRC 2017-10-06T21:21:26 *** matthias_bgg has joined #opensuse-admin 2017-10-06T21:25:08 *** matthias_bgg has quit IRC 2017-10-06T22:27:41 *** fvogt has quit IRC 2017-10-06T22:46:27 *** cboltz has quit IRC 2017-10-06T23:03:29 *** marxin has quit IRC 2017-10-06T23:07:23 *** marxin has joined #opensuse-admin