2018-01-12T00:03:40 *** cboltz has quit IRC 2018-01-12T00:14:57 *** BManojlovic has quit IRC 2018-01-12T00:19:57 *** nicolasbock has joined #opensuse-admin 2018-01-12T01:36:03 *** plinnell has quit IRC 2018-01-12T01:39:28 *** plinnell has joined #opensuse-admin 2018-01-12T03:03:25 *** Son_Goku has quit IRC 2018-01-12T03:31:49 *** okurz has quit IRC 2018-01-12T03:33:03 *** okurz has joined #opensuse-admin 2018-01-12T03:34:00 *** Son_Goku has joined #opensuse-admin 2018-01-12T03:57:53 *** plinnell has quit IRC 2018-01-12T04:04:26 *** plinnell has joined #opensuse-admin 2018-01-12T04:04:26 *** plinnell has joined #opensuse-admin 2018-01-12T04:16:54 *** Son_Goku has quit IRC 2018-01-12T04:21:17 *** nicolasbock has quit IRC 2018-01-12T04:28:11 PROBLEM: Hosts syslog on monitor.infra.opensuse.org - CRITICAL: Found files older than 600 minutes /var/log/opensuse/hosts//wikisearch.log was last modified on Wed Jan 3 18:24:51 2018 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=monitor.infra.opensuse.org&service=Hosts%20syslog 2018-01-12T05:32:28 *** plinnell has quit IRC 2018-01-12T05:51:32 *** plinnell has joined #opensuse-admin 2018-01-12T05:51:32 *** plinnell has joined #opensuse-admin 2018-01-12T07:29:29 *** asmorodskyi has joined #opensuse-admin 2018-01-12T07:53:05 *** asmorodskyi has quit IRC 2018-01-12T07:53:25 *** asmorodskyi has joined #opensuse-admin 2018-01-12T08:21:11 *** deneb_alpha has joined #opensuse-admin 2018-01-12T08:21:11 *** deneb_alpha has joined #opensuse-admin 2018-01-12T08:39:57 *** matthias_bgg has joined #opensuse-admin 2018-01-12T09:04:37 *** deneb_alpha has quit IRC 2018-01-12T09:12:32 *** deneb_alpha has joined #opensuse-admin 2018-01-12T09:13:48 *** Son_Goku has joined #opensuse-admin 2018-01-12T09:18:04 *** sven15 has joined #opensuse-admin 2018-01-12T09:19:26 *** ancorgs has quit IRC 2018-01-12T09:38:10 *** tigerfoot has joined #opensuse-admin 2018-01-12T10:09:09 *** ancorgs has joined #opensuse-admin 2018-01-12T10:29:47 *** BManojlovic has joined #opensuse-admin 2018-01-12T10:29:47 *** BManojlovic has joined #opensuse-admin 2018-01-12T10:57:20 *** tigerfoot has quit IRC 2018-01-12T11:08:44 *** BManojlovic has quit IRC 2018-01-12T11:12:27 *** BManojlovic has joined #opensuse-admin 2018-01-12T11:14:16 *** tigerfoot has joined #opensuse-admin 2018-01-12T11:20:57 *** cboltz has joined #opensuse-admin 2018-01-12T11:20:57 *** cboltz has joined #opensuse-admin 2018-01-12T11:36:36 *** nicolasbock has joined #opensuse-admin 2018-01-12T11:39:04 *** cthugha has joined #opensuse-admin 2018-01-12T11:42:17 *** ldevulder has quit IRC 2018-01-12T12:08:08 *** deneb_alpha has quit IRC 2018-01-12T12:45:00 *** cboltz has quit IRC 2018-01-12T12:49:55 *** matthias_bgg has quit IRC 2018-01-12T12:50:32 *** matthias_bgg has joined #opensuse-admin 2018-01-12T12:51:28 DimStar: can I update osc-collab to sp4? 2018-01-12T12:53:39 tampakrap: should be possible, yes 2018-01-12T12:54:01 do you want me to backup up something before? 2018-01-12T12:54:56 tampakrap: nah -the database on it can be recreated, as it's just mirroring stuff from various servers; not that I expect it to break, but if it does, we survive 2018-01-12T13:05:44 *** BManojlovic has quit IRC 2018-01-12T13:05:56 *** BManojlovic has joined #opensuse-admin 2018-01-12T13:05:56 *** BManojlovic has joined #opensuse-admin 2018-01-12T13:07:39 okay 2018-01-12T13:40:52 *** cthugha is now known as ldevulder 2018-01-12T13:42:14 *** Son_Goku has quit IRC 2018-01-12T14:27:12 *** Son_Goku has joined #opensuse-admin 2018-01-12T15:19:01 *** asmorodskyi_ has joined #opensuse-admin 2018-01-12T15:21:09 *** mcaj has quit IRC 2018-01-12T15:21:41 *** asmorodskyi has quit IRC 2018-01-12T16:06:56 *** Son_Goku has quit IRC 2018-01-12T16:37:27 *** mstrigl has quit IRC 2018-01-12T16:44:14 *** cboltz has joined #opensuse-admin 2018-01-12T16:44:14 *** cboltz has joined #opensuse-admin 2018-01-12T16:50:16 *** matthias_bgg has quit IRC 2018-01-12T16:59:22 *** asmorodskyi_ has quit IRC 2018-01-12T17:00:42 tampakrap: a "git pull" for our salt code just asked for gitlab@gitlab.infra.opensuse.org's password: 2018-01-12T17:00:55 that's new behaviour - did you change something? 2018-01-12T17:01:23 I updated the machine 2018-01-12T17:01:26 all machines actually 2018-01-12T17:02:40 did you also run highstate on it? 2018-01-12T17:03:38 https://paste.opensuse.org/80242089 2018-01-12T17:04:03 yes, I updated the formulas, ran highstate and zypper up on all machines 2018-01-12T17:04:25 and apparently the openssh-formula dropped default values 2018-01-12T17:06:14 ssh login as cboltz works, so the sshd config should in general be ok 2018-01-12T17:06:46 my guess would be that the highstate added the LDAP logins, and that this causes problems with the "gitlab" user 2018-01-12T17:08:49 I added back now manually all the options that salt removed 2018-01-12T17:08:51 and it works 2018-01-12T17:09:20 indeed :-) - thanks! 2018-01-12T17:09:43 the problem remains though, I'll file ticket 2018-01-12T17:10:16 *** sven15 has quit IRC 2018-01-12T17:11:40 https://progress.opensuse.org/issues/30277 2018-01-12T17:12:34 PROBLEM: memcached data stored on riesling.infra.opensuse.org - MEMCACHE CRITICAL: bytes is 211853590 200000000 - memcached 1.4.33 on 127.0.0.1:11211, up 5 days 4 hours ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=riesling.infra.opensuse.org&service=memcached%20data%20stored 2018-01-12T17:19:20 tampakrap: the gitlab web interface goes 500 :-( 2018-01-12T17:20:49 wtf 2018-01-12T17:20:56 it was working like 3 mins ago 2018-01-12T17:22:52 there's an improvement - the error code increased by 2 ;-) 2018-01-12T17:24:02 postgresql is fine? 2018-01-12T17:24:22 apparently not 2018-01-12T17:25:10 *** ancorgs has quit IRC 2018-01-12T17:27:23 so postgres is not working since I rebooted scar 2018-01-12T17:27:28 I'm texting darix 2018-01-12T17:27:43 *** sven15 has joined #opensuse-admin 2018-01-12T17:27:47 PROBLEM: PostgreSQL on mirrordb3.infra.opensuse.org - CRITICAL - no connection to mb_opensuse (FATAL: the database system is shutting down ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb3.infra.opensuse.org&service=PostgreSQL 2018-01-12T17:28:07 *** sven15 has quit IRC 2018-01-12T17:28:25 another HA database that isn't HA? 2018-01-12T17:28:41 * cboltz votes to downgrade to paper and pen 2018-01-12T17:47:47 RECOVERY: PostgreSQL on mirrordb3.infra.opensuse.org - OK - database mb_opensuse (0.004357 sec.) ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb3.infra.opensuse.org&service=PostgreSQL 2018-01-12T17:48:47 cboltz: the auto-update script broke it most probably 2018-01-12T17:50:26 I'll leave checking that to you ;-) 2018-01-12T18:02:31 tampakrap: gitlab is down again since a few seconds 2018-01-12T18:03:05 *** petracvv has quit IRC 2018-01-12T18:03:39 *** sven15 has joined #opensuse-admin 2018-01-12T18:04:20 *** petracvv has joined #opensuse-admin 2018-01-12T18:05:49 cboltz: yes took the opportunity that darix is on the other side of my phone to reboot difficult machines :) 2018-01-12T18:05:56 it should be back now 2018-01-12T18:06:14 ;-) 2018-01-12T18:06:21 yes, it works again 2018-01-12T18:07:13 I'll do status.o.o and the provo machines, and nothing else for today 2018-01-12T18:07:39 how do I ssh to status.o.o? 2018-01-12T18:08:25 hmm, what about "you ask me to add your ssh keys"? ;-) 2018-01-12T18:08:57 it's not even giving me a prompt 2018-01-12T18:09:15 ah now it does 2018-01-12T18:09:22 okay so add my key please 2018-01-12T18:10:53 done, try again 2018-01-12T18:10:58 which port it is? 22 or 2222? 2018-01-12T18:11:51 it's some easy math: use 22 once for status1, and 22 twice for status2 2018-01-12T18:12:18 ssh root@status1.opensuse.org 2018-01-12T18:12:20 ssh: connect to host status1.opensuse.org port 22: Connection refused 2018-01-12T18:13:20 strange, works here 2018-01-12T18:13:47 it was working here as well a few mins ago :P 2018-01-12T18:13:55 does it have fail2ban? 2018-01-12T18:14:16 yes 2018-01-12T18:14:29 213.151.95.130 add it to whitelist please 2018-01-12T18:14:47 it's the prague suse office IP 2018-01-12T18:16:41 try again 2018-01-12T18:17:01 still connection refused :P 2018-01-12T18:18:19 ah, there's a jail.conf and a jail.local ;-) 2018-01-12T18:18:53 does it work now? 2018-01-12T18:18:57 *** petracvv has quit IRC 2018-01-12T18:18:58 yep thanks! 2018-01-12T18:21:16 did you run highstate on the machine that runs the gitlab CI jobs? 2018-01-12T18:21:25 yes 2018-01-12T18:21:44 what's broken? 2018-01-12T18:22:07 that might explain why deploy_job fails - sudo asks for a password, and jenkins doesn't type it ;-) 2018-01-12T18:24:14 indeed, it says "update" instead of the deploy password 2018-01-12T18:26:13 *** petracvv has joined #opensuse-admin 2018-01-12T18:27:15 https://gitlab.infra.opensuse.org/infra/salt/blob/production/pillar/role/worker_gitlab.sls yep it's wrong 2018-01-12T18:34:08 *** Son_Goku has joined #opensuse-admin 2018-01-12T18:49:44 *** mstrigl has joined #opensuse-admin 2018-01-12T18:58:57 *** fvogt has joined #opensuse-admin 2018-01-12T19:12:37 *** matthias_bgg has joined #opensuse-admin 2018-01-12T19:13:01 *** Son_Goku has quit IRC 2018-01-12T20:39:48 PROBLEM: HAProxy on anna.infra.opensuse.org - HAPROXY CRITICAL - Active service sarabi is DOWN on elections proxy ! ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=anna.infra.opensuse.org&service=HAProxy 2018-01-12T20:39:49 PROBLEM: HAProxy on elsa.infra.opensuse.org - HAPROXY CRITICAL - Active service sarabi is DOWN on elections proxy ! ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=elsa.infra.opensuse.org&service=HAProxy 2018-01-12T20:39:50 PROBLEM: HAProxy on mufasa.infra.opensuse.org - HAPROXY CRITICAL - Active service riesling is DOWN on riesling proxy ! Active service narwal4 is DOWN on static proxy ! ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mufasa.infra.opensuse.org&service=HAProxy 2018-01-12T20:53:43 *** matthias_bgg has quit IRC 2018-01-12T22:33:45 *** fvogt has quit IRC 2018-01-12T23:03:59 *** fvogt has joined #opensuse-admin