2019-10-29T00:05:34  *** malcolmlewis <malcolmlewis!~malcolmle@opensuse/member/malcolmlewis> has joined #opensuse-admin
2019-10-29T02:20:25  *** boombatower <boombatower!~boombatow@drupal.org/user/214218/view> has quit IRC (Quit: Konversation terminated!)
2019-10-29T03:46:50  *** okurz_ <okurz_!~okurz@ppp-46-244-221-71.dynamic.mnet-online.de> has joined #opensuse-admin
2019-10-29T03:48:37  *** okurz <okurz!~okurz@ppp-46-244-204-76.dynamic.mnet-online.de> has quit IRC (Ping timeout: 240 seconds)
2019-10-29T03:48:37  *** okurz_ is now known as okurz
2019-10-29T06:55:47  *** maxlin <maxlin!~maxlin@60-251-47-116.HINET-IP.hinet.net> has joined #opensuse-admin
2019-10-29T07:17:44  *** marxin <marxin!marxin@nat/suse/x-dytgezivrlcnclop> has quit IRC (Quit: Leaving)
2019-10-29T07:23:14  *** marxin <marxin!marxin@nat/suse/x-lordqrfstgpuldtz> has joined #opensuse-admin
2019-10-29T07:30:24  *** moozaad <moozaad!~moozaad@opensuse/member/moozaad> has joined #opensuse-admin
2019-10-29T07:48:47  *** petracvv <petracvv!~quassel@fuduntu/support/petracvv> has quit IRC (Ping timeout: 268 seconds)
2019-10-29T07:50:32  *** petracvv <petracvv!~quassel@fuduntu/support/petracvv> has joined #opensuse-admin
2019-10-29T08:17:10  *** jadamek <jadamek!~julienA@176.164.241.21> has joined #opensuse-admin
2019-10-29T09:28:25  *** ldevulder <ldevulder!~ldevulder@176.167.63.5> has quit IRC (Quit: Leaving)
2019-10-29T10:00:39  *** petracvv <petracvv!~quassel@fuduntu/support/petracvv> has quit IRC (Ping timeout: 264 seconds)
2019-10-29T10:02:10  *** petracvv <petracvv!~quassel@fuduntu/support/petracvv> has joined #opensuse-admin
2019-10-29T10:17:49  *** ldevulder <ldevulder!~ldevulder@176.167.109.140> has joined #opensuse-admin
2019-10-29T10:29:02  *** ldevulder <ldevulder!~ldevulder@176.167.109.140> has quit IRC (Quit: Leaving)
2019-10-29T11:08:39  *** ldevulder <ldevulder!~ldevulder@176.167.109.140> has joined #opensuse-admin
2019-10-29T12:31:25  *** boombatower <boombatower!~boombatow@drupal.org/user/214218/view> has joined #opensuse-admin
2019-10-29T13:30:08  *** boombatower <boombatower!~boombatow@drupal.org/user/214218/view> has quit IRC (Remote host closed the connection)
2019-10-29T13:30:30  *** boombatower <boombatower!~boombatow@drupal.org/user/214218/view> has joined #opensuse-admin
2019-10-29T14:02:37  *** boombatower <boombatower!~boombatow@drupal.org/user/214218/view> has quit IRC (Remote host closed the connection)
2019-10-29T14:03:32  *** boombatower <boombatower!~boombatow@drupal.org/user/214218/view> has joined #opensuse-admin
2019-10-29T17:46:17  *** a-865k <a-865k!~ksirc@user-0c90gd5.cable.mindspring.com> has quit IRC (Ping timeout: 240 seconds)
2019-10-29T17:58:14  *** cboltz <cboltz!~cb@opensuse/member/Cboltz> has joined #opensuse-admin
2019-10-29T19:42:11  <cboltz> progress.o.o is down - does someone know what's going on, and how to fix it?
2019-10-29T19:43:36  <cboltz> it looks like there's no redmine process running
2019-10-29T19:44:05  <cboltz> rcredmine restart results in a green "done", but still no redmine process
2019-10-29T20:18:01  *** moozaad <moozaad!~moozaad@opensuse/member/moozaad> has quit IRC (Quit: Konversation terminated!)
2019-10-29T20:38:27  <klein> Hey cboltz
2019-10-29T20:38:38  <klein> did anyone touched this machine for maintenance or anything like that?
2019-10-29T20:38:49  <klein> kbabioch: maybe you?
2019-10-29T20:39:14  <kbabioch> nope, i didn't do anything to this machine :-)
2019-10-29T20:42:24  <klein> cboltz: just to be sure I am in the correct machine, it does have this IP addr 192.168.47.8 ?
2019-10-29T20:45:26  <klein> hummm I dont think so
2019-10-29T20:50:56  <cboltz> no, it's redmine.infra.o.o with IP 192.168.47.29
2019-10-29T20:51:17  <cboltz> .8 is the _future_ progress.o.o
2019-10-29T20:51:55  <klein> I am not on sudoers file :-(
2019-10-29T20:52:00  <klein> can you restart sssd ?
2019-10-29T20:52:23  <cboltz> doesn't help, it's an old SLE 11
2019-10-29T20:52:32  <klein> it is not salted?
2019-10-29T20:52:40  <klein> ow, let me see if I have the password in teampass
2019-10-29T20:52:48  <cboltz> it's in the heroes pass
2019-10-29T20:53:03  <cboltz> and "su" works ;-)
2019-10-29T20:55:11  <klein> last time someone touched this machine was 30/09
2019-10-29T20:55:25  <klein> is there any chance that this redmine is not working since then?
2019-10-29T20:56:20  <cboltz> it serves progress.o.o which worked until (at least) yesterday
2019-10-29T20:57:11  <klein> humm.. ok
2019-10-29T20:58:13  <klein> it has the abuild-online-update package installed
2019-10-29T20:58:36  <klein> *maybe* it did an upgrade last night and because of that it is not working?
2019-10-29T20:59:35  <klein> we have something from today on /var/log/zypper.log
2019-10-29T21:01:10  <klein> do you agree with me that the first thing we should do is remove abuild-online-update from this machine? (Karol and I have removed from mostly all machines already)
2019-10-29T21:26:31  <okurz[m]> progress.o.o definitely worked still today. Many teams are relying on it, I guess it's down no longer than 4h
2019-10-29T21:28:36  <okurz[m]> Would be awesome if someone could look into it
2019-10-29T21:38:38  <okurz[m]> klein: I would not know why you should remove the auto-update. I don't believe in manual updating if this is what you would propose :) anyway, "the first thing to do" is to fix that the redmine instance is down
2019-10-29T21:44:10  <klein> ok, I dont like to have auto-updates on things that can breake like this, I had a look in to the server, and I found nothing wrong with the machine itself
2019-10-29T21:44:20  <klein> and, sadly I have no experience with redmine
2019-10-29T21:46:15  <okurz[m]> Same as with all other server systems, take a look in logfiles :) sorry, can't take a look myself right now
2019-10-29T21:47:38  *** cboltz <cboltz!~cb@opensuse/member/Cboltz> has quit IRC (Ping timeout: 240 seconds)
2019-10-29T21:48:56  *** cboltz_ <cboltz_!~cb@opensuse/member/Cboltz> has joined #opensuse-admin
2019-10-29T21:51:01  <klein> we have no redmine or unicorn process runing, but there are files in /srv/www/vhosts/redmine/tmp/pids
2019-10-29T21:52:33  <cboltz_> do these files change if you (try to) restart redmine?
2019-10-29T21:55:47  <cboltz_> hmm, given the timestamps, at least unicorn.pid gets updated (it's just 4 minutes old)
2019-10-29T21:56:47  <klein> I tried to restart the service
2019-10-29T21:57:11  <klein> and yes, it creates a new pid file, but the process probably dies after start
2019-10-29T21:57:29  <cboltz_> yes, looks so
2019-10-29T21:57:41  <cboltz_> any idea if it writes a useful log anywhere?
2019-10-29T21:58:11  <cboltz_> in /var/log/redmine/unicorn.log, it says "check stderr"
2019-10-29T21:58:19  <cboltz_> but I have no idea where stderr ends up
2019-10-29T21:59:47  <klein> yeah, I saw that, and tried to run what init.d runs: startproc -u redmine -g redmine -l /var/log/redmine/unicorn.log /usr/bin/unicorn_rails -c /srv/www/vhosts/redmine/config/unicorn.rb -E production -D
2019-10-29T21:59:59  <klein> but got nothing
2019-10-29T22:00:34  <klein> let me try running "/usr/bin/unicorn_rails -c /srv/www/vhosts/redmine/config/unicorn.rb -E production -D" as user "redmine" to see what happens
2019-10-29T22:00:52  <klein> redmine@redmine:~> /usr/bin/unicorn_rails -c /srv/www/vhosts/redmine/config/unicorn.rb -E production -D
2019-10-29T22:00:55  <klein> master failed to start, check stderr log for details
2019-10-29T22:01:04  <klein> great... very usefull :-(
2019-10-29T22:01:13  <cboltz_> indeed :-/
2019-10-29T22:01:54  <klein> hah... without -D it shows something
2019-10-29T22:02:25  <klein> big ruby stack trace, do we have any paste on opensuse ?
2019-10-29T22:02:33  <klein> *pastebin
2019-10-29T22:02:34  <cboltz_> yes, paste.o.o
2019-10-29T22:03:20  <klein> https://paste.opensuse.org/b3f5b14d
2019-10-29T22:03:25  <cboltz_> as a sidenote - I can recommend a printout of the opensuse.org zone as bed lecture ;-)
2019-10-29T22:04:07  <klein> good idea
2019-10-29T22:04:21  <klein> anyway, I can see mysql connection errors, maybe that is our problem?
2019-10-29T22:05:12  <cboltz_> yes, looks like a ssl problem with it
2019-10-29T22:05:36  <klein> expired certificate on MySQL server maybe?
2019-10-29T22:05:41  <cboltz_> do you know if there was an upgrade on the mysql server(s)?
2019-10-29T22:06:59  <klein> no idea, need to search wich one redmine is using
2019-10-29T22:07:49  <cboltz_> mysql.infra.opensuse.org  (if I get the config file right)
2019-10-29T22:08:08  <cboltz_> and there's also a mention of ssl certificates
2019-10-29T22:08:36  <klein> yeah, I was reading the same config file :-)
2019-10-29T22:09:04  <klein> maybe we can change the config file to use mysql without SSL and see what happens?
2019-10-29T22:09:52  <klein> mysql runs on anna?
2019-10-29T22:10:13  <cboltz_> anna is the proxy, and forwards it to somewhere[tm]
2019-10-29T22:10:25  <cboltz_> openssl x509 -in /dev/stdin -text -noout < /etc/redmine/ssl/redmine.infra.opensuse.org.crt
2019-10-29T22:10:34  <cboltz_>             Not After : Nov  1 12:50:38 2019 GMT
2019-10-29T22:11:05  <cboltz_> so it's close to the expire date, but not there yet
2019-10-29T22:11:30  <klein> so if there is any cert prolem, it is on anna, because it is where redmine is connecting, right?
2019-10-29T22:13:24  <cboltz_> AFAIK anna does the forwarding with haproxy - I'm not sure if it does SSL termination or blindly forwards whatever it gets
2019-10-29T22:18:59  *** jadamek <jadamek!~julienA@176.164.241.21> has quit IRC (Quit: Leaving)
2019-10-29T22:19:00  <cboltz_> I don't see anything certificate-related in the mysql section of haproxy.cfg, therefore it probably just forwards whatever comes in
2019-10-29T22:21:41  <cboltz_> BTW: you mentioned zypper.log has something from today, but /v/l/zypp/history has the last change 2019-10-08
2019-10-29T22:34:32  <cboltz_> I just tried with the "mysql" command (which needs an interesting[tm] command line when using client certs)
2019-10-29T22:34:44  <cboltz_> it says   ERROR 2026 (HY000): SSL connection error: error:00000001:lib(0):func(0):reason(1)
2019-10-29T22:35:50  <cboltz_> BTW: use   screen -x   if you want to see what I tried ;-)
2019-10-29T22:38:27  <cboltz_> "at least" we now know that mysql is the problem
2019-10-29T22:40:05  <cboltz_> searching for the error message points to https://bugs.mysql.com/bug.php?id=75311
2019-10-29T22:40:52  <cboltz_> which lists cipher suites mismatch as a possible reason, and a revoked certificate as another option
2019-10-29T22:41:48  <klein> sorry, I lost my connection
2019-10-29T22:42:15  <klein> ok, so, can I talk with karol in the morning and maybe solve this? He might know more than us about it
2019-10-29T22:42:43  <cboltz_> mysql.infra.o.o gets forwarded to 172.16.42.4 (tarzan?) which AFAIK is a mysql cluster running in the SUSE network
2019-10-29T22:42:52  <cboltz_> (which also means I don't have access to it)
2019-10-29T22:43:35  <cboltz_> I'd guess someone updated it, and now it's incompatible with the old SLE11 openssl we have on the redmine server
2019-10-29T22:43:46  <klein> yeah... thats where we need Karol :-)... I may have access but, in 1,5 month in SUSE I haven't touched everything (yet)
2019-10-29T22:44:48  <cboltz_> I'd be surprised if you touched everything in the next 6 months ;-)
2019-10-29T22:45:39  <cboltz_> so talking to kbabioch sounds like a good idea
2019-10-29T22:46:55  <klein> and yes, we had an upgrade on the mysql/mariadb/galera/whatever cluster last week
2019-10-29T22:47:43  <cboltz_> that update might explain it, but then - why did progress.o.o survive until today?
2019-10-29T22:47:59  <klein> have no freakin idea :-)
2019-10-29T22:48:12  <klein> and, about the 6months, is that a challenge? hahaha
2019-10-29T22:48:17  <cboltz_> do you know if someone worked on the mysql config yesterday or today?
2019-10-29T22:48:36  <cboltz_> maybe someone thought "oh, we shouldn't allow SSLv3 anymore"?
2019-10-29T22:48:56  <klein> I don't think anyone has touched it, but I can ask tomorrow for everyone in the EngInfra Team
2019-10-29T22:50:27  <cboltz_> please tell me what you find out ;-)
2019-10-29T22:52:12  <klein> sure... and for today, thats all folks ;-)
2019-10-29T22:52:41  <cboltz_> good night ;-)
2019-10-29T23:24:08  *** cboltz_ <cboltz_!~cb@opensuse/member/Cboltz> has quit IRC ()