2018-01-03T00:24:53  *** cboltz has quit IRC
2018-01-03T01:06:19  *** katnip has quit IRC
2018-01-03T01:06:19  *** katnip has joined #opensuse-admin
2018-01-03T01:29:11  *** fvogt has quit IRC
2018-01-03T02:07:29  *** plinnell has quit IRC
2018-01-03T02:17:30  *** plinnell has joined #opensuse-admin
2018-01-03T03:01:30  *** Son_Goku has quit IRC
2018-01-03T03:40:27  *** okurz has quit IRC
2018-01-03T03:42:50  *** okurz has joined #opensuse-admin
2018-01-03T03:44:41  *** nicolasbock has quit IRC
2018-01-03T04:03:27  *** Son_Goku has joined #opensuse-admin
2018-01-03T05:16:42  *** tigerfoot has quit IRC
2018-01-03T05:57:54  *** plinnell has quit IRC
2018-01-03T06:00:56  *** plinnell has joined #opensuse-admin
2018-01-03T08:15:33  *** kollex has joined #opensuse-admin
2018-01-03T08:22:57  *** ldevulder has quit IRC
2018-01-03T08:33:44  *** ldevulder has joined #opensuse-admin
2018-01-03T08:50:10  *** sven15 has joined #opensuse-admin
2018-01-03T08:54:29  *** asmorodskyi has joined #opensuse-admin
2018-01-03T09:00:25  *** mcaj has joined #opensuse-admin
2018-01-03T10:00:10  <asmorodskyi> tampakrap: Hi . So I am able to reach monitor.infra.o.o but I need sudo there or supybot user password ( I mean linux user )
2018-01-03T10:12:05  <tampakrap> asmorodskyi: try sudo su - supybot
2018-01-03T10:12:10  <tampakrap> and use your freeipa password
2018-01-03T10:13:16  <asmorodskyi> tampakrap: ah this part was missing for me . so I should use my freeipa password
2018-01-03T10:13:42  <asmorodskyi> as I connected via ssh key I was not sure which password I have there ...
2018-01-03T10:16:46  <tampakrap> asmorodskyi: I just set up the sudo rule, so it wouldn't work either way :)
2018-01-03T10:16:54  <tampakrap> let me know if it works please so I can submit it also to salt
2018-01-03T10:20:17  <asmorodskyi> tampakrap: I am in
2018-01-03T10:22:32  *** heroes-bot has joined #opensuse-admin
2018-01-03T10:23:37  *** heroes-bot has joined #opensuse-admin
2018-01-03T10:24:07  <tampakrap> good
2018-01-03T10:25:02  <asmorodskyi> tampakrap: funny to add user I need to shutdown supybot , but I can't :) looks like some demon starting new instance everytime I am killing current one :\
2018-01-03T10:27:54  <tampakrap> https://gitlab.infra.opensuse.org/infra/salt/merge_requests/135
2018-01-03T10:30:34  <asmorodskyi> tampakrap: any idea how to shutdown bot for a while ?
2018-01-03T10:31:55  <tampakrap> you need sudo access to the service commands I suppose
2018-01-03T10:34:17  <tampakrap> asmorodskyi: sudo rcsupybot {status,restart,stop}
2018-01-03T10:37:13  <asmorodskyi> tampakrap: Sorry, user asmorodskyi is not allowed to execute '/usr/sbin/rcsupybot status' as root on monitor
2018-01-03T10:37:49  <tampakrap> try again, it was a typo
2018-01-03T10:38:19  <asmorodskyi> tampakrap: now it works
2018-01-03T10:40:02  <asmorodskyi> tampakrap: but  sudo rcsupybot stop does not have any effect
2018-01-03T10:41:01  <tampakrap> any error message?
2018-01-03T10:42:04  <asmorodskyi> tampakrap: output is "redirecting to systemctl stop supybot.service"
2018-01-03T10:42:42  <asmorodskyi> but I have no permission to execute "sudo systemctl stop supybot.service" so might be permission issue still
2018-01-03T10:42:44  <tampakrap> that's it?
2018-01-03T10:42:53  <asmorodskyi> yap
2018-01-03T10:42:57  <tampakrap> okay let's try
2018-01-03T10:43:15  <asmorodskyi> sudo rcsupybot stop
2018-01-03T10:43:17  <tampakrap> now you do
2018-01-03T10:43:17  <asmorodskyi> redirecting to systemctl stop supybot.service
2018-01-03T10:43:19  <asmorodskyi> asmorodskyi@monitor:/home/asmorodskyi> sudo systemctl stop supybot.service
2018-01-03T10:43:21  <asmorodskyi> Sorry, user asmorodskyi is not allowed to execute '/usr/bin/systemctl stop supybot.service' as root on monitor.
2018-01-03T10:44:46  <asmorodskyi> sudo rcsupybot status
2018-01-03T10:44:48  <asmorodskyi> Checking for service SUPYBOT                                                                                                                                                       running
2018-01-03T10:44:50  <asmorodskyi> ● supybot.service - LSB: Supybot daemon
2018-01-03T10:44:52  <asmorodskyi>    Loaded: loaded (/etc/init.d/supybot; bad; vendor preset: disabled)
2018-01-03T10:44:54  <asmorodskyi>    Active: inactive (dead)
2018-01-03T10:44:59  <asmorodskyi> but process is still running in a fact
2018-01-03T10:45:25  <asmorodskyi> I suspect that bot was started not in a usual way ...
2018-01-03T10:46:13  <tampakrap> maybe
2018-01-03T10:46:18  <tampakrap> try to kill the running process
2018-01-03T10:46:30  <asmorodskyi> I did it
2018-01-03T10:46:47  <asmorodskyi> but it get back in same second
2018-01-03T10:46:52  <asmorodskyi> with another PID
2018-01-03T10:47:08  <asmorodskyi> so something keep it alive
2018-01-03T10:49:47  *** heroes-bot has joined #opensuse-admin
2018-01-03T10:49:50  <asmorodskyi>  ps aux | grep supybot
2018-01-03T10:49:52  <asmorodskyi> supybot  19556  0.7  0.9 233212 55004 ?        Ssl  10:47   0:00 /usr/bin/python /usr/bin/supybot /home/supybot/supybot/heroes-bot.conf
2018-01-03T10:49:54  <asmorodskyi> root     20501  0.0  0.1 129348  7732 pts/1    S    10:48   0:00 sudo su - supybot
2018-01-03T10:49:56  <asmorodskyi> root     20502  0.0  0.0 106920  5340 pts/1    S    10:48   0:00 su - supybot
2018-01-03T10:49:58  <asmorodskyi> supybot  20503  0.3  0.1  24512  6564 pts/1    S    10:48   0:00 -bash
2018-01-03T10:50:00  <asmorodskyi> supybot  20654  0.0  0.0  55472  4056 pts/1    R+   10:49   0:00 ps aux
2018-01-03T10:50:02  <asmorodskyi> supybot  20655  0.0  0.0  10540  1632 pts/1    S+   10:49   0:00 grep --color=auto supybot
2018-01-03T10:50:04  <asmorodskyi> supybot@monitor:~> kill -9 19556
2018-01-03T10:50:06  <asmorodskyi> supybot@monitor:~> ps aux | grep supybot
2018-01-03T10:50:08  <asmorodskyi> root     20501  0.0  0.1 129348  7732 pts/1    S    10:48   0:00 sudo su - supybot
2018-01-03T10:50:10  <asmorodskyi> root     20502  0.0  0.0 106920  5340 pts/1    S    10:48   0:00 su - supybot
2018-01-03T10:50:12  <asmorodskyi> supybot  20503  0.1  0.1  24512  6572 pts/1    S    10:48   0:00 -bash
2018-01-03T10:50:14  <asmorodskyi> supybot  20971 34.7  0.9 232952 54716 ?        Ssl  10:49   0:00 /usr/bin/python /usr/bin/supybot /home/supybot/supybot/heroes-bot.conf
2018-01-03T10:50:16  <asmorodskyi> supybot  20994  0.0  0.0  55472  4168 pts/1    R+   10:49   0:00 ps aux
2018-01-03T10:50:18  <asmorodskyi> supybot  20995  0.0  0.0  10540  1628 pts/1    S+   10:49   0:00 grep --color=auto supybot
2018-01-03T10:54:32  <tampakrap> yep I see it as well
2018-01-03T10:55:47  *** heroes-bot has joined #opensuse-admin
2018-01-03T10:57:28  <tampakrap> systemctl says it is off now
2018-01-03T10:57:43  <tampakrap> although the process is still running
2018-01-03T11:01:09  <asmorodskyi> so we have a real hero bot , it servives everything including "kill -9" :)
2018-01-03T11:11:01  <tampakrap> asmorodskyi: I could reboot the machine with the supybot service not enabled and see what happens
2018-01-03T11:11:04  <tampakrap> sounds fine?
2018-01-03T11:19:15  *** cthugha has joined #opensuse-admin
2018-01-03T11:20:05  *** ldevulder has quit IRC
2018-01-03T11:29:57  *** cboltz has joined #opensuse-admin
2018-01-03T11:29:58  *** cboltz has joined #opensuse-admin
2018-01-03T11:36:13  *** nicolasbock has joined #opensuse-admin
2018-01-03T11:40:35  <asmorodskyi> sorry I was at lunch
2018-01-03T11:40:48  <asmorodskyi> yes , sounds fine
2018-01-03T11:42:32  *** heroes-bot has joined #opensuse-admin
2018-01-03T11:43:14  <tampakrap> asmorodskyi: rebooted, the service is still there
2018-01-03T11:44:59  <tampakrap> I mean the process
2018-01-03T11:47:07  <asmorodskyi> remove execution flag from /usr/bin/supybot ?
2018-01-03T11:47:14  <asmorodskyi> and kill process again
2018-01-03T11:48:08  <asmorodskyi> tampakrap: can I try ?
2018-01-03T11:49:52  <tampakrap> sure
2018-01-03T11:51:07  <asmorodskyi> tampakrap: file owned by root , can you please try ?
2018-01-03T11:52:15  *** cthugha has quit IRC
2018-01-03T14:37:23  *** heroes-bot has joined #opensuse-admin
2018-01-03T14:37:57  <asmorodskyi> tampakrap: ah , restart solve issue :)
2018-01-03T14:42:24  <asmorodskyi> tampakrap: and finally bot recognizes me !!! yahoo !
2018-01-03T14:45:30  *** asmorodskyi_ has joined #opensuse-admin
2018-01-03T14:46:48  <tampakrap> so it really needed to run without the --daemon
2018-01-03T14:46:56  *** asmorodskyi_ has joined #opensuse-admin
2018-01-03T15:15:51  <tampakrap> cboltz: so I'll reject all the fate tickets then
2018-01-03T15:23:18  <cboltz> I can do that - better spend the time to get me started with breaking haproxy ;-)
2018-01-03T15:24:08  <tampakrap> well the config is huge but it is quite straightforward
2018-01-03T15:24:34  <tampakrap> global and defaults sections are clear
2018-01-03T15:24:54  <tampakrap> the listen * sections are various interfaces that we listen to, to which ports etc
2018-01-03T15:25:09  <tampakrap> and it also specifies where is the ssl cert for the ssl ports
2018-01-03T15:25:34  <cboltz> let's start with a boring question - haporxy hosts are anna and elsa AFAIK
2018-01-03T15:25:38  <tampakrap> yes
2018-01-03T15:25:41  <cboltz> on which one do I need to change the config?
2018-01-03T15:25:57  <tampakrap> the configs are synced via csync2
2018-01-03T15:26:06  <tampakrap> they are not in salt and they should be :)
2018-01-03T15:26:15  <tampakrap> so whichever you change, it will get synced to the other
2018-01-03T15:26:25  <cboltz> ok
2018-01-03T15:26:32  <tampakrap> but to see the changes, you need to edit the one that has the vrrp
2018-01-03T15:26:48  <tampakrap> you can see the VRRPs on the keepalived config
2018-01-03T15:27:06  <tampakrap> and you can see which machine has the ips right now with the usual "ip a" command
2018-01-03T15:27:35  <tampakrap> if you check the keepalive configs, you'll see that anna is master and elsa is slave, so chances are that you need to touch anna
2018-01-03T15:27:37  <tampakrap> clear so far?
2018-01-03T15:27:41  <cboltz> did I already mention that I know nothing about keepalived? ;-)
2018-01-03T15:27:50  <cboltz> so - what exactly do I need to check?
2018-01-03T15:28:15  <tampakrap> so keepalived is doing a ping let's say between two machines
2018-01-03T15:28:32  <tampakrap> and it says that the one with the biggest priority will take the listed virtual IPs
2018-01-03T15:28:50  <tampakrap> so anna is master and has the virtual IPs
2018-01-03T15:29:02  <tampakrap> if anna is down, then elsa will take over the virtual IPs
2018-01-03T15:29:05  <tampakrap> clear?
2018-01-03T15:29:48  <cboltz> yes
2018-01-03T15:30:03  <tampakrap> on the keepalive config you can see who is the master, and which are the virtual IPs
2018-01-03T15:30:20  <cboltz> ah, found it :-)
2018-01-03T15:30:30  <tampakrap> this one is also in salt
2018-01-03T15:31:06  <cboltz> yes, I notied the "managed by salt" headers in keepalived.conf
2018-01-03T15:31:32  <tampakrap> okay go to the haproxy config now
2018-01-03T15:31:49  <cboltz> ok, I'm in /etc/haproxy on anna
2018-01-03T15:32:16  <tampakrap> check the sections that I explained already
2018-01-03T15:32:18  <cboltz> looks like there's a .git, and a quite big "git diff"
2018-01-03T15:32:56  <tampakrap> this git was probably done by me when we were setting up the provo haproxy
2018-01-03T15:32:58  <tampakrap> I'll remove it
2018-01-03T15:33:19  <tampakrap> ah no it existed from before
2018-01-03T15:33:49  <tampakrap> anyway, I'm removing it
2018-01-03T15:33:50  <tampakrap> it's dangerous
2018-01-03T15:34:00  <cboltz> I'd say even a cron.daily that does a   git commit -m 'daily haproxy config'   could be useful if someone breaks something ;-)
2018-01-03T15:34:11  <cboltz> why do you think it's dangerous?
2018-01-03T15:34:23  <tampakrap> i'd prefer to move them to salt instead
2018-01-03T15:34:48  <tampakrap> because I tend to do "git checkout ." in wrong windows :)
2018-01-03T15:35:04  <cboltz> ;-)
2018-01-03T15:35:22  <tampakrap> so, check the haproxy config
2018-01-03T15:36:30  <cboltz> oh, "only" ~900 lines? nice[tm]
2018-01-03T15:36:46  <tampakrap> don't worry, it's straightforward
2018-01-03T15:38:21  <cboltz> the first lesson is probably "skip the first 150 lines, the interesting stuff starts at "frontend http-ext-in"?
2018-01-03T15:38:30  <cboltz> (at least if I don't want to add new IPs)
2018-01-03T15:38:52  <tampakrap> so the listen * sections I explained before what they are
2018-01-03T15:39:00  <tampakrap> then we have frontend and backend
2018-01-03T15:39:25  <tampakrap> the frontend is a bind to an IP address and port
2018-01-03T15:39:33  <tampakrap> separate frontend for each interface
2018-01-03T15:40:17  <tampakrap> they take the http request and they try to match them to an acl rule
2018-01-03T15:40:19  <tampakrap> acl is_static       hdr(host) -i static.opensuse.org
2018-01-03T15:40:54  <tampakrap> this means that http requests coming to the ext-in interface (incoming connections to the external interface)
2018-01-03T15:41:02  <tampakrap> are getting the is_static acl rule
2018-01-03T15:41:04  <tampakrap> understood?
2018-01-03T15:41:19  <cboltz> I think so ;-)
2018-01-03T15:41:56  <tampakrap> then it assigns each acl rule to a backend
2018-01-03T15:41:58  <tampakrap> use_backend static           if is_static
2018-01-03T15:42:22  <tampakrap> and then you go to the backend section, and you see that the backend consists of one or more servers that take the request
2018-01-03T15:43:10  <tampakrap> so it is: interface -> IP -> domain -> acl -> select backend -> select server
2018-01-03T15:43:49  <tampakrap> the backend could have more than one server, because in case one is not online, it goes to the other
2018-01-03T15:43:55  <cboltz> looks like enough levels of indirection ;-)
2018-01-03T15:44:29  <cboltz> (there are probably reasons for that, but it still makes things slightly more interesting)
2018-01-03T15:44:48  <tampakrap> it can also do http requests to all the backend members every X seconds to a specific URL to see if they reply or not
2018-01-03T15:44:55  <tampakrap> as an example: option httpchk HEAD /robots.txt HTTP/1.1\r\nHost:\ conference.opensuse.org
2018-01-03T15:45:37  <cboltz> nice, makes sense to avoid broken servers
2018-01-03T15:45:57  <cboltz> hmm, what about
2018-01-03T15:46:01  <cboltz>     redirect code 301 prefix   https://cz.opensuse.org if is_dewiki
2018-01-03T15:46:13  <cboltz> "if is_dewiki" doesn't look correct...
2018-01-03T15:46:50  <cboltz> and, more interesting, cz.opensuse.org gives me download.o.o
2018-01-03T15:47:17  <tampakrap> it is indeed not correct, I'll leave it as an excercise to you to find the correct one
2018-01-03T15:47:25  <tampakrap> it gives you download.o.o because it failed to find a backend
2018-01-03T15:47:33  <tampakrap> and it goes to the default, which is download.o.o
2018-01-03T15:47:41  <tampakrap>     ## fallback
2018-01-03T15:47:43  <tampakrap>     default_backend download
2018-01-03T15:48:15  <tampakrap> now after you do a change, you need to do: haproxy -c -f /etc/haproxy/haproxy.cfg
2018-01-03T15:48:20  <tampakrap> to check if the config is correct
2018-01-03T15:48:27  <tampakrap> and also only reload, don't restart
2018-01-03T15:48:35  <tampakrap> restart will drop existing connections
2018-01-03T15:48:47  <tampakrap> we restart only in special cases (eg nothing else worked)
2018-01-03T15:48:51  <cboltz> any objections against putting this command into /etc/haproxy/check_config.sh ?
2018-01-03T15:49:02  <tampakrap> no objections
2018-01-03T15:49:13  <tampakrap> well, better to put it in /usr/local/bin
2018-01-03T15:49:28  <tampakrap> and even better in salt
2018-01-03T15:50:50  <cboltz> well, sometimes I like to have such little helper scripts next to the config files
2018-01-03T15:50:58  <cboltz> that avoids the need to remember the name ;-)
2018-01-03T15:51:17  <tampakrap> put a symlink then
2018-01-03T15:51:55  <cboltz> ok, good idea
2018-01-03T15:54:20  <cboltz>     redirect code 301 prefix   https://cs.opensuse.org if is_mediawiki_cz
2018-01-03T15:54:58  <cboltz> looks much better than redirecting to cz (that's what people already typed), and if is_mediawiki_cz makes more sense than is_dewiki
2018-01-03T16:00:08  <cboltz> looks like the redirects I just created work :-)
2018-01-03T16:00:31  <cboltz> diff -u haproxy.cfg-2018-01-03 haproxy.cfg    if you want to check my changes
2018-01-03T16:01:03  <tampakrap> checking
2018-01-03T16:01:44  *** asmorodskyi has quit IRC
2018-01-03T16:02:12  <tampakrap> they are fine
2018-01-03T16:02:26  <cboltz> :-)
2018-01-03T16:02:48  <cboltz> sounds like I can delete the backup file
2018-01-03T16:04:04  <cboltz> ah, grep is quite useful to understand the config, for example   grep '^[a-z]\|wiki_de' haproxy.cfg
2018-01-03T16:04:49  <cboltz> (much better than scrolling up to the section header each time ;-)
2018-01-03T16:05:29  <tampakrap> yep
2018-01-03T16:06:08  * cboltz puts that grep into another little script
2018-01-03T16:19:39  <cboltz> how often does csync2 run?
2018-01-03T16:19:46  <tampakrap> let's see if I can limit the tickets in 5 pages before I go home
2018-01-03T16:19:50  <cboltz> it looks like elsa still has the old haproxy.cfg
2018-01-03T16:19:56  <tampakrap> I don't remember, just put the stuff into salt :)
2018-01-03T16:22:54  <cboltz> I'm afraid that's a quite big task for haproxy (unless we do a simpe file.managed)
2018-01-03T16:23:04  <tampakrap> yes
2018-01-03T16:23:29  <tampakrap> bonus points for a test that runs haproxy -c -f
2018-01-03T16:25:08  <cboltz> it seems haproxy.service already does this (with an additional -q) in ExecStartPre and ExecReload
2018-01-03T16:25:42  <tampakrap> I think it stops the service eitherway
2018-01-03T16:25:48  <tampakrap> and I also think it doesn't do that on reload
2018-01-03T16:25:53  <cboltz> right, if you use "restart" you are lost
2018-01-03T16:26:15  <cboltz> for reload it should work in theory - but I'm not too keen to test it in production ;-)
2018-01-03T16:26:25  <tampakrap> it doesn't
2018-01-03T16:26:46  <tampakrap> test it on elsa
2018-01-03T16:27:35  <tampakrap> this is still valid? https://progress.opensuse.org/issues/20210
2018-01-03T16:28:04  *** kollex has left #opensuse-admin
2018-01-03T16:28:15  <cboltz> good question
2018-01-03T16:28:29  <cboltz> the wiki cookies are now restricted to each wiki's subdomain
2018-01-03T16:28:49  <cboltz> but I would be surprised if something on paste.o.o was fixed without updating the ticket
2018-01-03T16:29:10  <cboltz> so the bug in paste.o.o probably still exists, it's "just" hidden
2018-01-03T16:31:28  *** mcaj has quit IRC
2018-01-03T16:38:50  <cboltz> just tested - "systemctl reload haproxy" with a broken config file results in
2018-01-03T16:38:55  <cboltz> Job for haproxy.service failed because the control process exited with error code. See "systemctl status haproxy.service" and "journalctl -xe" for details.
2018-01-03T16:39:03  <cboltz> but haproxy is still running
2018-01-03T16:39:13  <cboltz> so reload is probably safe
2018-01-03T16:39:27  <tampakrap> cool
2018-01-03T16:39:33  <cboltz> BTW: haproxy.cfg on elsa is from Dec 7
2018-01-03T16:39:54  <cboltz> so csync didn't do anything for a month...
2018-01-03T16:39:59  <tampakrap> it was changed after updates then, because I am totally sure last time I tried it long time ago it didn't
2018-01-03T16:40:17  <tampakrap> so csync is not run automatically which is cool, more reasons to move the configs to salt :)
2018-01-03T16:40:38  <cboltz> lol
2018-01-03T16:41:04  <cboltz> I agree on the long term, but you'll hate yourself for saying that if elsa has to take over *now*
2018-01-03T16:44:27  <tampakrap> well, move them to salt now then :P
2018-01-03T16:48:26  <cboltz> the list of files mentined in csync2.cfg is a bit ;-) longer than what I want to handle ;-)
2018-01-03T16:52:21  <tampakrap> start with the ones you want to handle :)
2018-01-03T16:56:06  <cboltz> tell that to the other AppArmor devs - they already miss me ;-)
2018-01-03T16:57:08  <cboltz> besides that - I didn't setup all those unsalted services on anna and elsa *eg*
2018-01-03T17:08:49  <heroes-bot> PROBLEM: memcached data stored on riesling.infra.opensuse.org - MEMCACHE CRITICAL: bytes is 246335264  200000000 - memcached 1.4.33 on 127.0.0.1:11211, up 14 days 21 hours ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=riesling.infra.opensuse.org&service=memcached%20data%20stored
2018-01-03T17:09:30  <cboltz> looks like the monitoring noticed the changed memcached parameters ;-)
2018-01-03T17:11:15  *** sven15 has quit IRC
2018-01-03T17:12:12  * tampakrap goes home
2018-01-03T17:14:04  *** malcolmlewis has quit IRC
2018-01-03T17:21:28  *** malcolmlewis has joined #opensuse-admin
2018-01-03T17:21:28  *** malcolmlewis has joined #opensuse-admin
2018-01-03T17:31:48  <cboltz> tampakrap: was the due date 01/01/5000 on https://progress.opensuse.org/issues/2392 and the two related tickets intentional?
2018-01-03T18:00:49  <tampakrap> no
2018-01-03T18:04:30  <cboltz> then fix it ;-)
2018-01-03T18:05:46  <cboltz> BTW: you might like https://github.com/saltstack-formulas/haproxy-formula - but migrating our config to pillar data is probably a PITA
2018-01-03T18:06:13  <cboltz> (just to give you an impression - the jinja template for haproxy.cfg has 585 lines)
2018-01-03T19:36:13  <heroes-bot> PROBLEM: HTTP on conference.infra.opensuse.org - HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 922 bytes in 6.021 second response time ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=conference.infra.opensuse.org&service=HTTP
2018-01-03T19:41:35  <heroes-bot> PROBLEM: HTTP progress on redmine.infra.opensuse.org - HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 968 bytes in 0.011 second response time ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=redmine.infra.opensuse.org&service=HTTP%20progress
2018-01-03T19:48:12  <tampakrap> cboltz: ^^ that's you?
2018-01-03T19:48:35  <cboltz> no
2018-01-03T19:49:33  <tampakrap> okay I'll check
2018-01-03T19:53:03  <tampakrap> postgresql seems to be the issue
2018-01-03T19:53:43  <tampakrap> sorry, mysql
2018-01-03T20:03:01  <heroes-bot> PROBLEM: HAProxy on elsa.infra.opensuse.org - HAPROXY CRITICAL - Active service galera2 is DOWN on galera proxy ! Active service galera3 is DOWN on galera proxy ! Active service redmine is DOWN on redmine proxy ! ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=elsa.infra.opensuse.org&service=HAProxy
2018-01-03T20:03:02  <heroes-bot> PROBLEM: HAProxy on anna.infra.opensuse.org - HAPROXY CRITICAL - Active service galera2 is DOWN on galera proxy ! Active service galera3 is DOWN on galera proxy ! Active service redmine is DOWN on redmine proxy ! ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=anna.infra.opensuse.org&service=HAProxy
2018-01-03T20:08:52  *** cboltz_ has joined #opensuse-admin
2018-01-03T20:08:52  *** cboltz_ has joined #opensuse-admin
2018-01-03T20:10:49  *** cboltz has quit IRC
2018-01-03T20:12:29  *** cboltz_ has quit IRC
2018-01-03T20:12:52  *** cboltz has joined #opensuse-admin
2018-01-03T20:12:52  *** cboltz has joined #opensuse-admin
2018-01-03T20:13:35  <tampakrap> I am checking it now with darix, it seems that logrotate run at the same time on all three nodes and it crashed all dbs AND reset them
2018-01-03T20:13:57  <tampakrap> we have recent backups though
2018-01-03T20:14:17  <cboltz> sounds interesting[tm] :-(
2018-01-03T20:14:56  <tampakrap> it sucks a lot yeah
2018-01-03T20:40:03  <heroes-bot> PROBLEM: HAProxy on mufasa.infra.opensuse.org - HAPROXY CRITICAL - Active service narwal4 is DOWN on static proxy ! Active service riesling is DOWN on riesling proxy ! ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mufasa.infra.opensuse.org&service=HAProxy
2018-01-03T20:44:08  *** tigerfoot has joined #opensuse-admin
2018-01-03T21:01:35  <heroes-bot> RECOVERY: HTTP progress on redmine.infra.opensuse.org - HTTP OK: HTTP/1.1 200 OK - 8460 bytes in 0.104 second response time ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=redmine.infra.opensuse.org&service=HTTP%20progress
2018-01-03T21:06:07  <heroes-bot> RECOVERY: HTTP on conference.infra.opensuse.org - HTTP OK: HTTP/1.1 200 OK - 24126 bytes in 0.450 second response time ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=conference.infra.opensuse.org&service=HTTP
2018-01-03T21:17:41  <tampakrap> restored, darix saved the day once again
2018-01-03T21:31:31  <cboltz> thanks!
2018-01-03T21:32:02  <cboltz> 124 open tickets, so it looks like you managed to get the latest version :-)
2018-01-03T21:32:22  <tampakrap> yep I verified that as well, no data loss fortunately
2018-01-03T21:32:31  <tampakrap> I'm writing a postmortem now
2018-01-03T21:32:41  <katnip> how's it goin today?
2018-01-03T21:32:52  <cboltz> I'm looking forward to read it!
2018-01-03T21:33:15  <katnip> looks busy
2018-01-03T21:37:37  <cboltz> indeed, we just had a strange database failure, and from tampakrap's quick summary it sounded like the data was destroyed on all cluster nodes
2018-01-03T21:37:52  <cboltz> luckily he and darix somehow managed to restore everything :-)
2018-01-03T21:38:11  <cboltz> and I'm really looking forward to read about that "somehow"
2018-01-03T21:40:36  <tampakrap> cboltz: the data was not destroyed on the master node
2018-01-03T21:40:45  <tampakrap> it was destroyed on one of the slaves
2018-01-03T21:41:14  <cboltz> ah, that makes it less critical
2018-01-03T21:49:51  <katnip> that's good
2018-01-03T21:50:15  <katnip> do you use mysql?
2018-01-03T21:50:44  <katnip> i mean for everything
2018-01-03T21:51:46  <cboltz> depends on the service - some use postgresql, some mysql (actually mariadb)
2018-01-03T21:52:26  <katnip> ahh i see
2018-01-03T21:54:18  <katnip> sent a msg
2018-01-03T21:58:11  <katnip> just saw an update to mariadb in zypper dup
2018-01-03T23:00:31  *** tigerfoot has quit IRC