2018-04-28T00:16:09 *** Son_Goku has joined #opensuse-admin 2018-04-28T00:34:00 *** Son_Goku has quit IRC 2018-04-28T02:14:57 *** okurz has quit IRC 2018-04-28T02:15:46 *** okurz has joined #opensuse-admin 2018-04-28T03:36:17 *** Son_Goku has joined #opensuse-admin 2018-04-28T03:56:49 *** Son_Goku has quit IRC 2018-04-28T04:33:48 *** lcp has quit IRC 2018-04-28T05:36:21 *** victorhck has quit IRC 2018-04-28T05:39:34 *** victorhck has joined #opensuse-admin 2018-04-28T07:19:29 *** fvogt has joined #opensuse-admin 2018-04-28T07:29:11 goodmorning. 2018-04-28T07:45:11 *** tigerfoot has quit IRC 2018-04-28T07:45:35 *** tigerfoot has joined #opensuse-admin 2018-04-28T08:11:51 *** fvogt has quit IRC 2018-04-28T08:12:48 *** fvogt has joined #opensuse-admin 2018-04-28T08:34:05 *** fvogt has quit IRC 2018-04-28T08:41:56 *** plinnell has quit IRC 2018-04-28T08:46:13 anyone here? 2018-04-28T09:15:24 pjessen: 2018-04-28T09:15:26 i'm around 2018-04-28T09:15:30 saw your problem 2018-04-28T09:15:38 PTR are handled by MF :( 2018-04-28T09:15:43 I'm talking to darix atm 2018-04-28T09:15:51 maybe we just disable the new IPv6 address 2018-04-28T09:19:07 pjessen: fixed i migrated dns to the old v6 addresses 2018-04-28T09:19:12 on baloo 2018-04-28T09:19:17 sorry for inconvenience 2018-04-28T09:21:08 cool, tnx 2018-04-28T09:21:59 gibt nochmal ein "kleines hickup" wenn wir die "neue" range wegnehmen in 10-15min 2018-04-28T09:22:08 aber dann wirds sichs hoffentlich normalisieren 2018-04-28T09:24:42 das duerfte kein Problem sein. 2018-04-28T09:46:13 ok 2018-04-28T09:46:29 pjessen: die neuen ips sind weg 2018-04-28T09:46:31 sollte jetzt tun 2018-04-28T09:58:26 okay, mal schauen. 2018-04-28T09:59:55 *** lcp has joined #opensuse-admin 2018-04-28T10:07:13 thomic, bist du noch hier? auf baloo etwas zu aendern bringt nichts - was ist mit relay.infra.o.o ? 2018-04-28T10:46:59 *** lcp has quit IRC 2018-04-28T10:49:46 *** plinnell has joined #opensuse-admin 2018-04-28T10:49:46 *** plinnell has joined #opensuse-admin 2018-04-28T11:06:03 *** nicolasbock has joined #opensuse-admin 2018-04-28T11:23:34 pjessen: ja 2018-04-28T11:23:37 jetzt 2018-04-28T11:23:39 sorry 2018-04-28T11:23:40 :) 2018-04-28T11:23:55 samstag und nebenbei noch vorbereitungen für die konferenz 2018-04-28T11:24:01 ehm achso.. relay ist das problem? 2018-04-28T11:24:02 oO 2018-04-28T11:29:05 *** cboltz has joined #opensuse-admin 2018-04-28T11:34:46 *** lcp has joined #opensuse-admin 2018-04-28T11:50:45 *** tigerfoot has quit IRC 2018-04-28T12:00:01 ja, relay ist js der ausgang zum google 2018-04-28T12:02:21 *** cboltz has quit IRC 2018-04-28T12:35:30 *** mcaj_nb has joined #opensuse-admin 2018-04-28T12:36:30 *** mcaj_nb has joined #opensuse-admin 2018-04-28T12:38:27 *** lcp has quit IRC 2018-04-28T12:57:13 *** Son_Goku has joined #opensuse-admin 2018-04-28T13:38:39 *** lcp has joined #opensuse-admin 2018-04-28T14:38:23 *** lcp has quit IRC 2018-04-28T14:39:40 *** lcp has joined #opensuse-admin 2018-04-28T14:47:56 *** victorhck has quit IRC 2018-04-28T14:47:56 *** victorhck has joined #opensuse-admin 2018-04-28T15:07:22 *** lcp has quit IRC 2018-04-28T15:19:29 *** lcp has joined #opensuse-admin 2018-04-28T15:29:27 *** mcaj_nb has quit IRC 2018-04-28T15:31:16 *** cboltz has joined #opensuse-admin 2018-04-28T15:31:16 *** cboltz has joined #opensuse-admin 2018-04-28T15:40:33 *** lcp has quit IRC 2018-04-28T15:42:27 *** lcp has joined #opensuse-admin 2018-04-28T15:43:34 PROBLEM: NRPE on sarabi.infra.opensuse.org - connect to address 192.168.47.15 port 5666: Connection refused ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=sarabi.infra.opensuse.org&service=NRPE 2018-04-28T15:45:29 *** mcaj_nb has joined #opensuse-admin 2018-04-28T15:58:19 PROBLEM: SSH on rpmlint.infra.opensuse.org - SSH CRITICAL - OpenSSH_7.2 (protocol 2.0) version mismatch, expected OpenSSH_6.6.1 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=rpmlint.infra.opensuse.org&service=SSH 2018-04-28T16:43:02 *** lcp_ has joined #opensuse-admin 2018-04-28T16:43:33 *** lcp has quit IRC 2018-04-28T16:43:34 *** lcp_ is now known as lcp 2018-04-28T16:46:49 *** lcp has quit IRC 2018-04-28T16:47:28 *** lcp has joined #opensuse-admin 2018-04-28T16:49:10 *** mcaj_nb has quit IRC 2018-04-28T16:54:23 *** plusky has quit IRC 2018-04-28T16:54:55 *** plusky has joined #opensuse-admin 2018-04-28T17:46:24 *** lcp has quit IRC 2018-04-28T17:47:57 *** lcp has joined #opensuse-admin 2018-04-28T17:51:31 *** nicolasbock has quit IRC 2018-04-28T18:35:57 *** duncanmv has joined #opensuse-admin 2018-04-28T18:49:37 *** lcp has quit IRC 2018-04-28T18:52:25 *** lcp has joined #opensuse-admin 2018-04-28T18:52:49 OBS backend is not authenticating software.opensuse.org Net::HTTPForbidden 2018-04-28T19:23:04 so software.o.o is down once more - nice[tm] 2018-04-28T19:24:08 thomic: is this something you can fix? 2018-04-28T19:29:27 actually - duncanmv, since you have access to the error message, do you also have permissions to restart the service? 2018-04-28T19:30:08 last time this happened thomic did cboltz: donald:~ # systemctl restart software_opensuse_org.service 2018-04-28T19:38:26 cboltz: I am suspecting somethin gon the s.o.o side now 2018-04-28T19:38:49 cboltz: no, the app is running, it can't just connect to OBS backend. I restarted it already 2018-04-28T19:39:01 but I am debugging the code. Now the problem seems to be in memcached. Looking. 2018-04-28T19:39:12 the app can get the distribution list from OBS 2018-04-28T19:39:16 it just can't get it from the cache 2018-04-28T19:42:39 ok, it is up 2018-04-28T19:42:47 I know what happened 2018-04-28T19:42:52 :-) 2018-04-28T19:42:56 OBS was down somewhen 2018-04-28T19:43:14 thomic: fixed, continue to enjoy your weekend ;-) 2018-04-28T19:43:17 Rails.cache.fetch {block} stores in the cache and calls block if the key is not there 2018-04-28T19:43:26 but the block returned nil when OBS was down 2018-04-28T19:43:30 so the key was there, stored nil 2018-04-28T19:43:37 and that makes the app think OBS was down 2018-04-28T19:43:50 so basically cache poisoning? 2018-04-28T19:44:04 I would say very bad written code 2018-04-28T19:44:16 like methods returning member variables 2018-04-28T19:44:27 it is hard to read where side-effects happen 2018-04-28T19:44:47 at the same time modifying them 2018-04-28T19:45:04 silenting exceptions 2018-04-28T19:45:06 and returning nil 2018-04-28T19:45:24 oh, nice 2018-04-28T19:45:36 sounds like this code wears a big "rewrite me!" sign ;-) 2018-04-28T19:46:50 may I suggest another uptime improvement? 2018-04-28T19:47:05 I mean, if the app is supposed to go down if we cant get the distributions from OBS, why would we silent an exception when ApiConnect::get(/distributions') fails? 2018-04-28T19:47:20 cboltz: it would be helpful that status.opensuse.org does not show green when this happens 2018-04-28T19:47:27 cboltz: sure 2018-04-28T19:47:56 status.o.o gets updated manually (we have plans to change that), so as long as nobody updates it, it will stay green 2018-04-28T19:48:31 I know it's not perfect, but it's what what we have right now 2018-04-28T19:50:06 so back to what I'd like to suggest: 2018-04-28T19:50:12 the software.o.o main page and the distribution pages (Tumbleweed and Leap) are static, right? 2018-04-28T19:50:32 it would be a good idea to have them online even if OBS is down 2018-04-28T19:51:28 would that be doable without too much effort? 2018-04-28T19:52:01 the reason it goes down is because it inherits from the same controller which preloads some data 2018-04-28T19:52:11 I think it should be possible 2018-04-28T19:52:19 we can refactor obs_controller or something 2018-04-28T19:52:24 I will create an issue for that 2018-04-28T19:53:10 can you CC me, please? (I'm "cboltz" everywhere) 2018-04-28T19:54:04 yes 2018-04-28T19:54:13 right now though, the top priority is to find a leak we have 2018-04-28T19:54:26 I am about to deploy support to trace produciton in real time 2018-04-28T19:55:20 *** lcp has quit IRC 2018-04-28T19:55:59 yes, I heard about that - have fun hunting down this leak! 2018-04-28T19:56:29 deployed! 2018-04-28T19:57:52 software.o.o (including the search) still works :-) 2018-04-28T20:00:02 PROBLEM: HAProxy on elsa.infra.opensuse.org - HAPROXY CRITICAL - Active service dale is DOWN on dale proxy ! ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=elsa.infra.opensuse.org&service=HAProxy 2018-04-28T20:00:03 PROBLEM: HAProxy on anna.infra.opensuse.org - HAPROXY CRITICAL - Active service dale is DOWN on dale proxy ! ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=anna.infra.opensuse.org&service=HAProxy 2018-04-28T20:00:04 PROBLEM: HAProxy on mufasa.infra.opensuse.org - HAPROXY CRITICAL - Active service riesling is DOWN on riesling proxy ! ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mufasa.infra.opensuse.org&service=HAProxy 2018-04-28T20:02:24 *** lcp has joined #opensuse-admin 2018-04-28T20:04:01 cboltz: problem is that the main page does need OBS access 2018-04-28T20:04:08 to fill the search dropdown 2018-04-28T20:04:34 oh, but that one is the one that shows "OBS now available" 2018-04-28T20:04:39 "not available" 2018-04-28T20:06:17 I'd say that you could use aggressive caching for that - updating the search dropdown once per day would be fast enough IMHO 2018-04-28T20:07:08 yeah, what I mean, the main page is not down if OBS is down 2018-04-28T20:07:22 but for some reason the reverse proxy shows us in maintenance if OBS is down 2018-04-28T20:07:28 I need to undertand that logic 2018-04-28T20:08:08 ahhh we show a page, but with status 400 2018-04-28T20:10:47 I'd guess haproxy doesn't like the 400 (but don't know it good enough to know why it translates it to a 503) 2018-04-28T20:13:17 I cced you in the new issue 2018-04-28T20:14:38 thanks! 2018-04-28T20:18:48 * duncanmv will enable tracing 2018-04-28T20:47:11 *** lcp has quit IRC 2018-04-28T20:52:37 *** lcp has joined #opensuse-admin 2018-04-28T20:56:55 duncanmv: software.o.o is down again 2018-04-28T21:03:34 *** lcp has quit IRC 2018-04-28T21:06:05 cboltz: yes 2018-04-28T21:06:10 I took it down to get the dump 2018-04-28T21:06:15 but it looks like it will not work 2018-04-28T21:06:26 I ran out of disk 2018-04-28T21:08:27 up again 2018-04-28T21:08:33 and I managed to get the trace 2018-04-28T21:15:46 enjoy analyzing it ;-) 2018-04-28T21:20:07 cboltz: 2018-04-28T21:22:42 *** Son_Goku has quit IRC 2018-04-28T21:24:39 *** Son_Goku has joined #opensuse-admin 2018-04-28T21:30:11 *** Fraser_Bell has joined #opensuse-admin 2018-04-28T21:30:11 *** Fraser_Bell has joined #opensuse-admin 2018-04-28T21:30:42 something does not make sense 2018-04-28T21:45:29 define "something" please ;-) 2018-04-28T21:49:19 cboltz: IMHO, there are a lot of somethings in this world that fits that description. ;-) 2018-04-28T21:57:59 *** matthias_bgg has joined #opensuse-admin 2018-04-28T22:00:34 indeed ;-) 2018-04-28T22:02:51 *** fvogt has joined #opensuse-admin 2018-04-28T22:08:09 *** lcp has joined #opensuse-admin 2018-04-28T22:15:07 *** Fraser_Bell has quit IRC 2018-04-28T22:17:21 *** duncanmv has quit IRC 2018-04-28T22:24:44 *** Son_Goku has quit IRC 2018-04-28T22:26:36 *** duncanmv has joined #opensuse-admin 2018-04-28T22:38:43 *** Son_Goku has joined #opensuse-admin 2018-04-28T22:39:05 *** fvogt has quit IRC 2018-04-28T22:47:47 *** matthias_bgg has quit IRC 2018-04-28T22:48:55 *** Son_Goku has quit IRC 2018-04-28T22:55:51 *** maxlin has quit IRC 2018-04-28T22:56:07 *** maxlin has joined #opensuse-admin 2018-04-28T23:02:21 *** duncanmv has quit IRC 2018-04-28T23:20:44 *** cboltz has quit IRC 2018-04-28T23:34:22 *** Son_Goku has joined #opensuse-admin 2018-04-28T23:56:38 *** maxlin_ has joined #opensuse-admin 2018-04-28T23:56:51 *** maxlin has quit IRC