2018-01-10T00:18:13 ok 2018-01-10T00:18:18 got it 2018-01-10T00:55:26 *** Son_Goku has quit IRC 2018-01-10T00:58:24 *** Son_Goku has joined #opensuse-admin 2018-01-10T01:01:29 *** Son_Goku has quit IRC 2018-01-10T01:04:42 *** Son_Goku has joined #opensuse-admin 2018-01-10T01:08:31 *** cboltz has quit IRC 2018-01-10T01:22:51 *** Son_Goku has quit IRC 2018-01-10T01:25:40 *** Son_Goku has joined #opensuse-admin 2018-01-10T01:34:33 *** Son_Goku has quit IRC 2018-01-10T01:40:33 *** Son_Goku has joined #opensuse-admin 2018-01-10T01:50:18 *** Son_Goku has quit IRC 2018-01-10T01:51:26 *** Son_Goku has joined #opensuse-admin 2018-01-10T02:32:52 *** Son_Goku has quit IRC 2018-01-10T02:37:05 *** Son_Goku has joined #opensuse-admin 2018-01-10T02:42:15 *** plinnell_ has joined #opensuse-admin 2018-01-10T02:42:56 *** plinnell has quit IRC 2018-01-10T02:46:41 *** Son_Goku has quit IRC 2018-01-10T02:53:29 *** Son_Goku has joined #opensuse-admin 2018-01-10T03:01:06 *** plinnell_ is now known as plinnell 2018-01-10T03:01:27 *** plinnell has joined #opensuse-admin 2018-01-10T03:10:12 *** Son_Goku has quit IRC 2018-01-10T03:18:10 *** Son_Goku has joined #opensuse-admin 2018-01-10T03:30:02 *** Son_Goku has quit IRC 2018-01-10T03:34:49 *** okurz has quit IRC 2018-01-10T03:35:12 *** okurz_ has joined #opensuse-admin 2018-01-10T03:35:20 *** okurz_ is now known as okurz 2018-01-10T03:54:41 *** Son_Goku has joined #opensuse-admin 2018-01-10T04:28:12 PROBLEM: Hosts syslog on monitor.infra.opensuse.org - CRITICAL: Found files older than 600 minutes /var/log/opensuse/hosts//wikisearch.log was last modified on Wed Jan 3 18:24:51 2018 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=monitor.infra.opensuse.org&service=Hosts%20syslog 2018-01-10T04:43:52 *** Son_Goku has quit IRC 2018-01-10T08:27:59 *** matthias_bgg has joined #opensuse-admin 2018-01-10T08:47:53 *** sven15 has joined #opensuse-admin 2018-01-10T08:48:33 *** mcaj has joined #opensuse-admin 2018-01-10T09:01:14 *** asmorodskyi has joined #opensuse-admin 2018-01-10T09:02:22 *** deneb_alpha has joined #opensuse-admin 2018-01-10T10:23:05 *** asmorodskyi has quit IRC 2018-01-10T10:23:41 *** asmorodskyi has joined #opensuse-admin 2018-01-10T11:07:33 *** cboltz has joined #opensuse-admin 2018-01-10T11:07:33 *** cboltz has joined #opensuse-admin 2018-01-10T11:12:01 *** Son_Goku has joined #opensuse-admin 2018-01-10T11:38:47 *** cthugha has joined #opensuse-admin 2018-01-10T11:42:08 *** ldevulder has quit IRC 2018-01-10T11:47:36 *** cthugha is now known as ldevulder 2018-01-10T11:57:36 *** nicolasbock has joined #opensuse-admin 2018-01-10T11:59:10 *** ldevulder has quit IRC 2018-01-10T12:02:50 *** ldevulder has joined #opensuse-admin 2018-01-10T12:40:55 *** cboltz has quit IRC 2018-01-10T12:46:34 *** asmorodskyi has quit IRC 2018-01-10T12:46:58 *** asmorodskyi has joined #opensuse-admin 2018-01-10T14:03:42 *** Son_Goku has quit IRC 2018-01-10T14:25:02 *** asmorodskyi has quit IRC 2018-01-10T14:25:24 *** asmorodskyi has joined #opensuse-admin 2018-01-10T14:32:57 *** asmorodskyi has quit IRC 2018-01-10T14:33:18 *** asmorodskyi has joined #opensuse-admin 2018-01-10T14:36:53 *** Son_Goku has joined #opensuse-admin 2018-01-10T15:02:06 *** sven15 has quit IRC 2018-01-10T15:05:05 *** Son_Goku has quit IRC 2018-01-10T15:09:39 *** Son_Goku has joined #opensuse-admin 2018-01-10T15:38:39 *** Son_Goku has quit IRC 2018-01-10T16:07:51 *** asmorodskyi has quit IRC 2018-01-10T16:19:19 *** cboltz has joined #opensuse-admin 2018-01-10T16:19:19 *** cboltz has joined #opensuse-admin 2018-01-10T17:20:02 *** deneb_alpha has quit IRC 2018-01-10T17:22:34 RECOVERY: HAProxy on elsa.infra.opensuse.org - HAPROXY OK - freeipa (Active: 1/1) mickey (Active: 1/1) monitor (Active: 1/1) download-private (Active: 1/1) smt (Active: 1/1) lists (Active: 1/1) connect (Active: 1/1) mysql-int (Active: 1/1) download (Active: 1/1) static (Active: 3/3) gccstats (Active: 1/1) conference (Active: 1/1) duke (Active: 1/1) redmine (Active: 1/1) kruemel (Active: 1/1) osccollab (Active: 1/1) keyserver-recon (Active: 0/1) mirrorlist (Active: 1/1) elections (Active: 1/1) tarzan (Active: 1/1) icc (Active: 1/1) nuka (Active: 1/1) riesling (Active: 1/1) galera (Active: 3/3) community (Active: 1/1) etherpad (Active: 1/1) status (Active: 1/1) studioexpress (Active: 1/1) rpmlint (Active: 1/1) keyserver-db (Active: 0/1) keyserver (Active: 1/1) ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=elsa.infra.opensuse.org&service=HAProxy 2018-01-10T17:54:17 *** fvogt has joined #opensuse-admin 2018-01-10T18:15:02 *** matthias_bgg has quit IRC 2018-01-10T18:31:54 one hour until galera might crash again ;-) 2018-01-10T18:43:52 *** pjessen has joined #opensuse-admin 2018-01-10T18:48:01 *** pjessen has quit IRC 2018-01-10T19:36:13 PROBLEM: HTTP on conference.infra.opensuse.org - HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 922 bytes in 6.019 second response time ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=conference.infra.opensuse.org&service=HTTP 2018-01-10T19:37:51 damn, sometimes I hate it when I'm right :-( 2018-01-10T19:39:37 tampakrap: can you please fix galera once more? 2018-01-10T19:41:41 PROBLEM: HTTP progress on redmine.infra.opensuse.org - HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 968 bytes in 6.019 second response time ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=redmine.infra.opensuse.org&service=HTTP%20progress 2018-01-10T19:56:38 *** nicolasbock has quit IRC 2018-01-10T19:57:39 *** matthias_bgg has joined #opensuse-admin 2018-01-10T20:02:14 yey!! 2018-01-10T20:02:32 cboltz: interested to know how to recreate the cluster? 2018-01-10T20:02:37 *** matthias_bgg has quit IRC 2018-01-10T20:03:02 PROBLEM: HAProxy on elsa.infra.opensuse.org - HAPROXY CRITICAL - Active service redmine is DOWN on redmine proxy ! ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=elsa.infra.opensuse.org&service=HAProxy 2018-01-10T20:03:13 I hope we can stop practising that ;-) - but watching what you do in tmux can't hurt 2018-01-10T20:03:52 cool 2018-01-10T20:04:03 tmux is on 2018-01-10T20:05:43 attached on galera1 2018-01-10T20:06:00 I hope you have more fancy stuff than doing a tarball backup ;-) 2018-01-10T20:06:25 no, that's it 2018-01-10T20:06:40 you take a tarball and then it magically comes up 2018-01-10T20:06:50 lol 2018-01-10T20:07:33 BTW: you might want to watch https://github.com/codership/galera/issues/486 and https://jira.percona.com/browse/PXC-881 - they exactly match our log 2018-01-10T20:09:21 I also found http://msutic.blogspot.de/2015/10/confusion-and-problems-with-lostfound.html - but the precondition "/var/lib/mysql/lost+found/ exists" doesn't match in our setup 2018-01-10T20:14:02 ack thanks 2018-01-10T20:14:20 tomorrow morning I'll talk to the mysql packager to check the logs and I'll file a bug 2018-01-10T20:14:27 but meanwhile I don't know how to workaround it 2018-01-10T20:14:39 maybe not use the cluster, I'll ask darix 2018-01-10T20:15:05 my guess is still that we should try to disable OPTIMIZE in the backup script 2018-01-10T20:15:23 where is that? 2018-01-10T20:15:26 in sysconfig? 2018-01-10T20:15:32 this is done after the database dumps, and since the database dumps look, it's the most likely candidate 2018-01-10T20:15:52 yes, /etc/sysconfig/mysql-backupscript 2018-01-10T20:16:06 okay let's try it after the cluster is up again 2018-01-10T20:17:00 the backup script is independent of fixing the cluster, so we can change the sysconfig file now 2018-01-10T20:18:00 yes I meant to try to run it 2018-01-10T20:18:24 ah, ok 2018-01-10T20:18:27 makes sense 2018-01-10T20:19:22 sysconfig on galera2 changed - no more optimize 2018-01-10T20:20:38 sorry for the strange control chars in tmux - looks like it doesn't like mouse wheel scrolling ;-) 2018-01-10T20:31:48 cboltz: backup finished, are you watching? 2018-01-10T20:32:12 yes 2018-01-10T20:34:33 is there something special to check in SHOW STATUS? 2018-01-10T20:35:36 wsrep_local_state_comment | Synced 2018-01-10T20:35:38 this line 2018-01-10T20:35:46 boosters got corrupted again 2018-01-10T20:35:55 nice[tm] 2018-01-10T20:36:07 RECOVERY: HTTP on conference.infra.opensuse.org - HTTP OK: HTTP/1.1 200 OK - 24126 bytes in 0.227 second response time ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=conference.infra.opensuse.org&service=HTTP 2018-01-10T20:36:16 the good thing is that the dumps succeed, so we have an up-to-date backup ;-) 2018-01-10T20:36:23 (note that I didn't validate today's dumps) 2018-01-10T20:37:40 let's try it 2018-01-10T20:40:03 PROBLEM: HAProxy on anna.infra.opensuse.org - HAPROXY CRITICAL - Active service boosters is DOWN on connect proxy ! ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=anna.infra.opensuse.org&service=HAProxy 2018-01-10T20:40:05 PROBLEM: HAProxy on mufasa.infra.opensuse.org - HAPROXY CRITICAL - Active service riesling is DOWN on riesling proxy ! Active service narwal4 is DOWN on static proxy ! ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mufasa.infra.opensuse.org&service=HAProxy 2018-01-10T20:41:36 RECOVERY: HTTP progress on redmine.infra.opensuse.org - HTTP OK: HTTP/1.1 200 OK - 8450 bytes in 0.233 second response time ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=redmine.infra.opensuse.org&service=HTTP%20progress 2018-01-10T20:44:15 connect.o.o works fine now 2018-01-10T20:44:33 :-) 2018-01-10T20:45:12 ok, so let me start the backup script on galera2 2018-01-10T20:45:17 (feel free to tmux attach) 2018-01-10T20:45:43 we need to move the today's backups first right? 2018-01-10T20:46:04 indeed 2018-01-10T20:46:29 next attemp ;-) 2018-01-10T20:47:26 the problem is that it doesn't fail always, the previous times it was failing once per two or three days 2018-01-10T20:47:59 okay it finished, let's try again 2018-01-10T20:48:56 right, it didn't fail daily - but still quite often 2018-01-10T20:49:49 okay two attempts no crash 2018-01-10T20:49:58 let's hope for the best 2018-01-10T20:50:53 if you have a look at /var/log/mysql-backup.log you'll notice that the crashes were always two seconds after "Starting automatic repair and optimization of the databases/tables" 2018-01-10T20:51:11 therefore I hope disabling optimize really helps 2018-01-10T20:51:38 hmm, what about re-enabling optimize and running the backup script again to see if it crashes ;-) 2018-01-10T20:52:22 not really needed, we have enough evidence for my ticket already, I can set up a testing env for more breaking 2018-01-10T20:52:30 I asked also darix for his opinion on this 2018-01-10T20:52:44 ok 2018-01-10T20:53:54 I'll send a short update to the heroes ML later 2018-01-10T21:11:51 tampakrap: BTW: you really deserve a "told you so" - I already suspected the "optimize" yesterday ;-) 2018-01-10T21:36:36 I was chatting with darix yesterday all that time :( 2018-01-10T21:36:48 darix approves the optimize thing 2018-01-10T21:36:59 but I'm filing the bug tomorrow either way 2018-01-10T21:42:19 please CC me in the bugreport ;-) 2018-01-10T21:46:54 okay 2018-01-10T21:49:24 to avoid duplicate work - maybe the mail I just sent to heroes@ is useful as a template for that bugreport 2018-01-10T22:11:45 where is most of the usage of suse and opensuse come from? i was looking at that link cboltz in the mailing list 2018-01-10T22:12:04 that indicates russia? 2018-01-10T22:12:27 *** fvogt has quit IRC 2018-01-10T22:17:40 I assume you mean the statistics on release-tools.opensuse.org 2018-01-10T22:18:03 yes 2018-01-10T22:18:16 that's only used by a small part of the tumbleweed users, so I wouldn't trust those statistics too much to get a general overview for tumbleweed 2018-01-10T22:18:40 ok 2018-01-10T22:18:56 was curious 2018-01-10T22:19:42 I'm afraid I don't have access to better statistics 2018-01-10T22:20:18 I can only guess that there are lots of german users because SUSE is based in germany 2018-01-10T22:20:32 (well, originally - nowadays there are SUSE offices around the world) 2018-01-10T22:21:03 that doesnt surprise me too much knowing the history, but the russian use does 2018-01-10T22:21:16 *** Son_Goku has joined #opensuse-admin 2018-01-10T22:23:07 options are a) we have lots of russian users, b) russian users favor the frozen snapshots much more than the rest of the world and c) russia is big ;-) 2018-01-10T22:24:00 i do understand they have their own flavor of linux 2018-01-10T22:25:14 I don't know, but maybe openSUSE is better? ;-) 2018-01-10T22:26:12 i sense that they would make a choice between openSUSE and Redhat, or rather SLED or RHEL, they would use SLED 2018-01-10T22:27:32 RHEL is a very good distro however built for servers with good support, and owns Centos and Fedora 2018-01-10T22:27:58 does Suse have a Centos branding? 2018-01-10T22:28:06 like a free version? 2018-01-10T22:28:56 yes, Leap ;-) 2018-01-10T22:29:15 i thought that was like fedora? 2018-01-10T22:29:57 I know it's not an exact match 2018-01-10T22:30:06 ahhh ok 2018-01-10T22:30:09 i see 2018-01-10T22:30:34 Leap is SLE + additional packages (for example KDE) + in rare cases newer packages than in SLE (for example AppArmor) 2018-01-10T22:30:46 tw imo is better than fedora being rolling, i like it much better than fedora 2018-01-10T22:31:07 that summary is a bit simplified, but good enough to understand Leap 2018-01-10T22:31:49 i see, more stable too i assume 2018-01-10T22:32:30 yes, the SUSE customers test SLE before the stuff hits Leap ;-)) 2018-01-10T22:33:27 BTW: fedora uses openQA since a while - they started with that after Richard Brown tested fedora and sent them a big bunch of bugreports ;-) 2018-01-10T22:33:45 (with "tested" I mean "tested with openQA") 2018-01-10T22:34:31 ive seen the usage of fedora packages in other channels including debian, and wondered how that applies 2018-01-10T22:35:40 it's technically possible to convert RPMs to DEB packages and the other way round with "alien" 2018-01-10T22:35:55 oh that's right 2018-01-10T22:36:10 needless to say that this is not really recommended, but for simple packages it can work 2018-01-10T22:36:57 gotcha, i forgot about alien 2018-01-10T23:37:59 https://news.opensuse.org/2018/01/09/future-tumbleweed-snapshot-to-bring-yast-changes/