2018-01-09T00:21:44 *** BManojlovic has quit IRC 2018-01-09T02:07:13 *** Fraser_Bell has joined #opensuse-admin 2018-01-09T02:07:13 *** Fraser_Bell has joined #opensuse-admin 2018-01-09T02:37:01 *** dddh_ has joined #opensuse-admin 2018-01-09T02:58:20 *** nicolasbock has quit IRC 2018-01-09T03:14:48 *** Son_Goku has quit IRC 2018-01-09T03:31:59 *** Fraser_Bell has quit IRC 2018-01-09T03:35:30 *** okurz has quit IRC 2018-01-09T03:36:18 *** okurz has joined #opensuse-admin 2018-01-09T03:40:12 *** Son_Goku has joined #opensuse-admin 2018-01-09T03:59:56 *** Son_Goku has quit IRC 2018-01-09T04:28:12 PROBLEM: Hosts syslog on monitor.infra.opensuse.org - CRITICAL: Found files older than 600 minutes /var/log/opensuse/hosts//wikisearch.log was last modified on Wed Jan 3 18:24:51 2018 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=monitor.infra.opensuse.org&service=Hosts%20syslog 2018-01-09T06:46:45 *** tigerfoot has joined #opensuse-admin 2018-01-09T08:29:51 *** asmorodskyi has joined #opensuse-admin 2018-01-09T08:53:05 *** sven15 has joined #opensuse-admin 2018-01-09T09:02:28 *** mcaj has joined #opensuse-admin 2018-01-09T09:31:19 *** matthias_bgg has joined #opensuse-admin 2018-01-09T09:37:10 *** Son_Goku has joined #opensuse-admin 2018-01-09T09:42:57 *** sven15 has quit IRC 2018-01-09T09:44:54 *** Son_Goku has quit IRC 2018-01-09T09:52:58 *** Son_Goku has joined #opensuse-admin 2018-01-09T09:56:34 *** sven15 has joined #opensuse-admin 2018-01-09T10:02:10 *** Son_Goku has quit IRC 2018-01-09T10:02:32 *** deneb_alpha has joined #opensuse-admin 2018-01-09T10:02:32 *** deneb_alpha has joined #opensuse-admin 2018-01-09T11:13:00 *** Son_Goku has joined #opensuse-admin 2018-01-09T11:15:23 *** cboltz has joined #opensuse-admin 2018-01-09T11:15:23 *** cboltz has joined #opensuse-admin 2018-01-09T11:22:46 *** Son_Goku has quit IRC 2018-01-09T11:26:38 *** Son_Goku has joined #opensuse-admin 2018-01-09T11:49:45 *** cthugha has joined #opensuse-admin 2018-01-09T11:53:05 *** ldevulder has quit IRC 2018-01-09T12:00:04 *** matthias_bgg has quit IRC 2018-01-09T12:21:37 *** nicolasbock has joined #opensuse-admin 2018-01-09T12:37:51 *** asmorodskyi has quit IRC 2018-01-09T12:46:25 *** matthias_bgg has joined #opensuse-admin 2018-01-09T12:50:56 *** asmorodskyi has joined #opensuse-admin 2018-01-09T12:56:44 *** cboltz has quit IRC 2018-01-09T12:59:11 *** matthias_bgg has quit IRC 2018-01-09T13:50:27 *** matthias_bgg has joined #opensuse-admin 2018-01-09T13:54:23 *** Son_Goku has quit IRC 2018-01-09T14:02:39 *** deneb_alpha has quit IRC 2018-01-09T14:14:11 *** cthugha is now known as ldevulder 2018-01-09T14:24:51 *** asmorodskyi has quit IRC 2018-01-09T14:25:20 *** cthugha has joined #opensuse-admin 2018-01-09T14:28:16 *** default__ has joined #opensuse-admin 2018-01-09T14:28:41 *** ldevulder has quit IRC 2018-01-09T14:30:16 *** asmorodskyi has joined #opensuse-admin 2018-01-09T14:32:05 *** cthugha has quit IRC 2018-01-09T14:32:10 *** Son_Goku has joined #opensuse-admin 2018-01-09T14:56:23 *** default__ is now known as ldevulder 2018-01-09T15:02:30 *** nicolasbock has quit IRC 2018-01-09T15:43:49 *** Son_Goku has quit IRC 2018-01-09T15:49:49 *** Son_Goku has joined #opensuse-admin 2018-01-09T16:02:26 *** Son_Goku has quit IRC 2018-01-09T16:06:42 *** asmorodskyi has quit IRC 2018-01-09T16:34:46 *** cboltz has joined #opensuse-admin 2018-01-09T16:36:37 *** sven15 has quit IRC 2018-01-09T16:46:35 *** matthias_bgg has quit IRC 2018-01-09T16:53:47 *** mcaj has quit IRC 2018-01-09T17:47:33 *** fvogt has joined #opensuse-admin 2018-01-09T18:26:32 *** tigerfoot has quit IRC 2018-01-09T18:36:39 *** Son_Goku has joined #opensuse-admin 2018-01-09T18:41:40 *** Ada_Lovelace has joined #opensuse-admin 2018-01-09T18:54:21 hello partypeople 2018-01-09T18:56:06 *** mmaher_home has joined #opensuse-admin 2018-01-09T18:56:17 hello partyman 2018-01-09T19:00:01 so meeting time! 2018-01-09T19:00:04 who is here? 2018-01-09T19:00:12 hi everybody! 2018-01-09T19:00:17 I'm here 2018-01-09T19:01:00 Ada_Lovelace: ping? 2018-01-09T19:01:44 hi 2018-01-09T19:02:16 Hi 2018-01-09T19:02:30 cboltz: do you have the ticket handy? 2018-01-09T19:02:44 https://progress.opensuse.org/issues/28911 2018-01-09T19:03:13 thanks 2018-01-09T19:03:14 wanna chair? 2018-01-09T19:03:29 why not ;-) 2018-01-09T19:03:41 so first topic - questions and answers from the community 2018-01-09T19:04:10 does someone have a question? 2018-01-09T19:04:24 We had the topic election system in our Board meeting yesterday. 2018-01-09T19:04:36 I have one question: Why is there so much spam in admin tickets? 2018-01-09T19:04:45 tampakrap: It seems you want to pick it up? 2018-01-09T19:05:03 *** pjessen has joined #opensuse-admin 2018-01-09T19:05:07 okurz: because the admin@ are too common mail addresses 2018-01-09T19:05:17 could be 2018-01-09T19:05:20 sorry i', late 2018-01-09T19:05:28 okurz: I'm afraid the answer is "because mx*.suse.de isn't really good at blocking spam" 2018-01-09T19:05:35 Ada_Lovelace: well, tomas told me that you and cboltz offered to take over from where we left 2018-01-09T19:06:12 I missed yesterday's board meeting, so - good to know ;-)) 2018-01-09T19:06:26 ticket number for the election system is https://progress.opensuse.org/issues/25486 2018-01-09T19:06:32 cboltz: You are a Python expert. Do you want to do it? 2018-01-09T19:06:33 and update from my side: Sorry, because of many computer problems (and SLE bugs to test) today I did not find the time to check my openSUSE admin VPN and won't be able to do tomorrow, maybe thursday but I consider it also not urgent from my point of view. I want to start with understanding who progress.o.o is setup and then at best help tampakrap to update it and optionally extend with addons 2018-01-09T19:06:46 last two comments list what needs to be done 2018-01-09T19:07:23 Thanks, okurz! :) 2018-01-09T19:08:05 Ada_Lovelace: I have a lot in my plate these days so I'd like to avoid having to do this as well, my initial agreement would be to setup the python instance and give it to somebody to complete it 2018-01-09T19:08:28 so if possible I'd like to avoid doing it and focus on some security issues, like the progress.o.o upgrade that okurz mentioned 2018-01-09T19:08:47 and continue updating or retiring the rest of the sle11 services we have 2018-01-09T19:08:51 I was surprised, that it is assigned to you, tampakrap. 2018-01-09T19:09:23 it is assigned to me because I was working on it with tomas 2018-01-09T19:09:30 Ah... 2018-01-09T19:09:54 I never looked at helios on the technical side, therefore I hesitate to say yes instantly ;-) 2018-01-09T19:09:55 but I'd really appreciate it if someone could take over from here, the instance is somehow running either way 2018-01-09T19:10:14 well me neither :P 2018-01-09T19:10:42 I would say take a look at the current stuff we did and the remaining tasks and try to solve them, and as always feel free to come back to me for help 2018-01-09T19:10:51 cboltz: packages are finished. The connection for Members is missing. 2018-01-09T19:10:53 is that acceptable? 2018-01-09T19:11:35 Ada_Lovelace: in other words: the boring part is done, and "just" the difficult part remains ;-) 2018-01-09T19:11:46 ;-) 2018-01-09T19:11:54 it was quite hard to package it actually 2018-01-09T19:12:09 sounds interesting[tm] 2018-01-09T19:12:49 I'll have a look at it later - but that's all I'll promise for now ;-) 2018-01-09T19:13:05 Ok. 2018-01-09T19:13:32 okurz: Do you need any support by us or do you receive it at SUSE? 2018-01-09T19:14:41 okay but keep in mind that it is important and urgent as the elections are coming 2018-01-09T19:14:51 but you are both board members and you know that already :) 2018-01-09T19:15:01 indeed ;-) 2018-01-09T19:15:14 Ada_Lovelace: thank you I have enough support and want to *give* something myself :) 2018-01-09T19:15:30 I'll announce a Call for election commitee this week. And Richard meant April would be a good time for elections. 2018-01-09T19:16:08 okurz: Great! :) 2018-01-09T19:16:16 okurz: so after you finish your vpn, we can have a call one day so I can walk you around the current instance and we can work on the upgrade together 2018-01-09T19:16:40 okurz: the plugins are challenging as it will be a major upgrade, so we'll have to disable a lot of them, maybe all 2018-01-09T19:17:03 we'll have to ask various teams which ones of those plugins are still using etc, so it's going to be quite some work 2018-01-09T19:17:11 Sure, don't expect much from my side in the upcoming week though. My main goal is to understand the current setup of progress.o.o *readonly*, later one potentially *change* it 2018-01-09T19:17:31 and I'm still not sure how the db migration in such a big upgrade is going to be, so we'll need to give it to people for testing before we do the final dump 2018-01-09T19:18:01 okay 2018-01-09T19:18:05 Sure, at best I can rebuild a setup locally - potentially with a reduced database portion - and try it out first 2018-01-09T19:18:24 we have a staging host already waiting to be set up 2018-01-09T19:18:56 so... while reading here i was looking into the question ... as far as i see here... mx1 and mx2 just forward the mails to lists5-opensuse.suse.de for admin@ without filtering anything, because we don't do filtering on mx, we do it in later steps and/or rely on the spamfilters our users have in place... which means ... lists5 maybe should add a filtering for the admin@ address? 2018-01-09T19:19:03 keep in mind though that the admin work will not be a major part of my work so don't get your hopes up ;) I will just want to help solve "our own problems" instead of just hammering you with tickets 2018-01-09T19:19:04 sorry for writing offtopic =) 2018-01-09T19:20:04 thomic: we had a large discussion one/two meetings ago about setting up our own mx/relays, but it is frozen due to the heinlein agreement 2018-01-09T19:20:13 yay... 2018-01-09T19:20:14 thomic: might be an option, but not the best one - it will cause backscatter to whoever is the poor owner of the mail address the spammers use 2018-01-09T19:20:27 ^^ correct 2018-01-09T19:20:42 spam blocking should be done on mx*.suse.de to be able to _reject_ the spam instead of bouncing it 2018-01-09T19:20:59 that is not easy to do. 2018-01-09T19:21:26 you need to scan the email whilst it is inflight, i.e. not queued. 2018-01-09T19:21:45 cboltz: nope... it is not done on mx 2018-01-09T19:21:52 for no domains 2018-01-09T19:22:05 as far as i see .. i dont have the full view yet.. 2018-01-09T19:22:13 pjessen: right, but that's not really hard 2018-01-09T19:22:16 but afaik we do spam filtering in a later step 2018-01-09T19:23:27 thomic: "in a later step" sounds like a) "spam folder" or b) backscatter, and IMHO both are a bad idea (and a) is useless for admin@) 2018-01-09T19:23:54 i guess it is a topic for another day, but spending 10-15 seconds in spamassassin before the email is queued means holding up the client with NOOPs etc. 2018-01-09T19:24:56 cboltz: indeed we drop the mails one step later and/or have a filtering there.. but anyways I agree for admin@ it would not help there 2018-01-09T19:25:08 i just wanted to answer the question "why" 2018-01-09T19:25:14 it is because of the mailsetup atm :) 2018-01-09T19:26:26 next topic then? 2018-01-09T19:26:32 yes 2018-01-09T19:26:34 Yes 2018-01-09T19:26:39 status reports about everything 2018-01-09T19:27:20 tampakrap: maybe you should start with the galera cluster ;-) 2018-01-09T19:27:22 the salt and monitoring hosts lists are deviating, any volunteers to help me check them one by one and add them to the one or the other? 2018-01-09T19:27:57 too late, I started with something else :) 2018-01-09T19:28:32 silence, I take that as a no, I'll do it alone 2018-01-09T19:29:33 so galera cluster: we still need to do the logrotate.timer (currently off on galera1) and enable backups on galera3 2018-01-09T19:29:47 cboltz: you want me to replicate here the whole story that I sent on the mailing list? 2018-01-09T19:30:08 that would be superfluous ;-) 2018-01-09T19:30:30 The solution should be enough. ;-) 2018-01-09T19:30:52 I'm still surprised that logrotate (which might mean restarting mysql) causes the problem 2018-01-09T19:31:18 so any takers for those two tasks? 2018-01-09T19:31:22 I assume you compared the logrotate time with the time when mysql broke, but it still sounds strange 2018-01-09T19:31:47 I can help you with logrotate.timer 2018-01-09T19:32:03 *** Son_Goku has quit IRC 2018-01-09T19:32:12 what sounds strange? 2018-01-09T19:32:12 (maybe after the meeting) 2018-01-09T19:32:21 I guess logrotate was not to blame? 2018-01-09T19:32:39 that it only takes a log rotation (and restarting mysql) to break the cluster 2018-01-09T19:32:50 I'd expect it to be more stable 2018-01-09T19:33:31 why are we logrotating mysql?? 2018-01-09T19:34:13 why not? 2018-01-09T19:34:28 if we don't, one day we will end up with a log that fills up the disk space :) 2018-01-09T19:34:35 because mysql doesn't write any logs. 2018-01-09T19:34:48 the default logrotate.d/mariadb rotates /var/log/mysql/mysqld.log 2018-01-09T19:34:54 I never rotate any mysql logs. 2018-01-09T19:35:18 galera maybe does? there is definitely a /var/log/mysql/mysqld.log 2018-01-09T19:35:28 the logs written are usually query logs, but they have to turned on and off 2018-01-09T19:35:39 yes, the log is there, but is it growing? 2018-01-09T19:36:14 if it grows by e.g. 1kb a day, it should of course be rotated. 2018-01-09T19:36:34 logrotate uses /usr/bin/mysqladmin --local flush-error-log flush-engine-log flush-general-log flush-slow-log so mysql doesn't even get restarted 2018-01-09T19:36:43 which makes it even more surprising that the cluster breaks 2018-01-09T19:36:49 so apparently the logroate is not the issue, there is no logrotate running right now on the cluster 2018-01-09T19:36:54 and it went down an hour ago 2018-01-09T19:36:56 :) 2018-01-09T19:36:57 (all assuming that we are talking about the mariadb packages from 42.3) 2018-01-09T19:37:24 I'll fix it after the meeting 2018-01-09T19:37:41 PROBLEM: HTTP progress on redmine.infra.opensuse.org - HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 968 bytes in 6.020 second response time ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=redmine.infra.opensuse.org&service=HTTP%20progress 2018-01-09T19:37:46 ah no wait 2018-01-09T19:37:53 it logrotated again 2018-01-09T19:37:58 but how? logrotate.timer is down 2018-01-09T19:38:15 paste(bin) systemctl status logrotate.timer please 2018-01-09T19:38:26 let's move with the meeting and do that after? 2018-01-09T19:38:32 ok 2018-01-09T19:39:34 so next topic? 2018-01-09T19:40:10 depends - does someone else have a status report? 2018-01-09T19:41:05 doesn't look so, so next topic 2018-01-09T19:41:10 opensuse-education.org https://progress.opensuse.org/issues/29784 2018-01-09T19:41:34 ignore the link while progress is down ;-) 2018-01-09T19:42:07 basically the question is a) do we want to take over the opensuse-education.org domain and b) what do we want to do with it? 2018-01-09T19:42:13 PROBLEM: HTTP on conference.infra.opensuse.org - HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 922 bytes in 6.034 second response time ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=conference.infra.opensuse.org&service=HTTP 2018-01-09T19:43:01 The content currently provided via that domain will also be removed 2018-01-09T19:43:04 (Webpage as well as ISO images and RPMs provided via 2018-01-09T19:43:06 http://files.opensuse-education.org/ ). 2018-01-09T19:43:32 The ISO images and other files are currently mirrored to a host called 2018-01-09T19:43:34 community.opensuse.org (community.infra.opensuse.org). I leave it up to 2018-01-09T19:43:36 you to decide what should happen with the content on that mirror 2018-01-09T19:43:38 server. Feel free to delete it as well or leave it as it is for 2018-01-09T19:43:40 reference. 2018-01-09T19:43:42 The source of the webpage is currently available at GitHub: 2018-01-09T19:43:44 https://github.com/openSUSE-Education/www.opensuse-education.org 2018-01-09T19:43:46 Please tell me until 2018-01-31 if I should add someone as new Owner of 2018-01-09T19:43:48 the repository. 2018-01-09T19:43:50 this is the mail 2018-01-09T19:44:09 maybe the content could be copied to education.o.o ? 2018-01-09T19:44:22 it is outdated since 2015 2018-01-09T19:44:29 I see two options 2018-01-09T19:44:38 either somebody wants to take the project and care about the content 2018-01-09T19:44:46 or we don't deliver ISOs from 2015 2018-01-09T19:45:13 we can't deliver insecure, oldstable under official .o.o domain, as debian is doing ;) 2018-01-09T19:45:20 that's not our style i think 2018-01-09T19:45:41 saving the content somewhere (non-accessible) if somebody asks ... works for me 2018-01-09T19:45:52 but I won't publish it anywhere if it is outdated like this 2018-01-09T19:45:56 okay, I assumed somebody cared about the content. 2018-01-09T19:46:02 yes lars :) 2018-01-09T19:46:14 Somebody by Ubuntu wanted to have the Life name. openSUSE education staied at openSUSE. 2018-01-09T19:46:21 thomic: agreed, even if I'm not sure if I get your poke at debian ;-) 2018-01-09T19:46:48 cboltz: was unfair, at least they security-bugfix oldstable ;) 2018-01-09T19:46:56 so let it die? 2018-01-09T19:47:21 If nobody wants to maintain it... 2018-01-09T19:47:28 so we have time until end of this month 2018-01-09T19:47:36 do we go down the road asking for somebody? 2018-01-09T19:47:46 like on opensuse@ or opensuse-announce@? 2018-01-09T19:47:55 opensuse-project I'd say 2018-01-09T19:47:58 or do we just kill it? 2018-01-09T19:48:14 opensuse-project and opensuse should be right lists for that. 2018-01-09T19:48:30 unless someone creates updated ISOs, one option would be to make the domain a redirect to https://en.opensuse.org/Education (assuming that and the package repo are less outdated) 2018-01-09T19:48:33 okay any takers? 2018-01-09T19:48:41 * cboltz hates handing out domains to spammers 2018-01-09T19:49:16 ah interesting note 2018-01-09T19:49:18 the domain 2018-01-09T19:49:22 do we want to keep it? 2018-01-09T19:49:26 it belongs to lars privately 2018-01-09T19:49:34 if it dies, the domain dies as well 2018-01-09T19:49:37 just fyi 2018-01-09T19:49:53 two domains 2018-01-09T19:49:58 I didn't copy the first part sorry 2018-01-09T19:50:04 The domains "opensuse-education.org" and "opensuse-education.de" will 2018-01-09T19:50:07 be free after 2018-01-31. If you like, I can open a transfer request to 2018-01-09T19:50:08 anyone who wants to take over. 2018-01-09T19:50:30 for the .de address, you need an address in Germany 2018-01-09T19:50:51 or a registrar with a german proxy 2018-01-09T19:51:06 if we want to keep them 2018-01-09T19:51:09 i would take them over 2018-01-09T19:51:16 I'm a domain collecter anyways :D 2018-01-09T19:51:31 but I don't mind if we transfer them to the "official" pool 2018-01-09T19:51:36 i would even prefer that 2018-01-09T19:51:41 but no clue how to do that ;) 2018-01-09T19:52:03 to add them to freeipa you mean? 2018-01-09T19:52:07 my vote: if we are not going use them, we dont need them. 2018-01-09T19:52:30 tampakrap: maybe :) 2018-01-09T19:52:38 there is internet-x listed as sponsor on the education web page 2018-01-09T19:52:42 what do you mean by official pool then? 2018-01-09T19:52:48 we could ask them kindly to give us a free account 2018-01-09T19:53:06 tampakrap: yes, freeipa, but freeipa is only managing dns ... the domains belong to SUSE / Novell afaik 2018-01-09T19:53:19 which means "the official by company managed pool" 2018-01-09T19:53:29 it is more about law than about technology 2018-01-09T19:54:02 Is anyone feeling mad, if these domains die? 2018-01-09T19:54:07 If yes speak up now :) 2018-01-09T19:54:24 If not, I would say, lets ask once more on the opensuse-project 2018-01-09T19:54:25 okay my proposal: someone send a mail to the mailing lists asking for a new maintainer till 29/1 2018-01-09T19:54:30 if nobody replies, let it die 2018-01-09T19:54:32 if somebody wants to maintain 2018-01-09T19:54:46 if somebody replies, thomic takes the domains and the service remains alive for another 2 years 2018-01-09T19:54:47 i would ask internet-x for sponsorships or kick people in provo 2018-01-09T19:54:48 :) 2018-01-09T19:54:57 I'd keep the domains nevertheless 2018-01-09T19:55:10 ok so.. i start kicking anyways 2018-01-09T19:55:28 i will write them a very polite german mail ... :) 2018-01-09T19:55:52 thanks! 2018-01-09T19:56:19 next topic? 2018-01-09T19:56:28 yes and no 2018-01-09T19:56:38 next topic would be the new election website 2018-01-09T19:56:46 which we already discussed 2018-01-09T19:57:01 so anything else from anyone or can we close the meeting? 2018-01-09T19:58:18 well, maybe I have something 2018-01-09T19:58:40 now that Lars left the heroes, who will organize the offsite meeting? 2018-01-09T19:58:56 thomic: is this something you could do? 2018-01-09T19:59:26 cboltz: I'll let you know before the next month's team meeting on the mailing list 2018-01-09T19:59:45 ok 2018-01-09T20:00:04 PROBLEM: HAProxy on mufasa.infra.opensuse.org - HAPROXY CRITICAL - Active service riesling is DOWN on riesling proxy ! Active service narwal4 is DOWN on static proxy ! ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mufasa.infra.opensuse.org&service=HAProxy 2018-01-09T20:01:01 anything else? 2018-01-09T20:01:36 not from here 2018-01-09T20:02:05 looks like we can close the meeting 2018-01-09T20:02:12 thanks everybody! 2018-01-09T20:02:12 good, thanks everybody 2018-01-09T20:02:26 cboltz: you can ssh as root to the galera[1-3] nodes 2018-01-09T20:02:29 See you at FOSDEM or in the next meeting on Feb 6th ;-) 2018-01-09T20:02:32 on galera1 I have tmux open 2018-01-09T20:02:57 i'm not at FOSDEM :) 2018-01-09T20:02:58 *** Ada_Lovelace has quit IRC 2018-01-09T20:03:02 PROBLEM: HAProxy on elsa.infra.opensuse.org - HAPROXY CRITICAL - Active service redmine is DOWN on redmine proxy ! ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=elsa.infra.opensuse.org&service=HAProxy 2018-01-09T20:03:03 PROBLEM: HAProxy on anna.infra.opensuse.org - HAPROXY CRITICAL - Active service redmine is DOWN on redmine proxy ! ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=anna.infra.opensuse.org&service=HAProxy 2018-01-09T20:03:18 cboltz: Lars promised to organize the next offsite 2018-01-09T20:03:27 but I can do it with the dudes in Nbg if you want me to 2018-01-09T20:03:39 I can as well talk to Lars 2018-01-09T20:03:55 please talk to Lars ;-) 2018-01-09T20:04:21 see you later, everyone. 2018-01-09T20:04:37 *** pjessen has quit IRC 2018-01-09T20:05:52 tampakrap: mysqld.log on galera1 is interesting[tm] 2018-01-09T20:05:55 2018-01-09 19:31:28 0x7f0d75ce1700 InnoDB: Assertion failure in file /home/abuild/rpmbuild/BUILD/mariadb-10.2.11/storage/innobase/pars/pars0pars.cc line 817 2018-01-09T20:06:06 so we might have hit a bug 2018-01-09T20:06:49 I see it yes 2018-01-09T20:07:02 it happens every day at 19:31 UTC, I doubt it is a bug 2018-01-09T20:07:28 I have tmux on galera2 as well 2018-01-09T20:11:31 indeed, that would be a very timely bug ;-) 2018-01-09T20:12:59 systemctl list-timers lists systemd-tmpfiles-clean.timer (daily, last run 6h ago) and fstrim.timer (weekly, last run 1 day 20h ago) 2018-01-09T20:13:48 with --all (to list inactive timers) I see logrotate last run Sa 2018-01-06 00:00:01 UTC 2018-01-09T20:13:59 that doesn't look like 19:31 ;-) 2018-01-09T20:14:45 so darix found an interesting upstream bug 2018-01-09T20:14:53 which now leads that the backup job is the issue 2018-01-09T20:14:57 guess what time it runs :) 2018-01-09T20:15:01 https://jira.mariadb.org/browse/MDEV-12023 2018-01-09T20:16:07 causing a crash by doing a backup is... nice[tm] 2018-01-09T20:17:12 I never worked with galera, so - can I simply do a rcmysql restart on galera1, or is there something special I should know? 2018-01-09T20:17:40 there are steps to do 2018-01-09T20:17:57 leave it as it is please, I'll fix it after the backup finishes 2018-01-09T20:18:05 I am always taking a backup when the db crashes 2018-01-09T20:18:12 ok 2018-01-09T20:19:28 BTW: /var/spool/cron/lastrun/cron.daily has timestamp 19:00 2018-01-09T20:19:47 in galera1? 2018-01-09T20:19:55 yes 2018-01-09T20:20:22 19:30 on galera2 2018-01-09T20:20:42 and 20:00 on galera3 2018-01-09T20:21:42 so if I had to guess, I'd say running the backup on galera2 is the problem 2018-01-09T20:22:02 yes 2018-01-09T20:22:08 the backups are running only on galera2 2018-01-09T20:28:00 interestingly the dumps in /var/lib/mysql-files/backup/20180109/ look good (dumps for all databases exist, and each of them has a "Dump completed" at the end), so it _seems_ the crash happens late enough 2018-01-09T20:28:29 I didn't check if all tables are listed in the dumps 2018-01-09T20:28:58 so according to darix, mariadb is buggy and we should stop the backups for now :/ 2018-01-09T20:29:18 nice[tm] 2018-01-09T20:37:43 cboltz: there is an update available, I'll try that 2018-01-09T20:37:54 good idea 2018-01-09T20:38:09 it seems like the backup script also optimizes the database 2018-01-09T20:38:22 after doing the dumps 2018-01-09T20:38:30 since the dumps look good - could that be the reason? 2018-01-09T20:39:02 the bug says that it creates a temp table that goes on all dbs 2018-01-09T20:39:28 rephrasing: 2018-01-09T20:39:58 the backup creates a temp table which gets replicated, and that crashes the cluster 2018-01-09T20:40:09 so I'll try the .12 update and see what happens 2018-01-09T20:40:46 I'm slightly surprised - my guess would be that doing a backup is a read-only operation that shouldn't create a temp table 2018-01-09T20:41:16 but yeah, upgrade to .12 and then start the script in /etc/cron.daily manually 2018-01-09T20:41:49 okay 2018-01-09T20:41:54 if it crashes again, disable OPTIMIZE in /etc/sysconfig/mysql/mysql-backupscript and try again 2018-01-09T20:43:25 BTW: the backup script has a check for "today's backup dir already exists", so you'll need to rename the /var/lib/mysql-files/backup/20180109 directory 2018-01-09T20:50:06 looks like mysql started on galera2 (and only there) after the reboot 2018-01-09T20:51:12 yes I stopped it 2018-01-09T20:51:18 I forgot to disable it 2018-01-09T20:51:22 systemctl status -n100 mysql looks like there's some fun with the connect database 2018-01-09T20:52:04 oh, and [ERROR] Can't open shared library '/usr/lib64/mysql/plugin/ha_cassandra.so' (errno: 0, cannot open shared object file: No such file or directory) 2018-01-09T20:54:26 PROBLEM: MySQL WSREP connected on galera2.infra.opensuse.org - CRIT wsrep_connected (str) = OFF ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=galera2.infra.opensuse.org&service=MySQL%20WSREP%20connected 2018-01-09T20:58:46 something is wrong, the cluster didn't come up as expected :( 2018-01-09T20:59:08 waiting for darix to call me to save the day again 2018-01-09T20:59:35 i'm here, had a dr appt that i forgot about, what have i missed? 2018-01-09T21:00:05 tampakrap: do you need a second pair of eyes? 2018-01-09T21:00:13 i could stare at it at least :) 2018-01-09T21:02:04 * cboltz starts to wonder if a single mysql instance would be more HA than a cluster *g,d&r* 2018-01-09T21:17:36 RECOVERY: HTTP progress on redmine.infra.opensuse.org - HTTP OK: HTTP/1.1 200 OK - 8450 bytes in 0.085 second response time ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=redmine.infra.opensuse.org&service=HTTP%20progress 2018-01-09T21:17:55 may i ask an offtopic dead question in here or wait till later? 2018-01-09T21:20:56 just adk 2018-01-09T21:20:58 ask 2018-01-09T21:22:06 RECOVERY: HTTP on conference.infra.opensuse.org - HTTP OK: HTTP/1.1 200 OK - 24126 bytes in 0.241 second response time ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=conference.infra.opensuse.org&service=HTTP 2018-01-09T21:23:53 well, ive asked before, that is why i call it a dead question, how do you guys use @opensuse.org email with a relay? i am unable to find a solution unless i attempt to make my own on my suse computer, i'm very frustrated over this. i am even unable to unsubscribe to lists so i can subscribe with another address :( 2018-01-09T21:24:40 sorry for being repetitious 2018-01-09T21:26:44 my solution probably won't help you, but still - I'm admin on several mailservers, and can use any sender address I want 2018-01-09T21:26:59 the more helpful answer for you is: 2018-01-09T21:27:18 check if your mail provider allows you to add additional sender addresses 2018-01-09T21:27:32 it does not 2018-01-09T21:28:04 unless like G Suite, the domain is registered thru them 2018-01-09T21:29:38 i'm unable to unsubscribe to a list that i dont care much for 2018-01-09T21:30:10 i do not remember how i did iti before 2018-01-09T21:30:25 very angry with myself 2018-01-09T21:33:03 I found https://support.google.com/mail/answer/22370?hl=en - but it seems this requires you to enter the SMTP Auth data on the remote server which makes it somewhat useless :-( 2018-01-09T21:35:01 if your main problem is to unsubscribe from an openSUSE mailinglist, tell me which list and with which mail address you are subscribed, and I should be able to send a mail with your address as sender to ...+unsubscribe 2018-01-09T21:35:06 that is the problem, without the smtp or mx records, i'm screwed 2018-01-09T21:35:34 -offtopic for one 2018-01-09T21:36:36 -announce dont pertain to me 2018-01-09T21:36:51 cboltz: you can even go to baloo to do it with mlmmj commands 2018-01-09T21:37:20 tampakrap: or just remove him from the subscribers txt files :D 2018-01-09T21:37:32 tampakrap: you assume that I know anything about mlmmj ;-) 2018-01-09T21:37:39 is that safe? 2018-01-09T21:37:45 katnip your mail address is not here :) 2018-01-09T21:37:53 it has some internal db iirc 2018-01-09T21:37:54 tampakrap: no clue.. of course i prefer mlmmj stuff 2018-01-09T21:37:55 ;) 2018-01-09T21:38:00 so better not touch the txt files 2018-01-09T21:38:04 yay wait katnip 2018-01-09T21:38:06 katnip@opensuse.org 2018-01-09T21:38:09 :( 2018-01-09T21:38:18 now we have a mail address in public logs :( 2018-01-09T21:38:32 thats why i dont like public irc log 2018-01-09T21:38:45 is this channel logged? 2018-01-09T21:39:02 katnip: see topic :) 2018-01-09T21:39:17 shrug 2018-01-09T21:40:00 ive got my personal email in the logs too 2018-01-09T21:40:14 hrhr :) 2018-01-09T21:40:56 we should be able to use smtp settings at least 2018-01-09T21:44:07 the logs have a "Disallow" in robots.txt, so at least search engines shouldn't pick them up 2018-01-09T21:44:26 good 2018-01-09T21:44:52 i dont use google search anyway, ddg, but i do use chrome 2018-01-09T21:44:53 katnip: you should have happened :) 2018-01-09T21:45:01 got a confirmation? 2018-01-09T21:45:03 huh? 2018-01-09T21:45:10 i unsubscribed you from offtopic 2018-01-09T21:45:32 not yet 2018-01-09T21:46:07 now? :) 2018-01-09T21:46:14 nope 2018-01-09T21:46:20 hm maybe it takes a while 2018-01-09T21:46:25 however it was only offtopic? 2018-01-09T21:46:34 you seem to be not subscribed to announce 2018-01-09T21:47:44 if not, try either linuxmajic@gmail.com OR james@linuxmajic.com 2018-01-09T21:47:59 personal email again 2018-01-09T21:48:43 as i said query :) 2018-01-09T21:49:38 *** mmaher_home has quit IRC 2018-01-09T21:49:55 *** mmaher_home has joined #opensuse-admin 2018-01-09T21:50:46 what is the differnce between a suse.com and opensuse.org ? 2018-01-09T21:51:15 suse.com is with a mailbox from company 2018-01-09T21:51:19 opensuse.org is just an alias 2018-01-09T21:51:27 *** mmaher_home has quit IRC 2018-01-09T21:51:44 *** mmaher_home has joined #opensuse-admin 2018-01-09T21:52:00 ahh ok, suse.com is usually an employee? 2018-01-09T21:53:47 yup 2018-01-09T21:53:48 it is 2018-01-09T21:53:49 :) 2018-01-09T21:53:54 employee only 2018-01-09T21:54:12 ok, now how do we use debian in the packages? 2018-01-09T21:54:44 so .. this unsubscribe did not work really well :) 2018-01-09T21:54:45 i like 2018-01-09T21:55:03 in the email lists, ive seen apt as usable 2018-01-09T21:55:10 no it did not 2018-01-09T21:58:32 katnip: 2018-01-09T21:58:36 i cannot query you :) 2018-01-09T21:58:40 you blocked it 2018-01-09T21:58:41 *** mmaher_home has quit IRC 2018-01-09T21:59:27 katnip: ? 2018-01-09T21:59:52 ah ... you mean how apt-* is working 2018-01-09T21:59:56 it's just a wrapper for zypper 2018-01-09T22:00:17 how did i block it?? 2018-01-09T22:00:35 23:00:25-!- [card.freenode.net] katnip is in +g mode (server-side ignore.) 2018-01-09T22:00:38 23:00:25-!- [card.freenode.net] katnip has been informed that you messaged 2018-01-09T22:00:41 them. 2018-01-09T22:00:52 oh in here 2018-01-09T22:00:56 one sec 2018-01-09T22:01:24 katnip: https://build.opensuse.org/package/show/openSUSE%3ALeap%3A42.3%3AUpdate/patchinfo.7628 2018-01-09T22:01:25 should work now, if it's from you 2018-01-09T22:01:28 this is the wrapper 2018-01-09T22:01:34 for zypper 2018-01-09T22:01:40 used for convenience for debian users 2018-01-09T22:01:47 but it executes zypper 2018-01-09T22:02:04 still same error katnip 2018-01-09T22:02:07 so lets try here :D 2018-01-09T22:03:21 should work this time, grrrr 2018-01-09T22:05:14 *** fvogt has quit IRC 2018-01-09T22:05:33 katnip: i pinged you in query 2018-01-09T22:05:36 and wrote down stuff 2018-01-09T22:05:38 maybe you saw it 2018-01-09T22:05:42 at least for offtopic 2018-01-09T22:05:51 you should got a confirm now of unsubscription 2018-01-09T22:05:52 :) 2018-01-09T22:08:08 yes 2018-01-09T22:23:35 connect is heavily broken o.O ... all related to this databse fun tampakrap ? 2018-01-09T22:23:41 or already sleeping for tommorow? 2018-01-09T22:24:13 I'm trying to fix it without much luck 2018-01-09T22:24:32 tampakrap: quick hint where i set a users mail alias? 2018-01-09T22:24:39 just for my brain ... i forgot 2018-01-09T22:25:03 you need admin rights 2018-01-09T22:25:10 ah .. which i dont have 2018-01-09T22:25:12 but I'll bring it down again 2018-01-09T22:25:20 so katnip ^^ wait for connect to work again 2018-01-09T22:25:28 somebody will change your mail alias on connect 2018-01-09T22:25:36 and it is automatically synced to our mail aliases :) 2018-01-09T22:25:52 that's why i asked for a ticket, somebody else needs to do it katnip ^^ 2018-01-09T22:26:00 yay take it down tampakrap nevermind 2018-01-09T22:26:10 I'm home then tampakrap .. tell me if you need a hand 2018-01-09T22:26:14 i'm willing to help 2018-01-09T22:26:47 ack 2018-01-09T22:30:33 so 20180105 is the latest working backup which I used 2018-01-09T22:30:42 katnip: what do you need? 2018-01-09T22:31:10 it is about his mailalias changed 2018-01-09T22:31:15 no worries for now 2018-01-09T22:31:22 the mailinglist foo i already handled on baloo 2018-01-09T22:31:34 i need to move my alias from james@insiberia.net to james@linuxmajic.com 2018-01-09T22:31:39 tampakrap: damm, now we lost some spam on connect.o.o ;-)) 2018-01-09T22:32:09 :'( 2018-01-09T22:32:13 katnip: done 2018-01-09T22:32:21 ok thank you 2018-01-09T22:32:48 now, i am unable to log into 'daffy" ; The username/password combination you entered is invalid. Please try again, or recover it from here. (THIS IS DAFFY). 2018-01-09T22:33:46 where are you trying to log in? 2018-01-09T22:34:06 https://progress.opensuse.org/ICSLogin/auth-up 2018-01-09T22:34:43 well, it works for me 2018-01-09T22:35:38 403 2018-01-09T22:35:39 You are not authorized to access this page. 2018-01-09T22:36:34 on which page are you now? 2018-01-09T22:38:47 i seem to have found my way around :) 2018-01-09T23:14:53 thank you guys 2018-01-09T23:46:42 *** Son_Goku has joined #opensuse-admin 2018-01-09T23:52:04 opensuse-heroes+subscribe@opensuse.org < is this correct spelling for subscribing, it keeps coming back 2018-01-09T23:54:42 no, heroes+subscribe@opensuse.org