2020-07-16T02:20:07  *** okurz_ is now known as okurz
2020-07-16T04:45:40  <a-865k> mirrorbrain to dumb to find http://ftp5.gwdg.de/pub/opensuse/distribution/leap/15.2/repo/oss/noarch/gfxboot-branding-openSUSE-15.2.20200203-lp152.1.5.noarch.rpm
2020-07-16T04:45:48  <a-865k> had to fetch it manually
2020-07-16T05:03:21  * klein upgrading gitlab
2020-07-16T05:13:59  <klein> gitlab upgraded ;-)
2020-07-16T06:44:36  <pjessen> goodmorning
2020-07-16T06:45:18  <pjessen> kl_eisbaer: cool, thanks
2020-07-16T06:45:30  <pjessen> we seem to be having a dns issue on anna
2020-07-16T06:53:21  <pjessen> disregard, just 3000 queued mails, some due to DNS issues.
2020-07-16T07:16:36  *** ldevulder_ is now known as ldevulder
2020-07-16T07:24:01  <kl_eisbaer1> pjessen: I'm currently "zypper dup'ing" pontifex. Will look at the DNS issue after that, ok?
2020-07-16T07:33:37  <kl_eisbaer1> pjessen: news@myclash.de is indeed a fake. this domain does not exist (any longer?). I deleted those mails now - together with some others. Now we are down to 1410
2020-07-16T07:36:14  <pjessen> kl_eisbaer1: okay, funny - myclash.de did resolve for me and the address wis subscribed
2020-07-16T07:36:39  <kl_eisbaer1> I tried 8.8.8.8, 9.9.9.10 and 1.1.1.1 DNS servers without result
2020-07-16T07:37:04  <pjessen> I see mxes for myclash as birne.myclash.net and bohne.myclash.net
2020-07-16T07:37:16  <pjessen> yeah I also tried 8.8.8.8
2020-07-16T07:37:54  <kl_eisbaer1> hm. misterious. Worst case, they lost some Emails now. I'm sorry for that, but this address alone had 1023 "undeliverables"
2020-07-16T07:38:25  <kl_eisbaer1> JFYI: pontifex is now also 15.2.
2020-07-16T07:39:14  <pjessen> anyway, no DNS issue after all
2020-07-16T07:40:02  <kl_eisbaer1> which finally leaves only pinot, riesling, water, identification and dale running 15.1.
2020-07-16T07:40:35  <kl_eisbaer1> As I can not log in any longer to identification, I leave this one up to lcp
2020-07-16T07:41:05  <kl_eisbaer1> dale == events => henne
2020-07-16T07:41:27  <kl_eisbaer1> the rest is cboltz
2020-07-16T07:42:13  <pjessen> yeah, pinot and riesling sounds like him :-)
2020-07-16T07:42:21  <kl_eisbaer1> which on the other side means: 53 openSUSE infra machines run 15.2 already :D
2020-07-16T07:42:36  <pjessen> good going
2020-07-16T07:43:29  <kl_eisbaer1> jip. The most funniest (as usual) were the database servers, as we upgraded the database versions with that upgrade as well.
2020-07-16T07:44:04  <pjessen> any probs with that?
2020-07-16T07:44:32  <kl_eisbaer1> No, not really. But you need to plan that upgrade. Especially in a Galera-Cluster.
2020-07-16T07:44:59  <pjessen> ah yes
2020-07-16T07:48:28  <kl_eisbaer1> and postgres wants this /usr/lib/postgresql$version/bin/pg_upgrade gedöns ;-)
2020-07-16T07:49:03  <kl_eisbaer1> The good news: I just found some minor bugs. Sadly one even in one of my packages....
2020-07-16T08:09:53  <pjessen> have doublechecked dns for mx[12], looks good
2020-07-16T08:23:41  <kl_eisbaer1> pjessen: perfect! :-) So when do you plan to go online with the new MX?
2020-07-16T08:30:37  <pjessen> kl_eisbaer1: it's really ready to go, but I dont want to go live right before I go on vacation
2020-07-16T08:34:14  <pjessen> i'm back on 3 august, so maybe we announce the switch for 4 or 5 august ?
2020-07-16T08:53:11  <henne> kl_eisbaer1: feel free to update 🙂
2020-07-16T08:53:33  <henne> if something goes wrong with the deployment of OSEM, I'll check afterward
2020-07-16T08:55:47  <kl_eisbaer1> henne: thanks - omw
2020-07-16T08:57:56  <kl_eisbaer1> henne: FYI: I need to release your package lock on ruby for the upgrade - and libressl-devel will also go away. So there is a high chance that your webapp might be broken after the upgrade
2020-07-16T08:58:31  <kl_eisbaer1> => you might indeed need some time to get it up and running again.
2020-07-16T08:59:15  <kl_eisbaer1> henne: Last chance for a veto - otherwise I'll hit the enter button ;-)
2020-07-16T08:59:35  <henne> hit it!
2020-07-16T08:59:43  <kl_eisbaer1> done
2020-07-16T08:59:59  <henne> no risk, no fun
2020-07-16T09:00:08  <kl_eisbaer1> indeed
2020-07-16T09:00:34  <kl_eisbaer1> but I'm confident that the machine itself will survive the upgrade :D
2020-07-16T09:20:30  <kl_eisbaer1> henne: dale is up and running again - and awaits your love regarding the application level.
2020-07-16T11:05:27  <lcp99> kl_eisbaer1: I will look into customizing surveys somewhat
2020-07-16T11:05:42  <kl_eisbaer1> lcp99: thanks!
2020-07-16T11:05:52  <lcp99> I'm curious about why mousepad.ico on etherpad >:D
2020-07-16T11:06:03  <kl_eisbaer1> lcp99: But I guess you need to provide a complete limesurvey theme for that
2020-07-16T11:06:11  <lcp99> I do
2020-07-16T11:06:41  <kl_eisbaer1> lcp99: because I did not want always the same ico on all pages. That makes it harder to differentiate in the browser tabs ;-)
2020-07-16T11:07:04  <lcp99> technically the icons I did for chat should also be used for favicons
2020-07-16T11:07:19  <lcp99> but I need to write a script to render them at higher resolutions
2020-07-16T11:08:59  <lcp99> I will probably submit some of them later today for use with services
2020-07-16T11:15:40  <cboltz> FYI: pinot.i.o.o is running 15.2 since a minute
2020-07-16T11:17:08  <lcp99> bmwiedemann: can I bother you about getting oidc metadata for matrix again
2020-07-16T11:26:12  <kl_eisbaer1> cboltz: thanks!
2020-07-16T11:30:22  <cboltz> it was a boring update (as predicted by the test update on my test VM) ;-)
2020-07-16T11:32:43  <cboltz> (well, except that the slspath issue breaks highstate - on the positive side, I think I've seen a notice in bugzilla that an update for this is on its way)
2020-07-16T11:36:57  <lcp99> I reported that ages ago, I don't know why it has been taking this long tbh
2020-07-16T11:37:40  <cboltz> well, some things take longer than you'd hope ;-)
2020-07-16T11:38:46  <cboltz> I also had/have some fun with the AppArmor package - submitted late in the 15.2 cycle, and the paperwork prevented it from being included in 15.2
2020-07-16T11:38:55  <cboltz> and now I'm waiting to get it released as update
2020-07-16T11:39:15  <cboltz> luckily it's a not-too-serious issue ;-)
2020-07-16T12:49:31  <henne> events.o.o is back
2020-07-16T13:29:52  <henne> kl_eisbaer1: did you also do something to sql?
2020-07-16T13:30:34  <henne> some of the tables changed columns to `char(0)`
2020-07-16T13:30:50  <henne> which is a bit short for storing data ;-)
2020-07-16T13:35:43  <henne> on `mysql.infra.opensuse.org` database `conference_osem`
2020-07-16T13:44:44  <henne> basically all text columns change in that way
2020-07-16T13:44:49  <henne> is this some new default?
2020-07-16T13:51:43  <henne> anyone else who might know about the cluster?
2020-07-16T13:51:46  <pjessen> henne:  lars mentioned earlier that the upgrade to 15.2 also meant upgrding postgres.
2020-07-16T13:52:06  <henne> this is not postgre
2020-07-16T13:52:27  <pjessen> sorry, i meant mysql.
2020-07-16T13:54:22  <pjessen> he wrote "... as we upgraded the database versions with that upgrade as well."
2020-07-16T13:58:03  <kl_eisbaer1> henne: the mariadb update was done last week already.
2020-07-16T13:58:30  <kl_eisbaer1> henne: so I asume that this was not related (or nobody noticed over a whole week?)
2020-07-16T13:58:39  <kl_eisbaer1> do you have a local DB ?
2020-07-16T13:58:44  <henne> no idea. I noticed now
2020-07-16T13:58:49  <kl_eisbaer1> or parts from it?
2020-07-16T13:58:54  <henne> nope
2020-07-16T13:59:13  <henne> just using mysql.infra.opensuse.org
2020-07-16T13:59:18  <kl_eisbaer1> I can restore a backup from yesterday, to have a compare. Do you know if something changed from yesterday that we might loose?
2020-07-16T13:59:35  <henne> backup of what? the database content?
2020-07-16T13:59:41  <kl_eisbaer1> right
2020-07-16T14:00:00  <henne> this is not content. this is table/column definitions
2020-07-16T14:00:14  <kl_eisbaer1> Jul 16 03:01 conference_osem.sql
2020-07-16T14:00:17  <henne> I doubt the backup would change anything
2020-07-16T14:00:46  <henne> to me this sounds like a changed mariadb default
2020-07-16T14:00:51  <henne> or whatever is running there
2020-07-16T14:00:52  <pjessen> those char(0) columns should be visible in the backup ?  assuming it is a mysqldump
2020-07-16T14:01:00  <kl_eisbaer1> if I search for "char(0)" in the backup, I find nothing
2020-07-16T14:01:12  <kl_eisbaer1> yes, it's a mysqldump
2020-07-16T14:02:20  <henne> can you put the .sql file on dale?
2020-07-16T14:02:30  <kl_eisbaer1> ome
2020-07-16T14:02:57  <kl_eisbaer1> see root@dale:/root/conference_osem.sql
2020-07-16T14:03:51  <henne> grep -c 'char(0)' conference_osem.sql
2020-07-16T14:03:51  <henne> 45
2020-07-16T14:04:22  <kl_eisbaer1> hm: right
2020-07-16T14:04:29  <kl_eisbaer1> missed that one
2020-07-16T14:04:43  <henne> you have more backups?
2020-07-16T14:04:48  <henne> maybe from before the update?
2020-07-16T14:05:07  <kl_eisbaer1> I'm currently at 20200623 ....
2020-07-16T14:05:25  <henne> grep locked_by $file
2020-07-16T14:05:33  <kl_eisbaer1> 20200617/conference_osem.sql.xz:45
2020-07-16T14:05:34  <kl_eisbaer1> 20200618/conference_osem.sql.xz:45
2020-07-16T14:05:34  <kl_eisbaer1> 20200619/conference_osem.sql.xz:45
2020-07-16T14:05:34  <kl_eisbaer1> 20200620/conference_osem.sql.xz:45
2020-07-16T14:05:34  <kl_eisbaer1> 20200621/conference_osem.sql.xz:45
2020-07-16T14:05:51  <henne> that would be one of the columns that should not be defined a char(0)
2020-07-16T14:06:02  <kl_eisbaer> so this is in all backups :-(
2020-07-16T14:06:23  <kl_eisbaer> which definitively means it's not "just" since the update
2020-07-16T14:06:33  <henne> when was the update?
2020-07-16T14:06:54  <kl_eisbaer> Last Thursday, 2020-07-09
2020-07-16T14:08:04  <kl_eisbaer> sorry, that this went unnoticed. But I wonder, why it just get's noticed now.
2020-07-16T14:08:16  <pjessen> any chance those are unused columns ?
2020-07-16T14:08:59  <pjessen> ie. columns kept for backeard compatibilty?  kooing at the dump, I think that might the case
2020-07-16T14:10:41  <henne> no
2020-07-16T14:11:57  <pjessen> hmm.
2020-07-16T14:12:05  <henne> well, if I'm the only one with this problem
2020-07-16T14:12:13  <henne> I'm just going to change them...
2020-07-16T14:12:41  <henne> and file it under 'whatever'
2020-07-16T14:12:43  <kl_eisbaer> :thumbsup:
2020-07-16T14:13:18  <pjessen> I guess those columns are empty anyway?
2020-07-16T14:14:35  <henne> nope
2020-07-16T14:17:03  <henne> if we update the mysql host I guess it has something to do with the large mariadb changeset in 10.2.2
2020-07-16T14:19:21  <pjessen> interesting - so you have columns char(0) that contain data ..... wow
2020-07-16T14:24:40  <kl_eisbaer> JFYI: we went 10.3.20 -> 10.4.13 with the galera (mariadb) upgrade
2020-07-16T14:46:51  <henne> hm okay, that's not it then
2020-07-16T14:50:25  <henne> phew
2020-07-16T14:50:30  <henne> I have some things to do...
2020-07-16T14:50:31  <henne> https://paste.opensuse.org/view/raw/21207344
2020-07-16T14:51:26  <henne> - is what it should be. + is what is in the DB
2020-07-16T15:25:32  <kl_eisbaer> isn't Ruby known to handle all the DB magic behind closed doors for developers? *hideandduck*
2020-07-16T15:27:47  <henne> au contraire mon ami :)
2020-07-16T16:30:34  <henne> alright. everything should be back to normal...
2020-07-16T17:33:26  <kl_eisbaer> Thanks, henne!
2020-07-16T18:35:35  <lcp> hm
2020-07-16T18:35:41  <lcp> lcp's back
2020-07-16T18:36:02  <lcp> and lcp has business to do >:D
2020-07-16T18:36:25  <lcp> henne: did you have the time to have a look at freenode stuff I asked you?
2020-07-16T18:38:55  <lcp> kl_eisbaer1: you might know, because I don't know why, but some software refuses to connect to pgsql server using the ip
2020-07-16T18:39:10  <lcp> I wonder if you ever seen anything like it
2020-07-16T18:40:27  <pjessen> lcp: port# ?
2020-07-16T18:43:20  <lcp> it's the same as localhost would have
2020-07-16T18:43:55  <lcp> I'm kinda wondering if the adapter just wants a url and not an ip
2020-07-16T18:43:58  <lcp> I can test that quite easily tbh
2020-07-16T18:54:59  <cboltz> no more lcp$randomnumber? ;-)
2020-07-16T19:00:22  <cboltz> pjessen: since you looked at "my" hostnames earlier to day - do you know the story behind "water"?
2020-07-16T19:13:56  <lcp> cboltz: depends on whether or not I break our matrix again >:D
2020-07-16T19:14:21  <cboltz> ;-)
2020-07-16T19:15:30  <cboltz> from what I've seen in your upstream ticket, the never-ending queries were caused by duplicate rows, possibly caused by importing a dump twice
2020-07-16T19:16:05  <cboltz> even if that counts as user error, I wonder if it would be possible to make some fields UNIQUE to prevent such things
2020-07-16T19:16:29  <cboltz> maybe have a look at the database structure if it would be an option, and then ask upstream for it?
2020-07-16T19:17:39  * cboltz did several changes in PostfixAdmin based on real-world user errors to prevent that someone else can repeat that
2020-07-16T20:20:17  <lcp> cboltz: maybe I will at a later date
2020-07-16T23:03:33  <malcolmlewis> lcp, hey, on the create new login page can a note be added if they have an old forum account NOT to use the same email....?
2020-07-16T23:04:33  <lcp> that's proxy stuff, I don't have access to that
2020-07-16T23:23:27  <malcolmlewis> lcp, no the self service portal?
2020-07-16T23:23:51  <malcolmlewis> else who looks after the proxy stuff?
2020-07-16T23:54:18  <lcp> I also don't have permission to self service portal
2020-07-16T23:54:33  <lcp> and you would have to get somebody from SUSE-IT for that