2020-04-16T02:39:12  *** okurz_ is now known as okurz
2020-04-16T04:40:11  -heroes-bot- PROBLEM: PSQL locks on mirrordb2.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 232 * total waiting locks: 112 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb2.infra.opensuse.org&service=PSQL%20locks
2020-04-16T04:44:55  -heroes-bot- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total waiting locks: 4 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-04-16T04:50:12  -heroes-bot- RECOVERY: PSQL locks on mirrordb2.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=3 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb2.infra.opensuse.org&service=PSQL%20locks
2020-04-16T04:54:54  -heroes-bot- RECOVERY: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=8 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-04-16T08:14:28  <lethliel> vinzv_1: Can you please have a look at https://github.com/openSUSE/news-o-o/pull/35? I did a small modification of the wording.
2020-04-16T08:23:38  *** Martchus_ is now known as Martchus
2020-04-16T08:32:57  *** ldevulder_ is now known as ldevulder
2020-04-16T10:39:30  <tuanpembual> cboltz, testing period for new progress is over. and all request/issue already resolved.
2020-04-16T10:49:42  <cboltz> I'm happy to hear that :-)
2020-04-16T10:51:04  <cboltz> so the next step is actually switching over
2020-04-16T10:51:14  <cboltz> we'll need some coordination for that:
2020-04-16T10:51:20  <cboltz> - you need a fresh database dump
2020-04-16T10:52:10  <cboltz> - old progress.o.o needs to be shut down (so that nobody can do changes after we take the final database dump)
2020-04-16T10:52:21  <cboltz> - mail routing (admin@ etc.) needs to be changed
2020-04-16T10:53:08  <cboltz> - and finally, the haproxy config needs to be adjusted to the new server
2020-04-16T10:53:43  <cboltz> so now the question is - when can we do this?
2020-04-16T10:56:08  <cboltz> (for me, 18:00 UTC or later would be a good time, but you'll also need someone (maybe Lars?) for the database dump - and to make things more interesting, 18:00 UTC is late night for you)
2020-04-16T11:00:50  <tuanpembual> It is fine for me at 18.00 UTC :D
2020-04-16T11:01:37  <tuanpembual> add lists: sync files folder, migrate db for fix utf8 issue.
2020-04-16T11:01:56  <cboltz> right
2020-04-16T11:02:56  <cboltz> kl_eisbaer and klein - is one of you available tonight 18:00 UTC / 20:00 CEST to provide an up-to-date database dump of progress.o.o?
2020-04-16T11:03:35  <kl_eisbaer> cboltz: an up-to date database dump is always on the new progress.i.o.o machine. Pushed there every day once the daily DB backup is done
2020-04-16T11:04:03  <cboltz> I know, and that's good enough for testing
2020-04-16T11:04:27  <tuanpembual> that mean, new progress will using new db on db cluster?
2020-04-16T11:04:45  <tuanpembual> because for now, new progress still using local mysql as db.
2020-04-16T11:05:10  <cboltz> ah, ok - I didn't have that on my radar
2020-04-16T11:05:29  <kl_eisbaer> tuanpembual: IMHO that's the plan, yes.
2020-04-16T11:05:46  <kl_eisbaer> I can setup a new DB on the cluster once you say "go ahead" ;-)
2020-04-16T11:06:28  <tuanpembual> I am ok with that plan.
2020-04-16T11:06:30  <kl_eisbaer> I just don't want to import the database dump from the old into the new DB automatically, as I think this should be done manually for now
2020-04-16T11:07:32  <cboltz> kl_eisbaer: I'd say "go ahead" ;-)
2020-04-16T11:07:35  <tuanpembual> go ahead
2020-04-16T11:07:43  <kl_eisbaer> omw
2020-04-16T11:08:42  <tuanpembual> but we need notify milis, lets say we will do migrate progress, so service will unavailable form 18.00 utc until 20.00 utc
2020-04-16T11:08:55  <cboltz> tuanpembual: just to be sure, do a last test when you get access to that database (different DB server could in theory cause a "surprise")
2020-04-16T11:09:30  <tuanpembual> okay, omw.
2020-04-16T11:12:16  <tuanpembual> last db dump we have, Apr 12 06:00. 5 hours late
2020-04-16T11:13:00  <cboltz> that's good enough to test that everything works on the postgres cluster
2020-04-16T11:13:23  <cboltz> but we'll need a last-minute dump when we switch over to the new progress
2020-04-16T11:15:51  <tuanpembual> sure.
2020-04-16T11:18:25  <kl_eisbaer> tuanpembual: done
2020-04-16T11:18:36  <kl_eisbaer> mysql -h 192.168.47.4 -P 3307 -u redmine -D redmine -p
2020-04-16T11:18:45  <kl_eisbaer> should work now from your progress.i.o.o machine
2020-04-16T11:19:10  <tuanpembual> oke, will check it.
2020-04-16T11:19:11  <kl_eisbaer> just use the PW in /opt/redmine/config/database.yml
2020-04-16T11:20:13  <kl_eisbaer> just keep in mind that you have to define the port 3307, as we do not use the default port 3306 at the moment
2020-04-16T11:21:03  <tuanpembual> I can login. Will try migrate to use that db with last dump.
2020-04-16T11:21:07  <tuanpembual> will be back
2020-04-16T11:28:55  <cboltz> somehow I thought that it's a postgres database, but mysql is even better - I'm not really familiar with postgres, but I know how to do a mysqldump ;-)
2020-04-16T11:29:30  <cboltz> kl_eisbaer: this also means we don't necessarily need you to switch over, but of course you are welcome to help if you want to ;-)
2020-04-16T11:30:15  <kl_eisbaer> I'm always happy to help - but at the moment, I'm even more happy to leave this up to you
2020-04-16T11:31:50  <kl_eisbaer> cboltz: and in regards to postgresl: sed -i "s|myslqdump|pg_dump|g" ;-)
2020-04-16T11:32:36  <cboltz> that's the easy part, but - do they really use exactly the same parameters?
2020-04-16T11:34:29  <tuanpembual> the crucial point is part of ichain,
2020-04-16T11:34:51  <tuanpembual> it need the config from haproxy
2020-04-16T11:35:06  <tuanpembual> change from progress-test to progress.
2020-04-16T11:35:20  <cboltz> that's something I can do
2020-04-16T11:35:31  <tuanpembual> if we forget, user will cannot login.
2020-04-16T11:35:47  <tuanpembual> I see, i will be more easier.
2020-04-16T11:37:38  <tuanpembual> migrate is done.
2020-04-16T11:45:03  <cboltz> :-)
2020-04-16T11:45:31  <tuanpembual> clean local mysql done.
2020-04-16T11:52:55  <tuanpembual> All is good from me. Will be back at 18.00 UTC
2020-04-16T11:53:17  <cboltz> ok, see you later!
2020-04-16T11:59:40  -heroes-bot- PROBLEM: MySQL backup on mybackup.infra.opensuse.org - CRITICAL: No results from backup job ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mybackup.infra.opensuse.org&service=MySQL%20backup
2020-04-16T12:08:03  <cboltz> kl_eisbaer: ^^^ - can you please have a look at mybackup?
2020-04-16T13:15:18  <lcp> gitlab's down?
2020-04-16T13:43:14  <lcp> kl_eisbaer: I'm thinking we should just move freeipa to freeipa2 when all is done all things considered (and then move back to freeipa when we can run freeipa on Leap in a year or two)
2020-04-16T13:43:33  <lcp> kl_eisbaer: in any case, firefox will complain about cert issuer if we don't sync up both ipas
2020-04-16T13:44:47  <lcp> obviously the implication is that admin and user logins would merge into one thing, and I don't know if that's something that you deem a bad thing ;)
2020-04-16T13:59:56  <lcp> (we can sync up without merging tho)
2020-04-16T14:26:12  <lcp> cboltz: can I request permissions to openSUSE/freeipa?
2020-04-16T14:27:05  <lcp> (since I now maintain the SUSE ipaplatform upstream)
2020-04-16T15:16:42  <kl_eisbaer> mybackup => 2020-04-16 15:16:14 0 [Warning] mysqld: Disk is full writing '/var/lib/mysql-files/slave-bin/slave-bin.~rec~' (Errcode: 28 "No space left on device").
2020-04-16T15:17:27  <kl_eisbaer> so there is just some other monitoring missing....
2020-04-16T15:28:54  <lcp> cboltz: what is the preference for static.opensuse.org, separate repo with submodule or just adding to the existing repo
2020-04-16T15:58:12  -heroes-bot- PROBLEM: MySQL WSREP recv on galera1.infra.opensuse.org - CRIT wsrep_local_recv_queue_avg = 22.920000 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=galera1.infra.opensuse.org&service=MySQL%20WSREP%20recv
2020-04-16T17:12:58  <lcp> cboltz: up to change haproxy and srv for matrix again? >:D
2020-04-16T17:15:15  <lcp> cboltz: I will just add a reverse proxy from 8008 to 80 so it will be easier to setup with haproxy, and srv so it doesn't redirect to 8448 but to 443
2020-04-16T17:20:11  <cboltz> lcp: for static.o.o, just use the main repo - git submodules are annoying ;-)
2020-04-16T17:34:30  <lethliel> cboltz: should the gitfs on minnie now be updated?
2020-04-16T17:34:56  <cboltz> yes, should work now
2020-04-16T17:37:17  <lethliel> ok. I see that meet_infra_opensuse_org.sls is there Oo
2020-04-16T17:38:07  <lethliel> Specified SLS 'id.meet_infra_opensuse_org' in environment 'production' is not available on the salt master
2020-04-16T17:38:10  <lethliel> strange
2020-04-16T17:40:30  <cboltz> saltmaster log shows that there was some "fun" with fetching gitfs from gitlab
2020-04-16T17:40:51  <cboltz> I just triggered an update manually, can you try again?
2020-04-16T17:42:39  <cboltz> (as a sidenote, the saltmaster log shows a similar failure for id.mybackup-opensuse_infra_opensuse_org - but in this case we really have a wrong pillar/id/ filename)
2020-04-16T17:43:25  <lcp> cboltz: submodules are great, if your repo is very branching and you don't want the core repo to be a pain to download
2020-04-16T17:43:46  <lcp> but at this point this doesn't apply to static yet 😀
2020-04-16T17:45:35  <lethliel> cboltz: still the same error
2020-04-16T17:55:53  <tuanpembual> hi cboltz
2020-04-16T17:55:57  <cboltz> lethliel: another attempt - I just restarted saltmaster
2020-04-16T17:55:59  <cboltz> hi tuanpembual
2020-04-16T17:56:03  <tuanpembual> i am online. :D
2020-04-16T17:56:29  <tuanpembual> we start now, or still waiting someone..
2020-04-16T17:56:47  <cboltz> I think we can start ;-)
2020-04-16T17:57:11  <cboltz> before taking the final database dump, we (actually I) need to
2020-04-16T17:57:33  <cboltz> - change the postfix transport map to the new server (+ a backup address to prevent mail loss)
2020-04-16T17:57:45  <cboltz> - shutdown current progress so that nobody can change anything there
2020-04-16T17:57:53  <cboltz> - write a notice on status.o.o
2020-04-16T17:58:01  <cboltz> did I forget anything?
2020-04-16T17:58:29  <cboltz> also, can you please stop postfix on the new server so that delivery of incoming mails is delayed until we have the database moved?
2020-04-16T17:58:32  <tuanpembual> I prefer stop redmine service only at old progress
2020-04-16T17:58:51  <tuanpembual> I will stop postfix.
2020-04-16T17:58:58  <cboltz> right, I was a bit unclear about that ;-) - I'll only stop the service, not the whole VM
2020-04-16T17:59:43  <tuanpembual> postfix stopped.
2020-04-16T18:00:37  <lethliel> cboltz: did not work. But now I restarted the minion on meet.o.o and it seems to work Oo
2020-04-16T18:01:07  <lethliel> tuanpembual, cboltz and good luck by whatever you are doing ;-)
2020-04-16T18:01:23  <cboltz> lethliel: maybe the minion had the error cached
2020-04-16T18:01:25  <tuanpembual> thanks lethliel
2020-04-16T18:06:50  <cboltz> should have run postmap before sending a test mail/ticket ;-)
2020-04-16T18:08:00  <cboltz> hmm, looks like postfix didn't pick up the new virtual.redmine fast enough...
2020-04-16T18:08:46  <tuanpembual> run now?
2020-04-16T18:13:13  <cboltz> no, interestingly the mails/tickets still arrive on the old progress :-/
2020-04-16T18:14:46  <tuanpembual> maybe related because I stopped new redmine?
2020-04-16T18:15:04  <cboltz> no
2020-04-16T18:15:06  <tuanpembual> + postfix?
2020-04-16T18:15:13  <cboltz> I think I found it - admin@ gets routed via lists.o.o
2020-04-16T18:16:11  <tuanpembual> good
2020-04-16T18:16:22  <cboltz> I'll change that the hard way by editing the subscribers file on lists.o.o - not nice, but should work ;-)
2020-04-16T18:17:19  <tuanpembual> sometime hard way is accepted :)
2020-04-16T18:18:25  <cboltz> ah, that worked :-)
2020-04-16T18:20:01  <cboltz> mails now get queued (and I get a backup copy, just in case)
2020-04-16T18:20:10  <cboltz> I also shut down the webserver on old progress
2020-04-16T18:20:24  <cboltz> time to get the database dump ;-)
2020-04-16T18:20:46  <tuanpembual> you will send dump to new progress, correct?
2020-04-16T18:22:23  <cboltz> yes, give me a minute ;-)
2020-04-16T18:27:10  -heroes-bot- PROBLEM: HTTP progress on redmine.infra.opensuse.org - connect to address 192.168.47.29 and port 80: Connection refused ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=redmine.infra.opensuse.org&service=HTTP%20progress
2020-04-16T18:28:48  <cboltz>  /home/cboltz/opensuse_redmine.sql
2020-04-16T18:30:07  <tuanpembual> omw
2020-04-16T18:30:25  <tuanpembual> btw, we need sync files from old redmine
2020-04-16T18:30:35  <cboltz> yes, I'm already working on that
2020-04-16T18:33:01  <cboltz>  /home/cboltz/redmine-files.tar.gz
2020-04-16T18:34:16  <cboltz> should be 167 MB (du -h after unpacking)
2020-04-16T18:36:15  <tuanpembual> noted
2020-04-16T18:42:06  <tuanpembual> found error on migrate plugin.
2020-04-16T18:42:21  <tuanpembual> and not happen before :D
2020-04-16T18:42:25  <cboltz> nice :-/
2020-04-16T18:43:50  <cboltz> as a sidenote - I just noticed that my sqldump includes    USE `opensuse_redmine`;   which is the old database
2020-04-16T18:44:01  <cboltz> I hope this didn't cause too much trouble ;-)
2020-04-16T18:45:39  <tuanpembual> I found clue
2020-04-16T18:45:42  <tuanpembual> wait
2020-04-16T18:48:17  <tuanpembual> redmine started
2020-04-16T18:48:27  <tuanpembual> how about ha proxy?
2020-04-16T18:48:54  <tuanpembual> but still 503.
2020-04-16T18:49:23  <cboltz> I'll adjust haproxy now
2020-04-16T18:52:31  <cboltz> done, progress.o.o is available again (using the new server)
2020-04-16T18:53:12  <cboltz> don't forget to start postfix again ;-)
2020-04-16T18:53:36  <tuanpembual> postfix started
2020-04-16T18:53:59  <tuanpembual> I can loging
2020-04-16T18:54:02  <tuanpembual> :D
2020-04-16T18:54:22  <cboltz> yes, looks like everything works
2020-04-16T18:55:31  <cboltz> the queued mail also arrived as a ticket :-)
2020-04-16T18:55:57  <tuanpembual> looks good.
2020-04-16T18:57:42  <cboltz> a little detail - the mails get sent as redmine@o.o instead of admin@o.o
2020-04-16T18:57:50  <cboltz> can you change that for the admin project?
2020-04-16T18:57:56  <tuanpembual> okay
2020-04-16T18:58:21  <tuanpembual> done
2020-04-16T19:00:18  <tuanpembual> do you receive email for update ticket?
2020-04-16T19:00:24  <cboltz> yes
2020-04-16T19:00:38  <tuanpembual> cool.
2020-04-16T19:01:43  <cboltz> (and the sender address is admin@ now :-)
2020-04-16T19:02:34  <cboltz> I think you can close the test tickets ;-)
2020-04-16T19:03:20  <tuanpembual> sure. I update my tickets regarding redmine :D
2020-04-16T19:04:22  <cboltz>  /opt/redmine/files looks smaller than on the old server (only 143 MB, according to du -h) - did you update the files?
2020-04-16T19:04:57  <tuanpembual> my bad, I forgot.
2020-04-16T19:05:08  <tuanpembual> sorry
2020-04-16T19:05:10  <cboltz> no problem, the tarball is still there ;-)
2020-04-16T19:06:58  <tuanpembual> done
2020-04-16T19:07:53  <cboltz> thanks!
2020-04-16T19:08:03  <kl_eisbaer> guys, you rock!
2020-04-16T19:08:12  <cboltz> looks like there are only two tasks left:
2020-04-16T19:08:24  <cboltz> - update status.o.o to "all green"
2020-04-16T19:08:25  * kl_eisbaer is sad that his excuse for not doing tickets this evening is now gone... ;-)
2020-04-16T19:08:32  <cboltz> (I'll do that now)
2020-04-16T19:09:00  <cboltz> - write an announcement about the update - tuanpembual, I'll let that honor to you ;-)
2020-04-16T19:09:31  <tuanpembual> Finally, done :D
2020-04-16T19:09:44  <tuanpembual> almost 2 years.
2020-04-16T19:09:50  <okurz> ok, so I'm obviously not the first one anymore but still let me say: Thanks a ton! :)
2020-04-16T19:10:23  <tuanpembual> you are welcome
2020-04-16T19:10:51  <okurz> the first bad thing about the new redmine version – but maybe it's a custom stylesheet – is the more wasteful spacing on the screen. Is this coming from upstream?
2020-04-16T19:11:27  <okurz> oh, and there is one thing that seems to have been maybe lost? favicon is no friendly geeko!!!1
2020-04-16T19:11:49  <tuanpembual> it is custom themes
2020-04-16T19:12:10  <tuanpembual> I try using gecko logo, but my css skill is very bad
2020-04-16T19:12:11  <cboltz> I can inject the favicon via haproxy ;-)
2020-04-16T19:12:25  <tuanpembual> nice.
2020-04-16T19:12:53  <tuanpembual> github.com/tuanpembual/redmine-opensuse-theme/ here the repo.
2020-04-16T19:13:27  <lcp> adding geeko is fairly easy with css, I also looked into how exactly to implement theming
2020-04-16T19:13:34  <lcp> it will be kinda hard because it's js galore in many parts
2020-04-16T19:13:37  <cboltz> actually it's not that easy because the HTML references a custom favicon path
2020-04-16T19:14:06  <tuanpembual> for favicon, I can do it
2020-04-16T19:14:11  <cboltz> tuanpembual: please   wget https://static.opensuse.org/favicon.ico   and place it as  /themes/redmine-opensuse-theme/favicon/
2020-04-16T19:16:42  <tuanpembual> done.
2020-04-16T19:16:55  <cboltz> thanks!
2020-04-16T19:17:02  <cboltz> just noticed we have a progress-staging.opensuse.org that redirects to itsself in an endless loop - I guess we can simply drop that in haproxy, login proxy and DNS ;-)
2020-04-16T19:17:28  <okurz> much better favicon, thanks :)
2020-04-16T19:19:14  <tuanpembual> cboltz you will write an announcement. is it?
2020-04-16T19:20:03  <cboltz> no - you did all the work, so it should be you who writes it and gets the honor ;-)
2020-04-16T19:20:22  <cboltz> (if you want, I can review it before you send it out)
2020-04-16T19:21:01  <tuanpembual> oke, try to write.
2020-04-16T19:22:15  <cboltz> kl_eisbaer:   csync2 -xv   on anna tells me for some files "File stays in dirty state. Try again later..." and doesn't sync them. Any idea why, and how to fix it?
2020-04-16T19:29:25  <lcp> cboltz: willing to setup haproxy/srv for matrix today, got the homeserver pretty much going
2020-04-16T19:30:15  <lcp> I should probably note what changed in the issue
2020-04-16T19:30:21  <cboltz> yes, no problem
2020-04-16T19:30:28  <cboltz> just tell me what you need ;-)
2020-04-16T19:31:02  <lcp> there is also haproxy setup listed in synapse doc, maybe that helps https://github.com/matrix-org/synapse/blob/master/docs/reverse_proxy.md#haproxy
2020-04-16T19:32:27  <tuanpembual> https://etherpad.opensuse.org/p/progress_upgrade cboltz
2020-04-16T19:32:34  <lcp> I pretty much need what is in the doc
2020-04-16T19:32:41  <bmwiedemann1> Hi. Where can one update opensuse.org email forwardings? there are still some novell.com ones that stopped working.
2020-04-16T19:32:43  <tuanpembual> but not really good paragraf
2020-04-16T19:33:21  <cboltz> bmwiedemann1: that's done on connect.o.o - do you have admin permissions there (in the web interface)?
2020-04-16T19:38:22  <cboltz> lcp:   acl matrix-path path_beg /_matrix   matches paths starting with "_matrix", but probably independent of the domain - do you really need that?
2020-04-16T19:39:15  <cboltz> (that rule would for example match en.o.o/_matrix_foo ;-)
2020-04-16T19:39:35  <lcp> it should only be for matrix.o.o really
2020-04-16T19:39:56  <lcp> I don't need wikis to connect to matrix I don't think ;)
2020-04-16T19:40:00  <cboltz> ok, then the first rule with   hdr(host)   is enough ;-)
2020-04-16T19:42:49  <lcp> tuanpembual: the previous version was from 2014, right?
2020-04-16T19:43:06  <lcp> trying to make that note way too good I think
2020-04-16T19:43:19  <tuanpembual> I am not really sure.
2020-04-16T19:43:21  <bmwiedemann1> cboltz: only got freeipa groups assigned. not sure about connect.
2020-04-16T19:44:22  <cboltz> lcp: yes, the README.rdoc and some other files on the old redmine server is from 2014
2020-04-16T19:44:41  <tuanpembual> 6 years ago
2020-04-16T19:44:48  <cboltz> bmwiedemann1: how many aliases do you need to change?
2020-04-16T19:49:10  <tuanpembual> cboltz where we sent that announcement?
2020-04-16T19:49:48  <cboltz> I'd say opensuse-project@ with heroes@ in CC
2020-04-16T19:50:16  <cboltz> you can also write an article on progress (in the admin project) which then gets shown on planet.o.o
2020-04-16T19:53:28  <tuanpembual> tabs https://progress.opensuse.org/projects/opensuse-admin/activity some like broken.
2020-04-16T19:53:31  <tuanpembual> :/
2020-04-16T19:53:36  <tuanpembual> will check the log
2020-04-16T19:56:01  <bmwiedemann1> cboltz: so far I know of DDemaio
2020-04-16T19:56:18  <cboltz> yes, I've seen his ticket
2020-04-16T19:56:29  <bmwiedemann1> but there could be a hundred.
2020-04-16T19:57:18  <cboltz> s/could be/are/ - https://progress.opensuse.org/issues/64577
2020-04-16T19:58:13  <cboltz> in other news: you convinced me to give you admin permissions on connect ;-)
2020-04-16T19:59:03  <cboltz> quick howto: on the user's page (for example https://connect.opensuse.org/pg/profile/bmwiedemann ) click "Edit profile"
2020-04-16T19:59:24  <cboltz> then Ctrl-F "alias" and change the "opensuse.org email alias (target)" field
2020-04-16T19:59:34  <bmwiedemann1> that was fast level-up :-)
2020-04-16T20:00:10  -heroes-bot- RECOVERY: HAProxy on provo-proxy1.infra.opensuse.org - HAPROXY OK -  debuginfod (Active: 1/1) debuginfod-nginx (Active: 1/1) static (Active: 1/1) ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=provo-proxy1.infra.opensuse.org&service=HAProxy
2020-04-16T20:00:21  <cboltz> well, the price tag is that you may ;-) now fix quite some aliases that still point to @novell.com
2020-04-16T20:01:10  <cboltz> FYI: please keep the "Public" setting for all 3 "opensuse.org email alias" fields - setting it to something else will break the alias
2020-04-16T20:04:31  <bmwiedemann1> OK. fixed ddemaio now. Maybe easier to search in DB directly than waiting for tickets...
2020-04-16T20:05:09  <cboltz> if you know working addresses for the affected people, that's more than welcome ;-)
2020-04-16T20:06:45  <cboltz> do you know how to access the database?
2020-04-16T20:10:17  <bmwiedemann1> at least for the @novell.com addrs I would try @suse.com if they still work here.
2020-04-16T20:10:30  <bmwiedemann1> on what host is the DB?
2020-04-16T20:10:50  <bmwiedemann1> or maybe I can connect from connect.o.o there.
2020-04-16T20:11:39  <bmwiedemann1> hmm. I'm not in sudoers there.
2020-04-16T20:12:07  <cboltz> it's a SLE 11, don't even think about sudo ;-)
2020-04-16T20:13:50  <bmwiedemann1> oh, so that is why we want to get rid of it? I think, dup to SLE12 was not so easy either. needed booting off DVD with upgrade=1
2020-04-16T20:14:57  <lcp> bmwiedemann1: the hope is that we can use the new login system as a replacement for it, since it can pretty much do what we need in connect
2020-04-16T20:15:33  <lcp> and also noggin shows nice profiles for users so we aren't losing much there
2020-04-16T20:16:32  <lcp> I'm waiting for the vms to get that thing going too :/
2020-04-16T20:17:09  <cboltz> bmwiedemann1: see PM for database access
2020-04-16T20:18:38  <lcp> cboltz: hm, did you do the haproxy thing?
2020-04-16T20:19:36  <cboltz> I've seen too many things, but now I'll get back to haproxy ;-)
2020-04-16T20:20:10  <lcp> righty-o
2020-04-16T20:20:57  <lcp> the login system will be a great chance to get rid of the last pieces of infra cruft like connect and the old freeipa server
2020-04-16T20:21:23  <cboltz> https://paste.opensuse.org/82e1b596 is our current haproxy config for matrix
2020-04-16T20:21:36  <lcp> and also my own excuse to get freeipa ported to openSUSE
2020-04-16T20:22:05  <cboltz> so chat.o.o and dimension.o.o 443 get routed to matrix.i.o.o 80
2020-04-16T20:22:36  <cboltz> and matrix.o.o 443 gets routed to matrix.i.o.o 8008
2020-04-16T20:24:22  <lcp> that's good, I guess 8448 is waiting for its chance to shine later ;)
2020-04-16T20:24:41  <cboltz> right, that's the missing part ;-)
2020-04-16T20:25:20  <lcp> I still don't know what that 405 thing is about, I should probably take a look at it
2020-04-16T20:26:00  <cboltz> yes, might make sense ;-)
2020-04-16T20:26:46  <cboltz> regarding 8448 - our "bind ..." lines are slightly more interesting than the example, therefore I'd prefer if someone who knows haproxxy better does that
2020-04-16T20:30:05  <lcp> alright
2020-04-16T20:30:28  <lcp> it seems 405 haven't been happening since 11. April, at least not according to the access log
2020-04-16T20:37:31  <lcp> cboltz: hm, I wonder if the firewall isn't blocking requests there
2020-04-16T20:37:42  <lcp> for 8008 specifically
2020-04-16T20:38:20  <cboltz> the haproxy config is   external:443 -> matrix.i.o.o:8008
2020-04-16T20:38:50  <cboltz> (with "external" == matrix.o.o)
2020-04-16T20:40:31  <cboltz> http://matrix.infra.opensuse.org:8008/ says "connection failed" - please check if something is listening there
2020-04-16T20:42:39  * lcp sent a long message:  < https://matrix.org/_matrix/media/r0/download/matrix.org/hFluUTamcKbBNHoQoWijNXOu >
2020-04-16T20:42:45  <lcp> right
2020-04-16T20:44:01  <cboltz> curl -v localhost:8008   on matrix.i.o.o seems to work
2020-04-16T20:44:07  <cboltz> maybe it's only listening on localhost?
2020-04-16T20:44:51  <cboltz> anyway - is there a reason not to route this via nginx like dimension.o.o?
2020-04-16T20:45:07  <lcp> `bind_addresses: ['::1', '127.0.0.1']`, that's bizarre
2020-04-16T20:45:13  <lcp> I could do that, sure
2020-04-16T20:46:52  <lcp> it is on 80 now
2020-04-16T20:47:51  <tuanpembual> email sent.
2020-04-16T20:47:57  <tuanpembual> need back to sleep.
2020-04-16T20:48:08  <tuanpembual> thank you cboltz lcp
2020-04-16T20:48:42  <cboltz> we have to thank you ;-)
2020-04-16T20:49:39  <lcp> no worries, the work you did is very important
2020-04-16T20:49:50  <lcp> cboltz: yup, logged in
2020-04-16T20:49:54  <lcp> matrix works
2020-04-16T20:52:03  * lcp uploaded an image: Screenshot from 2020-04-16 22-51-41.png (75KB) < https://matrix.org/_matrix/media/r0/download/matrix.org/mjIvSrGNLhXoDMbcLgxFznWJ >
2020-04-16T20:54:41  <cboltz> matrix.opensuse.org gives a 404 - not too surprising since you only proxy_pass   /_matrix
2020-04-16T20:55:12  <cboltz> (as a sidenote - IIRC you wanted to have separate files for each vhost ;-)
2020-04-16T20:56:04  <lcp> I will do that, except gitlab doesn't load for me for some reason
2020-04-16T20:56:24  <cboltz> strange, works here
2020-04-16T20:59:29  <lcp> yeah, no idea why
2020-04-16T21:00:02  <lcp> in any case, I think we will wait with making this instance official until account system, so we can connect it to that ldap server or to saml idp
2020-04-16T21:00:53  <lcp> but it already works very well all things considered, better than I expected :P
2020-04-16T21:01:00  <cboltz> :-)
2020-04-16T21:01:20  <cboltz> speaking about auth stuff - any progress on the forums login?
2020-04-16T21:06:19  <lcp> cboltz: not any more than before, I would love to get those two fedora vms I requested so we can just setup noggin and ipsilon (since we underestimated the amount of effort required for porting)
2020-04-16T21:07:28  <lcp> now that we got a reality check, the way to go is to setup where we know those things work, we will handle the Leap version in a year or two, because then we can make sure everything works without any issues
2020-04-16T21:09:01  <cboltz> sounds like hoping to get ?noggin and ipsilon running on openSUSE within a few weeks was a bit too optimistic ;-)
2020-04-16T21:09:50  <lcp> well, it sure was, but at the start of it we weren't expecting to have only few weeks :P
2020-04-16T21:10:01  <lcp> the last meeting was quite a shock tbh
2020-04-16T21:10:11  <cboltz> yes, I can imagine
2020-04-16T21:10:25  <cboltz> actually I wonder if getting everything done on fedora VMs is also optimistic ;-) for that short time frame
2020-04-16T21:11:18  <lcp> well, we don't have to reinvent openSUSE for that to work, quite the opposite from what we would have to do to get them going there
2020-04-16T21:12:27  <lcp> it is quite shocking how little stuff we have actually in the repos related to this stuff too, so yeah, fedora is a way better idea
2020-04-16T21:13:09  <cboltz> still, setting up the servers and migrating all user accounts within a few weeks doesn't sound too easy to me
2020-04-16T21:13:21  <cboltz> even if you use a known-working distro as base
2020-04-16T21:14:52  <lcp> it won't be easy, for sure
2020-04-16T21:15:40  <lcp> bmwiedemann1: does SUSE have some tooling for importing the accounts ready? do you know schema of the dumps you are gonna be importing?
2020-04-16T21:21:40  <lcp> cboltz: I mean, the worst case is we are using SUSE accounts for a while before we figure out our own, that's not the worst alternative
2020-04-16T21:22:10  <lcp> that does cut out some people from using openid for a while, but nothing we can do if we don't manage
2020-04-16T21:22:42  <cboltz> right, that wouldn't be the end of the world ;-)
2020-04-16T21:30:03  <lcp> kl_eisbaer: trust me, if I thought we had enough time to implement using Leap, we would do it
2020-04-16T21:59:07  <lcp> cboltz: I will leave the federation port until we have the accounts figured out tho