2020-02-06T00:05:04  -heroes-bot- RECOVERY: PostgreSQL standby on mirrordb1.infra.opensuse.org - POSTGRES_HOT_STANDBY_DELAY OK: DB mb_opensuse2 (host:mirrordb2) 868592 and 0 seconds ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PostgreSQL%20standby
2020-02-06T00:39:03  -heroes-bot- PROBLEM: PostgreSQL standby on mirrordb1.infra.opensuse.org - POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB mb_opensuse2 (host:mirrordb2) 294830088 and 9 seconds ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PostgreSQL%20standby
2020-02-06T01:16:06  -heroes-bot- PROBLEM: MySQL WSREP recv on galera1.infra.opensuse.org - CRIT wsrep_local_recv_queue_avg = 22.344455 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=galera1.infra.opensuse.org&service=MySQL%20WSREP%20recv
2020-02-06T02:15:56  -heroes-bot- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 216 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-02-06T02:43:20  -heroes-bot- PROBLEM: HTTP l10n on nuka.infra.opensuse.org - CRITICAL - Socket timeout after 10 seconds ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=nuka.infra.opensuse.org&service=HTTP%20l10n
2020-02-06T02:45:56  -heroes-bot- RECOVERY: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=82 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-02-06T03:13:11  -heroes-bot- RECOVERY: HTTP l10n on nuka.infra.opensuse.org - HTTP OK: HTTP/1.1 200 OK - 41228 bytes in 0.928 second response time ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=nuka.infra.opensuse.org&service=HTTP%20l10n
2020-02-06T03:39:14  *** okurz_ is now known as okurz
2020-02-06T05:37:21  -heroes-bot- PROBLEM: HTTP l10n on nuka.infra.opensuse.org - CRITICAL - Socket timeout after 10 seconds ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=nuka.infra.opensuse.org&service=HTTP%20l10n
2020-02-06T09:02:49  *** ldevulder_ is now known as ldevulder
2020-02-06T10:37:03  -heroes-bot- PROBLEM: PostgreSQL standby on mirrordb1.infra.opensuse.org - POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB mb_opensuse2 (host:mirrordb2) 482050192 and 50 seconds ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PostgreSQL%20standby
2020-02-06T10:47:03  -heroes-bot- RECOVERY: PostgreSQL standby on mirrordb1.infra.opensuse.org - POSTGRES_HOT_STANDBY_DELAY OK: DB mb_opensuse2 (host:mirrordb2) 332528 and 0 seconds ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PostgreSQL%20standby
2020-02-06T13:22:15  <lcp> cboltz: hm, how should I go about getting packages into oS:infra on obs?
2020-02-06T13:22:25  <lcp> kinda wanna get matrix going
2020-02-06T13:27:14  *** paka is now known as Guest48959
2020-02-06T13:28:10  *** paka_ is now known as Guest3113
2020-02-06T13:36:34  <malcolmlewis> kl_eisbaer, are you in a position to integrate SSO for the forum software (at this point vB) into the openSUSE infrastructure, or is this a no-go in the short term?
2020-02-06T13:48:49  <malcolmlewis> adrianS, ^^ you were the other person mentioned that may be able to help with SSO?
2020-02-06T13:53:03  <adrianS> malcolmlewis: we can take care .... you are aware that our services are behind a so called "Login proxy" ?
2020-02-06T13:53:24  <adrianS> means the service is not getting the password, just has to check for the username in http header
2020-02-06T13:53:29  <adrianS> are you prepared for that?
2020-02-06T13:56:35  <lcp> adrianS: would OBS be able to support SSO solution?
2020-02-06T13:57:04  <lcp> (I haven't looked into OBS from that pov tbh)
2020-02-06T14:05:00  <adrianS> the login proxy is so far the company SSO solution
2020-02-06T14:05:08  <adrianS> saml is on the roadmap, but not there
2020-02-06T14:11:11  <Eighth_Doctor> adrianS: I'm hoping to stand up an Ipsilon instance (https://ipsilon-project.org/) in openSUSE infra sometime soon for testing, so it'd be nice if OBS could use either OpenID Connect (preferred) or SAML 2.0 (eh, less preferred, but works)
2020-02-06T14:11:47  <Eighth_Doctor> adrianS: main advantage of OIDC is that there are flows for supporting CLI use cases
2020-02-06T14:12:27  <Eighth_Doctor> and Fedora has some Python code for doing this with CLI tools since fedpkg(1) can authenticate this way
2020-02-06T14:15:12  <malcolmlewis> adrianS, so it might be better to use the vB login mechanism then?
2020-02-06T14:17:08  <lcp> Conan Kudo: not to mention, SSO limits the issues we have with the current infra where logging into one service logs out in the other
2020-02-06T14:17:26  <Eighth_Doctor> yes, that's incredibly annoying
2020-02-06T14:17:48  <Eighth_Doctor> adrianS: I strongly request you consider using OpenID Connect instead of SAML 2.0, though
2020-02-06T14:18:31  <Eighth_Doctor> OmniAuth rubygem supports this with the `omniauth-openid-connect` gem
2020-02-06T14:18:47  <lcp> synapse only has support for saml, so that's what we are gonna use when ipsilon happens, for now it's gonna be ldap
2020-02-06T14:19:02  <Eighth_Doctor> LCP: Ipsilon can do both simultaneously
2020-02-06T14:19:19  <Eighth_Doctor> most Fedora services use OIDC or FAS-OAuth2, but RHBZ uses SAML 2.0
2020-02-06T14:19:27  <lcp> yeah, I get it, I'm just saying what it supports
2020-02-06T14:19:31  <Eighth_Doctor> sure
2020-02-06T14:19:57  <lcp> https://github.com/matrix-org/synapse/blob/master/docs/saml_mapping_providers.md
2020-02-06T14:21:28  <malcolmlewis> lcp, adrianS at this point all (from a Forums perspective) is a definitive answer on what will be used, then can co-ordinate with MF to get the user data for integration
2020-02-06T14:21:52  <Eighth_Doctor> oh god
2020-02-06T14:22:00  <Eighth_Doctor> they did saml the braindead way
2020-02-06T14:22:30  <malcolmlewis> in saying that, we don't want to introduce something that locks us in going forward, if that make sense....
2020-02-06T14:23:36  <lcp> I mean, it was already annoying a year ago, now it's getting ridiculous
2020-02-06T14:24:17  <lcp> Conan Kudo: you can always suggest improvements
2020-02-06T14:24:56  <Eighth_Doctor> LCP: I'll have to look at it at some point
2020-02-06T14:25:14  <Eighth_Doctor> I was hoping it'd be easy to add oidc support, but if they did saml this way... :(
2020-02-06T14:26:11  <lcp> Conan Kudo: https://github.com/matrix-org/synapse/issues/6182
2020-02-06T14:26:26  <lcp> and there is a protocol implementation of jwt
2020-02-06T14:26:36  <lcp> https://github.com/matrix-org/synapse/issues/1504
2020-02-06T14:26:45  <lcp> although doesn't seem to be functional in synapse
2020-02-06T14:29:29  <lcp> Conan Kudo: ah, also worth noting https://github.com/matrix-org/synapse/blob/master/docs/password_auth_providers.md
2020-02-06T14:29:37  <lcp> that's how ldap plugin is implemented
2020-02-06T14:29:55  <lcp> you don't actually need to have much code for this, iirc ldap plugin is under 1000 lines long
2020-02-06T14:30:23  <lcp> 527 lines
2020-02-06T14:30:42  <lcp> https://github.com/matrix-org/matrix-synapse-ldap3/blob/master/ldap_auth_provider.py
2020-02-06T14:31:35  <Eighth_Doctor> hmm, that's not terrible
2020-02-06T14:42:00  <adrianS> malcolmlewis: MF is not involved here anymore
2020-02-06T14:44:43  <malcolmlewis> adrianS, not sure what you mean? When I say MF, I mean getting our Forum data (as in Forum content) and then we need to get user data over to openSUSE infra
2020-02-06T14:47:05  <adrianS> ah, thought you mean MicroFocus IT dep
2020-02-06T14:47:28  <adrianS> okay, you need them to the content, tru
2020-02-06T14:47:30  <adrianS> true
2020-02-06T14:47:42  <adrianS> sorry, to many parallel tasks
2020-02-06T14:49:06  <malcolmlewis> adrianS, :) so we have 40K Forum users/passwords how can we integrate this into openSUSE infra and pair this up with the vB content...
2020-02-06T14:50:01  <adrianS> the passwords are not in the forum software I hope
2020-02-06T14:50:10  <adrianS> I think they use also the classic login proxy
2020-02-06T14:50:25  <adrianS> we have basically the same proxy here, just apache based
2020-02-06T14:50:29  <adrianS> but you get the same headers
2020-02-06T14:50:41  <adrianS> so theoretically their software should work in the same way here
2020-02-06T14:50:42  <malcolmlewis> adrianS, no, MF Access Gateway
2020-02-06T14:50:54  <malcolmlewis> adrianS, SAML
2020-02-06T14:51:06  <adrianS> hm, that is a later step ....
2020-02-06T14:51:26  <adrianS> but I can check with our SUSE IT people who are building up the new account server
2020-02-06T14:51:36  <adrianS> they want saml also, but it is not their 1st prio
2020-02-06T14:51:55  <malcolmlewis> adrianS, so in the first instance I need to ask MF for the usernames/passwords in some format?
2020-02-06T14:52:23  <adrianS> they will not hand them over for good reasons
2020-02-06T14:52:31  <adrianS> at least if you are using the same database
2020-02-06T14:53:09  <malcolmlewis> adrianS, AFAIK they are, sans and MF emails (which is crazy)
2020-02-06T14:53:13  <malcolmlewis> and/any
2020-02-06T14:53:49  <malcolmlewis> adrianS, but the passwords are not in vB
2020-02-06T14:55:05  <malcolmlewis> all authentication is through MF
2020-02-06T15:15:03  -heroes-bot- PROBLEM: PostgreSQL standby on mirrordb1.infra.opensuse.org - POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB mb_opensuse2 (host:mirrordb2) 1555517008 and 276 seconds ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PostgreSQL%20standby
2020-02-06T15:35:03  -heroes-bot- RECOVERY: PostgreSQL standby on mirrordb1.infra.opensuse.org - POSTGRES_HOT_STANDBY_DELAY OK: DB mb_opensuse2 (host:mirrordb2) 270784 and 1 seconds ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PostgreSQL%20standby
2020-02-06T16:01:47  <adrianS> malcolmlewis: point is that the MF authentification will go away
2020-02-06T16:02:27  <adrianS> we build up currently a new one esp. for the build service, but we want actually also move as may opensuse services as well
2020-02-06T16:03:16  <adrianS> all services behind the login proxies (there are many instances) should not notice in theory
2020-02-06T16:03:43  <adrianS> because the proxies will get switched, so no change for the users and the services
2020-02-06T16:03:57  <adrianS> well, we will have a phase where people need to re-set their password
2020-02-06T16:04:03  <adrianS> or recover later
2020-02-06T16:16:57  <malcolmlewis> adrianS, sounds good, many thanks for the clarification :) I will concentrate on getting Forum data, vB instance (and license) etc up and running.
2020-02-06T16:33:14  <adrianS> malcolmlewis: yep, sounds like a good plan
2020-02-06T16:33:30  <adrianS> login names will not change....
2020-02-06T17:19:03  -heroes-bot- PROBLEM: PostgreSQL standby on mirrordb1.infra.opensuse.org - POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB mb_opensuse2 (host:mirrordb2) 1188823280 and 131 seconds ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PostgreSQL%20standby
2020-02-06T18:27:18  <lcp> cboltz: would you mind setting up yast-test?
2020-02-06T18:27:55  <cboltz> I already did some hours ago ;-)
2020-02-06T18:28:25  <lcp> hm, it doesn't work yet then I guess
2020-02-06T18:28:53  <lcp> it does, nvm
2020-02-06T18:29:58  <cboltz> I already wondered ;-)
2020-02-06T18:31:48  <lcp> cboltz: I'm hoping to be done with especially news by monday at this rate
2020-02-06T18:32:18  <lcp> I started noodling on paste-o-o too, it will just take me some time to get the theme going
2020-02-06T18:35:32  <cboltz> :-)
2020-02-06T18:36:45  <cboltz> just wondering - do you know if people.o.o (currently broken :-( ) and 101.o.o also use jekyll? (If yes, {c,sh}ould we also host them on jekyll.i.o.o?)
2020-02-06T18:39:33  <lcp> 101 does
2020-02-06T18:40:34  <lcp> I think people does too(?)
2020-02-06T18:41:26  <lcp> I was gonna do a revamp of 101 at some point though
2020-02-06T18:43:14  <lcp> especially since I was gonna add a list of github issues to it from all of our organizations, so we have some way to share potential issues for newcomers
2020-02-06T18:45:45  <cboltz> is there something you don't want to touch? ;-)
2020-02-06T18:46:14  <lcp> https://www.youtube.com/watch?v=HgzGwKwLmgM
2020-02-06T18:46:55  <cboltz> clear answer ;-)
2020-02-06T18:49:20  <cboltz> BTW: I checked people.o.o on github. It comes with a generator/run_me script that contains   require "haml"   and   require "kramdown"   so - not jekyll
2020-02-06T18:50:06  <cboltz> on the positive side, it looks like the generated html is on github, which means we could host it on narwal* (aka static.o.o)
2020-02-06T18:52:12  <lcp> this seems easy enough to move to jekyll
2020-02-06T18:52:35  <lcp> but yeah, all of the html is already there
2020-02-06T18:53:10  <cboltz> it was last changed 3 years ago, therefore I'd say keeping it as is is the easiest solution (unless you really want to move it to jekyll, of course ;-)
2020-02-06T18:53:25  <lcp> I will let it go ;)
2020-02-06T18:54:34  <cboltz> ok, then I'll add it to narwal*
2020-02-06T19:01:20  -heroes-bot- PROBLEM: HTTP l10n on nuka.infra.opensuse.org - CRITICAL - Socket timeout after 10 seconds ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=nuka.infra.opensuse.org&service=HTTP%20l10n
2020-02-06T19:06:51  -heroes-bot- PROBLEM: SSH on riesling.infra.opensuse.org - SSH CRITICAL - OpenSSH_7.9 (protocol 2.0) version mismatch, expected OpenSSH_7.2 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=riesling.infra.opensuse.org&service=SSH
2020-02-06T19:11:11  -heroes-bot- RECOVERY: HTTP l10n on nuka.infra.opensuse.org - HTTP OK: HTTP/1.1 200 OK - 41226 bytes in 0.592 second response time ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=nuka.infra.opensuse.org&service=HTTP%20l10n
2020-02-06T19:51:55  -heroes-bot- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 112 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-02-06T20:01:55  -heroes-bot- RECOVERY: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=88 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-02-06T21:17:53  <cboltz> FYI: people.o.o works again - now served by narwal* ;-)