2023-11-30T09:28:13  <andrii> many complains on slack that progress.opensuse.org shows 500 and indeed - some problem with cookies, because it can load in incognito and curl doesn't complain as well
2023-11-30T09:29:41  <acidsys> hi andrii, checking
2023-11-30T09:30:03  <acidsys> solved. was regression from id.o.o migration earlier today.
2023-11-30T09:32:26  <andrii> cool thx!
2023-11-30T09:32:30  <acidsys> :)
2023-11-30T10:16:20  <acidsys> hi henne, can you check hackweek/events? hackweek hangs when logging in, events runs into gateway timeout from the local apache
2023-11-30T10:16:34  <acidsys> I searched for logs on dale in the system journal and in /var/log but there is nothing
2023-11-30T10:20:29  <acidsys> ugh I eventually found the log file for events in /home/osem/events/releases/43/log/production.log .. what a location .. but doesn't show my attempts by the looks of it
2023-11-30T10:24:23  <DimStar> Hi all.. we have noticed that the collab server (osc-collab.infra.o.o) is no longer able to get the package list from OBS
2023-11-30T10:33:31  <acidsys> hi DimStar, where does it need to connect to for that and through which protocol?
2023-11-30T10:34:05  <DimStar> it used osc, so connects to api.opensuse.org
2023-11-30T10:35:32  <acidsys> ok, try now
2023-11-30T10:35:41  <DimStar> as for the protocols: I assume it connects to whatever resolver answers (but then, I also am limited on that machine for some reason now; can't even sudo -i anymore)
2023-11-30T10:36:14  <DimStar> interetingly, the error change:
2023-11-30T10:36:15  <DimStar> > osc ls
2023-11-30T10:36:15  <DimStar> Failed to reach a server (api.opensuse.org): [Errno -3] Temporary failure in name resolution
2023-11-30T10:36:46  <DimStar> used to be: > osc ls
2023-11-30T10:36:47  <DimStar> Failed to reach a server (api.opensuse.org): [Errno 101] Network is unreachable
2023-11-30T10:37:17  <acidsys> ok, I see the problme
2023-11-30T10:37:54  <acidsys> I remove there the broken "HW19" repository so our Salt states apply cleanly.
2023-11-30T10:38:39  <DimStar> that sounds like a bad idea - the hw19 repo is the essence of that server :)
2023-11-30T10:39:01  <acidsys> now api.opensuse.org works and your login/sudo problem should be resolved
2023-11-30T10:39:29  <acidsys> you can add it again but make sure refresh works
2023-11-30T10:40:18  <acidsys> use https://downloadcontent.opensuse.org/ for the URL
2023-11-30T10:40:34  <acidsys> or even better, add it via salt ^^
2023-11-30T10:41:00  <darix> do it in salt or it doesnt exist and will be removed again
2023-11-30T10:41:26  <DimStar> the repo exists (and always did): https://download.opensuse.org/repositories/home:/dimstar:/hw19/15.5/
2023-11-30T10:44:39  <darix> acidsys: moved the machines from last night on the etherpad page :)
2023-11-30T10:45:07  <darix> DimStar: if those tools are so essential wouldnt it make sense to have then in openSUSE:Tools or openSUSE:infrastructure?
2023-11-30T10:45:34  <darix> and acidsys only asked you to use downloadcontent.opensuse.org
2023-11-30T10:46:20  <acidsys> thx darix
2023-11-30T10:53:37  <DimStar> jaja - of course I add the downloadcontent.o.o as repo; that does not change the fact that the repo has not been broken in years
2023-11-30T10:57:53  <acidsys> sorry, I don't know what the exact problem was. there were lots of machines blocking my migration due to broken home or devel repositories. with the ones managed via Salt it was easy - I adjusted the needed settings centrally. with manually managed ones I had to be radical otherwise I would still be trying to message people next year
2023-11-30T10:59:51  <DimStar> as there were network issues which salt solved, this sounds like a deadlock (I myself would probably only have disabled the repos, not removed them.. but does not matter much; seems seem to be back in shape and I'll keep an eye on that machine)
2023-11-30T11:02:14  <acidsys> usually I disabled them and added a note to /etc/motd. seems not in your case. maybe it was already late
2023-11-30T11:03:12  <acidsys> yes I applied the highstate and it was missing various network settings so now it should be good .. otherwise let me know
2023-11-30T11:03:23  <DimStar> \o/ - could have been worse :) I knew where to find you guys, you helped, and that's all that matters in the end
2023-11-30T11:03:49  <acidsys> appreciated =)
2023-11-30T11:09:50  <acidsys> I am about to send my last poweroff command .. means as of today, openSUSE in Maxtorhof will be history
2023-11-30T11:10:23  <acidsys> make sure to note down the date in case you get asked in history class
2023-11-30T13:31:08  <acidsys> hi darix, can you help me with https://es.opensuse.org .. the other wikis work fine but this one returns 500 .. daffy2 error log prints "[Thu Nov 30 13:28:19.877237 2023] [proxy:error] [pid 26691:tid 139983612520192] [client 2a02:1748:f7df:9c80:8aa4:c2ff:fed6:4038:55412] AH00898: Error during SSL Handshake with remote server returned by /"
2023-11-30T13:31:34  <acidsys> curl -HIL 'Host: $lang.opensuse.org' https://atlas-login.infra.opensuse.org returns atlas maintenance page for any $lang
2023-11-30T13:32:04  <acidsys> but from remote client it works fine for other langs
2023-11-30T13:35:43  <acidsys> I did install valid certificates for atlas-login{,1,2}.infra.opensuse.org btw
2023-11-30T13:43:37  <darix> we are still on BBB
2023-11-30T13:44:33  <darix> my curl cmdline from daffy2 works
2023-11-30T13:44:56  <darix> curl -I -H "Host: es.opensuse.org" https://atlas-login.infra.opensuse.org/
2023-11-30T13:44:56  <darix> HTTP/2 301
2023-11-30T13:44:56  <darix> location: http://es.opensuse.org/Bienvenidos_a_openSUSE.org
2023-11-30T13:45:13  <darix> works from daffy1 too
2023-11-30T13:45:57  <darix> curl -I -H "Host: es.opensuse.org" https://atlas-login.infra.opensuse.org/Bienvenidos_a_openSUSE.org
2023-11-30T13:45:59  <darix> works too
2023-11-30T13:48:49  <acidsys> oh weird then what is the ssl handshake error about
2023-11-30T13:49:03  <acidsys> the vhost file seems to be using the same syntax as the others
2023-11-30T13:52:46  <darix> yeah i mean those are basically schema F :)
2023-11-30T13:53:16  <darix> is the ca-certificates-freeipa-heroes installed on daffy?
2023-11-30T13:54:36  <acidsys> I used Let's Encrypt certificates for now
2023-11-30T13:54:47  <darix> for the internal infra hsotname?
2023-11-30T13:55:10  <acidsys> yes I did not have the dehydrated instance for internal ca configured yet so it was easier
2023-11-30T14:13:22  <acidsys> we need to refactor the apache configuraton on ldap-proxy to offer TLS to atlas so I can proxy www.o.o/openid there without doing plain text .. but for that I need plain HTTP to let's encrypt which in the current non-vhost based config would also expose the whole id.o.o over plain text to the internet
2023-11-30T14:14:56  <acidsys> might be a good opportunity to use https for daffy -> ldap-proxy as well
2023-11-30T14:15:22  <darix> well
2023-11-30T14:15:27  <darix> we can keep the name internal
2023-11-30T14:15:37  <darix> and just setup TLS with acme against infra.o.o step-ca?
2023-11-30T14:20:47  <acidsys> would be cool but for that we need connectivity to acme.i.o.o over the SUSE WAN in place
2023-11-30T14:27:39  <darix> acidsys: events.o.o fixed
2023-11-30T14:28:19  <darix> acidsys: wrong ;) we only need to proxy acme via atlas on the login interface _runs_
2023-11-30T14:28:52  <acidsys> it seems daffy has a general problem with TLS, not just with es.opensuse.org. now that I changed the macro in id.opensuse.org vhost to use https:// to ldap-proxy that also returns the handshake error?
2023-11-30T14:29:46  <darix> not sure what you did
2023-11-30T14:30:24  <acidsys> I replaced http:// with https://
2023-11-30T14:30:30  <acidsys> in /etc/apache2/vhosts.d/id.opensuse.org.conf
2023-11-30T14:30:46  <acidsys> curl https://ldap-proxy.opensuse.org works from daffy2
2023-11-30T14:32:09  <acidsys> 84:39608] AH01084: pass request body failed to [2a07:de40:b280:87::1]:443 (ldap-proxy.opensuse.org)
2023-11-30T14:32:11  <acidsys> [Thu Nov 30 14:32:01.592706 2023] [proxy:error] [pid 29650:tid 139984073889536] [client 47.128.58.84:39608] AH00898: Error during SSL Handshake with remote server returned by /openidc/Authorization
2023-11-30T14:32:29  <darix> ok
2023-11-30T14:32:34  <darix> yeah i see that
2023-11-30T14:33:52  <darix> but you know for atlas-login it makes literally 0 sense why it works for all those other vhosts just not es.
2023-11-30T14:34:11  <darix> i did a diff against de. and the "es" vs "de" is the only difference
2023-11-30T14:34:33  <acidsys> it also made no sense why it worked even before I deployed the atlas-login certificate
2023-11-30T14:34:50  <darix> well
2023-11-30T14:37:10  <darix> the only sslproxy setting that is configured is engine on
2023-11-30T14:38:39  <acidsys> a no validation?
2023-11-30T14:39:15  <darix> https://httpd.apache.org/docs/2.4/mod/mod_ssl.html#sslproxyverify
2023-11-30T14:39:17  <darix> correct
2023-11-30T14:39:21  <darix> i turned it on now :D
2023-11-30T14:39:31  <darix> should I rcapache2 reload? :p
2023-11-30T14:39:45  <acidsys> sure
2023-11-30T14:40:10  <acidsys> but it would still be nice to figure out why id.o.o and es.o.o do not like it even without validation :|
2023-11-30T14:40:24  <darix> done
2023-11-30T14:40:43  <darix> now more things are dead :p
2023-11-30T14:40:45  <acidsys> the problem is id.o.o only has one interface so if I want to allow plain text http only from daffy I need to make it an acl inside the httpd config I cannot filter on host firewall level
2023-11-30T14:40:46  <darix> _runs_
2023-11-30T14:40:54  <acidsys> ah great
2023-11-30T14:41:13  <acidsys> then let's revert for now
2023-11-30T14:41:44  <acidsys> on my systems I sometimes had to adjust SSLProxyVerifyDepth to allow for intermediate certificates
2023-11-30T14:42:01  <acidsys> but not sure that is needed if verify is not enabled at all
2023-11-30T14:42:53  <acidsys> let's try to set SSLProxyCACertificateFile /etc/ssl/ca-bundle.pem ?
2023-11-30T14:43:12  <darix> i did already
2023-11-30T14:43:16  <darix> in opensuse-ssl.inc
2023-11-30T14:43:16  <acidsys> a okay
2023-11-30T14:43:47  <darix> as I said ... it talks to the very same haproxy on every other language ... where it works
2023-11-30T14:44:34  <acidsys> ye (except ldap-proxy)
2023-11-30T14:44:54  <darix> which doesnt make any sense either :p
2023-11-30T14:45:06  <acidsys> btw ldap-proxy is in /etc/apache2/vhosts.d/ipsilon-https.conf (not the cleanest I just merged the existing non-vhost configs and added the ssl parts)
2023-11-30T14:46:16  <acidsys> hm I could for now rever it now that I have a certificate.. but then it cannot renew
2023-11-30T14:47:10  <acidsys> or I make an acl in httpd that allows plain http only from daffy source addresses.. but that's even more complexity
2023-11-30T14:50:20  <darix> as long as it is behind daffy you can just use your step-ca
2023-11-30T14:50:32  <darix> do not overthink it and push letsencrypt for everything
2023-11-30T14:50:40  <darix> 🤷
2023-11-30T14:50:47  <acidsys> I cannot access step-ca at the moment
2023-11-30T14:50:51  <acidsys> I need something that works right now
2023-11-30T14:51:14  <darix> you remember how we created certs manually?
2023-11-30T14:51:24  <darix> where you can even make it last a long  time
2023-11-30T14:51:52  <acidsys> fine but why would that cert work for https daffy -> ldap-proxy if the current one does not work either
2023-11-30T14:52:38  <darix> no idea
2023-11-30T14:52:52  <darix> atm i have no clue what causes the handshake errors on those 2 vhosts
2023-11-30T14:53:52  <acidsys> I tried to set CipherSuite to the same as the one in the backend now as well .. also no luck
2023-11-30T14:54:16  <acidsys> hm okay then let me revert it now
2023-11-30T14:54:34  <bmwiedemann> pagure wants to connect to https://ev.opensuse.org:8080
2023-11-30T14:54:59  <darix> i mean all those settings are global?
2023-11-30T14:55:11  <darix> why would it work on vhost A connecting to atlas but not on B ? :D
2023-11-30T14:55:33  <darix> whatever ev.o.o is
2023-11-30T14:55:38  <darix> and what was running there
2023-11-30T14:55:56  <darix> acidsys: did you see slack?
2023-11-30T14:56:20  <acidsys> okay normal id.o.o works again now. now I need to think of some other hacks to get www.o.o/openid going .. I could ask try to proxy it through daffy but also will need to ask for that to be allowed in SUSE firewall
2023-11-30T14:56:34  <acidsys> yes
2023-11-30T14:56:37  <bmwiedemann> "Eventsource"
2023-11-30T14:56:58  <darix> acidsys: would it work if www.o.o just redirects to id.?
2023-11-30T14:57:11  <darix> instead of proxying
2023-11-30T14:57:30  <acidsys> hmm good question/idea. I'm not sure; maybe it would if it passes all the request parameters
2023-11-30T15:03:21  <darix> strictly speaking we could simplify the whole daffy config ... by having one wiki vhost
2023-11-30T15:03:35  <darix> but of course that would require us to redo the macro
2023-11-30T15:27:42  <acidsys> I could re-enable https on ldap-proxy and then switch id.o.o dns record to go there directly without going through daffy
2023-11-30T15:27:47  <acidsys> any objections?
2023-11-30T15:28:56  <darix> up to you
2023-11-30T15:29:42  <acidsys> okay then I think that'll be the easiest (assuming https atlas -> ldap-proxy works as originally designed .. should test first)
2023-11-30T15:31:54  <bmwiedemann> :+1:
2023-11-30T15:52:31  *** teepee_ is now known as teepee
2023-11-30T16:19:23  <acidsys> okay https://id.opensuse.org and https://www.opensuse.org/openid seem to both be working again now
2023-11-30T16:19:53  <acidsys> not sure which services to test for the latter, I tried openqa.o.o and login there seems to work
2023-11-30T16:20:14  <acidsys> until the TTL for id.o.o expired I have placed an /etc/hosts entries on atlas1. I will remove it in an hour
2023-11-30T16:20:22  <acidsys> now I need to run to pick up my food
2023-11-30T16:33:34  <darix> woot woot
2023-11-30T17:08:04  <DimStar> acidsys: seems the osc-collab server has another problem: we can't reach download.gnome.org:443 from there
2023-11-30T17:08:09  <DimStar> > curl --tlsv1  --fail https://download.gnome.org/core/
2023-11-30T17:08:10  <DimStar> curl: (7) Failed to connect to download.gnome.org port 443 after 1203 ms: Couldn't connect to server
2023-11-30T17:11:51  <DimStar> * connect to 2a04:4e42:600::347 port 443 failed: Permission denied
2023-11-30T17:15:22  <DimStar> similar errors reported for the server trying to download e.g http://www.cpan.org/modules/02packages.details.txt.gz
2023-11-30T17:16:44  <cboltz> meet.o.o gives me 3 404 columns and a connection error :-(
2023-11-30T17:31:58  <acidsys> DimStar: download.gnome.org/443, cpan.org/80, something else?
2023-11-30T17:32:07  <acidsys> cboltz: it's offline
2023-11-30T17:32:18  <DimStar> acidsys: pretty sure - that server is crawling out to find what needs to be updated
2023-11-30T17:32:44  <DimStar> pypi.org, repology.org are certainly on the list
2023-11-30T17:32:57  <acidsys> well please compile a full list and make a short ticket with it. then I can process it all together
2023-11-30T17:33:10  <DimStar> freea short list containing the internet?
2023-11-30T17:33:44  <DimStar> https://github.com/openSUSE/osc-plugin-collab/blob/master/server/upstream/upstream-tarballs.txt
2023-11-30T17:34:54  <acidsys> cool, will work something out
2023-11-30T17:35:33  <DimStar> node: the list is not static - with packages coming and going, this list can be changing too
2023-11-30T17:41:03  <acidsys> okay
2023-11-30T17:41:25  <acidsys> will work on it in a bit
2023-11-30T17:47:49  <acidsys> http/https/ftp it seems
2023-11-30T17:48:19  <acidsys> okay try now
2023-11-30T17:48:46  <acidsys> DimStar^
2023-11-30T17:50:31  <acidsys> darix: sorry to ask you, but will someone work on daffy{1,2} HA?
2023-11-30T17:58:20  <SFaulken> hm, I just got an e-mail kicked back from election-officials@lists.opensuse.org as spam, I thought that was fixed?
2023-11-30T18:17:51  <SFaulken> resent it, making sure its plain-text, see if it bounces back again
2023-11-30T20:08:16  <darix> https://about.gitlab.com/releases/2023/11/30/security-release-gitlab-16-6-1-released/
2023-11-30T20:16:16  <SFaulken> yup, plaintext did it.
2023-11-30T20:16:22  <SFaulken> bloody webmail…
2023-11-30T20:48:20  <SFaulken> can't login to the wiki for editing, is this new, or something being worked on?
2023-11-30T20:50:34  <acidsys> what's the problem?
2023-11-30T20:51:00  <acidsys> ah it hangs for me as well
2023-11-30T20:51:04  <acidsys> cboltz ^^^
2023-11-30T20:55:42  <SFaulken> got a 504 error, when trying to authenticate
2023-11-30T20:58:15  <acidsys> seems progress has an issue as well, after signing in via id.o.o it returns 503
2023-11-30T20:58:36  <acidsys> but not sure it's the same problem because it's two different authentication routes
2023-11-30T20:59:10  <acidsys> will check the progress one
2023-11-30T20:59:20  <SFaulken> yeah, I'm signed into the forums, mailing-lists and build.o.o just fine
2023-11-30T21:15:41  <acidsys> okay progress.o.o is repaired
2023-11-30T22:18:10  <cboltz> after waiting long enough, I get "Gateway Timeout" for https://en.opensuse.org/ICSLogin/auth-up
2023-11-30T22:18:31  <cboltz> I'm afraid this means the problem is somewhere in the login proxy :-(
2023-11-30T22:22:37  <cboltz> same "Gateway Timeout" also when trying to log in in elections.o.o -> clearly the login proxy
2023-11-30T22:25:12  <acidsys> does it need to "connect back" to the login proxy ?
2023-11-30T22:25:32  <darix> no
2023-11-30T22:25:54  <acidsys> good
2023-11-30T22:26:07  <acidsys> can you check darix ?
2023-11-30T22:58:55  *** teepee_ is now known as teepee
2023-11-30T23:54:33  <draeath> Is this an appropriate venue to ask about sign-in troubles with lists.opensuse.org? Looking at the Heroes page leaves me thinking it's not, but I wasn't sure if that's true.