2023-11-10T00:06:46 seems to work, it's loading now 2023-11-10T00:09:09 done 2023-11-10T00:13:50 thanks! 2023-11-10T06:32:55 *** teepee_ is now known as teepee 2023-11-10T09:49:43 *** teepee_ is now known as teepee 2023-11-10T09:53:51 *** cboltz_ is now known as cboltz 2023-11-10T13:09:21 shutting {backup,metrics}.i.o.o now 2023-11-10T14:17:13 acidsys, vpn doesn't let me in with AUTH_FAILED - anything changed there? 2023-11-10T14:18:13 hi anikitin, do you use OpenVPN natively or via NetworkManager ? 2023-11-10T14:18:25 from terminal 2023-11-10T14:19:36 how from the terminal? by calling openvpn or by calling networkmanager? 2023-11-10T14:20:51 calling openvpn 2023-11-10T14:21:42 ok what is the "remote" you are using? 2023-11-10T14:22:29 can I paste terminal output with ip here? 2023-11-10T14:22:47 yep 2023-11-10T14:22:55 but if you say IP, that is already the problem :) 2023-11-10T14:23:04 please use the DNS name (gate.opensuse.org) 2023-11-10T14:24:08 yeah i have `remote gate.opensuse.org 443` 2023-11-10T14:24:22 and https://paste.opensuse.org/pastes/0a50f9afaa04 2023-11-10T14:25:02 ping shows the same ip ping gate.opensuse.org 2023-11-10T14:25:02 PING gate.opensuse.org(2a07:de40:b27e:1102::a (2a07:de40:b27e:1102::a)) 56 data bytes 2023-11-10T14:25:28 interesting, bugfinder had an issue with TCP as well, could you try using the UDP please? 2023-11-10T14:25:50 here is a working config sample for UDP: https://progress.opensuse.org/issues/139280?issue_count=274&issue_position=3&next_issue_id=139256&prev_issue_id=139283#note-6 2023-11-10T14:29:57 hm with that udp config it is just hanging for me :) 2023-11-10T14:32:33 are you behind a firewall? 2023-11-10T14:32:54 just home router, I didn't change there anything 2023-11-10T14:33:22 ah let me try from another machine just in case 2023-11-10T14:35:40 the same, I can try modile connection if you wish, but this "(no shared cipher)" part looks susoicious 2023-11-10T14:36:20 ok just wondered because someone else had issues with tcp earlier (still need to check) whilst udp worked 2023-11-10T14:37:04 do you have both "ciphers" and "data-ciphers" (like in my example) in your file 2023-11-10T14:37:15 in paste ^, the same from laoptop and desctop. I didn't update the laptop for a month, so something changed or server side (or I forgot my password somehow :) - is there a way to verify the password without vpn? ) 2023-11-10T14:37:35 well the whole server changed yesterday 2023-11-10T14:37:49 password checking unfortunately isn't quite possible without the vpn 2023-11-10T14:38:11 but it uses the Heroes/IPA credentials 2023-11-10T14:39:07 let me check the tcp on my end in a minute as to rule that out 2023-11-10T14:39:16 ok data_chipers were commented out. now it works thx 2023-11-10T14:39:34 acidsys, thx, now works 2023-11-10T14:39:40 oh, great 2023-11-10T14:40:16 I'm surprised because this is a common issue but usually accompanied with a big warning message about deprecated ciphers in the log output. your output did not show it so I did not consider emphasizing data-ciphers more 2023-11-10T14:40:39 will have to update the sample configurations in the wiki soon 2023-11-10T14:40:51 let me know if ther are any other issues with the new infra 2023-11-10T14:43:32 acidsys, dns looks like a problem now. I have hardcoded ips there for anna and elsa iirc - anything changed there? 2023-11-10T14:44:05 > cat /etc/resolv.conf 2023-11-10T14:44:05 search infra.opensuse.org 2023-11-10T14:44:05 nameserver 192.168.47.102 2023-11-10T14:44:05 nameserver 192.168.47.101 2023-11-10T14:44:26 yes, for DNS please use 2a07:de40:b27e:64::c0a8:2f66 and 2a07:de40:b27e:64::c0a8:2f65 (this is anna/elsa's IPv6) until new internal DNS servers are ready 2023-11-10T14:47:04 acidsys, thx now pontifex2 doesn't let me in anymore 2023-11-10T14:47:18 can you ping it ? 2023-11-10T14:47:38 yes ssh asks for password - look my key is missing there 2023-11-10T14:48:52 I can ssh to mirrorcache at least 2023-11-10T14:50:11 acidsys, neither I can ssh to the salt master anymore, but that can wait 2023-11-10T14:51:10 anikitin: ldaps connectivity was broken, is now repaired and login should work again 2023-11-10T14:51:30 acidsys, yep thx 2023-11-10T15:00:14 :) 2023-11-10T15:12:26 download.opensuse.org not available , not just metrics 2023-11-10T15:12:44 machine is rebooting 2023-11-10T15:12:57 acidsys, nginx started on port 80 - is it intentional? 2023-11-10T15:13:12 on which machine? 2023-11-10T15:13:21 nothing started on 80 yet :) 2023-11-10T15:13:33 on pontifex apache says cannot bind 2023-11-10T15:13:36 colleague is checking 2023-11-10T15:15:34 download.o.o is back (thx @bugfinder) 2023-11-10T15:17:00 working 2023-11-10T15:17:03 confirmed 2023-11-10T15:17:22 I'm just adding the redirect domains on acme.i.o.o, and I wonder - with our haproxy setup, does it really make sense to have multiple domains in a certificate, or should we use one certificate per domain (and keep our-domains.txt more readable and sortable)? 2023-11-10T15:19:57 if you count that haproxy does not care what is in /etc/haproxy/pem if configured that way and will get correct one in any case 2023-11-10T15:21:42 I like grouping certificates based on their purpose. having the CN identify the service and the SAN's contain all related domains 2023-11-10T15:22:03 BManojlovic: yep that's what we are using .. pointing to a single directory 2023-11-10T15:22:57 an idea behind my question was that we might be able to mostly *) autogenerate our-domains.txt from the host_* haproxy config 2023-11-10T15:23:13 *) mostly, because expanding regexes is probably not worth the effort 2023-11-10T15:23:30 that would be much easier if we don't have multiple domains per line 2023-11-10T15:24:00 hm that's a nice idea but having a single certificate for every single host_* line sounds excessive 2023-11-10T15:24:36 would it hurt? ;-) 2023-11-10T15:24:48 potentially an easy way to get let's encrypt rate limited when they expire and renew all at once 2023-11-10T15:25:49 cboltz: i would do CN redirector.opensuse.org (first entry in domains.txt) and then add all the domains being redirected to that. 2023-11-10T15:28:17 that list will be quite long (~20 entries), but since you both prefer multi-domain certs, I'll do it that way 2023-11-10T15:28:37 luckily you can have up to 99 SAN's with LE :-) 2023-11-10T15:31:25 ;-) 2023-11-10T15:34:34 well another option is you think about which redirector entries you could kill 2023-11-10T15:34:37 🤷 2023-11-10T15:36:15 well we definitely want to keep https://rftm.opensuse.org 2023-11-10T15:36:23 rtfm* ... 2023-11-10T15:36:28 I clearly don't rtfm enough 2023-11-10T15:36:41 :) 2023-11-10T15:37:26 bonus questions: one of the redirect domains is (.*\.)?susestudio.com - but as far as I can see this isn't managed on chip 2023-11-10T15:37:53 so - who can update the IP for that domain and get a certificate for it to atlas*? 2023-11-10T15:37:54 it's managed in SUSE Route53 by the looks of it 2023-11-10T15:38:26 by saying that before waiting for your follow-up question I think I just got myself into another to-do 2023-11-10T15:38:49 we call this self own 2023-11-10T15:38:56 ;-) 2023-11-10T15:39:15 I don't like having to add code for http based challenge just for this single (legacy) domain though 2023-11-10T15:39:38 either it should be set to openSUSE nameservers or the redirect should happen at SUSE 2023-11-10T15:39:53 agreed 2023-11-10T15:50:12 will ask about it in the team .. 2023-11-10T15:53:17 DNS update for redirect domains done 2023-11-10T15:56:15 sweet 2023-11-10T15:57:55 the 3TB backup.i.o.o data disk is still copying, will check back on it later 2023-11-10T16:10:56 darix: can you paste the "new" way of serving errofiles? 2023-11-10T16:11:13 cboltz: ^^ for your conncheck patch 2023-11-10T16:15:45 It is used in the matrix backend 2023-11-10T16:16:11 For the well known files 2023-11-10T16:17:46 oh the same you say we should use for all files? 2023-11-10T16:18:51 return status 200 content-type ??/?? file /etc/haproxy/errorfiles/myerrorpage.html ? 2023-11-10T16:19:21 the matrix json response files are without headers, while the others include the HTTP headers. Does that need different handling (or removal of the headers from the response files)? 2023-11-10T16:19:49 See the haproxy code where the matrix files are used 2023-11-10T16:19:57 That will answer your questions 2023-11-10T16:21:41 it includes some hdr foo "foo value" - but with enough headers, this lines becomes long and hard to read IMHO 2023-11-10T16:21:50 Yes 2023-11-10T16:22:04 But this is the proper way 2023-11-10T16:23:02 Also some headers it will do for you 2023-11-10T16:25:34 which ones from conncheck.txt.http? (I could guess, but I'm not sure if that's the best idea ;-) 2023-11-10T19:06:45 cboltz: tbh i would just compare the header and see what you have in the cmdline and what haproxy calculated for you. i think content-length is the important autogenerated header 2023-11-10T19:08:41 hehe, not for conncheck which does a header-only reply ;-) 2023-11-10T19:39:08 https://docs.haproxy.org/2.8/configuration.html#http-request%20return 2023-11-10T19:39:11 as you can see there 2023-11-10T19:39:19 file parameter is optional 2023-11-10T19:41:34 it explains what it does 2023-11-10T19:43:00 indeed, that looks useful and helps to understand what's going on 2023-11-10T19:51:29 ok, I'm getting closer: 2023-11-10T19:51:35 http-request return status 204 content-type text/plain hdr "Cache-Control" no-cache hdr "X-NetworkManager-Status" online hdr "Connection" close 2023-11-10T19:52:14 content-type gets ignored (which probably isn't a problem with the empty response body) 2023-11-10T19:52:46 the more annoying issue is that the headers are all lowercased, for example I get x-networkmanager-status: online 2023-11-10T19:53:49 tell curl to do http 1.1 2023-11-10T19:53:54 you probably do http 2.0 2023-11-10T19:53:56 also 2023-11-10T19:54:01 string "" 2023-11-10T19:54:02 als body 2023-11-10T19:55:44 error detected in backend 'conncheck' while parsing 'http-request return' rule : 'string' expects as argument. - so string "" doesn't work 2023-11-10T19:57:27 curl -0 and curl --http1.1 give me the same lowercase headers 2023-11-10T19:57:50 and -v also tells you that it returns 1.x? 2023-11-10T19:58:15 yes, the reply is HTTP/1.1 2023-11-10T20:00:21 BTW: if you want to have a look yourself - I'm trying this on elsa 2023-11-10T20:04:02 how old is your haproxy? 2023-11-10T20:06:09 string "\n" works 2023-11-10T20:08:23 tbh 2023-11-10T20:08:32 do you really need a content-type for an empty response anyway? 2023-11-10T20:10:42 http-request return status 200 content-type text/json hdr Server "HAProxy Auto Reply/openSUSE is good for you" hdr Access-Control-Allow-Origin '*' hdr Cache no-cache hdr "X-NetworkManager-Status" online hdr "Connection" close 2023-11-10T20:10:45 looks fine by me 2023-11-10T20:10:55 i mean if you want to see if networkmanager is happy with it 2023-11-10T20:11:08 set conncheck.opensuse.org in /etc/hosts to elsa's IP and see? 2023-11-10T20:20:14 that's what I just did (in unbound, not in /etc/hosts, but - details) 2023-11-10T21:22:41 *** teepee_ is now known as teepee 2023-11-10T21:45:56 hey cboltz 2023-11-10T21:46:02 are you on Leap 15.5? 2023-11-10T21:46:11 no, Tumbleweed 2023-11-10T21:46:16 ah, ok 2023-11-10T22:27:07 cboltz: any ideas why haproxy "defaults" to our internal certificate if a bogus servername is passed? https://progress.opensuse.org/issues/139007 it's not a big problem but it's a bit ugly .. it should "default" to the atlas*.opensuse.org let's encrypt certificate 2023-11-10T22:27:22 (to reproduce: `echo|openssl s_client -showcerts -connect proxy-prg2.opensuse.org:443 -servername bogus.com 2>/dev/null|openssl x509 -noout -text`) 2023-11-10T22:31:34 no real idea, but a wild guess - sorting 2023-11-10T22:31:41 atlas.infra sorts before atlas.opensuse 2023-11-10T22:34:38 crazy idea, at least to test this: add aaaaaa.opensuse.org to the atlas.o.o cert ;-) 2023-11-10T22:39:55 uhm 2023-11-10T22:40:17 why not have a /etc/ssl/certs/internal /etc/ssl/external? 2023-11-10T22:40:19 also 2023-11-10T22:40:29 you can do "strict-sni" in the haproxy cfg 2023-11-10T22:40:32 bind line 2023-11-10T22:40:44 then it means "no matching cert no connection" 2023-11-10T22:41:29 why does atlas have an internal cert at all? 2023-11-10T22:41:31 just curious 2023-11-10T22:47:03 https://docs.haproxy.org/2.4/configuration.html crt-list says The first declared certificate of a bind line is used as the default certificate so my guess probably wasn't too wrong ;-) 2023-11-10T22:47:17 crtlist is yet another config file you have to maintain 2023-11-10T22:47:22 and not just drop cert into dir 2023-11-10T22:48:27 cboltz: ah that makes sense, I did not consider it 2023-11-10T22:48:30 I'd _guess_ that reading the cert dir is an implicit way for crt-list (at least the internal implementation) 2023-11-10T22:48:36 darix: strict-sni sounds like a great option 2023-11-10T22:54:28 you can test if it works with openssl 1.0 on my machine 2023-11-10T22:54:58 openssl version is not relevant for this 2023-11-10T22:55:23 thats not what the ticket said 2023-11-10T22:55:30 did you read my comment? 2023-11-10T22:55:53 the only difference is whether -servername is implied 2023-11-10T22:56:01 so you can just reproduce like in my snippet with -servername bogus.com 2023-11-10T22:56:20 well 2023-11-10T22:56:29 happy testing with strict-sni 2023-11-10T22:56:35 ok 2023-11-10T23:34:02 darix: I now have my recursor on hel, forward lookup forwarding works fine, but reverse forwarding (to our two ip6.arpa zones which we currently only have internally) yields servfail 2023-11-10T23:34:03 Nov 10 23:32:36 hel1.infra.opensuse.org pdns-recursor[30826]: msg="Sending SERVFAIL during resolve" error="Server Failure while retrieving DNSKEY records for ip6.arpa" subsystem="syncres" level="0" prio="Notice" tid="3" ts="1699659156.599" ecs="" mtid="9" proto="udp" qname="1.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.4.0.2.1.e.7.2.b.0.4.e.d.7.0.a.2.ip6.arpa" qtype="PTR" 2023-11-10T23:34:05 remote="[2a07:de40:b27e:5001::1005]:40638" 2023-11-10T23:34:39 do we have to add DNSKEY/DS records to e.7.2.b.0.4.e.d.7.0.a.2.ip6.arpa ? 2023-11-10T23:36:37 yeah 2023-11-10T23:36:48 same dance we did for infra 2023-11-10T23:38:04 okidoki 2023-11-10T23:41:42 https://paste.opensuse.org/pastes/6ec55966c179 2023-11-10T23:41:53 but it can then start to complain that the zone path is incomplete 2023-11-10T23:42:01 so maybe we should remove dnssec for now 2023-11-10T23:42:15 opensuse.org also shows these errors 2023-11-10T23:42:19 well warnings 2023-11-10T23:43:03 which is odd because you say it only worked after you added DNSKEY. but it writes that it ignores DNSKEY 2023-11-10T23:44:38 opensuse.org should work just fine 2023-11-10T23:44:40 we tested that 2023-11-10T23:46:00 I am talking about the warnings 2023-11-10T23:46:19 you say opensuse.org only worked after you added DNSKEY. but DNSKEY is ignored, hence it must still work without the DNSKEY record