2021-08-13T07:04:58 *** tigerfoot_ is now known as tigerfoot 2021-08-13T09:15:36 are there any indications of DNS problems? I am getting random resolve errors with the opensuse.org zone 2021-08-13T09:16:00 I cannot pinpoint it to a particular opensuse DNS server yet though 2021-08-13T09:23:39 danimo: I saw the same earlier. 2021-08-13T09:24:33 I restarted our local dns, that solved the problem for us, so far 2021-08-13T09:30:43 dnssec problem 2021-08-13T09:31:03 my bind says "broken trust chain" 2021-08-13T09:39:18 bmwiedemann: at what point? 2021-08-13T09:46:07 DNS started to fail around 10:20 CEST today 2021-08-13T09:48:46 Aug 13 09:46:10 named[15709]: validating meet.opensuse.org/A: bad cache hit (opensuse.org/DNSKEY) 2021-08-13T09:48:46 Aug 13 09:46:10 named[15709]: broken trust chain resolving 'meet.opensuse.org/A/IN': 213.133.99.99#53 2021-08-13T09:52:50 I see the first failures to resolve at around 10:09 CEST 2021-08-13T09:53:57 bmwiedemann: did the zone key expire? 2021-08-13T09:57:13 https://dnssec-analyzer.verisignlabs.com/opensuse.org looks good... 2021-08-13T10:29:36 hmm. when I tell my bind to not forward to Hetzner's DNS servers, it can resolve again 2021-08-13T11:01:54 is there some issue with d.o.o, or the network path between me and it is having a fit? I get a bunch of "Error code: Connection failed" for repomd.xml for several repos 2021-08-13T11:10:41 pedja: works from here 2021-08-13T11:11:06 pjessen, where is 'here' :) ? 2021-08-13T11:11:24 pedja: zurich, switzerland 2021-08-13T11:12:03 ah, OK. thanks. I guess something wrong on the path to these neck of the woods (.rs) 2021-08-13T11:15:46 I suspect a mirror issue, I just ran 'zypper ref' and only 3 repos fail now :) 2021-08-13T11:18:03 I wonder if I can persuade Belgrade University to add openSUSE repos to their mirror(atm, they mirror centos and *buntu, iirc) 2021-08-13T11:18:38 no harm in asking, I guess 2021-08-13T11:23:11 more mirrors are always welcome 2021-08-13T11:28:07 is the information at https://en.opensuse.org/openSUSE:Mirror_infrastructure correct/up-to-date? 2021-08-13T11:29:09 if mirror admins are game to add oS, I'd like to include that link in the e-mail to them 2021-08-13T12:16:27 https://twitter.com/LaF0rge/status/1426154667738812418 2021-08-13T12:16:37 something is def. weird with hetzner DNS and opensuse.org 2021-08-13T12:17:09 (at least) 2021-08-13T12:22:56 <_Marcus_> hi ... ui sent a mail to admin@ and heroes@ ... we seem to be blocking DNS traffic to ns.opensuse.org coming e.g. from Hetzner 2021-08-13T13:52:57 I've just read an answer on the /usr/bin/wall ;-) - while doing a DNSSec fix, the glue records got broken. A ticket to get them fixed is open at the registrar, but I have no idea how fast they'll work on it 2021-08-13T14:03:11 Greetings. I fielded reports this morning of DNS resolution failures (SERVFAIL) for download.opensuse.org. 2021-08-13T14:03:16 *** neirbowj_ is now known as neirbowj 2021-08-13T14:03:49 neirbowj: Came here for the same reason, and got pointed at: cboltz in #opensuse-admin:"[...]while doing a DNSSec fix, the glue records got broken. A ticket to get them fixed is open at the registrar[..]" 2021-08-13T14:04:02 ns{1,2,3,4}.opensuse.org were intermittently returning REFUSED for opensuse.org/DNSKEY 2021-08-13T14:18:47 The glue problem is very minor. The parent and child disagree about which IPs goes with which server, but all the servers and all the IPs appear in the referral and the authoritative answers. 2021-08-13T14:19:17 I must assume that the DNSSEC issue that required a fix is what caused the resolution failures. 2021-08-13T16:05:56 cboltz: https://twitter.com/LaF0rge/status/1426154667738812418 2021-08-13T16:06:04 not sure if anyone told heroes already 2021-08-13T16:07:49 https://progress.opensuse.org/issues/96830 2021-08-13T16:07:56 https://progress.opensuse.org/issues/96836 2021-08-13T19:30:35 I got a pcap from nue-ns1 and it has 1 query from Hetzner's NUE IPv6 range and 1 response packet (513 byte UDP) 2021-08-13T19:40:55 I'm out of ideas. Also it is hard to debug, it it is working from my IPs at hetzner. 2021-08-13T19:42:14 could it be that Hetzner "just" still has a not-yet-expired bad cache? 2021-08-13T19:43:04 but our TTLs are just 1800 (=30m) 2021-08-13T19:43:18 yeah, but what's the TTL of .org ? 2021-08-13T19:43:23 or are glue records using another TTL? 2021-08-13T19:44:00 I _think_ the TTL of .org applies for the glue records 2021-08-13T19:44:03 .org has 1d TTL 2021-08-13T19:44:27 ah, that could explain it :-/ 2021-08-13T19:44:28 but negative replies normally have lower TTL 2021-08-13T19:47:37 interestingly, sometimes (10%) I get a reply, but then on next try it is gone again 2021-08-13T19:50:48 wild guess: Hetzner wrote in the ticket that their outgoing DNS originates from a (v4) /26 subnet, but your server probably has 3 or 4 servers in resolv.conf. Could it be that they do some load balancing, and you hit a different DNS server each time (while 90% of them still having the error cached)? 2021-08-13T19:56:40 cboltz: no, I am using `host` to query their IPs explicitly 2021-08-13T19:57:43 maybe Hetzner has multiple DNS servers behind 1 IP? 2021-08-13T20:03:15 reply in the ticket and ask them? 2021-08-13T20:15:59 done. Maybe I should call them tomorrow 2021-08-13T20:18:03 which query would give you the glue records? 2021-08-13T20:18:48 dig +short NS opensuse.org 2021-08-13T20:18:53 doesnt show any IPs 2021-08-13T20:20:17 host -v has them in the additional section 2021-08-13T20:21:35 host -v -t soa opensuse.org. c0.org.afilias-nst.info. 2021-08-13T20:22:21 it does not answer the query, but points to the 4 authoritative NS 2021-08-13T20:23:11 i dont get any additional informations via my pdns 2021-08-13T20:23:14 will purge caches soon 2021-08-13T20:25:26 done 2021-08-13T20:25:33 still no extra ips listed 2021-08-13T20:26:45 are we really sure that the glue records are back 2021-08-13T20:26:48 because in doubt this means 2021-08-13T20:27:12 that the last server at hetzner that still replies is because is still has the active cache for the ns records 2021-08-13T20:28:47 dig +trace @a.root-servers.net opensuse.org 2021-08-13T20:28:52 If the GLUE is setup you should see a record that ends with: 2021-08-13T20:28:55 “Recevied XXX bytes from x.GTLD-SERVERS.NET.” 2021-08-13T20:29:07 “Recevied XXX bytes from x.GTLD-SERVERS.NET.” 2021-08-13T20:29:13 https://serverfault.com/questions/142344/how-to-test-dns-glue-record 2021-08-13T20:29:40 hmm 2021-08-13T20:29:47 intodns listet aber was 2021-08-13T20:29:58 https://intodns.com/opensuse.org 2021-08-13T20:30:22 INFO: GLUE was not sent when I asked your nameservers for your NS records.This is ok but you should know that in this case an extra A record lookup is required in order to get the IPs of your NS records. The nameservers without glue are: 2021-08-13T20:30:26 195.135.221.195 2021-08-13T20:30:28 195.135.221.196 2021-08-13T20:30:31 91.193.113.68 2021-08-13T20:30:33 62.146.92.204 2021-08-13T20:30:36 You can fix this for example by adding A records to your nameservers for the zones listed above. 2021-08-13T20:32:48 dig +norec @a0.org.afilias-nst.info. opensuse.org. NS 2021-08-13T20:32:51 ok 2021-08-13T20:32:55 that's the query you want 2021-08-13T20:32:59 and then it looks good 2021-08-13T20:51:31 https://de.wikipedia.org/wiki/NS_Resource_Record