2021-04-27T08:53:57  <a-865k> download.opensuse.org in USA is painfully slow
2021-04-27T08:54:24  <a-865k> has been for more than a day
2021-04-27T09:02:11  <a-865k> agony waiting on OSS repository retrieval
2021-04-27T09:48:55  <bmwiedemann> a-865k: try http://mirrorcache-us.opensuse.org/download/
2021-04-27T09:50:26  <bmwiedemann> Eighth_Doctor: I'm pushing a lot of Factory package git repos atm. I found a much faster way to split them within seconds (compared to 5 minutes with git subtree split).
2021-04-27T09:52:21  <bmwiedemann> and before that I found a way to break the pagure server by pushing repos that failed, but left data in /tmp until the partition was nearly full.
2021-04-27T10:56:32  <Eighth_Doctor> bmwiedemann: I don't know whether to congratulate you or sigh in sadness
2021-04-27T10:57:44  <openSUSEBot> <LCP> probably worth fixing though ;)
2021-04-27T11:12:11  <a-865k> bmwiedemann: I've tried mirrorcache before, usually with very disappointing results. USA & Canada don't have enough mirrors of TW and repositories/ :(
2021-04-27T11:12:59  <Eighth_Doctor> mirrorcache doesn't help for me either
2021-04-27T11:13:02  <Eighth_Doctor> I just kinda give up
2021-04-27T11:13:46  <pjessen> a-865k: tw should not be an issue in the US, more than 2/3s of US mirrors carry TW.
2021-04-27T11:15:25  <pjessen> repositories - yeah, for most mirrors, it is just too big.
2021-04-27T11:16:54  <Eighth_Doctor> pjessen: I stopped carrying TW because it never got used
2021-04-27T11:19:38  <pjessen> TW does seem less popular in some places.  No mirrors in Canada, none in Australia.  Still, 16 in the US isn't bad.
2021-04-27T11:20:55  <openSUSEBot> <LCP> yeah, the issue seems to be nobody actually gets redirected to those mirrors though, I hear people from even CA, not just NY are getting german mirrors instead of US
2021-04-27T11:22:48  <pjessen> LCP: it would be good to see some mirrorlist examples - for TW, there is really no reason for getting mirrors outside the US.
2021-04-27T11:23:12  <lcp> indeed
2021-04-27T11:28:19  <pjessen> apart from that, download.o.o is pretty busy right now,
2021-04-27T11:28:45  <lcp> a-865k, Conan Kudo: https://download.opensuse.org/tumbleweed/repo/oss/x86_64/zypper-1.14.43-1.2.x86_64.rpm.mirrorlist what do you see there
2021-04-27T11:29:14  <Eighth_Doctor> I see 13 US mirrors
2021-04-27T11:29:53  <Eighth_Doctor> several from kernel.org, two from provo-mirror, and 4 from universities far from me
2021-04-27T11:30:23  <Eighth_Doctor> and bmw's mirror
2021-04-27T11:30:41  <lcp> if you removed .mirrorlist from the end where does that redirect you?
2021-04-27T11:31:07  <Eighth_Doctor> kernel.org this time
2021-04-27T11:31:47  <lcp> hmmm
2021-04-27T11:35:58  <pjessen> yes, the provo mirror is counted twice, and I also counted Bernhards mirror :-)
2021-04-27T12:11:19  <Eighth_Doctor> Sasi Olin: it's basically a roundrobin
2021-04-27T12:11:31  <Eighth_Doctor> note that immediately out of the 13, the next mirror is from Japan
2021-04-27T12:11:53  <lcp> hmmm
2021-04-27T12:13:14  <Eighth_Doctor> basically what happens for me is that with a sufficiently big transaction, I wind up hitting one of these "bad" mirrors and everything gets stuck
2021-04-27T12:13:34  <Eighth_Doctor> this happens on copr so frequently that we have retries encoded in the mock config
2021-04-27T12:13:56  <Eighth_Doctor> I personally deal with it by using the metalink instead, because then DNF switches stuff around
2021-04-27T12:14:18  <Eighth_Doctor> but the problem is that our mirroring doesn't work correctly for guaranteeing some degree of consistency for metalink fetching
2021-04-27T12:17:21  <pjessen> Eighth_Doctor: Japan??  that doesn't sound right, different continent.
2021-04-27T12:17:29  <Eighth_Doctor> Yup
2021-04-27T12:18:10  <pjessen> which mirror is it?
2021-04-27T12:18:20  <Eighth_Doctor> nrt.edge.kernel.org
2021-04-27T12:19:17  <Eighth_Doctor> then a few NL ones, Swiss and German ones after, then tons of eastern Europe ones, and more Asian ones
2021-04-27T12:20:04  <pjessen> with nrt.edge, it has geo-location sensitive dns, when I just looked it up now, I got an address in new york
2021-04-27T12:21:01  <Eighth_Doctor> interesting
2021-04-27T12:22:05  <pjessen> when there are 12 mirrors locally, I would not expect any redirects to mirrors in other countries.
2021-04-27T12:22:18  <Eighth_Doctor> 🤷‍♂️
2021-04-27T12:24:48  <pjessen> s/countries/continents/
2021-04-27T12:26:17  <lcp> hm, I wonder if maybe it would be possible to write an hourly CI that does a check if the redirects land in other countries/continents and fetch mirrorlist to confirm
2021-04-27T12:27:33  <pjessen> the thing is - we should be seeing the same issue everywhere. I should be getting us mirrors - at least occasionally.
2021-04-27T12:27:55  <Eighth_Doctor> not necessarily
2021-04-27T12:28:32  <Eighth_Doctor> there are at least as many mirrors in Europe as there are for Asia and Americas combined
2021-04-27T12:28:41  <Eighth_Doctor> it depends on the thresholds for these things
2021-04-27T12:32:02  <pjessen> when I get a mirrorlist for something TW, I have 3 mirrors same country, 43 other countries on the same continent, and 25 elsewhere.
2021-04-27T12:32:56  <pjessen> with a US address, I see 12 mirrors plus 59 elsewhere (other continents).
2021-04-27T12:34:52  * Eighth_Doctor shrugs
2021-04-27T12:35:10  <pjessen> Eighth_Doctor: yeah.
2021-04-27T13:41:14  <a-865k> lcp: nominally, 13, of which 2 provo, 3 kernel.org
2021-04-27T13:41:25  <a-865k> map link is useless
2021-04-27T13:41:30  <a-865k> The Google Maps Platform server rejected your request. You must use an API key to authenticate each request to Google Maps Platform APIs. For additional information, please refer to http://g.co/dev/maps-no-account
2021-04-27T13:42:07  <a-865k> closest mirror to me is probably more than 300 miles away
2021-04-27T13:42:50  <a-865k> I routinely get directed to Europe
2021-04-27T13:43:10  <a-865k> historically that is
2021-04-27T13:44:08  <a-865k> over 90 minutes to upgrade 1200 TW packages yesterday
2021-04-27T13:44:38  <a-865k> No Packman in USA
2021-04-27T13:46:26  <pjessen> a-865k: yes, we have an open ticket on the map link.
2021-04-27T13:47:15  <pjessen> a-865k: the kernel.org mirrors should be at least two different locations.
2021-04-27T14:21:04  <pjessen> 12-13 mirrors ought to be plenty, but I guess the mb redirection uses a wider span for the round-robin selection.
2021-04-27T15:02:08  <pjessen> a-865k: fwiw, i have opened a ticket, progress#91836
2021-04-27T17:05:23  <a-865k> pjessen: :)
2021-04-27T22:01:40  <kl_eisbaer> pjessen: FYI: on mx1,2 there is a new file /etc/postfix/handling_special_recipients for the noreply@opensuse.org Email. Feel free to integrate it wherever your think it fits best :-)
2021-04-27T22:02:17  <kl_eisbaer> I switched OBS to use this Email for outgoing messages now. I hope this makes admin-auto more silent.