2024-12-22T00:32:00  *** teepee_ is now known as teepee
2024-12-22T01:44:33  <jshand> is first years server fallen over again.  i seemed to have gotten the debug packages ok
2024-12-22T01:45:53  <jshand> the release version is 20-12-2024
2024-12-22T02:07:06  <matrix-o-o> <Firstyear (Firstyear)> No, it's fine.
2024-12-22T02:30:57  <jshand> oh, ok. it must be under heavy load
2024-12-22T02:31:30  <jshand> it's too slow.  is the connection on fibre
2024-12-22T02:32:47  <matrix-o-o> <Firstyear (Firstyear)> It's on a gigabit connection.
2024-12-22T02:33:34  <matrix-o-o> <Firstyear (Firstyear)> I believe you will agree the load here looks fine.
2024-12-22T02:35:17  <jshand> if it's fine, why does it take hours to compltely sync th release
2024-12-22T02:36:43  <matrix-o-o> <Firstyear (Firstyear)> Because it doesn't _sync anything_. It's closer to a cdn where it "caches first hit". Normally the connection from that machine back to the "master" mirror is 100mbit to 200mbit. So any "slowness" is is from the master mirror on first hit.
2024-12-22T02:37:17  <matrix-o-o> <Firstyear (Firstyear)> Because the thing is jshand that most of the packages on the "mirrors" are never even accessed.
2024-12-22T02:37:27  <matrix-o-o> <Firstyear (Firstyear)> So syncing is a waste, only cache what's actually used.
2024-12-22T02:37:37  <matrix-o-o> <Firstyear (Firstyear)> We end up with a 90% or more hit rate because of this.
2024-12-22T02:38:22  <matrix-o-o> <Firstyear (Firstyear)> The setup is that anything that "is synced" to a mirror in aus, is directed to first, we only direct you to the mirror.fy.id.au for stuff that isn't "synced" locally, and then it caches it from the master
2024-12-22T02:40:13  <jshand> oh, ok.
2024-12-22T02:40:16  <jshand> interesting
2024-12-22T02:40:28  <matrix-o-o> <Firstyear (Firstyear)> Yeah, it's super effective in what it does actually.
2024-12-22T02:41:12  <matrix-o-o> <Firstyear (Firstyear)> There are other tricks it has too, like I coded in some "pattern" recognition where if you ask for the first repo file, we prefetch them all in the background to ensure their hot before zypper even makes the next request.
2024-12-22T02:41:18  <jshand> yeah, just tried doing a zypper dup and nothing has come down apart from the debug packages.  i'm in nz if that helps
2024-12-22T02:41:19  <matrix-o-o> <Firstyear (Firstyear)> It's why zypper ref is soooooo much faster now.
2024-12-22T02:41:58  <matrix-o-o> <Firstyear (Firstyear)> Are a lot of your packages from obs home repos perchance?
2024-12-22T02:42:56  <jshand> i don't think they are obs
2024-12-22T02:43:54  <matrix-o-o> <Firstyear (Firstyear)> Now, if you want to talk about a big problem we have, it's that whatever pushes content to other mirrors doesn't do it atomically.
2024-12-22T02:44:10  <matrix-o-o> <Firstyear (Firstyear)> slowroll?
2024-12-22T02:45:09  <jshand> i wish i knew what that problem is, i dont understand all this networking stuff
2024-12-22T02:47:49  <matrix-o-o> <Firstyear (Firstyear)> jshand: Oh the other thing is zypper still does some really bad behaviour wrt to connection management.
2024-12-22T02:47:53  <matrix-o-o> <Firstyear (Firstyear)> So that could be at play too
2024-12-22T02:47:55  <matrix-o-o> <malcolmlewis> jshand: Could be anywhere between NZ and the rest of the internet. You need to find someone there willing to setup/maintain a mirror. Even talk to your ISP about it...
2024-12-22T02:48:07  <matrix-o-o> <Firstyear (Firstyear)> I'm seeing no issues between mirror.fy.id.au to de though.
2024-12-22T02:48:29  <matrix-o-o> <Firstyear (Firstyear)> jshand: What variante of opensuse are you using?
2024-12-22T02:48:45  <jshand> well it seems my system is hooked into the firstyear server in aussie.
2024-12-22T02:48:55  <jshand> i don't think anyone in nz has a mirror
2024-12-22T02:49:09  <matrix-o-o> <Firstyear (Firstyear)> There are nz mirrors, but only for leap and some bits of tw iirc.
2024-12-22T02:49:32  <jshand> oh, probably won't help me
2024-12-22T02:49:39  <jshand> i guess i'll keep trying
2024-12-22T02:49:49  <matrix-o-o> <Firstyear (Firstyear)> jshand: The way it works is that zypper goes to mirrorcache-au.opensuse.org, and if a mirror like in nz has the file, you're sent there first. If _no mirrors_ in the region have the file, you fall back to mirror.fy.id.au as it can sync anything
2024-12-22T02:49:58  <matrix-o-o> <Firstyear (Firstyear)> jshand: What _version of opensuse do you use_.
2024-12-22T02:50:25  <matrix-o-o> <Firstyear (Firstyear)> And send me the output of "zypper lr"
2024-12-22T02:51:00  <jshand> i am using opensuse tumbleweed
2024-12-22T02:51:55  <matrix-o-o> <Firstyear (Firstyear)> So one of the only tw mirrors in the region is in nz.
2024-12-22T02:51:58  <matrix-o-o> <Firstyear (Firstyear)> So you should be sent there.
2024-12-22T02:52:52  <jshand> https://paste.opensuse.org/918db8f1c8ab
2024-12-22T02:53:21  <matrix-o-o> <Firstyear (Firstyear)> jshand: Right, so when you say "mirror.fy.id.au" is slow, how are you seeing that?
2024-12-22T02:54:21  <jshand> how do i get that information ie uris?
2024-12-22T02:54:37  <jshand> it seems to have fixed itself, it took a good 4 hours
2024-12-22T02:55:03  <matrix-o-o> <Firstyear (Firstyear)> How did you identify there was a problem in the first place? What were you seeing that made you think it was mirror.fy.id.au that was slow?
2024-12-22T02:55:16  <matrix-o-o> <Firstyear (Firstyear)> Like i want to know what you saw so I can actually work out what the problem is.
2024-12-22T02:57:12  <jshand> all i saw from the command line was zypper dup doing it's thing and getting no updates
2024-12-22T02:57:22  <jshand> no errors persey
2024-12-22T02:59:20  <matrix-o-o> <Firstyear (Firstyear)> jshand: It's better to work out and gather evidence, though I'll admin, zypper makes that hard. I think you need "zypper -v dup" to see the mirror it's using, and even then I think that can "lie" too because of how zypper does it's parallel downloads .... wireshark/tcpdump have sadly been the best ways to actually see what's up.
2024-12-22T03:03:09  <matrix-o-o> <malcolmlewis> jshand: run "time zypper -vvv ref -f" how long does that take?
2024-12-22T03:09:23  <jshand> https://paste.opensuse.org/cdbc7d4771e4
2024-12-22T03:11:59  <matrix-o-o> <malcolmlewis> jshand: do you need the debug repo enabled?
2024-12-22T03:12:17  <matrix-o-o> <malcolmlewis> jshand: Mine takes 15 seconds here on cdn...
2024-12-22T03:12:47  <jshand> oh i use the debug repo for debugging when apps crash
2024-12-22T03:13:15  <matrix-o-o> <malcolmlewis> jshand: so if you disable, does it speed up, only enable when needed
2024-12-22T03:14:09  <jshand> yes it does
2024-12-22T03:17:19  <matrix-o-o> <Firstyear (Firstyear)> malcolmlewis: Blocks updates.
2024-12-22T03:17:21  <matrix-o-o> <Firstyear (Firstyear)> That causes issues
2024-12-22T03:17:28  <matrix-o-o> <malcolmlewis> jshand: would suggest that as it's going elswhere for meta ddata
2024-12-22T03:17:48  <jshand> strange isn't it.
2024-12-22T03:18:02  <matrix-o-o> <Firstyear (Firstyear)> No, debuginfo isn't often synced to mirrors.
2024-12-22T03:18:07  <matrix-o-o> <malcolmlewis> Firstyear (Firstyear): so can you mirror the debug stuff too?
2024-12-22T03:18:13  <matrix-o-o> <Firstyear (Firstyear)> I do
2024-12-22T03:18:29  <matrix-o-o> <Firstyear (Firstyear)> Those repos happen to also have really large metadata etc.
2024-12-22T03:18:42  <matrix-o-o> <Firstyear (Firstyear)> And a lot of the cost isn't the dl, but zyppers slow local processing of the metadata.
2024-12-22T03:19:15  <matrix-o-o> <Firstyear (Firstyear)> Like here on my home network I have a 10gbit network, with my internal mirror on 10gbit. If everything is hot, and synced, there is 0 download time but zypper still takes forever to ref because it's so slow to process the metadata.
2024-12-22T03:20:52  <matrix-o-o> <Firstyear (Firstyear)> "zypper ref --force  13.32s user 1.83s system 61% cpu 24.597 total"
2024-12-22T03:20:58  <matrix-o-o> <Firstyear (Firstyear)> Almost all of that 25 seconds is zypper
2024-12-22T03:21:01  <matrix-o-o> <Firstyear (Firstyear)> Not network.
2024-12-22T03:21:38  <matrix-o-o> <Firstyear (Firstyear)> The other thing is that for jshand they'll still need to go via zypper for which repos to use etc, and that adds redirection and latency costs because zypper opens and closes a connection for every request.
2024-12-22T03:21:55  <matrix-o-o> <Firstyear (Firstyear)> * mirrorcache
2024-12-22T03:22:47  <matrix-o-o> <Firstyear (Firstyear)> So that adds a ton of time.
2024-12-22T03:23:56  <matrix-o-o> <malcolmlewis> So looks like a "It is what is is" thing...
2024-12-22T03:25:50  <matrix-o-o> <Firstyear (Firstyear)> More that there are many factors involved, and zypper is a lot of them.
2024-12-22T03:26:00  <matrix-o-o> <Firstyear (Firstyear)> I also assure you its better than it was
2024-12-22T03:26:08  <matrix-o-o> <Firstyear (Firstyear)> It used to take more than an hour to do a tw install in australia
2024-12-22T03:26:10  <matrix-o-o> <Firstyear (Firstyear)> now it's 20 minutes.
2024-12-22T03:26:28  <matrix-o-o> <Firstyear (Firstyear)> Because I sat down, profiled this all, raised issues, was ignored for months, then fixed it myself.
2024-12-22T03:28:26  <matrix-o-o> <Firstyear (Firstyear)> like why I spend a bunch of my own money to run mirrorcache's in areas outsid of europe.
2024-12-22T03:31:57  <matrix-o-o> <Firstyear (Firstyear)> malcolmlewis: There are quite a few stories of german SUSE employees coming to Australia, doing a zypper ref/dup and going "what the fuck, why is it that slow" and the australians turn around and say "this is a good day, it's quite fast". The difference is really staggering.
2024-12-22T03:32:53  <matrix-o-o> <Firstyear (Firstyear)> Today the fact that it only takes a minute to zypper ref is absolutely amazing
2024-12-22T03:33:02  <matrix-o-o> <Firstyear (Firstyear)> It used to be 3 or 4 minutes.
2024-12-22T03:34:16  <matrix-o-o> <malcolmlewis> Firstyear (Firstyear): the cdn network it trucking along here...
2024-12-22T03:34:26  <matrix-o-o> <Firstyear (Firstyear)> cdn is slower in au than not.
2024-12-22T03:34:33  <matrix-o-o> <Firstyear (Firstyear)> I tested it extensively when it was annouced.
2024-12-22T03:36:17  <matrix-o-o> <malcolmlewis> Firstyear (Firstyear): I only run one Tumbleweed machine, well Aeon and MicroOS, the rest are Leap systems
2024-12-22T03:36:27  <matrix-o-o> <Firstyear (Firstyear)> I can sit down and retest it sometime.
2024-12-22T03:36:56  <matrix-o-o> <Firstyear (Firstyear)> Leap has it's own issues, like drpms being slower than just downloading the packages _even here_ in aus/nz where dl speeds are terrible.
2024-12-22T03:37:03  <jshand> cool
2024-12-22T03:37:31  <matrix-o-o> <malcolmlewis> jshand: should be updated now....?
2024-12-22T03:38:08  <jshand> yeah, i got it all successfully.  thanks heaps.  i guess we are not europe lol
2024-12-22T03:38:14  <matrix-o-o> <malcolmlewis> Tumbleweed/Aeon and MicroOS/Hyprland all updated, nvidia drivers rebuilt etc
2024-12-22T03:38:23  <matrix-o-o> <Firstyear (Firstyear)> jshand: If you want to skip all the geoip chekcs, "sed -i -E 's/https?:\/\/download.opensuse.org/https:\/\/mirrorcache-au.opensuse.org/g' /etc/zypp/repos.d/*.repo "
2024-12-22T03:38:35  <matrix-o-o> <Firstyear (Firstyear)> That will change your mirrors to all be going via the australia redirector.
2024-12-22T03:38:55  <matrix-o-o> <Firstyear (Firstyear)> jshand: That's the underlying theme of this. If you aren't in europe, you don't matter. :(
2024-12-22T03:39:11  <matrix-o-o> <malcolmlewis> I'm in the US...
2024-12-22T03:39:35  <matrix-o-o> <Firstyear (Firstyear)> The US had tons of it's own issues too. I also worked to help fix them.
2024-12-22T03:39:45  <matrix-o-o> <Firstyear (Firstyear)> I run two mirror redirectors in us to help with your latency issues too
2024-12-22T03:40:01  <matrix-o-o> <Firstyear (Firstyear)> I also wrote a tool called mirrorsorcerer to let your machine automatically find the nearest mirror redirector and use that
2024-12-22T03:40:03  <matrix-o-o> <malcolmlewis> at least iso's download quick, around 2 minutes
2024-12-22T03:41:29  <matrix-o-o> <Firstyear (Firstyear)> The US has a better start by simply having more population -> more mirrors to choose from
2024-12-22T03:42:02  <matrix-o-o> <Firstyear (Firstyear)> Australia/nz are first, much much further away, and second, have the same land mass as US, but with only 30 million people, tops.
2024-12-22T03:42:33  <matrix-o-o> <Firstyear (Firstyear)> So we have less mirrors, and further to "travel" when we need to download something even locally, let alone to sync to eu etc.
2024-12-22T03:42:54  <matrix-o-o> <Firstyear (Firstyear)> malcolmlewis: What was bonkers in zypper would go to germany for every http request before, for every file and every package.
2024-12-22T03:43:01  <jshand> this might be the wrong channel.  how do i check for zombie processes
2024-12-22T03:43:15  <matrix-o-o> <Firstyear (Firstyear)> Of an install of tw, from 1 hour of it, 31 minutes was simply _waiting for a redirection from europe_
2024-12-22T03:43:47  <matrix-o-o> <Firstyear (Firstyear)> jshand: ps aux | grep 'Z'
2024-12-22T03:47:15  <jshand> yeah just found out.  i have an issue [zypak-sandbox] <defunct>
2024-12-22T03:47:29  <jshand> that seems to be a zombie
2024-12-22T03:47:44  <jshand> i do have flatpak installed running ferdium
2024-12-22T03:56:33  <matrix-o-o> <malcolmlewis> Firstyear (Firstyear): That's why I like MicroOS, just put the image onto the target medium, plug in my Combustion USB, boot, go make a coffee and all done...
2024-12-22T03:59:16  <matrix-o-o> <Firstyear (Firstyear)> I wish it could be that simple
2024-12-22T03:59:24  <matrix-o-o> <Firstyear (Firstyear)> But my issues with tech are a different problem.
2024-12-22T09:57:48  *** teepee_ is now known as teepee
2024-12-22T15:00:19  <acidsys> cboltz: try `salt mickey.infra.opensuse.org service.restart kanidm-unixd`
2024-12-22T15:01:06  <cboltz> the solution was more boring - it fixed itsself in the meantime
2024-12-22T15:01:21  <acidsys> also good. there is a poo for it
2024-12-22T15:01:44  <cboltz> I know, that's why I asked if you wanted to debug the broken state
2024-12-22T15:02:53  <acidsys> ah, thanks
2024-12-22T15:04:04  <cboltz> different topic - as far as I can tell, !2230 is ready for merging
2024-12-22T15:04:30  <cboltz> I didn't want to click "merge" for such a big change shortly before xmas, but I won't stop you ;-)
2024-12-22T15:45:51  <acidsys> thanks, I will check in a while
2024-12-22T20:13:24  *** teepee_ is now known as teepee
2024-12-22T20:16:07  <cboltz> FYI: I just deleted hundreds of non-member spam mails from commit@lists.o.o - and changed the rule for non-members to "reject" (was "hold")
2024-12-22T20:18:35  <cboltz> https://lists.opensuse.org/manage/lists/mirror.lists.opensuse.org/held_messages has (after deleting the spam) some "real" messages that are quite old, but might still be relevant for us (for example, mirror applications)
2024-12-22T20:19:25  <cboltz> can a mirror admin please have a look at them? (I'll happily add you as moderator to give you access.)