2023-03-23T04:30:56  *** teepee_ is now known as teepee
2023-03-23T04:46:51  *** Bitten[m] is now known as Bitten2949[m]
2023-03-23T08:44:16  <phoenix[m]> Hello Heroes, we still have some location unaccessible issues with download.opensuse.org on O3. Is there nothing what we can do about this?
2023-03-23T08:47:12  <bmwiedemann[m]> IIRC when I operated O3 in 2011, I had my own local rsync mirror to ensure reliability.
2023-03-23T08:49:39  <phoenix[m]> But isn't it that users are also affected, if O3 is affected as well? If O3 would be the culprit that causes those load issues I would fully agree, but O3 here is only the messenger.
2023-03-23T08:49:59  <bmwiedemann[m]> yes. They are.
2023-03-23T08:50:09  <phoenix[m]> I'm also willing to bring this up with our managers, if we need more/bigger hardware. I'm searching for a solution here 🙂
2023-03-23T08:50:44  <phoenix[m]> Just saying, if this happens when there is a big TW rebuild, we have reasons to believe this won't be a one time issue and we should do something about it, no?
2023-03-23T08:52:52  <bmwiedemann[m]> Part of the trouble is that nobody feels responsible. I had already setup some things that helped reduce the impact, but it seems not enough.
2023-03-23T08:54:01  <bmwiedemann[m]> Much of the load comes from rsync traffic and we could have most people rsync from the copy on stage3.o.o
2023-03-23T08:54:48  <phoenix[m]> Because download.opensuse.org is "just a mirror" of stage3.o.o?
2023-03-23T08:55:32  <bmwiedemann[m]> no. download.o.o = stage.o.o - and that is the central place from where 100 mirrors pull their repos.
2023-03-23T08:57:07  <pjessen> fwiw, I have just switched my mirror to stage3.  a drop in the ocean.  I second bmws opinion, the problem is that nobody feels responsible.
2023-03-23T08:58:32  <bmwiedemann[m]> There are some individuals like me, Andrii or per, but on the corporate level it is not recognized much.
2023-03-23T08:59:07  <pjessen> I used to look after it, but after the big re-engineering, i gave it up.
2023-03-23T10:13:42  <pjessen> I'm getting steady time-outs from stage3.o.o - my mirror is heavily loaded too, but I'm not even getting a drop from stage3.o.o
2023-03-23T10:14:40  <bmwiedemann[m]> with rsync rsync://stage3.opensuse.org/opensuse-full/ ?
2023-03-23T10:15:08  <pjessen> yeah, something like that.
2023-03-23T10:15:36  <bmwiedemann[m]> maybe some IPv6 routing issue?
2023-03-23T10:15:46  <pjessen> opensuse-full-really-everything, but with excludes
2023-03-23T10:15:59  <bmwiedemann[m]> or do you have MTU < 1500?
2023-03-23T10:16:22  <pjessen> nah, 1500 :-)
2023-03-23T10:16:33  <pjessen> I can ping it on ipv6
2023-03-23T10:17:32  <bmwiedemann[m]> and does rsync -4 rsync://stage3.opensuse.org/opensuse-full-really-everything/ work better?
2023-03-23T10:18:10  <pjessen> I haven't tried that yet - was going to be my next step.  Anyway, gotta go, be back after lunch.
2023-03-23T10:18:33  <bmwiedemann[m]> Guten Appetit and ttyl
2023-03-23T11:08:21  <acidsys> if corporate would care we would use SLE CDN for openSUSE :]
2023-03-23T11:12:32  <LubosKocman[m]> Hello team I do see a request to install copilot4prs in openSUSE on github
2023-03-23T11:13:19  <LubosKocman[m]> If it's related to AI generated code, I'd like to ask you not to install it. https://opensource.suse.com/legal/policy as of this moment forbids usage of AI generated code for SUSE employees.
2023-03-23T11:13:46  <LubosKocman[m]> We expect that this will change by end of the year, but specifically copilot is too risky at the moment.
2023-03-23T11:14:08  <LubosKocman[m]> If it's not GitHub copilot related, then please ignore my message
2023-03-23T11:14:30  <acidsys> +1 but maybe send it to heroes@ as well as not everyone checks chat backlog
2023-03-23T11:15:08  <acidsys> though I'm not sure who all is gh org maintainer
2023-03-23T11:15:34  <cboltz> that request came from @HollowMan6 so maybe contact him directly? https://github.com/HollowMan6 points to hollowman@o.o
2023-03-23T11:35:24  <pjessen> we have had offers of trying out CDN - cdn77 I think was one.
2023-03-23T11:36:30  <bmwiedemann[m]> I found their cname under the cdn.o.o entry, but it did not resolve.
2023-03-23T11:37:01  <cboltz> right (but that cdn77 offer was several years ago)
2023-03-23T11:37:32  <bmwiedemann[m]> one trouble is that download.o.o is not designed for caching proxies atm, so you get into problems of consistency if repomd.xml does not match the repomd.xml.asc signature
2023-03-23T11:38:15  <pjessen> bmwiedemann[m]: I left a tcpump running on the ipv6 addr of stage3 - I see *some* traffic, but with loooooong breaks.  (5minutes right now)
2023-03-23T11:39:11  <bmwiedemann[m]> not many are rsync'ing from there. Maybe you see the traffic where it pulls from stage?
2023-03-23T11:39:55  <pjessen> no, definitely opnly stage3
2023-03-23T11:40:35  <pjessen> weird with those long breaks, if its not busy.
2023-03-23T11:40:37  <bmwiedemann[m]> provo-mirror and opensuse-br-mirror pull from there regularly.
2023-03-23T11:41:49  <bmwiedemann[m]> There is also another IPv6 addr for some share of downloadcontent2. /var/log/nginx/access.log shows activity about it.
2023-03-23T11:42:36  <pjessen> hmm, I think maybe it has actually finished now.  another sync is scheduled in 5mins
2023-03-23T11:43:17  <bmwiedemann[m]> did you rsync with -v or -P so you see progress during sync?
2023-03-23T11:43:59  <bmwiedemann[m]> when you start rsync, it might also hang a while, because it checks all file mtimes and sizes
2023-03-23T11:44:10  <bmwiedemann[m]> depending on your options
2023-03-23T11:44:50  <pjessen> for now, it just runs in the background. I'll disable that for the next run
2023-03-23T11:47:03  <pjessen> yes, at first it is getting the filelist, but that would show both network traffic and strace activity. 774176 files to consider. now apparently taking a break.
2023-03-23T11:47:16  <pjessen> or checking files
2023-03-23T11:47:40  <bmwiedemann[m]> can you tell me your full rsync line?
2023-03-23T11:48:48  <pjessen> yup. two secs
2023-03-23T11:49:54  <pjessen> rsync -av --progress --delay-updates --delete-delay --no-motd  --exclude-from=../opensuse3.excludes stage3.opensuse.org::opensuse-full-really-everything/opensuse/ .
2023-03-23T11:50:41  <bmwiedemann[m]> looks just right.
2023-03-23T11:51:13  <pjessen> :-)
2023-03-23T11:51:45  <pjessen> hmm, it might even by my machine that is being slow.
2023-03-23T11:52:01  <pjessen> it is being hammered by clients from ch, es, it, fr
2023-03-23T12:00:04  <pjessen> I think I'll say it's my machine - loadavg 200/400/300 on a dual core :-)
2023-03-23T13:58:54  *** teepee_ is now known as teepee
2023-03-23T15:41:23  <acidsys> malcolmlewis: please do not set the priority of account change requests to "High". "High" for me means a service outage impacting several users. a single user wanting a cosmetic change is the lowest priority for me.
2023-03-23T15:42:45  <malcolmlewis> acidsys, I assigned a person... why not? The user needs help...
2023-03-23T15:43:58  <malcolmlewis> to me the priority needs to be defined according to the service, does it not?
2023-03-23T15:45:24  <acidsys> I take care of these requests voluntarily because I just so happen to have the access to do it. the priority should reflect the importance of the request, and a single change request is low priority in terms of service requests. the priority should be raised if there is a service outage.
2023-03-23T15:45:43  <acidsys> the users personal needs do not justify me rushing to my work machine :)
2023-03-23T15:46:07  <acidsys> I understand you want to get help to the person
2023-03-23T15:49:09  <malcolmlewis> acidsys, if I set the category correct, you should not get it?
2023-03-23T15:49:27  <malcolmlewis> I didn't set that
2023-03-23T15:50:16  <malcolmlewis> acidsys, I have now
2023-03-23T15:55:25  <acidsys> I get all tickets assigned to me or groups I am in, category being correct is a nice bonus but doesn't change much (only some categories have a default assignee)
2023-03-23T15:56:15  <acidsys> thanks for adding it
2023-03-23T15:59:32  <malcolmlewis> ok
2023-03-23T17:22:15  *** teepee_ is now known as teepee
2023-03-23T18:30:48  <Guest2> hello there, are you aware of the issues with download.opensuse.org?
2023-03-23T18:30:57  <Guest2> receiving HTTP 502 errors
2023-03-23T18:31:05  <Guest2> in general very slow
2023-03-23T18:31:32  <bmwiedemann[m]> yes. There is still heavy load from the last TW full rebuild.
2023-03-23T18:31:56  <bmwiedemann[m]> and our new mariadb VM was not up to the job
2023-03-23T18:33:48  <Guest2> im sorry, what is TW? I'm just a user trying to download some packages from download.opensuse.org through apt-get and receiving a lot of 502 :D
2023-03-23T18:34:07  <bmwiedemann[m]> TW=Tumbleweed
2023-03-23T18:34:43  <bmwiedemann[m]> oh, kernel problems. I guess, that means another 2m downtime of download.o.o
2023-03-23T18:36:06  <Guest2> any idea when things will go back to normal? its been 2 days now i think.
2023-03-23T18:36:31  <bmwiedemann[m]> maybe in 5m
2023-03-23T18:38:00  <Guest2> fingers crossed then :)
2023-03-23T18:39:18  <bmwiedemann[m]> try now
2023-03-23T18:40:16  <Guest2> wow, its a lot faster now I thought you were joking when you said in 5m
2023-03-23T18:41:10  <bmwiedemann[m]> will probably get slower again soon, once more users return.
2023-03-23T18:41:43  <Guest2> yep i just hit a 502 after 25 requests
2023-03-23T18:44:39  <Guest2> so is there a permanent fix planned?
2023-03-23T18:45:37  <bmwiedemann[m]> In June, there should be new faster hardware in the Prague DC to take over
2023-03-23T18:46:14  <bmwiedemann[m]> until then, I hope, the current spike will pass soon
2023-03-23T18:47:03  <bmwiedemann[m]> http://stage3.opensuse.org:17080/munin/opensuse.org/stage.opensuse.org/apache_accesses.html shows the impact of kernel crash
2023-03-23T20:03:29  <acidsys> speaking of prague dc
2023-03-23T20:04:11  <bmwiedemann[m]> any news there?
2023-03-23T20:05:01  <acidsys> openSUSE hardware mounted and labelled :) http://hugz.io/3m8dgbx/PRG2_oS_IMG_0550.jpg
2023-03-23T20:06:15  <acidsys> maybe I should have taken a picture from the back as well, it has these fancy PDU's
2023-03-23T20:08:27  <igorjagec78> hi there :)
2023-03-23T20:08:32  <ddemaio> hi
2023-03-23T20:09:09  <igorjagec78> irssi, how do you pronounce this?
2023-03-23T20:09:47  <acidsys> ei arr ess ess aye
2023-03-23T20:11:18  <bmwiedemann[m]> https://en.wikipedia.org/wiki/Irssi says [ˈirsːi]
2023-03-23T23:23:09  *** teepee_ is now known as teepee