2024-05-28T03:50:57 can somebody go feed the hamsters over at code.o.o? 2024-05-28T03:58:06 SFaulken: seems to be back already. But it is mysterious that service restarts take so long there... 2024-05-28T03:58:25 yeah, it seems to come and go, the last few days. 2024-05-28T05:43:38 pagure has strange log entries https://paste.opensuse.org/pastes/2199d79e6b51 2024-05-28T06:08:57 That pagure issue looks like this patch https://pagure.io/pagure/pull-request/5481#request_diff but in an early version were it was `cmd.extend` and not `cmd.append`. 2024-05-28T06:10:51 Yes. I see "cmd.extend("--end-of-options")" in /usr/lib/python3.6/site-packages/pagure/lib/repo.py 2024-05-28T06:11:19 Replace it with append and it works 2024-05-28T06:12:11 OK. hot-patched it for now 2024-05-28T06:12:20 do I need to restart something? 2024-05-28T06:13:10 And if you are brave, I released 5.14.1 last Friday https://pagure.io/pagure/releases which needs at least under EL8 those patches from yesterday too https://pagure.io/pagure/pull-request/5486 2024-05-28T06:13:44 No should be fine without a restart 2024-05-28T06:30:48 *** teepee_ is now known as teepee 2024-05-28T06:39:33 Now that I think about it, maybe the pagure_worker.service, but tbh not 100% sure. 2024-05-28T08:41:12 code.o.o seems down again atm with a 504 atleast from here in Australia 2024-05-28T08:42:41 In the journal I see some "May 28 08:41:44 pagure01 go-mmproxy[26883]: {"time":"2024-05-28T08:41:44.108224194Z","level":"DEBUG","msg":"connection broken","listenerNum":0,"protocol":"tcp","listenAdr":"[2a07:de40:b27e:1206::a]:2222","remoteAddr":"[2a07:de40:b27e:1204::11]:57418","localAddr":"[2a07:de40:b27e:1206::a]:2222","clientAddr":"UNKNOWN","targetAddr":"[::1]:2222","error":"readfrom tcp [::1]:44068->[::1]:2222:... 2024-05-28T08:42:47 ... splice: connection reset by peer","dropConnection":true}" 2024-05-28T09:19:09 Dominik Wombacher (wombelix): would you also have a hint for https://paste.opensuse.org/pastes/e87fdf6ae07b ? 2024-05-28T09:19:20 this could be behind the 504 gateway timeout problems 2024-05-28T09:21:57 that's pretty generic :-/ any other errors or hints in the logs? I guess the worker could run into any kind of exception in the background and therefore timeout 2024-05-28T09:22:33 nothing in the journal about pagure_worker.service 2024-05-28T09:29:40 Hmm maybe activating debug logs gives the necessary insights? https://docs.pagure.org/pagure/configuration.html?highlight=debug#logging how is the logging configured for code.o.o right now? 2024-05-28T09:41:21 I see various DEBUG messages from pagure in journalctl 2024-05-28T09:41:48 e.g. "May 28 09:41:36 pagure01 gunicorn[26016]: 2024-05-28 09:41:36,795 [DEBUG] pagure.lib.repo: Output: 7cc35babef7 Rename patches to the unified patches.suse directory" 2024-05-28T09:43:35 but OTOH I don't see any LOGGING entry in /etc/pagure/pagure.cfg 2024-05-28T09:46:03 nginx log has "2024/05/28 09:45:03 [error] 1477#1477: *7792859 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 64.71.166.35, server: code.opensuse.org, request: "GET /package/targetcli-fb/branches?branchname= HTTP/1.1", upstream: "http://unix:/srv/gitolite/.pagure_web.sock/package/targetcli-fb/branches?branchname=", host: "code.opensuse.org"" 2024-05-28T09:47:10 oh and ClaudeBot/1.0; +claudebot@anthropic.com is doing requests 2024-05-28T10:02:03 hmm no sorry, based on those entries not sure what causes the celery worker timeout :-/ Probably unrelated, but I see 'gitolite' in the path, do you still use it as backend? Asking because gitolite support is dropped in the major release 6.0 we work on, pagure becomes the new default backend. 2024-05-28T10:15:23 FWIW it's already unstable since some days: https://progress.opensuse.org/issues/160925 2024-05-28T10:16:31 the mmproxy is just for SSH access to Git, the "connection broken" log entries are from the HAProxy health checks 2024-05-28T10:57:17 acidsys: maybe https://gitlab.infra.opensuse.org/infra/salt/-/merge_requests/1890 helps 2024-05-28T11:16:40 Claudebot fetches many "time curl https://code.opensuse.org/kernel/kernel-source/history/patches.suse/crypto-chelsio-Fix-wrong-error-counter-increments.patch?identifier=17d45d2425238980bdd2c6115b6881488f06f0ee" and that gets a timeout after 31s. 2024-05-28T11:19:39 what is that file/path? it seems more problematic that there is such a path which brings down the server if requested to often than some tool requesting it 2024-05-28T11:22:22 too 2024-05-28T11:30:20 I guess, the kernel-source repo is just very large and pagure calls "git log --pretty=oneline --abbrev-commit --end-of-options $commit -- $file" on it 2024-05-28T11:31:16 and whenever the server is stuck, top shows 8 processes around 100% - 4x git log and 4x pagure 2024-05-28T11:36:51 sounds inefficient 2024-05-28T11:41:01 I will start pagure_mirror_project_in.timer again (I stopped it yesterday because it causes a load spike about every hour) 2024-05-28T11:42:33 it also complains about not reaching some external git services like framagit.org, which is expected but so far nobody requested this / complained about it 2024-05-28T11:45:26 but do you have an explanation why this problematic URL only started to become a problem some days ago 2024-05-28T11:46:06 see how it was quite low before the 23rd https://monitor.opensuse.org/grafana/goto/H23xHisSg 2024-05-28T11:50:25 I've seen a few reports of obsolete repodata pointers on slack https://progress.opensuse.org/issues/161078 2024-05-28T11:50:48 Not sure if there is anything else than cleaning cache that we can do 2024-05-28T11:54:39 There were four security issues in Pagure and my understanding is they were hot patched in the code.o.o instance. My guess is this was around the 23rd? When we know what changes were applied around that time it might easier to identify what causes the problem. I mean the history thing seems pretty obvious but I don't have a good explanation why it started to become a problem some days ago. The... 2024-05-28T11:54:45 ... general implementation is in Pagure since forever 2024-05-28T11:55:52 good point, I also thought of it correlating with the patches from Neal, but he didn't share what exactle was changed :D 2024-05-28T12:08:08 cboltz: ping (invite to -factory?) - we have some trouble around AA 4 / Factory, where you can possibly help with fixes to unblock TW 2024-05-28T12:13:35 https://pagure.io/pagure/pull-requests?status=Merged 5481 5482 5383 and 5484 are the security fixes. 5481 was at least applied by Neal, not sure about the others. 2024-05-28T12:13:56 * https://pagure.io/pagure/pull-requests?status=Merged, 2024-05-28T12:25:51 at least we know, which files were changed: https://paste.opensuse.org/pastes/06a70063f568 2024-05-28T13:31:02 Yeah I'm guessing the changes are mostly related to the PRs we were asked to include in a new 5.x release :) https://pagure.io/pagure/issue/5473#comment-911229 2024-05-28T13:31:59 So yeah, I'm happy to help to make code.o.o more reliable and fix problems, let's see if I can figure out something with the information we have so far 2024-05-28T13:35:50 What's the content of that MR you mentioned before? (https://gitlab.infra.opensuse.org/infra/salt/-/merge_requests/1890), I don't have access to the opensuse infra, maybe that should be changed so I can work directly on your pagure issues. 2024-05-28T13:36:15 it's mirrored to the public https://code.opensuse.org/heroes/salt/commits/production :) 2024-05-28T13:36:33 f3r5c0 is from !1890 2024-05-28T13:36:51 I added monitoring in my http://nagios.zq1.de/cgi-bin/nagios4/status.cgi?host=oopagure 2024-05-28T13:40:06 > it's mirrored to the public https://code.opensuse.org/heroes/salt/commits/production :) 2024-05-28T13:40:08 That's cool :) But I guess I'm blind, don't see f3r5c0, from when was that commit? 2024-05-28T13:40:27 f345c0 2024-05-28T13:40:43 (pagure does not let me select the text under this hyperlink with my mouse so I typed it off ... 2024-05-28T13:40:44 > I added monitoring in my http://nagios.zq1.de/cgi-bin/nagios4/status.cgi?host=oopagure 2024-05-28T13:40:46 In case you wanted to share it, access forbidden 2024-05-28T13:41:34 only when you try https there 2024-05-28T13:41:35 I know, stumbled across the same already, it's one of the million items on my personal pagure backlog :D 2024-05-28T13:41:53 yeah ok applying a robots file doesn't make it worse 2024-05-28T13:42:24 indeed, thanks chrome for automatically open it with https 2024-05-28T13:46:38 yeah so I don't see any changes in your salt stuff related to pagure that would explain the problems you see since ~1 week. So either something was done outside of Salt or it's a change from outside. Going back to the worker timeout. If something happens its ~30 seconds in the haproxy logs right? So if a request takes longer we seem to have this timeout? 2024-05-28T13:47:40 it times out in nginx 2024-05-28T13:47:54 we could increase the timeout but I think 30 seconds is already very long for a request 2024-05-28T13:48:13 or at least longer than an average user would wait for their tab to load 2024-05-28T13:49:38 yep that's true. Do you see the path of those requests that timeout? is it always https://code.opensuse.org/*/*/history/* ? 2024-05-28T13:54:57 btw, interacting with that (large?!) kernel source repo seem in general challenging. 2024-05-28T13:55:17 well yeah the repo is 4GB lol 2024-05-28T13:55:20 or at least, upstream linux kernel is 2024-05-28T13:55:56 :D right 2024-05-28T13:56:04 hey you get free stress testing 2024-05-28T13:58:09 `grep 'timed out' /var/log/nginx/error.log|grep -Po '\Krequest:(.*)'|sort -u` suggests it's not always just history 2024-05-28T14:00:38 *** teepee_ is now known as teepee 2024-05-28T14:02:23 yep make sense after the clone test 2024-05-28T14:29:31 I would make the assumption that it's something in the nginx/haproxy/infrastructure config. cloning a large repo on code.o.o fails, on pagure.io it works https://paste.opensuse.org/pastes/c07bbb785796 2024-05-28T14:29:56 which gives me the impression that pagure itself can handle it 2024-05-28T14:30:30 nginx is only involved with https, is it reproducible with ssh ? 2024-05-28T14:30:31 and yeah, a hard 30sec timeout in nginx seems problematic when dealing with large repos 2024-05-28T14:31:04 right with cloning it makes sense .. but with web gui requests? 2024-05-28T14:32:25 > nginx is only involved with https, is it reproducible with ssh ? 2024-05-28T14:32:25 Don't think I have the permissions to clone via ssh, let me see 2024-05-28T14:33:20 > right with cloning it makes sense .. but with web gui requests? 2024-05-28T14:33:20 My understanding is that the web ui makes problems when interacting with large repos? Then yes, that could make sense then. Let me see if I can trigger a long running web ui request on a large pagure.io repo 2024-05-28T14:45:21 I think so from what bmwiedemann1 wrote .. and the nginx log suggests as well that sometimes the backend does not respond. which might correlate with the gunicorn worker crashes 2024-05-28T14:49:22 OK i've must have miss that. I thought celery runs into a timeout but this is actually related to nginx that drops after 30sec. 2024-05-28T14:56:27 https://pagure.io/freebsd-src and https://pagure.io/linux-ark are fairly large repos on pagure.io and so far I don't see comparable problems when interacting with them. Let me do some more tests. The only thing I want to say: It's probably worth to understand the parts around pagure in the opensuse infra better :) 2024-05-28T15:02:24 code.o.o HTTPS flow: border -> atlas{1,2}.i.o.o (HAProxy) -> pagure01.i.o.o (nginx) -> local socket (gunicorn) 2024-05-28T15:03:03 code.o.o SSH flow: border -> atlas{1,2}.i.o.o (HAProxy) -> pagure01.i.o.o (go-mmproxy) -> local/dedicated sshd 2024-05-28T15:04:05 the HAProxy part is mostly transparent, it does not impose any limits 2024-05-28T15:05:03 https://monitor.opensuse.org/grafana/goto/W_u3aisIg 2024-05-28T15:06:40 the nginx part is rather .. generic? like any other reverse proxy in front of a wsgi backend 2024-05-28T15:07:48 the nginx configuration is here: https://code.opensuse.org/heroes/salt/blob/production/f/pillar/role/pagure.sls 2024-05-28T15:07:53 the pagure configuration is here: https://code.opensuse.org/heroes/salt/blob/production/f/salt/profile/pagure/files 2024-05-28T15:08:48 I think that's mostly all there is to it 2024-05-28T15:09:07 now monitoring complains 15:06 -- Notice(heroes-monitor): Alert: Host pagure01.infra.opensuse.org is low on disk space 2024-05-28T15:10:32 and magically already resolved. seems like someone briefly created a large file 2024-05-28T15:11:10 I bet that was me, I wanted to see what happens when I fork kernel-source. 2024-05-28T15:11:51 one thing I realized, you still use gitolite as backend 2024-05-28T15:12:36 so let's see how it takes to apply the ACLs ... https://code.opensuse.org/fork/wombelix/kernel/kernel-source 2024-05-28T15:13:12 +long 2024-05-28T15:13:24 I think partially? there is a pagure-gitolite-worker service which is stopped because Neal said we no longer use this. but I think some other gitolite parts are arctie 2024-05-28T15:13:29 active* 2024-05-28T15:14:09 think so too, because my fork says "The permissions on this repository are being updated. This may take a while. During this time, you or some of the project's contributors may not be able to push to this repository." which is related to gitolite and that the acl files get updated in the background by a celery worker 2024-05-28T15:15:04 the celery is printing interesting output too 2024-05-28T15:15:39 https://paste.opensuse.org/pastes/ba556b7ebb1e ugh why is it using /tmp on the root disk 2024-05-28T15:16:01 there is /srv/gitolite/tmp which is on a separate disk which it uses for some things but apparently not all 2024-05-28T15:16:36 ah it seems pagure-mirror_in uses /srv/gitolite/tmp, but pagure-fork uses /tmp 2024-05-28T15:16:50 can I change this? 2024-05-28T15:17:51 And /tmp was briefly running out of space I assume? 2024-05-28T15:18:29 yep (well, / did - /tmp is on the same file system) 2024-05-28T15:18:53 ok that makes sense and probably explains why git clone returned an error back to celery and failed 2024-05-28T15:19:36 so I would put that under "works as designed" then, if it would've have run out of space the fork probably would went through 2024-05-28T15:19:45 is there a native way to have it use a different directory? 2024-05-28T15:22:09 valid question, didn't saw anything yet to change temp folders, only the target location, let me check that 2024-05-28T15:22:45 okay, if not, I can add a BindPaths to the systemd unit as a workaround 2024-05-28T15:30:31 Doesn't seem to be customizable right now: https://pagure.io/pagure/blob/master/f/pagure/lib/git.py#_977 2024-05-28T15:31:20 ah, thanks for checking (I found tmpdirname = tempfile.mkdtemp() in tasks_services.py which does not seem to have a customizable dir= either) 2024-05-28T15:31:30 But I get the feeling we should allow to customize it so you can have it on a separate partition or something to avoid that your disk runs out of space 2024-05-28T15:31:30 guess another TODO ;-) let me add the bind mount 2024-05-28T15:32:26 * acidsys cries as `systemctl edit` reveals $EDITOR on this machine is nano 2024-05-28T15:32:30 yeah so the function would support a directory path as additional parameter. so I think that's worth a new config value that we pass to that function if it's set 2024-05-28T15:33:05 agree (and it can just default to None so that's handy) 2024-05-28T15:33:08 seem to be the same for mirroring too, there is also a temporary repo used 2024-05-28T15:35:38 added "BindPaths=/data/pagure/tmp:/tmp:norbind" to pagure_worker for now and restarted it. can we try again ? :) (I would but not sure what to click) 2024-05-28T15:37:01 https://pagure.io/pagure/issue/5488 2024-05-28T15:37:36 thanks for tracking it 2024-05-28T15:38:08 fun how login to id.fedoraproject.org now times out as well after clicking "log in" :D 2024-05-28T15:38:11 Looks like we identified a bug, because I have a fork now which is in an unhealthy state https://code.opensuse.org/fork/wombelix/kernel/kernel-source and in combination with gitolite that blocks it for ACL updates I can't even delete it 2024-05-28T15:38:53 oh you mean the broken fork earlier left a corrupt repository behind? 2024-05-28T15:39:47 yep, I have a fork but as we saw the fork doesn't have an actual repo, but in the UI it exists but I can't delete it in the settings 2024-05-28T15:39:53 nice zombie :-/ 2024-05-28T15:40:13 ouch 2024-05-28T15:40:14 yeah, I ran into that a few months back. 2024-05-28T15:40:27 can I delete it in the backend somehow? 2024-05-28T15:41:01 right I recall I did something similar for SFaulken. now how did I do that 2024-05-28T15:41:37 yeah whatever you did back then would be cool now :D I have to admit, idk how to do it, maybe via the pagure admin cli 2024-05-28T15:41:58 I realize I focused much more on the pagure code than pagure operations in the last time 2024-05-28T15:42:27 dev without ops 2024-05-28T15:44:10 so if someone want to try what happens when you fork the monster, login to code.o.o go to https://code.opensuse.org/kernel/kernel-source and click on the fork button :D 2024-05-28T15:44:43 this should then create a celery background worker tasks and a git clone command that runs and puts files into /tmp which now should be /data/pagure/tmp 2024-05-28T15:44:51 I found what I did - delete the directory and then "SELECT id,namespace,name FROM projects WHERE namespace = 'sfalken' AND name = 'Kalpa';" 2024-05-28T15:44:58 the problem is the fork does not show up in the projects table 2024-05-28T15:45:02 acidsys: there it is. 2024-05-28T15:45:09 SFaulken was a regular repository 2024-05-28T15:45:18 yeah. wasn't a fork. 2024-05-28T15:46:46 I think they are also in the projects table and have the flag "is_fork" so the namespace should then "forks" i guess? 2024-05-28T15:46:59 or "forks/wombelix" not exactly sure how it is put into the database 2024-05-28T15:47:14 hmm I did search for namespace='forks' but that has 0 results 2024-05-28T15:47:29 interesting, the fork is gone https://code.opensuse.org/fork/wombelix/kernel/kernel-source 2024-05-28T15:47:52 oh cool I checked after my `rm` and it was still there, but maybe some background task picked it up automagically 2024-05-28T15:48:04 might also explain why I no longer find it in the table :D 2024-05-28T15:48:06 but pagure still think I have one, I have a "view fork" button on https://code.opensuse.org/kernel/kernel-source :-/ 2024-05-28T15:48:13 oh 2024-05-28T15:48:48 I tried to look in user_projects but there are only ID numbers. so I try to find your user ID, but there is no user=wombelix in users 2024-05-28T15:48:50 * acidsys is very confused 2024-05-28T15:48:57 maybe there is still something in the table, maybe just search for kernel-source as name? that should return the main project and all forks 2024-05-28T15:49:23 https://paste.opensuse.org/pastes/0266d751fb5a 2024-05-28T15:49:30 there are three in namespace=kernel 2024-05-28T15:49:51 can you include the field "is_fork"? 2024-05-28T15:50:01 ooh! 2024-05-28T15:50:04 I assume for two it's "true" 2024-05-28T15:50:25 nice right so now I just need to find your user_id 2024-05-28T15:50:57 ok you are 341 2024-05-28T15:51:34 yep so I found it with "SELECT * FROM projects WHERE name = 'kernel-source' AND is_fork = 't' AND user_id = 341;" and now sent DELETE 2024-05-28T15:51:51 sounds good 2024-05-28T15:52:14 looks good, fork is gone, https://code.opensuse.org/kernel/kernel-source now allows me to fork it again 2024-05-28T15:53:29 kewl 2024-05-28T15:55:28 did the bindmount work? Otherwise it looks like setting it as env var would then picked up by python without adjusting the function based on the comment from pingou https://pagure.io/pagure/issue/5488#comment-912792 2024-05-28T15:55:59 I think it did, I see some pymp-xxxx thing in there 2024-05-28T15:56:11 but cool that sounds like a better solution 2024-05-28T15:56:15 let me change it 2024-05-28T15:57:33 set Environment=TMPDIR=/data/pagure/tmp now 2024-05-28T15:57:52 sounds reasonable, do you want me to fork the kernel source again? 2024-05-28T15:58:07 yes please, if you aren't bored of it yet 2024-05-28T15:58:37 no that's cool, it just looks like we still missed something somewhere: Your task failed: The requests repo forks/wombelix/kernel/kernel-source.git already exists 2024-05-28T15:58:55 sounds like an empty git repo was created on disk 2024-05-28T15:59:19 there is a new empty /srv/gitolite/repositories/forks/wombelix/kernel/ 2024-05-28T15:59:27 earlier I deleted the whole rm -r /srv/gitolite/repositories/forks/wombelix 2024-05-28T15:59:59 yeah maybe that cleanup was a bit too much :D in case I had any other forks there are now gone too ;P 2024-05-28T16:00:17 well I did check that there wasn't anything else in there 2024-05-28T16:00:30 is there a kernel-source.git directory somewhere? 2024-05-28T16:01:01 there is /srv/gitolite/repositories/kernel/kernel-source.git 2024-05-28T16:01:26 ha funny, I guess the first step when forking something is that a database entry is created and then the rest is done. because I seem to have another ghost fork again 2024-05-28T16:01:43 ah now /srv/gitolite/repositories/docs/forks/wombelix/kernel/kernel-source.git appeared too 2024-05-28T16:01:44 that's the original one, that's fine. I guess it's again a database entry, weird 2024-05-28T16:02:19 rm + 2024-05-28T16:02:21 DELETE again ? 2024-05-28T16:02:49 yeah try to delete "/srv/gitolite/repositories/docs/forks/wombelix/kernel/kernel-source.git" and then also my database entry for the fork 2024-05-28T16:03:23 it seems there are more .git directories per fork: https://paste.opensuse.org/pastes/1a61e4a343ea 2024-05-28T16:04:20 I guess I should delete */*/wombelix/kernel/kernel-source.git 2024-05-28T16:04:26 actually not, pagure saves tickets and stuff in git repos, only the one with "forks/" in the path are forks, the rest are other types 2024-05-28T16:04:37 ah okay 2024-05-28T16:04:44 ah forget it, you are right, now I understand what you say 2024-05-28T16:04:56 okay :D 2024-05-28T16:05:19 yes things like "/srv/gitolite/repositories/tickets/forks/wombelix/kernel/kernel-source.git" also need to be removed, you are right. so something lig */forks/wombelix/kernel/kernel-source.git 2024-05-28T16:05:27 * like 2024-05-28T16:06:04 ok, done 2024-05-28T16:07:49 let me try again to fork it 2024-05-28T16:07:53 https://paste.opensuse.org/pastes/d5430edeba51 for next time :P 2024-05-28T16:08:12 ok looks good, it's running, you should see a temp folder for it coming up 2024-05-28T16:08:25 yep it's there 2024-05-28T16:08:40 and growing 2024-05-28T16:08:40 it will further grow :D 2024-05-28T16:09:12 well we didn't fixed any performance issues yet but found a bug and some config to improve, better than nothing so far ;) 2024-05-28T16:09:16 it might actually help with speed too because the /srv/gitolite is on the same file system as the new temp dir 2024-05-28T16:09:41 hehe yep 2024-05-28T16:10:46 btw this was a git clone via ssh test on a fork of a large repo i did on pagure.io, worked fine: https://paste.opensuse.org/pastes/1e205bb1db2e 2024-05-28T16:11:09 I wanted to try to the same on code.o.o, that's why I tried to fork the kernel-source repo in the first place 2024-05-28T16:11:41 ah cool let's see how that goes then 2024-05-28T16:12:06 you will probably get lots of DEBUG output when doing ssh operations against code.o.o, we don't know how to disable it 2024-05-28T16:12:44 another thing to improve then :) 2024-05-28T16:13:54 think only for pushing though when it is doing the hooks, not for cloning 2024-05-28T16:13:56 y 2024-05-28T16:14:26 ok I can try both, still forking 2024-05-28T16:14:59 it's growing quite substantially, I think I should increase the disk size, if multiple people were to create such big forks at the same time it would run out 2024-05-28T16:15:48 well, yeah, I have every intention of increasing my usage of code.o.o for the stuff I'm working on. 2024-05-28T16:16:21 to be fair, the average user (hopefully?!) doesn't fork kernel-source all the time? :D 2024-05-28T16:16:40 but yeah, I mean, it can happen that it's done, a bit of disk space in reserve can't hurt 2024-05-28T16:17:04 :D 2024-05-28T16:17:26 all to make SFaulken happy 2024-05-28T16:17:39 I like it when people make me happy =P 2024-05-28T16:17:52 who doesn't 2024-05-28T16:17:54 I see you adjusted well to your board role :P 2024-05-28T16:18:06 I'm trying to. 2024-05-28T16:21:36 bmwiedemann1: https://gitlab.infra.opensuse.org/infra/salt/-/merge_requests/1892 2024-05-28T16:21:57 and it's still forking forking forking ... 2024-05-28T16:23:00 iotop says it's doing between 10-20 MB/s 2024-05-28T16:23:10 relatively consistent between read and write 2024-05-28T16:24:15 I'm sure it had a reason why it's implemented like that, with doing a TemporaryClone which then gets moved to the final location. But right now I don't understand why :D 2024-05-28T16:24:45 at least the final move should be relatively fast as it doesn't need to move between two disks 2024-05-28T16:24:59 but y would be curious too 2024-05-28T16:25:17 true, from a code perspective we really seem to do a git push --mirror into the fork repo 2024-05-28T16:25:23 interesting approach 2024-05-28T16:25:36 feels like an implementation with a lot of history :D 2024-05-28T16:26:29 heh 2024-05-28T16:33:44 it's still forking O_o wow... next time a 1GB test repo or something should be enough :D 2024-05-28T16:39:10 meanwhile, I increased the disk size from 100 too 300G 2024-05-28T16:40:17 but the IO heavy fork job earlier seems done already, it is sitting at 33G since a few minutes already 2024-05-28T16:41:43 ok 33GB is unexpected large, damn :O and, hmm, can you check if the repo in /srv/gitolite ... is growing? I guess it's now pushing 2024-05-28T16:42:31 this should be the path: /srv/gitolite/repositories/forks/wombelix/kernel/kernel-source.git 2024-05-28T16:45:38 that is sitting at 20K .. lsof says there is git-receive-pak running on it 2024-05-28T16:46:12 I need to head out now, will be back in ~1.5h 2024-05-28T16:46:26 thanks for the help so far! 2024-05-28T16:46:36 thanks for your help :) later 2024-05-28T16:47:06 I have the feeling we should talk about that I join the heroes, I feel bad that I block all your time to do some troubleshooting 2024-05-28T17:28:15 Dominik Wombacher (wombelix): heroes have upcoming meeting on 6th https://calendar.opensuse.org/teams/heroes/events/meeting so make sure to pop up there 2024-05-28T18:35:45 back 2024-05-28T18:35:52 DominikWombacher: would love to have you on board ;) 2024-05-28T18:36:37 git-receive-pack seems to have finished, the kernel-source.git fork is now at 2.1G and the temporary directory is gone 2024-05-28T19:06:06 I try to join the heroes meeting next week so we can get started :) 2024-05-28T19:06:13 perfect, then let's clone that beast 2024-05-28T19:07:22 I will be busy once again that Thursday :-( maybe we can vote on another weekday? 2024-05-28T19:09:28 vote on what? 2024-05-28T19:22:43 *** teepee_ is now known as teepee 2024-05-28T19:22:57 vote on which day of the week to have the heroes meeting 2024-05-28T19:24:34 ah, sure 2024-05-28T19:25:59 maybe make a survey.o.o + send to heroes@ ? maybe there's others who would like another day but haven't spoken up 2024-05-28T19:26:44 for me most of my evening blockers can be found on calendar.o.o ^^ 2024-05-28T19:32:24 no right to vote but I have to admit, 08:00 PM CEST made it always challenging for me, I had it on my to do list for months to join, but family responsibilities overlap pretty often... 2024-05-28T19:32:57 So, as expected, git clone via ssh took a while but no issues, I also understand the debug on push comment now, let me check how to disable that, it's really a bit annoying :O https://paste.opensuse.org/pastes/ea7c4dc4546f 2024-05-28T19:33:59 When we take a look how pagure.io is configured, there is no haproxy in front and they use apache. but with much higher timeouts https://pagure.io/fedora-infra/ansible/blob/main/f/roles/pagure/templates/0_pagure.conf wsgi 300, overall 600. 2024-05-28T19:34:36 interesting so you use mod_wsgi instead of gunicorn? 2024-05-28T19:34:37 And what I've learned today, most issues seem to come from the current 30sec timeout setting in nginx. So I recommend to increase them and then we go from there and see what other issues need to be fixed. 2024-05-28T19:35:32 The Fedora infra team does, I'm just the guy that takes care about pagure a bit. 2024-05-28T19:35:58 So not sure if the apache and wsgi setup is better or worse, but at least the timeout configuration is pretty different 2024-05-28T19:40:27 I see, I don't have experience with mod_wsgi, but I would be quite interested in trying it, the gunicorn workers bite me in multiple apps 2024-05-28T19:40:42 as for the timeouts, we can increase it, sure, I'm just not sure someone wants to wait so long for a website 2024-05-28T19:40:47 I do get it with https clone though 2024-05-28T19:41:32 It's interesting that Neal talks in the fedora infra channel about to prefer nginx and gunicorn over apache and wsgi in future for pagure :D 2024-05-28T19:41:50