2025-10-19T04:42:33  *** teepee_ is now known as teepee
2025-10-19T08:25:26  <matrix-o-o> <egotthold> acidsys: Could you kindly add me to the OBS project for fireactions? You forgot to put nameservers in and now DNS doesn't work. :D
2025-10-19T08:25:28  <matrix-o-o> <egotthold> localhost:~ # ping github.com
2025-10-19T08:25:55  <acidsys> egotthold: I will, but nameserver is set through ifcfg parameter, not hardcoded in image
2025-10-19T08:26:37  <matrix-o-o> <egotthold> Ah okay then I will need to adjust just that. I am currently starting to write everything up in the agreed upon Wiki page.
2025-10-19T08:27:31  <acidsys> interesting, in fireactions-runner-00c5e9a17805399534eb5ba4 it is missing
2025-10-19T08:30:41  <matrix-o-o> <egotthold> Yes that was how I noticed it.
2025-10-19T08:31:28  <matrix-o-o> <egotthold> Also I noticed the subcommand "fireactions microvm login" doesn't work. I will investigate that as well, as it is very useful (even more so with IPv6).
2025-10-19T08:37:00  <acidsys> it's supposed to pick them up from CNI and "ifcfg=eth0=2a07:de40:b27e:4003::129/64,2a07:de40:b27e:4003::1," place them after the last comma here
2025-10-19T08:41:36  <acidsys> microvm login suggests it looks for a pool "fireactions-runner" but we only have a pool "fireactions-performance" (not that I understand why it would be pool specific)
2025-10-19T08:45:52  <acidsys> ok, I solved nameservers
2025-10-19T08:47:28  <acidsys> I added it to !2623 which was still unreviewed anyways
2025-10-19T08:51:34  <matrix-o-o> <egotthold> I think the MR is valid. I can't say what the change in the GPG does content wise but I trust you that it is correct. :)
2025-10-19T08:51:57  <acidsys> unrelated to it, I noticed docker inside the VM to still suffer from https://bugzilla.opensuse.org/show_bug.cgi?id=1214443 even though the comment claims it should be resolved
2025-10-19T08:52:36  <acidsys> I prefer podman anyways :P
2025-10-19T08:52:52  <matrix-o-o> <egotthold> I wouldn't see this as suffering. We don't actually need any specific DNS, the Google ones are just fine for ours.
2025-10-19T08:53:40  <acidsys> you cannot reach Google nameservers, first because they are written with IPv4 addresses, second because it's not allowed outbound
2025-10-19T08:53:47  <matrix-o-o> <egotthold> Well podman is nice if you have a single container but Docker Compose is just too darn good. Quadlets are good as well but I prefer Docker Compose since everything is in a single file.
2025-10-19T08:53:56  <matrix-o-o> <egotthold> Ah okay well then we need the ones from the host...
2025-10-19T08:54:47  <acidsys> it's supposed to write the host nameservers into the container resolv.conf but isn't doing it
2025-10-19T08:55:18  <matrix-o-o> <egotthold> Well then I guess let's reopen the bsc and hand it back to our maintainer.
2025-10-19T08:57:03  <acidsys> need to investigate some more
2025-10-19T08:59:36  <acidsys> it seems my kernel is lacking something for legacy ip6tables
2025-10-19T09:04:13  <matrix-o-o> <egotthold> If you are done with that, I would love if you would be so kind and check https://progress.opensuse.org/projects/opensuse-admin-wiki/wiki/Apolloinfraopensuseorg for completness. I think I documented everything but I am unsure if I forgot something.
2025-10-19T09:16:32  <acidsys> ok docker in vm is working with network now after https://github.com/tacerus/opensuse-fireactions-kernel/commit/6818941e720194cd33eaceb7e2b90e6836bddb6e and https://build.opensuse.org/package/rdiff/openSUSE:infrastructure:fireactions/fireactions-runner-container?linkrev=base&rev=5. your article, the problem statement, for me at least, is not exclusively that we are using IPv6, but also that
2025-10-19T09:16:33  <acidsys> we would like to use trusted software instead of black magic off the internet :) besides that I will add some minor changes to the steps
2025-10-19T09:17:53  <acidsys> also SR workflow would probably be wise in the project for production
2025-10-19T09:18:19  <acidsys> oh you already wrote that, cool
2025-10-19T09:36:09  <matrix-o-o> <egotthold> Feel free do just edit the page how you see fit. :)
2025-10-19T09:39:12  <matrix-o-o> <egotthold> I'll be testing the setup now with a CI run. :)
2025-10-19T09:43:31  <acidsys> oki
2025-10-19T09:44:56  <acidsys> btw have you found a way to list all currently running VMs, `containerd-ctr -n fireactions-performance container ls` is always empty, and the log output will be a bit inconvenient to parse once it gets more busy :)
2025-10-19T09:45:18  <acidsys> the process listing does not tell much either
2025-10-19T09:45:24  <matrix-o-o> <egotthold> Well this is what "fireactions microvm list" is for.
2025-10-19T09:46:02  <acidsys> nice, didn't find that. thanks
2025-10-19T09:46:08  <matrix-o-o> <egotthold> Okay so something is still not working. :/
2025-10-19T09:46:12  <matrix-o-o> <egotthold> The runner isn't registering.
2025-10-19T09:46:53  <acidsys> Oct 19 09:33:16 localhost fireactions[892]: 2025-10-19 09:33:16Z: Runner connect error: The HTTP request timed out after 00:01:40.. Retrying until reconnected. this related?
2025-10-19T09:47:02  <acidsys> wish it would tell me http request to where
2025-10-19T09:48:08  <matrix-o-o> <egotthold> I'll try a new VM.
2025-10-19T09:48:16  <acidsys> ok
2025-10-19T09:49:19  <matrix-o-o> <egotthold> Okay new runner registered in GitHub but marked as offline.
2025-10-19T09:49:36  <matrix-o-o> <egotthold> Meaning Runner <--> GitHub connection is most likely not working.
2025-10-19T09:51:27  <matrix-o-o> <egotthold> Well I guess it is time for a pcap. :D
2025-10-19T09:52:28  <acidsys> https://github.blog/changelog/2023-08-29-github-actions-review-network-access-settings-for-the-self-hosted-runners/ is rather vague
2025-10-19T09:57:36  <matrix-o-o> <egotthold> Well that is all we will get.
2025-10-19T09:58:27  <matrix-o-o> <egotthold> There are not a lot of networks imposing the restrictions that we impose on our internal openSUSE infra networks.
2025-10-19T09:59:30  <matrix-o-o> <egotthold> This is even more complicated as their infra doesn't run in a dedicated IP-range but is (as you know) switching ranges which is only announced via their GitHub API.
2025-10-19T10:05:54  <matrix-o-o> <egotthold> Also I am confused why there are no repositories inside the MicroVM? In a Standard Leap Container there are repos... I see no steps in the Dockerfile that remove them?
2025-10-19T10:06:50  <matrix-o-o> <egotthold> Ah "zypper rs" was unkown to me.
2025-10-19T10:08:13  <acidsys> yeah that rs is because of https://github.com/openSUSE/obs-build/issues/1107 (one of the countless of leap 16 problems), if you want to preserve repos instead of the rs call, move the /etc/zypp/services.d/openSUSE.service (or whatever it's called) away, and move it back after all zypper operations
2025-10-19T10:10:26  <acidsys> or as an alternative rm and reinstall openSUSE-repos-Leap
2025-10-19T10:11:01  <matrix-o-o> <egotthold> Okay. I'll try and see what the packet capture says once I have tcpdump installed.
2025-10-19T10:11:51  <acidsys> regarding network, it would be acceptable for me to allow the runner VM subnet outbound https access to all internet destinations as long as I combine it with some traffic shaping, I will look into it
2025-10-19T10:12:55  <matrix-o-o> <egotthold> Thanks. If I can help, feel free to ask.
2025-10-19T10:13:42  <acidsys> sure, if you could verify that it's only 443/https it is trying to send that would be helpful ;)
2025-10-19T10:14:51  <matrix-o-o> <egotthold> Okay so I can't even use the Leap Repos...
2025-10-19T10:15:01  <matrix-o-o> <egotthold> localhost:~ # zypper ref
2025-10-19T10:15:13  <acidsys> you want to use http://download-prg.infra.opensuse.org/ as mirror (same as on the host)
2025-10-19T10:19:51  <matrix-o-o> <egotthold> It appears to go to do the following call:
2025-10-19T10:19:51  <matrix-o-o> 10:19:09.905851 IP6 2a07:de40:b27e:4003::12d.60660 > 2a07:de40:b27e:64::140a:e236.https: Flags [S], seq 707332222, win 64800, options [mss 1440,sackOK,TS val 2982164015 ecr 0,nop,wscale 7], length 0
2025-10-19T10:20:24  <matrix-o-o> <egotthold> How can I convert that to the IPv4 equivalent?
2025-10-19T10:22:40  <matrix-o-o> <egotthold> But I think it is indeed what you asked for atm: Only HTTPS traffic so far.
2025-10-19T10:22:58  <matrix-o-o> <egotthold> But let's see what we unlock once we have that opened up.
2025-10-19T10:36:25  <acidsys> cool, v4 to v6 is easy, to convert it back I'm honestly not sure, I do `dig -x 2a07:de40:b27e:64::140a:e236` and it will give you a CNAME to the IPv4 equivalent (whether there actually is a PTR record for the IPv4 address is not relevant for the purpose of finding the IPv4 address)
2025-10-19T10:36:44  <acidsys> (dig against one of our infra DNS servers, of course)
2025-10-19T10:45:01  <matrix-o-o> <egotthold> The result is not pleasing:
2025-10-19T10:45:25  <matrix-o-o> <egotthold> It means essentially that we need to allow Azure as a whole.
2025-10-19T10:45:52  <matrix-o-o> <egotthold> Or do I understand that result incorrectly?
2025-10-19T10:46:27  <matrix-o-o> <egotthold> Wait the CNAME points to AWS.
2025-10-19T10:46:51  <matrix-o-o> <egotthold> This is funny. An Azure Loadbalancer pointing to a AWS IP address. :D
2025-10-19T10:47:08  <matrix-o-o> <egotthold> That is the IP of a random EC2 instance in AWS: ec2-54-226-10-20.compute-1.amazonaws.com.
2025-10-19T11:04:38  <matrix-o-o> <egotthold> The tcpdump reveals that there are requests going to help asking for DNS records for pipelinesghubeus22.actions.githubusercontent.com. So I really do believe the only option is to allow all traffic towards actions.githubusercontent.com as those instances will be dynamically spawned and destroyed in one of the major clouds.
2025-10-19T13:39:54  <matrix-o-o> <egotthold> It appears that there is no way for nftables to filter by domain name (which I understand) but then nftables cannot reliably do the job we need it to do. I mean we can combine of course a script with a dedicated nftables set and then just list all IPs but that will not work in all cases I guess.
2025-10-19T13:40:44  <matrix-o-o> <egotthold> One more thing that complicates things is that not all DNS servers will all wildcard queries but I guess we can do some magic to fix that as well inside the script.
2025-10-19T13:48:04  <acidsys> I know, we have this tracked as https://progress.opensuse.org/issues/183734. as already said I am fine with allowing all outbound https for the vm network
2025-10-19T13:53:31  <matrix-o-o> <egotthold> Thanks. Is this already WIP?
2025-10-19T13:54:56  <acidsys> I was working on traffic control, my dependency
2025-10-19T13:58:51  <matrix-o-o> <egotthold> Okay so yes. Thank you.
2025-10-19T15:08:17  *** teepee_ is now known as teepee
2025-10-19T16:56:55  <matrix-o-o> <egotthold> acidsys: I see you are on apollo. I have to stop and nuke the thinpool. 1659 snapshots slows the host down considerbly.
2025-10-19T16:58:25  <acidsys> go ahead
2025-10-19T17:22:45  <matrix-o-o> <egotthold> I reboot the node. Something hangs in the config. It tries to find a deleted snapshot even though I nuked all folders.
2025-10-19T17:26:37  <matrix-o-o> <egotthold> Nooo it still wants a snapshot that isn't found. I don't get it....
2025-10-19T17:28:48  <acidsys> if it helps, I spent 2h trying to figure out why my test rules did not work when I realize I am not coming through the internet but through the vpn tunnel and watched the wrong interface the whole time
2025-10-19T17:32:39  <matrix-o-o> <egotthold> Ah I found it. Missed to delete a directory.
2025-10-19T17:32:41  <matrix-o-o> <egotthold> Now it works again.
2025-10-19T17:32:59  <matrix-o-o> <egotthold> System is yours again.
2025-10-19T17:33:27  <matrix-o-o> <egotthold> We need to somehow have monitoring for this. We can script setting it up again easily.
2025-10-19T17:33:40  <matrix-o-o> <egotthold> But not today. First we fix the network situation.
2025-10-19T17:35:33  <acidsys> anything prometheus-y works. native metrics/exporter preferred but script which generates textfile is also doable (find examples in salt/profile/monitoring/prometheus/files/textfile/{scripts,systemd}/)
2025-10-19T17:43:11  <matrix-o-o> <egotthold> No the metrics will be native. But the fixing will need to be a script.
2025-10-19T17:45:58  <matrix-o-o> <egotthold> Btw I see that you have a lot of MRs open in the openSUSE Salt. Are there any MRs that I can help you with? I would love to give a little back for all the time you spent this weekend for the issue with network connectivity and the fireactions packaging.
2025-10-19T17:53:21  <acidsys> my draft MRs have usually some piece missing or await testing, I'm not sure any of them are "easy" to complete (though if there are some which sound interesting to you I'm of course happy to provide context). more generally our ticket tracker has lots of free work ;)
2025-10-19T18:01:51  <matrix-o-o> <egotthold> Wonderful. I will see if anything catches my eye. :)
2025-10-19T19:21:14  <matrix-o-o> <egotthold> acidsys: approved
2025-10-19T20:03:19  <acidsys> apollo matches highstate now
2025-10-19T20:51:10  <cboltz_> "nice" - now the CI sync job fails for volva1 and volva2
2025-10-19T20:51:30  <cboltz_> last successful CI job was after merging the shaping branch
2025-10-19T20:52:37  <acidsys> I applied earlier change before, my change in f0974c3b316c64c58ba9497382d12c1331b07060 seems to be similarly senseless than the change it was supposed to fix :P
2025-10-19T20:53:37  <acidsys> on a second look, it should be fine because I did not touch the volva line below
2025-10-19T20:54:01  <cboltz_> right, that commit looks unrelated
2025-10-19T20:54:57  <cboltz_> As a side note: having/using a set containing all salt masters might make sense
2025-10-19T20:55:09  <acidsys> yep was thinking the same
2025-10-19T20:55:33  <acidsys> next templating idea: make sets based on hosts with roles ;-)
2025-10-19T20:55:58  <acidsys> https://paste.opensuse.org/pastes/349b9bc1d6bf
2025-10-19T20:56:26  <acidsys> it tried to route it through the wrong interface
2025-10-19T21:07:49  <acidsys> I found the issue, interesting we did not have it before, but solved for now
2025-10-19T21:08:36  <acidsys> egotthold: your requests are fulfilled now as well, please try the thingies again
2025-10-19T21:30:02  <cboltz_> what was wrong?
2025-10-19T21:30:34  <acidsys> explanation in latest commit message
2025-10-19T21:32:43  <cboltz_> you answer questions on IRC via MRs in gitlab? nice :-)
2025-10-19T21:33:23  <acidsys> yes and that I answer it before you even send the question on IRC is a bonus feature
2025-10-19T21:33:59  <cboltz_> indeed, IRC latency seems to be terrible ;-)
2025-10-19T21:38:24  <cboltz_> completely different topic - I don't understand french, but nevertheless, https://lists.opensuse.org/archives/list/users-fr@lists.opensuse.org/ looks *very* spammy :-/
2025-10-19T21:38:41  <acidsys> oui
2025-10-19T21:39:25  <acidsys> if the spammers were smart, they would use french text so it looks all normal to our moderators
2025-10-19T21:40:01  <cboltz_> I just configured moderation for mails sent via hyperkitty, but the archive clearly needs a cleanup
2025-10-19T21:40:59  <acidsys> not sure what's easiest, gui does not have option to select multiple emails and cli you need to filter for something
2025-10-19T21:46:10  <cboltz_> that, and I didn't find an obvious way for "show all mails sent by this user" which would be useful when looking for spam and spammers
2025-10-19T21:49:27  <acidsys> you can query for them and get the ids or similar
2025-10-19T22:31:22  <cboltz_> I clicked a bit through the users-fr archive, and needed to go to page 16 to find some non-spam mails (from Aug 2023) - since then, (nearly?) all mails were spam
2025-10-19T22:31:50  <cboltz_> I wonder why nobody reached out to report the spam until yesterday...