2025-10-24T00:23:15  *** teepee_ is now known as teepee
2025-10-24T07:16:53  <matrix-o-o> <egotthold> acidsys: It seems I cannot connect to apollo anymore. How can I access the BMC? I didn't find its address in Salt.
2025-10-24T08:19:23  <acidsys> egotthold: fd4b:5292:d67e:1001::e1 from devcon.i.o.o, yes monitor complains about it being down too
2025-10-24T08:21:59  <matrix-o-o> <egotthold> I'll check it out in a few minutes.
2025-10-24T08:50:35  <matrix-o-o> <egotthold> acidsys: Are you sure? I just get an empty Nginx dummy site on port 80 & port 443 returns nothing.
2025-10-24T08:50:44  <matrix-o-o> <egotthold> Command used "ssh -i ~/.ssh/id_ed25519 -L 8080:[fd4b:5292:d67e:1001::e1]:443 schoolguy@devcon.infra.opensuse.org"
2025-10-24T09:01:11  <acidsys> WORKSFORME https://paste.opensuse.org/pastes/8f6482719540 and ssh gets me a looping BIOS/GRUB
2025-10-24T09:02:29  <matrix-o-o> <egotthold> paste.o.o loads for ages...
2025-10-24T09:03:06  <acidsys> maybe you have more problems with your ghost nginx server
2025-10-24T09:07:57  <matrix-o-o> <egotthold> Now it works. I have no clue what the issue was. I saw you used the "-N" flag in SSH but well it works. So I am happy.
2025-10-24T09:08:55  <acidsys> N is just to avoid spawning a shell if I only want port forwarding, should not make a difference for that
2025-10-24T09:09:01  <acidsys> glad it works
2025-10-24T09:10:06  <matrix-o-o> <egotthold> Ahhh the issue may be Chrome. I can't open openSUSE Infra GitLab but in Firefox it works.
2025-10-24T09:37:25  *** teepee_ is now known as teepee
2025-10-24T10:42:26  <acidsys> monitoring complains about memory consumption on matrix.i.o.o, it seems it increased quite a bit over the last days https://monitor.opensuse.org/grafana/goto/S85MLpgDR?orgId=1 and https://monitor.opensuse.org/grafana/goto/q7i7LtgDR?orgId=1, could someone check? hellcp or emma maybe?
2025-10-24T10:42:44  <matrix-o-o> <Emma [it/its]> checking
2025-10-24T10:42:49  <acidsys> thanks!
2025-10-24T10:43:14  <matrix-o-o> <Emma [it/its]> oh woah that sure is a large memory
2025-10-24T10:44:30  <matrix-o-o> <Emma [it/its]> hm
2025-10-24T10:44:58  <matrix-o-o> <Emma [it/its]> initial observations say there's a whole 2 processes sitting at >1GB of RAM, and it's 2 synapse workers sitting at ~4.65G each
2025-10-24T10:46:20  <matrix-o-o> <Emma [it/its]> they are "federation_requests2" and "federation_requests1" respectively
2025-10-24T10:46:38  <acidsys> that seems to align with the two in the graph
2025-10-24T10:47:29  <matrix-o-o> <Emma [it/its]> "ls: cannot open directory '/var/log/matrix-synapse/': Permission denied"
2025-10-24T10:47:29  <matrix-o-o> blegh
2025-10-24T10:47:34  <acidsys> moment
2025-10-24T10:48:10  <matrix-o-o> <Emma [it/its]> though iirc the log format is messed up so im not sure how useful thatll be
2025-10-24T10:49:13  <acidsys> added you to matrix-admins, if you log out and back in you should be able to use sudo to root
2025-10-24T10:49:51  <matrix-o-o> <Emma [it/its]> neato
2025-10-24T10:50:10  <matrix-o-o> <Emma [it/its]> ah, we dont have lnav installed, alright
2025-10-24T10:50:54  <acidsys> right we have it only on the syslog server (which so far does not collect homeserver.log), but feel free to zypper install it if it helps you
2025-10-24T10:51:50  <matrix-o-o> <Emma [it/its]> yep, logs are still messy, only seems to be be logging GC calls and cache drops
2025-10-24T10:52:41  <matrix-o-o> <Emma [it/its]> interestingly, found a wave of larger cache drops followed by a bunch of pg connection errors, though unsure if related
2025-10-24T10:53:30  <acidsys> last maintenance on postgresql server was 12 days ago
2025-10-24T10:54:06  <matrix-o-o> <Emma [it/its]> "psycopg2.InterfaceError: connection already closed", "twisted.enterprise.adbapi.ConnectionLost", "psycopg2.OperationalError: SSL SYSCALL error: EOF detected" are the main ones
2025-10-24T10:54:15  <acidsys> I find some things like "2025-10-24 10:52:41.499 UTC [6584]: [38-1] db=matrix,user=matrix ERROR:  could not serialize access due to concurrent update"
2025-10-24T10:54:26  <acidsys> and "2025-10-24 10:53:14.628 UTC [26120]: [189-1] db=matrix,user=matrix ERROR:  duplicate key value violates unique constraint "users_name_key""
2025-10-24T10:54:29  <acidsys> not sure if that is related
2025-10-24T10:55:04  <matrix-o-o> <Emma [it/its]> sounds like migrations failing at startup due to parallel worker startup or something
2025-10-24T10:55:47  <matrix-o-o> <Emma [it/its]> though all of the psycopg2 errors are in the middle of runtime, not startup (ie. seeing references to "get_rooms_for_user" etc)
2025-10-24T10:56:15  <matrix-o-o> <Emma [it/its]> oh seems to have been relatively close to last restart
2025-10-24T10:56:38  <matrix-o-o> <Emma [it/its]> annoyingly, the logs dont have timestamps at all
2025-10-24T10:57:45  <acidsys> that's indeed quite annoying
2025-10-24T10:58:13  <matrix-o-o> <Emma [it/its]> do you think i should try just restarting the workers in question? i did notice that there's a quite noticeable spike in received events and soft-failed events around the time of the memory increase
2025-10-24T10:58:41  <acidsys> sounds like the windows-approach but if it helps, sure ;)
2025-10-24T10:58:57  <matrix-o-o> <Emma [it/its]> well to me it feels like a cache grew too large :P
2025-10-24T10:59:21  <acidsys> should it not clean itself up
2025-10-24T10:59:42  <matrix-o-o> <Emma [it/its]> it should, but it generally doesnt do a great job at it if it isnt told to by the config
2025-10-24T11:00:27  <acidsys> ah okey
2025-10-24T11:00:43  <matrix-o-o> <Emma [it/its]> checking config rq
2025-10-24T11:01:03  <matrix-o-o> <Emma [it/its]> hm, salt being unhelpful, makes sense lol
2025-10-24T11:01:14  <acidsys> we have it managed via salt/profile/matrix/files/homeserver.yaml etc
2025-10-24T11:01:33  <matrix-o-o> <Emma [it/its]> ah, i was looking at salt/pillar/role/matrix.sls
2025-10-24T11:01:55  <acidsys> ah yeah the homeserver configuration is not abstracted to pillar
2025-10-24T11:02:59  <matrix-o-o> <Emma [it/its]> ah right, we dont have kitty-terminfo on the server
2025-10-24T11:03:02  <matrix-o-o> <Emma [it/its]> vim just gave me a warning lol
2025-10-24T11:04:01  <acidsys> I prefer to use a terminal using standard terminfo or tmux for working on servers
2025-10-24T11:04:18  <matrix-o-o> <Emma [it/its]> hm, i wonder if home dirs are persisted
2025-10-24T11:04:43  <matrix-o-o> <Emma [it/its]> i might add a .bashrc to set TERM=xterm-256color
2025-10-24T11:05:00  <acidsys> yes /home/ are permanent (local to each machine though)
2025-10-24T11:07:11  <matrix-o-o> <Emma [it/its]> alias nvim='vim'
2025-10-24T11:09:37  <acidsys> alias vim=ex
2025-10-24T11:10:33  <matrix-o-o> <Emma [it/its]> lol
2025-10-24T11:11:09  <matrix-o-o> <Emma [it/its]> hm, we dont explicitly configure cache targets or garbage collection
2025-10-24T11:12:35  <matrix-o-o> <Emma [it/its]> (nor do we enable auto tuning)
2025-10-24T11:12:58  <matrix-o-o> <Emma [it/its]> my best guess is that it's seeing the hardware availability and going "oh i've got plenty of memory, lets do large caches"
2025-10-24T11:13:28  <matrix-o-o> <LCP> keep in mind that config is several years old and rarely updated
2025-10-24T11:13:37  <matrix-o-o> <LCP> so I'm not surprised it's not configured 😄
2025-10-24T11:18:17  <matrix-o-o> <Emma [it/its]> i gotta go do food, but i think ill prod it with a service restart when i get back
2025-10-24T11:19:19  <acidsys> enjoy food, you know where to find salt now
2025-10-24T11:24:47  <matrix-o-o> <LCP> there's not that much food in the pillar though, only some paste, lime and something dehydrated
2025-10-24T11:26:16  <acidsys> plenty cans of spam, but nobody likes those
2025-10-24T11:31:00  <cboltz> well, there's also some great ham on paste.o.o: https://paste.opensuse.org/pastes/a6c8c932d3ab
2025-10-24T11:38:33  <matrix-o-o> <Emma [it/its]> lol
2025-10-24T12:02:55  <acidsys> "friendly llama" fits well as a random for something that could be coming from an AI
2025-10-24T12:05:14  <matrix-o-o> <knurpht - Gertjan> An *AI* is a toothless mammal
2025-10-24T12:08:14  <matrix-o-o> <knurpht - Gertjan> https://de.wikipedia.org/wiki/Wei%C3%9Fkehl-Faultier
2025-10-24T12:44:04  <matrix-o-o> <Emma [it/its]> gave the federation requests workers a restart
2025-10-24T12:47:50  <matrix-o-o> <knurpht - Gertjan> Emma [it/its]: Photos ............ well why not real output?
2025-10-24T12:47:55  <matrix-o-o> <Emma [it/its]> cool, graphs are looking good
2025-10-24T12:48:42  <matrix-o-o> <Emma [it/its]> knurpht - Gertjan: ╭─┐²mem┌──────────────────────────────────┬
2025-10-24T12:50:00  <matrix-o-o> <knurpht - Gertjan> See, there we go
2025-10-24T12:50:03  <matrix-o-o> <Emma [it/its]> be careful what you wish for :P
2025-10-24T12:53:36  <matrix-o-o> <Emma [it/its]> hmmm learned a new terminal keyboard shortcut lol
2025-10-24T12:54:02  <matrix-o-o> <Emma [it/its]> ctrl+shift+h copies the entire screen buffer into a pager, allowing for easy copying
2025-10-24T15:51:43  *** teepee_ is now known as teepee
2025-10-24T16:43:09  <matrix-o-o> <egotthold> acidsys: The fireactions-runner package was installed by an OS update (that I was not aware of). As such the bootloop. Masked the service via systemd and now it looks better.
2025-10-24T16:43:35  <matrix-o-o> <egotthold> Although the thinpool is making trouble again after the reboot...
2025-10-24T16:43:35  <matrix-o-o> <egotthold> Over 700 Snapshots...
2025-10-24T16:44:28  <acidsys> well I told you you need to adjust the installed packages on host and container
2025-10-24T16:44:34  <acidsys> masking the service is very poor solution
2025-10-24T16:45:09  <acidsys> as either runner nor server was explicitly installed the conditional requires just picked one as part of update
2025-10-24T16:45:51  <matrix-o-o> <egotthold> https://gitlab.infra.opensuse.org/infra/salt/-/merge_requests/2646
2025-10-24T16:46:09  <matrix-o-o> <egotthold> I know and I am on it. I just wasn't aware of any auto-updates.
2025-10-24T16:46:29  <matrix-o-o> <egotthold> The systemd masking was just to investigate. Don't worry. :)
2025-10-24T16:46:56  <acidsys> aah I get it now, because the OnSuccess=reboot in -runner
2025-10-24T16:47:06  <acidsys> or whatever it was called
2025-10-24T16:47:06  <matrix-o-o> <egotthold> Yes
2025-10-24T16:48:10  <matrix-o-o> <egotthold> Although the issue with the tinpool still remains but that is something I need to clarify with upstream. I will write myself a small script.
2025-10-24T16:48:59  <acidsys> right
2025-10-24T16:59:18  <matrix-o-o> <egotthold> Do you have an idea how to automate "/root/fix-thinpool.sh" on apollo? It requires an interactive input for one of the lvm commands. I would be tempted once we can execute the script unattended to add the start of the systemd services and the execute it once a day.
2025-10-24T17:00:11  <matrix-o-o> <egotthold> (I have a testrun running atm, so please don't execute it until https://github.com/cobbler/cobbler/pull/3912 is finished.)
2025-10-24T17:02:23  <acidsys> which of the lines is the one asking for input?
2025-10-24T17:03:18  <acidsys> I assume dmsetup
2025-10-24T17:03:41  <matrix-o-o> <egotthold> If I am not mistakten it should be pvcreate line.
2025-10-24T17:03:46  <acidsys> oh
2025-10-24T17:04:03  <matrix-o-o> <egotthold> Did you execute it?
2025-10-24T17:04:08  <acidsys> no
2025-10-24T17:04:10  <matrix-o-o> <egotthold> :D
2025-10-24T17:04:29  <matrix-o-o> <egotthold> Ah okay. What was the "oh" then referring to?
2025-10-24T17:04:38  <acidsys> that my assumption was not right
2025-10-24T17:04:42  <acidsys> anyways try pvcreate with -y / --yes
2025-10-24T17:05:53  <matrix-o-o> <egotthold> Okay I can try that tomorrow once my CI run it through. Thanks for the tip!
2025-10-24T17:06:57  <matrix-o-o> <egotthold> If that works I will create a small RPM and put it into /usr/bin with a better name. Once that works I will add a Suggests to the fireactions server RPM.
2025-10-24T17:07:29  <matrix-o-o> <Emma [it/its]> hey im looking for some advice and options: anyone happen to know a good authoritative dns server service?
2025-10-24T17:07:40  <acidsys> good idea, fireactions service can then require it or similar
2025-10-24T17:07:47  <acidsys> emma: for hosting yourself or as a service
2025-10-24T17:08:05  <matrix-o-o> <Emma [it/its]> as a service is probably preferrable
2025-10-24T17:08:11  <acidsys> as a service I guess https://desec.io/
2025-10-24T17:08:19  <matrix-o-o> <Emma [it/its]> and im avoiding cloudflare in particular
2025-10-24T17:08:33  <acidsys> good)
2025-10-24T17:08:59  <matrix-o-o> <Emma [it/its]> just looked at my stats and it looks like i'll be expecting a ~50 EUR bill for just dns queries this month
2025-10-24T17:09:09  <acidsys> huh you even pay for it
2025-10-24T17:09:13  <matrix-o-o> <Emma [it/its]> assuming im remembering bunny pricing correctly
2025-10-24T17:11:55  <matrix-o-o> <Emma [it/its]> acidsys: i do yeah
2025-10-24T17:12:06  <matrix-o-o> <Emma [it/its]> even though i pretty much dont use any of their services outside of plain DNS
2025-10-24T17:12:39  <acidsys> well then good to switch away
2025-10-24T17:12:48  <matrix-o-o> <egotthold> Well I am a Hetzner Fanboy. You can edit their raw zonefiles if you know what you are doing. :)
2025-10-24T17:13:14  <matrix-o-o> <egotthold> And traffic for DNS doesn't cost you a penny. :D
2025-10-24T17:13:14  <matrix-o-o> <Emma [it/its]> i thought i was going to use their DNS scripting thing or maybe the CDN, but no
2025-10-24T17:39:39  <matrix-o-o> <Richard Rahl> is it enough to add the ssh key to kanidm to get it propagated? still can't login after couple hours
2025-10-24T17:39:51  <matrix-o-o> <Richard Rahl> s/login/ssh in/
2025-10-24T17:40:12  <acidsys> RichardRahl: yes it should be relatively immediate
2025-10-24T17:40:35  <acidsys> where are you trying to connect to?
2025-10-24T17:41:50  <matrix-o-o> <Richard Rahl> code-dev. we finally got the upstream version which includes the pagure migration path. and wanted to see if that update fixes also the issue with what you pointed out on progress
2025-10-24T17:46:25  <acidsys> I find one attempt that actually went to code-dev some hours ago and besides that only attempts on the jump server. try ssh to thor1.infra.opensuse.org now (so I can tail the output)
2025-10-24T17:47:28  <matrix-o-o> <Richard Rahl> acidsys: just tried and got permission denied
2025-10-24T17:48:40  <acidsys> you are trying to authenticate with SHA256:xYOIeNik2/sj2D+5n20eNS1jDl2sURTxLKZ2Of2YsBA but in your account is SHA256:6dEmrujL+SH9iE8utWwr3htFpORMIH4VPSgEAEJECOo
2025-10-24T17:51:37  <matrix-o-o> <Richard Rahl> acidsys: oO that would be news to me, as I just went through my keys manually to see which one it is.
2025-10-24T17:51:41  <matrix-o-o> <Richard Rahl> let me login again to idm
2025-10-24T17:53:15  <matrix-o-o> <Richard Rahl> acidsys: huh? so I added a key to idm.i.o.o around 17:00. which is not there anymore. so I guess I will readd the new key and see in a bit
2025-10-24T17:53:55  <matrix-o-o> <Richard Rahl> ahh, I guess I didn't click "save changes"
2025-10-24T17:58:27  <acidsys> now there's two
2025-10-24T17:58:58  <matrix-o-o> <Richard Rahl> acidsys: yeah thanks. that was the issue (not clicking save) am on code-dev no
2025-10-24T17:58:59  <matrix-o-o> <Richard Rahl> w
2025-10-24T17:59:11  <acidsys> kewl
2025-10-24T18:20:10  <matrix-o-o> <Richard Rahl> acidsys: if I remember correctly the mysql port got changed to 3306?
2025-10-24T18:20:21  <acidsys> no
2025-10-24T18:20:44  <acidsys> I added another listener with 3306 to use for new services but I did not change any existing ones
2025-10-24T18:20:44  <matrix-o-o> <Richard Rahl> ahh k dang
2025-10-24T18:21:06  <acidsys> shall I flush-hosts ?
2025-10-24T18:21:22  <matrix-o-o> <Richard Rahl> it can't connect to it. so that's why I am asking
2025-10-24T18:21:54  <acidsys> to mysql? that's because code-dev.i.o.o is blocked
2025-10-24T18:22:28  <matrix-o-o> <Richard Rahl> ahh, that makes even more sense
2025-10-24T18:24:05  <acidsys> FYI you can find out like this if you did not know (mariadb/mysql cli does not print much useful on its own) https://paste.opensuse.org/pastes/156623ab9472
2025-10-24T18:24:29  <acidsys> on the server it looks like this https://paste.opensuse.org/pastes/da9ffea91bd2
2025-10-24T18:24:49  <acidsys> I did flush-hosts now, so it will work again (for now)
2025-10-24T18:24:50  <matrix-o-o> <Richard Rahl> thanks :)
2025-10-24T19:21:44  <acidsys> emma: https://monitor.opensuse.org/grafana/goto/qgW6xhgDR?orgId=1 did not last long :)
2025-10-24T19:37:29  <matrix-o-o> <Emma [it/its]> oh come on D:
2025-10-24T20:38:06  <cornfeedhobo> does anyone understand what this denial message means? https://build.opensuse.org/requests/1313615
2025-10-24T20:45:06  <acidsys> cornfeedhobo: best to ask the person who wrote it directly, but he's saying that if at all the package should be linkpac'd to server:mail. with SR you would need to update it manually every time there is an update in openSUSE:Factory
2025-10-24T20:46:27  <cornfeedhobo> yeah, definitely not ideal, but i don't know a better way given the dependencies not being in the main repo
2025-10-24T20:47:00  <cornfeedhobo> uesrs could add both the language repo and the mail repo, but that has other implications too
2025-10-24T20:55:53  <cornfeedhobo> oh ... maybe i misunderstand your reply to. Linkpac'd? is there a way to link without importing?
2025-10-24T20:56:18  <cornfeedhobo> i only took this path because there are already lots of other perl-* modules in the server:mail repo
2025-10-24T20:56:35  <acidsys> a maintainer of the target project can use osc linkpac to achieve it
2025-10-24T21:00:16  <cornfeedhobo> ahhhh. hmm. since i don't see any of them on irc, what is the preferred communication channel to nudge them to do that?
2025-10-24T21:00:45  <cornfeedhobo> i'm probably going to take his advice to use rspamd, but you know, i'd like dependencies to work as they are required.
2025-10-24T21:06:14  <acidsys> darix you can find on oftc
2025-10-24T21:06:46  <acidsys> otherwise I guess email