2024-03-27T00:10:36  <matrix-o-o> <Firstyear (Firstyear)> Test
2024-03-27T00:10:42  <matrix-o-o> <SFaulken> Fail.
2024-03-27T00:10:58  <matrix-o-o> <Firstyear (Firstyear)> Good bye irc.
2024-03-27T00:11:04  <matrix-o-o> <SFaulken> At least we get your PFP, when you're here on Matrix.
2024-03-27T00:12:42  <acidsys> is pfp a less controversial abbreviation for profile picture ?
2024-03-27T00:13:04  <matrix-o-o> <SFaulken> yup, it's a discord term, as far as I know.
2024-03-27T00:13:28  <matrix-o-o> <SFaulken> at least that's the first place I ever encountered it.
2024-03-27T00:13:56  <acidsys> interesting. in my circles it's always been propic :p
2024-03-27T00:14:15  <matrix-o-o> <Firstyear (Firstyear)> I always heard DP for display picture but yeah. PFP works.
2024-03-27T00:19:24  <acidsys> better than hdmi
2024-03-27T00:20:33  <matrix-o-o> <Firstyear (Firstyear)> rofl
2024-03-27T00:20:45  <matrix-o-o> <Firstyear (Firstyear)> That was such a stupid debate.
2024-03-27T00:20:53  <acidsys> lol
2024-03-27T00:22:46  <matrix-o-o> <knurpht - Gertjan> Firstyear (Firstyear): Not a debate, nor stupid. It was mutual sharing of personal impressions in life
2024-03-27T01:00:29  <acidsys> if someone is still up and motivated, this came in: Notice(heroes-monitor): Alert! Low memory on matrix.infra.opensuse.org is firing .. otherwise I check later
2024-03-27T01:03:29  <matrix-o-o> <knurpht - Gertjan> acidsys: I have no access. Should I be scared that it forgets all my wonderful posts ?
2024-03-27T22:56:23  <matrix-o-o> <Firstyear (Firstyear)> acidsys: Which reminds me, speaking of monitoring how do I get enrolled for alerts on things.
2024-03-27T22:58:47  <acidsys> hi firstyear, critical alerts go to the admin-auto@ mailing list, warnings go to #opensuse-admin-alerts (still being worked on though), and https://alerts.infra.opensuse.org gets all categories (critical+warning+info)
2024-03-27T22:59:44  <acidsys> and besides alerts there are various pretty dashboards in https://monitor.opensuse.org/grafana/
2024-03-27T23:04:34  <matrix-o-o> <Firstyear (Firstyear)> Hmmm okay. I probably need to look at that more later to find a way to get IDM alerts.
2024-03-27T23:05:20  <acidsys> I want to monitor Kanidm, but I have to set up collection for OpenTelemetry first (we currently only use native Prometheus metrics)
2024-03-27T23:05:43  <acidsys> but you can already find basic machine and HAProxy data which might be useful
2024-03-27T23:07:27  <matrix-o-o> <Firstyear (Firstyear)> Ahhh.
2024-03-27T23:07:48  <matrix-o-o> <Firstyear (Firstyear)> You probably want to speak to yaleman about that's, he's the otel person.
2024-03-27T23:08:02  <acidsys> yep you already showed me once what they were working on
2024-03-27T23:08:48  <acidsys> it's quite cool but I will want to feed it into Prometheus (there's experimental otel support now I think) so I don't have to set up something like Tempo just for one service
2024-03-27T23:09:56  <matrix-o-o> <Firstyear (Firstyear)> For now, if we open up direct https/ldaps to kani1/kani2 and port 12346 on kani1 then i can just monitor it directly with my home nagios setup
2024-03-27T23:10:04  <matrix-o-o> <Firstyear (Firstyear)> They're all safe to open btw,
2024-03-27T23:10:33  <acidsys> I don't like that, what data would you get out of your nagios which you cannot get out of our monitoring?
2024-03-27T23:11:12  <matrix-o-o> <Firstyear (Firstyear)> Well, I have no idea how the monitoring for opensuse works 🤣
2024-03-27T23:11:22  <matrix-o-o> <Firstyear (Firstyear)> And how to enroll my self to only specific alerts
2024-03-27T23:11:59  <matrix-o-o> <Firstyear (Firstyear)> but the big one rn, is individual server health, but the port 12346 is for ipa ->  kani sync monitoring.
2024-03-27T23:13:09  <matrix-o-o> <Firstyear (Firstyear)> And we don't need otel for anything like that, we just nede "check ports"
2024-03-27T23:13:36  <acidsys> allowing individual people to receive personal alerts for individual machines was more a long term idea of mine, but if you already want it I can easily add an email address or similar to match your machines
2024-03-27T23:13:48  <acidsys> if you just want to check if a port is open that is relatively easy
2024-03-27T23:14:04  <matrix-o-o> <Firstyear (Firstyear)> Nah, there is actually some data to process on the 12346 port.
2024-03-27T23:14:12  <matrix-o-o> <Firstyear (Firstyear)> You need to check for specific responses.
2024-03-27T23:14:36  <matrix-o-o> <Firstyear (Firstyear)> The reason I suggested just opening it up now, is I already have all the templates for nagios good to go so we can monitor "today" and then solve it for the osuse monitoring as we go.
2024-03-27T23:14:53  <acidsys> well show me what you have
2024-03-27T23:15:52  <matrix-o-o> <Firstyear (Firstyear)> Getting it
2024-03-27T23:16:40  <matrix-o-o> <Firstyear (Firstyear)> define service {
2024-03-27T23:16:40  <matrix-o-o> ... long message truncated: https://matrix.opensuse.org/_matrix/media/v3/download/opensuse.org/CmYsLZmYFOrMUgnLSycxzqsc (22 lines)
2024-03-27T23:17:52  <matrix-o-o> <Firstyear (Firstyear)> This is just basic checks for now.
2024-03-27T23:19:11  <acidsys> theoretically we still have the legacy monitoring-plugins (collected with nrpe) running (which I work on replacing) but this seems simple enough to implement, parts I need for other services anyways
2024-03-27T23:19:38  <matrix-o-o> <Firstyear (Firstyear)> 👍
2024-03-27T23:19:59  <acidsys> so I will look into it the next days then we can do it properly to begin with
2024-03-27T23:21:12  <matrix-o-o> <Firstyear (Firstyear)> Sounds like a plan. Do you want me to help with the machine sssd -> kani client converts? I can do them "overnight" while most people are asleep etc.
2024-03-27T23:23:30  <acidsys> sure, though once we merge it into role.base we can just apply it everywhere
2024-03-27T23:24:11  <acidsys> I already wrote salt/profile/kanidm/client/legacy.sls to remove the old bits
2024-03-27T23:28:00  <matrix-o-o> <Firstyear (Firstyear)> Wouldn't we want to convert the machines a few at a time to catch any problems?
2024-03-27T23:28:08  <matrix-o-o> <Firstyear (Firstyear)> As much as I'm keen for "all in"
2024-03-27T23:30:27  <acidsys> I don't expect much issues myself but if you are motivated to do batches then of course you may ^^
2024-03-27T23:30:38  <matrix-o-o> <Firstyear (Firstyear)> hahaha
2024-03-27T23:30:50  <matrix-o-o> <Firstyear (Firstyear)> I don't expect issues, but the rule is "hope for the best, prepare for the worst"
2024-03-27T23:31:05  <matrix-o-o> <Firstyear (Firstyear)> How about I manually roll out to a few machines as a trial first, then we can just do the rest once we have a sample converted.
2024-03-27T23:31:26  <acidsys> sure
2024-03-27T23:31:32  <matrix-o-o> <Firstyear (Firstyear)> I'll temporarily need to be in the wheel group to do that though. But I'm happy to be removed after.
2024-03-27T23:31:47  <matrix-o-o> <Firstyear (Firstyear)> Okay, I'll look through gitlab and stuff today and check everything and I'll do a PR later.
2024-03-27T23:32:19  <matrix-o-o> <Firstyear (Firstyear)> When you're young, people are so keen to have admin on machines, and then you get old and know that access to machines is radioactive 🤣
2024-03-27T23:32:29  <acidsys> sure but you don't have to go to each machine, saltmaster-admins lets you use witch1.i.o.o
2024-03-27T23:33:03  <acidsys> then just `sudo salt <machine> state.apply profile.kanidm.client`
2024-03-27T23:33:44  <matrix-o-o> <Firstyear (Firstyear)> Ahhhh cool.
2024-03-27T23:34:08  <matrix-o-o> <Firstyear (Firstyear)> Is it called witch1 becausu it deploys curses to the broader population?
2024-03-27T23:34:42  <acidsys> the previous one was called minnie .. witch in disguise
2024-03-27T23:40:27  <acidsys> you can merge !1617 before or after .. I recommend before so as to not accidentally "roll back" some machines to FreeIPA
2024-03-27T23:41:42  <matrix-o-o> <Firstyear (Firstyear)> Lol.
2024-03-27T23:41:45  <matrix-o-o> <Firstyear (Firstyear)> yeah, lol.
2024-03-27T23:41:50  <matrix-o-o> <Firstyear (Firstyear)> I'll review that today.
2024-03-27T23:41:57  <acidsys> some potentially useful commands for you: https://paste.opensuse.org/pastes/03a22c3d193f, https://paste.opensuse.org/pastes/d0b7a9267726
2024-03-27T23:42:13  <matrix-o-o> <Firstyear (Firstyear)> Thank you!
2024-03-27T23:42:23  <acidsys> :)
2024-03-27T23:42:36  <matrix-o-o> <Firstyear (Firstyear)> I'll review all this, make a plan and have a look :)
2024-03-27T23:42:56  <matrix-o-o> <Firstyear (Firstyear)> I'm a bit of a boomer like that, I like to plan out all the commands and have them in notes before I start anything.
2024-03-27T23:44:16  <acidsys> it's good practice
2024-03-27T23:49:56  <matrix-o-o> <Firstyear (Firstyear)> Nahhhh. Just yolo everything to prod without backups and without testing.
2024-03-27T23:50:10  *** teepee_ is now known as teepee
2024-03-27T23:54:16  <acidsys> why without testing, if you yolo to prod you are effectively testing
2024-03-27T23:55:30  <matrix-o-o> <Firstyear (Firstyear)> rofl
2024-03-27T23:55:45  <matrix-o-o> <Firstyear (Firstyear)> "everyone has a dev environment, some of us are lucky its separute to production
2024-03-27T23:55:59  <acidsys> haha
2024-03-27T23:58:11  <acidsys> now matrix.i.o.o is running out of disk space in addition to out of memory
2024-03-27T23:58:50  <acidsys> 63G     /var/log/matrix-synapse