2024-07-19T06:22:58  <matrix-o-o> <Karthik (కార్తీక్)> acidsys: yes, i was about to say that 🙂
2024-07-19T07:24:45  <matrix-o-o> <tux93 (they/them)> are there any known issues with IDP login at the moment?
2024-07-19T07:24:45  <matrix-o-o> trying to log in to BOO gives me "wrong user name or password", logging in to OBS with the same credentials works fine
2024-07-19T07:25:41  <matrix-o-o> <Richard Rahl> Boo works for me
2024-07-19T07:27:12  <matrix-o-o> <tux93 (they/them)> hm, bugzilla sends me to SUSE IDP to log in, maybe that's my problem?
2024-07-19T07:28:31  <matrix-o-o> <Richard Rahl> It does also for me, so that shouldnt be the issue.
2024-07-19T08:58:53  <acidsys> cannot reproduce either. yes, bugzilla uses idp-saml.suse.com, that's correct
2024-07-19T10:36:59  <matrix-o-o> <Karthik (కార్తీక్)> acidsys: do we have any script on odin which can send emails ?
2024-07-19T11:13:21  <matrix-o-o> <Karthik (కార్తీక్)> the script is almost done. 2 things to figure out:
2024-07-19T11:23:05  <acidsys> hey Karthik, in shell we usually just use mailx, but maybe there's some easy native way in your Python script?
2024-07-19T11:23:27  <acidsys> it just needs to hand it to the local mailer
2024-07-19T11:24:04  <matrix-o-o> <Karthik (కార్తీక్)> acidsys: python has a mail module. i should explore that
2024-07-19T11:25:18  <matrix-o-o> <Karthik (కార్తీక్)> but, if there is a script / program which is preconfigured with smtp & related conf. I can use that as well
2024-07-19T11:26:37  <matrix-o-o> <Karthik (కార్తీక్)> i can call mailx from python if it's configured with required credentials
2024-07-19T11:32:22  <matrix-o-o> <Karthik (కార్తీక్)> here's how we do it with py https://docs.python.org/3/library/smtplib.html#module-smtplib
2024-07-19T11:33:36  <matrix-o-o> <Karthik (కార్తీక్)> but i wish to reply on OS programs if i have have an option of not dealing with secrets 😄
2024-07-19T11:33:42  <matrix-o-o> <Karthik (కార్తీక్)> * rely
2024-07-19T11:37:18  <acidsys> hm using the native python thing sounds cleaner than calling a subshell :-/ .. you don't need any authentication/secrets, there is a smtp server on localhost
2024-07-19T11:37:44  <acidsys> I can try it in a bit
2024-07-19T11:43:21  <acidsys> hmm I get relay access denied with both Python and mailx. I thought we could send outbound mail from machines .. will need to check mx's
2024-07-19T11:45:36  <acidsys> ah we need to add the os-odin network to pillar/role/mailserver.sls -> mynetworks
2024-07-19T11:49:07  <acidsys> after we've done that, something like the example here should be sufficient: https://docs.python.org/3/library/email.examples.html - no need to open a file, content can be inline
2024-07-19T11:52:55  <matrix-o-o> <Karthik (కార్తీక్)> acidsys: yeah, i will make it simple
2024-07-19T13:40:21  *** teepee_ is now known as teepee
2024-07-19T14:55:52  <acidsys> investigating storage problems in Prague
2024-07-19T15:23:18  <acidsys> things are coming back, falkor will need minor cleanup
2024-07-19T15:33:25  <acidsys> out of three mysql environments two survived, but our openSUSE one did not. I will restore it shortly
2024-07-19T15:39:13  <acidsys> now syncing galera1 -> galera2
2024-07-19T15:46:31  <acidsys> & 2 -> 3. but we still have a problem with falkor22, it failed to fence, potentially also because of i/o problems, I will reboot it once I have quorum for galera back
2024-07-19T16:08:48  <acidsys> it's a bit ugly because we still have falkor21 in standby because of the 15.6 issue. it's slowly balancing all to 20
2024-07-19T16:12:28  <lcp> is the mirrordb fine? matrix is screaming at me about it
2024-07-19T16:12:58  <acidsys> am still working on things, see past messages
2024-07-19T16:13:58  <lcp> oh, I thought you only mentioned galera, must have missed it
2024-07-19T16:14:30  <acidsys> sorry, I mean I did not yet get to check postgresql, but as it tries to move all to one node it might be slow
2024-07-19T16:15:03  <lcp> cool
2024-07-19T16:17:32  <acidsys> now 22 died on me ..
2024-07-19T16:17:58  <acidsys> (which includes vpn)
2024-07-19T16:24:20  <knurpht> Infra closed for the weekend. That'll teach them :P
2024-07-19T16:25:44  <acidsys> right. will label it as our new opening hours
2024-07-19T16:27:53  <cboltz> that, or blame Crowdstrike
2024-07-19T16:29:05  <acidsys> that one really struck the crowd
2024-07-19T16:31:34  <knurpht> Does the xz guy now work for them?
2024-07-19T16:33:15  <acidsys> now rebooting falkor22
2024-07-19T16:34:26  <acidsys> oh btw I forgot to post here, mid-debugging my desktop session crashed
2024-07-19T17:07:11  <a-865k> https://forums.opensuse.org/c/english/install-boot-login/18#:~:text=forums.opensuse.org%20took%20too%20long%20to%20respond. ERR_CONNECTION_TIMED_OUT
2024-07-19T17:12:09  <acidsys> it's not done yet
2024-07-19T17:25:44  <acidsys> ugh now more problems [system] The maximum number of pending replies for ":1.75" has been reached .. systemd-machined .. (max_replies_per_connection=128)
2024-07-19T17:39:56  <acidsys> at least we are not the only non-Windows people with issues today, bugs.freedesktop.org is down too
2024-07-19T18:02:02  <acidsys> ok, an eternity later, all VMs are started again. now one more round of galera syncing
2024-07-19T18:04:59  <acidsys> LCP: mirrordb2 should be fine again
2024-07-19T18:05:45  <acidsys> ah they will likely never see this: matrix systemd[1]: Failed to start Heisenbridge.
2024-07-19T18:10:39  <acidsys> I restarted it but I'm missing matrix-o-o in this channel
2024-07-19T18:15:31  <acidsys> galera/mysql is now back too
2024-07-19T18:18:48  <acidsys> I checked a few public services, etherpad needed a manual restart, others seem to have recovered by themselves
2024-07-19T18:22:16  <acidsys> quick systemd scan over all machines shows a few failing services which should be cleaned up but none sounding critical
2024-07-19T18:24:19  <acidsys> heisenbridge not sure why it crashes, prints "TimeoutError" without much useful information. I sent lcp a message in slack, but probably they won't read it there on a weekend
2024-07-19T19:58:56  *** teepee_ is now known as teepee
2024-07-19T20:08:23  <acidsys> matrix recovered itself
2024-07-19T21:24:23  <matrix-o-o> <LCP> hello matrix
2024-07-19T23:11:37  <matrix-o-o> <SFaulken> Hello LCP
2024-07-19T23:12:56  <matrix-o-o> <malcolmlewis> hello world