2024-07-23T07:45:52  <matrix-o-o> <Karthik (కార్తీక్)> acidsys: let me know when you allow odin to send mails, I will do a dry run
2024-07-23T08:19:22  *** teepee_ is now known as teepee
2024-07-23T10:02:33  <acidsys> Karthik: feel free to add a commit for it
2024-07-23T10:09:55  <matrix-o-o> <Karthik (కార్తీక్)> acidsys: i will try. should the .py extension -> to jinja ?
2024-07-23T10:11:29  <acidsys> for your script, it would be nice if you make it .py.jinja and add the {{ pillar['managed_by_salt'] }} on the top after the shebang (to have it render our banner)
2024-07-23T10:11:54  <acidsys> (not technically mandatory, just our convention)
2024-07-23T10:24:46  <matrix-o-o> <Karthik (కార్తీక్)> acidsys: done.
2024-07-23T10:24:51  <matrix-o-o> <Karthik (కార్తీక్)> acidsys: sure
2024-07-23T10:40:24  <matrix-o-o> <Karthik (కార్తీక్)> https://matrix.opensuse.org/_matrix/media/v3/download/poddery.com/GiEbEhbDWHykTklehdmneucT/image.png
2024-07-23T10:40:32  <matrix-o-o> <Karthik (కార్తీక్)> like this ^
2024-07-23T10:40:34  <matrix-o-o> <Karthik (కార్తీక్)> * ^?
2024-07-23T10:42:50  <acidsys> yep or just add it to your description comment block if you don't want it to get interpreted as python in your IDE
2024-07-23T10:54:03  <matrix-o-o> <Karthik (కార్తీక్)> acidsys: done.
2024-07-23T10:55:25  <matrix-o-o> <Karthik (కార్తీక్)> and pushed
2024-07-23T10:57:52  <acidsys> looks good
2024-07-23T18:49:57  <matrix-o-o> <malcolmlewis> acidsys: how long does the syncing usually take?
2024-07-23T18:50:10  <matrix-o-o> <Karthik (కార్తీక్)> feels like a bumpy day today
2024-07-23T18:50:19  <matrix-o-o> <smolsheep> Karthik (కార్తీక్): as long as build.o-o stays up I'm happy enough, even if everything else is rocky. but yes
2024-07-23T18:52:41  <cboltz> I'm afraid you can't be too happy then - the OBS main page works, but looking at a repo ends up with a timeout
2024-07-23T18:53:39  <matrix-o-o> <bmwiedemann> Is there a new big outage?
2024-07-23T18:53:48  <acidsys> we're almost back
2024-07-23T18:56:02  <acidsys> bmwiedemann: you could have the honors to check download.o.o please? :)
2024-07-23T19:01:01  <acidsys> heroes infra is fully back now
2024-07-23T19:01:17  <acidsys> bmwiedemann: nevermind, they are still working on cirrus
2024-07-23T19:04:32  <acidsys> team is still working on the OBS side of things
2024-07-23T19:04:51  <acidsys> keeping status.o.o incident open til then
2024-07-23T19:04:52  <chris_goe> Greetings. I was just about to ask what's the status with the outage, but channel logs, nice! So, godspeed then, hoping for a easy recovery! :-)
2024-07-23T19:05:21  <acidsys> hi chris_goe, we're mostly back besides download/OBS. thanks :)
2024-07-23T19:05:31  <chris_goe> \o/
2024-07-23T19:06:02  <cboltz> mailing lists are still listed as down, is this intentional? (lists.o.o works, but I guess we have a nice _external_ mail queue)
2024-07-23T19:06:56  <chris_goe> I'm surprised this hasn't hit the forums yet, that's where I looked first.
2024-07-23T19:09:26  <acidsys> chris_goe: the forums were gone too :D
2024-07-23T19:10:08  <acidsys> cboltz: thanks, overlooked. yes, mail queue will take a bit to recover
2024-07-23T19:17:53  <chris_goe> hah, OK. Sadly, IRC isn't linked on www.o.o, but it's good it was mentioned on status.o.o!
2024-07-23T19:18:32  <s|oO|0o|g> starting to look good! GJ to everyone involved
2024-07-23T19:19:39  <acidsys> :)
2024-07-23T19:21:12  <s|oO|0o|g> kinda curious about the original culprit... "storage backend software downgrade", is it something akin to HP Nimble arrays that is used? I would have guessed you'd run e.g. ZFS on white box hardware
2024-07-23T19:21:34  <s|oO|0o|g> maybe a Ceph cluster
2024-07-23T19:26:32  <s|oO|0o|g> is the storage backend imposed by the Prague DC (as in you rent capacity)?
2024-07-23T19:26:37  <acidsys> it's two NetApp storage clusters, and apparently a bug in their new software version causing it to crash
2024-07-23T19:26:44  <s|oO|0o|g> nvm nerding out a bit, none of my business :D
2024-07-23T19:27:04  <s|oO|0o|g> ah
2024-07-23T19:27:10  <acidsys> it's operated by SUSE. we as openSUSE can use it for our needs too
2024-07-23T19:27:43  <s|oO|0o|g> oh boy... so even the corpo side got hit?
2024-07-23T19:30:33  <acidsys> let's say, it was an interesting day for everyone
2024-07-23T19:32:53  <s|oO|0o|g> let's all sit back, grab some popcorn and watch the Finger Pointing Olympics? :D
2024-07-23T19:34:03  <s|oO|0o|g> "Why not during the Crowdstrike mess?!?! That could have flown under the radar!"
2024-07-23T19:34:25  <acidsys> luckily we don't have a finger pointing culture
2024-07-23T19:34:37  <acidsys> but yes people said they want to fix the netapp thing to finally rest in peace and just blame crowdstrike :P
2024-07-23T19:36:23  <s|oO|0o|g> no finger pointing, wow that's rare, cool! so you get some resources from SUSE's pool rather than have a dedicated testbed?
2024-07-23T19:37:06  <s|oO|0o|g> ffs here I go again, don't mind me
2024-07-23T19:37:15  <acidsys> human or technical resources?
2024-07-23T19:37:27  <s|oO|0o|g> technical
2024-07-23T19:38:45  <acidsys> from the technical side, SUSE provides fancy colocation including hardware for openSUSE
2024-07-23T19:39:10  <acidsys> at least that is how it's in Prague now. in Provo, we still have a legacy setup where SUSE operates the hardware, this will change after migration to a new location as well
2024-07-23T19:39:49  <s|oO|0o|g> ok
2024-07-23T19:42:25  <acidsys> download.o.o/OBS are now back as well, but the team is still doing some post-work
2024-07-23T19:47:13  <s|oO|0o|g> well, thanks all for your work and patience! cya around :)
2024-07-23T19:47:34  <acidsys> thanks for stopping by :)
2024-07-23T19:47:44  <s|oO|0o|g> np!
2024-07-23T21:25:09  <cboltz> FYI: I just created a basic AppArmor profile for apache on status2
2024-07-23T21:25:33  <acidsys> thank you!
2024-07-23T21:25:44  <cboltz> it covers the obvious things (restarting apache, people visiting status.o.o), but probably still needs some additions
2024-07-23T21:25:52  <cboltz> which also means it's in complain mode for now
2024-07-23T21:27:15  <cboltz> I also noticed there's a profile for /usr/sbin/mariadbd which couldn't be loaded because it included two non-existing abstractions (rpm, zypp)
2024-07-23T21:27:37  <cboltz> I commented out these abstractions, and switched the profile to complain mode to avoid that it breaks mariadb
2024-07-23T21:28:09  <acidsys> interesting why mariadbd would need rpm or zypp abstractions
2024-07-23T21:28:54  <cboltz> maybe one of them (while they existed) had something that matched a log event, and whoever created the profile mis-choose the abstraction
2024-07-23T21:30:14  <chris_goe> cool beans, thanks to everyone involved getting this all back online!
2024-07-23T21:30:30  <cboltz> I had to add a few rules to the mariadb profile after removing these abstractions, but nothing too bad
2024-07-23T21:32:11  <cboltz> for now, audit.log stays silent - but we should keep an eye on it in the next days
2024-07-23T22:00:08  *** teepee_ is now known as teepee