2024-07-11T00:37:46  *** teepee_ is now known as teepee
2024-07-11T04:17:08  *** teepee_ is now known as teepee
2024-07-11T06:27:12  <matrix-o-o> <Firstyear (Firstyear)> Can someone do me a favour and remove this mirror from mc-au.o.o? It's dead.
2024-07-11T07:10:10  <acidsys> good morning, seems SUSE side storage changes broke some things, I am looking into it
2024-07-11T07:15:35  <acidsys> it seems our openSUSE side things recovered themselves and the issues (like redmine not loading) are caused by dead id.o.o
2024-07-11T07:55:58  <matrix-o-o> <bmwiedemann> acidsys: https://id.opensuse.org/ is still down - anything I can help with?
2024-07-11T07:57:39  <acidsys> hi bmwiedemann, yes it is quite dead, let me send you an update
2024-07-11T08:01:03  <acidsys> bmwiedemann: https://paste.opensuse.org/pastes/0e161060564c
2024-07-11T08:29:28  <acidsys> id.o.o (and with that progress.o.o) are back
2024-07-11T08:44:58  *** teepee_ is now known as teepee
2024-07-11T12:35:40  <matrix-o-o> <Karthik (కార్తీక్)> acidsys: which from mail to use fo notifying a stale vpn user ?
2024-07-11T12:39:41  <acidsys> maybe just admin@o.o as to have it reply-able? wdyt cboltz? ^
2024-07-11T12:44:39  <matrix-o-o> <Karthik (కార్తీక్)> acidsys: i mean "from" address.
2024-07-11T12:46:21  <acidsys> me too
2024-07-11T12:46:46  <cboltz> agreed, From: admin@o.o makes sense
2024-07-11T12:48:38  <matrix-o-o> <Karthik (కార్తీక్)> is there any way to ping other users from the "kanidm" cmd ?
2024-07-11T12:49:10  <acidsys> ping how?
2024-07-11T12:49:58  <matrix-o-o> <Karthik (కార్తీక్)> one kanidm user can message the other ?
2024-07-11T12:50:17  <acidsys> no.. it's just an authentication system
2024-07-11T12:50:44  <matrix-o-o> <Karthik (కార్తీక్)> acidsys: got iy
2024-07-11T12:50:46  <matrix-o-o> <Karthik (కార్తీక్)> * it
2024-07-11T12:51:22  <acidsys> btw since I think you wanted to work with python, you might like https://idm.infra.opensuse.org/docs/swagger-ui/
2024-07-11T13:07:14  <matrix-o-o> <Karthik (కార్తీక్)> acidsys: nice, thanks
2024-07-11T13:08:13  <acidsys> :)
2024-07-11T16:43:33  <matrix-o-o> <Karthik (కార్తీక్)> acidsys: here's the progress so far. notifying the user & admin has to be implemented which can be done on server
2024-07-11T16:43:39  <matrix-o-o> <Karthik (కార్తీక్)> https://git.sr.ht/~kskarthik/suse-vpn/tree/master/item/notify_inactive_vpn_accounts.py
2024-07-11T16:51:11  <acidsys> hi Karthik, pretty cool :) I think we could maintain it in here: https://gitlab.infra.opensuse.org/infra/salt/-/tree/production/salt/profile/vpn/openvpn/files. only thing besides the email bits is I think we may need to check for users which do not have any log file as well
2024-07-11T16:53:29  <matrix-o-o> <Karthik (కార్తీక్)> yes. with some inspection, we can also cover that case.
2024-07-11T16:56:20  <cboltz> that, or adjust our salt code so that it creates an empty log file for each user
2024-07-11T16:56:57  <acidsys> there can theoretically be users without vpn access
2024-07-11T17:00:46  <matrix-o-o> <Karthik (కార్తీక్)> cboltz: this implies, when a user is added to kanidm, a log file is created irrespective of their login status ?
2024-07-11T17:01:34  <cboltz> we have salt code that creates openvpn config sniplets for each user, so creating an empty log file could be done at the same time
2024-07-11T17:01:49  <matrix-o-o> <Karthik (కార్తీక్)> acidsys: check users from logs against kanidm ?
2024-07-11T17:01:59  <acidsys> *for users in the "vpn" group
2024-07-11T17:02:07  <matrix-o-o> <Karthik (కార్తీక్)> yes
2024-07-11T17:02:56  <matrix-o-o> <Karthik (కార్తీక్)> check users in vpn logs against users in vpn group, difference are the one's who didn't log even once
2024-07-11T17:03:06  <acidsys> alternatively to querying the vpn group one can check the /etc/openvpn/ccd-{udp,tcp}/$user files, those are present for every user in the vpn group, regardless of whether they log in
2024-07-11T17:03:18  <acidsys> but I think querying the vpn group is more reliable?
2024-07-11T17:03:46  <acidsys> given we need to query kani anyways for the email address
2024-07-11T17:11:18  <matrix-o-o> <Karthik (కార్తీక్)> acidsys: we can. since i have the logs you sent in hand, I used this solution.
2024-07-11T17:17:56  <matrix-o-o> <Karthik (కార్తీక్)> ❯ ssh -i .ssh/id_suse -J kskarthik@thor.infra.opensuse.org kskarthik@odin.infra.opensuse.org
2024-07-11T17:17:58  <matrix-o-o> (kskarthik@thor.infra.opensuse.org) Password:
2024-07-11T17:18:22  <matrix-o-o> <Karthik (కార్తీక్)> acidsys: since the above issues is bugging me since inception 😄
2024-07-11T17:20:37  <acidsys> try with -F /dev/null ?
2024-07-11T17:21:42  <matrix-o-o> <Karthik (కార్తీక్)> ssh -F /dev/null -i .ssh/id_suse -J kskarthik@thor.infra.opensuse.org kskarthik@odin.infra.opensuse.org(kskarthik@thor.infra.opensuse.org) Password:
2024-07-11T17:22:13  <matrix-o-o> <Karthik (కార్తీక్)> no use
2024-07-11T17:23:22  <acidsys> if you can ssh to thor.i.o.o directly, I don't see a reason why it wouldn't work if used as jump server
2024-07-11T17:23:47  <acidsys> what about some other machine, like -J thor.i.o.o mx-test.i.o.o
2024-07-11T17:27:00  <matrix-o-o> <Karthik (కార్తీక్)> acidsys: asks thor's password
2024-07-11T17:27:05  <matrix-o-o> <Karthik (కార్తీక్)> same like before
2024-07-11T17:38:55  <acidsys> I applied the kbdinteractive change now so it should no longer give the bogus passphrase prompt. But it will likely not help the problem you are facing
2024-07-11T17:39:24  <acidsys> you can try the hop with -A .. not so ideal but maybe worth trying if you don't use the key for anything else ..
2024-07-11T17:40:10  <matrix-o-o> <Karthik (కార్తీక్)> ssh -i .ssh/id_suse -J kskarthik@thor.infra.opensuse.org kskarthik@odin.infra.opensuse.org
2024-07-11T17:41:05  <matrix-o-o> <Karthik (కార్తీక్)> the err changed but, no successful infiltration 😉
2024-07-11T17:41:38  <acidsys> it just prints "Connection closed by authenticating user kskarthik" .. I can enable more logging
2024-07-11T17:41:40  <matrix-o-o> <Karthik (కార్తీక్)> my public key does not exist on odin ?
2024-07-11T17:41:47  <acidsys> it exists everywhere
2024-07-11T17:41:55  <matrix-o-o> <Karthik (కార్తీక్)> hmm... strange
2024-07-11T17:42:04  <acidsys> and there is nothing printed on odin so I think it fails at thor
2024-07-11T17:43:04  <acidsys> ok, now have loglevel DEBUG ..if you want to try again (will still fail, but maybe print something more useful for me)
2024-07-11T17:43:21  <matrix-o-o> <Karthik (కార్తీక్)> acidsys: just attempted
2024-07-11T17:43:43  <matrix-o-o> <Karthik (కార్తీక్)> a couple of times
2024-07-11T17:44:57  <acidsys> noticed .. and now once directly/only to thor please
2024-07-11T17:46:14  <acidsys> you mentioned that worked in the past, but now it also prints "Failed publickey for kskarthik"
2024-07-11T17:47:17  <matrix-o-o> <Karthik (కార్తీక్)> acidsys: done
2024-07-11T17:47:33  <acidsys> ok let me compare
2024-07-11T17:48:57  <acidsys> so when you ssh directly your ssh client is presenting key ED25519 SHA256:eo7A0j7FQVOyGYEisKmlUzySW0eRrennV26Ah15xokI (correct)
2024-07-11T17:49:09  <acidsys> but when you ssh with jump your ssh client is presenting ED25519 SHA256:flRj2rf4LkTS6vCfJ6w5n3s3rWDd67sWcGZgJcCTRUg (not correct)
2024-07-11T17:57:55  <matrix-o-o> <Karthik (కార్తీక్)> oh!
2024-07-11T19:20:08  <cboltz> any idea why fontinfo.o.o gives me the "openSUSE Static Content" page instead of the actual fontinfo page in at least half of the reloads?
2024-07-11T19:33:54  <acidsys> when it does load the real fontinfo page it has lots of assets yielding 404
2024-07-11T19:35:26  <cboltz> my guess is that some of the narwals have a misconfiguration so that they deliver static.o.o instead of the actual fontinfo vhost (which also mean the results depend on haproxy roulette) - but I didn't see an obvious issue in the nginx config for fontinfo
2024-07-11T19:36:33  <cboltz> however, I noticed that some of the narwals make .git and .svn a 404, and some don't have these sniplets in the config - which is strange because the config is salt-managed, and a highstate didn't change it
2024-07-11T19:36:52  <acidsys> salt -N narwal cmd.run 'curl -H \'Host: fontinfo.opensuse.org\' localhost|head -n20' suggests that narwal8 + ipx-narwal1 have real fontinfo and 7,5,6,4 have static.
2024-07-11T19:37:35  <acidsys> only one in prg2 having the real thing would match the ratio of bad hits from the internet
2024-07-11T19:40:08  <acidsys> also probably not related but potentially worth mentioning https://progress.opensuse.org/issues/163739
2024-07-11T19:41:49  <cboltz> right, I'll have to fix that with a revert because Lars removed too much, including chameleon-3 which is used by lots of our websites (I'll comment on https://github.com/openSUSE/static.opensuse.org/issues/1 when done)
2024-07-11T19:42:18  <cboltz> as much as I dislike submodules, we were lucky this time because they prevented the change to get deployed
2024-07-11T19:43:27  <acidsys> cool protection ;)
2024-07-11T19:47:47  <cboltz> revert pushed, which should also stop the mails on admin-auto
2024-07-11T19:48:24  <acidsys> thanks!
2024-07-11T19:48:38  <cboltz> re fontinfo, I wonder why most narwals serve static instead, while the nginx config looks correct
2024-07-11T19:49:28  <cboltz> maybe I overlooked something, can you please have a look?
2024-07-11T19:50:41  <acidsys> if I do the same `curl -H 'Host: fontinfo.opensuse.org' localhost|head -n20` on narwal7 I get no entry in /var/log/nginx/fontinfo.access.log (last entry from 2023!!)
2024-07-11T19:51:34  <cboltz> I guess you'll find a log entry in static.access.log instead
2024-07-11T19:52:00  <acidsys> I'm also confused where the 192.168.x lines in /etc/nginx/vhosts.d/fontinfo.opensuse.org.conf come from on narwal7.
2024-07-11T19:52:20  <cboltz> (and I'm also surprised that we didn't get a ticket about fontinfo. Even if it isn't our most important website, I'd expect at least some users.)
2024-07-11T19:53:50  <cboltz> the configfile has a 2020 timestamp, which probably means it isn't salt-managed (most other vhosts.d/* files have a 2024 timestamp)
2024-07-11T19:54:12  <cboltz> at least that explains the outdated IPs
2024-07-11T19:54:32  <acidsys> right .. head -n1 pillar/role/web_static.sls :-(
2024-07-11T19:54:52  <acidsys> is that one of those websites which were broken with themigration and someone copy pasted something somewhere and deemed it works now
2024-07-11T19:56:31  <cboltz> ok, I can now explain why it's broken:   listen 80
2024-07-11T19:56:49  <cboltz> while the other working websites have   listen [::]:80;
2024-07-11T19:57:49  <cboltz> adding fontinfo to web_static should fix that - one of these important one-line fixes ;-)
2024-07-11T19:57:50  <acidsys> ah and in haproxy we only healthcheck staticpages backend once for all related pages
2024-07-11T19:57:54  <cboltz> give me a minute
2024-07-11T20:01:45  <cboltz> https://gitlab.infra.opensuse.org/infra/salt/-/merge_requests/1951
2024-07-11T20:17:20  <cboltz> trying to apply highstate, but
2024-07-11T20:17:22  <cboltz>      Comment: No Top file or master_tops data matches found. Please see master log for details.
2024-07-11T20:17:35  <acidsys> just `chown -R 477:479 /srv/salt-git`
2024-07-11T20:17:48  <cboltz> ah, the usual suspect...
2024-07-11T20:24:05  <cboltz> I just learned that "git revert" does not revert .gitmodules changes, therefore I had to do that somewhat manually. We'll see in ~40 minutes if it silences the admin-auto mails for static.o.o
2024-07-11T20:25:36  <cboltz> fontinfo is back :-)  (mostly - highstate on narwal4 is still running)
2024-07-11T20:30:04  <acidsys> kewl
2024-07-11T20:34:33  *** Acinonyx_ is now known as Acinonyx
2024-07-11T20:37:55  <cboltz> minor issue in an alert from Tuesday:
2024-07-11T20:37:59  <cboltz> Description:  Failed to update member aliases through get_member_aliases on mx2.infra.opensuse.org, would have removed  entries.
2024-07-11T20:38:09  <cboltz> I'd expect   ... removed $number entries
2024-07-11T20:44:07  <acidsys> the email is not much better "New member-aliases list (0 entries) would remove 741 aliases ?"
2024-07-11T20:45:16  <cboltz> actually it is better - "would remove all aliases" basically means that for some reason it didn't get a valid response from mysql
2024-07-11T20:45:44  <cboltz> ideally it would error out earlier (directly after the mysql failure), but - details ;-)
2024-07-11T20:49:23  <acidsys> I see, value is missing from the labels
2024-07-11T20:52:01  <acidsys> https://gitlab.infra.opensuse.org/infra/salt/-/merge_requests/1952
2024-07-11T20:53:04  <cboltz> thanks, set to automerge
2024-07-11T21:15:23  *** teepee_ is now known as teepee
2024-07-11T21:36:39  <cboltz> huh? -alerts looks like lots of things are down, any idea what's going on?
2024-07-11T21:40:53  <acidsys> seems there is some more major-ish problem
2024-07-11T21:44:13  <acidsys> marcus asked if we have some problems "too"
2024-07-11T21:44:38  <acidsys> but I did not quite get what is broken yet. I can reach some things but not all
2024-07-11T21:45:51  <cboltz> yeah, for a very small amount of "some" ;-)
2024-07-11T21:46:17  <cboltz> looks like at least one of the narwals is alive, which means the static pages are available
2024-07-11T21:46:33  <cboltz> but everything else I tested is down
2024-07-11T21:47:19  <cboltz> bugzilla, build.o.o and openqa also run into timeouts
2024-07-11T21:47:44  <cboltz> so the actually surprising thing is that the static pages still work ;-)
2024-07-11T21:51:20  <ignis> is download.opensuse globally out or just regionally in the affected dc?
2024-07-11T21:53:12  <acidsys> ignis: globally, download.opensuse.org is just one host
2024-07-11T21:53:33  <ignis> 👍, good luck guys
2024-07-11T21:58:54  <acidsys> thanks
2024-07-11T23:45:40  <acidsys> I think it is a storage issue
2024-07-11T23:46:00  <matrix-o-o> <Firstyear (Firstyear)> acidsys: What happened? Everyone's running around like the house is on fire.
2024-07-11T23:53:36  <acidsys> storage outage
2024-07-11T23:54:00  <acidsys> so pretty much house on fire
2024-07-11T23:59:42  <acidsys> now being recovered (thanks to colleagues)