2026-07-16T01:04:47  *** tian_ is now known as tian2992
2026-07-16T12:48:12  <matrix-o-o> <asmorodskyi> Hi . id.opensuse.org throws a 500 again this time for bugzilla.opensuse.org . should I report it or it is something known ?
2026-07-16T13:00:15  <acidsys> that's not logical because bugzilla.opensuse.org does not use id.opensuse.org
2026-07-16T13:00:38  <acidsys> could you share which exact url you get 500 from?
2026-07-16T13:43:53  <cboltz> the   Large flush mail queue   we get in -alerts are strange - I checked, and there aren't any mails queued - and for example on svn.i.o.o, there is no mail log activity since ~midnight
2026-07-16T13:44:01  <cboltz> could it be that this alert mis-detects something?
2026-07-16T14:03:04  <acidsys> hi cboltz, there is indeed a value of 2 recorded for the metric: https://prometheus.infra.opensuse.org/query?g0.expr=postfix_queue_length%7Binstance%3D%22svn.infra.opensuse.org%22%7D&g0.show_tree=0&g0.tab=table&g0.range_input=1d&g0.res_type=auto&g0.res_density=medium&g0.display_mode=lines&g0.show_exemplars=0, and `find /var/spool/postfix/flush -type f` does show 2 files
2026-07-16T14:04:33  <acidsys> the rule triggers if flush is > 0 on machines without a more specific rule, so it does alert as intended ..
2026-07-16T14:05:17  <acidsys> if you could check if those two files are supposed to be there, that would be great
2026-07-16T14:08:03  <cboltz> these two files each contain a queue id (well, ~10 times the same id) of mails that were successfully sent on Jul 1st
2026-07-16T14:08:31  <cboltz> so I guess I should temporarily stop postfix and delete these files
2026-07-16T14:10:33  <cboltz> an interesting side finding for one of the mails is   to=<systemd-status-mail@svn.infra.opensuse.org> [...] (delivered to mailbox)
2026-07-16T14:11:32  <cboltz> turns out it is about a mail to root that was bounced - maybe we should add   systemd-status-mail   to /etc/aliases so that it ends up at admin-auto like mails to root (or at least bounces the same way ;-)
2026-07-16T14:15:46  <acidsys> is it supposed to use that sender, FROM defaults to "systemd"
2026-07-16T14:16:39  <acidsys> is that on multiple machines? if only on svn, it might have been from testing ..
2026-07-16T15:03:33  <cboltz> alerting also reports "flush mail queue" for gcc-stats and gitlab-runner[12], I still have to check these
2026-07-16T16:07:35  <cboltz> similar state on these 3 machines, cleaned up now
2026-07-16T16:08:51  <cboltz> this all (especially the delivery issues logged for the mails listed in the flush queue files) looks related to the broken master.cf files I fixed some weeks ago - so hopefully this won't happen again
2026-07-16T19:32:37  <matrix-o-o> <asmorodskyi> acidsys: you right I mixed up https://idp-saml.suse.com/ with id.opensuse.org