2025-09-28T02:28:31 *** teepee_ is now known as teepee 2025-09-28T09:21:47 Hi, something I was not expecting to write - no more of those emails for maybe 18 hours now. Thanks! 2025-09-28T09:29:44 *** cboltz_ is now known as cboltz 2025-09-28T09:30:35 I did some more adjustments on the mailservers yesterday - looks like I finally found the right ones 2025-09-28T09:31:19 "luckily" one of my mails was also resent multiple times, which means I could check on the sending server what was going on - basically a timeout at the end of DATA 2025-09-28T09:32:21 which means the sending server wasn't sure if the mail was received, and - to be sure, and as defined in the SMTP protocol - assumed the worst, and did send the mail again (and did possibly run into another timeout, and therefore send again) 2025-09-28T09:33:24 the changes that finally helped were 2025-09-28T09:33:29 1) increase the limit of how many spampd processes may run in parallel (luckily only needs some additional CPU, but not more RAM) 2025-09-28T09:34:26 2) decrease the limit of smtpd processes so that it's close to the number of spampd processes - better let sending servers wait when trying to connect than after DATA 2025-09-28T09:36:42 for some statistics, see https://monitor.opensuse.org/grafana/d/rYdddlPWk/node-metrics?orgId=1&from=2025-09-25T22:36:01.957Z&to=2025-09-28T08:12:49.365Z&timezone=browser&var-DS_PROMETHEUS=default&var-job=nodes&var-node=mx1.infra.opensuse.org&var-diskdevices=%5Ba-z%5D%2B%7Cnvme%5B0-9%5D%2Bn%5B0-9%5D%2B%7Cmmcblk%5B0-9%5D%2B 2025-09-28T09:36:57 (doesn't include mail count, but the network traffic should give you an idea) 2025-09-28T09:45:40 Hmm, I have a 4-line script consisting of: free ; swapoff -a ; swapon -a ; free I have no idea whether that would be useful in your situation. 2025-09-28T09:46:31 well, personally I would shorten it to swapoff -a ;-) 2025-09-28T09:46:56 because my experience is that having swap typically means it takes longer until the OOM kill jups in 2025-09-28T09:57:02 Oh, I do want swap, but I prefer it not to be in use. Those lines typically clear out a bit of cache (as far as I can see). 2025-09-28T09:59:07 I don't think I've needed it since I went from 8GB to 16GB. 2025-09-28T10:58:13 *** kl_eisbaer1 is now known as kl_eisbaer 2025-09-28T11:12:58 *** teepee_ is now known as teepee 2025-09-28T11:53:10 *** kl_eisbaer1 is now known as kl_eisbaer 2025-09-28T14:07:05 *** teepee_ is now known as teepee 2025-09-28T18:31:38 cboltz: re: mail count: jfyi we have it in https://monitor.opensuse.org/grafana/d/Ds5BxBYGk/e-mail-experimental?orgId=1&from=2025-09-21T18:30:37.764Z&to=2025-09-28T18:30:37.764Z&timezone=browser&var-node=$__all&viewPanel=panel-15 2025-09-28T18:34:43 looks like the number of mails in the mailq 2025-09-28T18:35:25 I was looking for "number of mails sent" - and while the traffic isn't an exact metric for that, it at least gives some insight 2025-09-28T18:36:37 nevertheless, thanks for the link - that 15k queued mails spike (probably after the security-announce mails were released) is quite impressive 2025-09-28T18:38:06 ah ok I think postfix_smtpd_messages_processed_total would have those but seems still WIP (sigh) 2025-09-28T18:40:02 no problem ;-) 2025-09-28T18:48:00 cboltz: please check jid 20250928184437320256 on witch1, something you want to commit? 2025-09-28T19:00:03 indeed, some of these changes are the fixes to prevent timeouts at end of DATA 2025-09-28T19:03:15 (and seeing that authorized-exec would be installed explains why the certificate update doesn't work ;-) 2025-09-28T19:15:26 https://gitlab.infra.opensuse.org/infra/salt/-/merge_requests/2579 2025-09-28T19:22:14 https://gitlab.infra.opensuse.org/infra/salt/-/jobs/507299 - I see two issues there 2025-09-28T19:22:37 - unrelated to the MR: Jinja: PASS, but the check reported two errors 2025-09-28T19:22:53 - triggered by the MR: Jinja variables should have spaces before and after: '{{ var_name }}' 2025-09-28T19:23:03 1. I'm aware, j2lint already does not work since some time, 2. I already commented what you need to add 2025-09-28T19:23:22 ok, thanks! 2025-09-28T19:24:50 updated version pushed, let's see if the CI is happy now 2025-09-28T19:27:06 regarding j2lint, I'm not sure yet, it requires some patches, but maybe I want to remove it instead, the upstream mostly targets the jinja implementation in a product of theirs and not all jinja cases out of the box (which is understandable) 2025-09-28T19:27:49 you put "nooqa" (double o) 2025-09-28T19:28:11 we have nothing against oQA ;-) 2025-09-28T19:29:58 that's what I get from copy&paste ;-) - fixed 2025-09-28T19:30:33 oh, just wanted to check if you pay attention! 2025-09-28T19:30:43 ;-) 2025-09-28T19:53:34 deployed (including authorized_exec), so the certificate update should work on the next run 2025-09-28T20:04:05 thanks