2025-06-22T00:39:37  *** teepee_ is now known as teepee
2025-06-22T08:25:08  *** teepee_ is now known as teepee
2025-06-22T12:23:20  *** teepee_ is now known as teepee
2025-06-22T15:35:48  <matrix-o-o> <malcolmlewis> On the build service I'm getting a "<urllib3.connection.HTTPSConnection object at 0x7f3704ac0690>: Failed to establish a new connection: [Errno 111] Connection refused" error when downloading packages, could this be a result of recent changes, or a build service issue?
2025-06-22T15:36:29  <matrix-o-o> <malcolmlewis> it downloads a few packages, then errors out
2025-06-22T18:14:46  <cboltz> interesting find on paste.i.o.o: /v/l/salt/salt-minion was rotated at midnight, but salt-minion still has /var/log/salt/minion-20250622 open
2025-06-22T18:15:05  <cboltz> I'll manually restart the minion to fix this, but nevertheless - any idea what could cause this?
2025-06-22T18:17:06  <cboltz> oh, more fun - now the deleted file is no longer open - but instead   lsof   prints lots of   lsof: no pwd entry for UID 64507   lines
2025-06-22T18:18:22  <cboltz> ps aux   shows that kanidm_unixd runs under that uid - but I have no idea why the earlier lsof call (before restarting the minion) didn't show these error messages
2025-06-22T18:21:08  <cboltz> checking via salt, I find the still-open minion-20250622 log and the lsof error message on lots of VMs
2025-06-22T19:03:03  <acidsys> sounds like https://github.com/saltstack/salt/issues/55631 and I can reproduce it locally quite easily by mv'ing minion log file away
2025-06-22T19:03:28  <acidsys> however it will still create and write to the new minion log file as soon as there is any logful activity
2025-06-22T19:03:54  <acidsys> so while it is ugly, it is not really a problem particularly as we reboot often .. that of course leaves your interesting lsof output
2025-06-22T19:04:31  <cboltz> the problem is that the file also gets compressed, so everything after that basically gets logged to /dev/null
2025-06-22T19:05:12  <acidsys> it holds the old /v/l/salt/minion-<date> file open, but new output is written to a freshly created /v/l/salt/minion
2025-06-22T19:06:31  <cboltz> are you sure? at least on paste.o.o, the minion log starts with the minion restart I did
2025-06-22T19:06:47  <acidsys> well was there any minion output?
2025-06-22T19:07:17  <cboltz> good question, I don't know
2025-06-22T19:08:11  <acidsys> https://paste.opensuse.org/pastes/6fd314156c55
2025-06-22T19:10:00  <acidsys> generally if "minion" does not exist, there was nothing to report. also we - probably stupidly - log to journal and file at the same time
2025-06-22T19:10:42  <cboltz> that, or the minion still logged to the deleted minion-$date
2025-06-22T19:11:04  <cboltz> that's hard to verify now that I restarted all minions, so we should check if it happens again
2025-06-22T19:11:22  <acidsys> well I just run the above locally
2025-06-22T19:11:25  <acidsys> you can do the same on any machine
2025-06-22T19:12:33  <acidsys> and it will happen again, as it's a bug
2025-06-22T19:13:44  <cboltz> indeed, the file gets created
2025-06-22T19:14:02  <acidsys> tada :)
2025-06-22T19:14:19  <cboltz> that makes the bug less harmful (it will "only" block some disk space, but that shouldn't be a serious issue)
2025-06-22T19:14:40  <acidsys> right, our minion logs are generally quite low volume
2025-06-22T19:15:39  <acidsys> that being said it would be nice to check if we can disable the file logging as there's not much use for the duplication
2025-06-22T19:18:33  <cboltz> sounds like something for the TODO list, I'd put it near the end in the "if we are ever bored" section ;-)
2025-06-22T19:19:11  <cboltz> BTW: I also found such still-open rotated/deleted logs for mailman and matrix - fixed by restarting the services, but we should check in some days if it happens again
2025-06-22T21:42:52  *** teepee_ is now known as teepee