2022-12-17T00:13:27  <Luciano[m]> 😁
2022-12-17T05:34:51  *** teepee_ is now known as teepee
2022-12-17T10:50:47  <pjessen> has mailman died ?
2022-12-17T12:24:39  <hellcp> pjessen: yesterday we had both postfix and nginx die
2022-12-17T12:25:17  <hellcp> as far as I know mailman and mailman-web were stable through all of that (though obviously not particularly functional without nginx and postfix)
2022-12-17T12:28:57  <robin_listas> https://lists.opensuse.org/manage/lists/users-es.lists.opensuse.org/ in my firefox doesn't finish loading, it is waiting on seccdn.libravatar.org, whatever that is.
2022-12-17T12:29:52  <robin_listas> https://lists.opensuse.org/archives/list/users-es@lists.opensuse.org/thread/6C6WOSRIJQY7AKCQ3L3LMCAS4CDCJ4S7/ in my firefox doesn finish loading, it is waiting on beans.opensuse.org, whatever that is.
2022-12-17T12:29:56  <hellcp> it's the avatar service
2022-12-17T12:30:23  <hellcp> beans is a tracker, firefox blocks those by default afaik
2022-12-17T12:31:19  <hellcp> I don't think having a tracker loaded is all that essential, we will be fine without tracking you in particular
2022-12-17T12:32:10  <robin_listas> I don't think I enabled or disabled anything. But it did not happen three days ago, same machine.
2022-12-17T12:32:50  <robin_listas> FF was updated, though.
2022-12-17T12:33:47  <hellcp> no idea, I wouldn't really worry about it not loading though
2022-12-17T12:35:17  <robin_listas> Well, I see a 20% cpu load in FF. If I close those tabs, goes to 4%.
2022-12-17T12:36:17  <hellcp> that's a bit more concerning
2022-12-17T12:36:57  <robin_listas> Yep. Of course, this is a puny laptop, celeron cpu.
2022-12-17T12:37:43  <hellcp> hm, does the page finish loading all the content besides the beans?
2022-12-17T12:37:55  <hellcp> like, do you see rotating spinners?
2022-12-17T12:38:11  <robin_listas> Dunno, I can try. I'll open them again and comment back in a while
2022-12-17T12:39:31  <hellcp> I should pick up an older machine for this kind of testing, the older one around is intel broadwell midrange cpu, which is a bit too powerful for this kinda stuff
2022-12-17T12:39:39  <hellcp> oldest*
2022-12-17T12:40:53  <robin_listas> Hum. Those two links now loaded normally. Now it is https://lists.opensuse.org/archives/ which is busy, but doesn't say why.
2022-12-17T12:41:47  <hellcp> the network tab under f12 would probably answer that
2022-12-17T12:42:26  <hellcp> you are looking for an entry that doesn't have a status reported yet there
2022-12-17T12:42:48  <hellcp> (the number on the left)
2022-12-17T12:44:54  <robin_listas> Huh? I have nothing on f12. You mean ctrl-alt-f12? Fully black. New feature perhaps? I am on Leap.
2022-12-17T12:45:33  <hellcp> it's either f12 or ctrl shift i on firefox
2022-12-17T12:45:41  <robin_listas> Ah
2022-12-17T12:46:07  <hellcp> ctrl alt f12 is switching to tty12
2022-12-17T12:46:26  <hellcp> you can go back to tty2 with ctrl alt f2
2022-12-17T12:47:24  <hellcp> (it's certainly not a new feature, it has been around since linux was a thing as far as I know)
2022-12-17T12:47:30  <robin_listas> It shows to circle graphs, and says "waiting"
2022-12-17T12:47:42  <robin_listas> two circle...
2022-12-17T12:48:24  <robin_listas> Yeah, but I didn't see you meant f12 on FF.
2022-12-17T12:50:36  <robin_listas> Ok. Now I get a table. Most show green, a few are blank and showing "GET".
2022-12-17T12:51:09  <hellcp> the blank ones are the ones that aren't loaded
2022-12-17T12:51:50  <robin_listas> Several on list.opensuse.org, type "js", except two on static.opensuse.org
2022-12-17T12:52:03  <hellcp> or that are cached it looks like
2022-12-17T12:52:36  <hellcp> hm, right
2022-12-17T12:53:53  <hellcp> this may be a bit of a computationally expensive operation, but could you try ctrl+f5, that's a hard refresh without cache
2022-12-17T12:57:18  <pjessen> hellcp: I still can't logon but it's not important
2022-12-17T12:59:26  <pjessen> what did postfix and nginx die of?  oom killer?
2022-12-17T12:59:45  <hellcp> pjessen: would you create a ticket for inability to log in?
2022-12-17T13:00:12  <hellcp> I believe so, the machine has been really overwhelmed ever since the timeouts were extended
2022-12-17T13:01:02  <hellcp> it sometimes freezes up entirely and you just can't ssh into it or use the website
2022-12-17T13:02:14  <pjessen> hellcp: sure.  it's pretty weird, I just get "connection closed", much like thursday.  Thursday it then started wokring again later.
2022-12-17T13:03:02  <pjessen> something's gotta be wrong with our setup, I just don't know what it might be
2022-12-17T13:27:20  <pjessen> and now I could suddenly log in
2022-12-17T13:28:13  <acidsys> yes, OOM killer
2022-12-17T13:28:28  <acidsys> see kernel log
2022-12-17T13:28:32  <pjessen> right
2022-12-17T13:28:48  <pjessen> pretty weird for postfix to be killed
2022-12-17T13:29:01  <acidsys> last entry is about gunicorn being killed
2022-12-17T13:29:21  <acidsys> s/entry/entries/
2022-12-17T13:29:27  <pjessen> that makes more sense
2022-12-17T13:30:12  <pjessen> I see systemd-journal isn't too happy, taking coredumps
2022-12-17T13:31:26  <pjessen> I wish we could change those huge nginx mapfiles to a table lookup instead, like on apache.
2022-12-17T15:50:26  <Luciano[m]> pjessen: Jacob had the idea of using some sort of cgi script, but on HAproxy instead, to deal with those redirections. Do you think using a table would be better?
2022-12-17T16:54:47  *** teepee_ is now known as teepee
2022-12-17T17:03:26  <pjessen> Luciano[m]: I think a table would be less complicated. Essentially just a file that is read everytime someone wants something from the old archives.
2022-12-17T17:03:50  <pjessen> it means a little cpu time, I I think its worth trading it for memory
2022-12-17T17:03:58  <pjessen> it means a little cpu time, but I think its worth trading it for memory
2022-12-17T17:04:35  <pjessen> I don't know what nginx can do though, but apache has a really neat way of doing such maps.
2022-12-17T17:08:56  <acidsys> https://gist.github.com/danackerson/af7481a869396839b3da
2022-12-17T17:11:29  <cboltz> That table could also be a database table. A bit more overhead, but a database might be faster when doing the lookup than a self-made script scanning a text file
2022-12-17T17:11:44  <acidsys> cboltz: see my link ;-)
2022-12-17T17:14:49  <cboltz> I thought more about using mysql instead of an in-memory db, but - details ;-)  (and yes, I'm aware that redis also supports durability, even if I don't know the details)
2022-12-17T17:20:59  <Luciano[m]> Those are interesting options, as long as mailman3 can get a breath of fresh air...
2022-12-17T17:28:09  <Luciano[m]> Heh, mailman3 is scream once again in Icinga Web.
2022-12-17T17:29:30  <pjessen> cboltz: redis uses up memory too. I would be happy with using mysql, locally.
2022-12-17T17:31:11  <cboltz> I'd even use galera - less setup work (just another database), and read access never was a problem
2022-12-17T17:44:12  <cboltz> nice - mailman3 had a load of 73 (~ 10 minutes ago), and sssd was killed so that non-root ssh logins were broken
2022-12-17T17:44:39  <cboltz> restarting sssd (via salt) helped, ssh login works again, and the load is back to 0.something
2022-12-17T17:46:20  <Luciano[m]> Those tantrums seem to be intermittent, since yesterday.
2022-12-17T17:49:51  <Luciano[m]> And in the meantime, I think my Salt MR is good to go. For now, it's basically just adding the sudoers snippet file and putting the extra package repo in the config - it's already added in the VM.
2022-12-17T17:52:55  <cboltz> I'm afraid the CI pipeline disagrees ;-) - see the "validate" pipeline for the reason
2022-12-17T17:54:17  * Luciano[m] takes a look!
2022-12-17T17:56:14  <pjessen> 73 ?  did you happen to notice of what?  gunicorn and mailman are both limited
2022-12-17T17:56:59  <cboltz> unfortunately not, I requested /proc/loadavg and free -m via salt, and it took several minutes until I got the result
2022-12-17T17:57:56  <cboltz> (free reported about 3 GB as free and available, and buff/cache nearly empty - I guess OOM killer did its job)
2022-12-17T17:58:42  <pjessen> oom killed gunicorn which is fine.  systemd-journal still coredumps though
2022-12-17T18:10:34  <Luciano[m]> Yay, first time I'm seeing 3 tests giving green flag in the pipeline.
2022-12-17T18:27:33  <cboltz> merged
2022-12-17T18:28:54  <Luciano[m]> Thanks for that. 5 tests green... My newest record :^P
2022-12-17T18:29:51  <cboltz> ;-)
2022-12-17T19:11:09  <hellcp> Luciano: run highstate on jenkins machine, so you should be able to sudo on it presumably
2022-12-17T19:11:49  <Luciano[m]> Let me try it!
2022-12-17T19:12:01  <hellcp> hm, doesn't appear like it will work
2022-12-17T19:15:13  <hellcp> oh, apparently I had to run highstate twice
2022-12-17T19:15:13  <Luciano[m]> I issued salt-minion state.highstate, is that the right command?
2022-12-17T19:15:13  <hellcp> nvm
2022-12-17T19:15:27  <hellcp> just highstate should be enough
2022-12-17T19:16:02  <hellcp> highstate is an alias for the root user though I think
2022-12-17T19:16:35  <Luciano[m]> I'm getting "[ERROR   ] Could not cache minion ID: [Errno 13] Permission denied: '/etc/salt/minion_id'".
2022-12-17T19:16:37  <hellcp> indeed, you need sudo for that
2022-12-17T19:16:39  <hellcp> anyway, now you should be able to log in
2022-12-17T19:16:42  <hellcp> as root
2022-12-17T19:16:49  <hellcp> as in sudo
2022-12-17T19:17:09  <cboltz> FYI: there's also a   testhighstate   alias
2022-12-17T19:17:18  <Luciano[m]> OK, let's see.
2022-12-17T19:17:59  <hellcp> I'm running highstate for the third time because of errors from somebody else already messing about with that vm
2022-12-17T19:18:18  <hellcp> (there was an existing repo file with the contents we want)
2022-12-17T19:18:32  <hellcp> and now it's successful, perfect
2022-12-17T19:19:26  <Luciano[m]> There was, so I should've skipped that part in the salt configs?
2022-12-17T19:20:05  <hellcp> it's better for the state of the vm to be salted than not salted
2022-12-17T19:20:24  <hellcp> that clearly was not in the salt repo
2022-12-17T19:21:32  <Luciano[m]> Ah, nice!
2022-12-17T19:23:23  <Luciano[m]> No sudo for me, though.
2022-12-17T19:25:11  <Luciano[m]> Oh, I think I'm using the wrong password.
2022-12-17T19:25:52  <Luciano[m]> Yup, silly me. It's working.