2023-01-04T00:27:57  *** teepee_ is now known as teepee
2023-01-04T03:42:06  <malcolmlewis> hellcp, forum has died again....
2023-01-04T04:08:51  <Luciano[m]> ☹️
2023-01-04T04:09:41  <bmwiedemann[m]> do you need help to kick it back to life?
2023-01-04T04:12:01  <Luciano[m]> I don't think anyone is trying to, at the moment?!
2023-01-04T04:14:24  <bmwiedemann[m]> 2023/01/04 03:35:28 [crit] 10139#10139: *32419646 pwrite() "/var/lib/nginx/proxy//2/62/0000124622" failed (28: No space left on device) while reading upstream, client: 64.130.121.90, server: forums.opensuse.org, request: "GET /letter_avatar_proxy/v4/letter/c/b2d939/45.png HTTP/1.1", upstream: "http://unix:/srv/www/vhosts/discourse/tmp/sockets/puma.sock:/le
2023-01-04T04:16:22  <bmwiedemann[m]> so the 9GB disk ran full. but logs are only 1GB
2023-01-04T04:16:46  <Luciano[m]> Uhoh! I see there's no profile for this VM in Icinga yet. The only bleeding about space there is with rsync.
2023-01-04T04:17:05  <Luciano[m]> is about&
2023-01-04T04:27:09  <malcolmlewis> bmwiedemann[m], many thanks :) needs more space...
2023-01-04T04:27:54  <Luciano[m]> Now I'm curious to know how much space the old forums was taking.
2023-01-04T04:27:54  <bmwiedemann[m]> old one had only one 10GB disk
2023-01-04T04:27:55  <bmwiedemann[m]> resized discord01 vda to 20GB. Just needs a short reboot
2023-01-04T04:27:55  <bmwiedemann[m]> and with that, forums are back in business for a while.
2023-01-04T04:28:16  <malcolmlewis> seems to have dies again
2023-01-04T04:28:21  <malcolmlewis> bmwiedemann[m], ^^
2023-01-04T04:28:37  <bmwiedemann[m]> I'm installing updates and it updated discourse-2.8.13+git2.fbd6c300d-lp154.821.5.noarch
2023-01-04T04:28:51  <malcolmlewis> ahh ok
2023-01-04T04:31:42  <bmwiedemann[m]> and after another reboot it is back again
2023-01-04T04:34:50  <Luciano[m]> Very nice! I'm taking this full disk issue as a sign of prosperity.
2023-01-04T04:35:33  <bmwiedemann[m]> prosperity? looking at a dictionary...
2023-01-04T04:36:29  <bmwiedemann[m]> you mean, we had enough nginx log messages to fill up the space?
2023-01-04T04:38:06  <Luciano[m]> Oh, nginx log? Ouch.
2023-01-04T04:38:44  <bmwiedemann[m]> du -s /var/log/nginx/
2023-01-04T04:38:44  <bmwiedemann[m]> 924140  /var/log/nginx/
2023-01-04T04:41:34  <Luciano[m]> OK, so those errors - and maybe something else - filled up the disk.
2023-01-04T04:42:08  <bmwiedemann[m]> There was a 40MB crate left in /tmp but nothing else
2023-01-04T04:43:42  <Luciano[m]> I wonder whether that's expected from Discourse, or something bad is going on.
2023-01-04T04:44:31  <Luciano[m]> I mean, Discourse on nginx, of course.
2023-01-04T04:44:50  <bmwiedemann[m]> I think, most of the trouble comes from the nginx/puma interaction today
2023-01-04T04:45:46  <bmwiedemann[m]> by writing error log messages, it filled up the disk further
2023-01-04T04:48:31  <Luciano[m]> Let's see if those two misbehave again.
2023-01-04T04:52:43  <bmwiedemann[m]> currently it fills around 300kB/minute that would be 400MB/d. logrotate should compress things well, so maybe 40MB/d are left and at some point it should delete old logs.
2023-01-04T04:53:50  <bmwiedemann[m]> at this rate, disk should not run full in the next 100 days
2023-01-04T04:57:17  <bmwiedemann[m]> looking at while sleep 1m ; do date +%s ; df / ; done - let's see how that looks when europeans wake up.
2023-01-04T05:02:34  <Luciano[m]> Hmm, yeah.
2023-01-04T07:17:27  <hellcp> a European woke up
2023-01-04T08:36:53  <acidsys> Luciano[m]: well "pretty" to me is standard ports, so 80 for HTTP and 443 for HTTPS
2023-01-04T08:37:29  <bmwiedemann[m]> discord01 nginx is listening on 80
2023-01-04T08:38:06  <bmwiedemann[m]> or is it about a different service?
2023-01-04T08:38:19  <acidsys> jenkins-agent
2023-01-04T10:14:46  <bmwiedemann[m]> Luciano: disk usage is rising by 450 KB/min now. Still not bad.
2023-01-04T14:11:54  *** teepee_ is now known as teepee
2023-01-04T18:42:35  <pjessen> I don't really get this push for fedocal - why are we pushing it instead of a proper calendar?
2023-01-04T19:57:52  <LCP0968[m]> well, we need more than one calendar, we do have more than one team that has meetings
2023-01-04T19:58:17  <LCP0968[m]> also what does proper calendar mean, and how is fedocal not one?
2023-01-04T21:22:34  *** teepee_ is now known as teepee
2023-01-04T23:46:11  <Luciano[m]> acidsys: Not jenkins-agent, actually. Jenkins agents are meant to be workers, so people won't need to interact with it.
2023-01-04T23:46:29  <Luciano[m]> s/it/them
2023-01-04T23:49:56  * Luciano[m] acknowledges discord01's disk usage rise!