2025-07-03T14:06:35  <knurpht> For me forums are down, chat.o.o has intermittent connection issues. Is that known?
2025-07-03T14:13:36  <knurpht> forums seem back, chat.o.o is the same.
2025-07-03T14:16:57  <knurpht> Scratch that, forums down again
2025-07-03T14:20:01  <acidsys> right the monitor complains about various backend servers flapping similar as the other day
2025-07-03T14:27:20  <acidsys> in VPN I can partially work but partially things hang or return packet loss
2025-07-03T14:52:53  <acidsys> from fw and bastion I can reach some VMs feelingly consistently but some others only respond sporadically (for example hel{1,2}). I tested other VMs on the same hypervisors and in the same VLANs and the behavior is not consistent. checking on hel1 (as an example affected machine) does not show much interesting. inspecting network load I find lots of traffic to+from lnt (http from rproxy and
2025-07-03T14:52:55  <acidsys> postgres to hel) but not sure that is of relevance yet
2025-07-03T14:55:46  <acidsys> following the chain fw -> rproxy -> lnt -> rproxy -> database I find lots of "__vm_enough_memory: pid: 21424, comm: postgres, not enough memory for the allocation" on mirrordb1
2025-07-03T14:55:57  <acidsys> *mirrordb2
2025-07-03T14:57:40  <acidsys> but there are also backends flapping on atlas which don't have any dependency hel/postgres
2025-07-03T15:11:32  <acidsys>  I switched firewalls/routers now and will watch if that has any insightful effect (as a first effect it helped with making all smooth again)
2025-07-03T15:12:07  <acidsys> which reminds of poo#168487, but in that incident there was no v6 routing at all whereas here it is just inconsistent
2025-07-03T16:17:58  <matrix-o-o> <hendersj> That OOM-type issue with the database seems like it would be a reasonable cause for what I noticed the other day as well.  Is that db used for mediawiki and anything matrix-related as well?
2025-07-03T17:02:52  <acidsys> as said it also affects services not using postgresql
2025-07-03T18:09:37  <matrix-o-o> <bmwiedemann> acidsys: Richard Rahl cboltz I thought, we have heroes meeting today.
2025-07-03T18:09:42  <matrix-o-o> <Yoshio 🇺🇦 Sato> It's quite lonely here in the Heroes meeting...
2025-07-03T18:09:42  <matrix-o-o> Could make us company guys?
2025-07-03T18:18:32  <matrix-o-o> <hendersj> acidsys: Ah, I was thinking it was two separate concurrent issues.
2025-07-03T18:21:02  <acidsys> oh, bmwiedemann Yohi are you still thre
2025-07-03T18:21:22  <matrix-o-o> <Yoshio 🇺🇦 Sato> acidsys: We are
2025-07-03T18:57:30  *** teepee_ is now known as teepee