2020-04-19T00:51:07  -heroes-bot- PROBLEM: MySQL WSREP recv on galera3.infra.opensuse.org - CRIT wsrep_local_recv_queue_avg = 2.088299 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=galera3.infra.opensuse.org&service=MySQL%20WSREP%20recv
2020-04-19T01:48:54  -heroes-bot- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 147 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-04-19T01:58:55  -heroes-bot- RECOVERY: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=8 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-04-19T02:35:39  *** okurz_ is now known as okurz
2020-04-19T03:53:48  -heroes-bot- PROBLEM: Current Load on localhost - CRITICAL - load average: 20.35, 16.32, 9.02 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=localhost&service=Current%20Load
2020-04-19T04:42:15  -heroes-bot- PROBLEM: PSQL locks on mirrordb2.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 378 * total waiting locks: 185 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb2.infra.opensuse.org&service=PSQL%20locks
2020-04-19T04:42:54  -heroes-bot- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total waiting locks: 3 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-04-19T04:48:42  -heroes-bot- RECOVERY: Current Load on localhost - OK - load average: 2.21, 2.59, 2.95 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=localhost&service=Current%20Load
2020-04-19T04:52:16  -heroes-bot- RECOVERY: PSQL locks on mirrordb2.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=1 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb2.infra.opensuse.org&service=PSQL%20locks
2020-04-19T05:05:21  <lcp> cboltz: https://github.com/openSUSE/static.opensuse.org/pull/7 and https://gitlab.infra.opensuse.org/infra/salt/-/merge_requests/384
2020-04-19T10:33:52  <tuanpembual> cboltz I will create MR for new nginx config.
2020-04-19T10:34:03  * tuanpembual read salt doc
2020-04-19T11:02:00  <cboltz> in this case, you'll mostly need to understand nginx-formula - which shouldn't be too hard, just have a look at the existing pillar/role/web_progress.sls (and maybe some other pillar/role/web_* files)
2020-04-19T11:02:58  <tuanpembual> I see.
2020-04-19T11:05:46  <tuanpembual> does line order matter?
2020-04-19T11:06:34  <cboltz> in pillar/ - not really
2020-04-19T11:07:10  <cboltz> we often use alphabetic ordering, but if another sorting is better readable, feel free to use that
2020-04-19T11:49:40  <tuanpembual> MR created.
2020-04-19T12:15:20  <cboltz> looks good, and produces the expected config files :-)
2020-04-19T12:15:25  <cboltz> two minor nitpickings:
2020-04-19T12:15:36  <cboltz> - I'd prefer to keep the "HSTS" comment
2020-04-19T12:16:08  <cboltz> - please re-add the web_progress role in pillar/id/progress_infra_*
2020-04-19T12:18:04  <tuanpembual> client_max_body_size: 20m is enough ?
2020-04-19T12:18:29  <cboltz> I haven't heard complaints so far, so - yes ;-)
2020-04-19T12:19:25  <tuanpembual> on redmine itself, it use 20480, (on admin setting section)
2020-04-19T12:19:30  <tuanpembual> in KB
2020-04-19T12:20:58  <cboltz> 20480 (kB)/1024  = 20 MB, so that matches
2020-04-19T12:21:04  <tuanpembual> and at pillar/id should I remove thes comment.
2020-04-19T12:21:04  <tuanpembual> # currently progress.i.o.o has a hand-written nginx config that differs from the config in the web_progress role
2020-04-19T12:21:20  <cboltz> yes
2020-04-19T12:22:45  <tuanpembual> and should we remove web url https://progress-test.opensuse.org ?
2020-04-19T12:23:02  <cboltz> you'll also need to change   roles: []   to just   roles:   (the "[]" means empty, and that's no longer true/valid syntax after re-adding "- web_progress")
2020-04-19T12:24:12  <cboltz> depends - only remove it if you also want progress-test removed from the public DNS
2020-04-19T12:24:54  <tuanpembual> noted
2020-04-19T12:36:55  -heroes-bot- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 50 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-04-19T12:46:55  -heroes-bot- RECOVERY: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=16 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-04-19T15:58:11  -heroes-bot- PROBLEM: MySQL WSREP recv on galera1.infra.opensuse.org - CRIT wsrep_local_recv_queue_avg = 12.564907 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=galera1.infra.opensuse.org&service=MySQL%20WSREP%20recv
2020-04-19T16:09:03  <lcp> cboltz: https://gitlab.infra.opensuse.org/infra/salt/-/merge_requests/386 and https://github.com/openSUSE/static.opensuse.org/pull/8
2020-04-19T16:10:48  <lcp> it doesn't seem like the proxy pass is working though >:T
2020-04-19T16:12:33  <cboltz> maybe use a redirect instead?
2020-04-19T16:25:13  <lcp> cboltz: fixed it :P
2020-04-19T16:26:06  <lcp> as a sidenote, it seems the vm run out of space >:T
2020-04-19T16:26:46  <lcp> `/dev/vda2       10046440 9647180         0 100% /`
2020-04-19T16:26:52  <lcp> welp
2020-04-19T16:59:24  <lcp> cboltz: logs were taking up almost 4GB, rotating them helped
2020-04-19T17:06:55  <lcp> now everything works again
2020-04-19T17:07:14  <lcp> the conclusion is, we need more space and rotate logs often
2020-04-19T17:20:41  <malcolmlewis> Hi, any update on the likely solution for forum authentication? I see the SUSE Forums are going read-only on 04/23
2020-04-19T17:29:37  * lcp was gonna look at the authentification
2020-04-19T17:40:46  <lcp> righty-o, that's a confusing structure
2020-04-19T17:41:00  <lcp> malcolmlewis: what is SUSE doing with the forums on their side
2020-04-19T18:13:08  <malcolmlewis> lcp, no idea,  something new AFAIK
2020-04-19T18:14:38  <malcolmlewis> maybe they can import into the new platform?
2020-04-19T18:36:31  -heroes-bot- PROBLEM: HTTP progress on redmine.infra.opensuse.org - connect to address 192.168.47.29 and port 80: Connection refused ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=redmine.infra.opensuse.org&service=HTTP%20progress
2020-04-19T18:37:36  <lcp> malcolmlewis: I am mainly curious since they were using vb too afaik
2020-04-19T18:43:14  <malcolmlewis> lcp, yup, might find out more next week..
2020-04-19T21:59:43  <lcp> cboltz: one more thing https://gitlab.infra.opensuse.org/infra/salt/-/merge_requests/387
2020-04-19T22:02:08  <cboltz> shouldn't your addition be in the   {% if website == 'static' %}   section?
2020-04-19T22:02:16  <cboltz> I slightly doubt that for example shop.o.o needs it ;-)
2020-04-19T22:08:10  <lcp> ah, dang it
2020-04-19T22:10:22  <cboltz> better :-)
2020-04-19T22:25:00  <lcp> cboltz: I might also need to find where I made an error in here https://gitlab.infra.opensuse.org/infra/salt/-/jobs/8402 >:T
2020-04-19T22:25:12  <lcp> I am blind and the log is very unhelpful
2020-04-19T22:27:49  <cboltz>      [CRITICAL] Rendering SLS 'role.mailman3' failed, render error:
2020-04-19T22:28:06  <cboltz> that's the (pillar) file containing the error
2020-04-19T22:28:13  <cboltz>      found conflicting ID 'profile'
2020-04-19T22:28:19  <cboltz> and that's the problem ;-)
2020-04-19T22:29:13  <lcp> oh wow, I am blind after all
2020-04-19T22:29:48  <cboltz> or you are just unlucky - delete line 13 (!)
2020-04-19T22:30:48  <lcp> I did
2020-04-19T22:32:41  <cboltz> as a sidenote: as long as you work in your own branch, feel free to force-push a clean history ;-)
2020-04-19T22:35:30  <lcp> yup
2020-04-19T22:36:18  <lcp> that mailman stuff is still pretty much prototypes
2020-04-19T22:47:35  <cboltz> static.o.o config deployed
2020-04-19T22:54:31  <lcp> cboltz: makes me wonder if the config is correct now :/
2020-04-19T22:54:54  <lcp> I don't like cors tbh, it's hard to configure
2020-04-19T22:55:16  <lcp> (and the header might need https:// prefix that I didn't add)
2020-04-19T22:58:41  <lcp> much more worryingly, we will need cors setup on opensuse.org's .well-known >:T
2020-04-19T22:59:11  <cboltz> nice[tm]
2020-04-19T22:59:49  <lcp> I really don't like web server stuff tbh
2020-04-19T22:59:53  <cboltz> maybe we'll have to come back to Lars' idea of moving it to NBG and proxying /openid/ to the old server
2020-04-19T23:00:35  <lcp> well, if we get the fedora vms, maybe we will be able to have the openid endpoint locally too!
2020-04-19T23:01:03  <lcp> (we actually have packaging ready for this, as opposed to how it was going along for Leap)
2020-04-19T23:01:09  <cboltz> that would be even better :-)
2020-04-19T23:01:47  <cboltz> (nevertheless, we'll need to proxy /openid/ to $somewhere because I don't want to have that on the static.o.o VMs ;-)
2020-04-19T23:03:19  <lcp> it would be to sso.opensuse.org, yes
2020-04-19T23:08:21  <lcp> https://gitlab.infra.opensuse.org/infra/salt/-/merge_requests/388
2020-04-19T23:09:58  <lcp> cboltz: last think
2020-04-19T23:10:01  <lcp> I hope
2020-04-19T23:25:05  <cboltz> I'll check it tomorrow ;-) - good night!