2020-04-11T00:38:00 -heroes-bot- PROBLEM: MySQL WSREP recv on galera3.infra.opensuse.org - CRIT wsrep_local_recv_queue_avg = 1.591697 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=galera3.infra.opensuse.org&service=MySQL%20WSREP%20recv 2020-04-11T02:32:06 -heroes-bot- PROBLEM: NRPE on olaf.infra.opensuse.org - CHECK_NRPE: Error - Could not connect to 192.168.47.17. Check system logs on 192.168.47.17 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=olaf.infra.opensuse.org&service=NRPE 2020-04-11T02:42:05 -heroes-bot- RECOVERY: NRPE on olaf.infra.opensuse.org - NRPE v3.2.1 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=olaf.infra.opensuse.org&service=NRPE 2020-04-11T02:45:13 *** okurz_ is now known as okurz 2020-04-11T04:42:10 -heroes-bot- PROBLEM: PSQL locks on mirrordb2.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 238 * total waiting locks: 115 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb2.infra.opensuse.org&service=PSQL%20locks 2020-04-11T04:44:55 -heroes-bot- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total waiting locks: 4 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks 2020-04-11T04:52:09 -heroes-bot- RECOVERY: PSQL locks on mirrordb2.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=1 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb2.infra.opensuse.org&service=PSQL%20locks 2020-04-11T04:54:55 -heroes-bot- RECOVERY: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=4 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks 2020-04-11T08:58:55 -heroes-bot- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 50 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks 2020-04-11T09:08:54 -heroes-bot- RECOVERY: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=46 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks 2020-04-11T09:22:55 -heroes-bot- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 57 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks 2020-04-11T09:39:34 cboltz: well, there is another PR waiting :P 2020-04-11T09:58:36 https://gitlab.infra.opensuse.org/infra/salt/-/merge_requests/367 2020-04-11T13:16:25 -heroes-bot- PROBLEM: NRPE on svn.infra.opensuse.org - connect to address 192.168.47.25 port 5666: No route to host ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=svn.infra.opensuse.org&service=NRPE 2020-04-11T17:43:09 cboltz: now that this MR is merged, maybe I could get a haproxy setup 😛 2020-04-11T17:43:41 I can easily do matrix.o.o:443 2020-04-11T17:44:01 sure, that's enough to have a non-federated instance 2020-04-11T17:44:15 we can worry about 8448 later 2020-04-11T17:44:36 also remember there is dimension.o.o and chat.o.o in nginx there 2020-04-11T17:44:44 ok (8448 needs more work and firewall changes) 2020-04-11T17:46:02 to make sure I get everything right, can you give me a list of $subdomain.o.o -> matrix.i.o.o:$port (here or in https://progress.opensuse.org/issues/63463 )? 2020-04-11T17:49:04 matrix.o.o -> 8008 2020-04-11T17:49:04 chat.o.o -> 80 2020-04-11T17:49:04 dimension.o.o -> 80 2020-04-11T17:49:50 yeah, all of the appservices are on their own port, but they only communicate with the homeserver internally (it's a shame there is no socket setup there) 2020-04-11T17:53:12 https://gitlab.infra.opensuse.org/infra/salt/-/merge_requests/368 2020-04-11T18:00:58 DNS and haproxy done (and documented in the ticket checklist) 2020-04-11T18:01:35 awesome 2020-04-11T18:01:45 just need that small fix merged (whoops) 2020-04-11T18:01:54 I think it's the only issue 2020-04-11T18:02:07 already set to automerge, just wait for the pipeline ;-) 2020-04-11T18:08:35 as a sidenote - the 503 page now loads all images over https (one "mixed content" page less) 2020-04-11T18:25:38 nice 2020-04-11T18:25:48 * lcp sent a long message: < https://matrix.org/_matrix/media/r0/download/matrix.org/GDYaliqTFwwNiUAhDTZoYlbQ > 2020-04-11T18:25:56 salt is having a rough day 2020-04-11T18:27:38 it doesn't like using the same state names in two files 2020-04-11T18:28:09 it isn't using the same state names in two files though 2020-04-11T18:28:26 note that a deploy job failed, so you might have used an outdated version - and the last merged MR is still running the pipeline and not deployed yet 2020-04-11T18:28:30 unless it counts in telegram as being in appservices: in pillar, but that shouldn't be the case 2020-04-11T18:32:49 MR 353 would speed up the pipeline *a lot* - maybe I should cherry-pick it into a new branch and open a new MR for it (I hate to force-push to other people's branch) 2020-04-11T18:35:39 cboltz: here you go https://gitlab.infra.opensuse.org/infra/salt/-/merge_requests/369 2020-04-11T18:35:41 that was the error 2020-04-11T18:37:12 whitespace fun? nice ;-) 2020-04-11T18:41:04 Can you please review https://gitlab.infra.opensuse.org/infra/salt/-/merge_requests/370 ? 2020-04-11T18:44:01 hm, do we not have old distros anymore? 2020-04-11T18:44:37 only on a few servers, but not enough to keep salt tests for them ;-) 2020-04-11T18:44:53 besides that, these old servers typically don't see any changes, especially not via salt 2020-04-11T18:45:20 right, it looks fine to me 2020-04-11T18:45:58 :-) 2020-04-11T18:46:06 set to automerge 2020-04-11T19:26:17 352 set to automerge - I'll leave you the fun of running highstate / checking if reality and salt agree once it's merged and deployed ;-) 2020-04-11T19:30:39 in the meantime, feel free to review https://gitlab.infra.opensuse.org/infra/salt/-/merge_requests/371 2020-04-11T19:38:01 cboltz: mailman3 will gonna have fun with this, since it has cron for every occasion :P 2020-04-11T19:38:34 as long as successful cron runs don't flood admin-auto, everything is fine ;-) 2020-04-11T19:39:34 I am starting to get worried about the highstate on matrix.i.o.o, it has been running for a very long time 2020-04-11T19:43:53 ssh login took quite long, and it has/had a high load, but it seems to cool down again 2020-04-11T19:45:15 if highstate didn't give you a useful result, check /var/log/salt/minion 2020-04-11T19:55:38 yeah, it didn't because npm was trying to do terminal stuff to a text log 2020-04-11T19:55:40 so it's just filled with garbage 2020-04-11T19:58:18 nice ;-) 2020-04-11T19:58:37 deploy_job works like russian roulette - it works on gitlab-runner1, but fails on gitlab-runner2 claiming sudo is not installed there 2020-04-11T19:59:05 (but from what I can tell, it _is_ installed) 2020-04-11T20:00:51 well, dimension just hangs it seems 2020-04-11T20:01:48 https://github.com/angular/angular-cli/issues/5775 or not, I just have to wait a very long time 2020-04-11T20:03:37 salt kills it because it's taking too long though, so that's not good 2020-04-11T20:06:01 https://docs.saltstack.com/en/latest/ref/states/all/salt.states.cmd.html - Ctrl-F "timeout" 2020-04-11T20:06:27 the default timeout seems to be "None", therefore I'm a bit surprised... 2020-04-11T20:06:33 let me test a highstate 2020-04-11T20:07:18 also note that the minion log says (from your highstate run) 2020-04-11T20:07:21 npm run-script build:web && npm run-script build:app`\nnpm ERR! Exit status 137\nnpm ERR! \nnpm ERR! Failed at the matrix-dimension@1.0.0 build script.\nnpm ERR! This is probably not a problem with npm. There is likely additional logging output above.\n\nnpm ERR! A complete log of this run can be found in:\nnpm ERR! /var/lib/matrix-synapse/.npm/_logs/2020-04-11T19_42_10_495Z-debug.log' 2020-04-11T20:10:56 yeah, I'm testing it manually if it works if the process isn't killed 2020-04-11T20:11:04 but also that would mean the highstates would take half an hour 2020-04-11T20:11:13 might need to move all of that stuff into rpm instead 2020-04-11T20:11:56 if the build really takes half an hour, that might be a good idea 2020-04-11T20:12:33 (note: I started a highstate some minutes ago - does that conflict with your manual run?) 2020-04-11T20:16:19 shouldn't 2020-04-11T20:19:10 did you see the zypper conflict? 2020-04-11T20:19:13 Problem: installed matrix-synapse-1.11.1-lp152.122.2.noarch obsoletes python3-matrix-synapse < 1.11.1-lp152.122.2 provided by python3-matrix-synapse-0.28.1-lp152.4.5.noarch 2020-04-11T20:38:10 I have a patch ready 2020-04-11T20:38:19 I just wanna find a good fix for dimension 2020-04-11T20:40:13 -heroes-bot- PROBLEM: HAProxy on provo-proxy1.infra.opensuse.org - HAPROXY CRITICAL - Active service debuginfod is DOWN on debuginfod proxy ! ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=provo-proxy1.infra.opensuse.org&service=HAProxy 2020-04-11T20:47:26 cboltz: https://gitlab.infra.opensuse.org/infra/salt/-/merge_requests/372 I disabled dimension until I make an rpm for it 2020-04-11T20:50:02 what's the reason behind "build: True"? 2020-04-11T20:50:38 it looks like you added this for all appservices, so having this flag doesn't change much ;-) 2020-04-11T20:51:09 it's False for webhook 2020-04-11T20:51:37 ah, right - I should clean my glasses ;-) 2020-04-11T20:53:01 another question - why ffmpeg-3? what's the problem with plain ffmpeg? 2020-04-11T20:53:07 (and why not ffmpeg-4?) 2020-04-11T20:54:42 not in leap 2020-04-11T20:54:49 and it can't install based on provides 2020-04-11T20:54:54 which is really annoying 2020-04-11T20:56:38 indeed 2020-04-11T20:58:03 anyway - questions answered, set to automerge 2020-04-11T21:04:27 cboltz: killing the dimension process at the right time allows to continue without breaking things, but you can't really codify that without fancy stuff >:T 2020-04-11T21:04:48 it doesn't work anyway without user token though 2020-04-11T21:08:07 cboltz: I might need with your specialty, apparmor, because it doesn't want to work with synapse 2020-04-11T21:08:22 should be solvable ;-) 2020-04-11T21:08:42 neither apparmor nor synapse service wanna start for that reason tho 2020-04-11T21:09:48 looks like an easy one 2020-04-11T21:09:56 the matrix-synapse profile lacks 2020-04-11T21:10:04 #include 2020-04-11T21:10:33 which defines some variables (like @{PROC}) which are used in the abstractions 2020-04-11T21:10:58 alright, I will test it after this highstate is over 2020-04-11T21:15:32 cboltz: now it complains about the internals of turntables profile :P 2020-04-11T21:16:43 you misplaced the include - variable definitions have to be outside the profile 2020-04-11T21:19:00 anyone have success installing 389-ds server (leap 15.1)? I've installed/removed it several times because the ds commands are missing... any hints? 2020-04-11T21:23:08 cboltz: here you go https://gitlab.infra.opensuse.org/infra/salt/-/merge_requests/373 2020-04-11T21:23:41 much better :-) 2020-04-11T21:23:52 automerge 2020-04-11T21:42:17 cboltz: seems like synapse can't connect to the postgres server 2020-04-11T21:45:11 nice[tm] 2020-04-11T21:45:34 I could help with mysql, but I better keep my fingers off postgres ;-) 2020-04-11T21:45:49 sure 2020-04-11T21:49:00 but I do suspect it's the firewall 2020-04-11T21:49:36 LCP, cboltz: I've just stared at the code for package management in salt... I don't think there's enough eye bleach to undo the damage :o 2020-04-11T21:49:56 that code is awful 2020-04-11T21:51:17 cboltz: would you open port 5432? 2020-04-11T21:51:47 as a sidenote, for some reason nginx vhosts.d is unpopulated 2020-04-11T21:51:55 Eighth_Doctor: I rarely look at the salt internals, but - let's just say that the python code I prefer (and write) is more readable ;-) 2020-04-11T21:52:23 cboltz: I only looked because I thought I might be able to easily fix the lack of provides support 2020-04-11T21:52:34 but nope, it's all hardwired stupid 2020-04-11T21:56:13 lcp: for nginx, I'd guess you'll need to include profile.web.server.nginx 2020-04-11T21:56:46 that should be included 2020-04-11T21:56:56 oh wait 2020-04-11T21:58:52 cboltz: https://gitlab.infra.opensuse.org/infra/salt/-/merge_requests/374 2020-04-11T21:59:34 automerge ;-) 2020-04-11T22:07:31 cboltz: hm, what ports are open for matrix.i.o.o 2020-04-11T22:08:12 if you didn't setup a firewall, everything should be open in the internal network 2020-04-11T22:08:43 well then, pinging the postgresql server on the default port doesn't work 2020-04-11T22:08:58 "pinging" as in curl 2020-04-11T22:09:04 same thing really 2020-04-11T22:11:52 nmap postgresql.infra.opensuse.org (from my laptop) shows that 5432/tcp is open 2020-04-11T22:11:55 I'd be surprised (but that's just a guess) if it's available via VPN, but not from other VMs 2020-04-11T22:13:50 but helios uses it 2020-04-11T22:13:54 so that can't be it 2020-04-11T22:14:08 unless it has some wild config 2020-04-11T22:16:56 maybe the files in salt/profile/postgresql/ can answer your question about the config ;-) 2020-04-11T22:24:46 not at all, since that doesn't seem related to the server, it seems like a setup for mirrordb only 2020-04-11T22:27:10 postgresql.infra.opensuse.org points to anna/elsa, and pgbouncer is running there 2020-04-11T22:27:28 (and AFAIK redirects to the mirrordb* servers) 2020-04-11T22:27:39 hmm 2020-04-11T22:29:15 just checked - * = host=192.168.47.26 [...] - that IP is mirrordb1 2020-04-11T22:30:52 weirdly enough, pg can connect to mirrordb1 just fine 2020-04-11T22:31:27 (not just fine, since it's not permitted to actually connect from mirrordb1 side, but it can check this just fine) 2020-04-11T22:33:09 hmm, pgbouncer has a pg_hba.conf which seems to list several usernames and databases 2020-04-11T22:33:34 maybe you need an addition there, but I'd prefer if someone who understands postgres does that ;-) 2020-04-11T22:33:45 I will just leave a ticket to create the user and set up tables then 2020-04-11T22:40:23 ok 2020-04-11T22:41:06 FYI: the deploy_job for the nginx addition failed in the gitlab-runner roulette, I manually did the deploy_job on the saltmaster now 2020-04-11T22:46:46 cboltz: https://progress.opensuse.org/issues/65558 2020-04-11T22:47:19 added the identification stuff since I gonna need it too 2020-04-11T22:48:02 makes sense, getting started probably takes more time than actually creating the users and databases 2020-04-11T22:53:19 cboltz: hm, nginx should work but nothing happens on load of chat.opensuse.org (except for 503 that is) 2020-04-11T22:57:28 cboltz: but it sure does load on matrix.infra.opensuse.org 2020-04-11T22:57:36 something with haproxy by the looks of it 2020-04-11T22:57:48 yes, probably 2020-04-11T23:06:49 hmm, haproxy thinks matrix.i.o.o is down... 2020-04-11T23:07:09 Apr 11 23:06:15 anna haproxy[4949]: [WARNING] 101/230615 (4950) : Health check for server matrix/matrix failed, reason: Layer7 wrong status, code: 405, info: "Not Allowed", check duration: 1190ms, status: 0/2 DOWN. 2020-04-11T23:08:42 I disabled the check 2020-04-11T23:08:45 option httpchk OPTIONS / HTTP/1.1\r\nHOST:\ chat.opensuse.org 2020-04-11T23:08:49 and now it works 2020-04-11T23:09:07 but that's not something I'd call a permanent solution ;-) 2020-04-11T23:10:16 hm, what does it base that on? 2020-04-11T23:10:47 manual version: https://paste.opensuse.org/76404833 2020-04-11T23:11:47 oh, huh, gotta fix that 2020-04-11T23:13:59 you might also want to use a separate file for the dimension.o.o vhost 2020-04-11T23:14:06 (not technically needed, but less confusing) 2020-04-11T23:15:53 yep, you are right 2020-04-11T23:24:32 cboltz: actually, dimension script might be failing because of the lack of the user token 2020-04-11T23:24:46 gotta wait for the homeserver to be up then 2020-04-11T23:34:15 good night! 2020-04-11T23:37:22 night