2020-03-05T03:10:31 *** okurz_ is now known as okurz 2020-03-05T08:23:34 504 gateway time-out - might that be coming from haproxy ? 2020-03-05T08:41:01 504 gateway time-out - might that be coming from haproxy ? 2020-03-05T08:45:16 pjessen: where 2020-03-05T09:04:30 where can you see 504? ci.opensuse.org has a problem with Jenkins, i will jump on that as soon as I can finish the internal suse's maintenance window 2020-03-05T09:11:31 sorry guys, I was away. I get the 504 on forum.infra when trying to upgrade vbulletin. 2020-03-05T09:12:06 i'm runing a lot of database ops for the upgrade, but then I get stuck with this 504. 2020-03-05T09:12:55 I've looked at the logs an anna, but I didn't see anything interesting. I am wondering if it might be a long-running sql that causes it. 2020-03-05T09:14:24 it does report 50x, but it reports it from forum.infra and forums-nbg, so both without and behind the proxy, it must be something in the vm itself most likely 2020-03-05T09:15:15 where do you see the 504 ? 2020-03-05T09:16:14 hah, found it. thanks. 2020-03-05T09:17:10 fsockopen(): unable to connect to tcp://smtp2.provo.novell.com:26 (Connection timed out) 2020-03-05T09:17:31 weird. 2020-03-05T09:17:51 lcp: not sure if that is the 504. where do you see it? 2020-03-05T09:18:51 nah, the above is vbulletin trying to send an email. 2020-03-05T09:18:56 -heroes-bot- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 60 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks 2020-03-05T09:20:32 on the frontend 2020-03-05T09:20:57 that smtp thing should be easy enough to fix ;) 2020-03-05T09:23:19 lcp: yup, just looking through the database now. 2020-03-05T09:28:13 lcp: can you tell where the 504 is coming from? I am 99% certain it is the database (galera) 2020-03-05T09:28:48 I actually don't know 2020-03-05T09:30:00 hm, spitballing, but does the vm have access to the ports to connect to the database? 2020-03-05T09:30:42 (I can't really check considering I don't have the access to the server) 2020-03-05T09:30:55 lcp: sure, the database access works fine. I loaded up a few million records yesterday :-) 2020-03-05T09:33:40 hmm, i see a 30min timeout in haproxy.cfg. does anyone know if a timeout here would look like a 504 on the client end? 2020-03-05T09:38:56 -heroes-bot- RECOVERY: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=49 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks 2020-03-05T09:50:17 hmm, it's probably not haproxy. looks like maybe a timeout after 5mins. it's a long-running query, now at 500sec. 2020-03-05T09:52:55 -heroes-bot- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 50 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks 2020-03-05T11:06:23 Hey here! Is there someone who can give me access to the crtmgr.infra.opensuse.org. machine? 2020-03-05T11:06:34 Or provide me a copy of the config ? 2020-03-05T13:26:40 btw, ci.opensuse.org was not working because jdsn was upgrading it... it is working now 2020-03-05T13:46:20 klein: good to know, thanks 2020-03-05T13:47:03 pjessen: 504 might come from haproxy - but only if the backend is in http mode 2020-03-05T13:47:09 our galera cluster is in tcp mode 2020-03-05T14:13:42 kl_eisbaer: hmm, thanks. 2020-03-05T16:31:57 kl_eisbaer: I am making a lot of static sites for the project, trust me 😛 2020-03-05T16:32:24 some of them for years at this point 2020-03-05T16:40:28 *** ldevulder_ is now known as ldevulder 2020-03-05T17:04:08 -heroes-bot- PROBLEM: MySQL WSREP recv on galera1.infra.opensuse.org - CRIT wsrep_local_recv_queue_avg = 2.971752 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=galera1.infra.opensuse.org&service=MySQL%20WSREP%20recv 2020-03-05T17:34:16 lcp: :D 2020-03-05T17:35:33 lcp: FYI: your centos8.1 machine is currently installing ... 2020-03-05T17:35:49 excellent, can't wait 2020-03-05T17:41:54 lcp: machine is up - where can I find your SSH key ? 2020-03-05T17:42:17 on freeipa 2020-03-05T17:42:30 not on connect for some reason 2020-03-05T17:42:39 also on fedora accounts system >:D 2020-03-05T17:42:54 omw 2020-03-05T17:44:23 lcp: your key is on root@freeipa2.infra.opensuse.org 2020-03-05T17:45:43 excellent, thank you 2020-03-05T18:02:51 * lcp looks at gitlab pipeline 2020-03-05T18:02:57 still not working, eh? 2020-03-05T18:31:54 kl_eisbaer: I think I might try to configure it closer to how we do openSUSE vms with salt-minion and postfix relay, just because of convenience of it (and maybe we could reuse salt profile that way if I decide to write one for openSUSE version of freeipa when I get that done) ;) 2020-03-05T18:32:31 looking at our kiwi script, seems easy enough 2020-03-05T19:23:07 lcp: good idea 2020-03-05T19:32:20 * lcp uploaded an image: Screenshot from 2020-03-05 20-31-41.png (10KB) < https://matrix.org/_matrix/media/r0/download/matrix.org/fsTlaKwpjWAtozXmqsWLScrT > 2020-03-05T19:32:32 that's postfix done, they don't have sysconfig for postfix so it's quite a bit harder 2020-03-05T19:33:30 * cboltz tends to disagree with "harder" - the first thing I usually do is disabling sysconfig breakage for postfix 2020-03-05T19:35:19 I prefer setting 5 config options instead of ~30 2020-03-05T19:35:56 well, it probably depends on the config you want 2020-03-05T19:36:04 that's true 2020-03-05T19:36:13 for my own servers, I typically append ~40 lines to main.cf 2020-03-05T19:36:49 for the openSUSE infrastructure (where we basically only need to add "relayhost=..." with salt), things look different 2020-03-05T19:38:05 sendmail is so much better than mailx tho, I gotta say 2020-03-05T19:38:19 mailx always annoys me 2020-03-05T19:38:41 as long as you don't claim that sendmail (as a deamon) is easy to configure... ;-) 2020-03-05T19:39:02 as test client, I tend to use telnet $hostname 25 2020-03-05T19:39:29 sure 2020-03-05T19:40:02 methodology is just beyond my understanding at this point tbh ;) 2020-03-05T19:40:26 I used telnet once in my life, to configure a router 2020-03-05T19:40:37 as long as you don't want/need SMTP Auth, speaking SMTP is quite easy ;-) 2020-03-05T19:40:57 (and pop3 is even easier) 2020-03-05T19:41:36 I wonder what your reaction to jmap is ;) 2020-03-05T19:42:49 it even has a nice website https://jmap.io/ 2020-03-05T19:43:33 never looked at it before ;-) 2020-03-05T19:44:01 using json is nice, but typing json manually probably won't become my favorite method 2020-03-05T19:45:44 yeah, hopefully the usecase remains as an easy to use api for mobile/desktop and not usage in terminal ;) 2020-03-05T19:46:08 yes, probably 2020-03-05T19:46:22 nevertheless, simple protocols make manual testing much easier 2020-03-05T19:51:49 yup 2020-03-05T20:05:59 lcp: FYI freeipa2 is using sssd now (login via $user), chrony, ldap, grub (serial) and rsyslog are configured as well as auto-update (security only for now). Monitoring is missing for now (I guess this is ok, as you might want to play a bit.... ;-) 2020-03-05T20:06:32 I have to go now - just wanted to let you now that you are now "the one and only hero" on the machine ;-) 2020-03-05T20:06:39 have a lot of fun! 2020-03-05T20:15:01 thanks 2020-03-05T20:30:42 alright, seems like everything (?) is configured from the salt pov 2020-03-05T20:31:36 except for minion key that is 2020-03-05T20:37:04 give me a minute to accept the minion ;-) 2020-03-05T20:37:20 (and note that I have no idea how much of our salt code will work on fedora) 2020-03-05T20:38:13 key accepted, have fun ;-) 2020-03-05T20:39:13 it's centos, and probably none of it, but it is kind of fun trying to break it ;) 2020-03-05T20:40:03 * lcp sent a long message: < https://matrix.org/_matrix/media/r0/download/matrix.org/ymNepIwqSdGuwfFmXhsmeuRr > 2020-03-05T20:40:06 it's not that bad 2020-03-05T20:40:55 yeah, we don't have the "basic infrastructure" for centos in salt - shouldn't be too hard to add 2020-03-05T20:41:05 yup 2020-03-05T20:41:20 most things shouldn't break something (well, adding and later using openSUSE repos might be an exception, but most of the repos are defined in osfullname.* or osmajorrelease.*) 2020-03-05T20:41:28 first tho, broken pipeline in gitlab is a huge roadblock 2020-03-05T20:41:38 yes, indeed 2020-03-05T20:42:32 what do 2020-03-05T20:47:00 hope that someone (maybe Lars?) fixes it ;-) (unfortunately I don't know much about gitlab and have no idea how to fix it) 2020-03-05T20:47:28 until then, I should be able to merge things manually - just ping me if you need that ;-) 2020-03-05T21:12:58 thanks, I will 2020-03-05T21:23:39 funnily enough, 2/3 of the files for release stuff for CentOS will remain empty because there is little point in actually adding anything to them 2020-03-05T21:24:07 I only really need to add repos 2020-03-05T21:25:05 (that's also kinda pointless because you wouldn't even be able to use salt for provisioning without those repos, because salt doesn't exist in the main repo) 2020-03-05T21:33:25 ;-) 2020-03-05T21:33:38 maybe I could leave all of them empty >:D 2020-03-05T21:33:56 if it works with empty files, why not? ;-) 2020-03-05T21:34:29 it should 2020-03-05T21:38:04 oh god, Gitlab hates spaces in filenames 2020-03-05T21:38:38 you might want to replace it with an underscore in salt ;-) 2020-03-05T21:39:20 (I guess we are talking about osfullname?) 2020-03-05T21:40:49 yes 2020-03-05T21:42:22 basically change pillar/top.sls to adjust osfullname in a similar way it adjusts id 2020-03-05T21:43:04 (maybe already in the {% set part because osfullname is used more than once) 2020-03-05T21:44:57 nah, it's fine 2020-03-05T21:52:24 cboltz: there you go https://gitlab.infra.opensuse.org/infra/salt/-/merge_requests/354 >:D 2020-03-05T21:52:33 it should work 2020-03-05T21:56:34 yeah, looks good 2020-03-05T21:57:32 give me a minute to try to merge it ;-) 2020-03-05T22:03:32 in theory it should work now 2020-03-05T22:18:27 welp, some weird issue with chrony 2020-03-05T22:19:19 sounds like the pillar additions worked ;-) 2020-03-05T22:22:27 they did 😁 2020-03-05T22:30:31 I think I got it 2020-03-05T22:32:31 it's can't do chrony.config from salt/profile/regional.sls 2020-03-05T22:34:19 sounds like you should look at the chrony formula 2020-03-05T22:34:46 I am, and I see the issue 2020-03-05T22:35:02 config is trying to load /map.jinja, which is in a level lower directory 2020-03-05T22:35:28 so you get this: Rendering SLS 'production:chrony.config' failed: Jinja error: chrony/config/map.jinja 2020-03-05T22:35:39 when the correct dir is chrony/map.jinja 2020-03-05T22:36:24 interestingly I've never seen such a problem in openSUSE VMs - maybe centos has a different salt version with a bug too much? 2020-03-05T22:36:25 https://github.com/saltstack-formulas/chrony-formula/blob/master/chrony/config/file.sls 2020-03-05T22:37:00 most likely, that's salt's own repo, so it's basically the most current stable 2020-03-05T22:38:24 we use gitlab@gitlab.opensuse.org:saltstack-formulas/chrony-formula.git which might be older 2020-03-05T22:38:38 but I still wonder why it just works on openSUSE 2020-03-05T22:40:37 cboltz: found it https://github.com/saltstack/salt/issues/56119 2020-03-05T22:41:26 and upstream has that fixed https://github.com/saltstack-formulas/chrony-formula/blob/master/chrony/config/file.sls#L7 2020-03-05T22:41:34 so yeah, it's just a wrong variable 2020-03-05T22:44:04 reading the issue, it looks more like an unintended breaking change that will probably be reverted 2020-03-05T22:47:12 I mean, it will be reverted, and changed to the current behavior in the future after proper deprecation 2020-03-05T22:47:18 but yeah 2020-03-05T22:49:28 we'll see if it really gets deprecated ;-) 2020-03-05T22:50:09 updating the formula is an obvious option, but I'll first have to check what was changed upstream 2020-03-05T22:50:14 hopefully, who needs two vars pointing to the same place 2020-03-05T22:50:38 and I'm afraid I won't do this today while my eyes are only half open ;-) 2020-03-05T22:51:46 if keeping that superfluous var helps to not break salt code in the wild, I'd call that a good argument to keep it 2020-03-05T22:52:45 then maybe using tlppath for the thing the slspath turned this update would be a better idea ;) 2020-03-05T22:53:03 if it's undocumented, you might as well 2020-03-05T22:53:32 that breaks everything upstream, but its their own damn fault 2020-03-05T22:54:31 yeah, you can have quite some fun with changing undocumented stuff ;-)