2023-12-01T00:04:18 draeath, always check https://status.opensuse.org/ 2023-12-01T00:05:26 Ah, I see! If I continue to experience the problem when that's all Operational, would this be a valid place to talk about it? 2023-12-01T00:08:33 draeath, yes 2023-12-01T08:31:44 hi draeath, thanks for reporting, repaired it now 2023-12-01T09:05:53 Hi! I cannot login on wiki atm. Is it a known issue? 2023-12-01T09:33:23 guillaume_g: yes, the login servers moved to a new datacenter and something does not work yet 2023-12-01T09:34:12 bmwiedemann: ok, thanks for the info 2023-12-01T10:01:39 acidsys: Thanks; this seems to have worked (at least I observed an immediate effect on 'osc collab --project GNOME:Next todo' - it has information about what my next tasks should be again 2023-12-01T10:37:54 DimStar: great news :) 2023-12-01T10:38:40 the login for the wikis should be fixed 2023-12-01T10:39:48 are you saying you fixed it ? 2023-12-01T10:42:05 yes? 2023-12-01T10:42:18 ok, thank you 2023-12-01T10:42:37 can someone test? 2023-12-01T10:42:41 works for me 2023-12-01T11:40:57 DimStar: do you know if Ludwig still maintains rpmlint.o.o? it's not updating since the migration almost a month ago and I have not heard from anyone to check what it needs .. 2023-12-01T11:45:59 I think that server was originally setup by mliska, wasn't iot? 2023-12-01T11:46:19 rpmlint.o.o is defintiively a nice-to-have 2023-12-01T11:46:46 I'm not sure about the history, it just says lnuessel in https://github.com/openSUSE/heroes-salt/blob/production/pillar/id/rpmlint_infra_opensuse_org.sls 2023-12-01T11:47:37 then this should probably still be true. I'd assume this machine needs access to OBS to read the rpmlint logs 2023-12-01T11:55:11 okay good to know .. then I'll check if I find some obs related traffic there otherwise try to find Ludwig 2023-12-01T11:55:13 thanks! 2023-12-01T13:02:55 *** teepee_ is now known as teepee 2023-12-01T13:07:53 DimStar: FYI .. https://progress.opensuse.org/issues/151879 2023-12-01T13:08:43 DimStar: I was not aware of rpmlint.o.o It looks good but misses Factory:ARM ;) 2023-12-01T13:09:53 DimStar: Where is it maintained? To make a PR 2023-12-01T13:10:41 * acidsys hopes it is maintained at all 2023-12-01T13:10:54 I found answer to my last question as link at the bottom of the web page 2023-12-01T13:11:19 acidsys: last update: 26 April 2022 2023-12-01T13:12:56 but *.conf file is not up-to-date there 2023-12-01T13:31:48 acidsys: ouch.. that's worse than I thought.. but looking at the core repo, the whole thing was written by kraih; maybe he is simply not aware of the issue... will CC him on the progress ticket 2023-12-01T13:41:35 DimStar: known issue i'm afraid, seems IT forgot to update firewall settings when they moved the vm 2023-12-01T13:42:05 now the vm can't access obs anymore for updates, i have an SD ticket open 2023-12-01T13:45:24 acidsys: so seems like there WAS somebody reporting it needing attention by IT? Can you two work it out together? 2023-12-01T13:47:06 someone reported the problem to me like 2 weeks ago, i had no idea who moved the vm, so i just opened an SD ticket, which has been left untouched since 2023-12-01T13:48:51 acidsys: once the machine can access obs again it will just work 2023-12-01T13:55:10 kraih: https://github.com/openSUSE/build-check-statistics/blob/ec7917818c9c5b9ebca84bdd6e0995cc9830c728/build_check_statistics.conf#L2 2023-12-01T13:55:26 this might be another issue; I think the /public route is supposed to die 2023-12-01T13:56:00 also on OBS? 2023-12-01T13:56:23 ah, was that only an IBS topic? That's possible 2023-12-01T13:57:24 i've been scrambling to fix lots of IBS ssh auth issues over the past few weeks... hope this does not repeat for OBS :) 2023-12-01T14:02:39 honestly it's still a bit messy, since the auth mechanism is a bit unreliable and causes random auth failures every now and then 2023-12-01T14:20:22 ah, I didn't realize you were here as well kraih 2023-12-01T14:20:53 i'm lurking everywhere :) 2023-12-01T14:23:14 apparently not at the right time, because I did ask here at the time it happened :P 2023-12-01T14:23:16 2023-11-09 18:25:24 acidsys shutting rpmlint.i.o.o now 2023-12-01T14:23:18 2023-11-09 18:38:47 acidsys shutting obsreview.i.o.o now 2023-12-01T14:23:20 2023-11-09 19:04:55 acidsys rpmlint.o.o is back online but returns a 503 from the local webserver, who can check? 2023-12-01T14:23:30 anyways glad it's sorted now 2023-12-01T14:25:43 in my defense, i was sick that week with a really annoying stomach bug 2023-12-01T14:30:52 I'll accept that excuse :-) 2023-12-01T14:44:26 you're too kind 2023-12-01T15:14:42 :p 2023-12-01T18:01:44 *** teepee_ is now known as teepee 2023-12-01T18:20:26 so I've finally been able to log into the pagure VM though my connectivity is super slow 2023-12-01T18:20:40 it looks like nothing is actually reaching the pagure box from the outside 2023-12-01T18:20:47 and nothing seems broken inside of the VM itself 2023-12-01T18:21:07 so can someone help with making code.opensuse.org accessible again? 2023-12-01T18:26:32 Son_Goku: our proxy cannot reach your VM on port 80 2023-12-01T18:26:38 it seems you have a firewall installed which is blocking traffic 2023-12-01T18:26:41 [840375.219023] FINAL_REJECT: IN=os-code OUT= MAC=52:54:00:c9:d1:88:d2:e1:4b:98:46:1f:86:dd SRC=2a07:de40:b27e:1204:0000:0000:0000:0011 DST=2a07:de40:b27e:1206:0000:0000:0000:000a LEN=80 TC=0 HOPLIMIT=63 FLOWLBL=194634 PROTO=TCP SPT=36762 DPT=80 WINDOW=64800 RES=0x00 SYN URGP=0 2023-12-01T18:27:31 (I enabled LogDenied=all in /etc/firewalld/firewalld.conf on pagure01) 2023-12-01T18:27:34 acidsys: I added http and https to firewalld 2023-12-01T18:27:52 yay it works again 2023-12-01T18:28:02 how did you do that? 2023-12-01T18:28:14 firewall-cmd --add-service=http 2023-12-01T18:28:16 firewall-cmd --add-service=https 2023-12-01T18:28:22 firewall-cmd --runtime-to-permanent 2023-12-01T18:28:50 interesting, that route I would assume should work without a reload 2023-12-01T18:29:01 ok so I will disable logging again 2023-12-01T18:31:12 firewalld is nice in that you can apply rules at runtime without reload, and if it all works, you can save it 2023-12-01T18:31:14 or you can reset it 2023-12-01T18:31:17 if it doesn't work 2023-12-01T18:32:17 I know .. I just always dumbly ask because some people don't know that it's transient unless you add --permanent (or do runtime-to-permanent afterwards) and then have a machine that breaks the next reboot :P 2023-12-01T18:33:20 now there is still the problem I reported about comments not working 2023-12-01T18:33:23 the more interesting question is - why were http and https NOT allowed? And how did that work before? ;-) 2023-12-01T18:33:30 "The page isn’t redirecting properly" if you try to submit anything 2023-12-01T18:35:49 my suspicion is that I just did something temporary to make it work after the migration but then realized it does not fully work and made the ticket instead of continuing 2023-12-01T18:36:27 it already had problems before 2023-12-01T18:37:52 acidsys: I almost never use --permanent because that doesn't apply to the runtime :) 2023-12-01T18:38:03 it's a very weird flag and it's one of my quibbles with how firewall-cmd works 2023-12-01T18:38:30 btw ssh is not reachable from the proxy now 2023-12-01T18:39:04 .oO( zypper in nftables-service ; nft list ruleset > /etc/nftables.conf ; zypper rm firewalld ; systemctl enable --now nftables )o 2023-12-01T18:39:41 for simple host firewalls I don't mind firewalld 2023-12-01T18:39:53 it's also easy to manage with salt 2023-12-01T18:40:05 just for routing and perimeter firewalling I would not use it 2023-12-01T18:40:26 nftables.conf is easy too. simple file.managed :p _runs_ 2023-12-01T18:40:35 you are boring 2023-12-01T18:40:43 of course! 2023-12-01T18:40:47 >_> 2023-12-01T18:40:54 boring is good =P 2023-12-01T18:41:28 isn't there this weird community of people using the ufw package on openSUSE 2023-12-01T18:42:19 btw darix do you have a snippet for ssh health check in haproxy 2023-12-01T18:42:57 I only found this https://jakec007.github.io/2021-03-09-load-balancing-ssh-with-haproxy/, they check for rstring 2023-12-01T18:43:07 acidsys: there's a weird community of people running all sorts of silly stuff on openSUSE =P 2023-12-01T18:43:42 well true :p .. I guess the rule is, if it exists, it runs on openSUSE 2023-12-01T18:43:47 acidsys: looks good enough to me 2023-12-01T18:43:52 darix: ok thanks 2023-12-01T18:44:41 Son_Goku: so after I re-enabled firewalld logging and reloaded ssh from the proxy works as well 2023-12-01T18:44:51 does that mean I always need to temporarily enable logging to make things work? :p 2023-12-01T18:45:45 lol 2023-12-01T18:45:59 I did check to make sure ssh was enabled in the firewall 2023-12-01T18:46:04 I don't enjoy being locked out either :D 2023-12-01T18:46:45 heh 2023-12-01T18:46:52 well maybe before you leave today let's do one reboot of pagure01 and see if it all still sticks together 2023-12-01T18:47:02 that's a good idea 2023-12-01T18:47:15 generally though I'm concerned about how difficult it is for me to login through the jump host 2023-12-01T18:47:21 it timed out twice before it finally connected 2023-12-01T18:47:39 and the lag is significant 2023-12-01T18:48:27 I'm not sure the jump host is at fault. it's the same as any other VM there 2023-12-01T18:48:41 the last time you showed me debug output you had trouble with DNS 2023-12-01T18:49:57 yeah 2023-12-01T18:50:03 DNS seems to be okay now 2023-12-01T18:50:06 it's just really slow 2023-12-01T18:50:52 show ping for thor.infra.opensuse.org and ping for odin.opensuse.org 2023-12-01T18:51:38 better. use mtr. to see where it gets bad. 2023-12-01T18:54:54 btw proxy again cannot connect to ssh on pagure01 2023-12-01T19:09:23 seems it always works after `firewall-cmd --reload` for a minute or two 2023-12-01T19:14:56 well 2023-12-01T19:15:17 just a stupid idea ... the health check triggers some "oh you connected to me too often in the last few minutes" rules and then gets blocked? 2023-12-01T19:15:53 good idea 2023-12-01T19:15:55 `- Banned IP list:172.16.130.11 2a07:de40:b27e:1204::11 2023-12-01T19:16:07 so now the Q is how can we pass the real IP through 2023-12-01T19:16:17 SSH PROXY protocol? :P 2023-12-01T19:16:30 1. it wont help you for the health check. 2023-12-01T19:16:53 fine I can whitelist atlas in fail2ban 2023-12-01T19:18:23 I was more thinking if it will also be a problem for internet users 2023-12-01T19:18:46 that one person doesn't ban everyone 2023-12-01T19:21:52 https://serverfault.com/questions/620703/sshd-real-ip-behind-haproxy 2023-12-01T19:25:17 https://www.haproxy.com/blog/route-ssh-connections-with-haproxy 2023-12-01T19:26:17 ah bleh that last thing is we abuse openssl to tunnel it through 2023-12-01T19:26:45 so short answer no plain haproxy solution. 2023-12-01T19:26:48 but tproxy works 2023-12-01T19:30:11 mmproxy looks interesting 2023-12-01T19:30:19 but a bit unmaintained 2023-12-01T19:30:55 i have it packaged somewhere 2023-12-01T19:31:00 well I just noticed that as well 2023-12-01T19:31:05 in :playground 2023-12-01T19:31:10 https://www.loadbalancer.org/blog/configure-haproxy-with-tproxy-kernel-for-full-transparent-proxy/ 2023-12-01T19:31:46 I think this sounds interesting for database ACLs as well 2023-12-01T19:32:20 the proxy stuff? 2023-12-01T19:32:21 yeah 2023-12-01T19:32:24 yep 2023-12-01T19:33:22 the go-mmproxy seems better maintained :) https://github.com/path-network/go-mmproxy and probably cheesy to package 2023-12-01T19:33:43 yeah but why if we have in kernel stuff already? 2023-12-01T19:33:49 do we 2023-12-01T19:34:15 I tried the haproxy source ... usesrc ... and it crashed miserably. then again I did not read the docs fully yet 2023-12-01T19:36:23 I think our spec file is missing USE_LINUX_TPROXY 2023-12-01T19:36:49 in the make options 2023-12-01T19:37:40 https://docs.haproxy.org/2.8/configuration.html#5.1-transparent 2023-12-01T19:40:12 proxy should be default 2023-12-01T19:40:46 /usr/sbin/haproxy -vvv 2023-12-01T19:40:54 Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION -QUIC -QUIC_OPENSSL_COMPAT +RT +SHM_OPEN -SLZ +SSL -STATIC_PCRE -STATIC_PCRE2... 2023-12-01T19:41:00 ... +SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL +ZLIB 2023-12-01T19:41:00 see has tproxy 2023-12-01T19:41:02 at least on 2.8 2023-12-01T19:41:54 oh cool 2023-12-01T19:42:00 then I guess I have to play some more with it 2023-12-01T19:43:32 can you check if 2.4 has it as well? 2023-12-01T19:43:40 although didnt we switch atlas to 2.8 anyway? 2023-12-01T19:43:48 i thought we did 2023-12-01T19:43:52 we temporarily did 2023-12-01T19:44:07 i remember us fixing all the haproxy cfg stuff to modern age:) 2023-12-01T19:44:10 until we found what we needed worked in 2.4 anyways 2023-12-01T19:44:15 pfff 2023-12-01T19:44:21 yes that's still good because all the warnings are gone 2023-12-01T19:44:23 well can you check with -vvv if tproxy is on there? 2023-12-01T19:44:42 btw i added on more feature in haproxy formula and tagged a release 2023-12-01T19:44:46 also ... did you see slack? 2023-12-01T19:45:04 seems atlas still have haproxy 2.8 dangling from our experiment 2023-12-01T19:45:21 I need to clean it up 2023-12-01T19:45:36 no not recently, sorry 2023-12-01T20:02:11 btw if the haproxy formula needs any more keywords handled for tproxy let me know. i will hack them into the template quickly 2023-12-01T20:02:49 * acidsys coughs in "extra" 2023-12-01T20:03:37 that's ugly 2023-12-01T20:03:45 implementing keywords is super easy 2023-12-01T20:03:55 I know 2023-12-01T20:04:15 if you use extra let me know for which keywords and will do those 2023-12-01T20:06:19 ok 2023-12-01T21:33:20 acidsys: JFYI mirrorcache oid login should work now 2023-12-01T21:33:51 oh cool, I didn't realize it wasn't working 2023-12-01T21:34:05 confirmed 2023-12-01T21:34:09 it had a working firewall rule to id.o.o but not to www.o.o 2023-12-01T21:34:21 and id.o.o says "you will find some informations at www.o.o" :D 2023-12-01T21:34:42 ah I remember now, it was broken on the main mirrorcache.o.o 2023-12-01T21:34:54 Robert fixed it 2023-12-01T21:34:55 id.o.o shouldn't do that :x 2023-12-01T21:34:58 cool 2023-12-01T21:35:08 curl -I id.o.o :D 2023-12-01T21:35:10 juuuuuuuuuuuust saying 2023-12-01T21:35:40 smells like a bug 2023-12-01T21:36:11 having xrds discovery send you to a proxy which then sends you back to where you got it from 2023-12-01T21:37:08 also wiki.openid.net has an invalid certificate lol 2023-12-01T22:20:56 *** teepee_ is now known as teepee