2020-07-12T02:14:15  *** okurz_ is now known as okurz
2020-07-12T12:51:23  <cboltz> kl_eisbaer: our monitoring must be drunk ;-)
2020-07-12T12:51:31  <cboltz>     Updates CRITICAL : discontinued OS Release openSUSE 15.2
2020-07-12T12:53:03  <lcp79> cboltz: https://cdn.discordapp.com/attachments/366989996101730304/731855201774076004/unknown.png and https://cdn.discordapp.com/attachments/366989996101730304/731855314672156782/unknown.png show that the autoindex skin is more or less ready ;)
2020-07-12T12:54:09  <lcp79> I reported https://progress.opensuse.org/issues/68863 to fix the issues that arise when using it, either from my side or server side (although I can't fix everything from my side)
2020-07-12T12:57:06  <cboltz> lcp79: did you run highstate on jekyll.i.o.o for the monitor.o.o main page?
2020-07-12T12:57:25  <cboltz> I just tried to switch to it (the haproxy way), and got the news.o.o content
2020-07-12T12:58:49  <lcp79> I didn't
2020-07-12T12:59:02  <lcp79> news is the default server, so that's why
2020-07-12T12:59:35  <cboltz> guessed so ;-)
2020-07-12T13:00:06  <cboltz> please tell me when the monitor.o.o main page is ready so that I can remove a   #   in the haproxy config
2020-07-12T13:00:55  <cboltz> for download.o.o - you mentioned in the ticket that you need to adjust jekyll-theme to not use https, but I'm not sure if that is really needed
2020-07-12T13:01:17  <lcp79> it is, firefox/chrome will refuse to load https content on http page
2020-07-12T13:01:28  <lcp79> and vice versa for that matter
2020-07-12T13:02:20  <cboltz> really? that makes things more interesting
2020-07-12T13:02:49  <lcp79> kl_eisbaer: after 15.2 update on some servers highstate doesn't work because of faulty salt in main repos, it should be switched to openSUSE:infrastructure version
2020-07-12T13:02:50  <cboltz> I hope redirects to https aren't a problem?
2020-07-12T13:03:03  <lcp79> they sure are
2020-07-12T13:03:19  <lcp79> that's why I mentioned to have static accessible from http too
2020-07-12T13:03:44  <kl_eisbaer> cboltz:  "Updates CRITICAL : discontinued OS Release openSUSE 15.2" - it's not drunk, just a "bug" in the official package (too old version)
2020-07-12T13:04:27  <cboltz> "drunk" sounds better ;-)
2020-07-12T13:04:42  <kl_eisbaer> about the highstate on server: looks indeed like a salt problem.
2020-07-12T13:05:13  <kl_eisbaer> cboltz: I found 2 other bugs with monitoring packages in 15.2 already. Currently just collecting them to make one big update ;-)
2020-07-12T13:05:38  <cboltz> nice ;-)
2020-07-12T13:06:31  <kl_eisbaer> lcp79: thanks, btw., for your work on the download-o-o stuff. IMHO the https://build.opensuse.org/package/show/openSUSE:infrastructure/mirrorbrain-icons package should contain all available icons. If not, it's a bug ;-)
2020-07-12T13:07:15  <kl_eisbaer> FYI: ...and for mirrors.opensuse.org, I plan to extend https://build.opensuse.org/package/show/openSUSE:infrastructure/mirrorbrain-webpages with the template file as well
2020-07-12T13:07:24  <kl_eisbaer> so this could also end up in a git repo, if wanted.
2020-07-12T13:09:58  <cboltz> on which servers do you see the highstate failure? (one or two examples are good enough)
2020-07-12T13:10:34  <lcp79> jekyll a moment ago, I fixed it with `zypper in -f --from openSUSE:infrastructure salt*`
2020-07-12T13:10:49  <kl_eisbaer> The following "single page" pages might also worth an overhaul: https://kubic.opensuse.org/ https://monitor.opensuse.org/ https://osc-collab.opensuse.org/ https://rpmlint.opensuse.org/ https://yast.opensuse.org/ https://gcc.opensuse.org/ ;-)
2020-07-12T13:10:59  <kl_eisbaer> ah, and I forgot https://debuginfod.opensuse.org
2020-07-12T13:11:02  <lcp79> kubic is kinda on my radar
2020-07-12T13:11:12  <lcp79> yast is good imo
2020-07-12T13:11:24  <lcp79> and monitor is done >:D
2020-07-12T13:11:49  <lcp79> it is on jekyll.infra.opensuse.org already
2020-07-12T13:12:14  <lcp79> I made the current design of kubic, so that's why it's on my radar
2020-07-12T13:12:16  <kl_eisbaer> oh!? thanks!
2020-07-12T13:12:49  <lcp79> where is gcc hosted btw?
2020-07-12T13:12:58  <kl_eisbaer> on gcc.infra.opensuse.org
2020-07-12T13:13:36  <lcp79> I meant like git repo or something
2020-07-12T13:14:58  <kl_eisbaer> uh: that's something for the main operators
2020-07-12T13:15:41  <lcp79> aw, I had no idea setloc.js makes the links in breadcrumb happen, I would have rewrote it to better work with the theme
2020-07-12T13:15:50  <kl_eisbaer> => mmarek@o.o
2020-07-12T13:15:52  <lcp79> I kinda wish it was done on the server side too
2020-07-12T13:16:07  <lcp79> alright, I will contact them later
2020-07-12T13:17:38  <cboltz> kl_eisbaer: somewhat related - we have a pillar/id/gcc2_i_o_o ("future replacement for gcc.o.o") - but gcc2.i.o.o goes NXDOMAIN.
2020-07-12T13:18:00  <cboltz> is this a superfluous pillar/id/ file (= no replacement of gcc.i.o.o needed), or something else?
2020-07-12T13:18:13  <kl_eisbaer> cboltz: that one can go, as we managed to upgrade the current gcc to the "latest" Tumbleweed.
2020-07-12T13:18:41  <cboltz> ok, I'll delete it
2020-07-12T13:18:51  <kl_eisbaer> (the last update is a month old, so the next "zypper dup" is needed for the machine - that's why I say "latest" ;-)
2020-07-12T13:18:57  <kl_eisbaer> thanks.
2020-07-12T13:19:32  <kl_eisbaer> cboltz: if you are on it: I guess after the latest discussions with SUSE-IT, it might make sense to remove the daffy machines from salt as well - as SUSE-IT wants to manage them "on their own".
2020-07-12T13:19:55  <cboltz> while doing the tumbleweed update, please also check why ssh login as user doesn't work on gcc.i.o.o
2020-07-12T13:22:14  <kl_eisbaer> hm, ok.
2020-07-12T13:23:08  <cboltz> for daffy* - they are (at least partially, for example keepalived) salt-managed, so IMHO it would make sense to keep them in salt
2020-07-12T13:23:54  <lcp79> kl_eisbaer: I assume you don't know how the osc-collab front page is done? the osc-plugin-collab repo has an even older version
2020-07-12T13:24:22  <kl_eisbaer> cboltz: if I understand bmwiedemann and jdsn correctly, they did not want to have any openSUSE hero on those machines. That in turn would make it impossible to have them controlled by a salt-master under heroes control. But I might be wrong - so better ask back.
2020-07-12T13:24:47  <lcp79> cboltz: monitor on jekyll.i.o.o is ready
2020-07-12T13:25:00  <kl_eisbaer> lcp79: no, not really. I just look at those machines when i update/upgrade them or if something is broken that I may be able to fix if the original maintainer is not available
2020-07-12T13:25:16  <lcp79> alright, will ask dimstar then
2020-07-12T13:25:26  <lcp79> who apparently is here >:D
2020-07-12T13:25:30  <kl_eisbaer> but I like to stay away from all dirty details of those machines, if possible ;-)
2020-07-12T13:25:58  <lcp79> I also have cooverview to update too, it's a lot of stuff
2020-07-12T13:26:18  <cboltz> lcp79: thanks, haproxy config updated. The monitor.o.o main page now gets served from jekyll.i.o.o
2020-07-12T13:26:48  <lcp79> god bless I haven't finished the obs theming code, I would have way more work to do
2020-07-12T13:26:51  <kl_eisbaer> lcp79:  are you also trying to provide Grafana dashboards ?
2020-07-12T13:27:01  <kl_eisbaer> In that case, I have some requests :-)
2020-07-12T13:27:17  <lcp79> not currently, but you can line up to have that done as well >:D
2020-07-12T13:27:54  <kl_eisbaer> hehe: I have no problems to line up - especially, if I get something at the end :-)
2020-07-12T13:28:49  <lcp79> we should have a request tracker for websites
2020-07-12T13:29:21  <kl_eisbaer> lcp79: which one would you prefer?
2020-07-12T13:30:01  <lcp79> I don't know tbh, can I request a pagure instance? >:D
2020-07-12T13:30:39  * kl_eisbaer is thinking about a "o-o startpages" Github repo and subdirectories for all of the sub-domains...
2020-07-12T13:31:02  <lcp79> that sounds like a nightmare tbh
2020-07-12T13:31:29  <kl_eisbaer> lcp79: feel free to request one - just line up to have that done as well :-D
2020-07-12T13:31:47  <kl_eisbaer> hm: really? Why?
2020-07-12T13:31:56  <lcp79> I will deploy pagure one day
2020-07-12T13:32:06  <kl_eisbaer> That would at least help to keep an overview about all the different sub-domain pages
2020-07-12T13:32:21  <lcp79> it would end up as unmaintained as everything else >:D
2020-07-12T13:32:54  <kl_eisbaer> jip, but right now, the pages are also unmaintained - and on top of that, nobody knows where their original source comes from...
2020-07-12T13:33:35  <cboltz> kl_eisbaer: pillar/role/jekyll_master.sls has a list ;-)
2020-07-12T13:33:41  <kl_eisbaer> Which makes it even harder to contribute. With a repo on github, people could at least do spell checking or other stuff ;-)
2020-07-12T13:34:21  <kl_eisbaer> cboltz: can I btw. remind you to think about a documentation for a "simple heroes Salt setup" for testing at home ?
2020-07-12T13:34:44  <lcp79> kl_eisbaer: mmarek's openSUSE email leads to novell.com address
2020-07-12T13:34:51  <kl_eisbaer> lol...
2020-07-12T13:35:43  <cboltz> kl_eisbaer: that's already hidden in our internal wiki ;-)  (with the exception of my additional salt files, maybe I should just upload the tarball?)
2020-07-12T13:35:44  <kl_eisbaer> ...and I can't log in on gcc.infra.o.o to find out who logged in lately :-/
2020-07-12T13:35:57  <kl_eisbaer> cboltz: yes, please do that.
2020-07-12T13:37:00  <lcp79> kl_eisbaer: if I wanted to touch setloc.js from mirrorbrain-icons, should I also do it in the download-o-o repo?
2020-07-12T13:37:30  <kl_eisbaer> but I guess that matz@s.d might be able to help out
2020-07-12T13:38:07  <kl_eisbaer> lcp79: both are so far completely independent from each other. So feel free to do whatever is needed.
2020-07-12T13:38:32  <lcp79> I have no clue how that's even loaded tbh
2020-07-12T13:39:25  <lcp79> oh yeah, cboltz, would you mind giving me permissions to download-o-o, I lost them when moving the repo to openSUSE namespace
2020-07-12T13:40:04  <kl_eisbaer> lcp79: the pages on download.o.o are generated by apache2-mod_mirrorbrain
2020-07-12T13:40:27  <lcp79> well, mod_autoindex_mb, no?
2020-07-12T13:40:40  <kl_eisbaer> lcp79: you are correct, yes
2020-07-12T13:41:01  <kl_eisbaer> There should be a "README.html" and a "HEADER.html" in the package
2020-07-12T13:41:29  <cboltz> lcp79: I upgraded you from "write" to "admin"
2020-07-12T13:41:34  <kl_eisbaer> ...and in addition some js, which is loaded by the "IndexHeadInsert" command
2020-07-12T13:41:57  <lcp79> alright, I will replace that with head.html file to load instead then
2020-07-12T13:42:46  <kl_eisbaer> lcp79: just have a look at the /etc/apache2/mod_autoindex-mirrorbrain.conf file on pontifex
2020-07-12T13:43:18  <kl_eisbaer> and the files that end up in /usr/share/apache2/mirrorbrain-icons/
2020-07-12T13:43:18  <lcp79> alright, I will, if I can ssh there >:D
2020-07-12T13:43:53  <kl_eisbaer> lcp79: IMHO you should - as normal user + sudo
2020-07-12T13:52:19  <cboltz> kl_eisbaer: https://progress.opensuse.org/projects/opensuse-admin-wiki/wiki/Testing_with_local_saltmaster#section-5
2020-07-12T13:54:56  <kl_eisbaer> thanks! Will start testing, once I have my server back with nouveau :-/
2020-07-12T13:57:46  <cboltz> different question: csync2 on anna says "Updating /etc/pdns/recursor.conf on elsa ..." -> "File stays in dirty state. Try again later...". Any idea what could cause this?
2020-07-12T13:58:09  <cboltz> and, bonus question, how to handle it?
2020-07-12T14:28:24  <kl_eisbaer> cboltz: this is a result of the latest upgrade: after that, the pdns-recursor.conf permissions were wrong. Before I figured that out, I decided to install and configure dnsmasq on anna instead, to get us up and running again.
2020-07-12T14:28:57  <kl_eisbaer> darix promised to look at his "baby" (aka: pdns-recursor) on elsa (to not interrupt the working dnsmasq setup on anna)
2020-07-12T14:29:08  <kl_eisbaer> but I did not hear back from him.
2020-07-12T14:30:02  <cboltz> tell him that I'll replace it with unbound if he doesn't fix it ;-)
2020-07-12T14:30:07  <kl_eisbaer> As dnsmasq would allow us to even have a simple DHCP server in the intra network, I am currently thinking of making the temporary solution (dnsmasq on anna) a permanent one - and obsoleting pdns on elsa as well.
2020-07-12T14:30:29  <kl_eisbaer> cboltz: feel free to tell him yourself, once he is online again ;-)
2020-07-12T14:30:38  <cboltz> ;-)
2020-07-12T14:30:59  <kl_eisbaer> But about your question: I expect the pdns-recursor.conf on elsa to the correct one, that should be copied over to anna.
2020-07-12T14:31:51  <kl_eisbaer> But: on pdns is currently disabled on anna anyway in favor of dnsmasq, so - whatever you do - it should not influence any running service at the moment.
2020-07-12T14:32:10  <cboltz> ok, then I'll just ignore the csync2 warning
2020-07-12T14:32:30  <kl_eisbaer> just try again now
2020-07-12T14:33:16  <cboltz> no complaints anymore :-) - thanks!
2020-07-12T14:34:05  <kl_eisbaer> ok - and elsa is rebooting now for kernel update
2020-07-12T14:40:09  * cboltz has to leave, see you later!
2020-07-12T15:07:15  <malcolmlewis> lcp79, hi, any idea why forum response is slow get png files?
2020-07-12T15:54:33  <lcp15> malcolmlewis: what do you mean?
2020-07-12T16:00:30  <malcolmlewis> lcp15, https://susepaste.org/7088823
2020-07-12T16:00:56  <malcolmlewis> for me the forum is running oh so slow.....
2020-07-12T16:02:25  <malcolmlewis> 15 plus seconds to get to any page
2020-07-12T16:17:23  <pjessen> malcolmlewis: ditto, very slow
2020-07-12T16:19:21  <malcolmlewis> pjessen, what do you think it might be, or does it need a restart?
2020-07-12T17:07:05  <malcolmlewis> pjessen, lcp15 ^^
2020-07-12T17:07:14  <pjessen> malcolmlewis: dunno - I would suspect database access.
2020-07-12T17:07:57  <malcolmlewis> pjessen, any logs to look at?
2020-07-12T17:09:46  <pjessen> malcolmlewis: plenty :-)  the apache log will basically confirm the issue
2020-07-12T17:10:14  <malcolmlewis> pjessen, no access here, do you?
2020-07-12T17:11:52  <pjessen> yup.
2020-07-12T17:12:06  <pjessen> seems to be running fine
2020-07-12T17:12:14  <malcolmlewis> go figure
2020-07-12T17:12:47  <pjessen> I only get long waits when I access something I haven't accessed before ?
2020-07-12T17:13:46  <malcolmlewis> pjessen, yes, but it's every thread I look at 10-20 seconds for it to load
2020-07-12T17:15:24  <pjessen> malcolmlewis: Yeah, I see that too.
2020-07-12T17:22:37  <malcolmlewis> pjessen, something has changed in the last 24-36hrs...
2020-07-12T17:23:12  <malcolmlewis> pjessen, stuck maintenance cron jobs?
2020-07-12T17:24:09  <malcolmlewis> pjessen, or does that re-indexing need to occur? I think I posted the screenshots for you?
2020-07-12T17:43:09  <pjessen> malcolmlewis: tbh, dunno.
2020-07-12T17:47:01  <malcolmlewis> pjessen, can you see this page on our forum? https://www.vbulletin.com/docs/html/main/acp_settings_options_serverandoptimization
2020-07-12T21:26:16  <lcp15> cboltz: https://cdn.discordapp.com/attachments/366989996101730304/731982614390505533/unknown.png >:D
2020-07-12T21:26:41  <lcp15> I can promise you right now that if we ever get Bugzilla 6 deployed, I will make this happen :P
2020-07-12T21:30:50  <cboltz> looks good :-) - so let's wait for someone to update bugzilla
2020-07-12T21:31:07  <lcp15> well, Bugzilla 6 is also unreleased yet >:D
2020-07-12T21:31:23  <lcp15> it has been in development for the last 2ish years
2020-07-12T21:32:00  <lcp15> I'm kinda involved tho
2020-07-12T21:32:25  <lcp15> because of https://bugzilla.lcp.world/ and https://planet-bugzilla.lcp.world/
2020-07-12T21:36:44  <cboltz> nice :-) - I start to wonder if there's something you are _not_ involved ;-)
2020-07-12T21:37:30  <lcp15> It's very easy to get me on board