2021-04-04T14:10:44  <lcp> cboltz: would you point 1 of the old mailing list urls to mailman3?
2021-04-04T14:10:57  <lcp> I will do testing of rewriting with nginx
2021-04-04T14:11:14  <lcp> (I don't care which one tbh)
2021-04-04T14:11:36  <cboltz> I guess opensuse-test is a good candidate ;-)
2021-04-04T14:11:44  <lcp> yeah
2021-04-04T14:12:41  <lcp> I need to find some space on this vm because this file has over 300M and we barely manage to fit with the existing stuff
2021-04-04T14:13:00  <cboltz> haproxy config updated
2021-04-04T14:13:17  <lcp> thank you!
2021-04-04T14:13:27  <cboltz> sounds like you should request a bigger disk
2021-04-04T14:14:12  <lcp> we do have 120GB disk in addition to the base one
2021-04-04T14:14:20  <lcp> but that's taken up by xapian index
2021-04-04T14:43:49  <darix> during normal workhours it is possible to resize any disks or add more without downtime
2021-04-04T15:21:03  <lcp> cboltz: https://lists.opensuse.org/opensuse-test/2006-07/msg00000.html
2021-04-04T15:21:47  <lcp> so redirects from the mails are covered, now the lists
2021-04-04T15:21:56  <cboltz> :-)
2021-04-04T15:24:27  <lcp> nginx isn't particularly happy about this tbh, it doesn't like big hashmaps like this >:D
2021-04-04T15:24:53  <cboltz> I'm not too surprised ;-)
2021-04-04T15:25:13  <cboltz> what exactly does "not particularly happy" mean in practise?
2021-04-04T15:27:50  <lcp> it means I had to bump the default settings 16 or so times before it stopped complaining
2021-04-04T15:29:24  <cboltz> ;-)
2021-04-04T15:30:19  <lcp> also restarting nginx kills mailman webui
2021-04-04T15:30:40  <cboltz> oh, "great"
2021-04-04T15:30:59  <darix> why is that?
2021-04-04T15:31:22  <darix> are you running it within nginx' python support?
2021-04-04T15:35:05  <lcp> it's its own process so it's through uwsgi_pass
2021-04-04T15:35:18  <Eighth_Doctor> yeah, we're using nginx's uwsgi runner
2021-04-04T15:35:48  <Eighth_Doctor> (using uwsgi_pass means nginx is the controller process, as opposed to how pagure runs with simple proxy pass to gunicorn)
2021-04-04T15:48:44  <darix> i think we package the external uwsgi runners too
2021-04-04T15:49:05  <darix> it is even in the distro
2021-04-04T15:49:08  <darix> JFYI
2021-04-04T15:49:42  <darix> uwsgi-python3
2021-04-04T15:50:03  <Eighth_Doctor> yeah, I'm not sure why mailman is configured to use uwsgi through nginx itself
2021-04-04T15:50:30  <Eighth_Doctor> it also makes mailman slower, because the worker pool is in nginx rather than separately
2021-04-04T15:59:33  <lcp> I mean, all the config is in salt if you want to modify it
2021-04-04T16:44:02  <cboltz> lcp: I'm not sure if your lists.rewritemap is correct, especially (?<month>/[0-9]{2})
2021-04-04T16:44:15  <cboltz> the old archives use YYYY-MM, not YYYY/MM
2021-04-04T16:44:26  <lcp> uh, you are right
2021-04-04T16:45:00  <lcp> I am more worried about https://lists.opensuse.org/opensuse-test/ redirecting to users@l.o.o?
2021-04-04T16:45:06  <lcp> how? why?
2021-04-04T16:46:03  <cboltz> wild guess: add a $ to your regexes
2021-04-04T16:46:48  <lcp> that's probably a good idea
2021-04-04T16:47:48  <lcp> I don't exactly know where to add $ tbh, because I also wanna capture the random html files from mhonarc
2021-04-04T16:49:35  <lcp> how about `"~/opensuse-test(?<year>/[0-9]{4})?-?(?<month>[0-9]{2})?.*?$" "/archives/list/test@lists.opensuse.org$year/$month";`
2021-04-04T16:50:55  <cboltz> .*$ is the same as not having $ ;-)
2021-04-04T16:51:43  <lcp> that's true
2021-04-04T16:52:04  <lcp> I could do .*?(.html)?$ instead I guess?
2021-04-04T16:53:07  <lcp> maybe I should do more than 1 file for redirects, this is not gonna be very readable
2021-04-04T16:53:31  <cboltz> that's still all optional, and would allow .* _or_ .html
2021-04-04T16:53:48  <cboltz> maybe enforce a / after the date, or use   (/.*)?   ?
2021-04-04T16:54:32  <lcp> I will just go on baloo and check what those html files were
2021-04-04T16:56:10  <lcp> `(|threads.html|date.html|all.html|author.html)`
2021-04-04T16:56:50  <cboltz> nitpicking: \.html ;-)
2021-04-04T16:57:02  <lcp> good call
2021-04-04T16:57:19  <cboltz> well, I doubt there would be accidential matches for the . in this case
2021-04-04T16:58:11  <lcp> `"~/opensuse-test(?<year>/[0-9]{4})?-?(?<month>[0-9]{2})?/(|threads\.html|date\.html|all\.html|author\.html)$" "/archives/list/test@lists.opensuse.org$year/$month";`
2021-04-04T16:59:01  <cboltz> looks good
2021-04-04T17:03:06  <lcp> curl tells me it works, firefox has the redirect cached which makes sense considering it's 301
2021-04-04T17:03:39  <lcp> or is 302 permanent
2021-04-04T17:03:44  <lcp> it doesn't matter >:D
2021-04-04T17:04:04  <cboltz> 301 is permanent
2021-04-04T17:04:11  <lcp> ah
2021-04-04T17:04:30  <lcp> there are technically 2 redirects missing though
2021-04-04T17:04:40  <lcp> /stats and /search
2021-04-04T17:05:59  <cboltz>  /search already goes 404, are you sure about that?
2021-04-04T17:06:10  <cboltz> for /stats, I'd redirect that to /
2021-04-04T17:06:45  <lcp> it was cgi-bin/search maybe?
2021-04-04T17:07:11  <lcp> /cgi-bin/search.cgi
2021-04-04T17:07:20  <cboltz> right
2021-04-04T17:07:41  <lcp> but that requires rewriting parameters >:D
2021-04-04T17:08:01  <cboltz> I'd just redirect it to /
2021-04-04T17:08:06  <lcp> maybe ignore the parameters and just redirect to search
2021-04-04T17:08:45  <cboltz> no objections ;-)
2021-04-04T17:08:45  <lcp> why not /archives/search
2021-04-04T17:09:34  <cboltz> https://lists.opensuse.org/archives/search -> "Sorry but your query looks empty."
2021-04-04T17:09:40  <cboltz> which is not really surprising
2021-04-04T17:09:52  <lcp> because it is
2021-04-04T17:09:59  <cboltz> yes ;-)
2021-04-04T17:10:20  <cboltz> now the question is if it makes sense to redirect people to this error message, or if / would be more helpful
2021-04-04T17:16:26  <cboltz> just checked the access log - /cgi-bin/search has been accessed 260 times in March
2021-04-04T17:16:48  <cboltz> 150 of them have a lists.o.o referrer, and 100 have an empty referrer
2021-04-04T17:17:09  <cboltz> so I guess when the redirects for all lists are in place, /cgi-bin/search won't see many accesses anymore
2021-04-04T17:18:00  <lcp> where did the last 10 come from
2021-04-04T17:20:51  <cboltz> something like https://lists.opensuse.org/opensuse-factory/ - with various lists, sometimes also from a specific message
2021-04-04T17:21:11  <cboltz> so basically people use the search field on the old archive pages
2021-04-04T17:21:15  <lcp> ah
2021-04-04T17:21:45  <lcp> they can use the search engine on the new archive pages too >:D
2021-04-04T17:21:59  <cboltz> right
2021-04-04T17:22:10  <lcp> oh
2021-04-04T17:22:21  <lcp> I forgot about redirects to the archives
2021-04-04T17:22:31  <lcp> as in mboxes
2021-04-04T17:22:47  <cboltz> should be easy to add ;-)
2021-04-04T17:22:49  <lcp> and rss feed
2021-04-04T17:23:04  <lcp> to that regex? >:D
2021-04-04T17:23:09  <lcp> I doubt it
2021-04-04T17:23:39  <cboltz> you can make it a separate set of regexes if that's easier ;-)
2021-04-04T17:24:07  <cboltz> also just wondering - would it make sense to make the trailing / optional?
2021-04-04T17:24:18  <lcp> this sounds like you want me to actually try using the current regex >:D
2021-04-04T17:24:39  <lcp> in the source of the redirect?
2021-04-04T17:27:04  <lcp> I don't think so? since the current archives enforce trailing slash and otherwise 404 out
2021-04-04T17:27:39  <cboltz> https://lists.opensuse.org/opensuse-project - apache will redirect you to https://lists.opensuse.org/opensuse-project/
2021-04-04T17:27:54  <lcp> oh, I guess it depends then
2021-04-04T17:27:54  <cboltz> I guess replacing   /(|threads\.html|date\.html|all\.html|author\.html)$   with   (|/|/threads\.html|/date\.html|/all\.html|/author\.html)$   should work
2021-04-04T17:29:28  <cboltz> it's probably the usual directory redirect
2021-04-04T17:30:14  <lcp> yeah, maybe
2021-04-04T17:52:05  <lcp> did `"/opensuse-test/opensuse-test-(?<year>/[0-9]{4})-(?<month>[0-9]{2})\.mbox(\.gz)?" "/test@lists.opensuse.org/export/test@lists.opensuse.org-$year-$month\.mbox\.gz";` and `"/opensuse-test/mailinglist.rss" "/archives/list/test@lists.opensuse.org/feed/";`
2021-04-04T18:10:39  <cboltz> for mbox, <year>/ should be just <year>
2021-04-04T18:12:36  <cboltz> and unless you add a $, (\.gz)? is superfluous ;-)
2021-04-04T18:24:37  <lcp> I should add $ tbh
2021-04-04T18:25:55  <lcp> cboltz: I managed to crush ci with the size of this mr in gitlab
2021-04-04T18:34:37  <cboltz> I guess nobody expected a 340 MB config file ;-)
2021-04-04T18:38:29  <lcp> it's not even finished yet though >:D
2021-04-04T18:38:48  <lcp> we are still missing around 2400 mails there
2021-04-04T18:38:55  <cboltz> just wondering - does it really make sense to have a file of this size in git?
2021-04-04T18:39:11  <cboltz> or would packaging it (and limiting salt to touch'ing it) be better?
2021-04-04T18:40:25  <lcp> good question
2021-04-04T19:09:12  <cboltz> the problem I see is that this file will make our git repo much bigger, and that it will make the CI much slower (or completely break it)
2021-04-04T19:10:00  <cboltz> so even if managing this file outside git is in theory worse, it's probably better in practise
2021-04-04T19:48:35  <pjessen> lcp: how did you manage to cut back the disk space consumed on mailman3?
2021-04-04T19:49:50  <pjessen> 38% is way better than 98% :-)
2021-04-04T19:51:15  <openSUSEBot> <LCP> I don't really know? I didn't touch xapian at all
2021-04-04T19:51:52  <openSUSEBot> <LCP> honestly, we should try running index per list or for a selected period of time so it's not doing everything at once