2020-10-12T02:55:28  *** okurz_ is now known as okurz
2020-10-12T07:10:25  *** ldevulder_ is now known as ldevulder
2020-10-12T08:28:12  *** dddh__ is now known as dddh
2020-10-12T09:38:32  <klein> created an internal JIRA Ticket to deal with the creation of the 2 virtual machines (pagure and discourse), since I have no time to work on this, asked help to my team :-)
2020-10-12T10:47:52  <lcp> thanks!
2020-10-12T11:56:44  <lkocman> Hello team, can we please have a look at https://progress.opensuse.org/issues/73237 ?
2020-10-12T11:59:22  <okurz> lkocman: creating the progress ticket should be enough notification
2020-10-12T12:48:15  <darix> who decided that adding quad9 to the resolv.conf on heroes host is a good idea?
2020-10-12T14:22:52  <thomic> darix: seems like not a good idea
2020-10-12T14:22:57  <thomic> will you tell us why?
2020-10-12T14:24:19  <darix> /etc/resolv.conf is round robin
2020-10-12T14:24:54  <darix> so everytime the resolver picks 9.9.9.9 to resolve an internal hostname ... it will tell you "can not resolve host"
2020-10-12T14:25:16  <thomic> uff
2020-10-12T14:25:23  <thomic> well that should have been noticed :)
2020-10-12T14:26:49  <darix> it is the usual the workaround for "but if our resolver are unstable we have 9.9.9.9 at least "
2020-10-12T14:26:54  <darix> which is a very shitty workaround
2020-10-12T16:10:11  <darix> cboltz: who decided that adding quad9 to the resolv.conf on heroes host is a good idea?
2020-10-12T16:42:26  <lcp> darix: you may know stuff about https://progress.opensuse.org/issues/71563
2020-10-12T16:54:35  <darix> that still doesnt answer my question though
2020-10-12T16:54:46  <darix> I can only urge you not to mix internal and external resolvers
2020-10-12T16:54:57  <darix> unless you prefer random failures for testing
2020-10-12T17:28:12  <cboltz> darix: I only know that I didn't do that ;-)
2020-10-12T17:28:34  <cboltz> but I also know that resolv.conf(5) says   nameserver: [...]  If there are multiple servers, the resolver library queries them in the order listed.
2020-10-12T17:29:31  <cboltz> so as long as our internal DNS servers are listed first (and there's no "options rotate"), they should only be used as fallback
2020-10-12T17:30:54  <darix> cboltz: on anna/elsa it was configured like this as well and then commented out
2020-10-12T17:31:02  <darix> on pontifex we still found it active
2020-10-12T17:31:50  <cboltz> looking at pontifex - yes, there are external nameservers in resolv.conf, but no "options rotate"
2020-10-12T17:32:04  <cboltz> so the external nameservers should only be used as fallback
2020-10-12T17:32:38  <cboltz> (if the internal ones both fail or hit timeout:1)
2020-10-12T17:33:13  <darix> you do not want that
2020-10-12T17:33:23  <darix> it will lead to random errors
2020-10-12T17:33:28  <darix> (which is how we found it)
2020-10-12T18:10:31  <cboltz> darix: found the previous discussion about this: https://monitor.opensuse.org/heroes/%23opensuse-admin.2020-07-20.log (first ~20 lines)
2020-10-12T19:08:44  <lcp> cboltz: doing migration of archives up to september 30th
2020-10-12T19:08:56  <lcp> it is taking very long >:D
2020-10-12T19:10:47  <cboltz> We have tons of mails, therefore I'm not surprised ;-)
2020-10-12T19:41:47  <cboltz> speaking of mailman - when will you dare to deploy / highstate the changeme@example.com fix?
2020-10-12T20:14:48  <lcp> good point
2020-10-12T20:15:13  <lcp> I will when I'm not looking at it migrating >:D
2020-10-12T20:15:25  <lcp> it will probably finish in a few days >:D
2020-10-12T20:16:27  <lcp> there are some mails that aren't getting migrated, so I will have to look at why and migrate them individually later
2020-10-12T20:16:46  <cboltz> oh, that sounds interesting[tm]
2020-10-12T20:17:19  <cboltz> any pattern that causes the failures, or just random?
2020-10-12T20:18:49  <lcp> not random, it's the mlmmj appended message-id header
2020-10-12T20:19:41  <lcp> there was one that happened in obs-tests mailing list with message-id `<148875197121.28922.5341196545043271170@160.120.21.118>`
2020-10-12T20:19:55  <lcp> and there are a few hundred in opensuse mailing list
2020-10-12T20:21:11  <lcp> ah, and another issue, non-ascii sender header
2020-10-12T20:21:53  <lcp> I will collect all those issues and put them somewhere when it's done
2020-10-12T20:38:30  <lcp> openSUSE Paste is down
2020-10-12T20:38:59  <cboltz> hmm, works here
2020-10-12T20:39:27  <lcp> https://paste.centos.org/view/df3bf2a4 https://lists.opensuse.org/archives/list/users@lists.opensuse.org/ will need to be redone
2020-10-12T20:39:50  <lcp> the worst thing being that is the largest archive too
2020-10-12T20:40:08  <cboltz> well, what did you expect? ;-)
2020-10-12T20:40:39  <lcp> see, the issue is that there are like hundreds of emails with no date, so it assumes today
2020-10-12T20:40:48  <lcp> which is a nightmare
2020-10-12T20:40:59  <lcp> ugh
2020-10-12T20:41:13  <cboltz> that traceback in your first paste link looks like you should report it upstream - _header_value_parser.py breaks with empty strings
2020-10-12T20:41:39  <lcp> yup
2020-10-12T20:41:54  <lcp> I may just remove all the messages from october from opensuse mailing list
2020-10-12T20:42:04  <lcp> because those are just ones with wrong date
2020-10-12T20:44:25  <cboltz> just wondering - is https://lists.opensuse.org/archives/list/users@lists.opensuse.org/thread/O7FEY46453QJI6DBGWE3674ERAKAEPJT/ really a complete mail (modulo broken headers), or was this an attachment and the parsing made it a separate mail?
2020-10-12T20:44:39  <cboltz> (and yes, I know that searching for that in the mail archive might be funny[tm])
2020-10-12T20:47:55  <lcp> I can send it to you >:D
2020-10-12T20:48:11  <lcp> but you can download the mbox of it to see it in more detail in the sidebar
2020-10-12T20:48:43  <cboltz> I already did that download ;-)
2020-10-12T20:48:48  <lcp> that mbox tells me nothing >:D
2020-10-12T20:48:59  <cboltz> Content-Type: multipart/mixed
2020-10-12T20:49:05  <cboltz> with exactly one part
2020-10-12T20:49:59  <cboltz> searching the original mboxes is probably more helpful, but I'm afraid you'll get more than one match if you grep for   /misc    /etc/auto.misc
2020-10-12T20:50:18  <cboltz> and I'm not sure if   --===============2940320873304541671==   is also in the original mail
2020-10-12T20:51:00  <lcp> there are 3 occurrences of that body in the archive
2020-10-12T20:51:25  <lcp> but oom kills my editor when I load the archive sooo
2020-10-12T20:51:42  <cboltz> export EDITOR=vim   should help ;-)
2020-10-12T20:51:52  <lcp> I used vim
2020-10-12T20:51:57  <cboltz> oops ;-)
2020-10-12T20:52:14  <lcp> I know when to use what editor ;)
2020-10-12T20:53:54  <cboltz> I never managed to make vim go oom killed so I guessed you use $something_else ;-)
2020-10-12T20:54:10  <cboltz> try   grep -C50    - that should give enough context to include a date header
2020-10-12T20:56:20  <lcp> oh, I know what that is
2020-10-12T20:56:44  <lcp> it's an email attachment of /etc/auto.misc
2020-10-12T20:57:25  <lcp> but it's `grep: opensuse.mbox: binary file matches` according to grep
2020-10-12T20:58:19  <lcp> the other 3 occurrences are people talking about that file
2020-10-12T20:58:36  <cboltz> grep --text   then - that should force grep into text mode
2020-10-12T20:59:34  <cboltz> anyway - making an attachment a separate mail in hyperkitty sounds like the next bug to report ;-)
2020-10-12T21:00:47  <lcp> eh, I'm not so sure anymore
2020-10-12T21:00:56  <lcp> since that binary file was just an email
2020-10-12T21:01:29  <cboltz> do you know the date of that mail?
2020-10-12T21:06:43  <Lazy_Kent> Hello. I need an action about the IRC channel.
2020-10-12T21:07:18  <cboltz> define "an action" please ;-)
2020-10-12T21:09:25  <Lazy_Kent> cboltz: We decide to close the #opensuse-ru channel.
2020-10-12T21:09:55  <Lazy_Kent> It is not active.
2020-10-12T21:12:03  <cboltz> please send a mail to admin AT opensuse.org - a few lines are enough ;-)
2020-10-12T21:12:19  <cboltz> I'll try to find out how to close the channel in the next days
2020-10-12T21:12:41  <Lazy_Kent> The thread is there: https://forums.opensuse.org/showthread.php/540322-IRC-%D0%BA%D0%B0%D0%BD%D0%B0%D0%BB-opensuse-ru-%D0%9A%D0%BE%D0%BC%D1%83-%D0%BE%D0%BD-%D0%BD%D1%83%D0%B6%D0%B5%D0%BD
2020-10-12T21:16:55  <cboltz> lcp: I checked the mboxes on baloo, and probably found it in 2020-11: https://paste.opensuse.org/e53b44a2
2020-10-12T21:17:17  <cboltz> in mbox format,   From $mail $date   marks the beginning of a mail
2020-10-12T21:17:45  <cboltz> and that mail has (in the body, misinterpreted)   From /etc/auto.master:   and   From /etc/auto.misc:
2020-10-12T21:18:18  <cboltz> I'd argue that the mbox parser should be a bit more strict in what it accepts as valid "mail separator" From line
2020-10-12T21:26:00  <lcp> yikes, that's unfortunate
2020-10-12T21:26:21  <cboltz> Lazy_Kent: it seems https://ru.opensuse.org/openSUSE:%D0%9E%D0%B1%D1%89%D0%B5%D0%BD%D0%B8%D0%B5 still mentiones the IRC channel (actually #opensuse-ru is one of the few things I understand on that page ;-) so please remove it there
2020-10-12T21:26:45  <cboltz> bonus points if you also remove it from https://en.opensuse.org/openSUSE:IRC_list
2020-10-12T21:28:33  <Lazy_Kent> cboltz: I will try to do somehow.
2020-10-12T21:33:54  <cboltz> lcp: to make things worse - I guess *) the archive at the right date is also broken, it probably contains only the first half of the mail, ending at the misinterpreted From
2020-10-12T21:34:44  <cboltz> *) guess, because I'm not too keen to search for this mail by browsing the new archive, ~70 pages per month
2020-10-12T21:35:38  <cboltz> (search doesn't seem to work)
2020-10-12T21:39:33  <lcp> search didn't reindex yet
2020-10-12T21:40:07  <lcp> and you can select the month in the sidebar
2020-10-12T21:40:47  <lcp> oh, yeah, that's 70 pages
2020-10-12T21:40:48  <lcp> yikes
2020-10-12T21:47:00  <lcp> cboltz: here is the original email https://lists.opensuse.org/archives/list/users@lists.opensuse.org/message/DPKNJ34YZVRTTE2O33TN4FN4PHGJGRO7/, can't find the response to it though
2020-10-12T21:49:17  <cboltz> if you click "Back to the thread", you'll find the mail from corsepiu@ (the second mail on the page)
2020-10-12T21:49:35  <cboltz> and as I guessed, it ends with "Eg.:" :-/
2020-10-12T21:51:04  <lcp> oh yeah
2020-10-12T21:51:36  <lcp> well, I will finish migrating the rest and then take a look at the errors
2020-10-12T21:51:59  <lcp> since it seems the errors generate a million threads with no date, so they are easy to find >:D
2020-10-12T21:53:37  <cboltz> actually you can find these type of errors by grepping the mboxes on baloo:   zgrep --text "^From " *.gz | grep -v @ |grep -v '[0-9][0-9]$'
2020-10-12T21:53:50  <cboltz> 150 matches in the opensuse@ archive
2020-10-12T21:54:14  <lcp> whoop, that would more or less match the number of those empty emails
2020-10-12T21:54:27  <cboltz> interestingly only in 1999 and 2000
2020-10-12T21:54:52  <lcp> what backend did those list have back then?
2020-10-12T21:56:46  <cboltz> good question, the headers from 2000 are stripped to the bare minimum, no traces of any ML software left
2020-10-12T21:59:05  <cboltz> it looks like newer mboxes have those lines escaped as   >From   and https://en.wikipedia.org/wiki/Mbox confirms that escaping
2020-10-12T21:59:58  <cboltz> so maybe you should do this in your local copy of the mbox files, and then re-import the 1999 and 2000 archives
2020-10-12T22:00:23  <cboltz> (or re-import everything to be sure that no strange side effects are left)
2020-10-12T22:02:37  <lcp> I will reimport everything from 1999/2000
2020-10-12T22:05:29  <lcp> or just everything
2020-10-12T22:06:09  <cboltz> you might want to use some slightly better grep patterns than I did - my   grep -v   patterns so that it really only greps away the "real" mail separators
2020-10-12T22:06:24  <lcp> I will do it in vim
2020-10-12T22:06:32  <lcp> when I can get my hands on more ram >:D
2020-10-12T22:13:24  <cboltz> wild idea:   zgrep -C20  '^From.*[^0-9]$' opensuse-2000-12.mbox.gz > matches
2020-10-12T22:13:30  <cboltz> cp matches matches_ORIG
2020-10-12T22:13:39  <cboltz> vi matches
2020-10-12T22:13:47  <cboltz> diff -u matches_ORIG matches
2020-10-12T22:13:57  <cboltz> and then apply the patch to the big file
2020-10-12T22:14:12  <cboltz> it will give you *big* offsets, but in theory should work nevertheless
2020-10-12T22:18:29  <cboltz> (and again, a more restrictive grep pattern can't hurt ;-)
2020-10-12T22:21:38  <lcp> yup