2019-08-06T02:26:29  *** knurpht <knurpht!~knurpht@opensuse/member/Knurpht> has quit IRC (Ping timeout: 268 seconds)
2019-08-06T02:55:10  *** okurz <okurz!~okurz@ppp-46-244-216-85.dynamic.mnet-online.de> has quit IRC (Ping timeout: 246 seconds)
2019-08-06T02:55:48  *** okurz <okurz!~okurz@ppp-46-244-215-59.dynamic.mnet-online.de> has joined #opensuse-admin
2019-08-06T05:04:34  *** srinidhi <srinidhi!~srinidhi@opensuse/member/srinidhi> has joined #opensuse-admin
2019-08-06T06:16:46  *** moozaad <moozaad!~moozaad@opensuse/member/moozaad> has joined #opensuse-admin
2019-08-06T06:22:06  *** srinidhi <srinidhi!~srinidhi@opensuse/member/srinidhi> has quit IRC (Quit: Leaving.)
2019-08-06T06:23:01  *** srinidhi <srinidhi!~srinidhi@opensuse/member/srinidhi> has joined #opensuse-admin
2019-08-06T06:35:04  *** jadamek <jadamek!~julienA@i15-lef02-th2-89-83-245-73.ft.lns.abo.bbox.fr> has joined #opensuse-admin
2019-08-06T07:51:25  *** moozaad <moozaad!~moozaad@opensuse/member/moozaad> has quit IRC (Quit: Konversation terminated!)
2019-08-06T07:52:25  *** moozaad <moozaad!~moozaad@opensuse/member/moozaad> has joined #opensuse-admin
2019-08-06T09:32:27  *** lnussel <lnussel!lnussel@nat/novell/x-czvklnxoonbiitxn> has quit IRC (Ping timeout: 264 seconds)
2019-08-06T09:46:18  *** lnussel <lnussel!lnussel@nat/suse/x-etfanmmnuvdyrkps> has joined #opensuse-admin
2019-08-06T12:40:59  *** knurpht <knurpht!~knurpht@opensuse/member/Knurpht> has joined #opensuse-admin
2019-08-06T13:26:03  *** Eleventh_Doctor is now known as Pharaoh_Atem
2019-08-06T13:37:48  *** Pharaoh_Atem <Pharaoh_Atem!~neal@fedora/ngompa> has quit IRC (Remote host closed the connection)
2019-08-06T13:44:50  *** Pharaoh_Atem <Pharaoh_Atem!~neal@fedora/ngompa> has joined #opensuse-admin
2019-08-06T14:22:34  *** srinidhi <srinidhi!~srinidhi@opensuse/member/srinidhi> has quit IRC (Ping timeout: 246 seconds)
2019-08-06T14:28:26  *** srinidhi <srinidhi!~srinidhi@opensuse/member/srinidhi> has joined #opensuse-admin
2019-08-06T15:13:38  <okurz> hi, I just removed the permission "Delete issues" from the role "Developer" on progress.o.o as I was asked if that could be done as again someone by accident deleted a ticket. for https://progress.opensuse.org/projects/opensuse-admin that should be fine because it seems all that care about ticket deletion are "Manager" already
2019-08-06T15:13:46  <heroes-bot> <https://x0.no/4lfcj> (at progress.opensuse.org)
2019-08-06T15:15:03  *** cboltz <cboltz!~cb@opensuse/member/Cboltz> has joined #opensuse-admin
2019-08-06T15:34:06  *** mcaj <mcaj!mcaj@nat/suse/x-szwwukzkbljnzqre> has quit IRC (Quit: Leaving.)
2019-08-06T15:41:22  *** Eighth_Doctor <Eighth_Doctor!conankudom@fedora/ngompa> has quit IRC (Ping timeout: 240 seconds)
2019-08-06T15:41:46  *** Eighth_Doctor <Eighth_Doctor!conankudom@fedora/ngompa> has joined #opensuse-admin
2019-08-06T15:59:27  *** srinidhi <srinidhi!~srinidhi@opensuse/member/srinidhi> has quit IRC (Ping timeout: 264 seconds)
2019-08-06T16:06:15  *** srinidhi <srinidhi!~srinidhi@opensuse/member/srinidhi> has joined #opensuse-admin
2019-08-06T16:32:39  *** jadamek <jadamek!~julienA@i15-lef02-th2-89-83-245-73.ft.lns.abo.bbox.fr> has quit IRC (Quit: Leaving)
2019-08-06T17:40:38  *** srinidhi <srinidhi!~srinidhi@opensuse/member/srinidhi> has quit IRC (Quit: Leaving.)
2019-08-06T17:41:59  *** srinidhi <srinidhi!~srinidhi@opensuse/member/srinidhi> has joined #opensuse-admin
2019-08-06T17:53:05  <tuanpembual> Hi all
2019-08-06T17:57:12  *** mstroeder <mstroeder!~mstroeder@fttx-pool-37.60.0.76.bambit.de> has joined #opensuse-admin
2019-08-06T18:00:53  <cboltz> Hi!
2019-08-06T18:01:16  <cboltz> meeting time - who's here? ;-)
2019-08-06T18:01:26  <tuanpembual> me
2019-08-06T18:01:30  <mstroeder> meinereiner
2019-08-06T18:02:15  <cboltz> :-)
2019-08-06T18:02:43  <cboltz> while maybe a few more people wake up ;-) let's start with our first topic
2019-08-06T18:02:48  <bmwiedemann> Hi
2019-08-06T18:02:59  <cboltz> does someone from the community have a question?
2019-08-06T18:04:23  <cboltz> doesn't look so ;-)
2019-08-06T18:04:48  <cboltz> our next (and catch-all) topic is "status reports about everything"
2019-08-06T18:04:54  <cboltz> who wants to start?
2019-08-06T18:05:37  <tuanpembual> Me.
2019-08-06T18:05:54  <tuanpembual> update for new progress.
2019-08-06T18:06:11  <tuanpembual> *it is been a while where no update.
2019-08-06T18:07:38  <tuanpembual> cboltz suggest. We need "official" domain for testing progress to get routed via login2.opensuse.org
2019-08-06T18:08:21  <tuanpembual> domain like  "progress-test.opensuse.org"
2019-08-06T18:08:33  <tuanpembual> so who will can help me to do this?
2019-08-06T18:08:45  <tuanpembual> any idea @cboltz ?
2019-08-06T18:09:04  * tuanpembual will update ticket about this issue.
2019-08-06T18:09:06  *** robin_listas <robin_listas!~cer@70.red-88-9-30.dynamicip.rima-tde.net> has joined #opensuse-admin
2019-08-06T18:09:42  <tuanpembual> done from me.
2019-08-06T18:09:44  <cboltz> I'd say/hope that kbabioch_ or mcaj should have enough permissions to do the DNS entries, but they are both away
2019-08-06T18:09:49  <cboltz> so please open a ticket ;-)
2019-08-06T18:10:03  <tuanpembual> noted
2019-08-06T18:11:18  <cboltz> so - who's next?
2019-08-06T18:13:01  <cboltz> nobody? then I'll do a quick report about two recent downtimes
2019-08-06T18:13:05  <bmwiedemann> I'm somewhat worried that heroes are understaffed and tickets remain untouched for too long. I'd like to do more myself, but only got the VPN, no credentials
2019-08-06T18:13:58  <cboltz> sounds like something we should fix ;-)
2019-08-06T18:14:16  <cboltz> what exactly do you need?
2019-08-06T18:14:39  <cboltz> (for example, does the VPN connection work? Can you login as user on all VMs?)
2019-08-06T18:15:19  <bmwiedemann> VPN worked (last time I tried some weeks ago)... didnt try user login (was that documented somewhere?)
2019-08-06T18:16:25  <cboltz> you should be able to ssh to all VMs (see pillar/id/* in the salt repo for a list) as user
2019-08-06T18:16:47  <cboltz> to actually do something, someone needs to give you sudo permissions on specific machines
2019-08-06T18:17:07  *** srinidhi <srinidhi!~srinidhi@opensuse/member/srinidhi> has quit IRC (Ping timeout: 248 seconds)
2019-08-06T18:18:16  <cboltz> that's the quick summary - if you have questions, feel free to ask me ;-)
2019-08-06T18:18:48  <bmwiedemann> OK. ssh works :-)
2019-08-06T18:18:59  <cboltz> :-)
2019-08-06T18:22:03  <cboltz> let's continue while you look around on various VMs ;-)
2019-08-06T18:22:22  <cboltz> as I already mentioned, we had two recent downtimes:
2019-08-06T18:23:02  <cboltz> a) a mailinglist outage - I'm sure you've seen the mail on heroes@ (about two weeks ago)
2019-08-06T18:23:59  <cboltz> b) events.o.o was down for two days (after a reboot which "activated" a new ruby version) - kbabioch_ was able to hotfix it when we noticed the downtime, and in the meantime henne did a proper fix and updated the VM to Leap 15.1
2019-08-06T18:24:46  <cboltz> oh, and I migrated the meeting reminder (currently only used by the board) from an old SLE 11 to pinot.infra.o.o
2019-08-06T18:25:41  <cboltz> that's it from me
2019-08-06T18:25:54  <tuanpembual> ticket update: https://progress.opensuse.org/issues/27720
2019-08-06T18:27:07  <cboltz> thanks!
2019-08-06T18:27:14  <cboltz> does someone else have a status report or something to discuss?
2019-08-06T18:28:45  <mstroeder> Æ-DIR, should I stay or should I go? The test environment is up and running for one year now. But I guess nobody has the time to try it.
2019-08-06T18:29:19  <cboltz> yes, that's probably the problem :-(
2019-08-06T18:30:04  *** mcaj <mcaj!~mcaj@ip-37-188-255-169.eurotel.cz> has joined #opensuse-admin
2019-08-06T18:30:20  <mcaj> HI
2019-08-06T18:30:30  <cboltz> however, given the "fun" we have with sssd (it seems rebooting a 15.1 VM reliably breaks sudo unless you restart sssd - via the saltmaster, because sudo doesn't work) I'm more than open for switching to �dir
2019-08-06T18:30:57  <cboltz> hi mcaj
2019-08-06T18:31:24  <mstroeder> For the client side with Æ-DIR it's highly recommended to switch to aehostd (see https://www.ae-dir.com/aehostd.html).
2019-08-06T18:31:24  <cboltz> the last comment in  https://progress.opensuse.org/issues/27720  might be something for you (creating a DNS entry)
2019-08-06T18:31:25  <mcaj> Is it a bug of an error in config files ?
2019-08-06T18:32:03  <cboltz> restarting sssd "fixes" it
2019-08-06T18:32:18  <cboltz> so IMHO it shouldn't be an error in the config
2019-08-06T18:32:29  <bmwiedemann> maybe timing issue - something started too early in boot?
2019-08-06T18:32:41  <bmwiedemann> because systemd parallelizes as much as it can
2019-08-06T18:32:56  <bmwiedemann> journal should have some info?
2019-08-06T18:33:04  <cboltz> yes, could be
2019-08-06T18:33:23  <cboltz> I have to admit that I didn't check the boot log and "only" restarted sssd to get it working
2019-08-06T18:33:25  <mcaj> looks like missing dependency for sysctl conf
2019-08-06T18:34:12  <mcaj> I had simmilar problem with Haproxy and that time help me update the package to latest stable version
2019-08-06T18:34:47  <mcaj> but strange is that we sow this error on several machines but not all of them
2019-08-06T18:35:54  <cboltz> I didn't test everywhere, but on all 15.1 machines where I tested, sudo was broken after reboot, and restarting sssd "fixed" it
2019-08-06T18:36:26  <cboltz> I remember similar problems on 42.3 where sssd even broke ssh login as user
2019-08-06T18:36:41  <cboltz> but not always IIRC
2019-08-06T18:36:56  <cboltz> (now please don't argue that the situation improved on 15.1 because at least user login doesn't break ;-)
2019-08-06T18:37:02  <mstroeder> Hmm, I have customers running different versions of sssd. With years-old 1.9.6 there were several crashes during startup.
2019-08-06T18:38:24  <cboltz> systemctl status sssd   always says it's running, even in the cases it breaks sudo or ssh login
2019-08-06T18:39:35  <bmwiedemann> so it does not quit in case of errors
2019-08-06T18:39:35  *** mcaj <mcaj!~mcaj@ip-37-188-255-169.eurotel.cz> has quit IRC (Ping timeout: 268 seconds)
2019-08-06T18:39:54  <cboltz> if you or someone else wants to debug it - the "future TSP" VM is currently bored and nobody complains if it gets rebooted more often
2019-08-06T18:40:13  <mstroeder> Furthermore there were some bugs in the code using netlink detection. That's why I usually recommend to compile sssd with configure option --with-libnl=no
2019-08-06T18:40:18  <cboltz> for obvious reasons, I'll add your ssh pubkey to /root/.ssh/authorized_keys ;-)
2019-08-06T18:41:18  <mstroeder> Building with --with-libnl=no means that sssd won't detect network link for determining whether in online of offline mode.
2019-08-06T18:42:25  <mstroeder> sssd writes many log messages into various files beneath /var/log/sssd. Can you point me to an affected system?
2019-08-06T18:42:39  <cboltz> a quick look at the sssd.spec (in Factory) shows that it BuildRequires libnl, and that it doesn't use this configure option
2019-08-06T18:43:24  <cboltz> affected systems are probably all running Leap 15.1, for example narwal5, narwal6, narwal7, dale
2019-08-06T18:43:39  <mstroeder> libnl is automatically detected (sssd's configure default for --with-libnl is "auto")
2019-08-06T18:44:33  <cboltz> since /var/log/sssd/ is root-only, I'll copy those logs to /dev/shm on narwal7
2019-08-06T18:45:35  <cboltz> the log should show the problem after the last reboot
2019-08-06T18:50:54  *** mstroeder <mstroeder!~mstroeder@fttx-pool-37.60.0.76.bambit.de> has quit IRC (Read error: Connection reset by peer)
2019-08-06T18:51:50  *** mstroeder <mstroeder!~mstroeder@fttx-pool-37.60.0.76.bambit.de> has joined #opensuse-admin
2019-08-06T18:55:25  <mstroeder> sssd_nss.log shows "The Data Provider returned an error [org.freedesktop.sssd.Error.DataProvider.Offline]" which could mean that D-BUS is not ready yet during startup?
2019-08-06T18:55:46  <mstroeder> But it seems sssd is running with almost no logging at all.
2019-08-06T18:56:19  *** mcaj <mcaj!~mcaj@ip-37-188-238-197.eurotel.cz> has joined #opensuse-admin
2019-08-06T18:56:29  <mcaj> To me looks like a bug in Leap 15.1 ... see our frends thread at: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org/thread/OZOHTCTD4CQE6XB6ZQS3YLDYX76DW5XI/
2019-08-06T18:56:37  <heroes-bot> <https://x0.no/4lfjs> (at lists.fedorahosted.org)
2019-08-06T19:01:54  <cboltz> I see a familiar name there ;-)  - and yes, it looks like a valid explanation
2019-08-06T19:03:21  *** mstroeder <mstroeder!~mstroeder@fttx-pool-37.60.0.76.bambit.de> has quit IRC (Remote host closed the connection)
2019-08-06T19:05:04  *** mstroeder <mstroeder!~mstroeder@fttx-pool-37.60.0.76.bambit.de> has joined #opensuse-admin
2019-08-06T19:05:04  <mcaj> we can put there a fix in the config file but the bug should be fixed by opensuse or better up stream
2019-08-06T19:05:52  <cboltz> the sssd config is in pillar/common.sls
2019-08-06T19:06:07  <cboltz> and I'm sure you know how to write a bugreport ;-)
2019-08-06T19:10:19  <mstroeder> You mean I should write a bug report to the sssd bug tracker?
2019-08-06T19:11:04  <cboltz> you are the sssd expert, so assuming you think it's a bug - yes, please ;-)
2019-08-06T19:11:36  <mstroeder> Well, I rather gave up using sssd. ;-)
2019-08-06T19:12:08  <mstroeder> Maybe this is already fixed in sssd 2.x. Its change logs are huge.
2019-08-06T19:13:17  <cboltz> interesting question - Tumbleweed has 2.2.0, but Leap 15.1 is at 1.16.1
2019-08-06T19:13:47  <cboltz> (and I slightly ;-) doubt Leap and SLE want to do a major version upgrade)
2019-08-06T19:14:29  <mcaj> ok we will report it as a bug, fine, shall I open next topic ? Leap 42.3 machines ? ) ticket  https://progress.opensuse.org/issues/54968
2019-08-06T19:14:30  <mstroeder> IIRC Leap's sssd package is from SLE15.
2019-08-06T19:15:22  <cboltz> right, and that makes a version upgrade even less likely
2019-08-06T19:15:24  <mstroeder> Anyway, one of the reasons why I sat down implementing aehostd is getting rid of sssd's complexity.
2019-08-06T19:15:41  <mcaj> see https://en.opensuse.org/Lifetime
2019-08-06T19:17:41  <mcaj> Our Team will try to do some update soon but we definitly need Hereos to help us with it . (team = SUSE Eng-Infra)
2019-08-06T19:17:42  <cboltz> mcaj: I'm afraid your list is incomplete - a "salt \* grains.get osrelease" lists some more 42.3 machines
2019-08-06T19:18:54  <mcaj> yes, feel free to update the ticket, this was just what I found doring the kernel update on machines.. salt is rigts ..
2019-08-06T19:19:34  <mcaj> anyway zypper dup is need it ;)
2019-08-06T19:21:00  <cboltz> ticket updated
2019-08-06T19:21:37  <bmwiedemann> do zypper repos use $releasever already?
2019-08-06T19:22:30  <cboltz> no, you'll have to adjust them manually (or, better, adjust salt to switch to using $releasever ;-)
2019-08-06T19:23:47  <bmwiedemann> will do
2019-08-06T19:25:02  <cboltz> :-)
2019-08-06T19:27:21  <cboltz> speaking about salt - can someone who knows postgresql please have a look at https://progress.opensuse.org/issues/54503 ?
2019-08-06T19:27:41  <cboltz> I'd love not to have a highstate "timebomb" ;-)
2019-08-06T19:31:39  *** mstroeder <mstroeder!~mstroeder@fttx-pool-37.60.0.76.bambit.de> has quit IRC (Remote host closed the connection)
2019-08-06T19:33:42  *** robin_listas <robin_listas!~cer@70.red-88-9-30.dynamicip.rima-tde.net> has left #opensuse-admin
2019-08-06T19:38:49  *** mstroeder <mstroeder!~mstroeder@fttx-pool-37.60.0.76.bambit.de> has joined #opensuse-admin
2019-08-06T19:40:15  <bmwiedemann> made the $releasever salt MR
2019-08-06T19:43:13  <cboltz> any idea why I see "Could not retrieve the pipeline status" in your MR?
2019-08-06T19:43:52  <bmwiedemann> dont know
2019-08-06T19:44:34  <bmwiedemann> maybe the $ sign is special and needs escaping?
2019-08-06T19:49:17  <cboltz> no, AFAIK the $ sign doesn't have a special meaning in sls files
2019-08-06T19:50:03  <kbabioch_> the pipeline error is (probably) not related to $ -> https://gitlab.infra.opensuse.org/bmwiedemann/salt/-/jobs/4456
2019-08-06T19:50:11  <heroes-bot> <https://x0.no/4lfll> (at gitlab.infra.opensuse.org)
2019-08-06T19:50:18  <kbabioch_> "The requested URL returned error: 403"
2019-08-06T19:50:30  <kbabioch_> "
2019-08-06T19:50:30  <kbabioch_> remote: You are not allowed to download code from this project."
2019-08-06T19:51:02  <kbabioch_> bmwiedemann: if i see it correctly, you've forked the project ... not sure if this workflow is fully supported with the ci/cd ...
2019-08-06T19:52:46  <bmwiedemann> interesting... I thought that is the standard workflow (unless you want to give everyone write permissions on the main repo)
2019-08-06T19:53:23  * mcaj has 10% of the laptop batery :(
2019-08-06T19:53:56  <cboltz> bmwiedemann: making your fork more public might help
2019-08-06T19:54:00  <kbabioch_> bmwiedemann: yes, agreed, its the better way and what we all know from github, etc. pp. ... its just one thing i've noticed that is different from your merge request comapred to many others that do work fine ;-)
2019-08-06T19:54:39  <cboltz> some people use forks and some people create branches in the main repo
2019-08-06T19:54:46  <cboltz> so you'll see both ways depending on who does a MR
2019-08-06T19:55:38  <mcaj> time ti go CU next time
2019-08-06T19:56:02  <cboltz> BTW: I just tested (on a local test VM) and $releasever works as expected
2019-08-06T19:57:18  <bmwiedemann> kbabioch_: settings have it as "Private" and dont allow to change visibility
2019-08-06T19:59:12  <bmwiedemann> good night to you then. I'll get me some sleep, too.
2019-08-06T19:59:58  *** mcaj <mcaj!~mcaj@ip-37-188-238-197.eurotel.cz> has quit IRC (Ping timeout: 246 seconds)
2019-08-06T20:21:36  *** mstroeder <mstroeder!~mstroeder@fttx-pool-37.60.0.76.bambit.de> has quit IRC (Quit: Leaving)
2019-08-06T20:36:08  *** Son_Goku <Son_Goku!~King_InuY@fedora/ngompa> has joined #opensuse-admin
2019-08-06T21:05:10  *** Son_Goku <Son_Goku!~King_InuY@fedora/ngompa> has quit IRC (Read error: Connection reset by peer)
2019-08-06T21:06:35  *** Son_Goku <Son_Goku!~King_InuY@fedora/ngompa> has joined #opensuse-admin
2019-08-06T21:08:04  *** Son_Goku <Son_Goku!~King_InuY@fedora/ngompa> has quit IRC (Read error: No route to host)
2019-08-06T21:12:07  *** jadamek <jadamek!~julienA@176.164.254.202> has joined #opensuse-admin
2019-08-06T21:12:47  *** Sir_Gallantmon <Sir_Gallantmon!~King_InuY@fedora/ngompa> has joined #opensuse-admin
2019-08-06T21:17:35  *** Sir_Gallantmon <Sir_Gallantmon!~King_InuY@fedora/ngompa> has quit IRC (Quit: My MacBook has gone to sleep. ZZZzzz…)
2019-08-06T21:38:54  *** mcaj <mcaj!~mcaj@cst2-170-161.cust.vodafone.cz> has joined #opensuse-admin
2019-08-06T21:58:34  *** jadamek <jadamek!~julienA@176.164.254.202> has quit IRC (Quit: Leaving)
2019-08-06T22:11:52  *** mcaj <mcaj!~mcaj@cst2-170-161.cust.vodafone.cz> has quit IRC (Ping timeout: 244 seconds)
2019-08-06T22:12:20  *** cboltz <cboltz!~cb@opensuse/member/Cboltz> has quit IRC ()
2019-08-06T22:18:25  *** mcaj <mcaj!~mcaj@cst2-170-161.cust.vodafone.cz> has joined #opensuse-admin
2019-08-06T22:53:33  *** moozaad <moozaad!~moozaad@opensuse/member/moozaad> has quit IRC (Quit: Konversation terminated!)
2019-08-06T23:10:19  *** mcaj <mcaj!~mcaj@cst2-170-161.cust.vodafone.cz> has quit IRC (Remote host closed the connection)