2019-09-26T00:38:44  <a-865k> [zypp] Exception.cc(log):166 MediaSetAccess.cc(provide):266 CAUGHT:   Permission to access 'http://download.opensuse.org/tumbleweed/repo/oss/x86_64/libKF5ConfigGui5-5.61.0-1.1.x86_64.rpm' denied
2019-09-26T00:38:56  <heroes-bot> <https://x0.no/4nsc7> (at download.opensuse.org)
2019-09-26T00:40:15  *** knurpht <knurpht!~knurpht@opensuse/member/Knurpht> has joined #opensuse-admin
2019-09-26T00:49:44  <a-865k> zypper ref is getting access denied
2019-09-26T00:50:10  <a-865k> knurpht: same for you?
2019-09-26T00:52:30  <a-865k> using gwdg.de works, but not d.o.o.
2019-09-26T00:53:14  <a-865k> [zypp] Exception.cc(log):166 MediaSetAccess.cc(provide):266 CAUGHT:   Permission to access 'http://download.opensuse.org/tumbleweed/repo/oss/x86_64/libKF5ConfigGui5-5.61.0-1.1.x86_64.rpm' denied
2019-09-26T00:53:14  <heroes-bot> <https://x0.no/4nsc7> (at download.opensuse.org)
2019-09-26T01:06:16  <knurpht> a-865k: Same here. Earlier today too. Basically 403 on download.o.o But, could be a maintenance issue.
2019-09-26T01:06:45  <a-865k> https://progress.opensuse.org/issues/57383
2019-09-26T01:07:55  <a-865k> also https://progress.opensuse.org/issues/57359
2019-09-26T01:19:59  *** garloff_ <garloff_!~garloff@xdsl-87-78-139-162.nc.de> has quit IRC (Ping timeout: 265 seconds)
2019-09-26T01:22:33  *** boombatower <boombatower!~boombatow@drupal.org/user/214218/view> has quit IRC (Quit: Konversation terminated!)
2019-09-26T02:29:05  *** okurz_ <okurz_!~okurz@ppp-46-244-210-56.dynamic.mnet-online.de> has joined #opensuse-admin
2019-09-26T02:30:57  *** okurz <okurz!~okurz@ppp-46-244-197-4.dynamic.mnet-online.de> has quit IRC (Ping timeout: 240 seconds)
2019-09-26T02:30:57  *** okurz_ is now known as okurz
2019-09-26T02:32:00  *** knurpht <knurpht!~knurpht@opensuse/member/Knurpht> has quit IRC (Ping timeout: 265 seconds)
2019-09-26T05:54:40  *** srinidhi <srinidhi!~srinidhi@opensuse/member/srinidhi> has joined #opensuse-admin
2019-09-26T06:30:49  *** ldevulder__ <ldevulder__!~ldevulder@i15-lef02-th2-89-83-232-105.ft.lns.abo.bbox.fr> has joined #opensuse-admin
2019-09-26T06:30:52  *** jadamek <jadamek!~julienA@176.164.202.125> has joined #opensuse-admin
2019-09-26T06:34:38  *** ldevulder_ <ldevulder_!~ldevulder@176.164.222.118> has quit IRC (Ping timeout: 276 seconds)
2019-09-26T06:35:48  *** moozaad <moozaad!~moozaad@opensuse/member/moozaad> has joined #opensuse-admin
2019-09-26T06:47:19  *** marxin <marxin!marxin@nat/suse/x-iypazkihngclegfs> has joined #opensuse-admin
2019-09-26T06:59:34  *** srinidhi <srinidhi!~srinidhi@opensuse/member/srinidhi> has quit IRC (Quit: Leaving.)
2019-09-26T07:00:38  *** srinidhi <srinidhi!~srinidhi@opensuse/member/srinidhi> has joined #opensuse-admin
2019-09-26T07:41:22  *** ldevulder__ is now known as ldevulder
2019-09-26T07:44:03  *** lnussel <lnussel!lnussel@nat/suse/x-aijapppnuupjmstw> has quit IRC (Quit: Lost terminal)
2019-09-26T07:44:07  *** enavarro_suse <enavarro_suse!enavarro_s@nat/novell/x-brdxhabdrxtgocuc> has quit IRC (Quit: ZNC 1.7.2 - https://znc.in)
2019-09-26T07:44:07  *** agraul <agraul!agraul@opensuse/member/agraul> has quit IRC (Write error: Connection reset by peer)
2019-09-26T07:50:22  *** agraul <agraul!agraul@opensuse/member/agraul> has joined #opensuse-admin
2019-09-26T08:34:47  *** enavarro_suse <enavarro_suse!enavarro_s@nat/suse/x-vqssojgjqtupukgm> has joined #opensuse-admin
2019-09-26T09:27:04  *** garloff_1 <garloff_1!~garloff@xdsl-78-34-255-212.nc.de> has joined #opensuse-admin
2019-09-26T09:39:19  *** marxin <marxin!marxin@nat/suse/x-iypazkihngclegfs> has quit IRC (Quit: Leaving)
2019-09-26T12:23:32  *** srinidhi <srinidhi!~srinidhi@opensuse/member/srinidhi> has quit IRC (Ping timeout: 246 seconds)
2019-09-26T12:31:40  *** boombatower <boombatower!~boombatow@drupal.org/user/214218/view> has joined #opensuse-admin
2019-09-26T12:54:33  *** srinidhi <srinidhi!~srinidhi@opensuse/member/srinidhi> has joined #opensuse-admin
2019-09-26T13:51:58  *** srinidhi <srinidhi!~srinidhi@opensuse/member/srinidhi> has quit IRC (Remote host closed the connection)
2019-09-26T14:59:00  *** cboltz <cboltz!~cb@opensuse/member/Cboltz> has joined #opensuse-admin
2019-09-26T15:16:20  <kbabioch> is there anyone who knows the history of status1.o.o & status2.o.o
2019-09-26T15:16:24  <kbabioch> are both still used?
2019-09-26T15:16:29  <kbabioch> cert for status2.o.o expired a long time ago
2019-09-26T15:16:38  <kbabioch> i can access status1.o.o, but not status2.o.o
2019-09-26T15:16:42  <cboltz> I can explain ;-)
2019-09-26T15:16:51  * kbabioch listens
2019-09-26T15:17:08  <cboltz> normally we have a cname status -> status1, so this is where people end up usually
2019-09-26T15:17:41  <cboltz> but sometimes (for example there was a planned downtime of the whole SUSE datacenter in Nuremberg a while ago) we switch over the cname to status2
2019-09-26T15:18:04  <kbabioch> ok, and this is hosted where exactly? because ip looks "foreign"
2019-09-26T15:18:16  <cboltz> status2 is running in Provo
2019-09-26T15:18:20  <kbabioch> ah, okay
2019-09-26T15:18:26  <kbabioch> VERSION="42.3"
2019-09-26T15:18:34  <kbabioch> this is what i wanted to fix :-)
2019-09-26T15:18:46  <kbabioch> and are they managed via salt / freeipa? could you maybe add my key there ?
2019-09-26T15:18:49  <kbabioch> can't access it
2019-09-26T15:19:08  <cboltz> be careful - the web frontend doesn't work with the PHP version in 15.1
2019-09-26T15:19:15  <cboltz> so we'll need to update that first
2019-09-26T15:19:18  <kbabioch> ok
2019-09-26T15:20:19  <cboltz> (I already upgraded status2 which is "usually superfluous", which means it's broken right now)
2019-09-26T15:20:23  <kbabioch> :-)
2019-09-26T15:21:27  <cboltz> for getting access - status{1,2} are not in salt and don't use freeipa for the login, which means I'll have to upload your ssh key
2019-09-26T15:21:47  <kbabioch> i can access status1 (so someone put my key there) ... but not on status2 :-/
2019-09-26T15:21:57  <kbabioch> but if you can copy my key over, it would be appreciated
2019-09-26T15:21:58  <cboltz> the reason behind that was "if everything is burning, status.o.o should still work", which also means reducing dependencies on other services
2019-09-26T15:22:13  <cboltz> however I'm not sure if that's worth not having salt on status.o.o
2019-09-26T15:22:40  <kbabioch> well, at least i can understand this arguemnt for freeipa ... for salt it shouldn#t matter when the master isn't available for a while
2019-09-26T15:22:51  <cboltz> as long as we keep some ssh keys in /root/.ssh/authorized_keys, loosing the salt and freeipa connection shouldn't hurt ;-)
2019-09-26T15:23:35  <cboltz> for additional fun, status{1,2} were setup manually, so maybe it's a good idea to re-create them from scratch with salt
2019-09-26T15:24:13  <cboltz> (apache, php, mysql + Cachet - nothing really hard, it "just" will take some time)
2019-09-26T15:27:01  *** heroes-bot <heroes-bot!~openSUSE-@gate.opensuse.org> has joined #opensuse-admin
2019-09-26T15:27:13  <kbabioch> the sad thing is that nothing we do is "really hard" ... mostly a question of taking time and being strict
2019-09-26T15:27:23  <cboltz> right ;-)
2019-09-26T15:28:10  <cboltz> I remember a few exceptions / "war stories", but in general I agree
2019-09-26T15:29:10  <kbabioch> but we need to get away from this pet server mentality ... automation / configuration management all the way ;-)
2019-09-26T15:29:33  * kbabioch is also guilty of this sometimes :-/
2019-09-26T15:30:23  <cboltz> the funny thing is -
2019-09-26T15:30:40  <cboltz> in the first Heroes meeting in Nuremberg, the SUSE admins told us to use salt for everybody
2019-09-26T15:30:54  <cboltz> now guess who often doesn't do things with salt... ;-)
2019-09-26T15:31:59  <kbabioch> yeah ... but to be honest, the salt repo needs some general love and re-factoring
2019-09-26T15:32:23  <kbabioch> basically you are the only one that is currently actively working on it, and that of course, does not scale very well (although you do a great job, of course)
2019-09-26T15:32:46  <cboltz> I know, we have too many levels of indirection in our salt code (I blame tampakrap for that ;-)
2019-09-26T15:34:04  <kbabioch> btw: coming back to status2.o.o ... what is the status of the certificate?
2019-09-26T15:34:08  <kbabioch> the monitoring is complaining about it
2019-09-26T15:34:10  <kbabioch> should we fix it?
2019-09-26T15:34:18  <kbabioch> remove the monitoring?
2019-09-26T15:34:23  <kbabioch> sync the cert from status1.o.o?
2019-09-26T15:34:43  <cboltz> as long as we don't switch over to status2, it doesn't really hurt
2019-09-26T15:35:03  <cboltz> the problem is that it wants a cert for status.o.o, which unsurprisingly fails because that name points to status1
2019-09-26T15:35:50  <kbabioch> yeah, but still i would like to get rid of the notification ... so the r"edireciton" is happening via dns cname, right?
2019-09-26T15:36:13  <kbabioch> shouldn't it be sufficient to have a cert for "status.opensuse.org" on boht hosts ... and don't worry about status{1,2}.o.o?
2019-09-26T15:36:20  <kbabioch> at least for the certificate it should be good enough
2019-09-26T15:36:26  <cboltz> right, and that cname points to status1 nearly always
2019-09-26T15:37:03  <cboltz> that, or sync the *.o.o certificate to it
2019-09-26T15:43:47  <kbabioch> ok, so asked differently: do we actually need the "status1.o.o" and/or "status2.o.o" san in the certificates for anything?
2019-09-26T15:44:02  <kbabioch> otherwise i would change it to "status.o.o" only and sync it via a hook
2019-09-26T15:44:25  <cboltz> for our users, status.o.o is enough
2019-09-26T15:44:42  <kbabioch> and for any other service?
2019-09-26T15:45:00  <cboltz> for testing Cachet updates (on the currently-unused server) it's helpful to access it as status2.o.o
2019-09-26T15:45:22  <cboltz> but that's the only reason to have status1 and status2 in the certificate
2019-09-26T15:45:53  <kbabioch> hm, and would it be okay to change it anyway ... for testing you can still overrid and/or access it without tls (using ssh forwarding, etc.)
2019-09-26T15:46:00  <kbabioch> because this will make it simpler
2019-09-26T15:46:23  <cboltz> I'm a fan of "simple", so go ahead ;-)
2019-09-26T15:46:29  <kbabioch> alternatively we can also use crtmgr.infra.opensuse.org and request the certificates via dns
2019-09-26T15:46:35  <kbabioch> but this will be more complicated :-)
2019-09-26T15:46:43  <cboltz> exactly
2019-09-26T15:47:04  <cboltz> for testing, I know how to add status.o.o to /etc/hosts (actually to my local unbound, but that's a technical detail)
2019-09-26T15:47:40  <kbabioch> on a personal note: i don't quite like to expose details like "status1" and/or "status2" to the public (i.e. via the certificate) ... this should be an implementation detail and ther might be more machines coming, etc.
2019-09-26T15:48:24  <cboltz> it doesn't hurt IMHO, but I get your point
2019-09-26T15:49:39  <kbabioch> okay, thanks for your explanation ... will work on something based on this
2019-09-26T15:49:48  <kbabioch> at least i know understand more about the setup and ideas behind it
2019-09-26T15:50:31  <cboltz> there's another interesting detail:
2019-09-26T15:51:04  <cboltz> status1 and status2 are completely independent from each other, which also means they don't sync their databases
2019-09-26T15:51:27  <cboltz> most events/downtimes are probably in the status1 database
2019-09-26T15:52:38  <kbabioch> is there value in syncing them?
2019-09-26T15:53:29  <cboltz> it would be nice to have a full history of all our downtimes etc. in one database - but that's only "nice to have", not a requirement
2019-09-26T15:53:41  <cboltz> having both completely independent is more important IMHO
2019-09-26T15:55:26  <cboltz> (for completeness: users and subscriptions also get out of sync, but that's also not the end of the world)
2019-09-26T16:07:12  * Pharaoh_Atem waves
2019-09-26T16:23:21  *** knurpht <knurpht!~knurpht@opensuse/member/Knurpht> has joined #opensuse-admin
2019-09-26T16:39:41  *** knurpht <knurpht!~knurpht@opensuse/member/Knurpht> has quit IRC (Quit: Konversation terminated!)
2019-09-26T16:49:19  *** jadamek2 <jadamek2!~julienA@176.167.83.91> has joined #opensuse-admin
2019-09-26T16:53:18  *** jadamek <jadamek!~julienA@176.164.202.125> has quit IRC (Ping timeout: 265 seconds)
2019-09-26T16:53:47  *** lcp <lcp!~lcp@opensuse/member/hellcp> has quit IRC (Ping timeout: 265 seconds)
2019-09-26T17:02:07  *** jadamek2 <jadamek2!~julienA@176.167.83.91> has quit IRC (Ping timeout: 245 seconds)
2019-09-26T17:44:23  *** malcolmlewis <malcolmlewis!~malcolmle@opensuse/member/malcolmlewis> has joined #opensuse-admin
2019-09-26T17:54:55  *** moozaad <moozaad!~moozaad@opensuse/member/moozaad> has quit IRC (Quit: Konversation terminated!)
2019-09-26T18:27:24  *** moozaad <moozaad!~moozaad@opensuse/member/moozaad> has joined #opensuse-admin
2019-09-26T18:45:18  *** moozaad <moozaad!~moozaad@opensuse/member/moozaad> has quit IRC (Quit: Konversation terminated!)
2019-09-26T19:34:50  *** moozaad <moozaad!~moozaad@opensuse/member/moozaad> has joined #opensuse-admin
2019-09-26T20:58:46  *** moozaad <moozaad!~moozaad@opensuse/member/moozaad> has quit IRC (Quit: Konversation terminated!)
2019-09-26T22:16:27  *** cboltz <cboltz!~cb@opensuse/member/Cboltz> has quit IRC ()