2017-10-05T00:31:08 *** solevi|2 has quit IRC 2017-10-05T01:21:07 PROBLEM: Fail2Ban on status1.opensuse.org - CHECK FAIL2BAN ACTIVITY - CRITICAL - 1 detected jails with 2 current banned IP(s) ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=status1.opensuse.org&service=Fail2Ban 2017-10-05T02:42:57 *** okurz has quit IRC 2017-10-05T02:43:28 *** okurz has joined #opensuse-admin 2017-10-05T02:43:45 *** nicolasbock has quit IRC 2017-10-05T04:26:49 *** Son_Goku has joined #opensuse-admin 2017-10-05T05:07:42 *** Son_Goku has quit IRC 2017-10-05T05:31:43 *** kl_eisbaer has joined #opensuse-admin 2017-10-05T05:33:35 *** kl_eisbaer1 has joined #opensuse-admin 2017-10-05T05:34:15 *** kl_eisbaer1 has left #opensuse-admin 2017-10-05T05:36:01 *** kl_eisbaer has quit IRC 2017-10-05T06:42:16 *** kl_eisbaer has joined #opensuse-admin 2017-10-05T06:42:16 *** kl_eisbaer has joined #opensuse-admin 2017-10-05T07:01:08 *** lnussel has quit IRC 2017-10-05T07:22:37 *** heroes-bot has joined #opensuse-admin 2017-10-05T07:25:07 PROBLEM: HTTP connect on boosters.infra.opensuse.org - Temporary failure in name resolution ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=boosters.infra.opensuse.org&service=HTTP%20connect 2017-10-05T07:30:18 PROBLEM: HTTP download on pontifex3.infra.opensuse.org - Temporary failure in name resolution ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=pontifex3.infra.opensuse.org&service=HTTP%20download 2017-10-05T07:36:06 PROBLEM: rsync on pontifex3.infra.opensuse.org - rsync: getaddrinfo: stage.opensuse.org 873: Temporary failure in name resolution ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=pontifex3.infra.opensuse.org&service=rsync 2017-10-05T08:41:39 *** heroes-bot has joined #opensuse-admin 2017-10-05T08:44:58 RECOVERY: HTTP connect on boosters.infra.opensuse.org - HTTP OK: HTTP/1.1 302 Found - 524 bytes in 0.004 second response time ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=boosters.infra.opensuse.org&service=HTTP%20connect 2017-10-05T08:46:02 RECOVERY: rsync on pontifex3.infra.opensuse.org - OK: Rsync is up ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=pontifex3.infra.opensuse.org&service=rsync 2017-10-05T09:41:33 *** cboltz has joined #opensuse-admin 2017-10-05T10:12:17 *** Son_Goku has joined #opensuse-admin 2017-10-05T10:24:01 *** Son_Goku has quit IRC 2017-10-05T10:59:06 *** nicolasbock has joined #opensuse-admin 2017-10-05T11:05:21 *** nicolasbock has quit IRC 2017-10-05T11:05:43 *** nicolasbock has joined #opensuse-admin 2017-10-05T11:17:53 *** Son_Goku has joined #opensuse-admin 2017-10-05T11:26:33 *** asmorodskyi has quit IRC 2017-10-05T11:28:48 *** cboltz has quit IRC 2017-10-05T11:29:15 *** asmorodskyi has joined #opensuse-admin 2017-10-05T11:33:03 *** Son_Goku has quit IRC 2017-10-05T11:37:09 *** matthias_bgg has quit IRC 2017-10-05T11:47:32 *** Son_Goku has joined #opensuse-admin 2017-10-05T11:54:44 *** solevi has quit IRC 2017-10-05T11:55:06 *** solevi has joined #opensuse-admin 2017-10-05T12:04:12 *** solevi has quit IRC 2017-10-05T12:04:36 *** Son_Goku has quit IRC 2017-10-05T12:07:52 *** solevi has joined #opensuse-admin 2017-10-05T12:22:54 *** Son_Goku has joined #opensuse-admin 2017-10-05T13:00:19 tampakrap: do you know what is running behind 130.57.72.2 ? 2017-10-05T13:00:19 2017-10-05T13:00:51 kl_eisbaer: haproxy 2017-10-05T13:00:58 your key is there btw, you can ssh to it 2017-10-05T13:00:59 130.57.72.2 => proxy-prv1.opensuse.org. 2017-10-05T13:01:02 yes, but 2017-10-05T13:01:11 Host proxy-prv1.opensuse.org not found 2017-10-05T13:01:33 BUT: 130.57.72.1 => proxy-prv.opensuse.org. 2017-10-05T13:01:48 where is the machine running ? 2017-10-05T13:01:48 ah sorry, I'll fix it 2017-10-05T13:01:56 in bryce cluster 2017-10-05T13:02:02 it's also on racktables 2017-10-05T13:02:08 you mean the hydra1 ? 2017-10-05T13:02:37 no, hydra1 will be for suse dmz machines, this is for opensuse machines (same as anna/elsa in nue) 2017-10-05T13:02:39 that machine does not have an external interface, yet, but ping 130.57.72.2 works ... 2017-10-05T13:03:06 huh? 2017-10-05T13:03:08 let me check 2017-10-05T13:03:42 The other machines I see are rafiki, mufasa and narwal4 2017-10-05T13:04:00 it has an external interface 2017-10-05T13:04:08 mufasa is proxy-prv1.opensuse.org 2017-10-05T13:04:13 ah! :-) 2017-10-05T13:04:35 ok - is the firewall adapted for those 3 IPs already? 2017-10-05T13:05:05 give me a min to make the a record and I'll explain 2017-10-05T13:06:01 ok - I'll take a coffee :-) 2017-10-05T13:06:04 so: mufasa is provo's anna, public IP 130.57.72.2, it has open 80/443 from everywhere, and 22 only from nuremberg and prague office 2017-10-05T13:06:26 ok 2017-10-05T13:06:51 rafiki is provo's daffy (login2 opensuse proxy), not set up yet, it has open 80/443 from everywhere and 22 from suse network (no local firewall) 2017-10-05T13:07:07 narwal4 has only private interface, and mufasa as gateway 2017-10-05T13:07:20 ok 2017-10-05T13:07:25 narwal4 aka provo's static.o.o is fully set up as well btw 2017-10-05T13:07:49 Just updating https://etherpad.opensuse.org/p/NUE_office_downtime 2017-10-05T13:07:56 if you put on your /etc/hosts 130.57.72.2 static.opensuse.org conncheck.opensuse.org you will see them working 2017-10-05T13:08:02 the problem is that we are losing ipv6 though 2017-10-05T13:08:39 jip, but that's something MF-IT can not provide since years :-( 2017-10-05T13:08:48 we need to include this in the announcement 2017-10-05T13:09:00 ...and obviously remove it in DNS 2017-10-05T13:09:55 next steps: try to set up rafiki / login2, set up timon.o.o as gateway/openvpn (not urgent for the downtime), set up shenzi as master freeipa 2017-10-05T13:10:15 I have the storages ready for those VMs and I am planning today to set up the VMs as well 2017-10-05T13:10:23 but for freeipa I will need darix help 2017-10-05T13:10:24 ok 2017-10-05T13:10:25 I'll ping him now 2017-10-05T13:10:40 I will setup an additional mirrordb for download.o.o 2017-10-05T13:11:19 tampakrap: what is zazu planned for ? 2017-10-05T13:11:19 https://infra.nue.suse.com/Ticket/Display.html?id=92706 2017-10-05T13:11:36 => No permission to view ticket 2017-10-05T13:11:42 okay moment 2017-10-05T13:12:12 look at the etherpad 2017-10-05T13:12:37 ok 2017-10-05T13:12:54 so I'll add the mirrordb machine there, too. Any idea about the name for this machine ? 2017-10-05T13:13:19 I'd say stick with mirrordb now as I did with narwals 2017-10-05T13:13:24 mirrordb6 2017-10-05T13:13:43 hey, common, that's toooo easy 2017-10-05T13:14:04 http://disney.wikia.com/wiki/Nala 2017-10-05T13:15:04 give me a min I have a customer here 2017-10-05T13:16:30 yep go for it! 2017-10-05T13:21:27 nala up and running on bryce1 ... 2017-10-05T13:22:43 tampakrap: can I add the new machines to the monitoring already ? 2017-10-05T13:23:06 sure, I added them also in racktables and in infra.o.o zone 2017-10-05T13:23:20 I also filed MR for salt, but cboltz wants me to work more on it 2017-10-05T13:23:35 *** Son_Goku has quit IRC 2017-10-05T13:25:15 btw: what do you think about http://paste.opensuse.org/27122754 ? 2017-10-05T13:25:46 ^^ replace the 'myusage_like_scanner' part with something useful and you get a prompt like 2017-10-05T13:25:47 olaf (scanner):/etc # 2017-10-05T13:25:48 https://gitlab.opensuse.org/infra/salt/merge_requests/62 2017-10-05T13:26:00 checking 2017-10-05T13:26:09 ^^ server not found ;-) 2017-10-05T13:26:18 needs vpn 2017-10-05T13:26:29 ah the PS1 that you have in olaf 2017-10-05T13:26:34 yes very nice, I like it 2017-10-05T13:27:14 btw, provo.mirror currently does not support https - should be put the certs there, too ? 2017-10-05T13:29:40 I don't know how it works tbh 2017-10-05T13:29:42 Cannot create reverse record for "192.168.67.4": DNS reverse zone 168.192.in-addr.arpa. for IP address 192.168.67.4 is not managed by this server 2017-10-05T13:29:56 I just copied the certs in mufasa from anna 2017-10-05T13:30:08 tampakrap: you forgot to create a reverse zone for the new IP range in Provo 2017-10-05T13:30:08 ah yes I wanted to ask that as well, how to create new zones 2017-10-05T13:30:23 DNS Zones => add 2017-10-05T13:30:35 doing it now 2017-10-05T13:30:46 ok - just remember to mark it as reverse zone ;-) 2017-10-05T13:31:43 done 2017-10-05T13:32:10 I added mufasa 2017-10-05T13:33:57 oh, Racktables tells me that 67.4 is used by rafiki already, but freeipa tells me that rafiki uses 67.3 ? 2017-10-05T13:34:10 checking 2017-10-05T13:35:20 freeipa is wrong, I'm fixing it 2017-10-05T13:35:34 ok - just give me an IP for nala 2017-10-05T13:36:18 67.6 2017-10-05T13:42:24 ok: machine is ready, waiting for the gateway now ;-) 2017-10-05T13:47:38 the internal gateway you mean? 2017-10-05T13:47:46 you can use mufasa for now 2017-10-05T13:49:03 *** cboltz has joined #opensuse-admin 2017-10-05T13:51:40 ...and as DNS? 2017-10-05T13:52:18 a - let me take google 2017-10-05T13:53:22 JFYI: I switched "my" VMs in Provo to use provo-mirror.opensuse.org directly. This avoids rountrips ;-) 2017-10-05T13:54:12 mufasa/rafiki have 8.8.8.8 as well 2017-10-05T13:54:20 okay noted, I will add that to salt 2017-10-05T14:22:14 *** matthias_bgg has joined #opensuse-admin 2017-10-05T14:26:39 tampakrap: should we create different admin images for the locations ? IMHO you want to use a different salt master later anyway, right? 2017-10-05T14:27:35 by images you mean jeos? 2017-10-05T14:27:41 tampakrap: yes 2017-10-05T14:28:16 funny that you are asking 2017-10-05T14:28:32 there is a script in the jeos image /root/bin/initial_setup.sh, I am just now improving it to support locations 2017-10-05T14:29:29 and yes we will need a local saltmaster in provo 2017-10-05T14:29:45 ok 2017-10-05T14:29:47 my question is though how are we going to make the nuremberg machines and the provo machines to talk to each other 2017-10-05T14:30:06 last time we agreed to have an openVPN tunnel between them 2017-10-05T14:31:34 so this means that scar and timon (the future openvpn server in provo) will be openvpn servers and clients to each other? 2017-10-05T14:31:41 and we will set up the routing table properly on them? 2017-10-05T14:32:03 *** Son_Goku has joined #opensuse-admin 2017-10-05T14:32:31 just wondering - why do you think we need a saltmaster in Provo? IMHO we could just let the minions access the NBG saltmaster (for example via an openVPN tunnel) 2017-10-05T14:33:19 tampakrap: that's something I leave to the VPN experts :-) (maybe there needs to be other nodes for this, I don't know atm) 2017-10-05T14:33:39 cboltz: the next downtime will come ... ;-) 2017-10-05T14:33:48 cboltz: read about syndic 2017-10-05T14:34:16 in short: minnie will be syndic, there will be another syndic in provo, and a master of masters in nuremberg that will be able to connect to all minions 2017-10-05T14:34:54 kl_eisbaer: then we'll also need to mirror the gitlab repos ;-) 2017-10-05T14:35:08 cboltz: THAT's HA :-) 2017-10-05T14:35:33 cboltz: we could of course also say that we don't need this (like having one download.o.o for all) .... 2017-10-05T14:35:33 cboltz: why? 2017-10-05T14:36:38 Lars had "the next downtime" in mind, and if you want to cover that, it would be good if the Provo saltmaster wouldn't depend on gitlab in NBG 2017-10-05T14:37:23 ah 2017-10-05T14:37:30 yeah we could mirror the git repos as well, sure 2017-10-05T14:37:38 gitolite offers very nice mirroring solutions 2017-10-05T14:43:36 ok, guys: time for me to go offline. tampakrap: JFYI: I will start with the download.o.o setup in Provo tomorrow. If you have an idea meanwhile on how to get the 3 mirrordb machines connected to each other, just let me know :-) 2017-10-05T14:44:06 sure 2017-10-05T14:44:12 tomorrow I plan to work on the MX servers 2017-10-05T14:44:26 *** asmorodskyi has quit IRC 2017-10-05T14:45:36 *** kl_eisbaer has quit IRC 2017-10-05T14:45:41 just curious (and without knowing too much about mirrorbrain) - what needs to get connected between the mirrordb machines? 2017-10-05T14:46:50 mirrordb3 and mirrordb4 were the postgresql servers used only for mirrorbrain 2017-10-05T14:46:57 also, they were the only postgresql servers 2017-10-05T14:47:19 later those servers were moved to the atreju cluster, so that we can use them for other services as well 2017-10-05T14:47:36 so while the name is still mirrordb, lars' question was how to connect three postgresql servers 2017-10-05T14:48:06 especially with one being in another location 2017-10-05T14:48:46 yeah, the replication will obviously have some lag 2017-10-05T14:52:24 gitlab is broken :/ 2017-10-05T14:52:38 GitLab: Failed to authorize your Git request: internal API unreachable 2017-10-05T14:55:19 confirmed, "git pull" is broken 2017-10-05T14:55:53 the web interface still works 2017-10-05T15:02:03 PROBLEM: HTTP gitlab on mickey.infra.opensuse.org - HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 502 Bad Gateway ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mickey.infra.opensuse.org&service=HTTP%20gitlab 2017-10-05T15:09:04 *** asmorodskyi has joined #opensuse-admin 2017-10-05T15:11:14 it has a chain of issues, it is easier to update it to 10 2017-10-05T15:17:33 *** asmorodskyi has quit IRC 2017-10-05T15:17:59 *** asmorodskyi has joined #opensuse-admin 2017-10-05T15:27:50 fixed! 2017-10-05T15:28:43 cboltz: git pull works, but git clone doesn't here 2017-10-05T15:28:47 can you confirm please? 2017-10-05T15:30:04 both work here 2017-10-05T15:30:20 tested with git pull (obviously) and git clone gitlab@gitlab.opensuse.org:infra/salt.git 2017-10-05T15:32:06 RECOVERY: HTTP gitlab on mickey.infra.opensuse.org - HTTP OK: Status line output matched 302 - 534 bytes in 4.516 second response time ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mickey.infra.opensuse.org&service=HTTP%20gitlab 2017-10-05T15:41:49 *** Son_Goku has quit IRC 2017-10-05T15:42:29 ah yes 2017-10-05T15:42:34 I should be using gitlab.infra.opensuse.org 2017-10-05T15:43:14 okay all good 2017-10-05T15:43:16 it worked for me without .infra... 2017-10-05T15:43:30 you have something in your .git/config maybe? 2017-10-05T15:44:30 no, my ~/.gitconfig contains only unrelated entries 2017-10-05T15:44:40 .ssh/config sorry 2017-10-05T15:45:22 right, I have it aliased to gitlab.infra.o.o there 2017-10-05T15:45:49 that said - it would be nice if the web interface would offer the gitlab.infra.o.o location for copy&paste 2017-10-05T15:46:18 or we can move it completely to gitlab.i.o.o and get rid of gitlab.o.o 2017-10-05T15:49:30 some weeks ago, we discussed why we have it non-public and if we should make it public - which would mean to move it the other way round 2017-10-05T15:49:55 so for now I'd say keep gitlab.o.o 2017-10-05T15:50:28 *** asmorodskyi has quit IRC 2017-10-05T15:55:18 *** Son_Goku has joined #opensuse-admin 2017-10-05T15:56:48 *** nicolasbock_ has joined #opensuse-admin 2017-10-05T15:57:52 *** Son_Goku has quit IRC 2017-10-05T16:06:53 *** Son_Goku has joined #opensuse-admin 2017-10-05T16:12:00 *** Son_Goku has quit IRC 2017-10-05T16:43:18 *** fvogt has joined #opensuse-admin 2017-10-05T17:40:26 *** dddh_ has quit IRC 2017-10-05T17:42:57 *** matthias_bgg has quit IRC 2017-10-05T18:00:23 *** Son_Goku has joined #opensuse-admin 2017-10-05T18:11:21 *** fvogt has quit IRC 2017-10-05T18:14:49 *** fvogt has joined #opensuse-admin 2017-10-05T18:35:12 *** dddh_ has joined #opensuse-admin 2017-10-05T18:35:12 *** dddh_ has joined #opensuse-admin 2017-10-05T18:43:57 *** Son_Goku has quit IRC 2017-10-05T18:46:03 *** solevi|2 has joined #opensuse-admin 2017-10-05T18:49:15 *** dddh_ has quit IRC 2017-10-05T18:58:07 *** dddh_ has joined #opensuse-admin 2017-10-05T18:58:07 *** dddh_ has joined #opensuse-admin 2017-10-05T19:34:19 *** pjessen has quit IRC 2017-10-05T19:40:11 *** fvogt has quit IRC 2017-10-05T19:44:33 *** fvogt has joined #opensuse-admin 2017-10-05T20:00:01 PROBLEM: HAProxy on elsa.infra.opensuse.org - HAPROXY CRITICAL - Active service freeipa is DOWN on freeipa proxy ! ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=elsa.infra.opensuse.org&service=HAProxy 2017-10-05T20:00:02 PROBLEM: HAProxy on anna.infra.opensuse.org - HAPROXY CRITICAL - Active service freeipa is DOWN on freeipa proxy ! ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=anna.infra.opensuse.org&service=HAProxy 2017-10-05T20:20:17 *** fvogt has quit IRC 2017-10-05T20:24:36 *** fvogt has joined #opensuse-admin 2017-10-05T20:34:37 *** fvogt has quit IRC 2017-10-05T20:34:59 *** fvogt has joined #opensuse-admin 2017-10-05T20:40:02 RECOVERY: HAProxy on elsa.infra.opensuse.org - HAPROXY OK - icc (Active: 1/1) freeipa (Active: 1/1) lists (Active: 1/1) smt (Active: 1/1) nuka (Active: 1/1) crashdb (Active: 1/1) mysql-int (Active: 1/1) connect (Active: 1/1) mickey (Active: 1/1) mirrorlist (Active: 1/1) rpmlint (Active: 1/1) studioexpress (Active: 1/1) download (Active: 1/1) redmine (Active: 1/1) status (Active: 1/1) keyserver-db (Active: 0/1) gccstats (Active: 1/1) etherpad (Active: 1/1) tarzan (Active: 1/1) keyserver (Active: 1/1) monitor (Active: 1/1) kruemel (Active: 1/1) static (Active: 3/3) conference (Active: 1/1) osccollab (Active: 1/1) keyserver-recon (Active: 0/1) riesling (Active: 1/1) community (Active: 1/1) ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=elsa.infra.opensuse.org&service=HAProxy 2017-10-05T20:40:03 RECOVERY: HAProxy on anna.infra.opensuse.org - HAPROXY OK - lists (Active: 1/1) static (Active: 3/3) keyserver (Active: 1/1) keyserver-recon (Active: 0/1) redmine (Active: 1/1) keyserver-db (Active: 0/1) mysql-int (Active: 1/1) rpmlint (Active: 1/1) conference (Active: 1/1) nuka (Active: 1/1) gccstats (Active: 1/1) kruemel (Active: 1/1) riesling (Active: 1/1) studioexpress (Active: 1/1) status (Active: 1/1) icc (Active: 1/1) osccollab (Active: 1/1) community (Active: 1/1) tarzan (Active: 1/1) connect (Active: 1/1) mickey (Active: 1/1) crashdb (Active: 1/1) download (Active: 1/1) freeipa (Active: 1/1) mirrorlist (Active: 1/1) smt (Active: 1/1) monitor (Active: 1/1) etherpad (Active: 1/1) ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=anna.infra.opensuse.org&service=HAProxy 2017-10-05T20:56:12 *** fvogt has quit IRC 2017-10-05T21:22:16 *** cboltz has quit IRC 2017-10-05T21:25:51 *** cboltz has joined #opensuse-admin 2017-10-05T21:49:11 *** solevi|2 has quit IRC 2017-10-05T22:31:29 *** marxin has quit IRC 2017-10-05T22:32:40 *** marxin has joined #opensuse-admin 2017-10-05T22:37:32 *** marxin has quit IRC 2017-10-05T23:05:07 PROBLEM: Fail2Ban on status1.opensuse.org - CHECK FAIL2BAN ACTIVITY - CRITICAL - 1 detected jails with 2 current banned IP(s) ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=status1.opensuse.org&service=Fail2Ban 2017-10-05T23:18:08 *** cboltz has quit IRC 2017-10-05T23:28:42 *** marxin has joined #opensuse-admin