2017-11-13T01:48:36 *** plinnell has quit IRC 2017-11-13T02:35:41 *** dddh_ has quit IRC 2017-11-13T03:00:25 *** dddh_ has joined #opensuse-admin 2017-11-13T03:00:25 *** dddh_ has joined #opensuse-admin 2017-11-13T03:56:25 *** okurz has quit IRC 2017-11-13T03:57:37 *** okurz has joined #opensuse-admin 2017-11-13T04:18:07 PROBLEM: Hosts syslog on monitor.infra.opensuse.org - CRITICAL: Found files older than 600 minutes /var/log/opensuse/hosts//linux.log was last modified on Fri Nov 10 18:09:15 2017 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=monitor.infra.opensuse.org&service=Hosts%20syslog 2017-11-13T05:16:31 *** plinnell has joined #opensuse-admin 2017-11-13T05:33:14 *** Son_Goku has quit IRC 2017-11-13T05:34:58 *** Son_Goku has joined #opensuse-admin 2017-11-13T05:44:19 *** Son_Goku has quit IRC 2017-11-13T06:34:46 *** fvogt has joined #opensuse-admin 2017-11-13T06:45:19 *** fvogt has quit IRC 2017-11-13T06:52:06 *** tigerfoot has joined #opensuse-admin 2017-11-13T07:45:30 *** fvogt has joined #opensuse-admin 2017-11-13T07:59:55 *** tigerfoot has quit IRC 2017-11-13T08:01:27 *** tigerfoot has joined #opensuse-admin 2017-11-13T08:31:48 *** mcaj has joined #opensuse-admin 2017-11-13T09:09:15 *** sven15 has joined #opensuse-admin 2017-11-13T09:21:28 *** ldevulder has joined #opensuse-admin 2017-11-13T09:29:36 *** kl_eisbaer has joined #opensuse-admin 2017-11-13T09:29:36 *** kl_eisbaer has joined #opensuse-admin 2017-11-13T09:30:32 *** asmorodskyi has joined #opensuse-admin 2017-11-13T10:11:36 *** Son_Goku has joined #opensuse-admin 2017-11-13T10:20:55 *** Son_Goku has quit IRC 2017-11-13T10:37:15 *** Son_Goku has joined #opensuse-admin 2017-11-13T10:52:12 *** Son_Goku has quit IRC 2017-11-13T10:54:21 *** Son_Goku has joined #opensuse-admin 2017-11-13T11:03:39 *** Son_Goku has quit IRC 2017-11-13T11:11:12 *** cboltz has joined #opensuse-admin 2017-11-13T11:12:05 *** Son_Goku has joined #opensuse-admin 2017-11-13T11:12:48 *** asmorodskyi has quit IRC 2017-11-13T11:17:42 *** matthias_bgg has joined #opensuse-admin 2017-11-13T11:20:12 tampakrap: ping 2017-11-13T11:23:33 kl_eisbaer: I'm on an online training but tell me 2017-11-13T11:24:45 *** asmorodskyi has joined #opensuse-admin 2017-11-13T11:24:52 tampakrap: I'm searching for the documentation on how to setup a VPN for an openSUSE hero ;-) 2017-11-13T11:25:19 tampakrap: otherwise can I ask you (again) to create credentials for the new "monitor" user ? 2017-11-13T11:25:46 *** Son_Goku has quit IRC 2017-11-13T11:26:31 easyrsa gen-key $vpn_username nokey && easyrsa sign-req client $vpn_username 2017-11-13T11:26:43 password is at scar:/root/vpn_password 2017-11-13T11:27:14 I can do it as well yes 2017-11-13T11:30:55 kl_eisbaer: /etc/easy-rsa/pki/issued/monitor.crt /etc/easy-rsa/pki/private/monitor.key on scar 2017-11-13T12:16:53 *** Son_Goku has joined #opensuse-admin 2017-11-13T12:43:41 *** matthias_bgg has quit IRC 2017-11-13T12:44:27 *** matthias_bgg has joined #opensuse-admin 2017-11-13T12:46:21 *** cboltz has quit IRC 2017-11-13T13:08:21 tampakrap: thanks! 2017-11-13T13:11:02 tampakrap: FYI: https://progress.opensuse.org/projects/opensuse-admin-wiki/wiki/VPN#Creation-of-new-user-certificates 2017-11-13T13:27:30 *** kl_eisbaer has left #opensuse-admin 2017-11-13T13:31:11 *** matthias_bgg has quit IRC 2017-11-13T14:46:14 *** matthias_bgg has joined #opensuse-admin 2017-11-13T15:19:34 *** cboltz has joined #opensuse-admin 2017-11-13T15:19:44 *** cboltz has joined #opensuse-admin 2017-11-13T15:42:14 *** nicolasbock has joined #opensuse-admin 2017-11-13T15:47:58 *** Son_Goku has quit IRC 2017-11-13T16:06:33 *** asmorodskyi has quit IRC 2017-11-13T16:06:59 *** Son_Goku has joined #opensuse-admin 2017-11-13T16:08:25 *** kl_eisbaer has joined #opensuse-admin 2017-11-13T16:09:05 *** nicolasbock has quit IRC 2017-11-13T16:10:37 tampakrap: cboltz: a new mirrordb1 is ready - but I'm wondering, if there were plans to have 2 clusters or just one ? 2017-11-13T16:11:09 ...on the current one, we have the following interesting DBs: 2017-11-13T16:11:17 * conference_osem => new cluster 2017-11-13T16:11:24 * gcc_stats => new cluster (?) 2017-11-13T16:11:31 * mb_opensuse => new cluster 2017-11-13T16:11:44 * opensuse_discourse (???) 2017-11-13T16:11:57 * suse_hackweek => old cluster (?) 2017-11-13T16:12:03 * weblate => new cluster 2017-11-13T16:12:22 ah, and... 2017-11-13T16:12:28 * gitlab => new cluster 2017-11-13T16:12:52 anyone interested to discuss the next steps ? 2017-11-13T16:14:38 my *guess* is that mirrorbrain and download.o.o could be enough to keep a cluster busy, so a second one for "everything else" could make sense 2017-11-13T16:14:59 cboltz: no, sorry. all the DBs listed above are currently running on one cluster ;-) 2017-11-13T16:15:23 is this cluster bored or overloaded? 2017-11-13T16:15:39 cboltz: not overloaded since the latest tunings 2017-11-13T16:15:59 cboltz: but mirrordb indeed keeps it busy 2017-11-13T16:16:28 ATM I'm thinking if SUSE should sponsor 220G or just 20G for the 2nd cluster 2017-11-13T16:16:54 as - if we stay with one cluster, we don't need extra data partitions ... 2017-11-13T16:17:52 the mirrordb load sounds like a second cluster could make sense - even if "everything except mirrordb" probably won't generate noticable load 2017-11-13T16:18:14 but I have a feeling that you can judge on this much better than I can ;-) 2017-11-13T16:18:31 cboltz: I don't think the load would make trouble, just the permissions for some DBs. 2017-11-13T16:18:58 cboltz: as far as I can see, it's just the hackweek DB which might be "problematic" 2017-11-13T16:19:30 cboltz: here is where I need the GO from SUSE-IT folks on how to proceed 2017-11-13T16:20:07 if they decide to keep the hackweek DB separate (as it's more a SUSE related DB), we need to run 2 clusters anyway 2017-11-13T16:20:52 if they are fine to have this DB also maintained by heroes, we can start the discussion if it makes sense or not to run (and maintain) 2 independent clusters 2017-11-13T16:21:24 cboltz: firing up new machines is easy - keeping them secure, maintained and working is the tricky part ;-) 2017-11-13T16:22:30 did I already mention that using salt to do the setup makes this much easier? ;-) 2017-11-13T16:23:38 cboltz: "do the setup" != "keeping them secure, maintained and working" 2017-11-13T16:23:50 I know ;-) 2017-11-13T16:24:16 cboltz: or does salt deploy all the updates (incl. kernel), scans for correct configs and maybe even does VUL checks ? 2017-11-13T16:24:33 notices filles up partitions and automatically cleans up 2017-11-13T16:24:46 knows what to do if a kernel or process OOPses 2017-11-13T16:24:47 ... 2017-11-13T16:25:12 I'm happy, if you can do all of this (and more) with Salt 2017-11-13T16:25:22 for my (non-openSUSE) servers, I use salt cmd.run to install kernel updates - and doing a salt cmd.run 'init 6' to 5 or 10 machines at once is also nice ;_) 2017-11-13T16:25:31 but at the moment, I'm trying to get at least the services in a state that is useful 2017-11-13T16:25:48 I don't say salt can do everything, but it helps a lot and makes "one more machine" less painful than it would be without salt 2017-11-13T16:25:55 cboltz: IMHO this can/should be done for openSUSE, too 2017-11-13T16:26:36 cboltz: might be - but as far as I know, someone still needs the knowledge about services before he could write salt stuff, right? 2017-11-13T16:27:19 that, or two persons (one knowing the service, one knowing salt) have to do it together 2017-11-13T16:27:32 a nice side effect is that each of them learns something new ;-) 2017-11-13T16:28:16 cboltz: right 2017-11-13T16:28:44 cboltz: it just seem to be problematic to get those two persons together ;-) 2017-11-13T16:30:57 *** Son_Goku has quit IRC 2017-11-13T16:31:41 well, I already sent you an offer to salt check_zypper-ignores.txt 2017-11-13T16:32:04 cboltz: ? where? 2017-11-13T16:32:26 by mail, yesterday 17:43 2017-11-13T16:33:21 hm - looks like I missed that one 2017-11-13T16:33:42 should I resend it? 2017-11-13T16:33:45 ah - got it 2017-11-13T16:33:49 :-) 2017-11-13T16:33:53 spam folder ... 2017-11-13T16:33:56 sorry 2017-11-13T16:34:16 that's why I prefer "reject or inbox" and hate spam folders ;-) 2017-11-13T16:36:12 cboltz: getting examples or the information how it should look like is very easy for you (-: 2017-11-13T16:36:13 ssh root@riesling.infra.opensuse.org 2017-11-13T16:36:25 /usr/lib/nagios/plugins/check_zypper -h 2017-11-13T16:36:45 Just list one patch/package per line - example: 2017-11-13T16:36:45 ah, the help output 2017-11-13T16:36:45 patch:libtiff-devel 2017-11-13T16:36:45 # comment 2017-11-13T16:36:45 package:libtiff3 2017-11-13T16:36:45 package:libtiff-devel 2017-11-13T16:36:45 # comment 2017-11-13T16:36:45 whitelist:aaa_base 2017-11-13T16:36:46 # comment 2017-11-13T16:36:46 local_package:mypackage 2017-11-13T16:36:54 I only checked the actual file, and it was empty ;-) 2017-11-13T16:37:55 getting the files from all machines is something for salt ;-) 2017-11-13T16:38:14 I guess even any hero can do it via a ssh loop 2017-11-13T16:38:27 (the file belongs to root, but is normally readable by anyone) 2017-11-13T16:39:21 salt \* cmd.run 'cat /etc/nagios/check_zypper-ignores.txt ' on the saltmaster is probably faster ;-) 2017-11-13T16:39:41 well, my ssh-agent is also fast ;-) 2017-11-13T16:40:11 yes, but I'd have to accept the host key for maybe 20 hosts where I never logged in before ;-) 2017-11-13T16:40:41 cboltz: oh ;-) => just use the monitoring server :) 2017-11-13T16:41:13 ;-) 2017-11-13T16:41:38 but anyway - the good thing with the monitoring is: 2017-11-13T16:42:05 * the /etc/xinetd.d/check_mk and the /etc/xinetd.d/nrpe can be simple templates 2017-11-13T16:42:24 where just the "bind" address should be the one from the local machine 2017-11-13T16:42:54 * for the firewall, we could discuss if it's ok to open the two ports up to the whole internal network or just the monitoring server 2017-11-13T16:43:39 ...and for the special checks, that are executed via check_mk or nrpe: those are all single files for specific checks, so just a file deployment at the right position and you are done 2017-11-13T16:44:19 so it's very simliar to ntp.conf for example 2017-11-13T16:53:18 hmm, ssh connection to mufasa timed out, and it doesn't respond to ping 2017-11-13T16:53:27 is it intentionally down? 2017-11-13T16:54:14 same question for rafiki 2017-11-13T16:54:18 cboltz: something I can not tell, as mufasa is not monitored 2017-11-13T16:56:02 tampakrap: any idea why mufasa and rafiki are down? 2017-11-13T16:58:51 *** fvogt has quit IRC 2017-11-13T16:59:27 more interesting findings: 2017-11-13T17:00:25 gaston -> no route to host 2017-11-13T17:00:53 conference and mickey ask for my password, which means auto-importing the ssh key from freeipa doesn't work there 2017-11-13T17:01:10 minnie gives Permission denied (publickey), probably for a similar reason 2017-11-13T17:01:42 redmine: connection closed by remote host 2017-11-13T17:02:15 donald is NXDOMAIN, but listed in pillar/id 2017-11-13T17:02:44 and finally monitor and simba don't have a /etc/nagios/check_zypper-ignores.txt 2017-11-13T17:03:16 on the positive side, I now have a nice file listing all the ignores from most hosts we have in salt 2017-11-13T17:05:17 the salt code is easy - http://paste.opensuse.org/41805415 (not tested yet) 2017-11-13T17:05:45 the only thing that will take some time is putting all the packages etc. to ignore to the pillars - and ideally do this by role, not by hostname 2017-11-13T17:07:15 * tampakrap reads backlog 2017-11-13T17:09:25 kl_eisbaer: I would say separate db clusters for vlan42 and vlan47. On vlan42 there are I think two-three apps (openqa, maybe also software?) that run local postgres instances 2017-11-13T17:09:55 cboltz: mufasa I don't know, rafiki intentionally because it wasn't fully set up and lacks firewall 2017-11-13T17:10:02 *** tigerfoot has quit IRC 2017-11-13T17:10:11 gaston and donald are in the old vlan, if they are still in salt remove them please 2017-11-13T17:10:51 conference, redmine and every other sle11 machine still doesn't work with the freeipa keys. I think because of the order of the options in /etc/nsswitch.conf, but I never had time to debug 2017-11-13T17:11:18 mickey doesn't work because I was experimenting there with pam_ldap as replacement of pam_sss as it is lighter 2017-11-13T17:11:27 and mickey also I don't know, let's see 2017-11-13T17:11:37 minnie* I meant 2017-11-13T17:12:10 cboltz: mufasa is up, ssh root@proxy-prv1.opensuse.org 2017-11-13T17:13:10 can I have your ssh -vvv from minnie please? 2017-11-13T17:13:28 tampakrap: ok - I will plan with a shiny new 2 node cluster, than 2017-11-13T17:13:41 * kl_eisbaer is opening tickets now 2017-11-13T17:15:07 tampakrap: ssh -vvv minnie: http://paste.opensuse.org/24d3e0ad 2017-11-13T17:18:12 really no idea 2017-11-13T17:18:14 minnie:~ # /usr/local/bin/fetch_freeipa_ldap_sshpubkey.sh cboltz 2017-11-13T17:18:18 this returns your key fine 2017-11-13T17:18:24 all the configs are in sync 2017-11-13T17:19:52 it works for my account btw 2017-11-13T17:21:41 tampakrap: restarting the sshd on minnie with debug logging enabled ? 2017-11-13T17:23:36 cboltz: what's the ssh key you are normally using ? 2017-11-13T17:23:46 *** ldevulder has quit IRC 2017-11-13T17:23:50 I just see "no such file" errors in your paste output 2017-11-13T17:24:17 *** tigerfoot has joined #opensuse-admin 2017-11-13T17:24:20 debug logging started, cboltz try again please 2017-11-13T17:24:58 cboltz: maybe you have different user logins for openSUSE machines and local ? - and just forgot to adapt your .ssh/config ? 2017-11-13T17:29:31 *** mcaj has quit IRC 2017-11-13T17:41:43 I only have one key - ~/.ssh/id_dsa{,.pub} 2017-11-13T17:41:53 *** ldevulder has joined #opensuse-admin 2017-11-13T17:41:54 just tried again, so you should see something in the log 2017-11-13T17:42:57 my ~/.ssh/config sets the username for *.infra.o.o, but there's nothing special for minnie 2017-11-13T17:43:58 cboltz: without a look in the logs: if you explicitely try to login as cboltz, does that change something? 2017-11-13T17:44:47 ...or via the "-i ~/.ssh/id_dsa" option ? 2017-11-13T17:46:22 PROBLEM: SSH on minnie.infra.opensuse.org - connect to address 192.168.47.37 and port 22: Connection refused ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=minnie.infra.opensuse.org&service=SSH 2017-11-13T17:48:17 RECOVERY: Hosts syslog on monitor.infra.opensuse.org - OK: Tested /var/log/opensuse/hosts/ - no files older than 240 minutes found ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=monitor.infra.opensuse.org&service=Hosts%20syslog 2017-11-13T17:50:48 kl_eisbaer: login fails with cboltz@minnie, -i ~/.ssh/id_dsa, and a combination of both 2017-11-13T17:51:10 but at least the -vvv output differs slightly ;-) 2017-11-13T17:53:24 * cboltz wonders if minnie has something in the sshd config that denies dsa keys 2017-11-13T17:54:32 cboltz: might be - but I need to point you to tampakrap here. I'm currently working on security updates... 2017-11-13T17:54:55 checking 2017-11-13T17:55:47 it is identical to riesling 2017-11-13T17:56:43 that makes things interesting[tm] 2017-11-13T17:59:34 cboltz: if you add something like "-F /dev/null" to make sure that no wildcard in your config matches minnie ? 2017-11-13T17:59:35 *** ldevulder has quit IRC 2017-11-13T18:00:58 ssh -vvv cboltz@minnie.infra.opensuse.org -i ~/.ssh/id_dsa -F /dev/null -> Permission denied (publickey) 2017-11-13T18:03:49 cboltz: try now 2017-11-13T18:04:24 still doesn't work 2017-11-13T18:04:37 tampakrap: do you see something in the logs ? 2017-11-13T18:05:00 tampakrap: I'm logged in, but I don't see anything from sshd in /var/log/messages or /var/log/auth.log 2017-11-13T18:05:24 cboltz: minnie.infra.opensuse.org has address 192.168.47.37 <= is that correct for you ? 2017-11-13T18:05:58 ah, that's what causes problems :-( 2017-11-13T18:06:17 cool 2017-11-13T18:06:21 RECOVERY: SSH on minnie.infra.opensuse.org - SSH OK - OpenSSH_7.2 (protocol 2.0) ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=minnie.infra.opensuse.org&service=SSH 2017-11-13T18:06:22 tampakrap: btw: "Host logging.opensuse.org not found: 3(NXDOMAIN)" 2017-11-13T18:06:23 I'm going home 2017-11-13T18:06:26 I override that IP for my local saltmaster test setup 2017-11-13T18:06:40 tampakrap: the remote.conf on minnie is wrongly using this host 2017-11-13T18:07:11 tampakrap: I leave it as it is - but you might want to change your Salt profile to point to syslog.infra.opensuse.org instead 2017-11-13T18:07:14 sorry for not remembering that when seeing the problem :-( 2017-11-13T18:07:24 cboltz: tss... ;-) 2017-11-13T18:07:49 cboltz: at least now you know that your local salt master is secure ;-) 2017-11-13T18:08:14 rsyslog fixed on that host, added in my todo for today to put it in salt 2017-11-13T18:13:47 *** matthias_bgg has quit IRC 2017-11-13T18:21:21 tampakrap: just have a look at my Email to heroes@opensuse.org a few weeks ago ;-) 2017-11-13T18:21:53 tampakrap: especially - create an entry in /etc/hosts, please, so the machine has a valid rsyslog during boot 2017-11-13T18:22:17 have a nice evening! 2017-11-13T18:22:35 *** kl_eisbaer has left #opensuse-admin 2017-11-13T18:52:08 just wondering - does it really make sense to have a hostname in the rsyslog config + /etc/hosts entry, or would using the IP in the config file be better? 2017-11-13T18:52:13 *** fvogt has joined #opensuse-admin 2017-11-13T18:54:17 I was thinking exactly the same 2017-11-13T19:11:11 *** sven15 has quit IRC 2017-11-13T19:34:21 *** Son_Goku has joined #opensuse-admin 2017-11-13T19:46:08 cboltz: regarding your minnie issue, you can do salt-call --master=$yourmaster if it helps 2017-11-13T19:51:21 well, I'll probably override the IP via the VM's /etc/hosts instead of doing it in my local DNS server 2017-11-13T19:51:45 that fixes ssh to minnie, while the VMs will still find the local saltmaster 2017-11-13T19:51:46 so last MR on get_formulas regarding the CI runners https://gitlab.infra.opensuse.org/infra/salt/merge_requests/90 2017-11-13T19:52:12 every other change on it will be so that we can clone/pull/fetch repos locally, and update the remotes 2017-11-13T19:52:18 so we can also run the tests locally 2017-11-13T19:54:27 it is finally doing what I wanted from the beginning: git pull https://github... once, git pull https://gitlab.i.o.o the second time 2017-11-13T19:54:46 instead of pulling the github and adding the remote and checking out the production branch 2017-11-13T19:57:15 so why didn't you do that from the beginning? ;-) 2017-11-13T19:57:36 you are searching for signatures again? 2017-11-13T19:58:11 I'm _always_ searching for signatures ;-) 2017-11-13T19:58:41 I see 2017-11-13T19:58:51 I don't have a good answer to that question :( 2017-11-13T19:59:13 ;-) 2017-11-13T19:59:30 on a more serious note - !90 looks good :-) 2017-11-13T19:59:43 *** Son_Goku has quit IRC 2017-11-13T20:00:14 your thumbs up please and it's gone 2017-11-13T20:00:29 already done ;-) 2017-11-13T20:01:13 perfect 2017-11-13T20:01:24 just in case you are bored - I'm afraid there might be a bug in include_optional() 2017-11-13T20:01:28 {%- if salt['file.file_exists']('{0}/{1}/init.sls'.format(root, sls_file)) %} 2017-11-13T20:01:56 this covers whatever/init.sls - but it doesn't cover whatever.sls 2017-11-13T20:02:06 with the result that this pillar won't be included 2017-11-13T20:03:27 I don't fully get it 2017-11-13T20:03:49 how does it works for our roles then? 2017-11-13T20:03:58 *** Son_Goku has joined #opensuse-admin 2017-11-13T20:04:03 ehty are pillar/role/$role.sls 2017-11-13T20:04:07 they are not init.sls 2017-11-13T20:04:37 I tested only with existing VMs, so maybe salt just "did nothing" 2017-11-13T20:04:49 give me a minute to verify my theory... 2017-11-13T20:05:32 are you using vim btw? 2017-11-13T20:05:36 yes 2017-11-13T20:05:44 (what else?) 2017-11-13T20:05:54 I started using saltstack/salt-vim 2017-11-13T20:06:03 it is very nice 2017-11-13T20:06:43 doesn't cover plain jinja files though, haven't tried if it is supported with this or if we need a new one 2017-11-13T20:11:35 I'll have a look at it 2017-11-13T20:12:10 back to include_optional - I'm afraid you can rename it to include_never ;-) 2017-11-13T20:12:32 I tried by deleting the thumb.php symlink in my test VM, and salt did not re-create it 2017-11-13T20:12:46 if I re-add - role.{{role}} to top.sls, it works again 2017-11-13T20:12:56 *** Son_Goku has quit IRC 2017-11-13T20:14:39 so, it doesn't include the roles? 2017-11-13T20:14:41 there's also the difference of "Succeeded: 630" (with that addition to both top.sls) vs. "Succeeded: 67" (with only include_optional) 2017-11-13T20:14:51 exactly 2017-11-13T20:15:00 damn 2017-11-13T20:16:57 try this 2017-11-13T20:17:54 or salt['file.file_exists']('{0}/{1}.sls'.format(root, sls_file)) 2017-11-13T20:20:02 that would be too obvious to work - still only "Succeeded: 67" :-( 2017-11-13T20:20:57 why not to work though 2017-11-13T20:21:07 what does it return? 2017-11-13T20:21:09 I'll try it 2017-11-13T20:22:23 *** Son_Goku has joined #opensuse-admin 2017-11-13T20:22:36 *** tigerfoot has quit IRC 2017-11-13T20:24:09 *** tigerfoot has joined #opensuse-admin 2017-11-13T20:25:08 *** tigerfoot has joined #opensuse-admin 2017-11-13T20:25:13 *** Son_Goku has quit IRC 2017-11-13T20:28:05 cboltz: it works here 2017-11-13T20:29:43 {%- macro include_optional(sls_file) %} 2017-11-13T20:29:46 {%- for root in opts['pillar_roots'][saltenv] -%} 2017-11-13T20:29:48 {%- if salt['file.file_exists']('{0}/{1}/init.sls'.format(root, sls_file)) or salt['file.file_exists']('{0}/{1}.sls'.format(root, sls_file)) %} 2017-11-13T20:29:50 - {{ sls_file }} 2017-11-13T20:29:52 {% endif %} 2017-11-13T20:29:54 {%- endfor %} 2017-11-13T20:29:56 {%- endmacro %} 2017-11-13T20:30:11 you need the same on salt/macros.jinja btw 2017-11-13T20:30:20 otherwise your formula is not included I suppose 2017-11-13T20:32:07 *** Son_Goku has joined #opensuse-admin 2017-11-13T20:33:13 yes, of course 2017-11-13T20:33:27 your fix looks correct, but it still doesn't work for me :-( 2017-11-13T20:35:23 at least I found the reason now: 2017-11-13T20:35:25 {%- macro include_optional(sls_file) %} 2017-11-13T20:35:27 - opts_{{ opts['file_roots'][saltenv] }} 2017-11-13T20:35:43 (keep the remaining part of the macro unchanged) 2017-11-13T20:35:51 No matching sls found for 'opts_[]' in env 'production' 2017-11-13T20:36:11 so opts['file_roots'][saltenv] is empty for me 2017-11-13T20:36:28 what about opts['file_roots']? 2017-11-13T20:37:11 the parse error says - opts_{'production': []} 2017-11-13T20:38:01 http://paste.opensuse.org/26de0e72 <-- my /etc/salt/master.d/opensuse-production.conf 2017-11-13T20:39:00 I don't use gitfs, but that shouldn't make a real difference IMHO - and doesn't explain why production is empty 2017-11-13T20:41:03 yeah 2017-11-13T20:41:15 so what about opts? 2017-11-13T20:45:26 it has lots of keys - are you looking for something specific? 2017-11-13T20:45:54 (rendering all keys with their values fails because the syntax is too invalid ;-) 2017-11-13T20:46:59 I don't really know what I'm looking for 2017-11-13T20:47:07 we could print the output to a file 2017-11-13T20:52:29 what about a boring start - commit your addition (which looks like it should work) to git so that we see if it works on our production setup 2017-11-13T20:53:12 okay 2017-11-13T20:56:44 https://gitlab.infra.opensuse.org/infra/salt/merge_requests/91 2017-11-13T20:57:34 I doubt it will work with gitfs at all 2017-11-13T21:00:09 indeed, gitfs is another can of worms ;-) 2017-11-13T21:00:55 so I am merging it 2017-11-13T21:02:19 still only 67 states on riesling 2017-11-13T21:05:14 try it again please 2017-11-13T21:05:21 with test=True 2017-11-13T21:06:00 small improvement - 76 instead of 67 states 2017-11-13T21:06:49 so what changed? 2017-11-13T21:07:22 no real changes, just "Succeeded: 78" now while it was 67 before 2017-11-13T21:07:25 I got a small change on minnie as well 2017-11-13T21:07:38 yes I mean what are those additional ones that are found now? 2017-11-13T21:07:52 maybe the problem is the salt/macros.jinja only, while the pillar one works fine? 2017-11-13T21:07:56 although being gitfs 2017-11-13T21:09:35 good question - my "high" alias uses --state-output=changes to avoid flooding the screen, so I can't check the old state 2017-11-13T21:11:14 yep this is what happened 2017-11-13T21:11:20 the diff that I got in minnie 2017-11-13T21:11:30 comes from pillar/role/saltmaster.sls 2017-11-13T21:11:36 that's an irony because pillar allows "ignore_missing: True" since Feb 2015 - https://github.com/saltstack/salt/pull/19429 2017-11-13T21:11:53 but AFAIK nobody added that feature for salt/ yet 2017-11-13T21:13:16 try to get the value of opts in a file somewhere please 2017-11-13T21:13:21 it might give us a hint 2017-11-13T21:16:42 here we go: www.cboltz.de/tmp/dl/opts.txt 2017-11-13T21:17:38 grabbed with a file.managed with a template file that contains {{ opts }} 2017-11-13T21:19:11 I got it as well by just printing it on the macro 2017-11-13T21:20:27 I will set up a copy of minnie, so we can work on it with gitfs 2017-11-13T21:23:12 does the top file get parsed on the master or on the minion? 2017-11-13T21:23:33 from the behaviour we see, I'd guess it happens on the minion :-/ 2017-11-13T21:25:30 I suppose on the minion 2017-11-13T21:25:37 not entirely sure though 2017-11-13T21:26:19 that would explain our problem - there are no file_roots defined on the minion 2017-11-13T21:30:38 and pillar_roots is? 2017-11-13T21:31:01 :q 2017-11-13T21:31:11 sorry, wrong window 2017-11-13T21:31:30 'pillar_roots': {'base': ['/srv/pillar', '/srv/spm/pillar']} 2017-11-13T21:31:51 that looks like the default config - my config has /home/cb/susebeta/salt/pillar 2017-11-13T21:32:54 so I suppose it does some magic when using git_pillar 2017-11-13T21:35:40 I found a solution for salt/ - but I'm not sure if I like it... 2017-11-13T21:35:43 {%- if salt['file.file_exists']('/var/cache/salt/minion/files/production/{0}.sls'.format(sls_file)) %} 2017-11-13T21:39:26 yes there should be a variable for this 2017-11-13T21:42:07 {%- if salt['file.file_exists']('{0}/files/{1}/{2}.sls'.format(opts['cachedir'], opts['environment'], sls_file)) %} 2017-11-13T21:42:17 better, but I'm still not sure if I like it 2017-11-13T21:43:16 opts['environment'] should be saltenv like before 2017-11-13T21:43:19 did you try it already? 2017-11-13T21:52:13 yes, saltenv also works 2017-11-13T21:52:14 now the question is if we really want to rely on the cachedir layout of salt (no idea if it's documented and frozen, or if it changes tomorrow), or if we go back to having empty files 2017-11-13T21:52:50 *** Son_Goku has quit IRC 2017-11-13T21:54:46 I would say let's try this first 2017-11-13T21:54:59 I suppose the pillar_roots that we are using has the cachedir as well there 2017-11-13T21:55:31 opts[] includes the _expanded_ pillar data 2017-11-13T21:55:42 which means it gets rendered on the server 2017-11-13T21:56:06 so I only need to commit a fix for include_optional() in salt/ 2017-11-13T21:58:24 https://gitlab.infra.opensuse.org/infra/salt/merge_requests/92 2017-11-13T21:59:34 *** okurz[m] has quit IRC 2017-11-13T21:59:53 *** aboe[m] has quit IRC 2017-11-13T22:00:46 *** Son_Goku has joined #opensuse-admin 2017-11-13T22:05:42 *** aboe[m] has joined #opensuse-admin 2017-11-13T22:09:12 deployed 2017-11-13T22:09:15 much better now 2017-11-13T22:10:35 indeed, riesling is back to "Succeeded: 638" - looks good :-) 2017-11-13T22:10:57 and the openSUSE:infrastructure:wiki repo got its / added 2017-11-13T22:11:16 that's also good, because this is done in the role.wiki pillar 2017-11-13T22:13:56 *** Son_Goku has quit IRC 2017-11-13T22:17:40 *** Son_Goku has joined #opensuse-admin 2017-11-13T22:17:48 nice 2017-11-13T22:21:29 *** Son_Goku has quit IRC 2017-11-13T22:24:00 *** Son_Goku has joined #opensuse-admin 2017-11-13T22:25:20 *** fvogt has quit IRC 2017-11-13T22:25:46 *** fvogt has joined #opensuse-admin 2017-11-13T22:33:57 *** Son_Goku has quit IRC 2017-11-13T22:37:37 *** Son_Goku has joined #opensuse-admin 2017-11-13T22:46:16 *** fvogt has quit IRC 2017-11-13T22:49:00 *** Son_Goku has quit IRC 2017-11-13T22:51:48 *** okurz[m] has joined #opensuse-admin 2017-11-13T22:53:08 *** Son_Goku has joined #opensuse-admin 2017-11-13T23:29:42 *** cboltz has quit IRC