2018-06-25T00:05:29  *** Son_Goku has quit IRC
2018-06-25T00:23:36  *** Son_Goku has joined #opensuse-admin
2018-06-25T00:46:12  *** Son_Goku has quit IRC
2018-06-25T01:14:41  *** Son_Goku has joined #opensuse-admin
2018-06-25T03:22:39  *** Son_Goku has quit IRC
2018-06-25T03:51:45  *** Son_Goku has joined #opensuse-admin
2018-06-25T04:00:11  *** Son_Goku has quit IRC
2018-06-25T04:11:38  <heroes-bot> PROBLEM: HTTP rsync.o.o on widehat.opensuse.org - HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 714 bytes in 0.004 second response time ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=widehat.opensuse.org&service=HTTP%20rsync.o.o
2018-06-25T04:13:19  *** Fraser_Bell has joined #opensuse-admin
2018-06-25T04:15:53  *** Fraser_Bell has left #opensuse-admin
2018-06-25T04:42:54  *** Son_Goku has joined #opensuse-admin
2018-06-25T05:16:56  *** Son_Goku has quit IRC
2018-06-25T05:21:41  *** nicolasbock has joined #opensuse-admin
2018-06-25T06:23:22  *** ldevulder has joined #opensuse-admin
2018-06-25T06:49:47  *** enavarro has joined #opensuse-admin
2018-06-25T06:52:21  *** duncanmv has joined #opensuse-admin
2018-06-25T06:55:01  *** dddh has quit IRC
2018-06-25T06:59:35  *** sven15 has joined #opensuse-admin
2018-06-25T07:21:17  *** dakechi has joined #opensuse-admin
2018-06-25T07:25:39  *** dddh has joined #opensuse-admin
2018-06-25T07:25:39  *** dddh has joined #opensuse-admin
2018-06-25T07:33:52  *** matthias_bgg has joined #opensuse-admin
2018-06-25T07:37:26  *** mcaj has joined #opensuse-admin
2018-06-25T08:17:31  *** enavarro has quit IRC
2018-06-25T08:21:05  *** enavarro has joined #opensuse-admin
2018-06-25T09:03:44  *** lcp has quit IRC
2018-06-25T09:52:52  <heroes-bot> PROBLEM: PerfData on monitor.infra.opensuse.org - CRITICAL: Found files older than 2880 minutes /var/lib/pnp4nagios/perfdata//proc_Rsyslog_Process.xml was last modified on Thu Jun 21 09:42:54 2018 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=monitor.infra.opensuse.org&service=PerfData
2018-06-25T09:59:35  *** Son_Goku has joined #opensuse-admin
2018-06-25T10:07:56  *** Son_Goku has quit IRC
2018-06-25T10:08:55  *** sven15 has quit IRC
2018-06-25T10:11:06  *** cboltz has joined #opensuse-admin
2018-06-25T10:11:06  *** cboltz has joined #opensuse-admin
2018-06-25T10:11:08  *** Son_Goku has joined #opensuse-admin
2018-06-25T10:20:55  *** sven15 has joined #opensuse-admin
2018-06-25T10:25:20  *** Son_Goku has quit IRC
2018-06-25T11:01:25  *** Son_Goku has joined #opensuse-admin
2018-06-25T11:29:26  *** dakechi has quit IRC
2018-06-25T11:40:08  <cboltz> FYI: I'm just installing a kernel update on riesling and water, which means the wiki will be down for a minute soon
2018-06-25T11:41:23  *** dakechi has joined #opensuse-admin
2018-06-25T11:43:34  *** matthias_bgg has quit IRC
2018-06-25T11:44:12  *** matthias_bgg has joined #opensuse-admin
2018-06-25T11:44:24  <cboltz> done :-)
2018-06-25T12:10:14  *** cboltz has quit IRC
2018-06-25T13:07:01  *** dddh_ has joined #opensuse-admin
2018-06-25T13:07:01  *** dddh_ has joined #opensuse-admin
2018-06-25T13:24:27  *** Son_Goku has quit IRC
2018-06-25T13:48:32  *** Son_Goku has joined #opensuse-admin
2018-06-25T13:53:50  *** duncanmv has quit IRC
2018-06-25T14:07:46  *** duncanmv has joined #opensuse-admin
2018-06-25T14:11:36  *** sven15 has quit IRC
2018-06-25T14:29:22  *** Son_Goku has quit IRC
2018-06-25T14:55:08  *** sven15 has joined #opensuse-admin
2018-06-25T15:02:44  *** Son_Goku has joined #opensuse-admin
2018-06-25T15:23:08  *** duncanmv has quit IRC
2018-06-25T15:32:45  *** sven15 has quit IRC
2018-06-25T15:37:17  *** duncanmv has joined #opensuse-admin
2018-06-25T15:43:44  *** lcp has joined #opensuse-admin
2018-06-25T15:51:21  *** matthias_bgg has quit IRC
2018-06-25T15:52:35  *** ldevulder_ has joined #opensuse-admin
2018-06-25T15:54:17  *** enavarro has quit IRC
2018-06-25T15:56:10  *** ldevulder has quit IRC
2018-06-25T16:00:53  *** duncanmv has quit IRC
2018-06-25T16:05:18  *** Sir_Gallantmon has joined #opensuse-admin
2018-06-25T16:06:57  *** Son_Goku has quit IRC
2018-06-25T16:08:57  *** Sir_Gallantmon is now known as Son_Goku
2018-06-25T16:36:41  *** cboltz has joined #opensuse-admin
2018-06-25T16:36:41  *** cboltz has joined #opensuse-admin
2018-06-25T17:00:37  *** Sir_Gallantmon has joined #opensuse-admin
2018-06-25T17:03:40  *** Son_Goku has quit IRC
2018-06-25T17:05:15  *** Sixth_Doctor has joined #opensuse-admin
2018-06-25T17:06:53  *** Sir_Gallantmon has quit IRC
2018-06-25T17:13:52  *** Sixth_Doctor is now known as Son_Goku
2018-06-25T17:31:08  *** matthias_bgg has joined #opensuse-admin
2018-06-25T18:01:30  *** Son_Goku has quit IRC
2018-06-25T18:04:43  *** Son_Goku has joined #opensuse-admin
2018-06-25T18:24:41  <tampakrap> cboltz: you have root on community, just ssh as root
2018-06-25T18:42:35  *** duncanmv has joined #opensuse-admin
2018-06-25T18:52:36  <cboltz> damn, so I have to fix it myself... ;-))
2018-06-25T18:52:43  <cboltz> anyway, thanks!
2018-06-25T18:54:08  <cboltz> tampakrap: I'm afraid you had the wrong server in mind - the reminder bot runs on boosters, not community
2018-06-25T18:55:32  <cboltz> can you add my ssh key there too?
2018-06-25T18:58:29  <tampakrap> ah sorry
2018-06-25T18:59:10  <tampakrap> added
2018-06-25T18:59:26  <cboltz> thanks!
2018-06-25T19:16:52  <pjessen> no luck with widehat - what am I missing?
2018-06-25T19:17:09  *** matthias_bgg has quit IRC
2018-06-25T19:21:37  <tampakrap> pjessen: try anna please
2018-06-25T19:22:32  <tampakrap> probably widehat is having only a few public IPs whitelisted and not all
2018-06-25T19:22:39  <tampakrap> and baloo has different public IP than anna
2018-06-25T19:25:10  <pjessen> tried anna too
2018-06-25T19:25:22  <pjessen> tried anna first, then elsa
2018-06-25T19:25:27  <tampakrap> and it times out?
2018-06-25T19:25:35  <pjessen> nah, asks for a password
2018-06-25T19:25:56  <tampakrap> ah
2018-06-25T19:27:35  <tampakrap> it works here
2018-06-25T19:27:40  <tampakrap> it's the same key you have on baloo
2018-06-25T19:28:31  <pjessen> for anna and elsa, I use 'pjessen', for baloo, I use 'root' ?
2018-06-25T19:28:59  <tampakrap> root for everything
2018-06-25T19:29:03  <tampakrap> for anna and widehat
2018-06-25T19:29:17  <pjessen> anna asks for a password then
2018-06-25T19:29:31  <pjessen> elsa too
2018-06-25T19:29:45  <tampakrap> ah you mean that you log in to anna/elsa as pjessen?
2018-06-25T19:29:58  <tampakrap> the user for the jumphost doesn't matter
2018-06-25T19:30:21  <tampakrap> the important part is to have your ssh agent working, to log in to jumphost with ssh -A (whatever user)
2018-06-25T19:30:26  <tampakrap> and to log in to widehat as root
2018-06-25T19:30:31  <pjessen> right, ssh-agent is running
2018-06-25T19:31:12  <pjessen> I log into anna as pjessen, then try widehat as root or pjessen. Do I need ssh-agent on anna too??
2018-06-25T19:34:27  <tampakrap> no
2018-06-25T19:34:49  <pjessen> didnt think so
2018-06-25T19:35:05  <tampakrap> then your ssh agent is not working properly
2018-06-25T19:35:19  <tampakrap> give me a few
2018-06-25T19:35:32  <pjessen> hmm. it might be old. maybe I'll try from a different machine tomorrow.
2018-06-25T19:36:39  <tampakrap> do this
2018-06-25T19:36:45  <tampakrap> ssh -A anna (whatever user)
2018-06-25T19:36:51  <tampakrap> ssh-add -L
2018-06-25T19:36:55  <tampakrap> does it show your key?
2018-06-25T19:37:41  <pjessen> The agent has no identities.
2018-06-25T19:38:51  <tampakrap> https://stackoverflow.com/questions/26505980/github-permission-denied-ssh-add-agent-has-no-identities
2018-06-25T19:39:36  <tampakrap> either you're missing the public key from your machine
2018-06-25T19:39:43  <tampakrap> or the perms of those files are wrong
2018-06-25T19:39:56  <tampakrap> the ~/.ssh dir should be 700
2018-06-25T19:40:12  <tampakrap> ~/.ssh/id_rsa should be 600
2018-06-25T19:40:21  <tampakrap> and ~/.ssh/id_rsa.pub should be 644
2018-06-25T19:40:43  <pjessen> ~/.ssh - check
2018-06-25T19:41:10  <pjessen> yes, they're good.
2018-06-25T19:41:36  <pjessen> access to anna etc works fine too.
2018-06-25T19:42:48  <pjessen> hah. got it.
2018-06-25T19:43:06  <pjessen> neeede to run ssh-add on my local machine. :-)
2018-06-25T19:43:29  <tampakrap> so your agent failed to add the keys for some reason
2018-06-25T19:43:39  <pjessen> right
2018-06-25T19:43:41  <tampakrap> or you don't use keychain to add them automatically there
2018-06-25T19:44:33  <pjessen> tbh, I dunno.
2018-06-25T19:44:53  <tampakrap> eval `keychain --eval --nogui --quiet --timeout 360 --agents ssh,gpg id_rsa`
2018-06-25T19:44:56  <pjessen> it shoud have been automatic
2018-06-25T19:44:57  <tampakrap> I have this on my .zshrc
2018-06-25T19:45:06  <tampakrap> you need the keychain package
2018-06-25T19:45:29  <pjessen> I was just going to mention it.
2018-06-25T19:46:59  <pjessen> thanks. widehat is full, I wonder if the ports change has done it.
2018-06-25T19:47:32  <pjessen> cant even list /srv/ftp
2018-06-25T19:47:39  <tampakrap> also I took a look at provo-mirror, which has full /srv
2018-06-25T19:47:49  <tampakrap> and probably it doesn't sync with --delete
2018-06-25T19:48:02  <tampakrap> because it has 15TB full, while pontifex2.i.o.o has only 12TB
2018-06-25T19:48:19  <pjessen> widehat has been saying "xfs_log_force: error -5 returned." for a while
2018-06-25T19:48:30  <tampakrap> but I have no idea where the rsync script is, and after some time I got distracted
2018-06-25T19:49:29  <pjessen> must be pushed from pontifex?  I'll look at it tmorrow.
2018-06-25T19:49:39  <tampakrap> okay
2018-06-25T19:49:52  <tampakrap> so widehat is full and with broken fs?
2018-06-25T19:50:04  <pjessen> thats what it looks like.
2018-06-25T19:50:35  <pjessen> I dont know much about xfs - dunno if xfs_log_force: 5 is bad
2018-06-25T19:51:17  <tampakrap> from a quick google search, this is a very generic message
2018-06-25T19:51:27  <tampakrap> we need to figure out the exact issue by checking more logs
2018-06-25T19:52:11  <tampakrap> one guy fixed it by rebooting the server :P
2018-06-25T19:52:40  <pjessen> haha
2018-06-25T19:52:53  <pjessen> on june22 : XFS (sda): Internal error xfs_trans_cancel at line 990 of file ../fs/xfs/xfs_trans.c.  Caller xfs_create+0x3af/0x7e0 [xfs]
2018-06-25T19:53:13  <pjessen> Corruption of in-memory data detected.  Shutting down filesystem
2018-06-25T19:53:20  <pjessen> Please umount the filesystem and rectify the problem(s)
2018-06-25T19:54:35  <pjessen> will have to wait until tomorrow. I should have time to look at it.
2018-06-25T19:54:41  <tampakrap> okay
2018-06-25T19:54:47  <cboltz> tampakrap: for provo-mirror,   crontab -u mirror   looks like what you are looking for
2018-06-25T19:56:57  <tampakrap> cboltz: it has --delete?
2018-06-25T19:57:00  <pjessen> I'll be back with an update tomorrow.
2018-06-25T19:57:26  <cboltz> the scripts are in /home/mirror/bin/ - I'm just checking them
2018-06-25T19:58:35  <cboltz> --delete --delete-after
2018-06-25T20:00:02  <heroes-bot> PROBLEM: HAProxy on elsa.infra.opensuse.org - HAPROXY CRITICAL - Active service dale is DOWN on dale proxy ! Active service gccstats is DOWN on gccstats proxy ! ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=elsa.infra.opensuse.org&service=HAProxy
2018-06-25T20:00:03  <heroes-bot> PROBLEM: HAProxy on anna.infra.opensuse.org - HAPROXY CRITICAL - Active service dale is DOWN on dale proxy ! Active service gccstats is DOWN on gccstats proxy ! ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=anna.infra.opensuse.org&service=HAProxy
2018-06-25T20:00:12  <cboltz> I just started a manual run without --delete-after so that deletion happens first
2018-06-25T20:02:13  *** Fraser_Bell has joined #opensuse-admin
2018-06-25T20:02:13  *** Fraser_Bell has joined #opensuse-admin
2018-06-25T20:04:31  <cboltz> ah, nice - it looks like the script for rsyncing tumbleweed doesn't have --delete ;-)
2018-06-25T20:27:18  *** Fraser_Bell has quit IRC
2018-06-25T20:40:07  <heroes-bot> PROBLEM: HAProxy on mufasa.infra.opensuse.org - HAPROXY CRITICAL - HTTP error: 500 Internal Server Error (Cant connect to proxy-prv1.opensuse.org:443 (Bad hostname)) ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mufasa.infra.opensuse.org&service=HAProxy
2018-06-25T20:40:37  *** Caelum has quit IRC
2018-06-25T20:41:47  *** Caelum has joined #opensuse-admin
2018-06-25T20:42:15  *** Son_Goku has quit IRC
2018-06-25T20:56:52  *** Son_Goku has joined #opensuse-admin
2018-06-25T21:16:23  *** Son_Goku has quit IRC
2018-06-25T21:34:58  <tampakrap> cboltz: any progress?
2018-06-25T21:35:19  <cboltz> I removed all --delete-after (so that only --delete remains)
2018-06-25T21:35:33  <cboltz> which should help in case rsync hits an error
2018-06-25T21:35:41  *** lcp has quit IRC
2018-06-25T21:36:01  <cboltz> (with --delete-after, it will skip deletion if it couldn't sync a file for example)
2018-06-25T21:36:37  <cboltz> there's some free space now, but getting in sync again will probably take a while ;-)
2018-06-25T21:37:08  <cboltz> actually I killed all running rsync processes and started syncing the update repos manually
2018-06-25T21:37:13  <tampakrap> cool!
2018-06-25T21:37:36  <cboltz> when this is done, I'll install all updates (provo-mirror uses the repos from localhost, so there _will_ be updates ;-)
2018-06-25T21:37:51  *** nicolasbock has quit IRC
2018-06-25T21:40:11  *** lcp has joined #opensuse-admin
2018-06-25T21:40:52  <cboltz> oh, and I fixed the postfix config to use the real server name - with a @localhost sender, mx1.suse.de didn't like the mails ;-)
2018-06-25T22:21:05  *** duncanmv has quit IRC
2018-06-25T22:29:12  *** Son_Goku has joined #opensuse-admin
2018-06-25T23:06:27  *** cboltz has quit IRC
2018-06-25T23:26:40  *** Son_Goku has quit IRC
2018-06-25T23:33:32  *** plinnell has quit IRC