2018-04-26T02:12:56 *** plusky has quit IRC 2018-04-26T02:12:56 *** malcolmlewis has quit IRC 2018-04-26T02:13:53 *** plusky has joined #opensuse-admin 2018-04-26T02:16:59 *** dmacvicar[m] has quit IRC 2018-04-26T02:17:05 *** okurz has quit IRC 2018-04-26T02:17:44 *** aboe[m] has quit IRC 2018-04-26T02:17:44 *** okurz[m] has quit IRC 2018-04-26T02:18:36 *** okurz has joined #opensuse-admin 2018-04-26T02:23:48 *** malcolmlewis has joined #opensuse-admin 2018-04-26T02:33:15 *** malcolmlewis has quit IRC 2018-04-26T02:52:58 *** Son_Goku has joined #opensuse-admin 2018-04-26T03:00:50 *** malcolmlewis has joined #opensuse-admin 2018-04-26T03:00:50 *** malcolmlewis has joined #opensuse-admin 2018-04-26T03:23:44 *** Fraser_Bell has joined #opensuse-admin 2018-04-26T03:23:44 *** Fraser_Bell has joined #opensuse-admin 2018-04-26T03:28:00 *** Fraser_Bell has quit IRC 2018-04-26T03:33:02 *** nicolasbock has quit IRC 2018-04-26T03:51:40 *** Son_Goku has quit IRC 2018-04-26T04:08:11 *** lcp has joined #opensuse-admin 2018-04-26T05:02:00 *** lnussel has quit IRC 2018-04-26T05:14:54 *** Son_Goku has joined #opensuse-admin 2018-04-26T05:27:04 *** Son_Goku has quit IRC 2018-04-26T05:29:13 *** Son_Goku has joined #opensuse-admin 2018-04-26T05:38:48 *** Son_Goku has quit IRC 2018-04-26T05:40:36 *** Son_Goku has joined #opensuse-admin 2018-04-26T05:50:47 *** Son_Goku has quit IRC 2018-04-26T05:57:30 *** mcaj has joined #opensuse-admin 2018-04-26T06:26:23 *** maxlin_ has joined #opensuse-admin 2018-04-26T06:26:58 *** maxlin has quit IRC 2018-04-26T06:28:50 *** maxlin_ has quit IRC 2018-04-26T06:29:14 *** maxlin_ has joined #opensuse-admin 2018-04-26T06:35:59 *** maxlin_ is now known as maxlin 2018-04-26T06:53:28 *** mcaj has quit IRC 2018-04-26T06:54:31 *** duncanmv has joined #opensuse-admin 2018-04-26T06:58:07 *** deneb_alpha has joined #opensuse-admin 2018-04-26T06:58:08 *** deneb_alpha has joined #opensuse-admin 2018-04-26T07:14:18 *** ancorgs has joined #opensuse-admin 2018-04-26T07:14:30 *** cthugha has joined #opensuse-admin 2018-04-26T07:18:35 *** ldevulder has quit IRC 2018-04-26T07:26:48 *** a-865k has quit IRC 2018-04-26T07:27:39 *** maxlin_ has joined #opensuse-admin 2018-04-26T07:28:37 *** maxlin has quit IRC 2018-04-26T07:39:14 *** maxlin_ is now known as maxlin 2018-04-26T07:53:46 *** duncanmv has quit IRC 2018-04-26T08:05:52 *** duncanmv has joined #opensuse-admin 2018-04-26T08:08:42 *** mcaj_nb has joined #opensuse-admin 2018-04-26T08:17:49 *** mcaj_nb has quit IRC 2018-04-26T08:29:05 *** maxlin_ has joined #opensuse-admin 2018-04-26T08:30:04 *** maxlin has quit IRC 2018-04-26T08:30:21 *** Son_Goku has joined #opensuse-admin 2018-04-26T08:30:52 *** maxlin_ is now known as maxlin 2018-04-26T08:33:04 *** mcaj_nb has joined #opensuse-admin 2018-04-26T08:38:51 *** Son_Goku has quit IRC 2018-04-26T08:41:24 *** sven15 has joined #opensuse-admin 2018-04-26T08:44:08 *** duncanmv has quit IRC 2018-04-26T08:52:27 *** aboe[m] has joined #opensuse-admin 2018-04-26T08:52:59 *** matthias_bgg has joined #opensuse-admin 2018-04-26T08:58:43 *** duncanmv has joined #opensuse-admin 2018-04-26T09:05:17 *** mcaj_nb has quit IRC 2018-04-26T09:20:32 *** mcaj_nb has joined #opensuse-admin 2018-04-26T09:25:04 *** mcaj_nb has quit IRC 2018-04-26T09:26:07 *** a-865 has joined #opensuse-admin 2018-04-26T09:29:32 *** maxlin_ has joined #opensuse-admin 2018-04-26T09:30:19 *** maxlin has quit IRC 2018-04-26T09:31:53 *** maxlin_ is now known as maxlin 2018-04-26T09:33:09 *** a-865 is now known as a-865k 2018-04-26T09:36:58 *** maxlin_ has joined #opensuse-admin 2018-04-26T09:39:12 *** maxlin has quit IRC 2018-04-26T09:39:44 *** aboe[m] has quit IRC 2018-04-26T09:52:07 *** maxlin_ is now known as maxlin 2018-04-26T10:12:09 *** cboltz has joined #opensuse-admin 2018-04-26T10:12:09 *** cboltz has joined #opensuse-admin 2018-04-26T10:12:40 *** Son_Goku has joined #opensuse-admin 2018-04-26T10:22:28 *** aboe[m] has joined #opensuse-admin 2018-04-26T10:26:42 *** Son_Goku has quit IRC 2018-04-26T10:33:00 *** nicolasbock has joined #opensuse-admin 2018-04-26T10:51:03 *** Son_Goku has joined #opensuse-admin 2018-04-26T11:18:54 *** okurz[m] has joined #opensuse-admin 2018-04-26T11:19:00 *** dmacvicar[m] has joined #opensuse-admin 2018-04-26T12:04:18 *** cboltz has quit IRC 2018-04-26T12:05:59 *** matthias_bgg has quit IRC 2018-04-26T12:17:28 *** cthugha is now known as ldevulder 2018-04-26T12:48:18 *** matthias_bgg has joined #opensuse-admin 2018-04-26T13:04:36 eyevan: interestingly, locally, I am not seeing memory problems 2018-04-26T13:10:36 why did memory went down at 14:58:22 2018-04-26T13:12:22 duncanmv, I have no clue why the issue isn't reproducible locally. 2018-04-26T13:14:36 it was for me with no caching classes 2018-04-26T13:15:00 eyevan: what I am trying to do now is to get the value of GC.stat[:heap_eden_pages] / GC.stat[:heap_sorted_length] 2018-04-26T13:15:24 and get an idea of the fragmentation, to see if setting MALLOC_ARENA_MAX could help 2018-04-26T13:15:46 but I don't have fresh knowledge on what is the best way of debugging live without touching much stuff 2018-04-26T13:16:17 I was thinking to add some code that allows to start production with rbtrace enabled if an env variable is set 2018-04-26T13:16:42 that way I could edit /etc/software_opensuse_org.conf , set an env variable, restart and run ruby code live 2018-04-26T13:16:48 would you agree with such change? 2018-04-26T13:17:38 good morning everyone :-) 2018-04-26T13:20:21 * duncanmv tries MALLOC_ARENA_MAX without any evidence 2018-04-26T13:53:46 duncanmv, my idea was to look at the access logs and see which URLs are being accessed 2018-04-26T13:54:15 duncanmv, maybe there's some crazy search bot that goes through the package search, and maybe that leads to increased memory usage (or something like that). 2018-04-26T13:55:12 duncanmv, can we do some testing on the live instance? I.e., restart and then hit the / route several thousand times to see how that affects the memory. 2018-04-26T14:00:56 *** sven15 has quit IRC 2018-04-26T14:02:14 *** sven15 has joined #opensuse-admin 2018-04-26T14:12:26 eyevan: heh, the live instance gets ~10M requests/month, that is 236 req/min or 3.8/sec :-) 2018-04-26T14:12:34 eyevan: how do you expect to isolate your experiment? 2018-04-26T14:13:21 duncanmv, where's the number from? 2018-04-26T14:14:02 eyevan, it's probably not evenly distributed throughout the day, so in theory it could be isolatable :-) 2018-04-26T14:14:27 (I'm talking to myself, duh :D) 2018-04-26T14:14:54 eyevan: I did some log analysis when I tried to sell software.o.o to Microsoft ;-) 2018-04-26T14:15:16 it has 55k visits a month IIRC 2018-04-26T14:15:39 duncanmv, the logs right now are shared between multiple vhosts 2018-04-26T14:15:49 if you look a journalctl -u software_opensuse_org you can see it live 2018-04-26T14:15:56 -f 2018-04-26T14:16:14 ah but those are only puma 2018-04-26T14:16:43 well, puma's fine 2018-04-26T14:17:01 as long as it is the correct vhost 2018-04-26T14:17:26 eyevan: I added MALLOC_ARENA_MAX=2 to /etc/software_opensuse_org.conf 2018-04-26T14:17:50 it helped with fragmentation, everything is smaller now, but it still grows and grows 2018-04-26T14:18:35 eyevan: is there away that cache.classes is not working even if activated? 2018-04-26T14:18:50 duncanmv, let's fire up the production rails console and check 2018-04-26T14:19:13 it should be the same for the server and for the console. 2018-04-26T14:19:58 duncanmv, oh, by the way 2018-04-26T14:20:24 duncanmv, ExecStart=/usr/bin/bundle.ruby2.4 exec puma -e production <---- that might not be enough 2018-04-26T14:21:58 duncanmv, take a look here: https://github.com/openSUSE/software-o-o/blob/32ec8ac905cf45b1b91e917e16dc0344ffa05c2d/config/puma.rb#L9 2018-04-26T14:22:07 it expects RACK_ENV to be set as well 2018-04-26T14:22:16 if it's not set puma is running in development mode? :D 2018-04-26T14:25:37 duncanmv, it seems that systemd service is not in github repo. Is it tracked by the RPM package at all? 2018-04-26T14:25:46 *systemd service file 2018-04-26T14:32:59 yes 2018-04-26T14:33:29 eyevan: the systemd service has an EnvironmentFile directive 2018-04-26T14:33:32 pointing to /etc/software_opensuse_org.conf 2018-04-26T14:33:41 RAILS_ENV is defined there 2018-04-26T14:34:04 duncanmv, yeah, but it's not RAILS. 2018-04-26T14:34:11 duncanmv, it should be RACK_ENV 2018-04-26T14:35:00 duncanmv, let's add RACK_ENV=production to that file and restart? 2018-04-26T14:35:04 wait 2018-04-26T14:35:08 let me check something 2018-04-26T14:35:32 ExecStart=/usr/bin/bundle.ruby2.4 exec puma -e production 2018-04-26T14:35:37 that is in the service 2018-04-26T14:35:57 that is also rack related? 2018-04-26T14:36:18 should I try it, or will you try it? 2018-04-26T14:37:34 duncanmv, there's a possibility that `-e production` doesn't set RACK_ENV. I would suggest to add RACK_ENV=production to the config and see if it helps, it's a quick experiment to do. 2018-04-26T14:37:47 duncanmv, yeah, go ahead. 2018-04-26T14:38:10 hold on, I need to understand what I am doing 2018-04-26T14:38:26 puma config expects RAKE_ENV variable. 2018-04-26T14:38:36 I'm not sure if `puma -e production` sets it. 2018-04-26T14:38:45 it might be there from some ancient times. 2018-04-26T14:38:58 so why do we start the app with puma in systemd? 2018-04-26T14:39:02 and not with rails s? 2018-04-26T14:39:29 because puma -e sets RACK_ENV not RAILS_ENV 2018-04-26T14:39:30 in my tests in the GitHub PR I was testing with both RAILS_ENV=production and RAKE_ENV=production, and I had very nice memory situation :D 2018-04-26T14:39:41 but rails -e is RAILS_ENV afaik 2018-04-26T14:39:48 duncanmv, that's a good question. 2018-04-26T14:39:59 but RAILS_ENV is present in /etc/software_opensuse_org.conf 2018-04-26T14:40:10 yes, it is passed to puma then 2018-04-26T14:40:51 yeah, but RAKE_ENV status is still unknown. I'm not sure if `puma -e production` sets it. 2018-04-26T14:42:22 -e sets rack env 2018-04-26T14:42:26 -e, --environment ENVIRONMENT The environment to run the Rack app on (default development) 2018-04-26T14:42:54 but what is the difference of running puma vs running rails s 2018-04-26T14:44:24 well, rails should provide rack config so that puma can start it 2018-04-26T14:45:13 then, without touching anything yet, there is nothing strange 2018-04-26T14:45:20 RAILS_ENV is set, RACK_ENV is set 2018-04-26T14:46:03 I'm not sure that `puma -e production` will actually set ENV['RAKE_ENV'] 2018-04-26T14:46:25 and the config file specifically looks at the ENV, and not at the command line option. 2018-04-26T14:47:09 ENV variable != command line option 2018-04-26T14:47:32 I set it into the config file, and restarted 2018-04-26T14:47:41 oh, OK 2018-04-26T14:48:01 let's see what's going on with the memory then 2018-04-26T14:48:30 I will check again after my 5pm call 2018-04-26T14:58:32 eyevan: I have another clue 2018-04-26T14:58:39 duncanmv, yeah, it still seems to be growing. I'd like to grab some search queries from the access logs and try to run locally and see if maybe the search causing this. 2018-04-26T14:58:46 another clue 2018-04-26T14:58:59 you can see in the journal logs, a failed allocation happens after trying to get appdata from the cache 2018-04-26T14:59:08 prepare_appdata in application_controller 2018-04-26T14:59:52 basically pre-fills the cache on every request to search controller 2018-04-26T15:00:01 but if it is there 2018-04-26T15:00:06 it still gets it from the cache 2018-04-26T15:00:13 which means a huge allocation 2018-04-26T15:00:28 I will look into that clue 2018-04-26T15:00:42 duncanmv, yeah. Worth looking into. Previously even / route was eating memory locally, I didn't look into any other routes on the site. 2018-04-26T15:00:50 *** Son_Goku has quit IRC 2018-04-26T15:00:50 probably it just needs to look if the value is in the cache first, and then fill it 2018-04-26T15:00:57 instead of trying to get it by brute force 2018-04-26T15:01:11 how much memory is allocated in memcached by the way? 2018-04-26T15:02:30 it uses the namespace: 'software', but on the dashboard it should something like 60MB total for memcached? Seems weird. 2018-04-26T15:06:39 *it shows something like 60MB 2018-04-26T15:06:50 but it appears there no memcached config shipped in the RPM 2018-04-26T15:07:04 it uses some default settings apparently? :-/ 2018-04-26T15:08:11 probably yes 2018-04-26T15:11:02 *** sven15 has quit IRC 2018-04-26T15:13:14 well, it might make sense then to allocate more memory for memcached. 2018-04-26T15:13:41 (are any other sites also using it for caching?) 2018-04-26T15:38:02 *** tigerfoot has quit IRC 2018-04-26T15:43:41 *** tigerfoot has joined #opensuse-admin 2018-04-26T15:43:41 eyevan: I don't think so 2018-04-26T15:44:09 eyevan: but that should only affect performance (going back to the network, reparse a template, etc) right? 2018-04-26T15:45:40 duncanmv, yeah, but 60MB cache -- we might not use cache at all. 2018-04-26T15:45:48 duncanmv, the memory chart looks different now 2018-04-26T15:46:01 it is at 950M 2018-04-26T15:46:15 duncanmv, yeah, but now it has very sudden and steep changes 2018-04-26T15:46:51 duncanmv, seems like it stopped leaking memory from routes like /, but now it's probably allocating memory for prepare_appdata in big chunks 2018-04-26T15:48:12 at least that's the only explanation I can come up with :D 2018-04-26T15:48:47 although no, scratch that 2018-04-26T15:48:57 I'm looking at the wrong chart :D 2018-04-26T15:49:03 vmem instead of rss 2018-04-26T15:49:17 yeah, seems like it didn't help much 2018-04-26T15:49:43 so yeah, worth a shot looking at some controllers. 2018-04-26T15:50:11 if it's something with that ActiveXML gem -- I think it wouldn't show up in rbtrace anyway. 2018-04-26T15:50:47 "gem" 2018-04-26T15:50:53 :D 2018-04-26T15:50:54 it is a piece of code copied from OBS 2018-04-26T15:51:00 I tried to refactor it out so many times 2018-04-26T15:52:11 I'm gonna see if I can replicate the issue with memory on the weekend. 2018-04-26T15:54:48 *** ancorgs has quit IRC 2018-04-26T15:56:03 *** eyevan has quit IRC 2018-04-26T16:06:48 *** cboltz has joined #opensuse-admin 2018-04-26T16:06:48 *** cboltz has joined #opensuse-admin 2018-04-26T16:13:42 *** deneb_alpha has quit IRC 2018-04-26T16:27:37 *** duncanmv has quit IRC 2018-04-26T16:33:21 *** fvogt has joined #opensuse-admin 2018-04-26T16:49:04 *** matthias_bgg has quit IRC 2018-04-26T17:01:49 *** Son_Goku has joined #opensuse-admin 2018-04-26T17:03:21 *** victorhck has quit IRC 2018-04-26T17:05:46 *** victorhck has joined #opensuse-admin 2018-04-26T17:05:46 *** victorhck has joined #opensuse-admin 2018-04-26T17:08:51 *** fvogt has quit IRC 2018-04-26T17:09:09 *** fvogt has joined #opensuse-admin 2018-04-26T17:17:09 *** fvogt has quit IRC 2018-04-26T17:18:37 *** fvogt has joined #opensuse-admin 2018-04-26T17:57:08 *** Son_Goku has quit IRC 2018-04-26T19:14:40 *** duncanmv has joined #opensuse-admin 2018-04-26T20:00:14 *** fvogt has quit IRC 2018-04-26T20:01:41 *** fvogt has joined #opensuse-admin 2018-04-26T20:05:46 *** fvogt has quit IRC 2018-04-26T20:06:09 *** fvogt has joined #opensuse-admin 2018-04-26T20:28:35 *** fvogt has quit IRC 2018-04-26T20:55:45 *** Son_Goku has joined #opensuse-admin 2018-04-26T21:02:27 *** okurz has quit IRC 2018-04-26T21:07:07 *** okurz has joined #opensuse-admin 2018-04-26T21:22:24 *** duncanmv has quit IRC 2018-04-26T21:25:12 *** tampakrap has left #opensuse-admin 2018-04-26T21:25:19 *** tampakrap has joined #opensuse-admin 2018-04-26T21:43:37 *** Son_Goku has quit IRC 2018-04-26T22:19:35 *** Son_Goku has joined #opensuse-admin 2018-04-26T22:23:48 *** cboltz has quit IRC 2018-04-26T23:20:26 *** Fraser_Bell has joined #opensuse-admin 2018-04-26T23:20:27 *** Fraser_Bell has joined #opensuse-admin 2018-04-26T23:43:42 *** Fraser_Bell has quit IRC