Problem #1: A nagging memory problem
TroopTrack memory gets out of control every couple of days. It bumps up to 95% of available memory and stays there, triggering notifications from New Relic and slowing down traffic as TroopTrack uses swap. When this happens I will kick the web server over and life goes back to normal for a while.
Problem #2: A really slow dashboard
Yesterday I gave a TroopTrack demo to a small national scouting organization. The presentation included a demo of the backend tools I use to manage TroopTrack. I found myself apologizing for the 40 seconds it took to load the dashboard. In my embarrassment I silently chided myself for not figuring out what the problem was and fixing it.
Problem #3: A really slow help desk
I’ve recently started working on a set of features for TroopTrack I call Mister Smith. Mister Smith is my replacement for Zendesk. I have some ideas for how to make a helpdesk for SaaS that will better facilitate the sales and support process I want to have. These ideas don’t fit the models followed by Zendesk, Uservoice, etc. so I am rolling my own. I’ve already rolled out a “test” feature that lets users ask questions, but the speed of the active admin interface has been horrible, so much so that I’ve been considering dropping active admin for Mister Smith related features.
Trying to figure out problem #1 solves #2 & #3
On Tuesday I added a gem called Oink to my stack. It adds memory logging to my middleware and provides a tool for identifying which requests add the most memory to the heap. I let it sit in prod for a day, then ran it. Here are the results:
—- MEMORY THRESHOLD —-
THRESHOLD: 50 MB
— SUMMARY —
1. Dec 19 18:00:27, 655360 KB, admin/dashboard#index
2. Dec 19 05:12:47, 463092 KB, admin/questions#index
3. Dec 19 18:23:53, 394820 KB, admin/questions#index
4. Dec 19 20:33:34, 393216 KB, admin/questions#edit
5. Dec 20 04:35:28, 393216 KB, admin/dashboard#index
6. Dec 19 20:34:52, 327680 KB, admin/questions#edit
7. Dec 19 05:13:15, 323340 KB, admin/questions#edit
8. Dec 19 20:34:28, 262144 KB, admin/questions#index
9. Dec 19 04:04:51, 80156 KB, manage/users#show
10. Dec 19 12:00:58, 80156 KB, manage/manage#index
Worst requests #1-8 are problems 2 & 3. This got me thinking, and looking at what active admin was doing.
Fix #1: Not being excessive
My dashboard had a lot of stats that were counting instance that met certain requirements, like the number of troops that have an active paid subscription, etc. Loading the dashboard was running a ton of queries, a lot of which were for metrics that I thought might be useful when I added them but in practice were ignored. I cleaned those up and the page became acceptably quick.
Fix #2: Default filters for belongs_to relationships
The memory requirements for admin/questions/index confused me at first. This view uses pagination, and the table involved doesn’t have a lot of columns. You would expect it to be fast and not need much memory, but neither was true. So I watched the log and I saw a query that was something like this:
SELECT * from USERS
TroopTrack has 30,000 users and for some reason ActiveAdmin was loading every single one of them into memory. That’s when I noticed something I hadn’t even considered previously – the automatic filters on the right side of the view. Because a question has a belongs_to relationship with users, active admin was automatically adding a select filter for users by default. With 30,000 users, a drop-down list like that is not very useful anyway, so I just overrode the default filters with the ones I will actually use and boom, just like that the page loads fast and doesn’t hog the memory.
If I ever need a filter based on users then I will use some sort of autocomplete or name search instead of a dropdown, but for now just dropping the filter is enough.
Fix #3: Default forms for belongs_to relationships
If I hadn’t already figured out #2, then understanding the admin/questions/edit problem might have taken a while. I hadn’t overridden the form, so active admin was just creating default fields for every attribute on the model, including the user. I really don’t ever have any need to change the user associated with a question, so I just overrode the default form with one that doesn’t include the user field.
ActiveAdmin is awesome again!
I was starting to think that ActiveAdmin wasn’t going to scale up for me and that I would have to roll my own. I’m very glad I turned out to be stupid and wrong in this case. By simply digging in to figure out what was the real problem instead of merely chalking it up to “active admin is just slow”, I have been able to avoid what would have been a pointless and expensive development effort. Yay!
What about that memory problem?
After I realized that certain active admin pages were causing the memory problems I did a little test. After restarting the web server, I went to the trouble pages and watched the memory. Sure enough, it took off like a bat out of a hell. After I deployed the changes described above I repeated the test and memory held steady.