The longer I do this, the more I agree with @dhh
If you haven’t seen it yet, you should watch @dhh’s talk from RailsConf2015. In it, @dhh introduces the notion that “monolithic” applications aren’t really bad in and of themselves and rebrands them as integrated systems. As I listened to @dhh talk I realized that I have grown to feel much the same way.
I’m proud of my monolith
I started building TroopTrack in 2008. I think it began life as a Rails 1.8 application, but I’m not entirely sure. We were not using git back then and the sad truth is I no longer have the repo history going back to day 1.
TroopTrack is a monolithic application. It’s huge. It has 125+ models and well over 200 controllers. The routes file is almost 700 lines. It’s really, really big.
I think it’s beautiful.
That’s not to say it’s perfect. TroopTrack has been my personal Rails education, and you can still see some of my more painful learning experiences in the code. There are parts of it that absolutely suck. I once gave an impromptu presentation I called “the worst code you ever saw” and I pulled it straight from TroopTrack. A thousand lines of pure misery, all in one file. There are pockets of code in TroopTrack that are so horrible I dread looking at them and hope I don’t pull a story related to them.
Warts aside, working on TroopTrack is fun and generally not that hard. In spite of its size, I still enjoy making changes and adding new features to TroopTrack. I don’t feel as if I am suffocating under nearly eight years of legacy code that restricts my every move and leaves me worrying about the unforeseeable side effects of every change we make. TroopTrack’s code is generally well organized, easy to understand, and fun to work with. In spite of it’s size, we can still turn on a dime and execute a new feature or a re-design of an existing feature at top speed.
For example, we recently added the ability to selectively override certain views to make them work better on a small mobile device. I investigated a few different approaches, experimented with them, picked one, then rolled out an initial release that dramatically improved our usability on mobile devices. It took me about ten hours.
About the same time, we started work on a mobile app. To support it, we need an API. So we started adding end points as the mobile developer requested them. Turns out, that was pretty easy too.
The point I’m trying to make is simple. Life with a monolith doesn’t have to be bad. @dhh doesn’t live in a world with a different set of rules than the rest of us mere mortals – if he can build a majestic monolith that is fun and easy to work with, so can I. And so can you.
Here are some things we’ve learned along the way that I think are helpful for creating an integrated system that doesn’t crush you as it grows in size and complexity.
Solve the real problems
This code base is monolithic. We need to make it less monolithic
I’ve said this. I’ve broken an application (not TroopTrack) up into a collection of engines just so it would be “less monolithic”. I’ve seen developers look at a code base and say “Ugh. Monolith”.
The problem is not that an application is monolithic. That’s like saying that a car is bad because it’s blue. Blue is just a fact. It’s not a moral imperative. The fact that a code base is big doesn’t mean it’s bad.
What we usually mean when we say that a product has become monolithic is that it frightens us. It’s big. It seems complex. It appears to be near impossible to understand everything it does. The sheer effort required to grok the code seems impossible to produce.
The first step in making a monolithic application majestic is to deal with this fear and stop blaming the monolith. Regardless of how you break a product up into pieces, a product that does a thousand different things (for good reasons) is going to take some time and effort to understand. This is not impossible. It just takes time and effort, and if you organize your code well and follow the rails way as much as you can, you can make changes to the parts without completely understanding every aspect of the whole.
Instead of blaming the monolith for problems in your product, blame the problems. Then figure out how to make them better.
Optimize for understanding
I went through a phase a few years ago where I tried to be really clever with my code. I tried to do as much as I could in a single line of code. For some reason I don’t understand, I also didn’t hit the return key very often. Maybe my text editor at the time had line wrapping or something, I don’t really know. Years later when I run into some code from my clever phase, the horizontal scrolling and long chain of commands really tick me off and I immediately make changes to make it easier to read.
There’s really nothing wrong with being clever. But if there are two otherwise equal solutions to a problem, and one of them is easy to read and the other one is clever, the easy to read solution is better.
The clever solution is only ever better than the easy to read solution if it offers some additional value beyond sheer cleverness.
If you optimize your code around being easy to understand you won’t need to understand everything the application does in order to work on it. This makes it a lot easier to deal with the fear that developers often experience when they get their first glance at the innards of a monolith. You want their first deep dive into the code to leave them feeling better, not worse. If a new dev pulls a story, does it, and says something like “that wasn’t as bad as I thought it would be”, you should consider that a victory.
Use namespaces to reduce complexity
In TroopTrack we have six different user controllers. Each one deals with users from a different perspective – from the view of our helpdesk staff, a customer, a partner, etc. This really represents the fact that TroopTrack is actually several different applications that share some stuff: we have a helpdesk, a mobile app, a partner app, and our customer app. To make things even more interesting, our customer app alone is pretty huge – it has two user controllers of its own, along with 150+ other controllers.
We use namespaces to isolate behavior logically. We’ve got a namespace for our helpdesk, our partner app, our API, etc. Within our main customer app, we have divided the functionality into five namespaces that reflect the user interface design: plan, manage, communicate, achieve, and share. These namespaces correspond to the way the application is organized visually. This makes life easier for us because if you pull a story that involves changing a feature that is part of our “communicate” set of features, you already know where the code for that feature is as well as how to get there in the UI.
Namespaces can reduce the brainpower required to work on your integrated system dramatically. I like my namespaces to represent the groups of functionality within my system because it seems natural to me for my namespaces to reflect the general categories of stuff the system does. You may find other ways that suit you better and I’d be interested in hearing about them.
Don’t get trapped in the default folders
Models. Views. Controllers. Helpers. Assets. Mailers. These are a few of my favorite things. But they aren’t all the things. We shouldn’t feel compelled to squeeze everything we do into one of these folders. If you have a thing that doesn’t really fit in one of these boxes, make a new box. You don’t have to put everything you make into one of these categories.
In TroopTrack we have some things that kick off processes that begin with an email to customers based on their partner affiliation, membership status, and other things. They get initiated by scheduled jobs managed by resque and it’s considerably more complex than just a query and an email. I started out by writing this as a class in our app/workers directory, but became frustrated by the number of things we had in there and wanted it to be less confusing. So I made a new directory, called storks, and I put these features there.
Yeah, storks sounds kind of stupid, but my thinking was that what we were doing was kind of like the mythological delivery of babies by storks. The arrival of the storks wasn’t just a delivery, it was the beginning of a whole new phase for the recipients. This was also the case for our customers – the arrival of these particular emails meant the beginning of a membership application or renewal process. Pulling them out of the Workers folder and creating their own place for them makes it a lot easier to understand where things are based on what they do.
You don’t have to put everything you build in one of the default folders off app. You can add more folders. With subfolders even! In hindsight, this seems obvious, and frankly, kind of a stupid thing not to know, but I see it a lot in myself and in other devs. If your code will be better organized and easier to understand by stepping off the template, do it. Do it.
Note: You don’t have to make a folder off of apps/ for things like storks. In this case I decided to do that because I felt they were distinct enough to belong on their own, but I could have easily put them in app/models/storks. We frequently do this for things that aggregate data from lots of models into a single view (i.e. reports) and for things that encapsulate complicated business logic.
Make it easy to get started
If you hire a new dev or accept a new contributor and the process to get your product up and running takes more than an hour you have failed. Don’t let this get out of control – never let your bootstrapping routine get so complicated that it becomes a deterrent to a new contributor.
Resources != models
Remember when everyone started to realize that fat controllers were a problem? A lot of us ended up with fat models after that, at least for a little while.
One of the reasons my controllers got fat is because I would try to use the same controller to deal with lots of different types of actions on a single model. In some perverse version of DRY, I would try to squeeze everything related to an achievement, for example, through a single achievement controller. My actions got full of conditionals, even to the point where I was sometimes redirecting different ways based on the things that were changed. It was a mess.
You can solve this problem lots of different ways. I sometimes solve it by just creating a different resource for the different types of actions I would take. For instance, we have the ability to record achievements for a lot of users at once. This used to be a big fat method on the achievements controller with a special action defined in the routes file. I made it better by creating a bulk achievement resource in my routes, then a simple controller to go with it, and a plain old ruby object to hold all the logic. This approach of moving complex logic into a PORO is well known, but the thing that I think we often forget is that our resources don’t have to be models. Refactoring this code to be easy to understand started with a single line of code in routes.rb:
The point here is that we shouldn’t get trapped thinking that resources have to be models. It’s just the name of an endpoint for crying out loud. You can do whatever you want with resources, and they don’t have to correspond to a model.
Put the bad kids in the corner until you have time to deal with them
I mentioned some really bad code at the beginning of this post that is a thousand lines long. It used to be on our Troop model, and it made doing anything with our Troop model frustrating. The code works, in spite of itself, and we almost never need to change it. Because of what it does functionally (import data from a CSV file), and how it’s used (only my interns ever use it), it’s not really important that it be efficient or even 100% reliable. But it’s annoying. So I put it in the corner. I removed it from the Troop model and put it in its own class where it wouldn’t freak us out every time we changed the Troop model.
You might think this is a horrible practice. I admit it feels a bit underhanded to hide my bad code in the TroopTrack basement, but… it’s a pragmatic strategy. There is no compelling business case for making the code better. We wouldn’t make more money, we wouldn’t win more customers, and our product wouldn’t be more stable. In fact, there are hardly even any occasions that require us to read the code, much less touch it. So to the basement it goes. If we ever encounter a situation where that changes, we’ll deal with it. Until then… basement.
Be careful about dependencies
I love gems. They make our world a lot easier by giving us ready made features that we don’t have to build ourselves. The scope of gems that are available to a Rails developer is incredible. It’s awesome.
Gems totally suck. They get abandoned by their creators. They introduce dependencies on other things we might not want in our application. Sometimes those dependencies mean trouble when we try to upgrade rails, leaving us blocked and forced to either find a different solution, upgrade the gem ourselves, or wait.
Bite the bullet on Rails upgrades
TroopTrack started out as a Rails 1.8 application (I think). We’ve been through 3 major upgrades since then. Rails 3 was the hardest, partly because of self-inflicted wounds we earned by coupling the rails upgrade with a major redesign of TroopTrack. Don’t do that. Rails 4 was actually pretty pleasant. I’m over the moon about Rails 5 – Turbolinks 3 and Action Cable are going to be very important technologies in TroopTrack.
Falling far behind the current Rails version is disheartening and compounds the sense that a large code base can never hope to be majestic and beautiful. Don’t let this happen. It doesn’t always make business sense to be on the bleeding edge of Rails, but it’s never a good thing to fall more than six months behind a major release. Just bite the bullet and stay current. If you stay current on Rails, you will find it easier to feel good about your monolith and easier to enjoy events like RailsConf, where you will be able to walk away excited to use things like Turbolinks 3 and Action Cable in your real product, not just play with them in a prototype that will never see the light of day.
Target and repair bad code
Bad code is discouraging, whether it’s in a monolith or a micro service. When you find it, fix it. Don’t blame your overall system architecture for the fact that I got in a hurry and coupled some things that never should have been. Just fix it. Or yell at me until I fix it. Solve the real problem instead of imagining your problems would go away in a different design paradigm.
Thanks to @dhh and the Ruby and Rails communities at large
I love Ruby and Rails. Thanks to this amazing backpack of tools, I was able to leave a stupid soul-crushing occupation years ago, do work that I love, build a business that helps others, and bring some of my friends along with me. I am beyond grateful for the work others have done that have made my life so much better. So, here’s to you DHH, Matz, and everyone else who have stuffed my prepper backpack full of integrated system awesomeness: thanks a million. I love you all.