There’s a worn-out analogy in software development that you cannot build a skyscraper the same way you build a dog house. The idea is that techniques that will work on a small scale will not work on a larger scale. You need more formality to build large software systems.
The analogy is always applied in one direction: up. It’s always an exhortation to use techniques appropriate for larger projects.
But the analogy works in the other direction as well: it’s inappropriate to build a dog house the same way you’d build a skyscraper. It would be possible to build a dog house the way you’d build a skyscraper, but it would be very expensive. Amateur carpentry methods don’t scale up, but professional construction methods don’t scale down economically.
Bias for over-engineering
There’s a bias toward over-engineering because it works, albeit inefficiently, whereas under-engineering does not. You can use a sledgehammer to do a hammer’s job. It’ll be clumsy, and you might hurt yourself, but it can work. And there are tasks where a hammer just won’t get the job done.
Another reason for the bias toward over-engineering is asymmetric risk. If an over-engineered approach fails, you’ll face less criticism than if a simpler approach fails. As the old saying goes, nobody got fired for choosing IBM.
Context required
Simple solutions require context to appreciate. If you do something simple, you’re open to the criticism “But that won’t scale!” You have to defend your solution by explaining that it will scale far enough, and that it avoids costs associated with scaling further than necessary.
Suppose a group is debating whether to walk or drive to lunch. Someone advocating driving requires less context to make his point. He can simply say “Driving is faster than walking,” which is generally true. The burden is on the person advocating walking to explain why walking would actually be faster under the circumstances.
Writing prompt
I was using some database-like features in Emacs org-mode this morning and that’s what prompted me to write this post. I can just hear someone say “That won’t scale!” I often get this reaction from someone when I write about a simple, low-tech way to do something on a small scale.
Using a text file as a database doesn’t scale. But I have 88 rows, so I think I’ll be OK. A relational database would be better for storing million of records, but that’s not what I’m working on at the moment.
One of the acronyms/buzzwords I’ve seen in modern programming methods is YAGNI, “you ain’t gonna need it.” This discourages the temptation to add features for future expansion that may never be used.
The biggest problem I have with designing for scale is how often we are wrong about how users will really use the system. I have been complaining about over-engineering internally for a few years mainly because we introduce complexity when we don’t know that we will need it in that manner. We recently ran into this when we did not expect the data to grow in one area of the application so quickly. One of the pages of the application was terribly slow for the data that we had (40K rows). Someone wanted to add some “fancy” technology. I asked if anyone looked at the queries behind the page, nope nobody. Fixing the queries saved a lot of time. Sometimes it is the simple things.
Agree very much here.
For me this is why it’s so important to decide what the purpose of a thing is. Sometimes the purpose is just to successfully demo something to an investor, in which case you don’t need that sign up portal/database/React UI/…