Almost every engineer has been in a position where they are forced to choose between shipping a shoddy product to meet the deadlines vs taking time to do things right.
Oh the catch-22 between perfectionism and delivery.
Story time
Consultation at Essentia includes a fire-fighting mode, where we jump into alien codebases in dispair, at a moments notice and try to reslove the issues.
Some time recently we got an ailing product which comprised of a python based backend and a react-native based mobile application. Its usual technical-caretakers were unavailable, so it was brought to us. We were asked to get it working on a fresh AWS account.
We went in. Got access to the git repositories, aws account, app store etc. It took us a few hours but we were able to piece it back together and get it working again.
The owners were delighted.
In the process, we discovered some glaring abysmal blunders:
- images and videos uploaded by users were being saved in external managed mongodb gridfs.
- APIs were open for all. The user-authentication did not protect the APIs.
- base64 encoded files were then cached in a managed redis instance on AWS.
- no tests, no documentation, incomplete deployment automation.
We responsibly communicated it, to the owners the severity of the impending disasters which were very much likely when the application gained traction.
- someone might discover the open endpoints and try to mess with the data.
- there could be a huge bill due to data transfer charges on AWS and managed redis cluster.
Sadly, this was not the first app in despair that I came across and am sure won’t be the last.
Why does this usually happen?
No one is lining up to fund, purchase or subscribe to an ugly app even if it has a great backend. But a substandard backend kills the app before it can gain enough traction to become viable. In the real physical world too, investment towards new and shiny projects feels like progress where as money spent in the old infrastructure does not garner any excitement.
Backend and architecture only become a concern when it stops working or slows down or its vulnerabilities become public news. If its working, however shoddily, it is not an immediate priority for engineering teams to spend their time on.
It is a chicken and egg problem.
Here is what I hope might avoid some agony in the long term:
Here is some generic tech-stack agnostic advice for people starting new projects.
Finding the balance
Engineering, like life in general is about finding the balance between the pros and cons of difficult choices.
We might end up waiting forever if we do not accept that somethings won’t be perfect and that there will be bugs. At the same time, ignoring something does not make the problem go away. Usually problems compound with time.
Finding the time, money, energy to look at the problems is the prime prerogative of the technical leadership.
Use boring and simple tech.
Anuvrat of 12 years ago, bubbling with a fresher’s energy had balked at the suggestion from his seniors to use tried, tested, boring, stable pieces of technology for the new project he had been entrusted to make. Now, I seem to have turned into the grumpy old man screaming the same advice to the youngsters paying no heed. And the cycle will continue.
Microservices and Kubernetes: do you really need it?
Unless you can have a separate team of engineers dedicated to and owning every microservice for its maintenance you cannot afford microservices. As a single developer working on a shoestring budget microservices soon become unmanageable.
As for Kubernetes, there are too many memes about it. An app with a few thousand users running on K8s is like deploying a tank to kill a mosquito.
Serverless, lambda, cloud-functions
Serverless is a facade. There is a server, it is just that you are not administering it. The pros is that someone else is responsible to keep it alive. The cons is that you are not in control and cannot fix it if it does go down.
Managed Services
Services that promise to back-up your databases or deploy your code automatically from your git repo for a ridiculously small fee today won’t always remain cheap. Some day in the near future, in order to cover their costs, those services would have to charge you more or risk going out of business. In either case, you will be left high and dry.
Preferably learn to manage as much of your infra yourself, as you can. Managing all infra in-house will also not be practical if the team is short-staffed. So one has to be prudent and find the right balance.
Have a separate but identical local / staging / production environments
The whole application stack should be simple and lightweight enough to run on your personal machine. It will save you a ton of time while developing / debugging.
Tests and technical debt.
If you want your project to live long, write and maintain the tests.
The energy invested in writing and maintaining Tests is akin to regularly paying insurance premiums. It pinches when you pay up but have nothing to claim it for. But its absence pinches more, when you don’t have insurance and are staring at a huge bill eroding your life savings in one fell sweep.
Use version control
It is almost two decades since git came into being. Use it, or mercurial. There is no excuse for not doing so. And yet I find software projects without or with horrendous use of version control. Story time: this decade old company did all development live on their production server and “backed-up” their source code to their bitbucket account once every few weeks. :sigh: 😞
Document setup instructions and automate deployment
While setting up a new project, you think you would remember how individual components have been configured but that is not how our brain works. Stepping into a old project after just a few weeks requires a bit of poking and prodding to discover what we had done and why. For the sake of a new person stepping into the codebase or your future self looking back at it a few months later, document your setup, automate deployment and keep it up to date.
Better yet, use infrastructure as code solutions: ansible, terraform, salt etc.
;tldr
- Keep it simple.
- Aggressively use fewer pieces of technology.
- Be mindful of future cost of things you are using today.
- Be mindful of architecture and security. Seek feedback early on.
- Automate and document as much as possible.
- Write tests. Use version control.
- Stay away from shiny new pieces of technology.
Conclusion poetry
Excess of anything is bad. At the extremes, exists madness.
We always have to pay for our past sins. All debt comes due, someday.
Complexity costs more to maintain. Technology and life in general is all about choices and their tradeoffs.