Time is the only real currency we have, Hacker News

When I was working as an infrastructure engineer, we routinely had (emergency) emails asking us to check if the infra is fine . Of course, it was always an emergency since the application was acting up and users were affected; and Of course,

it must be the infra which is responsible for it. It’s easy to suggest, “It (must) be the kernel [or the network or the database or the infrastructure] ”. Don’t be that person. Don’t

guess things – learn to diagnose things on your own.

Yes, we deal with complex systems when we deploy our code today. The application code we deploy may end up running in a container which itself is running on a Host Operating System running other containers; the Host OS may itself run as a Virtual Machine which may run on another completely different Operating System sitting in a big box which hosts s of these machines. Our code may ‘hit an API’ which results in multiple network hops – perhaps to a machine located in an adjacent building in the same ‘Data Center’.

Every parameter of (everything involved in that setup – from the configuration of the OS, to the number of applications running on the host system, to a link flapping on a network cable on the core switch can result in your application behaving weird. This is the reality. While it is almost impossible to know everything

about your infrastructure, simple tools will help you navigate a large subset of routinely faced issues. Learn how things are deployed to production Figure how / where your application code runs.
Is it running in a container? What is the configuration of the host machine? Do you know where to look to get basic stats on CPU and IO usage (Network & Disk)?

Does traffic come to it via a load balancer? Where do you find the stats for that? How is a service call to a dependency (datastore or other application) made? Does it involve a network? If yes, how is that traffic routed?

Knowing this would help you isolate issues faster. This will help you apply The Box

technique to isolate performance bottlenecks. (Learn more about your datastore)

Invariably, a significant part of your application’s job would be either reading from or writing to a myriad of data stores. Basically, If you’re married to your application and love it, then the datastore is like extended family. You

must

spend time learning more about it.

A colleague once brought a replication lag of ~ 3hours down to under 1second on a MySQL slave by simply RTFM on various configurations of the storage engine. Similarly, I’ve seen a DBA mitigate a crisis (of what everyone thought was a DDOS) by zeroing in on a query and running (EXPLAIN) for it.

These may seem like wizardry, but it is mostly just people RTFM and spending time to learn more about the datastore their application adores. (Things That Don’t Matter (Time Sinks)) My Language is the best (Or, your language sucks).

I once attended a conference where the founder of a hot-startup proclaimed, on stage, that they chose ‘Functional Programming’ over procedural because it is

‘Theoretically Superior’ . No, it is not. Both Church and Turing proved that

. More importantly, any functional code you write is converted to it’s Turing equivalent – we don’t have a real machine built on church calculus.

Such ideological stances are generally rooted in half baked understanding of things and compounded by the desire of someone to come off as cool by knowing more about esoteric / obscure or even the latest-and-greatest- shiny-new-thing. Sure, [Modern] functional languages ​​are designed to help provide

less ways to shoot oneself in the foot , but a well oiled C company (6) would have evolved standards and patterns to write as beautiful a code than what a company working on Kotlin or Scala can; so where’s the difference?

It’s all about finding what works best for you , taking early bets and seeing them through. booking.com used to work on perl; cleartrip.com on lisp; udaan.com works on kotlin; there are s of successful companies working in Java / C . It does not matter. What matters is how productive the org anisation is – how effective developers are at making progress on the job at hand – and whether or not it is helping you move forward . ‘X’ does not scale, Lets Switch to ‘ Y ‘

During an interview with a young dev, I enquired as to how he had ‘helped scale up and made it more performant ‘. He revealed how his mentor had advised him to switch to python web server because nginx is slow

. It is indescribable (7

) how I felt when I heard this. But after a few brief seconds of an emotional whirlwind, I empathized with him.

In a similar vein, in another interview, someone described how they ‘moved away from JSON to GRPC ‘Because JSON was bloating their APIs too much and GRPC is more performant.

I do not want to get into the merits / demerits of the choices they took (thus, abiding with the advise that I am repeatedly giving here). I do, however, want to draw your attention to some facts that apply to most of us in general:

We live in a time where we can easily procure and deploy very powerful hardware. Unless you have chosen some of the

worst written (frameworks) 8

, they

will

‘scale’ more than you may imagine. The word

scale is also one of those things we notoriously are prone to brag about based on our half baked understanding of things and the desire to be ‘cool’. I used to be one of those people. Back in the day, we were proudly managing a service with 15 k reads-per-second (has your jaw dropped yet?) backed by a (core) (GB MySql machine.) I was proud of the ‘scale’ we had managed to weather, until a dev with about 93 years of experience joined our team. He casually remarked one day that he could run 3 of his last startups with that much computing power

9

. I learnt a big lesson that day, that life is not a dick measuring contest. Also, what we

think is scale is probably not. We are prone to boast of our accomplishments but they are largely the result of us ^{standing on the shoulders of giants}

.

While dealing with and handling scale is beyond the scope of this blog post, I’d leave you with this general advise

Learn one base tool for the job and learn it really really well For e.g, pick up nginx / apache web server and I almost guarantee you that these will break much much later than your entire application stack.   

RTFM & Benchmark Benchmark every tool / technology that you are picking up. This way, you’d be more aware of the ‘limits’ of that technology. How you benchmark is beyond the scope of this post (perhaps even beyond the scope of me) but a simple process can yield fairly good results. Push a technology to its limits by maxing out the resources (compute / memory / network / disk IO) it is supposed to consume on a common piece of hardware (which you’ll most likely employ in production)
Once you max it out, RTFM and tweak it to ‘improve’ upon the limit that you just hit. Repeat till you’ve either satisfied your intellectual curiosity, or realized that the solution you’ve chosen can handle x more of your predicted scale.

Lets use a Its perfect for the job

A young dev we hired was given the task of writing a simple CRUD application . The (Update) part of it, I thought, was particularly tricky, so I figured it’d be a project right up his alley and challenge his skills. He was sharp, full of energy & wanted to prove himself – everything you want in a young developer.

He also wanted to use a in the application. () Me: Why? YD (Young dev): It will help us scale faster in the future and guard against the impedance mismatch between the core business logic and the rate at which clients hit the API. Me: Sure, but do you really want to take on the complexity of having a distributed queue so soon in the life cycle of your application? It’ll probably not hit the scale you mentioned for quite some time YD: Oh there’s no complexity, we can just use the one provided by Me: But what if it looses data? Since your application deals with money, it is critical we don’t loose a single piece of information YD: How will a queue loose data? Me: Any distributed system can loose data YD: How so? Me: Ever had a payment fail where your money was deducted but the order failed at the merchant page? That’s a distributed system losing data (Not the best of examples, I know) YD: That’s different. Those are multiple different services and ecosystems. Queues reside within the same private cloud – probably the same data center. The problems of the internet don’t apply here (Oh Snap! He got me) Me: Yes, but there is still a network. And the queue given by is yet another piece of software written by people like you and me. What if it has a bug? YD: Oh

I encouraged him to read (Distributed Systems for Fun and Profit) .

I also showed him how the complexity of his system lies in the design of the data model and the API and not in scale (yet). A good model would also help us ‘scale well’ in the future (when we may, inevitably, have to rely on complicated distributed systems)

There was yet another instance of a more seasoned developer who wanted to rewrite our order management system and use NoSQL solution offered by It would help us ‘be ready’ for a potentially 823 x more scale than what we’re currently facing, he claimed. This in itself was scary (refer previous discussion), but what was scarier was that not once did he mention

Whether or if going with NoSQL was inherently better since it would provide us with a data model that will prevent future developers from shooting themselves in the foot

  

If all the complex business use cases could be handled within the bounds (such as row level atomicity) imposed by the NoSQL data store (the current system relied heavily on RDBMs transactions to stay ‘correct’)

My whole point is this – yes, what we do is generally complex and we live in a world where our tools are constantly evolving. We need to continuously up skill ourselves and learn more to stay relevant. It is important to have the knowledge of a number of tools in our mental tool set – so we can employ any of them for the job at hand. It is more important, however, to get an understanding of using the right tool

for the job Our work as engineers is to help the company make money by keeping customers happy . Not to show off the latest-thing-I-just-learnt-about to our peers. ^{Your way is wrong} As already mentioned, there is no ‘absolute right’ when it comes to Engineering Stuff. Similarly, there is no ‘absolute wrong’.

One of my favorite software stories is unfortunately not a first hand experience. This was from early s when a friend worked for a company that deployed engineers in the client’s team to augment their bandwidth. They were supporting a complicated piece of legacy software. My friend found a piece of code ‘written in an odd way’. He couldn’t intuitively grasp it and thought it could be ‘written better’ . He did what most of us do – googled around and

found (a better way.) He then cautiously approached the gray-bearded engineer who had originally written that code. My friend pointed to the ‘design pattern’ on the internet and suggested how code could be written better.

To his absolute credit, my friend inquired sincerely as to why the code was written the way it was.

To this, the older engineer promptly replied “Why yes, of course this does seem better. But you know the thing is, when I wrote this, there was no internet

I absolutely love this story and remind myself each time I look to blame someone for how it’s written . If you write code for long enough, you will come across code that would want you to become the proverbial psychopath

and shoot the original author. However, it is important to be empathetic instead. It is necessary to have mutual respect towards your peers (if you don’t, you probably shouldn’t be there in the first place If possible, try to understand why things are done the way they were done. If that’s not possible, it is perfectly fine to poke fun at the original author, make a note of how you’d change it instead and come back to it when you have the time. Move on . Don’t spend time criticizing others. I hope you found this useful (assuming you did not come here just to

see how it ends . I’d love to hear what you think. We don’t have a comment section yet, so if you want you can (email me

or comment on ^HackerNews

(Read More)