I’ve been an “active lurker” since 2008, occasionally submitting my own writing (had my earliest “first-pager” in 2012). For me, the most important part of HackerNews is the comments section.

If you open the comments section of almost any first-page link, you’ll usually find trolling, trashing, and other pessimistic behavior that clouds everything good. Over all these years, my instinct tells me that only a small percentage of people post most of the negativity. And I’m not talking about proper constructive criticism, but the usual internet rage that was so common during the early forum flame wars.

That can make one think that the place is not worth visiting. But when I looked closely, I found that although the amount of unhelpful comments had gone up, there were still the same quantity of gems to be found as I did over a decade ago.

While reading comments on an SICP post, I learnt that Sussman published a title in 2021 called “Software Design for Flexibility: How to Avoid Programming Yourself Into a Corner.” I got my copy of SICP while I was still in college, and had also viewed some of his MIT lectures. In an instant, I bought the hardcover to open my mind to more great ideas.

To whoever commented the above, you folks are the real gems of the community, and made me discover things that neither the first page of Google, or the supposed “RAG-based knowledge tools” are able to surface.

And just like that, the people who are willing to participate in the community…

…who are okay ignoring the negativity…

…who are okay disagreeing respectfully…

…will eventually dig a lot of gold. Convenient tools will just make others myopic.

Disconnecting from this community might cast a big blind spot in my potential for knowledge, because I’ll be disconnecting from a lot of smart folks whom I might never meet in real life.

Or, maybe I will?

First, it's compensation. If one is not paid market salary, the company can still offer him a title that the market won't. It's a legitimate strategy if the person can grow into the title eventually. But more often than not, the title is seen as an "end" rather than the beginning.

Second, the person's history. A CTO of a small startup would probably not want to become a SDE-III at a medium-scale startup. So the medium-scale startup invents a position between an engineer and the CTO.

Third, to make room for internal promotions. Everyone expects their designations and salaries to grow. But an organisation might just have junior & senior engineers, with a CTO at the top. And the only way to promote senior engineers would be to invent new titles yet again, even if the job responsibilities don't change.

All this leads to a designations not being comparable between companies. You have to look closely at the areas of responsibilities rather than the title alone.

I started learning to program as a kid. I got my first computer when I was 10ish, and since I liked playing games, one day I wondered, what would it be liked to create one? And that's how my programming journey started at the age of 11.

Initially, I started learning a tool called GameMaker, which was a free product created by Professor Mark Overmars. It was a fabulous piece of software. As I grew up, I learnt that GameMaker was acquired and is now published by YoYo Games. It was quite cool to see a hobby software turn into a great commercial product.

But it was not GameMaker itself as a software that inspired me to program, and continue program. No. The secret's something else. Before I get into it, though, I want to tell you that I struggled to learn programming like anyone else. It's like learning to balance a bicycle. When you try 100 things trying to find the one thing that works, you're gonna fail 99 times. And those kinds of failures are not really failures because you're still moving forward. However, if you give up, you've stopped, and that's real failure. And the most heart-breaking thing is thinking that something is impossible, even though you had more than enough in you to attain it.

One peculiar thing about the internet back then was that it was not as crowded. There were other kids out there my age who were also into programming. And since the internet was not cluttered, you didn't find a lot of "programming is insanely difficult," or "you really can't program" kind of negativity to kill your spirits. People talked about ideas and how to implement them. And the secret to me learning to program well was the GameMaker community/forum.

Unlike StackOverflow today, it was not a pointed question / pointed answer kind of a thing. Discussions were welcome, even about controversial choices. There was a section for people to showcase what hey had built, which inspired me further. But why does all that matter?

Because you get feedback, you feel useful. And that was important to me as a kid. Because even if I accomplished something small, and couldn't myself gauge how valuable it was, there were still a bunch of people who found my silly program useful. Who learnt from it. Or used it. Or liked my advice on how to improve something.

In that era, you'd talk to people through your work. Rather than just talking "at your work" as it happens today. Some feedbacks were good and inspiring, some were blunt or heart-crushing, but at least they were all honest.

We didn't have "titles." We were not even from the same continent. We were not all fluent in the same language. But we spoke the language of code and creation. Everyone created something, and everyone else responded to that creation. And that's all that mattered. That was our litmus test for knowledge: could we build it?

While today the focus is a lot on proxies for ability, like the number of social media followers, job title, years of experience, etc. But then, it was just pure build. Like Bobby Fischer said, "Play the board, not the man." We played the board. In today's media-happy world, we often don't.

In the subsequent years that technology has changed, I've found something profound. I can look at a piece of code, or a piece of software, or even a behemoth like AWS, and look at a screen for the very first time and still "solve" problems that a normal person couldn't. Why? Not because I was experienced with the AWS console. But I understood the principles by which that product had to work, even if I didn't know how the product was designed. It was like having a great superpower. And that power came from foundational thinking. First principles.

When you learn something vaguely you learn how it works. When you truly understand something you learn how it must work. And once you have the principles, you don't have to endlessly chase goals. Or new frameworks. Or new paradigms. They're all tradeoffs between different principles after all.

So what are those principles? In software engineering, it's only one. It's the ability to "put any puzzle together." In the case of writing code, the puzzle pieces are syntax, and the puzzle is creating a program.

In the case of systems design, the puzzle pieces are individual components that you use, and the puzzle is a system that behaves in a manner you want, while to the largest possible effect avoiding side-effects that you don't desire.

But syntax, components, algorithms, all of that change. Imagine what the world's first computer scientist must have known to become a computer scientist. They didn't have a syllabus. Nor a textbook. Nor ages of knowledge to assimilate before calling oneself a computer scientist. And that's the key. That something, that skill, that makes a person discover things. Even today, that's true of a computer scientist or a software engineer. A person who pushed the needle of knowledge forward, regardless of how much of it is already discovered. If a large stack is already discovered, that clouds what we are supposed to do. But we are supposed to move forward nonetheless.

Before we get into the waterfall vs. agile flame war, let's define what we mean by waterfall. Loosely, it is said that waterfall is akin to long-term planning while agile is responding to changes as they happen. But that's an overly simplistic definition of each.

Any endeavour will have a long-term vision and a set of short-term projects to support that vision. If you want to be a better engineer five years from today, you have different options; be it mastering algorithms, or learning TLA+ to improve your chops for distributed systems, or designing the next Rust.

Thus, long-term and short-term thinking in themselves are not the differentiating characteristics between waterfall and agile.

Planning happens at many levels. Just like a current flows through resistance, "heading" for a destination in the face of random obstructions, so does the execution of our plans. For example, the exact actions that'd contribute to your 5-year goal of becoming a better engineer could keep changing. It could be learning Rust today, prompt engineering tomorrow, and systems engineering the day after. Although your goal is still the same, your immediate actions depend on the immediate circumstances.

Navigating a goal is much like navigating traffic. You may go straight, take detours, or even go against the direction of a goal temporarily because the "road" was closed. But your destination is still the same. It's as if the goal was an abstract class and your immediate actions the most viable implementations.

So it's not long-term vs. short-term when we talk about waterfall and agile. It is the rigidity of the details. Earlier waterfall models insisted on having all the details, even the tiniest one, up-front. Which is akin to you deciding which specific courses you'll do for the next 5 years. But maybe by year 2, there is a better course for learning Rust, or the one you had in mind got outdated.

Or maybe your goal isn't "learning Rust." It's learning something cutting-edge. It happens to be Rust (and many other things) today. Tomorrow, when you actually sit down to do something new, it very well could be something else.

Thus, agile's response to changes has to do with the details, not the goal itself. Waterfall would be like a predefined plan, whereas Agile would be a policy that's applied to a changing landscape like a reinforcement-learning agent.

However, there is a way to take even agile too far. One can much as easily assume that having any kind of long-term goal is NOT agile because it locks you in. And we should be willing to change long-term goals too. True, that can sometimes happen. But you wouldn't want to work at a startup that was creating a database in the morning, a game in the afternoon, and finally pivoted to being a todo-list app in the evening.

Agile doesn't mean you shouldn't have long-term goals. But what it means is in the face of a policy biased toward some long-term goal, you should act intelligently on the actual circumstances rather than what you wished the circumstances were.

Waterfall tries to build a plan that can be greedily followed. Agile keeps the option open for maximising long-term outcomes, even if you make u-turns in the short term. And that's where the distinguishing characteristics of Iterative not being Incremental comes in.

Incremental is a greedy strategy. You had 5 lines of code yesterday, today you'll have 3 more, and tomorrow you'll have 4 more. They all add up to 14 lines of code.

Iterative looks a bit different. You have 5 lines of code today, which exposed some problems with the original design. Tomorrow, you have a different 4 lines of code as you threw out the 5 you had written previously. And so on. In the end, you end up with 3 lines of code that solved your problem, but only after you threw away 34 that didn't.

If we don't throw things away, then we'd be resistant to doing experiments to learn new things. And the iterative strategy often has us take detours, u-turns, for the long-term good, whereas incremental insists on short-term gains. If you have a feeling you're not making a lot of progress, you may benefit from viewing it from an iterative lens rather than an incremental lens.

Startup engineers are NOT usually taught how to interview. If someone is good at their job, it is assumed that he can evaluate potential candidates. You'll thus find people with limited technical experience, and still lacking the maturity of working with a diverse team, having the authority to determine who works for the company.

When people are not taught, they wing it. A primary mistake in interviewing is conflating concepts with knowledge. Unless someone is really good at software engineering, he has not yet learnt that syntax is NOT semantics, and the Kafka queue he learnt in a systems design Udemy course is not the only way to build this particular system.

The conflating of concepts with knowledge degrades the whole interview process to "how many different solutions have you memorised" instead of "do you have the ability to understand new problems and the skills to deal with them?" And the reason this is culturally acceptable is because this is how schools work.

Thus, it becomes important for your interviewer to hear things like "Kafka" or "Microservices" rather than answer the candidate's petty questions like "why do you need BigData for 100GB of data that you have?"

Because the emphasis is on arbitrary solutions rather than problems, there is never a good exchange of problems in an interview. Only comparing notes on "who has implemented a bigger solution so far in life."

For you to understand what a candidate has accomplished, it's important to understand the landscape of the problems that he dealt with. Sure, he wrote an app to work with Sqlite databases, which sounds very simplistic. However, when you look closer, you realise that he got the company migrated off-of ScyllaDB by rearchitecting a read-heavy application to use Sqlite over S3. And the primary motive was saving cost. Yeah, the solution does not look "cool," but he managed to pull off a major cost optimisation project. Usually, the arrogance of the interviewer prevents him from understanding the problem statement of the candidate, because it's not the interviewer who is getting interviewed.

On the flip side, inexperienced interviewers also see the world in black and white and think that their problem statements are the ONLY kind of problem statements that exist. They'll go ahead and ask, "How would you ensure that you rate-limit APIs properly?" The candidate, not knowing the details of the problem, and being rebuffed on asking more details, answers: "Well, I would just make sure the WAF takes care of any abnormal traffic; I wouldn't add anything more to it unless deemed necessary."

The above is not a bad answer in itself. It answers on the side of security (availability). But the interviewer's company rate-limits APIs because there is a non-public enterprise-tier API that is billed per call. But he never mentions this landscape to the candidate.

Now, many might jump to say, it's the candidate's job to fish out information. Well, most of the developing engineers don't have the skills to ferret out information either. Or else, there won't be such a big disconnect between the development and the product management team. The disconnect with the business teams is greater still on both sides. It's just that the interviewer has assumed what they know is what "programming is all about," and anyone who doesn't know it is "not a programmer."

That's the hidden standard right there. Whatever answers you give as a candidate, the interviewer adds additional "data points" to it, and then deems the final picture incoherent. But these hidden standards are not known to the candidate, even if he had the ability to solve the company's problem.

Additionally, the problems communicated to the candidate are usually described through the lens of the interviewer. Most of the times, I've seen good engineers solve problems because they viewed the problem differently, not because they computed differently on the exact same definition of the problem. So one must not set up the problem landscape and be so biased toward it that opposing views are dismissed even before being evaluated on their merits.

It is hyper-critical that we expect candidates to instantly jump to an answer with incomplete and biased information, whereas we ourselves would have taken days to internalise the requirements, weeks to fight over the timelines, and months making excuses about a buggy implementation.

Any time we're not able to evaluate skills, we degrade to evaluating memory in the name of experience.

The word waste has two meanings.

(1) waste: The act of expending something carelessly, extravagantly, or to no purpose.

(2) waste: Unwanted or unusable material, substances, or by-products.

When we hear the word waste, usually the first definition above comes to mind. And we immediately go into "avoiding waste mode," sometimes to the detriment of hoarding stuff we don't really need. And the society drills this "anti-waste" lesson very covertly. For example, a parent can accuse a child of wasting food, without drawing attention to the fact that the parent also contributed to the wastage by knowingly giving the child food that she hated.

However, the rest of the article is going to be about the second definition above. The unwanted or unusable things that we should definitely get rid of.

I'm not a productivity buff, but I am happier when I'm doing a lot of things. Doesn't matter if it's reading books, writing an intricate piece of software, or figure out the source code of DuckDB. Today, as I was re-reading David Allen's Getting Things Done®, I saw a phrase:

"Some executives I have coached have found it extremely useful to arrange for a large trash bin to be parked immediately outside their offices the day we work together!"

The bigger context above is that as you organise and write ideas on pieces of paper, there's a lot that you'll throw away because they didn't pan out. As I read that line, however, it occurred to me that an under-appreciated art is that of throwing unwanted things away.

For example, as a programmer, I know that many novices initially learn to "not waste code." They don't want to write something that they will have to rewrite. They focus heavily on getting the specs right, not chasing a moving goal post, and the likes. We all like "agile," but only as long as the development is incremental rather than iterative (they both are not the same thing).

Experienced programmers, however, do plenty of proofs-of-concepts with throwaway code to clarify a problem and experiment with new ideas. Once you obtain new understanding, you usually don't need original notes or apparatus in order to "keep that understanding." They might come in handy when you're communicating with someone else, but we're not going there.

Only in school do you need to show your work. In the real-world, you just need to show that it works, and how you got there usually is far more amorphous than a typical school-grade math problem.

In short, we tend to not throw away things that we should, assuming that it is the "unethical kind of wastage."

Technology appears to solve the waste management problem, but all it does is make it easier for us to hoard. We no longer have to make that judgement between important vs. unimportant, or necessary vs. unnecessary. We just keep them all.

For example, in the early days, emails had limited space. You had to delete stuff if you wanted to clear up space for new emails. The same was with SMSes and contact lists on our cell phones. In many cases, if you were into sales for example, the inbuilt capacity was not enough. And you found such people with organisers to keep track of people they've met, their history, etc.

However, ever since Gmail started offering a 1GB storage (and at the time, "ever increasing"), the necessity to go back and delete emails went down drastically. There is a difference between keeping something important for long-term, and avoiding "clearing up the waste basket because the basket is so big."

In this data-hungry world, we hoard things as much as possible, because "who knows when something might come in handy?" What goes for a toss is our judgement of what's useful and what's not.

We're so artful about finding shiny new things, that we forgot the art of throwing away the unwanted. And without the crucial skill to judge the importance of something, we won't recognise gold even if we see it, and we'd think a piece of copper is a 105 carat diamond. Quantity cannot solve quality.

To be more relaxed, throw unwanted things away. Maybe it's your unwanted inventory that's been keeping you from achieving what you truly want.

You may be reading a book, or deep diving into a course. Or maybe you’re perplexed at the overwhelming number of choices. Amongst the innumerable ways one can spin in circles trying to learn to code, what’s the optimal path to building your dream apps?

When I started to program, YouTube didn’t exist, and not a lot of people wrote technical content. We relied on hardcopy books and official programming language documentations. Also, the process was about learning principles so that you could use the documentation as a reference rather than a “tutorial” while you built your idea.

My first year in programming was an uphill battle. As a 11-year-old, I was excited about building something, but also overwhelmed with the complexities of making even a small program work. More than programming languages, even using computers in general was a technical challenge. But when I did come out the other side, I had a good foundational knowledge that was resilient to changes in programming languages, frameworks, technologies, and the likes. I was future-proof. What exactly did I do?

Because of a lack of content, I spent 80% of my time trying to write code, and just 20% of my time reading books or articles. For example, there’s only so much to learn about Lego® before you get your hands dirty trying to build something. The more you build, the more you understand each brick better. Same is with programming.

When I started, I too looked at a blank screen. But I knew what the problem was. My problem was not that I didn’t know C++. My problem was, “now that I know C++, how do I use it to build something?” And the only way to learn is by practicing it. So, how should you practice your Python chops?

Python has great a documentation. There is also a small tutorial about Python embedded in that documentation, called “The Python Tutorial.” It’s free, concise, and useful. Your first step must be perusing this document. This forms your 20%.

The second step is to “build something” from it. Here are some example projects that you can build by just using the Python documentation:

1. A program that asks you for your first name, and then last name, and then prints out “Hello, <first name> <last name>.”

2. A program that asks you for a principal amount, rate of interest, and no. of years, and prints the compound interest at the end of all the years.

3. A program that reads a menu (item + price) from a file and sorts them by ascending order of price (cheapest first).

The way to approach building something is to:

1. Read the Python Tutorial in its entirety first (you can skip the OOP and other advanced chapters).

2. Then, you pick a project up to build, and mentally sketch out what operations you might have to do in plain-English.

3. Go back to the Python Tutorial to hunt for specific information that you need.

When you follow the above process, you’ll realize that the “code” is not as important as your thinking process to “write the code” in the first place.

And it is going to be tough. At times you will stare at a blank screen for hours. But remember, your job is NOT to just to “figure out.” You need to learn to “figure out HOW to figure out.” That’s why the software building process appears so painful at the beginning. But soon enough you will reap exponential rewards if you persist.

Regardless of the learning resources that you use, spend 20% of your time reading books and watching courses, and 80% of your time experimenting. And when in doubt, just start with “The Python Tutorial.” Don’t shy away from a blank screen and keep persisting.