My engineering manifesto. On software engineering.
Chapter X. On fun, GPT and Do-It-Yourself.
Today’s Best Band Ever™ is Florist.
It’s the end of the year, and I’m slowly going through the Pitchfork’s list of best 50 albums of the year. Suddenly I find myself completely stuck with Florist. I’m listening to this album on repeat for a third day already and cannot listen to anything else.
Back to reality.
We’re another step closer to controlled fusion, globalisation is dead, ruzzia is close to count 100’000 dead soldiers and 300’000 wounded. They had 200’000 brought in for their 3-day blitzkrieg.
Let’s have it settled. Close to half a million are incapacitated. One could say, ruzzia is big and has 140 millions, but let’s have a look.
Out of 140 millions, ~50% are females. That’s 70 millions.
Out of the 70 millions, about 20 are children and old people. That’s 50 millions. So roughly a third of the population are men of working age.
There was COVID. Ruzzia had their own vaccine and general population was anti-vaccine. At 22 million COVID-related deaths, that’s an extra 7 million men of working age.
43 million. Reducing another third for alcohol, drugs and traditional “I’ll go to the doctor after I’m dead”, we’re at approximately at 30 million of men, capable of work and fit for military service. And we’re talking about 2% of those already lost. And these 2% are so significant, that they are already noticing it.
This started as a short one, triggered by two things that I could not leave unattended. Now it’s a full-fledged one.
Thing one. Consider working on genomics.
Thing two. ChatGPT.
While seemingly unrelated, I believe there’s a fundamental connection between the two (here, I’m referring to the ChatGPT’s ability to generate code). Let’s have a look.
Genomics in a nutshell
From software engineering perspective, the essense of computational genomics is about working with a very long strings composed from combinations of 4 special letters. Computational genomics is mostly about aligning and comparing strings (letters are a metaphor, so comparing is a little more complex than comparing if characters are the same). If you don’t believe me, have a read at https://learngenomics.dev/ and tell me what I’m missing.
With the particular case of genomics out of the way, let’s talk about the general case of the relationship between science and software engineering. Since we’re educated grown-ups talking to educated grown-ups, we’ll start with a picture pie chart.
Majority of scientific work is not exciting – documenting, running iterations after iterations of experiments, writing papers, peer reviews.
I suspect people go into science because of the cool stuff. The boring part of academia is often solved by employing PhD students to do the grunt work. It’s the good old academic hazing.
The problem that Clay McLeod is describing is that putting together a quick python script is the fun part. Making it into something sustainable, on the other hand, is the boring part. This is where first a PhD use is attempted and, when that inevitably doesn’t work, scientists reach out to the software engineers.
Don’t get me wrong, delegating is key to achieving bigger goals.
Now let’s have a look at what software engineering is made of:
If you haven’t seen “Engineering in a Hybrid World” report, and you’re interested in topics and data of reasonable KPI, product/matrix/tech teams, remote-first, software engineering metrics etc, I strongly suggest checking it.
On DevOps
There’s a popular misconception that DevOps is a department in an organisation. If you see one, you’re looking at simply renamed sysadmin department. DevOps is a methodology. One, where operational activities are integrated into software engineering pipeline. Which means that, ideally, DevOps activities are performed by software engineers. If one so desires a dedicated DevOps engineer (which can make sense at early stages, when it’s only established and knowledge is scarce), they must be integrated in the product/matrix team and their primary objective is not implementing DevOps tickets, but teaching the team to do that.
For the sake of this monologue, I’ll take the DevOps pipeline from that report:
In other words, writing code is only ⅙ of the entire process of developing software. Each step has its own proportion of boring and cool stuff, which contribute to the total.
When a scientist writes software, the moment it does what it’s supposed to do, it’s done. Not perfect, but 95% there. From scientific point of view, the hypothesis is proven.
So, reasonably, when they delegate the remaining 5% to a PhD or a software engineer, they see the remaining work as trivial, but fairly meaningless (since it serves no scientific purpose) task. They did most of the work, after all.
From the scientific point of view, they’re absolutely correct. Science stops at PoC. Product starts, after PoC.
Finally, the time for ChatGPT to enter the picture.
Since its release, the internet is flooded with fascinated comments that, so many things are going to be replaced by it. Some valid cases are writing emails and year-end reviews. Other cases are producing working code from an iterative text prompt.
And this is exactly where the misunderstanding is.
ChatGPT, generating code, is nothing but another (though very clever) no-code tool. We had those at least since MS Access. That’s early nineties.
Just like a scientist or ChatGPT, the software they produced is only a beginning in software engineering journey of making a product. It’s only a part of the first block out of the six above.
Whoever skips on any part of that ends up exactly in the situation, Clay McLeod is describing – the code is hard to maintain, the software is difficult to use.
And this is what software engineering is about – writing code and turning it into a product.
ChatGPT (GitHub Copilot, Amazon CodeWhisperer etc) are great tools for bootstrapping code. They’ll allow software engineers to be more efficient. Our industry is going through optimisation cycles:
we were punching cards in binary and switched to symbolic presentation of it
we were doing waterfalls and are switching to agile
we were writing login forms and have now OAuth and Passkeys
yeoman, create-react-app and scaffolding in general
ChatGPT is next to take over some boring parts
So if a problem can be solved by ChatGPT or other tool without involving software engineers, then it’s perfect. Finding the right tool is hard.
But the 80/20 rule is still applicable here – 20% of work takes 80% of time.
One, while not being a plumber, might be able to fix a leaky pipe in their home, and it will even hold. When it comes to entire plumbing of a building, hiring a professional is a better choice. Making something and selling it on Etsy is very different from building a factory and producing something at industrial levels.
In its essence, software engineering is about optimising someone else’s job at scale. What they were doing by hand (remember human computers?) is translated to be done by computers or robots. And this is great as it allows scaling (humans don’t scale), speeding up (“adding manpower to the project that is already late, makes it later”) and liberates humans from menial jobs, not worth the intelligence.
I’m not afraid that ChatGPT will take away my job. Since the dot-com boom, we’ve had an exponential amount of new software written and most of it was quite bad. If ChatGPT takes over the job of creating bad software, at least it will be uniform.
In some ways, this XKCD, while funny, is not that wrong, but shame on us if we don’t invest in optimising our own work.
Improving efficiency is not as glorious as discovering the possibility of cold fusion, but it was the gradual improvement from 70% to 120% that made the work in the National Ignition Facility so important.
Take care, improve efficiency and stop wars.
Stay safe (unless you’re an invader) and have a merry Christmas.
PS. This post is surprisingly still 100% GPT free.
I'm reading DevOps for Data Science - https://do4ds.com/ - which might hit both of your items as well