TDD and the Silver Bullet Express

So, there has been a lot of talk about TDD in the past half a year. It (possibly) started with David Heinemeier Hansson’s articles, TDD is Dead. Long Live Testing and Test-induced design damage. Those led to the ‘TDD is Dead’ Google Hangouts between Hansson, Martin Fowler and Kent Beck, which I highly recommend watching. And in the midst of all of it, as a response to the GOTO Fail / Heartbleed incidents, ex-Googler, Mike Bland posted Goto Fail, Heartbleed, and Unit Testing Culture.

Hansson’s sentiments resonate with me. He is not anti-testing; he merely asks you to consider the trade-offs of the TDD approach because, despite what TDD zealots might claim, they do exist. Unit-testing can prevent bugs. Unit-testing can also unnecessarily complicate/mangle your design and cost time/money that you might not have. And, bugs may or may not be acceptable, depending on what it is you’re implementing.

The GOTO Fail bug led to a furor of responses which, in enlightened hindsight (or rather, in knee-jerk rhetoric), have provided an excuse for every man and his dog to bleat on about how their favourite technique/tool/preference/plush toy could have prevented it. Bland’s article above attempts to demonstrate how unit-tests could have prevented GOTO Fail / Heartbleed. He’s simply trying to remind developers that they aren’t hopelessly subjected to the whims of software (i.e. the indifferent and feeble ‘bugs happen / shit happens’ attitude) and do have tools at their disposal which can help…

I don’t take issue with the article and I actually don’t disagree that unit-testing might have been an/the appropriate measure. I take issue with the TDD fanatics who misinterpreted it in the worst way possible. It’s all very easy, after a bug has been exposed, to be Captain Hindsight and write a unit-test which reveals it. I find it incredibly insulting to collective human intelligence that zealots think they can extrapolate from this singular demonstration, that all people everywhere should be operating under TDD in earnest, regardless of all other factors, and that it would prevent any/all bug like this and lead to developer salvation worldwide. Maybe unit-tests would have been a good fit for this chunk of security related code? It does not mean that TDD is the most suitable solution for all codebases everywhere, or even a viable/sensible solution. It does not mean that I should be aiming for 100% code coverage if I’m writing a local community website which is used by 10 people a year.

Because you know what else might have prevented the GOTO Fail bug?

  • Static code analysis, which would reveal dead code after the second accidental GOTO… and for the record, this would be infinitely cheaper than unit-testing.
  • Using a language which offers try/catch/finally and does not require the goto/cleanup pattern which is idiomatic in C.
  • Using a functional language which would have eliminated this imperative sprawl altogether
  • Mandating the use of braces around if/else code blocks (not that I agree with this)
  • Regulations/protocol to ensure that developers are more careful when merging code (as that appears to have been the source of this fault).
  • Code reviews
  • Community code review poetry sessions, where speakers attempt to shoehorn all code into iambic pentameter

It’s not just the TDD fanatics who were being sanctimonious. In fact, that event caused all sorts of fanatics everywhere to come out of the woodwork and engage bleating. And this brings me to my more general gripe with the software world… the existence of these silver bullets which promise to solve all your problems, and the rampant disingenuous brain-dead evangelism which soon follows. Dijkstra says that if you’re exposed to QBasic in your nascent days of programming, that you’ll be mentally mutilated beyond hope of regeneration. I say that if you blindly latch onto anything touted as a silver bullet, you’re in the same boat. Let’s take a moment to reflect on these panaceas which have been misappropriated.

TDD, BDD, DDD, user driven development, XP, RUP, Agile, Scrum, unit-testing, design patterns, OOP, immutability, the Cloud, NoSQL, ORMs, CMM, IDEs, convention over configuration, XML over/for everything, IOC containers, prototyping, static typing, dynamic typing, basically every application/web framework ever invented… I’ve only worked for a few years as a full time developer, but I’ve already heard these pearls of wisdom, often from senior people:

  • If you’re not using TDD, you can’t call yourself a software engineer.
  • What? Automated user interface testing. That’s not real testing!
  • You need to be using XML. It’s not declarative otherwise.
  • How can you be developing C# without IOC containers?
  • How are people going to understand code you write without UML?
  • (In the context of OOP), private methods are just classes waiting to burst free.
  • How can software developers possibly be self-managing without daily standups with the PM?

Now here’s the thing. These methodologies are not useless – I personally use some of them on a daily basis. Many of them actually have useful and great ideas! Ideas which can fantastically benefit you from a cost/value perspective, if, IF they are applied properly. with discretion and judgement. Often they’re formulated by smart people who aren’t terribly dogmatic themselves! But somewhere down the line, they’ve been warped and abused…

These bullets are good in intention. They’re generally groups of related best practices / techniques / technologies / ideas which, when applied correctly to the appropriate problem, will yield benefits (on average). It’s easy to see why they exist. They usually begin as a reaction to address something which is perceived as nasty. They’re a symptom of the state of the software world during a particular period in time. And it’s easy to see how they become popular. A few people start using them to great success, and they quickly spread because they adeptly solve the problem they were engineered to solve. Even more so if books are being written about them and marketing departments are getting behind them. Architects/senior devs/managers catch wind of them and start using/prescribing them too. They work well in some cases, and the adopters become very excited about them and tell their friends. Junior devs join a company and start using the bullets which are already being used. They like them, as they provide some sort of direction in the scary jungle… it’s probably better than blindly spraying code all over the place, right?

So where do the problems begin? What makes these bullets such an alluring target for abuse, warping them into annoying fads which proliferate our airspace? What is it about them which brings out the worst in people? Why must I tolerate the existence of people who refer to themselves as ‘Agile warriors’ or ‘Agile ninjas’?

  • People often lack the judgement not to apply the same hammer to every problem in cargo cult fashion. From their point of view, if it worked for the last project, there’s no reason not to use it again for their next project/job… and this time round, it should be even faster/easier! There’s often little consideration as to whether the new problem is even remotely similar to the previous problem, or misunderstanding as to what makes a solution a good fit for a problem. Bullets get misappropriated. A good example is JS templating frameworks… it’s relatively easy to learn something like Knockout… it’s slightly harder to have the prudence to know when to use it, and when not to.
  • People tend to project their own tasks/constraints onto others. They don’t consider that the problem they’re solving might be vastly different to the gamut of problems that other people are solving. A zealot who measures a man’s worth based on his OOP skills probably isn’t even considering embedded/game programmers, who often have little need for it. Hell, I think web development has less use for OOP than many other areas (but that’s another argument for another day).
  • It’s easier to feel invested in your existing trove of knowledge and defend it vehemently than to learn new ways. Provided that a technique you’ve used in the past resulted in an at least tolerable experience, you’ll probably favour it over new unknown techniques which you / your company have to invest time/money in learning. And besides, there simply isn’t enough time in the day to evaluate all the bullets.
  • Most devs tend to get locked into a particular ecosystem (whatever they happen to use at work) and ignore everything outside it… this leads to myopic adherence to certain bullets (think Microsoft vs Unix stack, relational vs NoSQL etc.). This kind of insularity is particularly prevalent in Microsoft land (although things are improving nowadays!). And when the minimal group paradigm shows us that even the slightest distinction between groups leads to discrimination, it’s easy to see how fanaticism starts!
  • Devs often have a tendency to take things to a logical extreme, leading them to religiously adhere to every aspect of a bullet, rather than retaining the useful parts and discarding the chaff. Just because I’m utilising four XP practices doesn’t mean I have to use the other 8.
  • While these bullets do generally solve the problem they were intended to solve, they often create other problems which didn’t originally exist; case in point is OOP, which can solve many problems but can also wreak havoc when in the hands of architecture astronauts. These trade-offs are often ignored or swept under the rug.
  • Some ideas aren’t optimal on an individual basis, but on average, produce a better result. E.g. I’m not fond of the rule that a method should be no longer than ‘x’ lines. Even if you ignore the arbitrary limit and treat it as a fuzzy guideline, it’s still dangerously non-optimal in many cases which might be better served by your own judgement. However, if you’re on a team full of people whose judgement leads them to write tangled thousand line methods everywhere, then having a rule like this is probably preferable, for the greater good.
  • Misattribution of benefits. The best example of this is people saying that TDD improves their design and using that as a unilateral argument for applying TDD to all parts of a codebase all the time. Yes, TDD forces you to think about your design/problem (to some extent). And yes, doing that will probably improve your design. But that doesn’t mean that TDD is a pre-requisite for thinking about your design/problem up front! If you find yourself unable to, you should probably re-evaluate that crutch.
  • Consultants, marketing departments and tech authors all tend to benefit when the technology/trend they know/write about is thriving. It’s how they make a living, so they have every reason to get behind it, regardless of all other factors. And let’s face it, a well reasoned and objective evaluation of competing solutions in an area isn’t going to sell as well as an extreme evangelistic acclamation of a single solution.

Fortunately, I work at a company (or at least in a team) where fads do not run amok. Instead, we attempt to use our judgement in order to make decisions which are sensible with consideration of the task at hand and people involved. And because this has proven to be effective for us, it clearly means it’ll be effective for everyone. So I’m going to add to the development methodology dog-pile with one of my own. I call it the Use Your Brain methodology, and here are its tenets which I shall furiously impose on you:

  • Test your code if you think it needs to be tested. Would tests offer much value in this case? Would a failure in that piece of code affect many people? What would a failure cost? Would a failure be catastrophic in consequences – does the code deal with security, credentials, money? Does it tick enough boxes in your ‘criteria of things which tend to require testing’ to justify the development/maintenance cost of writing tests. Think about it.
  • Use whatever frameworks you feel are appropriate.
  • Document code if it makes sense to document it.
  • Refactor if you feel it’s necessary. What will it cost you if you don’t?
  • Substance over style. Think about it.
  • Manage tasks/estimations/quality in a way which makes sense. Think about it.
  • Operate in a way which makes sense given the strengths/weaknesses of your team.

Zzz, I’m getting bored of writing about this banal process – you get the picture. Now, I’m going to proceed to shove it down everyone’s throats! 🙂 And in the meantime, I’m awaiting the Next Big Thing .

About Nathan Pitman

Undergrad software engineer at the University of Auckland
This entry was posted in General, Rants and tagged , , , , , , , . Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *