Is It Responsible to Not Test Your Code?
The Agile Weekly Crew discuss when it is responsible to not write tests.
Derek Neighbors, Jade Meskill, Clayton Lengel-Zigich and Chris Coneybeer have a conversation on when if ever it is responsible to not test.
Obie Blog Post
Crazy Bob Martin
Test everything all the time
To what degree
Not breaking things
Verifying value delivered
Market vs quality
What’s right level of coverage
BDD as a technique
How much risk can you tolerate
Whats your testing comfort level
Communities have different testing tool competency levels
Derek Neighbors: Hello, and welcome to another episode of the Scrumcast. I’m Derek Neighbors.
Jade Meskill: I’m Jade Meskill.
Clayton Lengel‑Zigich: I’m Clayton Lengel‑Zigich.
Chris Coneybeer: I’m Chris Coneybeer.
Derek: Today, we want to talk about is it ever responsible to not test?
[all men gasp]
Derek: I had to pick the guys up off the floor but…
Derek: This has come up a number of times. I remember personally arguing about this with James Shore and a few other people at, I believe it was, Agile Roots probably two years ago, maybe 2009. Then, recently, OBIEEs brought this up in a blog post as he’s doing his own start‑up, and he talks about maybe he shouldn’t be testing for his own start‑up in some ways, shapes, and forms.
I just wanted to, obviously, be in a group that’s pretty adamant about testing and pretty focused on quality, what we think about that.
Clayton: I read an interesting post the other day that was supposed to be an argument against TDD. It was one of those things where half of it was drawn in, and then the other half of it was, “To prove my point look at how crazy Bob Martin is. When he talks about testing, he’s crazy. That means testing’s crazy.”
Clayton: It was funny, but there were a lot of people that had commented on it that were like, “Oh, I’m so glad that someone finally came out and said this,” like it was some Oprah show or something where someone was revealing this great truth.
I thought that was interesting, but at the same time, no one was really getting into the like, “Well, hmm. Maybe it isn’t responsible to test everything all the time,” counteracting the Bob Martin or some of the people that are really into the test‑all‑the‑effing‑time kind of stuff. There’s something to be said for that.
Jade: A lot of it is the degree of which you’re testing. Advocating a position where you don’t test at all is very dangerous and completely irresponsible. But do you need unit tests that cover every single edge case, as well as, acceptance tests, as well as, request testing? There are so many levels of testing that you can get into, especially with a web‑based application.
When you’re running a Lean Start‑up and you’re in move‑fast mode, totally experimenting, and trying to figure things out, it can be quite a bit of overhead to do a complete full‑stack top‑to‑bottom, outside‑in testing.
Chris: Even if you’re doing your Lean Start‑up, there should still be some testing in there for acceptance. There should be some testing around what is the core criteria of the application I’m writing for, not of the unit testing, testing out every single line of code. I understand the coverage being a lot lower.
But I still think that, no matter what, if you’re doing acceptance tests, even integration tests and depending on what you’re building, what you’re doing, you’re going to think about a lot more things, and have more confidence in that code. Not confidence in being perfect but confidence in your not going too far away from what you’re trying to get to.
When you start writing everything with no test ‑‑ I just would have no faith that I couldn’t break something and not be aware of it. Then, if I’m a starter that could have a bad presentation to any end users or people I’m showing the product to.
Clayton: I heard a good way of describing testing, and a lot of people think of testing, especially if you read stuff on Twitter, like TDD. It’s all about, “Oh, TDD is so great because I realize that I broke something before I got it to production,” or whatever. So, testing becomes all about making sure you’re not breaking things.
The next level of that is, the code I heard was, “You should be doing testing so that you can verify that you’re delivering the value that you say you are with your application.” That would be really important if you said, “From my Lean Start‑up, I’m going to test the value parts and the rest of it is…” Maybe, I don’t do that. Maybe, not test those things. That could work.
Jade: One of the arguments that came into is, when you’re funding somebody as an investor, the truth is you don’t give a shit about testing. They want, you just don’t care. What you care about is, can you compete in the marketplace? At some point, testing or quality factors into that. But sometimes getting to market or getting to discovery, quality is not the most important thing.
So, if I say, “Clayton, I’m going to give you a hundred thousand dollars and I need this list of 15 features. If you can’t get those 15 features done, regardless of what the quality of those 15 features is, this project’s done and we’re not going any forward, and ending. If you deliver those 15 features to me, regardless of the quality, if you can demonstrate on some level that you can complete these 15 features, I’ll give you another million dollars to continue on with this product.”
That, in a Lean Start‑up mode, is where it starts to become a little difficult to say, what is that threshold? Is it OK to go four weeks without tests? Is it OK to go 10 weeks without tests? Is it OK to go 90 days without tests? Is it OK to only have 10 percent coverage or only have the value items covered?
I mean, there is a point and you could say the same thing, even not on Lean Start‑ups. When you’re in large enterprises that have millions of lines of code with no test and you want to do new development and it’s not reasonable to go back and write tests for every line of code that’s already written, at some point you have to make trade‑offs to say, “What’s the acceptable amount of testing that is responsible.”
That’s a word we use a lot but we drop real quick when we get into our wars, when it’s all testing all the time no matter what. We drop the whole like, “What’s responsible?” Maybe, we could talk about. What are some of the things that we see that are responsible or, maybe better, what we see that is irresponsible?
Chris: For me, I really embrace the core idea that BDD, a behavior driven development, is less about test coverage and the percentage of testing, but more as a technique to help me discover the problem I’m trying to solve. So when I approach it from that angle, I’m not really interested in necessarily ensuring that every single line of code is covered.
I’m more concerned about, “Am I creating a quality design that is good enough to solve the problem that I’m trying to solve?” There’s a bare minimum level of test coverage that you can implement when you’re going down that path that is going to cover you in 90 percent of cases, especially, as your software continues to grow and add new features. You have some level of confidence that you’re not completely destroying the application by changing a few things around.
So when I’m testing in Rails, I really like to do acceptance testing, some more outside‑in testing. I’m coming in as a user and following the path and making sure that things are working the way I anticipate them, but I can change around a lot of the implementation details behind the scenes without having to go back, refactor all my tests and worry about this whole gigantic test suit.
There’s a bare minimum level of coverage that you can get where the testing is pragmatic and simple but good enough for the time being. It really comes down to risk. How much risk are you willing to take and how much risk can the product or project that you’re working on afford to take?
If you have a hundred thousand dollars for the start‑up and it’s make it or break it, you can take a lot of risk with your software because the bigger risk is that you don’t deliver anything. If you spend all your money and engineering time doing testing, you’re not going to get to market anyway and it was just a waste of time
Derek: On your thousand dollar example, if you treat it as a prototype, people readily acknowledge that you should build the prototype and throw it away. If you can be responsible to do that, you should also recognize that maybe doing a full testing is irresponsible, because that’s something you’re going to throw away.
At the same time, I’m kind of torn, because I know that when you get to a certain point, I acknowledge that, there are benefits that you get from, say, TDD that are like architectural decisions. You get other benefits that are not really what you’re specifically testing. Those are harder to get and more people think that they understand those and get those than they do.
I could see a lot of people saying, “Well, you know, I do TDD because I have to test everything, because we’ve got all these great TDD benefits,” when maybe, they’re not really getting those and it’s a prototype that they should be throwing away anyway.
Chris: How many people do you think have the discipline to actually throw away that prototype?
Jade: To me, where the argument comes