Labels as inpediments to testing

Reading James Bach's post about labels for test techniques was a big relief. I find talking about what testing terms mean to be very limiting and frustrating. Very few "common" terms exist in testing and this makes it hard to talk amongst ourselves about what we do.

Names aren't bad, and they do facilitate discussion. Like them or hate them, programming patterns at least make it easy to have a conversation about common structures. Of course, this says nothing of people faking knowledge of the pattern under discussion. That is another post.

Establishing what terms mean for your team is a necessary first step. I'm going through this right now and so far, we have a lot of similar shared past experience and there is a lot of overlap and we haven't run into any serious misunderstandings or had any heated debates.

I second the recommendation of Gerald Weinberg's book. I need to read it again as a refresher on these topics.

Spend your time testing, not documenting

Every time I need to write a test plan or document test cases, I consider the role of documentation in testing. If you are in an industry segment that requires a lot of documentation, things are easy, produce the docs. If you aren't required by law to produce a certain level of documentation, it becomes harder.

The most consistent advice I've found is to document to the extent that you need to so you can give as good an accounting of the testing as you are required to.

When your project leadership asks what the state of the software is, can you give a good answer based on test data?

Do you know if you have addressed the risks in your dev cycle as best you can given your resources?

As I've mentioned previously, I'm building a new team and we are finding our way on processes. Documentation came up very early as something we would need to figure out. I keep spreadsheets and notebooks of my testing. Along with emails, bug reports, and a lean test plan, I feel like my work is adequately documented for our project. However, the other tester on my team wasn't documenting as much. So, when a chunk of functionality he had recently tested came back around after a major refactoring, I asked him to document his test cases in a spreadsheet. Basic, lightweight test documentation I'm sure most testers are familiar with. We are using Microsoft Team Foundation Server, but I'm not yet comfortable with the test documentation features to use it yet. I want some flexibility to enable us to learn more about how we are going to work with our test case assets.

Anyway, a few days passed and I went to check up on the testing progress from my other tester. He hadn't gotten very far.

"How much testing and how much documenting are you doing?"

"About 50/50".

Oh.

That struck me as too much documentation for our team. 80/20 seemed better. Our job is to give timely feedback to the dev team and get as much testing as we can get done. I'd like to think 80% testing and 20% documenting is a good balance. Analyzing the software and coming up with test cases is a separate activity that falls outside of this time recommendation. The 20% spent on documentation is actually writing out the tests, filing bugs, taking notes, etc. as you test.

I don't yet know how this will work out, but for now this is the heuristic I'm going to use.

Nice job kid!

One of the questions I ask in my phone screens for potential new tester hires is a classic: give me the test cases to test a Coke vending machine.

http://www.5min.com/Video/How-To-Hack-a-Soda-Machine-2497

I think I know where to look for my next hire!

The Ongoing Struggle with Tool Acquisition

Once again I find myself in the situation where a new tool was adopted without first seeing if it was able to support the end-to-end use cases of our team. This is not an uncommon problem, but is always frustrating.

We are using Team Foundation Server and MSBuild is the problematic piece. I'm a big believer in a "one button build" that goes from clean machine to installation media. If any step in-between fails, the entire build is considered to have failed. We are using WIX to create our installer and between it and MSBuild we are not able to sign our builds or have the build fail if the binary or installer build fails. So, we had to split things up and build the binaries and installer separately. And then do some deleting of binary builds so the installer build fails correctly. It really lowered my confidence in the builds. And good builds that you believe in are a foundation of testing.

I've seen the same problems with 3rd party libraries. "We don't have time to build this ourselves so we need to buy this fancy huge library package." Then it turns out you use 5-10% of the library and you would have been better off just building that small piece. Now you are now tied to the vendor's release cycle and Murphy will always make sure there is a bug right before you ship but six months away from the vendor's next release. Finding bugs in 3rd party code is my least favorite kind of bug.

I don't see the issue of tool acquisition getting any attention in the talk about what teams need to do better. Somehow were are all staggering around, grabbing things off the shelf that look about right, and suffering the consequences.

It is a lot of work to qualify a tool. It is just like testing your own applications. When has your team treated it as such and written acceptance tests for a new tool you are considering?

"It is not possible to tell the difference between high-quality code and poor testing."

For it is a dictate of common sense, that we can be under no obligation to do what it is impossible for us to do.
--Thomas Reid

"...defect removal in total has been the most expensive cost element of large software applications for more than 50 years."
--Capers Jones Estimating Software Costs

It's fair to say that current testing theory is based on the premise that "complete" testing is impossible. With this given, the idea is to make the most intelligent and practical selection of tests to run from all available tests given the time and resources available to you. The testing processes and techniques we employ in our projects are all a result of this practical necessity. We spend a lot of our time thinking about, and performing testing as the main activity to improve quality.

A new book, The Practical Guide to Defect Prevention, makes the argument for giving more attention to defect prevention. It seems to make argument that testing is necessary and fine as far as it goes, but it doesn't go nearly as far as we think given the amount of effort that we are willing to devote to it and the results we have gotten as an industry.

The authors argue that defect detection is the first and most common level of activity that teams undertake to improve software quality. "Test quality into the software". Many defects are detected this way, but nothing is done to prevent them recurring elsewhere in the application. The next level of software quality improvement is analysis. Previous defects are analyzed to see trends and find out why they weren't detected earlier. The long term goal and highest level is prevention. At this level active measures are used to identify and eliminate potential defects. The root causes of defects are found and used to eliminate them before they are introduced.

"The ultimate goal of any quality improvement effort should be to enable the development team to invest more time in defect prevention activity."

The low efficiencies of defect removal explain why there are often a long series of defect removal steps in software projects. When unit testing, function testing, integration testing, and system testing find less than 50% of the bugs present in an application, it is reasonable to question your quality improvement efforts and see if you are spending your time and resources most productively. I'll be giving defect prevention more consideration in my projects.

Should testers write "unit tests"?

"Some software is very difficult to test manually. In these cases, we are often forced into writing test programs."

A few weeks ago, one of the devs on my team went to the Patterns and Practices Summit where he heard James Newkirk say that testers should write unit tests. For some reason writing unit tests is a very seductive idea to me (because I enjoy it?), but I have a gut feel that testers shouldn't write them. I worry that it shifts a responsibility from the devs to the testers that the devs should reserve. Shouldn't the author of the code unit test it?

At PNSQC, Michael Bolton said that unit testing is "confirmatory testing". It is testing that is looking to prove that things work like the author intended and everything is fine. A unit test author doesn't want to see things go wrong. They want to see things go right, keep that bar green! A tester would want to see a red bar. This positive attitude is fine, as long as those involved are aware of this dynamic.

In the meeting I spoke up and said I wasn't sure about testers writing unit tests. The dev manager then described what I would call a functional test and said he thought testers writing this type of test would be fine.

Right away, we run into terminology issues which make taking about "unit testing" harder than it might otherwise be.

What is the definition of a "unit test"?

*A test that verifies the behavior of some small part of the system?

*A developer or programmer test. Contrast with a customer test.

"In computer programming, unit testing is a procedure used to validate that individual units of source code are working properly. A unit is the smallest testable part of an application."

Tests that the devs write such that the "...units being tested are a consequence of the design of the software, rather than being a direct translation of the requirements."

Now I'm confused. Since I'm not totally sure what a unit test is, how do I know that testers shouldn't write them? Hmm.

Well, I think of classic unit tests where the author writes some code, has some methods, and if it is OO software, some classes as a result, and then writes standard tests for those. Good solid tests like invalid values, boundary tests, that kind of thing. This seems like stuff the dev should write.

Stepping away from this, what about when you start to combine some subsystems and you are analyzing larger functionality that might be recognizable to a customer as at least a small part of the functionality they are expecting the software to provide to them? This starts to seem like it would be fair game for the testers to write this type of test. Are these unit tests? I don't think they are, at least not against what I think is the current understanding of unit tests. Maybe where SDETs are common, this type of boundary is understood better and they don't worry about it. Elsewhere, I think it is a gap that might often be missed.

Code reviews vs. Testing

On the Coding Horror blog today Jeff reminded us about these statistics from an earlier post of his regarding the gains from code reviews:

But don't take my word for it. McConnell provides plenty of evidence for the efficacy of code reviews in Code Complete:

.. software testing alone has limited effectiveness -- the average defect detection rate is only 25 percent for unit testing, 35 percent for function testing, and 45 percent for integration testing. In contrast, the average effectiveness of design and code inspections are 55 and 60 percent. Case studies of review results have been impressive:

* In a software-maintenance organization, 55 percent of one-line maintenance changes were in error before code reviews were introduced. After reviews were introduced, only 2 percent of the changes were in error. When all changes were considered, 95 percent were correct the first time after reviews were introduced. Before reviews were introduced, under 20 percent were correct the first time.

* In a group of 11 programs developed by the same group of people, the first 5 were developed without reviews. The remaining 6 were developed with reviews. After all the programs were released to production, the first 5 had an average of 4.5 errors per 100 lines of code. The 6 that had been inspected had an average of only 0.82 errors per 100. Reviews cut the errors by over 80 percent.

* The Aetna Insurance Company found 82 percent of the errors in a program by using inspections and was able to decrease its development resources by 20 percent.

* IBM's 500,000 line Orbit project used 11 levels of inspections. It was delivered early and had only about 1 percent of the errors that would normally be expected.

* A study of an organization at AT&T with more than 200 people reported a 14 percent increase in productivity and a 90 percent decrease in defects after the organization introduced reviews.

* Jet Propulsion Laboratories estimates that it saves about $25,000 per inspection by finding and fixing defects at an early stage.


Every time I see these statistics I get a little nervous because it seems like someone is trying to make an argument that tries to diminish the contribution testers make to a software project.

"Darn, we don't do code reviews, so I guess we have to have testers..."

But then I remind myself the issues are deeper and things are never as easy as they sound.

Jeff's previous post and the current one, make it sound like you just snap your fingers, start doing code reviews, and your problems are gone.

Problem is, reviewing code well is a hard skill to possess. Instead of the 80/20 rule it seems like the 20/80 rule: you spend all your time (however much you have) on code reviews finding the easy 20% of the bugs you could find while the other 80% lay dormant.

Part of what bugs me is the statistics and when they were taken. Different times, teams, languages, projects than yours. This doesn't wipe out the data, I still believe it, but it doesn't translate straight across either.

They say it takes 10,000 hours to become an expert at any particular thing. That works out to three hours a day for 10 years. How long have you spent doing detailed, thoughtful, deep code reviews? Here is the start of the problem.

I believe in code reviews and was very enthusiastic about them on a previous job. I worked really hard and got our team to start doing them. Then, ran into the problem that no one had developed the skill to get anywhere near the results of the statistics. And no one was interested in working very hard to improve their skills. Our code review experiment died a quiet death.

I do like being reminded of these statistics. Which ends up reminding me code review is a hard skill to learn and we all probably need more practice.

Toward that end, Scott Hanselman has a series on quality code to read. Worth checking out as a start.

Can I see you do that?

I wouldn't have an interview process that didn't include an audition. I've done about ten tester interviews with auditions now, and it tells you a lot about the candidate. You really see how they think.

For the audition, choose a small program to test. You can write it yourself or use something already out there. It doesn't have to be elaborate. You might add a pretend spec or readme.txt alongside the program.

The most interesting thing is how many different approaches people take testing the program. If ever you needed a rationale for having a team of testers, this is it. Get several people looking at software and you'll get some very different approaches. Some people dig into the spec and take it as the only truth. What it explicitly states is law, and what it doesn't mention is forbidden or unimportant. Others dive in and start typing in values for the sides of triangles. Some try special characters. Everyone does a lot of boundary testing. Some have a UI testing eye and make notes of proper tab order or improper resizing. None have run all the test Meyers notes in his book.

One of the best parts is at the end when I ask how much of the program they have tested on a scale of 0 to 100. Candidates have taken twenty to forty five minutes to test the program. I've gotten answers ranging from "10" to "90" (not correlated with longer or shorter times testing).

There is a lot of talk about interviewing and how imprecise it is. I don't subscribe to the "Blink interviewing hypothesis" entirely, having seen what an audition can show. While an audition won't tell you how hard the candidate will work, it does give you an idea of where they are at. Are they too trusting of authority, be it a spec or a developer? Do they run down all the leads they find while testing or stop at the first bug found?

At the PNSQC conference this year, Jon Bach gave a great presentation about traps testers fall into. In his presentation Jon lists the "Top Ten Tendencies that Trap Testers":

#10 Stakeholder Trust
#9 Compartmental Thinking
#8 Definition Faith
#7 Inattentional Blindness
#6 Dismissed Confusion
#5 Performance Paralysis
#4 Function Fanaticism
#3 Yourself, Untested
#2 Bad Oracles
#1 Premature Celebration

These are really good traps to consider when evaluating candidates or yourself.

Test plans?

I'm still not sure how I feel about test plans. As with all documents that help you think, the thinking part is great. I'm not sure how much value the plan as a document has once you are over the first thinking hurdle.

Of course, you should keep it a living document, but at what point does the maintenance cost exceed the remaining value of the plan? It's great to get you going, but at some point you know exactly what you need to do and the plan isn't necessary any longer. I've seen problems where this cutoff point wasn't respected and time was spent on the plan that would better have gone to testing.

I've had a lot of freedom in the test plans I've written. I've followed the IEEE template and I've made up numerous ones of my own. They all work if they get you to ask the right questions.

What exactly are we testing?
When do we expect the testing to happen?
What do we expect the testing to show us?
What are we going to consciously ignore because of project constraints?

Some of my development managers didn't care if I wrote a plan or not as long as the right testing got done. Others seemed to want the plan more than the testing. Biggest mistakes I've made are when I didn't realize what kind of manager I was dealing with in this regard. To some development managers, testing is a mystery and a plan removes some of that mystery. To others it is a false security blanket they'll later turn into a hammer and beat you with when you "missed" a bug that you should have "obviously" caught. No mention is made of the developers who coded the bug of course.

Maybe I know a little better what I think of test plans after this post. They are complicated and can get you into trouble if you aren't careful. If they ask the right questions they earn their keep, but don't let them eat into time better spent testing your software.

Hiring in a tight market

Right now is a great time to be a tester looking for a new job. The job market is really tight and it seems like one could take their pick of positions. The flip-side is that it is an awful time to be looking to hire testers because they have the freedom to be selective. The pool of developers seems to be much, much larger than the pool of testers out there. Probably no surprise, but if you are looking for job security, development might not be the obvious choice.

Seems like there are three major ways to source candidates:

Recruiting firms
Craigslist
Networking

I think you just might need all three.

Networking is probably the best, but also the hardest. It's great if you know solid people who know other quality testers. However, these people don't necessarily move around much, so they aren't the asset they might appear to be. But don't get me wrong, if someone in your network is looking for a new job and you are hiring this could be the ideal situation. Just don't hold your breath that your hiring window and their availability will overlap.

Some people don't network professionally, and don't use recruiting firms, so you'll find them from Craigslist. You'll also find the musicians whose gigs have dried up and are looking to make the move into software testing. But occasionally a great candidate will contact you and it makes it worth it. I just wouldn't put too much time into it. Also, I suspect there is a killer job description that attracts just the candidates you want. Unfortunately, at least for my open position, I'm unaware of what it is.

Recruiting firms seem like a good deal. They do all the legwork and send you perfectly matching resumes for your openings. You put your feet up on your desk, maybe grab a nice cup of coffee or a lemonade depending on the season, and spend a lazy morning deciding between candidates just so perfect you wish you could hire them all. That's the dream anyway.

These firms can send you people but they tend to specialize in the high end. If you need mid-level or junior people, they probably aren't going to be as helpful. Working with a good firm can be nice, but it isn't a guarantee of quick hiring.

When they say you should expect to spend three months filling a position, believe them. In a tight market with a small pool of specific talent, I might add some to that estimate. How much? I'll let you know when I get my hire. I'm going on two months...

Welcome

I've started this blog to talk about software testing and managing small testing teams.

It seems like testing is asking more of those who are working in the field because of the complexity of modern software and the challenges of software projects today.

I hope this blog can be an interesting conversation about what works and what is and isn't true any more in the field.