Where Are the Testers in XP?
Where Are the Testers in XP?
With Extreme Programming, programmers are taking responsibility for writing their own unit tests. What work does this leave for testers? Some people think that XP saves costs by eliminating the need for testers. Does programmer testing really take the place of tester testing? In this week's column, Bret Pettichord offers ways for testers to provide value to XP teams.
I've found that there are at least three things that people can mean by "XP":
- what the books say1
- what teams coached and trained by the XP founders actually do, and
- what people who've only read the books are doing.
According to the books, XP includes unit testing (done by the programmers) and acceptance testing (done by the "customers"). Programmers use unit tests merely to verify that the software works as they intended. The acceptance tests are necessary to validate that the software actually works the way the "customer" wants it to.2
Specifically, the "customer" is expected to write the user stories (similar to use cases) and then write tests for the user stories. The books are a little vague on who the "customer" is, which is why I've put it in quotes. The "customer" is the person who makes the business decisions. Outside XP, this person is often called the product manager or business analyst.3
Even teams well trained on XP often have trouble getting this "customer" to take the time to write the acceptance tests. So they end up being written by programmers or testers and they are often written late. Indeed, studies have shown that acceptance testing is one of the least-well-adhered-to practices that make up XP.4 To help repair this deficiency, XP co-founder Ward Cunningham has recently developed an open-source
testing framework for acceptance testing.5
The XP books don't claim that programmer testing supplants tester testing. Rather they claim that the total cost of testing is reduced, largely by avoiding the surprises of the interminable testing phases that often plague the final stage of software projects. But testing still must be done, both from programmer and customer perspectives.
There were some early claims that XP was going to put testers out of business. This was partly hyperbolic-meant to inspire programmers to do better unit testing. But I suspect it may also have been a response to experiences with testers who were obstreperous and unconcerned with the success of the larger project. Indeed, Lisa Crispin fought some early battles to find a place of respect for testers on XP teams; she later wrote a book describing her methods.6
With test-last programming, there was no doubt of the need for testers. But the XP context forces testers to articulate how they can be of service to a team. Many testers, trained in unsuitable methodologies, will fail.
Indeed, some believe XP tries to minimize the role of the black-box tester because of bad experiences with ineffective testers. Specific complaints are that testers
- object to process improvement that doesn't conform to their traditionalist views
- focus on bugs that are of little importance to the project
- lack technical skills needed for serious contribution, and
- lack education and insight into modern development methods and architectures.
Cem Kaner has said, "Unless the level of technical skill in our field improves substantially, we can expect programming teams to continue to actively plan ways to work around low-functioning test teams. If we continue to combine low technical skill with process traditionalism, judgmentalism, and righteous political activism on projects, I think we'll see ongoing, reasonable, and justified sympathy for excluding testers from any serious roles in projects, among programmers, project managers, and development executives."7
If you are a tester on a team adopting XP, here's how to demonstrate your value to the team:
- show how you provide a helpful perspective in the definition of software expectations (whether you call them "requirements" or "tests")-a perspective that is difficult for either programmers or "customers" but that makes a real difference for the success of the project
- show how you can be satisfied with an information-provider role, not insisting on being a gatekeeper or quality policeman8
- show how you can adapt to an iterative methodology, changing direction as the project changes, rather than insisting that the team stick to the plan9
- show how you can function with a minimum of formal specifications, asking for more information when you need it and taking the initiative to document key information yourself when you see the need
Interestingly, these are not only the criteria for how testers can provide value to XP teams; they are also the ingredients for exploratory testing.10,11
Acknowledgements
Thanks to Ståle Amland and Elisabeth Hendrickson for comments on early drafts.
References
1By "the XP books" I mostly refer to Extreme Programming Explained, by Kent Beck and Planning Extreme Programming, by Kent Beck and Martin Fowler
2Correspondence on "Agile-Testing" mailing list, Ron Jeffries, 29 April 2002
3Planning Extreme Programming, Beck and Fowler, p. 17
4 "Circle of Life, Spiral of Death," Ramachandran and Shukla, in XP/Agile Universe 2002 Proceedings, Wells & Williams, ed.
5Framework for Integrated Test, Ward Cunningham
6Testing Extreme Programming, Lisa Crispin and Tip House
7Correspondence on "Software-Testing" mailing list, Cem Kaner, 18 September 2002
8"Don't Become the Quality Police," Pettichord, Stickyminds.com, 1 July 2002
9"XP, Iterative Development, and the Testing Community," Kaner, Stickyminds.com, 21 October 2002
10"What is Exploratory Testing?" James Bach, Stickyminds.com, 29 January 2001
11"A Survey of Exploratory Testing," Brian Marick
Comments
#1 Submitted by Neal Gamache on Wed, 02/04/2004 - 6:57pm.
To comment on Gregory Daich's post on developer skills and the author's response: my wife and I are both software quality assurance professionals, working in different companies but very similar environments. We debate periodically the merits of getting in-depth in the code and technical design of the products we test. Her approach is much more end-user-experience based; mine is more code- or developer-based. In her experience, if she follows the logic path the developer created, she makes the same mistakes the developer did and misses problems. In my experience, if I can see how the developer was thinking I can try to specifically find the cases they haven't programmed around. I think both approaches are important (and I'm not just trying to avoid household arguments).Ultimately, however, it is my firm belief that a trained tester, and especially a quality assurance specialist, has skills and abilities developers do not. I think learning development techniques is a major help for many testers, or I wouldn't have bothered with it myself. However, it is not true that an ideal tester would be a developer who spent their full time testing. If a developer can be cross-trained in testing or quality assurance, I don't see that the jobs necessarily conflict, unless you're talking about a developer testing her own code.I know in my company the developers were stumped about how to properly test their applications--the task seemed enormous, and they knew they weren't doing a good job. I'm not an amazing tester with just four years of experience, but I have picked up enough skills that I thoroughly amazed the developers with the problems I found that they were simply incapable of finding. I think a experienced developer placed in a quality assurance role would have as much of a learning curve as an experienced tester placed in a developer role.
#2 Submitted by MikeValletti on Thu, 02/20/2003 - 11:35am.
To the readers comment from 1/28/03 - "The day for sitting back and waiting for the "perfect" spec has long gone".... Please tell that to NASA. The comments provided on this article by readers seem to be in agreement regarding a mad rush away from facts, details, concise information. While the "perfect" spec has really never existed I don't think we should completely abandon attempting to get it and take whatever is given. Funny how XP was written by programmers....
#3 Submitted by Gregory Daich on Wed, 02/19/2003 - 3:41pm.
Bret, Thanks for the article. Keep them coming. I believe you hit on an extremely important issue as you listed some of the complaints about testers. Having testers work more directly with developers will help them gain some of the technical skills but they will still need training and experience in some modern development methods and architectures (if they want it and desire to work in environments that require it). Rotating testers with developers might be one way to help but I haven't seen many organizations who do this successfully. It is expensive to provide training in many developer skills to testers. Your comment from Cem Kaner stated, "Unless the level of technical skill in our field improves substantially, we can expect programming teams to continue to actively plan ways to work around low-functioning test teams...". Thus, the current paradigm (that I've seen in the industry) about software tester training really needs to change to include more developer technical skills. Do you have any further advice for obtaining that type of training and experience for testers? I started out as a developer but I have not had the opportunity yet to work directly with many new development environments. I have to admit that it is also very difficult to keep up with many relevant developer skills from a consultant/trainer (in software testing practices) perspective. Furthermore, many testers come from non-technical backgrounds who aren't prepared for training in many developer skills. Those folks may be able to best contribute to final acceptance/user-focused testing and may be less able to support earlier integration and many types of system-level testing (not to mention entirely unable to contribute to unit testing via pairing with developers or even in reviewing unit tests). Thus, there are distinct and limited tester roles that many testers can support who don't have a development background. Yet I am often asked to provide software test training to testers with developer skills and testers without those skills. It is a bit of a challenge.
#4 Submitted by on Sun, 02/16/2003 - 4:27pm.
As a member of an XP team, we've found that management and the team members are supportive of testing. But sometimes it feels like we're reinventing the wheel. We practice test-first design, but don't always know the best way to test a particular feature. XP is all about breaking down barriers of communication. It is not about slapping together some code and throwing it over the wall to the QA department to test. I believe that if we could integrate members of the QA/Testing group into the development team, the overall quality of the application would drastically improve. Testers could pair-program with the developers on a daily basis, helping them write better unit-tests. Additonally, testers could work more closely with the "customer" to write automated acceptance tests. Daily interaction would breakdown many of the typical barriers between developers and QA.
#5 Submitted by Mike Dwyer on Thu, 01/30/2003 - 10:04am.
Two Thoughts come to mind.First, relying software that used a two stage testing model of unit test and acceptance is akin to working in a building where the bricks were checked and the building was walked through. Like any construction effort the truly critical aspect in the quality and safety lays in the validating the robustness of the connections between the components, which is why the additional costs and time have been put into Module, System and Integration testing.Second, HOORAY for a coding approach that encourages key pounders to review what they did. I hope that the lessons learned in this go-a-round can leverage off the experiences our profession had building previous generations of killer apps using the 'crusty' code ASSM, FTRAN, RPG, PL/1 and COBOL types of the 60's, 70's, and early 80's as well as the giddy naiveté we had with C and C++.
#6 Submitted by EdA-qa Mort-ora-y on Thu, 01/30/2003 - 4:15am.
Perhaps Im the odd one out on this, but I really think that the programmers should be doing more of the basic testing. I consider it a waste of my time to have to discover and report bugs that the XP method should have caught during programming. There are lots of serious issues relating to specification, fault tolerance, business logic failures, systems integration and performance, that XP style testing typically will not identify, those are the types of issues that the quality control team should be looking for. Having to continually report stuff like "string too long causes error" or "negative number is not rejected" is both a waste of quality testing time and generally makes the job monotonous and boring for the testers.
#7 Submitted by Peter Hawkins on Tue, 01/28/2003 - 7:17pm.
Hi, Interesting article and a timely reminder to read up on XP. I am not sure of the current situation with the testing community outside of Australia, but I would make the point that I encourage testers to be far more than just the narrow definition of a "black box tester". As part of the defining testing scope and coverage I encourage my testers to utilise systems analysis and design methods to not only assist them but to also clarify for the UAT team how the solution answers the requirement. Very often, it is the System Test Team that fills the gap left by the absence of a systems analyst. I would also mention that a "Full Life Cycle" testing methodology that encourages testing from requirement through to delivery and beyond would appear to be at odds with the XP concept. Having delivered many testing engagements I find this disturbing.
#8 Submitted by Mags on Tue, 01/28/2003 - 5:06pm.
Hi Bret,I thoroughly agree with your article. The day for sitting back and waiting for the "perfect" spec has long gone. The only way we can really make a difference to the quality of an end product is to be flexible and help out. This doesn't mean lowering our standards, it just means being more pro-active. XP allows us to update our skills and be part of a team rather than sitting outside a team waiting for things to happen.
#9 Submitted by Hugh Lippincott on Tue, 01/28/2003 - 10:53am.
As a long time SQA/tester, I would be pleased to work with an XP team and not have to be a quality gatekeeper. XP teams that ALL own the code SEEM to offer enough group pressure to keep code and unit tests up to the standard that needs no gatekeeper. Testers so often are the first who see the code after the developer that they automatically end up in that role.
#10 Submitted by on Tue, 01/28/2003 - 10:51am.
Greetings! As an XP user and a QA Analyst for a mid-size software company, I must comment that there will ALWAYS be a need for QA "testers", regardless of how "extreme" the programming of smart apps like XP becomes, for these simple reasons:1. Developers simply don't have time to adequately test their code. 2. QA analysts are trained to think "out of the box", which exposes far more bugs than does less rigorous testing.3. Developers cannot maintain a "user perspective" in addition to the creative testing by the QA team, because they are typically far removed from the customer. QA, on teh other hand, usually deals with teh customer through technical support, hence has a better view of customer needs and product expectations.For these reasons and others, there will always be a need for Quality Assurance, especially if a company hopes to achieve some level of industry certification, such as ISO 9000. Thank you.
#11 Submitted by on Tue, 01/28/2003 - 10:07am.
Excellent article, Bret. I hasten to point out, however, that you have a checklist beginning with "If you are a tester on a team adopting XP, here's how to demonstrate your value to the team:..." I'd suggest that the phrase "on a team adopting XP" is useful in the context of this essay, but unnecessary in the larger scheme of things. Those are all important things for a tester to do in any environment.Cheers,---Michael B.
#12 Submitted by Gene Fellner on Mon, 01/27/2003 - 12:00pm.
Another Plug for the SEI CMM. CMM Level Two requires a Software Quality Assurance function--that works. The SQA staff review testing standards, procedures, staffing, training, orientation, integration into the SDM, interaction with the project team, project manager, end users and IT organizational management. Then it reviews conformance and achievement of the goals of the testing process. The CMM is perfectly compatible with XP and there is no justification for the contention that with XP "there ain't no need for that stuff." SQA adds value; this is a perfect example.
#13 Submitted by Raymond Ellis on Mon, 01/27/2003 - 11:50am.
I recently attended a seminar given by a QA group in the Metro Detroit Area. The speaker was the president of an XP programming company. The disturbing impression that I left with, after speaking with him, was that feeling that he thought that running through every unit test before RTM phase was sufficient. I have seen enough results of unit tests, or lack of, to know that having unit tests is a good idea. But don't expect much improvement in this area from people being pushed so hard by their managers to get code out, much less unit tests. A lot of them only have programming and little testing experience. This by itself affects how in depth as well as the quality of the unit test. XP managers should make sure that they see the testing department as their 'customers' as well as the business analysts. Testing beyond unit testing is still required. Thanks for the article.
#14 Submitted by Gerold Keefer on Mon, 01/27/2003 - 5:06am.
as some of the XP publications expose a lack of basic testing knowledge and skills within the XP community, i consider the necessity of skilled testers in projects more on the rise than on the fall. what makes testing in the XP context hard is the absence of clear definitons of what is what. for example there is a lot of talk about unit testing, however, so far i have not seen a definition of what a "unit" in XP is. a lot of folks will therefore end up in poking around and call this "exploratory testing". notworthy, XP has fostered the idea of test automation as a byproduct and JUnit is certainly an excellent tool, however, everybody involved in efforts to extend that tool will agree that a fundamental assumption of the XP movement, namely that "code is enough", is rediculous. regards,
#15 Submitted by Lisa Crispin on Thu, 01/30/2003 - 7:03pm.
Bret, I think you are right on with the emphasis on information provider role. I know of 'XP' teams that have created a separate 'test team'. Testers on XP projects should insist on working together with the programmers as well as with the business experts so that they can provide the value that you describe here. Providing information after development is complete is not nearly as valuable as keeping the feedback loops going all through the process.