Edit Those Epics
Edit Those Epics
Stories Don’t Span Iterations in Agile
It can be tricky for managers and technical leaders to make the transition to agile. They’re likely accustomed to doing things a particular way. What’s more, they may try to squeeze what their old ways into the new, agile approach. Here, Johanna Rothman describes why that isn’t a good idea, especially regarding stories that are too big.
I've been working with folks making their transition to agile. One of the hardest transitions is for the managers and technical leaders.
Managers are accustomed to working in timeboxes. To them, the iteration is a timebox. But, they also are accustomed to features spanning multiple timeboxes, and that’s not OK in agile.
They are accustomed to predicting the end of the project, and they now want to use the team's velocity and the story sizes to predict the end of the project. That’s OK, but it’s not always wise. It assumes nothing will change, but agile is for fast change. The managers' fixed mindset is bumping up against the technical team's change mindset.
This leads me to a root cause. If you think your job is prediction, you don't change the size of the stories, and that means the stories are too big. The stories are epics.
It's fine to start with epics—very large stories. But, in order to have releasable product at the end of an iteration and to keep iterations short enough to get feedback often enough, you need small stories. That means you'll start with epics and decompose them into stories. At first, this seems impossible.
Here's one way to start. What's the first valuable thing—the smallest chunk that delivers business value to the customer, to the developers, to the testers, to someone—that we can do that we can show to someone at the end of a short iteration? Is that small thing a story? Is it small enough to be completed inside an iteration, maybe by the entire team? If so, complete that story in the iteration. If that thing is not a story, what would make it a story? If it is not small enough, can you decompose it further?
Making stories small enough to finish in a one- or two-week iteration is a common stumbling block for many teams in their transition to agile. Until a team can do that, they can't move to agile. They can't take advantage of kanban either, because their stories are too big—nothing will move across the board.
This thinking challenges several assumptions on the part of managers and technical leads:
- That there is one and only one right person to do that work. I'm not saying people are fungible. I'm suggesting members of a team swarm around the work.
- That work has to follow a specific order. I'm not saying the work is order-less. I'm suggesting you may have more flexibility than you think at first glance.
- That work has to be a big huge chunk in order to be done. I know that you can work in inch-pebbles. You have to do something first. You have to do something next. I know this.
- That work must only benefit one persona. I'm suggesting work can benefit not just the eventual customer, but the developers, the testers, the marketing people, the trainers, and even HR or legal. Please do not start your stories with "As a customer." Which customer?
Managers and leads are accustomed to thinking big about projects. They have needed to, in the past, because their managers asked them to estimate the entire darn project or, even worse, the entire program to see when it might end. They think they need to know velocity now to estimate the entire project. They still might.
But, they can't do that if stories span iterations. They can only be successful if the teams complete stories inside iterations. Teams cannot get partial credit for stories. Partial credit perpetuates work in progress, which is a very bad thing, no matter what agile or lean approach you are using.
If senior management needs to know how long the entire project will take, the best way is still to explain, "We need three iterations to be able to estimate. We will track our velocity for three iterations and give you a reasonable three-point estimate at the end of that time, OK?"
Anyone who won’t wait six weeks, or three two-week iterations, for a reasonable estimate wants a SWAG—a “scientific, wild-tush guess.” In that case, you can say, "Christmas," and don't provide a year. This is the "Happy Date" schedule game. You’ give them what they want to hear, because reality will intrude soon enough.
If you are transitioning to agile, here are guidelines I have found useful:
Make your stories small enough so that you, the team, can finish more than one story inside an iteration. My rule of thumb is that you should be able to complete a story every day or, at worst, every two days. If you can complete stories more often, that's even better. If you can't complete a story at least once every two days as a team, your stories are too big. Yes, there are exceptions, but you'd better have a darn good reason.
Remember that velocity is personal to a team. As soon as you change a team's makeup, velocity can change. So, don't change the people on a team if you want any sort of predictability about estimation.
If you really want some predictability, make the stories really small, so that people can pair on them and still finish them in a day or less. Then, all you need to do is count the stories to have a rough idea of how long the team needs to finish the project. "That's a lot of stories, JR!" Of course it is. That's why you're paying people to work on the project.
I no longer estimate my work. I make sure my chunks of work are tiny. That way, I just keep chugging along. That would work for your team, too.
But, whatever you do, give up the notion of stories spanning iterations. If your stories are big enough that they need to span iterations, stop right there. Stop with the partial credit. Put those stories on a diet. Separate the stories—not into tasks, but into multiple, independent, smaller stories that the responsible person or product owner can schedule into the most appropriate iteration and rank when it's time for that story to be implemented.
Comments
#1 Submitted by David Chassels on Sat, 12/10/2011 - 7:45am.
A new paradigm changes "agile"
Trick is to eliminate need for coders so the gap between developer and user is eliminated. Fact is business logic never changes it is people working individually and collectively to create business outcomes by doing something. This recognises the fact there are relatively few generic task types. This is “agile software” and represents a new alternative to COTS and custom coding for business solutions
Users are encouraged to contribute not just at the build but subsequently as change is readily supported as the core code never changes and no code generation or compiling. That is the "new paradigm" - read about it here on one pager to help buyers http://bit.ly/nQOAzE .
Sadly the IT business software industry has evolved in a chaotic and selfish manner that has resulted to two unsatisfactory characteristics. First is the complexity involving so many components to build solutions and perhaps the more serious is the resultant domination by a few large players whose "innovation strategy" is articulated in this article http://linkd.in/uqiXVE What Microsoft, Oracle, IBM, And SAP Don't Tell Customers.
"Agile" is an attempted to bring business and developers closer and as usual has spawned experts. But this is treating a symptom not curing the problem. Well in UK a minnow innovator Procession has cracked it and it is well proven. Build of solutions with all required business functionality including the most important the user interface. It is the start of a new journey with more to come. Legacy is used as required with the core focus on people working together to achieve their individual and collective outcomes. Accurate estimates can be given by counting the number of user forms in the process and that becomes the number of man-days to build the whole solution including workflow, rules etc.
Agile is a sound way to handle projects but many of the issues raised are readily overcome with the real step change to “agile software”.
#2 Submitted by Anonymous on Sat, 10/29/2011 - 5:27pm.
Good
Good
#3 Submitted by Sam Davis on Wed, 10/26/2011 - 7:01am.
Challenging the group think syndrome
To be quite honest (blunt actually), I'm tired of the Agile elitist attitude presented in this article. What do I mean? That if you are trying to project schedules, you just don't "get" the Agile concept. Maybe some of the Agile elitists just aren't "getting" it yet and eventually, they'll realize the fallacies, and even contradictions, in their sermons and they'll come around.
For example, as recently as a couple of years ago, the Agile world said that story points should be estimated purely on complexity. I was saying then that didn't make sense and we estimated story points on complexity, duration and risk. Now the mainstream is coming around.
But, we're still hearing elitist platitudes like this article preaching from a high tower of Agile wisdom. Breaking news... especially in software development companies, it's a business reality that Product Managers MUST have some type of anticipated completion date in order to arrange beta clients, implementations, etc. Agile is supposed to be focused on business value but current thought completely ignores this business need.
Another example of contradictions in current thought and in this article: User stories are supposed to provide some business value. This is extremely important because the Product Owner manages the backlog of user stories. In some instances such as building a new system, there is a minimal amount of functionality that MUST be provided in order for the end user to have ANYTHING of value. More often than not, early in the project, there is NO way this can fit into a single iteration. If you start artificially breaking down the first piece of usable functionality in order to comply with the Agile "rules", then your Product Owner starts prioritizing things that doesn't make sense for them to prioritize. The only reasonable thing to do is to break it into tasks and let the delivery team prioritize and schedule the tasks. Then there are a couple of things you can do about using velocity for planning purposes. One is to look at a longer term average. The other is maybe Agile blasphemy in your eyes, but we use it and it works... use metrics based on velocity of completing user stories AND tasks. BTW... this does not prohibit showing functionality at the end of the iteration and getting feedback, so the whole argument about frequent feedback doesn't hold water.
The process should serve the business, not the other way around. Agile is a tool to accomplish business goals and objectives, not a religion to be worshiped and making the business compromise because of artificially imposed process rules. Let's work on refining Agile to better serve the business rather than treating the original manifesto like it's gospel and it's blasphemy to question it and improve it and treating those who challenge it like they are inferior.
#4 Submitted by PhilJ on Wed, 08/08/2012 - 5:16pm.
Sam Davis
Well expressed Sam --- you're exactly correct in your appraisal.
#5 Submitted by Johanna Rothman on Wed, 10/26/2011 - 9:15pm.
Not about religion
Sam, I'm disappointed you think I was religious in the article. If you have epics, and your stories are larger than your iterations--something I see a lot in new-to-agile teams--it's useful to learn how to break the epics into stories. I don't see how you can accurately estimate a project composed of epics.
Once you have finished 3 iterations of stories you have some history. Now, you have some velocity to provide an estimate based on data. If you don't want an estimate based on data, you can provide an estimate that's a SWAG, which is probably what you do now.
How is that elitist? I am missing your point.
How is breaking stories into vertical story slices (which I did not say) not useful to the product owner? Have you tried it? If you have tried it three times and it hasn't worked for you, okay. But if you haven't tried it at all, and you only think it hasn't worked, then please don't claim it can't work.
You say that early on your new projects it's not possible for you to do anything of value that you can complete in an iteration. I'm paraphrasing you here. I have encountered that attitude on teams early in their transition, and I have found that if we rethought their stories and rethought their architecture, they could do useful work that gave them more architectural flexibility early in the project. They had made too many architectural decisions too early and that's why they could not finish small stories. However, I don't know your project, and maybe your project is different.
I am a big believer in doing what works for you. But here's what I don't believe:
- I don't believe that stories that span iterations works for teams.
- I don't believe you can take partial credit for stories.
- And, I don't believe you can accurately estimate epics for a project and provide anyone a single point project estimate.
We can agree to disagree. In fact, I invite you to write an article about what you do that does work. I will even work with you to publish it. I can only tell you what I have seen that does and does not work in my practice.
#6 Submitted by Sam Davis on Fri, 12/09/2011 - 10:43am.
Stories vs. Tasks (continued)
jr>> I don't see how you can accurately estimate a project composed of epics.
sd>> By breaking the stories into estimated tasks and having two sets of velocity metrics. One is business value delivered based on story points completed. The other is work completed based on task story point estimates which gets you to your velocity. It goes against your rule of "partial credit", but what the heck? It works. And your business value metric still includes only what was delivered of value.
jr>> How is that elitist? I am missing your point.
sd>> Because it's imposing a rule with no justification other than "that's what Agile says" and refusing to consider a better way of doing things.
jr>> How is breaking stories into vertical story slices (which I did not say) not useful to the product owner?
sd>> If the PO requires a minimal amount of functionality to provide something of value, it makes no sense and actually frustrates the PO to force them to prioritize things in smaller slices.
jr>> They had made too many architectural decisions too early and that's why they could not finish small stories. However, I don't know your project, and maybe your project is different.
sd>> Actually, I'm not even talking about architectural work. I'm talking about functional code. I am 100% convinced that at times you cannot provide things of true value to the end user in a single iteration. You may be able to provide pieces of things that when combined provide some value. But then, you're just artificially slicing things up to make them fit within an iteration and forcing the PO to priortize things that don't make sense. It's much more logical to slice it up into tasks and let the delivery team prioritize the tasks in a way to provide the thing of real value to the PO sooner. Again, that doesn't prohibit frequent inspection during iteration review for incomplete stories. This should absolutely be done.
I'm a big believer in not letting "units of work" (tasks) span iterations, but breaking stories into artificial slices just to pacify the terminology constraint that stories can't span iterations is senseless. The decision of whether something is a user story should be based upon whether the story provides something of true value to the PO as a discreet unit. Having the PO prioritize things that aren't of value in and of themselves is frustrating to the PO and prohibits the Delivery Team from taking the most efficient path to provide that thing which is of true value. It just doesn't make sense.
jr>> I don't believe you can accurately estimate epics for a project and provide anyone a single point project estimate.
sd>> If you break it down into smaller stories when possible and always break it down into estimated tasks that don't span iterations and track your velocity based on work completed, yes you can. We do it and have a great track record of delivering on time. Of course there are times when unexpected things happen and you can always go back to elaboration, re-estimate stories/tasks scheduled for future iterations, etc. But you do not have to break stories into smaller stories that are not of true value to the PO to get there.
#7 Submitted by Roger L. Cauvin on Wed, 10/26/2011 - 9:53am.
Splitting Stories into Tasks
Sam, you alluded to contradictions in the notion of delivering something valuable to customers while at the same time splitting stories to fit within an iteration. Once you split a story into pieces that can fit in a single iteration, it is certainly conceivable that the pieces aren't themselves valuable to customers. I do think most organizations purporting to use agile methods are unable to resolve this tension.
Rather than bite the bullet and break stories into tasks, I recommend splitting large stories into versions. User story versions iteratively strengthen the acceptance criteria for the story. I described this approach in my comment below as well as in my blog entry, "An Epic Conversation". Agile practitioners often overlook this story-splitting approach. It doesn't address every case in which large stories need splitting, but I've found it does work well more often than not.
#8 Submitted by Sam Davis on Fri, 12/09/2011 - 10:52am.
Story versioning
Roger, thank you for your reply and thoughtful response. I'm not sure I understand the "versioning" approach. If you are talking about slicing "like to have" vs. "must have" functionality into separate stories for future versions, then I'm 100% with you and our PO is already very adept at doing this. She's the one that often initiates it and is very supportive of the Agile mindset in that regard. If you're talking about breaking the work up into versioned pieces that still aren't anything that the PO would really want to use until another version is completed, then I stick by my previous assertion that these should be tasks because it doesn't make sense for the PO to prioritize it, but rather the Delivery Team members are the experts on how to break it up and how to prioritize it to get to the end game (something of true value) faster.
#9 Submitted by Kimber H. Miller on Wed, 11/02/2011 - 9:25am.
How do project peers respond to the Story versioning?
and how do you decide on the way to split a story? I would think that it would be difficult at times. For instance, I would not imagine that splitting into a UI/back-end would yield results that are immediately useful to the Business at the end of an iteration.
And, to JR's point -- if you're splitting a story into verrsions, aren't you just devolving the stories?
Is this more of a tactic to manage your Biz customer's expectations/perceptions to help them transition ideologically to an agile project environment?
Thanks much!
#10 Submitted by Teardrop on Wed, 10/26/2011 - 7:19am.
Careful what you wish for
Many of the people that make their money being 'agile experts' know this.
Don't worry, you're not really getting agile. You're getting the same old stuff with some bits moved around.
To sell something new into corporates you do need the 'special sauce' *99% snake oil. But to be invited back, you can't rock the boat too much.
To pretend true agile development can occur in most coroporate cultures is wishful thinking. But there's money to made pretending it can.
#11 Submitted by Teardrop on Wed, 10/26/2011 - 6:59am.
Agile in name only?
This depresses me. It's symptomatic of the industry wildly missing the point of agile. There may be a process transition occuring, but it's a transition to something that's agile in name only.
"Indivuals and interactions over processes and tools" - the Agile manifesto.
The user story and the iteration have sadly become the process here. The author has fallen into the 'corporate agile' trap of just implementing a 'mini waterall' process that merely resembles true agile development.
"Working software over comprehensive documentation" - the Agile manifesto
"Responding to change over following a plan" - the Agile manifesto
Epic user stories are just another form of comprehensive documentation, however much you edit them down (becoming ever more comprehensive with each of the hundreds, thousands, edits down to smaller stories). Having this level of user story may appear to be agile on the surface as it allows an 'iterative approach', but in fact this approach anti-agile. This level of written detail is the very enemy of change and interaction.
"Customer collaboration over contract negotiation" - the Agile manifesto
"Please do not start your stories with "As a customer." Which customer?
The answer in the world of agile is clear - "the customer you are collborating and interacting with!", the point you shouldn't need to be writing this stuff down like this, this should be implicit in the human interactions that are at the heart of true agile development.
This article makes many useful points in terms of managing a type of corporate software development process. Agile it isn't.
#12 Submitted by Johanna Rothman on Wed, 10/26/2011 - 9:21pm.
Mass Market Customers
Teardrop, when your customers are the mass market, you need personas to identify them. Some people are not good as identifying the particular customer. It's not the person you are collaborating and interacting with. That person is a surrogate for the persona.
I don't understand the other concerns you had with my article.
#13 Submitted by Teardrop on Thu, 10/27/2011 - 12:43am.
This isn't agile
I'm not sure if you're being disingenuous or really don't understand the concerns. You keep talking about 'agile'. Well, what your selling may be a better process than waterfall, but it's not agile. That is why I contrasted what you were describing with simple concepts described in the agile manifesto.
What you are peddling may bring improvement for some teams, but I wish you wouldn't use the word agile when what you propose really isn't. Iterative development would be a more honest term to use, rather than trying to leverage the agile buzzword inappropriately.
#14 Submitted by Anonymous on Thu, 11/03/2011 - 7:55am.
Agile needs to be agile
It seems to me that no methodology will apply 100% to every company or even every situation. We should be learning agile in its "pure" form an applying and adapting it based on experience. In the end, people write code, people test apps, and people use apps. If the method doesn't work for people, it just won't work at all.
#15 Submitted by Roger L. Cauvin on Tue, 10/25/2011 - 10:53am.
Decomposing Epics along Nonfunctional Lines
The big problem I've observed with story decomposition is that few people consider splitting a story by incrementally strengthening the acceptance criteria.
Typically, when someone attempts to split a story, she immediately begins to break the story down along functional lines. For example, if the larger story involves CRUD (the user creating, retrieving, updating, and deleting information), we might split the story into a "create" story, and "retrieve" story, an "update" story, and a "delete" story.
The problem with this sort of functional decomposition is that it sometimes renders many of the smaller stories completely useless. In many cases in which we provide users with CRUD functionality, it is a means to an end. I.e., users usually don't care much or at all about creating, retrieving, updating, or deleting. They are trying to achieve a larger goal and might be happy if they could bypass CRUD entirely.
We should strive to make all of our stories, no matter how small, valuable to end users. Often, the larger goal associated with an epic represents that value, at least from a functional point of view. When we need is a way of preserving that functional value while still decomposing the epic. We can do so by splitting the epic along nonfunctional lines.
To split a story along nonfunctional lines, we maintain the functional goal of the story, but we create several versions of the same story, each with incrementally more stringent acceptance criteria. For an e-commerce site, the first version of an end-to-end Browse and Purchase story might:
1. Relax the requirements surrounding browsing (e.g. no searching, just a list of products and descriptions).
2. Relax the requirements surrounding payment methods (e.g. give the user a telephone number to call and manually process her credit card information).
Versions tackled in subsequent iterations would tackle incrementally more ambitious acceptance criteria.
In a recently blog entry, "An Epic Conversation", I described (in the form of an epic!) many of the challenges teams face - and how they can overcome them - relating to story management and decomposition. I'm interested in what you think, Johanna!
#16 Submitted by Kimber H. Miller on Wed, 11/02/2011 - 10:28am.
but those ARE smaller stories embedded in the eipic
Roger, the example you gave seems flawed, to me.
The breakdown you gave lists two user stories embedded in the epic, "Browse and Purchase."
The first story could be, 1) User searches catalog using key words, and the second would be 2) User submits cc information via website as payment.
A third story would be the back=end processing of the CC to change the user's account and credit the vendor's.
It appears that you are decomposing along the lines of use CASES, rather than user STORIES. Even within those two use cases you outline are smaller Stories that could be teased out, as far as I understand.
Thoughts?
#17 Submitted by Johanna Rothman on Wed, 10/26/2011 - 9:27pm.
Incrementally strengthening acceptance criteria is a great idea
Roger, I like the idea of incrementally strengthening acceptance criteria. (I got a little lost in the conversation. I would have liked a short "do this, do that" at the end to net it out for me, since I'm traveling and jet lagged.)
#18 Submitted by Matt Block on Tue, 10/25/2011 - 10:04am.
Utilizing Story Cards
Great post JR. We are currently doing 1 week iterations and always strive to get stories done in a single sprint. As you alluded to, when breaking down stories the team constantly struggled with trying to use the INVEST mnemonic. We would get to a point such that we could break down a story further, but at that point it would provide no value to the user. I like how you point out that value can also be value to developers, testers, etc. We have started making a slight distinction in our backlog and creating what we call Story Cards. We use these when we are admitting that this item by itself won't be considered valuable to a user, however it will demonstrate progress towards providing value (the parent story). So it is still a complete "chunk" of work, completely developed, tested, etc. So while we always strive to have all items in an iteration deliver value to the user, in the occasions that isn't possible, we must at least be able to demonstrate progress towards value. So we do have stories that can span iterations, but that is an exception, not the norm, and when it happens the story cards still don't (or shouldn't) span iterations.
#19 Submitted by Sam Davis on Fri, 12/09/2011 - 11:08am.
Stories or Tasks
Matt, you are describing exactly what my posts are trying to address. Your post is full of terms such as "would provide no value to the user", "value to developers, testers, etc.", "progress toward providing value", "chunk of work", "demonstrate progress toward value".
All of these phrases are describing the characteristics of tasks, not stories, that should be prioritized by the delivery team and not by the Product Owner. I don't really care so much about the semantics ("stories" vs. "tasks") except that by calling them stories they are part of the product backlog (managed by the Product Owner) rather than part of the iteration backlog which is managed by the Delivery Team.
As I've stated in other posts, these can (and SHOULD) still be demonstrated in the review meeting in order to gain the benefit of frequent inspection.
What I object to is being forced to call these things stories so that the Agile "rule" is followed that stories don't span iterations and then by calling them stories, it causes the aforementioned disfunctional behavior. You can still get frequent inspection if they are managed as tasks. You can still get velocity by calculating a different metric based on task velocity.
I still have not heard ONE good point as to why this is wrong except the typical Agile purist lines like "you just don't get it" or "because Agile says so."
This seems to me like the ultimate hypocracy. In order to truly call yourself Agile, you have to rigidly defend the Agile "rules". You can't question or try to improve upon the Agile concept.
#20 Submitted by Kimber H. Miller on Wed, 11/02/2011 - 10:34am.
THIS sounds like the breakdown into "versions"....
...that Roger and others are attempting to describe.
The larger concept of tasks/scenarios/use cases still seems to provide some value in the case of gargantu-epic, just to sort out what the business goals are.
Then the stories can be identified and further decomposed according to what is possible in an iteration.
Or am I orbiting the planet that's no longer a planet?
#21 Submitted by Johanna Rothman on Tue, 10/25/2011 - 2:02pm.
Seems reasonable
Matt, this seems reasonable to me. Great idea.
#22 Submitted by Ken Katz on Tue, 10/25/2011 - 6:27am.
Great stuff, but here are the challenges that I face
1. My testers are not set up to do unit testing. So they can't test user stories, they only can test the entire application after the coding is complete, the application is "buttoned up" and migrated to the QA environment. I know that is not a good way to do things, but it's outside my control as scrum master / agile project manager.
2. Executive management gives me a non-negotiable completion date (inevitably negotiated with a client with no input by anybody who will have to actually the do the work) and a non-negotiable set of requirements (ditto with regard to client commitments). So "We need three iterations to be able to estimate. We will track our velocity for three iterations and give you a reasonable three-point estimate at the end of that time, OK?" is not OK, it is a non-starter.
#23 Submitted by Johanna Rothman on Tue, 10/25/2011 - 1:59pm.
Your challenges are not unique, unfortunately
Ken, why would your testers do unit testing? The developers should do unit testing. However, the testers *should* (maybe with training, maybe with hooks from development) do some feature-in-isolation testing.
Non-negotiable completion dates and non-negotiable feature sets guarantee that you will have: more defect, or more cost, or more people, or need a different environment. If you have more flexibility in other dimensions, great. If you have less flexibility, you know what is going to happen. You can take all the testers off that project now, because it doesn't matter what they find. If schedule and features are determined in advance with no basis in reality, don't waste the people.
That's a strong statement. You can tell I'm passionate about this topic! I think you and I have had this conversation before, too. That's why it's so important to know at the beginning what your project degrees of freedom are.
#24 Submitted by Kimber H. Miller on Wed, 11/02/2011 - 10:40am.
Isn't a wide range of flexibility required before Agile can be..
.....truly implemented as a methodology?
I mean, if one is in a corporate IT group (and now producing software to market), the basic supposition before implementing Agile is that the business WILL and MUST BE flexible about completion dates and scope.
As Ken describes for his environment, this would represent a change to the seminal way his corp does business.
And Ken -- you are SO not alone!
What do you think?