How to handle big stories / epics in Scrum?

Have you considered the leanStartup/XP way of defining a feature by releasing small increments of it to customers as an experiment? When you have an epic that's poorly defined and you have little agreement among customer reps, maybe you can produce a bit of it and see if the customers actually use it and recommend changes or spot problems.

Addressing more of the "how to fit preparation phase into Scrum" aspect (per your comment :-)

I've found two approaches work, but selecting between them is less about the situation and more about the team and "what they need"

Sprint 0 approach - I've found taking one sprint to dive in and really try to "unwrap" the idea or concepts that a customer wants. In this, it is -critically- important to actually get the customer in house or at least someone who is prepared to act totally like the customer (risky). Each day in the sprint looks like:

• customer presenting the idea (1 hour - yes, the customer must speak for the hour... 5 minutes won't cut it... this forces the customer to get specific)
• Q&A on the idea/concept (1 hour - timebox and manage the time on this... it can run over very easily!)
• "Break" - (2 hours) the customer can go away and refine the idea based on the questions (or take a much needed break!) and the team can begin to outline possible solutions. I like using the "Risk based approach" from old "Spiral" life cycle model where key risk items may get storyboarded or otherwise "mocked up" and then...
• Presentation to the customer by the team(s) - present the prototypes and get feedback. (2 hours) Facilitation skills here are key - take lots of notes and be prepared to parking lot items. You always have "tomorrow" for difficult items!
• Close out and set expectations for the next day (get feedback from the teams and agreement from the customer on what the customer will present the next day)

In this way, over 4-9 days (depending on sprint duration) you get to review the concept and have prototypes the customer has reacted to. Warning - This is a -very- draining phase. Customers might not be prepared to be hammered with the questions and teams might not be prepared to do "one hour prototypes" So tread lightly and be supportive.

Prototype and Iterate - I'm pretty sure this is what "the books" say to do with Scrum... Start with something that can be built, and build it. Begin iterating on the product delivered. The amount of work required by the Product Manager/Customer varies, but it is helpful if -some- concept of the system exists. I like using care examples, so imagine the customer said "build me a car, but all I can describe for you is the bumper" Well, build the bumper and begin iterating. If the team and customer are mature enough, they can deal with the stress of not having a complete architecture and the need for constant refactoring and design as the project progresses. In some cases, this is the only option you might have. The customer might really not know or be able to answer questions until they see something. These projects take longer and cost more as the discovery progresses, but that is true any time this situation exists. Take care to coach the team on it being "100% okay to not have an optimized solution for the build" the goal is always to get something in front of the customer that works and elicits more feedback. That is worth more than "good design" initially. Even for experienced teams, it can feel like "bad engineering" (and it might be... but it is decent project management! :-)

hope this helps!

How (1)

Before I dig in, I'd suggest that in your question when you comment "It can be done by arranging several meetings and working on the subject", you're actually headed in almost the ideal direction. No approach to working through problems with larger, fuzzy things that need built will be successful without facilitating the right conversations at the right time. These do NOT have to be "meetings", and should be focused on the right people being involved, often 1-on-1 or small group in the team room as the work is being done.

How (2)

Otherwise, I will typically come at this from two directions.

First, recognize that stories typically take longer for one of three categories of reasons, and thus derive higher story point estimates as a results. Those three categories are complexity, effort, and doubt. Complexity is the "We've got to figure this out" bit of the work, effort is "we know exactly how to do this, but it still takes time to go through the steps" and doubt is "we don't know how we're going to do this.

Second, I've observed that giving people a framework within which to decompose stories can often help immensely when they're working. Engineering types LOVE to create new approaches and processes, and that can be highly distracting from the effort to figure out how to approach the epic. As a result, I tend to suggest something like Dean's Epic - Feature - Story - Task decomposition model to organizations that are tying to coordinate many teams, while only using Epic - Story - Task for single teams or smaller organizations that don't need the extra complexity. The framework may not be ideal, but most work I've seen fits into it rather easily and it does provide a useful model for teams to get focused on their work.

Now, using those two perspectives, I'm able to work with teams to isolate the uncertainty. During release planning or backlog grooming time, the next large epic can be split up to understand which features are included and important (some teams just call these "child epics" instead of features). Each conversation around a split will indicate some things that are obvious, but hard (high effort), some that need figured out but are definitely possible (high complexity), and a few things that we're not even sure if we're headed in the right direction or if it's possible (high doubt). The relative value of these things can inform if you need to attack the doubt items first (because you might cancel that part of the app if it can't be done) or ignore them altogether (because you realize they're not important), while the complex items can be handled by planning time to have the right conversations just in time. High effort items just need done, and can occasionally be outsourced easily.

Having this understanding around each of the stories and features going into the work allows the team to focus on the right aspect at the right time.

When

The other part of your question seems to be "According to Scrum, when do I do this?"

In most implementations of Scrum, the team will need to spend some of its time assisting the product owner in grooming the backlog to the right level of detail. Most if not all of this time should be focused on stories that will be implemented in the coming sprint or two, as the learning from the conversations will fade quickly.

Conversations needed for complex items should occur just in time, during the sprint if at all possible (which is one reason Scrum requires a highly available product owner). Conversations around high-doubt items will need to occur earlier so that everybody understands exactly what is doubtful and what the decision points will be as the team attempts to resolve it. Many teams use spikes for these high-doubt items.

We've done something inside iterations, rather than strict Scrum, which might help you.

The reason we split the stories is not just to provide value earlier, but to get feedback.

Take all the ideas and work out which ones absolutely need to be delivered in order to get some value from the project. Then work out which of those areas is most risky. Dan North suggests we focus on the areas of greatest ignorance to identify the risk.

Within the riskiest area, find the places where the team know the least, and focus on those. Decide which ones you'll address each sprint, and which questions you'd like to answer.

You will end up with some stories which might not be end-to-end, but on which you can get feedback. Do whatever is necessary to tie them into something which proves the feedback is valid.

For instance, if you see the performance of a message bus as a risk, hard-code some messages which roughly match the kind of data you'll put into it then run some performance tests (architects are the stakeholders to which to showcase in this instance). If you're creating a new UI and you don't know whether it's usable, hard-code some data behind that then get the users to play with it using just the hard-coded data.

This will let the team get feedback on the most risky aspects of the code, early on. It won't be deliverable - but then, you identified the minimum deliverable in paragraph 3, so you know you need all those ideas (and if you didn't, eliminate the ones you don't need!)

Early on, everything will be something about which the team is ignorant, and which they probably haven't done before. Velocity will be all over the place and estimates will be impossible. The team will be discovering a lot. However, as the project progresses they'll be filling in the gaps which they do know, and velocity will stabilise, allowing the team to have as good an idea as possible as to whether they'll hit the deadline.

One of the main reasons for using estimates and velocity is to get the trust of stakeholders. It turns out that up-fronting the biggest risks is also a good way to do this.

Sounds like the existing scrum-based answers aren't getting you what you're looking for.

Consider skipping scrum in this situation and going towards a rolling-wave based approach. It is non-colliding with your scrum process and can deal with the factors you described.

Please consider the advice at http://agileinaflash.blogspot.com/2009/02/breaking-down-larger-stories.html and at http://blog.coryfoy.com/2011/03/breaking-down-features-to-user-stories/

You basically want to have individually demonstrable stories that each provide some kind of value to the end user. A minimal marketable feature may still be an interation's worth of work. Consider each story coming out of the epic to be a path that a user can take. Concentrate on the desired use of the story, and cut feature corners not quality corners.

In regard to fuzziness, you should not let your involvement with the team end at specification. Your stories can be fuzzy (represented on 3x5 cards even) if you come along with the story and work with developers. You get a better result, everyone learns, developers become more vested in the result, and specification takes less time.

For that matter, have the customers and developers help you break the epics down. You might be surprised at the excellent results!

One thing that intrigues me is that you say "they do well every time, when task is clearly defined". It seems there is a dependency on detail. This leads me to wonder if the team is clear on what the overall direction is. Epics are not standalone little projects they should be aligned to the overall product strategy. Is everyone clear on the vision? How about the product roadmap? The vision should help guide and steer decisions. A product roadmap will set expectations and avoid the last-loudest priority problem.

I may be completely missing the point and you might want to check on Bill Wake's excellent 20 Ways to Split Stories. He describes a couple of techniques for breaking down large stories that are not yet well understood. A great technique for working out whether we need to implement spikes or research stories is if the team cannot estimate stories. If you play planning poker and notice a large standard deviation in estimates you may need to do some more work before making commitments.

You're describing a situation in which a project needs to be planned and executed in an environment with high level of uncertainty.

There could be several areas of uncertainty:

1. Requirements uncertainty - what needs to be done, how done looks like, etc. This is mainly due to a lack of real customers with real spelled out needs.
   This kind of uncertainty could be handled by brainstorming and coming up with several approaches, alternatives, ideas that could be broken down to user stories the SCRUM team can develop in a sprint. These user stories should be defined as features/capabilities that when done can be presented to customers, or focus groups (UX testing) or any opinionated stakeholder. Plan for gathering feedback and adjusting the user stories in your backlog, that define the Epic according to the feedback that you get. Change and adjust the Epic from sprint to sprint until you reach a point in which all (or most) of the stakeholders are satisfied.

2. Technical uncertainty - how should the solution be formed, performance concerns, infrastructure concerns, etc.
   This kind of uncertainty can be handled by starting with a simple user story for building a slim end-to-end solution. In the first sprint you may encounter issues and problems. Then on the next sprint you will be able to define user stories that would handle issues around performance issues, architectural issues and more, by changing, refactoring and even throwing away code developed in the first sprint. Don't expect (and don't try) to be able to figure out all the technical aspect on day 1.

The same way you tackle any epic: break it down into smaller chunks.

There are many different stakeholders, so first try writing down the user story from each stakeholder viewpoint. If there are N stakeholders, that will generate N user stories.

Now, try to identify ONE user story of the N that can be fully fleshed out.

Tackle that one, and see what it tells you after the first iteration.

Tackling things from a stakeholder viewpoint is just one approach. Check out Google for other ways to deal with it.