Gil Zilberfeld, an agile practices expert, has posted on 4 warning signs that Agile is declining. I will not re-iterate his points; you should read the article for yourself. The overall theme though is this. The software establishment - including the managers, the consultants, the trainers, the vendors - has embraced Agile, to the extent that, according to Zilberfeld, more than half of software projects at least nominally use Agile methods. The results are disappointing though, because companies have simply absorbed bits of Agile into their existing top-down management culture. Therefore:

At this point, I feel Agile is declining into what TQM [Total Quality Management] was. A brilliant success in the beginning, and now just a history fact. In a few years, months even, the business side will wake up and say: Agile is snake oil. It doesn't deliver on its promise (and it doesn't matter if it's done wrong). The backlash will be grand.

I am reminded of something I learned from one of the excellent QCon conferences in London, which covers Agile in depth. It was not so much a specific speaker or talk, more a common theme running through many presentations. Software projects generally do not fail for technical reasons. They fail because the team - using the word team in its widest sense, to include all project stakeholders from users to executives - fails to communicate effectively. Many Agile techniques, such as the daily meeting which is part of the Scrum methodology, are designed to facilitate communication. I have also heard recommendations such as moving developers into the same office as managers in order to get them talking to each other. Another example, at a micro level, is Pair Programming, where two developers work side by side on the same code. You cannot do this without communicating your intentions, ideas and solutions to the person alongside you.

Kent Beck, one of the pioneers of Extreme Programming and Test Driven Development, highlights the human factor in software development. Take a look, for example, at his essay on Accountability in Software Development:

... while programming I offer accountability as a way of demonstrating my trustworthiness and encouraging my own best behavior. Pair programming; test-first programming; continuous integration; visible daily, weekly, and quarterly cycles; slack; and estimation are some of the way I make public commitments and render account of my activities. Knowing I will be honest and accountable affects how I do my work, just as knowing that I am hiding and concealing negatively affects how I do my work. Taking responsibility for my choices and actions deflects blame. There is no hidden shame if everything I do is above board.

What is important here is not the techniques in themeselves, but the trustworthiness and accountability they facilitate.

Human problems are harder to solve than technical problems, and seen in this light it is not surprising that companies which adopt bits of Agile methodology without changes in corporate and management culture will miss out on most of the benefits.

Another common experience at software development conferences is to talk to developers who enthuse over the insights they have received, but then lament that they cannot be applied in their workplace. The reasons are old and familiar: inflexible management, longstanding broken processes that nobody seems able to fix, little kingdoms which protect their borders at the expense of the effectiveness of the whole corporation, and so on.

A few thoughts in conclusion. First, if Agile projects are failing, that does not necessarily imply that something is wrong with Agile methodology. It all depends on how it is done and whether the people involved are embracing or resisting the change and communication that goes along with it. Second, irrespective of the methodology, effective communication is key to the success of a project; and if it is not possible to change the methodology or even the tools and technology, working on team communication may still yield amazing results.

Project Charters are a way to initiate projects. Sometimes charters are text documents. Sometimes charters are boxes or brochures that "sell" the product's benefits and features to others.

I like text documents as a first draft because they are easy, and are a fast way to bring a team together that has not worked together before. Once a team has worked together, try a box or a brochure. But if you're new, consider a text document.

Especially if you're new to project charters, take a minimalist approach. I like to think, "What's the minimum I can put into this project charter and still make it be enough for this project?" I offer a project charter from Manage It! Your Guide to Modern, Pragmatic Project Management. That charter has these parts:

1. Vision: Concise and compelling idea behind the project
2. Drivers, Constraints, and Floats: What's driving the project, what's constraining the project, and where you have freedom to organize the project as you need.
3. Goals: What you might want to accomplish with the project, but not required that you do so
4. Success Criteria: Capabilities your product will have at the end of the project
5. ROI Estimate: Potential return from your project (only if your organization requires it)

Write your vision with the project team. That way, you can make the project vision compelling to the entire team. If everyone starts yawning as they start writing, you know nothing about the vision is compelling! Ask them, "What do we have to do to make this vision compelling?" Maybe they need to start with objectives, not the vision. That's fine.

When you think about drivers, constraints, and floats, do not think of the iron triangle. There are more than three sides to the iron triangle. You need to consider the organizational constraints: cost, people and their capabilities, technical debt, and the environment of the project. And, you need to consider the specifics of this project: the desired release date, the desired feature set, and the desired defect levels. Only by balancing what's honestly driving the project, what's really constraining the project, and what's really in your degrees of freedom can you make decisions about the project. You might need a project driver matrix to make those decisions.

The team may want some goals for this project, that are separate from the organization's requirements. I often see this when teams want to start paying down some technical debt. "We want to automate 10% of our regression tests."

Success criteria are the capabilities the product will have once the system is done. Knowing what success looks like is a tricky definition. You'll probably use context free questions to define success.

ROI, return on investment is something few of us have to calculate, but if you do, the project charter is the place to put it. Of course, you'll be wrong, but that's ok. (You'll be wrong because it's a prediction based on what the project will cost and how many people will use your results. You cannot know the future! If you do, please go gamble in Vegas.)

The value of the charter is not so much in the text, but in the discussions to create it. Your discussions as a project manager with the team, the team's discussions among itself, the team's discussions with senior management, anyone's discussions about the project driver matrix. The discussions create an atmosphere of teamwork, and create the agreements so people can start.

You may decide that you need a cross between a project charter and a project plan. That's fine. Maybe all you need is a project vision and release criteria. For me, that's the absolute minimum that any project needs. You know where you are going and you know what done means.

The project charter gives you one small way to build a team. Try it.

I recently spoke with a Scrum Master of a team of 10 people. He is struggling, as is the team. They are having trouble connecting as a team. They are ok when they work as smaller sub-teams, but not as a team of 10. Well, that's not entirely surprising. There's research backed up by data to explain why.

I've observed over the years that teams of larger than 9 people didn't have what I call the "intimacy" of teams 9 people or smaller. Teams of 3 people or smaller didn't always have the ideas needed, but teams larger than 9 people lost something that was common to smaller teams--an ability to easily communicate with each other. There is a reason for that: the number of  communication paths in the team.

The number of communication paths in the team do not increase linearly as the team size increases;  team communication paths square when the team increases linearly. Here is the calculation where N is the number of people on the team: Communication Paths=(N*N-N)/2. That means you have these communication paths for these team sizes:

4 people, (16-4)/2=6
5 people, (25-5)/2=10
6 people, (36-6)/2=15
7 people, (49-7)/2=21
8 people, (56-8)/2=24
9 people, (81-9)/2=36
10 people (100-10)/2=45

That means the number of paths for up to 8 people is still manageable (24), but once you get to 9 people, it becomes 36 paths, which is high. For 10 people, the number of paths is 45, which is unmanageable for most of us. Can some people manage this number of communication paths? Sure. But not many of us, which is the problem. That's why "teams" of 10 people don't work.

If you have a group of 10 people, they will naturally divide into smaller sub-teams of people who can work together. They will not divide themselves evenly, as 5 and 5. Oh no, that would make life too easy. If you are very lucky, they will divide by features. If you are not lucky, they will divide by architecture or by function, or even worse, by clique.

So what does that mean you should do if you have a large team? Here are some ideas:

1. Organize by feature. If you have a feature team that's relatively small, as in fewer than 10 people, great. Keep the team that size and stop worrying. You're done.
   
2. If you have already organized by feature and the team is still larger than 9, first see if you have enough people on the team. Do you have enough testers, BAs, product owners, writers, whatever you need? If not, determine what you need. I often see that teams on the verge of becoming programs (collections of projects) do not realize they have to scale, and are trying to "make do" with who they have, which doesn't work at all.
3. Once you have enough people, and you've organized by feature, now, ask the people to organize the way that makes sense so that they can get the work done. Do not tell them, "Every team deserves 1.35 testers," because that's how you can divide the testers up. That doesn't make sense. Ask the teams to discuss it and let you know what they need and what does and does not make sense. They will let you know.

If you have a small team, a team of 3 or fewer people, watch out for insufficient ideas to solve the problems. That doesn't always happen, but it sometimes does. Too few people can be just as bad--although differently bad--than too many people. Team size matters.

I live in the Boston area, and we have had more snow in the last few weeks than we normally have. And, we have not had our normal thaws, so the snow hasn't gone anywhere. Our snowbanks at corners are high--so high that when you drive, you have to edge out into the already-narrowed streets to look around the snowbanks to see if it's safe to go. Of course, by that time, you're already halfway out into the street. You'd better commit to go!

I was thinking about that problem on my way home from the gym this morning. The streets are already narrowed from the snow, the snowbanks are high. There are times when I do not want to commit to going--such as when a bus is already too close for me to make a turn and get to a reasonable speed.

We see problems like this on our projects all the time. We need to stick our noses out to see if it's safe to try a direction before we fully commit to it. We may want to try one direction and then turn around if it's not working (not an easy thing to do on snow-narrowed streets).

We have options for managing our project risks that we do not have for our driving risks:

1. Try a Hudson Bay Start. For a Hudson Bay Start, you try to push through one very small piece of functionality, and see what it took to do that. The reasons you start with a small piece of functionality is that you want to try driving across the street, not the country. You want the equivalent of "Hello World" and no more, just to see where the risks are, and what you will need to do about them.
2. You can try a short iteration, say a week, maybe two, and see how far you can get. Now you have data that you can apply to the rest of the project or program.
3. For a design/architecture problem, apply a design spike. Take 2 or 3 people for no more than 6 hours, and discuss and code, and see where you are at the end of that time.
   

In software projects, we don't have to commit the way we do when driving--which is a good thing! We can do a little more exploring and see what data we can gather. Now, based on data, we can make a reasoned decision. We don't have to make a split-second decision on snowy streets the way I do right now when driving.

Wish me luck--we're expecting another foot of snow by the time you read this. The streets will be even narrower and the snowbanks even higher. One of my options will be to take a deep breath and go for it. We don't have to do that on our projects. Which is a Very Good Thing.

I'm a fan of agile approaches when they fit your project. But that doesn't mean I never use other approaches to projects.

I often use iterative approaches, when I want to try something and experiment a while. I can start with a small part of the system and iterate on that part until I understand it. If I'm the project manager, I can ask a team of people to do that show me the results.

I like incremental approaches too, where we build a piece of the system and then another piece and then another until we are done. I like seeing the system come together whether I'm using timeboxes or not.

Demos serve two purposes in any project: to see visible progress and to acquire feedback. If you're using a serial lifecycle, build in demos at least every three months. That way, you can't go too long without any feedback about the product, or feedback about the general architecture of the product.

If you use an iterative approach, consider a demo every two-to-four weeks. If you are iterating in longer cycles, your iterations are too long--you are trying to do too many experiments for the time. Break your experiments into smaller chunks, and then demo. (BTW, a strictly iterative lifecycle does not have timeboxes necessarily; you answer questions each iteration and then decide what to do. You want to make sure you don't have too many questions to answer for each iteration.)

With an incremental approach, you're working feature-by-feature, or feature set-by-feature set. I almost had a disastrous program when I used an incremental approach, and let the increments get too big. We waited almost two months for a demo. The project team was so pleased--and the product manager was not. We made our increments smaller and gave more frequent demos.

So, no matter what kind of project you have, consider a demo as part of your project management. You'll see the team's progress, the team will see its progress, and the product manager, sponsor, customer, whomever you've got will see what the team has done. Wins all the way around!

I meet a lot of managers, product owners, product managers who don't think they need to address technical debt. "Technical debt is a technical problem. We don't need to prioritize it, manage it, or anything else. You technical people, you do it."

I define technical debt as the debt you owe your product: decisions you make now that incur cost later. You can have architectural debt in the form of architecture that's too brittle, development debt when you don't separate the GUI from the API, testing debt when you don't have automated tests where you need them, and even management debt in the form of not making project portfolio decisions, leading to multitasking.

Well, management can attempt to ignore technical debt. But most often technical debt is the result of management decisions.  If management doesn't acknowledge the debt, you can't manage it in the future, and you can't assess your decisions about it.

It's not a weakness to accept that we accrue technical debt. Debt isn't evil, it buys you something, most often time. For example, if you use any project approach that saves defects as something to address later, you accrue technical debt.

One of the reasons Agile is scary to managers is because it makes technical debt transparent. The more transparency you have around technical debt, the more transparency you have in decision-making. This is where the current organization's culture may bump against a transition to agile. If you have a culture of burying technical debt because you need short term revenue, you can still do that, but the decision will be transparent. If management is not being transparent about decisions, agile will make that transparency happen.

Acquiring technical debt is a management decision. Make the decision transparent.

There's a tension in agile project/program management about product architecture: when do you make the architectural decisions?

If you make lots of architectural decisions early, you can create a brittle system that needs to be rearchitected. If you make the decision too late, you might be creating lots of technical debt. Is there a "Goldilocks" moment that is just right?

Maybe.

Part of the problem is you need to know what a minimally acceptable system is. Note that I said acceptable, not rejectable. Sometimes, the tolerance between minimally acceptable and not acceptable is quite small, as is the tolerance between minimally acceptable and fabulous.

Minimally acceptable is still acceptable. If you have minimally acceptable product, you might have a lot of technical debt, but still an acceptable product for this release. The real question is what happens for the next release? That's when the project/program team, and management needs to acknowledge the technical debt and put the debt into the backlog to manage and deal with.

If you have a product owner who only wants to deal with features, you have an unacceptable product owner. Product owners need to manage the technical debt as well as the desired feature set.

To discover your "Goldilocks" moment, you need data about the architecture. To get the data, you need to implement something: a walking skeleton, several features, alternative prototypes, something that provides you information about the potential system architecture.

So the way I know how to make these decisions is:

1. Know what minimally acceptable product is. This might be reflected in project release criteria or in success criteria.
2. Acquire some data about the proposed architecture.
3. Postpone the final decision until the last responsible moment. Sometimes that moment is earlier than you might like. Sometimes, if you're like me, it means you live with ambiguity longer than you might find comfortable.

Whatever you do, avoid making final architecture decisions at the very beginning of the product based only on pretty pictures. That's a fast way to create architectural technical debt.

Instead, consider decision-making based on data, and to see what problems this architecture resolves and which problems this architecture leaves open. Then, you'll have the least technical debt and it will be transparent to you, so you can continue making decisions about it.

If you are using agile or lean to manage your work, you are limiting work in progress. Agile limits work in progress by timeboxing--if the work doesn't fit into the timebox, you don't start it. Lean limits work in progress by limiting the number of items in any given state (started, in development, in testing, whatever).

When you limit work in progress, you give yourself a chance to finish one thing at a time. We, as human beings, are better at finishing something and going on to the next thing than we are at having many pieces underway. We miss problems with lots of work in progress. We are more likely to see problems when we finish (as well as we can) a given piece of work for now. If we have to return to it to make things better later, that's ok.

You can limit the work in progress regardless of the kind of project you have. If you are starting a serial lifecycle, explain that you want to finish the highest priority work first and then move to the next batch of work. You may have to ask the requirements people to timebox their work on requirements so you can move on to analysis and design. But even if you can't change the front of the project, there is nothing to stop you from implementing, integrating and testing as you go, once you are in development. Even in a serial lifecycle, there is no law against continuous integration!

Now, you and I both know that it's no longer a serial lifecycle if you start integrating and testing as you develop, but that's ok. Your customers, managers, and other stakeholders will be happy you did.

My experience, and I'm sure it varies from yours, is that once your stakeholders see the documents for requirements and architecture don't care much about your functional specs or other documents, such as design. Those documents are for the technical people. So if you must, start a project with a serial lifecycle, and then move into prototyping so you can understand the architecture, or, at the least, move into delivering chunks of work.

The lifecycle of "get some requirements, decide on an architecture, implement by feature, integrating and testing as you proceed" is an incremental lifecycle called staged delivery. It's a great way to limit work in progress and see your project state as you proceed. It's not strictly serial, but it works. And, isn't that what they pay us to do?

I've been working with some clients transitioning to agile. The senior managers want to know when the project will be done.

I can understand why they want to know. But before you start a project, you can't know with any certainty! What you can do is use rolling wave planning to know more as you proceed.

The way to use rolling wave planning for agile is this:

1. Lay out the major features you want in a product roadmap. You can organize the product roadmap by week, month, or quarter. I recommend no longer duration than one quarter.
2. Now, create the stories that go with the features in the order in which you want them done.
3. Work with the team to further refine the stories as they plan the first iteration.
   
4. The team completes the first iteration.
5. Now, assuming the team has worked together before, and that the product owner has refined the user stories as much as possible in the previous iteration, the team takes the time to do a gross estimation of all the remaining features in the roadmap. If the team has not worked together before or is new to agile, they need a few more iterations to practice estimating before they can provide a gross estimate that's at all useful.
6. Lay out, on a timeline, approximately when the team thinks they can finish the features, based on the gross estimate.
7. Now, lay out the major milestones (when you will have finished features) on a timeline. Every iteration, you note what you actually completed. When it makes sense, say every 4 weeks, re-estimate the entire backlog, if you need to provide a more accurate estimate. Since you'll be within +/- a few iterations at the beginning, I don't normally suggest you re-estimate all that often in the beginning. But you could.

Now, you have a rolling wave of what you think you can accomplish when. And, you are closing in on a date for your managers.