Gil Zilberfeld, an agile practices expert, has posted on 4 warning signs that Agile is declining. I will not re-iterate his points; you should read the article for yourself. The overall theme though is this. The software establishment - including the managers, the consultants, the trainers, the vendors - has embraced Agile, to the extent that, according to Zilberfeld, more than half of software projects at least nominally use Agile methods. The results are disappointing though, because companies have simply absorbed bits of Agile into their existing top-down management culture. Therefore:

At this point, I feel Agile is declining into what TQM [Total Quality Management] was. A brilliant success in the beginning, and now just a history fact. In a few years, months even, the business side will wake up and say: Agile is snake oil. It doesn't deliver on its promise (and it doesn't matter if it's done wrong). The backlash will be grand.

I am reminded of something I learned from one of the excellent QCon conferences in London, which covers Agile in depth. It was not so much a specific speaker or talk, more a common theme running through many presentations. Software projects generally do not fail for technical reasons. They fail because the team - using the word team in its widest sense, to include all project stakeholders from users to executives - fails to communicate effectively. Many Agile techniques, such as the daily meeting which is part of the Scrum methodology, are designed to facilitate communication. I have also heard recommendations such as moving developers into the same office as managers in order to get them talking to each other. Another example, at a micro level, is Pair Programming, where two developers work side by side on the same code. You cannot do this without communicating your intentions, ideas and solutions to the person alongside you.

Kent Beck, one of the pioneers of Extreme Programming and Test Driven Development, highlights the human factor in software development. Take a look, for example, at his essay on Accountability in Software Development:

... while programming I offer accountability as a way of demonstrating my trustworthiness and encouraging my own best behavior. Pair programming; test-first programming; continuous integration; visible daily, weekly, and quarterly cycles; slack; and estimation are some of the way I make public commitments and render account of my activities. Knowing I will be honest and accountable affects how I do my work, just as knowing that I am hiding and concealing negatively affects how I do my work. Taking responsibility for my choices and actions deflects blame. There is no hidden shame if everything I do is above board.

What is important here is not the techniques in themeselves, but the trustworthiness and accountability they facilitate.

Human problems are harder to solve than technical problems, and seen in this light it is not surprising that companies which adopt bits of Agile methodology without changes in corporate and management culture will miss out on most of the benefits.

Another common experience at software development conferences is to talk to developers who enthuse over the insights they have received, but then lament that they cannot be applied in their workplace. The reasons are old and familiar: inflexible management, longstanding broken processes that nobody seems able to fix, little kingdoms which protect their borders at the expense of the effectiveness of the whole corporation, and so on.

A few thoughts in conclusion. First, if Agile projects are failing, that does not necessarily imply that something is wrong with Agile methodology. It all depends on how it is done and whether the people involved are embracing or resisting the change and communication that goes along with it. Second, irrespective of the methodology, effective communication is key to the success of a project; and if it is not possible to change the methodology or even the tools and technology, working on team communication may still yield amazing results.

Some product owners don't want to address technical debt. They only want to add features to a product or system. And that's a huge problem.

The product owner has the responsibility for value of the entire system. That means that the product owner has to address features, technical debt, defects, whatever needs to be ranked so that the team knows what to do when. If the product owner does not rank the technical debt and the existing defects in addition to the features, the team will have trouble sooner or later delivering features.

So what's a team to do?

1. It's time for a heart-to-heart talk with the product owner. The product owner first needs to know that it is his/her job to rank everything for the iteration. It's possible he or she does not know.
2. If you are not tracking velocity, start. Remember, velocity is a trend, and will help your product owner see what is going on. Because your velocity will go down as your technical debt goes up, and your product owner needs to know that now.
3. If you are not tracking the number of defects found and fixed in an iteration, start now. Make this a trend, not a number. Trends are what's important. 
4. Consider tracking Fault Feedback Ratio, the number of bad fixes to total fixes as a trend on an iteration basis. This tells you if the developers and testers are making progress or spinning their wheels.
5. Make fixing technical debt a part of each feature. I don't understand how anyone can prevent you from doing this. I do understand how insufficient estimation can prevent initial test automation. But a team has the ability to say, "Wait a minute, we are not really done." 

Remember, the team has a the "done" card. If a team wants to play the done card and define what done means, and swarm around a story, making sure that they have prevented technical debt from accumulating, they can. I'm not talking about gold-plating, adding more features. I am talking about code review, architectural review, test development at all levels, whatever you need to know that the code works.

Not fixing technical debt is behavior you want to prevent, or stop early. But, if you can't stop it early, make sure the product owner sees the reality of his or her decisions. This is why short iterations are such a good idea. Your product owner might even have a good reason for his or her feature-itis. But if they acknowledge the reason for the feature-itis and explain when it will end, the team has a shot of retaining its sanity.

The more features a team adds without address technical debt or defects, the more technical debt a team adds. And, that's not acceptable, regardless of your lifecycle.

At Agile 2011 last week, we were treated to three wonderful keynotes, aside from over 260 sessions.

The first, on Tuesday morning was about positive emotions by Barbara Frederickson. I was afraid it was going to be one of those "be happy" nonsense things, but it wasn't. She discussed how to stay happy, and it was about challenging yourself, and learning how to have emotional resilience, something I'm quite interested in, and will be discussing in more detail on my createadaptablelife blog sometime in the next few weeks. I bought her book and will be reading it because I really liked her presentation. Her presentation is mostly pictures.

The second was entitled "Code" and delivered by Kevlin Henney. I hadn't heard of him, because I'm a management geek. Well, if you have not heard of him, go seek him out, find his presentation. He showed pictures of code. Yes, pictures! Aside from cute things you can do with code, one of the best analysis tools I saw was using wordle as a static analysis tool, to see what kinds of domain knowledge you have built into your code. Is that cool or what? Way cool.

But the best and last, was Linda Rising's keynote called, "The Power of an Agile Mindset." I should tell you, Linda and I are friends, as well as colleagues. I've known Linda for years, and volunteered as a shepherd on the Insights stage that she co-chaired for the conference.

 Linda discussed the fixed mindset where we go for perfection and get stuck if we don't achieve it versus the agile mindset where we adapt if discover a challenge and we try harder.

And, then came the zinger. As a society, we encourage smart little girls towards perfection in elementary school. We encourage little boys towards challenge. So, by the time we all get towards middle school/junior high, little girls do not want to fail. They don't want the challenge. They want perfection. All their lives they have been praised for perfection. How can they stop now? And boys? Why should they want perfection? All their lives they have been praised for rising to the challenge. Why should get all As? Why should they want the grades when they can find a bigger challenge?

This explains so much about why girls gravitate towards classes they can do well in, and why boys take classes they find a challenge--as a generalization. There will always be girls like me who rebel (rebelled) against society and boys who had such family pressure for perfection that they avoided the challenge.

If you are a parent now, you can help your children chart their way by encouraging them to challenge themselves by praising their effort, not their results. "Wow, look at those colors!" or "Wow, you must have worked really hard on that." or "I saw how hard you concentrated on that. Good for you!" Kids need to have their effort acknowledged, not rewarded. There's a big difference.

If you are a manager, same thing. Acknowledge the work, even if the result is not always what you expected. Some of you might be asking, "But why can't we take partial credit for partially complete stories at the end of the iteration, if we want to acknowledge effort?" Because we while we want to acknowledge effort, we do not want to reward it. That's the challenge.

I was one of the people who gave Linda a standing ovation. I will be urging her to share this presentation.

I've been meeting teams who complain to me the iteration is too long. "But, JR," they say, "we can't get anything done in four weeks. We have to move to six week iterations. We can't possibly commit to anything in four weeks. We need at least six weeks to finish anything."

They sit there, with their sincere faces. They look at me, expecting me to go along with them. I take a deep breath, and say, in my most sympathetic and sincere voice, "Oh boy. This is really hard. Ok, you've convinced me. The iteration is now two weeks long."

At first, they turn to each other congratulating each other that they've pulled something over on me. Then they turn back to me, and look at me like I've grown at least one more head. Or maybe that my head is turned around backwards. Inevitably, the tallest man stands up and tried to intimidate me physically. He doesn't realize that my stature is the smallest part of me, that I've been shorter than everyone else for so long that his attempt at intimidation is pitiful to me.

"Johanna, what do you mean, two weeks. Did you not hear us?"

"I heard you just fine. Do you really want to wait six weeks to discover why you can't release anything in six weeks either?'

That usually stops them in their tracks. "What do you mean?"

"If you don't think you can finish anything in four weeks, the chances are quite good you can't finish anything in six weeks either. If we make your iterations two weeks, you have to reduce the size of the work you take into your iterations. You will be more able to estimate the work. You will have less to test. You will have less opportunity to create technical debt. You have a shot at releasing what you created.

"But if you expand the timebox to six weeks, you won't get the feedback you need. You will never understand why you can't do anything in four weeks. You will never learn why you can't finish anything in four weeks. You will have a self-fullfilling prophecy: your four-week iteration will be too long."

At this point, they generally want to go back to a four-week iteration, because it looks better than a two-week iteration. But no, a four-week iteration is an excuse to do a waterfall iteration: one week of design, one to two weeks of development, one week of testing (which will get crunched into less than a week because the development will expand) and then quick, end the iteration. Some smart person will want a Gantt chart of the iteration, and another smart person will want a test plan. Their self-inflicted overhead will kill their agile efforts.

Now, maybe your agile pilot did succeed with four-week iterations. Good for you! But I'm meeting more and more teams whose pilots are not succeeding. They don't change what they are doing--they fit their silo'd teams into a four-week timebox and call it agile. They certainly get some value from the timebox. But they are not living the agile values and they are not transitioning to agile.

When you make the iterations shorter than you can imagine, you force yourself to change your practices. You force the team to consider continuous integration. You force the team to integrate the testers with the developers. You force the team to limit the number of stories in progress at one time. You force the team to reduce the number of stories under consideration for a timebox, so the team has a better idea of what they can commit to for an iteration.

Maybe each of these changes is not so big as a single change, but taken all together, they are a huge change for a given team. For the team who thought they could pull one over on me, each change was a big change, and all the changes together was a monumental shift in the way they worked. And, when I suggest we start the iteration on a Wednesday, I thought they were going to revolt, especially since it was Tuesday afternoon.

"Can't we start on Monday?" one of the developers plaintively asked.

"What do we gain by waiting?" I asked.

"We get accustomed to the ideas," he replied.

"You get to research and discover more ways you can push back and tell me why things won't work here," I answered."

Everyone laughed, nervously. I said, "Look, I'm asking you for two weeks. Starting tomorrow morning. The worst thing that happens is that you get through tomorrow morning and you hate me. Right? That's about me, not you. What do you have to lose? All you do is take it one step at a time. And I'm with you each step of the way. Ok?"

They agreed. I was concerned they would all be sick the next morning, but they all came into work on time. We did plan the next morning. And they did get through the next afternoon. They almost finished a small feature, which blew them away. By Friday afternoon, they had finished two stories off the backlog. And, they discovered what prevented them from making their integration continuous. And, they learned what the testers needed for an automated test system, and created a just-enough test automation system.

By the end of the iteration, they had finished eight of the ten stories on the backlog, discovered they had mis-estimated two of the stories, and had discovered five new stories. They had never worked like that before. When they provided a demo to the product owner, the PO was thrilled.

Not all teams have such great results. As with all changes, your mileage will vary. But when your team says, "the iteration is too long," consider shortening the duration, not lengthening it. The more opportunities the team has for feedback, the faster the team can learn.

I'm in Australia right now. I've been keynoting, paneling, and talking at Software Education's SDC conference. One of the participants asked the panel of speakers, "My management thinks I should be able to do the BA work and the Scrum Master work on my project. What do you think?" The other speakers were reasonable, explaining that because BAs are collaborative, they are a natural for the Scrum Master role.

I just about exploded out of my chair. "What is your management thinking? Do they want to do Scrum or do they want the project to fail? Or, maybe they want you to fail?" I ranted and raved a little more, but I'll save you from that. Here are the ideas behind my ranting and raving.

If you are new to agile and you want to do Scrum, then follow Scrum. There is a reason for the role of Scrum Master. That reason is that when you are new to agile, you need someone who advocate for the team to remove impediments at all levels of the organization, someone who can facilitate the process for the team. This person serves the team. This person, if serving the team, will spend all of their time serving their team. They will nudge the product owner into grooming the backlog. They will help people understand the acceptance criteria. They will facilitate the standup meetings and make sure those meetings occur, that the team collects the data and does not last longer than 15 minutes. That person ensures that the other problem-solving meetings occur. That person will solve problems on behalf of the team. In my experience, a new Scrum Master, for a new team spends a ton of time away from the team doing things like fighting the furniture police, fighting the network police, fighting the management police, so that the team can work. If this person also has deliverables to the team guess what? No one resolves the impediments. The team's velocity never settles into a rhythm. The team and the organization never sees the benefits of agile.

Someone says "Agile (or Scrum) doesn't work." Nope, that's not the problem. You're not doing agile or Scrum if you don't follow the directions. Don't adapt it if you don't first adopt it, lock, stock, and barrel.

When you try to play more than one role on an agile project, you are not adopting agile. You are adapting agile without understanding how to adopt it.

Multitasking on an agile project is just as bad as multitasking on any other project. You have to make it transparent. If you are asked to multitask, make it an obstacle. Show your lack of progress on the board. Consider using a kanban board inside a timebox if a number of people on the project are supposed to be multitasking. Ask your managers to start managing the project portfolio.

Because remember, it's not about the projects you start; it's about the projects you complete. It's not about the roles you take on; it's about the projects you complete. If you can successfully complete projects with multiple roles, great. That's a rare capability for rare projects. 

Keep your eye on the goal: finishing projects. Then you'll know what to do about this business of multiple roles.

There's a tension in agile project/program management about product architecture: when do you make the architectural decisions?

If you make lots of architectural decisions early, you can create a brittle system that needs to be rearchitected. If you make the decision too late, you might be creating lots of technical debt. Is there a "Goldilocks" moment that is just right?

Maybe.

Part of the problem is you need to know what a minimally acceptable system is. Note that I said acceptable, not rejectable. Sometimes, the tolerance between minimally acceptable and not acceptable is quite small, as is the tolerance between minimally acceptable and fabulous.

Minimally acceptable is still acceptable. If you have minimally acceptable product, you might have a lot of technical debt, but still an acceptable product for this release. The real question is what happens for the next release? That's when the project/program team, and management needs to acknowledge the technical debt and put the debt into the backlog to manage and deal with.

If you have a product owner who only wants to deal with features, you have an unacceptable product owner. Product owners need to manage the technical debt as well as the desired feature set.

To discover your "Goldilocks" moment, you need data about the architecture. To get the data, you need to implement something: a walking skeleton, several features, alternative prototypes, something that provides you information about the potential system architecture.

So the way I know how to make these decisions is:

1. Know what minimally acceptable product is. This might be reflected in project release criteria or in success criteria.
2. Acquire some data about the proposed architecture.
3. Postpone the final decision until the last responsible moment. Sometimes that moment is earlier than you might like. Sometimes, if you're like me, it means you live with ambiguity longer than you might find comfortable.

Whatever you do, avoid making final architecture decisions at the very beginning of the product based only on pretty pictures. That's a fast way to create architectural technical debt.

Instead, consider decision-making based on data, and to see what problems this architecture resolves and which problems this architecture leaves open. Then, you'll have the least technical debt and it will be transparent to you, so you can continue making decisions about it.

In Manage It! Your Guide to Modern, Pragmatic Project Management, I explained a schedule game called "We'll go faster now." That's the game where the project team thinks they can improve their speed, even though there is no data that says they can.

There's a similar management game, called "Double Your Velocity." Say a team is new to agile. They estimate they can complete 72 points in a two-week iteration. In the first iteration, they complete 30. Ok, they figure out what's wrong during the retrospective, and they complete 55 points in the second iteration. They learn more about their estimation, so they estimate the stories better, and they refine what done means, so they plan for 63 points in the third iteration. They make it! They keep proceeding, and eventually, they settle in at around 67-68 points per iteration.

Now, a senior manager wants to "help" the team. I see this often masquerading as "motivation." A senior manager comes to the team, gives them a pep talk, and says, "I want to see you double your points for the next iteration." Depending on the team, they may ignore the manager, flip the manager the bird, or placate the manager by doubling their points for each estimate.

This is a management velocity game. It's a game, because management can't motivate people to do more work. Teams can see what's slowing them down, remove those obstacles and maybe they can finish more work. But this business of "motivation"? What a bunch of nonsense. (Tell us how you really feel, Johanna :-))

The real problem is when the manager thinks that the placating team has increased velocity based on his or her actions or pep talk. That's when the team might think they can go faster now, or they keep playing with their estimation points.

Velocity is personal to a team. Period. Mucking with the points does not make the work get done faster.

I like tracking velocity for an iteration when I'm using agile. Teams often take several iterations to find their velocity--anywhere from 3 to 7 iterations. That's because the team has to relearn how to estimate, to make requirements user stories, to think about what done means, and how to make their stories smaller. I like estimating in points rather than hours or days, and that can be a challenge for people who are unaccustomed to thinking about relative size rather than duration. (See How to Estimate Without Time Units for a  quick summary.)

I find tracking velocity helpful in these ways:

1. Once we get past the first few iterations, velocity should be relatively constant, unless people are missing. If it is not, that's an indication that something might not be right.
   
2. A decrease in velocity, assuming no one is on vacation or away, may indicate technical debt.
3. An increase in velocity may indicate technical debt because the features are not really done.
4. Or, a velocity increase may be a result of the team revising how they create user stories or estimate.

Velocity changes are not good or bad, they are. And, if you track velocity, you will have more information about the team's progress.

I particularly like to track velocity as a burnup chart.

I'm just back from Adobe's partner conference in Amsterdam, where George Neill, Lead Experience Architect at Adobe Consulting, shows us this great slide depicting a woman processing mortgage applications. She has a PC on her desk which is running her organisation's app for managing mortgage applications. However, around here desk are multiple signs of usability failure. On her left, a paper calendar with names and phone numbers handwritten onto deadline days. On her right, an old-fashioned paper roll calculator. In front of her, a pile of paper forms colour coded with post-it notes. The app, Neill notes, should be handling all these tasks, but one glance at the user's working process is sufficient to expose its poor design.

All good stuff. The next thing we see is a nice application built with an Adobe Flex front-end and an Adobe LiveCycle back-end, with the not-so-subtle implication that these user-experience (UX) focused tools will help us create applications that fix this kind of design failure. There's also talk here at Adobe's partner conference of a new era in software built on customer experience, and how consumer technology from iPhone to Facebook and Twitter is raising expectations when it comes to business applications. There are even a few digs at developers, the people who, it is alleged, hate it when requirements change and who develop software geared towards the IT system which which it integrates, rather than towards users.

There is truth in all of this, but I'm cautious. It is easy to find a poor application and poke fun at it, but the idea of observing users and creating applications that improve their productivity is not a new one, and it is not necessary to use Adobe's stuff in order to do good software design. There are even times when it gets in the way. I recall spending time looking for a campsite on the web, for a holiday, and how it was in general the simple HTML sites rather than the "rich" Flash-driven ones that were easier to navigate. Admittedly the average campsite does not create web applications to Enterprise standard, but the underlying point is that there is a case for simple as well as a case for rich. Never forget "skip the intro".

The key question here: how do we define excellence in user experience? Let me state the obvious for the moment.

Applications that have functional deficiencies will not be rescued by any amount of eye candy or beautiful state transitions; and if the user is surrounded by post-it notes and antique calculators I'd suggest that this is not primarily a UX issue, unless we strip all meaning from the term and use it for every aspect of software design, features and performance.

Second, software can deliver a good UX without necessarily having gorgeous graphics and multimedia. As a business software user, I value applications that let me accomplish tasks quickly and easily. Question: think of a web application that changed the world, partly thanks to superb UX? Answer: Google search. Question: how much PhotoShop and Flash was used in designing the fantastic Google home page?

The case for user-centric software design is irrefutable; but we need rigour when it comes to working out what that means and how to achieve it. I like this comment which I found in an 2004 paper by Larry Constantine:

User-centered design is a good idea in need of improvement. The needed improvement is found in practices that put uses rather than users at the center of design and in changing the prime objective from enhancing user experience to enhancing user performance.

UP rather than UX resonates with me. It also reminds me of Kathy Sierra's mantra: creating passionate users. If the current rush towards UX puts the focus there, I am all in favour. If it means newly empowered designers imposing some sort of visual or multimedia experience on us whether we like it or not, count me out.

A client who's trying to move to agile is having trouble with the idea of literally standing at standup meetings. "I get the whole part of keeping the meeting short, but why do we have to stand up? My legs get tired."

I explained that when people don't stand, they are more likely to initiate and jump into sidebar conversations, aka rat-holes. "We don't do that here."

Um, yes, they did. And, we had the video to prove it. We were trying to do a feature-in-a-day, something I often do with clients, to see where the bottlenecks are, and how little they can actually do and still deliver valuable results. We were doing one-hour timeboxes with standups in between to see what was actually going on. We gained a number of valuable insights in our standups:

1. The one-hour timebox was too short. We should have done two-hour timeboxes. But we didn't because not everyone could free their schedules to participate every two hours, so we learned something about their projects and their daily organization.
2. When everyone stood up, the standups were no more than 5 minutes. When even one person sat down, we had tangents and rat-hole discussions.
   
3. They discovered that creating a project on their configuration management system cost them more than 30 minutes. Very interesting.
4. They learned that they had people who were even more specialized than they originally thought.
5. Almost everyone was quite willing to learn what other people do and how to help them.

If you are considering moving to agile, try a feature-in-a-day. And, as you do that, do practice standup meetings and standing up for them.