Motivation Archive

Introduction

In the previous posts of this series various benefits and obstacles of agile software development for distributed (offshored) teams were shown and solutions for these problems were provided. On basis of the Eclipse development process an implementation of this variation of the original agile principles was provided and it could be seen that such an approach can be a success.

In the final post the original defined twelve principles will be evaluated and it will be determined if each of them is still applicable in such a distributed environment.

You might also be interested in the other posts of this series:
Part 1: Motivation and Challenges
Part 2: Best Practices
Part 3: Case-Study on Eclipse
Part 4: Evaluation and Summary

Evaluation of the Agile Principles within Distributed Teams

To further demonstrate the challenges and to show drawbacks of agile distributed software development the twelve agile principles[BBvB+01] will be evaluated in the following section and it will be determined if these are still applicable within distributed teams.

  1. Our highest priority is to satisfy the customer
    through early and continuous delivery
    of valuable software.[SW07, 11]

    This principle can be achieved because it might be even possible to allocate more resources (manpower) in a distributed environment than with only one site [Bat09] and classic agile models which are often limited to very small teams (e.g. crystal clear 1-6 members [Coc04, 150]). Furthermore, as previously stated, the level of excellence of each team member might be increased which also results in a higher level of quality of the software (resulting in a more valuable software).

  2. Welcome changing requirements, even late in
    development. Agile processes harness change for
    the customer’s competitive advantage.[SW07, 11]

    The reduction of costs in an off shored or outsourced environment might lead to an increased flexibility concerning changed requirements [Bat09] but additional communication is required (e.g. visits of coordinators from different teams) to ensure that the changes are understood in the right manner. This communication overhead resulting from the dispersed teams might lead to an increased delay when reacting to changes compared to a single site. Thus this principle is feasible but it with the limitation that the changes might not be processed as fast as within a classic agile software development environment.

  3. Deliver working software frequently, from a
    couple of weeks to a couple of months, with a
    preference to the shorter timescale.[SW07, 11]

    As described at the first principle more resources might be available because of the distributed teams. But as a result of the hampered direct communication between distributed teams the release cycles might be increased compared to a co-located team.

  4. Business people and developers must work
    together daily throughout the project.[SW07, 11]

    The communication between the developers of sites which are not located close to the origin of the client might be hindered or even be impossible. This can be caused by two factors. The first might be the geographic distances. If these distances are too big the clients representative might not be willing to travel – and rotate – all the way to the other sites. The second can be a result of the different time zones. If the expert from the client is not willing to travel to the different sites and wants to communicate via e.g. telephone it might be hard to find a convenient time for both sides where this communication should take place. [Fow06,Bat09] Because of these factors it is hard to apply this agile principle to distributed teams but as Mikko Korkala et al. showed in their paper it is possible within globally distributed teams to establish frequent customer communication. [KPC10] Furthermore, they pointed out the importance of this principle in an agile globally distributed context:

    […] when adopting an agile approach in the globally
    distributed context […] the company should assure that […] at
    least some representatives from the customer group is always
    involved in the agile meetings.
    [KPC10, 46]

  5. Build projects around motivated individuals.
    Give them the environment and support they need,
    and trust them to get the job done.[SW07, 11]

    As a result of the increased pool of potential employees the motivation of each team member might be increased in distributed teams. But on the other hand it might be hard to control the motivation on the different sites. [Bat09] Additionally, because of the geographically dispersed sites it might be harder to support the team members and provide the required environment which they need to be fully productive. But with some controlling overhead it is possible to apply this principle to distributed teams.

  6. The most efficient and effective method of
    conveying information to and within a development
    team is face-to-face conversation.[SW07, 11]

    It’s impossible to apply this principle to geographically distributed teams. The only solution is a trade-off by using direct communication tools such as instant messenger, telephone or a video conference. But even with this tools it might still be very hard or impossible to establish a direct communication with a different site because of the different time zones and indirect communication mechanisms must be used e.g. a Wiki. [Fow06,Bat09]

  7. Working software is the primary measure of progress.[SW07, 11]

    As previously shown with the example of Continuous Integration additional attention must be put on the test and build system so that this principle can be applied. But in general this is feasible. [Fow06,Bat09]

  8. Agile processes promote sustainable development.
    The sponsors, developers, and users should be able
    to maintain a constant pace indefinitely.[SW07, 11]

    Overall this can also be applied to distributed teams (e.g. 40 hour weeks). But due to communication with sites located in a different time zone additional exhaustion might occur which has to be taken into account by the team leaders. [Bat09]

  9. Continuous attention to technical excellence
    and good design enhances agility.[SW07, 11]

    Because of the increased level of excellence within the teams resulting from the larger pool of potential software developers this principle might even be more practicable than with a single team.

  10. Simplicity-the art of maximizing the amount
    of work not done-is essential.[SW07, 11]

    This principle is feasible [Bat09] but it requires more attention because the communication might not be as direct as it would be with only one team which could lead to duplicated work.

  11. The best architectures, requirements, and designs
    emerge from self-organizing teams.[SW07, 11]

    The lack of face-to-face communication, the different cultural backgrounds of the teams and different holidays within the countries can result in a hindered self organization of the teams. To ensure a proper self organization additional measures have to be taken such as regularly exchanging developers between the teams and continuous awareness by the team members of the different cultural aspects. [Fow06,Bat09]

  12. At regular intervals, the team reflects on how
    to become more effective, then tunes and adjusts
    its behavior accordingly.[SW07, 11]

    Besides the already mentioned communication complications the application of this principle can also be hindered by unawareness of the cultural differences. If this challenge is not faced with the proper commitment activities like the retrospective where the whole team contributes to a process improvement will be hard to be realized successful [SW07,Fow06]. So to make this principle feasible within distributed teams the members with a different cultural background have to be introduced into these new aspects of the agile approach and then be encouraged to apply them.

Conclusions

It should be clear by now that there are several agile principles which are not applicable or limited when developing in distributed teams. These missing or restricted principles lead to many different challenges – where the main challenge lies within communication hindrances resulting from the lack of face-to-face communication and the physical dispersed teams – which makes it hard to apply agile principles to distributed teams. But when additional practices are introduced into the agile software development process it is possible to extend the classic co-located agile development to distributed teams. These additional practices are no replacement for the original practices but when followed with commitment and discipline a sufficient trade-off can be achieved. Especially the establishment and maintenance of communication channels is one key aspect of success in a distributed agile development process.

When a company introduces agile software development methologies to distributed teams they will face risks associated with this approach as Ade Miller concludes from the experiences of his team:

the decision maker(s) must understand that in doing so [distributing the project] they reduce the project’s likelihood of success, increase the delivery time, and reduce the team’s performance and increase its dysfunction. [Mil08, 18]

But the company can also profit from various advantages like the possibility to step into new markets or to increase the level of excellence within their employees.

Whether or not to use the agile methologies for distributed teams must thus be carefully balanced out for every project.

References

[Bat09]: Dinesh Batra. Modified agile practices for outsourced software projects. Commun. ACM, 52:143-148, September 2009.
[BBvB+01]: Kent Beck, Mike Beedle, Arie van Bennekum, Alistair Cockburn, Ward Cunningham, Martin Fowler, James Grenning, Jim Highsmith, Andrew Hunt, Ron Jeffries, Jon Kern, Brian Marick, Robert C. Martin, Steve Mellor, Ken Schwaber, Jeff Sutherland, and Dave Thomas. Manifesto for agile software development. Website, 2001. http://agilemanifesto.org/. Retrieved 28. May 2011.
[Coc04]: Alistair Cockburn. Crystal Clear: A Human-Powered Methodology for Small Teams. Addison-Wesley Professional, 2004.
[Fow06]: Martin Fowler. Using an agile software process with offshore development. Website, 2006. http://martinfowler.com/articles/agileOffshore.html. Retrieved 9. February 2011.
[KPC10]: Mikko Korkala, Minna Pikkarainen, and Kieran Conboy. A case study of customer communication in globally distributed software product development. In Proceedings of the 11th International Conference on Product Focused Software, PROFES ’10, pages 43-46, New York, NY, USA, 2010. ACM.
[Mil08]: Ade Miller. Distributed agile development at microsoft patterns and practices. Website, Article, 2008. http://www.microsoft.com/download/en/details.aspx?=amp&id=14916
[SW07]: James Shore and Shane Warden. The Art of Agile Development. O’Reilly Media, 2007.

by Kevin Kratzer

Introduction

The previous post showed some additional practices for agile software development with distributed teams. This post will give you an example of a project which is successfully developed for a long time using globally distributed teams and open source contributors.
Furthermore, the main agile practices will be highlighted and the additional practices which are followed for the distributed environment will be pointed out.

You might also be interested in the other posts of this series:
Part 1: Motivation and Challenges
Part 2: Best Practices
Part 3: Case-Study on Eclipse
Part 4: Evaluation and Summary

Case Study: Eclipse [Ste04,Gam05a,Gam05b]

Eclipse LogoAn example for agile software development which is applied to distributed teams is provided by the Eclipse Foundation. This foundation is an open source community which is using agile methologies within geographically dispersed teams to develop Eclipse (http://www.eclipse.org/)which is not only an integrated development environment including a java development environment but also a general tools and integration platform which can even be used as a general application platform to develop diverse applications. The contributers to the software are not only the open source developers but also six globally dispersed teams which are located in: Canada (Ottawa, Toronto), United States (Portland, Raleigh), France (St. Nazaire) and Switzerland (Zürich).

Practices Used

The Eclipse Foundation is using various practices within the development of Eclipse:


The orange boxes highlight the involvement of distributed contributors within the development process of eclipse.

In the following sections the core practices used will be further discussed.

Key Practices Used for Agile Software Development

Continuous Testing

Testing is already taken into account early within the development cycle and these tests are then performed frequently and automated to ensure working software. The tests do not cover only functionality but also the performance.

Continuous integration

Eclipse is using an automated build and testing environment to ensure a working software. The builds are performed on a nightly, weekly and milestone basis whereas successful milestone builds are released to the community.

Incremental Design

With changing requirements the design is adjusted to the needs but these changes are limited to ensure that the public API is downwards compatible. A stable API is important for Eclipse because it focuses on extensibility using the plug-in design pattern [Fow02].

Short Development Cycles

Within the release cycle which takes 6-15 months each iteration is limited to six weeks.

Retrospectives

After each release a so called decompression phase is performed which is time boxed to one month. Within this phase the lessons learned from the last cycle including achievements and failures are evaluated and the processes are refined accordingly. Furthermore, this phase is used to recover from the release to ensure that the developers can keep up a constant pace. In the end the next release and cycle are getting planned.

Key Practices Used for Distributed Agile Software Development

To apply the agile practices to the distributed and open source environment Eclipse is following various practices. There are three core practices which are explained in this section:

Milestones

Each Milestone represents a miniature development cycle including planning, execution, testing and a retrospective. To make it easier for the distributed teams and the open source community to follow the milestone and to see the progress of the software the milestone plans are published to publicity beforehand and after the milestone is reached the results which where achieved are getting advertised.

Community Involvement (Customer Involvement)

As Eclipse is an open source project the customer of the product is the community. So it is an important part of the distributed agile development that all aspects of the development is performed in a transparent way. As a result the community can actively participate in the development and provide feedback which is taken into account by the developers to adjust the software.

Component Architecture

The whole software is splitted up into different components. These components are then assigned to specific sites (e.g. SWT is developed in Ottawa and the team in Portland is developing the debugger). By this approach the development of single components can be performed in an classic agile manner within one team and the communication need with other teams is reduced. If there are cross component problems occurring these are handled by dynamic created teams which will handle this specific problem. The members of this team are key developers of the affected components which ensures that the knowledge base of the newly created team is sufficient to understand and solve the problem.

Summary

Eclipse is following various agile principles and is applying these not only to globally distributed teams but also on an open source environment and as Erich Gamma states apparently with good success:

Throughout the history of Eclipse the development team was successful in hitting
projected delivery dates with precision and quality. [Gam05a, 4]

As a result the software development process of the Eclipse project can be counted as a successful application of the agile software development methologies in an distributed context because quality software releases can be delivered on time since the beginning in 2001.

The final post will try first to evaluate the twelve original agile principles if these are still applicable in an distributed environment and then give a small summary on the content provided in this series of blog posts on agile distributed software development.

References

[Fow02]: Martin Fowler. Patterns of Enterprise Application Architecture. Addison-Wesley Professional, 2002.
[Gam05a]: Erich Gamma. Agile, open source, distributed, and on-time: inside the eclipse development process. In Proceedings of the 27th international conference on Software engineering, ICSE ’05, pages 4-4, New York, NY, USA, 2005. ACM.
[Gam05b]: Erich Gamma. Agile, open source, distributed, and on-time: inside the eclipse development process. Website, Presentation Slides, 2005. http://www.icse-conferences.org/2005/ConferenceProgram/InvitedTalks/GammaKeynote.pdf. Retrieved 18. May 2011.
[Ste04]: Christoph Steind, IBM Global Services Application Management Services. Distributed agile. Website, Presentation Slides, 2004. http://cf.agilealliance.org/articles/system/article/file/1420/file.pdf. Retrieved 31. May 2011.

by Kevin Kratzer

Introduction

The previous post on distributed agile software development – which can be a result of offshoring, outsourcing, multiple sites or just separated rooms within a company – discussed the benefits, motivation and the new challenges which will arise in such an environment. The follow-up post in this series will cover proven practices which can help to face these new challenges and lead to a successful project execution.

You might also be interested in the other posts of this series:
Part 1: Motivation and Challenges
Part 2: Best Practices
Part 3: Case-Study on Eclipse
Part 4: Evaluation and Summary

Best Practices

The agile manifesto describes different practices which have to be followed to ensure that the software development really is agile. With agile software development in distributed teams there are some practices which can’t be fulfilled (e.g. face-to-face communication). This problem is also stated by Ade Miller:

Agile approaches […] rely on a set of mutually supporting practices. Choosing to
drop one practice will weaken the remaining ones. Therefore when the communication bandwidth is
reduced teams must supplement weakened practices or replace them. Simply abandoning them […] will reduce the team’s level of agility.[Mil08, 7]

To avoid the weakening of other practices Martin Fowler [Fow06] and Ade Miller [Mil08] are providing solutions which have been successfully used within their distributed agile development environment. In the following the core practices which allowed this transformation of the agile methologies to distributed teams are explained and their impact on the previously shown challenges will be pointed out. These practices are meant as addition to existing agile software development methologies so that these can be used within a distributed environment.

Provide Multiple Communication Channels

As previously shown there are several challenges which arise from the communication hindrances. Because the required face-to-face communication is not possible within distributed teams it is necessary to provide multiple alternatives for this kind of communication.

Wiki

A Wiki is the perfect choice to contain information which should be written down like design documents, metrics or build processes. Because Wikis are usually unstructured it is possible for them to grow with the project in an agile manner and will thus hold the information in a way which fits the teams needs the best. Additionally, the version history of documents can represent the iterative change of the project (e.g. design documents). This version history of the documents can then later on be used to extract information why specific changes were made. Another benefit of a Wiki is the change notification mechanisms like RSS which can be used to keep every team member up-to-date. However, the agile growth of the Wiki might make it necessary to assign an employee to look over the Wiki to avoid needless posts.

Instant Messenger

Instant Messenger provide three positive aspects. The first is that it is easy to determine whether a developer is currently available for questioning or not because the messenger usually provide status information about the user. This can be a great advantage when the overlapping time between two sites is limited due to the different time zones. Secondly, communication through an instant messenger is fast (synchronous) and the overhead is limited due to the informal manner of this type of communication. As a result it serves well as a replacement for the informal communication between co-located developers in a classic agile environment. The last advantage of instant messengers is that today’s clients usually support group communication which can act as a replacement for osmotic communication because all involved developers will all receive the questions and answers and might decide to actively contribute to the communication if they are familiar with the topic. This can also be achieved by chat protocols like IRC.

E-Mail

E-mail can be used as an addition to the instant messenger as it provides an asynchronous way of communicating information to multiple receivers. Especially when combined with newsgroups the information published can be easily accessed later on and the wanted information can be extracted.

Voice and Video

Voice (including telephone) and video chats are an important form of communication for distributed teams. It must be ensured that it is easy for the developers to use these mechanisms by providing head sets and web cams and to tell the developers that they should not be concerned about costs for the communication because the impact of missing communication would be even worse than some expenses for a telephone call. With the help of voice and video communication it is possible to organize daily meetings by providing rooms which contain cameras, microphones and a beamer where the distributed teams then can exchange important information. Furthermore, it is also possible create video recordings of meetings where one developer team could not participate to share it with them later on. By this it will be easier to catch up and no additional documents have to be created which are documenting that meeting.

Telepresence Systems

Telepresence systems are robots which are placed on a specific site and then can be controlled by an operator over the internet. These systems are often mobile and include features like microphones, speakers, a video camera and a display which shows the operators webcam picture. Using these systems an advanced version of a video conference can be achieved. Furthermore, it is possible for the operator to interact with the employees at the remote site e.g. by moving the robot to their desk. Currently, there are only few telepresence robots available but in the future more systems might be developed. [Sim11]

Summary

It is an urgent matter to make the presented communication mechanisms available directly from the beginning of the project to avoid misunderstandings. However, these mechanisms cannot fully replace the direct communication but they provide a trade-off. If possible the synchronous communication mechanisms should be used but as a result of the different time zones it might be necessary to fall back to the asynchronous communication techniques which will add delay to the development process when compared to co-located agile software development.

Travel Frequently

There are two types of visits which should be considered: co-location visits and long-term visits.

Co-Location Visits

These visits should be executed within the first few iterations of a project. The characteristic of this kind of visit is that all team members are getting assembled within on location. During that time it is possible to build trust between the – under normal circumstances distributed – teams, create bonds and establish communication channels which can later be used to avoid misunderstandings. But not only the team members should be present it is also important that the project managers and an expert from the customer are present which will provide these co-location visits a working environment which is the same as it would be within a classic agile software development process. Especially for creating a release plan and the development within the first few iterations this can have a positive impact on the quality of the created software because many key design decisions are being made at the beginning of the project and it is better if the whole team and the customer can equally contribute to these decisions. It is also possible for every team member to reduce the vagueness concerning the requirement specification which can avoid later misunderstandings.

Every few months these visits should also be repeated periodically. It is sufficient for these periodic visits to take about one week but it is also recommended to plan longer visits if bigger changes within the software are being made or if some communication problems occurred. These visits can be used to refresh the bonds and trust which is fading over time within distributed teams. Furthermore, these periodic visits are a good place to reflect on the process, the achievements and the failures and to take appropriate actions like updating the process during a retrospective.

Additionally, such a visit should also be performed during the last few iterations before shipping a release to avoid delay in the development due to communication hindrances during that important time. If the whole team is assembled after the release is shipped this visit can then also be used to perform a retrospective.

To ensure the establishment of relationships during the visit some social events should be organized and the work pace should be relaxed. Additionally, gifts representing the different culture of the visitors will also help to break the ice and will help to initiate the communication. Martin Fowler emphasizes the importance of connections on a personal level:

Dinners and sightseeing can often be the most useful activity that the visitors do with the hosts. [Fow06, #UseContactVisitsToBuildTrust]

This will allow informal communication to take place during the visit which will later pay off because more communication links will be available after the visit is over.

Co-location visits are complex and could become very costly so it might not be possible to get all developers of the different sites together. If it is not possible at least the senior developers of the different sites should perform this type of visit at the beginning of the project to ensure profound design decisions.

Long-Term Visits

The key aspect of this visit type is that on every site there is always a representative from all the other sites present. The duration of such a visit should be ranging from a few weeks up to a few months depending on the representative being exchanged. The selection of the potential representatives must take the personal needs into account because it might not be possible for some to stay abroad for a long period. The main function of the representatives is to act as a catalyst for communication. Because he is familiar with the fields of specialization of his fellow team members he can establish communication links between both sites. Furthermore, he has access to the local informal communication in that team and can extract important information which he can redirect to his site of origin. Besides the benefits of establishing communication links the representatives is also able to build up trust between the geographically dispersed team members. A long-term visit also provides the project managers a good opportunity to get to know the whole team which is important when making decisions and resolving conflicts.

Martin Fowlers team extended this approach by not just sending developers as representatives but also by exchanging analysts who can communicate the requirements between the distributed teams [Mil08, 11].

To avoid misunderstandings and a lack of trust these representatives should be always present directly after the kick-off of the project.

Summary

The presented visits should be part of the distributed agile software development to compensate the lack of face-to-face communication, provide trust between the team members, to avoid misunderstandings and to counter the as Ade Miller states ”‘Us vs. Them’ attitude

Use Iterative Documentation

One of the new challenges lies within finding the correct amount and format of the formal documentation which is required within an agile distributed software development process. Because every development environment is different it is not possible to define these parameters for every case. As a result the process of determing the required formal documentation should be performed in an iterative way. A company should start with only little documentation and constantly request feedback from the distributed teams. This feedback can then be used to adjust the amount and the format of the formal documentation in a way that it is sufficient within their specific set up. To support this agile approach all documents should be kept in a repository to make it easier for every participant to access the latest version of a specific document. Furthermore, unstructured content management systems like the previously mentioned Wiki have proven to be useful.

Continuous Integration

Continuous Integration is a practice which originated from Extreme Programming and its main goal is to avoid problems when big separated work pieces are integrated back into the system by continuously integrating small changes every few hours. To ensure that the repository does not contain code which has unresolved problems every developer should first build the code within a sandbox and validate this build by running tests. If everything went smooth the changes can be integrated back into the main repository where a build server should perform an automated build followed by tests which validate that the integration did not break the system [SW07]. Of course integration problems also exists in a distributed environment but the impacts of failed builds are even worse because a failed build can prevent a team which is working in a different time zone from further developing until the team who broke the build is back at work. This can be avoided by using continuous integration. Additionally from avoiding such development dead locks, by always committing small changes it will be easier to detect misunderstandings within the requirement specification. This can be achieved because the repository will always contain a working version of the current development status which the customer can access and check if the requirements are implemented as desired. Thus the in the agile manifesto valued customer collaboration can be improved. Another benefit of continuous integration is that the new developed functions will always be well tested which results in a higher quality of the produced code.

That is why Continuous Integration should be used when applying agile software development to distributed teams.

Distribute the Work

When distributing work to several teams it often happens that specific components are always assigned to the same team. This might work for smaller projects and – according to Conway’s Law (Conway’s Law states that the architecture of a software will reflect the organizational structures within the company [Wik11]) – will have a positive impact on the software architecture because it will reduce the degree of coupling. This approach is suggested by Martin Fowler[Fow06]. Ade Miller on the other hand points out that this kind of distribution bears a big disadvantage for larger projects. As a result of the component-wise distribution islands of expertise will be created among the distributed teams. When new requirements arise it might be hard for a team which worked always on a specific component to quickly switch to a different component to develop a new feature within this component. He states:

Distributed teams […] need to […] think in terms of user stories not system components.
[Mil08, 15]

The impact of this disadvantage is that agility within the software development is lost which should be avoided. So it is necessary to deliberate for every project about whether to use the component-wise distribution or not.

Another approach which is often used for distributed teams is the activity-wise distribution. For example this can mean that testing is done on a different site than the development. This might work well in a classic software development process but it should be avoided when agile methologies are applied to distributed teams because by first creating test cases for the requirements it is possible for the developers to deepen their knowledge and thus helps them to understand the requirements better and a lack of clarity of the specification will be detected early. Furthermore, each developer will gain a better understanding of the whole software project which will also lead to better results when developing new features.

Summary

The presented practices mainly try to replace the direct communication within the team and the customer involvement throught the whole project which are core practices used in the agile software development but which cannot be achieved within an distributed environment. Although, the practices are only a trade-off which cannot replace the original practices it is important to adapt to these new practices starting as early as possible in the project to avoid misunderstandings within the teams.
Because every team is different it is not clear if these practices will completely fit your needs. You should start of with the presented practices and evolve your own practices from them so that the practices optimally fit your teams needs.

The next post will focus on the agile practices used in the development process of eclipse which is not only developed as an open source project but also within distributed teams.

References

[Fow06]: Martin Fowler. Using an agile software process with offshore development. Website, 2006. http://martinfowler.com/articles/agileOffshore.html. Retrieved 9. February 2011.
 [Mil08]: Ade Miller. Distributed agile development at microsoft patterns and practices. Website, Article, 2008. http://www.microsoft.com/download/en/details.aspx?=amp%3bamp&id=14916
[Sim11]: Tom Simonite. Vertretung im Büro. Technology Review Online Article, 2011. http://www.heise.de/tr/artikel/Vertretung-im-Buero-1326762.html. Retrieved 24. August 2011.
[SW07]: James Shore and Shane Warden. The Art of Agile Development. O’Reilly Media, 2007.
[Wik11]: Wikipedia. Conway’s Law. Website, 2011. http://en.wikipedia.org/wiki/Conway2011.

by Kevin Kratzer

Introduction

When looking at the 4th annual survey on the state of agile development conducted by VersionOne [Ver09] two trends within the world of software development can be identified:

  • Agile Development Practices: 84% of the surveys participants work in companies which are using agile practices for software development.
  • Distributed Development Teams: 58% of the surveys participants work in companies which have distributed teams.

However, by definition agile software development requires a close physical proximity of all participants ranging from the developers up to the customer. For example face-to-face communication is an important key aspects of agile software development [SW07] and it is obvious that this practice will not be feasible within distributed teams. How can these two fundamentally different software development paradigms be combined? What new challenges will the involved parties face? What drawbacks will arise with regard to the agility of the development when the agile principles are applied to distributed teams? This series of blog posts tries to provide answers to these questions by evaluating the experiences which where gathered within different agile distributed software projects.

The first part of this series will focus on the motivation and the new challenges which have to be faced when developing software in an agile and decentralized manner.

You might also be interested in the other posts of this series:
Part 1: Motivation and Challenges
Part 2: Best Practices
Part 3: Case-Study on Eclipse
Part 4: Evaluation and Summary

Motivation

There are several drivers for extending the agile methologies such that these can be applied within a distributed environment.

Cost Reduction

By off shoring or outsourcing a company can reduce the costs of the development process by relocating some of the work to countries with a lower wage level [BNK11]. Ade Miller states:

An offshore service provider may represent an apparent 25% cost saving over a domestic provider [Mil08, 5]

But he also points out that it’s difficult to calculate the effective cost reduction because additional costs due to increased traveling expenses and reduced productivity might occur.

Level of Excellence

Another benefit of distributed teams can be the increased pool of potential software developers which can be included within the company [BNK11,TBK05,Mil08]. As a result the company can hire more highly qualified software developers which leads to an increased level of excellence. This will not only raise the productivity of the company but it will also have a positive effect on two of the twelve agile principles as one of the values stated in the agile manifesto is to focus on Individuals [BBvB+01] this fact will have a positive effect on two of the twelve agile principles:

Build projects around motivated individuals. […] [SW07, 11]

Due to the increased competition during the hiring procedure it is possible to select more highly motivated employees.

Continuous attention to technical excellence and good design enhances agility.[SW07, 11]

With an increased amount of highly qualified software developers it is possible to raise the understanding for good design and technical excellence which makes it easier fulfill this principle.

New Markets

To be successful in new markets it is important for a company to get experience in these fields. Instead of accumulating this experience within the company by trainings – which might be expensive – this can also be achieved by buying up or forging alliances with companies which are experts in this new market.[Mil08]

Open Source

When following a distributed agile software development process it is also possible to integrate third party developers using open source software development. Open source software development provides many benefits including an extended developer base which will voluntarily contribute to the software to improve it [TBK05] and additional testers which will fill out bug reports resulting in an increased quality of the software. Furthermore, releasing the software as open source might lead to a domination of that market segment as Morkel Theunissen et al. shows on basis of JBoss:

[…] JBoss [is] using the OSS [Open Source Software] characteristic of its J2EE Web Application solution to try to become the preferred option for an installation and development base. [TBK05, 271]

Challenges of Agile Software Development within Distributed Teams

To understand the new challenges which arise when the agile methologies are extend to be used within distributed teams it is necessary to first look at the characteristic of both the agile and the classic distributed software development process.

Characteristics of the Classic Distributed Software Development Process

Control of Process

Usually distributed software development is controlled using formal processes like the waterfall model. In such formal processes the quality requirements and targets of the development process are explicitly defined before the development of the software actually begins. Furthermore, the software architecture is also specified during a planning phase and won’t be changed later on. This ensures that the work can be distributed in arbitrary well defined work pieces to the different distributed teams.[BY05,RCMX06]

Communication within the Teams

Within a classic distributed software development environment the communication is regulated by formal mechanisms. After every step within the software development model specific artifacts are created which serve as basis for further communication e.g. Requirements Document, Program Flowcharts or Test Description. As a result the communication between the different teams often occurs to be indirect. [BY05,RCMX06]

Trust within the Teams

As a result of the indirect communication between the individual teams it is likely that there exists a lack of trust. Each team identifies it self as an own unit with only limited relation to the other teams and the common goals of the company. [RCMX06]

Characteristics of the Agile Software Development Process

Control of Process

As the agile manifesto states agile software development is relying on lightweight processes with self-organizing teams. The continuous collaboration with the customer replaces an predefined contract where changes can be taken into account. [BBvB+01,RCMX06]

Communication within the Team

The main focus in an agile software development environment lies on direct communication inside the team and not on explicit communication through documents. [RCMX06] This is realized by various mechanisms like the physical proximity of the members to lower the hindrances of communication (e.g. pair programming [Coc01] and osmotic communication [Coc04]) or daily meetings (e.g. daily scrums [Pic10]) where the latest progress and the future work is discussed.

Trust within the Team

Because of the close collaboration within the team, the direct communication and the physical proximity of the team members the trust is build progressively and every team member can identify himself with the shared goals. [RCMX06]

New Challenges of Distributed Agile Development

When comparing the characteristics of the two approaches it is clear that new challenges will show up when both approaches are combined. The three core properties of both approaches (control of process, communication within the teams and trust within the teams) are realized in a completely different manner. Balasubramaniam Ramesh deduces the following conflict categories [RCMX06, 42]:

Communication need vs. communication impedance

The communication can be hindered by various factors. An obvious cause for this conflict lies within the geographically dispersed teams. Due to that it is impossible to perform face-to-face or osmotic communication throughout the whole development. Another reason for this problem can be the different time zones because there might be only few overlapping working hours where the teams could perform a direct communicating with the help of tools like a telephone.

Thus the communication can not be realized like it would be within the classic agile approach because of the distribution of teams. As a result further attention on this key aspect of agile development is thus required which defines one of the new challenges of agile distributed software development.

As a result of the hindered communication another challenge lies within integrating new developers. It will be harder for them to access the information which is spread among the different development sites and because direct communication is not always possible it will take longer until these developers can catch up.

Another aspect which is affected by the communication impedance is the customer collaboration. In a co-located agile project the customer can be integrated into the development process. Hence it is possible for the customer not only to see if the requirements where understood correctly by the developers but also if his requirements really reflect his needs or if it is necessary to adjust these. This direct customer collaboration is not possible with distributed teams and thus yields a new challenge.

Fixed vs. evolving quality requirements

Within the classic distributed software development process fixed quality requirements are used because it is hard to communicate requirements to the distributed teams. With changing requirements which is also one key aspect of agile development it will become more frequent to communicate the evolved requirements.

As shown in the communication category this can not be handled as usually when applying agile software development to distributed teams which leads to the challenge of how to effectively communicate new requirements so that misunderstandings are avoided.

People vs. process oriented control

Not only due to the classic distributed software development process but also because of the cultures of different countries the developers are often used to other organizational models. Especially the Asian cultures are used to hierarchical chain of commands which do not consider a control by people. Martin Fowler states:

[…] a major Indian contracting company defined management as “the science of control” [Fow06,#DontUnderestimateTheCultureChange]

Furthermore, he is pointing out the importance of handling cultural differences:

One of the hardest parts of introducing agile methods into an organization is the cultural change it causes. Indeed we’ve found that this is the major reason why organizations have problems with adopting agile methods. [Fow06, #DontUnderestimateTheCultureChange]

This demonstrates a core challenge which the distributed teams will face when trying to establish self-organizing control structures: Awareness of the cultural differences.

Formal vs. informal agreement

The targets within an agile development process are defined by informal agreements. Because it might be hard to communicate these informal agreements to the distributed teams it might be necessary to create more formal documentation.

The challenge which arises from this conflict is to determine the correct amount and format of the formal documentation which is required to ensure the specification of the requirements without falling back to the documentation overhead of classic distributed development.

Lack of team cohesion

As described in the trust section of the classic distributed software development it is hard to build trust between the dispersed teams. As a result the self-organization of the teams which is an important factor in the agile development won’t be easy to achieve. Furthermore, a lack of team cohesion can lead to a continuous degeneration of the communication.

One challenge of the agile distributed software development lies thus within breaking this loop to build up trust and cohesion between all distributed teams.

Another category which does not fit into the high level view defined by Ramesh is about the structural challenges.

Structural Challenges [Fow06]

It might be necessary to have all infrastructure related systems like code repositories, build and test servers online for 24 hours a day because these systems will be accessed from different sites and thus different time zones. This can make it hard to perform backups because the state of the systems is continuously being changed.

Furthermore, to avoid unnecessary delays within the development process it might be necessary to reproduce some systems on remote sites because the delay of transferring data over the internet can have a negative impact on the productivity and in the worst case it might happen that the network is unreachable for some time.

Summary

As one can see there are many good reasons why companies should use agile methodologies within distributed teams. However, the new challenges – mainly the lack of direct communication – which arise can not be neglected and a fitting strategy to counter these challenges must be found to avoid a failure of the project.

The next post will focus on best practices which can help to overcome the challenges of agile software development in distributed teams.

References

[BBvB+01]: Kent Beck, Mike Beedle, Arie van Bennekum, Alistair Cockburn, Ward Cunningham, Martin Fowler, James Grenning, Jim Highsmith, Andrew Hunt, Ron Jeffries, Jon Kern, Brian Marick, Robert C. Martin, Steve Mellor, Ken Schwaber, Jeff Sutherland, and Dave Thomas. Manifesto for agile software development. Website, 2001. http://agilemanifesto.org/. Retrieved 28. May 2011.
[BNK11]: Udayan Banerjee, Eswaran Narasimhan, and N. Kanakalata. Experience of executing fixed price off-shored agile project. In Proceedings of the 4th India Software Engineering Conference, ISEC ’11, pages 69-75, New York, NY, USA, 2011. ACM.
[BY05]: Lawrence Bernstein and C. M. Yuhas. Trustworthy Systems Through Quantitative Software Engineering. Wiley-IEEE Computer Society Press, 2005.
[Coc01]: Alistair Cockburn. Agile Software Development. Addison-Wesley Professional, 2001.
[Coc04]: Alistair Cockburn. Crystal Clear: A Human-Powered Methodology for Small Teams. Addison-Wesley Professional, 2004.
[Fow06]Martin Fowler. Using an agile software process with offshore development. Website, 2006. http://martinfowler.com/articles/agileOffshore.html. Retrieved 9. February 2011.
[Mil08]: Ade Miller. Distributed agile development at microsoft patterns and practices. Website, Article, 2008. http://www.microsoft.com/downloads/en/details.aspx?FamilyID=2143646D-BCC1-4D97-B9A1-968201917BAA
[Pic10]: Roman Pichler. Agile Product Management with Scrum: Creating Products That Customers Love. Addison-Wesley Longman, 2010.
[RCMX06]: Balasubramaniam Ramesh, Lan Cao, Kannan Mohan, and Peng Xu. Can distributed software development be agile? Commun. ACM, 49:41-46, October 2006.
[SW07]: James Shore and Shane Warden. The Art of Agile Development. O’Reilly Media, 2007.
[TBK05]: W. H. Morkel Theunissen, Andrew Boake, and Derrick G. Kourie. In search of the sweet spot: agile open collaborative corporate software development. In Proceedings of the 2005 annual research conference of the South African institute of computer scientists and information technologists on IT research in developing countries, SAICSIT ’05, pages 268-277, Republic of South Africa, 2005. South African Institute for Computer Scientists and Information Technologists.
[Ver09]: VersionOne. State of agile development survey 2009. Website, 2009. http://www.versionone.com/pdf/2009_State_of_Agile_Development_Survey_Results.pdf. Retrieved 5. July 2011.

by Kevin Kratzer