When looking at the 4th annual survey on the state of agile development conducted by VersionOne [Ver09] two trends within the world of software development can be identified:
- Agile Development Practices: 84% of the surveys participants work in companies which are using agile practices for software development.
- Distributed Development Teams: 58% of the surveys participants work in companies which have distributed teams.
However, by definition agile software development requires a close physical proximity of all participants ranging from the developers up to the customer. For example face-to-face communication is an important key aspects of agile software development [SW07] and it is obvious that this practice will not be feasible within distributed teams. How can these two fundamentally different software development paradigms be combined? What new challenges will the involved parties face? What drawbacks will arise with regard to the agility of the development when the agile principles are applied to distributed teams? This series of blog posts tries to provide answers to these questions by evaluating the experiences which where gathered within different agile distributed software projects.
The first part of this series will focus on the motivation and the new challenges which have to be faced when developing software in an agile and decentralized manner.
You might also be interested in the other posts of this series:
Part 1: Motivation and Challenges
Part 2: Best Practices
Part 3: Case-Study on Eclipse
Part 4: Evaluation and Summary
There are several drivers for extending the agile methologies such that these can be applied within a distributed environment.
By off shoring or outsourcing a company can reduce the costs of the development process by relocating some of the work to countries with a lower wage level [BNK11]. Ade Miller states:
An offshore service provider may represent an apparent 25% cost saving over a domestic provider [Mil08, 5]
But he also points out that it’s difficult to calculate the effective cost reduction because additional costs due to increased traveling expenses and reduced productivity might occur.
Level of Excellence
Another benefit of distributed teams can be the increased pool of potential software developers which can be included within the company [BNK11,TBK05,Mil08]. As a result the company can hire more highly qualified software developers which leads to an increased level of excellence. This will not only raise the productivity of the company but it will also have a positive effect on two of the twelve agile principles as one of the values stated in the agile manifesto is to focus on Individuals [BBvB+01] this fact will have a positive effect on two of the twelve agile principles:
Build projects around motivated individuals. […] [SW07, 11]
Due to the increased competition during the hiring procedure it is possible to select more highly motivated employees.
Continuous attention to technical excellence and good design enhances agility.[SW07, 11]
With an increased amount of highly qualified software developers it is possible to raise the understanding for good design and technical excellence which makes it easier fulfill this principle.
To be successful in new markets it is important for a company to get experience in these fields. Instead of accumulating this experience within the company by trainings – which might be expensive – this can also be achieved by buying up or forging alliances with companies which are experts in this new market.[Mil08]
When following a distributed agile software development process it is also possible to integrate third party developers using open source software development. Open source software development provides many benefits including an extended developer base which will voluntarily contribute to the software to improve it [TBK05] and additional testers which will fill out bug reports resulting in an increased quality of the software. Furthermore, releasing the software as open source might lead to a domination of that market segment as Morkel Theunissen et al. shows on basis of JBoss:
[…] JBoss [is] using the OSS [Open Source Software] characteristic of its J2EE Web Application solution to try to become the preferred option for an installation and development base. [TBK05, 271]
Challenges of Agile Software Development within Distributed Teams
To understand the new challenges which arise when the agile methologies are extend to be used within distributed teams it is necessary to first look at the characteristic of both the agile and the classic distributed software development process.
Characteristics of the Classic Distributed Software Development Process
Control of Process
Usually distributed software development is controlled using formal processes like the waterfall model. In such formal processes the quality requirements and targets of the development process are explicitly defined before the development of the software actually begins. Furthermore, the software architecture is also specified during a planning phase and won’t be changed later on. This ensures that the work can be distributed in arbitrary well defined work pieces to the different distributed teams.[BY05,RCMX06]
Communication within the Teams
Within a classic distributed software development environment the communication is regulated by formal mechanisms. After every step within the software development model specific artifacts are created which serve as basis for further communication e.g. Requirements Document, Program Flowcharts or Test Description. As a result the communication between the different teams often occurs to be indirect. [BY05,RCMX06]
Trust within the Teams
As a result of the indirect communication between the individual teams it is likely that there exists a lack of trust. Each team identifies it self as an own unit with only limited relation to the other teams and the common goals of the company. [RCMX06]
Characteristics of the Agile Software Development Process
Control of Process
As the agile manifesto states agile software development is relying on lightweight processes with self-organizing teams. The continuous collaboration with the customer replaces an predefined contract where changes can be taken into account. [BBvB+01,RCMX06]
Communication within the Team
The main focus in an agile software development environment lies on direct communication inside the team and not on explicit communication through documents. [RCMX06] This is realized by various mechanisms like the physical proximity of the members to lower the hindrances of communication (e.g. pair programming [Coc01] and osmotic communication [Coc04]) or daily meetings (e.g. daily scrums [Pic10]) where the latest progress and the future work is discussed.
Trust within the Team
Because of the close collaboration within the team, the direct communication and the physical proximity of the team members the trust is build progressively and every team member can identify himself with the shared goals. [RCMX06]
New Challenges of Distributed Agile Development
When comparing the characteristics of the two approaches it is clear that new challenges will show up when both approaches are combined. The three core properties of both approaches (control of process, communication within the teams and trust within the teams) are realized in a completely different manner. Balasubramaniam Ramesh deduces the following conflict categories [RCMX06, 42]:
Communication need vs. communication impedance
The communication can be hindered by various factors. An obvious cause for this conflict lies within the geographically dispersed teams. Due to that it is impossible to perform face-to-face or osmotic communication throughout the whole development. Another reason for this problem can be the different time zones because there might be only few overlapping working hours where the teams could perform a direct communicating with the help of tools like a telephone.
Thus the communication can not be realized like it would be within the classic agile approach because of the distribution of teams. As a result further attention on this key aspect of agile development is thus required which defines one of the new challenges of agile distributed software development.
As a result of the hindered communication another challenge lies within integrating new developers. It will be harder for them to access the information which is spread among the different development sites and because direct communication is not always possible it will take longer until these developers can catch up.
Another aspect which is affected by the communication impedance is the customer collaboration. In a co-located agile project the customer can be integrated into the development process. Hence it is possible for the customer not only to see if the requirements where understood correctly by the developers but also if his requirements really reflect his needs or if it is necessary to adjust these. This direct customer collaboration is not possible with distributed teams and thus yields a new challenge.
Fixed vs. evolving quality requirements
Within the classic distributed software development process fixed quality requirements are used because it is hard to communicate requirements to the distributed teams. With changing requirements which is also one key aspect of agile development it will become more frequent to communicate the evolved requirements.
As shown in the communication category this can not be handled as usually when applying agile software development to distributed teams which leads to the challenge of how to effectively communicate new requirements so that misunderstandings are avoided.
People vs. process oriented control
Not only due to the classic distributed software development process but also because of the cultures of different countries the developers are often used to other organizational models. Especially the Asian cultures are used to hierarchical chain of commands which do not consider a control by people. Martin Fowler states:
[…] a major Indian contracting company defined management as “the science of control” [Fow06,#DontUnderestimateTheCultureChange]
Furthermore, he is pointing out the importance of handling cultural differences:
One of the hardest parts of introducing agile methods into an organization is the cultural change it causes. Indeed we’ve found that this is the major reason why organizations have problems with adopting agile methods. [Fow06, #DontUnderestimateTheCultureChange]
This demonstrates a core challenge which the distributed teams will face when trying to establish self-organizing control structures: Awareness of the cultural differences.
Formal vs. informal agreement
The targets within an agile development process are defined by informal agreements. Because it might be hard to communicate these informal agreements to the distributed teams it might be necessary to create more formal documentation.
The challenge which arises from this conflict is to determine the correct amount and format of the formal documentation which is required to ensure the specification of the requirements without falling back to the documentation overhead of classic distributed development.
Lack of team cohesion
As described in the trust section of the classic distributed software development it is hard to build trust between the dispersed teams. As a result the self-organization of the teams which is an important factor in the agile development won’t be easy to achieve. Furthermore, a lack of team cohesion can lead to a continuous degeneration of the communication.
One challenge of the agile distributed software development lies thus within breaking this loop to build up trust and cohesion between all distributed teams.
Another category which does not fit into the high level view defined by Ramesh is about the structural challenges.
Structural Challenges [Fow06]
It might be necessary to have all infrastructure related systems like code repositories, build and test servers online for 24 hours a day because these systems will be accessed from different sites and thus different time zones. This can make it hard to perform backups because the state of the systems is continuously being changed.
Furthermore, to avoid unnecessary delays within the development process it might be necessary to reproduce some systems on remote sites because the delay of transferring data over the internet can have a negative impact on the productivity and in the worst case it might happen that the network is unreachable for some time.
As one can see there are many good reasons why companies should use agile methodologies within distributed teams. However, the new challenges – mainly the lack of direct communication – which arise can not be neglected and a fitting strategy to counter these challenges must be found to avoid a failure of the project.
The next post will focus on best practices which can help to overcome the challenges of agile software development in distributed teams.
- [BBvB+01]: Kent Beck, Mike Beedle, Arie van Bennekum, Alistair Cockburn, Ward Cunningham, Martin Fowler, James Grenning, Jim Highsmith, Andrew Hunt, Ron Jeffries, Jon Kern, Brian Marick, Robert C. Martin, Steve Mellor, Ken Schwaber, Jeff Sutherland, and Dave Thomas. Manifesto for agile software development. Website, 2001. http://agilemanifesto.org/. Retrieved 28. May 2011.
- [BNK11]: Udayan Banerjee, Eswaran Narasimhan, and N. Kanakalata. Experience of executing fixed price off-shored agile project. In Proceedings of the 4th India Software Engineering Conference, ISEC ’11, pages 69-75, New York, NY, USA, 2011. ACM.
- [BY05]: Lawrence Bernstein and C. M. Yuhas. Trustworthy Systems Through Quantitative Software Engineering. Wiley-IEEE Computer Society Press, 2005.
- [Coc01]: Alistair Cockburn. Agile Software Development. Addison-Wesley Professional, 2001.
- [Coc04]: Alistair Cockburn. Crystal Clear: A Human-Powered Methodology for Small Teams. Addison-Wesley Professional, 2004.
- [Fow06]Martin Fowler. Using an agile software process with offshore development. Website, 2006. http://martinfowler.com/articles/agileOffshore.html. Retrieved 9. February 2011.
- [Mil08]: Ade Miller. Distributed agile development at microsoft patterns and practices. Website, Article, 2008. http://www.microsoft.com/downloads/en/details.aspx?FamilyID=2143646D-BCC1-4D97-B9A1-968201917BAA
- [Pic10]: Roman Pichler. Agile Product Management with Scrum: Creating Products That Customers Love. Addison-Wesley Longman, 2010.
- [RCMX06]: Balasubramaniam Ramesh, Lan Cao, Kannan Mohan, and Peng Xu. Can distributed software development be agile? Commun. ACM, 49:41-46, October 2006.
- [SW07]: James Shore and Shane Warden. The Art of Agile Development. O’Reilly Media, 2007.
- [TBK05]: W. H. Morkel Theunissen, Andrew Boake, and Derrick G. Kourie. In search of the sweet spot: agile open collaborative corporate software development. In Proceedings of the 2005 annual research conference of the South African institute of computer scientists and information technologists on IT research in developing countries, SAICSIT ’05, pages 268-277, Republic of South Africa, 2005. South African Institute for Computer Scientists and Information Technologists.
- [Ver09]: VersionOne. State of agile development survey 2009. Website, 2009. http://www.versionone.com/pdf/2009_State_of_Agile_Development_Survey_Results.pdf. Retrieved 5. July 2011.