See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/48446071

Coordination in Large-Scale Software Teams

Article · January 2009

DOI: 10.1109/CHASE.2009.5071401

CITATIONS READS
45 462

4 authors, including:

Andrew Begel Lucas Layman

Microsoft Fraunhofer Center for Experimental Software Engineering
89 PUBLICATIONS 4,897 CITATIONS 50 PUBLICATIONS 1,918 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Andrew Begel on 28 May 2014.

The user has requested enhancement of the downloaded file.

NRC Publications Archive (NPArC)
Archives des publications du CNRC (NPArC)

Coordination in the Large-Scale Software Teams

Begel, Andrew; Nagappan, Nachiappan; Poile, Christopher; Layman,
Lucas

Web page / page Web

http://nparc.cisti-icist.nrc-cnrc.gc.ca/npsi/ctrl?action=rtdoc&an=12735084&lang=en
http://nparc.cisti-icist.nrc-cnrc.gc.ca/npsi/ctrl?action=rtdoc&an=12735084&lang=fr

Access and use of this website and the material on it are subject to the Terms and Conditions set forth at
http://nparc.cisti-icist.nrc-cnrc.gc.ca/npsi/jsp/nparc_cp.jsp?lang=en
READ THESE TERMS AND CONDITIONS CAREFULLY BEFORE USING THIS WEBSITE.

L’accès à ce site Web et l’utilisation de son contenu sont assujettis aux conditions présentées dans le site
http://nparc.cisti-icist.nrc-cnrc.gc.ca/npsi/jsp/nparc_cp.jsp?lang=fr
LISEZ CES CONDITIONS ATTENTIVEMENT AVANT D’UTILISER CE SITE WEB.

Contact us / Contactez nous: [email protected].

 Coordination in Large-Scale Software Teams

Andrew Begel, Nachiappan Nagappan Christopher Poile

Microsoft Research Edwards School of Business
Redmond, WA, USA University of Saskatchewan
[email protected], [email protected] Saskatoon, SK, Canada
[email protected]
Lucas Layman
National Research Council
Ottawa, ON, Canada
[email protected]

Abstract gram managers. We asked engineers how they coordinate

tasks with teams they depend on and with teams that depend
Large-scale software development requires coordination on them, and how they communicate with their dependen-
within and between very large engineering teams which cies when things go wrong. We then asked how develop-
may be located in different buildings, on different company ers feel about working with dependent teams to understand
campuses, and in different time zones. From a survey an- where they would like to see improvement. We described
swered by 775 Microsoft software engineers, we learned some aspects of this research in prior work [1].
how work was coordinated within and between teams and We find that it is important to consider the different roles
how engineers felt about their success at these tasks. The that people play on their teams when coordinating with oth-
respondents revealed that the most common objects of co- ers. Processes and tools intended for software developers
ordination are schedules and features, not code or inter- may not be appropriate for program managers. The two
faces, and that more communication and personal contact job roles “live” in different applications in their daily work;
worked better to make interactions between teams go more tools intended for one role’s applications just may not be
smoothly. used by the other. In addition, intra-team and inter-team co-
ordination communication modes are very different. It may
be easy to pay a personal visit to a team member who likely
1. Introduction sits on the same floor as you, but much more difficult and
socially awkward to visit a collaborator from another team,
Coordination between software development teams is especially when that collaborator is not a friend. We also
one of the most difficult-to-improve aspects of software en- find that the overhead of communication and maintaining
gineering. Kraut and Streeter argue that the software indus- relationships between individuals who coordinate on dif-
try has been in crisis mode for its entire existence, and a ferent teams is high, but necessary to getting work done
root cause is the difficulty in coordinating work between successfully. Many respondents to our survey wished they
teams of developers [9]. Researchers have studied pro- could get more information and action about their depen-
fessional software development teams empirically to gain dencies with less active communication requirements. Even
greater understanding of how software development pro- though coordination is difficult, we find that even in a cross-
cesses, tools, and people impact coordination. The im- section of one of largest software companies in the world,
portance of intra- and inter-team coordination is a fore- almost all engineers are required to coordinate with others
most concern as software development increasingly be- to get their work done.
comes globally distributed, and remains a persistent chal-
lenge in other disciplines as well. 2. Survey Design
To understand inter- and intra-team dependencies in
large-scale software development, we conducted a web- The research was conducted using an anonymous web-
based survey of 775 Microsoft developers, testers and pro- based survey offered over a period of two weeks in Au-
 Release schedule first question asked survey recipients what artifacts they
Features depended on from other teams, and what artifacts other
APIs
teams depended on from them. The responses are shown
Bugs
Documen on
in Figure 1. The respondents report 72% depend on an-
Code other team’s release schedule and 71% depend on the fea-
Prio on of work items tures of another team’s product. At a slightly lesser level are
Status
Other
APIs (62%), bugs (62%), documentation (61%) and code
(58%). Prioritization of work items and status are around
0% 10% 20% 30% 40% 50% 60% 70% 80%
50%. When considering what other teams depend on from
Depend on from other teams Other teams depend on this
them, however, status becomes the most frequent, rising to
62%, leaving features (57%), bugs (55%), release sched-
Figure 1. What engineers depend on from ules (55%) and documentation (52%) less frequent. Less
other teams, and what other teams depend than half of the respondents say that other teams depend on
on from them. their code (46%), prioritization of their work items (45%)
or their APIs (38%).
When asked how they kept track of the work items they
depended on from other teams (shown in Figure 2) most
gust 2007 inside the Microsoft Corporation. An invitation
respondents (69% overall) reported using email. 61% use a
was sent by email to 2,535 developers, testers, program
work item database and 56% talk about them at status meet-
managers (PMs), architects and user experience engineers,
ings. After that there is a large drop to keeping track of de-
consisting of a 10% random sample of employees in each
pendencies in your head (38%) followed by using Outlook
job role. At Microsoft, program managers gather customer
tasks (30%), Sharepoint websites (29%), using a point per-
requirements, write design specifications, and manage the
son in charge of keeping track of dependencies (27%) and
project schedule. Developers turn design specifications into
Excel spreadsheets (26%). Keeping a list on paper, text ed-
implementation specifications, and write the code. Testers
itor or personal whiteboard follows. Note that only 16% of
turn design specifications into testing specifications, write
people keep track of work items outside their team by read-
test harnesses, and conduct black box testing. Architects
ing the source code checkin messages, and very few (2%)
do long-range and product-wide planning, and user experi-
keep track of their work items on a public whiteboard.
ence engineers design user interfaces and conduct usability
studies. Respondents were offered a chance to win a single Notice, too, that program managers (PMs) use a greater
$250 gift certificate as incentive for completing the survey. diversity of tools to keep track of dependencies, almost
The survey questions were divided into three sections: 10% greater in many tools than developers and testers. The
demographics, details about how coordination occurs in biggest disparities are that PMs use Sharepoint websites
one’s team, and perceptions of how well coordination was twice as much as developers, and use Excel spreadsheets
practiced within one’s team and its dependencies. A Likert three times as much as developers. Developers use source
scale of “All of the time, Frequently, Occasionally, Rarely, code checkin messages 60% more often than PMs or testers.
Almost Never, N/A” was used. All of the survey questions Also illuminating are the ways to how engineers com-
reported on in this paper can be found in Appendix A. municate to unblock themselves from a critical dependency
(shown in Figure 4). When the dependency is within their
team, almost all (89% and 88%) respondents said they
3. Data and Results would send an email and pay a personal visit to the per-
son blocking their work. Instant messages and phone calls
We received 820 responses, of which 45 were invalid (we came in a distant second place (55% and 54%), followed by
removed duplicate and empty surveys), for an overall re- a setting up a one-time meeting (45%), escalation to their
sponse rate of 30.6% (775 / 2,535). In our sample of 775 re- manager (38%), and communicating via work item database
spondents, 39.2% are developers, 33.6% are testers, 20.3% (38%). When the dependency is outside their team, almost
are program managers, and the rest (6.9%) have other job all still use email (89%) to communicate, but paying a per-
roles. 76.5% of respondents were individual contributors; sonal visit drops to 48%. Instead, people use the phone
the rest (23.5%) were leads or managers. Respondents had (59%), escalation to a manager (52%) or one-time meetings
an average of 9.6 years (SD: 6.3) of experience as soft- (51%) as a substitute.
ware engineers, and spent 5.0 years (SD: 4.1) working at From the data in the chart, it appears that respondents
Microsoft. would use a point person to keep track of dependencies
We asked developers how they depended on other teams more for external rather than internal collaborations. In ad-
and how they coped when dependencies went awry. The dition, having a manager talk to another group’s manager is
 Dev PM Test
80.0%
70.0%
60.0%
50.0%
40.0%
30.0%
20.0%
10.0%
0.0%

Figure 2. The tools that engineers use to keep track of dependencies on other teams, divided by job
role. The ’other’ category is not shown for clarity of presentation.

Minimize code dependencies 63% anticipated and/or real problems with their dependencies.
Align product schedules 59% There were five strong responses (shown in Figure 3, three
Have a backup plan in order to ship without the
dependency
57% that minimize the dependency itself, and two that adjust the
Reprior ze a ected work items 53% project schedule. 63% of respondents would minimize all
Eliminate code dependencies 49%
code dependencies on other teams. 59% would align their
Switch to a new development methodology 12%
product’s schedule with their dependencies’ schedules in
Avoid unreasonable people 10%
order to ship only when their dependencies have finished
Eliminate a ected features 10%
their own work. This can be problematic if a dependency
Interact only with people I trust 9%
slips their schedule. 57% of respondents say that their team
Other 6%
makes sure to have a backup plan to ship their own product
Never tak cal dependencies 5%
without the problem dependency. 53% would reprioritize
Cancel project 5%
their work items, potentially slipping a feature or work item
Reorganize team 2%
to the next release. 49% would eliminate all code depen-
dencies, which could mean to “clone and own” (copy and
0% 10% 20% 30% 40% 50% 60% 70%
paste) the desired functionality from the other team’s code-
base. All of the other responses, including canceling the
Figure 3. How engineers mitigate (antici- project (5%) and reorganizing the team (2%) are much less
pated) problems with dependencies on other frequent.
teams.
Communication overhead and maintaining relationships
takes a big toll on coordination practices and effectiveness
(see Figure 5). Almost 50% of respondents say that they
much more common with external dependencies than inter-
need to proactively ask their colleagues for status frequently
nal ones (25% vs. 8%).
or all the time. Lack of communication causes problems
Notice also, that 98% of respondents depend on people as well for 23% of respondents who need frequently or all
outside their teams (only 2% report that they do not have the time find that work items they depend on have changed
any dependencies in Figure 4). without any notification. When communication does occur,
Since not all collaborations with other people and teams 25% of respondents say it is frequently or always difficult
go as planned, we asked participants how they mitigate to get a team they collaborate with to implement a change
 Personal Visit
Email
Instant Message
Phone
Set up a one- me m ng
Escalate to my manager or above
Work item database
Escalate to my point person in charge of tracking dependencies
Escalate to their manager or above
Escalate to their point person in charge of tracking dependencies
A nd their war room ng
One of my superiors talks to one of their superiors
Accept less than what I wanted
Find somebody else to do their work
Do not need to -- They do what I want when I ask
In mid on
Other
I rarely have dependencies on people
Give up

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Unblock dependency within your team Unblock dependency outside your team

Figure 4. How engineers communicate with one another when they need to unblock themselves from
a critical dependency.

they require. 48% of respondents find that must maintain that teams worry more about a feature shipping on a par-
constant contact with the team they depend on is the way ticular date than the details of how the functionality will be
to get what they want. This contributes greatly to over- implemented.
head in depending on other teams. It is also probable that The teams that provide libraries offer status as their main
much of this overhead is due to other team’s deprioritizing consumable, followed by features and schedules. Without
their dependent’s work items. Only 30% of respondents say being privy to the inner workings of other teams’ processes,
that their dependencies frequently or always tell them where status updates are one of the only ways for a team to manage
their needs fit into their dependencies’ priorities. This lack its dependency on another team’s software prior to the ship
of information often causes anxiety in teams that have many date. Will they finish in time? Will the requested features
of these dependencies. Even worse, if a team had a choice and work items be completed? The lack of this informa-
of teams to depend on, they would certainly choose the ones tion can make team leads anxious and unable to mitigate the
that place them number one on their priority list, rather than problems associated with teams that may not deliver what
a team that served too many masters. was promised. We see these problems in the perception data
A solution that many teams have found to work best for (see Figure 5) where pinging people for status occurs quite
them is to maintain personal connections with the people often. Respondents also noted that work items they de-
who work in the teams they depend on. 87.6% of respon- pended on were changed without any change notification.
dents agree with the statement “I feel that having personal We presume these were not changes to say that the work
connections with teams that I depend on is helpful to me.” items were done early or with more features than requested.
The change in status is one thing, but the lack of notification
4. Discussion of the change aggravates whatever anxiety the dependent
teams were already feeling about their dependency.
From the survey results in Figure 1, we can see that Another surprising result from the data (shown in Fig-
most respondents work on teams that consume software ure 2) is that the fourth most popular way to keep track
from others. This is consistent with the demographics of of dependencies is to use one’s memory. In cases where
Microsoft’s software teams. As a company, Microsoft cre- engineers have few and infrequent dependencies, this may
ates platforms and applications that depend on those plat- work just fine. We do not have data to support or refute this.
forms. There are necessarily fewer platforms than appli- We hope that when engineers must maintain more signifi-
cations. What was surprising from the survey responses is cant relationships between their team and others, they use
that most teams depend on other teams’ release dates and more robust forms of written material. The third most pop-
features more than other teams’ code or APIs. This implies ular tool is a status meeting, usually occurring face-to-face
 1. Work Items that I depend on are changed without my knowledge.

2. I need to ping the people I depend on for status.

3. It is di cult to get a team I depend on to implement a change I require.

4. I need to maintain constant contact with the team I depend on in order to get
what I want.
5. The teams I depend on tell me explicitly where my needs into their
priori s.

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

All of the me Frequently Occasionally Rarely Almost Never N/A

Figure 5. How engineers perceive the success of coordination communication.

or held via audio or video conferencing. This corroborates visit to someone whom one does not know, and interrupt
with the perception data (Figure 5), where respondents re- their work (which hopefully is unblocking the issue at the
port that in order to get what they want from another team, same time, but probably not) for something that may not be
they must maintain constant contact with them. Status, bug immediately important to the person. But from the other
triage, and similar kinds of meetings are held frequently by survey responses, we can see that attending another team’s
teams to manage their workflow. Attending another team’s status meeting is fairly accepted practice, and thus may be
meeting can give one great insight into what their priorities a good substitute for a personal visit. As can be observed
are and how they evolve over time. in Figure 4, almost all of the answers were chosen more
We present the data in Figure 2 split by job role to il- frequently by respondents with external dependencies than
lustrate the different distribution of answers. Program man- those with internal dependencies. Thus, the diversity of
agers (PMs) at Microsoft handle requirements, scheduling methods to unblock oneself is high, along with the asso-
and coordinate feature and work item completion with other ciated communication overhead required to avail oneself of
program managers, developers, and testers. Thus, it should them all. This communication overhead likely contributes
be unsurprising that they use tools to track dependencies to feelings that more work is required to manage external
simply more than developers or testers. However, PMs’ dependencies than internal ones.
greater dependence on tools is important for tool builders Finally, over half of the respondents report that their
who aim to help resolve coordination problems between teams have contigency plans to deal with unfulfilled depen-
teams. These tools will likely be most effective if they dencies, either delaying, replacing, or canceling requested
“live” in the same applications that program managers al- functionality. Only 5% report that their projects are can-
ready find themselves in. From the survey data, we can see celed for the lack of the dependency, thus, the projects re-
that this is often not in the code, itself. Thus, to most effec- ported about by these respondents appear modular enough
tively resolve team coordination issues with a tool, it is im- to adapt to unanticipated failures of collaboration between
portant for tool builders to note that non-programmer PMs teams. If, as reported in the perception data in Figure 5,
ought to be their target audience, rather than the developers more teams would tell their dependents explicitly where
who create the code or the testers who validate it. We can they stood in the team’s priorities, perhaps the numbers of
make this argument stronger by noting again that the main projects that are delivered with less functionality than de-
objects mediated in a dependency are features and product sired could be reduced by teams seeking out other less bur-
schedules, both items maintained by program managers, not dened teams to meet their needs.
by developers or testers.
From Figure 4, we can see that the way coordination 5. Threats to Validity
occurs within teams is very different than between teams.
Within teams, personal visits and email are the most pop-
ular ways to fix blocked dependencies. The parties know Our survey was conducted at Microsoft Corporation;
one another and are comfortable visiting in person to ex- while we imagine its results apply broadly to software de-
plain their issues and get them resolved. When dependen- velopers at other companies, studies at other sites would be
cies cross teams, however, the frequency of in-person visits useful to highlight behaviors which may be affected by Mi-
drops by almost half. It is likely quite awkward to pay a crosoft norms and culture.
6. Related Work [3] C. R. B. de Souza, D. Redmiles, and P. Dourish. ”break-
ing the code”, moving between private and public work
in collaborative software development. In Proceedings of
Coordination problems are inherent in any sort of dis- GROUP, pages 105–114, Sanibel Island, FL, 2003. ACM
tributed work situation. Many researchers have looked into Press.
coordination problems in software development organiza- [4] C. R. B. de Souza and D. F. Redmiles. An empirical study
tions and discovered difficulties caused by geographic dis- of software developers’ management of dependencies and
tance [7, 10], team size [5], lack of personal contact [8], changes. In ICSE ’08: Proceedings of the 30th international
lack of awareness [6], poor knowledge flow and commu- conference on Software engineering, pages 241–250, New
nication breakdowns [2], and architectural modularity [3]. York, NY, USA, 2008. ACM.
Kraut and Streeter [9] found that developers preferred to [5] J. Fred P. Brooks. The mythical man-month. In Proceedings
communicate frequently and informally to coordinate with of the international conference on Reliable software, page
193, New York, NY, 1975. ACM Press.
one another on schedules, bugs, tests and design reviews. [6] C. Gutwin, R. Penner, and K. Schneider. Group awareness in
de Souza and Redmiles [4] surveyed developers and cat- distributed software development. In Proceedings of CSCW,
alogued how they managed dependencies, both for poten- pages 72–81, Chicago, IL, 2004. ACM Press.
tial problems with coordination that they expected to expe- [7] J. D. Herbsleb and R. E. Grinter. Splitting the organiza-
rience and problems they expected to cause for others. tion and integrating the code: Conway’s law revisited. In
Proceedings of ICSE, pages 85–95. IEEE Computer Society
Press, 1999.
7. Conclusion [8] P. Hinds and C. McGrath. Structures that work: social struc-
ture, work structure and coordination ease in geographically
From these survey results, we can learn some lessons distributed teams. In Proceedings of CSCW, pages 343–352,
Banff, Alberta, Canada, 2006. ACM Press.
about how to improve intra- and inter-team coordination.
[9] R. E. Kraut and L. A. Streeter. Coordination in software
While most teams experience challenges in coordination, development. Communications of the ACM, 38(3):69–81,
the majority appear to get their tasks done, albeit with less 1995.
completed than they had hoped for. We saw that examining [10] A. Mockus, R. T. Fielding, and J. D. Herbsleb. Two
the needs and methods of engineers with different job roles case studies of open source software development: Apache
can help to focus process changers and tool builders on less and mozilla. ACM Transactions on Software Engineering
obvious audiences for their interventions, hopefully to ef- Methodology, 11(3):309–346, 2002.
fective results. Another takeaway message from our paper
is that creating and maintaining personal relationships be- Appendix A: Survey Questions
tween individuals on teams that coordinate is indicated by
many respondents as a good way to successfully collabo- 1. Which of the following items do you depend upon from other
rate with colleagues. However, while more communication teams? i.e. you need the following from them in order to
between teams can help improve coordination, it can also complete your task, or a change in the following with affect
increase process overhead. Creating tools that engender your task.
the right kinds of communication, such as automatic status (a) APIs
gathering and change notification could help people keep up (b) Code.
with their dependencies more efficiently. (c) Release schedule
For as long as specialization and collaboration has been (d) Bugs
around, many possible solutions to coordination problems (e) Features
(f) Prioritization of work items
have been tried and will continued to be invented. The data (g) Status
in this paper helps identify some of the potential target areas (h) Documentation
and audiences for such interventionary solutions. (i) Other
2. Which of the following items do other teams depend on you
References to provide? i.e., they need the following from you in order to
complete their task, or a change in the following will affect
their task.
[1] A. Begel. Effecting change: Coordination in large-scale
software development. In Workshop on Cooperative and (a) APIs
Human Aspects of Software Engineering, pages 17–20, (b) Code.
(c) Release schedule
Leipzig, Germany, May 2008. ACM. (d) Bugs
[2] B. Curtis, H. Krasner, and N. Iscoe. A field study of the (e) Features
software design process for large systems. Communications (f) Prioritization of work items
of the ACM, 31(11):1268–1287, 1988. (g) Status
 (h) Documentation (j) Escalate to my point person in charge of tracking de-
(i) Other pendencies
(k) Escalate to their point person in charge of tracking de-
3. How do you keep track of the items that you depend upon
pendencies
outside your team? (l) One of my superiors talks to one of their superiors
(a) Mental list (m) Intimidation (i.e. pay a visit and do not leave until you
(b) List on paper get what you want)
(c) List in text editor (n) Find someone else to do the work
(d) List on my personal whiteboard (o) Accept less than what I wanted
(e) List on a public whiteboard (p) Give up
(f) Excel spreadsheet (q) I rarely have dependencies on people within my team
(g) Source code checkin messages (r) Do not need to – They always do what I want when I
(h) Outlook tasks ask
(i) Work item database (s) Other
(j) Email
(k) Sharepoint website 6. What do you do in your own project to mitigate the effects
(l) Wiki (potential or actual) of depending on other teams?
(m) Status meetings
(n) Point person in charge of tracking dependencies (a) Minimize code dependencies
(o) The person I depend on reminds me (b) Eliminate code dependencies (e.g. “clone and own”)
(p) I do not keep track (c) Never take critical dependencies
(q) Other (d) Align product schedules
(e) Reprioritize affected work items
4. When you get blocked on a critical dependency within your (f) Avoid unreasonable people
team, in what ways do you communicate to unblock your- (g) Interact only with people I trust
self? (h) Cultivate in-person relationships (e.g. become
friendly, put a face to the name)
(a) Personal visit (i) Switch to a new development methodology (e.g.
(b) Email Scrum, Feature Crews)
(c) Phone (j) Have a backup plan in order to ship without the depen-
(d) Instant Message dency
(e) Work item database (k) Cancel the project
(f) Set up a one-time meeting (l) Reorganize the team
(g) Attend their bug triage meetings (m) Other
(h) Escalate to my manager or above
(i) Escalate to their manager or above 7. Likert-style questions. Answers: All of the time, Frequently,
(j) Escalate to my point person in charge of tracking de-
Occasionally, Rarely, Almost Never, N/A.
pendencies
(k) Escalate to their point person in charge of tracking de- (a) Work items that I depend on are changed without my
pendencies knowledge.
(l) One of my superiors talks to one of their superiors (b) I need to ping the people I depend on for status.
(m) Intimidation (i.e. pay a visit and do not leave until you (c) It is difficult to get a team I depend on to implement a
get what you want) change I require.
(n) Find someone else to do the work (d) I need to maintain constant contact with the team I de-
(o) Accept less than what I wanted pend on in order to get what I want.
(p) Give up (e) The teams I depend on tell me explicitly where my
(q) I rarely have dependencies on people within my team needs fit into their priorities.
(r) Do not need to – They always do what I want when I
ask
(s) Other

5. When you get blocked on a critical dependency outside your

team, in what ways do you communicate to unblock your-
self?

(a) Personal visit

(b) Email
(c) Phone
(d) Instant Message
(e) Work item database
(f) Set up a one-time meeting
(g) Attend their bug triage meetings
(h) Escalate to my manager or above
(i) Escalate to their manager or above

View publication stats