Elizabeth Hendrickson keynoting on eXtreme programming at scale, XP2016 Edinburgh. Elizabeth works as VP R&D for Cloud foundry development at Pivotal.
Some numbers: 59 teams, 251 engineers 7 locations and a bunch of timezones.
eXtreme Programming is supported by the management team, because Rob Mee, one of the founders, was an early adopter, he reviewed the first edition of eXtreme Programming explained before it came out.
Underlying principles
Breakfast and lunch together (I wonder how that works for people with kids, I like having some breakfast at home).
Practices
Mostly the 12 original xp practices. works great for a single team. What about more teams. More information about the transition from the VMWare official process to xp in the pub ;-).
Tragedy of the commons: when a whole bunch of teams have a whole bunch of repos, and nobody has an incentive to fix any of it.
“without intervention, work rolls down hill”. (picture of someone being chased down the hill by a big rock).
If XP is getting such good results, why is not everybody doing it? Because it requires fortitude. Every day we have to make decisions on how to deal with a thing. Are we going to decide to adopt a team branching policy, or are we going to take the pain and do continuous delivery with a massive number of teams. At the time it was just four, it takes fortitude to say we don’t want inventory of unshipped features. A feature does not count until it is in the hands of the customer.
“Why are you having two people doing the work of one” never gots asked (on pair programming). The cliffhanger:
We have 59 teams. Now what?
We have 59 teams of work rolling down hill. Underlying organizational principles:
- Use Conway’s law to our advantage. Your code will resemble your organisation. Arrange teams according to desired architecture.
- Teams own things. No feature teams. Teams own their destiny.
- The team that owns the thing tests the thing. It is your responsibility. Nobody else is responsible for this. Not out of a sense of purity, but because it is also the team that has to fix the bugs they produce.
- Relentlessly improve feedback cycles.
- Increase empathy. “It would be very easy to create a blaming culture”. It is painful to be in the field using something that works only in theory. Empathy is our strategic advantage.Rob Mee would be the interview screen. Do new hires have empathy (amongst other skills). After working with three or four teams it is really hard to say “I don’t care about the people in that team”. We don’t hire particularly for teams. We do hire specialists, but optimize for happiness of the individual, team and organization at the portfolio management level.
- Collaborate, not hand off (“with”, not “for”). Cultivate empathy by cross-team pairing (a little harder with field engineers). Sending on an ambassador. We never do just a pure hand-off.
“At Cloud Foundry I became the queen of awkward conversations”.
“External QA organizations don’t work”.
“I recently realised my Dilbert Peak. I got e-mails complimenting me on my e-mails.”
Reflecting and adapting without some mechanism to see whether you actually improved something or made it worse.
Internal tools
Visibility
Getting visibility across teams was absolutely crucial. There is a lot you can achieve you can achieve with color coding. Information all in one place, even if we have to write our own tools.
Build pipelines
Jenkins didn’t do pipelines as they see it, so they wrote their own, (Concourse)[https://blog.pivotal.io/pivotal-cloud-foundry/products/continuous-deployment-from-github-to-pws-via-concourse].
New leadership roles
It is a mesh, not a hierarchy.
- Engineering director / Leadership Liaison (“LL”). “The high availability cloud of engineering directors”. No little fiefdoms with you can’t rotate people from my team to your team. (I sometimes find this kind of job lonely, this seems like a great way to have more fun for the engineering directors to have more fun and less stress). This gives them more time to do the things they are good at besides being leaders (e.g. API design, testing). We have to let go of our egos. (“Egoless Management” ?). We knew what culture we wanted to grow. Every director started with a keyboard in their hands, not hired off the street. Hands on the keyboard, demonstrate leadership.
- Product Directors.
- Program Manager. Programme management that is in alignment with our agile values. Human centric. No release trains etc. Tweaking, adapting, reflecting, facilitating. Time and Scope trade-offs, no quality trade-offs or deathmarches. Takes care of a lot of the feedback cycle work.
“There is no failure, there is only learning”.
Every single day we have to make a choice, we have to take things forward. So in a couple of years this is going to be completely different.
Q&A
“You only get so many ‘I told you so’s’ a year”. We have disagreements, it just never got to this. This is a problem we need to solve, so what are we going to do about it?. Because we have the same values, we can come up with an answer. As long as you don’t always have to be right, you can get close enough to a consensus.
“Q: do you drive inspect and adapt by measuring and data, or gut feel?”. We measure build cycle time e.g. that we can get data on, others are hard to get data on. If we can experiment towards something that is measurable. We can also look at qualitative things over time.
Subscribe to our RSS feed