Docs

There, I said it. A four letter swear word. Something worse than the F’ word if the horror on the boss’ face is anything to go by.

We don’t do “documentation” anymore and besides, the agile manifesto says it’s immoral to write a word of documentation. The code is the documentation. That you have to get inside the twisted maze of my mind, and work out what drug infused insanity I was trying to convey at the time and which may or may not result in what was intended regardless… that’s your problem. 

Such nonsense pervades software development today. Although…

It’s not the lack of documentation per se but the lack of demonstrable thought that bugs me. A critical explanation of why things are the way they are.

Why has solution design been so poorly treated?

The answer to that lies in what design is trying to achieve.

Solution design aims to address the needs of the system, communicates how these needs are met and enables testing of the solution when the cost of change is lowest.

This testing is achieved by static walkthroughs and peer review.

However, in an agile world where we breakdown features and stories into small chunks that can be delivered rapidly and iteratively, the hefty tombs of yesterdays solution design documents are equally shrunk to focus on the few features and stories in scope.

Taken far enough it ends in a combined whiteboard design and review session with a handful of developers. And beyond taking a photo and circulating to the team, what's the point in doing anything more?

Firstly there's the supporting teams who are going to care for your solution through its early teething days, rebellious teenage years, into maturity and all the way to the grave – death being one of the few certainties in life. Making these guys bump their way around a darkened room trying to figure out how things hang together is unfair if not sadistic.

These guys don't work in scrum teams or story by story. They work from incident to incident, from shit-storm to fan-splattering shit-storm. They need a concise and holistic view of the solution for which a few poorly framed holiday photos doesn't cut it. Honestly, these guys are too nice to you.

Then there's the not insignificant matter of consistency. 

You can argue the marginal benefits of any technology but you need to justify any change which runs counter to the inherent design patterns of a system as contributing significant value to overcome the increased complexity and costs it brings.

Do things consistently and you get paybacks in terms of proven reliability, easier maintainability, reduced cognitive load, faster delivery and so on.

Do things inconsistently and whilst you may get to play with the latest  tech you're being pretty selfish - if that's your motivation. And if it's not then see above re justifying the increased complexity and costs.

Which brings me to my point on solution design.

Solution design is about more than addressing the needs of a few stories on a sprint by sprint basis. It's about addressing the broad needs of the system and providing a vision for the longer term. Defining the patterns which will be used time and again and which (should) enable that payback through consistency.

This isn't an argument that technological progress isn't a good thing or that we should never have crawled out of the sea. Or that your resident architect has all the answers. It's an argument to think about the broad needs, key decisions, responsibilities, patterns, principles, policies and costs - immediate and long term - that provide the foundation on which any system is based.

At times it may seem futile and time consuming but it's cheap to change this stuff early on rather than late on when it's most expensive.

And the way we do that best is through reasoned discourse. Discourse best expressed through quality documentation - words, diagrams and matrices.

The Con of Agile (or why agile reductionism is hard…)

Agile is, to a large extent, a radical breakdown of function into small incremental features delivered in a prioritised manner with rapid feedback to inform on the next evolutionary step to deliver ever greater customer value.

Executed as such you may not; probably should not, get what you originally intended and even if you fail then at least you’ve failed fast and saved yourself an expensive pipedream.

Bliss.

But agility is not easy. It is not something a tool, method or consultant will magically fix for you – despite what they tell you. Radical breakdown of function is hard. Reductionism is hard.

Most systems we build today are inherently complex with dependencies spread far and wide through the enterprise. Through reductionism we attempt to decompose these complex systems into their simpler component parts – a technique that has a long and successful history in software development – only to then recompose them in many and varied ways to deliver the outcomes we desire.

In a typical large organisation it’s not uncommon for a system to require integration with dozens of others and for any particular new feature to impact several of these at a time – in fact it’s rarer for a feature to be self contained within one system. Integration is the norm.

And here our problems start… 

According to Conways law an organisation will typically design systems that mirror the organisations communication structure. i.e. we end up with various systems mirroring the group and team structure of the organisation integrated along the same communication channels as the organisation.

The net result is that we have specialist teams with a deep understanding not just of the technology but the values and ethos of each system and the organisation they reflect. These teams own the system and have veto rights – rightly so – over what functionality they do or do not support. 

So on one hand we have a prioritised backlog of customer focused features and on the other we have a collection of disparate teams available to deliver them hell bent on aligning to the internal organisation structure. Something the customer usually doesn’t (and shouldn’t) give a toss about.

An instinctive reaction is for teams to start to breakdown features into stories that make sense from the organisational structure perspective. This can create a reflection of the organisation in the user journey and leads to stories which are really tasks because that’s easiest for teams to consume. And if you’re simply working off a prioritised list of tasks you’re not agile. You may be able to re-prioritise but you’ve lost the connection with why you’re doing something and that critical feedback loop from the customer so you can’t respond to their changing needs.

Perhaps worse still, a disparate collection of tasks spread across a plethora of teams does not provide a clear vision and direction for the people working in those teams. Treated this way people can start to feel like slaves to the machine, churning out widgets hour after hour with no clear understanding of why. Motivation and quality suffers and we no longer have small agile teams focused on delivering customer value but a collection of teams each with their own perspective on what we’re trying to achieve. 

As we breakdown features and stories into smaller chunks, maintaining a focus on customer value and the overall vision is hard – particularly in a large and complex organisation.

There’s now a twofold need both for well-articulated, customer focused and concise feature and story definition, and for solution designs which clearly stitches together the various parts of the organisation; and associated systems, connected to deliver those features.

The features and stories say what we’re trying to do and why we’re trying to do it.

The solution design says how we’re going to do it.

And that understanding of why we’re doing something is the “Ah-ha!” moment that gives us meaning and purpose, the reason we get out of bed in the morning. 

Ultimately it’s easy to explain what we’re trying to do and how we plan to do it – they’re concrete things we can action and it’s natural for us poor humans to want to focus on solving problems. It’s much harder to articulate why we’re trying to do something, and we all too often forget that others aren’t aware of the vision in the first place.

So as we break down capabilities and features into ever smaller chunks we need to consciously focus on the “why” we’re building this chunk and less so the “what” and the “how”. It may seem counter intuitive but solving problems is easy, finding them is harder.

As a consequence our features and stories should be heavy on the explanation of why we’re trying to do something and lighter on what we’re actually going to do. The “how” we’re doing something should be reserved for the solution design and provides traceability from the “what” and the “why” to the organisational structure, providing the foundation for that task list everyone seems to desire.

As we breakdown stories we also tend to reduce the number of teams involved and if we go far enough that they only impact one team then great. It won’t though always be the case and we shouldn’t try to twist stories to meet the internal organisation structure and make it fit us rather than the customer.

In short, as we breakdown features and stories we need to focus on why we’re doing something and what we’re going to do to address that need. Don’t worry about how we’re doing it – leave that to the technical team of architects and engineers – give them the easy task of connecting the dots and solving the problem. 

The trap in agile reductionism is that we, as human beings, have a tendency to want to solve the problem and in doing so lose the connection to the why that gives us meaning and purpose. We end up focusing on the solution and thereby reduce the solution to a problem, not the problem itself. The two are not the same. One is optimised for the organisation, the other for the customer.