Our thriving Integration Architecture team have interesting design challenges when working across multiple client sites. In this blog, Barry Phease, one of our highly skilled Integration Architects compares the advantages of informal and formal design and why companies should consider both options dependant on their overall integration needs.
Every one of the IntegrationWorks clients have different requirements in regard to design. Some of our clients want signed-off and detailed design documents before even thinking about development and others are satisfied with “as-built” when required documents. Some even think this is an overkill and prefer a fluid development and architectural phase that flows throughout the entire development phase. So who is right?
Why design?
Everything needs to be designed even if it is just made up on the fly by the developer(s). But how it is designed, whether methodically or fairly liberally is based on the needs of the client and the general outcomes required.
Design is a conversation between the various stakeholders in the project – both present and future. It may take the form of a formal design document, or it may be 3 people standing around a whiteboard. The important point is that they all finish with a shared understanding of what is to be built and why. The needs of current participants are easily satisfied, but what about future needs? With any format of design, a high regard needs to be placed on the future capability of the project and how it can adapt to digital transformation. Some prominent questions can be; does the design collateral help the people that need to maintain/extend the software? What happens if someone leaves and takes their design understanding with them?
Advantages of Formal Design Documents
In formal engagements a signed-off design document forms the basis of a contract between the project sponsor and the developers. The delivered software is tested against the requirements, but code-reviews are matched to the design. The design can be reviewed in advance to see that the planned approach meets the needs of the project and fits the infrastructure before the expense of building is incurred, and this can avoid costly rework.
Where a large number of different parties are involved (business, infrastructure, networks, application developers, testers etc) a formal design document gives everyone a clear picture of what their tasks are and how they are matched to the needs of the other parties. It allows a project manager to allocate work and check progress, while staying in scope and not blowing out a budget.
For enterprise requirements (e.g. security, performance, non-interference with other projects etc) a design document gives peripheral actors a chance to have their input. This is important where strong governance processes are in play and there is an increased need for control and quality assurance. Everyone can have confidence that their needs have been considered before signing off.
The design document provides a place to present the design decisions that were made. It is important for people who come later to understand the reasons for choosing various options. This avoids relitigating decisions and also allows a re-examination of the options when conditions change.
Advantages of Informal Design
In smaller projects a design document may not be needed. A daily standup can provide a forum to discuss any design decisions required that allows a fluid conversation to raise any issues in development and understand development priorities. The outcome should be recorded in the project backlog for future reference, but this is the only main administrative duties required in this type of design. Comments in the code and/or scripts should be used liberally to allow future participants (or current ones with non-exceptional memories) to understand why things are done and helps to create a reference point on when decisions were made by whom and when. This can provide the needs of documented design without the overheads or costly time structures.
The benefits of this include speed and time to market – a decision can be made in the morning and incorporated in code, checked in, built and tested all the same day. This way the documentation is always up to date and working in a more agile and fluid way. The documentation is always available as it is saved in the CMDB with the source and becomes mutually exclusive. Depending on the tooling used the documentation, can often be extracted (e.g. javadoc) and made available at the end of a project without the expense of developing as-built documents.
Where should design be stored?
Often organisations have formal document management or record keeping systems for design documents. It is often difficult to find the most up-to-date version and often even the most up-to-date version is not updated during the project so doesn’t reflect what was actually built. This can be a problem for later projects as they have to undertake an expensive discovery phase to find out what the real state is. Mature organisations can handle this, and it may be correct for them.
In general the documentation should be stored in the CMDB along with the code. The same versioning/branching/tagging processes that make source code management easy, ensure that the documentation matches the deployed artefacts. However, they don’t ensure that it is up-to-date. A design document should always be supplemented by access to code and deployed artefacts.
A good project-close-out process will have a supplementary document describing what was actually built which can be presented as an appendix to the design document describing variations and reasons for them. This can also be a separate as-built document. At the least it should be release notes with a manifest of deployed artefacts along with where to find the (commented & versioned) code/configuration items.
Design for Integration
Integration projects have particular needs and there are always several different products involved, multiple architectural procedures and development processes. A clearly defined interface is almost always required up-front to ensure quick coding and smooth transition.
If new integration applications are required then the project looks like any software project with infrastructure, software, networking requirements and the organisation’s normal design processes are used.
For service development, the design needs to articulate the different parties involved in the service and how they interact. Parts of the design (e.g orchestration, mediation) are often dependent on the product set used as they will normally have graphical design tools that match design to implementation. The interface and data design will usually use some sort of entity design tool like Sparx EA.
It is useful to have a catalogue of services in use and how they interact. This can be a spreadsheet, but often the linkages are more involved and a 2 dimensional table cannot easily represent them. A graph database may be an option to capture all the dependencies.
Conclusion
Each project, customer or site will have a different answer on how they document their design. Generally speaking though, there is always a requirement for a design, whether it be formal or informal – as it serves as a guiding light and risk mitigation for costly mistakes and time. With that in mind, sometimes it is only after the project is finished (and the next one underway) that you know whether your design strategy has been successful.
One thing is always true. Design is never “finished”. Even in a waterfall process with signed-off designs before coding starts there will always be further design decisions to be made as the project proceeds – the design process should recognise this and make sure that whatever is produced is still of value at the end of the project