I2182: Innovating Digital Systems and Services - Chapter 5

What are the five values of XP?

Communication (the people involved in the project effectively pass information to each other); Simplicity (follow the KISS principle by focusing on today's needs, not tomorrow's); Feedback (get user feedback early and often); Courage (own up to mistakes, trust co-workers, throwaway code that doesn't work); and Respect (discuss problems openly but nicely and seek to understand others)

DevOps

DevOps refers to the merger of concerns between IT development and Operations (maintenance and follow-up of products and services) to support faster innovation and better quality applications. Among other benefits, bringing people from Operations early on can help Development lower the failure rates when the product is in Operations and has been handed-off to them.

Making Organisations Agile

Having been introduced to agile methods it is important to reflect upon the consequence for such rapid and flexible system development for the organisation as a whole and the people within it. Especially as we consider systems as socio-technical systems it is important to keep in mind that change that is initiated through digital innovation often requires organisational change. Such change can vary in effort and complexity, e.g. it may involve the refocusing of the business's marketing strategy, the move to entirely new working practices and/or structures, the retraining of staff etc. It is argued that such 'people change' is not always easy and frequently lacks behind the technological innovation (Lashford 2016). Although the study of organisational change goes beyond the subject matter of this course, it is important that you are aware of the complexity behind it (and the amount of research carried out). Also it is important that you recognise that agile approaches are not limited to the development of IT systems. With an increasing need for flexibility and speed within organisations the idea of agility has also been adopted for aspects of organisational management and structure (McKinsey Quarterly 2017).

Differentiating XP and Scrum

Scrum and Extreme Programming (XP) are definitely very aligned. In fact, if you walked in on a team doing one of these processes you might have hard time quickly deciding whether you had walked in on a Scrum team or an XP team. The differences are often quite subtle, but they are important. I think there are four main differences between Scrum and XP: Scrum teams typically work in iterations (called sprints) that are from two weeks to one month long. XP teams typically work in iterations that are one or two weeks long. Scrum teams do not allow changes into their sprints. Once the sprint planning meeting is completed and a commitment made to delivering a set of product backlog items, that set of items remains unchanged through the end of the sprint. XP teams are much more amenable to change within their iterations. As long as the team hasn't started work on a particular feature, a new feature of equivalent size can be swapped into the XP team's iteration in exchange for the unstarted feature. Extreme Programming teams work in a strict priority order. Features to be developed are prioritized by the customer (Scrum's Product Owner) and the team is required to work on them in that order. By contrast, the Scrum product owner prioritizes the product backlog but the team determines the sequence in which they will develop the backlog items. I've never seen a Scrum team not choose to work on the highest-priority item. And a Scrum team will very likely choose to work on the second most important. However, at some point one of the high priority items may not be a good fit for the sprint being planned—maybe a key person who should work on it will be swamped by work on higher priority items. Or maybe it makes sense to work on a slightly lower priority item (let's say #10 on the product backlog instead of #6) because the team will be working in the code where #10 would be implemented. Scrum doesn't prescribe any engineering practices; XP does. I love the XP engineering practices, particularly things like test-driven development, the focus on automated testing, pair programming, simple design, refactoring, and so on. However, I think it's a mistake to say to the team "you're self-organizing, we trust you, but you must do these specific engineering practices...." This sends a mixed message to the team that causes confusion. I love the XP practices but don't like mandating them. I want teams to discover the value on their own. These are small and often subtle differences between Scrum and XP. However, they can have a profound impact on the team. My typical advice to teams is "start with Scrum and then invent your own version of XP." The XP practices are wonderful but they work best and teams commit to them the most stridently if they discover them themselves rather than having them mandated. I help teams do this in my coaching by asking questions like, "Would this bug have happened if we'd been doing test-driven development?" and "Would we have made that mistake if we were pairing?" I find true XP to be a small target off in the distance. If a team can aim at that and hit the bull's eye, wonderful. If not, however, they are likely hacking (e.g., refactoring without any automated testing or TDD). Scrum is a big bull's eye that on its own brings big improvements simply through the additional focus and the time boxed iterations. That's a good starting point for then adding the XP practices.

Agile challenges

There are a number of challenges to the adoption of Agile methods such as the necessary skills to implement them, difficulties when drafting contracts because of Agile's preference of direct communication with customers, and coordination of large teams and projects.

The practices of XP

This section considers the main practices that an XP team will use when developing software. You might come across more comprehensive lists when reading some of the vast literature on XP, but the practices presented here give you a solid foundation for how XP works. Note that the list below is a set of independent practices, and a development team will normally select those that are most suited to, and supportive of, their particular circumstances. Incremental development: iteration and release planning Following the main Agile Principle of frequent software releases to customers XP implements iteration by proposing short development cycles of 2 weeks. At the end of a cycle the system is demonstrated to the stakeholders for feedback (Martin 2014). A release plan is drawn up to map out the approximately six iterations ahead that are expected to be covered within the three months ahead. The plan normally consists of a list of functionalities (user stories - described below) that have been prioritised by the customer. The resulting release then constitutes a major deliverable that goes out to the customer for use and feedback. User stories A User story is a short description of a piece of functionality that the customer may request or need from the system under development, for example, 'Provide a button which calculates the total cost', or 'Allow 10 users to access the phonebook simultaneously'. As soon as these stories are written the development team should estimate the time it will take to complete them. This estimate can then be used to plan how to develop the software for the story, its cost- and to discuss with a customer whether they really want that feature at that price. Have a look at this article on creating user stories. http://www.alexandercowan.com/best-agile-user-story/ Pair programming Within XP all programming for the final software is supposed to be written in pairs - that is two people sitting around a single computer. The idea is that they will discuss their work, check each other's ideas, keep each other focused on the task, brainstorm new ideas, clarify ideas, and lower frustration by supporting the partner when they are stuck in a problem. Refactoring Code is subjected to constant changes, because new features are added and bugs are corrected. Such constant alterations lead code to degrade. XP suggests that such degeneration should be reverted through frequent refactoring (Martin 2014). Refactoring is 'the process of changing a software system in such a way that it does not alter the external behaviour of the code, yet improves its internal structure' (Martin Fowler). Refactoring often involves re-writing large chunks of code. This is important, because when the system is rapidly changing (as it is likely to with XP), refactoring ensures the software code does not get out of control. It also allows us to think about reuse of previous components, or perhaps looking at alternative ways of doing things. Refactoring means that we clean up the code so that it still does exactly the same thing, but does it in a more efficient and easy to understand way. Such improved code quality comes with code that is easier to understand (increased simplicity) and hence ensures that future necessary changes can be implement more easily. Unfortunately refactoring is often frightening to do as it involves breaking software which works (Martin Fowler, http://www.refactoring.com) Test-first development XP focuses heavily on testing as a central part of development (and you will learn more about software testing later in this guide). Radically, XP does not start with producing a piece of code and then writing a test to work out whether the code operates correctly: instead it reverses the order. It says that we should start by writing a test case which will test whether the task is successfully achieved by the code (which we have not yet started to write). By writing the tests first it makes it harder for programmers to get carried away and write more than they are meant to. Furthermore, having write code that aims at getting a test case to pass, encourages the creation of self-contained units of code - the units are decoupled from each other - which is good programming practice (more on this later in the guide). It also means that everyone can see whether a piece of code is ready, because otherwise it will fail the test, and it keeps everyone on track, because a test case provides an aim. Finally, the tests also support the refactoring process (introduced above). If some code has been refactored, the tests can be used to check whether the functionality has remained the same (Martin 2014). Customer Team member Rather than simply thinking about the software development team as a group of people to develop the software, we need to consider the mix of people within this team in detail. Crucially, the whole team must include 'customers', (i.e. representatives of the end-users of the system) who take responsibility for introducing the requirements for the team to implement. Paced work in an open workspace XP suggests that people who sit together a lot work better (this seems pretty obvious). As such it advocates organising a project's workspace such that people can sit and work together. Also, the workplace needs to be supportive of the work. This usually relates to the use of a wall to put up charts, posters or progress, pin boards for the story cards, tea and coffee. XP also normally does allow developers to work overtime. Working overtime is considered to be wrong and often leads to mistakes and lower productivity. Ten-minute build Building the whole system and running all the tests should take less than 10 minutes. This means that the source code needs to be automatically compiled into working software, and then have all the test cases run against it within 10 minutes. Why? Because if it takes longer than this to build the system developers are unlikely to do it as often, and this will mean it is not tested as often. Continuous integration Hand in hand with the need to build the system every 10 minutes is the need to integrate and test changes to the system every couple of hours. Rather than dividing the problem into sections, allocating them to people and asking them to go away and work on them, XP focuses on the pair of programmers integrating their changes into the system often, rather than at the end of the day. Source code management tools, such as a version control system, are used to keep track of the versions of code integrated by multiple team members.

Scrum

Like XP and other agile methods Scrum aligns with the Agile Manifesto and its underlying principles. However, it is often argued that whilst XP offers a tool set for the technical aspects of agile development, Scrum concentrates more on the project management aspects (Ashmore and Runyan 2015, Stephens 2015). As such, Scrum can be summarised as follows (see figure above): The product owner, who represents the customers, user and other stakeholders within the project, creates a product backlog. The product backlog consists of a prioritised list of backlog items (PBIs) represented as user stories (see XP practices above) that represent the project's goals. The development team then determines a manageable chunk from the top of the backlog (called sprint backlog) and plans how these user stories can be divided into tasks and implemented by the team. The idea of iteration is then realised by dividing the overall project into sprints. A sprint it limited to a time box of up to one month (typically 2 - 4 weeks) during which a sprint backlog must be implemented and tested. The concept of the sprint is supposed to ensure that the team works on tasks that are attainable and manageable, and daily meetings (known as daily scrums of 10-15 min) are supposed to provide additional focus and an opportunity for feedback. The end product of a sprint should be a potentially shippable (or releasable) increment (PSI). A central role within Scrum development is the role of the scrum master whose job it is to ensure that the team remains on track. A sprint ends with a sprint review meeting the sprint retrospective. The sprint review is concerned with the evaluation of the product. It allows the product owner to validate the deliverables against the original sprint backlog. The sprint retrospective reviews the development process. Here the sprint master and the development team identify problems and areas of improvement for the next sprint. The examples given above of the most significant agile methods should not only give you an idea of how agile methods work. More importantly they should confirm that agile development is not a single self-contained development model. Instead it offers principles and guidelines that aim to improve and enrich system development, so it is more adaptable to the demands of changing environmental factors. Indeed many agile development projects might borrow aspects from a number of methods.

Extreme Programming

The most significant attempt to create an agile method was probably the development of extreme programming (or XP) (Beck 1999). XP was developed for small teams who are confronted with vague and changing requirements. It is not considered to be appropriate for large projects. XP very much incorporates the idea of dividing development jobs between different members of the team, but at the same time being able to integrate all components quickly enabling several releases of the same system on one day. The central ideas that are associated with XP are communication, simplicity, feedback and courage. You will notice that these very much focus on the social aspects of the development process, i.e. on the way people work rather than on how best to model complex systems: Communication. XP suggests that a project must ensure that the people who need to know things about the project do know them. Simplicity. XP focuses on eliminating wasted complexity. Doing more than the minimum is often a waste of time, may lead to unnecessary complexity and degraded system quality. Feedback. Change is inevitable, and so feedback is necessary so that developers understand the changes. XP tries to ensure that the development team gets as much feedback from users/customers as possible, as easily as possible. Courage. XP highlights the need to have courage to foster the ideas above. Courage to admit to mistakes fosters good communication. Courage to throw away work that is failing may support simplicity, and courage to seek out answers will fosters good feedback. Respect. Within XP respect is viewed an important basis for successful teamwork that relies on discussion of problems. Also, mutual respect is obviously a good basis for courage.

Agile methods

The previous chapter has given you an insight into the motivation, the values and the principles that form the core of agile development. You will have noticed, however, that these values and principles do not come as clear instructions for developers who want to adopt agile development practices. Agile development does not come with a neat self-contained set of instructions, tools and steps unlike, for example, the waterfall model. Instead agile development is a more general term used that encompasses all iterative approaches to software development that embrace the values stipulated in the Agile Manifesto (Ashmore and Runyan 2015). However, the desire for agile development has led to the proposal of various agile methods that support the practicalities of the actual development process. Although these methods differ in detail they all agree that software should be developed incrementally, and they all follow the philosophy of the Agile Manifesto and Principles. The most prominent methods include: XP SCRUM lean software development feature-driven development dynamic system development Crystal Kanban (see chapter 6) In this chapter I will present the basic practices and ideas underlying XP and Scrum, because knowledge of these two agile methods will give you a good foundation to understand what agile development is all about and to grasp the numerous remaining agile methods out there. Also, XP serves as a good introduction to a range agile practices that are widely used in many agile methods today. Scrum is a good example of how agile development may be managed. You should also have some understanding of some of the remaining main agile methods. However, I will not describe these here. You will have to explore them yourself. The activity at the end of this chapter will give you guidance on how to do this.

Cloud computing and agility in organisations

You have been introduced to cloud computing in the course IS1181 Digital infrastructures for business. Let us briefly look at cloud computing in the context of agile development. You will recall that PaaS (Platform as a service) is a cloud computing idea that allows companies to run their development kits and database management tools on a host machine. This allows developers to then build and run software applications by using virtual resources provided by third-party sellers. Also, PaaS allows for the outsourcing of capabilities such as the construction and maintenance of a company's database and aspects of its security management. Furthermore, PaaS can support the need for fast expansion (and possible reduction) of IT capabilities (scalability). Popular PaaS providers include: Microsoft Azure, Force.com, Heroku Enterprise. For example, those developing Azure solutions allow Microsoft to worry about the hardware, storage, databases etc. in the cloud - they simply write basic software which uses the platform Microsoft provides. This means it is much easier to continue to develop their code - while Azure handles the complexity of keeping the system running for users. It becomes clear that cloud computing can hence also be viewed as a response to the need for agility (Sussna 2013). The need for speed and flexibility are both more easily addressed if (part of the) software is acquired as a service via the cloud. By taking advantage of having cloud services, companies can avoid having to invest time and effort into regular software jobs and maintenance. For example, jobs like database management operating system servicing and extended hardware acquisition can be outsourced. This is useful in times where development teams are expected to be more 'agile', i.e. to deliver frequent software releases in reaction to ever changing customer demands. Cloud computing now helps companies to concentrate on the optimisation of business processes. In short, cloud services can make the IT infrastructure more flexible thereby increasing the agility within today's organisations. In addition, because cloud computing allows access to 'virtual computers' quickly and easily, it is very easy for a development team to test out ideas using a new computer. For example, the developers might create a set of these virtual servers for testing, for development, and for running the real application (often called production). They might then create a new set of development 'virtual computers' just for a couple of days when they want to demonstrate their solution to some people (deleting them after the demo). They might create virtual computers to try out different ideas separate from the main development activity.