Waterfall to Agile Challenges

Waterfall to Agile challenges

Agile development methodology is aimed at maximizing team effectiveness, minimizing waste, and facilitating the continuous development and integration of software. Agile is a proven methodology for the efficient development of software in complex projects and is predominantly utilized by the software companies themselves. However, the reality for the transportation industry is that migrating from Waterfall to Agile is not straightforward. The vast array of Waterfall to Agile challenges require an organization-level journey that includes all aspects of development including team management, project management, processes, and the toolchain.

Table of Contents

• Comparing the software and transportation realms
• Advantages seen in Agile Development
• Challenges with Agile
• A Summary of Key Agile Enablers
• Summary

Comparing the software and transportation realms

One of the reasons Agile works better for pure software companies is because their hardware platforms are mostly fixed and most of their development work involves the creation and refinement of the software itself. In comparison, typically the transportation industry concurrently has hardware and software-dependent development underway, while conducting Verification and Validation (V&V) tasks that also must be completed. As a result, shorter continuous development cycles are much more difficult to plan and execute in the transportation realm. And V&V occurs multiple times over the life span of a piece of software, typically culminating in a need for road testing each time before an update can be released to the world.

LHP experts have experience helping businesses in the transportation industry navigate the complexities of their journey from traditional Waterfall methodology to Agile methodology and have helped them resolve many of the challenges encountered along the way. This article highlights the key advantages to turning to Agile as well as clarifying key challenges encountered by most organizations in the transportation industry.

Common issues with Waterfall

With traditional Waterfall development, some of the common issues that organizations must deal with include:

• Requirements that are not well defined.
• Difficulty with software development, workload planning, priorities, and estimating costs for software elements; this is a common theme in feedback received from various organizations.
• Keeping deadlines and tracking progress is challenging.
• It is difficult to find and resolve issues up front.
• Organizations have difficulty developing configurable products, e.g. where the software process needs to accommodate an array of different product lines.

Advantages seen in Agile Development 

In comparison to Waterfall, Agile development is truly a paradigm shift in the transportation industry. It is a completely unique method of development where both organizational changes and a value-oriented mindset are needed to be successful. However, it is not unusual for an organization to find it difficult to adapt, and there is typically resistance from various levels within the company. 

Despite any initial resistance, pushing through to successful adoption is worth it. The advantages include:

• Early and continuous delivery of software.
• Well-suited and effective for dealing with the complexity often found in new products involving complex software development combined with vague requirements and an ever-evolving methodology and toolchain.
• More capable of effectively dealing with changing requirements.
• The measure of progress is working software.
• Greater team efficiency: The team gets to tune itself, learn, and adjust at regular intervals.
• Developers are more accountable, and the work is more visible.
• Communication is forced; one cannot easily blame others for not doing work.
• Sponsors, developers, and users are a lot more involved in planning.
• Developers have more input into the ‘stories’ and what they plan to do.
• Increases the ability of the team to plan the software workload and fully utilize the available resources to address gaps.
• Early Verification: Allows for early system verification providing competitive advantage.
• Feature-based software development approach:
  • The team structure aligns with product features, increasing focus and efficiency.
  • Better suited for configurable products.
  • Facilitates maintaining a library of robust features and software components.
  • Allows companies to choose from an overall feature set to produce a composition.

Despite these advantages listed above, moving towards Agile methodology can be both disruptive and costly (overhead), and requires significant skill development and training. It really is an investment that could shift the organization to a much higher level of efficiency for software development, but it requires careful planning, a strategy roadmap, actively fostering an Agile culture, and selecting a suitable development toolchain and processes for the organization. 

Challenges with Agile

In our experience, some of the key challenges in the journey to Agile are:  

Agile/Scrum Culture

Scrum is a specific framework withing the Agile methodology. It uses all the core principles of Agile and defines specific methods, practices, and structure.

Agile is based on a culture with the values of Courage, Focus, Commitment, Respect and Openness. These are the five pillars of Agile adopted by the Scrum Teams.

These values are supposed to be manifested and promoted throughout all Scrum events, roles, and artifacts. However, a lot of organizations have difficulty adapting these values. The importance of Agile culture is often overlooked and doesn’t get the needed priority due to the organizations struggling with short term goals and urgent issues.

Agile coaching can greatly help with establishing and promoting these values.

Planning

Advanced Planning for the journey to Agile is important. It depends on setting effective organizational goals and creating a realistic road map based on available resources, both of which can be challenging.

• A plan needs to be developed for implementing pilot projects before the methodology can be rolled out or transitioned to mainstream development.
• Planning also needs to consider both product and process uncertainty.

When the development starts, requirements and architecture work need to take place before the development of the software. However, many organizations struggle with what is needed.

It is difficult to plan how to move toward Agile while keeping current production ongoing and uninterrupted. A progressive and gradual approach can be used but it also needs to align with the product roadmap. There needs to be a balance between the current release plan and longer-term process and methodology objectives.

During development, planning in every development phase is also a key component of the Agile/Scrum framework. Planning activities need to be well-defined and they require a team effort.

‘Big room planning’ for example is a key Scrum event, but usually sufficient time is not dedicated, and it is often conducted without first gathering sufficient and structured information. 

Organizational Structure

Agile is a unique way of development and organizational structural changes are often needed. This can be difficult, requiring considerable time and effort.

Agile/Scrum promotes a cross-functional team structure, whereas traditionally, organizations are usually organized based on functional units. Many OEMs and suppliers still have an extremely focused technical area mindset. In this traditional structure, different departments within these large organizations stay isolated (silos). Breaking these silos to create a truly cross-functional teams focused on specific features of the products, can be difficult.

In helping organizations with these transitions, this process must be viewed and accepted as truly being a fundamental change in thinking. It can be difficult for an organization to adapt, and there may be a lot of resistance from all of the distinct levels in the organization.

In a Scrum Team, typically there are 5-10 people that are Cross-functional (e.g. Systems, Controls, UI Designers, Software Developers, Application Engineers, etc.). Members should be full-time but some exceptions may be accommodated for certain roles like System Administration. Teams are self-organizing, and membership should not change during a sprint.

Even within an Agile structured organization, conflict of interests can arise e.g. scrum master (SM) reporting to the product owner (PO).

Experts and Agile coaches can play a pivotal role in supporting the needed transition of the organizational structure.

Global Teams

Agile development can be even more challenging when there are global teams in multiple different time zones with varying language skills and development capabilities across different disciplines.

In the ‘Big Room Planning’ event in Scrum, the same physical location is preferred and there is a lot of face-to-face interaction that need to happen between team members and across different teams.

In our experience, to deal with some of these challenges, it is possible to:

• Shift some of the global teams’ timing to provide better overlap to increase the efficiency of team communications.
• Use the time difference and different holiday schedules to your advantage. Distribute the workload and coordinate teams in different time zones effectively.
• Invest in telepresence facilities to improve interactions and communication.

Functional Safety

Functional safety development according to ISO26262 (as well as Cybersecurity according to ISO21434 and ASPICE) appears to be best suited for a ‘V Development’ model.

Functional Safety for example seems to follow the more traditional waterfall, V-Cycle or phased project-based approach.

There are different schools of thought. Some argue that these two cannot work together and that that compliance with the Functional Safety standard for complete software lifecycle (including requirements, architecture development, documentation, software development and testing) while following an Agile framework is not possible.

However, in our experience, they can work together very well with careful planning, project structuring and ‘Process Tailoring’. Both of these processes provide tailoring ability for the complete software lifecycle. This allows achievement of functional safety standards without compromising on creativity and continuous development of Agile.

‘Adaptation of the concepts’ and ‘Careful Structuring’ is needed of both processes to work well together.

• While following the ISO26262 workflow including all of the main development phases of ISO 26262 (i.e., Part 3 Concept Phase; Part 4 Product Development at the System Level; Part 5. Product Development at HW Level; and Part 6. Product Development at the Software Level) can utilize Scrum methodology with a Scrum-based Teams structure, Program Iterations, and Sprints. Continuous delivery doesn’t necessarily mean that at the end of each sprint there needs to be a software deliverable. It means that at the end of each sprint there need to be useful work products. Those could be work products other than software itself (e.g., requirements, documents, hardware, software architecture, and validation results). Agile is an iterative approach but each Sprint doesn’t need to produce the same type of work products.
• The keys to successful co-implementation are awareness, training, and process tailoring of both methodologies. SAFe (Scaled Agile Framework) for example, scales Agile principles and facilitates tailoring at various levels including Portfolio, Program and Team levels.

Estimation and Measurement

Task estimation, the measurement of progress, and reporting, are all key aspects of Agile development, but in reality, their execution can be difficult. Consideration need to be given not only to historical data and expert opinions, but also individual team member capabilities and uncertainty.

Story point estimation in Scrum for example can be difficult. In our experience,

• Stories are often weak; acceptance criteria are not well defined.
• Tasks are not detailed enough, not decomposed properly, and do not have the right structure.
• Stories are typically just issues resulting from some issue tracking or links to some system level requirements.

Some of the above issues are inherited from system level requirements.

Measurement of the scrum team progress and overall project progress is also difficult. Scrum typically uses Burn Down Charts for measurable progress. A Sprint Burn Down Chart shows the total Sprint Backlog hours remaining per day and the estimated amount of time to a Release. Ideally the ‘Remaining Effort’ should burn down to zero at the end of the Sprint. A Product Burndown Chart Is a “big picture” view of project’s progress (all the releases). These burn-down charts are not useful unless story point estimation is done properly.

And, correct metrics need to be used considering team members capacity, skill levels and uncertainty.

Feature-based Approach

A Feature-based approach uses a product backlog structure that is customer feature-oriented. It is based on a ‘Product Capability’ and ‘Product Value’-based development mindset.

In this approach, the Product Backlog is a features-based list of all necessary and sufficient items (Features, Functions, Services, Constraints, Behaviors, Bugs/Defects, Use Cases, Non-functional requirements) that needs to be delivered to the customer. It is a customer-facing, ‘Single Source’ of truth that details what is planned in the product.

The Product Backlog is decomposed and linked with program goals and sprint backlogs with bidirectional traceability, to higher level portfolio backlogs and lower-level sprint backlogs.

This approach is most effective when both the product backlog and the team structure are Feature-Based. At a higher level, this approach works well, but decomposition to lower levels, maintaining traceability, and maintaining project team and product structures can be difficult.

A summary of Key Agile enablers

Agile Culture, Organizational Structure, Training, Feature-based Product Backlog

Core Scrum Practices:

• Roles: Product Owner, Scrum Master, Development Team
• Activities: Big Room Planning, Sprint Execution (Planning, Daily Scrum, Development, Review, Retrospective and Product Backlog Adjustments)
• Artifacts (Product Backlog, Sprint Backlog, Product Deliverables)

A development partner such as LHP with our significant industrial experience, can help to considerably speed up the process and minimize the disruption to your ongoing software development and production cycle. 

As a development partner LHP can provide (our services) .

Our process, toolchain and methodology expert consultants have industry-wide experience in the migration towards Agile, and they can help you throughout this journey.

• Organizational Consulting for Migration to Agile
• 'Show the way' by being part of the process as a Development Partner
• Conduct a Pilot Project
• Process Implementation Support
• Development Support for Agile Culture 

Solutions Provided

LHP has technical experts who not only have many years of experience in software development within the Agile methodology, but they have also facilitated major OEMs and Tier-1s in their journeys from Waterfall to Agile.

LHP’s experts have delivered projects (e.g. some of the largest US OEM's and startups) developing software in Agile as well as supporting the migration to Agile. A summary of the solutions provided can be found below.

Key Activities:

• Assessed customers' current software development methodology and reviewed short and long-term software delivery targets.
• Created organizational requirements for all aspects including structure, key roles, and training.
• Performed Gap Analysis and identified Key Enablers.
• Helped adapt a feature-based top-down architecture and development approach.
• Helped create and maintain efficient product backlogs.
• Helped establish a development roadmap.
• Helped evolve the toolchain and processes to optimize program iterations and sprints.
• Organized team training.
• Structured and utilized a truly global team with many time zones efficiently.
• Performed pilot projects in a parallel manner for streamlining the process and toolchain while minimizing disruption to software releases. Helped with the adoption of the new process/toolchain after each pilot project.
• Helped move the whole organization towards Agile in an incremental way, by gradually expanding the approach that has been demonstrated successfully (in the previous step) in the pilot projects.
• Helped mature the Agile methodology as a development partner for multiple production applications spanned over several years. 

Summary

The Agile development methodology maximizes team effectiveness, minimizes waste, and facilitates the continuous development and integration of software. Agile is a proven methodology for the efficient development of software in complex projects. However, in the transportation industry, migrating from Waterfall to Agile is worthwhile but it can be a challenging journey. It requires an organization-level commitment that includes all aspects of development. With LHP as your development partner, we can help speed up the process considerably and minimize the disruptions to your ongoing business. Our significant and proven experience, delivered by our team of highly qualified experts, can help you navigate this complex landscape, avoid missteps and risks, and significantly ease your journey. LHP spares you from unnecessary pain, risk, and expense, and takes the mystery out of achieving success.