Software development has long been chaotic, whether from technical missteps or changing requirements. But now with more than 40,000 new software projects launched daily, this chaos is reaching new levels as the pressure mounts to deliver on ever-changing customer needs while keeping pace with competitors integrating artificial intelligence (AI) and other new technologies.
Organizations often seek to embed new frameworks or methodologies into their processes in an effort to reduce chaos. However, the real issue is not just processes – it’s visibility, argues Shannon Mason, Chief Strategy Officer at Tempo Software.
Development teams experience chaos when engineering leaders and other stakeholders influencing software projects lack clear visibility into priorities, dependencies, and resource allocation. Chaotic environments, in turn, drive misalignment that can undermine delivery and quality and strain individual developers, time, and budget, Mason says.
Techopedia sat down with Mason to discuss the source of the chaos in software development and how teams can improve visibility within their own workflows as well as across the broader organization, all the way up to the C-suite.
Key Takeaways
- The Covid-19 pandemic, economic uncertainty, and a lack of employee engagement have contributed to the escalating chaos.
- Visibility is a leadership issue, not just a team issue – when stakeholders, from developers to the C-suite, aren’t on the same page, delivery timelines, product quality, and team morale suffer.
- Frameworks and methodologies can help, but organizations need to provide the right environment for productivity, which requires focus and time.
- Small steps, like clearer reporting, integrated workflows, and stronger cross-team communication, can reduce chaos without adding unnecessary burden.
- AI can help organizations gather insights from data so that engineering teams are free to do the work of building good products that generate revenue.
With over a decade of experience in enterprise software, Shannon is known for aligning product innovation with customer outcomes, especially in high-growth, high-change environments.
Before joining Tempo, Shannon held senior roles at Digital.ai, GitLab, and CA Technologies. Her experience spans product management, solution architecture, and strategic operations – helping some of the world’s largest organizations adopt agile and DevOps practices at scale.
Shannon is passionate about connecting organizational psychology in vision to execution, and enabling teams to move with clarity and purpose. She regularly shares insights on agile transformation, product leadership, and strategic portfolio management.
Why Today’s Development Environment Feels More Chaotic Than Ever
Q: What has changed in the last few years that has amplified the chaos?
A: We are seeing a shift in dynamics from the pandemic to now. There was a rebalancing of what kept people happy during the pandemic, and that shifted us towards making sure employees were healthy and wanted to come to work. It was the biggest point of employee empowerment.
We emerged from that, and now some organizations are creating hostile environments so that they can reduce headcounts and they don’t have to pay employees severance. On top of that, the United States is creating a lot of chaos globally from an economic standpoint. There is a shift towards making sure that we are optimizing revenue, which is not necessarily different, but it is a response to the pandemic.
Add AI on top of that, so there is this underlying threat of jobs being replaced. It leads to employees feeling less empowered and less engaged. When there is that added layer of chaos from leadership, you end up with less care. Less care means that employees are not necessarily going to really commit to and follow the company’s plans.
The chaos has more to do with these factors than with AI itself.
We are seeing enough data to show that organizations are spending billions of dollars on AI investments, but they are not seeing the results. I don’t think we can blame AI – only the belief that it’s going to be a silver bullet to create speed, connection, knowledge, effectiveness, efficiency, and then ultimately revenue. What we are seeing now is that it is not the case either, although AI companies are seeing those things.
Q: How do you define what visibility means, especially within the context of software development?
A: What I refer to as the “squishy middle” has been maligned. Those folks who traverse between strategic conversations and how to execute against strategy get squished by both sides. They try to make sure their teams and their executive leadership have the right information. They are acutely aware of what is possible to do, what is not possible, the morale of the teams, and how the organization is truly doing.
We have seen over the last couple of years that we don’t need global middle management; we can remove that layer. But that is the first piece where you start to lose visibility, because they are focused on making sure that they have visibility and that they are keeping things moving forward, associated with the strategy for an organization.
There is a proliferation of tooling on top of that in organizations. What we are seeing is in reaction to that shift that happened during the pandemic, when we were all distributed – the need for tools for visibility, because employees were remote, ballooned out. Companies focused on the tools and not on establishing a process that worked for them. They didn’t focus on what it takes to keep people organized, motivated, and focused.
And then they removed the “squishy middle,” or made them feel like their jobs are at threat. Add on top of that a less motivated employee base, and they are not going to do the things that create visibility, because they want to hide – the more that people know about their project not doing well, the riskier it is for their jobs.
So, there is a set of factors leading to what feels like a more chaotic environment, but it is a reaction to cultural shifts in organizations that are not being well attended to.
Q: Many organizations assume new frameworks or methodologies will solve the chaos. Why do those approaches often fall short?
A: The first problem is that they tend to have lots of different things that need to be followed, rather than a philosophical practice that employees all agree to.
When organizations expect to solve underlying challenges with a framework and some tooling, they miss one of the key pieces, which is the people part. We can say ‘people, process, tooling’ over and over, but we skip the people part all the time. We do it because it is the hardest part, to get people through a change process.
In addition to ignoring the people and the change challenge, one of the other aspects is that change processes and framework implementation take time and repetition. It takes failure, because it is usually a shift in how we organize and relate to each other as humans that is then going to impact the work.
That requires time, because usually there is an underlying lack of trust, and it needs to be rebuilt. Most organizations don’t have the patience for that. Or worse, they fire people too regularly for any change process to see results.
Q: Where do you most often see visibility break down – at the team level, across departments, or at the executive level?
A: Teams usually have good visibility on what they are doing. But we continue to see data that teams usually feel disconnected from the global strategy. They still do not understand how their work is connected to what the organization is trying to do.
A simplistic framework process would be to make sure that it is happening. An organization doesn’t need to implement a big global strategy framework; it just needs to make sure that there is a thing they are doing and employees understand how their work is connected to that. That is a great place to start.
When teams lack visibility into how their work is connected to the bigger picture, they can get distracted easily because they don’t understand what is important. And then the executive leadership tends not to understand the reasonableness of asks. That is why the middle squishy layer is so important, because they do a lot of work to connect those dots.
That is also when you see the overabundance of tools come in, because the executives feel they lack visibility and need tools to solve it. But that overburdens the engineering teams with answering questions and documentation, when they want to spend as much time as possible in the flow state, building beautiful products.
Gaining Clarity Without Adding Complexity
Q: What are the first steps leaders can take to improve visibility without overburdening teams with more tools?
A: Ultimately, everybody wants to make wonderful products, regardless of whether it is software, a car, or a plush toy.
One of the first things we can do is step back and recognize the fact that usually an organization’s biggest line item is its employees. Are we providing the right environment for productivity, which requires focus and time? We are also asking people to solve challenges that require creative thinking, which means you have to understand the problem to be able to solve it.
The other piece is at the leadership level. Leaders must first recognize there’s a problem, then decide how the organization should work and what outcomes they want to achieve.
Visibility is always one of the top three things that people want from any sort of change management and change process. There are many different things that folks could do to alleviate the chaos, to increase the visibility, but the biggest thing is to recognize what they are trying to do in the first place, which is to build great products and then develop processes and ways of working that truly support that.
Providing intelligence on top of large data sets associated with how companies do planning is a great place to bring in AI.
AI is not built for large enterprises that have giant, monolithic code bases that are complicated. But it is effective for understanding the information sprawl and connecting the dots inside of an organization, and if that leads to more flow state, that is a perfect use.
There are predictable patterns of behavior that happen in organizations. One of the things that we can leverage artificial intelligence for is its objectivity. You need to give it boundaries, but if you ask it specific questions about what it sees in the data, it is going to give you an objective response.
Humans are usually going to give you as close to an objective response as possible, if you ask them to, but it is still going to have a bit of emotional opinion. And if it goes really bad it will be outright manipulative.
Q: What role should the C-suite play in creating visibility into projects and priorities?
A: Leaders need to be clear about their goals for the team and the business, then communicate them in a way everyone in the organization can understand.
You don’t need a 920-page strategy document, but you should write the story of your year that can be summarized in a one-page strategy overview that should be accessible to your organization.
Written information is one way of learning; visual information is another way. You are going to have a group of people who need to start doing to understand how things are connected with each other, which is why iterative development is so beneficial – they see the little pieces come together.
A lot of executive teams tend to check boxes and move along without coming back to what the organization said it was going to do and how that changed. They need to communicate that to the rest of the business, because the employees feel those changes.
In the current chaotic situation, leadership teams are undertaking a significant amount of external work to minimize challenges, and as a result, they have lost some of their focus on the internal repetition that is necessary to keep their teams moving forward.
Q: Looking ahead, how do you see the visibility challenge evolving as software development accelerates further?
A: The pace of technological change is so fast that even our human brains have trouble keeping up with it.
I hope that we end up doing less work about the work, because visibility is just understanding how work is moving through the system. It feels chaotic because the moment we start executing against something, it changes.
One of the things that organizations can do, augmenting their data with intelligence, is to focus less on tracking work and more on providing information about the purpose of the work. That will lead to the big trifecta – happy employees build great products, great products make happy customers, and ultimately that leads to more revenue.
You get to that state by allowing your teams to build things and do what they came to work to do, which isn’t to answer questions about status. It is a false narrative because as soon as we start executing against the plan, the plan changes. There is never truly a static status.
The Bottom Line
Software development chaos isn’t only caused by broken processes or outdated frameworks – it stems from a lack of visibility. When teams and leaders don’t have a clear view of priorities, dependencies, and resources, even the best methodologies can fail, leading to misalignment, slower delivery, and lower product quality.
To solve visibility problems, companies need to simplify, not complicate. Clear communication, connected workflows, and smart use of AI can help leaders give their teams the focus they need to create great products – and, ultimately, better business results.