By Joe Pasqua, Executive Vice President of Products, MarkLogic
The high profile outages that happened almost concurrently at the NYSE, United Airlines, and the Wall Street Journal this past summer, serve as a stark reminder of our universal dependence on software. Even the best run companies in the world are vulnerable because software is growing in complexity to the point where it is virtually unmanageable. Systems are so large, so byzantine, and in some cases, so old, that they are beyond the scope of any individual to understand. For companies that acquire or merge with other companies, the situation is even worse. Integrating two of these behemoth systems can take years and the result is rarely an improvement. Typically, the complexity is multiplied.
I can’t overstate how harmful complexity can be to an organization. A complex system makes it very difficult to predict circumstances that may result in an outage. We’ve seen that repeatedly. It undermines the security of the system since security professionals don’t understand the full attack surface or interaction of all of the components. Perhaps most importantly, complexity stifles innovation because every little change requires navigating a quagmire of code that is poorly understood, and wasn’t so much designed as accreted.
The biggest source of complexity in today’s enterprise software is a mismatch between the problem to be solved and the technology used to solve it. We think of IT as a very fast paced arena where technology and expectations change on a daily basis. That’s true, but the fact is that some of the core technologies in widespread use today were conceived over 30 years ago. We’re using rotary phones in the age of iPhones. We’re sticking with software designed in the 20th century for requirements that didn’t even come into existence until the 21st. It’s just not a good fit any more.
Nowhere is this truer than in the financial sector. In the aftermath of the global financial crisis new regulations have required an unprecedented 360-degree view of the business. That requirement is in turn requiring that systems and data be integrated in ways that were never intended or foreseen. They are trying to do this with legacy technologies, like relational databases, that are rigid by design and not adaptive by nature. The result is high costs, long time lines, failed projects, and even more complexity. They are paving over potholes along the way, but what they need is a serious rethink of their infrastructure.
We’re sticking to this legacy software because it served us extremely well for many years. At one time it brought the revolutionary new paradigm that allowed us to leap forward. When we were faced with new requirements, we stuck with the tools and technologies that made us successful even when it wasn’t a great fit. We added layers of tools and processes to help adapt the old model to the new reality. Before we knew it, we ended up with Rube Goldberg machines that were costly to maintain, fragile, error prone, and hard to improve. That’s where we are today.
This isn’t a technology challenge. Innovation is flourishing and there are new technologies that are every bit as revolutionary today as the legacy systems were in their time. This is a cultural challenge. The organizations that succeed are those with a leader or leadership team who is willing to buck the status quo and bring in the right technology for the job. We saw this play out very publicly with Healthcare.gov. It wasn’t until the project leadership decided to abandon legacy approaches that the system got onto the very solid footing that it has today. When everyone is telling you, “This is the way we’ve always done it, this is the way everyone does it,” it takes vision and guts to go in a different direction.
Most large organizations think in terms of multiple years and millions of dollars for a new solution. Their entire procurement process is predicated on that fact so they tend to be slow, methodical, and risk averse. This favors incumbency. Often times, new technologies and new implementation methodologies can implement entire systems in the time it would take to create a proof of concept using a legacy technology. Once again, leaders who are willing to embrace a new approach can see dramatically faster time to results while also reducing complexity and the pain that comes with it.
The cost of sticking with legacy solutions is only going to grow and new technologies and approaches will be adopted. The only question is when. It’s up to the technology leaders in our organizations to drive this and counteract the inertia of legacy software.