The Extinction Event In Software
By Gurjeet Singh, PhD
While much of the discussion around AI deals with the societal implications of this transformative technology, very little is spent on the subject of what it means for the software industry. Given that software is the primary vehicle for AI and software will be its UX, there are massive implications for this multi-trillion dollar industry.
AI will create a massive extinction event in the world of software. Unlike the Cretaceous–Tertiary extinction event, we can see this one coming. Further, we understand the forces that will make it happen. Let’s take a look at them in turn:
- Compute is in its golden age. Advancements in compute and the adjacent storage space have revolutionized our capacity to solve problems — across enterprises large and small. With an afternoon of tutorial and the cost of a Latte Grande, a high school student can harness the power of a supercomputer. GPUs are readily available in the cloud. Amazon is offering easy and cheap access to FPGAs. Even more specialized hardware designed for key machine learning tasks are making their way to market. Quantum computing is not only real, but looks to contribute within the decade — well ahead of projections (although, the debate on quantum supremacy is far from settled).
The result is that we are able to tackle challenges that seemed preposterously hard only a decade ago. We tend to lose focus of this acceleration of acceleration, but it is very tangible in compute and it will power a whole generation of applications.
- The development of the field of AI. While the AI hype cycle is in full cyclone mode at this point, the truth of the matter is that some of it is completely justified. There has been extraordinary growth in the field — evidenced in the number of papers, the startups working on key problems and the breakthroughs. Even in the face of an “AI winter,” there is enough going on in the field, enough avenues of pursuit, and enough low hanging fruit to revolutionize entire industries — even if new and better techniques were not developed.
AI technology has effectively outpaced the talent inside most organizations. They can’t consume what already exists — which means that the opportunity is as much in making it easier to consume than it is to develop new approaches.
The consumption opportunity is what will rewrite every software application in existence.
- The growth of the UX mindset. We are ardent believers in the concept that UX matters immensely in AI. Our friends at Google share our perspective on the matter. It cannot be understated. AI, without UX, has an audience size no larger than the state of Tennessee — that is to say, hard core data scientists and researchers. Marry AI and UX, and the audience is the entire globe. The evidence is likely in your home. Voice is the UX for NLP + NLG (both branches of AI) in your Alexa, Cortana, Siri, and Google devices. The results are pretty amazing:
- The rise of component architectures. It has never been easier to be an AI company than it is in 2018. This is truly an “on the shoulders of giants” moment in history but with a twist. It is less about a handful of towering intellects as it is about thousands of great intellects — all of whom share their code, their approaches and their algorithms on Github, Stackoverflow and the like. These full-fledged machine learning libraries provide exceptional capabilities to larger and larger audiences. Further, in the race for cloud dominance, Google, Amazon and Microsoft are incorporating and democratizing AI and machine learning capabilities. Granted, there are drawbacks to some of these approaches (lack of sophisticated unsupervised learning, the ability to use compute resources efficiently without dumbing down the math), but the power available to the technically inclined is stunning.
- The arrival of justification. We are firm believers that for AI to succeed, that it needs to be explainable on a level far greater than currently articulated by the explainable AI crowd. Identifying what algorithm was used and what parameters were learned do not provide any intuition as to what is going on. It allows one to “check the math,” but it is not valuable to know that your computer can do matrix multiplication or basic linear algebra operations. The concept of justification is far more robust and is what’s required to move AI into production. While Justification also identifies what algorithm was used and what parameters were applied, it goes further, providing the additional ingredient of intuition — the ability to see what the machine is thinking; “when x, because y.”
This is the number one requirement for enterprise class AI, and it is solvable with the right technology.
- Small, Fast, and Intelligent vs. Big and Slow. There are some giants in the software industry. They make the lion’s share of revenue for the industry. They are also at the greatest risk. The future is not in massive, monolithic pieces of software with billions of lines of code. The future is in intelligent applications that focus on discrete business problems and can interact with other applications. This is classic innovator’s dilemma territory. The big guys won’t care if a small part of the supply chain business gets carved out by an intelligent app. Over time (a very short time), though, that same app is going to grow into adjacent areas and will, in the blink of an eye, have co-opted entire processes. Look at what Workday has built, what Concur built. These companies seemingly “came out of nowhere” to swallow up large segments of the market. Now they are the hunted.
The future belongs to those companies that can rapidly construct intelligent applications. The data is there, waiting to be learned, but most giant enterprise applications can’t do anything with it.
All of these forces are conspiring to create an extinction event the likes of which the software industry has never seen. Every application, every model, every process will change to either call on an AI platform or execute as an AI application.
From an evolutionary biology perspective, the current class of software, written with human logic and brittle rules, simply won’t be able to compete and will cease to exist. Many of those applications will cease to exist because the companies that run them cease to exist; also caught up on the wrong side of the accelerating evolutionary race.
We recently paged through a software titan’s product listing. More than 52 percent of its existing software applications would be obsoleted by AI technology. The rest would be obsoleted by things other companies did.
Core components like databases are already automating and becoming intelligent. The conversation won’t be around in-memory vs. disk-optimized, it will be around intelligent vs. not intelligent, and unlike the in-memory vs. disk-optimized, there won’t be much of conversation.
While human-authored rules will not survive this wave, human involvement in the development of, and interaction with, software will continue. This will be the driver of requirements around UX and justifiability. This is not to suggest that software will write itself, but it does suggest that software will learn more rapidly, optimize on problems more efficiently, and recognize changes to the underlying data more effectively.
While some will question the societal impact of this tectonic shift, suffice it to say, that in the short and medium term there is a tremendous amount of work to do — not just to write this next generation of software but to operationalize it, to build the user experiences, and to QA it. Make no mistake, however, this change is coming and will rewrite the industry.
About The Author
Gurjeet Singh is Ayasdi's CEO and co-founder. He leads a technology movement that emphasizes the importance of extracting insight from data, not just storing and organizing it. Dr. Singh developed key mathematical and machine learning algorithms for Topological Data Analysis (TDA) and their applications during his tenure as a graduate student in Stanford’s Mathematics Department where he was advised by Ayasdi co-founder, Professor Gunnar Carlsson. Dr. Singh is the author of numerous patents, and has published in a variety of top mathematics and computer science journals. Before starting Ayasdi, he worked at Google and Texas Instruments. Dr. Singh serves on the Technology Advisory Board at HSBC and was named to the Silicon Valley Business Journal’s “40 Under 40” list in 2015. Dr. Singh holds a B.Tech from Delhi University and a Ph.D. in Computational Mathematics from Stanford University. He lives in Palo Alto with his wife and two children, and develops multi-legged robots in his spare time.