Development's Dirty Secret – Nobody Knows Their Data

By Jay Valentine, ContingencySales.com

Movement to the cloud accelerated the desire to rebuild applications with lower costs, apps more responsive and competitive benefits.  It is not happening.

There are several reasons a reported 70% of application transformations fail. 

One is that the cloud is just someone else’s data center using the same technology stack the apps previously used.  The much bigger issue is the world of relational and even non-SQL databases has disconnected people from understanding their data.

Let’s take an example.

A recent conversion with a multi-billion dollar company centered on the 330 relational tables they used to calculate customers’ bills.  No human can comprehend 330 tables with any understanding of what to do next.

Why were there 330 tables? 

The billing equation looks like this:  there is a customer, who uses a service, which has a unit price, and that unit price times the number of units is added to a bill, with some taxes and sent out.  How hard is that?

Sure, there may have been a million or so customers.  So what?  Why are there 330 tables?  Why scores of joins?  Why a struggle for primary and secondary keys?

The dirty little secret, which is the key to application non-transformation today, is that the vast majority of those tables are required simply because a relational, index driven data structure is not the optimal way to manipulate data for bill calculations.

The tools being used by current IT and cloud providers are 40 years old.  A lot has changed in 40 years, thus no wonder we live in a world of non-transformation. 

One approach that isn’t 40 years old is System Oriented Programming, the next step in microservices.

System Oriented Programming imagines data differently.  Let’s go there.

First, System Oriented Programming does not care about the code for that current billing system.  No code review, no spreadsheets or diagrams.  System Oriented Programming wants to see only data – input data, all of it, and output data, all of it.

Try this exercise:

Instead of 330 tables, create three.

One bucket is the data about the customer.  Everything known about the customer, even from the marketing department spreadsheets from the last reach-out event goes in one location.

A second bucket contains what that customer does.  She uses her phone, consumes minutes, is charged an agreed-upon price depending on time of use or other rules.

The third bucket is what the company wants to do about it.

The firm applies the price, business rules about rebates or specials, puts them on a bill, adds them up, adds tax and sends a bill.

Imagine for a minute all the data for a massively complex billing system is structured in this way.  Anyone can wrap their heads around 3 data locations.  Complexity is gone; transformation becomes not only possible but predictable.

System Oriented Programming expects that most data comes from somewhere else.  It has a sophisticated ETL function to smoothly accept data in multiple formats, from many locations, process it and emit the results. 

For 40 years software vendors, particularly those from the database world, believed their data stores were the center of the computing universe.  These vendors continue to hold many IT shops hostage to painful maintenance fees for essentially obsolete technology.  It may be obsolete, unable to support transformation, but it is the infrastructure of the modern enterprise.

These vendors’ self-interest is in complexity, expensive licenses and onerous tech people support.  They like those 330 tables and want to build even more indices spanning them, eating machine resources.

There is another way.

System Oriented Programming frees firms from the need for relational databases and their attendant overhead.  For the first time in a generation, perhaps two, a company can actually understand their data in those 3 tables.

Even the business exec understands:  the customer, what they do, what do we want to do about it?  Three data locations.

Transformations fail because the dirty little secret is nobody can understand their data; it is arbitrarily scattered in hundreds of tables no human can comprehend.  There is no economical transforming with current technology stacks.

System Oriented Programming delivers the most complex apps in a single business quarter because one finally understands one’s data.

About The Author

Jay Valentine is the CEO of ContingencySales.com, bringing disruptive tech products to market without venture capital and the VP of Sales for portfolio company Cloud-Sliver.