BI Roadblock #6: High Data Latency

By Nicole Hitner, Exago

Data latency as it applies to business intelligence is “how long it takes for a business user to retrieve source data,” whether it be in the form of a report or dashboard. Even the most sophisticated data systems experience some form of latency, as it takes time to pass data from one server to another; but some are especially prone to latency issues.

Data exporting and analysis via a spreadsheet is one such system. The manual work involved dramatically increases time to insight and increases the likelihood that a record or calculation will be obsolete by the time it reaches a decision maker. High-volatility, high-stakes data sets should be the easiest to retrieve and the least latent.

Symptoms

How latent is too latent will depend on the use case. One company may have no trouble waiting a few hours for a static report to be manually updated while another might take serious issue with a dashboard needing more than four seconds to load. It all depends on context.

Signs you or your organization may be struggling with data latency include:

• Speed of data retrieval is slowing business operations.
• Speed of data retrieval is slowing customer service or otherwise adversely impacting customer experience.
• The most up-to-date reports and dashboards available are displaying old data.

Causes

One or more steps along your organization’s data processing pipeline could be contributing to the data latency issue. Here’s a scenario common to young organizations and business departments:

Data exports can take anywhere from a few seconds to several minutes, depending on the system and the volume of data being exported. Still, the bulk of retrieval time is likely going to the manual analysis stage. The raw data isn’t usable to the stakeholder in this scenario, so an analyst must either design a new report or update an existing report with fresh information before emailing it.

In another processing pipeline, the cause of latency could be entirely different.

Here, assuming ETL (Extract, Transform, Load) is configured to take place overnight, querying is the likely holdup. The volume of data being retrieved might be exceptionally high or the BI solution poorly configured to perform that type of query.

Data latency causes should be assessed on a case-by-case basis, but here are some common ones to consider:

• A manual step in the process is slowing retrieval.
• ETL is not being performed often enough or at the appropriate times.
• The database or warehouse server(s) is/are being overtaxed and cannot handle the processing load.
  • The warehouse architecture is not optimized.
  • The database needs to be scaled to handle load.
• The BI web server(s) is/are being overtaxed and cannot handle the processing load.
  • The BI application’s configuration is not optimized.
  • The BI application needs to be scaled to handle load.

Risks

Data latency can cause operational delays, inefficiencies, and in the most severe cases result in data staleness and misinformation. Delays and inefficiencies can adversely affect customer satisfaction and, therefore, impact revenue. Misinformation often results in poor decision making and, depending on the decision, can have a lasting impact on the business. Users of high-latency BI solutions are also likely to desert the application in favor of other systems, reducing the return on your BI investment.

Solutions

Organizations that rely on manual data processing are at much higher risk of experiencing data latency fallout, particularly if their data is highly volatile. Adopting a BI solution can dramatically reduce the time spent turning raw data into usable insights by automatically refreshing existing reports with fresh information.

Those already leveraging BI can work to optimize their data processing pipelines by first identifying the bottleneck and then working to alleviate it. As indicated in the Causes section, this is likely to involve scaling and/or configuring your data source(s) and/or BI application server to handle additional load more efficiently.

In rare cases, organizations may find their BI solution is simply incapable of meeting their latency goals. In the search for an alternative application, it’s helpful to read about assessing product performance. Enterprises need access to technology that will scale with them and be able to meet their evolving data latency requirements.

About The Author

Nicole Hitner is Content Strategist at Exago, Inc., producers of embedded business intelligence for software companies. She manages the company’s content marketing, writes for their blog, hosts their podcast Data Talks, and assists the product design team in continuing to enhance Exago BI. You can reach her at content@exagoinc.com.