How dirty is your data? As business intelligence experts know, the answer to that question can have a tremendous impact on your organization’s efficiency. And although it may sound unsavory, the truth is that unreliable information, aka dirty data, is a common business intelligence problem for large companies.
Dirty data becomes a problem as businesses grow over time. As stores, offices, warehouses, and plants are established and enhanced, data systems multiply. Lack of standardization among these systems may allow users to produce inconsistent, invalid, or duplicated information, or simply fail to enter critical information.
While it may not have been a practical or intentional choice, lack of standardization allows for the adoption of different business rules and data systems across the company. Sites may use different sets of codes to represent similar products, services, or entities such as vendors, customers or lines of business. Data systems in use at some locations may collect information from users into free-form entry fields instead of using a “pick list” of options, increasing the likelihood of incorrect entries or multiple versions of a single value resulting from misspelling and typos. Furthermore, these systems may allow certain values such as names, addresses, or dates to be unexpectedly overwritten, limiting the ability to track historical events or create accurate business intelligence trend reporting.
When dirty data finds its way into reports, the results can be inaccurate and unusable, with important information distorted, excluded, or even multiplied many times over. When this happens, the hunt for meaningful, actionable business intelligence can become a laborious, never-ending quest.
Companies may not be aware of the full extent of their business intelligence problems, or may accept them as an unfortunate fact of life. Even if you know where the problems lie, you may have to follow elaborate and repetitive procedures to gather the business intelligence you need to succeed. Dirty data makes it difficult to consolidate and measure business results. Inaccurate and missing information prevents decision-makers from getting answers to their important questions about your business.
To evaluate the health of your data, consider the following business intelligence questions:
- Do you find yourself doing the same repetitive tasks each month to compile clean data?
- Is it difficult to answer questions about your business using your data?
- Do you regularly follow extensive procedures to retrieve and consolidate companywide data?
- Do you spend too much time analyzing your business and not enough on decisions that will move the company forward?
If your best analysts are spending the majority of their time compiling the data required to create reports, and the results always arrive at the last minute or past deadlines, it’s time to look into the role of business intelligence in your organization. Tackle the problem head-on with an experienced business intelligence expert who can identify and address dirty data issues, homogenize data across different systems and standards, and consolidate company-wide data so it is accessible and reportable whenever you need it. You’ll be on your way to creating a more strategic and action-oriented environment where those over-worked analysts can focus their efforts on analyzing results instead of preparing them.
To begin the process of addressing data quality, your business intelligence partner will need to understand your organization’s goals. Where do you want to be in five years? What has been keeping you awake at night? Identifying strategic priorities will help you outline a plan which focuses on the most relevant and critical information.
To establish high-quality, reliable data, the business intelligence team will filter, merge, standardize, and validate records, all with the end goal of simplifying life for the end user. But creating a great business intelligence solution involves much more than database management and coding. The process will likely involve identifying areas where company standards and business rules can be added or improved, and getting consensus from stakeholders to define the rules that will prevent future problems.
Working with the right business intelligence partner, you’ll ultimately find the treasure that every large business dreams of: a centralized, authoritative source of data, married to a business intelligence system that analyzes your business to pinpoint critical problems and highlight favorable opportunities, and then reports the results you need to measure progress and plan for the future.
Business Intelligence, Operational Efficiency and Dirty Data
by Paul Ousterhout, Business Support Expert