Finding “One Source of the Truth” is the holy grail for those who manage big data. “One Source of the Truth”, or OSOT, the the concept of having a single, authoritative, and reliable source of information (or data sets).
Organizations desire the OSOT because it allows all the individuals and systems who use their data to have a single point of reference. This mitigates issues that arise from data inconsistencies, discrepancies, and confusion that may arise from multiple, and potentially conflicting, sources of data.
The Benefits of OSOT
There are several major benefits of OSOT for organizations. These include:
- Vastly improved organizational insights
- Better decision making
- Reductions in operational costs and complexity
- Improved confidence in the data
As such, organizations have looked at several ways in which they can achieve this. In general, these have focussed on data replication technology such as data warehouses and data lakes. Typically, this involved taking data from various heterogeneous sources, replicating it into a centralized location, and standardizing it.
However, for reasons discussed in this blog post, data repositories have not yielded the results that they promised, given the significant investment they necessitate. In fact, 75% of Data Repositories projects fail, and/or do not meet expectations.
New Technology, New Approach – Lift and Shift
Many SaaS platforms are being specifically designed to tackle some of the challenges we’ve discussed previously in big data. They provide new tools and approaches for moving data to a central location.
Some of these tools offer data warehousing solutions that are designed to handle very large volumes of data, quicker query performance (traditionally a pain point for dealing with large data), as well as the ability to support a large variety of data analytics and business intelligence tasks.
The result?
The ability to scale computing power, irrespective of data volumes, means faster response times to the demands of the business. Scaling power usually comes at a cost.
Not All New Approaches Achieve OSOT
While “lift and shift” data platforms have some clear benefits, their approach, much like the previously mentioned data warehousing/data lake projects, fails to achieve OSOT.
Not only does OSOT require technology integration – but also an agreement of data definitions, processes, and governance across the organization. Having data in a central location is a small part of the OSOT equation. The true challenge lies in establishing agreed definitions, meaning, and context for the data. This is a significant challenge as different groups define and use the same data in different ways, for their own, unique reasons.
In other words, data is nuanced. Let’s consider what “sales revenue” means to different business users to garner some insight into the nature of data nuances:
- For salespeople – the monetary value of a product or service with consideration to gross, net, and region
- For finance people – definitions aligned with accounting principles with consideration to discounts, returns, etc…
- Marketing – the number of products shipped or delivered, aligned with product distribution
- Executive leadership – strategic goals and growth targets with a view over longer periods of time.
As you can see from the examples above, context and nuance shape how data is interpreted. This highlights the importance of establishing common data definitions that take into account the definitions of all the various stakeholders.
Factors to Consider when Establishing Common Data Definitions
- The contextual differences in data expressions
- Operational focus: Different user groups have different operational requirements. As a result, differing priorities can lead to irregular or variance in data interpretations
- Technical expertise: Users with varying levels of technical expertise may interpret data differently
- Subjectivity: Data interpretation can involve a degree of subjectivity
- Tool usage: Different software may represent or interpret data differently from each other
- Historical practices: Pre-defined departmental data definitions can become entrenched and hard to change
- Communication gaps: When communicating across an organization, interpretations may get lost in translation or suffer from a case of broken telephones
- Lack of governance: A strong framework for data governance is essential
- Evolving processes: As businesses change over time, so too do data definitions
If OSOT can be achieved, it provides significant value. However, traditional approaches to date – data repositories – have attempted to achieve OSOT but have a high failure rate. On the other hand, SaaS platforms for data management succeed at data movement and consolidation but not in terms of OSOT – they don’t tackle data definitions, which is a major organizational challenge that never ends.
Is OSOT Even Achievable?
I hate to leave you on a cliff hanger like this – but you’ll have to look out for my next blog where I discuss this in far greater detail, and if its not achievable, what alternatives can organizations consider.
Sign up for our newsletter and receive posts like this in your email
