Managing Big Data – A Brief Introduction to Data Mesh, Data Fabric, & Beyond

There is a tongue-in-cheek quote that predicted at some point in time, librarians will rule the world. This was because they are the only people that know how to organize, manage, and use data.  Anyone who has tried to reconcile their own personal data e.g., contact lists, photos, or emails between their various devices, can relate to this comment.  For companies, this challenge is exponential and increasingly daunting.

The Problem is Not Only Existing Data but the Creation of New Additional Data  & How to Process It

The proliferation of digital technologies, the internet, connected devices, and the growing reliance on data-driven processes have contributed to exponential growth in data generation.  For example:

• By 2025 there will be a predicted 75 billion IoT-connected devices
• Artificial Intelligence (AI) and Machine Learning (ML) demand enormous amounts of data to train models and make predictions
• Video and multimedia data is exploding via Social Media and User-Generated Content

Simply stated, data will grow exponentially.

This means the storage and processing of data processing tools we have today will have to deal with double that amount in the next period of measurement – a year, a month, a week, or even a day!

We can’t ignore the production of new data –  It provides information and insights that are used to make informed decisions, gain knowledge, solve problems, and understand patterns or trends.

We Need Data. Data is Raw, Unprocessed Facts and Figures

However, at the same time, data is hard to manage. Consider the analogy of a library. It starts with a modest collection of books.  As time goes on, more books are added, covering various topics and genres.  As the library collects books from diverse sources and genres, data is generated from various digital sources such as social media, sensors, transactions, and online activities.  As more books (data) are added to the library, it becomes increasingly challenging to organize, manage, and extract insights from the vast collection.  The library becomes, and is, an increasingly valuable resource for knowledge and information.  New investments in approaches, technology, storage, and usage mediums are all required just to keep up.

Similarly, Big Data continues to hold and drive immense potential for organizations and society to gain insights, make informed decisions, and drive innovation. However, managing Big Data presents many challenges.  Examples of these are:

1. Volume: Big data has extremely large and complex datasets which exceed the storage and processing capacities of traditional data management systems, creating a need for new ways to deal with more data. 
2. Velocity: Big data is generated at a high velocity – it is produced rapidly and continuously, contributing to the growth in the volume of data. 
3. Variety: Managing and integrating data from diverse sources and formats requires flexible data management approaches, which is counter to the rigidity of software and systems.
4. Veracity: Big data can be corrupt, incomplete, or unreliable.  Evaluation and correction of the data are paramount and typically consume a significant amount of time (and cost). 
5. Scalability: Scaling up infrastructure and processing capabilities to accommodate growth without sacrificing performance requires new ways of looking at the problem.  
6. Skills and expertise: Managing and understanding big data requires a specialized skill set of expensive and in-demand talent.

Different Approaches to Managing Data: Data Fabric & Data Mesh

There are different data management approaches or architectures for dealing with Big Data.  Two of the more recent approaches are Data Fabric and Data Mesh.

Data Fabric is an Architectural Framework 

Data Fabric aims to provide a unified and integrated view of distributed and disparate data across an organization. It enables seamless data access, management, and integration across different systems, locations, and formats.

The concept of data fabric recognizes that in modern organizations, data is distributed across various sources, such as on-premises databases, cloud services, SaaS applications, IoT devices, and more. These data sources may have different formats, protocols, and storage technologies. Data fabric seeks to overcome the challenges of fragmented data by creating a cohesive layer that allows organizations to access and utilize their data effectively.

Key Characteristics and Components of a Data Fabric Typically Include:

1. Data Integration: Data fabric integrates data from multiple sources, including databases, data lakes, streaming platforms, and cloud services.
2. Data Virtualization: Data fabric provides a virtualized view of data, abstracting away the underlying complexities of data storage and formats
3. Data Orchestration: Data fabric manages the movement and flow of data across different systems and platforms. It orchestrates data pipelines, transformations, and workflows to ensure data consistency, quality, and reliability.
4. Data Governance: Data fabric includes governance mechanisms to enforce data policies, security, and compliance across the organization. It ensures that data is accessed, used, and shared according to predefined rules and regulations.
5. Metadata Management: Metadata plays a crucial role in data fabric by providing information about the data sources, schema, quality, and lineage. Effective metadata management allows users to understand the data’s context, improve data discovery, and facilitate data integration.
6. Scalability and Performance: Data fabric is designed to scale horizontally and handle large volumes of data. It leverages distributed computing technologies and optimization techniques to ensure performance and responsiveness, even with massive datasets.

The goal of a data fabric is to enable organizations to have a unified and consistent view of their data, regardless of where it resides. It helps in breaking down data silos, improving data accessibility, and enabling data-driven decision-making across the enterprise.

Data fabric solutions are typically implemented through a combination of technologies, such as data integration tools, data virtualization platforms, metadata management systems, and data governance frameworks.

Another Emerging Approach is Called Data Mesh

Data Mesh is an architectural approach and organizational model for managing data in large, complex organizations. It aims to address the challenges of data silos, centralization, and scalability by promoting decentralized data ownership, domain-oriented teams, and self-serve data infrastructure. Data Mesh principles focus on enabling data as a product and empowering domain experts to take ownership of their data assets.

Key Characteristics and Principles Associated with Data Mesh:

1. Domain-Oriented Teams: Data Mesh suggests organizing teams around specific business domains rather than centralizing data functions. These domain-oriented teams have end-to-end ownership of their data, including data collection, storage, processing, and governance. This approach allows teams to have deep domain knowledge and responsibility for their data assets.
2. Federated Data Ownership: Data Mesh promotes the idea that data should be owned by the domain teams responsible for generating and using it. Instead of a centralized data team controlling all data, ownership and decision-making authority are distributed among domain experts who understand the context and requirements of their specific data.
3. Self-Serve Data Infrastructure: Data Mesh advocates for providing self-serve data infrastructure and tools to domain teams. This enables teams to have direct control and responsibility for their data pipelines, storage, and processing. They can use standardized infrastructure components and shared platforms to avoid reinventing the wheel while tailoring solutions to their specific needs.
4. Data as a Product: Data Mesh treats data as a product, shifting the mindset from data being a byproduct of software systems to a valuable asset with its own lifecycle and stakeholders. Data is seen as a valuable resource that should be discoverable, understandable, and usable across the organization.
5. Federated Governance and Standards: While domain teams have ownership of their data, Data Mesh emphasizes the need for federated governance and standards. Cross-functional collaboration is important to establish data standards, quality guidelines, security measures, and compliance requirements. Centralized governance bodies and communities of practice can facilitate coordination and knowledge sharing.
6. Data Mesh Architecture: Data Mesh architecture promotes the use of decentralized, domain-specific data platforms. These platforms enable domain teams to manage their data lifecycle, including data storage, processing, access controls, and quality monitoring. Common infrastructure components, such as data catalogs, data discovery mechanisms, and metadata management systems, help facilitate data interoperability and collaboration.

The aim of Data Mesh is to foster a culture of data empowerment, collaboration, and autonomy while reducing bottlenecks and dependencies on centralized data teams. By embracing domain-oriented teams and self-serve data infrastructure, organizations can unlock the potential of distributed data assets, enable innovation, and improve the overall effectiveness of their data-driven initiatives.

The Limitations of Data Meshes and Data Fabric & Future Solutions

While Data Mesh and Data Fabric are two approaches that can assist in dealing with the problem and extraction of the value of Big Data, they are both (as well as other approaches) restricted by the Business-IT gap – the difference in the time between the business need and when IT can deliver the solution.  The reason for the gap can be summarized by:

• The need for deep IT skills to implement, set up and maintain the data management toolset.
• Wide and varied requirements to manage Big Data require a similarly wide and varied toolset, all from different vendors with different standards, etc.

Data mesh and data fabric have been a positive development in dealing with ever-increasing data volumes and complexity.  Looking to the future, we can anticipate the development of new data management tools that address the challenges of Big Data and provide significant value. These tools take a fresh approach to managing data by incorporating several key features.

Firstly, they enable the access of raw data from various sources in real-time, eliminating the need for interim data repositories such as data warehouses. This real-time access ensures that the data is always up to date and readily available for analysis and decision-making.

Secondly, users can selectively choose specific data elements or fields instead of retrieving complete tables. This selective approach enhances efficiency by focusing on relevant data and reducing unnecessary processing and storage requirements.

Additionally, these tools provide a comprehensive toolset for data cleansing and transformation, ensuring the data is prepared and optimized for the required solutions. This includes processes such as removing inconsistencies, resolving errors, and transforming data formats to meet the specific needs of analysis or integration.

In summary, these future data management tools revolutionize the way in which Big Data is handled and thus make source data more accessible and closer to the end user.  Software platforms such as r4apps offer real-time access to raw data, selective retrieval of data elements, and a robust toolset for data cleansing and transformation. Additionally, they extend their reach by integrating the data with the other digital assets and processes in an enterprise.  By incorporating these features, these tools mean that a business user has immediate access and the ability to work with all digital assets (data, software, and process) in an organization.  The result?  Greatly enhanced efficiency, improved data quality, and empowered organizations to derive maximum value from their data and related assets.