When compared to data repositories, data source technologies are proving to be a promising technology with significant benefits and value. These include the speed of implementation, the realization of benefits, and reduced cost and risk. But Data Repository technology is still very dominant despite the high costs, long implementations, and high failure rates. Could this be due to the fact that data from source technologies are disruptive?
Data Source Technologies Change The Current Approach to Managing Big Data – and That Causes Resistance
Similarly, the introduction of 3D printing technology resulted in similar resistance. The process for manufacturing something was to design, engineer, and program a machine to create the part, (in scale) and then packed, ship, and delivered it to the end user. 3D printing capability resulted in an immediate process change. It starts and finishes with the end user. The end user obtains a schematic online which has previously been designed and engineered. Via 3D printing, the end user now manufactures the part (or parts) for immediate availability and use. Scale is no longer a consideration and custom modifications can be made on the fly. The result – faster, cheaper, and a process that focuses and is executed by the most important person – the end user. As the technology was tested, understood, and introduced, resistance was broken down as people realized where it provided the greatest value versus a wholesale replacement of the existing process.
The Traditional Approach to Managing Big Data is Through a Data Repository
The data is copied from the source and ingested into the repository via a process such as extract, transform, and load (ETL). Users are then able to access the data for whatever purpose is required – reporting, analysis, or further consumption. However, this is a time-consuming and costly process, that is fraught with risk, and if successful, only provides partial realization of the planned value (as evidenced by the high failure rate of data repository implementations). Even if successful, the data in the repository is a copy of the original, and therefore not current. Additionally, this technology struggles with the volume of new (Big Data) due to the ingestion process. A large number of professionals are involved in the Data Repository delivery process, including a variety of I.T. skills, as well as business analysts and users. These resources must work in serial, and thus one step must finish successfully before the next can begin, lending itself to long implementations and difficult error correction.
Data Management via Data From Source & Technologies
The process with data visualization tools, just like 3D printing, starts and finishes with the individual who will create value – the end user. Data is accessed from the source by an end user and transformed as required (similar to ETL, but from the source and without the need for a load). The transformed data can then be consumed as required, for example, report writer, or further processing via the algorithm. Like 3D printing, the process is much simpler, far less expensive, and much more responsive to the needs of the business/user, and the risk is greatly reduced. Most importantly, the process is executed by the value provider, the end user as many of the previous IT skills are no longer required.
So why there is still a predominant focus on data repository technology versus data from source? 3D printing was seen as a disruptive technology. This resulted in fear that this technology would completely replace the traditional manufacturing process. This would change business models, jobs, economies, etc. However, to date, this is not what occurred. 3D printing was initially primarily used for innovation projects, to produce spare parts that were no longer in inventory. As the technology was tested, understood, and introduced, companies and people adapted to take advantage of the technology, where it provided the greatest value versus a wholesale replacement of the existing process (and also delivered value).
In the case of Data from Source technologies are we experiencing the same reaction? In my experience to date, the resistance to this technology is high, particularly from the IT world. Considering that the Data Repository has such a high rate of failure, and the data from source technology indicates the potential for significant value, I find this resistance curious. Of all the potential reasons I find the personal reaction of those in positions of power as the most influential:
- Fear of the unknown: Disruptive change can upend established practices, and norms, leading to uncertainty and resistance.
- Loss of control or power: Individuals or organizations that hold positions of power or benefit from the status quo may resist change that threatens their authority or influence.
- Job displacement: Disruptive change often involves automation or the introduction of new technologies that can replace human labor.
So as Data from Source technologies continue to develop, it’s important to note that while resistance to disruptive change can hinder progress, it also serves as a check-and-balance mechanism. it is important to provide an open and favorable message, particularly to the IT world, as an alternative and complementary technology to the data repository. These messages should include:
- Look like a Hero (Part 1): With the right project, business users can be wowed with never seen delivery times, very low budgets, and business user control and responsibility.
- Look like a Hero (Part 2): With the removal of many of the “heavy lifting activities” due to the real-time capabilities of Data from Source technologies, a considerable budget is freed up for other high value-added and strategic activities.
- My Job Just Got Better: Many IT activities are mundane and boring. Again, with the elimination of these activities, IT professionals can focus on more rewarding roles that include functional expertise, consulting, innovation, and other direct value-creating activities.
Data from source technologies does not mean wholesale replacement of the data repository technology, process, and infrastructures. Instead, it presents a significant opportunity to work with and shape a disruptive technology for individual environments, to ensure the highest and best use of all your digital assets.
