Predictions about data for 2023 and beyond. End of the year: it’s the time for predictions. Let’s have a look at some predictions regarding data. There are many predictions for Machine Learning, Deep Learning, and AI – explainability, professionalisation, and automation are often covered. The article lists some other topics that are hot for 2023 and beyond.
Data Management predictions 2023
Some data management predictions for 2023.
Data Engineering Teams Will Spend More Time On FinOps / Data Cloud Cost Optimisation
The cloud is mainstream, and the costs are increasing. Storage may be cheap, but data-intense workloads will cause high costs, especially if there is an estimated waste of around 30%. Cost optimisation is required.
IMO, cost optimisation is just putting out a place: Proper data architecture is necessary – a cost optimisation afterwards can only minimise some pains.
Data contracts move to early-stage adoption
Collaboration and sharing of data require proper data contracts. A one-time data delivery might be easy, but a continuous data pipeline is required with defined SLAs, agreed schema changes, known data quality, etc.
Adoption of data fabric and data mesh accelerates
Data fabric and data mesh are two approaches to managing distributed, decentralised data. Data fabric is a stack of data management technologies connecting endpoints on a network. The data mesh is a mindset and process to manage data within domains and share it across domains. For more information, see tdwi.org or dbta.com.
BARC Data, BI and Analytics Trend Monitor 2023
The BARC Data, BI and Analytics trend monitor contains a comprehensive description of various trends for 2023 compared to the last years. The Top 3 trends are:
- Master data/DQ management: Correct decisions can only be drawn if trustworthy, reliable, and consistent data are available.
- Data-driven culture: Data is the key driver for decisions. Data sharing and data collaboration are a must in contrast to keeping data in sealed silos.
- Data governance: An overall data governance approach is necessary compared to focusing on applications only. Data ownership, for example, must be defined for data as a whole and not for individual systems.
All trends are shown in the diagram below.
New Vantage Partners Data and AI Executive survey
The executive survey runs yearly since 2012 with data from responses from mainly CDOs and CIOs. Some takeaways from the study f0r 2023:
- Becoming data-driven and building a data culture remain aspirational objectives for most organizations
- Cultural factors continue to be the greatest obstacle to delivering business value from data investments
- Companies continue to fall short in attention and commitment to data ethics policies and practices
Primary areas of investment in 2023:
- Data modernization
- Data products
- AI/ML
- Data quality
- Data Mesh/Fabrics, and Data Literacy follow as lower priority
All surveys can be found on the overview page.
Gartner: 100 Data and Analytics Predictions Through 2026
The Gartner webinar „100 Data and Analytics Predictions Through 2026“ contains predictions for the next years on a higher level. The screenshot below summarises data management solutions.
The data landscape will continue to be distributed across different vendor clouds and on-premises. Gartner expects that by 2025 50% of the enterprise-managed data will be created and processed outside of cloud and classical data centre infrastructure. Edge use cases will have an increasing usage of Machine Learning, up to 65% in 2027 compared to 10% in 2021.
The Adoption of Machine Learning Will Resemble the Adoption of Databases
No prediction for 2023, but a long-term prediction contains the article „The Adoption of Machine Learning Will Resemble the Adoption of Databases“.
The article shows different periods of adoption of databases. Today (nearly) every application has a database. Database know-how is essential for every software engineer. If the adoption of ML takes a similar path as databases, every software engineer will need to understand the fundamentals of how to work with data. But adoption of ML may not take that long. Data competency will become the most important expertise in 2030, according to an article in WiWo.
ChatGPT & GPT-4
ChatGPT is part of The Verge’s predictions „The use of ChatGPT in education will spark a national conversation about AI“, and Forbes asks, “Does GPT-4 come pretty soon?“. I‘m giving lectures on data management and Data Warehouse
at Cooperative State University DHBW. Therefore, I observe ChatGPT with a high interest: what about seminar papers? Will such papers still make sense, or will students outsource the complete work to ChatGPT?
ChatGPT may help to produce some basic parts if the user is able to ask really good questions. But there is more needed in a good paper that will not be covered by ChatGPT (yet), e.g.:
- structuring the paper having a golden thread
- critical thinking about the given topic
- bringing in new ideas and aspects
Finally, let’s ask ChatGPT about the top 3 data management predictions for the next years. Note that ChatGPT is currently trained without the latest data for 2022.
