Artificial intelligence and data science are big news in 2023. The rise of generative artificial intelligence has, of course, led to this dramatic increase in visibility. So what could happen to this area in 2024 to keep it on the front page? And how will these trends impact business?
Over the past few months, we’ve conducted three data and technology leaders surveys. Two of them attended the MIT Director of Data and Information Quality Symposium—one sponsored by Amazon Web Services (AWS) and the other by Thoughtworks. The third survey was conducted by Wavestone, formerly NewVantage Partners, whose annual surveys we have previously written about. More than 500 senior executives took part in the new surveys, with perhaps some overlap in participants.
Surveys don’t predict the future but suggest what people closest to data science and companies’ artificial intelligence strategies and projects are thinking and doing. Based on this data, here are five major emerging issues that deserve your close attention:
Generative AI sparkles but needs to deliver value.
As mentioned, generative artificial intelligence has attracted enormous attention from businesses and consumers. But does it bring economic value to the organizations that use it? The survey results show that while interest in this technology is high, its value has not yet been realized. Many respondents believe that generative AI has transformational potential; 80% of AWS survey respondents said it will transform their organizations, and 64% of Wavestone survey respondents said it is next-generation technology. The vast majority of respondents are also increasing investment in technology. However, most companies are simply experimenting at the individual or departmental level. Only 6% of companies in the AWS survey have any production generative AI applications, and only 5% in the Wavestone survey have large-scale production deployments.
Production adoption of generative AI will, of course, require more investment and organizational change, not just experimentation. Business processes must be redesigned, and employees retrained (or perhaps, in some cases only, replaced by generative AI systems). New AI capabilities need to be integrated into existing technology infrastructure.
Perhaps the most significant changes are related to data – managing unstructured content, improving data quality, and integrating different sources. In an AWS survey, 93% of respondents agreed that data strategy is critical to benefiting from generative AI, but 57% have yet to make any changes to their data.
Data science is shifting from artisanal to industrial.
Companies are feeling the need to accelerate the creation of data science models. What were previously craft activities are becoming increasingly industrialized. Companies are investing in platforms, processes and methods, feature stores, machine learning operations systems (MLOps), and other tools to improve productivity and speed of deployment. MLOps systems monitor the health of machine learning models and determine whether they are still making accurate predictions. Otherwise, models may need to be retrained using new data.
Most of these capabilities are provided by external providers, but some organizations are now building their platforms. While automation (including automated machine learning tools, which we’ll discuss below) helps improve productivity and enable greater participation in data science, data science’s most tremendous boon to productivity is arguably the reuse of existing data sets, functions, variables, and even the entire model.
Two versions of data products will dominate.
According to a ThoughtWorks study, 80% of data and technology leaders said their organizations use or consider using and managing data products. By product data, we mean packaging data, analytics, and artificial intelligence in a software product offering for internal or external customers. From concept to implementation (and continuous improvement), it is managed by data product managers. Examples of information products include recommendation systems that tell customers what products to buy next and price optimization systems for sales teams.
However, organizations view information products differently. Less than half (48%) respondents said they include analytics and artificial intelligence capabilities in their data product vision. About 30% view analytics and artificial intelligence separately from data products and use the term only for reusable information assets. Only 16% say they don’t think about analytics and artificial intelligence in a product context.
We favor a definition of data products that includes analytics and artificial intelligence because that is where data can be helpful. However, what is essential is that the organization consistently defines and discusses information products. If an organization prefers a combination of “data products” and “analytics and artificial intelligence products,” that can work, too, and this definition retains many of the positive aspects of product management. However, without definitions, organizations can become confused about what product developers should be building.
Data scientists will become less attractive.
Dubbed “unicorns” and occupying “the hottest jobs of the 21st century” for their ability to succeed in all aspects of data science projects, data scientists have seen their star power. Some innovations in data science create alternative approaches to managing essential parts of work. One such change is the increase in the number of related roles that can address data science problem areas. This expanding pool of specialists includes data engineers to process the data, machine learning engineers to measure and integrate models, translators, connectors to collaborate with business stakeholders. And data product managers to oversee the entire initiative.
Another factor reducing the need for professional data scientists is the rise of citizen data science, where quantitative people create models or algorithms themselves. These people can use AutoML or automated machine learning tools to do most of the heavy lifting. What helps citizens the most is the modeling capability available in ChatGPT called Advanced Data Analytics. With a very short prompt and a loaded dataset, it can handle almost every step of the model creation process and explain its actions.
Of course, many more aspects of data science require professional data scientists. For example, developing entirely new algorithms or interpreting how complex models work are challenges that will never disappear. The role will still be need. But perhaps not as much as before and without the same level of power and brilliance.
Data, analytics, and AI leaders are becoming less independent.
Last year, we noticed that many organizations were reducing the number of technology and data “leaders” they had, including chief data and analytics officers (and sometimes heads of artificial intelligence). This CDO/CDAO role, although increasingly common in companies, has long been characterize by short tenure and confusion about responsibilities. We don’t see the roles of data and analytics leaders disappearing; instead, they increasingly refer to a broader set of technology, data, and digital transformation functions managed by a “super-tech chief” who typically reports to the CEO. Titles for this position include chief information officer, chief information officer, and chief digital officer. Real-world examples include TIAA’s Sastri Durvasula, First Group’s Sean McCormack, and Travelers’ Moighan Lefebvre.
This evolution of C-suite roles was the focus of the ThoughtWorks survey, with 87% of respondents (primarily chief data officers, but also some technology leaders) agreeing that people in their organizations can be mainly holistic or somewhat confused. . where to go for services and questions related to data and technology. Many senior executives said collaboration with other technology-focused leaders in their organizations is relatively low, and 79% agreed that a lack of collaboration in the past has hampered their organizations.
We believe that by 2024. We will see more shared technology leaders who can create value from data and the technology professionals who report to them. They will continue to emphasize analytics and artificial intelligence because that is how organizations understand data and use it to create value for employees and customers. Most importantly, these leaders must be entirely focus on the business, be able to discuss strategy with their senior management colleagues. And translate it into systems and ideas that will make that strategy a reality.
