Machine Learning and Business Intelligence
The purpose of business intelligence (BI) is to find relationships in one’s historical data that provide insight into one’s customers’ behavior. That is also one of the major purposes of machine learning (ML) when used in a business setting.
Need for Data That Has Clear Meaning
Both BI and ML rely on the availability of usable data. If historical data, either in a data lake or a data warehouse, is poorly documented or is of low quality, then BI and ML both become very difficult.
It is therefore essential to build agile approaches to data management into one’s development flow. See Defining a Data Strategy.
Are Both Applied Research Activities
Both BI and ML are fundamentally applied research activities, not production or development activities. They are research because one cannot predict if or when one will find exploitable relationships in the data. And if one does find relationships, developing effective BI or ML models that make is not a given, and requires deep expertise as well as a lot of experimentation.
Need for Experts
Both BI and ML require deep expertise. Business analysts who are effective at analyzing data are usually necessary to be able to leverage one’s business data. For ML, off-the-shelf models can usually be used effectively by a talented programmer, but a custom model requires an ML expert, usually with a PhD.
In addition, data about real people often contains sensitive information, and data often contains biases. There are legal implications with regard to data privacy and data biases that are very difficult to manage at a technical level, requiring experts.
Need an Effective Interface Between Research and Development
Interfacing a research activity with a development stream is challenging. Not only are the ways of working of each activity very different, but one is unpredictable whereas the other needs to be predictable.
It is usually very valuable to have some experienced people who have a working knowledge of the research area and the development area. For example, an experienced developer who has taken courses in machine learning model building will have enough understanding of the ML work to be able to have effective conversations with both the research and development teams, and serve as a bridge for facilitating discussion on how to link the two. Alternatively, an ML expert who has worked as a developer can play that role as well.
One caution is that one should not assume that ML experts are developers. They know how to program, because they need to program in order to simulate their models, but their programming knowledge is usually very narrow and not team-oriented. They rarely know the methods that professional developers use.