New Working Paper: The Perks and Perils of Machine Learning in Business and Economic Research

On October 10, Prof. Lars Hornuf presented a new new research paper at the Faculty of Economics and Business at KU Leuven, which you can download here: SSRN.  Tom Dudda and Prof. Hornuf had a look at more than 50,000 papers from the FT50 journals and asked:

(1) How much more Machine Learning (ML) research is there recently in business and economics and which sub-areas are leading?

• In the FT50 publications between 2010 and Q2 2023 we found 1,058 applied ML studies and 223 studies that aimed to predict a variable of interest using ML.
• Adoption of ML models most frequent in information systems, marketing, and operations research/management journals.
• Adoption of ML models less often in human relations, organization, economic, and accounting journals.

(2) Do researchers honestly report a traditional benchmark model or do they simply leave it out, perhaps because it performs after all better than their fancy ML model?

• In total, 28% of predictive ML studies do not report the benchmark performance of traditional models.
• Not reporting varies by discipline, e.g., 6% in finance vs 38% in marketing.
• Reporting of traditional benchmark models is partly explainable through the use of novel data types (e.g., text, images).

(3) How much better do ML models perform in business and economics research and why?

• Most studies report an outperformance in favor of the best-performing ML model, but only 69% of the average ML model perform relatively better.
• Thus, researchers must put some effort in finding a better performing ML model.
• Using complex ML models might become inefficient (in terms of money, energy, and explainability) in cases where research questions are simple and relationships linear.

(4) Does it pay to report traditional models honestly; i.e., do these papers get more citations

• Studies that are transparent about the performance of traditional methods receive, on average, 3 more citations per year than studies that do not report results for traditional models.

© TD