Machine Learning in Gamification and Gamification in Machine Learning: A Systematic Literature Mapping

Abstract

Albeit in different ways, both machine learning and gamification have transfigured the user experience of information systems. Although both are hot research topics, so far, little attention has been paid to how these two technologies converge with each other. This relation is not obvious as while it is feasible to enhance gamification with machine learning, it is also feasible to support machine learning with gamification; moreover, there are applications in which machine learning and gamification are combined yet not directly connected. In this study, we aim to shed light on the use of both machine learning in gamification and gamification in machine learning, as well as the related topics of using gamification in machine learning education and machine learning in gamification research. By performing a systematic literature mapping, we not only identify prior works addressing these respective themes, but also analyze how their popularity evolved in time, investigate the areas of application reported by prior works, used machine learning techniques and software tools, as well as the character of research contribution and the character of evaluation results for works that presented them.

1. Introduction

Machine learning (ML) is “a field of computer science that studies algorithms and techniques for automating solutions to complex problems that are hard to program using conventional programming methods” [1], whereas gamification consists in “using game-based mechanics, aesthetics and game thinking to engage people, motivate action, promote learning, and solve problems” [2]. These two approaches can be combined to help attain a certain goal—a good example of which is improving user experience (UX). Already in 2015, Jordan and Mitchell have indicated that ML algorithms can be used to create personalized experiences for individuals and groups of individuals [3], whereas three years later, Hsu and Chen explored how gamification improves the user experience [4]. Applications combining machine learning and gamification thus exploit both these ways for boosting the UX improvement (see, e.g., [5]).

But machine learning and gamification can also be combined in other ways, in particular, in which one merely supports the other (or vice versa). For instance, machine learning may be applied to support gamification by adapting it in real-time based on applying machine learning techniques on user interaction data (an approach known as data-driven gamification design, DDGD) [6]. Conversely, gamification can support machine learning, for instance, in the process of elicitation of training data (see, e.g., [7]).

Despite the prevalence of both machine learning and gamification applications, surprisingly little effort has been made to map the vivid field of research on combining them. At the moment of writing these words, we are only aware of two publications having attempted this. The first one is by Khakpour and Colomo-Palacios, who investigated 32 papers extracted in a systematic approach from IEEE Xplore, ACM Digital Library, SpringerLink, and Science Direct digital libraries [8]. The identified papers were analyzed with respect to the usage area, distinguishing the usage of ML in gamification (unfortunately, only three areas were considered there: learning, personalization, and behavioral change, and their delimitation was somewhat ambiguous, considering that one application could at the same time belong to all of these areas), and the usage of gamification in ML; reported effects of using gamification in ML; aspects of gamification affected by ML; and benefits and challenges (unfortunately, aggregate data were only provided for the usage of ML in gamification, the review in the remaining dimensions was only performed on selected examples). The period covered in the review ends in 2019, omitting the last four years of research output, thus making it, to some extent, outdated.

The second review, by Schultz Garcia da Luz et al., covers a one-year-longer period (ending in 2020), and is based on querying five data sources (Google Scholar, Science Direct, ACM, IEEE, and Scopus), but is limited in both its scope, as it only dealt with research on gamification applied to education, and quantity, as the authors analyzed only five published works [9]. These five works were analyzed with respect to the game style, the area of education and its level, the game architecture, and the machine learning algorithms and techniques used.

The limitations of the two existing reviews provide a strong motivation for performing a new, up-to-date mapping of literature on combining machine learning and gamification. Partly inspired by the dimensions of analysis performed in the prior studies, partly addressing their deficiencies, we have developed 10 research questions, which we arranged into three categories: the general characteristics of identified studies (RQ1–3), the covered topics (RQ4–7), and the research contribution (RQ8–10):

RQ1.

How did the interest in the field evolve in time?

RQ2.

How is the interest in the field distributed geographically?

RQ3.

Is the research output in the field confined to specific publication venues?

RQ4.

What are the main themes of the identified publications relevant to the field and how did they evolve in time?

RQ5.

What are the human activity areas addressed by the identified publications?

RQ6.

What are the machine learning types, techniques, and software tools used in applications described in the identified publications?

RQ7.

What are the main technologies (other than machine learning) that the applications described in the identified publications are based on?

RQ8.

What is the character of the research contribution of the identified publications?

RQ9.

What are the evaluation results reported by those of the identified publications that report such results?

RQ10.

Which of the identified publications made the largest research impact so far?

The arrangement of research questions mirrors the structure of Section 3, presenting the results addressing the respective questions, which are then discussed in Section 4. Before that, in the following section, we describe how the data were collected, and how the respective analytical categories have been defined.

2. Materials and Methods

The search for the relevant publications has been performed using the Web of Science (https://www.webofscience.com/wos/woscc/advanced-search (accessed on 7 August 2023)) platform. It has been chosen for both being currently the largest of the generally accessible bibliographic databases (with over 211 million records [10] compared to over 90 million records of Scopus [11]) and ensuring the indexed publication venues meet the minimum quality standards (thus simplifying the analysis by avoiding the manual assessment of scientific quality of retrieved works, a necessary step for sources that do not observe any standards of quality).

Note that Web of Science indexes most of the publications (almost all of those published at high-quality venues) indexed by platforms used in prior research (i.e., IEEE Xplore, ACM Digital Library, SpringerLink, and Science Direct). For this reason, it was pointless to include any of them in the search.

The search term has been defined to target all peer-reviewed publications (i.e., articles, books, chapters, and proceedings papers) in English that featured the terms gamification and machine learning in their titles, abstracts, or keywords specified by their respective authors:

• ((( ((TI=(gamification)) OR AB=(gamification)) OR AK=(gamification))
  AND ( ((TI=(‘‘machine learning’’)) OR AB=(‘‘machine learning’’))
  OR AK=(‘‘machine learning’’)) )
  AND LA=(English))
  AND DT=(Article OR Book OR Book Chapter OR Proceedings Paper)

Figure 1 shows the flow of the data collection and qualification process. Although only one bibliographic database has been used, one duplicate item has been found in the results. Also, although only peer-reviewed publication types were targeted by the search query, three non-peer-reviewed publications have been found. Moreover, 29 papers mentioned both machine learning and gamification in their abstracts, but were actually devoted to only one of these two. All such irrelevant papers have been removed, leading to a total of 94 papers qualified for further analysis.

The main point of our research was the mapping of publication themes. This kind of analysis can be performed in two ways: bottom-up or top-down. In the first way, the starting point is the list of topics indicated by the respective publications’ authors themselves, possibly as keywords, which are then clustered to reveal more general themes (see, e.g., [12], especially Section 3.2 therein, for an example of following this scheme). The bottom-up approach is, however, poorly suitable for an investigation like the one performed here, where the focus is on the intersection of two research areas and the character of the relations between them, which is rarely reflected in the most frequently used authors’ keywords (which form the seeds of the clusters). For instance, in our research, the top keywords by frequency included, apart from the search keywords (Gamification and Machine Learning), the indications of the application area (Education, STEM), the supported processes (Crowdsourcing, Adaptive Learning, Affective Computing, Natural Language Processing), the supported technologies (Augmented Reality, Virtual Reality), and the enabling environments (Big Data) (see Figure 2; the terms included in the query are shown in red color).

For the reason given above, we decided to choose the top-down approach, in which the starting point is the list of themes most relevant to the research questions, which is then possibly extended if publications not matching any of them are encountered. Following the title of this paper, the two primary themes were defined as:

• Machine learning supporting gamification, where ML algorithms could be used to, e.g., customize challenges and/or difficulty level for individual users, provide high-level insights for managers or teachers based on users’ activity in the gamified system, or automatically generate content and/or challenges;
• Gamification supporting machine learning, primarily for motivating crowds to volunteer training and test data, or to assess the results of having applied ML algorithms.

Naturally, we expected there could also be papers reporting:

• Combining machine learning and gamification, e.g., describing applications that make use of both ML and gamification, but in which one is not directly supporting the other, such as introducing an e-learning system in which gamification is used to motivate the students and ML algorithms are used to profile them;
• Machine learning coexisting with gamification, e.g., describing approaches or processes in which both applications featuring gamification and those featuring ML could be used alternatively or at different stages, e.g., introducing a novel educational approach that involves the use of various support tools, some of them gamified, some of them ML-based.

During the analysis of the collected papers, we identified a substantial number of papers that could not be assigned to any of the four themes listed above but were somehow related to both machine learning and gamification. For these, two additional themes were defined:

• Using gamification in machine learning education, i.e., describing solutions that apply gamification yet do not apply ML algorithms, but their goal is to support ML education;
• Using machine learning in gamification research, i.e., reporting research that apply ML algorithms not to enhance gamified applications in any way, but to collect findings that push forward the theory of gamification and/or may be exploited in the future applications of gamification.

There is one more theme considered: Mixed, to which papers addressing more than one of the six themes listed above belong (this one is particularly useful for overview or review papers describing a number of unrelated applications of gamification and ML). All papers that fit none of these seven themes, i.e., did not cover either gamification or machine learning, were considered as not relevant to this study.

As regards the mapping of application areas, we started it with a list of a handful of high-level areas, which we extended with each area mentioned in a paper that could not be assigned to any of the existing ones. In the second phase, the similar or closely related areas were joined to shorten the list.

A similar procedure has been applied to the mapping of machine learning techniques: we started with a list of the most-known techniques extending it with other techniques found in the analyzed papers, and in the second step, similar techniques were joined (e.g., different kinds of artificial neural networks were combined to form a single category). Note that many papers mentioned the use of various methods, and in such cases, they were assigned to all matching buckets.

As for the more general distinction of machine learning types, we followed the traditional approach [13] and considered three of them (supervised, unsupervised, and reinforcement learning). As few papers mentioned various approaches, we have consequently added the “various” category.

Regarding the used machine learning software, we made no a priori assumptions and simply counted the software tools or libraries mentioned by the authors. A part of the papers reported using the software developed by their respective authors themselves—if only the name of the used programming language has been given, these were labeled as “Unspecified (name of the programming language)”, so that the instances of use of specific programming languages could be counted.

A similar approach has been applied to the mapping of the main technologies (other than machine learning) that the applications described in the identified publications were based on. Here, we noted the type of technology and also any proper names of technology or software mentioned.

For the mapping of the character of the research contribution, the following labels were defined:

• Presentation—papers reporting a new application of machine learning and gamification;
• Evaluation—papers reporting an evaluation of a machine learning/gamification application (new or existing) in any aspect (e.g., performance, usability, attaining of learning objectives in educational applications);
• Comparison—papers reporting a comparison of two or more applications of machine learning/gamification in any aspect;
• Method—papers introducing a novel method, technique, or procedure, enhancing machine learning and/or gamification;
• Literature review—papers reviewing prior work relevant to machine learning and gamification;
• Overview—papers giving a general introduction to machine learning and gamification (e.g., an encyclopedia entry or an introductory chapter of a book).

The Evaluation papers were further mapped into three categories: those reporting Positive, Negative, and Mixed results of the evaluation. While the first two categories are straightforward, the Mixed category was applied to papers reporting evaluation in several dimensions or from several experiments, of which only a part has been evaluated positively, and the remainder negatively.

3. Results

3.1. General Characteristics of Identified Studies

3.1.1. Distribution of Studies in Time

Before we examine the contents of the identified studies, we shall first look at their general characteristics, starting with their form of publication and distribution in time. Figure 3 shows how the number of relevant publications evolved in the subsequent years.

The two earliest identified relevant works were published in 2014 [14,15]. Both were conference papers, the first journal papers appearing only two years later [16,17,18]. The number of published conference papers surpassed the number of journal articles in every considered year. In the entire period, conference publications constituted almost $2/3$ of the relevant papers analyzed.

The largest growth in the scientific output has been recorded in 2018. Since that year on, the number of publications steadily kept growing, but at a somewhat lesser pace (note that 2023 is the year in progress, and complete data for it are not available at the moment of writing these words).

3.1.2. Geographic Distribution of Studies

As can be observed in Figure 4, the analyzed publications were written by authors hailing from 43 countries from all inhabited continents. The largest pool of papers has been delivered by authors from the United Kingdom (15 publications in total), followed by the USA (14 publications) and Germany (11 publications). No other country has surpassed the threshold of 10 contributed publications.

3.1.3. Publication Venues

We have found an unexpectedly high diversity among the journals in which the 35 relevant journal papers have been published: there were 32 of them, with only one journal having published 3 articles (Sensors) and another one having published 2 articles (Computer Applications in Engineering Education). The remaining journals have published only one paper each: ACM Transactions on Interactive Intelligent Systems, Applied Intelligence, Applied Sciences, Artificial Intelligence, Cartography and Geographic Information Science, Computers in Human Behavior, Computers in Industry, Education and Information Technologies, Electronics, Frontiers in Artificial Intelligence, Frontiers in Psychology, IEEE Access, IEEE Systems Journal, IEEE Transactions on Games, IEEE Transactions on Learning Technologies, Interaction Design and Architectures, Interactive Learning Environments, International Journal of Architectural Computing, International Journal of Information and Learning Technology, Journal of Information Science and Engineering, Journal of Intelligence, Journal of Medical Internet Research, Mathematical Problems in Engineering, Multimedia Tools and Applications, Physiological Measurement, Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies (IMWUT), Schizophrenia Research-Cognition, Simulation & Gaming, Smart Cities, and Sustainability.

3.2. Topic Analysis

3.2.1. Publication Themes

As can be observed in Figure 5, in the analyzed set, there are many more reports on gamification supporting machine learning (39%) than machine learning supporting gamification (19%). Papers reporting combining machine learning and gamification constituted 16% of the analyzed set. A little above $1/10$ of the analyzed papers were devoted to using gamification in machine learning education, whereas a little less than $1/10$ of the analyzed papers were devoted to using machine learning in gamification research. Only 3% of the analyzed papers described machine learning coexisting with gamification, and the same share was taken by papers mixed in character.

If we look at how the distribution of publication themes evolved in time (Figure 6), we can see that gamification supporting machine learning has been the dominant theme in all years since 2017 with the exception of 2020. Since 2018, we observe a growth of interest in machine learning supporting gamification (we remind the reader that the data for 2023 are incomplete). The theme of combining machine learning and gamification had low-to-moderate popularity with the exception of two years: 2016 and 2020, when it peaked. The theme of using gamification in machine learning education peaked in 2018 and 2022, whereas the theme of using machine learning in gamification research peaked in 2016. There were no notable peaks of interest for the remaining themes.

3.2.2. Areas of Application

Analyzing the collected publications, we were able to determine 16 activity areas targeted by the application of machine learning and gamification described therein (note that some papers addressed more than one area). As can be observed in Figure 7, the most popular area was education (addressed by over 40% of the papers in the analyzed pool), followed by research and development (a little above $1/4$ of the papers), and healthcare (about 23% of the papers); no other area has been addressed by more than seven papers.

Below, selected examples of applications of gamification and machine learning in areas for which at least two applications have been identified are presented.

An interesting example of an application in the area of education (yet also Augmented Reality) is ARsinoe, which combines gamification and machine learning in an Augmented-Reality mobile app helping K12 students in studying Egyptian culture and history, and in the learning of languages with non-Latin writing systems [19].

In the area of research and development (yet also pertaining to healthcare), an interesting application of gamification and machine learning has been developed within the SUITCEYES project in the form of an intelligent assistive wearable empowering people with deafblindness, helping them recognize faces and detect objects [20].

Another example of an application targeted at healthcare worthy of mention is GameAAL, a system for monitoring the elderly daily life activities and providing neurocognitive stimulation games, promoting social interaction and mobility of its end-users using gamification and machine learning techniques [5].

Among the several applications dedicated to Human Resources Management (yet also pertaining to R&D), one particularly interesting is the one described by Liapis et al., in which Deep-Reinforcement-Learning agents emulate human-like behavior based on the OCEAN personality traits model, navigating, interacting, and solving tests in a three-dimensional online escape room [21].

An example of an application in the area of entertainment is provided by Parekh et al., who developed an automatic generator of quiz games based on a given topic, leveraging the DBpedia database and using machine-learned ranking algorithms for recommending similar resources given a particular resource, at the same time serving the purpose of cleaning the publicly maintained data repositories of DBpedia and Wikipedia [22].

As regards the area of UX improvement, 360-MAM-Affect aims at this goal in the context of a recommender system for educational video content, utilizing ML for sentiment analysis and gamification techniques to increase user participation with digital content [23].

In the area of environment protection and sustainability, the survey on smart grid mobile apps by Chadoulos et al. covers not one but five applications combining gamification and machine learning with an aim to lower and shape energy consumption patterns to achieve peak load reduction, load smoothing, and hence carbon emission curtailment [24].

Regarding the cybersecurity area, Ashktorab et al. investigate the problem of backdoor attacks consisting in creating a vulnerability in a machine learning model by “poisoning” the training set by selectively mislabeling images containing a backdoor object. They introduce the Backdoor Game, in which users can interact with different poisoned classifiers and upload their own images containing backdoor objects in an engaging way. Thus, obtained results can be helpful in determining the effectiveness of different backdoor objects and increasing the safety of future AI systems [25].

The Think!First framework targets both Customer support/Recommender system and Environment protection/Sustainability areas, as it aims at encouraging online shoppers to behave in a more targeted and environmentally conscious manner while purchasing goods [26].

Sensitive Assembly is an application in the area of architecture, which takes the form of a wall from which players are asked to remove and replace building blocks to create windows, challenging its stability. Machine-learning techniques are employed to predict an approaching collapse [18].

An example of an application aimed at the area of Augmented Reality, regardless of its context, is described by Ogi et al. It addresses the problem of the need for an AR marker or location information by replacing it with a machine-learning system fed with learning data automatically collected from the users with the help of gamification [27].

Map Safari is an example of an application in the area of Geographic Information Systems, which proposes a machine-learning-supported workflow making use of gamification to encourage volunteers to sustain the contribution of map data for humanitarian organizations without degrading their quality [28].

3.2.3. Applied Machine Learning Methods

Over $2/3$ of the analyzed papers that reported the application of machine learning of a specific type indicated supervised learning, whereas over $1/5$ of them indicated unsupervised learning, with merely 6% indicating the use of reinforcement learning, and half of that share of papers referring to various types of machine learning (see Figure 8).

The machine learning technique most frequently applied in the analyzed works was artificial neural networks of various kinds, with the second place taken by random forests and extremely randomized trees. However, the use of many other techniques has been reported as well (including ensemble algorithms), yet with much lesser frequency. The ten ML techniques that were most frequently reported as used are depicted in Figure 9.

Only less than 1/3 of the analyzed papers specified the used ML software tool or library. The retrieved information is visualized in

Table 1. Used machine learning software.

ML Software	Number of Publications
scikit-learn	10
TensorFlow (including 1 with Keras), Unspecified (Python)	3
Grasshopper (including 1 with Octopus add-on), WEKA	2
Azure Machine Learning, CatBoost, caTools, e1071, GATE, Google prediction API, klaR, nnet, Py-Feat, PySyft, Snatchbot, XGBoost, Unspecified (R)	1

. As can be observed, there is a clear leader: scikit-learn library [29], reported to be used by 10 papers. The second-place TensorFlow [30] was mentioned merely by three papers. Overall, 18 distinct ML software tools or libraries have been reported as used in the analyzed papers. In terms of programming languages used, Python is the clear leader: its use has been reported in seventeen papers vs. six reporting the use of R and four reporting the use of Java.

3.2.4. Used Technologies

In Figure 10, one can see the main technologies (other than machine learning) on which the applications described in the identified publications were based: it includes only those technologies that were mentioned by more than one paper. As can be observed therein, the most often mentioned was web technology, followed by mobile technology (both surpassing the threshold of 20 publications). The only proper name of a technology that was mentioned in more than one paper of the analyzed set was Unity (by 10 publications). The other two technologies that made it to the chart were augmented reality (AR) and virtual reality (VR). Note that the technologies are not disjunct (the same application can be based on web technology, virtual reality, and Unity at the same time).

3.3. Research Contribution

3.3.1. Character of Contribution

A lot of novel applications of machine learning and gamification have been introduced in the analyzed literature (about $2/3$ of the considered papers included this element). About 40% of the analyzed papers reported results of any kind of evaluation of a machine learning/gamification application. About $1/3$ of the analyzed papers introduced a novel method, technique, or procedure, enhancing machine learning and/or gamification. Only nine papers included a comparison of different machine learning/gamification applications (see Figure 11).

3.3.2. Evaluation Results

As shown in Figure 12, among the 38 papers that included the evaluation of machine learning/gamification applications, the vast majority (87%) reported positive results. Only 3% reported negative results, with the remainder reporting neutral or mixed results.

3.3.3. Research Impact

The most common measure of a paper’s research impact is the number of citations it received (see [31] and works cited therein). In

Table 2. Top 10 most cited works in the analyzed dataset.

Ref.	Title	Cited
[32]	Interactive machine learning: experimental evidence for the human in the algorithmic loop: A case study on Ant Colony Optimization	99
[33]	Kids making AI: Integrating Machine Learning, Gamification, and Social Context in STEM Education	27
[34]	A virtual environment for learning computer coding using gamification and emotion recognition	23
[16]	Early Prediction of Student Profiles Based on Performance and Gaming Preferences	23
[35]	A process for designing algorithm-based personalized gamification	22
[17]	Knowledge discovery of game design features by mining user generated feedback	22
[36]	Impact of Gamification on Code review process—An Experimental Study	19
[37]	Gamification and Machine Learning Inspired Approach for Classroom Engagement and Learning	12
[38]	Toward Personalized Adaptive Gamification: A Machine Learning Model for Predicting Performance	12
[39]	Miniaturized Pervasive Sensors for Indoor Health Monitoring in Smart Cities	11

, we list ten works of the analyzed set that have received the highest number of citations.

Interestingly, the top ten cited works give a good outline of the diversity of the whole analyzed set in terms of the considered themes, as they include papers describing gamification supporting machine learning (two papers), machine learning supporting gamification (two papers), combining gamification with machine learning (one paper), using gamification in machine learning education (one paper), as well as using machine learning in gamification research (a staggering four papers). Compared to Figure 5, one can notice an over-representation of the last theme among the most-cited papers, primarily at the expense of the theme of gamification supporting machine learning, which was addressed by the largest share of papers in the analyzed set; nonetheless, the top-cited paper belongs to this theme, as it describes how a human interaction performed via a gamified interface can improve the results of Ant Colony Optimization [32]. The second-cited paper represents the theme of using gamification in machine learning education, describing how open-ended gamified challenges consisting in an automatic classification of mango fruits helped to make the concept of machine learning accessible to Thai middle-school students having little or no computer engineering background [33]. The third-cited paper also deals with education, but has been assigned to the combining gamification with machine learning theme, as it describes a virtual environment for learning computer programming featuring both gamification elements used for motivating the students and machine learning techniques used for recognizing students’ emotions, so that the system could respond to them adequately [34]. The fourth-cited paper addresses the area of education as well, yet this one belongs to the theme of using machine learning in gamification research, as it describes how machine learning algorithms can be applied to classify students attending a gamified college course into one of four types, characterized by different levels of performance, engagement, and behavior with almost 80% accuracy [16]. Also, the most-cited paper belonging to machine learning supporting gamification theme (taking the 8th spot) has its focus on education, as it describes using machine learning in the form of an adaptive neuro-fuzzy inference system to improve the distribution of rewards in a gamified framework addressing the issue of disengagement of higher-education-level students during classroom learning [37].

4. Discussion

The key observations based on the results presented in the preceding section are as follows:

• The research combining the use of machine learning and gamification is a vivid and still growing field as indicated by the rising number of papers published each year since at least 2018. [RQ1]
• This research is not limited to a handful of countries, and even though there are leaders, as for now, they have not achieved a substantial advantage against the rest of the countries in terms of the research quantity. [RQ2]
• The numerical prevalence of conference publications over journal articles is an expected phenomenon in a quickly developing research field, as conference publications usually have a much shorter submit-to-publish time than most journals. [RQ3]
• There is not a single scientific journal that could be considered by authors writing on combining machine learning and gamification as their default venue of publication. This diversity in journal selection can only be partly explained by different application areas addressed by the analyzed papers. We hope this observation will motivate members of the gamification/machine learning research community to propose a new thematic section or Special Issue to attract future publications in this vein and thus create a hub for sustained scientific discussion. [RQ3]
• There was a considerable research interest identified in five of the six considered themes (and few publications mixing more than one theme). Gamification supporting machine learning was the most frequently addressed theme for almost the entire period considered. The second in popularity was machine learning supporting gamification, which has gained a lot of interest in the most recent years. The themes of combining machine learning and gamification, using gamification in machine learning education, and using machine learning in gamification research (which took the subsequent spots in popularity ranking) were characterized by subsequent ups and downs with no clear trend emerging. [RQ4]
• The research on the application of machine learning and gamification is not confined to a single area of activity: in fact, we were able to identify 15 distinct areas targeted by the described applications. While finding education to be the most popular of them was predictable, much less so were the areas that took the subsequent spots: research and development, and healthcare. Nonetheless, there are many areas that could benefit from applications combining machine learning and gamification, yet were omitted (e.g., software development) or under-represented (e.g., tourism), suggesting the directions of future research in this vein. [RQ5]
• The papers reporting the application of supervised learning have a strong numerical advantage over those reporting the application of unsupervised learning, with an even lesser share indicating the use of reinforcement learning. Artificial neural networks of various kinds were most often indicated as the applied machine learning techniques, though they are far from dominating the scene (15% of analyzed papers), with a gamut of other techniques also in use. The most typical software environment for the application of machine learning and gamification was Python (indicated in 17 papers out of 29 that provided such data) with scikit-learn library (indicated in 10 papers). [RQ6]
• The applications of machine learning and gamification relied slightly more often on web than mobile technology, and (with a somewhat less frequency) augmented reality than virtual reality. The only technological platform whose name has been mentioned by more than one paper (in fact, 10 of them) was Unity. [RQ7]
• The analyzed literature brought a considerable scientific contribution, primarily consisting in the introduction of new applications of machine learning and gamification, and their evaluation, but also in the introduction of new methods, techniques, or procedures, enhancing machine learning and/or gamification. Nonetheless, further progress in the development of the field requires standardization of evaluation dimensions and scales as proposed by the GATUGU framework [40]. [RQ8]
• The combination of machine learning and gamification was found to be effective in multiple applications, as indicated by the vast majority of papers that reported positive evaluation results. This, however, is not enough to confirm that the vast majority of machine learning and gamification applications are effective, as the reported results may be affected by publication bias, with the researchers and journals more inclined to publish positive results than negative ones and statistically significant ones over non-significant ones [41]. [RQ9]
• The scientific discussion regarding the combination of machine learning and gamification still seems to be in an early stage of its development, as only one publication of the analyzed set ([32]) received almost 100 citations (according to Web of Science), and only nine others surpassed the threshold of 10 citations. This is also indicated (and, to some extent, caused at the same time) by the small number of papers dedicated to comparing and reviewing existing applications—this, however, is naturally limited by the wide spread of areas at which the described applications were addressed. [RQ10]

5. Conclusions

In the presented study, we investigated the existing research on applications involving both machine learning and gamification. Despite both of them being hot research topics, there was much less than adequate attention paid so far to how these two technologies converge with each other.

Our study revealed a substantial literature on this topic, differing in character, applied methods, and the addressed areas of application. We have managed to analyze it considering a number of dimensions, helping to map the achievements attained so far.

On the strategic level, the presented results confirm that the combination of machine learning and gamification has proven effective in numerous cases, encouraging further research on its applications. Looking at the detailed results, they not only indicate the beaten paths (in terms of addressed problems and verified tools and methods) which can be easy to follow for both researchers and practitioners in their future projects involving machine learning and gamification, but also indicate the uncharted research directions (similarly, in terms of problems remaining to be addressed and methods that have not been tried yet) still waiting to be explored.

This work shares its limitations with all research based on the analysis of prior work. The main threats to its validity lie in (1) the selection bias—with possibly important relevant research existing outside of the Web of Science database and/or not revealing gamification and machine learning in its title, abstract, or keywords, and (2) the publication bias—as a large part of applications involving gamification and machine learning may be omitted in the scientific literature. Nonetheless, despite its limitations, the presented study is an important step forward, updating the knowledge of the state of this relatively under-researched area, and setting up a base for further research.