Automatic Literature Mapping Selection: Classification of Papers on Industry Productivity

Round 1

Reviewer 1 Report

Comments for author File: Comments.pdf

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

As the number of manuscript publications has increased rapidly, it is important to design new methods for the analysis of the publications and for extracting useful information out of the vast amount of materials.

The authors proposed four ML methods for this task. The experimental results are encouraging. But, as said in the authors’ conclusion, it would be interesting to broaden the data scope and integrating diverse evaluation metrics. Indeed, this would strengthen the argument and support of the authors’ current manuscript. It is desirable to test with more datasets in the revised version, instead of the future work.

Some formula are not displayed properly, like the one for calculating SSS.

It would also be very desirable if better and clearer graphical presentation (like Fig. 4) can be used in the revised version.

Authors should also pay attention to regular spacing between the words like:

“To streamline catego-rization, we leveraged four Machine Learning models: Multinomial Naive Bayes

        Classification (MNBC), Stochastic Gradient Descent (SGD), Support Vector Machines

(SVM), and Decision Tree Classifier (DTC). Among these, the Decision Tree Classifier”

The English is easy to follow.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

## Review summary

This work seems interesting at first reading, but I have some major concerns. The study used a familiar methodology but lacked scientific novelty and methodological soundness. The introduction section should be rewritten to present the motivation more clearly. The state-of-the-art literature is large and expressive in the examined domain, and the manuscript would benefit from introducing a separate section describing recent works more systematically. References should be extended and updated. I strongly suggest that the authors publish programming code and thus enable the reproducibility of the described approach.

## Major comments

1. Introduction: Make an explicit connection between literature mapping and a proposed machine learning task.

2. Accuracy as a measure of classification performance is not relevant in your settings. The group sizes (i.e., marginal frequencies in Figure 5) are heavily imbalanced, and I strongly suggest referring only to the F-measure as a performance measure.

3. The study design should be modified to include at least a simple sort of human evaluation. The manuscript would greatly benefit if you included a manually curated gold-standard dataset into the loop. In addition to the results presented in sections 3.1 and 3.2, at least several hundred examples should be manually evaluated.

4. You interchangeably use the terms “classification” and "categorization." Please define both terms and/or use them strictly according to the machine learning literature.

5. Section 2.6.1, in which you define the Z-score and standard deviation, is not necessary. Both concepts are well known, and it is enough if you cite any elementary statistics textbook.

6. The “Bibliometrix package from RStudio” -> RStudio is an IDE; please cite the original paper here.

Please see my comments above.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

see attached file.

Comments for author File: Comments.pdf

Author Response

Please see the attachment.

Author Response File: Author Response.pdf