An Architecture for Workplace Learning Analytics (WLA) to Support Lifelong Learning in Sustainable Smart Organisations

Round 1

Reviewer 1 Report

As an introduction to workplace learning analytics (WLA), this paper presents a comprehensive and well-structured discussion of its topic. By proposing a layered WLA architecture, practitioners and researchers can improve the success of WLA implementations in corporate organizations by filling a gap in the existing literature.

The authors should consider the following revisions to further strengthen the manuscript: Provide more details regarding the specific data analysis methods utilized for focus group discussion data. Describe the limitations or challenges that may arise in implementing the proposed WLA architecture in various organizational contexts, as well as its scalability and adaptability in response to future technological advances. The importance of data governance, privacy, and security in the management of data should be elaborated on in the architecture. Furthermore, as highlighted in the literature, the authors can also discuss the human-related factors and skills necessary to achieve successful implementation of such an architecture, and more recent references are encouraged to be added to this work, as among all the 40 references only 10 of them were published after 2020. By revising the paper, some of the gaps identified can be addressed and its contribution to the field can be further enhanced.

Author Response

Please see attached file for the improvements made related to Reviewer 1 plus some general improvements made.

Author Response File: Author Response.docx

Reviewer 2 Report

The purpose of this paper is to present the design of a workplace learning analytics architecture that can be used in organisations to impact successful WLA, which is an interesting topic. However, I have following questions regarding methodological approach which needs to be taken care of by authors.

1. What is the reason that authors selected papers for review from 2011 till 2019. why not until 2023? is there no architecture after 2019, otherwise extend it 2023.

2. There was only one focus group, as discussed as one of the limitation of the study the findings may not be generalizable. Maybe at least another FGD could be done to have at least around 10 members.

3The second phase of evaluation was done by practitioner-researcher which may be biased can it be validated in FGD?

4. Why three attributes were taken only from the Information Quality dimension in the D&M IS Success Model and other dimensions were not considered.

Author Response

Please find attached the response to Reviewer 2

Author Response File: Author Response.docx

Reviewer 3 Report

The article is devoted to studying Workplace Learning Analytics architecture and finding or designing an architecture suitable for WLA.

The study and the resulting article have several major problems. In particular,

1. The chief problem is that the research question was formulated very broadly: "What architecture is suitable for WLA?" The authors don't define their meaning of "suitable", and there is no justification for choosing a single architecture for that question: many different architectures can be all suitable for WLA. That leads to the lack of clear and critical discussion on what proves that the architecture the authors present in the section 7 is indeed suitable (or more suitable than other architectures?) Section 7 is titled "Discussion" but it mostly describes the new architecture instead of the critical discussion of the authors' findings. So it all remains relatively subjective without strict definition and criteria of suitability. Please, address the issue by either re-formulating the research question or providing a concise defition and proof of suitability of your architecture.

2. In section 2, Focus Groups, which is a relatively well-known method, is discussed throughly while other methods ("pragmatic systematic approach" and "D&M IS Success Model" are described in much less detail and just given references to. Please, balance the description of your methods and describe the less commonly used methods first.

3. Among the inclusion criteria of focus group, there were subjective criteria like "Have a clear understanding of the how the learning data is pushed or pulled" What does it mean? How did you evaluate understanding and its clarity? Please provide more data on selection criteria like that. (In contrary, the selection criteria like years of experience are easily measured and non-subjective).

4. In Section 3.1. the literature review isn't reported in a systematic manner. While this is tolerable for a regular literature review providing context for a study, you wrote that conducting a literature review was a part of your study. Then this review must be performed and reported systematically: please provide the databases and search strings you used, the number of found records and duplicates, inclusion criteria and so on. If your review isn't systematic, then how can you be sure that you didn't miss any useful architecture?

5. You write that you reviewed papers published between 2011 and 2019, but your article mentions ethics clearance number obtained at the end of 2022, which shows the time of research. Why did you exclude the papers published in 2020-2022? Please, explain this or include the relevant papers in your review

6. Section 5 contains some comparison of the architectures found in literature and what is used in the company designated as ERPCo. However, it is unclear if the differences mean that the company isn't up to the stadard provided in the architecture and should improve its processes or the difference shows the deficiency in the architecture while the company does it better. It is possible that for different lines of the list in pages 11-12 the different answers apply. Please, discuss that and justify your opinion on who was better where: the company or the studied architecture.

7. The study is based on the opinion of employees of one organisation.  While they may contain important insights, one company is still one company; it is quite possible that specialists in other companies will have a different view, different goals and priorities and so on.  However, there is no critical discussion of the generalizability of the study results: which kind of companies can use them and what proves that? How different the evaluation will be if you'd take a few other companies? I don't see any data on that, so it looks just one random measurement, which is hard to use outside of the original company - which severely limits the article's impact. Please, discuss the range of companies for which your study results are valid and why do you think so. Threats to study validity must be thoroughly discussed.

8. The article has too many abbreviations defined in it,  which makes it hard to follow. Please limit the abbreviation to the most important terms.

Addressing those problems will help to increase the article's quality and its impact.

English quality is uneven in different sections of the manuscript: some of them are written well, while others have problems. Examples of problematic sentences are:  "A pragmatic systematic approach [19] and a three-phase evaluation took place (??) to meet this objective." (aproaches don't take places)

"a thorough review was conducted of papers published between 2011 and 2019" (likely misplaced "was conducted")

"set of requirements for WLA were identified" set (singular)... were (plural)

And so on.

Author Response

Please find response to reviewer 3's comments.

Author Response File: Author Response.docx

Round 2

Reviewer 2 Report

The authors have improved the paper and it can be accepted.

Author Response

Reviewer 2 was happy with our responses from Round 1

Reviewer 3 Report

The article was significantly improved. However, some issues were not addressed:

1. In section 2, only a small remark was added to describe the criteria "Have a clear understanding of the how the learning data is pushed or pulled" - the authors added "(in other words to be data literate)." Please, explain the procedure, how did you decide if your study participant was data literate or not. You should have used some procedure for that.

2. There is no discussions of threats to validity and limits of the research. Regardless of how well the results are based on the previous literature, this study studied people working in one company only. Do you claim that the results are applicable to every company all around the world? What makes you think like that? If not, what are geographical, cultural, economical (etc.) limitations of your findings? To which kinds of organizations it can be safely generalized and which kinds of organizations require further research? Scientists must be aware of limits of their knowledge and theories.

A technical moment: you added new references and rennumbered them in the article's text, but the references in the captions of figures aren't shown in red so they could have remained unchanged. Please verify that they are correct.

Author Response

Please see attached file.

Author Response File: Author Response.docx