Methodology of Labeling According to 9 Criteria of DSM-5

Abstract

Depression disorder is a disease that causes a deterioration of daily function and can induce thoughts of suicide. The Diagnostic and Statistical Manual of Mental Disorders Fifth Edition (DSM-5), which is the official reference of the American Psychiatry Association and is also used in Korea to identify depressive disorders, sets nine criteria for diagnosing depressive disorders. The lack of counseling personnel, including psychiatrists, and negative social perceptions of depressive disorders prevent counselors from being treated for depressive disorders. Natural language processing-based artificial intelligence (AI) services such as chatbots can help fill this need, but labeled datasets are needed to train AI services. In this study we collected data from AI Hub wellness consultations and crawls of the Reddit website to augment and build word dictionaries and analyze morphemes using the Kind Korean Morpheme Analyzer and Word2Vec. The collected datasets were labeled based on word dictionaries built according to nine DSM-5 depressive disorder diagnostic criteria.

1. Introduction

Depressive disorder is a disease that causes various cognitive and mental and physical symptoms, resulting in a decrease in daily function. Severe depressive disorder can lead to suicidal thoughts as well as repetitive thoughts of death 194 [1]. The prevalence of depressive disorders in Korea is 36.8%, and four of every 10 people in Korea suffer from depressive disorders or depression, which is the highest among OECD countries 194 [2].

The fifth edition of the Diagnostic and Statistical Manual of Mental Disorders Fifth Edition (DSM-5) is the official reference of the American Psychiatric Association (APA). The DSM-5 is also used around the world, and depressive disorders are included in its classification of mental disorders. Depressive disorders are determined by comprehensively considering the nine diagnostic criteria of the DSM-5, psychiatrists’ clinical experiences, clinical interviews, psychological test results, and treatment progress.

Fear caused by negative social perceptions of mental disorders can prevent counselors, including psychiatrists, from being treated for depressive disorders. In addition, it can be difficult to intervene in a timely manner due to the lack of infrastructure to cope with depressive disorders when they are accompanied by a physical disease [3]. Accordingly, natural language processing-based artificial intelligence (AI) services that can be consulted in a timely manner are emerging [4,5]. Among them, conversational AI agents known as chatbots can help predict mental illness conditions by consulting a counselor’s textual records.

The term chatbot refers to a robot that can converse with users. Natural-language processing and machine learning are essential for chatbots to carry on meaningful and complex conversations. In this case, supervised machine learning is the primary tool, and it is essential to label datasets to ensure correct answers are used during training sessions. In addition, because perceived empathy with users can lead to a positive evaluation of a chatbot’s responses, it is effective to use empathy datasets [6].

This paper labels an empathetic dataset according to DSM-5 depressive disorder diagnostic criteria. A wellness consultation dataset provided by AI Hub [7] and a dataset obtained by crawling the depression subreddit channel of the Reddit internet forum were used. In addition, the corresponding datasets were classified and labeled through the use of a word dictionary that classifies words expressing the elements of depressive disorders according to nine criteria.

Artificial intelligence is becoming a part of society, and human–computer interaction technology is developing that automatically detects human emotions or gives appropriate responses to questions or utterances through chatbots. [8,9,10,11,12] However, there are currently no labeled datasets according to the DSM-5 depressive disorder diagnostic criteria. DSM-5, which is used in many countries, including Korea, has nine criteria for diagnosing true depressive disorder: 1. depressed mood, 2. loss of interest/pleasure, 3. weight loss or gain, 4. insomnia or hypersomnia, 5. psychomotor agitation or retardation, 6. fatigue, 7. feeling worthless or excessive/inappropriate guilt, 8. decreased concentration and 9. suicidal thoughts. If more than five of the nine symptoms, including one of No. 1 and No. 2, persist for more than two weeks, it is diagnosed as a depressive disorder. Accordingly, this paper labeled the dataset using the DSM-5 depression disorder diagnostic criteria, which are widely used by the medical community in various fields. In medicine, the accuracy of diagnosis can be improved through the use of corresponding datasets and machine-learning models, and in natural-language processing, the criteria can be used in chatbots programmed for depression counseling. For chatbots specifically, empathy datasets and actual counseling datasets can help elicit positive evaluations by revealing an empathetic attitude toward the user.

2. Previous Research

Kang Seung-sik et al. [13] analyzed the vocabulary usage characteristics of the short messaging service (SMS) character corpus, which is a spoken corpus, the Naver movie review corpus, and the Korean written primitive corpus, which is a written corpus. To measure the strength of the performance, a discriminatory performance vocabulary analysis methodology was used, and adjectives that appear mainly in the spoken and written forms were identified. In addition, a word-embedding technique was used to automatically build an emotional vocabulary dictionary based on adjectives with high performance intensity. A total of 343,603 emotional vocabulary dictionaries were automatically constructed using this technique. However, related research has not been able to manually select a large number of automatically constructed professional vocabularies.

Kim et al. [14] proposed a machine learning-based emotional-analysis system that detects user depression through SMS messages. Using an emotion dictionary of words related to depression and other emotions, search keywords were selected from text data on Twitter to build a learning dataset consisting of 1297 sentences that express depression-related emotions and 1032 sentences without them. Finally, a circulatory neural network, short- and long-term memory, and gate circulation units were compared and evaluated. Among them, a model based on gate circulation units achieved an accuracy of 92.2%. However, Kim et al. limited depression-related emotions to “sadness”.

Seo et al. [15] collected conversation datasets from 2016 to February 2019 from users who tended to be depressed and those who were not and analyzed the subject and vocabulary characteristics of the dataset. The periods before and after the depression trend were compared, and the depression trend was analyzed by period. In addition, topic modeling, simultaneous word analysis, and emotional analysis were used to identify the characteristics and differences among topics of conversation of users with respect to depression and non-depression trends and to analyze the vocabulary used. However, the Seo et al. team’s biased thoughts may have affected the outcome when classifying words associated with depression and non-depression.

Chin et al. [16] collected conversation datasets related to depressed emotions in chats between people and chatbots. They analyzed their data using text-mining techniques to identify conversation topics related to depression-related emotions. In addition, through qualitative analysis, the types of depression that users described when confiding in chatbots were categorized and classified, and differences were identified by comparing them with depression-related data on Twitter.

Płaza et al. [8] developed and validated a method to use machine learning algorithms to detect and classify customer emotions in contact center applications. This enabled companies to obtain useful information that can help improve their customer experience. The advantage of this study was that we develop and test emotion classification algorithms using data collected from real-world voice and text channels, which can help companies improve their customer experience. However, the limitations of this study are the problem of data imbalance and that the performance of some classifiers was low.

Rathnayaka et al. [9] proposed a meta-cognitive technology that leverages artificial intelligence chatbots to provide custom behavior activation and remote health monitoring, proposed a conceptual framework by synthesizing state-of-the-art research and technological advances, and dealt with the design, development, and participatory evaluation of chatbots. The advantage of this paper was that it can provide customized support through chatbots, and the limitations were that it had only been tested in small sample sizes and laboratory environments. In addition, it is necessary to verify its effectiveness in other cultural contexts.

Prottasha et al. [10] dealt with how to improve accuracy using transfer learning in BERT-based emotional analysis, and experiments were conducted using the Bangla language. Experiments showed that combining Bangla-BERT with LSTM yields 94.15% accuracy, and LSTM showed the most important results among the four word-embedding systems. This paper collected data from various sources, including Internet sites, social networking sites, online retailer product reviews, and film and book reviews, and provided useful insights into emotional analysis using transfer learning. However, this study only conducted experiments on the Bangla language, and there were limitations because it used only one language and one dataset. In addition, this study did not conduct a qualitative analysis of the results.

Graterol et al. [11] proposed a general emotion recognition framework that allowed social robots to detect and respond to human emotions using natural language processing (NLP) transformers and emotion ontology, which can improve social robot-to-human interaction and social behavior modeling.

Roca et al. [12] analyzed the performance impact of the slot tagging and training data length of joint natural language understanding models in Spanish drug management scenarios and presented design considerations for better NLU model development, but the study had limitations in that it did not consider the impact of other factors such as training data quality or pre-trained language models.

3. Related Techniques

3.1. Kind Korean Morpheme Analyzer

Morphological analysis refers to classifying sentences into morphemes and grasping the linguistic structure; it can distinguish roots, prefixes, suffixes, and parts of speech. The little morpheme analyzer [17] is a Korean morpheme analyzer that is less sensitive to spacing errors. For speed, an adjacent condition-inspection method using a mood dictionary was used, and for quality, a probability model based on the Heuristic and Hidden Markov Model was used. In the case of a heuristic model, morphemes were analyzed by considering the characteristics of the language, such as the part-verb of the previous word and the part-verb of the next word. Morpheme analysis results were then sorted to ensure the most appropriate information was obtained. A hidden Markov model is a probability model with both hidden and observed states. In the little morpheme analyzer, each morpheme is modeled as a hidden state, and the parts of the morpheme are modeled as observable. Morpheme analysis estimates the probability of the appearance of morphemes. Figure 1 depicts an example of the morpheme analysis process using a little morpheme analyzer.

3.2. Word2Vec

Word2Vec [18] is a word-embedding technology presented by Google in 2013. Word2Vec distributes the meaning of a word across multiple dimensions, expresses it as a vector, and calculates the similarity between word vectors. At this time, analysis of words that appear in similar contexts assumes that they are similar. In addition, after making the word blank, the researchers guessed the word to place in the blank using the word adjacent to the blank in the sentence and one of two methods. The continuous bag of words (CBOW) methods leaves the surrounding words and middle words blank, and Skip-Gram methods predict surrounding blank words as intermediate words. An example of the Word2Vec expression is shown in Figure 2.

3.3. Dataset Labeling and Classification

Dataset labeling provides information about the data, and dataset classification assigns data to groups. In general, machine-learning models use labeled datasets to solve classification problems. For machine-learning models to learn efficiently and perform predictions, high-quality labeled datasets are required. Datasets can be labeled manually or automatically. Manual labeling assigns datasets to experts and other categories and is often highly accurate, but it can be costly, in terms of both money and time. Semi-graphic learning combines hand-labeled and non-labeled data, which can reduce labeling costs at the expense of mislabeled or missing data.

4. Dataset

4.1. Dataset Description

“Wellness counseling data” were created by processing records of mental health counseling at Severance Hospital in Sinchon. Only data related to mental symptoms were used. A total of 5231 questions and 1034 responses were included. The “intent” column has 19 labels: depression, sadness, loneliness, anger, lethargy, emotional control abnormalities, loss, loss of appetite, appetite, insomnia, anxiety, fatigue, guilt, concentration, self-confidence, despair, suicide impulse, and anxiety.

In the wellness counseling data, the average length of a counselor’s utterance is 28 characters (minimum 2, maximum 117), and the average length of a counselor’s utterance is 31 characters (minimum 5, maximum 67). Figure 3 shows the results of visualizing words that mainly appear in each counselor’s speeches and responses as a word cloud.

Data from the r/depression subreddit channel were extracted from Reddit, a mega-social community site in the United States. In this paper, 30,000 posts and comment data from 2010 to 2016 were randomly extracted. In the “depression” subreddit, any violation of the notice and detailed rules will be deleted regardless of the number of recommendations. The notice and detailed rules are shown in Figure 4.

Users of r/depression can only ask for help with depressive disorders for themselves or someone close to them, and all answers should be sympathetic. In addition, answers to posts asking for help are only allowed in public comments, and personal answers and posts are not allowed. Responses without meaningful content are not allowed, and responders should not pose as a role model or someone with a superior perspective. Meaningless encouragement, promises that cannot be kept, and medical knowledge, advocacy, and opposition to treatments are also not permitted. In addition, it is impossible to certify self-harm methods and self-destructive content, and content that deviates from the subject or goes against the rules should not be encouraged. Finally, it is impossible to request and provide discussions, unauthorized surveys, self-promotion, and money, goods, services, etc.

4.2. Dataset Preprocessing

In the case of wellness counseling data, the “utterance” column, which involves counselors’ utterances, and the “response” column, which involves counselors’ responses, were used. After that, the missing value was deleted. In r/depression data, there is data in which two adjectives are bound by “/”. If “/” is contained, there is no clear analysis, so preprocessing is required. The way we came up with it was deletion. Thus, we deleted sentences containing adjectives on either side of a “/”, such as “depressed/anxious”. An example of a sentence is shown in Figure 5.

Translations were made using the Google Translate API. An example of an original text and the translation result is shown in Figure 6.

5. Dictionary of Expression Words

A word dictionary was created based on the nine DSM-5 diagnostic criteria for depressive disorder. Medical staff working in the Department of Mental Health Medicine collected expressions for each of the nine depressive disorder criteria and generated a dictionary, as shown in Figure 7.

The number of words in the word dictionary for each criterion is 59, 45, 44, 33, 37, 27, 45, 26, and 47. Figure 8 provides an example of the first of the nine criteria: “most of the day, almost daily depressed mood”.

6. Methodology

6.1. Morphological Analysis

Each sentence of the preprocessed data was analyzed using a little morpheme analyzer. The preprocessed word dictionary was also morphemically analyzed and stored for each of the nine criteria. An example of each result is shown in Figure 9.

6.2. Morphological Screening

Korean is an agglutinative language. In an agglutinative language, one morpheme performs a function; natural language processing in Korean is usually carried out on a morpheme basis. Korean morphemes are divided into real morphemes and formal morphemes depending on the presence or absence of lexical meanings. Morphemes with lexical meanings are called real morphemes, and morphemes with grammatical meanings are called formal morphemes. To extract only words with specific meanings, we selected the noun, the proverb, the root, and the adverb corresponding to the real morpheme. Among the formal morphemes, prefixes, which are morphemes that give new meanings to vocabulary, were also added and selected. The morpheme analysis results of the DSM-5 word dictionary can be found in Figure 10. These were applied to the part-time tag table of the little morpheme analyzer (Figure 11).

6.3. Data Classification

Each sentence of the consultation dataset was searched to determine if the words were included in the dictionary, and automatic classification was performed accordingly in a cloud-based Jupiter Notebook (made by Jupyter, New York, NY, USA) environment using a Tesla T4 (made by NVDIA, Santa Clara, CA, USA) graphical processing unit accelerator. An example of the classification process is shown in Figure 12.

6.4. Data Evaluation

In Section 7.1, it can be confirmed that the number of datasets belonging to diagnostic criteria 3, 4, 6, 7, and 8 was small. Therefore, we augmented the words, and the whole process of word augmentation is shown in Figure 13.

The similarity of words for the corresponding diagnostic criteria was therefore searched using Word2Vec. The hyperparameters of Word2Vec were set as follows. The larger the window size, the better it is to capture topic or domain information [19]. Therefore, we used five left and right words of the word as context by setting the window to 5. min_count is a parameter that specifies the frequency to be used for learning. We set min_count to 5, the default value of genism, and excluded words that appeared fewer than 5 times from learning. Finally, the CBOW algorithm with a relatively high learning speed was used by designating sg, a parameter that specifies the algorithm, as 0. Word2Vec was used to calculate word similarity and extract words filtered by the set parameters into similar words. Figure 14 shows the results of searching for words similar to “lonely”, “terrible”.

After that, similar words were visualized using TensorBoard Embedding Projector (www.projector.tensorflow.org, accessed on 20 March 2023), and secondary selection was performed. Figure 15 shows the results of visualizing words similar to ‘tough’.

Secondary selected expressions were added to the DSM-5 expression word dictionary to increase the data.

7. Result

7.1. Data Labeling

Classified data sets were labeled according to the DSM-5 depression disorder diagnostic criteria. For example, if the word dictionary for the third diagnostic criterion, “loss or gain weight without diet, loss or increase of appetite almost every day”, contained a “pig” in the counseling dataset sentence, the sentence was given a label of number 3, which means the third diagnostic criterion. Figure 16 shows the results of labeling wellness consultation data according to the nine diagnostic criteria and visualizing the frequency.

7.2. Comparing the Frequency of Each of the 9 Criteria

Figure 17 compares the labeling result of wellness counseling data before and after word pre-augmentation, and Figure 18 compares the labeling result of r/depression data from Reddit before and after augmentation. As can be seen from the figures, there was no significant difference before and after the word dictionary was augmented.

Combining the wellness consultation data and Reddit data produced the following frequency visualization (Figure 19).

7.3. Frequency Result Inference

As a result of establishing a null hypothesis that there is a difference in data before and after augmentation and conducting a t-test, the p-value was over 0.8, which was greater than 0.05. Therefore, we confirmed that there was no significant difference before and after word dictionary augmentation. No significant difference was evident in the results before and after augmentation of the word dictionary because many words were not added during dictionary enhancement. In addition, we also identified that there was no significant difference in the counseling dataset collected in this study due to the lack of data corresponding to the label.

However, criteria 1 and 2 are “most of the day, almost every day of depressed mood”, and “most of the day, almost every day of interest or pleasure in almost all daily activities is significantly reduced”. This indicates that the expression words are more ambiguous than other criteria and are often used. Compared with other criteria, the number of expressions such as “loss or gain weight without diet control, loss or increase of appetite almost every day” is small, but clear. However, diagnostic criteria 1 include “tears”, “sad”, and “disappointed”, which are ambiguous and frequently used in other contexts. As a result, we concluded that the number of data is inevitably high when the expression word is an ambiguous and widely used label.

8. Conclusions

This paper proposes a methodology for classifying and labeling datasets based on DSM-5 diagnostic criteria for depressive disorder. Wellness counseling data and Reddit crawling data consisting of empathy-related content were used to augment AI services for depressive counseling. A word dictionary built on medical knowledge was used, and we analyzed the dataset and word dictionary using a little morpheme analyzer, with the resulting dataset labeled 1 to 9. In addition, if the number of words was insufficient, the word dictionary was augmented using Word2Vec.

We deleted the entire sentence if it contained “/” in the data preprocessing process. In future work, only one of the front and back adjectives of “/” can be used or preprocessed in other ways. Reddit data are originally English data. Since we used this English data translated into Korean, the accuracy of the analysis may have been slightly reduced. In the future, it is expected that more accurate research will be conducted if Korean data that show depression are used.

A word dictionary built by medical staff and based on the DSM-5 depression disorder diagnosis criteria used by the APA was employed in this study. The proposed methodology can therefore be used in both medical care and natural language processing. Guidance learning techniques are widely used to create chatbots. Guidance learning or semi-learning can also be performed using datasets classified according to the DSM-5 depressive disorder diagnostic criteria, which are proven diagnostic criteria. In addition, through our proposed methodology, we can easily label new data to enable better learning, so we can expect excellent answers from natural language processing-based AI services such as chatbots. In addition, this approach can lead to a combination of AI and medical services. Training the model using these labeled datasets will enable early diagnosis in the case of not visiting the hospital due to negative gaze or cost problems, and it will be easy to monitor after diagnosis. If a sentence contains multiple emotions, it would be more reasonable to perform multi-label classification in future studies because even one sentence can have multiple labels.