A Modified Quad Q Network Algorithm for Predicting Resource Management

Abstract

As the resource management systems continues to grow, the resource distribution system is expected to expand steadily. The demand response system enables producers to reduce the consumption costs of an enterprise during fluctuating periods in order balance the supply grid and resell the remaining resources of the product to generate revenue. Q-learning, a reinforcement learning algorithm based on a resource distribution compensation mechanism, is used to make optimal decisions to schedule the operation of smart factory appliances. In this paper, we proposed an effective resource management system for enterprise demand response using a Quad Q Network algorithm. The proposed algorithm is based on a Deep Q Network algorithm that directly integrates supply-demand inputs into control logic and employs fuzzy inference as a reward mechanism. In addition to using uses the Compare Optimizer method to reduce the loss value of the proposed Q Network Algorithm, Quad Q Network also maintains a high accuracy with fewer epochs. The proposed algorithm was applied to market capitalization data obtained from Google and Apple. Also, we verified that the Compare Optimizer used in Quad Q Network derives the minimum loss value through the double operation of Double Q value.

1. Introduction

Enterprise production systems are attracting serious concern across the economy due to their negative impacts on the industrial environment and market change. The global economy is experiencing an unprecedented demand for resources to seek for new investments with reinforcement and expansion of smart factory infrastructures and large adoption of renewable industrial resources [1]. The global economy is experiencing an unprecedented demand for resource seeking new investments for the reinforcement and expansion of smart factory infrastructures and the large adoption of renewable enterprise resources [2].

As a result, companies around the world are experiencing continuous global restructuring, enacting basic rules and laws for development direction and business expansion. A.I. programs are being introduced in some of the smart factories as resource options for reducing the supply of enterprise during certain periods of time balance supply and demand [3].

However, due to various factors, the number of data type parameters and the accuracy of artificial intelligence models are clearly limited for introducing them into the field of practice [4]. Lack of quantity of datasets and errors in datasets as representative thresholds presents a serious obstacle to learning artificial intelligence. In this paper, we chose reinforcement learning as a solution to the problem of deficient datasets. The reward system for reinforcement learning varies from one learning system to another by weight [5]. We used an applicable DQN for fuzzy logic in this paper to demonstrate the uncertainty of reward schemes. Although the fuzzy logic was characterized by high levels of uncertainty, complexity, and nonlinearity [6], it was able to solve real-world uncertainty by simulating human experience in the form of rules [7]. Fuzzy logic was introduced by Zadeh. It is a mathematical discipline used to express human reasoning in terms of rigorous mathematical notations. Human beings can make decisions even in the presence of imprecise and incomplete knowledge. Fuzzy logic can facilitate the approximation of human reasoning, beginning with a set of user-supplied human actions. It then builds rules [8]. The fuzzy system can convert these rules into their mathematical equivalents. Additional benefits of the fuzzy logic include its simplicity and its flexibility that can aid in solving problems consisting of imprecise and incomplete data. It can also be used to model nonlinear functions of arbitrary complexities [9].

The potential of fuzzy logic has been explored in numerous fields of artificial intelligence, including signal processing, speech recognition, aerospace, robotics, embedded controllers, networking, business, and marketing. Moreover, fuzzy logic in reinforcement learning has been shown to be a promising technique due to its ability to efficiently facilitate the combination and evaluation of various parameters [10].

However, the reinforcement learning theory was mainly directed at improving the accuracy of the reward value, and there was insufficient evidence to minimize the loss value. The concern with these loss values is often associated with Overfitting issues, which creates further issues. In this paper, we find a solution to the problem when deriving the Loss value from the Double Dueling DQN algorithm. The output of the proposed Quad Q Network algorithm is derived from the Compare Optimizer task, which reduces the loss value. We also demonstrate our performance experimentally using two Sample Data, Total Resource on Google and Apple, and mathematical proofs of the derivation of Double Q value.

The rest of the paper is presented as follows. Section 2 describes previous researches in this field. Section 3 discusses the fundamentals of Dueling Deep Q Network and Double Dueling Deep Q Network as well as the proposed Quad Q Network. Section 4 presents the proposed experiment and results. Section 5 presents a summary of this work and future directions.

2. Literature Review: Related Work

2.1. The Latest Trends in Fuzzy Logic and Reinforcement Learning

In recent years, reinforcement learning has shown the ability to achieve good results in games such as fuzzy fields and robotic manipulators. It has become increasingly popular in the field of motion planning due to its characteristics of not requiring much labeling data. Fuzzy systems have found applications such as control engineering [11], medicine [12], and autonomous agents [13]. In recent years, some researchers have tried to combine reinforcement learning and fuzzy control to address several complex issues in the field of robot control [14]. Fuzzy systems typically have several control parameters that are outputted from the system. These parameters are then transformed into exact numerical values by diffusing to deal with dependencies between outputting different commands, thus providing a solution to improve the accuracy of predictions in Q-learning [15]. Deep reinforcement learning, which employs neural networks to infer state values, has received a great deal of interest.

It is capable of dealing with large complex unknown conditions [16,17,18,19], including very large state and activity spaces. Deep reinforcement learning has achieved numerous breakthroughs in complex spectrum access in recent years owing to its inherent advantages. The works in [20] may be regarded as the pioneer work in the investigation of Deep reinforcement learning. It proposes a dynamic channel access scheme based on the Deep Q Network with experience replay [21]. The user selects one of the M channels to transmit its packet in each slot. Since the user only learns the channel state after selecting it, the related optimization decision problem can be expressed as a partially observable Markov decision method. Deep Q Network can promptly look for the best approach based on experience [22,23,24], thus it is commonly used in a variety of enterprises. Deep Q Network [25] is being investigated to solve both complex channel access and interference management. The Deep Q Network in [21] continues to pursue the learned policy over time slots and avoids learning a suitable policy.

Actual Reinforcement Environments Learning in the channel research is complex, and requires retraining of the Deep Q Network. To fix this problem, an adaptive Deep Q Network scheme [26] is proposed, which evaluates the cumulative incentives of the current policies over time. When the incentive falls below a certain level, the Deep Q Network is retrained to find new good policies. The simulation results show that when the channel state change, the scheme can detect it and begin relearning to achieve high returns. A Deep Reinforcement Learning system based on actor-critics is suggested, which can improve performance in a large number of channels [27]. Most of the methods listed above are limited to instances of only one user rather than multi-user or multiple channel setting, where collisions are common. Taking this into consideration, a fully distributed model [28] is used for users to perform dynamic multi-user spectrum access. For Dynamic Spectrum Access, a deep multi-user reinforcement learning method [29] is developed. This method can effectively represent the overall complex environment while avoiding the high computational costs incurred by the broad state space and inefficient observability. The spectrum access scheme proposed in [30] is based on Q learning and can assign idle spectrum holes without interaction with users. proposes multi-agent reinforcement learning based on Q learning, which can effectively avoid interference and achieve good sensing performance. Each radio in this model considers the actions of other radios to be a part of the environment [31]. Each radio tries to avoid transmissions from other wideband autonomous cognitive radios while also avoiding a jammer signal that sweeps through the entire spectrum band of interest. However, it is more difficult to achieve a stable policy in a volatile and complicated climate. To achieve stable policies in complex environments, research to reduce the loss value is needed, research in this area is growing [32].

2.2. Introduction to the Algorithms of Deep Q Learning

Deep learning network consists of numerous float values, weights, and biases [33]. These values determine the deep learning output, and the initialization process is required because these values are not defined before the learning begins. Although the most commonly used method is to initialize with random values, Glorot Initialization allows one to create an initial weight value with more stable results. This method is also called Xavier Initialization because it was proposed by Xavier Glorot. In Deep Q Network (DQN), the idea of target network is applied. If one looks again at the expression that updates the Q value in Q-Learning, one can see formula $Q(s,a)=Q(s,a)+\alpha (R+\gamma maxQ(s^{\prime },a^{\prime })-Q(s,a))$. The DQN has an idea called target network [34]. In the formula that updates Q value in Q-Learning, the $\alpha$ represents the learning rate. In the update formula, Q(s,a) is modified by multiplying $R+\gamma maxQ(s^{\prime },a^{\prime })-Q(s,a)$ by $\alpha$. If $\alpha$ is 0, then $Q(s,a)$ will not change, and if the $\alpha$ is 1, then $Q(s,a)$ will be $R+\gamma maxQ(s^{\prime },a^{\prime })$. Because $\alpha$ is usually a value between 0.0 and 1.0, $Q(s,a)$ tends to $R+\gamma \left(maxQ(s^{\prime },a)\right)$ as the learning continues. Here, the problem of using network weights with the same goal as the current value arises if weights and biases of the Q Network are expressed as $\theta$. The weight of the Q Network changes each time one learns. Goals change as well, thus, one might have difficulty in converging to a certain value. Therefore, we need to create a separate network with the same structure as the Q Network called target network, and mark it with a symbol ${\theta }^{-}$, as in the formula $Q(s,a;\theta )\to R+\gamma maxQ(s^{\prime },a^{\prime };{\theta }^{-})$.

The target network retains a fixed value during the original Q Network learning for a few times and periodically resets it to the original Q Network value [35]. This can be an effective way of learning because Q Network can be closer to a fixed target network. DeepMind’s David Silver cited three major improvements after on the DQN, which he has presented to Nature in his fourth talk, “Deep Reinforcement Learning”. They are Double DQN, Prioritized replay, and Dueling DQN. This paper introduces about Double DQN and Dueling DQN with direct calculation change. Double DQN was originally an algorithm using two Q Networks to improve the problem of Q-Learning in 2010, before DQN came out. Double Q-Learning shows that Q-learning is sometimes poor because Q values tend to be overvalued. If one looks closely at the $maxQ(s^{\prime },a^{\prime })$ in the original Q-Learning expression, one can see that this value, given the status s’, selects a’, which has the highest Q value in the Q Network, and multiplies the Q value by gamma to create a target value for $Q(s,a)$.

However, the problem is that one has to use the same Q Network when selecting and importing Q values [36]. The selected Q value is usually large value. Once it is imported, the Q value goes in an increasing direction. However, if the Q value becomes larger than necessary, the performance of the Q Network will be greatly reduced. To prevent this, the first method proposed by Hadoopan Haselt is to use two Q Networks [37]. The key to Double DQN is that these two Q Networks can help prevent unnecessary growth of Q values. As we can see above, DQN already had the idea of target network.

The DQN of formula $Q(s,a;\theta )\to R+\gamma maxQ(s^{\prime },a^{\prime };{\theta }^{-})$ is replaced by the Double DQN of formula $Q(s,a;\theta )\to R+\gamma maxQ(s^{\prime },argmaxQ(s^{\prime },a^{\prime };\theta );{\theta }^{-})$. Once one action is selected, using the existing Q network target network, Q value is obtained. Dueling DQN is the result of the network before one prompts for A value Q. A and V are then divided by the idea back together. The important thing is that one can calculate the V here. For each state by calculating the V, Q should be added as $Q(s,a)=V\left(s\right)+A(s,a)-\frac{1}{\left|A\right|}{\sum }_{a}A(s,a)$.

If you obtain a Q value, the Q value of the state with a higher value becomes a higher value, and the Q value of the state with a lower value becomes a lower value. Therefore, the agent moves to a state led by a high value and avoids the behavior of giving a low value. The idea of separating information about the value and the behavior has been similarly applied to the Quad Q Network algorithm presented in this paper. Unlike the Double DQN above, Dueling DQN changes the network structure. It is the same as combining into an output layer, which selects the behavior at the end. V and A are calculated separately in the middle. Finally the Q value is calculated to select the behavior.

3. Description of the Quad Q Network Algorithms and Functionalities

In the Deep Q Network, the idea of target network is the key. If one looks at the expression that updates the Q value in Q-Learning, it’s the same as $Q(s,a)=Q(s,a)+\alpha (R+\gamma maxQ(s^{\prime },a^{\prime })-Q(s,a))$. In the Deep Q Network, the target is $(R+\gamma maxQ(s^{\prime },a^{\prime })-Q(s,a))$. The idea of Dueling DQN is to divide the result value of the network by V and A before obtaining the Q value and then merge it again. Figure 1 shows the combination of V (value function) and A (advantage) at the bottom.

The concept of Dueling DQN starts with splitting Q value into two. The existing Q value refers to the value when an action is taken at a given state, which is divided into two terms. One may wonder why we first divide then merge later. In order to see where the agent is interested in obtaining future rewards from a value stream perspective, to maximize future rewards, the agent ends up on the way to score and resource forecasting and resource forecasting. Based on the advantage stream, if no action is taken right away, one’s reward is affected. Thus it is necessary to pay attention to the preceding value. If you dualize Q-learning into two streams, one can establish which state is valuable without having to learn the results by trying to act on each state. In a normal DQN, the agent must calculate the Q value for each action in a state when the value function is bad. Calculation of the Q value is done even in situations where the state is different or not best to get a reward. Despite performing all actions possible in the states, a bad results can be obtained. The reason for dueling is because normal DQN calculates Q-learning with state and action pairs, If either of these values is bad, then calculating the Q value with them will only prevent us from learning the optimal policy, thus, the priorities are re-arranged priorities. This will reduce unnecessary computational time and allow learning with slightly higher quality samples. Figure 2 below shows a schematic diagram of the Dueling DQN algorithm written as a control in this study.

In the Figure 2, $q_{\theta s^{\prime }}$ assumes value as the following array (1), calculated from Value stream and Advantage stream.

$$q_{\theta s^{\prime }}=\left[\begin{array}{cccc}4.1& 12.4& 3.1& 2.1\\ 3.7& 3.2& 14.2& 1.0\\ 10.2& 2.1& 1.2& 7.2\end{array}\right]$$

(1)

In addition, in Figure 2, the $argmax\left(q_{\theta s^{\prime }}\right)$ value can be presented as in the following array (2) values in terms of the weight for each row.

$$argmax\left(q_{\theta s^{\prime }}\right)=\left[\begin{array}{ccc}1& 2& 0\end{array}\right]$$

(2)

These Algorithms can fully play a role in increasing the accuracy of the existing Deep Q Network process. In addition, existing algorithms have the advantage of being easy to apply to Double Deep Q Network algorithms. The process is shown in Figure 3 and $q_{{\theta }^{\prime }{s}^{\prime }}$ was assumed to be array (3).

$$q_{{\theta }^{\prime }{s}^{\prime }}=\left[\begin{array}{cccc}4.1& 2.4& 5.1& 2.1\\ 3.7& 4.2& 1.2& 1.0\\ 1.2& 2.1& 1.2& 8.2\end{array}\right]$$

(3)

$argmax\left(q_{\theta s^{\prime }}\right)$ through $q_{\theta s^{\prime }}$ is applied to $q_{{\theta }^{\prime }{s}^{\prime }}$, and $Double{q}_{s’}$ is derived using Formula (4).

$$Double{q}_{s^{\prime }}=q_{{\theta }^{\prime }{s}^{\prime }}[range\left(batc{h}_{size}\right),argmax\left(q_{\theta s^{\prime }}\right)]$$

(4)

The above $Double{q}_{s’}$ calculated using Formula (4) gives the following array (5) when it is calculated based on the example.

$$Double{q}_{s^{\prime }}=\left[\begin{array}{ccc}2.4& 1.2& 1.2\end{array}\right]$$

(5)

The calculated $Double{q}_{s’}$ is substituted in Formula (6), and the calculated Formula (6) is substituted in Formula (7) in order to calculate the final $los{s}_{s^{\prime }}$.

$$Target{q}_{s^{\prime }}=R+\gamma *Double{q}_{s^{\prime }}$$

(6)

$$Los{s}_{s^{\prime }}={[Target{q}_{s^{\prime }}-q_{\theta s}*onehot\left(A\right)]}^2$$

(7)

The following calculated loss value can be clearly reduced compared to the one obtained before the verification process using the Q2 Network. Naturally, the performance varies widely depending on the parameter value, datasets feature, and the environment. However, the above algorithm clearly has a limit to simply determine the $argmax\left(q_{\theta s^{\prime }}\right)$ value through $q_{\theta s^{\prime }}$ alone. Therefore, Quad Q Network suggested in this paper was created by duplicating Q1 and Q2, as shown in Formula $q_{\theta s^{\prime }}$ and $q_{{\theta }^{\prime }{s}^{\prime }}$ were then computed separately before deriving a loss value using the Formula process shown above. The detailed Quad Q Network algorithm’s drawings are shown as in Figure 4.

The Quad Q Network proposed in this paper does not reverse the structure of the existing Dueling Deep Q Network algorithm, thus, a similar precondition was established for the same Dataset, and the $Los{s}_{s’}$ value was calculated by substituting the idea of Double Deep Q Network derived only from one precondition, $q_{\theta s’}$. However, the comparison of two loss values to derive the loss value through the calculation of $q_{\theta ‘s’}$ can amplify the performance by selecting the side with less loss value. In addition, in Figure 4, the $argmax\left(q_{{\theta }^{\prime }{s}^{\prime }}\right)$ value is presented by array (8) values as the weight for each row.

$$argmax\left(q_{{\theta }^{\prime }{s}^{\prime }}\right)=\left[\begin{array}{ccc}2& 1& 3\end{array}\right]$$

(8)

$argmax\left(q_{{\theta }^{\prime }{s}^{\prime }}\right)$ value through $q_{{\theta }^{\prime }{s}^{\prime }}$ is applied to $q_{{\theta }^{\prime }{s}^{\prime }}$, and $Double{q}_{\theta ’}$ is derived from Formula (9).

$$Double{q}_{{\theta }^{\prime }}=q_{{\theta }^{\prime }{s}^{\prime }}[range\left(batc{h}_{size}\right),argmax\left(q_{{\theta }^{\prime }{s}^{\prime }}\right)]$$

(9)

The above $Double{q}_{\theta ’}$ calculated by Formula (9) gives array (10) when calculation is based on the example.

$$Double{q}_{{\theta }^{\prime }}=\left[\begin{array}{ccc}3.1& 3.2& 7.2\end{array}\right]$$

(10)

The calculated $Double{q}_{\theta ’}$ is substituted in Formula (11), and the calculated Formula (11) is substituted in Formula (12) to calculate the final $los{s}_{{\theta }^{\prime }}$.

$$Target{q}_{{\theta }^{\prime }}=R+\gamma *Double{q}_{{\theta }^{\prime }}$$

(11)

$$Los{s}_{{\theta }^{\prime }}={[Target{q}_{{\theta }^{\prime }}-q_{\theta s}*onehot\left(A\right)]}^2$$

(12)

The Quad Q Network, proposed by comparing initially computed $Los{s}_{s’}$ and $Los{s}_{{\theta }^{\prime }}$, proceeds with the learning to derive performance improvements through lowering of the loss value.

4. Experiment and Results

4.1. Datasets of Industrial Google Resource Value

In this paper, we conducted a study using Total Resource on Google from 28 March 2014 to 10 November 2017. The configuration of the datasets consisted of Date, Price, and Volume as shown in

Table 1. Total Resource on Google from 28 March 2014 to 10 November 2017.

Date	Price	Volume
28 March 2014	559.99	41,003
31 March 2014	556.97	10,772
1 April 2014	567.16	7932
2 April 2014	567.00	146,697
3 April 2014	569.74	5,087,500
4 April 2014	543.14	6,377,600
7 April 2014	538.15	4,368,717
8 April 2014	554.90	3,148,563
9 April 2014	564.14	3,323,579
⋮	⋮	⋮
3 November 2017	1032.48	1,076,350
6 November 2017	1025.90	1,124,765
7 November 2017	1033.33	1,112,146
8 November 2017	1039.85	1,088,395
9 November 2017	1031.05	1,244,886
10 November 2017	1028.07	720,674

.

In order to visualize this dataset, it is presented as in Figure 5 below in order to visualize the overall value of Total Resource on Google.

4.2. DQN, Dueling DQN, Double Dueling DQN, and QQN Learning Results on Google’s Total Resource

Based on Table 1, we conducted a study in this paper. Datasets of Total Resource were obtained from Google, and a learning process was conducted by measuring the reward based on these datasets.

Figure 6 presents learning results by substituting the DQN algorithm based on the proposed Total Resource on Google. After the learning process, the minimum reward was −118, the maximum reward was 8, the maximum loss value was approximately 92 K, and the minimum loss value was 19.

Figure 7 presents learning results by substituting the Dueling DQN algorithm based on the proposed Total Resource on Google. After the learning process, the minimum reward was −50, the maximum reward was 21, the maximum loss value was approximately 805, and the minimum loss value was 4.5.

Figure 8 presents learning results by substituting the Double Dueling DQN algorithm based on the proposed Total Resource on Google. After the learning process, the minimum reward was −44, the maximum reward was 28, the maximum loss value was approximately 8322, and the minimum loss value was 3.6.

Figure 9 presents learning results by substituting the Quad Q Network algorithm based on the proposed Total Resource on Google. After the learning process, the minimum reward was −6, the maximum reward was 29, the maximum loss value was approximately 56, and the minimum loss value was 1.4.

Figure 10 shows a comparative analysis of the proposed Q Network reward of Total Resource on Google. We can see that the reward values have stabilized quickly in the order of DQN, Dueling DQN, Double Dueling DQN, and Quad Q Network. The proposed Quad Q Network algorithm presents an enhanced performance compared to the number of epochs based on the reduction of Loss value.

4.3. DQN, Dueling DQN, Double Dueling DQN, and QQN Learning Results on Apple’s Total Resource

Since the model was customized for specific datasets, the datasets type was changed and Total Resource on Apple was used. The learning process was achieved by measuring the reward based on this datasets.

Figure 11 presents learning results by substituting the DQN algorithm based on the proposed Total Resource on Apple. After the learning process, the minimum reward was −57, the maximum reward was 53, the maximum loss value was approximately 4257, and the minimum loss value was 9.6.

Figure 12 presents learning results by substituting the Dueling DQN algorithm based on the proposed Total Resource on Apple. After the learning process, the minimum reward was −228, the maximum reward was 7, the maximum loss value was approximately 176 K, and the minimum loss value was 43.

Figure 13 presents learning results by substituting the Double Dueling DQN algorithm based on the proposed Total Resource on Apple. After the learning process, the minimum reward was −39, the maximum reward was 58, the maximum loss value was approximately 2814, and the minimum loss value was 7.

Figure 14 presents learning results by substituting the Quad Q Network algorithm based on the proposed Total Resource on Apple. After the learning process, the minimum reward was 0, the maximum reward was 42, the maximum loss value was approximately 118, and the minimum loss value was 0.6.

Figure 15 shows comparative analysis of the proposed Q Network reward of Total Resource on Apple. We observe that reward values stabilize quickly in the order of DQN, Dueling DQN, Double Dueling DQN, and Quad Q Network. Depending on Datasets. It is absolutely difficult to judge the performance of DQN and Dueling DQN. However, in the case of the proposed Quad Q Network, experiments using previous datasets also confirmed that fast performance derivation could result from loss value reduction, further resulting in fast stabilized reward values.

5. Conclusions

Fuzzy reasoning mimics human reasoning and tests random actions that agents might take as reward functions in reinforcement learning. However, there has been no works on additional tasks particularly in loss value calibration tasks in conventional reinforcement learning. In this paper, loss value correction with Quad Q Network has are shown to achieve superior performance. The presented simulation scenarios demonstrate that the proposed Quad Q Network algorithm could induce smooth supply and demand of enterprise resources and prevent use of surplus resources and resource missing events. This paper proposes a supply-demand response algorithm to minimise enterprise resource and resource price by shifting load supply-demand, in response to enterprise resource price signal and consumer preferences, from peak periods when the resource price is high to off-peak demand when the resource price is low. In this study, an effective household resource management tool was developed using Q-learning to deal with dynamic resource prices and different resource consumption patterns without compromising the users’ lifestyle or preferences. The proposed quad Q Network performed better than other techniques through renewal of Loss value using Target Q value as a task for minimizing the loss value to predict enterprise resources. Quad Q Network algorithm was proven to have good performances of loss reduction through other datasets. These algorithms could be used for predicting resources in other fields such as predicting future electricity demand, predicting computer resources, and predicting network communication resources in the future.