Research on Human-Computer Interaction Technology of Large-Scale High-Resolution Display Wall System

Abstract

As an effective solution for data visualization and analysis, the large-scale high-resolution display wall system has been widely used in various scientific research fields. On the basis of investigating existing system cases and research results, this paper introduces the SHU-VAS system (60 screens, 120 megapixels) for data visualization and analysis. In order to improve the efficiency of human-computer interaction in large-scale high-definition display wall systems, we propose an interaction framework based on gesture and double-precision pointing technology. During the interaction process, an adaptive mode switching method based on user action features is used to switch between rough and precise control modes. In the evaluation experiments, we analyzed the characteristics of different interaction methods in movement and navigation interaction tasks, and verified the effectiveness of the gesture-based interaction framework proposed in this paper.

1. Introduction

With the increasing scale and complexity of data, the visual analysis technology of data is receiving more and more attention. A large-scale high-resolution display wall system allows multiple users to observe and analyze a large amount of information at the same time, and can provide enough physical space to realize multi-person collaborative work [1,2]. This type of display wall system has a wide application prospect and can be used for image analysis and processing [3,4], high-resolution image display [5,6], medical research [7], financial transaction data analysis [8], traffic management [9] and other fields. It is an effective tool for visualization and analysis of large and complex datasets and data-intensive tasks [10].

The appearance of the scalable cluster-driven display wall system makes the scale of the display wall system larger and larger. However, it also brings a series of new problems, such as how to achieve the best display effect, how to design appropriate human-computer interaction methods and how to achieve efficient multi-person collaboration [11]. The large-scale virtual reality hybrid system CAVE2 developed by the Electronic Visualization Laboratory of the University of Illinois at Chicago in 2012 is the first display wall system that meets the Hybrid Reality Environment (HRE) standard [12]. The HRE standard has 5 features. The display method: (1). Conforms to the natural visual characteristics of human beings [13]; (2). Supports the stereoscopic display of 2D/3D data sets; (3). Supports natural human-computer interaction; (4). Supports multi-person collaborative workspace; (5). Uses a set of software that can support the effective operation of the system. The HRE standard provides guidance for the design and implementation of similar systems.

In terms of human-computer interaction, traditional display wall systems usually use a mouse-and-keyboard-based interaction method, which limits the operator’s interaction range to a certain extent, and the large-scale movement of the mouse reduces the interaction efficiency. In recent years, with the development of sensors and artificial intelligence technology, gesture-based interaction technology has gradually become a hot spot in the field of human-computer interaction research in display wall systems. Although many research works have successfully implemented gesture-based human-computer interaction in display wall systems, work comparing gesture interaction with other interaction methods in different types of interaction tasks is relatively rare. It is therefore difficult to evaluate the superiority of gesture-based interaction methods. To solve these problems, the main work of this paper is:

• On the basis of fully investigating the characteristics of typical display wall systems in the past, we designed and implemented an interactive high-resolution display wall system SHU-VAS (60 screens, 120 megapixels) for visual analysis of big data.
• A general human-computer interaction framework for large-scale high-resolution display wall systems is proposed, which uses gesture-based adaptive double-precision interaction technology to achieve efficient human-computer interaction.
• We designed different types of evaluation experiments to compare the human-computer interaction method proposed in this paper with other interaction methods, and verified the effectiveness of the human-computer interaction framework proposed in this paper.

The organizational structure of the following parts of this paper is as follows: Section 2 introduces the development of the current high- resolution display wall system research and human-computer interaction methods in detail, and summarizes the main field of research on current high-definition display wall systems. Section 3 introduces the system structure of SHU-VAS. Section 4 introduces the human-computer interaction framework proposed in this paper. Section 5 introduces the adaptive double-precision gesture interaction method in the human-computer interaction framework. Section 6 designs and implements two types of evaluation experiments to verify the effectiveness of the human-computer interaction framework in this paper. Section 7 summarizes the work of this paper and presents future work plans.

2. Related Work

2.1. Introduction of Existing Display Wall Systems

Early display wall systems mostly used projection display methods [14], such as the CAVE [15] system. With the development of display equipment, the display screen using LCD panels has become the mainstream display solution for building a display wall system because of its better display effect, higher cost performance ratio and lower maintenance cost. As shown in

Table 1. Configuration of large-scale high-resolution display wall systems.

	STALLION [16]	CAVE2 [10]	RealityDeck [17]	HORNET [18]	DO [19]	SHU-VAS
Shape (Top View)
Layout	Flat 11.2 (L), 2.2 m (H)	Circular 3.65 m (R), 320°	Rectangle 10 m (L), 5.8 m (W)	Curve 7 m (R), 60°	Circular 3 m (R), 313°	Curve 9 m (R), 80°
Screen type (LCD)	30″, 2560 × 1600	46″, 1366 × 736	27″, 2560 × 1440	46″, 1920 × 1080	46″, 1920 × 1080	46″, 1920 × 1080
Number of screens	80 (16 × 5)	72 (18 × 4)	416 (52 × 8)	35 (7 × 5)	64 (16 × 4)	60 (12 × 5)
Number of pixels (megapixels)	328	72	1500	72	132	124
Number of render nodes	23	36	18	3	Ⅰ: 32 Ⅱ: 5	Ⅰ: 24 Ⅱ: 3 Ⅲ: 3
CPU	2 × 6-core Intel Sandy Bridge 2667 W	16-core Xeon E5-2690	2 × 6-core Xeon E5645	2 × Intel Xeon E5-2609	Ⅰ: 6-core Xeon Ⅱ: 2 × (12/16)-core Xeon	Ⅰ: 4-core Xeon E5-2623 Ⅱ: 2 × 12-core Xeon E5-2650 Ⅲ: 2 × 12-core Xeon E5-2650
GPU	2 × Nvidia Quadro K5000	Nvidia GTX 690	4 × AMD FirePro V9800	3 × Nvidia GTX 780	Ⅰ: Nvidia Quadro K5000 Ⅱ: 2 × AMD FirePro W8100	Ⅰ: Nvidia Quadro M5000 Ⅱ: 2 × AMD FirePro W9100 Ⅲ: Nvidia Quadro P5000
Stereo mode	N/A	Passive mode	N/A	N/A	N/A	N/A
Tracking system	N/A	10 × Infrared camera	24 × Infrared camera	7 × Infrared camera	N/A	6 × Depth sensor
Sound system	N/A	20.2	24.4	N/A	16.3	14.2

, many large-scale high-resolution display wall systems have been used in practical application fields. Although these display wall systems have different shapes and layouts, they are all equipped with powerful computing and rendering systems that can visualize data with extremely high detail and quality. These display wall systems belong to scientific research institutions such as universities, and are mainly used for visualization and analysis of data-intensive scientific research tasks.

The structure of large-scale display wall systems is complex and needs to be composed of various hardware systems and software systems. Therefore, there are many research directions for large-scale display wall systems, including: researching appropriate visualization methods, supporting software for large-scale high-definition display wall systems, and human-computer interaction and collaboration methods. As can be seen from Table 1, most display wall systems are equipped with a tracking system, and the research on human-computer interaction technology for large-scale high-definition display wall systems is a hot spot in this field, and it is also one of the characteristics of display wall systems that are different from traditional display systems.

2.2. Human-Computer Interaction Method

In order to make full use of the advantages of the display wall system, the researchers conducted in-depth research on the key technologies of the display wall system. Human-computer interaction methods constitute one of the important research areas for display wall systems. Although a larger display area is helpful for the visualization of data, it also brings some difficulties to human interaction. With the help of software tools such as Synergy, the cross-machine operation of keyboard and mouse can be realized, but it is difficult to use because the mouse pointer icon is too small and too sensitive. Therefore, the operation efficiency of traditional keyboard and mouse is greatly reduced on large-scale screens [20]. In order to solve this problem, researchers have proposed a variety of solutions, including using additional interactive devices, graphical information and composite interaction methods.

Using additional interactive devices is an effective solution to achieve human-computer interaction, such as using personal smartphones with display wall systems [21,22]. Kim et al. designed a haptic device to assist visually impaired users in target selection during interaction with the display system [23]. However, there is still a lack of unified interfaces and standards for additional interactive devices, making it difficult to achieve cross-system use, which may bring additional learning costs.

Graphical information can provide more auxiliary information for the interactive process at the software layer, and can improve the operation efficiency of users in selecting application windows and targets [24]. For example, the GIAnT system provides a rich set of visual elements to support the study of multi-user interaction behaviors on display wall systems [25]. Similar to the problems of additional interactive devices, there is currently a lack of unified standards for graphical information, so various types of visual information may cause confusion during user interaction.

Combination interaction methods can use multiple interaction methods comprehensively. For example, the TALARIA system combines touch operation with hover operation to help users interact with objects that cannot be directly touched [26]. Combination interaction can make full use of the advantages of various interaction methods, but it also creates higher requirements for the design of combination methods.

Among various human-computer interaction methods, the gesture-based interaction is very suitable for large-scale high-resolution display wall systems. For example, Klein et al. implemented a gesture recognition framework that supports multiple input devices to meet different application requirements and realize the interaction with the display wall system [20]. Mathieu Nance et al. proposed the key technology of designing mid-air interaction [27], and further proposed a solution including an adjustable acceleration function and double-precision technical framework to improve the efficiency of mid-air pointing technology [28]. These interaction methods all need to convert the user’s device-based or gesture-based interaction into the movement and operation of the mouse pointer. This research shows that different pointing techniques can effectively accomplish different tasks [29].

Based on these research results, the basic principles for designing human-computer interaction methods in display wall systems can be obtained. First, compared with traditional small-scale displays, when users interact with the display wall system, users tend to use gesture-based interactive operations with larger interactive areas. Then, if the interactive gestures used are familiar to users, it can simplify the process of learning interactive gestures [30]. Finally, the system should provide interaction techniques that do not require a lot of walking, because too much movement will cause physical fatigue and is not suitable for long-term interactive operations [1].

On the other hand, due to the limitations of a single depth sensor in terms of viewing angle and detection distance, in order to realize gesture interaction control on a large-scale display wall system, it is necessary to design a synchronous cooperation and asynchronous alternate control method of multiple depth sensors. To solve this problem, we design and implement a gesture-based human-computer interaction framework called SHU-VAS by referring to the above design principles. The interaction framework uses distributed multiple depth sensors, which can help users realize a wide range of human-computer interaction in the display wall system.

3. System Introduction

The human-computer interaction method of the display wall system is closely related to the structure and shape design of the system. Therefore, before introducing the human-computer interaction framework proposed in this paper, it is necessary to introduce the display wall system used in this paper, including the system structure and layout.

3.1. System Structure

The system structure of SHU-VAS is shown in Figure 1, which is mainly composed of a visual interaction system and a data center. As can be seen from Table 1, the Reality Deck system has reached 1.5 gigapixels. However, the research work of Papadopoulos shows that when the number of pixels in the display wall system exceeds 600 M, the visualization effect of the system will not be further improved, but the search efficiency of visual targets will be significantly reduced [31]. Therefore, SHU-VAS does not pursue a higher number of pixels, but achieves good visualization effects and efficient human-computer interaction under the constraints of limited physical space and funds.

The visual interaction system mainly includes display wall, rendering cluster and desktop console. The display wall consists of 60 46-inch Full HD (1920 × 1080) splicing screens in a 5 × 12 arc layout, with a total resolution of 23,040 × 5400 and a total of 124 megapixels. The frame thickness between the two screens is less than 3.5 mm, and all screens are connected to the desktop console by daisy chain. The rendering cluster includes 30 high-performance graphics workstations equipped with NVIDIA Quadro series and AMD FirePro series professional graphics cards, which are responsible for driving the entire display array and providing multiple working modes. The desktop console consists of four 4K resolution screens in a 2 × 2 layout. Users can manage the content displayed by the system through the console, and can also control and manage each screen, including switching power supply, switching signal sources, and adjusting display parameters. SHU-VAS includes a gesture interaction system composed of 6 Kinect V2.

With the continuous development of artificial intelligence technology, the scale of various data sets is getting larger, and the complexity of algorithms is also getting higher. In order to better meet the needs of current artificial intelligence algorithms for data set scale and computing power, and to support remote rendering of complex 3D scenes and high-resolution images [32], SHU-VAS includes a data center.

The data center includes CPU computing clusters, GPU computing clusters, network storage and network devices to meet the needs of different application scenarios. Specifically, it includes: 9 CPU computing nodes for high-performance parallel computing; 4 GPU computing nodes equipped with NVIDIA Tesla V100, providing sufficient computing power for AI algorithms; 4 GPU rendering nodes equipped with NVIDIA Quadro P6000, providing the platform with sufficient rendering capacity; 160 TB network storage system, meeting the data storage requirements of various applications. The various components of SHU-VAS are connected by a Gigabit Ethernet network and a 10 Gigabit optical fiber network. The Gigabit Ethernet network is used for the transmission of control signals, and the 10 Gigabit optical fiber network is used for data transmission.

3.2. System Layout

In order to help SHU-VAS have a good display effect, we designed the layout of the screen according to the natural visual characteristics of human beings.

Takashima et al. designed a deformable display wall system, which can change the layout of the screen according to the needs of different application scenarios [33]. Although this design is not yet suitable for large-scale display wall systems, their research work shows that the curved surface structure can meet the needs of most application scenarios. It can also be seen from Table 1 that many display wall systems adopt a curved or circular layout. The flat layout of the STALLION system is not suitable for users to observe all displayed content at medium and close distances. The rectangular layout of Reality Deck makes it difficult to achieve a smooth visual transition at the corners. The circular layout of CAVE2 and DO can provide users with a good sense of immersion, but it also means that users need more turning and walking actions to observe the full picture of the displayed content. In order to enhance the user’s observation effect on the overall display content, SHU-VAS adopts a curved layout with a radius of 9 m and an opening angle of 80 degrees.

3.3. System Observation Area

We analyze the observation area of the system according to the two physiological characteristics of human vision: Visual Acuity (VA) and Field of View (FOV).

The visual sensitivity of the human eye limits the Pixel Per Degree (PPD) that the human eye can recognize to 60. That is to say, within the range of 1° viewing angle, the human eye can distinguish up to 60 pixels in one direction. When the PPD is greater than 60, the human eye will not be able to perceive the existence of pixels [34], such as the retina screen used by Apple.

Let p be the distance between two pixels on the display. Let w be the width of the display screen, and h_r be the horizontal resolution of the display screen, then:

$$p=\frac{w}{h_r},$$

(1)

Assuming that the distance between the user and the display screen is d, the PPD can be calculated by the following formula:

$$PPD\approx \frac{d·\tan 1^{°}}{p}=\frac{d·h_r\tan 1^{°}}{w}\ge 60,$$

(2)

Then:

$$d\ge \frac{60w}{h_r·\tan 1^{°}}.$$

(3)

The parameters of the 46-inch screen used by the system in this paper are w = 1.02 m, h_r = 1920 px. Substituting them into the above formula, d_min ≈ 1.8 m. Therefore, when the distance between the user and the display screen exceeds 1.8 m, the user will not be able to perceive the pixels and achieve the best visual effect. In addition, limited by the size of the room, d_max ≈ 4 m.

When keeping the eyeballs still, the horizontal FOV of human eyes is about 180°, the vertical FOV is about 130°, and the color perception range is about 100° according to the spatial distribution of cone cells [35]. The area with the highest visual accuracy is only 5° from the optical axis. Although the range of visual information that can be recognized with high precision in the visual field is small, the influence of the peripheral visual field on the user’s attention should not be ignored. The research work of Sigitov et al. shows that the working area of the display wall system needs to be carefully designed because the peripheral vision may affect the user’s attention and reduce work efficiency [36]. Based on the range that the color perception area can cover on the display wall, we analyzed the observation area of the system, as shown in Figure 2.

Figure 2a is a side view of the display wall; the bottom baffle H_bottom = 0.3 m, and is used to install 3 Kinect V2 sensors at the bottom. The height of the 46-inch screen is about 0.58 m, so the height of the display area is H_wd = 0.58 × 5 = 2.9 m. The height of the top LED strip is H_top = 0.3 m. Assuming that the user’s height is H_user = 1.8 m, the user’s horizontal line of sight is in the center area of the display screen, and when d_min ≤ d ≤ d_max, the display area is all within the color perception range of the human eye in the vertical direction.

As shown in Figure 2b, both the bottom baffle and the top LED light bar use dark color panels, while the roof and the ground use light color panels. By visually setting suggestive boundaries, the user’s attention can be focused on the interior of the display area.

Figure 2c is a top view of the display wall. When the user is at point A, the distance from the screen d = d_min = 1.8 m, and the user will not be able to distinguish the pixels and achieve good observation effect. However, the user’s viewing range is relatively small at this time, and the user needs to turn the head and body in a large range to observe other areas of the display wall. When the user is at point B, the distance from the screen d = d_max = 4 m, the user can directly observe most areas of the display wall, and only needs to turn the head in a small range to observe the entire display area. Considering the supplementation of the peripheral vision on both sides, the user will obtain the best observation effect at this time. When the user is at point C, the entire display area can be directly observed, but the observation effect will be affected by the distortion caused by the excessive angle between the line of sight and the display wall.

The effective recognition distance of Microsoft Kinect V2 in the default working mode is 0.8 m to 4 m, and the recommended distance is 1.2 m to 3.5 m. Therefore, when the user is in the system observation area, the Kinect V2 used by the system can better capture the user’s skeletal node information. It should be noted that the research work of Hamed et al. [37] shows that when multiple Kinect V2 systems using ToF technology work at the same time, there will be a certain degree of interference, and the median drift reaches 5 mm. In some applications that require high precision, such as high-precision 3D reconstruction, this interference error will have a significant impact on the results. However, in the application based on skeleton node information, the interference error caused by multiple Kinects working together can be ignored. In fact, in order to obtain stable skeletal node information, multiple Kinects with different angles can be used at the same time [38]. In the human-computer interaction framework of the system in this paper, the interference error will also be ignored.

4. Human-Computer Interaction Framework

4.1. Design Principles

Human-computer interaction has always been a research hotspot in the field of large-scale display wall systems. It is meaningful to realize natural and efficient interactive operation with visual objects in a large-scale display wall system. According to the description in Section 2.2 of this paper, we believe that the interaction method for large-scale display wall systems should be simple, direct and reliable. As in the traditional WIMP interaction paradigm, users only need to combine simple operations such as clicking keyboard buttons, clicking mouse buttons and dragging the mouse to realize all interactive operations.

SHU-VAS does not use a display screen that supports touch operation, but we have designed a human-computer interaction framework based on gestures. Compared with previous systems, such as CAVE2, DO, etc., SHU-VAS does not use an infrared camera-based tracking system, but uses 6 Kinects to form a gesture interaction system. The advantage of using Kinect is that users can interact with visual objects using natural gestures without the need for additional markers or equipment. The collaborative use of multiple devices can expand the working area and overcome the occlusion problem to a certain extent.

4.2. Workflow

As shown in Figure 3, in our gesture interaction framework, MSCube SDK is used to collect data from 6 Kinects, and ZeroMQ is used to transmit data to the management server through the network. MSCube is an open source project for processing multi-kinect data, supporting other frameworks such as Cinder, NodeJS, etc. ZeroMQ is a message queue-based multithreaded networking library that uses an asynchronous IO model and provides sockets across multiple transport protocols. Considering the network bandwidth and delay, the system in this paper uses the skeleton node information as the data source of gesture interaction.

After capturing the user’s skeletal node information, we use the gesture recognition algorithm we have proposed to recognize the user’s interactive gestures [39], then convert the gestures into the operational semantics that the user wants to express, and finally map them to the corresponding system events and call the function interfaces. In order to respond to interactive gestures, applications need to provide customized application interfaces. In order to make the gesture interaction framework of SHU-VAS compatible with more existing applications, we map the user’s interaction gesture semantics to the system messages in the traditional WIMP human-computer interaction paradigm. The interaction between the user and the application is realized by converting the user’s operational semantics into a series of system messages.

4.3. Uniform Coordinate System

In order to facilitate subsequent data processing, we need to make uniform the 3D coordinate information of the skeleton nodes captured by the 6 Kinect V2 into one coordinate system, as shown in Figure 4. To avoid noise interference, all captured skeletal node information is preprocessed using median filtering.

C_i represents the coordinate system centered on the ith Kinect. We take multiple sets of checkerboard calibration board images with different poses in the overlapping area of the two Kinects, and then use stereo calibration method to calculate the transformation matrix H_i,j between C_i and C_j.

$$\{\begin{array}{c}{P}_l=R_l·P_w+T_l\\ P_r=R_r·P_w+T_r\end{array},$$

(4)

where P_w represents the coordinates of a point P on the checkerboard calibration board in the world coordinate system. R_l and T_l represent the rotation matrix and translation matrix of point P relative to the left Kinect position, respectively. R_r and T_r represent the rotation matrix and translation matrix of point P relative to the right Kinect position, respectively. P_l and P_r, respectively, represent the coordinates of the left and right Kinects in the world coordinate system. With P_w, Formula (4) can be rewritten as:

$${(R_l)}^{-1}·(P_l-T_l)-{(R_r)}^{-1}·(P_r-T_r)=0.$$

(5)

Solving Formula (5), we can get:

$$P_r=\left[R_r{(R_l)}^{-1}\right]P_l+\left[T_r-R_r{(R_l)}^{-1}{T}_l\right].$$

(6)

The position relationship of the right Kinect relative to the left Kinect can be expressed as:

$$P_r=R·P_l+T.$$

(7)

By combining Formulas (6) and (7), we can get:

$$\{\begin{array}{l}R=R_r{R}_l^T\hfill \\ T=T_r-R{T}_l\hfill \end{array}.$$

(8)

The transformation matrix H can be expressed as:

$$H=\left|\begin{array}{cc}R& T\\ 0& 1\end{array}\right|.$$

(9)

The point P_i,k in the coordinate system C_i can obtain the coordinate P_j,k of this point in the coordinate system C_j by transforming the matrix H_i,j, that is P_j,k = H_i,j P_i,k. We take C₁ as the coordinate system of the interaction area of the whole system. By obtaining the transformation matrix between every two Kinects, we can unify the points in other coordinate systems into the C₁ coordinate system, as follows:

$$\{\begin{array}{l}{P}_{1,0}=H_{0,1}{P}_{0,0}\hfill \\ P_{1,2}=H_{2,1}{P}_{2,2}\hfill \\ P_{1,3}=H_{3,4}{H}_{4,1}{P}_{3,3}\hfill \\ P_{1,4}=H_{4,1}{P}_{4,4}\hfill \\ P_{1,5}=H_{5,4}{H}_{4,1}{P}_{5,5}\hfill \end{array}.$$

(10)

The user’s skeletal nodes may be captured by n Kinects at the same time, where n ≥ 1. Then, after the coordinates of the skeletal joint point in each Kinect coordinate system are unified to the C₁ coordinate system by Formula (10), the coordinate of the skeletal node P can be expressed as:

$$P=\frac{1}{n}{\displaystyle \sum _{i=1}^n{P}_i},i=1,2\dots n.$$

(11)

The advantage of this processing is that, on the one hand, the stability of the coordinate P can be improved through multi-coordinate system fusion. On the other hand, when the user is performing interactive operations, if some skeletal nodes are blocked relative to a certain Kinect, the spatial coordinates of the node can be obtained with the help of other Kinects, so the robustness of the interaction can be improved to a certain extent.

4.4. Calculating Directivity

After obtaining the stable spatial coordinates of the skeletal nodes, we take the intersection of the spatial straight line determined by the user’s right hand elbow coordinates P_er and wrist coordinates P_wr and the corresponding spatial surface of the display wall as the user’s pointing target, as shown in Figure 5. In the WIMP interactive paradigm, this intersection represents the position of the mouse on the screen. The coordinate value of the space intersection point can be calculated by the following formula:

$$\{\begin{array}{l}\frac{x-p_{er}(x)}{p_{wr}(x)-p_{er}(x)}=\frac{y-p_{er}(y)}{p_{wr}(y)-p_{er}(y)}=\frac{z-p_{er}(z)}{p_{wr}(z)-p_{er}(z)}\hfill \\ x^2+{(z-R)}^2=R^2\hfill \\ 0\le y\le 2.9\hfill \end{array},$$

(12)

where R represents the radius of the arc surface of the display wall, and R = 9 m.

Then, the space coordinates are mapped to plane coordinates for subsequent conversion into screen coordinates:

$$\{\begin{array}{l}F(x)=[{\sin }^{-1}(\frac{F^{\prime }(x)}{R})]\frac{W}{2}+\frac{W}{2}\hfill \\ F(y)=F^{\prime }(y)\hfill \end{array},$$

(13)

where W is the length of the display wall, and W = 12.57 m.

5. Adaptive Double-Precision Gesture Interaction

5.1. Double-Precision Pointing Technique

Position-based pointing techniques require a transfer function to map input motion captured by the device to cursor motion. The most common transfer function uses Control-to-Display (CD) gain, defined as the ratio between cursor movements, usually measured in terms of linear distance and input change.

The research work of Nancel et al. has theoretically proved that the single-mode remote pointing technology is not suitable for large-scale display wall systems, so a double-precision technical framework supporting the acceleration function was proposed. The double-precision pointing technology divides the pointing task into two stages: (1) the coarse positioning stage, where the user controls the interactive object to quickly approach the target; (2) the precision positioning stage, where the user can precisely control the interactive object to move to the target area.

Both coarse and precise modes can be absolute or relative. Absolute controls are variable speed and easy to use, as the previous cursor position does not affect the input movement, i.e., the cursor can be teleported. They have a fixed gain, however, and thus have limited precision. The pointer acceleration transfer function and wider CD gain range can be applied to the relevant controls, but the offset between the input and cursor position may increase, resulting in variable speed action.

Currently, most research work on double-precision pointing technology uses absolute control for coarse positioning, relative control for precise positioning, and an explicit switching method to switch between coarse and precise modes. We believe that using an explicit switching method will affect the fluency of the user interaction process. In addition, it is necessary to define new interactive gestures or use other devices to realize mode switching, which will increase the user’s burden. Therefore, we have designed an adaptive mode switching method based on the moving speed of user interaction gestures.

5.2. Adaptive Mode Switching

By observing the user’s interaction process, we found that the user’s cursor moves faster in the coarse positioning stage in order to quickly approach the target area. When the interactive object approaches the target area, the user will naturally slow down the moving speed for fine-tuning the position, as shown in Figure 6. According to this operating habit, we switch between coarse and precise modes according to the change of cursor movement speed. After entering precise mode, there is a proportional factor f (0 < f ≤ 1) between the movement gain of the interactive object and the expected movement gain of the user-controlled object. If user stay in the precise mode for a long time, there will be a significant difference between the position of the interactive object on the display wall and the position pointed by the user, which will bring a sense of strangeness to the user. Therefore, it is necessary to switch the control mode of the position of the interactive object from relative control mode to absolute control mode before making a significant difference.

The moving speed at time t is v_t; then, the average moving speed v within $\Delta t$ time can be expressed as:

$$\overline{v}=\frac{1}{\Delta t}{\displaystyle {\int }_{t-1}^t{v}_{t}dt}.$$

(14)

If the current moving speed v is lower than a certain percentage of the average speed $\overline{v}$ in the previous $\Delta t$ range, it will automatically enter the precise control mode. In this mode, the movement gain of the interactive object is determined by multiplying the absolute gain of the user’s pointing by the scaling factor f_v.

When the user controls the movement distance d of the interactive object in the precise movement mode to be greater than the threshold l, in order to avoid a significant difference between the position of the interactive object and the pointing position of the user, the coarse control mode is restored and the position of the interactive object is corrected to the current position pointed by the user. The switching process between each state is shown in Figure 7.

6. Evaluation Experiments

In order to evaluate the effectiveness of the human-computer interaction framework proposed in this paper, we designed two types of evaluation tasks: movement and navigation. The effectiveness of the interaction method proposed in this paper is evaluated by comparing the time to complete the task, operation accuracy and other metrics with different interaction methods.

6.1. Experiment 1 Movement Task

6.1.1. Experiment Setup

We configure the Sage2 service deployed on the server, and configure the 60 screens of the display wall system into a complete 5 × 12 unified desktop. The system uses 27 rendering nodes to drive the 60 screens. Among them, there are 23 type I rendering nodes, each driving 2 screens, and 3 type III rendering nodes, each driving 4 screens. Users can interact in the following 3 ways:

• Desktop console. The user interacts with the Sage2 console using the keyboard and mouse on the desktop console.
• Tablet PC with touch screen. Users use a Microsoft Surface Pro 4 tablet computer to log in to the Sage2 server and use the touch screen for interactive operations.
• Gesture interaction. Users use the human-computer interaction framework proposed in this paper to perform interactive operations through gestures.

6.1.2. Operational Flow

The operation process is shown in Figure 8. Randomly select a grid position in the starting area of 5 × 6 grids on the left to generate an interactive object with a red background, and randomly select a grid position in the target area of 5 × 6 on the right to generate a dotted frame as the moving target of the interactive object. The timing starts when the user selects the interactive object, and stops when the user moves the interactive object near the target area and releases it.

6.1.3. Evaluation Metrics

We define the accuracy rate acc of the operation as the ratio of the overlapping part of the moving object and the target area to the area of the target area:

$$acc=\frac{\Delta w·\Delta h}{w·h},$$

(15)

where w and h denote the width and height of the target area, respectively. $\Delta w$ and $\Delta h$ represent the width and height of the overlapping part of the interactive object and the target area, respectively.

We believe that the distance moved by the grid and the time used are important factors affecting the score, and the score of the operation is defined as:

$$score=acc·\frac{d}{t},$$

(16)

where d represents the moving distance, and t represents the time used to complete the movement task.

6.1.4. Experiment Results and Analysis

The experimental results on the movement task are shown in

Table 2. Experimental results of movement task.

	Source Area (row,col)	Target Area (row,col)	d	t	Acc	Score
Desktop Console	(1,4)	(8,3)	7.212	2.43	0.997	2.959
	(2,3)	(7,2)	5.201	1.89	0.96	2.642
	(4,1)	(9,4)	5.948	2.61	0.972	2.215
	(3,3)	(9,2)	6.204	2.73	0.977	2.220
	(4,2)	(11,1)	7.212	2.9	0.982	2.442
	(4,5)	(11,3)	7.426	3.05	0.991	2.413
	(2,3)	(11,2)	9.236	3.06	0.968	2.922
	(5,2)	(8,3)	3.226	3.47	0.971	0.903
	(2,3)	(9,2)	7.212	2.35	0.967	2.968
	(1,2)	(10,4)	9.404	2.64	0.989	3.523
Surface	(5,4)	(10,3)	5.201	1.62	0.702	2.254
	(1,5)	(9,4)	8.224	2.52	0.779	2.542
	(4,5)	(12,5)	8.160	1.36	0.818	4.908
	(5,4)	(11,4)	6.120	2.36	0.918	2.381
	(1,3)	(9,4)	8.224	2.75	0.967	2.892
	(4,2)	(11,5)	7.768	3.02	0.722	1.857
	(6,3)	(7,2)	1.442	2.66	0.874	0.474
	(5,1)	(8,4)	4.327	2.31	0.703	1.317
	(4,1)	(8,1)	4.080	1.1	0.757	2.808
	(5,3)	(9,1)	4.562	1.91	0.924	2.207
Gesture	(1,4)	(11,1)	10.649	5.17	0.659	1.357
	(6,2)	(10,4)	4.562	3.69	0.653	0.807
	(4,2)	(7,2)	3.060	3.54	0.773	0.668
	(4,1)	(7,5)	5.100	4.16	0.921	1.129
	(4,5)	(8,1)	5.770	2.94	0.781	1.533
	(5,1)	(12,4)	7.768	3.74	0.679	1.410
	(4,4)	(11,1)	7.768	4.75	0.873	1.428
	(6,5)	(10,5)	4.080	2.63	0.766	1.188
	(4,4)	(12,1)	8.715	3.61	0.74	1.786
	(2,5)	(12,3)	10.402	4.36	0.844	2.014
Gesture (double-precision)	(3,1)	(11,3)	8.411	7.26	0.961	1.113
	(6,1)	(10,3)	4.562	6.72	0.918	0.623
	(2,3)	(9,3)	7.140	5.92	0.915	1.104
	(1,2)	(7,2)	6.120	4.8	0.886	1.130
	(3,5)	(10,5)	7.140	5.04	0.973	1.378
	(4,2)	(12,4)	8.411	6.61	0.885	1.126
	(3,2)	(12,5)	9.677	6.38	0.916	1.389
	(4,1)	(10,3)	6.451	6.48	0.875	0.871
	(1,2)	(8,4)	7.426	5.46	0.919	1.250
	(5,4)	(9,4)	4.080	4.75	0.942	0.809

, and the statistical analysis results are shown in

Table 3. Statistics table of movement task results.

	t			acc			Score
	Min	Max	Avg	Min	Max	Avg	Min	Max	Avg
Desktop Console	1.89	3.47	2.713	0.96	0.997	0.977	0.903	3.523	2.521
Surface	1.1	3.02	2.161	0.702	0.967	0.816	0.474	4.908	2.364
Gesture	2.63	5.17	3.859	0.653	0.921	0.769	0.668	2.014	1.332
Gesture (Double-Precision)	4.75	7.26	5.942	0.875	0.973	0.919	0.623	1.389	1.0793

and Figure 9.

In terms of interaction time, the keyboard and mouse-based desktop console is closer to the touch-screen-based Surface, while the gesture-based interaction takes longer. Although the user is more familiar with the operation of the traditional keyboard and mouse, the physical distance for the user’s finger to move on the Surface touch screen is shorter, so the time consumption of the two interaction methods is relatively close. Since gesture interaction using double precision technology consumes more time in precise control mode, the total time consumption is significantly higher than other methods. We believe that users can improve interaction efficiency after practicing interactive gestures for a period of time.

In terms of accuracy, the accuracy and stability of the desktop console method is significantly better than other methods. The results of Surface and gesture interaction are relatively close. The accuracy rates of these two interaction methods for multiple rounds of operation are quite different, and the stability is low. We believe this is due to the user’s low stability of body control. After combining double-precision pointing technology, the accuracy and stability of gesture interaction have been significantly improved.

The score is a combination of interaction time and accuracy. The desktop console and Surface scored higher due to their advantage in interaction time. Affected by the accuracy rate, the score of gesture interaction is significantly lower than other methods. While gesture interaction combined with double-precision pointing technology improved accuracy, the score decreased slightly due to the overall time-consuming increase. This shows that compared with other interaction methods, gesture interaction in movement task does not have advantages in interaction efficiency.

In addition, we counted the time consumption of 10 groups of double-precision gesture interactions, as shown in Figure 10. It can be seen that the time consumption of the precise control mode accounts for about 32.5% of the total time consumption Due to the introduction of the precise control mode, the average time spent on interaction has increased by 54%, and the average accuracy rate has increased by 19.5%, which is close to the accuracy rate of desktop console and better than Surface touch operations. This shows that gesture interaction using double-precision technology is feasible in application scenarios where there is no limit on the interaction time.

6.2. Experiment 2 Navigation Task

On the SHU-VAS platform, we built the Liquid Galaxy [40] system based on the Master-Slaver structure. The Liquid Galaxy system was originally built to run Google Earth to create an immersive visual experience for users [41]. Using the interaction framework of the SHU-VAS, in addition to the traditional keyboard and mouse, 3D mouse, and Leap Motion interaction methods, we have also realized gesture-based human-computer interaction. Users can directly use gestures to interact with the virtual globe, such as zooming, rotating, etc.

6.2.1. Experiment Setup

The structure of the Liquid Galaxy system built based on the system in this paper is shown in Figure 11. The system uses 6 high-performance rendering nodes to drive the 60 screens. Among them, there are 3 type II rendering nodes, each driving 16 screens; 3 type III rendering nodes, each driving 4 screens. Users can use the following four methods for interactive operations:

• Desktop console. On the console of the display wall system, the user uses the keyboard and mouse to perform interactive operations on the Master side of Google Earth.
• 3D mouse. On the console of the display wall system, the user uses a three-dimensional mouse to perform interactive operations on the Master side of Google Earth.
• Leap Motion Controller. The user uses the Leap Motion controller to interact with the Master side of Google Earth on the console of the display wall system.
• Gesture interaction. Users use the human-computer interaction framework proposed in this paper to perform interactive operations using gestures, as shown in Figure 12.

6.2.2. Operational Flow

Set the current viewing angle center as the starting position on the Master side of Google Earth: 35°00′00.00″ ± 5″ north latitude, 100°00′00.00″ ± 5″ east longitude, and 10,000 ± 100 km viewing angle altitude. Target position: 31°19′00.00″ ± 5″ north latitude, 121°23′30.00″ ± 5″ east longitude, viewing angle altitude 1000 ± 100 m. When the user navigates the center of view to the target position through various interactive methods, the navigation task is completed and the timing is stopped.

6.2.3. Evaluation Metrics

Navigation operations can generally be divided into two types: moving and zooming. Let N_move represent the number of moving operations, and N_zoom represent the number of zooming operations. The total number of operations is N = N_move + N_zoom, and the smaller N is, the fewer adjustments need to be made during the navigation process, and the higher the navigation efficiency is.

Define the score of an operation as:

$$score=\frac{C}{N·t},$$

(17)

where C is a constant, and C = 100. t represents the time taken to complete the navigation task.

6.2.4. Experiment Results and Analysis

The experimental results of the navigation task are shown in

Table 4. Experimental results of navigation tasks.

	Nmove	Nzoom	t	Score
Desktop Console	6	4	7.227	1.384
	6	8	7.815	0.914
	5	5	7.443	1.344
	6	6	7.694	1.083
	6	7	12.084	0.637
	4	7	6.293	1.445
	4	8	6.588	1.265
	4	4	9.961	1.255
	8	4	7.954	1.048
	7	8	12.282	0.543
3D Mouse	4	4	11.446	1.092
	4	6	10.781	0.928
	4	6	11.09	0.902
	6	4	9.838	1.016
	6	4	10.67	0.937
	3	6	9.12	1.218
	6	7	9.313	0.826
	3	7	9.799	1.021
	4	6	11.807	0.847
	6	8	10.813	0.661
Leap Motion	3	5	11.566	1.081
	3	5	7.525	1.661
	6	5	11.403	0.797
	4	6	11.309	0.884
	3	6	12.565	0.884
	6	6	11.696	0.712
	4	5	7.824	1.420
	6	5	8.644	1.052
	6	4	12.878	0.777
	6	5	11.609	0.783
Gesture	7	9	8.476	0.737
	6	10	9.807	0.637
	7	7	10.236	0.698
	7	6	9.493	0.810
	5	7	8.811	0.946
	9	10	9.528	0.552
	8	7	9.67	0.689
	7	7	10.72	0.666
	9	8	9.389	0.627
	8	9	8.804	0.668
Gesture (double-precision)	6	6	10.976	0.759
	8	7	12.738	0.523
	6	10	9.452	0.661
	7	7	12.621	0.566
	9	6	9.29	0.718
	6	8	11.448	0.624
	9	7	13.131	0.476
	5	10	12.841	0.519
	6	9	9.263	0.720
	6	8	10.756	0.664

, and the statistical analysis results are shown in

Table 5. Statistics table of navigation tasks.

	Nmove + Nzoom			t			Score
	Min	Max	Avg	Min	Max	Avg	Min	Max	Avg
Desktop Console	8	15	11.7	6.293	12.282	8.534	0.543	1.445	1.092
3D Mouse	8	14	10.4	9.12	11.807	10.468	0.661	1.218	0.945
Leap Motion	8	12	9.9	7.525	12.878	10.702	0.712	1.661	1.005
Gesture	12	19	15.3	8.476	10.72	9.493	0.552	0.946	0.703
Gesture (double-precision)	12	16	14.6	9.29	13.131	11.252	0.476	0.759	0.623

and Figure 13. It can be seen that although there is a certain learning cost for using the 3D mouse and Leap Motion for navigation operations, users can still achieve operating efficiency similar to that of traditional mouse operations. We believe that the interactive devices used in these three interaction methods are placed on the console desktop in a relatively stable manner, and the offsets of each user operation are relatively similar, so the average number of operations required to complete the navigation task is similar.

Gesture operation has a relatively low overall score because the number of interactive operations is significantly more than other methods. The introduction of double-precision gesture operation allows faster reaching of the target in the final fine-tuning stage and reduces the average number of operations. However, due to the increase in total time consumption, the score is reduced. Therefore, for navigation tasks, there is no advantage in gesture operation in terms of interaction efficiency.

7. Conclusions and Outlook

This paper introduces the large-scale high-resolution display wall system SHU-VAS (60 screens, 120 megapixel) for big data visual analysis. This paper designs and implements a human-computer interaction framework for a large-scale high-resolution display wall system. The framework uses gesture-based double-precision interaction technology to achieve efficient human-computer interaction, and uses an adaptive mode switching method to avoid the impact of explicit mode switching on the fluency of the interaction process.

The evaluation experiments of movement task and navigation task show that, compared with other interaction methods, the human-computer interaction framework proposed in this paper is close to the traditional keyboard-mouse interaction method in terms of operation accuracy, and has practical application value.

As large-scale display wall systems are more and more widely used, related software and hardware technologies are also constantly developing. There are still many aspects of the SHU-VAS system that deserve further research and improvement. Since this paper only uses objective evaluation indicators, it lacks the subjective evaluation of users. In the future work plan, on the one hand, we will further improve the evaluation criteria, such as using the SUS (System Usability Scale) evaluation method. On the other hand, we will further expand the human-computer interaction framework to provide more interaction methods, such as using facial tracking to assist gesture interaction and supporting voice interaction.