A Novel Method Using 3D Interest Points to Place Markers on a Large Object in Augmented Reality

Abstract

Multiple markers are generally used in augmented reality (AR) applications that require accurate registration, such as medical and industrial fields. In AR using these markers, there are two inevitable problems: (1) geometric shape discrepancies between a real object and a virtual object, and (2) the relative positions of the markers placed on the virtual object and markers placed on the real object are not consistent. However, studies on applying multiple markers to a large object are still insufficient. Additionally, most studies did not consider these inevitable problems because the markers were subjectively placed (hereafter conventional method). In consideration of these problems, this paper proposes a method for placing multiple markers to provide accurate registration on a large object. The proposed method divides a virtual object evenly and determines the positions of multiple markers automatically using 3D interest points within the divided areas. The proposed method was validated through a performance comparison with the conventional method of subjectively placing markers, and it was confirmed to have more accurate registration.

1. Introduction

Augmented reality (AR) is a computer graphics technology that (1) combines a virtual image with a real environment, (2) enables real-time interaction, and (3) registers a virtual object in a three-dimensional space [1]. The technology to register an augmented virtual object to an object in the real environment (hereafter real object) is not only a core technology of AR but also an index for ensuring performance [2].

Registration is conducted by using markers to combine the real world and the virtual world (hereafter marker-based AR), or by using features between the real and virtual world through sensors (hereafter markerless AR). A marker generally refers to an object that can be identified and tracked through digital devices, such as camera sensors. There are various types of markers, such as simple flat images [3,4,5,6], infrared markers [7,8], and optical markers [9,10]. Markers are used as reference points for registration not only in marker-based AR but also in markerless AR. For AR applications where accurate registration is essential, such as in medical and industrial fields, marker-based registration is still required [11,12].

If the target is a large object for registration in AR using markers (e.g., vehicles, robots, and factory machines), the registration error of the marker can be increased according to the scale of the object. This error can be complemented by using more than one marker (hereafter multiple markers). This is because other markers can correct errors that may arise in a certain marker, thereby providing more accurate registration [13,14,15].

In AR using these markers, there are two inevitable problems: (1) geometric shape discrepancies between a real object and a virtual object (hereafter geometric discrepancy problem), and (2) the relative positions of markers placed on the virtual object and markers placed on the real object are not consistent (hereafter relative position consistency problem).

The geometric discrepancy problem arises from the issue that the virtual objects designed by 3D/CAD/CAM engineers are not 1:1 identical to the real objects printed out into the real world. This is because errors inevitably occur during the production process due to the state of the machinery, environment, materials, costs, etc. As it is impossible to solve these errors, acceptable errors known as tolerance are generally applied. Registration errors owing to this problem can be reduced by placing multiple markers evenly on the surface of the object.

The relative position consistency problem arises from the issue that, when the marker (A) is placed on the real object based on the position of marker (B) that is placed on the virtual object, there is no guarantee that A and B have relatively consistent positions. This is because the process of referencing A (e.g., subjective interpretation by human or image processing result by machine) may be inaccurate, and errors may occur in the process of placing markers. Registration errors owing to this problem can be reduced by placing multiple markers in clearly identifiable positions.

However, studies on applying multiple markers to a large object are still insufficient [5,14,15]. Boonbrahm et al. (2020) proposed a method to reliably augment a large object using multiple markers for remote collaboration [5]. In the proposed method, five markers were used to augment a large object, but the registration aspect was not considered because it was augmented in an actual box that was not related to the actual object to be registered. Bruno et al. (2020) developed and proposed an AR system based on multiple markers. In the proposed method, multiple markers were placed 1.5 m apart around the perimeter of a large object. However, in this study, the registration was performed again each time a marker was tracked, so it was considered a single marker-based registration [14]. Sheng et al. (2023) proposed a method for 3D scanning actual a large object using AR head-mounted display. Although a virtual object was not augmented in the proposed method, multiple markers were used for stable scanning. The markers were placed around the edge of the actual object, and the distance between them was set to about 25 cm [15].

Additionally, marker placement criteria, such as the number of markers used and positions of markers, were arbitrarily selected in most studies; therefore, these inevitable problems were not considered. This means that, even if previous studies are replicated, it is difficult to obtain the same results (e.g., accuracy of registration) as previous studies because of human subjectivity in the process of placing markers.

In consideration of the above-mentioned problems, this paper proposes a method for placing multiple markers to provide accurate registration on a large object. The proposed method (1) divides a virtual object evenly considering the geometric discrepancy problem, (2) uses identifiable positions based on 3D interest points considering the relative position consistency problem, and (3) automatically determines the positions of multiple markers. This paper is organized as follows. Section 2 describes the proposed method in detail. Section 3 presents two experiments conducted to validate the proposed method, along with discussions, followed by conclusions in Section 4.

2. Proposed Method for Placing Multiple Markers

The proposed method consists of a preprocessing step (Section 2.1) to remove unnecessary geometric information from the mesh of the virtual object, and a scoring step (Section 2.2) to calculate scores that indicate which positions on the surface of the mesh are suitable for marker placement. In the proposed method, the position of the face among the components of the virtual object mesh is considered as a candidate for marker placement. This is because a surface is required to attach the marker directly to the real object in AR, as shown in Figure 1.

2.1. Preprocessing Step

This step converts the surface of the mesh precisely to consider various candidates and remove unnecessary candidates. This step is performed through the following processes: (1) upscaling to precisely convert the surface of the mesh, (2) eliminating the inner surfaces to remove unnecessary candidates, and (3) applying a region of interest (ROI) to remove candidates that are difficult to attach markers to or observe.

2.1.1. Upscaling

The surface of a precise mesh is composed of numerous small faces (hereafter the face is denoted as f, the faces are denoted as f′s, and the set of faces is denoted as F). Considering this, f is sliced so that every edge is smaller than the threshold $\epsilon$ (hereafter the edge is denoted as e, and the set of edges is denoted as E). In this paper, 1% of the mesh scale expressed in the norm was arbitrarily used for the $\epsilon$ value to be sufficiently precise relative to the overall mesh scale, as expressed in Equation (1).

$$\epsilon =\frac{\sqrt{{\left\{\max \left(\mathit{X}\right)-\mathrm{m}\mathrm{i}\mathrm{n}\left(\mathit{X}\right)\right\}}^2+{\left\{\max \left(\mathit{Y}\right)-\mathrm{m}\mathrm{i}\mathrm{n}\left(\mathit{Y}\right)\right\}}^2+{\left\{\max \left(\mathit{Z}\right)-\mathrm{m}\mathrm{i}\mathrm{n}\left(\mathit{Z}\right)\right\}}^2}}{100}$$

(1)

where X denotes $\left\{x | x \mathrm{v}\mathrm{a}\mathrm{l}\mathrm{u}\mathrm{e}\mathrm{s} \mathrm{o}\mathrm{f} \mathrm{v}\mathrm{e}\mathrm{r}\mathrm{t}\mathrm{i}\mathrm{c}\mathrm{e}\mathrm{s}\right\}$, Y denotes $\left\{y | y \mathrm{v}\mathrm{a}\mathrm{l}\mathrm{u}\mathrm{e}\mathrm{s} \mathrm{o}\mathrm{f} \mathrm{v}\mathrm{e}\mathrm{r}\mathrm{t}\mathrm{i}\mathrm{c}\mathrm{e}\mathrm{s}\right\}$, Z denotes $\left\{z | z \mathrm{v}\mathrm{a}\mathrm{l}\mathrm{u}\mathrm{e}\mathrm{s} \mathrm{o}\mathrm{f} \mathrm{v}\mathrm{e}\mathrm{r}\mathrm{t}\mathrm{i}\mathrm{c}\mathrm{e}\mathrm{s}\right\}$, max is a function that finds the maximum value in a set, and min is a function that finds the minimum value in a set.

If some of the e constituting a specific i th f (hereafter ${\mathbf{f}}_{\mathit{i}}$) is longer than $\epsilon$, the existing ${\mathbf{f}}_{\mathit{i}}$ is sliced into four similar triangles using the triangle mid-segment theorem. Subsequently, new vertices and f′s are added accordingly (hereafter the vertex is denoted as v, the vertices are denoted as v′s, and the set of vertices is denoted as V). This operation is repeated for each newly added f, and then duplicate v′s are removed. Figure 2 shows an example of this process performed on a car model, which is a specific virtual large object.

Figure 2a, where this process is not applied, is not precise enough to recognize the shapes of f′s on the surface (e.g., the glass part). On the other hand, Figure 2b, where this process is applied, is shown to have a surface so precise that it is difficult to confirm whether it is composed of f′s.

2.1.2. Eliminating Inner Faces

A virtual large object may be a combination of more than two small- or medium-sized objects. For example, if a virtual large object is a car, it might be a combination of the following objects: side outer, fender, handle, tire, etc. In this case, since the marker must be attached to the outer surface of the real object, the inner f′s are excluded from the candidates.

When rays are emitted from the center of ${\mathbf{f}}_{\mathit{i}}$ and its constituent v′s with its normal direction, if one is hit by other f′s, ${\mathbf{f}}_{\mathit{i}}$ is determined to be an inner surface; otherwise, it is determined to be an outer surface. Figure 3 shows an example for determining whether the surfaces are inner or outer. The f′s determined to be the inner surface are removed because they are not suitable positions for placing markers, and then unused v′s are removed accordingly.

2.1.3. Applying ROI

If the marker is placed too high or too low, it is difficult to observe through AR devices. In consideration of this, a person′s knee height (hereafter $h_{knee}$) to eye height (here after $h_{eye}$) is set as the ROI. The $h_{knee}$ and $h_{eye}$ are the median measurements of males and females of ages 20s–50s from the Korean anthropometric survey conducted by the Korean Agency for Technology and Standards (

Table 1. Mean knee heights of males and females in their 20s–50s (unit: mm).

Age	Male	Female
20s	442.662	408.077
30s	438.340	397.864
40s	432.436	397.931
50s	423.298	392.718

and

Table 2. Mean eye heights of males and females in their 20s–50s (unit: mm).

Age	Male	Female
20s	1615.147	1489.162
30s	1595.316	1464.283
40s	1571.064	1451.561
50s	1547.258	1434.260

).

If the vertical value (generally y) of all v′s constituting ${\mathit{f}}_{\mathit{i}}$ is within the ROI, ${\mathit{f}}_{\mathit{i}}$ is maintained. Otherwise, ${\mathit{f}}_{\mathit{i}}$ is removed and new f′s are created using v′s within the ROI and positions ($p^{\prime }, p^{″}$) at the boundary, as shown in Figure 4a. The creation of new f′s depends on the number of v′s outside the ROI, as shown in Figure 4b. After removing the unused v′s, the preprocessing is complete.

2.2. Scoring Step

In this step, the score for each candidate is calculated to determine which of all candidates of the preprocessed mesh is suitable as a position for the marker placement (hereafter MPS score, Marker Placement Suitability score). The proposed method automatically places the marker in the candidate with the highest calculated score. Based on the conditions for reducing the registration error due to the inevitable problems mentioned in the introduction, two hypotheses were considered for the calculation of the score.

Hypothesis 1.

From each area of an evenly divided virtual object, the face closest to the center of the area (CoA) would be the most suitable candidate. This is because the closer the markers are to the CoA, the more evenly they are placed.

Hypothesis 2.

Face with more interest points around it would be the most suitable candidate. This is because the more interest points around it, the easier it is to identify.

This step is performed through the following processes: (1) centrality computation of segmented area representing scores for candidates based on hypothesis 1, (2) density computation of interest points representing scores for candidates based on hypothesis 2, and (3) calculation of the MPS scores by considering these comprehensively.

2.2.1. Centrality of Segmented Area

There are various conventional methods for segmenting meshes, but most of the purposes are to classify them into meaningful parts as [16,17,18]. Among these segmentation methods, the main purpose of K-means clustering is to divide into k parts rather than to classify into meaningful parts [19,20]. Therefore, this paper considers K-means clustering as a method for segmenting meshes.

To compensate for the randomness of this method, K-means++ is used. This method ensures that the centers of the initial clusters are the farthest away from each other, resulting in a more even division. Clustering is performed using V, and the cluster of specific ${\mathbf{f}}_{\mathit{i}}$ is determined to be the highest percentage cluster in the three v′s that constitutes it. Figure 5 shows the segmentation result using K-Means++ after preprocessing on a specific virtual large object (a car model).

From Hypothesis 1, the specific i th f in the j th area among the segmented areas (hereafter this f is denoted as ${\mathbf{f}}_{\mathit{j},\mathit{i}}$) is placed more evenly as it is closer to the j th CoA, making it an appropriate candidate. But the f′s in the j th area (hereafter the set of these f′s is denoted as ${\mathit{F}}_{\mathit{j}}$) have three-dimensional geometric information, the mean position of ${\mathit{F}}_{\mathit{j}}$ may not represent the CoA. For example, if ${\mathit{F}}_{\mathit{j}}$ is shaped like a sphere convexly protruding from a circular plane, as shown in Figure 6, the CoA is expected to be the top of the sphere. However, the mean position of ${\mathit{F}}_{\mathit{j}}$ is located below the center of the sphere. As a result, the f closest to this position is not the closest to the expected CoA.

To solve this problem, the dimensionality of ${\mathit{F}}_{\mathit{j}}$ is reduced to ${\mathbb{R}}^2$ using principal component analysis. Dimension reduction is performed by excluding the smallest eigenvector among eigenvectors for V. Based on this, the CoA is defined as the mean position of ${{\mathit{F}}^{\mathit{\prime }}}_{\mathit{j}}=\left\{{{\mathbf{f}}^{\mathbf{\prime }}}_{\mathit{j},\mathit{i}}|{{\mathbf{f}}^{\mathbf{\prime }}}_{\mathit{j},\mathit{i}}\in {\mathbb{R}}^2, 1\le i\le \mathrm{n}({\mathit{F}}_{\mathit{j}})\right\}$, expressed as Equation (2).

$$\mathrm{C}\mathrm{o}\mathrm{A}\left({\mathit{F}}_{\mathit{j}}\right)=\frac{1}{n}\sum _{i=1}^n\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\left({{\mathbf{f}}^{\mathbf{\prime }}}_{\mathit{j},\mathit{i}}\right)$$

(2)

where n denotes $\mathrm{n}\left({\mathit{F}}_{\mathit{j}}\right)$, ${\mathbf{f}\mathbf{\prime }}_{\mathit{j},\mathit{i}}$ denotes ${\mathbf{f}}_{\mathit{j},\mathit{i}}$ dimensionally reduced to ${\mathbb{R}}^2$, and position is a function that obtains the position of ${\mathbf{f}\mathbf{\prime }}_{\mathit{j},\mathit{i}}$.

The farther ${\mathbf{f}}_{\mathit{j},\mathit{i}}$ is from $\mathrm{C}\mathrm{o}\mathrm{A}\left({\mathit{F}}_{\mathit{j}}\right)$, the less suitable it is as a candidate, and the closer it is, the more suitable it is as a candidate. We define the degree of this suitability as centrality of segmented area in this paper (hereafter CSA), as expressed by the normalized value in Equation (3).

$$\mathrm{C}\mathrm{S}\mathrm{A}\left({\mathbf{f}}_{\mathit{j},\mathit{i}}\right)=1-\frac{‖\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\left({{\mathbf{f}}^{\mathbf{\prime }}}_{\mathit{j},\mathit{i}}\right)-\mathrm{C}\mathrm{o}\mathrm{A}\left({\mathit{F}}_{\mathit{j}}\right)‖}{\mathrm{m}\mathrm{a}\mathrm{x}\left(\forall {\mathbf{f}}^{\mathit{″}}\in {{\mathit{F}}^{\mathit{\prime }}}_{\mathit{j}}, ‖\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\left({\mathbf{f}}^{\mathbf{″}}\right)-\mathrm{C}\mathrm{o}\mathrm{A}\left({\mathit{F}}_{\mathit{j}}\right)‖\right)}$$

(3)

where ${{\mathit{F}}^{\mathit{\prime }}}_{\mathit{j}}$ denotes ${\mathit{F}}_{\mathit{j}}$ dimensionally reduced to ${\mathbb{R}}^2$, and ${\mathbf{f}}^{\mathbf{″}}$ denotes f dimensionally reduced to ${\mathbb{R}}^2$ by iterating through ${{\mathit{F}}^{\mathit{\prime }}}_{\mathit{j}}$.

Figure 7 shows the CSA computed by Equation (3) at a specific ${\mathbf{f}}_{\mathit{j},\mathit{i}}$.

2.2.2. Density of Interest Points

The 3D interest points are characteristic positions extracted from the three-dimensional geometric information of the mesh (hereafter 3D interest point is referred to as interest point). There are various methods for extracting interest points [21,22,23], most of them extract partial features by analyzing the overall geometry of the mesh. This means that if interest points are extracted from a large object, various features may not be reflected from the perspective of the segmented area.

Therefore, Mesh Saliency, which computes the degree of saliency for every vertex, is used to extract interest points [21]. The threshold for extracting interest points is the mean value of the total saliency at all vertices, as in [24]. In addition, because it is a large object, the local maxima are not considered to derive the interest points as much as possible. Figure 8 shows interest points extracted after preprocessing car models, which are specific virtual large objects.

From Hypothesis 2, the more densely the interest points are around ${\mathbf{f}}_{\mathit{j},\mathit{i}}$, the more likely it is to be a suitable candidate. We define the degree of this suitability as density of interest points in this paper (hereafter DoI), which is expressed as Equation (4).

$$\mathrm{D}\mathrm{o}\mathrm{I}\left({\mathbf{f}}_{\mathit{j},\mathit{i}}\right)=\forall \mathrm{p}\in \mathrm{i}\mathrm{n}\mathrm{t}\mathrm{e}\mathrm{r}\mathrm{e}\mathrm{s}\mathrm{t}\left({\mathit{F}}_{\mathit{j}}\right), \sum \left\{\begin{array}{c}1, ‖\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\left({\mathbf{f}}_{\mathit{j},\mathit{i}}\right)-\mathrm{p}‖<r\\ 0, ‖\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\left({\mathbf{f}}_{\mathit{j},\mathit{i}}\right)-\mathrm{p}‖\ge r\end{array}\right.$$

(4)

where $\mathrm{i}\mathrm{n}\mathrm{t}\mathrm{e}\mathrm{r}\mathrm{e}\mathrm{s}\mathrm{t}\left({\mathit{F}}_{\mathit{j}}\right)$ denotes all the interest points in ${\mathit{F}}_{\mathit{j}}$, p denotes a point iterating through these interest points, and r is the radius for determining whether the interest points are around ${\mathbf{f}}_{\mathit{j},\mathit{i}}$ (in this paper, twice the value of $\mathsf{\epsilon }$ in the preprocessing is used).

As with the CSA represented by normalization, the DoI is also normalized and is expressed as in Equation (5).

$$\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{z}\mathrm{e}\left(\mathrm{D}\mathrm{o}\mathrm{I}\left({\mathbf{f}}_{\mathit{j},\mathit{i}}\right)\right)=\frac{\mathrm{D}\mathrm{o}\mathrm{I}\left({\mathbf{f}}_{\mathit{j},\mathit{i}}\right)}{\mathrm{m}\mathrm{a}\mathrm{x}\left(\forall {\mathbf{f}}^{\mathit{″}}\in {\mathit{F}}_{\mathit{j}}, \mathrm{D}\mathrm{o}\mathrm{I}\left({\mathbf{f}}^{\mathit{″}}\right)\right)}$$

(5)

where ${\mathbf{f}}^{\mathit{″}}$ denotes a f iterating through ${\mathit{F}}_{\mathit{j}}$.

Figure 9 shows the normalized DoI computed by Equation (5) at a specific ${\mathbf{f}}_{\mathit{j},\mathit{i}}$.

2.2.3. MPS Score

The MPS score, which indicates the suitability of a specific ${\mathbf{f}}_{\mathit{j},\mathit{i}}$ as a position for placing a marker, is expressed as Equation (6) using a weighted mean of both Equations (3) and (5) derived from each hypothesis.

$$\mathrm{M}\mathrm{P}\mathrm{S}\left({\mathrm{f}}_{j,i}\right)=\left(1-w\right)·\mathrm{C}\mathrm{S}\mathrm{A}\left({\mathrm{f}}_{j,i}\right)+w·\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{z}\mathrm{e}\left(\mathrm{D}\mathrm{o}\mathrm{I}\left({\mathrm{f}}_{j,i}\right)\right)$$

(6)

where MPS denotes a function that calculates the MPS score at ${\mathbf{f}}_{\mathit{j},\mathit{i}}$, and w denotes the weight used in the weighted mean.

Multiple markers are placed at f′s with the highest MPS score in each area. Figure 10 shows the MPS scores calculated according to the number of segments k and weight w on a specific virtual large object, the car model.

In Figure 10, green color indicates a high MPS score, and white color indicates a low MPS score. When no weights are considered (w = 0, second column in Figure 10a–c), it is shown that the closer to the CoA of each segmented area, the higher the MPS score, and the further away from the CoA, the lower the MPS score. On the other hand, when only the weights are considered (w = 1, fourth column in Figure 10a–c), the featured positions of each segmented region have higher MPS scores regardless of the CoA, and lower MPS scores otherwise. It is also shown that the number of segments (k) affects the MPS score because the segmented areas and CoAs are changed by k.

3. Experiment and Discussion

3.1. Experimental Environment and Methodology

In this paper, a preliminary experiment to determine a valid weight w (described in Section 3.2) and a main experiment to validate the proposed method (described in Section 3.3) were performed. In this section, we introduce the experimental environment and methodology used in both experiments.

3.1.1. Experimental Environment

Both experiments were conducted using Microsoft HoloLens 2, which is capable of spatial awareness of surroundings [25]. This device provides a more reliable camera pose estimation than popular AR devices, such as ARKit and ARCore [26]. In addition, this device is capable of robust pose estimation (for example, in [27], the mean error between a real and captured space when moving about 287 m in an indoor environment was 2.4 cm).

The experiments were conducted in an indoor parking lot, as shown in Figure 11, to ensure stable illumination. Because the illumination of this environment is determined by indoor lighting, it is relatively more stable than outdoor environments, which change in real time due to the sun, clouds, etc.

The experiments were conducted after sunset and before sunrise to maintain constant lighting. The civil twilight period was also excluded to minimize the influence of sunlight. The illuminance of the experimental location was measured between 100 and 200 lx during these times, and HoloLens′s ability to augment virtual objects does not degrade at these illuminations according to a previous study [28].

• Real objects

In the experiments, cars, which are commonly seen in everyday life, were used as large objects. Two types of cars with different sizes were used as large objects, as shown in Figure 12: Kia-Morning (=Picanto) 3rd generation 1st facelift (hereafter DM) and Hyundai-Avante 6th generation early model (hereafter DA). The specifications for each real object are listed in

Table 3. Specifications of real objects (unit: mm).

Type	${\mathbf{D}\mathbf{M}}_{\mathbf{R}}$	${\mathbf{D}\mathbf{A}}_{\mathbf{R}}$
Overall length	3595	4570
Overall width	1595	1800
Overall height	1485	1440

.

2.

Virtual objects

We used models created by measuring actual cars, from the 3D model providers CGTrader [29] and Hum3D [30], as virtual objects for each real object. CGTrader is the world′s largest supplier of 3D models, and it has been used in many studies. In particular, Ref. [31] studied autonomous driving using a real car and a corresponding virtual object in the CGTrader. Hum3D is a company that aims to create realistic modeling for applications such as AR and visualization, and 3D experts examine whether the modeling results match actual objects. This company is primarily used in research on cars [32,33,34].

As virtual objects, 3D models identical to the car models (e.g., Kia-Picanto 3rd generation 1st facelift) of the real objects were used [35,36]. To verify the equivalence of the virtual models to the real objects, it was checked whether they met the tolerances in Article 115 “Tolerance of Specifications” of the “MOLIT Ordinance: Regulations for Performance and Safety Standards of Motor Vehicle and Vehicle Parts”.

The results showed that they all met the tolerances, as listed in

Table 4. Comparison of ${\mathrm{D}\mathrm{M}}_{\mathrm{R}}$ and ${\mathrm{D}\mathrm{M}}_{\mathrm{V}}$.

DM	Dimensions
	Width	Height	Length
Real (mm)	1595	1485	3595
Virtual [35]	1596.99	1511.19	3606.88
Error (mm)	1.99	26.19	11.88
Tolerance (mm)	±30	±50	±40
Error rate (%)	0.125	1.764	0.330

where an object with the subscript V denotes a virtual object.

and

Table 5. Comparison of ${\mathrm{D}\mathrm{A}}_{\mathrm{R}}$ and ${\mathrm{D}\mathrm{A}}_{\mathrm{V}}$.

DA	Dimensions
	Width	Height	Length
Real (mm)	1800	1440	4570
Virtual [36] (2×)	1803.89	1456.39	4558.64
Error (mm)	3.89	16.39	11.36
Tolerance (mm)	±40	±60	±50
Error rate (%)	0.216	1.138	0.249

(note that Ref. [36] was created at a 1/2 scale, so the comparison was performed after scaling up to 2×). Except for the overall height, all the error rates were less than 1%. This is likely due to the influence of consumable components such as tires on this value.

Because they are virtual models, they may differ from real objects owing to options and consumables, as well as the geometric discrepancy problem. For this reason, consumable wipers, tires, and license plates were removed from the virtual models, and the issue of options is considered as the geometric discrepancy problem in this paper.

3.1.2. Methodology

• Content for augmentation and measurement

To validate the proposed method through experiments, content that augmented the virtual object based on multiple markers was developed. The content was developed by using the 3D content development tool ‘Unity (2020.3.12f1 version)’ and the officially supported HoloLens 2 content development library ‘mixed reality tool ‘kit (2.7.3 version)’.

The developed content (a) augmented a virtual sphere at the index fingertip in real time through a pointing gesture with the right hand (HoloLens 2 has the highest hand tracking accuracy when the pointing gesture is maintained in the right hand [37,38]), and (b) placed a virtual marker at the position of the augmented sphere by controlling a wireless mouse with the left hand, as shown in Figure 13.

In a simple test using the real right hand and a physical marker, it was confirmed that when the virtual sphere augmented at the index fingertip was positioned at the end of the marker, an error occurred approximately 0.4 to 1.5 cm, as shown in Figure 14. This error is considered as the relative position consistency problem in this paper.

The virtual marker was then placed on the real object by referring to a marker position image recorded for the virtual object, as shown in Figure 15. When the placement of the markers was completed, the content verified whether all the markers were at the initial input positions. If any markers were different from the initial position, the content allowed to reposition them using the right index finger and mouse.

Once it was confirmed that all the markers were placed correctly, the developed content performed registration based on the iterative closest point (ICP) using the relationship between the placed markers and the markers positioned relative to the virtual object, and finally augmented the virtual object. Figure 16 shows the results of the augmented registration on the DM and DA.

The developed content added a World Anchor property to the virtual object when the augmentation and registration of the virtual object were completed. This property is a technology that maintains the position and rotation of virtual objects placed in the real world, even if camera pose estimation errors occur and accumulate in markerless environment [39,40,41,42]. HoloLens 2 uses the perceived surrounding space and sensors on the device to anchor a virtual object to the real world. Figure 17 shows the spatial awareness of the real object. Then errors, which are the registration performance of the augmented virtual object, were measured using this content.

2.

Method for measuring registration error

Before the experiments, the surrounding spaces of the real object were collected for approximately 30 min so that the HoloLens 2 could sufficiently recognize them. Figure 18 shows the process of collecting the surrounding spaces.

The reference points for measuring the error between the position of the real object and the position of the virtual object (hereafter, registration error) were selected at positions that were clearly recognizable to the naked eye, as shown in Figure 19.

Four reference points were selected, as shown in Figure 20a, and each reference point is described in

Table 6. Detailed description of reference points used to measure registration errors.

Point	Description
a	The top end of the panel where the door connects to the fender (front, left).
b	The top end of the panel where the door connects to the fender (front, right).
c	The top end of the panel where the door connects to the fender (back, left).
d	The top end of the panel where the door connects to the fender (back, right).

. The registration error was measured by recording the difference between the position of the reference point on the real object and the reference point on the virtual object, when pointing with the right index fingertip at each reference point on the real object and controlling the mouse with the left hand. Figure 20b shows the process of measuring registration errors.

3.2. Preliminary Experiment

Before validating the performance of the proposed method, the preliminary experiment was conducted to determine the valid value of the weight. Its procedure is shown in Figure 21, where the number of markers k and weight w were considered as follows.

• The number of markers k

Previous studies on marker-based AR mentioned that at least three markers are required to solve the perspective-n-point (PnP) problem for camera pose estimation [7,43]. The ICP algorithm, which is primarily used for registration with multiple markers, also requires at least three markers [44]. Meanwhile, previous studies on markerless AR, which is relatively free from the PnP problem, often used more than three markers for the initial registration [45,46,47]. Therefore, in consideration of previous studies, the number of markers was used in this experiment from 4 to 2x as many as 8.

2.

Weight w

The weights in this experiment were used with a total of 5 values, from 0 which excludes DoI to 1 which excludes CSA, split into quartiles:0, ¼, ½, ¾, and 1.

This experiment was conducted by changing k and w and measuring the registration errors between the real object and the augmented virtual object. This process was repeated twice.

Results of Preliminary Experiment

As a result of the preliminary experiment, the mean of the registration errors (hereafter MRE), which are measured through all reference points, was calculated in Equation (7), as shown in Figure 22.

$$\mathrm{M}\mathrm{R}\mathrm{E}=\frac{1}{n}\sum ‖{\mathrm{p}}_{\mathrm{r}}-{\mathrm{p}}_{\mathrm{v}}‖$$

(7)

where n denotes the number of reference points, ${\mathrm{p}}_{\mathrm{r}}$ denotes the measured position of the reference point on the real object, and ${\mathrm{p}}_{\mathrm{v}}$ denotes the measured position of the reference point on the virtual object.

• Impact of weights

It was assumed that the MRE would be measured the least when a certain weight was used, regardless of the number of markers. Accordingly, the analysis was performed using 5 weight groups with a total of 20 MRE: 2 (number of models) × 5 (number of markers) × 2 (number of iterations).

The Shapiro–Wilk test was performed to examine whether each weight group followed a normal distribution (with a 95% confidence interval), and the results are listed in

Table 7. Shapiro–Wilk test results for the five weight groups.

w	Statics	p-Value
0	0.796	0.001
0.25	0.830	0.003
0.5	0.637	7.28 × 10−6
0.75	0.807	0.001
1	0.838	0.003

.

As a result of this test, p-values for all groups were less than 0.05, indicating that the normal distribution was not valid. Therefore, the non-parametric Kruskal–Wallis test was performed to confirm significant differences between the groups (with a 95% confidence interval). This test confirmed a statistically significant difference between the weight groups (H = 35.772, p = 3.22 × 10⁻⁷ < 0.001). As listed in

Table 8. Means and standard deviations of the five weight groups.

w	M (mm)	STD (mm)
0	3.271	0.978
0.25	2.959	1.065
0.5	1.950	0.578
0.75	1.683	0.331
1	2.130	0.475

The gray row indicates the lowest MRE (M/STD) when the w is 0.75.

, it was confirmed that the MRE was the lowest when the weight was 0.75, and accordingly, the proposed method used w = 0.75 as the default value.

2.

Impact of the number of markers

Because many previous studies indicated that the registration error decreases as the number of markers increases, an additional analysis was performed to confirm whether it also holds true for w = 0.75. For this analysis, five groups based on the number of markers were used, each consisting of four MRE: 2 (number of models) × 2 (number of iterations). Owing to the small number of data, the non-parametric Kruskal–Wallis test was performed. As a result of this test, there was no statistically significant difference between the groups (H = 9.300, p = 0.054 > 0.05), but this is likely due to the small sample size of the MRE comprising each group.

3.3. Main Experiment

The main experiment was conducted to validate that the performance of the multiple markers determined by the proposed method with the weight 0.75 is more effective for registration than that of multiple markers placed by the conventional methods with human subjectivity. The experimental procedure is shown in Figure 23.

Participants for the subjective placement of multiple markers were randomly recruited from among males and females in their 20s who were enrolled in or graduated from the Korea University of Technology and Education. Recruitment was conducted until the total number of participants was at least five, the minimum number used for evaluation in the field of Human–Computer Interaction [48], and a total of 10 participants were recruited (male: 8, female: 2).

Before this experiment, each participant was introduced to the purpose of the experiment and the role of the markers in AR. Each participant then placed multiple markers using a subjective marker placement program (described in Section 3.3.1) and was rewarded to encourage participation. In this experiment, the number of markers used was the minimum (4), median (6), and maximum (8) of the number of markers which were used in the preliminary experiment.

An analysis to validate the performance was performed by comparing 60 MRE from the multiple markers set by the participants, with 60 MRE from multiple markers determined by the proposed method.

3.3.1. Subjective Marker Placement Program

As shown in Figure 24, the program, which was provided to the participants, had functions of manipulating (moving, rotating, and zooming) the visualized virtual models with the mouse and functions of placing markers where it was deemed suitable. To prevent participants from making mistakes in this process, the creation and removal of markers were simply performed using left- and right-clicks in this program. In the process of being guided before the experiment, the participants were also introduced to how to use the program.

3.3.2. Results of Main Experiment

The respective MRE, which were measured 10 times after placing the markers using both the conventional subjective method (hereafter ${\mathrm{M}\mathrm{R}\mathrm{E}}_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{a}\mathrm{l}}$) and the proposed method (hereafter ${\mathrm{M}\mathrm{R}\mathrm{E}}_{\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{e}\mathrm{d}}$), are shown in Figure 25, and it was confirmed that the proposed method has relatively less volatility.

• Performance analysis for registration accuracy

An analysis was performed to confirm whether the proposed method significantly reduced the MRE compared with the conventional method. The total MRE, which included the models (DM and DA), number of markers, and number of iterations, was used to confirm the normal distribution of the MRE for each method. The results of the Shapiro–Wilk test for the total MRE for each method are listed in

Table 9. Shapiro–Wilk test results for the total MRE.

Method	Statics	p-Value
conventional	0.862	6.95 × 10−6
proposed	0.470	2.16 × 10−13

(with a 95% confidence interval), and it was confirmed that the normal distribution was not valid.

Therefore, the non-parametric Mann–Whitney U test was performed to confirm significant differences between the two groups (with a 95% confidence interval). This test confirmed that the proposed method had a significantly lower MRE than the conventional method, regardless of the number of markers. Figure 26 shows the mean differences and significance intervals between the conventional and the proposed methods.

2.

Performance analysis according to the number of markers

To confirm whether the performance of the proposed method improved as the number of markers increased, as in previous studies, ${\mathrm{M}\mathrm{R}\mathrm{E}}_{\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{e}\mathrm{d}}$ according to k was analyzed. As a result of the Kruskal–Wallis test between groups separated by each k (with a 95% confidence interval), it was confirmed that statistically significantly more accurate registration is performed as k increases (H = 10.344, p = 0.006 < 0.01).

Table 10. Overall comparison between ${\mathrm{M}\mathrm{R}\mathrm{E}}_{\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{e}\mathrm{d}}$ by number of markers k.

k	M (mm)	STD (mm)
4	2.184	0.662
6	1.770	0.298
8	1.581	0.328

lists the results of this test, which are consistent with results of previous studies.

To confirm whether this result was equally applicable to all models, ${\mathrm{M}\mathrm{R}\mathrm{E}}_{\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{e}\mathrm{d}}$ was split into DM and DA, and this analysis was repeated (with a 95% confidence interval). The results are listed in

Table 11. Individual comparisons between ${\mathrm{M}\mathrm{R}\mathrm{E}}_{\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{e}\mathrm{d}}$ by number of markers k (mean ± standard deviation).

k	DM (p = 5.23 × 10−5)	DA (p = 0.393)
4	2.802 ± 0.292	1.566 ± 0.165
6	1.966 ± 0.258	1.573 ± 0.185
8	1.691 ± 0.394	1.471 ± 0.189

. The MRE seemed to decrease as k increased, but there was no statistically significant difference in the DA (DM: H = 19.719, p = 5.23 × 10⁻⁵ < 0.001, DA: H = 1.866, p = 0.393 > 0.05).

3.4. Discussion

It was verified that, to determine the position of multiple markers, the proposed method performed better than the conventional method which is subjectively determined by humans.

The weight determined by the preliminary experiment (w = 0.75) means that the DoI is more effective than the CSA to minimize the inevitable problems that arise in AR. It was also confirmed that the registration error decreased as the proportion of the DoI increased. This means that the more interest points, the easier for humans or sensors to recognize. However, the registration error increased when the CSA was completely excluded (w = 1). This means that even a small proportion must be reflected to minimize the geometric discrepancy problem.

When the minimum number of markers (k = 4) was used, ${\mathrm{M}\mathrm{R}\mathrm{E}}_{\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{e}\mathrm{d}}$ was approximately 59.8% less than ${\mathrm{M}\mathrm{R}\mathrm{E}}_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{a}\mathrm{l}}$, and the reduction in error increased as k increased. This means that the proposed method can be applied in various ways, from environments using a small number of multiple markers to environments using a large number of multiple markers.

In the performance comparison analysis of the proposed method using the number of markers, as in results of previous studies, the error decreased significantly as the number of markers increased. However, a significant reduction in the specific object DA was not observed. This means that the using only four markers in DA is no different than using six or eight, so it is not necessary to use many markers; and this result means that there may be an optimal number of markers depending on the geometry of a specific model.

Lastly, because the proposed method automatically determines the positions of multiple markers used for registration, it can be used as a comparison group in all subsequent and extended experiments to compare registration performance.

4. Conclusions

This paper proposed a multiple marker placement method to minimize the inevitable problems when placing markers in AR and validated its performance. The difference between the proposed and conventional methods is that, unlike the conventional method that randomly positions the markers, the proposed method automatically determines the marker positions by reflecting the geometric features of the virtual object.

The proposed method calculates the CSA and DoI within the segmented area by dividing the virtual object as evenly as possible and using the interest points. To consider all of these, the MPS score formula based on the weighted mean was derived, and the scores for all faces were calculated. The positions of multiple markers were determined at the center of the face with the highest MPS score in each segmented area.

In the proposed method, the weight for calculating the MPS score was determined to be 0.75 through the preliminary experiment. The registration error was measured after augmenting the virtual object by changing the weight ratio of the proposed method in the preliminary experiment. As a result of the preliminary experiment, it was confirmed that the CSA should not be completely excluded, but the DoI should be considered more.

The performance of the proposed method was validated by comparing it with the conventional method in which markers were subjectively placed. It was confirmed that the proposed method performs better than the conventional method as follows. (1) The proposed method can provide more consistently accurate registration than the conventional method. (2) Regardless of the number of markers, the proposed method provides more accurate registration.

This means that the proposed method can be applied in various ways, from environments using a small number of multiple markers to those using a large number of multiple markers, and it can become a baseline for placing multiple markers in research related to registration.

The advantage of the proposed method is that it can replace the subjective human judgment when positioning multiple markers. Since human subjective judgment is highly volatile depending on the surrounding conditions or environments, it may have limitations in consistency and accuracy. However, the proposed method can ensure consistent performance and accuracy as it determines the positions of multiple markers by considering the CSA and the DoI of the object.

The following studies are considered in future works. First, as the methods for extracting the interest points of virtual objects continue to evolve, studies are required to compare the performance of the proposed method using various interest point extraction methods.

Second, a study is needed to verify the effect of the two thresholds used in the proposed method on registration performance based on the MPS Score. The proposed method used threshold $\epsilon$ for upscaling in the preprocessing step, and threshold 2$\epsilon$ for calculating the DoI in the scoring step. Since these two thresholds are involved in the calculation of the MPS score, the effect of the threshold values should be closely examined in future research.

Third, further research is needed to perform multifaceted sensitivity analyses by applying the proposed method to multiple large objects. Since the proposed method was carried out on two specific models with similar sizes, the set weight may be overfitting for vehicles. This implies that the parameters or formulas of the proposed method can be improved depending on the type or size. To compensate for this, it is necessary to apply the proposed method to multiple large objects of various types, not limited to vehicles, and to perform sensitivity analyses.

Fourth, further research should be conducted to compare registration based on the proposed method with registration based on markerless methods. The registration performance of the proposed method was confirmed by comparing it with only the subjectively conventional method. However, it was not compared to markerless-based registration methods.

Finally, the K-means++ clustering used in this paper for area segmentation complements the K-means clustering to minimize randomness, but it is not completely resolved. This means that, although the results from the proposed method are mostly similar, they do not always guarantee the same results. Therefore, it is necessary to improve the method for segmenting areas to ensure that the same results can always be derived.