Utilizing Stereography to Compare Cultural Heritage in the Past and Now: An Interactive AR Application

Abstract

Due to the astonishing speed in technological progress for new heterogeneous experiences, the Mixed Reality continuum represents nowadays an invaluable tool for cultural heritage applications, and the possibility of seemingly natural interaction with virtual elements is paving the way for more immerse, ve and deeply enjoyable experiences, both in dedicated spaces, such as museums and exhibitions, and in public spaces via the latest mobile AR technologies. This paper illustrates the complete workflow followed for the design and implementation of an Augmented Reality application on a holographic table, along with critical latency-quality tradeoffs encountered when combining both high-fidelity surveys and latency-sensibility devices and experiences. A workaround is provided in order to mitigate the impact of low-importance models on the overall latency utilizing a software toolchain employing Google Maps/Earth web applications; each software comprises the toolchain Free and Open-Source Software (FOSS) for a more adaptive workaround with a wider range of suitable applications. The application is then tested with the students of LaborA, which is the modeling laboratory of “Politecnico di Milano”.

1. Introduction

In the field of cultural heritage, the study of digitization and surveying of architecture is essential for understanding their integral role within broader contexts and the formation of landscape concepts [1]. However, effectively preserving and transmitting these contexts to future generations via digitization is not always straightforward [2]. The utilization of 3D spaces, particularly via immersive technologies like Virtual Reality (VR) or Extended Reality (XR), still faces significant challenges due to limitations in bandwidth and hardware capabilities [3,4,5]. Achieving satisfactory immersive conditions remains complex, and certain areas of cultural heritage are particularly vulnerable to risks such as climate change, which may lead to their disappearance or severe damage [6,7,8,9].

The spatiality of natural features like parks, waterways, rural areas, and vast environments that are moderately influenced by human activities present unique challenges. Existing technologies may not always leverage the most advanced tools for digitizing open spaces. Although mobile laser scanning and drone-based methods are viable solutions, they pose considerable economic and post-processing challenges for researchers and digitization practitioners. Additionally, these approaches may not always necessitate the same level of accuracy as architectural digitization but rather require an effective restitution, even if it is slightly less precise.

To address these issues, this research proposes the application of open-source XR methodologies, including surveys conducted with proprietary software, on a large scale for the reconstruction of a monumental park, and on a more specific scale for digitizing selected artifacts to be used in holographic mode. The objectives of this study are as follows:

• Develop a toolchain that can automatically extract the constituent elements of a large environment, enabling the holographic projection of individual architectural objects within it;
• Create a holographic context environment that combines separate workflows for the overall context and for individual architectural objects;
• Enable the simultaneous immersive experience of a park-scale environment.

This paper provides a comprehensive description of all the technical operations necessary to design an immersive 3D model with a high level of detail of the main architecture located in the surrounding environment while minimizing processing time.

Background Researches

This paper is related to the stereometric perception of the projection of mesh models from previous processing and aims to focus mainly on describing the quality of the input data in the stereographic visualization and the end-user experience. In the first aspect, the formation of the stereometric image introduced in the experience of the representation of the existing comes from two different areas of training: on the one hand, the existing mesh derived from Google Maps, a low polymesh, and on the other hand, the mesh derived from the survey sessions operated with laser scanner instruments. In the second aspect, the subjective responses of users when faced with the joint representation of the two domains were evaluated. The literature in the field of Image Quality Assessment has been growing strongly in recent years [10] and includes using systems for assessing the responsiveness of an image to reality compared with both objective and subjective parameters related to the subject’s domains of perception related to visual experience.

In the field of Image Quality Assessment (IQA), existing metrics have been categorized from the beginning into two main methods [11]: objective and subjective methods. Objective methods provide a quantifiable index or numerical value obtained via computational algorithms, to automatically measure image quality. Subjective methods rely on human observers to evaluate the quality of an image based on their subjective perception.

Given that human observers are the ultimate recipients of the visual information conveyed by a digital model, subjective methods directly involving human observers were considered the most reliable approach in the proposed case study. A correct balance with respect to latency issues has been observed. The two meshes utilized in the general model are also very different; the results have been compared in terms of saving time in the construction of the immersive experience and appreciation of the human viewers with similar scenarios surveyed. Even if this approach is time consuming and not easily feasible for many scenarios, it has been considered the most efficient. The demand for objective quality metrics that can accurately correlate with the case study has too many variables to be considered in the case study and would result in an ungovernable system of parameters oscillating between the survey (3D registration of the real objects) [12], the treatment of meshes, the insertion of two different systems into one view, the use of the refraction system, the heaviness of the 3D model, and last but not least the need to deal with a 3D space where the model is projected [13].

There are comprehensive surveys available that focus on Screen Content Quality Assessment (SCQA), which can certainly serve as methodological references for investigating potential scientific judgment workflows. However, the reference to the two-dimensionality of the image and the display medium is still prevalent, even when considering an immersive environment, which inherently relies on a two-dimensional support system, in the case of VR [14,15]. Therefore, this paper aims to make the first small contribution to the examination of the perception of a stereometric environment organized with two different degrees of detail related to a park and monumental architectures and two different scales of objects. To accomplish this, an available tool has been used: a holographic table equipped with a specific mode of interaction with the projected object, and a wand used to manipulate, rotate, or scale the model. Without the tool, it is not possible to interact with the displayed object, despite this limitation being apparent [16].

2. Materials and Methods

Monza Park [17] is a historical monumental park located 11 Km from Milan; it has more than 7 square meters of green area, divided between lawn and woods, 110,000 tall trees, 13 farmhouses, 3 historic villas, 13 m of fences, and 90,000 visitors on spring Sundays [18]. It was established on 14 September 1805 by Emperor Napoleon with the purpose of making it a model agricultural estate and hunting preservation. It is one of Europe’s largest historic parks. It retains the original walled enclosure of about 14 km, making it the third largest walled park in Europe; with the Royal Gardens, it constitutes a complex of historical and architectural landscape value, having enhanced its interior with the restyling of farmsteads and some new construction by architects Canonica and Tazzini [19]. Above all, it has the dual layout of a romantic park, near the Royal Palace, and an agricultural park, which enhances its presence in the regional territory. In recent seasons, climate change has been putting a strain on the park’s green assets; periods of great drought have been occurring more and more frequently, and sudden strong winds have put a strain on the oldest plants. The preservation and enjoyment of a natural environment like this is as much a necessity as that of preserving the memory of a place that demonstrates all its fragility with each passing day. The proposed case study of the built here concerns a part of the park with a so-called Visconti Tower, designed by Luigi Canonica in the first two decades of the 19th century during his work in the gardens of the Villa Reale in Monza.

The historical map gadget allows users to compare the actual environment view with historical cartography and constitutes an important feature of cultural heritage. This feature enables users to explore the changes that have occurred over time in a particular location, allowing for a deeper understanding of the historical significance of the location and the cultural context in which it developed. By overlaying a historical map on the actual model of the location, users can compare the features of the location as they exist today with their historical counterparts. This allows for a visual comparison of changes that have occurred over time, such as the addition or removal of buildings, or changes in the layout of streets and public spaces.

This feature is essential in this application. By examining the changes that have occurred over time, users can gain insights into the forces that have shaped the location and the cultural context in which it developed. Furthermore, this feature provides an innovative and engaging way to explore cultural heritage. Rather than relying solely on textual descriptions or static images, users can interact with a dynamic and immersive model that highlights the historical significance and cultural context of the location.

2.1. Survey Workflow

The surveying phase (Figure 1) was developed using a Leica RTC360 laser scanner, a static laser able to acquire scans of 2 million points per second, and HDR image; using an internal VIS (Visual Inertial System), automated targetless field registration is possible. In this way, the phenomena of object splay, tapering, and out-of-plumb are accurately detected and returned.

After target-less pre-alignment in the proprietary Leica Cyclon FIELD (Release 2021.1.1) application, an additional alignment phase was performed in the proprietary Leica Cyclone Register 360 software (Release 2021.1.1), followed by a cleaning phase of the 10 setups. The outcome is a point cloud with a total of 461.086.484 points with an error of 0.003 m. The total amount of the point cloud can be described as follows: 461.086.484 points, 10 setups, and a bundle error of 0.003 m.

The automatic registration in the field without targets (based on VIS technology, Visual Inertial System where the SLAM (Simultaneous Localization and Mapping algorithm) computes the 3D coordinates of the tracked features from two or more positions and uses these coordinates for the determination of the following position) and the automated transfer of data to the office, further maximizing productivity while minimizing survey time. The survey was carried out in less than three hours and produced a point cloud exported in the proprietary software Leica Cyclon Register 360 and a mesh prepared in Leica 3DR (Release 2021.1) [20].

2.2. High Accuracy Surveys

Several point clouds decimated from this could be created to experiment with the most suitable and compliant one in terms of data heaviness and accuracy in describing the architectural element.

Several experiments have been developed to work out a workflow that is based on the development of NURBS to compare different geometries via modeling processes between manual and semi-automated ones based on the use of different algorithms, but the whole survey, with overlapping parts cleaned and vegetation erased, has been chosen.

2.3. Low Accuracy Survey Toolchain

While the workflow previously illustrated may yield high accuracy and quality models, their large sizes and dimensions may not be practical for many applications, particularly those with dynamic and latency-sensitive requirements. To address this limitation, this paper proposes a toolchain that leverages Free and Open-Source Software (FOSS) for surveys that prioritize lighter dimensions and smaller latency over top-notch accuracy. The toolchain consists of four main steps:

• Utilizing “Google Maps/Google Earth” as a viable compromise between model database quality and availability;
• Capturing the 3D models using the “RenderDoc (Release 1.19) ” injection functionality;
• Modifying the resulting mesh for customized usage in Blender (Release 3.5 2023);
• Importing the resulting model to Unity for further integration.

The historical map functions allow users to compare the actual environment view with historical cartography and constitute an important feature of cultural heritage.

2.3.1. 3D Model Sources

This study leverages the utilization of Google Maps/Google Earth as a convenient source for retrievable 3D models. These web applications are interchangeable in terms of model quality, as they offer comparable levels of accuracy and detail. Therefore, researchers and practitioners can seamlessly utilize either platform based on their preferences and accessibility. This flexibility allows for the efficient retrieval of 3D models while maintaining the desired level of quality and accuracy, without being limited to a specific platform.

2.3.2. Storing the Model

The next step involves recording and storing the 3D model from Google Maps/Earth utilizing RenderDoc (Figure 2), a powerful frame debugger tool capable of capturing API calls in specific frames. RenderDoc is injected into the Google Chrome GPU process, and during Google Maps/Earth navigation, a specific frame is captured; provided that no mesh optimization is performed during that frame, RenderDoc captures every API call for Google Maps/Earth 3D model rendering into a proprietary .rdc file. To avoid mesh optimization, it is enough to force the web application to refresh models during those frames, usually by moving the scene or zooming in/out. The proprietary “.rdc” file can then be read by a custom Blender plugin in order to replicate those API calls to recreate the model in a manageable workspace.

2.3.3. Mesh Modification Workspace

The next step involves importing the Google Maps capture in a user-friendly framework. Blender constitutes a top choice for this step because of its robust support for a wide range of file formats and its flexibility and customizable user interface.

Blender is employed in this workflow via the utilization of an open-source plugin “MapsModelImporter”; this add-on allows to import “.rdc” files, which is the final result of the step illustrated in Section 2.3.2 with RenderDoc (Figure 3).

2.3.4. Exporting Process

This workflow employs Unity for application development, and its powerful and versatile game engine nature offers significant advantages for developing augmented reality applications in the cultural heritage sector. First of all, Unity’s extensive capabilities include built-in support for interfacing with other devices. A Unity package is available for interfacing a physical scene with a holographic table, allowing for 3D visualization over a dedicated device.

For models to be utilizable in the Unity workspace, they need to be exported in either .fbx or .obj format.

3. Results

The beforementioned toolchain allows for low-detailed model retrieval useful for context environment models at low size cost; two main metrics then need to be evaluated for assessing the results of this paper:

• Quality detail level
• Performance savings

The quality detail level represents a metric hardly evaluated in objective terms; the most suitable method to assess it would consist of employing models obtained with the beforementioned toolchain in a practical application with high details. On the other hand, performance savings can be objectively assessed via an analysis of the running application FPS, evaluating both cases of high details and low details.

Firstly, an Augmented Reality (AR) application was developed to enable the immersive and interactive exploration of a cultural heritage site. The application’s tools and functionalities facilitate on-site visual comparisons between different periods of the site, as well as highly detailed surveys conducted using state-of-the-art techniques. This practical feature of the study showcases the tangible application of AR technology in the context of cultural heritage preservation and exploration.

Secondly, this paper presents an analysis of the causes and symptoms of motion sickness in Augmented Reality applications. This theoretical aspect delves into the factors that contribute to motion sickness in AR experiences and proposes a workaround for reducing proneness to motion sickness. The proposed approach involves utilizing low-fidelity surveys of overall environments, which provides insights into the potential strategies for mitigating motion sickness in AR applications. This theoretical feature of the paper sheds light on the challenges and potential solutions related to user comfort and well-being in AR experiences.

Together, these two distinct features of this study contribute to a comprehensive understanding of both the practical application and theoretical considerations of Augmented Reality in the context of cultural heritage exploration and motion sickness reduction.

3.1. The AR Application

The developed Augmented Reality application is designed specifically for use on the holographic table at LaborA, Politecnico of Milan (Figure 4). The holographic table is an AR device based on the principle of stereoscopy, which utilizes multiple projections of 2D images and lightweight glasses to create visual 3D artifacts. Unity has been selected as the primary workspace for this application due to its versatility and ability to seamlessly communicate with the holographic table and other software components involved in the survey scan workflows.

The choice of Unity as the main development environment is driven by its compatibility with the holographic table and its capability to facilitate smooth communication and integration with the AR device. Unity’s flexible and powerful features enable the creation of interactive and immersive AR experiences while also accommodating the integration of various survey scan workflows.

3.1.1. Usage

The applications presented here can be used in all fields of cultural heritage documentation that are best described from a landscape view that is broader than the monument itself. It is sometimes needed to express the relationships between objects in a park, as in this case, but also the spatial relationships between distant architectures and urban complexes in densely built contexts, where the modeling part of the whole area would require too much effort to accomplish. It is usable in all fields of landscape education and the fulfillment of contextualized design experiments [21,22].

3.1.2. Functionalities

In accordance with Section 3 of the paper, the AR application includes multiple tools that serve two specific purposes: providing a visual comparison of the cultural heritage site during different time periods and offering highly detailed surveys with minimal motion sickness factors. This is achieved via a combination of highly detailed surveys for specific buildings/environments and low-detailed surveys for overall environment surveys.

The highly detailed survey involves a laser scan of the “Torretta Viscontea”, located in “Parco di Monza”, Italy, with a level of detail of approximately 5 mm. This detailed survey provides a comprehensive representation of a specific building, capturing intricate details for an immersive and accurate AR experience. In addition, a lightweight survey utilizing the Google Maps workflow is employed to obtain an overall view of the entire “Parco di Monza” environment with minimal computational cost on the AR application.

To facilitate interactive exploration of the cultural heritage site, four tools have been implemented in the AR application. The first tool enables switching between different models of the site during different time periods, allowing users to visually compare the site’s appearance over time. The second tool allows activating or deactivating a high-fidelity model of the environment, providing users with control over the level of detail in the AR experience. The third tool is a map tool that visually overlays historical maps of the current environment/site, providing additional context and information. Lastly, a warp system has been implemented to enable easy switching between the models/objects to be visualized on the holographic table, enhancing the user’s ability to interact with the AR environment and navigate between different elements of the cultural heritage site.

Overall, the combination of highly detailed and lightweight surveys, along with the implementation of various tools, enhances the AR application’s capabilities in providing an immersive, interactive, and informative exploration of the cultural heritage site with minimal motion sickness factors [23].

Timeline switch: This tool provides users with the ability to navigate the AR model of the cultural heritage site across different time periods. When utilized, the tool instantiates a different time period model in place, effectively substituting the current state of the site with the corresponding historical representation (Figure 5).

This allows users to visually compare the site’s appearance and characteristics during different epochs, thereby providing a dynamic and interactive exploration of its historical evolution.

High-detail structures: Another implemented tool allows users to activate or deactivate a highly detailed structure within the environment. The tool was introduced to address the challenge of managing large-scale surveys (Figure 6) while maintaining a fluid and seamless user experience. By activating or deactivating the highly detailed structure on demand, users can control the level of detail they want to view in the AR environment, providing a balance between visual fidelity and performance and ensuring that the application remains responsive and fluid even when dealing with complex and data-intensive models.

Map overlay: The AR application also includes a tool that enables users to overlay a historical map onto the cultural heritage site. The map overlay feature has been implemented via manipulation of the transparency of the map, allowing users to control the level of opacity by translating the input wand in the physical space (Figure 7). This intuitive interaction technique provides a natural and immersive way for users to adjust the transparency of the historical map, seamlessly blending the virtual and physical environments, and allowing users to visually compare the site with its historical map, thereby gaining insights into the site’s historical context and evolution. Its interactive and dynamic nature enhances users’ engagement and understanding of the cultural heritage site, creating a more interactive and informative AR experience.

Warp across models: The AR application also includes a custom navigation tool that enhances the intuitive navigation system across different models. This tool offers two ways for users to switch between models (Figure 8):

• Placeholder System: Users can activate a placeholder system that generates clickable placeholders on the map. Each placeholder is positioned and linked to a specific model. By clicking on a placeholder, the scene automatically translates into the linked model, providing a seamless transition between different models. This approach allows users to easily explore different areas of the cultural heritage site by selecting the corresponding placeholders on the map.
• Toolbar Selection: Users can also switch between linked models using the toolbar. The toolbar provides a visual representation of the available models, and users can simply select a model from the toolbar to instantly switch to that model. This method offers a convenient and straightforward way for users to choose and switch between different models without relying solely on the map placeholders.

These two navigation options provide users with flexibility and convenience in exploring the different models of the cultural heritage site, allowing for a more immersive and interactive experience.

3.2. Latency Sensibility Analysis

The AR application has been developed specifically for a holographic table, which is an Augmented Reality device that utilizes 2D elements for 3D visualization. The stereoscopic principle is employed to create a sense of depth perception for users. This is achieved by providing a pair of slightly different 2D images to the user, which the brain processes into a single image with the perception of depth.

During the exploration of the application, motion sickness symptoms [24,25,26] have been observed in some users. This subsection of the paper aims to analyze the causes and symptoms of motion sickness in the context of the AR application, as well as provide a quantitative analysis of the impact of model sizes on the overall latency of the application.

The analysis will include quantitative data analysis of the impact of different model sizes on the overall latency of the application, as latency can potentially contribute to motion sickness symptoms. This analysis will provide insights into the factors contributing to motion sickness in the AR application and propose potential strategies or workarounds for reducing motion sickness proneness in users. The findings from this analysis can contribute to the overall understanding of the user experience with the AR application and inform potential improvements to mitigate motion sickness and enhance user comfort during exploration.

This paper focuses mainly on FPS analysis in order to evaluate the user experience comfort. It is worth noting that user comfort during the immersive experience depends on a multitude of factors; however, the FPS has been chosen as the main focus for this study given the performance-related task of obtaining 3D models. Size-increase performance requirements often worsen the application performance after a critical point of FPS rendering. This analysis will evaluate the impact on the application of the newly obtained 3D model regarding FPS amount.

The obtained models can then be optimized using techniques such as retopology and normal maps in order to further improve the application performance.

3.2.1. Causes

The holographic table recreates the 3D sense of depth by exploiting the stereoscopic principle, like the basic way the human eyes work. The user is provided with a pair of slightly different images, and the 3D is then recreated by the user’s brain perspective. Given its functioning, the system presents a critical sensibility on the application latency: the images provided to the user require a constant and precise update to maintain a feasible 3D model from the user’s perspective.

High-fidelity surveys have the best visual performance but with a non-negligible cost: their size and level of details represent a computational cost that can result in worse performances for the application. Furthermore, their computational cost is concurrent with the holographic table for rendering, making the 3D model not stable and accurate, resulting in visual artifacts causing motion sickness to the user.

3.2.2. Symptoms

Non-accurate images do not let the user’s brain to fully elaborate a 3D view of the model; the partial depth perception is then perceived by the user as a blurred view caused by physical factors, causing symptoms like, but not limited to:

• Nausea;
• Wanes;
• Cold sweating.

On the other hand, it is worth noticing that the holographic table motion sickness presents less severe symptoms than counterparts such as Virtual Reality headsets and biomechanical simulators [27] due to their lightweight glasses and the still perception of reality. By testing the application on several students attending the laboratory, the most common symptom resulted in dizziness, causing the user to remove the lightweight glasses and rapidly regain consciousness over the surrounding environment.

3.2.3. Analysis Mesh-Latency

The analysis of latency variability in response to variable model size and level of details is a crucial step in understanding the performance of the AR application on the holographic table and its potential impact on user experience and motion sickness symptoms. The four models that were analyzed include the following (

Table 1. File size and characteristics of the tested models.

	Vertices	Faces	Size [MB]	File ext.	Google
High quality	8.510.529	17.000.000	1.810	Obj	No
Low quality	580.488	913.723	731	Obj	No
“Torretta Viscontea”	5.215	5.025	1.42	Fbx	Yes
Monza Park	487.780	335.890	14.	Fbx	Yes

):

• Two highly detailed surveys of “Torretta Viscontea” in Monza, Italy, performed with professional instruments and tools, exported with different levels of detail;
• One low-detailed survey of “Torretta Viscontea” using a toolchain workaround for comparison;
• One low-detailed survey of the entire “Parco di Monza” with a total surface of 688 km² using the same toolchain workaround.

The recorded latency and performance data of the application, in conjunction with user responses and potential motion sickness symptoms, provide valuable insights into the relationship between the model size, level of details, application performance, and user comfort. By analyzing the data, patterns and correlations can be identified to understand how the different models impact the application’s performance and user experience.

The analysis reveals that the highly detailed surveys, due to their larger size and higher level of details, result in excessively increased computational costs, rendering performance, and latency, which potentially critically impact the overall stability and accuracy of the 3D model on the holographic table. This may lead to visual artifacts and motion sickness symptoms in users. On the other hand, the low-detailed surveys, despite their lower level of details, have better performance and lower latency due to their smaller size and reduced computational costs.

The findings of the analysis provide potential strategies for optimizing the performance of the application, such as optimizing rendering processes, reducing computational costs, and mitigating visual artifacts, in order to enhance user comfort and reduce the likelihood of motion sickness. Additionally, the analysis aims at helping to identify the optimal balance between the model size, level of details, and application performance to provide a smooth and enjoyable user experience without compromising the stability and accuracy of the 3D model on a holographic table [28].

The analysis of file and model size differences between high-accuracy and low-accuracy models is essential to understand the applicability and quality of lightweight low-accuracy models in the context of the AR application on the holographic table. To assess the quality of those models, Cyclone 3DR was used to run a quality analysis by comparing the high-accuracy point clouds of “Torretta Viscontea” with their low-accuracy counterparts obtained using the FOSS (Free and Open-Source Software) toolchain [29].

The first comparison performed was based on the covered surface of the models. It was observed that the high-accuracy surveys covered a surface of 1879 square meters, with significantly high accuracy in purely quantitative terms. This indicates that the low-accuracy models can capture a large portion of the surface details of the high-fidelity models, despite their reduced file size and lower level of details.

The next comparison was performed in terms of adherence, which measures how accurate the low-accuracy models are compared to the most precise digitized version of the model. The results (Figure 9) reveal that 81% of the model mesh has an adherence within a 0.66 m range from the point cloud model, and 94.9% of the model mesh has an adherence within a 1.31 m range from the point cloud. This indicates that the low-accuracy models maintain a reasonably high level of adherence to the high-accuracy models, despite their lower level of details.

These findings suggest that low-accuracy models obtained using the FOSS toolchain are capable of providing a minimum quality level that is suitable for the AR application on a holographic table. While they may not capture all the fine details of the high-fidelity models, they still provide a visually accurate representation of the model with a high level of adherence to the original point cloud data (Figure 10). This supports the feasibility of using low-accuracy models to optimize the application’s performance by reducing the file size and computational costs, while still maintaining an acceptable level of quality for user experience.

To assess the performance of the application, a series of experiments were conducted using the Unity Analysis Profiler. Since rendering was a crucial task in the application, the main performance metric analyzed was the FPS amount during the rendering phase, which is critical for performance-related motion sickness. Five separate experiments with both low-detailed and highly detailed models employed were conducted with different setups to determine the most significant ones for the results presented in this paper. The experiments included the following (Figure 11):

• Loading the most highly detailed survey model, “Torretta Viscontea”, directly into the application with an approximate file size of 1.81 GB.
• Loading the less highly detailed survey model, “Torretta Viscontea”, directly into the application, with an approximate file size of 0.73 GB.
• Loading only the low-resolution environment of “Parco di Monza”, with a tiny file size of only 14 MB.
• Loading the low-resolution environment of “Parco di Monza” and the lowest high-fidelity survey of “Torretta Viscontea”.
• Loading the low-resolution environment of “Parco di Monza” and subsequently both high-fidelity surveys of “Torretta Viscontea”.

The results of the experiments, including the maximum, minimum, and average frame rates, as well as the total frames processed within a 30 s time span, are shown in

Table 2. Experimental results of the application latency over various models.

	Latency MIN [ms]	Latency MAX [ms]	Latency AVG [ms]	Tot. Frames
High Quality	31.38	144	87.69	170
Low Quality	3	18.61	10.81	2000
Parco	4.8	7.38	6.09	2000
Parco + Low	6.58	28.06	17.32	1754
Parco + Low + High	10.42	189	99.71	158

. Since the application is closely related to video games, the Frame Per Seconds (FPS) requirement can be safely applied as a metric for this analysis [30]. The minimum FPS for an overall enjoyable experience is typically considered to be 30 FPS, although more experienced users in mixed reality and gaming applications may require a minimum of 60 FPS [31]. The FPS for each experiment during the 30 s time span is shown in the following figures.

It is important to note that a maximum buffer of 2000 frames was maintained with the Unity Analysis Profiler, resulting in 2000 frames processed if the minimum FPS requirement was met. This allowed for a comprehensive analysis of the application’s performance during the experiments.

4. Conclusions

4.1. Discussion

This project has been centered on applying a FOSS toolchain to gain low-detail models for environment rendering; the toolchain comprises Google Maps utilized as a satellite-view 3D model library, RenderDoc as the frame recorder for storing the 3D information related to the Google Maps 3D view, and Blender as the main work environment for mesh pre-processing procedures, such as mesh cleaning and vertex optimizations.

The resulting models are then utilized in concomitance with high-detail models obtained with professional tools and procedures. The goal is to make it possible to enjoy large spaces, both built and unbuilt, where the object of study finds its reason above all in its relationship with its surroundings in a holographic situation. This system can facilitate storytelling experiences not only in holography but also in VR environments by saving time with good accuracy of results.

Referring to the quality detail level, it is worth noting that it is impossible to discuss it without correlating the quality level with the physical dimensions of the analyzed area: with “Parco di Monza” covering approximately 7.32 square kilometers, achieving professional quality details on the complete model would lead to unmanageable performance requirements; thus, the proposed tradeoff of different level of details on green parts and architectural parts provide a more complete experience with manageable models, providing the natural location and surroundings of the architectural parts obtained using professional tools and techniques for the best possible quality.

On the other hand, the performance analysis was critical in importance, given the impact of the overall FPS on motion sickness-related matters, such as lag and incoherent image rendering, essential for the third-dimension illusion of the AR device (Hologram Table).

4.2. Future Works

This project work achieved a fluid visualization experience at 60+ FPS with highly physical dimensions models such as the entire “Parco di Monza”, in order to accomplish obtaining the most complete experience, a combination of both highly detailed models at architectural scale and lowly detailed models at urban scale has been employed, thereby achieving the highest level of detail possible for Point Of Interests (POI) while utilizing environment contexts for the spatial location and green areas.

Future studies on the application comprise four possible areas:

The first area for future studies is related to content creation. The application at its current state allows for easy widget insertion inside the models via the Unity interface, user experience can be improved with new scenarios correlating the past and the present of POIs, and the beforementioned Google Maps toolchain allows for easily retrievable 3D models of high-scale dimensions with little to no cost and effort. Furthermore, for a more personalized experience around the personas involved, it is possible to create new widgets with custom behaviors, linking them to the present interface optimized for the most scalable solution with new widgets.

Another future study area that can be employed involves performance optimization. The proposed toolchain succeeds in obtaining low-size models from Google Maps, but both low- and high-detail models can be improved for performance by size-reduction techniques, which include, but are not limited to, retopology for a cleaner and lighter representation while matching the same physical shape, texture optimization, normal maps, and so on. Other performance-related improvement involves utilizing LOD logic for better optimization of the process of allocating computational power to render the focused building, silently reducing the level of detail of surroundings and border objects.

Another interesting future study area may involve personalization over the application. Different personas may be interested in different points of view and POIs, for which a recommender system [32] may be employed for further personalization according to the user’s taste and interest [33].

Lastly, the Quality Of Experience (QoE) of the user depends on multiple factors. Although latency-related causes were addressed in the “Results” section, it is worth noticing that the quality of the image plays a key role in the user comfort analysis. The holographic table device and 3D rendering of the 2D screens may introduce image distortions that could lead to user discomfort, loss of visual data, and an overall worse quality of experience. It goes without saying that this factor needs to be analytically measured and evaluated in importance related to the specific experience; these metrics require comprehending distortion factor introduction, analysis, and methodologies to overcome their negative effects, such as Blind Multiple Pseudo Reference Image (BMPRI)-based measure and the Blind Pseudo Reference Image (BPRI) [34,35].

Image quality assessment is a challenging task for this kind of application, as the visual representation of the environment is often subject to distortions due to hardware capabilities (AR tracking glasses, holographic table infrared tracker cameras, etc.) or user-dependent causes (depth perception, eyes related problems, etc.); assessing the quality of the image may be conducted with techniques such as Blind Pseudo Reference Image and Blind Multiple Pseudo Referenced Image.

MPRI is a general-purpose Blind Image Quality Assessment (BIQA) method for distortion estimation. The blind characteristics make it Opinion Unaware (OU), and is thus suitable for applications with a high volume of information subject to variable distortion. On the other hand, BMPRI is a metric for measuring the QoE of the user employing images subjected to distortion, and it is obtained via the extension of pseudo-referenced images (PRI) to the beforementioned Multiple Pseudo Referenced Image (MPRI).