The Design of a Virtual Prototyping System for Authoring Interactive Virtual Reality Environments From Real-World Scans

We are living in an era where substantial efforts are being made to create seamless pathways between the physical reality and its digital counterpart. Due to its tremendous potential to capture the reality in authentic self, the digital counterpart can be made useful in a variety of applications, such as remote collaboration, hologram teleconferencing, and simulated training [1–3]. Consequently, the past decade has witnessed the emergence of virtual reality (VR) technologies and applications in various domains, e.g., manufacturing, construction, sports, etc., that provide the domain users (DUs) with realistic and immersive platforms to explore and interact with the digital counterparts of the realities [4]. Despite the spatial affordances provided by the immersive VR applications in allowing situated and hands-on experiences for the DUs, there is high barrier for adoption of these technologies in domain-specific fields [5]. This is partly due to the lack of accessible methods for the DUs to author these applications for their respective domains.

Conventional methods of digital content creation for VR applications are tedious and fragmented, involving stacks of various technologies, e.g., VR programming software and technical modeling [6,7]. The creation process is usually limited to the hands of experts in VR programming (EVRPs), and thereby alienates novices in VR programming (NVRPs) from the authoring process [5]. Most importantly, the requirement of VR programming expertise creates an entry barrier for the DUs, who are oftentimes also found to be NVRPs, to author field-specific virtual applications. Considering the above problems that the DUs face to utilize the technology to their benefit, our work is focused on designing an accessible system workflow that can assist DUs in authoring interactive VR environments from real-world scans without needing technical expertise.

To make the scene reconstruction more accessible to NVRPs, prior works have introduced numerous authoring systems for VR applications [8–10]. The idea is to provide users with a modeling tool that effectively uses the reality scene as a modeling reference for derivative scene reconstruction with added interactive functionalities. However, the amount of user interaction and technical modeling knowledge required is still substantial. On the other hand, recent works in the artificial intelligence (AI) area have been using deep learning to automatically reconstruct the digital scene from 3D scan of the real world [11–15]. However, these automated approaches usually lack the ability to provide user-in-the-loop customization. Inspired by the previous works, we choose an approach that balances user interaction with AI automation.

Because of the state-of-the-art sensing hardware and computer vision algorithms, real-world digitization can be easily done by commercially available 3D scanners and red, green, blue plus depth (RGB-D) cameras [16]. The results from these devices are usually the point cloud or the mesh model reconstruction of the reality which provides a unique benefit of laying out the objects reasonably with realistic dimensions and unaltered spatial relationships. However, they can only serve as a spatially situated visual reference for the user without any interactive functionalities. In order to preserve the interactivity of the scanned real-world scene for a desired VR experience, it requires professional developers to manually recreate the virtual scene while using the scanned results as a background modeling reference [17]. Accompanied by the affordances provided by our system, users can spatially interact in these visually immersive virtual worlds in an embodied fashion [18,19] for authoring applications.

There are a few different interface platforms for users to interact with the digital version of the real world, including 2D (as in mobile, tablet, and desktop-based interfaces) and 3D (as in VR-based interfaces). In this paper, we focus on VR due to its capability to provide immersive experiences while supporting in situ spatial and embodied interactions from the users [18]. The choice of 3D platform affords better 3D perception and direct testing of the authored content within the execution environment, without needing to switch between 2D and 3D interfaces. Moreover, 2D interfaces bring interaction issues while adding the extra dimension in unnatural and unconventional ways, and restrict an effective “flow” of using hands to work in virtual environments [20]. Considering these advantages of VR, our work is focused on creating an accessible and immersive system for users to transform physical reality into meaningful virtual environments.

We present VRFromX, an end-to-end system design that supports the authoring of interactive VR scenes from real-world scans. Our workflow is illustrated in Fig. 1: the user starts from scanning a real-world scene (a welding workshop in this case) into point cloud using a handheld 3D scanner (Fig. 1(a)); then replaces the point cloud objects with their corresponding virtual models via embodied interaction with AI-assisted object retrieval (Fig. 1(b)); the user then attaches functionality onto the virtual objects and defines logical connections between them (Fig. 1(c)); finally the virtual scene is ready for VR interactions (this particular scene can be used for virtual welding training) (Fig. 1(d)). The contributions of this paper are as follows:

• The system workflow design of VRFromX that enables the creation of interactive VR scenes, by scanning real-world environments into point cloud scans, then transforming the point cloud objects into virtual 3D models, and establishing functionalities and logical connections among virtual contents.

• The design of user interaction that supports point cloud segmentation and editing via embodied interactions, the AI assistant that guides the object retrieval and alignment, and the spatial and visual interface for functionality and logic authoring.

• A detailed system development of a welding use case scenario, evaluation results obtained from a three-session user study, in addition to two example use case demonstrations to show the breadth and nature of potential applications of VRFromX.

In order to evaluate the usability of the system, we conducted a user study using a welding use case with 20 DUs, comprising of 12 NVPRs and eight EVRPs. Through the findings of the user study, we wanted to answer the following research questions:

• To what extent does the users’ expertise in VR programming affect their performance and experiences in using VRFromX to author virtual environments?

• How does the domain knowledge of users affect their perception and experience in using VRFromX to author virtual environments?

Our study results indicated high user ratings about the accessibility, engaging, and intuitive aspects of the system to author virtual experiences. A comparative evaluation across the EVRPs versus NVRPs indicated similar ratings towards perception and experiences in using the system, across the two groups with no significant differences. Two other use cases in the (1) 3D-printing and (2) Robot-IoT domains were further explored to envisage the potential applications of VRFromX.

Traditional 3D reconstruction techniques that targeted transfer of real-world scenes to virtual environments have mostly lived in desktop-based environments. Computer vision-based methods used in these environments have explored reconstructing virtual scenes using RGB/RGB-D images [21–28], point clouds [29–32], and 360 deg images [33]. However, these computer vision techniques are usually not adaptive to users’ interaction and the 2D interfaces also limit immersion from the users. On the other hand, prior work have highlighted the immersive affordance provided by the 3D platforms [34] that has been used in VRFromX to facilitate intuitive interaction from users while leveraging their spatial reasoning skills during the scene reconstruction.

Over the past few years, research works have deployed 3D reconstruction techniques to automatically reconstruct virtual scenes within VR using scanned RGB-D information [35–39]. Reality Skins [35] uses consumer-level RGB-D cameras to pre-scan and process the user’s physical surrounding to translate it directly into a virtual environment with the best object layout. Oasis [36] procedurally generates an interactive VR experience using pre-scanned data from the physical environment as a template. By allowing the seamless transition between the physical and virtual realities, RealityCheck [37] selectively shows real-world objects as part of a situated VR experience. Dreamwalker [38] leverages RGB-D sensors, windows mixed reality (MR) tracking, and global positioning system (GPS) to substitute real-world walking experience with a dynamic virtual environment. VRoamer [39] scans the physical environment in real-time and dynamically instantiates pre-authored virtual experiences that fit the physical world. These systems mainly focus on employing RGB-D information to provide realistic immersion within virtual spaces, ensure safe walking in the physical world, and construct pre-authored virtual environments. However, the above construction techniques do not provide interaction capabilities for the user to create or modify the authored environments, thereby restricting their abilities for prototyping purposes. Moreover, these techniques result in the creation of virtual environments without any semantic meaning associated with the virtual objects, and thus do not facilitate any interaction from the end-users. On the other hand, VRFromX considers the RGB-D information as a medium where users can interact with to transform static point clouds into virtual experiences, which allows user involvement to create meaningful experiences.

In the context of object manipulation in the reconstructed scenes, authoring tools have been developed that allows users to capture and manipulate scene scans of physical environments [40–47]. Remixed Reality [40] performs a live 3D reconstruction with multiple external RGB-D cameras and provides direct manipulation of the reconstructed virtual scene. AffordIt! [41] offers a solution to assign affordances and constraints to virtual objects which do not have intrinsic behaviors after 3D reconstruction and segmentation. Prouzeau et. al. [42] developed an authoring tool for building management using digital twins in VR where users can design situated and embedded augmentedreality (AR) visualizations to 3D models, and later, view those visualizations in AR in the physical space. ARchitect [10] reconstructs an interactive virtual scene by scanning the physical environment and aligning virtual proxies to physical objects with similar affordances. DistanciAR provides an authoring interface that allows the user to scan an environment at a remote site, add and manipulate virtual objects, and create AR experiences for the remote environment [43]. PhotoShopAR allows users to quickly prototype design ideas in a spatial context using point cloud editing operations and later experience the design via AR [46]. ScalAR allows users to create AR experiences and define semantic behaviors of AR contents with the help of a VR authoring interface, while using existing scenes as semantic references [44]. However, the manipulation techniques offered by the above authoring tools are specifically restricted to the spatial affordances provided by the virtual objects by virtue of their position and location in the environment and thus limits the semantic representation of the reconstructed scene. Although VRFromX bears some similarities with the above tools in the mode of reconstructing the virtual scene, the nature of scene manipulation techniques offered by the proposed system is strikingly different. In contrast to the above methods, VRFromX allows users to manipulate and interact with the reconstructed scenes to facilitate the prototyping of context-aware and interactive virtual experiences. A logic-authoring interface is employed using which users can link different virtual objects in the scene and embed contextually relevant functionalities to associate semantic meaning to the virtual objects.

In the context of embedding interactive functionalities in virtual scenes, prior work has also studied spatially situated authoring tools to attach logic constructs onto virtual objects [8,9,48]. Ens et al. [8] developed an application to enable end-users with limited programming skills to author internet-of-things (IoT) devices in virtual environments with multiple logic nodes. Ng et al. [9] provided a game authoring system that enables users to add virtual contents to the physical world and attach game-logics to virtual objects. Flowmatic provides an immersive authoring tool that allows programmers to specify reactive behaviors that react to discrete events such as user actions, system timers, or collisions [48]. The tool introduces primitives for programmatically creating and destroying new objects, for abstracting and re-using functionality, and for importing three-dimensional (3D) models, while allowing their direct representation in VR. However, the above authoring tools disconnect the users from the physical reality by utilizing pre-authored virtual scenes [8] and/or still require technical experience from the users to be able to program the interactive behaviors [48]. Although the logic-authoring interface developed in VRFromX draws insights from the above works, the system workflow differs from the prior work by providing DUs with an end-to-end workflow to create interactive and meaningful virtual experiences from real-world scans.

In the last decade, research works have merged automated modeling and reconstruction techniques with human interaction workflows to explore the different degrees and benefits of human and AI involvement in authoring virtual content [49–53]. While AI can aid the user by reducing the requirement of skills, time, and effort during the content transformation and creation process, bringing the user in-the-loop adds more control and customization over the content design, and drives the transformation in a more goal-oriented fashion. Inspired by the prior works in this area, VRFromX combines the benefits of human-in-the-loop systems and AI techniques to create context-aware VR environments. The AI-assisted human-in-the-loop interactive capability in VRFromX is specifically utilized in the scene reconstruction where user performs sketching operations in the point cloud scans. And accordingly, the AI algorithms return the corresponding virtual objects that closely resemble the inputs from the user.

Since early times, humans have used sketching as a natural and free-form input modality to convey visual information [54,55]. Because of the easy to use and intuitive aspects of the sketch-based interactions coupled with the inherently fine-grained visual descriptions [56], sketch-based retrieval methods have witnessed a substantial interest within the computer vision community for virtual content creation. SemanticPaint [51] offers a classical 3D reconstruction pipeline in which the user can physically touch any desired object in the real world to enable real-time segmentation and labeling of scanned environments. Shao et al. [57] proposed an interactive approach for semantic modeling of indoor scenes where the user can draw strokes on the RGB-D image of the scene to alter results from an automatic segmentation. Recent research has also explored immersive sketching in mid-air to retrieve 3D models for virtual content creation. Giunchi et al. [49] proposed a multi-view convolutional neural network (CNN)-based retrieval method where free-form sketches within a virtual environment can be used as queries for searching virtual models. Zhu et al. [58] proposed a skeleton-based approach to draw 3D models in head mounted display (HMD)-based virtual environments using convolution surface-based brushes. Li et al. [59] implemented a CNN-based 3D sketch-based shape retrieval system, which allows users to freely draw 3D sketches in the air to retrieve relevant 3D models. On the same note, Luo et al. [60] studied the retrieval problem from sparse human VR sketches by training on human and synthetic sketches. BuildingSketch [61] allows users with limited modeling knowledge to draw freehand strokes in mid-air which subsequently gets processed by deep neural networks to generate detailed building models. Similar to prior work in the context of sketch-based object retrieval, VRFromX leverages a PointNet-based architecture [62] to facilitate interactive object retrieval for virtual content creation using free-form mid-air sketches and point cloud selections as query inputs inside a virtual environment.

Our current work directly builds upon our prior work published in ACM Conference on Human Factors in Computing Systems (CHI) extended abstracts [63] which presented the system workflow design of VRFromX. In addition to providing a clearer description concerning the design details, justifying the design choices and carefully situating the work with respect to relevant literature, the current version conducts a revised evaluation of the system to justify the research objectives. The study results provide novel insights on how the domain and technical expertise of the users affect their perception and performances in utilizing VRFromX to author VR experiences.

Considering the difficulties faced during the authoring process for transforming physical reality to VR and insights gathered from relevant prior work, the following design goals of VRFromX were identified to make the system accessible to DUs while providing them with an enjoyable experience.

• Accessibility: System features of VRFromX should be easy to understand and use, and eliminate the need of any modeling and programming expertise. This can enable users with varied technical backgrounds and expertise levels to easily develop familiarity with the system features.

  First, hardware complexity is kept low involving the use of OculusQuest^™, which is chosen due to its easier configuration compared to other VR devices [64]. Next, easy adoption of the authoring process is facilitated by the use of freehand sketching on the point clouds as input modalities from the users [65], and easy-to-use drag-and-drop user interfaces (UI) for attaching affordances [66] to virtual objects. While the point cloud sketching can provide users with a visual and interactive means of interacting with the spatial data [67,68], the drag-and-drop-based interfaces allow users to simply drag elements and drop them into specific areas or targets by adopting simple motor skills [69]. Furthermore, such interactions and interfaces have shown to reduce learning curve by reducing cognitive load, enhance understanding of the UI, and facilitate easy user interaction. Furthermore, the sketching and behavioral modeling tools developed in VRFromX have been designed with user friendly interfaces and features that are easy to find, use, and navigate to further lower the floors in accommodating different abilities of the users during the authoring process.

• Engaging: System features of VRFromX should provide an immersive experience and provide opportunities for active user engagement during the authoring process. This is achieved by providing an in situ authoring environment with gamified and dynamic interactions, and options for user customization.

  Prior work has shown enhanced sense of immersion, engagement, and motivation in users during authoring using VR as compared to 2D-based platforms such as desktop-based environments and tablets [70]. The 3D authoring environment in VRFromX enables free and embodied interactions from users while allowing them to directly experience the resulting application, a concept coined by Lee et al. as WYXIWYG: What You eXperience Is What You Get [70]. The system provides a prompt interface for users to embrace natural and free-form creation methodologies such as 3D sketching in VR [49,65]. Active user engagement in the point cloud environment is supported with the AI assistance during point cloud sketching. The sketching interface in VRFromX easily adapts to the user’s choices and modifications, and works on the changes to provide desired outputs in an iterative manner. Prior work has shown the impact of such AI-assisted interactions to induce a sense of motivation and immersiveness in users by enhancing the scope of creativity and support, and promoting sustained interest during the authoring process [71].

• Effective: The interaction procedures offered by VRFromX should be intuitive, goal-oriented, and make sense to the DUs about what they are authoring.

  By bringing in the digital twin of the real world into VR, users can more closely relate to the real-world tools and interactions while authoring using the system. As evident in the research conducted by prior literature [67], the point cloud data can provide a visual representation of the spatial reality, and thus make it easier for DUs to understand and analyze the complex 3D information contained within it. Using the sketching tools and AI-assisted object retrieval in VRFromX, users can (1) manually segment or edit specific points or regions on the point cloud scan to retrieve the desired virtual counterpart and (2) personalize and/or iterate on their sketches or selections. Such interaction flows with human-in-the-loop and AI assistance have shown to improve decision-making in users during the authoring process [71,72].

  Being able to understand and relate the tasks in the virtual environment to real-world activities can add more meaning to the creation process. In order to achieve this, the behavioral modeling interface employed in VRFromX allows users to attach contextually relevant properties to the virtual objects. Such context-aware authoring can enhance user motivation and confidence during the authoring process [73]. Finally, the outcomes of the authoring process, i.e., the resulting VR application, should behave as expected by the DUs. After completion of authoring, the system allows the users to test and verify the virtual environments that they have created.

We designed VRFromX to enable users to transfer a scanned point cloud scene into an interactive virtual experience (Fig. 2). The major components of our system include: (1) an interactive interface for embodied interaction on point cloud for AI-assisted object retrieval and (2) behavioral modeling for virtual models. Point cloud acquisition of a real environment is performed with the help of an inbuilt light detection and ranging (LIDAR) in iPadPro, using an application called 3d Scanner App^™ [74]. This raw point cloud serves as a medium for the user to interact within a virtual environment. Spatial and color information of the point cloud helps the user in perceiving various objects present inside the scene. The user then employs a brush tool to segment regions of interest in the point cloud scene. Using the segmented point cloud as the input query, AI algorithms implemented in the back-end assist the user in retrieving corresponding 3D models. Finally, the user shifts to the behavioral modeling mode to assign functions to the virtual models with the aid of an affordance recommender.

In VRFromX, we designed a system for users to interact with a scanned point cloud in an embodied manner inside a virtual environment. A brush tool which enables users to select regions on the point cloud or sketch in mid-air in a free-form manner was introduced. The various features of the brush tool (Fig. 3(A)) are explained as follows.

Select, delete, and add (Figs. 3(d), 3(c), and 3(b), respectively) are used for selecting, deleting, and adding points in the point cloud scene, respectively. Clear all (Fig. 3(e)) is used to clear all selections or sketches in the scene. Query (Fig. 3(f)) is used to send the point cloud selection or sketch as query input to the back-end neural networks. Scale (Fig. 3(g)) is used to scale or orient the point cloud scene as per convenience. Tip size (Fig. 3(a)) and tip shape (Fig. 3(h)) are used to change the size and shape of the brush tip (Fig. 3(i)), respectively. To prevent confusion, the brush tip changes color on selection of different features, e.g., blue for selection, white for adding points, and red for deleting points.

After forming a clear perception of the context of the scanned scene and the objects present inside it, the users can exploit the functionalities offered by the brush tool for content creation inside the virtual environment. They can manually segment regions of interest inside the point cloud or modify them (by adding or deleting points) to get corresponding virtual models from the database. The tool also allows the users to sketch the object shape in mid-air if they do not find the objects in the partial point cloud. Keeping in consideration the accessibility of regions inside the scene, the scale feature enables users to manually scale the environment or orient it to a desired setting to suit requirements using simple hand gestures.

Neural networks are implemented to assist the user in object retrieval and model alignment in an interactive manner.

Object retrieval in VRFromX is performed as a two-step process, object classification followed by similarity check. After user selects a region of interest in the point cloud (Fig. 4(b)) or sketches the object shape, the point cloud selection or sketch can be fed to the back-end system as an input query. A PointNet [75] based classification network deployed in the back-end provides the user with the top five possibilities for object classes present in the database (Fig. 4(c)). When the user selects the desired class, the network performs a L2-distance-based similarity check and returns the top five plausible results of the object models that closely resemble the input query (Fig. 4(d)). The L2-distance is calculated using the feature vector of each object extracted by the classification network. The retrieval process is also adaptive to the user’s selections, i.e., if the user decides to change the shapes of the input query using the brush tool features (Figs. 4(e) and 4(f)), the AI algorithms adapt to the input and modify the search results accordingly (Figs. 4(g) and 4(h)). After the desired model is selected, an alignment network based on PointNet-LK [76] assists the user by automatically aligning the 3D virtual model with the region of interest according to scale, position, and orientation (Fig. 4(h)). Users can also perform simple hand gestures to grab and alter the position, orientation, and scale of the virtual models based on their requirements. Further details on the back-end implementation are provided in Sec. 5.

It is worth mentioning that the interaction capabilities provided by VRFromX can also enable users to re-purpose the point cloud to retrieve models of interest. In Fig. 5, a user re-purposes the point cloud of a table to retrieve a 3D model of a chair. As explored in prior literature [,46,77], such re-purposing can be particularly helpful in redesigning physical spaces, creativity exploration, etc.

After setting the environment with the virtual models retrieved from the database, users can attach affordances to those objects to make them interactable. According to Gibson [78], the term affordance is defined as the property of an object that defines its uses, or an action that can be performed over the object by an agent in an environment. In our work, we defined such behaviors of virtual objects to be categorized under three categories: animations, displays, and controls. Animations and displays are used to understand and visualize the different states of the virtual object(s). Animations are used to symbolize the various states of a virtual object in terms of motion or effects (sound and visual). As shown in Fig. 6(b), the light or glow animation is used to determine the states (ON and OFF) of the virtual lamp. Displays are panels used to visualize data for various attributes of virtual objects. For example, the display panel in Fig. 6(c) is used to display the intensity of the virtual lamp. Controls are used to perform actions that enable transition between the different states of the virtual object(s). These affordances can exist in the form of other virtual objects or widgets, and can trigger actions upon collisions between objects, or user manipulation of the widgets. (For more details on selection of widgets for the controls affordances, see Sec. 4.3.1.) For example, the user manipulates the switch (Fig. 6(d)) and slider (Fig. 6(e)) controls to switch ON the lamp and change its intensity, respectively. Finally, links are used to represent the logic connections between the virtual objects and their affordances, and fall under two categories: trigger link and data flow link. The trigger link (Fig. 6(d)) represents the input connections from the controls affordances to the virtual objects, whereas the data flow link (Fig. 6(c)) represents the output connections from the virtual objects to the displays.

We provide an affordance recommender for the user to choose the behavioral attributes of a virtual object. The interface design of the affordance recommender is shown in Fig. 6(a). Upon grabbing the virtual model of interest, users can enable the affordance recommendation by pressing a virtual button that appears on the left hand. Object properties for specific classes are predefined and stored in JSON format along with the 3D models in the database. The recommender extracts these properties for the target object and provides the users with possible options. Users can grab one or more functions from the list of choices displayed by the recommender and attach them to the corresponding virtual object.

Buttons, levers, and switches are ubiquitous to control physical machines, thus it would be valuable to include them as digital metaphors for controls widgets in an immersive interface. The mapping affordances of the real buttons, levers, and switches can be relied upon for interactivity, and most users would already be familiar with them. Thus, this mapping reduces the effort of learnability for the UI. To identify the most frequently used objects to model the prefabs for controls widgets, we performed a small-scale study of 100 different commonly used machines. Machines such as 3D printers, LASER cutter, lathe, household printers, computer numerical control (CNC) mills, water-jet cutter, tungsten inert gas (TIG) welder, miter saw, table saw, smart lights, etc. were chosen for the study, and the same machines from different companies were also identified and grouped as part of the study.

The list of physical interfaces that are frequently used such as buttons, levers, and switches were identified and categorized based on their functions. All items identified were part of one of the following three categories: toggle, linear, and discrete. A toggle is a device that provides only two states of affordances (Figs. 7(b), 7(e), and 7(f)), while a linear device can provide a range of states within a continuous function with limits (Fig. 7(d)). A discrete device provides output with more than one state but is not continuous (Fig. 7(c)). Each one of these categories serves a different purpose, at least one device was chosen from each category. Figure 7(a) shows the list of seven chosen button prefabs that were identified, and digital metaphors of which were developed.

The implementation of the system workflow design was based on a welding use case scenario. Welding is a hard skill which involves a set of tools and interactions under controlled settings. We chose the welding use case to test the capability of VRFromX to author a virtual metal inert gas (MIG) welding simulator. We adjusted the system requirements in terms of object models and their functions in database to enable an experienced welder for authoring the welding simulator. We created a welding dataset with object classes that can be typically found in an actual MIG-welding setting (see Fig. 8) and trained the back-end neural networks on these object classes. Object properties were defined such that the authoring process could involve replicating machine setup (amperage/voltage), creating a realistic welding puddle with accurate sounds and movements, simulation of sparks and slag, and tracking key weld parameters, such as work angle and contact-to-tip distance [79]. In order to obtain the point cloud for the implementation, the welding station at a local welding training company was scanned using an iPadPro. A researcher walked around the scene holding the iPadPro scanning every possible detail using the 3d Scanner App^™ [74]. The process was repeated five times each of which took around 10 min and the best scan was used for the implementation. The best scan was identified by the same researcher upon subjective consideration of the visual details captured in the scans.

We used the Oculus Quest HMD with Oculus Link as well as Oculus Touch controllers to perceive the virtual experience. The system ran on a laptop PC with a Processor Intel^® Core^™ i7-8700K, NVIDIA GeForce RTX 2070 super graphics card and 64 GB of RAM. The raw point cloud obtained was directly rendered and used in Unity^™, and no data augmentation techniques were utilized during this process. Front-end in the VRFromX system was developed within Unity Engine^™ in C# and was connected with the pre-loaded back-end neural networks through a separate service thread in Unity^™.

The back-end implementation of the neural networks employed in VRFromX is as follows. The implementation of the PointNet [75] based classification network was based on a PyTorch version of the original PointNet [62]. The classification network was trained 250 epochs on the ModelNet40 dataset [80], which includes 40 classes of objects. The network was fine-tuned with 200 epochs on the welding dataset, including 14 classes of welding equipment, each consisting of 50 models (see Fig. 8). The fine-tuning with the welding dataset was performed to prepare the system for the welding use case and improve the classification accuracy. The network’s accuracy for top five results was over 98%, with the average testing time of 1.8 s. Accuracy was calculated by running predictions on the testing dataset which contained a subsample of the training dataset. The alignment network based on PointNet-LK [76] was implemented using a PyTorch version of the original PointNet-LK [81] on the ModelNet40 dataset with a data augmentation method that applies to random rotation and variation. The average testing time of the network for aligning an input with 2000 points was 1.3 s.

In order to evaluate the ability of the system to author interactive virtual experiences from real-world scans, a user study was conducted using the welding use case. The study was conducted with 20 participants (three female, 17 male) with ages between 18 and 35 years, from which 12 users (three female, nine male) were NVRPs, and the rest eight users were EVRPs (zero female, eight male). All users had prior hands-on experience with welding, and thus can also be considered as DUs. Furthermore, 11 users were experts in welding. For more details on the user demographics, please see Table 1. Participants were recruited using word of mouth and the study was approved under Institutional Review Board (IRB) protocols. During this study, we observed the usability of VRFromX while examining how the users interacted with the system and what features they expected to achieve using the system. During the study, we also tried to evaluate the potential of the virtual welding simulator to be used for training purposes based on the comments of the welding experts.

At first, users were provided with an iPadPro and were asked to experience the scanning process by scanning any surrounding region using the application 3d Scanner App^™ [74]. This step was included in the user study to give the user an idea about the point cloud and the ease of its acquisition using a consumer-level product. This was followed by giving the user a virtual tour of the welding station in VR by placing virtual markers at appropriate positions in the scan showing the pictorial representation of the objects in the actual scene. After the users had developed a proper understanding of the scene and objects present in the scan, the actual study process started. A researcher explained the instructions verbally about the functions of the physical buttons on the oculus controllers (OCs) and virtual buttons in the system. Using a video demonstration, the objectives of each task were explained to the user. The entire session was divided into three tasks: (1) object retrieval using the brush tool, (2) behavioral modeling of virtual objects, and (3) interaction with the final scene. During the first task, users were asked to either select the objects from the scanned point cloud or sketch in mid-air using the brush tool to retrieve models from the database. After building the virtual environment with the models, the next task was to assign functions to the objects. The final task was to interact with the objects inside the welding scene to perform virtual welding. After each task, users recorded their experience about the system using a 7-point Likert survey (1 = strongly disagree, 7 = strongly agree). After completion of the three tasks, we interviewed the users by asking about their experience with the system, comments on improvement, and applications where the system could be used. Data were collected for each user in terms of (i) time required to finish each task, (ii) 7-point Likert scale-based questionnaire to test the usability of the system, and (iii) qualitative feedback. Each user study session was recorded from three camera views, first person camera view in VR, third person camera view in VR, and third person camera view in real environment. We segmented the first person camera view in VR to record the objective results in terms of time needed to perform the tasks. We audio-recorded all the subjective comments and suggestions from the users for post-study analysis and summary. The subjective feedback is later used in the paper to explain the study results and inspire for future design insights. The study lasted for approximately 60 min. Participants were given $20 compensation for taking part in the study.

All participants were able to complete every task. Results from questionnaire surveys were analyzed to identify users’ scores on each individual aspect of the system, how usable were the interaction techniques, how relatable were the actions as compared to the real-world entities, and how immersive was the experience. All Likert scale questions are reported with mean (M) and standard deviation (SD) in Table 2. We also provide user’s reactions that were gathered during the conversational interview in Sec. 7, depicting their experience while performing the tasks.

Task 1: Point Cloud Interaction for Object Retrieval: During this task, users used the brush tool for setting up the virtual welding environment. The time of completion for the task was first extracted from the video analysis, and then checked for any presence of outliers. No outliers were observed for the time of completion of task 1 across the DUs. See Fig. 9(a) for the box plot for the data distribution of the time of completion. The average time (in minutes) needed to finish this task was found to be M = 24.24, SD = 10.16. User ratings from the questionnaire survey following this task were grouped into three categories to check the accessibility (A1–A6), engaging (A12–A17), and effectiveness (A7-A11) aspects of the point cloud interaction using VRFromX. See Fig. 10 for more details.

Task 2: Behavioral Modeling for Virtual Objects: With the help of the affordance recommender, users attached various functions to the virtual objects to make the virtual scene interactable. The time of completion for the task was first extracted from the video analysis, and then checked for any presence of outliers. Two outliers were observed for the time of completion of task 2 across the DUs, with values of 19 and 17. See Fig. 9(a) for the box plot for the data distribution of the time of completion. The presence of the outliers was thoroughly examined to ensure they were not data collection errors. It was observed that the high values of the outlier data points accounted from (a) the corresponding users having low familiarity using VR applications, (b) the corresponding users being novices in VR programming, and (c) the system’s ability to allow users to iterate upon the authored prototypes, e.g., users had the flexibility to revisit the point cloud interaction during task 2 to retrieve object models if needed. Thus, the outliers were retained in the analysis due to their potential relevance to the research, the use of clear instructions and standardized procedures for data collection across all users, and the use of non-parametric analysis methods (Mann–Whitney U analysis) which are less sensitive to outliers. The average time (in minutes) taken to complete this task was found to be M = 8.46, SD = 3.56 (including outliers), and M = 7.40, SD = 1.46 (excluding outliers). User ratings from the questionnaire survey following this task were grouped into three categories to check the accessibility (B1–B3), engaging (B11–B16), and effectiveness (B4–B10) aspects of the behavioral modeling using VRFromX. See Fig. 11 for more details.

Task 3: Virtual Welding: During this task, users interacted with the virtual scene that they created during the previous tasks. The time of completion for the task was first extracted from the video analysis, and then checked for any presence of outliers. An outlier was observed for the time of completion of task 3 across the DUs, with value of 30. See Fig. 9(a) for the box plot for the data distribution of the time of completion. The presence of the outliers was thoroughly examined to ensure they were not data collection errors. It was observed that the high values of the outlier data point accounted from (a) the corresponding user having low familiarity using VR applications, (b) the corresponding user being novice in VR programming, and (c) the system’s ability to allow users to interact with the authored scenes as desired. In the case of the outlier, the corresponding user performed virtual welding eight times to achieve a desired weld, while keeping track of the work parameters. Thus, the outlier was retained in the analysis due to the user’s potential relevance to the research, the use of clear instructions and standardized procedures for data collection across all users, and the use of non-parametric analysis methods (Mann–Whitney U analysis) which are less sensitive to outliers. The average time (in minutes) taken to complete this task was found to be M = 12.18, SD = 7.00 (including outlier), and M = 11.25, SD = 5.76 (excluding outlier). User ratings from the questionnaire survey following this task were grouped into two categories to check the engaging (C8–C14) and effectiveness (C1–C7) aspects of the virtual welding using VRFromX. See Fig. 12 for more details.

The results from survey questionnaires and video analysis were divided into two groups based on VR programming expertise of the users. This analysis was performed to examine the effect of prior VR programming experience on the user perception and performance during authoring using VRFromX. It was verified that the normal distribution assumption was not met by conducting Shapiro-Wilk normality test on the data (p < 0.05). Therefore, Mann–Whitney U test was conducted on the survey results and the quantitative data for time of completion for the three tasks to compare the performance of the two user groups. The mean (M), SD, U-scores, and p-values from the comparative analysis are reported in Table 2 with statistical significance indicated by p < 0.05. From the analysis results, the user experience and perception of EVRPs versus NVRPs were almost similar for the three tasks during the study with no significant differences. An on-the-border significance was found for the user rating if the retrieval results were obtained as expected during the task 1 (U = 22.00, p = 0.04), where the average rating obtained for EVRPs and NVRPs were 5.75 (SD = 1.28) and 4.50 (SD = 0.90), respectively. The time of completion for the three tasks, extracted from the video analysis, was checked for any presence of outliers across the two user groups. For task 2, two outliers were observed for the time of completion of task 2 across the NVRPs, with values of 19 and 17. See Figs. 9(b) and 9(c) for the box plots for the data distribution of the time of completion across the two user groups. EVRPs took an average of 16.22 (SD = 7.36), 6.65 (SD = 1.21), and 6.96 (SD = 3.85) minutes to complete tasks 1, 2, and 3, respectively. NVRPs took an average of 29.58 (SD = 8.13), 9.67 (SD = 4.12), 8 (SD = 1.41), and 15.67 (SD = 6.50) minutes to complete tasks 1, 2 (including outliers), 2 (excluding outliers), and 3, respectively. There was a significant difference observed in the time of completion of the three tasks in both groups: (i) task 1 (U = 8.00, p* = 0.00), (ii) task 2 (U = 18.50, p* = 0.02) (including outliers), and (iii) task 3 (U = 9.00, p* = 0.00). Given the potential relevance of the outliers, the use of clear instructions and standardized procedures for data collection across all users, and the utilization of non-parametric analysis methods (Mann–Whitney U analysis), the outliers were retained in the analysis. The results are further analyzed in Sec. 7 in greater detail.

We now summarize the main outcomes of the user study. The results obtained from the questionnaire survey indicate the feasibility of VRFromX as an authoring platform to transform physical environment of a welding station into a virtual welding simulator. Figure 13 shows the interactions of participants with the virtual welding simulators that they created during the user study (from the third person view in Unity^™). User ratings from the survey questionnaire proved that the interaction procedures offered by the system are simple, and easy to understand and use. Most importantly, the execution of the different interactions do not necessarily demand skill expertise in VR programming, and thus NVRPs could use the system for creating meaningful virtual experiences. Moreover, the entire process can be completed in the same environment without needing the users to switch between platforms. The free and embodied interactions inside the 3D environment did not cause fatigue to the users, rather they rated the authoring experience to be enjoyable and immersive. Being already familiar with the real welding settings and processes from before, the welding DUs were more attached to the authoring process since they could connect their activities in the virtual world to the corresponding real-world tools and interactions. The perceived response from the users regarding the weight of the OC affecting the overall performance during the tasks was low. Additionally, the perceived accuracy of the system and OC during the authoring process was rated highly. This further implies that the users enjoyed the authoring process in VR and were satisfied with the system response to their actions.

From the findings of the study and the qualitative feedback from the conversational interview, we now try to answer the research questions that were posed before. We also discuss future extensions and directions for evaluating VRFromX, and some limitations therein.

1. To what extent does the users’ expertise in VR programming affect their performance and experiences in using VRFromX to author virtual environments? From the comparative analysis shown in Table 2, it can be observed that NVRPs and EVRPs provided positive ratings of their experiences with VRFromX during all three tasks. This indicates that prior experience in VR programming is not a requisite for interacting with the system. This is in accord with the system interface of VRFromX being designed with easy-to-use and immersive features that are accessible across users irrespective of their programming skills. It was observed that both groups of users were highly enthusiastic about trying the different features and enjoyed their experience with the system. Both groups felt that the system features were easy to use and provided them with an immersive experience. There was a significant difference found in the times of completion for the three tasks for EVRPs versus NVRPs. This can be reasoned due to the lack of familiarity with using VR devices and applications in the case of NVRPs. This resulted in NVRPs taking more time to get acclimated to the immersive scenes and the button controls, as compared to the EVRPs. Despite the high completion times, the positive ratings in the survey indicate that the longer duration did not quite affect the user experience that NVRPs gained using the system. Next, we attempt to relate each of the system components to their underlying causes of providing users with accessible and immersive experiences.

   Point Cloud Interaction and Three-Dimensional Sketching in Virtual Reality: VRFromX enables users to use hands freely in their surrounding space to create virtual scenes. The free sketching affordance provided by the system enables the users to portray their thoughts in the 3D space, which in its entirety acts as an unrestricted canvas to the curious minds. From the study results, it was observed that users were enthusiastic to use the 3D canvas during their interactions with the system. As pointed out by a user, “I did not know we could do point cloud sketching to retrieve [3D] models. 3D sketching is different from 2D sketching, [it seems] very easy for beginners, [there is] no need to learn any new software, [and] we don’t need experts to do that.” This can be reasoned by stating that sketching is very natural to human beings, and is something one learns from childhood [82]. A user remarked, “I liked the freeform nature of the drawing, drawing was usually smooth. Matching the drawing (may be the shape of the object) to get objects was a cool feature.” So, it would not take much time to get accustomed to the 3D sketching as an input modality for content creation. Some users also commented on the cognitive benefits that sketching would provide when used as the input modality during the content creation, “[It can be useful,] Especially if you know what an object looks like. Maybe people would remember things better if they would sketch to get the objects.”

   The easy and free interactions enabled by the sketching modalities along with the simple features of the brush tool further helped users to select point clouds or sketch in mid-air without much effort. The brush tool interface, being attached to the user’s hands, is easy to find and reach, whenever necessary. The features of the tool span across the basic operations are needed to select point clouds or sketch in mid-air to retrieve 3D models from the database. Users could get hold of the features easily and were able to successfully construct the virtual welding scene using the brush tool. The point cloud environment also simplified the task for the user by providing a reference to build upon. Being able to move freely inside the virtual environment and using the system features to manipulate the position and scale of the virtual scene, the user could easily reach desired locations in the scene.

   Users felt fascinated by the idea of turning their sketch into a virtual model and the interactive feature provided by the system. An user pointed out, “[I] Like the 3D sketching aspect, drawing things in mid-air and the object detection, trying to turn the sketch into the object that I was trying to convey.” Through the experiment, it was shown that the object retrieval performed using point cloud selection and 3D sketch as the query input provided acceptable results in terms of the accuracy of 3D models retrieved from the database and time of retrieval using our system. The action–reaction scheme facilitated by the AI assistance was perceived well by the users and accounted further towards enhancing the accessibility and user experience of our system. Interactions with the point cloud are also interactive, meaning the system responds to the manipulations made by the user. As pointed out by another user, “[The system could] Easily detect the sketch, simple sketching detects the objects – [The system interface is] user friendly.” Although both EVRPs and NVRPs highly rated the efficiency of the retrieval system to fetch the objects as expected, there was an on-the-border significance result achieved between the two groups. This could be possibly explained by the observation from the demographics survey which indicated that the EVRPs had higher sketching skills as compared to the NVRPs (Table 1). This calls for potential improvements to the point cloud interface component of the system to further enhance its accessibility.

   During our user study, some users pointed out augmenting the capabilities of the brush tool to provide some geometrical features to create simple shapes to assist during sketching. “Use some features to draw straight lines, circles, rectangles; [We need] not draw them ourselves. Some features which are used in tools like paint, put simple shapes together to build complex shapes.” This can be achieved by adding features in the brush tool to enable creation of constraint-based geometric shapes in 3D [83,84]. According to some users, similar-looking shapes were a bit harder to retrieve. “But sometimes it (the object retrieval system) got confused. Like I was trying to get a gas cylinder, the network was giving me fire extinguisher. Maybe color and the finer details could be used to compute the differences between tall gas cylinder and short fire extinguisher.” Currently, the PointNet network used for retrieving the object models from database only use (x, y, z) coordinates as the point’s channels [62]. To improve the accuracy of the classification network, extra feature channels such as color, normal, etc. can be used. Furthermore, using additional modalities such as voice input [50] can be explored as viable solutions to improve the system’s accuracy in such cases.

   Another direction to look into is to make the human and AI interaction in VRFromX more effective. AI algorithms can be used to assist the users in performing the semantic segmentation of the point cloud scans [51]. While selecting point cloud using the brush tool, the network can provide some suggestions for areas to select to the users based on their intent. On the same note, the network can recommend features for sketch completion when the users are performing 3D sketching in VR [72]. Further collaboration between humans and AI could be targeted towards improving the accuracy and scalability of the retrieval networks through providing scope for active learning during point cloud interaction in VRFromX [85]. Using active learning techniques would make the human–AI collaboration symbiotic, meaning the AI-based networks could learn better using human interaction. For example, these techniques can especially be useful in case of unseen object classes, where user can segment and label point cloud scans for the AI-based networks to learn, and in return user interaction can be enhanced by the improved outputs from the system. We also plan to explore the potential of VRFromX in reconstructing shapes by making use of shape primitives [86] or geometric features in the form of planes, surfaces, etc. present in the point cloud [87] for object reconstruction in case of unseen classes.

   Affordance Assignment in Virtual Reality: Attaching affordances to virtual objects in VRFromX was facilitated by simple drag and drop interactions. The interaction methods also created a direct connection between the user and the authoring process by inducing a sense of realism and presence inside the immersive environment. Users could spatially attach the various functions to the virtual objects in terms of animations, displays, and controls while perceiving the spatial relationship between the object(s), their parts, and functions with respect to their virtual self. The three-dimensional authoring provides freedom to the user to perform free transformations of self or objects with respect to each other and other objects, thereby the sense of spatial awareness experienced by the users during the authoring process can be very closely related to the similar settings in real environments. For example, while attaching the functions to the virtual welding gun during the user study, the users could correctly position the spark animation to the tip of the welding gun.

   Interaction With the Virtual Environment: User interactions with their authored content added more significance to the value of the authoring process, resulting in enhanced user motivation and confidence in using the system for transforming physical reality to virtual experiences. The system enables DUs to directly test the authored content in the virtual environment without needing to switch between multiple platforms. During the testing phase, if the user figures out any scope for modifications to the existing content, the user can go back to the authoring stage effortlessly by pressing buttons on the OC and make necessary changes to the content.

2. How does the domain knowledge of users affect their perception and experience in using VRFromX to author virtual environments?

   Point Cloud Interaction and Three-Dimensional Sketching in Virtual Reality: During the point cloud interaction for object retrieval, prior experience in welding enabled the users to get familiarized with the point cloud scan of the welding station quickly and in no time. Users could make use of the spatial and color information offered by the scan to recall and recognize the welding objects found in a typical workplace setting, e.g., tools like clamp, hammer, and wirebrush were found above the welding table; the welding gun was found wired to the welding machine; the red fire extinguisher attached to the wall was an easy find.

   Affordance Assignment in Virtual Reality: The nature of the functions displayed by the affordance recommender, being very similar to the real-world processes, draw the DUs more towards the functionality authoring process, by exercising their motor and cognitive faculties to transfer the knowledge and skills gained from their real-world experience. As pointed out by a user, “[I] Liked dragging and attaching functions. [It] was easy. Which components [were] meant for each tool was obvious. Graphs were nice. Overall the attaching components part was smooth.” DUs can directly make use of and transfer their understanding that they have developed during the real-world interactions and processes easily in the virtual environment. For example, during the user study, the users could figure out the nature of the functions needed to be attached to the welding gun, e.g., spark animation to represent the graphical effects during virtual welding, angle and distance displays to visualize the welding parameters, and welding plate and trigger button controls to trigger virtual welding. Therefore, the transfer of understanding and perception can reduce the overall cognitive effort required during the process. A user pointed out, “It was very interesting. [It] Works very smoothly. [It is] Easy to use, and very easy to figure out.”

   Interaction With the Virtual Environment: When performing virtual welding, the realistic content and high fidelity of the affordances associated with the virtual objects enhanced the user perception about the virtual welding simulator. As pointed out by a user, “Virtual welding behavior [was] somewhat realistic, that part is somewhat reasonable.” Being aware of the real-world processes creates a sense of satisfaction when the user can test that the authored content in VR behave as the real-world version of the object would behave in the real world. A user pointed out, “[The processes are] Relatable to the real world interactions, because I have done it before.” Few users even highlighted that the virtual welding simulator could be used for basic MIG welding training purposes, “When you are holding the gun, you can see the angle and improve the performance. [It is] Cool how you can track that and get some feedback. That’s the part which can make it good for training. It was cool to see the feedback. You can try it in VR first before doing it in real.” Some welding experts also mentioned about the use of the virtual welding simulator to provide beginner level experiences to welding novices, e.g., develop knowledge regarding welding related objects, setting up the welding space, etc. A welding expert pointed out, “[The virtual welding simulator can be] Good first space [to teach] how to set up a welding space, what the various parts are, [and] for the beginners, to get used to the system.”

   Insights were gained from the study to improve the efficacy of the virtual welding trainer. Based on the comments of the DUs, the 3D models in the virtual welding environment looked cleaner and visually more attractive as compared to real-world settings. The issue could be fixed by adding textures to the list of affordances for the virtual objects. As pointed out by a welding expert, “[There might be some] Trouble translating from real to VR, [especially, the] Way to hold the welding gun. [It feels] Sensitive not having the actual way of holding the welding gun in your hand.” One way to solve and address this issue can be achieved by replacing the OC from user’s hands with an IoT-enabled real welding gun. The effect of haptic perception on the user experience inside the virtual welding environment can be explored as a scope of future work [88]. Furthermore, future research can explore the effectiveness of such authored applications for training purposes and evaluate the transfer of skills from VR to the real environment [88,89].

   A limitation of the current work is related to the absence of statistical power analysis to estimate the sample size for the user study, which was due to constraints in finding users with welding experience for the study. Despite this limitation, we believe that the evaluation results from the study still provide valuable insights into evaluating the effect of technical expertise and domain-specific experience towards interacting with the system components of VRFromX. Although the sample size was constrained, we made efforts to mitigate potential biases by keeping welding experience as the only criteria for the user selection, utilizing clear instructions and standardized procedures for data collection across all users, and utilizing non-parametric statistical analysis methods. We believe that the insights gained from this study contribute to the existing body of knowledge in the field of providing accessible methods for adoption of the immersive VR technology for domain-specific applications, particularly in the context of accessible authoring of VR applications [7]. Nevertheless, we acknowledge the benefits of future studies with a proper sample size estimation using statistical power analysis to further validate and extend the current findings, and hope that our work provides design insights to explore this area in greater detail.

We further analyze insights from prior work and our own observations during the user study to derive potential applications of our system.

Potential Applications: The accessibility of authoring of the VR-based applications should keep in pace with the increasing trend of use of the technology in various sectors such as training and education, gaming and entertainment, and industry. By enabling easy authoring of virtual content, VRFromX facilitates easy accessibility of the technology to create useful applications in VR. During the conversational interview, users commented about the use of VRFromX to author applications for (1) training in industry, e.g., “Training purposes – drill mills, saw, CNCs,” “In my old factory, I could use this for setting parameters, training for setting parameters. Training how to do set up parameters for machines. Useful when the task might be dangerous. Especially when they are doing some dangerous tasks, VR takes the danger out,” “A lot of potential in medical usage. Training people to perform surgery;” (2) education, e.g., “Comfortable way to teach a soft skill, before you actually try on a physical set-up,” “Lab scenario – students can do it at home, can go to lab for data collection, rather than having problems with the set-up;” (3) entertainment, e.g., “Escape room-It’s a game so that people can escape from room to any mysterious world;” (4) safety, e.g., “Trial and error, process is expensive to perform in real. Scan the process, bring in VR, do in VR for user experience, save the cost. Process is hazardous, VR would help there;” (5) assembly, e.g., “Assemblies and stuff like that where you don’t have multiple choices;” and (6) human–robot interactions (HRIs), e.g., “Stories about up and coming robotic devices, [where] surgeon [is] in remote location and patient [is] with the robot. Surgeons with basically the joystick and perform the procedure remotely,” etc.

Motivated by the insights collected from the user study, we implemented two different use case scenarios with an aim to justify the capabilities of our system to perform activities in MR environments, such as (1) remote 3D-printing and (2) robot-IoT task planning.

Remote Three-Dimensional Printing: The second use case demonstrates an example integration of VRFromX in maker education to enhance students’ learning experiences. Prior work has shown that maker education, which emphasizes hands-on based experiential learning and student creativity [90], can be enriched using the MR technology by offering a more interactive and engaging learning experience [91,92]. First, MR learning environments can allow students to visualize and manipulate objects in three-dimensional space. Second, the seamless integration of real-world elements into the virtual space allows students to bridge the gap between abstract ideas and tangible creations, making their learning more tangible and meaningful.

In this context, the second use case demonstrates an example of MR fabrication using IoT-enabled devices (Fig. 14). User interacts with the scanned point cloud of a 3D-printing area to retrieve a virtual 3D-printer and assigns functions to facilitate interaction with the IoT-enabled 3D-printer in the real world. Using the brush tool, virtual models are retrieved using point cloud selections or sketching operations. When the user puts the selected virtual models on the plate of the virtual printer, the models are sent to the physical 3D-printer for remote printing. Such applications can specifically be used in maker-based educational settings to encourage high school students with little knowledge in technical modeling. Using sketching features, they can retrieve 3D designs for maker-based prototyping and print models remotely without the need of going into the complexity of constraint-based 3D modeling softwares [93].

Robot-Internet-of-Things Task Planning: Addressing the challenge of effective communication of intent between humans and robots in smart environments has been a longstanding issue [94–96]. Previous studies have presented promising solutions for HRI by leveraging the surrounding environment as a shared canvas for visual and spatial cues [97], and enabling humans to convey instructions to robots for performing tasks in smart environments that incorporate multiple IoT devices [98]. In this context, the third use case aims to leverage the advantages of VRFromX in HRI specifically for authoring robot-IoT tasks, fostering seamless interactions among humans, robots, and the smart environment.

The third use case is demonstrated in Fig. 15. Using the scanned point cloud overlaid in the virtual environment, user specifies the path that the robot has to follow by selecting waypoints and authors the robot tasks at desired waypoints. Using the brush tool features, user retrieves the models of IoT-enabled devices (TV and 3D-printer) that the physical robot has to interact with and assigns functions to them (Fig. 15(a)). Finally, the physical robot navigates on the path and interacts with the IoT-enabled devices in the real world as authored by the human (Fig. 15(b)). With intuitive and user-friendly interfaces, VRFromX can empower users to scan any physical environment for task authoring and remotely control the robot-IoT tasks at their convenience. By directly programming the robot through simple spatial interactions, humans assume the role of the robot’s “brain,” guiding it to perform the intended tasks within the smart environment [98].

We have presented the system design and workflow of VRFromX that transforms physical reality into interactive virtual experiences inside a VR environment. We have demonstrated how the system design facilitates an embodied user interaction on scanned point cloud to create virtual content using AI assistance. We also demonstrated the usability of the functionality and logic-authoring tool to attach affordances to the virtual objects for a meaningful end-user experience. The design concepts used in VRFromX were successfully tested using a welding use case on 20 domain users. The study results confirmed accessibility, effectiveness, and engaging aspects of the system in authoring virtual experiences irrespective of the users’ VR programming expertise. We further implemented two more use cases to demonstrate the diversity of potential applications of VRFromX in mixed reality scenarios. By leveraging direct knowledge transfer from DUs to create VR applications, VRFromX provides an accessible platform for easy and greater adoption of the authoring process by allowing DUs to become the makers of VR applications themselves. By leveraging cloud-based architectures inside VRFromX, the system’s potential can be further extended to foster collaborative work where multiple participants from interdisciplinary fields can author virtual content together. Such collaboration, while facilitating the scalability of the authoring process across inter-related fields, would also promote creativity by supporting the joint exploration of design spaces in the virtual world.

This work was partially supported by the National Science Foundation under Grant FW-HTF 1839971. We also acknowledge the Feddersen ChairFunds. We thank Hyung-gun Chi and Asim Unmesh for their help regarding this work. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the funding agency.

There are no conflicts of interest.

The authors attest that all data for this study are included in the paper.