The Challenge

The Client required a Mixed Reality (MR) solution to the problem of assisting with the assembly of a vehicle gearbox as a training exercise. This called for the trainee to be able to have detailed instructions displayed in front of them as well as ‘seeing’ certain gearbox parts overlaid on the real areas of the system to aid assembly

The Solution

Yeovil College contacted Unity Developers about the possibility of developing Meta Quest 3 MR App that would help engineering students to become familiar with complex engineering tasks. In this case, the task was the assembly of a gearbox with certain components being replicated in virtual 3D form.

To achieve this, Unity Developers realised that the Meta Quest 3 in passthrough mode would be the best system to deliver this content, but there were a number of challenges that would need to be overcome to make it work.

There are numerous advantages to employing MR for student training. These technologies offer a secure and regulated environment for users to acquire and refine new skills, without the threat of real-world repercussions. Additionally, they provide a more immersive and engaging experience that can be replicated again and again.  However, a development of this nature had a number of uncertainties that needed to be addressed. The main issues were:

• The precision use of passthrough to place parts accurately.
• The use of virtual hands with full hand tracking.

Developing virtual models of the replacement gearbox parts required the use of precision modelling packages and was fairly straightforward, but enabling the use of passthrough and ensuring that the virtual hands could interact effectively with those parts required a lot more effort.

The big problem came with the need to track the virtual parts alongside the physical model and ensure that the passthrough feature worked perfectly in all cases.  Even the slightest mismatch would be obvious to the user and would make the simulation seem unreal, thereby losing the point of it.  To combat this Unity Developers developed a bespoke methodology to ‘lock’ the virtual models to the physical parts using advanced spatial and visual tracking.  Precision movement was key to this, and became the major challenge.

The second major challenge that needed to be overcome was that Meta Quest controllers were insufficiently precise to allow for the grasping and positioning of small parts.  To ensure both spatial precision and reality of the task, Unity Developers needed to integrate full hand tracking into the App. With this enabled and calibrated, it was possible for the trainee to grasp and manipulate any of the virtual parts that were needed for the assembly.

In addition, Quest 3 MR app needed to have detailed instruction on the current operation at every step, so that the trainee could walk through the rebuild operation in a logical manner and get each step right.  Unity Developers were able to achieve this using a combination of spatial awareness and advanced full hand tracking.

Yeovil College wanted an effective training system for Engineering students, and Unity Developers were able to deliver that. Thanks to the success of this project, there is scope for more projects of this nature, making training more enjoyable and hands on.

The MR engineering App demonstrates the power of both the Meta Quest 3 and Unity software in creating MR experiences, and the Unity Developers team are experts at developing this kind of educational content.