Immersive Engineering Project
Snapshot Overview
Project Dates: January 2025 through July 2026
Technical Skills Used:
-
Circuit Design & Analysis
-
Machine Learning Model Design
-
Real-Time ML Model Deployment
-
Sensor Integration
-
Computer Aided Design
-
Additive Manufacturing
Project Outcome(s):
-
Design of an easily maintainable animatronic dragon figure with a rolling trolley mount.
-
Incorporation of a pose estimation machine learning model to drive the gaze following mechanism.
-
Electrical harness designed to withstand transportation and repeated use.
Interactive Animatronic Figure
Here be dragons - and the engineering to prove it! The development of an interactive dragon animatronic was sparked by a desire to include immersive experiences to tours around the campus of New Mexico Institute of Mining and Technology (NMT). I decided to mix this desire with my passion for mechatronic system design and machine learning to develop my interactive animatronic dragon figure. The animatronic figure itself needed to be mobile enough to relocate to various spots on NMT campus while maintaining size large enough to provide immersive experiences to larger tour groups. It was important that the figure be easily maintained, as it is meant to last well past when my time at NMT comes to an end.
​
To achieve the goals previously stated, I designed the animatronic dragon figure atop a trolley with attached lockable caster wheels. The trolley system benefitted the system as both a form of transportation as well as a perfect mounting position for the camera system being used to drive the machine learning algorithm trained for this project. Additionally, the design of each actuation point was meant to be easily replicated in a smaller scale for discussing with students during tours. This simplicity in demonstration was important to me, as I want students to be inspired and see a project they could successfully produce on their own.
Image Coming Soon
Machine Learning Driven Interactions
The dragon tracks and follows visitors in real time using a custom-trained pose estimation model. I trained and implemented the model onto a NVIDIA Jetson AGX Orin which was used to drive the entire figure. Real-time video input is collected using an Intel RealSense RGB-D camera which is provided to the trained model for inference. Key point locations for both eyes of detected individuals is noted and used to determine an output angle for driving eye mechanism of the figure. Eye key point locations were temporally smoothed using an exponential moving average to reduce jitter in angle output calculations. An example of this functional algorithm in action can be seen in the video to the left.
