Lessons Learned
- Split big complex problems into small digestable ones
- A little time preparing saves a lot of time doing
---
This might be one of my favorite projects to date. Since I was little I've been facinated with non-traditional languages
and the expressive nature of American Sign Language (ASL), so it seemed like a fantastic challenge to build a machine Learning
model trained on more dynamic signs. I had messed around with machine learning in the past, building a
facial recognition from scratch, but that was with static images while the goal for this was
to recognize dynamic movements. I opted to use libraries for this one which made it a lot simpler, but one of the biggest improvements I made
since my last attempt at machine learning was code structure and documentation. A lot of that was from my time at John Deere.
There were a couple problems I faced with this project:
- How do I store such a large amount of data?
- Where do I find enough data?
These were addressed, in a way, by decreasing the amount of "noise" in the data. This was done by using the mediapipe hand landmarker model. By passing the data through this, the data was reduced from the thousands of pixel values that are associated with each frame to less than 100 data points per frame. This process not only allowed the model to focus only on the important parts, but also largely addressed the issue of storing the data.
A frame of the sign "hello" with hand landmarks shown
At one point I played with a posture recognition model to further enhance the data, but found this to be overkill and removed it for efficiency. As an additional step in the process, to double the speed and coverage of data collection, I added the option to flip the data over the y-axis to mimic someone signing from the opposite side. This reinforced the idea that it's easier to split big complex problems into small digestable ones.
While working on this project I found that a large amount of my time was taken up by waiting for TensorFlow (the ML library I used) to load when I wanted to do any testing or data creation which prompted me to combine the GUI for each step into a single application. This significally sped up development. A little time preparing saves a lot of time doing. Converting the scripts to a single application led to me changing the goal, instead of being about successful recognition of ASL it became creating a tool which highlights the process of machine learning and neural networks. This change prompted the addition of allowing the model to be defined within the application.