Activity 2: ProtoPie Connect

I followed along with all the instructions for this activity and I only briefly struggled with having my laptop unable to connect to my phone due to me running a background function and missing the step to turn it off. I also think that this kind of work with the Arduino works great with the set-up I have, a windows gaming laptop and an apple phone. I really enjoyed seeing just how functional ProtoPie can become as I really liked how in depth the features are.

Activity 3: TouchDesigner and Projection

This activity got me thinking more about what I could actually create with physical computing. I started exploring different ideas and researching how things like movement tracking, gesture recognition, and projection mapping could work together. I really like the idea of combining physical and digital interactions, so I wanted to see just how far I could take that. I still have a lot to figure out, but I'm excited to keep exploring these ideas and see what I can make out of them.

Research Activity

---

Consumer Trend Research

I researched two consumer trends: Hybrid Play and Responsive, Human-Centred Tech. Both pointed me towards projects that mix physical and digital play and that people feel comfortable using.

Personas Research

From the trends I built two personas, one for each category, using the statistics from my sources to describe who they are and what they care about.

HMI Research

For each trend I wrote a How Might I (HMW) statement and identified two problem spaces to design around.

Project Path

To work out which direction to take, I looked at open-source Arduino projects that overlap with my ideas, researched projection mapping for inspiration, and then sketched several concepts.

Research: AI Room Tracker

Many of my ideas hinge on the idea of being able to somehow see what the user is doing in order to prompt a predetermined response. This AI Room Tracker while fulfilling a different purpose, has the same base functionality of needing to survey the room for movement. I feel like keeping this circuit diagram in mind when planning how my projects will function will be important, it gives me key context of the things I need to design around. I also really like that the components used and the code is accessible as something I can reference.

Research: Joystick-Controller

A core proponent of many of my interactive ideas is some kind of controller, so I figured this would be a good reference if I need in the future. It also gives me information regardless of if I use a controller in my final project, this allows me to see what a potential idea could look like. I feel like understanding how something like a joystick fundamentally works with the Arduino helps me understand better what I’m working with regardless of if I use it or not.

Research: Follow-Me Tracking

This one tracks a person in real time and uses that to move something so it follows them, which is pretty close to what I want for my own projects, just for a different purpose. It combines colour and depth data to find the person and then adjusts its speed to keep up. I thought it was interesting that it runs in simulation, so I can see how the tracking works without needing a full robot. I feel like that could be a good way to test an idea before building anything physical. I also really like that the code and documentation are open for me to reference, but I’ll need to check that the board it uses (the Arduino VENTUNO Q) is something I can actually get.

Research: Hand Gesture Recognition

Gesture sensing is something I’ve been wanting to look into, since it could let someone control a projection without touching anything. This project takes a live video feed, finds the hand, and figures out which gesture is being made. What stood out to me is that everything happens on the device itself, so it’s fast and doesn’t send video anywhere. I feel like that matters for my project because I don’t want people to feel watched. It runs on Qualcomm Dragonwing boards and not a regular Arduino, so I’m not sure I could use it directly. The code and a blog post are available, so it’s still a good reference for how gesture detection is put together.

Research: Pose Estimation

This project finds the joints in a person’s body in real time and draws a skeleton over the video. I think this could be useful for my idea of picking a doll for someone based on how they move around. It uses an AI model running on the Arduino VENTUNO Q and, like the follow-me project, it comes with code and a documentation PDF I can reference. The tricky part is that it needs more powerful hardware than the Arduino I’ve been working with, so I’d have to check what’s available.

Research: Projection Mapping Inspiration

I wanted to look at projection mapping to see what’s possible. School of Motion shared a few of their favourite examples, and what I liked most is how different they are. Box uses camera movement and perspective to make it feel like a magic trick, Projections in the Forest projects onto living plants instead of a flat wall, and the Cavs one happens right on the basketball court. There was also a real-time face tracking example from a team in Japan. I feel like the face tracking one is closest to what I want, since the projection reacts to the person. I also like the idea of projecting onto something that isn’t a plain screen, since that could make my doll feel more physical.

Ideation: Help Me Get Dressed

For this idea I was thinking about old flash games from when I was a kid and how fun the dress up aspect of them is. The doll would be shown in her bedroom asking the user to help her get dressed for an outing and then the user would select all the outfit components. The doll would react to the choices with predetermined responses just like in those older games with phrases like “oh..” or “this is NOT my colour!” For how the user interacts with the projection I did not have much in mind, maybe a simple remote, maybe even hopscotch.

Ideation: Do I Look Like You?

I briefly pondered the idea of projection mapping being used for a “scary” or “creepy” idea rather than something focused on beauty or accessibility. My idea was that maybe the user would answer some sort of questionnaire using ProtoPie and then depending on the answers like a personality quiz a specific “scary” doll would appear. The doll would then taunt them in some way and then attempt to “break out” by pounding on the screen. Ultimately I decided that I am not that attached to this concept and it is something a broad audience may not tolerate.

Ideation: Baby Monitor

I was also considering creating a baby monitor that uses a combination of tracking movement and sound to alert family members of the baby’s movement. It would also have a special much more intensive alarm for if strange movement is detected or no movement. I do not have a baby or know one right now, but I appreciate all the effort that goes into making a child’s first couple years fun, safe, and entertaining. If the child is old enough to have the doll in the crib or attached to the side of it I think that would be my best target demographic. I like this idea, but it is not as fun as I would like.

Ideation: Is This Dog Friendly?

One of my first ideas was to use projection mapping and maybe some kind of gesture detection to have users interact with dogs. I have always had dogs and worked at a dog daycare the past couple summers but I find it quite alarming how little the average person knows about interacting with them. I would have like a book of dog posture and behaviours with what that means for the dog (eg. stiff tail is not happy) in front of the projection mapping for users to reference. I don’t like how many people interact with dogs clearly displaying behaviours asking to be left alone and that leads to bites, injuries, and fights.

Ideation: Select a Doll

This idea while being one of my more vague concepts is also one of my favourites. I feel like this could be possible through projection mapping, maybe sensors underneath the table that the physical objects are set on, and perhaps something like sensing gestures. The concept for this project would be having the user choose a doll that they like and then having that doll appear on the projection screen in front of them. The user would then dress the doll they are holding like playing barbies as a child. I really like the idea of combining digital and physical play, I am passionate about this idea.