Overview

This project was a collaboration between 21 of the top students from across the school of design who were challenged to create an interactive sculpture for the Gulfstream Center for Design to showcase the possibilities within the field of design. Below is a glimpse inside our 20 week journey of going from paper prototypes to the towering mechanical creatures that we have today. 

The Spark

The catalyst that sparked this project into existence was the idea that a robotic arm could be driven with a pulley system instead of the conventional motor on every joint that you see in assembly robots. This not only slims down the arm by reducing the amount of parts needed but it also makes the movements more flowing and organic like the tentacle of a sea creature. While just seeing it move gracefully on its own would be fun to watch, we thought we could engage our audience of prospective designers by adding an interactive component. 

With the idea of an interactive tentacle creature as a centerpiece, there was no shortage of excitement in the air as ideas started to pour in. While an abundance of ideas is a blessing, our time constraints wouldn’t allow us to pursue every idea. We needed a way to sort them all out…

Flimsy Beginnings

We found the best way to do this was to actually execute the idea, but not at its full scale. In an effort to understand if an idea was feasible without sacrificing too much time or money, we employed rapid prototyping methods with cheap materials such as paper. We could then see the problems that we couldn’t see as a conceptual idea and if we even liked the idea. From there we could make the informed decision whether to move forward or scrap the idea.

Bite Size Code

We treated the code in the same way: starting with small tutorials for our tracking library MediaPipe, a multi-language library that allows for the implementation of AI and Machine learning solutions through a standard webcam. To move the motors on the Arduino side we used the AccelStepper library which gives precision control over position, speed, and acceleration of a stepper motor – very similar to how a 3d printer operates. After figuring out these two bite-sized ideas, we did a similar small test with I2c communication between RaspberryPi as the controller and Arduino as the peripheral. Only after we understood these small examples could we begin to think about combining them together. It wasn’t about just making something work. We needed to critically consider multiple solutions to make it as efficient as possible, not only for the code we are currently working on, but for compatibility of functionality developed in parallel – as well as in the future. 

Puppeteering

Without the mechanisms to move, there is nothing to code. Not only that, but how it moves influences the difficulty of the code moving the mechanism so it was essential that the programming team work with the kinetic team to consider all variations of movement and what is most suitable for future development.  

While a standard front, back, left, and right configuration of the tensioning cables is easier to program, a configuration with two cables on the left and two on the right produced more life-like movements, sacrificing some intuitiveness on the coding side but ultimately creating a more engaging experience.   

Plan of Interaction

Much like the common pitfall “scope creep” in game development, or the gradual growth of project requirements beyond the initial scope, the abundance of ways we could interact with the sculpture, while no doubt cool, could stall progress if we pursued each one. Our team decided to create a tiered system to prioritize the essential parts of the interaction that the project couldn’t exist without first. This gave us a structure but provided goals to strive for.   

Follow My Lead

As shown in the chart above, following the user in some way was the most essential part of the interaction, so I got to work using MediaPipe and AccelStepper to move a pair of Nema 34 steppers. My goal was to get a side-to-side movement through a reciprocating motion where the direction that it pulled towards would be tensioned by pulling up on the cable while the other motor would be rotated the opposite direction relieving tension by the same degree. 

After a lot of flipping motor positions and math calculation I was finally able to attach it to our tentacle mechanism and create our first tracking prototype running two motors. Although motion was slow this was a huge moment for the entire team because we could finally see a somewhat complete result, strengthening our confidence that our grand ideas were achievable. 

First prototype with motion tracking

Continuing our method of adding small changes and making sure they work, we started to add the second two steppers to control the upper tentacle section side-to-side while the previous two adjusted side-to-side of the bottom section. The compound movement that this created elevated the engagement of the movement but added an additional level of complexity. If the top segment was at one extreme or the other, the bottom segment’s tracking position would be different if it was in the center versus looking on one side or the other resulting in the tentacle not pointing directly at the user when not in the center of the camera’s field of view. To solve this, we adjusted the bottom segment’s range of motion proportionally to the positions of the top segment motors. If the tentacle’s top section was to either extreme, the bottom segment’s range would be limited more than if the top segment was centered to ensure the tentacle pointed to the user at any location. For example, if the top section is centered, the bottom segment can use its full range of motion to point at the target left or right. If the top section is at either extreme, the left and right movement of the bottom stage is cut in half in order to look at the same height as when the top stage was in the center. But that only covers the side-to-side movement. In addition to horizontal movement, we added the ability for it to look up and down by adjusting the already clamped range’s position proportionally depending on the height of the tracking point.   

Organization for Optimization

With such a large scale project with multiple people working on the same code over a long period of time it’s easy for code to get confusing and for time to be wasted trying to rewrite ,find, or understand previously written code. In order to prevent this we needed an overall organization structure, a way of sharing current code quickly, and ways of communicating functionality. To organize the python files we separated the scripts by functionality and placed critical processes on separate threads such as the getting and displaying of frames from the webcam feed. We used Git to track the changes to the code as well as used a remote repository on Github to quickly share code between multiple machines. Finally, commenting our code gave a quick summary of how things were working, while discussions during work sessions allowed us to support each other and keep up to date with what is happening in other aspects of the project. All of these methods gave us a strong organizational foundation that complimented our philosophy of small incremental changes to maintain a stable code base. 

Usability

User testing gets harder when there is a physical prototype involved, especially at Biome’s large scale. Testers have to come to one location and, since multiple teams are working with one prototype, there is no guarantee that it will be available. Luckily, the project gathered enough attention from curious passersby in the shop and through social media that we were never short of a fresh perspective to test our interaction. I took the same approach with everyone who inquired: first not giving them any information on how to control the tentacle to see if they could figure out how to control it. From our idea of the interaction outlined in the interaction plan, we wanted it intuitive enough to interact with that you could approach it and immediately have control, but still need a little skill  in order to provoke different responses, adding a sense of discoverability like a wild animal. This adds replayability as people have a connection with the project and are eager to discover more or bring in others by showing their friends what they can do with it. 

Here are some of our biggest findings from user testing:

Features

  • The amount of pull on the bottom segment motors is proportional to how far the top segment is to one side of the other so the arm will not overshoot its target when the top segment is tensioned to either extreme.
  • The speed and acceleration of the movement can be changed at any point in the code including mid tracking. 
  • To prevent injury and encourage optimal distance for the best experience, the tentacle will deactivate when a person gets too close (their hip position becomes lower than the bottom of the screen) and still tracked within the camera (It doesn’t catch everything. It could lose tracking on them and move on to the next target)
  • The animation system takes predetermined animations which movements are organized into frames in a .json file and can be called anywhere within the code. The speed, acceleration, and delay can be changed for every frame. 
  • Animation positions can be absolute or relative. An absolute positioned animation goes to the same position regardless of current position while a relative positioned animation will move based off of the last position it was in before the animation started.
  • Different animations are played depending on the zone it is when it starts the animation. If it is on the right side it will play one set of animations vs. on the left it will play a different set. 
  • To prevent one person from hogging the interaction, the tracking will deactivate after tracking has been active for a set interval giving it a chance to pick a new target
  • Since steppers don’t remember their position when they don’t have power, Hall effect sensors detect a magnet on the shafts of the motors to find the 0 positions of the motors. This is done automatically on startup by cycling each motor and finding the center of magnetic field by finding the rising and falling edge of the hall effect sensor activation.  
  • The tentacle will track the hand that is higher than the other. This is because in most situations the user raises or extends the hand upwards when they want that hand to control. An exception is when they are recording a video on their phone and controlling with the other, sometimes the controlling hand is lower than the hand holding the phone.

The Team

Students:

Aaron Allen, Furniture Design

Alex Arnold, Industrial Design

Ana Luiza Fortes Viana, Furniture Design, MFA

Annie McElree, Furniture Design

Bryce Fuller, Industrial Design

Charlotte Tinguely, Industrial Design

Chloe King, Industrial Design

Drew Hart, Furniture Design

Ciera Dylan Cummings, Industrial Design

Emma Wosje, Industrial Design

Evan Baker, Furniture Design

Grant Given, Industrial Design, MFA

Javier Chau, Industrial Design

Jonathan Hall, UX Design

Luke Langford, Industrial Design

Macy McCarthy, UX Design, Industrial Design

Marta McWhorter, Furniture Design

Max Juliao, Furniture Design

Saskia Eberman, Industrial Design

Soso Leung-Wolf, Production Design, MFA

Professors:

Aaron Heisler, Furniture Design

Clark Delashmet, UX Design

Reflection

Biome is one of my favorites that I did at SCAD. While traditional classes focus on assessment oriented tasks that restrict projects, this project’s focus on producing a final result allowed me to pursue topics that interest me and implement them with the guidance from the professors and students when I needed it. 

In projects such as this that are striving for a particular result beyond a grade in a class, an idea – whether good or bad – might not see the light of day in the final product no matter how many hours are put into it. It can be extremely discouraging for the people who have put the time into it, often feeling like no progress has been made. While it might seem like it on the surface, these efforts can actually propel the project forward by creating constraints to work within and create a log of experienced ideas that can be referred back to and pulled from with ease in future situations. This is just as important to realize as someone whose work is not directly impacted as it is for those whose work is directly impacted. As a leader it’s important to acknowledge the work that has been put in by others and understand the progress that has been made in case one needs to return to it in the future. As someone who has had their work rejected, it’s important to understand the progress that has been made so if the opportunity does arise you can share your findings for the benefit of the rest of the project. 

My philosophy when it comes to these kinds of struggles is that when pursuing something that hasn’t been done the decisions being discussed haven’t been thought of, so in the journey into this unknown there is inherent tension. That’s how you know you are on the right track. It’s how you deal with the problems that make a project great.