Making Motion

So sorry we left you hanging with our last post, but it took us longer to get packed than we anticipated.  So thanks for waiting patiently.  But with coffee in hand, we are now ready to hit the road to the Land of 3D Air Mice.  Our first stop along the way is the venerable PC mouse since computer mouse dynamics provide the underpinning for today’s 3D air mice.

Believe it or not, the first mouse prototype was developed way back in 1963 by U.S. inventor Douglas Engelbart working at SRI International.  It was wooden and had two metal wheels.  Engelbart first called it an X-Y position indicator for a display system.  Then he and his team had the better sense to call it a mouse because, well, it looked like a mouse.

The first public demonstration of the PC mouse was held in December 1968 at what is now known as the Mother of All Demos.  The SRI team showcased videoconferencing, hyperlinks, text editing, networked collaboration and of course the mouse.  (There is no truth to the rumor that Engelbart also demonstrated Facemash which is a lie propagated by the Winklevoss twins.  Ok, we just made that up.)  If you have a historical interest, you can find videos of the public demonstration here.

So that’s great to know, but what does this have to do with our 3D air mice?  Well we are glad you asked.  There are two really important aspects to computer mouse dynamics that we are certain you haven’t given much thought.  It’s because they work so well, you probably don’t even realize it.

First, the computer mouse provides a relative pointer.  By this we mean it provides a cursor that is bound to the screen.  A bounded cursor always remains visible on the screen no matter how far in any direction you move the mouse. The distinct advantage of this approach is that the cursor is always visible on the screen and never disappears, so the user doesn’t need to “find the cursor”.  This approach is in stark contrast to absolute pointing systems with a fixed frame of reference.  For example, try navigating the menus on the Wii.  Unless you maintain pointing the remote control directly at the sensor bar, the cursor goes off the screen and now you must go hunting for it.  We really love playing games but we really hate cursor hunting.

Now the challenge of a relative pointer is that you may need to re-center the mouse to use it comfortably which can be hard for novice users.  For example, try this for an experiment.  With your PC mouse to the right hand side of your PC keyboard, slide your mouse to the right by 12 inches. Now you can continue using the mouse over at this long distance away, but it would become cumbersome to use after a while unless you are an NBA basketball player and have arms that are like 5 feet long.  So if you want to use the mouse more comfortably you now have two choices.  You can pick the mouse off the table and set it down closer to the keyboard.  Or you can just move the mouse back 12 inches to the left and basically use the left edge of the screen to bound the cursor on the way back.  Either way will work.

As we said, children and other new users of the PC mouse often take a little time to get accustomed to this.  But we are all quick learners and eventually re-centering the cursor becomes second nature.  Or we become so good at it, that we rarely need to do it.  But either way, the key thing is that no thought is required.

The second important mouse dynamic is non-linear ballistics.  This is a fancy way of saying that the mouse velocity is not constant.  When the user moves the mouse, the system measures its speed and then applies a gain to it – a multiplier.  The multiplier varies depending on the speed.  The gain curves are configured to apply acceleration as you move the mouse quickly (gain > 1) and deceleration as you move the mouse slowly (gain < 1).  This helps in two ways.  The acceleration means less effort is required to move the cursor long distances across the screen.  The deceleration enables the user to more effectively hit smaller targets on the screen.  The effect of this behavior is that the cursor will not always appear in the same position as the mouse moves around.

If you don’t believe us, try this experiment.  Position your cursor on the left side of the screen.  Move your mouse quickly to the right by 3-4 inches.  (Try not to hit the right edge of your screen with the cursor.) Then move your mouse back very slowly to the left again until the cursor reaches the left edge of the screen again.  We’ll bet that the mouse isn’t in the same position as when you started.

It’s the combination of a bounded cursor and nonlinear gain which makes the PC mouse both efficient and easy to use.  And it’s this relative pointing feature which will form the basis for highly usable 3D air mice.

So that’s our quick trip for today!  Now let’s get on the road again.  We’re going to pick up some fast food and eat on the bus to save time.  Next stop, we’ll learn more about the living room and important considerations for 3D air mice.

Ahh road trips! What’s more idyllic than the thought of a fun getaway to a new place? It harkens me back to trips with friends to Boston to the see the Red Sox and the Green Monster. Or to Wilmington, NC to lie on the beach and listen to Jimmy Buffet.

Well friends, we are going to take a road trip ourselves. Not just any road trip mind you. Today and for the next few blog entries we are going to the land of 3D Air Mice. No, these air mice aren’t found in Oz.  Those would be monkeys. This is the technology that enables a remote control to move in the air and control a cursor on a PC or Smart TV.  LG’s Smart TV products have an air mouse known as Magic Motion.

Now I know what you’re thinking. This isn’t quite the road trip you’re expecting. Surely we can’t be comparing trips to Boston or Chicago or even Jacksonville to 3D air mice. Can we?

Well of course the answer is yes. See we can’t afford to take all of our friends to get Deep Dish pizza in Chicago and catch a Cubs games. As much as we’d want to do just that, our bosses said that wouldn’t be a great use of company funds.

So we are going to do the next best thing which is show you that it is no easy feat creating a usable 3D mouse for the TV. And this is important because unless you’ve been too busy checking Facebook on your smart phone for the last 3 years, you’ll note that remote control technology for TVs is undergoing a massive change. Gone are the days of simple candy bar remote controls with channel, volume, and up-down-left-right controls. See today’s Smart TVs are much more sophisticated. They have features like web browsers and Angry Birds that demand more than the old remote control can offer.

So we’ll explore the land of 3D air mouse and what’s involved with creating cursor control for Smart TV. We’ll start by talking about the basics of how a PC mouse works. And then we’ll probably discuss challenges with 3D air mice. We say probably because this is a road trip and somewhat spontaneous and we’ll see how things go.

So we’ll get the bus and you bring the music. And hopefully we’ll have a good time and not get into too much trouble.

Have you ever worked really hard at something only to have the smallest thing be your downfall? I remember taking an engineering exam in college. I studied for hours, made meticulous notes, and made sure to get a good night sleep. I arrived early, found good a seat and popped open my diet Coke to give me the energy to get through the exam. But then it hit me. I forgot my calculator. Doh!!!

A similar problem can happen with your attempts at great sensor fusion on your Smart TV or mobile phone product. You’ve picked a set of high quality inertial sensors. You’d made sure to build your electronic circuit appropriately. And you’ve obtained a sophisticated sensor fusion library. Everything is great. Except that you forgot the simple step of calibration. Now your fused sensor data isn’t going to be so accurate. Kahn!!!

See the problem is that our inertial sensors aren’t perfect. We described this in an earlier post about the “seedy underbelly of motion sensing devices”. Our sensors have structural errors in their outputs that affect the accuracy of our readings. Error components include bias, scale and misalignments. Some of these errors are static and linear, but many are non-linear and vary depending on time (age of device) and temperature.

As an example, all gyroscopes produce an output known as Zero Rate Offset or ZRO. ZRO is the reported output voltage when the angular velocity is zero. Of course different products from different vendors will have their own unique ZRO values, but you’d hope that ZRO would be consistent across a particular product over time and temperature. Unfortunately, it’s not. The figure below shows different ZRO readings at various temperatures for several well-known industry gyroscopes.

So we need a process to account and adjust for these errors to have a more accurate measurement. This is where calibration comes to the rescue. Just like you would calibrate a scale to ensure that weight measurements are accurate, we need to do something similar with our sensors. And we’ll need both static and dynamic calibration mechanisms. Static calibration is done ahead of time and adjusts for parameters that have important variation between devices but smaller variation with respect to environmental conditions. Dynamic calibration is needed for parameters that exhibit a large variation over the lifetime of the device, such as ZRO.

At Hillcrest, we built a state of the art sensor qualification lab to explicitly measure the static and dynamic characteristics of our different sensors and also to measure their responsiveness to time and temperature effects. Our basic configuration includes a rotating 2-axis robotically controlled gimbal that allows precise movement of a device under test containing the sensors to different positions and with varying angular velocity. The gimbal system is located inside a temperature chamber that allows characterization over temperatures from 0 to 50C.   The chamber and gimbal are mounted on a vibration isolation table to ensure no interference from the external environment.

So when our customers ask us to qualify a new sensor, we put it through its paces. We get a large set of sensor samples from a vendor, place these on electronic circuit boards, and run them through our qualification lab. From this we generate a nominal (single) static calibration record for the particular sensor product.

However, we often are better served to generate a calibration record on a per-device basis as some of the sensor characteristics can vary significantly between individual devices. This then becomes an engineering tradeoff question since calibrating an individual device requires a longer manufacturing process. How many measurements do we need and how does this affect the accuracy of our final results? Consider these four options for calibrating an accelerometer on an electronic circuit board or embedded on a final product.

1. No calibration.
2. Simple 2 point calibration. Here we make two measurements of our sensor values, one with the board face up and one with the board face down.
3. More complex 2 point calibration. Again we make two measurements but we attempt to better vary the board’s position along its three possible rotations around roll, pitch, and yaw. So we would make a reading at (0°, 0°, 0°) and another at (45°, 45°, 45°) by using a gimbal.
4. N-point calibration. Here we actually make 32 measurements using our gimbal to get more precise readings.

Our figure below shows our error modeling four these 4 different approaches. With no calibration, we estimate our mean angular error to 1.93 degrees with max error of 5 degrees. A simple 2 point calibration improves our mean error by 15%. A more complex 2 point calibration improves our mean error by almost 50%. Finally, a 32-point calibration improves our mean error an order of magnitude – a 90% improvement.

The final choice depends on the use case for the device and what amount of error is tolerable. So choose wisely, but above all else, don’t forget to calibrate.

Tell me if you’ve heard this before. You’re meeting someone in the city but are running late. And you aren’t familiar with the streets and have somehow gotten turned around. So you pull out your handy dandy smart phone. Launch your favorite map application. Click on the compass indicator to show your heading. Then ruh roh. The dreaded compass interference message!

The Dreaded Compass Interference Message

Luckily you’re given the very helpful instructions to wave your phone in a figure 8 motion. Yes, of course, the magic figure 8. Why didn’t we think about that before? That should be on everyone’s problem solving list right after “uh, let me try that again” and right before “I think I’ll reboot and pretend it didn’t happen”.

Well, if you are like me, you probably wonder what’s going on. Maybe this is the design of some evil software programmer at Apple or Google that gets a kick out of seeing people idiotically wave their phone in the air. While that may still be true, the real problem is that your magnetometer hardware isn’t very smart. It does one thing really well which measures the earth’s magnetic field. Scratch that. It picks up all magnetic fields including other permanent magnets like the speaker in your phone. In the industry we call these hard iron effects – effects due to other magnetic sources.

And as you could probably guess – because we are quite clever – there are soft iron effects too. Soft iron effects are ones that warp the existing magnetic field but don’t generate magnetic fields themselves. Actually, the electronic circuits in your phone contribute to soft iron effects. So do other things in your environment made of iron, steel, and nickel.

The figure shown below illustrates the effects of hard and soft iron on our magnetometer readings. An ideal reading of the earth’s magnetic field forms a circle in 2 dimensions as we rotate the phone or a sphere in 3 dimensions. The radius of the circle is the magnetic field strength – which of course would vary depending where on earth we are located. Hard iron effects are additive magnetic sources which cause an offset in the field readings: a shifting of the ideal circle. Soft iron effects are more complicated. They cause a distortion in the field which turns our ideal circle or sphere into an ellipse or ellipsoid.

Magnetic Field Distortions from Hard and Soft Iron

In order to compensate for the interference and calibrate the magnetometer, we need to solve for the equation of the ellipsoid and find its scale, skew, and offset. And the best way to map out the ellipsoid is to move your phone in a figure eight pattern in the air so that the magnetometer gets good readings in different dimensions. Hence, the magic figure 8.

So the next time you get caught with the dreaded compass interference message and someone sees you waving your phone in the magic figure 8 pattern, channel your inner Sheldon. Tell them you are calibrating your phone’s magnetometer to remove hard and soft iron effects so you can solve the equation of an ellipsoid in 3D space.

So in our last 2 entries, you’ve experienced the highs and lows of our motion sensing devices. First we got you all excited when we covered the 4 most popular motion sensing devices. But then we brought you crashing down to earth while describing the seedy underbelly of our motion sensors – all their challenges and limitations. Yes, life is indeed bittersweet.

But today friends, we are ready to level up! So hold on tight as we learn the Wondrous Things™ we can do by combining our sensors together. Think of accelerometers as strawberries and gyroscopes as cream. Or chocolate and peanut butter. Or better yet, Gulab jamun and ice cream. Mmm Gulab jamun.

Ok, the real point is that when we combine the outputs of sensors together we can do much better than those sensors can alone. This is what people in the industry call sensor fusion. Combining accelerometers, magnetometers, gyros, and cameras together can create better performing and more intuitive applications. Let’s look at a couple of examples.

Air Mouse

Using a gyroscope embedded in a remote control, one can easily generate movements that can control the display of a cursor on a PC or TV screen. And the first air mice did just this by mapping rotational velocities into 2D cursor movements. Unfortunately, these solutions had significant usability problems.

To see why, imagine yourself standing directly in front of the TV about 5 feet away. You’ll hold the remote control completely parallel to the ground and make wrist movements up-down, left-right, zig-zag to control the cursor. It may work ok. But soon you’ll get tired and want to sit down. Or worse, your family will yell at you to get out of the way.

So you’ll lie down on your comfy sofa holding the remote control in your hand and resting on your torso. And at some point you’ll move your remote control up in the air to move the cursor up on the screen. But since the remote control is now held sideways the cursor moves left-right instead of up-down. This is annoying. Very seriously annoying. So annoying that you’ll contemplate throwing your remote control at the TV screen to make it stop. Trust us – you’ll soon want your money back.

So when we created our first air mouse, we combined the data from a 3D accelerometer and a 2D gyroscope to create a more responsive and usable in-air pointer. If you remember, the accelerometer measures the force of gravity which helps us determine the remote control’s orientation. So with this data, no matter how you hold the remote control (right side up, upside down, tilted any which way), if you moved your hand up vertically in the air, the cursor moves vertically in the air too. So now you can sit up, lie down, or stretch out on recliner with your legs up, beer in one hand and remote control in the other. The world is now your oyster.

This feature has a fancy name called orientation compensation. It means that the algorithm for generating the cursor location compensates for the device’s orientation before doing so. Of course there is more to creating a usable air mouse than just orientation compensation. For one, we’ll need to deal with tremor and other unintended motions that will make the cursor jumpy. But now we are getting ahead of ourselves as this is more fodder for a later post.

Racing Game

Let’s explore the use of motion data to control the movements of a car in a racing game. You are now Speed Racer and your car is the Mach 5 and you are taking on all contestants in your race of a lifetime with Chim-chim in the trunk. To control the car’s movements, we’ll embed a 3-axis accelerometer in a game controller or remote control and manipulate the controller like a steering wheel to make turns and avoid other car’s that have been outfitted with sharp knives.

The mental model for this is the controller as a steering wheel. You hold the steering wheel out in front of you and turn it left and right to turn the car. So we basically need to know when the controller is centered, i.e. not turning.  And when turned, we want to know how many degrees from center we moved so we can control the amount of turn.

Our accelerometer will work for this purpose but imposes a few challenges. First, if we hold the controller still, the raw accelerometer data will be noisy and we will have a hard time keeping our car on a straight path. Second, when we turn our remote control, we’ll have a hard time isolating the force of gravity from the linear acceleration of the movement until we settle the movement which means our turn won’t be smooth.  Third, this all works decent if you hold the controller straight in front of you, but if you put it down on a flat surface and turn it back and forth, you won’t be able to determine the orientation since the dimensions of turn aren’t affected by the forces of gravity. (This is similar to problem of changing the portrait and landscape mode on your phone while it’s resting on a flat surface.)

We can solve the first two problems by heavily filtering (i.e. removing) the linear acceleration data from our raw accelerometer data. So now we can control the steering wheel much more smoothly. But we’ve now traded off smooth and steady control for performance. Filtering the data will require us to delay the output so when we turn our remote control to the right, our car won’t start turning immediately but rather after some delay. Ouch! Our Mach 5 has just fallen off the edge of the cliff because we didn’t turn in time.  Poor Chim-chim.

A better approach would be to combine a gyroscope and accelerometer together to control the steering. We use the gyroscope data to control the turning since our gyro provides a fast and accurate measurement of angular velocity. Of course the gyro data alone won’t tell us the orientation of the remote control; it only tells us how fast it’s moving and in which direction. So we’ll use the accelerometer data to periodically calculate our orientation, correcting our gyroscopic positional estimate over time.

Now we have a responsive and smooth steering control for our Mach 5. We still have problem #3 – if we lay down our remote control on a table and twist it, it won’t work correctly. To solve this, we could add a magnetometer. Using both an accelerometer and magnetometer we can now determine our device’s orientation in all three dimensions. (Unless we are located at the earth’s magnetic north pole, but that’s a story for another day.)

So our high performing Mach 5 steering controller is now complete.  And we can get back to racing and finding out the identity of Racer X.  Or better yet, grab some Gulab jamun.

Up to now, life has been grand. We’ve discussed the most popular motion sensing devices and how these can be used to detect the 4 types of motion. Unfortunately, all our sensing devices have issues. Every single one of them. Sorry to burst your bubble, but we can’t hide the truth. This is what we call “the seedy underbelly” of our motion sensing devices. (Yes, we have a penchant for being dramatic. And we’ve been waiting a long time to use the words “seedy underbelly”.)

To start, all sensors have structural errors in their outputs which manifest themselves as a difference between the expected reading and the measured reading. These structural errors can be attributed to the sensor’s internal design or to deviations from an ideal manufacturing process. Yes, nothing in life is perfect.

Error components include bias, scale, and misalignments. Bias is a basically an additive term between the expected and measured reading, while scale is a multiplicative term. Misalignment errors relate to the interdependency of measurements for a multi-axis device. For example, a 3-axis accelerometer would ideally measure linear acceleration for the x, y, and z axes which are separated by exactly 90 degrees. If they are not exactly 90 degrees apart then the readings are not completely independent.

What makes these errors even more challenging is that they are not always linear and can change depending on temperature and time / sensor aging. To handle these errors we typically introduce a static and perhaps dynamic calibration process. This is a complex topic in its own right and so we’ll cover this in a future article.

Let’s now take a look at each of our individual sensors.

Accelerometer Issues

Accelerometers are great devices indeed. They are small, inexpensive and low power which is why they were the first of our sensors used in smart phones and game controllers. And they do one thing really well which is measure the force of gravity. Knowing the force of gravity tells us which way is down and we can “mostly” determine the device’s orientation. We say mostly because if we put our cell phone on a table we will know whether the device is face up or face down, but we’ll have no idea whether the device is facing towards us or away from us in portrait or landscape mode. This is why you need to tilt your phone in order to change the screen’s orientation. So really our 3-axis accelerometer can only give us 2 dimensions of tilt, roll and pitch, the only two axes affected by gravity.

And of course, if our device is moving, then all bets on gravity are off. On movement, an accelerometer measures the combined forces of gravity and the linear acceleration due to movement. It’s much more difficult to isolate the force of gravity and determine orientation. Actually we exaggerate. It’s not really more difficult because we can just heavily filter the output which is a fancy way of saying we can perform a weighted average of the sensor’s values over time which should remove the effects of linear acceleration. But this means we need to delay our final output which means we are now slow. This is why many of the original accelerometer-only tilt based cell phone games had low responsiveness.

And don’t even think about using the linear acceleration to determine absolute position. If you remember your calculus, we can integrate the acceleration over time to calculate linear velocity. Integrate being the fancy word for add. And if we do it twice – the infamous double integral – we can calculate linear position. Except that small errors in our tilt estimate will have enormous effects on linear position. For example a small error in our tilt estimate of roughly 1 degree can make our linear position estimate off by multiple meters per second! We would show you the math but we promised to keep this blog simple, so you’ll have to trust us on that one.

Magnetometer Issues

Magnetometers are also great devices: small, inexpensive, and low power. And they also do one thing really well which is measure the earth’s magnetic field to help us determine heading. However, they are also exceptional at picking up the magnetic field effects of components in your phone like the speaker and internal circuit board as well as external transient sources like large metallic structures in the environment. What this means is that a magnetometer is a great device for determining heading except when it doesn’t work at all. We’ll eventually need to use some advanced software to compensate for these effects.

The other challenge is that in order to effectively calculate heading we need to compensate for tilt which means we need to know what the flat level plane is. The mag won’t give us that information. We’ll eventually need to fuse information from the accelerometer to do this. But we are getting ahead of ourselves.

Gyroscope Issues

Unlike mags and accelerometers, gyros tend to cost more and use more power which isn’t good. But they are very responsive devices providing a fast and accurate measurement of angular or rotational velocity. For this reason they tend to be the workhorses of motion tracking. And if we integrate their output over time, we can determine angular position. Which is great, except that we don’t know our starting angular position based on our gyro alone since it has no frame of reference like gravity or the earth’s magnetic field.

The other way of looking at the gyro is that it is great at providing short term information but not so great at long term information. This is the exact opposite of our accelerometers and magnetometers which produce great long term stable information from gravity and the earth’s magnetic field, but less usable short term information.

If we integrate our gyro output to determine angular position, over time we’ll encounter drift which is an alignment problem between our measured position and the real position. So to do this well, we’ll periodically need to introduce correction terms from our accelerometer and magnetometer readings. The good news is that we can do Wondrous Things™ if we combine our sensors together which is our next topic.

So far we’ve examined the four types of motion and motion’s 6 degrees of freedom by considering your life as an airline pilot. This week we’ll discuss how to detect linear and angular motion using the most popular motion sensing devices in consumer electronics products: accelerometers, gyroscopes, magnetometers and cameras.

As a refresher refer to our figure of the mobile phone and the 6 degrees of freedom we detect – linear x, y and z as well as roll, pitch and yaw. And if you look really carefully, you’ll notice that we flipped the X and Y axes on you from last week since the mobile people decided they needed to be different from the navigation people.  (Sorry, your time as a jet setter was short lived. Now back to reality.)

Motion's Six Degrees of Freedom Mapped On To Smart Phone

Accelerometers

The Wii and the iPhone popularized the use of motion sensing accelerometers and allowed people to play natural motion games like tennis and tilt-based games such as Doodle Jump. Accelerometers also played an important function by supporting the landscape / portrait rotation on your phone.

Accelerometers are devices that measure force along an axis. Our figure below shows the basics of an accelerometer: a mass is suspended by springs and free to move about an axis. The sensor measures displacement or distance of the mass relative to center. The displacement enables the sensor to determine the linear acceleration of the mass in that direction. Today’s accelerometers typically measure the various forces along our three linear axes and not just the two shown below.

Accelerometer - Mass on a Spring

When the device containing the accelerometer is completely still, the mass on the spring will be offset by the force of gravity pulling it down to earth. So accelerometers are great at measuring the force of gravity – at least when completely still. And knowing this, we can derive the tilt of an object, which is its roll and pitch. Unfortunately yaw isn’t affected by gravity and can’t be directly measured with an accelerometer.

Gyroscopes

So accelerometers are great; they can measure the linear acceleration of an object including gravity, but they can’t measure twisting or rotating motions very well. This is where a gyroscope is really handy. A gyro is an electronic sensor that determines angular velocity around our three axes of roll, pitch and yaw. The physical properties of a gyroscope are more complex to explain than an accelerometer, so if you want to learn more we suggest you look here.

Gyroscope

With our gyroscope we can also determine the device’s orientation in 3D space. Knowing our angular velocity, we can compute our angular position over time. At least we could if we knew the starting point. Which a gyro doesn’t provide since it has no frame of reference like gravity. So we’ll probably need to combine our gyro and accelerometer data together to do Interesting Things™ – like Nintendo did with their Wii Motion Plus accessory.

Magnetometers

One thing that our accelerometers and gyros can’t do by themselves is determine absolute heading to answer questions like “Are we flying towards London England or London Ontario?” Or more pragmatically, “Which way am I facing when looking at the map on my smart phone?”

A magnetometer on the other hand can give us direction. A mag – the cool name for magnetometer – is a device that measures the strength or direction of a magnetic field. Sort of like a compass. Or exactly like a compass. So with a mag we know which way is north.

Hall Effect Magnetometer

There are various types of magnetic sensors and our figure shows the basics of a simple Hall effect sensor. Electrons passing through a perpendicular magnetic field are affected by the this field – they don’t follow a straight line path and instead are pulled towards a side. From this we can determine the direction of the field. These sensors also generate a voltage that is proportional to the strength of the magnetic field.

Cameras

Finally, we have optical and image sensors that can be used to measure distance and direction from an object. So let’s say we mount a camera in a TV set and look at a person’s hand (like Microsoft’s Kinect) or at a gaming remote control (like the Wii Remote). The camera intrinsically knows its field of view and based on the size and location of the object it captures, can determine both the angular direction and distance from the TV.

Camera Basics

Putting the Pieces Together

So we’ve briefly talked about the four most popular motion sensing devices. Our accelerometers, gyroscopes, and magnetometers are inertial sensing devices meaning they measure motion data from the point of view of the object in motion. Cameras are non-inertial and they compute motion from an external reference point such as the TV set or personal computer. Sometimes inertial and non-inertial sensors are needed depending on the use case and which motion parameters are really important. We’ll cover the All Important Use Cases™ in a future blog post. And we’ll also talk about how these devices can be used together to solve interesting problems.

Last week we examined the four types of motion: natural motion, pointing, gestures and virtual controls. This week we’ll discuss the elements of motion; those parameters we measure and track to develop our motion-enabled applications like a game console golf game or augmented reality mobile app.

So let’s start by imagining yourself as a private airplane pilot. Just roll with it for a minute. There’s nothing you love more than taking to the sky and being one with the clouds. Got it?

Ok, now you plan a trip. You’re a present day Charles Lindbergh and are jet setting across the Atlantic from New York to London to see the Olympics. (Yes we know that Lindbergh actually flew to Paris. Try to focus.)

As we fly, we’ll track our plane’s linear position along the path from New York to London. That position can be described in three dimensions on x, y and z axes. The x-axis tracks our path along the “straight line” from New York to London. The y-axis tracks a perpendicular or sideways movement from the New York to London path. Maybe there is a storm we are trying to avoid so we don’t take a direct route. In this case the y-axis represents our offset from the straight line path. Finally, our z-axis is our elevation in the air which hopefully changes as we take off and land.

So at any point during our flight we can describe our location with three points in a 3D space. Not only can we track our position, but also our velocity and acceleration – well since planes move and this is about making motion. Velocity is a measure of how fast we are moving in our three different axes. So if we are flying at constant height and directly on a straight line path, our velocity is completely along the x-axis and essentially zero along the y or z axes. Acceleration is the change in velocity and is a measure of whether we are speeding up or slowing down. Which we’ll probably need to do when we land.

Great, so now we have our linear position, velocity and acceleration. Are we done? Heck no. Our linear data is only 3 degrees of freedom (x, y, z axes) and we need 6. So where do we get the other three?

Motion's 6 Degrees of Freedom

The other motion we haven’t yet considered is angular or rotational motion. This will be useful to describe our plane’s full orientation. Roll is a rotational movement around the x-axis and enables us to bank our plane in and out of turns. Or show off to our friends and family.

There is also pitch which is a rotation along the y-axis and enables us to point the nose of our plane up and down as we take off and land. Finally, yaw is a rotation around the z-axis and would describe our movement if we decided to spin our plane around in a circle like a top – which we probably want to avoid if we want to arrive in London in one piece.

Our angular position is described in terms of the angle of rotation about our three linear axes. And along with position, we can also measure our angular velocity and angular acceleration which can be used to predict our future orientation.

So now we can describe our position and motion with 6 degrees of freedom: 3 linear and 3 angular. Using these we can plot both the position and precise orientation of our jet as we travel to London. And watch the U.S. men’s national soccer team take on the world’s best. Or maybe not.

Next week, we’ll look at the 4 most popular motion sensing devices and how these can be used to detect and track motion.

So what is motion and how should we characterize it? For the purposes of this blog, we refer to motion in the context of applications on our personal entertainment devices. These are ones you use on your TV, game console, smart phone, and PC. Of course, motion also extends into health, fitness and industrial products and we may spend a little time on these too.

Before we define motion, let’s dispel one myth which is that motion means gestures. Today, gestures are all the rage; you can’t read articles about the future of UIs without the seeing the word gesture. And many consumer electronics marketers now use the word gesture to broadly describe all types of motions. As if humans were just one big gesture making machine.

Nonsense! Sure, waving hello with your hand is a gesture. But swinging a baseball bat is not a gesture nor is making a menu selection to rent a movie. (By the way, we place sole blame on Tom Cruise for this problem.)

So how should we categorize motion? From our perspective there are really four categories of motion: (1) natural motion, (2) pointing, (3) gestures, and (4) virtual controls.

The broadest of these categories is what we call natural motion. When we walk into a room or jump in the air we are invoking natural motions; these motions are inherent in the human structure and have been learned in conjunction with everyday activities. Swinging a baseball bat or golf club is also natural motion. (Although don’t look at my golf swing; it’s hardly natural. But that’s my problem.)

The other characteristic that differentiates natural motion from the others is the application and motion tracking system must be sensitive enough to distinguish different performance levels. So if we are designing a golf game, there should be some performance difference in how Tiger Woods or Rory McElroy swing a golf club compared to how I swing one so that the game is meaningful. So with natural motion, we need to detect and highlight the differences while with the other motions, we need to tolerate and smooth the differences.

A second type of motion is pointing. Now pointing is also a natural motion but it has very specific characteristics which is why we treat it separately. People learn to point before they learn to speak and infants and toddlers use pointing as a fundamental means of communicating very early in life. If you don’t believe us, ask a 1 year old child “Where’s Mommy?” or “Where’s Daddy?” and see what they do. (Just make sure you know the child or Mommy and Daddy might get upset.)

It’s also not a coincidence that we call the cursor on your PC a “pointer”. Most computer interfaces today are pointing-based, whether via a mouse or through touch control. One of the most important characteristics of pointing and cursor control is that it requires a high level of performance or it leads to frustration. Pointing must have very low latency and a high degree of accuracy to be effective. Imagine if typing in text on the touchscreen of your Smart Phone wasn’t very accurate and how frustrated you would feel. Ok, that was a bad example. No one likes typing text on their smart phone for exactly this reason!

The third category of motion is gestures. A gesture is a shortcut; in essence it is a verb used to drive a specific action in a user interface. A good example is the “pinch” to zoom operation on your phone or tablet. Gestures are like sign language and it has been shown in academia that the number of gestures or shortcuts should be limited to something close what our short term memory can retain. The iPhone, for example, really only requires 7 gestures – four directional swipe operations to scroll content as well as pinch in, pinch out, and double-tap  to control the zoom level. That’s it. Enough said!

The final category of motion is virtual controls. Virtual controls are a hybrid type of motion that is both in one sense natural and in another sense gesture-like, but again has unique characteristics. Traditionally, hardware electronics products have been controlled with knobs, dials, and sliders. We can enable more functional skeuomorphic interfaces by tracking motions corresponding to virtual controls. The easiest way to describe this is imagining how a scroll wheel can be used to adjust a volume slider up and down. Now instead of using a scroll wheel, use an air mouse and move your wrist up and down to move the slider. For this to be effective, the control needs to smoothly track motion and enable a one-to-one tracking of your hand and the UI element on the screen.

So to summarize, we distinguish four types of motions:

1. Natural motion: Must detect individual performance levels to be effective
2. Pointing: Very high degree of performance required or leads to frustration
3. Gestures: Simple small number of control mechanisms or shortcuts
4. Virtual controls: Enables skeuomorphic interfaces and must track smoothly

And the next time you hear someone confuse the terms motion and gesture think to yourself “Not so fast my friend”. Motion is more than just gestures. And motion sensing software needs to be sophisticated enough to distinguish the differences. Or I’ll be joining the PGA tour.

Have you ever played a tilt game like Doodle Jump on your smart phone? Or boxed someone on the Wii? Or danced with your friends using your Xbox Kinect? Well, unless your name is Wilson and you’ve been stuck on an island for the past 5 years, you’ve probably been exposed to motion sensing and tracking technology.

Of course motion sensing systems are not new. They have been used for decades in industrial, military and aerospace applications. But recent advances in fabrication technology, processing performance and software algorithms have made these systems available at low cost and for mass market use. So now you can put sensors in your favorite pair of sneakers to track your running performance. Or wave a remote control in the air to move a cursor on a television set. Or even know when your significant other has left the toilet lid up.

The use of motion in natural user interfaces, health, sports, games, and entertainment is becoming ubiquitous.   Motion works because it is intuitive and natural.  People use motion everyday whether to wave hello to a friend, point at something interesting or just plain walk and chew gum.  And so if you’ve ever wondered about the technology behind motion, you’ve come to the right place.

Making Motion is a new blog from Hillcrest Labs, where we aim to make sense of motion. Our goal is to demystify motion sensing technologies and help you understand how to build valuable and usable products that incorporate motion.  And we don’t expect you to be an applied physicist to read this blog either.  Mentions of Euler’s Theorem and Discrete Cosine Transforms are off limits – although you may see the occasional stray reference to a quaternion or two.

So literally Making Motion is “Everything you wanted to know about motion but were afraid to ask.”  Speaking of which, don’t be afraid to. If there are topics you’d like to see covered or a question you’ve always wanted answered, please follow up in the comments or reach out to us on twitter @makingmotion.

Enjoy!