## Sensor Fusion on Android Devices: A Revolution in Motion Processing

Google Tech Talk

August 2, 2010

ABSTRACT

Presented by David Sachs.

Gyroscopes, accelerometers, and compasses are increasingly prevalent in mainstream consumer electronics. Applications of these sensors include user interface, augmented reality, gaming, image stabilization, and navigation. This talk will demonstrate how all three sensor types work separately and in conjunction on a modified Android handset running a modified sensor API, then explain how algorithms are used to enable a multitude of applications.

Application developers who wish to make sense of rotational motion must master Euler angles, rotation matrices, and quaternions. Under the hood, sensor fusion algorithms must be used in order to create responsive, accurate, and low noise descriptions of motion. Reducing sensing errors involves compensating for temperature changes, magnetic disturbances, and sharp accelerations. Some of these algorithms must run at a very high rate and with very precise timing, which makes them difficult to implement within low-power real-time operating systems. Within Android specifically, this involves modifying the sensor manager, introducing new APIs, and partitioning motion processing tasks.

David Sachs began developing motion processing systems as a graduate student at the MIT Media Lab. His research there led him to InvenSense, where he continues this work with MEMS inertial sensors used in products such as the Nintendo Wii Motion Plus. David's designs incorporate gyroscopes, accelerometers, and compasses in various combinations and contexts including handset user interfaces, image stabilizers, navigation systems, game controllers, novel Braille displays, and musical instruments.

Regarding the gravity signal, he says "it's actually mostly gyroscope data, but with the accelerometer used to correct drift" (at 17:59). I can't find any information on how this is done.

The W3C Motion Sensors Explainer hints at the same thing: "On most sensor hubs, gravity is isolated from the accelerometer using the gyroscope" (https://www.w3.org/TR/motion-sensors/#relative-orientation), but also lacks any detail.Even 8 years later, it seems to be the most comprehensive talk about sensors and sensor fusion I have ever watched. Thank you!

Good presentation, explains the things behind!

Although now in the latest APIs there are composite sensors like

- ROTATION VECTOR (accelerometer + magnetometer + gyroscope)

- GAME ROTATION VECTOR (accelerometer + gyroscope)

- GEOMAGNETIC ROTATION VECTOR (accelerometer + magnetometer)

which are using SensorFusion technique by default and their output can be used in methods like "GetRotationMatrixFromVector", "GetQuaternionFromVector".

See also:

https://source.android.com/devices/sensors/sensor-types#attitude_composite_sensorsvery good video, helps me so much writing my master thesis. additionally he has so a clear pronunciation

sorry, I know this is inappropriate, but can I see your thesis?

Check out Ken Wheeler "Missing Secrets of Magnetism" 3rd edition.

what is the "inegral" ??

Anyone want to give definitions for all the terminologies used here?

Interesting to compare the real world design vs the point of view of abstract University teacher who wrote the wikipedia pages about what should be the same subject.

How the GPS work so well in any cell phone? The real data from the satellite is not precise enough to detect that you arrive at a street in 50 feet. But try to discover how the cell phone work from any web site or youtube video..

Nobody knows. Wikipedia just mention the expensive equipment that are mandatory in commercial airplanes. They know the cell phone in their pocket appear to match the performance of these devices. But all they can say is : thanks to Einstein, science works. The black hole in their knowledge is discretely ducted under the rug.

This video published in August 2010 explain the difficulty of sensing linear displacement from the 6 sensors. A good engineer can fill the gaps, extrapolate what the device do starting from the noisy input sensors to the smooth animation that Google map and other similar application draw of the cell phone.I saw this video again after three months and theres still so much to learn about this. This is so far the best resource on Motion Sensors and sensor fusion I have come across.

what is the data format of 9-axis combination data?

5:00 street view is run by the same technology pioneered by the video game Mist.

Even 6 years later, it seems to be the most comprehensive talk about sensors and sensor fusion I have ever watched. Thank you!

Even almost 9 years later...

Indeed ...

Even almost 8 years later, I totally agree :)

Thank you!

Can someone explain how the pan change worked at 38:00 assuming I have got the rotationMatrix from the sensor library, how to I pan the 2D image on my mobile screen?

What we ripped out was the headphone jack. Apple 7 copy pasta.

I was looking for this ^^

What we ripped out was the headphone jack. Apple 7 copy pasta.

38:30 "Yes, it's pronounced "Oiler". I'm not sure why."

MAYBE BECAUSE HE WAS SWISS?? DURRR... Goddamn american dumbasses... Holy shit, that was retarded.aa ok, it's a Swiss name. Then when it's translated to English it should be spelled in English.

An engineer at MIT doesn't speak German, I'm triggered.

26:44 I doesnt get it ...why is use Kalman filter, and what was about pedometer. Someone can me explain it ,please?

it's for the positioning of the character..for example if you want to

move your character by moving your device or want to map your footsteps

as input to your character's step... he was actually giving alternate to

the Accelerometer (to avoid double integer errors) and proposed

pedometer and Kalman filter to use for the above mention task.yet still the nexus 6p has tons of problems with its sensors. some gyro axis are not on zero when static, gravity value of the linear acceleration virtual sensor is off and there is no way to calibrate any of them...

It was incredible. This video help me a lot to approach my thesis based on staft related with acc. Thanks. GreatJob

The best explanation of how to interpret quaternions I've seen. Thank you!

42:17 Hi im little bit dumb. How does he get 9 numbers from 3 axis points? How does the twist work to give 6 extra numbers? "Those axis twist around with the teapot..." i dont get it.

+saxman58b

Thank you very very much for explaining! :)+TheiLame I believe what he meant was that each of the three rows of the 3x3 matrix are the vectors describing the new x, y and z axis relative to the original xyz dimensions. For example, if you wanted to just rotate around the z axis by 90 degrees then the matrix to convert old dimensions to new would be:

[0 -1 0 (vector describing the old x axis relative to the new coordinate system after rotation, i.e. x' = -y)

1 0 0 (vector describing the old y axis relative to the new coordinate system after rotation, i.e. y' = x)

0 0 1] (Z axis doesn't change so all vectors keep their same z coordinate)

To test it, pick an arbitrary position in space, for example (1, 2, 3). If you multiply the matrix above by this vector you get a resultant vector of (-2, 1, 3), which is exactly what was intended, a 90 degree rotation around the z axis.

This was a simple example, but it should work for any 3D rotation. Just make each row of the rotation matrix equal the vector describing the new direction of the three local x, y, z axis of the object. And unit vectors are required in the matrix, or else the space / object will expand or contract. Thus a 45 degree rotation version of the above would be:

[sqrt(1/2) -sqrt(1/2) 0

sqrt(1/2) sqrt(1/2) 0

0 0 1]This is 5 years old video, is it still up to date in terms of what they use? :) anyone knows?

This is 5 years old video, is it still up to date? :) anyone knows?

+TheiLame The math and physics all still makes sense. What has changed is that most phones now have all of this hardware built in, and on the iPhone at least (I only do IOS development) the software functions are now built into the standard libraries. So no need to hack the OS or add hardware to get the gravity vector, user acceleration, attitude (via Euler, quaternion and rotation angles) etc. It's almost all there and ready to use with little or no extra processing.

22 people love Jesus

They are 36 now.

youtube(google) should integrate IQ and Knowledge into thumbs-up/down. because that way when you see the ratio you'll have a better chance of correctly evaluate yourself!

Cool video. See our Smartphone Garage Door Opener with webcam ⛔ #iHued-Garage ⛔ P.S. I Liked your video hope you can do the same, thanks ;-)

Euler was Swiss, this is why his name is pronouced "Oiler".

This is how we could do position in VSN. Thank you!

This video helped in my project. Omg thanks!!

## Comments