The sensitive side of Android

Ankur Kotwal, Tim Bray, Tony Chan

Android Developer Advocates
View this session at
https://developers.google.com/events/io/sessions/gooio2012/108/
Demo
Sensitive Blackjack
Dealer vs Player
Aim is reach 21
- Ace is 1 or 11
- King, Queen, Jack are 10
Touch
Touch events
5
public boolean onTouchEvent(MotionEvent event)
Easy? Yeah, right.
GestureDetector VelocityTracker
Gesture builder sample
Recipe for multi-finger pinch (1)
6
public boolean onTouchEvent(MotionEvent event) {
if (event.getAction() != MotionEvent.ACTION_CANCEL
&& event.getAction() != MotionEvent.ACTION_UP) {
// Capture initial touch. Changed counts are new initial
if (initialTouches == null || initialTouches.size() !=
event.getPointerCount()) {
for (int i = 0; i < event.getPointerCount(); i++) {
// Capture each touch and calculate maximum distance
initialDistance = ...
}
} else {
// Capture new touches and calculate distance.
for (int i = 0; i < event.getpointerCount(); i++) {
currentDistance = ...
}
}
Recipe for multi-finger pinch (2)
7
} else {
// Cancelled motion or last finger up. Clear state.
initialTouches.clear();
currentTouches.clear();
}
if (callback != null) {
static final double DISTANCE_THRESHOLD = 0.3f;
static final int MIN_POINTERS = 3;
if (currentDistance < initialDistance * DISTANCE_THRESHOLD &&
initialTouches.size() >= MIN_POINTERS) {
callback.onMultiPinchShrink();
} else if ((initialDistance * DISTANCE_THRESHOLD <
currentDistance * DISTANCE_THRESHOLD) &&
(currentTouches.size() >= MIN_POINTERS)) {
callback.onMultiPinchGrow();
}
Gotcha - close pointers can merge!
Telepathy
Tailor your rate
Unregister aggressively
Best practices for using Sensors
9
sensorManager.registerListener(listener,
sensorManager.getDefaultSensor(Sensor.TYPE_LIGHT),
SensorManager.SENSOR_DELAY_UI);
sensorManager.unregisterListener(this);
Proximity sensor gestures
Found on phones to turn o! screen
Continuous or binary values
- Safest to assume binary results
- Sensor.getMaximumRange()
10
Light sensor gestures (1)
Lumens per sq. meter
- Continuous data
Typically used for adjusting brightness
11
Illuminance (lux) Light source
0.27 Full moon on a clear night
50 Family living room
320-500 O!ce lighting
32,000 - 130,000 Direct sunlight
Light sensor gestures (2)
12
Light sensor gestures (3)
13
Recipe for wave gesture
14
private static final int DATA_SIZE = 100;
private static long TIME_THRESHOLD = 500000000L; // 500 ms
private static float GESTURE_THRESHOLD = 0.2f;
public void onSensorChanged(SensorEvent event) {
mData[mIndex] = (int) event.values[0];
if (++mIndex >= mData.length) {
mIndex = 0;
mDataFull = true;
}
// Calculate max light but only after array is full.
if(mDataFull == true && event.timestamp - mTimeOfCalc > TIME_THRESHOLD)
for(int point : mData)
if(point > mCurrentMaxLight)
// Found new maxLight
if(event.values[0] < mCurrentMaxLight * GESTURE_THRESHOLD)
// Wave gesture triggered
}
Kinetics
Idea: Controls-free/clean-screen apps
16
Tap!
Shake!
Chop!
Wave!
17
Kinetics-related Sensors
Accelerometer
Gravity
Gyroscope
Linear Acceleration
Magnetic "eld
Orientation
Rotation Vector
18
since API 3, includes gravity, very common
since API 3, less common
since API 3, A.K.A. compass
deprecated, unreliable, dont use it
since API 9, synthetic
since API 9, synthetic
since API 9, synthetic, Accelerometer - Gravity
Synthetic sensors got a lot better in ICS... if theres a gyroscope.
19
Galaxy Nexus
(with gyroscope)
Nexus One
(no gyroscope)
The gyroscope
matters
Co-ordinate Systems
20
Device World
Watch out for rotations!
21
Power consumption
Sampling rate
Static and random
variation
Accelerometer data
(unless youre in orbit)
More gotchas
22
23
Detecting flip-up: Four chops
Detecting Flip-up (1)
24
private class Listener implements SensorEventListener {
@Override
public void onSensorChanged(SensorEvent event) {
switch (event.sensor.getType()) {
case Sensor.TYPE_ROTATION_VECTOR:
handleRotation(event);
break;
case Sensor.TYPE_GRAVITY:
handleGravity(event);
break;
}
}
}
Detecting Flip-up (2)
25
private void handleRotation(SensorEvent event) {
final float[] last = mFlipper.last(), next = mFlipper.next();
mFlipper.flip();
SensorManager.getRotationMatrixFromVector(next, event.values);
if (last == null) {
return;
}
SensorManager.getAngleChange(mValues, next, last);
final float deltaX = mValues[1]; // [z,x,y]
final boolean plusX = (deltaX > THRESHOLD_ANGLE);
final boolean minusX = (deltaX < -THRESHOLD_ANGLE);
. . .
Detecting Flip-up (3)
26
switch (mState) {
case STATE_AT_REST:
// ignore first few events when arriving at rest;
// stabilizing after a gesture
if (mStateDuration < THRESHOLD_REST_DURATION) {
mStateDuration++;
} else if (minusX) {
mState = STATE_MOVING_FORWARD;
mStateDuration = 1;
} else if (plusX) {
mState = STATE_MOVING_BACKWARD;
mStateDuration = 1;
}
break;
. . .
Detecting Flip-up (4)
27
private void send(int kinetic) {
stop();
if (!mCustomer.kineticRecognized(kinetic)) {
start();
}
}
case STATE_MOVING_FORWARD:
if (minusX) {
mStateDuration++;
} else {
if (mStateDuration > THRESHOLD_MOVING_DURATION) {
send(FLIP_UP);
}
mState = STATE_AT_REST;
mStateDuration = 1;
}
break;
Which way is up?
28
private void onSensorChanged(SensorEvent e) {
// avoid NaN's provoked by wonky sensor readings
double gy = e.values[1] / SensorManager.GRAVITY_EARTH;
if (gy > 1)
gy = 1;
// correct the range so it goes smoothly from 0 to 180
// degrees. The X value tells you, roughly speaking,
// whether youre leaning left or right. The Y value
// goes from 0 at the top to +90 whichever way you lean
// the device
double theta = Math.acos(gy);
if (e.values[0] < 0)
theta = (Math.PI/2) - theta;
else
theta = (Math.PI/2) + theta;
setTilt(mAverage.add(theta));
Recipe for Tilt-tracking
29
Tilting a Mostly-flat device
1.Collect ACCELEROMETER triples.
2.If the device is held #at and tilted, the X
and Y values can be used directly as
acceleration numbers.
3.These can be used to describe Verlet
integration, see
lonesock.net/article/verlet.html
4.Useful for games! See example at
AccelerometerPlayActivity.java in the
SDK samples.
30
Light Saber!?!
1.Collect LINEAR_ACCELERATION triples, but...
2.Unfortunately, the acceleration values are
insu$ciently accurate and steady to enable the
use of Verlet, but...
3.You can usefully measure total acceleration,
sqrt(aX
2
+ aY
2
+

aZ
2
), to detect shakes and so on,
but...
4.Itll never rival a Wii or Kinect using
current typical mobile-device sensors.
31
private float[] accel;
private float[] mag;
public void onSensorChanged(SensorEvent event) {
if (event.accuracy == SensorManager.SENSOR_STATUS_UNRELIABLE) {
return;
}
switch (event.sensor.getType()) {
case Sensor.TYPE_ACCELEROMETER:
accel = event.values.clone();
break;
case Sensor.TYPE_MAGNETIC_FIELD:
mag = event.values.clone();
break;
}
Recipe for a Compass (1)
32
if (accel != null && mag != null) {
float[] r = new float[16];
float[] outR = new float[16];
float[] euler = new float[3];
if (true == SensorManager.getRotationMatrix(r, null, accel, mag)) {
// Correct for orientation
SensorManager.remapCoordinateSystem(r, SensorManager.AXIS_X,
SensorManager.AXIS_Z, outR);
SensorManager.getOrientation(outR, euler);
// Convert to degrees
compassValue = euler[0] * 180 / ((float) Math.PI);
}
}
Recipe for a Compass (2)
33
Audio
DSP
Demo
Heartbeat Monitor
Heartbeat Monitor
36
Sampling
Filtering
Down
Sampling
Heartbeat Monitor App
Headphone Jack
Microphone
MediaRecorder - API 1, Android 1+
- Capturing and encoding a variety of common audio formats
- Use for regular audio recording
AudioRecord - API 3, Android 1.5+
- If you need to process the audio signals
37
Sound Fundamentals
Human hearing range: 20Hz - 20kHz
Sound waves - Pitch and loudness
38
Sound Wave - superposition of multiple sinusoidal signals
Sound Fundamentals
39
+
All Android compatible devices support 44.1kHz sampling rate
To avoid aliasing, sampling rate must be > 2 * highest frequency component
of your signal
-1 0 1 2 3 4 5 6 7 8
Sampling
40
fOriginal = 0.9Hz
Original
All Android compatible devices support 44.1kHz sampling rate
To avoid aliasing, sampling rate must be > 2 * highest frequency component
of your signal
-1 0 1 2 3 4 5 6 7 8 -1 0 1 2 3 4 5 6 7 8
fSampling = 1Hz
Sampling
40
fAlias = 0.1Hz fOriginal = 0.9Hz
Alias Original
All Android compatible devices support 44.1kHz sampling rate
To avoid aliasing, sampling rate must be > 2 * highest frequency component
of your signal
-1 0 1 2 3 4 5 6 7 8 -1 0 1 2 3 4 5 6 7 8
Sampling
40
fAlias = 0.1Hz fOriginal = 0.9Hz
Alias Original
fSampling = 2Hz
All Android compatible devices support 44.1kHz sampling rate
To avoid aliasing, sampling rate must be > 2 * highest frequency component
of your signal
-1 0 1 2 3 4 5 6 7 8
Sampling
40
-1 0 1 2 3 4 5 6 7 8
fAlias = 0.1Hz fOriginal = 0.9Hz
Alias Original
fSampling = 2Hz
Buffering
Choice of bu!er size
- Responsiveness
- Memory/CPU cycles
- Tolerance to failures
Common pitfalls
- Always use getMinBufferSize(int sampleRateInHz, ...)
- For ENCODING_PCM_8BIT, use read(byte[] audioData, ...)
- For ENCODING_PCM_16BIT, use read(short[] audioData, ...)
41
Filter noises
Spectral analysis
Hard to do it in time domain but easy in frequency domain
Audio Signal Processing
42
Time
Time Domain
Hz 0 2 4 6 8
Frequency Domain
Computing Intensive
In practice: Fast Fourier Transform (FFT)
Things to look out for
- Garbage Collection
- In-place algorithm
- Native code
Discrete Fourier Transform (DFT)
43
double[] real = new double[buffer.length];
double[] imaginary = new double[buffer.length];
Complex[] complex = new Complex[buffer.length];
- Garbage Collection
Computing Intensive
In practice: Fast Fourier Transform (FFT)
Things to look out for
- Garbage Collection
- In-place algorithm
- Native code
Discrete Fourier Transform (DFT)
43
FFT.fft(real, imaginary);
lowPassFilter(real, imaginary);
FFT.ifft(real, imaginary);
- In place algorithm
Computing Intensive
In practice: Fast Fourier Transform (FFT)
Things to look out for
- Garbage Collection
- In-place algorithm
- Native code
Discrete Fourier Transform (DFT)
43
FFT.fft(real, imaginary);
lowPassFilter(real, imaginary);
FFT.ifft(real, imaginary);
Consider using NDK
- Native code
Thresholds
Time
Statistics
- Mean, Median, Mode, Range, etc
- Moving/Weighted Average
if (magnitude >= HEARTBEAT_AMPLITUDE_THRESHOLD) { // Detect heartbeat peaks
if (diff < HEARTBEAT_FREQUENCY_THRESHOLD) return; // Filter peaks too close
if (count > HEARTBEAT_REPEAT_THRESHOLD) { // Set new heartbeat rate
// if it repeats X times
Making Sense out of your Sensor Data
44
Thresholds
Thresholds
Time
Statistics
- Mean, Median, Mode, Range, etc
- Moving/Weighted Average
if (magnitude >= HEARTBEAT_AMPLITUDE_THRESHOLD) { // Detect heartbeat peaks
if (diff < HEARTBEAT_FREQUENCY_THRESHOLD) return; // Filter peaks too close
if (count > HEARTBEAT_REPEAT_THRESHOLD) { // Set new heartbeat rate
// if it repeats X times
// Calculate heartbeat rate
long now = System.currentTimeMillis();
long diff = now - mLastHeartBeatTime;
int heartbeat = (int) ((1000.0 / (now - mLastHeartBeatTime)) * 60)
Making Sense out of your Sensor Data
44
Time
Thresholds
Time
Statistics
- Mean, Median, Mode, Range, etc
- Moving/Weighted Average
if (magnitude >= HEARTBEAT_AMPLITUDE_THRESHOLD) { // Detect heartbeat peaks
if (diff < HEARTBEAT_FREQUENCY_THRESHOLD) return; // Filter peaks too close
if (count > HEARTBEAT_REPEAT_THRESHOLD) { // Set new heartbeat rate
// if it repeats X times
// Calculate heartbeat rate
long now = System.currentTimeMillis();
long diff = now - mLastHeartBeatTime;
int heartbeat = (int) ((1000.0 / (now - mLastHeartBeatTime)) * 60)
Making Sense out of your Sensor Data
44
Statistics are your friend
Statistics
- Mean, Median, Mode, Range, etc
- Weighted/Moving Average
Visualization
android.media.audiofx - Visualizer
Custom visualizer
- Down-sample data for display
- Use a circular bu!er to store the display data to achieve scrolling e!ect
45
Thank You!
http://developer.android.com/training
+Android Developers on Google+
+Tim Bray
+Tony Chan
+Ankur Kotwal
View this session at
https://developers.google.com/events/io/sessions/gooio2012/108/
!"#"$%&"'(