Machine Learning Models on Mobile Devices
1 like307 views
The document discusses the advantages of implementing machine learning models on mobile devices, highlighting privacy, availability, and speed. It outlines various models compatible with CoreML and TensorFlow Lite, including Inception V3 for object detection and XOR examples for logic operations. Additionally, the document explains the process of model conversion and optimization techniques, such as quantization, for efficient deployment on mobile platforms.
![Inception V3 and CoreML
func resHandler(request: VNRequest, _: Error?) {
let results = request.results
as! [VNClassificationObservation]
let percent = Int(results[0].confidence * 100)
let identifier = results[0].identifier
resultLabel.text = "(percent)% (identifier)"
}
resHandler
results
results
results](https://image.slidesharecdn.com/m3machinelearningmodelsonmobiledeviceslarsgregori-181015162738/85/Machine-Learning-Models-on-Mobile-Devices-17-320.jpg)
![Inception V3 and CoreML
@IBAction func predict(_ sender: UIButton) {
selectedImage.image = sender.currentImage
let image = sender.currentImage!.cgImage!
let handler = VNImageRequestHandler(cgImage: image,
options: [:])
try? handler.perform(self.requests)
}
@IBAction
sender.currentImage!.cgImage!
handler.preform
handler
image =
image
self.requests](https://image.slidesharecdn.com/m3machinelearningmodelsonmobiledeviceslarsgregori-181015162738/85/Machine-Learning-Models-on-Mobile-Devices-19-320.jpg)
![Inception V3 and TF Lite
private void predict(Context context, int imgId) {
Bitmap bitmap = BitmapFactory.decodeResource(...,
imgId, ...);
image.setImageBitmap(bitmap);
ByteBuffer imgBuf = convertBitmapToByteBuffer(bitmap);
float[][] labelProb = new float[1][labels.size()];
tflite.run(imgBuf, labelProb);
>>>
predict
imgBuf
imgBuf
labelProb
labelProbtflite.run](https://image.slidesharecdn.com/m3machinelearningmodelsonmobiledeviceslarsgregori-181015162738/85/Machine-Learning-Models-on-Mobile-Devices-22-320.jpg)
![Inception V3 and TF Lite
String label = "";
Float percent = -1.0f;
for(int index = 0; index < labels.size(); index++) {
if (labelProb[0][index] > percent) {
percent = labelProb[0][index];
label = labels.get(index);
}
labelProb
labelProb
index
index
index
index
labels
labels](https://image.slidesharecdn.com/m3machinelearningmodelsonmobiledeviceslarsgregori-181015162738/85/Machine-Learning-Models-on-Mobile-Devices-23-320.jpg)
![XOR Keras Model
training_data = np.array([[0,0], [0,1], [1,0], [1,1]])
target_data = np.array([ [0], [1], [1], [0]])
model = Sequential()
model.add(Dense(8, input_dim=2, activation=sigmoid))
model.add(Dense(1, activation=sigmoid))
model.compile(loss=mean_squared_error, optimizer=SGD(lr=1.0))
model.fit(training_data, target_data, epochs=2000)
>> Jupyter Notebook
Jupyter Notebook: https://github.com/choas/Machine_Learning_Models_In_Apps/blob/master/XOR/models/XOR.ipynb
SGD - stochastic gradient descent - https://en.wikipedia.org/wiki/Stochastic_gradient_descent
Sigmoid - https://en.wikipedia.org/wiki/Sigmoid_function](https://image.slidesharecdn.com/m3machinelearningmodelsonmobiledeviceslarsgregori-181015162738/85/Machine-Learning-Models-on-Mobile-Devices-28-320.jpg)
![XOR Keras Model
print model.predict(training_data)
[[0.03998361]
[0.9534271 ]
[0.9549079 ]
[0.05220655]]](https://image.slidesharecdn.com/m3machinelearningmodelsonmobiledeviceslarsgregori-181015162738/85/Machine-Learning-Models-on-Mobile-Devices-29-320.jpg)
![XOR and CoreML
func predict(_ a: NSNumber, _ b: NSNumber) -> NSNumber {
let input_data = try! MLMultiArray(shape:[2], …)
input_data[0] = a
input_data[1] = b
let xor_input = xorInput(input: input_data)
let prediction = try! xor().prediction(input: xor_input)
return prediction.result[0]
}
predict
input_data
input_data
input_data
input_data
xor().predictionprediction
prediction.result[0]
xorInput](https://image.slidesharecdn.com/m3machinelearningmodelsonmobiledeviceslarsgregori-181015162738/85/Machine-Learning-Models-on-Mobile-Devices-34-320.jpg)
![XOR and TF Lite
private String prediction(int a, int b) {
float[][] in = new float[][]{{a, b}};
float[][] out = new float[][]{{0}};
tflite.run(in, out);
return String.valueOf(out[0][0]);
}
prediction
in
in
out
out
out[0][0]
tflite.run(in, out)](https://image.slidesharecdn.com/m3machinelearningmodelsonmobiledeviceslarsgregori-181015162738/85/Machine-Learning-Models-on-Mobile-Devices-36-320.jpg)
Machine Learning Models on Mobile Devices
- 1. Machine Learning Models on Mobile Devices Lars Gregori
- 2. Why Machine Learning Models on Mobile Devices? •Privacy •Availability •Speed
- 5. What Models (CoreML)? convolutional Caffe Keras random forests scikit-learn XGBoost multiclass classification logic regression https://developer.apple.com/documentation/coreml/converting_trained_models_to_core_ml
- 6. How? •CoreML •TensorFlow Lite •Other e.g. Caffe2Go
- 7. CoreML •Apple •WWDC 2017 •iOS, macOS, watchOS, tvOS
- 8. Application Vision NaturalLanguage GameplayKit CoreML Accelerate und BNNS MetalPerformanceShaders HardwareCPU GPU CoreML BNNS - Basic neural network subroutines https://developer.apple.com/documentation/accelerate/bnns
- 9. TensorFlow Lite •Google •Based on TensorFlow Mobile •Android, iOS and Raspberry Pi
- 10. TensorFlow Lite .tflite fileAndroid App Java API C++ API Interpreter Android Neural Network API Operators .tflite fileiOS App C++ API Interpreter Operators
- 12. 95.1% African elephant, Loxodonta africana 4.2% tusker 0.4% Indian elephant, Elephas maximus 0.0% moped 0.0% warthog
- 13. Inception V3 •Object detection (96.5% top-5) •Google Brain, arXiv:1512.00567 •1000 classes; ILSVRC2012* ILSVRC2012 - ImageNet Large Scale Visual Recognition Challenge 2012
- 14. Inception V3 and CoreML https://developer.apple.com/machine-learning/build-run-models/
- 15. Inception V3 and CoreML
- 16. Inception V3 and CoreML override func viewDidLoad() { super.viewDidLoad() let model = try! VNCoreMLModel(for: Inceptionv3().model) let request = VNCoreMLRequest(model: model, completionHandler: resHandler) self.requests.append(request) } Inceptionv3().model resHandler self.requests
- 17. Inception V3 and CoreML func resHandler(request: VNRequest, _: Error?) { let results = request.results as! [VNClassificationObservation] let percent = Int(results[0].confidence * 100) let identifier = results[0].identifier resultLabel.text = "(percent)% (identifier)" } resHandler results results results
- 18. Inception V3 and CoreML
- 19. Inception V3 and CoreML @IBAction func predict(_ sender: UIButton) { selectedImage.image = sender.currentImage let image = sender.currentImage!.cgImage! let handler = VNImageRequestHandler(cgImage: image, options: [:]) try? handler.perform(self.requests) } @IBAction sender.currentImage!.cgImage! handler.preform handler image = image self.requests
- 20. Inception V3 and TF Lite https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/lite/g3doc/models.md
- 21. Inception V3 and TF Lite String tffile = “inception_v3.tflite”; protected void onCreate(Bundle savedInstanceState) { tflite = new Interpreter(loadModelFile(tffile)); button1.setOnClickListener(new View.OnClickListener() { @Override public void onClick(View v) { predict(context, R.drawable.image1); Interpreter setOnClickListener predict
- 22. Inception V3 and TF Lite private void predict(Context context, int imgId) { Bitmap bitmap = BitmapFactory.decodeResource(..., imgId, ...); image.setImageBitmap(bitmap); ByteBuffer imgBuf = convertBitmapToByteBuffer(bitmap); float[][] labelProb = new float[1][labels.size()]; tflite.run(imgBuf, labelProb); >>> predict imgBuf imgBuf labelProb labelProbtflite.run
- 23. Inception V3 and TF Lite String label = ""; Float percent = -1.0f; for(int index = 0; index < labels.size(); index++) { if (labelProb[0][index] > percent) { percent = labelProb[0][index]; label = labels.get(index); } labelProb labelProb index index index index labels labels
- 24. Image Retraining 37.7% strawberry 11.9% ice cream, icecream 7.9% pineapple, ananas 2.8% ear, spike, capitulum 2.6% trifle
- 25. Image Retraining •Transfer learning •Already trained model •New classification layer on top https://www.tensorflow.org/hub/tutorials/image_retraining
- 26. Example XOR
- 27. XOR •0 xor 0 = 0 •0 xor 1 = 1 •1 xor 0 = 1 •1 xor 1 = 0
- 28. XOR Keras Model training_data = np.array([[0,0], [0,1], [1,0], [1,1]]) target_data = np.array([ [0], [1], [1], [0]]) model = Sequential() model.add(Dense(8, input_dim=2, activation=sigmoid)) model.add(Dense(1, activation=sigmoid)) model.compile(loss=mean_squared_error, optimizer=SGD(lr=1.0)) model.fit(training_data, target_data, epochs=2000) >> Jupyter Notebook Jupyter Notebook: https://github.com/choas/Machine_Learning_Models_In_Apps/blob/master/XOR/models/XOR.ipynb SGD - stochastic gradient descent - https://en.wikipedia.org/wiki/Stochastic_gradient_descent Sigmoid - https://en.wikipedia.org/wiki/Sigmoid_function
- 29. XOR Keras Model print model.predict(training_data) [[0.03998361] [0.9534271 ] [0.9549079 ] [0.05220655]]
- 30. Save Model •CoreML Tools (Keras, …) •tfcoreml (Tensorflow) •TensorFlow Lite Optimizing Converter (TOCO)
- 31. DEMO XOR
- 32. XOR Example
- 33. XOR and CoreML override func viewDidLoad() { super.viewDidLoad() out00.text = "(predict(0, 0))" out01.text = "(predict(0, 1))" out10.text = "(predict(1, 0))" out11.text = "(predict(1, 1))" } predict(0, 0) predict(0, 1) predict(1, 0) predict(1, 1)
- 34. XOR and CoreML func predict(_ a: NSNumber, _ b: NSNumber) -> NSNumber { let input_data = try! MLMultiArray(shape:[2], …) input_data[0] = a input_data[1] = b let xor_input = xorInput(input: input_data) let prediction = try! xor().prediction(input: xor_input) return prediction.result[0] } predict input_data input_data input_data input_data xor().predictionprediction prediction.result[0] xorInput
- 35. XOR and TF Lite protected void onCreate(Bundle savedInstanceState) { // … tflite = new Interpreter(loadModelFile("xor.tflite")); result00.setText(prediction(0, 0)); result01.setText(prediction(0, 1)); result10.setText(prediction(1, 0)); result11.setText(prediction(1, 1)); } tflite prediction(0, 0) prediction(0, 1) prediction(1, 0) prediction(1, 1)
- 36. XOR and TF Lite private String prediction(int a, int b) { float[][] in = new float[][]{{a, b}}; float[][] out = new float[][]{{0}}; tflite.run(in, out); return String.valueOf(out[0][0]); } prediction in in out out out[0][0] tflite.run(in, out)
- 37. Quantization
- 38. Quantization •A discrete set of values. •TensorFlow Lite: 8-bits •CoreML 2: Weight Quantization
- 39. Quantization - Inception V3 https://ai.googleblog.com/2016/08/improving-inception-and-image.html
- 40. Quantization - Inception V3 Convolution https://ai.googleblog.com/2016/08/improving-inception-and-image.html
- 42. 97% original Inception V3 97% 16-bit linear 98% 8-bit linear 2% 4-bit linear (pillow is wrong)
- 43. Summary
- 44. Summary •Why •How •What will you build?
- 45. Thank you @choas