How I Made Zoom In and Enhance - Seattle Mobile .NET
0 likes702 views
This document discusses convolutional neural networks (CNNs) and how they can be used for image zoom and enhancement. It provides an overview of CNN theory, including layers, training, and generalization. It then describes using a pix2pix CNN model to generate training examples for zooming faces, training the model, and testing it on new images. Future work discussed includes porting CNN libraries and models to mobile.
How I Made Zoom In and Enhance - Seattle Mobile .NET
- 1. HOW I MADE ZOOM IN AND ENHANCE CONVOLUTIONAL NEURAL NETWORKS
- 5. ?
- 6. 1. SOMEONE ELSE DID IT 2. FINALLY THERE WAS A LIBRARY I COULD UNDERSTAND
- 7. THEORY
- 8. THEORY ▸ A way to compute a function inspired by the human brain ▸ Much simpler than the brain! ▸ Input -> Network -> Output ▸ Comprised of many neurons ▸ These neurons are interconnected ▸ Some are “inputs” and some are “outputs”
- 9. THEORY DEEP NETWORKS ▸ Nowadays we have so many neurons that we no longer describe the networks with them ▸ Instead, we have layers that contain many neurons ▸ When layers are connected to each other, the constituent neurons get connected ▸ Neurons in different types of layers behave differently from those in other types
- 10. TEXT CONVOLUTIONAL NEURAL NETWORKS ▸ CNNs use a layer type called a Spatial Convolution to operate on images ▸ This is a way of connecting neurons so that nearby pixels in 2D images are connected to nearby neurons ▸ The operation the neurons perform is called convolution and is a generalized technique for manipulating images ▸ The benefits are that local information gets local connections and the operation itself is very powerful
- 11. TEXT RECURRENT NEURAL NETWORKS ▸ RNNs use feedback to predict the future! ▸ Most networks are pure functions that process inputs and produce outputs ▸ RNNs feed outputs back into the network to model time ▸ It’s scarily effective
- 12. THEORY NETWORK NOTATION ▸ Networks are written like a chemistry formula ▸ Instead of atoms, layer types are used
- 13. THEORY TRAINING ▸ We train the network by showing it a bunch of inputs and desired outputs ▸ The training algorithm is called back propagation and involves a lot of number crunching ▸ Neurons assign weights to the neurons they’re connected to ▸ These weights control how much the neighbors influence that neuron ▸ Training is the process of determining these weights ▸ The number of times a pair have been reused is called an epoch
- 14. THEORY MINIMIZE ERROR BY ADJUSTING WEIGHTS
- 15. THEORY GENERALIZING ▸ What’s the point of training a network if we already have a bunch of inputs and outputs? ▸ The hope is that the network will learn how to solve problems it hasn’t yet seen ▸ When we train, we always reserve a batch of validation input and outputs that it never sees ▸ We then use those inputs and outputs to rate the network
- 16. PRACTICE
- 17. PRACTICE NEURAL NETWORK DISTRIBUTION ▸ While the concepts are universal… ▸ Neural networks are released as source code that can train and execute the network ▸ The code can be in any number of languages and use any number of support libraries ▸ Python with TensorFlow ▸ Lua with Torch ▸ Networks may only work with some hardware ▸ NVIDIA CUDA is used a lot (they invest in academia) ▸ Cloud solutions exist varying from virtualized hardware to proprietary languages
- 18. PRACTICE PIX2PIX https://github.com/phillipi/pix2pix ▸ 2 Networks: Generator + Discriminator
- 19. PRACTICE PREREQUISITES - HARDWARE LOTS OF PROCESSING POWER
- 21. PRACTICE INSTALLATION ▸ Install NVIDIA drivers ▸ CUDA - GPU programming SDK ▸ CUDNN - GPU libraries to help writing nets ▸ I did this on Mac and Linux ▸ Install Torch ▸ Lua libraries that can use CUDNN ▸ Install pix2pix
- 22. PRACTICE TRAINING $ DATA_ROOT=./datasets/facades name=facades_generation which_direction=AtoB th train.lua
- 24. PRACTICE TRAINING COMPLETE ▸ We get a trained model file ▸ Contains the structured of the network ▸ Along with the learned weights
- 25. PRACTICE RUNNING THE NETWORK ▸ Now that the network is trained, we can run it against new inputs ▸ Put images you want to test in a special folder ▸ Run test.lua instead of train.lua ▸ Now you have outputs!
- 26. PRACTICE
- 27. BACK TO ZOOM AND ENHANCE
- 28. ZOOM AND ENHANCE GOOD TRAINING EXAMPLES ▸ It some fooling around to learn what the network can and can’t do ▸ You can’t just throw images at it and hope for the best ▸ You must spend time to give it good training examples ▸ Inputs and outputs should only differ by what you want the network to learn ▸ Other differences will cause slow or impossible learning ▸ Examples: aspect ratio, cropping / region of interest, backgrounds
- 29. ZOOM AND ENHANCE FACE ZOOMING EXAMPLE GENERATION ▸ Narrowed the task down to zooming in on faces ▸ Wrote an app that extracts faces from images using Apple’s CoreImage framework ▸ Wrote another app that down samples those faces by 8x, then 16x ▸ This only simulates the problem of Z&E since noise is drastically reduced by this downsampling ▸ Ideally, my inputs and outputs would be taken with the same camera using two different zoom levels. But alas…
- 30. ZOOM AND ENHANCE 300 EXAMPLES - NOT JUST ME
- 35. FUTURE
- 36. GOTTA GITIT WORKIN ON MOBILE
- 37. FUTURE MOBILE LIBRARIES ▸ TensorFlow is a C++ library that runs on Android & iOS ▸ Miguel de Icaza is binding TensorFlow to .NET (hope it works on mobile!) ▸ https://github.com/migueldeicaza/TensorFlowSharp ▸ Apple includes Metal Performance Shaders that contains basic CNN routines ▸ I’m porting Torch, for now…