Structuring Data from Unstructured Things. Sean Lorenz
3 likes1,294 views
The document discusses the importance of structuring data in the Internet of Things (IoT) era, emphasizing the need for adaptive applications and effective data manipulation using tools like the Python library Pandas. It highlights organizing various data types and managing time-series data efficiently for building data-driven IoT applications. Additionally, it touches on predictive modeling and multimodal sensor fusion for health monitoring and management.
Structuring Data from Unstructured Things. Sean Lorenz
- 1. Structuring Data from Unstructured Things Sean Lorenz, Founder & CEO, Senter @seanlorenz | @SenterIoT
- 2. WE CURRENTLY LIVE IN THE INTERNET OF THING ERA
- 3. IS THIS AS GOOD AS THE IOT GETS?
- 4. WE WANT ADAPTIVE APPS FROM CROSS-MANUFACTURER DATA
- 5. APIs ARE THE LIFEBLOOD OF THE IOT
- 6. WHAT DO YOU DO WITH ALL THAT DATA?
- 7. NO MORE UNTAPPED DATA LAKES
- 8. NO MORE BAD IoT DASHBOARDS
- 9. MORE DATA STRUCTURING & SENSEMAKING
- 10. BUILDING DATA-DRIVEN IOT APPS 1 2 3 4 INGEST ORGANIZE PREDICT ACCESS/STORE 2 ORGANIZE
- 11. timeseries images text sparse binary sparse analog INGEST - PROCESS DIFFERENT DATA TYPES
- 12. ORGANIZE - THE SOLUTION??
- 13. ORGANIZE - Python Data Analysis Library (pandas) • A fast and efficient DataFrame object for data manipulation; • Tools for reading and writing data between in-memory data structures and different formats; • Intelligent data alignment and integrated handling of missing data • Columns can be inserted and deleted from data structures for size mutability; • High performance merging and joining of data sets; • Time series-functionality: date range generation and frequency conversion, moving window statistics, moving window linear regressions, date shifting and lagging. Even create domain-specific time offsets and join time series without losing data http://pandas.pydata.org/
- 14. ORGANIZE - Python Data Analysis Library (pandas) http://pandas.pydata.org/
- 15. ORGANIZE - Handling Time Series Data A few things to remember: • Schema design that minimizes memory, disk I/O • How often do you aggregate the data? • Read/write to a database needs to be fast, reliable, scalable, adaptable • Dealing with uneven time period data inputs • How much of the raw data do you keep? • Appending existing vs. creating new DataFrames
- 16. ORGANIZE - 3 Examples Twitter, Fitbit, Temperature
- 17. timeseries images text sparse binary sparse analog Deep RNN & LSTM coding of electrical activity to categorize activity peaks Deep RBM coding of facial anomaly detection from security cameras Deep RBM coding of CRM keywords & phrases for concept clustering Sparse PCA & LASSO of ERP system data for delivery probability Sparse Bayesian coding of IoT sensor data for smart trigger event notifications PREDICT - NOT ALL ALGORITHMS ARE CREATED EQUALLY
- 18. PREDICT - Google TensorFlow + LSTM RRN + time series data
- 19. ACCESS/STORE - So, so, so many options….
- 20. multimodal sensor fusion w/ cognitive deep learning IoT home and health, phone app, & EHR data The Hub for Adaptive Connected Home Health +
- 21. LocationStates Activity Patterns HzBands O2Prediction MedClusters MedUsage Prediction Patient Health States PREDICTIVE HOME HEALTH IoT EXAMPLE CARE PLAN ACTION
- 22. Thanks.