Open and Automated Machine Learning
1 like503 views
This document summarizes an presentation about OpenML, an online platform for sharing machine learning data and experiments. OpenML allows users to search datasets, build machine learning models using various tools/APIs, run experiments on tasks, and automatically upload results. This facilitates reproducibility, benchmarking, and reuse of prior work. OpenML also aims to advance automated machine learning through meta-learning techniques that leverage the large amount of shared data and experiments.
![Search (API)
import pandas as pd
datalist = pd.DataFrame.from_dict(openml_list)
datalist[datalist.NumberOfInstances>10000
].sort_values(['NumberOfInstances'])
Python, R, Java, C#](https://image.slidesharecdn.com/openmltechtalk-180828065132/85/Open-and-Automated-Machine-Learning-15-320.jpg)
![Get (API)
dataset = oml.datasets.get_dataset(1471)
dataset.description[:500]
Python, R, Java, C#](https://image.slidesharecdn.com/openmltechtalk-180828065132/85/Open-and-Automated-Machine-Learning-17-320.jpg)
![Get (API)
X, y, attribute_names = dataset.get_data(
return_attribute_names=True)
Python, R, Java, C#
eeg = pd.DataFrame(X, columns=attribute_names)
eeg['class'] = y](https://image.slidesharecdn.com/openmltechtalk-180828065132/85/Open-and-Automated-Machine-Learning-18-320.jpg)
![Fit and share pipelines
Uploaded to http://www.openml.org/r/7943199
from sklearn import pipeline, ensemble, preprocessing
from openml import tasks,runs, datasets
task = tasks.get_task(59)
pipe = pipeline.Pipeline(steps=[
('Imputer', preprocessing.Imputer()),
('OneHotEncoder', preprocessing.OneHotEncoder(),
('Classifier', ensemble.RandomForestClassifier())
])
flow = oml.flows.sklearn_to_flow(pipe)
run = oml.runs.run_flow_on_task(task, flow)
myrun = run.publish()](https://image.slidesharecdn.com/openmltechtalk-180828065132/85/Open-and-Automated-Machine-Learning-30-320.jpg)
![Fit and share deep learning models
import keras
from keras.models import Sequential,
from keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPooling
from keras.layers.core import Activation
model = Sequential()
model.add(Reshape((28, 28, 1), input_shape=(784,)))
model.add(Conv2D(20, (5, 5), padding=“same", input_shape=(28,28,1),
activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
model.add(Conv2D(50, (5, 5), padding="same", activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
model.add(Flatten())
model.add(Dense(500))
model.add(Activation(‘relu'))
model.add(Dense(10))
model.add(Activation('softmax'))
model.compile(loss=keras.losses.categorical_crossentropy,
optimizer=keras.optimizers.Adadelta(),
metrics=['accuracy'])](https://image.slidesharecdn.com/openmltechtalk-180828065132/85/Open-and-Automated-Machine-Learning-31-320.jpg)
![Download, reuse runs
myruns = oml.runs.list_runs(task=[14951])
sns.violinplot(x="score", y="flow", data=pd.DataFrame(](https://image.slidesharecdn.com/openmltechtalk-180828065132/85/Open-and-Automated-Machine-Learning-36-320.jpg)
Open and Automated Machine Learning
- 1. OpenML O P E N , A U T O M AT E D M A C H I N E L E A R N I N G J O A Q U I N VA N S C H O R E N , T U / E @open_mlwww.openml.org
- 2. OpenML You can be part of this presentation :) Follow the code examples: • On Google Colab: goo.gl/VwbKb4 • On Github: https://git.io/fA3eL J O A Q U I N VA N S C H O R E N , T U / E @open_mlwww.openml.org
- 5. (Not so) Automatic Machine Learning It’s hard to find and learn from prior machine learning data
- 6. (Auto)ML: manual work, unnecessary friction Hard to find and reuse prior results No standards / hubs for sharing and organizing results Scattered, ill-described, datasets Manual searching, reformatting, making assumptions Hard to automate model building end-to-end Requires automated data organization, clean APIs,… Myriad algorithms, versions, languages Write code, set up experiments, store results,… Reproducibility is hard Manually tracking every detail is error-prone
- 7. Easy to use: Integrated in many ML tools/environments Easy to contribute: Automated sharing of data, code, results Organized data: APIs to find & reuse data, models, experiments Reward structure: Track your impact, build reputation Self-learning: Learn from many experiments to help people OpenML S H A R E A N D R E U S E M A C H I N E L E A R N I N G D ATA O N L I N E
- 9. OpenML: Components Flows: Pipelines/code that build ML models Run locally (or wherever), auto-upload all results Datasets: Auto-annotated, organized, well-formatted Find the datasets you need, share your own Tasks: Auto-generated, machine-readable Everyone’s results are directly comparable Runs:All results from running flows on tasks All details needed for tracking and reproducibility Evaluations can be queried, compared, reused
- 10. It starts with data
- 11. Data (tabular) easily uploaded or referenced (URL) It starts with data
- 12. Data can remain in existing repositories -> registered via URL, transparent to users interoperability For now: only tabular data -> ARFF or CSV import (auto-annotate features) -> FrictionlessData support in the works
- 13. auto-versioned, analysed, organised online
- 14. Search (API) import openml as oml openml_list = oml.datasets.list_datasets() Python, R, Java, C#
- 16. Search (API) datalist.query('name == "eeg-eye-state"') Python, R, Java, C# data id
- 17. Get (API) dataset = oml.datasets.get_dataset(1471) dataset.description[:500] Python, R, Java, C#
- 18. Get (API) X, y, attribute_names = dataset.get_data( return_attribute_names=True) Python, R, Java, C# eeg = pd.DataFrame(X, columns=attribute_names) eeg['class'] = y
- 20. Fit (API) Python, R, Java, C# from sklearn import neighbors clf = neighbors.KNeighborsClassifier() clf.fit(X, y)
- 21. Complete code to build a model, automatically, anywhere import openml as oml from sklearn import neighbors, tree dataset = oml.datasets.get_dataset(1471) X, y = dataset.get_data() clf = neighbors.KNeighborsClassifier() clf.fit(X, y)
- 22. Tasks contain data, goals, procedures. Auto-build + evaluate models correctly All evaluations are directly comparable optimize accuracy Predict target T Tasks benchmarking and collaboration 10-fold Xval
- 23. 10-fold Xval Predict target T Collaborate in real time online optimize accuracy
- 25. Search mytasks = pd.DataFrame.from_dict(task_list) mytasks.query('name=="eeg-eye-state"') task id
- 27. Auto-run algorithms/workflows on any task Integrated in many machine learning tools (+ APIs) Flows Run experiments locally, share them globally
- 28. Integrated in many machine learning tools (+ APIs) import openml as oml from sklearn import tree task = oml.tasks.get_task(14951) clf = tree.ExtraTreeClassifier() flow = oml.flows.sklearn_to_flow(clf) run = oml.runs.run_flow_on_task(task, flow) myrun = run.publish()
- 29. Fit and share (complete code) Uploaded to http://www.openml.org/r/9204488 import openml as oml from sklearn import tree task = oml.tasks.get_task(14951) clf = tree.ExtraTreeClassifier() flow = oml.flows.sklearn_to_flow(clf) run = oml.runs.run_flow_on_task(task, flow) myrun = run.publish()
- 30. Fit and share pipelines Uploaded to http://www.openml.org/r/7943199 from sklearn import pipeline, ensemble, preprocessing from openml import tasks,runs, datasets task = tasks.get_task(59) pipe = pipeline.Pipeline(steps=[ ('Imputer', preprocessing.Imputer()), ('OneHotEncoder', preprocessing.OneHotEncoder(), ('Classifier', ensemble.RandomForestClassifier()) ]) flow = oml.flows.sklearn_to_flow(pipe) run = oml.runs.run_flow_on_task(task, flow) myrun = run.publish()
- 31. Fit and share deep learning models import keras from keras.models import Sequential, from keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPooling from keras.layers.core import Activation model = Sequential() model.add(Reshape((28, 28, 1), input_shape=(784,))) model.add(Conv2D(20, (5, 5), padding=“same", input_shape=(28,28,1), activation='relu')) model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2))) model.add(Conv2D(50, (5, 5), padding="same", activation='relu')) model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2))) model.add(Flatten()) model.add(Dense(500)) model.add(Activation(‘relu')) model.add(Dense(10)) model.add(Activation('softmax')) model.compile(loss=keras.losses.categorical_crossentropy, optimizer=keras.optimizers.Adadelta(), metrics=['accuracy'])
- 32. Fit and share deep learning models Uploaded to https://www.openml.org/r/9204337 task = tasks.get_task(3573) #MNIST flow = oml.flows.keras_to_flow(model) run = oml.runs.run_flow_on_task(task, flow) myrun = run.publish()
- 34. reproducible, linked to data, flows, authors and all other experiments Experiments auto-uploaded, evaluated online Runs Share and reuse results
- 35. Experiments auto-uploaded, evaluated online
- 39. OpenML Community 5600+ registered users, 120000+ yearly users
- 40. A U T O M L : M E TA - L E A R N I N G • Find similar datasets • 20,000+ versioned datasets, with 130+ meta-features • Instead of starting from scratch, start from configurations that worked well on similar datasets
- 41. A U T O M L : M E TA - L E A R N I N G • Find similar datasets • 20,000+ versioned datasets, with 130+ meta-features • Instead of starting from scratch, start from configurations that worked well on similar datasets • Auto-sklearn (AutoML challenge winner, NIPS 2016) • Lookup similar datasets, start with best pipelines Matthias Feurer et al. (2016) NIPS
- 42. A U T O M L : M E TA - L E A R N I N G • Find similar datasets • 20,000+ versioned datasets, with 130+ meta-features • Instead of starting from scratch, start from configurations that worked well on similar datasets • Auto-sklearn (AutoML challenge winner, NIPS 2016) • Lookup similar datasets, start with best pipelines
- 43. A U T O M L : M E TA - L E A R N I N G • Reuse (millions of) prior model evaluations: • Benchmark new algorithms against state-of-the-art • Meta-models: E.g. predict performance or training time • MIT AutoML system (ICBD 2017) • Uses and compares against OpenML results
- 44. A U T O M L : M E TA - L E A R N I N G • Reuse (millions of) prior model evaluations: • Benchmark new algorithms against state-of-the-art • Meta-models: E.g. predict performance or training time • Runtime prediction
- 45. A U T O M L : M E TA - L E A R N I N G • Reuse (millions of) prior model evaluations: • Benchmark new algorithms against state-of-the-art • Meta-models: E.g. predict performance or training time • Faster TPOT (in progress) • Build meta-models (Random Forest works well) • Focus on fast configurations first
- 46. A U T O M L : M E TA - L E A R N I N G • Reuse results on many hyperparameter settings • Surrogate models: predict best hyperparameter settings • Study hyperparameter effects/importance • Amazon’s multi-task learning AutoML (NIPS 2017) • Trains surrogate models per task • On new tasks: learns how to combine them with neural net
- 47. A U T O M L : M E TA - L E A R N I N G • Reuse results on many hyperparameter settings • Surrogate models: predict best hyperparameter settings • Study hyperparameter effects/importance • Hyperparameter space design • Use OpenML data to learn which hyperparameters to tune Jan van Rijn et al. (2017) AutoML@ICML
- 48. A U T O M L : M E TA - L E A R N I N G • Never-ending Automatic Machine Learning: • AutoML methods built on top of OpenML get increasingly better as more meta-data is added • Faster drug discovery (QSAR) • Meta-learning to build better models that recommend drug candidates for rare diseases ChEMBL DB: 1.4M compounds, 10k proteins,12.8M activities Molecule representations MW #LogP #TPSA #b1 #b2 #b3 #b4 #b5 #b6 #b7 #b8 #b9# !! 377.435 !3.883 !77.85 !1 !1 !0 !0 !0 !0 !0 !0 !0 !! 341.361 !3.411 !74.73 !1 !1 !0 !1 !0 !0 !0 !0 !0 !…! 197.188 !-2.089 !103.78 !1 !1 !0 !1 !0 !0 !0 !1 !0 !! 346.813 !4.705 !50.70 !1 !0 !0 !1 !0 !0 !0 !0 !0! ! ! ! ! ! ! !.! ! ! ! ! ! ! !: !! 16,000 regression datasets x52 pipelines (on OpenML) meta-model all data on new protein optimal models to predict activity (Olier et al., Machine Learning 107(1), 2018)
- 49. Learning to learn Bots that learn from all prior experiments Automate drudge work, help people build models
- 50. Join us! (and change the world) Active open source community We need more bright people - ML/DB experts - Developers - UX
- 51. Support is welcome! Workshop sponsorship (hackathons 2x/year) Donations: OpenML foundation Compute time Project ideas
- 52. E I N D H O V E N U N I V E R S I T Y Looking for: • PhD Students • Scientific programmer
- 53. O P E N M L H A C K AT H O N Paris, September 17-21, 2018 meet.openml.org Co-located with COSEAL
- 55. Questions?