SlideShare a Scribd company logo
© 2017 NAVER LABS. All rights reserved.
Matthias Gallé
Naver Labs Europe
@mgalle
Human-Centric Machine Learning
Rakuten Technology Conference 2017
Advanced
Chess
Supervised Learning
Where f typically such that
𝑓 = argmin 𝑓∈𝐹
1
𝑁
෍
𝑖=1
𝐿 𝑓 𝑥𝑖 , 𝑦𝑖 + 𝜆𝑅 𝑓
I know what I want
(and can formalize it)
I have time & money to label lots of data
X,Y f(x)
Example: Machine Translation
Given a text s and its proposed translation p, how to measure its distance with
respect to a reference translation t ?
BLEU: n-gram overlap between t and p
typically: 1 ≤ 𝑛 ≤ 4, precision only, brevity penalty
METEOR
bonus points for matching stems and synonyms
use paraphrases
Statistical Machine Translation
P Koehn
(www.statmt.org/book/slides/08-
evaluation.pdf)
Consequences of not formalizing correctly
Users do not use your model
Computer-Assisted Translation used rule-based systems for years
Ad-hoc solutions
Quality Prediction
Automatic Post Edition
Unsupervised Learning
Where Z(X) capture some prior:
• Compression
• Clustering
• Coverage
• ….
I am not sure what I want
I have a (big) corpus with assumed patterns
X Z(X)
Example: Exploratory Search
Whenever your task is:
• Ill-defined:
– Broad / under-specified
– Multi-faceted
• Dynamic:
– Searcher’s understanding inadequate at the beginning
– Searcher’s understanding evolves as results are gradually retrieved.
The answer to what you search is “I know it when I see it”
https://en.wikipedia.org/wiki/I_know_it_when_I_see_it
Interactive Learning
Exploratory Search: examples
E-Discovery
Sensitivity Review
• Vo, Ngoc Phuoc An, et al. "DISCO: A System Leveraging Semantic Search in Document Review." COLING (Demos). 2016.
• Privault, Caroline, et al. "A new tangible user interface for machine learning document review." Artificial Intelligence and Law 18.4 (2010): 459-479.
• Ferrero, Germán, Audi Primadhanty, and Ariadna Quattoni. "InToEventS: An Interactive Toolkit for Discovering and Building Event Schemas." EACL 2017 (2017): 104.
Example: Active Learning
Give initiative to the algorithm
allow action of type: “please, label instance x”
Cognitive effort of labeling a document 3-5x higher than labelling a word [1]
Feature labelling:
• type(feedback) ≠ type(label)
• information load of a word label is small
• word sense disambiguation
[1] Raghavan, Hema, Omid Madani, and Rosie Jones. "Active learning with feedback on features and instances." Journal of
Machine Learning Research7.Aug (2006): 1655-1686.
Conclusion
If you really want to solve a problem, don’t be prisoner of your
performance indicator
Ask yourself:
1. Does it really capture success?
does it align with human judgment?
2. What does the [machine | human] best?
3. Can you remove the burden from humans by smarter algorithms?
Further reading & Acknowledgments
Jean-Michel RendersMarc Dymetman Ariadna Quattoni
http://www.europe.naverlabs.com/Blog
Q&A
© 2017 NAVER LABS. All rights reserved.
Appendix
© 2017 NAVER LABS. All rights reserved.
Statistical Machine Translation
P Koehn
(www.statmt.org/book/slides/08-
evaluation.pdf)

More Related Content

What's hot (10)

PDF
Machine Learning part 3 - Introduction to data science
Frank Kienle
 
PDF
A General Overview of Machine Learning
Ashish Sharma
 
PPT
Introduction to Machine Learning
butest
 
PPTX
Day 2 (Lecture 5): A Practitioner's Perspective on Building Machine Product i...
Aseda Owusua Addai-Deseh
 
PDF
Data science as career
Manjunath Sindagi
 
PDF
The Predictron: End-to-end Learning and Planning
Yoonho Lee
 
PPTX
real life application in numerical method
Daffodil international University
 
PPTX
Session 04 communicating results
bodaceacat
 
PPTX
Research project ppt for students
KavitaPatil65
 
PPTX
application of numerical method
Shaikat Saha
 
Machine Learning part 3 - Introduction to data science
Frank Kienle
 
A General Overview of Machine Learning
Ashish Sharma
 
Introduction to Machine Learning
butest
 
Day 2 (Lecture 5): A Practitioner's Perspective on Building Machine Product i...
Aseda Owusua Addai-Deseh
 
Data science as career
Manjunath Sindagi
 
The Predictron: End-to-end Learning and Planning
Yoonho Lee
 
real life application in numerical method
Daffodil international University
 
Session 04 communicating results
bodaceacat
 
Research project ppt for students
KavitaPatil65
 
application of numerical method
Shaikat Saha
 

Viewers also liked (20)

PDF
Life of an enginner in rakuten osaka diarmaid lindsay
Rakuten Group, Inc.
 
PDF
Predictions and Hard Problems With AI
Rakuten Group, Inc.
 
PDF
トラブルシューティングのあれこれ Yoshihiko kamata
Rakuten Group, Inc.
 
PDF
Rakuten Technology Conference 2017 A Distributed SQL Database For Data Analy...
Rakuten Group, Inc.
 
PDF
WannaEat: A computer vision-based, multi-platform restaurant lookup app
Rakuten Group, Inc.
 
PDF
はてなのインフラの歴史、そしてMackerelへ至る道とこれから
Rakuten Group, Inc.
 
PDF
AI based language learning tools
Rakuten Group, Inc.
 
PDF
Don't manage too hard!
Rakuten Group, Inc.
 
PDF
COBOL to Apache Spark
Rakuten Group, Inc.
 
PDF
AI AND FUNDAMENTAL GAME TECHNOLOGIESIN FINAL FANTASY XV
Rakuten Group, Inc.
 
PDF
Rakuten app productivity initiative for developers marcus saw
Rakuten Group, Inc.
 
PDF
What i learned from translation of the sre ryuji tamagawa
Rakuten Group, Inc.
 
PDF
Value Delivery through RakutenBig Data Intelligence Ecosystem and Technology
Rakuten Group, Inc.
 
PDF
Rakutenとsreと私 yanagimoto koichi
Rakuten Group, Inc.
 
PDF
Challenge for statup's cto from big company nagaaki hoshi
Rakuten Group, Inc.
 
PDF
One Hundred Languages
Rakuten Group, Inc.
 
PDF
時間がないといって、オペレーション改善を怠るな~オペレーション改善奮闘記~ Emi muroya
Rakuten Group, Inc.
 
PDF
cloudera Apache Kudu Updatable Analytical Storage for Modern Data Platform
Rakuten Group, Inc.
 
PDF
Java ee7 with apache spark for the world's largest credit card core systems, ...
Rakuten Group, Inc.
 
PDF
Building your own static site Using Hugo
Rakuten Group, Inc.
 
Life of an enginner in rakuten osaka diarmaid lindsay
Rakuten Group, Inc.
 
Predictions and Hard Problems With AI
Rakuten Group, Inc.
 
トラブルシューティングのあれこれ Yoshihiko kamata
Rakuten Group, Inc.
 
Rakuten Technology Conference 2017 A Distributed SQL Database For Data Analy...
Rakuten Group, Inc.
 
WannaEat: A computer vision-based, multi-platform restaurant lookup app
Rakuten Group, Inc.
 
はてなのインフラの歴史、そしてMackerelへ至る道とこれから
Rakuten Group, Inc.
 
AI based language learning tools
Rakuten Group, Inc.
 
Don't manage too hard!
Rakuten Group, Inc.
 
COBOL to Apache Spark
Rakuten Group, Inc.
 
AI AND FUNDAMENTAL GAME TECHNOLOGIESIN FINAL FANTASY XV
Rakuten Group, Inc.
 
Rakuten app productivity initiative for developers marcus saw
Rakuten Group, Inc.
 
What i learned from translation of the sre ryuji tamagawa
Rakuten Group, Inc.
 
Value Delivery through RakutenBig Data Intelligence Ecosystem and Technology
Rakuten Group, Inc.
 
Rakutenとsreと私 yanagimoto koichi
Rakuten Group, Inc.
 
Challenge for statup's cto from big company nagaaki hoshi
Rakuten Group, Inc.
 
One Hundred Languages
Rakuten Group, Inc.
 
時間がないといって、オペレーション改善を怠るな~オペレーション改善奮闘記~ Emi muroya
Rakuten Group, Inc.
 
cloudera Apache Kudu Updatable Analytical Storage for Modern Data Platform
Rakuten Group, Inc.
 
Java ee7 with apache spark for the world's largest credit card core systems, ...
Rakuten Group, Inc.
 
Building your own static site Using Hugo
Rakuten Group, Inc.
 
Ad

Similar to Human-Centric Machine Learning (20)

PDF
ChatGPT in academic settings H2.de
David Döring
 
PPTX
Machine_Learning.pptx
shubhamatak136
 
PDF
Make Learning Big Data Work For You
Jessie Chuang
 
PDF
Xiangen Hu - WESST - AutoTutor, an implementation of Conversation-Based Intel...
NUS Institute of Applied Learning Sciences and Educational Technology
 
PPTX
The Corpus of Business Discourse
ACBSP Global Accreditation
 
PDF
Xiangen Hu - WESST Keynote - Conversational Tutors and the Experience API
NUS Institute of Applied Learning Sciences and Educational Technology
 
PDF
MSRA 2018: Intelligent Software Engineering: Synergy between AI and Software ...
Tao Xie
 
PDF
Machine learning with in the python lecture for computer science
jayasreepalani02
 
PPTX
Advancing Foundation and Practice of Software Analytics
Tao Xie
 
PPT
Hci techniques from idea to deployment
John Thomas
 
PPTX
Ai in Higher Education
Stephen Murgatroyd, PhD FBPsS FRSA
 
PPTX
Interoperability: Scientific Foundations
Yannis Charalabidis
 
PPTX
Movie Recommendation System.pptx
randominfo
 
DOC
taghelper-final.doc
butest
 
PDF
Introduction to Machine Learning
Eng Teong Cheah
 
PDF
Learning Content and Usage Factors Simultaneously
Arnab Bhadury
 
PDF
ISEC'18 Keynote: Intelligent Software Engineering: Synergy between AI and Sof...
Tao Xie
 
PPT
The impact of standardized terminologies and domain-ontologies in multilingua...
AIMS (Agricultural Information Management Standards)
 
PPTX
The Data Science Product Management Toolkit
Jack Moore
 
PPT
Experimenting with eXtreme Design (EKAW2010)
evabl444
 
ChatGPT in academic settings H2.de
David Döring
 
Machine_Learning.pptx
shubhamatak136
 
Make Learning Big Data Work For You
Jessie Chuang
 
Xiangen Hu - WESST - AutoTutor, an implementation of Conversation-Based Intel...
NUS Institute of Applied Learning Sciences and Educational Technology
 
The Corpus of Business Discourse
ACBSP Global Accreditation
 
Xiangen Hu - WESST Keynote - Conversational Tutors and the Experience API
NUS Institute of Applied Learning Sciences and Educational Technology
 
MSRA 2018: Intelligent Software Engineering: Synergy between AI and Software ...
Tao Xie
 
Machine learning with in the python lecture for computer science
jayasreepalani02
 
Advancing Foundation and Practice of Software Analytics
Tao Xie
 
Hci techniques from idea to deployment
John Thomas
 
Ai in Higher Education
Stephen Murgatroyd, PhD FBPsS FRSA
 
Interoperability: Scientific Foundations
Yannis Charalabidis
 
Movie Recommendation System.pptx
randominfo
 
taghelper-final.doc
butest
 
Introduction to Machine Learning
Eng Teong Cheah
 
Learning Content and Usage Factors Simultaneously
Arnab Bhadury
 
ISEC'18 Keynote: Intelligent Software Engineering: Synergy between AI and Sof...
Tao Xie
 
The impact of standardized terminologies and domain-ontologies in multilingua...
AIMS (Agricultural Information Management Standards)
 
The Data Science Product Management Toolkit
Jack Moore
 
Experimenting with eXtreme Design (EKAW2010)
evabl444
 
Ad

More from Rakuten Group, Inc. (20)

PDF
EPSS (Exploit Prediction Scoring System)モニタリングツールの開発
Rakuten Group, Inc.
 
PPTX
コードレビュー改善のためにJenkinsとIntelliJ IDEAのプラグインを自作してみた話
Rakuten Group, Inc.
 
PDF
楽天における安全な秘匿情報管理への道のり
Rakuten Group, Inc.
 
PDF
What Makes Software Green?
Rakuten Group, Inc.
 
PDF
Simple and Effective Knowledge-Driven Query Expansion for QA-Based Product At...
Rakuten Group, Inc.
 
PDF
DataSkillCultureを浸透させる楽天の取り組み
Rakuten Group, Inc.
 
PDF
大規模なリアルタイム監視の導入と展開
Rakuten Group, Inc.
 
PDF
楽天における大規模データベースの運用
Rakuten Group, Inc.
 
PDF
楽天サービスを支えるネットワークインフラストラクチャー
Rakuten Group, Inc.
 
PDF
楽天の規模とクラウドプラットフォーム統括部の役割
Rakuten Group, Inc.
 
PDF
Rakuten Services and Infrastructure Team.pdf
Rakuten Group, Inc.
 
PDF
The Data Platform Administration Handling the 100 PB.pdf
Rakuten Group, Inc.
 
PDF
Supporting Internal Customers as Technical Account Managers.pdf
Rakuten Group, Inc.
 
PDF
Making Cloud Native CI_CD Services.pdf
Rakuten Group, Inc.
 
PDF
How We Defined Our Own Cloud.pdf
Rakuten Group, Inc.
 
PDF
Travel & Leisure Platform Department's tech info
Rakuten Group, Inc.
 
PDF
Travel & Leisure Platform Department's tech info
Rakuten Group, Inc.
 
PDF
OWASPTop10_Introduction
Rakuten Group, Inc.
 
PDF
Introduction of GORA API Group technology
Rakuten Group, Inc.
 
PDF
100PBを越えるデータプラットフォームの実情
Rakuten Group, Inc.
 
EPSS (Exploit Prediction Scoring System)モニタリングツールの開発
Rakuten Group, Inc.
 
コードレビュー改善のためにJenkinsとIntelliJ IDEAのプラグインを自作してみた話
Rakuten Group, Inc.
 
楽天における安全な秘匿情報管理への道のり
Rakuten Group, Inc.
 
What Makes Software Green?
Rakuten Group, Inc.
 
Simple and Effective Knowledge-Driven Query Expansion for QA-Based Product At...
Rakuten Group, Inc.
 
DataSkillCultureを浸透させる楽天の取り組み
Rakuten Group, Inc.
 
大規模なリアルタイム監視の導入と展開
Rakuten Group, Inc.
 
楽天における大規模データベースの運用
Rakuten Group, Inc.
 
楽天サービスを支えるネットワークインフラストラクチャー
Rakuten Group, Inc.
 
楽天の規模とクラウドプラットフォーム統括部の役割
Rakuten Group, Inc.
 
Rakuten Services and Infrastructure Team.pdf
Rakuten Group, Inc.
 
The Data Platform Administration Handling the 100 PB.pdf
Rakuten Group, Inc.
 
Supporting Internal Customers as Technical Account Managers.pdf
Rakuten Group, Inc.
 
Making Cloud Native CI_CD Services.pdf
Rakuten Group, Inc.
 
How We Defined Our Own Cloud.pdf
Rakuten Group, Inc.
 
Travel & Leisure Platform Department's tech info
Rakuten Group, Inc.
 
Travel & Leisure Platform Department's tech info
Rakuten Group, Inc.
 
OWASPTop10_Introduction
Rakuten Group, Inc.
 
Introduction of GORA API Group technology
Rakuten Group, Inc.
 
100PBを越えるデータプラットフォームの実情
Rakuten Group, Inc.
 

Recently uploaded (20)

PDF
Responsible AI and AI Ethics - By Sylvester Ebhonu
Sylvester Ebhonu
 
PDF
RAT Builders - How to Catch Them All [DeepSec 2024]
malmoeb
 
PPTX
Farrell_Programming Logic and Design slides_10e_ch02_PowerPoint.pptx
bashnahara11
 
PDF
Presentation about Hardware and Software in Computer
snehamodhawadiya
 
PPTX
Agile Chennai 18-19 July 2025 | Emerging patterns in Agentic AI by Bharani Su...
AgileNetwork
 
PPTX
Introduction to Flutter by Ayush Desai.pptx
ayushdesai204
 
PDF
State-Dependent Conformal Perception Bounds for Neuro-Symbolic Verification
Ivan Ruchkin
 
PDF
TrustArc Webinar - Navigating Data Privacy in LATAM: Laws, Trends, and Compli...
TrustArc
 
PDF
CIFDAQ's Market Wrap : Bears Back in Control?
CIFDAQ
 
PDF
The Future of Artificial Intelligence (AI)
Mukul
 
PDF
MASTERDECK GRAPHSUMMIT SYDNEY (Public).pdf
Neo4j
 
PPTX
Applied-Statistics-Mastering-Data-Driven-Decisions.pptx
parmaryashparmaryash
 
PPTX
Dev Dives: Automate, test, and deploy in one place—with Unified Developer Exp...
AndreeaTom
 
PDF
Peak of Data & AI Encore - Real-Time Insights & Scalable Editing with ArcGIS
Safe Software
 
PDF
Data_Analytics_vs_Data_Science_vs_BI_by_CA_Suvidha_Chaplot.pdf
CA Suvidha Chaplot
 
PDF
A Strategic Analysis of the MVNO Wave in Emerging Markets.pdf
IPLOOK Networks
 
PDF
How ETL Control Logic Keeps Your Pipelines Safe and Reliable.pdf
Stryv Solutions Pvt. Ltd.
 
PPTX
The Future of AI & Machine Learning.pptx
pritsen4700
 
PDF
Generative AI vs Predictive AI-The Ultimate Comparison Guide
Lily Clark
 
PDF
Make GenAI investments go further with the Dell AI Factory
Principled Technologies
 
Responsible AI and AI Ethics - By Sylvester Ebhonu
Sylvester Ebhonu
 
RAT Builders - How to Catch Them All [DeepSec 2024]
malmoeb
 
Farrell_Programming Logic and Design slides_10e_ch02_PowerPoint.pptx
bashnahara11
 
Presentation about Hardware and Software in Computer
snehamodhawadiya
 
Agile Chennai 18-19 July 2025 | Emerging patterns in Agentic AI by Bharani Su...
AgileNetwork
 
Introduction to Flutter by Ayush Desai.pptx
ayushdesai204
 
State-Dependent Conformal Perception Bounds for Neuro-Symbolic Verification
Ivan Ruchkin
 
TrustArc Webinar - Navigating Data Privacy in LATAM: Laws, Trends, and Compli...
TrustArc
 
CIFDAQ's Market Wrap : Bears Back in Control?
CIFDAQ
 
The Future of Artificial Intelligence (AI)
Mukul
 
MASTERDECK GRAPHSUMMIT SYDNEY (Public).pdf
Neo4j
 
Applied-Statistics-Mastering-Data-Driven-Decisions.pptx
parmaryashparmaryash
 
Dev Dives: Automate, test, and deploy in one place—with Unified Developer Exp...
AndreeaTom
 
Peak of Data & AI Encore - Real-Time Insights & Scalable Editing with ArcGIS
Safe Software
 
Data_Analytics_vs_Data_Science_vs_BI_by_CA_Suvidha_Chaplot.pdf
CA Suvidha Chaplot
 
A Strategic Analysis of the MVNO Wave in Emerging Markets.pdf
IPLOOK Networks
 
How ETL Control Logic Keeps Your Pipelines Safe and Reliable.pdf
Stryv Solutions Pvt. Ltd.
 
The Future of AI & Machine Learning.pptx
pritsen4700
 
Generative AI vs Predictive AI-The Ultimate Comparison Guide
Lily Clark
 
Make GenAI investments go further with the Dell AI Factory
Principled Technologies
 

Human-Centric Machine Learning

  • 1. © 2017 NAVER LABS. All rights reserved. Matthias Gallé Naver Labs Europe @mgalle Human-Centric Machine Learning Rakuten Technology Conference 2017
  • 3. Supervised Learning Where f typically such that 𝑓 = argmin 𝑓∈𝐹 1 𝑁 ෍ 𝑖=1 𝐿 𝑓 𝑥𝑖 , 𝑦𝑖 + 𝜆𝑅 𝑓 I know what I want (and can formalize it) I have time & money to label lots of data X,Y f(x)
  • 4. Example: Machine Translation Given a text s and its proposed translation p, how to measure its distance with respect to a reference translation t ? BLEU: n-gram overlap between t and p typically: 1 ≤ 𝑛 ≤ 4, precision only, brevity penalty METEOR bonus points for matching stems and synonyms use paraphrases
  • 5. Statistical Machine Translation P Koehn (www.statmt.org/book/slides/08- evaluation.pdf)
  • 6. Consequences of not formalizing correctly Users do not use your model Computer-Assisted Translation used rule-based systems for years Ad-hoc solutions Quality Prediction Automatic Post Edition
  • 7. Unsupervised Learning Where Z(X) capture some prior: • Compression • Clustering • Coverage • …. I am not sure what I want I have a (big) corpus with assumed patterns X Z(X)
  • 8. Example: Exploratory Search Whenever your task is: • Ill-defined: – Broad / under-specified – Multi-faceted • Dynamic: – Searcher’s understanding inadequate at the beginning – Searcher’s understanding evolves as results are gradually retrieved. The answer to what you search is “I know it when I see it”
  • 11. Exploratory Search: examples E-Discovery Sensitivity Review • Vo, Ngoc Phuoc An, et al. "DISCO: A System Leveraging Semantic Search in Document Review." COLING (Demos). 2016. • Privault, Caroline, et al. "A new tangible user interface for machine learning document review." Artificial Intelligence and Law 18.4 (2010): 459-479. • Ferrero, Germán, Audi Primadhanty, and Ariadna Quattoni. "InToEventS: An Interactive Toolkit for Discovering and Building Event Schemas." EACL 2017 (2017): 104.
  • 12. Example: Active Learning Give initiative to the algorithm allow action of type: “please, label instance x” Cognitive effort of labeling a document 3-5x higher than labelling a word [1] Feature labelling: • type(feedback) ≠ type(label) • information load of a word label is small • word sense disambiguation [1] Raghavan, Hema, Omid Madani, and Rosie Jones. "Active learning with feedback on features and instances." Journal of Machine Learning Research7.Aug (2006): 1655-1686.
  • 13. Conclusion If you really want to solve a problem, don’t be prisoner of your performance indicator Ask yourself: 1. Does it really capture success? does it align with human judgment? 2. What does the [machine | human] best? 3. Can you remove the burden from humans by smarter algorithms?
  • 14. Further reading & Acknowledgments Jean-Michel RendersMarc Dymetman Ariadna Quattoni http://www.europe.naverlabs.com/Blog
  • 15. Q&A © 2017 NAVER LABS. All rights reserved.
  • 16. Appendix © 2017 NAVER LABS. All rights reserved.
  • 17. Statistical Machine Translation P Koehn (www.statmt.org/book/slides/08- evaluation.pdf)