IEEESSCI2017-FOCI4-1039
1 like267 views
The document explores the tighter upper bound of the real log canonical threshold (RLCT) in non-negative matrix factorization (NMF) and its implications for Bayesian inference. It highlights the improvements over previous bounds, the challenges with traditional statistical methods in hierarchical models, and the establishment of a learning theory for NMF. The research achieves a significant advancement by deriving an exact value for the RLCT in cases of rank 2 or less, suggesting tighter bounds for theoretical applications.
![• In general [Watanabe, 2001]
– Let n be the sample size
– Bayesian generalization error 𝑮 𝒏 has
an asymptotic behavior:
𝔼 𝑮 𝒏 =
𝝀
𝒏
+ 𝒐
𝟏
𝒏
• Learning coefficient
𝝀 depends on the model
• 𝝀 is called real log canonical
threshold (RLCT)
2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 15
When does RLCT appear?](https://image.slidesharecdn.com/hayashi1039-180314084003/85/IEEESSCI2017-FOCI4-1039-15-320.jpg)
![Error: Bayes<<Freq.
• In hierarchical structure model, Bayesian 𝝀 is
smaller than frequentist’s one and maximum
posterior one [Watanabe,2001 and 2009]
• Bayesian inference is effective for reducing
the generalization error
• We consider Bayesian inference framework
– Bayesian inference for NMF has been proposed
[Cemgil, 2009] Rem: ← is only discrete
2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 16](https://image.slidesharecdn.com/hayashi1039-180314084003/85/IEEESSCI2017-FOCI4-1039-16-320.jpg)
![Def. RLCT
• RLCT is characterized as a learning coefficient
• It is defined by the largest pole of the
following complex function:
𝜻 𝒛 = න𝑲 𝒘 𝒛 𝝋 𝒘 𝒅𝒘,
where 𝑲 is KL-divergence from true distribution
to learning machine and 𝝋 is prior.
• A statistical model selection method that uses
RLCTs has been proposed [Drton, et al. 2017]
2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 18](https://image.slidesharecdn.com/hayashi1039-180314084003/85/IEEESSCI2017-FOCI4-1039-18-320.jpg)
![Def. RLCT
• RLCT is characterized as a learning coefficient
• It is defined by the largest pole of the
following complex function:
𝜻 𝒛 = න𝑲 𝒘 𝒛 𝝋 𝒘 𝒅𝒘,
where 𝑲 is KL-divergence from true distribution
to learning machine and 𝝋 is prior.
• A statistical model selection method that uses
RLCTs has been proposed [Drton, et al. 2017]
2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 19
known as sBIC
(singular BIC)](https://image.slidesharecdn.com/hayashi1039-180314084003/85/IEEESSCI2017-FOCI4-1039-19-320.jpg)
![Research Goal
• Constructing learning theory of NMF
→focus theoretical generalization error
→focus RLCT of NMF
• Recently, we derived an upper bound of
RLCT [Hayashi, et. al. 2017]
• We used algebraic geometrical method
(singularity resolution)
2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 21](https://image.slidesharecdn.com/hayashi1039-180314084003/85/IEEESSCI2017-FOCI4-1039-21-320.jpg)
![Formalizing and Setting
• Data matrices: 𝑾 𝒏 = 𝑾 𝟏, … , 𝑾 𝒏 ; 𝑴 × 𝑵(× 𝒏)
– For general, we treat not only n=1 but also n>1.
2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 25
[Kohjima et al. 2016/6, modified]
𝑾𝑴
𝑵](https://image.slidesharecdn.com/hayashi1039-180314084003/85/IEEESSCI2017-FOCI4-1039-25-320.jpg)
![Formalizing and Setting
• Data matrices: 𝑾 𝒏 = 𝑾 𝟏, … , 𝑾 𝒏 ; 𝑴 × 𝑵(× 𝒏)
– For general, we treat not only n=1 but also n>1.
• True factorization: 𝑨; 𝑴 × 𝑯 𝟎, 𝑩; 𝑯 𝟎 × 𝑵
• Learner factorization: 𝑿; 𝑴 × 𝑯, 𝒀; 𝑯 × 𝑵
2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 26
[Kohjima et al. 2016/6, modified]
𝑾𝑴
𝑵 𝑯 𝟎 𝑵
𝑯 𝟎
𝑴 𝑨
𝑩](https://image.slidesharecdn.com/hayashi1039-180314084003/85/IEEESSCI2017-FOCI4-1039-26-320.jpg)
![Formalizing and Setting
• Data matrices: 𝑾 𝒏 = 𝑾 𝟏, … , 𝑾 𝒏 ; 𝑴 × 𝑵(× 𝒏)
– For general, we treat not only n=1 but also n>1.
• True factorization: 𝑨; 𝑴 × 𝑯 𝟎, 𝑩; 𝑯 𝟎 × 𝑵
• Learner factorization: 𝑿; 𝑴 × 𝑯, 𝒀; 𝑯 × 𝑵
• What is the Bayesian framework of ↑?
2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 27
[Kohjima et al. 2016/6, modified]
𝑾𝑴
𝑵 𝑯 𝟎 𝑵
𝑯 𝟎
𝑴 𝑨
𝑩](https://image.slidesharecdn.com/hayashi1039-180314084003/85/IEEESSCI2017-FOCI4-1039-27-320.jpg)
![• Assume
𝒒 𝑾 ∝ 𝐞𝐱𝐩 −
𝟏
𝟐
𝑾 − 𝑨𝑩 𝟐 ,
𝒑 𝑾 𝑿, 𝒀 ∝ 𝐞𝐱𝐩 −
𝟏
𝟐
𝑾 − 𝑿𝒀 𝟐
,
and prior 𝝋 is strictly positive and bounded in a
neighborhood of 𝑨, 𝑩 .
• Remark: Poisson and exponential dist. can be
also applied [Hayashi, et al. 2017].
2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 29
Formalizing and Setting](https://image.slidesharecdn.com/hayashi1039-180314084003/85/IEEESSCI2017-FOCI4-1039-29-320.jpg)
![Robustness on Dist.
• In the prior work [Hayashi, et. al. 2017],
we proved that same zeta function can be
applied to Poisson and exponential distribution:
𝒒 𝑾 ∝ 𝐏𝐨𝐢 𝑾 𝑨𝑩 ,
𝒑 𝑾 𝑿, 𝒀 ∝ 𝐏𝐨𝐢 𝑾 𝑿𝒀 ,
𝒒 𝑾 ∝ 𝐄𝐱𝐩𝐨 𝑾 𝑨𝑩 ,
𝒑 𝑾 𝑿, 𝒀 ∝ 𝐄𝐱𝐩𝐨 𝑾 𝑿𝒀 ,
𝜻 𝒛 = ඵ 𝑿𝒀 − 𝑨𝑩 𝟐 𝒛
𝒅𝑿𝒅𝒀 .
2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 44
Even if
We can use](https://image.slidesharecdn.com/hayashi1039-180314084003/85/IEEESSCI2017-FOCI4-1039-44-320.jpg)
IEEESSCI2017-FOCI4-1039
- 1. Tighter Upper Bound of Real Log Canonical Threshold of Non-negative Matrix Factorization and its Application to Bayesian Inference Naoki Hayashi* (TokyoTech, Dept. of MCS) Sumio Watanabe (TokyoTech, Dept. of MCS) 12017/11/28 IEEE SSCI 2017 FOCI, Hawaii
- 2. Slide • This slide is available at http://watanabe-www.math.dis.titech.ac.jp/~nhayashi /pdf/hayashi1039.pdf 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 2
- 3. Index • Introduction • Main Theorem • Discussion • Conclusion • (Appendix: Sketch of Proof) 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 3
- 4. 1. INTRODUCTION 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 4
- 5. Index • Introduction – Non-negative Matrix Factorization – Real Log Canonical Threshold – Research Goal • Main Theorem • Discussion • Conclusion • (Appendix: Sketch of Proof) 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 5
- 6. NMF has been applied • Non-negative Matrix Factorization (NMF) has been applied to many field • E. g. – Purchase basket data → Consumer analysis – Image, sound,… → Signal processing – Text data → Text mining – Microarray data → Bioinformatics ↑ Knowledge/Structure Discovery NMF: data → knowledge 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 6
- 7. Suffering • NMF has hierarchical structure • Likelihood cannot be approximated by Gaussian function • Traditional statistics cannot be used 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 7 HOWEVER AIC BIC
- 8. Suffering • NMF has hierarchical structure • Likelihood cannot be approximated by Gaussian function • Traditional statistics cannot be used Hierarchical structure causes non-identifibility : 𝑿𝒀 = 𝑿𝑷𝑷−𝟏 𝒀; 𝐟𝐨𝐫 ∃𝑷 ≠ 𝑰; 𝑿, 𝒀, 𝑿𝑷, 𝑷−𝟏 𝒀 ≥ 𝟎 𝟏 𝟑 𝟏 𝟑 𝟏 𝟒 𝟏 𝟏 𝟒 𝟓 𝟏 𝟒 = 𝟏 𝟑 𝟏 𝟑 𝟏 𝟒 𝟐 −𝟑 𝟏 𝟐 𝟐 −𝟑 𝟏 𝟐 −𝟏 𝟏 𝟏 𝟒 𝟓 𝟏 𝟒 = 𝟏 𝟕 𝟓 𝟑 𝟓 𝟑 𝟔 𝟓 𝟏𝟕 𝟓 𝟐𝟎 𝟗 𝟏 𝟒 = 𝟏𝟔 𝟒 𝟏𝟔 𝟏𝟔 𝟒 𝟏𝟔 𝟐𝟏 𝟓 𝟐𝟎 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 8 HOWEVER AIC BIC
- 9. Suffering • NMF has hierarchical structure • Likelihood cannot be approximated by Gaussian function • Traditional statistics cannot be used Hierarchical structure causes non-identifibility : 𝑿𝒀 = 𝑿𝑷𝑷−𝟏 𝒀; 𝐟𝐨𝐫 ∃𝑷 ≠ 𝑰; 𝑿, 𝒀, 𝑿𝑷, 𝑷−𝟏 𝒀 ≥ 𝟎 𝟏 𝟑 𝟏 𝟑 𝟏 𝟒 𝟏 𝟏 𝟒 𝟓 𝟏 𝟒 = 𝟏 𝟑 𝟏 𝟑 𝟏 𝟒 𝟐 −𝟑 𝟏 𝟐 𝟐 −𝟑 𝟏 𝟐 −𝟏 𝟏 𝟏 𝟒 𝟓 𝟏 𝟒 = 𝟏 𝟕 𝟓 𝟑 𝟓 𝟑 𝟔 𝟓 𝟏𝟕 𝟓 𝟐𝟎 𝟗 𝟏 𝟒 = 𝟏𝟔 𝟒 𝟏𝟔 𝟏𝟔 𝟒 𝟏𝟔 𝟐𝟏 𝟓 𝟐𝟎 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 9 HOWEVER AIC BIC One matrix, (at least) pairs about the NMF
- 10. Suffering • NMF has hierarchical structure • Likelihood cannot be approximated by Gaussian function • Traditional statistics cannot be used 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 10 HOWEVER • Strongly depending on initial value • Suffering from many local minima – It seldom reaches to the global minimum. In Addition + AIC BIC
- 11. Learning Theory of NMF • NMF has been used for ``data → knowledge’’ 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 11
- 12. • NMF has been used for ``data → knowledge’’ • Mathematical property is unknown – Learning theory has not been yet established – Prediction accuracy has not been yet clarified No guarantee for correctness of numerical calculation No method for theoretical hyperparameter tuning 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 12 Learning Theory of NMF
- 13. • NMF has been used for ``data → knowledge’’ • Mathematical property is unknown – Learning theory has not been yet established – Prediction accuracy has not been yet clarified 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 13 Constructing its theory is an important problem Learning Theory of NMF
- 14. Index • Introduction – Non-negative Matrix Factorization – Real Log Canonical Threshold – Research Goal • Main Theorem • Discussion • Conclusion • (Appendix: Sktech of Proof) 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 14
- 15. • In general [Watanabe, 2001] – Let n be the sample size – Bayesian generalization error 𝑮 𝒏 has an asymptotic behavior: 𝔼 𝑮 𝒏 = 𝝀 𝒏 + 𝒐 𝟏 𝒏 • Learning coefficient 𝝀 depends on the model • 𝝀 is called real log canonical threshold (RLCT) 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 15 When does RLCT appear?
- 16. Error: Bayes<<Freq. • In hierarchical structure model, Bayesian 𝝀 is smaller than frequentist’s one and maximum posterior one [Watanabe,2001 and 2009] • Bayesian inference is effective for reducing the generalization error • We consider Bayesian inference framework – Bayesian inference for NMF has been proposed [Cemgil, 2009] Rem: ← is only discrete 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 16
- 17. RLCT of NMF is unknown • NMF has been used for ``data → knowledge’’ • Mathematical property is unknown – Learning theory has not been established – Prediction accuracy has not been clarified ↑ means that the RLCT of NMF has not been clarified 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 17
- 18. Def. RLCT • RLCT is characterized as a learning coefficient • It is defined by the largest pole of the following complex function: 𝜻 𝒛 = න𝑲 𝒘 𝒛 𝝋 𝒘 𝒅𝒘, where 𝑲 is KL-divergence from true distribution to learning machine and 𝝋 is prior. • A statistical model selection method that uses RLCTs has been proposed [Drton, et al. 2017] 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 18
- 19. Def. RLCT • RLCT is characterized as a learning coefficient • It is defined by the largest pole of the following complex function: 𝜻 𝒛 = න𝑲 𝒘 𝒛 𝝋 𝒘 𝒅𝒘, where 𝑲 is KL-divergence from true distribution to learning machine and 𝝋 is prior. • A statistical model selection method that uses RLCTs has been proposed [Drton, et al. 2017] 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 19 known as sBIC (singular BIC)
- 20. Index • Introduction – Non-negative Matrix Factorization – Real Log Canonical Threshold – Research Goal • Main Theorem • Discussion • Conclusion • (Appendix: Sktech of Proof) 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 20
- 21. Research Goal • Constructing learning theory of NMF →focus theoretical generalization error →focus RLCT of NMF • Recently, we derived an upper bound of RLCT [Hayashi, et. al. 2017] • We used algebraic geometrical method (singularity resolution) 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 21
- 22. Research Goal • Constructing learning theory of NMF →focus theoretical generalization error →focus RLCT of NMF • In this research, we newly derive the exact value of the RLCT of NMF in the case rank ≦ 2 • Using the above exact value, we make the upper bound tighter than previous one 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 22
- 23. 2. MAIN THEOREM 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 23
- 24. Index • Introduction • Main Theorem – Bayesian Framework of NMF – Main Result • Discussion • Conclusion • (Appendix: Sketch of Proof) 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 24
- 25. Formalizing and Setting • Data matrices: 𝑾 𝒏 = 𝑾 𝟏, … , 𝑾 𝒏 ; 𝑴 × 𝑵(× 𝒏) – For general, we treat not only n=1 but also n>1. 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 25 [Kohjima et al. 2016/6, modified] 𝑾𝑴 𝑵
- 26. Formalizing and Setting • Data matrices: 𝑾 𝒏 = 𝑾 𝟏, … , 𝑾 𝒏 ; 𝑴 × 𝑵(× 𝒏) – For general, we treat not only n=1 but also n>1. • True factorization: 𝑨; 𝑴 × 𝑯 𝟎, 𝑩; 𝑯 𝟎 × 𝑵 • Learner factorization: 𝑿; 𝑴 × 𝑯, 𝒀; 𝑯 × 𝑵 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 26 [Kohjima et al. 2016/6, modified] 𝑾𝑴 𝑵 𝑯 𝟎 𝑵 𝑯 𝟎 𝑴 𝑨 𝑩
- 27. Formalizing and Setting • Data matrices: 𝑾 𝒏 = 𝑾 𝟏, … , 𝑾 𝒏 ; 𝑴 × 𝑵(× 𝒏) – For general, we treat not only n=1 but also n>1. • True factorization: 𝑨; 𝑴 × 𝑯 𝟎, 𝑩; 𝑯 𝟎 × 𝑵 • Learner factorization: 𝑿; 𝑴 × 𝑯, 𝒀; 𝑯 × 𝑵 • What is the Bayesian framework of ↑? 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 27 [Kohjima et al. 2016/6, modified] 𝑾𝑴 𝑵 𝑯 𝟎 𝑵 𝑯 𝟎 𝑴 𝑨 𝑩
- 28. • Notation of probability density function (PDF) – 𝑞 𝑊 : true distribution, – 𝑝 𝑊 𝑋, 𝑌 : learning machine, – 𝑝∗ 𝑊 : predictive distribution, whose domains are Euclidian sp. – 𝜑 𝑋, 𝑌 : prior distribution, – 𝑝 𝑋, 𝑌 𝑊 𝑛 : posterior distribution given data, whose domains are compact subsets of Euclidian sp. 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 28 Formalizing and Setting data parameter
- 29. • Assume 𝒒 𝑾 ∝ 𝐞𝐱𝐩 − 𝟏 𝟐 𝑾 − 𝑨𝑩 𝟐 , 𝒑 𝑾 𝑿, 𝒀 ∝ 𝐞𝐱𝐩 − 𝟏 𝟐 𝑾 − 𝑿𝒀 𝟐 , and prior 𝝋 is strictly positive and bounded in a neighborhood of 𝑨, 𝑩 . • Remark: Poisson and exponential dist. can be also applied [Hayashi, et al. 2017]. 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 29 Formalizing and Setting
- 30. Bayesian Framework • The posterior is defined by 𝒑 𝑿, 𝒀 𝑾 𝒏 = 𝟏 𝒁 𝒏 ෑ 𝒊=𝟏 𝒏 𝒑 𝑾𝒊 𝑿, 𝒀 𝝋 𝑿, 𝒀 where 𝒁 𝒏 is normalizing constant. • The predictive distribution is defined by 𝒑∗ 𝑾 = න𝒑 𝑾 𝑿, 𝒀 𝒑 𝑿, 𝒀 𝑾 𝒏 )𝒅𝑿𝒅𝒀 . 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 30
- 31. Bayesian Framework • The Bayesian generalization error is defined by KL-divergence from true to predictive dist. : 𝑮 𝒏 = න 𝒒 𝑾 𝐥𝐨𝐠 𝒒 𝑾 𝒑∗ 𝑾 𝒅𝑾. • This depends on the training data thus it is a random variable. • Its expected value among the overall data has an asymptotic behavior: 𝔼 𝑮 𝒏 = 𝝀 𝒏 + 𝒐 𝟏 𝒏 . 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 31
- 32. Index • Introduction • Main Theorem – Bayesian Framework of NMF – Main Result • Discussion • Conclusion • (Appendix: Sketch of Proof) 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 32
- 33. Def. RLCT of NMF • The RLCT of NMF is defined by the minus maximum pole of the following zeta function: 𝜻 𝒛 = ඵ 𝑿𝒀 − 𝑨𝑩 𝟐 𝒛 𝒅𝑿𝒅𝒀 . 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 33
- 34. • The RLCT of NMF is defined by the minus maximum pole of the following zeta function: 𝜻 𝒛 = ඵ 𝑿𝒀 − 𝑨𝑩 𝟐 𝒛 𝒅𝑿𝒅𝒀 . • 𝜻 𝒛 can be analytically continued to the entire complex plane and its poles are negative rational numbers. • The largest pole of 𝜻 𝒛 equals −𝝀 . Then, 𝝀 is the RLCT of NMF. 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 34 Def. RLCT of NMF
- 35. • The RLCT of NMF is defined by the minus maximum pole of the following zeta function: 𝜻 𝒛 = ඵ 𝑿𝒀 − 𝑨𝑩 𝟐 𝒛 𝒅𝑿𝒅𝒀 . • 𝜻 𝒛 can be analytically continued to the entire complex plane and its poles are negative rational numbers. • The largest pole of 𝜻 𝒛 equals −𝝀 . Then, 𝝀 is the RLCT of NMF. 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 35 𝐎 𝐗 𝐗 𝐗 𝐗 𝐗 𝒛 = −𝝀 Def. RLCT of NMF ℂ
- 36. Main Theorem • The RLCT of NMF 𝝀 satisfies the following inequality: 𝝀 ≤ 𝟏 𝟐 𝑯 − 𝑯 𝟎 𝐦𝐢𝐧 𝑴, 𝑵 + 𝑯 𝟎 𝑴 + 𝑵 − 𝟐 + 𝜹 𝑯 𝟎 , where 𝜹 𝑯 𝟎 = ቊ 𝟏 (𝑯 𝟎 ≅ 𝟏, 𝐦𝐨𝐝 𝟐) 𝟎 (𝒐𝒕𝒉𝒆𝒓𝒘𝒊𝒔𝒆) . The equality holds if 𝑯 = 𝑯 𝟎 = 𝟏 𝐨𝐫 𝟐 or 𝑯 𝟎 = 𝟎. 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 36
- 37. 3. DISCUSSION 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 37
- 38. Index • Introduction • Main Theorem • Discussion – Tightness – Theoretical Application – Numerical Experiment and Conjecture • Conclusion • (Appendix: Sketch of Proof) 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 38
- 39. Tighter than previous • Main Theorem shows an upper bound of the RLCT of NMF. • We have derived another bound of it in previous research. • How tight is the new bound? 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 39
- 40. Tighter than previous • In previous work, 𝝀 ≤ 𝝀 𝒑𝒓𝒗 = 𝟏 𝟐 𝑯 − 𝑯 𝟎 𝐦𝐢𝐧 𝑴, 𝑵 + 𝑯 𝟎 𝑴 + 𝑵 − 𝟏 . • In this paper, 𝝀 ≤ 𝝀 𝒏𝒆𝒘 = 𝟏 𝟐 𝑯 − 𝑯 𝟎 𝐦𝐢𝐧 𝑴, 𝑵 + 𝑯 𝟎 𝑴 + 𝑵 − 𝟐 + 𝜹 𝑯 𝟎 . • By comparing them, we improve the bound 𝝀 𝒏𝒆𝒘 − 𝝀 𝒑𝒓𝒗 = 𝟏 𝟐 𝑯 𝟎 − 𝜹 𝑯 𝟎 . • True dist. is more complex, bound is tighter. 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 40
- 41. Index • Introduction • Main Theorem • Discussion – Tightness – Theoretical Application – Numerical Experiment and Conjecture • Conclusion • (Appendix: Sketch of Proof) 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 41
- 42. Bound of Error • Main Theorem shows an upper bound of the Bayesian generalization error via 𝔼 𝑮 𝒏 = 𝝀 𝒏 + 𝒐 𝟏 𝒏 . • Actually, we have 𝔼 𝑮 𝒏 ≤ 𝟏 𝟐𝒏 𝑯 − 𝑯 𝟎 𝐦𝐢𝐧 𝑴, 𝑵 + 𝑯 𝟎 𝑴 + 𝑵 − 𝟐 + 𝜹 𝑯 𝟎 + 𝒐 𝟏 𝒏 . – This gives guarantee of accuracy! • What distribution can we bound the error? 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 42
- 43. Robustness on Dist. • Main Theorem assumes that elements of parameter matrices are subject to normal distribution: 𝒒 𝑾 ∝ 𝓝 𝑾 𝑨𝑩 , 𝒑 𝑾 𝑿, 𝒀 ∝ 𝓝 𝑾 𝑿𝒀 . • Can Main Theorem be used even for other distributions? 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 43
- 44. Robustness on Dist. • In the prior work [Hayashi, et. al. 2017], we proved that same zeta function can be applied to Poisson and exponential distribution: 𝒒 𝑾 ∝ 𝐏𝐨𝐢 𝑾 𝑨𝑩 , 𝒑 𝑾 𝑿, 𝒀 ∝ 𝐏𝐨𝐢 𝑾 𝑿𝒀 , 𝒒 𝑾 ∝ 𝐄𝐱𝐩𝐨 𝑾 𝑨𝑩 , 𝒑 𝑾 𝑿, 𝒀 ∝ 𝐄𝐱𝐩𝐨 𝑾 𝑿𝒀 , 𝜻 𝒛 = ඵ 𝑿𝒀 − 𝑨𝑩 𝟐 𝒛 𝒅𝑿𝒅𝒀 . 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 44 Even if We can use
- 45. Robustness on Dist. • The above result is derived by the fact that I-divergence and Itakura Saito-divergence have same RLCT as square error. 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 45 Distribution Normal Poisson Exponential Similarity Sq. error I-divergence IS-divergence 𝜻 𝒛 = ඵ 𝑿𝒀 − 𝑨𝑩 𝟐 𝒛 𝒅𝑿𝒅𝒀 We can use Same RLCTs
- 46. Robustness on Dist. • The above result is derived by the fact that I-divergence and Itakura Saito-divergence have same RLCT as square error. 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 46 Distribution Normal Poisson Exponential Similarity Sq. error I-divergence IS-divergence 𝜻 𝒛 = ඵ 𝑿𝒀 − 𝑨𝑩 𝟐 𝒛 𝒅𝑿𝒅𝒀 We can use Same RLCTs Main Thm. is attained!
- 47. Index • Introduction • Main Theorem • Discussion – Tightness – Theoretical Application – Numerical Experiment and Conjecture • Conclusion • (Appendix: Sketch of Proof) 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 47
- 48. • We carried out experiment to estimate the exact value of RLCT. – (+) to compare with the RLCT of reduced rank regression (RRR); non-restricted matrix factorization. • The posterior cannot be analytically derived. →Markov Chain Monte Carlo(MCMC) – We used Metropolis Hastings method. 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 48 Numerical Experiment
- 49. • We made artificial data and set the following cases: – 1. exact value of RLCT of NMF is known. – 2. exact value is unknown and rank = rank+. – 3. exact value is unknown and rank ≠ rank+. • rank+ : minimal inner dimension of NMF – ↑ is called non-negative rank. – In general, rank+≧ rank holds. – If min{rows, columns} ≦ 3 or rank ≦2, rank =rank+. – There is a non-negative matrix s.t. rank＜rank+. 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 49 Non-negative Rank
- 50. • Sample size n=200 (parameter dimension≦50) • The number of data sets D=100 → we empirically calculated the RLCT using 𝔼 𝑮 𝒏 = 𝝀 𝒏 + 𝒐 𝟏 𝒏 → 𝝀 ≈ 𝒏𝔼 𝑮 𝒏 ≈ 𝒏 𝑫 𝒋=𝟏 𝑫 𝑮 𝒏 𝒋 • MCMC sample size K=1,000 – Burn-in=20,000, thin=20, i.e. sampling iteration is 40,000. • For calculating 𝑮 𝒏, we generated T=20,000 test datas from the true distribution. • Total: 100*(40,000+1,000*20,000) ≈ 𝑶 𝑫𝑲𝑻 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 50 Condition of Experiments
- 51. 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 51 Numerical Result 𝝀 𝑵 𝝀 𝝀 𝑩 𝝀 𝑹 Numerical calculated The exact value in NMF The upper bound in NMF The exact value in RRR r: true rank
- 52. 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 52 Numerical Result 𝝀 𝑵 𝝀 𝝀 𝑩 𝝀 𝑹 Numerical calculated The exact value in NMF The upper bound in NMF The exact value in RRR Numerical results equal theoretical value: 𝝀 𝑵 = 𝝀. Numerical calculation is correct! r: true rank
- 53. 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 53 Numerical Result 𝝀 𝑵 𝝀 𝝀 𝑩 𝝀 𝑹 Numerical calculated The exact value in NMF The upper bound in NMF The exact value in RRR r: true rank If rank = rank+, then numerical results equal RRR case: 𝝀 𝑵 = 𝝀 𝑹. It seems that if rank = rank+, then the RLCT of NMF 𝝀 = 𝝀 𝑹.
- 54. 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 54 Numerical Result 𝝀 𝑵 𝝀 𝝀 𝑩 𝝀 𝑹 Numerical calculated The exact value in NMF The upper bound in NMF The exact value in RRR r: true rank If rank ≠ rank+, then num. results are larger than RRR case: 𝝀 𝑵 > 𝝀 𝑹. If rank ≠ rank+, then the RLCT of NMF is larger than RRR case: 𝝀 𝑵 > 𝝀 𝑹.
- 55. 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 55 ConjectureFrom the paper:
- 56. 4. CONCLUSION 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 56
- 57. Index • Introduction • Main Theorem • Discussion • Conclusion • (Appendix: Sketch of Proof) 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 57
- 58. • (Main contribution) We mathematically improved the upper bound of the RLCT of NMF. – This made the bound of the generalization error in Bayesian NMF tighter. • (Minor contribution) We carried out experiments and suggested conjecture about the exact value of the RLCT: – ･ rank = rank+ ⇒ RLCT of NMF = RLCT of RRR. – ･ rank ≠ rank+ ⇒ RLCT of NMF > RLCT of RRR. 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 58 Conclusion
- 59. APPENDIX: SKETCH OF PROOF FOR MAIN THEOREM 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 59
- 60. • We have already derived the exact value in the case 𝑯 𝟎 = 𝟎 and 𝑯 = 𝑯 𝟎 = 𝟏. • We newly clarified the exact value in the case 𝑯 = 𝑯 𝟎 = 𝟐 by considering the dimension of algebraic subvariety in the parameter space. • We bound the RLCT in the case 𝑯 = 𝑯 𝟎 by using the clarified exact value mentioned above. • We bound the RLCT in general case by using the above results. 2017/11/28 IEEE SSCI 2017 FOCI, Hawaii 60 Sketch of Proof