Physics-driven Spatiotemporal Regularization for High-dimensional Predictive Modeling
2 likes121 views
Rapid advancement of distributed sensing and imaging technology brings the proliferation of high-dimensional spatiotemporal data, i.e., y(s; t) and x(s; t) in manufacturing and healthcare systems. Traditional regression is not generally applicable for predictive modeling in these complex structured systems. For example, infrared cameras are commonly used to capture dynamic thermal images of 3D parts in additive manufacturing. The temperature distribution within parts enables engineers to investigate how process conditions impact the strength, residual stress and microstructures of fabricated products. The ECG sensor network is placed on the body surface to acquire the distribution of electric potentials y(s; t), also named body surface potential mapping (BSPM). Medical scientists call for the estimation of electric potentials x(s; t) on the heart surface from BSPM y(s; t) so as to investigate cardiac pathological activities (e.g., tissue damages in the heart). However, spatiotemporally varying data and complex geometries (e.g., human heart or mechanical parts) defy traditional regression modeling and regularization methods. This talk will present a novel physics-driven spatiotemporal regularization (STRE) method for high-dimensional predictive modeling in complex manufacturing and healthcare systems. This model not only captures the physics-based interrelationship between time-varying explanatory and response variables that are distributed in the space, but also addresses the spatial and temporal regularizations to improve the prediction performance. In the end, we will introduce our lab at Penn State and future research directions will also be discussed.
![Introduction Methodology Experiments References
Experiments - Simulation in a Two-sphere Geometry
Dynamic distributions of electric potentials on the inner surface
x(s, t) and outer surface y(s, t) are calculated analytically
x(s, t) =
1
4πσ
p(t) · rH (s)
r2
BrH
[
2rH
rB
+ (
rB
rH
)2
]
y(s, t) =
3
4πσ
p(t) · rB(s)
r3
B
Gaussian noise ∼ N(0, σ2) is added to y(s, t)
(a) (b)
Figure: (a) Parameters of the two-sphere geometry; (b) Each sphere is triangulated
with 184 nodes and 364 triangles
Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 30 / 42](https://image.slidesharecdn.com/stregularizationhy-171126033426/85/Physics-driven-Spatiotemporal-Regularization-for-High-dimensional-Predictive-Modeling-30-320.jpg)
More Related Content
What's hot (20)
Similar to Physics-driven Spatiotemporal Regularization for High-dimensional Predictive Modeling (19)
Recently uploaded (20)
Physics-driven Spatiotemporal Regularization for High-dimensional Predictive Modeling
- 1. Physics-driven Spatiotemporal Regularization for High-dimensional Predictive Modeling Bing Yao and Hui Yang 杨 徽 Associate Professor Complex Systems Monitoring, Modeling and Control Lab The Pennsylvania State University University Park, PA 16802 November 25, 2017
- 2. Outline 1 Introduction 2 Physics-driven Spatiotemporal Regularization 3 Experimental Design and Results 4 References
- 3. Introduction Methodology Experiments References Research Roadmap Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 3 / 42
- 4. Introduction Methodology Experiments References Research Projects Manufacturing Processes, Precision Machining Publications: Pattern Recognition, IEEE Transactions, Chaos, IIE/ASME Transactions, J. Mfg Sys Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 4 / 42
- 5. Introduction Methodology Experiments References Research Projects Electro-mechanical Processes, Biomanufacturing Publications: IEEE Transactions, Physical Review, Physiological Measurements, Scientific Reports Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 5 / 42
- 6. Introduction Methodology Experiments References Predictive Modeling Shrinkage methods: ridge regression, LASSO, and elastic net ... Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 6 / 42
- 7. Introduction Methodology Experiments References Advanced Sensing 108.5°C <60.0°C CInfrared Camera Thermal Image Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 7 / 42
- 8. Introduction Methodology Experiments References Advanced Sensing (*from Yoram Rudy Lab @ WUSTL) Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 8 / 42
- 9. Introduction Methodology Experiments References High-dimensional Predictive Modeling BSPM y(s,t) Heart-surface Potential Mapping x(s,t) Inverse Forward Y (s, t) = RX(s, t) + Traditional regression is not generally applicable! Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 9 / 42
- 10. Introduction Methodology Experiments References Challenges Spatially-temporally big data Dimensionality Velocity - sampling in milliseconds Veracity - data uncertainty Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 10 / 42
- 11. Introduction Methodology Experiments References Challenges Complex structured systems Complex geometries of AM builds Complex torso-heart geometry (*from CIMP-3D @ PSU) Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 11 / 42
- 12. Introduction Methodology Experiments References Challenges Y (s, t) = RX(s, t) + Outer surface profiles y(s, t) ⇒ Inner surface profiles x(s, t) Transfer matrix R ? Physical principles Additive manufacturing - Heat transfer model Heart - Electrical wave propagation Ill-conditioned system Linear systems involving high-dimensional data Condition number: cond(R) = R R−1 A measure of the relative sensitivity of the solution to changes in y ∆x x cond(R) ∆y y Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 12 / 42
- 13. Introduction Methodology Experiments References State of the Art Tikhonov regularization min x(s,t) { y(s, t) − Rx(s, t) 2 2 + λ2 Γx(s, t) 2 2} L1 regularization min x(s,t) { y(s, t) − Rx(s, t) 2 2 + λ2 Γx(s, t) 1} Zeroth-order Γ = I Directly penalize the magnitude of x(s, t) Sparsity vs. Regularity Not account for spatial or temporal correlations Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 13 / 42
- 14. Introduction Methodology Experiments References State of the Art First-order Regularization The first-order derivative: Γx(s, t) = ∂x(s, t)/∂τ Align x(s, t) in one column as {x(s1|t), x(s2|t), ..., x(sN |t)}T Apply the bidiagonal gradient matrix Normal derivative operator: Γx(s, t) = ∂x(s, t)/∂n Γ = −1 1 −1 1 ... ... −1 1 n τ Γx(s,t) Tangent plane Need to fill the gaps Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 14 / 42
- 15. Introduction Methodology Experiments References Physics-driven Spatiotemporal Regularization Objective function min x(s,t) T t=1 { y(s, t)−Rx(s, t) 2 +λ2 s ∆sx(s, t) 2 +λ2 t t+ w 2 τ=t− w 2 x(s, t)−x(s, τ) 2 } Parameter Matrix R - physics-based interrelationship Spatial regularity - handle approximation errors by spatial correlation Temporal regularity - model robustness to measurement noises Algorithm - generalized dipole multiplicative update method Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 15 / 42
- 16. Introduction Methodology Experiments References Parameter Matrix R Divergence theorem: if F is a vector field which is continuously differentiable and defined on a volume V ⊂ R3 with a piecewise-smooth boundary S, then V ( · F )dV = S (F · n)dS Electric Field Body Surface SB Heart Surface SH Green’s second identity: If φ and ψ are twice continuously differentiable on V , and let F = φ ψ − ψ φ, then S (φ ψ − ψ φ) · ndS = V (φ 2 ψ − ψ 2 φ)dV Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 16 / 42
- 17. Introduction Methodology Experiments References Parameter Matrix R Heart - a bioelectric source Torso - a homogeneous and isotropic volume conductor (*from marvel.com) Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 17 / 42
- 18. Introduction Methodology Experiments References Parameter Matrix R Heart - a bioelectric source Torso - a homogeneous and isotropic volume conductor Green’s second identity: S (φ ψ − ψ φ) · ndS = V (φ 2 ψ − ψ 2 φ)dV ψ = 1/r; φ = electric potentials No electrical source between SH and SB: 2 φ = 0 Electric field outside SB is negligible: φ = 0 on SB SH n SB n ∇2 φ = 0 ∇y(s,t)=0 y(s,t) x(s,t) dΩBB dSB dΩBH dSH Heart Surface Body surface Volume conductor Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 18 / 42
- 19. Introduction Methodology Experiments References Parameter Matrix R Boundary element method Body surface potential on SB y(s, t) = − 1 4π SH x(s, t)dΩBH − 1 4π SH x(s, t) · n rBH dSH + 1 4π SB y(s, t)dΩBB Heart surface potential on SH x(s, t) = − 1 4π SH x(s, t)dΩHH − 1 4π SH x(s, t) · n rHH dSH + 1 4π SB y(s, t)dΩHB Numerical integration ABBy(s, t) + ABHx(s, t) + MBHN(s, t) = 0 AHBy(s, t) + AHHx(s, t) + MHHN(s, t) = 0 Parameter matrix R: R = (ABB − MBHM−1 HHAHB)−1 (MBHM−1 HHAHH − ABH) Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 19 / 42
- 20. Introduction Methodology Experiments References Physics-driven Spatiotemporal Regularization Objective function min x(s,t) T t=1 { y(s, t)−Rx(s, t) 2 +λ2 s ∆sx(s, t) 2 +λ2 t t+ w 2 τ=t− w 2 x(s, t)−x(s, τ) 2 } Parameter Matrix R - physics-based interrelationship Spatial regularity - handle approximation errors by spatial correlation Temporal regularity - model robustness to measurement noises Algorithm - generalized dipole multiplicative update method Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 20 / 42
- 21. Introduction Methodology Experiments References Spatial Regularity Surface laplacian ∆s for a square lattice Surface laplacian at the node p0 x1 = x0 + a ∂x ∂v p0 + 1 2 a2 ∂2x ∂v2 p0 x2 = x0 − a ∂x ∂u p0 + 1 2 a2 ∂2x ∂u2 p0 x3 = x0 − a ∂x ∂v p0 + 1 2 a2 ∂2x ∂v2 p0 x4 = x0 + a ∂x ∂u p0 + 1 2 a2 ∂2x ∂u2 p0 ⇒ x1 + x2 + x3 + x4 = 4x0 + a2 ( ∂2x ∂u2 + ∂2x ∂v2 ) p0 = 4x0 + a2 ∆x0 ⇒ ∆x0 = 1 a2 ( 4 i=1 xi − 4x0) = 4 a2 (¯x − x0) Laplacian matrix for the square lattice ∆ij = − 4 a2 , if i = j 1 a2 , if i = j, pj ∈ neighborhood of pi 0, otherwise Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 21 / 42
- 22. Introduction Methodology Experiments References Spatial Regularity Linear interpolation (imaginary nearest neighbor): xt(j) = xt(i) + ¯di dij (xt(j) − xt(i)) dij is the distance between pi and pj ¯di = 1 ni ni j=1 dij ni is the number of neighbors of pi Surface laplacian at the node pi ∆sxt(i) = 4 ¯di 2 ( 1 ni ni j=1 xt(j) − xt(i)) = 4 ¯di ( 1 ni ni j=1 xt(j) dij − ( 1 di )xt(i)) Laplacian matrix for 3D triangle mesh ∆ij = − 4 ¯di ( 1 di ), if i = j 4 ¯di 1 ni 1 dij , if i = j, pj ∈ neighborhood of pi 0, otherwise Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 22 / 42
- 23. Introduction Methodology Experiments References Physics-driven Spatiotemporal Regularization Objective function min x(s,t) T t=1 { y(s, t)−Rx(s, t) 2 +λ2 s ∆sx(s, t) 2 +λ2 t t+ w 2 τ=t− w 2 x(s, t)−x(s, τ) 2 } Parameter Matrix R - physics-based interrelationship Spatial regularity - handle approximation errors by spatial correlation Temporal regularity - model robustness to measurement noises Algorithm - generalized dipole multiplicative update method Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 23 / 42
- 24. Introduction Methodology Experiments References Temporal Regularity Spatiotemporal data x(s, t) and y(s, t) - dynamically evolving over time and have temporal correlations T t=1 t+w 2 τ=t−w 2 x(s, t) − x(s, τ) 2 Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 24 / 42
- 25. Introduction Methodology Experiments References Physics-driven Spatiotemporal Regularization Objective function min x(s,t) T t=1 { y(s, t)−Rx(s, t) 2 +λ2 s ∆sx(s, t) 2 +λ2 t t+ w 2 τ=t− w 2 x(s, t)−x(s, τ) 2 } Parameter Matrix R - physics-based interrelationship Spatial regularity - handle approximation errors by spatial correlation Temporal regularity - model robustness to measurement noises Algorithm - generalized dipole multiplicative update method Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 25 / 42
- 26. Introduction Methodology Experiments References DMU Algorithm Objective function - both spatial and temporal terms, and is difficult to be solved analytically. Iterative algorithm - traditional multiplicative update method requires x(s, t) to be nonnegative Heart surface - negative and positive electric potentials A new dipole multiplicative update algorithm for generalized spatiotemporal regularization xt = x+ t − x− t , x+ t = max{0, xt} x− t = max{0, −xt} Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 26 / 42
- 27. Introduction Methodology Experiments References DMU Algorithm If we define A = A+ − A− = RT R + λ2 s∆T s ∆s + 2λ2 t wI B = yT t R + 2λ2 t t−1 τ=t− w 2 xT τ + 2λ2 t t+ w 2 τ=t+1 xT τ The objective function can be rewritten as: J = T t=1 {xT t Axt − Bxt − xT t BT } = ((xT t )+ )A+ x+ t − ((xT t )+ )Ax− t − ((xT t )− )Ax+ t − ((xT t )+ )A− x+ t +((xT t )− )A+ x− t − ((xT t )− )A− x− t − B(x+ t − x− t ) − (x+ t − x− t )T BT Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 27 / 42
- 28. Introduction Methodology Experiments References DMU Algorithm If we define a+ i = (2A+ x+ t )i a− i = (2A+ x− t )i b+ i = −(2Ax− t )i − 2BT i b− i = −(2Ax+ t )i + 2BT i c+ i = (2A− x+ t )i c− i = (2A− x− t )i New update rules (x+ t )i ← −b+ i + (b+ i )2 + 4a+ i c+ i 2a+ i (x+ t )i (x− t )i ← −b− i + (b− i )2 + 4a− i c− i 2a− i (x− t )i Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 28 / 42
- 29. Introduction Methodology Experiments References DMU Algorithm Table: The Proposed Dipole Multiplicative Update Algorithm for STRE 1: Set constants λs, λt and w. Let A = A+ − A− = RT R + λ2 s∆T s ∆s + 2λ2 t wI B = yT t R + 2λ2 t t−1 τ=t− w 2 xT τ + 2λ2 t t+ w 2 τ=t+1 xT τ 2: Initialize {x+ t } and {x− t } as positive random matrices. 3: Repeat 4: for i = 1, . . . , T do (x+ t )i ← (Ax− t )i+Bi+ ((Ax− t )i+Bi)2+4(A+x+ t )i(A−x+ t )i (2A+x+ t )i (x+ t )i (x− t )i ← (Ax+ t )i−Bi+ ((Ax+ t )i−Bi)2+4(A+x− t )i(A−x− t )i (2A+x− t )i (x− t )i 5: end for 6: until convergence 7: Solution: ˆxt = x+ t − x− t Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 29 / 42
- 30. Introduction Methodology Experiments References Experiments - Simulation in a Two-sphere Geometry Dynamic distributions of electric potentials on the inner surface x(s, t) and outer surface y(s, t) are calculated analytically x(s, t) = 1 4πσ p(t) · rH (s) r2 BrH [ 2rH rB + ( rB rH )2 ] y(s, t) = 3 4πσ p(t) · rB(s) r3 B Gaussian noise ∼ N(0, σ2) is added to y(s, t) (a) (b) Figure: (a) Parameters of the two-sphere geometry; (b) Each sphere is triangulated with 184 nodes and 364 triangles Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 30 / 42
- 31. Introduction Methodology Experiments References Results σε (a) (b) Tikh-0thTikh-1st L1-1st STRE RE 0 0.05 0.1 0.15 0.1 0.2 0.3 0.4 0.5 RE 0.08 0.1 0.12 0.14 0.16 0.18 0.2 Tikh-0th Tikh-1st L1-1st STRE Figure: (a) The comparisons of relative error (RE) between the proposed STRE model and other regularization methods (i.e., Tikhonov zero-order, Tikhonov first-order and L1 first-order methods) in the two-sphere geometry when there is no noise on the potential map y(s, t) of the outer sphere; (b) The comparisons of RE for different noise levels σ = 0.1; 0.2; 0.3; 0.4; 0.5 on the potential map y(s, t) of the outer sphere. Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 31 / 42
- 32. Introduction Methodology Experiments References Results Dynamic distribution of electric potentials on the inner sphere x(s, t) Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 32 / 42
- 33. Introduction Methodology Experiments References Results Potential mapping on the inner sphere x(s, t), t = 150ms Reference Tikh_0th RE=0.1475 Tikh_1st RE=0.1026 L1_1st RE=0.1025 STRE RE=0.006 Tikh_0th RE=0.208 Tikh_1st RE=0.1528 L1_1st RE=0.1569 STRE RE=0.0769 (a) (b) (c) Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 33 / 42
- 34. Introduction Methodology Experiments References Experiments - Realistic Torso-heart Geometry Heart surface - 257 nodes and 510 triangles Body surface - 771 nodes and 1538 triangles y(s, t) - body area sensor network Data uncertainty - gaussian noise ∼ N(0, σ2). Five different noise levels: σ = 0.005, 0.01, 0.05, 0.1, 0.2 (a) (b) Front Back Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 34 / 42
- 35. Introduction Methodology Experiments References Results σε (a) (b) Tikh-0th Tikh-1st L1-1st STRE RE 0 0.05 0.1 0.15 0.2 0.25 0.3 0 0.05 0.1 0.15 0.2 RE 0.5 1 1.5 2 2.5 3 Tikh-0th Tikh-1st L1-1st STRE Figure: (a) The comparisons of relative error (RE) between the proposed STRE model and other regularization methods (i.e., Tikhonov zero-order, Tikhonov first-order and L1 first-order methods) in the realistic torso-heart geometry when there is no extra noise on the potential map y(s, t) of the body surface; (b) The comparisons of RE for different noise levels σ = 0.005; 0.01; 0.05; 0.1; 0.2 on the potential map y(s, t). Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 35 / 42
- 36. Introduction Methodology Experiments References Results Dynamic distribution of electric potentials on the heart surface x(s, t) Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 36 / 42
- 37. Introduction Methodology Experiments References Results Potential mapping on the heart surface x(s, t), t = 50ms Reference Tikh_0th RE=0.2488 Tikh_1st RE=0.2839 L1_1st RE=0.2735 STRE RE=0.0997 STRE RE=0.2386 Tikh_0th RE=0.557 Tikh_1st RE=0.972 L1_1st RE=1.248 (a) (b) (c) Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 37 / 42
- 38. Introduction Methodology Experiments References Summary Challenges Spatiotemporal data (predictor and response variables) Complex-structured system Ill-conditioned system Methodology: Physics-driven spatiotemporal regularization Parameter Matrix R - physics-based interrelationship Spatial regularity - handle approximation errors by spatial correlation Temporal regularity - model robustness to measurement noises Algorithm - generalized dipole multiplicative update method Significance A novel approach to solve ECG inverse problem A new dipole multiplicative update algorithm for generalized spatiotemporal regularization Broad applications: thermal effects in additive manufacturing Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 38 / 42
- 39. Introduction Methodology Experiments References References B. Yao, R. Zhu, and H. Yang*, “Characterizing the Location and Extent of Myocardial Infarctions with Inverse ECG Modeling and Spatiotemporal Regularization,”IEEE Journal of Biomedical and Health Informatics, page 1-11, 2017, DOI: 10.1109/JBHI.2017.2768534 B. Yao and H. Yang*, “Physics-driven spatiotemporal regularization for high-dimensional predictive modeling,”Scientific Reports 6, 39012, 2016. DOI: www.nature.com/articles/srep39012 B, Yao and H. Yang*, “Mesh Resolution Impacts the Accuracy of Inverse and Forward ECG problems,”Proceedings of 2016 IEEE Engineering in Medicine and Biology Society Conference (EMBC), August 16-20, 2016, Orlando, FL, United States. DOI: 10.1109/EMBC.2016.7591615 Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 39 / 42
- 40. Introduction Methodology Experiments References Acknowledgements NSF CAREER Award NSF CMMI-1617148 NSF CMMI-1646660 NSF CMMI-1619648 NSF IIP-1447289 NSF IOS-1146882 James A. Haley Veterans’ Hospital Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 40 / 42
- 41. Introduction Methodology Experiments References Contact Information Hui Yang, PhD Associate Professor Complex Systems Monitoring Modeling and Control Laboratory Harold and Inge Marcus Department of Industrial and Manufacturing Engineering The Pennsylvania State University Tel: (814) 865-7397 Fax: (814) 863-4745 Email: [email protected] Web: http://www.personal.psu.edu/huy25/ Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 41 / 42
- 42. Introduction Methodology Experiments References Questions? Hui Yang (PSU) Spatiotemporal Regularization November 25, 2017 42 / 42