COMPUTATIONAL ENGINEERING

TA C K L I N G T U R B U L E N C E W I T H ( S U P E R ) C O M P U T E R S
A BRIEF OVERVIEW OF CFD
IN A NUTSHELL
REMINDER: NS EQUATIONS

∂⃗
u 2 Newton's 2nd law
+( ⃗
u⋅∇ ) ⃗
u =ν ∇ u⃗ −∇ p
∂t

∇⋅⃗
u =0 Mass conservation

Mathematical Fundamentals for CFD 3/30

REMINDER: NS EQUATIONS

∂⃗
u 2 Newton's 2nd law
+( ⃗
u⋅∇ ) ⃗
u =ν ∇ u⃗ −∇ p
∂t

∇⋅⃗
u =0 Mass conservation

Mathematical Fundamentals for CFD 4/30

THREE PILARS OF CFD

CFD (Computational Fluid Dynamics)

Turbulence modelling
We will study this
during this course

Numerical Methods

We will NOT study

HPC (High Performance Computing) this. Just a very
brief overview
(today)

Mathematical Fundamentals for CFD 5/30

HIGH PERFORMANCE COMPUTING (HPC)
Or how to use efficiently modern (super)computers

MareNostrum4 supercomputer
Picture downloaded from https://www.bsc.es/marenostrum/marenostrum (accessed February 27, 2019).

Mathematical Fundamentals for CFD 6/30

HIGH PERFORMANCE COMPUTING (HPC)
Strong speed-up (problem size remains constant)

Figure extracted from A. Gorobets, F.X. Trias, R. Borrell, O. Lehmkuhl, and A. Oliva. “Hybrid MPI+OpenMP parallelization
of an FFT-based 3D Poisson solver with one periodic direction”. Computers & Fluids, 49:101–109, 2011.

Mathematical Fundamentals for CFD 7/30

HIGH PERFORMANCE COMPUTING (HPC)
Weak speed-up (load per CPU remains constant)

Figure extracted from A.Gorobets, F.X. Trias, M.Soria and A.Oliva, “A scalable parallel Poisson solver for three-
dimensional problems with one periodic direction”, Computers and Fluids, 39:525–538, 2010.

Mathematical Fundamentals for CFD 8/30

HIGH PERFORMANCE COMPUTING (HPC)

Mathematical Fundamentals for CFD 9/30

HIGH PERFORMANCE COMPUTING (HPC)

Mathematical Fundamentals for CFD 10/30

HIGH PERFORMANCE COMPUTING (HPC)
Why modern supercomputers are so inefficient for many applications?

https://www.top500.org/lists/2019/11/ (accessed February 18, 2020). https://www.hpcg-benchmark.org/custom/index.html?lid=155&slid=302 (accessed

February 18, 2020).

Mathematical Fundamentals for CFD 11/30

HIGH PERFORMANCE COMPUTING (HPC)
Why modern supercomputers are so inefficient for many applications?
Because we cannot feed the beast!!!
Roofline model
Not enough bandwidth (BW)
Floating point performance (GFLOP/s)

Peak performance Compute bound

nd
ou
yb
or
em
M

BW × AI

Arithmetic Intensity (FLOP/byte)

Mathematical Fundamentals for CFD 12/30
HIGH PERFORMANCE COMPUTING (HPC)
Why modern supercomputers are so inefficient for many applications?
Because we cannot feed the beast!!!
spmv: sparse matrix-vector product
unry: unary vector operation (e.g. scaling)
Not enough bandwidth (BW)
bnry: binary vector operation (e.g. linear combination of vectors)
sred: scalar reduction (e.g. dot product)

Mathematical Fundamentals for CFD 13/30

THREE PILARS OF CFD

CFD (Computational Fluid Dynamics)

Basic theory (K41)

Turbulence modelling (RANS, LES or DNS)

(FVM, FDM or FEM)

Numerical schemes
Numerical Methods (spatial and
temporal)...

HPC (High Performance Computing) DONE!

Mathematical Fundamentals for CFD 14/30

TURBULENCE MODELING: PREVIEW OF IDEAS

Mathematical Fundamentals for CFD 15/30

ERRORS, STABILITY AND CODE
VERIFICATION
TA M I N G T H E B E A S T

   
“Programming is thinking, not typing.”—Casey Patton
ERROR LANDSCAPE

Validation Verification
Solve the right equations Solve the equations right

∂φ ∂2 φ d φh
+Γ =f + D φ h=f h A x=b ⇒ x≈ A
−1
b
∂t ∂x
2
dt

Numerical
Physical realm Math model Discrete eqs.
solution

Modelization Discretization Solver residual

errors errors Round-off errors
● Local truncation error
● Global truncation error
● Dispersion error
● Aliasing error

Mathematical Fundamentals for CFD 17/30

 CLASSICS: LOCAL TRUNCATION ERROR

∂φ
∂x | xi

Taylor series expansion around x i :

φi =φ i
hi hi +1 h 2i+1 h3i+1
φi +1=φ i +hi +1 ∂ x φi + ∂ φ+ ∂ φ +...
2 xx i 3 ! xxx i
x i−2 x i−1 xi x i+1 x i+ 2 2
hi
3
hi
φi −1=φi −hi ∂ x φ i + ∂ xx φ i− ∂ xxx φ i +...
2 3!

∂φ
|
∂x x
=
φ i+ 1−φ i−1
hi +hi +1
i
1
+O (h )
2 2 2 2

|
∂φ h φ i i+ 1+(h i+1−hi ) φ i −h i+1 φ i−1 2
= +O (h )
∂x xi hi hi +1 (hi + hi+1 )

Mathematical Fundamentals for CFD 18/30

 CLASSICS: LOCAL TRUNCATION ERROR

∂2 φ
∂ x2 | xi

Taylor series expansion around x i :

φi =φ i
hi hi +1 h 2i+1 h3i+1
φi +1=φ i +hi +1 ∂ x φi + ∂ φ+ ∂ φ +...
2 xx i 3 ! xxx i
x i−2 x i−1 xi x i+1 x i+ 2 2
hi
3
hi
φi −1=φi −hi ∂ x φ i + ∂ xx φ i− ∂ xxx φ i +...
2 3!

( )( ) ( ) ( ) ( )
1 1 ci −1
1 0 ci −1 h i+1
1
−h i 0 hi +1
ci = 0 ⇒ ci = −(hi + hi+1 )
2 2 hi hi+1 ( hi +hi +1)
h i−1 / 2 0 h i+1 / 2 c i +1 1 c i +1 hi

|
hi φ i+ 1−( hi+ 1+ hi ) φi + hi+1 φ i−1
2
∂ φ 2
= +O (h )
2
∂x x i
hi hi+1 ( hi +hi +1)

Mathematical Fundamentals for CFD 19/30

 GOING TO HIGHER ORDER...

∂2 φ
∂ x2 |
xi
...using Richardson
extrapolation

Taylor series expansion around x i :

Hi H i+1 φi =φ i
hi hi +1 h 2i+1 h3i+1
φi +1=φ i +hi +1 ∂ x φi + ∂ φ+ ∂ φ +...
2 xx i 3 ! xxx i
x i−2 x i−1 xi x i+1 x i+ 2 2
hi
3
hi
φi −1=φi −hi ∂ x φ i + ∂ xx φ i− ∂ xxx φ i +...
2 3!
Local truncation error

φ i+ 1−2 φ i + φi −1 ⏞
|
2 2
∂ φ h 4
2
= 2
+ ∂ φ
xxxx i +O (h ) × 4/3
∂x x h 4!
i

|
2
∂2 φ φ i+ 2−2 φ i + φ i−2 ( 2 h) 4
= + ∂ xxxx φ i +O (h ) ×−1/ 3
2
∂x x i
H
2
4!
4
O ( h ) approximation
Mathematical Fundamentals for CFD 20/30
CODE VERIFICATION...

…using the Method of Manufactured Solutions (MMS)

manufactured solution!

Extracted from F.X. Trias, A. Gorobets and A. Oliva. "A simple approach to discretize the viscous term with spatially varying
(eddy-)viscosity", Journal of Computational Physics, 253 (1): 405-417, 2013.

Mathematical Fundamentals for CFD 21/30

 STABILITY

A simulation might blow up, but you MUST know why

Preview of ideas:
n +1 n
φ =a φ is stable if |a|≤1

Stable Marginally Unstable

n
stable n
φn φ φ
|a|<1 |a|=1 |a|>1

n n n
Mathematical Fundamentals for CFD 22/30
 STABILITY

So, at the end, we just need to know the spectral radius

of our transfer function
n +1 n
φ =f ( φ ) is stable if |f |=|λmax|≤1
n +1 n
φ =a φ is stable if |a|≤1
n +1 n
φ h = A φ h is stable if |A|=|λ max|≤1

Hence, it is basically an eigenvalue problem.

Nothing more, nothing else!

Mathematical Fundamentals for CFD 23/30

STABILITY

Most typical example is explicit time-integration

First-order Euler scheme φ n+ 1−φ n
φ t =λ φ =λ φ n
Δt
transfer function
n +1 n
φ =(1+ λ Δ t ) φ
I
~
λ≡λ Δ t +1

−2 ℝ

−1
|1+~
λ|≤1 Stability region

Mathematical Fundamentals for CFD 24/30

STABILITY

Most typical example is explicit time-integration

2nd-order Adams-Bashforth φ n+ 1−φ n 3 n 1 n−1
φ t =λ φ =λ ( φ − φ )
Δt 2 2

( )( )( )
φ n+1 ~ ~ φn
1+3 λ / 2 − λ / 2 ~
n = λ≡λ Δ t
φ 1 0 φ n−1
transfer function

Figure 1 in F.X. Trias and O. Lehmkuhl. “A self-adaptive

strategy for the time-integration of Navier-Stokes equations”.
Numerical Heat Transfer, part B, 60(2):116–134, 2011.

Mathematical Fundamentals for CFD 25/30

 STABILITY

So, what is CFL? A very popular (inaccurate) way to do it

Δ t ≤C conv ( )
Δx
U min
Δ t ≤C diff ( Δ x2
ν )
min

1
What is Δ x and U in an unstructured mesh? 5 4
ANSWER: This is the typical "1D uniform mesh"
6 2
idea applied to "3D unstruct"
d uh 3
Ω +C ( uh )u h=D u h−Ω G ph M u h=0h
dt

−C (uh )+ D
Instead we can compute the
spectral radius of this operator

Mathematical Fundamentals for CFD 26/30

BONUS: LINEAR SOLVERS

sol −1 sol
A x=b x ≈A b where ‖A x −b‖≤ε
ini
A x ε b x sol

Easy to test!
A
rand rand sol sol rand
x b= A x x x ≈x
ε

Mathematical Fundamentals for CFD 27/30

BONUS: LINEAR SOLVERS

sol −1 sol
A x=b x ≈A b where ‖A x −b‖≤ε

How to choose a solver?

What type of matrix/problem do I have?
T
dense or sparse? A= A ? diagonal dominant?...
circulant? block diagonal? condition number?...
A x i =bi , i=1. .. N ; Is N large or small?
What is ε ?
do I need machine accuracy or ε=0.001 is enough?
ini
How are x and b ?
ini sol
is x a good guess of x ? what is the spectrum of b ?
How is my computer?
parallel? memory shared or distributed?
Mathematical Fundamentals for CFD 28/30
 BONUS: LINEAR SOLVERS

Typical linear solvers in CFD: To know this

Diagonalizable: is a MUST!
Direct e.g. circulant matrix (easily solved with a FFT )
−1 −1
LU-based: A x =LU x=b ⇒ y=L b ⇒ x =U y
T
e.g. TDMA, band-LU, Cholesky (if A= A ),
Schur complement (for block/partitioned A )
Stationary: A=~ A+( A−~ A) ⇒ ~ A x i +1=b−( A−~ A) x i
Iterative e.g. Jacobi ( ~
A=D ), Gauss-Seidel ( ~ A=D + L ) ,.. .
T
Non-stationary: e.g. Krylov subspace (CG if A= A ),
Multigrid solvers:
Smoother (e.g. Jacobi) easily solves high freqs
Idea: transform low freqs into high freqs ( restriction )
solve them and come back ( prolongation )

29/30
Take-away messages

● Navier-Stokes equations is an excellent mathematical

model of turbulence.
● CFD is based on three pilars: Turbulence modelling

– Turbulence modelling Numerical Methods

– Numerical methods
HPC (High Performance Computing)
– HPC
● Codes must be well-structured, efficient, simple, readable
and modular.

Floating point performance (GFLOP/s)

Peak performance Compute bound

nd
ou
yb
or
CFD codes are memory bound
em
●

M
BW × AI

Arithmetic Intensity (FLOP/byte)

Mathematical Fundamentals for CFD 30/30