May 16 - 20, 2016

On the Application of a Multivariate

Metric for Multiple Set Points in
Validation Exercises of
RANS solvers

L. Eça (IST-Lisbon, MARIN Academy)

R. Lopes (IST-Lisbon)
H.Abreu (IST-Lisbon)
F.Pereira (MARIN Academy, IST-Lisbon)
G.Vaz (MARIN)

May 16 - 20, 2016

 May 16 - 20, 2016

Contents
1. Introduction
2. ASME V&V 20 Validation Metrics
3. Multivariate Metric
4. Examples
5. Conclusions

May 16 - 20, 2016

 May 16 - 20, 2016

1. Introduction
• Validation exercises of mathematical models
solved by CFD are often performed for
functional quantitions as for example force
coefficients or local quantities
• For such cases, ASME V&V 20 proposed a
Validation procedure that estimates an interval
that contains the modelling error 95 out of 100
times

May 16 - 20, 2016

 May 16 - 20, 2016

1. Introduction
• There are several cases where modelling errors
are evaluated at multiple set points, as for
example lift coefficient of an airfoil for a range of
angles of attack, airfoil surface pressure
distribution or velocity field in the wake of a ship
• For such cases, ASME V&V 20 is proposing
a “Multivariate Metric” using results from
Multiple Validation Set Points

May 16 - 20, 2016

 May 16 - 20, 2016

1. Introduction
• This presentation includes several examples of
application of the Multivariate Metric
- Flow over a flat plate
- Flow around the NACA 0012 airfoil
- Flow around the Eppler 387 airfoil
- Flow around a circular cylinder
- Flow around the KVLCC2 tanker

• All calculations are performed with

(www.refresco.org)

May 16 - 20, 2016

 May 16 - 20, 2016

2. ASME V&V 20 Validation Metric

• ASME V&V 20-2009 Validation procedure is
based on the comparison of two quantities
- Comparison error E =S−D
- Validation uncertainty U val = U D2 + U input
2 2
+ U num

UD Experimental uncertainty
U num Numerical uncertainty
U input Parameter uncertainty

May 16 - 20, 2016

 May 16 - 20, 2016

2. ASME V&V 20 Validation Metric

• Estimated interval that contains the modelling
errorδ model is given by
E − U val ≤ δ model ≤ E + U val
• It is not a pass/fail method! For example,
E = 2% 
 ⇒ 1% ≤ δ model ≤ 3%
U val = 1%
E = 4% 
 ⇒ −6% ≤ δ model ≤ 14%
U val = 10%

May 16 - 20, 2016

 May 16 - 20, 2016

3. Multivariate Metric
T −1
• Vector of weighted differences r = r Vval r
based on the vector of differences at
n set points
 S1 − D1 
r=

 
 Sn − Dn 
and covariance matrix
Vval = VD + Vinput + Vnum

May 16 - 20, 2016

 May 16 - 20, 2016

3. Multivariate Metric
• Covariance matrix depends on numerical
uncertainties U num
2
0 ... 0 
 1

Vnum = 0 ...U numi ...0 
2

 
0 ...
2
U numn  U D21 0 ... 0 
 
experimental uncertainties VD = 0 ...U Di ...0 
2

 2

0 ... U Dn 

and parameter uncertainties that are assumed

to be zero in the present exercises Vinput ≅ 0

May 16 - 20, 2016

 May 16 - 20, 2016

3. Multivariate Metric
• If the uncertainties in the n estimated δ model are
represented by normal distributions, the
2
expected value of r is obtained from a χ
distribution that leads to a reference value
given by rref = n + 2n
• If r / rref > 1 , the modelling errors are in general
larger than the validation uncertainty
• Otherwise, modelling errors are globally within
the Validation uncertainty
May 16 - 20, 2016
 May 16 - 20, 2016

4. Flow over a Flat Plate

• Reynolds-Averaged Navier-Stokes (RANS)
equations (time-averaging)
• Three versions of k-ω eddy-viscosity model with
and without transition modelling (SST, W98,
W06)
• Quantity of interest is the skin friction coefficient
Cf τw
Cf =
1 2 ρV∞2

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow over a Flat Plate

• Experimental data from the ERCOFTAC Classic
Database (T3A, T3AM, T3B) and the Stanford
Conference 1968
• 2% uncertainty assumed from variability in the
different techniques used to measure
• Experimental data for the inlet turbulence
intensity for the ERCOFTAC cases and forced
transition for the Stanford Conference data

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow over a Flat Plate

• Reynolds number based on plate length L and
incoming velocity V∞ equal to Re=107
C p =0
y ∂φ
1.5L =0
∂x
U x =1
U y =0 0 .25L
k = ki
ε = εi x
0 . 25L L 0 .25L

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow over a Flat Plate

• 9 geometrically similar grids ranging from
1537×193 to 385×49 (1≤hi/h1≤4)
0.2
y/L
L

0.1

0 0 0.5 1
x/L
• Calculations converged to maximum normalized
residual of all transport equations below 10-7

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow over a Flat Plate

8 Wieghardt SST W98 W06T
T3AM, I=0.874%
7 T3A, I=3.3%
T3B, I= 6.5%
6
5
3
10
Cf×1

4
3
2
1
0 4 5 6 7
10 10 10 10
Rex

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow over a Flat Plate

SST W06
150 Wieghardt
T3AM, I=0.874%
T3A, I=3.3%
T3B, I=6.5%
Cf)ref)%

W98
100
E(Cf / (C

50

4 5 6 7
10 10 10 10
Rex

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow over a Flat Plate

100 SST
W98
W06
80

60
r/rreef

40

20

0
Wieghardt T3AM T3A T3B

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow over a Flat Plate

20
SST
Wieghardt data W98
W06
10
Cf)ref)%
E(Cf / (C

-10

-20 5 6 7
10 10 10
Rex

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow over a Flat Plate

8 8
T3AM, I=0.874% 8 T3AM, I=0.874% T3AM, I=0.874%
T3A, I=3.3% T3A, I=3.3% T3A, I=3.3%
7 T3B, I=6.5% 7 T3B, I=6.5% 7 T3B, I=6.5%
νti= 0.1ν νti= ν νti= 10ν νti= 100ν νti= 0.1ν νti= ν νti= 10ν νti= 100ν νti= 0.1ν νti= ν νti= 10ν νti= 100ν
6 6 6
5 5 5
3

3
3
xti1
Cf×10

Cf×10
Cf×10
4 4 4
3 3 3
2 2 2
LCTM W98T W06T
1 1 1
0 0 0
104 105 106 107 104 105 106 107 104 105 106 107
Rex Rex Rex
8 T3AM, I=0.874% 8 T3AM, I=0.874% 8 T3AM, I=0.874%
T3A, I=3.3% T3A, I=3.3% T3A, I=3.3%
7 T3B, I=6.5% 7 T3B, I=6.5% 7 T3B, I=6.5%
νti= 0.1ν νti= ν νti= 10ν νti= 100ν νti= 0.1ν νti= ν νti= 10ν νti= 100ν νti= 0.1ν νti= ν νti= 10ν νti= 100ν
6 6 6
5 5 5
3

3
xti 2
Cf×10

Cf×10

Cf×10
4 4 4
3 3 3
2 2 2
LCTM W98T W06T
1 1 1
0 0 0
104 105 106 107 104 105 106 107 104 105 106 107
Rex Rex Rex

8 T3AM, I=0.874% 8 T3AM, I=0.874% 8 T3AM, I=0.874%

T3A, I=3.3% T3A, I=3.3% T3A, I=3.3%
7 T3B, I=6.5% 7 T3B, I=6.5% 7 T3B, I=6.5%
νti= 0.1ν νti= ν νti= 10ν νti= 100ν νti= 0.1ν νti= ν νti= 10ν νti= 100ν νti= 0.1ν νti= ν νti= 10ν νti= 100ν
6 6 6

xti 3 5 5 5
3

3
Cf×10

Cf×10

Cf×10
4 4 4
3 3 3
2 2 2
LCTM W98T W06T
1 1 1
0 0 0
104 105 106 107 104 105 106 107 104 105 106 107
Rex Rex Rex

SST+LCTM W98T W06T

May 16 - 20, 2016
 May 16 - 20, 2016

4. Flow over a Flat Plate

100
SST+LCTM
W98T
W06T
80

60
r/rreef

40

20

0
T3AM T3A T3B

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow over a Flat Plate

8 T3AM, I=0.874%
T3A, I=3.3%
7 T3B, I= 6.5%
LCTM W98T W06T
6
5
3
10
Cf×1

4
3
2
1
0 4 5 6 7
10 10 10 10
Rex

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow over a Flat Plate

SST W06
150 T3AM, I=0.874%
T3A, I=3.3%
T3B, I=6.5%
W98
Cf)ref)%

100
E(Cf / (C

50

4 5 6 7
10 10 10 10
Rex

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow around the NACA 0012

• Reynolds-Averaged Navier-Stokes (RANS)
equations (time-averaging)
• Three versions of k-ω eddy-viscosity model with
transition modelling (SST+LCTM, W98T, W06T)
• Quantities of interest are lift C L and C D drag
coefficients and surface pressure distribution C p
at 0, 2, 4, 6 and 8 degrees of angle of attack α

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow around the NACA 0012

• Experimental data from two different sources,
but only one used to determine E
• Estimated uncertainties from the information
available in the reports: 1% for C L; 3% for C D and
10% for surface pressure coefficient C p
• Inlet oturbulence conditions tuned to match C Dat
α =0
• Two variants of previous conditions tested

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow around the NACA 0012

• Reynolds number based on chord c and
incoming velocity V∞ equal to Re=2.88×106
Pressure imposed at one point of the inlet boundary Free Slip

10

5 Streamwise
Everything derivatives
y/c

specified 0 equal to
except zero
pressure -5

-10
-10 -5 0 5 10 15 20
x/c
Free Slip

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow around the NACA 0012

• 5 geometrically similar grids ranging from
232320 to 929280 or 90750 to 363000 cells
(1≤hi/h1≤2)

• Calculations converged
to maximum normalized
residual of all transport
equations (including
transition model LCTM variables) below 5×10-6

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow around the NACA 0012

14 Exp
13 SST+LCTM
W98T
12 W06T
3

11
CD×100

10
9
8
7
6
0 2 4o 6 8
α

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow around the NACA 0012

1
Exp
SST+LCTM
0.8
W98T
W06T
0.6
CL

0.4

0.2

0
0 2 4o 6 8
α

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow around the NACA 0012

0.8
α = 2o
0.6 Exp
SST+LCTM
W98T
0.4 W06T
-Cp

0.2

-0.2

0 0.2 0.4 0.6 0.8 1

x/c

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow around the NACA 0012

4
α = 8o
Exp
3 SST+LCTM
W98T
W06T
2
-Cp

0 0.2 0.4 0.6 0.8 1

x/c

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow around the NACA 0012

8

7 SST+LCTM
W98T
W06T
6

5
r/rreef

0
CD CL α = 0o α = 2o α = 4 o α = 6 o α = 8o

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow around the Eppler 387

• Reynolds-Averaged Navier-Stokes (RANS)
equations (time-averaging)
• Three versions of k-ω eddy-viscosity model with
transition modelling (SST+LCTM, W98T, W06T)
• Quantities of interest are lift C L and C D drag
coefficients and surface pressure distribution C p
at -2, -1, 0, 1, 2, 3, 4, 6 and 7 degrees of angle
of attack α

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow around the Eppler 387

• Experimental data from two different sources,
but only one used to determine E
• Estimated uncertainties from the information
available in the 2 reports are very different:
2% for C L ; 0.12% for C D and 0.25% for C p
5.5% for C L and 5.73% for C p
• Two variants of inlet turbulence conditions
tested

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow around the Eppler 387

• Reynolds number based on chord c and
incoming velocity V∞ equal to Re=3×105
Pressure imposed at one point of the inlet boundary
Free Slip
10

5 Streamwise
Everything derivatives
y/c

specified 0 equal to
except zero
pressure -5

-10
-10 -5 0 5 10 15 20
x/c
Free Slip

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow around the Eppler 387

• 5 geometrically similar grids ranging from
232320 to 929280 or 90750 to 363000 cells
(1≤hi/h1≤2)

• Calculations converged
to maximum normalized
residual of all transport
equations (including
transition model LCTM variables) below 5×10-6

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow around the Eppler 387

16 Exp
SST+LCTM
14 W98T
W06T
3
CD×100

12

10

6
-2 0 2 o 4 6 8
α

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow around the Eppler 387

1.2

0.8
CL

0.6 Exp
Exp
SST+LCTM
0.4
W98T
W06T
0.2

0
-2 0 2 o 4 6 8
α

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow around the Eppler 387

1
α = 0o Exp
Exp
SST+LCTM
W98T
0.5 W06T
-Cp

-0.5

-1
0 0.2 0.4 0.6 0.8 1
x/c

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow around the Eppler 387

1
α = 3o Exp
Exp
SST+LCTM
W98T
0.5 W06T
-Cp

-0.5

-1
0 0.2 0.4 0.6 0.8 1
x/c

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow around the Eppler 387

SST+LCTM
W98T
40 α = 0o W06T

α = 2o
r/rreef

20 α = 4o
α = 7o

0
CD C L α = −1o α = 1o α = 3o α = 6o

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow around a circular cylinder

• Reynolds-Averaged Navier-Stokes (RANS)
equations (ensemble-averaging)
• Three two-equation eddy-viscosity models
tested (SST, KSKL, W98T)
• Quantities of interest are the average drag
coefficient(C D )avg and the average pressure C p and
skin friction C f coefficients on the cylinder surface

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow around a circular cylinder

• Experimental data taken from Achenback E.,
“Distribution of Local Pressure and Skin Friction
around a Circular Cylinder in Cross-flow up to
Re = 5×106”, Journal of Fluid Mechanics, Vol.
34,No 64,1968, pp. 625-639”
• Estimated uncertainties from symbols size and
asymmetry of the solution

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow around a circular cylinder

• Reynolds numbers based on the diameter D
and incoming velocity V∞ equal to 105, 2.6×105,
8.5×105 and 3.6×106
Pressure imposed at one point of the inlet boundary Free Slip

Everything Streamwise
specified derivatives
except equal to
pressure zero

Free Slip

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow around a circular cylinder

• 4 geometrically similar grids ranging from
8.5×104 to 2.5×105 cells
(1≤hi/h1≤1.71)

• Calculations converged
to maximum normalized
residual of all transport
equations at each time
step below 10-6

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow around a circular cylinder

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow around a circular cylinder

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow around a circular cylinder

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow around a circular cylinder

8
SST
3.6 ×106
7 KSKL
W98 2×105
6

5 105 8.5 × 105

105
r/rreef

3
2× 1058.5 × 105
2

1 3.6 ×10 6
0
(CD )avg C p Cp Cp Cp Cf Cf Cf Cf

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow around the KVLCC2 tanker

• Reynolds-Averaged Navier-Stokes (RANS)
equations (time-averaging)
• 14 turbulence models tested including one and
two-equation eddy-viscosity models and Explicit
Algebraic Reynolds-Stress models
• Quantities of interest are the resistance
coefficient C T and the flow field at the propeller
plane

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow around the KVLCC2 tanker

• Experimental data from towing tank and wind
tunnel tests (different boundary conditions at
free surface and different blockage)
• Estimated uncertainties reported in the papers

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow around the KVLCC2 tanker

• Reynolds number based on ship length LPP and
incoming velocity V∞ equal to Re=4.6×106
Symmetry
Pressure imposed

Everything
specified
except
pressure
Everything
specified
except
pressure

Free Slip

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow around the KVLCC2 tanker

• 5 nearly geometrically similar grids ranging from
5×106 to 2×107 cells (1≤hi/h1≤2)
• Calculations converged
to maximum normalized
residual of all transport
equations below 10-8
for all turbulence models
tested

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow around the KVLCC2 tanker

Flow field at the propeller

plane
May 16 - 20, 2016
 May 16 - 20, 2016

4. Flow around the KVLCC2 tanker

Flow field at the propeller plane

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow around the KVLCC2 tanker

Flow field at the propeller plane

May 16 - 20, 2016

 May 16 - 20, 2016

4. Flow around the KVLCC2 tanker

Flow field at the propeller plane

May 16 - 20, 2016

 May 16 - 20, 2016

4. Conclusions
• The multivariate metric proposed by the ASME
V&V 20 is a useful tool to assess the modelling
accuracy of multiple set points
• It has a direct application in the comparison of
the modelling accuracy of different turbulence
models
• It should be “handled with care”, i.e. as for the
V&V 20 2009 procedure, it must be analysed
taking into account the level of Validation
uncertainty
May 16 - 20, 2016