AAPM TG-18

Executive Summary
(2005)
Introduction What is TG-18?
Quality assurance of digital image displays

• In the 90’s and early 2000’s, medical imaging transitioned from film
and film-screen systems to digital imaging

• Soft copy vs Hard copy images

• Much more efficient means to:
• Produce images
• Archive images
• Retrieve images
• Transport images
• Display images
3
Quality assurance of digital image displays

• In the midst of this transition, several studies showed that electronic

display devices can compromise image quality and interpretations
• S. J. Ackerman, et al., “Receiver operating characteristic analysis of fracture and pneumonia detection: comparison of laser-digitized
workstation images and conventional analog radiographs,” Radiology 186, 263–268 (1993).
• W. W. Scott, et al., “Subtle orthopedic fractures: Teleradiology workstation versus film interpretation,” Radiology 187, 811–815 (1993)
• W. W. Scott, et al., “Interpretation of emergency department radiographs by radiologists and emergency medicine physicians:
teleradiology workstation versus radiograph readings,” Radiology 195, 223–229 (1995).

• How can we ensure the image quality a film-based system offers is not
lost or degraded due to a digital image or display?

4
 Quality assurance of digital image displays

• AAPM determined that the assessment and performance of electronic

displays falls within the responsibilities of medical physicists

• AAPM TG-18 suggests standard guidelines and criteria for acceptance

testing and quality control of medical display devices
• Intended audience is medical physicists, engineers, researchers, and radiology admin
• The report addresses CRT and LCD monitors, the dominant technology of the time

5
Display Technologies

• Cathode Ray Tube (CRT) displays

• Display images through continuous raster scanning of a phosphor
screen with an electron beam

• The electron beam is generated by a filament, and is accelerated

towards the phosphor screen via a series of electrostatic lenses

• A magnetic deflection yoke alters the path of the beam to produce

the raster scanning, at a rate of 60-80 fps

• As the beam is scanned, its intensity is modulated as appropriate

6
Display Technologies

• Liquid Crystal Displays (LCDs)

• Displays images using the light polarization properties of liquid
crystals in conjunction with thin-film transistors (TFTs)

• Each liquid crystal cell produces its own modulated light source,
which all together generate the image

• The display consists of a series of thin components coupled

together, with a uniform fluorescent light source on the back end

7
General Prerequisites
for Display Assessments
Classification of Display Devices

• In accordance with ACR and FDA guidelines, TG-18 characterizes

displays as either Primary or Secondary

• Primary displays: Used for the official interpretation of medical images

• Secondary displays: Used for viewing medical images for purposes other than
official interpretation
• Modality displays
• Post-interpretation displays used by non-radiology physicians

9
Required Tools: Instrumentation

• Although many display tests are qualitative, certain quantitative tests

require special tools

• Luminance meter – display brightness

• Illuminance meter – ambient light
• Colorimeter – display chromaticity
• Magnifying glass – visual resolution

10
Required Tools: Test Patterns

• TG-18 specifies a series of test patterns for different display

assessments, designated by TG18-xyz where xyz denote the type and
variant of a pattern

• When displaying TG18 patterns, the window leveling should allow the
full range of values to be displayed
• 16-bit patterns: WL = 2048, WW = 4096
• 8-bit patterns: WL = 128, WW = 256

11
Required Tools: Software

• Not essential, but for physicists who will assess many displays of
different makes/models, software tools can be very helpful

• Some software can have test patterns pre-loaded for display

• Some software can perform automatic or manual calibrations

• Some software can even generate evaluation reports for each display!

12
Preparing for display assessment

• Are test patterns available on the workstation or readily available to be

uploaded and displayed in proper format?
• It is very inefficient (and frustrating) to attempt to assess a display without having
access to test patterns or knowing where the patterns are located

• Is the display device in a location optimal to assess performance?

• Ensure the display is in a location where ambient light or light reflections will not
affect evaluation
• For CRT monitors, ensure no magnetic fields are present that can cause
distortion
• Ensure the display and workstation is configured ergonomically 13
 Preparing for display assessment

• Has the display been allowed to warm up?

• Displays should be allowed to warm up for ~30mins prior to evaluation
• A quick review of the TG18-QC pattern should be used to ensure functionality
• During warm up, the display should be checked and cleaned of dust and smudges
• For new displays, any plastic coating or protective layer should be removed

14
 Preparing for display assessment

• Is the ambient lighting level optimal while being close to operating conditions?
• The ambient light (illumination) should be reduced to an appropriate level and should be
reflective of operating conditions
• Illuminance should be recorded, tracked, and compared to reference measurements

15
 Preparing for display assessment

• Are the luminance settings configured properly?

• The display area size should be correctly established
• The minimum and maximum luminance (𝐿 and 𝐿 ) should be measured using TG18-
LNz-1 and TG18-LNz-18 to ensure the luminance ratio (LR) is appropriate
• If the luminance ratio is not appropriate, the display should be calibrated or serviced prior
to assessment

• Is the display a new system?

• If so, the specifications should be verified against the purchase agreement
• System information (make, model, software or test pattern availability) should be recorded
• Acceptance test results should serve as baseline/reference values for routine QC 16
Assessment of
Display Performance
Geometric Distortions

• Mostly a concern in CRT display devices

• Qualitative evaluation
• The TG18-QC or the TG18-LPV/LPH test patterns should be used, filling the entire display area
• Patterns should be viewed from 30cm
• Patterns should appear with straight lines, proper scaling, and without significant distortion

• Some small barrel and pincushion distortions are normal for CRT monitors, but should not be
excessive

18
Geometric Distortions

• Mostly a concern in CRT display devices

• Quantitative evaluation
• With the TG18-QC pattern displayed, use a flexible ruler to measure square areas of the pattern
to determine the percent deviations between measurements in the vertical and horizontal
directions
• The measured spatial deviations shall be < 2% and < 5% for primary and secondary displays,
respectively

• For workstations that use more than one display, the displays should have spatial deviations
matched within 2%

19
Display Reflection

• Electronic displays have specular and diffuse reflection that can

reduce image contrast and degrade image quality
• Ambient light reflections are more pronounced in CRTs (thick faceplates) than
LCDs (thin faceplates)

20
Display Reflection

• Qualitative evaluation
• Specular Reflections
• With the display in sleep mode or off, the display
should be viewed at 30-60cm within +/- 15 ̊
• No high contrast patterns or sources of light should
be visible in the display

• Diffuse Reflections
• The TG18-AD pattern can be viewed from 30cm, in
normal ambient light conditions and in near
darkness conditions. The quality of TG18-AD should
not change between the two ambient light
conditions

21
 Display Reflection – Quantitative Eval
Diffuse Reflection Characteristics Specular Reflection Characteristics
𝐿 @𝑑
𝑅 =
𝐿 𝐿 @𝑑 +𝑑
𝑅 = 𝐿 /𝐿

𝑑
𝐿
𝐿
𝐿 𝑑

22
Luminance Response

• The human visual system perceives brightness in a non-

linear fashion
• So, the luminance response of a display should match this non-
linear response

• The DICOM Grayscale Standard Display Function (GSDF)

• Applies a specific LUT to display values, so that the display values present
equally discriminable levels of brightness

23
Luminance Response

• Qualitative Evaluation
• The TG18-CT pattern can be inspected to quickly
evaluate luminance response
• Viewed from 30cm, the central halfmoon
targets and the four low-contrast targets in
each corner of the 16 luminance patches
should be visible

24
Luminance Response

• Qualitative Evaluation
• The TG18-MP pattern can also be inspected to
evaluate the bit-depth resolution
• Viewed from 30cm
• No contrast reversals should be visible
• Horizontal white markers (small: 10-bit,
large: 8-bit) represent bit depth transitions
• Location of contouring bands should
correspond to white markers

25
Luminance Response
• Quantitative Evaluation
• The DICOM GSDF applies a specific LUT of display
values to luminance for the human visual system,
so we want to compare a monitors response
function to the GSDF

• The TG18-LN patterns are 18 luminance

patches ranging from gray value 0 to 255 (8-
bit) or 0 to 4095 (12-bit)

• Measure the monitor’s 𝐿 =𝐿 +𝐿

and luminance at each TG18-LN pattern

• Calculate the response function using the

DICOM GSDF lookup tables
26
• DICOM part 3.14
Luminance Dependencies

• While the luminance response evaluation characterizes the

display’s luminance in one location, luminance may also vary
due to non-uniformity or angular dependence

27
Luminance Dependencies

• Qualitative Evaluation – Display Uniformity

• The TG18-UN10 and TG18-UN80 can be used to
assess uniformity
• Viewed from 30cm, both patterns should be free
of gross nonuniformities from the center to the
edges

28
Luminance Dependencies

• Quantitative Evaluation – Display Uniformity

• The TG18-ULN10 and TG18-ULN80 can be used
to assess uniformity
• Luminance is measured at the center and four
corners of the display (in the boxes)
• The maximum luminance deviation for each
pattern should be calculated and <30%

• %𝑑𝑖𝑓𝑓 = 200 ∗

29
Luminance Dependencies

• Qualitative Evaluation – Angular Dependency

• The TG18-CT pattern can be used to assess
angular variation

• Viewed from 30cm at the display norm, at what

viewing angle is visual performance of the TG18-
CT pattern significantly degraded?

• The acceptable viewing angle cone should be

documented for the display

30
Luminance Dependencies

• Quantitative Evaluation – Angular Dependency

• The TG18-LN1 (𝐿 ) and TG18-LN18 (𝐿 ) patterns
should be used
• Using a telescopic luminance meter 30cm from the
display surface, measure the Luminance Ratio (LR) at
different viewing angles
• As viewing angle increase from norm the LR of the
display should not reduce by more than 30%, and the
• The LR must be >175 at angle for primary displays
• The LR must be > 70 at angle for secondary displays

• The acceptable viewing cone for the display should

be documented 31
Display Resolution

• Resolution is the ability of a display to present the spatial

details of a displayed image, and is related to the number
of pixels, the spatial extent of each pixel, and the
optical/electronic processes of the pixel

32
Display Resolution
• Qualitative Evaluation
• The TG18-QC pattern contains line-pair
patterns as well as “Cx” patterns which can be
assessed for display resolution

• The viewer should use a magnifying glass to

assess the line pair and Cx patterns

• The Cx patterns should score between 0 and 4

for primary displays

• The Cx patterns should score between 0 and 6

for secondary displays
33
 Display Resolution
• Quantitative Evaluation
• The MTF of the display should be calculated using a
digital camera and a test pattern that provides line or
edge inputs such as the TG18-RV, TG18-RH, and TG18-NS
patterns

• Camera lens flare should be reduced using a cone

• The line/edge should be oversampled, at least 64:1

camera pixels to display pixel

• Camera pixel array should be angled 2-5 degrees

• 30 images should be acquired for an average MTF

34
Display Noise

• Noise refers to statistical fluctuations in the image that either

vary spatially or temporally.
• Temporal noise is difficult to characterize outside of a lab and is
less well understood, but spatial noise is much simpler to
characterize

35
 Display Noise
• Qualitative Evaluation
• The TG18-AFC pattern can be used to assess
spatial noise effects on a display

• Viewed from 30cm, the TG18-AFC pattern

has 4 quadrants with low contrast targets of
varying sizes.

• For primary displays, the low contrast targets

should be visible in at least 3 quadrants

• For secondary displays, the low contrast

targets should be visible in at least 2
quadrants

36
Display Noise
• Quantitative Evaluation
• The noise power spectrum (NPS) of a display can
be used to characterize spatial noise of a display
using the TG18-NS patterns

• A scientific grade digital camera should be used to

acquire images of uniform areas of the test
patterns, with a time integration of ~1s

37
Veiling Glare

• Veiling glare is a light-spreading phenomenon in a display

that causes a degradation of image contrast

• In CRTs, veiling glare is caused by internal light-scattering

processes, light leakage, and electron backscattering

• In LCDs, electronic cross-talk can be viewed as a form of veiling

glare
38
Veiling Glare
• Qualitative Evaluation
• The TG18-GV and TG18-GVN patterns can
be used to assess veiling glare
• The display should be adjusted so that the
white region has a diameter of 20cm
• Using a cone such that the viewer can only
see the inner circle, the low contrast targets
should be assessed

• For primary displays, 3 targets should be

discernible
• For secondary displays, at least 1 target should be
discernible

39
 Veiling Glare TG18-GQB

• Quantitative Evaluation
• The TG18-GQ, TG18-GQB, and TG18-GQN TG18-GQ
can be used to calculate a glare ratio GR

𝐿
• Using a highly collimated luminance meter
with a light blocking cone to measure 𝐿, 𝐿 ,
and 𝐿 TG18-GQN

• 𝐺𝑅 = 𝐿

• For primary displays, GR > 400

• For secondary displays, GR > 150
40
Display Chromaticity

• Chromaticity refers to the magnitude and uniformity of color tint of

the device when displaying a monochrome image

• In CRTs, color tint is dictated by the phosphor type

• In LCDs, color tint is dictated by the color temperature of the backlight

• Color tint is usually considered a preference issue, but it can be a cause of

distraction, especially in workstations where multiple monitors have
mismatched color tints

41
Display Chromaticity

• Qualitative Evaluation
• The TG18-UN80 should be used to visually
assess color uniformity

• The pattern should be displayed on all monitors

at a workstation, and the relative color uniformity
of each display and across displays should be
assessed

• For primary displays, no significantly perceivable

color differences should be present across
displays or within a single display
42
 Display Chromaticity
• Quantitative Evaluation
• The TG18-UNL80 should be used to quantify color
nonuniformities

• The pattern should be displayed on all monitors at a

workstation, and a colorimeter should be used to
measure the (𝑢 , 𝑣′) color coordinates at the center and
corner of each display

• Using all measurements, the distance between any pair

of measurements should be calculated
• 𝐷= 𝑢 −𝑢 + 𝑣 −𝑣

• The distances should all be uniform within 1% for primary

displays 43
Miscellaneous tests

• There are several other display performance tests outlined

by the TG18 report such as video artifacts, moire artifacts,
color artifacts, physical defects, flicker, and electronic cross-
talk

• These assessments are considered more niche and not

universally necessary
44
Overall Evaluations

• A more general/overall assessment of display performance

can be achieved using the TG18-QC and TG18 anatomical
patterns

45
 Overall Evaluations – TG18-QC pattern

• General image quality and artifacts

• Evaluate the overall appearance of the pattern
• Note any non-uniformities or artifacts
• Verify the ramp bars appear continuous without any contour lines

• Geometric distortion
• Verify the borders and lines of the pattern are visible and straight
• Ensure the pattern appears centered in the display area

• Luminance, reflection, noise, and glare

• Verify all 16 luminance patches are distinguishable
• Verify the 5% and 95% patches are visible
• Verify the appearance of low contrast letters and targets

• Resolution
• Evaluate the Cx patterns at the center and the corners of the pattern
• Verify the visibility of the line pair patterns at the center and corners 46
Overall Evaluations –
Anatomical patterns
• A radiologist should evaluate the
overall clinical image quality of the
TG18 anatomical patterns

• There is a chest radiograph, knee

radiograph, and two digital
mammograms
• The radiologist should evaluate the
images according to specified
criteria and record any significant
discrepancies

47
AAPM TG-18 Conclusions

• Electronic displays are a key component of the imaging

chain and have essentially replaced film as the medium
through which radiologists view images

• Acceptance testing and QC testing is essential for ensuring

quality patient care, and TG18 provides specific test
procedures and criteria towards that end
48
Technological and Psychophysical
Considerations for Digital
Mammographic Displays

Ehsan Samei, PhD (2005)

Introduction

• This article:
• Briefly introduces CRT and LCD display technologies (already
covered)

• Discusses some psychophysical issues related to soft-copy viewing

of mammograms

• Specific testing methodologies for digital mammographic displays

50
Psychophysical Factors
Display Luminance and Contrast Factors

• Mammography is a low-contrast modality

• Higher-brightness images cause reduce noise in the human visual
system (more light to eyes), which improves the contrast threshold.

• Mammographic displays should have a luminance values of

𝟐
𝒎𝒊𝒏
𝟐
𝒎𝒂𝒙

Mammographic displays should be calibrated per the DICOM

GSDF, with extra attention at low luminance levels 52
Display Luminance and Contrast Factors

• Mammography requires optimized image contrast

• Display reflection, both specular and diffuse, can degrade contrast
at low luminance levels

• Displays with high reflection coefficients and low 𝒎𝒊𝒏 values

are most prone to contrast degradation due to reflection

53
Display Luminance and Contrast Factors

• Mammography requires optimized image contrast

• Display veiling glare degrades contrast similarly to reflections

• CRTs are more prone to veiling glare effects

• Glare tends to degrade contrast in dark regions surrounded by

bright scenes

• For mammography, it is best to use displays with minimal glare 54

Resolution Factors

• Smaller pixel sizes allow for higher intrinsic resolution, but ideally
resolution should match the limits of the human visual system

• Currently, the clinical consensus is that for a 30x40cm display, the

display should contain 5 megapixels

55
Noise Factors

• Luminance noise describes variations in pixel luminance

when a uniform pattern is displayed
• In CRTs, luminance noise is mostly due to phosphor granularity
• The amount of luminance noise in a CRT display will be in part due to the
type of phosphor screen

• In LCDs, luminance noise is generally low and due to electronic

cross talk, which can be corrected for via gain calibration

56
 Noise Factors

• Structured noise describes spatially fixed, recognizable

structures within the image
• CRTs are prone to “raster lines” in uniform regions
• LCDs produce noise due to the pixel structure
• LCDs go through a “pre-whitening” process which can reduce structured
noise

57
 Presentation Considerations

• Beyond device specifications of displays, how displays present images is also

important
• Viewers are likely to have viewing preferences
• MLO on left with CC on right or vice versa
• Four images displayed per display vs one or two
• Having two monitors per workstation is optimal to review mammograms
• Workstations should have conveniently accessible tools
• Pan/zoom
• Distance measurement
• Image data
58
Testing Mammographic Displays
Testing Mammographic Displays

• Follow AAPM TG-18!

• The quality assurance of mammographic displays is regulated by

21 CFR 900 (MQSA)

• TG-18 describes standards and procedures for testing diagnostic

displays, including those use for mammography

60
Conclusions

• Since mammography is primarily a low contrast

modality:
• Mammographic displays should be selected and tested based on
their ability to deliver adequate image quality
• high display luminance, sufficient image contrast, high spatial resolution, low
image noise

61