Integrated Cache Training –

Netscaler Webinar

Mark Hillick
Agenda of Cash, sorry Cache
• Goals of Caching

qWhy & What

• Cache Policies

• How long to cache

• Memory for Cache

• IC configuration
example

• Cache Statistics

• Troubleshooting IC
© 2007 Citrix Systems, Inc. — All rights reserved 2
Presentation Goal

Please be aware that this a technical presentation with lots of detail

and it is intended that it can be used as a reference guide at a later
date (e.g. when you're on-site).

Sadly, it s not like my usual Zen presentation with lots of funny

pictures and jokes, sorry L

© 2007 Citrix Systems, Inc. — All rights reserved 3

Caching Goals

• Caching would be useless if it did not significantly improve

performance.

• Goal => to eliminate the need to send requests in many cases.

qReduces the number of network round-trips required for many

operations - we use an "expiration" mechanism for this
purpose

• Goal => remove the need to send full responses in many other
cases.

q The latter reduces network bandwidth requirements - we use

a "validation" mechanism for this purpose
© 2007 Citrix Systems, Inc. — All rights reserved 4
Caching is Used Everywhere

• Local cache in your browser

• Cache in the office (Forward Proxy or Transparent)

• Cache at the ISP (Transparent)

• Cache at the origin (Reverse Proxy or Transparent)

• Cache integrated inside a traffic manager – much better J

© 2007 Citrix Systems, Inc. — All rights reserved 5

Caching Benefits.

1. Saves Bandwidth

2. Reduces traffic to back-end server & so saves processing

resources

3. Protects back-end servers from flash crowds

4. Improves the response time & Increases site capacity

P.S. Even more benefits when integrated within a LB J

© 2007 Citrix Systems, Inc. — All rights reserved 6

IC in Packet-Processing
TCP Processing

SSL Decrypt

HTTP Parsing

AAA Processing

Cache Redirection

Content Switching

Application Firewall

Responder Action?

Cache Hit?

HTTP Cache Req Policies

Rewrite Policy Eval

Load Balancing

Content Filtering

© 2007 Citrix Systems, Inc. — All rights reserved 7

Non-Caching Proxy

GET /foo/index.html HTTP/1.1 GET /foo/index.html HTTP/1.1

Host: www.example.com Host: www.example.com

Proxy

HTTP/1.1 200 OK HTTP/1.1 200 OK

Last-Modified: Thu, ... Last-Modified: Thu, ...
Content-Length: 3688 Content-Length: 3688
Content-Type: text/html Content-Type: text/html

© 2007 Citrix Systems, Inc. — All rights reserved 8

Caching Proxy: Miss

GET /foo/index.html HTTP/1.1 GET /foo/index.html HTTP/1.1

Host: www.example.com Host: www.example.com

Proxy

HTTP/1.1 200 OK HTTP/1.1 200 OK

Last-Modified: Thu, ... Last-Modified: Thu, ...
Content-Length: 3688 Content-Length: 3688
Content-Type: text/html Content-Type: text/html

Proxy  Cache
(Saves  copy)

© 2007 Citrix Systems, Inc. — All rights reserved 9

Caching Proxy: Hit

GET /foo/index.html HTTP/1.1

Host: www.example.com

Proxy

HTTP/1.1 200 OK
Last-Modified: Thu, ...
Content-Length: 3688
Content-Type: text/html

Proxy  Cache
(Fresh  copy!)

© 2007 Citrix Systems, Inc. — All rights reserved 10

HTTP Response after a hit
• wget -S --header="If-None-Match:23834-b1-4951a45768b8d" -U
Mozilla http://10.90.196.68/index.html

• HTTP/1.1 200 OK
………

• Via: NS-CACHE-9.2: 65
• ETag: "23834-b1-4951a45768b8d"
• Server: Apache/2.2.14 (Ubuntu)
......... OR .........

• ETag: "23834-b1-4951a45768b8d"
• 2011-04-26 18:22:56 ERROR 304: Not Modified.
………

© 2007 Citrix Systems, Inc. — All rights reserved 11

HTTP Response after a miss

• $ wget -S --header="If-None-Match:23834-b1-4951a45768b8d" -U
Mozilla http://10.90.196.68/index.html
……….
• HTTP/1.1 200 OK
• Date: Wed, 27 Apr 2011 09:30:59 GMT
• Server: Apache/2.2.14 (Ubuntu)
• Last-Modified: Mon, 15 Nov 2010 16:52:53 GMT
………..
• Saving to: `index.html.41

© 2007 Citrix Systems, Inc. — All rights reserved 12

What Not to Cache

• Cache-Control: no-store =>applies to entire message (req/resp)

• Cache-Control: no-cache =>server cannot respond with cached
response
• Cache-Control: Private => not by shared cache
q More info see -
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
• POST request.
• Don t cache if set-cookie exists
• Don t cache all 5xx response codes, example 503 (Gateway time out)

• By default, NS ignores cache-control headers in requests -> CG

Settings
© 2007 Citrix Systems, Inc. — All rights reserved 13
Freshness

• Describes how long the associated representation is fresh for.

qNo need to send a request to server if object has not expired

qSaves time

• Expires: Fri, 30 Oct 1998 14:19:41 GMT

• Cache-Control: max-age=3600

© 2007 Citrix Systems, Inc. — All rights reserved 14

Validation

• Validation – once expired, validator can be used to check with the

server if the object is still valid.

q Send conditional request and get a short “304 Not Modified”

response
q Saves bandwidth

• If-Modified-Since <Date>

• If-None-Match <Etag>

© 2007 Citrix Systems, Inc. — All rights reserved 15

Caching Proxy: Validation

GET /foo/index.html HTTP/1.1 GET /foo/index.html HTTP/1.1

Host: www.example.com Host: www.example.com
If-Modified-Since: Thu, ...
Proxy

HTTP/1.1 200 OK HTTP/1.1 304 Not Modified

Last-Modified: Thu, ... Date: Fri, ...
Content-Length: 3688
Content-Type: text/html

Proxy  Cache
(Saves  copy)

© 2007 Citrix Systems, Inc. — All rights reserved 16

Cacheability Policies
• Specifies if response is cacheable.
q Request or Response-based policy
• CG is DEFAULT by default J but can be changed.
• Action types
q CACHE: Transaction is cacheable. Store the object
q NOCACHE: Transaction is non-cacheable.
q MAY_NOCACHE
o For request based policies only
o The response may be cacheable. The default if no response
cacheability policy matches is to not cache the response.
q MAY_CACHE
o For request based policies only
o The response may be cacheable. The default if no response
cacheability policy matches is to cache the response.
q INVAL
o Mark the object as invalid
© 2007 Citrix Systems, Inc. — All rights reserved 17
Cache Decision Time

• Cache, Nocache, Inval

q Cache decision @ request time & unless something that prevents decision
like CL mistmatch, no policy eval @ response time

• MAY_CACHE, MAY_NOCACHE
q Cache decision @ response time

© 2007 Citrix Systems, Inc. — All rights reserved 18

Invalidate versus Expire

• Invalidate
q Invalidate will invalidate the object/"all objects in contentgroup".
q But the way it is done is it is optimized such a way that we don't look at
each object and remove it but when you run expire command or when new
request comes for the invalid object, it is then removed.

q We serve a full response (200) to client regardless of cache hit.

• Expire
q Only mark the object as expired
q Request comes in & instead of getting full response (i.e what we do for
invalid object) we will send if-modified-since/if-none-match conditional
request to back-end server. i.e can receive a 304.

© 2007 Citrix Systems, Inc. — All rights reserved 19

Content Group
• Every cached object => member of CG
q No CG, object => Default
q minhit – no. of hits before store
• Properties of objects can be controlled per contentgroup
• Dynamic versus Static - parametised
• Configurations per content-group are
q Expiry settings
q Portions of object considered for caching as well as Invalidation
[Parameterized Caching]
q Pre-fetch objects before it gets expired (event-driven)
q Inserting headers (e.g. Via header)
q Memory related configuration for the objects and for this whole groups

© 2007 Citrix Systems, Inc. — All rights reserved 20

Flash Cache (FC)
• FC
q Improves performance –> 1 req to back-end server
q Concurrent requests queued on NS & single response served
q set/add cache contentgroup <group_name> -flashCache YES

q POWERFUL

q Cannot enable with PET (ensures IC polls origin before serving)

© 2007 Citrix Systems, Inc. — All rights reserved 21

Some Other CG settings
• Expiry settings (blank by default)
qRelexpiry – relative -> overrides server response settings
qheurexpiryParam - %
qExpire after complete response received
• If no expiry settings & none in response => cache miss but can do
validation (non-fc)

• Cache controls
qNo cache depending on size or hits
qMem limit usage per group
• Works with FC J but probably not wise on significant groupsExpire
after complete response received

© 2007 Citrix Systems, Inc. — All rights reserved 22

Cache Policies

• Built-In (begin with “__” & “ctx_”)

q Greatly enhanced in 9.3 – should be sufficient for most

• Bound globally or per vServer

q Easy-To-Administer & View within Global Policy Manager

• PI-based expression engine like other NS features

© 2007 Citrix Systems, Inc. — All rights reserved 23

Cache

• Built-In (begin with “__” )

q Greatly enhanced in 9.3 – should be sufficient for most

• Bound globally or per vServer

q Easy-To-Administer & View within Global Policy Manager

• PI-based expression engine like other NS features

© 2007 Citrix Systems, Inc. — All rights reserved 24

Verification

• Three Verification Options (again sh cache pa)

q Hostname (Sufficient for Most Companies)

q Hostname & IP (Default)

q DNS (Most Secure)

© 2007 Citrix Systems, Inc. — All rights reserved 25

Flushing the Cache
• Flush Cache
q Remove stale content manually
q Troubleshooting (preferred flush, disable, enable)
q CLI – either CG or Object
o flush cache contentGroup content_group_html
o flush cache contentGroup ALL
o flush cache object –locator <0x….>
q GUI – IC > Cache Objects

• Automatic CG flush once CG configuration is modified

© 2007 Citrix Systems, Inc. — All rights reserved 26

 Cache Memory Management (1)
• BSD & PPE First
• 7500 with 8gb RAM & 3 PE
q 2gb for kernel, 3gb for PE & max 3gb for IC
q Each PE has minimum of 1gb

• 17500 – 48gb mem & 11 PE

q 2gb for kernel, 23gb to PE & IC can take up to 23gb
q Each PE cannot exceed 3.2gb address space

• IC takes up what’s left (!>0.5)

© 2007 Citrix Systems, Inc. — All rights reserved 27

 Cache Memory Management (2)
• > set cache pa –memLimit 254
• > sh cache pa | grep -i mem
Memory usage limit: 10000 MBytes

Memory usage limit (active value): 10000 Mbytes => reboot

Maximum value for Memory usage limit: 15662 MBytes

• > sh cache contentgroup

• Memory Allocation Issues
qcli> stat cache –detail | grep alloc

Memory allocation failures 5741 => cache misses

© 2007 Citrix Systems, Inc. — All rights reserved 28

Cache Selectors
• Best practice to configure selectors (preferred over parameters)
• Named filter – locates particular objects in CG, not exact match
• Associate selector based on the request, not with responses
• Sample selectors
• http.req.url; http.req.method; client.ip.dst etc
• Use cases
qOne file only cached once regardless of hostname
qTwo copies of same file – compressed & uncompressed
qSame file, same hostname but different VIPs & CGs

© 2007 Citrix Systems, Inc. — All rights reserved 29

“Not Cache” Best Practice
• Not Cache policies first
q Once IC is turned on, it will begin to cache as per the default
policies.

o This may cause confidential content to be cached by

accidentThen cache known, good content

q Start with images – jpegs, gifs

q Check statistics for
o Memory allocation errors
o Hits
o Misses
o Expiries

© 2007 Citrix Systems, Inc. — All rights reserved 30

Cache HTTP Callout
• Callouts to external servers
q Cache saves multiple requests

• add cache policy pol_callout -rule "CLIENT.IP.SRC.EQ

(127.128.129.130 )" -action CACHE –storeInGroup

• bind cache global pol_callout –priority 100 –gotoPriorityExpression END

–type REQ_DEFAULT

• sh cache object –locator <0x…>

HTTP callout cell: YES
HTTP callout name: callout1
HTTP callout type: TEXT
HTTP callout response: NO

© 2007 Citrix Systems, Inc. — All rights reserved 31

Soft-Cache Expiry
• When a back-end server does not respond quickly to requests, an
upstream NetScaler can serve responses from the cache.

q Liberal interpretation of RFC

q Protects back-end server further (e.g. lash-type event)
q Serves stale data for a minimal amount of time

• Applies to 9.2.52.3 & above

© 2007 Citrix Systems, Inc. — All rights reserved 32

Configuration through Wizard (1)
• Easy-to-use J

© 2007 Citrix Systems, Inc. — All rights reserved 33

Configuration through Wizard (2)

© 2007 Citrix Systems, Inc. — All rights reserved 34

Configuration through Wizard (3)

© 2007 Citrix Systems, Inc. — All rights reserved 35

Example Configuration
• enable feature IC

• set cache parameter –memlimit 512

• add cache contentgroup cg_image –relExpiry 600 –memLimit 100 –

maxResSize 512

• add cache policy cp1 –rule http.req.url.contains(“jpeg”) –action cache

–storeingroup cg_image

• bind cache global cp1 –priority 1 –type REQ_DEFAULT

cli> sh cache object

0x000000078b4200000004 cg_image //10.102.30.16:80/abc.jpeg

© 2007 Citrix Systems, Inc. — All rights reserved 36

Statistics via the CLI (1)

• cli> sh cache object

0x000000078b4200000004 cg_image //10.102.30.16:80/abc.jpeg

• cli> sh cache object –locator 0x000000078b4200000004

q Address in Memory

• nscachemgr –a/-g <contentgroup>

q Shell Equivalent

• Use grep – POWERFUL for parsing output for hits, misses etc.

© 2007 Citrix Systems, Inc. — All rights reserved 37

Statistics via the CLI (2)

© 2007 Citrix Systems, Inc. — All rights reserved 38

Statistics via the GUI (1)

© 2007 Citrix Systems, Inc. — All rights reserved 39

Statistics via the GUI (2)

© 2007 Citrix Systems, Inc. — All rights reserved 40

Statistics via the GUI (3)

© 2007 Citrix Systems, Inc. — All rights reserved 41

Troubleshooting Tools1

• Browser -> inconsistencies & can be very frustrating L

qFF (HTTP Fox & Tamper Data Extensions but FF4 issues)
qIE (7, 8, 9) -> Fiddler
qOpera & Chrome -> as standard, no extensions

• Tracing & PCAP ->

qNstrace & Wireshark as usual J

© 2007 Citrix Systems, Inc. — All rights reserved 42

Troubleshooting Tools2

• Paros (as a proxy in the middle) -> I now use Zap (Owasp project fork of
Paros)

• CLI -> no user-cache to worry about J

qwget
• wget –S –header=“If-None-Match:etag_value” http://test.com/file.jpg
qcurl
o curl –header “If-None-Match: etag_value” test.com/index.html

© 2007 Citrix Systems, Inc. — All rights reserved 43

Troubleshooting

• Cache turned on but no cache objects

q sh cache pa != Memory usage limit: 0 bytes
q Zero memory = cache misses
o Warning msg in 9.2 when sh cache object
q Check expiry settings? NS? Server?

• Blank entries in nscachemgr output?

q //10.90.196.76/
q //10.90.196.76/citrix_logo.jpg
q Default caching properties of GET / HTTP/1.1

© 2007 Citrix Systems, Inc. — All rights reserved 44

CTX Articles (just a snapshot)

• http://support.citrix.com/article/CTX124553 - how to IC
• http://support.citrix.com/article/CTX123753 - compressed
content to unadvertised clients
• http://support.citrix.com/article/CTX126557 - caching
uncompressed & compressed content of same file
• http://support.citrix.com/article/CTX128801 - same
hostname/object but different vip twice
• http://support.citrix.com/article/CTX129118 - has my HTTP
callout been cached
• http://support.citrix.com/article/CTX124718 - cache callout
response
• http://support.citrix.com/article/CTX129734 - Soft Cache
Expiry

© 2007 Citrix Systems, Inc. — All rights reserved 45

Extra Reading

• You can use perl, curl, wget, browser add-ons but I like wget –
q http://www.cyberciti.biz/tips/linux-wget-your-ultimate-
command-line-downloader.html
q http://www.computerhope.com/unix/wget.htm
q Google “wget site:www.linuxquestions.org”
q Easy to write a basic shell script around it & doesn’t need
to be too complicated

• RFC 2616 (HTTP) VERY IMPORTANT

q Section 13 (Caching)
q Section 14 (Headers)

© 2007 Citrix Systems, Inc. — All rights reserved 46

So with IC you can, take over….

© 2007 Citrix Systems, Inc. — All rights reserved 47

Sláinte

© 2007 Citrix Systems, Inc. — All rights reserved 48

© 2007 Citrix Systems, Inc. — All rights reserved 49