ISP Network Design

ISP Workshops

These materials are licensed under the Creative Commons Attribution-NonCommercial 4.0 International license
(http://creativecommons.org/licenses/by-nc/4.0/)

Last updated 27th February 2015 1

Acknowledgements
p  This material originated from the Cisco ISP/IXP
Workshop Programme developed by Philip Smith
& Barry Greene

p  Use of these materials is encouraged as long as

the source is fully acknowledged and this notice
remains in place

p  Bug fixes and improvements are welcomed

n  Please email workshop (at) bgp4all.com

Philip Smith
2
ISP Network Design
p  PoP Topologies and Design
p  Backbone Design
p  Upstream Connectivity & Peering
p  Addressing
p  Routing Protocols
p  Security
p  Out of Band Management
p  Operational Considerations

3
Point of Presence
Topologies

4
PoP Topologies
p  Core routers – high speed trunk
connections
p  Distribution routers and Access routers –
high port density
p  Border routers – connections to other
providers
p  Service routers – hosting and servers
p  Some functions might be handled by a
single router
5
PoP Design
p  ModularDesign
p  Aggregation Services separated according
to
n  connection speed
n  customer service
n  contention ratio
n  security considerations

6
 Modular PoP Design
Other ISPs
Web Cache
ISP Services Hosted Services &
(DNS, Mail, News, Datacentre
FTP, WWW)

Backbone link Backbone link

to another PoP to another PoP

Network
Core

Consumer cable,
Consumer xDSL and
Dial Access wireless Access

Business customer MetroE customer

aggregation layer aggregation layer
Network
Operations
Centre
Channelised circuits GigE fibre trunks
for leased line circuit delivery for MetroE circuit delivery
7
Modular Routing Protocol Design
p  Modular IGP implementation
n  IGP “area” per PoP
n  Core routers in backbone area (Area 0/L2)
n  Aggregation/summarisation where possible
into the core
p  Modular iBGP implementation
n  BGP route reflector cluster
n  Core routers are the route-reflectors
n  Remaining routers are clients & peer with
route-reflectors only
8
Point of Presence Design

9
PoP Modules
p  Low Speed customer connections
n  ADSL2, Cable, Public Wireless, PSTN access
n  Low bandwidth needs
n  Low revenue, large numbers

p  Business customer connections

n  10Mbps to 100Mbps speed range
n  Delivery over SDH or MetroEthernet links
n  Medium bandwidth needs
n  Medium revenue, medium numbers

10
PoP Modules
p  Highband Business customer connections
n  From 100Mbps to over 1Gbps
n  Ethernet, Fibre, or high end SDH
n  High bandwidth needs
n  High revenue, low numbers

11
PoP Modules
p  PoP Core
n  Two dedicated routers
n  High Speed interconnect
n  Backbone Links ONLY
n  Do not touch them!

p  Border Network
n  Dedicated border router to other ISPs
n  The ISP’s “front” door
n  Transparent web caching?
n  Two in backbone is minimum guarantee for
redundancy 12
PoP Modules
p  ISP Services
n  DNS (cache, secondary)
n  News (still relevant?)
n  Mail (POP3, Relay, Anti-virus/anti-spam)
n  WWW (server, proxy, cache)

p  Hosted Services/DataCentres
n  Virtual Web, WWW (server, proxy, cache)
n  Information/Content Services
n  Electronic Commerce

13
PoP Modules
p  Network Operations Centre
n  Consider primary and backup locations
n  Network monitoring
n  Statistics and log gathering
n  Direct but secure access

p  Out of Band Management Network

n  The ISP Network “Safety Belt”

14
 PSTN & Broadband Access Module

Web Cache

DSLAM
Telephone Network BRAS
Access Network
Gateway Routers
IP, ATM

Cable RAS
To Core Routers
Cable System

Primary Rate
T1/E1

Digital Modem SSG, DHCP, TACACS+

Access Servers or Radius Servers/Proxies,
DNS resolver, Content
15
Business Customer Access Module

Aggregation Edge

Channelised T1/E1

Metro Ethernet
To Core Routers

Direct Fibre Links

16
High Speed Access Module

Aggregation Edge

Metro Ethernet

Direct Fibre Links

To Core Routers

Channelised OC3/OC12

17
ISP Services Module
To core routers

Service Network
Gateway Routers

WWW DNS POP3s SMTP SMTP DNS

cache Secondary IMAPs Host Relay Resolver
(in) (out)

18
Hosted Services Module
To core routers

Hosted Network
Gateway Routers

vlan11 vlan12 vlan13 vlan14 vlan15 vlan16 vlan17

Customer 1 Customer 3 Customer 5 Customer 7

Customer 2 Customer 4 Customer 6

19
Border Module

To local IXP ISP1 ISP2

NB: router has no default route +
local AS routing table only

Network
Border Routers

To core routers

20
 NOC Module
Critical Services
To core routers Module
Corporate LAN
Out of Band
Hosted Network
Management Network Gateway Routers Firewall

async terminal server

Billing, Database
and Accounting
Systems
NetFlow TACACS+ SYSLOG Primary DNS
Analyser server server

Network Operations Centre Staff 21

 Out of Band Network

Out of Band
Management Network

Router
consoles
Terminal server
To the NOC
NetFlow
enabled
routers
NetFlow
Collector

Out of Band Ethernet

22
Backbone Network
Design

23
Backbone Design
p  Routed Backbone
p  Switched Backbone
n  ATM/Frame Relay core network
n  Now obsolete

p  Point-to-point circuits
n  nx64K, T1/E1, T3/E3, OC3, OC12, GigE, OC48,
10GigE, OC192, OC768, 100GE
p  ATM/Frame Relay service from telco
n  T3, OC3, OC12,… delivery
n  Easily upgradeable bandwidth (CIR)
n  Almost vanished in availability now
24
Distributed Network Design
p  PoP design “standardised”
n  operational scalability and simplicity
p  ISP essential services distributed around
backbone
p  NOC and “backup” NOC
p  Redundant backbone links

25
 Distributed Network Design
Customer
ISP Services connections

Backup
Operations Centre POP Two

Customer Customer
connections connections

ISP Services

POP Three POP One

ISP Services
External External
connections connections
Operations Centre
26
Backbone Links
p  ATM/Frame Relay
n  Virtually disappeared due to overhead, extra
equipment, and shared with other customers
of the telco
n  MPLS has replaced ATM & FR as the telco
favourite
p  Leased Line/Circuit
n  Most popular with backbone providers
n  IP over Optics and Metro Ethernet very
common in many parts of the world

27
Long Distance Backbone Links
p  These usually cost more
p  Important to plan for the future
n  This means at least two years ahead
n  Stay in budget, stay realistic
n  Unplanned “emergency” upgrades will be
disruptive without redundancy in the network
infrastructure

28
Long Distance Backbone Links
p  Allowsufficient capacity on alternative
paths for failure situations
n  Sufficient can depend on the business strategy
n  Sufficient can be as little as 20%
n  Sufficient is usually over 50% as this offers
“business continuity” for customers in the case
of link failure
n  Some businesses choose 0%
p  Very short sighted, meaning they have no spare
capacity at all!!

29
Long Distance Links

POP Two
Long distance link

POP Three POP One

Alternative/Backup Path

30
Metropolitan Area Backbone Links
p  Tend to be cheaper
n  Circuit concentration
n  Choose from multiple suppliers

p  Think big
n  More redundancy
n  Less impact of upgrades
n  Less impact of failures

31
Metropolitan Area Backbone Links

POP Two
Metropolitan Links

POP Three POP One

Metropolitan Links

Traditional Point to Point Links 32

Upstream Connectivity
and Peering

33
Transits
p  Transit provider is another autonomous system
which is used to provide the local network with
access to other networks
n  Might be local or regional only
n  But more usually the whole Internet
p  Transit providers need to be chosen wisely:
n  Only one
p  no redundancy
n  Too many
p  more difficult to load balance
p  no economy of scale (costs more per Mbps)
p  hard to provide service quality
p  Recommendation: at least two, no more
than three
Common Mistakes
p  ISPs sign up with too many transit providers
n  Lots of small circuits (cost more per Mbps than larger
ones)
n  Transit rates per Mbps reduce with increasing transit
bandwidth purchased
n  Hard to implement reliable traffic engineering that
doesn’t need daily fine tuning depending on customer
activities
p  No diversity
n  Chosen transit providers all reached over same satellite
or same submarine cable
n  Chosen transit providers have poor onward transit and
peering
Peers
p  A peer is another autonomous system with which
the local network has agreed to exchange locally
sourced routes and traffic
p  Private peer
n  Private link between two providers for the purpose of
interconnecting
p  Public peer
n  Internet Exchange Point, where providers meet and
freely decide who they will interconnect with
p  Recommendation: peer as much as possible!
Common Mistakes
p  Mistaking a transit provider’s “Exchange”
business for a no-cost public peering point
p  Not working hard to get as much peering
as possible
n  Physically near a peering point (IXP) but not
present at it
n  (Transit sometimes is cheaper than peering!!)

p  Ignoring/avoidingcompetitors because
they are competition
n  Even though potentially valuable peering
partner to give customers a better experience
Private Interconnection
p  Twoservice providers agree to
interconnect their networks
n  They exchange prefixes they originate into the
routing system (usually their aggregated
address blocks)
n  They share the cost of the infrastructure to
interconnect
p  Typically each paying half the cost of the link (be it
circuit, satellite, microwave, fibre,…)
p  Connected to their respective peering routers

n  Peering routers only carry domestic prefixes

38
Private Interconnection
Upstream
Upstream

PR ISP2
PR

ISP1

p  PR = peering router
n  Runs iBGP (internal) and eBGP (with peer)
n  No default route
n  No “full BGP table”
n  Domestic prefixes only
p  Peering router used for all private interconnects39
Public Interconnection
p  Serviceprovider participates in an
Internet Exchange Point
n  It exchanges prefixes it originates into the
routing system with the participants of the IXP
n  It chooses who to peer with at the IXP
p  Bi-lateral peering (like private interconnect)
p  Multi-lateral peering (via IXP’s route server)

n  It provides the router at the IXP and provides

the connectivity from their PoP to the IXP
n  The IXP router carries only domestic prefixes

40
Public Interconnection
Upstream
ISP6-PR
ISP5-PR

IXP ISP1-PR ISP1

ISP4-PR

ISP2-PR
ISP3-PR

p  ISP1-PR = peering router of our ISP

n  Runs iBGP (internal) and eBGP (with IXP peers)
n  No default route
n  No “full BGP table”
n  Domestic prefixes only
p  Physically located at the IXP 41
Public Interconnection
p  The ISP’s router IXP peering router needs careful
configuration:
n  It is remote from the domestic backbone
n  Should not originate any domestic prefixes
n  (As well as no default route, no full BGP table)
n  Filtering of BGP announcements from IXP peers (in and
out)
p  Provision of a second link to the IXP:
n  (for redundancy or extra capacity)
n  Usually means installing a second router
p  Connected to a second switch (if the IXP has two more more
switches)
p  Interconnected with the original router (and part of iBGP mesh)
42
Public Interconnection
Upstream
ISP6-PR
ISP1-PR2
ISP5-PR

IXP
ISP1-PR1 ISP1
ISP4-PR

ISP2-PR
ISP3-PR

p  Provision of a second link to the IXP means

considering redundancy in the SP’s backbone
n  Two routers
n  Two independent links
n  Separate switches (if IXP has two or more switches)
43
Upstream/Transit Connection
p  Two scenarios:
n  Transit provider is in the locality
p  Which means bandwidth is cheap, plentiful, easy to
provision, and easily upgraded
n  Transit provider is a long distance away
p  Over undersea cable, satellite, long-haul cross
country fibre, etc
p  Eachscenario has different considerations
which need to be accounted for

44
Local Transit Provider

AR ISP1
BR

Transit

p  BR = ISP’s Border Router

n  Runs iBGP (internal) and eBGP (with transit)
n  Either receives default route or the full BGP table from
upstream
n  BGP policies are implemented here (depending on
connectivity)
n  Packet filtering is implemented here (as required) 45
Distant Transit Provider

AR1

ISP1
Transit BR

AR2

p  BR = ISP’s Border Router

n  Co-located in a co-lo centre (typical) or in the upstream provider’s
premises
n  Runs iBGP with rest of ISP1 backbone
n  Runs eBGP with transit provider router(s)
n  Implements BGP policies, packet filtering, etc
n  Does not originate any domestic prefixes
46
Distant Transit Provider
p  Positioninga router close to the Transit
Provider’s infrastructure is strongly
encouraged:
n  Long haul circuits are expensive, so the router
allows the ISP to implement appropriate
filtering first
n  Moves the buffering problem away from the
Transit provider
n  Remote co-lo allows the ISP to choose another
transit provider and migrate connections with
minimum downtime
47
Distant Transit Provider
p  Other points to consider:
n  Does require remote hands support
n  (Remote hands would plug or unplug cables,
power cycle equipment, replace equipment, etc
as instructed)
n  Appropriate support contract from equipment
vendor(s)
n  Sensible to consider two routers and two long-
haul links for redundancy

48
Distant Transit Provider

AR1

ISP1
Transit BR1

AR2
BR2

p  Upgrade scenario:
n  Provision two routers
n  Two independent circuits
n  Consider second transit provider and/or turning up at
an IXP
49
Summary
p  Design considerations for:
n  Private interconnects
p  Simple private peering
n  Public interconnects
p  Router co-lo at an IXP
n  Local transit provider
p  Simple upstream interconnect
n  Long distance transit provider
p  Router remote co-lo at datacentre or Transit
premises

50
Upstream Connectivity
and Peering Case Study
How Seacom chose their
international peering locations
and transit providers

51
Objective
p  Obtain high grade Internet connectivity for
the wholesale market in Africa to the rest
of the world
p  Emphasis on:
n  Reliability
n  Interconnectivity density
n  Scalability

52
Metrics Needed in Determining
Solution (1)
p  Focusing on operators that cover the destinations
mostly required by Africa
n  i.e., English-speaking (Europe, North America)
p  Include providers with good connectivity into
South America and the Asia Pacific.
p  Little need for providers who are strong in the
Middle East, as demand from Africa for those
regions is very, very low.

53
Metrics Needed in Determining
Solution (2)
p  Split the operators between Marseille (where the
SEACOM cable lands) and London (where there is
good Internet density)
n  To avoid outages due to backhaul failure across Europe
n  And still maintain good access to the Internet
p  Look at providers who are of similar size so as
not to fidget too much (or at all) with BGP tuning.
p  The providers needed to support:
n  10Gbps ports
n  Bursting bandwidth/billing
n  Future support for 100Gbps or N x 10Gbps

54
Metrics Needed in Determining
Solution (3)
p  Implement peering at major exchange points in
Europe
n  To off-set long term operating costs re: upstream
providers.

55
Implementing Solution
p  Connected to Level(3) and GT-T (formerly
Inteliquent, formerly Tinet) in Marseille
p  Connected to NTT and TeliaSonera in London
p  Peered in London (LINX)
p  Peered in Amsterdam (AMS-IX)
p  BGP setup to prefer traffic being exchanged at
LINX and AMS-IX
p  BGP setup to prefer traffic over the upstreams
that we could not peer away
p  No additional tuning done on either peered or
transit traffic, i.e., no prepending, no de-
aggregation, etc. All traffic setup to flow naturally56
End Result
p  50% of traffic peered away in less than 2x
months of peering at LINX and AMS-IX
p  50% of traffic handled by upstream providers
p  Equal traffic being handled by Level(3) and GT-T
in Marseille
p  Equal traffic being handled by TeliaSonera and
NTT in London
p  Traffic distribution ratios across all the transit
providers is some 1:1:0.9:0.9
p  This has been steady state for the last 12x
months
n  No BGP tuning has been done at all 57
Addressing

58
Where to get IP addresses and AS
numbers
p  Your upstream ISP
p  Africa
n  AfriNIC – http://www.afrinic.net
p  Asia and the Pacific
n  APNIC – http://www.apnic.net
p  North America
n  ARIN – http://www.arin.net
p  Latin America and the Caribbean
n  LACNIC – http://www.lacnic.net
p  Europe and Middle East
n  RIPE NCC – http://www.ripe.net/info/ncc
59
Internet Registry Regions

60
Getting IP address space
p  Take part of upstream ISP’s PA space
or
p  Become a member of your Regional Internet
Registry and get your own allocation
n  Require a plan for a year ahead
n  General policies are outlined in RFC2050, more
specific details are on the individual RIR website
p  There is no more IPv4 address space at IANA
n  APNIC and RIPE NCC are now in their “final /8” IPv4
delegation policy phase
n  Limited IPv4 available
n  IPv6 allocations are simple to get in most RIR regions
61
What about RFC1918 addressing?
p  RFC1918 defines IPv4 addresses reserved for
private Internets
n  Not to be used on Internet backbones
n  http://www.ietf.org/rfc/rfc1918.txt
p  Commonly used within end-user networks
n  NAT used to translate from private internal to public
external addressing
n  Allows the end-user network to migrate ISPs without a
major internal renumbering exercise
p  ISPs must filter RFC1918 addressing at their
network edge
n  http://www.cymru.com/Documents/bogon-
list.html 62
What about RFC1918 addressing?
p  There is a long list of well known problems:
n  http://www.rfc-editor.org/rfc/rfc6752.txt
p  Including:
n  False belief it conserves address space
n  Adverse effects on Traceroute
n  Effects on Path MTU Discovery
n  Unexpected interactions with some NAT implementations
n  Interactions with edge anti-spoofing techniques
n  Peering using loopbacks
n  Adverse DNS Interaction
n  Serious Operational and Troubleshooting issues
n  Security Issues
p  false sense of security, defeating existing security 63
techniques
Private versus Globally Routable IP
Addressing
p  Infrastructure Security: not improved by using
private addressing
n  Still can be attacked from inside, or from customers, or
by reflection techniques from the outside
p  Troubleshooting: made an order of magnitude
harder
n  No Internet view from routers
n  Other ISPs cannot distinguish between down and broken
p  Summary:
n  ALWAYS use globally routable IP addressing for ISP
Infrastructure

64
Addressing Plans – ISP
Infrastructure
p  Address block for router loop-back interfaces
p  Address block for infrastructure
n  Per PoP or whole backbone
n  Summarise between sites if it makes sense
n  Allocate according to genuine requirements, not historic
classful boundaries
p  Similar allocation policies should be used for IPv6
as well
n  ISPs just get a substantially larger block (relatively) so
assignments within the backbone are easier to make

65
Addressing Plans – Customer
p  Customers are assigned address space
according to need
p  Should not be reserved or assigned on a
per PoP basis
n  ISP iBGP carries customer nets
n  Aggregation not required and usually not
desirable

66
Addressing Plans – ISP Infrastructure
p  Phase One
223.10.0.0/21

223.10.0.1 223.10.6.255 /24

Customer assignments Infrastructure Loopbacks

p  Phase Two
223.10.0.0/20

223.10.0.1
223.10.5.255 /24 /24
223.10.15.255

Original assignments New Assignments 67

Addressing Plans
Planning
p  Registrieswill usually allocate the next
block to be contiguous with the first
allocation
n  Minimum allocation could be /21
n  Very likely that subsequent allocation will
make this up to a /20
n  So plan accordingly

68
Addressing Plans (contd)
p  Document infrastructure allocation
n  Eases operation, debugging and management
p  Document customer allocation
n  Contained in iBGP
n  Eases operation, debugging and management
n  Submit network object to RIR Database

69
Routing Protocols

70
Routing Protocols
p  IGP – Interior Gateway Protocol
n  Carries infrastructure addresses, point-to-point
links
n  Examples are OSPF, ISIS,...

p  EGP – Exterior Gateway Protocol

n  Carries customer prefixes and Internet routes
n  Current EGP is BGP version 4

p  No connection between IGP and EGP

71
Why Do We Need an IGP?
p  ISP backbone scaling
n  Hierarchy
n  Modular infrastructure construction
n  Limiting scope of failure
n  Healing of infrastructure faults using dynamic
routing with fast convergence

72
Why Do We Need an EGP?
p  Scaling to large network
n  Hierarchy
n  Limit scope of failure

p  Policy
n  Control reachability to prefixes
n  Merge separate organizations
n  Connect multiple IGPs

73
Interior versus Exterior Routing
Protocols
p  Interior p  Exterior
n  Automatic neighbour n  Specifically configured
discovery peers
n  Generally trust your IGP n  Connecting with outside
routers networks
n  Prefixes go to all IGP n  Set administrative
routers boundaries
n  Binds routers in one AS n  Binds AS’s together
together

74
Interior versus Exterior Routing
Protocols
p  Interior p  Exterior
n  Carries ISP n  Carries customer
infrastructure addresses prefixes
only n  Carries Internet
n  ISPs aim to keep the prefixes
IGP small for efficiency n  EGPs are independent
and scalability of ISP network topology

75
Hierarchy of Routing Protocols
Other ISPs
BGP4

BGP4
and OSPF/ISIS

BGP4 Static/BGP4

Customers
IXP
76
Routing Protocols:
Choosing an IGP
p  OSPF and ISIS have very similar properties
n  Review the “ISIS vs OSPF” presentation
p  Which to choose?
n  Choose which is appropriate for your operators’
experience
n  In most vendor releases, both OSPF and ISIS have
sufficient “nerd knobs” to tweak the IGP’s behaviour
n  OSPF runs on IP
n  ISIS runs on infrastructure, alongside IP
n  ISIS supports both IPv4 and IPv6
n  OSPFv2 (IPv4) plus OSPFv3 (IPv6)

77
Routing Protocols:
IGP Recommendations
p  Keep the IGP routing table as small as possible
n  If you can count the routers and the point-to-point links
in the backbone, that total is the number of IGP entries
you should see
p  IGP details:
n  Should only have router loopbacks, backbone WAN
point-to-point link addresses, and network addresses of
any LANs having an IGP running on them
n  Strongly recommended to use inter-router
authentication
n  Use inter-area summarisation if possible

78
Routing Protocols:
More IGP recommendations
p  To fine tune IGP table size more, consider:
n  Using “ip unnumbered” on customer point-to-
point links – saves carrying that /30 in IGP
p  (If customer point-to-point /30 is required for
monitoring purposes, then put this in iBGP)
n  Use contiguous addresses for backbone WAN
links in each area – then summarise into
backbone area
n  Don’t summarise router loopback addresses –
as iBGP needs those (for next-hop)
n  Use iBGP for carrying anything which does not
contribute to the IGP Routing process
79
Routing Protocols:
iBGP Recommendations
p  iBGPshould carry everything which
doesn’t contribute to the IGP routing
process
n  Internet routing table
n  Customer assigned addresses
n  Customer point-to-point links
n  Access network dynamic address pools,
passive LANs, etc

80
Routing Protocols:
More iBGP Recommendations
p  Scalable iBGP features:
n  Use neighbour authentication
n  Use peer-groups to speed update process and
for configuration efficiency
n  Use communities for ease of filtering
n  Use route-reflector hierarchy
p  Route reflector pair per PoP (overlaid clusters)

81
Security

82
Security
p  ISP Infrastructure security
p  ISP Network security
p  Security is not optional!
p  ISPs need to:
n  Protect themselves
n  Help protect their customers from the Internet
n  Protect the Internet from their customers
p  The following slides are general recommendations
n  Do more research on security before deploying any
network

83
ISP Infrastructure Security
p  Router & Switch Security
n  Use Secure Shell (SSH) for device access &
management
p  Do NOT use Telnet
n  Device management access filters should only
allow NOC and device-to-device access
p  Do NOT allow external access
n  Use TACACS+ for user authentication and
authorisation
p  Do NOT create user accounts on routers/switches

84
ISP Infrastructure Security
p  Remote access
n  For Operations Engineers who need access
while not in the NOC
n  Create an SSH server host (this is all it does)
p  Or a Secure VPN access server
n  Ops Engineers connect here, and then they can
access the NOC and network devices

85
ISP Infrastructure Security
p  Other network devices?
n  These probably do not have sophisticated security
techniques like routers or switches do
n  Protect them at the LAN or point-to-point ingress (on
router)
p  Servers and Services?
n  Protect servers on the LAN interface on the router
n  Consider using iptables &c on the servers too
p  SNMP
n  Apply access-list to the SNMP ports
n  Should only be accessible by management system, not
the world
86
ISP Infrastructure Security
p  General Advice:
n  Routers, Switches and other network devices
should not be contactable from outside the AS
n  Achieved by blocking typical management
access protocols for the infrastructure address
block at the network perimeter
p  E.g. ssh, telnet, http, snmp,…
n  Use the ICSI Netalyser to check access levels:
p  http://netalyzr.icsi.berkeley.edu
n  Don’t block everything: BGP, traceroute
and ICMP still need to work!
87
ISP Network Security
p  Effective filtering
n  Protect network borders from “traffic which
should not be on the public Internet”, for
example:
p  LAN protocols (eg netbios)
p  Well known exploit ports (used by worms and
viruses)
p  Drop traffic arriving and going to private and non-
routable address space (IPv4 and IPv6)
n  Achieved by packet filters on border routers
p  Remote trigger blackhole filtering

88
ISP Network Security – RTBF
p  Remote trigger blackhole filtering
n  ISP NOC injects prefixes which should not be accessible
across the AS into the iBGP
n  Prefixes have next hop pointing to a blackhole address
n  All iBGP speaking backbone routers configured to point
the blackhole address to the null interface
n  Traffic destined to these blackhole prefixes are dropped
by the first router they reach
p  Application:
n  Any prefixes (including RFC1918) which should not have
routability across the ISP backbone

89
ISP Network Security – RTBF
p  Remote trigger blackhole filtering example:
n  Origin router:
router bgp 64509
redistribute static route-map black-hole-trigger
!
ip route 10.5.1.3 255.255.255.255 Null0 tag 66
!
route-map black-hole-trigger permit 10
match tag 66
set local-preference 1000
set community no-export
set ip next-hop 192.0.2.1
!

n  iBGP speaking backbone router:

ip route 192.0.2.1 255.255.255.255 null0
90
ISP Network Security – RTBF
p  Resulting routing table entries:
gw1#sh ip bgp 10.5.1.3
BGP routing table entry for 10.5.1.3/32, version 64572219
Paths: (1 available, best #1, table Default-IP-Routing-Table)
Not advertised to any peer
Local
192.0.2.1 from 1.1.10.10 (1.1.10.10)
Origin IGP, metric 0, localpref 1000, valid, internal, best
Community: no-export
gw1#sh ip route 10.5.1.3
Routing entry for 10.5.1.3/32
Known via "bgp 64509", distance 200, metric 0, type internal
Last update from 192.0.2.1 00:04:52 ago
Routing Descriptor Blocks:
* 192.0.2.1, from 1.1.10.10, 00:04:52 ago
Route metric is 0, traffic share count is 1
AS Hops 0
91
ISP Network Security – uRPF
p  Unicast Reverse Path Forwarding
p  Strongly recommended to be used on all
customer facing static interfaces
n  BCP 38 (tools.ietf.org/html/bcp38)
n  Blocks all unroutable source addresses the
customer may be using
n  Inexpensive way of filtering customer’s connection
(when compared with packet filters)
p  Can be used for multihomed connections too, but
extreme care required

92
What is uRPF?
FIB:
172.16.1.0/24 fa0/0
192.168.1.0/24 se0/1
src=172.16.1.1 fa0/0 se0/1
router
src=192.168.1.1

p  Router compares source address of incoming

packet with FIB entry
n  If FIB entry interface matches incoming interface, the
packet is forwarded
n  If FIB entry interface does not match incoming interface,
the packet is dropped 93
Security Summary
p  Implement RTBF
n  Inside ISP backbone
n  Make it available to BGP customers too
p  They can send you the prefix you need to block with a
special community attached
p  You match on that community, and set the next-hop to the
null address
p  Implement uRPF
n  For all static customers
p  Use SSH for device access
p  Use TACACS+ for authentication

94
Out of Band Management

95
Out of Band Management
p  Not optional!
p  Allows access to network equipment in
times of failure
p  Ensures quality of service to customers
n  Minimises downtime
n  Minimises repair time
n  Eases diagnostics and debugging

96
Out of Band Management
p  OoB Example – Access server:
n  modem attached to allow NOC dial in
n  console ports of all network equipment
connected to serial ports
n  LAN and/or WAN link connects to network
core, or via separate management link to NOC
p  Fullremote control access under all
circumstances

97
Out of Band Network
Equipment Rack Equipment Rack

Router, switch
and ISP server
consoles

(Optional) Out of band

WAN link to other PoPs

Modem – access
to PSTN for out of
band dialin

Ethernet
98
to the NOC
Out of Band Management
p  OoB Example – Statistics gathering:
n  Routers are NetFlow and syslog enabled
n  Management data is congestion/failure
sensitive
n  Ensures management data integrity in case of
failure
p  Fullremote information under all
circumstances

99
Test Laboratory

100
Test Laboratory
p  Designed to look like a typical PoP
n  Operated like a typical PoP
p  Used to trial new services or new software
under realistic conditions
p  Allows discovery and fixing of potential
problems before they are introduced to
the network

101
Test Laboratory
p  Some ISPs dedicate equipment to the lab
p  Other ISPs “purchase ahead” so that
today’s lab equipment becomes
tomorrow’s PoP equipment
p  Other ISPs use lab equipment for “hot
spares” in the event of hardware failure

102
Test Laboratory
p  Can’t afford a test lab?
n  Set aside one spare router and server to trial
new services
n  Never ever try out new hardware, software or
services on the live network
p  Every major ISP in the US and Europe has
a test lab
n  It’s a serious consideration

103
 Operational
Considerations

104
Operational Considerations

Why design the world’s best

network when you have not
thought about what operational
good practices should be
implemented?
105
Operational Considerations
Maintenance
p  Never work on the live network, no matter how
trivial the modification may seem
n  Establish maintenance periods which your customers are
aware of
p  e.g. Tuesday 4-7am, Thursday 4-7am
p  Never do maintenance on the last working day
before the weekend
n  Unless you want to work all weekend cleaning up
p  Never do maintenance on the first working day
after the weekend
n  Unless you want to work all weekend preparing

106
Operational Considerations
Support
p  Differentiate
between customer support
and the Network Operations Centre
n  Customer support fixes customer problems
n  NOC deals with and fixes backbone and
Internet related problems
p  Network Engineering team is last resort
n  They design the next generation network,
improve the routing design, implement new
services, etc
n  They do not and should not be doing support!

107
Operational Considerations
NOC Communications
p  NOC should know contact details for
equivalent NOCs in upstream providers
and peers
p  Or consider joining the INOC-DBA system
n  Voice over IP phone system using SIP
n  Runs over the Internet
n  www.pch.net/inoc-dba for more information

108
ISP Network Design
Summary

109
ISP Design Summary
p  KEEP IT SIMPLE & STUPID ! (KISS)
p  Simple is elegant is scalable
p  Use Redundancy, Security, and
Technology to make life easier for yourself
p  Above all, ensure quality of service for
your customers

110
ISP Network Design
ISP Workshops

111