1

Securing Internet Routing

(with RPKI)

2 v1.0
Headlines

• Not so funny L – 1 April 2020

q AS12389 (Rostelecom) hijacks/leaks 8K+
more specifics
§ Facebook, Cloudflare, AWS, Akamai, Google,
Digital Ocean….
§ ~200 ASNs
q Some peers accepted/propagated the leaks:
§ AS20764 (Rascom) à AS174 (Cogent) à AS3356
(Level3)

https://blog.qrator.net/en/how-you-deal-route-leaks_69/

3 v1.0
Headlines

• BGP Optimizers impact Internet – June 2019

q AS13335 hosted sites were not
reachable during the leak
§ About 15% of their global traffic!!
§ ~ 120mins

https://twitter.com/atoonk/status/1143143943531454464/photo/1

4 v1.0
Headlines

• BGP Optimizers impact Internet (contd…)

q What & How?

104.16.16.0/20
AS33154
[DQE] AS3356
[Level3]
104.16.16.0/21
104.16.24.0/21
104.16.16.0/20
AS396531
[ATI] AS13335
104.16.16.0/21
104.16.24.0/21

AS701
Internet
[Verizon]

5 v1.0
Headlines

• Google prefix leaks – Nov 2018

q Google services (G-Suite, Google search
and Google analytics) affected by the leak
§ Traffic dropped at AS4809 (China Telecom)
§ ~ 74mins

6 v1.0
Headlines

• Google prefix leaks (contd…)

q How?
§ AS37282 (MainOne) leaked to AS4809 (CT) at IXPN, who leaked it to
others like AS20485 (TransTelecom)

https://blog.thousandeyes.com/internet-vulnerability-takes-down-google/

7 v1.0
Headlines

• Amazon (AS16509) Route53 hijack – April2018

q AS10279 (eNET) originated more specifics (/24s) of Amazon
Route53’s prefix (205.251.192.0/21)
205.251.192.0/24 ……. 205.251.199.0/24
https://ip-ranges.amazonaws.com/ip-ranges.json

q Its peers, like AS6939 (HE), shared these routes with 100s of their
own peers…

q The motive?
§ During the period, DNS servers in the hijacked range only responded to queries
for myetherwallet.com
§ Responded with addresses associated with AS41995/AS48693

8 v1.0
Headlines

• Route53 hijack (contd…)

q Resolvers querying any Route53
managed names, would ask the
authoritative servers controlled
through the BGP hijack
§ Possibly, used an automated cert issuer
to get a cert for myetherwallet.com

q use _THEIR_ crypto to end-users to

see everything (including passwords)
https://blog.cloudflare.com/bgp-leaks-and-crypto-currencies

9 v1.0
Headlines

• Bharti (AS9498) originates 103.0.0.0/10 - Dec 2017

q ~ 2 days
q No damage done – more than 8K specific routes!

• Google brings down Internet in Japan - Aug 2017

q ~ 24 hours
q Google (AS15169) leaked >130K prefixes to Verizon (AS701) in
Chicago
§ Normally ~ 50 prefixes
§ ~25K of those were NTT OCN’s (AS4713) more specifics
§ which was leaked onwards to KDDI and IIJ (and accepted)

q Everyone who received the leaked more specifics, preferred the

Verizon-Google path to reach NTT OCN!

10 v1.0
Headlines

• Google leak (contd…)

After leak
(JP->JP)

Before leak (JP->JP)

After leak
(EU->EU)

https://dyn.com/blog/large-bgp-leak-by-google-disrupts-internet-in-japan/

11 v1.0
Headlines

• YouTube (AS36561) Incident - Feb 2008

q ~ 2 hours
q AS17557 (PT) announced 208.65.153.0/24 (208.65.152.0/22)
§ Propagated by AS3491 (PCCW)

12 v1.0
Why do we keep seeing these?

• Because NO ONE is in charge?

q No single authority model for the Internet
q No reference point for what’s right in routing

13 v1.0
Why do we keep seeing these?

• Routing works by RUMOUR

q Tell what you know to your neighbors, and Learn what your
neighbors know
q Assume everyone is correct (and honest)
§ Is the originating network the rightful owner?

14 v1.0
Why do we keep seeing these?

• Routing is VARIABLE
q The view of the network depends on where you are
§ Different routing outcomes at different locations

q ~ no reference view to compare the local view L

15 v1.0
Why do we keep seeing these?

• Routing works in REVERSE

q Outbound advertisement affects inbound traffic
q Inbound (Accepted) advertisement influence outbound traffic

16 v1.0
Why do we keep seeing these?

• As always, there is no E-bit (evil!)

q RFC3514 J

q A bad routing update does not identify itself as BAD

q All we can do is identify GOOD updates
q But how do we identify what is GOOD???

17 v1.0
Why should we worry?

• Because it’s just so easy to do bad in routing!

By Source (WP:NFCC#4), Fair use,

https://en.wikipedia.org/w/index.php?curid=42515224

18 v1.0
How do we address these?

• Good Hygiene ~ Filter Filter Filter!

q your peers, upstream(s), and customers
§ Prefix filters/Prefix limit
§ AS-PATH filters/AS-PATH limit
§ RFC 8212 – BGP default reject or something similar

19 v1.0
Current practice

LOA Check

Peering/Transit Filters (in/out)

Request

20 v1.0
Tools & Techniques

LOA Check

Whois Letter of
IRR (RPSL)
(manual) Authority

21 v1.0
Tools & Techniques

• Look up whois
q verify holder of
a resource

22 v1.0
Tools & Techniques

• Ask for a Letter of Authority

q Absolve from any liabilities

23 v1.0
Tools & Techniques

• Look up (or ask to enter)

details in internet routing
registries (IRR)
q describes route origination and
inter-AS routing policies

24 v1.0
Tools & Techniques

• IRR
q Helps auto generate network
(prefix/as-path) filters using RPSL
tools
§ Filter out route advertisements not
described in the registry

25 v1.0
Tools & Techniques

• Problem(s) with IRR

q No single authority model
§ How do I know if a RR entry is genuine and correct?
§ How do I differentiate between a current and a lapsed entry?
q Many RRs
§ If two RRs contain conflicting data, which one do I trust and use?
q Incomplete data - Not all resources are registered in an IRR
§ If a route is not in a RR, is the route invalid or is the RR just missing data?
q Scaling
§ How do I apply IRR filters to upstream(s)?

26 v1.0
Tools & Techniques

• Automating network filters (IRR filters) - Caution

q IRR filters only as good as the correctness of the IRR entries

§ Might require manual overrides and offline verification of resource holders

§ Good idea to use specific sources (-S in bgpq3, -s in rtconfig) when

generating filters, assuming mirrors are up to date

27 v1.0
Back to basics – identify GOOD

• Could we use a digital signature to convey the authority to

use?
q Private key to sign the authority, and
q Public key to validate the authority

• ~ If the holder of the resource has the private key, it can

sign/authorize the use of the resource

28 v1.0
How about trust?

• How do we build a chain of trust in this framework??

q Follow the resource allocation/delegation hierarchy

IANA à RIRs à NIRs/LIRs à End Holders
|
V
End Holders

§ To describe the address allocation using digital certificates

29 v1.0
RPKI Chain of Trust

Trust Anchor
Allocation Certificate
Hierarchy
Cert
(CA)
ARIN AFRINIC APNIC LACNIC RIPE-NCC

Cert
Certificate (CA)
chain NIR
mirrors the
allocation
Cert Cert Cert
hierarchy (EE)
Cert
(EE) (EE) (EE)
ISP ISP ISP ISP

Image 4

30 v1.0
RPKI Chain of Trust

• RIRs hold a self-signed root certificate for all the resources

they have in the registry
§ they are the Trust Anchor for the system

• The root certificate signs the resource certificates for end-

holder allocations
§ binds the resources to the end-holders public key

• Any attestations signed by the end-holder’s private key, can

now be validated up the chain of trust

31 v1.0
RPKI profile ~ Resource Certificates

CA
RFC 3779 extensions – binds a list
Signed by parent’s private key

X.509 CERT •
of resources (IPv4/v6,ASN) to the
RFC 3779 subject of the certificate (private
EXTENSION key holder)
IP RESOURCES
(ADDRESS & ASN) • SIA (subject information access)
contains a URI that identifies the
SIA
(URI WHERE THIS PUBLISHES)
publication point of the objects
signed by the subject of the cert.
OWNER’S PUBLIC KEY

32 v1.0
Resource Certificates

• When an address holder A (*IRs) allocates resources (IP

address/ASN) to B (end holders)

q A issues a resource certificate that binds the allocated address with

B’s public key, all signed by A’s (CA) private key

q The resource certificate proves the holder of the private key (B) is
the legitimate holder of the number resource!

33 v1.0
Route Origin Authorization (ROA)

• (B) can now sign authorities using its private key

q which can be validated by any third party against the TA

• For routing, the address holder can authorize a network

(ASN) to originate a route, and sign this permission with its
private key (~ROA)

34 v1.0
Route Origin Authorization (ROA)

• Digitally signed object

q Binds list of prefixes and the nominated ASN

q can be verified cryptographically

Prefix 203.176.32.0/19
Max-length /24
Origin ASN AS17821

• ** Multiple ROAs can exist for the same prefix

35 v1.0
What can RPKI do?

• Authoritatively proof:
q Who is the legitimate owner of an address, and
q Identify which ASNs have the permission from the holder to
originate the address

• Can help:
q prevent route hijacks/mis-origination/misconfiguration

36 v1.0
RPKI Components

• Issuing Party – Internet Registries (*IRs)

q Certificate Authority (CA) that issues resource certificates to end-holders
q Publishes the objects (ROAs) signed by the resource certificate holders

APNIC
publication
RPKI Repository
Engine
rpki.apnic.net

MyAPNIC GUI

37 v1.0
RPKI Components

• Relying Party (RP)

q RPKI Validator that gathers data (ROA) from the distributed RPKI repositories
q Validates each entry’s signature against the TA to build a “Validated cache”

rsync/RRDP
IANA Repo

rpki.apnic.net
rsync/RRDP
APNIC RIPE Repo
Repo
rsync/RRDP RP
(RPKI Validated
rsync/RRDP Validator) Cache
LIR Repo LIR Repo

38 v1.0
RPKI Service Models

• Hosted model:
q The RIR (APNIC) runs the CA functions on members’ behalf
§ Manage keys, repo, etc.
§ Generate certificates for resource delegations

• Delegated model:
q Member becomes the CA (delegated by the parent CA) and operates
the full RPKI system
§ JPNIC, TWNIC, CNNIC (IDNIC in progress)

39 v1.0
Route Origin Validation (ROV)

65551 65550 17821

2406:6400::/48

2406:6400::/48 65551 65550 17821 i

YOU
2406:6400::/48 65553 65552 i

2406:6400::/48

65553 65552

40 v1.0
Route Origin Validation (ROV)

Global
(RPKI)
Repository
65551 65550 17821
ROA
rs 2406:6400::/32-48
yn
c/ 17821
RR 2406:6400::/48
DP
RPKI-to-Router 2406:6400::/48 65551 65550 17821 i Valid
(RTR)
2406:6400::/48 65553 65552 i Invalid
2406:6400::/32-48
Validator
17821

2406:6400::/48

65553 65552

41 v1.0
Route Origin Validation

• Router fetches ROA information from the validated RPKI cache

q Crypto stripped by the validator

• BGP checks each received BGP update against the ROA

information and labels them

42 v1.0
Validation States

• Valid
q the prefix (prefix length) and AS pair found in the database.

• Invalid
q prefix is found, but origin AS is wrong, OR
q the prefix length is longer than the maximum length

• Not Found/Unknown
q No valid ROA found
§ Neither valid nor invalid (perhaps not created)

43 v1.0
Validation States

ASN Prefix Max Length

ROA 65420 10.0.0.0/16 18

BGP Routes

ASN Prefix RPKI State

65420 10.0.0.0/16 VALID

65420 10.0.128.0/17 VALID

65421 10.0.0.0/16 INVALID

65420 10.0.10.0/24 INVALID

65430 10.0.0.0/8 NOT FOUND

44 v1.0
Possible actions - RPKI states

• Do Nothing
• Tag
q If you have downstream customers or run a route server (IXP)
q Ex:
§ Valid (ASN:65XX1)
§ Not Found (ASN:65XX2)
§ Invalid (ASN:65XX3)

• Modify preference values

q RFC7115

• Drop Invalids
q ~6K IPv4 & ~1.5K IPv6 routes (might want to check your top flows)
https://rpki-monitor.antd.nist.gov/index.php?p=0&s=0

45 v1.0
Are ROAs enough?

• What if I forge the origin AS in the AS path?

q Would be accepted as good – pass origin validation!

• Which means, we need to secure the AS path as well

q AS path validation (per-prefix)

• We can use RPKI certificates for this

46 v1.0
AS keys (per-router keys)
CA
APNIC Training

202.125.96.0/24
AS45192

Public Key
Cert
(CA)

APNIC Prefix EE AS Cert CA

CA 202.125.96.0/24 AS45192
APNIC Training

Cert
202.125.96.0/24 Public Key Public Key
(CA) AS45192 Encodes
APNIC Training ASN and
Public Key ROA router ID
Router EE
202.125.96.0/24 Router EE
Router EE
AS45192
AS45192
rtr-00
AS45192
rtr-00
AS45192 rtr-00
Public Key
Public Key
Public Key
47 v1.0
BGPsec (RFC8205)
AS1 -> AS2
(Signed AS1)
AS2->AS3
(signed AS2) AS3

AS1 AS2
AS1 -> AS2
AS1 -> AS2
(Signed AS1) (Signed AS1)
AS2->AS4 AS4
(signed AS2)

§ AS1 router crypto signs the message to AS2

§ AS2 router signs the message to AS3 and AS4, encapsulating AS1’s message

q A BGPsec speaker validates the received update by checking:

§ If there is a ROA that describes the prefix and origin AS
§ If the received AS path can be validated as a chain of signatures (for each AS
in the AS path) using the AS keys
48 v1.0
So why NOT BGPsec?

• Cannot have partial adoption

q Cannot jump across non-participating networks

• More HW resources
q CPU - high crypto overhead to validate signatures, and
q Memory
§ Updates in BGPsec would be per prefix
§ New attributes carrying signatures and certs/key IDs for every AS in the AS
path

• No clarity on how to distribute the collection of certificates

required to validate the signatures

49 v1.0
AS Provider Authorization (draft but promising)

• ASPA is digitally signed object that binds

q a Set of Provider ASNs to a Customer ASN (for a specific AFI),

• For Routing, the ASPA is an attestation

q that the AS holder (Customer ASN) has authorized the set of
provider ASNs to propagate its announcements onwards….

50 v1.0
ASPA Validation/Verification ~ simplified

• For a received route (v4/v6):

q If there is no valid ASPA for the Customer AS (AS(0)) UNKNOWN

q If there is an ASPA with the customer AS, and:

§ if AS(I) in the AS_PATH attribute (AS_SEQ{AS(I), AS(I-1)}) is in the SPAS VALID

§ Else, INVALID

51 v1.0
RPKI Further Reading

X.509 PKI Certificates

5280

Extensions for IP Addresses and ASNs

3779

Resource Public Key Infrastructure

6481-
6493

52 v1.0
Acknowledgement

• Geoff Huston, APNIC

• Randy Bush, IIJ Labs/Arrcus
5280

53 v1.0
Implementation

54 v1.0
1. Create & publish your ROA

• Login MyAPNIC
§ Need to activate the RPKI engine to create ROAs
§ Go to Resources à Resource certification à RPKI (see image below)

55 v1.0
Create & publish your ROA

• Then go to the Routes page

§ Go to Resources à Route Management à Routes (see image below)

https://www.apnic.net/wp-content/uploads/2017/12/ROUTE_MANAGEMENT_GUIDE.pdf
56 v1.0
Create (publish) your ROA

• Select Create route (as shown below)

57 v1.0
Create (publish) your ROA

• Example for IPv6 below

58 v1.0
Create (publish) your ROA

59 v1.0
Create (publish) your ROA

• Example for IPv4

60 v1.0
Create (publish) your ROA

• Your ROAs are

ready!

61 v1.0
Check your ROA

http://nong.rand.apnic.net:8080/roas

62 v1.0
Check your ROA

# whois -h rr.ntt.net 2001:df2:ee00::/48

route6: 2001:df2:ee00::/48
descr: RPKI ROA for 2001:df2:ee00::/48
remarks: This route object represents routing data retrieved from the RPKI
remarks: The original data can be found here: https://rpki.gin.ntt.net/r/AS131107/2001:df2:ee00::/48
remarks: This route object is the result of an automated RPKI-to-IRR conversion process.
remarks: maxLength 48
origin: AS131107
mnt-by: MAINT-JOB
changed: [email protected] 20180802
source: RPKI # Trust Anchor: APNIC RPKI Root

63 v1.0
Check your ROA

# whois -h whois.bgpmon.net 2001:df2:ee00::/48

Prefix: 2001:df2:ee00::/48
Prefix description: APNICTRAINING-DC
Country code: AU
Origin AS: 131107
Origin AS Name: APNICTRAINING LAB DC
RPKI status: ROA validation successful
First seen: 2016-06-30
Last seen: 2018-01-21
Seen by #peers: 97

# whois -h whois.bgpmon.net " --roa 131107 2001:df2:ee00::/48"

------------------------
ROA Details
------------------------
Origin ASN: AS131107
Not valid Before: 2016-09-07 02:10:04
Not valid After: 2020-07-30 00:00:00 Expires in 2y190d9h34m23.2000000029802s
Trust Anchor: rpki.apnic.net
Prefixes: 2001:df2:ee00::/48 (max length /48) 202.125.96.0/24 (max length /24)

64 v1.0
Check your ROA

https://bgp.he.net/

65 v1.0
Check your ROA

https://rpki.cloudflare.com/

66 v1.0
Global ROA Status

67 v1.0
Global ROA Status

68 v1.0
Rise of Invalids L

https://blog.apnic.net/2020/04/10/rise-of-the-invalids/
69 v1.0
2. Deploy RPKI Validator

• Many options:
q RIPE RPKI Validator
https://www.ripe.net/manage-ips-and-asns/resource-management/certification/tools-and-resources

q Dragon Research Labs RPKI Toolkit

https://github.com/dragonresearch/rpki.net

q Routinator
https://github.com/NLnetLabs/routinator

q OctoRPKI & GoRTR (Cloudflare’s RPKI toolkit)

https://github.com/cloudflare/cfrpki

q Fort (NIC Mexico’s Validator)

https://github.com/NICMx/FORT-validator

70 v1.0
3. Router Configuration (IOS)

• Establishing session with the validator

router bgp 131107
bgp rpki server tcp <validator-IP> port <323/8282/3323> refresh 120

• Note:
q Cisco IOS by default does not include invalid routes for best path selection!
q If you don’t want to drop invalids, we need explicitly tell BGP (under respective address
families)

bgp bestpath prefix-validate allow-invalid

71 v1.0
Configuration (IOS)

• Policies based on validation:

route-map ROUTE-VALIDATION permit 10

match rpki valid
set local-preference 200
!
route-map ROUTE-VALIDATION permit 20
match rpki not-found
set local-preference 100
!
route-map ROUTE-VALIDATION permit 30 OR route-map ROUTE-VALIDATION deny 30
match rpki invalid match rpki invalid
set local-preference 50
!

72 v1.0
Configuration (IOS)

• Apply the route-map to inbound updates

router bgp 131107
!---output omitted-----!
address-family ipv4
bgp bestpath prefix-validate allow-invalid
neighbor X.X.X.169 activate
neighbor X.X.X.169 route-map ROUTE-VALIDATION in
exit-address-family
!
address-family ipv6
bgp bestpath prefix-validate allow-invalid
neighbor X6:X6:X6:X6::151 activate
neighbor X6:X6:X6:X6::151 route-map ROUTE-VALIDATION in
exit-address-family
!

73 v1.0
3. Router Configuration (JunOS)

• Establishing session with the validator

routing-options {
autonomous-system 131107;
validation {
group rpki-validator {
session <validator-IP> {
refresh-time 120;
port <323/3323/8282>;
local-address X.X.X.253;
}
}
}
}

74 v1.0
Configuration (JunOS)

• Define policies based on the validation states

policy-options {
policy-statement ROUTE-VALIDATION {
term valid {
term invalid {
from {
from {
protocol bgp;
protocol bgp;
validation-database valid;
validation-database invalid;
}
}
then {
then {
local-preference 200;
local-preference 50;
validation-state valid;
validation-state invalid;
accept;
accept;
}
}
}
}
term unknown {
}
from {
}
protocol bgp;
validation-database unknown;
}
OR
then {
then {
local-preference 100;
validation-state invalid;
validation-state unknown;
reject;
accept;
}
}
}

75 v1.0
Router Configuration (JunOS)

• Apply the policy to inbound updates

protocols {
bgp {
group external-peers { group external-peers-v6 {
#output-ommitted #output-ommitted
neighbor X.X.X.1 { neighbor X6:X6:X6:X6::1 {
import ROUTE-VALIDATION; import ROUTE-VALIDATION;
family inet { family inet6 {
unicast; unicast;
} }
} }
} }
}

76 v1.0
RPKI Verification (IOS)

• IOS has only

#sh bgp ipv6 unicast rpki ?

servers Display RPKI cache server information
table Display RPKI table entries

#sh bgp ipv4 unicast rpki ?

servers Display RPKI cache server information
table Display RPKI table entries

77 v1.0
RPKI Verification (IOS)

• Check the RTR session

#sh bgp ipv4 unicast rpki servers

BGP SOVC neighbor is X.X.X.47/323 connected to port 323

Flags 64, Refresh time is 120, Serial number is 1516477445, Session ID is 8871
InQ has 0 messages, OutQ has 0 messages, formatted msg 7826
Session IO flags 3, Session flags 4008
Neighbor Statistics:
Prefixes 45661
Connection attempts: 1
Connection failures: 0
Errors sent: 0
Errors received: 0

Connection state is ESTAB, I/O status: 1, unread input bytes: 0

Connection is ECN Disabled, Mininum incoming TTL 0, Outgoing TTL 255
Local host: X.X.X.225, Local port: 29831
Foreign host: X.X.X.47, Foreign port: 323

78 v1.0
RPKI Verification (IOS)

• Check the RPKI cache

#sh bgp ipv4 unicast rpki table
37868 BGP sovc network entries using 6058880 bytes of memory
39655 BGP sovc record entries using 1268960 bytes of memory

Network Maxlen Origin-AS Source Neighbor

1.9.0.0/16 24 4788 0 202.125.96.47/323
1.9.12.0/24 24 65037 0 202.125.96.47/323
1.9.21.0/24 24 24514 0 202.125.96.47/323
1.9.23.0/24 24 65120 0 202.125.96.47/323

#sh bgp ipv6 unicast rpki table

5309 BGP sovc network entries using 976856 bytes of memory
6006 BGP sovc record entries using 192192 bytes of memory

Network Maxlen Origin-AS Source Neighbor

2001:200::/32 32 2500 0 202.125.96.47/323
2001:200:136::/48 48 9367 0 202.125.96.47/323
2001:200:900::/40 40 7660 0 202.125.96.47/323
2001:200:8000::/35 35 4690 0 202.125.96.47/323

79 v1.0
Check routes (IOS)
#sh bgp ipv4 unicast 202.144.128.0/19
BGP routing table entry for 202.144.128.0/19, version 3814371
Paths: (1 available, best #1, table default)
Advertised to update-groups:
2
Refresh Epoch 15
4826 17660
49.255.232.169 from 49.255.232.169 (114.31.194.12)
Origin IGP, metric 0, localpref 110, valid, external, best
Community: 4826:5101 4826:6570 4826:51011 24115:17660
path 7F50C7CD98C8 RPKI State valid
rx pathid: 0, tx pathid: 0x0

#sh bgp ipv6 unicast 2402:7800::/32

BGP routing table entry for 2402:7800::/32, version 1157916
Paths: (1 available, best #1, table default)
Advertised to update-groups:
2
Refresh Epoch 15
4826
2402:7800:10:2::151 from 2402:7800:10:2::151 (114.31.194.12)
Origin IGP, metric 0, localpref 100, valid, external, best
Community: 4826:1000 4826:2050 4826:2110 4826:2540 4826:2900 4826:5203
path 7F50B266CBD8 RPKI State not found
rx pathid: 0, tx pathid: 0x0

80 v1.0
RPKI Verification (JunOS)

• Check the RPKI cache

>show validation session

Session State Flaps Uptime #IPv4/IPv6 records
X.X.X.46 Up 75 09:20:59 40894/6747

>show validation session 202.125.96.46

Session State Flaps Uptime #IPv4/IPv6 records
X.X.X.46 Up 75 09:21:18 40894/6747

81 v1.0
RPKI Verification (JunOS)

• Check the RPKI cache

>show validation database
RV database for instance master

Prefix Origin-AS Session State Mismatch

1.9.0.0/16-24 4788 202.125.96.46 valid
1.9.12.0/24-24 65037 202.125.96.46 valid
1.9.21.0/24-24 24514 202.125.96.46 valid
1.9.23.0/24-24 65120 202.125.96.46 valid

----------
2001:200::/32-32 2500 202.125.96.46 valid
2001:200:136::/48-48 9367 202.125.96.46 valid
2001:200:900::/40-40 7660 202.125.96.46 valid
2001:200:8000::/35-35 4690 202.125.96.46 valid
2001:200:c000::/35-35 23634 202.125.96.46 valid
2001:200:e000::/35-35 7660 202.125.96.46 valid

Would have been nice if they had per AF!

82 v1.0
RPKI Verification (JunOS)

• Can filter per origin ASN

>show validation database origin-autonomous-system 45192

RV database for instance master

Prefix Origin-AS Session State Mismatch

202.125.97.0/24-24 45192 202.125.96.46 valid
203.176.189.0/24-24 45192 202.125.96.46 valid
2001:df2:ee01::/48-48 45192 202.125.96.46 valid

IPv4 records: 2
IPv6 records: 1

IOS should have something similar!

83 v1.0
Check routes (JunOS)

>show route protocol bgp 202.144.128.0

inet.0: 693024 destinations, 693024 routes (693022 active, 0 holddown, 2

hidden)
+ = Active Route, - = Last Active, * = Both

202.144.128.0/20 *[BGP/170] 1w4d 21:03:04, MED 0, localpref 110, from

202.125.96.254
AS path: 4826 17660 I, validation-state: valid
>to 202.125.96.225 via ge-1/1/0.0

>show route protocol bgp 2001:201::/32

inet6.0: 93909 destinations, 93910 routes (93909 active, 0 holddown, 0

hidden)
+ = Active Route, - = Last Active, * = Both

2001:201::/32 *[BGP/170] 21:18:14, MED 0, localpref 100, from

2001:df2:ee00::1
AS path: 65332 I, validation-state: unknown
>to fe80::dab1:90ff:fedc:fd07 via ge-1/1/0.0

84 v1.0
Propagating RPKI states to iBGP peers

• To avoid every BGP speaker having an RTR session, and

• Ensure all BGP speakers have consistent information

q Relies on non-transitive extended BGP community (RFC8097)

0x4300:0:0
0x4300:0:1
0x4300:0:2
§ Sender (one with RTR session) attaches the extended community to Updates, and receiver derives
the validation states from it
§ Must be enabled on both sender and receiver!

85 v1.0
Propagating RPKI states (IOS)

• Sender (one with RTR session)

router bgp 131107
bgp rpki server tcp <validator-IP> port <323/8282/3323> refresh 120
!---output omitted-----!
address-family ipv4
neighbor X.X.X.X activate
neighbor X.X.X.X send-community both
neighbor X.X.X.X announce rpki state
exit-address-family
!
address-family ipv6
neighbor X6:X6:X6:X6::X6 activate
neighbor X6:X6:X6:X6::X6 send-community both
neighbor X6:X6:X6:X6::X6 announce rpki state
exit-address-family
!

86 v1.0
Propagating RPKI states (IOS)

• Receiver (iBGP peer)

router bgp 131107
!---output omitted-----!
address-family ipv4
neighbor Y.Y.Y.Y activate
neighbor Y.Y.Y.Y send-community both
neighbor Y.Y.Y.Y announce rpki state
exit-address-family
!
address-family ipv6
neighbor Y6:Y6:Y6:Y6::Y6 activate
neighbor Y6:Y6:Y6:Y6::Y6 send-community both
neighbor Y6:Y6:Y6:Y6::Y6 announce rpki state
exit-address-family
!

§ If announce rpki state is not configured for the neighbor, all prefixes received
from the iBGP neighbor will be marked VALID!

87 v1.0
Propagating RPKI states (JunOS)

• Sender (router with an RTR session)

policy-statement ROUTE-VALIDATION {
term valid {
from {
protocol bgp;
validation-database valid; term unknown {
} from {
then { protocol bgp;
local-preference 200; validation-database unknown;
validation-state valid; }
community add origin-validation-state-valid; then {
accept; local-preference 100;
} validation-state unknown;
} community add origin-validation-state-unknown;
term invalid { accept;
from { }
protocol bgp; }
validation-database invalid; }
} }
then {
local-preference 50;
validation-state invalid;
community add origin-validation-state-invalid;
accept;
}
}

88 v1.0
Propagating RPKI states (JunOS)

• Receiver (iBGP peer)

policy-statement ROUTE-VALIDATION-1 {
term valid {
from community origin-validation-state-valid;
then validation-state valid;
}
term invalid {
from community origin-validation-state-invalid;
then validation-state invalid;
}
term unknown {
from community origin-validation-state-unknown;
then validation-state unknown;
}

89 v1.0
Operational Considerations

• Max-length
q Make sure the value covers your BGP announcements

• minimal ROAs
q Reduce spoofed origin-AS attack surface
§ https://tools.ietf.org/html/draft-ietf-sidrops-rpkimaxlen-03
§ ROAs should cover only those prefixes announced in BGP

90 v1.0
Operational Considerations

• When RTR session goes down?

q Validation state changes to Not Found for all routes (including

Invalids) after a while (JunOS/SROS: 1hr, IOS-XE: 5mins)

§ at least two RTR sessions and careful filtering policies

q What happens if the two validators don’t agree??

91 v1.0
Operational Considerations

• During a router reload?

q Will we receive ROAs first or BGP updates first?
§ If BGP update is faster than ROA, invalid routes to iBGP peers

92 v1.0
Operational Considerations

• Default routes?
q Even if you drop Invalids, default route will match anything

93 v1.0
Operational Considerations

• iBGP State Propagation? ~ Multivendor

q IOS as BR, propagating states to JunOS iBGP peers

unknown iana 4300

q Options(<JunOS 17.4R3, 18.2R3, 18.4R2):

§ Either act on the states at the border, or

§ Match and tag them with custom communities before propagating

94 v1.0
Operational Considerations

• iBGP State Propagation?

q What if one eBGP router sends Valid tag, and the other sends Not
Found:
§ If the RTR session is down on one eBGP speaker, and/or
§ Different ROA cache entry flush timers for different router vendors
(JunOS/SROS: 1hr, IOS-XE: 5mins)

q What is the router best path selection?

§ XE behaviour seems to pick validation state over other BGP attributes

95 v1.0
Other developments

• ROA with AS-0 origin (RFC6483/RFC7607)

q Reserved by IANA for non-routed networks

q Negative attestation: no valid ASN has been granted authority

§ Not to be routed (Ex: IXP LAN prefixes)
§ Overridden by another ROA (with an origin-AS other than AS-0)
§ APNIC ~ Nov 2018

q Prop-132: unallocated/unassigned APNIC space

§ ~ RFC6491 for special use/reserved/unallocated

96 v1.0
https://www.apnic.net/community/security/resource-certification/#routing

97 v1.0
Any questions?

98 v1.0