Joining your first Internet Exchange

Last time, I covered the process of announcing your very first route to the Internet via BGP, but that’s only the beginning. I promised to dive into the process of joining an Internet Exchange to bring better connectivity to a fledgling network, and now is the time.

For this exercise, I will be connecting the BGP VM running AS200351 to the Ontario Internet Exchange (ONIX), which has gladly provided me with a port for my test network to help me write this post. While ONIX isn’t the biggest Internet Exchange in Toronto, for that honour belongs to TorIX, it nevertheless has a reasonable amount of big peers, such as Hurricane Electric, which will…

Without further ado, let’s dive in.

What is an Internet Exchange?

I’ve written about this in more detail as part of my series on BGP, especially when I discussed autonomous systems. To summarize, an Internet Exchange (IX) is conceptually a really big switch into which many networks are connected, forming a massive peering LAN. Through this peering LAN, many networks can achieve a direct connection to each other and set up BGP sessions with each other without having to run a dedicated wire between each pair of networks. Instead, every network simply needs to run one wire to the IX. This reduces the number of wires required to be linear to the number of interconnected networks from quadratic.

With BGP sessions, networks can announce routes to their own networks to each other, a process called peering. However, naïvely running direct BGP sessions between all networks still requires a quadratic number of BGP sessions. To help reduce this, most IXes offer route servers (RSes). Every member of the IX peers with the route server, announcing their own routes, and the route server sends back routes that all other peers announce to it. This reduces the number of BGP sessions to be linear to the number of networks on the IX.

For this reason, most networks peer with the route servers. However, some networks would prefer more fine-grained control over route distribution, and thus set up bilateral BGP sessions between each other anyway. These are frequently referred to as bilats. Large networks typically prefer this approach.

The IX peering LAN can also be used for more than just peering. For example, one network might offer another network BGP transit over a bilateral BGP session, which involves announcing every route on the Internet and providing full connectivity to the Internet.1

How to not get banned?

Before we talk about how to join an Internet Exchange, it is very important to discuss how to stay joined, i.e. not getting banned from an IX. You must have a firm grasp of this before connecting yourself to the IX.

While an Internet Exchange is effectively just a big LAN, it must not be carelessly treated as if it were a normal network. It is very important to follow several principles when dealing with Internet Exchanges, or you risk causing disruption to the entire Internet. I assure you, you do not want to wake up to news headlines of your network being accused of causing a major Internet outage. Such violations of the rules will also get you yelled at by the operators, resulting in your port getting disabled, or with repeated offences, get you permanently banned.

While the exact policy depends on the IX in question, and you are highly encouraged to read the relevant policies, there are several principles by which basically all IXes operate:

Only sending IPv4, IPv6, and ARP traffic. Specifically, this is usually framed as an EtherType restriction to:
0x0800: IPv4;
0x86dd: IPv6; and
0x0806: Address Resolution Protocol (ARP), a protocol to discover the MAC address associated with an IPv4 address.

An Internet Exchange is designed to exchange IP traffic, not other random traffic!

No multicast or broadcast traffic, since they go to every member, and sending a lot of them can easily threaten the stability of the IX itself. The following exception is carved out, as it’s required for MAC address discovery:
ARP packets; and
ICMPv6 neighbour discovery (NDP), specifically neighbour solicitation (ICMPv6 type 135). Note that this does not permit you to send router solicitation or router advertisement messages! Doing so will cause a lot of problems.

In fact, most IXes will very strongly prefer that you use very long timeouts for ARP and NDP caching to avoid generating too much broadcast traffic.

No link-local protocols other than the aforementioned ARP and NDP protocols. Technically, the previous two requirements already forbid all of these, but it’s worth specifically calling out certain especially troublesome ones:
Spanning Tree Protocol (STP), i.ie. IEEE 802.1d and IEEE 802.1w;
This also includes related protocols such as Cisco’s Unidirectional Link Detection (UDLD);
ICMP Router Discovery Protocol (IRDP);
ICMPv6 router solicitation and router advertisement;
ICMP redirects;
Vendor-specific discovery protocols, including but not limited to CDP, MNDP, and LLDP;
Interior routing protocols, including but not limited to OSPF, IS-IS, or IGRP. Only BGP should be used over IXes;
BOOTP/DHCP, both client and server, since IX ports should always use static IP addresses; and
Protocol-Independent Multicast (PIM).

Instructions for Linux will be included in this post. If you are joining from any hardware router or L3 switch, make sure you consult the vendor documentation on all the protocols enabled by default. Depending on the IX, sending a single forbidden packet may result in your port getting instantly disabled.

Do not hijack other people’s IPs or MACs. Only ever use IP addresses that you have been assigned by the IX. Do not use ARP/NDP proxying. Do not respond to ARP/NDP for IPs that you haven’t been assigned to.

Do not offer L2 access to the peering LAN to others, unless previously authorized by the IX to operate an extension or resell ports. Many IXes enforce a limit of one MAC address per port and have strict ACLs for MACs.

Do not announce the peering LAN via BGP. The peering LAN is a private network between members of the IX and should not be reachable over the Internet.

Do not point default routes at other IX members. Specifically, you should only ever send traffic to a member if the destination address is covered by a route announced to you by that member, either directly or through the route servers.

Only send your routes in your cone to peers and route servers. The norm for peering is to send your cone, i.e. routes belonging to your network and your downstream networks, as authorized by IRR and RPKI. For more details, see my post on route authorization. You should not send all routes in your routing table to peers or the route servers. You may offer transit over bilateral BGP sessions, in which case you may send the full Internet routing table, but you must never offer route servers transit.

Do not sniff traffic between any other IX members. This should go without saying, but do not violate the privacy of other IX members.

Note that this is by no means an exhaustive list. Doing anything crazy that causes instability for the IX or any network on it will have consequences. To learn more about bad IX traffic, you can consult this post from Ben Cartwright-Cox.

Gaining access to the peering LAN

Okay, with all the dangers out of the way, you can apply to join an IX and eventually obtain access to the peering LAN. This will depend heavily on how you plan to join the IX, mostly depending on whether you are using a VM or a dedi, and whether your server provider is a reseller for the IX.

Renting servers from an IX reseller

The easiest way to join an IX, and the one used by most hobby networks, is renting a VM (or dedicated server) from a provider that also resells access to a desired IX. Usually, this will be advertised as an option on the VMs, and some providers may specifically sell IXP access VMs that are specifically designed to be hooked up to an IX.

When joining an IX this way, you would typically open a ticket with your server provider requesting access to the IX, and they will send you a form to fill out. Fill out the provided form, pay the relevant fees, and at some point, the peering LAN will be delivered to your server.

Typically, with VMs, the IX peering LAN will be delivered as a separate virtual interface on the VM. With dedicated servers, it’s usually delivered as a VLAN on the main network uplink, although it’s also possible for it to be plugged into a separate network port, if you have those available.

For this exercise, I asked ONIX if I could “resell” the connection I have on my dedicated server to my VM, and they said yes, then allocated an IPv6 address for AS200351 on the peering LAN: 2001:504:125:e1::bde. We will be using this later.

Apply to an IX on your own

This is definitely the harder option, but it’s necessary if you wish to join an IX that your provider doesn’t resell. I had to go through this process recently to join AMS-IX.

The first thing you should figure out is where your VM or dedicated server is located, and whether the IX is available in the same location. In my case, I am using iFog GmbH in Amsterdam, and the server is located in Digital Realty’s AMS17 datacentre, where AMS-IX is available through several “partners”, i.e. resellers.

If the desired IX is not available, not all is lost, since it might be possible to connect in a different location and have the traffic transported (or “hauled”) through someone else’s connection between datacentres. You would typically find a provider that would give you an L2 connection between two datacentres. You then connect to the provider, who then connects to the IX in the other location.

You should then figure out how much it takes to run a cross-connect (XC), which is basically a piece of optical fibre that goes between your server provider’s rack and the AMS-IX reseller’s (or directly to AMS-IX). In my case, an XC in DRT AMS17 came with a prohibitive monthly price, and therefore, it didn’t make sense to connect to AMS-IX in DRT AMS17.

However, not all is lost. I spoke with iFog GmbH for other options, and since they already had a connection between DRT AMS17 and NIKHEF (a nearby datacentre with no monthly fees for XCs), I can simply get an XC with AMS-IX in NIKHEF and the traffic hauled to me through iFog’s connection. This is a much more affordable option, and it’s the one I went with.

Knowing how to connect, I spoke to AMS-IX and they directed me to one of their resellers, A2B Internet. Once I sorted out the administrative stuff with them, they sent over a letter of authorization (LOA) authorizing an XC to be run between A2B Internet and Dynamic Quantum Networks in the next 30 days after issuance. I forwarded the LOA over to iFog, who then got someone in NIKHEF to run the XC between A2B’s rack and iFog’s.

With the XC in place, iFog hauled the connection as a VLAN through a bunch of his switches until it reached my server. Once done, I informed A2B of the fact, and they asked for the MAC address of my IX interface to configure the ACL. Once I produced that, I got onto AMS-IX’s peering LAN.

Okay, that would be the case if the XC worked. For some reason, iFog couldn’t see any light on the fibre, and I spent the next two weeks channelling messages between iFog and A2B, both 6 hours ahead of me due to timezones, until we finally determined that one of the fibre strands was damaged. After fixing that, I finally got onto AMS-IX. These things do happen occasionally, and I guess I just had bad luck. Most people I talked to seemed to have a much easier time with XCs, but you should be prepared for the worst if you are dealing with such real world things.

Configuring the IX interface

Like last time, we’ll use ifupdown2 to configure the interface. In this example, ONIX is on ens19, and we will bring it up by appending the following content to /etc/network/interfaces:

auto ens19
iface ens19 inet static
# To prevent IP hijacking by people blindly copying this config,
# this is not my actual IP on ONIX.
# Change this to the IP and mask allocated by the IX.
address 2001:db8:1234::5678/64
# And if you are using IPv4, put your IP here, or otherwise remove this:
address 192.0.2.123/24

# Disables multicast, which is undesired on IX peering LANs.
up ip link set multicast off dev "$IFACE"

Note that this specifically uses static IPs. We do not want DHCP on the IX peering LAN, as previously mentioned. If you are running any sort of DHCP server (such as dnsmasq) or route advertisement daemon (such as radvd) on the host, make sure it’s disabled for the peering LAN.

Before bringing up the interface, configure the following sysctls by creating /etc/sysctl.d/onix.conf with this content:

# Prevents the kernel from answering ARP requests with addresses from
# other interfaces, which is undesirable on the IXP peering LAN.
net.ipv4.conf.ens19.arp_filter = 1
net.ipv4.conf.ens19.arp_ignore = 1
net.ipv4.conf.ens19.arp_announce = 1

# Disables IPv6 autoconfiguration on this interface.
net.ipv6.conf.ens19.autoconf = 0

# Do not accept any IPv6 route advertisements on this interface.
# These shouldn't exist on the peering LAN, but some IXes don't ban people
# as quickly as we'd like.
net.ipv6.conf.ens19.accept_ra = 0

# Disables router solicitations.
net.ipv6.conf.ens19.router_solicitations = -1

# Bump ARP and NDP timeouts to 4 hours to avoid generating broadcast traffic.
net.ipv4.neigh.ens19.base_reachable_time_ms = 14400000
net.ipv6.neigh.ens19.base_reachable_time_ms = 14400000

With that configured, we can now reload sysctl config and bring up the interface:

sudo systemctl restart systemd-sysctl.service
sudo ifup ens19

If all goes well, you should now be able to ping the route servers. You can look up the IP addresses on the IX’s website, but in the case of ONIX, it’s 2001:504:125:e1::1 and 2001:504:125:e1::2. Let’s ping the first IX:

$ ping 2001:504:125:e1::1
PING 2001:504:125:e1::1 (2001:504:125:e1::1) 56 data bytes
64 bytes from 2001:504:125:e1::1: icmp_seq=1 ttl=64 time=0.275 ms
64 bytes from 2001:504:125:e1::1: icmp_seq=2 ttl=64 time=0.173 ms
^C
--- 2001:504:125:e1::1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1025ms
rtt min/avg/max/mdev = 0.173/0.224/0.275/0.051 ms

We are online!

If this doesn’t work, make sure your IP configuration is correct. If that still doesn’t work, you should run tcpdump -i ens19 and check if you see any broadcast traffic. If you aren’t receiving anything, speak with your server provider to see if the port is connected correctly.

Checking for bad traffic

Before we continue, we should take this opportunity to check that we aren’t sending forbidden traffic to the IX. For this, you will need tcpdump:

sudo apt install tcpdump

Now, to sanity check that it can receive traffic:

$ sudo tcpdump -e -i ens19 'broadcast || multicast'
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens19, link-type EN10MB (Ethernet), snapshot length 262144 bytes
21:12:45.728543 f0:64:26:f5:b2:0b (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 60: Request who-has 149.112.50.178 tell hurricane-electric.as6939.ip.onix.cx, length 46
21:12:45.769472 50:6b:4b:3e:33:a5 (oui Unknown) > 33:33:ff:00:00:17 (oui Unknown), ethertype IPv6 (0x86dd), length 86: paradoxnetworks.as52025.ip6.onix.cx > ff02::1:ff00:17: ICMP6, neighbor solicitation, who has private-user.as51019.ip6.onix.cx, length 32
21:12:45.798912 e4:8d:8c:fb:c0:62 (oui Unknown) > 33:33:ff:00:00:15 (oui Unknown), ethertype IPv6 (0x86dd), length 86: smishcraft.as210667.ip6.onix.cx > ff02::1:ff00:15: ICMP6, neighbor solicitation, who has as17290.onix.cx, length 32
21:12:45.811583 bc:24:11:f1:22:57 (oui Unknown) > 33:33:ff:00:00:61 (oui Unknown), ethertype IPv6 (0x86dd), length 86: as213768.onix.cx > ff02::1:ff00:61: ICMP6, neighbor solicitation, who has bgptools-rc.as212232.ip6.onix.cx, length 32
...

Yes, we can see broadcast ARP and NDP packets on ONIX. Now, let’s filter it down to the outgoing bad stuff:

$ sudo tcpdump -Qout -e -i ens19 '(broadcast || multicast) && !arp && !(icmp6 && ip6[40] == 135)'
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens19, link-type EN10MB (Ethernet), snapshot length 262144 bytes

Leave this running for a while, say at least an hour. Make sure it doesn’t print anything more. Then press Ctrl+C to interrupt it, and make sure it prints 0 packets captured, like this:

0 packets captured

If you see any bad traffic, please address it immediately, before you get banned by the IX.

Set preferred IPs

To avoid making requests from IPs on the IX peering LAN on routes received from the route servers, you must set krt_prefsrc on protocol kernel to the desired default IP.

For example, we can uncomment the commented block in the IPv6 kernel protocol in /etc/bird/bird.conf from last time, resulting in something like this:

protocol kernel {
scan time 60;
ipv6 {
export filter {
if source = RTS_STATIC && proto = "default_v6" then reject;
# FIXME change the IP to one of yours that you want to prefer
if source = RTS_BGP then krt_prefsrc = 2602:fa43:f0::1;
accept;
};
};
}

You will need to do so for IPv4 as well, if you have IPv4.

Setting up BGP sessions with route servers

First, we should define an identifier for ONIX in filter_bgp.conf, in the section # FIXME: define your IXPs here. For example:

define IXP_ONIX = 100;

This will be used in BGP communities generated by the library to identify routes originated from the IX’s route servers.

We can now append new content to the bird.conf we constructed last time.

First, we define a bird protocol template for BGP with our ONIX IPs to avoid repeating the same IPs over and over again:

# Drop this entire section if you don't have IPv4.
# FIXME change the template name if joining something not ONIX.
template bgp onix_v4 {
# Replace this IP with your ONIX IPv4
local 192.0.2.1 as 200351;
default bgp_local_pref 100;
}

# FIXME change the template name if joining something not ONIX.
template bgp onix_v6 {
# Replace this IP with your ONIX IPv6
local 2001:db8:1234::5678 as 200351;
default bgp_local_pref 100;
}

Note that we set bgp_local_pref to 100 for all IX peers by default, since that’s what we said in the previous part we were using for direct peers.

Now, let’s define templates for route servers:

# Drop this entire section if you don't have IPv4.
# FIXME change the template name if joining something not ONIX.
template bgp onix_rs_v4 from onix_v4 {
ipv4 {
import keep filtered;
# FIXME change IXP_ONIX to the constant defined for the IX
# you are joining.
import where import_ixp_trusted(IXP_ONIX);
# FIXME change the ASN here to that of the IX route servers.
export where export_cone(57369);
};
}

# FIXME change the template name if joining something not ONIX.
template bgp onix_rs_v6 from onix_v6 {
ipv6 {
import keep filtered;
# FIXME change IXP_ONIX to the constant defined for the IX
# you are joining.
import where import_ixp_trusted(IXP_ONIX);
# FIXME change the ASN here to that of the IX route servers.
export where export_cone(57369);
};
}

Note that helper functions are included from filter_bgp.conf, which we downloaded last time from my bird-filter library. You can look there for the exact implementations if you are interested in the details.

Now, let’s define the BGP sessions with the route servers:

# Drop this entire section if you don't have IPv4.
# FIXME change the protocol and template names if not joining ONIX.
protocol bgp onix_rs4a from onix_rs_v4 {
# FIXME change the description if not joining ONIX.
description "ONIX Route Server A (IPv4)";
# FIXME change the IPv4 and ASN to that of the first route server.
neighbor 149.112.50.1 as 57369;
}

# Drop this entire section if you don't have IPv4.
# FIXME change the protocol and template names if not joining ONIX.
protocol bgp onix_rs4b from onix_rs_v4 {
# FIXME change the description if not joining ONIX.
description "ONIX Route Server B (IPv4)";
# FIXME change the IPv4 and ASN to that of the second route server.
neighbor 149.112.50.2 as 57369;
}

# FIXME change the protocol and template names if not joining ONIX.
protocol bgp onix_rs6a from onix_rs_v6 {
# FIXME change the description if not joining ONIX.
description "ONIX Route Server A (IPv6)";
# FIXME change the IPv6 and ASN to that of the first route server.
neighbor 2001:504:125:e1::1 as 57369;
}

# FIXME change the protocol and template names if not joining ONIX.
protocol bgp onix_rs6b from onix_rs_v6 {
# FIXME change the description if not joining ONIX.
description "ONIX Route Server B (IPv6)";
# FIXME change the IPv6 and ASN to that of the second route server.
neighbor 2001:504:125:e1::2 as 57369;
}

If you are joining an IX with only one route server, then omit the protocol block for the B route server.

Now let’s reload our configuration:

$ sudo birdc configure
BIRD 2.17.1 ready.
Reading configuration from /etc/bird/bird.conf
Reconfigured

Now we should see the route server BGP sessions:

$ sudo birdc s p
BIRD 2.17.1 ready.
Name       Proto      Table      State  Since         Info
...
onix_rs6a  BGP        ---        up     22:43:37.758  Established
onix_rs6b  BGP        ---        up     22:43:37.713  Established

And we are now peered with the route servers!

Common issues

If your BGP session isn’t showing as established, then something has gone wrong. There are several things to try (this also applies to bilateral sessions, which we will demonstrate later):

Double check the IP addresses you are using. Make sure the ones on your side and the peer’s side are both correct.
Check if you can ping the BGP neighbour address, as seen in birdc s p a. If you can’t ping the IP, it means the IP might be incorrect or the other side is down. Double check the IP and talk to the peer or the IX for more information if you believe the IP is correct.
If you have a firewall, ensure TCP port 179 connections are allowed, as it is used for BGP.
Check BIRD logs with journalctl -u bird for any errors.

If you can’t figure this out, ask your upstream, or ask for help in the IPv6 Discord.

Setting up bilateral BGP sessions

For fun, I will also show how to configure a bilateral BGP session over ONIX, using AS54148 as an example:

# Drop this entire section if you don't have IPv4.
# FIXME change the protocol and template names if not joining ONIX.
protocol bgp onix_dqn4 from onix_v4 {
# FIXME change the description if not peering with me.
description "Dynamic Quantum Networks (IPv4)";
# FIXME change the IP and ASN to that of your desired peer.
neighbor 149.112.50.51 as 54148;

ipv4 {
import keep filtered;
# FIXME change the ASNs below to that of your desired peer.
import where import_peer_trusted(54148);
export where export_cone(54148);
};
}

# FIXME change the protocol and template names if not joining ONIX.
protocol bgp onix_dqn6 from onix_v6 {
# FIXME change the description if not peering with me.
description "Dynamic Quantum Networks (IPv6)";
# FIXME change the IP and ASN to that of your desired peer.
neighbor 2001:504:125:e1::51 as 54148;

ipv6 {
import keep filtered;
# FIXME change the ASNs below to that of your desired peer.
import where import_peer_trusted(54148);
export where export_cone(54148);
};
}

For simplicity’s sake, I will not use any IRR or RPKI filters, but note that this is not good practice. I may cover this later on my blog, but Instructions are available in the README for my bird-filter repository.

Now let’s reload bird and validate the session:

$ sudo birdc configure
BIRD 2.17.1 ready.
Reading configuration from /etc/bird/bird.conf
Reconfigured
$ sudo birdc s p
BIRD 2.17.1 ready.
Name       Proto      Table      State  Since         Info
...
onix_dqn6  BGP        ---        up     23:20:17.861  Established

Check the common issues section above if you run into trouble.

Hurricane Electric

Hurricane Electric (AS6939) is a major global ISP, and you can peer with them over ONIX. In fact, they will typically reach out to you directly when your port goes up with something like this:

Hello AS200351,

We noticed you are live at ONIX and would like to initiate peering. If you would like to establish peering over this exchange or any other of our common exchanges please send me the details and when you have configured the session on your side. If you have any specific questions feel free to let me know.

I’ve included our information at the end of this message. If you need additional information, let me know and I’ll get it out to you ASAP.

[ONIX] AS6939 2001:504:125:e1::3

[ONIX] AS200351 2001:504:125:e1::bde

We look forward to hearing from you.

Thank you

Mindy Mosher Hurricane Electric AS6939

In which case, you can just reply and say yes. If not, you can email peering@he.net for a BGP session as well.

You can set up the session on your end like any other peers, as before:

# Drop this entire section if you don't have IPv4.
# FIXME change the protocol and template names if not joining ONIX.
protocol bgp onix_he4 from onix_v4 {
description "Hurricane Electric (IPv4)";
# Update the IP here if not peering with HE over ONIX.
neighbor 149.112.50.3 as 6939;

ipv4 {
import keep filtered;
import where import_peer_trusted(6939);
export where export_cone(6939);
};
}

# FIXME change the protocol and template names if not joining ONIX.
protocol bgp onix_he6 from onix_v6 {
description "Hurricane Electric (IPv6)";
# Update the IP here if not peering with HE over ONIX.
neighbor 2001:504:125:e1::3 as 6939;

ipv6 {
import keep filtered;
import where import_peer_trusted(6939);
export where export_cone(6939);
};
}

However, they will also happily offer you free IPv6 transit, allowing you to get a second upstream to multi-home your network. If you say yes to their offer, you can use the following block instead for IPv6:

# FIXME change the protocol and template names if not joining ONIX.
protocol bgp onix_he6 from onix_v6 {
description "Hurricane Electric Transit (IPv6)";
# Update the IP here if not peering with HE over ONIX.
neighbor 2001:504:125:e1::3 as 6939;
# We use 50 for upstreams, as mentioned last time.
# You can use a slightly bigger number to prefer HE, or a smaller number
# to deprefer them.
default bgp_local_pref 50;

ipv6 {
import keep filtered;
import where import_transit(6939, false);
export where export_cone(6939);
};
}

Once done, run sudo birdc configure to reconfigure bird. The BGP session will be down until HE configures it on their end. For this exercise, it took HE a grand total of 2 minutes to set up the BGP session after I emailed them, which is rather impressive. Anyways, once it’s configured on both ends:

$ sudo birdc s p
BIRD 2.17.1 ready.
Name       Proto      Table      State  Since         Info
...
onix_he6   BGP        ---        up     23:44:43.056  Established
$ sudo birdc s p a onix_he6
...
Routes:         224535 imported, 0 filtered, 3 exported, 155101 preferred

Since this is a transit session, you should see over 200k routes.

Once again, check the common issues section above if you run into trouble.

After a while, you should see AS6939 as a direct transit provider on bgp.tools for your prefix, e.g. for 2a07:54c1:d351::/48. This might take a day or two, just like how long it previously took the route to appear on the Internet for the first time.

Conclusion

Hopefully, you’ve successfully joined your first Internet Exchange and made your network multi-homed. You might even have found yourself a backup transit provider. Note that networks other than Hurricane Electric might also offer free IP transit over IXes; you’ll just have to ask around.

Remember to double check that you aren’t sending random multicast or broadcast traffic with the tcpdump command above!

You can follow the same procedure to join more Internet Exchanges, if more are available in your location, or if you decide to get another server.

Here are some providers I use that sell IX access:

iFog (non-affiliate): sells access to FogIXP, ONIX, FREMIX, NVIX, Frys-IX, NL-ix, SpeedIX, LSIX, ERA-IX, and more; ticket to get a quote for pricing;
Xenyth (non-affiliate): sells access to ONIX and NVIX;
Lagrange Cloud (non-affiliate): sells access to LINX and LONAP; and
F4 Networks (non-affiliate): sells access to KCIX, STLIX, HOUIX, F4IX, and more.

Finally, if you run into trouble, feel free to ask for help in the IPv6 Discord.

Notes

This is usually the case, but due to several high-profile peering disputes, not every transit provider is able to provide the complete Internet routing table. ↩