\documentstyle[12pt,twoside]{article} \def\TITLE{Tunnels over IP} \input preamble \begin{center} \Large\bf Tunnels over IP in Linux-2.2 \end{center} \begin{center} { \large Alexey~N.~Kuznetsov } \\ \em Institute for Nuclear Research, Moscow \\ \verb|kuznet@ms2.inr.ac.ru| \\ \rm March 17, 1999 \end{center} \vspace{5mm} \tableofcontents \section{Instead of introduction: micro-FAQ.} \begin{itemize} \item Q: In linux-2.0.36 I used: \begin{verbatim} ifconfig tunl1 10.0.0.1 pointopoint 193.233.7.65 \end{verbatim} to create tunnel. It does not work in 2.2.0! A: You are right, it does not work. The command written above is split to two commands. \begin{verbatim} ip tunnel add MY-TUNNEL mode ipip remote 193.233.7.65 \end{verbatim} will create tunnel device with name \verb|MY-TUNNEL|. Now you may configure it with: \begin{verbatim} ifconfig MY-TUNNEL 10.0.0.1 \end{verbatim} Certainly, if you prefer name \verb|tunl1| to \verb|MY-TUNNEL|, you still may use it. \item Q: In linux-2.0.36 I used: \begin{verbatim} ifconfig tunl0 10.0.0.1 route add -net 10.0.0.0 gw 193.233.7.65 dev tunl0 \end{verbatim} to tunnel net 10.0.0.0 via router 193.233.7.65. It does not work in 2.2.0! Moreover, \verb|route| prints a funny error sort of network unreachable'' and after this I found a strange direct route to 10.0.0.0 via \verb|tunl0| in routing table. A: Yes, in 2.2 the rule that {\em normal} gateway must reside on directly connected network has not any exceptions. You may tell kernel, that this particular route is {\em abnormal}: \begin{verbatim} ifconfig tunl0 10.0.0.1 netmask 255.255.255.255 ip route add 10.0.0.0/8 via 193.233.7.65 dev tunl0 onlink \end{verbatim} Note keyword \verb|onlink|, it is the magic key that orders kernel not to check for consistency of gateway address. Probably, after this explanation you have already guessed another method to cheat kernel: \begin{verbatim} ifconfig tunl0 10.0.0.1 netmask 255.255.255.255 route add -host 193.233.7.65 dev tunl0 route add -net 10.0.0.0 netmask 255.0.0.0 gw 193.233.7.65 route del -host 193.233.7.65 dev tunl0 \end{verbatim} Well, if you like such tricks, nobody may prohibit you to use them. Only do not forget that between \verb|route add| and \verb|route del| host 193.233.7.65 is unreachable. \item Q: In 2.0.36 I used to load \verb|tunnel| device module and \verb|ipip| module. I cannot find any \verb|tunnel| in 2.2! A: Linux-2.2 has single module \verb|ipip| for both directions of tunneling and for all IPIP tunnel devices. \item Q: \verb|traceroute| does not work over tunnel! Well, stop... It works, only skips some number of hops. A: Yes. By default tunnel driver copies \verb|ttl| value from inner packet to outer one. It means that path traversed by tunneled packets to another endpoint is not hidden. If you dislike this, or if you are going to use some routing protocol expecting that packets with ttl 1 will reach peering host (f.e.\ RIP, OSPF or EBGP) and you are not afraid of tunnel loops, you may append option \verb|ttl 64|, when creating tunnel with \verb|ip tunnel add|. \item Q: ... Well, list of things, which 2.0 was able to do finishes. \end{itemize} \paragraph{Summary of differences between 2.2 and 2.0.} \begin{itemize} \item {\bf In 2.0} you could compile tunnel device into kernel and got set of 4 devices \verb|tunl0| ... \verb|tunl3| or, alternatively, compile it as module and load new module for each new tunnel. Also, module \verb|ipip| was necessary to receive tunneled packets. {\bf 2.2} has {\em one\/} module \verb|ipip|. Loading it you get base tunnel device \verb|tunl0| and another tunnels may be created with command \verb|ip tunnel add|. These new devices may have arbitrary names. \item {\bf In 2.0} you set remote tunnel endpoint address with the command \verb|ifconfig| ... \verb|pointopoint A|. {\bf In 2.2} this command has the same semantics on all the interfaces, namely it sets not tunnel endpoint, but address of peering host, which is directly reachable via this tunnel, rather than via Internet. Actual tunnel endpoint address \verb|A| should be set with \verb|ip tunnel add ... remote A|. \item {\bf In 2.0} you create tunnel routes with the command: \begin{verbatim} route add -net 10.0.0.0 gw A dev tunl0 \end{verbatim} {\bf 2.2} interprets this command equally for all device kinds and gateway is required to be directly reachable via this tunnel, rather than via Internet. You still may use \verb|ip route add ... onlink| to override this behaviour. \end{itemize} \section{Tunnel setup: basics} Standard Linux-2.2 kernel supports three flavor of tunnels, listed in the following table: \vspace{2mm} \begin{tabular}{lll} \vrule depth 0.8ex width 0pt\relax Mode & Description & Base device \\ ipip & IP over IP & tunl0 \\ sit & IPv6 over IP & sit0 \\ gre & ANY over GRE over IP & gre0 \end{tabular} \vspace{2mm} \noindent All the kinds of tunnels are created with one command: \begin{verbatim} ip tunnel add mode [ local ] [ remote ] \end{verbatim} This command creates new tunnel device with name \verb||. The \verb|| is an arbitrary string. Particularly, it may be even \verb|eth0|. The rest of parameters set different tunnel characteristics. \begin{itemize} \item \verb|mode | sets tunnel mode. Three modes are available now \verb|ipip|, \verb|sit| and \verb|gre|. \item \verb|remote | sets remote endpoint of the tunnel to IP address \verb||. \item \verb|local | sets fixed local address for tunneled packets. It must be an address on another interface of this host. \end{itemize} \let\thefootnote\oldthefootnote Both \verb|remote| and \verb|local| may be omitted. In this case we say that they are zero or wildcard. Two tunnels of one mode cannot have the same \verb|remote| and \verb|local|. Particularly it means that base device or fallback tunnel cannot be replicated.\footnote{ This restriction is relaxed for keyed GRE tunnels.} Tunnels are divided to two classes: {\bf pointopoint} tunnels, which have some not wildcard \verb|remote| address and deliver all the packets to this destination, and {\bf NBMA} (i.e. Non-Broadcast Multi-Access) tunnels, which have no \verb|remote|. Particularly, base devices (f.e.\ \verb|tunl0|) are NBMA, because they have neither \verb|remote| nor \verb|local| addresses. After tunnel device is created you should configure it as you did it with another devices. Certainly, the configuration of tunnels has some features related to the fact that they work over existing Internet routing infrastructure and simultaneously create new virtual links, which changes this infrastructure. The danger that not enough careful tunnel setup will result in formation of tunnel loops, collapse of routing or flooding network with exponentially growing number of tunneled fragments is very real. Protocol setup on pointopoint tunnels does not differ of configuration of another devices. You should set a protocol address with \verb|ifconfig| and add routes with \verb|route| utility. NBMA tunnels are different. To route something via NBMA tunnel you have to explain to driver, where it should deliver packets to. The only way to make it is to create special routes with gateway address pointing to desired endpoint. F.e.\ \begin{verbatim} ip route add 10.0.0.0/24 via dev tunl0 onlink \end{verbatim} It is important to use option \verb|onlink|, otherwise kernel will refuse request to create route via gateway not directly reachable over device \verb|tunl0|. With IPv6 the situation is much simpler: when you start device \verb|sit0|, it automatically configures itself with all IPv4 addresses mapped to IPv6 space, so that all IPv4 Internet is {\em really reachable} via \verb|sit0|! Excellent, the command \begin{verbatim} ip route add 3FFE::/16 via ::193.233.7.65 dev sit0 \end{verbatim} will route \verb|3FFE::/16| via \verb|sit0|, sending all the packets destined to this prefix to 193.233.7.65. \section{Tunnel setup: options} Command \verb|ip tunnel add| has several additional options. \begin{itemize} \item \verb|ttl N| --- set fixed TTL \verb|N| on tunneled packets. \verb|N| is number in the range 1--255. 0 is special value, meaning that packets inherit TTL value. Default value is: \verb|inherit|. \item \verb|tos T| --- set fixed tos \verb|T| on tunneled packets. Default value is: \verb|inherit|. \item \verb|dev DEV| --- bind tunnel to device \verb|DEV|, so that tunneled packets will be routed only via this device and will not be able to escape to another device, when route to endpoint changes. \item \verb|nopmtudisc| --- disable Path MTU Discovery on this tunnel. It is enabled by default. Note that fixed ttl is incompatible with this option: tunnels with fixed ttl always make pmtu discovery. \end{itemize} \verb|ipip| and \verb|sit| tunnels have no more options. \verb|gre| tunnels are more complicated: \begin{itemize} \item \verb|key K| --- use keyed GRE with key \verb|K|. \verb|K| is either number or IP address-like dotted quad. \item \verb|csum| --- checksum tunneled packets. \item \verb|seq| --- serialize packets. \begin{NB} I think this option does not work. At least, I did not test it, did not debug it and even do not understand, how it is supposed to work and for what purpose Cisco planned to use it. \end{NB} \end{itemize} Actually, these GRE options can be set separately for input and output directions by prefixing corresponding keywords with letter \verb|i| or \verb|o|. F.e.\ \verb|icsum| orders to accept only packets with correct checksum and \verb|ocsum| means, that our host will calculate and send checksum. Command \verb|ip tunnel add| is not the only operation, which can be made with tunnels. Certainly, you may get short help page with: \begin{verbatim} ip tunnel help \end{verbatim} Besides that, you may view list of installed tunnels with the help of command: \begin{verbatim} ip tunnel ls \end{verbatim} Also you may look at statistics: \begin{verbatim} ip -s tunnel ls Cisco \end{verbatim} where \verb|Cisco| is name of tunnel device. Command \begin{verbatim} ip tunnel del Cisco \end{verbatim} destroys tunnel \verb|Cisco|. And, finally, \begin{verbatim} ip tunnel change Cisco mode sit local ME remote HE ttl 32 \end{verbatim} changes its parameters. \section{Differences 2.2 and 2.0 tunnels revisited.} Now we can discuss more subtle differences between tunneling in 2.0 and 2.2. \begin{itemize} \item In 2.0 all tunneled packets were received promiscuously as soon as you loaded module \verb|ipip|. 2.2 tries to select the best tunnel device and packet looks as received on this. F.e.\ if host received \verb|ipip| packet from host \verb|D| destined to our local address \verb|S|, kernel searches for matching tunnels in order: \begin{tabular}{ll} 1 & \verb|remote| is \verb|D| and \verb|local| is \verb|S| \\ 2 & \verb|remote| is \verb|D| and \verb|local| is wildcard \\ 3 & \verb|remote| is wildcard and \verb|local| is \verb|S| \\ 4 & \verb|tunl0| \end{tabular} If tunnel exists, but it is not in \verb|UP| state, the tunnel is ignored. Note, that if \verb|tunl0| is \verb|UP| it receives all the IPIP packets, not acknowledged by more specific tunnels. Be careful, it means that without carefully installed firewall rules anyone on the Internet may inject to your network any packets with source addresses indistinguishable from local ones. It is not so bad idea to design tunnels in the way enforcing maximal route symmetry and to enable reversed path filter (\verb|rp_filter| sysctl option) on tunnel devices. \item In 2.2 you can monitor and debug tunnels with \verb|tcpdump|. F.e.\ \verb|tcpdump| \verb|-i Cisco| \verb|-nvv| will dump packets, which kernel output, via tunnel \verb|Cisco| and the packets received on it from kernel viewpoint. \end{itemize} \section{Linux and Cisco IOS tunnels.} Among another tunnels Cisco IOS supports IPIP and GRE. Essentially, Cisco setup is subset of options, available for Linux. Let us consider the simplest example: \begin{verbatim} interface Tunnel0 tunnel mode gre ip tunnel source 10.10.14.1 tunnel destination 10.10.13.2 \end{verbatim} This command set translates to: \begin{verbatim} ip tunnel add Tunnel0 \ mode gre \ local 10.10.14.1 \ remote 10.10.13.2 \end{verbatim} Any questions? No questions. \section{Interaction IPIP tunnels and DVMRP.} DVMRP exploits IPIP tunnels to route multicasts via Internet. \verb|mrouted| creates IPIP tunnels listed in its configuration file automatically. From kernel and user viewpoints there are no differences between tunnels, created in this way, and tunnels created by \verb|ip tunnel|. I.e.\ if \verb|mrouted| created some tunnel, it may be used to route unicast packets, provided appropriate routes are added. And vice versa, if administrator has already created a tunnel, it will be reused by \verb|mrouted|, if it requests DVMRP tunnel with the same local and remote addresses. Do not wonder, if your manually configured tunnel is destroyed, when mrouted exits. \section{Broadcast GRE tunnels''.} It is possible to set \verb|remote| for GRE tunnel to a multicast address. Such tunnel becomes {\bf broadcast} tunnel (though word tunnel is not quite appropriate in this case, it is rather virtual network). \begin{verbatim} ip tunnel add Universe local 193.233.7.65 \ remote 224.66.66.66 ttl 16 ip addr add 10.0.0.1/16 dev Universe ip link set Universe up \end{verbatim} This tunnel is true broadcast network and broadcast packets are sent to multicast group 224.66.66.66. By default such tunnel starts to resolve both IP and IPv6 addresses via ARP/NDISC, so that if multicast routing is supported in surrounding network, all GRE nodes will find one another automatically and will form virtual Ethernet-like broadcast network. If multicast routing does not work, it is unpleasant but not fatal flaw. The tunnel becomes NBMA rather than broadcast network. You may disable dynamic ARPing by: \begin{verbatim} echo 0 > /proc/sys/net/ipv4/neigh/Universe/mcast_solicit \end{verbatim} and to add required information to ARP tables manually: \begin{verbatim} ip neigh add 10.0.0.2 lladdr 128.6.190.2 dev Universe nud permanent \end{verbatim} In this case packets sent to 10.0.0.2 will be encapsulated in GRE and sent to 128.6.190.2. It is possible to facilitate address resolution using methods typical for another NBMA networks f.e.\ to start user level \verb|arpd| daemon, which will maintain database of hosts attached to GRE virtual network or ask for information dedicated ARP or NHRP server. Actually, such setup is the most natural for tunneling, it is really flexible, scalable and easily managable, so that it is strongly recommended to be used with GRE tunnels instead of ugly hack with NBMA mode and \verb|onlink| modifier. Unfortunately, by historical reasons broadcast mode is not supported by IPIP tunnels, but this probably will change in future. \section{Traffic control issues.} Tunnels are devices, hence all the power of Linux traffic control applies to them. The simplest (and the most useful in practice) example is limiting tunnel bandwidth. The following command: \begin{verbatim} tc qdisc add dev tunl0 root tbf \ rate 128Kbit burst 4K limit 10K \end{verbatim} will limit tunneled traffic to 128Kbit with maximal burst size of 4K and queuing not more than 10K. However, you should remember, that tunnels are {\em virtual} devices implemented in software and true queue management is impossible for them just because they have no queues. Instead, it is better to create classes on real physical interfaces and to map tunneled packets to them. In general case of dynamic routing you should create such classes on all outgoing interfaces, or, alternatively, to use option \verb|dev DEV| to bind tunnel to a fixed physical device. In the last case packets will be routed only via specified device and you need to setup corresponding classes only on it. Though you have to pay for this convenience, if routing will change, your tunnel will fail. Suppose that CBQ class \verb|1:ABC| has been created on device \verb|eth0| specially for tunnel \verb|Cisco| with endpoints \verb|S| and \verb|D|. Now you can select IPIP packets with addresses \verb|S| and \verb|D| with some classifier and map them to class \verb|1:ABC|. F.e.\ it is easy to make with \verb|rsvp| classifier: \begin{verbatim} tc filter add dev eth0 pref 100 proto ip rsvp \ session D ipproto ipip filter S \ classid 1:ABC \end{verbatim} If you want to make more detailed classification of sub-flows transmitted via tunnel, you can build CBQ subtree, rooted at \verb|1:ABC| and attach to subroot set of rules parsing IPIP packets more deeply. \end{document}