[Openswan Users] site to site ipsec VPN. local endpoint replies with a host unreachable.

Discussion:

Michael Closson

2013-11-24 04:43:08 UTC

Hello Openswan experts.

Site to site VPN, ping from local site, local VPN endpoint generates ICMP
host unreachable.

I'm using CentOS 6.4
Openswan IPsec U2.6.32/K2.6.32-358.23.2.el6.x86_64...

The remote side is the IaaS company SoftLayer.

Here is the network map

172.16.0.0/16
|
172.16.1.201==24.212.xxx.xxx --- 173.192.251.47
|
|
???
|
|
10.100.128.128/26

If I run 'ping 10.100.128.130' on 172.16.1.18, then 172.16.1.201 replies
with ICMP host unreachable.

I don't think its an iptables problem since I've set the default rule in
the filter/INPUT and filter/FORWARD to be ACCEPT. And there are no rules
about replying with a host unreachable.

I think the problem is with the local ipsec endpoint (172.16.1.201). I
attach tcpdump to eth0 on that machine (the public interface) and there
is no ESP traffic to the remote ipsec endpoint. Just the periodic
'isakmp: phase 2/others ? inf[E]'
which--I believe--is expected.

I am following up in parallel with softlayer support. But I suspect that
if they don't know openswan/netkey/linux kernel, they will not be able to
help.

I don't know how to debug this further. Any suggestions will be
appreciated!

Paul Wouters

2013-11-24 17:45:52 UTC

Permalink

Post by Michael Closson
Site to site VPN, ping from local site, local VPN endpoint generates ICMP
host unreachable.
I'm using CentOS 6.4
Openswan IPsec U2.6.32/K2.6.32-358.23.2.el6.x86_64...
The remote side is the IaaS company SoftLayer.
Here is the network map
172.16.0.0/16
|
172.16.1.201==24.212.xxx.xxx --- 173.192.251.47
|
|
???
|
|
10.100.128.128/26
If I run 'ping 10.100.128.130' on 172.16.1.18, then 172.16.1.201 replies
with ICMP host unreachable.
Here is my ipsec config file. (lambda is 172.16.1.201)
conn softlayer
type=tunnel
authby=secret
auto=start
left=%defaultroute
leftsubnet=172.16.0.0/16
leftsourceip=172.16.1.201
right=173.192.251.47
rightsubnet=10.100.128.128/26
pfs=yes

That looks good.

Post by Michael Closson
src 173.192.251.47 dst 24.212.xxx.xxx
proto esp spi 0xd04fb5e2 reqid 16385 mode tunnel
replay-window 32 flag 20
auth hmac(sha1) 0xc8e849ee498bbe3453dfa2a9a84e837926fe9675
enc cbc(aes) 0xa98dee1f3574083b2d19ed95699f92e4
src 24.212.xxx.xxx dst 173.192.251.47
proto esp spi 0xd3636bcf reqid 16385 mode tunnel
replay-window 32 flag 20
auth hmac(sha1) 0xf28bd49f9608e03e0130784a829054e1038a3fe4
enc cbc(aes) 0xc73abb30f3078aa3a382ec2daad50ea5
src 172.16.0.0/16 dst 10.100.128.128/26
dir out priority 2598 ptype main
tmpl src 24.212.xxx.xxx dst 173.192.251.47
proto esp reqid 16385 mode tunnel
src 10.100.128.128/26 dst 172.16.0.0/16
dir fwd priority 2598 ptype main
tmpl src 173.192.251.47 dst 24.212.xxx.xxx
proto esp reqid 16385 mode tunnel
src 10.100.128.128/26 dst 172.16.0.0/16
dir in priority 2598 ptype main
tmpl src 173.192.251.47 dst 24.212.xxx.xxx
proto esp reqid 16385 mode tunnel

Same here, look good. Perhaps there is an ESP filter somewhere? Try adding
forceencaps=yes to force ESPinUDP?

Paul

--
Libreswan Developer - https://libreswan.org/
Red Hat Security - http://people.redhat.com/pwouters/
Personal Blog - https://nohats.ca/

Michael Closson

2013-11-24 18:49:54 UTC

Permalink

Hi Paul.

Thank-you for your reply!

After (on a whim) trying host to site VPN rather than site to site VPN, I can
confirm that ESP isn't being filtered. See below for the details.

The problem remains that the local VPN endpoint is generating a ICMP host
unreachable.

Is there anyway I can enable some kernel level debugging? I'll check and see
what google can suggest.

Some additional information. (3 things)

1)

I tried your suggestion. Same symptoms. Seems that the enc part works
because tcpdump showed the expected output.

13:25:13.761769 IP 173.192.251.47.4500 > 24.212.xxx.xxx.4500: NONESP-encap: isakmp: phase 2/others ? inf[E]
13:25:13.761978 IP 24.212.201.xxx.xxx > 173.192.251.47.4500: NONESP-encap: isakmp: phase 2/others ? inf[E]
13:25:17.765236 IP 24.212.201.xxx.xxx > 173.192.251.47.4500: isakmp-nat-keep-alive
13:25:17.765258 IP 24.212.201.xxx.xxx > 173.192.251.47.4500: isakmp-nat-keep-alive
13:25:18.801498 IP 173.192.251.47.4500 > 24.212.xxx.xxx.4500: NONESP-encap: isakmp: phase 2/others ? inf[E]
13:25:18.801732 IP 24.212.xxx.xxx.4500 > 173.192.251.47.4500: NONESP-encap: isakmp: phase 2/others ? inf[E]
13:25:23.850583 IP 173.192.251.47.4500 > 24.212.xxx.xxx.4500: NONESP-encap: isakmp: phase 2/others ? inf[E]

But the local VPN endpoint still sends out ICMP host unreachable.

2)

On a whim, I tried host to site VPN. On the softlayer side I configured
24.212.xxx.xxx/32 as the local side. I used the following config on the
local side.

[***@lambda ipsec.d]# cat softlayer.conf
conn softlayer
type=tunnel
authby=secret
auto=start
left=%defaultroute
#leftsubnet=172.16.0.0/16
#leftsourceip=172.16.1.201
right=173.192.251.47
rightsubnet=10.100.128.128/26
pfs=yes
forceencaps=yes

Now I can ping hosts inside softlayer.

[***@lambda ipsec.d]# ping 10.100.128.131
PING 10.100.128.131 (10.100.128.131) 56(84) bytes of data.
64 bytes from 10.100.128.131: icmp_seq=1 ttl=61 time=49.1 ms
64 bytes from 10.100.128.131: icmp_seq=2 ttl=61 time=39.8 ms
64 bytes from 10.100.128.131: icmp_seq=3 ttl=61 time=44.8 ms
^C
--- 10.100.128.131 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2405ms
rtt min/avg/max/mdev = 39.829/44.615/49.155/3.815 ms

Here is the relevant tcpdump.

13:29:05.636046 IP 24.212.xxx.xxx.4500 > 173.192.251.47.4500: NONESP-encap: isakmp: phase 2/others ? inf[E]
13:29:09.820978 IP 24.212.xxx.xxx.4500 > 173.192.251.47.4500: UDP-encap: ESP(spi=0xd3637cf1,seq=0x6e), length 132
13:29:09.870074 IP 173.192.251.47.4500 > 24.212.xxx.xxx.4500: UDP-encap: ESP(spi=0x8b12936d,seq=0x26), length 132
13:29:09.870074 IP 10.100.128.131 > 24.212.xxx.xxx: ICMP echo reply, id 18988, seq 1, length 64
13:29:10.822214 IP 24.212.xxx.xxx.4500 > 173.192.251.47.4500: UDP-encap: ESP(spi=0xd3637cf1,seq=0x6f), length 132
13:29:10.862001 IP 173.192.251.47.4500 > 24.212.xxx.xxx.4500: UDP-encap: ESP(spi=0x8b12936d,seq=0x27), length 132
13:29:10.862001 IP 10.100.128.131 > 24.212.xxx.xxx: ICMP echo reply, id 18988, seq 2, length 64
13:29:11.824134 IP 24.212.xxx.xxx.4500 > 173.192.251.47.4500: UDP-encap: ESP(spi=0xd3637cf1,seq=0x70), length 132
13:29:11.868952 IP 173.192.251.47.4500 > 24.212.xxx.xxx.4500: UDP-encap: ESP(spi=0x8b12936d,seq=0x28), length 132
13:29:11.868952 IP 10.100.128.131 > 24.212.xxx.xxx: ICMP echo reply, id 18988, seq 3, length 64

woohoo! progress.

Looks like softlayer is doing some NATting on their end.

[***@lambda ipsec.d]# ssh ***@10.100.128.131
***@10.100.128.131's password:
Last login: Wed Nov 20 12:11:01 2013 from 10.0.80.184
[***@xxx-private ~]# who
root pts/0 2013-11-24 12:31 (10.1.22.46) <== HERE
[***@xxx-private ~]# tty
/dev/pts/0

Seems that softlayer re-wrote the source IP from 24.212.xxx.xxx to 10.1.22.46.

This is a problem because connections initiated from inside softlayer to
the local-side host (24.212.xxx.xxx) don't go through the tunnel, but go
unencrypted over the internet.

3)

I also tried host to site but without forcing encapsulation and was able to
ping and ssh to hosts on the softlayer side.

13:40:24.860619 IP 24.212.xxx.xxx > 173.192.251.47: ESP(spi=0xd3637da8,seq=0x2c), length 100
13:40:26.190404 IP 24.212.xxx.xxx > 173.192.251.47: ESP(spi=0xd3637da8,seq=0x2d), length 132
13:40:26.229591 IP 173.192.251.47 > 24.212.xxx.xxx: ESP(spi=0x92886450,seq=0x21), length 132
13:40:26.229591 IP 10.100.128.131 > 24.212.xxx.xxx: ICMP echo reply, id 35117, seq 1, length 64
13:40:27.191744 IP 24.212.xxx.xxx > 173.192.251.47: ESP(spi=0xd3637da8,seq=0x2e), length 132
13:40:27.233203 IP 173.192.251.47 > 24.212.xxx.xxx: ESP(spi=0x92886450,seq=0x22), length 132
13:40:27.233203 IP 10.100.128.131 > 24.212.xxx.xxx: ICMP echo reply, id 35117, seq 2, length 64
13:40:28.193343 IP 24.212.xxx.xxx > 173.192.251.47: ESP(spi=0xd3637da8,seq=0x2f), length 132
13:40:28.233588 IP 173.192.251.47 > 24.212.xxx.xxx: ESP(spi=0x92886450,seq=0x23), length 132
13:40:28.233588 IP 10.100.128.131 > 24.212.xxx.xxx: ICMP echo reply, id 35117, seq 3, length 64

So ESP is definitely not being filtered.

Post by Paul Wouters

That looks good.

Same here, look good. Perhaps there is an ESP filter somewhere? Try adding
forceencaps=yes to force ESPinUDP?
Paul
--
Libreswan Developer - https://libreswan.org/
Red Hat Security - http://people.redhat.com/pwouters/
Personal Blog - https://nohats.ca/

Paul Wouters

2013-11-25 17:02:44 UTC

Permalink

Post by Michael Closson
After (on a whim) trying host to site VPN rather than site to site VPN, I can
confirm that ESP isn't being filtered. See below for the details.
The problem remains that the local VPN endpoint is generating a ICMP host
unreachable.
Is there anyway I can enable some kernel level debugging? I'll check and see
what google can suggest.

Try this in /etc/sysctl.conf and run sysctl -p:

net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
#
net.ipv4.conf.default.rp_filter = 0
#
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.icmp_ignore_bogus_error_responses = 1
#
net.ipv4.conf.default.log_martians = 1
net.ipv4.conf.all.log_martians = 1

Paul

--
Libreswan Developer - https://libreswan.org/
Red Hat Security - http://people.redhat.com/pwouters/
Personal Blog - https://nohats.ca/

Michael Closson

2013-11-26 01:17:08 UTC

Permalink

Hey Paul. Thanks again for your time!

I sent an email earlier but I think I used a wrong sender address,
so it may not have made it onto the mailing list.

I was able to get it working by disabling the iptables masquerade feature
on the public interface. That is,

*nat
:PREROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
#-A POSTROUTING -o eth0 -j MASQUERADE <<== comment out this line.

After making that change, the site to site VPN worked perfectly.
Seems that IP masquerading and openswan site-to-site are not compatible.
I suppose there is a good reason for it. The code that is responsible for
doing the masquerade will have to be aware of which IP ranges are part of
an openswan tunnel, and not do the address translation on them. Muddies
the otherwise clean separation of responsibilities. Perhaps this is an
advantage of KLIPS over NETKEY. With KLIPS there is a different interface
for the IP sec packets, so netfilter won't try to NAT those packets.

Thanks for your help.
Michael Closson

Post by Paul Wouters

net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
#
net.ipv4.conf.default.rp_filter = 0
#
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.icmp_ignore_bogus_error_responses = 1
#
net.ipv4.conf.default.log_martians = 1
net.ipv4.conf.all.log_martians = 1
Paul
--
Libreswan Developer - https://libreswan.org/
Red Hat Security - http://people.redhat.com/pwouters/
Personal Blog - https://nohats.ca/

Paul Wouters

2013-11-26 02:59:55 UTC

Permalink

Post by Michael Closson
I was able to get it working by disabling the iptables masquerade feature
on the public interface. That is,
*nat
:PREROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
#-A POSTROUTING -o eth0 -j MASQUERADE <<== comment out this line.
After making that change, the site to site VPN worked perfectly.
Seems that IP masquerading and openswan site-to-site are not compatible.
I suppose there is a good reason for it. The code that is responsible for
doing the masquerade will have to be aware of which IP ranges are part of
an openswan tunnel, and not do the address translation on them. Muddies
the otherwise clean separation of responsibilities. Perhaps this is an
advantage of KLIPS over NETKEY. With KLIPS there is a different interface
for the IP sec packets, so netfilter won't try to NAT those packets.

You can exclude your remote network from being masqueraded:

iptables -t nat -I POSTROUTING -d remotenet/mask -j RETURN

Be sure to Insert the rule before the above masquerade rule.

Paul

--
Libreswan Developer - https://libreswan.org/
Red Hat Security - http://people.redhat.com/pwouters/
Personal Blog - https://nohats.ca/

Simon Deziel

2013-11-26 03:32:58 UTC

Permalink

It's true that with KLIPS you can still use -o to decide what is
masquerade or not but you can also use Paul's suggestion to base that
decision on the destination address.

You can also tell iptables to avoid masquerading when dealing with IPsec
packets:

-A POSTROUTING -m policy --dir out --pol ipsec -j RETURN

Just make sure to put that one above your MASQUERADE rule.

Regards,
Simon

u***@lists.openswan.org

2013-11-24 14:17:47 UTC

Permalink

Rescued from the spam bucket. Please remember to subscribe to the mailing list before posting to it.

From: Michael Closson <***@lambda.closson.ca>
Subject: Re: [Openswan Users] site to site ipsec VPN. local endpoint replies with a host unreachable.
Date: November 24, 2013 at 2:17:39 PM EST
To: ***@lists.openswan.org

Oh. I found the problem.

After disabling IP masquerading on the local side, site to site VPN works now.

*nat
:PREROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
#-A POSTROUTING -o eth0 -j MASQUERADE

Success!!

Michael Closson

Post by Michael Closson
Hi Paul.
Thank-you for your reply!
After (on a whim) trying host to site VPN rather than site to site VPN, I can
confirm that ESP isn't being filtered. See below for the details.
The problem remains that the local VPN endpoint is generating a ICMP host
unreachable.
Is there anyway I can enable some kernel level debugging? I'll check and see
what google can suggest.
Some additional information. (3 things)
1)
I tried your suggestion. Same symptoms. Seems that the enc part works
because tcpdump showed the expected output.
13:25:13.761769 IP 173.192.251.47.4500 > 24.212.xxx.xxx.4500: NONESP-encap: isakmp: phase 2/others ? inf[E]
13:25:13.761978 IP 24.212.201.xxx.xxx > 173.192.251.47.4500: NONESP-encap: isakmp: phase 2/others ? inf[E]
13:25:17.765236 IP 24.212.201.xxx.xxx > 173.192.251.47.4500: isakmp-nat-keep-alive
13:25:17.765258 IP 24.212.201.xxx.xxx > 173.192.251.47.4500: isakmp-nat-keep-alive
13:25:18.801498 IP 173.192.251.47.4500 > 24.212.xxx.xxx.4500: NONESP-encap: isakmp: phase 2/others ? inf[E]
13:25:18.801732 IP 24.212.xxx.xxx.4500 > 173.192.251.47.4500: NONESP-encap: isakmp: phase 2/others ? inf[E]
13:25:23.850583 IP 173.192.251.47.4500 > 24.212.xxx.xxx.4500: NONESP-encap: isakmp: phase 2/others ? inf[E]
But the local VPN endpoint still sends out ICMP host unreachable.
2)
On a whim, I tried host to site VPN. On the softlayer side I configured
24.212.xxx.xxx/32 as the local side. I used the following config on the
local side.
conn softlayer
type=tunnel
authby=secret
auto=start
left=%defaultroute
#leftsubnet=172.16.0.0/16
#leftsourceip=172.16.1.201
right=173.192.251.47
rightsubnet=10.100.128.128/26
pfs=yes
forceencaps=yes
Now I can ping hosts inside softlayer.
PING 10.100.128.131 (10.100.128.131) 56(84) bytes of data.
64 bytes from 10.100.128.131: icmp_seq=1 ttl=61 time=49.1 ms
64 bytes from 10.100.128.131: icmp_seq=2 ttl=61 time=39.8 ms
64 bytes from 10.100.128.131: icmp_seq=3 ttl=61 time=44.8 ms
^C
--- 10.100.128.131 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2405ms
rtt min/avg/max/mdev = 39.829/44.615/49.155/3.815 ms
Here is the relevant tcpdump.
13:29:05.636046 IP 24.212.xxx.xxx.4500 > 173.192.251.47.4500: NONESP-encap: isakmp: phase 2/others ? inf[E]
13:29:09.820978 IP 24.212.xxx.xxx.4500 > 173.192.251.47.4500: UDP-encap: ESP(spi=0xd3637cf1,seq=0x6e), length 132
13:29:09.870074 IP 173.192.251.47.4500 > 24.212.xxx.xxx.4500: UDP-encap: ESP(spi=0x8b12936d,seq=0x26), length 132
13:29:09.870074 IP 10.100.128.131 > 24.212.xxx.xxx: ICMP echo reply, id 18988, seq 1, length 64
13:29:10.822214 IP 24.212.xxx.xxx.4500 > 173.192.251.47.4500: UDP-encap: ESP(spi=0xd3637cf1,seq=0x6f), length 132
13:29:10.862001 IP 173.192.251.47.4500 > 24.212.xxx.xxx.4500: UDP-encap: ESP(spi=0x8b12936d,seq=0x27), length 132
13:29:10.862001 IP 10.100.128.131 > 24.212.xxx.xxx: ICMP echo reply, id 18988, seq 2, length 64
13:29:11.824134 IP 24.212.xxx.xxx.4500 > 173.192.251.47.4500: UDP-encap: ESP(spi=0xd3637cf1,seq=0x70), length 132
13:29:11.868952 IP 173.192.251.47.4500 > 24.212.xxx.xxx.4500: UDP-encap: ESP(spi=0x8b12936d,seq=0x28), length 132
13:29:11.868952 IP 10.100.128.131 > 24.212.xxx.xxx: ICMP echo reply, id 18988, seq 3, length 64
woohoo! progress.
Looks like softlayer is doing some NATting on their end.
Last login: Wed Nov 20 12:11:01 2013 from 10.0.80.184
root pts/0 2013-11-24 12:31 (10.1.22.46) <== HERE
/dev/pts/0
Seems that softlayer re-wrote the source IP from 24.212.xxx.xxx to 10.1.22.46.
This is a problem because connections initiated from inside softlayer to
the local-side host (24.212.xxx.xxx) don't go through the tunnel, but go
unencrypted over the internet.
3)
I also tried host to site but without forcing encapsulation and was able to
ping and ssh to hosts on the softlayer side.
13:40:24.860619 IP 24.212.xxx.xxx > 173.192.251.47: ESP(spi=0xd3637da8,seq=0x2c), length 100
13:40:26.190404 IP 24.212.xxx.xxx > 173.192.251.47: ESP(spi=0xd3637da8,seq=0x2d), length 132
13:40:26.229591 IP 173.192.251.47 > 24.212.xxx.xxx: ESP(spi=0x92886450,seq=0x21), length 132
13:40:26.229591 IP 10.100.128.131 > 24.212.xxx.xxx: ICMP echo reply, id 35117, seq 1, length 64
13:40:27.191744 IP 24.212.xxx.xxx > 173.192.251.47: ESP(spi=0xd3637da8,seq=0x2e), length 132
13:40:27.233203 IP 173.192.251.47 > 24.212.xxx.xxx: ESP(spi=0x92886450,seq=0x22), length 132
13:40:27.233203 IP 10.100.128.131 > 24.212.xxx.xxx: ICMP echo reply, id 35117, seq 2, length 64
13:40:28.193343 IP 24.212.xxx.xxx > 173.192.251.47: ESP(spi=0xd3637da8,seq=0x2f), length 132
13:40:28.233588 IP 173.192.251.47 > 24.212.xxx.xxx: ESP(spi=0x92886450,seq=0x23), length 132
13:40:28.233588 IP 10.100.128.131 > 24.212.xxx.xxx: ICMP echo reply, id 35117, seq 3, length 64
So ESP is definitely not being filtered.

Post by Paul Wouters

That looks good.

_______________________________________________
https://lists.openswan.org/mailman/listinfo/users
Micropayments: https://flattr.com/thing/38387/IPsec-for-Linux-made-easy
http://www.amazon.com/gp/product/1904811256/104-3099591-2946327?n=283155