Summary:
I am putting this post together to help others out there that are trying to create a site to site VPN tunnel between different Openstack regions OR even Cloud Providers (Openstack). The VyOS configuration portions can be leveraged for those trying to setup AWS VPC to VPC IPSEC tunnels as well. Everything I have posted in here can be found on the internet, but the information wasn't all in once easy to find place.
Requirements:
- openstack >= Icehouse/Juno
- Neutron
- allowed address pairs
- floating IP's
- ability to create and update your private network and subnet settings
- cli tools installed and can run commands in your tenant space
- VyOS 1.1.7 from here VyOS Image
- above VyOS converted to a qcow image and uploaded into glance (if I receive enough requests I can post either the instructions or a working image)
Example Diagram:
Openstack Project setup:
Go ahead and create your network, subnet and router similar to the above diagram if you haven't done so already.
Next launch an instance using the image you have already created and uploaded into glance, if people are having a lot of trouble doing this, I will see if I can add the instructions to this post on how to do this. If people are concerned about HA setups, I can look to add that later as well, it isn't difficult to do.
We now need to create a security group that will allow the correct traffic to and through this VM so the VPN will work, please modify to fit your needs, this is NOT a secure setup and will allow all traffic from both sides of the VPN to traverse. Replace <VYOS2_FIP>/CIDR with the appropriate value for your setup.
Security Group Setup: using neutron cli
neutron security-group-create vyos-dc1
neutron security-group-rule-create --direction ingress --ethertype ipv4 --protocol tcp --port-range-min 1 --port-range-max 65535 --remote-ip-prefix <LOCAL_SUBNET_2>/<CIDR> vyos-dc1
neutron security-group-rule-create --direction ingress --ethertype ipv4 --protocol udp --port-range-min 1 --port-range-max 65535 --remote-ip-prefix <LOCAL_SUBNET_2>/<CIDR> vyos-dc1
neutron security-group-rule-create --direction ingress --ethertype ipv4 --protocol tcp --port-range-min 1 --port-range-max 65535 --remote-ip-prefix <LOCAL_SUBNET_1>/<CIDR> vyos-dc1
neutron security-group-rule-create --direction ingress --ethertype ipv4 --protocol udp --port-range-min 1 --port-range-max 65535 --remote-ip-prefix <LOCAL_SUBNET_1>/<CIDR> vyos-dc1
neutron security-group-rule-create --direction ingress --ethertype ipv4 --protocol tcp --port-range-min 1 --port-range-max 65535 --remote-ip-prefix <VYOS2_FIP>/<CIDR> vyos-dc1
neutron security-group-rule-create --direction ingress --ethertype ipv4 --protocol udp --port-range-min 1 --port-range-max 65535 --remote-ip-prefix <VYOS2_FIP>/<CIDR> vyos-dc1
neutron security-group-rule-create --direction ingress --ethertype ipv4 --protocol icmp --remote-ip-prefix 0.0.0.0/0 vyos-dc1
Setup: VyOS Region1
You will need to use the console access from horizon in order to run the below commands to get ssh working and DHCP cannot be used as the VyOS doesn't seem to support that with IPSEC tunnels, it doesn't seem to be able to determine the IP, see references at the bottom.
VyOS Server 1; Horizon Console Access
configure
set interfaces ethernet eth0 address <LOCAL_VYOS1_IP>/<CIDR>
set service ssh port '22'
set protocols static route 0.0.0.0/0 next-hop <LOCAL_NETWORK_GATEWAY> distance '1'
set protocols static route <LOCAL_SUBNET_2>/<CIDR> next-hop <VYOS2_FIP>
commit
save
SSH to the VyOS server in order to configure it and run the below configuration commands to setup the IPSEC tunnel
VyOS Server 1 Config: SSH access
configure
set vpn ipsec esp-group office-srv-esp compression 'disable'
set vpn ipsec esp-group office-srv-esp lifetime '1800'
set vpn ipsec esp-group office-srv-esp mode 'tunnel'
set vpn ipsec esp-group office-srv-esp pfs 'enable'
set vpn ipsec esp-group office-srv-esp proposal 1 encryption 'aes256'
set vpn ipsec esp-group office-srv-esp proposal 1 hash 'sha1'
set vpn ipsec ike-group office-srv-ike ikev2-reauth 'no'
set vpn ipsec ike-group office-srv-ike key-exchange 'ikev1'
set vpn ipsec ike-group office-srv-ike lifetime '3600'
set vpn ipsec ike-group office-srv-ike proposal 1 encryption 'aes256'
set vpn ipsec ike-group office-srv-ike proposal 1 hash 'sha1'
set vpn ipsec ike-group office-srv-ike dead-peer-detection action 'restart'
set vpn ipsec ike-group office-srv-ike dead-peer-detection interval '30'
set vpn ipsec ike-group office-srv-ike dead-peer-detection timeout '30'
set vpn ipsec auto-update '60'
set vpn ipsec ipsec-interfaces interface 'eth0'
set vpn ipsec site-to-site peer <VYOS2_FIP> authentication mode 'pre-shared-secret'
set vpn ipsec site-to-site peer <VYOS2_FIP> authentication pre-shared-secret '<CHANGEME>'
set vpn ipsec site-to-site peer <VYOS2_FIP> ike-group 'office-srv-ike'
set vpn ipsec site-to-site peer <VYOS2_FIP> local-address '<LOCAL_VYOS1_IP>'
set vpn ipsec site-to-site peer <VYOS2_FIP> authentication remote-id '<LOCAL_VYOS2_IP>'
set vpn ipsec site-to-site peer <VYOS2_FIP> tunnel 0 allow-nat-networks 'disable'
set vpn ipsec site-to-site peer <VYOS2_FIP> tunnel 0 allow-public-networks 'disable'
set vpn ipsec site-to-site peer <VYOS2_FIP> tunnel 0 esp-group 'office-srv-esp'
set vpn ipsec site-to-site peer <VYOS2_FIP> tunnel 0 local prefix '<LOCAL_SUBNET_1>/<CIDR>'
set vpn ipsec site-to-site peer <VYOS2_FIP> tunnel 0 remote prefix '<LOCAL_SUBNET_2>/<CIDR>'
commit
save
Openstack Region 1 CLI Commands:
Allowed Address Pairs is required to allow traffic to flow over the neutron port, it works in conjunction with OpenVswitch, also if you were to leverage a VIP (pacemaker or keepalived) this would be required to assign multiple IP's to a single port.
First we need to obtain the neutron port ID for the instance we need to update.
Command: using neutron cli
neutron post-list |grep <LOCAL_VYOS1_IP>
Output:
| 536ca7a9-6569-41a9-ad44-8e33d8519472 | | fa:16:3e:e2:b3:6a | {"subnet_id": "56b874bd-488b-4fd1-a419-c62d26d84ca8", "ip_address": "<LOCAL_VYOS1_IP"} |
The first section between the two pipes | | highlighted in yellow is the neutron port ID we are looking for.
Update port command: using neutron cli
neutron port-update <port-id> --allowed-address-pairs type=dict list=true ip_address=<LOCAL_SUBNET_2>
We now need to update the subnet settings in order to automatically insert routes for the other network over the VPN setup, this will be added through dhcp. We need the subnet_id in order to rung the update you can either grab the subnet ID from the command above which is the GUID following "subnet_id", or run the command below.
Subnet List: neutron cli
neutron subnet-list |grep <LOCAL_SUBNET_1>
Subnet Update: neutron cli
neutron subnet-update <LOCAL_SUBNET_1_ID> --host_routes type=dict list=true destination=<LOCAL_SUBNET_2>/<CIDR>,nexthop=<LOCAL_VYOS1_IP>
Region 2 Setup:
You can for the most part replicate the above in the other project/data center, changing the IP's and networks for the new site, but I have replicated it below for completeness
If not then as we started above for configuring the Openstack project we must do the same here, if there isn't one already create your network, subnet and router similar to the above diagram.
Next launch an instance using the image you have already created and uploaded into glance.
Then create a security group as we did for the first region, again please modify to fit your needs, this is NOT a secure setup and will allow all traffic from both sides of the VPN to traverse. Replace <VYOS1_FIP>/CIDR with the appropriate value for your setup.
Security Group Setup: using neutron cli
neutron security-group-create vyos-dc2
neutron security-group-rule-create --direction ingress --ethertype ipv4 --protocol tcp --port-range-min 1 --port-range-max 65535 --remote-ip-prefix <LOCAL_SUBNET_2>/<CIDR> vyos-dc2
neutron security-group-rule-create --direction ingress --ethertype ipv4 --protocol udp --port-range-min 1 --port-range-max 65535 --remote-ip-prefix <LOCAL_SUBNET_2>/<CIDR> vyos-dc2
neutron security-group-rule-create --direction ingress --ethertype ipv4 --protocol tcp --port-range-min 1 --port-range-max 65535 --remote-ip-prefix <LOCAL_SUBNET_1>/<CIDR> vyos-dc2
neutron security-group-rule-create --direction ingress --ethertype ipv4 --protocol udp --port-range-min 1 --port-range-max 65535 --remote-ip-prefix <LOCAL_SUBNET_1>/<CIDR> vyos-dc2
neutron security-group-rule-create --direction ingress --ethertype ipv4 --protocol tcp --port-range-min 1 --port-range-max 65535 --remote-ip-prefix <VYOS1_FIP>/<CIDR> vyos-dc2
neutron security-group-rule-create --direction ingress --ethertype ipv4 --protocol udp --port-range-min 1 --port-range-max 65535 --remote-ip-prefix <VYOS1_FIP>/<CIDR> vyos-dc2
neutron security-group-rule-create --direction ingress --ethertype ipv4 --protocol icmp --remote-ip-prefix 0.0.0.0/0 vyos-dc2
VyOS Server 2 (Region 2)
Like above with VyOS server 1 we need to start configuring the server, firstly we will set up the address on the eth0 interface (adjust for your interface)
VyOS Server 2; Horizon Console Access
configure
set interfaces ethernet eth0 address <LOCAL_VYOS2_IP>/<CIDR>
set service ssh port '22'
set protocols static route 0.0.0.0/0 next-hop <LOCAL_NETWORK_GATEWAY> distance '1'
set protocols static route <LOCAL_SUBNET_1>/<CIDR> next-hop <VYOS1_FIP>
commit
save
SSH to the VyOS server in order to configure it and run the below configuration commands to setup the IPSEC tunnel
VYOS Server 2 Config:
set vpn ipsec esp-group office-srv-esp compression 'disable'
set vpn ipsec esp-group office-srv-esp lifetime '1800'
set vpn ipsec esp-group office-srv-esp mode 'tunnel'
set vpn ipsec esp-group office-srv-esp pfs 'enable'
set vpn ipsec esp-group office-srv-esp proposal 1 encryption 'aes256'
set vpn ipsec esp-group office-srv-esp proposal 1 hash 'sha1'
set vpn ipsec ike-group office-srv-ike ikev2-reauth 'no'
set vpn ipsec ike-group office-srv-ike key-exchange 'ikev1'
set vpn ipsec ike-group office-srv-ike lifetime '3600'
set vpn ipsec ike-group office-srv-ike proposal 1 encryption 'aes256'
set vpn ipsec ike-group office-srv-ike proposal 1 hash 'sha1'
set vpn ipsec ike-group office-srv-ike dead-peer-detection action 'restart'
set vpn ipsec ike-group office-srv-ike dead-peer-detection interval '30'
set vpn ipsec ike-group office-srv-ike dead-peer-detection timeout '30'
set vpn ipsec auto-update '60'
set vpn ipsec ipsec-interfaces interface 'eth0'
set vpn ipsec site-to-site peer <VYOS1_FIP> authentication mode 'pre-shared-secret'
set vpn ipsec site-to-site peer <VYOS1_FIP> authentication pre-shared-secret '<CHANGEME>'
set vpn ipsec site-to-site peer <VYOS1_FIP> ike-group 'office-srv-ike'
set vpn ipsec site-to-site peer <VYOS1_FIP> local-address '<LOCAL_VYOS2_IP>'
set vpn ipsec site-to-site peer <VYOS1_FIP> authentication remote-id '<LOCAL_VYOS1_IP>'
set vpn ipsec site-to-site peer <VYOS1_FIP> tunnel 0 allow-nat-networks 'disable'
set vpn ipsec site-to-site peer <VYOS1_FIP> tunnel 0 allow-public-networks 'disable'
set vpn ipsec site-to-site peer <VYOS1_FIP> tunnel 0 esp-group 'office-srv-esp'
set vpn ipsec site-to-site peer <VYOS1_FIP> tunnel 0 local prefix '<LOCAL_SUBNET_2>/<CIDR>'
set vpn ipsec site-to-site peer <VYOS1_FIP> tunnel 0 remote prefix '<LOCAL_SUBNET_1>/<CIDR>'
commit
save
Openstack Region 2 Commands:
neutron post-list |grep <LOCAL_VYOS1_IP>
Output:
| 778675b7-a0fc-4c18-b350-89fa888d9f83 | | fa:16:3e:fa:2d:a5 | {"subnet_id": "2f485fbf-b17b-4915-be10-8c4c4a32a9ec", "ip_address": "<LOCAL_VYOS2_IP>"} |
The first section between the two pipes | | highlighted in yellow is the neutron port ID we are looking for
Update port command: using neutron cli
neutron port-update <port-id> --allowed-address-pairs type=dict list=true ip_address=<LOCAL_SUBNET_2>
We now need to update the subnet settings in order to automatically insert routes for the other network over the VPN setup, this will be added through dhcp. We need the subnet_id in order to rung the update you can either grab the subnet ID from the command above which is the GUID following "subnet_id", or run the command below
Subnet List: neutron cli
neutron subnet-list |grep <LOCAL_SUBNET_2>
Subnet Update: neutron cli
neutron subnet-update <LOCAL_SUBNET_2_ID> --host_routes type=dict list=true destination=<LOCAL_SUBNET_1>/<CIDR>,nexthop=<LOCAL_VYOS2_IP>
We should reset the connection on each side to make sure we have a good connection setup
VyOS Server 2: ssh as vyos user
reset vpn ipsec-peer <VYOS1_FIP>
VyOS Server 1: ssh as vyos user
reset vpn ipsec-peer <VYOS2_FIP>
Testing:
Run the below on each VyOS server to see if the first phase is up (
first line only)
vyos@vyos:~$ sh vpn ike sa
Peer ID / IP Local ID / IP
------------ -------------
<VYOS1_FIP> <LOCAL_VYOS1_IP>
State Encrypt Hash D-H Grp NAT-T A-Time L-Time
----- ------- ---- ------- ----- ------ ------
up aes256 sha1 5 no 1289 3600
The state shows up so far so good..
Next run the below on each VyOS server (
first line only)
vyos@vyos:~$ sh vpn ipsec sa
Peer ID / IP Local ID / IP
------------ -------------
<VYOS1_FIP> <LOCAL_VYOS1_IP>
Tunnel State Bytes Out/In Encrypt Hash NAT-T A-Time L-Time Proto
------ ----- ------------- ------- ---- ----- ------ ------ -----
0 up 78.7K/78.9K aes256 sha1 no 711 1800 all
As you see the above the tunnel should show as up on each side
Lastly you should be able to ping each sides local interface, if all passes, launch a new instance on each side and test trying to ping one another (assuming you allow ICMP on your security groups for the newly created VM's)
References:
