Terraform an HA-VPN between GCP and Cisco

Doing Infrastucture-as-Code (IaC) with Ansible has given me a headache – so I’ve recently been playing around with Terraform as an alternative to Ansible for certain tasks that require Cloud IaaS interactions.

The goal of this blog post is to build an HA-VPN solution between GCP and an on-premises Cisco IOS-XE device (CSR) using Terraform. BGP will be established over the VPN in order to exchange routes dynamically. GCE compute instances will be deployed in GCP for testing connectivity over the VPN.

Let’s get started.

Table of contents:

Assumptions

This article assumes that you have already created a GCP account and have access to the GCP console. You should also have an understanding of GCP fundamentals and routing fundamentals.

GCP HA-VPN Overview

Google cloud offers numerous types of hybrid connectivity, and two (2) types of VPN connectivity – Classic or HA. The HA Cloud VPN is the recommended approach. When deploying the HA VPN solution you’ll achieve an SLA of 99.99% service availability out-of-the-box. This is because the HA Cloud VPN solution is multi-zonal and can handle failure within a region.

In a production environment, you’d likely have at least two VPN gateways on-premises to provide maximum redundancy. However, I only have a single IP address at my house, and a single router to play with. I’ll be choosing a redundancy type of “single IP, internally redundant”, meaning that GCP will allocate two separate public IP addresses attached to two separate interfaces on the HA VPN Gateway and will build tunnels from each to a single remote VPN gateway. My on-prem router will in turn build a tunnel to each of these.

A GCP Cloud Router will handle the BGP adjacencies and routing across the tunnel and internally within the GCP VPC.

For more detailed information about Google’s solution, please reference the documentation here:

Cloud VPN overview | Google Cloud

The Topology

Here is the topology I’ll be following for the remainder of this blog post:

GCP Preparation

There are a few things we must do manually in GCP to prepare for deploying Infrastructure-as-Code. First, we must create a project in GCP. Projects are at the highest level of organization in GCP and this is where billing occurs. Every resource in GCP must belong to a project. If you already have a project you want to use – great! If not, please create one now – I called mine “terraform-testing”. Go ahead and copy/paste the project ID somewhere for future use with the Terraform configuration script.

After choosing or creating a project, go over to IAM & Admin > Service accounts

You should see a single default service account

Click “Create Service Account” and give your new service account a name, then click “Create”

Next, add the following roles to the service account:

  • Project Editor
  • Compute Admin
  • Compute Network Admin

You can be more prescriptive here if you like but these roles work for me.

Click “Continue” then click “Create Key”

Choose “JSON” and click “Create”

This will download the JSON key file to your local machine. Keep this secure since it contains your private key. We’ll be referencing this file when running the Terraform script.

Terraform Installation

If you haven’t done so already, you need to install Terraform. You’ll need to update your $PATH as mentioned in the link provided, or you can move the binary to a directory that’s already in your path, which is what I did on my Mac:

[13:28:57] dvarnum:~ $ sudo mv /Users/dvarnum/terraform/terraform /usr/local/bin
Password:

[13:30:23] dvarnum:~ $ which terraform
/usr/local/bin/terraform

[13:30:28] dvarnum:~ $ terraform --version
Terraform v0.12.24

Terraform Overview

Straight from Terraform’s website:

“Terraform is used to create, manage, and update infrastructure resources such as physical machines, VMs, network switches, containers, and more. Almost any infrastructure type can be represented as a resource in Terraform.”

You create a Terraform configuration, which can either be a single file or set of files that describe the infrastructure to deploy and manage. The Terraform documentation is really good so I won’t repeat everything here but I’d like to talk about a few basics so you understand the script.

Terraform Providers

Providers are used for creating and managing resources. They handle the API interactions with IaaS providers. Since some of the services we’re deploying in GCP are considered “beta” we’ll need to use two (2) providers: google and google-beta. It is here that we specify top-level parameters such as the project, default region, default zone, and credentials.

provider "google" {
  project = var.gcp_project
  region = var.gcp_region
  zone = var.gcp_zone
  credentials = var.gcp_credentials_file
}

provider "google-beta" {
  project = var.gcp_project
  region = var.gcp_region
  zone = var.gcp_zone
  credentials = var.gcp_credentials_file

}

More information on the Google provider: Provider: Google Cloud Platform – Terraform by HashiCorp

Terraform Variables

The var. references seen in the provider above are pointing to static variables. Static variables can be stored in the same file as the main Terraform configuration or in a separate file. A sample variable looks like this:

variable gcp_region {
  description = "Default to US Central."
  default = "us-central1"
}

Terraform Resources

Resources are where everything happens in Terraform. A resource defines a piece of infrastructure to be deployed or configured. When declaring a resource block, you specify the resource type such as ”google_compute_network”, and a local resource name name such as ”tf_vpc_net1” , followed by the block of code. The name has no significance outside of the scope of the configuration script.

Example resource:

resource "google_compute_network" "tf_vpc_net1" {
  name                    = "tf-vpc-net-1"
  routing_mode            = "GLOBAL"
  auto_create_subnetworks = false
}

This same resource can be referenced later using the given name. For example, if I created the VPC network above, I could then create a subnet inside of that VPC reference the resource name:

resource "google_compute_subnetwork" "tf_vpc_net1_subnet1" {
  name          = "ha-vpn-subnet-1"
  ip_cidr_range = "10.0.1.0/24"
  region        = "us-central1"
  network       = google_compute_network.tf_vpc_net1.self_link
}

Terraform uses interpolation to handle complex references and dependencies between objects. I’m blown away at how much simpler it is to handle complexity in Terraform versus Ansible when it comes to dynamic variables. Not only that, it makes writing the script much easier since the order doesn’t really matter. Terraform knows what is dependent upon each other and executes in an intelligent way, abstracting that painful logic away from you.

Terraform Data

Although I’m not using data objects in the configuration script outlined in this blog post, it’s important to know that data can be used to retrieve information from a remote system.

Terraform Output Values

You can output various values pulled during configuration. You’ll see an example of this during the configuration execution below. More on outputs here: Output Values – Configuration Language – Terraform by HashiCorp

Terraform Configuration Overview for Deploying HA VPN on GCP

If you’d like to follow along, you can pull the code from the GitHub repo here:

git clone https://github.com/varnumd/terraform-gcp-cisco-vpn

In this directory you’ll see two (2) terraform files:

[17:11:35] dvarnum:gcp-cisco-vpn $ tree
.
├── gcp_variables.tf
└── main.tf

The gcp_variables file needs to be updated with the values of your choice. The only two that are absolutely required for you to update is gcp_projectand on_prem_ip1. Update the others as you deem necessary.

Note that the first variable here is a prompt that will ask you for your credential file when you run terraform apply. You could instead remove the “type = string” and use “default = file(“account.json”)” where “account.json” is the name and location of the key file you downloaded earlier.

variable "gcp_credentials_file" {
  description = "Locate the GCP credentials file."
  type = string
}

variable gcp_project {
  description = "GCP Project"
  default = "[UPDATE-WITH-YOUR-PROJECT-ID]"
}

variable gcp_region {
  description = "Default to US Central."
  default = "us-central1"
}

variable gcp_zone {
  description = "Default to US Central1c."
  default = "us-central1-c"
}

variable on_prem_ip1 {
  description = "The IP of the on-prem VPN gateway"
  default = "[UPDATE-WITH-YOUR-VPN-PUB-IP]"
}

variable gcp_asn {
  description = "BGP ASN or GCP Cloud Router"
  default = 64997
}

variable on_prem_asn {
  description = "BGP ASN or On-Prem Router"
  default = 65000
}

variable gcp_shared_secret {
  description = "VPN shared secret"
  default = "d0v3r1a1d"
}

The next file called main.tf is where all of the Terraform configuration happens. Let’s take a look.

First are the providers:

provider "google" {
  project     = var.gcp_project
  region      = var.gcp_region
  zone.       = var.gcp_zone
  credentials = var.gcp_credentials_file
}

provider "google-beta" {
  project     = var.gcp_project
  region.     = var.gcp_region
  zone        = var.gcp_zone
  credentials = var.gcp_credentials_file
}

Next we’ll create the HA VPN Gateway by calling the google_compute_ha_vpn_gateway resource. This is a beta feature so I must specify the google-beta provider at the time of this writing. Documentation for this resource can be found here: Google: google_compute_ha_vpn_gateway – Terraform by HashiCorp

resource "google_compute_ha_vpn_gateway" "ha_gateway1" {
  provider = google-beta
  region   = "us-central1"
  name     = "ha-vpn-1"
  network  = google_compute_network.tf_vpc_net1.self_link
}

Don’t worry about the network statement just yet – we’ll get to that.

Next we’ll specify the external gateway that GCP will be connecting to (my on-prem router). Note the redundancy type that I chose since I only have a single on-prem router but still want to achieve SLA 99.99% on the GCP-side. Other options can be found here: Creating an HA VPN gateway to a Peer VPN gateway | Cloud VPN

Documentation for this Terraform resource google_compute_external_vpn_gateway can be found here: Google: google_compute_external_vpn_gateway – Terraform by HashiCorp

resource "google_compute_external_vpn_gateway" "external_gateway" {
  provider        = google-beta
  name            = "hq-cisco"
  redundancy_type = "SINGLE_IP_INTERNALLY_REDUNDANT"
  description     = "An externally managed VPN gateway"
  interface {
    id            = 0
    ip_address = var.on_prem_ip1
  }
}

Configure a custom VPC network. Unlike AWS and Azure, VPCs in Google are global. We still need to set the routing mode to global to make sure that routes are propagated appropriately.

resource “google_compute_network” “tf_vpc_net1” {
  name                    = “tf-vpc-net-1”
  routing_mode            = “GLOBAL”
  auto_create_subnetworks = false
}

Configure subnets in the VPC. Notice the network value is google_compute_network.tf_vpc_net1.self_link. This is a special pointer to the VPC that is being created as a part of this configuration. This is the same pointer that was used when configuring the HA VPN Gateway resource.

resource “google_compute_subnetwork” “tf_vpc_net1_subnet1” {
  name          = “ha-vpn-subnet-1”
  ip_cidr_range = “10.0.1.0/24”
  region        = “us-central1”
  network       = google_compute_network.tf_vpc_net1.self_link
}

resource “google_compute_subnetwork” “tf_vpc_net1_subnet2” {
  name          = “ha-vpn-subnet-2”
  ip_cidr_range = “10.0.2.0/24”
  region        = “us-west1”
  network       = google_compute_network.tf_vpc_net1.self_link
}

Configure a Cloud Router

resource "google_compute_router" "router1" {
  name     = "ha-vpn-router-1"
  network  = google_compute_network.tf_vpc_net1.name
  bgp {
    asn = var.gcp_asn
  }
}

Configure two (2) VPN tunnels. The first tunnel will map to interface 0 on the VPN gateway, the next will map to interface 1 on the VPN gateway. More information about this resource can be found here: Google: google_compute_vpn_tunnel – Terraform by HashiCorp

resource "google_compute_vpn_tunnel" "tunnel1" {
  provider              = google-beta
  name                  = "ha-vpn-tunnel1"
  region                = "us-central1"
  vpn_gateway           = google_compute_ha_vpn_gateway.ha_gateway1.self_link
  peer_external_gateway = google_compute_external_vpn_gateway.external_gateway.self_link
  peer_external_gateway_interface = 0
  shared_secret         = var.gcp_shared_secret
  router                = google_compute_router.router1.self_link
  vpn_gateway_interface = 0
}

resource "google_compute_vpn_tunnel" "tunnel2" {
  provider              = google-beta
  name                  = "ha-vpn-tunnel2"
  region                = "us-central1"
  vpn_gateway           = google_compute_ha_vpn_gateway.ha_gateway1.self_link
  peer_external_gateway = google_compute_external_vpn_gateway.external_gateway.self_link
  peer_external_gateway_interface = 0
  shared_secret         = var.gcp_shared_secret
  router                = google_compute_router.router1.self_link
  vpn_gateway_interface = 1
}

Configure two (2) router interfaces, with one mapped to each tunnel. You must use /30 prefixes in the 169.254.0.0/16 CIDR block for peering.

# Router1 Interface 1 - Tunnel 1

resource "google_compute_router_interface" "router1_interface1" {
  name       = "router1-interface1"
  router     = google_compute_router.router1.name
  region     = "us-central1"
  ip_range   = "169.254.0.1/30"
  vpn_tunnel = google_compute_vpn_tunnel.tunnel1.name
}

resource "google_compute_router_peer" "router1_peer1" {
  name                      = "router1-peer1"
  router                    = google_compute_router.router1.name
  region                    = "us-central1"
  peer_ip_address           = "169.254.0.2"
  peer_asn                  = var.on_prem_asn
  advertised_route_priority = 100
  interface                 = google_compute_router_interface.router1_interface1.name
}

# Router1 Interface 2 - Tunnel 2

resource "google_compute_router_interface" "router1_interface2" {
  name       = "router1-interface2"
  router     = google_compute_router.router1.name
  region     = "us-central1"
  ip_range   = "169.254.1.1/30"
  vpn_tunnel = google_compute_vpn_tunnel.tunnel2.name
}

resource "google_compute_router_peer" "router1_peer2" {
  name                      = "router1-peer2"
  router                    = google_compute_router.router1.name
  region                    = "us-central1"
  peer_ip_address           = "169.254.1.2"
  peer_asn                  = var.on_prem_asn
  advertised_route_priority = 100
  interface                 = google_compute_router_interface.router1_interface2.name
}

Configure a GCE VM instance in each subnet which will be used for testing connectivity across the tunnel.

resource "google_compute_instance" "vm_instance1" {
  name = "terraform-intance-1"
  machine_type = "f1-micro"
  zone         = "us-central1-a"

  boot_disk {
    initialize_params {
      image = "debian-cloud/debian-9"
    }
  }

  network_interface {
    # A default network is created for all GCP projects
    # Using terraform interpolation, we'll reference the self_link here pointing to the newly created network
    subnetwork = google_compute_subnetwork.tf_vpc_net1_subnet1.self_link
    access_config {
    }
  }
}

resource "google_compute_instance" "vm_instance2" {
  name = "terraform-intance-2"
  machine_type = "f1-micro"
  zone         = "us-west1-b"

  boot_disk {
    initialize_params {
      image = "debian-cloud/debian-9"
    }
  }

  network_interface {
    # A default network is created for all GCP projects
    # Using terraform interpolation, we'll reference the self_link here pointing to the newly created network
    subnetwork = google_compute_subnetwork.tf_vpc_net1_subnet2.self_link
    access_config {
    }
  }
}

By default GCP custom networks do not have any firewall rules attached to them. Here’s a sample of a generic firewall rules that will allow access via some common protocols to the VM instances. The firewall rules are defined at the VPC level but enforced for each instance.

resource "google_compute_firewall" "tf_firewall" {
  name    = "terraform-firewall-base"
  network = google_compute_network.tf_vpc_net1.self_link

  allow {
    protocol = "icmp"
  }

  allow {
    protocol = "tcp"
    ports    = ["22", "80", "443"]
  }

  source_ranges = ["0.0.0.0/0"]
}

Lastly, we have some output values to display certain pertinent information to the terminal after the terraform configuration is applied.

output "gcp_external_vpn_address_1" {
  value = google_compute_ha_vpn_gateway.ha_gateway1.vpn_interfaces.0.ip_address
}

output "gcp_external_vpn_address_2" {
  value = google_compute_ha_vpn_gateway.ha_gateway1.vpn_interfaces.1.ip_address
}

output "gcp_instance1_external_ip" {
  value = google_compute_instance.vm_instance1.network_interface.0.access_config.0.nat_ip
}

output "gcp_instance1_internal_ip" {
  value = google_compute_instance.vm_instance1.network_interface.0.network_ip
}

output "gcp_instance2_external_ip" {
  value = google_compute_instance.vm_instance2.network_interface.0.access_config.0.nat_ip
}

output "gcp_instance2_internal_ip" {
  value = google_compute_instance.vm_instance2.network_interface.0.network_ip
}

Terraform in Action

A Terraform workflow looks like this:

  1. Initialize Terraform by executing terraform init in the directory where the Terraform configuration exists. This will initialize the provider plugins.
  2. (Optional) Validate your configuration syntax by running terraform validate. This will make sure everything checks-out from a syntax perspective.
  3. (Optional) Execute terraform plan to get a view of everything that will change in the environment. This is a preview and does not actually make any changes. Although it’s optional, it’s recommended to do this.
  4. Apply the configuration by executing terraform apply. This command will apply the configuration that was detailed in the plan.

Once in the directory, run terraform init to initialize the plugins.

[14:29:04] dvarnum:gcp-cisco-vpn $ terraform init

Initializing the backend...

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "google" (hashicorp/google) 3.15.0...
- Downloading plugin for provider "google-beta" (terraform-providers/google-beta) 3.15.0...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.google: version = "~> 3.15"
* provider.google-beta: version = "~> 3.15"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Validate the syntax

[20:23:21] dvarnum:terraform-gcp-cisco-vpn git:(master*) $ terraform validate
Success! The configuration is valid.

Now when we run terraform plan we’ll see a preview of the changes. This step logs in to GCP via the API using the service account created earlier. You’ll need to pass in the key when running this command. There are many ways you can bypass having to enter the credentials key. You can export a variable like export GOOGLE_CLOUD_KEYFILE_JSON=“/opt/terraform/account.json” or explicitly call it out in the gcp_variables.tf file. For more information see this document: Google Provider Configuration Reference – Terraform by HashiCorp

[20:29:21] dvarnum:terraform-gcp-cisco-vpn git:(master*) $ terraform plan
var.gcp_credentials_file
  Locate the GCP credentials file.

  Enter a value: /Users/dvarnum/Downloads/terraform-testing-272920-df6f321fa2bf.json

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # google_compute_external_vpn_gateway.external_gateway will be created
  + resource "google_compute_external_vpn_gateway" "external_gateway" {
      + description     = "An externally managed VPN gateway"
      + id              = (known after apply)
      + name            = "hq-cisco"
      + project         = (known after apply)
      + redundancy_type = "SINGLE_IP_INTERNALLY_REDUNDANT"
      + self_link       = (known after apply)

      + interface {
          + id         = 0

...truncated

Plan: 15 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

[20:30:45] dvarnum:terraform-gcp-cisco-vpn git:(master*) $ 

If you get the error similar to the following when applying, it means your permissions haven’t been set properly. Revisit the role assignments earlier in this blog for more information.

Error: Error creating Network: googleapi: Error 403: Required 'compute.networks.create' permission for 'projects/terraform-testing-272920/global/networks/tf-vpc-net-1', forbidden

  on main.tf line 35, in resource "google_compute_network" "tf_vpc_net1":
  35: resource "google_compute_network" "tf_vpc_net1" {

If everything looks good – time to run terraform apply

20:36:11] dvarnum:terraform-gcp-cisco-vpn git:(master*) $ terraform apply

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

…truncated

Plan: 15 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

google_compute_external_vpn_gateway.external_gateway: Creating...
google_compute_network.tf_vpc_net1: Creating...
google_compute_external_vpn_gateway.external_gateway: Still creating... [10s elapsed]
google_compute_network.tf_vpc_net1: Still creating... [10s elapsed]
google_compute_external_vpn_gateway.external_gateway: Creation complete after 12s [id=projects/terraform-testing-272920/global/externalVpnGateways/hq-cisco]
google_compute_network.tf_vpc_net1: Still creating... [20s elapsed]
google_compute_network.tf_vpc_net1: Creation complete after 23s [id=projects/terraform-testing-272920/global/networks/tf-vpc-net-1]
google_compute_ha_vpn_gateway.ha_gateway1: Creating...
google_compute_subnetwork.tf_vpc_net1_subnet2: Creating...
google_compute_router.router1: Creating...
google_compute_subnetwork.tf_vpc_net1_subnet1: Creating...
google_compute_firewall.tf_firewall: Creating...
google_compute_ha_vpn_gateway.ha_gateway1: Still creating... [10s elapsed]
google_compute_subnetwork.tf_vpc_net1_subnet2: Still creating... [10s elapsed]
google_compute_router.router1: Still creating... [10s elapsed]
google_compute_subnetwork.tf_vpc_net1_subnet1: Still creating... [10s elapsed]
google_compute_firewall.tf_firewall: Still creating... [10s elapsed]
google_compute_subnetwork.tf_vpc_net1_subnet2: Creation complete after 12s [id=projects/terraform-testing-272920/regions/us-west1/subnetworks/ha-vpn-subnet-2]
google_compute_instance.vm_instance2: Creating...
google_compute_firewall.tf_firewall: Creation complete after 13s [id=projects/terraform-testing-272920/global/firewalls/terraform-firewall-base]
google_compute_router.router1: Creation complete after 13s [id=projects/terraform-testing-272920/regions/us-central1/routers/ha-vpn-router-1]
google_compute_ha_vpn_gateway.ha_gateway1: Creation complete after 13s [id=projects/terraform-testing-272920/regions/us-central1/vpnGateways/ha-vpn-1]
google_compute_vpn_tunnel.tunnel2: Creating...
google_compute_vpn_tunnel.tunnel1: Creating...
google_compute_subnetwork.tf_vpc_net1_subnet1: Still creating... [20s elapsed]
google_compute_instance.vm_instance2: Still creating... [10s elapsed]
google_compute_vpn_tunnel.tunnel1: Still creating... [10s elapsed]
google_compute_vpn_tunnel.tunnel2: Still creating... [10s elapsed]
google_compute_subnetwork.tf_vpc_net1_subnet1: Creation complete after 23s [id=projects/terraform-testing-272920/regions/us-central1/subnetworks/ha-vpn-subnet-1]
google_compute_instance.vm_instance1: Creating...
google_compute_instance.vm_instance2: Still creating... [20s elapsed]
google_compute_vpn_tunnel.tunnel2: Still creating... [20s elapsed]
google_compute_vpn_tunnel.tunnel1: Still creating... [20s elapsed]
google_compute_instance.vm_instance1: Still creating... [10s elapsed]
google_compute_vpn_tunnel.tunnel1: Creation complete after 23s [id=projects/terraform-testing-272920/regions/us-central1/vpnTunnels/ha-vpn-tunnel1]
google_compute_router_interface.router1_interface1: Creating...
google_compute_vpn_tunnel.tunnel2: Creation complete after 24s [id=projects/terraform-testing-272920/regions/us-central1/vpnTunnels/ha-vpn-tunnel2]
google_compute_router_interface.router1_interface2: Creating...
google_compute_instance.vm_instance2: Creation complete after 25s [id=projects/terraform-testing-272920/zones/us-west1-b/instances/terraform-intance-2]
google_compute_instance.vm_instance1: Creation complete after 14s [id=projects/terraform-testing-272920/zones/us-central1-a/instances/terraform-intance-1]
google_compute_router_interface.router1_interface1: Still creating... [10s elapsed]
google_compute_router_interface.router1_interface2: Still creating... [10s elapsed]
google_compute_router_interface.router1_interface1: Creation complete after 13s [id=us-central1/ha-vpn-router-1/router1-interface1]
google_compute_router_peer.router1_peer1: Creating...
google_compute_router_interface.router1_interface2: Still creating... [20s elapsed]
google_compute_router_peer.router1_peer1: Still creating... [10s elapsed]
google_compute_router_interface.router1_interface2: Creation complete after 26s [id=us-central1/ha-vpn-router-1/router1-interface2]
google_compute_router_peer.router1_peer2: Creating...
google_compute_router_peer.router1_peer1: Still creating... [20s elapsed]
google_compute_router_peer.router1_peer2: Still creating... [10s elapsed]
google_compute_router_peer.router1_peer1: Still creating... [30s elapsed]
google_compute_router_peer.router1_peer2: Still creating... [20s elapsed]
google_compute_router_peer.router1_peer1: Creation complete after 38s [id=projects/terraform-testing-272920/regions/us-central1/routers/ha-vpn-router-1/router1-peer1]
google_compute_router_peer.router1_peer2: Still creating... [30s elapsed]
google_compute_router_peer.router1_peer2: Still creating... [40s elapsed]
google_compute_router_peer.router1_peer2: Creation complete after 47s [id=projects/terraform-testing-272920/regions/us-central1/routers/ha-vpn-router-1/router1-peer2]

Apply complete! Resources: 15 added, 0 changed, 0 destroyed.

Outputs:

gcp_external_vpn_address_1 = 35.242.126.28
gcp_external_vpn_address_2 = 35.220.87.218
gcp_instance1_external_ip = 34.71.134.246
gcp_instance1_internal_ip = 10.0.1.2
gcp_instance2_external_ip = 34.83.193.162
gcp_instance2_internal_ip = 10.0.2.2

Configuring the Cisco IOS-XE device

In my case I’m just using a CSR. The output values at the end of the terraform apply execution help us with building the IKEv2 configuration on our on-prem router. You can see the output at any time by running:

[20:38:58] dvarnum:terraform-gcp-cisco-vpn git:(master*) $ terraform output
gcp_external_vpn_address_1 = 35.242.126.28
gcp_external_vpn_address_2 = 35.220.87.218
gcp_instance1_external_ip = 34.71.134.246
gcp_instance1_internal_ip = 10.0.1.2
gcp_instance2_external_ip = 34.83.193.162
gcp_instance2_internal_ip = 10.0.2.2

Here is the sample Cisco IOS-XE configuration I’m using. Replace the values with “<<<<<<<<“ next to them with the appropriate values from the terraform output.

crypto ikev2 proposal VPN_IKEV2_PROPOSAL
 encryption aes-cbc-256 aes-cbc-192 aes-cbc-128
 integrity sha256
 group 16
!         
crypto ikev2 policy VPN_IKEV2_POLICY
 proposal VPN_IKEV2_PROPOSAL
!
crypto ikev2 keyring VPN_KEYRING
     peer GCP1
      address 35.242.126.28      <<<<<<<< ! Update this with gcp_external_vpn_address_1
      pre-shared-key d0v3r1a1d   ! Update this with your PSK
    !
     peer GCP2
      address 35.220.87.218      <<<<<<<< ! Update this with gcp_external_vpn_address_2
      pre-shared-key d0v3r1a1d   ! Update this with your PSK
    !
crypto ikev2 profile VPN_IKEV2_PROFILE
 match address local interface GigabitEthernet3
 match identity remote any
 identity local address X.X.X.X   ! I'm using NAT-T so I needed to specify my public NAT here
 authentication local pre-share
 authentication remote pre-share
 keyring local VPN_KEYRING
 lifetime 36000
 dpd 60 5 periodic
!
crypto ipsec security-association lifetime seconds 3600
crypto ipsec security-association replay window-size 1024
!
crypto ipsec transform-set VPN_TS esp-aes 256 esp-sha-hmac
 mode tunnel
!
crypto ipsec profile VPN_VTI
 set security-association lifetime seconds 3600
 set transform-set VPN_TS
 set pfs group16
 set ikev2-profile VPN_IKEV2_PROFILE
!
interface Tunnel1
 ip address 169.254.0.2 255.255.255.252
 ip mtu 1400
 ip tcp adjust-mss 1360
 tunnel source GigabitEthernet3
 tunnel mode ipsec ipv4
 tunnel destination 35.242.126.28     <<<<<<<< ! Update this with gcp_external_vpn_address_1
 tunnel protection ipsec profile VPN_VTI
!
interface Tunnel2
 ip address 169.254.1.2 255.255.255.252
 ip mtu 1400
 ip tcp adjust-mss 1360
 tunnel source GigabitEthernet3
 tunnel mode ipsec ipv4
 tunnel destination 35.220.87.218     <<<<<<<< ! Update this with gcp_external_vpn_address_1
 tunnel protection ipsec profile VPN_VTI
!
router bgp 65000
 bgp log-neighbor-changes
 neighbor 169.254.0.1 remote-as 64997
 neighbor 169.254.0.1 timers 20 60 60
 neighbor 169.254.0.1 description BGP session over Tunnel1
 neighbor 169.254.1.1 remote-as 64997
 neighbor 169.254.1.1 timers 20 60 60
 neighbor 169.254.1.1 description BGP session over Tunnel2
!
address-family ipv4
 network 10.200.0.1 mask 255.255.255.255
 neighbor 169.254.0.1 activate
 neighbor 169.254.1.1 activate
 exit-address-family

Once applied, we can start verifying.

Verification

VPC and subnets are created:

VPN Tunnels are up and BGP sessions established:

Cloud Router shows both peers up:

We’re looking the advertised 10.200.0.1/32 network advertised from the CSR:

Both VM instances have been deployed.

From the CSR, both crypto sessions are up.

R1#show crypto session 
Crypto session current status

Interface: Tunnel1
Profile: VPN_IKEV2_PROFILE
Session status: UP-ACTIVE     
Peer: 35.242.126.28 port 4500 
  Session ID: 3295  
  IKEv2 SA: local X.X.X.X/4500 remote 35.242.126.28/4500 Active 
  IPSEC FLOW: permit ip 0.0.0.0/0.0.0.0 0.0.0.0/0.0.0.0 
        Active SAs: 2, origin: crypto map

Interface: Tunnel2
Profile: VPN_IKEV2_PROFILE
Session status: UP-ACTIVE     
Peer: 35.220.87.218 port 4500 
  Session ID: 3293  
  IKEv2 SA: local X.X.X.X/4500 remote 35.220.87.218/4500 Active 
  IPSEC FLOW: permit ip 0.0.0.0/0.0.0.0 0.0.0.0/0.0.0.0 
        Active SAs: 2, origin: crypto map

Both Tunnel interfaces are up

R1#sh ip int b | i T
Tunnel1                169.254.0.2     YES manual up                    up      
Tunnel2                169.254.1.2     YES manual up                    up      

BGP is up and learning the GCP prefixes of both subnets – one in the central1 region and one from the west1 region:

R1#show ip bgp summ

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
169.254.0.1     4        64997      67      73       20    0    0 00:21:02        2
169.254.1.1     4        64997      67      75       20    0    0 00:21:05        2

R1#sh ip route bgp

      10.0.0.0/8 is variably subnetted, 5 subnets, 2 masks
B        10.0.1.0/24 [20/100] via 169.254.1.1, 00:22:16
B        10.0.2.0/24 [20/336] via 169.254.1.1, 00:22:16

Validate with some pings

R1#show ip int brie loopback 200
Interface              IP-Address      OK? Method Status                Protocol
Loopback200            10.200.0.1      YES manual up                    up      

R1#ping 10.0.1.2 source loopback 200
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.1.2, timeout is 2 seconds:
Packet sent with a source address of 10.200.0.1 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 61/70/105 ms

R1#ping 10.0.2.2 source loopback 200
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.2.2, timeout is 2 seconds:
Packet sent with a source address of 10.200.0.1 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 91/93/99 ms

Terraform State

Terraform created a terraform.tfstate file after applying the configuration. This state file is critical; it keeps track of the IDs of created resources so that Terraform knows what it is managing. This is unlike some other automation platforms such as Ansible where there really isn’t state tracking. For production environments, this state file should be on a shared repo and shared with anyone else who might need to manage or change the environment managed by Terraform. You can inspect the current state using terraform show

Making live changes with Terraform

Say I wanted to change something I deployed or add something new. All I’d need to do is make the changes in the terraform configuration files and then execute another terraform apply .

I’ll update the firewall policy to remove icmp and I’ll create a new vm instance.

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create
  ~ update in-place

Terraform will perform the following actions:

  # google_compute_firewall.tf_firewall will be updated in-place
  ~ resource "google_compute_firewall" "tf_firewall" {
        creation_timestamp      = "2020-04-02T20:37:08.949-07:00"
        destination_ranges      = []
        direction               = "INGRESS"
        disabled                = false
        enable_logging          = false
        id                      = "projects/terraform-testing-272920/global/firewalls/terraform-firewall-base"
        name                    = "terraform-firewall-base"
        network                 = "https://www.googleapis.com/compute/v1/projects/terraform-testing-272920/global/networks/tf-vpc-net-1"
        priority                = 1000
        project                 = "terraform-testing-272920"
        self_link               = "https://www.googleapis.com/compute/v1/projects/terraform-testing-272920/global/firewalls/terraform-firewall-base"
        source_ranges           = [
            "0.0.0.0/0",
        ]
        source_service_accounts = []
        source_tags             = []
        target_service_accounts = []
        target_tags             = []

        allow {
            ports    = [
                "22",
                "80",
                "443",
            ]
            protocol = "tcp"
        }
      - allow {
          - ports    = [] -> null
          - protocol = "icmp" -> null
        }
    }

  # google_compute_instance.vm_instance3 will be created
 ...truncated

Plan: 1 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

google_compute_firewall.tf_firewall: Modifying... [id=projects/terraform-testing-272920/global/firewalls/terraform-firewall-base]
google_compute_instance.vm_instance3: Creating...
google_compute_firewall.tf_firewall: Still modifying... [id=projects/terraform-testing-272920/global/firewalls/terraform-firewall-base, 10s elapsed]
google_compute_instance.vm_instance3: Still creating... [10s elapsed]
google_compute_firewall.tf_firewall: Modifications complete after 12s [id=projects/terraform-testing-272920/global/firewalls/terraform-firewall-base]
google_compute_instance.vm_instance3: Creation complete after 14s [id=projects/terraform-testing-272920/zones/us-west1-b/instances/terraform-intance-3]

Apply complete! Resources: 1 added, 1 changed, 0 destroyed.

We now have the new instance:

And I can no longer ping:

R1#ping 10.0.2.2 source loopback 200
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.2.2, timeout is 2 seconds:
Packet sent with a source address of 10.200.0.1 
.....
Success rate is 0 percent (0/5)

Terraform Destroy

When you’re done playing around (if that is all you’re doing) you can destroy everything that the terraform configuration is managing by using terraform destroy.

[21:30:50] dvarnum:terraform-gcp-cisco-vpn git:(master*) $ terraform destroy
Plan: 0 to add, 0 to change, 16 to destroy.

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

..truncated
Destroy complete! Resources: 16 destroyed.

Leave a Reply