Skip to main content

Using Private Clusters in GCP

Private cluster support in GCP is provided through network peering. GCP creates a VPC peering between the GKE cluster and a managed network owned by Google.

Wayfinder requires the public endpoint for Kubernetes Cloud API to remain enabled. So to achieve private clusters, you enable the private cluster setting in Wayfinder, place the workloads behind a gateway, and lock down the public endpoint (with the ACL that GCP provides) to the external addresses of the Wayfinder instance.

This topic provides details of this process.

Summary of the private cluster process​

Enabling private cluster support in GCP includes these high-level tasks. Click the links for more information.

  • Create an IP allocation to manage the address space.
  • Create a cluster plan to enforce private clusters settings.
  • Create a peering rule specifiying how users can access the private cluster.

The sections below provide an example of these and other tasks.

Example assumptions​

For the purposes of the example in this topic, we assume the following requirements:

  • Connectivity between Wayfinder and private clusters will be provided by direct peering.
  • We have been allocated a network range of 10.20.0.0/16, which will be sliced up and allocated on demand to the clusters.
  • We intend to use a dedicated virtual network 10.80.0.0/24, residing in another account, where a VPN will be situated for users.
  • We need to peer all our clusters with our CI network 10.70.0.0/24, which is running in another account.
  • We want to lock down the authentication module to only speak to CI and VPN.

Create the IP allocation for clusters​

This step is optional by highly recommended. Skipping this step places the responsibility of address space management on the workspaces.

To create an IP allocation for GKE:

  1. Create a resource file and apply it using the CLI:

wf apply -f FILENAME

Here is an example for GKE:

---
apiVersion: networks.appvia.io/v1alpha1
kind: AssignableNetwork
metadata:
  name: gke
  namespace: ws-admin
spec:
  provider: gke
  networks:
  - defaultMask: 21
    max: 21
    min: 14
    range: 10.200.0.0/16
    type: node
  - defaultMask: 18
    max: 20
    min: 14
    range: 10.80.0.0/16
    type: pod
  - defaultMask: 20
    max: 22
    min: 14
    range: 10.100.0.0/16
    type: service

The above IP allocation specifies IP ranges for nodes, pods, and services.

For more information, see IP Management.

Create a plan for private clusters​

This procedure creates a plan to enforce cluster settings. Here, we are going to ensure that:

  • The GKE public Kubernetes API can only be accessed by Wayfinder.
  • All clusters are private.
  • We lock down the authentication module to CI and VPN.

To create a plan for private clusters on GKE:

  1. Retrieve the external addresses of the Wayfinder instance:

wf -w admin get netfab management -o yaml

The addresses are listed in the status of the resource.

This address range is plugged into authorizedMasterNetworks below. For the purpose of this example we assume the address range is 192.78.32.10/32.

  1. Create a resource file, of kind: Plan, and apply it with the CLI command wf apply -f FILENAME.

Here are example settings for a GKE cluster plan named gke-default.

apiVersion: compute.appvia.io/v1alpha1
kind: Plan
metadata:
  name: gke-default
  namespace: ws-admin
spec:
  allocation:
    type: all
  policies:
  - editable: false
    path: spec.networking.authorizedMasterNetworks
  - editable: true
    path: spec.enablePrivateCluster
  - editable: true
    path: spec.providerDetails.gke.masterIPV4Cidr
  - editable: false
    path: spec.maintenanceWindow
  - editable: false
    path: spec.region
  - editable: false
    path: spec.version
  - editable: false
    path: spec.enableAutoUpgrade
  - editable: false
    path: spec.networking.authorizedNetworks
  - editable: false
    path: spec.networking.clusterIPV4Cidr
  - editable: false
    path: spec.networking.servicesIPV4Cidr
  - editable: false
    path: spec.networking.podsIPV4Cidr
  summary: Default GKE Cluster
  template:
    enableAutoUpgrade: true
    enablePrivateCluster: true
    maintenanceWindow: "03:00"
    networking:
      authorizedMasterNetworks:
      - cidr: 192.78.32.10/32
        name: wayfinder
      authorizedNetworks:
      - cidr: 10.80.0.0/24
        name: vpn
      - cidr: 10.70.0.0/24
        name: ci
      clusterIPV4Cidr: auto
      podsIPV4Cidr: auto
      servicesIPV4Cidr: auto
    nodePools:
    - autoscaling:
        enabled: true
        maxSize: 10
        minSize: 1
      diskSize: 30
      image: COS
      machine: n1-standard-2
      maxPodsPerNode: 110
      name: compute
      size: 1
    plan: gke-default
    provider: GKE
    providerDetails:
      gke:
        masterIPV4Cidr: 172.16.0.16/28
      type: GKE
    region: europe-west2
    version: regular

Create a cloud account and peering rule for VPN and CI​

For the purpose of this example we are assuming the VPN and CI reside in the same project, but in different networks. As such, we need a cloud account with the correct permissions to handle automatic acceptance.

note

If the networks resided in the same account as Wayfinder we could skip this step and reuse the ws-admin/management cloud account instead. Also, auto acceptance of external networks (CI and VPC under this example) is not a strict requirement. You can disable the feature and perform the necessary peering acceptance and routing change out-of-band. In this scenario the cluster(s) will be in a holding pattern waiting for the peering to be accepted.

Create the cloud account for VPN and CI​

To create the cloud account:

  1. Create a cloud account resource using yaml, and apply the file using the Wayfinder CLI:

wf apply -f FILENAME

Here is a sample file:

apiVersion: cloudaccess.appvia.io/v1alpha1
kind: CloudAccount
metadata:
  name: shared-services
  namespace: ws-admin
spec:
  accountType: shared
  allocation:
    type: all
  cloud: gcp
  defaultRegion: europe-west2
  identifier: wf-shared-dev
  identityCred:
    name: cloudidentity-gcp
    namespace: ws-admin
  name: wf-shared-dev
  orgIdentifier: ""
  providerDetails:
    type: GCPProject
  stages:
  - nonprod
  - prod

  1. Create a binding between the managed cloud identity Wayfinder runs under and the roles in the shared-services project:

    1. Switch to a gcloud configuration that has permissions to create IAM roles.

    2. Run the following CLI command:

    $ wf setup roles -w admin --cloud-account wf-shared-dev --feature Provisioning

For more information, see:

Define the peering rules for CI and VPN​

This procedure creates the peering rules to connect back to CI and VPN.

To define the peering rule:

  1. Create resource files of kind: PeeringRule, one for VPN and one for CI (see examples below). Then apply them with the CLI command:

wf apply -f FILENAME

note

When creating these resource files, use the fully qualified external network reference for identifier: <GCP_NETWORK_SELFLINK>. To retrieve these details, use the Google Cloud CLI to run:
gcloud compute networks describe NETWORK-NAME.

Example peering rule for VPN:

---
apiversion: networking.appvia.io/v1alpha1
kind: PeeringRule
metadata:
  name: vpn
spec:
  filters:
    allocation:
      type: all
    selectors:
      matchexpressions:
        - key: appvia.io/provider
          operator: in
          values: ["gcp"]

  # use the credential created above
  cloudaccount:
    name: shared-services
    namespace: ws-admin

  # define the connection to the vpc network
  connection:
    type: peering
    peering:
      enableAutoApproval: false
      network:
        account: <GCP_PROJECT>
        identifier: <GCP_NETWORK_SELFLINK>
        location: global # defaults to global

Example peering rule for CI:

---
apiversion: networking.appvia.io/v1alpha1
kind: PeeringRule
metadata:
  name: ci
spec:
  filters:
    allocation:
      type: all
    selectors:
      matchExpressions:
        - key: appvia.io/provider
          operator: in
          values: ["gcp"]

  # use the credential created above
  cloudaccount:
    name: shared-services
    namespace: ws-admin

  # define the connection to the network - here we are not automatically
  # accepting the request, this must be done out-of-baud
  connection:
    type: peering
    peering:
      enableAutoApproval: true
      network:
        account: <GCP_PROJECT>
        identifier: <GCP_NETWORK_SELFLINK>
        location: global # defaults to global

Create a private cluster​

Once the steps above are complete, workspaces can create private clusters, using the cluster plan gke-default (a plan that has private cluster enabled):

Last updated