Skip to main content
Version: 2.3

Using Private Clusters in GCP

Private cluster support in GCP is provided through network peering. GCP creates a VPC peering between the GKE cluster and a managed network owned by Google.

Wayfinder requires the public endpoint for Kubernetes Cloud API to remain enabled. So to achieve private clusters, you enable the private cluster setting in Wayfinder, place the workloads behind a gateway, and lock down the public endpoint (with the ACL that GCP provides) to the external addresses of the Wayfinder instance.

This topic provides details of this process.

Summary of the private cluster process​

Enabling private cluster support in GCP includes these high-level tasks. Click the links for more information.

  • Create an IP allocation to manage the address space.
  • Create a cluster plan to enforce private clusters settings.
  • Create a peering rule specifiying how users can access the private cluster.

The sections below provide an example of these and other tasks.

Example assumptions​

For the purposes of the example in this topic, we assume the following requirements:

  • Connectivity between Wayfinder and private clusters will be provided by direct peering.
  • We have been allocated a network range of 10.20.0.0/16, which will be sliced up and allocated on demand to the clusters.
  • We intend to use a dedicated virtual network 10.80.0.0/24, residing in another account, where a VPN will be situated for users.
  • We need to peer all our clusters with our CI network 10.70.0.0/24, which is running in another account.
  • We want to lock down the authentication module to only speak to CI and VPN.

Create the IP allocation for clusters​

This step is optional by highly recommended. Skipping this step places the responsibility of address space management on the workspaces.

To create an IP allocation for GKE:

  1. Create a resource file and apply it using the CLI:

    wf apply -f FILENAME

    Here is an example for GKE:

    ---
    apiVersion: networks.appvia.io/v1alpha1
    kind: AssignableNetwork
    metadata:
      name: gke
      namespace: ws-admin
    spec:
      provider: gke
      networks:
      - defaultMask: 21
        max: 21
        min: 14
        range: 10.200.0.0/16
        type: node
      - defaultMask: 18
        max: 20
        min: 14
        range: 10.80.0.0/16
        type: pod
      - defaultMask: 20
        max: 22
        min: 14
        range: 10.100.0.0/16
        type: service

    The above IP allocation specifies IP ranges for nodes, pods, and services.

For more information, see IP Management.

Create a plan for private clusters​

This procedure creates a plan to enforce cluster settings. Here, we are going to ensure that:

  • The GKE public Kubernetes API can only be accessed by Wayfinder.
  • All clusters are private.
  • We lock down the authentication module to CI and VPN.

To create a plan for private clusters on GKE:

  1. Retrieve the external addresses of the Wayfinder instance:

    wf -w admin get netfab management -o yaml

    The addresses are listed in the status of the resource.

    This address range is plugged into authorizedMasterNetworks below. For the purpose of this example we assume the address range is 192.78.32.10/32.

  2. Create a resource file, of kind: Plan, and apply it with the CLI command wf apply -f FILENAME.

    Here are example settings for a GKE cluster plan named gke-default.

    apiVersion: compute.appvia.io/v1alpha1
    kind: Plan
    metadata:
      name: gke-default
      namespace: ws-admin
    spec:
      allocation:
        type: all
      policies:
      - editable: false
        path: spec.networking.authorizedMasterNetworks
      - editable: true
        path: spec.enablePrivateCluster
      - editable: true
        path: spec.providerDetails.gke.masterIPV4Cidr
      - editable: false
        path: spec.maintenanceWindow
      - editable: false
        path: spec.region
      - editable: false
        path: spec.version
      - editable: false
        path: spec.enableAutoUpgrade
      - editable: false
        path: spec.networking.authorizedNetworks
      - editable: false
        path: spec.networking.clusterIPV4Cidr
      - editable: false
        path: spec.networking.servicesIPV4Cidr
      - editable: false
        path: spec.networking.podsIPV4Cidr
      summary: Default GKE Cluster
      template:
        enableAutoUpgrade: true
        enablePrivateCluster: true
        maintenanceWindow: "03:00"
        networking:
          authorizedMasterNetworks:
          - cidr: 192.78.32.10/32
            name: wayfinder
          authorizedNetworks:
          - cidr: 10.80.0.0/24
            name: vpn
          - cidr: 10.70.0.0/24
            name: ci
          clusterIPV4Cidr: auto
          podsIPV4Cidr: auto
          servicesIPV4Cidr: auto
        nodePools:
        - autoscaling:
            enabled: true
            maxSize: 10
            minSize: 1
          diskSize: 30
          image: COS
          machine: n1-standard-2
          maxPodsPerNode: 110
          name: compute
          size: 1
        plan: gke-default
        provider: GKE
        providerDetails:
          gke:
            masterIPV4Cidr: 172.16.0.16/28
          type: GKE
        region: europe-west2
        version: regular

Create a cloud account and peering rule for VPN and CI​

For the purpose of this example we are assuming the VPN and CI reside in the same project, but in different networks. As such, we need a cloud account with the correct permissions to handle automatic acceptance.

note

If the networks resided in the same account as Wayfinder we could skip this step and reuse the ws-admin/management cloud account instead. Also, auto acceptance of external networks (CI and VPC under this example) is not a strict requirement. You can disable the feature and perform the necessary peering acceptance and routing change out-of-band. In this scenario the cluster(s) will be in a holding pattern waiting for the peering to be accepted.

Create the cloud account for VPN and CI​

To create the cloud account:

  1. Create a cloud account resource using yaml, and apply the file using the Wayfinder CLI:

    wf apply -f FILENAME

    Here is a sample file:

    apiVersion: cloudaccess.appvia.io/v1alpha1
    kind: CloudAccount
    metadata:
      name: shared-services
      namespace: ws-admin
    spec:
      accountType: shared
      allocation:
        type: all
      cloud: gcp
      defaultRegion: europe-west2
      identifier: wf-shared-dev
      identityCred:
        name: cloudidentity-gcp
        namespace: ws-admin
      name: wf-shared-dev
      orgIdentifier: ""
      providerDetails:
        type: GCPProject
      stages:
      - nonprod
      - prod

  2. Create a binding between the managed cloud identity Wayfinder runs under and the roles in the shared-services project:

    1. Switch to a gcloud configuration that has permissions to create IAM roles.

    2. Run the following CLI command:

      $ wf setup roles -w admin --cloud-account wf-shared-dev --feature Provisioning

For more information, see:

Define the peering rules for CI and VPN​

This procedure creates the peering rules to connect back to CI and VPN.

To define the peering rule:

  1. Create resource files of kind: PeeringRule, one for VPN and one for CI (see examples below). Then apply them with the CLI command:

    wf apply -f FILENAME

    note

    When creating these resource files, use the fully qualified external network reference for identifier: <GCP_NETWORK_SELFLINK>. To retrieve these details, use the Google Cloud CLI to run:
    gcloud compute networks describe NETWORK-NAME.

    Example peering rule for VPN:

    ---
    apiversion: networking.appvia.io/v1alpha1
    kind: PeeringRule
    metadata:
      name: vpn
    spec:
      filters:
        allocation:
          type: all
        selectors:
          matchexpressions:
            - key: appvia.io/provider
              operator: in
              values: ["gcp"]

      # use the credential created above
      cloudaccount:
        name: shared-services
        namespace: ws-admin

      # define the connection to the vpc network
      connection:
        type: peering
        peering:
          enableAutoApproval: false
          network:
            account: <GCP_PROJECT>
            identifier: <GCP_NETWORK_SELFLINK>
            location: global # defaults to global

    Example peering rule for CI:

    ---
    apiversion: networking.appvia.io/v1alpha1
    kind: PeeringRule
    metadata:
      name: ci
    spec:
      filters:
        allocation:
          type: all
        selectors:
          matchExpressions:
            - key: appvia.io/provider
              operator: in
              values: ["gcp"]

      # use the credential created above
      cloudaccount:
        name: shared-services
        namespace: ws-admin

      # define the connection to the network - here we are not automatically
      # accepting the request, this must be done out-of-baud
      connection:
        type: peering
        peering:
          enableAutoApproval: true
          network:
            account: <GCP_PROJECT>
            identifier: <GCP_NETWORK_SELFLINK>
            location: global # defaults to global

Create a private cluster​

Once the steps above are complete, workspaces can create private clusters, using the cluster plan gke-default (a plan that has private cluster enabled):

Last updated on 18 Jul 2023