Skip to main content

Using Private Clusters in AWS

Private cluster support in AWS can be achieved using one of these methods:

  • Direct peering between the control plane and the workspace networks
  • Hub and spoke model provided by AWS Transit Gateway

This topic provides a high-level summary for private cluster support in AWS, and a worked example of the process.

Summary of the private cluster process​

Enabling private cluster support in AWS includes these high-level tasks. Click the links for more information.

The sections below provide an example of these and other tasks.

Example assumptions​

For the purposes of the example in this topic, we assume the following requirements:

  • Connectivity between Wayfinder and private clusters will be facilitated by direct peering.
  • We have been allocated a network range of 10.20.0.0/16, which will be sliced up and allocated on demand to the clusters.
  • We intend to use a dedicated virtual network 10.80.0.0/24, residing in another account, where a VPN will be situated for users.
  • We need to peer all our clusters with our CI network 10.70.0.0/24, which is running in another account.
  • The CI solution is managed by another team that wants to manually approve peering.
  • We want to lock down the authentication module to only speak to CI and VPN.

Create the IP allocation for clusters​

This step is optional, but highly recommended.

An IP allocation for the cloud provider sets up a pool of address spaces, which is sliced up and allocated on demand to the clusters. This ensures that you don't create peers with overlapping address spaces, or networks that may overlap with shared services.

To add an IP allocation using the UI or CLI, see detailed instructions in Add a network allocation.

note

Currently we have no support for IP management of the EKS service address range, which defaults to 172.20.0.0/16. When creating a peering between networks, ensure that you don't overlap this range. Otherwise, the result will be a peering status of success, but the internal routing for kubernetes will ignore the whole range.

To create an IP allocation for EKS:

  1. Create a resource file and apply it using the CLI:

wf apply -f FILENAME

Here is an example for EKS:

---
apiVersion: networks.appvia.io/v1alpha1
kind: AssignableNetwork
metadata:
  name: eks
  namespace: ws-admin
spec:
  provider: eks
  networks:
  - defaultMask: 21
    range: 10.20.0.0/16 # we should be able to fit 32 clusters in the range
    type: node
  - defaultMask: 18
    min: 21
    max: 24
    type: node

For more information, see IP Management.

Create a plan for private clusters​

This step is optional, but if skipped, it is the responsibility of the workspaces to ensure the clusters are defined as private, i.e., spec.enablePrivateCluster is set to true in yaml, or the Private cluster setting is enabled in the UI.

To create a plan for private clusters on EKS:

  1. Create a resource file, of kind: Plan, and apply it with the CLI command:

wf apply -f FILENAME.

Below is a sample plan. You can prevent workspaces from editing a parameter when creating a cluster by setting editable to false for that parameter.

apiVersion: compute.appvia.io/v1alpha1
kind: Plan
metadata:
  name: eks-default
  namespace: ws-admin
spec:
  allocation:
    type: all
  policies:
  - editable: false
    path: spec.networking.authorizedNetworks
  - editable: false
    path: spec.networking.authorizedMasterNetworks
  - editable: false
    path: spec.enablePrivateCluster
  - editable: false
    path: spec.networking.clusterIPV4Cidr
  summary: Default EKS Template
  template:
    addons:
      ingress:
        enabled: false
    enablePrivateCluster: true
    networking:
      authorizedMasterNetworks:
      - cidr: 0.0.0.0/0
        name: default
      authorizedNetworks:
      - cidr: 10.70.0.0/24
        name: ci
      - cidr: 10.80.0.0/24
        name: vpn
      clusterIPV4Cidr: auto
    nodePools:
    - autoscaling:
        enabled: true
        maxSize: 10
        minSize: 1
      diskSize: 30
      image: AL2_x86_64
      machine: t3.medium
      name: compute
      size: 2
      spot:
        enabled: true
    provider: EKS
    region: eu-west-2
    version: latest

View instructions to create cluster plans.

Create a peering connection from workspace clusters back to Wayfinder​

Wayfinder requires direct connectivity to the Kubernetes API. This is the cloud managed Kubernetes API, not the authentication module. Since we've just removed all public access to the endpoint we need to create a peering back to the management network, shown in the example below.

---
apiVersion: networking.appvia.io/v1alpha1
kind: PeeringRule
metadata:
  name: management
spec:
  filters:
    allocation:
      type: all
    selectors:
      matchExpressions:
        - key: networking.appvia.io/peering
          operator: In
          values: ["true"]
        - key: appvia.io/provider
          operator: In
          values: ["aws"]
  connection:
    type: peering
    peering:
      enableAutoApproval: true

Create the cloud account for VPN and CI​

For the purpose of this example, we can assume the VPN and CI system reside in the same AWS account, but in different networks. So we need a cloud account with the correct permissions to handle automatic acceptance.

note

If the networks resided in the same account as Wayfinder we could skip this step and reuse the ws-admin/management cloud account instead. Also, auto acceptance of external networks (CI and VPC under this example) is not a strict requirement. You can disable the feature and perform the necessary peering acceptance and routing change out-of-band. In this scenario the cluster(s) will be in a holding pattern waiting for the peering to be accepted.

To create the cloud account for VPN and CI:

  1. Create a cloud account using yaml, and apply the file using the Wayfinder CLI:

wf apply -f FILENAME

Here is a sample file. The important element to note here is the role required in spec.roles.

---
apiVersion: cloudaccess.appvia.io/v1alpha1
kind: CloudAccount
metadata:
  name: shared-services
  namespace: ws-admin
spec:
  accountType: shared
  allocation:
    type: none
  cloud: aws
  defaultRegion: eu-west-2
  features:
  - NetworkManager
  identifier: <AWS_ACCOUNT_ID>
  identityCred:
    name: management
    namespace: ws-admin
  name: shared
  orgIdentifier: ""
  providerDetails:
    type: AWSAccount
  roles:
  - role: NetworkManager
  stages:
  - nonprod
  - prod

The above creates a cloud account linked to the identity that Wayfinder runs under. We don't have access yet.

  1. We need a user with permissions in the account to delegate the NetworkManager role to us.

    1. Switch into an AWS profile/session that has IAM access in the target account.
    2. Run the command: wf setup roles -w admin --cloud-account shared-services --feature NetworkManager.

The required role is provisioned behind the scenes in the target account, and a trust policy is created between the account and the cloud managed identity Wayfinder is running under. (The cloud managed identity is provided by EKS pod identity service.)

Create the peering rules for CI and VPN​

The following example creates the peering rules necessary for CI and VPN.

To create peering rules for CI and VPN:

  1. Create resource files of kind: peeringRule, one for VPN and one for CI (see examples below). Then apply the files using the CLI:

wf apply -f <FILENAME>

Example peering rule for VPN:

---
apiVersion: networking.appvia.io/v1alpha1
kind: peeringRule
metadata:
  name: vpn
spec:
  filters:
    allocation:
      type: all
    selectors:
      matchExpressions:
        - key: appvia.io/provider
          operator: in
          values: ["aws"]

  # use the credential created above
  cloudAccount:
    name: shared-services
    namespace: ws-admin

  # define the connection to the vpc network
  connection:
    type: peering
    peering:
      enableAutoApproval: true
      network:
        account: <aws_account_id>
        identifier: <aws_vpc_id>
        location: <aws_region>
        routes:
          - 10.80.0.0/24
        routeTableSelectors:
          tag:name: vpn-main

Example peering rule for CI:

---
apiVersion: networking.appvia.io/v1alpha1
kind: peeringRule
metadata:
  name: ci
spec:
  filters:
    allocation:
      type: all
    selectors:
      matchExpressions:
        - key: appvia.io/provider
          operator: in
          values: ["aws"]
        - key: networking.appvia.io/peering
          operator: in
          values: ["true"]

  # use the credential created above
  cloudAccount:
    name: shared-services
    namespace: ws-admin

  # define the connection to the vpc network - here we are not automatically
  # accepting the request, this must be done out-of-baud
  connection:
    type: peering
    peering:
      enableAutoApproval: false
      network:
        account: <aws_account_id>
        identifier: <aws_vpc_id>
        location: <aws_region>
        routes:
          - 10.70.0.0/24

Create a private cluster​

Once the steps above are complete, workspaces can create private clusters, using the cluster plan eks-default (a plan that has private cluster enabled):

Last updated