Version: 2.3

Using Private Clusters in Azure

Private cluster support in Azure is provided via network peering. Azure supports transitive routing which can be taken advantage of to great effect when creating hub and spoke networks. Though Wayfinder is flexible enough to support a variety of topologies:

Direct peering back to the management network
Peering to a hub and spoke transit network
Peering to a hub and spoke and using the hub as the default gateway
A combination of all the above

Summary of the private cluster process

Enabling private cluster support in Azure includes these high-level tasks. Click the links for more information.

Create an IP allocation to manage the address space.
Create a plan to enforce private clusters settings, see plans.
Create a peering rule to define how we will access the management network.

The sections below provide an example of these and other tasks.

Example assumptions

For the purposes of this quickstart guide we will assume the following requirements. Though if you want to look at hub & spoke setups visit here

Connectively between the Wayfinder 10.170.0.0/22 network and clusters will be provided by direct peering.
We have been allocated a network range of 10.20.0.0/16, which we will slice up and provide to the private clusters.
We intend to use a dedicated network 10.80.0.0/24 in another account for VPN for users.
We need to peer all networks to our CI 10.70.0.0/24, running in another account.
We want to lock down the authentication module to only speak to CI and VPN.

Create the IP Allocation for clusters

note

Currently we have no support for IP management of the AKS service address range, which defaults to 172.20.0.0/16. When creating a peering between networks ensure you don't overlap this range.

Create a resource file and apply it using the CLI:

wf apply -f FILENAME

Here is an example for AKS:

---
apiVersion: networks.appvia.io/v1alpha1
kind: AssignableNetwork
metadata:
  name: aks
  namespace: ws-admin
spec:
  provider: aks
  networks:
  - defaultMask: 21
    max: 21
    min: 14
    range: 10.200.0.0/16
    type: node

For more information, see IP Management.

Create a plan for private clusters

This procedure is optional. This creates a plan to enforce the cluster settings.

To create a plan for private clusters on AKS:

Create a resource file, of kind: Plan, and apply it with the CLI command wf apply -f FILENAME.

Here are example settings for an AKS cluster plan named aks-default.

apiVersion: compute.appvia.io/v1alpha1
kind: Plan
metadata:
  name: aks-default
  namespace: ws-admin
spec:
  allocation:
    type: all
  policies:
  - editable: false
    path: spec.networking.authorizedMasterNetworks
  - editable: true
    path: spec.enablePrivateCluster
  - editable: false
    path: spec.maintenanceWindow
  - editable: false
    path: spec.region
  - editable: false
    path: spec.version
  - editable: false
    path: spec.enableAutoUpgrade
  - editable: false
    path: spec.networking.authorizedNetworks
  - editable: false
    path: spec.networking.clusterIPV4Cidr
  summary: Default AKS Cluster
  template:
    enableAutoUpgrade: true
    enablePrivateCluster: true
    maintenanceWindow: "03:00"
    networking:
      authorizedMasterNetworks:
      - cidr: 10.170.0.0/22
        name: wayfinder
      authorizedNetworks:
      - cidr: 10.80.0.0/24
        name: vpn
      - cidr: 10.70.0.0/24
        name: ci
      clusterIPV4Cidr: auto
      # we do not support auto assignment on the service range yet
      servicesIPV4Cidr: 172.16.0.0/20
    nodePools:
    - autoscaling:
        enabled: true
        maxSize: 10
        minSize: 1
      diskSize: 30
      image: Linux
      machine: Standard_D2_v2
      maxPodsPerNode: 110
      name: system
      size: 1
      providerDetails:
        aks:
          mode: system
        type: AKS
    plan: aks-default
    provider: AKS
    region: uksouth
    version: latest

Create a peering connection from workspace clusters back to Wayfinder

Wayfinder requires direct connectivity to the kubernetes API. This is the cloud managed Kubernetes API, not the authentication module.

To define the peering rule:

Create a resource file of kind: PeeringRule, and apply it with the CLI command wf apply -f FILENAME.

Here's an example:

---
apiVersion: networking.appvia.io/v1alpha1
kind: PeeringRule
metadata:
  name: management
spec:
  filters:
    allocation:
      type: all
    selectors:
      matchExpressions:
        - key: networking.appvia.io/peering
          operator: In
          values: ["true"]
        - key: appvia.io/provider
          operator: In
          values: ["azure"]
  connection:
    type: peering
    peering:
      enableAutoApproval: true

Create a cloud account and peering rule for VPN and CI

For the purpose of this example we are assuming the VPN and CI reside in the same subscription, but in different networks. As such, we need a cloud account with the correct permissions to handle automatic acceptance.

note

If the networks resided in the same account as Wayfinder we could skip this step and reuse the ws-admin/management cloud account instead. Also, auto acceptance of external networks (CI and VPC under this example) is not a strict requirement. You can disable the feature and perform the necessary peering acceptance and routing change out-of-band. In this scenario the cluster(s) will be in a holding pattern waiting for the peering to be accepted.

Create the account for VPN and CI

To create the cloud account:

Create a cloud account resource using yaml, and apply the file using the Wayfinder CLI:

wf apply -f FILENAME

Here is a sample file:

apiVersion: cloudaccess.appvia.io/v1alpha1
kind: CloudAccount
metadata:
  name: shared-services
  namespace: ws-admin
spec:
  accountType: shared
  allocation:
    type: all
  cloud: azure
  defaultRegion: ukwest
  features:
  - Provisioning
  identifier: <SUBSCRIPTION_ID>
  identityCred:
    name: wf-azure
    namespace: ws-admin
  name: shared-services
  orgIdentifier: ""
  providerDetails:
    type: AzureSubscription
  roles:
  - role: None
  - role: ClusterManager
  - role: NetworkManager
  - role: DNSZoneManager
  stages:
  - prod
  - nonprod

Create a binding between the managed cloud identity Wayfinder runs under and the roles in the shared-services project:
1. Switch your Azure CLI session to one that has permissions to create IAM roles in the designated subscription above.
2. Use the Wayfinder CLI to run:
  $ wf setup roles -w admin --cloud-account shared-services --feature Provisioning

For more information, see:

wf create cloudaccounts for the CLI method
Cloud Accounts for detailed information

Define the peering rules for VPN and CI

This procedure creates the peering rules to peer CI and VPN.

To define the peering rules:

Create resource files of kind: PeeringRule, one for VPN and one for CI (see examples below). Then apply them with the CLI command:

wf apply -f FILENAME

Note the following when creating these resource files:

When referring to the external virtual network in these resource files, use the fully qualified resource, for example: /subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP/providers/Microsoft.Network/virtualNetworks/NAME
To obtain a VNET_ID, run the CLI command:
az network vnet list --resource-group RESOURCE_GROUP_NAME

Example peering rule for VPN:

---
apiVersion: networking.appvia.io/v1alpha1
kind: PeeringRule
metadata:
  name: vpn
spec:
  filters:
    allocation:
      type: all
    selectors:
      matchExpressions:
        - key: appvia.io/provider
          operator: in
          values: ["azure"]

  # use the credential created above
  cloudAccount:
    name: shared-services
    namespace: ws-admin

  # define the connection to the vpc network
  connection:
    type: peering
    peering:
      enableAutoApproval: false
      network:
        account: <SUBSCRIPTION>
        identifier: <RESOURCE_GROUP>/<VNET_ID>
        location: <REGION>

Example peering rule for CI:

---
apiversion: networking.appvia.io/v1alpha1
kind: PeeringRule
metadata:
  name: ci
spec:
  filters:
    allocation:
      type: all
    selectors:
      matchExpressions:
        - key: appvia.io/provider
          operator: in
          values: ["azure"]

  # use the credential created above
  cloudAccount:
    name: shared-services
    namespace: ws-admin

  # define the connection to the network - here we are not automatically
  # accepting the request, this must be done out-of-baud
  connection:
    type: peering
    peering:
      enableAutoApproval: true
      network:
        account: <SUBSCRIPTION>
        identifier: <RESOURCE_GROUP>/<VNET_ID>
        location: <REGION>

Create a private cluster

Once the steps above are complete, workspaces can create private clusters, using the cluster plan aks-default (a plan that has private cluster enabled):

CLI: wf create cluster CLUSTERNAME --plan aks-default. See wf create cluster.
UI: See Create a cluster.

Summary of the private cluster process​

Example assumptions​

Create the IP Allocation for clusters​

Create a plan for private clusters​

Create a peering connection from workspace clusters back to Wayfinder​

Create a cloud account and peering rule for VPN and CI​

Create the account for VPN and CI​

Define the peering rules for VPN and CI​

Create a private cluster​

Summary of the private cluster process

Example assumptions

Create the IP Allocation for clusters

Create a plan for private clusters

Create a peering connection from workspace clusters back to Wayfinder

Create a cloud account and peering rule for VPN and CI

Create the account for VPN and CI

Define the peering rules for VPN and CI

Create a private cluster