Anthos on AWS is GA!

Anthos ona AWS in now GA an Anthos on Azure is in a preview. GKE on AWS is a simplified installation of the Admin control plane. It uses AWS Elastic Compute Cloud (EC2)Elastic Block Storage (EBS), and Elastic Load Balancer (ELB) services. The bellow diagrams shows high level architecture. The deployment is done using Terraforms.

The details about architecture you can found here:

The installation process can be found her:

Note: the docs are still pending update as it shows „Beta”

The war between Anthos and VMware Tanzu is getting more exciting to watch :).

Anthos 1.3 is GA on March 23, 2020!

There are some revolutionary features coming to GKE-OP:

  • A new installer helps you create and prepare the admin workstation.
  • In bundled load balancing mode, GKE on-prem provides and manages the Seesaw load balancer.
  • The Authentication Plugin for Anthos has been integrated into and replaced with the Google Cloud command-line interface, which improves the authentication process and provides the user consent flow through gcloud commands.
  • vSphere credential rotation is enabled. Users can now use Solution User Certificates to authenticate to GKE deployed on-prem.
  • Preview Feature: Introducing User cluster Nodepools.
  • gkectl automatically uses the proxy URL from config.yaml to configure the proxy on the admin workstation.

Full list of features is available here:

In the follow up post we will have a closer look at those new features!

Anthos 1.2 GKE-OP backup script issues

To backup your Anthos GKE-OP cluster Google provides a nice script you can schedule to run as a crone job.

The only problem is that it misses last line where you actually copy the snapshot of the admin cluster ETCD database.

Add the bellow line to make it work:

kubectl --kubeconfig=${ADMIN_CLUSTER_KUBECONFIG} cp kube-system/${admin_etcd}:admin_snapshot.db $BACKUP_DIR/

Once added it work like a charm 🙂

ubuntu@admin-workstation2:~/backup$ ls
admin_snapshot.db gke-03-usercluster01_snapshot.db pki

In the output you might see some errors related to the issue I explained here: but you can ignore it

Anthos 1.x issue when running sudo commands

When you login to GKE-OP nodes and try to run sudo command you will get the following warning:

sudo: unable to resolve host [nodename]

Your command will still execute but will show this warning. It is related to Ubuntu OS settings. To resolve it add the following line into the /etc/hosts file on the node: [node-name]

Hope this will be solved soon as Google has already identified this issue. I guess they will add the record in the provisioning process for the nodes.

Installing Istio on GKE-OP for Anthos


GKE-OP 1.1.2 supports open source Istio version 1.1.13. To perform the installation you require a user cluster to be installed and validated. The procedure of installation can be found here:

In this article we will show hot to install Istio and a simple microservice application. We will generate some traffic to that application and visualise the flows with Kiali.

The high level steps are as follows:

  • install Helm
  • deploy Istio CRDs
  • deploy Istio
  • expose Telemetry services
  • install BookInfo application

All the steps are performed from the Admin workstation

Installing Helm

Download Helm running:

curl --output helm-v2.16.1-linux-amd64.tar.gz

Unzip it, move to the bin folder and see if you can check the version

tar -zxvf helm-v2.16.1-linux-amd64.tar.gz

mv linux-amd64/helm /usr/local/bin/helm

Helm version

Install CRDs

helm template install/kubernetes/helm/istio-init --name istio-init --namespace istio-system | kubectl apply -f -

Setup Kiali password

KIALI_USERNAME=$(read -p 'Kiali Username: ' uval && echo -n $uval | base64)

KIALI_PASSPHRASE=$(read -sp 'Kiali Passphrase: ' pval && echo -n $pval | base64)

when prompted pass the username and password

cat <<EOF | kubectl apply -f –

apiVersion: v1

kind: Secret


  name: kiali

  namespace: $NAMESPACE


    app: kiali

type: Opaque


  username: $KIALI_USERNAME

  passphrase: $KIALI_PASSPHRASE


Install Istio using the the demo pattern – this icludes Kiali, Grafana and Jeagger.

helm template install/kubernetes/helm/istio --name istio --namespace istio-system \ --values install/kubernetes/helm/istio/values-istio-demo.yaml | kubectl apply -f -

Check that services are running

kubectl get service -n istio-system

kubectl get pods -n istio-system

Edit the Istio ingress gateway to assing IP address the Istio Gateway.

kubectl edit svc -n istio-system istio-ingressgateway



 loadBalancerIP: <IP_Address>

Check that IP is assigned

kubectl get service -n istio-system

Expose Kiali service

For reference you can use:

cat <<EOF | kubectl apply -f –


kind: Gateway


  name: kiali-gateway

  namespace: istio-system



    istio: ingressgateway


  – port:

      number: 15029

      name: http-kiali

      protocol: HTTP


    – „*”


kind: VirtualService


  name: kiali-vs

  namespace: istio-system



  – „*”


  – kiali-gateway


  – match:

    – port: 15029


    – destination:

        host: kiali


          number: 20001


kind: DestinationRule


  name: kiali

  namespace: istio-system


  host: kiali



      mode: DISABLE


Connect to Kiali

Deploy the application

kubectl apply -f <(istioctl kube-inject -f samples/bookinfo/platform/kube/bookinfo.yaml)

watch kubectl get pods

kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml

Issues with F5 BIG-IP load balancer in GKE-OP Anthos 1.x – K8s APIs not responding

When using F5 BIG-IP load balancer of GKE On-Prem you might be wanting to use evaluation license. Keep in mind that this license has a restriction of 2MBps bandwidth in total. GKE-OP even with one user cluster can cause saturation and slowness of K8s API response. With multiple cluster and Istio installed the API can stop response at all. Note that F5 might not be showing the bandwith is saturated when you use the CLI tools.

Resolution: use full license or request 10GBps evaluation license.

Problems creating pre-check VM in Anthos 1.2 GKE-OP

With Anthos 1.2 there is a new feature that creates a test VM to check connectivities before you deploy your GKE-OP clusters. It helps to avoid issues during the installation.

When installing you GKE On-Prem using the following documentation: you perform checks with the following commands

gkectl check-config --config [PATH_TO_CONFIG]

you will get an error as bellow:

  • Validation Category: F5 BIG-IP
    • [FAILURE] Admin Cluster VIP and NodeIP: Failed to create VM: failed to create VM (not retriable): failed to find VM template "gke-on-prem-osimage-1.14.7-gke.24mage-1.14.7-gke.24-20191120-f71f9a709b' not found
    • [FAILURE] User Cluster VIP and NodeIP: Failed to create VM: failed to create VM (not retriable): failed to find VM template "gke-on-prem-osimage-1.14.7-gke.24-age-1.14.7-gke.24-20191120-f71f9a709b' not found

Root cause: This is cause by the image not being present on the datastore. The installation steps in the GCP docs have wrong sequence.

Solution: run

gkectl prepare --config [CONFIG_FILE] --validate-attestations

After that the VMs get created and connectivity checks can be performed