The Atomist Kubernetes integration requires that the k8svent utility be installed in your Kubernetes cluster. Once the k8svent utility is installed and running in your cluster, it will send events to Atomist. If your Atomist Kubernetes integration page is reporting that it has not received any events from k8svent, something may have gone wrong with the installation. Below are some tips to help troubleshoot why Atomist is not receiving events from your Kubernetes cluster.

Installation failure

If you see errors like the following when you try to install the Atomist Kubernetes integration to your Kubernetes cluster,

Error from server (Forbidden): error when creating "cluster-wide.yaml": clusterroles.rbac.authorization.k8s.io "k8s-sdm-clusterrole" is forbidden: attempt to grant extra privileges: [...] user=&{YOUR_USER [system:authenticated] map[]} ownerrules=[PolicyRule{Resources:["selfsubjectaccessreviews"], APIGroups:["authorization.k8s.io"], Verbs:["create"]} PolicyRule{NonResourceURLs:["/api" "/api/*" "/apis" "/apis/*" "/healthz" "/swagger-2.0.0.pb-v1" "/swagger.json" "/swaggerapi" "/swaggerapi/*" "/version"], Verbs:["get"]}] ruleResolutionErrors=[]

then your Kubernetes user does not have administrative privileges on your cluster. You will either need to ask someone who has admin privileges on the cluster to create the RBAC resources or try to escalate your privileges in the cluster. To attempt to provide your Kubernetes user with cluster admin privileges, run the following command, replacing USER with your Kubernetes user name:

$ kubectl create clusterrolebinding USER-cluster-admin-binding \
--clusterrole=cluster-admin --user=USER

If you are using GKE, you can find your user name using the gcloud command-line utility.

$ kubectl create clusterrolebinding \
$(gcloud config get-value account | sed 's/@.*//')-cluster-admin-binding \
--clusterrole=cluster-admin \
--user=$(gcloud config get-value account)

Once your Kubernetes user has the necessary privileges, run the command to install the Atomist Kubernetes integration again.

kubectl

If installation was successful and Atomist still reports that it is not receiving events from your cluster, make sure the k8svent pod container is running and healthy in your cluster. To do this, run the following kubectl command.

$ kubectl get pods -n k8svent
NAME READY STATUS RESTARTS AGE
k8svent-65d576999b-nqx99 1/1 Running 0 5m

If your output does not look something like the above output, try deleting the k8svent pod and allow Kubernetes to create its replacement.

$ kubectl delete -n k8svent \
$(kubectl get pods -n k8svent -o name | grep /k8svent-)

If that does not help, check the pod logs.

Logs

The k8svent pod logs can help to diagnose issues. You can access the logs for k8svent the same way you would for any Kubernetes pod.

$ kubectl logs -n k8svent \
$(kubectl get pods -n k8svent | \
awk '$1 ~ /^k8svent-/ && $2 == "1/1" { print $1; exit }')

The logs should start something like

{"host":"k8svent-65d576999b-nqx99","level":"info","msg":"k8svent version 0.17.0-2-gcec2dd2 starting","service":"k8svent","time":"2020-08-01T20:17:05Z"}
{"host":"k8svent-65d576999b-nqx99","level":"info","msg":"Creating Kubernetes API client set","service":"k8svent","time":"2020-08-01T20:17:05Z"}
{"host":"k8svent-65d576999b-nqx99","level":"info","msg":"Starting to vent","service":"k8svent","time":"2020-08-01T20:17:05Z"}
{"host":"k8svent-65d576999b-nqx99","level":"info","msg":"Using Docker image tag 'latest' for digest check","service":"k8svent","time":"2020-08-01T20:17:05Z"}
{"host":"k8svent-65d576999b-nqx99","level":"info","msg":"Posting to 'https://webhook.atomist.com/atomist/resource/XYZ'","pod":"api/a-5bbf64648d-gwg7n","service":"k8svent","time":"2020-08-01T20:17:06Z"}
{"host":"k8svent-65d576999b-nqx99","level":"info","msg":"Posting to 'https://webhook.atomist.com/atomist/resource/XYZ'","pod":"api/b-6bbf64648d-mhddx","service":"k8svent","time":"2020-08-01T20:17:06Z"}

within a few seconds you should see messages confirming successful delivery of the webhook payloads that look like


{"host":"k8svent-65d576999b-nqx99","level":"warning","msg":"Failed to extract correlation ID from https://webhook.atomist.com/atomist/resource/XYZ response: response '{}' has no property 'correlation-id'","pod":"api/a-5bbf64648d-gwg7n","service":"k8svent","time":"2020-08-01T20:17:11Z"}
{"code":200,"correlation-id":"","host":"k8svent-65d576999b-nqx99","level":"info","msg":"Posted to 'https://webhook.atomist.com/atomist/resource/XYZ'","pod":"api/a-5bbf64648d-gwg7n","service":"k8svent","time":"2020-08-01T20:17:12Z"}
{"host":"k8svent-65d576999b-nqx99","level":"warning","msg":"Failed to extract correlation ID from https://webhook.atomist.com/atomist/resource/XYZ response: response '{}' has no property 'correlation-id'","pod":"api/b-5bbf64648d-mhddx","service":"k8svent","time":"2020-08-01T20:17:11Z"}
{"code":200,"correlation-id":"","host":"k8svent-65d576999b-nqx99","level":"info","msg":"Posted to 'https://webhook.atomist.com/atomist/resource/XYZ'","pod":"api/b-5bbf64648d-mhddx","service":"k8svent","time":"2020-08-01T20:17:12Z"}

You can ignore the warnings about failing to extract a correlation ID. If you do not see log messages showing the "code":200 response, there is likely network issues between the k8svent pod and the Atomist webhook endpoint. Check network policies, routing, and firewall rules to ensure nothing is blocking traffic between the pod and Atomist.

Network policies

If installation was successful and Atomist still reports that it is not receiving events from your cluster, network policies might be an issue. If your Kubernetes cluster has network policies enabled and there are network policies restricting egress in the k8svent namespace, you will have to add a policy or modify an existing policy to allow the k8svent pod egress traffic to webhook.atomist.com.

Support

If you are still having trouble, please do not hesitate to contact us via email at support@atomist.com, in our community Slack at https://join.atomist.com/, or on Twitter @atomist. Happy automating!

Did this answer your question?