Kubernetes cert-manager and letsencrypt doesn't generate certificates anymore http to https redirect

You've probably set up a Kubernetes cluster, which contains a load balancer such as Traefik or Nginx. You've easily set up cert-manager to generate your SSL TLS certificates via LetsEncrypt.

Everything worked fine until one day you realize your certificates aren't updated and generated anymore! Now, something is wrong but you don't really but you don't really know what.

• Has your Kubernetes cluster been compromised?
• Have you been banned by Letsencrypt? 
• Has cert-manager failed?
• Is your loadbalancer not working anymore and are you getting 404 errors?

Are you're looking for a guide on how to setup wildcard subdomains that get their own Letsencrypt TLS SSL certificates? If yes leave a comment.

Setting up cert-manager with ingress-shim for normal certificates breaks when traefik 1.7  "forcefully" redirects http to https on a global basis. It seemed nginx had a similar issue in the past.
I've tested various configurations and spent 2 days on this issue trying to figure out a compromise and which settings would work. I'll spare you all the configuration I did so I'll resume some interesting things I found out during experimentation.

I've eventually came to the realization that I was not alone in this struggle and that it had been detected by others before.

The Logs

Traefik did give 404 errors. By looking in the logs I found out that that the secrets containing the certificates had the "tls.crt" empty ("") meaning that it failed to put a certificate and thus didn't render the service correctly.

Standard Traefik settings http to https redirects use the status code 302 Found. This doesn't tell automated systems anything at all and they'll probably ignore the Location: header.
We'd need either 301 or 308 redirect to properly indicate redirects. Letsencrypt indicated that their ACME bots normally follow 301.
Luckily, traefik has a setting just for that.
If you're using Helm Charts you can set ssl.permanentRedirect = true. Or you can set permanent = true under the redirect strategy in the traefik toml file.

The status code is now 302. However, that didn't seem to generate certificates either. Something else was wrong. Even if the service is up and running it seemed to give 404 errors. After triple checking the individual pods and containers, everything was working. When I disabled TLS in the ingress that pesky 404 disappeared.

This seems like a catch 22 kind of issue.

After +/-2 days of pushing on this really hard I decided to just disable the global http to https redirect. I figured out it's just not worth any more time and I'll do a redirect at the application level.
I've tried playing with the specific annotations to redirect requests like before, they simply didn't seem to work:

traefik.ingress.kubernetes.io/redirect-entry-point: httpstraefik.ingress.kubernetes.io/redirect-permanent: "true"

After looking in the Traefik Kubernetes Ingress annotations I said to test 1 more thing.
The solution is to add the following annotation which redirects http to https and returns a 301 status.

This does the same thing as the previous 2 annotations combined yet this time it just worked!

A fully working example of an ingress with the proper annotations:

apiVersion: extensions/v1beta1kind: Ingressmetadata:  name: testing-site  namespace: sparta  annotations:    kubernetes.io/ingress.class: "traefik"    ingress.kubernetes.io/ssl-redirect: "true"    cert-manager.io/cluster-issuer: "letsencrypt-production"spec:  rules:  - host: testing.example.com    http:      paths:      - backend:          serviceName: spartan-site          servicePort: 8100  tls:  - hosts:    - testing.example.com    secretName: testing-example-prod

 

This should work if you use a combination of the following:

• Traefik
• k8s or k3s cluster
• cert-manager

Cleanup of non-working certificates and empty secrets

After creating tens of certificates and secrets they just filled up my traefik logs so i've fiddled with kubectl and jsonpath to delete them.
You might want to tweak the following commands for your own cluster/namespaces.

kubectl -n sparta delete cert $(kubectl -n sparta get cert -o=jsonpath='{.items[?(@.status.conditions[0].status=="False")].metadata.name}')kubectl -n sparta delete secret $(kubectl -n sparta get secret -o=jsonpath='{.items[?(@.data.tls\\.crt}=="")].metadata.name}')

Conclusion

Such a small setting can have such dire consequences!

Your Kubernetes cluster can now make use of cert-manager and Letsencrypt  to generate TLS certificates and you can still have http to https redirects by using the following annotation:

ingress.kubernetes.io/ssl-redirect: "true"

Good luck!