Secure TLS to Thanos sidecar

[Kellen275]

Hi team! I'm deploying the Thanos sidecar via the kube-prometheus-stack (chart version 61.8.0, which uses the image thanos/thanos:v0.36.0 for the sidecar).

My goal is to enable secure TLS on the sidecar such that eventually I have an external Thanos query service securely query the sidecar. However, I haven't introduced Thanos query yet to keep the problem simple. For now I'm just running grpcurl ... grpc.health.v1.Health.Check to test connections.

I've been able to successfully demonstrate non-TLS communications with the following kube-prometheus-stack values

prometheus:
  thanosService:
    enabled: true

With this I can run the following from an arbitrary pod with grpcurl

$ grpcurl -plaintext kube-prometheus-stack-thanos-discovery.prometheus-system.svc.cluster.local:10901 grpc.health.v1.Health.Check
{
  "status": "SERVING"
}

I've attempted to enable TLS via the following chart values

prometheus:
  prometheusSpec:
    thanos:
      grpcServerTlsConfig:
        caFile: "/etc/ssl/thanos-sidecar-tls/ca-certificates.crt"
        certFile: "/etc/ssl/thanos-sidecar-tls/server.crt"
        keyFile: "/etc/ssl/thanos-sidecar-tls/server.key"
      volumeMounts:
        - name: tls-certs
          mountPath: "/etc/ssl/thanos-sidecar-tls"

However, if I then run grpcurl with certs, I receive the following

$ grpcurl -cacert ca-certificates.crt -cert server.crt -key server.key kube-prometheus-stack-thanos-discovery.prometheus-system.svc.cluster.local:10901 grpc.health.v1.Health.Check
Failed to dial target host "kube-prometheus-stack-thanos-discovery.prometheus-system.svc.cluster.local:10901": EOF

I can at least confirm the additional chart values have changed behavior of the sidecar, as my previous plaintext command no longer works

$ grpcurl -plaintext kube-prometheus-stack-thanos-discovery.prometheus-system.svc.cluster.local:10901 grpc.health.v1.Health.Check
Error invoking method "grpc.health.v1.Health.Check": rpc error: code = Unavailable desc = failed to query for service descriptor "grpc.health.v1.Health": upstream connect error or disconnect/reset before headers. reset reason: connection termination

Any tips would be greatly appreciated!

[dosubot[bot]]

Hey @Kellen275! I'm here to help you with your Thanos sidecar TLS issue. Let's figure this out together!

To enable secure TLS on the Thanos sidecar and resolve the EOF error with grpcurl, you can follow these steps:

1. Ensure Correct TLS Configuration: Double-check that the paths to your certificate files (caFile, certFile, keyFile) are correct and accessible by the Thanos sidecar.

2. Use Per-Endpoint mTLS Configuration: Utilize the --endpoints.config option in Thanos Querier to configure mTLS for specific endpoints. This allows you to manage TLS settings without needing additional proxies like Envoy.

3. Example Configuration:
   Create a YAML configuration file for --endpoints.config specifying the TLS settings for each endpoint. Include paths to certificate files and server names.

   - tls_config:
       cert_file: "/etc/ssl/thanos-sidecar-tls/server.crt"
       key_file: "/etc/ssl/thanos-sidecar-tls/server.key"
       ca_file: "/etc/ssl/thanos-sidecar-tls/ca-certificates.crt"
       server_name: "kube-prometheus-stack-thanos-discovery.prometheus-system.svc.cluster.local"
     endpoints:
       - "kube-prometheus-stack-thanos-discovery.prometheus-system.svc.cluster.local:10901"

4. Automatic Certificate Rotation: Use Kubernetes cert-manager to handle automatic certificate rotation. Configure Thanos to redo the TLS handshake periodically using the --grpc-server-max-connection-age option.

5. Verify Certificates: Ensure that the certificates are valid and correctly signed for the server name you are using.

By following these steps, you should be able to establish secure TLS communication for Thanos sidecar queries, resolving the EOF error with grpcurl ^[1][2].

To continue talking to Dosu, mention @dosu.