Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker-compose fails with ERROR: SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661) #38

Closed
colindembovsky opened this issue Mar 3, 2017 · 23 comments

Comments

@colindembovsky
Copy link

(Logging here as per @RoopeshNair on microsoft/VSTS-Docker-Preview#14)

I have created a release with a several docker tasks. I have an endpoint to a docker host machine. I am able to run docker commands successfully (like docker run) against the docker host. However, the docker-compose up commands fail with ERROR: SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661). (In the task I'm using the "Run Services" action).

I can repro this on my client machine by running docker-compose ps against the same host - I get the cert error. However, if I specify the tls settings explicitly like this:

docker-compose -H $dockerHost --tls --tlscacert ~/.docker/ca.pem --tlscert ~/.docker/cert.pem --tlskey ~/.docker/key.pem ps

then the command succeeds. Is there some issue with the environment variables that the task is using to run docker-compose up?

@logankoester
Copy link

I'm seeing the same thing.

@jitekuma
Copy link

jitekuma commented Mar 6, 2017

It seems that the certificates used in Docker Container Host Endpoint and the one saved in ~/.docker/ folder are different.

The certificate stored in Docker Container Host Endpoint is saved in ~/.dockercerts/ by task and used for execution. Can you verify if these certificates are same.

~/.dockercerts/ gets deleted once the task execution completes. You can take the certificate while task is executing to validate.

@colindembovsky
Copy link
Author

Hi @jitekuma - I only have one set of certs. And I can't see how they can be different because docker commands work using the endpoint. It's just the docker-compose command that fails. I suspect that the openssl on my Windows Bash and in the VSTS agent container is old - I'm going to try to update it to see if that resolves the issue.

@SemionPar
Copy link

I am having the same issue.

DOCKER_TLS_VERIFY = "1"
DOCKER_HOST = "tcp://some.host:2376"
DOCKER_CERT_PATH = "/home/xyz/.docker"

docker --tlsverify ps executes just fine, while docker-compose --tlsverify up -d --force-recreate gives me an error:
SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)

@RoopeshNair
Copy link
Member

RoopeshNair commented Mar 7, 2017 via email

@colindembovsky
Copy link
Author

I've tried to update python and install docker-compose via pip (since some other threads seem to resolve this issue that way) but that doesn't seem to work for me.

@jitekuma
Copy link

jitekuma commented Mar 8, 2017

@colindembovsky You dont need to pass --tls or --tlsverify option in the docker-config path as the task already sets DOCKER_TSL_VERIFY environment varaible. I debugged docker-compose and docker-py library and verified that if you pass any flag --tls or --tlsverify flag it tries to create tslConfig object out of options and not from environment and hence either ca_cert object or verify is none in TFSConfig file. You can use the task without using these flags.
Check the variable getting set here https://github.com/Microsoft/vsts-docker/blob/cdccda9323abe8e861d0891f03922d0e3f8fd8d1/src/tasks/dockerCompose/dockerComposeConnection.ts
Let me know if you need more details.

@colindembovsky
Copy link
Author

colindembovsky commented Mar 8, 2017

@jitekuma I'm not setting the env variables during the build/release - I just noted that I can repro the task behavior if I do so locally.

I'm still not sure if this is an issue with the docker task or the Azure cli that created the certificates in the first place. Bottom line: if I pass the certs in I get a successful connection - when I use env variables, I get failures. Here's what happens when I run some docker commands from my machine:

Docker commands succeed when passing --tls:

colin@colinsurface31 ~> set vmName "cd-dockerhost"
colin@colinsurface31 ~> set location "westus"
colin@colinsurface31 ~> set dockerHost "tcp://$vmName.$location.cloudapp.azure.com:2376"
colin@colinsurface31 ~> docker -H $dockerHost --tls ps
CONTAINER ID        IMAGE                                             COMMAND             CREATED             STATUS              PORTS               NAMES
90497ba28b92        microsoft/vsts-agent:ubuntu-16.04-docker-1.11.2   "./start.sh"        4 days ago          Up 4 days                               jovial_nightingale

Docker commands fail when using env vars:

colin@colinsurface31 ~> set -x DOCKER_HOST $dockerHost
colin@colinsurface31 ~> set -x DOCKER_TLS_VERIFY 1
colin@colinsurface31 ~> docker ps
FATA[0000] An error occurred trying to connect: Get https://cd-dockerhost.westus.cloudapp.azure.com:2376/v1.18/containers/json: x509: certificate is valid for *, not cd-dockerhost.westus.cloudapp.azure.com

Docker-compose succeeds when passing in certs:

colin@colinsurface31 ~> docker-compose -H $dockerHost --tls --tlscacert ~/.docker/ca.pem --tlscert ~/.docker/cert.pem --tlskey ~/.docker/key.pem ps
/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
Name   Command   State   Ports
------------------------------

Docker-compose fails when using env vars:

colin@colinsurface31 ~> set -x DOCKER_TLS_VERIFY 1
colin@colinsurface31 ~> set -x DOCKER_HOST $dockerHost
colin@colinsurface31 ~> docker-compose ps
ERROR: SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)

@jitekuma
Copy link

jitekuma commented Mar 8, 2017

@colindembovsky You also need to set DOCKER_CERT_PATH environment variable.

DOCKER_TLS_VERIFY = "1"
DOCKER_HOST = $dockerHost
DOCKER_CERT_PATH = "~/.docker/"

then simply do docker-compose ps this will work

or specify all the tls parameter like

docker-compose -H $dockerHost --tls --tlscacert ~/.docker/ca.pem --tlscert ~/.docker/cert.pem --tlskey ~/.docker/key.pem ps

@colindembovsky
Copy link
Author

@jitekuma - the default is correct in my case (~/.docker). But even if I set that I get the same behavior:

colin@colinsurface31 ~> set vmName "cd-dockerhost"
colin@colinsurface31 ~> set location "westus"
colin@colinsurface31 ~> set dockerHost "tcp://$vmName.$location.cloudapp.azure.com:2376"
colin@colinsurface31 ~> set -x DOCKER_HOST $dockerHost
colin@colinsurface31 ~> set -x DOCKER_TLS_VERIFY 1
colin@colinsurface31 ~> set -x DOCKER_CERT_PATH ~/.docker
colin@colinsurface31 ~> docker ps
FATA[0000] An error occurred trying to connect: Get https://cd-dockerhost.westus.cloudapp.azure.com:2376/v1.18/containers/json: x509: certificate is valid for *, not cd-dockerhost.westus.cloudapp.azure.com
colin@colinsurface31 ~> docker-compose ps
ERROR: SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)

I think my point is that I think the task should be passing the varialbes into the command rather than relying on the env vars - especially given that passing the certs in works reliably while using the env settings appears to be flaky for some reason.

@jitekuma
Copy link

jitekuma commented Mar 8, 2017

@colindembovsky now i think this is a certificate issue can you put --tlsverify option in docker-compose

docker-compose -H $dockerHost --tls --tlsverify --tlscacert ~/.docker/ca.pem --tlscert ~/.docker/cert.pem --tlskey ~/.docker/key.pem ps

I think this should also fail. Let me know the output.

@colindembovsky
Copy link
Author

@jitekuma Yes! It failed indeed (same as the error with docker):

colin@colinsurface31 ~> docker-compose -H $dockerHost --tls --tlscacert ~/.docker/ca.pem --tlscert ~/.docker/cert.pem --tlskey ~/.docker/key.pem --tlsverify ps
ERROR: SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)

The thing is that I didn't create the certs by hand - I used azure vm docker create which created the certs. Is there an issue with that process??

@jitekuma
Copy link

jitekuma commented Mar 8, 2017

No. I dont think so. Might be due to some command ran during development messed up with you certificate.

@colindembovsky
Copy link
Author

Hmm I didn't do anything after creating the dockerHost. I think this might be the issue:

An error occurred trying to connect: Get https://cd-dockerhost.westus.cloudapp.azure.com:2376/v1.18/containers/json: x509: certificate is valid for *, not cd-dockerhost.westus.cloudapp.azure.com

I'm going to delete this host, clean the certs folder and run the azure vm docker create command again to make sure everything is squeaky clean. Then I'll see if I get the same behavior.

@jitekuma
Copy link

jitekuma commented Mar 8, 2017

@SemionPar in docker-compose command you can not mix environment variable and command option. You can specify setting in env variable and then just use docker-compose ps. The connection will be secured with TLS protocol if DOCKER_TLS_VERIFY variable is set.

@jitekuma
Copy link

jitekuma commented Mar 8, 2017

@colindembovsky also please check the path where certificates are created in the log, usually its different than ~/docker .

@colindembovsky
Copy link
Author

I figured out what the problem was - I used azure vm docker create to create the dockerhost. By default the CN for the generated certificates is "*". I just set it to hostname.location.cloudapp.azure.com using --docker-cert-cn "cd-dockerhost.westus.cloudapp.azure.com" when calling create and now I can connect without cert errors.

@ashish2rathi
Copy link

I have installed the docker toolbox and I am facing the same problem when try to build an image. It shows me the following error message:

"ERROR: SSL error: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:661)"

@amandoabreu
Copy link

Same here: @ashish2rathi are you on windows?

@ashish2rathi
Copy link

@amandoabreu - yes, I am using windows 7

@ASnow
Copy link

ASnow commented Apr 1, 2019

vmware/vic-product#955

@kumarvt
Copy link

kumarvt commented Jul 17, 2019

I was getting this on windows ERROR: SSL error: HTTPSConnectionPool(host='192.168.99.101', port=2376): Max retries exceeded with url: /v1.30/info (Caused by SSLError(SSLError(1, u'[SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:727)'),))

export COMPOSE_TLS_VERSION=TLSv1_2 fixed the issue

@KMDSOUF
Copy link

KMDSOUF commented Apr 27, 2021

this one did it for me docker-machine regenerate-certs --client-certs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

10 participants