Upgrade from 2025.01.31 to 2025.4.26 broke certificate

[2-5]

this request worked in 2025.01.31 but fails in 2025.4.26:
https://api.hbdm.com/api/v1/contract_contract_info

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.hbdm.com', port=443):
Max retries exceeded with url: /api/v1/contract_contract_info
(Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)')))

The request works in Firefox/Chrome.

[alex]

Does the request work in Firefox nightly?

[2-5]

I don't know, I don't use nightly.

[alex]

Ok, I hypothesize that it's broken there.

Looking at https://www.ssllabs.com/ssltest/analyze.html?d=api.hbdm.com&s=104.17.167.187&latest, I believe the problem is that "AAA Certificate Services" has been removed from the Mozilla trust store, and the server does not send the "SSL.com TLS Transit RSA CA R2" intermediate.

[2-5]

I don't know about these things. However this is a major bitcoin exchange. I have pinned my certifi to the previous version until this is solved by the website/certifi.

[alex]

Ok, well, since this appears to be working as intended, there's no action for us to take here.

[sfc-gh-dlisouski]

@alex Can you please also check the following: Cert with Fingerprint 1465fa205397b876faa6f0a9958e5590e40fcc7faa4fb7c2c8677521fb5fb658 has been removed in 2025.4.26 but is still in the Mozilla trust store -- https://ccadb.my.salesforce-sites.com/mozilla/IncludedCACertificateReport

[alex]

Please see: https://bugzilla.mozilla.org/show_bug.cgi?id=1957701

[sfc-gh-dlisouski]

Thank you. Out of curiosity and for my own understanding: am I correct that 2025.4.26 includes changes for anticipated updates to the Mozilla trust store? In other words, the Mozilla trust store itself has not yet been updated, but certifi has already incorporated those changes? I'm curious how certifi and the Mozilla trust store are generally kept in sync. Thank you!

[alex]

We update when the NSS trust store migrates to the Firefox repo, which is to say roughly when changes go into nightly. This is before they're in a final Firefox release.

[phrfpeixoto]

I'm having similar issues with Shopify's GraphQL APIs
This is really NOT great

[alex]

Have you contacted Shopify?

[phrfpeixoto]

I am doing that right now.
EDIT
Though:

Also, downgrading to 2025.1.31 works.

[alex]

It appears that all the impacted websites are behind Cloudflare. I have reached out to folks there.

[ni-todo-spot]

I got here by following snowflake's status page: https://status.snowflake.com/incidents/txclg2cyzq32#:~:text=Customers%20hosted%20in%20AWS%20regions%20may,the%20certifi%20library%20to%20version%202025.4.26.

my project is using snowflake

Do you plan to fix this 🤔 ?

[BI-MarcB]

We are also affected by this issue. We use dbt (data build tool, a Python-based ETL tool) with Snowflake and S3 buckets. Our production pipeline broke because we are now unable to upload local files. This is the error we are encountering, which is related to the communication from Snowflake shared above.

Encountered an error while running operation: Runtime Error
254007: The certificate is revoked or could not be validated: hostname=sfc-eu-ds1-customer-stage.s3.amazonaws.com

[alex]

I'm fairly perplexed, sfc-eu-ds1-customer-stage.s3.amazonaws.com appears to be serving a valid cert chain that should validate.

Can you share what version of OpenSSL your Python is using? python3 -c "import ssl; print(ssl.OPENSSL_VERSION)"

[BI-MarcB]

Our dbt version uses

OpenSSL 3.0.15 3 Sep 2024

which is sourced from "pyOpenSSL" at version 24.3.0 via the following dependency requirement

pyOpenSSL<25.0.0,>=22.0.0

[alex]

Hmm, I was hoping (for certain values of hoping) the answer would be an exceptionally old OpenSSL (1.0.1), because then there'd be a known bug that'd explain this.

Can you reproduce with requests.get("https://sfc-eu-ds1-customer-stage.s3.amazonaws.com")? For me, that correctly returns an HTTP 403 (after verifying the cert), so there's something slightly more at work here than that domain merely not working.

As a meta note: I understand and appreciate folks frustration here, it's never the goal with a new release for things to break. We're working hard to identify impacted usage patterns and find fixes (as I mentioned, I'm in touch with Cloudflare), and I'm appreciative of folks who are continuing to help us debug and identify the causes of issues. That said, there's no obvious step we can take to just "fix this" ourselves. For now, impacted folks may want to temporarily revert -- though I encourage you to make sure you're in touch with service providers running the impacted services.

[2-5]

I do wonder why does certifi releases certificate changes before they get into Firefox stable?

It would seem to me a much better approach would be to do it a few days after Firefox stable, since the stable release would quickly surface many such issues and allow websites a few days to fix them.

Why the hurry?

[alex5207]

I am doing that right now. EDIT Though:

Also, downgrading to 2025.1.31 works.

Exact same experience

[BI-MarcB]

I cannot replicate the issue using requests. It returns a 403.
I tried the following, which lead to - I think - expected results:

import certifi
print(certifi.__version__)

import ssl
import socket
from urllib.parse import urlparse

def print_cert_chain(url, port=443):
    # Parse the URL to get just the hostname
    parsed_url = urlparse(url)
    hostname = parsed_url.netloc
    if not hostname:  # If the input wasn't a URL, assume it's a hostname
        hostname = url
    
    print(f"Checking certificate for: {hostname}")
    
    context = ssl.create_default_context()
    with socket.create_connection((hostname, port)) as sock:
        with context.wrap_socket(sock, server_hostname=hostname) as ssock:
            cert = ssock.getpeercert(binary_form=False)
            print("Certificate details:")
            print(f"Subject: {dict(x[0] for x in cert['subject'])}")
            print(f"Issuer: {dict(x[0] for x in cert['issuer'])}")
            print(f"Version: {cert.get('version')}")
            print(f"Serial Number: {cert.get('serialNumber')}")
            print(f"Not Before: {cert.get('notBefore')}")
            print(f"Not After: {cert.get('notAfter')}")
            if 'subjectAltName' in cert:
                print("Subject Alternative Names:")
                for type, value in cert['subjectAltName']:
                    print(f"  {type}: {value}")

print_cert_chain("sfc-eu-ds1-customer-stage.s3.amazonaws.com")

Output:

2025.04.26
Checking certificate for: sfc-eu-ds1-customer-stage.s3.amazonaws.com
Certificate details:
Subject: {'commonName': '*.s3.amazonaws.com'}
Issuer: {'countryName': 'US', 'organizationName': 'Amazon', 'commonName': 'Amazon RSA 2048 M01'}
Version: 3
Serial Number: 0D4B8BB60137B46B99C9908B9C54B069
Not Before: Feb 14 00:00:00 2025 GMT
Not After: Feb 7 23:59:59 2026 GMT
Subject Alternative Names:
DNS: *.s3.amazonaws.com
DNS: s3.amazonaws.com

Then I tried implementing the actual put command via the snowflake.connector and DEBUG logs.
This is the result, showing the link to the changes to Starfield, but I am unsure how in interpret them. @sfc-gh-dlisouski Are you able to comment on the ocsp json and if something needs to be adjusted there?

DEBUG:snowflake.connector.cursor:query execution done
DEBUG:snowflake.connector.cursor:SUCCESS
DEBUG:snowflake.connector.cursor:PUT OR GET: True
DEBUG:snowflake.connector.file_transfer_agent:UPLOAD
DEBUG:snowflake.connector.file_transfer_agent:command type: UPLOAD
DEBUG:snowflake.connector.file_transfer_agent:no file encoding was detected: file=C:\Users\Marc.Behrens\work\eds\debug../target/manifest.json
DEBUG:snowflake.connector.vendored.urllib3.util.retry:Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
DEBUG:snowflake.connector.vendored.urllib3.util.retry:Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
DEBUG:snowflake.connector.network:Session status for SessionPool 'b'sfc-eu-ds1-customer-stage.s3.amazonaws.com'', SessionPool 1/1 active sessions
DEBUG:snowflake.connector.storage_client:storage client request with session <snowflake.connector.vendored.requests.sessions.Session object at 0x0000019C48A35D30>
DEBUG:snowflake.connector.vendored.urllib3.connectionpool:Starting new HTTPS connection (1): sfc-eu-ds1-customer-stage.s3.amazonaws.com:443
DEBUG:snowflake.connector.ssl_wrap_socket:OCSP Mode: FAIL_OPEN, OCSP response cache file name: None
DEBUG:snowflake.connector.ocsp_snowflake:ocsp_response_cache_uri: file://C:/Users/Marc.Behrens/AppData/Local/Snowflake/Caches/ocsp_response_cache.json
DEBUG:snowflake.connector.ocsp_snowflake:OCSP_VALIDATION_CACHE size: 396
DEBUG:snowflake.connector.ocsp_snowflake:OCSP response cache server is enabled: http://ocsp.snowflakecomputing.com/ocsp_response_cache.json
DEBUG:snowflake.connector.ocsp_snowflake:OCSP dynamic cache server RETRY URL: None
DEBUG:snowflake.connector.ocsp_snowflake:validating certificate: sfc-eu-ds1-customer-stage.s3.amazonaws.com
DEBUG:snowflake.connector.ocsp_asn1crypto:# of certificates: 4
DEBUG:snowflake.connector.ocsp_asn1crypto:subject: OrderedDict({'common_name': '*.s3.amazonaws.com'}), issuer: OrderedDict({'country_name': 'US', 'organization_name': 'Amazon', 'common_name': 'Amazon RSA 2048 M01'})
DEBUG:snowflake.connector.ocsp_asn1crypto:subject: OrderedDict({'country_name': 'US', 'organization_name': 'Amazon', 'common_name': 'Amazon RSA 2048 M01'}), issuer: OrderedDict({'country_name': 'US', 'organization_name': 'Amazon', 'common_name': 'Amazon Root CA 1'})
DEBUG:snowflake.connector.ocsp_asn1crypto:subject: OrderedDict({'country_name': 'US', 'organization_name': 'Amazon', 'common_name': 'Amazon Root CA 1'}), issuer: OrderedDict({'country_name': 'US', 'state_or_province_name': 'Arizona', 'locality_name': 'Scottsdale', 'organization_name': 'Starfield Technologies, Inc.', 'common_name': 'Starfield Services Root Certificate Authority - G2'})
DEBUG:snowflake.connector.ocsp_asn1crypto:subject: OrderedDict({'country_name': 'US', 'state_or_province_name': 'Arizona', 'locality_name': 'Scottsdale', 'organization_name': 'Starfield Technologies, Inc.', 'common_name': 'Starfield Services Root Certificate Authority - G2'}), issuer: OrderedDict({'country_name': 'US', 'organization_name': 'Starfield Technologies, Inc.', 'organizational_unit_name': 'Starfield Class 2 Certification Authority'})
DEBUG:snowflake.connector.ocsp_asn1crypto:reading certificate bundle: C:\Users\Marc.Behrens.virtualenvs\eds-XlS1n9Wm\Lib\site-packages\certifi\cacert.pem
DEBUG:snowflake.connector.ocsp_asn1crypto:not found issuer_der: OrderedDict({'country_name': 'US', 'organization_name': 'Starfield Technologies, Inc.', 'organizational_unit_name': 'Starfield Class 2 Certification Authority'})
DEBUG:snowflake.connector.ocsp_snowflake:{'driver': 'PythonConnector', 'version': '3.13.2', 'eventType': 'RevocationCheckFailure', 'eventSubType': 'CertificateExtractionFailed', 'sfcPeerHost': 'sfc-eu-ds1-customer-stage.s3.amazonaws.com', 'certId': None, 'ocspRequestBase64': None, 'ocspResponderURL': None, 'errorMessage': None, 'insecureMode': False, 'failOpen': True, 'cacheEnabled': True, 'cacheHit': False}
DEBUG:snowflake.connector.network:Session status for SessionPool 'b'sfc-eu-ds1-customer-stage.s3.amazonaws.com'', SessionPool 0/1 active sessions
Connection error occurred:
Error type: OperationalError
Error message: 254007: 254007: The certificate is revoked or could not be validated: hostname=sfc-eu-ds1-customer-stage.s3.amazonaws.com
Error code: 254007
Detailed message: 254007: 254007: The certificate is revoked or could not be validated: hostname=sfc-eu-ds1-customer-stage.s3.amazonaws.com
DEBUG:snowflake.connector.storage_client:cleaning up tmp dir: C:\Users\MARC~1.BEH\AppData\Local\Temp\tmp9j9_q797

[phrfpeixoto]

I'm fairly perplexed, sfc-eu-ds1-customer-stage.s3.amazonaws.com appears to be serving a valid cert chain that should validate.

Can you share what version of OpenSSL your Python is using? python3 -c "import ssl; print(ssl.OPENSSL_VERSION)"

I'm using OpenSSL 1.1.1w 11 Sep 2023 with pyOpenSSL==24.2.1

Your request test produces:

Python 3.12.10 (main, Apr  9 2025, 00:30:32) [GCC 10.2.1 20210110]
Type 'copyright', 'credits' or 'license' for more information
IPython 9.2.0 -- An enhanced Interactive Python. Type '?' for help.
Tip: Use the IPython.lib.demo.Demo class to load any Python script as an interactive demo.

In [1]: import requests

In [2]: response = requests.get("https://sfc-eu-ds1-customer-stage.s3.amazonaws.com")

In [3]: response.raise_for_status()
---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
Cell In[3], line 1
----> 1 response.raise_for_status()

File /usr/local/lib/python3.12/site-packages/requests/models.py:960, in Response.raise_for_status(self)
    957     http_error_msg = u'%s Server Error: %s for url: %s' % (self.status_code, reason, self.url)
    959 if http_error_msg:
--> 960     raise HTTPError(http_error_msg, response=self)

HTTPError: 403 Client Error: Forbidden for url: https://sfc-eu-ds1-customer-stage.s3.amazonaws.com/

In [4]: response.text
Out[4]: '<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>ZQ05XYVPYMK6DWBH</RequestId><HostId>Itv+oNXPEeEFTQeB+Zi2ijYNr0ikRwW1ymE6LBGkYL7GRsWcY9GOuH1K4ugxCwjVzv6n+mKMdljkynlnuipSLV+WZfvlw9QV</HostId></Error>'

In [5]: response.headers
Out[5]: {'x-amz-bucket-region': 'eu-central-1', 'x-amz-request-id': 'ZQ05XYVPYMK6DWBH', 'x-amz-id-2': 'Itv+oNXPEeEFTQeB+Zi2ijYNr0ikRwW1ymE6LBGkYL7GRsWcY9GOuH1K4ugxCwjVzv6n+mKMdljkynlnuipSLV+WZfvlw9QV', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Date': 'Mon, 28 Apr 2025 14:26:51 GMT', 'Server': 'AmazonS3'}

[csvec-chwy]

So came here from Snowflake as well, and I'd focus more on that than the s3 issue right off the bat (as that seems to be related to how snowflake integrates with S3, and the cert itself is fine, but OCSP may be failing?). The issue isn't the cert isn't trusted exactly, it's that OCSP fails. It seems related to the removal of starfield class 2, which AWS did last year in theory: https://aws.amazon.com/blogs/security/acm-will-no-longer-cross-sign-certificates-with-starfield-class-2-starting-august-2024/ but the snowflake URLs still reference them, and I'm not sure how the OCSP logic in their libraries works. But if we disable OCSP in their connection libs, this is happy again, as the cert trust is entirely fine.

This is likely different from the top level error having to do with HDBM failing cert verify entirely. and probably wants to be a different issue, even if both are related to the trust store updates in some way.

i:C = US, O = Amazon, CN = Amazon RSA 2048 M03
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Jan 16 00:00:00 2025 GMT; NotAfter: Feb 14 23:59:59 2026 GMT
1 s:C = US, O = Amazon, CN = Amazon RSA 2048 M03
i:C = US, O = Amazon, CN = Amazon Root CA 1
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Aug 23 22:26:04 2022 GMT; NotAfter: Aug 23 22:26:04 2030 GMT
2 s:C = US, O = Amazon, CN = Amazon Root CA 1
i:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: May 25 12:00:00 2015 GMT; NotAfter: Dec 31 01:00:00 2037 GMT
3 s:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2
i:C = US, O = "Starfield Technologies, Inc.", OU = Starfield Class 2 Certification Authority
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Sep 2 00:00:00 2009 GMT; NotAfter: Jun 28 17:39:16 2034 GMT

For sanity and completion as an edit in case the maintainers see this: I'm more and more convinced the OCSP issue with snowflake is only related to your update, but not your fault. These URLs all seem to pass OCSP checking manually. Only the snowflake connectors are claiming otherwise from testing.

[alex]

Ok, the knowledge that what's going on here is that Snowflake does OCSP and it's verifying the OCSP cert that is failing is a new wrinkle!

Does anyone by chance know where in the snowflake code the OCSP request machinery is?

[edgarrmondragon]

Ok, the knowledge that what's going on here is that Snowflake does OCSP and it's verifying the OCSP cert that is failing is a new wrinkle!

Does anyone by chance know where in the snowflake code the OCSP request machinery is?

I wonder if they're in the process of fixing it: snowflakedb/snowflake-connector-python@2431336

[alex]

I don't particularly have enough context to evaluate that code, but it does suggest to me that perhaps the issue here is the intersection of the change in the Mozilla trust store _combined_ with a bug in the snowflake client.

…

On Mon, Apr 28, 2025 at 6:00 PM Edgar Ramírez Mondragón < ***@***.***> wrote: *edgarrmondragon* left a comment (certifi/python-certifi#349) <#349 (comment)> Ok, the knowledge that what's going on here is that Snowflake does OCSP and it's verifying the OCSP cert that is failing is a new wrinkle! Does anyone by chance know where in the snowflake code the OCSP request machinery is? I wonder if they're in the process of fixing it: ***@***.*** <snowflakedb/snowflake-connector-python@2431336> — Reply to this email directly, view it on GitHub <#349 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAGBG6E6D6EZX3NWFTBJT232QI5AVCNFSM6AAAAAB35WUKI6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMZWHAZDQNBQGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- All that is necessary for evil to succeed is for good people to do nothing.

[marc-barry]

We hit this issue today. I think dbt-labs/dbt-adapters#1027 is an attempt to address the issue with dbt.

This is our error:

ERROR:snowflake.connector.result_batch:Failed to fetch the large result set batch data_0_2_1_2 for the 9 th time, backing off for 14s for the reason: '254007: The certificate is revoked or could not be validated: hostname=sfc-oh-ds1-15-customer-stage.s3.us-east-2.amazonaws.com'
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.12.10/x64/lib/python3.12/site-packages/snowflake/connector/result_batch.py", line 331, in _download
    response = session.request("get", **request_data)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.10/x64/lib/python3.12/site-packages/snowflake/connector/vendored/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.10/x64/lib/python3.12/site-packages/snowflake/connector/vendored/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.10/x64/lib/python3.12/site-packages/snowflake/connector/vendored/requests/adapters.py", line 485, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.10/x64/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.10/x64/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/connectionpool.py", line 404, in _make_request
    self._validate_conn(conn)
  File "/opt/hostedtoolcache/Python/3.12.10/x64/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/connectionpool.py", line 1058, in _validate_conn
    conn.connect()
  File "/opt/hostedtoolcache/Python/3.12.10/x64/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/connection.py", line 419, in connect
    self.sock = ssl_wrap_socket(
                ^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.10/x64/lib/python3.12/site-packages/snowflake/connector/ssl_wrap_socket.py", line 89, in ssl_wrap_socket_with_ocsp
    raise OperationalError(
snowflake.connector.errors.OperationalError: 254007: The certificate is revoked or could not be validated: hostname=sfc-oh-ds1-15-customer-stage.s3.us-east-2.amazonaws.com

It points to the OCSP stuff being in ssl_wrap_socket.py: https://github.com/snowflakedb/snowflake-connector-python/blob/main/src/snowflake/connector/ssl_wrap_socket.py#L89

snowflakedb/snowflake-connector-python#2299 looks to be the fix.

[sfc-gh-sviswanathan]

Hey all, Snowflake Engineering team here. We acknowledge that the impact to Snowflake-to-S3 use cases here stems from a bug in our implementation of chain validation in the Snowflake Python connector. The code ought to validate against the Amazon Root CA trust anchor instead of checking for the validity of all the certificates in the certificate chain. We are fixing this in the Python Connector and are on track to release a patch by Apr 30.

[alex]

Thanks for sharing the update, and thanks for your work here. Sorry to be the forcing function for this work!

…

On Mon, Apr 28, 2025, 8:04 PM Srikanth Viswanathan ***@***.***> wrote: *sfc-gh-sviswanathan* left a comment (certifi/python-certifi#349) <#349 (comment)> Hey all, Snowflake Engineering team here. We acknowledge that the impact to Snowflake-to-S3 use cases here stems from a bug in our implementation of chain validation in the Snowflake Python connector. The code ought to validate against the Amazon Root CA trust anchor instead of checking for the validity of all the certificates in the certificate chain. We are fixing <snowflakedb/snowflake-connector-python@2431336> this in the Python Connector and are on track to release a patch by Apr 30. — Reply to this email directly, view it on GitHub <#349 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAGBHPKOYXQ4KVHXDKSKL2326ZNAVCNFSM6AAAAAB35WUKI6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMZXGA2TKMBYG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[phrfpeixoto]

I'm sorry, but my issue is not with snowflake. I'm using httpx to talk to Shopify.

I reached out to them, and they are interest to hear what Cloudfront had to say about this.

Please advise.

[leocxy]

I have the same issue. When I try to get the access token from Shopify (Custom app).
Once I downgrade the package from 2025.4.26 to 2025.1.31 everything works again.

[blazyng]

I have the same issues using requests in GCF (Google-Cloud-Functions) + ShopifyGraphQL API.
Using certifi==2025.1.31 in my requirements.txt and everything works again.

[borutmrak]

@2-5 hbdm.com changed their certificate so the issue is fixed (for this specific address :)

@alex It's true that they are behind CF, but they are either using their own certificate(s) or there is an option to reissue via CF with different roots, because the new chains/roots are totally different. I'm just guessing (not a CF customer here), but when I got to the root of the problem it got fixed in a few hours after reporting to them.

[nraffuse]

I'm also experiencing this using HTTPX against Shopify. Instead of version pinning I opted to use the system's root bundle.

using pytest

import ssl
import httpx

def test_will_fail():
    r = httpx.get("https://EXAMPLE.myshopify.com/admin/api/2025-04/graphql.json")
    assert r.status_code == 401  # unauthorized

def test_will_pass():
    ctx = ssl.create_default_context()
    ctx.load_verify_locations("/etc/ssl/certs/ca-certificates.crt")  # Debian
    # ctx.load_verify_locations("/etc/ssl/certs/ca-bundle.crt")  # Rocky
    client = httpx.Client(verify=ctx)
    r = client.get("https://EXAMPLE.myshopify.com/admin/api/2025-04/graphql.json")
    assert r.status_code == 401  # unauthorized

[borutmrak]

I'm also experiencing this using HTTPX against Shopify. Instead of version pinning I opted to use the system's root bundle.

That's OK - for now.

Sooner or later the root certs will also be removed from the system roots - this one is on the cert owners to fix, in the meantime we need workarounds. Either an older version of certifi or using some other roots to verify the validity. Or not verifying, but that's not something that should be done. Might work in a pinch though, especially if the data retrieved is not really private.

[sminnee]

For those interested

• certifi gets its cert from https://mkcert.org/
• which gets it from Firefox trunk at https://hg-edge.mozilla.org/mozilla-central/raw-file/tip/security/nss/lib/ckfw/builtins/certdata.txt
• which gets it from the Mozilla NSS project https://nss-crypto.org/ source at https://hg-edge.mozilla.org/projects/nss/
• The change related to these bugs
  • https://bugzilla.mozilla.org/show_bug.cgi?id=1957685
  • https://bugzilla.mozilla.org/show_bug.cgi?id=1937338

The same file on the released version of Firefox is available here:

https://hg-edge.mozilla.org/releases/mozilla-release/raw-file/tip/security/nss/lib/ckfw/builtins/certdata.txt

I've confirmed that Firefox 139 beta has this issue in the end-user app, and Firefox 138 (the current stable) does not

I wonder if it's worth amending this repo (or mkcert.org) to use https://hg-edge.mozilla.org/releases/mozilla-release/raw-file/tip/security/nss/lib/ckfw/builtins/certdata.txt as the source of the cert file rather than Firefox trunk?

[ruck94301]

@alex I'm coming across this discussion 3 weeks after the fact. I observed this trouble on 4/26, and circled back on 5/16 to reproduce and debug. But I couldn't reproduce and now I see why... this was a temporary fouling, outside of certifi's purview.
Thanks for the quick attention, investigation, & information on 4/26.
FWIW, the target URL for the sw that I was using was not using snowflake or cloudflare or aws, AFAIK: (oops, wrong, see [1])
https://dl.fbaipublicfiles.com/detectron2/COCO-InstanceSegmentation/...

[1] Oh wait, google ai says: "The domain dl.fbaipublicfiles.com is owned by Meta (formerly Facebook) and appears to be hosted on Amazon S3". And "Amazon S3 (Simple Storage Service) is a specific cloud storage service offered by Amazon Web Services (AWS)".
Okay, so, I just have another example of certifi cert-trouble when hitting AWS, on 4/26 only.

@sminnee I love that you're looking for an improvement to avoid the reoccurrence of this fouling. Did you get a sense that some process improvement will take place? Either what you recommended, or, something else?

[alex]

While we're still working with various service providers here, I wanted to share that curl's ca-certificates package released today, also removing these certificates. As a result, I expect pressure to accelerate for service providers to take action.

[sminnee]

@alex what do you think about this to avoid users of this package being on the bleeding edge in the future?

I wonder if it's worth amending this repo (or mkcert.org) to use https://hg-edge.mozilla.org/releases/mozilla-release/raw-file/tip/security/nss/lib/ckfw/builtins/certdata.txt as the source of the cert file rather than Firefox trunk?

[alex]

I don't have a strong view one way or the other. Perhaps using Mozilla's beta train is a sensible compromise. (In any event, we need to update for Mozilla's migration to git)

[sminnee]

Fair enough, yeah, that would be the this URL:

https://hg-edge.mozilla.org/releases/mozilla-beta/raw-file/tip/security/nss/lib/ckfw/builtins/certdata.txt

Or on github:

https://raw.githubusercontent.com/mozilla-firefox/firefox/refs/heads/beta/security/nss/lib/ckfw/builtins/certdata.txt

I could raise a PR that fetches from the github URL if you want?

[alex]

Sure, that sounds good.

…

On Mon, May 26, 2025 at 12:35 AM Sam Minnée ***@***.***> wrote: *sminnee* left a comment (certifi/python-certifi#349) <#349 (comment)> Fair enough, yeah, that would be the this URL: https://hg-edge.mozilla.org/releases/mozilla-beta/raw-file/tip/security/nss/lib/ckfw/builtins/certdata.txt Or on github: https://raw.githubusercontent.com/mozilla-firefox/firefox/refs/heads/beta/security/nss/lib/ckfw/builtins/certdata.txt I could raise a PR that fetches from the github URL if you want? — Reply to this email directly, view it on GitHub <#349 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAGBGEG7JZKWIFW3OJYXD3AKK27AVCNFSM6AAAAAB35WUKI6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSMBYGQ4TIOJZGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- All that is necessary for evil to succeed is for good people to do nothing.

[sminnee]

I'll raise an issue on mkcert first and see if they're interested in changing it upstream. Otherwise the change will be more involved

Update: it looks like Lukasa is happy to make the change there, so no need to change this repo. We're set it to beta initially.