Skip to content

Commit d31cebd

Browse files
chore: make cleanup_cloud_functions test fixture more robust to flakiness (#720)
* chore: make `cleanup_cloud_functions` test fixture more robust to flakiness In response to a recent test failure in load test session just because this optional cleanup code failed. ``` ____________ ERROR at setup of test_read_gbq_sql_large_results[1gb] ____________ [gw1] linux -- Python 3.12.0 /tmpfs/src/github/python-bigquery-dataframes/.nox/load/bin/python ... tests/system/conftest.py:1119: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ tests/system/utils.py:349: in delete_cloud_function operation = functions_client.delete_function(request=request) .nox/load/lib/python3.12/site-packages/google/cloud/functions_v2/services/function_service/client.py:1499: in delete_function response = rpc( .nox/load/lib/python3.12/site-packages/google/api_core/gapic_v1/method.py:131: in __call__ return wrapped_func(*args, **kwargs) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = (name: "projects/bigframes-load-testing/locations/us-central1/functions/bigframes-ab103e98fa51cfc67a56d754b7e5068e-u8owkeqc" ,) kwargs = {'metadata': [('x-goog-request-params', 'name=projects/bigframes-load-testing/locations/us-central1/functions/bigframe...6d754b7e5068e-u8owkeqc'), ('x-goog-api-client', 'bigframes/1.7.0 ibis/8.0.0 gl-python/3.12.0 grpc/1.64.0 gax/2.19.0')]} @functools.wraps(callable_) def error_remapped_callable(*args, **kwargs): try: return callable_(*args, **kwargs) except grpc.RpcError as exc: > raise exceptions.from_grpc_error(exc) from exc E google.api_core.exceptions.ServiceUnavailable: 503 502:Bad Gateway .nox/load/lib/python3.12/site-packages/google/api_core/grpc_helpers.py:78: ServiceUnavailable ``` * include warning about failed cleanup * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
1 parent 085fa9d commit d31cebd

File tree

1 file changed

+10
-1
lines changed

1 file changed

+10
-1
lines changed

tests/system/conftest.py

+10-1
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
import textwrap
2121
import typing
2222
from typing import Dict, Generator, Optional
23+
import warnings
2324

2425
import google.api_core.exceptions
2526
import google.cloud.bigquery as bigquery
@@ -1129,13 +1130,21 @@ def cleanup_cloud_functions(session, cloudfunctions_client, dataset_id_permanent
11291130
# successfully, while the other instance will run into this
11301131
# exception. Ignore this exception.
11311132
pass
1132-
except google.api_core.exceptions.ResourceExhausted:
1133+
except Exception as exc:
1134+
# Don't fail the tests for unknown exceptions.
1135+
#
11331136
# This can happen if we are hitting GCP limits, e.g.
11341137
# google.api_core.exceptions.ResourceExhausted: 429 Quota exceeded
11351138
# for quota metric 'Per project mutation requests' and limit
11361139
# 'Per project mutation requests per minute per region' of service
11371140
# 'cloudfunctions.googleapis.com' for consumer
11381141
# 'project_number:1084210331973'.
11391142
# [reason: "RATE_LIMIT_EXCEEDED" domain: "googleapis.com" ...
1143+
#
1144+
# It can also happen occasionally with
1145+
# google.api_core.exceptions.ServiceUnavailable when there is some
1146+
# backend flakiness.
1147+
#
11401148
# Let's stop further clean up and leave it to later.
1149+
warnings.warn(f"Cloud functions cleanup failed: {str(exc)}")
11411150
break

0 commit comments

Comments
 (0)