error reporting still broken at 0.24

[asilversempirical]

After updating to 0.24 to fix #3167 I'm still unable to report exceptions.

Ubuntu xenial w/ Python 3.5.2

client.report_exception(http_context=error_reporting.build_flask_context(flask.request))

On google-cloud-error-reporting <0.23 the above line works fine. At 0.24 I get:

Apr  1 15:52:51   File "/usr/local/lib/python3.5/dist-packages/google/cloud/error_reporting/client.py", line 347, in report_exception
Apr  1 15:52:51     user=user)
Apr  1 15:52:51   File "/usr/local/lib/python3.5/dist-packages/google/cloud/error_reporting/client.py", line 276, in _send_error_report
Apr  1 15:52:51     self.report_errors_api.report_error_event(error_report)
Apr  1 15:52:51   File "/usr/local/lib/python3.5/dist-packages/google/cloud/error_reporting/_gax.py", line 75, in report_error_event
Apr  1 15:52:51     ParseDict(error_report, error_report_payload)
Apr  1 15:52:51   File "/usr/local/lib/python3.5/dist-packages/google/protobuf/json_format.py", line 384, in ParseDict
Apr  1 15:52:51     parser.ConvertMessage(js_dict, message)
Apr  1 15:52:51   File "/usr/local/lib/python3.5/dist-packages/google/protobuf/json_format.py", line 415, in ConvertMessage
Apr  1 15:52:51     self._ConvertFieldValuePair(value, message)
Apr  1 15:52:51   File "/usr/local/lib/python3.5/dist-packages/google/protobuf/json_format.py", line 501, in _ConvertFieldValuePair
Apr  1 15:52:51     raise ParseError('Failed to parse {0} field: {1}'.format(name, e))"   

[lukesneeringer]

Hi @asilversempirical,
That is frustrating. We will try to figure out what the problem is.

[dhermes]

@waprin I am re-opening this since you said "probably" in #3299. Feel free to close if you think you nailed it (but again, prefer a system test).

[asilversempirical]

Based on #3315 I was thinking this was fixed in 0.24.1, but I'm still having issues (albeit new ones).

I'm now seeing:

Apr 20 21:30:45   File "/usr/local/www/server/server.py", line 95, in error_handler
Apr 20 21:30:45     flask.request))
Apr 20 21:30:45   File "/usr/local/lib/python3.5/dist-packages/google/cloud/error_reporting/client.py", line 346, in report_exception
Apr 20 21:30:45     user=user)
Apr 20 21:30:45   File "/usr/local/lib/python3.5/dist-packages/google/cloud/error_reporting/client.py", line 275, in _send_error_report
Apr 20 21:30:45     self.report_errors_api.report_error_event(error_report)
Apr 20 21:30:45   File "/usr/local/lib/python3.5/dist-packages/google/cloud/error_reporting/_gax.py", line 76, in report_error_event
Apr 20 21:30:45     self._gax_api.report_error_event(project_name, error_report_payload)
Apr 20 21:30:45   File "/usr/local/lib/python3.5/dist-packages/google/cloud/gapic/errorreporting/v1beta1/report_errors_service_client.py", line 208, in report_error_event
Apr 20 21:30:45     return self._report_error_event(request, options)
Apr 20 21:30:45   File "/usr/local/lib/python3.5/dist-packages/google/gax/api_callable.py", line 419, in inner
Apr 20 21:30:45     return api_caller(api_call, this_settings, request)
Apr 20 21:30:45   File "/usr/local/lib/python3.5/dist-packages/google/gax/api_callable.py", line 407, in base_caller
Apr 20 21:30:45     return api_call(*args)
Apr 20 21:30:45   File "/usr/local/lib/python3.5/dist-packages/google/gax/api_callable.py", line 368, in inner
Apr 20 21:30:45     return a_func(*args, **kwargs)
Apr 20 21:30:45   File "/usr/local/lib/python3.5/dist-packages/google/gax/retry.py", line 126, in inner
Apr 20 21:30:45     ' classified as transient', exception)" 

I'd blame myself for doing something weird with credentials, but this is with identical code + config that works in <023.0. Has something else changed with credential handling?

[waprin]

Still in process of taking another pass at testing this (sorry for delay). <0.23.0 was quite different since it was defaulting to writing to Stackdriver via Logging instead of using gRPC to the API directly. So it's honestly not a huge surprise we're running into some issues, it was a big change.

Error Reporting constructor takes a _use_grpc that falls back to the old approach. You can also set it with GOOGLE_CLOUD_DISABLE_GRPC environment variable enabled. Guessing that will workaround the issue until we dig into what's going on with you here.

Any more information to help reproduce always appreciated.

[asilversempirical]

I don't think we're doing anything particularly strange. Our call is just:

from google.cloud import error_reporting

def error_handler(e):
    client = error_reporting.Client()
    client.report_exception(
        http_context=error_reporting.build_flask_context(flask.request))

app.register_error_handler(500, error_handler)

Our credentials file is provided via $GOOGLE_APPLICATION_CREDENTIALS

If there are other details you think would be useful I'm happy to provide them, but I think we're using it in a pretty straightforward manner.

Thanks for the consistently quick responses!

[theacodes]

It seems error_reporting isn't specifying the scopes needed for the credentials:

https://travis-ci.org/GoogleCloudPlatform/python-docs-samples/builds/225318153

@waprin / @dhermes is this the same issue?

[dhermes]

@jonparrott It seems very possible.

[Fkawala]

@jonparrott I reproduce the scope issue. Below a docker-ized MRE.
@dhermes Is it related to #2934 ? Is there a workaround ?

main.py

import google.cloud.error_reporting as error_reporting
import google.cloud.logging

if __name__ == '__main__':
    try:
        # First verify that the logging works.
        client = google.cloud.logging.Client()
        logger = client.logger('log_name')
        logger.log_text('A simple entry')
        # Genererate some error to report.
        raise RuntimeError("I'm dead.")
    except RuntimeError:
        client = error_reporting.Client(service="repro", version='v-xyz')
        client.report_exception()
        print('reported')

Dockerfile

# Dockerfile extending the generic Python image with application files for a
# single application.
FROM ubuntu:latest

# Add graph tool sources.
RUN apt-get update

# Install much needed packages.
RUN apt-get install -y \
    python3 \
    python3-pip

# Install python packages.
RUN pip3 install google-cloud-error-reporting

# Add the app.
ADD . /app

# Set CWD.
WORKDIR /app

# Set ENVs.
ENV GCLOUD_PROJECT project-name
ENV GOOGLE_APPLICATION_CREDENTIALS /app/ServiceAccount.json

CMD python3 /app/main.py

Sample output

E0427 15:45:48.679450186       6 plugin_credentials.c:74]    Getting metadata from plugin failed with error: Traceback (most recent call last):
  File "/app/main.py", line 11, in <module>
    raise RuntimeError("I'm dead.")
RuntimeError: I'm dead.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 154, in grpc._cython.cygrpc.plugin_get_metadata (src/python/grpcio/grpc/_cython/cygrpc.c:6989)
  File "/usr/local/lib/python3.5/dist-packages/grpc/_plugin_wrapping.py", line 106, in __call__
    AuthMetadataPluginCallback(wrapped_cygrpc_callback))
  File "/usr/local/lib/python3.5/dist-packages/google/auth/transport/grpc.py", line 73, in __call__
    callback(self._get_authorization_headers(context), None)
  File "/usr/local/lib/python3.5/dist-packages/google/auth/transport/grpc.py", line 61, in _get_authorization_headers
    headers)
  File "/usr/local/lib/python3.5/dist-packages/google/auth/credentials.py", line 118, in before_request
    self.refresh(request)
  File "/usr/local/lib/python3.5/dist-packages/google/oauth2/service_account.py", line 310, in refresh
    request, self._token_uri, assertion)
  File "/usr/local/lib/python3.5/dist-packages/google/oauth2/_client.py", line 143, in jwt_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/usr/local/lib/python3.5/dist-packages/google/oauth2/_client.py", line 109, in _token_endpoint_request
    _handle_error_response(response_body)
  File "/usr/local/lib/python3.5/dist-packages/google/oauth2/_client.py", line 59, in _handle_error_response
    error_details, response_body)
google.auth.exceptions.RefreshError: ('invalid_scope: Empty or missing scope not allowed.', '{\n  "error" : "invalid_scope",\n  "error_description" : "Empty or missing scope not allowed."\n}')

Traceback (most recent call last):
  File "/app/main.py", line 11, in <module>
    raise RuntimeError("I'm dead.")
RuntimeError: I'm dead.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/google/gax/retry.py", line 120, in inner
    return to_call(*args)
  File "/usr/local/lib/python3.5/dist-packages/google/gax/retry.py", line 68, in inner
    return a_func(*updated_args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/grpc/_channel.py", line 507, in __call__
    return _end_unary_response_blocking(state, call, False, deadline)
  File "/usr/local/lib/python3.5/dist-packages/grpc/_channel.py", line 455, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAUTHENTICATED, Traceback (most recent call last):
  File "/app/main.py", line 11, in <module>
    raise RuntimeError("I'm dead.")
RuntimeError: I'm dead.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 154, in grpc._cython.cygrpc.plugin_get_metadata (src/python/grpcio/grpc/_cython/cygrpc.c:6989)
  File "/usr/local/lib/python3.5/dist-packages/grpc/_plugin_wrapping.py", line 106, in __call__
    AuthMetadataPluginCallback(wrapped_cygrpc_callback))
  File "/usr/local/lib/python3.5/dist-packages/google/auth/transport/grpc.py", line 73, in __call__
    callback(self._get_authorization_headers(context), None)
  File "/usr/local/lib/python3.5/dist-packages/google/auth/transport/grpc.py", line 61, in _get_authorization_headers
    headers)
  File "/usr/local/lib/python3.5/dist-packages/google/auth/credentials.py", line 118, in before_request
    self.refresh(request)
  File "/usr/local/lib/python3.5/dist-packages/google/oauth2/service_account.py", line 310, in refresh
    request, self._token_uri, assertion)
  File "/usr/local/lib/python3.5/dist-packages/google/oauth2/_client.py", line 143, in jwt_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/usr/local/lib/python3.5/dist-packages/google/oauth2/_client.py", line 109, in _token_endpoint_request
    _handle_error_response(response_body)
  File "/usr/local/lib/python3.5/dist-packages/google/oauth2/_client.py", line 59, in _handle_error_response
    error_details, response_body)
google.auth.exceptions.RefreshError: ('invalid_scope: Empty or missing scope not allowed.', '{\n  "error" : "invalid_scope",\n  "error_description" : "Empty or missing scope not allowed."\n}')
)>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/main.py", line 14, in <module>
    client.report_exception()
  File "/usr/local/lib/python3.5/dist-packages/google/cloud/error_reporting/client.py", line 346, in report_exception
    user=user)
  File "/usr/local/lib/python3.5/dist-packages/google/cloud/error_reporting/client.py", line 275, in _send_error_report
    self.report_errors_api.report_error_event(error_report)
  File "/usr/local/lib/python3.5/dist-packages/google/cloud/error_reporting/_gax.py", line 76, in report_error_event
    self._gax_api.report_error_event(project_name, error_report_payload)
  File "/usr/local/lib/python3.5/dist-packages/google/cloud/gapic/errorreporting/v1beta1/report_errors_service_client.py", line 208, in report_error_event
    return self._report_error_event(request, options)
  File "/usr/local/lib/python3.5/dist-packages/google/gax/api_callable.py", line 419, in inner
    return api_caller(api_call, this_settings, request)
  File "/usr/local/lib/python3.5/dist-packages/google/gax/api_callable.py", line 407, in base_caller
    return api_call(*args)
  File "/usr/local/lib/python3.5/dist-packages/google/gax/api_callable.py", line 368, in inner
    return a_func(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/google/gax/retry.py", line 126, in inner
    ' classified as transient', exception)
google.gax.errors.RetryError: GaxError(Exception occurred in retry method that was not classified as transient, caused by <_Rendezvous of RPC that terminated with (StatusCode.UNAUTHENTICATED, Traceback (most recent call last):
  File "/app/main.py", line 11, in <module>
    raise RuntimeError("I'm dead.")
RuntimeError: I'm dead.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 154, in grpc._cython.cygrpc.plugin_get_metadata (src/python/grpcio/grpc/_cython/cygrpc.c:6989)
  File "/usr/local/lib/python3.5/dist-packages/grpc/_plugin_wrapping.py", line 106, in __call__
    AuthMetadataPluginCallback(wrapped_cygrpc_callback))
  File "/usr/local/lib/python3.5/dist-packages/google/auth/transport/grpc.py", line 73, in __call__
    callback(self._get_authorization_headers(context), None)
  File "/usr/local/lib/python3.5/dist-packages/google/auth/transport/grpc.py", line 61, in _get_authorization_headers
    headers)
  File "/usr/local/lib/python3.5/dist-packages/google/auth/credentials.py", line 118, in before_request
    self.refresh(request)
  File "/usr/local/lib/python3.5/dist-packages/google/oauth2/service_account.py", line 310, in refresh
    request, self._token_uri, assertion)
  File "/usr/local/lib/python3.5/dist-packages/google/oauth2/_client.py", line 143, in jwt_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/usr/local/lib/python3.5/dist-packages/google/oauth2/_client.py", line 109, in _token_endpoint_request
    _handle_error_response(response_body)
  File "/usr/local/lib/python3.5/dist-packages/google/oauth2/_client.py", line 59, in _handle_error_response
    error_details, response_body)
google.auth.exceptions.RefreshError: ('invalid_scope: Empty or missing scope not allowed.', '{\n  "error" : "invalid_scope",\n  "error_description" : "Empty or missing scope not allowed."\n}')
)>)

[dhermes]

The culprit is that error_reporting.Client has no scope. BLERG! Compare to a typical Client.

@waprin We DESPERATELY need system tests here. Or we should just push a release that results in:

>>> from google.cloud import error_reporting
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../error_reporting/google/cloud/error_reporting/__init__.py", line 18, in <module>
    raise ImportError('This package is currently under construction.')
ImportError: This package is currently under construction.