Skip to content

Commit 8ba0023

Browse files
arbrowngcf-owl-bot[bot]release-please[bot]daniel-sanche
authored
feat: Add support for library instrumentation (#551)
* Add .python-version to .gitignore * Add initial class/test for instrumentation_source * Add version and truncate logic * Add instrumentation tests and severity info * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Add method to update and validate existing info * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Add .python-version to gitignore * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Implement hook to add instrumentation for logger * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Add tests for logger instrumentation logic * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Update structured log handler to emit info * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Refactor structured log and add unit test * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Add side effect to unit test * Ensure that instrumentation info is only called once * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Update to environment submodule * Fix linter errors * chore(main): release 3.1.0 (#479) Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com> * docs: Change button in README to .png file (#554) * fix: Change button to .png file * Change the "Guide Me" button to a .png file to work with RST `image` * Avoids `raw` directive disabled by PyPi * Fixes #553 * Fix unexpected unindent in docs Add copy of png file to render correctly in GitHub readme and Sphinx generated docs. * chore(main): release 3.1.1 (#557) Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com> * Update env-tests submodule * Minor format update * Fix system test to skip diagnostic log entry * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Update truncation logic based on feedback * Update environment tests * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Fix broken unit test * Fix broken unit test * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Change default name/version * Refactor add_instrumentation Return a new list instead of a mutated original Do not return after first log without info * Add more documentation to validation methods * Refactor add_instrumentation to be more pythonic * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Update environemnt tests * Refactor _is_valid and add test * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Add more detail to method documentation * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Move methods to private * Change instumentation_added to private * Fix some issues with validation method Add a test * Fix bug in _add_instrumentation * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Simplify string truncation * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Remove unused import to fix lint * Remove validate_and_update_instrumentation Simplify code by adding a single instrumentation entry instead of validating potential existing entries * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Remove _is_valid code (no longer checked) * Run nox blacken * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Remove extraneous unit test Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com> Co-authored-by: Daniel Sanche <[email protected]>
1 parent a2eed8c commit 8ba0023

File tree

8 files changed

+260
-2
lines changed

8 files changed

+260
-2
lines changed

google/cloud/logging_v2/__init__.py

+2
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,8 @@
4141
"""Query string to order by ascending timestamps."""
4242
DESCENDING = "timestamp desc"
4343
"""Query string to order by decending timestamps."""
44+
_instrumentation_emitted = False
45+
"""Flag for whether instrumentation info has been emitted"""
4446

4547

4648
__all__ = (
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
# Copyright 2022 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
"""Add diagnostic instrumentation source information to logs"""
16+
from google.cloud.logging_v2.entries import StructEntry
17+
from google.cloud.logging_v2 import __version__
18+
19+
_DIAGNOSTIC_INFO_KEY = "logging.googleapis.com/diagnostic"
20+
_INSTRUMENTATION_SOURCE_KEY = "instrumentation_source"
21+
_PYTHON_LIBRARY_NAME = "python"
22+
23+
_LIBRARY_VERSION = __version__
24+
25+
_MAX_NAME_LENGTH = 14
26+
_MAX_VERSION_LENGTH = 14
27+
_MAX_INSTRUMENTATION_ENTRIES = 3
28+
29+
30+
def _add_instrumentation(entries, **kw):
31+
"""Add instrumentation information to a list of entries
32+
33+
A new diagnostic entry is prepended to the list of
34+
entries.
35+
36+
Args:
37+
entries (Sequence[Mapping[str, ...]]): sequence of mappings representing
38+
the log entry resources to log.
39+
40+
Returns:
41+
Sequence[Mapping[str, ...]]: entries with instrumentation info added to
42+
the beginning of list.
43+
"""
44+
45+
diagnostic_entry = _create_diagnostic_entry(**kw)
46+
entries.insert(0, diagnostic_entry.to_api_repr())
47+
return entries
48+
49+
50+
def _create_diagnostic_entry(name=_PYTHON_LIBRARY_NAME, version=_LIBRARY_VERSION, **kw):
51+
"""Create a diagnostic log entry describing this library
52+
53+
The diagnostic log consists of a list of library name and version objects
54+
that have handled a given log entry. If this library is the originator
55+
of the log entry, it will look like:
56+
{logging.googleapis.com/diagnostic: {instrumentation_source: [{name: "python", version: "3.0.0"}]}}
57+
58+
Args:
59+
name(str): The name of this library (e.g. 'python')
60+
version(str) The version of this library (e.g. '3.0.0')
61+
62+
Returns:
63+
google.cloud.logging_v2.LogEntry: Log entry with library information
64+
"""
65+
payload = {
66+
_DIAGNOSTIC_INFO_KEY: {
67+
_INSTRUMENTATION_SOURCE_KEY: [_get_instrumentation_source(name, version)]
68+
}
69+
}
70+
kw["severity"] = "INFO"
71+
entry = StructEntry(payload=payload, **kw)
72+
return entry
73+
74+
75+
def _get_instrumentation_source(name=_PYTHON_LIBRARY_NAME, version=_LIBRARY_VERSION):
76+
"""Gets a JSON representation of the instrumentation_source
77+
78+
Args:
79+
name(str): The name of this library (e.g. 'python')
80+
version(str) The version of this library (e.g. '3.0.0')
81+
Returns:
82+
obj: JSON object with library information
83+
"""
84+
source = {"name": name, "version": version}
85+
# truncate strings to no more than _MAX_NAME_LENGTH characters
86+
for key, val in source.items():
87+
source[key] = (
88+
val if len(val) <= _MAX_NAME_LENGTH else f"{val[:_MAX_NAME_LENGTH]}*"
89+
)
90+
return source

google/cloud/logging_v2/handlers/structured_log.py

+13
Original file line numberDiff line numberDiff line change
@@ -16,10 +16,13 @@
1616
"""
1717
import collections
1818
import json
19+
import logging
1920
import logging.handlers
2021

2122
from google.cloud.logging_v2.handlers.handlers import CloudLoggingFilter
2223
from google.cloud.logging_v2.handlers.handlers import _format_and_parse_message
24+
import google.cloud.logging_v2
25+
from google.cloud.logging_v2._instrumentation import _create_diagnostic_entry
2326

2427
GCP_FORMAT = (
2528
"{%(_payload_str)s"
@@ -84,3 +87,13 @@ def format(self, record):
8487
# convert to GCP structred logging format
8588
gcp_payload = self._gcp_formatter.format(record)
8689
return gcp_payload
90+
91+
def emit(self, record):
92+
if google.cloud.logging_v2._instrumentation_emitted is False:
93+
self.emit_instrumentation_info()
94+
super().emit(record)
95+
96+
def emit_instrumentation_info(self):
97+
google.cloud.logging_v2._instrumentation_emitted = True
98+
diagnostic_object = _create_diagnostic_entry().to_api_repr()
99+
logging.info(diagnostic_object)

google/cloud/logging_v2/logger.py

+9-1
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@
2323
from google.cloud.logging_v2.entries import TextEntry
2424
from google.cloud.logging_v2.resource import Resource
2525
from google.cloud.logging_v2.handlers._monitored_resources import detect_resource
26+
from google.cloud.logging_v2._instrumentation import _add_instrumentation
2627

2728
import google.protobuf.message
2829

@@ -134,6 +135,7 @@ def _do_log(self, client, _entry_class, payload=None, **kw):
134135
kw["log_name"] = kw.pop("log_name", self.full_name)
135136
kw["labels"] = kw.pop("labels", self.labels)
136137
kw["resource"] = kw.pop("resource", self.default_resource)
138+
partial_success = False
137139

138140
severity = kw.get("severity", None)
139141
if isinstance(severity, str) and not severity.isupper():
@@ -155,7 +157,13 @@ def _do_log(self, client, _entry_class, payload=None, **kw):
155157
entry = _entry_class(**kw)
156158

157159
api_repr = entry.to_api_repr()
158-
client.logging_api.write_entries([api_repr])
160+
entries = [api_repr]
161+
if google.cloud.logging_v2._instrumentation_emitted is False:
162+
partial_success = True
163+
entries = _add_instrumentation(entries, **kw)
164+
google.cloud.logging_v2._instrumentation_emitted = True
165+
166+
client.logging_api.write_entries(entries, partial_success=partial_success)
159167

160168
def log_empty(self, *, client=None, **kw):
161169
"""Log an empty message

tests/system/test_system.py

+5
Original file line numberDiff line numberDiff line change
@@ -888,6 +888,11 @@ def test_update_sink(self):
888888

889889
@skip_for_mtls
890890
def test_api_equality_list_logs(self):
891+
import google.cloud.logging_v2
892+
893+
# Skip diagnostic log for this system test
894+
google.cloud.logging_v2._instrumentation_emitted = True
895+
891896
unique_id = uuid.uuid1()
892897
gapic_logger = Config.CLIENT.logger(f"api-list-{unique_id}")
893898
http_logger = Config.HTTP_CLIENT.logger(f"api-list-{unique_id}")

tests/unit/handlers/test_structured_log.py

+24
Original file line numberDiff line numberDiff line change
@@ -438,3 +438,27 @@ def test_format_with_json_fields(self):
438438
self.assertEqual(result["message"], expected_result)
439439
self.assertEqual(result["hello"], "world")
440440
self.assertEqual(result["number"], 12)
441+
442+
def test_emits_instrumentation_info(self):
443+
import logging
444+
import mock
445+
import google.cloud.logging_v2
446+
447+
handler = self._make_one()
448+
logname = "loggername"
449+
message = "Hello world!"
450+
451+
record = logging.LogRecord(logname, logging.INFO, "", 0, message, None, None)
452+
453+
with mock.patch.object(handler, "emit_instrumentation_info") as emit_info:
454+
455+
def side_effect():
456+
google.cloud.logging_v2._instrumentation_emitted = True
457+
458+
emit_info.side_effect = side_effect
459+
google.cloud.logging_v2._instrumentation_emitted = False
460+
handler.emit(record)
461+
handler.emit(record)
462+
463+
# emit_instrumentation_info should be called once
464+
emit_info.assert_called_once()

tests/unit/test__instrumentation.py

+65
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
# Copyright 2022 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
import unittest
16+
import google.cloud.logging_v2._instrumentation as i
17+
18+
19+
class TestInstrumentation(unittest.TestCase):
20+
21+
TEST_NAME = "python"
22+
# LONG_NAME > 14 characters
23+
LONG_NAME = TEST_NAME + "789ABCDEF"
24+
25+
TEST_VERSION = "1.0.0"
26+
# LONG_VERSION > 16 characters
27+
LONG_VERSION = TEST_VERSION + "6789ABCDEF12"
28+
29+
def _get_diagonstic_value(self, entry, key):
30+
return entry.payload[i._DIAGNOSTIC_INFO_KEY][i._INSTRUMENTATION_SOURCE_KEY][-1][
31+
key
32+
]
33+
34+
def test_default_diagnostic_info(self):
35+
entry = i._create_diagnostic_entry()
36+
self.assertEqual(
37+
i._PYTHON_LIBRARY_NAME,
38+
self._get_diagonstic_value(entry, "name"),
39+
)
40+
self.assertEqual(
41+
i._LIBRARY_VERSION, self._get_diagonstic_value(entry, "version")
42+
)
43+
44+
def test_custom_diagnostic_info(self):
45+
entry = i._create_diagnostic_entry(
46+
name=self.TEST_NAME, version=self.TEST_VERSION
47+
)
48+
self.assertEqual(
49+
self.TEST_NAME,
50+
self._get_diagonstic_value(entry, "name"),
51+
)
52+
self.assertEqual(
53+
self.TEST_VERSION, self._get_diagonstic_value(entry, "version")
54+
)
55+
56+
def test_truncate_long_values(self):
57+
entry = i._create_diagnostic_entry(
58+
name=self.LONG_NAME, version=self.LONG_VERSION
59+
)
60+
61+
expected_name = self.LONG_NAME[: i._MAX_NAME_LENGTH] + "*"
62+
expected_version = self.LONG_VERSION[: i._MAX_VERSION_LENGTH] + "*"
63+
64+
self.assertEqual(expected_name, self._get_diagonstic_value(entry, "name"))
65+
self.assertEqual(expected_version, self._get_diagonstic_value(entry, "version"))

tests/unit/test_logger.py

+52-1
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,12 @@ class TestLogger(unittest.TestCase):
3434
LOGGER_NAME = "logger-name"
3535
TIME_FORMAT = '"%Y-%m-%dT%H:%M:%S.%f%z"'
3636

37+
def setUp(self):
38+
import google.cloud.logging_v2
39+
40+
# Test instrumentation behavior in only one test
41+
google.cloud.logging_v2._instrumentation_emitted = True
42+
3743
@staticmethod
3844
def _get_target_class():
3945
from google.cloud.logging import Logger
@@ -975,6 +981,43 @@ def test_list_entries_folder(self):
975981
self.assertIsNone(entry.logger)
976982
self.assertEqual(entry.log_name, LOG_NAME)
977983

984+
def test_first_log_emits_instrumentation(self):
985+
from google.cloud.logging_v2.handlers._monitored_resources import (
986+
detect_resource,
987+
)
988+
from google.cloud.logging_v2._instrumentation import _create_diagnostic_entry
989+
import google.cloud.logging_v2
990+
991+
google.cloud.logging_v2._instrumentation_emitted = False
992+
DEFAULT_LABELS = {"foo": "spam"}
993+
resource = detect_resource(self.PROJECT)
994+
instrumentation_entry = _create_diagnostic_entry(
995+
resource=resource,
996+
labels=DEFAULT_LABELS,
997+
).to_api_repr()
998+
instrumentation_entry["logName"] = "projects/%s/logs/%s" % (
999+
self.PROJECT,
1000+
self.LOGGER_NAME,
1001+
)
1002+
ENTRIES = [
1003+
instrumentation_entry,
1004+
{
1005+
"logName": "projects/%s/logs/%s" % (self.PROJECT, self.LOGGER_NAME),
1006+
"resource": resource._to_dict(),
1007+
"labels": DEFAULT_LABELS,
1008+
},
1009+
]
1010+
client = _Client(self.PROJECT)
1011+
api = client.logging_api = _DummyLoggingAPI()
1012+
logger = self._make_one(self.LOGGER_NAME, client=client, labels=DEFAULT_LABELS)
1013+
logger.log_empty()
1014+
self.assertEqual(api._write_entries_called_with, (ENTRIES, None, None, None))
1015+
1016+
ENTRIES = ENTRIES[-1:]
1017+
api = client.logging_api = _DummyLoggingAPI()
1018+
logger.log_empty()
1019+
self.assertEqual(api._write_entries_called_with, (ENTRIES, None, None, None))
1020+
9781021

9791022
class TestBatch(unittest.TestCase):
9801023

@@ -1645,7 +1688,15 @@ class _DummyLoggingAPI(object):
16451688

16461689
_write_entries_called_with = None
16471690

1648-
def write_entries(self, entries, *, logger_name=None, resource=None, labels=None):
1691+
def write_entries(
1692+
self,
1693+
entries,
1694+
*,
1695+
logger_name=None,
1696+
resource=None,
1697+
labels=None,
1698+
partial_success=False,
1699+
):
16491700
self._write_entries_called_with = (entries, logger_name, resource, labels)
16501701

16511702
def logger_delete(self, logger_name):

0 commit comments

Comments
 (0)