Repository navigation
feat(storage): add OpenTelemetry standard metrics for Async gRPC client - #18762
shradhakatyal wants to merge 2 commits into
Conversation
…d helpers Implement core OpenTelemetry standard metrics infrastructure for GCS: - Add _MetricsRecorder with the 9 standard metric instruments - Add custom latency (165 buckets) and byte-size (45 buckets) histogram views - Add classify_error_type for low-cardinality error.type mapping - Add normalize_url_template for bounded HTTP url.template routing - Add start_operation ContextVar for outermost logical operation tracking and nested call suppression [Generated-by: AI]
Implement OpenTelemetry standard metrics for AsyncGrpcClient, AsyncMultiRangeDownloader, AsyncAppendableObjectWriter, AsyncReadObjectStream, AsyncWriteObjectStream, and _StreamMultiplexer following the GCS O11y specification. [Generated-by: AI]
There was a problem hiding this comment.
Code Review
This pull request implements OpenTelemetry metrics support for the Google Cloud Storage asyncio client using gRPC. It introduces standard metric instruments, custom histogram bucket boundaries, canonical error classification, and route normalization. Additionally, it adds gRPC client interceptors and call tracking proxies to record physical attempt metrics, and integrates logical operation tracking across various asyncio stream and writer classes. The feedback suggests implementing __getattr__ in _TrackedCallBase to delegate unhandled attribute access to the underlying call object, ensuring robust compatibility with gRPC's internal implementation details.
| ) | ||
| self._network_path = network_path | ||
| self._recorded = False | ||
| self._closing_cleanly = False |
There was a problem hiding this comment.
To ensure full compatibility with gRPC's internal implementation details, other interceptors, or future gRPC updates that might access private or undocumented attributes on the call object, implement __getattr__ in _TrackedCallBase to transparently delegate any unhandled attribute or method access to the underlying self._call object.
self._closing_cleanly = False
def __getattr__(self, name: str) -> Any:
return getattr(self._call, name)
Summary
Implements OpenTelemetry Standard Metrics instrumentation for the Async gRPC client (
AsyncGrpcClient) and Bidi streaming abstractions ingoogle-cloud-storage:_opentelemetry_metrics.py):gcp.storage.operation.count,gcp.storage.operation.duration,gcp.storage.operation.retry.count,gcp.storage.operation.transfer_size,gcp.storage.api.request.count,gcp.storage.api.request.duration,gcp.storage.api.request.body_size,gcp.storage.api.response.body_size,gcp.storage.stream.idle_duration), histogram bucket views, and attribute keys.extract_grpc_server_address_and_network_path), gRPC status/error classification (extract_grpc_status_code,classify_error_type), and Bidi stream metrics (_GrpcBidiStreamMetricsContext).async_grpc_client.py):_metrics_managerinAsyncGrpcClient.__init__using the configured gRPC transport endpoint and user options (enable_metrics,meter_provider,metrics_exporter).delete_object,get_object_metadata,compose_object,restore_object,move_object,update_object_metadata,lock_bucket_retention_policy,start_resumable_write,query_write_status) withstart_operationandstart_api_request, wiring GAPICAsyncRetry(on_error=...)callbacks to record retry counts and per-attempt API request metrics.download_object(server-streamingReadObject) with operation and API request contexts, recordingrecord_response_bytesper chunk.upload_object(BidiWriteObject) with operation and Bidi stream metrics, recordingrecord_request_bytesandstream.idle_duration.async_multi_range_downloader.py,async_appendable_object_writer.py,async_read_object_stream.py,async_write_object_stream.py,_stream_multiplexer.py):AsyncMultiRangeDownloader(open,download_ranges,_reconnect),AsyncAppendableObjectWriter(open,append,flush,close,simple_flush,_reconnect_and_recover),AsyncReadObjectStream,AsyncWriteObjectStream, and_StreamMultiplexerwith operation, Bidi stream, transfer size, retry, and idle duration metrics.test__opentelemetry_metrics.pyandtests/unit/asyncio/(348 unit tests passing).[Generated-by: AI]