Repository navigation
Grpc uds rust - #18761
Grpc uds rust#18761raj-prince wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces google-cloud-storage-uds, a high-performance Rust gRPC-over-UDS accelerator server and a thin Python client wrapper for Google Cloud Storage, supporting zero-copy shared memory transfers. The code review highlights several critical correctness, stability, and compatibility issues. Specifically, zero-copy reads with zero length are silently truncated and should return an error instead. Additionally, spawning the sidecar subprocess with an unread stdout pipe can lead to deadlocks, and closing active mmap regions while memoryview slices are held causes application crashes. Finally, Unix-specific calls like os.sysconf and os.getuid() must be wrapped with cross-platform fallbacks to prevent crashes on Windows.
|
|
||
| // Zero-copy shared-memory (`mmap`) path: write directly into `/dev/shm` | ||
| // and return only `actual_lengths` over the UDS socket. | ||
| if !req.shm_path.is_empty() { |
There was a problem hiding this comment.
Zero-copy shared memory reads with length = 0 (unbounded reads) are silently truncated to 1024 bytes because cap is hardcoded to 1024 when r.length == 0. This is a critical correctness bug that will cause silent data truncation. The server should return an error if a zero-copy read is requested with an unbounded/zero length.
if !req.shm_path.is_empty() {
for r in &ranges {
if r.length == 0 {
return Err(Status::invalid_argument(
"Zero-copy shared memory reads require explicit, non-zero range lengths"
));
}
}
let mut offsets_and_caps = Vec::with_capacity(ranges.len());| self._process = subprocess.Popen( | ||
| cmd, | ||
| stdout=subprocess.PIPE, | ||
| stderr=subprocess.PIPE, | ||
| text=True, | ||
| bufsize=1, | ||
| ) |
There was a problem hiding this comment.
Spawning the subprocess with stdout=subprocess.PIPE without actively consuming/reading from the pipe while the process is running can lead to a deadlock/hang if the OS pipe buffer fills up (typically 64KB on Linux). Since stdout is never read in the codebase, any standard output from the server (such as the READY signal or other prints) will eventually block the server process. Redirect stdout to subprocess.DEVNULL to prevent this.
| self._process = subprocess.Popen( | |
| cmd, | |
| stdout=subprocess.PIPE, | |
| stderr=subprocess.PIPE, | |
| text=True, | |
| bufsize=1, | |
| ) | |
| self._process = subprocess.Popen( | |
| cmd, | |
| stdout=subprocess.DEVNULL, | |
| stderr=subprocess.PIPE, | |
| text=True, | |
| bufsize=1, | |
| ) |
| if self._mmap is not None: | ||
| self._mmap.close() | ||
| self._mmap = None |
There was a problem hiding this comment.
Calling self._mmap.close() while the caller still holds an active/exported memoryview slice from a previous read will raise a BufferError: cannot close exported pointers exist and crash the application. To prevent this, catch BufferError and keep the old mmap alive in a retired list (which should be initialized in __init__ to avoid defensive hasattr checks) so that the active mapping can be safely recreated while existing views remain valid.
if self._mmap is not None:
try:
self._mmap.close()
except BufferError:
self._retired_mmaps.append(self._mmap)
self._mmap = NoneReferences
- Do not use defensive getattr/hasattr checks for attributes that should be initialized in the class's init method.
| A `ProcessMetrics` snapshot with cumulative CPU time in seconds and | ||
| resident memory in MiB. | ||
| """ | ||
| clk_tck = os.sysconf("SC_CLK_TCK") |
There was a problem hiding this comment.
os.sysconf is Unix-specific and is not available on Windows platforms. Calling it directly will raise an AttributeError when running on Windows. To ensure cross-platform compatibility (e.g., when importing or running telemetry on Windows), wrap this call in a try-except block or check for attribute existence.
| clk_tck = os.sysconf("SC_CLK_TCK") | |
| try: | |
| clk_tck = os.sysconf("SC_CLK_TCK") | |
| except (AttributeError, ValueError): | |
| clk_tck = 100 |
| credentials and HTTP/gRPC pools before reporting ready. | ||
| startup_timeout_s: Maximum seconds to wait for server startup. | ||
| """ | ||
| uid = os.getuid() |
Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:
Fixes #<issue_number_goes_here> 🦕