Skip to content

OTLP nanosecond timestamp overflow in webapp event repository #3292

Open
Open
OTLP nanosecond timestamp overflow in webapp event repository#3292
Labels
bugSomething isn't working

Description

@nicktrn
nicktrn
opened on Mar 29, 2026

Several places in the webapp multiply epoch milliseconds by 1,000,000 before converting to BigInt, which causes IEEE 754 precision loss (~256ns errors in ~0.2% of cases). The result exceeds Number.MAX_SAFE_INTEGER (~9e15) since epoch-ms * 1e6 is ~1.7e18.

The correct pattern is BigInt(ms) * BigInt(1_000_000) (multiplication after BigInt conversion). This already exists in convertDateToNanoseconds() in the same file - the buggy functions just didn't use it.

Affected locations:

  1. apps/webapp/app/v3/eventRepository/common.server.ts:24 - getNowInNanoseconds()

    // Bug: multiplication in float-land
return BigInt(new Date().getTime() * 1_000_000);
// Fix:
return BigInt(new Date().getTime()) * BigInt(1_000_000);

  2. apps/webapp/app/v3/eventRepository/common.server.ts:38 - calculateDurationFromStart()

    // Bug:
const duration = Number(BigInt($endtime.getTime() * 1_000_000) - startTime);
// Fix:
const duration = Number(BigInt($endtime.getTime()) * BigInt(1_000_000) - startTime);

  3. apps/webapp/app/v3/eventRepository/index.server.ts:217 - recordRunDebugLog()

  4. apps/webapp/app/v3/runEngineHandlers.server.ts:431 - retry event recording

Impact: Low. The errors are +-256 nanoseconds, which is unlikely to cause visible issues. But it's an easy fix and prevents any edge cases with span ordering.

Context: Found during compute workload manager PR review (#3114). The supervisor's msToNano() was fixed in that PR using a split approach that also preserves sub-ms precision from performance.now() arithmetic.

Linear: TRI-8269

Activity

  1. nicktrn
    added
    bugSomething isn't working
    on Mar 29, 2026
  2. linear
    closed this as duplicateon Mar 29, 2026
  3. nicktrn
    reopened this on Mar 29, 2026
  4. nielskaspers
    added a commit that references this issue on Mar 29, 2026

    fix: convert to BigInt before multiplying to avoid nanosecond precisi…

    8176a6b
  5. Yanhu007
    added a commit that references this issue on Apr 12, 2026

    fix: convert epoch-ms to BigInt before nanosecond multiplication

    26de74f
  6. aayushbaluni
    added 2 commits that reference this issue on Apr 15, 2026

    fix: prevent IEEE 754 precision loss in OTLP nanosecond timestamps

    170c1f3

    fix: convert to BigInt before nanosecond multiplication in all instances

    65e5e38
  7. avasis-ai
    added a commit that references this issue on Apr 15, 2026

    fix: OTLP nanosecond timestamp overflow from IEEE 754 precision loss

    c10ca6d

  8. aayushbaluni commented on Apr 15, 2026

    @aayushbaluni
    aayushbaluni
    on Apr 15, 2026

    I've submitted PR #3381 fixing all instances of the BigInt overflow pattern, including the one at runEngineHandlers.server.ts:432 that was missed in the previous PR #3378. However, the PR was auto-closed because I need to be vouched as a contributor. Could a maintainer please help with that?

  9. deepshekhardas
    added a commit that references this issue on Jun 17, 2026

    fix: prevent nanosecond overflow in OTLP timestamps

    8a26457
  10. okxint
    added a commit that references this issue on Jul 30, 2026

    fix(webapp): fix BigInt nanosecond precision loss in retry event star…

    2ab0c81

  11. AkhilTrivediX commented on Aug 4, 2026

    @AkhilTrivediX
    AkhilTrivediX
    on Aug 4, 2026

    Hi! I've written a fix for this precision loss issue and submitted PR #4499. However, the PR was automatically closed because I am not on the vouched contributors list yet. Could a maintainer please vouch for me so I can reopen the PR? Thank you!

  12. robinthedev-a11y commented on Sep 9, 2026

    @robinthedev-a11y
    robinthedev-a11y
    on Sep 9, 2026

    Hi — noticed this is still open and the buggy BigInt(ms * 1e6) pattern is still on main in eventRepository/common.server.ts. Prior spam PRs were closed; no open claim.

    Happy to ship a clean TS fix: switch all ms→ns sites to BigInt(ms) * BigInt(1_000_000) (matching convertDateToNanoseconds), plus a small regression test.

    If useful, could you /bounty $125 on this issue via Algora? I’ll exclusive-PR within 48–72h as @robinthedev-a11y.

    Happy to adjust amount/scope. Thanks for the OSS + past Algora payouts.

  13. bilalhusain-dev commented on Oct 12, 2026

    @bilalhusain-dev
    bilalhusain-dev
    on Oct 12, 2026

    🔍 Root Cause Analysis & Complete Fix across all 4 Call Sites

    Hey team,

    I investigated the OTLP nanosecond timestamp precision loss on the current main branch.

    Why this happens

    In JavaScript, standard numbers are IEEE 754 64-bit floats with Number.MAX_SAFE_INTEGER = 9,007,199,254,740,991 (9e15). An epoch millisecond timestamp (1.78e12) multiplied by 1_000_000 produces ~1.78e18, which far exceeds Number.MAX_SAFE_INTEGER. Multiplying prior to BigInt conversion causes truncation/rounding errors (±256ns) in ~0.2% of timestamps.

    The safe pattern (already used in convertDateToNanoseconds) is to cast to BigInt first:
    BigInt(ms) * BigInt(1_000_000)

    Proposed Fix (All 4 Affected Locations)

    diff --git a/apps/webapp/app/v3/eventRepository/common.server.ts b/apps/webapp/app/v3/eventRepository/common.server.ts
index e23a45b..b89c102 100644
--- a/apps/webapp/app/v3/eventRepository/common.server.ts
+++ b/apps/webapp/app/v3/eventRepository/common.server.ts
@@ -24,3 +24,3 @@ export function getNowInNanoseconds(): bigint {
-  return BigInt(new Date().getTime() * 1_000_000);
+  return BigInt(new Date().getTime()) * BigInt(1_000_000);
 }
@@ -38,3 +38,3 @@ export function calculateDurationFromStart(
-  const duration = Number(BigInt($endtime.getTime() * 1_000_000) - startTime);
+  const duration = Number(BigInt($endtime.getTime()) * BigInt(1_000_000) - startTime);

diff --git a/apps/webapp/app/v3/eventRepository/index.server.ts b/apps/webapp/app/v3/eventRepository/index.server.ts
index c45d12a..d78e341 100644
--- a/apps/webapp/app/v3/eventRepository/index.server.ts
+++ b/apps/webapp/app/v3/eventRepository/index.server.ts
@@ -211,3 +211,3 @@ export async function recordRunDebugLog({
-      startTime: BigInt((startTime?.getTime() ?? Date.now()) * 1_000_000),
+      startTime: BigInt(startTime?.getTime() ?? Date.now()) * BigInt(1_000_000),

diff --git a/apps/webapp/app/v3/runEngineHandlers.server.ts b/apps/webapp/app/v3/runEngineHandlers.server.ts
index a12bc34..f56ef89 100644
--- a/apps/webapp/app/v3/runEngineHandlers.server.ts
+++ b/apps/webapp/app/v3/runEngineHandlers.server.ts
@@ -560,3 +560,3 @@ export async function recordRetryEvent({
-          startTime: BigInt(time.getTime() * 1000000),
+          startTime: BigInt(time.getTime()) * BigInt(1_000_000),
  14. oziass
    added a commit that references this issue on Oct 14, 2026

    fix(webapp): convert epoch ms to BigInt before nanosecond multiplication

    33f68e7

  15. oziass commented on Oct 14, 2026

    @oziass
    oziass
    on Oct 14, 2026

    /attempt #3292

    Hey team — I put together a fix and colocated unit tests for this in PR #5027 (auto-closed by the vouch bot as a first-time contributor here):

    • Updated getNowInNanoseconds and calculateDurationFromStart in common.server.ts to use convertDateToNanoseconds so epoch ms is converted to BigInt before multiplying by 1_000_000.
    • Updated the remaining BigInt(ms * 1_000_000) sites in index.server.ts and
      unEngineHandlers.server.ts to BigInt(ms) * BigInt(1_000_000).
    • Added real-clock unit tests in �pps/webapp/app/v3/eventRepository/common.server.test.ts (and resolved both Devin review notes in 33f68e7).

    Could a maintainer vouch @oziass so #5027 can be reopened? Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions