Bifrost Market Fixes — Investigation & Resolution

Date: 2026-03-02 Branch: fix/bifrost-market-fixes Commit: da846674 Reported by: Michael (Bifrost Team)

---

Issues Reported

1. Potentially too aggressive recovery API usage
2. Market odds updates getting stuck on the membersite
3. Specific reproduction: South Africa odds changing 70→72→74→76→78→80→82 on Bifrost, but membersite stuck at 76, then eventually 80, then stuck at 80 when 82 was pushed

---

Investigation

Evidence

Image 1 — UI stuck at 80 when Bifrost had 82:

• South Africa: 80 (membersite) vs 82 (Bifrost API)
• New Zealand: 106 (stale)

Image 2 — RabbitMQ queue spike:

• forsyt.market-book.queue spiked to ~5,000 messages in one minute
• Publish rate peaked at ~750 msg/s
• Consumer ACK rate: ~63/s
• Queue drained to 0 over ~80 seconds

Root Cause Chain

Periodic recovery runs every 10 minutes
  → recoverAllMarkets({ includeMarketBook: true })
  → Bifrost floods forsyt.market-book.queue with ALL current market states
  → Queue spikes to ~5,000 messages (image 2)
  → Consumer drains the entire backlog in rapid succession
  → 70→72→74→76→78→80 all arrive within milliseconds of each other
  → Leading-edge throttle fires at 76 → publishes to frontend
  → 78, 80 arrive within the 100ms throttle window → dropped with no retry
  → Frontend stuck at 76 until the next real-time update (80) arrives
  → Same pattern causes 82 not to appear after 80

Secondary Bug

Inside the marketUpdate event listener, the throttle check used return inside a for...of loop over fixtureIds. This exits the entire event listener function rather than continuing to the next fixture — silently skipping other fixtures when the first is throttled.

---

Files Changed

backend/src/exchanges/adapters/bifrost/BifrostRecoveryManager.ts

Fix: No market books on periodic recovery

includeMarketBook in recoverAllMarkets() and recoverPhantomEvents() is now a parameter instead of always true.

• trigger === 'initial' (startup/reconnect): includeMarketBook = true — need full snapshot
• trigger === 'periodic' (every 10 min): includeMarketBook = false — market books stream in real-time via RabbitMQ push, no need to request them again

This eliminates the periodic queue flood entirely. The 10-minute periodic cycle now only refreshes event catalogues (new fixtures, cancelled events), which is its primary purpose.

// Before:
await this.recoverAllMarkets(); // always includeMarketBook: true

// After:
const includeMarketBook = trigger === 'initial';
await this.recoverAllMarkets(includeMarketBook);

Signature changes:

• recoverAllMarkets() → recoverAllMarkets(includeMarketBook: boolean)
• recoverPhantomEvents() → recoverPhantomEvents(includeMarketBook: boolean)

---

backend/src/exchanges/adapters/bifrost/BifrostAdapter.ts

Fix 1: Trailing-edge throttle

The old throttle was leading-edge only: fire immediately on the first update in the window, drop everything else. With a queue flood, "drop" meant the latest price was permanently lost.

The new pattern is leading edge + one scheduled trailing-edge timer per window:

Update arrives → within throttle window?
  NO  → publish immediately, cancel any pending timer
  YES → schedule ONE timer (if not already scheduled) to publish after window expires

When the timer fires, it calls publishOddsForFixture() which reads the current cache state at that moment — by then the cache has processed all queued versions and holds the latest price.

// Before (leading-edge only):
if (now - lastPublish < throttleMs) return; // drops update permanently

// After (leading + trailing edge):
if (now - lastPublish < throttleMs) {
  if (!this.pendingThrottleTimers.has(fixtureId)) {
    const delay = throttleMs - (now - lastPublish);
    const timer = setTimeout(() => {
      this.pendingThrottleTimers.delete(fixtureId);
      this.publishOddsForFixture(resolvedEventId, fixtureId);
    }, delay);
    this.pendingThrottleTimers.set(fixtureId, timer);
  }
  continue; // not return
}
// Cancel pending trailing timer if we're firing now (no longer needed)

Fix 2: return → continue in fixtureIds loop

The throttle check used return inside for (const fixtureId of fixtureIds), which exits the entire event listener. Changed to continue so remaining fixtures in the loop are still processed.

Fix 3: Extracted publishOddsForFixture() helper

Both the immediate and trailing-edge paths share one private method that always reads fresh data from cache at call time:

private publishOddsForFixture(resolvedEventId: string, fixtureId: string): void {
  const markets = this.buildMarketDeltaForEvent(resolvedEventId, fixtureId);
  const payload = JSON.stringify({
    fixtureId,
    source: 3,
    timestamp: Date.now(), // fresh timestamp on every call
    markets: markets.length > 0 ? markets : undefined,
  });
  redis.publish('odds:updated', payload).catch(...);
}

Fix 4: Cleanup of pending timers

Added pendingThrottleTimers: Map<string, NodeJS.Timeout> to class fields. All pending timers are cleared in:

• shutdown() — on adapter shutdown
• Periodic cleanup interval — orphaned timers with no matching throttle entry

---

Behaviour After Fix

Scenario	Before	After
Periodic recovery (every 10 min)	Floods queue with all market books	Only refreshes catalogues
Queue spike from recovery	~5,000 messages, ~80s to drain	No spike
Price 76→78→80 in rapid burst	Frontend stuck at 76	76 published immediately, 78/80 published at end of throttle window
Price change after quiet period	Works correctly	Works correctly (unchanged)
Multiple fixtureIds, first throttled	All fixtures silently skipped	Only first fixture skipped, rest processed

---

What Was NOT Changed

• Recovery interval (10 minutes) — unchanged, no need to reduce frequency
• Throttle window values (100ms in-play, 500ms pre-play) — unchanged
• Queue prefetch settings — unchanged (handler is synchronous, prefetch is not the bottleneck)
• Any frontend code — issue was entirely in backend pub/sub pipeline
• Financial queues (bets.snapshot, bets.outcomes) — unaffected

---

Cross-System Impact

System	Impact
backend	BifrostAdapter.ts, BifrostRecoveryManager.ts — 2 files
frontend	Receives correct real-time prices (consumer of fix, no code changes)
services	None
infra	RabbitMQ forsyt.market-book.queue no longer spikes every 10 minutes

---

Testing Checklist

• Deploy to bhdev/staging
• Verify RabbitMQ forsyt.market-book.queue stays flat during periodic recovery (no spike at 10-minute mark)
• Manually update odds on a live market — verify membersite reflects the change within 100–500ms
• Confirm initial startup still loads all market books correctly (cold start with includeMarketBook=true)
• Restart backend mid-match — verify odds hydrate from Redis cache and queue correctly