Skip to main content

Bifrost Integration Chat — 2026-03-13 (Friday)

Channel: #forsyt-bm-integration (Suntech Innovation Slack)

Bhargav — 10:41 AM

Hey @Michael Cheremuhin

For voiding and resettlement, All flows are implemented and deployed:

  1. Bet-level void/unvoid (pre and post-settlement) — via BetSnapshot
  2. Market-level unsettlement — via MarketBook status change (SETTLED → non-SETTLED)
  3. Resettlement — via new BetOutcome at higher version with different outcome
  4. Market-level unsettlement also depends on seeing the SETTLED statusvfirst (to detect the transition to non-SETTLED). If our service restarted between those two MarketBook messages, the previous status isn't cached.
  5. On the fancy line, we lock the line at click time so it shouldn't change.

Thread (1 replies):

Michael Cheremuhin — 2:48 PM hi @Bhargav. thanks for the update, passing further

Michael Cheremuhin — 3:12 PM

@Bhargav something regarding login options. will we be able to use email login (not gmail) in prod?

Thread (1 replies):

Bhargav — 3:12 PM Yes

Michael Cheremuhin — 5:53 PM

@Bhargav seems like updates started to apply with delay or are getting skipped, can you please check. also post settlement voiding doesn't seem to work

Thread (20 replies):

Bhargav — 5:56 PM What updates? Odds?

Bhargav — 6:14 PM found and fixed the 2nd part of the issue. Post-settlement voiding, unsettlement, and resettlement should all work now on bhdev. Can you re-test?

Michael Cheremuhin — 6:38 PM @Bhargav can you please remind your usage of recovery flow. seems like you are ddosing it which causes tons of extra unneeded data to be requested and it's causing the delay on odds

Bhargav — 7:04 PM We use Bifrost recovery APIs only on startup — 3 calls total (events, then markets with market books, then phantom sub-events). No periodic recovery.

We restarted the container ~45 min ago for a fix, which triggered the startup recovery requesting market books for all cached events (~3600 IDs). That's likely what caused the extra load. It's a one-time burst, not ongoing.

Michael Cheremuhin — 7:10 PM so you dont do that for normal work, just for service restart/recovery?

Bhargav — 7:15 PM Yes, only on service restart/recovery. During normal operation we receive everything through the RabbitMQ queues. No recovery API calls for now.

Bhargav — 7:31 PM We wanted to test the full flow with queues as you were little reluctant for us to use in the happy flow.

We can enable the recovery now, as per your suggestion and business requiremnets based on our internal testing.

Michael Cheremuhin — 7:33 PM can't get your point what actions are needed from our side

Bhargav — 7:37 PM General recommendation on when to use your recovery API.

Bhargav — 7:38 PM But I'm concerned more on the DDoS style handling of updates on our server. Do we know any other reasons on why this would happen?

Michael Cheremuhin — 7:48 PM from what we observed we were receiving tons of recovery request from you, each triggers messages back to you

Michael Cheremuhin — 7:48 PM if you request 3k markets, you will receive 3k messages

Michael Cheremuhin — 7:49 PM that's why I'm saying that it's a "recovery" api, which should not be used if there is any way to avoid it

Bhargav — 8:18 PM You're right — our startup recovery sent all ~3600 cached event IDs with market books in one call, which triggered thousands of messages back. That was the flood. How would you recommend we handle recovery on restart? Any best practice on scoping the request (e.g. only active events, specific time window)?

Michael Cheremuhin — 8:19 PM can you please remind why are you doing it overall? because you have only inmemory part of the cache that you need to fill in?

Bhargav — 8:21 PM Yeah.. We have only in-memory store of events.

Bhargav — 8:24 PM Actually we have Redis-backed persistence too — on startup we hydrate from Redis first (events, catalogues, market books, all mappings). The recovery call was meant for cold starts when Redis is empty, but currently it fires every restart even when Redis already has the data. Probably that's not needed. We'll skip recovery when Redis hydration is sufficient.

Michael Cheremuhin — 8:24 PM i think you need to have some endpoint to trigger the cold start, but for normal app restarts you shouldnt do any recovery

Michael Cheremuhin — 8:25 PM given that fresh updates will sit in rabbitmq, you only need to keep your current context. streaming recovery should not be used as a replacement for context

Bhargav — 2:26 PM @Michael Cheremuhin We've made two changes:

  1. Normal restarts no longer trigger recovery. If Redis cache has data, we skip recovery entirely and rely on RabbitMQ queue catch-up
  2. Recovery is now admin-only via dashboard buttons. We have two tiers:
    • Selective: recovers only active fixtures (open bets + live matches). ~34 event IDs today, completed in 8 seconds.
    • Full: events + all markets + phantom sub-events. ~6300 event IDs, for cold starts only.

We just tested both on our dev environment. Both worked cleanly.

Michael Cheremuhin — 8:37 PM

post settlement looks wrong. If i settle a loser, and then void it, it turns it into a winner instead of a 0 bet if i settle as a winner, then void it, it turns it into a loser instead of a 0 bet

Thread (3 replies):

Michael Cheremuhin — 8:42 PM post settlement voiding worked this time. maybe a timing thing

Michael Cheremuhin — 8:46 PM looks rather good

Bhargav — 2:29 PM Can you confirm if you say any issues for any reason here? We want to be 100% sure.