A doctor sees the wrong price. The patient waits. The team gets blamed. Here's how I fixed the root cause.

The Problem Nobody Talks About

It started with a support ticket.

A doctor at one of our clinics had finished a consultation and opened the billing screen. The tariff for a blood test had been updated in the central lab system two days ago. But the clinic's software still showed the old price.

The front desk had to call the lab. The lab blamed the software. The software blamed the sync. And the patient sat there waiting.

This is the kind of bug that doesn't throw an exception. It doesn't show up in your error logs. It just quietly erodes trust — one wrong number at a time.

Our platform connects multiple clinics to a central lab system. Each clinic manages hundreds of clients. Each client has test tariffs that need to stay in sync with a master record. The stakes are real: wrong tariffs mean wrong bills, disputes, and in a healthcare context, compliance issues.

We had a cron job. It ran every night. It "synced everything."

Until it didn't.

Why the Cron Job Failed Us

The cron job worked fine in staging. It worked fine in the first month of production. Then the edge cases showed up:

What if a doctor manually updates a client's tariff mid-day and needs it reflected immediately? The cron job said: wait until midnight.

What if the sync job crashes halfway through 300 clients? The cron job said: nothing — it just silently moved on.

What if the server is already under heavy load when the job fires? The cron job said: run anyway, make things worse.

What if someone accidentally disables the scheduler and nobody notices for a week? The cron job said: no record of this ever happening.

Every one of these scenarios happened in production. Each one eroded confidence in the system. I needed something that could handle the happy path and survive the chaos.

Rethinking the Architecture

The insight that changed everything: the problem wasn't that sync was slow. The problem was that nobody knew when sync had failed, or why, or for how long.

So I rebuilt it around three principles:

1. Every sync request is a first-class record — not a fire-and-forget task, but a tracked entity with a status, a priority, a retry count, and an owner.

2. The system must recover itself — no human should need to intervene when something stalls.

3. The system must know when to do nothing — an overloaded server running a sync job is worse than a delayed sync.

Here's what I built.

The Architecture

1. The Priority Queue

Every sync request — whether triggered manually by a doctor or scheduled automatically — gets enqueued as a record with two key fields: a priority number and a timestamp.

Manual triggers from the frontend get high priority. A doctor can't wait until midnight. Scheduled background jobs run at normal priority and fill the gaps.

When it's time to dequeue, the rule is simple:

• Pick the highest priority first

• Tie? Oldest one wins — pure FIFO

This means urgent requests always jump the queue, but nothing starves indefinitely.

// Manual sync from frontend — HIGH priority
await dbLabAPI.EnqueueSyncRequestAsync(clinicId, clientCode, priority: High);

// Scheduled background sync — Normal priority
await dbLabAPI.EnqueueScheduledSyncAsync(); // normal priority, all clinics

2. Stale Lock Recovery

This is the piece most background job implementations skip — and it's the one that bites you hardest.

When a sync job picks up a request, it marks it as Processing. But what if the job crashes? Or the pod restarts? The record stays Processing forever. Nobody picks it up again. The sync silently never happened.

Every cycle, before doing anything else, the background service checks for stale locks:

await dbLabAPI.RecoverStaleLocksAsync(staleAfterMinutes: 30);

The logic:

• If a request has been Processing for more than 30 minutes → something went wrong

• Reset it to Pending, increment retryCount

• If retryCount hits 3 → mark as PermanentlyFailed and stop retrying

Processing (>30 min) → retryCount < 3 → Pending (retryCount++)
Processing (>30 min) → retryCount = 3 → PermanentlyFailed

PermanentlyFailed is important. It means the record is still there, visible, queryable. It's not lost. You can investigate it, alert on it, and manually retry it if needed. Nothing disappears silently.

3. The Automatic Fallback

Manual syncs cover urgent cases. But what about the normal flow — a tariff update that no doctor noticed, that just needs to propagate overnight?

The background service checks: has any sync been triggered manually today?

If not, it automatically enqueues all clients for all clinics at 06:00 and 18:00.

private static readonly TimeSpan[] ScheduledSyncTimes = {
    new TimeSpan(6, 0, 0),
    new TimeSpan(18, 0, 0),
};

This is a safety net, not the primary mechanism. It ensures that even if the frontend is never touched, data stays fresh within half a day. The 12-hour window was chosen based on how frequently tariffs actually change in our domain — tune this for your own context.

4. The CPU Gate

This one saved us from a cascading failure.

Before the service dequeues anything, it checks two resource metrics:

System-wide CPU (via /proc/stat on Linux):

var systemCpu = await GetCpuUsageAsync();
if (systemCpu > 75.0) {
    _logger.LogWarning("System CPU at {Cpu:F1}% — skipping cycle.", systemCpu);
    return;
}

Process-level CPU (via Process.TotalProcessorTime):

var processCpu = await GetProcessCpuUsageAsync();
if (processCpu > 50.0) {
    _logger.LogWarning("Process CPU at {Cpu:F1}% — skipping cycle.", processCpu);
    return;
}

Why two separate checks?

• System CPU catches when the host is struggling — maybe the database is hammering the CPU, or another service is spiking. No point adding more load.

• Process CPU catches when our own service is already busy — maybe a previous job is still running cleanup tasks. This is a secondary guard.

Both checks sample over a 500ms window, not instantaneously — a single spike shouldn't block the entire queue.

The thresholds I settled on after tuning:

5. Dequeue and Execute

Once all checks pass, the service dequeues the next request and calls the sync API:

var request = await dbLabAPI.DequeueNextAsync();

var syncResponse = await dbLabAPI.SyncAttuneTestMasterTariff(
    request.ClinicId,
    request.ClientCode
);

var status = syncResponse.Result
    ? QueueRequestStatus.Completed
    : QueueRequestStatus.Failed;

await dbLabAPI.UpdateSyncRequestStatusAsync(request.Id, status, syncResponse.Message);

Notice the separation: the sync API is just an API call. It doesn't know anything about queues, priorities, or retries. The background service owns all of that complexity. This keeps each layer focused.

6. DisallowConcurrentExecution

The entire job runs inside a Quartz.NET IJob with one critical attribute:

[DisallowConcurrentExecution]
public class BackgroundSyncClientMaster : IJob

No matter how frequently Quartz fires the trigger, only one instance runs at a time. Combined with the process CPU check, this is a hard guarantee against runaway concurrency.

The Full Flow

Frontend: Sync Client / Sync All Clients
         ↓
[HIGH PRIORITY] → Enqueue request (Pending)
         ↓
Background Service (every N seconds):
  1. RecoverStaleLocksAsync()        ← rescue stuck jobs
  2. TryEnqueueScheduledSyncAsync()  ← auto-fill if no manual sync today
  3. Check system CPU > 75%?         ← skip if yes
  4. Check process CPU > 50%?        ← skip if yes
  5. DequeueNextAsync()              ← highest priority, then FIFO
  6. Call SyncAttuneTestMasterTariff()
  7. Update status → Completed / Failed

What I'd Do Differently

Looking back, there are two things I'd add from day one:

A RAM check. CPU gates without memory gates are incomplete. A sync job that loads large payloads can cause GC pressure and OOM crashes that don't show up in CPU metrics. Add a check against /proc/meminfo and skip if free memory drops below 15%.

Persistent last-sync timestamp. The _lastScheduledSync field is currently static — it resets on every app restart. If the service restarts at 07:00, it might re-trigger the 06:00 scheduled sync. Storing this in MongoDB or a distributed cache would make it restart-safe.

Key Takeaways

The cron job failed not because it was poorly written, but because it was designed only for the happy path. The rewrite was successful not because of clever algorithms, but because it was designed to fail gracefully.

A few principles that carry beyond this specific problem:

• Every async task is a record. If it doesn't have a status you can query, it doesn't exist when it fails.

• Recovery is not an afterthought. Stale lock recovery was the single highest-value addition to this system.

• Know when to do nothing. The CPU gate prevented two production incidents I can directly point to.

• Visibility over cleverness. PermanentlyFailed is more valuable than a retry loop that silently swallows errors.

The doctor's billing screen is now accurate. The patient doesn't wait anymore.

And the support tickets stopped.

Stack

• .NET 7 — background service

• Quartz.NET — job scheduling with [DisallowConcurrentExecution]

• MongoDB — sync request queue storage

• Linux /proc/stat — system CPU monitoring

Have questions about the queue implementation or the CPU sampling approach? Drop a comment below — happy to go deeper on any part of this.