Batching and Polling

Process large datasets efficiently with controlled batching and monitor long-running operations with polling strategies.

Batch Processing
Polling Strategies
Real-World Patterns
Performance Optimization
Error Handling in Batches
Progress Tracking

Batch Processing

The Problem: Processing Large Datasets

// DON'T: Process 10,000 payments all at once
async function processPayments(payments: Payment[]) {
  return Promise.all(payments.map(p => stripe.charges.create(p)))
  // Problems:
  // - Memory explosion
  // - API rate limits hit
  // - All-or-nothing failure
  // - No progress visibility
}

The Solution: Controlled Batching

import { batch, mapAsync } from '@ontologyio/receta/async'

async function processPayments(payments: Payment[]) {
  return batch(payments, (group) =>
    // batch hands over 100 payments at a time, mapAsync limits requests within them
    mapAsync(group, (payment) => stripe.charges.create({
      amount: payment.amount,
      currency: payment.currency,
      customer: payment.customerId
    }), { concurrency: 10 }), // 10 concurrent requests per batch
  {
    batchSize: 100,            // Process 100 at a time
    delayBetweenBatches: 1000  // Wait 1s between batches
  })
}

Batch Function Overview

Basic Usage

fn runs once per batch and receives the whole batch, which suits bulk APIs. You get one result per batch:

import { batch } from '@ontologyio/receta/async'

// Simple batching
const results = await batch(
  items,
  async (group) => saveAll(group),
  { batchSize: 50 }
)

// Several batches at once
const results = await batch(
  items,
  async (group) => saveAll(group),
  {
    batchSize: 100,
    concurrency: 5
  }
)

// A pause between batches
const results = await batch(
  items,
  async (group) => saveAll(group),
  {
    batchSize: 100,
    delayBetweenBatches: 500
  }
)

// batch has no progress callbacks, so count inside fn
let completed = 0

const results = await batch(
  items,
  async (group) => {
    const saved = await saveAll(group)
    completed += group.length
    console.log(`Progress: ${completed}/${items.length} (${(completed/items.length*100).toFixed(1)}%)`)
    return saved
  },
  { batchSize: 100 }
)

Type Signature

Signature

function batch<T, U>(
  items: readonly T[],
  fn: (batch: readonly T[]) => Promise<U>,
  options?: BatchOptions
): Promise<Result<U[], BatchError>>

interface BatchOptions {
  batchSize?: number           // Items per batch (default: 10)
  concurrency?: number         // Batches running at once (default: 1)
  delayBetweenBatches?: number // Delay between batches in ms (default: 0)
}

interface BatchError {
  type: 'batch_error'
  batchIndex: number
  itemsInBatch: number
  underlyingError: unknown
}

A delayBetweenBatches above 0 runs batches one at a time and ignores concurrency. batchIndex is -1 when the failed batch can't be identified: with a delay, or when concurrency covers every batch.

Chunk Utility

Breaking Arrays into Batches

import { chunk } from '@ontologyio/receta/async'

// Split array into chunks
const items = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
const batches = chunk(items, 3)
// => [[1, 2, 3], [4, 5, 6], [7, 8, 9], [10]]

// Process each chunk manually
for (const batch of batches) {
  await processBatch(batch)
  await sleep(1000) // Delay between batches
}

Combining with Remeda

import * as R from 'remeda'
import { chunk, mapAsync } from '@ontologyio/receta/async'

// Process in chunks with transformation
const results = await R.pipe(
  largeDataset,
  chunk(100),
  (batches) =>
    mapAsync(
      batches,
      (group) => mapAsync(group, processItem), // Items in a chunk run together
      { concurrency: 1 }                       // Chunks run one at a time
    )
)

Batch Size Optimization

Finding the Right Batch Size

// Rule of thumb: Start with these defaults and adjust
const defaultBatchSizes = {
  // API Calls
  externalApi: 100,      // Most APIs handle 100/batch well
  database: 1000,        // Databases can handle larger batches
  emailSending: 50,      // Email services often rate-limit

  // Data Processing
  imageProcessing: 10,   // CPU/memory intensive
  fileUploads: 5,        // I/O bound operations
  dataTransform: 5000,   // Pure computation can be large
}

// Adaptive batching based on processing time
async function adaptiveBatch<T, U>(
  items: T[],
  fn: (item: T) => Promise<U>
): Promise<U[]> {
  let batchSize = 100
  const results: U[] = []

  for (let i = 0; i < items.length; i += batchSize) {
    const batchItems = items.slice(i, i + batchSize)
    const startTime = Date.now()

    const batchResults = await Promise.all(batchItems.map(fn))
    results.push(...batchResults)

    const duration = Date.now() - startTime

    // Adjust batch size based on duration
    if (duration < 1000) {
      batchSize = Math.min(batchSize * 1.5, 500) // Increase
    } else if (duration > 5000) {
      batchSize = Math.max(batchSize * 0.5, 10)  // Decrease
    }
  }

  return results
}

Delays Between Batches

Rate Limit Compliance

import { batch, mapAsync } from '@ontologyio/receta/async'

// Stripe: 100 requests per second
async function batchStripeCharges(charges: ChargeRequest[]) {
  return batch(charges,
    (group) => mapAsync(
      group,
      (charge) => stripe.charges.create(charge),
      { concurrency: 10 }
    ),
    {
      batchSize: 100,
      delayBetweenBatches: 1000  // 1 second delay = 100 req/sec
    }
  )
}

// SendGrid: 10,000 emails per hour
async function batchSendEmails(emails: Email[]) {
  const perBatch = 100
  const batchesPerHour = 10000 / perBatch // 100 batches
  const delayBetweenBatches = (60 * 60 * 1000) / batchesPerHour // ~36 seconds

  return batch(emails,
    async (group) => {
      const sent = await sendgrid.send(group)
      console.log(`Batch complete. Next batch in ${delayBetweenBatches}ms`)
      return sent
    },
    {
      batchSize: perBatch,
      delayBetweenBatches
    }
  )
}

Exponential Backoff Between Batches

async function batchWithBackoff<T, U>(
  items: T[],
  fn: (item: T) => Promise<U>,
  batchSize: number
): Promise<U[]> {
  const batches = chunk(items, batchSize)
  const results: U[] = []

  for (const [index, batch] of batches.entries()) {
    const delay = Math.min(1000 * Math.pow(2, index), 30000)
    if (index > 0) {
      await sleep(delay)
    }

    const batchResults = await Promise.all(batch.map(fn))
    results.push(...batchResults)
  }

  return results
}

Polling Strategies

Poll Function Overview

import { poll } from '@ontologyio/receta/async'

// Poll until condition is met
const result = await poll(
  async () => {
    const job = await api.getJob(jobId)
    return job.status === 'completed' ? job : null
  },
  {
    interval: 1000,   // Check every 1 second
    timeout: 60000,   // Give up after 60 seconds
    maxAttempts: 100, // Or after 100 attempts
  }
)

Type Signature

Signature

function poll<T>(
  fn: () => Promise<T | null | undefined | false>,
  options?: PollOptions
): Promise<Result<T, PollError>>

interface PollOptions {
  interval?: number      // Time between checks (default: 1000)
  maxAttempts?: number   // Max number of attempts (default: 10)
  timeout?: number       // Max total time (no limit by default)
  shouldContinue?: (attempt: number) => boolean
  onPoll?: (attempt: number) => void
}

type PollError =
  | { type: 'max_attempts'; attempts: number }
  | { type: 'timeout'; elapsed: number }
  | { type: 'stopped'; attempt: number }

null, undefined and false mean "not yet". Anything else, including 0 and '', stops polling and becomes the Ok value.
If fn throws, poll rejects instead of returning an Err. Return failures as values, as the examples below do.
maxAttempts defaults to 10, so raise it along with a long timeout, or the attempts run out first.
The wait is interval every time. There's no backoff.
After a timeout, poll returns, but the loop keeps calling fn until it runs out of attempts.

Working with PollError

poll returns a Result, so you can tell the failure modes apart without exceptions:

Real-World: CI/CD Job Monitoring with Result

import { poll } from '@ontologyio/receta/async'
import { isErr } from '@ontologyio/receta/result'

type JobStatus = {
  id: string
  status: 'pending' | 'running' | 'success' | 'failed'
  result?: string
  error?: string
}

const monitorCIJob = async (jobId: string) => {
  const result = await poll(
    async () => {
      const response = await fetch(`/api/jobs/${jobId}`)
      const job: JobStatus = await response.json()

      if (job.status === 'success' || job.status === 'failed') {
        return job
      }
      return null // Keep polling
    },
    {
      interval: 5000,
      maxAttempts: 60, // 5 minutes max
      timeout: 300000,
      onPoll: (attempt) => {
        console.log(`Checking job ${jobId} (attempt ${attempt})`)
      }
    }
  )

  // Handle different error types
  if (isErr(result)) {
    if (result.error.type === 'timeout') {
      return { status: 'timeout', message: `Job timed out after ${result.error.elapsed}ms` }
    }
    if (result.error.type === 'max_attempts') {
      return { status: 'failed', message: `Max polling attempts (${result.error.attempts}) reached` }
    }
    return { status: 'stopped', message: 'Polling stopped' }
  }

  return { status: result.value.status, result: result.value.result }
}

Real-World: Order Processing with Result

import { poll, retry } from '@ontologyio/receta/async'
import { type Result, isErr, mapErr } from '@ontologyio/receta/result'
import * as R from 'remeda'

type Order = { id: string; status: 'processing' | 'complete'; total: number }

const waitForOrder = async (orderId: string): Promise<Result<Order, string>> => {
  // Poll with retry on network failures
  const pollRes = await poll(
    async () => {
      const res = await retry(
        () => fetch(`/api/orders/${orderId}`).then(r => r.json()),
        { maxAttempts: 2, delay: 500 }
      )

      if (isErr(res)) return null // Retry polling

      const order = res.value as Order
      return order.status === 'complete' ? order : null
    },
    { interval: 2000, maxAttempts: 30 }
  )

  return R.pipe(
    pollRes,
    mapErr(error => {
      if (error.type === 'max_attempts') {
        return `Order ${orderId} processing timed out`
      }
      return `Order polling failed: ${error.type}`
    })
  )
}

Benefits

✅ Discriminated error types - Know exactly why polling failed ✅ No exceptions - Pure functional polling ✅ Composable - Chain with other Result operations ✅ Full context - Access attempts, elapsed time, etc.

Polling Patterns

Job Status Polling

import { poll, type PollError } from '@ontologyio/receta/async'
import { type Result, err, flatten, ok } from '@ontologyio/receta/result'

// AWS Batch job monitoring
async function waitForBatchJob(
  jobId: string
): Promise<Result<JobResult, PollError | string>> {
  const finished = await poll(
    async () => {
      const job = await batchClient.describeJobs({ jobs: [jobId] })
      const status = job.jobs[0]?.status

      if (status === 'SUCCEEDED') {
        return ok({ jobId, status, output: job.jobs[0].container.output })
      }
      if (status === 'FAILED') {
        return err(`Job failed: ${job.jobs[0].statusReason}`)
      }

      return null // Still running
    },
    {
      interval: 5000,    // Check every 5 seconds
      maxAttempts: 720,  // Enough attempts to reach the timeout
      timeout: 3600000,  // 1 hour timeout
      onPoll: (attempt) => {
        console.log(`Checking job status (attempt ${attempt})...`)
      }
    }
  )

  return flatten(finished)
}

CI/CD Pipeline Monitoring

// GitHub Actions workflow polling
async function waitForWorkflow(
  owner: string,
  repo: string,
  runId: number
): Promise<Result<WorkflowRun, PollError>> {
  return poll(
    async () => {
      const { data } = await octokit.actions.getWorkflowRun({
        owner,
        repo,
        run_id: runId
      })

      if (data.status === 'completed') {
        return data
      }

      return null // Still in progress
    },
    {
      interval: 10000,   // Check every 10 seconds
      maxAttempts: 180,  // Enough attempts to reach the timeout
      timeout: 1800000,  // 30 minute timeout
      onPoll: (attempt) => {
        if (attempt % 6 === 0) { // Log every minute
          console.log(`Workflow still running (${attempt * 10}s elapsed)...`)
        }
      }
    }
  )
}

Payment Processing Status

// Stripe charge confirmation polling
async function waitForChargeConfirmation(
  chargeId: string
): Promise<Result<Charge, PollError | string>> {
  const finished = await poll(
    async () => {
      const charge = await stripe.charges.retrieve(chargeId)

      if (charge.status === 'succeeded') {
        return ok(charge)
      }
      if (charge.status === 'failed') {
        return err(`Charge failed: ${charge.failure_message}`)
      }

      return null // Still pending
    },
    {
      interval: 2000,
      maxAttempts: 30  // 60 seconds max
    }
  )

  return flatten(finished)
}

Combining Poll with Timeout

Time Limit Option

poll takes a timeout option, so there's no separate promise to race. Give it enough attempts that the timeout is what stops it:

import { poll, type PollError } from '@ontologyio/receta/async'

async function pollWithTimeout<T>(
  fn: () => Promise<T | null>,
  pollIntervalMs: number,
  timeoutMs: number
): Promise<Result<T, PollError>> {
  return poll(fn, {
    interval: pollIntervalMs,
    maxAttempts: Math.ceil(timeoutMs / pollIntervalMs) + 1,
    timeout: timeoutMs
  })
}

// Usage
const result = await pollWithTimeout(
  async () => {
    const status = await checkJobStatus(jobId)
    return status.complete ? status : null
  },
  1000,
  30000
)

Real-World Patterns

Pattern 1: Data Import/Export

// CSV export with batching and progress
async function exportUsersToCSV(
  filters: UserFilters
): Promise<Result<string, ExportError>> {
  const users = await db.user.findMany({ where: filters })

  if (users.length === 0) {
    return err({ type: 'no_data' })
  }

  const csvRows: string[] = ['id,email,name,created_at']

  const exported = await batch(users,
    async (group) => {
      const rows = group.map((user) => [
        user.id,
        user.email,
        user.name,
        user.createdAt.toISOString()
      ].join(','))

      csvRows.push(...rows)
      console.log(`Exported ${csvRows.length - 1}/${users.length} users`)
    },
    { batchSize: 1000 }
  )

  if (isErr(exported)) {
    return err({ type: 'export_failed' })
  }

  const csv = csvRows.join('\n')
  const filename = `users_export_${Date.now()}.csv`

  await fs.writeFile(filename, csv)
  return ok(filename)
}

Pattern 2: Email Sending Batches

// SendGrid bulk email with rate limiting
interface EmailCampaign {
  recipients: EmailAddress[]
  template: string
  subject: string
}

async function sendEmailCampaign(
  campaign: EmailCampaign
): Promise<Result<CampaignReport, SendError>> {
  const report: CampaignReport = {
    sent: 0,
    failed: 0,
    errors: []
  }

  await batch(campaign.recipients,
    async (group) => {
      const sent = await mapAsync(group, async (recipient) => {
        const result = await tryCatchAsync(async () => {
          await sendgrid.send({
            to: recipient.email,
            subject: campaign.subject,
            templateId: campaign.template,
            dynamicTemplateData: recipient.data
          })
        })

        if (isOk(result)) {
          report.sent++
        } else {
          report.failed++
          report.errors.push({
            recipient: recipient.email,
            error: result.error
          })
        }
      }, { concurrency: 10 })

      console.log(`Sent batch. Total sent: ${report.sent}`)
      return sent
    },
    {
      batchSize: 100,            // SendGrid batch limit
      delayBetweenBatches: 1000  // Rate limit compliance
    }
  )

  return ok(report)
}

Pattern 3: Database Migrations

// Backfill database records in batches
async function backfillUserScores(): Promise<Result<MigrationReport, MigrationError>> {
  const usersWithoutScores = await db.user.findMany({
    where: { score: null },
    select: { id: true }
  })

  console.log(`Found ${usersWithoutScores.length} users to backfill`)

  const report: MigrationReport = {
    processed: 0,
    failed: 0,
    errors: []
  }

  await batch(usersWithoutScores,
    async (group) => {
      const backfilled = await mapAsync(group, async (user) => {
        const result = await tryCatchAsync(async () => {
          const score = await calculateUserScore(user.id)
          await db.user.update({
            where: { id: user.id },
            data: { score }
          })
        })

        if (isOk(result)) {
          report.processed++
        } else {
          report.failed++
          report.errors.push({ userId: user.id, error: result.error })
        }
      }, { concurrency: 20 })

      const completed = report.processed + report.failed
      const percent = ((completed / usersWithoutScores.length) * 100).toFixed(1)
      console.log(`Migration progress: ${completed}/${usersWithoutScores.length} (${percent}%)`)

      return backfilled
    },
    {
      batchSize: 500,           // Database can handle larger batches
      delayBetweenBatches: 100  // Small delay to not overwhelm DB
    }
  )

  return ok(report)
}

Pattern 4: Order Processing

// Process pending orders in batches
interface OrderProcessor {
  processPendingOrders(): Promise<Result<ProcessingReport, ProcessingError>>
}

const orderProcessor: OrderProcessor = {
  processPendingOrders: async () => {
    const pendingOrders = await db.order.findMany({
      where: { status: 'pending' },
      include: { items: true, customer: true }
    })

    const report: ProcessingReport = {
      processed: 0,
      skipped: 0,
      failed: 0,
      totalRevenue: 0
    }

    await batch(pendingOrders,
      async (group) => {
        const handled = await mapAsync(group, async (order) => {
          // Check inventory
          const inventoryCheck = await checkInventory(order.items)
          if (!inventoryCheck.available) {
            report.skipped++
            return
          }

          // Process payment
          const paymentResult = await processPayment(order)
          if (isErr(paymentResult)) {
            report.failed++
            await db.order.update({
              where: { id: order.id },
              data: { status: 'failed', error: paymentResult.error }
            })
            return
          }

          // Update order
          await db.order.update({
            where: { id: order.id },
            data: { status: 'completed', processedAt: new Date() }
          })

          report.processed++
          report.totalRevenue += order.total
        }, { concurrency: 5 })

        console.log(`Processed: ${report.processed}, Failed: ${report.failed}, Skipped: ${report.skipped}`)
        return handled
      },
      {
        batchSize: 50,
        delayBetweenBatches: 500
      }
    )

    return ok(report)
  }
}

Performance Optimization

Memory Management

// DON'T: Load all results in memory
async function processLargeDataset(items: Item[]) {
  const results = await batch(items, processGroup, { batchSize: 1000 })
  return results // 1M items = huge memory usage
}

// DO: Stream results to disk
async function processLargeDataset(items: Item[]) {
  const outputStream = fs.createWriteStream('results.json')
  outputStream.write('[')

  let first = true
  await batch(items,
    async (group) => {
      for (const result of await processGroup(group)) {
        const json = JSON.stringify(result)
        outputStream.write(first ? json : `,${json}`)
        first = false
      }
    },
    { batchSize: 1000 }
  )

  outputStream.write(']')
  outputStream.end()
}

Optimal Concurrency

import os from 'os'

// Auto-detect optimal concurrency
function getOptimalConcurrency(taskType: 'cpu' | 'io'): number {
  const cpuCount = os.cpus().length

  if (taskType === 'cpu') {
    return cpuCount // CPU-bound: match CPU cores
  } else {
    return cpuCount * 4 // I/O-bound: can handle more
  }
}

// Usage
await batch(items, processGroup, {
  batchSize: 100,
  concurrency: getOptimalConcurrency('io')
})

Error Handling in Batches

Fail Fast vs. Continue on Error

// Strategy 1: Fail fast
async function batchFailFast<T, U>(items: T[], fn: (item: T) => Promise<U>) {
  return batch(
    items,
    // unwrap throws, which is what batch counts as a failure
    async (group) => unwrap(await mapAsync(group, fn)),
    { batchSize: 100 }
  )
  // The first failed batch stops the rest
}

// Strategy 2: Collect errors, continue processing
async function batchContinueOnError<T, U>(
  items: T[],
  fn: (item: T) => Promise<U>
): Promise<{ results: U[]; errors: Array<{ item: T; error: unknown }> }> {
  const results: U[] = []
  const errors: Array<{ item: T; error: unknown }> = []

  await batch(items,
    (group) => mapAsync(group, async (item) => {
      const result = await tryCatchAsync(() => fn(item))
      if (isOk(result)) {
        results.push(result.value)
      } else {
        errors.push({ item, error: result.error })
      }
    }),
    { batchSize: 100 }
  )

  return { results, errors }
}

Retry Failed Items

type BatchFailure = { message: string; failedItems: unknown[] }

async function batchWithRetry<T, U>(
  items: T[],
  fn: (item: T) => Promise<U>,
  maxRetries = 3
): Promise<Result<U[], BatchFailure>> {
  let currentItems = items
  let allResults: U[] = []

  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    const { results, errors } = await batchContinueOnError(currentItems, fn)

    allResults.push(...results)

    if (errors.length === 0) {
      return ok(allResults)
    }

    if (attempt === maxRetries) {
      return err({
        message: `${errors.length} items failed after ${maxRetries} retries`,
        failedItems: errors
      })
    }

    // Retry only failed items
    currentItems = errors.map(e => e.item)
    console.log(`Retrying ${currentItems.length} failed items (attempt ${attempt + 1})`)
  }

  return err({ message: 'No attempts ran', failedItems: [] })
}

Progress Tracking

Console Progress Bar

import { batch, mapAsync } from '@ontologyio/receta/async'

async function processWithProgress<T, U>(
  items: T[],
  fn: (item: T) => Promise<U>
) {
  const total = items.length
  let done = 0

  return batch(items,
    (group) => mapAsync(group, async (item) => {
      const result = await fn(item)
      done++
      const percent = ((done / total) * 100).toFixed(1)
      const bar = '█'.repeat(Math.floor(done / total * 40))
      const empty = '░'.repeat(40 - bar.length)
      process.stdout.write(`\r[${bar}${empty}] ${percent}% (${done}/${total})`)
      return result
    }),
    { batchSize: 100 }
  )
}

Structured Progress Reporting

interface ProgressReport {
  stage: string
  completed: number
  total: number
  percent: number
  estimatedTimeRemaining: number
  errors: number
}

class BatchProgressTracker {
  private startTime = Date.now()
  private completed = 0
  private errors = 0

  constructor(
    private total: number,
    private stage: string,
    private onUpdate: (report: ProgressReport) => void
  ) {}

  update(completed: number, errorOccurred = false) {
    this.completed = completed
    if (errorOccurred) this.errors++

    const elapsed = Date.now() - this.startTime
    const rate = completed / elapsed
    const remaining = this.total - completed
    const estimatedTimeRemaining = remaining / rate

    this.onUpdate({
      stage: this.stage,
      completed,
      total: this.total,
      percent: (completed / this.total) * 100,
      estimatedTimeRemaining,
      errors: this.errors
    })
  }
}

// Usage
async function processWithTracking<T, U>(
  items: T[],
  fn: (item: T) => Promise<U>
) {
  const tracker = new BatchProgressTracker(
    items.length,
    'Processing items',
    (report) => {
      console.log(`${report.stage}: ${report.completed}/${report.total}`)
      console.log(`ETA: ${Math.round(report.estimatedTimeRemaining / 1000)}s`)
      if (report.errors > 0) {
        console.log(`Errors: ${report.errors}`)
      }
    }
  )

  let completed = 0

  return batch(items,
    (group) => mapAsync(group, async (item) => {
      const result = await tryCatchAsync(() => fn(item))
      completed++
      tracker.update(completed, isErr(result))
      return result
    }),
    { batchSize: 100 }
  )
}

Summary

When to Use Batching

Use batch() when:

Processing large datasets (>100 items)
Calling rate-limited APIs
Managing memory usage
Need progress tracking
Want controlled concurrency

When to Use Polling

Use poll() when:

Waiting for async jobs to complete
Monitoring external system status
Processing webhooks/callbacks
Checking for data availability
Long-running operations

Best Practices

Start conservative: Small batch sizes, low concurrency
Monitor and adjust: Track performance, optimize iteratively
Handle errors gracefully: Continue on error, retry failed items
Respect rate limits: Add delays between batches
Track progress: Keep users informed on long operations
Test edge cases: Empty arrays, all failures, timeouts
Use backoff where it exists: retry has it, while poll waits a fixed interval

Batching and Polling

Table of ContentsCopy Anchor Link

Batch ProcessingCopy Anchor Link

The Problem: Processing Large DatasetsCopy Anchor Link

The Solution: Controlled BatchingCopy Anchor Link

Batch Function OverviewCopy Anchor Link

Basic UsageCopy Anchor Link

Type SignatureCopy Anchor Link

Chunk UtilityCopy Anchor Link

Breaking Arrays into BatchesCopy Anchor Link

Combining with RemedaCopy Anchor Link

Batch Size OptimizationCopy Anchor Link

Finding the Right Batch SizeCopy Anchor Link

Delays Between BatchesCopy Anchor Link

Rate Limit ComplianceCopy Anchor Link

Exponential Backoff Between BatchesCopy Anchor Link

Polling StrategiesCopy Anchor Link

Poll Function OverviewCopy Anchor Link

Type SignatureCopy Anchor Link

Working with PollErrorCopy Anchor Link

Real-World: CI/CD Job Monitoring with ResultCopy Anchor Link

Real-World: Order Processing with ResultCopy Anchor Link

BenefitsCopy Anchor Link

Polling PatternsCopy Anchor Link

Job Status PollingCopy Anchor Link

CI/CD Pipeline MonitoringCopy Anchor Link

Payment Processing StatusCopy Anchor Link

Combining Poll with TimeoutCopy Anchor Link

Time Limit OptionCopy Anchor Link

Real-World PatternsCopy Anchor Link

Pattern 1: Data Import/ExportCopy Anchor Link

Pattern 2: Email Sending BatchesCopy Anchor Link

Pattern 3: Database MigrationsCopy Anchor Link

Pattern 4: Order ProcessingCopy Anchor Link

Performance OptimizationCopy Anchor Link

Memory ManagementCopy Anchor Link

Optimal ConcurrencyCopy Anchor Link

Error Handling in BatchesCopy Anchor Link

Fail Fast vs. Continue on ErrorCopy Anchor Link

Retry Failed ItemsCopy Anchor Link

Progress TrackingCopy Anchor Link

Console Progress BarCopy Anchor Link

Structured Progress ReportingCopy Anchor Link

SummaryCopy Anchor Link

When to Use BatchingCopy Anchor Link

When to Use PollingCopy Anchor Link

Best PracticesCopy Anchor Link

On this page

Table of Contents

Batch Processing

The Problem: Processing Large Datasets

The Solution: Controlled Batching

Batch Function Overview

Basic Usage

Type Signature

Chunk Utility

Breaking Arrays into Batches

Combining with Remeda

Batch Size Optimization

Finding the Right Batch Size

Delays Between Batches

Rate Limit Compliance

Exponential Backoff Between Batches

Polling Strategies

Poll Function Overview

Type Signature

Working with PollError

Real-World: CI/CD Job Monitoring with Result

Real-World: Order Processing with Result

Benefits

Polling Patterns

Job Status Polling

CI/CD Pipeline Monitoring

Payment Processing Status

Combining Poll with Timeout

Time Limit Option

Real-World Patterns

Pattern 1: Data Import/Export

Pattern 2: Email Sending Batches

Pattern 3: Database Migrations

Pattern 4: Order Processing

Performance Optimization

Memory Management

Optimal Concurrency

Error Handling in Batches

Fail Fast vs. Continue on Error

Retry Failed Items

Progress Tracking

Console Progress Bar

Structured Progress Reporting

Summary

When to Use Batching

When to Use Polling

Best Practices