Batch Orchestra is an easy-to-use library for reliably and scalably performing many like operations such as DB migrations and periodic batch jobs. It solves many problems for these scenarios, such as controllably parallel execution and fair pagination.

What features and benefits does it have?

It helps solve many problems that tend to come up for batch jobs.

• Can run tens or hundreds of processes in parallel.
• Eliminates unevenly-sized pages.
• Only scans through your database one time--no need for an initial pass to find the page boundaries.
• Scales "infinitely"
• Controllable parallelism to limit impact on downstream systems.
• Pauses and gradual throughput rampups.
• Feature-rich retries.
• Built-in progress tracking.
• Debuggable, via Temporal's UI and log statements.
• Rich failure handling:
  • Code deploys can fix your running batch without need for other remediations.
  • Built-in failure and stuck pages tracking.
  • Stuck pages do not block overall progress on other pages.
  • [coming soon] Allows you to avoid head-of-line blocking.

Check out the samples to get a better idea.