Hatchet, a background task orchestration platform built on PostgreSQL, has officially launched its v1 release. The platform aims to solve the common challenges developers face when managing background tasks at scale, offering an alternative to traditional task queue systems like Celery or BullMQ, as well as workflow orchestrators like Temporal and Airflow.
Background tasks are essential for modern applications, helping offload work from web servers and ensuring critical operations complete even during traffic spikes. While many developers start with simple Redis or RabbitMQ-backed queues, these solutions often become difficult to debug and monitor as applications grow in complexity.
PostgreSQL-Based Architecture With Impressive Scalability
Hatchet's team made a significant architectural shift in their v1 release, moving away from the traditional FOR UPDATE SKIP LOCKED pattern commonly used in PostgreSQL-based queues. According to the developers, this approach didn't scale well beyond 25,000 queries per second, particularly when dealing with many workers and a large task backlog.
We originally had combined some of our workflow tables with our monitoring tables... This table used a UUID as the primary key, because we wanted UUIDs over the API instead of auto-incrementing IDs. The UUIDs caused some headaches down the line when trying to delete batches of data and prevent index bloat.
The team implemented several optimizations, including separating monitoring tables from queue tables to prevent issues like table bloat, buffering reads and writes to reduce database load, and switching high-volume tables to use identity columns instead of UUIDs. These changes have significantly improved performance and stability under heavy loads.
Flexible Task Execution Models
Hatchet supports multiple task execution patterns, catering to different developer needs. The platform offers both traditional task queues for simple background jobs and more complex workflow orchestration through directed acyclic graphs (DAGs) and durable execution patterns.
For developers coming from systems like Celery or SQS, Hatchet provides familiar queue functionality with improved monitoring. For those needing more complex workflows, the platform supports child spawning and durable execution patterns similar to Temporal's workflows and activities.
One user highlighted the platform's first-class Pydantic support in the v1 SDK as a major advantage for developers switching from Celery, noting that the type safety significantly improves the developer experience. The combination of Python and TypeScript SDKs also allows for greater flexibility in implementation, with Go support available as well.
Hatchet Key Features
- Queues: Durable task queue for reliable task execution
- Language Support: Python, TypeScript, and Go SDKs
- Task Orchestration: Support for DAGs (Directed Acyclic Graphs)
- Flow Control: Conditional logic and branching
- Scheduling: Time-based task execution
- Concurrency Management: Dynamic concurrency control
- Rate Limiting: Control task execution rates
- Monitoring: Real-time web dashboard
- Deployment Options: Cloud service or self-hosted
Deployment Methods
- Hatchet Cloud: Managed service
- Hatchet Lite: Bundled self-hosted option
- Docker Compose: Multi-container deployment
- Kubernetes: Helm chart available
Comparison Points
- vs Temporal: More focus on application developers vs workflow orchestration
- vs Task Queues (BullMQ, Celery): Better monitoring, more durable execution
- vs DAG Platforms (Airflow, Prefect, Dagster): More focused on application needs than data pipelines
- vs Cloud Tasks: More complex orchestration capabilities but similar webhook support
Developer Experience and Monitoring
Hatchet places significant emphasis on observability and developer experience. Unlike many task queue systems where monitoring becomes challenging at scale, Hatchet separates its monitoring infrastructure from its queue tables, allowing developers to query both active queues and historical task data without performance degradation.
The platform includes a real-time web dashboard that offers improved visibility compared to tools like Celery's Flower. This focus on monitoring addresses a common pain point raised by users - understanding what happens when tasks fail and how to debug issues effectively.
Some users have noted rough edges in the dashboard and documentation, though the team appears responsive to feedback. The separation of monitoring from queue tables also enables the option to run monitoring on a completely separate database if necessary, providing additional flexibility for high-scale deployments.
Deployment Options and Integration
Hatchet is available both as a cloud service and as a self-hosted solution. For self-hosting, the team provides a hatchet-lite option that bundles all internal services, as well as more complex deployment options via Docker Compose and Helm charts for Kubernetes.
The platform also supports webhook workers, allowing tasks to trigger HTTP endpoints rather than requiring dedicated long-running workers. This approach provides flexibility similar to services like Google Cloud Tasks, though some users noted that this functionality appears less prominently documented than other features.
While Hatchet's primary focus is on Python, TypeScript, and Go developers, the team is tracking requests for additional language SDKs. The platform's architecture using gRPC for worker communication may present challenges for direct API integration without using the official SDKs.
As background task management continues to be a critical component of modern application architecture, Hatchet's approach of combining the reliability of PostgreSQL with advanced orchestration features positions it as a compelling option for development teams looking to scale their background processing capabilities.
Reference: Run Background Tasks at Scale