A decade-old approach to job processing in Elixir is experiencing renewed interest as developers explore alternatives to mainstream solutions. The discussion centers around building custom job runners using GenStage, Elixir's built-in producer-consumer framework, rather than relying solely on established libraries.
The conversation highlights a persistent challenge in the Elixir ecosystem: market adoption. Despite technical advantages, developers continue to face resistance when proposing Elixir-based solutions in enterprise environments. One developer shared their experience of building a production Pub/Sub notification system that has been running successfully for years, yet still encounters skepticism from customers who prefer more conventional technologies like Java, .NET, Python, or PHP.
GenStage Architecture Components:
- Producer: Generates messages/jobs (UUIDs in the example)
- Consumer: Processes jobs pulled from producers
- Supervisor: Manages process lifecycle and restarts
- Built-in Buffer: Stores items in memory before processing
Oban Dominates Production Deployments
While custom GenStage implementations offer learning value, the community strongly gravitates toward Oban for production use. This established job processing library provides both open-source and professional versions, with the latter offering advanced features like workflows, rate limiting, and concurrency controls. However, some developers express concern about the feature limitations in the free version, particularly when building open-source projects that might benefit from Pro features.
The discussion reveals a practical tension between the comprehensive nature of commercial solutions and the flexibility of custom implementations. While Oban handles most production requirements effectively, developers interested in specific architectural patterns or learning exercises find value in understanding the underlying GenStage mechanics.
Oban Version Comparison:
- Open Source: Basic job processing, scheduling, error handling
- Pro Version: Advanced features including workflows, rate limiting, concurrency controls, and multi-worker coordination
PostgreSQL Integration Opportunities
An interesting development direction emerged around PostgreSQL-based job processing. Developers are exploring integration with PostgreSQL extensions like pgmq and pg_cron, similar to approaches used by pgflow. This direction aligns with broader trends toward database-centric job processing, leveraging PostgreSQL's advanced features for queue management and scheduling.
I'd love to see one for Elixir built around the Postgres pgmq and pg_cron extensions similar to what pgflow is doing
The PostgreSQL approach offers potential advantages in environments already heavily invested in PostgreSQL infrastructure, particularly those using Supabase's realtime features, which are powered by Elixir and monitor PostgreSQL's Write-Ahead Log.
PostgreSQL Integration Options:
- pgmq extension: Message queue functionality
- pg_cron extension: Scheduling capabilities
- pgflow approach: Database-centric job processing
- FOR UPDATE SKIP LOCKED: Concurrent job processing with row-level locking
Educational Value and Long-term Code Stability
The revisiting of decade-old GenStage code demonstrates Elixir's stability and the enduring relevance of its concurrency model. Developers noted how well the original implementation held up over time, suggesting that foundational Elixir patterns remain robust and maintainable. This stability contrasts with the rapid evolution seen in many other technology stacks.
The educational aspect resonates particularly with developers learning Elixir's distributed computing capabilities. The ability to build sophisticated job processing systems with relatively straightforward code showcases Elixir's strengths in concurrent and distributed applications.
The ongoing discussion reflects broader themes in the Elixir community: technical excellence coupled with adoption challenges, the balance between custom solutions and established libraries, and the continuous evolution of PostgreSQL integration patterns. As developers continue exploring these approaches, the combination of proven stability and modern requirements continues to drive innovation in Elixir job processing solutions.
Reference: Writing A Job Runner (in Elixir) (Again) (10 years later)