The tech community is actively discussing different approaches to solve S3's latency problems after RisingWave published details about their hybrid caching solution called Foyer. While S3 offers unlimited storage at low costs, its 50-150ms latency makes it unsuitable for real-time applications that need fast data access.
S3 vs Local Storage Latency Comparison
- S3 time to first byte: 50-150ms
- Memory access: ~0ms
- Disk access: 0.2-1ms
- S3 Express One Zone: Single-digit milliseconds
Swap vs Explicit Cache Management
A key debate emerged around whether developers need complex caching systems or can rely on simpler solutions. Some developers argue that using system memory with swap provides a unified interface without explicit cache management. Materialize.com reportedly switched from custom disk cache management to using swap and saw significant performance improvements. This approach lets the operating system handle memory spilling to disk automatically.
However, others point out serious drawbacks with the swap approach. For latency-sensitive workloads, page faults and swapping create unpredictable performance spikes that hurt response times. Many production environments intentionally disable swap to prevent these issues, making explicit cache management more reliable across different deployment scenarios.
Cache Management Strategies
- Explicit Management: Custom memory and disk cache coordination
- Swap-based: Let OS handle memory spilling to disk automatically
- Hybrid Approach: Combine memory cache with disk cache layers
- Trade-offs: Complexity vs performance predictability
Real-World Performance Impact
The performance difference between S3 and local storage is substantial. S3's time to first byte ranges from 50-150ms, while serving from memory drops to nearly zero, and disk access takes just 0.2-1ms. This massive gap explains why caching becomes essential rather than optional for applications built on S3.
AWS does offer S3 Express One Zone with single-digit millisecond latency, but this premium service targets different use cases where the same objects need frequent access across multiple instances. For single-instance repeated access, local caching remains more cost-effective.
Open Source Solutions Gaining Traction
Several projects are building on Foyer, the Rust-based hybrid caching library inspired by Facebook's CacheLib. Distributed Chroma uses Foyer extensively in their open-source vector database, while other developers have integrated it into file systems and storage services. The library offers both in-memory and hybrid modes, letting developers start simple and add disk caching later without changing application logic.
Foyer Hybrid Cache Features
- Written in Rust, inspired by Facebook's CacheLib
- Supports both in-memory and hybrid modes
- Zero-copy in-memory cache abstraction
- Multiple disk engines for different workloads
- Used by Distributed Chroma and other open-source projects
Cost Considerations and Trade-offs
While S3 GET requests are relatively cheap, the real costs come from latency impact on application performance rather than request pricing. The complexity of hybrid caching systems requires careful planning around cache invalidation, write-through behavior, and handling concurrent requests for the same data.
The discussion reveals that choosing between different caching strategies depends heavily on specific workload requirements, deployment constraints, and performance targets. As more applications adopt S3-first architectures, these caching decisions become increasingly critical for maintaining acceptable user experiences.
Reference: The Case for Hybrid Cache for Object Stores