Record and replay debugging is a powerful technique that allows developers to capture program execution and replay it exactly, making it easier to track down elusive bugs. However, one of the most popular open-source tools for this purpose, rr, has had a significant limitation: it requires access to CPU Hardware Performance counters, which are typically disabled in cloud VMs and container environments. A new breakthrough aims to change that.
Software Counters Mode Brings rr to More Environments
The rr debugger has been modified to run without requiring access to CPU Hardware Performance counters, making it usable in cloud virtual machines and containers where such access is typically restricted. This new variant, called Software Counters mode rr, uses lightweight dynamic and static instrumentation to track program execution instead of relying on hardware counters. This development opens up record and replay debugging to many more environments, potentially solving one of the biggest obstacles users have faced with rr.
I have tried SO hard to get rr to work for me, including buying a separate PC just to use it...but it just consistently fails so I've basically abandoned it. Something like this would absolutely be a godsend.
The modification addresses a fundamental challenge in record/replay systems: tracking program execution progress to ensure accurate replay. Traditional rr uses hardware performance counters (like counting CPU branch instructions) to measure this progress, but Software Counters mode rr achieves the same goal through code instrumentation.
Tradeoffs and Current Limitations
While this breakthrough expands rr's usability, it comes with some tradeoffs. According to the developer, Software Counters mode is slower than using hardware counters due to the overhead of dynamic and static instrumentation. The approach may also introduce some fragility to the record/replay process.
Currently, only x86-64 support has been publicly released, though the developer mentions having aarch64 support working internally. Another limitation is that Software Counters mode rr currently runs only on recent Linux distributions with robust debuginfod support, including Fedora 40/41, Debian Unstable, and Ubuntu 24.10.
Despite these limitations, the community response has been enthusiastic, with many users eager to try a version of rr that works in environments where the original tool fails. The ability to use record/replay debugging in cloud VMs could be particularly valuable as more development work shifts to cloud-based environments.
Key Limitations of Software Counters Mode rr
- Performance: Slower than using rr with Hardware counters due to instrumentation overhead
- Fragility: Dynamic and static instrumentation can make record/replay more fragile
- Platform support: Currently only x86-64 architecture supported publicly
- Distribution requirements: Runs only on recent Linux distributions with robust debuginfod support:
- Fedora 40/41
- Debian Unstable
- Ubuntu 24.10
- Shared limitations with original rr: No io_uring support
Future Prospects and Integration
There's currently no definitive plan for merging Software Counters mode back into the main rr project. The developer notes that this modification introduces a significantly different approach to tick counting through dynamic/static instrumentation, which represents a major change from rr's traditional methodology.
The developer is encouraging users to try the software and report their experiences, suggesting that with sufficient positive feedback, a stronger case could be made for upstream integration. Some users have also inquired about compatibility with other tools in the ecosystem, such as Pernosco (a debugging platform built on rr), which would likely require technical modifications to support recordings with soft ticks.
It's worth noting that certain limitations of the original rr remain unchanged in Software Counters mode. For example, io_uring support is still missing due to fundamental architectural challenges in how rr interacts with the kernel/userspace boundary.
For developers interested in trying Software Counters mode rr, the code is available at https://github.com/sidkshatriya/rr.soft, along with detailed documentation on how it works and how to build and run it.
Reference: Bringing Record and Replay everywhere