DeepSeek AI's recently released DeepGEMM library has captured the attention of the technical community with its innovative optimization techniques for FP8 matrix operations. While the library offers several performance improvements for General Matrix Multiplications (GEMMs), it's the FFMA SASS interleaving technique that has particularly impressed technical experts, delivering performance gains exceeding 10% in some cases.
The Magic Behind FFMA SASS Interleaving
The DeepGEMM team discovered a performance improvement in CUTLASS FP8 kernels between NVCC 12.2 and 12.3 compiler versions. Through careful analysis of compiled SASS (Streaming Assembly) code, they identified that a specific bit in FADD instructions was being flipped in an interleaving pattern. This bit controls the yield functionality, which essentially allows the current warp to yield execution, potentially enhancing warp-level parallelism by letting other warps work.
Building on this discovery, the team developed a script to modify FFMA (Fused Floating-point Multiply-Add) instructions in the compiled binary. They not only manipulated the yield bit but also flipped the reuse bit, as registers cannot be reused if the warp is yielded. This seemingly small modification creates more opportunities to overlap Matrix Multiply-Accumulate (MMA) instructions with promotion FFMA instructions, resulting in significant performance improvements.
I would say it is really mind-blowing.
Specialized Optimization for Critical AI Infrastructure
The community discussion highlights that while such performance optimizations are typical in matrix math when performance is critical, they haven't been widely applied to this specific problem by other AI companies. As one commenter noted, most AI players rely on high-performance GEMM operations but typically settle for standard implementations like CUTLASS or cuBLAS rather than leveraging undocumented features.
This level of optimization demonstrates the lengths to which AI companies are willing to go to squeeze out every bit of performance from expensive GPU clusters. Even a 10% performance improvement can translate to significant cost savings when operating at scale. As the discussion points out, such gains could potentially pay for a lot of people's salaries when companies are investing hundreds of millions in GPU infrastructure.
DeepGEMM Key Features and Requirements
-
Performance Gains: Up to 2.7x speedup compared to optimized CUTLASS 3.6 implementation
-
Optimization Techniques:
- Persistent warp-specialization
- Hopper TMA (Tensor Memory Accelerator) features
- Unified block scheduler with rasterization
- Fully JIT design
- Unaligned block sizes
- FFMA SASS interleaving
-
Hardware Requirements:
- Hopper architecture GPUs with sm_90a support
- Python 3.8+
- CUDA 12.3+ (12.8+ recommended)
- PyTorch 2.1+
- CUTLASS 3.6+
Industry Impact and Accessibility
DeepGEMM's open-source release appears strategically positioned to benefit the industry at large, particularly larger providers serving AI models. The library requires Hopper architecture GPUs (with sm_90a support) and is specifically designed for scenarios like those in DeepSeek-V3, supporting both normal and Mix-of-Experts (MoE) grouped GEMMs.
Some community members have already attempted to test the library on consumer hardware like the RTX 5080, encountering limitations related to shared memory capacity. The library is explicitly designed for NVIDIA Hopper tensor cores, making it primarily relevant for enterprise-grade AI infrastructure rather than consumer applications.
The technical depth of DeepGEMM highlights the growing sophistication of AI infrastructure optimization. As AI models continue to grow in size and complexity, these seemingly minor optimizations at the hardware instruction level become increasingly valuable for organizations pushing the boundaries of what's possible with current hardware.
Reference: DeepGEMM: clean and efficient FP8 GEMM kernels with fine-grained scaling