Virtually every high-end GPU and AI accelerator relies on high bandwidth memory (HBM), which can shuffle data around at multiple terabytes a second but can only reach into the gigabytes, with models often needing to be shared across multiple processors. However, an emerging storage technology could change that, boosting accelerator memory capacity from hundreds of gigabytes to terabytes. The technology, called high-bandwidth flash (HBF), is being developed by Sandisk and SK Hynix and aims to provide SSD-like capacities at HBM-like speeds. Peeling back HBF’s layers Conceptually, high-bandwidth flash looks and sounds a lot like HBM. It’s assembled by stacking multiple layers (16 in the case of Sandisk’s first-gen modules) of memory together, which boosts capacity and bandwidth. But where HBM uses DRAM, HBF aims to use NAND flash. Sandisk claims its first generation of high-bandwidth flash will supposedly achieve read bandwidths up to 1.6 TB/s [PDF], making it a bit faster than HBM3e but significantly slower than HBM4, which is already hitting 2.5 TB/s per 12-high stack. Future HBF generations are expected to push bandwidth to over 2 TB/s and eventually 3.2 TB/s. While bandwidth makes HBF interesting as an alternative to HBM, its real party trick is capacity. Because it’s built using NAND, Sandisk says it can achieve capacities up to 256 Gb per die, which translates to 512 GB per 16-high module. That’s more than 14 times the capacity of the HBM4 used in AMD and Nvidia’s latest accelerators. Continuing with the similarities, HBF modules share similar packaging requirements to HBM, which means you can expect them to be fused to the GPU die using advanced packaging techniques like TSMC’s CoWoS, or Intel’s EMIB and Foveros tech. Nothing particularly exotic as AI accelerators go. What’s more, the storage vendor doesn’t expect the modules to come at a power or price premium over HBM. And from a bits per dollar standpoint, HBF looks like a stellar option. If all this sounds a bit too good to be true, that’s because for all of HBF’s benefits, it comes with some rather significant compromises. NAND still isn’t DRAM The main trade off, as we understand it, is write endurance and access latency. HBF may perform like HBM on paper, but it’s still using NAND, which has a finite write endurance before it wears out and has access latencies measured in microseconds as opposed to tens of nanoseconds for DRAM. If you were to swap HBM for HBF, it (probably) wouldn’t perform very well and it’d wear out pretty quickly, rendering that $50,000-plus GPU of yours a paperweight — not ideal for a product that’s being asked to serve longer to suit hyperscalers' depreciation schedules. Instead, Sandisk and SK Hynix propose using HBF to supplement HBM to make inference more cost effective. HBM handles all the write intensive stuff while HBF takes care of the read heavy parts of the pipeline. While we talk about inference as one job, it's really a collection of many that can broadly be broken into two categories, one that’s compute intensive and another that’s bandwidth bound. The first of these phases, called prefill, involves tokenizing and embedding prompts, feeding them through the model in one big forward pass, generating the key value caches used to track state, and outputting the first token. The second, called decode, reads the entirety of the model's weights, or in the case of a mixture of experts (MoE) models, their active parameters, from memory over and over again for each token generated. Because of this, how quickly an AI system can churn out tokens is directly proportional to how fast its memory is. Prefill is comparatively write heavy, so it makes sense to do as much of that in HBM as possible. But the decode phase is almost entirely read, which makes HBF an ideal medium for storing model weights as write endurance really isn’t a factor. It becomes a sort of write-once, read-many scenario, which is perfect for NAND flash since reads are essentially free. You could almost think about HBF a bit like a rewritable ROM cartridge for models. And because HBF is non-volatile, it becomes a bit like Intel’s Optane persistent memory. There is no need to wait for weights to reload from storage into GPU memory; they’re already there and ready to go. Sandisk’s slides propose a couple of different options including one that would feature 3.12 TB of memory across two stacks of HBM and six stacks of HBF. Oh the things you can do with all that memory That much memory has implications for model and inference architectures. Most frontier models at this point employ a mixture-of-experts (MoE) architecture, which means the model is really a collection of routed sub-models called experts, a small selection of which are used to generate each token. This has allowed model devs to build models larger than would otherwise be practical to serve due to memory bandwidth constraints. Because HBM’s capacity is so limited, these experts usually have to be spread across multiple GPUs connected by extremely high-speed interconnects. But with high-bandwidth flash, even multi-trillion-parameter models, like Kimi K3, could be packed into a single accelerator, mitigating any of the performance bottlenecks induced by the chip-to-chip interconnects. On the flip side, HBF could allow a 72-GPU rack to run some truly massive models measuring hundreds of trillions of parameters. Training such a model presents its own unique set of challenges, and the number of active experts/parameters would be limited by HBF’s bandwidth, but it could work. So when? So when can we expect to see HBF deployed in datacenters? If Sandisk is to be believed, the first samples should go out later this year with the first AI inference devices based on HBF available early next year. But for a variety of reasons previously highlighted by our sibling site Blocks and Files, we’re not holding our breath. One of the biggest factors is standardization. Memory is a commodity business and blazing your own trail with a proprietary technology rarely pans out — just look at Optane if you need evidence of that. Sandisk and SK Hynix officially kicked off this process earlier this year under the auspices of the Open Compute Project. “The key to AI infrastructure is to go beyond the performance competition of individual technologies and to optimize the entire ecosystem,” Ahn Hyun, president and chief development officer at SK Hynix, said at the time. There are also manufacturing considerations to be made. HBF is going to require a lot more dies per module than your typical flash storage device. What’s more, from what we understand, the specific kind of NAND used to make these modules is different from the kind used to make SSDs and other flash storage. Those modules will need to be copackaged with accelerators, which means SK Hynix and Sandisk will need to get buy-in from GPU and ASIC makers, which is going to take time. In fact, it wouldn’t be surprising for these modules to be ready years before the first chips designed to take advantage of them enter production.®
/s3/static.nrc.nl/wp-content/uploads/2026/07/30151451/300726VER_2035601660_sonita.jpg)
