A $50B Flash Tsunami is Coming!
A powerful tsunami of flash adoption is breaking over the enterprise and subsequent waves of adoption look to be even larger. Flash technology initially got attention in the enterprise as a performance play. But flash media is rapidly gaining momentum because of its many other benefits relative to traditional hard disks including management savings, maintenance reductions, and consolidation for power, cooling, and rack space.
This article explores how new hyper-converged infrastructure (HCI) systems speed the adoption of flash storage in the enterprise by combining the performance of server-side architectures with the economics of shared storage. HCI introduces a new way for all-flash systems to be deployed so that tiered flash can deliver both performance and management benefits affordably. HCI looks to be a powerful engine fueling the next wave of flash adoption.
Flash Cache Started the First $200M Wave
Flash was initially deployed as a cache in the late 90’s when flash solid-state performance was available only in small densities at high cost. The flash-as-cache model quickly became the norm for enterprise server and storage systems because a small amount of expensive flash could boost the performance of slow but cheap mechanical hard disk drives.
Flash and HCI are great equalizers, allowing IT generalists to configure and manage complex storage systems with ease
It would take years for Moore’s Law to catch up with the market need for higher capacity flash.
Server-side Flash Fueled a $1B Wave
Flash manufacturing ramped up quickly in the late 2000’s to meet the explosive growth of mobile phones. This surge in large-scale flash manufacturing resulted in higher capacity solid-state flash devices that were far superior to existing mechanical disk drives for performance applications.
Innovators like Fusion-io saw the opportunity to repurpose consumer-grade solid-state flash devices for local servers to accelerate latency-sensitive applications such as databases. And while server-side flash remained expensive in capacity terms, the technology improved performance economics by one to two orders of magnitude over hard disks on performance metrics such as dollar per latency or dollar per bandwidth.
The challenge with server-side flash was that performance was isolated to a single server. What worked for one database administrator didn’t help the overall enterprise. So while server-side flash quickly became a $1B market, it did not drive mass adoption because it did not solve the need to share flash performance affordably.
Storage-side Flash Flooded A $10B Wave
The next market boost for flash came once in-line storage efficiency technologies were introduced by flash storage vendors to reduce the “effective” cost of solid-state media. Unlike traditional disk-based storage systems that relied on post-process compression and deduplication, all-flash arrays deployed new in-line efficiency techniques to reduce the actual raw amount of expensive flash capacity required. In-line efficiencies typically reduced the amount of flash capacity 2-10x and fewer write operations allowed vendors to use less expensive flash devices.
Inline processing of flash forever changed the optimal design of large-scale storage systems. Traditional storage systems employed a “scale-up” design where a small number of storage controllers, typically two, managed up to hundreds of high-latency disk drives. But, scale-up designs were wholly unsuited for all-flash storage systems where controller performance needed to scale linearly with capacity to keep up with deduplication and compression.
Newer flash systems like EMC xTremIo and NetApp SolidFirefavor “scale-out” designs where storage controllers are added linearly with flash capacity so that compression and deduplication do not comprise the low-latency performance of the underlying flash media.
For storage buyers, flash had found its mark. However, the storage market is small relative to the server market. Roughly 14 million servers are sold each year while less than 1 million SAN and NAS storage systems server are deployed. So flash adoption in the enterprise market depends on an economic model where the server market can integrate in-line flash efficiencies and scale-out designs with server economics.
HCI Serverizes Storage for a $50B Flash Wave
Hyper-converged infrastructure, or HCI, unites performant server-side flash with affordable shared flash storage to completely reset the economics of investing in on-premises infrastructure. What makes HCI possible is software-defined storage software applied to the raw horsepower of x86 servers containing low-cost flash and low-latency network hardware. In other words, the nexus of high-performance compute, storage, and network hardware resources provides an unmatched time for storage software to uniquely integrate these resources in a way that provides a new price-performance alternative to traditional storage systems.
The optimum architectural design point for the software defined data center is the hypervisor. Early HCI systems such as Lefthand Networks, Nutanix and Pivot3 provided storage management outside the hypervisor. But HCI implementations outside the hypervisor are inherently inefficient and require multiple management interfaces for virtualized compute, storage and network resources.
Only VMware and Microsoft offer HCI solutions that are directly integrated in mainstream hypervisors. By our estimates, the VMware vSAN and Microsoft Storage Spaces Direct software-defined storage solutions will together drive over $50B in enterprise flash sales over the next 5 years. For sure, customers will continue to use enterprise SAN and NAS storage systems for custom high-performance and high-capacity use cases. But mainstream virtualization environments will quickly move to HCI for management simplicity.
Flash and HCI are Digital Transformation Tools
Flash in HCI systems is particularly compelling because IT generalists simply do not have time to dwell on storage configurations. Flash eliminates the need to plan ahead and obviates the difficult process of diagnosing performance issues. In fact, for many customers, flash eliminates the need to rewrite poorly architected applications.
Today’s storage administrators have to anticipate application needs and they typically over-provision systems based on years of experience that inform choice of RAID levels, queue depths, stripe sizes, disk speeds, and cache tier values. With flash, the value of this esoteric knowledge evaporates in the same way that knowing how to buy a horse disappeared once gasoline and cars became affordable.
Flash and HCI are great equalizers, allowing IT generalists to configure and manage complex storage systems with ease. HCI allows server experts to configure complex storage systems with minimal budget commitments and with simple scale-out flexibility. For CIOs, the combination of HCI and flash reduces initial investment commitment and the risk of errors.
All-Flash HCI Creates Competitive Advantage for CIOs
HCI offers a compelling new IT model that reasserts IT as a competitive differentiator. The budget implications of hyper-converging server and storage hardware designs are broad. Storage systems and server systems have typically been over-provisioned as storage has been configured, sized and managed as a separate silo from compute. HCI allows customers to scale storage capacity and performance in small increments tied to increased compute needs. This vm-centric budgeting led Gartner to conclude that software-defined HCI saves 50 percent over traditional systems.
For CIOs looking at how to excel in a world of digital transformation, HCI offers a server-centric approach to infrastructure that taps into the economics of Intel systems that can be managed by IT generalists. What is critical is to select a HCI architecture that is integrated in the hypervisor to maximize economic and management benefits. The power of flash in simplifying infrastructure management are magnified as HCI solutions marry the performance of server-side products with the scale of storage-side systems in a single scale-out solution that can be managed by IT generalists.