Enterprise Computing is the driving force behind many of the major Cloud Providerâs strategy today. Sure, it was small businesses and startups that drove the early innovation on Amazon Web Services but the Enterprise category is responsible for audacious predictions by the likes of Gartner, forecasting $107billion in Public Cloud spending by 2016. Enterprise Computing is roughly an $800billion business so itâs not surprising that weâre seeing this sort of momentum given the industry is migrating to the cloud on-masse. HPC on the other hand is a relatively niche market today. The overall spend is somewhere around $34billion. Cloud adoption has been modest to-date but this is rapidly accelerating. Big Data, a term almost as overloaded and ambiguous as Cloud, is the fastest growing IT sector right now. At least in hyperbole if nothing else. Analyst CAGR numbers are typically around 40% which is pretty remarkable.
There are two disruptive forces at play that will ultimately see the convergence of HPC, Big Data and Enterprise computing.
The hype surrounding Big Data has a lot to do with the value this offers to the Enterprise. As the quantity and complexity of unstructured data increases, so does the need to process it. Businesses are finding clever and innovative ways of turning this data into a source of revenue, not directly, but through a deeper understanding of their business, their customers and the habits exhibited by them.
Typically large silos of structured and unstructured data are spread across various divisions and applications within the enterprise. Big Data technology allows questions of this data to be more easily asked to gain insight that allows companies to serve their customers better and compete more effectively.
So Big Data is about performing analysis and computations against large amounts of data. What about HPC? Yep, same thing. Sure, there are some important distinctions and subtleties that I wonât delve into here but the point is weâre already seeing a strong convergence of Big Data and HPC technology.
Cost and Accessibility
Putting Big Data hype aside for a moment, the second, and most important, disruptive force at play is Cloud Computing. The cloud offers many advantages over traditional on premise computing that make HPC-like capabilities far more accessible and affordable. Here is a summary of some of the benefits;
Cost: For spikey workloads, the cost savings in the cloud can be significant. Whatâs more, the cost is an Operational rather than Capital Expense so is often more palatable for many businesses as costs can be attributed to a particular project.
Ease of use: The cloud can facilitate easy dynamic provisioning for specific workloads. The ability to have OS/Software configurations particular to a workload is a key benefit.
Speed of deployment: The ability to rapidly provision new environments/clusters in minutes is incredibly empowering to many businesses.
Scalability: Elastically scaling out to meet increased capacity demands is a powerful concept. The public cloud promises âinfiniteâ scale. The reality is somewhat different: there are some real limits even in the cloud. But the computing capacity that can be obtained from large providers such as Microsoft Azure is far greater than what most users can muster with internal hardware.
Resiliency: The ability to snapshot workloads as they are running can allow for check-pointing of MPI workloads. Combine this with active monitoring and the ability to dynamically move a guest VM from one physical host to another, and your workloads can keep running even in the face of hardware failure.
Portability: The ability to move a workload from one cloud platform to another on the fly without any application changes allows for powerful options such as for bursting from a private cloud out to a public cloud, High Availability where a workload is run on multiple clouds simultaneously, scaling across multiple clouds to meet extremely high resource requirements or, to take advantage of shifts in the spot pricing market.
In summary, enterprises across many industries have much to gain in adopting Big Compute and Big Data technology and taking advantage of the many benefits Cloud Computing offers for these applications. This is why I predict wider adoption and convergence of these technologies is inevitable. Software Vendors and Cloud Service Providers that are focusing solutions to this market and address challenges around data management and governance will be well positioned.
Follow me on twitter:@dave_fellows