Big data storage: the real cost

Published on Oct 6, 2015 in Quantum

These days big data is a buzz word that risks becoming a cliché; what exactly is it? How much data is “big”? A few years ago a terabyte seemed an inconceivable amount, today nobody thinks twice about it. Where is the line between regular data and big data? Perhaps the important question is not “what is big data?” but rather, “can I easily manage the data that I have at a price that makes sense – both now and in the future”? At Trams we take great pains to ensure that whether the customer is investing in 20 Terabytes or 500 Terabytes, and projected to rise into the Petabytes, we take the same careful approach – and the answer to that question will be “yes”.

Too often we meet customers who have purchased a system that was sold as simple and expandable, but sadly they didn’t read the small print. These systems often come with a hidden future cost because the base system wasn’t as flexible as the customer had originally hoped. For example, the system cannot scale up but only scale out, or cannot easily be backed up once it reaches 200TB so a replication solution is then required; “Ker-ching” – buy the same amount of disk again Mr. Customer, because there is no other choice. Sadly we see all of these things happen and a long list of others all too frequently.

We have therefore selected Quantum StorNext as our platform of choice for big data. StorNext is now in its fifth iteration. With over 50,000 installations worldwide and performance at the top of the curve, StorNext meets the requirement for scalability, reliability and unrivalled connectivity choices to your Enterprise data. StorNext can deliver the broadest range of connectivity and performance requirements whilst scaling massively into multi-petabyte file-systems containing billions of files:

  • 􀀠 Connectivity via IP using industry standard NFS, CIFS and AFP
  • Cluster access via DFS or Linux clustering to multiplex bandwidth and redundancy
  • Block level connectivity to the filesystem via StorNext Distributed LAN Client gateway servers for the fastest most demanding scenarios over 1GbE and 10GbE
  • Block level connectivity to the filesystem via FC, FCoE and iSCSI for best performance, easily scaling to deliver gigabytes per second performance to individual connected servers and workstations

Crucially, all the above connection and performance flexibility can be achieved at the same time, to the same filesystem from multiple diverse connected operating systems, Linux, Windows, Mac, AIX and Solaris.

But we can do a lot more than that. Certainly whilst Trams can supply StorNext Appliances and storage (disk and tape) from Quantum if required, we can also leverage your existing investment in enterprise storage so that you do not have to buy new. Globally deployments exist involving disk from Hitachi Data Systems, EMC, NetApp, Hewlett Packard, Data Direct, IBM, Promise, Infortrend, Dell, Nexsan and many others. First implemented over 15 years ago, StorNext has evolved to be the industry leader in many fields where high performance and rock solid reliability are key.

And that is just the StorNext Filesystem. Beneath that comes the data moving HSM “smoke and mirrors” of StorNext Storage Manager. Most disk-based systems contain large quantities of data that are little used. These systems cost money and yet the data upon them is very often largely at rest. This represents a very poor return on the investment cost of expensive infrastructure. StorNext is capable of tiering data from a relatively small pool of high performance disk onto much cheaper tape systems. So when accessed, the data is returned to disk for user access – consider it in this case as a very fast data cache where only the important accessed data is in the cache. But if the delay inherent in tape retrieval is too great to tolerate, StorNext can also use StorNext Lattus M object storage for tiering and deliver data back in near real time at throughputs of 2-3 gigabytes per second and upwards, whilst storing the data on disk that scales into the petabytes. This groundbreaking Lattus M storage system encompasses ‘store forever’ reliability and a migration roadmap that ensures that as disks become too old to continue in production, Lattus M can keep your data alive safely and quickly. It is even possible to geo-spread your data to enable entire site-loss tolerance.

Trams’ staff has experience with StorNext that spans over 7 years. With a great deal of pride Trams has been Quantum’s Big Data reseller of the year in EMEA four years running (2012-2015). In addition, Quantum recently selected Trams’ Technical Head to act as lead System Integrator on their first Lattus Object Storage deployment worldwide, which was successfully completed in August this year. “We replaced our Xsan just over a year ago with Quantum M330. When the renewal came up for the support, a year after the installation, we realised we hadn’t switched it off in that time and it had caused us absolutely no bother whatsoever.”

In addition to this first Object Storage project, other projects from Trams have recently involved an IP based solution to deliver 40 gigabit connectivity to a filesystem for access by over 100 high performance video render blades, at a major satellite broadcaster. This will deliver near wire speed access for render farming. Sitting upon 120 IBM primary enterprise disks, this system will bring new levels of performance to the render farm.

Another recent project has involved migrating an entire advertising agency’s creative and administrative data to a StorNext Managed Filesystem so that backups become a thing of the past, data is protected, replicated to a DR site and protected again with almost zero staff input. Remember changing tapes? Those guys no longer have to, it is a thing of the past now.

Another major customer has revolutionised their video on demand service for playout via cable and satellite by replacing NAS based systems with fibre channel connectivity to StorNext,; the 10GbE NAS they used before was just too slow and StorNext leaves it standing.

Another recent customer was also suffering with a well-known scale out NAS solution for their Data Analytics, it just couldn’t meet the performance that StorNext could deliver – over 13,000 file creates per second (if your disk is fast enough!) with StorNext version 5.

Is any of this big data? That depends on your perspective, but is StorNext a credible and cost-effective challenger to the big data crown? Absolutely. So if you need it big and fast (or big and not so fast), scale up or scale out, or replicated, or deduplicated, protected, connected to via IP, FC, FCoE and iSCSI, native on Windows, Linux, Mac, AIX and Solaris – AT THE SAME TIME – then you should take a look at it. Tell us what you want to achieve and we will tell you how we can bring the power of StorNext to bear upon your data – what have you got to lose?

Warren Peel, Managing Director, Trams / QsmartStorage