It’s now the end  Movember which for me was a time to focus on men’s health, and in particular mental health. Most of you will know that I can be a moody old thing from time to time and some of you will know the impact that depression has on young men in particular,  and how that caused me to lose a number of good friends.

To show my commitment, I donated my face to the cause by growing a moustache for the entire month of November. My Mo sparked a number conversations, and no doubt generated some laughs; all in the name of raising vital awareness and funds for prostate cancer and male depression.

Why am I so passionate about men’s health?

• 1 in 9 men will be diagnosed with prostate cancer in their lifetime
• This year 20,000 new cases of the disease will be diagnosed
• 1 in 8 men will experience depression in their lifetime

I’m asking you to support my Movember campaign by making a donation by either:

• Donating online at: http://mobro.co/JohhMartin
• Writing a cheque payable to ‘Movember’, referencing my Registration ID: 2207632 and mailing it to: Movember, PO Box 60, East Melbourne, VIC, 8002

If you’d like to find out more about the type of work you’d be helping to fund by supporting Movember, take a look at the Programs We Fund section on the Movember website: http://au.movember.com/about

Thank you in advance for supporting my efforts to change the face of men’s health.

This post Originally appeared in the ABC Tech and Games blog

The IT transformation currently occurring in the market thanks to cloud computing and the wide adoption of shared IT infrastructure seems like it’s predominantly affecting the large enterprise sector. But while this big-business IT revolution is going on, there is a flow-on effect to the mid-size market which is also tackling unprecedented data growth whilst struggling to assess the benefits of a move to cloud computing. The technology challenge for MSEs is in understanding how best to optimise their IT environments for both efficiency and scale so they can be comfortable in the knowledge they’ve made the right decisions for their business in the longer term. What is a mid-size business? Generally 100-1000 staff 1-3 IT staff who are generalists, not specialists IT staff typically have responsibility for the entire IT infrastructure. For the most part, the IT Vendor community in Australia tends to focus on the big end of town with multinational organisations dominating the landscape. When you then consider that 73 per cent (source: Reckon 2010 Annual SMB Business Survey) of the Australian economy is based on small businesses it seems unsurprising that mid-size enterprises which fall between these two market segments often seem to be forgotten. This means many MSEs are either reaching the limits of technology designed for small business with the resulting reliability and management headaches, or they’re paying too much to use specialist driven IT solutions aimed at the big end of town in an effort to avoid the problems they’ve just escaped from trying to do too much with SMB focused technology. These high end technologies with dedicated and siloed functionality aren’t well suited to mid-sized enterprises, not only because of their inherently high costs and inefficiencies, but also because the IT employees in these organisations are usually generalists. They need to know how all the company’s systems work, and how to fix them if they break and they simply don’t have the time to gain the specialist expertise needed to get the most out of these solutions. All of this puts the purchaser of IT for mid-size enterprises in an unenviable position. With many vendors rapidly jumping onto the cloud bandwagon, the mid-sized enterprise who still needs internal IT, but not at the kind of scale that would allow “internal clouds” are seeing a lot of turbulence in the supplier marketplace, none of which seems to be helping them. Many vendors are changing the way they are going to market and moving their focus away from their products for mid-sized business towards “cloudy” futures, and no longer investing in the kind of innovation required by this challenging business environment. While this is a worry, many of the traditional solutions offered by technology vendors to mid-sized enterprises often never really met their specific needs and challenges effectively. In an effort to win the business in these very budget-conscious organisations, mid-size businesses are often offered commodity-based solutions with low upfront costs, without being fully informed that many of these solutions cannot continue to meet their business needs as their company grows. In the midst of this gloom, one piece of good news is that in general, unlike many other areas of the marketplace that are facing budget constraints, mid-size business budgets are still reasonably healthy, though not infinite. Over the last few years, many MSEs have successfully focused on cost containment. For example, a large percentage have already taken the virtualization path, in fact, the virtualization trend is moving faster now than ever before. MSEs have seen great savings from these efforts, but most are now seeing that they’ve saved about as much as they can from consolidating and virtualizing their compute infrastructure while at the same time they are seeing a steady increase in the amount of money and percentage of their IT budget they spend on data storage. As the focus moves towards optimising data storage costs, mid-size businesses are looking for ways to reproduce the cost savings benefits they have gained in virtualizing their compute capacity. They are also looking for ways to optimize their environments to achieve more, address the data growth they are seeing and gain competitive advantage. These factors have huge implications on a company’s IT and in particular their data storage infrastructure requirements. However, these issues also provide a great opportunity to enhance IT systems to poise the company for growth. By demanding solutions that solve important and difficult challenges without undue complexity, and are powerful and scalable enough for the future, MSEs can gain great return on investment and get more from their suppliers. The test for mid-size companies is to make smart decisions on technology that is genuinely efficient, provide simplicity and offer scalability to meet growth requirements. The goal of MSEs needs to be focused around a technology foundation that will maintain pace with the growing business demands and allow the company to do more with fewer resources. So, what’s the lesson? Mid-size businesses are poised to continue growth and dominate in the market. The things they need to look out for in the technology arena are: solutions that can scale with their business needs – up and down – to protect their initial IT investment simple technologies that can be managed by IT generalists, yet still provide good cost of ownership and advanced enterprise-level capabilities partner businesses who can help them make smart decisions about the longer-term IT strategy, so it aligns properly with business objectives

–This has been revised based on some comments I’ve received since the original posting, check the comment thread if you’re interested what/why–

I came in this morning with an unusually clear diary, and took the liberty to check the newsfeeds for NetApp and EMC, this is when I came across an EMC press release entitled  ”EMC VNX SETS PERFORMANCE DENSITY RECORD WITH LUSTRE —SHOWCASES “NO COMPROMISE” HPC STORAGE“.

I’ve been doing some research on Lustre and HPC recently, and that claim surprised me more than a little, so I checked it out, maybe there’s a VNX sweetspot for HPC that I wasnt expecting. The one thing that stood out straight away was . “EMC® is announcing that the EMC® VNX7500 has set a performance density record with Lustre—delivering read performance density of 2GB/sec per rack” (highlight mine)

So today I found out that we’d broken a few records of our own few days ago, which was, at least from my perspective associated with surprisingly little fanfare with the associated press release coming out late last night. I’d like to say that the results speak for themselves, and to an extent they do. NetApp now holds the top two spots, and four out of the top five results on the ranking ladder. If this were the olympics most people would agree that this represents a position of PURE DOMINATION. High fives all round, and much chest beating and downing of well deserved delicious amber beverages.

So, apart from having the biggest number (which is nice), what did we prove ?

Benchmarks are interesting to me because they are the almost perfect intersection of my interests in both technical storage performance  and marketing and messaging. From a technical viewpoint, a benchmark can be really useful, but it only provides a relatively small number of proof points, and extrapolating beyond those, or making generalised conclusions is rarely a good idea.

For example, when NetApp released their SPC-1 benchmarks a few years ago, it proved a number of things

1. That under heavy load which involved a large number of random writes, a NetApp arrays performance remained steady over time

2. That this could be done while taking multiple snapshots, and more importantly while deleting and retiring them while under heavy load

3. That this could be done with RAID-6 and with a greater capacity efficiency as measured by  RAW vs USED than any other submission

4. That this could be done at better levels of performance than an equivalently configured  commonly used “traditional array” as exemplified by EMCs CX3-40

5. That the copy on write performance of the snapshots on an EMC array sucked under heavy load (and by implication similar copy on write snapshot implementations on other vendors arrays)

That’s a pretty good list of things to prove, especially in the face of considerable unfounded misinformation being put out at the time, and which, surprisingly is still bandied about despite the independently audited proof to the contrary. Having said that, this was not a “my number is the biggest”, exercise which generally proves nothing more than how much hardware you had available in your testing lab at the time.

A few months later we published another SPC-1 result which showed that we could pretty much doubl the numbers we’d achieved in the previous generation at a lower price per IOP with what was at the time a very competetive submission.

About two years after that we published yet another SPC-1 result with the direct replacement for the controller used in the previous test (3270 vs 3170). What this test didnt do was to show how much more load could be placed on the system, what it did do was to show that we could give our customers more IOPS at a lower latency with half the number of spindles .   This was the first time we’d submitted an SPC-1e result which foucussed on energy efficiency. It showed, quite dramatically how effective our FlashCache technology was under a heavy random write workload. Its interesting to compare that submission with the previous one for a number of reasons, but for the most part, this benchmark was about Flashcache effectiveness.

We did a number of other benchmarks including Spec-SFS benchmarks that also proved the remarkable effectiveness of the Flashcache technology, showing how it could make SATA drives perform as better than Fibre channel drives, or dramatically reduce the number of fibre channel drives required to service a given workload. There were a couple of other benchmarks done which I’ll grant were “hey look at how fast our shiny new boxes can run”, but for the most part, these were all done with configurations we’d reasonably expect a decent number our customers to actually buy (no all SSD configurations).

In the mean time EMC released some “Lab Queen” benchmarks, at first I thought that EMC were trying to prove just how fast their new X-blades were for processing CIFS and NFS traffic. They did this by configuring the back end storage system in such a rediculously overengineered way as to remove any possibility that they could cause a bottleneck in any way, either that or EMC’s block storage devices are way slower than most people would assume. From an engineering perspective I think they guys in Hopkington who created those X-blades did a truly excellent job, almost 125,000 IOPS per X-Blade using 6 CPU cores is genuinely impressive to me, even if all they were doing was  processing NFS/CIFS calls. You see, unlike the storage processors in a FAS or Isilon array, the X-Blade, much like the Network Processor in a SONAS system, or an Oceanspace N8500 relies on a back end block processing device to handle RAID , block checksums, write cache coherency and physical data movement to and from the disks, all of which is non-trivial work. What I find particularly interesting is that in all the benchmarks I looked at for these kinds of systems, the number of back end block storage systems was usually double that of the front end, which infers to me either that the load placed on back end systems by these benchmarks is higher than the load on the front end, or  more likely that the front end / back end architecture is very sensitive to any latency on the back end systems which means the back end systems get overengineered for benchmarks. My guess is after seeing the “All Flash DMX” configuration is that Celerra’s performance is very adversly affected by even slight increases in latency in the back end and that we start seeing some nasty manifestations of little law in these architectures under heavy load.

A little while later after being present at a couple of EMC presentations (one at Cisco Live, the other at a SNIA event, where EMC staff were fully aware of my presence), it became clear to me exactly why EMC did these “my number is bigger than yours” benchmarks. Ther marketing staff at corporate created a slide that compared all of the current SPC benchmarks in a way that was accurate, compelling and completely misleading all at the same time, at least as far as the VNX portion goes. Part of this goes back to the way that vendors, including I might say Netapp, use an availability group as a point of aggregation when reporting peformance numbers, this is reasonably fair as adding Active/Active or Active/Passive availability generally slows things down due to the two phase commit nature of write caching in modular storage environments. However, the configuration of the EMC VNX VG8 Gateway/EMC VNX5700 actually involves 5 separate availability groups (1xVG8 Gateway system with 4+1 redundancy, and and 4x VNX5700 with 1+1 redundancy). Presenting this as one aggregated peformance number without any valid point of aggregation smacks of downright dishonesty to me. If NetApp had done the same thing, then, using only 4 availabilty groups, we could have claimed over 760,000 IOPS by combining 4 of our existing 6240 configurations, but we didnt, because frankly doing that is in my opinion on the other side of the fine line where marketing finesse falls off the precipice into the shadowy realm of deceptive practice.

Which brings me back to my original question, what did we prove with our most recent submissions, well three things come to mind

1. That Netapp’s Ontap 8.1 Cluster mode solution is real, and it performs briliiantly

2. It scales linearly as you add nodes (more so than the leading competitors)

3. That scaling with 24 big nodes gives you better performance and better efficiency than scaling with hundreds of smaller nodes (at least for the SPEC benchmark)

This is a valid configuration using a single vserver as a point of aggregation across the cluster, and trust me, this is only the beginning.

As always, comments and criticism is welcome.

Regards

John

One Red Fly Green Dress with Puppy

There’s a lot of talk of “Big Data”, how it can help make businesses more efficient, uncover correlations between diet, income, location and and health outcomes, and advance science and human endeavor in thousands of ways. From a storage vendors perspective Big Data also changes some of the fundamental assumptions about the value of data storage and the architectures and asumptions of shared and network storage.

What I didn’t expect though was how relevant Big Data was to the small business owner. Last week my wife started Tivoli2moro, a  new online fashion business for girls (yes this is blatant plug, if you have a daughter between the ages of 8 – 15 or know someone who does, check it out, I’m very proud of what she’s achieved) . While doing the market research for things like Google Adwords, Facebook ads, and demographic trends the quality of the information she had at her fingertips truly surprised me. All of this information she had relied on years of data gathering from millions of data points, the infrstructure she leveraged would have cost millions of dollars, the storage requirement I suspect would be measured in hundreds of Terabytes.

While this kind of big data may not save lives, it does help change the competitive business landscape by giving small business access to the kinds of research data that would have been unimaginable until a few years ago. Similar access to other kinds of datasets may also change the way social activism and politics is run in the future, which makes me believe that helping people build these Big Data infrastructures really can help make make my wife happier, and this planet a better place to live, and that,  means the world to me.

A couple of days ago I saw an email asking “what is the recommendation for maximum capacity utilization that will not cause performance degradation”. On the one hand this kind of question annoys me because for the most part it’s borne out of some the usual FUD which gets thrown at NetApp on a regular basis, but on the other, even though correctly engineering storage for consistent performance rarely, if ever, boils down to any single metric, understanding capacity utilisation and its impact on performance is an important aspect of storage design.

Firstly, for the record, I’d like to reiterate that the performance characteristics of every storage technology I’m aware of that is based on spinning disks decreases in proportion to the amount of capacity consumed.

With that out of the way, I have to say that as usual, the answer to the question of how does capacity utilisation affect performance is, “it depends”, but for the most part, when this question is asked, it’s usually asked about high performance write intensive applications like VDI, and some kinds of online transaction processing, and email systems.

If you’re looking at that kind of workload, then you can always check out good old TR-3647 which talks specifically about a write intensive high performance workloads where it says

The Data ONTAP data layout engine, WAFL®, optimizes writes to disk to improve system performance and disk bandwidth utilization. WAFL optimization uses a small amount of free or reserve space within the aggregate. For write-intensive, high-performance workloads we recommend leaving available approximately 10% of the usable space for this optimization process. This space not only ensures high-performance writes but also functions as a buffer against unexpected demands of free space for applications that burst writes to disk

I’ve seen other benchmarks using synthetic workloads where a knee in the performance curve begins to be seen at between 98% and 100% of the usable capacity after WAFL reserve is taken away, I’ve also seen performance issues when people completely fill all the available space and then hit it with lots of small random overwrites (especially misaligned small random overwrites). This is not unique to WAFL, which is why it’s a bad idea generally to fill up all the space in any data structure which is subjected to heavy random write workloads.

Having said that for the vast majority of workloads you’ll get more IOPS per spindle out of a netapp array at all capacity points than you will out of any similarly priced/configured box from another vendor

Leaving the FUD aside, (the complete rebuttal of which requires a fairly deep understanding of ONTAP’s performance achitecture)  when considering capacity and its effect on performance on a NetApp FAS array it’s worth keeping the following points in mind.

1. For any given workload, and array type you’re only ever going to get a fairly limited number transactions per 15K RPM disk, usually less than 250
2. Array performance is usually determined  by how many disks you can throw at the workload
3. Most array vendors bring more spindles to the workload by using RAID-10 which uses twice the amount of disks for the same capacity, NetApp uses RAID-DP which does not automatically double the spindle density
4. In most benchmarks (check out SPC-1), NetApp uses all but 10% of the available space (in line with TR-3647) which allows the user to use approximately 60% of the RAW capacity  while still achieving the same kinds of IOPS/drive that more other vendors are only able to do using 30% of the RAW capacity. i.e at the same performance per drive we offer 10% more usable capacity than the other vendors could theoretically attain using RAID-10.

The bottom line is, that even without dedupe or thin provisioning or anything else you can store twice as much information in a FAS array for the same level of performance as most competing solutions using RAID-10

While that is true, it’s worth mentioning it does have one drawback. While the IOPS/Spindle is more or less the same, the IOPS density measured in IOPS/GB on the NetApp SPC-1 results is about half that of the competing solutions, (same IOPS , 2x as much data = half the density). While that is actually harder to do because you have a lower cache:data ratio, if you have an application that requires very dense IOPS/GB (like some VDI deployments for example), then you might not be allocate all of that extra capacity to that workload.  This in my view gives you three choices.

1. Don’t use the extra capacity, just leave it as unused freespace in the aggregate which will make it easier to optimise writes
2. Use that extra capacity for lower tier workloads such as storing snapshots or a mirror destination, or archives etc, and set those workloads to a low priority using FlexShare
3. Put in a FlashCache card which will double the effective number of IOPS (depending on workload of course) per spindle, which is less expensive and resource consuming than doubling the number of disks

If you dont do this, then you may run into a situation I’ve heard of  in a few cases where our storage efficiencies allowed the user to put too many hot workloads on not enough spindles, and unfortunately this is probably the basis for the  “Anecdotal Evidence”  that allows the Netapp Capacity / Performance FUD to be perpetuated. This is innacurate because it has less to do with the intricacies of ONTAP and WAFL, and far more to do with systems that were originally sized for a workload of X having a workload of 3X placed on them because there was still capacity available on Tier-1 disk capacity, long after all the performance had been squeezed out of the spindles by other workloads.

Keeping your storage users happy, means not only managing the available capacity, but also managing the available performance. More often than not, you will run out of one before you run out of the other and running an efficient IT infrastructure means balancing workloads between these two resources. Firstly this means you have to spend at least some time measuring, and monitor both the capacity and performance of your environment. Furthermore you should also set your system up to it’s easy to migrate and rebalance workloads across other resource pools, or be able to easily add performance to your existing workloads non disruptively which can be done via technologies such as Storage DRS in vSphere 5, or ONTAP’s Data motion and Virtual storage tiering features.

When it comes to measuring your environment so you can take action before the problems arise, NetApp has a number of excellent tools to monitor the performance of your storage environment. Performance Advisor gives you visualization and customised alerts and thresholds for the detailed inbuilt performance metrics available on every FAS Array, and OnCommand Insight Balance provides deeper reporting and predictive analysis of your entire virtualised infrastructure including non-NetApp hardware.

Whether you use NetApp’s tools or someone elses, the important thing is that you use them, and take a little time out of you day to find out which metrics are important and what you should do when thresholds or high watermarks are breached. If you’re not sure about this for your NetApp environment, feel free to ask me here, or better still open up a question in the Netapp communities which has a broader constituency than this blog.

While I appreciate that it’s tempting to just fall back to old practices, and overengineer Tier-1 storage so that there is little or no possibility of running out of IOPS before you run out of capacity, this is almost always incredibly wasteful and has in my experience resulted in storage utilisation rates of less than 20%, and  drives the costs/GB for “Tier-1″ storage to unsustainable and uneconomically justifiable heights. The time may come when storage administrators are given the luxury of doing this again, or you may be in one of those rare industries where cost is no object, but unless you’ve got that luxury, it’s time to brush up on your monitoring and storage workload rebalancing and optimisation skills. Doing more with less is what it’s all about.

As always, comments and contrary views are welcomed.

I’m in the middle of digesting what was actually released in EMC’s recent launch. For the most part there isn’t anything really that new: lots of unsupported hype like, “3 times simpler, 3 times faster.” Faster than what, exactly? From a technical perspective the only thing that’s really interesting or surprising is the VNXe and that was less interesting than I expected because I thought they were going to refresh their entire range using that technology. So it looks like they’ve given up trying to make that scale for the moment.

So much of what they’ve done copies or validates what we’ve already done at NetApp:

• Simplified software packaging
• Launching a lot of stuff at the same time
• New denser shelves with small form-factor drives
• An emphasis on storage efficiency
• An emphasis of flash as a caching layer
• The ideal match between unified storage and virtualized environments

The biggest change that I see is that they now appear to be shipping all their controllers with unified capability from the start, enabled via a software upgrade which is something EMC has criticised us for in the past. Now they acknowledge that the only way to compete with NetApp effectively is to try to be as much like us as they possibly can. This might explain why EMC in Australia isn’t going to sell the “Block only” VNX 5100. SearchStorage.com.au had this report:

EMC’s new VNX 5100 (pictured), a block-only storage device, won’t go on sale in Australia becaus “We did not see great enough demand to see that particular system,” according to Mark Oakey, the company’s Marketing Manager for Storage Platforms in Australia and New Zealand. “We’ll continue with the Clariion CX4 120,” he told SearchStorage ANZ. “It has more or less the same capabilities.”

Most of the interesting capabilities they’re touting came last year with FLARE 30 and DART 6.0 (two of their operating systems). Even the VMax stuff they’re pushing during the launch came out via a software upgrade without a lot of fanfare in December, so as far as I can see their “record breaking announcement” consists of announcing a whole bunch of things they’d already done along with some new tin.

Things they didn’t announce:

1. Multistore equivalency
2. V-Series equivalency
3. Unified replication capabilities
4. A commercial grade VMware based “Virtual Storage Array”- The new low end box is based on Linux
5. A scale out roadmap for their “Unified” platform
6. Any significant change in their management software strategy or offering
7. Block level deduplication for their unified arrays
8. Clarification on where their newly acquired scale out Isilon systems fit within their new “Unified” ecosystem.

Overall EMC did a catch up release to try and maintain pace with NetApp innovation, and nothing they’ve done or released represents a significant new threat. If this is

“the most significant midrange announcement in EMC’s 30-year history”

according toi Rich Napolitano, President, Unified Storage Division at EMC, then EMC will continue to play catch up as NetApp redefines Unified Storage and its role in shared infrastructure.

Preamble

During a discussion I had at the SNIA blogfest, I mentioned that I’d written a whitepaper around archiving and I promised that I’d send it on. It took me a while to get around to this, but I finally dug it out from my archive, which is implemented in a similar way to the example policy at the bottom of this post.,  it only took me about a minute to search and retreive it once I’d started the process of looking for it from a FAS array that was far enough away from me to incur a 60ms RTT latency. Overall I was really happy with the result.

The document was in my archive because I wrote it almost two years ago, since then a number of things have changed, however the fundamental principals have not, I’ll work on updating this when things less busy, probably sometime around January ’11. On a final note, because I wrote this a couple of years ago when my job role was different than it is today, this document is considerably more “salesy” than my usual blog posts, it shouldnt be construed as a change in direction for this blog in general.

Introduction

There are a number of approaches that can be broadly classified as some form of Archiving, including Hierarchical Storage Management (HSM), and Information Lifecycle Management (ILM).  All of these approaches aim to improve the IT environment by

• Lowering Overall Storage Costs
• Reducing the backup load and improving restore times
• Improving  Application Performance
• Making it easier to find and classify data

The following kinds of claims are common in the marketing material promoted by vendors of Archiving software and hardware

“By using Single Instance Storage, data compression and an ATA-based archival storage system (as opposed to a high performance, Fibre Channel device), the customer was able to reduce storage costs by $160,000 per terabyte of messages during the first three years that the joint EMC / Symantec solution was deployed. These cost savings were just the beginning, as the customers were also able to maintain their current management headcount despite a 20% data growth and the time it took to restore messages was drastically reduced. By archiving the messages and files, the customer was also able to improve electronic discovery retrieval times as all content is searchable by keywords.”

These kinds of results while impressive assume a number of things that are not true in NetApp implementations

• A price difference between primary storage and archive storage of over $US160,000 per TB.
• Backup and restores are performed from tape using bulk data movement methods
• Modest increases in storage capacities require additional headcount

In many NetApp environments, the price difference between the most expensive tier of storage and the least expensive simply does not justify the expense and complexity of implementing an archiving system based on a cost per TB alone.

For file serving environments, many file shares can be stored effectively on what would be traditionally thought of as “Tier-3” storage with high-density SATA drives, RAID-6 and compression / deduplication. This is because unique NetApp technologies such as WAFL and RAID-DP provide the performance and reliability required for many file serving environments. In addition, the use of NetApp SnapVault replication based data protection, for backup and long term retention means that full backups are no longer necessary. The presence or absence of the kinds static data typically moved into archives has little or no impact on the time it takes to perform backups, or make data available in the case of disaster.

Finally, the price per GB and IOPS for NetApp storage has fallen consistently in line with the trend in the industry as a whole. Customers can lower their storage costs by purchasing and implementing storage only as required. NetApp FAS array’s ability to non-disruptively add new storage, or move excess storage capacity and I/O from one volume to the other within an aggregate makes this approach both easy, and practical.

While the benefits of archiving for NetApp based file serving environments may be marginal, archiving still has significant advantages for email environments, particularly Microsoft Exchange. The reasons for this are as follows

1. Email is cache “unfriendly” and generally needs many dedicated disk spindles for adequate performance.
2. Email messages are not modified after they have been sent/received
3. There is a considerable amount of “noise” in email traffic (spam, jokes, social banter etc)
4. Small Email Stores are easier to cache, which can significantly improve performance and reduce the hardware requirements for both the email servers and the underlying storage
5. Email is more likely to be requested during legal discovery
6. Enterprises now consider Email to be a mission critical application and some companies still mandate a tape backup of their email environments for compliance purposes.

Choosing the right Archive Storage

It’s about the application

EMC and NetApp take very different approaches to archive storage, each of which works well in a large number of environments. An excellent discussion on the details of this can be found in the NetApp whitepaper WP-7055-1008 Architectural Considerations for Archive and Compliance Solutions. For most people however, the entire process of archive is driven not at the storage layer, but by the archive applications. These applications do an excellent job of making the underlying functionality of the storage system transparent to the end user, however the user is still exposed to the performance and reliability of the storage underlying the archives.

Speed makes a difference

Centera was designed to be “Faster than Optical” and while it has surpassed this relatively low bar, its performance doesn’t come close to even the slowest NetApp array. This is important, because the amount of data that can be pushed onto the archive layer is determined not just by IT policy, but also by user acceptance and satisfaction with the overall solution. The greater the user acceptance, the more aggressive the archiving can be, which results in lower TCO and faster ROI.

Protecting the Archive

While the archive storage layer needs to be reliable, it should be noted that without the archive application and its associated indexes, the data is completely inaccessible, and may as well be lost. It might be possible to rebuild the indexes and application data from the information in the archive alone, often this process may be unacceptably long. Protecting the archive involves protecting the archive data store, the full text indexes, and the associated databases in a consistent manner at a single point in time.

Migrating from an Existing Solution

Many companies already have archiving solutions in place, but would like to change their underlying storage system to something faster and more reliable. Fortunately archiving applications build the capability to migrate date from one kind back-end storage to another into their software. The following diagrams show how this can be achieved for EmailXtender and DiskXtender to move data from Centera to NetApp.

Some organizations would prefer to completely replace their existing archiving solutions including hardware and software. For these customers NetApp collaborates with organizations such as Procedo (www.procedo.com), to make this process fast and painless.

SnapVault

As mentioned previously, the cost and complexity of traditional archiving infrastructure may not add sufficient value to a NetApp file-serving environment, as many of the problems it solves are already addressed by core NetApp features. This does not mean that some form of storage tiering could not or should not be implemented on FAS to reduce the amount of NetApp primary capacity.

One easy way of doing this is by taking advantage of the flexibility of the built in backup technology. This is an extension of the “archiving” policy used by many customers, where the backup system is used for archive as well.  The approach of mixing backup and archive is rightly discouraged by most storage management professionals, the reasons for doing so in traditional tape based backup environments don’t apply.

The reasons for this are

• Snapshot and replication based backups are not affected by capacity as only changed blocks are ever moved or stored
• The backups are immediately available, and can be used for multiple purposes
• Backups are stored on high reliability disk in space efficient manner using both non-duplication and de-duplication techniques
• Files can be easily found via existing user interfaces such as Windows Explorer or external search engines

In general, SnapVault destinations use the highest density SATA drives with the most aggressive space savings policies applied to them. These policies and techniques, which may not be suitable for high performance file sharing environments, provide the lowest cost per TB of any NetApp offering. This combined with the ability to place the SnapVault destination in a remote datacenter may relieve the power, space and cooling requirements of increasingly crowded datacenters.

An example policy

Many companies file archiving requirements are straightforward, and do not justify the detailed capabilities provided by archiving applications. For example, a company might implement the following backup and archive policy

• All files are backed up on a daily basis with daily recovery points kept for 14 days, weekly recovery points will be kept for two months and monthly recovery points kept for seven years.
• Any file that has not been accessed in the last sixty days will be removed from primary storage and will need to be accessed from the archive

This is easily addressed in a SnapVault environment through the use of the following

• Daily backups are transferred from the primary system to the SnapVault repository
• Daily recovery points (snapshots) are kept on both the primary storage system and the SnapVault repository for 14 days
• Weekly recovery points (snapshots) are kept only on the SnapVault repository
• Monthly recovery points (snapshots) are kept only on the SnapVault repository
• A simple shell script/batch file is executed after each successful daily backup which deletes any file from the primary volume that has not been accessed in thirty days
• Users are allocated a drive mapping to their replicated directories on the SnapVault destination.
• Optionally the Primary systems and SnapVault repository may be indexed by an application such as the Kazeon IS1200, or Google enterprise search.

Users then need to be informed that old files will be deleted after thirty days, and that they can access backups of their data, including the files that have been deleted from primary storage by looking through the drive that is mapped to the SnapVault repository, or optionally via the enterprise search engines user access tools.

By removing the files from primary storage, instead of the traditional “stub” approach favoured by many archive vendors, the overall performance of the system will be improved by reducing the metadata load, and users will be able to more easily find active files by having fewer files and directories on the primary systems.

Conclusion

Many Organisations archiving requirements can be met by simply adding additional SATA disk to the current production system replicated via SnapMirror to the current DR system – rather than managing separate archive platforms.

This architecture provides flexibility and scalability over time and reduces management overhead. Tape can also be used for additional backup and longer term storage if required. SnapLock provides the non-modifiable WORM like capability required of an archive without additional hardware (a software licensable feature, see more detail at http://www.netapp.com/us/products/protection-software/snaplock.html ).

During some recent discussions on Twitter, the subject of disk drive rebuild times for very large drives in excess of 10TB has raised the subject of urecoverable read errors also known as UER, which is sometimes blamed on something called  “bit rot”  however,  two NetApp sponsored studies shows that bit rot is far less of a problem for storage array reliability than many other factors.

The best publically available data on bit rot and it’s impact compared to other causes I’ve found is contained in “A Highly Accurate Method for Assessing Reliability of Redundant Arrays of Inexpensive Disks (RAID) by Jon G. Elerath and Michael Pecht  in IEEE TRANSACTIONS ON COMPUTERS, VOL. 58, NO. 3, MARCH 2009 http://media.netapp.com/documents/rp-0046.pdf”. The following information summarizes and paraphrases the information found in that document.

What Bit rot is and why you should care

Bit rot is a concern for two main reasons, for the home user with no RAID protection, it results in the inconvenience of a lost or corrupted file, or possibly a machine that wont boot, for the enterprise user, bit rot raises the specter, not just of a lost or corrupted file, but of the potential to completely lose an entire RAID group after the failure of a single drive due to the “Media Error on Data Reconstruct” problem. The less catastrophic issue on a enterprise calss array is far less because the additional error detection and correction available through the use of RAID and block level checksums means the chances of bit rot causing the loss or corruption of a file is vanishingly remote.

What I believe most people mean by bit rot, could be more accurately described as latent media errors rather “bit rot” which is more strictly caused by degradation of the magnetic properties of the media.

The reason for this is that most early RAID reliability models assumed that data will remain undestroyed except by “bit rot”. Although it is correct that the magnetic properties of the media can degrade, this failure mechanism is not a significant cause. Data can become corrupted any time the disks are spinning, even when data are not being written to or read from the disk.  The failure mechanisms outlined below here are not unknown, but neither are they readily available from HDD manufacturers

Common Causes for losing data

Four common causes for losing data after its been correctly written are “Thermal asperities”, scratches and smears, and corrosion.

• Thermal asperities are instances of high heat for a short durations caused by head-disk contact. This is usually the result of heads hitting small “bumps” created by particles embedded in the media surface during the manufacturing process. The heat generated on a single contact may not be sufficient to thermally erase data but may be sufficient after many contacts.
• Although disk heads are designed to push particles away, but contaminants can still become lodged between the head and disk, hard particles used in the manufacture of an HDD, can cause surface scratches and data erasure any time the disk is rotating.
• Other “soft”materials such as stainless steel can come from assembly tooling. Soft particles tend to smear across the surface of the media, rendering the data unreadable.
• Corrosion, although carefully controlled, can also cause data erasure and may be accelerated by thermal asperity generated heat

Why data is sometimes not there in the first place

A latent defect can also be caused by data that was incorrectly, or incompletely written to the disk in the first place, this can happen, this can happen because of the inherent “Bit Error Rate” or BER, writing to damaged media, or too much lubrication and “high-fly writes”

• The bit error rate (BER) is a statistical measure of the effectiveness of all the electrical, mechanical, magnetic, and firmware control systems working together to write (or read) data. Most bit errors occur on a read command and are corrected, but since written data are rarely checked immediately after writing, bit errors can also occur during writes.

• BER accounts for a fraction of defective data written to the HDD, but a greater source of errors is the magnetic recording media that coats the disks. Writing on scratched, smeared, or pitted media can result in corrupted data. The reasons for scratches and smears where covered earlier, however “pits and voids are caused by particles that were originally embedded in the media during the manufacturing process and subsequently dislodged during the polishing process or field use.

• The final common cause for poorly written data is the “high-fly write.” The heads are aerodynamically designed to have a negative pressure and maintain the small, fixed distance above the disk surface at all times. If the aerodynamics are disturbed, the head can fly too high, resulting in weakly (magnetically) written data that cannot be read. In addition to “wind gusts” inside the disk, all disks have a very thin film of lubricant on them to help protection from head-disk contact. While this lubrication helps mitigate the effects of “thermal asperities”, lubrication build-up on the head can increase the flying height, resulting in weak or incomplete writes.

Where’s my data ?

Finally, all the data may have been written correctly, but the disk may not be able to “find” it, because of damage to special “servo” tracks which help keep the heads correctly aligned to the data on the disk. In some cases, it’s not damage to the servo tracks but wear and tear on the motor and disk head bearings, noise, vibration and other electromechanical errors can cause the head positioning to take too long to lock onto a track which ultimately also causes “latent block errors”

How to protect yourself

There are two main ways of dealing with these kinds of latent block errors, the first is to perform disk scrubs, which is something every reputable array vendor does, the problem is however that as disk sizes get larger and larger, the time taken to perform a full disk scrub can take too long for the protection to be as effective as it should. The other method is to use additional levels of RAID protection such as RAID-6 which allows for higher levels of resiliency and error correction in the event of hitting a latent block error when reconstructing a RAID set. NetApp uses both approaches as studies have shown that the risk of losing data through these kinds of events is thousands of times higher than predicted by most simple “MTBF” failure models.

This is an edited copy of a comment I posted on storagezilla’s blog (the second of two which said more or less the same thing, the comment moderation there doest seem to indicate whether the comment is posted or queued).  There was a secondary request to EMC which dilutes the essence of the apology and on reflection and feedback from other’s I’ve decided to remove it.

You’re right, it was 62 vBlock accounts not 62 vBlock’s sold as I tweeted

“1 Year in and just over 60 V-Blocks sold. I’ll wager that there will be many more FlexPod deployments in 12 months time”

and

“@DanMoz 63 in production or deployment according to the figures I saw. But you are right, the concept has been sold well.”

Thanks for pointing out the inaccuracy of these statements (though inferring that I’m dumb or a liar is a little harsh), and I fully recant/withdraw the comment and apologise for the dumb error, both here, on twitter where I made the statement, and on my own blog. I will be more careful in the future.

<Request removed .. JM >

Let the truth prevail.

Regards John Martin

@life_no_borders

On a third reading of this, even my apology was inaccurate, I claimed 63 vBlocks had been sold, not 62 … d’oh ! Time to get more sleep and up my game.