Share this post

October 12, 2015

Atlantis Stretched Cluster

Priyadarshi Prasad - Atlantis


There is a saying popular among storage solution architects that goes something like this — everyone wants zero downtime and zero data loss until they see the check.

In storage speak, an HA & DR solution is architected around two key concepts: Recovery Point Objective (RPO) and Recovery Time Objective (RTO).

RPO/RTO vs. Cost

The ideal scenario would be a solution with zero RPO (no data loss) and zero RTO (no downtime), and the good news is that it is possible. The solution is called stretched cluster, wherein a single cluster is deployed across two data centers, typically at a metropolitan distance (hence also called metro cluster solution). If properly implemented with the right products, such a solution protects against any data loss in an event of a disaster. Furthermore, the failover from one site to the other happens automatically (no manual intervention).

It is truly a nice solution — highly available. The problem — you guessed it — is often the prohibitive cost of such a solution.

With Atlantis Stretched Cluster, we have brought a solution that delivers high availability without the associated high costs. The secret sauce is the USX architecture that lends deduplication-awareness to Stretched Cluster capability — fundamentally reducing the cost of this solution at multiple levels.

To understand how we did it, let’s take a look at the solution itself:

Atlantis Stretched Cluster

Atlantis Stretched Cluster adds a layer of high availability on top of the natively resilient USX solution. With Stretched Cluster now, customers can tolerate failure of an entire datacenter without experiencing any downtime on their applications and without worrying about data loss.

The solution depicted above protects against:

  • Component failure (Drives, Nodes, Volumes) ➞ Handled by Atlantis native resiliency

  • Network failure b/w data centers ➞ Handled by Atlantis Stretched Cluster

  • Site failure ➞ Handled by Atlantis Stretched Cluster

    • Failover is automatic with no manual intervention required

USX Stretched Cluster is a three-site solution. Site 1 and Site 2 are in synchronous mirror relationship with each other and on the third side, we deploy a Tiebreaker. While the Tiebreaker is nothing but a simple VM, it is an important piece of the solution that provides the tier-1 availability and predictability customers seek from such solutions.

The Secret Sauce

So back to the original challenge — how did we reduce the cost of stretched cluster solution? To understand this, let’s dissect the solution components that go in building such a solution from a storage standpoint. There are three major components:

  1. Site 1: Includes cost of compute, hypervisor, storage and local networking

  2. Site 2: Similar to Site 1

  3. Network bandwidth between the sites

With Atlantis’ famous inline deduplication, we routinely deliver 70% or more capacity savings to customers. So deploying Atlantis SDS or hyper-converged solution can massively reduce customers’ storage spend on Site 1 and Site 2. Things become even more interesting when you look at (3) — network bandwidth.

You see, one of the main requirements for stretch cluster solution is that the two sites need to be in a synchronous mirror relationship with each other. Makes sense, since that guarantees that there is absolutely no loss of data in the event of a disaster. However, it also means that every single write that happens on one site needs to be transmitted to the other. In other words, stretched cluster hogs massive amounts of bandwidths — forcing customers to dedicate leased high speed lines between the sites, and pay through the noses for bandwidth consumed every month.

This is where Atlantis Stretched Cluster shines by making its synchronous mirroring deduplication-aware. In other words, only unique blocks of writes are sent over the wire. For all duplicate writes (conservatively around 70% of writes), only metadata updates are being sent over. That’s huge in terms of savings from a bandwidth standpoint, and can easily cut the bandwidth requirements in half. That’s 50% less that customers will have to pay every month for their leased line.

Moreover, dedup-awareness comes in handy during resync operations (after a failover) as well. During resync, we only send changed unique blocks and metadata updates. This makes the resync light on the network and also fast, restoring high availability to the overall solution quickly.

A good architecture is a gift that keeps on giving, and Atlantis USX is certainly proving that point.

Make Failover the last option, not the first

Failing over from one site to another is expensive from a customer standpoint due to its operational implications. So our natively resilient design resorts to site failover only as the last option, while still assuring availability and data integrity. We accomplish this by keeping our failover granular to a Volume level.

Consider some situations:

Volume Failover Scenario

  1. USX Volume failure — We simply failover that volume to its corresponding HA Volume on the remote site. All operations continue with no additional failover necessary.

  2. Node/Component failure — USX native resiliency kicks in and ensures that other nodes/components in the cluster handle that node failure. No failover necessary.

  3. Link b/w Site 1 and Site 2 fails — No failover is necessary since the Tiebreaker can still communicate with both the sites.

  4. Link b/w Tiebreaker and one (or both) of the sites fails — No failover necessary since the two sites can communicate with each other.

  5. One of the sites goes down or gets isolated — In this scenario, Tiebreaker would let the surviving site take over as expected. The site that is isolated, if still up, will detect loss of quorum and will gracefully shutdown.

  6. All sites get isolated — If all communication goes down, then we freeze both sites by default (with an option to keep them both running) and let the administrator choose one of the sites to be up and running. This situation is extremely rare since it requires that all networks to go down at the same time. In a practical scenario, the failure/network loss is cascaded and gives one of the sites enough time to take ownership and keep serving IOs.

Great Stuff — When is it available and for how much

Atlantis Stretched Cluster is available as part of our USX 3.1 release. And true to our form, this capability is available at no extra license. Our entire software stack comes with a single all-inclusive license — no nickel and diming with editions like Advance, Ultimate etc.

Finally in the true competitive spirit, here is my take on the competitive picture around stretched cluster implementations. I have tried to be factual, and have also reviewed this with industry veterans for accuracy. However should there be any inaccuracy, please let me know and I will be happy to correct it.

Stretched Cluster competitive landscape

We will be holding a webinar describing this capability more in detail on Oct 22, 2015 at 9AM Pacific time. Register now.

For more discussions, always available at:

@AtlantisSDS, or @Priyadarshi_Pd
Current rating: 3.8 (8 ratings)