Issues Moving to Amazon’s Elastic Load Balancer

Kris Rasmussen

Many of you noticed that the Asana service was occasionally unavailable for brief periods of time, lasting less than one minute, on Thursday and Friday last week. We apologize for the inconvenience these connectivity issues caused, and want to let you know what we are doing to prevent similar issues from occurring again in the future.

Asana’s infrastructure runs almost entirely on top of Amazon Web Services (AWS). AWS provides us with the ability to launch managed production infrastructure in minutes with simple API calls. We use AWS for servers, databases, monitoring, and more. In general, we’ve been very happy with AWS.

A month ago, we decided to use Amazon’s Elastic Load Balancer service to balance traffic between our own software load balancers. We did this for two reasons:

Scale: To evenly distribute requests between our own load balancers.

Reliability: To ensure that a single server failure does not result in a percentage of requests failing until we fix the server.

The Elastic Load Balancer was accomplishing both of these goals for nearly a month when it suddenly stopped forwarding all HTTP requests to our servers. The issue lasted less than a minute, but it was long enough to trip our monitoring system and briefly disrupt the workflow of users who have come to rely on Asana to get work done.

The first time this occurred, we assumed it was a random hiccup that was unlikely to happen again. When it occurred a second time, we got in touch with Amazon for assistance. Amazon thought they resolved the underlying problem, but it re-occurred twice more later that night and early in the morning on the following day. At that point, we decided to replace the Elastic Load Balancer with DNS Round Robin. Since doing so, the problems have gone away completely.

DNS Round Robin isn’t without its own set of issues, but none of them should impact your ability to access our service. We hope to use the Elastic Load Balancer again in the future, but not until Amazon provides us with enough information to properly diagnose the problem and address it.

  1. avatarFMartinez

    Hi
    Do you still use your own round robin engine ? If yes, would you mind let me know what you use ?
    Best Regards
    FMartinez

  2. avatarKris Asana Team Member

    Hey FMartinez,

    We just use DNS round robin, which is part of DNS itself. It does not require any special software on our part. Your DNS provider just needs to allow you to specify multiple DNS records for the same name entry.

    Best,
    Kris

  3. avatarJay fellows

    Are you having issues with the service this morning.

    I login, see my work spaces and projects but nothing shows up in the projects including tasks.

    Please advise.

    Jay fellows

  4. avatarDaniel

    Hm, I’m having a DNS problem right now, in October, so it seems like something isn’t working. app.asana.com is giving me a DNS error and has been for about 90 minutes so far! Is this possibly related?

  5. avatarJFK Airport Limo Service

    Air port Barro Assistance NEW YORK CITY Cape Assistance JFK Airport terminal Limousine Program LaGuardia Limousine Program Via Newark Flight terminal

Leave a comment