Release the Kraken! An open-source pub/sub server for the real-time web

Asana Kraken

Today, we are releasing Kraken, the distributed pub/sub server we wrote to handle the performance and scalability demands of real-time web apps like Asana.

What problem does Kraken solve?

One of the key promises of real-time web applications is that users will see changes made by other users as they happen, without hitting reload or refresh. Many of the up-and-coming reactive web frameworks initially accomplish this by periodically re-executing queries against the database. When we were building the early prototypes of Asana, our reactive framework (Luna) was no different.

Unfortunately, most production-ready databases, from Mongo to Mysql, are unable to keep up with the query volume that even a moderate userbase creates when you frequently poll the database for changes. We recognized this problem early, and worked around it by designing an algorithm that incrementally updates queries in real time without putting any additional load on the database server. Key to this solution is the requirement that every query be notified about changes to objects that could potentially match, or stop matching the result set of each query.

That’s where Kraken comes into play. Kraken is responsible for routing data-invalidation messages between the processes that are running these queries so that they stay up-to-date.

Why is Kraken the right solution?

Before building Kraken, we searched for an existing open-source pub/sub solution that would satisfy our needs. At the time, we discovered that most solutions in this space were designed to solve a much wider set of problems than we had, and yet none were particularly well-suited to solve the specific requirements of real-time apps like Asana. Our team had experience writing routing-based infrastructure and ultimately decided to build a custom service that did exactly what we needed – and nothing more.

The decision to build Kraken paid off. For the last three years, Kraken has been fearlessly routing messages between our servers to keep your team in sync. During this time, it has yet to crash even once. We’re excited to finally release Kraken to the community!

You can get Kraken yourself at Github

Would you recommend this article? Yes / No
  • Kamran Mackey
    Does that mean we will be able to contribute to it? :)
  • Scott Andreas
    Very clean code, nice no-surprises OTP design, and excellent comments throughout – this looks like an excellent project. Nice work, and congrats on releasing it!

    – Scott

  • jesse
    Good to see another great project in Erlang.
  • Tevya
    By why did you name it “Kraken”??? Seems to inspire very different emotions than “Asana.” :)
    • Jason Diller
      It’s their drink of choice at the office. We love the Kraken too!

      Cheers my friends.

  • Arek
    I wonder if it is possible that in future you can offer offline apps for desktops (maybe Kraken helps to manage that in terms of synchronization?)
    • Kamran Mackey
      Arek: it would take the Asana team weeks/months to develop desktop apps for Asana. I think they are just focused on making improvements and adding new features to the Web/iPhone/Android apps right now.
  • Pingback: Scott Banwart's Blog › Distributed Weekly 197()

  • Paul
    Awesome’s a word I’ve seen a lot around here… Once again, awesome job guys!
  • Geneva Sopko
    Spot on with this write-up, I really suppose this web site wants way more consideration. I�ll most likely be once more to learn way more, thanks for that info.
  • Pingback: Infrastructure at Asana: what’s next?()

  • Pingback: Asana: Alle taken van een organisatie op één plek()