Oct
20

Clarifying AMQP

By kellabyte  //  Architecture  //  16 Comments

AMQP (Advanced Message Queuing Protocol) is an open standard protocol for message-oriented middleware. Like most standards, the direction has taken a few twists and turns during the process of standardization and is in the final stages of being ratified as AMQP 1.0 within the OASIS body. From there AMQP will move on to become an international standard through ISO/IEC.

The biggest contributor to the confusion around AMQP is due to the fact AMQP took a drastic turn in direction between 0.9.1 and 1.0.

Why does AMQP Matter?

Like any standard (remember WS-*?) the AMQP standard may not affect or matter to you but for some it will be important to understand. I’ve been informed that specific industries will require AMQP 1.0 which means if you are already or want to do business in these areas you should get informed about this standard and your available options.

Example

Let’s create a fictional situation (I’m not proposing this is how you would solve this problem) where we want to send log messages to 2 different services. One service will receive critical error messages to dispatch to support staff while another service will receive warnings and info messages for dashboard displays. The composition of the messaging topology may look something like this.

 

broker-01

 

With this diagram we can highlight the areas the different AMQP specifications take place.

AMQP 0.9.1

Up to AMQP 0.9.1 the AMQP group was focused on standardizing client to broker wire level protocol and broker messaging capabilities.

Messaging capabilities

The message broker capabilities allow composing message routing using exchanges, queues and bindings.

Exchanges receive messages from client applications via the network wire-level protocol and route messages to queues based on the routing rules defined by bindings. Routing rules using bindings are created on message properties or content.

Message queues receive and store messages from clients or exchanges. Messages are removed from a client pulling off the queue.

Bindings define the routing rules and relationship between an exchange and message queues.

This is called the “AMQ model” which attempts to provide a standard for the following:

  • Guarantee interoperability between conforming implementations.
  • Provide explicit control over the quality of service.
  • Complete configuration of server wiring via the protocol.

The AMQP 0.9.1 specification has commands for creating and managing exchanges, queues and bindings.

Highlighted in red are the messaging capabilities (AMQP model) defined in AMQP 0.9.1.

amqp-0-9-1-messaging-capabilities

 

 

Network wire-level protocol

This portion of the standard defined client to broker communications. The standardization of the wire-level protocol attempts to provide the following:

  • Compact, using a binary encoding that packs and unpacks rapidly.
  • Messages of any size without significant limit.
  • Multiple channels across a single connection.
  • Long-lived, with no significant in-built limitations.
  • Asynchronous command pipe-lining.

Highlighted in red is the network wire-level protocol.

amqp-0-9-1-network-wire-level-protocol

 

 

In summary, AMQP 0.9.1 describes the bytes going over the wire and the API’s available for configuring the message routing and quality of service of the broker.

    AMQP 1.0

    The AMQP group decided to go down a very different direction than what had been done up to 0.9.1 before ratification and because of that AMQP 1.0 is entirely different. Some message-oriented middleware had already been introduced into the market that was built conforming to 0.9.1 and previous versions of the standard. A few examples of this are RabbitMQ, Apache Qpid, StormMQ and RedHat Enterprise MRG (based on Apache Qpid).

    This change has created a pretty big divide in the community from my observations. Some people involved in earlier versions consider it a big step backwards while others consider it a simplification to build on that introduces more flexibility (more on that later).

    AMQP 1.0 is a much more narrow standard than the broad standards in the previous versions. It only defines the network wire-level protocol for the exchange of messages between two endpoints. The standard supports exchanging messages in peer-to-peer or by brokered topologies.

    The same protocol can be used to communicate client-to-client, client-to-broker or broker-to-broker with all of the delivery guarantees available.

    AMQP 0.9.1 heavily influenced broker internals. AMQP 1.0 does not define the internals of the broker implementation and can be used with any broker. This means that existing 0.9.1 implementations (like the ones mentioned above) can add the AMQP 1.0 network wire-level protocol support. An example is an experimental plugin for RabbitMQ that provides AMQP 1.0 endpoint support even though RabbitMQ is an AMQP 0.9.1 based broker.

    The AMQP 1.0 network wire-level protocol is not compatible with the AMQP 0.9.1 network wire-level protocol.

    Important part lost

    In AMQP 0.9.1 the protocol itself has support for creating and managing exchanges, queues and bindings. This means the broker behaviours are fixed and limited by what the standard defines.

    In AMQP 1.0 the broker management is shifted from a protocol-level command to a message you transport over a standard message. This means brokers can define their own management commands that any AMQP client library can send. Broker capabilities do not have to be part of the standard that clients must implement to be compatible.

    Since AMQP 1.0 defines how to communicate to an endpoint and has the concept of an address, a broker could define an XML or JSON message to send to this address for broker management commands such as creating exchanges, queues and routing behaviour.

    In theory, if FooMQ was a broker that was AMQP 1.0 compliant you could send a message to amqp://hostname/queues/bar-queue that was a JSON message with the command to create a new durable queue named “bar-queue”.

    There will be AMQP 1.0 related projects and products announced and in the market soon.

    Summary

    In my opinion, by AMQP 1.0 moving the broker specifics out from the protocol level makes it simpler to write client libraries and for these libraries to have increased longevity. Brokers also gain flexibility.

    This move is similar to what we see in popular HTTP API’s today (RESTful ones included) versus WS-* of yesterday. Less standardizing of the protocols themselves and more generic communication and loose coupling. Tighter coupling won a few battles but loose coupling won the war.

    A wise person once told me history has shown loose coupling usually wins.

  • Jason Williams

    By reducing AMQP to a glorified transport, it encourages fragmentation amongst brokers, particularly the commercial ones to ensure lock in.

    Sure it reduces the load on client writers because it pushes that load on to the app writer. They now have to care what broker they’re talking to.

  • http://www.swiftmq.com Andreas Mueller

    Great article. Let me add a few points.

    1) AMQP 1.0 adds some important features which are missed in 0.9.1, e.g. Session windows for outstanding acks, producer and consumer flow control via link credits, different message types (not only binary), support for message encryption, pluggable message selectors.

    2) As you pointed out, AMQP 1.0 throws away the broker behavior specification. This is very important as it prevented existing messaging middleware (such as JMS brokers) in the past to implement AMQP because it required to implement these exchanges. Now SwiftMQ implements 1.0 and Windows Azure ServiceBus and ActiveMQ will soon implement it. This will create the required momentum.

    3) We will see a lot of AMQP 1.0 libraries in future which can be used to implement brokers and peer-to-peer clients. One example is Proton of Apache Qpid.

  • Kyle

    This change severely reduces the utility of AMQP. Part of the selling point to the original AMQP was that you could take any AMQP provider and swap it for any other one. If I’m using RabbitMQ and decide I want to use StormMQ I don’t need to make any changes to my code, just spin up a new instance of StormMQ and adjust some configuration properties. With this change we’re going to get into a situation like SQL faces, where everyone implements AMQP, but that doesn’t actually mean anything and you’re still going to need an extra translation layer to sit inbetween your AMQP service and your app. This ISN’T LOOSE COUPLING, it’s tight coupling because now my app has to care not just that it’s talking to something with AMQP, but WHICH PARTICULAR FLAVOR of AMQP this service supports. Just because the details of interacting with the service are no longer part of the protocol doesn’t mean we don’t have to worry about them, it just means the protocol provides fewer guarantees, and thus the service provider and the client now have to deal with them in an arbitrary and inconsistent manner.

    The proper way to handle this would have been to split the standards in two. Leave the original AMQP standard the way it was, but define a new AMQP Wire standard. Then in the AMQP standard simply specify that clients and services should communicate using the new AMQP Wire standard.

    We were moving towards a single message standard that everyone could agree on, but now with this change we’re back to nobody being compatible with each other and being locked into a particular flavor of message queue.

  • http://www.swiftmq.com Andreas Mueller

    Your comment is wrong.

    The only implementation of 0.9.1 is RabbitMQ (Qpid is at 0.10). So 0.9.1 is actually a proprietary RabbitMQ protocol. Not only this – RabbitMQ has made “extensions” to the protocol like publisher-nack (simply by modifying the protocol XML declarations!!) plus at least one different data type interpretation. This locks you even more into that implementation. So from this standpoint I really understand why the RabbitMQ folks don’t want 1.0. ;-)

    On the other hand, AMQP 1.0 is a true vendor independent standard. We do a lot of interop tests (between Qpid, Proton, Windows Service Bus, SwiftMQ) and you are free to see one live on stage at ApacheCon Europe. This works fine without changes. All we need to do to connect to an AMQP endpoint from a different vendor is to change hostname and user credentials.

    AMQP 1.0 standardizes message exchange between 2 endpoints. We have also prepared an AMQP Global Addressing standard (not yet published) which will enable you to route messages through different implementations and so make AMQP 1.0 federations and global routing possible.

    So everything is there, no lock-ins.

  • http://qpid.apache.org Gordon Sim

    Thank you for your excellent & thoughtful article!

    The rules of interaction between parties in reliable, flow-controlled exchange of well defined messages is a distinct concern. The handling of messages when one party is an intermediary – a broker, a proxy or relay, a bridge etc – is another. Standardising the former provides *interoperability*, the crucial foundation that enables a higher level of network to emerge; standardisation efforts focused on the latter are aimed at defining interchangeable components from which such networks can be composed.

    The evolution of AMQP can be seen as increasing refinement and separation of these abstractions. The 1.0 specification as published actually covers both, though the focus is unsurprisingly on laying down the interoperable foundation. Further standardisation around management of brokers and ‘broker behaviour’ is anticipated, but it is worth pointing out that SwiftMQ and Qpid have demonstrated interoperability over the range of functionality required by JMS (something that had not been done prior to 1.0). This did require the use of extensions to the standard for ‘no-local’ filtering and selectors. These features have required extensions in previous versions also however. Further there is now a process by which proposed extensions such as these are registered to promote interoperability pending any further standardisation work. (Extension filters have also been registered that correspond with the matching algorithms as defined by the pre 1.0 exchange ‘types’).

    So I would argue that AMQP 1.0 is not narrower than previous versions; it is simply less restrictive.

    Both models are defined in such a way as to allow extensible behaviour. Where earlier versions centred around the ‘exchange’ as the core abstraction, 1.0 defines a more generic (and more symmetric) ‘node’ abstraction that is not limited simply to a matching algorithm. This allows greater flexibility and richer behaviour.

    I believe AMQP 1.0 will see greater adoption, will achieve greater interoperability and will be applicable to a wider set of use cases. Change is not always easy or universally popular, but sometimes it is necessary. I am confident that users of previous versions of the protocol will be well taken care of by communities such as Qpid and RabbitMQ, enabling transition where possible and supporting older protocols where it is not.

  • Kyle

    @Andreas
    The problem is, as part of most applications using AMQP they’re naturally going to be doing a bunch of configuration on startup to make sure the MQ service is configured with all the right queues and exchanges. In the pre-1.0 version, when this was all part of the standard, the application didn’t need to worry about which AMQP service it was talking to (even if RabbitMQ was the only one that supported it at the time, in theory others could have as well). Now that that configuration is no longer part of the standard suddenly applications DO need to care about which service they’re talking to, which means we now need a new translation layer that tries to detect which MQ service we’re talking to, and translate the exchange and queue configuration messages into the appropriate flavors for that version of AMQP service.

    The situation we’re going to be in now is going to be like the situation with SQL, we’re going to need something like Hibernate that sits inbetween our applications and the AMQP service to translate the configuration messages into the proper flavor since AMQP no longer makes any guarantees about what’s supported.

    As for routing between AMQP services, I honestly don’t care a bit about that, it’s a nice feature I suppose, but if you’re still trying to work out HOW to even manage the configurations, getting two different services talking with each other is fairly low down the priority list.

    If the desire was to decouple the wire protocol from the configuration protocol, that’s fine, but it should have been handled by splitting the standard in two, not by dropping half the standard and hoping someone else picks up the slack.

  • http://qpid.apache.org Gordon Sim

    @Kyle,

    I think there are two categories of use there. In one, broker entities (such as queues perhaps) are created on demand by applications sending or receiving messages from them. In the other, you want to programmatically ‘pre-configure’ or provision a broker.

    The former is actually supported by the core protocol – nodes can be dynamically created when they are linked to (this is how temporary queues work for example). This is actually the primary reason declare_queue/declare_exchange was added to earlier versions of the spec (hence declare rather than create).

    Admittedly the 1.0 approach doesn’t fit as well with the latter case. However pre-1.0 was in fact quite limited in its ‘management capabilities’, 0-9-1 especially so (there is no way to query what exchanges there are, or what the type of a given exchange is, or whether a queue is bound to an exchange). The standard model was also very limited (no standard policies for limiting queue size for example, inadequate and confusing notions of ‘in-use’ for automatically cleaned up resources etc).

    Standardised management *is* anticipated and is certainly valuable. In my view though it makes more sense to layer that, and take a slightly more holistic view of management.

  • http://www.swiftmq.com Andreas Mueller

    With AMQP 1.0 you have just addresses and your clients will be portable among different AMQP 1.0 implementations. This is the key point. You are portable. Write once, run with anyone.

    Wiring broker configurations inside a client is not a clean approach. That’s why JMS left that outside the spec. These entities (queues, topics, connection factories) are called administered objects and are handled by broker admin tools because they are too broker specific. And JMS is currently the only messaging standard that works.

    However, even the management and monitoring will be standardized by AMQP 1.0. You will communicate with “services” (if those services are offered) through exchange of standardized messages.

  • Brett

    This may be of interest: http://www.google.com/trends/explore#q=rabbitmq%2C%20qpid%2C%20zeromq%2C%20swiftmq%2C%20amqp&cmpt=q. Plenty of scope for robust discussion there I think.

  • Pingback: Distributed Weekly 178 — Scott Banwart's Blog

  • John O’Hara

    Thank you for a well written, balanced article.

    I’m the person that started the AMQP effort back around 2003, and I co-chair the OASIS AMQP group.

    AMQP is the work of dozens of people working diligently and closely together across companies. The AMQP crew really are outstanding individuals and it has been a privilege and a pleasure to work with them.

    With AMQP 1.0 we have:
    1) A big community that works really well together at OASIS.
    2) An industrial-grade wire transport for messaging
    3) A protocol that can be retro-fitted onto existing MOM’s and enable migration
    4) It will be a Standard, and big customers will be asking for AMQP1.0 by name
    5) A way to make “exchanges and bindings” work with 1.0 (the “Link” construct is the key)
    6) Future work to standardize API’s, SOAP/AMQP, management and profiles of broker behavior

    AMQP has in some way been a part of millions of people’s lives already; they just don’t know it! That’s a theme that’s only going to get bigger.

    One person didn’t want to come on that journey. The most vocal opponent of AMQP is Pieter Hintjens. I hired Pieter’s company to work with my expert team in the early days. I wanted to make sure that AMQP was a group effort from the beginning. However, as the size of the AMQP community grew and Pieter’s contributions were diluted he became very bitter and withdrew. His passionate retaliatory effort has now itself forked into Crossroads IO (http://www.infoq.com/news/2012/03/Crossroads-IO).

    AMQP is all about robustly connecting with others.
    John

  • Pieter Hintjens

    Dear John, you called my name? And three times in a row?
    Beware of what you call for. Well, heck, let’s start this show.

    You once told me, Pieter, to be rich, stop writing code,
    it’s men who do the politics who pick up all the gold.

    Investment banker ethics, you said when we first met,
    You proved that many times, I’ll always owe a debt.

    You chased hard after money, you chased power, glory, fame.
    So ten years’ on we’re here again, still playing at this game.

    But rich friends and their stooges won’t make you a good designer,
    New kitchen, car and flat TV won’t make your work smell finer.

    Repeat the tired old promises, your dreams of endless glory,
    because that’s what they are, they’re dreams, they’re just a story.

    You built a massive castle, and raised up a higher wall
    and invited the kings and the queens to a fancy costume ball.

    And outside, we raw peasants, we toiled in the mud,
    and we built a real sprawling city, the future, my lord.

    Your fortress sits quite splendid, tricked out in purest gold
    but inside those high walls it’s empty, and brutal, and cold.

    The sycophantic circle jerk is awesome entertainment,
    I’m breathless for the next reply, if you can maintain it.

    We get it, your deep hatred, your anger, and your fear,
    these are normal emotions when your fate is crystal clear.

    We’re the peasant zombies, the 99% unseen,
    the dirty unwashed masses, the community, unclean.

    We argue and we bicker, and we have our little wars,
    but we’re the quiet storm that’s breaking down your doors.

    The future may remember you, John, if we care at all,
    a footnote to remind us: pride comes before a fall.

  • http://lucina.net/ Martin Lucina

    @John O’Hara

    Excuse me, but the Crossroads I/O fork has nothing whatsoever to do with [Pieter's] “passionate retaliatory effort” [against AMQP]. And the “expert team in the early days” working on the AMQP *protocol design* was Pieter and yourself, no one else.

  • http://www.mynitor.com Ma Diga

    Thanks for the article.
    I need to know what you used to create the diagrams. I’m assuming powerpoint but could be something else?

    Thanks!

  • Damien

    Hello everyone,

    Very interesting things here !
    Could someone explain how to secure AMQP exchanges ?
    I mean: does AMQP have intrinsical secure functionnalities ? I’m thinking about SASL for example (not specifical AMQP ;) but also a way to adapt ws-security in AMQP.
    More clearly, i would like to be able to :
    - filter content into AMQP message (ex: .exe file forbidden,ex: max file size)
    - authentication between Broker and clients (and between brokers) based on token like ws-security
    - etc.
    What is the best way to secure AMQP exchanges ?

    A good solution could be to have a similary “secure proxy” AMQP like WAF http which could analyze the whole exchanges content (filter L3, TCP,AMQP), and NECESSARILY could act as a bounder network by:
    - (1) analyze and filter inbound messages(msg) and AMQP exchanges (RFC compliance),
    - (2) extracting the content of inbound messages (performs a network partitionning)
    - (3) build a new message a with the content of the extract from (1)

    Scheme:
    ———————————– ———————————————– —————————–
    | AMQP client | ——> | ~~~~~ AMQP filter ~~~~ | —->| society broker |
    | sends message | | authenticates client by token | | receive secured msg|
    | (X) = message payload | | enpoint to SASL or IpSec | ——————————
    | | | uses (X) to build the new msg |
    ———————————– ———————————————–

    Scope : AMQP 1.0

    The last by not least question : how to build the FILTER module in the above scheme ?

    Thanks everyone for reading and if possible for helping me to answer my questions.

    Best regards

    Damien (french self-educated).

  • Pingback: Hour 5 | My Distributed Systems Notes