May
26

Continuous Client: Our multi-device dream but how do we build it?

image

Rewind pre-iPhone and computing life was far simpler and predictable for most people. You had a work computer and a home computer. Your work computer may have been a laptop which you carry with you home or on the road. If you had a BlackBerry, all you realistically did with it was e-mail. You could probably count on one hand how many times in a day you switched machines. Fast forward back to present day and things are far more chaotic in workflow.

Our needs for computing workflow have completely changed. Services like DropBox or Instapaper are narrow solutions to the real problem. DropBox lets us sync our data so that we can access it on multiple devices but it’s not addressing the workflow issue. DropBox doesn’t carry over the context of what we were doing when we shifted devices.

I decided to count how many times I switched devices between noon and 6pm, so a 6 hour period. I switched devices 37 times.

Some scenarios that occured:

  • I was checking Twitter on my phone and got an instant message. We sent a few IM’s back and forth. I switched back to my desktop and continued the IM conversation and ended it shortly after.
  • I was in another chat conversation on the desktop and I switched to an iPad, continued the IM convo on the iPad while reading a word document.
  • Someone sent me a text message to my phone, which I wanted to view on my iPad so I typed in the URL on the iPad.
  • I started drafting an email but then was running late for a meeting so I grabbed my iPad to take notes and ran off to the meeting (I would have liked to be able to finish the email on the iPad while waiting in the meeting room).

My multi-device workflow is far more crazy in the evenings or weekends where I’m all over my home with who knows what device. If I’m out, its usually a singular device situation with just my smart phone. But what if I ran out of the door while in the middle of something?

I wrote a blog post back in July where I touched on how I thought devices really needed to be more aware of the users situation not just about file syncing and the cloud could help enable that. Certainly files may play a big role but it’s really only a part of the story.

Ideally we want our multiple devices to understand the context of what we were doing and be able to exchange this state with each other. Optionally we may want to configure how this works and we can debate how the state carries over, what the user experiences are visually, that’s another discussion for another time.

Engadget came up with a proposal and coined the term Continuous Client so from now on I have been referring to this multi-device workflow handoff as Continuous Client since it is a great name for it I think.

Trillian is the only application/service I know of that has truly tackled Continuous Client. If I chat with someone on my PC, iPad or iPhone and switch devices, the Trillian service is keeping track and keeps the same chats open on each device. Although I love this feature, it’s only on a single application/service level.

What we really want is our devices to understand us.

In the post from July I knew the cloud would play a big part of solving this problem but I had no real clue how to achieve it to to the scale and coverage I was wanting. Sure we could write a ton of state tracking hooks everywhere in our code but that’s ugly and a lot of work. Recently I’ve started studying various patterns such as CQRS and enterprise integration patterns (messaging) and I’ve really started to realize how messaging really changes the game for what you can enable in your software. I think Continuous Client fits right in there.

CQRS with event sourcing defines events as domain events that represent state changes in your system. This ensures all mutations of the domain model are tracked with domain events. This is really important because it means your state modifications only occur from events and nothing else. Your system is natively built around events and not a bunch of hooks on the side to facilitate tracking. This ensures when you re-play events your system behaves no differently than it did when it actually was running live.

Greg Young did a talk about CQRS, not just for server systems where he spoke about using CQRS to sync occasionally connected devices. What we want is the same goals. The principals he speaks about in CQRS with event sourcing/syncing with merge conflict resolution are what we want to establish in our applications not just for domain state in the case of CQRS but across the user experience.

Instead of syncing documents and files Greg speaks about syncing domain state change events so that you can re-apply them in another copy of the same domain that lives somewhere else.

Sounds familiar doesn’t it? We want to take a bunch of user interactions (events) and we want it to follow us.

I think one could certainly implement this within CQRS with event sourcing but that’s a big system and if you don’t need those benefits it may be too much ceremony for what you want. I’ll let you be the judge. It’s one of those “it depends” calls. For the sake of this post we will focus on events by themselves.

Let’s look at Greg’s CQRS sample to see how a state modification is implemented to enforce that domain events are the native way the system handles changes. In this example pretend a command comes into the system, which calls a method on the domain object which fires an event into itself to handle. Within the event handler the actual state change occurs. An event is required for the change to occur.

public class InventoryCommandHandlers
{
    public void Handle(DeactivateInventoryItem message)
    {
        var item = repository.GetById(message.InventoryItemId);
        item.Deactivate();
        repository.Save(item, message.OriginalVersion);
    }
}

public class InventoryItem : AggregateRoot
{
    private Guid id;
    private bool activated = true;
    
    public void Deactivate()
    {
        if (!activated)
            throw new InvalidOperationException("already deactivated");

        ApplyChange(new InventoryItemDeactivated(id));
    }

    private void Apply(InventoryItemDeactivated e)
    {
        activated = false;
    }
}

What if I told you many of you have already designed your applications in such a way that they are already capable of having Continuous Client support or would require little work to get there? Are you developing any applications with the MVVM pattern? Are you using MVVM Light’s Messenger, Caliburn.Micro’s Event Aggregator or any other Event Aggregator? If you are an iOS deveoper are you using NSNotificationCenter? If you are using these mechanisms to publish and subscribe UI interactions you are well on your way already.

The more your software is designed around messages the easier your system can be automated without special code required because your system is natively written to handle these messages. The last step is we need a way to push these messages to a service that multiple devices can subscribe to.

Essentially we want this:

image


Let’s pretend we are using Twitter apps that we want our workflow to follow us no matter what device we are on whether it is a PC or smart phone. The steps the user takes are as follows:

  1. User opens the Twitter application on the desktop PC.
  2. The user starts typing a new tweet but is interrupted and has to leave their PC.
  3. The user wants to finish the tweet so they open the Twitter on their smart phone, finish typing and send.

This is how it could look. Keep in mind the software vendor could all be one, but to play devils advocate, what if the messages were standard for a particular service so that 2 vendors could cross their workflows?

image


The key to this like in CQRS with event sourcing is that it is vital that you design your system around using decoupled pub/sub messaging. When a user clicks a button to do something on an iPhone it shouldn’t be talking directly to the UINavigationController to push the next view on the screen. You should push a message through NSNotificationCenter that someone subscribes to which then pushes the new view on to the screen. This way when you sync/merge events you can pump them through NSNotificationCenter and your application doesn’t react any differently than if the user did it live.

If we were going to implement the action of when a user clicks a button and the compose a tweet view is displayed in Objective-C or something similar in C# may look like this:

- (void) displayComposeTweetView
{
    [navigationController pushViewController:composeTweetViewController];
}

What we really want is only behaviors to occur if an event has happened and we might use NSNotificationCenter to facilitate Pub/Sub like this:

- (void) displayComposeTweetView
{
    [[NSNotificationCenter defaultCenter] 
        postNotificationName:@"composeTweetViewOpened" 
        object:self 
        userInfo:tweetViewOpenedEvent];
}

- (void) composeTweetViewOpenedNotification:(NSNotification *) notification
{
    [navigationController pushViewController:composeTweetViewController];
}

Now our behaviour is message driven by nature. For the MVVM crowd in C# the code would be similar using a pub/sub event mechanism of your choice such as MVVM Light’s Messenger or an Event Aggregator.

The added benefit is the components within your application are now very decoupled and have a level of automation built in. You could easily write unit tests that you read these messages and pump them through just as if they came from the cloud service or the user interactions. Another benefit is you have a history of what a user did if you wish to have an audit trail for bug reports and replay for debugging.

One caveat is you would probably want to control what events you end up allowing to go outside the application boundary to the cloud service, you wouldn’t want every key press to go to the cloud service.

In a dream world the OS would provide us a messaging mechanism that all applications use for this kind of functionality rather than implementing it differently within each with the support for syncing against a cloud service.

Microsoft had this initiative already with Live Mesh before it got killed off. Live Mesh had much more than just file sync. Take a look at this blog post about Live Mesh as a platform. I think it could have been a game changer for user experiences on Microsoft platforms or in the industry if it would have been pursued. Live Mesh was supposed to ship on mobile devices along with desktops. Unfortunately it won’t see the light of day. Android has what is called Intents and Intent Filters but that is on just one OS so it doesn’t help us much. Also to note the IEEE has a document up about stream oriented computing.

Design the native behaviours of your applications to leverage messaging and you might just strike a user experience that your users will thank you for.

  • http://muddypa.ws/blog Nick Portelli

    Good post.
    I have trillian on my Android, but rarely use it. Mainly because I use Google Talk most and on Android it does exactly as you describe, almost. I can pick up conversations from gmail or other desktop client and see the sent and received messages on my phone. I don’t see the value of sending a half written IM or Tweet to other devices, they are short enough where you can just finish your thought quick or start over. Where it does make sense is email and documents and the like. And gmail and gmail for Android do that too. I can start an email on gmail and save the draft(or autosave) and it will be on my phone in the draft folder almost instantly. Of course that is only in the Google ecosystem as far as I know.

    I agree that it would be nice if there was a spec that help facilitate all this. If only the mobile OS owners would listen.

    (You need Disqus comments :) )

  • Dave Caswell @dcaswell

    Excellent!
    Very well done, Kelly. So many people in mobile are starting to throw the word ‘ecosystem’ around to talk about the entirety of their platform but really that’s only half the story and doesn’t truly reflect how people use mobile. It’s not its own separate entity, we use it as just a piece of our connected lives. The platform isn’t really the ecosystem when you look at real world use cases, the entirety of the systems we use to stay ‘connected’ is the ecosystem. Great stuff.

  • http://correspondence.codeplex.com Michael L Perry

    I’ve been working on this problem for a while now. I completely agree that event sourcing and pub/sub are the keys that make it all work.

    But I think we can take it one step further. Rather than exposing the message bus as an application feature, we can rely upon declarative behavior. In my library, you don’t subscribe to messages and then respond imperatively to them. You express your desired outcome, and the messenger determines what parts of it need to update.

    Using Trillian as the example, you just say “this list box shows all messages in the conversation”. When a new message arrives, the list box updates. You don’t write a HandleMessage method.

    I have plans to expand this to all devices, but for now I’ve bridged the gaps among Windows Phone, Silverlight Web, and desktop. The real challenge for getting this system onto Android and iPhone will be the lack of data binding.

  • http://mikescode.info Mike Griffith

    Meebo also does this as well, at least on the iphone and web clients. I’m not sure if it has an android app, but I’d imagine it does. Come to think of it there are a few others like that as well, ziplist, dropbox, and of course evernote.

  • Gene

    Your solution is interesting, but I wonder if the problem won’t exist by the time a practical solution is available. If I understand correctly, you are looking for a way to continuously synch service clients so a user can switch between them as needed while performing a single action. This supposes that the average user now and for the foreseeable future needs more than one device to achieve their “digital goals.” I would argue that there are a number of industries striving to put the one device in their customers’ hands that satisfies their daily requirements. I think as form factors change, human computer interfaces evolve, power reqs drop and capabilities rise, you will see fewer devices per person (as in one) and the need for this application of the design pattern going away.

    That being said, I agree that the other benefits you discuss make adopting much what you describe worthwhile. Whichever way it turns out (advances making it a non-issue or a continuous client implementation), I think the software community as a whole will benefit from continued research in this area.

    Thanks for writing the post.

  • Pingback: PLM: From Work To Home via Microsoft and Open Cloud

  • Pingback: PLM: From Work To Home via Microsoft and Open Cloud « Daily PLM Think Tank Blog

  • vijay

    Meebo totally does this.

  • http://gdroggi.blogspot.com Rudy Barbieri

    This is a very interesting post, but I think that this kind of design use a lot o resource without a real necessity. If you are sitting in front of your monitor, in your room writing something, it’s not necessary to deliver event information (typing) to a cloud server. Normally you’ll end your work without any interruption. Probably the best way is to syncronize the remote only when the user request to do it, and eventually executing some siyncronization when the data size is bigger enough in order to avoid latency to retrive data in the other device.
    Also in this kind of application we face a moral problem related to the pollution, ’cause it will create a huge amount of redoundant data use a lot of electricity normally generated by plants generating a lot of pollution.
    I hope my english was good enough ’cause I’m Italian

  • http://www.solcher.net Daniel Solcher

    Excellent, I totally agree on the benefits of continuous client. It’s an idea whose time has come.

    For now, I would image that every client-side information can be “saved” on a server. Now that cloud is here, it makes more sense to leave as many as client-side information in the cloud as well.

    I was wondering does that Evernote app qualifies as early type of continuous client concept? I thought so, too.

    Great post!

    Dan

  • http://physionconsulting.com Barry Wark

    Great post, with some serious food for thought.

    I wonder how you would handle a situation where replay of the incoming events would violate an invariant of the current device’s UX. For example, if the user has started to enter a new tweet, before the “start new tweet”, followed by “set text” events arrive from the cloud, the user should be prompted to optionally discard their existing tweet. If they choose No, but then the “tweet sent” events arrives, what should happen? The system shouldn’t provide the “tweet sent” animation, for example, since the user just said *not* to overwrite the tweet they’re working on and haven’t sent yet. This is a subtle issue, perhaps, but I think it’s indicative of the challenges of merging paths through a state machine where the events you would want to send to make the client feel continuous and the transitions that can be made atomically and idempotently are not one and the same.

  • Pingback: Chris Love's Official ASP.NET Blog : To Say Mobile is to Set A False Limit on Your Reach