Jun
26

Continuous Client: My first attempt at multi-device user experience transitions

By kellabyte  //  Architecture, User Experience  //  12 Comments

Let’s show you the goods before you read how I achieved it. I present my first attempt at continuous client in the form of a video player that allows user experience transitions to cross between a Windows Phone 7 device, an iPhone and a Silverlight web player.

Is that not crazy awesome? I was amazed at how simple it was. Below is how I did it. I can’t wait to work on a more elaborate demo. Stay tuned! Follow my Twitter and this blog.

How I got to this point

My experiments in continuous client architecture has started to gain a lot of traction since I have been tweeting and creating working code and videos. My user experience rant on the potential of continuous client has spanned almost a calendar year now and for the people just following along recently based on my last video I’d like to recap a bit of history.

In July 2010 I posted my thoughts of how I thought the cloud plays a key role in continuous client. The cloud helps bridge the gap between cloud hosted code and on-premise hosted code. On-premise in the case of continuous client is devices such as PC’s, tablets, smart phones and any other device. At the time I wasn’t exactly sure of a way to implement it architecturally without introducing large amounts of complexity and synchronization code.

In the last 5 months I’ve been researching and learning enterprise integration patterns and a pattern called CQRS. Fast forward to a month ago (May 2011) and the stars seemed to align in my head one day. I wrote a post about combining some principals of both which helps us build an architecture that allows us to fairly easily implement some continuous client experiences. It dawned on me that the challenge is really an integration issue.

How did I build it

Enterprise integration patterns brings the value of integrating multiple apps, services and devices through messaging patterns. CQRS gives us the value of building an event driven architecture centered around our state changes. In CQRS the domain mutations only occur as a result of an event being handled. This allows you to support event sourcing where you can get replay ability. In this case we aren’t so much worried about a domain but the user experience state.

Let’s put these together to enable continuous client:

  • Every user interaction generates an event. The application handles the event and mutates internal state.
  • The event gets sent as a message to a service bus hosted in the cloud with a copy per device we want the user experience to be capable of transitioning to.
  • The cloud hosted service bus enqueues these messages in a separate queue per device. This allows for each device to event source what happened on the other devices.

It would look something like this:

image

This gives us a system where event sourcing is essentially automating the user experience and since our application code is developed to handle events natively it operates the same. Another key thing to note is that the cloud infrastructure is holding the queue of events. Devices don’t have to be running the application to take part in the state transitions. Reliable messaging allows us to consume these messages whenever the application starts up.

To prove this out I developed a video player demo as my first attempt at continuous client. The video player has 2 basic functions which are play and pause. I created 2 port’s. One for Windows Phone 7 and Silverlight and my friend Martin Pilkington helped with the iOS iPhone port (thanks Martin!).

We utilized the new Azure Service Bus cloud platform for the messaging infrastructure. The upcoming Azure Service Bus platform has support for topics. I used a topic per user and separate subscription per device. This gives a queue per device, each having their own copy of the messages pushed into the topic. Filtering based on device id was used so that a device would only get messages generated by other devices and not self-generated messages.

In a weekend I had my first continuous client demo working. Most of that time was spent learning Azure Service Bus and writing ports. The actual application code was very simple and I used the Greg Young’s CQRS sample as a reference point.

Show us the code

One thing to keep in mind here in the Windows Phone 7 and Silverlight code is I did not implement this using the MVVM pattern. I’m a fan of MVVM but the purpose is to keep the sample code as simple as possible. Implementing this in MVVM doesn’t change a whole lot.

Here is the WP7 and Silverlight web application code. It is exactly the same and shared between both:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113
using System;
using System.Windows;
using System.Windows.Media;
using Microsoft.Phone.Controls;
 
using VideoDemo.Messaging;
 
namespace VideoDemo
{
public partial class MainPage : PhoneApplicationPage,
IHandle<PlayMessage>,
IHandle<PauseMessage>
{
private EventAggregator events;
private TimeSpan duration;
 
public MainPage()
{
InitializeComponent();
 
this.Loaded += new RoutedEventHandler(MainPage_Loaded);
 
this.buttonPlay.Click +=
new RoutedEventHandler(buttonPlay_Click);
this.buttonPause.Click +=
new RoutedEventHandler(buttonPause_Click);
media.CurrentStateChanged +=
new RoutedEventHandler(media_CurrentStateChanged);
}
 
void MainPage_Loaded(object sender, RoutedEventArgs e)
{
events = App.Container.Resolve<EventAggregator>l;();
events.Init();
events.Subscribe(this);
 
media.Source = new Uri("http://someurl/tron.wmv");
 
}
 
void media_CurrentStateChanged(object sender, RoutedEventArgs e)
{
switch (media.CurrentState)
{
case MediaElementState.Playing:
media.Position = duration;
break;
}
}
 
void buttonPlay_Click(object sender, RoutedEventArgs e)
{
// Here we publish the interaction as an event to Azure
// ServiceBus. We don't do any mutation of state without
// the event going through our event mechanism. This
// ensures all our code when event sourced from Azure
// ServiceBus runs as-is with the native behaviors of the app.
events.Publish(new PlayMessage(media.Position));
}
 
void buttonPause_Click(object sender, RoutedEventArgs e)
{
// Same as above.
events.Publish(new PauseMessage(media.Position));
}
 
public void Handle(PlayMessage message)
{
// Since we are handling the even there we can mutate
// state. Live interactions or event sourced events from
// Azure ServiceBus will all go through this path. It is the
// native behavior of the app.
duration = message.Duration;
media.Play();
}
 
public void Handle(PauseMessage message)
{
// Same as above.
duration = message.Duration;
media.Pause();
}
}
}
 
// Here are the message types.
public class PlayMessage : Message
{
public TimeSpan Duration { get; set; }
 
public PlayMessage()
{
}
 
public PlayMessage(TimeSpan duration)
{
this.Duration = duration;
}
}
 
public class PauseMessage : Message
{
public TimeSpan Duration { get; set; }
 
public PauseMessage()
{
}
 
public PauseMessage(TimeSpan duration)
{
this.Duration = duration;
}
}

Notice here when the interaction event occurs all we do is fire an event. This event is self handled. The events published to the EventAggregator get sent to Azure Service Bus so that all messages are delivered to the topic which each device subscribes to.

I am receiving messages from Azure Service Bus over HTTP. I serialize messages as JSON and when a new message is received we deserialize the JSON into the message type and pump them through the EventAggregator.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
subscription.MessageRecieved +=
new EventHandler<SubscriptionMessageEventArgs>(messageRecieved);
 
subscription.ReceiveMessage(10);
 
void messageRecieved(object sender, SubscriptionMessageEventArgs e)
{
EnvelopeMessage msg =
JsonConvert.DeserializeObject<EnvelopeMessage>(e.Message);
 
Type type = MessageMapper.GetType(msg.MessageId);
 
object message = JsonConvert.DeserializeObject(
msg.MessageContent.ToString(), type);
 
Publish(message);
subscription.ReceiveMessage(10);
}

Since a user interaction is publishing a message to the EventAggregator and our Azure Service Bus topic subscription is also publishing to the EventAggregator our application code is now architected to be natively event driven and our application code has no concept of the source of the event.

Messages have an id that is canonical so that it is not specific to any platform. I chose to use a URI format such as http://kellabyte.com/samples/VideoDemo/Messages/PlayMessage and we use a MessageMapper to get the app specific type that message relates to for deserialization purposes since we can’t share types across platforms who don’t share languages. This could probably be done cleaner, I didn’t spend much time here since it wasn’t the primary focus of the sample.

I’m not covering all the Azure Service Bus code in this post there is already a lot of content out there for Azure. If you feel I should blog about it I can do a specific post for that, let me know!

It’s really very simple and straight forward. As long as you adhere to the rules your application operates the exact same whether it is interactions as a result of a user action or event sourced from the cloud.

Here’s Martin’s Objective-C iOS application code for the iPhone:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149
#import "Video_DemoViewController.h"
#import "AWSServiceController.h"
 
NSString *M3PlaybackTime = @"Duration";
NSString *M3PlayMessage =
@"http://kellabyte.com/samples/VideoDemo/Messages/PlayMessage";
NSString *M3PauseMessage =
@"http://kellabyte.com/samples/VideoDemo/Messages/PauseMessage";
 
@interface Video_DemoViewController ()
 
- (MPMoviePlayerController *)_moviePlayer;
 
@end
 
@implementation Video_DemoViewController
 
@synthesize serviceController;
@synthesize contentView;
 
/***************************
Set up
**************************/
- (id)initWithCoder:(NSCoder *)aDecoder {
if ((self = [super initWithCoder:aDecoder])) {
[self addObserver:self
forKeyPath:@"serviceController"
options:NSKeyValueObservingOptionOld
context:NULL];
}
return self;
}
 
/***************************
Update notifications when the service controller changes
**************************/
- (void)observeValueForKeyPath:(NSString *)keyPath
ofObject:(id)object
change:(NSDictionary *)change
context:(void *)context {
if ([keyPath isEqualToString:@"serviceController"]) {
[[SAFE_NULL([change objectForKey:NSKeyValueChangeOldKey])
notificationCenter] removeObserver:self];
 
id centre = [[self serviceController] notificationCenter];
 
[centre addObserver:self
selector:@selector(playMovie:)
name:M3PlayMessage
object:nil];
 
[centre addObserver:self
selector:@selector(pauseMovie:)
name:M3PauseMessage
object:nil];
}
}
 
/***************************
Set up the movie player
**************************/
- (void)viewDidLoad {
MPMoviePlayerController *player = [self _moviePlayer];
[[player view] setFrame:[[self contentView] bounds]];
[[self contentView] addSubview:[player view]];
[super viewDidLoad];
}
 
/***************************
Set up and return the movie player
**************************/
- (MPMoviePlayerController *)_moviePlayer {
if (!moviePlayer) {
NSURL *url = [NSURL URLWithString:
@"http://someurl/tron.m4v"];
moviePlayer = [[MPMoviePlayerController alloc]
initWithContentURL:url];
 
[moviePlayer setShouldAutoplay:NO];
[moviePlayer setControlStyle:MPMovieControlStyleNone];
}
return moviePlayer;
}
 
/***************************
Handle a play notification
**************************/
- (void)playMovie:(NSNotification *)aNote {
NSNumber *time = [[aNote userInfo] objectForKey:M3PlaybackTime]
CGFloat playbackTime = [time floatValue];
 
[[self _moviePlayer] setCurrentPlaybackTime:playbackTime];
[[self _moviePlayer] play];
}
 
/***************************
Handle a pause notification
**************************/
- (void)pauseMovie:(NSNotification *)aNote {
NSNumber *time = [[aNote userInfo] objectForKey:M3PlaybackTime]
CGFloat playbackTime = [time floatValue];
 
[[self _moviePlayer] setCurrentPlaybackTime:playbackTime];
[[self _moviePlayer] pause];
}
 
/***************************
Post a play notification when the user presses the play button
**************************/
- (IBAction)play:(id)sender {
CGFloat time = [[self _moviePlayer] currentPlaybackTime];
NSNumber *timeObj = [NSNumber numberWithFloat:time];
id info = [NSDictionary dictionaryWithObject:timeObj
forKey:M3PlaybackTime]
 
id centre = [[self serviceController] notificationCenter];
[centre postNotificationName:M3PlayMessage
object:nil
userInfo:info];
}
 
/***************************
Post a pause notification when the user presses the pause button
**************************/
- (IBAction)pause:(id)sender {
CGFloat time = [[self _moviePlayer] currentPlaybackTime];
NSNumber *timeObj = [NSNumber numberWithFloat:time];
id info = [NSDictionary dictionaryWithObject:timeObj
forKey:M3PlaybackTime]
 
id centre = [[self serviceController] notificationCenter];
[centre postNotificationName:M3PauseMessage
object:nil
userInfo:info];
}
 
/***************************
Clean up
**************************/
- (void)dealloc {
[contentView release];
[super dealloc];
}
 
- (void)viewDidUnload {
[self setContentView:nil];
[super viewDidUnload];
}
@end

Keep an eye out because I will be working on a more elaborate continuous client sample! I’ve begun work on a mobile RSS reader so that I can prove that the architecture shown here can carry over to all kinds of applications while enabling all sorts of user experience transitions.

If you would like to help port it to a specific platform and work with me on the continuous client concepts feel free to contact me on Twitter Smile

Shout out to Martin Pilkington for working on the iOS port in parallel with me and the Azure Service Bus team (Clemens Vasters and Will Perry) for answering my questions on Twitter and logging some bugs that I encountered. I really enjoyed working with Azure Service Bus. It’s very easy and handy to use on multiple platforms.

I’m excited about this progress. Stay tuned for more continuous client!

  • http://www.sumitmaitra.com Sumit

    Nice summary of all those tweets :-) .

    I had asked you on twitter (@sumitkm) about the location of the movie and you mentioned it was on the cloud but the client wouldn’t care. I am curious how would the client(s) handle it if indeed the source wasn’t the cloud. Wouldn’t the source at-least need to be a common location that all the intended clients can access right? Might be a basic/dumb question, wanted to know what you were thinking. I totally get the idea when the source is in the cloud/commonly accessible location, but if that’s not the minimum requirement I am missing something!

    On the CC idea on the whole, I think you’ve gone a long way in de-mystifying google docs’ collaboration feature :-) . Democratizing collaborative document creation :-) . Looking forward to the RSS Reader.

    The Media player idea does urge me to write a CC client that can control movies from MBP to my iDevices and vise-versa (without Apple TV).

    Cheers,
    Sumit.

  • http://correspondence.codeplex.com Michael L Perry

    With the queue-per-device approach, do you have to know all of your devices in advance? Or can a new device join a conversation already in progress?

  • Pingback: Windows Client Developer Roundup 074 for 6/27/2011 - Pete Brown's 10rem.net

  • Pingback: Kelly Sommers first attempt at multi-device user experience transitions – www.nalli.net

  • http://gdroggi.blogspot.com Rudy Barbieri

    This post is really interesting.
    In this sample you don’t need to keep the full event list ’cause you simply need the last event, so i would like to suggest you to put a “quality” related to a single event in order to specify a special behavior (for example: keep only the last one).
    How would you like to manage the case where two device are sending events at the same time (for example: i have play the video on my mobile phone, than i put it on the desk and play the same video on my desktop computer).
    Ciao
    Rudy

  • Pingback: pligg.com

  • Pingback: Linksammlung 9/7/2011 | Silverlight, WPF & .NET

  • http://nicholascloud.com ncloud

    First, +1 for the Tron trailer. Your geekfu is strong.

    Second, this makes total sense. I’ve not worked with the Azure SB yet, but it seems like a natural fit for a CQRS architecture.

    This would be an excellent way to let remote users or locations participate in a slide-show presentation, where a single speaker is broadcasting audio (maybe via phone), but everyone has a client that will “keep up” with slide changes as the speaker moves through the talk.

    Or it might even have applications in gaming. You might have “The Grid” where players could jump into a fun game of Light Cycles and the message bus would broadcast “LeftTurn”, “RightTurn”, and “Collision” messages for each player.

    Very cool.

  • Pingback: Windows Azure and Cloud Computing Posts for 12/12/2011+ - Windows Azure Blog

  • Anonymous

    Is there a link to the fully working source code for this one?

  • SWM

    Hi Kelly,
    Is the source code / solution file for WP7 and the Web Page available for download somewhere

    Thanks

  • SWM

    Hi Kelly, I love your continuous client sample. Wld it be possible to get a src code or project download pls ? Both the WP7, Azure Bus and the Web Application pieces, please ?