Azure Table Storage Client Extensions

Azure Table Storage is my storage technology of choice for my Windows Azure App. For anyone who’s used this technology, you’ll be well aware of its limitations. One in particular is its lack of relational functionality (aka table joins). For example, take the following LINQ query:

from o in context.Orders
join c in context.Customers on o.CustomerId equals c.CustomerId
select new
{
    o.OrderNo,
    c.FirstName,
    c.LastName,
    o.Total
};

This query will simply fail with a The method 'Join' is not supported. error (not to mention I’m also trying to use projection which will also fail).

But I was thinking that wouldn’t it be cool if a query like this just worked, and was smart enough to perform the required asynchronous queries behind the scenes to return the data.

Now of course there would be a performance hit in doing this, but for small datasets the overhead should be more than acceptable. For large data sets, the only workable option is to denormalise your data beforehand.

So I’d like to introduce my (highly experimental) XTableServiceContext:

Download from CodePlex

XTableServiceContext is a replacement to the standard TableServiceContext class provided by Microsoft. It aims to provide identical functionality, but without the limitations. At the time of writing this blog, the following additional features are supported:

• Relationships via Join()
• Projection via Select()

However I hope to keep filling in the gaps (Union, GroupBy etc) as I need more functionality.

Test Harness

Thanks to the guys at Microsoft Australia for giving me some extra CTP tokens, I'm able to publish the test harness while the CTP is still available:

http://storageclientext.cloudapp.net/

Important Note!

Remember that we are still limited by the underlying technology provided by Table Storage, and that this class retrieves data using the most efficient method available. This may mean performing multiple transactions behind the scenes so please keep this in mind.

Please send me your questions, feedback and suggestions!

Development Storage Sync (Azure November 2009 SDK)

With the November 2009 release of the Windows Azure SDK, Microsoft have changed the development storage to support dynamic schemas - just like the real Azure environment.

Because of this change, there is no longer a need to run the Create Test Storage Tables function (hoorah!) - which also means my existing Development Storage Sync tool is now defunct too.

However not all is sunshine and fairies. In order for development storage to update its schema, you must first insert an entity with the most recent properties (see this forum). If you don't, you'll get a crash if you attempt to query on a property that's not in the schema.

Essentially this means that you still need to update your development db each time you change an entity!

To make life easier, I've updated my existing Development Storage Sync tool to do this. Using reflection, it will update the schema for your entire context in one action:

Download Development Storage Sync

Installation

The application requires a reference to your assemblies that contain your TableServiceContext - which means you can't just double-click the app to run it. However a really easy way to run the app is to create an External Tool in Visual Studio. To do this:

1. Click Tools > External Tools...

2. Add a new External Tool:

   

   The Command field should point to DevStorageSync.exe in your installation directory.

   The Arguments field should point to your assembly that contains your TableServiceContext (surround with talking marks, separate multiple assemblies with a semicolon).

Now, whenever you want to synchronise your development db, just click Tools > Sync Development Storage.

Please let me know of any bugs or feedback.

Cheers,
Anthony.

Parsing Multipart Form Data in a WCF Service

Recently I needed a web page that uploaded files directly to a Windows Communication Foundation (WCF) service. On the face of it this seemed achievable: plonk a file upload field on my web page, and write a WCF service that accepts a file Stream.

So I did this, and it sort of worked - however the file that arrived on my server was all garbled! Then I remembered that a browser does not just post the binary contents of a file, it posts "multipart form data" which can include any other field data included in the form.

SURELY there is some sort of low-level .NET API that can read multipart form data, and just give me my file stream...right? Well I bet somewhere there is, but I couldn't find anything that I could use within a WCF service. Ugh.

So I'm left to deal with it myself. By inspecting the contents of the multipart data, I get something like this:

------------cH2ae0GI3KM7GI3Ij5ae0ei4Ij5Ij5
Content-Disposition: form-data; name=\"Filename\"

PB020344.jpg
------------cH2ae0GI3KM7GI3Ij5ae0ei4Ij5Ij5
Content-Disposition: form-data; name=\"Filedata\"; filename=\"PB020344.jpg\"
Content-Type: application/octet-stream

BINARY DATA IS HERE
------------cH2ae0GI3KM7GI3Ij5ae0ei4Ij5Ij5
Content-Disposition: form-data; name=\"Upload\"

Submit Query
------------cH2ae0GI3KM7GI3Ij5ae0ei4Ij5Ij5--

The binary data starts after the Content-Type, and ends with a line break. So I wrote a parsing class that uses some regular expressions to extract the binary data. It also retrieves the posted file name, and the content type.

Download from CodePlex

It works great in my scenario, however there may be scenarios where it will fail - for example posting multiple files in a single form.

To use the class in a WCF service, your implementation would look something like this:

public string Upload(Stream stream)
{
    MultipartParser parser = new MultipartParser(stream);

    if(parser.Success)
    {
        // Save the file
        SaveFile(parser.Filename, parser.ContentType, parser.FileContents);
    }
    else
    {
        throw new WebException(System.Net.HttpStatusCode.UnsupportedMediaType, "The posted file was not recognised.");
    }
}

Cheers,
Anthony.

Non-Destructive Test Storage Table Creation

For those of you working with Windows Azure Table Storage, I'm sure you're aware of the Create Test Storage Tables feature in Visual Studio:

This function looks through your cloud application for classes derived from DataServiceContext, and then creates matching tables in your local SQL 2008 database.

The frustrating thing is, every time you run Create Test Storage Tables, it blasts away your entire DB, then re-creates it. When you're building a large application, it can be incredibly time consuming to re-instate your test data.

Argh!

What I'd prefer is if this function synchronised your classes with your current DB, without destroying your data. So I've had a stab at building a replacement program which does just this.

Download Development Storage Sync

Development Storage Sync is an alternative to the standard DevtableGen.exe that ships with the Windows Azure SDK (this is what is executed when you click Create Test Storage Tables. Just like DevtableGen, it looks for classes derived from DataServiceContext in your cloud app. However instead of re-building your DB, it compares each class and property with each table and column to see what's changed, and displays a summary of changes required:

Clicking Sync will then execute the changes, while maintaining any existing data.

Installation

After installing the application, you need to tell your cloud app to run DevStorageSync.exe, instead of DevtableGen.exe. To do this, follow these steps:

1. Open your cloud app project file in a text editor. This is the project file that ends .ccproj.
   
   
2. Locate the following line:
   

     <Import Project="$(CloudExtensionsDir)Microsoft.CloudService.targets" />
   

3. Insert the following directly below this line:
   

     <Target Name="_CreateDevStorageTable" DependsOnTargets="Build">
         <ItemGroup>
           <TableAssemblies Include="%(Roles.Identity)**\*.dll" />
         </ItemGroup>
         <Exec Command='"C:\Program Files\DevStorageSync\DevStorageSync.exe" "/config:$(ServiceHostingSDKBinDir)DevtableGen.exe.config" "/database:$(DevTableStorageDatabaseName)" "@(TableAssemblies)"' />
     </Target>
   

   Note: Ensure the path in the Command attribute points to where you installed DevStorageSync.exe.

What does this do? The Microsoft.CloudService.targets file contains a build target called _CreateDevStorageTable this is the target that is run when you click Create Test Storage Tables (it launches DevtableGen.exe). By adding the same target to your project file, we are overriding the target to point to DevStorageSync.exe

Running DevStorageSync

Just click Create Test Storage Tables as you normally would. You will still get a warning dialog that your existing database will be deleted - just ignore this as this seems to be hard-coded to the menu item in Visual Studio.

Bugs & Feedback

There's sure to be some issues with this, so please report any bugs to me!

Cheers,
Anthony.

Row Grouping for YUI Data Table

I've previously harped on about how great the YUI Data Table is so I'll spare the introduction.

For all its bells & whistles, what the data table doesn't offer is grouping of rows - i.e. you're limited to 1 level of data. But thanks to the YUI data table's extensibility, there's plenty of opportunity to roll your own - which is what I did:

Download from GitHub

The YUI data table provides a custom row formatter hook, which allowed me to insert a group header at the relevant points. I had to use a bit of CSS trickery to insert the headers, as they are floating DIV elements, not table rows. The reason for this is that inserting rows caused the YUI data table to get confused if row selection is enabled.

Update 4 Nov 2009: Thanks to the hard work of contributor Mark Mansour many updates have been made to the Grouped Data Table. Mark has made the component inherit from the standard DataTable, plus many other bug fixes & enhancements.

The extension also supports collapsing of groups, and group selection:

Usage is pretty straightforward, firstly include the groupeddatatable.js file on your page:

    <script type="text/javascript" src="groupeddatatable.js"></script>

There is also a set of default styles based on the yui-sam look & feel:

    <link rel="stylesheet" type="text/css" href="assets/datatablegrouper.css">

GroupedDataTable inherits from the standard YUI DataTable, so create an instance of it as per the standard YUI component:

    // Create a GroupedDataTable based on the supplied columns & data source
    var myDataTable = new YAHOO.widget.GroupedDataTable("dtContainer", myColumnDefs, myDataSource, { groupBy: "state" });

Notice the extra groupBy configuration parameter.

That's it. There's a much more detailed example provided with the script.

As always, there's bound to be issues with the script as there are too many scenarios for me to test. So please let me know what you come across.

Cheers, Anthony.