Azure Table Storage Client Extensions

Azure Table Storage is my storage technology of choice for my Windows Azure App. For anyone who’s used this technology, you’ll be well aware of its limitations. One in particular is its lack of relational functionality (aka table joins). For example, take the following LINQ query:

from o in context.Orders
join c in context.Customers on o.CustomerId equals c.CustomerId
select new
{
    o.OrderNo,
    c.FirstName,
    c.LastName,
    o.Total
};

This query will simply fail with a The method 'Join' is not supported. error (not to mention I’m also trying to use projection which will also fail).

But I was thinking that wouldn’t it be cool if a query like this just worked, and was smart enough to perform the required asynchronous queries behind the scenes to return the data.

Now of course there would be a performance hit in doing this, but for small datasets the overhead should be more than acceptable. For large data sets, the only workable option is to denormalise your data beforehand.

So I’d like to introduce my (highly experimental) XTableServiceContext:

Download from CodePlex

XTableServiceContext is a replacement to the standard TableServiceContext class provided by Microsoft. It aims to provide identical functionality, but without the limitations. At the time of writing this blog, the following additional features are supported:

• Relationships via Join()
• Projection via Select()

However I hope to keep filling in the gaps (Union, GroupBy etc) as I need more functionality.

Test Harness

Thanks to the guys at Microsoft Australia for giving me some extra CTP tokens, I'm able to publish the test harness while the CTP is still available:

http://storageclientext.cloudapp.net/

Important Note!

Remember that we are still limited by the underlying technology provided by Table Storage, and that this class retrieves data using the most efficient method available. This may mean performing multiple transactions behind the scenes so please keep this in mind.

Please send me your questions, feedback and suggestions!

Development Storage Sync (Azure November 2009 SDK)

With the November 2009 release of the Windows Azure SDK, Microsoft have changed the development storage to support dynamic schemas - just like the real Azure environment.

Because of this change, there is no longer a need to run the Create Test Storage Tables function (hoorah!) - which also means my existing Development Storage Sync tool is now defunct too.

However not all is sunshine and fairies. In order for development storage to update its schema, you must first insert an entity with the most recent properties (see this forum). If you don't, you'll get a crash if you attempt to query on a property that's not in the schema.

Essentially this means that you still need to update your development db each time you change an entity!

To make life easier, I've updated my existing Development Storage Sync tool to do this. Using reflection, it will update the schema for your entire context in one action:

Download Development Storage Sync

Installation

The application requires a reference to your assemblies that contain your TableServiceContext - which means you can't just double-click the app to run it. However a really easy way to run the app is to create an External Tool in Visual Studio. To do this:

1. Click Tools > External Tools...

2. Add a new External Tool:

   

   The Command field should point to DevStorageSync.exe in your installation directory.

   The Arguments field should point to your assembly that contains your TableServiceContext (surround with talking marks, separate multiple assemblies with a semicolon).

Now, whenever you want to synchronise your development db, just click Tools > Sync Development Storage.

Please let me know of any bugs or feedback.

Cheers,
Anthony.

Parsing Multipart Form Data in a WCF Service

Recently I needed a web page that uploaded files directly to a Windows Communication Foundation (WCF) service. On the face of it this seemed achievable: plonk a file upload field on my web page, and write a WCF service that accepts a file Stream.

So I did this, and it sort of worked - however the file that arrived on my server was all garbled! Then I remembered that a browser does not just post the binary contents of a file, it posts "multipart form data" which can include any other field data included in the form.

SURELY there is some sort of low-level .NET API that can read multipart form data, and just give me my file stream...right? Well I bet somewhere there is, but I couldn't find anything that I could use within a WCF service. Ugh.

So I'm left to deal with it myself. By inspecting the contents of the multipart data, I get something like this:

------------cH2ae0GI3KM7GI3Ij5ae0ei4Ij5Ij5
Content-Disposition: form-data; name=\"Filename\"

PB020344.jpg
------------cH2ae0GI3KM7GI3Ij5ae0ei4Ij5Ij5
Content-Disposition: form-data; name=\"Filedata\"; filename=\"PB020344.jpg\"
Content-Type: application/octet-stream

BINARY DATA IS HERE
------------cH2ae0GI3KM7GI3Ij5ae0ei4Ij5Ij5
Content-Disposition: form-data; name=\"Upload\"

Submit Query
------------cH2ae0GI3KM7GI3Ij5ae0ei4Ij5Ij5--

The binary data starts after the Content-Type, and ends with a line break. So I wrote a parsing class that uses some regular expressions to extract the binary data. It also retrieves the posted file name, and the content type.

Download from CodePlex

It works great in my scenario, however there may be scenarios where it will fail - for example posting multiple files in a single form.

To use the class in a WCF service, your implementation would look something like this:

public string Upload(Stream stream)
{
    MultipartParser parser = new MultipartParser(stream);

    if(parser.Success)
    {
        // Save the file
        SaveFile(parser.Filename, parser.ContentType, parser.FileContents);
    }
    else
    {
        throw new WebException(System.Net.HttpStatusCode.UnsupportedMediaType, "The posted file was not recognised.");
    }
}

Cheers,
Anthony.

Non-Destructive Test Storage Table Creation

For those of you working with Windows Azure Table Storage, I'm sure you're aware of the Create Test Storage Tables feature in Visual Studio:

This function looks through your cloud application for classes derived from DataServiceContext, and then creates matching tables in your local SQL 2008 database.

The frustrating thing is, every time you run Create Test Storage Tables, it blasts away your entire DB, then re-creates it. When you're building a large application, it can be incredibly time consuming to re-instate your test data.

Argh!

What I'd prefer is if this function synchronised your classes with your current DB, without destroying your data. So I've had a stab at building a replacement program which does just this.

Download Development Storage Sync

Development Storage Sync is an alternative to the standard DevtableGen.exe that ships with the Windows Azure SDK (this is what is executed when you click Create Test Storage Tables. Just like DevtableGen, it looks for classes derived from DataServiceContext in your cloud app. However instead of re-building your DB, it compares each class and property with each table and column to see what's changed, and displays a summary of changes required:

Clicking Sync will then execute the changes, while maintaining any existing data.

Installation

After installing the application, you need to tell your cloud app to run DevStorageSync.exe, instead of DevtableGen.exe. To do this, follow these steps:

1. Open your cloud app project file in a text editor. This is the project file that ends .ccproj.
   
   
2. Locate the following line:
   

     <Import Project="$(CloudExtensionsDir)Microsoft.CloudService.targets" />
   

3. Insert the following directly below this line:
   

     <Target Name="_CreateDevStorageTable" DependsOnTargets="Build">
         <ItemGroup>
           <TableAssemblies Include="%(Roles.Identity)**\*.dll" />
         </ItemGroup>
         <Exec Command='"C:\Program Files\DevStorageSync\DevStorageSync.exe" "/config:$(ServiceHostingSDKBinDir)DevtableGen.exe.config" "/database:$(DevTableStorageDatabaseName)" "@(TableAssemblies)"' />
     </Target>
   

   Note: Ensure the path in the Command attribute points to where you installed DevStorageSync.exe.

What does this do? The Microsoft.CloudService.targets file contains a build target called _CreateDevStorageTable this is the target that is run when you click Create Test Storage Tables (it launches DevtableGen.exe). By adding the same target to your project file, we are overriding the target to point to DevStorageSync.exe

Running DevStorageSync

Just click Create Test Storage Tables as you normally would. You will still get a warning dialog that your existing database will be deleted - just ignore this as this seems to be hard-coded to the menu item in Visual Studio.

Bugs & Feedback

There's sure to be some issues with this, so please report any bugs to me!

Cheers,
Anthony.

Row Grouping for YUI Data Table

I've previously harped on about how great the YUI Data Table is so I'll spare the introduction.

For all its bells & whistles, what the data table doesn't offer is grouping of rows - i.e. you're limited to 1 level of data. But thanks to the YUI data table's extensibility, there's plenty of opportunity to roll your own - which is what I did:

Download from GitHub

The YUI data table provides a custom row formatter hook, which allowed me to insert a group header at the relevant points. I had to use a bit of CSS trickery to insert the headers, as they are floating DIV elements, not table rows. The reason for this is that inserting rows caused the YUI data table to get confused if row selection is enabled.

Update 4 Nov 2009: Thanks to the hard work of contributor Mark Mansour many updates have been made to the Grouped Data Table. Mark has made the component inherit from the standard DataTable, plus many other bug fixes & enhancements.

The extension also supports collapsing of groups, and group selection:

Usage is pretty straightforward, firstly include the groupeddatatable.js file on your page:

    <script type="text/javascript" src="groupeddatatable.js"></script>

There is also a set of default styles based on the yui-sam look & feel:

    <link rel="stylesheet" type="text/css" href="assets/datatablegrouper.css">

GroupedDataTable inherits from the standard YUI DataTable, so create an instance of it as per the standard YUI component:

    // Create a GroupedDataTable based on the supplied columns & data source
    var myDataTable = new YAHOO.widget.GroupedDataTable("dtContainer", myColumnDefs, myDataSource, { groupBy: "state" });

Notice the extra groupBy configuration parameter.

That's it. There's a much more detailed example provided with the script.

As always, there's bound to be issues with the script as there are too many scenarios for me to test. So please let me know what you come across.

Cheers, Anthony.

Select Filter for ADO.NET Data Services

UPDATE 2 Sept 2009: Microsoft have now released CTP2 of ADO.NET Data Services, which now supports projections. You'd be crazy to use my solution now, so I've removed it from CodePlex. You can download CTP2 from here: http://blogs.msdn.com/astoriateam/archive/2009/08/31/ado-net-data-services-v1-5-ctp2-now-available-for-download.aspx Anthony.

If you want to build a service layer for your app in a flash, ADO .NET Data Services is the bee’s knees in my opinion. That said, being a new technology there are a few shortfalls I’ve come across.

The first is a lack of support for retrieving a count of records, which is a must if you intend to do paging of data. Thankfully, Microsoft has released a v1.5 CTP which addresses this (see my previous blog post).

The second glaring omission is the inability to only return selected properties from an entity in a query. This is otherwise known as projection, but I think of it as a traditional SELECT statement in SQL. Currently, a Data Service will always return all properties of all entities (or in my SQL analogy, SELECT *).

When you remember that you’re sending all data over HTTP from the server to the client, this can have a massive performance impact as you are almost always retrieving way more data than you need.

No way José! I’m sure Microsoft is well aware of this and are beavering away on a great solution, but in the meantime I’ve come up with a very dodgy, but equally effective solution in the form of a Http Module.

Download Select Filter for ADO.NET Data Services

After spending considerable time fiddling with Query Interceptors in my data services, there really is no way to only return selected properties from an entity. The only workaround at a code level would be to create custom entities that only have the properties you need – but that ‘aint pretty.

So I ended up using more of a “brute force” solution, by letting the Data Service do its thing, but then intercepting the Http response before it’s sent back to the client. By creating a custom Http Module, I was able to catch the response, remove the unwanted properties, and then stream the modified response to the client.

Note #1: I’m not using the ADO .NET Data Service Client libraries to consume my services, I call the services directly using AJAX. If you’re using the client libraries through Silverlight etc, I doubt this solution will be any good for you.

Note #2: My solution only works with JSON responses at the moment, as this is all I use. However it would be very easy to add support for XML if required.

Here’s how to get up and running:

1. Download the library and add a reference to AntsCode.DataServiceSelect.dll in your Data Services project.
2. Add the following HttpModule to your Web.config:

<httpModules>
    <add name="DataServiceModule" type="AntsCode.DataServiceSelect.DataServiceModule"/>
</httpModules>

That’s it really. Now, when you make a data service request, you can add a new select parameter, specifying the properties to return:

http://yourserver.com/YourService.svc/Customers?select=FirstName,LastName

The HttpModule will intercept the response, and filter out unwanted properties:

{
    d: [
    {
        “FirstName”: “John”,
        “LastName”: “Bloggs”
    },
    {
        “FirstName”: “Anne”,
        “LastName”: “Jones”
    }
]}

How is the data filtered? While I could have deserialized the JSON response into an object, and somehow removed the unwanted properties programmatically; I instead opted to use regular expressions and just strip out the unwanted text as I felt this would be much more efficient (I told you the solution was dodgy!!) You can see this in action in the DataServiceJsonFilter class in the source code.

Of course I’m really hanging out for an update from Microsoft to do this properly, at which point I’ll promptly throw this code in the bin. In the meantime, if you have any feedback or bugs, please report them on the CodePlex site.

Anthony.

YUI DataTable with ADO.NET Data Services

The Yahoo User Interface library (YUI) is an awesome client-side UI library, and ADO.NET Data Services is an equally awesome server-side framework for exposing a service layer for your data.

However, there is little documentation on how to get the 2 technologies to talk to one-another. Microsoft are pitching ADO.NET Data Services to marry up with Sliverlight - and rightly so, but like a lot of people, my skills are firmly grounded in AJAX.

Searching around, Jim Fiorato has written a great article on integrating a data service with a YUI chart but that's about it at this stage.

So far I've managed to get a YUI DataTable to display data from an ADO.NET Data Service, along with paging and sorting.

ADO.NET Data Services v1.5 CTP1

I need to highlight a shortfall in the current version of ADO.NET Data Services (1.0) - it does not return a count of all rows. Without this, displaying pages of data is nigh impossible as we simply don't know how many pages to display.

Thankfully, the issue has been resolved in the v1.5 CTP, you can download it here. Note that this is a pre-release version so I'm really hanging out for the final release.

Sample Project on CodePlex

I've created a sample project that contains a simple ADO.NET Data Service, and a sample page that implements a YUI DataTable. Note: You must first install the v1.5 CTP for it to compile.

I'm hoping to add more samples of other YUI components down the track.

Instead of writing a huge blog post explaining all the steps, I've added detailed code comments in the sample project.

Anthony.

Strip Formatting on Paste using YUI Rich Text Editor

I’ve been using the Yahoo User Interface Library (YUI) in my web app, and one particularly cool component is the YUI Rich Text Editor: its cross-browser compatible, fully extensible, and best of all it’s free :)

For all its greatness, one thing I’ve struggled with is that if you copy & paste stuff from another app into the YUI Editor, all of the original formatting is maintained. Most of the time for us, the “other app” is Microsoft Word, which does a particularly heinous job of generating HTML from a formatted document. This almost always wreaks havoc if the user subsequently tries to change text styles in the editor, as the underlying HTML is a total mess.

So, the solution for us was to try and strip out all of the formatting when somebody pastes stuff into the Editor, resulting in nice clean HTML that plays well with the YUI Editor formatting functions. Now, unfortunately no such feature exists in the YUI Editor, and Googling around just led to dead ends, so I was left to build my own.

CleanPaste for YUI Rich Text Editor

Download from CodePlex

For information on what’s supported, please read the notes on CodePlex, as I will keep this updated as I make bug fixes etc.

To use the CleanPaste script, follow these steps:

1. Ensure you’ve already installed the YUI components & created a Yahoo Editor on your page.
2. Place the CleanPaste.js file somewhere in your project.
3. Include the script in your page using the following code:
   
   <script type="text/javascript" src="CleanPaste.js"></script>
   
   Ensure the src attribute points to the directory where the CleanPaste.js script is located.
   
4. In the Javascript where you create your Yahoo Editor object, create an instance of the CleanPaste object, passing in the editor as the parameter:
   
   var myEditor = new YAHOO.widget.Editor('editor', myConfig); myEditor.render();
   
   var cleanPaste = new CleanPaste(myEditor);

That’s it, the editor should now strip the formatting out of pasted text.

Of course this is still under development, so if you have any problems or feedback please post on CodePlex and I will get back to you.

Cheers, Anthony.

Nested Mail Merge Regions using Aspose.Words

I've been looking into various solutions for merging Word documents server-side, without resorting to Microsoft Word Automation as it is apparently the devil.

After trialing a handful of solutions on the web, I've concluded that Aspose.Words has some killer features, such as doc and docx support, and PDF file export. Furthermore, unlike competing products, I won't have to sell a kidney to pay for it. Perfect!

That said, once getting into the nuts and bolts of things I discovered one shortfall that is thoroughly discussed in their forums: No support for nested mail merge regions. A region is something Aspose created to allow a portion of a Word document to grow dynamically based on a set of data, e.g. a Table.

Compared to what Microsoft Word can do, a region is pretty powerful stuff. However, not powerful enough for what I need as our app has lots of hierarchical data. For example, imagine a food order that has multiple delivery times:

10.30am
Bagels
Fruit Juice

12.30pm
Sandwiches
Tea & Coffee

If I wanted to define this in a Word template, my regions would look like this:

{TableStart:DeliveryTimes}
{TableStart:Items}
{TableEnd:Items}
{TableEnd:DeliveryTimes}

Now, for anyone who's tried, Aspose.Words will simply throw an exception: Nested mail merge regions are not yet supported.

I figured that since Aspose.Words provides a full Document Object Model for a document, I could write my own class. A bit of looping here and there... easy!

Hardly. I can now see why Aspose has been holding out on such a function. It actually turned out to be a right nightmare to write the class. That said, I got there in the end so I thought I'd share it with the rest of you so that you too don't need to toil over this for days on end.

Introducing AdvancedMailMerge

Download the source code from CodePlex.

You may need to add a reference to your copy of Aspose.Words in the WordsExtensions project in order for it to compile.

In terms of how to set up your Word document regions, I've used the same syntax as Aspose.Words. Create merge fields called TableStart:TableName and TableEnd:TableName.

To perform the merge, it's pretty straightforward:

1. Load your Word file into an Aspose.Words Document object.
2. Create an instance of the AdvancedMailMerge object.
3. Pass the Document into the Load method of AdvancedMailMerge. The Load method will validate the regions in the document.
4. Check the Errors property to ensure there are no structural errors in the document.
5. Call AddRegionData to add as many data sources as you require. There are special overloads of this function for defining child/parent relationships so that your data is filtered as per your data relationships.
6. Call the Merge function. This will do it's magic and merge the nested regions with your data sources.
7. You can then call the Save method of the Document object, outputting a PDF or whatever you like.

User data filtering

There is also a groovy filtering feature that allows users to filter region data by adding syntax to a TableStart merge field. If you add the \b switch to a field, followed by a standard ADO select query, the data will be filtered in the document, e.g.:

{MERGEFIELD TableStart:Items \b Title = 'Bagel' }

Switch support

Since I can't use the standard Aspose.Words MailMerge class, I'm left to support Mail Merge switches myself (e.g. \* Upper to make the result uppercase etc). I've created a MergeFieldSwitch class to handle this. However there are too many switches for me to support. I've added the basics, however they haven't really been tested. Be prepared for bugs if you want to use these.

Please Contribute

I have a lot of testing still to do on this, and I know there will be some issues. If you find any problems, please post them on the CodePlex site and I will look into it.

I'll post my own bug fixes to CodePlex as I come across them.

Cheers,
Anthony.

Obfuscating .NET code on the cheap

I recently needed to obfuscate a .NET component I wrote, and never having done this before I started looking into the various options available. Being a small side project I really didn't want to spend big bucks.

The obvious first choice was to try out the Dotfuscator Community Edition that ships with Visual Studio as it's free. I obfuscated my component, and then disassembled it using Red Gate's Reflector.

It turned out that all it did was obfuscate my namespaces & method names, however it didn't touch the control flow of the methods, meaning that all of my logic source code was still clearly visible.

These features are available in their professional edition which is way out my budget.

Googling around, I stumbled across Eazfuscator.NET, a free .NET obfuscator written by Oleksiy Gapotchenko. Installing the app was a breeze, it even integrated with the Visual Studio tools menu.

It has some very powerful features, including control flow obfuscation, assembly merging, and re-signing strongly named assemblies. For a free app this is very impressive, as you would pay thousands for this elsewhere.

Upon running my obfuscated component through Reflector, my source code had been completely obfuscated, making me a very happy man.

So far I'm wrapped in what this app can do, and I'll keep you posted as I discover more.