Non-Destructive Test Storage Table Creation

For those of you working with Windows Azure Table Storage, I'm sure you're aware of the Create Test Storage Tables feature in Visual Studio:

This function looks through your cloud application for classes derived from DataServiceContext, and then creates matching tables in your local SQL 2008 database.

The frustrating thing is, every time you run Create Test Storage Tables, it blasts away your entire DB, then re-creates it. When you're building a large application, it can be incredibly time consuming to re-instate your test data.

Argh!

What I'd prefer is if this function synchronised your classes with your current DB, without destroying your data. So I've had a stab at building a replacement program which does just this.

Download Development Storage Sync

Development Storage Sync is an alternative to the standard DevtableGen.exe that ships with the Windows Azure SDK (this is what is executed when you click Create Test Storage Tables. Just like DevtableGen, it looks for classes derived from DataServiceContext in your cloud app. However instead of re-building your DB, it compares each class and property with each table and column to see what's changed, and displays a summary of changes required:

Clicking Sync will then execute the changes, while maintaining any existing data.

Installation

After installing the application, you need to tell your cloud app to run DevStorageSync.exe, instead of DevtableGen.exe. To do this, follow these steps:

1. Open your cloud app project file in a text editor. This is the project file that ends .ccproj.
   
   
2. Locate the following line:
   

     <Import Project="$(CloudExtensionsDir)Microsoft.CloudService.targets" />
   

3. Insert the following directly below this line:
   

     <Target Name="_CreateDevStorageTable" DependsOnTargets="Build">
         <ItemGroup>
           <TableAssemblies Include="%(Roles.Identity)**\*.dll" />
         </ItemGroup>
         <Exec Command='"C:\Program Files\DevStorageSync\DevStorageSync.exe" "/config:$(ServiceHostingSDKBinDir)DevtableGen.exe.config" "/database:$(DevTableStorageDatabaseName)" "@(TableAssemblies)"' />
     </Target>
   

   Note: Ensure the path in the Command attribute points to where you installed DevStorageSync.exe.

What does this do? The Microsoft.CloudService.targets file contains a build target called _CreateDevStorageTable this is the target that is run when you click Create Test Storage Tables (it launches DevtableGen.exe). By adding the same target to your project file, we are overriding the target to point to DevStorageSync.exe

Running DevStorageSync

Just click Create Test Storage Tables as you normally would. You will still get a warning dialog that your existing database will be deleted - just ignore this as this seems to be hard-coded to the menu item in Visual Studio.

Bugs & Feedback

There's sure to be some issues with this, so please report any bugs to me!

Cheers,
Anthony.

Row Grouping for YUI Data Table

I've previously harped on about how great the YUI Data Table is so I'll spare the introduction.

For all its bells & whistles, what the data table doesn't offer is grouping of rows - i.e. you're limited to 1 level of data. But thanks to the YUI data table's extensibility, there's plenty of opportunity to roll your own - which is what I did:

Download from GitHub

The YUI data table provides a custom row formatter hook, which allowed me to insert a group header at the relevant points. I had to use a bit of CSS trickery to insert the headers, as they are floating DIV elements, not table rows. The reason for this is that inserting rows caused the YUI data table to get confused if row selection is enabled.

Update 4 Nov 2009: Thanks to the hard work of contributor Mark Mansour many updates have been made to the Grouped Data Table. Mark has made the component inherit from the standard DataTable, plus many other bug fixes & enhancements.

The extension also supports collapsing of groups, and group selection:

Usage is pretty straightforward, firstly include the groupeddatatable.js file on your page:

    <script type="text/javascript" src="groupeddatatable.js"></script>

There is also a set of default styles based on the yui-sam look & feel:

    <link rel="stylesheet" type="text/css" href="assets/datatablegrouper.css">

GroupedDataTable inherits from the standard YUI DataTable, so create an instance of it as per the standard YUI component:

    // Create a GroupedDataTable based on the supplied columns & data source
    var myDataTable = new YAHOO.widget.GroupedDataTable("dtContainer", myColumnDefs, myDataSource, { groupBy: "state" });

Notice the extra groupBy configuration parameter.

That's it. There's a much more detailed example provided with the script.

As always, there's bound to be issues with the script as there are too many scenarios for me to test. So please let me know what you come across.

Cheers, Anthony.

Select Filter for ADO.NET Data Services

UPDATE 2 Sept 2009: Microsoft have now released CTP2 of ADO.NET Data Services, which now supports projections. You'd be crazy to use my solution now, so I've removed it from CodePlex. You can download CTP2 from here: http://blogs.msdn.com/astoriateam/archive/2009/08/31/ado-net-data-services-v1-5-ctp2-now-available-for-download.aspx Anthony.

If you want to build a service layer for your app in a flash, ADO .NET Data Services is the bee’s knees in my opinion. That said, being a new technology there are a few shortfalls I’ve come across.

The first is a lack of support for retrieving a count of records, which is a must if you intend to do paging of data. Thankfully, Microsoft has released a v1.5 CTP which addresses this (see my previous blog post).

The second glaring omission is the inability to only return selected properties from an entity in a query. This is otherwise known as projection, but I think of it as a traditional SELECT statement in SQL. Currently, a Data Service will always return all properties of all entities (or in my SQL analogy, SELECT *).

When you remember that you’re sending all data over HTTP from the server to the client, this can have a massive performance impact as you are almost always retrieving way more data than you need.

No way José! I’m sure Microsoft is well aware of this and are beavering away on a great solution, but in the meantime I’ve come up with a very dodgy, but equally effective solution in the form of a Http Module.

Download Select Filter for ADO.NET Data Services

After spending considerable time fiddling with Query Interceptors in my data services, there really is no way to only return selected properties from an entity. The only workaround at a code level would be to create custom entities that only have the properties you need – but that ‘aint pretty.

So I ended up using more of a “brute force” solution, by letting the Data Service do its thing, but then intercepting the Http response before it’s sent back to the client. By creating a custom Http Module, I was able to catch the response, remove the unwanted properties, and then stream the modified response to the client.

Note #1: I’m not using the ADO .NET Data Service Client libraries to consume my services, I call the services directly using AJAX. If you’re using the client libraries through Silverlight etc, I doubt this solution will be any good for you.

Note #2: My solution only works with JSON responses at the moment, as this is all I use. However it would be very easy to add support for XML if required.

Here’s how to get up and running:

1. Download the library and add a reference to AntsCode.DataServiceSelect.dll in your Data Services project.
2. Add the following HttpModule to your Web.config:

<httpModules>
    <add name="DataServiceModule" type="AntsCode.DataServiceSelect.DataServiceModule"/>
</httpModules>

That’s it really. Now, when you make a data service request, you can add a new select parameter, specifying the properties to return:

http://yourserver.com/YourService.svc/Customers?select=FirstName,LastName

The HttpModule will intercept the response, and filter out unwanted properties:

{
    d: [
    {
        “FirstName”: “John”,
        “LastName”: “Bloggs”
    },
    {
        “FirstName”: “Anne”,
        “LastName”: “Jones”
    }
]}

How is the data filtered? While I could have deserialized the JSON response into an object, and somehow removed the unwanted properties programmatically; I instead opted to use regular expressions and just strip out the unwanted text as I felt this would be much more efficient (I told you the solution was dodgy!!) You can see this in action in the DataServiceJsonFilter class in the source code.

Of course I’m really hanging out for an update from Microsoft to do this properly, at which point I’ll promptly throw this code in the bin. In the meantime, if you have any feedback or bugs, please report them on the CodePlex site.

Anthony.

YUI DataTable with ADO.NET Data Services

The Yahoo User Interface library (YUI) is an awesome client-side UI library, and ADO.NET Data Services is an equally awesome server-side framework for exposing a service layer for your data.

However, there is little documentation on how to get the 2 technologies to talk to one-another. Microsoft are pitching ADO.NET Data Services to marry up with Sliverlight - and rightly so, but like a lot of people, my skills are firmly grounded in AJAX.

Searching around, Jim Fiorato has written a great article on integrating a data service with a YUI chart but that's about it at this stage.

So far I've managed to get a YUI DataTable to display data from an ADO.NET Data Service, along with paging and sorting.

ADO.NET Data Services v1.5 CTP1

I need to highlight a shortfall in the current version of ADO.NET Data Services (1.0) - it does not return a count of all rows. Without this, displaying pages of data is nigh impossible as we simply don't know how many pages to display.

Thankfully, the issue has been resolved in the v1.5 CTP, you can download it here. Note that this is a pre-release version so I'm really hanging out for the final release.

Sample Project on CodePlex

I've created a sample project that contains a simple ADO.NET Data Service, and a sample page that implements a YUI DataTable. Note: You must first install the v1.5 CTP for it to compile.

I'm hoping to add more samples of other YUI components down the track.

Instead of writing a huge blog post explaining all the steps, I've added detailed code comments in the sample project.

Anthony.

Strip Formatting on Paste using YUI Rich Text Editor

I’ve been using the Yahoo User Interface Library (YUI) in my web app, and one particularly cool component is the YUI Rich Text Editor: its cross-browser compatible, fully extensible, and best of all it’s free :)

For all its greatness, one thing I’ve struggled with is that if you copy & paste stuff from another app into the YUI Editor, all of the original formatting is maintained. Most of the time for us, the “other app” is Microsoft Word, which does a particularly heinous job of generating HTML from a formatted document. This almost always wreaks havoc if the user subsequently tries to change text styles in the editor, as the underlying HTML is a total mess.

So, the solution for us was to try and strip out all of the formatting when somebody pastes stuff into the Editor, resulting in nice clean HTML that plays well with the YUI Editor formatting functions. Now, unfortunately no such feature exists in the YUI Editor, and Googling around just led to dead ends, so I was left to build my own.

CleanPaste for YUI Rich Text Editor

Download from CodePlex

For information on what’s supported, please read the notes on CodePlex, as I will keep this updated as I make bug fixes etc.

To use the CleanPaste script, follow these steps:

1. Ensure you’ve already installed the YUI components & created a Yahoo Editor on your page.
2. Place the CleanPaste.js file somewhere in your project.
3. Include the script in your page using the following code:
   
   <script type="text/javascript" src="CleanPaste.js"></script>
   
   Ensure the src attribute points to the directory where the CleanPaste.js script is located.
   
4. In the Javascript where you create your Yahoo Editor object, create an instance of the CleanPaste object, passing in the editor as the parameter:
   
   var myEditor = new YAHOO.widget.Editor('editor', myConfig); myEditor.render();
   
   var cleanPaste = new CleanPaste(myEditor);

That’s it, the editor should now strip the formatting out of pasted text.

Of course this is still under development, so if you have any problems or feedback please post on CodePlex and I will get back to you.

Cheers, Anthony.