Menu

An update to “Remote File Sync using WCF and MSF”

This is a follow-up to Bryant Likes’ post where he gave a prototype solution for file synchronization over WCF. I converted the code to Microsoft Sync Framework 2.0 so now it compiles and seems to run well enough. But you have to keep in mind that it isn’t a complete example (it wasn’t that in the original code either): it only does upload sync, it doesn’t have any conflict resolution logic etc. It is my opinion that this is not really worth pursuing any further, because one would need to develop two complete custom providers - and all that (just for copying files?) when there’s an existing FileSyncProvider in the framework which knows how to cooperate with other providers so it should be able to communicate over WCF… On the other hand, if you do pull the heroic act of completing this code, please let me know because I’m (obviously) very interested in it. I tried to keep the modified code as close to the original as possible so that a simple diff (e.g. WinMerge) can show what I’ve done, because I’m not sure I got it all right (as I’m afraid Bryant wasn’t, too).

Here’s the complete solution: http://www.8bit.rs/download/samples/RemoteSync converted to MSF 2.0.zip

Replicating self-referencing tables and circular foreign keys with Microsoft Sync Framework

Self-referencing tables – or, at least circular foreign key references between tables – are probably a common thing in all but the simplest database designs. Yet Microsoft Sync Framework doesn’t have a clear strategy on how to replicate such data. I found various suggestions on the net: order rows so that the parent records come before children – this is usable for self-referencing tables (although not endorsed by Microsoft because the framework doesn’t guarantee it will respect this order), but not nearly good enough for circular references – if you have two rows in two tables pointing at each other, ordering them cannot solve the problem. On an MSDN forum there was a suggestion to temporarily disable foreign key constraints: this I cannot take seriously because it opens my database to corruption, all it takes is one faulty write while the constraint is down and I have invalid data in the database (unless I lock the tables before synchronization, and I’m not sure how to do this from within the Sync Framework).

So, when all else fails, you have to sit and think: what would be the general principle for solving this, Sync Framework notwithstanding? Exactly - do it in two passes. The problem is present only when inserting rows, if the row contains a reference to another row that wasn’t yet created, we get a foreign key violation… Our strategy could be to insert all rows without setting foreign key field values, then do another pass to just connect the foreign keys. If we do this after all tables have finished their first pass (inserts, updates, deletes and all), we also support the circular references because required rows are present in all tables. Ok, that was fairly easy to figure out (not much harder to implement either, but more on that later). We have another issue here that is not so obvious, deleting the rows… There may be other rows referencing the one we are deleting that haven’t yet been replicated. Since the Sync Framework applies the deletes first, we can be fairly certain that the referencing rows are yet to be replicated - they will either be deleted or updated to reference something else. So we can put a null value in all fields that reference our row. (Note that this will probably mark the other rows as modified and cause them to be replicated back – this is an issue I won’t go into in this post, but I’m quite certain there needs to be a global mechanism for disabling change tracking while we’re writing replicated data. I currently use a temporary “secret handshake” solution: I send a special value - the birth date of Humphrey Bogart - in the row’s creation/last update date fields that disables the change tracking trigger).

Ok, on to the code. I won’t give you a working example here, just sample lines with comments. You’ve probably figured out by now that it will be necessary to write SQL commands for the sync adapter by hand. I don’t know about you, but I’m no longer surprised by this: many of the tools and components we get in the .Net framework packages solve just the simplest problems and provide nice demos – if you need anything clever, you code it by hand. My solution was to create my own designer/code generator, and now I’m free to support any feature I need (also, I am able to do it much faster than Microsoft, for whatever reason: it took me a couple of days to add this feature… It may be that I’m standing on the shoulders of giants, but the giants could really have spared a couple of days to do this themselves). For simplicity, I’ll show how to replicate a circular reference: there’s an Item table that has an ID, a Name, and a ParentID, referencing itself. For replication, I split the table into two SyncAdapters: Item, that inserts only ID and Name and has a special delete command to eliminate foreign references beforehand, and Item2ndPass, which has only the insert command – but the only thing insert command does is wiring up of ParentID’s, it does not insert anything. I’ve deleted all the usual command creation and parameter addition code, the point is only to show the SQL’s, since they hold the key to the solution.

[Serializable]
public partial class ItemSyncAdapter : Microsoft.Synchronization.Data.Server.SyncAdapter
{
	partial void OnInitialized();

	public ItemSyncAdapter()
	{
		this.InitializeCommands();
		this.InitializeAdapterProperties();
		this.OnInitialized();
	}

	private void InitializeCommands()
	{
		// InsertCommand
		// 1899-12-25 00:00:00.000 is a 'Humphrey Bogart' special value telling
		// the change tracking trigger to skip this row
		this.InsertCommand.CommandText =  @"SET IDENTITY_INSERT Item ON
INSERT INTO Item ([ID], [Name], [CreatedDate], [LastUpdatedDate]) VALUES (@ID, @Name,
@sync_last_received_anchor, '1899-12-25 00:00:00.000') SET @sync_row_count = @@rowcount
SET IDENTITY_INSERT Item OFF";

		// UpdateCommand
		this.UpdateCommand.CommandText = @"UPDATE Item SET [Name] = @Name,
CreatedDate='1899-12-25 00:00:00.000', LastUpdatedDate=@sync_last_received_anchor WHERE
([ID] = @ID) AND (@sync_force_write = 1 OR ([LastUpdatedDate] IS NULL OR [LastUpdatedDate]
<= @sync_last_received_anchor)) SET @sync_row_count = @@rowcount";		

		// DeleteCommand
		this.DeleteCommand.CommandText = @"UPDATE Item SET [ParentID] = NULL
WHERE [ParentID] = @ID DELETE FROM Item WHERE ([ID] = @ID) AND (@sync_force_write = 1 OR
([LastUpdatedDate] <= @sync_last_received_anchor OR [LastUpdatedDate] IS NULL))
SET @sync_row_count = @@rowcount";

		// SelectConflictUpdatedRowsCommand, SelectConflictDeletedRowsCommand
		// skipped because they are not relevant

		// SelectIncrementalInsertsCommand
		this.SelectIncrementalInsertsCommand.CommandText = @"SELECT  [ID],
[ParentID], [CreatedDate], [LastUpdatedDate] FROM Item WHERE ([CreatedDate] >
@sync_last_received_anchor AND [CreatedDate] <= @sync_new_received_anchor)";

		// SelectIncrementalUpdatesCommand
		this.SelectIncrementalUpdatesCommand.CommandText = @"SELECT  [ID],
[ParentID], [CreatedDate], [LastUpdatedDate] FROM Item WHERE ([LastUpdatedDate] >
@sync_last_received_anchor AND [LastUpdatedDate] <= @sync_new_received_anchor AND
[CreatedDate] <= @sync_last_received_anchor)";

		// SelectIncrementalDeletesCommand
		this.SelectIncrementalDeletesCommand.CommandText = @"SELECT FirstID
AS ID FROM sys_ReplicationTombstone WHERE NameOfTable = 'Item' AND DeletionDate >
@sync_last_received_anchor AND DeletionDate <= @sync_new_received_anchor";
	}

	private void InitializeAdapterProperties()
	{
		this.TableName = "Item";
	}

} // end ItemSyncAdapter 
[Serializable]
public partial class Item2ndPassSyncAdapter : Microsoft.Synchronization.Data.Server.SyncAdapter
{
	partial void OnInitialized();

	public Item2ndPassSyncAdapter()
	{
		this.InitializeCommands();
		this.InitializeAdapterProperties();
		this.OnInitialized();
	}

	private void InitializeCommands()
	{
		// InsertCommand
		this.InsertCommand.CommandText =  @"UPDATE Item SET [ParentID] = @ParentID,
CreatedDate='1899-12-25 00:00:00.000', LastUpdatedDate=@sync_last_received_anchor WHERE ([ID] =
@ID) AND (@sync_force_write = 1 OR ([LastUpdatedDate] IS NULL OR [LastUpdatedDate] <=
@sync_last_received_anchor)) SET @sync_row_count = @@rowcount";

		// SelectIncrementalInsertsCommand
		this.SelectIncrementalInsertsCommand.CommandText = @"SELECT  [ID],
[ParentID] FROM Item WHERE ([CreatedDate] > @sync_last_received_anchor AND [CreatedDate] <=
@sync_new_received_anchor)";

	}

	private void InitializeAdapterProperties()
	{
		this.TableName = "Item2ndPass";
	}

} // end Item2ndPassSyncAdapter

In this case, it would be enough to setup the second-pass sync adapter to be executed after the first one. For circular references, I put all second-pass adapters at the end, after all first-pass adapters. Notice that the commands for selecting incremental inserts and updates read all columns - this is probably suboptimal because some fields will not be used, but it's much more convenient to have all field values handy than to rework the whole code generator template for each minor adjustment.

UPDATE (24.11.2010): I’ve added a source file with an illustration for this solution. I haven’t tested it (although it does compile and may well work) but it could be useful in showing the overall picture. It was created by extracting one generated sync adapter from my application, hacking away most of our specific code and then making it compile. http://www.8bit.rs/download/samples/ItemSyncAgentSample.cs Note that the file contains a couple of things that stray away from standard implementation, like using one table for all tombstone records, using the sample sql express client sync provider etc. Just ignore these. One thing, though, may be of interest (I may even do a blog post about it one day): the insert command knows how to restore autoincrement ID after replication, so that different autoincrement ranges can exist in different replicated databases (no GUIDs are used for primary keys) and identity insert is possible. This is necessary because SQL server automatically sets the current autoincrement seed to the largest value inserted in the autoincrement column. Keep in mind that this (as well as the whole class, for that matter) may not be the best way to do things – but I’ve been using it for some time now and haven’t had any problems.

Sync Framework 2 CTP2 – no SqlExpressClientSyncProvider yet

Ok, I’ve finally gotten around to installing the CTP2 of the Sync framework. Let's see what new and interesting stuff I got with it:

1. A headache.
2. Erm... anything else?

All witticism aside, a lot of details have probably changed, but the main gripe I had still stands: there is no support for hub-and-spoke replication between two SQL servers etc. (Oh, and the designer is still unusable… Two gripes).

As for the first thing, I hoped I was finally going to get rid of my SqlExpressClientSyncProvider debugged demo but no such luck. It turns out that nothing of the sort is (yet?) included in the sync framework. Judging by a forum post, v2 is soon due to be released, but any questions regarding the Sql Express provider are met with a dead silence. It doesn’t seem it will be included this time (9200 downloads of the SqlExpressClientSyncProvider demo are obviously not significant for these guys). You almost literally have to read between the lines: regarding information about this CTP, there was a very sparse announcement, and a somewhat misleading one at that (and since this is the only information you get, any ambiguity can lead you in the wrong direction).

The CTP2 release announcement said:

# New database providers (SqlSyncProvider and SqlCeSyncProvider)
Enable hub-and-spoke and peer-to-peer synchronization for SQL Server, SQL Server Express, and SQL Server Compact.

So, does SqlSyncProvider work in hub-and-spoke scenario? I thought yes. How would you interpret the above sentence?

The truth is that SqlSyncProvider cannot be used as local sync provider in a SyncAgent (that is, in a hub-and-spoke scenario as I know it) because it is not derived from ClientSyncProvider. The SyncAgent explicitly denies it - and throws a very un-useful exception that says, essentially “ClientSyncProvider”… Translated, this means: “use the Reflector to see what has happened”, which I did. The code in the SyncAgent looks like this:

public SyncProvider LocalProvider 
{ 
    get 
    { 
        return this._localProvider; 
    } 
    set 
    { 
        ClientSyncProvider provider = value as ClientSyncProvider; 
        if ((value != null) && (provider == null)) 
        { 
            throw new InvalidCastException(typeof(ClientSyncProvider).ToString()); 
        } 
        this._localProvider = provider; 
    } 
}

(Someone was too lazy to write a meaningful error message… How much effort does it take? I know I wouldn’t tolerate this kind of behavior in my company.)

So there’s no chance for it to work (or I’m somehow using an old version of the SyncAgent). Is there any other way to do a hub-and-spoke sync, without a SyncAgent? I don’t know of it. But the docs for the CTP2 say:

SqlSyncProvider and SqlCeSyncProvider can be used for client-server, peer-to-peer, and mixed topologies, whereas DbServerSyncProvider and SqlCeClientSyncProvider are appropriate only for client-server topologies.

I thought that client-server is the same as hub-and-spoke, now I’m not so sure… At the end, after hours spent researching, I still don't know what to think.

A macro to find missing files in Visual Studio Solutions

Or: how to solve the setup project message “ERROR: An error occurred while validating.  HRESULT = '80004005'” It seems that for a large part of features in Visual Studio .Net, the development stops at the point where they are mostly usable and effectively demo-able. The Microsoft people are very eager to show you how easy it is to solve a trivial problem with a couple of clicks, but they are very reserved once something really serious has to be done. A case in point: the Setup/Deployment projects in Visual Studio. (Yeah, the ones, Zero Click Deployment - they make it sound like it reads your thoughts - and the like). If you have a missing file in your solution, and if the file is of a non-critical type (e.g. a resource) so that the solution compiles, you have a big problem because the setup won’t. It will fail with a moronic message “ERROR: An error occurred while validating.  HRESULT = '80004005'”. Which file is missing? Well, if you really really care – go through all files in your projects and check (don’t forget to expand the controls, some RESX child file may be the culprit!) Another thing that can happen is that a reference in one of the projects is bad. For example, you had the project reference another project and you moved the other project out of the solution… The first project doesn’t use anything from it so that the build passes but the setup is not as forgiving. In this case, you need to check all references. If, like me, you have a solution that consists of thirty project and thousands of files, this would mean a major headache. One way to solve it would be to create a new, temporary setup project and add to it one by one all outputs from the original setup. Build after each step and when the error appears you’ll know which project is at fault. Now, if the first is the case (a missing source file), one possible solution is to automate the manual search by using a macro. Below you’ll find one, modified from the example on the excellent MZ Tools site. As for the missing reference, it is somewhat easier to find since there are considerably less references in a solution than project files (you can detect these using the one-by-one method described above). I’m leaving it to you as a TODO: improve this script to detect missing references, post it on your blog and let me know so I can add a link to it.

Option Strict Off
Option Explicit Off
Imports System
Imports EnvDTE
Imports EnvDTE80
Imports EnvDTE90
Imports System.Diagnostics
Imports System.Windows.Forms

Public Module Module2

    Sub FindMissingFiles()

        Dim objProject As EnvDTE.Project

        Try
            If Not DTE.Solution.IsOpen Then
                MessageBox.Show("Please load or create a solution")
            Else
                For Each objProject In DTE.Solution.Projects
                    NavigateProject(objProject)
                Next
            End If
        Catch objException As System.Exception
            MessageBox.Show(objException.ToString)
        End Try

    End Sub

    Private Sub NavigateProject(ByVal objProject As Project)

        Dim objParentProjectItem As ProjectItem

        Try
            objParentProjectItem = objProject.ParentProjectItem
        Catch
        End Try

        NavigateProjectItems(objProject.ProjectItems)

    End Sub

    Private Sub NavigateProjectItems(ByVal colProjectItems As ProjectItems)

        Dim objProjectItem As EnvDTE.ProjectItem

        If Not (colProjectItems Is Nothing) Then
            For Each objProjectItem In colProjectItems
                For i As Integer = 1 To objProjectItem.FileCount
                    Dim fileName As String = objProjectItem.FileNames(i)

                    If fileName <> "" And Not System.IO.File.Exists(fileName) _
                            And Not System.IO.Directory.Exists(fileName) Then
                        MessageBox.Show("File missing: " + fileName)
                    End If
                Next

                If Not (objProjectItem.SubProject Is Nothing) Then
                    ' We navigate recursively because it can be:
                    ' - An Enterprise project in Visual Studio .NET 2002/2003
                    ' - A solution folder in VS 2005
                    NavigateProject(objProjectItem.SubProject)
                Else
                    ' We navigate recursively because it can be:
                    ' - An folder inside a project
                    ' - A project item with nested project items (code-behind files, etc.)
                    NavigateProjectItems(objProjectItem.ProjectItems)
                End If
            Next
        End If

    End Sub

End Module
Subscribe to this RSS feed