JSON DateTime Serialisation Gotchas

Posted on March 22, 2012 by N

DateTimes are a bit nasty, really. They appear deceptively elementary and unthreatening, leading generations of programmers to misuse them, or even grossly underestimate them and attempt to roll their own datetime libraries, only to end up in no man’s land with war stories galore. At work, we recently encountered a problem where the dates on our front-end lagged the dates in our database by a day. This was immediately indicative of a time zone issue, but we weren’t sure exactly where it was coming from.

It turns out that the discrepancy was partially a result of our MVC3 web service serialising datetimes using its default JavaScriptSerializer, and then our front-end deserialising them using DataContractSerializer – they use different formats. But worse still, after doing a bit of testing, I discovered that JavaScriptSerializer’s serialisation and deserialisation operations aren’t inverse. That’s to say that if you use it to serialise a DateTime of a non-UTC kind (i.e. local or unspecified), then deserialise the result, you’ll end up with a different date to the one you started with.

Here’s what happens when you serialise a DateTime of 1/1/2000 12 am with its Kind set to UTC and Local respectively. A bigger number means a later date/time. Note that NZ is UTC+13, so 1/1/2000 Local is equivalent to 31/12/1999 11am UTC. As this is earlier than 1/1/2000 UTC, the Local  number is smaller than the UTC one.

JavaScriptSerializer DataContractSerializer
UTC \/Date(946684800000)\/ \/Date(946684800000)\/
Local \/Date(946638000000)\/ \/Date(946638000000+1300)\/

Prior to serialisation, both serialisers convert the DateTime to UTC if it’s Local/Unspecified – and in the case of NZ, this means subtracting time. The difference is that the DataContractSerializer keeps track of this subtraction by way of a timezone annotation, so that you can reliably deserialise it later. And it shows, because the DataContractSerializer will spit your original DT back at you if you point it at either of its serialised forms above. The same can’t be said for the JavaScriptSerializer, which treats everything as UTC, and hence, only succeeds with UTC DTs.

This won’t be an issue for people who always convert their DateTimes to UTC before sending them across the wire. It caught us out because we were transmitting Unspecified DateTimes which are treated the same as local dates, causing the JavaScriptSerializer to  convert them to UTC, subtracting 13 hrs in the process. Since we wanted to pass around semantically unspecified DateTimes (i.e. unknown time zone), it didn’t make sense to convert to UTC. So we decided to instead serialise them as strings in the format YYYY-MM-DDThh:mm:ss, a variation on a W3C suggested format found here.

Posted in C#, Web | Tagged , , , , , , , , , | 2 Comments

How to Add a User Control to a SharePoint Web Part

Posted on February 22, 2012 by N

By default, when you create a visual web part in a SharePoint 2010 project, Visual Studio adds a user control for you and performs the necessary plumbing to ensure it gets deployed to the Control Templates directory in the 14 hive. This is all very fine until you want to model a dialog or secondary display mode as another user control; VS does not offer an easy or obvious way to add a user control inside a web part.

I figured out the procedure for this a few months ago at work, and it has been living on our internal wiki since then, so I thought I’d post it here. It assumes you want to create a user control called UserControl.ascx either directly inside a web part, or inside a folder inside the web part (given by OptionalFolderPath).

  1. Right click your project and add a new user control.
  2. Drag the user control from the default location (ControlTemplates\ProjectName) to your web part.
  3. Edit the web part .spdata file in the following way:
    • The target for the new user control is probably set to ProjectName\WebPart instead of CONTROLTEMPLATES\ProjectName\WebPart, like the original user control. Change it to match.
    • The source for the new user control is probably set to something like ‘..\WebPart\OptionalFolderPath\UserControl.ascx’. If you want, you can remove the first two redundant segments.
  4. Change the namespace of the user control from the godawful ControlTemplates one, to one of your choice (generally, the same as your web part). If you have Resharper, you can do this cleanly in one swipe. Otherwise, manually:
    • Rename the namespace in your user control code-behind
    • Update the Control directive near the top of the ascx accordingly.
  5. Reference your new user control. The path will be something like “~/_CONTROLTEMPLATES/ProjectName/WebPart/OptionalFolder/UserControl.ascx”:
    • If you’re referencing it from your web part .cs, call TemplateControl.LoadControl with the above path.
    • If you’re referencing it from your original user control, add a reference like: <%@ Register TagPrefix=”Controls” tagName=”UserControl” src=above path %>
Posted in ASP .NET, SharePoint, Web | Tagged , , , , , , | 2 Comments

CSS Content Generation

Posted on September 17, 2011 by N

So, with all the excitement over the W3C CSS 2.1 spec being finalised just a few months ago (a mere 13 years after the advent of CSS2!), I recently decided to have a quick skim over it. In my defence, I was bored and the spec isn’t as long or dry as specs usually are… with the exception of the section which distinguishes block-level elements, block-level boxes, block container boxes and block boxes 😉

While reading it, I stumbled across something I knew very little about… CSS content generation. The idea is that you can use CSS to insert content into your document like this:

.greetings {
    counter-reset: personCount;
}

.greetings .greeting {
    counter-increment: personCount;
}

.greetings .greeting:before {
    content: "Hi " attr(name) "! ";
}

.greetings .greeting:after {
    content: counter(personCount) ". ";
}

<div class="greetings">
  <span class="greeting" name="Bob Loblaw" >You are #</span>
  <span class="greeting" name="Tobias Fünke" >You are #</span>
</div>

The above CSS uses content attributes which are fed by raw strings, HTML attribute values and CSS counters in order to render something like the following:
Hi Bob Loblaw! You are #1. Hi Tobias Fünke! You are #2.

Now I’m sure that many consider these features to be evil, as the entire purpose of CSS is to separate content from presentation, right? Well, I’d argue that their use is entirely justified in at least a couple of cases:

  • When you have no control over the rendering of some component in your webpage and you want to inject some content without resorting to Javascript. I’ve had to use this on more than one occasion, while taming the beast that is SharePoint.
  • When the content you’re inserting is presentational, or highly structured but trivial. Inserting chevrons before elements is one example. Another is using nested counters to prepend 1.1, 1.2… to items inside hierarchical lists.

Food for thought!

Posted in Web | Tagged , , , , , , | Leave a comment

Using SPWebConfigModification to Modify your Web.Configs

Posted on July 31, 2011 by N

SharePoint allows you to use SPWebConfigModification to programmatically make web.config changes which propagate across your farm. Typically, you do this by creating a SP feature and adding your code to its FeatureActivated and FeatureDeactivating event handlers. The advantage of this approach is that you can (ideally) keep the web.configs on your farm up to date and consistent without a lot of manual work. And if a new server is added to your farm, you (or even a sysadmin) should be able to update its web.configs by simply deactivating and reactivating a feature or two.

Unfortunately, SPWebConfigModification is scantly documented, difficult to use, non-transparent and notoriously prone to destroying your web.configs. After spending far too much time working with it, my advice right now is to avoid it like the plague. But if for some reason you decide to use it, hopefully this post will help you out.

So here are some of my observations/tips on the topic:

  • SPWebConfigModification::Sequence seems to do nothing as far as I can tell. If you want to override the ordering on modifications, you should append [1=1] , [2=2] etc. to the Path. This makes no real difference to the XPath expression but takes advantage of the alphabetic ordering which is applied.
  • When it comes to SPWebConfigModification::Type, do not use EnsureSection. This option creates web.config nodes which are are near impossible to remove. I tend to use EnsureChildNode.
  • SPWebConfigModification::Name is actually an XPath relative to Path and is used to uniquely identify that node after it is added. Consider the below appsetting. 
    <add key="CachePeriod" value="300" />

    We could add this node by creating a SPWCM and setting its name to add[@key=’CachePeriod’]. So now this node is uniquely identifiable by the combination of Path (its parent path) and Name (representing a node called Add with an attribute ‘key’ with the value ‘CachePeriod’. This app setting value could later be updated by applying an SPWCM with the same Path+Name and a different Value. If we didn’t want this appsetting to be updateable, we could incorporate its value in the XPath identifier.

  • If you use SPWebConfigModification through a web-app scoped feature, be sure to disable auto activation. Because unless your WSP contains web-app resources, it may end up being deployed globally and if auto-activation is enabled, this will result in the modifications being applied to all of your web apps rather than the one you want to target.
  • Don’t be surprised if any of the following occur:
    • Race conditions
    • SPWCMs being applied inconsistently across web app zones
    • Modifications being ‘remembered’ by SharePoint, even after you’ve done everything you can to destroy them.

And if you’re still wanting to use SPWebConfigModifications after reading all of this, don’t say I didn’t warn you 😉 If you need discouragement, you need only look at any of the hundreds of blog posts written by disillusioned programmers who have been forced to jump through hoops in order to insert a bit of text and a few settings into their config files. The very existence of simulators and wrapper utilities for SPWCM attests to its nebulous and capricious nature, and in most cases, I’d advise that you save yourself the trouble and use WinMerge in conjunction with a canonical web.config.

Posted in ASP .NET, C#, SharePoint, Web | Tagged , , , , , | 1 Comment

Should C# Offer the With Construct?

Posted on July 23, 2011 by N

As repulsive as I find the syntax of VB .NET, I recently stumbled upon one of its language features which piqued my interest – the With keyword, for which no direct equivalent exists in C#. It basically allows you to write code like the following (in VB).

With New Car
    .Colour = "Red"
    .Name = "Bob"
    .Start()
    .Drive(20, Direction.South)
    Console.Write(.DistanceTravelled)
End With

There seems to be quite a lot of controversy over this language feature and I get the feeling I’m one of the few C# coders who sees anything positive in it. So I’ll justify what I like about it:

  • Brevity:
    As in the example, you can instantiate a class and refer to it right there using a single period. Or maybe instead of referring to a new Car, you want to refer to an object buried deep in a hierarchy (i.e. house.Suburb.City.Country), but you don’t want to type out the entire structure in each of the five consecutive lines where you use it? The counter-argument is of course to create a temporary variable with a shorter name and to use that instead… but I find this approach (in some cases) to be cleaner and preferable for other reasons too.
  • Readability:
    With this code, the intention is obvious. It says “Ok, I have my object and now I’m going to make use of it in this clearly delineated and semantically cohesive block of code”.
  • Scoping:
    In the above example, the car cannot be referred to outside the With block… and provided that you’re finished with it by that point, this approach makes perfect sense. You can achieve the same thing in C# by simply placing all the code inside some braces in order to induce a new scope. But again, I’d hardly consider that readable or obvious.

Now naturally, code readability is a fairly subjective matter, and I’m sure that many C# programmers would still balk at a C# version of a With construct. But is it really so different in its virtues and appearance to object initialisers? They, also, achieve brevity and readability by grouping together operations on a single object, while offering what is effectively a shorthand way of accessing it. And what about using blocks? While working towards their purpose of simplifying resource management, they often incidentally achieve the readability and scope limiting I mentioned above.

So it strikes me that VB With could be considered as a generalised version of existing C# constructs which serve more specific purposes. And I’d say that it could add some value if used judiciously, where existing constructs aren’t applicable and refactoring into methods would be overkill. But just as GOTO is a more generalised and versatile version of practically every control flow structure, and is all the more abuseable for it, I imagine that With could be easily misused/overused, leading to wretched code packed with atrocities such as hulking nested With blocks and contrived attempts to write every single method as a sequence of With clauses. So handle with care?

Posted in C# | Tagged , , , , , | Leave a comment

It’s the Time of the Season (for Moving)

Posted on July 12, 2011 by N

So… some of you may have noticed that I’ve recently migrated this blog from WordPress to my very own domain. Now, whether this warrants a blog post named after a Zombies song is another matter altogether, but anyway…

One of the luring factors of the self-hosted solution was the ability to use WordPress plugins such as the Syntax Highlighter plugin for displaying code. Another factor was the possibility that I might one day want to host some files which aren’t strictly WordPress blog pages or even related to my blog at all… hence, why it’s living in its own subdirectory. Not to mention, it gives me the excuse to have a cheesy homepage! I decided to keep my old blog up (with a permanent redirect here) and maintain the old structure and hyperlink formats in the hope that the migration wouldn’t kill the SEO or lead to a proliferation of broken links. So let me know if you see something which has gone awry.

I apologise for being a bit slack with the frequency of blog updates – I blame it on my being ultra-busy with work, work-provided SharePoint training and various other RL things. I also blame it on a bunch of new DVDs which I’ve purchased in the past month 😀 But there’s a lot of things I want to write about, so expect something soon! Until then, I better get some sleep as I have a 6 am website deployment to do tomorrow.

Posted in General | Leave a comment

TFS Pending Changes – Ignoring Files which are Identical to the Originals

Posted on May 1, 2011 by N

Since posting about TFS a few weeks ago, I noticed I’ve received a lot of Google traffic in relation to a specific TFS annoyance I mentioned; the fact that when you view pending changes or check in files for a solution, it presents you with a slew of files that you’ve checked out at some stage, but haven’t actually modified. If you do a Compare on them, TFS will tell you “The files are identical”. Well since then, I’ve figured out how to purge these unwanted squatters.

You need to install TFS Power Tools and execute tfpt uu /noget /r * in the root of the branch. As a result, TFS will go through and undo checkouts for any unchanged files whilst leaving your modified files untouched. TFS is silly!

Posted in Version Control | Tagged , , , , , | 14 Comments

Migrating an SVN Repository to GIT

Posted on April 20, 2011 by N

Well, it’s been a week since my last post and it seems I’m writing about version control again, but this time with a more positive spin on things. SVN has served me well over the past couple of years but I’ve decided to make the jump to GIT for various reasons, not the least of which is the friendly peer pressure from some of my open source mates 🙂 Some of GIT’s features which appealed to me include its distributed model (which allows for flexible workflows), improved ignore settings, staging/stashing and seamless move/delete operations (which in my experience can be painful using TortoiseSVN).

So I got a GitHub account, installed MSysGit and TortoiseGit for Windows, had a go at creating my own repository and committing a few things, and all was well. So of course, the very next thing I tried was migrating the SVN repository for one of my projects (a Foobar plugin) whilst preserving the revision history. I first attempted this using git-svn clone (the recommended way) and it proved to be considerably more difficult and bug-ridden than I’d hoped. In the end, after hours of trying different combinations and hacking through Perl (I don’t even know Perl!), I switched to a more manual method. Nevertheless, I’m documenting a few of the bugs/errors I encountered because even though it didn’t work out for my particular SVN repo, chances are it’ll work for others, perhaps with the help of one of the workarounds below.

Migration can potentially be as simple as this, where SVN_REPO is the path to your existing central SVN repository and GIT_REPO is the path where the new GIT repository will be created.

git svn clone SVN_REPO GIT_REPO

 

Error 1:

Unable to open an ra_local session to URL: Expected FS format '2' found format '4'This was the first error I encountered. As it turns out, git-svn is built against older Subversion libs (1.4-1.5) whereas my SVN repo was 1.6. This can be resolved by doing mirroring your repo in compatibility mode like this:

svnadmin dump SVN_REPO svn_repo.svn_dump
svnadmin create --OPTION COMPAT_SVN_REPO
svnadmin load --pre-1.4-compatible COMPAT_SVN_REPO < svn_repo.svn_dump

 

Error 2:

Ignoring error from SVN, path probably does not exist: (160013)I never really figured this one out… even though it says it checked through 22 revisions, my GIT repo was empty. But by switching from using an svn:// URL to a file:// one, this error disappeared.

 

Error 3:

So I tried this:

And it seemed to do the job… until I encountered this at the end:

Permission denied: Can't open '/tmp/report.tmp' : Permission denied atI tried to diagnose the problem by tinkering with SVN repo settings and tracing back through source code to no real avail. But I noticed that if I called clone with an -r1 option to grab only the first revision, and then git-svn fetched each one individually using the same option, that error never crops up (in fact, none do)… but upon checking the GIT repo, it’s clear that its contents never advanced beyond r1.

So at that point I gave up and decided I’d be better off ‘migrating’ my repository manually… by creating an empty GIT repo to which I would commit the exported versioned files for each revision of the SVN repo, along with associated checkin comments. Painfully mundane, but 22 simulated checkins and half an hour later, I was reminded that the dumb solution is sometimes a decent option. And automating it through scripting would be a better option if your repo has a few gazillion revisions.

Posted in Version Control | Tagged , , , , , | Leave a comment

My Take on TFS

Posted on April 12, 2011 by N

So, for the past couple of months at work, I’ve been living in Microsoft land doing SharePoint /ASP .NET web development. Now I’m not some blind Linux evangelist bemoaning the fact that I’m tethered to a large Microsoft stack during the daytime. In fact, I quite like the Microsoft languages and development resources and have always found them to be top-notch (MSDN documentation excluded). That was until I came across TFS (Team Foundation Server) which we use at work. Now I’m sure that TFS is all very fine for project and release management, but it’s an absolute train-wreck for version control… and as such, I cannot fathom how it managed to be coupled with such a great IDE as Visual Studio.

Where better to start than TFS’ archaic checkout model which insists that you check out a file with the server before you can edit it? It implements this in a very primal fashion by adding a read-only attribute to all versioned files and removing it when checking out. This means you need to manually check out a file in Visual Studio before attempting to alter it using an external tool (heaven forbid you should want to use anything outside VS!). Furthermore, if you manually remove the read-only attribute yourself and edit it outside VS, TFS won’t pick up your pending changes come commit time (and this does make me wonder how it would handle being cut off from a TFS server in a disconnected/offline scenario – I haven’t tried). I really don’t understand what’s wrong with the idea of scanning for file changes like most other version control tools? But the best part comes when you go to check in your files, only to be confronted with a slew of files which you’ve checked out at some stage but haven’t actually modified. If you compare with latest, TFS will tell you that the files are identical. Gee, thanks TFS! EDIT: I’ve since posted a workaround for this here.

Another thing I love about TFS is when it gets into a state where “Get Latest” doesn’t actually get the latest. Or rather, TFS appears to update to the latest revision and presents you with the latest files in Team Explorer, but miraculously, your local files in Solution Explorer are completely outdated. This is a consequence of its centralised model where the server basically says “My files haven’t changed since your system last grabbed them so I’m not going to give them to you”… well, it’s a bit annoying when it gets that wrong. The workaround is to select Get Specific Version and force an overwrite.

TFS seems to have its share of niggling usability issues. Why is it that right clicking a file and selecting Undo sometimes undoes the pending changes for that file, and sometimes brings up a new window allowing you to undo pending changes over multiple files? And why can’t I simply right click a file and undo pending changes? Half of the motive for using TFS is that it’s (meant to be) fully integrated with VS… so when you View History for a solution in Solution Explorer, shouldn’t it present you with the pending changes for all files in the solution rather than the changes for the specific VS .sln file which is almost never what a dev is after?

And don’t even get me started on the default merge tool. When I first started, a few of my colleagues did warn me to avoid it, expressing the opinion that it was downright dangerous. So after trying it and confirming that for myself, the very next thing I did was download WinMerge and use that instead.

I realise that I’m only scratching the surface of TFS, but I really hope I find something to like about it because at the moment, I’d happily swap it for command line SVN.

Posted in Version Control | Tagged , , , , , , | 1 Comment

Generic Blog Update

Posted on March 27, 2011 by N

Well as it’s been so long since my last blog entry, it would be remiss of me not to provide at least some sort of update on what I’ve been doing. When I last posted, I had just finished my final exam ever and was enjoying some well-earned time off. The end of university also marked the closure of two interesting projects for me:

The first of these was my fourth year project on algorithm visualisation which I undertook with David Olsen. I’ve briefly mentioned it in the past and will probably blog about it  (and hopefully do some more work on it) in the future. For now, you’ll have to accept the quick and abstruse description of our work as an algorithm visualisation framework; basically a library which instruments C++ programs with code that animates their dataflow during execution. And the programmer need not use special data structures or manually invoke drawing routines; it’s just standard C++ with STL and a couple of extra header includes. It earned us first prize in the Software Applications category. 🙂

I worked on the second project with David and Sutirtha Basak for our advanced graphics class. We developed a building sketch program which generates 3D models of buildings from 2D sketch input. Most of the credit goes to David whose model generation algorithm impressed our lecturer to the extent that he asked us to co-author a paper on it, which we submitted to an international computer graphics conference. My part of it mainly involved detecting the best-fitting line of symmetry in the user’s sketch and manipulating the sketch by reflecting half of it accordingly, to make it truly symmetric.

Since then, I’ve been doing a bit of work on my personal project, an MP3 tagging plugin for the Foobar music player. But the more important news is that I got a job as a software developer for a large company in NZ. In fact, I’ve been working there for two months now, doing web development using C#/ASP .NET and (battling with) the SharePoint platform. I’ve enjoyed my time so far and it has been an eye-opening experience, given that I was 100% an application programmer before starting. But I’ve been picking things up reasonably quickly, and it feels good to be contributing valuable work straight off the mark, despite how apprehensive I may have been about my knowledge (or lack thereof) of all things webby…

Posted in General | Leave a comment