All the downloadable stuff is already available (mostly in Spanish) here: http://blogs.msdn.com/gardel/archive/2008/11/20/pdc-buenos-aires-materiales-y-links.aspx.

My friends Kzu and Vga were also presenting a nice preview of VS2010.

It was a great experience, thanks Miguel!

Geotagging items is not a new thing, and many web sites already supports adding geographic information to the objects their handle. For example, Flickr allows you to set the coordinates where the picture was taken. Wikipedia also has structured information that contains the latitude and longitude for articles about a place in the world. On the other side, specs like GeoRSS can be used to augment the information given by a feed. However, even though all these new geo-related features are being widely adopted, there are still much information out there, that would need a human reading the text to understand which places is it mentioning.

So, we decided to make this process automatically as most as possible, extracting the information from the text itself. This is know as "location extraction", and is actually a branch of a more general thing named "entity extraction".

The location extraction process can be splitted in two stages:

Infer the possible location mentions in the text

The first step in location extraction is to obtain the possible terms in the text that could be talking about a location in the world.

This is being accomplished using a gazetteer plus some heuristics. A gazetteer it's like a world location database, and in this case I used the information that can be downloaded from the site of GeoNames. This database contains dozens of million of places all around the world, each one with alternative names, kind of place and geographical coordinates between some other things.

The matching cannot be done simply by looking the words in a dictionary because most place names are composed by more than just one term (i.e.: Buenos Aires). This is why I used a variant of the Aho-Corasick algorithm in which the tokens are whole words instead of characters. A good description of this algorithm can be read in these slides. This means that I had to pre-process the GeoNames database (consisting in a bunch of zipped text files) and construct a directed graph. I decided to use SQLite as the store for this huge graph as it is really fast and lightweight.

The construction of this database graph is a slow and memory hungry process. My first approach was to implement it with a Python script and it was enough to process test data from one or two countries. But for the final database that implies millions of nodes, the lightweight of C++ made a good job and I end up with a process that takes around 20 minutes in my Core 2 Duo notebook at 2.2 GHz. I think it's important to mention that I used a ramfs file system to construct the SQLite database. This really makes a big difference. As I taken only the most important places for my database, the final version sizes around 500Mb.

Once the database is ready, the graph can be traversed as a kind of state machine with the words being read from the text. Every time a match is found, it's saved in a list for later disambiguation. The Aho-Corasick algorithm finds all the occurrences of substrings, so those that overlaps must be discarded. For instance if we read "San Fernando" we have a positive match that must be later discarded if we read "San Fernando del Valle de Catamarca". This is very straightforward if we save the starting word index and length of each location mention, so for each match we look up the list of previous matches from the bottom, removing all those entries that would overlap with the new one.

To improve the accuracy of the matching algorithm I added these heuristics to the process:

1. Every place must start with a capital letter: If the name is composed by more than one word, the following words are not enforced to start with upper case. This doesn't work for all languages, but at this stage I only need it does with English and Spanish.
2. Uppercase chains must not be broken: this means that consecutive uppercase words must be taken together or not taken at all.

Resolve ambiguities

Before an specific location name extracted from the text can be placed on a map we have to disambiguate it from different meanings. For example, there are dozens of places named 'Springfield' and not only in US. So the meaning of a particular name must be inferred from the context. For this part of the process I used mainly this paper.

The central idea is calculate a weight for each of the senses of every location. This weights are calculated taking into account the other senses of other locations coocurring in the same text, and the distance of these terms in the text.

To calculate the weight of a sense we use the following formula:

Where "num" is the number of mentions of a place, "dist" represents the distance in words between two locations, and "weight" represents the influence of a location sense to other sense of a different location, and is calculated with the following table:

Some heuristics are also applied in this stage of the process. For example, if a location name can be interpreted as a country, other meanings are discarded. Also, if the same location name appears many times in the same text, the sense with maximum weight is taken and propagated to the other mentions, based on the "one sense per discourse" principle.

To improve the accuracy and based on my own tests, I introduced these modifications to the original formula:

1) Increase weight based on the location inside the text: the weight of the locations found at the beginning of the text, or even at the title, are multiplied by a factor that decreases exponentially as we advance in the document.

2) Reduce the weight based on a blacklist: Many locations has names of common words. The weight of these locations are reduced if are found in a blacklist to avoid potential mismatches. This blacklist can also be populated based on feedback of the users of the final system.

Once the algorithm has finished, only those places with weights above the 5% of the maximum weight are considered for the final result.

Next steps

The algorithm described here is far from being perfect. There is some much work done out there about this subject and I need to introduce some interesting stuff. I found that the biggest effort should be placed on the first part of the process that currently is introducing some noise because of false positive matches.

These are some of the points in my wishlist:

1) Use pattern matching before the disambiguation algorithm for patterns like "<city>, <state>" or "<city>, <country>", or even "city of <city>".

2) Prepare a better blacklist based on statistics. These can be obtained looking for common words appearing in hundead of texts.

3) Improve the mention lookup with previous processing. For example using sentence splitting and POS tagging.

I hope you found this article interesting, as it was to me to implement such process. The results are truly amazing!

In fact the implementation supports not only double clicks, but multiple clicks being able to specify the max number of clicks we want to receive, and the number of actual clicks are received in the event handlers.

There were previous approaches but the Ary's one feels much more natural to use.

I wish Microsoft adds support for this kind of basic events in the final release. However in this way we don't have to miss these useful gestures.

Good job Ary!

In that list we can find two useful options related to text selection, that has different
default values depending on the platform (Windows, Linux, Mac).

The first option "layout.word_select.stop_at_punctuation" controls whether the selection
when you do a double-click or use ctrl+arrows to select text, stops at dots, slashes and
other symbols. Enabling this option is specially useful when trying to change some part of
the URL in the address bar. Otherwise the entire URL is selected, thing that can still be
done triple-clicking on the address.

This option is selected by default in Windows version of Firefox, but is not in the Linux
version. There is a long discussion about how it would be in future versions.

The second option "layout.word_select.eat_space_to_next_word" is related to the selection
of surrounding spaces when double-clicking on a word on a page text. I think this option
should be always set to false, because sometimes is really annoying specially when copying
and pasting usernames and passwords. This time the value I consider more useful is the
default in Linux and the opposite happens in Windows. No exact idea of what happens on the
Mac's world .

There are many mock frameworks available for .NET but all of them have one or both of the following problems:

1. The framework rely on the member names as strings, making really difficult the refactoring sessions.
2. The mock objects are created based on the interaction and sequences of calls.

The last problem is probably subjective, but I believe the unit tests should not rely on the interaction of the tested entity with the outside. Otherwise, these tests would be tightly coupled with the implementation making possible refactoring and optimizations almost impossible without rewriting the entire test suite.

So, how to solve these issues and what LINQ has to do with all this? Moq is the answer! And you can read the "official" announcement in the kzu's weblog. I've been working with him in this project as was completely amazing to have the first prototype working in just an hour of pairing and having written just a bunch of dozens of code lines! LINQ rocks!

Many examples are given in the project site, but I'll copy some of them so you can get excited:

var mock = new Mock<ICloneable>();
var clone = new object();

mock.Expect(x => x.Clone()).Returns(clone);

Assert.AreEqual(clone, mock.Instance.Clone());

I think the framework interface is self explanatory. Even for more complicated examples like this one with callbacks:

var mock = new Mock<IFoo>();
bool called = false;
mock.Expect(x => x.Do1()).Callback(() => called = true).Returns(1);

Assert.AreEqual(1, mock.Instance.Do1());
Assert.IsTrue(called);

Or even very flexible filters (powered by LINQ):

var mock = new Mock<IFoo>();

mock.Expect(x => x.Duplicate(It.Is<int>(value => value < 5 && value > 0))).Returns(() => 1);

Assert.AreEqual(1, mock.Instance.Duplicate(3));

Enjoy!

Now with .NET 3.5 the WCF framework is extended to support natively these kind of scenarios and is really easy and clean to use. There are some good tutorials describing these extensions, like this one.
So, I've put hands on these exciting tools, and migrated my previous example to use the new framework. Ideally I would just remove my customizations, configure the endpoint with a webHttpBinding and a webHttp behavior. But this wont work because in my service contract I have some operations receiving unsupported argument types like int?, DateTime? and string[]. Fortunately, one more time the WCF team did and excellent job providing us the extensibility points we need .

Instead of configuring directly the webHttp behavior, I've created a custom DeliciousWebHttpBehavior (extending WebHttpBehavior) and overriding only the GetQueryStringConverter method. The QueryStringConverter is responsible of converting the arguments to a string representation and back again. Creating a DeliciousQueryStringConverter that supports our special types is fair enough.

After that I only have to change the attributes in the operations, so for example where in the previous example says:

[OperationContract(Action = "/v1/posts/get")]
PostsResponse GetPosts(string tag, DateTime? dt, string url);

in the new version is changed by:

[OperationContract]
[WebGet(UriTemplate = "/v1/posts/get?tag={tag}&dt={dt}&url={url}")]
PostsResponse GetPosts(string tag, DateTime? dt, string url);

Happy RESTing! .

Download: wcfrestpox35.zip

Public Sub RemoveSortAndSaveSelectedItems()
    Try
        DTE.ExecuteCommand("Edit.RemoveAndSort")
    Catch
    End Try
    DTE.ExecuteCommand("File.SaveSelectedItems")
End Sub

Now this will be on my list of the very first things to do when reinstalling a development machine .

I remember from my high school times, the internal spontaneous competitions looking for the "fastest" prime generators. We were really far from the greatest and well known prime generators, but things like that one gave us funny ways to learn interesting stuff. The algorithm used in this post is a naïve version, that has nothing to do with "fast".

Since C# 2.0, we can write custom enumerators very easily and create "generators" like this one:

IEnumerable PositiveIntegers
{
  get {
    for (int i = 1; ; i++)
      yield return i;
  }
}

Now with LINQ, is very easy to write lazy evaluated sequences like this one:

var primes = from n in PositiveIntegers.Skip(1)
             where
                 PositiveIntegers.Skip(1)
                     .TakeWhile(x => x <= Math.Sqrt(n))
                     .All(x => (n % x) > 0)
             select n;

or even using subqueries:

var primes = from n in PositiveIntegers.Skip(1)
             where
                 (from m in PositiveIntegers.TakeWhile(x => x <= Math.Sqrt(n))
                 where m >= 2 && (n % m) == 0
                 select m).FirstOrDefault() == 0
             select n;

Now you can assume that primes contains all the infinite series of primes! In fact it's a lazy evaluated enumerator, that searches for the next prime every time you call the MoveNext method.

So, you can search the primes less that 100:

foreach (var x in primes.TakeWhile(n => n < 100))
    Console.WriteLine(x);

the first 100 primes:

foreach (var x in primes.Take(100))
    Console.WriteLine(x);

or look for the 200th prime:

var prime200 = primes.ElementAt(200);

Of course, don't try to iterate the entire sequence .

Kenny Wolf describes in a post how to reuse an implementation of an ASMX Web Service, "double-decorating" the classes to support both ASMX and WCF with the same sources.

Kirk Allen Evans extends that idea allowing the clients remain unchanged in a first phase. That means the web services' URL remain with the same .asmx extension replacing the ASP.NET build provider:

<buildProviders>
  <remove extension=".asmx"/>
  <add extension=".asmx" type="System.ServiceModel.Activation.ServiceBuildProvider, System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"/>
</buildProviders>

Based on my experience, I can add some comments:

1. Action names

The default action names that WCF assigns for each operation is different than ASMX ones. ASMX uses the namespace appended by the message name, while WCF uses the namespace, the service name and the operation name. This means that if you want to be fully compatible with old clients, you have to specify the Action name:

[WebService(Namespace = "http://mynamespace/")]
[ServiceContract(Namespace = "http://mynamespace/")]
public class MyService
{
   [WebMethod]
   [OperationContract(Action = "http://mynamespace/TheWebMethod")]
   public void TheWebMethod()
   {
      // . . .
   }
}

2. Message names

If you used the MessageName argument for a web method, then you have to specify the corresponding Name argument in the operation contract. Also, the action name must be modified:

[WebService(Namespace = "http://mynamespace/")]
[ServiceContract(Namespace = "http://mynamespace/")]
public class MyService
{
   [WebMethod(MessageName = "MyWebMethod")]
   [OperationContract(Action = "http://mynamespace/MyWebMethod", Name = "MyWebMethod")]
   public void TheWebMethod()
   {
      // . . .
   }
}

3. Xml serialization

In the most common cases you also created your own "data contracts" for ASMX, creating Xml serializable clases. In order to support the same serialization you have to add the XmlSerializerFormat attribute:

[WebService(Namespace = "http://mynamespace/")]
[ServiceContract(Namespace = "http://mynamespace/")]
[XmlSerializerAttribute]
public class MyService
{
   [WebMethod]
   [OperationContract(Action = "http://mynamespace/TheWebMethod")]
   public MyDataType TheWebMethod()
   {
      // . . .
   }
}

4. ASP.NET compatibility

In those cases where you used ASP.NET specific features, or just for example want to have access to the current HttpContext and don't want to change all the source code to make it in the WCF way, you can take advantage of the ASP.NET compatibility mode of WCF.

<system.serviceModel>
  <serviceHostingEnvironment aspNetCompatibilityEnabled="true" />
<system.serviceModel>

The problem with this mode, is that if you replaced the .asmx build provider as I mention before and pointed by Kirk your services will fail miserably with an error like this:

Unable to cast object of type 'System.Web.Compilation.BuildResultCustomString'
to type 'System.Web.Compilation.BuildResultCompiledType'.

Fortunately, there is a very simple solution for this. Just replace the .asmx HTTP handler with the WCF one:

<httpHandlers>
  <remove path=".asmx" verb="*" />
  <add path="*.asmx" verb="*" type="System.ServiceModel.Activation.HttpHandler, System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" validate="false" />
</httpHandlers>

There are also some other important points to take into account in order to be fully compatible with old clients. For example, ASMX allows thin clients to invoke services using HTTP GET and/or PUT request with arguments encoded in the query string and the request body respectively. There are some improvements for this in Orcas that would support this out of the box, but in order to be fully compatible, all this should be supported by the same endpoint at the same time without forgetting the SOAP protocol. I had to deal with this also, but that gives enough material for a future post .

In this example I created a client behavior that communicates with a REST/POX service. Clemens Vasters have an excelent series of articles on how to create REST/POX services with WCF. There is also two articles here and there in MSDN but what I don't like about it is that you loose the type safe contracts, having to deal with Message class in order to send a request or decode the response.

What I did to avoid this problem is use a series of extension points in order to customize the request and response messages. In the example I used the del.icio.us API.
These are the WCF hooks I've implemented:

• IEnpointBehavior: attaches the message inspector and also configures itself as an operation behavior in every contract operation.
• IOperationBehavior: configures the message formatter.
• IClientMessageInspector: customizes the request message suppressing the body, changing the HTTP method and headers, and setting the query string previously created by the message formatter. This is done setting an instance of HttpRequestMessageProperty in the message properties. This is later used by the HTTP channel.
• IClientMessageFormatter: This implements both the request and response customizations. In the SerializeRequest method, it creates the query string and attach it as a property in an empty message. This is later used by the message inspector to create the final request. It also uses the Action value of the OperationContract to use as the relative URI. In the DeserializeReply method, the received message is deserialized as a plain XML using the XmlSerializer.

Once the behavior is attached to the client endpoint, the contract methods can be defined as:

[OperationContract(Action = "/v1/posts/recent")]
PostsResponse GetRecentPosts(string tag, int? count);

That makes a request to the address defined in the Action, with two arguments (tag and count) in the query string.

IMPORTANT NOTE: Take special care in the usage of this example. It doesn't take into account any of the important notes about the del.icio.us api usage, and you can be throttled. Use under your own risk. Again, this is just an example.

In order to use this example, first you have to provide your del.icio.us username and password to WCF, setting it in the Program.cs.

There is a number of things to improve. For example, right now it only supports GET requests. It should also handle POST method serializing the arguments as a POX request. Also, the query string arguments formatting should be customizable.

Download the sources: WcfRestPox.zip