More Fun with C# 4.0 - Dynamic REST Service Calls

This followup post on the C# 4.0 dynamic feature demonstrates taking dynamic JSON further to dynamic or proxyless REST service calls turning dynamic method calls into HTTP requests...

In my last post, I blogged about the upcoming C# 4.0 dynamic feature and using it to work with JSON data in a more natural late-bound manner. I also alluded to where I was heading with the idea - issuing calls to REST services through a dynamic interface.

Specifically, I was thinking of being able to have a dynamic object represent a proxy to a REST service, and having its encapsulated late-binding behavior turn method calls into actual HTTP requests, serializing parameters into URL query string parameters, and finally processing responses (JSON or XML) into a dynamic object for the caller to poke into and extract interesting pieces of data. This post builds on the previous one, so be sure to check that out.

Also, do check out the comments. There was a lot of interesting commentary about how c# and dynamic don't mix together. It is an interesting debate no doubt, and I thought I should summarize my perspective. I personally tend to think dynamic is just another tool, and when used appropriately it is quite nice. Often times there is a small bit of code where late-binding would help, and its nice to be able to do so in c#, without switching to another dynamic language and losing all static typing in the process, or having to split your code artificially. Instead I envision using and encapsulating late-binding code within an object, and then using that object itself in a strongly-typed, compile-time-bound manner in the rest of the app.

Back to REST services... it is interesting to look at an example, such as Flickr.

Looking at its API documentation page, you can see Flickr offers around 100 or so APIs. Every single API has numerous permutations of parameters (just look at search for example). It would be an fairly big and non-trivial task to create a full-featured toolkit, and keep it up to date as the service evolves. More often than not an app just wants a handful of APIs. What if there was a generic and dynamic REST proxy that could out-of-the-box support typical REST services. That would at least facilitate quick and dirty prototyping even if you do decide later to turn just the APIs you care about into a strongly typed REST client library for the actual implementation. The same pattern repeats itself with other services such as Amazon, Facebook and so on.

Once I had this generic REST client coded up, I could write the following code to work against Flickr:

internal static class FlickrSample {
    private static void WritePhotos(dynamic list) {
        foreach (dynamic photo in list.photos.photo) {
            Console.WriteLine(photo.title);
            Console.WriteLine(String.Format("http://farm{0}.static.flickr.com/{1}/{2}_{3}.jpg",
                                            photo.farm, photo.server, photo.id, photo.secret));
            Console.WriteLine();
        }
    }

    public static void Run() {
        dynamic flickr = new RestClient(AppSettings.Default.FlickrUri, RestClientMode.Json);
        flickr.apiKey = AppSettings.Default.FlickrApiKey;

        Console.WriteLine("Searching photos tagged with 'seattle'...");
        dynamic photosOptions = new JsonObject();
        photosOptions.tags = "seattle";
        photosOptions.per_page = 4;
        dynamic searchResponse = flickr.Photos.Search(photosOptions);
        WritePhotos(searchResponse);

        Console.WriteLine("Searching interesting photos...");
        dynamic interestingList = flickr.Interestingness.GetList();
        WritePhotos(interestingList);
    }
}

I could also use the same exact generic REST client to work against Amazon, which returns XML rather than JSON (but that difference is handled through my library by creating a dynamic layer over System.Xml.Linq):

public static void Run() {
    dynamic amazon = new RestClient(AppSettings.Default.AmazonUri, RestClientMode.Xml);
    amazon.subscriptionID = AppSettings.Default.AmazonSubscriptionID;

    dynamic searchOptions = new JsonObject();
    searchOptions.SearchIndex = "Books";
    searchOptions.Keywords = "Dynamic Programming";

    dynamic bookList = amazon.ItemSearch(searchOptions);

    foreach (dynamic book in bookList.SelectAll("Item")) {
        Console.WriteLine(book.ASIN + " : " + book.ItemAttributes.Title);
    }
}

REST client itself doesn't know about Flickr or Amazon. It simply takes in the URI handed to it in its constructor, and does a few things:

  • Replace named tokens like apiKey={apiKey} in the URI with properties set on the RestClient object (eg. apiKey in the Flickr sample and subscriptionID in the Amazon sample)
  • Derives the REST operation name from the method call name in the code (Photos.Search, Interestingness.GetList and ItemSearch)
  • Serializes the JSON object passed in containing parameters as query string values
  • Deserializes the response using either the JsonReader for JSON responses or XDocument for XML responses into either a JsonObject or XmlNode (both are dynamic themselves).

You can check out the code and if you have the VS2010 and C# 4.0 CTP, you can run the sample (you'll need to get your own Flickr API key and Amazon subscription ID and place them in App.config before running them). The interesting places to put a breakpoint to see whats happening are in the GetMember(), SetMember() and Call() implementations of RestClient.

As a side note, I am dabbling a bit with github for source code hosting - the source code for the project is shared in a github repository, where you can browse the code online, follow any future changes, or even fork the code to experiment with your own ideas.


[ Tags: | | | | | ]
Posted on Wednesday, 11/5/2008 @ 3:02 PM | #Projects


Comments

10 comments have been posted.

Mike Amundsen

Posted on 11/5/2008 @ 4:55 PM
Nice examples. Good to see the dynamic aspects of C# being applied to REST-like coding. It'll be even better when the REST-like coding shows dynamic serialization types based on client-initiated content negotiation. Compiling the serialization format into the code is not only un-RESTful, it lowers the value and lifetime of the proxy.

Pedroafa

Posted on 11/11/2008 @ 12:59 AM
does dymanic feature work with CSV Rest Services too?.

Nikhil Kothari

Posted on 11/11/2008 @ 9:10 AM
Whats a CSV REST service? Something that returns data in csv format would be my guess...

As coded, the sample doesn't support CSV. However adding this support would be trivial - simply attach a CsvReader to the stream and create JsonObjects for each record. The source code is shared on github as noted in the post - if there is enough interest or someone willing to contribute it, I can certainly incorporate this.

Whats a good example of an interesting service that does this? Partly curious, and partly, that would certainly help make the case for supporting this.

Justin Grant

Posted on 11/17/2008 @ 1:16 AM
Hi Nikhil - how will RestClient deal with the usual requirements for ensuring performance and reliability of ASP.NET apps using remote calls, e.g. timeouts, batching, async, making multiple calls in parallel, etc. ? It's frustrating in today's .NET Framework how seemingly simple scenarios require large app changes. For example, if I make two webclient calls in parallel, I'd use this code:
string s1 = client1.DownloadString(url1)
string s2 = client2.DownloadString(url2)

Assume I want to make those two calls in parallel (and cancel them if they take too long). And (unlike today's *Async pattern on WebClient) I don't want to have to rewrite my app to split all processing into before-async-data and after-async-data-returned methods. I just want to make two calls in parallel. Will there be a way to do this? I'm OK with having to burn a web server thread while the calls are executing-- I generally only need async to reduce end-user latency, and never have enough traffic on the web server where another thread is a problem.

In other words, today the .NET Framework forces users who want async to make pretty radical app changes-- either to split processing into different methods, or to downshift to lower level classes like HttpWebRequest. Would be nice to have a middle ground where easy-to-use classes offered easier-to-use async.

Nikhil Kothari

Posted on 11/22/2008 @ 7:50 PM
For handling async, this is what I wanted to do using Amazon as a sample:
dynamic amazon = new RestClient(...);
dynamic searchOptions = new JsonObject();
// setup search options
dynamic callOptions = new JsonObject();
callOptions.callback = <callback function>;
callOptions.callbackContext = <some optional object>

dynamic searchCall = amazon.ItemSearch(searchOptions, callOptions);

whre searchCall has members such as asyncResult, cancel(), wait().

If you're ok burning a thread, then its a bit easier using the background worker sort of pattern, where on the thread you just do sync work. This could be easier in asp.net by optimizing for this out-of-the-box, but it isn't as burning a thread on the server usually isn't an option. Even if it is, you might as well burn the request thread - why spin up another?

Finally I truly behave the answer to more async and more parallel isn't in just frameworks, but in new language support and additionally the ability to model more of the intent and specify less of the specific implementation.

Joseph

Posted on 12/5/2008 @ 1:16 PM
sorry...just posted this q to the JSON article:
Is it possible to use the compiled RestClient binary in a .net 3.5 app or is dynamic keyword/feature tied to the .net 4 runtime?

Nikhil Kothari

Posted on 12/6/2008 @ 5:36 AM
The dynamic keyword compiles down to framework code that only exists in .net 4.0.

Chris Hay

Posted on 12/17/2008 @ 5:09 AM
On a seperate point Nikhil, I was thinking about what if there was a Microformat for REST API's. The documentation is already produced today by most folks offering REST API's it would be a small step to produce the documentation with a Microformat.

The advantage of this is that you could fire a client proxy generator (e.g. Visual Studio) at it, and have it autogenerate a proxy for you (giving you lots of nice intellisense).

Anyways, that was just the ramblings of someone who was up very late (my blog has more details)

http://silverlightuk.blogspot.com/2008/12/silverlight-rest-api-microformat.html

Nikhil Kothari

Posted on 12/19/2008 @ 12:10 AM
@Chris - I've actually spent some time thinking of a REST description language (RSDL) and even went partially down the path of trying to define it and implement it. There are a few challenges:

The least of which is REST services tend to differ in terms of structure from one service to the next. However there seems to be some general unification in terms of style when you look at things like Flickr, Facebook etc.
The more impactful challenge is that there are several combinations of using a service in terms of parameter groupings and values etc. that are hard to capture in order to make an intelligent and helpful proxy.

The flip side is having a generic proxy. This C# dynamic-powered approach gives you that for a large spectrum of services. Furthermore there are toolkits available for most of the interesting services in a variety of languages and developer frameworks, where someone has done some level of intelligent structuring of the APIs for the meaningful set of scenarios. This reduces the need for a generic description model and proxy generator.

Putting these together has left me wondering if there is in fact much value in something like RSDL. However I agree that if there is such value and it can be applied consistently across a broad range of services, a microformat is an interesting way to markup a documentation page meant for humans to also be consumable by machines...
Post your comment and continue the discussion.