Google Calendar API v3 Undocumentation

Over the many collective months that I’ve been working with Google Calendar API v3, I’ve noticed many undocumented “features” and behaviors. I thought I should sum up my experiences here so that someone else struggling to use this “simple” API would have at least some forewarning of the adventure they’re about to embark upon.

The documentation up on Google’s site is woefully incomplete. Yes, it lists all the methods and most of the parameters and such, the reference documentation; that’s great for starters, but it’s the bare minimum amount of useful information. What is completely missing is the documentation of behaviors you will encounter, what I call the narrative documentation. Google seems to be very bad about narrative documentation in general.

Uniqueness Constraints

What seems to be completely missing from the documentation is uniqueness constraints of objects (or resources, as Google calls them).

For example, an event is unique by its iCalUID property. Is this important fact mentioned on the event insert method‘s documentation page? Not at all. In fact, iCalUID is not even mentioned on this page. You have to go to the generic events resource page to find the first mention of iCalUID at all. Is the uniqueness constraint mentioned there either? Nope.

While we’re on the subject of inserting events, there’s also the import event, which I have no idea about what it does differently than the insert method, other than that they’ve declared that an iCalUID is required for the import method. The summary documentation is a useless one-liner: “imports an event.” Thanks, Sherlock; that was real helpful.

Furthermore, the only documentation about the iCalUID states that it should be an “Event ID in the iCalendar format.” I’ve found this to be completely untrue. That’s probably what the field is intended for, but there is absolutely no format validation for an iCalUID. You can put anything you want in here. (TWSS)

Colors

Google Calendar API gives you the ability to assign colors to events and also assign colors to whole calendars. What they don’t tell you is that assigning colors to events is completely useless unless the calendar the events are contained within is your own personal calendar. In other words, assigning event colors to a calendar intended to be shared with others is pointless. The only control over colors you have in that circumstance is to assign the whole calendar a single color specific to the user’s calendar list. If you really want colorization of events being shared with multiple users, your only choice is to split events across multiple calendars and assign the colors at the calendar level per each user. And of course, don’t forget the uniqueness constraint on the Summary property of the calendars you create!

Also, what they don’t tell you about colors is that there is one global palette of two kinds of colors: calendar colors and event colors. They do tell you there are two palettes, but they do not indicate whether they are global palettes or user-specific palettes. The two (calendar and event) palettes are not the same palette and an event colorId is not interchangeable with a calendar colorId and vice versa. Why use the same type name “colorId” to refer to two incompatible types? Why not just call one an “eventColorId” and the other a “calendarColorId”? Would that be so hard? To be fair, the documentation does make the distinction but it’s not obvious at first glance that the distinction is meaningful.

Furthermore, when Google duplicates events on your behalf (and they do – see the Side Effects section below), they don’t necessarily duplicate all properties, including the colorId property.

Recurring Events

Creating recurring events is extremely frustrating and fraught with many gotchas and stingers. I don’t even want to go into it here; avoid it at all costs if you value your sanity.

Side Effects

WARNING! Side effects of regular Google Calendar API v3 usage may include:

  • Adding email addresses as attendees copies the event to the attendees’ personal calendars. This creates a completely different event with its own eventId, unrelated to the one you created via the API. As far as I can tell, there is no programmatic way to determine if this duplicated event originated from the event you created via the API.
  • Deleting a user which owns calendars that are shared with other users will create a private copy of the shared calendars in each users’ calendar list and will only delete the original calendars owned by the user being deleted.
  • Deleting an event causes it to be marked as a dual “deleted/cancelled” state. I simply cannot figure out the difference between deleted and cancelled, if there is one.
  • Trying to re-create a previously deleted event will cause a 409 Conflict response. You must instead resurrect the deleted/cancelled event which has the same uniqueness properties as the one you are trying to create (e.g. the iCalUID must match).
    • When fetching the event list for a calendar, always set the “showDeleted” parameter to true. This way you can detect if you’re trying to recreate an already existing yet deleted event.

Types

  • /events/list accepts timeMin and timeMax arguments and these are simply stated as accepting a ‘datetime’ argument. Of the myriad possible standardized date-time formats, I have discovered that this value should be a UTC date-time (with offset explicitly set at 00:00) formatted as RFC3339 (yyyy-MM-ddThh:mm:ss.sss+00:00).

There are many more issues than I can list here, but these are fresh in my memory.

Revisions

UPDATE (2012-10-14): removed the bit about calendars being unique by Summary as that was not true.

UPDATE (2012-10-19): added Types section to document timeMin and timeMax arguments

Google Calendar API access with 2LO (2-legged OAuth) and .NET

Getting this scenario to work has been one of the most frustrating experiences of my development career as yet.  I’m writing this incredibly detailed and informative blog post to save others from self-inflicted premature baldness.

Your scenario is that you want to write a background task or service in .NET to communicate with Google’s servers specific to your Google Apps domain for business or education. Your service will not have any external exposure to end-users of your apps domain. You want to synchronize data in bulk to Google on behalf of your apps domain users. You want your service to use the latest and greatest APIs that Google advertises that are not deprecated. At the time of this writing that is the V3 API for things like Calendar (http://code.google.com/apis/calendar/v3/getting_started.html). I will use the Calendar API as my example here since that’s what I was first interested in using for my project.

Google seems to want to make this relatively simple scenario unnecessarily difficult to find any information on how to do it correctly. All of the documentation does one of the following: (a) ultimately redirects you to a page talking about a V2 API, not a V3 API, (b) does not talk at all about 2LO and instead obsesses over 3LO, (c) is misleading and woefully incomplete with regards to the practical information you need to know to avoid the dreaded 401 Unauthorized response from the API servers.

Let me assume that you have created a Google Apps domain for business or education already and that you have superadmin access to your Google Apps domain. If you do not, please do so and find the responsible parties in your organization to grant you superadmin access (if only to a development-specific domain).

Required set-up steps: (every single one is critical; do not skip one thinking you know better or you will fail)

  1. Create an API Project at https://code.google.com/apis/console while logged in as your superadmin account for your apps domain (honestly I’m not sure if it matters which user account you create the project with but it doesn’t hurt to be consistent here just in case).
  2. Go to the API Access section.
  3. Create an OAuth 2 client named “whatever you want” at “https:// wherever you want; it doesn’t matter for this scenario”.
    1. Note that you don’t need to specify an icon and that the name of your client doesn’t matter as no other living soul (a.k.a. end-user in your apps domain) will ever see it.
  4. Copy down the generated client ID (somenumberhere.apps.googleusercontent.com) and the client secret (lots-oflettersanddigits-here).
  5. Go to the Services section while still in the APIs Console and enable your specific services (e.g. Calendar).
  6. Open https://www.google.com/a/cpanel/yourdomainhere.com/ManageOauthClients (need superadmin rights to your google domain here)
  7. Add the client ID (somenumberhere.apps.googleusercontent.com) from step #4 and specify “https://www.google.com/calendar/feeds/” for your scope (assuming you want to work with Calendar API) and click Authorize.
    1. For other APIs, list the proper scopes here, comma-delimited. I just need Calendar API and this works for me. Be sure to specify https, not just http.

     

  8. Go to https://www.google.com/a/cpanel/yourdomainhere.com/SetupOAuth
  9. Un-check the “Allow access to all APIs” under “Two-legged OAuth access control”.
    1. Yes, un-check it. This is per http://groups.google.com/group/google-tasks-api/msg/c8dd0ac7c8f320dc. As of 2012-01-25, this is still relevant and required. Perhaps this will change in the future but for now it is required.
  10. Save changes.

Now you should have a properly set-up apps domain and API project and the two are linked together.

Let’s move on now to your .NET code that will be your task/service that runs in the background on behalf of your users in your apps domain.

Firstly, I do not recommend using the open-source Google client library for .NET. I’ve had bad experiences with it, namely that it has been known to leak memory. The issue I reported with them on this matter was claimed to be resolved but I haven’t been back to check it out. I had to make progress on my project and waiting for them to resolve the issue was not an option.

I wrote my own T4 template (Visual Studio code generator) to generate a client library for Google (and other RESTful APIs which use OAuth 1.0 or 2.0) that has no memory leaks and is both ridiculously fast and efficient: https://github.com/JamesDunne/RESTful.tt. It supports both synchronous and asynchronous methods of I/O. Its code is up to date as of 2012-11-12.

Check out the project locally and open the TestClient project’s Program.cs file. This is a simple console application designed to demonstrate the simplicity of the code-generated API client for Google and using OAuth 1.0 for shared-secret authentication.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using WellDunne.REST;
using WellDunne.REST.Google;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;

namespace TestClient
{
    class Program
    {
        static void Main(string[] args)
        {
            // Create the authentication module to use oauth 1.0 2LO:
            // NOTE: Replace 'key' and 'secret' with your testing parameters, else you'll get a 401 (Unauthorized) response.
            var oauth = new OAuth10("myclientidhere.apps.googleusercontent.com", "mysecretkeyhere");

            // Create the client against googleapis.com:
            // NOTE: This client instance is reusable across threads.
            var client = new WellDunne.REST.Google.V3.CalendarServiceAsyncClient(new Uri("https://www.googleapis.com"), oauth);

            // Create the request to get the user's calendar list:
            // NOTE: Each request object is NOT reusable.
            var req = client.GetMyCalendarList(null, null, null, /*requestorID:*/ "username@example.com");

            // Fetch the request synchronously:
            var rsp = req.Fetch();

            // Write the response JSON object to Console.Out:
            using (var conWriter = new JsonTextWriter(Console.Out))
                rsp.Response.WriteTo(conWriter);
            Console.WriteLine();
        }
    }
}

NOTE: This test program is specific to Google Calendar. If you are working with a different API, you’ll have to edit the RESTful/Google/Restful.tt T4 template to declare the API methods you need access to. It couldn’t hurt to define some Newtonsoft.Json-enabled classes to deserialize the response data to.

For Google Calendar API testing, simply paste in the values from your API Console (client ID and client secret) into the `new OAuth10(“myclientidhere.apps.googleusercontent.com”, “mysecretkeyhere“)` expression.

Then paste in an actual user provisioned in your domain into the `client.GetMyCalendarList(null, null, null, /*requestorID:*/ “username@example.com“)` expression.

Run the program and you should see a raw JSON dump of the response retrieved from Google.

For example, I get this output (id and summary are sanitized):

{"kind":"calendar#calendarList","etag":"\"bt6uG7OvVvCre70u9H5QXyrDIXY/5P7Dh-jUGpT56O5EBhfgecrj2pU\"","items":[{"kind":"calendar#calendarListEntry","etag":"\"bt6uG7OvVvCre70u9H5QXyrDIXY/HstY1Kh3cCrbvmn0afdroRd44BQ\"","id":"owner@example.org","summary":"owner@example.org","timeZone":"America/Los_Angeles","colorId":"15","backgroundColor":"#9fc6e7","foregroundColor":"#000000","selected":true,"accessRole":"owner","defaultReminders":[{"method":"email","minutes":10},{"method":"popup","minutes":10}]}]}

query.ashx

I put together a web-based SQL query tool written using ASP.NET’s IHttpHandler interface, packaged as a single query.ashx file. Its main feature is that it guarantees that you cannot construct a query that inserts or updates data. The SELECT query form is forced upon you and any means of escaping that form via SQL injection is detected and the query is rejected in that case. This makes it safe to deploy internally so that your developers may query data in a read-only manner.

Unfortunately I can’t link you to any public deployment of this tool for a demo because it queries SQL databases. I don’t have any SQL databases out in the wild and if I did I certainly wouldn’t want any random jackhole off the internet querying data from them anyway. You’ll just have to deploy it yourself for your own demo. It’s not hard to set it up since all you need is just some form of ASP.NET host that can compile and execute an ashx file. Virtually any standard IIS host will be able to handle this in its default configuration. The tool runs under the security context of the app pool you place it in so if you use connection strings with Integrated Security=SSPI, beware of that.

Features

  • SQL query builder that allows only SELECT queries (Query tab)
    • Try to break it and find a way to UPDATE or INSERT data!
    • Strips out all SQL comments
    • Actively prevents you from overriding the forced separation of query clauses, e.g. you cannot put a FROM into the SELECT clause unless it’s part of a subquery
  • Results tab
    • Queries are forcibly executed in READ UNCOMMITTED transaction isolation level
    • SET ROWCOUNT is set to 1000 by default but can be overridden with rowlimit=# query string parameter.
    • Dynamic show/hide of SQL column type headers
    • Execution time tracked in msec
    • Results grid with left/right text alignment set per column type (numerals are right-aligned, text is left-aligned, etc.)
    • Binary data is displayed as 0xHEXADECIMALSTRINGS
    • Shows generated SQL query
    • Link to share query with someone else (opening link automatically executes the query)
    • Links to JSON and XML output formats with appropriate Content-Type headers. (Try it in Chrome’s Advanced REST Client app!)
  • Custom Connection String support (Connection tab)
    • The drop-down is sourced from web.config’s connectionStrings section
  • Recording of query history (Query Log tab)
    • Stores host name of client that executed query
    • Paged view
    • Click “GO” link to execute query
  • Parameterized queries
    • Parameter values are saved in the query log
    • Probably want to store a library of queries in addition to the query history log
    • Limited set of parameter types supported, but most common ones used should be there
  • Multiple forms of JSON and XML formatted output for programmatic consumption of query results
    • Add query string params to produce less verbose output: no_query=1, no_header=1
  • Single-file deployment
  • Self-update feature that pulls latest code from github
  • Tip jar!

I rather enjoy writing developer tools like this. I especially enjoy the ashx packaging mechanism. For an ashx file, you write all your tool code in one file that contains a primary class that implements the IHttpHandler interface (or IHttpAsyncHandler). You have complete control over the HTTP response in this way at a very low level yet you still get all the convenience of the HttpContext class with its Request and Response structures that ease the pain of dealing with HTTP headers, URLs, query strings, POST form variables, etc. at such a low level.

IVO-CMS – Part 2 – The New

Any good system architecture is based on the concept of layering. A basic premise of layering is that one layer should not concern itself with the details of any other layer. With the proprietary CMS described in Part 1, my failure to realize that fact was the critical design and implementation flaw of the system. The implementation of the system is obsessed with the revision control part. However, with IVO-CMS, I’ve designed the content management system aspect to be ignorant of the revision control system. Think of it as a CMS wrapped in a revision control system. The entire CMS system itself can still function with a non-versionable implementation of the revision control system.

This is possible because the meat of the revision control system is simply a file system that stores blobs. Virtually any system can be designed with such an organization mechanism. The contents of the blobs and their relation to one another are what the design of the CMS is concerned with.

We’re no longer limited by the relational schema of a database. Our data structures are simply serialized to and from blobs stored in the revision controlled file system.

IVO-CMS uses XML as its serialization formation in its blobs. This is a natural choice because of the ability for an HTML5 document to be serialized as XML. HTML is our primary output for a web content management system, so a clean ability to manage and output it must be central to the design of the system.

IVO-CMS does not define any traditional CMS concepts at its core. Things like “page”, “content”, “navigation”, etc. are never mentioned. At its core, IVO-CMS is simply an HTML renderer with an extensible content processing engine.

The most basic concept at play is the blob, for lack of a better term. A blob is the recipe, written in XML, for rendering an HTML document fragment or even a complete document. IVO-CMS’s blob maps directly onto IVO’s blob.

The content rendering engine for IVO-CMS simply starts up a streaming XML reader on a blob and copies the XML elements read directly to the output, with a defined path for handling custom elements.

All custom processing elements of IVO-CMS start with ‘cms-’. The most basic processing elements built-in are:

  • cms-import
  • cms-import-template
    • cms-template
    • cms-template-area
    • cms-area
  • cms-scheduled
  • cms-conditional
  • cms-link

Any XML element that starts with ‘cms-’ is sent to a pluggable provider model to be parsed and handled.

Let’s start with cms-import. When a cms-import element is found in a blob, it should have the form <cms-import path=”/absolute/path/to/another/blob” /> or <cms-import path=”../relative/path/to/../../another/blob” />. Both absolute and relative paths are allowed to describe the location of the blob to import. The imported blob is sent through the content rendering engine and its output is directly injected into the output of the currently rendering blob. The relative path is relative to the currently rendering blob’s absolute path.

An imported blob must be a valid document fragment with one or many root elements that are fully closed. In other words, it cannot contain any unclosed elements which makes its usefulness in rendering partial HTML content limited. This is why cms-import-template was invented.

Think of cms-import-template as importing a template which has areas that can be overridden. This is analogous to the Page/Master Page concept of ASP.NET’s web forms. The page is the currently rendering blob and the master page is the imported template blob. Only certain blobs may be imported as templates – those that contain a single root element: cms-template. Unlike ASP.NET’s web forms, multiple templates may be imported into a single blob and templates may even import each other.

The cms-template blob may contain templateable areas with cms-template-area, uniquely identified with an ‘id’ attribute. The blob importing the template may override these template areas’ contents with a cms-area element and an ‘id’ attribute that matches the template. The order the cms-areas are defined in is important since all XML elements are processed in a streaming fashion and there is no back-filling of content.

Now we come to cms-scheduled. This is an element that allows part of a blob (or the entire thing, if so desired) to be rendered on a scheduled basis. It must first contain some <range from=”date” to=”date” /> elements that define the date ranges when the <content> element should be rendered. An <else> element may also be present to render content for when the current date/time does not fall into any of the date ranges.

Next up is the cms-conditional element which can primarily be used for selectively targeting content to specific audiences. It presents the content author with a simple system of if/else-if/else branching logic to determine which content to render for whichever audience. The inner elements are <if>, <elif>, and <else>. The attributes on the <if> and <elif> elements make up the conditional expressions.

The system evaluates the conditional expressions (a dictionary of key/value pairs pulled directly from the element attributes) to a single true/false value by using a “conditional provider” class. This class may be a custom implementation provided by the site implementer since it is best left up to him/her to define exactly how audiences may be defined and evaluated based on the user that the content should be rendered for.

However, that may be asking too much of the site implementer because it would potentially involve defining a domain-specific language for evaluating expressions. I may provide a default implementation that allows for defining complex boolean logic expressions, e.g. <if expr=”(role = ‘manager’) and (dept = ’23′)”>Hello, managers in dept 23!</if>. The values of variables ‘role’ and ‘dept’ would be provided by a provider model implementation that the site implementer could more easily develop.

Finally, the cms-link element is responsible for allowing the content author to simplistically create anchor links (i.e. the <a> tag) to other blobs without having to worry about the details of how the URL gets mapped to the referenced blob. This is primarily for SEO purposes so that you don’t have to force an implementation of your URL rewriting scheme into your site’s content. The site implementer can write a custom provider that takes the linked-to blob’s absolute path and rewrite it into a URL that should pull up that blob as its own page or as a wrapped article page or whatever other linking scheme he/she wishes to implement. This lets your content be internally consistent without worrying about URL details. Changing your SEO strategy for your content should be as simple as rewriting the link provider.

Now that we have the nitty-gritty details on how the content rendering engine and its basic processing elements work we can talk about how such a low-level engine can be integrated with an existing site, but that’s for next time!

IVO-CMS – Part 1 – Mistakes of the Past

For the first time, I’m starting a series of blog posts. This series will focus on a web content management system (IVO-CMS) that I’m currently designing and developing, partly out of curiosity to see if it can be done, partly out of the fun in implementing something new, and partly out of a need to correct my own mistakes of the past. This first post will explain the past mistakes in designing an existing proprietary CMS for my current employer; one you’ve never heard of, never seen, and never used due to its proprietary nature. I will tell you all this in order to give you context around how the design of the new system was influenced by the failures of the current one.

Let me start out by describing the general system architecture and then demonstrating the weaknesses of such a system. Ironically, this blog post is probably the most accurate documentation of this system’s architecture that currently exists.

Since the system allows each piece of content it tracks to be revision-controlled, I started out by designing a rather simplistic revision control system.

Each versionable object domain is represented in the database by a pair of tables: the “versioned root” table and the version table. What I have termed the “versioned root” is nothing more than a conceptually meaningless container that keeps track of the currently published version record by foreign key reference and is used to tie all the versioned records together by a common ancestor/container. Its basic schema is (VersionedRootID int PK, PublishedVersionID int FK, IsPublished bit).

The version table is what contains all of the versionable data that may change over time and have a history kept track of. Its basic schema is (VersionID int PK, VersionedRootID int FK, ProjectID int FK, all other data columns that are versionable).

For instance, let’s say we have XHTMLContentFragment (versioned root table) paired with XHTMLContentFragmentVersion (version table). The XHTMLContentFragmentVersion contains all of the useful system data and the versioned root table is there to tie all of those version records together and assign an identity to each versionable content fragment object.

A project is a container of versioned items; all versioned records point back to a specific ProjectID. Creating a project must be done first before creating any new versions. Think of it as a “changeset” that’s created ahead of time to associate a set of changes that are going to be applied simultaneously. A project is what gets published, and doing so simply bumps all of the PublishedVersionID FKs on the versioned root records up to the VersionedID that is contained in the to-be-published ProjectID. All of this is done in a transaction so as to be atomic. The idea being that the entire live site gets updated at once and that it all goes live or nothing goes live.

In my eagerness to implement this design, I neglected to account for a few implementation details.

Firstly, with this schema, a versionable item is not able to be deleted from the system in an atomic fashion by doing a publish. I did not implement an IsDeleted bit on the versioned root table to keep track of which objects are still alive. Furthermore, I failed to implement a companion table that records the publish history. There is no way to know what previous VersionID was published before the current PublishedVersionID.

Furthermore, the versioned root concept is flawed in that these containers are meaningless. They do not lend themselves to performing cross-system content merging. In fact, they actively hinder such a useful action. Imagine you have several database environments, like development, testing, staging, and production. Content changes in production would need to be merged down into development so that developers have the latest content to edit for system integration purposes (adding links to new features, etc.) while the production content maintainers can continue to make content changes to the live system.

The reason these containers are meaningless is primarily because of the choice to use auto-incrementing database identity values as primary keys. VersionedRootID 132 has no meaning to anyone except the actual versioned records that point back to it as their container. Its identity value 132 represents nothing semantically useful to the system. If I try to merge content from one system’s VersionedRootID 132 to another system’s VersionedRootID 132, that means nothing. I cannot know that those two containers are the same across the two independent systems.

Finally, the nail in the coffin is that I failed to track the parent VersionID when creating a new version. The system simply copies the content from the currently-published version record and makes a new version record with it then lets the user edit that. The lack of a recorded parent VersionID means that there is no way to tell what was published at the time of object creation. Projects may be created by any user of the system at any time and there may be multiple active projects being worked on simultaneously. Without an accurate history, there’s no way to figure out how to accurately merge this project’s change with another project’s change to the same versionable object. For instance, project B gets created based off published version A and project C gets created based off published version A as well. Let’s say that project B gets published first. This means project C is still based off published version A, and not version B. If project C gets published, it effectively overwrites changes that were made in project B. Without recording the parent VersionIDs, we can have, at best, a baseless merge.

In summary, with this CMS we have a weak history in that all the previous project versions are stored, but there’s no way to know in what order each project was published in. We also don’t know whether content from one project was accidentally overwritten with changes in a second project that was created before that first project was published but was then later published. There is no way to delete versioned items. There is no clean way to merge content across independent systems/environments because an accurate history is not kept and the container structure for the versioned items is meaningless.

All of these problems got me thinking about a better solution. That solution came, somewhat surprisingly, with git… but that’s for next time!

Immutable Versionable Objects (IVO) a.k.a. The Git Object Model

Git is a distributed, content-addressable and versionable file system that is primarily used as a revision control system for project source code. That’s great and all, but we’re not really interested in what git does, but more specifically how it does what it does; we’re interested in its internal object model. I started a simple C#(5) .NET 4 project that implements git’s internal object model as a reusable library.

Git

As a slight change of pace, I will now redirect you to first read Git For Computer Scientists (if you haven’t already) as a basic yet effective crash course on the git object model. This understanding is critical to understanding this blog post and by extension the purpose of the project I created. Seriously, go read it and come back here. I’ll wait.

You’re back now? Good. Do you understand what git’s object model is all about? In summary, it’s composed of a very small set of primitive objects that link together to form a very complicated and powerful revision-controlled file system.

The key concept of git is content-addressable objects, surprisingly. Content-addressable means that a hash is used as a unique key to identify an object based solely on its contents. The hash must be generated by concatenating all content stored in the object together in a predictable and consistent manner, and then running a well-known cryptographically-secure hashing function (such as SHA-1) over that concatenation. “Cryptographically secure” may not be the best name for such a requirement, but a hash function that produces a very good distribution and that maintains entropy of the data in its hash are the desirable qualities.

IVO

The project I have created is named IVO (pronounced /ee’-vho/), short for Immutable Versionable Objects. It is hosted on github here. Sorry, I’m bad at naming projects. I want to call everything either Orion or some combination of the names that make up the project’s uniqueness.

As I stated above, the purpose of this project is to implement git’s internal object model in a completely reusable and implementation-flexible way. The current implementation as I write this is based on persisting the object data in a SQL Server 2008 database (currently using SQL Express as a development environment). The API is defined with interfaces so any back-end provider (including regular filesystem storage like how git does it) can be implemented with ease. In fact, implementations should be cross-compatible with one another so you can dump a SQL database containing a repository into a set of flat files or vice versa.

Asynchrony and Immutability

Consistent with my latest kick for asynchronification, I’ve designed the IVO API do be entirely Task-based using the TPL (task parallel library; `System.Threading.Tasks`) part of the .NET 4 framework. The SQL Server 2008 back-end implementation executes all its database queries asynchronously using SqlConnection’s `BeginExecuteReader` / `EndExecuteReader` standard async pattern.

Hopefully, the Task-based API is easily degradable to synchronous execution where that is desired, but why would you not want it? :) I’m fairly certain it’s easier to downgrade asynchronous to synchronous than the inverse.

Consistent with the asynchronous nature of the API, and regardless of object model logical immutability, all CLR objects that represent part of the object model are immutable data structures in order to be asynchrony-friendly. Nested Builder objects are used to construct a mutable object as a work-in-progress (e.g. when being constructed from a persistence store) and is then converted to its parent immutable object type.

As for object model logical immutability, data is only ever added to the system in the form of new records (and records can be removed when proven orphaned). Data is never updated in-place except where that object model is considered logically mutable, e.g. a `ref`. As a general rule, all content-addressable objects are immutable and are never updated after their original creation.

Solutions

Once you fully understand the power of the object model implemented, you can solve all sorts of interesting problems that used to be considered hard, like implementing a content-management system, a remote filesystem synchronization system, a source control system, a document-tracking system, a historical record tracking system, and yes, even a revision controlled native file system driver (although probably not easily achievable via .NET)! The possibilities are nearly endless. What’s apparent to me is that this object model lends itself well to the design of systems that have to deal with revision control or synchronization, or a combination of the two. I’m sure there are many other problems that this can help solve that I’m not aware of.

You don’t even have to expose the gritty internal details of commits, trees, blobs and all that jazz to the end user. You can implement it transparently on their behalf. Then when it comes time to handle that complicated merge operation, you can present the user with “Oh look! I kept track of all your revisions! How would you like me to merge your changes for you?” It’s my opinion that the more a (savvy) user knows how a system is implemented the more he/she can be useful and take advantage of the system.

Distributed Workflow

A distributed workflow model is a virtually “free” benefit of this object model as well. You can have multiple users of your system working simultaneously and independently of one other. The only difficulty is in the implementation of a semantic diff/merge utility specific to the kind of data you’re working with. You will quickly find that you need to provide users the ability to merge their “branches” together to produce one common main-line branch that represents the accepted state of the system. To do that, you’ll need that diff/merge utility to handle combining independent changes. The object model can easily track your merge commit with multiple parent commitids, but how you handle merge the actual content is entirely up to you.

Summary

In summary, IVO is a framework for solving a larger problem that wants to handle revision control or synchronization in a powerful and flexible manner. I, myself, am writing a web content management system based off of it named, quite naturally, IVO-CMS. That will be the subject of future blog posts. I felt I should blog about IVO first as a platform for introducing IVO-CMS. I’m implementing and improving the two in tandem so IVO will get more API usability benefits as a result of implementing IVO-CMS along the way.

Feather: A new language based on asynchrony

I’ve spent most of my free time over the last few weeks in pursuit of designing a new programming language, one designed for asynchrony from the ground up. I call this language “Feather,” in the hope that it will be lightweight, simple, elegant, and just might possibly enable one to fly.

My core goals for this new language are:

  • asynchronous execution by default with explicit mechanisms to revert to synchronous execution.
  • immutable data and no ability to share mutable state between independent threads of execution.
  • static typing and complete type-safety.

Keeping this goal list very short will allow me to actually achieve all of these goals relatively easily with a final, working reference implementation of a compiler and runtime system.

Asynchronous execution:
This is the primary and most important goal of the language. Where possible, functions must be allowed to execute asynchronously. That is, the completion of one function need not depend on the completion of another function. This assumes execution independence of functions. Of course, this will not always be possible in every case, since one function may rely on the computed results of another function in order to complete, creating a dependency. There may also be times where execution of certain functions need to take place in a specific sequence, but in general I believe these to be special cases.

No shared, mutable state:
The main problem with allowing just any random set of functions to execute asynchronously with respect to one other has to do with shared mutable state. For instance, if just two functions can modify the same shared state and both are executing asynchronously without any synchronization between the two, the final effects on the shared state are undefined. The simplest solution to this problem is to disallow the sharing of mutable state between functions.

Combining these two goals as core to the language has never been done before in any programming language I’ve ever used. Sure, other languages support asynchrony and even fewer may support immutable data, but not to the extent of putting them at the core of the language’s design. The concept of defining data as immutable is not the same as forcing that mutable data cannot be shared across threads of execution.

I hope that defining these goals up front justifies the language’s existence enough for me to further purse its design and development. For once, though, I think I’ll end this post before it gets too lengthy. I have much more to talk about in regards to this language and its features than one post can contain. So, until next time!

RabbitMQ .NET client API is terrible (rant)

I’ve seen some pretty poorly-designed APIs in my time, but RabbitMQ’s .NET client API has got to be among the top worst that I’ve seen lately. Specifically, I’m talking about the RabbitMQ.Client library that appears to be officially sanctioned by the RabbitMQ project and “documented”.

The very first problem you’ll quickly see when you get into using this “API” is that everything (e.g. return values and parameters) is typed as a string. Strings are problem enough in and of themselves, so why exacerbate the problem by requiring them everywhere?

Let’s dive in to this API and look at a few examples in the IModel interface:

void ExchangeDeclare(string exchange, string type, bool durable, bool autoDelete, IDictionary arguments);

See that ‘type’ parameter? What does that mean? What is an exchange ‘type’ in this context? Can I create my own types? It turns out, no. This string value is expected to be one of (“direct”, “fanout”, “topic”). How do we know this from the method signature? We don’t! Not a clue. No, we have to pore over the user guide “documentation” to find that we’re expected to use one of the ExchangeType class’s const string members here.

Here would be a perfect candidate for an enum. This is essentially an untyped string “enum”, primarily used when calling ExchangeDeclare function. I haven’t done a full search of the API, but I’d want to say that this is probably the only place these values are used. Not to mention this ExchangeType “static” class isn’t even declared as static. Sure, you can new up an ExchangeType! What would that get you? Nothing. Not only did they do it wrong, they did it wrong the wrong way.

Continuing on, the ExchangeDeclare method returns `void`, like most methods in this API. This is also terrible in that it gives you absolutely no information back as to the success/failure status of the operation; that would be the bare minimum expectation for a reasonably dull API. What would be preferred is a more complex type system that propagates contextual information through its methods. Now there are some good situations to use void-returning methods in an API, but given this context, I don’t think this is one of them.

Why is the void-returning such a big deal? It gives you no information as to what operations should logically follow next after “declaring” an “exchange”. The API designer has expectations on how you, the user, should use this API and has documented elsewhere that the expected pattern of invocation is ExchangeDeclare, QueueDeclare, and QueueBind, but you would never be able to discover this yourself without significant trial-and-error at runtime.

The API itself should expose strong hints as to how it is designed to be used, not the documentation. Documentation should be used to clarify what and/or how a specific piece of functionality works/does. The API should scream at you (via its types) how it is to be used correctly. Documentation should scream at you how it should NEVER be used incorrectly.

What would have been nice is if this ExchangeDeclare method gave back some semblance of an Exchange type that has its own methods, maybe something like `QueueBinding BindQueue(Queue queue, …)`. That also implies there is a Queue type returned from QueueDeclare. All of this typing information should let you logically come to the conclusion that, “Oh, I need both a Queue and an Exchange declared first, and then the Exchange lets me bind queues to it!”

Instead, what we’re given is three seemingly independent methods defined on `IModel` (which is a terrible name for something that’s supposed to mean a channel, BTW) with void-returning semantics that accept only string-typed parameters and the order of operations is left undefined for you to guess at.

There is also absolutely no XML parameter documentation in this API either, which is more-or-less a de-facto standard in the C# world. Now, there are method <summary>s in the XML documentation, but the parameter documentation is entirely missing. Intellisense would have significantly helped out the discoverability of this API has there been any XML documentation on the parameters. Going back to ExchangeDeclare, at least a NOTE saying that this string ‘type’ parameter expects its values to come from the ExchangeType class would have been more useful than saying nothing whatsoever.

string QueueDeclare(string queue, bool durable, bool exclusive, bool autoDelete, IDictionary arguments);

Why would you use the pre-generics IDictionary type with absolutely no explanation as to what types it expects to see in the dictionary, what it is used for, or even if I can safely pass in a null reference if I don’t care right now about extra ‘arguments’? There are MUCH better strongly-typed ways of conveying optional information to an API programmatically than using an IDictionary. This should simply not be done, no matter if it be with the pre-generics framework types or with the generic ones (i.e. IDictionary<TKey, TValue>).

What does this method return? A string? Seriously? What do I do with that? What information is in this string? What other methods are looking for this particular string value? This return value is in its own domain, and I would hope obviously not to be confused with any other `string`-typed parameters in this terrible API. I should NOT have to run the program (in which I don’t even know if I’ve done anything right so far) in order to discover what different domains of string values there are and where each one is expected. This is the entire point of a typing system, whether it be dynamic or static. At least with a dynamic typing system there is some structure around your data. Here we just have strings, the lowest of the low.

void QueueBind(string queue, string exchange, string routingKey);

Now we have a new ‘routingKey’ string parameter. What exactly is the proper formatting of this key? Should I care? Are there any special characters reserved? Why not wrap this up in a RoutingKey struct, at the very least, that’s responsible for constructing only-valid routing keys? Nope, we’re completely on our own to create valid routing key values that should hopefully conform to the established AMQP standard’s definition of a routing key.

string QueueDeclare(string queue, bool durable, bool exclusive, bool autoDelete, IDictionary arguments);

We can declare queues! Great. Any restrictions on the queue name? Is that validated by this API? Again with the IDictionary crap? At least we have some bools now, and they’re not `string`s whose valid set of values would be (“true”, “TRUE”, “false”, “FALSE”) but not “True” or “False”. That last part was a joke, but I certainly wouldn’t be surprised if that were the convention here too. Why not be consistent and just STRING everything? That was also a joke. I hope you’re still with me here. Still, boolean parameters are generally frowned upon.

string QueueDeclarePassive(string queue);

Okay, now apparently there are ‘passive’ queues. I guess that passive queues cannot be defined as durable, exclusive, or autoDelete. What, no extra IDictionary of arguments this time? Does that mean I declare ‘active’ queues with QueueDeclare? What’s the difference?

All these strings everywhere bring me to another point: case-sensitivity. Is “MYQUEUE” different from “myQueue”? Are routing keys case-sensitive? Also, what if I want to use a non-ASCII character in my queue name, like “ಠ_ಠ”? Will that wreak havoc with this system? It seems this is the only face I’m able to make towards this API’s design. Hmm… what if I have some extra whitespace in my string? Are all string values Trim()ed, or just some, or none?

Summary:
I generally don’t like writing public rants, but it seems no one else has had anything negative to say about this API yet. I feel dumber having used this API. I award the designer no points, and may God have mercy on his/her soul. This makes me not want to use RabbitMQ at all.

RunQuery commandline tool

As a useful companion to LinqFilter (which I have given considerable attention to over the last few weeks), I put together a tool called RunQuery, which, you guessed it, runs SQL queries from the console… BUT (and here’s the kicker) it formats the results in an escaped, TAB-delimited format for direct use by LinqFilter.

For tool interoperability, I created (read: quite possibly reinvented) a useful, near-universal data interchange file format which uses TAB characters as column delimiters but also backslash escapes characters in each column to eliminate the possibility of actual TAB chars in the data interfering with the format. The backslash escaping rules are a subset of C#’s own string escaping rules. Also, null-valued columns are rendered as NUL (\0) chars in the format.

This data interchange format is accessible from LinqFilter via the SplitTabDelimited and JoinTabDelimited static methods. These methods are cloned in the RunQuery tool to ensure parity between the two tools. All of this makes it so we can easily exchange tabular data between tools using simple text pipes and/or files without having to worry about escaping, encoding, etc.

When you use RunQuery with default settings, it outputs two rows of header information. First, the column names, and second, the column types. All subsequent rows are column values. No row count is currently reported. There is no limit to number of rows returned either.

Aside from simply executing static SQL queries, RunQuery has a special “batch” mode which can execute a parameterized query whose parameter values are supplied via the stdin pipe. This mode expects the stdin lines to be fed in a particular order and formatting, namely the TAB-delimited encoding described above.

In this batch mode, the first line of input must be a row of parameter names where each column is a parameter name and must begin with a ‘@’ character. The next line of input is a row of columns that represent parameter types. Acceptable values are the enum members of SqlDbType.

After these first two rows of metadata are consumed and validated, all subsequent rows are treated as columns of parameter values. For each row, the query is executed against the given parameter values until EOF is reached.

Example:

$ cat > params
@n
int
1
2
3
^D

$ RunQuery -E -S myserver -d mydatabase -b -q ‘select @n’ < params
—————————————————————————
select @n
—————————————————————————

int
1
2
3

For example, batch mode may be used for streaming query results from one server to another for something simple and naive like a poor man’s data replication system or cross-database queries whose servers are not linked. One can also stream these parameter inputs from a LinqFilter query which may be processing input from a file or perhaps even consuming data from a web service. The possibilities are endless here.

Running RunQuery with no arguments displays some brief usage text.

LinqFilter Code Injection

Unless you’ve been living under a rock, you may have heard of such things as SQL Injection and XSS (cross-site scripting) attacks, just to name the popular forms of the attack. These types of attacks are language specific forms of the general “Code Injection” attack. These types of attacks happen when developers concatenate source code with unchecked user input and get to the stage where that fully trusted source code is eventually executed.

SQL Injection happens when a developer concatenates user input into a SQL Query but the user input may contain special characters like single-quotes which changes the meaning of the query. You could do this in a safe way by either properly escaping the raw user input or passing it in to the SQL Server out-of-band in a parameterized query. But I’m not here to blather on about SQL Injection attacks.

Today I’d like to talk about C# Code Injection, specifically as it applies to LinqFilter.

In LinqFilter, I naively concatenate what should be untrusted user input LINQ code directly into an implicity trusted C# code template embedded in the program. I invoke the C# compiler provider on that resulting string and execute the code in the compiled assembly. By concatenating unchecked user input into an existing code template, I’ve opened the door to C# Code Injection. The user has numerous ways of defeating the requirement that the query be a valid C# 3.5 expression.

One can easily terminate the expression with a ‘;’ and go on to write regular C# statements. LinqFilter will naively concatenate your code that it assumes is an expression into the code template and compile it. The compiler doesn’t care what happened so long as the code compiles as a valid C# program. This defeats the purpose (somewhat) of the LinqFilter tool so it’s generally not wise to do so and you’d only be hurting yourself, unless of course you’re executing someone else’s malicious LinqFilter scripts.

What I’d really like to do is put in a safety check to validate that the user input is indeed a C# 3.5 expression. This involves getting a parser for the C# 3.5 language, not an easy task. All I need to do is simply validate that the input parses correctly as an expression and only an expression. I don’t need to do any type checking or semantic analysis work; the compiler will handle that. I just need to know that the user can’t “escape out” of the expression boundaries.

Unfortunately, there are no prepackaged C# parsers that I could find. This means using one of the plethora of available parser-generator toolkits or writing my own. Neither options are sounding very attractive to me at the moment so I’ll probably just hold off entirely on the work since it provides no benefit to myself. I know how to use LinqFilter effectively and there’s clearly no community around it (yet) that are sharing scripts so there’s really no security problem. If someone else is concerned enough about the problem, I gladly welcome patches :).

I think I’d also like to figure out how the CAS (code access security) feature works and incorporate that into the generated code so that queries are executed in a safe context and won’t be going out and destroying your files by invoking File.Delete() or something sinister, but this is a low priority as well.