Why?

Obvious question, really. There are lots of web frameworks out there for .NET already, which all offer different things, so why build a new one? The primary answer is: none of them worked quite the way I wanted. There were things I liked about some of them, but not all in the same package. I loved WCF Web API while it was still called that, when Glenn Block was working on it, but by the time it became ASP.NET Web API it wasn’t the same thing any more. It’s had a Controller base class foisted upon it, for a start, and the routing was all completely different, and it broke my (current, start-up, commercial) project, which soured the relationship. I needed something else for my project, and nothing I knew of did the things I really needed right away.

The other thing was that I read a couple of blog posts around the theme that the MVC pattern breaks the rules of SOLID, most notably the Single Responsibility Principle. In a traditional MVC application, a Controller represents a resource type, and has Actions for all the possible operations against that type of resource: Create, Read, Update, Delete and so on. That’s multiple responsibilities right there. And this impacts on performance as well as maintainability, because you have to inject the dependencies for all the possible actions in the constructor, even though none of them may be needed. Search for “single action controller” and you’ll find dozens of posts warming to this theme and offering that workaround, but it got me thinking, what if all the controllers were single action? What if, instead of using a method to represent an action (or a URI), you used a class? How would a framework like that look?

Design goals

I decided to try building a framework like this in tandem with the commercial project, primarily focusing on getting it to do the things I needed it to do for that. My primary goals for the initial version were:

• Keep it simple;
• It should be ridiculously easy to do TDD and/or BDD;
• Support asynchronous, non-blocking handling of requests;
• Make it easy to build a proper RESTful, hypermedia-driven application and services, including supporting content-type negotiation for all requests, including HTML;
• Be really open and extensible, because if people can write plug-ins and add-ons easily, I don’t have to build all that stuff in.

To that end, the framework has been built to be extensible, but currently provides a limited set of options. The only hosting option at the moment is on top of ASP.NET. The only view engine supported is Razor. It uses JsonFx for handling JSON, and a very rough system built on the WCF DataContractSerializer for XML. And the only IoC container I’ve built support for is Ninject. These are all personal preferences, and while I’ve tried to make sure they’re all abstracted in a way that will make it very easy for additional plug-ins to be built, I fully expect to have to make at least a few core changes if and when people want to build them. In particular, the work I did on refactoring out the hosting code into a separate project for ASP.NET probably took too much code with it.

Keeping it simple

I’ve tried to make everything obvious, and cut out boilerplate as much as possible. In particular, there are no base classes for handlers, although you are free to create your own if you want to. Everything is done through interfaces. Lots and lots of interfaces, each one very small, in accordance with the Interface Segregation Principle. Also, everything gets put together automatically, with no complex routing tables and brittle dependencies.

Facilitating TDD

The biggest barrier to automated testing of web apps is the need to fake, mock or stub the accoutrements of HTTP in order to fool the object under test into thinking that it’s running in a web server. In Simple.Web, handler objects know nothing about HTTP. They implement interfaces to tell the framework what information they need from the request, and what information they provide for the response. The framework takes care of mapping the various properties and methods to HTTP. This means a handler can be created and have those properties set in very simple “arrange” code, and you never need to mock another Request object in order to set a header or a cookie or a request body. Handlers can stick to implementing application and business logic, and you can stick to testing that logic, which is how it should be.

Building the handlers in this way has had other positive effects on the system and the code. It’s made for some very small, modular types; it’s made the HTTP<->handler mapping into an extension point which developers can hook into really easily; and it’s meant that the runtime system can be highly optimised, since all the information about what a handler needs and does is right there in its type declaration.

Supporting asynchrony

This is as important on a web server as it is on a client machine. In general, web servers spend most of their time waiting on I/O operations, pointlessly blocking on threads while a database or another web service does its thing. If this time spent waiting can instead be spent processing other requests, we can service more clients with fewer servers. This is, of course, the principle behind Node.js, but it turns out it works just as well in .NET, thanks to the Task Parallel Library and Windows’ kernel-level evented I/O. ASP.NET MVC 4 allows you to return a Task<ActionResult>, and when combined with the awesome new async/await features in C# 5, it’s really easy to write non-blocking web code.

In Simple.Web, the primary method on each handler is taken from the HTTP method represented by the handler interface. For example, a GET operation is handled by a type which implements the IGet interface, and that declares a Get method which returns a Status. It’s all very getty. Like this:

[UriTemplate("/")]
public class GetIndex : IGet
{
    public Status Get()
    {
        return Status.OK;
    }
}

Don’t worry about where the output comes from, we’ll come to that. The important thing at this stage is that for every HTTP method interface, there is an asynchronous variant, where the relevant method returns a Task<Status> instead, and that would look like this:

[UriTemplate("/")]
public class GetIndex : IGetAsync
{
    public Task<Status> Get()
    {
        return DoLongRunningThing()
            .ContinueWith(t => Status.OK);
    }
}

Except of course you’d have some error checking and stuff in there. That’s where you want to be using C# 5 and async/await, otherwise it’ll bake your brain.

The point is, the decision to make something asynchronous is not a big one, and you’re free to mix and match synchronous and asynchronous handlers in a project; it’s up to you.

REST and hypermedia

The most important feature for implementing good, flexible, robust web apps and APIs is proper content-type handling. This is one of the biggest problems with ASP.NET MVC: actions do not return resources; they return specific representations of resources. The action decides whether to render a view or some XML or some JSON, when in fact it should be the client telling the action what representation it’s after. To do this, you need a content-type negotiation system, so Simple.Web has one, although it’s in a fairly embryonic state at this point.

A Simple.Web handler outputs an object, or a collection of objects, and lets the content-type negotiation system work out what to do with it. If the client is a browser asking for HTML, then the view-engine system kicks in, and tries to find a view. It can do this using the @model directive if only one view in the application declares that model type:

@model Sandbox.Form
<!DOCTYPE html>
<html>
    <body>
        <div>
            <h1>@Model.Text</h1>
        </div>
    </body>
</html>

or, if that is not specific enough, or if there is no model type for a given handler, there is a new @handler directive to support directly linking between view and handler:

@handler Sandbox.GetForm
<!DOCTYPE html>
<html>
<body>
<h1>@Handler.Title</h1>
<form action="/submit" method="POST"><input type="text" name="Text" /><input type="submit" /></form>
</body>
</html>

This also results in the handler being a property in the view, with all its public variables available; no more need for ViewBags.

The other aspect to HTTP and hypermedia-based apps and APIs is that all resources should come with a complete, human- and machine-readable set of the available state transitions for the resource, represented as links. There is a system for building these links within Simple.Web, and I’ll be covering that in an upcoming post.

Be open and extensible

Over the last year and a half of maintaining Simple.Data, I’ve been really impressed by the can-do attitude of people who have built and maintained their own adapters and providers, and submitted code back to the core project when they needed something. There’s an astonishing number of databases, both SQL and NoSQL supported, and mostly that support is nothing to do with me.

I want to let the same kind of people do that with Simple.Web too, whether it’s for view engines, or hosting systems, or content-type handlers. And with a web framework, there’s an additional driver for being open and extensible, because there are plenty of points within a specific project where you might want to change the way something works.

So the framework comes out of the box with some plug-ins, which are included in the Simple.Web package for the time being for convenience, but I’m hoping that fans of other hosts/view-engines/serializers will pop up with their own alternatives, at which point I’ll separate out the JsonFx and the Razor, for instance.

But Simple.Web is extensible in smaller ways, too. For example, there are interfaces for GET, PUT, POST, PATCH, etc., but I haven’t created an interface for MERGE, which some systems use. What I have done is make it easy for you to add your own HTTP method interfaces, which will get picked up by the framework and integrated just like the built in ones. So you can add MERGE, or TRACE, or make up your own methods, like SMITE and BESTOW.

And then there’s all those interfaces which map information between the HTTP infrastructure and your handlers. Those are all built on a very simple pattern, using public types and attributes, and you can create your own interfaces which will get integrated into the pipeline in exactly the same way that the built-in ones do. If you have a custom header called X-User-Mood, you can create a “behavior” interface and some “implementation” code, and it will get hooked in for you, so you never need to pollute your handler code with HTTP contexts and requests and responses.

Getting started

OK, that’s enough waffle for now. If it sounds interesting, then the Skills Matter video has a lot more of me blathering on about it and doing some demos, many of which fail spectacularly. You can install the package from NuGet, or download the source from Github, where there is also a demo application which demonstrates some of the main features. There’s a Google Group for people who want to use the framework to build applications, and another one for people who want to build plug-ins or add-ons or contribute to the core in some way (hint: Razor needs some love).

Also, because this one is all statically-typed, with no dynamic oddness, IntellSense is your friend, and there are documentation comments all over the place so it should be reasonably easy to pick up. I’m looking into something that generates actual help from the comments and isn’t Sandcastle, so hopefully there’ll be a manual soon.

Finally, I’ll be putting up some more detailed posts about various features over the next few days and weeks, so watch this space.

The first little pig was lazy and frivolous, so he built his house out of straw. When the Big Bad Wolf arrived, he huffed and he puffed and he blew the house down, and ate the little pig, because this is the Grimm version.

The second little pig was very serious, so he decided to engage a large consultancy firm to handle the building of his house for him. The consultancy firm assigned four of their best Big Fat Pigs to the project, and they scheduled a very expensive meeting with the little pig to discuss requirements. The little pig told them he wanted a nice, comfortable house with modern conveniences which would keep the Big Bad Wolf at bay. The Big Fat Pigs told the little pig that he should plan ahead more, and recommended that he build a large skyscraper in case he ever decided to run a multi-national corporation from the house. The little pig worried that this would be expensive, but the big fat pigs assured him that costs would be kept down by using cost-effective materials and off-shoring the actual building of the skyscraper. The little pig didn’t really understand much about building houses, and the big fat pigs seemed to be very clever and had lots of experience, so he signed a very long contract and waited for his skyscraper. Two months later the Big Fat Pigs came back to tell the little pig that they had had a pre-pre-meeting meeting meeting, and had clearly defined an early draft of the project plan. In only eight months they would have the final draft of The Plan, and would be ready to select an architect, who would then submit three different designs for the skyscraper along with thousands of pages of documentation detailing the wolf-proofing measures, so the little pig could be absolutely sure he was going to get the very bestest skyscraper that had ever been built. The Big Fat Pigs had a Gantt chart that covered three years and 142 participants and stakeholders, all with clearly defined activities and responsibilities so they knew exactly where they would be and what they would be doing at any time over the three year project. The Gantt chart was printed in colour on A0 paper and everything.

Sadly, the little pig never got to see the beautiful Gantt chart, because he had been eaten by the Big Bad Wolf. Luckily for the Big Fat Pigs, a clause in the contract he had signed meant that the project could continue in spite of his death, so they carried on. Nine years later, as the final prefabricated part of the skyscraper was fitted for the seventh time (because the previous six parts had not fitted correctly when they arrived from the land far away), the entire skyscraper collapsed because, it turned out, it had been built from substandard parts made from cheap materials in several different countries by unqualified workers who were working from different versions of the design document, which was two thousand pages long. Oh, and somebody had forgotten to dig the foundations, but by the time this was noticed, it was too late to do anything about it, so a complicated system of tensile steel cables and tent pegs was added to the base of the skyscraper to stop it falling over.

(The Big Fat Pigs were very surprised when the tent pegs didn’t work, despite the fact that they had used the same idea on every project they’d ever done and it had never worked. They thought maybe it would work next time, if they incorporated the learnings from this project.)

The local council insisted that the little pig’s family clean up the mess that the collapsing skyscraper had created, and make good the damage to neighbouring properties. The Big Fat Pigs put in a tender for the clean-up project, and got it. That was four years ago. The rusting remains of the skyscraper are still where they landed; the Big Fat Pigs have succeeded in designing a new system for reporting on the progress of the clean-up operation, and are currently waiting for executive approval so that they can move on to the pseudo code stage.

The third little pig hired a really smart architect, who knew a very good builder. The little pig told them he needed a house to protect him from the wolf, so the architect drew up a simple, strong design and the builder built it. After he had moved in, the little pig wanted some improvements, so he went back to the architect and the builder to ask for them. Because the house was very simple, and the builder had built it properly, they were able to make the new improvements quickly and easily, and the third little pig lived happily ever after.

THE END

Another post giving the 501 Developer Manifesto more attention than it deserves. I honestly wouldn’t waste my time reading it if I were you.

…

Why are you still here? If you’ve seriously got nothing better to do than read this, I pity you. I really, really pity you. Because you’re pitiable.

“Pity” is a derogatory word

Let’s get this one out of the way for a start. There’s a reason why “I don’t want your pity” is one of the hallmark clichés of badly written dialogue. If there is probably some pity in your attitude toward someone, it implies that you feel lucky not to be that person; that their situation is deficient in some way that yours is not. If you are making specific reference to an aspect of that someone’s life that they consider important and special and even defining, then you shouldn’t be surprised when they are insulted, and paraphrase Fight Club in Hacker News comments.

A clarification of terms

I shouldn’t really need to write this, as this post on Scott Hanselman’s blog (the first link on the 501 Manifesto page) explains very clearly what is intended by the epithet “501 developer”. It has nothing to do with what time you leave work, or how hard you work while you are there. Being a 501 developer means that you have no great enthusiasm for your work; no desire to learn new and interesting things; no ambition within your chosen career. It means you’re probably just grinding the levels until you get to a better-paid management role, which you will then carry out very badly because you don’t really understand thing one about software development.

If you:

• Never read technical blogs
• Are not aware of open source projects (unless they’re forced on you at work)
• Have never attended any kind of developer-focused event
• Don’t own any books about coding or productivity
• Aren’t quite sure why Github is called Github
• Couldn’t* care less about trying to be better at your job

…then you are a 501 developer, and no, we don’t respect you for it. If you do any of those things, you’re not a 501 developer, regardless of what time you leave work, and I respect you as a professional and as a human being.

*Yes, Americans, the term is “couldn’t care less”. “I could care less” means “I care”.

It is perfectly possible to arrive at work at 9am, leave work at 5 or 5:30pm, and yet still read blogs and discuss software development on the internet and follow internet-famous software people Twitter and go to your local user group when there’s a speaker talking about something that interests you and read Clean Code and maybe do a little bit of extra-curricular coding with something new, and still spend plenty of time with your family and your Moog synthesiser and collection of blank cassette tapes.

That’s the real issue here, is that by taking the time to register a domain and create a web page and attach a blog to it, and by linking to Computer Zen and Coding Horror and making apparent his knowledge of the Agile manifesto, the author of the 501 manifesto has identified himself as anything but a 501 developer. Maybe the problem is that he’s lucky enough to have never worked with an actual 501 developer, in a massive room full of 501 developers for a 501 company in a 501 vertical market, so he doesn’t understand just how bad it can be.

Because you don’t have to be obsessive about it. You don’t have to self-identify as a ninja or a rock-star or a temporally-displaced Olympian demi-god (no? Just me then*). You don’t have to write the blogs, or speak at the user groups, or write the books, or make the next big thing to avoid being classified 501 (but the people who do those things would prefer it if you didn’t make fun of them for it). You just need to care, just a little bit, that’s all.

*Irony, dipshit.

One last thing

If you do have an obsession, and it’s not your job, what are you doing to rectify that situation? Because if you have one all-consuming passion and it’s not what you do all day, every day, and the thing that you do do all day every day keeps you from that passion, that sucks.

It sucks to be in that situation. And you can write a manifesto and you can think you’re “taking back the word” and you can convince yourself the people you work with who love what they’re doing and can’t believe somebody’s paying them to do it are pitiable, but at the end of the day you’re the one being kept from doing the thing you really love by having to do something you merely tolerate because you have two kids and an oppressive mortgage.

#501manifesto? Here’s a better idea: Find a job you love and you’ll never work a day in your life.

— Mark Rendle (@markrendle) April 17, 2012

As I browse around fantasy-laptop-shopping, I see lots of laptops which have features that I really, really want, but I’ve never found one laptop that has everything. So I’m going to outline the spec I want, comprised entirely of things that are either available in current laptops or, at least, announced and imminent parts.

Currently I have a late-2009 15” MacBook Pro, which has a 2.53Ghz Core 2 Duo CPU, and has been upgraded with 8GB of RAM and a 512GB Crucial m4 SSD. I also have access to a 17” HP Envy, on loan from my last employer while I am doing some freelance work for them.

I do most of my own development, including open source stuff and some speculative projects, on the MacBook, using Parallels 7 to virtualise Windows 7, because when I had it running under Snow Leopard’s Boot Camp it kept blue-screening at inopportune moments. Like five minutes from the end of my presentation at DDD SouthWest 2011.

The MacBook also gets used for most of my speaking engagements, although I have been known to use the Envy when I’m talking about Azure because I’m not sure about running the Azure Emulator inside an emulator.

The Spec

Form factor

I’m happy with the size and weight of the MacBook Pro, although obviously thinner and lighter is always welcome. Remains to be seen whether those Air-style MacBook Pro rumours are true or not. The other good thing about Macs is their small, light power supplies. The brick that comes with the Envy seems to weigh almost as much as the laptop itself, and it insists on you dragging the three-pronged not-quite-a-kettle lead around too. It all adds to the mass and bulk, especially on long trips to overseas conferences. Just make it as thin and light as you can; whatever it adds to the cost I’ll easily make up in savings on chiropractic bills.

CPU

I’m running Visual Studio 2010 and 2011, plus SQL Server 2008 R2 and 2012 and IIS 7.5 or IIS 8 running as services, and emulators for Azure and Windows Phone and VMs for Linux and suchlike, and a whole bunch of other cool beta stuff because I’m an incurable neophile, so I want a beefy CPU. I’m thinking Intel Core i7-3820QM, which is supposed to be available next month and is based on the new 22nm Ivy Bridge architecture. Shouldn’t be too much of a problem to get that.

Hard disk

Needs to be solid-state, obviously. In pure fantasy-land, I’d have a pair of 512GB SSDs set up in RAID/0, but I’d settle for one 512GB SSD (which I already own) and a 750GB Seagate Momentus XT secondary drive. So two drive bays, obviously.

Before we move on, let me just suggest one extra thing: can I get an easily-accessible HDD/SDD caddy system for the primary drive (maybe a bit like a PCMCIA), so I can switch between operating systems by swapping drives? That would be awesome.

Memory

The Lenovo ThinkPad W520 is the existing example here: you can spec that up with 32GB of RAM. That should do it. But can I please have that in a laptop that weighs less than my head? And a tip: if adding 24GB of RAM to your default 8GB spec in your “Configure your laptop” page costs more than buying 32GB of compatible RAM from Crucial, guess which I’m going to do? Yeah.

GPU

I don’t play games on PC, because I have an Xbox 360 and a PS3 and a Wii and a DS and a PS Vita so I’m really OK for that stuff. But there are other uses for GPUs these days: I want to play with C++ AMP, and make backups of my Blu-ray and DVD discs, which takes much less time when you’ve got a few CUDA cores kicking around. So I’d like an nVidia GPU, preferably one of the 500M series, although I’d accept an NVS 4200M (which appears to be the new Quadro). Obviously the GPU should be switchable, so it can fall back to the integrated graphics in the CPU when running unplugged.

Update: Peter Bright has pointed out that Quick Sync is better than GPU for video encoding/transcoding, which is true. I still want a GPU for playing with C++ AMP, though.

Screen

One of these, please. 15”, and 16:10 (or 8:5) format. 16:9 is all very well for watching movies, but when you’re working on text, that extra height is really nice to have. Oh, and make it multi-touch with Gorilla Glass 2 while you’re at it. Ta.

Keyboard

A nice, backlit, chiclet keyboard, obviously. With a physical Insert key, because I use ReSharper and there’s a ton of neat things hiding under the Alt-Ins keyboard shortcut. Oh, and if you’re Apple, put bloody # on a key already. Seriously.

If you want to do the Razer thing and bung some programmable OLED keys on there, that would be awesome, but stick them on the top row, not to one side where I’ll keep hitting them instead of Esc or Ctrl or Backspace.

Track-pad

A decent track-pad, which goes *click* when you click it, is essential, and it should support gestures like two-finger scrolling and… actually, just two-finger scrolling would be fine. Yes, I know the screen is touch, but it’s all the way over there; I can’t be reaching for that all the time. Again, Apple is the exemplar here.

Connectivity

A built-in VGA output would be useful for attaching to all those 1024×768 projectors, but stick a Mini DisplayPort on there too, like HP have done on my Envy.

A decent set of USB 3.0 ports (at least two), maybe with a couple of 2.0 for keyboard and mouse.

Really good WiFi is more important than a physical Ethernet port, and frankly, I’m not that fussed about being able to fit a 3G SIM until I can find a carrier who will give me multiple SIMs for the same account/allowance. Until that frabjous day, I will make do with my MiFi thing. Bluetooth, though, I need Bluetooth for communicating with the Lego Mindstorms robots.

Personally, I don’t care whether it’s got an SD/XD/CF/BS slot or not, but people seem to like them, so I have no objection to there being one. Put one on, or don’t, it’s all good. Its presence, or lack thereof, is not going to stop me buying this thing, is what I’m saying.

Docking station

Now, what you could do here, Mr/Ms Laptop Manufacturer, is take a leaf out of Sony’s Vaio Z book and offer an optical-cable-connected docking station which has got a full desktop GPU in it and can drive 3 monitors. But for preference, make it an nVidia GPU. If you can do that, then the on-board graphics can be the NVS chip.

Optical drive

Yeah, no.

In summary, then:

I want the moon on a stick. But if you build the moon on a stick, I will buy it, and so will others like me. And there are enough like me to justify building this. So go on. What are you waiting for?

Following the well-received half-day Introduction to Nancy and Simple.Data that I presented with Steven Robbins (@Grumpydev) at the Progressive .NET Tutorials 2011, Wendy at SkillsMatter approached me and asked if I’d be interested in doing a full two-day course. I’ve always been very impressed with the courses they offer, so I jumped at the chance to get involved. So you can now sign up for a full-on, two day deep dive into building web applications with minimal code.

The course will quickly cover the basics, such as View Engines, Dependency Injection, Nancy’s TDD/BDD test helpers and Simple.Data’s built-in data mocking functionality, then move on to more interesting things like: security, authentication (including OAuth) and authorisation; building RESTful web APIs (proper ones, not just HTTP web services); using different data stores with Simple.Data; and different methods for validating data.

It’ll be a heavily interactive couple of days. You can either turn up with an idea for a web application that you want to get started on, or you can work on an app I’ve designed to exercise the skills you’ll be learning. Either way, at the end of the course, that code is yours to take away and use as a starting-off point or a useful reference.

The first instance of the course is on the 12th and 13th of March, and there are still places available. Lots of places. All of them, in fact. So go sign up for it.

It’s taken me so much longer to get here than I originally expected, but I’ve released the first 1.0 beta version of Simple.Data.

If this post is the first you’ve heard of Simple.Data, then head over to the GitHub page and browse back through previous posts to find out more.

New features

I didn’t do a post for the 0.14 release, so I’ll cover the changes from that as well as what’s new in the 1.0 beta.

Eager-loading with With

Since very early on, Simple.Data has supported lazy-loading when you reference a joined property from the dynamic record type. There were two issues with that: firstly, if you assigned the record to a static type, the joined properties were not hydrated; secondly, multiple selects do not make DBAs happy, and should be avoided if possible.

Now you can use the With method to load the joined data at the same time as the main record.

var db = DatabaseHelper.Open();
Customer actual = db.Customers.FindAllByCustomerId(1).WithOrders().FirstOrDefault();

This uses a single SQL select statement to pull all the Customer rows, plus the Order detail rows, and groups the data in-memory; generally, that’s more efficient than running two select operations. When the record is converted to the Customer type, it sets the ICollection<Order> property at the same time, either creating a new instance of List<Order>, or populating an existing instance if the property is readonly.

The inverse works too:

var db = DatabaseHelper.Open();
Order actual = db.Orders.FindAllByOrderId(1).WithCustomer().FirstOrDefault();

If the property name is not the same as the database table name, you can use an alias to tweak it:

var db = DatabaseHelper.Open();
var actual = db.Orders.FindAllByOrderId(1).With(db.Orders.OrderItems.As("Items"));

And if there’s no referential integrity in the database, you can specify an explicit join separately:

dynamic manager;
var q = _db.Employees.All()
    .OuterJoin(_db.Employees.As("Manager"), out manager)
    .On(Id: _db.Employees.ManagerId)
    .With(manager);

(Oh, yeah, and check out the OuterJoin method. Finally.)

Of course, if there’s no referential integrity, it’s hard for Simple.Data to work out whether the joined property is a collection or a complex object, so you can specify WithOne or WithMany to help it out:

dynamic manager;
var q = _db.Employees.All()
    .OuterJoin(_db.Employees.As("Manager"), out manager)
    .On(Id: _db.Employees.ManagerId)
    .WithOne(manager);

And of course, you can mix and match all these. This test gives you a good idea of some of the heavy lifting that’s going on for you with this feature, just with the SQL:

public void MultipleWithClauseJustDoesEverythingYouWouldHope()
{
    const string expectedSql = 
        "select [dbo].[employee].[id],[dbo].[employee].[name]," +
        "[dbo].[employee].[managerid],[dbo].[employee].[departmentid]," +
        "[manager].[id] as [__withn__manager__id]," +
        "[manager].[name] as [__withn__manager__name]," +
        "[manager].[managerid] as [__withn__manager__managerid]," +
        "[manager].[departmentid] as [__withn__manager__departmentid]," +
        "[dbo].[department].[id] as [__with1__department__id]," +
        "[dbo].[department].[name] as [__with1__department__name]" +
        " from [dbo].[employee] left join [dbo].[employee] [manager] " +
        "on ([manager].[id] = [dbo].[employee].[managerid])" +
        " left join [dbo].[department] " +
        "on ([dbo].[department].[id] = [dbo].[employee].[departmentid])";

    dynamic manager;
    var q = _db.Employees.All()
        .OuterJoin(_db.Employees.As("Manager"), out manager)
        .On(Id: _db.Employees.ManagerId)
        .With(manager)
        .WithDepartment();

    GeneratedSqlIs(expectedSql);
}

(Never mind the logic involved in turning that result set into the correct in-memory object graphs.)

Now, this works on multi-record queries, but not on single-record ones such as FindById or the key-driven Get method, and that’s more problematic since those methods don’t return a query you can modify, just a record. In the past I did actually toy with having the SimpleRecord type do lazy self-evaluation, but the fact that the NUnit Assert.IsNull test wouldn’t accept that the object was null even when it swore black-was-blue that it was put me off. (It works for Nullable<T>; no fair.)

Instead of that, and this is only in the 1.0 beta release, you can specify your With clause before the Get or FindBy:

var db = DatabaseHelper.Open();
Customer actual = db.Customers.WithOrders().Get(1);

So now you get all that goodness for single records, where it’s arguably more useful anyway.

Upsert

Inserting and updating is all very well, but sometimes you’ve got some data and you just don’t know whether it’s in the database already or not. If it is, you want to update the row with some new values; if it’s not, you want to insert it. Boring.

To save you the time and trouble, Simple.Data now provides the Upsert method. Give it a record, and it will do all the checking to see if it exists or not. And in beta2, there’ll be back-end database-specific optimizations; for example, if you’re using the SQL Server provider with SQL 2008 or later, it will use the MERGE operation.

Upsert returns the record as it is in the database following the operation, with any database-specified values intact.

var db = DatabaseHelper.Open();
var user = new User {Id = 2, Name = "Charlie", Password = "foobar", Age = 42};
var actual = db.Users.Upsert(user);

That’s one example, but there are plenty of other ways to use Upsert. Take a look at the tests to see the others.

NuGet and SemVer

I’ve pushed this release to NuGet using the Semantic Version number for pre-release (1.0.0-beta1), which NuGet added support for in 1.6. Using this form means that NuGet knows that it’s pre-release software, and you’ll have to explicitly tell it that that’s what you want. So to get the 1.0 beta releases, remember to use the –Pre flag for the Install-Package command. The great thing about this is that when we get to 1.0 RTW, I’ll start on 1.1.0-alpha1 and both package types will be available from the repository.

If you’re waiting for the Mono build, I hope to have it out as a tgz by the end of this weekend.

What’s still to do?

So with those two features, I’m drawing a temporary line in the sand and focusing on getting everything to release quality. That means implementing some optimizations around object creation and database tricks, but more importantly, working on test coverage, refactoring some messy code, and making the documentation comprehensive. Help on that last one would be much appreciated!

Come to that, if anybody wants to make a really nice website…

This does not, however, mark in any way an end to my relationship with Dot Net Solutions. I will remain in my current position as Principal Consultant and will continue to work closely on major projects and supporting Microsoft as an active member of the VTSP (Virtual Technology Specialist Program) performing consulting and conducting ADSs (Architecture Design Sessions).

From my perspective, there are things one cannot achieve as a singleton, and this continuing relationship will enable me to pursue potential work that would otherwise be out of reach.

Dot Net Solutions has been a great company to work for, and over the last two years they have been very supportive of my community and OSS work. I have learned a great deal, and I like to think that I have made a valuable contribution too. As a company, DNS are in an excellent position to grow this year and beyond, and I am very excited to remain part of that process.

If you would like to discuss what I can do for your project, team, or company, please use the Contact Me page to get in touch.

In case you don’t know, eager loading is a pattern for getting data from the database in an optimal fashion, with the minimum number of requests or least-possible server load. It’s the opposite of lazy loading, which Simple.Data already does.

The implementation syntax will look like this:

    IEnumerable<Customer> customers = _db.Customers.FindAllByRegion("South")
                .With(_db.Customers.Orders)
                .Cast<Customer>();

This will create the customer objects and populate an ICollection<Order> Orders property in each one, assuming there is one.

There are two directions in which a class may have relationships with other classes: it may have “parent” (or “lookup”) classes, where there is at most a single instance of that parent class within the instance of the class in question; and it may have “child” classes, where there is a collection of instances of the child class. Eager loading of parent classes is trivial; you just add the necessary join(s) to the master table(s) and marshal the data into the right places in code.

Eager loading of child classes is similarly easy when there is only a single child collection to be loaded; you join to the detail table and then process the results so that a single instance of the main table row is created, and instances of the detail table rows are grouped into collections.

For example, loading user objects with a list of phone numbers:

SELECT User.Id, User.Name, Phone.Number
FROM User
LEFT JOIN Phone ON User.Id = Phone.UserId

Returned rows:

From this we create a User object for Alice with her two phone numbers, and an object for Bob with his. Even if there are a lot of rows in the detail table, this is still the preferred way of handling eager loading for this scenario as it involves the fewest round-trips to the database. Often, you can handle a parent-child-child relationship using this algorithm, too, although beyond that point you’re getting a bit scary.

The complexity arises when you want to load two unrelated child collections, say, load Customers with their Invoices and Orders:

SELECT Customer.*, Invoice.*, Order.*
FROM Customer
LEFT JOIN Invoice ON Customer.Id = Invoice.CustomerId
LEFT JOIN Order ON Customer.Id = Order.CustomerId

This creates what is technically called an “outer Cartesian product”, where a row is returned for every possible combination of the rows from the two outer-joined tables. So if you’re grabbing the details for a single valuable customer who has placed a thousand orders and generated a thousand invoices, you get a million rows back from the database. I have a hunch this might not be optimal.

So this is my Request for Comments: what would you like to see done here?

The options are:

1. Only allow one detail-table With clause to be specified per query. Throw an exception if there is more than one.
2. Allow multiple detail-table With clauses, and use additional SELECT operations for all except the first.
3. Add a parameter to the With method allowing the developer to specify the technique on a per-detail-table basis:

    var customers = _db.Customers.FindAllByRegion("South")
                .With(_db.Customers.Orders, WithUsing.Join)
                .With(_db.Customers.Orders.Items, WithUsing.Join)
                .With(_db.Customers.Invoices, WithUsing.ExtraQuery);

If only one detail table is specified, or one detail with a detail below it (e.g. _db.Customers.Orders.Items), the default behaviour when no WithUsing parameter is supplied will be to use JOIN. Multiple detail tables without a WithUsing will cause the ADO adapter to throw an exception (other adapters, such as the one for MongoDB, can use the With clause as they see fit).

I think I’ve rubber ducked here a bit, as Option 3 is looking like a no-brainer, but I’ve typed it all now, so if you do have any comments or suggestions, please do leave them below. I’m planning on starting code for this on Sunday morning, so there’s plenty of time to change my mind.

I originally built Simple.Data as a proof-of-concept when there was a big web-scuffle over what was then called Microsoft.Data (now WebMatrix.Data). David Fowler, a developer I have since come to respect greatly, posted an example on his blog which used concatenated SQL, and everybody lined up to tear him a new one. I’m a big fan of code over cockfights, so I had a go at implementing a simple data access library which made SQL Injection impossible by eliminating SQL strings. Thus Simple.Data. You told it what you wanted, and provided the necessary values, and then it wrapped them safely up in parameters and executed them. I stuck it on GitHub, and posted about it on my blog.

(In fact, the first version did allow you to execute SQL strings, much like Dapper and Massive, because the functionality was extremely limited otherwise. As I’ve iterated on it, I’ve been able to expose nearly all the required functionality using the dynamic approach, so the only way you can execute SQL directly now is by getting an IDbConnection object from the AdoAdapter and using ADO.NET against it.)

The most recent project that we’ve used it on at Dot Net Solutions is the Met Office open data thing that was announced on Tuesday. And that forced me to bring forward an optimisation that’s been way down my to-do list, partly because I didn’t really just how much of an optimisation it would be.

The Met Office project involves inserting something like 8 million records a day into a SQL Azure database, which isn’t a huge amount, but enough to need you to be smart about how you do it. The version of Simple.Data that was on NuGet when we started supported bulk inserts, but it wasn’t friendly to the error handling we needed and it assumed it needed to return the inserted records, doing that whole ‘select just-inserted-record’ thing, which is often completely unnecessary.

(So it turns out that when you’re handling TryInvokeMember in a DynamicObject, you can actually find out whether the return value is used by the caller, and not bother if it isn’t. But that’s another blog post.)

Anyway, I tweaked a couple of things and shaved off a fraction of the time it was taking, but it was a small fraction, and things were still far too slow. So we did what we should have done in the first place, and used SqlBulkCopy.

If you haven’t used this (SqlClient-specific) method, you should read up on it and keep it in your mental list of “things that are good that I might need some day”. It lets you prepare a big batch of rows in a DataTable (turns out they’re still good for something) and then insert them in a single operation, and man, it’s quick.

But it’s SQL Server specific, so I couldn’t support it in the generic ADO adapter code.

I’ve exposed a few interfaces in the Simple.Data.Ado assembly which providers can optionally implement if they need to do something a little differently or can do something better. The first instance was ICustomInserter, which is implemented in the Oracle provider to handle fetch-backs in a world without IDENTITY columns. Since then I’ve added more as I went along, and IBulkInserter was one of them because, as I said earlier, I had half a mind to implement this. And now I have.

Anyway, I’ll stop blathering now and just post the comparison code I wrote (measures time to insert 10,000 records, five times) and the before and after results.

Code:

namespace BulkInsertComparison
{
    using System;
    using System.Collections.Generic;
    using System.Diagnostics;
    using Simple.Data;

    class Program
    {
        static void Main(string[] args)
        {
            var db = Database.OpenConnection("data source=.;initial catalog=BulkInsertTest;integrated security=true");

            for (int i = 0; i < 5; i++)
            {
                Console.WriteLine(TimeInsert(db));
            }
        }

        private static TimeSpan TimeInsert(dynamic db)
        {
            var stopwatch = Stopwatch.StartNew();
            db.Target.Insert(GenerateItems(10000));
            stopwatch.Stop();
            return stopwatch.Elapsed;
        }

        static IEnumerable<Item> GenerateItems(int number)
        {
            for (int i = 0; i < number; i++)
            {
                var guid = Guid.NewGuid();
                yield return new Item(0, guid, guid.ToString("N"));
            }
        }
    }

    class Item
    {
        private readonly int _id;
        private readonly Guid _guid;
        private readonly string _text;

        public Item(int id, Guid guid, string text)
        {
            _id = id;
            _guid = guid;
            _text = text;
        }

        public string Text { get { return _text; } }
        public Guid Guid { get { return _guid; } }
        public int Id { get { return _id; } }
    }
}

Before:

00:00:16.9799819
00:00:17.1971797
00:00:18.0744958
00:00:19.1514537
00:00:17.3798541

After:

00:00:00.4616911 (First run includes MEFing IBulkInserter)
00:00:00.2757802
00:00:00.2852119
00:00:00.2504587
00:00:00.2453277

Totally worth it.

0.12.2 on NuGet now. Mini-roadmap: eager-loading (0.14) and upserts (0.15).