The first little pig was lazy and frivolous, so he built his house out of straw. When the Big Bad Wolf arrived, he huffed and he puffed and he blew the house down, and ate the little pig, because this is the Grimm version.

The second little pig was very serious, so he decided to engage a large consultancy firm to handle the building of his house for him. The consultancy firm assigned four of their best Big Fat Pigs to the project, and they scheduled a very expensive meeting with the little pig to discuss requirements. The little pig told them he wanted a nice, comfortable house with modern conveniences which would keep the Big Bad Wolf at bay. The Big Fat Pigs told the little pig that he should plan ahead more, and recommended that he build a large skyscraper in case he ever decided to run a multi-national corporation from the house. The little pig worried that this would be expensive, but the big fat pigs assured him that costs would be kept down by using cost-effective materials and off-shoring the actual building of the skyscraper. The little pig didn’t really understand much about building houses, and the big fat pigs seemed to be very clever and had lots of experience, so he signed a very long contract and waited for his skyscraper. Two months later the Big Fat Pigs came back to tell the little pig that they had had a pre-pre-meeting meeting meeting, and had clearly defined an early draft of the project plan. In only eight months they would have the final draft of The Plan, and would be ready to select an architect, who would then submit three different designs for the skyscraper along with thousands of pages of documentation detailing the wolf-proofing measures, so the little pig could be absolutely sure he was going to get the very bestest skyscraper that had ever been built. The Big Fat Pigs had a Gantt chart that covered three years and 142 participants and stakeholders, all with clearly defined activities and responsibilities so they knew exactly where they would be and what they would be doing at any time over the three year project. The Gantt chart was printed in colour on A0 paper and everything.

Sadly, the little pig never got to see the beautiful Gantt chart, because he had been eaten by the Big Bad Wolf. Luckily for the Big Fat Pigs, a clause in the contract he had signed meant that the project could continue in spite of his death, so they carried on. Nine years later, as the final prefabricated part of the skyscraper was fitted for the seventh time (because the previous six parts had not fitted correctly when they arrived from the land far away), the entire skyscraper collapsed because, it turned out, it had been built from substandard parts made from cheap materials in several different countries by unqualified workers who were working from different versions of the design document, which was two thousand pages long. Oh, and somebody had forgotten to dig the foundations, but by the time this was noticed, it was too late to do anything about it, so a complicated system of tensile steel cables and tent pegs was added to the base of the skyscraper to stop it falling over.

(The Big Fat Pigs were very surprised when the tent pegs didn’t work, despite the fact that they had used the same idea on every project they’d ever done and it had never worked. They thought maybe it would work next time, if they incorporated the learnings from this project.)

The local council insisted that the little pig’s family clean up the mess that the collapsing skyscraper had created, and make good the damage to neighbouring properties. The Big Fat Pigs put in a tender for the clean-up project, and got it. That was four years ago. The rusting remains of the skyscraper are still where they landed; the Big Fat Pigs have succeeded in designing a new system for reporting on the progress of the clean-up operation, and are currently waiting for executive approval so that they can move on to the pseudo code stage.

The third little pig hired a really smart architect, who knew a very good builder. The little pig told them he needed a house to protect him from the wolf, so the architect drew up a simple, strong design and the builder built it. After he had moved in, the little pig wanted some improvements, so he went back to the architect and the builder to ask for them. Because the house was very simple, and the builder had built it properly, they were able to make the new improvements quickly and easily, and the third little pig lived happily ever after.

THE END

Another post giving the 501 Developer Manifesto more attention than it deserves. I honestly wouldn’t waste my time reading it if I were you.

…

Why are you still here? If you’ve seriously got nothing better to do than read this, I pity you. I really, really pity you. Because you’re pitiable.

“Pity” is a derogatory word

Let’s get this one out of the way for a start. There’s a reason why “I don’t want your pity” is one of the hallmark clichés of badly written dialogue. If there is probably some pity in your attitude toward someone, it implies that you feel lucky not to be that person; that their situation is deficient in some way that yours is not. If you are making specific reference to an aspect of that someone’s life that they consider important and special and even defining, then you shouldn’t be surprised when they are insulted, and paraphrase Fight Club in Hacker News comments.

A clarification of terms

I shouldn’t really need to write this, as this post on Scott Hanselman’s blog (the first link on the 501 Manifesto page) explains very clearly what is intended by the epithet “501 developer”. It has nothing to do with what time you leave work, or how hard you work while you are there. Being a 501 developer means that you have no great enthusiasm for your work; no desire to learn new and interesting things; no ambition within your chosen career. It means you’re probably just grinding the levels until you get to a better-paid management role, which you will then carry out very badly because you don’t really understand thing one about software development.

If you:

• Never read technical blogs
• Are not aware of open source projects (unless they’re forced on you at work)
• Have never attended any kind of developer-focused event
• Don’t own any books about coding or productivity
• Aren’t quite sure why Github is called Github
• Couldn’t* care less about trying to be better at your job

…then you are a 501 developer, and no, we don’t respect you for it. If you do any of those things, you’re not a 501 developer, regardless of what time you leave work, and I respect you as a professional and as a human being.

*Yes, Americans, the term is “couldn’t care less”. “I could care less” means “I care”.

It is perfectly possible to arrive at work at 9am, leave work at 5 or 5:30pm, and yet still read blogs and discuss software development on the internet and follow internet-famous software people Twitter and go to your local user group when there’s a speaker talking about something that interests you and read Clean Code and maybe do a little bit of extra-curricular coding with something new, and still spend plenty of time with your family and your Moog synthesiser and collection of blank cassette tapes.

That’s the real issue here, is that by taking the time to register a domain and create a web page and attach a blog to it, and by linking to Computer Zen and Coding Horror and making apparent his knowledge of the Agile manifesto, the author of the 501 manifesto has identified himself as anything but a 501 developer. Maybe the problem is that he’s lucky enough to have never worked with an actual 501 developer, in a massive room full of 501 developers for a 501 company in a 501 vertical market, so he doesn’t understand just how bad it can be.

Because you don’t have to be obsessive about it. You don’t have to self-identify as a ninja or a rock-star or a temporally-displaced Olympian demi-god (no? Just me then*). You don’t have to write the blogs, or speak at the user groups, or write the books, or make the next big thing to avoid being classified 501 (but the people who do those things would prefer it if you didn’t make fun of them for it). You just need to care, just a little bit, that’s all.

*Irony, dipshit.

One last thing

If you do have an obsession, and it’s not your job, what are you doing to rectify that situation? Because if you have one all-consuming passion and it’s not what you do all day, every day, and the thing that you do do all day every day keeps you from that passion, that sucks.

It sucks to be in that situation. And you can write a manifesto and you can think you’re “taking back the word” and you can convince yourself the people you work with who love what they’re doing and can’t believe somebody’s paying them to do it are pitiable, but at the end of the day you’re the one being kept from doing the thing you really love by having to do something you merely tolerate because you have two kids and an oppressive mortgage.

#501manifesto? Here’s a better idea: Find a job you love and you’ll never work a day in your life.

— Mark Rendle (@markrendle) April 17, 2012

As I browse around fantasy-laptop-shopping, I see lots of laptops which have features that I really, really want, but I’ve never found one laptop that has everything. So I’m going to outline the spec I want, comprised entirely of things that are either available in current laptops or, at least, announced and imminent parts.

Currently I have a late-2009 15” MacBook Pro, which has a 2.53Ghz Core 2 Duo CPU, and has been upgraded with 8GB of RAM and a 512GB Crucial m4 SSD. I also have access to a 17” HP Envy, on loan from my last employer while I am doing some freelance work for them.

I do most of my own development, including open source stuff and some speculative projects, on the MacBook, using Parallels 7 to virtualise Windows 7, because when I had it running under Snow Leopard’s Boot Camp it kept blue-screening at inopportune moments. Like five minutes from the end of my presentation at DDD SouthWest 2011.

The MacBook also gets used for most of my speaking engagements, although I have been known to use the Envy when I’m talking about Azure because I’m not sure about running the Azure Emulator inside an emulator.

The Spec

Form factor

I’m happy with the size and weight of the MacBook Pro, although obviously thinner and lighter is always welcome. Remains to be seen whether those Air-style MacBook Pro rumours are true or not. The other good thing about Macs is their small, light power supplies. The brick that comes with the Envy seems to weigh almost as much as the laptop itself, and it insists on you dragging the three-pronged not-quite-a-kettle lead around too. It all adds to the mass and bulk, especially on long trips to overseas conferences. Just make it as thin and light as you can; whatever it adds to the cost I’ll easily make up in savings on chiropractic bills.

CPU

I’m running Visual Studio 2010 and 2011, plus SQL Server 2008 R2 and 2012 and IIS 7.5 or IIS 8 running as services, and emulators for Azure and Windows Phone and VMs for Linux and suchlike, and a whole bunch of other cool beta stuff because I’m an incurable neophile, so I want a beefy CPU. I’m thinking Intel Core i7-3820QM, which is supposed to be available next month and is based on the new 22nm Ivy Bridge architecture. Shouldn’t be too much of a problem to get that.

Hard disk

Needs to be solid-state, obviously. In pure fantasy-land, I’d have a pair of 512GB SSDs set up in RAID/0, but I’d settle for one 512GB SSD (which I already own) and a 750GB Seagate Momentus XT secondary drive. So two drive bays, obviously.

Before we move on, let me just suggest one extra thing: can I get an easily-accessible HDD/SDD caddy system for the primary drive (maybe a bit like a PCMCIA), so I can switch between operating systems by swapping drives? That would be awesome.

Memory

The Lenovo ThinkPad W520 is the existing example here: you can spec that up with 32GB of RAM. That should do it. But can I please have that in a laptop that weighs less than my head? And a tip: if adding 24GB of RAM to your default 8GB spec in your “Configure your laptop” page costs more than buying 32GB of compatible RAM from Crucial, guess which I’m going to do? Yeah.

GPU

I don’t play games on PC, because I have an Xbox 360 and a PS3 and a Wii and a DS and a PS Vita so I’m really OK for that stuff. But there are other uses for GPUs these days: I want to play with C++ AMP, and make backups of my Blu-ray and DVD discs, which takes much less time when you’ve got a few CUDA cores kicking around. So I’d like an nVidia GPU, preferably one of the 500M series, although I’d accept an NVS 4200M (which appears to be the new Quadro). Obviously the GPU should be switchable, so it can fall back to the integrated graphics in the CPU when running unplugged.

Update: Peter Bright has pointed out that Quick Sync is better than GPU for video encoding/transcoding, which is true. I still want a GPU for playing with C++ AMP, though.

Screen

One of these, please. 15”, and 16:10 (or 8:5) format. 16:9 is all very well for watching movies, but when you’re working on text, that extra height is really nice to have. Oh, and make it multi-touch with Gorilla Glass 2 while you’re at it. Ta.

Keyboard

A nice, backlit, chiclet keyboard, obviously. With a physical Insert key, because I use ReSharper and there’s a ton of neat things hiding under the Alt-Ins keyboard shortcut. Oh, and if you’re Apple, put bloody # on a key already. Seriously.

If you want to do the Razer thing and bung some programmable OLED keys on there, that would be awesome, but stick them on the top row, not to one side where I’ll keep hitting them instead of Esc or Ctrl or Backspace.

Track-pad

A decent track-pad, which goes *click* when you click it, is essential, and it should support gestures like two-finger scrolling and… actually, just two-finger scrolling would be fine. Yes, I know the screen is touch, but it’s all the way over there; I can’t be reaching for that all the time. Again, Apple is the exemplar here.

Connectivity

A built-in VGA output would be useful for attaching to all those 1024×768 projectors, but stick a Mini DisplayPort on there too, like HP have done on my Envy.

A decent set of USB 3.0 ports (at least two), maybe with a couple of 2.0 for keyboard and mouse.

Really good WiFi is more important than a physical Ethernet port, and frankly, I’m not that fussed about being able to fit a 3G SIM until I can find a carrier who will give me multiple SIMs for the same account/allowance. Until that frabjous day, I will make do with my MiFi thing. Bluetooth, though, I need Bluetooth for communicating with the Lego Mindstorms robots.

Personally, I don’t care whether it’s got an SD/XD/CF/BS slot or not, but people seem to like them, so I have no objection to there being one. Put one on, or don’t, it’s all good. Its presence, or lack thereof, is not going to stop me buying this thing, is what I’m saying.

Docking station

Now, what you could do here, Mr/Ms Laptop Manufacturer, is take a leaf out of Sony’s Vaio Z book and offer an optical-cable-connected docking station which has got a full desktop GPU in it and can drive 3 monitors. But for preference, make it an nVidia GPU. If you can do that, then the on-board graphics can be the NVS chip.

Optical drive

Yeah, no.

In summary, then:

I want the moon on a stick. But if you build the moon on a stick, I will buy it, and so will others like me. And there are enough like me to justify building this. So go on. What are you waiting for?

Following the well-received half-day Introduction to Nancy and Simple.Data that I presented with Steven Robbins (@Grumpydev) at the Progressive .NET Tutorials 2011, Wendy at SkillsMatter approached me and asked if I’d be interested in doing a full two-day course. I’ve always been very impressed with the courses they offer, so I jumped at the chance to get involved. So you can now sign up for a full-on, two day deep dive into building web applications with minimal code.

The course will quickly cover the basics, such as View Engines, Dependency Injection, Nancy’s TDD/BDD test helpers and Simple.Data’s built-in data mocking functionality, then move on to more interesting things like: security, authentication (including OAuth) and authorisation; building RESTful web APIs (proper ones, not just HTTP web services); using different data stores with Simple.Data; and different methods for validating data.

It’ll be a heavily interactive couple of days. You can either turn up with an idea for a web application that you want to get started on, or you can work on an app I’ve designed to exercise the skills you’ll be learning. Either way, at the end of the course, that code is yours to take away and use as a starting-off point or a useful reference.

The first instance of the course is on the 12th and 13th of March, and there are still places available. Lots of places. All of them, in fact. So go sign up for it.

It’s taken me so much longer to get here than I originally expected, but I’ve released the first 1.0 beta version of Simple.Data.

If this post is the first you’ve heard of Simple.Data, then head over to the GitHub page and browse back through previous posts to find out more.

New features

I didn’t do a post for the 0.14 release, so I’ll cover the changes from that as well as what’s new in the 1.0 beta.

Eager-loading with With

Since very early on, Simple.Data has supported lazy-loading when you reference a joined property from the dynamic record type. There were two issues with that: firstly, if you assigned the record to a static type, the joined properties were not hydrated; secondly, multiple selects do not make DBAs happy, and should be avoided if possible.

Now you can use the With method to load the joined data at the same time as the main record.

var db = DatabaseHelper.Open();
Customer actual = db.Customers.FindAllByCustomerId(1).WithOrders().FirstOrDefault();

This uses a single SQL select statement to pull all the Customer rows, plus the Order detail rows, and groups the data in-memory; generally, that’s more efficient than running two select operations. When the record is converted to the Customer type, it sets the ICollection<Order> property at the same time, either creating a new instance of List<Order>, or populating an existing instance if the property is readonly.

The inverse works too:

var db = DatabaseHelper.Open();
Order actual = db.Orders.FindAllByOrderId(1).WithCustomer().FirstOrDefault();

If the property name is not the same as the database table name, you can use an alias to tweak it:

var db = DatabaseHelper.Open();
var actual = db.Orders.FindAllByOrderId(1).With(db.Orders.OrderItems.As("Items"));

And if there’s no referential integrity in the database, you can specify an explicit join separately:

dynamic manager;
var q = _db.Employees.All()
    .OuterJoin(_db.Employees.As("Manager"), out manager)
    .On(Id: _db.Employees.ManagerId)
    .With(manager);

(Oh, yeah, and check out the OuterJoin method. Finally.)

Of course, if there’s no referential integrity, it’s hard for Simple.Data to work out whether the joined property is a collection or a complex object, so you can specify WithOne or WithMany to help it out:

dynamic manager;
var q = _db.Employees.All()
    .OuterJoin(_db.Employees.As("Manager"), out manager)
    .On(Id: _db.Employees.ManagerId)
    .WithOne(manager);

And of course, you can mix and match all these. This test gives you a good idea of some of the heavy lifting that’s going on for you with this feature, just with the SQL:

public void MultipleWithClauseJustDoesEverythingYouWouldHope()
{
    const string expectedSql = 
        "select [dbo].[employee].[id],[dbo].[employee].[name]," +
        "[dbo].[employee].[managerid],[dbo].[employee].[departmentid]," +
        "[manager].[id] as [__withn__manager__id]," +
        "[manager].[name] as [__withn__manager__name]," +
        "[manager].[managerid] as [__withn__manager__managerid]," +
        "[manager].[departmentid] as [__withn__manager__departmentid]," +
        "[dbo].[department].[id] as [__with1__department__id]," +
        "[dbo].[department].[name] as [__with1__department__name]" +
        " from [dbo].[employee] left join [dbo].[employee] [manager] " +
        "on ([manager].[id] = [dbo].[employee].[managerid])" +
        " left join [dbo].[department] " +
        "on ([dbo].[department].[id] = [dbo].[employee].[departmentid])";

    dynamic manager;
    var q = _db.Employees.All()
        .OuterJoin(_db.Employees.As("Manager"), out manager)
        .On(Id: _db.Employees.ManagerId)
        .With(manager)
        .WithDepartment();

    GeneratedSqlIs(expectedSql);
}

(Never mind the logic involved in turning that result set into the correct in-memory object graphs.)

Now, this works on multi-record queries, but not on single-record ones such as FindById or the key-driven Get method, and that’s more problematic since those methods don’t return a query you can modify, just a record. In the past I did actually toy with having the SimpleRecord type do lazy self-evaluation, but the fact that the NUnit Assert.IsNull test wouldn’t accept that the object was null even when it swore black-was-blue that it was put me off. (It works for Nullable<T>; no fair.)

Instead of that, and this is only in the 1.0 beta release, you can specify your With clause before the Get or FindBy:

var db = DatabaseHelper.Open();
Customer actual = db.Customers.WithOrders().Get(1);

So now you get all that goodness for single records, where it’s arguably more useful anyway.

Upsert

Inserting and updating is all very well, but sometimes you’ve got some data and you just don’t know whether it’s in the database already or not. If it is, you want to update the row with some new values; if it’s not, you want to insert it. Boring.

To save you the time and trouble, Simple.Data now provides the Upsert method. Give it a record, and it will do all the checking to see if it exists or not. And in beta2, there’ll be back-end database-specific optimizations; for example, if you’re using the SQL Server provider with SQL 2008 or later, it will use the MERGE operation.

Upsert returns the record as it is in the database following the operation, with any database-specified values intact.

var db = DatabaseHelper.Open();
var user = new User {Id = 2, Name = "Charlie", Password = "foobar", Age = 42};
var actual = db.Users.Upsert(user);

That’s one example, but there are plenty of other ways to use Upsert. Take a look at the tests to see the others.

NuGet and SemVer

I’ve pushed this release to NuGet using the Semantic Version number for pre-release (1.0.0-beta1), which NuGet added support for in 1.6. Using this form means that NuGet knows that it’s pre-release software, and you’ll have to explicitly tell it that that’s what you want. So to get the 1.0 beta releases, remember to use the –Pre flag for the Install-Package command. The great thing about this is that when we get to 1.0 RTW, I’ll start on 1.1.0-alpha1 and both package types will be available from the repository.

If you’re waiting for the Mono build, I hope to have it out as a tgz by the end of this weekend.

What’s still to do?

So with those two features, I’m drawing a temporary line in the sand and focusing on getting everything to release quality. That means implementing some optimizations around object creation and database tricks, but more importantly, working on test coverage, refactoring some messy code, and making the documentation comprehensive. Help on that last one would be much appreciated!

Come to that, if anybody wants to make a really nice website…

This does not, however, mark in any way an end to my relationship with Dot Net Solutions. I will remain in my current position as Principal Consultant and will continue to work closely on major projects and supporting Microsoft as an active member of the VTSP (Virtual Technology Specialist Program) performing consulting and conducting ADSs (Architecture Design Sessions).

From my perspective, there are things one cannot achieve as a singleton, and this continuing relationship will enable me to pursue potential work that would otherwise be out of reach.

Dot Net Solutions has been a great company to work for, and over the last two years they have been very supportive of my community and OSS work. I have learned a great deal, and I like to think that I have made a valuable contribution too. As a company, DNS are in an excellent position to grow this year and beyond, and I am very excited to remain part of that process.

If you would like to discuss what I can do for your project, team, or company, please use the Contact Me page to get in touch.

In case you don’t know, eager loading is a pattern for getting data from the database in an optimal fashion, with the minimum number of requests or least-possible server load. It’s the opposite of lazy loading, which Simple.Data already does.

The implementation syntax will look like this:

    IEnumerable<Customer> customers = _db.Customers.FindAllByRegion("South")
                .With(_db.Customers.Orders)
                .Cast<Customer>();

This will create the customer objects and populate an ICollection<Order> Orders property in each one, assuming there is one.

There are two directions in which a class may have relationships with other classes: it may have “parent” (or “lookup”) classes, where there is at most a single instance of that parent class within the instance of the class in question; and it may have “child” classes, where there is a collection of instances of the child class. Eager loading of parent classes is trivial; you just add the necessary join(s) to the master table(s) and marshal the data into the right places in code.

Eager loading of child classes is similarly easy when there is only a single child collection to be loaded; you join to the detail table and then process the results so that a single instance of the main table row is created, and instances of the detail table rows are grouped into collections.

For example, loading user objects with a list of phone numbers:

SELECT User.Id, User.Name, Phone.Number
FROM User
LEFT JOIN Phone ON User.Id = Phone.UserId

Returned rows:

From this we create a User object for Alice with her two phone numbers, and an object for Bob with his. Even if there are a lot of rows in the detail table, this is still the preferred way of handling eager loading for this scenario as it involves the fewest round-trips to the database. Often, you can handle a parent-child-child relationship using this algorithm, too, although beyond that point you’re getting a bit scary.

The complexity arises when you want to load two unrelated child collections, say, load Customers with their Invoices and Orders:

SELECT Customer.*, Invoice.*, Order.*
FROM Customer
LEFT JOIN Invoice ON Customer.Id = Invoice.CustomerId
LEFT JOIN Order ON Customer.Id = Order.CustomerId

This creates what is technically called an “outer Cartesian product”, where a row is returned for every possible combination of the rows from the two outer-joined tables. So if you’re grabbing the details for a single valuable customer who has placed a thousand orders and generated a thousand invoices, you get a million rows back from the database. I have a hunch this might not be optimal.

So this is my Request for Comments: what would you like to see done here?

The options are:

1. Only allow one detail-table With clause to be specified per query. Throw an exception if there is more than one.
2. Allow multiple detail-table With clauses, and use additional SELECT operations for all except the first.
3. Add a parameter to the With method allowing the developer to specify the technique on a per-detail-table basis:

    var customers = _db.Customers.FindAllByRegion("South")
                .With(_db.Customers.Orders, WithUsing.Join)
                .With(_db.Customers.Orders.Items, WithUsing.Join)
                .With(_db.Customers.Invoices, WithUsing.ExtraQuery);

If only one detail table is specified, or one detail with a detail below it (e.g. _db.Customers.Orders.Items), the default behaviour when no WithUsing parameter is supplied will be to use JOIN. Multiple detail tables without a WithUsing will cause the ADO adapter to throw an exception (other adapters, such as the one for MongoDB, can use the With clause as they see fit).

I think I’ve rubber ducked here a bit, as Option 3 is looking like a no-brainer, but I’ve typed it all now, so if you do have any comments or suggestions, please do leave them below. I’m planning on starting code for this on Sunday morning, so there’s plenty of time to change my mind.

I originally built Simple.Data as a proof-of-concept when there was a big web-scuffle over what was then called Microsoft.Data (now WebMatrix.Data). David Fowler, a developer I have since come to respect greatly, posted an example on his blog which used concatenated SQL, and everybody lined up to tear him a new one. I’m a big fan of code over cockfights, so I had a go at implementing a simple data access library which made SQL Injection impossible by eliminating SQL strings. Thus Simple.Data. You told it what you wanted, and provided the necessary values, and then it wrapped them safely up in parameters and executed them. I stuck it on GitHub, and posted about it on my blog.

(In fact, the first version did allow you to execute SQL strings, much like Dapper and Massive, because the functionality was extremely limited otherwise. As I’ve iterated on it, I’ve been able to expose nearly all the required functionality using the dynamic approach, so the only way you can execute SQL directly now is by getting an IDbConnection object from the AdoAdapter and using ADO.NET against it.)

The most recent project that we’ve used it on at Dot Net Solutions is the Met Office open data thing that was announced on Tuesday. And that forced me to bring forward an optimisation that’s been way down my to-do list, partly because I didn’t really just how much of an optimisation it would be.

The Met Office project involves inserting something like 8 million records a day into a SQL Azure database, which isn’t a huge amount, but enough to need you to be smart about how you do it. The version of Simple.Data that was on NuGet when we started supported bulk inserts, but it wasn’t friendly to the error handling we needed and it assumed it needed to return the inserted records, doing that whole ‘select just-inserted-record’ thing, which is often completely unnecessary.

(So it turns out that when you’re handling TryInvokeMember in a DynamicObject, you can actually find out whether the return value is used by the caller, and not bother if it isn’t. But that’s another blog post.)

Anyway, I tweaked a couple of things and shaved off a fraction of the time it was taking, but it was a small fraction, and things were still far too slow. So we did what we should have done in the first place, and used SqlBulkCopy.

If you haven’t used this (SqlClient-specific) method, you should read up on it and keep it in your mental list of “things that are good that I might need some day”. It lets you prepare a big batch of rows in a DataTable (turns out they’re still good for something) and then insert them in a single operation, and man, it’s quick.

But it’s SQL Server specific, so I couldn’t support it in the generic ADO adapter code.

I’ve exposed a few interfaces in the Simple.Data.Ado assembly which providers can optionally implement if they need to do something a little differently or can do something better. The first instance was ICustomInserter, which is implemented in the Oracle provider to handle fetch-backs in a world without IDENTITY columns. Since then I’ve added more as I went along, and IBulkInserter was one of them because, as I said earlier, I had half a mind to implement this. And now I have.

Anyway, I’ll stop blathering now and just post the comparison code I wrote (measures time to insert 10,000 records, five times) and the before and after results.

Code:

namespace BulkInsertComparison
{
    using System;
    using System.Collections.Generic;
    using System.Diagnostics;
    using Simple.Data;

    class Program
    {
        static void Main(string[] args)
        {
            var db = Database.OpenConnection("data source=.;initial catalog=BulkInsertTest;integrated security=true");

            for (int i = 0; i < 5; i++)
            {
                Console.WriteLine(TimeInsert(db));
            }
        }

        private static TimeSpan TimeInsert(dynamic db)
        {
            var stopwatch = Stopwatch.StartNew();
            db.Target.Insert(GenerateItems(10000));
            stopwatch.Stop();
            return stopwatch.Elapsed;
        }

        static IEnumerable<Item> GenerateItems(int number)
        {
            for (int i = 0; i < number; i++)
            {
                var guid = Guid.NewGuid();
                yield return new Item(0, guid, guid.ToString("N"));
            }
        }
    }

    class Item
    {
        private readonly int _id;
        private readonly Guid _guid;
        private readonly string _text;

        public Item(int id, Guid guid, string text)
        {
            _id = id;
            _guid = guid;
            _text = text;
        }

        public string Text { get { return _text; } }
        public Guid Guid { get { return _guid; } }
        public int Id { get { return _id; } }
    }
}

Before:

00:00:16.9799819
00:00:17.1971797
00:00:18.0744958
00:00:19.1514537
00:00:17.3798541

After:

00:00:00.4616911 (First run includes MEFing IBulkInserter)
00:00:00.2757802
00:00:00.2852119
00:00:00.2504587
00:00:00.2453277

Totally worth it.

0.12.2 on NuGet now. Mini-roadmap: eager-loading (0.14) and upserts (0.15).

I’ve got Simple.Data running on Mono 2.10.6. Most tests pass. YMMV. Download it here.

Ritalin version

Simple.Data on Mono is something that people have been asking about, and I’ve been meaning to sort out, pretty much since the project went from a proof-of-concept to an actual OSS product. One way or another I’ve never gotten round to it, but a couple of things have made it seem more relevant. Firstly, there are providers for lots of OSS databases now (MySQL, SQLite, PostgreSQL); secondly, a certain @fekberg has been very persistent in his status update requests, so I’ve taken the time this weekend to make it work.

The challenges

For the most part, I’ve been pleasantly surprised by how easy it’s been. I’ve found what I think is a bug in the Mono implementation of either dynamic or LINQ or the combination of the two, which I’m going to file with Xamarin once I’ve created a simple repro project. I’ve come up with a workaround involving old-school class-based IEnumerable/IEnumerator implementations, and the performance doesn’t seem to be affected, so that’s fine.

The hardest thing about the whole process is that MonoDevelop just isn’t Visual Studio 2010 + ReSharper. It’s not a bad IDE by any standards – I’d still take it over Eclipse any day – but I work day in, day out with VS and jumping into any other IDE just feels like somebody moved all the cheese. Add to that the fact that I’m running it under OSX, so even the standard Windows keyboard shortcuts don’t work, and it’s a bit like running in treacle. As I understand it, the Mono Tools for Visual Studio don’t support 2.10; Miguel de Icaza tells me that they’re working on some awesome new VS tooling, so I’m really looking forward to that.

The other big challenge was testing the ADO adapter against a real database. I only own the SQL Server and SQL Compact providers, so I really wanted to test with SQL Server to keep things simple and let me step-debug if necessary. I was expecting not to have fun with this, but it turned out fine. I run my Windows development environment on my MacBook using Parallels 7, so I set up the Host-Guest networking (easy) and opened inbound port 1433 in Windows Firewall (also easy) and that was it. There were a couple of failing tests, one calling a stored procedure with a DataTable and one involving a scalar function, but the rest just passed. I’m guessing that the majority of people who want to use Simple.Data on Mono will be using one of the OSS DB providers, so hopefully this won’t be a problem.

Releases

I don’t know what the Mono NuGet situation is, so I’ll be releasing Mono builds as tgz downloads from the GitHub project page. For the time being, there are differences between the Mono build of Simple.Data.SqlServer and the Microsoft .NET build, so if you want to use that provider, don’t use the NuGet version.

Right now, there’s a 0.11.4 build in the Downloads section which I hope works. However, I haven’t really exercised it to any great degree, so if you encounter any problems please raise an issue if you think it’s a bug, or ask on the mailing list if you aren’t sure.

If you are one of the developers of an adapter or provider, you might want to test against Mono. If, for any reason, that’s not an option, then let me know and I’ll try to find time to test it for you.

Going forward, I will support (in the OSS sense of the word) both Microsoft .NET and Mono for all releases.