Simple.Data 0.11

Some API changes and enhancements

After the slow-down in development caused by all that InMemoryAdapter stuff, there were a few important things I needed to address quickly. One of these will have broken third-party adapters (but not providers) so let’s talk about that one first.

Get

var db = DatabaseHelper.Open();
var user = db.Users.Get(1);
Assert.AreEqual(1, user.Id);

I’m not really sure why Get wasn’t already there, to be honest. Part of the problem is that it requires a new abstract method internally, and until adapter authors implement that method, their users are stuck on <0.11, which I try to avoid where possible.

For the ADO adapter, Get will use the table’s primary key to construct the query (once; it’s then cached internally, no worries about performance). For the MongoDB adapter, I’d expect it to use the built-in id value that Mongo assigns to all records. Somebody is working on an OData adapter, for which, e.g., Customers.Get(1001) will resolve to the /Customers(1001) URL.

Get is supported in the InMemoryAdapter, but you’ll have to configure the key(s) for each table:

var adapter = new InMemoryAdapter();
adapter.SetKeyColumn("Test", "Id");
Database.UseMockAdapter(adapter);
var db = Database.Open();
db.Test.Insert(Id: 1, Name: "Alice");
var record = db.Test.Get(1);
Assert.IsNotNull(record);
Assert.AreEqual(1, record.Id);
Assert.AreEqual("Alice", record.Name);

Trace configurability

The ADO adapter has been writing all generated SQL to the Trace output at the point of execution for a while now. While this is often very useful, I’ve had a couple of people ask if I could make it turn-off-and-on-able, so I have. You can do this in two ways:

In code:

Database.TraceLevel = TraceLevel.Off;

In config:

<?xml version="1.0" encoding="utf-8" ?> <configuration> <configSections> <sectionGroup name="simpleData"> <section name="simpleDataConfiguration"                type="Simple.Data.SimpleDataConfigurationSection, Simple.Data"/> </sectionGroup> </configSections> <simpleData> <simpleDataConfiguration traceLevel="Error"/> </simpleData> </configuration>

(Gotta love XML.)

ADO SQL output will happen with the trace level set to Info, Warning or Error.

More ADO connection control

I occasionally see how Simple.Data performs compared to other ORM/micro-ORM tools, using the PerformanceTests project from Dapper. I was running this through the other day, and I realised that Simple.Data was losing a lot of time opening and closing connections, while the other test cases were mostly using an open connection for the duration of the test. I’ve had a few comments that they’d like more control over the connection, or that Simple.Data is too aggressive in closing connections, so I decided to improve my standing in the Dapper smack-down and hopefully help some real people out too.

Start using an open connection like this:

SqlConnection connection = Program.GetOpenConnection();
((AdoAdapter) db.GetAdapter()).UseSharedConnection(connection);

And stop using it again like this:

((AdoAdapter) simpleDb.GetAdapter()).StopUsingSharedConnection();

And that’s it. In my performance test project (which is in the solution on Github), this knocked 20-30% off the runtime, with 500 FindById operations taking ~80ms, versus ~50ms using plain ADO.

I’ve also tried to tone down the aggression a little when it comes to closing connections, doing it as soon as possible, instead of (occasionally) before.

Immediate road-map

I want to get the Azure Table Service adapter done, and help with the OData adapter, and I’ve got a cool website to build using Simple.Data and Nancy, so Core will go into maintenance while I do those things for a while. I’ll try to fix any problems in a timely fashion, as usual. If you’ve got anything still outstanding as of 0.11.1 (now on Nuget) please gently remind me on Twitter.

Simple.Data 0.10

Deserves its own post, even though I’m about to write the 0.11 one.

This release took much longer than I would have liked, mainly because I bit off more than I could chew. I had a requirement to provide poly-fills that implement query operations against in-memory datasets, on behalf of adapters for data-stores that don’t support certain operations. Think aggregates, grouping and so on. At some point, it occurred to me that this would make it really easy to create an in-memory adapter which would just have all query functionality implemented by these poly-fills.

Aside: From very early on, Simple.Data has had the XmlMockAdapter which could be used for providing a quick and dirty database for test environments. At the time I wrote that, I wasn’t totally confident in the Simple.Data model, so a way to instantiate that data without using the API seemed important. Since then, of course, the API has become much more stable and I’ve gained confidence, so it seems perfectly OK to use it to set up the test data as well as consume it. Consider XmlMockAdapter deprecated.

Simple.Data.InMemoryAdapter

The InMemoryAdapter lives directly in the Simple.Data namespace in the Simple.Data.dll assembly from the Simple.Data.Core package. It is a first-class component, because testing is a first-class concern.

You can use the adapter like this:

Database.UseMockAdapter(new InMemoryAdapter());
var db = Database.Open();
db.Test.Insert(Id: 1, Name: "Alice");
var record = db.Test.FindById(1);
Assert.IsNotNull(record);
Assert.AreEqual(1, record.Id);
Assert.AreEqual("Alice", record.Name);

That UseMockAdapter method is now directly on the Database class. After calling that, all subsequent calls to any of the Database.Open methods will return a database using that adapter.

Any tables you reference in the database will be “created” if they don’t already exist. Internally, the tables are just lists of dictionaries. Any data you insert will retain its CLR type, so the adapter can handle byte arrays and so on quite happily.

Joins are supported:

var adapter = new InMemoryAdapter();
adapter.ConfigureJoin("Customer", "ID", "Orders", "Order", "CustomerID", "Customer");
Database.UseMockAdapter(adapter);
var db = Database.Open();
db.Customer.Insert(ID: 1, Name: "NASA");
db.Customer.Insert(ID: 2, Name: "ACME");
db.Order.Insert(ID: 1, CustomerID: 1, Date: new DateTime(1997, 1, 12));
db.Order.Insert(ID: 2, CustomerID: 2, Date: new DateTime(2001, 1, 1));

var customers = db.Customer
    .FindAll(db.Customer.Orders.Date < new DateTime(1999, 12, 31)).ToList(); Assert.IsNotNull(customers); Assert.AreEqual(1, customers.Count);

And most query operations work OK.

YMMV: This is probably not perfect, but it was stalling other development so I took the decision to release it. So far in my own projects it has not caused any problems; I’ve had one reported which is fixed in 0.11 (more on that in the next post).

Bug fixes and such

There were a bunch of fixes for various issues in 0.10, most of which were contributed by Richard Hopton, who has also released an ADO provider for Sybase SQL Anywhere.

I’m going 100% digital in 2012

I know it’s early for New Year resolutions, but I’m going to share this one now partly so I don’t forget in six weeks.

I buy a lot of media. I have an entire 6′x3′ bookcase full of technical books, most of which are out of date by at least one version. There’s a cubic hectare of Blu-ray and DVD boxes lying around the place, and I’m pretty sure there are boxes of CDs (remember those?) in the loft somewhere. Probably half the available storage space in my house is taken up with things which would actually fit on a few terabytes of hard disk.

It’s not like everything in the house isn’t already digital. We’ve got a 160GB iPod Classic, iPhones, iPads, Kindles, three laptops, an Xbox 360, a PS3 (with a 320GB HD), a 50Mbps internet connection, and on Saturday we bought an Apple TV.

It’s ridiculous. It has to stop. It ends, starting in January. From then on, where a digital purchase option is available, I shall avail myself of it. I will invest in a home server with a few TB of storage on mirrored drives; I’m thinking of the Lacie 5big Network 2, which will take up to 10TB. Then I can “back up” my movie collection to that, and install iTunes on it for music streaming. Paper books will go to charities and libraries; if I want to read any of them again, I’ll buy them in ebook format. Magazines through Newsstand and papers through the Times iPad app.

Now, I just need Sony and Microsoft to distribute all PS3 and Xbox games digitally, and I need never darken the door of another shop. Not that I do anyway. All this crap comes from Amazon.

Just a quick note about Tests

I’ve been fixing a few bugs in Simple.Data over the last couple of days, and I’m feeling the need to post something about the benefits of a good test suite.

For most feature development on Simple.Data, I do test-driven development. The dynamic nature of my API lends itself really well to this approach; this is one of the real reasons TDD is popular with dynamic languages like Ruby and Python. When I want to add a new Query operator or method, I can write a test that uses the syntax I’m aiming for, and the Behaviour test project will compile and run, and I’ll get a failing test, either with a failed assert or an exception. I really like the latter form of failure, since I get a stack trace and I can just dive into the code at that point and start working out what’s wrong.

So that’s great, but the thing that makes me want to write this post is my “QA” process. When someone reports a bug, like this one, it means my test suite is incomplete. So again, the first thing I do is to create a test which reproduces the bug. Then I fix the code so that test passes. Then I do a Release build, and run the full set of tests (currently 560+ including integration tests) to make sure that the fix hasn’t broken anything else. And then, and this is the really awesome part: if all the tests pass, I package the build and push it to NuGet.

I can do this because I trust those tests to be verifying all the behaviour that Simple.Data users are relying on in their applications. If the tests pass, I’m not going to break anybody’s system when they update to the new version. On the (rare) occasions when this has gone wrong, it’s been because I didn’t have the right tests, and I’ve gone back and added them (would you believe the SQL Server test project didn’t test delete’s until yesterday? Epic fail).

When people look at the Simple.Data repository, and say “wow, you’ve got lots of tests”, it’s as if they think that’s a discipline thing, that I’m just really conscientious about coverage. But that’s not it. I couldn’t do this without the tests. And neither can you, whatever you’re working on.

OK, so I think HTML/JS for Metro apps is a good idea

(Just a quick post while I wait for Anders’s talk to start. Hope he confirms destructuring assignment.)

I turned up to BUILD with a negative attitude to this whole idea of using HTML and JavaScript for creating Windows applications. And I held onto that attitude throughout Tuesday’s keynote and Big Picture sessions. I went back to my hotel that evening and threw together an app which pulls issues from Github and displays them in Metro style, and I did it in C# and XAML. And it worked.

Then I watched yesterday’s session on Using the Windows Runtime (WinRT) from C#, and my attitude changed. It’s very easy to create WinRT components in C# which can be consumed from C++ and JavaScript apps. Really, really easy. You barely have to think about it. And the support in this new Windows JavaScript for consuming WinRT components is very, very good, with binding and everything. You can effectively write an MVVM application, but use HTML and CSS instead of XAML.

This is completely brilliant.

Why?

Well, at Dot Net Solutions we do a lot of WPF and Silverlight work. And we’re cracking engineers, but we’re not designers. We get outside resource in to make our software look beautiful. And finding design studios with excellent XAML and Blend expertise is very, very difficult. But finding design studios with excellent HTML, CSS and JavaScript? Well, that’s much easier.

So my Windows app development paradigm in this brave new Win8 world is going to be this: all the internal logic, the Data Access Layer, the Models and ViewModels and so on can be built in C#, with type safety and unit tests and the performance gains. Then those ViewModels are exposed across the WinRT ABI boundary and bound to HTML views, styled with CSS and with all the UI-specific code (transitions, animations etc) implemented in JS by people whose idea of a good colour scheme isn’t “blue”.

And if there’s performance-critical code, I’m going to write it in C++ using the AMP library.

Right. I’m going to watch this talk, then I’m refactoring my Github app and do the UI in HTML.

What I look for in developer candidates

I just sent this to our recruiter (our only recruiter, so please don’t bother if you’re not him), and it occurred to me that it might make an interesting mini-post, so here it is, copied and pasted from my email.

The main thing we look for is strong core programming skills, ideally in C# but will also consider Java and C++ programmers. The interview will include technical questions about Object Oriented programming and whichever skills the candidate claims to have.

Web development is the second most important thing for us these days. Again, there will be questions about web applications, ASP.NET fundamentals and so on.

The third thing we expect is database skills, and I’ll ask questions about database design and SQL. I don’t want people who only know ORM abstraction layers.

Obviously the level to which the questions go varies according to the level we’re interviewing for, but the basics of all three would be covered for all levels.

My big bugbear is when I ask a question which the candidate’s CV implies they should know the answer to, but it turns out they don’t.

Any code questions are asked and answered on the whiteboard, so individuals who rely totally on IntelliSense are in trouble.

If you’re looking for a developer role with a great company in Windsor, UK, and the above doesn’t put you off, please get in touch!

On tour

TL;DR

From October 10th to 13th, I will be doing a mini-tour of Scottish .NET User Groups, in association with DevExpress and Dot Net Solutions.

DevExpress          Dot Net Solutions

It’s not advertising; it’s sponsorship.

The Ritalin version

Way back in May Gary Park asked if I would come and present at the Aberdeen Developers .NET User Group, and I said “where’s Aberdeen”, and he said “in Scotland”, and I said “where’s Scotland” and he said “hundreds of miles away” and I said “um”. But then we thought if I could visit a few groups that I wouldn’t normally get the opportunity to present at, that would be good, and so Gary went off and convinced some other group organisers that this was a cracking opportunity to book some top-notch blathering from quite possibly the softest Southerner south of 55°N, and they said “it’s not that Rendle, is it?” and he said “no” and they said “oh, all right then”, and my Caledonian Code Crusade was conceived. And by the time they realised it was that Rendle, it was too late to book anybody else, so ha.

Completely awesomely, the lovely Rachel Hawley at DevExpress has arranged some sponsorship for this endeavour, and saved me from a fate exactly as bad as freezing to death on park benches. Huge thanks to Rachel and DevExpress for that.

Thanks also to Dot Net Solutions for helping out with fuel costs and allowing me to work from various coffee shops and motorway services for these few days.

So, for the week following DDD North (at which I hope to be presenting (hint, hint)), I’ll be at a user group near you, as long as you live in or near Glasgow, Edinburgh, Dundee or Aberdeen. I’m still waiting to confirm the talks for a couple of groups, so I’ll fill in the details below as and when I get them.

Monday 10th, Glasgow

In Glasgow, I’ll be sharing some tips on How to Manage Your Manager, and previewing a talk I’m doing at Øredev in November, which is based on Zen and the Art of Motorcycle Maintenance and is all about Quality in software and software development. Details here.

Tuesday 11th, Edinburgh

Code-heavy night in Edinburgh, with the Functional Alchemy talk and CoffeeScript 101, which is premiering at DDD North. Should be fun.

Wednesday 12th, Dundee

I’ll be talking about the Windows Azure Table Storage Service in Dundee, looking at the NoSQL, schema-less nature of the beast, and sharing some ideas on how to take advantage of that fact. Then I’ll share the ever-popular tips on How To Manage Your Manager.

Thursday 13th, Aberdeen

This will be my Functional Alchemy talk, which is usually good fun (and I’ve got a couple of new examples since the last time I gave it); and How to Manage Your Manager, which is inexplicably popular given the uniformly excellent quality of all the managers I’ve worked with over the last 20 years. Oh yes.

Internal for a reason

One of the things about working on an open source project is that everybody can go and hunt through the code and see all the private, protected and internal things. I occasionally get requests to make certain types or methods public, because the requester can see a way to use them to achieve some piece of functionality that they are otherwise unable to implement. Sometimes these requests have an undertone of “why are you hiding this stuff in the first place?” So I thought I’d take a moment to explain the reason why we have these visibility modifiers at all, why I use them the way I do, and also to highlight a couple of times where I was over-protective (or over-internalist) and did actually make the change.

You can’t depend on me

When a type is internal, or a method isn’t public, it means your code can’t take a dependency on it. Think about how much we try to avoid firm dependencies just within a single codebase with a single programmer maintaining it. We do this to avoid the code becoming brittle and hard to maintain; the fewer things that directly depend on something being the way it is, the fewer places we need to make changes when we change the thing they depend on. This is just good practice.

For a public API, it becomes even more important. Every publically accessible member in an API gets an arbitrary number of dependencies the moment you publish the library. So you hide everything, just as a matter of course, and only make things public if you really have to.

As an example, in Simple.Data, I have a type called ExpressionHelper, which turns an IDictionary<string. object> into a SimpleExpression composed of straight-forward equality comparisons. I use it to generate the criteria for all the FindBy/FindAllBy methods. I got a change request asking me to make this type public, and I rejected it, because I think there might be a better approach so I want to reserve the right to completely replace ExpressionHelper and remove it from the code-base entirely. I’m not even sure there are any direct tests against ExpressionHelper; it’s probably covered through the higher-level behaviour tests, and that’s the way it should be.

Now, if you want to copy ExpressionHelper out of my code-base and include it in your own source, well, that’s fine, and more power to you. That’s one of the beauties of Open Source. But if I’ve marked something as private or internal, it means I don’t think it’s a required part of the API, and I want to reserve the right to change it, or remove it, as and when the need arises.

An example of when I changed my mind

There is a public method on the Database class called “GetAdapter”. Until a few versions ago, this method was internal. Then Bobby Johnson (@NotMyself) started having some issues with his SQLite provider when using SQLite’s in-memory mode. He needed to be able to get to the underlying ADO connection to do initial setup for tests. My initial reaction was doubt, because my immediate take on what he wanted to do was

“Get the IDbConnection from the Database object”

which would be bad, not least because the Database object doesn’t necessarily have an IDbConnection; it might be using the Mongo adapter, or the Azure Table Service adapter (I’m working on it) or any of the adapters I hope people will write in the future.

But what Bobby actually wanted to do was

“Get the Adapter from the Database, and if it’s an AdoAdapter, get the IDbConnection from that.”

which is fine, and it’s why we have the as keyword in C#. And it wouldn’t just apply to AdoAdapter, there might be other adapters with arbitrary public methods which could be used in this way. This was a case of me being over-private, and I made the change, and I think the API is much better for it.

Tonight’s movie

will be Falling Down. An unemployed defense worker frustrated with the various flaws he sees in society, begins to psychotically and violently lash out against them.

That is all.

Simple.Data does not need POCOs

This is a Public Service Announcement

TL;DR

Simple.Data returns dynamic objects. You don’t have to use statically-typed POCOs unless you want to.

And in full for those without Attention Deficit disorders

I just caught this post on Google+, about using Simple.Data with MongoDB. (I’m really pleased that there’s a MongoDB adapter, because (a) MongoDB is awesome, and (b) it’s the poster child for the fact that Simple.Data is not exclusively an RDBMS-targeting solution. The adapter works really well, although some query operations don’t make sense in MongoDB; I’m working on some code for Core which provides polyfills for any operations which adapters might not be able to interpret meaningfully, such as grouping and aggregates and so forth.)

Anyway, back to the main point of this short post. The comments from Jak Charlton about static typing made me realise that there is potentially a fundamental misunderstanding about what Simple.Data actually does, and a belief that it requires static POCO types to work. It doesn’t, but I haven’t made this clear enough, so sorry about that, and this is me rectifying that situation.

So this misunderstanding probably arises from the fact that most of the demo apps and blog posts that have been written do use POCOs, doing things like this:

User user = db.Users.FindByEmail(email);
VerifyPassword(enteredPassword, user.Password, user.PasswordSalt);

That code works, but in terms of clarity it hides a step in the process. If I split it into two steps, it becomes clearer:

var record = db.Users.FindByEmail(email);
User user = record;
VerifyPassword(enteredPassword, user.Password, user.PasswordSalt);

The first line creates an object of type SimpleRecord, and the type inferred for var is dynamic. (SimpleRecord is very similar to the BCL ExpandoObject type, but with some data-access-specific intelligence.)

The second line performs a kind of “implicit cast” of the dynamic SimpleRecord object to the static User type. (SimpleRecord handles this by overriding the DynamicObject.TryConvert method, if you’re interested.)

The Very Important Thing that I want you to take away from reading this post is that the second line is not required. This code will work every bit as well:

var user = db.Users.FindByEmail(email);
VerifyPassword(enteredPassword, user.Password, user.PasswordSalt);

Here, user will be compiled as dynamic, created as a SimpleRecord instance, and will return values for the Password and PasswordSalt properties when asked. You just won’t get IntelliSense™ for the object.

Other things you can do with these dynamic objects is pass them to your View for rendering, or bind to them in WPF applications. If you don’t want the “statically typed noose dragging you down” (and personally I really don’t), then Simple.Data is your new best friend.

Tonight’s movie

will be Pitch Black. A group of marooned space travellers struggle for survival on a seemingly lifeless sun-scorched world.

That is all.

Progressive .NET Tutorials

Only a month to go now until the 4th Progressive .NET Tutorials at Skills Matter, put together in collaboration with Ian Cooper of the London .NET User Group. They’ve got a packed 3 days with a bunch of excellent speakers giving in-depth tutorials and workshops on some real bleeding-edge technologies and techniques.

OK, so I’m biased. I’m honoured to have been asked to present one of the tutorials, an Introduction to NancyFx and Simple.Data. I’ve presented on this serendipitous pairing of low-ceremony web framework and ultra-light data access library a few times now, and the feedback I get from people is amazingly positive. I’m really looking forward to the opportunity to dig a little deeper in this whole-afternoon workshop. Regular readers of my blog will know that Simple.Data is my project, and the session will be co-hosted by Steven Robbins, aka @Grumpydev, who (with project founder Andreas Håkansson (@TheCodeJunkie)) writes NancyFx, so you’ll be getting the latest information straight from the horses’ mouths.

I’ll be there for all three days of the event. On the Monday morning I’ll either be in Christian Hassa’s and Gasper Nagy’s all-day workshop on Gherkin Acceptance Criteria and the Automation thereof, or watching Ian Cooper and Seb Lambla talking about Package Management in the morning and Dylan Beattie’s tutorial on building great UI, which he’s aiming at people like me who are happier in the guts of an app than up near the shiny things at the top. Yeah, I should probably go to that one. Maybe I’ll learn how to use colours that aren’t blue.

I’m going to have to try and clone myself for Tuesday, because I honestly want to see all four talks. Simon Brown on Load Testing for Developers, Damjan Vujnovic on Test-Driven Development in JavaScript, Paul Stack (a personal show-biz friend of mine) on Continuous Integration and Continuous Delivery, and Adam Granicz on Advanced WebSharper, which if you haven’t heard of it is a very cool web framework based on F#.

For the Wednesday, I definitely want to see if I can get through three and a half hours of Jon Skeet’s in-depth talk on Async in C# 5 without parts of my brain melting, although that means I’ll miss Ian Robinson’s introduction to building RESTful web services with the WCF Web API (which is what Pocket C’#’s servers are running on, btw). I can’t watch Nathan Gloyn’s talk on Agile because he’s on at the same time as me, but I work with Nathan and he knows what he’s talking about, so I promise you that being in either room on Wednesday afternoon will enrich your developer life.

All this awesome cannot be free, sadly, not least because Skills Matter must generate some revenue to help cover all the events they host which are free. But the entry price for three intensive days of learning with real experts in their fields is just £425, and that’s a bargain by any standard. So beg, borrow or steal* the money and sign up through the “book online” link on the event page at Skills Matter: http://skillsmatter.com/event/open-source-dot-net/progressive-dot-net-tutorials-2011/cs-2277

Hope to see you there. I’ll be the one wearing the Statler head.

*Don’t steal. codeface does not endorse or encourage acts of larceny.

Follow

Get every new post delivered to your Inbox.