Turbulent Intellect

You'll POCO Your Eye Out!

2012-01-17T09:00:00.000-06:00

POCO
noun
1. A Plain Old CLR Object that does not derive from a base class, or implement an interface, which is defined as an integration point for a framework.

Victory

Infrastructure frameworks often brag these days about being POCO-compatible. This is in some ways a victory, because it used to be quite the opposite. In order to use an ORM, or an MVC framework, or really anything involving a "model", you needed to inherit some framework base class before it would recognize and respect the role of these objects in your application. But we're past that now. Today you can use POCOs even with frameworks that come directly from Microsoft, like the Entity Framework.

"Convention over configuration" is a similar principle of framework and infrastructure design that has been lauded for a few years now, and is finally hitting more mainstream shops. It trades on the ability to reduce the number of design decision points, allowing you to write less code, and do it faster. In other words, why take all the time and effort to deal with a complicated configuration mechanism capable of expressing myriad options, and then a complicated engine capable of processing all those options, when any one project is only going to use a slice of it all. With convention-based frameworks, just follow some simple rules, and let the framework handle all the details.

Tradeoff

For a long time, I was really happy about this. But today I realized that for all the benefits of simplification and boilerplate elimination, there are some pretty serious tradeoffs being made. Yes, the benefits can be absolutely huge. The gains in consistency, speed, reliability, and flexibility can't be overstated. It saves you from thinking and worrying about a lot of things that are fairly arbitrary in the context of the larger business problem.

Yet for all my appreciation, some shine is starting to come off this penny for one big reason I like to call "context-smearing". Both POCOs and conventions gain their structural homogeneity and rapid development by diluting object responsibilities, and trading away discoverability and expressive design. They take contextual info that ought to be localized, and smear it out between layers, across multiple or even many different locations in code. I am becoming increasingly careful about making this trade-off. I have seen the price it exacts, and I think there are definitely cases where that cost is probably not worth the gain.

Danger

So the big problem with both POCOs and conventions is a loss of context. What's so bad about that? In a word: coupling. Regardless of the absence of tangible framework attachments like base classes and interfaces, your POCOs and conventional objects are still tightly coupled to the framework. But instead of looking at the object and seeing the coupling, it is made almost completely invisible.

When you write a POCO that a framework needs to process, it needs to be simple. Anything too complicated and the framework won't understand it or will make an assumption that doesn't hold, and then the whole thing comes crashing down. Under this pressure your POCOs get dumber and dumber. Unless you're very diligent, before long they are not much more than simple property bags. And a bag of properties is no model. It's just a bunch of data.

With the behavior driven out, there's no context with which to understand the meaning of the data, the relationships, and the references. There's no indication why this property is reproduced over there, or why that container seems to exist solely to hold a single other object. Instead, this knowledge is encoded into all the places where the objects are used. It is distributed and diffused throughout the application, so that in no one place--or even handful of places--can you look to see the true meaning of what that object's role is.

The consequence of this is that points of consumption become fountains of boilerplate and redundancy in your code. At least it's consistent, though. So naturally you'll turn to conventions in order to scrub away some of this homogeneous noise. A good convention-based framework can look at your model, look at your usage, and determine all it needs to know to glue them together. This really streamlines things. But soon enough you've compounded the problem, because conventions actually encourage the exact same diffusion of context. Only conventions grind context down to an even finer sprinkling of dust, and then scatter it on the wind.

It should be clear enough why this is dangerous. It's very easy to slip from here into what John Cook calls "holographic code", which he describes far more elegantly than I have here.

Mitigation

In my opinion, none of this means you should shun POCOs and conventions. Heck, I'm even working on a convention-based Javascript framework of my own called copper.js. They do provide real benefits, especially when the ability to iterate rapidly and share coding and design responsibilities is more important than hi-fi business modeling. Both are highly effective tools in the context of such priorities. And lets be honest, that's the environment that most programmers are working in.

But take the sales pitch with a grain of salt. Conventions will make some of your code simpler, but they can also easily obscure the problem and the solution--or even just the actual model--behind the patterns mandated by the mechanism. An experienced "framework X developer" can see right past that noise. But should we really need to be framework experts just to understand the business problems and solutions? Solid and inviting framework documentation can go a long way in helping people see the signal in all the noise.

Likewise, POCOs are tremendously light-weight, quick, and flexibile. This commonly comes at a cost of design confusion or opacity of purpose when seen separated from their points of consumption. Establishing context is crucial in avoiding confusion and misuse. POCOs should be small and focused, not general and reusable. And they should be located as close as possible to the consumers.

Acceptance

While the problem could get very bad if you're not careful, it's rarely inevitable. And the speed you gain for the 80% case can often be invaluable. You just need to be aware of the pathologies in play, and take steps to mitigate them. If you're lucky, wise decisions on the part of the framework designers can reduce the number of compromises you have to make. But even more important--because it's actually under your control--wise decisions on the part of your development team can contextualize the anemic and opaque bits, limiting their reach.

My advice is to be aware of the problem, and tackle it head-on. Whether by careful coding or clear documentation, be diligent in providing context where the frameworks will tend to suck it away.

WinRT Details: TL;DR version

2011-09-17T15:38:00.000-05:00

Ian Griffiths has written a very deep and informative post about what WinRT really is, and what is meant by all this "projection" vs "native" business. It's a great read, and you should dig in. Especially if you're an old COM hand, like myself.

I have seen a lot of important questions floating around lately (most notably from @kellabyte) that this post answers. But it takes a long time and many diversions before it gets there. So for those who don't have the time or the background to wade through Ian's post, I humbly present a TL;DR bullet list summary.

WinRT appears to be the new Windows API. It is probably the next evolution of Win32.
WinRT is native, and just as fast as native has always been.
WinRT is rooted in the next evolution of COM, not .NET
WinRT involves just as much indirection (and "inefficiency", compared to COM-less native code) as COM always has.
WinRT "projections" are language-specific. For C++, they're native.
WinRT projections do no more and no less than smooth over much of the API for consuming languages, putting a curtain around the pain and noise of working with COM and HRESULTs directly.
WinRT projections are not the next .NET windowing framework. They do not replace WinForms, WPF, Silverlight.

I'll also add to this list my own prediction: We will probably not want to directly consume WinRT any more than we wanted to directly consume Win32. There will be value in a GUI framework to abstract that away, just as there always has been. I haven't heard any mention of the Metro-based successor to WPF/Silverlight, but I would expect one to be delivered with the next version of VS and .NET. I look forward to hearing more from DevDiv now that all the WinRT details are starting to trickle out.

The Limits of Templating as a Declarative View Solution

2011-09-12T10:00:00.000-05:00

I've been playing a little bit lately with AngularJS. Angular is one of a spate of new JavaScript frameworks that provide powerful tools for building interactive web sites and web apps by capitalizing on mature patterns and practices that have served well in the "native client" space. Most of these frameworks provide some sort of separation of concerns between the "view" and the "model".

The "big four" frameworks in this space are AngularJS, KnockoutJS, Backbone.js, and Batman.js. All four of these frameworks are primarily concerned with finding ways to purify and simplify two parts of the architecture of most modern applications: the UI (a.k.a. view, in this case, the HTML), and the model that sources the information in the UI. The primary obstacle in the way of achieving this goal is something called data-binding, which is a terse and useful term for synchronization of the dynamic information displayed in your UI to the state of an object model in your application.

AngularJS goes about this in a strikingly different way from the other three frameworks. The developers of Angular elected to use HTML "templating" as the mechanism of data-binding. Which is to say, you take a "template" of your web page, poke holes in it where you want your dynamic information to go, and then place in those holes small bits of syntax which tell the templating engine how to build the HTML to fill the holes at runtime. Below is a simple example of a template, as it would appear in an application built with AngularJS. If it looks familiar, that's because it's my own modified version of the front page demo from the AngularJS website. Be sure to note the fidelity of the template to the HTML it will produce. It's very clean, and you can see clearly the features of the document in it.

{{todo.task}}

Templating is great for producing a document on demand using data that isn't known until runtime. It's a technique that has been around for quite a while now, and we've mostly figured out some really convenient syntax and conventions for doing it. Heck, ASP.NET MVC is a template-based solution for creating HTML on the fly. It works fantastic. But MVC is not dynamic once that document leaves the server. It builds the HTML once and sends it off to the client, never to revisit it again. The next document it serves up will be a whole new document in response to a whole new request. This is very different from the way Angular uses templating on the client, where the same document may be changed and re-rendered hundreds of times, or more.

Here I feel this solution starts to strain, and this bothers me about Angular as a general solution for web apps. Let me be clear, I think it may be a fantastic way of building static pages, interactive forms with a dash of business logic, or generally anything where there isn't need for complex logic, or generation of large swaths of HTML from nothing. But as pages become more dynamic after the initial rendering, the templates can become bloated and less emblematic of the produced document structure. Today web pages are commonly less about content, layout, and flow, and increasingly about interactivity, dynamism, and tasks. This impacts a templating solution in that expressions expand and dominate, and eventually replace, their HTML content containers. The bulk of logic starts to move further away from the template and deep into javascript and framework internals start to leak out. As an example, see the introduction page on creating your own "custom elements" in AngularJS. We've obviously stepped far away from a "templating" solution here.

For extremely dynamic sites where most of the elements on the page are subject to change in some way or another and very little can be considered "static", I think more conventional solutions are more appropriate. At that point, it becomes appropriate to move logic as much as possible into code, where it can be encapsulated in objects and layers with clear roles and identities. Here it may be prudent to start considering Backbone, Batman, Knockout, or maybe even something else.

For myself, I'd love to see a JavaScript framework that employed MVVM in an unobstrusive way, by treating the HTML document as a passive view which is bound to at run time. I can always drop in Mustache templates for creating HTML in small bits where needed. And "declarative bindings" can just as easily be achieved outside of the HTML, via a nice fluent API. Does anyone know of a framework like that?

Going off the Reservation with Constructor Logic, Part 2

2011-08-08T23:05:00.000-05:00

First I want to say thanks to Alex for the feedback on my previous post. I did eventually manage to clean up my design quite a bit, but couldn't really put my finger on why it was so hard. After reading Alex's comment it occured to me that I hadn't really thought very hard on where the layer boundaries are. I had (and maybe still have) my persistence, my service, and my infrastructure layers somewhat intertwined. I'm ashamed to have been bitten by this considering how much of a proponent of intentional design and layering as I've been. But I was on a spike to tackle some unfamiliar territory, as well as fighting with a bit of an uncooperative helper framework, so it seems I bit off more than I could chew.

The problem I was trying to deal with was a multi-threaded service, with some threads producing data, and others consuming it to store to a DB. Between device sessions, threads, DB sessions, and domain transactions, I had a lot of "resources" and "lifetimes" that needed very careful managing. It seemed like the perfect opportunity to test the limits of the pattern I've been fond of lately, of creating objects whose primary responsibility is managing lifetime. All services which operate within the context of such a lifetime must then take an instance of the lifetime object. This is all well and good, until you start confusing the context scoping rules.

Below is a sample of my code. I'd have to post the whole project to really give a picture of the concentric scopes, and that would be at least overwhelming and at worst a violation of my company's IP policies. So this contains just the skeleton of the persistence bits.

public class DataContext : IDataContext
{
public DataContext(ISessionProvider sessionProvider)
{
this.SessionProvider = sessionProvider.ThrowIfNullArgument("sessionProvider");
}

private ISessionProvider SessionProvider { get; set;}

protected ISessionFacade Session
{
get
{
return this.SessionProvider.Session;
}
}

public virtual IQueryable CreateQuery()
{
// ...
}

public virtual TModel Fetch(object id)
{
// ...
}

public virtual void TrackChanges(TModel model)
{
// ...
}

public virtual void Delete(TModel model)
{
// ...
}
}

public class PointReadingRepository : IPointReadingRepository
{
public PointReadingRepository(IDataContext context)
{
this.DataContext = context.ThrowIfNullArgument("context");
}

protected IDataContext DataContext { get; private set; }

public void Add(PointReading reading)
{
this.DataContext.TrackChanges(reading);
}
}

public class SessionProvider : ISessionProvider
{
public SessionProvider(IUnitOfWork unitOfWork)
{
// ...
}

private ISessionFacade _Session;
public ISessionFacade Session
{
get
{
return _Session;
}
}
}

public class UnitOfWork : IUnitOfWork
{
public UnitOfWork(ISessionManager sessionManager)
{
this.SessionManager = sessionManager.ThrowIfNullArgument("sessionManager");
}

public ISessionManager SessionManager { get; private set; }

public virtual bool HasStarted { get; private set; }

public virtual bool HasFinished { get; private set; }

public virtual bool WasCancelled { get; private set; }

public virtual void Start()
{
// ...
}

public virtual void Finish()
{
// ...
}

public virtual void CancelChanges()
{
// ...
}

public virtual void SubmitChanges()
{
// ...
}

public virtual void Dispose()
{
if (!this.HasFinished)
{
this.Finish();
}
}
}

The important part of this picture is the following dependency chain (from dependent to dependency): Repository->DataContext->SessionProvider->UnitOfWork. What this would indicate is that before you can get a Repository object, you must have a Unit Of Work first. But it's common in our web apps to basically just make the UoW a singleton, created at the beginning of a request, and disposed at the end. Whereas in a service or desktop app the most obvious/direct implementation path is to have different transactions occurring all over the place, with no such implicit static state to indicate where one ends and the next begins. You get a common workflow that looks like this: Create UoW, save data to Repo, Commit UoW. This doesn't work nicely with a naive IoC setup because the UoW, being a per-dependency object now, is created after the Repo, which already has its own UoW. The changes are in the latter, but it's the former that gets committed. So either the UoW must be created at a higher scope, or the Repo must be created at a lower scope, such as via a separate factory taking a UoW as an argument. I'm not super fond of this because it causes the UoW to sort of vanish from the code where it is most relevant.

One final option is that the dependency chain must change. In my case, I realized that what I really had was an implicit Domain Transaction or Domain Command that wasn't following the pattern of having an object to manage lifetime scope. The fact that this transaction scope was entirely implicit, and its state split across multiple levels of the existing scope hierarchy, was one of the things that was causing me so much confusion. So I solved the issue by setting up my IoC to allow one UoW and one Repo per "lifetime scope" (Autofac's term for child container). Then I created a new domain command object for the operation which would take those two as dependencies. Finally the key: I configured Autofac to spawn a new "lifetime scope" each time one of these domain command objects is created.

Again this illustration is a bit simplified because I was also working with context objects for threads and device sessions as well, all nested in a very particular way to ensure that data flows properly at precise timings. Building disposable scopes upon disposable scopes upon disposable scopes is what got me confused.

I have to say that at this point, I feel like the composability of the model is good, now that I've got layer responsibilities separated. But that it feels like a lot is still too implicit. There is behavior that has moved from being explicit, if complex, code within one object, to being emergent from the composition of multiple objects. I'm not convinced yet that this is an inherently bad thing, but one thing is for certain: it's not going to be covered properly with just unit tests. If this is a pattern I want to continue to follow, I need to get serious about behavioral/integration testing.

Going off the Reservation with Constructor Logic

2011-08-05T17:08:00.001-05:00

I'm noticing a strange thing happening as I make classes more composable and immutable. Especially as I allow object lifetime to carry more meaning, in the form of context or transaction objects. This shift has made object initialization involve a lot of "real" computation, querying, etc. Combined with a desire to avoid explicit factories where possible, to capitalize on the power of my IoC container, this has led to an accumulation of logic in and around constructors.

I'm not sure how I feel about so much logic in constructors. I remember being told, I can't remember when or where, that constructors should not contain complex logic. And while I wouldn't call this complex, it certainly is crucial, core logic, such as querying the database or spinning up threads. It feels like maybe the wrong place for the work to happen, but I can't put my finger on why.

It's important to me to be deliberate, and I do have a definite reason for heading down this path. So I don't want to turn away from it without a definite reason. Plus, pulling the logic out into a factory almost certainly means writing a lot more code, as it steps in between my constructors and the IoC container, requiring manual pass-through of dependencies.

I'm not certain what the alternative is. I like the idea of object lifetimes being meaningful and bestowing context to the things below them in the object graph. Especially if it can be done via child containers and lifetime scoping as supported by the IoC container. But it is really causing me some headaches right now.

My guess at this point is that I am doing too much "asking" in my constructors, and not enough telling. I've done this because to do the telling in a factory would mean I need to explicitly provide some of the constructor parameters. But I don't want to lose the IoC's help for providing the remainder. So I think I need to start figuring out what my IoC can really do as far as currying the constructor parameters not involved in the logic, so that I can have the best of both worlds.

Maybe it will bite me in the end and I'll pull back to a more traditional, less composable strategy. Maybe the headaches are a necessary consequence of my strategy. But I think it's worth trying. At least if I fail I will know exactly why not to do this, and I can move on to wholeheartedly looking for alternatives.

Thoughts, suggestions, criticisms? Condemnations?

IoC Containers are Application Composers

2011-05-17T09:30:00.040-05:00

I've been working with Jeremy D Miller's StructureMap IoC container lately. Over the weekend I also did some research on Castle Windsor, in order to properly offer an opinion to Krysztof Kozmic about whether he should remove nested containers from it. Previous to all this, the only IoC I'd really had the opportunity to use on a real project was Autofac. Maybe it's just the bias of first exposure, but I still like Autofac the best. But I can point to one big tangible reason that I like Autofac better, which the other two options don't have: lifetime scopes.

Lifetime scopes are the primary unit of context in Autofac. When the framework has to resolve a dependency, it doesn't pull it from a container. Rather, it pulls it from a lifetime scope. A lifetime scope is different from a container in that you can't change or query its configuration at runtime. It is a purely resolution-oriented interface. But more importantly, in concert with a few other features, it offers a very convenient way to establish what I call "resolution contexts" within your application.

A resolution context is a "bubble" within your application with some or all of its own resolution rules. You can dictate that within a given bubble, certain objects are guaranteed unique, and will be cleaned up when the context is disposed of. Or you can ensure that dependencies are resolvable inside the bubble, but not outside of it. By nesting bubbles that do this, you can enforce inter-layer dependency rules ensuring, for example, that the domain services layer does not depend on the UI layer. You can see the obvious reason that the word "scope" is used in the name: scoping is their explicit purpose.

The features that combine with lifetime scope to make it a truly powerful idiom are tags, and tagged registration. You can tag a lifetime scope with an object that identifies that scope. This value can also be specified in a given type registration to mark it as being unique within a scope having that tag.

Combined, these features allow you to tie instances of objects to a scope, and ensure that anything resolved within that scope gets a particular object. This can be particularly useful for managing things such as database transactions, or domain units of work. Or really any transaction-like operation consisting of a set of sub-operations involving resources which need only, or should only, exist for the duration of the parent operation.

Without this feature, it falls to the developer to explicitly write an object whose responsibility is to manage the lifetime or reach of these things, and another whose responsibility it is to generate and dispose of these scoping objects. This is essentially simple, but for an inexperienced dev it is a tough problem. For an experienced dev, it's annoyingly repetitive.

The thing I love about Autofac is that it supports tagged scopes which make it trivially easy to solve this problem. The thing I hate about Autofac is that you can't define as part of the registration process where you want these nested lifetime scopes to be generated. Instead, you must use a hook, a direct framework reference, a relationship type or some other extension mechanism, to dictate when nested lifetime scopes are introduced.

The reason I have these conflicting feelings is that Autofac approaches, but falls short of what I have come to consider the ideal role of an IoC container: to be the composer of the application. The use to which most of us put IoC containers today diverges from the implications of the name "container". While they do serve as containers in the sense that they hang onto shared (singleton) instances of objects, they also fill a factory role. And with Autofac's nested lifetime scopes, they can draw boundaries around self-contained operations and object graphs. Windsor prides itself on powerful, flexible "release" mechanisms, for cleaning up objects that are no longer needed. Again, this goes far beyond the role of a "container".

Yet despite this, the configuration API of most IoC frameworks still is highly container-oriented. We "register" types and interfaces, attaching bits of metadata here and there as needed. These are "stored" in the container, which can be queried or overridden at run-time by adding or removing registrations from the container.

But an IoC is at its most effective when it is acting not as a container, but as a rules engine, to regulate when new objects are created, and when existing ones are reused. Or how one object is chosen from many that implement a common interface. Or when and how object graphs are disposed of and resources cleaned up. These roles all build toward an overarching theme of "composing" the application. That is to say, building it up out of different blocks of functionality, fitting them together according to a pattern which is clearest when viewed as a whole.

This is why the ServiceLocator block from Microsoft makes me angry. While you can still configure the composition of your application using your IoC of choice, ServiceLocator exposes it as if it were a simple abstract-to-concrete mapping operation. And beyond that, the base idea of a centralized "service locator" completely undermines the goal of IoC in the first place, which is to invert the control. Don't ask for what you think you want. Just expose the dependency declaratively, and let it be provided to you as deemed appropriate by someone whose whole job is to know what you need.

I say we should embrace the composer idiom. I would love to see an IoC framework that took this role to heart, and wore it on its sleeve, so to speak, by expressing it naturally in its API. Autofac comes closest of any container I've used or researched, but it still shows some artifacts of its container heritage. One big improvement that could be made is a declarative configuration-time way of establishing when new scopes should be spun up. I posted a gist to this effect over the weekend and shared it with Krysztof Kozmic, hoping he'd take some inspiration from it for Windsor. But in truth, Windsor also still has a lot of other aspects that are tied to its identity as a "container".

Maybe it's time for someone to take another crack at the IoC problem. The ideal functionality is mostly all available out there. All that's lacking is an API that makes its true role obvious and natural.

Responsibility - The Quantum of Design

2011-05-03T09:30:00.005-05:00

It turns out I really can't bang this drum enough.

This weekend I tweeted three simple rules for determining if a class's responsibility has been properly assigned. Class responsibilities were on my mind for a significant portion of the day on Saturday. It wasn't anything conscious or intentional. Rather I think it was a sort of run-off from a lot of responsibility talk over the course of the previous week.

Lately I find that whenever someone asks my opinion about a particular design decision, the question of responsibility is the first place I go. It's far and away the most effective and powerful thing to ask, as it easily gets the most guaranteed bang for the buck.

At both the class and the method level, the most important factor in deciding where and how to implement a particular bit of functionality is whose responsibility it is. Likely, the whole picture of the task is made up of several bits of responsibility that belong in several places. If your application is architected in such a way that responsibilities are well isolated and assigned to layers and clusters of classes, then breaking the task down into those constituent responsibilities gives you a draft blueprint for how to get the job done.

If your application is not fortunate enough to have such a clear and well articulated architectural scheme... well I'd advise you to make it a goal. Start by learning about onion architecture. But in the meantime, try to keep in mind the rules of thumb below and evaluate your classes against them as you work. If they don't measure up, see if they can be improved incrementally as you work. Leave them cleaner than you left them.

Now, without further ado, the rules.

If you can't sum up a class's responsibility in 2 sentences, shift your level of granularity. It almost certainly has more than one responsibility, and you need to think hard about why you've combined them.
If you can't sum up a class's responsibility in a single sentence, then you have lots of room for improvement. Don't be satisfied until you have a clear, concise statement of the class's responsibility.
If you can sum up a class's responsibility in a single sentence, go a step further and think hard about whether it has another unstated or assumed responsibility. Unit tests can really help here.

The message here is not subtle. Every class needs to have a clear answer to the question "what is my responsibility". This is, in my experience, the most important factor in establishing clean, clear, composable, testable, maintainable, comprehensible application design and architecture. It's not the end of the road, but it's a darn important step on the way.

Don't Stop at DRY

2011-04-19T22:25:00.006-05:00

I've been thinking a lot about the DRY principle lately. You know the one. DRY: Don't Repeat Yourself. It's a principle that was first made popular in the book "The Pragmatic Programmer". Unlike much of the rest of the content of the book, DRY is something that appears to have penetrated fairly deeply into the collective consciousness of the greater programming community. In a certain respect, this is a huge success on the part of Andy Hunt and David Thomas. To have disseminated so completely such a fundamental rule of programming is surely something to be proud of.

Note what I just said. DRY is "a fundamental rule of programming." I suspect very few people with appreciable experience would reasonably and honestly dispute this claim. But a rule is a funny thing, and a fundamental one doubly so. Rules are a first-pass at quality. They establish a minimum level of ensured quality by being a reasonable course of action in the vast majority of cases, at the expense of being a sub-optimal course of action in many cases, and even a plain bad option in some.

I am by no means saying that DRY is a bad thing. I'm not even saying not to do it. But I am saying that applying DRY is a beginner's first step toward good design. It's only the beginning of the road, not the end.

Eliminating repetition and redundancy is a very natural and obvious use for programming. It's especially hard to deny for so many people who slide into the profession by way of automating tedious and repetitious manual labor. It just makes sense. So when you tell a new programmer Don't Repeat Yourself, they embrace it whole-heartedly. Here is something that a) they believe in, b) they are good at, and c) will automatically achieve a better design. You'd better believe they are going to pick up that banner and wave it high and proud.

This is a good thing. If you can get your team to buy in to DRY, you know you'll never have to worry about updating the button ordering on that dialog box only to find that it hasn't changed in 3 scenarios because they're using a near identical copy in another namespace. You know that you'll never have to deal with the fallout of updating the logic that wraps up entered data for storage to the database and finding inconsistent records a week later because an alternate form had repeated the exact same logic.

What you might run into, however, is:

A method for rendering currencies as user-friendly text which is also used to format values for XML serialization. The goal being to keep all text serialization centralized regardless whether it's meant for display or storage.
Views, controllers, configuration, and services all using the same data objects, regardless of the dramatically different projections necessary for every operation. This in the interest of avoiding redundant structures.
You might find that your views, controllers, and persistence layers all depend directly on a single tax calculation class. Of course here the goal is very simply to centralize the business logic, but actively works against establishing a proper layering in the application.

These are all very sub-optimal, or even bad, design decisions. They are all examples that I have seen with my own eyes of decisions made in the name of DRY. But DRY itself is not the cause. The problem is that the people responsible for these decisions have unknowingly pitted DRY against other quality design rules and principles.

DRY focuses on behavior. That is to say algorithms. "Algorithms", along with "data", are the fundamental building blocks of applications. They are the substance our work is made of. But trying to build complex software while thinking only or primarily at this elementary level is like trying to map a forest one tree at a time.

At some point, the programmer must graduate from the rule of DRY to the nuance of responsibility. The Single Responsibility Principle (SRP) is the cardinal member of the SOLID family. It's premier position is not simply a coincidence of the acronym. It's also the next most important rule to layer onto your understanding of good design.

Responsibility is about more than function. It's also about context. It's about form and purpose. Identifying and isolating responsibilities allows you to take your common DRY'ed functionality, and cast it through various filters and projections to make it naturally and frictionlessly useful in the different places where it is needed. In a very strict sense, you've repeated some functionality, but you've also specialized it and eliminated an impedance mismatch.

Properly establishing and delimiting responsibilities will allow you to:

Take that currency rendering logic and recognize that readability and storage needs present dramatically different contexts with different needs.
See that the problem solved by a data structure is rarely as simple as just bundling data together. It also includes exposing that data in a form convenient to usage, which may vary dramatically from site to site.
Recognize that calculation is worth keeping DRY, but so is the responsibility to trigger such calculation. It can be both triggered and performed in the business layer, and the result projected along with contextual related business data to wherever it needs to be.

By layering responsibility resolution on top of the utilitarianism of DRY, you take a step back from the trees and can begin to manage a slightly wider view of the problem and solution spaces. This is the key and crucial lesson that all beginning programmers must learn, once they've mastered the art of DRY. Once again, DRY is fundamental and indispensable. It's one step along the path to wisdom and, in general, should not be skipped or omitted. But it's not an end in itself. So don't stop when you get DRY. Start looking for the next step. And consider responsibility resolution as a strong candidate.

Mindfulness

2011-04-12T09:30:00.001-05:00

You are faced with an unfamiliar coding problem. You know what you're "supposed to do"--use this pattern, follow that convention--but you don't know why. And the guidance seems ever so slightly... off. The context is just a bit different. The constraints are a bit tighter here. A bit looser there. The peg doesn't quite fit in the hole, but everyone is telling you it should.

Everyone is telling you "this is how you do it" or "this is how we've always done it". You think you can follow the rule, but you don't think it will be pretty. What do you do?

I have witnessed many people in this situation make their choice. Most commonly they do whatever it takes to follow the rule. If the peg doesn't fit, they pound it till it's the right shape. If the cloth won't cover, they'll stretch, fold, and tear creatively until they can make a seam. Somewhat less often, they just toss out the advice and do it the way they are comfortable with. Often that just means "hacking out" an ad hoc solution.

In either case, the person learns nothing. They will repeat their struggle the next time they face a situation that is ever so slightly different. They will sweat, stress, and hack again. Over and over, until someone shows them a new rule to follow.

As students we are taught to learn first by rote, then by rule. If we are lucky, we are eventually tested on whether we have learned to think. And most commonly we manage to slog it out and do well enough to pass without actually being required to think. It is so very easy to become comfortable with this model of learning. A particular someone, vested with the responsibility of fertilizing our minds and nurturing the growth of understanding, knows the answers. We don't understand, but someone can at least tell us when we are right or wrong, and often that's enough.

We settle into dependence. And by settling, we establish roadblocks in our path before we even set foot on the road. By settling, we refuse to take ownership of our own knowledge. We put ourselves at the mercy of those who know more and are willing to share of their time and understanding to help us overcome obstacles of our own creation.

This situation is not inevitable. You can avoid dooming yourself to toil under it with a very simple determination. But make no mistake, this simple determination will require determination. When you face the prospect of doing something that you do not understand, stop, take note, and ask "Why?" Refuse to continue on with any course of action until you know why you are doing it.

I'm not talking about questioning authority here (though that's good too). I am advocating understanding. If you think there's any chance you may have to face a similar situation again, then as a professional developer it behooves you to understand what you're doing this time, and why. This prepares you firstly to defend your actions, and secondly to tackle similar but different problems in the future.

By recognizing the reasoning behind a particular prescribed course of action, when you encounter a similar situation in the future you will be able to identify the subset of the problem that is subject to the prescription. Seeing this allows you to conceptually distinguish that part of the problem from the rest. From this vantage point you can decide whether the remainder is just noise that has to be accommodated, or something more significant. You will be able to start to consider whether there is another layer or dimension to the problem which might be better served by a different or additional pattern. You will be able to think, intelligently, and intentionally, about the problem both as a whole, and in part.

Lack of mindfulness is the scourge of intellectual pursuits (and a great many other things in life). Whether in programming, in health, in investment, etc., it binds you to the service of rules and systems. It puts you at the mercy of those who have understanding, and under the thumb of those who own the systems. Benevolent or otherwise, do you really want your own success and satisfaction to come at the whim of someone else, for no other reason than that you couldn't be bothered to put in the effort to understand? Do you want to spend your career tromping the same grounds over and over again, never coming to any familiarity or understanding of the landscape?

Always ask, "Why?" Then take the time to understand. Always be deliberate and intentional about applying any solution. Don't just follow the directions of the crowd, or some authority. You're not a patient taking orders from your doctor. This is your domain. Own your knowledge. Your future self will thank you.

Take "Single Responsibility" to the Next Level

2011-04-05T10:25:00.001-05:00

The Single Responsibility Principle (SRP) is a crucial tool in your toolbox for managing complexity. Bob Martin has a great essay on the Single Responsibility principle which expresses one of the biggest benefits that it can deliver to you. The SRP is predicated upon the unfortunate reality that changing code is the biggest source of bugs. By extension, the easiest way to avoid bugs is to ensure that whenever you have to make a change, it affects as little of the code as possible.

This is well known among experienced developers, but as Martin notes at the end of his essay, it's extremely difficult to get right. In my experience, very few devs take the principle as far as they should. Especially considering the fact that most of us were taught that Object-Oriented Design is all about bundling up code that works together, it can be easy to get lulled into confidence about what it takes to truly adhere to SRP.

Martin says, "conjoining responsibilities" comes naturally to us. Programmers are finely-honed pattern matching machines. It often comes far too easily. Most of us begin our careers weaving responsibilities and dependencies throughout our code, in the interest of solving a problem as quickly and efficiently as possible... with bonus points for being clever. Then we start learning the SOLID principles and start to adjust our coding habits. We take SRP to heart and break up responsibilities along reasonable lines, even if it means writing a bit more code, and synchronizing state that could theoretically be shared if all things were equal.

Then we stop.

We move our logging out of exception handlers and into service classes. But we leave severity conditions and entry formatting mixed in with the filestream management. We separate our business logic from our UI rendering and input handling. But we pile up unrelated domain operations in our controllers and presentation models. We shrink our class sizes from 1500 lines to 500 and claim victory.

This is not enough.

For one thing, responsibilities that often seem naturally elemental can usually be broken down yet further. The log example is a perfect one. File stream management is a responsibility complex enough to be given its own class. Text formatting is yet another. And severity handling is something that can be configured apart from the other two aspects. Oh, and don't forget that interfacing with an unmockable framework resource class such as System.IO.Filestream is a worthy role all its own. Each of these is a responsibility that can be wrapped up in a small class of just 100 or so lines, and exposed with a simple fine-grained interface of just a few methods. Compose these together and you have a logging service that's flexible, highly testable in all aspects, and most importantly, can evolve and be maintained independently along several orthogonal dimensions, without interfering with the other functionality. And on top of all this, you get the automatic benefit that it's much more friendly to dependency injection and unit testing.

The other important lesson to learn is that SRP doesn't just apply to classes but also to methods. It's almost never necessary for a method to be more than 30 or so lines long, on the outside. A method of such restricted size will inevitably have fewer arguments to worry about, for starters. And further, it almost inherently prevents spaghetti code. Purely by the act of breaking out small operations of a handful of steps apiece, and giving each a highly specific and expressive name, you can avoid the "flying V" of nested loops and conditionals. You can avoid long-lived context variables and status flags. You can avoid stretching tightly coupled cause and effect relationships across multiple screens-worth of code. And you'll likely find yet more places to naturally slice up a class into separate responsibilities.

All of these help to control sprawling complexity. This sounds unintuitive, because if you follow the rough rule of 30 lines per method, 200 lines per class, you'll almost certainly end up writing far more classes, and probably more code in general. But you will always know exactly where to find code related to any particular thing that may go wrong. And you can always be certain that the portion of the application that is affected by a change to a particular unit of functionality will be highly constrained by virtue of the reduced number of interactions and dependencies that any one unit needs to worry about.

Consider what you can do to take the next step with SRP. Don't be satisfied with the first-order effects of a naive application of the principle. Rededicate yourself, try it out, shrink your units, and see the benefits in your code.

A Taxonomy of Test Doubles

2011-03-29T09:30:00.014-05:00

Many words have been written in the TDD community about the myriad ways of mocking in service of unit tests. After all this effort, there remains a great deal of confusion, ambiguity, in the understanding of many--maybe even most--developers who are using mocks.

No less than the likes of the eminently wise Martin Fowler has tackled the subject. Fowler's article is indispensible, and it in large part built the foundation of my own understanding of the topic. But it is quite long, and was originally written several years ago, when mocks were almost exclusively hand-rolled, or created with the record/replay idiom that was popular in mocking frameworks before lambdas and expressions were added to C# and VB.NET with Visual Studio 2008. Add to that the fact that the article was written in the context of a long-standing argument between two different philosophies of mocking.

Unfortunately these arguments continue on even today, as can be seen in the strongly-worded post that Karl Seguin wrote last week. Looking back now, with several more years of community experience and wisdom in unit testing and mocking behind us, we can bring a bit more perspective to the discussion than what was available at that time. But we won't throw away Fowler's post completely. Within his post, there are firm foundations we can build on, in the definitions of the different types of mocks that Fowler identified.

There are four primary types of test doubles. We'll start with the simplest, and move through in order of ascending complexity.

Dummies

A dummy is probably the most common type of test double. It is a "dumb" object that has no real behavior. Methods and setters may be called without any exception, but without any side-effect, and getters will return default values. Dummies are typically used as placeholders to fill an argument or property of a specific type that won't actually be used by the test subject during the test in question. While a "real" object wouldn't actually be used, an instance of a concrete type may have strings attached, such as dependencies of its own, that would make the test setup difficult or noisy.

Dummmies are most efficiently created using a mock framework. These frameworks will typically allow a mock to be created without actually configuring any of the members. Instead they will provide sensible defaults, should some innocuous behavior be necessary to satisfy the subject.

Stubs

A stub is a test double which serves up "indirect input" to the test subject. An indirect input is information that is not provided to an object by the caller of its methods or properties, but rather in response to a method call or property access by the subject itself, to one of its dependencies. An example of this would be the result of a factory creation method. Factories are a type of dependency that is quite commonly replaced by a stub. Their whole purpose is to serve up indirect input, toward the goal of avoiding having to provide the product directly when it may not be available at the time.

Stubs tend to be quite easy to set up even with more primitive mocking frameworks. Typically, all that is needed is to specify ahead of time the value that should be returned in response to a particular call. The usual simplicity of stubs should not be taken as false comfort that the doubles are not too complicated, however. Stubs can get quite complex if they need to yield a variety of different objects multiple calls. The setup for this kind of scenario can get messy quick, and that should be taken as a sign to move on to a more complex type of double.

Mocks

A mock is a type of test double that is designed to accept and verify "indirect output" from the subject class. An indirect output is a piece of information that is provided by the test subject to one of its dependencies, rather than as a return value to the caller. For example, a class that calls Console.WriteLine with a message for printing to the screen is providing an indirect output to that method.

The term "mock" for a particular type of test double is in a certain way unfortunate. In the beginning there was no differentiation. All doubles were mocks. And all the frameworks that facilitated easy double creation were called mocking frameworks. The reason that "mock" has stuck as a particular type of double is because in those beginning times, most test doubles tended to take a form close to what we today still call a "mock". Mocks were used primarily to specify an expectation of a particular series of method calls and property access.

These "behavioral mocks", or "classical mocks" as Fowler calls them, gave birth to the record/replay idiom for mock configuration that reached its peak in the days of RhinoMocks. And due to the tendency of inexperienced developers to create complicated object interactions and temporal coupling, mocks continue to be a very popular and common form of test double. Mocking frameworks make it far easier to unit test classes that rely on these types of coupling. This has led many to call for the abolishment of mocks and mocking frameworks in a general sense, claiming that they provide a crutch that makes it too easy to leave bad code in place. I'm sympathetic to the sentiment, but I think that this is throwing the baby out with the bathwater.

Fakes

Fakes are the most complicated style of test double. A fake is an object that acts simultaneously as both a stub and a mock, providing bidirectional interaction with the test subject. Often fakes are used to provide a substantial portion of the dependency's interface, or even all of it. This can be quite useful in the case of a database dependency, for example, or a disk storage service. Properly testing an object that makes use of storage or persistence mechanisms often requires testing a full cycle of behavior which includes both pushing to and pulling from the storage. An in-memory fake implementation is often a very effective way of avoiding relying on such stateful storage in your tests.

Given their usefulness, fakes are also probably the most misused type of test double. I say this because many people create fakes using a mocking framework, thinking they are creating simple mocks. Or worse, they knowingly implement a full-fledged fake using closures around the test's local variables. Unfortunately, due to the verbosity of mocking APIs in static languages, this can very easily become longer and more complex code than an explicit test-specific implementation of the interface/base class would be. Working with very noisy, complicated, and fragile test setup is dangerous, because it's too easy to lose track of what is going on and end up with false-passes. When your test's "arrange" step starts to overshadow the "act" and the "assert" steps, it's time to consider writing a "hand-rolled fake". Hand-rolled fakes not only remove brittle and probably redundant setup from your tests, but they also often can be very effectively reused throughout all the tests for a given class, or even multiple classes.

It's not Just Academic

These are the primary categories into which nearly all, if not all, test doubles can be grouped. Fowler did a great job of identifying the categories, but I think this crucial information is buried within a lot of context-setting and illustration that doesn't necessarily offer great value today. Mocking is ubiquitous among the subset of developers that are doing unit testing. But too many people go about unit testing in an ad hoc fashion, rather than deliberately with a plan and a system for making sense of things. I believe that a simple explanation of the major types and usages of test doubles, as I've tried to provide here, can aid greatly in bringing consistency and clarity of intent to developers' unit tests. At the very least, I hope it can instill some confidence that, with a little discipline, pattern and reason can be found in the often messy and overwhelming world of unit testing.

YAGNI Abuse

2011-03-22T09:30:00.044-05:00

Have you ever proposed a code change or a course of action in a project for the purpose of improving the stability and maintainability of the code base, only to have someone dispute the need on the basis of YAGNI? I was flummoxed the first time this happened to me. Since then I've learned that it's not at all rare, and in fact may even be common.

The YAGNI principle is a wonderful thing. Used properly, it can have a huge beneficial impact on your productivity, your schedule, and on the maintainability of your product. But like so many other important ideas in the history of software development, YAGNI has become a poorly understood and misused victim of its fame. Through constant abuse it has become difficult to communicate the sentiment that it was intended for without a thorough explanation. And I can't count the number of times I've heard YAGNI cited in a completely incorrect or even dangerous way.

The term "YAGNI" has fallen prey to a similar disease as "agile". People often invoke it as an excuse not to do something that they don't want to do. Unfortunately, this quite often includes things that they should do. Things that have long constituted good design, and good software engineering practice. A few examples of things that I have personally been horrified to hear disputed on YAGNI grounds include:

These are all activities that are strongly valued and diligently practiced in the most productive, successful, and small-A-agile software development organizations and communities. For myself and many of you out there, it's patently obvious that this is a subversion and abuse of the YAGNI strategy. Your first instinct in response to this kind of misuse is to say, with no little conviction, "that's not what YAGNI means."

This, of course, will not convince anyone who has actually attempted to use the YAGNI defense to avoid good engineering practices. But to refute them, you needn't rely solely on the forceful recitation of the principle as they do. Fortunately for us, YAGNI is not an elemental, indivisible tenet of software engineering. It did not spring fully-formed from Ron Jeffries' head. Rather it is based in experience, observation, and analysis.

It is clear from reading Jeffries' post on the XP page that the key to sensible use of the YAGNI principle is remembering that it tells you not to add something that you think you might or probably will need, or even certainly will need, in the future. YAGNI is a response to the urge to add complexity that is not bringing you closer to the immediate goal. Particularly common instances of true YAGNI center on features that haven't been identified as either crucial or wanted, such as configurability, alternative logging mechanisms, or remote notifications.

Looking at my original list, it is clear that none of these things truly add complexity in this way. A very naive metric of complexity such as "number of code entities" may seem to indicate the opposite. But these are actually all established and reliable methods for controlling complexity. What these techniques all have in common is that they restrict the ways in which parts of your program are allowed to interact with other parts of your program. The interaction graph is the most dangerous place for complexity to manifest in a program because it compounds the difficulty of changing any one part of the application without affecting the rest of it like a row of dominoes. The practices I identified above, which are so often refuted as "adding complexity", are some of the many ways to guide your application toward this:

And away from this:

There is a multitude practices, patterns, and design principles that help keep your modules small, their scopes limited, and their boundaries well-defined. YAGNI is one of them, but not the only one. Claiming YAGNI to avoid this kind of work is "not even wrong". Not only are you gonna need it, but you do need it, right from day one. Working without these tools is seeding the ground of your project with the thorns and weeds of complexity. They provide you with a way to keep your code garden weed-free. In this way they are kin to YAGNI, not its enemy. Claiming otherwise reveals either a disrespect for, or a lack of understanding of, the benefits of good design and engineering practices in a general sense. So next time someone sets up this contradiction in front of you, don't let them get away with it. Show your knowledge, and stand up for quality and craft.

Retrospective on a Week of Test-First Development

2011-03-15T09:30:00.005-05:00

Any programmer who is patient enough to listen has heard me evangelizing the virtues of Test-Driven Design. That is, designing your application, your classes, your interface, for testability. Designing for testability unsurprisingly yields code which can very easily have tests hung onto it. But going beyond that, it drives your code to a better overall design. Put simply, this is because testing places the very same demands on your code as does incremental change.

You likely already have an opinion on whether that is correct or not. In which case, I'm either preaching to the choir, or to a brick wall. I'll let you decide which echo chamber you'd rather be in, but if you don't mind hanging out in the pro-testability room for a while, then read on.

Last week I began a new job. I joined a software development lab that follows an agile process, and places an emphasis on testability and continuous improvement. The lead architect on our development team has encouraged everyone to develop ideally in a test-first manner, but I'm not sure how many have taken him up on that challenge. I've always wondered how well it actually works in practice, and honestly, I've always been a bit skeptical of the benefits. So I decided this big change of environment was the perfect opportunity to give it a shot.

After a week of test-first development, here are the most significant observations:

Progress feels slower.
My classes have turned out smaller, and there are more of them.
My interfaces and public class surfaces are much simpler and more straightforward.
My test have turned out shorter and simpler, and there are more of them.
I spent a measurable amount of time debugging my tests, but a negligible amount of time debugging the subject classes.
I've never been so confident before that everything works as it is supposed to.

Let's break these out and look at them in detail.

1. Progress feels slower.

This is the thing I worried most about. Writing tests has always been an exercise in patience, in the past. Writing a test after writing the subject means sitting there and trying to think about all the ways that what you just wrote could break, and then writing tests for all of them. Each tests include varying amounts of setup, and dependency mocking. And mocking can be tough, even when your classes are designed with isolation in mind.

The reality this week is that yes, from day to day, hour to hour, I am writing less application code. But I am re-writing code less. I am fixing code less. I am redesigning code less. While I'm writing less code, it feels like each line that I do write is more impactful and more resilient. This leads very well into...

2. & 3. My classes have turned out smaller, and there are more of them.
My interfaces and public class surfaces are much simpler and more straightforward.

The next-biggest worry I had was that in service of testability, my classes would become anemic or insipid. I thought there was a chance that my classes would end up so puny and of so little presence and substance that it would actually become an impediment to understandability and evolution.

This seems reasonable, right? Spread your functionality too thin and it might just evaporate like a puddle in dry heat. Sprinkle your functionality across too many classes and it will become impossible to find the functionality you want.

In fact the classes didn't lose their presence. Rather I would say that their identities came into sharp and unmistakable focus. The clarity and simplicity of their public members and interfaces made it virtually impossible to misuse them, or to mistake whether their innards do what they claim to. This enhances the value and impact of the code that consumes it. Furthermore it makes test coverage remarkably achievable, which is something I always struggled with when working test-after. On that note...

4. My tests have turned out simpler, and there are more of them.

The simple surface areas and limited responsibilities of each class significantly impacted the nature of the tests that I am writing, compared to my test-after work. Whereas I used to spend many-fold more time "arranging" than "acting" and "asserting", the proportion of effort this step takes has dropped dramatically. Setting up and injecting mocks is still a non-trivial part of the job. But now this tends to require a lot less fiddling with arguments and callbacks. Of course an extra benefit of this is that the test are more readable, which means their intent is more readily apparent. And that is a crucial aspect of effective testing.

5. I spent a measurable amount of time debugging my tests, but a negligible amount of time debugging the subject classes

There's not too much to say here. It's pretty straightforward. The total amount of time I spent in the debugger and doing manual testing was greatly reduced. Most of my debugging was of the arrangement portions of tests. And most of that ended up being due to my own confusion about bits of the mocking API.

6. I've never been so confident before that everything works as it is supposed to.

This cannot be overstated. I've always been fairly confident in my ability to solve problems. But I've always had terrible anxiety when it came to backing up correctness in the face of bugs. I tend to be a big-picture thinker when it comes to development. I outline general structure, but before ironing out all the details of a given portion of the code, I'll move on to the interesting work of outlining other general structure.

Test-first development doesn't let me get away with putting off the details until there's nothing "fun" left. If I'm allowed to do that then by the time I come back to them I've usually forgotten what the details need to be. This has historically been a pretty big source of bugs for me. Far from the only source, but a significant one. Test-driven design keeps my whims in check, by ensuring that the details are right before moving on.

An Unexpected Development

The upshot of all this is that despite the fact that some of the things I feared ended up being partially true, the net impact was actually the opposite of what I was afraid it would be. In my first week of test-first development, my code has made a shift toward simpler, more modular, more replaceable, and more provably correct code. And I see no reason why these results shouldn't be repeatable, with some diligence and a bit of forethought in applying the philosophy to what might seem an incompatible problem space.

The most significant observation I made is that working like this feels different from the other work processes I've followed. It feels more deliberate, more pragmatic. It feels more like craft and less like hacking. It feels more like engineering. Software development will always have a strong art component. Most applied sciences do, whether people like to admit it or not. But this is the first time I've really felt like what I was doing went beyong just art plus experience plus discipline. This week, I feel like I moved closer toward that golden ideal of Software Engineering.

Strategy vs. Tactics in Coding Standards

2011-03-08T09:30:00.000-06:00

I'm starting a new job on March 7. As I wrapped up my time with the company where I've spent the past 3 years, I spent some time thinking about the decisions and responsibilities that I was given, and that I took on, while there. One of the reasons I was hired was to bring to the development team my more formal education and professional experience as a software developer. And one of the responsibilities I was allowed was to lead regular discussions with the other developers. The goal of these discussions was to help our team to become more professional in the way we developed software. Of course it was only a matter of time before someone on the team brought up coding standards.

I've had enough experience with coding standards to have developed some skepticism to the way that they are typically applied. It sounds good, in theory. Bring some consistency, order, and predictability to the source code produced by the group. But in my experience, the practice rarely delivers on the promise. It's easy to start a fight on this topic, of course. It tends to be a "religious" topic in that people have strong and entrenched opinions, and are rarely shaken from them regardless of argument. This is unfortunate, because I think there is a way to make them work. But it requires thinking about coding standards in a different kind of way.

The challenge of establishing standards, and the reason standards tend to be so fractious, is that "what works" must necessarily vary depending on the case. But very rarely does an organization who establishes coding standards allow for much variance in response to new problems or process friction. To figure out why this is, and how we can deliver on some of the promise of standards without resorting to brainwashing, let's take a closer look at the types coding standards we see in the wild.

In my experience, coding standards tend to fall into one of two broad categories. There are strategic coding standards, which are general and outcome-oriented. And there are tactical coding standards, which are specific and mechanics-oriented.

I'll address the latter first, as these are the kinds of standards that tend to start religious wars. Here are some examples of tactical coding standards:

All method headers must include a description, pre-conditions, side-effects, argument descriptions, and return value description.
Use C#'s "var" keyword only for variables of anonymous data types.
Place each method argument on its own line.

At this point you're probably at least nodding your head in recognition. Maybe some of you are nodding in satisfied approval, while others are holding back an outburst of disgust. When most people hear the words "coding standards", the tactical ones are the ones they think of. And these often evoke strong emotional responses. Their most common adherents tend to be managers or project leads, and often regular developers who've been around the block a few times.

The reason people want tactical coding standards is simple. They bring a modicum of consistency to a wild and woolly world. You often can't trust that the guy writing code in the next cube can code his way out of a paper bag. So when you have to take over his code after he's gone it's nice to know that, at the very least, he followed the standards. If all else fails, there is one thing that will provide sure footing as you plumb the dangerous depths of their code.

I understand this, and I sympathize. But at the same time, I chafe under these standards. They feel like micromanagement to me. It makes me feel like my coworkers don't trust me to do the thing that I was hired to do. There are organizations where this can't be taken for granted. But my own personal response to those environments is to leave them. I want to work with people I can trust at this level, and that can trust me the same.

It's not all bad for tactical standards, and we'll circle back in a bit. But for now let's look at some examples of strategic coding standards.

Write informative class headers.
Endeavor to keep methods to less than 25 lines in length.
Use whitespace and indenting to identify related code.

Strategic coding standards are broad. They do not tell the developer exactly what to write. Rather they set an expectation of the kind of code that should be written, and leave it to the developer as to how exactly that kind of code should be created. In my experience, these are the kinds of standards that are okay to mandate and enforce from a management perspective. Evoke the general feel of the code you want your team to produce, and then let them use their skill, experience, and professionalism to decide how to make it happen.

You can probably tell that in general I feel a lot better about this type of standard than the other. For one, they stay out of your way on a minute-to-minute basis. They're also far easier to get people to agree upon. Rather than being a cynical defensive measure against bad code, they empower the developer while fostering good coding habits.

Now having said all that, I do think there is a place for tactical standards. One big reason that I named these categories as I did is because I think that the role of establishing each type of standard can map to the organization similarly to how the roles map in the military, from which these terms are taken. Strategic standards can be mandated by leadership without stifling the developers. While the particular tactics can be determined by the people "on the ground".

There are many cases where tactical coding standards are beneficial. But it almost always serves the organization better to let the developers establish and enforce them. Take a team with multiple people all sharing equally in ownership of a codebase, each working from day to day on code written by the others. Imagine further that the code is sensitive in some way that can't abide regression bugs or the like. Maybe there is also a huge amount of code, or a large number of coders.

In these types of situations, it's crucial that the developers be able to pick up a strange piece of code and get a quick picture of what's going on. And to be able to modify the code without introducing a foreign-looking island of code to trip up the next guy. To do this, the developers have to be comfortable with the low-level style aspects of the code, both to read and to write. The best hope a team has of developing this comfort is to, as a self-directed group, come to agreement on common-denominator style standards over time and through shared experience.

There is a balance to be struck here. We want to ensure quality output from our teams, but we don't want to turn our developers into code-generators that don't take responsibility for their own work product. I think this balance is best struck by the proper application of each type of coding standard, in the proper context. If leaders can establish strategic standards that express high-level goals for clean code and understandable designs, then they can empower their teams to find solutions to the lower-granularity challenges such as understanding each other, and reducing the friction involved in cooperation on a shared codebase. This is strategy and tactics in practice, and at its best.

Anatomy of a Windows Service - Part 4

2011-03-01T16:30:00.005-06:00

Deciding how to trigger the work of the service usually comes down to some sort of threading solution. And threading in Windows services is, as everywhere else, a thorny issue. The most basic need we have is to be able to respond to orders from the Service Control Manager (SCM) without blocking processing at an inconvenient spot. Beyond that, we may want to have a bit of work kicking off periodically, with a short period, or we may want to respond to some environmental event.

Responding to environmental events is fairly easy when the framework provides a callback mechanism for it, such as the FileSystemWatcher. But if there's no such facility, then you're stuck with polling, which is in the same boat as the guy who needs to do work on a quick periodic schedule. This leads us to worker threads, and all the headaches that go along with that.

I have no intention of getting deep into threading concepts and strategies here. Not least because I am woefully uninformed and inexperienced with them. If you need to be highly responsive, highly predictable, or highly parallel, that's an entirely new problem space. So it will have to suffice to say here that you should not just hack something together quickly. Do your research and do it right.

What I have done in the past is to use a Thread.Timer to fire a callback on my desired period. But I use a flag to make sure multiple work units do not process in parallel, and I don't worry about missing a period by a nanosecond simply because the work completed just after I checked the flag. Instead I just wait for the next one. This limits the responsiveness of my service, but it's simple and reliable because it sidesteps the issues that arise out of mutually writeable shared state and truly parallel work.

For our sample project here, I introduced this mechanism by creating a worker class that manages the timer. Let's take a look at that.

using System;
using System.Threading;

namespace HelloSvc
{
public class GreetServiceWorker : IGreetServiceWorker
{
protected virtual IGreeter Greeter
{
get;
private set;
}

public GreetServiceWorker(IGreeter greeter)
{
Greeter = greeter.ThrowIfNull("greeter");
}

protected virtual Boolean Running
{
get;
set;
}

public virtual Boolean Paused
{
get;
protected set;
}

protected virtual Timer GreeterTimer
{
get;
private set;
}

protected virtual void TimerPulse(Object state)
{
if (!Paused && !Running)
{
Running = true;

try
{
Greeter.SayHello();
}
finally
{
Running = false;
}
}
}

protected virtual Int32 IntervalMilliseconds
{
get { return 5000; }
}

public virtual Boolean Started
{
get { return GreeterTimer != null; }
}

public virtual void Start()
{
if (GreeterTimer == null)
{
GreeterTimer =
new Timer(
TimerPulse,
null,
IntervalMilliseconds,
IntervalMilliseconds);
}
}

public virtual void Pause()
{
Paused = true;

// Stop the timer from firing.
GreeterTimer.Change(Timeout.Infinite, Timeout.Infinite);
}

public virtual void Continue()
{
Paused = false;

// Reset the timer to normal.
GreeterTimer.Change(IntervalMilliseconds, IntervalMilliseconds);
}

public virtual void Stop()
{
// Stop the timer from firing.
GreeterTimer.Change(Timeout.Infinite, Timeout.Infinite);

// Dispose the timer, waiting for current iteration to
// finish. We don't want the GC
// to clean up the worker while it's still working.
var waitHandle = new AutoResetEvent(false);
GreeterTimer.Dispose(waitHandle);
waitHandle.WaitOne();

GreeterTimer = null;
}
}
}

Note the flag properties, which are used as a guard on the timer event to make sure that processing doesn't re-enter before the previous iteration completes. The timer interval is set to start things rolling, and cleared to stop them. The added complexity in the Stop method is crucial to understand as well. Timers are tricky to get rid of without blocking the thread. We use a wait handle to do it. Wait handles are worth reading up on if you deal with threading at all. Another nice feature of this worker class is that an object of this type can be reused: Start can be called to spin things up again after Stop has finished.

The worker presents essentially the same interface as the service itself, which may seem a bit redundant. All I am really trying to achieve by this is to respect the separate responsibilities of management of service as a whole, as opposed to management of the work timer and triggers. The service itself is just processing events from the SCM. It need not know, when it tells the worker to start, whether the worker is firing up a new thread to get started, waiting for the moment the current one finishes, or just taking a pass on this period.

The service class will now delegate to the worker. This is very straightforward. I just had to replace the IGreeter property with a property for the worker. Now in OnStart, we spin up the worker, and in OnStop we finally have some code as well, to spin down the worker.

internal class GreetService : ServiceBase
{
protected virtual IGreeter Greeter
{
get;
private set;
}

protected virtual ILogger Logger
{
get;
private set;
}

public GreetService(IServiceNameProvider serviceNameProvider, ILogger logger, IGreetServiceWorker worker)
{
serviceNameProvider.ThrowIfNull("serviceNameProvider");

ServiceName =
serviceNameProvider.ServiceName
.ThrowIfNullOrEmpty("serviceNameProvider.ServiceName");

Logger = logger.ThrowIfNull("logger");
Worker = worker.ThrowIfNull("worker");

CanStop = true;
AutoLog = true;
}

protected IGreetServiceWorker Worker
{
get;
set;
}

protected override void OnStart(String[] args)
{
Logger.Message("Starting service.");

if (Worker.Started)
{
try
{
Worker.Start();

Logger.Message("Service started.");
}
catch (Exception ex)
{
Logger.ExceptionAlone(ex);
}
}
else
{
Logger.Message("Cannot start service. Service is already running.");
}
}

protected override void OnStop()
{
Logger.Message("Stopping service.");
Worker.Stop();
Logger.Message("Service stopped.");
}
}

You can find the source code for our service project at this point tagged in the repo for this post on GitHub.

Now that we've got the service doing everything it's supposed to do, we may decide that we'd like to be able to change exactly how it does it's job. Essentially, we want to make it configurable. The only real parameters of our service at this point are the work interval and the file name, and those are hard-coded. We could easily make the service more flexible by taking these hardcoded values and making them configurable.

One way to do that would be to use the service argument array, which operates just like the regular app argument array. We don't have much to configure here, so that would probably work for us. But to imitate a real-world scenario where there may be lots to configure, let's pretend that won't be sufficient.

Another option would be to use a config file. Services, being normal .NET executables, can have app.config files just like a command line or Windows app does. I'm personally not fond of the default .NET config mechanism though, so I think I'll roll my own, because that can be quick and easy too, especially when our needs are as simple as they are here. Let's throw together a little XML format for our two settings.

We can load this pretty easily without relying on the magic of the .NET configuration mechanism, by creating a serializable XML data object. We can put it behind a general settings interface, and have a loader class to populate it. The interface will be enough to shield the rest of the app from the persistence details.

Here is the data class, and the settings interface.

internal interface IConfigSettings
{
String GreetingFileName { get; }
Int32 WorkIntervalSeconds { get; }
}

[XmlRoot("GreeterSettings")]
public class ConfigSettings : IConfigSettings
{
public ConfigSettings ()
{
}

[XmlAttribute]
public String GreetingFileName
{
get;
set;
}

[XmlAttribute]
public Int32 WorkIntervalSeconds
{
get;
set;
}
}

The loader class is nearly as simple. Standard XML deserialization.

internal interface IConfigLoader
{
IConfigSettings LoadConfig();
}

internal class ConfigLoader : IConfigLoader
{
public ConfigLoader()
{
}

protected String ConfigFilePath()
{
return
Path.Combine(
Path.GetDirectoryName(
Assembly.GetExecutingAssembly().Location),
"settings.xml");
}

protected FileStream OpenConfigFile()
{
return
new FileStream(
ConfigFilePath(),
FileMode.Open,
FileAccess.Read,
FileShare.Read);
}

public IConfigSettings LoadConfig ()
{
var settings = null as IConfigSettings;

using (var fs = OpenConfigFile())
{
var xml = new XmlSerializer(typeof(ConfigSettings));
settings = xml.Deserialize(fs) as ConfigSettings;
}

return settings;
}
}

We can ensure that the code that uses these classes is kept simple, by registering these properly with the IoC container.

protected virtual Config.IConfigSettings ResolveConfigSettings(IComponentContext context)
{
return
context.Resolve<Config.IConfigLoader>()
.LoadConfig();
}

protected virtual void RegisterConfig(ContainerBuilder builder)
{
builder
.RegisterType<Config.ConfigLoader>()
.As<Config.IConfigLoader>()
.InstancePerLifetimeScope();

builder
.Register(ResolveConfigSettings)
.As<Config.IConfigSettings>()
.InstancePerLifetimeScope();
}

This registration will ensure that any IConfigSettings-typed dependency within a given Autofac lifetime context will receive the same ConfigSettings instance. The first resolution of that ConfigSettings instance will be created using the ConfigLoader service.

After identifying what we want to be configurable, we can go about defining some settings providers. I'll remind you that the value of settings providers is that they allow you to define dependencies on a given configuration value, while also decoupling the dependent class from the other unrelated settings that happen to cohabit the configuration source.

Below the work interval settings provider. I'll spare you the very similar code found in the file name settings provider. The IoC registration is likewise straightforward.

internal interface IWorkInterval
{
Int32 IntervalSeconds { get; }
}

internal class ConfigWorkIntervalProvider
{
public Int32 IntervalSeconds
{
get;
private set;
}

public ConfigWorkIntervalProvider(Config.IConfigSettings settings)
{
IntervalSeconds = settings.WorkIntervalSeconds;
}
}

With the service now configurable, we can perform one final trick. When a service stops, the process may not necessarily stop. So we need some way to establish when the configuration is reloaded. The natural place for this to happen is when the OnStart event is triggered. Since the config file is loaded when a new IConfigSettings dependency is resolved for the first time in a context, we just need a new context to be spun up with each call to OnStart.

We can make this happen by taking advantage of Autofac's relationship types again. A Func<T> dependency will re-resolve each time the function is executed. And an Owned<T> dependency will cause a new lifetime scope to be created each time the dependency is resolved. So all we need to do is combine the two.

Here is the new service class, with the Func<Owned<T>> dependency.

internal class GreetService : ServiceBase
{
protected virtual IGreeter Greeter
{
get;
private set;
}

protected virtual ILogger Logger
{
get;
private set;
}

protected virtual Func<Owned<IGreetServiceWorker>> WorkerFactory
{
get;
private set;
}

public GreetService(IServiceNameProvider serviceNameProvider, ILogger logger, Func<Owned<IGreetServiceWorker>> workerFactory)
{
serviceNameProvider.ThrowIfNull("serviceNameProvider");

ServiceName =
serviceNameProvider.ServiceName
.ThrowIfNullOrEmpty("serviceNameProvider.ServiceName");

Logger = logger.ThrowIfNull("logger");
WorkerFactory = workerFactory.ThrowIfNull("workerFactory");

CanStop = true;
AutoLog = true;
}

protected IGreetServiceWorker CurrentWorker
{
get;
set;
}

protected override void OnStart(String[] args)
{
Logger.Message("Starting service.");

if (CurrentWorker == null)
{
try
{
CurrentWorker = WorkerFactory().Value;
CurrentWorker.Start();

Logger.Message("Service started.");
}
catch (Exception ex)
{
Logger.ExceptionAlone(ex);
}
}
else
{
Logger.Message("Cannot start service. Service is already running.");
}
}

protected override void OnStop()
{
Logger.Message("Stopping service.");
CurrentWorker.Stop();
CurrentWorker = null;
Logger.Message("Service stopped.");
}
}

When OnStart is called, assuming the service isn't already running, we create a new worker, from the delegate dependency. With this new worker comes a new scope, so it's dependency on IConfigSettings will be resolved with a new object, and the config file will be loaded on the spot. Conversely, OnStop allows the worker to go out of scope, resulting in the lifetime scope being cleaned up.

With that, we have finally completed our Hello World Windows service. The guts are all there, and it's cleanly designed around interfaces for maximum testability should we desire to have a unit test sweet. And finally, we use Autofac to conveniently manage a relevant object lifetime scope, making sure our settings are loaded and unloaded at the appropriate time. The actual core behavior of the service itself can be as simple or as complex as it needs to be, while remaining easily accessible and manageable via the worker object and it's lifetime scope.

The full source for the service in its final incarnation can, like all the waypoints, be found in this post's GitHub repo. Thanks for bearing with me through this month of Windows service architecture! I'll see if I can't find something new and different to tackle for next week.

Anatomy of a Windows Service - Part 3

2011-02-22T09:30:00.016-06:00

After going through the remaining work I had planned for our service specimen, I realized that the resulting post would be very long, and so I have decided to extend the series by one more post. This week we give the app a job, and add logging.

We've spent enough time working on this service so far that we should probably give it a job at this point. You may have guessed from the name by this point that I've been intending to make this a "hello world" for services. The types of things that services typically do are either to respond to system events or network communications, or to process units of work as they become available. A nice easy job we can give to our service is to touch a file on a very short interval--an interval short enough that a scheduled task isn't appropriate. Let's say 5 seconds.

We'll worry about handling the interval later. For now, let's just add the behavior so that it happens once, when the service starts.

First we create a straightforward little task-oriented class and interface. This should require no explaining. And of course we'll register this with the IoC container as well. You should be familiar with that code by now, so from here on out, I won't mention the IoC container unless something out of the ordinary is required for the registration or resolution.

With this done, we add a dependency to the interface in our GreetService constructor, and then a call to the task in OnStart.

This will cause the C:\Hello\World.txt file to be created when the service starts. The service will then be running, according to the Service Control Manager (SCM), and can be stopped. But while running it will do nothing more. If you want to see the full sample project source at this point of progress, I have tagged it in the GitHub repo.

This is a good time to add some logging. If we have problems later on, logging will help us to troubleshoot what's going on. Right now, if the service were to fail somehow, we wouldn't know why. If AutoLog were set to true in the service contructor, then the service would log to the Application event log when it crashed, but not with much useful information. But we don't want that anyway, if our log is an important one. Rather it should have it's own log, so we can always be certain we're looking at the right info and we don't have to sift through hundreds of other entries to find it.

Adding a custom event log is actually quite simple. There are two parts to the task. One is to make sure it gets created when the service is installed. The other is to use it. Creating an event log on install is trivial, mostly because it's already being done, and we can just hook our own data onto that. That's right, the ServiceInstaller class by default contains an EventLogInstaller that is configured to install an event source for our service in the Application event log. All we need to do is override that with our own log and source name.

Since both the installer and the logger class we'll write will need to know those two pieces of text, we'll create a settings provider for them. This will allow us to control the values for both usages in a single place.

Now that we've got a source from which to obtain the proper names, we can add a dependency to our service installer, and use it to override the default event log settings. All we need to do is pull the default EventLogInstaller out of the ServiceInstaller.Installers collection, and change the properties.

When we go to install the service using InstallUtil we will see it output that it is creating a new event source and event log. And if you go to the Windows Event Viewer in the Administrative Tools, you'll find a log called GreeterService.

Of course in order for anything to show up in there, we'll need to log some things to it. The safest way to do that is to throw in a logger object that can try to log to the event logs, but which won't crash out if it can't. The last thing we want is for the service to crash due to a log issue if the work itself is humming along. (Unless, of course, we need an audit trail or something, but that's a different story.) Note the explicit exception suppression in the logger class below for this purpose.

This logger could be enhanced in a few ways, for more serious services. One good thing to do is to support different "levels" of logging. I have four standard levels that I use. In increasingly voluminous order they are: fatal errors, exception cases, progress, and verbose. An enum parameter added to the logging methods would allow the logger to check to see if it's in a mode that should allow an entry for the specified level. A step further, we might replace the String parameters with Func callbacks. The callers can place the actual work of assembling the message into these callbacks, and the logger would only call them if it had determined that the logging would be performed. By only doing the string manipulation if it is absolutely necessary this keeps performance up and avoids unnecessary failure points.

With the logger object available it would be a good idea to add some logging when the service is starting and stopping. Here are the updated OnStart and OnStop methods from our GreetService class:

Now you can install the service using InstallUtil.exe, and when you start and stop via the SCM, you'll see entries show up in the GreeterService event log.

We'll stop here for now, as this post is getting long and there is much more to do. You can see the full service code to this point so far where I have tagged it in the GitHub repo. Next time will be our final (I promise) post in this series. We'll bring it home by figuring out the best way to make sure the service does its work when we want it to, and then making it easily configurable, and reconfigurable.

Satisfying A Pair of Same-Typed Dependencies with Autofac

2011-02-15T09:30:00.176-06:00

We'll take a break from the series on Windows services this week and cover some more Autofac / IoC / DI material. I hope to finish the services series with the final part next week.

Back in December I wrote a rather long post about the different Autofac relationship types. I covered what I considered to be the big ones, but there were a few that I didn't cover: IIndexed, and Keyed. I had an objection to these types that prevented me from examining them any further.

My Initial Objection

It's pretty simple. I don't want to have to reference the the IoC container framework in my classes. Accepted wisdom on IoC container usage is that you shouldn't reference the container, and I viewed this as an extension of that. Indeed I prefer to avoid referencing such infrastructure frameworks in core code wherever possible.

I feel the same way about the BCL's TypeConverterAttribute, and the XML serialization attributes, for example. It's an insidious form of coupling, and I really prefer to avoid having these little signposts sprinkled throughout the code of what framework I'm working with. Especially if they represent things that I can't sub out for testing, as is usually the case with the BCL.

In addition to the framework reference objection, I felt that this was just not a responsibility that belonged to the subject class. Using special parameter types or the like to denote such information was "cheating", akin to service location, and undermined the principle of Inversion of Control.

The purist in me had dug in on two separate fronts, and was ready for a fight. But recently I had an experience that taught me the value of these relationship types: I found I needed to inject two different implementations of the same service in the same place. I initially shied away from these relationship types. Instead, I decided I could solve the problem with a clever registration/configuration. After all, deciding who gets what implementations is what that's all about, though typically in a more generalized sense.

The Problem

My situation was that I had needed two different runtime instances of an image cache. One for full-size images, and one for thumbnails. And most commonly I needed to reference both of them in the same scope. If the two use cases existed in different contexts, I could possible use Autofac's nested container support to identify each one appropriately when needed. But in my scenario, I actually had a couple of classes that needed to make use of both caches. I gave the constructor for that class two parameters for injection, both having a datatype of the cache interface type. Then I went to work setting up the registration.

Without a major redesign, the solution has to involve attaching some metadata to each of two separate instances registered with the container. The container needs some way to differentiate between the instances. Since Autofac provides it, there's really no reason at all not to use the keying facility for registration. Given an interface ICache and an implementor Cache, the registration would look as below.

Now let's introduce another class that needs to use both of these caches.

Again, my instinct is to say that the class shouldn't have to say anything else. It's the container's job to figure out how to fill those dependencies. But what information does the ImageBrowser class expose that the container registration could use to determine which argument gets which instance of ICache? Well right now, there's not much there. Pretty much just the argument names. Can we use those to assign the ICache instances? Yes, but it ain't pretty.

OnPreparing¹

The way to access the parameter names for resolution is to hook the OnPreparing event, as seen here:

Our implementation of OnPreparingImageBrowser then looks like this:

This looks harmless enough, if a bit verbose. But is a problems. We have encoded parameter names in strings in our registration code. If we ever have to change those parameter names, or add more dependencies to the constructor, this registration code for ImageBrowser will be broken, but it will still compile. So unless you have some pretty magnificent tests for your IoC registration (which it seems is not easy to do), this is a regression bug waiting to happen. Furthermore, this code is very likely useless even if we have another class with these same dependencies. If the constructor arguments are named differently, we'll need another delegate for that set of constructor arguments.

So obviously argument names aren't the ideal way for the subject to communicate information about multiple dependencies of the same service type. And even using the argument names undermines my objection in spirit. It just does it implicitly, rather than explicitly. So let's just accept that we've already taken the first step and go the rest of the way. How can we communicate the necessary information in a sustainable way, that doesn't require us to go mucking around in the registration code if we shuffle things around a bit in ImageBrowser?

Three options spring to mind. One would be to use an attribute applied to the arguments of the constructor. But there's no native support for recognizing these in Autofac, so we'd have to write our own registration and resolution conventions. While it would be more resistant to change than the named parameter mechanism, it would require even more code on the registration side of things, to set up a convention-based registration that keys on the attributes. Not to mention having to define the attributes. This actually sounds like a fun exploration for another day. But today, we just want to get this working with a small amount of intuitive code.

Autofac has two mechanisms which we can use to reduce the amount of code we need here: IIndexed and Meta.

IIndexed<TKey, TComponent>²

Using Autofac's IIndexed mechanism is a simple change from where we are now. It involves a small change to the constructor of ImageBrowser, and removing code from the bootstrapper. That's right, a net reduction in code, without losing functionality. And there's nothing better than a good codectomy. So grab the scalpel and sculpt our new bootstrapper.

Note the removal of the OnPreparingImageBrowser method, and the reference to it in the ImageBrowser registration. That's the only change here. It's completely unnecessary code. The cache registries don't need to change because the IIndexed mechanism hooks right into the keyed registration just like our old strategy did.

The ImageBrowser constructor needs to change a bit to trigger resolution using the keyed registration. Here's the updated class:

The only change here is that we consolidated the separate cache arguments into one dictionary-style argument.

Meta<TComponent>³

Autofac's Meta mechanism works a little differently in that it allows you to attach data to a registration, and then to express a dependency on that metadata, in the subject class. The data takes the form of name/value pairs. If we wanted to use this, it would change our registration as follows:

We've replaced the call to Keyed with a call to WithMetadata. We also are essentially passing a name-value pair. There's also the option to attach an actual custom metadata object, and use the property names and values as the name/value pairs. Since we only need the one value, and it's not going to be changing at runtime, we can just directly assign the name and value easily.

Once again our constructor will need to change, of course. Here's the version that uses the metadata:

Here we're using the IEnumerable relationship type to get all the metadata objects, and then explicitly picking out the ones with the two with the values we're looking for. The CacheTypeChecker method is just a little comparison delegate generator that saves a bit of repeated code in the constructor.

The Lesser of Two Evils

Now this isn't necessarily the problem that Meta or IIndexed are meant to solve. It's really more meant for cases where the needed component is going to depend on runtime conditions. In that case, a simple service finder/locator is sometimes exactly what you want. There are often other ways to get the job done even then however, for example using lifetime contexts and contextual state objects to give the container all the information it needs.

In our case though, the dependency that's actually being injected bothers me a bit. It's sort of a limited service locator. The problem with that is that the specific dependencies aren't explicit. ImageBrowser is basically saying, "Er, I can't tell you exactly what I want, but if you give me everything like this, I'll pick out the ones I want". That is definitely not a pure inversion of control in this case, because we need exactly 2 instances of a very particular type.

We're basically limited to choosing the lesser of two evils. Do we couple to constructor argument names, or take a dependency on a service locator? I'm not sure I feel more strongly about either one. But the IIndexed option is definitely the least code, so that's the one I'll go with. And you can bet that if I run across a situation where the decision of which instance I need is made at runtime, I'll look to these solutions first.

Somewhere down the line I'll investigate that attribute convention option I mentioned in passing. I tend to dislike attributes, but if it's our code (rather than framework code) that does both the defining and the consuming... then maybe I can live with it.

GitHub Source Code References

Anatomy of a Windows Service - Part 2

2011-02-08T09:30:00.061-06:00

This post contains gist code samples. My experiments indicate that these do not display in Google Reader, so if you are reading this there, you may want to click through to the post.

Last time, we dissected a Windows service template project to get a handle on the different moving pieces. We identified an installer class, as well as two run-time configured installer objects, a service class, and the framework injection point for spinning up a service.

The template is a very simple project, and it gets the job done. There are some deficiencies though. From an architectural standpoint, the design doesn't lend itself well to testability, or composability. It also doesn't provide any good place to bootstrap an IoC container for the installer portion, which would be beneficial to have if we want to have any custom behavior happening at install. From a feature standpoint, there's no logging, and no worker thread to separate the service control events from the actual behavior of the app.

Let's start out by improving the composability, which will put us on a good footing for testability as well. The biggest obstacle to that in the template project was all of the direct instantiation and configuration of objects. For example, in the template project, the ProjectInstaller class's constructor, which is an entry point that is triggered directly by msiexec or installutil, was responsible for creating and initializing the sub-installers. In our application, we'll defer to an IoC container, rather than construct these objects directly.

Let's add a reference to my personal favorite IoC container, Autofac, and then create a bootstrapper that will do the registration necessary for the installer portion of the app. We'll leave it empty to start, because we haven't established what we'll need from it yet.

Now lets create the ProjectInstaller. The constructor, since it's an application entry point, will have a direct reference to the IoC container. We'll leave out the needless code we identified last time, but don't forget the crucial RunInstaller attribute. The constructor is left responsible for setting up the Installers collection. If we assume that we'll be registering all our installers with the IoC container, then Autofac makes this easy for us with the IEnumerable relationship type, which will resolve all implementors of the specified service that are available at the resolution location. Let's put all that together and see what we get.

Compared to the template project, we've eliminated some explicit object instantiation, but we've also lost some explicit configuration. That has to be exported to somewhere else. There are a couple of options for re-creating this functionality with our IoC container in place. One is to give these exact configuration responsibilities, i.e. the setting of properties after construction, to the IoC container. This is not a strong-point of most containers--even Autofac. It can be done, but it's not really straightforward. It results in complex registration code that doesn't communicate well what's going on.

Alternatively, we could move these responsibilities into factories. I've used this strategy before, and it can be advantageous if there are classes that will have dependencies on the factories themselves. But in this case, the only thing that would need to know about the factories is the IoC. So instead, I'd recommend deriving special-purpose sub-classes of the desired installer base classes, and putting the configuration in those constructors instead. Here's what that looks like.

Simple, straightforward, self-contained, and single responsibility. Now we just need to update the IoC registration to include them, and they'll get picked up automatically by the ProjectInstaller constructor. We could take advantage of Autofac's fluent registration API and auto-wiring to do this without referencing the individual types. But I have these two classes in separate namespaces (the service installer is in the same place as the service, because these must come in pairs), and the ProjectInstaller derives from the same base, it would actually take more code to deal with these special circumstances. So explicit registration it is.

Note also here that we've registered these as InstancePerLifetimeScope, because there's really no reason more than one of either of these should be floating around.

That covers the install portion of the service. We now have separated the three installation components into their own separate places. Changes to the configuration of any one of them are isolated in their own locations in the code. And as a bonus we've made the ProjectInstaller constructor trivially simple.

Let's shift gears and take a look at the service portion of the application. We'll set up a blank bootstrapper for this portion as well. I'll refrain from posting the code as it's essentially identical to what I showed you for the other one.

The remaining two significant pieces that need to be implemented to replicate the functionality we saw in the template project are the service class itself, and the program mainline. The mainline is going to be simple just like the ProjectInstaller. In fact it's structure mirrors the installer very closely, just with different type names in a few strategic places.

I've chosen to request an IEnumerable, even though we only have one implementation. Maybe we'll add more services to this assembly some day, and maybe we won't. To make a decision on that basis triggers my YAGNI reflex. I used an IEnumerable resolver just for my own aesthetic sense, because the framework runner takes a collection anyway.

Next let's look at the service class itself. This is another class that, in the template project, had a lot of noise that we identified as unnecessary. So in this iteration, it's going to be comparatively short and sweet.

There's very little to see here, and that's very much as it should be, since our service doesn't actually do anything yet. We just cut out the noise and were left with something trivially simple. So let's move on to the last piece left to put in place: the service bootstrapper.

We're just blasting through code files here, as this one is another very simple one, with just one registration. We'll add more, but for right now we have completely matched the functionality of the template project.

Before we put the project to bed for another week, though, there's a small refactoring we can make that will make a nice little illustration of the benefits of how we've organized the project. You may have noticed that we have some hard-coded strings in the constructors of the service and the service installer. And particularly, there's one that's intentionally repeated, since it has to be the same in both places. We can eliminate the hard-coded string and make sure that the values in both locations always match, by using the setting injection pattern I described a few months ago.

With just a few lines of code we can set up the necessary setting dependency.

With this class and interface in place, we can add a dependency to the constructors of the installer and the service.

And finally, in order to support the dependency injection, we just add one new registration to each of the bootstrappers.

That's the end of our work for this week. We have replicated the template project's functionality, such as it is, and run through a lot of simple little snippets of code. Structurally, the Solution Explorer doesn't look a ton different, but I think the code is simpler and more straightforward. Plus we have built a good foundation on which to add the functionality that people expect of a service. All the code for the project we worked on in this post can be found on GitHub at https://github.com/cammerman/turbulent-intellect/tree/master/HelloSvc.

Next week, we'll add logging to an event log, and we'll make it actually do some work. Finally, we'll figure out how that work needs to get done, inline, or in one or more worker threads.

Anatomy of a Windows Service - Part 1

2011-02-01T09:30:00.016-06:00

This post contains gist code samples. My experiments indicate that these do not display in Google Reader, so if you are reading this there, you may want to click through to the post.

I recently had to estimate the effort for creating a one-off Windows Service for massaging some document indexing data as it passed between a capture system and an ECM system. I was quite confident in my ability to code up the "massaging" part, but the service part was unfamiliar territory. So I decided to do some investigation and figure out just what it takes to get a service set up, what special considerations need to be made for logging and injecting configuration, and how to preserve good design sensibilities through it all.

The easiest way to create a Windows service is to use the associated project template in Visual Studio. I specifically wanted to work in .NET 4.0 for the investigation, as I would be on the actual project, but unfortunately I don't personally have a license for the full version of Visual Studio 2010, and VS Express 2010 doesn't have a project template for Windows services. Fortunately, I do have VS 2008, and the BCL infrastructure for services hasn't changed between 3.5 and 4. So I decided to generate what I could in VS 2008, add some googling, and use the combined information to create a service from scratch in VS Express.

This week, we'll analyze what VS will give us for free with the project template and special code-generating actions. We'll identify the purpose of the different pieces of code, and make note of how we can use this information to build our own service, complete with installers, from scratch.

So to start, let's take a look at the structure that the project template creates for us. Here's a shot of what the initial solution looks like in a brand new Windows Service project.

Note the two structural differences between this template project and a blank or console project. There's an extra reference to System.ServiceProcess.dll, which we'll need to remember, and there is an extra component class called Service1. There's also a .Designer.cs file for Service1, but this is really just because Visual Studio automatically adds these any time an item that is a subclass of Component is added to your project, either via a template, or the "Add New Item" command.

Before we take a look inside the service class itself, we should see how the Program class differs from other types of project templates. Here's the Program class.

There are a few differences to note here. The biggest difference is obviously that the main method contains some generated code. It actually looks similar to what is generated with the new Windows Forms project template. We have the service class, representing the actual core of the program, being instantiated and then passed to a runner method provided by the framework that handles spinning up the appropriate context.

Now lets take a look at the service class. It comes in two parts: the main class file, and the designer file. First the main class file.

The most prominent features here is the base class, ServiceBase, and the two override methods, OnStart and OnStop. ServiceBase is, of course, the common base class that all .NET windows service classes must derive from, in order to take advantage of the framework support. The overrides are two of the possible entry points into the service. They correspond directly to the actions that you can take on a service in the Services control panel. OnStart is where you would place the code that processes the arguments, spins up workers, etc. Naturally OnStop is the other end of things, where you'd put your cleanup code to prepare for termination. There's also a call to InitializeComponent in the constructor, which is typical of generated Component-derived classes. We'll see if there's anything interesting happening there next, in the designer file.

There's a bunch of generated code here, but there's actually less going on. So what is all this doing? We can easily see an implementation of a Dispose idiom, which is disposing of any components contained inside this one. But when we look back at the InitializeComponent method, the Container is constructed and then left empty. The only significant code here is the assignment of the ServiceName property. We'll need to remember this for when we create our own service class from scratch. Everything else in this file could be dropped. The ServiceName initialization could just as easily be put directly into the constructor.

What we've seen so far is the absolute minimum service. If you put some behavior into the OnStart method, you could run from the command-line, and it would operate. But it wouldn't show up in the Services control panel, because it doesn't have an installer to register it. And that means you couldn't Stop it once it was started.

What does it take to register a service with the OS? VS offers a shortcut to generate the code for this as well. If you right-click the service class Service1.cs in the solution explorer, the context menu has an "Add Installer" option, which adds a new class to the solution, called ProjectInstaller. This is the last piece of our example puzzle, and it's another two-parter. First the base class file.

There's not much code here either. Like the service, we see this is a Component-derived class, by way of Installer) with an InitializeComponent method called in the constructor. We'll take al ook inside there in a moment. First, make note of the attribute on the class. The RunInstaller attribute is a signpost for the InstallUtil.exe installer tool that comes with the .NET Framework, or for the VS Setup Package Custom Action Installer. The Boolean true argument tells the installers that that they should indeed invoke this installer class when they find it in the assembly.

The attribute is important, but it's not the meat of the class. That's found in the designer file.

The form of this file is very similar to that of the service's designer file (and any designer file, really). Once again we see the disposal idiom, and once again, the container that it disposes is never populated with anything. The body of the InitializeComponent method is concerned with creating and initializing two more installer objects. One is a ServiceInstaller, which is a class that knows how to install a particular service in the registry and such, given a ServiceBase representing it. The ServiceProcessInstaller handles install tasks common to all services, and so there should be only one inside a given project installer.

The MSDN documentation doesn't go into any more detail about what the responsibility differences are between these two types of installer, or why they are peers, as far as the main Installer is concerned. But it is clear about how they should be used. The installer objects need to have properties initialized to indicate how the install should proceed, and then they must be placed in the Installers collection. This also could just as easily be done in the constructor, eliminating the need for the designer file.

One important thing the documentation highlights is that the ServiceName property on the ServiceInstaller must match the same property on the service which it is associated with, since this is the only thing that links the service registration with the actual implementation. We'll examine the other installation-guiding properties, on both objects, in detail when we put together our own service from scratch.

For now, we've covered everything in the project template. What we have here is a complete, spartan service that does nothing. It will install, simply by compiling the executable, and then running InstallUtil.exe from the .NET framework folder against it. And it will run from the service control panel... but it has no visible side-effects.

Next time, we'll get to work replicating the moving pieces of this solution in a spanking new, blank C# 4 project. It will look a little different, structurally, but all the prominent features will still be there.

P.S. All the code that appears in this series will be available on my github page. For now, I have it in this repo: https://github.com/cammerman/turbulent-intellect. The code for this post in particular is found here: https://github.com/cammerman/turbulent-intellect/tree/master/ServiceExample35. I expect to reorganize this a bit in the future, but I'll update this link when I do so.

New Domain!

2011-01-27T10:16:00.000-06:00

Just a little off-schedule post today to alert anyone who didn't notice the redirect: I've changed the URL of the blog to be its own domain. I purchased the domain a couple years ago, when Blogspot's support for custom domains was... primitive. It's a lot better now and was a breeze to set up. So here we are! Blogspot says the old URL will continue to redirect to the new location, if you don't feel like changing.

Now back to normalcy. Expect a post on Tuesday as usual.

Glints of Profession

2011-01-25T09:30:00.005-06:00

Recently I caught a few tweets from Patrick Welsh where he briefly mentioned pair-programming with his young daughter while she was home sick from school. In particular the second one referenced being "green", as in all tests passing. It struck me on a couple of different levels, even beyond the "awww" factor.

(Let me not understate the "awww" factor. Stories like this set anticipation brewing in me. I can't wait to be able to share my passion for programming with my own son or daughter. Hopefully she listens better than I did when my dad attempted to enact the rituals of fatherhood with me at his workbench, under the hood of the car, and in the kitchen. Sorry Dad! If I could do it again, I'd pay attention this time around.)

This brief snapshot of their interaction conjured visions of the apprentice at the elbow of the master craftsman, absorbing subtle waves of well-worn trade wisdom. We're talking about more than just "measure twice, cut once" type of stuff here. The equivalent of that might be "fail fast". Very good, very fundamental advice, to be sure. But testing and TDD are at least one level above that.

At the level of TDD, we're dealing with explicable, transferrable techniques. Not just muscle memory, but practices. Practices that, when followed, guide the hands of the initiate on the path toward skill, consistency, and quality. To me, this is a huge indicator of mature and professional behavior on the teacher's part. And such things are themselves markers of the continuing evolution of programming from simply a job to a profession.

Compare these types of hallmarks to the ad hoc, piecemeal, on-demand way that most programmers still learn the tricks of the trade. Most of us are still tossed into the job without mentorship to speak of. We're lucky if we've been taught theory in school, but this is incomplete knowledge at best. (This is like the difference between knowing how a car works, and being able to build one.) But a relevant degree is not the path most commonly arrived by for programmers anyway. Most programmers learn through the work alone, maybe supplemented by online information gleaned from Stack Overflow questions, or blog posts where people have documented their own unique travails. Lessons are gleaned through individual trial and error, with the occasional vicarious horror story of warning.

This story is still common. Though happier alternatives are becoming more common. Many of us make quite a show of seeking to make programming a true profession. There's a Craftsmanship Manifesto. There are apprenticeships. There are actual, literal journeymen, as well as prolific, sough-ought masters. But these are all bright trappings, and it is quite easy to lose track of substance.

Profession is not found in titles and roles, or documents of good intent. Profession is not even found in the practices themselves, for TDD does not a craftsman make. Profession is found in the fact that there ARE practices which are known to improve the quality and consistency of work. Profession is found in the wisdom that is passed on by mentors, in concert with the mechanical skills of the job. Profession is found in the common platform of discipline, integrity, and mutual education upon which quality work, worthy of pride, is built.

I haven't seen what happens in Obtiva's apprenticeships, or in Uncle Bob's classes, or when Corey Haines pairs with his hosts. I trust that they do bear forward, at least in part, the evolution of our chosen trade. But in the words and excitement of Patrick Welsh's little apprentice, I can see for myself the rough diamond of profession glinting in the light.

Throwaway Projects

2011-01-18T09:30:00.004-06:00

A big obstacle in my learning of new technologies, patterns and methods, has been my reluctance to do throwaway projects. I have a really hard time with them. Always have, in all their many forms. At times I have even actively disdained them as a useful way to learn, for various reasons.

There have been a lot of things I've been curious about and wanted to learn. I'm a programmer, so that really goes without saying. And goodness knows that, relatively unencumbered from social obligations, as geeks tend to be, I've had enough free time in my life that I could have dedicated some of it to working on some little test or toy projects to get a handle on these things. But I've always managed to convince myself it wasn't necessary, or even that it was a waste of time. Usually I accomplish this by claiming that I learn more and better from doing "useful things". Throwaway projects don't impress real constraints on you, and so they let you take liberties or make compromises that wouldn't fly when trying to actually get something done. And I was convinced that this meant the knowledge gained would be less useful or applicable to real problems.

Today I had the revelation that I couldn't possibly have been more wrong.

I was faced very recently with a number of opportunities to work on some new things, to "get stuff done", involving some technologies I'm not really at all familiar with. I initially thought to myself that this is a great opportunity to finally ramp up on some things I've been wanting to learn. But after a bit more thinking, I found myself in one case completely unsure whether I could accomplish the goal on the necessary timeline, and in the others quite certain there was no way that I could.

That's when it hit me. I realized what teachers of all stripes have known for centuries. While I've learned an awful lot about programming on the job, those fundamental lessons that my knowledge are rooted in were very rarely learned in the active pursuit of an important functional goal.

Let's first acknowledge the obvious and ubiquitous example of homework. Homework is nothing but throwaway problems. I learned the value of homework (finally) when I moved from high school, where I breezed through on in-class osmosis, to college, where info came so fast and furious that I couldn't absorb it in the desk.

In grade school, I did rote program transcription in Apple Basic. In middle school I messed around for hours, daily, in QBasic on my parents' desktop. There I played with graphics and sounds. I poked and prodded at parts of the QBasic games that were packaged with DOS. In high school I moved on to trigonometry, large-number math, a text-based RPG in the vein of LORD, 3D wireframing, and even created a graphing calculator that would automatically scale to fit one cycle of a sinusoidal function.

One useful thing I did do in QBasic was to write a file chunker that would split a file into either a particular number of pieces, or pieces of a particular size. The goal of that was to more easily share games that wouldn't fit on a single floppy, or that would take several hours to transfer over 14.4kbps modem. So, okay that was functional, and I learned a bit about file access in QBasic from it, but nothing much. Until I started to learn C++ in college, when the very first thing I did was to port my file-splitter from QBasic. I did this knowing that not only was it not very useful, but there were many freeware apps that did it better than mine. But I learned about the fundamentals of C++ syntax, style, and common pitfalls. It gave me a starting point and saved me from what would have otherwise been embarrassingly slow, trial-and-error progress on the first project of my internship.

What I missed was the now-obvious truth that being free to take liberties and make compromises that you wouldn't do in production is the perfect sandbox for making mistakes, which is a crucial part of the learning process. I had it just plain backwards.

So here I sit. Mind blown. Feeling like an idiot. I still hate doing throwaway projects, but now I know I can't afford not to. I just need to man up and fight that voice in my head that nags at me saying, "why are you writing the world's millionth blog engine when you could be working on real problems that are actually still in need of better solutions?" Because the truth is that before I can reinvent banking, for example, there are a few things I need to learn before I'll even know where to start. And the best way to do that is to play in the sandbox for a while.

Dependency Relationships and Injection Modes: Part 2

2011-01-11T09:30:00.003-06:00

Last time, I wrote about primal dependencies and the constructor injection mode. I also mentioned that in Part 2 I would cover "the other mode". In contemplating this other mode, however, I came to the conclusion that there are in fact two other modes. I'll cover both of them here in Part 2, but first we'll tackle mutator injection and optional dependencies.

Mutator Injection

Let's get the terminology out of the way first. By mutator, I mean a property or method on a class or object that is used to change some bit of state in that class or object. By optional dependency, I simply mean one which is not required in order for the object to accomplish its primary purpose.

In most code you'll read, mutator injection is ubiquitous. However, I would argue that most are misused because the usage, the need, or both, do not fit the above description of an optional dependency. So what are people doing with dependency mutators that they shouldn't be? And what should they be doing instead? Most people use mutation when they are presented with an apparent hurdle to using another, preferable injection mode. Often this comes down to an inconvenient design choice external to the class that creates an environment where it is hard to do the right thing. The appropriate response is of course to fix the context, whenever possible.

Why You Should Prefer Alternatives

Often, a dependency mutator is used to handle post-construction initialization of a primal dependency, where the appropriate dependency instantce is not known or available when the consumer is constructed. I would argue that this is very rarely an inevitable state of affairs, and is commonly due to a dependency cycle between two layers of the architecture. (I.e. an object in one layer depends on an object in the layer below it, and another object in that layer depends on an object in the first layer.)

Intra-layer dependency cycles are almost always a bad idea, not just because it makes it tough to avoid mutator injection. Maintaining a single direction of intra-layer knowledge also helps to prevent unnecessary coupling, and it aids in keeping each layer cohesive. But an extra benefit of fixing this situation is that it will typically allow for the primal dependency to be created or retrieved when it is needed for the constructor injection, rather than by the consumer or a manager object.

Why does this matter so much? Most simply, because mutator injection is a state change. And state changes are big potential points of failure. It's also well-established that it's easier to reason about immutable data structures. I've found this to be true in my own experience as well. It's easier to compose behavior out of pieces that exhibit immutability because it simplifies the interaction graph. Furthermore, when things go wrong fewer state changes means a shorter list of possible places where the bad state may have been triggered. So when designing my classes, I avoid mutators wherever I can, and that means preferring constructor injection and site injection over mutator injection whenever possible.

The other way that mutator injection is mis-used is in a situation I briefly mentioned in Part 1: when only a subset of the operations on a class actually make use of the dependency. Mutators are often seen as convenient in this situation because they cut down long constructor argument lists, which are seen as messy or complex. The fact that the object has at least a limited use without the dependency makes it possible to get away with setting it later.

In Part 1 I said that the situation itself is often due to a violation of the Single Responsibility Principle. It would be facile to cite this reason for all instances of mutator injection. Sometimes it's true and sometimes not. When it really isn't true, there are a couple of other options available before you have to resort to mutator injection.

The first option is again to drop back to constructor injection, but to do so with a sort of lazy proxy in its place. The lazy proxy exists for the sole purpose of deferring the resolution of the actual service dependency until it's needed. This is its Single Responsibility. It can then either provide an instance on demand, or it can be more transparent and just implement the same interface and delegate to the underlying object.

In a static language like C#, this isn't always possible either. Sometimes there's no interface, and some of the members are not virtual, so they can't be overridden in a derived class. If you control the code of this object, I highly recommend refactoring to an interface to resolve this problem. But if you don't, it might be time to look at the third injection mode: site injection.

Temporal Dependencies and Site Injection

Site injection is a fancy name for a method argument. It describes the use of a service as an argument to the operation for which it will be used. Such a dependency, that is passed directly to an operation, and for which no reference is maintained after that operation completes, is called a "temporal dependency". The reference is fleeting, and not actually part of the state of the object that depends on it.

There are a couple consequences of this type of injection. First is that the calling object or objects become the primary dependents of the service. It may or may not be easier to manage the dependency there. Hopefully it is, but sometimes it isn't, especially if there are many different objects that may call on this operation.

The other consequence of site injection is that it can add noise to the interface/API. If there are multiple operations in the subset of the dependent object that make use of the service, the repeated passing of the service into the objects can feel like unnecessary overhead. Not performance-wise, but in terms of keeping the code clean and readable. Expression of intent is paramount here. If the extra arguments communicate the true relationship between these two pieces of your architecture, then it's not truly noise. Instead it's a signal to readers of what's going on, and who's doing what.

There are lots of places where site injection and temporal dependencies are very natural. So many that you probably use them without even thinking about it, and it's often the most appropriate thing to do.

Truly Optional Dependencies

That's probably enough about what isn't an optional dependency. So what is? I would argue very few things are. Usually an optional dependency crops up as something that can be null when you want to do nothing, or can have an instance provided when you want to do something. For example, one step of an algorithm template, or a logging or alert solution with a silent mode. Often, even these can be treated as primal dependencies from the consumer's perspective, with a bonus of avoiding null-check noise. Just provide a Null Object by default, and make a setter available to swap the instance out as necessary. If you can say with certainty that this is not appropriate or feasible, then you may have a truly optional dependency.

A Good Start

I believe these three dependency relationships, and the injection modes associated with them, cover most of the scenarios that you'll encounter. And I've found that the advice I've included for normalizing to these modes has helped improve the design of my code in a very real and tangible way. But I'm certain my coverage isn't complete. There are surely more dependency relationships that I haven't identified here, especially if you dig into the nuances. I'd love to hear about other types of dependency relationships you've encountered if you're willing to share them in the comments.

Dependency Relationships and Injection Modes: Part 1

2011-01-04T09:30:00.001-06:00

This week I'm going to continue my trend of adding to the cacophony of people talking about IoC and DI. There's a lot of talking, and a lot of it is noise. Not inaccuracy, mind you. Just an awful lot of shallow coverage, one step beyond assumptions and first principles, with not a lot of real analysis. So let me bring a bit of deliberate consideration and analysis to the table, and hopefully improve the signal to noise ratio.

What's different this week from the recent past is that I am specifically going to talk about DI for once, rather than IoC in general. The particular topic I want to address is something that I've seen addressed a few times, but usually in passing reference in the context of some other argument. And most often it's being both taken for granted and dismissed out of hand, by the respective debaters.

In most languages there are essentially two injection modes available: via a constructor argument, or via a mutator such as a property setter or method. I've found that each option is most appropriate in different situations. A constructor argument dependency expresses a relationship that is very different than that expressed by a mutator.

Conveniently, there are also two broad types of dependencies that map fairly well to these two modes. The first is sometimes termed a "primal dependency". (I'd love to credit the inventor of this term, but all i can find in a Google search is Jimmy Bogard, Tim Barcz, and Steve Harman all using the term as if it already existed.) The second type of relationship is most often called an "optional dependency", which is a much less evocative term, but it's a descriptive one nonetheless.

The relationship that is being evoked by the term "primal dependency" is one of raw, fundamental need. A primal dependency is one that must be satisfied in order for the class to even begin to do its duty. When a primal dependency is unfilled, the dependent object is essentially incomplete and nonfunctional. It's in an invalid state and can't properly be used.... At least not if the SRP, or Single Responsibility Principle (PDF link), is being obeyed.

The SRP is one reason why I advocate using constructor injection for primal dependencies. Constructor injection makes it explicit and unavoidable that a given dependency is a requirement for proper functioning of the object. If a dependency could be provided via a mutator without affecting the post-construction usefulness of the object, then really you have one or both of two situations. Either it's not really a primal dependency, or your object is responsible for more than one thing and this dependency is only required for some of those things. Using constructor injection avoids the possibility of this signal being hidden behind the guise of post-construction initialization.

When people oppose constructor injection, it's often on the claim that it puts an undue burden on the object performing the instantiation. While it's true that a burden is being transferred, I'd argue that it's not undue. Construction and initialization of an object into a fully valid state should be considered to be two parts of the same operation, the same responsibility. Furthermore, this responsibility is separate from the primary purpose for the existence of the object itself, and so it doesn't belong to the object itself. This is what DI is meant to resolve. Yet, the logic must be taken a step further.

If construction and initialization is truly a multi-step process pulling from different domains, and/or which can't cleanly be handled in one place, then it's a responsibility that's complex enough to base on entire object around. In short, you should strongly consider setting up a factory or builder object to handle this task. The only extra burden this leaves is on the brain of the programmer who is unfamiliar with the pattern. But trust me: just like with any exercise, the burden gets lighter with repetition.

I have a confession to make: I had never actually used the builder pattern until I came to this revelation. I didn't really understand what the benefit was over a factory. And I'd only taken the most rudimentary advantage of factories, for implementing concrete implementations of abstract base classes. But now each is as clear, distinct, and obvious of purpose to me as a screwdriver and a hammer. You'll find they come in quite handy, and really do clean up the consuming code, once you acknowledge that construction and initialization should often be a proper object responsibility all on its own.

In Part 2 of this topic, I'll address optional dependencies and the mutator injection mode. Like today's topic, they're deceptively simple to describe. But there is just as much, if not more, nuance to be found in the manner and purpose of their implementation.

Data Plus Behavior: A Brief Taxonomy of Classes

2010-12-28T14:45:00.000-06:00

When I was first learning about classes and object-oriented design in college, classes were explained to us cutely as "data plus behavior". This is oversimplified and inadequate in the same way as would be an explanation of the Cartesian Plane as "a horizontal axis plus a vertical axis". This neat package doesn't begin to encompass all that the Cartesian Plane is, especially compared with its constituent parts.

Through my own work and experience, I've found it useful to break classes down into four different categorizations that better express the continuum of data/behavior relationships. Each category serves a different class of needs, and each interacts with other classes and categories in consistent ways. The four categories, as I have identified them, are data, algorithms, stateful algorithms, and models. Let's take a look at each of these in turn. We'll go over the traits that qualify each, the roles they serve, and their typical relationships with other classes.

Data

Data classes are quite straightforward. They are "pure" data, with little to no behavior attached. Your typical "data transfer objects", primitives, structs, event argument classes, etc. fall into this category. There are few, if any, methods defined on this type of class. When a data class does have methods, they will typically be related to type conversion or projection, simple derived value calculation, or state change validation. However, methods offering complex mutability or interactions with other objects would push a class out of this category and toward the "model" category.

Classes in this category are most often composed completely of other data classes. When I was initially considering other qualifications, I thought immutability would a be a big one, but I've found that's not necessarily true. Rather, it's dependency on services (especially stateful algorithms) that almost certainly indicates that a class is a model rather than a data class. Data classes very rarely have more than a passing dependency on external behavior, though they may have multiple dependencies on external state, made primarily of other data classes. This makes sense, since the most common thing to do with algorithms, stateful or otherwise, is to compose them into more complex behavior.

Data classes are the quanta of information in your application. They will be passed around and between other classes, created and processed by them in various ways. They exist first and foremost to be handled. They are rarely seen as dependencies except indirectly via configuration layers, but rather show up most commonly as state, or as operational inputs.

Algorithms

An algorithm is a class that is essentially functional, at least in spirit. A strong indicator that you have an algorithm on your hands is when a development tool such as Resharper suggests that methods, or the class itself, be made static. This generally indicates a near-complete lack of state.

One common application of algorithm classes is as a replacement for what would be a method library in a language which allows loose methods. For example a string library, or a numeric processing library. In fact, data processing in general is one big problem space that lends itself to algorithm classes. Also, many of the methods seen on primitives in languages such as C# could very easily be extracted into algorithm classes.

Algorithms rarely have public properties and almost never have public setters. In general algorithm classes have the potential to be expressed even without private properties, i.e. as truly functional code. Usually this means handling data that would otherwise be stored in properties as arguments instead. But often the realities of memory and processing power limitations mean that this isn't actually possible. So complex algorithm classes do often have some private state.

Depending on the amount of internal state required for performant execution of the algorithm, such classes may be expressed as either static/singleton, or by instance. For algorithms requiring no internal state, I prefer a container-enforced singleton pattern (as described in my post on lifecycle control via IoC containers). Static classes encourage tight coupling, and are very difficult to sub out for testing, even with a powerful isolation tool such as Microsoft's Moles,Telerik's JustMock, or TypeMock Isolator. As such, I avoid statics like the plague.

Instanced algorithm classes are usually lightweight and treated as throw-away objects, repeatedly created and disposed. Some may also choose to implement them with a reset/clear mechanism to allow reuse of the same object for different inputs. However, given the memory and garbage collection power currently available, I view this as an optimization, and avoid it unless it provides clear, needed performance benefits. Otherwise, it simply adds complexity to the implementation and muddies the interface with methods that don't directly relate to its purpose.

Algorithm classes often have dependencies on other algorithms. Data dependencies, however, are usually just another form of argument for the algorithm. Something that affects the output of the work from one instance to another, but is likely constant within one instance. Usually this is data that simply isn't going to be available at the call site where the algorithm will be triggered. Environmental state is one example. Mode switches are another. Anything more complicated or mutable than this is an indicator that you might in reality have a stateful algorithm on your hands.

Stateful Algorithms

The simple explanation of a stateful algorithm is that it is a bundle of behavior with a limited reliance on internal or external state. It differs from a pure algorithm in that the internal state isn't necessarily directly related to the processing itself. The most common application of a stateful algorithm is a class whose primary purpose is to expose the behavior of a model, while internally managing and obscuring the state complexities of that model. The goal of this being to encapsulate in a service layer all the concerns that the other layers need not worry about explicitly.

IO services are common instances of these types of classes. Some examples include file streams, scanner services, or printer services. Any time a complex API is wrapped so as to expose a simple interface for bringing in or flushing out data in the natural format of the application, a stateful class is the most likely mechanism. Looking at the file stream example, the state involved might consist of a buffer, a handle to the endpoint of the output, and parameters regarding how the data should be accessed (such as the file access sharing mode).

Stateful algorithms may take dependencies on any other type of class, but more commonly pure or stateful algorithms. Stateful algorithms are more likely to interact with data classes and models as arguments, or internally in the course of API wrapping.

Models

Models are a diverse, messy bunch. They're probably the closest to fitting the classical "data plus behavior" description. A class in this category will have intertwined state and behavior. The state will typically be of value without the behavior, but the behavior exists only in the context of the state. Most often, a model class comes about not because extracting the behavior and state into separate classes is impossible, but rather because it muddies the expressiveness of the design. Domain models, UI classes, and device APIs are the places where the model category of classes tend to serve best. Note, however, that these spaces also tend to attract convoluted coupling, inscrutable interfaces, unintuitive layering, and a host of other design pathologies.

These warts proliferate because it is difficult to make generalizations or rules about models. The foremost rule I keep in dealing with them is to try to avoid them. Not in the "model classes are considered harmful" sort of way. If you look at a program I've written, you won't find it devoid of model classes. In fact, they aren't even particularly rare. When I say I try to avoid them, all I really mean is that before I write a model class I make sure that the role I'm trying to fill isn't better addressed by some at least relatively simple combination of the other categories.

It's common for the behavior of a model class to depend on some private aspect of the state. Separating the behavior from the state would thus also mean dividing the state into public and private portions, to be referenced by the consumers of the model, and the model's services, respectively. This can be a nightmare to maintain, and the service implementor must often make a decision between downcasting a public data interface to a compound public/private one, or internally mapping public state objects or interfaces to their associated private state objects or interfaces. You see this type of thing crop up repeatedly in ORM solutions, where the otherwise internal state of the change and identity tracking is pulled out of the model and maintained by the ORM.

This separation is difficult and messy to make. If you find yourself facing this choice, you can be fairly certain that a model solution is at least an acceptable fit for the role. There are benefits to the separation, but often the cost of implementation is high, and should be deferred until necessary. But beware as you forge ahead with the consolidated model, because every model is different. I find that the cleanest way to incorporate a model into a project is to establish a layer in which that model and its service classes will live, and make heavy use of data transfer objects and projections to inter-operate with other layers.

A model can, and often does take dependencies on any category of class. Because of the varied roles models can serve and the many ways that state and behavior may be mixed in them, it's very difficult to identify any patterns or rules as far as how dependencies typically arise and how they should be handled. Any such effort would likely hinge on further subdividing the categories into common roles and basing the rules and patterns at that level instead. I'll leave that as an exercise for the reader, for now. =)

A Guideline

As I've said, I find these categories to be a useful guideline in my own programming efforts. It's quite easy and natural to mix state and behavior together in most every class you write, simply because it's convenient on the spot to have behavior neighboring the state it's relevant to. But, it tends to be far easier to reason about the first three categories in the abstract than it is to do so about models. This is likely because the former tend to enable and encourage immutability, which minimizes and simplifies a whole host of problems from concurrency, to persistence, to testing. For this reason I try to identify places that lend themselves well to the extraction of a data class or an algorithm class.

This strategy seems to make my code more composable, and my applications more discrete in terms of purpose and dependency. Which in turn makes the divisions between the different layers and functionality regions more clear and firm. This not only discourages leakage of concerns, but also makes the design more digestible to the reader. And that is always a good thing, even if the reader is just yourself 6 months later.

What Your IoC Container Won't Tell You About Inversion of Control

2010-12-21T09:30:00.004-06:00

As I've written about recently, the term "Inversion of Control" has come to be closely associated with "Dependency Injection". Indeed many people tend to use the two terms interchangeably. The reality is that DI is just one way of achieving one kind of limited IoC.

It's worth considering that something cannot be inverted without there being an original, un-inverted state. So what is that state, and why did it become so important to invert it? According to Martin Fowler's Bliki post on IoC, the state in question is which code is in control of the flow of the program within a particular scope. Fowler credits Ralph Johnson and Brian Foote's 1988 paper Designing Reusable Classes, with originally identifying the boundaries between system layers, or between a framework and the consumer of that framework, as being particularly valuable places to invert control.

The goal of the inversion, as stated by Fowler, Johnson, and Foote, is essentially to let a service remain in control of the pieces of its execution, rather than to micromanage it from the outside, like a boss who insists on being kept "in the loop". I would say this is dramatically different from what IoC has come to mean in the common parlance today.

IoC in its common usage has become divorced from the Hollywood Principle, which was about allowing a service to retain control of its own execution. Instead it has been married to DI, which is about a service divesting responsibility for ancillary concerns. The latter is equally important. But only equally. When they are brought together, they enhance each other and the resulting clarity of the design is more than the sum of its parts.

There is a duality here that is important to keep in mind when designing classes and interfaces. A duality of scope, a duality of perspective, a duality of responsibility, a duality of control. There is both give and take. By giving one thing, we take another. IoC is not just about giving up control and responsibility for choosing and creating dependency implementations. It's also about taking control and responsibility for the cohesion of the solution, the algorithm, and the facade that's offered by a service, a framework, or a layer.

It is far too easy, when one first delves into DI, especially aided by the power tool that is an IoC container, to forget this other side of things. We become satisfied that as long as we don't have a "new" in our constructor, we must be on the right track. But sooner or later you will end up in a situation where the discipline of DI becomes very difficult to maintain. Without the context of where DI came from, the limits of its purpose, and the parallel concerns that were meant to be addressed alongside, it won't be clear what is the proper way to proceed. You may realize that it's time to "break the rules", but you won't know what new rules then take over.

Speaking from experience I can tell you that this way lies madness. Or at least messiness. Potentially maddening messiness. Far too many people simply shed the rules as they pass this boundary and revert back to wild and woolly frontier-style programming. But the truth is that you aren't passing from civilization into the wilderness. You may be leaving one regime, but you're also entering another. And the guiding light of this new regime isn't just "whatever works". Most likely it's simply time to broaden your perspective beyond the issue of class dependencies and Separation of Concerns.

Start to look at the surface of the layer, model, framework, or module that the problem classes are a part of. Consider all of its responsibilities, dependencies, entry points, hooks, and outputs. If you are encountering problems making DI work, it's very possible that a mess has begun to accrete in one of these other aspects. If you step back to get the broader perspective, it may be easier to identify the problem as a flaw in the cohesion of it as a whole.

In particular, look at the ways your objects are working together, and the way the whole layer interacts with its consumers. A clear story should show up somewhere. Yes you may have a hundred small classes running around taking care of little chunks of the work. But someone should be "conducting" these bits as they work together to accomplish the goal, and that someone should be chosen and designed deliberately.

It's a common mistake to break responsibilities down into bits and then just toss them all together with no real clear roadmap emerging from the noise. This is a real problem, and the cause is a proliferation of delegation in the interest of decoupling, combined with a failure to take the reigns of regional control that ensures cohesion. Weaving together the dual needs of decoupling and cohesion is what Inversion of Control is really about.

On the Dimensions of My Ignorance

2010-12-15T09:35:00.001-06:00

Yesterday I posted what some may view as a rant, on how troublesome it is to get started with REST web services in .NET. It wasn't intended to be a rant so much as a cathartic walk through the research I had done and the decisions I had made, over several hours of trying to find an easy on-ramp to REST on .NET.

One of the things I mentioned was the fact that I downloaded the OpenRasta source, but was unable to build it due to a couple compiler errors. I had been in loose contact with Sebastien Lambla, the author of OpenRasta, about this situation over Twitter. Yesterday he read my post and, not undeservedly, jabbed me about not trying to fix the bug myself, instead opting to waste time "touring frameworks" to find what I need.

I got defensive, of course. I felt like I clearly conveyed my situation, and why I went "touring", rather than just settling on one and pushing through the hurdles. But looking back I can see that information is distributed throughout the post, and a skimmer would likely see only that I had looked at, and whiningly dismissed, each option in turn. Which is factual, though not really complete.

So here I will reiterate that I am:

A REST noob
A web services noob
An ASP.NET noob
An OSS noob
A Git noob

And that doesn't even consider Sinatra, which brings to bear the additional facts that I am:

A Ruby noob
A Linux noob

I know my limitations. Or rather, I know where my knowledge ends. The reason I went "touring" is because I have ideas and I want to start to make them real. I didn't want to go burrowing into some rabbit hole not knowing if it was even the right one. I want to find a framework that minimizes the number of dimensions of my ignorance that are likely to limit or undermine my ability to produce something minimally functional.

There are a lot of things I am experienced with and good at, but the list above consists of things that I don't know enough about to know when I am making a mess for myself. Without guidance, I don't even know where to start. When OpenRasta wouldn't build, I didn't even bother to look at the code, because I didn't want to get lost in the weeds due to my multiple dimensions of ignorance. It turned out to be something that I probably could have handled, so shame on me for not at least taking a closer look. I'll take responsibility for that and tuck the critique away for future growth potential.

It's tough enough to be a noob about one thing. But ignorance compounds and grows exponentially. So it's downright painful and demoralizing to be on a path where, every time you hit a roadblock, you don't even know which of the multiple gaping holes in your experience is the culprit. It's far too easy to end up chasing down red herrings or just plain make matters worse for yourself in that situation. And I just want to get stuff done.

Fortunately for me, Sebastien says that the OpenRasta stable branch is actually stable now, so I can take another bash at that. I also learned just an hour after my last post went up that there are two more .NET REST frameworks to check out. RestPoint appears to be a young, attribute-based framework. And Nina claims to be a mature, Sinatra-inspired, lambda-based framework.

I'll admit that I really don't like attribute-based APIs, but compared to the challenges I'd face with all the other options I've looked at, I can probably stomach it. But I am a big fan of lambda-based APIs, and I'm still stoked about the Sinatra approach, so I'm pretty excited to take a look at Nina. It looks like it might just be what I was hoping Nancy would be.

And on a parallel track, I plan to start filling in at least one of my other knowledge gaps: Git. I have the PragProg book on Git, but I think I may also invest in the TekPub videos on Git to get started.

.NET REST Frustrations

2010-12-14T09:30:00.003-06:00

I've had a frustrating few days, and I'm going to vent about it a little bit here, because the experience has been something that I think many programmers have to deal with from time to time.

This weekend I found myself with 2 evenings home by myself, while my wife works her new 2nd shift job. While I miss having my evenings with her, part of me had been looking forward having some evenings all to myself, to really dig into some extra-curricular programming projects.

I'm in an unusual position right now. Unusual for myself anyway: I have a lot of ideas of things to work on. Some are for myself, and some are for work, to get a little extra satisfaction out of the job, and to make things more tolerable during the 9 to 5 when I'm on the hook for some less-than-enjoyable tasks. One thing most all of these ideas have in common is that they revolve around creating web services. Which is something I've never done before.

A few weeks back I wrote a meandering rumination, documenting my thoughts as I ventured into the web services world... Specifically, the REST world. I found myself leaning toward a particular approach, and this weekend I had an the time and opportunity to attempt to start down that road. Unfortunately, I hit some roadblocks. OpenRasta is just not going to work for me right now, I'm afraid. If I'm lucky that will change in the near future, but for now it's the state of things. So today I went back to the drawing board. Starting with whether I should persevere on the goal of using REST.

I didn't spend much time back at the very beginning here. I know that it is fairly easy to whip up an RPC-style web service with WCF. But I also know that RPC is (for most use cases) becoming a far less appropriate option. I've been doing a lot of reading on REST, and it is starting to make some fundamental sense to me. It's a paradigm I *want* to work in because it feels natural in so many ways. But I look at all the resources on REST in WCF and think to myself that it really just seems like a lot more work than it needs to be. Not to mention the fact that the whole point of WCF is to be protocol-agnostic, which is of course the opposite of REST, which embraces the HTTP protocol. Using WCF for REST seems like a frivolous waste of resources: a framework for its own sake, rather than for any value derived from the generality. So, I really wanted to avoid WCF for the learning curve and for the design conflict. But what are my other options?

I've read about how awesomely intuitive and simple the Sinatra framework is for REST, but I don't know Ruby, and really have zero comfort with Linux from a programming, admin, or server standpoint. So Sinatra really just has too many hurdles at this point. That's a lot of learning I'd have to do before I can make a simple hello world web service. Not sure I am willing to wait so long before I get some ROI. I'll learn Ruby properly, eventually. But preferably not when I'm also trying to learn 4 other things from scratch.

There are a couple of .NET frameworks around that claim to be heavily inspired by Sinatra. First there was Martin, and now Nancy. Unfortunately, I can't really find any recent activity on or around Martin. Maybe it was just a whim that didn't get the necessary care and investment.

Nancy looks more promising. It looks like a really nice start. I hope it will be a viable platform in the future, and I'll be keeping an eye on it. But right now it's brand new and not ready for production usage. Which puts it out of the running for all but my personal projects. It seems like a train I'd want to be on, and I'd love to contribute to the effort. But well, here I've got a couple more hurdles. Once again, I'm a web services n00b. I have my sense of design and elegance, and not much else. No real appreciation for the realities and complexities of the problem space or the underlying technologies. This is a journey I'm now looking to embark on for the first time. I'm far from ready to try blazing a trail for others. The second hurdle is that it's hosted on Github, and alas I'm still a Git n00b as well. I have successfully installed Git on one machine, and haven't actually successfully used it a single time yet.

What's left for REST? OpenRasta again. Which I've already established won't work. But why? It looks so... right. I can't just give up, can I? OpenRasta looks really nice in a lot of the same ways as Nancy, except that it's a far more mature product. And the author, Sebastien Lambla, has a ton of OSS experience. It's on Github too, like Nancy, but given a stable branch/build that shouldn't be a problem. But here's where things go bad. The stable binaries seem to have vanished from the place they are supposed to be, at ohloh.net. And the two ostensibly stable branches are currently in a state of... instability... as in they won't build. None of which contributes to a n00b-friendly situation. So here we are indeed back to the same obstacles I have with Nancy.

It's important for me to say that I don't begrudge Sebastien or his OSS compatriots for this situation. He's got a lot of irons in the fire with his OSS efforts, and a lot of people with more important complaints than mine are looking for their issues to be attended to. But it makes me sad, because this seemed like my best bet, and I feel like it's hanging there taunting me just outside my reach.

So... I'm coming out of the weekend very discouraged. I have all these things I want to do, and I feel like I'm being foiled at every turn by a combination of my own ignorance and unfortunate circumstance. My ideas are frozen until I can overcome some of these challenges, but right now I feel like I really need the motivation of progress on these ideas to propel me through the work of ramping up on... well, on anything. I have enough other things to do with my time that it's a challenge to prioritize learning new tech if I won't also be making progress toward producing something.

I feel like I have to make a choice between doing things the way I want and having to spend a long time digging through the muck of learning unfamiliar territory compounded with the pitfalls of immature tools, or I can compromise my design sense and go with WCF in the interest of making some material progress on my ideas.

At least WCF is a place to start where there is an established "on ramp" with controlled unknowns. I could get some preliminary, controlled exposure to the problem space and the technology stack while learning WCF, while still preserving a safety net of sorts, in the form of MS's mature and thorough framework support. Then once I have cleared those learning curves, I can revisit the question and do things the way I'd prefer to next time. So alas, that is probably what I will do.

It's just very discouraging to end up back at the place I started. The one place I didn't want to be. But that's life. Sometimes you can't fight the constraints. Sometimes you have to accept them and just do what you have to "ship", even if it means choosing a technology you'd rather not rely on.

Taking IoC beyond DI with Autofac Part 2: Relationships

2010-12-07T09:30:00.006-06:00

In my last post I began to discuss the applications of Autofac as a tool for accomplishing true Inversion of Control, beyond just simple Dependency Injection. Specifically, last time I focused on creation strategies and lifetime control.

This week I want to talk a little more about lifetime control, and a lot more about categories of dependency, also known as relationship types. Nicholas Blumhardt, the principal author of Autofac, has a great blog post on what he calls “the relationship zoo” which I think goes a long way toward covering this space. I was originally going to do something similar, but his post is far more authoritative, and I’m quite certain I couldn’t do better. So instead, I'll abstain from the explanation and code samples and stick to analysis and rumination. So go read his post, then please do come back here and I’ll expand on it with some of my own thoughts.

…

The reality is that I take issue with some of the dependency abstractions provided by Autofac. It can be quite dangerous to your design to rely on some of them without very good reason. Used properly and prudently, they can absolutely address some particularly painful problems, especially when the dependencies consist of code you don't own and can't change. But it’s also easy to let them creep in wherever they appear to be convenient, and severely corrupt your design and architecture by doing so.

Let me start with the warnings, and then I’ll move on to extoll some virtues that I think Nicholas neglected to identify.

The first relationship type to be wary of is Lazy<T>. The intended purpose, as Nicholas explains, is to avoid expensive operations or construction until or unless it’s necessary. The idea of a lazy object is an old one. It’s a pattern that has been around for a while. My opposition to the usage of this relationship type is primarily that the need for laziness is usually a function not of the usage by the dependent class, but rather of the implementation of the module that is depended upon. Where possible, the consumer should be ignorant of implementation details of its dependencies. Let’s recognize that expensive operations are rarely truly transparent or ignorable, but the fact remains that the responsibility for this situation lies with the module being depended on, not on the consumer.

I strongly believe that if the code of the module that will be resolved for this dependency is under your control, then it behooves you to wrap the laziness around this functionality elsewhere... either building it into the implementation, or wrapping it with some sort of facade. Where Lazy<T> comes in handy is when you don’t have control over this code, and the class poorly encapsulates its functionality such that it can’t sufficiently be wrapped in a facade. At that point the consumer cannot pretend to ignore the situation, and may benefit from delegating the responsibility for the laziness to the container.

I’ll attach my second warning to the Func<T> relationship. I’m wary of the plain Func<T> because it overlaps a great deal with both Lazy<T>. It shares the same issues as Lazy<T> while adding the sin of looking suspiciously like a service locator. Service locators can be dangerous because they subvert the goal of inverting control. A service locator hands control back to the consumer by saying, “just call when you need X”, rather than handing off the dependency and saying “I know you need X so here, use this for that”. This is very rarely the appropriate way of expressing the relationship. The exception would be in the case that the consumer knows for certain that it will require multiple instances of the service it depends on.

Let’s spin things back to the positive by looking at Func and the like. How does adding the extra generic arguments change this from a smell to proper IoC? It expresses a very particular type of dependency. Specifically, it says that this class depends on at least one instance of T, but which instance or instances are needed is a function of a few arguments which won’t be known until each instance is actually needed. This is useful if you have a service which must be constructed for each use and requires some primitive types to guide its behavior, such as an encryption object which requires a key at construction. Or if you have a few different implementations of the a service which are mapped to different run-time states, such as a set of logging mechanisms for different error severities.

The IEnumerable<T> relationship is similar to this last scenario in that it offers a way to say “I depend on all implementations of T, no matter what they are”. This is probably a rarer scenario. Usually a service will have one key implementation, or a couple with very specific and differing purposes which will be used separately in different places. The most likely way for an inclusive but generalized need like this to arise is in an add-on or plug-in scenario. And in that case, you’re likely going to need some up-front processing to get things loaded up before the implementations can be passed off as dependencies to other objects.

I should note that it is probably not all that unusual for IEnumerable arguments to show up in constructors. But more often than not, this will be in data classes which will be instantiated directly, or a single step removed via a factory method, rather than resolved by the container. These aren’t truly “services” and are very unlikely to be registered with the IoC container. The factory may be a service, but what it creates in this case is more of a data structure than anything. Data structures mean state, and usually highly contextual ones at that. Autofac more context-sensitive than many IoC containers, but even it has its limits. With data structures, usually the best solution is either a direct constructor call, or a light factory.

One more small step along the path is Meta<T, M>. This is relationship type that specifies a dependency not only on a service of type T, but on some piece of metadata of type M about the module that will be provided at runtime. This metadata may be used to decide whether or not to go through with making use of the service, or how to do it. In fact, metadata is a great way to handle the special considerations a consuming object may need to make for a lazy service implementation involving a long-running operation. Maybe 90% of the time, the app can simply halt for an operation, but for certain implementations of the service, it’s more prudent to display a friendly “please wait” message and a progress bar. Attaching metadata at registration is a great way to enable these types of decisions intelligently without hard-coding a universal expectation one way or the other.

The final relationship type to address is Owned<T>. This one can be quite handy. At first blush it seems like this might violate the same principles as Lazy<T>, but if you think about it, they are actually quite different. Owned<T> indicates that the object expects to be solely responsible for the fate of the dependency passed to it. This tells the programmer, “I’m going to use this up, and it won’t be any good when I’m done with it, so just clean it up afterward.” Believe it or not, this fits in perfectly with the recommended implementation pattern for the .NET IDisposable interface. That is, that at some point, some object takes ownership of the resources, and responsibility for calling IDisposable, absolving all other transient handlers of the same. Ownership is sort of the dual, or inverse, of a constructor dependency. The constructor dependency says “I know I’m going to need this object”, and the ownership claim says “when I’m done with it, it can be tossed out.” And Autofac happily obliges, releasing the dependencies when the consumer itself is released, calling IDisposable.Dispose as appropriate.

After a twitter conversation with Nicholas himself, it became obvious to me that Owned<T> is probably the better solution to my qualms about InstancePerLifetimeScope. By establishing that the consumer “owns” its dependencies, we have essentially established exactly the limited and definite lifetime context that I was asking for! Behind the scenes, a lifetime scope is spun up when the object is instantiated, with nested resolution rules being applied as a matter of course. And when the object is released, then so is the lifetime scope and anything that was resolved as a new instance in that scope. However, we do have a symmetrical limitation here. Both the creation and the destruction of the context are tied to this object. Once the context is created, it can’t and/or shouldn’t be shared with anything either above or adjacent to this object in the dependency graph, or the deterministic disposal Autofac prides itself on will subverted and become unreliable.

In these past two posts, I’ve covered the whole continuum of dependency creation, lifecycle, and relationship strategies that go beyond simple Dependency Injection to fill out the breadth of what Inversion of Control really means. And they’re all available via Autofac to be handled (for the most part) separate from the implementation of the consumer, and without writing boilerplate factory classes or resource management classes to make it work. I hope that in reading these posts some people may see that IoC is a broad space of patterns and solutions, and that IoC containers are powerful and useful and far beyond the naive use and even abuse that people put them to for simple DI.

Taking IoC beyond DI with Autofac Part 1: Lifecycle Control

2010-11-30T20:19:00.008-06:00

People who have to listen to me talk about programming know that I’m a big proponent of Inversion of Control (IoC) containers and what they can do to clean up your code. Most people get introduced to IoC containers via Dependency Injection (DI). DI is another great way to clean up your code. It’s hard to argue against it, really. It decouples your code in a big way. Not only does this make it more testable, but because it aids in separating concerns/responsibilities, this also makes it easier for you to track down bugs when they do show up. But people rightly point out that you don’t need an IoC container to use DI and get these benefits.

I’m usually both happy and sad to hear that argument. On the positive side, it means that people are acknowledging the benefits of DI, which is great. The more people are using DI, the less god classes full of spaghetti code there are out there for me to unearth in future maintenance efforts. Another reason I’m happy to hear that argument is because it means that people aren’t confusing the means for the end. DI is a good thing, for the reasons I established above, not because, as some people seem to think, IoC containers are good and DI is what IoC containers do.

That last sentence there leads into the reason that I’m sad to hear people dismiss IoC containers as being unnecessary for DI. The problem is, DI isn’t the only benefit of IoC containers. DI isn’t what IoC containers do. IoC containers, as their name would indicate, invert control. They take a number of concerns that have traditionally been assigned to the class under consideration, and extract them out to the context in which that class is used. That context may be the immediate consumers of the class, or it may be coordinating code, infrastructure, data access, or any number of other locations in the application. But the point is that the responsibilities are removed from the class itself and given to other classes whose business it is to know what should be created, when, with what initialization, and how it should be disposed of.

A good IoC container does more than just wire up constructor dependencies. It goes beyond that and lives up to the breadth of this definition of IoC. And that is why I laud and evangelize the glories of IoC containers.

Lets take a look at one particular container that I’ve come to know and love: Autofac. It’s an amazing framework that offers an answer to nearly everything that the principles of IoC ask of it. Autofac has features for lifecycle control, object ownership, and deterministic disposal. It has features for nested scoping, contextual dependency resolution, and varied construction mechanisms. These are all concerns that are a function not of the consumer of a dependency, but of the cloud of code and functionality that surrounds it, of the nature and design of your application as a composition. And Autofac gives you ways to deal with them on those terms.

Autofac boasts strong support for robust lifecycle control. In the project wiki you’ll find it laid out under the topic “Deterministic Disposal”, but it’s about creation as much as it’s about disposal. When you register a module with an Autofac container, you have the opportunity to specify a scope strategy. This strategy will determine whether a new instance is created upon the request, or whether an existing one is pulled from the container. Furthermore, it will also determine when references to the instance or instances are released and Dispose called on IDisposables. I’ll be doing some explaining, but if you want to do your own reading on the available strategies, you can do so here: http://code.google.com/p/autofac/wiki/InstanceScope

The two simple lifetime scopes that everyone tends to be conceptually familiar with are found n Autofac’s SingleInstance and InstancePerDependency scopes. The former is roughly equivalent to the function of a singleton pattern implementation, while the latter corresponds to a factory pattern implementation. Autofac goes well beyond this, however, and gives you two more scoping strategies that let you manage creation and disposal of your components in a more nuanced and more powerful way.

Both of the two more nuanced scope strategies depend on Autofac’s support for “nested containers”. A nested container is essentially a scoping mechanism, similar to a method or class definition, or a using block. Nested containers are useful for establishing the architectural layer boundaries of your application. For example, in a desktop application you may have many windows coming in and out of existence, all operating on the same domain objects, persisting them via the same handful of repository classes. These windows may be created in nested container contexts that are spun up as needed, and disposed when the windows are closed. Some objects will be created new each time this happens, while others are unique and shared across the entire UI. The nested container is what allows Autofac to make the appropriate distinction.

Imagine you are writing a file diff’ing application. You have to show two documents at once in side-by-side windows. They are the same thing, in terms of functionality and data, and so could just be two different instances of the same object.... But they will share some dependencies, and have references to their very own copies of certain others.

Lets pick out a few pieces of this puzzle and tie them to Autofac’s features. You will probably have a file access service that allows you to operate on the files that you have loaded. There’s no reason to have more than a single copy of this in the entire app, so it can effectively be a singleton. The way you would express this to Autofac is via the registry.

Given the class and interface definition:

You would register the component as a singleton like this:

The run-time behavior you would see based on this registration is that only one instance of the FileAccess component will ever be produced by the container. Subsequent requests will just return the one that’s already constructed.

The next layer on top of that is the UI. You decide that you may want to be able to have multiple comparison windows open at once, without having to run separate instances of the app. But each of those windows is still essentially a full instance of your apps interface. They’re not exactly singletons, but they should be unique within their separate stacks. Whatever sub-context they are in, there should be only one.

Given the class and interface definition:

You would register the component using the InstancePerMatchingLifetimeScope strategy, like this:

One weakness of Autofac for this use case is that in order to establish the proper resolution contexts, you have to refer directly to the container. In this case it’s probably okay, since you can be fairly certain you’ll only ever need a Left context and a Right context. So you can probably create the lifetime scopes up front, store them away somewhere, and explicitly resolve these objects from the separate contexts when needed. A sample setup for this can be seen below.

Then you’d need to make sure that the LeftContainer and RightContainer were used explicitly to resolve the left and right ComparisonWindow components.

This works for us in this situation. But it’s not at all difficult to imagine a scenario, maybe even in this same app, where the contexts aren’t predetermined and static. For example, you may want have a worker thread pool, where each thread has its own resolution context. In fact this is a situation addressed explicitly in the Autofac wiki. Even there, it seems to be accepted that an explicit container reference in the thread pool is necessary. It doesn’t seem like there’s a great solution to this challenge at this time, though I will surely be keeping an eye out for one. This is messy, concern-leaking infrastructure code that I would really prefer not to have to write.

There’s another scoping type that’s related to this one. In fact it’s use is a bit simpler. This is the InstancePerLifetimeScope strategy. Note the subtle lack of the “Matching” adjective in the name. What this indicates is that the context is implied rather than explicit. The behavior specified by this strategy is that at most one instance will be created within the resolution context where the resolution happens, at whatever level of nesting that happens to be. Functionally, this differs from InstancePerMatchingLifetimeScope in that it doesn’t search the context stack for one particular context in which to do the resolution.

This strategy can be effective when you have a set architecture with a trivial tree structure. Well-defined layering is crucial. All the leaf nodes of your context tree need to be at the same depth in order to be certain when a new instance will be created and when not. For example, a website where you have a base layer for the whole web app, and a leaf layer nested within for each individual web requests. In our diffing app, if we can be certain that we need no more deeply nested containers beyond our LeftContainer and our RightContainer, and all significant service resolution will happen at those layers, then we may have a use for this strategy for the dependencies of our Left and Right windows and controllers

The registration for this strategy is very similar to the others. Given a class and interface definition such as this:

The registration would look like this:

The final scoping strategy is InstancePerDependency. As noted before, the behavior is roughly equivalent to an implementation of a factory pattern. Every resolution request for a service registered as InstancePerDependency will result in the creation of a new instance. In our diffing app, we may find use for this strategy with something like an alert dialog. There’s no need to keep an alert around when it’s not being shown, and in fact it should almost certainly *not* carry any state from one alert to the next.

So given a class and interface such as this:

The registration would look like this:

That covers most all of Autofac’s lifecycle control functionality. There are four strategies available: SingleInstance which approximates Singleton, InstancePerDependency which approximates Factory, and the more subtle and, honestly, difficult to use, InstancePerLifetimeScope and InstancePerMatchingLifetimeScope which are heavily contextual. I really wish that these last two were more manageable and directable. If they were, I think that Autofac could claim to easily address most any lifecycle control need with very little overhead and requiring few concessions to the framework. This would be a very noble goal. But as it is, their behavior will tend to be circumstantial rather than controlled and intentional. And the only hope of improving that situation lies in taking great pains to organize your design to account for the container’s shortcomings and then go on to break the rule of not referencing the container.

Despite these shortcomings, I believe there are many benefits to be found in relying on Autofac for lifecycle control where it’s possible and not overly problematic. Certainly I’ve saved myself some headaches in doing so. And we haven’t even begun to explore the dependency relationship patterns that Autofac supports out of the box. We'll dive into to those next time!

A Noob's Scattered Thoughts on REST

2010-11-23T10:00:00.008-06:00

Note: I hope you'll bear with me as I meander through my still very disorganized thoughts on the topic. This post is very much for myself, with the goal of distilling the information floating around in my head into something more concrete and which I can actually make use of...

I have a few projects on the horizon, both at work and on my own, which will involve web services in some form or another. I have to admit, I've never been much of a "web guy" before. My career to this point hasn't demanded that this change, but it looks like it's time for me to move off the desktop at least part-time. So I've dived into web services.

I develop for .NET at work. So while I may explore something else for my personal projects, for work I have to focus on .NET frameworks. Of course REST is all the rage these days, so it's obligatory that I research that. So I've read a number of blog posts describing what REST is about. My assessment of the REST is that it seems like a nice idiomatic design philosophy for a web service that deals primarily with persistent data entities and operations over them.

It seems clear to me though that there must be problems which might be addressed via a web service which aren't necessarily natural to model using REST. Of course the fallback, if one were to decide that the barrier is too great, would be to use some sort of RPC style. RPC comes natural to us developers, of course, because it's essentially just a web-enabled version of the idioms we use ubiquitously in code.

My own feelings about RPC as compared to REST are less critical than many. I understand the philosophical imperative to maintain the webbishness of services operating over the web. However, I also hold very strong to the opinion that solutions should be created using metaphors that lend themselves naturally to the problem space. REST, as defended by many, seems to me to be less about the problem space than it is about the mechanism the solution is built on.

However, if REST can be said to consist of two pillars, resource-orientation and mapping operations to the HTTP verbs, then it seems the later is really the part that seems limiting. After all, a resource is not so different from the "objects" that permeate our code as programmers. But imagine if, in your object-oriented design, you were allowed only 4 methods per object, and each of them had to fit one of four very specific patterns. That feels arbitrarily limiting, doesn't it? It does for me.

Now, having said that, I do see value in keeping things simple. The web is already a high level abstraction. And building convoluted metaphors on top of it that don't map easily to the mechanisms that underly it can cause a lot of unnecessary headaches. I also believe that it may not be unfair to compare RPC-heavy designs to the old style of procedural programming. That is, that they have all the nasty kinds of coupling and inertia, and few of the good kinds. Especially when the data entities involved are mutable.

That last point hits home hard for me, though. And I think it will end up strongly informing how my designs play out. The reason for this has largely to do with how my overall coding style has evolved over the past couple of years. I've found lately that, with the exception of DTOs and the primary domain objects in my solutions, a great many of the objects I work with tend to be immutable. And the algorithmic bits that operate on them tend to come in nuggets of functionality that are highly composable. This strikes me as being fairly compatible with REST's resource-oriented design.

Another factor I have to consider is the frameworks that are available to me. Comparing what I can find on the web about WCF and ASP.NET, the primary non-REST web service frameworks, and OpenRasta.... I have to say that the code and configuration of OpenRasta solutions seems to be much simpler, more elegant, and clearer of intent. This appeals to me greatly. It seems like it would be much easier not only to spin up something from scratch into a working solution, but also to evolve smoothly from my first fumbling attempts through to a final product that still has clarity of structure and intent. If I've learned anything in my career so far, it's the importance of that.

So my initial forays, at least, will likely be based in OpenRasta. If it turns out that OpenRasta can't give me what I need, or that there's too much friction, I'll look for help of course. But I also won't be afraid to try to weave some RPC into my design if it's called for.

Clean Injection of Individual Settings Values

2010-11-15T21:19:00.014-06:00

The Setup

Today I again encountered a challenge that I have dealt with numerous times while working on the product I develop for my employer. It's not insurmountable an insurmountable challenge. Honestly, it's not even all that challenging. But it has been an irritant to me in the past because all the solutions that I have come up with felt unsatisfactory in some way.

That challenge is the injection of simple settings values into the classes that need to refer to them. By simple I mean single-value settings embodied by an integer, a string, or a floating-point, for example. This seems like a simple problem. And honestly, I thought this was going to be a short and sweet post, until I realized that there is value in delineating all the dead ends I've followed on the road to my current favored solution. As you'll see, I've put a fair amount of thought and analysis into it.

Being a hip programmer, I use an IoC container to constructor-inject all my complex dependencies. And my IoC container of choice, Autofac, is only too happy to auto-wire these so I don't have to worry about them. But mixed in among these are the simple settings values that guide the functionality of certain classes. A component/service model doesn't really apply to these values. No one string value is implementing the "string" service. The values can't be easily auto-wired because unlike the complex classes there is certain to be more than one important value floating around for the primitive type in question.

Our app processes image. Large files, and large volume. And worse, the working sets are very large, so we can't handle things piecemeal. We have a serious need in a few different places for disk caching of "active" entities that are still very much in flux. Having multiple caching points necessarily means that there are several parameters that may need to be tweaked to keep things optimized. For each cache we want to be able to set the disk location for the cache and the in-memory entity limit, just to start.

Taking a simple case with two cache points, we have two string settings and two integer settings. So already we are in a position where in order to inject the values, we'll need to do some sort of "switching". Some run-time decision of which string and which integer go where.

Pitfalls and Red Herrings

As I noted in my opening, I have solved this in several ways in the past. One is to hook into the IoC via Autofac's "OnPreparing" event, where I can supply values for particular constructor parameters. This is nice, because it means I can avoid setter injection. But it complicates the IoC bootstrapper by adding exceptions and special handling for particular classes. Just as undesirable, it couples the IoC bootstrapper directly to the settings mechanism.

What about setter injection? Autofac provides a post-construction OnActivated event that is perfect for setter injection, but this is subject to the exact same disadvantages as the pre-construction event. We could leave the setters alone and let some other object fill them in, but that leaves us with a couple different problems. First, there's just as much coupling, it's just outside the IoC, which may or may not be a marginal improvement depending on how it's implemented. If you end up with some class that must be aware of both the app settings mechanism and the class(es) that receive those settings then this is really not much of an improvement.

But beyond that, refraining from providing the values until after the components are obtained is undesirable for yet a few more reasons. First and foremost, it means that your services will exist in a state of incomplete initialization. The risk of getting hold of an incompletely initialized service makes calling code more brittle. And protecting against the possibility makes it more complex. Furthermore, setter injection for these particular types of values is undesirable because it implies they are variants. The truth is that the last thing you want is for some errant code to change the cache location on disk after a bunch of files have been stored there. And putting in protection against such post-initialization changes is pathologically unintuitive: it subverts the very nature and purpose of a setter.

So we've established that these direct injection routes are problematic in a number of ways. Let's move on to indirect injection. What does that mean? Basically it means putting a provider object in the middle. Our classes can take a dependency on the setting provider, which can wrap the settings mechanism itself, or act as a facade for a bundle of different mechanisms.

The option that at first appears simplest is to have a single settings provider object through which all the app's settings can be accessed. The classes can all depend on this object, which the IoC can provide with a singleton lifecycle if we desire, for maximum consistency. But now what we have essentially done is created a service locator for settings. This is another thing that's good to avoid for two reasons. For one, it creates a huge common coupling point, and for two, it violates "tell, don't ask". Why should my dependent class have to depend on a generic interface and worry about asking for the appropriate thing, when all it cares about is just that one thing?

This is especially dangerous if the app-side interface to your settings keeps them grouped just as they are in the user-side (i.e. the config file), as the build in .NET config mechanism is wont to do. The needs of the user for the purpose of managing settings individually or en masse are vastly different than the needs of the application whose behavior is driven by those settings. While a user likely thinks the most intuitive arrangement is for all the 15 of the paths to be bundled together, it's highly unlikely that any particular class in the application is going to care about more than one or two individual paths. And if the class doesn't need them, then they shouldn't be offered to it.

A Light in the Dark

So where do we go from here? If you can believe it after all this meandering and rambling, we're very close. From here we take a little tip from the DDD community: eschew primitives. If you think back to the beginning, the whole problem centers on the fact that primitives are just too darn generic. The type doesn't mean something specific enough for it to be an indicator of what exactly the dependency is. How do we fix this? Encapsulate the individual setting in a type specific to that need. Given that the explicit purpose of these classes will be to provide particular settings to the classes that need them, it is appropriate for these to couple to the configuration mechanism, whatever that may be, and more importantly, encapsulate it in useful bite-size chunks. And because the providers will themselves be injected where needed, the coupling is one-way, one level, down the layer hierarchy, which is arguably the best kind of coupling.

Show Me The Code

Enough talking. Now that I've set the stage, here's some code to furnish and light it.

First, the setting providers. As you can see, they tend to be nice and short and sweet.

Next, the caches that depend on them. Note how the setting providers are used in the constructors.

Finally, I'll show just how easy it can be to wire these up. If you bundle all your setting providers in one namespace, you can even safely auto-register them all in one fell swoop!

Objections?

There are a few things that I can anticipate people would object to. One is the potential for proliferation of tiny classes. I don't see this as a bad thing at all. I think it's fairly well established that small classes, and methods, with laser-focused responsibilities are far easier to maintain, evolve, and comprehend. I can say from personal experience that I am utterly convinced of this. And if anecdote isn't good enough for you, I'll add an appeal to authority to it =) Go read up on what the most respected programmers out there today are saying, and you'll see them express the same sentiment, and justify it very well.

Another thing I expect people to object to is that when taken as a whole, this pile of small classes looks like a bit of a heavy solution. And it is heavy in this context, where none of the actual app code is surrounding it. But nestle it inside a 50K line desktop app with hundreds of classes and it will start to look a lot better. For one, those classes and their namespace create sort of a bubble. It's a codespace that has boundaries and purpose. You know what's inside it, and you know what's not. It's a goal-oriented mental anchor to latch onto while you code, and that's a darn useful thing.

Community Service

2009-06-01T11:12:00.006-05:00

As I see it, programming questions tend to take one of two forms:

How do I do X?
How do I do X in a clean, maintainable, and elegant way?

Matt Gemell has a great response to questions of the first form. I have nothing to add to his excellent post on that topic.

But I think it's crucially important to make a distinction between the two forms. The second form is a far more difficult question, asked far less commonly, and to which good answers are even rarer. When questions of this form are asked in forums, people are usually given a link to a page describing a pattern, or if they are really "lucky" a page with a sample implementation of that pattern. To be fair, patterns usually are the answer to these types of questions. But what we find written up on the web, or even in books, is most commonly pathetically oversimplified, without context, and often even without guidance on what support patterns are necessary to obtain the benefit or when alternatives may be preferable.

Essentially, most developers are left with no choice but to apply Matt's answer to form 1, to questions of form 2, in a much less information-rich environment. I contend that, while it may be one of those proverbial activities that "build character", it is ultimately more likely to be harmful to their immediate productivity--possibly even to their continued professional growth.

What we end up with is a pandemic of developers trying to hack out pattern implementations, being discouraged by the fact that the pattern seems to have no accounting for any possible deviations or complications in the form of the problem. Worse, developers are often dismayed to find that the one pattern they were told to use is merely the tip of a huge iceberg of support patterns without which the first may actually be more problematic than an ad hoc solution. Most often the developer in this position will end up determining that their deadlines will never allow them to go through the painful trial and error process on every one of these patterns, and accordingly drop back to the ad hoc solution.

It's time that we acknowledge that software development, whether you consider it an engineering discipline, an art, or a craft, has a history--albeit a short one. Things have been tried. Some work, some don't. There do exist "solved problems" in our problem space. To say that every developer should try and fail at all these efforts on their own ignores and devalues the collective experience of our community. Worse, it stunts the growth of the software development industry as a whole.

Yes, one can learn these things by trial and error. Yes, the understanding gained in this way is deeper and stronger than that gained initially by being tutored on how to apply the solution. And yes, there's a certain pride that comes with getting things done in this way. But this is not scalable. Putting each person forcibly through this crucible is not a sustainable strategy for creating experienced, productive, wise programmers. Every hour spent grappling in isolation with a question of "why won't this pattern do what I've been told it will" is an hour that could be spent creating functionality, or heaven forbid solving a new problem.

That is why those of us who have managed to obtain understanding of these "solved problems" must be willing to shoulder the responsibility of mentoring the less experienced. Being willing to explain, willing to discuss specifics, mutations, deviations, exceptions. Willing to discus process, mindset, methodology. These are the things that make the distinction between a programmer and a software developer, between a software developer and a software engineer.

The internet may very well not be the appropriate place to seek or provide this type of mentoring. I suspect it's not. And unfortunately, there are too many development teams out there buried in companies whose core competencies are not software, and consisting solely of these discouraged developers, lacking an experienced anchor, or even a compass. There are online communities that attempt to address the problem at least partially. ALT.NET is one such, for .NET technologies. But there really is no substitution for direct, personal mentorship.

So I would encourage young developers out there to seek out more experienced developers, and ask them the tough, form 2 questions. And I would even more strongly encourage experienced developers to keep a watchful eye for those in need of such guidance. Maybe even consider proactively forming a local group and seeking out recruits. Be willing, able, and happy to provide this guidance, because it benefits all of us. Every developer you aid is one less developer creating an ad hoc solution which you or I will be condemned to maintain, overhaul, or triage somewhere down the line.

Incidental Redundancy

2009-02-18T11:17:00.002-06:00

A note: I originally composed this blog post in June 2008, but lost it in my backlog and it never got posted. In the interim, Robert C. "Uncle Bob" Martin has addressed the issue in his Clean Code Tip of the Week #1. He describes the issue and the dilemma far more concisely than I have here, and even provides a strategy for dealing with it. By all means feel free to skip my post here and consider him the "official source" on this topic. =)

I am a big fan of "lazy programming". By that of course I mean avoiding repetition and tedium wherever possible. I mean, that's what most programming is really about, right? You have a problem: some process that is annoying, unwieldly, or even impossible to perform with existing tools. And the solution is to write a program that takes care of all the undesirable parts of that process automatically, based on a minimized set of user input or configuration data.

The realities of programming on an actual computer rather than in an idealized theoretical environment rarely allow this ascending staircase of productivity to be climbed to the top step. But it is nevertheless a goal we should always strive for.

Hence Jeff Atwood's recent post about eliminating redundancies.

Anything that removes redundancy from our code should be aggressively pursued

An honorable goal, to be sure. But no rule is without exception, right?

I humbly submit to you the concept of "incidental redundancy". Incidental redundancy is a repetition of code syntax or semantics that tempts the programmer to refactor, but if carried out the refactoring could damage the elegance and discoverability of the program.

The difference between incidental redundancy and regular redundancy in code is that the redundancy does not arise because of any substantive, or at least relevant, similarity between the two problems in question. Here are two ways I can think of off the top of my head for this to happen:

The solutions you have come up with to this point for each situation just happen to have taken a similar form. Given a different creative whim, or even just an alternative refactoring, the commonality may never have arisen.
The problems are, in fact, similar at some level. But the level at which they are similar is far above or below the level where you are working, and so not truly relevant or helpful to the immediate problems you are trying to solve.

The first situation should be acceptable enough to anyone who has spent a decent amount of time programming. Sooner or later, you will spend some precious time carefully crafting a solution to a problem only to later discover that a year and a half previous, you had solved the same exact problem, in a completely different and maybe even better way.

The second situation may sound a little incredible. But allow me to point out an example from the world of physics. Please bear with me as this is an area of personal interest, and it really is the best example that comes to mind.

There are four forces in the known universe which govern all interactions of matter and energy, at least one of which I'm sure you've heard of: the electromagnetic, weak nuclear, strong nuclear, and gravitational forces. It is known now that the first two of those apparently very different forces are in fact two different aspects of the same phenomenon (the electroweak force), which only show up as different when ambient temperature is comparatively low. Most physicists are pretty sure that the third force is yet another aspect of that same phenomenon that splits off only at even higher temperatures. And it is suspected that gravity can be unified with the rest at temperatures higher still.

The point of all this is that there are four different phenomena which in fact bear undeniable similarities in certain respects, and these similarities continue to drive scientists to create a generalized theory that can explain it all. But no one in his right mind would try to design, for example, a complex electrical circuit based entirely on the generalized theory of the electroweak force.

The analogy to programming is that, were we to try to shift up to that higher level and formulate an abstraction to remove the redundancy, the effect on the problem at hand would be to make the solution unwieldly, or opaque, or verbose, or any number of other undesirable code smells. All at the cost of removing a little redundancy.

What we have in both physics and in our programming dilemma, is noise. We are straining our eyes in the dark to find patterns in the problem space, and our mind tells us we see shapes. For the moment they appear clear and inevitable, but in the fullness of time they will prove to have been illusions. But making things worse, the shadow you think you see may really exist, it's just not what you thought it was. That's the insidious nature of this noise: what you see may in fact be truth in a grander context. But in our immediate situation, it is irrelevant and problematic.

This concept is admittedly inspired heavily, though indirectly, by Raganwald's concept of "incidental complexity", where a solution takes a cumbersome form because of the act of projecting it upon the surface of a particular programming language, not unlike the way the picture from a digital projector becomes deformed if you point it at the corner of a room.

The real and serious implication of this is that, to put it in Raganwald's terms, if you refactor an incidental redundancy, the message your solution ends up sending to other programmers, and to yourself in the future, ceases to be useful in understanding the problem that is being solved. It starts sending a signal that there is a real and important correlation between the two things that you've just bound up in one generalization. When in fact, it's just chance. And so when people start to build on top of that generalization with those not quite correct assumptions, unnecessary complexities can quite easily creep in. And of course that inevitably impacts maintenance and further development.

Noise is ever-present in the world of programming, as in other creative and engineering disciplines. But it doesn't just come from the intrusive environment of our programming language or our tools as Raganwald pointed out. It can come from our own past experience, and even come from the problem itself.

So be wary. Don't submit unwittingly to the siren song of one more redundancy elimination. Think critically before you click that "refactor" button. Because eliminating an incidental redundancy is like buying those "as seen on tv" doodads to make your life "easier" in some way that was somehow never an issue before. You think it's going to streamline some portion of your life, like roasting garlic or cooking pastries. But in your quest to squeeze a few more drops of efficiency out of the tiny percentage of your time that you spend in these activities, you end up out some cash, and the proud owner of a toaster oven that won't cook anything flat, or yet another muffin pan, and a bunch of common cooking equipment that you probably already own.

So remember, redundancy should always be eliminated, except when it shouldn't. And when it shouldn't is when it is just noise bubbling up from your own mind, or from the problem space itself.

The First Rule of Extension Methods

2009-02-12T09:49:00.007-06:00

Extension methods are a tremendously useful feature of C# 3. Briefly, they allow you to bundle new behavior into an existing class that wasn't included by the class's original author, but without opening up the implementation internals of the class. In a very general sense, this is useful if you come up with useful behavior related to a class and the best place for it is IN that class, but you don't want to bend or break encapsulation by inheriting. I won't spend any more words on an introduction, but rather offer a caution, for those of you who have seen their usefulness and would like to start taking advantage.

Before you even begin to think about whether a particular behavior belongs on an existing class, or as a service, or what have you, you should internalize one cardinal rule of dealing with extension methods. Thankfully this First Rule of Extension Methods is nothing like the First Rule of Fight Club, because if it were, I wouldn't be able to help you at all. No, the First Rule of Extension Methods is: DO NOT export extension methods in the same namespace as your other public classes.

The reason for this is very very simple: odds are very good that if you have come up with an idea of including a particular new behavior on an existing class, someone else has or will as well. There will only be a finite number of relevant names to give this new method, which means that if it's a simple method, especially one without parameters, there's a decent chance that the signatures of your method and this other programmer's will be identical. And if both you and this other programmer happen to include the method on public static classes in a namespace with other functionality the user needs, then someone trying to use both libraries could very likely run up against identifier collisions with the two methods.

Note that the rule begins with "do not export..." This is important, because you can be 100% certain to save your users any collision headaches if you just don't export your extension method. Why wouldn't you export your extension method? Well, there's a very good chance that just because you found your extension method to be useful and, dare I say, clever (Tyler Durden: "how's that working out for you?"), that doesn't mean the consumer of your library will. So consider carefully whether you should even make the method public at all, or rather just keep it internal.

If you decide that your method is just so darned handy that to keep it locked up would just be sadistic, then make a separate namespace for each batch of related extension methods you want to export. If the user decides they'd like to use them too, they can import the namespace as necessary. This will minimize the chance that they will need to fully qualify the namespaces of the other library with the identically named method whose author wasn't as responsible as you were and polluted their namespaces with extension methods.

This tactic won't guarantee anything, because there's always a chance some other library could use the same namespace identifiers as well. But with each additional namespace, you dramatically increase possible naming permutiations, and proportionally decrease the chance of collisions.

Useful Extension Methods 1 through 3 of N

2009-02-04T15:33:00.011-06:00

Quite often when I'm writing code I'll notice a very small bit of logic that keeps popping up all over the place. It's usually something so trivial that most people barely notice it. But it's also usually something that shows up so often that, despite being small and ignorable, it constitutes a fairly constant level of noise in the code. By noise I simply mean something that takes up more characters than it needs to, obscuring the real meat of the logic of your code. Common functionality like this that everyone knows and understands should just get out of the way, fade into the background, and let the unique logic stand out.

You may have also heard of this noise idea by another name: accidental complexity.

As Reg Braithwaite has pointed out, looping syntax is one of these things. With the ubiquity of IEnumerable and the advent of extension methods in C#, there is almost never a good reason to write an explicit for loop anymore. Looping is a ubiquitous bit of logic that nonetheless takes up quite a lot of characters. Even the vaunted foreach loop is now officially more verbose than it very often needs to be.

Microsoft made it easy to get rid of the explicit loop when what you are doing is essentially a mapping operation, with the inclusion of the IEnumerable.Select extension method.

Say your Foo class has a static function taking a Bar and returning a Foo, and you want to use this function to take a collection of Bars and create a collection of Foos.

You could do this:

IEnumerable<Bar> bars = GetAllBars();
List<Foo> foos = new List<Foo>();

foreach (var bar in bars)
foos.Add(Foo.FromBar(bar));

Or you could do this, which is obviously much more concise:

IEnumerable&lt;bar> bars = GetAllBars();
IEnumerable&lt;foo> foos = bars.Select(Foo.FromBar).ToList();

Note that the Select function completely takes care of the looping logic. Once you know that, this code reveals itself as being extremely elegant. But what if the action you're taking doesn't return anything? You're "stuck" writing an explicit loop, right? Not at all.

Take this code:

IEnumerable<Foo> foos = bar.GetAllItems();

foreach (var foo in foos)
PrintToScreen(foo);

To start removing the noise, first define an extension method for IEnumerable called ForEach. This is extension method #1.

public static IEnumerable<X> ForEach<X>(this IEnumerable<X> lhs, Action<X> func)
{
foreach (X x in lhs)
{
   func(x);
   yield return x;
}
}

Then rewrite:

bar.GetAllItems()
.ForEach(PrintToScreen)
.ToList();

Now there's just the issue of that nasty little ToList call. Right now, we need that in order to force the collection to be iterated. The yield return syntax essentially causes a function's execution to be deferred until an element is actually requested. This is actually potentially useful even if none of the things you need to do will return values. You can chain together a bunch of actions on the collection by chain-calling ForEach with different delegates. But it's still silly to create and throw away a List just to do this.

So we create an Evaluate function that does a simple explicit iteration and nothing more, to force the iterator to be evaluated. This is extension method #2.

public static void Evaluate<X>(this IEnumerable<X> lhs)
{
 foreach (X x in lhs) ;
}

And now you can replace ToList with Evaluate, which will iterate the collection without allocating a new List.

bar.GetAllItems()
.ForEach(PrintToScreen)
.Evaluate();

This is nice, and is going to be useful if we need to chain ForEach calls. But when we don't need that there's still that Evaluate call at the end that's going to be repeated every time we want this functionality, which could be an awful lot. So, let's get rid of that too.

To do that we define a Visit function (named for the Visitor Pattern), that will call ForEach with the given delegate, and then Evaluate as well. This is extension method #3.

public static void Visit<X>(this IEnumerable<X> lhs, Action<X> func)
{
  lhs
      .ForEach(func)
      .Evaluate();
}

Now we can finally get all this done in a single function call:

bar.GetAllItems()
  .Visit(PrintToScreen);

This takes some getting used to. But it really has the potential to condense your code. It isn't readily apparent looking at one bit of code, but once you start talking about nested loops or consecutive loops, you'll see the difference. Not to mention the total effect it will have across your codebase. Loops are everywhere. Shave off 50 characters from each one and your talking about a lot of characters in aggregate.

For my part, after I determined to avoid explicit loops whenever possible, the comparatively verbose explicit looping syntax became almost painfully extraneous to my eyes. I feel like function calls are much more elegant.

Decoupling Domain Model from Persistence

2009-01-29T14:55:00.007-06:00

For the last several months, I've been working on a Windows desktop application. This application has a number of pretty common aspects: file manipulation, local data repository, GUI, user settings, collection/dataset manipulation (think drag-and-drop listview type stuff), communication with peripheral devices, and communication with remote services (such as a web service). I think just about every desktop application has a good cross-section of these aspects, and maybe a few others that are slipping my mind at the moment. As a result, there are patterns of application architecture that will arise, and which the development efforts of well-designed, robust applications will have in common. I'm not talking about the typical "design patterns" here à la Gang of Four (GoF), but rather "application architecture patterns". Big patterns. A set of 4 or 5 patterns that, when meshed together, encompass the big, meaty chunks accounting for 95% or better of your application.

I know this is true. I know these patterns exist. But as this is my first effort in developing a desktop app of this complexity, I don't really know what the patterns are. One that GoF did account for was MVC/MVP, and sure I know those... But honestly, up until recently (and maybe even still) my knowledge of that pattern was very hazy and academic. Actually implementing it was something I'd never had to do before. It took a lot of blood, sweat, and tears--lots of trial and error--to get to a point where I really feel like I'm starting to grasp the "right" way to put together an MVP structure.

So now I've moved on to persistence. I have my domain model. But I need to be able to persist the objects in my domain model to and from disk (not a database!). I've been struggling to find ways to add in this functionality without tainting the domain model with persistence logic. Granted, this is a valid way of doing things, as is evidenced by Martin Fowler's ActiveRecord pattern. But it really doesn't make for ease of unit testing. Your business logic and persistence logic get all mixed up in the same classes and you can't easily isolate one from the other for testing purposes.

Then I saw Fowler's Repository pattern. I thought, okay, maybe that will help. I can create an interface that I can implement which will take care of all the logic for disk access to all the right places, based on what objects or collections are being retrieved. And I can just mock the interface for purposes of testing the domain model and business logic. I have a simple three-level hierarchy of domain interfaces, so it would be fairly simple to implement all the querying and CRUD I need for each domain class on the repository interface. But then I thought about how we plan to have multiple implementations of these interfaces in the future, with different permutations of under-the-covers data to be persisted, and thought maybe it would be best to have the repository be metadata-driven, rather than hard-coding the query logic for the different domain classes. This of course naturally leads to a record-based design, and that would mean I'd want data mappers to translate from my domain classes to repository records. And of course, the repository class should be strictly independent of the persistence mechanism itself (in this case the disk) so that I can unit test the repository logic without worrying about maintaining a test repository just for that purpose.

And suddenly I realized that I had mentally re-created ADO .NET. DataSets and DataTables are the repository mechanism, (with typed datasets and the ADO designer even supporting generating data mappers for you). DataAdapters are the persistence service.

So where this leaves me is with the question of whether I should just design my on-disk storage schema, build myself an app-specific data adapter mechanism, and let ADO .NET do all the work. If I really wanted to minimize the amount of mapping and repository-access code I have to write, I might even be able to use the Entity Framework to do my dirty work for me. Admittedly I have no idea how much work it would take to implement my own data adapter. The interfaces seem straightforward enough, but I have a nagging feeling that's really just the tip of the iceberg.

After all my radio silence here, I don't know if anyone is still listening... But if anyone has any thoughts on the wisdom of this approach, I'm listening intently.

Don't Give Up Assembly Privacy For Sake of Unit Testing

2008-10-08T16:34:00.013-05:00

Just a little PSA to other .NET unit testing newbs out there like me. This info is available various places on the internet, but you'll be lucky to find it without just the right search terms. So hopefully adding another blog post to the mix will make it easier to stumble on.

Unit testing frameworks need to instantiate your types, in order to run unit tests. What this means is that they need to be able to see them. The easiest way to do this is to make your classes public and/or place the tests right inside your project.

Placing the tests right in your project means that you'll very likely have to distribute the unit test framework assemblies along with your product. This might give more information to potential hackers than you would like. And making your classes public brings with it the often undesirable side-effect of opening up essentially all the types in your assembly to be used by anyone who knows where the assembly file is, in essentially any way they like.

But you don't have to make these concessions. You can move the tests out of your assembly, to keep them out of the deployment package, and still keep your classes internal (though not private). The .NET framework allows an assembly to declare "friend" assemblies that are allowed to see its internal classes. (Yes, very similar to the old C++ friend keyword). This is accomplished by adding an assembly attribute called InternalsVisibleTo to your AssemblyInfo.cs file.

If your unit test project does not have a strong name, it's as simple as referencing its assembly name:

[assembly: InternalsVisibleTo("MyCoolApp.UnitTests")]

However, I strongly recommend giving your unit test assembly a strong name. A strong name is a name involving a public-private key pair, and which is used by the .NET framework along with some hashing and encryption technology to prevent other people from creating assemblies that can masquerade as your own. Furthermore, if you give your app itself a strong name (which you should if you plan to distribute it), any libraries it references will need strong names, including the ones it just allows to see its internals.

So, if you decide to give your unit test project a strong name, you'll need the public key (not just the token) as well:

[assembly: InternalsVisibleTo("MyCoolApp.UnitTests, PublicKey={Replace this, including curly braces, with the public key}")]

(If you need to learn about strong names, and/or how to extract the public key from your assembly, this is a good place to start: http://msdn.microsoft.com/en-us/library/wd40t7ad.aspx.)

Once you've done this, you should be able to compile and run your unit tests from a separate project or even solution, and still keep the classes you're testing from being "public".

This is all well and good, but if you're working with mocks at all, you probably have another problem on your hands. The most popular .NET mock frameworks (e.g. RhinoMocks, Moq, and NMock) use the Castle Project's DynamicProxy library to create proxies for your types on the fly at runtime. Unfortunately, this means that the Castle DynamicProxy library ALSO needs to be able to reference your internal types. So you might end up with an error message like this:

'DynamicProxyGenAssembly2, Version=0.0.0.0, Culture=neutral, PublicKeyToken=a621a9e7e5c32e69' is attempting to implement an inaccessible interface.

Complicating this fact is that the Castle DynamicProxy library places the proxies it generates into a temporary assembly, which you can't just run the strong name tool against, because the temporary assembly doesn't exist as a stand-alone file. Fortunately, there are programmatic ways of extracting this information, and the work has been done for us. The public key for this assembly, at the time of this writing, has been made available, here and here. You might find some code at those links that could help you extract the public key from any future releases of Castle as well.

The important information is basically that, as of today, you need to add this to your AssemblyInfo.cs file, without line breaks:

[assembly: InternalsVisibleTo("DynamicProxyGenAssembly2, PublicKey=002400000480000094000000060200000024000052534131
0004000001000100c547cac37abd99c8db225ef2f6c8a360
2f3b3606cc9891605d02baa56104f4cfc0734aa39b93bf78
52f7d9266654753cc297e7d2edfe0bac1cdcf9f717241550
e0a7b191195b7667bb4f64bcb8e2121380fd1d9d46ad2d92
d2d15605093924cceaf74c4861eff62abf69b9291ed0a340
e113be11e6a7d3113e92484cf7045cc7")]

Caution: The name and public key of this temporary assembly was different in earlier versions, and could change again in later versions, but at least now you know a little more about what to look for, should it change.

So remember: You don't have to open wide your assembly to just anyone who wants to reference your types, just for sake of unit testing. It takes a bit of work, but you don't need to compromise.

Surviving WinForms Databinding

2008-07-02T01:00:00.003-05:00

I've rarely had the freedom in my career to implement a Windows application using MVC-style separation of concerns. Generally I am told "just get it working". Now, if I already knew how to tier out an application, this wouldn't be a problem. But since I don't, and it would take me a good deal of time to figure out a satisfactory way of doing it, I haven't been able to justify spending the time, on the company dime.

But recently, I've been fortunate enough to work on a new software product without a hard ship-date, and having other obligations on my plate. This has given me the freedom to not spend every minute producing functionality. So I've been experimenting with implementing MVC with Windows Forms.

There are essentially two ways to get this done. You can:

Write a bunch of manual event code to trigger view changes from model changes, and vice versa.
Use databinding and save yourself the explicit event code.

This works great if all your datasources are DataTables, or DataViews, or other such framework classes that are designed from the ground up to work with WinForms databinding. But should you have the misfortune of not dealing with any record-based data, you'll find you have a much tougher road to walk.

If you, like I, choose option 2, and you happen to be working on anything more complicated than a hello world application, and you are truly committed to doing MVC both correctly, and with as little unnecessary code as possible, then you will undoubtedly spend, like I have, a lot of time banging your head against a brick wall, trying to figure out why your databinding isn't doing what it is supposed to. This is a terrible shame, because if databinding in WinForms worked properly and was easy to use, it would be a spectacular tool for saving time and shrinking your codebase.

Truth is, you can still same time and code. But not as much as you might think when first introduced to the promise of databinding. If you can find any decent information on the obstacles you'll encounter. Compounding the above hurdles is the fact that what information can be found online about them is scattered to the four winds, and no one bit references the rest. So, I've decided to blog the headaches I encounter, and my resolutions, as I find them. This should increase the searchability of the issues at least a bit, by tying together the separate references with my blog as a link between them, and also by containing the different bits of information within one website. Or it would, if I had readers...

My results so far have produced 5 rules, to help you preserve your sanity while using WinForms databinding.

Rule 1: Use the Binding.Format and Binding.Parse events.

This first rule isn't actually too hard to find information on. Format and Parse essentially let you bind to a datasource property that doesn't have the same data type as the property on the control. So you can bind a Currency object to a TextBox.Text property, for example.

The Format event will let you convert from your datasource property type to your control property type, and Parse will let you convert from your control property type to your datasource property type. The MSDN examples at the links above are pretty good. If you use them as a template, you won't go wrong. But if you start to switch things up, beware Rule 2...

Rule 2: If you use Format or Parse events, DO NOT add the Binding to the control till after register with the events.

I honestly don't know what the deal is with this one. I just know that if you add your events to your Binding object after you've already passed it to the Control.DataBindings.Add function, they won't get called. I don't know why this should be, unless the control only gets a copy of your actual Binding object, not a reference to it.

Unfortunately, I have lost the references I had to the forum posts that talked about this. There were several, and now I can find none of them. I know I saw them, though, and I saw the symptoms of the other ordering, so as for me, I'm going to make sure to follow this rule.

Rule 3: Use INotifyPropertyChanged and/or INotifyPropertyChanging.

I ran across this info in a post on Rick Strahl's blog. I ran across this information during a desparate scramble to find out why my datasource ceased to be updated by user actions on the controls, after the initial binding occurred. The INotifyPropertyChanged interface is intended to be implemented by a datasource class that has properties that will be bound to. It provides a public event called PropertyChanged, which is registered with the data binding mechanism when you add the Binding to your Control. Your class then calls this event delegate in the desired property setters, after the new property value has been set. Make sure to pass "this" as the sender, and the name of the property in the event arguments object. Notice that the property name is provided as a String, which means that there is reflection involved. This will become relevant in Rule 4. Also note that there is an INotifyPropertyChanging interface, which exposes an event that is meant to be raised immediately before you apply the datasource change. This is generally less useful for databinding, but I include it here to save some poor soul the type of frustration I have recently endured.

Rule 4: If you implement INotifyPropertyChanged, don't include any explicit property change events ending with "Changed".

As I mentioned, the databinding mechanism uses reflection. And in so doing, it manages to outsmart itself. There is a very good chance you're going to run into a situation for which these databinding mechanisms aren't useful, and you'll have to implement your own explicit property change events on your datasource class. And of course, you're going to name these events in the style of "NameChanged", "AddressChanged", "HairColorChanged", etc. However, the binding mechanism things it's smart, and rather than just registering the INotifyPropertyChanged.PropertyChanged method, it will also register with any public event whose name ends with "Changed". And if you didn't happen to make your event follow the standard framework event signature pattern--that is, void delegate(Object sender, EventArgs e)--then you will get errors when the initial binding is attempted, as the mechanism attempts to register it's own standard-style delegates with your custom events, and you get a casting error.

I solved this one by following a crazy whim, but I also tried to verify the information online. All I could find was one old post buried in an obscure forum somewhere.

Rule 5: Don't bind to clickable Radio Buttons

I know how great it would be if you could just bind your bunch of radio buttons to an enum property. I really do. You think you're just going to hook up some Format and Parse events to translate back to your enum, and all will be well. It would be so darn convenient, if it actually worked. But WinForms just isn't cut out for this. For 3 full releases now (or is it 3.5 releases?), this has been the case. It's because of the event order, which is not something that MS can go switching up without causing thousands of developers to get really cheesed off.

The problem really comes down to the fact that unlike other controls' data properties, the Checked property of a radio button doesn't actually change until focus leaves the radio button. And as with all WinForms controls the focus doesn't actually leave the radio button until after focus is given to another control, and in fact not until after the Click event of the newly focused control has fired. The result of this, as it pertains to radio buttons, is that if you try to bind to them, the bound properties in your datasource will actually lag your radio buttons' visual state by one click. If you have just two radio buttons, the datasource will be exactly opposite the visible state, until you click somewhere else that doesn't trigger an action that references those datasource properties. Which can make this a really infuriating bug to track down. I almost thought I was hallucinating.

Now, in all honesty, it's possible to make it work. But it is the kludgiest kludge that ever kludged. Okay maybe it's not that bad... but it's a messy hack for sure. It takes a lot of work for something that really should already be available. As near as I can tell, the only way to solve this problem without giving up the databinding mechanism is to essentially make your own RadioButton control, with a property change and event order that is actually useful. You can either write one from scratch, or sub-class RadioButton and override all the event logic with custom message handling.

So....

There's the result of 3 weeks of frustration. I hope it helps someone else out there someday. I'll make list addendum posts if/when I come across any other mind-boggling flaws in the WinForms databinding model. And in the meantime, I welcome any additions or corrections that anyone is willing to contribute in the comments.

Is There a Place for Deletionism in Wikipedia?

2008-06-15T18:12:00.006-05:00

Just dipping a toe in the water with something too big for Twitter, but that I definitely have a few thoughts on which I am willing to pitch out onto the internet.

Yesterday evening and this morning I watched (Or overheard? No idea what the appropriate term here is...) an exchange over Twitter between Tim Bray and Jeff Atwood about "deletionism". Tim's thoughts on the matter were apparently too strong to be held within the restrictive confines of Twitter and so he wrote a strongly worded post on his blog as well.

The post is obviously... passionate. But putting aside the bulk of the post that simply expresses that passion, he makes a couple really strong points.

The first point is that deletionists are just your garden variety elitists.

"the arguments from the deletionists are jargon-laden (hint: real experts use language that the people they’re talking to can understand)"

The other point really could have comprised the whole post and, IMHO, would have been a sufficient argument all by itself.

"What harm would ensue were Wikipedia to contain an accurate if slightly boring entry on someone who was just an ordinary person and entirely fame-free? Well, Wikipedia’s “encyclopedia-ness” might be impaired... but I thought the purpose of Wikipedia was to serve the Net’s users, not worry about how closely it adheres to the traditional frameworks of the reference publishing industry?" [emphasis added]

This, to me, undermines the deletionists' whole platform. That is what Wikipedia was supposed to be: a new model of information archival. A model not subject to the sensibilities of some authoritarian arbiters of what is "notable" or "interesting". And the deletionists are unequivocally trying to undermine this goal, by deciding what "deserves" to be archived.

I say, if they want to take that old dead-tree encyclopedia model and just port it to the web, they can go find their own site, and their own content, to do it with. I've even got a couple recommendations for them.

Identity Crisis in Computer Science Education

2008-01-29T10:06:00.000-06:00

A while back, the seeds of a post started rolling around in my head, inspired by my lack of satisfaction with the preparation my education provided me for a programming career. But I didn't quite know what I thought. Then not long ago, Joel Spolsky presented a radical repackaging of Computer Science degrees, supposedly geared toward producing better programmers, and my thoughts started to gel.

I knew what my personal complaint was, and I knew that it was connected to a larger problem with the state of computer science education in general. But I didn't know exactly what the larger problem was, let alone have any ideas worth sharing on what might be done about it.

Now that seemingly the entire remainder of the blogosphere has weighed in on this and related topics, I think I am finally ready to throw my two cents in. I'm going to barrage you with links in the following paragraphs, to ensure that I credit everyone I read who assisted me in coming to my final conclusions. Feel free not to click through, but be aware that they represent a rich cross-section of an important discussion.

It took me several weeks to realize that what we have going on is essentially a three-way tug of war from people in different regions of the vast sphere of software development, who need very different things out of their workers, and hence out of their education. Below I will give a run-down of some of the claims made, expressing the different forces pulling on CS graduates these days. You'll quickly see it's no wonder that the schools are so confused....

The Artisan Programmer

Joel laments the uselessness of theory courses in computer science curricula, saying "I remember the exact moment I vowed never to go to graduate school" and then proceeding to recall a terrible experience he had with a Dynamic Logic class. Jeff Atwood insists that real-world development environments need to be in place and mandatory, including, but not limited to, source control, bug tracking, deployment, and user feedback. Then, as mentioned above, Joel proposes offering BFAs in software development, to make darn well sure that none of the academic (in the pejorative sense) theory stuff gets mixed in unnecessarily. The upshot of most of these points are that computer science / programming degrees should spend as much time as possible teaching people what they need to know to go into a career in software development, writing business software or software products.

The Computer Scientist

Brian Hurt comes in from another direction entirely, and in the process makes some very good points about the true purpose of higher education. He lays the blame for the flood of single-language programmers entering the workforce at the feet of schools who do just exactly what Joel and Jeff are asking for. He makes some great points. And while he sounds more than a little reminiscent of the classic Joel post about the perils of java schools, his argument is much more thorough than just blaming the tools. Chris Cummer joins this party, wishing that his theory foundations had been firmer, and making an excellent analogy to the difference between someone who studies a language, and someone who studies language. We also have the respectable Raganwald, who although he has admirably pointed out good points from all sides, doesn't shy from offering his opinion that programmers ignore CS fundamentals at risk of their own career advancement.

The Software Engineer

But thats not all. Several people have weighed in from yet another direction. Robert Dewar and Edmond Schonberg wrote one of the posts that started off this blog firestorm. Along with alluding to a similar sentiment as Hurt and Cummer, they heavily criticize the state of software engineering education for focusing too much on how to use specific, limited tools, when there are more sophisticated ones available. They claim understanding these will allow software engineers to easily pick up the other tools that may come along and direct them to appropriate purposes. Ravi Mohan stops short of calling the education satisfactory, instead sensibly pointing out simply that an engineer who doesn't use standard engineering tools such as system modeling, isn't really an engineer. Mohan comes on a little too strong for me in the comments, but the posts themselves (of which there are also a precursor and a successor, and should soon be one more) are worth reading. Like the others he makes valid points.

Resolving the Crisis

Mark Guzdial is one of the few people that really puts his finger near the pressure point. Though maybe not near enough to feel the pulse beneath it when he did so. At risk of quoting a little too heavily....

Rarely, and certainly not until the upper division courses, do we emphasize creativity and novel problem-solving techniques. That meshes with good engineering practice. That does not necessarily mesh with good science practice.

Computer scientists do not need to write good, clean code. Science is about critical and creative thinking. Have you ever read the actual source code for great programs like Sketchpad, or Eliza, or Smalltalk, or APL 360? The code that I have seen produced by computational scientists and engineers tends to be short, without comments, and is hard to read. In general, code that is about great ideas is not typically neat and clean. Instead, the code for the great programs and for solving scientific problems is brilliant. Coders for software engineers need to write factory-quality software. Brilliant code can be factory-quality. It does not have to be though. Those are independent factors.

And there it is.... Different environments require different mindsets/approaches/philosophies. Research requires one mindset/philosophy of work, engineering requires another, and in-the-trench-based programming requires yet a third.

When a person suffers from a personality fracture, the resolution is often to merge the personalities by validating each as part of a whole. Fortunately, since we are not dealing with a person, we have the freedom to go another direction: make the split real and permanent.

Associate of Science in Computer Programming

To fill Joel and Jeff's need, the student who wants to work in the craft of software development / computer programming, who wants to be an artisan, needs to have an appropriate degree. It needs to provide them with an exposure to the generalized idea of the programming platforms and tools that they will have to deal with for the rest of their career. Lose the lambda calculus, compiler-writing projects, etc. These things not necessary for them to get stuff done in the trenches. But they do need to be exposed to the fundamental generalities that pervade programming. And they need to be prepared to learn at an accelerated rate while in the field. That just comes with the territory. Focus on core programming skills like program analysis, debugging, and test practices. Introduce industry-standard tools (emphasizing generality and platform-independence) such as source-control, bug tracking, etc.

I think a two-year associate degree is perfect for the code-monkeys and business programmers that just love to dig in and mess around with code, and don't want to concern themselves with the overarching concerns. Especially with these jobs increasingly being pushed offshore, computer science grads are rapidly being priced out of the market. An associate degree is cheap enough to be worth the investment for a lower-paying programming job. And it doesn't carry the overhead of any unnecessary theoretical content that they may not be interested in learning. It should be noted though that this type of programming job enters the realm of the trades, with all the associated benefits and drawbacks.

If you're looking for a more well-rounded individual capable of moving up out of this position into a lead position, or even management, then a 4-year bachelor of science (or Joel's BFA, but I tend not to think so) may be a viable option as well.

Bachelor of Science in Computer Science

There's not much to say about this degree, because if you look at all the schools that are famous for their CS degrees, this is pretty much what you'll find. Lighter on general studies, heavy on theory, heavy on math. Light on tools because the students will be expected to find (or make) tools that work for them. Light on specific language education because students will be expected to adapt to whatever language is necessary for their problem domain.

This is a degree that will produce people primed for going on to masters and doctorates. They will end up in research, "disruptive" startups, or working on new languages, OSes, etc. This degree is designed for people who want to work at the edge of things. Who want to solve new problems and push the boundaries. They are people upon whom will be placed the burden of pushing the state of knowledge in CS into the next era.

Bachelor of Science in Software Engineering

I am hesitant to propose this degree, because I am not certain that the practice of Software Engineering has evolved to the point where we have 4 years worth of general knowledge that's worth teaching, and that won't be out of style by the time the student's graduate.

It seems that some people, when they talk about Software Engineering, are talking about architecture and design, and others are talking about process, resource allocation, estimation, etc. To be frank, I don't think the former qualifies as a true engineering discipline. At least not yet. I don't know how much the modeling of programs that Ravi Mohan talks about is going on out there in the industry. I suspect that it happens more in the process of really big projects, and maybe in digital security. The second type of engineering people think of, however, I think is very similar to what we see in manufacturing, with industrial and process engineers. These are people who get an intimate knowledge of the domain, and then figure out ways to get everything to run smoother, more efficiently, and producing higher quality.

I can definitely see some education possibilities here, though I am not sure myself how to fill out the whole degree. It should at least encompass a good portion of the Associate of Science in Computer Programming, because they need to understand the intricacies involved. I can also see this degree teaching some of the more established measurement and estimation techniques found among the industry's established and experienced software project managers. Generally more management-related topics such as resource allocation, planning, product design, feature negotiation, etc. might fit in well here. Different project processes, testing/QA models, and of course an ability to keep up to date with technologies and platforms, are all par for the course as it's all critical for making decisions in the industry.

Conclusion

I really, honestly believe that Computer Science education as a whole needs a makeover. It needs more structure, more integrity in the vision of what each degree means, across schools. When someone has one of these degrees, you need to be able to reliably assume they should have learned certain things, regardless what school they went to. Many of the degrees currently on offer don't satisfactorily prepare their students for any one of the possible careers discussed above. I'm not saying their hand needs to be held from enrollment right on through to their first job. That's not the purpose of college. The purpose of college is to provide a cohesive education, directed to some relatively well-defined goal of capability and knowledge. Today this is tragically non-uniform at best, and absent altogether at worst.

So I see plenty of room in software development education for a clarification of purpose, and a readjustment of goals and curricula. A few different tracks, each geared toward a distinct section of the sphere with different goals and different responsibilities. And if we resolve to use existing terminology with some respect for the historical meaning of the words, we can re-use our existing nomenclature. But there can be no more of this muddy slurry of computer science, craft of programming, and software engineering all overlapping in claims of purpose, treading on each others' territory without care. Everyone can have what they are asking for. They just need to accept that no one can claim the "one true way".

I am not a Computer Scientist

2008-01-14T15:19:00.000-06:00

Prepare yourselves. I have an embarrassing and melodramatic admission to make.

My career is a sham.

Although my degree and education are in a field that is typically referred to as "computer science". I am not actually a "scientist". Nor do I "practice science". But I won't be satisfied to go down alone for this charade. I'll going on record saying that I am convinced that for the vast majority of people who were educated in or work in the field of "computer science", the ubiquitous presence of the word "science" in proximity to our work or education, is a tragic misnomer.

I don't know how long this has been on my mind, but I know almost precisely when I became conscious of it. It was a couple months ago. I was newly exposed to devlicio.us, and perusing the blogs hosted there, when I came across a post by Bill McCafferty about a lack of respect and discipline in our field.

Early in the post, Bill reveals an injustice he encountered during his education.

...When I started my undergrad in this subject, I recall reading articles debating whether it should be called a science at all. Gladly, I do not see this argument thrown around much anymore.

I think I am probably not going to make the exact argument here that he disagreed with back then. The things we all studied in school are definitely part of a nebulous field of study that may rightfully be called "computer science". As Bill points out,

"From Knuth's classic work in The Art of Computer Programming to the wide-spread use of pure mathematics in describing algorithmic approaches, computer science has the proper foundations to join other respected sciences such as physics, concrete mathematics, and engineering. Like other sciences, computer science demands of its participants a high level of respect and pursuit of knowledge."

I have no argument with any of this. He's right on. Donald Knuth (who is indeed my homeboy in the sense that we share our hometown) studied and practiced computer science (which if you know anything about Knuth, you'll know is an almost tragic understatement). And thousands of people who have followed in Knuth's foot steps can lay the same claim. However, that's not me. And it's not more than 99% of all programmers in the field today.

Computer science suffers the same type of misnomer as many other disciplines who have adopted the word "science" into their name, such as political science, social science, animal science, food science, etc. And it seems that most such fields, if not all, have done so because the very validity of the field of study itself was subject to severe criticism at some point in the past. So we take on the term "science" to get it through people's heads that there is a root in formal practices and honest intellectual exploration. But to then blanket every profession that derives from this root with the term "science" is a misappropriation of the term.

I can think of a number of examples.... The programmer working for the bank to develop their website, or for the manufacturing company to manage their transaction processing system is no more necessarily a "computer scientist" than the election commentator is necessarily a "political scientist". When someone gets an electrical engineering degree and goes to design circuits for a living we do not say he "works in electrical science". We say he is an electrical engineer. When someone gets a technical degree in mechanics and then goes to support or produce custom machinery, we do not say he "works in mechanical science". We say he is a mechanic, or a technician. Why, then, when someone gets an education that amounts to a "programming degree", and then goes to work doing programming, do we say that he "works in computer science"? It's a uselessly vague and largely inappropriate label.

By contrast, if you have a doctorate in computer science, I'm prepared to say you deserve the label. If you write essays, papers, articles, books, etc. for use by the general practitioner, you probably deserve the label. If you do research, or work on the unexplored fringes of the field--if you are exploring the substance and nature of the information or practices that the rest of us simply consume and implement, then in all likelihood you deserve the label.

Please, please understand that I am by no means belittling the value of our work, or the nobility of our profession. Often we simply consume the information produced by true "computer scientists". But we transform it from theory into practice. We resolve the concrete instances of the abstract problems that the true scientists formally define. We take the pure thought-stuff produced by scientists and turn it into tangible benefit.

This is not trivial. It is not easy. It deserves respect, discipline, study, and care. But it is not "practicing science".

I should say in closing that I am not as upset about all this as the tone of this post might imply. I don't even really have a big problem with the use of the word "science" to refer to a field of study or work that largely does not include research-type activities. I don't like it, but I accept that it happens. But "computer science" has a problem that other similar "sciences" don't. When someone says they work in "political science" or "food science", you can make a guess as to the type of work they do, and it's hard to be significantly incorrect. Though maybe it's my outsider's naïveté that allows me to make this claim. At any rate, "computer science" as a field is so broad and vague that I don't think the term communicates a useful amount of information. But you wouldn't know that by talking to programmers, who seem only too ready to attempt to take hold of the term and own it for themselves.

I think this is one small facet of a larger and far more critical issue in our field in general, which I fully intend to write more about very soon. But until then, lets take the small step of starting to consider what we really mean when we use much of the popular but often ambiguous terminology when discussing our profession.

I work in the field of computer science. This tells you nothing except that I am unlikely to be a prime specimen of the wondrous human physiology. But.... I am a programmer. I have a degree in Computer Engineering. I am interested in programming theory. I work as a software development consultant. And now, you know something about what I know and what I do.

Now what about you?

Update: I forgot to note that in McCafferty's blog entry, he himself makes use of "trade" terminology to categorize different levels of reading materials. Which belies the uncertain nature of programming as a profession. We certainly wouldn't say that a carpenter works in the "wood sciences", would we?

What is a Senior Programmer?

2007-11-07T20:20:00.000-06:00

My friend and co-worker Nate recently wrote about some hurdles he has encountered in pursuing his professional ambitions as a software developer. I know what he's going through because I entered both my internship and my first post-college job with tragic misconceptions (non-conceptions really, in the case of my internship) as to how my career would develop.

My first mistake was that, as I began my internship, I had no idea how my career would progress, how it should progress, or how active a participant I would be in whatever progression did occur. I knew that I enjoyed twiddling around with code, and that I seemed to have more of a natural talent for it than most of my classmates. And I figured that if I was going to make a career out of anything, it should be something that I did at least passingly well, and that I enjoyed.

As I entered my first post-college job, I had decided I most definitely did not want to become a manager. I enjoyed programming too much to give it up, for one thing. Further, managers had so far stood mostly as obstacles to my involvement in interesting work, and as incriminating figures more prepared to remonstrate me for my professional flaws rather than empower me to better myself as a programmer. So I wanted nothing to do with that. Instead I set a near-term goal of becoming a "senior developer", at which point I would re-evaluate my career trajectory and adjust course if necessary.

An important question in gauging the advancement of your career as a programmer is to ask what exactly it means to be a senior programmer. I have come to see this title as being tied to the professional respect that one has accumulated in a programming career. A senior programmer is not someone who has served a certain amount of time "in the trenches". Nor is it even someone with a broad experience base.

No, I think there is something a bit more intangible, that identifies someone deserving of the "senior developer" title. Something less measurable. Something that is probably sensed, but not necessarily explicable by those with less experience. But something that would be conspicuous by its absence.

As I see it, what distinguishes someone deserving of the "senior developer" title is a sense of stability. A senior developer is someone who can stand in the middle of the chaos that arises in a project, and exert a calming influence on the people and efforts swirling around him. A senior developer is an anchor to which a project can be tied to keep it from drifting into dangerous waters. He is a sounding board against which claims will ring true or false and goals will ring possible or impossible. He is the steady hand, not necessarily on the rudder of your project's ship, but wherever that hand is needed most. And he is a strong voice that reliably calls out your bearing relative to your destination.

Of course, these metaphors sound a bit grandiose. But the general picture I think is accurate. A senior developer is someone that you put on a project to ensure there's some measure of certainty in the mix. It doesn't mean your project is guaranteed to succeed. But it should mean that you can sleep a little easier knowing that where you think the project is, is where it really is. And that if it's not where it needs to be, that there is someone involved who has a decent idea of how to get it there.

Naturally, these things do come with time and experience. So what I said earlier isn't completely true, in that a senior developer is someone with tenure and experience. However, these are necessary, but not sufficient conditions. Not everyone with 20 years of experience on 5 platforms and 15 languages qualifies. And not everyone with 5 years of experience on 1 platform and 2 languages doesn't qualify. Rather, if you show that you can learn from experiences both positive and negative, port technical and non-technical knowledge from one domain to another, and educate, inspire, or empower colleagues and junior developers.... Then you are showing yourself to have what it takes.

If you're like me, and Nate, you don't feel that you're there yet, but you hope one day to proudly contribute this kind of value. Don't lose hope. Every new experience, technical and non-technical, is a growth opportunity. But it is important to broaden your horizons in both of those respects. If you hope to educate, inspire, and empower others, you must first learn to do so for yourself. And if there's one thing I've learned, it's that you can't do that if you feel stagnant. In that case, your first responsibility to yourself is to educate your boss of the professional value you could offer with a little broader exposure. And if that doesn't work, address the issue yourself, dedicating a bit of time outside work. If you can find them, working together with some like-minded friends or co-workers can be very encouraging, like what we have done with our "book club". Remember, nothing changes if things just stay the same.

LINQ: Not just for queries anymore

2007-10-12T10:56:00.000-05:00

Take a look at this LINQ raytracer, discovered via Scott Hanselman's latest source code exploration post.

For those of you who don't know what the heck LINQ is, it was created as a language extension for .NET to allow querying of datasets using an in-place SQL-inspired notation. This was a huge step up from maintaining separate bodies of stored queries, or hard-coding and dynamically assembling queries as strings in your source code.

Lately LINQ has been enhanced and expanded upon even further to become a more broadly usable miniature functional/declarative language within .NET. This is impressively illustrated by the raytracer code. The author disclaims that it's probably not the best way to write it, which is probably true. But this in no way detracts from its illustration of the power, expressiveness, and flexibility of LINQ.

I love it! It's a great example of taking a deceptively simple language and showing its power by doing things with it that aren't strictly within the purview of its design. It reminds me of some SQL code that I've written for my current employer, to get functionality out of inefficient PL/SQL and into blazing-fast pure SQL. Stuff that it was said couldn't be done in pure SQL. Of course, this is usually said by people who think that SQL is a second-class citizen among languages, not realizing the power of its declarative foundations. This is something that I hope to write about more extensively in the future, either here or in a new SQL blog I'm considering starting with a friend and coworker.

Anyway, I'm tempted to say that there's a bit of a down side in that this LINQ raytracer is mostly comprised of LETs, which feel more imperative than declarative. However, I've long claimed that just such functionality in actual SQL would make it a tremendously more compact and elegant language, so I won't complain too much about that. =)

My gut reaction upon seeing this code is that it feels like a crazy hybrid of LISP and SQL, but it's written in C#. Which all makes my head spin, but in a good way. I love that C# is becoming such a great hybrid of procedural and functional programming paradigms.

Where will your programming job be in 7 years?

2007-09-17T01:18:00.000-05:00

There's been a lot of noise recently about the future of programming, as a profession. Okay, let's be honest, people have been talking about the imminent programming crash for a long time. But they didn't know what they were talking about. When I graduated college a few years back, we were warned that many of us would not find jobs, but this fear turned out to be overblown.

But now the noise is coming from some different directions. It's not analysts, or people who've been layed off, or the people who got screwed in the "dot.com bust". Though to be fair, the analysts are still saying it, louder than ever. But it's also now coming from the people who hire programmers. And unfortunately, if you are a programmer, those are the people you care about. So, what's different this time? Why don't companies need programmers anymore?

Well, they do need programmers. But they need programmers who can do more than just program. So odds are good that if your job title is "programmer", or "programming" constitutes 90% or better of your responsibilities on the job, they're looking at you like day-old tuna salad.

From the ComputerWorld article:

"It's not that you don't need technical skills, but there's much more of a need for the business skills, the more rounded skills"

"Crap," you say to yourself. "I hate business stuff." Or maybe you're saying "Crap. Why didn't I go for that business minor or second major back in school?" Easy, cougar. Don't get too worked up yet. Let's look at the context of this statement. We already know who's saying it: "business people". People working in IT in companies throughout the nation. Not software companies. Not consulting companies. Just regular companies. This is crucial information. It means the positions we are talking about are mostly going to be be "business programming" jobs.

Now we have a few more questions. Where are the jobs going? Why did I put quotes around "business programming"? And why are these jobs going away for real this time?

First answer: they're going to go to people with Associate degrees, and people who are self-taught, and consultants. Some to high-quality on-shore resources, but an awful lot to people who speak Hindi or Urdu (or Chinese, or Korean). Wages and education requirements will be cut for in-house employees, and others jobs are going to consultants. Offshore in most cases.

Next... The reason I put quotes around "business programming" is to distinguish these jobs from other types of programming positions. If you are a "business programmer", you're part of what is currently a pretty huge job market. You're part of a population that does a lot of work for a lot of companies, but is often looked at as an unfortunate necessity by other people at these companies. "Business programming" is the type of programming for which, in the past, a company who sells shoes, would hire some people on full-time and in-house to do. It's order processing, financial reporting, CRM, network management, database scripting, and so on and so forth. And for a long time, management types have had an uneasy feeling that they were getting kind of a raw deal on these guys. And I hate to break it to any business programmers out there.... but they were right.

According to the Occupational Outlook Handbook entry for computer programmers, there are very specific and very real reasons that programming jobs are being phased out.

"Sophisticated computer software now has the capability to write basic code, eliminating the need for many programmers to do this routine work. The consolidation and centralization of systems and applications, developments in packaged software, advances in programming languages and tools, and the growing ability of users to design, write, and implement more of their own programs mean that more of the programming functions can be transferred from programmers to other types of information workers, such as computer software engineers.

Another factor limiting growth in employment is the outsourcing of these jobs to other countries. Computer programmers can perform their job function from anywhere in the world and can digitally transmit their programs to any location via e-mail. Programmers are at a much higher risk of having their jobs outsourced abroad than are workers involved in more complex and sophisticated information technology functions, such as software engineering, because computer programming has become an international language, requiring little localized or specialized knowledge. Additionally, the work of computer programmers can be routinized, once knowledge of a particular programming language is mastered."

I would add to this list that "business programming" is almost inherently redundant. Every company out there that employs in-house developers is reinventing the wheel. 99% of the problems their programmers are solving have been solved by thousands of other programmers at other companies around the country. When I look at it that way, it feels like such a tremendous waste of money and time. These programmers could be working on real problems like true AI, building Skynet, or bringing about the rise of the machines and their subsequent domination over the human race.

So essentially, the big reason is that there is finally a way for a company to separate their technical needs from their business needs. Packaged software has finally come to a point where it solves the general problems, while still providing a minimum amount of flexibility necessary to handle a company's critical peculiarities. When they have a need that isn't fulfilled by some packaged solution out there, contracting resources have become plenteous and cheap enough to fill that need. The company can move the business knowledge into more generic manager roles that are more useful to the company. (Roles with better pay, and job titles such as "software engineer" and "system analyst", and "project manager".) And the technical knowledge can be moved mostly out of the company into a "unpluggable" resource that they only need to pay as long as the work is actually being performed.

So, what's a programmer to do?

Well, first of all, stop being just a programmer. Don't be a code monkey. Yes, the under-appreciated coder-geeks of the world have "owned" that pejorative term. But in the eyes of corporate America, the words "code monkey" are always preceded by the words "just a". Code monkeys abound. They are replaceable. If you don't evolve (pun intended) you're going to end up obsolete when your company finds an offshore contractor that doesn't suck. (Yes, they do exist!)

One way you can evolve is by taking the advice of all the articles and reports I've linked to in this post. You can take courses, get certifications, etc., and become a "software engineer" or a "system analyst" or a "project manager". The numbers show there are plenty of companies out there that will be willing to pay you to do this as long as you don't suck at it. (And some that will even if you do suck. But I strongly advise against sucking.)

Many programmers go this route at some point. And there's no shame in it, if you can let go of coding every day (or at all) as part of your job and not hate yourself for it. I think I might go that route one day, but I'm not ready for it yet. For one thing I don't feel my experience base is either broad or deep enough to be as effective as I would want to be. But also, there are just too many cool languages and libraries out there. Too many programs that haven't been written yet. Too many problems that haven't been solved.

So what is there for me, and those like me? Those who don't want to give up programming as the core of our job, but don't want to end up teaching programming to high school kids by day while we sate our code-lust at night in thankless open source efforts? (No not all open source is thankless. But there are a lot of projects that go nowhere and just end up abandoned.)

Two answers:

Consulting. Not everyone is willing to send the work around the world, dealing with language barriers, time-zone difficulties, and security uncertainties, to get it done. There's a definite place in the market for on-shore, face-to-face consulting. This spot is getting tighter though, so you had better make sure you're a top-shelf product.
Software companies. The companies that sell the software that's putting code monkeys out of jobs are still raking it in. And they're actually solving new problems, building new technologies, etc. All the exciting stuff I dream of working on as I churn out yet another database script for a client.

These are your new targets. The sooner you make the jump the better. Not only will you be trading up, as far as job satisfaction (and probably pay), but you'll also be contributing your own small part to the further expansion of software technology. Both by actually working on it, and by condemning the tremendously redundant field of "business programming" to the death it has deserved for so long. You'll probably still have to learn that business and management stuff, but you can also probably avoid it taking over your job.

A new kind of Democracy

2007-09-08T09:25:00.000-05:00

Friday night I was listening to the Boston-based NPR show On Point. The episode was about how reading affects the brain. It was interesting, but that's not what I want to talk about. There were two guests on the show. The main guest is a professor of child development at Tufts, and the other a professor of educational communication and technology at Wisconsin's very own UW Madison. This latter guest was brought in as a counterpoint to the main guest's fear that people spending more time with videogames and on the internet were missing out on many of the amazing benefits that reading bestows upon the brain.

Given the UW prof's area of focus, the conversation of course eventually came around, briefly, to blogs. She claimed that just as the Guttenberg printing press enabled the "democratization of knowledge", blogging will bring about another fundamental shift. One where, as before, some very nice things will be left behind, forgotten by most, in sacrifice to the emergence of a new mode of interaction that will bring about it's own useful new evolutions and societal impacts. I found this a tremendously interesting proposition, and hoped they would follow this line of thought a bit more, but unfortunately the host had latched onto some earlier point and swung back around to that, never to return.

I'm not sure I heard another word of the broadcast anyway though because my mind was off and racing.

Many people have already staked a claim on the blogosphere as the next frontier in journalism. Rejoicing has already begun that journalism will migrate into the blogosphere and morph into something new and different than ye olde journalisme, bringing enlightenment to all and ushering in the Age of Aquarius.... Certainly it will offer a new value proposition, but I am not yet convinced of the extent to which blog journalism will supplant traditional journalism. Regardless of that, however, these people are thinking too narrowly.

Let's go back for a minute to that statement about "democratized knowledge". I've heard this turn of phrase several times in reference to the rise of the printing press, but never really stopped to think about what is really being expressed there. The history of human communication seems to me to track very closely the history of significant "advancements" in civilization, and with a bit of thought I think it becomes abundantly clear why this should be.

When speech and body language were all that was available, the spread of knowledge was a social affair. You needed to have at least two people, in close proximity, interacting more or less directly. The advent of writing freed knowledge from the bounds of proximity in space and time. But consumption was serial because the recordings required a huge time investment. First one person, then another, then another.... If the recording was not lost or stored away in a private library. The printing press resolved these issues. It became trivial to make multiple copies of a manuscript. Copies could be made as long as there was demand for them. If some were destroyed others would survive.

The printing press made knowledge available to all those who desired to consume it. It was no longer the privilege of rich or powerful men. If you could read, you could learn, regardless whether someone was available to teach you personally. One man's idea could be made available to thousands, across time and space, and generally without concern for the social position of the consumer. This is what is meant by the democratization of knowledge, and it really was a significant turning point in human history.

I contend that the rise of blogging is the next great evolution of human communication. Sir Isaac Newton is quoted as having said "If I have seen farther, it is only by standing on the shoulders of giants." How do we know they were giants? Because they had the time and resources to make recordings of their ideas, and people valued them enough to demand and preserve them. In the past there were generally two ways you could get your ideas disseminated to humanity at large. If you were privileged enough to have the time and resources, you could finance a recording of your idea regardless of its perceived value. Or if you really believed in your idea, you could make a big sacrifice to get it recorded, and pray that people valued it enough to compensate you for your investment.

With blogging, these restrictions are largely abolished. If you have access to a computer, you can create a blog, for free, as I've done here. You can record your ideas as quickly as you can type them, and once you post it, it is instantly available for billions of people to simultaneously search for, stumble upon, consume, discuss, and share.

Newton and Einstein stood on the shoulders of giants. With blogging, we no longer have need of the giants. For you and I to see farther, we can assemble a great pyramid of normal human beings, each with a small contribution that in aggregate has potential that the "little people" of the past could only dream of participating in.

We can also move beyond ideas and into expression and experience. Blogging allows us all to share with the world our unique experiences, our viewpoints, our disasters, our epiphanies, our humanity. No matter how mundane our individualities may be judged by most, we now have the potential to find the other people out there who are weird in the same ways we are. With increasing frequency and ease we can now connect online with someone else who has shared our joys and sufferings, our beliefs and needs. Someone with whom we can identify and be comforted. Or we can peer into the "mundane" individualities of others, and experience the wide breadth of simple but beautiful diversities of humanity. These are often things that are difficult to get published in print, unless the presentation is particularly engaging, or the story is something that we expect will touch large groups of people. Online these barriers do not exist. And we need not simply consume the others' narratives either. We are empowered to interact with the speaker and participate in their narrative.

I could go on into more specifics, but instead maybe I will dedicate another post to that sometime down the line when I've thought about it a bit more.

To put it succinctly, Gutenberg's printing press enabled the democratization of the consumption of knowledge. With the emergence of blogging, we are witnessing the democratization of the production of knowledge and of expression of the human experience.

Maybe not the beginning of the Age of Aquarius.... But I think it's a bit of a "big deal" nonetheless. Though I am probably late to the party on this.

My Grand Entrance

2007-09-07T23:49:00.000-05:00

I have officially joined the blogosphere. I'm still not sure I like that word. It sounds like something Dan Simmons would have made up. But here I am nonetheless. My entry was unceremonious, not surprisingly, as my online presence has heretofore consisted solely of my college Senior Design project page and scattered posts on a few unrelated forums in various corners of the internet (some of which I am now thoroughly ashamed of).

I created this blog as an outlet, for my many and varied interests. It has been exceedingly rare to find someone who is willing to listen to me talk about them, let alone actually hold up one end of a discussion. So, hopefully this will provide me a pressure valve, where I can let out some of the thoughts that build up in my head without thoroughly irritating my friends and coworkers.

If I'm lucky, someone will read one of my posts all the way through before realizing they've wasted their time on the ramblings of some idiot on the internet.

And if I'm really lucky, it'll turn out that there's another person out there who actually cares enough about the same things as I do to tell me how ridiculous my ideas are and that I should just close up shop and go back to my day job.

Anyway, here are a few of the things I'll likely blog about:

Software Development
Programming Languages (and theory)
Technology and Culture
Theoretical Physics
Education
Math
Professional Development
Language / Linguistics

If these obscure and esoteric topics look interesting to you, God help you.... I mean stay tuned!