Oh, Verifiability, We Miss Thee

While stepping through some code in the debugger I encountered a wonderful reminder of how nice it is to inhabit the managed world of Type Verifiable code.

There's a library of functions (library A) that gets statically linked into another library of functions (library Z). Lib Z defines type FOO (a class with several methods longer than several thousand lines of code each, but that's a rant for another day).

Unfortunately lib A also defines a type FOO. The lib A version is an earlier incarnation of lib Z's type FOO. Being an earlier version, lib A's FOO has not kept up with the times.

When a lib Z function calls a lib A function and passes it a pointer to FOO, what exactly does lib A operate on? Of course it uses its own version of FOO (you don't seriously think we're using namespaces to solve this problem do you?) which, being out of date, has all its parts in different places.

So the other developer working on this problem spends hours of frustration trying to figure out why FOO->SomeCount goes from the low 400s to the millions as soon as it enters lib A's FOO. In despair he decides its probably related to crossing a process boundary (lib A used to be dynamically linked to lib Z but is no longer) and so opts to copy the code from lib A into lib Z.

This solves the immediate problem of course since now the function is using the version of type FOO that the developer expects.

But it creates another, more subtle problem. One that will not rear its ugly head until many months, possibly years, into the future. In fact, it was exactly this sort of decision, taken many years ago in the past, that created the current problem. Why is type FOO defined in 2 separate places? This fork in the code led to the divergent types that led the misdiagnosis that lead to yet another fork in the code.

All because the compiler in the native world can't verify that the type you request matches the type you're applying it to. Anything can be cast to anything; at best the compiler can warn you. It's the slow accumulation of nuisances like this that leads to the Big Ball of Mud. Software that's so brittle it can't be modified and must eventually be rebuilt from scratch.

In the managed world this type mismatch would never have passed compilation.

Hacker vs Professional

Even with the luxury of highly detailed specifications there arise moments where the developer has to make choices about how to implement a feature. The decisions made during these moments provide a pretty good indication of whether or not the developer is of the hacker variety or of the professional variety.

Let me provide, in lieu of a formal definition of a hacker, a few examples that illustrate the hacker mindset.

The user needs a way to enter a password with the text masked as it's entered. The framework you're using, e.g., Windows Forms, doesn't have a built in way to do this. How do you go about meeting this need?

The Hacker's instincts will involve one or more of the following steps:
  • Using the debugger to identify the internal structure of the textbox.
  • Using Reflector to decompile the framework and reverse engineer its logic.
  • Using Spy++ (or ManagedSpy) to reverse engineer the communication patterns.
  • Hooking directly into the underlying API on which the framework is based and, using the knowledge acquired in the previous steps, finding a way to intercept keystrokes then replace their display with asterisks.

The Hacker may not settle on this approach but those are his instincts. To him there is an inestimable joy in unraveling the framework, peeling back its layers, identifying what its inventors overlooked and plugging into the framework in a clever way likely not envisioned by its creators. The fact that its creators apparently didn't think of this obviously important feature is evidence of their mere mortality. And since the Hacker is also a mere mortal, he's got as much reason as anyone to fix the shortcomings in the framework. He might even get a little kick out of subverting the confines of the framework.

The Professional is also a mere mortal and like other mortals also has instincts. His instincts will typically involve one or more of the following:

  • Using the framework's documentation to identify any events exposed by the framework that can be used.
  • Mining the samples that ship with the framework for examples of solutions to similar or related problems.
  • Mining the guidance/overviews that ship with the framework for ways to plug-in and/or extend the textbox.
  • Lobby for an alternative solution (e.g., a popup that quickly disappears) that can be built with stock components included in the framework.

The Professional, like the Hacker, may also end up settling on a different solution but his instincts are radically different than the Hacker's instincts. To him the framework, shortcomings and all, represents a genuinely better way to do things. He remembers the bad old days of MFC and the worse old days of Win32 and doesn't miss the oodles of lines of plumbing necessary just to perform tasks the framework allows him to do without writing a single line of code. He'll go "hack into" the framework if he absolutely has to but would much rather find a way to do things the "framework" way.

The Hacker and the Professional have a lot in common. They're both writing software. They'll often use many of the same tools (e.g., debuggers, profilers, tracers, IDEs, etc...) but their instincts will drive them to use them at different times and in different ways. The Professional will tend to think of the debugger as a way to identify what's wrong with his own code. The Hacker will tend to think of the debugger as a way to figure out what the framework is doing "under the hood"; a way to find the secret spot that he can use to plug-in and do things the way he thinks they should have been done in the first place.

Say what you will about Hackers a few things are, in my experience, pretty universal to them:

  • They're very smart. Sometimes they're even geniuses.
  • They like to write code.
  • They have a strong sense of curiosity.

The big problem with the Hacker is that he tends to write code that is very difficult to maintain. Because the knowledge the Hacker gains by reverse engineering is usually very incomplete, often in ways germane to the way he intends to solve the problem, his solutions are plagued by seemingly inexplicable errors. His solutions work well 90% of the time then, for no known reason, crash or hang the other 10%. He might have been able to piece together what was going on "under the hood" at the time he was looking but is usually not able to reverse engineer all of the assumptions in effect.

Over time, the Hacker's hacks accumulate. He forgets the insight he had while stepping through the decompiled framework method. The next version of the framework changes whatever internal peculiarity allowed his hack to work causing it to stop working long after anyone even remembers the hack. As these hacks accumulate they start to impact the rate and quality of development. The rate slows down because making changes requires accommodating hacks that must first be deciphered. The quality goes down because the accumulated hacks interfere with each other.

Given this indictment of the Hacker, does the Professional do things any better? I'll cover that angle in the next "Hacker vs Professional" post.

Using the Recovery Console to fix "validation failed for C:\WINDOWS\system32\VSINIT.dll" error

Jump To Solution - How to manually disable Zone Alarm

From time to time files get corrupted. If you're lucky this happens in an innocuous place like a text file or a picture that you're never going to need.

If you're unlucky, it can happen in internal data files needed by some software running on your computer.

If you're extremely unlucky, it can happen in files needed by software that embeds itself in practically every operation that executes on your computer.

If you're astronomically unlucky, that particular file will be so important to that particular piece of software that it will refuse to run if the file has been changed. Since corruption counts as change, corruption prevents this piece of software from running.

Unfortunately this piece of software, AntiVirus software from Zone Labs, is then faced with a conundrum. If its data files were tampered with then it can't be sure that it's operating properly. But if it doesn't operate properly, because it embeds itself in practically every action you take on your computer, then the user won't be able to use their computer (not easily anyway).

To make matters worse, the AntiVirus software has to run to uninstall itself. But it can't do that when it thinks one of its critical files has been compromised. So you're left with a PC that crashes everytime explorer executes one of its shell hooks (e.g., you open a context menu to, say, delete a folder).

AntiVirus software, being rationally defensive, isn't easy to uninstall manually. Otherwise viruses would just uninstall it before wreaking havoc on the infected machines. Sometimes AntiVirus software embeds itself so deeply into the PC that you can't uninstall it manually even from safe mode.

How does it do that? I'm not exactly sure, but I believe in this case, a virtual device driver (vsdatant.sys) has something to do with it.

Since Safe Mode wasn't safe enough to uninstall the AntiVirus device driver or monitoring service (vsmon), I decided to give the recovery console a try.

The recovery console is an option that appears when you boot from the Windows XP setup disk. After slowly loading enough drivers to provide basic functionality, the setup disk displays a blue screen with a few options. One of the options is to, of course, install Windows XP. Another option, selected via R, is to enter the recovery console.

It's a stripped down environment with only a command prompt and a few commands. But those commands are precious! The disable and enable commands can be used to disable or enable both services AND device drivers. So all it takes to disable a crippled Zone Alarm (so that it can be manually uninstalled then reinstalled) is:

disable vsdatant
disable vsmon

That's it. After a reboot Zone Alarm can be manually uninstalled, the certificate store updated and Zone Alarm reinstalled.

To make this easier in the future I installed the recovery console as a boot option using the handy winnt32.exe /cmdconsole command in the i386 directory of the XP Setup CD.

Upgrading from VS2005 to VS2008

Migrating a few fairly large solutions from VS2005 to VS2008. A few gotches:

mt.exe, the manifest generator tool, has moved. Looks like they're taking a more standardized approach; instead of shipping a custom SDK inside the visual studio directory they're installing it in the program files\Microsoft SDKs directory. That plural looks a little weird but at least it's ALMOST the same location as when you download the Windows SDK separately and install it.

Hardcoding CRT/MFC Side-by-Side assembly versions in manifest files can result in strange SxS errors when you upgrade the compiler (upgrading to VS2008 implicitly upgrades the VC++ compiler).

Using pre-build and post-build events to do things that can be done through project settings slows down the conversion. Since VS isn't aware of these settings when paths and directories change they're not automatically corrected. In general, anything that can be done through the Project Settings should be.

Not sure why but one C++ project, a unit test project designed to test interop, could no longer statically link the c runtime library (a stream of offset errors was returned from the linker). Since it's a unit test it only runs in the build environment which has the runtime library dlls so switching this to use them was the fix.

At least on a small team, the developers can keep on working as long as they're not making project file changes. Code changes aren't, usually, affected by the upgrade.

Exploring LINQ

The opportunity has FINALLY arisen to make use of a .NET 3.5/C# 3.0 feature that I've wanted to use for quite a while. That feature is Language Integrated Query.

I'm using a pretty typical approach to learning this technology:
  • Immersion in documentation. In this case, the local version of MSDN is up on my rightmost monitor. It's open to "Language-Integrated Query (LINQ)".

  • Scratch projects for quick experimentation. Visual Studio is up on the main monitor. Only up to ConsoleApplication3 but this will go up as time goes on.

  • Text document open for taking notes. UltraEdit is up on the leftmost monitor with "linq notes.txt" open.

The .NET framework in many ways represents a unification of basic semantic constructs that "should" be available to any modern programming language (technically the CLI does that but the point still stands). So it should come as no suprise that LINQ, a part of the .NET Framework, does a bit of unifying on its own.

I've seen a lot of data access abstractions: SQLJ, PERL-DBI, Jet, ADO, ADO.NET, SAX, DOM to name a few but LINQ takes it to a higher level of abstraction. Instead of providing a way to query these data structures in their "native" language (e.g., xpath for XML, SQL for RDBMS) it provides a single query notation/syntax for querying any data (or collection).

This is a brilliant unification. While I personally enjoy learning the nuances of SQL or XPath, most developers are loathe to have to learn a new query language and will often delay it, to the detriment of the product, for as long as possible.

I am usually very skeptical of anything that smacks of automatic object-relational mapping because a lot leaks through that boundary. At some level that's unavoidable but I'm hoping that LINQ to SQL at least, provides some way to "hint" about access paths to the underlying provider.

Using Windows Search to find files

Windows Search, formerly Windows Desktop Search, has become an indispensable part of my computing experience. It's so convenient that I no longer bother organizing email; I use search to find whatever I'm looking for instead.

However, I've found Windows Search to be less than ideal when the goal is to search for files with a particular name (or pattern) as opposed to searching the contents of files. Since it searches content by default filename searches would yield so many results that wading through them to find the file was no longer convenient.

The former search mechanism, Search Companion, was slow but had built in support for searching file names instead of file contents.

Turns out you CAN do filename searches/globs by limiting the store ala:

store:file *.reg

will search for all filenames ending in the .reg extension (registry edit files).

A few other search qualifiers I'm finding useful are:

kind:email - when you only want to search through email (although this will also search emails saved to file).
from:"firstname lastname" - when you know who sent the message.
sent:last week - obvious from the qualifier.

By adding the platform SDK include directories to the list of indexed locations Windows Search becomes a great way to search through header files (e.g., WM_NC to search for certain window messages).

With the help of the store: qualifier I can finally bid Search Companion farewell.

How to detect when a WinForms Form is Maximized

The Form.FormWindowState enumeration (will either be Maximized, Minimized or Normal).

A more subtle question is when to detect that the form has been maximized. In my case I needed a way to detect when a form was maximized and/or minimized exactly once because the operation to be performed was somewhat expensive.

Control.Layout event to the rescue! It's raised whenever a control needs to adjust its layout (e.g., resized, child control added or removed, etc...). If one of your controls doesn't know how to lay itself out (because, for example, you're handling its positioning) then the trick is to handle its parent container's Layout event.

By using the LayoutEventArgs.AffectedControl you can make sure that the expensive operation is only performed when its container control is affected.

LINQ would be so very helpful in this scenario

While working on an explorer like app that provides a way to filter the elements being displayed it occurs to me that this is exactly the kind of problem that Language Integrated Query (LINQ) was designed to solve.

Right now, being limited to the 2.0 framework, filtering is done manually (foreach/for loops). As the size of the filtering framework grows this manual approach will become harder and harder to manage. The underlying elements are already objects; by switching to LINQ all of the bugs that will creep up in the increasingly convoluted manual filtering could be avoided.

LINQ to Objects would allow me to query the various collections instead of manually merging/filtering them. That would free up time to get the element object model right. Getting the object model right is 80% of the battle.

Just a little in-the-field confirmation that LINQ solves *real* problems.

Favoring Data over Code

An old rule-of-thumb states that it's easier to modify data than code. So when a given feature can be accomplished by modifying data or modifying code one should tend to favor modifying data.

One implication of this heuristic for design is to create code so that some behavior can be modified without having to modify code. At the application level configuration files are one way to accomplish this. Preferences dialogs, also at the application level, are a more-user-friendly way to accomplish this.

This rule-of-thumb is often useful at the level of individual pieces of an application. TreeViews display representations of hierarchically related elements. When it comes to providing behavior based on different circumstances of the underlying element one can take a function-oriented approach or a data-oriented approach (or some other approach).

A function-oriented approach relies on function calls to examine the element in question along with as much context as is necessary to determine what action to take. This approach starts out being easier to code, especially when there isn't much information associated with each element. In the case of the tree view, having a function that parses the TreeNode.FullPath property to figure out the corresponding filesystem location is a function-oriented approach.

A data-oriented approach relies on each element carrying enough information so that it can be acted upon based on direct examination. With respect to the preceding example, a data oriented approach derives a class from TreeNode that stores its filesystem location. So the filesystem location can be examined directly instead of being computed.

The data oriented approach ends up being much easier to manage as more and more types are represented. The problem with the functional approach is that you often have to reinterpret the code long after you've forgotten its peculiarities because types tend to be added over time. Minimizing use of global state can make this easier but in the end it's still harder to parse logic than to interpret data that's much closer to the problem domain.

Applying Artificial Intelligence

An interesting question on stackoverflow.com, a question-and-answer site that I frequent, came up about applying artificial intelligence (AI). The question was in the context of trading algorithms but since it's a question that could easily be asked in many other domains I wanted to explore it generally.

Applying AI is an entirely different proposition than creating, discovering or even understanding AI. For the most part one could profitably apply AI to a problem without understanding the internal mathematics of the AI algorithm.

A classical example of this is teaching neural networks how to perform a logical XOR (exclusive or). Exclusive OR is a function that returns 1 if the inputs differ and 0 if they are the same. This example is popular for a number of reasons but, for the purpose of applying AI, the key points are:
  • Neural networks take a vector of numerical inputs and produce a vector of numerical outputs.
  • XOR is a problem that takes a vector of numerical inputs and produces a single numerical output.
  • Applying AI involves training the neural network on a bunch of input vectors until its output matches the expected output.

When it comes to applying the neural network there's really no need to understand exactly how the neural network "learns" the XOR algorithm. (See my master's thesis for an example of spending way too much time on explaining the mathematics behind backpropagation neural networks.) For the purpose of applying the neural network to a given problem the functioning of the network itself can be considered a black box.

Of course, in the real world things are rarely that simple. Most of the problems you encounter in the real world don't present themselves as a vector of numerical inputs. Nor will your boss/client accept a vector of numerical outputs as a solution.

So most of the work of applying neural networks in the real world is that of translating the problem into a form that the AI algorithm can understand then translating its output into a form that your boss/client/audience can use. If you're lucky the solutions will be very good at least 2/3rds of the time.

There are many other techniques in AI (genetic algorithms, fuzzy logic, support vector machines, nearest-neighbor algorithms) but for the most part they all require numerical input, produce numerical output and generally do not require you to develop a formal model of your problem and its solution. If you've already got a formal model of your problem and its solution then you're probably better off using numerical or statistical methods.

Compiler Warnings

While working on a project with a medium-to-largish codebase (>10000 lines) I've run across a slew of compiler warnings. Most of these warnings are due to the following practice that I'm going to call an anti-pattern though it should probably be called a mini-anti-pattern:

catch (Exception ex)
// execute code that never references ex

Usually the unused exception ex is there to help during debugging. I do it all the time. But once you're no longer debugging that particular block, a simple // does the trick; makes it obvious to the next person that you were debugging here and makes it easy to re-use should you need to debug there again.

In this case, there were more than 60 compiler warnings streaming across the output screen due to this mini-anti-pattern. So what? Unfortunately that stream was masking a "detected unreachable code" warning. This was bad because not only had I assumed the code was reachable another process also assumed the code was reachable (and being reached!).

On the bright side, at least this provides extra motivation to clean up compiler warnings whenever it's feasible.