Ode to Notepad++

Slowly making the transition away from UltraEdit to Notepad++. UltraEdit is awesome and has been my main editor for probably going on a decade now. I've bought 3 upgrades.

Unfortunately it doesn't play nice with the revision control system at work. It takes a while to exit after editing a file during an interactive resolve session. And by a while I mean longer than Outlook! Outlook is notorious for its process taking forever to exit after you've exited the application.

Anyway, I'm finding all sorts of nifty stuff with Notepad++. Like being able to stuff a string into a bunch of lines at once. Alt-Click-drag (column mode) across the text you want to replace on all of the lines and start typing - presto! The text is inserted at the same location in all of the selected lines! Great for manually transforming text from copy/paste across different systems.

Another nifty feature is the highlight every word that matches the currently selected word. Great for finding things in log files.

Unlike UltraEdit, which threw everything but the kitchen sink into the app all at once, Notepad++ seems to be relatively minimalist. It relies on extensions to add extra functionality. So, for instance, when I wanted to navigate back and forth from whichever file and line I've been at previously (CTRL+-/CTRL+SHIFT+-), I was disappointed that it wasn't built into the app. My disappointment was premature - just looking up that plugin (there's a built-in minimalist plugin manager) exposed me to all the other plugins out there. The plugin (Location Navigate) installed quickly and worked immediately.

I still keep UltraEdit installed, mainly for things that I don't know how to do in Notepad++ yet. But that list is diminishing. And Notepad++ has already replaced it as my %EDITOR%.

From Tolerating Faults To Being Recovery Oriented

Many years ago I got into the habit of rebooting my work PC at least weekly. I don’t recall the exact circumstances which prompted this; after all, one of the bragging points for an operating system is how long you can go between reboots.

But I was raised back in the late Win 3.1 and Windows 95 days. Once had a job walking to each computer in every lab that we maintained with floppy disks to update network card drivers. We digress but the point is that back in the early days rebooting was a daily, if not multiple times daily, affair.

Another motivating factor was the transition from CGI to FastCGI back in the late 1990s. Instead of starting a massive executable to handle a single request, FastCGI allowed a single process to handle multiple requests. 

Porting a massive (at the time, to me at least) executable that:

  1. Was riddled with the simplifying assumption that it will only ever process a single request at a time.
  2. And was therefore full of global variables (some even initialized statically/at-compile-link-time).
  3. Was exceptionally brittle as these assumptions weren't explicit.
  4. Was developed by brilliant but inexperienced engineers and therefore re-discovered some of the errors a basic CS education (formal or informal) helps you avoid.
One of the things I learned back then was that even after it was mostly successful (single process successfully handling multiple requests without crashing) part of its success came from limiting the number of requests a single process handled before being recycled.

This brings us back to the title of this post. It struck me that limiting the number of requests was basically an admission that errors were not just unavoidable. Errors were going to happen so we will build in a way to firewall off many of them by starting from scratch on a regular basis.

Strictly speaking an algorithm that doesn't behave deterministically is, usually, incorrect. The lack of determinism is due to one or more errors (aka bugs) that must be rooted out. At least according to accepted wisdom.

Well it turns out that another way to increase the determinism of the system is to figure out when, on average, the error rate goes unacceptably high. Then start over before getting there. As an aside this strikes me as analogous to the trend in 20th century mathematics to prove the correctness of a given construct in the limit as opposed to correctness always and everywhere.

This is a profound alternative with implications for the design of large software systems. And by large I mean complex since size in-and-of-itself creates a kind of complexity. This alternative, Recovery Oriented Computing, basically acknowledges that at some point the complexity of a software project exceeds some threshold beyond which errors are guaranteed.

Once this is accepted, and it is a painful thing to accept for me at least, all sorts of other often overlooked considerations come to the fore. Things like:

  • How long does this thing take to start after a crash?
  • How easy is it for us to figure out when we should restart to avoid crashing?
  • Maybe we should emphasize architectural principles that optimize towards minimizing the length of time it takes to start (and restart).
  • Ditto for shutdown/stop.

Fare Ye Well, You Engineering Prophet

My boss recently changed teams (but thankfully not companies). I didn't get to work for him for very long but it was obvious almost immediately that he was one of those sui generis types that can make the difference between success taking a year and success taking 3+ years.

Over the years I've worked in several different companies: small, medium and large. A variety of industries: e-commerce/retail, professional services automation, telecom, ip video and have even been fortunate enough to work in the semiconductor industry - the source of that awesomeness that makes life as a programmer possible. But I digress...

There are always a few, and I emphasize few, that get it. From the 10ft up-close view, to the 100ft bit-of-distance view, to the 30,000ft god's eye view. And perhaps most importantly, they get what is and isn't important to focus on. Sometimes these people are in the right place at the right time and take a project that used to launch at 3am in the morning because it was so buggy that they needed that much time to put out fires to ship/go-live - every month, to a project that can ship multiple times per month with no one arriving before 7am and fires being unusual instead of the norm.

He's one of those kinds of people. An Engineering Prophet.

Designing For The Future

Agility has been a long and hard fought lesson learned over the past few decades. In CS classes we're told about ancient DOD style projects with names like Wooden Man, Stone Man and Iron Man. The pattern is pretty clear: Requirements Definition, Design, Implementation, Maintenance and Disposal (though we don't learn much about the last 2 as they're naturally somewhat difficult to teach given the time constraints of a semester).

When this theory hits practice what I've found is that the value of detailed up-front design is inversely related to how far into the future you're trying to design. Put another way: across many disciplines, not just our own, our ability to see into the future is *verifiably* not very good. Excellent works on the failure of prediction include Daniel Kahneman's "Thinking Fast and Slow" and "The Signal and the Noise: Why So many Predictions Fail - But Some Don't" among many others.

So, it's reasonable to ask, why shouldn't software be developed like other engineering disciplines? If you spend more time up-front in design then aren't you both being conscientious and efficient (problems caught in design are much cheaper to fix than in later stages)? And isn't the alternative (assuming there's only one) a recipe for death marches?
But, to choose another engineering discipline, don't mechanical engineers have the same problem? This is true but software, being soft, operates in an artificial environment. If an architect thought the Eiffel Tower would look better if only the Pyramids of Giza were off in the distance, possibly just ahead of the snow-capped peaks of the Swiss Alps no one would take him seriously. The alps aren't movable (yet), nor are the pyramids (yet).
Yet an analogous request in software, precisely because it's soft, can't be dismissed outright because software lacks (usually) such hard (and obvious) constraints. Built a program that knows how to read log files? Excellent! We've got tons of database logs that we need to parse. It wasn't designed for reading database log files? No worry, it's software, we can change it.

Essentially the softness of software, its malleability, guarantees that requirements defined will be incomplete (=incorrect) the second after the ink dries (assuming you still print them out). The Pyramids of Giza can be moved to France, the mountains can be twisted into a more photogenic orientation and nearly any idea is not only possible but a wonderful morass of design into which us software engineers happily but, usually unprofitably, dance.

A natural response to this is the impulse to "nail down the requirements and don't allow them to change". Unfortunately malleability is possibly the biggest value proposition that software brings to the table. Software is valuable precisely because it can be easily changed without having to reorient an assembly line, stamp out possibly millions of replacement widgets, ship them, track them, etc...

So we have this intrinsic tension. Software is valuable because it's easy to change. Accommodating change requires design. Design is essentially trying to predict the future. Prediction is hard because of change and change, as a fundamental value proposition, is and will always be the norm in software. Ergo, we should expect our ability to predict the future to be even worse for software than other disciplines (for which malleability is a less significant part of their value proposition).

So what do you do when you must see into a future to satisfy requirements well but that future is almost certainly guaranteed to have changed the next time you look? Look more frequently and don't spend too much time looking too far into the future.

This, to me, is the reason Agile gained prominence in the mid-90s. It's also, IMHO, the driving force behind shorter release cycles, earlier feedback and many of the other practices associated with Agile. We can't predict the future since, in our business, that future does not really exist yet. It's created in the iteration cycles during which stakeholders mold and shape the product; the molding and the shaping itself molds and shapes the future as stakeholders discover what they like, what they don't like, what they thought they'd like but don't, what they don't like but works so well they'll take, etc...

The further into the future we engage in detailed design the more time and energy we spend on a future that will be very different by the time we get there. The lack of obvious artificial constraints means that few blind alleys can be discarded outright. The future is often so different when we get there that instead of helping ourselves by designing for our expected future, we've designed ourselves into a corner with loads of technical debt created to get out of that corner.

Unit Tests, Robocopy and Shallow Mirroring

To increase the fidelity of a set of unit tests I needed to mirror a directory structure. The catch is that there are huge files that I don't want to copy. My first thought was to write a program that only copies the first N bytes of each file while mirroring the directory structure.

On a lark I figured I'd check if robocopy could do something like this. Turns out that it can! Dozens of gigs of bits conserved! Thanks robocopy!

robocopy srcdir destdir /E /CREATE

Thread view in Touchdown Exchange Android Client?

Finally getting around to reading through the manual for the exchange client I use on android (touchdown). Found a nifty feature I've wanted for ages: Thread view!

I find it much easier to keep the context of discussions by having them listed together.

Now if only they gave me an easy way to either save a .eml file or attach a message in response to another message. When there are lots of recipients it's a pain to have to add them and breaks the thread you're trying to update...