Monthly Archives: April 2007

You down with FPP?

Steven is wondering why IT recruitment agents are not well respected:

http://searchniche.blogs.com/nzrecruiter/2007/03/why_recruiters_.html

Here is a story from my time in London, as told to me by a flatmate, which might have some of the answers. I’ve changed the names to protect the fictional.

Scene: Jim is a recruitment agent. Bob is an IT professional looking for work.

[Jim's phone rings]

Jim: Hello
Bob: Hi, my name is Bob. I’m calling about the ASP job you have advertised on Jobserve

[An uncomfortable pause]

Jim: Oh, the ASP job. Well, actually, that position has just been filled.

Which is a recruitment agents way of saying that the job never existed in the first place. Actually he was just using the advert to try and build up a database of candidates that he could then pimp to his clients.

Anyhow, the conversation continues …

Bob: Okay, what else do you have at the moment that might suit?
Jim: Actually I have another client who might be looking for somebody with ASP experience. How many years’ ASP do you have?
Bob: [sigh]

How has tenure come to be so important in recruitment? Why do agents talk in terms of “having” a technology? Surely there are 1,000 shades of grey when it comes to technical experience? I never answered those questions satisfactorily during my time overseas.

Bob: I have been using ASP for about 3 years.
Jim: Really, is that all?
Bob: Yeah, but I’ve used a lot of different technologies prior to that. I think I know ASP pretty well after 3 years.
Jim: I’ve spoken to a lot of candidates this morning who have 5 years’ ASP experience, so they would probably be preferred.
Bob: 5 years eh? Is that really 5 years’ experience, or is it 1 year repeated 5 times?
Jim: What?
Bob: Never mind.

Bob decides that he’s probably wasting his time, so an experiment is in order…

Bob: What other web development positions do you have open at the moment?
Jim: We have loads. What technologies do you have? HTML? CSS? SQL?
Bob: Yeah all of those.
Jim: Okay, great.
Bob: I also have about 5 years’ MMP experience. And a little bit of FPP experience.
Jim: A ha. Okay, great.

At this point Bob has confirmed that he is wasting his time with this agent. To a New Zealander MMP and FPP are two alternative electoral systems. To a dumb London recruitment agent they are two more TLAs to enter into their database – independent of their relevance to an actual web development position.

Bob: Yeah, do you get many web development jobs requiring MMP or FPP experience?
Jim: Yeah, all the time. You know what gov, just doing a quick search, we don’t have any open positions at the moment, but I’ll add you to our database and call you as soon as something comes in. Actually I’m going to see a client this afternoon who is probably quite interested in …

[Bob hangs up]

A battle on multiple fronts

When Adobe acquired Macromedia (for US$3.4 billion!) I have to admit I was sceptical.

Between them they have two of the most installed applications that exist on the fringe of web content: Acrobat and Flash.

But, I doubted their ability to integrate the various product lines and generally get their act together. However, get their act together they have.

All of a sudden they are being seriously compared to Microsoft.

Adobe has momentum. Just like that Microsoft is fighting a battle on multiple fronts.

This week is Microsoft’s turn in the spotlight (or is that Silverlight?)

Will be interesting to watch…

UPDATE: Ryan Stewart has some more details on Silverlight.

Children Of Men

How did they do that?

If you haven’t seen Children Of Men, starring Clive Owen and directed by Alfonso Cuarón, you should.

Perhaps it’s just my curious nature, but I always get distracted watching a film like this, constantly thinking: how did they do that?

In this case there are a number of scenes which are filmed in a long single take. This short documentary has the behind the scenes story of how they achieved it:

http://www.youtube.com/watch?v=EjNk-nxHjfM

Interesting!

#2

Just a quick note to announce the arrival of baby #2.

He (it’s another boy!) was born yesterday morning.

So, it’s going to be a bit quiet on here for a few days.

Take care,
Rowan.

:-)

Enhanced metafiles

How hard is it to copy a diagram into a PowerPoint presentation in a reliable format?

Copy and paste is a pretty fundamental operation. When you copy and paste a file, the file that is created is a new file with no relationship to the file that was copied, apart from the fact that it is initially a copy. Any changes you make to the new copy are not applied to the original and vice versa.

But, for a reason that I cannot explain, Office messes with that simple model. Instead the default paste tries to embed the original file.

Why is this the default?

Is this really what most people expect to happen when they click paste?

I doubt it.

To get around this broken default you need to select the ‘Paste Special’ option. And then, God help you!

There are seven different options to choose from:

Enhanced metafile dialogue

I have a Computer Science degree and the only ones that make sense to me are the four that appear to map to different image formats (Bitmap, GIF, PNG, JPEG). Even then it’s not clear to me which of these formats would make the best choice.

What the hell is an ‘Enhanced Metafile’? Why would I choose a ‘Windows Metafile’ format when I could have the enhanced version? Should I prefer the Windows flavour of Metafiles or the Enhanced ones (how exactly are they enhanced)? And, while I’m at it, what is the sort order on this list?

The picture below is taken from a random Google image search.

Marjorie

I don’t know this women. But for arguments sake let’s call her Marjorie and think for a second what she would make of this dialogue.

When are those of us who build these tools going to start putting ourselves in the shoes of people that don’t speak C#?

How to hire really smart developers?

The best web developers don’t ever look in the situations vacant.

Not even online.

But they do look at source code!

So, a while back I quietly added this HTML comment to the top of the Trade Me homepage:

<!--
If you're looking at our HTML source,
you're exactly the person
who should send us your resume.
View current vacancies at
http://www.trademe.co.nz/careers
-->

It’s been surprisingly effective at targeting the type of smart ASP.NET web developers we want to hire. But, not effective enough it would seem.

Alan, who heads up the development team at Trade Me, is advertising a couple of specific roles at the moment:

We’re also currently advertising for a Web Designer.

If you’re a smart developer or designer there are lots of opportunities at the moment. There definitely seems to be more vacancies around than people to fill them.

So, as a candidate, how do you decide what’s the best place to work? I would have thought that the chance to work on NZs largest web site would be pretty attractive.

Recently closed tabs

As of this morning I have a new favourite Firefox feature:

History > Recently closed tabs

As the name suggests it lists any tabs that have been closed in the last wee while. It’s like an undo button for when you accidentally close the wrong tab.

Simple, but brilliant!

(extra: there is even an extension available which allows you to add this function directly to the toolbar)

One wonderful day

We’ve recently been talking about the next round of under-the-covers clean up work at Trade Me.

In the last couple of years we’ve migrated all of our sites from VBScript to ASP.NET and made some pretty major changes to our development and testing process. It’s been a lot of hard work, but the results have made it well worthwhile.

However, the ground is constantly shifting. There are always further improvements to be made. You’re never finished. Sometimes it can feel like you need to run just to stay still.

In all of this I was reminded of an excellent 37signals post which contained this insight:

“The business is not going to slow down to allow you to clean all these things up one wonderful day. It just won’t happen. “

Sometimes the amount of work involved in these sort of improvements is daunting – for example, thinking about adding automated unit tests to a large code base that has never been written with testability in mind can seem like an impossible challenge. But, unless you change the way you are working now the gap between where you are and where you want to be is only ever going to get bigger and bigger.

To make an application more capable over time it’s important to include enough time in the schedule to remove scar tissue. But you also need to stop cutting yourself.

Find out what your audience is after

Tom Beard (of WellUrban fame) says …

“I’m always amazed, and sometimes amused, by what my visitors have been searching for. Sometimes, a steady stream of search terms can be a good reminder to write about something you’ve had on the back burner for a while.”

And …

“You should always write about what matters to you, but it’s good to know what’s newsworthy to your readers.”

Following that advice, and taking a good look at the logs for this blog for the first time, I have to conclude that the #1 reason people come here is to find out how to access Trade Me from work when it’s blocked. How depressing!

Some others are looking for information on monkey suits or advice on how to get a real job.

One guy (or maybe girl?) ended up here after search for “Where does Sam Morgan live”! :-)

I’m also getting quite a bit of traffic from people searching for information about Hamish Carter. How did both Rod and I end up on the first page of Google search results for his name?

All of this reminds me of an excellent post from Eric Lippert a few years back, which I link here for your reading pleasure.

(and, if you like that, also check out his follow-up posts: part 2 and part 3)

Seriously, thanks to everybody who has discovered this blog over the last few months. If there is anything you’d like me to talk about here I’d be keen to hear from you.

And, if you’ve found it interesting perhaps you’d like to subscribe to my RSS feed? :-)

Questions from Tim Haines, Part II

This is Part II in a two-part series. Part I covers the Trade Me application architecture.

Tim’s second lot of questions are about our dev tools and process:

Q: Any third party tools in the software or the dev/management process?

Q: What source control software do you use, and how do you use it?

Q: How do you manage roll outs? Dev/Staging/Live?

Q: Do you use pair programming, or adopt any other methodologies from the agile world?

The answers to these questions are just a snapshot, capturing how we do things today (early in April, 2007).

I go far enough back to remember when our “development environment” was Jessi’s PC (Jessi at that stage was our entire Customer Service department!) Back then there was no source control as such, we all shared a single set of source files. To deploy a change we would simply copy the relevant ASP files directly onto the live web server and then quickly make the associated database changes.

Somehow it worked!

Ever since then we’ve been constantly tweaking the tools and processes we use, to accommodate a growing team and a growing site. As our application and environment has evolved and become more complex our tools and process have had to change also.

This change will continue, I’m sure. So, it will be interesting to come back to this post in another 8 years and see if the things I describe below sound as ridiculous then as the things I described above do now.

Also, the standard disclaimer applies to these ideas: what makes sense for us, in our environment and with our site, may not make sense to you in yours. So, please apply your common sense.

Tools

Our developers use Visual Studio as their IDE and Visual SourceSafe for source control.

All of our .NET application code and all of our stored procedures are kept in a SourceSafe project. Developers tend to work in Visual Studio and use the integration with SourceSafe to check files in and out etc.

Thus far we’ve used an exclusive lock approach to source control. So, a developer will check out the file they need to make changes to and hold a lock over that file until the changes are deployed.

However, as the team gets bigger this approach has started to run into problems – for example, where multiple developers are working on changes together, or where larger changes need to be made causing key files to be blocked for longer periods.

To get around these issues, we’re increasingly working on local copies of files and only checking those files out and merging in their changes later. I imagine we will shortly switch to an edit-merge-commit approach, and that will require us to look again at alternative source control tools (e.g. SourceGear’s Vault, Microsoft’s Visual Studio Team System or perhaps Subversion – we’d be interested to hear from anybody who’s had experience with any of these).

Release Manager

At the centre of our dev + test process is a tool we’ve built ourselves called the ‘Release Manager’.

This incorporates a simple task management tool, where tasks can be described and assigned to individual developers and testers. It also hooks into source control, and allows a developer to associate source code changes with the task they are working on.

This group of files, which we call a ‘package’, may include ASPX files, VB class files as well as scripts to create or replace stored procedures in the database.

The tool also incorporates reports which help us track tasks as they progress through the dev + test process. These are built using SQL Reporting Services.

Environments

We have four environments:

  1. Dev: this includes a shared database instance and local web servers for each developer.
  2. Test: this includes a production-like database (actually databases, as we now have multiple instances in production) and a separate web server.
  3. Stage: our pre-production environment, again with it’s own web server
  4. Production: our live site, which actually incorporates two environments currently, one in Wellington and one in Auckland.

Developers typically work on changes individually. We have a code-review process, so any code changes have two sets of eyes over them before they hit test.

Once a code change is completed, the developer will create the package in Release Manager and set the task to be ‘ready to test’ so it appears on the radar of the test team.

We have a web-based deployment tool which testers can use to deploy one or more packages into the test environment. This involves some Nant build scripts which get the source files for the selected packages, copy these into the test environment and then build the .NET assemblies on the test server. The build script also executes any associated database changes that are included, and then updates the status of the package/s to ‘in test’.

The deploy tool is able to use the data from Release Manager to identify any dependencies between packages. Where dependencies exist we’re forced to deploy packages in a certain order, but in the general case we’re able to deploy packages independently of each other, which provides a great degree of flexibility and allows us to respond quickly where required (e.g. when there is an urgent bug fix required).

Production

Once a package has been tested the test team use the same deploy tool to move the package into the stage environment ready for go-live.

From there the responsibility switches to the platform team, who manage our production web servers. They have automated scripts, again built using Nant, which deploy from stage to our production environment/s. These scripts update configuration files then copy the required production files to the various web server locations. It also manages the execution of database scripts. The idea is to get everything as close to the brink as possible (which is the time consuming part of the deploy process) and then tip everything over the edge as quickly as possible, so as to minimise disruption to the site.

Typically we do two production releases each day, although this number varies (up and down) depending on the specific packages. In most cases these releases are done without taking the site offline.

The bigger picture

Our dev + test process is just one part of a much bigger product management process, which is roughly represented by the diagram below (click for a larger view):

Product Management Process

The other parts of this process are probably fodder for a separate post, but it’s important to note that there is a loop involved here.

Most of the changes we make to the site are influenced heavily by previous changes. In many cases very recent changes. This only works like it does because our process allows us to iterate around this loop quickly and often.

While we don’t follow any formal agile methodology, our process is definitely lightweight. We don’t produce lots of documentation, which is not to say that we don’t design changes up-front, just that we don’t spend too much time translating that thinking into large documents (it’s not uncommon for screen designs to be whiteboard printouts for example).

While we do make larger changes from time to time (for example, the DVD release which went out last week) the vast majority of changes we make are small and seemingly insignificant. Again, this only works because each of these small changes is able to flow through with minimal friction added by the tools and processes.

I’d also hate to give you the impression that this process is perfect. There is massive room for improvement. The challenge for us is to continue to look for these opportunities.

More?

That’s it for Tim’s questions. I hope some of that was useful?

If you have any other questions, ask away. You can either place a comment below or contact me directly. My email address is in the sidebar to the right.

eBay architecture

Here is a really interesting slide deck about the eBay software development process and the evolution of their architecture (via Matt from the Trade Me database team):

http://www.addsimplicity.com/downloads/eBaySDForum2006-11-29.pdf

As I’ve cautioned previously unless you happen to run a really really big website many of the approaches they describe here are probably more interesting than practical.

As far as I can tell from the architecture diagrams included in this deck what we have today at Trade Me is a mixture of their v2 architectures. Those maxed out for them at around 50m registered users, so that would suggest we have some more growth to go before we need to start looking too seriously at what they did in v3.

But, nonetheless, it’s interesting to think ahead to the challenges down the track – for example around the architecture of the application code, and the implementation of search within the database.

DVDs on Trade Me

Last week we released some changes to the DVD category on Trade Me.

Here is the site announcement about this change:

“We’ve made some exciting changes to the DVDs & Movies category designed to make life easier for buyers and sellers.

To place a listing sellers just need to enter the DVD title and we will automatically include all of the other details from our catalogue of over 10,000 titles. This includes cover art, genre, synopsis, director, cast, classification, etc.

Buyers can browse all of the items for sale from the new DVDs & Movies page. Listings are categorised by genre such as Drama or Action, or by other criteria such as New Releases, Top 100 and New Zealand movies.

Important information for sellers:

  • Sellers who have exceeded their free listing allowance will not be charged listing fees for DVDs listed using the catalogue. However, successful DVD sales will continue to count towards your overall free listing allowance.
  • DVDs listed using the catalogue can be listed for up to 14 days at no extra charge.
  • We encourage DVD sellers to list with a start price equal to reserve and to include a Buy Now price. A 25c reserve fee will apply to auctions with a reserve price higher than the start price.
  • DVDs listed using the catalogue receive free gallery. Bold and featured listings are no longer available. All sellers with DVD listings at the time of this change have had promotional fees refunded.
  • All of the titles in the catalogue are classified for sale in NZ. However, sellers must confirm that the specific DVD they are selling is a legitimate copy with a NZ classification sticker.”

(although, of course you already subscribed to the RSS feed for these announcements, right?)

If you haven’t yet, take a minute and check it out.

This change is great for buyers. It makes it much easier to browse the DVD category, as the focus is on the titles rather than on individual listings. So, for example, you can quickly find all of the movies starring Brad Pitt, or all of the movies directed by Tim Burton. If you’re after a specific title you can use to the search on the sidebar to track it down.

It will be interesting to see what happens to prices over the next couple of weeks, once the majority of listings are within this catalogue. For example, take Once Were Warriors. As I type there are 8 copies available on the site. The Buy Now prices range from $15 up to $24. It’s hard to believe that the $24 copy will sell now that it’s easy for buyers to quickly find and compare all copies for sale like this.

We’ve also introduced a ‘reserve fee’ of 25c in this category, which should encourage sellers to run auctions with start = reserve (i.e. we don’t expect many sellers to pay this fee) and we encourage sellers to specify fixed shipping costs and a Buy Now price so that buyers can complete the purchase in one visit.

But, it’s not all about the buyers. As described in the announcement this change also removes just about all of the pain from the sell process too. Where as in the past you’d need to track down the details for the listing (e.g. from IMDB) and take photos etc, it’s now very quick and easy to place a listing. You just enter the title of the DVD and we do the rest!

What do you think?

Paul Graham

A while back Natasha (who works with us at Trade Me and is also one of the people behind Webstock) asked me to nominate my dream speaker for their next conference.

For me it was an easy question to answer: Paul Graham.

If you’re not familiar with his writing, please stop embarrassing me by reading this relative drivel and go and read some of his stuff instead.

His latest article is here:

Why to Not Not Start a Startup

Here is a quote that appealed to me:

“If you took a nap in your office in a big company, it would seem unprofessional. But if you’re starting a startup and you fall asleep in the middle of the day, your cofounders will just assume you were tired.”

:-)

VB.NET – arrghh!

I get really annoyed when smarmy C# developers look down their noses at VB developers.

If I’m feeling like an argument I’ll ask them to write something in C# that I couldn’t replicate in VB. I’m not saying it’s impossible, but so far nobody has been able to do it (feel free to post a comment below if you’d like to try).

All of the Trade Me code is VB, and on the whole it does a perfectly good job. (btw, there is a good reason why we decided to migrate to VB rather than C#)

But, really, sometimes VB just doesn’t help itself.

Witness this recent email from Annie, one of the developers at Trade Me …

From: Annie
To: Development
Subject: Rounding numbers in VB.NET and SQL Server

Hey guys,I’ve just been debugging a situation in the sell process (for way too long) where two decimal numbers (9.5 and 10.5) were BOTH rounding to 10 by a call to exactly the same code: a simple call to good ol’ CLng.After searching around, it turns out that CLng, CInt, CByte, CCur and Round all implement what is known as “Banker’s Rounding” in VB.NET. This is also sometimes called “round to nearest”, or “round to even”. Basically, Banker’s rounding rounds 0.5 up sometimes and down other times. Apparently the convention is to round to the nearest EVEN number. Therefore, 1.5 and 2.5 would both round to 2. Likewise, 3.5 and 4.5 would both round to 4.If you’re interested, the rational behind this is that if you were rounding large sets of numbers, constantly rounding 0.5 UP would result in a bias as only 4 of 9 numbers (0.1, 0.2, 0.3 and 0.4) would cause a round DOWN and the remaining 5 numbers (0.5, 0.6, 0.7, 0.8 and 0.9) would cause a round UP.

ANYWAY, as you can imagine, this causes what seems to be slightly erratic un-deterministic behaviour to those of us who are used to “arithmetic rounding”. And unfortunately, it looks like there are quite a few calls to CLng particularly in the sell process. Calls to CLng are currently being used when checking for minimum or maximum values for attributes and eventually for storing and displaying integer values entered for said attributes. For example, if I enter 49.5m2 for the floor area of the apartment I’m trying to sell, it will round it up to 50m2. But if I enter 50.5m2, it will also round to 50m2. I dunno about you guys but I was always taught to round 0.5 UP in school, without exception. Anyhoo, some users might consider this to be a bug (I know I find it weird).

To add to the confusion, I just did a bit of a test in SQL Server and it looks like casting a decimal value to an int just truncates the number to the most precise digit (e.g. cast(10.9 as int) = 10).

So, basically just use a bit of caution when using CLng, CInt, CByte, CCur and Round in VB.NET and cast in SQL Server as they may be returning results quite different to what you’d expect, and worse yet, different results when calculated in the code as opposed to a stored proc.

Apparently Format and FormatNumber perform “arithmetic rounding” and although they spit out a string, in many of our cases these functions will do the trick as we tend to treat most of our integers in the sell process as strings anyway.

But yeah, just thought I’d share this with all of you. Please feel free to add to this discussion/rant.

/Annie