There is one type of interview question I see pretty frequently that drives me absolutely bonkers. I’ve never actually been given this queston myself, but I’ve worked at a lot of companies that like to ask it (I’ve been lucky). Just recently, a friend of mine who has been interviewing mentioned some of the questions he was given; one of them fit this description and all I could do was feel bad for the poor guy. There’s no way he answered it correctly, despite being a very capable software engineer.

I refer to this question as the “Career In A Bottle Question”

The Career In A Bottle Question

Very often this question is actually given prior to an interview, as something you submit to a company after passing an initial phone screening but before they bring you in. The question takes a form similar to the following:

Write a program that will [anything]. Please write it as though it were going into production.

It’s the second sentence that does a candidate in. Why? Because there is no right way to answer the question.

More specifically, there is a right way to a single interviewer, but you have a 50/50 chance.

Allow me to elaborate by making this concrete. Imagine the question is as follows:

Write a program that calculates what phase the moon is in for a given date (the input) as though your code were going to be used in production.

If I, as an interview candidate, were to see a question like this, my mind would go through the following thought process:

Well, that’s actually a really simple algorithm. Read a date from commandline, do a little calculation, print out the phase. But wait, they said they want it to be like production code. So I should probably design it in a more object-oriented modular manner than just some procedural script. And when I write production code, I unit test it, so I should write some unit tests. And with all that code, I’m going to need a build file… Jesus, this seems like overkill for a 10 line script.

It’s the last sentence that destroys me, and many other capable programmers. On the one hand, production code needs tests, it needs a build, it needs to be well designed, and it needs to work. On the other hand, the problem I was given is a 10-liner, and it seems like overengineering to do anything more.

If I write the simple script version, will the interviewer think that I write one-off production code without tests?

If I write the full system, will the interviewer think that I overengineer everything I write and anyone who maintains my code will have to deal with an overly complex system?

I have no way to know.

I have no idea what the interviewer’s expectations are, so I have to guess. I have, essentially, a 50/50 shot at guessing correctly. To make matters worse, my answer will likely go through a number of different interviewers, and I have a 50/50 shot at having guessed correctly with each of them. Assuming that a single “no” from one of the interviewers means I don’t get a job offer, having 2 interviewers gives me a 25% chance of success. Three interviewers gives me a 12.5% chance. A team of 6 or 7 interviewers (extremely common in up-and-coming companies) gives me virtually no chance at all.

Generally the conversation about the interviewee’s qualifications goes like this:

Interviewer 1:

I really liked his answer to this question. He has a good design, good unit test coverage, and a fast build.

Interviewer 2:

Yeah but he wrote this future-proof ten-class system for a simple math function. Why did he write code for requirements he wasn't given? I don't want to hire someone that's going to spend 2 weeks writing something that should take a day.

Interviewer 1:

Yeah, I guess I can see that. He did write an awful lot of code.

Interviewer 2:

Let's keep looking.

You can’t possibly have written something to please both of these interviewers without hitting the magical, arbitrary middle-ground exactly between the two of them. If there was a third interviewer, you’d have virtually no chance.

What’s Going On Here?

Fundamentally, the reason this question is bad is that there is a mismatch between two of the constraints. On the one hand, the interviewer is told to code as though it were something he were assigned at work, but at the same time a programmer would never get such a simplistic task.

Interviews are tricky, because you can’t tell your candidate to reinvent your product for you in the interview. Questions need to be simplified to make the interview process manageable, but once something becomes simplified it leaves the realm of something you’d actually have to write in production. As a result, there is no way to word a question like this without making it conflict with itself.

Essentially, this question breaks down because it’s testing too many of the candidate’s skills simultaneously. What the interviewer wants to know here is:

• How are the candidate’s algorithmic skills?
• How are the candidate’s design skills?
• How are the candidate’s testing skills?
• How are the candidate’s Java/Ruby/C#/Whatever skills?

Occasionally, these questions are worded in a way to also test a candidates familiarity with a framework as well, such as an additional constraint of “Use Spring” or something of that nature.

The problem is that you can’t get the answer to all of these questions while keeping the questions manageable in scope and size. Good design skills mean a high degree of modularity and encapsulation, but testing modular code means writing more tests than is appropriate for an interview. Good algorithm skills mean writing simple procedures that perform a task, but simple procedures do not lend themselves to the scope that give insight into a candidate’s design skills or language knowledge.

How To Change The Question

If you’re a company looking to hire, don’t ask the Career In A Bottle question or you’ll wind up passing on a number of highly qualified candidates. The only people who will get in are the ones who just so happen to hit the magic compromise between the ideal responses of all of your team members. It turns what should be a game of skill into a game of luck.

Instead, figure out what you’re trying to ascertain, and ask direct, focused questions centering around those topics.

If you want to know a candidate’s algorithm skills, ask a question fitting this template: “Write a method that takes a [something] and determines [something else]. Use any language you wish.” By saying that you only want a method, you are removing any concerns the candidate might have about your expectations regarding class design and testing. By saying the candidate can pick any language, you are making sure they aren’t wasting mental power on remembering language specifics or APIs. The answer you want from the candidate should be whiteboarded pseudocode, easy to read and functionally correct. If you need to say something like “pretend that there’s no something-or-other method,” feel free.

If you want to know a candidate’s design skills, describe the requirements of a large-scale system and ask that they draw UML describing the classes in the system, how they interact, what extends what, what’s an interface, etc. The question should sound similar to “Sketch the design a system that [something].” By saying sketch, you’re making it clear that you don’t want any code. You want method names, class names, and so forth. The candidate should give you boxes, arrows, and method signatures on a whiteboard, though I wouldn’t be too strict about the adherence to the UML standard unless it’s important to your company.

If you want to know a candidate’s testing skills, then YOU should be the one drawing boxes and arrows on a whiteboard. Sketch out a system with classes and methods and ask how the candidate would test your system. Ask about how he or should would strive for complete test coverage on a unit test. Ask about functional tests and integration tests that test multiple aspects of the design. The candidate should answer this question with, in my opinion, words only. Descriptions and methodologies, without writing anything down.

If you want to know a candidate’s language skills, then and only then should you give an assignment. Pre-interview assignments aren’t good at determining anything other than a candidates ability to write code in a specific language and work with the API for that language. Asking language and API questions during an in-person interview is pointless since it tests a skill that developers don’t need unless your company’s internet connection dies and all of your books catch fire suddenly. The best way to give a programming assignment is for YOU to actually provide some code. Provide an interface and a test or a program that uses an implementation of that interface. Ask the candidate to provide the implementation. This way, the candidate can actually compile and run the code to make sure it does what it should (an ability he or she would have in real life). It also gives you control over what you want to make sure the candidate is familiar with. If it’s important to you that the candidate be familiar with the java collections library, write an interface that passes various collections around.

Weeding Out

The biggest objection someone may have to my suggestions above is probably that the purpose of the assignment is to weed out candidates before everyone has to take time out of their day to interview someone in person.

I believe that you can actually convert the information you wish to gather from a programming assignment into information you can easily gather with a phone screening. The algorithm question can simply have the complexity of the question decreased to a point where you could describe the algorithm with words alone, over the phone. The same goes for the design and test questions - you can make the system under discussion less complex and the questions and answers can be transmitted through speech alone. And you can still do language/API/framework familiarity testing by giving a programming assignment following the specifications described above.

In my experience, following these suggestions may make the interview process take slightly longer than you may like, but the end result is that you severely decrease the number of false negatives. Those qualified candidates that you aren’t hiring because of bad questions are getting jobs elsewhere, and it may be your competition.

Don’t throw good programmers away because you’re too impatient to ask narrowly-scoped questions.

Don’t ask the “Career In A Bottle” question.

Update: Some people have pointed out that you can simply ask the interviewer what he or she is expecting from you. While this is usually a decent workaround, I’ve often seen the attitude taken that figuring out what “production ready” means is part of the question. One particular horror story I’m aware of from a friend has him asking the interviewer ‘Do you want me to include tests?’ and having him respond ‘Do you not normally test your production code?’

If you were to insist on asking a question like this, adding additional information to the description such as “I’m expecting unit tests and a build file” makes it far more palatable.

I still feel that the question is fundamentally flawed because it attempts to ask too many questions simultaneously, and would be better split into a few more narrowly-focused questions.