Archive for March, 2007

More rejected candidates

Friday, March 30th, 2007

I rejected 4 job candidates in as many days, using the new technique of “Give them an assignment, see what happens.” The difficult question right now is what kind of assignment I should give. I sometimes feel like what I’m giving is too hard for a junior and if I had interviewed myself out of college I wouldn’t have hired myself – a mistake I should avoid.

3 of the techniques I tried are:

1. Look at this shader code I wrote and give me feedback on how to optimize or improve it
2. This lookup is using a linear search. Change the data structure to an optimized list.
3. Write code to send a list to the server, have the server sort it, and send it back

The answers I’ve gotten are:
1. Uh, it looks OK. (On further code review the guy didn’t understand basic C++)
2. Implemented using a map, instead of an ordered list, and was incorrect as he didn’t understand how to use C style strings
3. Had the client sort the list instead of the server, and attempted to write quicksort on the spot rather than using the existing ordered list class I provide, or stl, and also couldn’t make a project for RakNet after 1.5 hours even though I give example projects and instructions on it.

So far there’s only one guy I rejected out of hand due to price, which was $35 an hour where I knew I couldn’t talk him down. He was also the best candidate by an order of magnitude, with real experience. I’m on the edge about that one. I can’t afford it, but I feel someone like that could make real progress in the game.

Sound effects

Friday, March 30th, 2007

I’ve been trying to contract out sound effects. I have over 200 sound effects for my game, most of which are repeated variations. For example, for the cloaked mine weapon I have

CloakedMineActivate.wav
CloakedMineBulletAmbientSound.wav
CloakedMineBulletHitAsteroidSound.wav
CloakedMineBulletHitDestroyableSound.wav
CloakedMineBulletHitFighterSound.wav
CloakedMineBulletHitWallSound.wav
CloakedMineFailOnAmmo.wav
CloakedMineFailOnEnergy.wav
CloakedMineShrapnelAmbientSound.wav
CloakedMineShrapnelHitAsteroidSound.wav
CloakedMineShrapnelHitDestroyableSound.wav
CloakedMineShrapnelHitFighterSound.wav
CloakedMineShrapnelHitWallSound.wav

For the 20 different items and weapons I have I got a quote back of $17,000, which is about $12,000 more than I was expecting. So at this point I can either cut things down to some important component (enough to ship), keep looking for another company, or only have them do critical ones, and do the remainder myself.

I didn’t expect sound to be this expensive – I guess it’s another lesson I should have learned before I started.

New interview technique seems to work

Monday, March 26th, 2007

Today in my interview I didn’t even bother with the usual process of asking technical questions. All I did was say “Build the game” and after that was done “Implement this task” Building the game took an average amount of time, but the task was a 10-15 minute task he took an hour to do, and did wrong, so I ended the interview and there were no doubts from either of us that I made the right decision.

Total time taken away from me was probably 15-20 minutes, and I got a perfect measure of the candidates suitability.

When I think back on my previous bad hires, this approach would have screened out all but one of them, and it would have even worked there if I had given a more complex task.

It seems obvious now, but I can see how my past approach of asking lots of technical questions wasn’t sufficient in itself. All that did was keep away guys that didn’t know the performance differences between a linked list and a binary search tree. Since most candidates do, this didn’t help me that much.

The best approach I have right now is:

1. Prescreen with technical questions that require good fundamentals (use of data structures and 3D math). This should only take the minute or so to send an email, and another 5 minutes to scan over the reply. Also require salary requirements and start availability in the reply.

2. Give a short but real task, that involves working with other code, that is typical of the kind of tasks you might actually give. This will tell you if the candidate looks at what he is doing carefully, can read and work with existing code, can follow and understand instructions, and how fast he works. I think speed is a good measure of ability.

3. Hire on an hourly basis at first, with a clear understanding that the first few months are probation. This will save you from guys that are lazy, or who got past step 2 and shouldn’t have.

I’m happy that I saved a lot of time and money today with this new technique. Too bad I didn’t know it months ago ­čÖü

New interview technique

Sunday, March 25th, 2007

I got another bad hire a couple of days ago. One thing I noticed is that even though I spent 2 hours carefully interviewing him, it only took me 5 minutes of working with him before I started seeing red flags and warning signs. Among other things it took him 5 hours to compile, download, and run the game, when I’ve had others in the past do it in 30 minutes. That’s raw experience right there.

I’m going to try a new interviewing technique, which is to give a real programming task that

1. Interfaces with a larger system, so you have to work with someone else’s code
2. Has multiple solutions, some of which are better than others
3. Takes enough time to require real though and design, but not so much that it would take someone competent longer than an hour, and designed such that a great person could do it in minutes.

I’m trying to think of some good problems. Right now I’m partial towards giving an assignment to extend RakNet in some way. I have a guy scheduled for an interview tomorrow and will see how this goes.

What I’ve learned about the hiring process

Wednesday, March 21st, 2007

6 months ago, when I started my Indy game company Jenkins Software, I had virtually no interviewing experience and made some hiring mistakes. Here’s what I’ve learned, now that I know a bit more about the process. I can’t go back in time to send this to myself, but hopefully others starting companies can learn from my mistakes.

It takes time and costs money to find the good people.

Originally, I thought posting a job ad on Gamasutra would net me hundreds of applications, of which some small percentage (5%) would be great guys. And I’d hire those guys.

As it turns out everyone wants the great guys, not just me, and the other companies have a lot more money and perks to offer than I do. So great guys are rarely on the market, and when they are they tend to be snatched up very quickly. The only reason I’m competitive at all is that I offer telecommuting, which is such a perk that it outweighs everything else for many people. On my own resume I offer to work for $10K less a year if I am allowed to telecommute. I also allow everyone in the company to have a say in the game design, another huge perk most other companies do not offer.

My job ad on Gamasutra only got me 8 resumes in the end, only 1 of which I would say is a professional level candidate (where I was expecting hundreds of professional level candidates) and even then I don’t know if he was one of those top 5% great guys I was looking for.

I’m starting to understand the recruiter market. Because the pool of great candidates is so small, recruiters are there to pull people away from their existing companies and not so much to search for people who are already available. They charge a ridiculous amount for that small service (15%-50% of the first year’s salary) and I’d never use one for that reason but I can at least see the theoretical point now.

In 6 months of looking, I only ever found one good guy and just hired another, who will hopefully turn out to be good – we’ll have to see.

If I could do things over again, I’d start the hiring process about 1 month before I even started the company. Get some names down, some people to sign up for you for a start date two weeks later. And I’d continue hiring and looking, because some of those people won’t work out anyway.

A bad hire can ruin the company

Before I started the company I read a lot of management books and spent a lot of time trying to motivate the people I did hire. So when people didn’t perform, I tried to be a better manager, and blamed myself for their failings, so would respond with motivation and help, and kept giving more time, and more chances. As it turned out, the problem wasn’t bad management but a lack of raw talent – the bad management part was in hiring the wrong people to begin with. I confused inexperience with incompetence. The first is easily fixed, and in a way is positive because you can get someone cheaper and train them. The second is impossible to fix and will only drag your company down. If someone says “I don’t know how to …” this is OK – spend a day training them and you have a more valuable worker. If someone says “I can’t …” and they actually can, this is incompetence and you might have made a hiring mistake. For new hires, this is very important to watch for.

Hiring the wrong people is a triple whammy. You waste your time training, they waste your money learning, and you end up even later and more behind schedule.

This may sound harsh, but it’s true. There is a 100X difference between the best and the worst programmers and the curve is towards the bad end. So there’s a lot more bad programmers looking for jobs than good ones, and it makes it very important to not keep the bad ones.

You won’t know who is a bad hire until after you hire them

Maybe I’m not good enough at interviewing, but some people are good at interviewing but bad at programming. It takes me between a week and a month to sort out the difference. So when you hire, you should overhire, because some of the people you hire aren’t going to work out anyway.

Quality is more important than cost

With 100X the difference between the best and worst programmers, it’s worth paying double for someone 10X better. Of course it’s always a risk – say you hire a guy at double the cost, he doesn’t work out after a month, and now you are out 2 months of salary. But generally speaking, if you find a great guy just take him right away, despite the extra cost. This will save you money in the long run. Plus, the good programmers know they are good and tend to ask for more money. A very low asking price is a warning sign.

One of the mistakes I made early on was to pass up a very experienced guy in the US because he was slightly too expensive. As it turned out, I ended up paying far more in the long run because of this, and had to hire people nearly as expensive anyway.

Get the money straightened away first

Once I interviewed with a company that spent probably $1000 on the interview. They flew me in, put me up at a nice hotel, spent a lot of time on interviews, had a producer drive me around showing me the local area, and spent time doing paperwork. Their final offer was 20% below my best offer, and was also below the minimum I put on my resume. The offer was fair for the area, but was too low for me to accept given the number of other companies that also had offers for me.

To a lesser degree I had the same problem, spending a lot of time interviewing guys that wanted double what I could afford to pay. I may have liked them in the end, but it was a waste of time nonetheless.

Setup interviews in stages, and don’t be afraid to cut an interview short

Similar to above, a lot of guys just can’t cut it and it’s not worth the time to do a 2 hour interview with a weak candidate. It’s not rude, it’s business, if you have to just cut an interview off after a short time.

Go with your gut instincts

Lastly, if someone gives you a bad feeling but you can’t say exactly why, no hire. Every time I’ve hired someone because I had no other reason not to, I regretted it a few weeks later.

Energy Bar

Thursday, March 15th, 2007

I have temp art for the energy bar in. It uses an animated texture to flip lights on and off of the base of the bar. The center part interpolates between the ’empty’ texture or the ‘full’ texture, picking along the UV channel based on the percentile of my total remaining energy.

Photoshop:
Energy Bar Photoshop

In the game
Energy Bar In-Game

Game programmer hiring

Tuesday, March 13th, 2007

I posted an job ad on Gamasutra about 4 days ago. It’s $350 for a single posting. I’ve only gotten 3 responses so far and my posting is the most appealing one there – work from home, high pay, small company. That’s pretty disappointing, I’m inclined to ask for a refund.

Of the 3 that applied, I’m now sending a pre-interview test. It’s based on how smart you are, rather than how much you can research stuff, and is a good indicator of programming ability.

1. Suppose you want to find the median of a list of a billion numbers. You have a cluster of networked computers at your disposal. You need to find the median as fast as possible. Describe a technique to do so.

2. I have a large set of data where each datum is a key/data pair. I need the operations insertion, deletion, lookup, and linear access and all these operations must be as fast as possible. How should I represent this data?

3. You are tasked with writing a logger class in a 3D game that does not already have a logger. It will be used for a wide range of features, writing everything from networked chat logs to graphics errors events to database lookup errors. You only have one day to write it. How would you architect this class? What features would you support?

4. You are planning to write a client/server game with an unattended server. Because the server is unattended, any crashes will mean the players cannot play for days. You have one week to implement whatever features / security you want to deal with this problem. What would you do?

In the past, I’d hire based on how badly I needed somebody, and how much I wanted to save money. So if the position really needed to get filled, and cheaply, I’d end up hiring a bad programmer for little money. This actually made me worse off – the bad programmer would waste my time (costing money) and accomplish nothing (costing money) and the job wouldnt’ get done (so now I needed somebody even more).

Now I’m hiring solely based on “Is this someone I would like to work with, and who would provide value for the money?” I’d rather pay 4X more for someone 10X more productive, and as we know there is a 100X difference between the best and worst programmers, so this isn’t hard to make happen.

Indy Game programming: India outsourcing postmortem

Friday, March 9th, 2007

Ever since I started in the games industry, I, and most games professionals I know, have a pet game design, and a dream to start our own companies. Most of us never have a chance to do that, but with a promise to my wife to ‘only’ spend 3 months and 20K (so about 50K counting my own lost salary), I started down the path of potential riches. That was 5 months ago.

The initial schedule called for 4 months of game programming work for myself. In order to cut that down to the required 2 months, I decided to hire 3 juniors, plus one senior graphics programmer so my game would have all the cool new graphics effects. Of course I can’t afford that for 20K, and I heard that India was the best outsourcing location and that the programmers there were cheap and hardworking. So I decided to give outsourcing a try.

To get my programmers, I posted on a general job site, naukri.com. I thought I would get 20-30 resumes, interview maybe half of them, and pick the best of the bunch. What actually happened was I got got flooded with about 800 resumes, about 200 of which came in the first 24 hours. Because of the way their site is designed, anyone can apply to any job, regardless of qualification, and without reading the job description, and with a single click. You can imagine the resulting average quality applicant I got. I also had no idea what Indian programmers were worth at the time. I got prices ranging from $250 a month to $5000 a month, all of whom claimed a college education, and most of whom could not even program “Hello World” in a real-life application.

I ultimately hired 4 guys at, on average, $1500 a month. One disappeared on the day he was supposed to start work. One was fired after wasting 3 days just trying to compile the game and had to get help even opening and copying files. Another supposedly worked for two days, then disappeared, and told me he took another job else a week later. The last was the best qualified of the bunch: my former graphics programmer – a diamond in the rough comparatively speaking. He claimed to be a former lead programmer at nVidia, claimed 6 years of experience, could speak well, and could answer most of my general interview questions.

Yet at the same time, his graphics knowledge wasn’t what I was expecting. I’m a networking guy, not a graphics guy, so was prepared to interview a graphics programmer who knew so much more than I did it was like Chuck Norris beating up a schoolyard bully. What actually happened was closer to an untrained sumo wrestling match in a mud ring. I couldn’t really get a grip on what he knew, I knew some things he didn’t, and overall we were pretty evenly matched as graphics programmers. But I attributed this lack due to in part the language barrier, in part because he was awesome compared to the 849 buffoons before him, and lastly because I was so desperate to save money. My own inexperience in hiring didn’t help either.

I thought hiring an ‘expert’ with 6 claimed years of experience would enable me to be very hands-off about the graphics, and focus just on the gameplay and networking. Don’t meddle in affairs you know nothing about, right? Unfortunately, this approach messed things up even more. He put a lot of hours in – easily 7 days a week, over 10 hours a day. He got work done, from the standpoint of writing code and checking it in. Yet the game never made much graphical progress. The problem he solved were the wrong problems, solved the wrong way, and even that inefficiently and poorly written. Many tasks ended up taking 4X longer than they should have – once to explain every minor nuance, once for him to program it, once for him to redo it after I point out obvious errors by glancing at the code for 30 seconds, and the final time for me to give up in exasperation and do it myself.

As time progressed, and this kept happening, I had to learn more and more about graphics programming in Ogre 3D just to keep the game from degrading. Eventually it reached the critical threshold where it was easier and faster to do everything myself. I forced him to resign, and only regret not doing so sooner.

What conclusions can one draw from this? I have a good sample set of applicants and interviewees – the average applicant on a general job site is worthless. To be fair, this is probably also true on general job sites in the US as well. There is a huge variation among asking salaries – a twenty-fold difference between the cheapest and most expensive. I’ve found that you get what you pay for – ignoring the outrageous endpoints, the more expensive guys are clearly better.

I hired 4 guys – not a statistically meaningful sample set, but for what it’s worth, all 4 of the guys that did work worked hard and put a lot of time in. The comments about Indian programmers being cheap and hardworking is actually true. It’s only their competence that is lacking.

If I have to pick one common problem among all the applicants and workers I’ve had from India it’s literalism. If you say “Write a ship explosion effect” they will – but it will look stupid, be inefficient, and ignore the game design you just told them about two days before. And they honestly won’t know what’s wrong. So if you be more specific, and say “Write a ship explosion effect and that looks like this image, and do so efficiently” they will bombard you with a dozen stupid questions. It’s like programming a computer – you have to give ALL the information. Whatever information you don’t give will produce arbitrary and meaningless results, such as only supporting 1 lights in shaders (I didn’t say we needed more than one, I just said we needed light support), or doing 3D effects in 2D (Our game is in 3D, but I didn’t say the effect had to be in 3D), or making a cloaking effect where the enemy player’s 3D name still shows up (I didn’t say the name had to be invisible, only the ship).

Not all outsourcing is bad – I have some artists and a programmer in Europe that are doing an excellent job. But outsourcing to India has been a definite bust for me.

Fortunately, the project is not at risk, since graphics is just marketing, 90% of graphic quality is your artists anyway, and I have fantastic artists. But you definitely get what you pay for. Next time I’ll just hire someone in the US, at full salary.

Outsourcing again: Webpage design

Wednesday, March 7th, 2007

I’ve been looking for a company to do webpage design for me now. I’m looking for something on-par with http://www.guildwars.com or http://www.worldofwarcraft.com. Pretty high standards, but this is my main marketing venue other than graphics.

One thing I’ve found is you can get a pretty good idea of the quality of a company in the first 5 minutes. How quickly do they respond to an inquiry? Do they ask intelligent questions? Are they going to detail their work, or do the bare minimum to meet the requirements?

I don’t have that big of a sample set yet, but from what I’ve seen US companies are a lot more likely to ignore inquiries than foreign countries. I think the foreign countries are hungrier for your dollars (which is good).

Maybe it’s just bad luck with Brazil, but for 2 / 2 times I’ve talked to companies there now, they’ve tried to take advantage of me. First was the art team that wanted $60K to do the art to me, the equivalent of $240K in US purchasing power. Just nuts. Then I spoke to a web design firm, who while professional, wanted $11K for the website at $25 an hour. In terms of purchasing power, again that’s like asking for $100 an hour, which is unreasonable. Why would I bother outsourcing to be charged US rates? Also the hours don’t make sense – they put down 320 hours for the coding alone. How the heck does writing HTML take 320 hours? The art, I can see it, because time = quality. HTML is trivial though.

I’ve gotten pretty good responses from India, the Ukraine, and especially the US. When my inquiries are not ignored, I seem to get the most intelligent and professional responses from the US, and the best service. The prices are higher, but are are still surprisingly often competitive with other countries. I think this is because the other countries tend to raise their rates up to US standards, rather than charging what is locally competitive. But with some extra work I’ve found I can still find the good deals and get good quality work done (as per my current art team).

Debris from reactors

Wednesday, March 7th, 2007

I have debris from shooting reactors now. When the reactor mesh loads, I analyze the mesh and generate a face list, which I store in a cache map with the filename of the mesh as the key. When you shoot the reactor, I take the bullet position and find the nearest face. I then find the nearest point on that face to the bullet. Lastly, I randomly rotate the face normal within a certain tolerance and use it the vector to shoot out the debris from. I had to account for some transformation problems, such as the face list is in local space, while the bullet is in world space, but I got it working in the end. What this means is that when you shoot a reactor, a piece of debris comes out from the surface of the reactor where you shot it, out away from the surface. Most games just fake it, by spawning debris from the bullet position, but I have damage from a proximity radius, so the bullet doesn’t always directly hit the reactor. This way, the debris still comes out of the reactor.

This technique can be extended more usefully for walls which explode outwards.

I’m not sure if anyone will actually appreciate this, given how far out the game is zoomed, but it’s good practice for me. Just 6 months ago I would have really struggled with this. But with my 3D programmer training at The Collective I not only did it, but in passing reinvented the function to get the nearest point on a plane just from thinking about it for a few minutes.

Next up is when reactors explode, the lights go out and you switch spawn points. This should be no work at all, but it seems like every time I start some minor thing I get sidetracked with a dozen little bugs and changes.