Last week we looked at how much the services industry has grown. We also talked about all the different criteria we look at when evaluating studio talent. When you put this info together, you have a solid foundation for partner selection. But it leaves a pretty big question sitting on the table: how do we actually do our evaluations?

Below is a high-level look at a typical CDR procurement process for a known need–all the things we do to help you hire the best partner for your project. The steps and details will vary depending on the project, but the process below will give you a good overview of how we approach and think through partner evaluations.

I view this entire process as an information gathering exercise. What information do you need to make a great hiring decision, and how can we design a process to most effectively capture that information. The goal is to fairly and efficiently get the information you need to make the best choice for your company.

Step 1: Understand Your Need

This is the first thing I got wrong when doing ExDev–I didn't take the time to consider which capabilities our partners required to be successful with the projects we were giving them. As a result, I hired a bunch of studios that were really talented…just not talented with the things we actually needed them to be good at.

I know it sounds silly, but it's the truth. And it’s not just me, I see it happen all the time. For example, I’ll bet you've seen someone say something to the effect of: "does anyone know any good animation studios?"

The range of skillsets that could answer "yes" to this question is enormous. Are they looking for 2D animation or 3D animation? Are they looking for help with making their combat animations more readable in Unreal, or narrative-driven cinematics in Maya?

And it's not just skillsets I'm talking about. I've heard of studios being hired to do a port, only to realize when it came time to kickoff that the developers didn't have access to the necessary development kits!

To be fair, even if I had done this kind of analysis back then, I didn't have any understanding of the market or know any service studios to call in the first place, so it wouldn't really have mattered. The options my friends referred to me was all I had to go on, so I took them.

And as a result, we spent a lot of time and money learning the hard way–a.k.a. having to fire hard-working studios we put in impossible losing positions–which skillsets were actually important for our project.

Step 2: Make a Plan

Start with what success looks like and work backwards. Define timeline, budget, and any project constraints: list all assumptions, list any known unknowns, etc.

I usually like to create the vendor documentation at this point (usually an RFP) so the client knows how we're positioning the project to external parties. This also forces you to collect and evaluate all available project documentation. Once it's together, you can review it and ask:

• Is this project feasible? If not, we may need to have some hard conversations about budget, timeline, etc.
• Is the scope clear? If not, you either need more documentation or a different approach.
• What type of partnership model makes the most sense in context of the overall plan?
• What type of evaluation process do you want to run to test candidates on your top priority capability requirements?
• Who are all the internal stakeholders for the hiring decision, and are they appropriately involved?

How much time you invest in evaluation should be proportional to risk inherent in the project itself. And by risk, I'm not talking about the budget. I'm talking about the downsides if things go wrong.

You could have a massive budget for a partnership, but if it’s long-term and the contract is easy to terminate, the consequences of a bad hire are relatively low. At worst, you might lose a month of work.

However if a partnership is highly leveraged - where failure is systemic, such as with a critical piece of middleware - you should invest significant time in evaluating the partner. Ensuring they can deliver reliably with high confidence is crucial, even if the budget itself isn’t large.

Step 3: Identify Top Providers Based on Your Requirements and Solicit Availability

Once we have a plan in place and are aligned on the overall constraints of the project–budget, timezones, headcount required, top capability requirements–we can start to think about which studios would be best positioned to execute on the project.

To start, we usually ask different members of the CDR team to create their own potential candidate lists. This is done via:

• Who we personally know and have worked with in the market/our personal networks.
• Querying and filtering the CDR Database (we've done a lot of work and iteration on our search functionality, I'll write more on this later.)

Our initial shortlist is usually pretty large, 40 or 50 companies, which we then whittle down as we take a closer look at their portfolios and all the notes CDR has on past performance/referrals.

Next, we'll start to reach out to candidates on the shortlist with a very high level brief of the project, such as:

"We're looking for a co-development partner for a California based studio, initially scope is for X game engine developers and artists, likely to later grow to X if successful, will start with a 3 month contract to hire. Are you available and interested?"

If they are, we will get them under NDA and schedule a call to ask more in-depth details on their relevant past experience, as well as share more information about the project and procurement process we're about to run.

The goal at this point is to make sure the potential team is a good fit from an experiential and values point of view, then we get confirmation that they'd like to participate in the process.

Most of the time we target between 4-8 studio bids, depending on the size and complexity of the project. I like to have a mix of studios I've worked with in the past with some new promising candidates–whether newly founded studios or companies we haven't yet worked directly with in the past.

Step 4: Proposal Stage

After carefully evaluating past experience at the shortlist stage, the proposal stage is where things get exciting–you finally get to meet the actual team members who would work on your project and see the all-important pricing details.

Everything we've done so far builds to this moment: creating healthy competition by requesting proposals from multiple qualified candidates simultaneously. To make this work smoothly, preparation has to be rock-solid with candidates under NDA, requirements well-documented, and procurement timeline set.

Why a structured process?

• It creates a level playing field where everyone gets the same information and timeframe to prepare their best offer.
• When studios compete for your business, it naturally leads to higher quality proposals and fair pricing.
• Reviewing all proposals at once makes comparison easier and speeds up your decision-making process.

During this stage, candidates will ask a lot of questions (a good sign they're taking your project seriously). I've found it works best to schedule two specific Q&A periods during the process. This lets us batch and deduplicate questions for your internal teams and share answers with all participants, ensuring everyone has access to identical information.

Proposals are also a good opportunity to request some initial creative work from the team to evaluate their approach, like art concepts, a draft production schedule, or simply some thoughts on tackling a specific technical challenge. Just remember that what you can reasonably request depends on your project's budget and scope. The larger the potential contract, the more pre-work studios will be willing to provide.

Step 5: Interview / Assessment Stage

Once we receive proposals, it's time to level the bids. This means converting all objective measures to the same base units (like daily rates vs monthly rates, USD vs Euros), and putting them in a leveling sheet to compare side by side. The final result will look something like this:

Once we're able to evaluate apples to apples, there are usually a few outlier bids that can be easily eliminated and a handful that seem like they'd be a great fit. Then, it's on to the next step: interviews and tests.

(Side note: In cases where we don't have direct working knowledge with a candidate, CDR also performs reference checks with prior clientele, either by asking the studio for introductions or via backchannel feedback if we already have connections with some of the brands listed on their past client slides.)

When interviewing teams, I like to prepare questions in advance, share the topics with the studios participating in the interview process so they can schedule the appropriate personnel assigned to the project on the call, and then ask more or less the same questions to each candidate so you can more easily compare and contrast responses.

When performing tests such as art tests, a few things are important:

• Keep the scope tight: what is the minimum unit of work necessary for a studio to demonstrate a skillset.
• Schedule in revisions. Half of the value of the test, beyond testing capabilities, is to get an understanding of what the working relationship with this studio will be like: How do they respond to feedback? What type of decisions do they make when requested to "fill in the blank" creatively?

If you're asking a studio to do work for an evaluation, this should be paid. You should also make sure to allocate the time necessary for your internal team to appropriately manage and give feedback to each studio participating.

Step 6: Final Decision

Once evaluations are complete, I ask each decision-maker to write down a brief synoposis on who they believe is the best fit for the project. This allows us to review each other’s takeaways without being influenced by other answers.

I'll also usually try and score each candidate using previously ranked capabilities, but candidly by this point if you've done a good job in your evaluation process the decision is usually pretty clear.

Keep in mind, each project is different so each evaluation process is different. Oftentimes studios don’t know exactly what type of help they need, which means part of the process is clarifying the need to begin with. A well-structured approach not only helps you find the best fit but also ensures you’re solving the right challenge in the first place.

The next step is to make sure you're set up to effectively onboard and manage your new partner…a topic for another newsletter!

‍