Research FAQs: A Guide to Common Questions

The fixed goal of the Johnson O’Connor Research Foundation is pure scientific measurement…the basis, we hope, for a happier mankind.

Johnson O’Connor

Research is integral to the work that we do at the Johnson O’Connor Research Foundation. We’re not just an aptitude testing service; we’re also a nonprofit research institute with a full-time, in-house research department. Our researchers continuously evaluate our tests and data so that we can provide clients with the most accurate assessment of their aptitudes and how to use them. We’re proud of our 100+ years of research and encourage anyone who wants to learn more to visit the Research section of our site.

In this article we’ve worked with one of our researchers, Dr. Zara Wright, to compile answers to the most common client questions. You’ll find a downloadable document elaborating on our research at the bottom of this page.

“How Do You Study Careers?”

When clients ask “Where does your research come from?” or “How do you study careers?,” they’re usually wanting to know how we conduct Career Validation Studies.

Our occupational validation studies demonstrate evidence for the significant relationships between specific aptitudes and the types of careers we would expect to be related to those aptitudes. We conduct these studies by analyzing:

• data from our existing client pool (see an example of types of occupation by aptitude [2019] here), and
• data from occupational groups (see an example of a study on engineering students [2012] here).

In both cases, this involves careful selection of participants, ensuring they have sufficient experience in their field, and focusing on those who report job satisfaction.

Data from Client Pool

Studies using our main population of clients allow us to demonstrate the way aptitude testing can be useful across a variety of occupational fields. For example, when comparing the aptitudes of individuals satisfied with their careers to those who were dissatisfied within the same profession, we see a modest relationship suggesting that people with more alignment between their aptitudes and the demands of their professions tend to report higher job satisfaction.

Data from Occupational Groups

Studies examining aptitude scores within a certain profession help us map aptitude patterns across different occupational and educational groups. When validating the tests for a particular profession, we typically include data from individuals who have worked in their field for at least one year and report being either satisfied or neutral about their job. To make reliable generalizations, we aim to include at least 100 participants from the major occupational group, although we also supplement this with case studies and smaller groups.

“How Accurate Are the JOCRF Tests?”

For all tests that the Johnson O’Connor Research Foundation uses to assess aptitudes, our researchers conduct studies to ensure standards of validity (how accurately we are measuring what we set out to measure) and reliability (how precisely and consistently we are capturing whatever we are measuring).

Certifying the validity and reliability of our aptitude tests is a continuous process that begins when a test is initially created and continues periodically as tests are revised.

Validity

We use several standards of validity including:

• construct validity – evidence that our tests are accurately measuring the aptitude that they are supposed to measure, and
• criterion-related validity – evidence that our tests can accurately predict expected real-world outcomes, such as vocational field.

The method(s) that is used to establish validity varies somewhat from test to test based on differences in the tests’ content and construction. You can see an example of a typical study to establish test validity here.

Reliability

Our researchers focus on the two most common measures of reliability:

• internal consistency – the degree to which the individual items on a test are related to all other items on the test, and
• split-half reliability – the degree to which two halves of a test are related to one another.

We aim for a minimum standard of reliability of .80 which is above the generally accepted .70 threshold. The JOCRF test battery has reliabilities ranging from .72-.97, with only three tests under the .80 JOCRF standard.

“Will My Aptitude Test Scores Change Over Time?”

One of the unique elements of our assessments is that the scores from our aptitude tests are relatively stable over time. That means an individual only needs to go through our service once. The information will remain valid throughout the course of their lifetime and can always be used to make decisions. This is in contrast to the results of other assessments, like interest surveys, that can change at different life stages.

Stability

The Johnson O’Connor Research Foundation conducts research to assess how well our test scores remain stable over time, relative to an individual’s age.

Since performance on our tests may vary across different age cohorts, an individual’s raw score at age 20 may differ from their score on the same test at age 50. However, aptitudes generally have rank-order stability, meaning test scores remain stable when an individual’s score is compared to others in their age group, as seen here in our most recent study on stability.

To evaluate test stability, we requested that our follow-up clients retake one of the tests, allowing us to compare performance over time. This process enabled us to establish both short-term and long-term reliability coefficients for 18 of our tests.

“Who Is in Your Testing Population?”

We test, on average, about 5,000 people per year in our offices across the United States. We have been administering most of our tests for at least 50 years, meaning some of our tests have been administered to hundreds of thousands of people.

Our scoring is norm-based, meaning that clients are awarded percentile scores based on how their performance on a test compares with that of others of about the same age. Our norming is all based on the group of clients who come to us for testing. The norm groups vary by size and make-up from test to test, but most comparison groups are at least several hundred people, with many comparison groups into the thousands or even tens of thousands. We examine score distributions regularly and update norms as needed to account for population changes.

Below you’ll see measurements of what our most recent norming population typically looks like, based on the 15,000+ clients who have been in for testing between July 2020 and July 2024. Key takeaways from the examination of our most testing population shows that:

• over half of our clients are between 14 and 22 years of age,
• 54.9% of our testing population are biological females, and
• our testing population tends to be more educated than the national average.

“What’s the Success Rate of Aptitude Testing?”

We’re frequently asked about the “success rate” of aptitude testing. While it is challenging to formally track the long-term impact of our aptitude testing services, we are making strides in gathering more structured data through follow-up surveys.

We recently launched a new program to collect 1-year, 5-year, and 10-year follow-up data for all JOCRF clients in an effort to collect more targeted insights into career-based outcomes. With this added context, we will gain a deeper understanding of how each individual aptitudes predicts vocational field, career success, and career satisfaction, thus improving our support for criterion-related validity of each of our tests.

What JOCRF Clients Say about Aptitude Testing

In 2022, we conducted a survey of alumni and received over 800 responses. When asked whether aptitude testing significantly influenced a major career or life decision, over 84% of respondents indicated “somewhat” or “yes.” This positive response suggests that our aptitude testing provides valuable guidance, although we recognize that the benefits are not always about changing career paths.

Aptitudes and Career Satisfaction

We have conducted client follow-up surveys to explore the relationship between aptitude alignment and job satisfaction.

A series of these surveys was distributed in the 2010s to better quantify how well clients felt their aptitudes were being utilized in their work and how this correlated with job satisfaction. These surveys found that clients who frequently use their aptitudes at work report higher job satisfaction and are more likely to agree with positive statements about their career. In contrast, those who rarely or never use their aptitudes experience lower satisfaction, highlighting the impact of aptitude alignment on career fulfillment.

Aptitude Testing and Neurodivergence

We do not systematically collect data on learning differences or neurodivergence, so it is unclear the extent to which testing results, satisfaction with the testing experience, and aptitude profiles may be influenced by diagnoses such as ADHD, dyslexia, dysgraphia, verbal and non-verbal learning disabilities, and other types of neurodivergence. Since our normative data is primarily based on neurotypical clients, comparisons for neurodivergent individuals may not always be as meaningful.

Nevertheless, even though we do not know the frequency of neurodivergent diagnoses in our norming data, we can expect that (given the large size of our sample and the rate of incidence of neurodivergence in the population) neurodivergence is represented in our population.

Thus, we believe that some individuals with learning disabilities would still be able to gain valuable insights from testing results. Please visit our FAQs page for more information about the aptitude testing experience and autism and/or learning differences.

How Can I Find Out More?

Our Research department offers free PDFs of research publications, when available.

Please direct requests for copies, or any other inquiries, to research@jocrf.org.

You can also read or download our 2025 Research FAQs report, which expands upon some of the topics from this article.