Request for a Diagnostic Instrument

ERI conducted an extensive survey of nursing programs throughout the United States to determine whether there was interest in the professional, nursing community for a critical thinking test. The results clearly revealed that the academic community desired a diagnostic instrument which would evaluate critical thinking within a nursing environment, yet stress intellectual processing skills over nursing content mastery. These nursing programs clearly called for a critical thinking test, not just another exit test that measured or depended on nursing content mastery for student success with test items.

The Examination Committee of Educational Resources (ECER) appointed a Task Force (TF) to explore the development of a Critical Thinking Process Test (CTPT). This standing subcommittee is composed of nursing faculty who are actively involved in both teaching and clinical practice. The ECER members in their professional preparation and experience represent the five regions of the country: West, South, Southwest, Midwest and Northeast. These experts had a threefold task:

w Help the test bureau decide the purpose, scope, processes, constructs, and what the outcomes should measure in the CTPT;

w Develop and review the test blueprint for the CTPT; and then

w Help the test bureau determine the weighting that should be given to the blueprint outcomes.

Survey of Literature

This task force performed an extensive review of the published, professional literature on the topic of critical thinking and critical processing behaviors thought to be necessary for successful nursing practice. Topics, such as the following, were explored:

Establish Test Plan Draft

A general test plan evolved that reflected this literature survey.

Validity of Test Plan

Referencing both the literature review and the topics outlined above, the Special committee then reviewed the overall blueprint draft and the suggested the proportion of questions that should be assigned to each critical thinking skill or critical processing topic. Modifications were made in the test plan by the task force and returned, repeatedly, to members of ECER until a final test blueprint was accepted which is believed to reflect both the literature and the needs of nursing students preparing to enter practice.

Assignment of Test Items

Item writers were then selected by the Task Force who were considered knowledgeable and experts within the academic community of nursing. The selection of item writers was based upon their credentials, the type of nursing program they represented, and their expertise within the area of the critical thinking and critical processing skills.

Writing of Test Items

Questions measuring appropriate critical thinking skills within a nursing environment were written, and then each question was evaluated for consideration in the blueprint of the CTPT. Each question was checked to make certain that both current nursing textbooks and journals, and practice, supported the correct answers. After this screening of questions, they were sent for review and criticism to faculty in representative schools of nursing throughout the United States. These faculty members reviewed the questions, selected correct answers, noted whether the desired critical thinking or processing skill was properly assessed, indicated any need for revision, and recommended to the Task Force which questions should be either kept or omitted. The results of this screening were then checked and recommendations acted upon by the Task Force.

After the literature review, blueprint draft reviews, and item writing, 75 questions were completed that satisfied the stringent criteria for item writing endorsed by the National Council of State Boards of Nursing and other recognized professional organizations. From these 75 questions would emerge 50 test items that actually appear in the CTPT.

Index of Item Discrimination

An Item Discrimination test was utilized to provide a true reflection of each item's contribution to the functioning of the CTPT. This statistical measurement determined how well actual item performance correlated with the criteria. The point-biserial correlation method was used to investigate the test items for correlation of high and low functioning students. A study was completed on 788 students involved in the norming of the CTPT, representative of the different nursing programs in each region of the United States. Each item was calibrated according to how the upper 20% and the lower 20% of the testing sample who answered each test item. If the test item displayed a correlation favoring the weaker student (below .15), the test item was either discarded or the distracters were revised by the Task Force. The current CTPT contains only questions with a positive point-biserial correlation which indicated the question was a strong predicator for both high scoring and low scoring nursing students.

Trial Questions

These questions were next administered to a randomly selected sample of 155 generic nursing students who were completing the final semester of their formal nursing education prior to writing NCLEX. These try-out questions were used to collect information about the test items. This procedure assured, again, that the test questions were clearly written and relevant to both the nursing environment and critical thinking criteria that was desired.

Data from the trial questions identified sources of ambiguity by indicating questions for which many nursing students selected the same wrong answer. Even though questions were thoroughly reviewed before they were included as trial questions, occasionally a question was interpreted in a way that was not intended when it was written. Such a test item was then corrected or discarded.

Question Clarity

The questions on the CTPT were not designed to be confusing or tricky. Instead, considerable effort was devoted to make the questions as clear as possible; the main purpose of the try-out questions and of the extensive editing of questions was to remove possible sources of confusion in wording. Statistical analysis of the trial test questions were reviewed by the members of the Task Force. If, with a given question, there was conflict between an item's statistical quality and the critical processing level the Committee wished to sample, statistical quality took precedence.

In the end, the crucial consideration was this: Does this question distinguish between the varying levels of cognitive abilities of a nursing student who demonstrates mature intellectual functioning and one who does not? Questions which did not make this distinction were not used. This system of checks and balances was used to ensure that only questions testing essential, critical thinking skills were used. Every effort was made to ensure that the questions tested important critical thinking processes, rather than nursing content. There are no trick questions; there are no questions with double meanings; there are no unanswerable questions.

Currency of Test Content

The Task Force, with the assistance of the staff of Educational Resources, Inc. Testing Bureau, continue to evaluate the questions used in this assessment instrument. Trial questions, at times, may make-up five percent of the questions in the CTPT, but they are not computed as part of a student's or nursing program's diagnostic scores. The questions that do count are the same questions in all test booklets sent to a specific nursing program. This method of continually evaluating questions builds a test pool for future revisions and forms of the CTPT. The scores for all students and programs are continually assimilated with the established norms for the tests and reflect current norms.

In the development of the CTPT a series of standardization procedures were followed, both to assure that the content was appropriate and to provide an efficient means of interpreting test performance. The following describes these procedures and provides evidence of the reliability and validity of the CTPT.

Standardization

The original Alpha testing of the CTPT was based on the testing of 788 nursing students who were from all five geographical regions of the U.S.

The final standardization population consisted of 943 students who were able to complete all questions of the CTPT. Students who did not complete all questions in the CTPT were deleted from the test results.

Reliability

The reliability of the CTPT for students who participated in the standardization was examined by the parallel forms method. The forms were created by treating the odd numbered questions from the CTPT as Form A and the even numbered questions as Form B. These forms were not separated from the original forms of the CTPT, except by the computer which read the items in scoring as Form A and Form B. The content consistency of the two forms was then evaluated. The average reliability coefficient was .93. The coefficient indicated that there was little internal variation in a student's performance from one form (half) of a CTPT to the next.

Validity

Validity is an integrated evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores or other modes of assessment. During the development of the CTPT, the validity of each test was examined. The information presented here provides two types of validity evidence: (1) content validity, which concerns the appropriateness of the items on the CTPT for evaluating related critical thinking process behaviors within a nursing environment; (2) diagnostic validity, which concerns the diagnostic instrument's usefulness in identifying related critical skill deficits.

Content Validity

Content validity was evaluated by showing how well questions of the CTPT sampled the critical thinking and critical processing levels from which conclusions were to be drawn about the proficiency of individual nursing students. Content validity was built into the CTPT through its design specifications. In the development of diagnostic items, test items were written that followed the blueprint. Again, the blueprint evolved for each CTPT topic from a survey of literature within nursing academia which explored, defined and promoted the development of critical thinking and processing skills for nursing students. Test items, therefore, were included by ERI's Task Force only if the questions emphasized critical thinking and critical processing skills identified to be essential for nurses preparing to enter nursing practice.

Diagnostic Validity

It was also of interest to assess the diagnostic validity of the CTPT. The question asked here was whether the CTPT could identify students who were expected to have deficiencies in those skill areas tapped by the CTPT.

To test this hypothesis, the CTPT was administered to 459 randomly selected students who had just completed their course in Nursing Fundamentals. Students must have completed Nursing Fundamentals or they would not have the basic vocabulary necessary to work with the questions of the CTPT which are written within a nursing environment.

It was expected that lower scores would be achieved by students with critical thinking and critical processing skills when these students were tested near the beginning of course work in nursing content, and that the same students would have increased success in answering these same questions as they completed their academic and clinical program in nursing education. Generic baccalaureate students were evaluated at the beginning of their junior year and at the end of their senior year.

A t-test was then completed that compared student performance on the appropriate CTPT given as a pretest and the same CTPT given as a posttest. The t-values and their levels of significance show that for each CTPT the successful performance of beginning nursing students was significantly below the average of the norms established for students after they completed their nursing programs.

Conclusion

The technical data lends support to the validity of the CTPT. There is always an interest by the Examination Committee for studies involving the CTPT in correlation with assessment instruments offered by other test bureaus. Independent studies show that the CTPT is a reliable and valid assessment instrument for evaluating mastery by students of critical thinking skills. However, the validity and reliability of any instrument must be continually re-evaluated. For this reason, users of the Critical Thinking Process Test are encouraged to submit studies that pertain to the validity of this assessment tool.

Test Development Panel

Norma Anderson RN, BS, BSN, MSN, EdD
Rockford, IL
St. Joseph College of Nursing

Linda Booth RN, BSN, MS
Shawnee Mission, KS
Director of Test Development
and Curriculum Integration

Joe Catalano RN, MSN, PhD
Ada, OK
East Central University
Professor of Nursing

Lurelean Gaines RN, BSN, MSN
Los Angeles, CA
East Los Angeles College
Chairperson, College of Nursing

Margaret Horner RN, BSN, CNOR, MSN, CS-MS
UMKC, BSN & MSN Completion Programs
Kansas City, MO

Helena McBride RN, MS, PhD
San Antonio TX
Primary CareNet of Texas

Sarajane McCormick RN, BSN, MSN, PhD
Little Rock, AR
Baptist Medical Center
School of Nursing
Asst.Director, Student Development/Special Projects

Virginia (Beth) Richardson RN, MSN, DNS, CPNP
Indianapolis, IN
Indiana University, College of Nursing
Assistant Dean, Student Affairs

Marva Roddy, RN, BSN, MA
Imperial Beach, CA
East Central University
Health Facility Evaluator/Professor, College of Nursing

Larry Simmons, RN, BSN, MSN, PhD
UMKC BSN & MSN Completion Programs &
Olathe Medical Ctr, Director,
Home Health/ Hospice
Shawnee Mission, KS

Ann Straughn, RN, BSN, MSN, RNP
Little Rock, AR
University of Arkansas
Professor, College of Nursing

Sandra Underwood RN, PhD, FAAN
Gurnee, IL
University of Wisconsin-Milwaukee
Professor, College of Nursing

Jacki Witt, RN, JD, BSN, MSN, WHNP
UMKC's MSN Program
Kansas City, MO

previous · content