Introduction

The benefits of participation in paid work for the individual with a spinal cord injury (SCI) are multifaceted and go beyond monetary compensation [1]. Society also benefits from high return to work rates, for example by avoiding expenses for unemployment benefits [2]. It is known however that persons with SCI encounter difficulties in returning to paid work after the onset of their condition. Hence, return to work is an important goal in SCI rehabilitation medicine and vocational rehabilitation (VR) gains attention worldwide.

The Work Rehabilitation Questionnaire (WORQ) was developed to assess work-related functioning and consists of a selection of categories from the International Classification of Functioning, Disability, and Health (ICF) [3,4,5]. The WORQ can be used to describe a person’s levels of functioning on work-related domains at the start of VR; to set VR goals and monitor functioning over time; to facilitate interdisciplinary and client communication in VR and to stimulate the client to actively participate in his/her VR process [6]. Moreover, the WORQ is free and easy to use. The first version of the WORQ was interview administered. The initial psychometric evaluation of the WORQ showed high test-retest reliability and good internal consistency [3]. To improve the practicality of its use, a self-report version was developed [7]. This version has been translated in many languages [8] and showed good usability and test-retest reliability and promising construct validity on relatively small samples with varying pathologies [9,10,11]. Furthermore, in a validity study in a different population (musculoskeletal disorders), professionals stated that they gained significant information on work-related functioning but suggested that the WORQ would be most valuable when used in complex diagnoses, possibly because the low median sum score of the WORQ in their study [12].

A previous study on the content validity of the WORQ for use in persons with SCI in the early post-acute phase suggested evidence for content validity, but also some gaps and the authors recommended further research on the utility of the WORQ considering concepts identified as lacking in their study (d410 changing basic body position, d445 hand and arm use, d460 moving around in different locations, d640 doing housework, d760 family relationships, e115 products and technology for personal use in daily living, e120 products and technology for personal indoor and outdoor mobility and transportation, e150 design, construction and building products and technology of buildings for public use, e155 design, construction and building products and technology of buildings for private use) [7]. They also suggested to further evaluate its validity in persons with post-acute and chronic SCI [7].

To our knowledge, no study on content validity of the WORQ from the perspective of SCI rehabilitation professionals has been published. The aim of our study was to investigate the content validity of the Dutch self-report version of the WORQ for use in persons with post-acute and chronic SCI, by identifying relevant, irrelevant and lacking categories.

Methods

Design

This study used a mixed-methods design by combining findings from semi-structured interviews with persons with SCI and two group meetings with SCI professionals with a list of SCI specific ICF categories (qualitative study) and with findings from the WORQ filled out by the persons with SCI (quantitative study). The study was performed at a university-based Rehabilitation Center in The Netherlands.

Participants

Persons with SCI and active paid employment (≥1 h per week [13]) prior to sustaining the SCI were included in this study. Potential participants were purposefully selected according to SCI level and American Spinal Injury Association (ASIA) Impairment Scale [14]. Four subgroups were selected: ≥C8 AIS A or B (tetraplegia-motor complete), ≥C8 AIS C or D (tetraplegia-motor incomplete), ≤T1 AIS A or B (paraplegia-motor complete), or ≤T1 AIS C or D (paraplegia-motor incomplete). We aimed to include 4 participants in every subgroup (16 persons in total): 2 still undergoing inpatient rehabilitation; 1 at least 1-year post-SCI who had finished rehabilitation and returned to work and 1 at least 1-year post-SCI who had finished rehabilitation and did not return to work.

Eight professionals with ≥6 years of experience in SCI rehabilitation were selected to represent all relevant disciplines. The following team members participated in the 2 group meetings: vocational counselor, occupational therapist, physiotherapist, social worker, psychologist, rehabilitation nurse, activity therapist, and rehabilitation physician.

Procedures

Potential participants with SCI were selected by the first author, rehabilitation physician on the SCI ward, and invited for the study between May and October 2019. They received written and oral information and provided written informed consent. Those who agreed to participate completed the WORQ questionnaire first and participated in a semi-structured interview afterwards. The first six interviews were performed by both the physician and a master-level medical student to assure a similar method of interviewing. The subsequent interviews were performed by either the student or the physician.

An ICF-list was introduced to the SCI rehabilitation professionals who also received the WORQ beforehand as background information. A group meeting was held in which the professionals were asked to fill out on the ICF-list which ICF categories they considered important or irrelevant for use in VR in persons with SCI without consulting others. Responses were analyzed afterwards by the first author. Categories were considered as important if 75% or more of the professionals identified the item as important. To decide on the categories that were affirmed by more than 40% and less than 75% of the professionals, a second group meeting was held under presence of an additional rehabilitation professional who is a practicing VR counselor and also expert in research (3rd author) to reach consensus on the ICF categories by having a second voting round [15].

Instruments

The Dutch self-report version of the WORQ was developed by adapting the Flemish self-report version of the WORQ [10], in collaboration with the Flemish researchers [11]. The instrument consists of a part 1 with 17 socio-demographic and work situation-related questions and a part 2 with 40 functioning questions based on the ICF categories of the ICF Core Set for VR. Questions in part 2 were posed as ‘Overall in the past week, to what extent did you have problems with …?’. Answers were scored from 0 (no problem) to 10 (complete problem).

The open-ended questions of the patient interview (see supplementary appendix) were based on a previous study and aim to explore the vocational-related experiences of participants by investigating body functions, activity and participation, environmental factors (barriers and facilitators), and personal factors [7].

An ICF-list with a total of 72 ICF categories was composed by merging the categories represented by the items of the self-report WORQ based on the WORQ development study and a previous content validity study [3, 7], the brief ICF core set for SCI for post-acute care [16] and the brief ICF core set for SCI for chronic situation [17] into one list. To keep the level of detail consistent across all categories, only second-level ICF categories were included in this list. Consequently, several items of the WORQ representing more detailed ICF categories within the same second-level category were merged:

  • items Temper, Self-confidence and Irritable, into b126 Personality functions,

  • items Sad/depressed and Worried/anxious into b152 Emotional functions,

  • items Seeing Object and Seeing Person into b210 Seeing functions,

  • items Lifting and carrying objects weighing up to 5 kg and Lifting and carrying objects weighing more than 5 kg into d430 Lifting and carrying objects

  • items Walking a short distance and Walking a long distance into d450 Walking.

Data collection about the persons with SCI included socio-demographics (age, sex), the date of interview, SCI subgroup and time since SCI onset (less or more than 1 year), employment characteristics at the time of interview (performing paid work or not) and their scores on the WORQ part 2. Regarding the professionals, data collection included years of experience in SCI rehabilitation and their rating of the importance of the ICF categories.

Data analysis

The aim of our study was to investigate the content validity of the Dutch self-report version of the WORQ for use in persons with post-acute and chronic SCI, by identifying relevant, irrelevant, and lacking categories.

To identify a category as important we used several criteria: included in the Brief ICF Core Sets for SCI, mentioned in interviews by persons with SCI and consensus on importance by the SCI professional team.

Qualitative data analysis

The interview was audio-recorded and transcribed. Two researchers independently coded text fragments of the first three transcripts using the ICF list, compared the results of their coding, discussed the differences, and reached consensus on how to finally code the text. Further transcripts were first read and coded by one researcher and afterwards checked by the second researcher. For 2 ICF categories on this list, the Dutch translation of these 2 ICF categories is not completely aligned with the English version and therefore the coding was performed according to the English version. The codes mentioned during the interviews that did not match a category of the ICF-list were listed as “not included”. These codes were linked to another ICF category if possible. The number of interviews in which the ICF category occurred was reported. The frequency of the ICF categories per interview was not analyzed.

Furthermore, we determined what ICF categories were quoted as important or irrelevant in ≥ 75% by the SCI professionals.

Quantitative data analysis

Quantitative data were analyzed using SPSS for Windows, version 23.0 (IBM Corp., Armonk, NY). Distributions of scores from the participants on the functioning categories of the WORQ part 2 were expressed as Median (IQR). If a participant provided 2 scores per question, the average of the two scores was used. If item 34 (driving) was marked as not applicable, this was treated as a missing value. If less than 25% of the item scores were missing, the missing values were replaced by the mean of the valid WORQ item scores.

Results

At the time of performing the interviews, only one person with motor-complete SCI in inpatient rehabilitation could be included in this study and therefore this group was extended by recruiting persons with similar but less recent SCI. Fourteen of the intended 16 interviews were conducted. The sociodemographic and injury-related characteristics of the participants are presented in Table 1.

Table 1 Sociodemographic and injury-related characteristics of the participants.
Full size table

Table 2 shows the ICF list and the number of interviews in which the category occurred; the median scores of the ICF categories as filled out on the WORQ part 2 by the persons with SCI; and the categories that are important according to 75% or more of the SCI professionals. One participant did not return the WORQ-questionnaire on time but several months after the interview, the analysis of this questionnaire was considered missing. Some ICF categories (e.g., d850 remunerative employment, e310 immediate family, e330 people in positions of authority) are included in the WORQ part 1 by multiple-choice questions and those concepts do not have a median score.

Table 2 ICF list with frequencies of categories in interviews and findings from group meetings professionals and WORQ.
Full size table

Thirty seven out of 46 categories (80%) of the WORQ were confirmed by both sources: mentioned in interviews with persons with SCI and considered important by the SCI professional team. Those categories are marked in bold in Table 2. Of the other 9 ICF categories, four were mentioned by three or more persons with SCI, but were not identified by consensus as important by the professionals. Moving around (d455) obtained the maximum median score of 10 and was mentioned by 4 participants with SCI, but not considered important by the professionals. Five categories were considered important by the SCI professionals, but not mentioned in the interviews with persons with SCI. Of those categories the median score (if applicable) was rather low (≤3).

Table 3 shows the ICF categories, part of the brief ICF core set for SCI for acute care and/or chronic situation but not of the WORQ, that were mentioned in the interviews with persons with SCI or considered relevant by the SCI professionals. The categories in bold were mentioned by both sources. The ICF categories that were mentioned in the interviews with persons with SCI only (marked with * in the text box) were mentioned in one (s120 and d510) or 6 interviews (e155).

Table 3 ICF categories not part of WORQ and considered relevant by persons with SCI and/or SCI professionals.
Full size table

Table 4 shows the frequency of concepts mentioned in the interview with the persons with SCI but not included in the ICF-list used in this study. Some concepts can be linked to an existing ICF category and some categories remain undefined. Some are part of the WORQ part 1 or are covered in questions 41 and 42 of the WORQ (“Overall in the past week, how long did it take you...”).

Table 4 Not included and undefined concepts.
Full size table

Discussion

The aim of our study was to examine the content validity of the WORQ for use in persons with post-acute and chronic SCI. In our study, 80% of the ICF categories of the WORQ were mentioned by both sources. A previous study with persons with SCI in an early post-acute context showed similar results with 80% of WORQ ICF categories mentioned during interviews with persons with SCI [7]. In this previous study seven categories of the WORQ were not contained in any source [7], whereas in our study all categories were contained in at least one source. If an ICF category is part of the WORQ-NL, mentioned during the interview with a person with SCI and considered important by the SCI professionals, we can consider it undoubtedly as relevant for VR in persons with SCI. Four categories (d455 moving around, d570 looking after one’s health, d870 economic self-sufficiency, and e310 immediate family) were not mentioned by the SCI professional team but were mentioned in at least 3 interviews. The rehabilitation professionals considered categories such as “moving around” irrelevant. Potentially this can be explained as this category includes “running, climbing, jumping, etc.”, activities not directly related to employment, and does not include “walking or moving around using equipment”. The ICF categories mentioned by the SCI professionals only are categories often not considered problematic in SCI (e.g., “seeing functions”). In fact, all of those categories had a low median score (≤3). This does not necessarily mean that those categories are redundant for use in VR in persons with SCI. Professionals deemed these categories important possibly because they have a comprehensive view at the return to work process including the importance of underlying conditions e.g., seeing, sleeping and education whilst the persons with SCI were mainly focusing on their (previous) employment activities solely.

In our study furthermore, 14 ICF categories of the brief ICF core set for acute and/or chronic situation were mentioned in interviews with persons with SCI and considered important by the SCI professionals; however, they are not part of the WORQ, and therefore, they might be considered lacking for use in SCI. This is in line with a previous study where 9 categories were identified as lacking in the WORQ-SELF for use in SCI [7]. The common 5 lacking categories from both studies are: d410 changing basic body position; d445 arm and hand use; e115 products and technology for personal use in daily living; e120 products and technology for personal mobility and transportation and e150 design, construction, and building products and technology of buildings for public use. In this previous study the category “e155 design, construction, and building products and technology of buildings for private use” was furthermore mentioned as lacking in the WORQ. This item was mentioned in 6 interviews with persons with SCI, but was not considered important to the SCI professional team in our study. In hindsight the professionals in our study probably quoted this item as related to the “own private house” not considering the fact that possibly some persons work (partially) from home. Knowing that SCI often involves significant levels of impairment resulting in limited physical functioning and aid- and wheelchair dependence, our findings suggest that the above mentioned categories (see also Table 3) have to be considered when using the WORQ in VR for persons with SCI. The lacking categories describe typical aspects relevant for persons with SCI or severe physical disability. The WORQ may have been developed for the gross, not necessarily physically limited, population needing VR. In addition, the interviews in our study revealed several concepts not included in the ICF list but worthwhile considering for use in VR in persons with SCI. Some of those concepts belong to an existing ICF category and seem very specific to spinal cord injury e.g., b265 touch function, d415 sitting, and e135 products and technology for employment. The concept ‘time’ has been mentioned in 11 of the 14 interviews, however this concept is already present in the WORQ (questions 41 and 42 that are not part of the total score). Interestingly, the category “e425 Individual attitudes of acquaintances, peers, colleagues, neighbors, and community members” was mentioned in 6 interviews and, even if not SCI specific, could potentially be relevant for adding to the WORQ.

Some limitations were encountered during this study. Firstly, only fourteen of the 16 interviews were conducted, two persons with recent low complete SCI and one person with recent high complete SCI were missing. To compensate, one person with semi-recent low complete SCI was included. This resulted in a small sample of recent complete SCI and therefore the aimed variance in participants regarding level of SCI and time since injury was potentially not fully achieved. In addition, coding of the interviews was done in second-level ICF categories, which is less specific.

In conclusion, our study confirms that most categories of the WORQ are important to consider for VR in persons with SCI, however, there are ICF categories that are absent in the WORQ and deemed relevant for use in VR in persons with SCI. Consequently, the content validity of the WORQ without additional items is insufficient for persons with SCI.

For future use in clinical practice in VR of persons with SCI, we advise adding an SCI-specific appendix with the above-described lacking categories. In this way, additions can be developed to serve needs specific for diagnostic groups, and the WORQ core remains intact and can still benefit from compiling evidence resulting from future studies. Also, further research is warranted to investigate the value of using the additional categories described as lacking for the SCI population in this study.