Validity and reliability of the Italian version of the oral assessment guide

ucositis is a general term that describes the inflammatory response of mucosal epithelial cells to the cytotoxic effects of chemotherapy and radiation therapy (Duncan et al., 2003). Any part of the mucous membrane covering surfaces from the mouth to the rectum can be affected by mucositis (Wojtaszek, 2000). Oral mucositis is a frequent complication of the chemotherapy and radiotherapy regimens commonly used in oncology practice. About 40% of patients treated with conventional chemotherapy and more than 70% of the patients with bone marrow transplantation develops oral complications following treatment (Djuric et al., 2006; Silverman et al., 2007). Patients with haematological malignancies develop oral problems even at two or three times the rate of solid tumours and this is believed to be related to the degree of immunosuppression experienced by this population (Silverman, 2007). Oral complications are M Validity and reliability of the Italian version of the oral assessment guide


INTRODUCTION
ucositis is a general term that describes the inflammatory response of mucosal epithelial cells to the cytotoxic effects of chemotherapy and radiation therapy (Duncan et al., 2003).Any part of the mucous membrane covering surfaces from the mouth to the rectum can be affected by mucositis (Wojtaszek, 2000).Oral mucositis is a frequent complication of the chemotherapy and radiotherapy regimens commonly used in oncology practice.About 40% of patients treated with conventional chemotherapy and more than 70% of the patients with bone marrow transplantation develops oral complications following treatment (Djuric et al., 2006;Silverman et al., 2007).
Patients with haematological malignancies develop oral problems even at two or three times the rate of solid tumours and this is believed to be related to the degree of immunosuppression experienced by this population (Silverman, 2007).Oral complications are Professioni Infermieristiche responsible for oral discomfort, burning sensation, pain, nutritional difficulties and can also increase the likelihood of oral or systemic infections (Castaño et al., 2005).
Mucositis is associated with longer periods of hospitalization, significant health and financial costs, and may interfere with the regular administration and dosage of antineoplastic agents (Niscola et al., 2007;D'Angelo et al., 2012).Mucositis induced by chemotherapy can also affect the gastrointestinal and genitourinary tract, causing oesophagitis, gastroenteritis and cystitis.Since the mouth is readily accessible and visible, its assessment can serve as an indication of mucosal tissue damage in other parts of the body (Eilers et al., 2004) Oral mucosa is the first line of defence against infection in the oral cavity so its assessment and hygiene can improve the state of oral health (Eilers et al., 2007).Oral assessment is an essential starting point for the effective management of oral complications, so it is essential to identify mucositis at an early stage and take adequate measures (Sadler et al., 2003).The primary purpose of an oral assessment is to collect data that can be used as a basis for oral care interventions (Potting et al., 2006).Evidence-based clinical practice recommends a regular use of valid and reliable tools to assess oral cavity health, including both self-reporting and professional examination (Harris et al., 2008).
A critical analysis of current literature on mucositis emphasizes that instruments assessing oral cavity should be sensitive to changes (Sonis et al., 1999).A valid and reliable rating tool for routine oral assessment is needed to improve outcomes in patients affected by oral mucositis (Potting et al., 2006).
The Oral Assessment Guide (OAG) (Eilers et al., 1988) is a Multiple Variable Scale and represents a comprehensive instrument that assesses both oral cavity functions and its physical aspects.It provides a more detailed and precise score in the evaluation of oral cavity that is helpful for decision making in nursing care (Sonis et al., 2004).This tool, with an established validity (content, construct) and reliability (Eilers et al., 1988), was considered to be appropriate for daily clinical practice (Glenny et al., 2010).The OAG consists of eight items: saliva, voice, lips, tongue, mucous membrane, gums, teeth/dentures and swallowing.Each item has three progressive descriptors that are scored from one, when the considered variable is healthy, to three, when it has is a severe alteration.The final score, that represents the sum of the individual scores, ranges between 8 and 24, where the higher score indicates the worse oral condition.
To our knowledge, Italian clinicians and researchers lack a valid and reliable instrument to assess mucositis (Catania, 2007;Longo M, et al., 2008).Although some studies have used scales for this purpose (World Health Organization, 1979;Trotti et al., 2000;Orlando et al., 2008), these scales are the results of consensus statements among co-operation groups or small numbers of experts in the field.No studies have tested the reliability and validity of instruments assessing the oral status in Italian patients with haematological malignancies treated with chemotherapy.
The aim of this study was to establish the reliability and validity of the Italian version of the Oral Assessment Guide for assessing mucositis in patients with haematological malignancies undergoing chemotherapy.A reliable and valid tool to assess mucositis will be useful both to improve patients' quality of life and research in this important field.

METHODS
Several phases were followed to establish the psychometric properties of the Italian version of the OAG:

First Phase (Translation and Back-translation)
The OAG was translated into Italian and back-translated into English by two bilingual Italian-English teachers with expertise in medical terminology.The final version was approved by a multidisciplinary panel comprising three oncologists, three oncology nurses, two dental hygienists and a dentist, who discussed in detail the meaning of some English terms of the original OAG items in order to reach their equivalent meaning in Italian.
This process led to change the categories "gums", "teeth/dentures" and "swallow" (reported in the original version) to improve their Italian understanding.For example the score one in the category "Gum" is when gums are "pink and stippled and firm".This sentence was translated into Italian using the sentence "a buccia d'arancia" (literally "orange peel") because this is the exact term used by Italian oral care professionals to describe the normal aspect of gums.
In the "Teeth/dentures" category the OAG uses the terms "debris" and "plaque".These terms were initially translated respectively with the Italian words "residuo" and "placca" that correspond exactly to the original English words.
Then the panel decided to use the terms "residuo alimentare" (food debris) and "placca batterica" (bacterial plaque) because this allowed oral health care professionals to avoid making mistakes in their assessments.In the "Swallow" category, the score is one when patients complain "some pain" while swallowing.This description was changed into "difficulty in swallowing" in order to remove any reference to pain from the OAG.

Second Phase (Establishing Content Validity)
After obtaining the final version of the Italian OAG, the tool underwent to content validity according to the procedure suggested by Lynn (1986) and by Polit and Beck (2006).As successfully done also by Gibson et al. (2006) a panel of experts composed by 12 professionals (three oncologists, three dentists, three nurses, and three dental hygienists) evaluated the relevance of the OAG items and the descriptors to assess and rate oral cavity status respectively.Both items and descriptors were rated for relevance by the panel of experts using a Likert Scale from 1 = "not relevant" to 4 = "very relevant".Then, Likert options were dichotomised according to the procedure by Lynn (1986) and Polit and Beck (2006) to compute the Content Validity Index (CVI) for each item and descriptor.

Third Phase (Recruitment of Sample and OAG Assessment)
Fourteen patients were selected for OAG assessment.A total of 60 pairs of assessments were made by dental hygienists (120 assessments in total).
Patients of our study was selected according to the following criteria: -18 years and over; -agree to participate to the study; -diagnosed with haematological malignancy (acute leukaemia, non Hodgkin's lymphoma, and Hodgkin's disease) and treated with chemotherapy.
Patients suffering from any muco-cutaneos disease were excluded from the study.
All patients participating in the study were treated with an internal standardized oral care protocol consisting of rinsings with a sodium bicarbonate and Nystatin solution four times a day and brushing oral cavity with a soft toothbrush until the start of chemotherapy.To test the responsiveness to change of the tool according to the oral cavity condition, the two dental hygienists examined the patients' mouth in different moments: just before, during and after chemotherapy.In this manner they carried out for each moment 20 pairs of assessments.

Fourth Phase (Inter-Rater Reliability)
In order to establish the inter-rater reliability, the assessment of oral cavity using the OAG was made by a pair of dental hygienists with expertise in oral cavity assessment in oncology patients.Dental hygienists independently examined patients before, during and after chemotherapy.The assessment took place in a systematic manner in 5 minutes maximum.Before each assessment all the patients received oral hygiene to avoid mistakes.To complete the OAG, oral examination was performed according to Gerpen's protocol (2003) using a halogen light source, spatula and gauze and non-sterile gloves.Examinations were carried out in the patient's own room, with the patient either lying in bed or sitting on a chair.Assessors were instructed to examine each patient individually within 4 hours as suggested by Eilers (2004).All assessments were registered on separate forms and were collected after each assessment by the first author.

Ethics
The Institutional Review Board approved the study.All the patients received written information and signed the informed consent form.All the approached patients accepted to participate in the study.

Statistical Analysis
The internal consistency reliability was calculated using Cronbach's alpha.Inter-rater agreement between dental hygienist assessments was calculated using Cohen's Kappa coefficient.Also agreement, sensibility and specificity of assessments were calculated.To establish the validity of the OAG, the Content Validity Index of OAG items and descriptor were calculated according to the Lynn formula (1986).Repeated measure analyses of the variance (ANOVA) were used to determine differences in mean values between the assessments of the dental hygienists.The SPSS 13.0 for Windows was used to analyze the data.

RESULTS
A total of 120 assessments (60 pairs) were made on 14 patients by the two dental hygienists.Table 1 shows the description of the patients whose oral cavity was examined with the OAG.Patients had a mean age of 55 years, and most were males.

Professioni Infermieristiche
Nine patients were affected by Acute Leukaemia, the others suffered from Non-Hodgkin's and Hodgkin's Lymphoma.All patients were treated with Chemotherapy.The oral assessment score ranged between 8 and 20 (mean 14.02).
Table 2 shows the number of patients who presented alterations of the oral cavity and the total frequency of these alterations according to the OAG items.These alterations were similar to other studies carried out on mucositis (Eilers et al.,1988;Borowski, et al.,1994;Andersson et al., 1999;Djuric et al., 2006;).The most common problem in the oral cavity complained by patients were related to changes in the mucous membrane (67%), saliva (65%) and gums (50%).Other common changes in the OAG index were related to lips (48%) and teeth (42%).

Reliability
The reliability of the OAG was tested for internal consistency and inter-rater reliability.The internal consistency measured by Cronbach's alpha was 0.84.Inter-rater reliability between the assessments of the two Dental Hygienists measured by Cohen's Kappa coefficient and agreement between assessments are reported in Table 3.As suggested by Landis and Koch (1997) Cohen's Kappa value < 0.20 is considered "poor", between 0.21 and 0.40 "fair", between 0.41 and 0.60 "moderate", between 0.61 and 0.80 "good" and > 0.80 "very good".In our study Cohen's Kappa, ranged from "good" to "very good" with the best agreement for "swallowing" (0.87) and the worst for "teeth" (0.65).
Considering the couples of assessment, sensitivity and specificity were calculated for descriptors of each OAG item (Table 4).The difference between the means of OAG evaluations made by the two assessors were not statistically significant (p = 0.613).Test-retest reliability was not evaluated because of the rapid changes in the patients' health status during chemotherapy.

Validity
Table 5 shows the CVI for each item of the OAG according to the evaluation of the 12 experts.CVI ranged from 0.83 to 1.00 demonstrating that the items of the OAG highly contributed to measure the oral cavity status.

OAG items
Patients with alterations (n=14) Frequency of alterations in all assessments (n=120; %) As successfully done by Gibson et al. (2006), also the CVI of each item descriptors of the OAG was calculated (Tables 6 -8).Descriptors with score 1 (that define the healthiest oral status) had a CVI very high (for four items the coefficient was 1).Only the descriptors "Normal" in the "Voice" item was 0.67.Even better results were obtained for the descriptors with score 2 and 3: these last descriptors reached a coefficient of 1 in seven of the eight item of the OAG.

DISCUSSION
Oral mucositis is one of the main complications in non-surgical cancer treatments (Alterio et al., 2007).Assessment is essential in the prevention and treatment oral mucositis (Graham et al., 1993) because this improves outcomes in cancer patients (Jaroneski, 2006).Systematic mouth examination with a valid and reliable instrument provides healthcare professionals with a lot of information to guide appropriate care (Eilers et al., 2004;Eilers et al., 2007).Furthermore, oral assessment allows early detection of mouth alterations and suggests the most appropriate intervention to improve the oral status and decrease the risk of infection (Andersson et al., 1999).
The process of developing and validating an instrument largely focuses on reducing errors in the measurement process (Sitzia et al., 1997;Sili et al., 2010).Instruments should be both valid and reliable to produce data to guide practice and improve outcomes, thus it is important that these instruments limit the amount of errors in measuring and capturing changes (Keefe et al., 2007).
To our knowledge there are no valid and reliable tool to assess mucositis in Italy.To establish the reliability and validity of the Italian version of the OAG we followed a rigorous procedure outlined in the literature (Lynn, 1986;Polit and Beck, 2006).
Back translation and peer review by experts are considered a valid approach to assess semantics in instrument translation (Sitzia et al., 1997).In order to reach the same meaning of the original version, some descriptors were not literally translated but changed with terms usually used by dental care professionals.This translation was done according to Lindhe's terminology outline (Lindhe et al., 2006).
During the second phase, all experts agreed on the importance of the eight categories of the OAG and this was confirmed by the high score of the Content Validity Index per item, ranging from 0.83 to 1.
In the original study, Eilers (1988) identified three descriptors to rate each of the eight items, so that these descriptors could be considered as a Likert scale.In the present study, the Content Validity Index for descrip- tors was high (ranging from 0.67 to 1), and experts appreciated the scale for its clarity, wording, efficiency and simplicity.This characteristic is essential for an instrument to be used in clinical practice and research (Potting et al., 2006).The panel did not recommend any major modification to the version they reviewed.Despite this, the limit of the OAG is that it lacks of pain assessment notwithstanding patients with mucositis had pain (Cella et al., 2003;Duncan et al., 2003).In order to consider also this variable, the panel recommends to use a separate scale of pain to achieve a comprehensive assessment of oral cavity health, as suggested also by literature (Parulekar et al.,1998;Blijlevens et al., 2006;Gibson et al., 2006;Eilers et al., 2007) Inter-rater reliability between the assessments made by the dental hygienists as measured by the Cohen's Kappa coefficient ranged from 0.64 to 0.86.This is in agreement with Eilers' study (1988) where the interrater score was similar.Swallowing had the best agreement between the pairs of assessment (0.87), instead teeth, the worst (0.65).This is similar to the results of other studies (Eilers et al., 1988;Andersson et al., 1999;Andersson et al., 2002;Knöös et al., 2010) and might indicate that dental hygienists find more difficult to assess the teeth than the swallowing.
The percentage of assessment errors of a one point was 8.2%, of two errors was 5.8% and of three errors was 1.9%.Discordances of more than one point was likely due to simple measurement errors or the lack of sufficient site definition, which may have led to mistaken scoring of contiguous areas.For example, if a red oedematous area was present on the gums and this spread to the mucous membrane, one assessor may have given a score related to the gums but not to the mucous membrane, whereas the second assessors did the opposite.
In our study most of assessment disagreements were in relation to the severity of mucositis, so it was easier for assessors to evaluate a healthy mouth than one with problems.However, it is also reasonable to think that repeated examinations could be troublesome for patients suffering from severe mucositis, and this could have influenced the second examiner's ability to inspect all the designated sites.
We found that the sensitivity and specificity of the OAG (Table IV) was good for each descriptor and this is important for a tool designed to measure the impact of specific interventions.
Only three patients, just before chemotherapy, had the lower OAG score and these were also the youngest of the sample.In fact, we noticed that the patients' oral status was not healthy.Of the patients, 8 (57%) showed alterations of the oral cavity even before chemotherapy, with high levels of plaque and perio-dontal disease and an OAG score ranging between 10 and 14.Many studies suggest that pre-existing oral disease and poor oral hygiene are contributing factors to mucositis, and that comprehensive oral care, including the maintenance of good oral hygiene during therapy, may reduce the severity of oral mucositis (Borowski et al., 1994;Cheng et al., 2001;Djuric et al., 2006;Bernardi et al., 2010).Patients should be informed on the importance of practicing oral hygiene.
The most common alterations in the oral cavity, during chemotherapy, were related to changes in the mucous membrane (67%), saliva (65%) and gums (50%).Alterations in the mucous membrane may lead to infections, especially if patients are immunocompromised, while the reduction of saliva production may increase the risk of mucous infections because saliva plays an important role in protecting the mucous membranes against bacterial and fungal attacks.
In this type of patients, gums status changed rapidly in relation to leucocytes, thrombocytes value and the length of time elapsing since the course of treatments (Djuric et al., 2006).
The patients' oral status changed quickly after beginning chemotherapy, as found by several authors (Duncan et al., 2003;Eilers et al., 2004;Castaño et al., 2005;Djuric et al., 2006;).These changes occurred steadily from the best to the worst record scores.Therefore, assessments should be made more regularly and accurately during this period, so changes can be rapidly identified.

LIMITATIONS
Our study also presented some limits.First, to gain a better understanding of the validity of the OAG in the clinical setting, we believe that dental hygienists' assessments ought to be compare with the nurses' assessment, and see if they agree.Nurses have a key role in identifying signs and symptoms of oral mucositis because they take care of patients on a 24-hour basis (Mark W, et al., 2005) Other studies have made these comparisons (Parulekar et al., 1998;Andersson et al., 1999;Andersson et al., 2002) and in general showed that nurses' assessment of patients' oral status was better than those of dental hygienists.The second limit was the lack of a fixed day to asses mucositis.We also divided the assessment process into different phases, before, during and after chemotherapy, but the evaluations were not done on the same day for each patient.This made it difficult to establish exactly when the onset of mucositis occurred in most patients (i.e. on day 10, after beginning chemotherapy, rather than on day 14).
The third limit is due to the small sample size of patients with similar malignancies.But being the aim of the study the reliability testing of the tool we focused more on the assessments rather than on the number of patients.Because the sample was composed only with hematological patients, questionable could be the validity and reliability of the OAG for solid neoplasms.Further studies should be carry out to test the OAG validity and reliability in other different clinical settings to confirm our results.
Notwithstanding these limits, this study was important because OAG is the first tool in Italy to be validated for the assessment of oral mucositis in haematological patients.It will contribute to a better assessment of patients and consequently to improve the prevention and treatment of mucositis also in palliative care settings (D'Angelo et al., 2013).In addition, the OAG will be also useful for future research to improve patient outcomes.

Table 2 .
Number of Patients With Alteration of the Oral Cavity and Frequency of Alteration According to the Oral Assessment Guide Items Table 3. Inter-rater Agreement of Assessments for Each Oral Assessment Guide Item Among Dental Hygienists

Table 4 .
Sensitivity and Specificity of Descriptors for Each Item of the Oral Assessment Guide Between Assessments of Dental Hygienists

Table 5 .
Content Validity Index for Each Item of the Oral Assessment Guide

Table 6 .
Content Validity Index of the Descriptors With Score 1 of Each Item of the Oral Assessment Guide

Table 8 .
Content Validity Index Score for the Descriptors with Score 3 of Each Item of the Oral Assessment Guide

Table 7 .
Content Validity Index of the Descriptors with Score 2 of Each Item of the Oral Assessment Guide