Theofilou, Aroni, Ralli, Gouzou, and Zyga: Measuring Health: Related Quality of Life in Hemodialysis Patients. Psychometric Properties of the Missoula-VITAS Quality of Life Index (MVQOLI-15) in Greece

Measuring Health: Related Quality of Life in Hemodialysis Patients. Psychometric Properties of the Missoula-VITAS Quality of Life Index (MVQOLI-15) in Greece

Abstract

Different measuring tools have been used to understand the outcomes of end-stage renal disease (ESRD) therapies. However, survival, cost-effectiveness and quality of life (QOL) are the main parameters to evaluate treatment of ESRD. The current study meant to assess the psychometric properties (reliability and validity) of the Missoula-VITAS Quality of Life Index (MVQOLI-15) translation to Greek in patients undergoing hemodialysis (HD). A total sample of 79 HD patients voluntarily participated in this pilot study. Domain analysis of MVQOLI-15-Greek was conducted based on the collected data at initial assessment. The reliability properties of the instrument were tested using the following measures; internal consistency, repeatability, test-retest reliability and convergent validity. Domain analysis demonstrated that all domains of the questionnaire had good variability. MVQOLI-15-Greek internal consistency was satisfactory with an overall Cronbach’s a at 0.74. Pearson’s r and intraclass correlation coefficient revealed strong correlations (ranging from 0.91 to 0.98) between initial assessment and re-assessment. MVQOLI-15-Greek convergent validity analysis indicated that the domains were strongly related to the same construct. The findings of the study indicate that the Greek version of MVQOLI-15 provided satisfactory psychometric properties supporting its use within pathological populations and in the context of national QOL measurement.





Introduction

Health-related quality of life (HRQOL) is increasingly recognized as an important patient-reported outcome in health care research. However, the use is still restricted and several questions remain about the value and feasibility of using measures of HRQOL in routine health care.1 In the medical field health status measurement, quality of life (QOL) and HRQOL assessment are interchangeably adopted to describe an evaluative process in which the instruments concern about the effects that a disease, treatment or a more complex intervention may have on a person’s physical or emotional performance in everyday activities.2 The impact of end-stage renal disease (ESRD) on a patient’s QOL has become increasingly recognized as an important outcome measure.3-5 Some generic measures such as the 36-item Short Form health survey (SF-36) are used to assess HRQOL. However, generic instruments are broad and produce scores for all domains of QOL.6 They try to cover each area specifically and may not even address the primary symptoms. It is obvious that various QOL measurement tools have been designed but most may not be ideal for use in palliative care patients, whose QOL assessment should focus on areas for which palliative care is most effective, such as psychosocial and spiritual problems.7,8

In response to the need for a QOL measure that assesses the individual experience of people nearing the end-of-life (EOL), Byock and Merriman created the Missoula-VITAS Quality of Life Index (MVQOLI).8 The MVQOLI is an assessment instrument that gathers patient-reported information about QOL during advanced illness. Maintaining optimal QOL is a core goal of palliative and hospice care, and information gathered via the MVQOLI assists health care professionals in identifying and addressing patient concerns that affect QOL. The MVQOLI has been used in many different healthcare settings including hospice, hospital, home health, long-term care (including assisted living), outpatient palliative care, disease management and pre-hospice programs.9

The framework of the MVQOLI is based on Ira Byock’s work regarding growth and development at the end of life and the concepts of landmarks and tasks of life closure.9 The MVQOLI asks patients about 5 dimensions or domains of QOL: symptoms, function, interpersonal, well-being and transcendence. There is also a total score. The instrument is specifically designed to assess the patients personal experience in each of these dimensions, hence the MVQOLI items are constructed with highly subjective language and no scores appear on the version of the tool seen by patients. The tool seeks to describe the qualitative and subjective experience of QOL in a way that can be quickly interpreted by professional caregivers.10

Within each dimension, three kinds of information are gathered from respondents in order to illuminate their overall experience:10

  • Assessment (A): subjective measurement of actual status or circumstance (What it is.) Example: I feel sick all the time.

  • Satisfaction (S): degree of acceptance or mastery of actual circumstance (How much does it bug you?) Example: I am satisfied with current control of my symptoms.

  • Importance (I): degree to which a given dimension has an impact on overall QOL (How much does it matter?) Example: physical discomfort overshadows any opportunity for enjoyment.

Each dimension is defined by the patient’s perception and/or experience not the judgment of caregivers (family or professional). The definitions for the dimensions and examples of items for each response category are shown below.

Symptoms-experience of the physical discomfort associated with progressive illness; perceived level of physical distress.

(A) 1. I feel sick all the time.

(S) 2. I am satisfied with current control of my symptoms.

(I) 3. Physical discomfort overshadows any opportunity for enjoyment.

Function-perceived ability to perform accustomed functions and activities of daily living, experienced in relation to expectations.

(A) 4. I am no longer able to do many of the things I like to do.

(S) 5. I accept the fact that I cannot do many of the things that I used to do.

(I) 6. My contentment with life depends upon being active and being independent in my personal care.

Interpersonal-degree of investment in personal relationships and the perceived quality of one’s relations with family and friends.

(A) 7. I have recently been able to say important things to the people close to me.

(S) 8. At present, I spend as much time as I want with family and friends.

(I) 9. It is important to me to have close personal relationships.

Well-being - self-assessment of an internal condition; subjective sense of emotional wellness or disease; contentment or lack of contentment with self.

(A) 10. My affairs are not in order; I am worried that many things are unresolved.

(S) 11. I am more satisfied with myself as a person now than I was before my illness.

(I) 12. It is important to me to be at peace with myself.

Transcendent-experienced degree of connection with an enduring construct; degree of experienced meaning and purpose in life.

(A) 13. I have a better sense of meaning in my life now than I have had in the past.

(S) 14. Life has lost all value for me; every day is a burden.

(I) 15. It is important to me to feel that my life has meaning.

Each item uses a five-point Likert scale recorded so that the lowest score always indicated the least desirable situation and vice versa. The questions are general, which means that the MVQOLI provides information about the domains that detract from or augment the patient’s QOL. The MVQOLI also incorporates a single item quality-of-life status question (global score), which was used to assess the convergent validity of the MVQOLI.11

There are two versions of the MVQOLI: -15 item and -25 item. The instrument was initially designed with 25 items. Clinicians reported that the tool was too long for some patients to complete. Using data from the original study of reliability and validity, a 15-item revised version was constructed that has a correlation coefficient of 0.93 with the 25-item version, indicating that little information is lost when only 15 items are used. The newest versions of the tool included with this guide have been revised using simpler language and item formats to make it easier to use for both patients and staff.

The MVQOLI can be scored using an EXCEL program or manually. Its scoring protocol is designed to turn the qualitative subjective experience of the patient into quantitative information that can be easily interpreted by the care team. The unique scoring system has the advantage of revealing how much each domain affects QOL. For example, efforts to make a patient comfortable may contribute little to QOL if that domain is not important to them. In addition, small changes in any domain may affect QOL a lot if that domain is very important to the patient.

The MVQOLI items are scored as follows:

Assessment (-2 to +2)

Satisfaction (-4 to +4)

Importance (1 to 5)

Where: (Assessment + Satisfaction) x Importance = QOL in each dimension

Note that the assessment and satisfaction scores can range from -6 to +6 and indicate whether the patient assess his/her situation positively or negatively. When multiplied by the importance factor, the overall dimension score is magnified by how important that domain is. The final score in each dimension reflects the overall impact of that domain on QOL.

  1. Negative dimensions are reducing QOL,

  2. Positive dimensions are increasing QOL

  3. The size of each dimension reflects the amount of impact.

Most questionnaires used to evaluate the QOL were developed for English speaking populations. In consequence, they are rarely adequate in terms of correct translation or correspondence to the reality of other countries.12 So, these questionnaires have to be validated in order to be applied to the reality of each specific population.

A literature review in PubMed/Medline revealed that, among instruments for QOL/HRQOL measurement in nearing the EOL, the only tool specifically aimed at measuring the HRQOL of chronic disease patients in advanced stages is the MVQOLI. No studies are found on the psychometric properties of instruments for advanced disease patients in Greece. Given the lack of this type of instrument in our country, the present research aimed to assess the validity and reliability of the Missoula-VITAS Quality of Life Index-15 revised item (MVQOLI-15R).

Materials and Methods

Cultural adaptation

Adaptation and translation of the MVQOLI was done according to the criteria for translation and adaptation of generic health-related QOL measures.13

The MVQOLI-15R version was translated from the source language (English) to the target language (Greek). Translation was done according to the guidelines for adapting instruments in multiple languages and cultures.14 Translators who were conversant with both the source and target languages, and had skills in cross-cultural adaptation of instruments, made two independent forward translations and two independent backward translations. The final version was independently reviewed and translated by a bilingual health psychologist without previously seeing the original MVQOLI. The back translated version had very close concordance with the original MVQOLI, as verified by a professional linguist fluent in both the English and Greek languages. A social scientist conversant with both languages carried out the final step of smoothing out the language. This involved editing the target language version of the instrument in a consistent writing style. This helped to ensure that patients could easily understand the modified version of the MVQOLI. A HRQOL expert reviewed the final instrument to check for omissions. Field-testing of the provisional version included its completion by a group of patients undergoing hemodialysis (HD) (n=10), by means of one-to-one interviews, in order to examine the potential distribution of responses, check comprehension, and to ensure linguistic and content validity.15

Study population

A cohort of 79 HD patients was recruited from three General Hospitals in the broader area of Peloponnese. Selection criteria included: i) >18 years of age; ii) ability of communication in Greek; iii) diagnosed with ESRD; iv) dialysis treatment at least for a year; v) satisfying level of cooperation and perceived ability.

The rate of response was very high, reaching 100%. Thus, the total sample includes all patients of these three units, consisting of 43 males (54.4%) and 36 females (45.6%), with a mean age of 62.43 years ±15.91. Fifty patients (63.3%) were married, 16 (20.3%) single and 13 (16.4%) divorced or widowed. Further, the majority of patients (60.8%) had elementary education, 29.1% had secondary and 10.2% had university education. The mean duration of treatment was 4.80 years ±4.44. Participants were Greek adults having signed a consent form for participation. All subjects had been informed of their rights to refuse or discontinue participation in the study according to the ethical standards of the Helsinki Declaration. Ethical permission for the study was obtained from the scientific committees of the participating hospitals. The study took place between September 2012 and December 2012.

Procedure

At initial assessment (day-1), the Greek version of the MVQOLI-15R questionnaire was given to all participants and completed by themselves, under the supervision of one of the members of the research team. The MVQOLI-15R was re-administered to all participants by the same examiner two weeks after the first appointment day (day 14). Between assessments, no variation in individuals’ clinical status was recorded and no treatment interventions were received.

Data analysis

A P value of 0.05 or less was considered to indicate statistical significance. All analyses were performed with the Statistical Package for the Social Sciences (SPSS 13.0 for Windows). Kolmogorov-Smirnov tests were performed in order to check whether the values of the sample would fall within a normal distribution.

Domain analysis of the MVQOLI-15R was used to examine the variability among the 5 domains, the total score as well as the global score and to identify if any of these domains on the questionnaire did not have a positive monotonic trace when plotted against the total score.16 Domain analysis was carried out using the mean and standard deviation data of the MVQOLI-15R domains from the initial assessment (day 1).

The reliability of the MVQOLI-15R was evaluated by assessing the instrument’s internal consistency, repeatability and its test-retest reliability. Internal consistency evaluates how well different questions (domains) that test the latent structure of the instrument should give consistent results.17 The internal consistency of the questionnaire was assessed with Cronbach’s alpha coefficient (Cronbach’s a) using the data obtained from the initial assessment.18 A threshold value of 0.70 was chosen, which indicates sufficient reliability for research purposes.19,20 In the present study, the Cronbach’s a if item deleted was used as an additional evaluation test of the MVQOLI-15R internal consistency.21 Repeatability was defined as the stability of participants’ responses over time, that is, the ability of the instrument to give consistent results whenever it is used.22 The MVQOLI-15R repeatability was determined by calculating Pearson’s product moment correlation coefficient (Pearson’s r) between the initial and re-assessment total scores of the questionnaire. The Pearson correlation coefficient values were specified as follows: 0.00-0.19 = very weak correlation; 0.20-0.39 = weak correlation; 0.40-0.69 = moderate correlation; 0.70-0.89 = strong correlation; and 0.90-1.00 = very strong correlation.23 The test-retest reliability of the instrument was defined as the degree to which the participants maintained their opinion in the repeated measurements of the MVQOLI-15R questionnaire, taking into account the error in measurements as a proportion of the total variance. Test-retest reliability was evaluated using the intra-class correlation coefficient (ICC) with 95% confidence interval (CI). The ICC, which is the most suitable statistical test for the assessment of reliability ranges from 0 to 1, with 1 indicating perfect reliability.24-26 The Cronbach’s a and ICC correlations were characterized as follows: 0.00-0.25 = little, if any, correlation; 0.26-0.49 = low; 0.50-0.69 = moderate; 0.70-0.89 = high; and 0.90-1.00 = excellent.27 In addition, the scores of the two assessments were tested for systematic differences using the paired t-test, because the ICC does not correct for systematic differences and agreement by chance.

Finally, the convergent validity of the MVQOLI-15R was evaluated by examining the correlations between the total score of the scale and the domain scores, at initial assessment. Acceptable convergent validity should be indicated by high or excellent (0.70 to 1.00) domain intercorrelations for all domain pairings. This would provide evidence that all 5 domains of the MVQOLI-15R are related to the same construct.28

Results

Descriptives

The values of the total cohort were found to pass the normality distribution test. At initial assessment, 79 participants completed the questionnaire. The mean MVQOLI-15R total score was 17.36 (SD±3.76), ranging from 6.50 to 24.50. There were no missing items for the MVQOLI-15R score. At the second administration (day 14), all participants completed the questionnaire and the mean MVQOLI-15R total score was 17.21 (SD±3.66), ranging from 6.50 to 23.40.

Domain analysis

Domain analysis for the MVQOLI-15R instrument demonstrated that all domains had a positive monotonic trace when plotted against the total score apart from the domain of well-being. Domain analysis statistics are presented in Table 1, with domain means (average response for each domain) ranging from 7.48 (domain of well-being) to 16.26 (domain of interpersonal). There was good variability in relation to the means (SDs ranged from 8.94 to 15.17).

Reliability

The internal consistency of the MVQOLI-15R was satisfactory, with an overall Cronbach’s a at 0.74, ranging between 0.69 (domain of well-being) and 0.74 (domain of function) (Table 2). Most values were higher than the chosen threshold value of 0.7, suggesting that most MVQOLI-15R domains are interdependent and homogeneous in terms of the construct they measure. The paired samples t-test between the MVQOLI-15R total score at initial assessment and re-assessment indicated no statistically significant difference, (P=0.10, Table 3). Pearson’s r and the ICC coefficient revealed excellent correlations between initial assessment and re-assessment (Table 3). Our results indicated that the total score of the MVQOLI-15R was remarkably consistent between the two measurements.

Convergent validity

Table 4 summarizes the correlations between the MVQOLI-15R total score and the domain scores of the questionnaire at initial assessment (domains/total score correlations). All domains showed quite satisfactory correlation coefficients, ranging from 0.35 (domain of function) to 0.75 (domain of transcendent), indicating that MVQOLI-15R domains were same construct.

Discussion

This is the first study to examine the psychometric properties of the MVQOLI-15R questionnaire in a Greek population cohort. The constructed Greek version of the MVQOLI-15R was tested in patients with ESRD undergoing hemodialysis and was found to have good internal consistency, excellent repeatability, very high test-retest reliability and satisfactory convergent validity properties.

Domain analysis

Domain analysis was used in order to determine the participants’ level of QOL. By examining the mean values of the responses in every MVQOLI-15R domain, QOL limitations of the selected population group were studied. The high mean values of the responses in domains like interpersonal could be explained by the fact that our selected population group was comprised of married people with a supportive social network. The low mean values of the responses in domains like symptoms, well-being and function could be explained by the fact that HD group was comprised of older people.

Reliability

Our results indicated that the MVQOLI-15R was consistent, stable and highly repeatable between the two assessments. Analysis of the internal consistency of the MVQOLI-15R showed that the domains of the scale quite satisfactorily measure all aspects of the participants’ QOL. The MVQOLI-15R overall Cronbach’s a coefficient was quite similar to that of the Uganda version (0.83),11 and the English version (0.77) altogether indicating a good internal consistency for the MVQOLI-15R scale.29 The values of Pearson’s r indicated excellent repeatability of an individual’s response over time. The MVQOLI-15R questionnaire’s test-retest reliability was found to be very high, with a low standard error of measurement, in line with the results of Namisango et al.11 for the Uganda version as well as Schwartz et al.29 for the English version.

Convergent validity

The domains-total score correlations of the MVQOLI-15R provided evidence that all 5 domains converge on the same construct.

Strengths and limitations

The standardized methods that were used in all phases during the cross-cultural adaptation of the original MVQOLI-15R scale and the random selection of the participants from a well-defined and homogenous target population are important strengths of this study. In addition, testing the reliability of the MVQOLI-15R instrument using four standardized statistical measures added statistical power to our results. However, there is a potential limitation associated with the present study; The MVQOLI-15R was examined for reliability only in HD group. Therefore, it remains necessary to extend the study of the instrument’s clinimetric properties to other ESRD groups, like patients undergoing peritoneal dialysis. Documentation of the validity properties of the MVQOLI-15R questionnaire is the purpose of an ongoing study by the same research team.

Conclusions

In the present study, the MVQOLI-15R questionnaire was used to evaluate QOL of HD patients. The instrument demonstrated excellent reliability properties, comparable to that of the Uganda and English version. Overall, it can be suggested that the MVQOLI-15R might be a reliable tool for assessing QOL issues in patients with ESRD who are on maintenance hemodialysis.

Acknowledgments

The authors would like to thank the patients for their participation in the study and acknowledge the support given by the health professionals and the administration personnel of the dialysis participating units.

References

1. 

P Theofilou. Outcome measurement in palliative care: quality of life. J AIDS Clinc Res 2012;3:1-6.

2. 

G Apolone, P. Mosconi Review of the concept of quality of life assessment and discussion of the present trend in clinical research. Nephrol Dial Transplant 1998;13:65-69.

3. 

P Theofilou. Self-reported functional status: an important predictor of mental health outcomes among chronic dialysis patients. Eur J Psychol Assess 2012;28:1-7.

4. 

P Theofilou. Association of insomnia with kidney disease quality of life reported by patients on maintenance dialysis. Psychol Health Med 2013;18:70-8.

5. 

P Theofilou. Medication adherence in Greek hemodialysis patients: the contribution of depression and health cognitions. Int J Behav Med 2012 Mar 10. [Epub ahead of print].

6. 

DV Joshi, N Mooppil, FYJ Lim. Validation of the kidney disease quality of life-short form: a cross-sectional study of a dialysis-targeted health measure in Singapore. BMC Nephrology 2010;11:36.

7. 

SR Cohen, BM Mount. Quality of life in terminal illness: defining and measuring the subjective well-being in the dying. J Palliat Care 1992;8:40-5.

8. 

IR Byock, MP Merriman. Measuring quality of life for patients with terminal illness: the Missoula-VITAS quality of life index. Palliat Med 1998;12:231-44.

9. 

IR Byock. The nature of suffering and the nature of opportunity at the end of life. Clin Geriatr Med 1996;12:237-52.

10. 

P Theofilou. Assessing heath-related quality of life using the Missoula - Vitas Quality Of Life Index (MVQOLI). J Clin Trials 2012;2:1-2.

11. 

E Namisango, E Katabira, C Karamagi, P. Baguma Validation of the Missoula-Vitas Quality of-Life Index among patients with advanced AIDS in Urban Kampala, Uganda. J Pain Symptom Manage 2007;33:189-202.

12. 

F Guillemin, C Bombardier, D. Beaton Cross cultural adaptation of health-related quality of life measures: literature review and proposed guidelines. J Clin Epidemiol 1993;46:1417-32.

13. 

A Bowden, JA Fox-Rushby. A systematic and critical review of the process of translation and adaptation of generic health-related quality of life measures in Africa, Asia, Eastern Europe, the Middle East, South America. Soc Sci Med 2003;57:1289-306.

14. 

RK Hambleton. Issues, designs and technical guidelines for adapting tests into multiple languages and cultures. In: RK Hambleton, PF Merenda, CD Spielberger, eds. Adapting educational and psychological tests for cross-cultural assessment. London: Routledge; 2004. pp. 3-38.

15. 

P Theofilou, F Kapsalis, H. Panagiotaki Greek version of MVQOLI-15: translation and cultural adaptation. Int J Caring Sci 2012;5:289-94.

16. 

Red River College. Item analysis. Available from: http://xnet.rrc.mb.ca/tomh/item_analysis.htm. Accessed: 15 March 2012.

17. 

WMK Trochim. Types of reliability. Internal consistency. 2006. Available from: http://www.socialresearchmethods.net/kb/reltypes.php. Accessed: 19 March 2012.

18. 

LJ Cronbach, RJ Shavelson. My current thoughts on coefficient alpha and successor procedures. Educ Psychol Meas 2004;64:391-418.

19. 

LJ Cronbach. Coefficient alpha and the internal structure of tests. Psychometrica 1951;16:297-334.

20. 

JC Nunnally. Psychometric theory. New York: McGraw-Hill; 1978.

21. 

JA Gliem, RR Gliem. Calculating, interpreting, and reporting Cronbach’s alpha reliability coefficient for likert-type scales. Available from: https://scholarworks.iupui.edu/bitstream/handle/1805/344/Gliem+&+Gliem.pdf?sequence=1.

22. 

KE Berg, RW Latin, eds. Measurements and data collection concepts. Essentials of research methods in health, physical education, exercise science, and recreation. 2nd edition. Baltimore-Philadelphia: Lippincott Williams & Wilkins, 2004; 165.

23. 

J Fowler, P Jarvis, M. Chevannes Practical statistics for nursing and health care. West Sussex, England: John Wiley & Sons, Ltd; 2002.

24. 

PE Shrout, JL Fleiss. Intraclass correlations: uses in assessing rater reliability. Psychol Bull 1979;86:420-8.

25. 

G Dunn, B. Everitt Clinical biostatistics: an introduction to evidence-based medicine. London: Edward Arnold, 1995.

26. 

KJ Ottenbacher, SD Tomchek. Reliability analysis in therapeutic research: practice and procedure. Am J Occup Ther 1992;47:10-6.

27. 

L Portney, M. Watkins Foundation of clinical research: applications to practice. Upper Saddle River: Prentice-Hall, 2000.

28. 

Web center for social research methods. Knowledge base. Convergent and discriminant validity. Available from: http://www.socialresearchmethods.net/kb/convdisc.php. Accessed: 18 March 2012.

29. 

EC Schwartz, PM Merriman, G Reed, I. Byock Evaluation of the Missoula-VITAS Quality of Life Index-revised: research tool or clinical tool? J Palliat Med 2005;8:121-35.

Table 1.

Domain analysis of the Missoula’s Vita Quality of Life Index-15 Revised.

Domain Mean ± SD
Global score (global QOL) 3.25±1.11
Symptoms 3.83±11.15
Function 4.26±8.94
Interpersonal 16.26±12.52
Wellbeing 7.48±13.58
Transcendent 6.75±15.17
Total score 17.36±3.76

[i] QOL, quality of life; SD, standard deviation.

Table 2.

Internal consistency of the Missoula’s Vita Quality of Life Index-15 Revised (overall alpha: 0.74).

Domain Cronbach’s a
Global score (global QOL) 0.75
Symptoms 0.71
Function 0.74
Interpersonal 0.70
Wellbeing 0.69
Transcendent 0.70
Total score 0.56

[i] QOL, quality of life; SD, standard deviation.

Table 3.

Reliability properties of the Missoula’s Vita Quality of Life Index-15 Revised.

Property Measure Value Significance
Internal consistency Cronbach’s a 0.74 -
Repeatability Pearson’s r (0.91-0.99) <0.001
Test-retest reliability I ICC (95% CI) 0.84 <0.001
Test-retest reliability II Paired samples t-test 17.36±3.76*
17.21±3.66° NS (P=0.10)

[i] ICC, intraclass correlation coefficient; CI, confidence interval.

[ii] *MVQOLI-15R at initial assessment

[iii] °MVQOLI-15R at re-assessment.

Table 4.

Convergent validity of the Missoula’s Vita Quality of Life Index-15 Revised (domain-total score correlations).

Domain Pearson’s r
Global score (global QOL) 0.51
Symptoms 0.59
Function 0.35
Interpersonal 0.60
Wellbeing 0.67
Transcendent 0.75

[i] Global QOL, global quality of life.

Abstract views:
666

Views:
PDF
348
APPENDIX
105
HTML
1781

Article Metrics

Metrics Loading ...

Metrics powered by PLOS ALM


Copyright (c) 2013 Paraskevi Theofilou, Adamantia Aroni, Maria Ralli, Maria Gouzou, Sophia Zyga

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
 
© PAGEPress 2008-2017     -     PAGEPress is a registered trademark property of PAGEPress srl, Italy.     -     VAT: IT02125780185