Exploring Reliability

Exploring Reliability

Alexander Castillo

[Institutional Affiliation(s)]

Exploring Reliability

This paper emphasizes on the different types of reliability and validity that are used in the Values and Motives Questionnaire. Moreover, this will be also including the strengths and concerns that are related to the VMQ by framing an opinion about the test within the limitations of the available data. Furthermore, this paper will aim to convey information relating to the sample size, nature of the population, which can make an impact on the constructs being tested by the test.

Exploration of Reliability and Validity

The principle of validity and reliability is accepted by the researchers, and this is mostly used in the qualitative research methodology ADDIN ZOTERO_ITEM CSL_CITATION {"citationID":"m3y0mVF2","properties":{"formattedCitation":"(Roberts & Priest, 2006)","plainCitation":"(Roberts & Priest, 2006)","noteIndex":0},"citationItems":[{"id":1813,"uris":["http://zotero.org/users/local/F0XOCTdk/items/US8ZKUJA"],"uri":["http://zotero.org/users/local/F0XOCTdk/items/US8ZKUJA"],"itemData":{"id":1813,"type":"article-journal","title":"Reliability and validity in research","container-title":"Nursing standard","page":"41-46","volume":"20","issue":"44","author":[{"family":"Roberts","given":"Paula"},{"family":"Priest","given":"Helena"}],"issued":{"date-parts":[["2006"]]}}}],"schema":"https://github.com/citation-style-language/schema/raw/master/csl-citation.json"} (Roberts & Priest, 2006). Value Motives Inventory can be also related to the Values and Motives Questionnaire, which helps to examine, the motivation of individuals, concerning their daily based activities. The VMQ ensures a clear understanding of values by assessing three main areas, that include interpersonal, intrinsic, and extrinsic values. Intrinsic values are the combination of individuals' belief systems and personal behavioral patterns. According to ADDIN ZOTERO_ITEM CSL_CITATION {"citationID":"88LVK1mZ","properties":{"formattedCitation":"({\\i{}Values and Motives Inventory}, n.d.)","plainCitation":"(Values and Motives Inventory, n.d.)","dontUpdate":true,"noteIndex":0},"citationItems":[{"id":1807,"uris":["http://zotero.org/users/local/F0XOCTdk/items/R22DMWEK"],"uri":["http://zotero.org/users/local/F0XOCTdk/items/R22DMWEK"],"itemData":{"id":1807,"type":"webpage","title":"Values and Motives Inventory","URL":"https://www.psytech.co.za/images/PsytechSA/VMI/VMIMan.pdf","accessed":{"date-parts":[["2020",1,28]]}}}],"schema":"https://github.com/citation-style-language/schema/raw/master/csl-citation.json"} Values and Motives Inventory, (n.d.), interpersonal values emphasize the relationship concerns and links of an individual with the other members. Lastly, extrinsic values are those factors, which provide motivation at the workplace.

Based on the content available in the VMQ, two types of reliability are being discussed which include inter-rater reliability and internal consistency. Inter-rater reliability in the literature review of VMQ is used to portray an agreement over the same data by different individuals ADDIN ZOTERO_ITEM CSL_CITATION {"citationID":"o5dcoFD4","properties":{"formattedCitation":"({\\i{}Values and Motives Inventory}, n.d.)","plainCitation":"(Values and Motives Inventory, n.d.)","noteIndex":0},"citationItems":[{"id":1807,"uris":["http://zotero.org/users/local/F0XOCTdk/items/R22DMWEK"],"uri":["http://zotero.org/users/local/F0XOCTdk/items/R22DMWEK"],"itemData":{"id":1807,"type":"webpage","title":"Values and Motives Inventory","URL":"https://www.psytech.co.za/images/PsytechSA/VMI/VMIMan.pdf","accessed":{"date-parts":[["2020",1,28]]}}}],"schema":"https://github.com/citation-style-language/schema/raw/master/csl-citation.json"} (Values and Motives Inventory, n.d.). Another type of reliability includes internal consistency that focuses on the methods of engagement of each item being associated positively in scoring on the available scales. Researchers have utilized the Cronbach's Alpha coefficient for the measurement of reliability ADDIN ZOTERO_ITEM CSL_CITATION {"citationID":"piHusO3X","properties":{"formattedCitation":"(Heale & Twycross, 2015)","plainCitation":"(Heale & Twycross, 2015)","noteIndex":0},"citationItems":[{"id":1798,"uris":["http://zotero.org/users/local/F0XOCTdk/items/7M6PKQKP"],"uri":["http://zotero.org/users/local/F0XOCTdk/items/7M6PKQKP"],"itemData":{"id":1798,"type":"article-journal","title":"Validity and reliability in quantitative research","container-title":"Evidence-Based Nursing","page":"66-67","volume":"18","source":"ResearchGate","journalAbbreviation":"Evidence-Based Nursing","author":[{"family":"Heale","given":"Roberta"},{"family":"Twycross","given":"Alison"}],"issued":{"date-parts":[["2015",1,1]]}}}],"schema":"https://github.com/citation-style-language/schema/raw/master/csl-citation.json"} (Heale & Twycross, 2015). Furthermore, researchers have used sampling and construct validity in the VMQ. First type of validity is a sampling that ensures that a wide range of areas is being covered by the measurement in VMQ.

Areas of Concerns and Strength

The following section will be providing an understanding regarding the areas of concern and strengths that concerns to VMQ. Based on the data available in the VMQ, the range of co-efficient ranges from .52 to .83, while a correlation ranging between +/- .3 to +/- .5 are meant as moderate correlation ADDIN ZOTERO_ITEM CSL_CITATION {"citationID":"OakMPM77","properties":{"formattedCitation":"({\\i{}Values and Motives Inventory}, n.d.)","plainCitation":"(Values and Motives Inventory, n.d.)","noteIndex":0},"citationItems":[{"id":1807,"uris":["http://zotero.org/users/local/F0XOCTdk/items/R22DMWEK"],"uri":["http://zotero.org/users/local/F0XOCTdk/items/R22DMWEK"],"itemData":{"id":1807,"type":"webpage","title":"Values and Motives Inventory","URL":"https://www.psytech.co.za/images/PsytechSA/VMI/VMIMan.pdf","accessed":{"date-parts":[["2020",1,28]]}}}],"schema":"https://github.com/citation-style-language/schema/raw/master/csl-citation.json"} (Values and Motives Inventory, n.d.). A strong correlation is the one, which is above +/- .5, and more than .7 concludes that two constructs can be utilized interchangeably. Moreover, higher interest correlation means that co-efficient alpha will be also higher and these can possibly show errors in a particular problem. As constructs of VMQ and 16PF are different, so the validity of these require different constructs for the purpose to provide varying scores. When there is a lesser correlation coefficient (.5), then it is said to be invalid ADDIN ZOTERO_ITEM CSL_CITATION {"citationID":"067CKFgJ","properties":{"formattedCitation":"({\\i{}Values and Motives Inventory}, n.d.)","plainCitation":"(Values and Motives Inventory, n.d.)","noteIndex":0},"citationItems":[{"id":1807,"uris":["http://zotero.org/users/local/F0XOCTdk/items/R22DMWEK"],"uri":["http://zotero.org/users/local/F0XOCTdk/items/R22DMWEK"],"itemData":{"id":1807,"type":"webpage","title":"Values and Motives Inventory","URL":"https://www.psytech.co.za/images/PsytechSA/VMI/VMIMan.pdf","accessed":{"date-parts":[["2020",1,28]]}}}],"schema":"https://github.com/citation-style-language/schema/raw/master/csl-citation.json"} (Values and Motives Inventory, n.d.). Based on the VMQ and 16 PF correlation co-efficient, for privacy, it is -.41, and impression management counts with the correlation coefficient of .4. The constructs for the VMQ and opposition (OPP), indicates the two scoring data, which has a .7 co-relation co-efficient ADDIN ZOTERO_ITEM CSL_CITATION {"citationID":"WpytZqo9","properties":{"formattedCitation":"({\\i{}Values and Motives Inventory}, n.d.)","plainCitation":"(Values and Motives Inventory, n.d.)","noteIndex":0},"citationItems":[{"id":1807,"uris":["http://zotero.org/users/local/F0XOCTdk/items/R22DMWEK"],"uri":["http://zotero.org/users/local/F0XOCTdk/items/R22DMWEK"],"itemData":{"id":1807,"type":"webpage","title":"Values and Motives Inventory","URL":"https://www.psytech.co.za/images/PsytechSA/VMI/VMIMan.pdf","accessed":{"date-parts":[["2020",1,28]]}}}],"schema":"https://github.com/citation-style-language/schema/raw/master/csl-citation.json"} (Values and Motives Inventory, n.d.). This is the indication of the strong relation of scores, while the result is that .58 is the co-efficient of the constructs of social desirability and .73 for affiliations.

Sample Size and Nature of Population

While conducting research, deciding the sample size, and nature of the population are an integral part of a reliable research. Lower standard error means that there is a larger population being included in the research. The validity of the test is reduced when the value for the Standard Error Measurement is higher. This represents that scores differ from those analyzed in the total population.

The VMQ provides general information about the internal consistency reliability and Standard Error Measurement, whereas coefficient alpha is used for the measurement of the internal consistency ADDIN ZOTERO_ITEM CSL_CITATION {"citationID":"adL2IDFk","properties":{"formattedCitation":"({\\i{}Values and Motives Inventory}, n.d.)","plainCitation":"(Values and Motives Inventory, n.d.)","noteIndex":0},"citationItems":[{"id":1807,"uris":["http://zotero.org/users/local/F0XOCTdk/items/R22DMWEK"],"uri":["http://zotero.org/users/local/F0XOCTdk/items/R22DMWEK"],"itemData":{"id":1807,"type":"webpage","title":"Values and Motives Inventory","URL":"https://www.psytech.co.za/images/PsytechSA/VMI/VMIMan.pdf","accessed":{"date-parts":[["2020",1,28]]}}}],"schema":"https://github.com/citation-style-language/schema/raw/master/csl-citation.json"} (Values and Motives Inventory, n.d.). The overall range for the coefficients is noted as .5, but they all are comprised of varying ranges. The internal consistency is the source that fulfills the needs of a reliable instrument in the research ADDIN ZOTERO_ITEM CSL_CITATION {"citationID":"uZ7Xhgfb","properties":{"formattedCitation":"(Merk et al., 2017)","plainCitation":"(Merk et al., 2017)","noteIndex":0},"citationItems":[{"id":1801,"uris":["http://zotero.org/users/local/F0XOCTdk/items/GD9Z3QXY"],"uri":["http://zotero.org/users/local/F0XOCTdk/items/GD9Z3QXY"],"itemData":{"id":1801,"type":"article-journal","title":"The Motivational Value Systems Questionnaire (MVSQ): Psychometric Analysis Using a Forced Choice Thurstonian IRT Model","container-title":"Frontiers in Psychology","volume":"8","source":"PubMed Central","abstract":"This study presents a new measure of value systems, the Motivational Value Systems Questionnaire (MVSQ), which is based on a theory of value systems by psychologist Clare W. Graves. The purpose of the instrument is to help people identify their personal hierarchies of value systems and thus become more aware of what motivates and demotivates them in work-related contexts. The MVSQ is a forced-choice (FC) measure, making it quicker to complete and more difficult to intentionally distort, but also more difficult to assess its psychometric properties due to ipsativity of FC data compared to rating scales. To overcome limitations of ipsative data, a Thurstonian IRT (TIRT) model was fitted to the questionnaire data, based on a broad sample of N = 1,217 professionals and students. Comparison of normative (IRT) scale scores and ipsative scores suggested that MVSQ IRT scores are largely freed from restrictions due to ipsativity and thus allow interindividual comparison of scale scores. Empirical reliability was estimated using a sample-based simulation approach which showed acceptable and good estimates and, on average, slightly higher test-retest reliabilities. Further, validation studies provided evidence on both construct validity and criterion-related validity. Scale score correlations and associations of scores with both age and gender were largely in line with theoretically- and empirically-based expectations, and results of a multitrait-multimethod analysis supports convergent and discriminant construct validity. Criterion validity was assessed by examining the relation of value system preferences to departmental affiliation which revealed significant relations in line with prior hypothesizing. These findings demonstrate the good psychometric properties of the MVSQ and support its application in the assessment of value systems in work-related contexts.","URL":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5611709/","DOI":"10.3389/fpsyg.2017.01626","ISSN":"1664-1078","note":"PMID: 28979228\nPMCID: PMC5611709","shortTitle":"The Motivational Value Systems Questionnaire (MVSQ)","journalAbbreviation":"Front Psychol","author":[{"family":"Merk","given":"Josef"},{"family":"Schlotz","given":"Wolff"},{"family":"Falter","given":"Thomas"}],"issued":{"date-parts":[["2017",9,20]]},"accessed":{"date-parts":[["2020",1,28]]}}}],"schema":"https://github.com/citation-style-language/schema/raw/master/csl-citation.json"} (Merk et al., 2017). While the standard deviations get impacted but he unreliable and unequal distribution of scores in any test, but the deviation in the scores is acceptable and there is no as such high correlation in the test results.

For the research, researchers have selected those participants who tend to possess higher education ration as compared to other populations. This population includes students from psychology and MBA students, which portrays that study is based on a specific population ADDIN ZOTERO_ITEM CSL_CITATION {"citationID":"ZFGFZ9Sq","properties":{"formattedCitation":"({\\i{}Values and Motives Inventory}, n.d.)","plainCitation":"(Values and Motives Inventory, n.d.)","noteIndex":0},"citationItems":[{"id":1807,"uris":["http://zotero.org/users/local/F0XOCTdk/items/R22DMWEK"],"uri":["http://zotero.org/users/local/F0XOCTdk/items/R22DMWEK"],"itemData":{"id":1807,"type":"webpage","title":"Values and Motives Inventory","URL":"https://www.psytech.co.za/images/PsytechSA/VMI/VMIMan.pdf","accessed":{"date-parts":[["2020",1,28]]}}}],"schema":"https://github.com/citation-style-language/schema/raw/master/csl-citation.json"} (Values and Motives Inventory, n.d.). However, VMQ does not include any comparison of any specific culture and country. Specific cultures as an instrument are beneficial for the comparison of the demographics, and this ensures the inclusion of all populations. The selection of a specific population for the research means that not everyone is included to contribute to the research, and the results achieved by the VMQ do not allow us to deduce a generalized outcome or a result ADDIN ZOTERO_ITEM CSL_CITATION {"citationID":"y7JIzhTk","properties":{"formattedCitation":"(Taherdoost, 2016)","plainCitation":"(Taherdoost, 2016)","noteIndex":0},"citationItems":[{"id":1812,"uris":["http://zotero.org/users/local/F0XOCTdk/items/HAFXR3TL"],"uri":["http://zotero.org/users/local/F0XOCTdk/items/HAFXR3TL"],"itemData":{"id":1812,"type":"article-journal","title":"Validity and reliability of the research instrument; how to test the validation of a questionnaire/survey in a research","container-title":"How to Test the Validation of a Questionnaire/Survey in a Research (August 10, 2016)","URL":"https://www.researchgate.net/publication/319998004_Validity_and_Reliability_of_the_Research_Instrument_How_to_Test_the_Validation_of_a_QuestionnaireSurvey_in_a_Research","DOI":"DOI: 10.2139/ssrn.3205040","author":[{"family":"Taherdoost","given":"Hamed"}],"issued":{"date-parts":[["2016"]]}}}],"schema":"https://github.com/citation-style-language/schema/raw/master/csl-citation.json"} (Taherdoost, 2016).

Summary

The overall values used in the VMQ look interesting and this includes the assessment of value systems of individuals and their subjective feelings, which cannot be empirically proved. I believe that the inclusion of various cultures and populations could have provided a reliable result for generalizability. The inclusion of more population groups will be beneficial to provide a wider range of information. Nevertheless, the reliability and validity of the instruments used in the VMQ cannot be rejected because this connects to a concern for conducting research based on higher co-relations.

References

ADDIN ZOTERO_BIBL {"uncited":[],"omitted":[],"custom":[]} CSL_BIBLIOGRAPHY Heale, R., & Twycross, A. (2015). Validity and reliability in quantitative research. Evidence-Based Nursing, 18, 66–67.

Merk, J., Schlotz, W., & Falter, T. (2017). The Motivational Value Systems Questionnaire (MVSQ): Psychometric Analysis Using a Forced Choice Thurstonian IRT Model. Frontiers in Psychology, 8. https://doi.org/10.3389/fpsyg.2017.01626

Roberts, P., & Priest, H. (2006). Reliability and validity in research. Nursing Standard, 20(44), 41–46.

Taherdoost, H. (2016). Validity and reliability of the research instrument; how to test the validation of a questionnaire/survey in research. How to Test the Validation of a Questionnaire/Survey in a Research (August 10, 2016). https://doi.org/DOI: 10.2139/ssrn.3205040

Values and Motives Inventory. (n.d.). Retrieved January 28, 2020, from https://www.psytech.co.za/images/PsytechSA/VMI/VMIMan.pdf