Validity analysis refers to how well a tool or method of measurement can accurately measure the intended things. The higher the agreement between the measurement results and the content being measured, the greater the validity; conversely, the lower the agreement, the lower the validity. There are three types of validity: Content validity, Criterion validity, and Construct validity.
Content validity evaluates the relevance and representation of test items to measure the desired behavior or content. It is established through expert judgment and empirical evidence.Criterion validity assesses the meaningfulness of the data obtained from the scale by comparing it with values from other selected variables. It is measured using correlation analysis or significant difference tests, but finding an appropriate criterion can be challenging.Construct validity examines the extent to which the measurement results reflect a specific structure and corresponding values. This is evaluated using factor analysis, which extracts common factors from all the variables in the scale. The cumulative contribution rate, common degree, and factor loading are used to evaluate the validity of the structure. The cumulative contribution rate reflects the effectiveness of the common factor, the degree of commonality measures the effectiveness of the original variable, and the factor loading shows the correlation between the original variable and a common factor.
On this platform, factor analysis is employed to assess the structural validity of questionnaires. However, before conducting factor analysis, it is essential to perform the KMO test and Bartlett’s sphericity test to assess the suitability of the original variables for factor analysis. A KMO value greater than 0.9 is highly appropriate, 0.8-0.9 is appropriate, 0.6-0.8 is mediocre, 0.5-0.6 is not ideal, and less than 0.5 is extremely unsuitable. The rejection of the null hypothesis in Bartlett’s sphericity test indicates that factor analysis can be carried out, while the acceptance of the null hypothesis implies that the variables may be independent and not suitable for factor analysis.
Indicator description:
KMO value: often used as a measure to determine if a dataset is appropriate for factor analysis. A KMO value greater than 0.6 is generally considered acceptable.
Bartlett’s sphericity test: If the p-value is below 0.05, it indicates that the dataset is appropriate for conducting factor analysis.
Characteristic root: A measure is utilized to automatically determine the most appropriate number of factors to extract, usually starting from a minimum of 1. Nevertheless, users have the option to manually specify the number of factors they desire to extract.
Variance Explanation Rate: A weight value can be used to represent the amount of information that is extracted by a factor.
Cumulative Variance Explanation Rate: An indicator is commonly used to represent the amount of information extracted by all factors, and it typically has a value higher than 50%.
Varimax rotation method (Varimax): The rotation method used by default is usually the maximum variance method.
Common degree (common factor variance): Factor loading is a measure often used to evaluate the significance of an item in factor analysis. It is generally considered low if it is below 0.4, indicating that the item may not be contributing meaningfully to the analysis. In this situation, the item may need to be eliminated from the analysis.
Factor Loading Coefficient: Factor loading is a frequently used measure to assess the relationship between the factors and the items in factor analysis. A high factor loading is indicated by an absolute value exceeding 0.4 and suggests that the item is likely to belong to the corresponding factor.
References：
- Hadi, N. U., Abdullah, N., & Sentosa, I. (2016). An easy approach to exploratory factor analysis: Marketing perspective. Journal of Educational and Social Research, 6(1), 215.
- Javid, J. (2015). Measurement Validity and Reliability. [PowerPoint Slides] SlideShare.net. Retrieved April 2023 from https://www.slidehshare.net/JonathanJavid/measurement-validity-and-reliability.
- Watkins, M. W. (2018). Exploratory factor analysis: A guide to best practice. Journal of Black Psychology, 44(3), 219-246.