The reliability test on DiVoMiner provides a comprehensive process, starting from the preparation phase and progressing through the entire reliability testing procedure. This ensures that users can systematically conduct reliability test on the platform, with each step being transparent and traceable, ultimately leading to the completion of the entire testing process.
Below is the specific procedure for calculating the reliability:
After completing the reliability coding, users can view the reliability coefficients immediately.
After all coders have completed the test coding, in the [Reliability Calculation] section, select the coders and reliability index. Select the “Bot” option in coder if you like to view the results of machine coding, then click “Calculation” to obtain the reliability results. The platform currently offers four commonly used reliability index for everyone to use!
In addition, the DiVoMiner features with [Calculate the co-coded data only] function, which is a perfect solution to the problem of inconsistent progress in coding test among coders while researchers need to view the current reliability results.
What is the ideal level of reliability? In academic discourse, there is no definitive consensus on the rigid standards for reliability. Reliability varies depending on research requirements and category difficulties (especially when subjective categories present significant challenges).
Utilizing [Coding Tracking] for coding corrections
If the inter-coder reliability reaches an acceptable level, formal encoding can commence. If the reliability coefficient is not ideal, the coding test need to be re-conducted until the reliability coefficient reaches a credible level.
TIPS:
For data items with low reliability coefficients, it’s necessary to retrain the coders before re-conducting reliability test.
The method for rebuilding the test library is, firstly, click [Empty Test Library] in the [Test Library] section, which will clear all coded results. Secondly, create a new [Test Library] in [Coding Library], then each coder can re-do the coding independently in the new test library.
