We are in the process of developing a core codeset to serve as a first pass through the interviews we have collected. Making them consistent and useful across the entire collaboratory is not a trivial task. Interrater-reliability is difficult enough within a small group, but with a much larger, multi-site, mutli-domain research group this is exacerbated. We basically decided to not worry about traditional interrater-reliability, but to think about coding as a flag of attention for particular topics. So we created a protocol that will be somewhat consistent across the sites and still provide access.

Rather than have each site code based on an emergent coding system or codes and then try to merge them or find cross-linkable codes between the sites, it was decided that having a lighter codeset would lower the barrier to entry and support integration across sites. Plus, using NVivo’s collaborative tools has not been as supportive as we would have hoped. The light coding scheme, we started with 7 codes and have now increased to roughly 27, which is do-able, and an NVivo protocol that filters by the person who created the code to maintain some level of data hygiene for the central data repository.

Codes need to fit the interviews and how we view the research questions, but also serve as useful retrieval points for our research directions. The rubber will hit the road on this when we actually try searching across the coded passages and see if we can answer our questions using these codes. This is a particular concern when the tool wasn’t really designed for collaborative work, and using more features and making it more central in our work will further tax it and reveal shortcomings.

Lesson: Don’t put too much faith in the tool – don’t lean on it too hard.