Measuring and Analyzing for Results

Definition: Understanding of key research methods including measures, sampling theory, experiment design and statistical analysis in order to design and implement an evaluation process on an intervention.

You can’t manage what you can’t measure

Peter Drucker

Sample Work

Intended Measure vs. Actual Measure: This comparison visual came from a presentation we conducted in ORGD 6050 where we analyzed a survey and made recommendations. I elected to review my organization’s staff engagement survey to determine if the questions included were really measuring what we intended to measure. And, provide enhancements to capture more valid and actionable data. The full report can be found here: Survey Improvement Project

Experiment Design: The above figure was the research design I proposed for a project in ORGD 6050. Given that my organization was dealing with concerns about turnover, I was curious to see if components of workplace engagement (vigor, dedication and absorption) were actually predictors of turnover intention. This proposal project was helpful in clarifying how to craft a research question, how to look at the relationship between different measures and to consider how to best measure that relationship. The full Survey Research Proposal can be found in the embedded link.

Multiple Regression Analysis: The regression analysis data was run on a survey sample we created for our ORGD 6050 class. The survey was comprised of multiple constructs aggregating all of the variables based on a variety of research proposals. The results of the regression analysis confirmed two of our hypotheses. Prior to this report I had not completed a regression analysis or other statistical tests. I found the process fairly straightforward, given our clear instructions, supportive texts and fantastic support from Dr. Brooks. The full Survey Data Report can be found by clicking on the embedded link.

EMOD Experience

I must admit that this portion of the EMOD outline, when we reviewed it at orientation, gave me the most angst. I interpreted this competency and particularly ORGD 6050-Data Collection and Feedback (somewhat correctly) as statistics veiled in language to make me feel better about taking statistics. What I found upon opening the box into the world of measurement and analysis is that I truly enjoyed the learning about and applying the various techniques, tools, methods for collecting, analyzing and feedback information to a client. So much so, that I recall asking Dr. Brooks for some additional information about the Protestant Work Ethic Construct and I found myself so enthralled with the article, and giddy that I was able to understand the charts and analyses that I could not help but share it with my whole team.

In-Class Work: The first assignment that took us on a deep dive into measurement tools occurred in ORGD 6050 with Dr. Brooks. The assignment was a survey analysis and improvement project where we reviewed a survey we were familiar with and made recommendations based on our textbooks, producing a Survey Improvement Project (Paper) and a revised Survey with Improvements. I elected to use my organization’s staff engagement survey, having intimate familiarity with the survey as the project manager for that project the year prior. It may have been an audacious choice to select a 56 item survey prepared by a survey vendor as my first attempt at making recommendations for improvements, but I approached the assignment with gusto and found myself getting energized thinking about how to create questions that might provide the data we were looking for as an organization. My biggest learning in this assignment was the importance of designing questions to be good measures (i.e. reliable and valid) of whatever it is one is trying to measure.

This deep dive into a survey tool supported several assignments including our Diagnostic & Dialogic team assignment in ORGD 6040, as well as our current capstone. For the Dialogic and Diagnostic team assignment we explored the interest level and feasibility of an EMOD Alumni cohort. I was on the Dialogic team and we elected to take an Appreciative Inquiry approach to our interviews with key stakeholders including alumni, current students, faculty and community members. As we designed our Dialogic Team AI Interview Guide, we carefully crafted the questions by role recognizing that different stakeholder groups have different experiences with the EMOD program. As such, we wanted to ensure our questions provided opportunity to explore that experience – in the style of the Dialogic frame – and yet strike a balance with questions that would provide information we could codify into themes related to our initial question of interest and feasibility of an Alumni group. In our Dialogic Team Final Paper we took many of the lessons we learned from our assignments and text in Dr. Brooks’ class into our work in ORGD 6040. A really important lesson from ORGD 6050 was how to provide a quantitative analysis with qualitative data which is referenced in our Results section in the paper. In the end we were able to extract a lot of data from both our interview and intervention processes and the qualitative data that we gathered proved to be supportive of the more quantitative data that the diagnostic team discovered through their survey process.  

The Dialogic Team assignment from 6040 has influenced our capstone project as we think about our data collection process, approaching it from both a dialogic and diagnostic approach. A subgroup of our team working with TAF created an interview protocol and surveys for various stakeholder groups. Meanwhile, we are creating a strategic planning session that will both support one of our intended outcomes for the client, as well as serve as a dialogic process for gathering additional data about the organization including but not limited to; the organization’s perception of internal strengths and weaknesses, environmental influencers, who they see as their core customer, and how effectively they communicate during the process. We have several team members who will be supporting the event facilitation which will allow us multiple perspectives to take in all of these various data points and analyze them to get a more holistic picture of the organization.

Two additional assignments that we conducted in ORGD 6050 that supported my understanding of how to set up a good research question and then how to conduct the experiment and analyze the data were our Survey Research Proposal and Survey Data Report Team Project. As we were designing our research proposals, I wanted to connect my research to turnover intention given that it had been a perennial concern for my organization. When we began to discuss different constructs to include in our collective class survey, I found myself wanting to look at the data every which way. However I kept myself focused on those measures related to things that an organization might be able to have influence over, either through the hiring process or through organizational support mechanisms. Ultimately a small group of us selected my research proposal looking at three dimensions of engagement (vigor, dedication and absorption) and added an additional construct (psychological safety) to our analysis. During the assignment I was so steeped in the language of constructs and statistical tests that I felt very comfortable analyzing the data and speaking to the report we produced. I do recall that Dr. Brooks provided a significant amount of support in selecting the appropriate statistical test for our study, as that was not an area where I felt confident. Once we received the data however, I felt comfortable interpreting it and making recommendations for how that data might influence my work, for example recommended changes to our staff engagement survey or interview process. Now having some distance from the class and insufficient practice, I would need to revisit our textbooks in order to feel comfortable putting together another research proposal and selecting the appropriate methods to analyze the data, but I have comfort with the core concepts and know where and when to employ trusted resources to do so.

Application Outside of Class: While I have not conducted a full scale research project outside of our classwork, I will say that the terms, tools and methods that we have explored through the program do influence my work and life. Having this exposure has caused me to be more critical of data, being curious about the sample size, when data was collected, what was filtered out, etc., before accepting the analysis at face value. Additionally, when reviewing or creating surveys, I am analyzing multiple aspects of the questions, rating scale, instructions, method of delivery, etc. to ensure that the method of data collection will produce more valid and reliable data. Additionally, our OD team at my current employer recently completed a certification with KornFerry to deliver their 360 results. KornFerry is very much a research focused organization and they provided a number of statistical charts and data points as supplementary information. I found that based on my level of knowledge, I was able to ask questions about the information that helped to increase my competence and confidence in the data and reports we will be sharing with the organization.