Choosing 360: A Guide to Evaluating Multi-rater Feedback Instruments for Management Development. John Fleenor W.
Читать онлайн книгу.
individual feedback settings. These are instruments that have not been developed or tested for other purposes—such as making selection or promotion decisions.
STEP 4:EXAMINE THE FEEDBACK SCALES
In evaluating individual instruments, you should begin by examining the scales on which feedback will be received. Are you comfortable with what it measures?
There is a detailed discussion of scales in step 6, but what you need to know at this point is that the scales are made up of several items on the instrument and represent the content or competencies on which managers will be evaluated. Each individual scale represents a slice of managerial work (for example, planning) or a single kind of competency (for example, decisiveness); as a whole the scales provide a portrait of leadership or managerial effectiveness. Using the sample feedback you have obtained, you should consider the following when looking at the scales:
• Is your organization wedded to a particular way of representing what it takes to be effective in your business or do you have a particular model underlying management-development efforts?
• Does the range of scales fit with what you see as relevant competencies for managers in your target group?
• Does the number of scales seem reasonable?
• If, in your judgment, an instrument does not have enough scales that seem relevant to your target group, or if it has too many that seem irrelevant, drop it from further consideration.
STEP 5:FAMILIARIZE YOURSELF WITH THE INSTRUMENT-DEVELOPMENT PROCESS
In order to know how to identify quality instruments, you must understand the basics of sound instrument development.
The development process can be seen as occurring in four stages:
• developing instrument items and feedback scales,
• assessing reliability and validity,
• designing the feedback display, and
• creating supporting materials.
At each stage different issues are being addressed.
When items and scales (fully defined in step 6 below) are being developed, the author must identify, as much as possible, the full range of behaviors or skills that he or she believes represents management or leadership competency. Another question at this stage is whether items of behavior or competency cluster in groups that are internally consistent, distinct from each other, and useful for feedback purposes.
To assess reliability, the author of an instrument must consider whether the measurement of these skills or competencies is stable in a variety of ways. To assess validity, the author must determine whether the scales really measure the dimensions they were intended to measure and whether they are related to effectiveness as a manager or leader. Because 360-degree-feedback instruments are primarily intended for individual development, the question of whether the areas assessed can be developed also must be considered.
When designing feedback, the author should try to maximize the manager’s understanding of the data to enhance its impact. In creating supporting materials, the aim of the author is to help the feedback recipient gain deeper understanding of the theory or research behind the instrument and thereby enhance the ability to interpret and work with the data. Your task as an evaluator is to assess the work completed in each of these four stages and balance what you find against the needs of your target group.
STEP 6:LEARN HOW ITEMS AND FEEDBACK SCALES WERE DEVELOPED
Instruments that assess managerial competence or leadership effectiveness are dealing with complicated phenomena. These phenomena cannot be adequately represented by a single behavior or characteristic because they are comprised of many closely related behaviors and skills. To adequately measure these complex capacities, instruments must have scales that are made up of several items.
The process of instrument development typically begins with the writing of items that represent behaviors or characteristics believed to be related to effective management or leadership.
Items can come from a variety of places. Sometimes the author refers to a theory (leadership theory, theory of managerial work, competency models) to develop specific behavioral statements or statements describing characteristics or skills. At other times researchers create descriptions of characteristics or skills based on data they have collected. Another way items can be written is by basing them on the organizational experience of the author(s). People who frequently work in training or consulting with managers may feel they can capture in a set of statements the essence of the leadership or management effectiveness they have observed.
The better instruments tend to be those that have used a combination of approaches in their development. A basis in theory provides an instrument with a set of validation strategies, while empirical research can provide data from working managers. Ultimately, the quality of the final product depends on a combination of the quality of the theory, research, and experience of the developer; his or her skill in translating theory, research, and experience into written items; and the attention paid to instrument development and feedback design. A complete evaluation on your part will reveal the level of quality at all these stages.
The nature of items can vary, regardless of their origin. Items can be phrased behaviorally (for example, “Walks around to see how our work is going”), phrased as skills or competencies (for example, “Is good at influencing the right people”), or phrased as traits or personal characteristics (for example, “Is highly motivated”).
Instrument feedback is usually presented to the target manager as scores on scales (groups of items). Because scales tend to be more abstract than items (for example, “Resourcefulness”), it may be difficult for target managers to set goals for change based on this type of data. To help managers process their data, some instruments provide scores on the individual items that comprise these scales.
Feedback on behavioral items may be easiest for managers to use in setting goals for change because they are the most concrete. Behavioral changes are the easiest for co-workers to see as well. Change on this type item, however, can be the most superficial in terms of enhancing personal development. At the other extreme, feedback on characteristics such as motivation can be the most difficult to use, and change on this type item can be the hardest to observe. But change on these items may be more likely to enhance personal development. Feedback on specific skills probably falls somewhere between these two extremes: It is moderately easy to use when changes are observable and it involves some real skill development.
The items discussed above are good examples. If one receives a low score on a behavioral item such as “Walks around to see how our work is going,” it will be relatively easy to change (that is, “Walk around more”) but will probably lead to little in the way of personal development for that manager. If one receives a low score on a skill-based item such as “Is good at influencing the right people,” it will be harder to change, because the manager will have to find out how to become better and then will need to improve. But the result can be important skill development. Finally, receiving a low score on an item such as “Is highly motivated” can be the hardest of all to change. Change will require the manager to reflect and discover why motivation is low, and to decide what it will take to feel more motivated. Then the manager will have to make whatever personal or life changes are necessary. This kind of change, however, can be the more developmental.
If individuals are left on their own to process feedback (no trainer or facilitator is available), or if an instrument is not accompanied by comprehensive interpretive and development materials, the clarity of item content is critical. The harder items are to interpret, the more difficulty managers will have in benefiting from the feedback and the more important the quantity and quality of support becomes.
Once items are created, instrument development proceeds to the task of constructing the scales on which feedback will be given. Multiple items are grouped together to