Scenarios



Ed 510 Applications of Educational Research
return to the course schedule
Antonia D'Onofrio, Ph.D.

Introduction

You are going to spend the semester with Mary Q. Peoples, Ed. D., an educational psychologist and faculty member at Superior State University, School of Education.  Dr. Peoples studies the relationship between personality variables and leadership.  Recently she has turned her interest to the influence of personality and learning on leadership.

 
Dr. Peoples has been granted a large sum of money by the commonwealth to design, conduct and evaluate workshops targeted at public school principals.  The workshop curriculum, a series of 10 inservice meetings, integrates lessons that reinforce leadership skills with personal reflection and experimentation.  Dr. Peoples has crafted a number of units and lessons in which principals will take stock of their predominant personality traits as problem solvers and try to understand how the decisions they make are influenced by personality.  Principals who attend will develop many action plans in which they simulate school decision making.

 
So much for the curriculum.

 
Because this is a new line of investigation for Dr. Peoples, she has never studied principals before, the evaluation of the workshops confronts her with specific problems.  In fact, she is in many ways in the same situation as a beginning researcher.  Of course Dr. Peoples has published many studies in the past, but this is a new area for her.  In spite of her prior knowledge of research technique, which she can apply in this new area of research, Dr. Peoples really is starting anew.

 
When you read these scenarios, put yourself in her position.  Think with Dr. Peoples about the design decisions she must make.  Then go to the Internet readings listed on the course schedule and consider how they could be applied to solving some of Dr. Peoples problems.

 
The scenarios take you from Week 1 through Week 12. These scenarios are designed to stir you into thinking about possibilities.  They do not have all the information you need to converge on a correct answer.  That is not their purpose.  Their purpose is to have you think broadly about possible answers and to motivate you to read the internet selections that will help you to fill in the blanks.   Remember to read the Internet selections that are listed for the week in question.  In class we will continue the task of solving Dr. Peoples' research problem.  At the end of a class you will achieve more closure and have a better idea of what Dr. Peoples needs to do.

 
Week 1

 
Dr. Peoples brings past experience to her research investigation.  She has already studied the relationship between personality, learning and leadership for several years.  That means that she has accumulated many copies of articles and chapters, as well as scholarly papers, in these three areas.  She has read important work completed by other experts in this area.  Her problem now is threefold.  How does she organize what she does know and the published resources that she already has?  How does she focus the relationship between personality and leadership so that it fits a population that is new to her, public school principals?  How does she add to her understanding of basic literature which connects personality variables to leadership variables so that her focus in contemporary and up to date?

 
Because this is a new area of research she contemplates whether to conduct a quantitative study or a qualitative one.  She also has concerns about the evaluation.  She would like to be able to demonstrate that the workshop was effective - that it met its objectives.

 
Based on the internet readings and review of class notes, what do you think are possible options for Dr. Peoples?

 
Week 2
 

Dr. Peoples currently holds over 300 scholarly articles and papers on the subject of personality and learning and their relationship to leadership.  She refers to the topic as the PLL connection to save time.  Her most important priority is to organize the material she already has.

Dr. Peoples has been tempted to begin asking research questions based on her current level of understanding about the relationship between personality, learning and leadership.  However, Dr. Peoples also must add another dimension to the literature she already has.  That is her need to link what she knows to what she wants to study, a new population of publish school principals.  She must also find a way to apply this knowledge to a study of the workshop outcomes.

 
Her major concern at this time is to determine what the direction of her study will be.  She will use the literature she now has to guide her search for new literature before developing her research questions.

 
She decides that she will have to come up with two kinds of strategies to help her with her task  First she constructs a diagram of what she believes she can already say about the PLL connection.  Then she edits her map, looking for places where questions about the connection with public school principals might lie.  She next decides she will review literature on leadership and public school principals.  This is a good start.

 
What should she be using to make her review of the literature more efficient?  Should she be content to summarize the newly found studies, or are there several ways to examine that literature more efficiently and effectively?

 
Week 3
Dr. Peoples, like many of us at the beginning stages of research, is contemplating her problem in the abstract.  Personality, learning and leadership are concepts.  They are related according to much literature, in a number of ways.  But to develop a research design Dr. Peoples has to be much more specific about her meaning of personality, learning and leadership, respectively.  It is too soon to worry about measuring these concepts.  But it is not too soon to be thinking about how to make personality, learning and leadership more objective and therefore open to empirical study.

 
Dr. Peoples must think of ways to turn personality and learning and leadership into variables.  What are the steps that she will have to take to do this?

 
What are some ways to redefine personality and learning so that they can become measurable variables?
And what about leadership?  Doesn't Dr. Peoples have to go back and look at the curriculum of her workshop and base her measurement of leadership on the workshop outcomes?  What might leadership variables be called when they are considered workshop outcomes?

 
Week 4

 
Dr. Peoples believes that she has focused her investigation enough to begin writing a few research questions.

 
Question 1:  Are public school principals who exhibit reflective personality traits are more likely to test action plans that simulate school decision making than principals who exhibit active experimental traits.
Question 2:  Will both reflective and actively experimental principals who attend the PLL workshops include more categories of  transformational leadership in simulations of school decision making than a similar group of principals who did not attend the workshop series?
Try to engineer backward from these two questions.  How may Dr. Peoples have finally organized her literature and focused her literature review?  How might she have redefined the concepts of personality, learning and leadership so that she could get to the point to ask the two questions listed above?

 
Week 5

 
Dr. Peoples must tackle the problem of measuring her variables.  She decides to begin by measuring the outcomes of the workshop.  You recall that the principals who attend the workshop will be busy devising action plans that would simulate school decision making.  They will be exposed each week to lessons developed by Dr. Peoples that introduce the principals to the lessons of transformational leadership.  In these lessons principals will explore the benefits of communicating pedagogical values, personal accountability, communication skill and an experimental attitude toward

change.
 
Dr. Peoples concludes that she must develop a series of four local tests that will assess how much the principals will have learned about these four areas of transformational leadership.  She also decides that she will test the principals at the following levels of knowledge:  Information and Concepts, Principles and Rules, Application and Evaluation.  She builds a time measurement into her tests as well.  The items that test Evaluation are placed at the end of each test and because the test is taken on a desktop computer, this section can be timed.
How might Dr. Peoples begin the task of designing four objective tests that will evaluate the principals' workshop learning?

 
Week 6

 
Dr. Peoples must also devise ways to summarize the results of the local tests of transformational leadership.
Thus she decides to calculate measures of central tendency and measures of dispersion.  What will each of the following statistics describe, once Dr. Peoples has collected her data?

 
Week 7

 
It is natural for any researcher to wonder whether a newly designed test is providing credible information.

 
One question Dr. Peoples considers is whether scores received on any one of the individual areas of leadership learning (i.e., information and concepts, theories and principles, application and evaluation, respectively) are correlated with the other areas.  If they are correlated then each of the sub areas on the tests of workshop outcomes are statistically related.  Should that occur, Dr. Peoples can be more confident that all parts of an objective test are working together consistently.  Dr. Peoples will use the Pearson product moment correlation coefficient to see if that is the case.

 
How would Dr. People evaluate the statistical relationships between her assessments of the four areas of transformational leadership?

 
Another way to determine whether the items on a local test are good indicators of an overall score or measure of performance on the total test is to correlate the items one by one with total scores.  Dr. Peoples plans to do just that.  This procedure is often referred to as an item discrimination procedure.  Dr. Peoples expects that when individuals get any item on a test correct, they will probably also get most items correct, and therefore get higher total scores.  When individual items meet this expectation they are considered good discriminators of the total score.  Items with good discrimination are retained, poor discriminators are eliminated from the calculation of a final total value.  A nonparametric correlation is used to test item discrimination.  Your internet readings explain this procedure.

 
Week 8

 
Dr. Peoples has reached the point where she is ready to decide how to select principals for her workshop project.  There is a fair amount of interest among principals of local elementary schools in this project because the commonwealth has just decided that all professional school people must continue their professional education in order to keep their certificates active.  These workshops will generate a number of valuable of continuing education units, and the curriculum has practical value.

 
According to the contract that Dr. Peoples has accepted with the commonwealth, the first cohort will consist of principals drawn from all counties in the state.  Once an initial group of volunteers has been identified, then Dr. Peoples will select 50 individuals to participate.

 
Dr. Peoples has a number of sampling strategies to consider.  Some strategies depend on random selection and as such will provide a more statistically accurate view of workshop outcomes.  Other sampling strategies will provide a more realistic picture.

 
Your internet readings explore a number of sampling strategies and discuss the trade-offs that pertain to accuracy versus realism.  What are Dr. Peoples' choices?  How would she implement each one of them?  Are there some sampling strategies that are rather impractical?  After reading about these sampling alternatives, you should decide on the strengths and limitations of any of the choices that Dr. Peoples might make.

 
Week 9

 
Dr. Peoples has not yet explored the problem of measuring the two personality traits that she expects the principals will exhibit.  You recall that she hypothesized that reflective principals and actively experimental principals would respond differently to the simulation problems.

 
In her review of the literature, Dr. Peoples discovered that reflective and active experimentation are two personality dispositions that are related to how we learn.  These traits can be measured using an instrument known as the Personality Profiler (Golden and Golden).  According to the firm that developed this instrument, the Personality Profiler is both reliable and valid.

 
In reading the test manual, the document that describes the publisher's attempts to document statistical reliability and validity, Dr. Peoples will want to have a number of questions answered?

 

 
 
 
 
 
 
 
 
 

      • What questions should be answered about content validity?
      • What should she be able to find out about statistical reliability?
      • What information regarding statistical validity should be available to Dr. Peoples

 
 
Week 10

 
Dr. Peoples has already decided that there will be measurable differences in the time principals spend on testing their action plans.  If you recall, she hypothesized that reflective principals would spend more time testing their plans than would the actively experimental principals.  Dr. Peoples drew this conclusion in a preliminary way based on her reading of the literature on the subject of leadership. However her hypothesis is just a hypothesis.  As such it is open to statistical evaluation - that is a test.

 

 
 
 
 
 
 
 

When hypotheses are tested or  evaluated for the statistical likelihood that the predicted findings could not occur by chance alone.  Here are some important concepts:
 

What is the other alternative hypothesis?  What makes you think you are correct?

 
Week 11

 
When Dr. Peoples has gathered all her test scores together at the end of each testing period and at the end of the workshop series she will construct a data base of test scores.  This data base will consist of a number of scores.  There will be four scores based on the local tests that Dr. Peoples herself designed.  These scores are the outcome scores for tests of Pedagogical Values.

 
In addition Dr. Peoples will have an additional score, a personal score that indicates whether a principal is reflective or actively experimental

 
Did you think she will have a set of scores for only 50 principals, those who attended the workshop?
Not quite.  You will recall that Dr. Peoples was also going to use a control group.  She will also record the same five scores for another group, a group of 50 principals who did not attend the workshop but were part of the study.  This control group will be used to test Dr. People's second hypothesis.

 
Public school principals who exhibit reflective personality traits are more likely to test action plans that simulate school decision making than principals who exhibit active experimental traits,

 
Dr. Peoples will use a test of 2 independent samples.  Use your Internet readings to find out more about this statistical procedure.

 
In order to test the second hypothesis:
Both reflective and actively experimental principals who attend the PLL workshops include more categories of  transformational leadership in simulations of school decision making than a similar group of principals who did not attend the workshop series.

 
Dr. Peoples will also use a test of 2 independent samples.  But this time the principals who attended the workshop will be compared to those in the control group.

 
Week 12

 

 
 
 
 
 
 
 
 
 

Dr. Peoples wants to evaluate yet one more pattern in the workshop data that has not been part of her research plan to date.  She believes that she will have a stronger study if she systematically compares the control group performance to the performance of the workshop group.  She has personality scores for all members of the control group.  Thus she can make a number of different comparisons with the control group.
 
 

Group Reflective Actively Experimental
Workshop attendees Workshop outcome as scores Workshop outcome as scores
Control group member Workshop outcome as scores Workshop outcome as scores

 
 
Can you list as many as 3 separate comparisons between the control group and treatment group using only the Pedagogical values score?  What are they? What would they be for the other workshop outcome measures?

 
What is the advantage of comparing both the reflective and actively experimental workshop attendees to control group subjects who have been grouped according to the same personality traits?

 
Dr. Peoples will use a F test to compare four groups.  Your Internet readings will help you understand what she is going to do.

 
Page created January 4, 2001.  Page modified January 14, 2002.  Copyright - Antonia D'Onofrio - 2001/2002/2003.