| Terms |
Definitions |
|
conceptual criterion
|
overarching, subjective, verbal statement of the definition of success
|
|
instrumentation threat
|
when instruments (or raters) are systematically varying with a condition
|
|
differential attrition
|
when different kinds of people drop out from the conditions such that the composition of the groups is no longer equivalent
|
|
active learning
|
broad approach to training, emphasizes (1) active exploration, (2) error encouragement, and (3) emotion control strategies, see (Bell & Kozlowski, 2008)
|
|
metric invariance
|
when the weightings of these correlations are the same between raters; e.g., rater 1 and 2 agree that the dimensions of Construct 1 are ranked B, A, C
|
|
measures of effectiveness (MOE)
|
outcomes that theoretically/logically contribute to the perception of the conceptual criterion
|
|
halo error
|
a \"general impression,\" where a rater rates a ratee similarly across all dimensions
|
|
true halo
|
when there is real correlation between the constructs (see Viswesvaran et al 2005)
|
|
configural invariance
|
when the pattern of correlations of dimenions to higher-order constructs are consistent between raters; e.g., rater 1 and 2 agree that Construct 1 consists of A, B, and C
|
|
personal construct
|
an individual's mental model of a given concept
|
|
implicit trait policy
|
an individual's belief that certain traits are most effective in certain situations
|
|
2. Closed-ended questions
|
- Possibilities chosen in advance. - Yes/no or True/false.- Categorical or multi choice (mult ans or choose one)- Likert (scale usu 1-5, usu disagree vs agree)- Usefal data with right ques; MUST come up w/ all an or 'other' category becomes open-ended. - Complete w/ outside help.
|
|
criterion relevance
|
the degree to which what you measure is actually related to the criterion
|
|
general factor
|
similar to the concept of true halo, when shared variance occurs between factors that is not due to measurement error but because of a higher-order latent construct, see Viswesvaran et al 2005)
|
|
training needs analysis
|
when you look at person, task, and organization to ensure that the training will train the appropriate things (task), the right people (person), and that the organization is conducive to the training
|
|
D. Biased sampling1. refusal/response rates2. limiting factors
|
Not representative. 1) May be systematic ex. lazy people do not want to respond to surveys; monitor reason for not responding; low response rate requires reflection. 2) Where list acquired. Ex. People only using cell phones (no directory), problem for polic polling, no phone service, people at work, people with caller ID, newsweek list (Liberal magazine) vs US News and World Report (Conserv). Location.
|
|
Total Variance in Y
|
Percent Variance in Y Accounted For by X (of total, 100% variance, how much differing, explain/predict) Not nec percent of people but discrepancies. PRE btwn 0 and 1.
|
|
II. Calculating Probability A. Calculating single event probability
|
A) p(successful outcome A) = # possible successful outc/# possible outc. Ex. complete deck with jokers has 54 cards and 13 diamonds.
|
|
III. Calculating Correlations: Pearson’s rVariance:
|
- Integral or ratio data (ordinal, nominal, diff types of stat) - Var: How var x varies across distrib. (Σ (X – M) (X – M)) / (N)
|
|
C. Constructing good survey questions1. using unbiased language in your questions
|
(Valid/reliable; adequate rep; do not unintentionally push for a certain ans) 1) ex. push polls (ans in ques, soc desirability; word choice: Control, Unfair; labeling)
|
|
behavior modeling training (BMT)
|
consists of (1) defining the behavior, (2) providing models, (3) giving practice opportunities, and (4) providing feedback.
|
|
Error variance invariance
|
when raters have the same amount of measurement error variance; error variance variance is not a threat to validity as long as you have configural and metric invariance (Woehr 2005)
|
|
moderators of BMT
|
BMT is most effective when: (1) mixed models are presented, (2) trainee generated scenarios, (3) superiors are trained as well, (4) trainees instructed to set goals, and (5) rewards/consequences for transfer (Taylor et al 2005)
|
|
2) Expressing significance:
|
p is less than versus equal to (rules changing); p is less than 5% chance for errors indicates nothing about importance; never use 'significance' unless statistical.
|
|
II. Cautions and Pitfalls with Correlational and Differential DesignsA. Correlation does not equal causation
|
- Cannot usu word Effect/Affect (assoc, predic). 1) The third var problem; may be multiple 'third' variables. Cannot det diff. in correl research. A third variable, C, causes both A and B, and therefore makes A and B look like they’re directly related. May not know how to measure "C"; particular confluence of third var perhaps.
|
|
Regression/Prediction Part III. Error in PredictionA. Estimating and InaccuracyB. Sum of Squared Errors: Error in comparison to the regression line
|
A) Other factors involved, will be wrong to some extent, measurement error, if y is off, a is off, etc.B) Random variation; 1) HOW MUCH error? 2) ACCURACY of predic. (Predic vs actual values of y?)
|
|
Calculating Correl: Pearson's r (integral or ratio data) Covariance:
|
Σ (X – Mx) (Y – My)-------------------- Nx and y varying across distrib & relation btwn x and y. How x and then y deviate around means.
|
|
Predictor Variable- Regression constant
|
(X) vs criterion var (y). - Baseline number (a); fixed value. Regression=Prediction.
|
|
2. selecting the appropriate comparison group3. confounds
|
2) Ex. Divorced parents vs not comparison group? (Due to number of parents, loss of parent, absence of father as a role model...?) Can Choose Several for multiple competing hypoth. Will not completely narrow down interferences. 3) Anything inteferring with int. validity, vary together. Just rep your construct? C may actually cause B, tend to occur together. But Variable A and Variable C tend to occur together (they are related to each other, and thus confounded)When we are predicting from Variable A to Variable B, if we haven’t measured Variable C and included it in our model, we can’t rule out that B was actually caused by C, not by A.Ex...news studies often Not experimental research.
|
|
B. AttenuationC. Rank ordering, not absolute stability
|
B) Reduc in Magnitude due to reliability in measures, specifically. Error is random noise on construct: imprecise; unreliability reduces correl. Must improve measures, same outcome as restric in range. C) Esp for longitudinal: stable overtime? Not indicated by correl: only rank ordered relative to mean; are partic varying together? Do not knowo if partic are staying the same. All partic may dec in reaction time at same rate relative to mean, for ex, with a high correl. Must interprete as changing the same way; ex. height stable relative to peer group vs not growing. Not absolute stability through correol.
|
|
Multiple Regression Equation
|
Ŷ = a + b1X1 + b2X2 + b3X3...Ex height, age, gender predict weight. Multiple IVs for one DV (Limit IVs by sample size, usu 2-4) Simulataneous equations for one line, calc b diff than simple reg. *Diff reg coeff per var but still have intercept. Explains more of total var but Overlap; x1 and x3, x2 and x2, etc.
|
|
exercise or trait factors
|
when the dimensions in an AC load onto either exercise or dimension/traits
|
|
C. Choosing a sample1. nonprobability or haphazard sampling
|
1) Ex. first 40 person at the location chosen. Fast. Good setting yields ok results. Interested population responding. Time, excluding people in various ways. Region; over/ underestimating?
|
|
III. "Developmental" DesignsA. 1. Cross-sectional studies2. cohort effects
|
- Change over time, for Any type of data collec vs just surveys. A. 1) Different ages at the same time (10, 15, 20 on Oct 30), cross sec of ages (IV), compare for DV. Cannot prove but make inferences; assoc; form of differential research. 2) Born at same time; diff societal experiences may result in psy diff. Ex cold war and nuclear war risks (age not the cause but time period) Differential experiences coming with ages.
|
|
3. avoiding response sets4. making room in your scale for a variety of opinions *Give permission to ans negatively.
|
- Keeping ans in same way w/out reading carefully. -> REVERSE items: shift from high to low.- People do not like extreme ans, give room to be honest w/out feeling bad. Likert: most in middle, yes/no (do not want to look bad, not good for soc desirability ques, ex. I would cheat...)
|
|
B. Practical applicationsII. Basic Concepts in RegressionA. Bivariate prediction
|
Have x, only score y -> to estimate y. Ex grades in college and grades in college; univ. use x to predict y based on previous data from Both var (have equation) II. A) One IV, one DV, collect data on both to figure out how to describe scatter of data. Fit regression line to scatter plot; line not going through origin; generally nonzero y-int/
|
|
Stanford Prison Study- Are effects ok if positive? - Is it possible to have no effects?
|
- How to brief prisonders? Unantic risks: standard debriefing may help.- Responsibilities of other citizens? Ethically?- Adverse event: must contact IRB w/in 24 hrs for help.- Today: asking about childhood is risky. -(Just a coping mechanism?) - No efects: Yes, for some types no long term effects; but manip emot well-being; ellicit emotions. Must simply best protect (cannot eliminate) risks, IRB for more possible risks, balance. Difficult to give control to partic, few guidelines for guards. Must remain in control of exp.
|
|
D. Measures of Error and Prediction1. Multiple R
|
(similar to PRE) 1) Correl btwn all of IVs and DVs at same time; multiple correl. Always POS, from 0-1, the higher the better, NOT DIREC.
|
|
III. Computing Regression [For every unit of change in x, how many units of change for y] (b)A. Raw score regression equation
|
b = rxy (SDY/SDX)b is the correl btwn x and y times SD of Y div by SD of x. Takes into account variability of Y and X. Correl go through origin but it becomes adjusted. b is the sum of the dev products for x and y and the denonminator is the sum of the dev for X.
|
|
2. using neutral toned statements or questions
|
- Ineffec ques vs better. - Shocking? Need to disagree? Ex. selfish only children? (Neg) Just as altruistic? (pos). No rapport: can all go towards one end of distrib, Bad for statis/study.
|
|
Error when predicting Ŷ = MY (subscript y) (everyone scores at mean)
|
Error = Y – MYError2 = (Y – MY)2SStotal = Σ (Y – MY)2(If we guessed around mean, how far off? Sum of dev scores for y, squared)
|
|
B. Multiple event probability1. Probability of either A or B occurring2. Probability of both A and B occurringClass examples:
|
1) p(A or B) = p(A) + p(B). Must be indep where one will not affect the likelihood of the other vs Diamond and a three. 2) p(A and B) = p(A) x p(B). Ex two aces in a row, assuming REPLACEMENT to restore original condition. Ex: Not EXPected rel Freq (long run expec freq) but rel freq (real number of occurences).
|
|
Error when predicting Ŷ = a + bX (everyone scores on the regression line)Error =
|
Error = Y – Ŷ, Error2 = (Y – Ŷ)2; SSerror = Σ (Y – Ŷ)2- Everyone scores on the reg line; Predic value of y differs in each case; For every value for x, y hat changes. Compute y hat for every data point. Sum to zero (all errors), square, SSe, (How far off? How good an est is line? Not standardized) Relative error: vs not knowing but approx know y; compare to mean.
|
