747
1 IN THE UNITED STATES DISTRICT COURT
NORTHERN DISTRICT OF ILLINOIS
2 EASTERN DIVISION
3
ADAMS, et al., ) Docket No. 94 C 5727
4 )
Plaintiffs, )
5 )
vs. )
6 )
CITY OF CHICAGO, ) Chicago, Illinois
7 ) March 11, 1996
Defendant. ) 9:00 o'clock a.m.
8
9 VOLUME V
TRANSCRIPT OF PROCEEDINGS - Hearing
10 BEFORE THE HONORABLE JOHN A. NORDBERG
11
APPEARANCES:
12
13 For the Plaintiff: LAW OFFICES OF KENNETH N. FLAXMAN, P.C.
BY: MR. KENNETH N. FLAXMAN
14 122 South Michigan Avenue, Suite 1850
Chicago, Illinois 60603
15
16 For the Defendant: CITY OF CHICAGO
BY: MS. DARKA PAPUSHKEWYCH
17 MS. SHONA B. GLINK
MR. JAY KERTEZ
18 30 North LaSalle Street, Room 1020
Chicago, Illinois 60602
19
20 Court Reporter: MR. JOSEPH A. RICKHOFF
219 S. Dearborn Street, Room 1738
21 Chicago, Illinois 60604
22 * * * * * * * * * * * * * * * *
PROCEEDINGS RECORDED BY
23 MECHANICAL STENOGRAPHY
TRANSCRIPT PRODUCED BY COMPUTER
24
25
748
1 THE CLERK: 94 C 5727, Adams vs. The City of Chicago.
2 For hearing.
3 THE COURT: All right. Good morning all.
4 MS. PAPUSHKEWYCH: Good morning.
5 MR. FLAXMAN: Good morning.
6 THE COURT: Looks like a beautiful day outside,
7 anyway.
8 Are we ready to continue on now with Dr. Barrett's
9 testimony?
10 MS. PAPUSHKEWYCH: Your Honor, we have a couple of
11 preliminary matters in terms of the attempts to reach
12 stipulation on the exhibits.
13 THE COURT: Okay.
14 MR. FLAXMAN: I propose we take those up after we're
15 done with Dr. Barrett, because he is --
16 MS. PAPUSHKEWYCH: Well, we would like for plaintiffs
17 to finally rest their case in chief. And what has been
18 happening, your Honor, is that, despite repeated attempts over
19 the weekend to get some stipulations in this case, we are no
20 closer; in fact, I think we are going backwards in this.
21 THE COURT: Okay.
22 MS. PAPUSHKEWYCH: And, at this point, I think that we
23 need to address it, because we would like the plaintiffs --
24 THE COURT: I know, but --
25 MS. PAPUSHKEWYCH: Dr. Barrett --
749
1 THE COURT: -- unless he is able to come back at a
2 later time --
3 MS. PAPUSHKEWYCH: Dr. Barrett is here for the full
4 day, your Honor.
5 THE COURT: Right. So, it seems to me we ought to try
6 to finish his testimony.
7 MS. PAPUSHKEWYCH: We will. We would just like to be
8 able to determine where we are on some of the exhibits. It
9 will not take a long time.
10 THE COURT: I think as soon as we finish with
11 Dr. Barrett, that is what we ought to go to.
12 MS. PAPUSHKEWYCH: Okay.
13 THE COURT: Because, as I understand it, we have no
14 more live witnesses from the plaintiff. And, so --
15 MS. PAPUSHKEWYCH: That's what he has assured us.
16 THE COURT: Right. So, we need to go forward with the
17 balance of the evidence.
18 MS. PAPUSHKEWYCH: Okay.
19 THE COURT: All right.
20 Doctor, if you will come forward.
21 DR. GERALD BARRETT, DEFENDANT'S WITNESS, PREVIOUSLY SWORN
22 THE COURT: You remain under oath. Make yourself
23 comfortable; and, counsel, you may proceed when you are ready.
24 DIRECT EXAMINATION (Resumed)
25 BY MS. GLINK:
Barrett - direct
750
1 Q. I would just like to remind the Court and Dr. Barrett where
2 we left off on testimony, which, I believe, was the morning of
3 March 7th.
4 When we last had you here, Dr. Barrett, we were
5 talking about challenges and the appellate -- the appeals
6 process, the challenges to the test items. After the test
7 battery was -- test components were administered, was there any
8 mechanism -- and, I mean, when I say "the test battery," I
9 mean, the written job knowledge test, the in-basket simulation
10 and oral briefing exercise -- were there any mechanisms by
11 which an applicant can challenge any of the items on the test?
12 A. Yes.
13 Q. And on what basis could an individual challenge an item?
14 A. Any basis he or she wanted to.
15 Q. And why did you have such an appeals process?
16 A. Well, it's a way to be sure the test is fair. In other
17 words, if there was a mistake, items miskeyed or some
18 ambiguity, it's an opportunity for the testee to state his or
19 her case, and then the review panel will make a decision based
20 upon the facts.
21 Q. Is this sort of appeals process something that you have
22 some familiarity with in other tests that you've developed?
23 A. Yes.
24 Q. Did you receive any challenges to the written job knowledge
25 portion or test component of your test battery?
Barrett - direct
751
1 A. I recall we had, I believe, seven total challenges. I
2 think there may have been one challenge to the written test.
3 Q. When you say "seven total challenges," do you mean for all
4 three components of the examination?
5 A. I believe that was the in-basket and the written -- job
6 knowledge written test.
7 Q. Okay.
8 So, for the first two components of the test, you
9 received approximately seven challenges to items on those
10 examinations?
11 A. Yes.
12 Q. Can you describe for me the appellate process -- the
13 appeals process once you receive challenges?
14 A. We had a panel, which consisted of Dr. Cellar, Ron
15 Carabini, Sophia Sembiekos, Chief Cadigan and Deputy Klein. We
16 would receive the challenge, review it, meet to discuss it.
17 They would report to me in terms of what they had decided
18 preliminarily. In a number of cases, we had to go back for
19 more information from Arthur Andersen.
20 In some case, I asked Dr. Doverspike and Mr. Sawyer to
21 do some statistical analyses to see what the effect of the
22 challenge might be.
23 Q. Let me stop you there, so we can clear up some issues.
24 Mr. Carabini -- Ron Carabini, Sembiekos and Cellar,
25 are they members of Barrett & Associates' staff?
Barrett - direct
752
1 A. Yes.
2 Q. And are these individuals familiar or have some training in
3 industrial psychology?
4 A. Yes.
5 Q. And were they involved in the test development process for
6 the 1993 police sergeant's examination?
7 A. Yes.
8 Q. Were they involved from the beginning with the job
9 analysis?
10 A. Yes.
11 Q. And all the way through the test writing development phase?
12 A. Yes.
13 Q. And as well as with the scoring of the examination?
14 A. Yes.
15 Q. And Chief Cadigan and Lieutenant Klein, those are the same
16 -- are those the same two individuals that came to review the
17 test items towards the end of your test development process
18 prior to the administration of the written job knowledge test
19 and the in-basket simulation?
20 A. Yes.
21 Q. And Dr. Doverspike, can you tell the Court who
22 Dr. Doverspike is?
23 A. He is a member of Barrett & Associates. He has his Ph.D.
24 in industrial and organizational psychology.
25 Q. And you mentioned that Dr. Doverspike and -- I'm sorry --
Barrett - direct
753
1 somebody else --
2 A. Sawyer.
3 Q. Sawyer.
4 Does this person also have a background in industrial
5 psychology and psychometrics?
6 A. Yes.
7 Q. And you mentioned that you asked them to do some
8 statistical analyses?
9 A. Yes.
10 Q. On the items that had been challenged?
11 A. Only on one or two, to see what the effect might be, yes.
12 Q. Okay.
13 Let's go back to the initial meeting. You mentioned
14 Arthur Andersen. Can you describe your understanding of what
15 Arthur Andersen's role was in the challenge process?
16 MR. FLAXMAN: Let me object to the form of the
17 question. If he knows what their role was, he could testify to
18 it. If he doesn't know, his understanding is not relevant.
19 THE COURT: Well, all of the testimony to date
20 indicates that he worked directly with them in the arranging
21 for the giving of the testing. You are requiring that some
22 foundation be laid as to --
23 MR. FLAXMAN: I'm just objecting to the form. If the
24 question was: "What did Arthur Andersen do on the test," I
25 wouldn't have an objection.
Barrett - direct
754
1 But if the question is: "What's your understanding of
2 what they did" --
3 THE COURT: Well, let us lay a foundation as to how --
4 I do not know how clear that foundation has been --
5 MS. GLINK: For the record, Mr. Flaxman questioned
6 Dr. Barrett on Arthur Andersen's role in the process already,
7 so that's already part of the record.
8 THE COURT: Right.
9 MS. GLINK: But I would be happy to lay more of a
10 foundation.
11 BY MS. GLINK:
12 Q. Was Arthur Andersen involved in any way in the challenge
13 process?
14 A. Yes.
15 Q. And what involvement did they have?
16 A. Arthur Andersen was the firm hired to administer the test.
17 We coordinated our activities with them in terms of test
18 administration, but they had the responsibility of obtaining
19 the facility, the chairs, the tables, proctors, monitors. The
20 whole process was their responsibility, which we coordinated
21 with them.
22 They also were responsible for maintaining certain
23 records. They maintain the records, for example, of the
24 proctors. They maintained the records of the -- maintained the
25 answer sheets. They maintained anything which had to do with
Barrett - direct
755
1 the administration process.
2 And we obtained additional information from them
3 because, as I recall, the majority of the challenges had to do
4 with administration issues. For example, a candidate might say
5 something was missing from their packet of material or they
6 need more time or some other issues such as this.
7 Q. And were those issues similarly addressed through this
8 appeals process?
9 A. Yes, they were.
10 Q. Can you describe for me the type of challenges that you
11 would receive -- some of the challenges that you received to
12 these seven items? Can you give me a general understanding of
13 what the nature of the challenges were?
14 A. Well, the bulk of the challenges had to do with
15 administration, basically. There were one or two having --
16 saying the item was not clear, for example, there was some
17 ambiguity, some other answer was correct.
18 Q. And you mentioned that there was an initial meeting between
19 Carabini, Sembiekos, Cellar, Chief Cadigan and Lieutenant
20 Klein; is that correct?
21 A. Yes.
22 Q. And, then, what then happened after that initial meeting?
23 A. Each made their comments. They made a tentative decision.
24 I reviewed the decision, and then we all agreed this was the
25 decision to be made.
Barrett - direct
756
1 Then we drafted a letter for Arthur Andersen to
2 actually send. We did not send the letter, as I recall,
3 ourselves. I believe it was sent by Arthur Andersen, which
4 indicated what was the outcome of the appeal process. And I,
5 again, I recall -- I think Item 60 on the in-basket was double
6 keyed. Two responses were both counted as correct.
7 I believe another item on the in-basket we also
8 decided to make two items correct. And I think one item on the
9 job knowledge test we counted correct for everyone, also.
10 So, there were some changes made as a result of our
11 total review process.
12 Q. And when you say an item was double keyed, can you explain
13 what that means?
14 A. Both A and B was counted as correct.
15 Q. Or whatever the two right answers --
16 A. Yes.
17 Q. -- or alternatives that were determined to be correct?
18 A. Yes.
19 Or, in one case, I believe all alternatives were said
20 to be correct, so everyone got credit for that item.
21 Q. And --
22 THE COURT: Can we pin down what we are talking about
23 here, since we have had a lot of testimony about individual
24 questions?
25 MS. GLINK: Yes.
Barrett - direct
757
1 BY MS. GLINK:
2 Q. I believe that -- do you recall which particular items on
3 the written job knowledge test, if any, were double keyed?
4 A. From memory, all I can recall, the 60th item on the
5 in-basket.
6 Q. That would be Item No. 60?
7 A. Yes, and the job knowledge test, for some reason, I recall
8 it was 11, but I could be mistaken.
9 Q. Okay.
10 And there was --
11 THE COURT: Why do we not take the time. It will be
12 helpful to me trying to make some --
13 MS. GLINK: That's fine, your Honor.
14 THE COURT: -- factual findings if I know what the
15 position of the defendants are with respect to various
16 questions that have been criticized here.
17 MS. GLINK: That's fine, your Honor.
18 THE COURT: If Dr. Barrett is not going to respond, I
19 suppose you have other witnesses that could respond. But if
20 you don't choose to respond, I only have one side's view of the
21 validity of the questions then.
22 MS. GLINK: I believe that Plaintiffs' Exhibit 54 or
23 55, one of the two, contains Item No. 19. And that item, I
24 believe, it reflects on that exhibit was double keyed, D and E,
25 or there were two scores given on that item.
Barrett - direct
758
1 I also believe the answer sheet for the in-basket
2 simulation that has been presented by the plaintiffs in this
3 case, although it does not reflect that Item 60 was double
4 keyed, despite the fact we had told counsel about that fact
5 back in September of '95, it does reflect that one of the items
6 was given credit for all of them.
7 So, we can get those for you. I believe that those
8 items will be in the record, and there will be a record -- a
9 paper record of what items were double keyed and what items
10 were not double keyed.
11 THE COURT: Of the decision on each of these
12 challenges, there is a paper record in the report, like Exhibit
13 D?
14 MS. GLINK: Plaintiffs have --
15 THE COURT: If it is already going to be in evidence,
16 anyway, I do not require it be duplicated, but I need to know
17 both sides, if possible. It is a big help to me in trying to
18 make some decisions.
19 MS. GLINK: I believe the record will reflect that
20 Item 19 on the written job knowledge test was double keyed.
21 The record will reflect that Item 60 on the in-basket
22 simulation was double keyed. The record will also reflect --
23 THE COURT: We cannot have you testifying, so --
24 MS. GLINK: No, no, I am saying there will be a paper
25 record of it.
Barrett - direct
759
1 THE COURT: All right.
2 MS. GLINK: In the documents that will be submitted to
3 this Court for evidence.
4 THE COURT: You do not need his testimony then. All
5 right.
6 MS. GLINK: We will also review and see if there is
7 other paperwork that will help the Court in determining which
8 items reviewed, if that is something the Court would wish to
9 see.
10 MR. FLAXMAN: Judge, we would be agreeing to two of
11 those contentions, but requiring there be proof of the third
12 contention about Question 60 on the in-basket, which -- I mean,
13 we would be stipulating that Question 19 on the written test,
14 that there were two correct; that Questions B and C were
15 correct.
16 MS. GLINK: I would like to call -- this is something
17 I wanted to bring up this morning, your Honor. I sent a letter
18 to the plaintiffs' counsel on September of 1995, letting him
19 know that the answer key that I sent him was improperly scored,
20 that we had made a mistake, and that there was full credit
21 given for Item 19 --
22 THE COURT: Do you wish to mark this?
23 MS. GLINK: Yes, I will now mark it as an exhibit --
24 that the answer key I had sent him was improperly scored, that
25 we had made a mistake and that there was full credit given for
Barrett - direct
760
1 Item 19 --
2 THE COURT: Do you wish to mark this?
3 MS. GLINK: Yes, I will now mark it as an exhibit.
4 And that he knowingly submitted false testimony to
5 this Court on the stand by eliciting testimony on Item 60 and
6 how it was improper yesterday. And I would move to strike all
7 testimony about Item 60 on the in-basket stimulation on the
8 grounds counsel put in --
9 THE COURT: I am not going to strike testimony, but it
10 is helpful to have testimony from the other side, so that --
11 MS. GLINK: Your Honor, yesterday he put on the --
12 THE COURT: -- so that I can evaluate it.
13 MS. GLINK: Last week he put on the stand one of his
14 subject matter experts who took the examination. He testified
15 that we had improperly scored --
16 THE COURT: Can I say something? It is important to
17 complete his testimony. He has to be completed today. These
18 matters can be handled later on, it seems to me. Let us a go
19 ahead and complete his testimony first. And, then, if we need
20 to have a hearing on a particular motion, we will go ahead and
21 do that. But we do need to have everything marked, so, for the
22 record, we know --
23 MS. GLINK: I will mark that as -- I think we are on
24 Defendant's Exhibit L.
25 THE COURT: Okay.
Barrett - direct
761
1 I do not mean to interfere with your examination. I
2 am just saying that where considerable time has been spent on
3 attacking the validity of the answers to various questions, it
4 is helpful to me to find out if the defendants have any
5 contrary view to that.
6 MS. GLINK: I understand, your Honor, but when we do
7 spend a lot of time attacking the validity of questions, I
8 would ask that the plaintiffs' counsel do so properly and in
9 good faith before this Court. That's all that I'm asking.
10 THE COURT: Okay.
11 If we can get a number for that letter; and, then, if
12 we can continue on now with whatever you were going to be
13 asking the expert, so we can complete his testimony.
14 MS. GLINK: That will be marked as Defendant's Exhibit
15 L.
16 THE COURT: L?
17 MS. GLINK: L, that letter.
18 BY MS. GLINK:
19 Q. You testified that Dr. Doverspike and another individual at
20 Barrett & Associates conducted some statistical analyses on two
21 of the items on -- that have been challenged. Can you explain
22 the nature of the statistical analysis?
23 A. I don't recall the exact details now. Just to verify
24 various options which might be possible. It had to do with the
25 issue of, I believe, one testee did not receive information
Barrett - direct
762
1 about one item on the in-basket. Other testees also alleged
2 that they had been given instructions which was in error.
3 And we just looked at alternative ways. What if, for
4 example, if we gave everyone credit? What if we used a mean
5 for that person, used that for a score? What were the
6 alternatives?
7 We actually did not use or make any of those decisions
8 based on that analysis, but we -- it was just something I
9 looked at and reviewed. In effect, we made the decision to
10 give everyone credit, as I recall. And for that one person, we
11 gave him credit for all of the material which was missing from
12 his packet.
13 But we looked at alternatives. So, it was a review
14 process we went through on each one before we reached a final
15 decision.
16 Q. And did you receive any challenges to any portion of the
17 oral briefing exercise?
18 A. I don't recall receiving any challenges to that.
19 Q. How did the number of challenges that you received in this
20 case compare to your experience in the past when you've had an
21 appeals process?
22 MR. FLAXMAN: Objection.
23 THE COURT: Well, if it is for lack of foundation, I
24 will sustain the objection. It seems to me that, if there is
25 sufficient similarity, it may have some relevance, but a
Barrett - direct
763
1 foundation would have to be laid.
2 BY MS. GLINK:
3 Q. Have you had other examinations where there were appeals
4 processes?
5 A. Yes.
6 Q. And were they similar to the appeals process that was in
7 place for the Chicago Police Department?
8 A. Yes.
9 Q. In those other instances when you received challenges, how
10 many challenges, approximately, would you receive?
11 MR. FLAXMAN: Objection. I don't think it's enough
12 foundation.
13 THE COURT: It is just conclusory, and there is no way
14 to evaluate how similar.
15 MS. GLINK: I'll move on, your Honor. I don't think
16 it's that important of a point.
17 THE COURT: Well, you have the right to ask him to
18 describe exactly whether we are dealing with police sergeants,
19 whether we are dealing with police, whether we are dealing with
20 any law enforcement or something else.
21 MS. GLINK: That's fine.
22 BY MS. GLINK:
23 Q. These past examinations in which you had an appellate
24 process for challenging items on your examination, did they
25 involve safety force examinations?
Barrett - direct
764
1 A. Yes, they did.
2 Q. Did any of them involve examinations for police sergeant?
3 A. Yes.
4 Q. And in the examinations that you had an appeals process to
5 items in the past examinations for police sergeant, has it been
6 your experience -- in your experience, would you generally get
7 more challenges than you received in this examination?
8 MR. FLAXMAN: Objection.
9 THE COURT: Well, you need to know numbers. You need
10 to know whether the test was a similar -- I mean, there is a
11 lot of things you need to know in order to have it relevant.
12 MS. GLINK: That's fine, your Honor. I don't -- I'm
13 happy to move on.
14 BY MS. GLINK:
15 Q. Did you form any opinion about the challenges that you
16 received -- or the number of challenges that you received in
17 this examination?
18 A. Yes.
19 Q. And what opinion did you form?
20 A. We received far fewer challenges than would be normal for
21 this type of examination.
22 Q. Professor -- Dr. Barrett, when you finished administering
23 the examination, how was a candidate's final score on the
24 examination calculated?
25 A. We added together, unit weighted the scores of the three
Barrett - direct
765
1 examinations and obtained a total score, and rank ordered each
2 person by total score.
3 Q. When you said "unit weighted," what do you mean by that?
4 A. That means each of the tests received the same weight.
5 Q. And you combined them to create a composite or final score?
6 A. Yes.
7 Q. And once you calculated the final scores on the
8 examination, did you make any recommendations to the City of
9 Chicago regarding how the examination results should be used?
10 A. Yes.
11 Q. And what recommendation did you make?
12 A. We --
13 MR. FLAXMAN: Could we have more foundation how he
14 made the recommendation, when and to whom he made it?
15 THE COURT: I will overrule that objection. You have
16 the right of cross-examination on that. There is enough
17 foundation for him to answer that.
18 Go ahead.
19 BY MS. GLINK:
20 Q. Do you recall the question that I've asked?
21 A. Not exactly.
22 Q. Okay.
23 I asked you, after you had calculated the final scores
24 on the examination, did you make a recommendation to the City
25 of Chicago regarding how the examination scores should be used?
Barrett - direct
766
1 A. Yes.
2 Q. And what was that recommendation?
3 A. Rank order.
4 Q. Okay.
5 And why did you recommend that the examination results
6 be used in rank order?
7 A. Well, there are a number of reasons. Before we even began
8 the process, we were determined to develop a content-valid
9 test. And following proper procedures, we, in fact, did
10 develop a content-valid test.
11 According to the Uniform Guidelines, according to the
12 Standards and according to the Principles, when you have a
13 content-valid test and you have job analysis evidence, for
14 example, that a higher score relates to better job performance,
15 then, in fact, you can rank order.
16 Q. Let me stop you there. You said "Standards" and
17 "Principles." Can you be more clear what you mean by that?
18 A. It's the Standards for Educational and Psychological
19 Testing, Uniform Guidelines of 1978, which has been drafted and
20 written by EEOC and other federal agencies, and the Principles
21 for Personnel Selection, which has been published by The
22 Society for Industrial and Organizational Psychology.
23 Q. And are these the Standards and Principles and Guidelines
24 that are accepted in your field of industrial psychology as
25 setting out the framework for how a test should be developed
Barrett - direct
767
1 and validated?
2 A. Yes.
3 Q. And you mentioned that you had evidence from your job
4 analysis. Can you explain what you meant by that?
5 A. Well, it was clear from our job analysis, just to give one
6 example, the more knowledge you had, the more of the different
7 types of major work behaviors you can perform the task.
8 For example, if you had no knowledge of the Illinois
9 Criminal Code, there is many tasks you could not do as a police
10 sergeant. You couldn't review reports. You couldn't do a lot
11 of things. You couldn't give guidance to police officers. So,
12 it was clear from that that there was that sort of
13 relationship.
14 We also had the linkages between each item on the job
15 knowledge test and the work behaviors. And if you would take
16 away that knowledge, there would not be the prerequisite
17 knowledge to allow the sergeants to make a variety of decisions
18 which they are required to make.
19 Q. Other than this evidence, were there any other reasons why
20 you recommended rank order use of the examination results from
21 the 1993 sergeant's examination?
22 A. We have a wealth of scientific research showing that test
23 scores relate in a linear fashion to job performance.
24 Q. What do you mean by that?
25 A. That means the higher the test score, the higher the
Barrett - direct
768
1 probability of being a successful job performer. And, in
2 particular, for example, there have been content-valid job
3 knowledge tests which have been studied and related to job
4 performance.
5 Q. Now, those tests, did they involve the Chicago Police
6 Department?
7 A. No.
8 Q. What kinds of studies are these?
9 A. Well, there are some which have been done in the military,
10 others -- at least one that's been done in a police
11 department. Tests where we did look at job knowledge and found
12 there was a relationship between the job knowledge and job
13 performance.
14 Q. In your field of expertise, is that the kind of evidence
15 that an industrial psychologist relies on in reaching decisions
16 about how to use examinations?
17 A. Yes.
18 Q. Other than that evidence that we've discussed, were there
19 any other reasons why you recommended rank order use of the
20 results of this examination?
21 A. Yes.
22 After we administered the test to the 4,700 people --
23 I'm now referring to the job knowledge test and the in-basket
24 -- we could look at the psychometric properties of the test.
25 Q. What do you mean by "psychometric properties of the test"?
Barrett - direct
769
1 A. These are various statistical indices which indicate how
2 adequate the test is for the purpose for which it's been
3 constructed.
4 In other words, we devised a content-valid test, which
5 was relevant to the job of police sergeant. It was never
6 pretested. We never went out and took a sample of patrol
7 officers to see what would happen; in other words, would it be
8 reliable? So, you have to do this after the fact.
9 In other words, you hope by your procedures that you
10 followed -- good item writing practice, good test construction
11 practice, all the things we've talked about previously -- that
12 you would obtain a test which has good psychometric
13 properties. The first -- and we've talked about some before --
14 is the test reliable? As I recall, the test had a reliability
15 of about .86, .87, which is high reliability for this type of
16 test.
17 So, we knew that, if a person would take the test
18 over, again, they would tend to get the same test score,
19 again. This gives you confidence that you are measuring
20 something in a reliable fashion.
21 We had other indices that we looked at for the test.
22 For example, what percent of the items were answered correctly
23 by individuals? We found we had a range. Not everyone got all
24 the questions correct, for example.
25 Q. And what does that mean to you, Dr. Barrett?
Barrett - direct
770
1 A. Well, you could differentiate among individuals based on
2 the knowledge. In other words, we had a test where it wasn't
3 too easy, and it wasn't too hard, where everyone had scores
4 which were very, very low. We had a range of scores.
5 What it meant was, in an operational sense, we didn't
6 have any extreme bunching of scores, say, at the top end or the
7 low end. It did differentiate among individuals, say, on the
8 knowledge they possessed.
9 A third indicator was the fact that, when we examined
10 the item statistics looking at each -- let's just use the job
11 knowledge test as an example. It had 150 items. We could take
12 for Item No. 1 individual responses and correlate that response
13 to Item No. 1 to their total score on that test.
14 In general, you expect to have a low positive
15 correlation. Now, some items, for a variety of reasons, could
16 be 0, perhaps even negative; but, in general, you expect to
17 find some positive relationship, say around .2, between a
18 response to one item, one small piece of behavior, to the 150
19 samples of behavior. And, in fact, we did find that there was
20 that sort of positive relationship between each item and the
21 total test score.
22 So, that, again, gives you some confidence that you're
23 meeting the standards of what is a good test from a
24 psychometric point of view.
25 Q. And these are the reasons that you have just spoken for why
Barrett - direct
771
1 you recommended rank ordering?
2 A. Well, there's one other thing we did, also. We also looked
3 at a process called banding, and to see if that would be an
4 alternative way of scoring the examination which might be a
5 better way of doing it.
6 Q. Let me stop you there and ask you to explain to the Court
7 what banding is.
8 A. Banding is a relatively new controversial technique in the
9 field, where you assume -- it's an assumption -- that you
10 observe scores -- say, you have scored between 90 and 100. You
11 pretend like those scores are all the same, they don't
12 differentiate. So, you assume that everybody who has a score
13 between 90 and 100 really has a score of 100, for example.
14 And there are many different types of banding, but --
15 and we looked at a number of different alternatives. And, of
16 course, the problem with banding is, once you have people in a
17 band, how do you select from those individuals in a band?
18 Q. So, the band would include -- could include many
19 individuals?
20 A. It could include many individuals. So, you're still left
21 with the decision: How do you make the final decision of
22 exactly what individuals you are going to promote to sergeant?
23 THE COURT: Well, I do not understand why anybody
24 would look at banding? What has banding got to do with
25 anything? I mean, there is no explanation as to the value of
Barrett - direct
772
1 analyzing --
2 MS. GLINK: I will elicit that, your Honor.
3 THE COURT: -- using the banding techniques.
4 BY MS. GLINK:
5 Q. Is there a reason why you looked at banding as an
6 alternative to rank order use?
7 A. Well, there's always a possibility that it would have
8 reduced adverse impact perhaps, and there might be some other
9 justifications. Say, well, this is a better process.
10 For example, if the scores had been very bunched
11 together at the top end, one might argue, "Well, wait a
12 minute. You have two" -- in one court case, for example,
13 two-thirds of the scores were bunched within four -- within
14 four items, like 97 to 93, for example. And you might say,
15 "Wait a minute. Your test is not doing a very good job from a
16 psychometric point of view of differentiating individuals."
17 And, so, maybe the best thing to do in a situation
18 like that is assume that the scores between 93 and 97 are
19 actually the same score. So, there are reasons to look at
20 this. And one of the main reasons, of course, is it is a
21 technique which has been ordered in some consent decrees.
22 For example, most notably in San Francisco, where
23 there has been a consent decree, they said, "All right. Here
24 is a finding of discrimination, and let's use an alternative
25 technique for selecting individuals for positions."
Barrett - direct
773
1 Q. And after you had analyzed --
2 THE COURT: I mean, is this a way to avoid what
3 otherwise might be adverse impact by just changing the scores?
4 That is what you are doing, I guess.
5 THE WITNESS: It's a hope, but the reality is it often
6 does not because they actually don't use a strict banding.
7 First they use a rank order, and I'm talking about operational
8 use by the court. They'll use a rank order, for example, in
9 San Francisco. Then, after they go from a rank order, they
10 will take so many people and they will use some other process
11 to select individuals from that group. So, you still have
12 adverse impact often, but it may be less severe in some cases.
13 So, it's not a panacea is all I can say.
14 BY MS. GLINK:
15 Q. And is your attempt to look at banding set forth anywhere
16 in your technical report, which has been marked as Exhibit D?
17 A. I think it's Appendix S.
18 Q. And after you analyzed banding, was that something that you
19 recommended to the City of Chicago?
20 A. No, it was of no value to the City of Chicago because, in
21 fact, in one of our analyses, you could have fewer minorities
22 selected because, once you put people in a band, one technique
23 is to draw out at random, and it crosses -- this is a lottery.
24 I mean, when you go into a lottery situation, we actually found
25 that in some cases we had fewer minorities selected than if
Barrett - direct
774
1 they would have used the rank order approach.
2 Q. Dr. Barrett, do you recall testifying last week about
3 criterion-related validation?
4 A. Yes.
5 Q. And do you also recall testifying that criterion-related
6 validation involves the correlation of test scores to some
7 measure of job performance?
8 A. Yes.
9 Q. In your opinion, do you have to conduct a criterion-related
10 validation study before using the results of a content-valid
11 test in rank order?
12 A. No.
13 Q. And what's the basis of your opinion?
14 A. Again, as I've previously stated, the professional
15 literature on various Uniform Guidelines, the Principles and
16 the Standards and our prevailing knowledge in the field about
17 the fact that content-valid tests do, in general, relate to job
18 performance.
19 Q. In your professional opinion, in your field of expertise,
20 would you say that this type of empirical evidence, which is --
21 when I'm saying "empirical evidence," I mean, a correlation
22 between test scores and job performance -- is generally viewed
23 as not required in this instance when you have a content-valid
24 test?
25 MR. FLAXMAN: Objection. I don't think "generally
Barrett - direct
775
1 viewed" is something that's --
2 THE COURT: I will sustain the objection as to the
3 form of question, but not the subject matter.
4 And we are getting an awful lot of conclusory
5 statements without any indication of any support by literature
6 in the field or tests that he was involved with or anything
7 else. I will not say that they are not there. It is just not
8 getting into the evidence.
9 MS. GLINK: That's fine.
10 BY MS. GLINK:
11 Q. Dr. Barrett, have you been in the field of industrial
12 psychology for over 40 years?
13 A. Yes.
14 Q. And have you been doing research in the field of industrial
15 psychology for 40 years?
16 A. 40 years? I thought you said 30 years.
17 Q. 30 years, I'm sorry.
18 A. 30 years would be probably more accurate.
19 Q. And during the course of your career as an industrial
20 psychologist, have you analyzed -- have you become familiar
21 with literature in your field, including your own, which
22 analyzes the need or looks at the need for empirical evidence
23 when you have a content-valid test?
24 A. Yes.
25 Q. And are these studies set forth in the affidavit that you
Barrett - direct
776
1 prepared -- are some of these studies that you reviewed set
2 forth in the affidavit that you prepared on March 3rd for this
3 case?
4 A. Yes.
5 THE COURT: Do we have an exhibit number on that?
6 MS. GLINK: It's attached as Exhibit E, your Honor, to
7 the --
8 THE COURT: It's Exhibit E.
9 MS. GLINK: It's Exhibit E to what you have.
10 THE COURT: Oh, okay.
11 MS. GLINK: And I'm going to show Dr. Barrett.
12 (Document tendered.)
13 BY MS. GLINK:
14 Q. And have you reviewed the studies that are set forth in
15 Exhibit E?
16 A. Yes.
17 Q. And do you agree with the research techniques and
18 conclusions reached by these individuals?
19 A. In general --
20 MR. FLAXMAN: Objection. Whether he agrees is not
21 relevant.
22 THE COURT: Well, it has something to do with the
23 basis -- the foundation for his responses. I will overrule the
24 objection and permit the answer to stand. You can pursue it
25 further in cross-examination.
Barrett - direct
777
1 BY MS. GLINK:
2 Q. I'm going to direct your attention to Paragraph 10 of
3 Exhibit E, and Paragraph 11, in particular, of Exhibit E.
4 Is it your opinion, based on your years of experience
5 in the field of industrial psychology and psychometrics and
6 your review of the relevant research in your field, that you do
7 not need to conduct an empirical -- a criterion-related
8 validation study before utilizing examination results in rank
9 order?
10 A. Yes.
11 Q. You mentioned earlier that there are recognized procedures
12 in the field of industrial psychology and psychometrics for
13 analyzing the psychometric properties of an examination; is
14 that correct?
15 A. Yes.
16 Q. Did you apply these procedures to the written job knowledge
17 test that was part of the 1993 sergeant's exam that was
18 administered to 4700 candidates?
19 A. Yes.
20 Q. And did you analyze the results of the written job
21 knowledge test to determine the psychometric properties of the
22 test and the test results?
23 A. Yes, I did.
24 Q. And do you have an opinion, based on this analysis,
25 regarding whether this examination meets accepted standards in
Barrett - direct
778
1 your field of industrial psychology and psychometrics?
2 A. Yes.
3 Q. And what is that opinion?
4 A. It does meet all of the standards for a good psychometric
5 test.
6 Q. And did you conduct a similar analysis of the psychometric
7 principles of the in-basket simulation and the oral briefing
8 exercises that were part of the test battery for this
9 examination?
10 MR. FLAXMAN: Could we have one question at a time?
11 One test at a time?
12 MS. GLINK: Fine, your Honor.
13 BY MS. GLINK:
14 Q. Did you conduct a similar analysis of the psychometric
15 properties of the results of the in-basket simulation?
16 A. Yes.
17 Q. And did you reach any conclusions as a result of that
18 analysis?
19 A. Yes.
20 Q. And what conclusions did you reach?
21 A. It was -- had good psychometric properties.
22 Q. And did you conduct a similar analysis of the psychometric
23 properties of the results of the oral briefing exercise that
24 was the third component part of the 1993 police sergeant's
25 examination?
Barrett - direct
779
1 A. Yes.
2 Q. And do you have an opinion based on that analysis?
3 A. Yes.
4 Q. And what opinion do you have?
5 A. It had good psychometric properties.
6 Q. Dr. Barrett, does the technical --
7 THE COURT: I am not sure that I know what "good
8 psychometric properties" means. Does it mean he thinks it is
9 valid or invalid or that it could be valid? I mean, I just --
10 I do not have any definition of what good psychometric
11 properties is.
12 BY MS. GLINK:
13 Q. Could you define that for us, Dr. Barrett?
14 A. It goes back, again, to the three things I mentioned, three
15 statistical indices. The first index is the reliability of the
16 test. If a test is not reliable, it cannot be valid.
17 THE COURT: Then it is capable of repetition or the
18 same --
19 THE WITNESS: Repetition, yes.
20 THE COURT: -- answer? Okay.
21 THE WITNESS: And you can calculate it a number of
22 different ways, but, conceptually, it is. If you took the same
23 exam over, again, would people be in the same rank order. It's
24 really a correlation coefficient and a measure of association.
25 BY MS. GLINK:
Barrett - direct
780
1 Q. And did you find that this examination was reliable?
2 A. Yes, I did.
3 Q. You said there's two other indices that you looked at. Can
4 you tell us what the second one was?
5 A. The second indice was the item total correlation. For each
6 of the 150 items, we calculated a correlation.
7 Q. This would be on the written job knowledge test?
8 A. Yes.
9 For each of the 150 items on the written job knowledge
10 test, we calculated a correlation between the item score and
11 the total score. In general, we found a positive relationship
12 between even one item and the total score.
13 Q. Can you explain that? When you say -- what's the purpose
14 of looking at one item to the total score? Is that for a
15 particular individual?
16 A. Well, it's across all individuals, of course. So, all
17 4,700 were in the sample.
18 Q. And what's the purpose of that analysis?
19 A. To determine if there is an association between the items
20 and the total score. If, for example, you had most of your
21 items being negative, that, on this item, that people who do
22 well on the item do poorly on the test, we would have some
23 concern about the properties of that item and, in general,
24 about the test.
25 In general, you want to see a small positive
Barrett - direct
781
1 relationship between one type of behavior -- the item, one item
2 -- and the response on the 150 items. And, in fact, we did
3 find that we did have that sort of relationship, in general,
4 between the items and the total score.
5 The third psychometric property I mentioned was the
6 percent correct. We look at each item. And what percent of
7 the total group obtained a correct answer on that item. If,
8 for example, we have most of our items which are very easy and
9 99 percent of the people obtained the correct response, it's
10 not going to be a very good test.
11 Conversely, if only five percent of the people, in
12 general, get the items correct, it's a very hard test and it's
13 not a test which differentiated among individuals. So, it's
14 not keyed at an appropriate level.
15 Q. With this third analysis, do you expect that, on some
16 items, the majority of the people will get it correct?
17 A. Yes, there should -- there will be a range. In other
18 words, some items a lot of people get it correct, and some
19 items can be more difficult for people. So, there's going to
20 be a range of a percent correct on any one item.
21 Q. Now, you mentioned the other day that there were are
22 distractors built into your examination on this written portion
23 of your examination. Do distractors play a role in any way in
24 whether an individual gets an item right or wrong?
25 A. Yes.
Barrett - direct
782
1 Q. Can you explain the role that distractors play in the test
2 development process?
3 A. For the job knowledge test, we had, for each of the 150
4 items, one alternative which was keyed to be correct. We had
5 four which were designed to be plausible distractors.
6 Q. And why do you design distractors to be plausible?
7 A. Because if, in fact, they were not plausible, someone with
8 no knowledge of, say, the Illinois Criminal Code or the general
9 orders, could answer the question without the prerequisite
10 knowledge. Because one item is designed to tap one domain of
11 knowledge, and, if all it was was distinguishing between
12 plausible and implausible items, then we're not testing for
13 that domain of knowledge by that item.
14 MS. GLINK: Your Honor, is that sufficient for your
15 understanding?
16 THE COURT: It is helpful.
17 So, I take it that those three items of statistical
18 reliability are meant by good psychometric properties or are
19 included in good psychometric properties?
20 THE WITNESS: Yes, they are included in that.
21 THE COURT: Okay.
22 BY MS. GLINK:
23 Q. In your opinion, Dr. Barrett, is the test that you
24 developed for the City of Chicago a valid test?
25 A. Yes.
Barrett - cross
783
1 MS. GLINK: I have nothing further, your Honor.
2 THE COURT: Do you need a couple minutes?
3 MR. FLAXMAN: Actually, I do need a couple minutes.
4 THE COURT: All right.
5 Let us take a five-minute break.
6 (Whereupon, a recess was taken.)
7 THE COURT: All right. Court is back in session, and
8 our witness can resume his position.
9 And, Mr. Flaxman, you may proceed when ready.
10 MR. FLAXMAN: Thank you, Judge.
11 CROSS-EXAMINATION
12 BY MR. FLAXMAN:
13 Q. Now, Dr. Barrett, right at the end of your direct
14 examination, you were talking about good psychometric
15 properties; do you remember that part of your testimony?
16 A. Yes.
17 Q. And you told us, I think, that there were three indices of
18 good psychometric properties that were relevant to your work on
19 this promotional test; is that right?
20 A. Yes.
21 Q. The first was the reliability that deals with repeated
22 measures; is that correct?
23 A. Yes.
24 Q. The second was the item total correlations; is that right?
25 A. Yes.
Barrett - cross
784
1 Q. Were there item total correlations for the oral
2 examination?
3 A. No.
4 Q. Were there item total correlations for the in-basket
5 examination?
6 A. Yes.
7 Q. And you looked -- you also said the third index was percent
8 correct?
9 A. Yes.
10 Q. That was something that was applicable to the written job
11 knowledge test; is that right?
12 A. Yes.
13 Q. It was also applicable to the in-basket test; is that
14 right?
15 A. Yes.
16 Q. Was it applicable to the oral test?
17 A. In some ways. Of course, the oral test was ratings by
18 trained raters.
19 Q. So, is it correct then that the only good measure of
20 psychometric properties that was applicable to the oral test
21 was reliability?
22 A. In the traditional sense, that's correct.
23 Q. And, then, you also talked about bunching of scores; is
24 that right?
25 A. Yes.
Barrett - cross
785
1 Q. And you said -- does the test have bad psychometric
2 properties if there's a bunching of scores?
3 A. If the total test scores for the three test components were
4 all bunched together, of course, we couldn't differentiate
5 among individuals.
6 Q. Well, what if the scores on any one of those components
7 were all bunched together? Would that also be true?
8 A. No, because we're looking -- we're making a decision based
9 upon all three components.
10 Q. So, if 50 percent of the people got between -- got 12, 13,
11 14 or 15 on the oral test, that wouldn't interfere with your
12 opinion that the oral test had good psychometric qualities?
13 A. No, it would not.
14 Q. Is that about what the numbers were on the oral test,
15 Dr. Barrett?
16 A. I don't recall exactly what it was on the oral test.
17 Q. Well, is it correct that, on the oral test, the scores
18 range from -- the theoretical scores range from 0 to 15?
19 A. Yes, that's correct.
20 Q. And the practical scores were -- did not get as low as 0,
21 did they?
22 A. No, I don't believe anyone received a 0 on the oral
23 examination.
24 Q. As a matter of fact, the lowest score that anyone received
25 was a 5; is that right?
Barrett - cross
786
1 A. I don't recall.
2 Q. Well, are there any documents, any part of your report that
3 would refresh your recollection about what the actual
4 distribution of scores was on the oral exam?
5 A. I don't believe we have -- we could have. I don't recall.
6 Q. Well, would you like to look at your report and see if you
7 could find it for us? Do you have it with you?
8 THE COURT: Does counsel know? Is it in E?
9 MS. PAPUSHKEWYCH: If Mr. Flaxman is looking at a
10 document, if he could identify it, maybe it would expedite the
11 proceedings here.
12 THE COURT: If you have one, then we should not really
13 take the time.
14 MR. FLAXMAN: My point is it's not in the report, and
15 I am going to show that that's important that it's not in the
16 report. That's my -- that's -- but I don't want to testify
17 it's not in the report because maybe I missed it, and I don't
18 want to be accused of misrepresenting the report.
19 THE COURT: No, I am not suggesting that. So, in
20 other words, you are not aware of where it is?
21 MR. FLAXMAN: That's correct.
22 THE COURT: And the witness has said he is not aware.
23 BY MR. FLAXMAN:
24 Q. Well, could you find it in the report if it's there?
25 A. Well, again, I don't recall putting it in the report.
Barrett - cross
787
1 Q. So --
2 A. It could be, but I don't recall putting it in the report.
3 THE COURT: I have a copy of the report. It would be
4 Exhibit D; is that what we are referring to here?
5 MR. FLAXMAN: That's correct.
6 THE COURT: You can take a look at that, if that will
7 --
8 THE WITNESS: Well, I don't recall --
9 THE COURT: -- simplify it.
10 THE WITNESS: -- putting the distributions in the
11 report of any of the tests, but it would be in the Appendix if
12 I did, but I don't recall it being in any of our appendices.
13 BY MR. FLAXMAN:
14 Q. Well, could you take a minute and see if it's there?
15 (Brief pause.)
16 BY THE WITNESS:
17 A. No, I don't believe it's in the report.
18 BY MR. FLAXMAN:
19 Q. Okay.
20 Did you ever look at the distribution of scores on the
21 oral test?
22 A. Yes, after we administered the test, yes.
23 Q. And could you tell us why you didn't include it in the
24 technical report?
25 A. We didn't include, I don't believe, any distributions of
Barrett - cross
788
1 anything in the technical report.
2 Q. Could you tell us why you didn't include the distribution
3 of scores on the oral test in the report?
4 MS. GLINK: Asked and answered, your Honor.
5 THE COURT: Well, he has not answered it. Objection
6 overruled. He may answer.
7 BY THE WITNESS:
8 A. We can't include everything in the report. We include the
9 most important material.
10 BY MR. FLAXMAN:
11 Q. So, what's in the report is the most important material; is
12 that right?
13 A. At that point in time, yes.
14 Q. And the distribution of scores on the oral test is not as
15 important as the material that's in that report; is that right?
16 A. I would guess we made that judgment at that point in time;
17 that's correct.
18 Q. Okay.
19 Now, in that report, you have Appendix R; is that
20 right? Could you look at the report? We're talking about -- I
21 think it's Defendant's Exhibit E.
22 MS. GLINK: D, Ken?
23 THE COURT: Yes, Defendant's Exhibit D.
24 MR. FLAXMAN: D, I'm sorry.
25 THE COURT: And, then, you are talking about the
Barrett - cross
789
1 Appendix to that.
2 MR. FLAXMAN: Appendix R.
3 THE COURT: R, okay.
4 BY MR. FLAXMAN:
5 Q. Do you see Appendix R?
6 A. Yes, I do.
7 Q. And Appendix R is entitled, "Weighting Rationale Report";
8 is that right?
9 A. Yes.
10 Q. And it was important to include this Weighting Rationale
11 Report in the report -- final report; is that correct?
12 A. Well, it was important to be in the Appendix, I guess. So,
13 I guess we did include it; that's correct.
14 Q. Okay.
15 And it was more important to include this Weighting
16 Rationale Report than it was to include a report of the
17 distribution of scores on the oral test; is that right?
18 A. Well, I guess we made that judgment at that point in time,
19 yes.
20 Q. Now, at Page R-6 of this Weighting Rationale Report, you
21 said you reviewed the literature for an actual figure defining
22 the performance difference between blacks and whites on the
23 test battery; do you see that?
24 A. I'm not sure what you're referring to.
25 Q. Well, it's the first sentence -- first full sentence. This
Barrett - cross
790
1 was accomplished by reviewing -- do you have the same --
2 A. Yes.
3 Q. Okay.
4 Do you see that sentence?
5 A. "This was accomplished by reviewing literature for an
6 actual figure defining the performance difference between
7 blacks and whites on the test battery."
8 Q. Does that mean you looked at published articles to get a
9 number for the performance difference between blacks who took
10 the test and whites who took the test?
11 A. Now, keep in mind this does not refer, of course, to the
12 sergeant's examination.
13 Q. Can you --
14 A. The next sentence says --
15 Q. Could you answer my question, Dr. Barrett?
16 A. I'm trying to answer your question.
17 Q. My question, Dr. Barrett, is: Did you look at published
18 articles to determine an actual figure defining the performance
19 difference between blacks and whites on the test battery?
20 A. Well, I want to be clear what we're talking about.
21 Q. Well --
22 A. The test battery is not referred to, the sergeant's test
23 battery.
24 Q. Okay.
25 Well, when you reviewed literature, what did you look
Barrett - cross
791
1 at?
2 A. Well, the next sentence tells you, I believe, if I'm
3 reading it correctly. It says, "Sackett, DuBois & Noe, N-o-e
4 (1991) stages difference as approximately .47 standard
5 deviation units favoring whites."
6 Q. What is Sackett, DuBois & Noe?
7 A. It's --
8 Q. Is that an article?
9 A. Yes.
10 Q. Is that an article that appeared in a journal?
11 A. Yes.
12 Q. And is that the type of article that's generally relied
13 upon by experts in your field to reach conclusions?
14 A. Well, it contains some scientific knowledge, some
15 conclusions in our field, yes.
16 Q. So, is it correct, then, that, when you prepared the 1993
17 sergeant's test, you relied on the finding of Sackett, DuBois &
18 Noe that there's a difference of .47 standard deviation units
19 favoring whites?
20 A. No, I didn't rely upon it. I don't recall relying upon
21 that particular article in terms of preparing our examination
22 at all.
23 Q. Why did you include it in your report, Dr. Barrett?
24 A. We were doing an abstract simulation to look at different
25 potential weighting schemes for the three-test battery. And
Barrett - cross
792
1 using an abstract computer simulation -- this is the first time
2 we had ever done this sort of thing -- there's always an issue
3 about how you can theoretically maintain a test battery which
4 has equal validity, but has less adverse impact.
5 So, one alternative might be if somehow you could
6 devise a situation where your testing process is one where now
7 you have an alternative which has, for example, the same
8 validity as what you plan to use, but had less adverse impact.
9 You would want to use the one with the less adverse impact with
10 equal validity.
11 Q. So, abstract simulation, that means you didn't actually use
12 it in preparing the test; is that right?
13 A. No, it was done afterwards. It was not -- we didn't use it
14 to prepare the test, no.
15 Q. And you didn't use it to assign the weights for the three
16 components of the test, did you?
17 A. Well, we considered it in terms of one possibility, but it
18 was not actually -- no. I don't know how right the word "use
19 it" is, but it was an alternative procedure would he looked at,
20 same as we looked at banding as a possibility; other
21 considerations.
22 Q. And it was more important to include details about this
23 abstract simulation than to include a chart of the distribution
24 of scores on the oral test; is that right?
25 A. Well, I guess at that point in time we made that decision
Barrett - cross
793
1 this was more important, yes.
2 Q. Now, it says, "The Sackett article states this difference
3 approximately .47 standard deviation units favoring whites."
4 What does that mean, Dr. Barrett?
5 A. Well, I don't recall the exact article in the sense of
6 every detail.
7 Q. What does ".47 standard deviation units favoring whites"
8 mean?
9 A. That was a study -- I believe, a meta-analysis -- where
10 they looked at various measures of performance and found a
11 difference between blacks and whites on performance.
12 Q. So, do you mean that that study found that whites did
13 better on the job, whatever job it was, than blacks?
14 A. As I recall, that was their conclusion.
15 Q. And .47 standard deviation units, what does that mean as a
16 number?
17 A. It's an abstract number which indicates that 0 would be no
18 difference and 1 would be a relatively large difference.
19 Q. And .47 would be half of a large difference?
20 A. Yes.
21 Q. And, then, you used this difference between black and white
22 performance on the job in the simulation; didn't you,
23 Dr. Barrett?
24 A. Yes, it was one of the variables which we included, among
25 others, in the simulation.
Barrett - cross
794
1 Q. And you looked at -- you assumed that there was a situation
2 where whites performed one-third of the standard deviation
3 better than blacks on the job; is that right?
4 A. Yes, in our simulation, we had a case -- situation like
5 that.
6 Q. And you looked at another simulation where whites performed
7 one-half of the standard deviation better than blacks on the
8 job?
9 A. That's correct.
10 Q. And you looked at another simulation where whites performed
11 one standard deviation better than blacks on the job?
12 A. I think that's correct.
13 Q. How many sergeants did you talk to when you were involved
14 in preparing the promotional test?
15 A. I don't recall the exact number.
16 THE COURT: Could I just clarify, though, is all of
17 this related solely to the analysis as to what should be the
18 weighting of the three parts of the test, or does this relate
19 to something else?
20 I thought the witness had said this related to
21 weighting of the three-part -- the results of the three parts
22 of the test. Somebody -- I do not -- you do not have to do it,
23 but one side or the other, I would like to have that clarified
24 whether this relates only to weighting or whether it relates to
25 something else; and, if so, what.
Barrett - cross
795
1 Okay. You may proceed.
2 BY MR. FLAXMAN:
3 Q. You reviewed -- "you" meaning Barrett & Associates --
4 reviewed each test question with Chief Cadigan and Mr. Klein;
5 is that correct?
6 A. Yes.
7 Q. Did Chief Cadigan tell you that whites performed better on
8 the job of police sergeant than blacks do?
9 A. No.
10 Q. Did Mr. Klein tell you that?
11 A. No.
12 Q. Anybody from the Chicago Police Department tell you that
13 whites perform better on the job than blacks do?
14 A. No.
15 Q. Did you ever obtain any data from your job analysis to show
16 that whites perform better on the job than blacks do?
17 A. No.
18 Q. Now, the simulation that's reported in Appendix R-9, is it
19 your testimony today that that Weighting Rationale Report was
20 not the basis for the decision to weight each component of the
21 test equally?
22 A. Well, it was one consideration we used, among other
23 things. We looked at that evidence, also.
24 Q. Now, have you been assisting the City of Chicago in
25 responding to discovery in this case?
Barrett - cross
796
1 A. Yes.
2 Q. And have you been signing interrogatory answers --
3 A. Yes.
4 Q. -- in this case?
5 Let me show you what's been marked as Plaintiffs'
6 Exhibit 80.
7 THE COURT: Plaintiffs' Exhibit E?
8 MR. FLAXMAN: 80.
9 THE COURT: 8-0.
10 MR. FLAXMAN: 8-0, and I have a copy for the Court.
11 (Document tendered to the Court and witness.)
12 THE COURT: Thank you.
13 BY MR. FLAXMAN:
14 Q. Does your signature appear on the next to the last page?
15 A. Yes.
16 Q. And did you review this document before you signed it?
17 A. Yes.
18 Q. And do you see on Page 6, response to Interrogatory 6(b):
19 "The decision that the three components of the examination
20 would be weighted equally was based upon a simulation conducted
21 by Barrett & Associates, Inc. to determine weighting procedures
22 for the promotional process. See Appendix R of the Barrett
23 report"?
24 A. Yes.
25 Q. And was that language there when you affixed your signature
Barrett - cross
797
1 to Page 11?
2 A. Yes.
3 Q. And by Appendix R, the Barrett report, is that the thing --
4 the Weighting Rationale Report that we've been talking about?
5 A. Yes.
6 Q. So, is it correct, then, that the weighting decision, the
7 decision that the three components of the examination be
8 weighted equally, was based in part on the difference in
9 performance between blacks and whites of -- that whites perform
10 better than blacks?
11 A. No.
12 Q. Now, is there anything in your report, Defendant's Exhibit
13 D, which states that you're not -- you're rejecting the finding
14 of Sackett, DuBois & Noe that -- the difference in performance
15 between blacks and whites?
16 A. I don't recall we discussed it in the body of the report,
17 anything about Sackett, et al.'s results.
18 Q. Okay.
19 Now, you told us, I think, that, in your opinion,
20 someone who gets a higher score on the written job knowledge
21 test will do better as a sergeant than somebody who gets a
22 lower score because the test is job related; is that right?
23 A. Yes, that's, in part, true.
24 Q. And you can offer that opinion --
25 MR. FLAXMAN: Strike that.
Barrett - cross
798
1 BY MR. FLAXMAN:
2 Q. And you have offered that opinion without any
3 criterion-related study; is that right?
4 A. We have performed no criterion-related studies specifically
5 with any of the three components for the police sergeant in the
6 City of Chicago.
7 Q. And you believe that the written job knowledge test
8 measures the job knowledge required to be a sergeant; is that
9 right?
10 A. Yes, it's a --
11 Q. And --
12 A. -- sample of that domain.
13 Q. And the more -- excuse me?
14 A. It's a sample of that domain, yes.
15 Q. And the more job knowledge you have, the better you'll be
16 as a sergeant; is that right?
17 A. Yes.
18 Q. Now, could you explain -- now, the oral test, does that
19 measure job knowledge?
20 A. No, it does not.
21 Q. It measures how well you can do in that particular test,
22 doesn't it?
23 A. Well, it measures communication skills.
24 Q. Well, do you have any data that relates performance on the
25 oral examination with performance on any job?
Barrett - cross
799
1 A. We don't have any empirical data on that specific oral
2 communication exercise with any specific job, no.
3 Q. And is there any generally recognized text which discusses
4 that particular oral exercise and relates scores on that oral
5 exercise with performance on any job?
6 A. I don't know of any one text or one article to be used as
7 our specific oral communication exercise that relates it to job
8 performance.
9 Q. Now, the in-basket test, do you have any data which relates
10 scores on the in-basket test with performance on the job?
11 A. We have no data on the specific in-basket tests we
12 developed for the City of Chicago for police sergeant which
13 relates it to job performance as a police sergeant.
14 Q. Now, your in-basket test is -- is it correct to say it's
15 the Barrett in-basket test?
16 A. I don't know what you mean by the "Barrett in-basket test."
17 Q. Well, was the procedure of reading materials and answering
18 written short answer questions something that was developed by
19 Barrett & Associates?
20 A. Well, the basic process has been around since at least the
21 1940s in terms of an in-basket, and it's the most widely used
22 assessment center device or test. So, I wouldn't want to claim
23 credit for developing the in-basket.
24 Q. Well, is your in-basket the same kind of in-basket that was
25 used by the Office of Strategic Services in World War II?
Barrett - cross
800
1 MS. GLINK: Objection, your Honor. No foundation.
2 What is this test? Is there such a test?
3 THE COURT: Well, if the witness knows what it is,
4 fine. If he does not know, then we will have to have further
5 questions. I will overrule the objection and permit the
6 witness to answer, if he knows.
7 BY THE WITNESS:
8 A. As I recall, I think it was a 1940 book, and I don't recall
9 the exact structure of the in-basket test by OC -- OSS used.
10 Q. Well, in Appendix R of your report, you refer to some
11 published literature which discusses in-basket tests; is that
12 right?
13 A. Yes.
14 Q. And one of those published articles that you relate to is
15 an article by Wollowick & McNamara.
16 THE COURT: What is the first?
17 MR. FLAXMAN: W-o-l-l-o-w-i-c-k and McNamara.
18 BY MR. FLAXMAN:
19 Q. Is that right?
20 A. Yes.
21 Q. And that reference appears at Page R-4 of your Appendix R;
22 is that right?
23 A. Yes.
24 Q. And Wollowick & McNamara describe a study that they did,
25 among other things, including an in-basket; is that right?
Barrett - cross
801
1 A. I don't recall the exact -- at this point, what they
2 exactly did.
3 Q. Let me show you what's been marked as Plaintiffs' Exhibit
4 7.
5 MR. FLAXMAN: I have a copy for the Court.
6 (Document tendered to the Court and witness.)
7 BY MR. FLAXMAN:
8 Q. Is Plaintiffs' Exhibit 7 a copy of the -- with a blank page
9 -- of the article that's referred to in Page R-4 by Wollowick
10 & McNamara?
11 A. Yes, it appears to be the same article.
12 Q. And at some point in your almost 40-year career, you read
13 this article; is that right, Dr. Barrett?
14 A. Yes, I have read this article.
15 Q. And when you read this article, did you see on Page 349 in
16 the second column on the right, the first full paragraph where
17 if describes the type of in-basket that was used in this study?
18 A. Yes.
19 Q. And that type of in-basket is different than the type of
20 in-basket that was used in the 1993 sergeant's promotional
21 test; is that right?
22 A. In reading it, it appears it's different in the sense that
23 they had an interview process for 30 minutes after they took
24 their actions, that's correct. And we didn't have an interview
25 process.
Barrett - cross
802
1 Q. Well, in the Wollowick & McNamara study, there was no
2 written short answer test, was there?
3 A. For the in-basket you're referring to?
4 Q. That's correct.
5 A. No, they are rated from the interview process.
6 Q. That's in -- the in-basket process was involved dealing
7 with internal and external correspondence which has accumulated
8 since its predecessor was assigned to another position; is that
9 right?
10 A. Yes.
11 Q. And the in-basket in this article was rated by a subjective
12 scorer; is that right?
13 A. It was rated. I'm not sure who it was by, who actually did
14 the ratings, but it was rated, yes.
15 Q. And it rated characteristics of oral communication,
16 planning and organizing, self-confidence, written
17 communications, decision making, risk taking and administrative
18 ability; is that right?
19 A. Yes.
20 Q. That's entirely different than answering written short
21 answer questions; isn't that correct, Doctor?
22 A. Not necessarily, no.
23 Q. Do you have any data that shows answering written short
24 answer questions is similar to being rated on oral
25 communications, planning and organizing, self-confidence,
Barrett - cross
803
1 written communications, decision making, risk taking and
2 administrative responsibility?
3 A. First, we make no attempt in the in-basket to rate --
4 Q. Dr. Barrett --
5 A. -- oral communication --
6 Q. -- my question was: Do you have any data which shows that
7 what's measured by your in-basket is the same as what's
8 measured in the Wollowick & McNamara study?
9 A. I never conducted a study with their in-basket that related
10 to our in-basket.
11 Q. Okay.
12 And you've never conducted a study with your in-basket
13 with any criterion or performance measure; have you, Doctor?
14 A. We have never taken the Chicago sergeant's in-basket and
15 related it to any other performance measure; that's correct.
16 Q. Have you tested it with garbage collectors?
17 A. No, we have not used the Chicago sergeant's in-basket with
18 garbage collectors.
19 Q. Okay.
20 Now, one of the other studies about in-baskets that
21 you referred to in Appendix R is and article by Turnage &
22 M-u-c-h-i-n-s-k-y, published in 1984; is that correct?
23 A. Yes.
24 Q. And at some point in your career, did you have occasion to
25 read that article?
Barrett - cross
804
1 A. Yes.
2 Q. Now, in Appendix R-4, you report the results of that
3 article by saying, "There was an N of 799." Does that mean the
4 article reported data for 799 subjects?
5 A. Yes.
6 Q. And, then, it says, "An R equaled .08"; is that right?
7 A. I would have to check.
8 Q. Why don't you look at R-4.
9 (Brief pause.)
10 BY THE WITNESS:
11 A. Yes, that's shown in the table.
12 BY MR. FLAXMAN:
13 Q. What does an R of .08 mean?
14 A. It's a correlation between the -- I assume. I haven't
15 reviewed the article recently; but, as it's here, it's a
16 correlation coefficient.
17 Q. Now, correlation coefficient means the relationship between
18 something and something else; is that --
19 A. Yes.
20 Q. -- what correlation is?
21 And a correlation of .08, is that a low correlation?
22 A. Yes, that's a low correlation.
23 Q. Does that mean that the observed and the measured or --
24 well, what does that mean, a correlation of .08?
25 A. It just means a low association between the test score and,
Barrett - cross
805
1 in this case, a measure of job performance.
2 Q. Does it mean that the test score explains less than one
3 percent of the variance in job performance?
4 A. In terms of coefficient of determination, that would be
5 correct.
6 Q. Now, did you cite this Turnage & Muchinsky article because
7 you accepted it as research in your field?
8 A. Yes, it's research in our field.
9 Q. And it's research that was published in a referee journal;
10 is that correct?
11 A. I will have to check and see. I assume it was, but I will
12 check and see.
13 Q. Well, could you tell us where it was published?
14 A. That's what I'm looking for.
15 It was published in the Journal of Applied Psychology.
16 Q. Is that a recognized journal in your field?
17 A. Yes, it is.
18 Q. Okay.
19 And when you cited this article by Turnage and
20 Muchinsky, did you understand that a direct and literal
21 interpretation of the findings of Turnage and Muchinsky would
22 cause one to question seriously the value of assessment centers
23 to forecast the future job success of employees?
24 MS. GLINK: I object to the form of the question.
25 THE COURT: Objection overruled. He may answer.
Barrett - cross
806
1 BY THE WITNESS:
2 A. This was one piece of information we used in a
3 meta-analysis, as I recall. I don't recall the exact details
4 of that study at this point in time.
5 BY MR. FLAXMAN:
6 Q. Well, is it correct --
7 THE COURT: Could I ask what a meta-analysis is? You
8 may have explained it, but I am dealing with lots of new
9 words. It is like a new language here. So, if that could be
10 clarified, what he meant by saying it was a meta-analysis.
11 MR. FLAXMAN: Well, let me come back to that after, if
12 I may --
13 THE COURT: All right. I just -- I do not have any
14 idea what the words mean. That is why I am asking.
15 MR. FLAXMAN: Well, all right. Well, I'll accommodate
16 the finder of fact, Judge.
17 BY MR. FLAXMAN:
18 Q. The Weighting Rationale Report, Appendix R, does that
19 report, among other things, a meta-analysis?
20 A. Yes.
21 Q. Now, is a meta-analysis a technique for combining the
22 results of many studies into one result?
23 A. Yes, it is.
24 Q. And, for example, if you had a hundred studies dealing with
25 caffeine usage and suicide, you could get the data from all of
Barrett - cross
807
1 those studies and combine those and produce one overall result;
2 is that right?
3 A. That's the general idea, yes.
4 Q. And meta-analysis is used often in analyzing data
5 information about drugs and medication?
6 A. Yes, it's used in that area.
7 Q. And it's also used by you in your work in industrial
8 psychology; is that right?
9 A. Yes.
10 Q. And there are people who believe meta-analysis is a good
11 way to do industrial psychology analyses; is that correct?
12 A. Well, it's one technique that's used in industrial
13 psychology.
14 Q. And is the profession of industrial and organizational
15 psychology divided on the utility of meta-analysis?
16 A. In some sense, yes.
17 Q. Okay.
18 Now, when you do a meta-analysis, is it important to
19 gather all of the available data that has previously been
20 acquired on whatever it is you're studying?
21 A. Well, it's according to what your purpose is, but you do
22 try at that point in time to gather as much as you can.
23 Q. And you look at data that has been in published studies; is
24 that right?
25 A. Yes.
Barrett - cross
808
1 Q. And you look at data that's been in unpublished studies; is
2 that right?
3 A. Yes.
4 Q. And you try to combine it all together; is that right?
5 A. Yes.
6 Q. Now, you did a meta-analysis as part of your work in
7 preparing the 1993 sergeant's test; is that right?
8 A. Not really in preparing the test. We didn't do a
9 meta-analysis preparing the test. It was after the test was
10 prepared, as I recall, we did the meta-analysis, I think it
11 was.
12 Q. Well, anyway, in Plaintiffs' Exhibit 80, you said that,
13 "The decision that the three components would be weighted
14 equally was based upon a simulation conducted by Barrett &
15 Associates to determine weighting procedures for the
16 promotional process"; is that right?
17 A. That was one piece of evidence we used, that is correct.
18 Q. And that simulation was -- involved a meta-analysis; is
19 that right?
20 A. Yes, that was one element to the simulation, was a
21 meta-analysis.
22 Q. Now, when you looked at the Turnage & Muchinsky article,
23 were you gathering data to be used in the meta-analysis?
24 A. It was one article we did use, yes.
25 Q. And do you recall that -- and did you read the entire
Barrett - cross
809
1 Turnage & Muchinsky study?
2 A. At one point, I'm sure I did.
3 Q. And when you read that, did you come across the statement
4 that, "The profession of industrial and organizational
5 psychology seems divided on the value of assessment centers"?
6 A. I don't recall that exact term, no.
7 Q. Well, let me show you what's been marked as Plaintiffs'
8 Exhibit 5.
9 (Document tendered to the Court and witness.)
10 BY MR. FLAXMAN:
11 Q. Is Plaintiffs' Exhibit 5, the Turnage & Muchinksy study
12 that you've referred to in your meta-analysis?
13 A. Yes.
14 Q. Let me ask you to look at Page 600, the right-hand column
15 under "Discussion."
16 A. I'm sorry, which page?
17 Q. 600.
18 Do you see where it says, "The profession of
19 industrial and organizational psychology seems divided on the
20 value of assessment centers"? Do you see that?
21 A. Yes, I see that sentence.
22 Q. Do you agree with that sentence?
23 A. Well, you have to read -- I think, read it in total context
24 of the article.
25 Q. Well, my question, Dr. Barrett: Is it correct -- do you
Barrett - cross
810
1 agree with, "The profession of industrial and organizational
2 psychology seems divided on the value of assessment centers"?
3 A. I don't think they're divided on the actual value of
4 assessment centers. I think they are divided perhaps about
5 certain technical details about assessment centers, but I don't
6 -- I wouldn't fully agree that they're divided on the value of
7 assessment centers.
8 Q. Well, when you read the Turnage & Muchinsky article, did
9 you see where they stated, as a result of their research, that
10 there is not a strong relationship between job performance and
11 --
12 MR. FLAXMAN: Well, strike that.
13 BY MR. FLAXMAN:
14 Q. Did you see the first sentence of the next paragraph under
15 "Discussion", "A direct and literal interpretation of our
16 findings would cause one to question seriously the value of
17 assessment centers to forecast the future job success of
18 employees"?
19 A. I'm sorry, what line are you on now?
20 Q. I'm on the second paragraph under "Discussion," the
21 right-hand column on Page 600: "The literal and direct
22 interpretation of our findings would cause one to question
23 seriously the value of assessment centers to forecast the
24 future job success of employees."
25 A. Yes, I see that sentence.
Barrett - cross
811
1 Q. And is that an accurate -- well, do you agree with that
2 statement?
3 A. No, I wouldn't agree because right -- if you go to your
4 left, on Table 2, you can see they have a somewhat dimensional
5 and overall ratings is .35, for example. Above that is some
6 promotional problem ratings .31, the in-basket is .27.
7 So, I think you have to look at the word "literal."
8 And, so, I would say that they've probably overstated their
9 actual findings because their actual data, one can see, is much
10 more positive than what they seem to be saying.
11 So, I'm not sure -- again, you have to read a
12 discussion in total context of what's being said. Again, they
13 say most certainly assessment centers evaluations do predict
14 those who will get ahead in the organization.
15 So, I'm just saying that it's a mistake when you read
16 a scientific article and take little sound bites out of an
17 article or sentences. And, again, a .35 correlation is one
18 that you would expect to find in literature. It's not unusual
19 to find that sort of relationship between an assessment center,
20 for example, and performance.
21 Q. Do you see the left-hand page -- left-hand column, the
22 paragraph that starts at the bottom, "In total, from all the
23 analyses"? Do you see that?
24 A. I'm sorry, where are you now?
25 Q. On Page 600, the left-hand column, the paragraph that
Barrett - cross
812
1 starts at the bottom, "In total from all the analyses, we
2 arrive at the following conclusions"; do you see that?
3 A. Yes.
4 Q. Now, the next sentence that says, "I think neither
5 assessment center evaluations, nor traditional external
6 predictor variables are strongly related to actual job
7 performance," is that another sound bite, Dr. Barrett?
8 A. Yes, it is; because, again, you have to take the words in
9 the total context of the study and what's being said, and what
10 the purpose of an assessment center is in terms of what's being
11 done.
12 Q. Do you have any data that relates high scores -- or that
13 relates scores on your type of in-basket exercise to actual job
14 performance?
15 A. We have never taken the police sergeant in-basket and
16 related it empirically using a criterion-related study to any
17 measure of job performance.
18 Q. Now, you gave the same type of in-basket that was used in
19 Chicago in some other municipalities; isn't that correct?
20 A. We have used the in-basket approach in different
21 jurisdictions, yes.
22 Q. And did you use it in Akron, Ohio?
23 A. As I recall, we did, yes.
24 Q. When you used it in Akron, Ohio, did you ever relate scores
25 on the in-basket with performance on the job?
Barrett - cross
813
1 A. I don't recall that we did.
2 Q. Now, when you gave the in-basket in Akron, Ohio, you gave
3 it to 126 police sergeants; is that right?
4 A. I don't -- to be honest about it, I don't -- what date are
5 you referring to?
6 Q. Well, in your Appendix R-4, Table 1 relates to Akron police
7 sergeant's exam; do you see that?
8 A. Are we through with this?
9 Q. Yes.
10 A. I'm on Page R-4. You mean, under "3"? Is that what you're
11 referring to? 3-C is what you're referring to?
12 Q. I'm on Page R-4.
13 A. I am, too.
14 Q. Okay.
15 Do you see where it refers to Akron police sergeant's
16 exam?
17 A. And I'm referring to 3-C; is that what you mean? This 126,
18 R equals .39? I am at 3-C on Page R-4. It says, "Akron police
19 sergeant's exam: N equals 126, r equals .39"; is that what you
20 mean.
21 Q. Well, no, I'm looking -- let's start at the top. The top
22 has -- shows a correlation between the written test and oral
23 briefing?
24 A. Okay. I'm confused. I thought -- I'm sorry. I thought
25 you were talking about the in-basket. I'm sorry.
Barrett - cross
814
1 Q. Let's look at Page R-4.
2 On Page R-4, do you see a table there that's called,
3 "Table 1, Correlations Used in Meta-analysis"?
4 A. Yes.
5 Q. And under that table, there's a number "1," and it's
6 called, "Correlation Between the Written Test and the Oral
7 Briefing"?
8 A. Yes.
9 Q. And, then, under that, there's, "See Akron police
10 sergeant's exam"?
11 A. Yes.
12 Q. Now, on that Akron police sergeant's exam that's referred
13 to in Table 1, did you administer a written job knowledge test?
14 A. Yes.
15 Q. And was that the same type of test that was used in the
16 City of Chicago in 1993 or sergeant's promotional test?
17 A. When you say "type," I assume you mean it was a job
18 knowledge test. The knowledge being tapped is quite different,
19 and the items were all different.
20 Q. Was it a multiple-choice test?
21 A. Yes.
22 Q. Was it a test that was derived from the written rules and
23 regulations and statutes and ordinances that pertain to the job
24 of an Akron police sergeant?
25 A. I don't recall the details, but I assume it did, yes.
Barrett - cross
815
1 Q. Okay.
2 Was there also an in-basket on the Akron police
3 sergeant's exam?
4 A. Yes.
5 Q. Was that the same type of in-basket that was used in the
6 City of Chicago sergeant's promotional test?
7 A. In the sense that it did measure for the decision making,
8 it would be the same type. It would different items, different
9 exercises, different content.
10 Q. Was the format the same? Did you get many pieces of paper
11 to review for a period of time and then multiple-choice
12 questions to answer in a particular amount of time?
13 A. Yes.
14 Q. How many multiple-choice questions were there in Akron?
15 A. I don't recall.
16 Q. And do you know how long a time was allowed in Akron?
17 A. I don't recall.
18 Q. Okay.
19 Now, when you gave the in-basket test to the Akron
20 police sergeants, did the whites do better as a group than the
21 blacks?
22 A. At this point, I don't recall the exact analyses or what
23 the differences were.
24 Q. Well, let me show you what's been marked as Plaintiffs'
25 Exhibit 79.
Barrett - cross
816
1 (Document tendered to the Court and witness.)
2 BY MR. FLAXMAN:
3 Q. Have you ever seen this before?
4 A. It appears to be Defendant's Response to Plaintiffs' Fourth
5 Request to Admit.
6 Q. Now, my copy has a poorly Xeroxed Page 8. Is your Page 8
7 any better, or is it also deformed or incomplete?
8 A. It's incomplete. There's just --
9 Q. Well, can you see a portion --
10 THE COURT: It slices off --
11 MR. FLAXMAN: Yes.
12 BY MR. FLAXMAN:
13 Q. Do you see a portion of your signature on Page 8?
14 A. I see an e-t-t.
15 Q. And is that your e-t-t?
16 A. I wouldn't want to testify to that, but it looks like it.
17 Q. Okay.
18 Did you review -- is Plaintiffs' Exhibit 79,
19 Defendant's Response to Plaintiffs' Fourth Request to Admit,
20 did you review it before you signed it?
21 A. Yes.
22 Q. Okay.
23 Now, when you reviewed it, did you check the
24 correctness of the assertions that had been contained in the
25 request to admit?
Barrett - cross
817
1 A. If you mean if I personally ran the data myself, no. If
2 you mean if I read the material and reviewed it, yes.
3 Q. Okay.
4 And was it correct, then, the assertion that's in
5 Paragraph 2(b), that, "Scores on the in-basket component of the
6 Akron police sergeant data ranged from 7 to 28, the median
7 score was 20, 38 percent of the white applicants received
8 scores below 20, and 24 percent" -- "24 of the 31
9 African-American applicants, or 77 percent, received scores
10 below 20"?
11 THE COURT: What page is this on now?
12 MR. FLAXMAN: Page 2.
13 THE COURT: Page 2.
14 MS. GLINK: Your Honor, I am going to object on
15 relevancy grounds. I don't see what -- why it's relevant
16 whether that examination had adverse impact when adverse impact
17 is not even at issue in this case. I just don't see the
18 relevance of it.
19 THE COURT: Well, I am going to overrule the
20 objection, but subject to your bringing a subsequent motion to
21 strike, once we have gone through this. So --
22 BY MR. FLAXMAN:
23 Q. Were those numbers --
24 THE COURT: -- you may proceed now.
25 BY MR. FLAXMAN:
Barrett - cross
818
1 Q. Those numbers were correct; weren't they, Dr. Barrett?
2 A. As far as I know, those numbers are correct.
3 Q. Now, in your meta-analysis on Page R-4, you relied on a
4 correlation between scores on the written test and the
5 in-basket on the Akron police sergeant's exam; is that right?
6 A. I'm sorry, what page are you on now?
7 Q. Page R-4 of your meta-analysis.
8 A. Okay, I have R-4 now.
9 Q. Okay.
10 You relied on a correlation of scores between the
11 written test and the in-basket on the Akron police sergeant's
12 exam under 2(c) of --
13 A. Yes, that is a correlation we did use in the meta-analysis.
14 Q. And that correlation was a relatively high correlation; is
15 that right?
16 A. Well, it's usually called a medium correlation. Yes, it's
17 .54.
18 Q. Now, when you administered the written test and the
19 in-basket in the City of Chicago for the police sergeant's
20 test, did you also compute the correlation between the written
21 job knowledge test and the in-basket?
22 A. I assume we did.
23 Q. Was it .54?
24 A. I don't recall what it was.
25 Q. Was it minus .04?
Barrett - cross
819
1 A. I don't recall what it was.
2 Q. Well, would you look at Page 146 of your report?
3 THE COURT: That is Defendant's Exhibit D --
4 MR. FLAXMAN: D.
5 THE COURT: -- as in dog?
6 MR. FLAXMAN: Yes.
7 BY MR. FLAXMAN:
8 Q. Does that show a correlation between the written job
9 knowledge examination and the in-basket simulation?
10 A. Yes, it does.
11 Q. And is it .52?
12 A. No, it's not .52.
13 Q. What is it, Dr. Barrett?
14 A. It's a negative .04.
15 Q. Does that mean that the written job knowledge test and the
16 in-basket test in Akron were measuring different things than
17 the written job knowledge test and in-basket in Chicago?
18 A. Well, it's difficult to compare the two because there's
19 something called restriction of range. In Akron, we had
20 everyone take the oral examination and everyone took the
21 written test.
22 In the City of Chicago, we had, of the 4700 people who
23 took the written test, 1937 who took both the written job
24 knowledge test -- I'm sorry, that would not be true. This is
25 the written test and -- no, this -- yes, I'm sorry.
Barrett - cross
820
1 THE COURT: You have got to start over because you
2 lost me.
3 THE WITNESS: Okay. Lost me, too.
4 BY THE WITNESS:
5 A. So, I'm saying what we have here is a restricted sample in
6 the sense of we don't have the total number of people. We are
7 looking at a sub-sample here of 1937 out of 4,700. All this
8 shows is that, basically, a written job knowledge exam is
9 measuring something different from the in-basket exam.
10 BY MR. FLAXMAN:
11 Q. In Chicago, it's measuring something different; is that
12 right?
13 A. Well, no. Again, don't confuse the issue with what I
14 said. In Akron, we used a total sample. That .54 is the total
15 sample score.
16 Q. Could you have computed the correlation for the total
17 sample in Chicago?
18 A. We could have. In effect, I think we did somewhere.
19 Q. Do you know what it is?
20 A. No, I don't recall offhand what it is.
21 Q. Do you know if it's reported in Defendant's Exhibit D?
22 A. You mean, my report?
23 Q. Right.
24 A. I don't think it's there.
25 Q. Is that something that's important?
Barrett - cross
821
1 A. I don't -- not in the context of whether or not the test is
2 valid, no, it's not important.
3 Q. Okay.
4 Now, when you made that decision that only the top
5 1937 candidates would go on to take the oral test, were you
6 making a judgment that the top 1937 were qualified and everyone
7 else was not qualified to be promoted?
8 A. No.
9 Q. Well, how did you make that judgment that the top 1937
10 would go on to take the oral part of the test?
11 A. It was based upon a number of factors, and the factors were
12 in terms of how many people we could administer an oral
13 examination to; second, the cost of administration; and, third,
14 we let everyone go through who had any probability of being
15 promoted from the list.
16 In other words, we took the -- we assumed, for
17 example, that the person who was the lowest scoring on the
18 combined in-basket and written test, what would happen if they
19 got a perfect score on the oral? Would this be, in effect, a
20 high enough score to give them some probability of being
21 promoted?
22 Q. Now, when you made that judgment that the top 19 -- top
23 2,000 or so would go on to take the oral, you had already
24 decided that promotions would be made in rank order from the
25 results of the test; hadn't you, Dr. Barrett?
Barrett - cross
822
1 A. That was our assumption or analysis at that point in time,
2 that it would probably be rank order, yes.
3 Q. And that was your assumption at the time you submitted your
4 proposal to the City of Chicago back in March of 1993; wasn't
5 it, Dr. Barrett?
6 A. It probably was our assumption it will be rank order, yes.
7 Q. Now, you also administered the same type of in-basket to
8 Akron fire lieutenants; is that right?
9 A. Yes.
10 Q. And did you ever do any investigation to determine if there
11 was a relationship between scores on the in-basket and
12 performance as an Akron fire lieutenant?
13 A. No.
14 Q. Okay.
15 And you also administered, did you not, the same type
16 of in-basket to persons employed by the Washington Metropolitan
17 Area Transit Authority; is that correct?
18 A. That's correct.
19 Q. That was a sample of 34 people; is that right?
20 A. Yes.
21 Q. And did you ever do any investigation to determine whether
22 or not there was a relationship between scores on the in-basket
23 and performance in whatever -- well, let's go back.
24 What position was it on the Washington Metropolitan
25 Area Transit Authority for which you were giving tests that's
Barrett - cross
823
1 reported in Appendix R-4?
2 A. As I recall, it would be transit police.
3 Q. Was it an entry-level position?
4 A. I don't recall at this point. I think it was the
5 sergeant's position.
6 Q. Okay.
7 Well, let's -- did you ever do any work to relate
8 scores on the Washington Metropolitan Area Transit Authority
9 sergeant's in-basket that you administered and performance as a
10 sergeant?
11 A. No.
12 Q. The oral component of the 1993 City of Chicago sergeant's
13 exam, you -- is that similar in format to the examination that
14 you administered for Akron police -- for the Akron police
15 sergeant's exam?
16 A. Yes.
17 Q. Did you ever do any investigation to relate scores on that
18 oral component of the Akron police sergeant's exam with
19 performance on the job?
20 A. No, we didn't -- we did not perform a criterion-related
21 study.
22 Q. And that oral examination that's part of the police --
23 Chicago police sergeant's examination, is that also similar to
24 the oral examination you administered for Akron fire
25 lieutenant's promotional exam?
Barrett - cross
824
1 A. Only in the sense that there would be the same general
2 format, the ratings, but the content would be quite different.
3 Q. Did you ever do an investigation to relate scores on the
4 oral examination for Akron fire lieutenants with performance on
5 the job?
6 A. We have conducted no criterion-related study on the oral
7 for fire lieutenants.
8 Q. And did you -- is the oral examination that you
9 administered for the Chicago sergeant's promotional test
10 similar to the oral examination you administered for the
11 Washington sergeant transit people -- let me get that right --
12 the Washington Metropolitan Area Transit Authority sergeants?
13 A. It was similar in the sense that there was a presentation
14 which is rated. The content would be quite different.
15 Q. And did you ever do any study to relate performance on that
16 oral examination to performance on the job at the WMATA?
17 A. We performed no criterion-related study at the WMATA.
18 Q. Now, you talked about reliability, and you said you did
19 measures of reliability; is that right?
20 A. Yes.
21 Q. Now, on the written multiple-choice job knowledge test, did
22 you do something called split-half reliability?
23 A. Yes.
24 Q. And could you explain very briefly what that is?
25 A. Briefly, it is taking one-half the test and correlating it
Barrett - cross
825
1 to the other half of the test. Correlating one-half the test
2 to the second half of the test.
3 Q. And is that the same kind of reliability measure that you
4 used on the in-basket?
5 A. No, we used the raters' reliability for the in-basket.
6 Q. And what is raters' reliability?
7 A. How consistent the raters rated the individuals who are
8 being rated.
9 Q. The in-basket --
10 A. I'm sorry, the in-basket -- I'm sorry, I thought you were
11 talking about the oral this time. Are we talking about the
12 oral or the in-basket?
13 Q. Okay.
14 Well, let's -- so the record is clear, when you talked
15 about rater reliability, you were talking about the oral; is
16 that right?
17 A. Yes.
18 Q. And the oral, you had two raters each listening to the same
19 tape; is that right?
20 A. I thought it was three raters.
21 Q. Okay.
22 Three raters listening to the same tape; is that
23 right?
24 A. Yes.
25 Q. And, then, you did a statistical analysis to see how
Barrett - cross
826
1 closely each of the three raters agreed on the way they were
2 rating the tape?
3 A. Yes.
4 Q. Now, on the in-basket, the 60 multiple-choice questions
5 based on the materials in the in-basket, how did you measure
6 reliability?
7 A. Again, I think it would be the split-half technique.
8 Q. Now, when you use -- is the split-half technique also
9 called a single trial reliability coefficient?
10 A. Yes. You only have one trial.
11 Q. Okay.
12 Is it correct that a single trial reliability
13 technique is not applicable to a speeded test?
14 A. No.
15 Q. What is a speeded test?
16 A. That is a test where the score depends solely upon how fast
17 you do something.
18 For example, you might see how fast you can make
19 check-marks on a piece of paper.
20 Q. Well, there was a time limit for answering those 60
21 multiple-choice questions on the in-basket; is that right?
22 A. Yes, there's a time limit on the test.
23 Q. Excuse me?
24 A. Yes, there is.
25 Q. And do you know what that time limit was?
Barrett - cross
827
1 A. I think it was an hour and 15 minutes, but I sometimes get
2 different administrations confused.
3 Q. Well, what kind of investigation or work did you do --
4 MR. FLAXMAN: Strike that.
5 BY MR. FLAXMAN:
6 Q. Did you determine how long the test should take to complete
7 --
8 MR. FLAXMAN: Strike that.
9 BY MR. FLAXMAN:
10 Q. Did you determine how much time would be allowed for
11 completion of the in-basket test?
12 A. At some point in time, yes, we made a determination.
13 Q. And did you pilot test that examination on Chicago police
14 officers to see how long it would take them to complete it?
15 A. No, we did not.
16 Q. Did you -- how -- did you pilot test it on graduate
17 students to see how long it would take them to complete it?
18 A. As I recall, we did, yes.
19 Q. By "graduate students," you mean, people who are psychology
20 graduate students at the University of Akron?
21 A. I think most were, yes.
22 Q. How many --
23 A. I don't recall.
24 Q. -- graduate students were pilot tested for this?
25 A. I don't recall how many.
Barrett - cross
828
1 Q. Is this recorded, this pilot testing mentioned in your
2 report, anywhere?
3 A. I assume it is. I'm not sure.
4 Q. Could you --
5 MR. FLAXMAN: This might a good time for a morning
6 break, if Dr. Barrett could find for us, if it's there.
7 THE COURT: All right.
8 We are at 11:15, and I was going to see how you wanted
9 to handle the noon break. Partly, that will depend, I suppose,
10 on how far along we are. What would you recommend for --
11 MR. FLAXMAN: I might be done at a reasonable noon
12 break.
13 THE COURT: If we go until, say, 1:00 o'clock, is that
14 what you suggest?
15 MR. FLAXMAN: I think there is a good chance I will be
16 done, yes.
17 THE COURT: All right.
18 In any event, we will take a short break at this time,
19 then, and we will resume shortly.
20 (Whereupon, a recess was taken.)
21 THE COURT: All right. You may proceed.
22 BY MR. FLAXMAN:
23 Q. Dr. Barrett, did you have a chance to look through your
24 report, Defendant's Exhibit D?
25 A. Yes, I did.
Barrett - cross
829
1 Q. Is there something in there that describes the pilot
2 testing of the in-basket?
3 A. Yes, there is.
4 Q. And where is that?
5 A. On Page 104 and 105, I believe.
6 THE COURT: Page 104 --
7 THE WITNESS: And Page 105.
8 THE COURT: Okay, and that's Defendant's Exhibit D.
9 BY MR. FLAXMAN:
10 Q. How many pilot test participants were there on September
11 12, 1993?
12 A. We didn't report the number.
13 Q. And how many -- do you know how many there were on
14 September 18th, 1993?
15 A. We didn't report the number in the report.
16 Q. And --
17 THE COURT: What you are saying is you don't recall?
18 THE WITNESS: No, I don't recall, but it's not in the
19 report. We mentioned we -- pilot testing, but we don't say the
20 number who were pilot tested.
21 BY MR. FLAXMAN:
22 Q. Now, Page 105 states that, "On September 12th, the pilot
23 testing was just answering the questions without the in-basket
24 materials"; is that correct?
25 A. That's correct.
Barrett - cross
830
1 Q. And that's not a procedure that would give you some
2 indication of how long it should take a Chicago police officer
3 to answer the 60 multiple-choice questions; is that right?
4 A. That's right.
5 Q. And in September 18th -- and what -- well, could you
6 describe for us the pilot testing that occurred on September
7 18th?
8 A. That's where they had the full materials to review and they
9 answered the questions.
10 Q. And do you know how many questions they were required to
11 answer on September 18th?
12 A. Well, we had -- we had 60 questions at that point in time.
13 Q. And how much time were they allowed to answer the 60
14 questions?
15 A. As I recall, we gave them whatever time they wanted,
16 basically, to answer the questions.
17 Q. So, you didn't set the time at 75 minutes to answer the 60
18 questions, did you?
19 A. Well, we got some -- an idea of what -- how long it took
20 for them to do it -- actually do it.
21 Q. Was there a technique that was available to you in
22 September 18th, 1993, to get some indication of how much time
23 should be allowed to answer the 60 multiple-choice --
24 60-question multiple-choice test?
25 A. You mean, beyond what we actually did or --
Barrett - cross
831
1 Q. Well, no, was there a technique available to you that could
2 have given you the indication of how much time should be
3 required to answer the test?
4 A. Not beyond what we actually did.
5 Q. Well, could you have allowed --
6 THE COURT: Could I interject something, so that you
7 could ask about that or the other side could ask about it? I
8 understand that, because of some testing that I have had in my
9 own educational background and the like, the fact that some
10 tests are set so that you cannot possibly answer them all.
11 They want to see how far you can go on the testing of it.
12 Is it -- I do not know whether it is -- there is any
13 standards as to whether it is more desirable to have a test
14 that everybody can complete or whether it is more desirable to
15 have the test that cannot be completed.
16 I just throw that out because -- this is an area,
17 obviously, that I am the least expert of anybody in the room,
18 and, so, I need all the help I can get from the witnesses.
19 BY MR. FLAXMAN:
20 Q. Well, was the in-basket test designed to be a speeded test?
21 A. Not in the traditional sense of speeded, no. It had a time
22 limit, but it was not a speeded test in how it's traditionally
23 defined.
24 Q. Just so it's clear, is a speeded test one in which
25 individual differences depend entirely on speed of performance?
Barrett - cross
832
1 A. That would be a good general definition.
2 Q. And is a speeded test constructed with items of uniformly
3 low difficulty all of which are well within the ability level
4 of the persons for whom the test is designed?
5 A. That would be a good general definition.
6 Q. And is the time limit on a speeded test made so short that
7 no one can finish all the items?
8 A. Yes.
9 Q. And on a speeded test, does each person's score reflect
10 only the speed from which he or she worked?
11 A. That would be the usual definition.
12 Q. Now, is the opposite of a speeded test -- or not the
13 opposite, at the other end of the spectrum, something that's
14 known as a power test?
15 A. That's correct.
16 Q. And is it correct that a power test has a time limit long
17 enough to permit everyone to attempt all items?
18 A. Well, sometimes there's no time limit at all on a power
19 test. So, it's not -- yes, the answer would probably be yes.
20 Q. And on a power test, is the difficulty of the items steeply
21 graded, so that the test includes some items too difficult for
22 anyone to solve, so that no one could get a perfect score?
23 A. That would be a typical definition of a power test, yes.
24 Q. Now, was the in-basket test intended to be a power test?
25 A. No.
Barrett - cross
833
1 Q. Was the in-basket test intended to be a speeded test?
2 A. No.
3 Q. Was it important to determine how long it should take a
4 Chicago police officer to complete the 60 questions on the
5 in-basket test before the test was administered?
6 A. It was important to have a reasonable amount of time for
7 people to complete most of the items on the test.
8 Q. And did you do any pilot testing to determine a reasonable
9 amount of time?
10 A. Yes, a second pilot testing on the 18th, 9-18-93, helped us
11 determine -- to make that determination.
12 Q. Well, you -- September 18th, 1993, you told us, I think,
13 that there was no time limit; isn't that correct?
14 A. Yeah, but we timed how long it took people to complete.
15 Q. And these people who were completing it, these were the
16 graduate students; is that right?
17 A. Yes.
18 Q. Now, was there a technique that was available to you back
19 in September 18th, 1993, to provide some measure of whether the
20 time you were allowing was too short or too long?
21 A. I can't think of any practical technique beyond what we
22 actually did. There is, of course, under ideal conditions,
23 which, of course, we did not have, you could have taken a large
24 sample of Akron -- I'm sorry, of Chicago police officers and
25 administered the test to them and obtained some reasonable
Barrett - cross
834
1 measure of -- for a time for that test.
2 But, of course, that would not be practical because
3 then the test contents would be revealed and the validity of
4 that in-basket will be destroyed. So, we had no practical or
5 feasible way that I know of beyond what we actually did.
6 Q. You're familiar with split-half techniques of measuring
7 test reliability; is that right?
8 A. Yes.
9 Q. And you used the split-half technique where you split half
10 the questions; is that right?
11 A. Yes.
12 Q. Could you also have used a split-half technique where you
13 split the test in terms of time, rather than in terms of items?
14 A. I'm not sure what you mean.
15 Q. Well, could you have administered one-half of the test with
16 a separate time limit, and then administer the second half of
17 the test with a separate time limit?
18 A. I'm not sure what -- what value that would be in this type
19 of test.
20 Q. Well, are you telling us that that is not an alternate
21 method of determining the reliability of a test, by splitting
22 it in terms of time, rather than in terms of items?
23 A. I don't recall a technique like that, I guess. I'm not
24 sure what you're saying, but I don't --
25 Q. Well --
Barrett - cross
835
1 THE COURT: What do you mean by "reliability of
2 test"?
3 MR. FLAXMAN: Okay.
4 BY MR. FLAXMAN:
5 Q. You did a measure of reliability where you compared the
6 answers on 30 questions with the answers on another 30
7 questions; is that right?
8 A. And this was after the test had already been administered
9 to the 4700 people. It was after the fact.
10 Q. Right.
11 And you did some kind of statistical analysis looking
12 at the odd questions and the even questions?
13 A. Correct.
14 Q. And is that -- you did that to give you some estimate of
15 how people would do if they took the test on Day One and then
16 took it, again, on Day Seven --
17 A. Yes.
18 Q. -- is that right?
19 A. Yes.
20 Q. You could also have done -- theoretically, you could have
21 done a test-retest approach to determining reliability; is that
22 right?
23 A. In theory, but not in practicality for that type of test.
24 Q. Now, what I was asking you about, Dr. Barrett, was to take
25 a -- take the in-basket test and administer the first 30
Barrett - cross
836
1 questions in one hour, and then, at the end of one hour, you
2 tell the applicant to answer the next 30 questions. So, you
3 are splitting the test up by time. Would that have been
4 possible to have administered in the City of Chicago?
5 A. I'm still not -- I guess I'm missing what you're driving
6 at. Anything is possible.
7 Q. Well --
8 A. But you're making some assumptions about the two.
9 MS. GLINK: Your Honor, I am going to object to this
10 line of questions. What's at issue here is what we did, not
11 the hundred different ways we could have developed or
12 administered this test. And I'm not clear where this line of
13 questioning is going or the relevancy at all.
14 THE COURT: Well, that is probably part of the design
15 of the cross-examination. You are not necessarily to know
16 yet. I am acting on faith that we have good professionals on
17 both sides here.
18 I am going to overrule the objection at this point,
19 subject to subsequent motion to strike.
20 But I do not understand your question. Are you
21 referring to, in effect, that everybody taking the test would
22 have this or are we talking about some sort of sample group?
23 MR. FLAXMAN: No.
24 BY MR. FLAXMAN:
25 Q. I'm saying, when you gave the test to 4700 different
Barrett - cross
837
1 people, you could have given them two answer sheets -- the
2 first answer sheet is Hour One, and second answer sheet at Hour
3 Two; in Hour One you answer the first 30 questions, in Hour Two
4 you answer the second 30 questions.
5 That could have been done; couldn't it, Dr. Barrett?
6 A. Anything is feasible, and what you described could have
7 been done.
8 Q. And that would be a way of splitting the test in terms of
9 time, rather than in terms of items; is that right?
10 A. I never heard of a procedure like this, so it's hard to
11 respond to you. I don't understand --
12 THE COURT: He did not split it according to items.
13 MS. GLINK: That's right, your Honor. The reliability
14 coefficient is something that was done after the fact. That's
15 the testimony.
16 MR. FLAXMAN: I'm asking him about this alternative
17 method of reliability, which he --
18 THE COURT: Oh, okay. I see now.
19 BY MR. FLAXMAN:
20 Q. So, are you telling us that, in your view, what I have
21 described by splitting the test into two 30-question tests
22 administered with separate time limits is, in your opinion, not
23 an alternate method of determining the reliability of the test?
24 A. I've never heard of it being done that way, no.
25 Q. Are you familiar with any generally-accepted texts on
Barrett - cross
838
1 testing?
2 MR. FLAXMAN: I will strike that.
3 BY MR. FLAXMAN:
4 Q. Are there any generally-accepted texts on testing?
5 A. Yes.
6 Q. Is there a book by someone called an Anastasi,
7 A-n-a-s-t-a-s-i?
8 A. Yes, there is.
9 Q. Is that a generally-accepted text on testing?
10 A. It's a text on testing, yes.
11 Q. Is it generally-accepted in your field?
12 A. It's used quite often in our field.
13 Q. Have you ever looked at it?
14 A. It's been a number of years, but I have looked at it in the
15 past.
16 Q. Is Anastasi a recognized authority on psychological
17 testing?
18 A. Yes, she is.
19 Q. Okay.
20 And do you agree that, "If individual differences in
21 test scores depend not on errors, but on speed, the measure of
22 reliability must obviously be based on consistencies and speed
23 of work"?
24 MS. GLINK: Your Honor, can he please show the witness
25 what he's looking at? This is a 400-page book that he's
Barrett - cross
839
1 referring to one piece of one line out of a book.
2 THE COURT: Yes, if the witness needs to look at it.
3 I have difficulty comprehending exactly what that
4 question was, again, but we will leave it up to the witness
5 whether he heard the quote and needs further assistance or not.
6 BY THE WITNESS:
7 A. Yeah, I would like to see the quote.
8 BY MR. FLAXMAN:
9 Q. Oh, sure.
10 MR. FLAXMAN: Should we mark this as an exhibit?
11 THE COURT: Well, you are not going to be offering the
12 treatise in evidence.
13 MR. FLAXMAN: No, but -- well --
14 MS. GLINK: We don't have a copy, but we really -- our
15 expert can take a look at it. That's fine.
16 THE COURT: You've identified it in part.
17 "Psychological Testing," is the title, I guess, of this book.
18 MR. FLAXMAN: I'm directing the witness to Page 128,
19 129.
20 (Brief pause.)
21 BY THE WITNESS:
22 A. I think I've read -- I am not sure what paragraph you
23 wanted me to focus on, but I've read down to the middle of Page
24 129. Is that far enough or should I keep on reading?
25 BY MR. FLAXMAN:
Barrett - cross
840
1 Q. Well, do you disagree with the statement in Anastasi that,
2 "Single trial reliability coefficients such as those found by
3 odd, even or Kuder-Richardson techniques are inapplicable to
4 speeded tests"?
5 A. Well, I think you must understand that you left a word out
6 when you wrote -- when you gave that quotation to me before.
7 What they're talking about is a pure speed test versus
8 a pure power test. And the word "pure" is not what you -- is a
9 word you sort of omitted.
10 And, so, what I'm saying is she goes on to explain
11 that the typical test is neither a pure power, nor a pure speed
12 test, and that she is referring now on Page 129 to a speeded
13 test. She says in that first paragraph "speeded test."
14 So, we did not have anything which approached a pure
15 speeded test.
16 Q. Well, how did you determine, Dr. Barrett, that an hour and
17 15 minutes was long enough for a Chicago police officer who is
18 capable of being promoted to sergeant to answer those 60
19 multiple-choice questions?
20 A. Well, we had, certainly, two pieces of information, among
21 others. One would be our pilot test results and, second, our
22 past experience in giving this type of test.
23 Q. Now, this past experience, is that the Akron police
24 sergeants that we talked about?
25 A. That would be in part true, yes.
Barrett - cross
841
1 Q. And that's also the Akron fire lieutenants we talked about?
2 A. Yes.
3 Q. And the WMATA that we talked about?
4 A. Yes.
5 Q. Could you tell us what you did in the Akron police
6 sergeant's examination to determine how long it should take an
7 Akron police officer to complete the multiple-choice questions
8 in the in-basket test used in that exam?
9 A. We had followed generally the same procedure, as I recall.
10 I don't recall the exact details.
11 Q. And that exact procedure is to do nothing; isn't that
12 right, Dr. Barrett?
13 A. I'm confused about that.
14 Q. Okay.
15 Now, how about -- can you tell us what procedures you
16 used to determine how long it should take the fireperson in the
17 Akron fire lieutenant's promotional exam to complete the
18 written in-basket multiple-choice test there?
19 A. Well, again, I don't recall the exact procedure, but I'm
20 making an assumption. We usually do a pilot test.
21 Q. And you do a pilot test with fire lieutenants or fire
22 sergeants?
23 A. I assume we did.
24 Q. Do you know how many -- well, how many did you use in that
25 pilot test?
Barrett - cross
842
1 A. I don't recall.
2 Q. Why did you do a pilot test in the Akron fire lieutenant's
3 promotional test, but not in the Chicago police sergeant's
4 test?
5 A. We had -- I guess, I'm getting confused, again, because I
6 think our testimony was that we did do a pilot test.
7 Q. By "pilot test," you mean, administering the
8 multiple-choice test to graduate students; is that right?
9 A. Yes.
10 Q. And when you administered it, the pilot test for Chicago
11 did not have a time limit; isn't that your testimony?
12 MS. GLINK: Your Honor, I believe that he's
13 mischaracterizing the testimony and he is being argumentative.
14 THE COURT: Well, it is cross-examination. He
15 certainly can be probing, anyway. I will overrule the
16 objection and permit the witness to answer.
17 Do you have the question?
18 THE WITNESS: No, I don't have a question.
19 BY MR. FLAXMAN:
20 Q. The pilot testing of the graduate students for the Chicago
21 sergeant's in-basket test, there was no time limit for how long
22 those graduate students had to answer the multiple-choice
23 questions; was there, Dr. Barrett?
24 A. To the best of my recollection, we obtained the time it
25 took them to finish the examination.
Barrett - cross
843
1 Q. Excuse me?
2 A. We obtained each individual's time to finish is my best
3 recollection at this point in time.
4 Q. And do you know whether each individual finished it in more
5 than an hour and 15 minute