555
1 IN THE UNITED STATES DISTRICT COURT
NORTHERN DISTRICT OF ILLINOIS
2 EASTERN DIVISION
3 ERNEST T. BROWN, et al., )
)
4 Plaintiffs, )
) No. 95 C 1890
5 v. ) Chicago, Illinois
) November 24, 1997
6 CITY OF CHICAGO, ) 9:30 a.m.
)
7 Defendant. )
8 VOLUME 4
9 TRANSCRIPT OF PROCEEDINGS - TRIAL
10 BEFORE THE HONORABLE ROBERT W. GETTLEMAN
11 APPEARANCES:
12 For the Plaintiffs: KENNETH N. FLAXMAN, P.C.
122 South Michigan Avenue
13 Suite 1850
Chicago, Illinois 60603-6107
14 BY: MR. KENNETH N. FLAXMAN
15 and
16 FUTTERMAN & HOWARD, CHTD.
122 South Michigan Avenue
17 Suite 1850
Chicago, Illinois 60603
18 BY: MR. CRAIG FUTTERMAN
19
For the Defendant: MAYER, BROWN & PLATT
20 190 South LaSalle Street
Chicago, Illinois 60603
21 BY: MR. JAMES HOLZHAUER
MR. JEFFREY S. PIELL
22 MS. ANGELA K. DORN
MR. ANDREW NICELY
23
Official Court Reporter: JENNIFER S. COSTALES, CSR, RMR
24 219 South Dearborn Street
Room 1744-A
25 Chicago, Illinois 60604
(312) 427-5351
556
1 (Proceedings heard in open court.)
2 THE CLERK: 95 C 1890, Ernest Brown versus City of
3 Chicago; on trial.
4 THE COURT: Are we ready?
5 MR. HOLZHAUER: Your Honor, we have one brief -- good
6 morning. We have one brief preliminary matter that Mr. Flaxman
7 and I --
8 THE COURT: Dr. Barrett, you can have a seat.
9 MR. HOLZHAUER: Sorry.
10 THE COURT: Go ahead.
11 MR. HOLZHAUER: I was informed after evidence closed
12 last week that the paralegal whose working with us on this
13 matter, one of the paralegals, Colleen Manley, is a good
14 personal friend of Katy Blakey, who I understand is a law clerk
15 of yours. I don't think that presents a problem. Mr. Flaxman
16 agrees, but I thought it would be --
17 THE COURT: Do you want me to fire Katy, is that it?
18 MR. HOLZHAUER: We're certainly not proposing
19 anything like that, sir.
20 MR. FLAXMAN: I doubt that the friendship will have
21 any impact on your decision in this case.
22 THE COURT: Some of my best former friends have
23 appeared before me as lawyers. I don't think those sorts of
24 things can get in the way. As long as they're not parties, I
25 don't have a problem.
557
Barrett - cross
1 MR. HOLZHAUER: Were they former friends before or
2 after they --
3 THE COURT: I chose my words carefully.
4 MR. HOLZHAUER: Thank you, Your Honor.
5 THE COURT: Thank you for informing me. I don't
6 think it's a problem. All right.
7 Dr. Barrett, good morning. You're still under oath.
8 GERALD BARRETT, DEFENDANT'S WITNESS, PREVIOUSLY SWORN
9 CROSS-EXAMINATION (Resumed)
10 BY MR. FLAXMAN:
11 Q. When we left, Dr. Barrett, we were talking about the
12 in-basket as I recall.
13 Was the in-basket exercise intended to measure
14 judgment?
15 A. To some degree, yes.
16 Q. Excuse me?
17 A. To some degree, yes, in terms of analysis of material.
18 Q. Was it intended to measure decision making?
19 A. Yes, they had to make decisions.
20 Q. Isn't it correct, Dr. Barrett, that a content validity
21 approach to test validation is not appropriate for a test which
22 is intended to measure judgment?
23 A. No, that's not correct.
24 Q. Isn't that what the EEOC guidelines say, Dr. Barrett?
25 A. No. I think you're misinterpreting the guidelines and the
558
Barrett - cross
1 standards and professional practice in the field.
2 Q. Well, the guidelines are written down, aren't they,
3 Dr. Barrett?
4 A. Yes.
5 Q. And as a matter of fact, they have been marked as an
6 exhibit, is that right?
7 A. That's correct.
8 Q. And I think they have been marked as exhibit number --
9 well, I don't know, 24 I think.
10 MR. FLAXMAN: Is that correct?
11 THE COURT: Whose exhibit 24?
12 MR. FLAXMAN: Defendant's exhibit 24 -- no. It's not
13 24. 27.
14 MR. PIELL: That's correct.
15 MR. HOLZHAUER: That's correct.
16 MR. FLAXMAN: Should I wait for Your Honor?
17 THE COURT: I have it.
18 MR. FLAXMAN: Okay.
19 BY MR. FLAXMAN:
20 Q. There is sections of EEOC guidelines that deal with content
21 validity, isn't that right?
22 A. Yes.
23 Q. And there is a section that deals with technical standards
24 for content validity, is that right?
25 A. Yes.
559
Barrett - cross
1 Q. And is there a sentence in the guidelines on page 208 of
2 the exhibit that says that, "A selection procedure based upon
3 inferences about mental processes cannot be supported solely or
4 primarily on the basis of content validity"?
5 A. There may be.
6 THE COURT: Can you direct my attention to that?
7 MR. FLAXMAN: Yes, page 208, where on the left column
8 there is a heading "Technical Standards for Content
9 Validation." Let me direct everyone's attention to the next
10 paragraph which starts out, "A selection procedure."
11 BY MR. FLAXMAN:
12 Q. Do you see that sentence that I just read, Dr. Barrett?
13 A. Yes.
14 Q. Have you found that?
15 A. Yes.
16 Q. Do you also see the sentence that says, "Thus, a content
17 strategy is not appropriate for demonstrating validity of
18 selection procedures which purport to measure traits or
19 constructs such as intelligence, aptitude, personality, common
20 sense, judgment, leadership and spatial ability"?
21 A. Yes.
22 Q. So your opinion, Dr. Barrett, is different than EEOC
23 guidelines about content validity measuring judgment, is that
24 right?
25 A. No, it's not.
560
Barrett - cross
1 Q. Okay. And we've established, you've agreed, don't you,
2 that the in-basket measured judgment, is that right?
3 A. Yes. Partly it measures decision making in that context,
4 in the context of specific content valid problems.
5 Q. Well, Dr. Barrett -- could I have the guidelines back?
6 Is it your testimony that the in-basket reflects
7 actual tasks performed by Chicago police lieutenants?
8 A. I'm saying that those are problems in the in-basket which
9 are representative of problems faced by Chicago police
10 lieutenants.
11 Q. Dr. Barrett, my question is: Was the in-basket intended to
12 reflect actual tasks performed by Chicago police lieutenants?
13 A. I'm not sure how you're defining "tasks." It was actual
14 work behavior problems which they face on the job.
15 Q. Well, Dr. Barrett, my question is: Was the in-basket
16 intended to reflect actual tasks performed by Chicago police
17 lieutenants?
18 A. As a simulation it was, a content valid simulation, yes.
19 Q. Dr. Barrett, can you answer my question?
20 A. Well, I'm trying to. See, I'm not sure what you are saying
21 in terms of it certainly was a simulation of the tasks, of the
22 work behavior of the Chicago police lieutenants.
23 Q. Dr. Barrett, my precise question is: Was the in-basket
24 intended to reflect the actual tasks performed by Chicago
25 police lieutenants?
561
Barrett - cross
1 A. It reflects some actual tasks performed by Chicago police
2 lieutenants.
3 Q. It was intended to reflect the actual tasks?
4 A. To some degree, yes. It's a simulation. I'm just saying,
5 you say "actual." I'm not sure what you're meaning. If we're
6 going out in the field in the context of it's not in the sense
7 of -- it is still a test.
8 Q. Well, so it wasn't intended to reflect the actual tasks, is
9 that what you are telling us?
10 A. It was reflected to be a content-valid test reflecting the
11 tasks which are performed by Chicago police lieutenants.
12 Q. Let me rephrase the question. Maybe it will help you.
13 Was the in-basket intended to approximate a realistic
14 work situation for a Chicago police lieutenant?
15 A. It was a simulation of realistic work situations, yes.
16 Q. Well, what do you mean, "simulation"? It means it was
17 different than what lieutenants really do, is that what you are
18 saying?
19 A. No. I'm saying that it is still a test, all right, with
20 all the limitations of a test. It's not actually walking out
21 into the field, if that's what you are implying, and going into
22 a patrol car and watching the behavior. It's not that sort of
23 thing. It is a simulation of the behavior and tasks.
24 Q. Well, when we talked about content validity as a typing
25 test, typing is an actual test done by typists, is that right?
562
Barrett - cross
1 A. That's correct.
2 Q. And that you could have a content-valid test if you give a
3 typing test to someone who is going to be a typist, is that
4 right?
5 A. Yes, you can give a content valid typing test to typists.
6 Q. Well, doesn't content validity require a close link between
7 the test content and the job content?
8 A. Yes. There should be a link between the two.
9 Q. And that to be content valid, the tests should consist of
10 representative samples of tasks, behaviors, or knowledge
11 actually used on the job, isn't that right?
12 A. Yes.
13 Q. And to be content valid, the job domain should be defined
14 through job analysis by identifying the important tasks,
15 behaviors, or knowledges of which the test is a representative
16 sample of, is that right?
17 A. Yes.
18 Q. That's not what the in-basket is, is it, Dr. Barrett? The
19 in-basket is a simulation?
20 A. It is a content-valid test which does simulate those
21 important activities which involve administrative skills of a
22 Chicago police lieutenant.
23 Q. Well, that's different than content validity as it's
24 understood in the SIOP guidelines, isn't that right,
25 Dr. Barrett?
563
Barrett - cross
1 A. No, that's not correct at all.
2 Q. Are you familiar with the SIOP guidelines?
3 A. Yes, I am.
4 Q. Isn't it true that the SIOP guidelines refer to what you
5 are defining as content validity as something that's treated as
6 construct validity?
7 A. No.
8 Q. It's not true?
9 A. Not true at all.
10 Q. Okay. Don't you believe that all validity is construct
11 validity?
12 A. I believe that the basis -- and this is certainly the
13 modern psychometric view -- that the basis of all of our tests
14 are first, identify some constructs in a general sense. And
15 that is reinforced by the standards, for example, on page 11,
16 where it says, "You should not make artificial distinctions
17 between content and constructs." You would go to Messick,
18 1989, which has --
19 MR. FLAXMAN: I move to strike the answer as
20 nonresponsive. I asked him, "Isn't it true that," and I'm
21 getting a lecture.
22 MR. HOLZHAUER: Well, I believe --
23 THE COURT: "Don't you believe" -- and I think he's
24 trying to tell you what he believes. You basically asked him
25 for his beliefs. I think you opened the door to this one.
564
Barrett - cross
1 MR. FLAXMAN: All right.
2 THE COURT: Do you want to finish? Were you
3 finished?
4 THE WITNESS: No, I'm not finished.
5 THE COURT: Go ahead.
6 BY THE WITNESS:
7 A. Messick, 1989, for example, has a treatise on construct
8 validity which takes the modern approach to the issue. And
9 clearly constructs underlie all of our tests.
10 And I wrote a paper in 1992 which is more specific to
11 the area of industrial psychology, also reiterating that point
12 of view. In question 81 of the EEOC guidelines, there is a
13 definition and a distinction which talks about
14 transportability, construct validity, and how these are
15 distinguished from criteria-related validity. And clearly
16 there are differences.
17 But one could argue, for example, based upon the EEOC
18 guidelines, based upon the past research, that I could defend
19 the in-basket we developed solely on construct validity based
20 on the past research.
21 BY MR. FLAXMAN:
22 Q. Doesn't construct validity require an empirical
23 relationship between test scores and actual performance?
24 A. I think, again, you should refer yourself to question 81 of
25 the EEOC guidelines questions and answers.
565
Barrett - cross
1 MR. FLAXMAN: Judge, could I have an answer to this
2 question?
3 THE COURT: Can you answer that yes or no?
4 BY MR. FLAXMAN:
5 Q. Doesn't construct validity require an empirical
6 relationship between test scores and job performance?
7 A. No.
8 Q. Doesn't construct validity require empirical data?
9 A. It's according, again, how you define empirical data and
10 how you define construct validity, because I think what you are
11 saying is that you have the same misconception Dr. York had.
12 He said you have to have a criterion-related study for that
13 job. That is not accurate. And this conforms to guidelines of
14 professional practice.
15 Q. Well, you keep saying that, Dr. Barrett.
16 Now, there aren't any questions in the EEOC
17 guidelines, are there?
18 A. In the questions and answers which came out in 1979 and
19 1980, they clarified the 1978 EEOC guidelines.
20 Q. So you're referring to questions and answers. You're not
21 referring to the EEOC guidelines when you talk about questions,
22 is that right?
23 A. Yes, I'm referring to the questions and answers which were
24 put out by the EEOC to clarify the 1978 guidelines.
25 Q. Now, let's -- are you familiar with the materials that made
566
Barrett - cross
1 up the in-basket?
2 A. In general, yes.
3 Q. Were you responsible for approving those before they were
4 given?
5 A. Yes.
6 Q. And did you conclude that the in-basket is representative
7 of important aspects of performance on the job for which police
8 lieutenants do?
9 A. Yes.
10 Q. Let me ask you to look at the in-basket test, which is
11 Defendant's Exhibit 15. If you turn to page, Bates stamped
12 page 08452, do you see the instructions that were given to the
13 person taking the in-basket?
14 A. Yes.
15 Q. And those instructions tell the person taking the test to,
16 "Assume that you're a watch commander who is taking over at
17 11:30 in the morning for another watch commander who became ill
18 at 11:00 o'clock," is that right?
19 A. Yes.
20 Q. Does your job analysis indicate how often that situation
21 occurs?
22 A. No.
23 Q. Do you have any personal knowledge of whether or not that
24 situation has ever occurred in the history of the Chicago
25 police department?
567
Barrett - cross
1 A. I've been told it has.
2 Q. You've been told it has?
3 A. Yes.
4 Q. Is that documented anywhere in your report that you've been
5 told it has?
6 A. No.
7 Q. Okay. Now, the instructions continue by saying that "You
8 can sort the in-basket materials in" -- there is another
9 instruction that "do not separate any stapled pages." Do you
10 see that? Have you found that, Dr. Barrett?
11 A. Is it on the next page?
12 Q. Yes.
13 A. Yes.
14 Q. Does your job analysis indicate that watch commanders are
15 not permitted to unstaple material -- unstaple materials when
16 they do their work?
17 A. No.
18 Q. Now, just to clarify, when this material was given to the
19 person taking the test, they didn't receive it in the form that
20 you have it, did they, Dr. Barrett? Some of the pages were
21 stapled together, is that right?
22 A. I don't recall exactly, but I assume so.
23 Q. Well, let me show you what I have one copy of, which I'll
24 call Plaintiffs' Exhibit 160. I'll put it in a folder.
25 MR. HOLZHAUER: Your Honor, we don't have a copy of
568
Barrett - cross
1 this as it is. We'll stipulate that things were stapled
2 together.
3 THE COURT: What is it?
4 MR. FLAXMAN: This is the actual in-basket that was
5 given to me in discovery. Actually, a duplicate original that
6 was actually administered.
7 THE COURT: This is Defendant's Exhibit 16 as it
8 existed at the time the test was given?
9 MR. FLAXMAN: That's correct.
10 THE COURT: Is that accurate, Mr. Holzhauer? Take a
11 look at it.
12 MR. HOLZHAUER: Can I look at it, please?
13 MR. FLAXMAN: Sure.
14 THE COURT: Let's make sure it's the right thing.
15 (Pause.)
16 MR. HOLZHAUER: It appears to be correct, Your Honor.
17 THE COURT: Okay.
18 BY MR. FLAXMAN:
19 Q. Let me show you Plaintiffs' Exhibit 160, which we've now
20 agreed is the in-basket that was actually administered. There
21 are some packets in there that are stapled together, is that
22 right?
23 A. Yes.
24 Q. And if you look through it, you see there is some scratch
25 paper, is that right?
569
Barrett - cross
1 A. Yes.
2 Q. Then you get to page 7. There is a to/from report, is that
3 right?
4 A. Yes.
5 Q. And did anything in your job analysis indicate that watch
6 commanders start their tour of duty with files in an in-basket?
7 A. Not in that format, no.
8 Q. Well, that's a yes or no question, Dr. Barrett.
9 Did anything in your job analysis indicate that watch
10 commanders start their tour of duty with files in an in-basket?
11 MR. HOLZHAUER: Asked and answered, Your Honor. He
12 said no.
13 THE COURT: I think he said no.
14 MR. FLAXMAN: Thank you.
15 THE COURT: Is that what you meant?
16 THE WITNESS: Yes.
17 BY MR. FLAXMAN:
18 Q. Now, in those materials, there are two weapons discharge
19 investigations for the watch commander. Do you see that?
20 A. What page are you on now? Oh, page 10?
21 Q. Well, it's the to/from on page 7 of the material.
22 A. This seems to be out of order. This is out of order. I
23 understand. Yes, I see. You're talking about page 7, the
24 first one, weapons discharge investigations.
25 Q. Right. That gives -- now, that's on Bates stamp 8462 of
570
Barrett - cross
1 the exhibit, I believe. That assigns six major tasks that the
2 watch commander has to complete, is that right?
3 A. Yes.
4 Q. Is there anything in your job analysis that indicates that
5 watch commanders do two weapons discharge investigations on a
6 daily basis?
7 A. No.
8 Q. And if we look further on that sheet, there is an activity
9 called scheduling. Do you see that, No. 6?
10 A. Yes.
11 Q. Does anything in your job analysis indicate that watch
12 commanders do scheduling without clerical assistance?
13 A. No.
14 Q. And to answer the questions that were later asked on the
15 in-basket, the person taking the test had to do scheduling
16 without clerical assistance, is that right?
17 A. That's correct.
18 Q. And you were trying to measure the ability to do
19 scheduling, is that right?
20 A. Or supervise scheduling, yes.
21 Q. You're trying to measure supervisory skills, is that right?
22 A. Supervisory behavior would be more accurate probably.
23 Q. Let's look at page 16 of the materials which is Bates
24 stamped 8471. Do you see that? That's a witness statement
25 from Paul Phillips?
571
Barrett - cross
1 THE COURT: What was the page?
2 MR. FLAXMAN: 8471.
3 BY MR. FLAXMAN:
4 Q. Have you found that?
5 A. Yes.
6 Q. Does anything in your job analysis indicate that watch
7 commanders review witness statements in this format?
8 A. In this format?
9 Q. That's correct, Dr. Barrett.
10 A. I'm not sure of the exact format. But they do review
11 witness statements.
12 Q. Anything in your job analysis indicate that this is the
13 format of Bates stamp 8471 in which watch commanders review
14 witness statements?
15 A. I don't recall anything specific in my job analysis, my job
16 description that says that, no.
17 Q. As a matter of fact, this isn't the form as used in the
18 Chicago Police Department, is it, Dr. Barrett?
19 A. I don't recall.
20 Q. Now, let me ask you to go on to page 20, which is Bates
21 stamped 8475, that's called "Lieutenant Roberta Joseph's
22 notes." Have you found that, Dr. Barrett?
23 A. Page 20, yes.
24 Q. Anything in your job analysis indicate that watch
25 commanders maintain notes in this type of format?
572
Barrett - cross
1 A. This type of format, no.
2 Q. As a matter of fact, this format was made up for your
3 simulation test, is that right?
4 A. We used this for our simulation, that's correct.
5 Q. Now, to take the in-basket, the person had to review the
6 materials and then answer multiple-choice questions, is that
7 right?
8 A. Yes.
9 Q. And there were 60 multiple-choice questions?
10 A. Yes.
11 Q. And you had 90 minutes to do those, is that right?
12 A. Yes.
13 Q. Do you have -- now, if you look at that exhibit, I think
14 you'll find the questions. They're in a separate packet. And
15 they start at Bates stamp 8403 of the exhibit. Let me ask you
16 to look at --
17 A. I'm sorry. I'm still -- I have the original.
18 Q. Well, if you look through the original, I think you'll find
19 the exact format in which the questions exist.
20 THE COURT: 8403?
21 MR. FLAXMAN: 8430.
22 BY MR. FLAXMAN:
23 Q. If you'd like me to find it for you, Dr. Barrett, I'll be
24 glad to point it out.
25 A. Yes, sure.
573
Barrett - cross
1 Q. I'm not pointing it out. Let me show you 8430. Do you see
2 question 3?
3 A. Yes.
4 Q. Anything in your job analysis which indicate that Chicago
5 watch commanders use language of "a discrepancy among the five
6 statements which may impact the investigation"?
7 A. No.
8 Q. You don't know, do you, whether or not that language is
9 used by Chicago police lieutenants, do you?
10 A. That specific language, no.
11 Q. You don't know whether -- that's not -- as far as you know,
12 that's not Chicago police jargon to determine whether there is
13 "a discrepancy among statements which may impact an
14 investigation"?
15 A. I don't recall anything like that, no.
16 Q. Okay. And that same language about "a discrepancy which
17 may impact the investigation," that was used in question 6 on
18 Bates stamp 8431, is that right?
19 A. I don't know.
20 Q. Let me show you question 6.
21 A. Yes, "discrepancy" is a term, "discrepancy" is in question
22 6, that's correct.
23 Q. And "a discrepancy which would impact the investigation,"
24 is that the phrase that was used in question 6?
25 A. No.
574
Barrett - cross
1 Q. What phrase was used about discrepancy, Dr. Barrett?
2 A. "There is discrepancy in the witness statements regarding."
3 Q. Do you know if Chicago police lieutenants talk about
4 discrepancies?
5 A. No.
6 Q. Let me show you question 11, Bates page 8433. Is that a
7 question which asked again about "a discrepancy which would
8 impact an investigation"?
9 A. Yes. It says "a major discrepancy" on item 11.
10 Q. Then the same thing as to question 13, about another
11 "discrepancy which would impact an investigation"?
12 A. Yes, number 13 says "a major discrepancy."
13 Q. Now, so this isn't the kind of language that Chicago police
14 watch commanders deal with, is that right?
15 A. I said I didn't know. It doesn't seem an usual term to me,
16 but I don't have any direct knowledge of the use of that term.
17 Q. As a matter of fact, you don't have any direct knowledge at
18 all of what's important for a Chicago police lieutenant to
19 know, do you, Dr. Barrett?
20 A. That's not correct.
21 Q. Okay. Well, let me ask you to look at question 47 which is
22 on Bates page 8445. Have you had a chance to look at that?
23 A. Yes.
24 Q. You don't have any personal knowledge, do you, Dr. Barrett,
25 that Chicago watch commanders actually make the kind of
575
Barrett - cross
1 decision that's asked for in question 47?
2 A. From our job analysis and job description, we know that
3 they do review that material, supervise that material, and they
4 may, in fact, make decisions based upon this material.
5 Q. They may, in fact, Dr. Barrett? Is that your testimony?
6 A. Yes.
7 Q. So as far as you know, all those decisions, all that work
8 is done by a watch secretary, is that right, by a clerical
9 person?
10 A. I'm not saying that.
11 Q. Well, you don't know whether or not it's done by the watch
12 commander or by a civilian personnel, do you?
13 A. I know it's all reviewed by the watch commander.
14 Q. Now, does that question ask the watch commander to review
15 something? What does question 47 say?
16 A. "According to Lieutenant Joseph's notes, Sergeant Woodward
17 has requested one day compensatory time in December to visit
18 her mother. The best day for Sergeant Woodward's day off is
19 blank December '94."
20 Q. Now, as far as you know, no watch commander has ever made
21 that kind of decision in the first instance in the history of
22 the Chicago Police Department?
23 A. I'm not -- that exact decision?
24 Q. That's right.
25 A. No, probably not. I don't have any knowledge of that.
576
Barrett - cross
1 Q. How about question 48? What's question 48, Dr. Barrett?
2 A. "In order to satisfy the personnel needs for security at
3 the Kraft Festival on 13 December '94, one of the day off
4 requests by the officers would need to be denied. Request of
5 Officer Blank should be denied first."
6 Q. And, again, you don't have any data from your job analysis
7 to show that watch commanders make that kind of decision in
8 their work as watch commanders?
9 A. We have data they, in fact, review decisions and may make
10 this decision, yes.
11 Q. And may make this decision?
12 A. Yes.
13 Q. What data do you have that shows they may make this
14 decision?
15 A. I don't have any exact data that I can point to in terms of
16 our job analysis in terms of saying this is some statement that
17 they made that certain decision on a certain day, no.
18 We do have reviews by subject matter experts, though,
19 which indicated that was the sort of behavior which watch
20 commanders did and were involved in.
21 Q. Was Mr. Klein one of those subject matter experts?
22 A. Yes.
23 Q. Did you ask Mr. Klein whether or not watch commanders make
24 the kind of decision that's called for in question 48?
25 A. I don't recall an exact question. We ask in general, and
577
Barrett - cross
1 we went through each and every item.
2 Q. Did you ask him specifically about the questions that were
3 used on the test, the in-basket test?
4 A. Yes.
5 Q. And did he tell you specifically that watch commanders
6 answer the kind of question that's asked for in question 48?
7 A. I don't recall that specific question, no.
8 Q. Well, was Commander Cadogan another one of the subject
9 matter experts who reviewed the in-basket?
10 A. You mean Chief Cadogan?
11 Q. That's right.
12 A. Yes, he reviewed the in-basket.
13 Q. Did you ask Chief Cadogan whether watch commanders did the
14 kind of scheduling that's called for in question 47?
15 A. We asked in general if the behavior we're testing for was
16 important and relevant.
17 Q. Did you ask him --
18 A. I don't recall exactly asking that specific question, no.
19 Q. Did you ask him specifically whether or not this is the
20 kind of thing that a lieutenant does?
21 A. I don't recall asking that specific question, no.
22 Q. I've asked you the same about Mr. Shaw. Did you ask him
23 specifically whether question 47 is the kind of thing that a
24 lieutenant does? Would your answer be the same, "I don't
25 recall"?
578
Barrett - cross
1 A. No. I don't recall asking specifically about 47, no.
2 Q. Now, after the in-basket was administered, you had an
3 opportunity to look at the results, is that right?
4 A. Yes.
5 Q. And you saw what percent got each question right, is that
6 right?
7 A. Yes.
8 Q. And some of the questions 90 percent of the people taking
9 the test got right?
10 A. I don't recall the exact figures, but it sounds accurate.
11 Q. And some of the questions less than 50 percent of the
12 people taking the test got right, is that right?
13 A. I assume that's correct, yes.
14 Q. Do you know that 30 percent of the people taking the test
15 got question 48 correct, 70 percent got it wrong?
16 A. No, I didn't know that. Well, I shouldn't say I didn't
17 know it. I may have known it at one point in time. Sitting
18 here today, I don't know that.
19 Q. I think there is a stipulated exhibit which shows that.
20 Now, your in-basket used an objective scoring system,
21 is that right?
22 A. Yes.
23 Q. And the scoring was based on how many questions somebody
24 answered correct on a multiple-choice test, is that right?
25 A. Yes.
579
Barrett - cross
1 Q. There was no penalty for guessing, is that correct?
2 A. Yes, that's correct. There was no penalty for guessing.
3 Q. You told us, I think when we broke last week, that your
4 objective scoring system was based on research that had been
5 reported by a man named Lopez in 1962, is that right?
6 A. Well, that's not quite accurate. I think I referenced
7 Lopez as a research article, research which in fact did use an
8 objective technique. I'm not saying I used his technique, I
9 don't believe.
10 Q. So you just cited it because he used objective scoring, is
11 that -- is that what your testimony is, Dr. Barrett?
12 A. Well, I don't recall your exact question, but I think there
13 was a discussion about traditional, objective and other types
14 of in-baskets.
15 And I just mentioned the fact that there is a long
16 history of psychometric, objective type scoring systems for
17 in-baskets. I think that's what I recall I said.
18 Q. Well, are you telling us that Lopez reported an objective
19 scoring system for an in-basket that was similar to the one
20 that you used in the 1994 police lieutenants test?
21 A. I'm not sure how you are using the word "similar," because
22 I'm sure, or as I recall, he used a variety of objective
23 in-baskets.
24 In fact, he reported that one of his concerns was
25 that the more mechanical approach might turn people off, like
580
Barrett - cross
1 vice-presidents. As I recall, he said even vice-presidents
2 accepted the more objective approach as being appropriate. So
3 I'm not sure what you are asking, I guess.
4 Q. Well, I'm asking you, Dr. Barrett, is your scoring system
5 based on anything that Felix M. Lopez reported back in the
6 1960s?
7 A. I'm not sure if it's based upon. I'm saying that, yes, he
8 used an objective approach also, which was computer scored as I
9 recall. If that's what you are asking, there is that
10 similarity.
11 Q. Let me take that back.
12 Let me show you what I'll mark as Plaintiffs' Exhibit
13 161. Is this the Lopez article that you are referring to,
14 Dr. Barrett?
15 A. Yes.
16 Q. Now, that's not an article that appeared in a peer-reviewed
17 journal, is it, Dr. Barrett?
18 A. No. This appeared as basically a technical report put out
19 by the American Management Association.
20 Q. Now, when you talked about the in-basket test, you said it
21 was a simulation of work that was performed by lieutenants, is
22 that right?
23 A. Yes.
24 Q. Do you agree with the Lopez definition of simulation as "an
25 operating representation of central features of reality"?
581
Barrett - cross
1 A. That could be one definition. I don't recall it, what he
2 said.
3 Q. Well, let me ask you to look at page 14 and see if you can
4 find where it says, "Social scientists define simulation simply
5 as an operating representation of central features of reality"?
6 A. I see "'Simulation,' 'role playing' and 'game' are terms
7 often used interchangeably in the literature to denote various
8 types of training and testing methods."
9 Is that what you mean?
10 Q. Yeah. Then he says, "Social scientists define simulation
11 simply as an operating representation of central features of
12 reality."
13 A. Yes, I see that.
14 Q. Now, is that -- do you agree with that definition of
15 simulation, Dr. Barrett?
16 A. I don't see it's -- yeah, I don't see anything wrong with
17 that definition particularly. It's his definition.
18 Q. And the in-basket was a simulation, is that right?
19 A. Sure. Yes, it was.
20 Q. And the oral exercise was also a simulation, isn't that
21 right.
22 A. Yes.
23 Q. Now, Mr. Lopez defines an in-basket at page 17 where he
24 says, "The in-basket exercise presents the participant with the
25 hypothetical work situation in which he just makes decisions on
582
Barrett - cross
1 a series of letters, memos, and other documents deposited as
2 incoming mail on his in-basket."
3 Do you see that?
4 THE COURT: What?
5 BY THE WITNESS:
6 A. No.
7 THE COURT: What page was that?
8 MR. FLAXMAN: Page 17.
9 THE COURT: Where on the page?
10 MR. FLAXMAN: I'm short one copy.
11 THE COURT: You don't have a copy?
12 MR. HOLZHAUER: The last paragraph of the second
13 column.
14 THE COURT: The last paragraph.
15 MR. HOLZHAUER: Second to last, the last full
16 paragraph, it starts with "the in-basket exercise," and you
17 split the last sentence, it starts about six lines from the
18 bottom of that paragraph, the words ETS.
19 THE COURT: I have it. Thank you.
20 BY MR. FLAXMAN:
21 Q. Have you found that, Dr. Barrett?
22 A. I find -- the one you're talking about ETS, that paragraph?
23 Q. "The in-basket exercise presents the participant with the
24 hypothetical work situation in which he just makes decisions on
25 a series of letters, memos and other documents deposited as
583
Barrett - cross
1 incoming mail in his in-basket."
2 A. Yes, I see that.
3 Q. That's not what happened in your in-basket for the 1994
4 lieutenants test, did it, Dr. Barrett?
5 A. Let me see now. It was a hypothetical work situation.
6 Q. Could you just --
7 A. That is true. You're asking me, I assume, the parts say
8 does this correspond with what we did? Yes. It was a
9 hypothetical work situation, our in-basket. Do you make
10 decisions? Yes. And they were based upon written
11 documentation. That's true.
12 Q. So in your opinion answering the multiple-choice questions
13 is the same as making decisions on a series of letters, memos
14 and other documents as in the in-basket that Mr. DeLopez --
15 Mr. Lopez is describing, is that what you are telling us?
16 A. I'm saying they could not answer the questions without
17 making decisions. Yes, they made decisions based on that
18 material. If they didn't have the material, they could not
19 make decisions.
20 Q. Now, Dr. Barrett, are you telling us that it is your
21 opinion that answering those 60 multiple-choice questions on
22 your in-basket is the same as what Mr. Lopez is describing at
23 page 17, the excerpt that I just read? Is that what you are
24 telling us?
25 A. What I've read so far, it seems to be the same.
584
Barrett - cross
1 Q. And do you remember Mr. Lopez describes an in-basket
2 exercise with lieutenants in the Port Authority Police
3 Department, is that right?
4 A. Yes.
5 Q. Is it your opinion that the in-basket for those lieutenants
6 was the same as your 60 multiple-choice question in-basket?
7 A. I've never seen the actual Port Authority police lieutenant
8 in-basket, so I would have to see it to compare. I don't know
9 exactly what --
10 Q. Well, isn't it described in Mr. Lopez's article,
11 Dr. Barrett?
12 A. It's described in general, yes.
13 Q. And it's not at all answering 60 multiple-choice questions,
14 is it, Dr. Barrett?
15 A. I don't recall how many questions they had to answer.
16 Q. Do you recall how long, how much time there was to do the
17 Port Authority in-basket?
18 A. No, I don't.
19 Q. Do you recall what format the answers took for the Port
20 Authority lieutenant test, in-basket test?
21 A. Again, it's been some time since I read it, but I think
22 they had to mark the reasons for the various actions, I
23 believe. But I could be mistaken.
24 Q. They had to complete a "reasons for action report,"
25 Dr. Barrett, isn't that correct?
585
Barrett - cross
1 A. Well, again, as I said, that's what I recall. They had to
2 indicate what were the appropriate actions to be taken.
3 Q. They had to explain what action they would take on
4 particular, on a particular situation, is that right?
5 A. No.
6 Q. Excuse me?
7 A. No.
8 Q. They didn't have to explain what action they took for each
9 piece of work and why they did it?
10 A. No.
11 Q. That's, that's -- okay.
12 A. I'm using the word "explain" to say they had made no verbal
13 explanation or written explanation. They just marked, "this is
14 the explanation I use for this decision," as I recall. So I'm
15 saying I think you're taking it out of context what they
16 actually did.
17 Q. Let me ask you to look at page 75 of the Lopez report.
18 There is some material I think on the right-hand column which
19 is quoted from one of the people who took the test where he's
20 describing his experiences on the test. Have you found that,
21 Dr. Barrett?
22 A. You're on page -- it says, "like a group of doomed men who
23 proceeded into the testing room." Is that what you are
24 referring to?
25 Q. Yes.
586
Barrett - cross
1 Do you see where it says, "We also had to explain
2 what action we took with each piece of work and why we did so"?
3 THE COURT: Where are you?
4 MR. FLAXMAN: There is a right-hand column. There is
5 excerpted material.
6 THE COURT: Where are you in that?
7 MR. FLAXMAN: If I could have somebody's copy, I'll
8 --
9 THE COURT: I see it.
10 MR. FLAXMAN: All right.
11 THE COURT: Second to the last paragraph, the last
12 sentence.
13 BY MR. FLAXMAN:
14 Q. Do you see that, Dr. Barrett?
15 A. Yes. I see, "After three strenuous hours during which --
16 three strenuous hours," this is what you are referring to?
17 Q. No. I'm referring to the sentence, "We also had to explain
18 what action we took with each piece of work and why we did so."
19 A. Yes, I see that sentence.
20 Q. In your in-basket for the 1992 lieutenants test, the only
21 explanation that the people taking the test could make was
22 whether -- which choice to make on the 60 questions, is that
23 right?
24 A. I'm sorry. What's your question?
25 Q. On your in-basket, the in-basket simulation, there was no
587
Barrett - cross
1 place to write why a particular applicant thought a question
2 was making a particular choice, is that right?
3 A. That's correct.
4 Q. That's different than the lieutenants test for the Port
5 Authority described in the Lopez article, isn't that right?
6 A. Well, again, I think I have to read it in context and go to
7 page 74. I think they might have been -- this might have been
8 the situation where they were comparing both the more objective
9 form and the more subjective form and showing of the same.
10 So if you start on page 74, I think you'll see, in
11 fact, at the top of page 75, they talk about the 47 problems
12 that they had to solve. And so I think what it could be would
13 be the fact that they're comparing the two. I don't know. I'd
14 have to read it in context, because what they did is show that
15 in fact we could do a rating or subjective approach, but you
16 would obtain the same results if you would just check off here
17 are the reasons for actions after they made a decision. Your
18 more objective approach which could be computer scored. So
19 again, I'd have to read the whole report in context, but I
20 think that's what he's referring to.
21 Q. Well --
22 A. I'd have to read on.
23 Q. Well, why don't you read on, Dr. Barrett, to page 79.
24 A. Okay.
25 Q. Where there is -- page 79 to 80. Do you see the
588
Barrett - cross
1 description there of the objective in-basket and a reference to
2 an 867-item questionnaire?
3 THE COURT: Where are you?
4 MR. FLAXMAN: I knew I needed one more copy.
5 BY MR. FLAXMAN:
6 Q. Do you see on page 80 underneath the horizontal line, the
7 third line down, by filling out the in-basket action report, an
8 867-item questionnaire?
9 A. Yes, I see. I believe he described his handling of the
10 problem by filling out the in-basket action report, an 867,
11 867-item questionnaire.
12 Q. That's different than a 60-question multiple-choice test,
13 isn't it, Dr. Barrett?
14 A. Yes. That's probably much more difficult.
15 Q. It's different, isn't it, Dr. Barrett?
16 THE COURT: Don't raise your voice, please.
17 MR. FLAXMAN: I'm sorry.
18 MR. HOLZHAUER: Asked and answered. He said yes.
19 MR. FLAXMAN: All right.
20 BY MR. FLAXMAN:
21 Q. And if you keep looking at page 80, do you see where it
22 says, "If a breakthrough had occurred in the scoring
23 procedures, it may be possible to develop in-baskets for other
24 positions and perhaps to use it to evaluate the effectiveness
25 of management training programs"?
589
Barrett - cross
1 A. Yes.
2 Q. Lopez wasn't saying he had made a breakthrough, was he,
3 Dr. Barrett?
4 A. He's saying what he's saying.
5 Q. All right. And if you go on in the Lopez, go to page 94
6 where he talks about another Port Authority test. Do you see
7 where he says, "The scoring system was a modified version of
8 the one developed in earlier research efforts. It consisted of
9 an analysis of two separate variables: One, the actual content
10 of the response, including the propriety and accuracy of the
11 actions taken by the candidate and his knowledge of the rules,
12 regulations, and technical aspects of police work; and, two,
13 the style with which he performed his actions, such as the way
14 he communicated, organized his work, made decisions and
15 delegated"?
16 Do you see that, Dr. Barrett?
17 A. No, I'm sorry. I didn't catch exactly -- which column are
18 you in, the first or second column?
19 Q. I'm on the left column of page 94.
20 A. Left column, okay.
21 Q. The first full paragraph.
22 A. First full paragraph, okay.
23 Q. Starting with the second sentence, "The scoring system"?
24 A. "As administrating of the test," is that what you're
25 concerned, that paragraph?
590
Barrett - cross
1 Q. The scoring system utilized in the police lieutenant
2 examination was a modified version?
3 A. Yes.
4 Q. Now, why don't you just read that sentence and the next
5 sentence?
6 A. "The scoring system utilized" --
7 Q. You can read it to yourself.
8 (Pause.)
9 BY MR. FLAXMAN:
10 Q. Have you had a chance to read that?
11 A. Yes.
12 Q. That's not the way the 1994 in-basket for the Chicago
13 police lieutenant was scored, is it, Dr. Barrett?
14 A. It's not the way it was scored, that's correct.
15 Q. And in your -- the report that you wrote about the 1994
16 lieutenants test, do you recall in appendix Q you refer to some
17 other scholarly works about in-baskets?
18 A. I guess I recall we may have discussed these issues.
19 Q. Well, let me direct your attention to page 321 of
20 Defendant's Exhibit 1, appendix Q, page 4. Do you see that,
21 Dr. Barrett?
22 A. Yes.
23 Q. And that was a reference to some earlier work that had been
24 done with in-baskets, is that right?
25 A. Yes. This is part of our weighting rationale report.
591
Barrett - cross
1 Q. And that earlier work with in-baskets, some of it achieved
2 something -- the validity coefficient that we talked about
3 before, is that right?
4 A. Yes.
5 Q. What's the first in-basket study that you refer to in that
6 appendix? I think it's Q-4?
7 A. Q-4 -- well, table 1, number 5, the correlation between the
8 in-basket and job performance?
9 Q. Right.
10 Well, do you refer -- could you tell us what the
11 first in-basket study is that you are referring to?
12 A. We have there Bentz 1968.
13 Q. Was the Bentz 1968 in-basket a 60-multiple-choice-question
14 test?
15 A. No.
16 Q. Was the Bentz in-basket what Dr. York referred to as the
17 "traditional in-basket"?
18 A. As I recall, it was more traditional, yes.
19 Q. Excuse me?
20 A. I believe it was.
21 Q. Is that one of those tests where the individual was asked
22 to assume that he had just been appointed manager of a store,
23 the previous manager had died, and the new manager had to come
24 in and perform many tasks? Is that the Bentz in-basket?
25 A. It sounds like the format that John Bentz used to use, yes.
592
Barrett - cross
1 Q. Okay. That's different than the format that Gerald Barrett
2 used in 1994 in the City of Chicago for promotions to
3 lieutenant, correct?
4 A. The instructions are almost identical.
5 Q. Excuse me?
6 A. The instructions are almost identical in terms of the
7 format. Just take the word "lieutenant" and substitute "store
8 manager," and it would be identical instructions.
9 Q. Well, was the Bentz in-basket graded by answering
10 multiple-choice questions?
11 A. I don't believe it was.
12 Q. As a matter of fact, it was graded by looking at the way in
13 which the work was performed, isn't that right?
14 A. I don't recall exactly now, but it could be true.
15 Q. Okay. Well, is there another -- there is another in-basket
16 study that you referred to in that report, is that right?
17 A. Yes.
18 Q. And what's the next one?
19 A. Kesselman 1982.
20 Q. Was the Kesselman 1982 the same type of in-basket as used
21 in the 1994 police lieutenants exam?
22 A. I need to see the exact reference. It's not on this paper,
23 so I can't --
24 Q. So as you sit here now, you don't know whether or not it
25 was, is that right?
593
Barrett - cross
1 A. Well, the reason I'm saying this is because I believe the
2 Kesselman articles appeared in Public Administration Quarterly,
3 I think it might have been, which did use an objective
4 in-basket.
5 And if this is the same one -- again, I'm not sure, I
6 don't recall right now, because it's been some time, but --
7 Q. Well, did you also rely on an in-basket study by Wallowick?
8 A. Yes. Is that one of those 60 multiple-choice
9 objectively-graded in-baskets?
10 A. I believe that was the IBM research. And I don't think
11 they did use objectives.
12 Q. Excuse me?
13 A. I don't believe they did at IBM use an objective in-basket
14 at that point in time.
15 Q. As a matter of fact, it wasn't an objective in-basket. It
16 was what Dr. York referred to as the traditional in-basket, is
17 that right?
18 A. As I recall, I think that's correct.
19 Q. Do you recall that Mr. Lopez in that 1960 article described
20 objective scoring "as quite promising but not overwhelmingly
21 successful"?
22 A. No.
23 Q. Do you still have it in front of you, Dr. Barrett?
24 A. Yes.
25 Q. Let me ask you to look at page 111 right at the top. Do
594
Barrett - cross
1 you see where it says, "In-basket exercise was deemed to be
2 quite promising"?
3 A. Yes.
4 It starts on page 110. It says, "This first approach
5 to group scoring of an in-basket exercise was deemed to be
6 quite promising, although not overwhelmingly successful."
7 Q. Thank you.
8 Now, do you agree that paper-and-pencil tests --
9 well, do you agree that the in-basket was a paper-and-pencil
10 test?
11 A. No.
12 Q. Excuse me?
13 A. No.
14 Q. Do you agree that the written job knowledge test was a
15 paper-and-pencil test?
16 A. Yes.
17 Q. Okay. Was reading comprehension important in doing the
18 in-basket?
19 A. I'm not sure what you mean by "important." But you had to
20 be able to read and comprehend the material in the in-basket,
21 yes.
22 Q. Well, would you agree that someone with better reading
23 comprehension would do better on that in-basket test?
24 A. I haven't done any studies that show it to be true. It was
25 written at the eleventh grade level, the instructions and the
595
Barrett - cross
1 question. So if the person was a high school graduate, that
2 would be adequate reading comprehension to perform the task.
3 So I'm not sure how much better they would do with better
4 reading comprehension.
5 Q. Well, you told us, I think, that you believe that if you
6 had more time to do the in-basket, you'd do better on it?
7 A. I don't think there is any question that since it's an
8 open-book test and the fact that if you have an unlimited
9 amount of time, you could get all of the 60 questions correct.
10 Q. And you can't -- you don't have any opinion at all about
11 whether reading comprehension, good reading comprehension would
12 help you do that test in the 90 minutes, answer those 60
13 questions?
14 A. Well, I said I've done no empirical study on that. But I
15 did say that I have -- we did check the reading level. It's
16 eleventh grade reading level.
17 Q. Well, you've done no empirical study on whether the final
18 scores predict performance as a police lieutenant, isn't that
19 right?
20 A. I said that we had accomplished no criterion-related study.
21 I didn't say we didn't do any empirical study.
22 Q. Now, the written-multiple-choice test was a job knowledge
23 test, is that right?
24 A. Yes.
25 Q. And do you have any evidence from your job analysis that
596
Barrett - cross
1 somebody who can answer 150 questions on that test correctly
2 would do better as a lieutenant than somebody who can answer
3 149?
4 A. Yes.
5 Q. Do you have any -- and do you have any data that shows
6 that, Dr. Barrett?
7 A. Well, I think not only from the job analysis, but also what
8 we did was to go back and for the challenged items of
9 Mr. Bishop, we devised our flow charts, which shows quite
10 graphically both the complexity of the knowledge required for
11 the police lieutenant's job in order to solve these problems
12 successfully.
13 Q. Maybe you didn't understand.
14 A. Okay.
15 Q. I'm sorry. Maybe you didn't understand my question.
16 Do you have any empirical evidence that knowing the
17 correct answers to 150 questions is correlated with better
18 performance as a police lieutenant?
19 A. Mr. Flaxman, again, you are saying correlated is empirical.
20 If that's what you are saying, if you're asking is there a
21 criterion-related study, I'm saying we have no
22 criterion-related study on the job knowledge test.
23 Q. I'm asking you about empirical evidence.
24 A. I'm saying, yes, we have empirical evidence from the job
25 analysis and from the flow charts.
597
Barrett - cross
1 Q. And the job analysis is this judgment that you're making,
2 is that right?
3 A. It's observation of behavior, observation of interviews.
4 It's -- it is evidence which we can use.
5 Q. Now, you told us that you don't have a criterion-valid
6 study or you didn't try to do criterion validation, is that
7 right?
8 A. I said we did not do a criterion-related study, that's
9 correct.
10 Q. And if you had done a criterion-related study you would
11 have ended up with a validity coefficient, is that right, or
12 number?
13 A. That's correct.
14 Q. And you briefly said on direct that it was impossible to do
15 a criterion-valid study, is that right?
16 A. I'm not sure I used the word "impossible." I'm not sure I
17 used that term.
18 Q. Well --
19 A. I might have said it was not feasible. There are many ways
20 that you could do it. I think I might have said if you did do
21 it, under even the best conditions, you would compromise the
22 test. But I'm not sure I said impossible.
23 Q. Well, you said it's not feasible and would compromise the
24 test? Did you propose to the City of Chicago that you
25 validate -- that you prepare a promotional test on a
598
Barrett - cross
1 criterion-valid theory?
2 A. I did not propose to the City of Chicago that I would do a
3 criterion-related study, that's correct.
4 Q. Did you consider doing a criterion-valid study?
5 A. Yes.
6 Q. And why did you not do a criterion-valid study?
7 A. I didn't do a criterion-related study for a number of
8 reasons. There are two types of criterion-related studies.
9 One is called predictive. To do a predictive study, I would
10 had to have -- what is called a perfect predictive study,
11 randomly selected sergeants --
12 THE COURT: I think we've been into this. I've heard
13 this before.
14 MR. FLAXMAN: I didn't ask the question.
15 THE COURT: Well, you asked him why he didn't do it.
16 Now he's telling you. I think he's already said that once.
17 MR. FLAXMAN: Well, he didn't tell you why he
18 couldn't do it on cross-examination. He gave his explanation
19 of predictive and -- but he didn't --
20 MR. HOLZHAUER: I believe he did once on direct and
21 once on cross.
22 MR. FLAXMAN: I don't think I've done this on cross.
23 I was saving this for the end.
24 THE COURT: Well, since you say it's the end, I'm
25 going to let you do it.
599
Barrett - cross
1 MR. FLAXMAN: All right.
2 BY MR. FLAXMAN:
3 Q. Without giving us the explanation that you were just giving
4 us, why did you not attempt to do a criterion-validated study
5 for the City --
6 MR. HOLZHAUER: Object, Your Honor. I don't think he
7 can instruct him not to give the explanation that answers the
8 questions he's just given.
9 THE COURT: Well, he could say, other than what
10 you've already said, is there any other reason?
11 MR. HOLZHAUER: Okay.
12 BY MR. FLAXMAN:
13 Q. Other than what you've already said, is there any other
14 reason why you did not do a criterion-valid study?
15 A. Not having recalled what I said last week, I think that
16 some of the main points had to do with compromising the test.
17 Q. Is it --
18 A. The feasibility, the issues having to do with being able to
19 obtain an adequate measure of job performance from the
20 supervisors. So there are a variety of reasons why we could
21 not do it.
22 Q. Well, are you telling us that you can't measure how well
23 somebody is doing on the job as a police lieutenant?
24 A. I didn't say that. I said it's very difficult in the
25 present climate to get reliable measures of job performance.
600
Barrett - cross
1 I usually say that I had tried this approach in the
2 past in the case in Akron where I attempted to obtain a good
3 measure of job performance. We pilot tested it. But once we
4 started -- tried to use it operationally, there were charges of
5 bias, and not only charges of bias, but there were grouping of
6 scores.
7 In effect, the whole process fell apart because of
8 the issues of a supervisor having to rate subordinates and
9 potentially being accused of bias in that rating process.
10 Q. So are you telling us that as a practical matter you can't
11 measure job performance of police lieutenants?
12 A. I'm not saying -- I'm saying as a practical matter it's
13 very difficult when it's being used in the way it's being used
14 in the courtrooms.
15 Q. Didn't you do a criterion-valid study for Akron police
16 officers?
17 A. Yes, in the early 1970s I did a number of criterion-related
18 studies for police and fire. I did for promotional exams also,
19 I did criterion-related studies, yes, I did.
20 Q. Did you do a test that was administered in June of 1974 to
21 police force applicants in the City of Akron?
22 A. That sounds like the right date. Yes, I probably did.
23 Q. And you did a criterion-valid study with that test, didn't
24 you, Dr. Barrett?
25 A. As I recall, I did, yes.
601
Barrett - cross
1 Q. And that criterion-valid test was challenged in court, is
2 that right?
3 A. Yes, it was.
4 Q. And that criterion-valid test was upheld by the Court,
5 isn't that right, Dr. Barrett?
6 A. I believe that was Arnold v. Ballard in 1975.
7 Q. Did you also, after Arnold v. Ballard, do a criterion-valid
8 test for police sergeants in the City of Akron?
9 A. As I recall, I did, yes.
10 Q. And that criterion-valid test was also upheld by the
11 courts, isn't that right?
12 A. That's correct.
13 Q. You also did a criterion valid -- criterion-related test
14 for police lieutenants in the City of Akron, is that right?
15 A. Yes, I did.
16 Q. And that criterion-related test was also upheld by the
17 courts, is that right?
18 A. Yes, it was.
19 Q. And you also did a criterion-related test for entry-level
20 fire fighters in Akron, is that right?
21 A. Yes, I did.
22 Q. That was also upheld by the courts, is that right?
23 A. Yes, it was.
24 Q. You did another criterion-related test for fire lieutenant,
25 is that right?
602
Barrett - cross
1 A. Yes, I did.
2 Q. That was in Akron, is that right?
3 A. That's correct.
4 Q. That was also upheld by the courts?
5 A. That's correct.
6 Q. And you did another criterion-related test for fire
7 captains in Akron, is that right?
8 A. Yes.
9 Q. That was also upheld by the courts?
10 A. That's correct.
11 Q. And then you come to Chicago, Dr. Barrett, and you can't do
12 any criterion-related studies, is that right?
13 A. There's been a time lapse of 20 years, I think, between the
14 ones I did in 1974 and the present, 1994.
15 As I said in that time period things have changed in
16 terms of the perception and the cooperation from safety forces.
17 It's been due to the fact of the continued litigation over this
18 past 20 years.
19 Q. Well, Dr. Barrett, the decision about police officers in
20 Akron, that was in 1975, is that right?
21 A. Yes.
22 Q. And the decision about police sergeants in Akron, that was
23 also in 1975, is that right?
24 A. Yes.
25 THE COURT: The decision or the test?
603
Barrett - cross
1 MR. FLAXMAN: The decision.
2 BY THE WITNESS:
3 A. I'm not sure. It's probably correct.
4 BY MR. FLAXMAN:
5 Q. Now, police lieutenants in Akron, that was 1987, isn't
6 it -- wasn't it?
7 A. I think you're referring to when we changed the format to
8 the format we have now in terms of the oral, job knowledge and
9 in-basket. I think that would be '87. But I'm not -- I cannot
10 be 100 percent sure.
11 Q. Didn't you do a test in 1996, a criterion-related test in
12 Akron for fire captain?
13 A. No.
14 There was a decision in Black Fire Fighters
15 Association v. the City of Akron, a Sixth Circuit appeal case,
16 (unintelligible) 1996, which I believe -- which did verify that
17 our tests were valid. But those tests were the ones, same
18 format we used here. It was based on content validity. It was
19 not based upon criterion-related validity.
20 Q. They weren't criterion-related tests, Dr. Barrett?
21 A. Not the ones we used -- not for the decision in 1996, as I
22 recall.
23 Q. Well, are you familiar -- you're a member of the SIOP, is
24 that right?
25 A. Yes.
604
Barrett - cross
1 Q. And the SIOP is -- for the court reporter -- the Society of
2 Industrial and Organizational Psychologists, is that right?
3 A. Yes.
4 Q. And the SIOP has a newsletter, is that right, Dr. Barrett?
5 A. Yes, it does.
6 Q. What's it called?
7 A. The TIP, I believe
8 Q. The Industrial Psychologist?
9 A. Yes.
10 Q. And from time to time you write for that newsletter, don't
11 you, Dr. Barrett?
12 A. Yes, I do.
13 Q. And when you write for that newsletter, you're truthful, is
14 that right?
15 A. Yes.
16 Q. And you're accurate, is that right?
17 A. I try to be.
18 Q. And everything you say in the articles you write for the
19 newspaper is correct, is that right?
20 A. Well, I wouldn't go quite that far. I try to be accurate.
21 Q. Well, you don't lie in the newsletter, do you?
22 A. No.
23 Q. Now, do you have an article in, I think it's the October
24 1997 TIP, called "An Historical Perspective on the Nassau
25 County Police Entrance Examination, Arnold versus Ballard,
605
Barrett - cross
1 B-a-l-l-a-r-d, 1975 Revisited"?
2 A. Yes, I do.
3 Q. And in that article, you refer to the work you did in Akron
4 as criterion-related studies, isn't that right?
5 A. I think I refer to some of the studies I did as being
6 criterion related, that's correct.
7 Q. Well, you refer to Akron -- cases that were reported as
8 Akron Fire Fighters versus City of Akron as a criterion-related
9 study, is that right?
10 A. Some of the early cases were criterion related, there is no
11 question about it.
12 Q. Well, Barrett & Associates is your organization, is that
13 right?
14 A. That's correct.
15 Q. And when they send out faxes, do they put their fax phone
16 number at the top?
17 A. I don't know.
18 Q. Do you know what your fax phone number is at Barrett &
19 Associates?
20 A. No.
21 Q. All right. Let me show you what we'll mark as 162.
22 MR. HOLZHAUER: Do you have one?
23 MR. FLAXMAN: I only have one.
24 MR. HOLZHAUER: Your Honor --
25 THE COURT: Take a look at it.
606
Barrett - cross
1 Mr. Flaxman, if you are going to be using exhibits,
2 you ought to have more than one copy. You ought to have one
3 for your opponent, one for the witness, one for me and one for
4 yourself.
5 MR. FLAXMAN: You're right. And I don't. And I was
6 doing better before. And I blew it with this one. I'm sorry.
7 THE COURT: Are you almost finished with cross?
8 MR. FLAXMAN: Well, I'm finished. And I want to
9 confer, but I think I'm done.
10 MR. HOLZHAUER: Are you just going to refer to this
11 paragraph right here?
12 MR. FLAXMAN: Yes.
13 BY MR. FLAXMAN:
14 Q. Is Plaintiffs' 162 a copy of your article that appeared in
15 TIP?
16 A. This doesn't look like a copy of an article. It looks like
17 a draft at least. But it's not -- is this the actual article
18 that appeared?
19 THE COURT: Keep your voice up a little bit.
20 THE WITNESS: I'm sorry.
21 BY THE WITNESS:
22 A. I'm not sure it's the actual article that's printed.
23 BY MR. FLAXMAN:
24 Q. Well, in that draft, do you see on the first page where it
25 refers to criterion-valid tests that you developed for the City
607
Barrett - cross
1 of Akron in police and fire?
2 A. I see that the concurrent validation study was completed in
3 six months and resulted in a battery of four tests. The test
4 was administered in June of 1974 to police force applicants, et
5 cetera. Is that what you mean?
6 Q. Yes. In that article you don't refer to content-valid work
7 that you did in Akron, do you?
8 A. No. I refer at that point, right in this paragraph, to
9 1974, what we did, yes.
10 Q. And you continue with cases that are reported through 1996,
11 is that right?
12 A. I say, for example, on page 2, "The police preacademy
13 course was conducted for six weeks. I administered a
14 content-valid test."
15 Q. No. Dr. Barrett, you're not --
16 A. Well, you asked in terms of what I talked about. It seems
17 like I talked about both.
18 Q. My question, Dr. Barrett, is: In this article, somewhere
19 in the article, isn't it true that you refer to the work that
20 you did in Akron as involving criterion-related studies?
21 A. In 1974, that's correct.
22 Q. And in that article, don't you also refer to, after the
23 reference to the 1974 test, to subsequent criterion-related
24 studies?
25 A. I probably did, yes. We did a subsequent criterion-related
608
Barrett - cross
1 study, that's correct.
2 Q. And some of those subsequent criterion-related studies
3 extend through the 1990s, isn't that right, Dr. Barrett?
4 A. No, that's not correct. As far as I can recall, we didn't
5 have any criterion-related studies conducted in 1990s. No,
6 that's not correct.
7 MR. FLAXMAN: I think I'm done, but I want to confer.
8 THE COURT: Let's take a short break.
9 I had the Criminal Code up here. Does anybody know
10 what happened to it? You didn't take it back, did you?
11 MR. PIELL: I think we might have taken it back after
12 it was handed up.
13 THE COURT: Don't come up to the bench and take
14 things off the bench without asking me first.
15 MR. HOLZHAUER: I think George gave that to --
16 THE COURT: Well, I want it back. Ask me, not
17 George. All right. Five minutes.
18 (Recess.)
19 MR. FLAXMAN: Very few.
20 THE COURT: Proceed.
21 BY MR. FLAXMAN:
22 Q. Let's go back to the in-basket very briefly, Dr. Barrett.
23 The in-basket that you administered in Chicago was
24 the same type of in-basket that you had administered to Akron
25 fire lieutenants, is that right?
609
Barrett - cross
1 A. The same format, yes.
2 Q. Did you ever do any investigation to determine if there is
3 a relationship between scores on the in-basket and performance
4 as an Akron police lieutenant?
5 A. I'm not sure I did one for Akron. I thought for fire. I'm
6 not sure I did it at this point. No, I've never done a
7 criterion-related study on the in-basket in the City of Akron.
8 Q. Did you also administer the same type of in-basket to
9 persons employed by the Washington Metropolitan Area Transit
10 Authority?
11 A. Yes.
12 Q. Did you ever do any investigation to determine whether or
13 not there was a relationship between scores on the in-basket
14 and performance in the Washington transit police at the
15 Washington Metropolitan Area Transit Authority?
16 A. We conducted no criterion, or criterion-related study in
17 Washington, D.C., with the transit police.
18 Q. Now, the in-basket when it was administered for the Akron
19 lieutenants and the Washington Metropolitan Area Transit
20 Authority was given under different physical conditions than in
21 Chicago, isn't that right, Dr. Barrett?
22 A. They were all administered, as I recall, in a setting of a
23 classroom-type setting or training room-type setting.
24 Q. Did the people taking the test in Akron and in Washington,
25 D.C., have a tablet chair to put their materials on?
610
Barrett - cross
1 A. I'm not sure. I don't believe so. I think we had tables
2 in other places.
3 Q. You didn't have tables in Chicago, did you, Dr. Barrett?
4 A. No, we did not.
5 Q. And isn't it correct, Dr. Barrett, that because of
6 logistics problems in the City of Chicago, you modified the
7 administration process for the in-basket from the way it had
8 been -- you had administered it in other cities to the way it
9 was administered in the City of Chicago?
10 A. I don't think -- I think our administration procedures were
11 exactly the same.
12 Now, I think what we did do in the City of Chicago, I
13 think what we did is I recall buying a lot of clipboards. I
14 think that we bought a lot of clipboards to help individuals
15 who were taking the in-basket test in the City of Chicago. I'm
16 not sure we had clipboards in other locations.
17 Q. Okay. Did you have Manila folders in any other location?
18 A. I don't recall.
19 Q. Isn't it true, Dr. Barrett, that each person taking the
20 Chicago test was given 15 Manila folders?
21 A. At this point, I don't recall the exact number. But that
22 could be correct if that was part of the instructions.
23 Q. And that was different than it had been in Akron and in
24 Washington, D.C., is that right?
25 A. I don't, I don't recall if we used Manila folders or not in
611
Barrett - cross
1 those two locations.
2 Q. Now, the Manila folders in Chicago, were your instructions
3 that each person should receive 15 folders, all with the same
4 tab? I mean 15 folders for the left tab or 15 folders for the
5 right tab or 15 folders with the tab in the middle?
6 A. I don't recall.
7 Q. Let me show you what's been marked as Plaintiffs' 162.
8 Have you ever seen this before?
9 A. Yes.
10 Q. Could you tell us what it is.
11 A. It's a letter from one of my associates, senior associates,
12 Mr. Carabini, to a Marion Brzykcy, B-r-z-y-k-c-y, of Arthur
13 Andersen. It's in regard to a telephone call received
14 concerning administration of the test.
15 Q. And the first sentence refers to a telephone call you
16 received from Adrian Brandt about one tablet chair, is that
17 right?
18 A. Yes.
19 Q. You had wanted, had you not, more than one tablet chair for
20 each person taking the test, is that right?
21 A. Actually, as I recall, what I wanted was a table.
22 Q. And you didn't get a table, did you?
23 A. No, I did not.
24 Q. And then because you weren't getting a table -- well, let's
25 go back.
612
Barrett - cross
1 In Akron when the test was administered, did the
2 people taking the test have a table?
3 A. I can -- I don't recall exactly, but I think we always
4 requested a table. I know that when the test was given in our
5 office building we had, we had folding tables. I recall that.
6 But I don't recall for each and every application, if we did or
7 not.
8 Q. Now, so when you learned that Chicago wasn't providing
9 tables, you modified the administration process, is that right?
10 A. I wouldn't call it modifying the administration process.
11 We provided additional aids to the people.
12 Q. Well, were the modifications set out in this letter made in
13 the administration process?
14 A. Yes. We said we would provide additional items. This
15 would be one clipboard and 15 Manila file folders.
16 Q. The paragraph after that talks about each person getting 15
17 folders with the same position tab, one person would get right
18 tab folders and another person left tab folders, a third person
19 middle tab folders, is that right?
20 A. Yes.
21 Q. Now, you're not telling us, are you, that your job analysis
22 indicated that watch commanders have to work with file folders
23 that are all tabbed in the same place, are you, Dr. Barrett?
24 A. No, I don't recall that.
25 Q. Okay. And did you do any study to determine whether or not
613
Barrett - cross
1 administering the test this way with the clipboard and file
2 folders would provide the same simulation as administering the
3 test with the table?
4 A. We didn't do any criterion-related studies, if you're
5 asking that question.
6 Q. Did you pilot test it with the clipboard and file folders?
7 A. Yes.
8 Q. Was that in November of 1994?
9 A. Once we learned about that, we -- as I recall, we said what
10 can we do to aid the individuals? Of course, it's the same
11 administration, the same clipboard, the same file folders for
12 everyone. So everyone has the same materials. But we said
13 what can we do to make it perhaps easier?
14 And so we decided to -- decided to use clipboards and
15 file folders. As a matter of fact, that's now become our
16 standard procedure since that time where, in fact, we do
17 provide those things to people.
18 Q. And you don't have any data which shows whether or not
19 administering the test with the table or with 15 file folders
20 and a clipboard produces the same result?
21 A. No. I've done no study of that type.
22 Q. And that's not a criterion type of study, is it,
23 Dr. Barrett?
24 A. Well, it could be if you wanted to make it one.
25 Q. Well, but you could do that -- that kind of study without
614
Barrett - cross
1 any criterion, what you talked about criterion validation,
2 isn't that right?
3 A. I'm not sure what you are asking. It would be, it would be
4 -- if you want to try to formulate a study at this point in
5 time?
6 Q. Well, what I mean, Dr. Barrett, is you could have two
7 groups of people. One group could take the test with a table
8 and another group could take the test with the clipboard and
9 file folders, and you can see if there is any differences in
10 how they do on the test. Is that a traditional kind of study,
11 Dr. Barrett, testing a hypotheses?
12 A. It's traditional only in the sense, yes, you could do a
13 study. I've never heard of anyone worried about if you use a
14 clipboard and file folders if there is going to be a
15 difference, because the conditions are the same for everyone.
16 If there is any difference, it's a constant across individuals.
17 Q. Now, you're not saying that working on a table is the same
18 work sample as working on a clipboard with 15 file folders, are
19 you, Dr. Barrett?
20 A. I'm saying it's the very same decision-making process, the
21 same administrative skills are called for to make decisions.
22 Q. And you could, that's a hypothesis that could be tested, is
23 that right?
24 A. I'm not sure what you are asking. Again, if you are asking
25 if we took 100 people and used a table, 100 people using the
615
Barrett - cross
1 tabular chair, clipboards and file folders, we could do a study
2 to see if there's a mean difference between those two groups in
3 terms of number correct, for example. Yes, we could do a study
4 like that. It's possible.
5 Q. You haven't done one, have you, Doctor?
6 A. No, I have not.
7 Q. Thank you.
8 Now, in the course of your work in preparing the 1994
9 lieutenants test for the City of Chicago, did you become aware
10 of something called the Mayor's Blue Ribbon Panel on Police
11 Testing and Hiring and Promotion, Reports and Recommendations?
12 A. Yes.
13 Q. And as a matter of fact, in your report about your test,
14 you make reference to that Mayor's Blue Ribbon Panel report, is
15 that right?
16 A. Yes.
17 Q. Did you consider that report when you devised your test?
18 A. Yes.
19 Q. Now, as you sit there now, were you aware that the Blue
20 Ribbon Panel recommended that attendance, medical and
21 disciplinary records should play a role in promotions?
22 A. I believe there was testimony earlier on in the trial, I
23 believe, about that, yes.
24 Q. Let me show you what's been marked as Plaintiffs' Exhibit
25 60.
616
Barrett - cross
1 MR. FLAXMAN: I think we all have a copy of this?
2 THE COURT: What is this?
3 MR. FLAXMAN: This is the Mayor's Blue Ribbon Panel
4 on Police Testing, Hiring and Promotion.
5 THE COURT: Defendant's 60?
6 MR. FLAXMAN: Plaintiffs' 60.
7 THE COURT: Go ahead.
8 BY MR. FLAXMAN:
9 Q. Do you see there's a second sentence under number 4 where
10 it says, "Some weight should also be given to the applicant's
11 attendance, medical, and disciplinary records"?
12 A. Yes.
13 Q. You didn't follow that recommendation, did you,
14 Dr. Barrett?
15 A. No.
16 Q. Now, also in that report there is a statement that at page
17 6 talking about hiring that, "The nearly unanimous view of
18 testing experts is that it is plainly incorrect to regard one
19 candidate as better qualified than another or more deserving of
20 a job simply because he or she scored slightly higher on a
21 written test."
22 Do you see that?
23 A. Yes.
24 Q. In the 1994 lieutenants promotional test, is it correct
25 that one candidate is ranked higher than another because he or
617
Barrett - cross
1 she scored higher on the test?
2 A. That's correct.
3 Q. And you're not one of those testing experts, are you, who
4 agrees that it's plainly incorrect to regard one candidate as
5 better qualified than another simply because he or she scored
6 slightly higher on a written test?
7 A. No.
8 Q. As a matter of fact, you're one of those testing experts
9 who believes that somebody who scored 99.91 will do better on
10 the job than somebody who scored 99.90 on one of your tests, is
11 that right?
12 A. What I said it is a general scientific proposition,
13 following the linear model, that the probability is that the
14 people who score higher on the test in general will score --
15 will perform better on the job, is what I've said.
16 Q. And it's your opinion that people who got a higher final
17 score on your test will do better on the job if promoted to be
18 lieutenant, is that right?
19 A. As a generalization, a scientific generalization, that's
20 correct.
21 Q. And you don't have any empirical data to support that
22 opinion, do you, Dr. Barrett?
23 A. Yes, I do.
24 Q. Do you have any measure of police performance that you
25 could relate scores on the test to, Dr. Barrett?
618
Barrett - redirect
1 A. If you're asking, again, if I did a criterion-related study
2 in Chicago, I never have.
3 Q. No.
4 My question, Dr. Barrett, is: You don't have any
5 measure of police performance to relate to scores on your test,
6 do you?
7 A. No, I've never measured police performance in Chicago.
8 MR. FLAXMAN: Thank you. I have nothing further.
9 REDIRECT EXAMINATION
10 BY MR. HOLZHAUER:
11 Q. Dr. Barrett, I believe you testified on cross-examination
12 that it would be a mistake if the written job knowledge exam
13 contained a question based on a general order that was not on
14 the reading list, is that correct?
15 A. Yes.
16 Q. And you were shown the linkage charts indicating that one
17 of the questions was linked to general order 84-7, is that
18 correct?
19 A. Yes.
20 Q. And you were also shown the version of the reading list
21 that appeared in your report which did not include general
22 order 84-7, is that correct?
23 A. Yes.
24 Q. I'd like to show you several exhibits now. And because
25 they come from different binders, I've made new copies for you
619
Barrett - redirect
1 and new copies for the Court and new copies for Mr. Flaxman.
2 I need the reading list, which is appendix F of your
3 report, the linkage chart, appendix K of your report, general
4 orders 84-7 to 94-5. This is the reading list, the linkage
5 chart and the general orders, general orders 84-7 to 94-5.
6 MR. FLAXMAN: Could we have a minute break, Judge?
7 Maybe I could just let him lead and not object or make a
8 representation, if I could just have a minute to look this over
9 before he asks questions. It's not a jury trial. If there is
10 a mistake --
11 THE COURT: But it is a trial, and my time is short.
12 Go ahead and take a look at it.
13 Aren't these all from the books?
14 MR. HOLZHAUER: These are all from the books, Your
15 Honor. I just thought it would be hard to have four or five
16 books going at once.
17 THE COURT: I appreciate that. It is.
18 (Pause.)
19 MR. FLAXMAN: Thank you very much, Your Honor.
20 MR. HOLZHAUER: I have another document, Your Honor,
21 that we've marked Defendant's Exhibit 29. Mr. Flaxman has that
22 as well. This is new.
23 BY MR. HOLZHAUER:
24 Q. Dr. Barrett, first I'd like for you to look at the linkage
25 chart that I handed you. Do you have that up there?
620
Barrett - redirect
1 A. Yes.
2 Q. In particular, look at question 8.
3 A. Yes.
4 Q. Okay. Can you tell me --
5 THE COURT: What exhibit is this from?
6 MR. HOLZHAUER: This is appendix K of Exhibit 1, Your
7 Honor.
8 BY MR. HOLZHAUER:
9 Q. Can you tell me what question -- what general order
10 question 8 is linked to in that linkage report?
11 A. It's linked to GO 84-7.
12 Q. Okay. Now, I'd like you to look for a second at the
13 reading list, which was part of appendix F of your report.
14 Does general order 84-7 appear on that reading list?
15 A. No, it does not.
16 Q. And Mr. Flaxman questioned you about that, is that correct?
17 A. That's correct.
18 Q. Looking at the reading list, what does it say -- perhaps
19 you can just read for us what it says in that paragraph on the
20 top that's marked number 1?
21 A. "Department directives as listed blow. The written
22 examination will be based only on directives issued prior to
23 December 1st, 1994, with the exception of GO 94-5 (payroll and
24 timekeeping), which will be issued September 27th, 1994."
25 Q. Thank you.
621
Barrett - redirect
1 Now, I'd like you quickly to look at general order
2 84-7 that I handed you a minute ago. That's the one you
3 couldn't find on the reading list. Do you have that there?
4 A. Yes, I do.
5 Q. What's the topic of that general order?
6 A. It's payroll and timekeeping.
7 Q. There is something handwritten on page 1. Can you read
8 that for us?
9 THE COURT: Maybe I'm having trouble here.
10 MR. HOLZHAUER: I'm sorry, Your Honor. It's the
11 white general order.
12 THE COURT: It's the white one?
13 MR. HOLZHAUER: Yes.
14 THE COURT: All right. Sorry.
15 BY THE WITNESS:
16 A. It says, "Rescinded by general order 94-5 on 27 September
17 '94."
18 BY MR. HOLZHAUER:
19 Q. Okay. Now, I'd like you to look at 94-5, the blue general
20 order that I handed you. What's the topic of that general
21 order?
22 A. Payroll and timekeeping.
23 Q. Now, I'd like you to look at what I've marked as
24 Defendant's Exhibit 29, what I've marked and handed up to you
25 with the new sticker on it.
622
Barrett - redirect
1 A. Yes.
2 Q. Can you tell me what that is?
3 A. That's question 8, which goes through our justification
4 procedure and also indicates the source is GO 94-5.
5 Q. Did you keep this -- is this a record that you kept as part
6 of the process of test development?
7 A. Yes.
8 Q. Were these entries made at the time of the events that are
9 reflected on the form?
10 MR. FLAXMAN: There is no time on the form.
11 MR. HOLZHAUER: The dates.
12 MR. FLAXMAN: There is no date. I'm looking at Bates
13 stamp 7488. I don't see a date.
14 THE COURT: 7488?
15 MR. FLAXMAN: Right, but I will agree that --
16 MR. HOLZHAUER: This comes up.
17 MR. FLAXMAN: Well, I'll agree that I was wrong about
18 the cross-examination, about the question, if that's what the
19 point of all this is.
20 THE COURT: Okay.
21 MR. HOLZHAUER: Well, that's the point.
22 But I would like to move Exhibit 29 into evidence so
23 that we have it. It is -- it comes in as a record. It also
24 comes in under 106. It's part of the set of documents that
25 you've already produced for a variety of questions.
623
Barrett - redirect
1 THE COURT: You're stipulating that you were wrong
2 regarding the cross-examination on question 8?
3 MR. FLAXMAN: That's correct, Your Honor. That it
4 is, in fact, based upon a general order that was on the reading
5 list. And I apologize for missing that.
6 THE COURT: Okay.
7 MR. HOLZHAUER: I'd like to move Exhibit 29 into
8 admission.
9 THE COURT: All right. It will be admitted.
10 (Defendant's Exhibit 29 received in evidence.)
11 BY MR. HOLZHAUER:
12 Q. Give me back that Exhibit 29 and keep the other things that
13 you have in front of you.
14 Now, looking back at that linkage chart that was from
15 appendix K, do you recall Mr. Flaxman asking you some questions
16 about question 5 on the linkage chart?
17 A. Vaguely.
18 Q. Do you recall acknowledging that the linkage for question 5
19 seemed to be in error?
20 A. Yes.
21 Q. What source is question 5 linked to?
22 A. It's linked to 93-3, addendum 5, two -- Roman numeral II,
23 D, capital D-2.
24 Q. I'd like to show you another new exhibit which we've marked
25 Defendant's Exhibit 30. Can you tell me what that is?
624
Barrett - redirect
1 A. This is our documentation for item No. 5.
2 Q. This is similar to the record we just looked at for item
3 No. 8?
4 A. Yes.
5 Q. And it relates to question -- which question?
6 A. This question No. 5.
7 MR. HOLZHAUER: I'd like to move Defendant's Exhibit
8 30 into evidence, Your Honor.
9 MR. FLAXMAN: I think there should be more foundation
10 for this document. There is no date. There is no --
11 MR. HOLZHAUER: I can go through it.
12 MR. FLAXMAN: There is nothing about it.
13 THE COURT: What is this document, Dr. Barrett?
14 THE WITNESS: This is our documentation of our
15 rationale and our justification for an item which we generate
16 as we write an item.
17 And the first part is the stem of the item, and then
18 we have the five alternatives. The star is the correct answer.
19 And it indicates that the source was GO 84-7.
20 Then we have the rationale and justification of why
21 the answer is correct and why the other alternatives or all
22 alternatives in this case, B, C, D and E, are not correct.
23 MR. HOLZHAUER: Your Honor, we were about to go
24 through the foundation of this for the first exhibit, which was
25 identical to this or identical as to another question when
625
Barrett - redirect
1 Mr. Flaxman stipulated that that was okay.
2 These are exactly the same as a series of documents
3 that Mr. Flaxman has introduced as to the questions that
4 Mr. Bishop has challenged. And I'll be glad to lay the
5 foundation if Mr. Flaxman would like me to.
6 THE COURT: Are they part of your report, Doctor?
7 THE WITNESS: No.
8 THE COURT: What are they? Where are they kept?
9 THE WITNESS: They're kept as working files in our
10 offices, in our office. And we gave one example, I believe, of
11 this in the report in the appendix.
12 THE COURT: Are they kept in the ordinary course of
13 your business?
14 THE WITNESS: Yes.
15 THE COURT: As part of your regular business?
16 THE WITNESS: Yes.
17 THE COURT: Are you in charge of the keeping of these
18 documents?
19 THE WITNESS: Yes.
20 THE COURT: I'm trying to save some time here,
21 Mr. Flaxman.
22 MR. FLAXMAN: I just would like to -- the missing
23 link is when was this thing prepared, at what point in the test
24 process?
25 THE COURT: That's a fair question.
626
Barrett - redirect
1 When was it prepared, Dr. Barrett?
2 THE WITNESS: I can't give you the exact date. It
3 was prepared after we had determined the source list and before
4 the final exam date, this ongoing process between those dates.
5 MR. FLAXMAN: I don't think that's enough foundation
6 to actually get it in as a business record.
7 MR. HOLZHAUER: I think it is, Your Honor, and --
8 THE COURT: I do, too. I do, too.
9 MR. HOLZHAUER: It's ongoing process.
10 THE COURT: It will be admitted.
11 (Defendant's Exhibit 30 received in evidence.)
12 BY MR. HOLZHAUER:
13 Q. Okay. Looking at the first page on which there is writing,
14 Bates stamp 7470, can you tell me what the correct answer to
15 this question is?
16 A. The correct answer has been marked as A.
17 Q. And can you tell from the form what general order the
18 answer is linked to?
19 A. Yes, it's GO 84-7.
20 Q. Now, looking back at your reading list, is 84-7 on the
21 list?
22 A. No.
23 Q. Is this the order that we just discussed that was
24 superseded by 94-5?
25 A. Yes.
627
Barrett - redirect
1 Q. Is 94-5 on the list?
2 A. Yes.
3 Q. Now, back to the linkage chart, could you look at question
4 6.
5 A. Yes.
6 Q. What source is question 6 linked to on the linkage chart?
7 A. 84-7.
8 Q. I'd like to show you another exhibit that I've marked
9 Defendant's Exhibit 31. Can you tell us what that is?
10 A. This is item 6 -- it's our documentation for item 6, which
11 contains the stem, correct answers and alternatives.
12 Q. Can you tell me, is this the same form, same business
13 record that we discussed and identified as No. 29 and No. 30?
14 A. Yes.
15 Q. And this is as to a different question?
16 A. Yes.
17 Q. No. 6?
18 MR. HOLZHAUER: I'd like to move 31 into evidence.
19 MR. FLAXMAN: Same objection.
20 THE COURT: Same ruling. It will be admitted.
21 (Defendant's Exhibit 31 received in evidence.)
22 BY MR. HOLZHAUER:
23 Q. Looking at the first page, Bates stamped 7476, can you tell
24 me what general order this is linked to?
25 A. 93-3.
628
Barrett - redirect
1 Q. Okay. Back to the linkage chart. Again, looking at No. 5
2 and No. 6.
3 A. Yes.
4 Q. Can you now tell me what the error is in No. 5 and in No.
5 6?
6 A. It looks like they are reversed in terms of the source;
7 that the source for 5 should be 6, and the source for 6 should
8 be 5.
9 Q. Those two items were transposed?
10 A. It looks that way, yes.
11 Q. Thank you.
12 THE COURT: You're saying they all have 93-3 as their
13 source?
14 THE WITNESS: No. I'm just saying that they're
15 transposed.
16 THE COURT: But all these questions seem to have 93-3
17 as their source, is that right? Am I wrong about that?
18 MR. HOLZHAUER: All of the alternatives?
19 THE COURT: Question 8, I'm sorry, question 8 is
20 94-5.
21 MR. HOLZHAUER: Right.
22 THE COURT: Question 5, question 5 and 6 both have
23 93-3 as their source.
24 MR. HOLZHAUER: Right.
25 No, Your Honor, I think question 5 has 94-5, which
629
Barrett - redirect
1 used to be called 84-7, as its source on the form. Question 6
2 has 93-3 on this form. You look at the linkage chart, what
3 appears next to 5 should appear next to 6, and what appears
4 next to 6 should appear next to 5.
5 THE COURT: I see. Okay.
6 BY MR. HOLZHAUER:
7 Q. Dr. Barrett, I'd like you to look at question 13 on the
8 linkage chart.
9 A. Okay.
10 Q. Do you recall Mr. Flaxman asking you some questions about
11 the chart relating to question 13?
12 A. Yes.
13 Q. Do you recall acknowledging that the linkage for question
14 13 also seemed to be in error?
15 A. Yes.
16 Q. What is the source listed for question 13 on your chart?
17 A. 87-7.
18 Q. Dr. Barrett, I would like to show you another document that
19 I have marked Defendant's Exhibit 32. Can you tell me what
20 that is?
21 A. This is item 13. And, again, it's the same business record
22 we keep as we generate our questions.
23 Q. As to which item?
24 A. It's to item 13.
25 MR. HOLZHAUER: Your Honor, I'd like to move
630
Barrett - redirect
1 Defendant's 13 -- 32 into evidence.
2 MR. FLAXMAN: Same objection.
3 THE COURT: Same ruling.
4 (Defendant's Exhibit 32 received in evidence.)
5 BY MR. HOLZHAUER:
6 Q. Now, looking at the first page, Bates stamped number 7522,
7 can you tell me from what general order -- what general order
8 this answer is linked to?
9 A. 88-15.
10 Q. Okay. Now, looking back at your reading list, is 88-15 on
11 the reading list?
12 A. Yes.
13 Q. I'd like to show you another document which is marked
14 Defendant's Exhibit 33.
15 But before I do that, I'd like you to look at your
16 linkage chart for No. 14.
17 A. Yes.
18 Q. Can you tell me what source No. 14 is linked to?
19 A. 88-15.
20 Q. Dr. Barrett, I just handed you a document I've marked as
21 Defendant's Exhibit 33. Can you tell me what that is?
22 A. This is our business record for item 14.
23 Q. The same kind of business record we've discussed for the
24 previous questions?
25 A. Yes.
631
Barrett - redirect
1 MR. HOLZHAUER: Your Honor, I'd like to move
2 Defendant's 33 into evidence.
3 MR. FLAXMAN: Same objection.
4 THE COURT: It will be admitted.
5 (Defendant's Exhibit 33 received in evidence.)
6 BY MR. HOLZHAUER:
7 Q. Looking at the first Bates stamped -- first page with words
8 on it, Bates stamped 7527, can you tell me from this form what
9 general order No. 14 is linked to?
10 A. 87-7.
11 Q. Now, again looking at your linkage chart, can you explain
12 the error that Mr. Flaxman found?
13 A. Again, it appears to be an identical error transposing 13
14 and 14 in terms of the source.
15 Q. Okay. I'd like you to turn to the second page of your
16 linkage chart. Look at No. 34.
17 A. Yes.
18 Q. Can you tell me what -- do you recall Mr. Flaxman asking
19 you questions about No. 34?
20 A. Yes.
21 Q. What source is question 34 linked to on the chart?
22 A. 91-14.
23 Q. I'd like to show you a document that we've marked
24 Defendant's Exhibit 34. Dr. Barrett, can you tell us what this
25 exhibit is?
632
Barrett - redirect
1 A. This is a business record we maintain for writing items.
2 In this case it's item No. 34.
3 MR. HOLZHAUER: Your Honor, I'd like to move
4 Defendant's Exhibit 34 into evidence.
5 THE COURT: It will be admitted.
6 MR. FLAXMAN: Same objection.
7 THE COURT: It will be admitted.
8 (Defendant's Exhibit 34 received in evidence.)
9 THE COURT: You have a standing objection,
10 Mr. Flaxman.
11 MR. FLAXMAN: Thank you.
12 BY MR. HOLZHAUER:
13 Q. Looking at the first page with words, it's Bates stamped
14 7624. Can you tell me what general order this item should be
15 linked to?
16 A. 85-1-IV-E-2-b.
17 Q. Looking back to the linkage chart, is the linkage chart
18 incorrect as to question 34?
19 A. Yes.
20 Q. Looking back at your reading list, is 85-1 on your reading
21 list?
22 A. Yes.
23 Q. So question 34 is based on a source in the reading list?
24 A. Yes.
25 Q. Okay. Let's go on to a new topic. Perhaps it would be
633
Barrett - redirect
1 easier if you just handed me back those forms.
2 Dr. Barrett, do you recall that on cross-examination
3 Mr. Flaxman asked you whether Diane Thompson would have been
4 promoted had she achieved a score of 60 on the in-basket?
5 A. Yes.
6 Q. Do you recall how many questions there were on the
7 in-basket exercise?
8 A. 60.
9 Q. So would 60 have been a perfect score?
10 A. Yes.
11 Q. Do you know if anybody received a 60?
12 A. I don't believe so.
13 Q. I'd like to show you Defendant's Exhibit 22.
14 MR. HOLZHAUER: That's the big spreadsheet, Your
15 Honor.
16 BY MR. HOLZHAUER:
17 Q. Dr. Barrett, we've stipulated that that's a list of scores
18 for all candidates. Do you see the column setting out the
19 in-basket scores?
20 A. Yes.
21 Q. Which column is it? It might be easier if you count from
22 the right, I think.
23 A. One, two, three, four, five, six.
24 Q. Can you go through that list now and tell me if anyone got
25 a perfect score of 60?
634
Barrett - redirect
1 MR. FLAXMAN: Judge, I think we all know what the
2 highest score was. We can stipulate to that.
3 THE COURT: Go ahead. Do it.
4 MR. FLAXMAN: 56.
5 MR. HOLZHAUER: Go ahead.
6 MR. FLAXMAN: I believe 56 was the highest score
7 anybody got on the in-basket.
8 MR. HOLZHAUER: We'll agree to that.
9 MR. FLAXMAN: Am I wrong?
10 MR. HOLZHAUER: I think it's right. I haven't gone
11 through it myself.
12 THE COURT: It's not 60?
13 MR. HOLZHAUER: It's not 60.
14 BY MR. HOLZHAUER:
15 Q. Dr. Barrett, do you remember Mr. Flaxman asking you about
16 your fees in this matter?
17 A. Yes.
18 Q. On direct I asked you about any outstanding invoices to the
19 City. Do you recall what your answer to that was?
20 A. I think it was 136 --
21 Q. Okay. I'd like to show you -- excuse me?
22 A. 136,000, as I recall.
23 Q. I'd like to show you a set of documents, Dr. Barrett. Can
24 you look at those and tell me what they are?
25 A. These -- it's a letter and an invoice sent to the -- from
635
Barrett - redirect
1 my office to the City of Chicago.
2 THE COURT: Are these marked?
3 MR. HOLZHAUER: No. I haven't marked these as
4 exhibits, Your Honor.
5 THE COURT: For what purpose are you showing them to
6 the witness?
7 MR. HOLZHAUER: I'm trying to -- we could mark them
8 as exhibits. But we're trying to clarify the point that
9 Mr. Flaxman made about additional --
10 THE COURT: You don't have to say anything else. Are
11 you refreshing his memory?
12 MR. HOLZHAUER: Yes.
13 THE COURT: Have you exhausted his memory?
14 MR. HOLZHAUER: I can exhaust his memory.
15 THE COURT: All right. Do you want to stipulate to
16 these?
17 MR. FLAXMAN: I've never seen these before.
18 THE COURT: All right. You don't want to stipulate
19 to these. Sit down, sit down. You don't want to stipulate.
20 Sit down. Exhaust his memory and use them properly. But mark
21 them. Give me a number.
22 MR. FLAXMAN: I want to object to their being used.
23 THE COURT: Well, he's not using them until -- sir, I
24 want you to turn them over right now. They're not marked.
25 First, he has to exhaust his recollection. Then he's allowed
636
Barrett - redirect
1 to use anything he wants to refresh it.
2 MR. FLAXMAN: Well, I'm not sure if he could use
3 things that were requested in discovery that were said we don't
4 have them and then produced them in trial.
5 MR. HOLZHAUER: Your Honor, Mr. Flaxman is incorrect
6 about that. We did produce these documents to Mr. Flaxman.
7 THE COURT: Well, I'm not getting into that. He can
8 use anything he wants to refresh the witness' recollection. It
9 doesn't mean they're coming into evidence, Mr. Flaxman.
10 He can use, you know, yesterday's Tribune if he wanted to.
11 MR. HOLZHAUER: That's next.
12 THE COURT: I hope not.
13 BY MR. HOLZHAUER:
14 Q. Dr. Barrett, do you recall on cross-examination that you
15 were asked whether there were any additional amounts you were
16 paid already for your work since the exam administration
17 regarding participation and your role in this litigation above
18 and beyond the $136,000 invoice already outstanding?
19 A. Are you referring to the 350, $350,000 for the actual
20 project itself?
21 Q. No. This $350,000 for the project?
22 A. Right.
23 Q. There is a $136,000 outstanding invoice now?
24 A. Yes.
25 Q. I believe Mr. Flaxman asked you a series of questions that
637
Barrett - redirect
1 related to other amounts that you had been paid above and
2 beyond the 350,000 that related to your work on this case since
3 the exam was given. Do you recall that?
4 A. Vaguely, yes.
5 Q. Do you recall whether you have sent the City invoices for
6 additional amounts of work other than the 350,000 and other
7 than the 136,000?
8 A. Yes, we have.
9 Q. Have you been paid for those amounts?
10 A. As I recall, yes.
11 Q. Do you recall what those amounts were?
12 A. No.
13 Q. I'd like to show you some documents to refresh your
14 recollection about that. Can you tell me what those documents
15 are?
16 A. The first document is a letter from, from one my associates
17 to Mr. Pat Rocks, which includes an invoice for $21,000. And
18 the date is 2-20-96.
19 Q. That's for additional work on this, this matter?
20 A. Yes. It says, "Work completed in the matter of Brown v.
21 City of Chicago."
22 Q. Okay. Would you keep going, please.
23 A. There's a second letter dated March 21st, 1996, again from
24 an associate, with an invoice which is in the amount of over
25 $54,000. And, again, it's for work completed in the matter of
638
Barrett - redirect
1 Brown v. City of Chicago.
2 Q. Okay. Can you keep going through all of these documents,
3 please.
4 A. The next is a letter of July 6th, 1995, from one of my
5 associates, including an invoice. The invoice is for a little
6 over $3,000. And again, it has to do with Brown v. City of
7 Chicago.
8 There is a letter, again from my associate dated
9 August 9th, 1995, which includes an invoice, and the invoice
10 amount is $62.50, again, for Brown v. City of Chicago.
11 And the last letter is from an associate which
12 includes an invoice, and I don't see a total. Oh, there it is.
13 Total invoice, $136,000, again for Brown v. City of Chicago.
14 Q. Is this last one the amount that's still outstanding?
15 A. Yes.
16 Q. So the others are amounts that have already been paid?
17 A. Yes.
18 Q. Looking at those together, you've been paid roughly $80,000
19 in addition to the 350,000 in addition to the 136,000, is that
20 correct?
21 A. That sounds correct.
22 Q. Do you recall Mr. Flaxman then trying to draw a comparison
23 between the amounts the City has paid you and the additional
24 cost of subjective scoring that Dr. York said would be needed
25 to make your test the traditional in-basket test?
639
Barrett - redirect
1 A. Yes.
2 Q. I'd like to show you a document we marked Defendant's
3 Exhibit 3. Can you tell me what that is?
4 A. This is a response to the expert witness reports and
5 depositions of Dr. York, Koziol, K-o-z-i-o-l, and Levitt, and
6 Mr. Bishop in the matter of Brown v. City of Chicago. The date
7 is October 13th, 1997.
8 Q. I'd like you to turn to page 27 and 28 of that report.
9 A. Yes.
10 Q. Will you tell me what the analysis on those pages is?
11 A. I made an analysis of what it would cost to use live
12 assessors in what Dr. York has termed a more traditional
13 process. And there I used, I believe, his rate of compensation
14 and determined what it would cost the City to perform that
15 process.
16 Q. And what was your estimate, Dr. Barrett?
17 A. Yes, it was approximately $6 million.
18 Q. Now, can you tell us what the table 3 on page 28 is?
19 A. That's my estimated costs of a traditional in-basket.
20 Q. Now, can you tell me approximately how many hours would be
21 spent on scoring the traditional in-basket eliminating the
22 training time of the raters?
23 A. How many hours, with no training you're saying?
24 Q. Yes. Eliminate the training.
25 A. It's 12, 15, 18,000 hours.
640
Barrett - redirect
1 Q. And do you know how much that would come to at --
2 A. That would be $3.6 million.
3 Q. Okay. Do you know what hourly rate you generally charge
4 for raters?
5 A. How I charge?
6 Q. Yes.
7 A. It would be -- according to who is doing it, our rate would
8 vary between probably 70 to $150 an hour.
9 Q. So to calculate what it would cost for your raters to do
10 this job, you would multiply something between 70 and 150 times
11 18,000?
12 A. Yes, if I used those people. But I'm talking about rating
13 for -- in our shop, yes.
14 Q. Was cost the primary reason for using multiple-choice
15 answers for your in-basket?
16 A. No.
17 Q. What was the primary reason?
18 A. In terms of objectivity and reliability, elimination of
19 bias.
20 Q. Let me take that book out of your way.
21 On the written job knowledge test, you testified
22 earlier that there was a pilot test, is that correct?
23 A. Yes.
24 Q. Do you recall Mr. Flaxman asking you a number of questions
25 about the time allotted for the various exam components?
641
Barrett - redirect
1 A. Yes.
2 Q. I'd like to have you look at your report, Exhibit 1. Look
3 at page 116. Can you tell us what is described on page 116?
4 A. The pilot testing of the written job knowledge exam.
5 Q. How many questions were on the pilot test?
6 A. 186.
7 Q. Do you know how much time the test takers were given?
8 A. Three hours.
9 Q. How does that work out in minutes per question?
10 A. One minute.
11 Q. Did you consider the pilot test results when you decided
12 whether the timing of the job knowledge test was appropriate?
13 A. Yes.
14 Q. Was there a problem with the amount of time given during
15 the pilot test?
16 A. No.
17 Q. Dr. Barrett, do you recall that Mr. Flaxman asked you on
18 cross-examination whether a three-part test consisting of an
19 in-basket exercise, an oral briefing exercise, and a job
20 knowledge component is the standard package offered by Barrett
21 & Associates?
22 A. Yes.
23 Q. Does every promotional exam developed by Barrett &
24 Associates consist of an in-basket exercise, an oral briefing
25 exercise and a written job knowledge test?
642
Barrett - redirect
1 A. No.
2 Q. What other exam designs have you used?
3 A. The most common exam is often -- a very common exam is the
4 job knowledge test by itself.
5 Q. In what circumstances do you use that kind of exam?
6 A. There is -- I've got to be very specific, I guess. In the
7 City of Jacksonville, Florida, for example, that's the design
8 we used, have been using for a number of years, because there
9 is a concern by the fire fighters that any other process will
10 not be as objective.
11 MR. FLAXMAN: Can I object to the -- his explanation
12 because of what other people think?
13 THE COURT: Hearsay?
14 MR. FLAXMAN: Yes.
15 THE COURT: All right. It won't be accepted for its
16 truth. It's accepted for his reasons for doing it, though.
17 Whether it's true or not doesn't matter.
18 BY MR. HOLZHAUER:
19 Q. So that was in the City of Jacksonville, did you say?
20 A. Yes.
21 Q. And in what force was that?
22 A. It's the fire fighters.
23 Q. Was that promotional or entry level?
24 A. It's promotional exams.
25 Q. Okay. And when was that exam given?
643
Barrett - redirect
1 A. We've been giving that exam now for a number of years. I'm
2 not sure exactly when we started, around early 1990s, I
3 believe.
4 Q. How recently has it been given?
5 A. As recently as about two months ago, I think.
6 Q. So all of the recent public safety promotional exams you've
7 given have not used the similar three-part exam?
8 A. That's correct.
9 Q. Is it possible that a job analysis of some police or fire
10 job might reveal that that three-part structure would not be
11 appropriate for that job?
12 A. You could infer, yes.
13 Q. What would you do under those circumstances?
14 A. Only use what was appropriate.
15 Q. When you develop promotional exams for different police and
16 fire departments, do you use the same questions and the same
17 resource materials each time?
18 A. No.
19 Q. Has your standard three-component test been found to be
20 content valid in every case in which its validity was
21 challenged?
22 MR. FLAXMAN: Objection, not material, especially if
23 they're different questions, different tests. It's his
24 reputation that he's trying to prove up here.
25 MR. HOLZHAUER: Let me ask one foundational question
644
Barrett - redirect
1 and try it again, Your Honor.
2 THE COURT: Okay.
3 BY MR. HOLZHAUER:
4 Q. Do you follow the same content validation process in the
5 other promotional exams for police and fire departments?
6 MR. FLAXMAN: Objection. This sounds like habit
7 testimony.
8 THE COURT: It sounds like what?
9 MR. FLAXMAN: Habit testimony.
10 THE COURT: Habit.
11 MR. HOLZHAUER: The issue in this case is whether his
12 process is appropriate.
13 THE COURT: I agree. Overruled.
14 BY THE WITNESS:
15 A. Yes.
16 BY MR. HOLZHAUER:
17 Q. Has your standard three-component test been found to be
18 valid content in every case where its validity has been
19 challenged?
20 MR. FLAXMAN: Same objection.
21 THE COURT: Overruled.
22 BY THE WITNESS:
23 A. Yes.
24 THE COURT: Your objection is to weight more than
25 admissibility, I think.
645
Barrett - redirect
1 BY MR. HOLZHAUER:
2 Q. Dr. Barrett, Mr. Flaxman asked you several questions about
3 the challenge procedure administered by Arthur Andersen. Do
4 you recall that?
5 A. Yes.
6 Q. What is the Educational Testing Service?
7 A. It's a nonprofit organization which develops a wide variety
8 of tests in the academic area and in the public and private
9 sectors.
10 Q. Could you tell us what the best known Educational Testing
11 Service tests are?
12 A. Probably the Scholastic Aptitude Test, the SAT, is probably
13 the widest known.
14 Q. Does it also do employment tests?
15 A. Yes. They do employment tests.
16 Q. Do you have any knowledge -- is the Educational Testing
17 Service a large organization?
18 A. It's very large.
19 Q. Among organizations that design and develop tests, how
20 would you say that its size ranks?
21 A. I would have to estimate, make an estimate from -- I've
22 been invited to its campus in New Jersey as a discussant on a
23 panel. They just do a wide variety. It's a very large group
24 of -- there's hundreds of people involved in that organization.
25 Q. Do you have any knowledge about the challenge procedures
646
Barrett - redirect
1 that are implemented by the Educational Testing Service?
2 MR. FLAXMAN: Objection, relevance.
3 THE COURT: Sustained, sustained.
4 BY MR. HOLZHAUER:
5 Q. Dr. Barrett, to your knowledge, did some candidates
6 challenge particular questions on the lieutenants exam?
7 A. Yes.
8 Q. As a result of those challenges, were some questions
9 eliminated?
10 A. Yes.
11 Q. Were some questions modified as to the correct answer?
12 A. Yes.
13 Q. What form did that modification take?
14 A. I believe in one case two alternatives, A and B, for
15 example, were both counted as correct.
16 Q. Do you recall for how many questions were either -- were
17 eliminated?
18 MR. FLAXMAN: I think we did this on direct.
19 THE COURT: Do it on direct or cross? I know it
20 sounds very familiar.
21 MR. HOLZHAUER: It is. And I have one follow-up
22 question on it. I think it will be useful to have the evidence
23 of the number of questions, Your Honor.
24 THE COURT: All right. Go ahead.
25 BY THE WITNESS:
647
Barrett - redirect
1 A. As I recall, there were two on the job knowledge test and
2 two on the in-basket which were challenged and we made some --
3 some modifications were made, either in terms of giving
4 everyone, giving everyone points or giving two options of the
5 points.
6 BY MR. HOLZHAUER:
7 Q. So three questions altogether?
8 A. Four.
9 Q. Four.
10 Did it surprise you that changes were necessary in
11 that many questions?
12 MR. FLAXMAN: Objection.
13 THE COURT: Sustained.
14 BY MR. HOLZHAUER:
15 Q. Does the number of challenges in this case, how does the
16 number of challenges in this case compare with your experience
17 in other cases?
18 MR. FLAXMAN: Objection.
19 THE COURT: Overruled.
20 BY THE WITNESS:
21 A. It's far fewer.
22 MR. HOLZHAUER: Thank you.
23 BY MR. HOLZHAUER:
24 Q. Now, you testified on cross that candidates were not
25 allowed to take their answer sheets with them or to make notes
648
Barrett - redirect
1 and take them out of the test -- test room. Do you recall
2 that?
3 A. Yes.
4 Q. Why do you do that?
5 A. To maintain the confidentiality of the test and the
6 procedure.
7 The a COURT: How much longer do you have?
8 MR. HOLZHAUER: Probably a very few minutes. Less
9 than five minutes, I think.
10 BY MR. HOLZHAUER:
11 Q. I'd like to refer you to Defendant's Exhibit 27. I can
12 take that other book.
13 Dr. Barrett, before we get into that exhibit, would
14 you just tell me why you maintain the confidentiality of test
15 questions?
16 A. It's often the standard procedure of a civil service
17 organization to do that. It's more of a policy decision of the
18 organization. It's not always done on a content-valid test in
19 terms of maintaining confidentiality.
20 Q. I'd like you to look at page 208 of this document.
21 A. Okay.
22 Q. What is this document?
23 A. This is the EEOC guidelines of 1978.
24 Q. Do you recall Mr. Flaxman asking you questions about
25 whether content validation can be used on the in-basket tests
649
Barrett - redirect
1 insofar as it involved judgment and decision making?
2 A. Yes.
3 Q. And you answered that it could?
4 A. Yes.
5 Q. And you also answered you didn't think your view was
6 contrary to the EEOC guidelines?
7 A. Yes.
8 Q. Can you explain that for me.
9 A. In that section he was referring to, it was, I think,
10 referring to the type of test which you take off the shelf,
11 such as a general intelligence test or a general test of
12 judgment of a very generic nature, where you're using a test to
13 measure a construct.
14 Again, the best example is an intelligence test of
15 where you are measuring a construct called intellectual
16 ability, and there's a standardized test which you can take off
17 the shelf and use for selection purposes.
18 Q. Thank you. I can take that from you.
19 Dr. Barrett, do you recall testifying both on direct
20 and cross-examination that your in-basket exam is a simulation?
21 A. Yes.
22 Q. And do you recall that Mr. Flaxman pointed to the in-basket
23 exercise and asked you whether you knew if a lieutenant would
24 have to do two weapons discharge incidents in one day?
25 A. Yes.
650
Barrett - redirect
1 Q. And you also asked him whether they would ordinarily be
2 doing a number of scheduling matters without clerical
3 assistance in that day?
4 A. Yes.
5 Q. Why do you test on more than what would occur in a
6 particular day?
7 A. Because it is a psychometric test. As I said early on last
8 week, it's a compression into a short period of time a number
9 of events which the lieutenant would actually do on the job.
10 But you're compressing it into a short period of time and
11 making it into a psychometric test.
12 THE COURT: Psychometric?
13 THE WITNESS: Psychometric, yes.
14 BY MR. HOLZHAUER:
15 Q. Dr. Barrett, do you recall questions about question No. 3,
16 No. 6, No. 11, and No. 13 of the in-basket exam that used the
17 term "discrepancy," and "discrepancy that had an impact"?
18 A. Yes.
19 Q. Was the term "discrepancy" in those questions intended by
20 you to be a replication of police jargon?
21 A. Not necessarily, no.
22 Q. What was it intended to be?
23 A. Give clear directions to the testees in terms of what was
24 being asked.
25 Q. Now, Mr. Flaxman also asked you questions about question 47
651
Barrett - redirect
1 and 48 and said that some of the issues as to scheduling and
2 time off that were raised by 47 and 48 would be made by
3 clerical employees, not by lieutenants. Do you recall that?
4 A. Yes.
5 Q. If they are made by clerical employees, why would they be
6 on your test?
7 A. Well, one reason, if in fact he's correct, that the
8 responsibility of lieutenant is to supervise these activities,
9 and, in fact, ensure that what has ever been done by clerical
10 employees is accurate and correct.
11 Q. Thank you.
12 Now, you testified, you were asked on
13 cross-examination at length about the Lopez monograph, is that
14 correct?
15 A. Yes.
16 Q. Were there any other studies of objective scoring systems
17 with in-basket that you relied upon?
18 A. Yes.
19 Q. Can you tell us what they were?
20 A. As I said, there is a long history of that starting with
21 Fredrickson and Lopez, of course. Kesselman has an article, as
22 I recall, was one which had it. And since that time, there has
23 been research by Hakstian up to the present day looking at the
24 more objective in-baskets.
25 Q. The Hakstian article was published in what year, do you
652
Barrett - redirect
1 recall?
2 A. 1997 is the most recent. But he had one in the late 1990s.
3 Of course, there is a review also by Shipman, et al.,
4 in Personnel Psychology, I believe, which is a review of
5 various in-basket techniques where they do mention the
6 objective approach.
7 MR. HOLZHAUER: Two more questions.
8 BY MR. HOLZHAUER:
9 Q. Do you recall that there were questions regarding reading
10 comprehension that you were asked by Mr. Flaxman?
11 A. Yes.
12 Q. And you testified that you looked at the reading level of
13 the exam questions?
14 A. Yes.
15 Q. Can you tell us how you did that with the process of
16 evaluating what the reading level was?
17 A. It's a standard Flesch count type approach where you
18 basically let the computer scan the material and you get an
19 index in terms of grade level.
20 Q. Did you match the grade level of the exam questions with
21 the grade level of something else?
22 A. Yes, I did.
23 Q. What was that something else?
24 MR. FLAXMAN: Objection. This is now hearsay about
25 what the computer told him something is. I don't know if Your
653
Barrett - redirect
1 Honor heard the beginning of --
2 THE COURT: I'm reading it.
3 (Pause.)
4 THE COURT: Overruled.
5 BY THE WITNESS:
6 A. I compared it with the source lists we used.
7 BY MR. HOLZHAUER:
8 Q. The material that lieutenants read on the job?
9 A. Yes.
10 Q. And how did it compare?
11 A. The reading level of our test is lower than the reading
12 level of material, material the lieutenants are required to
13 read on their job.
14 Q. Is that all described in your report?
15 A. Yes, it is.
16 MR. HOLZHAUER: Thank you. No further questions.
17 THE COURT: All right. We'll take an hour break.
18 (Recess at