360
1 IN THE UNITED STATES DISTRICT COURT
NORTHERN DISTRICT OF ILLINOIS
2 EASTERN DIVISION
3 ERNEST T. BROWN, et al., )
)
4 Plaintiffs, )
) No. 95 C 1890
5 v. ) Chicago, Illinois
) November 20, 1997
6 CITY OF CHICAGO, ) 9:30 a.m.
)
7 Defendant. )
8
9 TRANSCRIPT OF PROCEEDINGS - TRIAL
10 BEFORE THE HONORABLE ROBERT W. GETTLEMAN
11 APPEARANCES:
12 For the Plaintiffs: KENNETH N. FLAXMAN, P.C.
122 South Michigan Avenue
13 Chicago, Illinois 60603-6107
BY: MR. KENNETH N. FLAXMAN
14
FUTTERMAN & HOWARD, CHTD
15 122 South Michigan Avenue
Chicago, Illinois 60603
16 BY: MR. CRAIG B. FUTTERMAN
17 For the Defendant: MAYER, BROWN & PLATT
190 South LaSalle Street
18 Chicago, Illinois 60603
BY: MR. JAMES HOLZHAUER
19 MR. JEFFREY S. PIELL
MS. ANGELA K. DORN
20 MR. ANDREW NICELY
21
22 Official Court Reporter: JENNIFER S. COSTALES, CSR, RMR
219 South Dearborn Street
23 Room 1744-A
Chicago, Illinois 60604
24 (312) 427-5351
25
361
Barrett - direct
1 (Proceedings heard in open court.)
2 THE CLERK: 95 C 1890, Ernest Brown versus City of
3 Chicago, on trial.
4 MR. HOLZHAUER: Your Honor, when we closed yesterday
5 you were asking us some questions about --
6 THE COURT: Somebody close the door, please.
7 MR. HOLZHAUER: You were asking us some questions
8 about schedule. Mr. Flaxman and I have conferred. Our
9 witnesses are all here and ready to go. I think, judging by
10 -- depending how long the cross is, we'll probably spend most
11 if not all of today with Barrett.
12 THE COURT: Remember, we break at 4:00 o'clock
13 today.
14 MR. HOLZHAUER: We break at 4:00. I think that will
15 probably be about the conclusion of Dr. Barrett, perhaps.
16 We have one long witness, Lieutenant Klein, who will
17 be going Monday. Our last two witnesses are very short
18 witnesses, depending on the length of cross.
19 We certainly feel we'll finish our case Monday or
20 perhaps Tuesday.
21 THE COURT: You know, if Dr. Barrett is almost
22 through today, but not quite, I had some matters cancel
23 tomorrow, so I could give you an hour or so tomorrow if you
24 just have to wrap him up.
25 MR. HOLZHAUER: All right. He has a plane going out
362
Barrett - direct
1 -- I think we'll be able to wrap him up.
2 THE COURT: I'm just saying that I have that
3 flexibility.
4 MR. FLAXMAN: It looks like we're going to end all
5 of the testimony a week from next Tuesday, which would give
6 you the Wednesday and Thursday of that week which you had
7 allocated to us to do other things.
8 THE COURT: Okay. Good.
9 MR. HOLZHAUER: I also understand, Mr. Flaxman, that
10 we might have mornings only on that Monday or Tuesday, judging
11 from your witnesses.
12 MR. FLAXMAN: We have one witness on Monday and one
13 on Tuesday, and we might have another one Monday afternoon.
14 And that's it.
15 THE COURT: All right. Well, fine. Okay.
16 Dr. Barrett, please take the stand. You are still
17 under oath, sir.
18 DR. GERALD V. BARRETT, DEFENDANT'S WITNESS, PREVIOUSLY SWORN
19 DIRECT EXAMINATION
20 RESUMED
21 BY MR. HOLZHAUER:
22 Q. Good morning, Dr. Barrett.
23 A. Good morning.
24 Q. When we ended yesterday, I believe we were reviewing the
25 oral briefing exercise; is that right?
363
Barrett - direct
1 A. Yes.
2 Q. How was the oral briefing exercise scored?
3 A. It was scored by three trained raters. The raters sat in
4 a room and independently rated each tape following a very
5 structured rating procedure.
6 After they had each rated each tape, they then
7 conferred and reached a consensus. If it was required, they
8 would play the tape back any number of times to be sure they
9 all agreed on the final score.
10 Q. You say a structured rating. Did they have a form that
11 they worked with?
12 A. Yes.
13 Q. How did you decide upon that scoring method for the oral
14 exercise?
15 A. It evolved over ten years in terms of our use of that
16 sort of procedure. Basically it's a very objective procedure
17 where there is little or no possibility of bias.
18 We have conducted research in the area of oral
19 communication, and we have found that in fact you can be
20 biased if there's -- if there is some lack of structure in the
21 rating process. So basically it's our -- based on our past
22 research and past experience, that's the reason we evolved
23 that sort of objective technique.
24 Q. You said that the raters first went through the
25 structured rating process. Was that independent, or did they
364
Barrett - direct
1 work together on that structured part?
2 A. That was independent.
3 Q. And then did they work together?
4 A. Yes. If there was any inconsistencies in their rating
5 process.
6 Q. Were the raters provided any information about the
7 identity of the voices on the tapes?
8 A. No. All you had was a test identification number.
9 Q. Was any analysis performed to assess whether the three
10 different versions of the exercise of the oral briefing
11 exercise were equivalent?
12 A. Yes.
13 Q. What was that?
14 A. We had forms X, Y and Z, and after all the ratings were
15 accomplished, we did a statistical approach to see if there
16 was any significant differences based upon form X, Y or Z. We
17 found no significant difference.
18 Q. Now, you testified earlier that the two other components
19 of the test were the written job knowledge component and the
20 written in-basket exercise. Were they administered on the
21 same day?
22 A. Yes.
23 Q. Well, let's start with the written job knowledge test.
24 Why did you decide to include a written job knowledge test in
25 the exam?
365
Barrett - direct
1 A. For several reasons. First, past research has shown the
2 importance of job knowledge for all jobs; and second, our job
3 analysis, which is consistent with other job analyses, showed
4 that in fact job analysis was critical and important for many
5 major work behaviors.
6 Q. Could you briefly describe for us the format of the test,
7 how many questions, how many choices, how much time and so
8 forth?
9 A. The job knowledge test consisted of 150 items. The
10 people, the candidates were allowed two and a half hours to
11 take the exam.
12 Q. Was this a multiple choice test?
13 A. Yes, it was a multiple choice test, a stem, and five
14 alternatives, one of which was correct.
15 Q. I'd like to show you a document that's been marked
16 Defendant's Exhibit 14, and it's been received in evidence.
17 I'd like you just to let me know what it is for
18 identification, please.
19 A. This is the Chicago police lieutenant written
20 examination.
21 Q. Thank you, Dr. Barrett.
22 Let's review the process by which you developed this
23 component of the exam.
24 Was the written exam designed to test any particular
25 body of job knowledge, skills and abilities?
366
Barrett - direct
1 A. Yes.
2 Q. What was it?
3 A. The domain of job knowledge was that which was contained
4 in our source material or reading list, which was given to all
5 candidates.
6 Q. How did you go about drafting those questions?
7 A. We had trained item writers who would review the
8 documents, review the job analysis and job descriptions, and
9 also look for areas where our job analysis indicated were
10 critical events, things which police lieutenants do do on the
11 job. So the stem often reflected an event, which did require
12 underlying knowledge to actually perform that activity.
13 Q. Dr. Barrett, you're using the term "stem." What do you
14 mean by that?
15 A. That's really the question, the question stem.
16 We had a very structured process in writing items.
17 Once an item was drafted, the alternatives, it was in a form,
18 and it was passed around and reviewed by item reviewers, it
19 was reviewed by four subject matter experts, and then the very
20 final review was my own review. I reviewed all the items.
21 Q. Who were these trained exam or question writers?
22 A. They were employees of Barrett and Associates. They all
23 had at least a master's degree in industrial psychology, had
24 all been trained to write items.
25 Q. Where were they from?
367
Barrett - direct
1 A. Akron, Ohio.
2 Q. They weren't Chicago police officials?
3 A. No.
4 Q. Did the item writers go through any particular training
5 for this job?
6 A. Yes, we have a structured proprietary training program.
7 Q. What did that consist of?
8 A. Basically it goes through all the item writing rules:
9 Here is how you should write an item in terms of the
10 structure, what the alternatives should be, how the
11 alternatives should be plausible, how you document the
12 process.
13 Q. Do they receive any diversity training?
14 A. Yes. We had a diversity training program for everyone,
15 just not item writers, but everyone who was working on the
16 project.
17 Q. Did you test for every bit of information on the reading
18 list material?
19 A. No.
20 Q. Would there be general orders and special orders and
21 other items on the reading list that were not the subject of
22 any test question?
23 A. Yes.
24 Q. How did that happen?
25 A. There's probably two main reasons. When you review the
368
Barrett - direct
1 material, you might find that the order is not amenable to
2 writing the question. Maybe it's a little bit vague or
3 whatever it might be.
4 A second consideration might be an item which was
5 written may be rejected as not being good enough by an
6 internal review process or by the subject matter experts. And
7 so you try to put in the most important material.
8 Q. Does the fact that some items on the reading list were
9 not tested diminish the validity of the test?
10 A. No.
11 MR. FLAXMAN: Objection. That assumes that there is
12 validity of the test.
13 THE COURT: I think the question is a fair
14 question. I understand the question. Overruled.
15 BY THE WITNESS:
16 A. No.
17 BY MR. HOLZHAUER:
18 Q. How did you decide what material to cover and how many
19 questions to ask regarding particular items on the reading
20 list?
21 A. Well, again, we tried to develop those situations for
22 which job knowledge was required and we could write a good,
23 professional item.
24 Q. Approximately how many written job knowledge questions
25 did your staff prepare?
369
Barrett - direct
1 A. We wrote several hundred. We pilot tested 186 before the
2 SMH reviewed it. So I don't know the exact number because
3 it's an additive process.
4 Q. Why did you select the multiple choice format?
5 A. This is a format which is very objective. There is no
6 possibility of bias.
7 Q. What do you mean by "objective"?
8 A. That means that there is one correct answer, it's
9 computer scored. It seemed to be very fair by candidates.
10 Q. Did you perform the same kind of linkage analysis that
11 you performed for the oral briefing exercise for the written
12 job knowledge test?
13 A. Yes.
14 Q. Once again, was the linking analysis -- what was the
15 linking analysis designed to demonstrate?
16 A. Just to show that everything we did test on was relevant,
17 we could link it to major work behaviors; that, in fact, one
18 concern is that you're going to be testing on something which
19 is not relevant to the job of the lieutenant. Of course, our
20 basic source knowledge list, of course, was by itself very
21 relevant, but just to be sure we were able to demonstrate this
22 was true.
23 Q. Were you able to link every one of the test items to
24 major work behaviors, knowledge, skills, and abilities?
25 A. Yes.
370
Barrett - direct
1 Q. Were the test questions and answers checked against the
2 source documents from the reading list?
3 A. Yes.
4 Q. By whom?
5 A. By our item writers and by the subject matter expert.
6 Q. Now, you mentioned a minute ago a pilot test. You ran a
7 pilot test of this exam?
8 A. Yes.
9 Q. How was that conducted?
10 A. We had ten, again, graduate students, eight whites, two
11 blacks, who went through the exam and took it looking for
12 inconsistencies, typos, various types of errors.
13 Q. Where did that take place?
14 A. That took place in Akron, Ohio, September 28 to 29th,
15 1994.
16 Q. Did you take any steps to ensure confidentiality of the
17 exam questions?
18 A. Yes. We took the same steps we did for the oral, of
19 people who were admitted into our secure quarters without any
20 sort of paper, pencil, et cetera. They were informed of the
21 confidential nature of what they were going to be doing. They
22 also signed a confidentiality agreement. And before they left
23 we were sure no pages of any of the tests were ever taken out
24 of the office.
25 Q. What use did you make of the pilot test results?
371
Barrett - direct
1 A. It helped us just to refine the test items.
2 Q. Did anyone from the Chicago Police Department review the
3 proposed test questions?
4 A. Yes.
5 Q. Who were they?
6 A. It was Chief Cadogan, Deputy Chief Shaw, Commander
7 DeLopez, and Lieutenant Klein. They came to Akron on October
8 1st through the 6th for the first review session.
9 Q. What did they do at that session?
10 A. We had the item writers individually go through each of
11 the items of the source material so they could see if in fact
12 the items were relevant. If they were any mistakes, they had
13 the source materials available, they could check the sources.
14 So, again, it was a check to be sure that the items
15 we wrote were relevant to the job of a Chicago police
16 lieutenant, and it was being -- the review was being performed
17 by people who were very knowledgeable about that position.
18 Q. You testified earlier that you actually prepared a good
19 deal of -- a number larger than 150 test questions initially.
20 Did the subject matter experts review all of those
21 questions?
22 A. Yes.
23 Q. Did they make recommendations that any of the questions
24 be revised or deleted?
25 A. Yes, they did.
372
Barrett - direct
1 Q. Did you follow those recommendations?
2 A. Yes, we did.
3 Q. Did they recommend any additional subject matter areas
4 that should be added to the test?
5 A. Yes.
6 Q. Did you follow those recommendations?
7 A. Yes.
8 Q. And you developed -- did you develop additional
9 questions?
10 A. Yes, as a matter of fact, we developed additional
11 questions, and in fact we revised for about two weeks, and
12 they came back for a second review session around the --
13 October 18th and 19th they came back, of 1994, for a second
14 review session, to look at the final form of the written job
15 knowledge test.
16 Q. So they reviewed the final form in that second review
17 session?
18 A. Yes.
19 Q. Did they review each and every item on the test?
20 A. Yes.
21 Q. You testified earlier that the written job exam was
22 administered on November 19, 1994. Were you present for that
23 administration?
24 A. Yes.
25 Q. Can you briefly describe how the test was administered?
373
Barrett - direct
1 A. Well, as always, the administration process was under the
2 control of Arthur Andersen. We came as a crisis team to
3 handle any unusual problems.
4 So again, it was a very structured process, highly
5 controlled, where we delivered the test booklets. We actually
6 produced the test booklets at our Akron office, transported by
7 truck with our people, taken to the Arthur Andersen facility,
8 which is also secure. It was delivered to the -- I believe it
9 was the Daley College, by Arthur Andersen.
10 Standardized instructions were given to the people,
11 the packets were passed out with the computer scoring sheet,
12 and they were given two and a half hours to complete.
13 There were monitors in all of the rooms. Security
14 was very, very tightly controlled.
15 Q. We'll come back to the timing in a second, but you
16 mentioned that you were there in case any particularly urgent
17 things came up.
18 Did any particularly urgent things come up during
19 the administration?
20 A. There were always perhaps a few things, nothing which was
21 -- compromised the validity of the testing process. There
22 were some complaints about a couple things about the rooms
23 perhaps, but nothing major for that written test.
24 Q. You said that -- how long did they have to complete that
25 exam?
374
Barrett - direct
1 A. That was -- the written test was two and a half hours.
2 Q. How did you decide on that time?
3 A. That's based on our past practice, and our pilot
4 studies.
5 Q. Did the Chicago Police Department subject matter experts
6 review the time allotted?
7 A. Yes.
8 Q. Did they give you any feedback on it?
9 A. They felt the time was correct, two and a half hours.
10 Q. Okay. Now, how was the test scored?
11 A. The test was computer scored by -- first I should say --
12 go back. What as I recall we did, the answer sheets were
13 scanned separately by ourselves and Arthur Anderson. We
14 brought a test scanner, and that data was entered onto a
15 computer file.
16 They separately -- Arthur Andersen scored the answer
17 sheets and retained the answer sheet. We went back to Akron
18 for our computer file. They went to their offices. They were
19 scored separately, and the scores were identical.
20 Q. So they were compared, the Arthur Andersen and the
21 Barrett and Associates scores were compared?
22 A. Yes.
23 Q. And you said they were identical?
24 A. Yes.
25 Q. Did the written job knowledge test test for knowledge
375
Barrett - direct
1 that a lieutenant must know in order to be effective on the
2 job?
3 MR. FLAXMAN: Objection. There is no basis for him
4 being able to answer this question.
5 MR. HOLZHAUER: This is based on his expertise, on
6 his job analysis, as well as a test developer.
7 MR. FLAXMAN: There should be a foundation for an
8 opinion of this nature.
9 THE COURT: I think he told us all the work he did
10 to develop the test, and how it related, in his opinion, to
11 the job of a police lieutenant. I think he is entitled to
12 give his opinion on this.
13 He may not like it or he may attack it, but I don't
14 understand why this would not be admissible.
15 MR. FLAXMAN: It's our position that there hasn't
16 been a foundation under Daubert, as some people pronounce it,
17 for this kind of opinion from this witness about this subject
18 matter.
19 THE COURT: I'm not asking you to respond. You're
20 reciting magic words here. What do you mean by that? He's
21 been -- don't you think he's qualified?
22 MR. FLAXMAN: No, I don't. I think there has to be
23 some real science before he can give an opinion about this.
24 He's being asked for his opinion about something which there
25 hasn't been an indication that there is a basis for the
376
Barrett - direct
1 opinion.
2 He's an expert about making up tests, but before he
3 can give an opinion that's admissible about whether something
4 is important for a lieutenant to know, there has to be some
5 scientific validity for his being able to give that opinion.
6 Daubert doesn't allow people who have gray hair and
7 who have been in the field 40 years to pontificate about
8 things without some scientific validity for their opinion.
9 THE COURT: Are you saying that a chemist couldn't
10 give an opinion about a chemical reaction unless he's actually
11 done some sort of experiment for me; is that it?
12 MR. FLAXMAN: No, a chemical reaction is much
13 different than this kind of social science.
14 THE COURT: Social science is subject to Daubert,
15 and it's subject to the normal inquiries about a person's
16 expertise. He's got the expertise. What would you have him
17 do?
18 MR. FLAXMAN: I think there has to be more of a
19 foundation. Certainly it would not be unreasonable for your
20 Honor to hear this subject to our further exploring the
21 Daubert foundation, but I don't think it would be --
22 THE COURT: You don't want to just let it in without
23 an objection, so that you can preserve your right later on to
24 cross examine him and attack his expertise and opinion?
25 MR. FLAXMAN: That's -- yes.
377
Barrett - direct
1 THE COURT: All right. I'm going to overrule the
2 objection.
3 MR. FLAXMAN: Can I have a standing objection?
4 THE COURT: Yes, you can have a standing objection.
5 Certainly I'm going to give you all the opportunity you want
6 to cross examine him and argue that the opinion should not be
7 given weight or whatever else you want to argue, or that it
8 doesn't meet Daubert. Right now I think it meets the
9 threshold of Daubert for me to hear it.
10 MR. FLAXMAN: I understand your Honor's ruling.
11 THE COURT: Thank you.
12 BY MR. HOLZHAUER:
13 Q. Dr. Barrett, did the written job knowledge test test for
14 knowledge that a lieutenant must know in order to be effective
15 on the job?
16 A. Yes.
17 Q. Now let's look briefly at the oral -- at the in-basket
18 exercise. Could you just briefly describe what the in-basket
19 exercise entailed?
20 A. The in-basket exercise entails a test of administrative
21 behavior, and the actual substance are a number of pages which
22 contains memos, directives, notes from a lieutenant.
23 The scenario is that a lieutenant becomes ill, he
24 becomes ill in the middle of a number of investigations, and
25 this is the instructions. So the new lieutenant is assigned
378
Barrett - direct
1 the job of taking over from the lieutenant who became ill, and
2 complete a number of assignments which the Chicago police
3 lieutenant actually perform on the job: review arrest
4 reports, for example, or analysis, apply rules. We included
5 in the packet various rules and regulations which were germane
6 to the problems we presented to the candidates.
7 They had two and a half hours to review the
8 material, read the memos. There were directives which said
9 basically -- from the commanding officer saying lieutenant,
10 role they are playing, you are to do X, Y, Z, various
11 functions.
12 And at the end of two and a half hours, they were
13 given 60 multiple choice questions based upon the events they
14 were doing, working on, the problems they were working on, and
15 they were given one and a half hours to complete that.
16 Q. Now, why did you decide to use an in-basket exercise?
17 A. For several reasons. Again, the general research shows
18 that in-basket exercise is the most frequently used assessment
19 center exercise, and there are numerous studies showing that
20 it's valid.
21 Second, our job analysis in the Chicago Police
22 Department showed the importance of these administrative
23 functions, and our past experience in police departments also
24 showed the importance of these functions.
25 Q. Did the in-basket exercise assess something different
379
Barrett - direct
1 from the written job knowledge test and the oral briefing
2 exercise?
3 A. Yes. This was all self-contained. You could answer all
4 the questions by the material in the packet they were given.
5 Q. What was it designed to assess?
6 A. The administrative ability, the ability to make these
7 decisions. Could they analyze, could they apply rules, could
8 they schedule.
9 Q. Dr. Barrett, I'd like to show you a document that's been
10 marked and received into evidence as Defendant's Exhibit 15,
11 just for identification purposes.
12 Can you tell me what that is?
13 A. This is a Chicago police lieutenant in-basket
14 simulation.
15 Q. So this is the in-basket test that you prepared?
16 A. Yes.
17 Q. What was the format for the answers to this exercise?
18 A. It was a multiple choice format with a computer score.
19 There was one right answer.
20 Q. Why did you --
21 A. I'm sorry. There was one right answer.
22 Q. Why did you decide to use a multiple choice format for
23 the in-basket exercise?
24 A. Because it's objective, there is no bias, and other
25 research had shown that it was a valid technique, and you
380
Barrett - direct
1 would obtain -- you obtain reliability and no possibility of
2 bias in the scoring of the test.
3 Q. Isn't it unusual to use multiple choice questions in
4 conjunction with an in-basket exercise?
5 A. It's not particularly unusual, no.
6 Q. Do you know whether other test developers use multiple
7 choice formats in in-basket exercises?
8 MR. FLAXMAN: Objection. This doesn't go to the
9 validity. This goes to what other people -- I don't know what
10 this goes to.
11 THE COURT: It goes to custom, usage.
12 MR. FLAXMAN: I don't know why that's relevant to
13 scientific validity.
14 THE COURT: I think your own witness testified about
15 very similar matters, answered very similar questions.
16 Overruled.
17 MR. FLAXMAN: Could we have more specificity rather
18 than just the general question about why other people do these
19 things: who they are, when they did it, where they did it?
20 THE COURT: I'm sure Mr. Holzhauer is going to
21 follow up to make it meaningful.
22 MR. HOLZHAUER: Yes.
23 BY MR. HOLZHAUER:
24 Q. Do you know whether other test developers used multiple
25 choice format for in-basket?
381
Barrett - direct
1 A. Yes.
2 Q. Can you tell us about that?
3 A. One of the first studies by Lopez in 1966, or 30 years
4 ago, used a structured format. In fact it was for, I believe,
5 a police lieutenant in the New York Port Authority.
6 And so that technique has been used right long as --
7 I suppose you could call it a psychometric technique, a
8 psychometric tradition for the in-basket.
9 There are articles more recent as 1977 by Haskinson,
10 which appeared in educational -- psychological measurement,
11 for example, showing the validity of a structured approach.
12 Q. Did you hear Dr. York testify that the test was not a
13 traditional in-basket test because it was a multiple choice
14 test?
15 A. Yes.
16 Q. Do you agree that it's not traditional?
17 A. Well again, the problem is at least with the word
18 traditional. As I said, there are at least in the literature
19 three different types. One is the more subjective rating
20 approach, but even that has what's called action items where
21 you actually rate the approach of psychometric approach, the
22 one I'm using, which is much more structured, and a
23 combination of the two.
24 I would say in the private sector you would often
25 have for hiring positions than what a structured approach
382
Barrett - direct
1 might be used.
2 Q. Does the fact that this used a multiple choice question
3 format diminish the reliability of the exam?
4 A. No.
5 Q. Does it change the reliability of the exam?
6 A. It tends to make the reliability higher.
7 Q. Why is that?
8 A. Because it's objective, objectively scored, and there is
9 no rater error in terms of two or three people rating and not
10 coinciding in the ratings of the responses.
11 Q. Does the fact that it used multiple choice questions
12 diminish its validity?
13 A. No.
14 Q. Why not?
15 A. Because, again, it's an objective approach, and when you
16 have a highly reliable objective approach, your validity tends
17 to increase.
18 Q. Did you hear Dr. York testify that in what he called
19 traditional in-basket exercises, after the exercise was
20 completed the raters discussed the results with the
21 candidate?
22 A. Yes.
23 Q. Did you consider using that approach for the Chicago
24 police lieutenants exam?
25 A. Yes.
383
Barrett - direct
1 Q. Why didn't you use it?
2 A. There was really no reason to use that approach. That is
3 a developmental approach, which is often a teaching approach
4 where you do an in-basket with managers in some organization,
5 and you then critique their style of management, for example,
6 and give feedback.
7 I've never heard of it being done in the public
8 sector. There is also practical considerations of the time
9 involved. It would be enormously expensive and
10 time-consuming.
11 Q. Did the candidates have to rely on job knowledge in the
12 in-basket test that was not included in the in-basket material
13 in order to complete the exercise?
14 A. No.
15 Q. Was this, in effect, an open-book test?
16 A. Yes.
17 Q. If it didn't require any knowledge that wasn't included
18 in the materials in front of the candidate, what did the test
19 assess? What did this exercise assess?
20 A. Well, it assessed basically their ability to perform
21 administrative functions of the Chicago police lieutenant.
22 Q. Were the materials patterned after materials actually
23 encountered on the job of the Chicago Police Department?
24 MR. FLAXMAN: Could we not have leading, Judge, at
25 this stage.
384
Barrett - direct
1 THE COURT: Sustained.
2 BY MR. HOLZHAUER:
3 Q. How were the materials designed?
4 A. As I previously testified, we had in our files all of the
5 arrest reports, various forms which were used by Chicago
6 police lieutenants.
7 We usually used the exact form that they would use
8 in the police department for our reports and other material.
9 The one exception, notable exception, would be the notes which
10 were left by the lieutenant who was ill. We thought about
11 having them handwritten. They were all typed.
12 In other words, for investigations they were
13 involved in, here were the notes the lieutenant had taken in
14 the process of that investigation but had not completed, so
15 the candidates could read the material.
16 Q. Why did you do it in that form, a typed out --
17 A. It's a very standardized way of doing it so every
18 candidate has the same -- pardon me. The same opportunity to
19 gain information.
20 It would be possible to give it in an oral fashion,
21 of course, but you have to have role players, which means it's
22 not as standardized.
23 Q. If you were to learn that watch commanders and other
24 Chicago lieutenants typically don't review a hundred pages of
25 documents at one time, would that reflect negatively on the
385
Barrett - direct
1 validity of your examination?
2 A. No.
3 Q. Why not?
4 A. Because every in-basket is a simulation. It's a test. I
5 don't care what type it is, what you are really doing is
6 compressing into a short period of time days or months of
7 work.
8 As an example, I started developing content valid
9 tests in 1962 while working for Goodyear Aerospace, and one of
10 my first jobs was to develop a test for the Mark 114 control
11 system. It's a control system for atomic submarines.
12 Now, you could have simulation where you're going
13 out in the Persian Gulf and you have no events. What you do
14 is you have model events occurring, so the person is very
15 active and busy.
16 The same philosophy is used in all in-baskets where
17 you in effect compress in a short period of time a lot of
18 material for the person to handle.
19 Q. Now, if you were to learn that watch commanders and other
20 Chicago police lieutenants don't typically review notes that
21 were prepared by other officers, would that reflect negatively
22 on the validity of the exam?
23 A. No.
24 Q. Why not?
25 A. Because the notes we had were standardized communication,
386
Barrett - direct
1 gave information which was needed for the candidate to solve
2 the problem, and it might be given in the actual situation in
3 oral form, but we wanted to be very standardized.
4 Q. Now, if you were to learn that watch commanders and other
5 Chicago lieutenants don't really have in-baskets on their
6 desks, would that reflect negatively on the validity of the
7 exam?
8 A. No.
9 Q. Why not?
10 A. Because we're talking about a process of solving
11 administrative problems.
12 Q. And if you were to learn that watch commanders and other
13 Chicago lieutenants don't have to answer multiple choice
14 questions on the job, would that reflect negatively on the
15 validity of the exam?
16 A. No.
17 Q. Why not?
18 A. Because to have a correct answer, they have to go through
19 and do the analysis, or do the scheduling, or apply the rules
20 to the problem. And once you have done that, then all they
21 have to do is, in fact, find in a multiple choice question,
22 mark the right response, correct response.
23 Q. You mentioned earlier that one simple example of a
24 content valid test would be a typing test for typists; is that
25 correct?
387
Barrett - direct
1 A. Yes.
2 Q. To be content valid, the tests have to replicate the
3 exact job --
4 MR. FLAXMAN: Object to the leading, Judge. The
5 witness can testify about this stuff, if that's his opinion.
6 MR. HOLZHAUER: I don't see that as leading, your
7 Honor.
8 THE COURT: I don't either. Overruled.
9 BY MR. HOLZHAUER:
10 Q. Do you recall the question?
11 THE COURT: Start over, please.
12 BY MR. HOLZHAUER:
13 Q. To be content valid, the tests have to replicate the
14 exact job tests, like a typing test does?
15 A. No, it does not.
16 Q. Why not?
17 A. Well, a typing test is a good example. There would be
18 very few typists who all they did was sit and type for eight
19 hours a day. In fact, clerical personnel perform a variety of
20 functions. They might answer the phone, might greet visitors,
21 they might do other clerical functions. And typing might be a
22 very important part of the job, but it's not all of the job.
23 Q. Was the in-basket exercise designed to be a precise
24 simulation of something Chicago police lieutenants actually
25 do?
388
Barrett - direct
1 A. It was a psychometric test. It was a simulation, yes.
2 Q. Was it designed to be similar to something a lieutenant
3 might do one day on the job, might happen to a lieutenant one
4 day on the job?
5 MR. FLAXMAN: Objection. Again, who's testifying
6 here?
7 THE COURT: Overruled.
8 BY THE WITNESS:
9 A. No. As I said before, it's a simulation, it's
10 compressed. It would not be one day on the job. It would be
11 a compressed activity which might occur over an extended
12 period of time to make it a psychometrically reliable and
13 valid test.
14 Q. How was the in-basket exercise developed?
15 A. We again relied upon our past experience, literature
16 reviews, and also, of course, the job analysis, which gave us
17 important information concerning the administrative activities
18 of a Chicago police lieutenant.
19 Q. Who designed the materials that were provided in the
20 in-basket?
21 A. We had a team who wrote or designed the in-basket, but
22 the actual material mostly came from passing out the forms,
23 and events. They came from our job analysis.
24 Q. Did you also conduct a pilot test of the in-basket
25 simulation?
389
Barrett - direct
1 A. Yes.
2 Q. Who took that pilot test?
3 A. Again, I believe there were nine graduate students, seven
4 whites and two blacks.
5 Q. Did you learn anything from that pilot test?
6 A. Yes.
7 Q. What did you learn?
8 A. We learned that, again, we fine-tuned and modified the
9 test.
10 Q. Did the pilot test-takers critique the exercise?
11 A. Yes.
12 Q. Did that result in any modification of the exercise?
13 A. Yes.
14 Q. Now, were the materials that were used and given to
15 candidates as part of the in-basket exercise, were they
16 reviewed by anyone at the Chicago Police Department?
17 A. Yes. Again, we had the two reviews those dates in
18 October, 4th, 5th and 6th were the first review where they
19 actually took the in-basket simulation, and that would be the
20 same four individuals: Chief Cadogan, Deputy Chief Shaw,
21 Commander DeLopez, and Lieutenant Klein. And they, based on
22 that, gave us feedback about the exercise.
23 Q. You mentioned they actually took the in-basket exercise
24 at that point?
25 A. Yes.
390
Barrett - direct
1 Q. Did they review all of the materials?
2 A. Yes.
3 Q. Did they give you any -- did they recommend any changes
4 in that material?
5 A. Yes, they recommended we cut down some of the materials,
6 not have so much material in the in-basket, and increase the
7 time.
8 Q. Did you follow that recommendation?
9 A. Yes, that's the reason they came back for a second review
10 period.
11 Q. Did they recommend that any of the materials that were
12 retained be modified?
13 A. What they would do is be sure it was realistic, yes. If
14 we were not quite on point, they wanted to be sure it
15 reflected what the Chicago police lieutenant actually did.
16 Q. You said they came back. When they came back, did they
17 review the actual exercise?
18 A. Yes.
19 Q. The exercise as given?
20 A. Yes.
21 Q. Did you perform a linkage analysis of the in-basket
22 exercise?
23 A. Yes.
24 Q. Were you able to link all of the test items to major work
25 behaviors and knowledge, skills, and abilities?
391
Barrett - direct
1 A. Yes.
2 Q. Is that linkage analysis contained in your validation
3 study?
4 A. Yes, as an outline in that study.
5 Q. I don't think I asked that question as to the job
6 knowledge test. Is the job knowledge test linkage contained
7 in your validation study?
8 A. Yes.
9 Q. What was the in-basket exercise designed to assess?
10 MR. FLAXMAN: Objection. Asked and answered.
11 THE COURT: Sustained.
12 BY MR. HOLZHAUER:
13 Q. You mentioned earlier that the in-basket exercise was
14 administered on the same day as the written job knowledge
15 exam.
16 Were you present for that exercise as well?
17 A. Yes.
18 Q. Can you briefly describe how the in-basket exercise was
19 administered?
20 A. Again, it was a very standardized process. The same
21 truck which brought the written job knowledge test brought the
22 in-basket.
23 Arthur Anderson had monitors, trained monitors at
24 the site. Each candidate was given their own desk in a room,
25 there was monitors in each and every room, and we were there
392
Barrett - direct
1 as a crisis team.
2 Q. Did any crises occur?
3 A. The usual administration complaints about maybe a piece
4 of paper is missing or something.
5 Q. Were those crises dealt with?
6 A. Yes.
7 Q. How long did the candidates have to review the
8 materials?
9 A. Two and a half hours.
10 Q. And how long did they have to complete the 60 questions?
11 MR. FLAXMAN: Objection. It's been covered. 60
12 minutes for the 60 questions -- an hour and a half.
13 THE COURT: An hour and a half.
14 MR. HOLZHAUER: I don't believe it has been covered
15 by this witness.
16 THE COURT: Yes, it has.
17 MR. HOLZHAUER: On this part?
18 THE COURT: Yes.
19 BY MR. HOLZHAUER:
20 Q. How did you decide on those times?
21 MR. FLAXMAN: That's been covered also.
22 MR. HOLZHAUER: Not on this exercise.
23 THE COURT: I don't think so, but let him go ahead.
24 If he did, he did.
25 BY THE WITNESS:
393
Barrett - direct
1 A. Based on our past experience, our pilot study and the
2 recommendations of the subject matter experts.
3 THE COURT: Were you there when the tests were
4 taken?
5 THE WITNESS: Oh, yes, I was physically there. All
6 the tests.
7 BY MR. HOLZHAUER:
8 Q. You mentioned the subject matter experts. Did they make
9 any recommendations on the timing?
10 MR. FLAXMAN: That I wrote down the answer to, and
11 they said it should be longer. That was two minutes ago.
12 THE COURT: Let him say it again just so I
13 understand it.
14 BY THE WITNESS:
15 A. Yes. Our first review they said we should increase the
16 time.
17 THE COURT: From what to what?
18 THE WITNESS: I don't recall the exact time.
19 THE COURT: And the time you ended up with was their
20 recommendation?
21 THE WITNESS: Yes.
22 BY MR. HOLZHAUER:
23 Q. Were the candidates allowed to refer to the materials
24 while they were completing the questions?
25 A. Yes.
394
Barrett - direct
1 Q. How many choices were there in the multiple choices?
2 A. Five.
3 Q. And how was this in-basket exercise scored?
4 A. It was computer scored also. The same process I
5 described for the written job knowledge test was used for the
6 in-basket, a computer scoring sheet. We individually scored
7 it in a personnel -- Arthur Andersen individually scored it on
8 their separate machines. We took our computer file back to
9 Akron, they took theirs to their offices, we scored them
10 separately, and we found they were identical.
11 Q. For both the written job knowledge test and the in-basket
12 exercise was there just one correct answer among the multiple
13 choice questions?
14 A. Yes.
15 Q. Did the in-basket exercise test for knowledge that a
16 Chicago police lieutenant needed to have in order to be
17 effective on the job?
18 MR. FLAXMAN: Objection. I think he's already
19 answered this question about what it was intended to test.
20 THE COURT: Overruled.
21 BY THE WITNESS:
22 A. Yes.
23 BY MR. HOLZHAUER:
24 Q. Dr. Barrett, I'd like to look at your report for a
25 minute, and I'd particularly like you to turn to Appendix L of
395
Barrett - direct
1 Exhibit 1.
2 Can you tell me what that is?
3 A. Item sample not used, item developed for the written job
4 knowledge examination.
5 Q. Turn to page L 2, please. You said, "item sample not
6 used." Was this a question not used then?
7 A. Yes.
8 Q. Why was it not used?
9 A. Because it overlapped with another item.
10 Q. Okay. Now, starting off on page L 2, there is a question
11 marked 52, and five possible multiple choice answers. Then at
12 the bottom it says 52 B. What does that mean, 52 B?
13 A. That means that the right answer is B.
14 Q. Following that it says general order 91-3, what does that
15 mean?
16 A. That's a source of the knowledge being tapped by this
17 item.
18 Q. Following that, there's a section called rationale, and
19 at the very end there are some -- a notation NWB: 2M4. Can
20 you tell us what all that means?
21 A. That refers to the major work behaviors which will be
22 found in the master job description we covered yesterday.
23 Q. Following that there's a section marked justification.
24 What does that entail or involve?
25 A. That's where the item writer goes through each of the
396
Barrett - direct
1 five alternatives and indicates why it is correct or not
2 correct.
3 Q. Now, the next page there is a second called reviewers
4 comments. What are they?
5 A. These are the reviewers comments which are internal, and
6 you'll also see SMEs which are the external reviewers.
7 Q. I notice in the reviewers comment section there is a
8 reviewer 2, reviewer 3, reviewer 4, reviewer 5. Who are they?
9 A. They are employees of Barrett and Associates.
10 Q. Different people?
11 A. Yes.
12 Q. Then there's a notation that says "929 pilot" and some
13 comments after that. What does that mean?
14 A. That was the result of our pilot study on that date, and
15 that is the P value or percent who got it correct, and looking
16 at the various alternatives, which were or were not endorsed.
17 Q. You said P value of .70. What does that mean?
18 A. That means that 70 percent of the people got it correct.
19 Q. Now, next there is -- it says 10-4 SME 1, and following
20 that 10-4 SME 2, and 10-10 SME 1 and 2. What is that all
21 about?
22 A. That refers to the subject matter experts. Those were
23 the four individuals from the Chicago Police Department who
24 reviewed all of the items. And what we did was we actually
25 wrote down their comments.
397
Barrett - direct
1 Q. Now, following there is something 10-13 reviewer No. 2.
2 What does that reflect?
3 A. Again, someone didn't like the item, liked something
4 better. Looks like we decided to use No. 18 instead of this
5 item.
6 Q. Now, was a similar documentation done for all of the
7 other written job knowledge questions?
8 A. Yes.
9 Q. Including the written job knowledge questions that
10 actually wound up on the exam?
11 A. Yes.
12 Q. Was a similar documentation done for the other components
13 of the exam?
14 A. Yes.
15 Q. Now --
16 THE COURT: Is this the only one that wasn't used?
17 THE WITNESS: No, no. Many were not used. There
18 were probably, I would guess, 50 which were not used.
19 BY MR. HOLZHAUER:
20 Q. Dr. Barrett, after all this was done, all this
21 administration took place, the three components were scored,
22 how did you deal with the scores?
23 A. The scores were standardized separately by ourselves and
24 Arthur Anderson, and they were added together.
25 Q. Standardized, what do you mean by that?
398
Barrett - direct
1 A. That means it's a process of taking one test, the written
2 job knowledge test had 150 items, maximum score, they all had
3 15. You put it on the same scale, basically, so it all had
4 the same weight. It's a standard procedure so that they all
5 have the same mean standard deviation.
6 Q. Is that a technical term for that standard procedure?
7 A. This was a Z score we used, transformation.
8 Q. Z score. Now, what was the purpose of exchanging the raw
9 score of each component to a Z score?
10 A. So we could equally weight the exams, so it would all be
11 on the same scale.
12 Q. How did you decide to equally weight the three
13 components?
14 A. Based on a number of factors. The first is the factor
15 that -- if you review the literature on weighting, you'll find
16 that there is a strong presumption from criterion-related
17 studies that equal weighting is the best way to proceed to
18 predict job performance.
19 The second is our job analysis, which indicated that
20 equal weighting could be appropriate. We also did a more
21 formal approach looking at the linkages from our job
22 analysis. We also did a computer simulation to look at some
23 possibilities.
24 Q. Was the weighting decided before or after the exam was
25 given?
399
Barrett - direct
1 A. Before.
2 Q. Why was that?
3 A. As part of the blue ribbon panel's recommendation, and as
4 our standard procedure, we always inform the candidates ahead
5 of time how the test is going to be weighted and scored.
6 Q. Was any thought given to modifying the weighting after
7 the exam results were obtained?
8 A. No.
9 Q. Did you run any computer simulations to test the
10 different weighting schemes?
11 A. Yes.
12 Q. Can you describe those computer simulations?
13 MR. FLAXMAN: Objection. He testified no thought
14 was given to it, so what he did is not relevant.
15 MR. HOLZHAUER: He testified that he decided on the
16 weightings based on computer simulations, among other things.
17 I would like to explore what those computer simulations were.
18 THE COURT: Overruled.
19 BY THE WITNESS:
20 A. For the sergeants examination we developed a theoretical
21 approach, really abstract, to try to take into account the
22 intercorrelations between various tests hypothetically before
23 the fact, their expected validity before the fact, and the
24 potential adverse impact. And we did a computer run using
25 those factors.
400
Barrett - direct
1 BY MR. HOLZHAUER:
2 Q. Were those simulations based on the actual exam results?
3 A. No.
4 Q. Were they based on the results of pilot exams?
5 A. No, not really.
6 Q. What do you mean by not really?
7 A. Well, we had some basic data from other tests and an
8 analysis, which we summarized. But it was not based on the
9 pilot data that we had collected in developing the test.
10 Q. I'd like you to turn to Appendix Q of the report that you
11 have in front of you.
12 What does that consist of?
13 A. That's our simulation. It's titled Chicago Police
14 Examination Weighting Rationale Final Report.
15 Q. Thank you. Following the administration of the test,
16 were any items challenged by candidates?
17 A. Yes.
18 Q. Did Barrett and Associates have any role in resolving
19 those challenges?
20 A. Yes.
21 Q. Can you describe how those challenges were resolved?
22 A. As I recall, from November 20th to about December 5th was
23 what was called an appeal period, where they could phone in to
24 Arthur Anderson objections or appeals.
25 I recall there were twelve appeals. Seven had to do
401
Barrett - direct
1 with administration of the test, four with the items itself.
2 There was one appeal concerning the in-basket, and three
3 concerning the written job knowledge test.
4 Q. Were any exam items modified or eliminated as a result of
5 the challenges?
6 A. Yes.
7 Q. Can you tell us about that?
8 A. A panel reviewed all the challenges, and I believe it was
9 a Chief Cadogan, Deputy Chief Shaw, Commander DeLopez, I
10 believe Commander Schraeder, and Dr. Cellar from our Chicago
11 office.
12 They found -- recommended to me that, I believe, two
13 items on the job knowledge test everyone should get correct
14 answers for it. And for the in-basket test, one item should
15 -- everyone should get a correct answer for it, points for
16 it, and one answer should have two correct answers.
17 Q. So for the job knowledge test everybody was given credit
18 for those two challenged questions?
19 A. Yes.
20 Q. And the other two were on the in-basket test?
21 A. Yes.
22 Q. For one, what happened?
23 A. For one everyone was given credit, and for the second
24 one, they were given two correct options. If you got A and B
25 correct, for example, you would get credit for that item.
402
Barrett - direct
1 Q. A or B?
2 A. Yes.
3 Q. Dr. Barrett, in your view, did the three-part examination
4 test a reasonable sample of the knowledge, skills, and
5 abilities required of police lieutenants?
6 MR. FLAXMAN: Could we note my objection on the
7 Daubert grounds?
8 THE COURT: I thought you said you wanted a standing
9 objection, and I said you could have one.
10 MR. FLAXMAN: I'll sit down. Thank you.
11 BY THE WITNESS:
12 A. Yes.
13 BY MR. HOLZHAUER:
14 Q. Did it test all of the knowledge, skills, and abilities
15 required of a Chicago police lieutenant?
16 A. No.
17 Q. Is it necessary, in your professional opinion, for a
18 promotional exam to test all of the knowledge, skills, and
19 abilities needed for the job?
20 A. No.
21 Q. Is it possible to assess all of the knowledge, skills,
22 and abilities required for a job?
23 A. Well, it's according to how complex a job it is.
24 Q. For the job of a Chicago police lieutenant.
25 A. I don't think you could assess everything, no.
403
Barrett - direct
1 Q. Could you give me a few examples of knowledge, skills and
2 abilities that you don't think it would be possible or
3 practical to assess?
4 A. Well, as Mr. Bishop indicated, he would like an integrity
5 test. I think it would be very difficult to develop an
6 integrity test and test the candidates' integrity to be a
7 Chicago police lieutenant.
8 Q. Is it possible to test for integrity?
9 A. We do have integrity tests, but there is an issue of its
10 reliability, its validity, and the stigma attached to someone
11 being labeled as lacking integrity to be a Chicago police
12 lieutenant.
13 Q. Dr. Barrett, your professional opinion, do the three
14 components that you developed collectively test for a
15 significant portion of job domain of lieutenant?
16 A. Yes.
17 Q. In your opinion, does the fact that certain knowledge,
18 skills, and abilities were not directly tested on the exam
19 diminish the content validity of the exam?
20 A. No.
21 Q. In your opinion, does the test that you developed conform
22 with the EEOC guidelines on selection procedure?
23 A. Yes.
24 Q. Do you discuss that in your report?
25 A. Yes.
404
Barrett - direct
1 Q. Dr. Barrett, I'd like to go over that section of your
2 report with you.
3 I believe it starts on page 137 of Exhibit 1. Is
4 that correct?
5 A. Yes, page 137 talks about professional guidelines.
6 MR. HOLZHAUER: I'm looking at page 137 of the
7 report, your Honor, which is Bates 144 of Exhibit 1.
8 BY MR. HOLZHAUER:
9 Q. I'd also like you to look at Appendix P of your report.
10 Stay on 137, but turn for a minute to Appendix P and tell me
11 what that is.
12 A. That Appendix P is a point-by-point reference to the EEOC
13 guidelines.
14 Q. Now, I see on page 137 at the end of the first paragraph
15 you indicate that a summary of a point-by-point reference to
16 the standards of the EEOC guidelines is presented in Appendix
17 O. Is that a typo?
18 A. Yes, it should be P.
19 Q. Now, I see there is a series of questions beginning on
20 page 137, with your answers. What are those questions based
21 on?
22 A. They're based upon a summary of the EEOC guidelines.
23 Q. Now, the first question is: "Were explicit linkages made
24 between the test's content and major work behaviors associated
25 with police lieutenants?" How do you answer that?
405
Barrett - direct
1 A. Yes.
2 Q. Are these the linkage tables you testified about
3 earlier?
4 A. Yes.
5 Q. The second question, which is on the next page, I
6 believe: "Were explicit linkages made between the frequency
7 or importance of major work behaviors for the master job
8 description and test items in the written job knowledge
9 examination?" How do you answer that?
10 A. Yes.
11 Q. Were explicit linkages made between the master job
12 description and the in-basket simulation and oral briefing
13 exercise?
14 MR. FLAXMAN: Objection. I think we covered this
15 once.
16 THE COURT: It's in the exhibit, the exhibit is in
17 evidence, right?
18 MR. HOLZHAUER: The exhibit is in evidence.
19 THE COURT: What good does it do to have him just
20 repeat it? If you want him to clarify something in it that
21 you think might need clarification or amplification, that's
22 fine, but let's not waste time.
23 BY MR. HOLZHAUER:
24 Q. Let me summarize it at least.
25 Dr. Barrett, did you go through the inquiries that
406
Barrett - direct
1 you believe were required by the EEOC guidelines in this
2 section of your report?
3 A. Yes.
4 Q. And did you believe that your test complied or met all of
5 those inquiries?
6 A. Yes.
7 Q. Now, in your opinion, does the test that you developed
8 conform with the APA standards for educational and
9 psychological testing?
10 A. Yes.
11 Q. In your opinion, does the test you developed conform with
12 the SIOP principles that you discussed earlier?
13 A. Yes.
14 Q. Dr. Barrett, you testified quite a bit ago that you were
15 using a content validation strategy to validate the
16 lieutenants exam; is that correct?
17 A. Yes.
18 Q. Were the candidates -- how were the candidates selected
19 for promotion off the list that resulted from this exam?
20 A. A rank order.
21 Q. Is content validation adequate to support use of an
22 examination for rank order promotions?
23 MR. FLAXMAN: Objection. The question is too
24 nonspecific to be relevant to this case. If it's made more
25 specific, then I stand on my Daubert objection, but I think
407
Barrett - direct
1 right now it's just posed as an abstract question which is not
2 necessarily material to this case.
3 THE COURT: I don't understand your objection.
4 MR. FLAXMAN: I think the general question of
5 whether content validity can be enough to support rank order
6 selection may or may not be -- is not what's at issue in this
7 case. What's at issue in this case is whether the content --
8 whether in this particular instance content validity is enough
9 for promotions in rank order.
10 And I would object to the general question without
11 being specific, and then I reassert the grounds that are
12 preserved with my standing objection.
13 THE COURT: Overruled. Ask the question again.
14 BY MR. HOLZHAUER:
15 Q. Is content validation adequate to support use of an
16 examination for rank order promotions?
17 A. Yes.
18 Q. On what do you base that opinion?
19 A. Base it on a number of factors. The first is the general
20 scientific literature which shows a positive relationship
21 between test scores and job performance.
22 Second, we have our job analysis, which shows in
23 some detail that more job knowledge, greater administrative
24 skills, oral communications, the more effective you are in
25 that, the more effective you expect to be on the job itself.
408
Barrett - direct
1 We also have the guidelines, EEOC guidelines,
2 standards and the SIOP principles all support that position.
3 We also have the fact that we did test a broad
4 representative basis of knowledge, skills, and abilities. In
5 other words, we did not just test job knowledge, we tested job
6 knowledge plus administrative skills, plus the oral
7 communication component of the job.
8 We also have evidence from the fact that once we
9 gave the test, it had the desirable psychometric properties
10 which allows you to rank order. For example, it was very
11 reliable. I think our overall reliability of test battery
12 is .93.
13 Second, we had good spread between the test scores
14 from high to low. Scores were not all bunched up at the top
15 and at the bottom end. So we did differentiate among the
16 candidates.
17 Those are the main supports for rank ordering.
18 Q. Are there any scholars, other scholars who support the
19 position that content validation strategy can be used for rank
20 order promotions?
21 A. Yes.
22 Q. Who are they?
23 A. Several come to mind. Dr. Landy, for example, Dr. Frank
24 Landy; Dr. Marvin Denette; Dr. Jenerette.
25 Q. Now, you mentioned the EEO guidelines. Did the EEOC
409
Barrett - direct
1 guidelines specifically address this issue?
2 A. Yes, they do.
3 Q. And the SIOP principles, do they specifically address
4 this issue?
5 A. Yes, they do.
6 Q. Do you recall briefly how the SIOP principles address
7 this issue?
8 A. I believe what they talk about in the SIOP principles is
9 if it's a broad representative sample of the job knowledge,
10 skills, and abilities, it being reliable, and that there is
11 some differentiation.
12 Q. Dr. Barrett, in an exam given to, let's say, 700 or more
13 candidates, would it be statistically possible to demonstrate
14 that the person who finished first on the list would perform
15 better than the person who finished second, who would perform
16 better than the person who finished third, and so on down to
17 No. 700?
18 A. Oh, in theory it's possible, but in reality it would
19 never happen. You'd have to have some indication that your
20 criterion or your major job performance is perfectly reliable,
21 your test is perfectly reliable, and your correlation between
22 test scores and job performance is 1.0. But that is not
23 feasible.
24 Q. Could you briefly describe what criterion -- did you
25 consider using a criterion-related strategy for this
410
Barrett - direct
1 examination?
2 A. Yes.
3 Q. Why didn't you use that kind of strategy?
4 A. Well, there were a number of reasons. As I described
5 yesterday, there are two different types of criterion-related
6 approaches. One would involve what's called concurrent
7 validation. I would have to take -- I have to develop my
8 test, give my test to all of the current lieutenants in
9 Chicago, give them the test, score it, and then obtain some
10 measure of their job performance.
11 The problem with that approach is at least twofold.
12 One is the confidentiality of the exam would be destroyed.
13 They would -- lieutenants would read questions, job knowledge
14 questions, they would memorize them, they would go back and
15 write them down and give them to their friends who are
16 sergeants.
17 And this effect occurred in the sergeants
18 examination. A matter of a very short time after we gave the
19 actual test, it was reproduced. So you have the idea of
20 confidentiality being destroyed.
21 The second problem in the modern police departments
22 is that people are very reluctant to rate people's job
23 performance knowing it might be reviewed in court. They feel
24 they might be accused of bias, for example. So that's a very
25 difficult process. It also would be a time-consuming process
411
Barrett - direct
1 to do that.
2 The second type of validation which is criterion
3 related would be a predictive study. That would mean we would
4 have to take a sample of sergeants with the test we have, a
5 random sample, give them the test, promote them to lieutenant,
6 and a year later see how well they perform as a lieutenant.
7 Again, we have the same issues about
8 confidentiality, and it's just not feasible nor practical.
9 Q. Dr. Barrett, did you ultimately see the results of this
10 exam?
11 A. Yes.
12 Q. Did you review the adverse impact of those results?
13 A. Yes.
14 Q. What were your reactions to the adverse impact?
15 A. I was disappointed that we had adverse impact.
16 Q. Now, after the results were received, did you do any
17 statistical analysis on alternative ways of using the test
18 results?
19 MR. FLAXMAN: Objection.
20 THE COURT: Before you object, I'd like to ask you a
21 question. Was there an adverse impact with all three
22 components?
23 THE WITNESS: Yes.
24 THE COURT: Okay. Thank you.
25 Ask your question again.
412
Barrett - direct
1 BY MR. HOLZHAUER:
2 Q. After the results were received, did you do any
3 statistical analysis on alternative ways of using the test
4 results?
5 MR. FLAXMAN: My objection is that he testified
6 previously that they never considered reweighting and any
7 statistical analysis would be irrelevant to have.
8 THE COURT: This is after the fact, and I think it
9 goes to your argument of alternatives, so I'm taking it in
10 that context.
11 BY MR. HOLZHAUER:
12 Q. Did you do any statistical analysis on alternative ways
13 of using the test results?
14 A. Yes.
15 Q. What are those alternative ways that you considered?
16 A. One way was a banding.
17 Q. Did you evaluate the adverse impact of banding?
18 A. Yes.
19 Q. Could you briefly describe what people in your field mean
20 when they refer to banding?
21 A. Well --
22 MR. FLAXMAN: Let me object and ask that he describe
23 what he means as banding.
24 THE COURT: That's a good objection. Sustained.
25 MR. FLAXMAN: Thank you.
413
Barrett - direct
1 BY MR. HOLZHAUER:
2 Q. Can you describe what you mean by banding? And I'll have
3 a follow-up question.
4 A. As Mr. Joyce described it yesterday, it's a relatively
5 new technique, and I call it a controversial technique, and
6 there is no one type of banding.
7 What you're in effect doing is using certain
8 statistical procedures to say, now I believe that the scores
9 between 90 and 100 should be treated the same. And in effect
10 you're, from my point of view, introducing random error into a
11 process. But that's the basic concept.
12 Q. Is there a substantial professional literature on
13 banding?
14 A. I'm not sure substantial, but at least there are a dozen
15 articles, I would guess, or half a dozen for sure.
16 Q. Does the substantial literature discuss banding in the
17 same way that you have discussed it?
18 A. They discussed a number of different types of banding,
19 yes.
20 Q. I would like you to look at Appendix R of your report.
21 Can you tell me what that is?
22 A. It's the Chicago police lieutenant banding report.
23 Q. Now, you mentioned that you evaluated the possibility of
24 banding for this test.
25 A. Yes.
414
Barrett - direct
1 Q. Look at page R 2. What size of band did you consider?
2 A. The first band included 50 people.
3 Q. How did you calculate that band size?
4 A. We went to the standard textbooks and used a standard
5 error of measurement to get a band, and we used -- applied
6 that to a lower band.
7 Q. And looking at page R 4, table R 5, did you examine what
8 the racial breakdown of that candidate in the first band would
9 be?
10 A. Yes. The first two bands you're saying, or first band?
11 Yes. The first band --
12 Q. Start with the first band.
13 A. The first band would be 47 white and three African
14 Americans, for a total of 50.
15 Q. Now, looking at that first band, would that banding
16 approach reduce adverse impact?
17 A. No, it would still be substantial adverse impact.
18 Q. Did you also calculate a second band?
19 A. Yes.
20 Q. What statistical technique did you use for that?
21 A. The same one we mentioned before.
22 Q. How many people are in that second band?
23 A. 153.
24 Q. Did you look at the racial breakdown of candidates in
25 that band as well?
415
Barrett - direct
1 A. Yes.
2 Q. Is that also in table R 5?
3 A. Yes.
4 Q. And what was that?
5 A. It would be 153 white, six African American, six
6 Hispanic, and no Asian Americans.
7 Q. Let's look back at table R 2. Can you tell me what that
8 shows?
9 A. This is the gender and ethnic group breakdown using a top
10 down selection procedure.
11 Q. Top down, do you mean the same thing as rank order?
12 A. Yes.
13 Q. And top down how far?
14 A. We were going down to 200.
15 Q. Did you compare that rank order group with what had
16 happened using statistical bands?
17 A. Yes.
18 Q. Can you look at tables R 6 and R 7 on page R 5. What are
19 they?
20 A. I'm sorry. What page are you on?
21 Q. Page R 5, tables R 6 and R 7.
22 A. We're looking at the expected numbers of each group from
23 band 2, R 7, expected number of candidates in each sub group,
24 approximately 200 candidates.
25 Q. Do they show the calculations determining the likely
416
Barrett - direct
1 composition of the first 200 promotees using the banding
2 approach?
3 A. Yes.
4 Q. And what's the result?
5 A. Well, basically you are actually -- have more adverse
6 impact. You have one less African American being chosen,
7 looks like.
8 Q. More adverse impact than what?
9 A. Than the top down.
10 Q. So would this banding approach reduce the adverse
11 impact?
12 A. No.
13 Q. From your analysis, was the adverse impact of this test
14 attributable to the fact that candidates were selected in rank
15 order?
16 A. No.
17 Q. In your professional opinion, do the three components of
18 the 1994 lieutenants exam collectively constitute a content
19 valid promotional exam?
20 A. Yes.
21 Q. In your opinion, do they collectively constitute a
22 content valid promotional exam when used for rank order
23 promotion?
24 A. Yes.
25 Q. In your professional opinion, was the written job
417
Barrett - direct
1 knowledge component a content valid exam component?
2 A. Yes.
3 MR. FLAXMAN: It's been asked and answered.
4 THE COURT: It has.
5 MR. HOLZHAUER: Have I asked it for all of the
6 components?
7 THE COURT: I think so.
8 BY MR. HOLZHAUER:
9 Q. May we establish that each of the components was content
10 valid, Dr. Barrett?
11 A. Yes.
12 Q. Thank you. In your professional opinion, would a
13 candidate receiving a higher score on this exam be likely to
14 show better job performance than one receiving a lower score
15 on the exam?
16 A. Yes.
17 Q. I'd like to show you a document we've labeled Defendant's
18 Exhibit 3. Can you tell me what that is, Dr. Barrett, when
19 you have a chance to look at it.
20 A. This is my response to the expert witness reports and
21 depositions of Drs. York, Koziol, and Levitt, and Mr. Bishop,
22 in the matter of Brown v. Chicago, October 13, 1997.
23 Q. So it includes your response to Mr. Bishop as well as the
24 others?
25 A. Yes.
418
Barrett - direct
1 Q. I'd like you to turn to page 52, table 5. Can you tell
2 me what that is?
3 A. This is the table rank of plaintiffs on job knowledge
4 from the Chicago police lieutenant test, where we eliminate
5 challenged items.
6 Q. What is column R 11 of this chart?
7 A. This is Bishop's critique of, I believe, all his
8 critiques, which includes test wiseness, not actual procedure,
9 reference to not essential, ambiguous, watch commander, police
10 officer, assorted items. I believe it's a compilation of all
11 his critiques.
12 Q. What does R 11 have -- do with those critiques?
13 A. What we're doing is -- say all right, what would happen
14 if we take out all the items that Mr. Bishop critiqued and
15 reranked the people on the new test.
16 In other words, we're saying all right, Mr. Bishop,
17 we're going to see if -- what's going to occur if we take out
18 all of the items you say are not good, will this actually
19 change or modify the ranking on this test.
20 MR. FLAXMAN: My objection to further questions
21 about this is it's not relevant now that we know what it's
22 about.
23 THE COURT: Why?
24 MR. FLAXMAN: We don't have to prove that anybody,
25 that any plaintiff would have been promoted but for any
419
Barrett - direct
1 particular questions. We have to prove that the test had an
2 adverse impact on minorities. We proved that, that's been
3 conceded. On the question of whether leaving out three
4 questions would have had a different result is not a defense.
5 It's not material to either the defense in this case or the
6 plaintiffs' case.
7 THE COURT: Well, you say that, but why? They have
8 to prove it's content valid, and they're doing that by showing
9 that it would be the same results, I assume, or similar
10 results even if they met your criticism, which you put on in
11 your case. They're meeting your case.
12 MR. FLAXMAN: I don't think that's part of our
13 case. Our case --
14 THE COURT: Well, that's what your witness testified
15 to. They went through these questions one by one and said
16 they were not related to the job. Now they're saying all
17 right, we'll take them out and let's see if it affects the job
18 at all. It may not be conclusive, but I think it's probative.
19 MR. FLAXMAN: I don't think it's even material. I
20 think this is something we have to educate you about as we go
21 on with this, and I --
22 MR. HOLZHAUER: Your Honor, we may be able to deal
23 with this in the briefs, but I think Mr. Flaxman takes the
24 position basically if he can show one question is not job
25 related, even if 100 percent of the exam candidates got it
420
Barrett - direct
1 right, that the exam is not valid. I don't agree with that.
2 THE COURT: I'm overruling the objection.
3 BY MR. HOLZHAUER:
4 Q. Dr. Barrett, as a result of eliminating all the questions
5 Mr. Bishop challenged in column R 11, did any of the
6 plaintiffs move up to the top 200 on that list?
7 A. No.
8 Q. Let's look at the first line, just as an example. We're
9 not going to go through this list, certainly.
10 The first line lists Diane Thompson, who testified
11 earlier. How would elimination of those questions affect her
12 ranking on the job knowledge test?
13 A. On R 1, on that column, across, you'll see she's ranked
14 200.5. You go across to R 11, her rank drops to 262. So she
15 would drop in rank when you eliminate the questions challenged
16 by Mr. Bishop.
17 Q. Were some of the other plaintiffs also moved down as a
18 result of eliminating those questions?
19 A. Yes.
20 Q. Were some of them moved up?
21 THE COURT: Excuse me. Can you tell me again, what
22 is R 1?
23 THE WITNESS: That's the actual rank as the test was
24 given, all 150 items. We're not taking any out. That's how
25 it was actually scored and used for rank orders.
421
Barrett - direct
1 THE COURT: And rank 11 would be taking -- rank 12
2 would be taking all of them out that they criticized?
3 THE WITNESS: Yes. Basically we're taking all the
4 items out in R 11, that column.
5 THE COURT: R 11?
6 THE WITNESS: Yes. R 12 I believe includes
7 plaintiff Holly Robinson had additional challenges, I
8 believe. I believe from her deposition, I assume.
9 THE COURT: Okay.
10 BY MR. HOLZHAUER:
11 Q. Now, I believe you testified that other plaintiffs did
12 move down. My next question was: Did other -- some of the
13 plaintiffs move up as a result?
14 A. Yes.
15 Q. Did you look at how many improved as a result and how
16 many lost ground?
17 A. Yes.
18 Q. What did you find?
19 A. About half improved, about half lost ground. I believe
20 one stayed the same.
21 Q. But nobody went into the top 200?
22 A. No.
23 Q. Dr. Barrett, were you in the courtroom when Diane
24 Thompson testified?
25 A. Yes.
422
Barrett - direct
1 Q. Did you hear her testify that she would have been
2 promoted had the in-basket been eliminated?
3 A. Yes.
4 MR. FLAXMAN: I don't think -- I didn't hear her
5 testify to that, Judge. She testified if she had gotten 60 in
6 the in-basket she would have been promoted.
7 THE COURT: I think you're wrong. I think she said
8 that: "If I hadn't taken it, I would have been promoted." I
9 remember that. Let's assume that's the case. If I'm wrong,
10 we'll do it as a hypothetical.
11 MR. FLAXMAN: If I'm wrong, I apologize.
12 BY MR. HOLZHAUER:
13 Q. Did you do any analysis over the last two days to see
14 what would have happened had the in-basket been eliminated?
15 A. Yes.
16 Q. Did you review that analysis in preparation for your
17 testimony today?
18 A. Yes.
19 MR. FLAXMAN: I would ask that we defer this
20 questioning until that analysis has been produced to me and I
21 have a chance to look at it and think about it rather than
22 having it been sprung in direct examination.
23 THE COURT: I think that's fair. Let's wait until
24 after lunch, and I'll let you reopen your examination. I
25 think that's fair.
423
Barrett - direct
1 MR. HOLZHAUER: Okay.
2 I have no further questions, Dr. Barrett.
3 MR. FLAXMAN: Can we take a short break before we
4 start.
5 THE COURT: Why don't we. We'll take a short
6 break.
7 Why don't you give him that analysis now,
8 Mr. Holzhauer. Maybe then we could start with you're asking
9 him those questions, and then go right into cross
10 examination.
11 Let's make it around five minutes.
12 (Short recess.)
13 THE COURT: Did you get your material, Mr. Flaxman?
14 MR. FLAXMAN: Yes. I'm satisfied that he could ask
15 those questions about the new exhibit.
16 THE COURT: All right. Dr. Barrett, please take the
17 stand. Please be seated.
18 All right. Mr. Holzhauer, you want to ask a few
19 more questions?
20 MR. HOLZHAUER: Yes, I do.
21 BY MR. HOLZHAUER:
22 Q. Dr. Barrett, if the in-basket had been eliminated, would
23 Diane Thompson have been promoted among the top 108 scores?
24 A. No.
25 Q. If the in-basket had been eliminated, would any of the 44
424
Barrett - direct - cross
1 plaintiffs have been promoted among the top 108 scores?
2 A. No.
3 Q. Did you see what happened or look to see what happened to
4 Sergeant DeSalvo if the in-basket was eliminated?
5 A. Yes.
6 Q. Do you recall what happened to Sergeant DeSalvo?
7 A. He would drop dramatically in the ranking.
8 Q. Did you also, over the last several days, look at the
9 rank ordering for the entire exam battery that would result
10 from elimination of the 25 items that Mr. Bishop challenged on
11 the stand?
12 A. Yes.
13 Q. As a result of elimination of those items, would any of
14 the plaintiffs have been promoted among the top 108 scores?
15 A. No.
16 MR. HOLZHAUER: No further questions, your Honor.
17 THE COURT: Mr. Flaxman?
18 MR. FLAXMAN: Thank you, Judge.
19 CROSS EXAMINATION
20 BY MR. FLAXMAN:
21 Q. Let's stay with this computation of scores while we're
22 there.
23 Did you have occasion to recompute where --
24 THE COURT: Are you talking about page 52 of
25 Exhibit --
425
Barrett - cross
1 MR. FLAXMAN: I'm not talking about any exhibit, I'm
2 just talking about the general question of recomputation.
3 BY MR. FLAXMAN:
4 Q. Did you have occasion to recompute where Ms. Thompson
5 would be if she scored 60 on the in-basket?
6 A. No.
7 Q. Is that possible to do, Dr. Barrett?
8 A. Yes.
9 Q. How would you go about doing that?
10 A. I would enter in the computer program for the in-basket
11 the score of 60, and have an appropriate Z score
12 transformation made and add them together, and get a final
13 rank order for her.
14 Q. Would you need a computer to do that, Dr. Barrett?
15 A. I would need a computer.
16 Q. Well, are you familiar with the procedures that the
17 computer would follow to come up with that final score?
18 A. In general.
19 Q. Well, specifically, the computer would compute the
20 standard score for Ms. Thompson's written score; is that
21 right?
22 A. You mean the job knowledge test?
23 Q. Right.
24 A. Yes.
25 Q. And it would compute the standard score for the orals,
426
Barrett - cross
1 for her oral -- for her score on the oral component; is that
2 right?
3 A. Yes.
4 Q. And the computer would also compute the in-basket
5 standard score for a score of 60; is that right?
6 A. Yes.
7 Q. And is it complicated to compute the standard score,
8 Dr. Barrett?
9 A. Not too complicated.
10 Q. Well, could you tell us how you would compute the
11 standard score for the written score, the score on the job
12 knowledge test?
13 A. You would take the mean of distribution minus the score
14 of the individual and derive the standard deviation in
15 general.
16 Q. Now, in your report, Defendant's Exhibit 1, you report,
17 don't you, what the mean of the job knowledge test score was,
18 don't you?
19 A. I think we do.
20 Q. And do you recall that the mean was 109.89?
21 A. No.
22 Q. Do you think if you looked at the report it would refresh
23 your recollection?
24 A. It probably would.
25 Q. Let me show you Defendant's Exhibit 1.
427
Barrett - cross
1 Do you already have a copy in front of you, sir?
2 A. The report, you mean?
3 Q. Yes.
4 A. Yes, it's here.
5 Q. Why don't you look at the one you have and see if you can
6 tell us if in that report you reported what the mean was on
7 the written job knowledge test.
8 MR. HOLZHAUER: Your Honor, while the doctor is
9 doing that, I had given Mr. Flaxman a new edition of this big
10 spread sheet, which I think is much clearer to read, and I
11 would give you a copy as well.
12 MR. FLAXMAN: This is Defendant's Exhibit --
13 MR. HOLZHAUER: 22.
14 MR. FLAXMAN: 22, and we are in agreement it's much
15 easier to read.
16 MR. HOLZHAUER: And it's been entered by
17 stipulation; is that correct?
18 MR. FLAXMAN: That's my understanding.
19 THE COURT: We're substituting it for Exhibit 22.
20 Is that right? This is Defendant's Exhibit 22, is it not?
21 Correct?
22 MR. HOLZHAUER: Correct.
23 THE COURT: Can you direct the witness's attention
24 to --
25 BY MR. FLAXMAN:
428
Barrett - cross
1 Q. Let me ask you to look at page 135, Dr. Barrett.
2 A. I'm in Appendix I 2, which says the mean is 109.89.
3 Q. And Appendix I 2 actually has that formula that you were
4 talking about; is that right?
5 A. Yes.
6 Q. So if you want to find the standard score for Diane
7 Thompson on the job knowledge test, and we've stipulated that
8 she received 121, you would take 121, subtract 109.89 from it,
9 and divide that by 14.95; is that right?
10 A. Yes.
11 Q. And we've stipulated that your chart shows that the
12 standard score for that is .74. Does that make sense to you,
13 Dr. Barrett?
14 A. Makes sense.
15 Q. And you could do the same thing with the oral briefing,
16 you have -- the mean is a different number. What's the mean,
17 Dr. Barrett?
18 A. I'm sorry.
19 Q. What is the mean? We use that word "mean." What does
20 that mean? What does the mean mean?
21 A. It's a number you divide by some numerator.
22 Q. Is the mean the average?
23 A. Yes, the average.
24 Q. So the mean on the oral briefing exercise was 12.29, that
25 means that the average score for the 765 candidates was 12.29;
429
Barrett - cross
1 is that right?
2 A. That's correct.
3 Q. And if the standard deviation -- what's the standard
4 deviation, Dr. Barrett?
5 A. That's a spread around the mean.
6 Q. Is it --
7 A. 66 percent of the score would fall around that -- plus or
8 minus that score.
9 Q. So if the -- could you tell us again what standard
10 deviation -- how do you compute the standard deviation,
11 Dr. Barrett?
12 A. Basically you're looking at the deviations around the
13 mean. So it's plus or minus -- that indicates that 66 percent
14 of the scores will be plus or minus one standard deviation.
15 Q. You have everybody's score on the written test; is that
16 right?
17 A. On the computer, yes.
18 Q. And on the computer how do you go about finding out what
19 the standard deviation is?
20 A. Put it in the program, and it comes out.
21 Q. Well, do you say computer, give me standard deviation?
22 A. Yes.
23 Q. Do you know of your own knowledge how to compute the
24 standard deviation?
25 A. I used to be able to do it.
430
Barrett - cross
1 Q. Is that something you learned when you went to school,
2 Dr. Barrett?
3 A. Yes, it is.
4 Q. That's something that you knew when you wrote your
5 dissertation, Dr. Barrett?
6 A. Yes.
7 Q. And it is something that you knew when you became a
8 diplomat of the SIOP?
9 A. Yes.
10 Q. But it's not something you know today when you're
11 testifying in court; is that right?
12 A. I know the concept.
13 Q. Well, could you explain to us in understandable words, if
14 you can, the concept of standard deviation?
15 A. As I said before, it's a spread around the mean, and it's
16 a number which indicates that you're going to have around that
17 mean 66 percent of the scores plus or minus.
18 Q. Well, let's say the standard deviation was 1.69 and the
19 mean was 12.29. What does that mean? What's the significance
20 of having a standard deviation of 1.69?
21 A. Well, you would add 1.69 to 12.29, and you should have 33
22 percent of the scores there. You subtract 1.69, you have 33
23 percent of the scores there, so you have between those two
24 bands 66 percent of the scores.
25 Q. Do you always have 66 percent of the scores between plus
431
Barrett - cross
1 or minus one standard deviation around the mean?
2 A. That's the concept, yes.
3 Q. My question, Dr. Barrett, is do you always have in real
4 world data 66 percent of the scores within plus or minus one
5 standard deviation around the mean?
6 A. Well, it's according to the distribution of scores. You
7 could have it very high, for example, and you would not have
8 that.
9 Q. So you're telling us that 66 percent is just some
10 abstract concept that may or may not apply to this data; is
11 that right?
12 A. No, it applies to this data.
13 Q. Where does it say in your report that it applies to this
14 data? Or does it say that in your report, that 66 percent
15 applies to this data?
16 A. I don't think it says anywhere in this report.
17 Q. And do you have any documentation that is in the report
18 that shows where you measured whether or not it applies to
19 this data?
20 A. Nothing at all.
21 Q. And it's just your opinion that it applies to this data;
22 is that right?
23 A. I'm not sure -- yes, it's my opinion.
24 Q. Now, let me ask you the same questions about the written
25 job knowledge test. I 2 says the standard deviation is 14.95;
432
Barrett - cross
1 is that right?
2 A. Yes.
3 Q. Do you have any data that shows that 66 percent of the
4 scores on the written job knowledge test are within plus or
5 minus 14.95 of the mean, which you say is 109.89?
6 A. No.
7 Q. And then the in-basket, it's on page I 3, you say the
8 mean is 45.02; is that right?
9 A. Yes.
10 Q. And you say -- you report the standard deviation as being
11 7.14; is that right?
12 A. Yes.
13 Q. And again, you don't have any analysis that shows that 66
14 percent of the scores on the in-basket are within plus or
15 minus 7.14 or 45.02, do you?
16 A. No.
17 Q. Now, to compute Diane Thompson's final Z score, if she
18 had scored 60 on the in-basket, would you take -- you take her
19 Z score on the written, Z score on the oral as reported in
20 Defendant's Exhibit 22, and then compute her Z score on the
21 in-basket, assuming that it was 60, and take those three
22 numbers and divide by three. Add those three numbers and
23 divide by three; is that right?
24 A. Yes.
25 Q. And you didn't try doing that, did you?
433
Barrett - cross
1 A. No.
2 Q. Okay. Do you think you'll have an opportunity to do that
3 over the lunch break and tell us what number you got?
4 A. I could try.
5 Q. Okay. Let me talk about Holly Robinson.
6 On Exhibit 22, Holly Robinson -- it shows that Holly
7 Robinson received 117 on the written and 56 on the in-basket.
8 And would it be possible -- it would be possible, would it
9 not, to take Holly Robinson's Z score on the written, or Z
10 score on the in-basket, and add to it a Z score for different
11 oral scores, let's say 13, 14 or 15, and see what Holly
12 Robinson's final Z score would have been if she had scored 13,
13 14 or 15 on the oral; is that right?
14 A. Yes.
15 Q. And do you think you'd have a chance to do that over the
16 lunch break?
17 A. Sounds like a lot of work over the lunch break.
18 Q. Well, what does it take to compute the Z score for one
19 exam component, Doctor? Do you have to take one number,
20 subtract another number, and then divide it?
21 THE COURT: What's the Z score again, Doctor?
22 THE WITNESS: A Z score is a standard technique for
23 making the scores equivalent when the raw scores are not
24 equivalent. It's spelled out, the formula is given in
25 Appendix I, which is --
434
Barrett - cross
1 BY MR. FLAXMAN:
2 Q. Now, when was it that you were retained to work on the
3 lieutenants test for the City of Chicago, Dr. Barrett?
4 A. We started, I believe, April of 1994, discussing it with
5 the City of Chicago Law Department.
6 Q. And did you submit a written proposal to the City of
7 Chicago for the contract to prepare the test?
8 A. Yes.
9 Q. Did you submit that written proposal before you had
10 prepared the test?
11 A. I don't know the exact date. The reason I don't, because
12 it is not unusual for us to begin work before a formal
13 contract is signed.
14 Q. Well, before submitting the formal proposal for the
15 lieutenants test, had you been hired by the city to prepare
16 the sergeants test?
17 A. Yes.
18 Q. And did you submit a formal proposal to do -- to prepare
19 the sergeants test?
20 A. Yes.
21 Q. And did you enter into the contract with the city to
22 prepare the sergeants test before you started work on the
23 sergeants test?
24 A. I don't recall. We may have.
25 Q. Okay. Did you propose to prepare the same kind of test
435
Barrett - cross
1 for lieutenant as you had proposed preparing for the
2 sergeants?
3 A. It was the same types of test, yes. Not the same type of
4 content.
5 Q. By the same type of test, do you mean that you proposed
6 preparing a test that included a job knowledge component, an
7 oral component, and an in-basket exercise?
8 A. Yes.
9 Q. As a matter of fact, that's the standard test that
10 Barrett and Associates has been administering for the last ten
11 years; is that right?
12 A. That standard format that we've used in many different
13 police and fire departments, yes.
14 Q. By Barrett and Associates; is that right?
15 A. Yes.
16 Q. And you've used that technique with the Akron police
17 sergeants; is that right?
18 A. As I recall we did, yes.
19 Q. You used it with Akron fire lieutenants; is that right?
20 A. As I recall we did, yes.
21 Q. And you've used it with Washington, D.C., Metropolitan
22 Transit police sergeants; is that right?
23 A. Yes.
24 Q. You also used it with the St. Louis Fire Department; is
25 that right?
436
Barrett - cross
1 A. Yes.
2 Q. And this three-component test is the Barrett standard
3 packet, is that fair to say?
4 A. No.
5 Q. Okay. Well, is it fair to say that in this
6 three-component test the in-basket exercise, whether it's used
7 in Akron or St. Louis, is intended to measure administrative
8 skills?
9 A. Yes.
10 Q. And there's an oral briefing exercise that was used in
11 Akron, and used in St. Louis and Chicago, and that was
12 intended to measure oral communication skills; is that right?
13 A. Yes.
14 Q. And the multiple choice job knowledge test, that's used
15 in -- that was used in Akron in police and fire; is that
16 right?
17 A. Yes.
18 Q. And it was used in the Washington Metropolitan Area
19 Transit Authority?
20 A. Yes.
21 Q. And it was used in St. Louis?
22 A. Yes.
23 Q. And it was used in Chicago?
24 A. Yes.
25 Q. And that's intended to measure job knowledge; is that
437
Barrett - cross
1 right?
2 A. Yes.
3 Q. And the view, your views that measuring those -- using
4 those three tests provides a good indication of who should be
5 promoted; is that right?
6 A. Yes.
7 Q. And that's true for the City of Chicago and its
8 lieutenants; is that right?
9 A. Yes.
10 Q. And it's true for Akron police sergeants; is that right?
11 A. Yes.
12 Q. Is the job of being a police sergeant in Akron, Ohio,
13 different than being a lieutenant in Chicago?
14 A. I'm not sure what you mean by "different." There are
15 certainly different contents in terms of the job knowledge, if
16 that's your question.
17 Q. Well, did you do a job analysis of the job of police
18 sergeant in Akron?
19 A. Yes.
20 Q. And from your job analysis, did you conclude that the job
21 of policing as a sergeant in Akron is the same as the job of
22 policing of a lieutenant in Chicago?
23 A. No.
24 Q. They're different jobs, aren't they?
25 A. Yes.
438
Barrett - cross
1 Q. And being a police -- being a fire lieutenant in Akron is
2 different than being a police lieutenant in Chicago; is that
3 right?
4 A. Yes.
5 Q. And being a fire fighter in St. Louis is different than
6 being a police lieutenant in Chicago; is that right?
7 A. Yes.
8 Q. And all of those jobs you assess with this three
9 component test, the job knowledge test, the in-basket exercise
10 and oral briefing exercise; is that right?
11 A. Those forms of tests were used in all those locations,
12 yes.
13 Q. Now, when you prepared the job knowledge test for the
14 police lieutenants in the City of Chicago, you had this source
15 list, this list of documents; is that right?
16 A. Yes.
17 Q. And were you the person who picked out those general
18 orders, and special orders, and statutes, and rules?
19 A. You mean personally picked them out?
20 Q. Yeah.
21 A. No.
22 Q. Who did?
23 A. Well, we had a team approach, and we had, as I recall,
24 around May, interviews with 11 different police lieutenants
25 who helped us generate potential relevant source documents.
439
Barrett - cross
1 We had our job analysis team, who also would often
2 have suggestions about relevant material. We had, of course,
3 the expert panel from the police department, which I recall
4 included Commander Wedgbury, Captain Montague, Lieutenant
5 Schraeder, I believe -- I think it's Commander Schraeder. We
6 also had input from Lieutenant Klein.
7 So we had multiple sources of what was relevant.
8 Q. Well, these 11 lieutenants that were interviewed, did you
9 interview any of those 11 lieutenants, Dr. Barrett?
10 A. I don't think I did.
11 Q. Do you have personal knowledge of what was said to any of
12 those 11 lieutenants?
13 A. Personal knowledge?
14 Q. Yes.
15 A. In the sense of -- no, I don't have personal knowledge.
16 Q. Do you know which, if any, of those 11 lieutenants were
17 asked to identify the important general orders of the Chicago
18 Police Department?
19 A. No, I don't.
20 Q. Do you know which if any of those 11 lieutenants were
21 asked to identify the important special orders of the police
22 department City of Chicago?
23 A. No, I don't.
24 Q. Now, you talked about the expert panel. Did you ever
25 meet with the expert panel?
440
Barrett - cross
1 A. No. Dr. Cellar met with the expert panel.
2 Q. Who is Dr. Cellar?
3 A. He's a vice president in Barrett and Associates. He has
4 his office here in Chicago.
5 Q. Is he one of your graduate students that went on to other
6 things?
7 A. Yes.
8 Q. Aside from his employment with Barrett and Associates,
9 does he have a day job?
10 A. Yes.
11 Q. What's that?
12 A. He's in DePaul University.
13 Q. Is he in the City of Chicago at this time?
14 A. I'm sorry?
15 Q. Is he in the City of Chicago at this time? Is he out of
16 the country, is he here?
17 A. I don't know if he's actually in the city. He might be
18 in his home, which is not in Chicago.
19 Q. Excuse me?
20 A. He might be at his home. I don't know where he is right
21 now.
22 Q. Is his home in the Chicago area?
23 A. Yes.
24 Q. Okay.
25 THE COURT: What is he doing at DePaul?
441
Barrett - cross
1 THE WITNESS: He's an associate professor.
2 BY MR. FLAXMAN:
3 Q. And he met with Wedgbury, Minogue, and who else did he
4 meet with?
5 A. I'm not sure. For some reason I think it was Commander
6 Schaffer or Schraeder. I've forgotten now.
7 Q. Anybody else?
8 A. Are you talking about now the review panel?
9 Q. Right.
10 A. Beyond the people I've already mentioned, is that what
11 you're asking?
12 Q. Well, I don't know. I don't think you've already
13 mentioned anybody else. I want to know who Dr. Cellar met
14 with.
15 A. On the review panel?
16 Q. Yes.
17 A. I thought I said Commander Wedgbury, Captain Montague, I
18 believe Shaffer or Schraeder --
19 Q. Anybody else?
20 A. Well, Dr. Cellar did a lot of interviews, as I recall, so
21 I don't know specifically who he met with.
22 Q. Well, do you have personal knowledge of whether anybody
23 that Dr. Cellar met with told him which orders were important?
24 A. I'm sorry.
25 Q. Do you have personal knowledge of whether anybody that
442
Barrett - cross
1 Dr. Cellar met with told him which general orders were
2 important for a lieutenant to know?
3 A. All I have is his report to me and the recommendations.
4 Q. Is his report to you and recommendation contained in
5 Defendant's Exhibit 1 that's been admitted into evidence?
6 A. I think the final recommendation would be whatever was
7 posted by the police department and sent to the candidates.
8 That would be the official list.
9 Q. Is that the form that you received your report from
10 Dr. Cellar?
11 A. I don't recall the exact form I received it in.
12 Q. Did you ever receive a report from Dr. Cellar?
13 A. Yes, I don't recall when. I recall receiving, yes,
14 here's is a recommendation in terms of the panel. And I
15 recall Lieutenant Klein made an addition to it.
16 Q. Well, let's go back to this report from Dr. Cellar. That
17 report is not part of Defendant's Exhibit 1; is that right?
18 A. It is in the sense I don't -- I assume, I don't know
19 what's in your exhibit, but as I recall, there is an official
20 posting. I don't know what exhibit number that is.
21 Q. Well, Doctor, we're trying to find out where that
22 official posting came from. Who prepared that official
23 posting?
24 A. I'm not sure what you mean.
25 Q. Who typed it? Where was it typed up?
443
Barrett - cross
1 A. I don't know where it was typed up.
2 Q. Do you know whether it was typed in Ohio or Chicago?
3 A. I don't know.
4 Q. Do you know whether it was prepared by Mr. Joyce or
5 somebody in his office?
6 A. Mr. Joyce testified yesterday he did not prepare it.
7 Q. That was not my question, Dr. Barrett.
8 Do you know whether it was prepared by Mr. Joyce or
9 somebody in his office?
10 MR. HOLZHAUER: Objection. He already testified he
11 doesn't know. How many times can he ask the same question a
12 different way?
13 THE COURT: I think you made your point.
14 BY MR. FLAXMAN:
15 Q. Did you talk to Lieutenant Klein about the important
16 general orders?
17 A. Yes, we discussed the important issues.
18 Q. When did you have this discussion with Lieutenant Klein
19 about which general orders were important?
20 A. I talked to Lieutenant Klein many times. I don't recall
21 any one specific date.
22 Q. Do you recall any particular conversation with Lieutenant
23 Klein about which general orders are important?
24 A. No.
25 Q. You told us that Lieutenant Klein made an addition to the
444
Barrett - cross
1 list. Do you recall that testimony?
2 A. Yes, I do.
3 Q. What did he add to the list? What addition did he
4 propose to the list?
5 A. I believe, now, this is -- I believe he suggested the
6 community policing bulletin, as I recall. Again, it's been
7 three years, so I could be wrong about that.
8 Q. Well, is Mr. Klein's recommendation about something to
9 add documented anywhere in Defendant's Exhibit 1?
10 A. Are you referring now -- is this my report you're
11 referring to?
12 Q. That's correct.
13 A. I doubt it.
14 Q. Well, when you agreed to prepare this test for the City
15 of Chicago, did you agree to create a paper trail of the work
16 that you were doing?
17 A. Yes.
18 Q. Could you tell us where the paper trail is for
19 Mr. Klein's recommendation about including community policing?
20 A. There is no -- as far as I know, there is no paper trail
21 on that.
22 Q. Did Mr. Klein make any other recommendations that things
23 should be added?
24 A. Again, I don't recall specifically what he -- beyond what
25 I just said, he might have added.
445
Barrett - cross
1 Q. In the course of your work in preparing this examination
2 and the sergeants examination, did you become aware of
3 something called the rules and regulations of the Chicago
4 Police Department?
5 A. Yes.
6 Q. And did you become aware that there are rules and
7 regulations that are promulgated by the Chicago police board?
8 A. I believe that's correct.
9 Q. Did you become aware that some of those rules and
10 regulations are vitally important for each police officer from
11 the rank of patrolman to the rank of superintendent?
12 A. I don't recall that.
13 Q. Well, did you conclude that those rules were not
14 important for Chicago police lieutenants to know?
15 A. Again, this is three years ago, but I think that there
16 was some belief that they were outdated. This is just my
17 recollection.
18 Q. Whose belief was that, Dr. Barrett?
19 A. Well, again, we relied upon the subject matter experts.
20 In other words, they are the ones who recommend to us these
21 sources.
22 Q. Did Mr. Klein tell you that the rules and regulations of
23 the Chicago Police Department were outdated?
24 A. I don't recall.
25 Q. Did Captain Minogue tell that you the rules and
446
Barrett - cross
1 regulations of the Chicago Police Department were outdated?
2 A. I don't recall.
3 Q. Did Commander Wedgbury tell you that the rules and
4 regulations of the Chicago Police Department were outdated?
5 A. I don't recall.
6 Q. Did Commander Schraeder tell you that the rules and
7 regulations of the Chicago Police Department were outdated?
8 A. I don't recall.
9 Q. And is it your opinion, based on whenever it's based on,
10 that those rules and regulations are outdated?
11 A. That was a decision that we reached, I'm sure if that's
12 what decision we did reach, at that point in time, yes.
13 Q. And the rules and regulations were not tested in the
14 lieutenants test; is that right?
15 A. I think that's correct.
16 Q. There wasn't any question about that on the written job
17 knowledge?
18 A. As I recall, that's correct.
19 Q. Now, in your report you include the job announcement from
20 the 1987 lieutenants test. Do you recall that inclusion?
21 A. Yes.
22 Q. That's at page Bates stamped 182, Defendant's Exhibit 1.
23 Why did you include that report? Excuse me. Why did you
24 include that job announcement in your report, Dr. Barrett?
25 A. I'm sorry. What page are you on?
447
Barrett - cross
1 Q. 182.
2 A. This was included because the EEOC regulations asked you
3 try to look at the previous exam process, and I believe there
4 was very little information obtained about that previous
5 process.
6 So the reason for including that is that's all we
7 could actually obtain at that point in time, because I don't
8 recall us being able to obtain a technical report, I believe
9 we asked for one, for that previous examination.
10 Q. Defendant's Exhibit 1 is dated on page 1 of that report,
11 October 10, 1995. Do you see that?
12 A. I'm sorry. What page are you on now?
13 Q. Page 1. The very first page. Page -- the cover of
14 Defendant's Exhibit 1 is what I'm referring to right now. See
15 where it says report on the development that administration --
16 do you see page 1?
17 A. Oh, you mean of the report itself?
18 Q. Yes.
19 A. I thought you meant of the -- the cover page you're
20 talking about?
21 Q. That's right. There's a date on the cover page; is that
22 right?
23 A. Yes.
24 Q. What's the date?
25 A. October 10, 1995.
448
Barrett - cross
1 Q. Was that after the exam had been administered?
2 A. The exam was administered on November 19th, 1994, so it
3 would be after the exam was administered.
4 Q. So it took 11 months to complete this report after the
5 exam was administered?
6 A. That's what the date says.
7 Q. Now, this lawsuit, do you know whether or not any
8 lawsuits had been filed about your Chicago promotional test
9 before October 10, 1995?
10 A. No, I don't know the dates when you actually filed the
11 suit.
12 Q. Well, did you ever become aware that lawsuits were filed
13 in Chicago about the promotional test?
14 A. Yes, I was -- became aware of that.
15 Q. Were any of those lawsuits filed before October 10,
16 1995?
17 A. I don't recall the exact date you filed the lawsuits.
18 Q. Did you learn about those lawsuits on or about the date
19 they were filed?
20 A. I assume that the City of Chicago law department called
21 me.
22 Q. Now, you told us on direct examination that you have been
23 paid -- maybe I misremembered or misheard it -- $130,000 for
24 your work in this litigation.
25 A. I think, again, I believe there is an invoice for that
449
Barrett - cross
1 amount outstanding, I think is what I said.
2 Q. Well, is that the total amount for which you billed the
3 City of Chicago for your post -- for your services in
4 connection with litigation concerning the test?
5 A. I don't believe so. I'm just -- other invoices which
6 have been paid.
7 Q. What's the total amount that you've billed the City of
8 Chicago for work that you've done after the tests were
9 administered, litigation work?
10 A. I don't recall the exact figure.
11 Q. Well, is it more than 130,000?
12 A. Yes.
13 Q. Is it more than 200,000?
14 A. I actually don't recall the exact figure.
15 Q. Is there any way that you could find out?
16 A. I thought we sent you the invoices.
17 Q. Well, my recollection is that those invoices show that
18 you are -- I'm not sure, but that there was like $80,000 over
19 the 130 that you've been paid. I don't want to testify, so
20 we'll move on.
21 That time has been billed at your rate of $250 an
22 hour?
23 A. Yes.
24 Q. So you worked nearly a thousand hours in connection with
25 the litigation that's been filed about your test?
450
Barrett - cross
1 A. No. That's not accurate.
2 Q. Well, how many hours have you worked?
3 A. I don't know.
4 Q. Well, if we take the total amount that you billed and
5 been paid and divide it by 250, would we determine that amount
6 of time?
7 A. No, it would not be accurate.
8 Q. Why is that?
9 A. Because we also billed for other people in the firm, at
10 which we always bill at the lowest possible rate.
11 Q. How much was Barrett and Associates paid to develop the
12 test?
13 A. I believe it was 350,000.
14 Q. It hasn't cost quite that much yet to defend the test,
15 has it, Dr. Barrett?
16 A. I don't know what cost it took to defend the test.
17 Q. You told us that it would be very expensive to administer
18 an assessment center where there was -- which was graded in a
19 different way than the in-basket exercise was in the test; is
20 that right?
21 A. Yes.
22 Q. Would it cost $150,000?
23 A. I'm not clear. You want me to estimate -- I made an
24 estimate, as I recall, in my report for Mr. York replying to
25 -- Dr. York, where I made an estimate of what it would cost.
451
Barrett - cross
1 Q. Let's go back to the 1987 job posting. Let's look at
2 page 183.
3 Did you look at that at any time before you
4 completed your work in the job knowledge test?
5 THE COURT: Page 183 of what?
6 MR. FLAXMAN: Of Defendant's Exhibit 1. I'm sorry.
7 MR. HOLZHAUER: What page?
8 MR. FLAXMAN: 183.
9 BY MR. FLAXMAN:
10 Q. Do you have that?
11 A. Yes.
12 Q. Did you look at that before the job -- before the reading
13 list was posted of the 1994 test?
14 A. I don't recall.
15 Q. Did you look at it after the reading list was posted of
16 the 1994 test?
17 A. I looked at it in some point in time. I don't recall the
18 exact time I saw it.
19 Q. Did you ever notice that some general orders were on the
20 1987 list that weren't on the 1994 list?
21 A. I never made a point-to-point comparison.
22 Q. Now, if you keep going in that exhibit to page 234, we
23 get to the reading list for the 1994 test; is that right?
24 A. Yes, it's labeled that way, yes.
25 Q. And then somewhere there is a linkage chart for the
452
Barrett - cross
1 written job knowledge test in this exhibit; is that right
2 also?
3 A. Yes.
4 Q. And the linkage chart tells you each question and what
5 general order it's based on.
6 Let me ask you to look at page 283. Is that the
7 linkage chart?
8 A. Yes.
9 Q. If you look at the linkage chart, question 54, that says
10 it's based on special order 7816; is that right?
11 A. Yes.
12 Q. And question 70 is also based on special order 7816; is
13 that right?
14 A. I'm sorry. I didn't hear your number.
15 Q. Question 70.
16 A. 70. Yes.
17 Q. Now, was there anything that came up in your job analysis
18 that demonstrated that there should be two questions on
19 general order 7816 on the job knowledge test?
20 A. I assume something on the job analysis may have come up
21 that way, or our review of the material may have found that it
22 was appropriate.
23 Q. Is there anything in your report that explains why there
24 are two questions on that general order?
25 A. No.
453
Barrett - cross
1 Q. As you sit here now, could you tell us why there are two
2 questions on that general order?
3 A. No.
4 Q. Now, the reading list, there is also a general order 86-2
5 called city license premises. Do you see that? It's on page
6 234 of Exhibit 1.
7 A. 234, and the question was --
8 Q. Do you see where the reading list says one of the general
9 orders is 86-2, city license premises?
10 A. 86-2. Yes, I see that.
11 Q. Are there any questions on the written test that were
12 linked to general order 86-2?
13 A. I don't know.
14 Q. Could you take a look and see if you find any?
15 A. Looks like 32.
16 Q. Okay. Anything else?
17 THE COURT: If it's there, can you point it out,
18 Mr. Flaxman?
19 MR. FLAXMAN: I'm proving a negative, Judge.
20 THE COURT: Well, it doesn't seem that it's a
21 negative.
22 THE WITNESS: I only found in my quick review one
23 item.
24 BY MR. FLAXMAN:
25 Q. How was this linkage chart prepared, Dr. Barrett?
454
Barrett - cross
1 A. One of my associates probably took from our review sheets
2 the item number and the source and linked that to the work
3 behavior.
4 Q. So are you telling us that if we look at general order
5 86-2, we'll find the answer to question 32; is that right?
6 A. Unless there's a mistake in the documentation.
7 Q. Are there any mistakes in your documentation,
8 Dr. Barrett?
9 A. Yes. Based upon the testimony of one of your sergeants,
10 I believe, we did find a mistake.
11 Q. Well, didn't you check all of your work before you gave
12 it into the City of Chicago?
13 A. Yes, we tried to check all our work.
14 Q. How many mistakes did you make, Dr. Barrett?
15 A. Well, it can only be one if I don't --
16 Q. How many lieutenants have been promoted because of a
17 mistake, Dr. Barrett?
18 A. I don't think any have been.
19 Q. You don't think 108 have been because of a mistake?
20 A. No.
21 Q. So this linkage chart is a mistake where it says question
22 32 is based on general orders 86- --
23 MR. HOLZHAUER: Objection. He didn't testify to
24 that. The jury is not impressed, Mr. Flaxman.
25 THE COURT: Sustained. Let's go to the facts.
455
Barrett - cross
1 You'll have a chance to argue. Let's go to the facts.
2 Is it or isn't it?
3 BY MR. FLAXMAN:
4 Q. Let me ask you to look at question -- the linkage chart
5 for item No. 5.
6 What general order does the linkage chart say that
7 item No. 5 is based on?
8 A. 93.3, addendum 5.
9 Q. And general order 93.3 is in fact on the reading list; is
10 that right?
11 A. Yes.
12 Q. So are you telling us then if we go look at general order
13 93-3 we could find the answer to question 5; is that right?
14 A. Unless there's a mistake. Yes.
15 Q. Well, there were only -- how many questions were there
16 based on general orders on this test, Dr. Barrett? 53?
17 A. Looks like there's 53.
18 Q. 53?
19 A. 53, yes, that's correct.
20 Q. And linking the general orders to the work behaviors was
21 an important part of the content validation process?
22 A. It's just one more piece of evidence to show -- you
23 really don't have to do it, but yes, it's a piece of evidence.
24 Q. How many mistakes were made in preparing that one piece
25 of evidence, Dr. Barrett?
456
Barrett - cross
1 A. I don't know.
2 Q. Well, was general order 84.7 on the reading list?
3 A. No.
4 Q. If I told --
5 A. Not on the list -- what we have here, this is not the
6 official list, by the way.
7 Q. The one we stipulated to is not the official list?
8 A. This was not the list which was sent out to the people.
9 It doesn't look like the list.
10 Q. Were different orders on the list that was sent out to
11 the people?
12 A. I don't know.
13 Q. Do you know who has the list that was sent out to the
14 people?
15 A. No.
16 THE COURT: What exhibit are you referring to, so
17 the record is clear on this?
18 MR. FLAXMAN: Exhibit 1.
19 THE COURT: Where is the list on Exhibit 1? The
20 witness is looking at a document. What document is he looking
21 at?
22 BY MR. FLAXMAN:
23 Q. Are you looking at page 234 of Exhibit 1, which is
24 entitled Chicago Police Lieutenant Promotional Exam Reading
25 List?
457
Barrett - cross
1 A. Yes. This is an appendix in our report, yes.
2 Q. And you're telling us that this is not the reading list
3 that was sent out to police lieutenants?
4 A. Well, I don't believe this -- it looks like something was
5 typed in our office, is my first thought for an appendix. It
6 does not look like what would be sent out. Usually in my
7 experience there is more on an official reading list sent out
8 by a city, but --
9 Q. Did you ever see what the city sent out to lieutenants,
10 to sergeants who wanted to take the lieutenants test?
11 A. Yes, I did.
12 Q. And was it different than this?
13 A. I didn't do a point-by-point correspondence check.
14 Q. Did you include a copy of what the city sent out in your
15 report, Defendant's Exhibit 1?
16 A. No, what we have is what I have here in this appendix F.
17 Q. Were there any differences between what the city sent out
18 and the reading list that's in your report?
19 A. As far as I know there's not, but there might be.
20 Q. Well, the reading list that's in your report does not
21 show general order 84-7 as being one of the things that
22 sergeants were told that the test would be based on; is that
23 right?
24 A. That's -- there is no 84.7 in our list, in the appendix,
25 that's correct.
458
Barrett - cross
1 Q. And it was your intent in preparing the written test that
2 each question on that written test would be based on materials
3 identified in the reading list; is that right?
4 A. Yes, the official reading list that went out, yes, that's
5 correct.
6 Q. So would it be another mistake if there was a test
7 question that was based on a general order that was not on the
8 reading list?
9 A. If it's not the