421
1 IN THE UNITED STATES DISTRICT COURT
NORTHERN DISTRICT OF ILLINOIS
2 EASTERN DIVISION
3
ADAMS, et al., ) Docket No. 94 C 5727
4 )
Plaintiffs, )
5 )
vs. )
6 )
CITY OF CHICAGO, ) Chicago, Illinois
7 ) March 7, 1996
Defendant. ) 9:00 o'clock a.m.
8
9 VOLUME III
TRANSCRIPT OF PROCEEDINGS - Hearing
10 BEFORE THE HONORABLE JOHN A. NORDBERG
11
APPEARANCES:
12
13 For the Plaintiff: LAW OFFICES OF KENNETH N. FLAXMAN, P.C.
BY: MR. KENNETH N. FLAXMAN
14 122 South Michigan Avenue, Suite 1850
Chicago, Illinois 60603
15
16 For the Defendant: CITY OF CHICAGO
BY: MS. DARKA PAPUSHKEWYCH
17 MS. SHONA B. GLINK
MR. JAY KERTEZ
18 30 North LaSalle Street, Room 1020
Chicago, Illinois 60602
19
20 Court Reporter: MR. JOSEPH A. RICKHOFF
219 S. Dearborn Street, Room 1738
21 Chicago, Illinois 60604
22 * * * * * * * * * * * * * * * *
PROCEEDINGS RECORDED BY
23 MECHANICAL STENOGRAPHY
TRANSCRIPT PRODUCED BY COMPUTER
24
25
422
1 THE CLERK: 94 C 5727, Adams vs. City of Chicago. For
2 hearing.
3 MR. FLAXMAN: Kenneth Flaxman for the plaintiffs.
4 MS. PAPUSHKEWYCH: Darka Papushkewych.
5 MS. GLINK: Shona Glink, good morning, your Honor.
6 MR. KERTEZ: Jay Kertez.
7 THE COURT: Right.
8 Good morning all. I know we have had difficult
9 weather. It just creates a real problem on how we are going to
10 handle the scheduling, but we are not going to spend so much
11 time worrying about the scheduling that we do not give the
12 witness a chance to testify. So, we will cover as much ground
13 as we can, and you have to tell us or the witness has to tell
14 us when we cut off, so we can be sure that he gets whatever
15 flight he needs to get because I assume the airports are
16 functioning in normal fashion.
17 MS. PAPUSHKEWYCH: Well, your Honor, the only thing I
18 want to bring up is that this prejudices the city's case. We
19 all know that we are under certain time restraints. I am not
20 sure we are going to get through our direct testimony with the
21 witness today.
22 THE COURT: I think that could very well be, and I am
23 confident that you are not going to get through
24 cross-examination with the expert.
25 MR. FLAXMAN: Well, I would --
423
1 THE COURT: Well, let me indicate the other rulings
2 now.
3 With respect to the testimony by the individual
4 sergeants who were testifying as to -- were permitted to
5 testify as to the scope of their duties and the duties of the
6 sergeants generally in the Chicago Police Department as a job
7 description analysis testimony, in reviewing what has been
8 testified to during the course of the trial and in getting a
9 better perspective on it, it seems to me that I am in error in
10 barring the plaintiff from having the witnesses testify as to
11 the specific questions that they claim are not related, in
12 effect, to the job duties of sergeant of the Chicago Police
13 Department and/or that the answers are in error or that they do
14 not -- they are ambiguous to the extent that a police officer
15 would not understand what the question was, so that they could
16 intelligently answer the question.
17 It seems to me that while I adhere to the decision
18 that these -- the fact that they are sergeants and have
19 experience does not qualify them as an expert so far as either
20 fashioning questions or determining the type of questions to be
21 asked and the like. As I ruled, that requires an expertise
22 that they do not have.
23 On the other hand, it seems to me that plaintiffs have
24 the right to test the content validity of the questions. And
25 in order to test the content validity of the questions,
424
1 witnesses should be able to testify to the fact that a
2 particular question may not relate to the operation of the
3 Chicago Police Department and the like.
4 I am not going to take the time to try to be more
5 detailed about that, but I think that, in retrospect now in
6 thinking about it last night, that I am in error in limiting
7 plaintiffs' witnesses from testifying specifically as to
8 individual questions and whether they relate to the actual job
9 requirements of sergeant of the Chicago Police Department.
10 Similarly, of course -- and I had already indicated
11 this could be done, but it clearly indicates that
12 cross-examination of Dr. Barrett as to individual questions is
13 certainly appropriate as to whether it is related to, again,
14 the accurate job description. Because plaintiffs have the
15 right to challenge the job description that has been prepared
16 and has been relied on in the proceeding.
17 All of this, of course, is subject to the deferral. I
18 do not want to indicate that I have lost track of the motion
19 that has been made by the city as to whether there is -- the
20 necessary threshold requirements can be met for purposes of
21 preliminary injunction. But it seems to me that it's going to
22 be in everybody's interest because all of the evidence that is
23 developed here will be applicable to any permanent injunction
24 hearing. So, we are not going to have to do this twice.
25 And, so, I think we ought to completely complete the
425
1 presentation of this evidence to have as complete a record as
2 possible, both for possible appeal on the preliminary
3 injunction, as well as for ultimate resolution of the case.
4 More specifically now, how to deal with the handling
5 of the questions. There is considerable testimony now
6 indicating that, I believe, both sides take the position that
7 the test in a sense is already compromised.
8 MS. PAPUSHKEWYCH: That's not our position, your
9 Honor.
10 THE COURT: Well, we have a record of everything
11 that's been stated.
12 MS. PAPUSHKEWYCH: Yes.
13 THE COURT: It is clear that you are not going to ever
14 use this specific test, again. That has been announced. There
15 is been considerable testimony about the risks and dangers of
16 compromise and the fact that you already know of examples of
17 what has been done with respect to the test.
18 So, it seems to me that dealing with certain specific
19 questions that are being challenged is not the same thing as
20 fully revealing the test because that is certainly going to
21 make it very difficult for anybody in the future to prepare
22 another test that would not have the problem of compromise from
23 the outset.
24 On the other hand, I think it creates a serious
25 problem for both plaintiff and defendant -- oh, I am going to
426
1 jump from this for one second because I also want to
2 acknowledge the fact that I have prevented the defendant from
3 any testimony by the various Chicago police officers,
4 lieutenants and the like, who were utilized by Dr. Barrett in
5 forming the job description and in reviewing the questions to
6 see whether they were, in fact, related to the activities of
7 sergeants in the police department.
8 So, in my opinion, I have committed error both with
9 respect to the plaintiff and the defendant on that same subject
10 matter. Obviously, both sides should have a right to present
11 detailed testimony with respect to that.
12 Getting back as to how to do it, it seems to me that
13 because of the circumstances that we have and the difficulty,
14 while, in theory, the idea of using the question number and so
15 on and so forth conceivably could be done and if we were
16 dealing with atomic secrets, perhaps something like that
17 certainly would -- and beyond would be required.
18 But it seems here that it is going to make it so
19 difficult for both sides, as well as the witnesses, to really
20 effectively testify, that with respect to the questions that
21 are being challenged, it seems to me everyone has to have the
22 ability to deal with at least the substance of the language of
23 the questions. I guess we can still prohibit the exact reading
24 of it, so that that is not done.
25 Also, it seems to me that because of the importance of
427
1 not handcuffing future panels and future experts in preparing
2 future exam questions and not to lose the substantial funds
3 that have been expended in preparing what is available now to
4 the Chicago Police Department, it seems to me that it should be
5 reasonable for the Court to require that anyone who is going to
6 be present in the courtroom during the time of the questioning,
7 one: Sign the protective agreement, so that they are subject
8 to penalties, contempt of court. And I can assure anybody that
9 violates that, it would be my view it is a very serious
10 violation and would be in contempt of court. So, each person
11 in the courtroom is going to be bound by the protective order.
12 And, two, nobody takes any notes. There will be no
13 paper and pencil or pen and pencil to be utilized throughout
14 the testimony with respect to the exact questions, so that
15 there will be no written record made.
16 The Court will order that that portion of the
17 testimony be placed under seal. So, we will make it as
18 difficult as possible for anybody to try to reconstruct them.
19 But we still are going to permit the normal usage of
20 questioning and cross-examining, except it seems to me that,
21 unless a specific request is made, we may have to then take a
22 sidebar for that, that we not read the exact language of it;
23 that rather a summary statement or substance of it or the
24 portion of it that you are relying or questioning about only be
25 referred to, so that that precaution be taken, as well.
428
1 Now, the fact that I am going to be permitting
2 additional testimony to come in -- and I do not think in a way
3 the offer of proof is really adequate for -- and, certainly,
4 the defense has not had any opportunity to do this, as well. I
5 am well aware that it is going to extend this hearing to some
6 extent, but it seems to me that the importance of having
7 something to be complete, having both sides have a fair
8 opportunity to fully present their respective cases, should
9 overshadow a few days.
10 It seems to me that that is not going to create any, I
11 will use the expression, "unfair hardship," on the Chicago
12 Police Department with the understanding that we will complete
13 this hearing and the Court will complete its decision just as
14 rapidly as possible.
15 Now, I was not aware that Dr. Barrett would not be
16 available, maybe not Friday, but can we inquire of him to see
17 if he would be available on Monday or Tuesday of next week, so
18 that --
19 MS. PAPUSHKEWYCH: He will --
20 THE COURT: Because it would be my view that his
21 testimony is going to take longer than either side has
22 indicated, I think, in order to be complete.
23 MS. PAPUSHKEWYCH: He is currently scheduled to be in
24 Newark starting Friday, tomorrow, on a project.
25 THE COURT: For how long?
429
1 MS. PAPUSHKEWYCH: For a week -- over a week.
2 THE COURT: Well, I mean, I --
3 MS. PAPUSHKEWYCH: And we will --
4 THE COURT: It just seems to me that this is one of
5 the problems that expert witnesses always have. Unfortunately,
6 we --
7 MS. PAPUSHKEWYCH: No, I understand that, your Honor,
8 but these promotions were announced six weeks ago, and we have
9 225 individuals in the academy waiting for your decision. The
10 fact that Mr. Flaxman determined that he would proceed in
11 another courtroom before coming here should not work at
12 hardship on the City of Chicago in this case. And --
13 THE COURT: Well, all right. I really do not know
14 that there is a particular hardship. And part of the
15 additional time that is going to be required is the fact that I
16 did not get from the defense a picture of the use to which
17 these witnesses would be made. I particularly was not aware of
18 the fact of the sort of review committee of sergeants and the
19 like that reviewed his questions and the fact that he removed
20 questions that they objected to saying that they were not
21 related to the activities that took place. So that these
22 motions in limine being in advance without sufficient
23 background, that is going to be an extension that in a sense is
24 the fault of the Court, but I just did not have enough of a
25 background to see clearly until last night really when
430
1 reviewing it all.
2 MS. PAPUSHKEWYCH: And if you would allow me to make
3 my record as to why your ruling now is incorrect?
4 THE COURT: Now. Well, I will, but I do want to go
5 ahead and get as much testimony today with respect to the
6 doctor as we can.
7 Do you want to take a minute, though, to confer --
8 MS. PAPUSHKEWYCH: Just a moment.
9 THE COURT: -- with him to see, because we do need him
10 available next week.
11 MS. PAPUSHKEWYCH: If you would allow us to.
12 THE COURT: Okay.
13 MR. FLAXMAN: And could I apologize on the record for
14 the snow. I didn't want to be late, Judge.
15 THE COURT: No, I understand, and I do not fault you
16 for it. It is just -- it is not going to solve the problem
17 that we have here in any event.
18 MS. PAPUSHKEWYCH: He will make himself available on
19 Monday. Monday only.
20 THE COURT: Okay.
21 Well, that -- we should certainly be able to complete
22 --
23 MS. PAPUSHKEWYCH: We would hope so.
24 THE COURT: -- by Monday.
25 Will he be able to be in by 9:00, or does he need a
431
1 little later time period flying in?
2 MS. GLINK: He can come in Sunday night, your Honor,
3 and be here at 9:00.
4 THE COURT: Oh, okay. That would be fine.
5 MR. FLAXMAN: If there is snow, again, I will come in
6 Sunday, too.
7 THE COURT: Hopefully, this is the last blast of March
8 here.
9 All right. Now, does someone have a copy of the
10 standard form order -- the protective order form to be
11 executed, so we can run off some copies and have all of those
12 who are going to be present in the courtroom when we get to
13 that point sign the orders, and we can verify visually that no
14 one is taking notes.
15 And I would ask that the litigants of both sides
16 assist the Court in being sure that nobody takes any notes.
17 MR. FLAXMAN: I would suggest that you actually post
18 that because we might have press people come in who might want
19 to appeal that order.
20 THE COURT: Well, that is what I am trying to avoid,
21 obviously, and I do not want to complicate things any more than
22 we have. And it is only for the limited purpose of seeking to
23 preserve whatever remains of the sanctity of that test.
24 MS. PAPUSHKEWYCH: Although the defendants would move,
25 your Honor, to exclude the plaintiffs from this testimony,
432
1 since they are individuals who may potentially be taking the
2 next sergeant's exam.
3 THE COURT: Well, I do not see how we can
4 differentiate between the plaintiffs and anyone else. It seems
5 to me that they should all be treated equally than to -- in my
6 opinion, the way we are doing this is, I assume and am
7 confident, that, certainly, with advice of counsel, they are
8 not going to violate the protective order. So, I would not be
9 willing to exclude the plaintiffs. I think that just adds an
10 added problem to a case that we do not need to have.
11 All right. Where are we on -- so, he will be
12 available --
13 MS. PAPUSHKEWYCH: He will be here Monday.
14 THE COURT: -- for Monday.
15 And you wish to make some additional argument?
16 MS. PAPUSHKEWYCH: Yes, I do.
17 THE COURT: Or can you just indicate that you have an
18 additional argument to make and will make it after Dr. Barrett
19 has raced to the airport, so we do not lose any additional
20 time?
21 MS. PAPUSHKEWYCH: That would be fine.
22 THE COURT: Okay.
23 All right. With that, we are ready then for
24 re-calling Dr. Barrett then for direct examination, is that
25 right, in the defendant's case in chief out of order?
433
1 MS. GLINK: I would like to call to the stand
2 Dr. Gerald Barrett.
3 THE COURT: All right.
4 DR. GERALD BARRETT, DEFENDANT'S WITNESS, PREVIOUSLY SWORN
5 THE COURT: Now, what I would ask is if somebody could
6 give me just a blank copy of that form, that one-page form, and
7 then Laura would ask if we could get somebody to run some
8 copies of that, so we will have that.
9 MR. FLAXMAN: I do not have any of those forms with
10 me, and it's my form. I don't know if the city has a copy.
11 THE COURT: We were trying to look this morning. I
12 know we have got one someplace. I just, at the moment, I
13 cannot put my finger on it. There was a motion to bar --
14 motion in limine the city filed, and I wonder if a copy of the
15 protective order and the form was attached to that motion. I
16 am not --
17 MS. GLINK: We did it orally.
18 THE COURT: Somebody attached it to something, and I
19 cannot recall what it was now.
20 MR. FLAXMAN: It wasn't a court-approved form. It was
21 a form that I had prepared.
22 THE COURT: No, but it was one that had generally been
23 used and -- oh, I know when it came up, and I know I saw it at
24 the time that we had that emergency hearing --
25 MS. GLINK: Right.
434
1 THE COURT: -- considering the -- with respect to the
2 letter and what would be --
3 MS. PAPUSHKEWYCH: Your Honor, we will contact our
4 office and have somebody bring over a copy of it. We have it
5 back in the office.
6 THE COURT: Okay. Well, if that can be done --
7 MS. PAPUSHKEWYCH: We have somebody doing it.
8 THE COURT: Okay. That will be helpful.
9 All right. And you have been placed under oath,
10 again, as I recall --
11 MS. GLINK: Your Honor, I would also like to --
12 THE COURT: -- from yesterday. So, you remain under
13 oath.
14 MS. GLINK: I'm sorry, I didn't mean to interrupt you.
15 I would like to request the right to re-call
16 Dr. Barrett on direct after he has had the opportunity to
17 listen to the subject matter experts go through his questions.
18 So, in fairness, the purpose for us calling Dr. Barrett at this
19 point was we had already had the subject matter experts'
20 testimony. So --
21 THE COURT: All right.
22 Now -- but that did flag the individual questions and
23 the arguments or the gist of the criticism of those questions.
24 So, I thought that there is enough information that was
25 developed that you would be able to continue on with his
435
1 testimony. Now, if that is not the case --
2 MS. GLINK: Well, particularly with the in-basket
3 exercise and the oral briefing exercise, we had very general
4 testimony on --
5 THE COURT: I know it was general, and I am now
6 confessing error to that.
7 MS. GLINK: Right.
8 THE COURT: Now, what I would hope that we could do
9 then is once he has to leave, that we would then go back and
10 seek to have plaintiff cover in more detail then the particular
11 matters that were referred to in the offer of proof.
12 MS. GLINK: If we could get a copy of the transcript
13 in time for Dr. Barrett to review it before he is back here on
14 Monday, so he has an opportunity to see what's been said here
15 so, when he comes back on Monday, he will have a chance to
16 respond to some of those arguments that have been raised.
17 THE COURT: All right.
18 THE COURT: What is a fair request that can be made --
19 can be done on trying to make a transcript on the testimony
20 that would be coming on both this afternoon and tomorrow?
21 THE COURT REPORTER: As far as I'm concerned, Judge, I
22 already have transcript that I'm working on for them; and,
23 plus, I'll have this morning. And I'm out of here at 11:00
24 this morning.
25 THE COURT: Right.
436
1 THE COURT REPORTER: So, that will be up to the
2 following reporters.
3 THE COURT: Do we know who that is going to be?
4 THE COURT REPORTER: I'm not for certain. I believe
5 it may be the woman that was here two days ago that I
6 relieved.
7 THE COURT: All right.
8 Now, there is one thing that we can do, counsel.
9 There is one thing that we can do on this. We tape-record, as
10 well as have the -- our court reporter take it down manually.
11 I do not see any problem in making available the audiotapes.
12 Now, this is not -- I know you can read faster than you can
13 hear it, but at least the audiotapes would be available and
14 someone over the weekend might be available to take some notes
15 from that.
16 MS. GLINK: The audiotape is fine, your Honor, just so
17 he has an opportunity to hear the testimony or see it in some
18 way.
19 THE COURT: So, what could be prepared will be
20 available, and we will work it out with the other reporters;
21 but, otherwise, we will see to it that they have enough
22 audiotapes so separate tapes could be made available.
23 MS. GLINK: We would appreciate that, your Honor.
24 THE COURT: All right.
25 With that, you may proceed, then.
Barrett - direct
437
1 DIRECT EXAMINATION (Resumed)
2 BY MS. GLINK:
3 Q. Dr. Barrett, I am going to remind you and the Court what we
4 were talking about at the end of the day yesterday. I believe
5 we had just started to talk about whether or not your test
6 tested for an applicant's ability to listen, and you had
7 commented that it did not directly. Can you refresh our
8 recollection as to why you -- what you meant by the word
9 "indirectly"?
10 A. I believe I testified yesterday that there's been research
11 which indicates that both reading comprehension and listening
12 comprehension are very similar. So that, if you understand
13 what you read, you can also hear and understand it. And
14 because we have a central information processor called a brain,
15 it doesn't really matter which channel the information comes
16 in, if it comes in auditorially or visually.
17 Q. And what's the basis of this opinion that the brain --
18 reading comprehension and listening comprehension is processed
19 similarly?
20 A. Well, it's based upon past research, including some
21 research of my own.
22 THE COURT: It would be helpful for me to have some
23 reference to either generally-accepted treatises in the area or
24 that would refer to tests that took place or any other material
25 that has met peer review before publication with respect to the
Barrett - direct
438
1 position that the witness is testifying to. In other words,
2 any scientific basis for this opinion.
3 BY MS. GLINK:
4 Q. Do you know off --
5 THE COURT: I know you may not have coordinated on
6 this, but, to the extent that he can recall at this point, it
7 will be especially helpful, I think, for the record to have
8 references to actual scientific verification.
9 BY MS. GLINK:
10 Q. Do you recall any of the studies, including your own, which
11 analyzed reading comprehension and listening skills that you've
12 just discussed?
13 A. In the article by Barrett, 1992, which has repeatedly been
14 brought into the courtroom, it was published in "Human
15 Performance." And, in that article, I discuss some of my
16 research.
17 THE COURT: The title of the article, again, is?
18 THE WITNESS: I believe it was "Construct Validity."
19 I believe Mr. Flaxman has a copy.
20 MS. GLINK: I believe it was attached to, actually,
21 one of the submissions by plaintiffs in this case. Let me
22 approach the witness.
23 (Document tendered.)
24 BY MS. GLINK:
25 Q. Is that a copy of the article that you are discussing?
Barrett - direct
439
1 A. Yes, this is called, "Clarifying Construct Validity:
2 Definitions, Processes and Models." It was published in "Human
3 Performance" in 1992, Volume 5, Pages 13 to 58.
4 And while it is not directly related in the sense of
5 what was the main focus of this paper, in it I do review my
6 research with a test called the auditory selective perception
7 test and shows how this test predicts driving accident
8 involvement, for example. How the auditory channel can predict
9 what is basically a visual function. So, that does discuss
10 some of the research in that article.
11 Q. And have you reviewed other studies in your field which
12 similarly analyze the processes that the brain undertakes in
13 analyzing, listening -- when someone listens or when somebody
14 -- reading comprehension skills?
15 A. Yes.
16 Q. Are these studies -- do you recall any of the names of
17 these studies?
18 A. No, I don't.
19 Q. Are these studies, in your recollection, prepared by
20 individuals who are experts in your field of industrial
21 psychology or in the field that would be the specialty for
22 somebody who would be analyzing how a brain processes
23 information, like reading and listening skills?
24 A. Well, some of these studies are back in the 1970s. So,
25 they were in the context usually of saying, "Can we reduce
Barrett - direct
440
1 adverse impact by giving a listening test, instead of a written
2 test?"
3 And I don't recall the exact authors' names at this
4 point, but that was the purpose of the research.
5 Q. And the purpose of the research looked at listening tests
6 --
7 A. Yes.
8 Q. -- and their impact?
9 A. Yes.
10 Q. Can you explain that in more detail, please?
11 MR. FLAXMAN: Could we first have a foundation as to
12 what specific research she's talking about?
13 THE COURT: Yes, to the extent that he can recall, let
14 us pin down exactly what type of research it was that --
15 BY MS. GLINK:
16 Q. Do you recall reviewing these articles that you are
17 discussing?
18 A. I cannot recall the exact authors' names or where it
19 appeared because it was back in the 1960s and early 1970s. All
20 I can recall is the results from those studies.
21 Q. And based on your 30 years of experience --
22 THE COURT: Well, you might just ask him the type of
23 publication or what it might have been in or something. Just
24 try to get --
25 BY MS. GLINK:
Barrett - direct
441
1 Q. Do you recall whether it was in a peer-reviewed article or
2 journal?
3 A. As I recall, it was, yes.
4 Q. And can you explain what a peer-reviewed article or journal
5 is?
6 A. It's one where the paper is submitted and reviewed by
7 people in that discipline to see if it meets the standard for
8 publication. And if it does, it is published.
9 Q. And to the best of your recollection, these reviews that
10 you are referring to all went through that peer review process?
11 A. As far as as much as I recall, yes.
12 Q. Based on your professional opinion, is it generally
13 accepted in the field of your expertise that listening skills
14 and reading comprehension skills tap similar functions or
15 abilities of an individual?
16 A. It's usually not discussed very much, to be honest about
17 it. I don't have an opinion about -- or opinions on this
18 issue.
19 Q. Your review of these articles and, in your experience, do
20 direct listening tests have a tendency to impact any particular
21 group?
22 MR. FLAXMAN: Let me object to the form of the
23 question. I don't know what a direct listening test is.
24 BY GLINK:
25 Q. Can you describe for me what kind of -- how you would test
Barrett - direct
442
1 for somebody's listening skills?
2 A. Well, you are asking a general question. You could, I
3 guess, do it a number of different ways. But one way, of
4 course, is you could have someone listen to a tape of a
5 conversation and then indicate what they believed had -- was
6 being said.
7 In other words, could they comprehend what was being
8 said -- the message? You have to do it in some objective
9 format. You have to score in some way.
10 You can do it, obviously, a number of different ways.
11 So, it could be a process where, I guess, again, you could
12 listen to a tape; and, in theory, you could have them respond
13 into a tape-recorder, again, and say, "Here is what I believe
14 they said." And you could have somebody else rate that
15 understanding.
16 Or you could listen to a tape of a conversation and
17 have them answer questions in the written format. So, I guess
18 there is a number of different ways you could do it
19 potentially.
20 Q. Have you ever developed a listening -- a test that would
21 tap somebody's ability to listen?
22 A. Well, again, I mentioned the Auditory Selective Attention
23 Test, ASAT.
24 Q. And in what context did you develop that test?
25 A. I'm sorry?
Barrett - direct
443
1 Q. In what context did you develop that test?
2 A. We were developing that test to predict, among other things
3 -- our main focus at first was to predict the ability of
4 someone to avoid accidents, actual accidents on the road,
5 driving a car accidents.
6 Q. And why did you choose to use this type of test in that
7 situation?
8 A. There had been more fundamental research in the field of
9 information processing, which indicated that was a very
10 efficient way to determine how people focussed on events and
11 actually focussed their attention on events and responded to
12 them.
13 So, it's basically a test where you have earphones
14 on. It's called a dichotic listening test, and you have
15 messages coming in both ears. And you have to say, for
16 example, is the number the same or different. It's a very
17 abstract type of test. And based upon the errors people make,
18 I could predict whether or not they would be involved in actual
19 accidents on the roadway, driving cars or trucks. You start
20 out with truck drivers first.
21 Q. Had you identified listening skills as something that was
22 important for a sergeant?
23 A. Yes. Certainly, you have to -- you have to be able to
24 listen and comprehend on the job.
25 Q. And why did you choose not to test listening skills
Barrett - direct
444
1 directly?
2 A. Well, again, it would be a more complex way of doing it;
3 and, in fact, we were testing for it indirectly through our
4 comprehending written material.
5 Q. Are there any other reasons why you chose not to use the
6 type of test that you have just described for me in this
7 situation?
8 A. There's also a practical aspect of developing a test like
9 that. There's the issue of reliability. If you did ratings,
10 for example, it would be an issue of reliability. If you make
11 it all interactive, it would become extremely complex and not
12 standardized. So, there's a number of practical
13 considerations.
14 Also, we had unusually good coverage of the content
15 domain. As I said before, many jurisdictions might only use a
16 job knowledge test.
17 Q. When you say a "job knowledge test," that would be like the
18 written multiple-choice job knowledge test that we've been
19 discussing here?
20 A. Yes, but we had, of course, the oral comprehension test,
21 and we had also had the administrative decision-making test.
22 Q. And that's the in-basket simulation?
23 A. Yes.
24 Q. Did the sergeant's examination that you developed test for
25 an applicant's ability to write clearly and concisely?
Barrett - direct
445
1 A. We didn't test directly for the ability to write clearly
2 and concisely, no.
3 Q. Is that a task that had been identified during your job
4 analysis process as being something that a sergeant has to do?
5 A. Yes, a sergeant does have to write at times.
6 Q. Why did you choose not to include an essay or short answer
7 type of examination in this case?
8 A. Well, there are a number of problems with that. First,
9 there's psychometrics problems of -- that those type of tests
10 tend to have low reliability. In other words, again, it
11 depends upon somebody rating whether or not they believe
12 whatever was written was written correctly.
13 Q. When you say "low reliability," since we're not industrial
14 psychologists, can you explain what you mean by that and what
15 problems may be associated with that?
16 A. There have been a number of studies on how people judge
17 essays. And when we say -- I talk about reliability, I'm
18 talking about two different people in this case judging the
19 same essay, for example, or the same report.
20 And what we have found over and over, again, is that
21 people have different judgments of how good an essay is or even
22 how good a short report might be. So, you have two different
23 raters, and they will come to different conclusions about how
24 good that essay is. And, so, you have low reliability between
25 the two different raters. So that causes a problem for you
Barrett - direct
446
1 from a psychometric point of view.
2 There's also a practical point of view about, if you
3 have everybody do this process, it would be very time-consuming
4 to go through and rate each person's essay. So, you have a
5 constraint in terms of time that they could do it and expense.
6 Q. At the time you were determining what component parts to
7 include in the test, what -- how many expected applicants --
8 how many applicants did you expect may take this examination?
9 MR. FLAXMAN: Object. Could we have foundation as to
10 what time this was that -- when he was --
11 THE COURT: Okay.
12 I will require you to lay a more detailed foundation
13 for that.
14 BY MS. GLINK:
15 Q. At the time that -- at what point in the test process did
16 you make the ultimate decision that you were going to include
17 the three component parts of the test that ultimately were part
18 of this test: The in-basket, the written short answer and the
19 oral briefing exercise?
20 A. Well, we did -- we made that decision before the posting
21 notice went out.
22 Q. And the posting notice went out -- do you recall when the
23 posting notice went out?
24 A. I believe it was September of '93, I believe.
25 Q. And, in September of '93, at that point in time when the
Barrett - direct
447
1 posting notice went out, that's the examination announcement
2 telling people when to apply and how to apply to take your
3 examination?
4 A. Yes.
5 Q. And, at the time that that posting notice went out in
6 September of 1993, how many applicants had you anticipated may
7 sign up to take this test?
8 A. I was told there could be as many as 8,000 applicants.
9 Q. And did the number of applicants that you had expected
10 might sign up to take this test in any way play a role in your
11 determination not to include a written short answer or essay
12 component?
13 A. Yes.
14 Q. And can you explain why that is?
15 A. Well, again, as I said before, it's the time and expense of
16 actually scoring an essay-type approach. You also have an
17 issue about what is a right and wrong answer in terms of
18 someone challenging this. They would say, "My essay is good."
19 How do you actually go back and determine what is right or
20 wrong?
21 And, as I said, it would be very time-consuming to do
22 this and expensive.
23 Q. Based on your experience, what are the results -- what kind
24 of results do you generally get in terms of the scoring of
25 these types of examinations?
Barrett - direct
448
1 MR. FLAXMAN: Object to the form of the question.
2 What types of examinations?
3 MS. GLINK: Essay and short answer examinations.
4 MR. FLAXMAN: And, again, I would like more foundation
5 as related to police officers or just --
6 THE COURT: All right.
7 I will sustain the objection as to form, but not the
8 subject matter. If we could just be a little more specific.
9 BY MS. GLINK:
10 Q. Are there any other problems with written short answer
11 questions that you are aware of in terms of the scores that
12 applicants tend to receive on essay and short answer types of
13 questions?
14 A. Well, there's always a possibility that they would be
15 bunched together at one end of the spectrum and it wouldn't
16 differentiate individuals.
17 Q. Do you have experience giving essay and written short
18 answer examinations?
19 A. Not in the employment context. We don't usually do that in
20 the employment context.
21 Q. And is there a reason why you as a test developer who's
22 been developing tests for 30 years have decided not to use
23 these types of essay and written short answer tests in the
24 employment sector?
25 A. Well, it goes back, again, to the psychometric issues and
Barrett - direct
449
1 reliability. If you have low reliability, you have low
2 validity. But more basically, it goes back to research which
3 has been done over the last 20 or 30 years in comparing
4 constructed response tests versus structured tests.
5 Q. Okay.
6 Can you explain the difference between a constructive
7 response test and a structure test?
8 A. A prototype of a constructive response test would be an
9 essay, for example. In other words, the testee has freedom to
10 respond anyway he or she wants to.
11 Conversely, a test which is structured would be one,
12 again, where you would say it's multiple choice. You choose
13 which is the best response; and, so, there is a right and a
14 wrong answer.
15 And what that research has shown is basically two
16 things. The first thing it has shown is that the structured
17 approach is much more reliable, and if you have high
18 reliability, you, therefore, can have higher validity.
19 And, second, it's much more economical to score a
20 structured approach.
21 And, third, the hope was, at one point, that if you
22 used constructed response tests like essays, you would have
23 less adverse impact. In fact, the research has shown you may
24 have more adverse impact. So, giving a constructed response
25 test does not assure you that you are going to have less
Barrett - direct
450
1 adverse impact. In fact, it could be more adverse impact.
2 Q. That plays a role in your decision as to -- has played a
3 role in the past in your decision as to whether to include such
4 a test?
5 A. Yes.
6 Q. You mentioned the relationship between reliability and
7 validity. Can you explain what you meant by, if you have a
8 higher reliability, you will have more validity?
9 A. Well, at the extreme, if no one agrees about how good the
10 essay is, the two raters, how do you even score it? What's the
11 score mean?
12 So, the general psychometric principle is your
13 validity is limited by the reliability. If you have very low
14 reliability, you can't have a valid test.
15 Conversely, if you have high reliability, there is the
16 potential to have high validity. It doesn't mean, of course,
17 if you have high reliability, that ensures the test is valid.
18 But it is a prerequisite for a valid test.
19 Q. In your opinion, did the three test components of the 1993
20 examination that you developed test for a significant portion
21 of the job domain for the position of sergeant on the Chicago
22 Police Department?
23 A. Yes.
24 Q. And you've mentioned it before, but can you explain what
25 the basis of that opinion is?
Barrett - direct
451
1 A. Well, the basis of my opinion is, first, on job analysis,
2 which we conducted, which it identified major work behaviors
3 and identified the knowledge, skills and abilities required for
4 each of the major work behaviors; and, the fact that we, in
5 effect, could and did develop, following the content-validity
6 process, three tests which did tap those domains which are
7 important for a police sergeant.
8 And, of course, there's a general body of literature,
9 of course, which indicates that these types of tests, in
10 general, do predict job performance.
11 Q. In your professional opinion as a test developer, is the
12 fact that certain skills, like listening skills, were not
13 directly tapped, or writing skills, were not tested on this
14 examination in any way effect your opinion that the examination
15 tested for a significant portion of the job domain?
16 A. No, it does not.
17 Q. In your opinion, do you believe that the 1993 sergeant's
18 examination that you developed conformed with the uniform
19 guidelines on employee selection procedure?
20 A. Yes, it did.
21 Q. Did you make any attempts during this process to ensure
22 that you were following the requirements of the uniform
23 guidelines?
24 A. Yes.
25 Q. And is that set forth anywhere in your technical report?
Barrett - direct
452
1 A. It's in an appendix of my report.
2 Q. I believe the technical report is still in front of you,
3 Dr. Barrett. I'd like to direct your attention to --
4 THE COURT: We are talking about the December 13,
5 1994, report?
6 MS. GLINK: Yes.
7 THE COURT: Okay.
8 BY MS. GLINK:
9 Q. I'd like to direct your attention to Appendix P.
10 THE COURT: P, as in Peter?
11 MS. GLINK: P, as in Peter.
12 THE COURT: Okay.
13 BY MS. GLINK:
14 Q. Can you tell the Court what Appendix P is?
15 A. It's a point-by-point reference to the Guidelines on
16 Employee Selection Procedures of 1978.
17 Q. And that Appendix is 18 pages -- actually, it's not -- 18
18 pages long; is that right?
19 A. Yes.
20 Q. Okay.
21 And can you -- let's turn to the second page of it,
22 P-2. Can you just explain to me briefly what information is
23 contained in this Appendix?
24 A. Well, for example, the first column indicates a section of
25 the Uniform Guidelines in terms of issues, and it references a
Barrett - direct
453
1 section of the Uniform Guidelines and an issue to be addressed
2 in this case, in content and validity:
3 "A selection procedure is justified by showing it
4 representatively samples significant parts of the job, such
5 as a typing test for a typist."
6 And the next column is our response, where we, in
7 effect, say how this corresponds to what a content-valid test
8 is.
9 THE COURT: And you are reading from a page marked
10 P-2?
11 MS. GLINK: That's right.
12 BY MS. GLINK:
13 Q. And what was the purpose of doing this point-by-point
14 response to the Uniform Guidelines?
15 A. It would allow the reader to focus on the important part of
16 the guidelines; at the same time, focus upon our response as to
17 why we believe this to be a content-valid test.
18 Q. And are test developers in your field required to follow,
19 in your opinion, the Uniform Guidelines in developing tests?
20 A. Yes.
21 Q. Did you also take steps to ensure that your examination
22 that you developed, that you followed a process that would be
23 supported by the standards -- professional standards in your
24 field?
25 A. Yes.
Barrett - direct
454
1 Q. And can you tell us for the record, again, what the two
2 professional standards -- what the professional standards are
3 in your field that dictate how a test should be developed or
4 the steps a test developer should follow?
5 A. One is the "Principles for Employment Testing," put out by
6 the Society for Industrial and Organizational Psychology, and
7 the second is "Standards for Psychological Testing," put out by
8 the American Psychological Association, among others.
9 Q. And you followed those standards, as well?
10 A. Yes.
11 Q. In your opinion, Dr. Barrett, is the examination that you
12 developed content valid?
13 A. Yes.
14 Q. And is your determination that the sergeant's examination
15 that you developed is content valid dependent in any way on the
16 review of the two subject matter experts from the Chicago
17 Police Department -- Chief Cadigan and Lieutenant Klein?
18 A. No.
19 Q. And why isn't it dependent in any way on their review?
20 A. Before they came for the final review of our set of items,
21 we had developed what we considered to be a content-valid test
22 for police sergeant. In fact, in many jurisdictions in which
23 we work, there is no review of the content-valid test by
24 anybody in the jurisdiction, except ourselves. And the reason
25 is, as I have mentioned before, there is a great fear of
Barrett - direct
455
1 unfairness and a fear that the test will be compromised. So,
2 it's not unusual for us to develop a content-valid test, and
3 there is no review by anyone.
4 In this case, we had limited the review to only two
5 individuals. We were assured that there would be no compromise
6 of the test items. And, so, this was just a final check on
7 what we had already developed as a content-valid test.
8 Q. For clarification of some issues that were brought up
9 yesterday, I just had a few follow-up questions. When you're
10 developing items -- did you develop the items that -- all of
11 the items that were ultimately used on the three test
12 components of your test battery for police sergeant?
13 A. Are you asking did I use all the items --
14 Q. Did you develop all of the items that were considered in
15 developing your test?
16 THE COURT: Individually or by his --
17 MS. GLINK: Was he responsible and people at Barrett &
18 Associates that work for him?
19 BY THE WITNESS:
20 A. Yes, all items that were developed were developed by myself
21 or with the team of individuals assigned to the project.
22 BY MS. GLINK:
23 Q. And those individuals all worked for Barrett & Associates?
24 A. Yes.
25 Q. And when you're developing test items for an examination,
Barrett - direct
456
1 do you develop more than the number of test items that you
2 ultimately use on the examination?
3 A. Yes.
4 Q. And what -- why do you do that?
5 A. Well, as you go along in the process, you'll find or you
6 gain more information because it's a continual process, and you
7 -- of review.
8 You find many items you have developed at first are
9 not appropriate for a variety of reasons -- technical reasons
10 in terms of item writing. We feel that an item might be more
11 relevant.
12 So, basically, we develop a pool of items which we
13 believe will tap that domain, for example, of knowledge. And
14 this gives the chance to review and refine the item pool and
15 select from that pool those items which are most appropriate
16 for the test.
17 Those which will be representatively sampled, for
18 example, a domain of knowledge, which is required for police
19 sergeant.
20 Q. Can you explain that a little bit more thoroughly? What do
21 you mean by a representative sample of a job domain?
22 A. Well, it's probably simplest to refer back to the source
23 list for the written job knowledge test.
24 Q. And is that source list referenced in your technical
25 report?
Barrett - direct
457
1 A. Yes, it is.
2 Q. I'm going to direct your attention to Appendix F of Exhibit
3 D, so that when you're explaining this process, we'll be able
4 to refer to the source list.
5 And is Appendix F the source list that you are
6 discussing?
7 A. Yes.
8 Q. Okay.
9 So, can you continue to explain what you mean by a
10 representative sample of a job domain?
11 A. As you recall from yesterday, I said that we had a process
12 of going through and determining from the job analysis and from
13 other individuals in the police department where we could find
14 the knowledge they use to perform the job of police sergeant,
15 and we narrowed the list down to what you see in Appendix F.
16 As you might recall, also, we mentioned the fact that
17 there were parts of the general orders and other parts of the
18 criminal code which were not used.
19 Q. And those are set forth in Appendix G?
20 A. Yes.
21 So, we had materials which we thought were not as
22 pertinent for a variety of reasons, and we put down the
23 rationale for excluding those sources and the content from
24 being our operationally-defined domain of knowledge.
25 Q. So, the sources listed in Appendix F, are those the sources
Barrett - direct
458
1 that, in your opinion, based on the test developer, are the
2 sources that a sergeant needs to know?
3 A. Yes.
4 Q. And when you said "representative sample," can you explain
5 now what you're talking about?
6 A. Well, what -- let's go back to more basic tests. That may
7 be the easiest way to explain what I'm trying to say -- is that
8 in basic test theory, a good test is not based upon one item
9 itself; but, if I were to go through here and pick out one item
10 and say, "This is a test," obviously, it has no reliability,
11 it's not valid. It does not sample the domain of knowledge.
12 The real secret about developing a good test, in terms
13 of test theory, you have a number of items -- a large number of
14 items which sample that domain of knowledge. It's the
15 aggregation of the correct responses on those large number of
16 items which allows you to say, "Yes," and make the inference
17 that this individual who scores very high has a mastery of this
18 job knowledge which is required for performing well as a police
19 sergeant.
20 Conversely, if they can answer very few items, that
21 means, of course, that they do not have the knowledge. And,
22 empirically, we have shown that does not relate to job
23 performance in other studies.
24 THE COURT: It does not?
25 THE WITNESS: Does not. In other words, low test
Barrett - direct
459
1 scores mean low performance, in general.
2 THE COURT: Okay.
3 BY THE WITNESS:
4 A. In other words, any one item itself is not too important.
5 It's the aggregation. In this case, we had 150 items which
6 sampled the domain.
7 Now, clearly, we cannot test practically and have a
8 question for every piece of knowledge or every order. So, we
9 want a representative sample of this in terms of general
10 orders, specific orders, Illinois Criminal Code, and so forth.
11 So, part of our process is to see that, in fact, we do
12 attempt to write items which are going to cover this domain.
13 And any one item might represent theoretically a much greater
14 bit of knowledge which a sergeant should have.
15 Q. Can you explain what you mean by that?
16 A. In other words, if you don't know this item, okay, it
17 really represents the fact there is a broader domain of
18 knowledge the person probably doesn't know, either.
19 Q. Can you give me a specific example like a test area of
20 knowledge you might test on your test, put an item in on a
21 particular knowledge area that would be representative of an
22 understanding of a larger body?
23 A. Well, for example, in the criminal code, if we found that
24 the individuals could not answer questions about the Illinois
25 Criminal Code, this would indicate not only questions about
Barrett - direct
460
1 what we ask, but also a broader domain of the Illinois Criminal
2 Code which we did not test on.
3 Q. So, that they don't -- it's not that they just don't -- in
4 other words, it's not that they just don't know this particular
5 crime, but the -- or this particular part of the code that you
6 are testing on, but, as a test developer, from your
7 perspective, it indicates they might not know any of it?
8 A. Exactly, the domain.
9 MR. FLAXMAN: Objection to the form of the question.
10 I am not sure who is testifying there.
11 THE COURT: Well, I will overrule the objection
12 because I think it sought to restate what his answer was, and I
13 do not understand that she was adding anything or subtracting
14 anything from his prior testimony. So, it is difficult in this
15 area partly because of the specialized vocabulary, but partly
16 just because the detail we are dealing with. So, I will
17 overrule the objection and permit the question and answer to
18 stand.
19 BY MS. GLINK:
20 Q. Do you have to make judgments as a test developer about
21 what knowledges or what skills or what abilities should be
22 tapped by the items in your test battery?
23 A. Yes, I do.
24 Q. Can you explain the type of -- what factors you may take
25 into consideration in this process?
Barrett - direct
461
1 A. Well, there are a number of factors; but, what, basically,
2 I do, I review each and every item. In other words, for each
3 item, we have a process we go through where it's been refined.
4 There may be a page or two in terms of talking about what and
5 the rationale for each item.
6 So, I actually make a decision about whether or not
7 this item should be used or not. I have to make a decision
8 about how many items we should have in each domain. In other
9 words, we wouldn't want to have a large number of items tapping
10 only one area of the domain.
11 We also have to make a decision about, from the source
12 materials, can we really write a respectable item from that
13 domain -- from that general order, for example. Maybe a
14 general order which was deemed relevant doesn't really lend
15 itself from a testing point of view for us to be able to write
16 a good item. Some areas are much more difficult than others to
17 write good items from.
18 Q. How do you --
19 A. So, all these things have to be taken into account.
20 Q. When you say "good item," how do you make a -- when you say
21 you have to write a good item, can you explain what you mean by
22 a "good item"?
23 A. Well, a good item is one where the testee can -- it's a
24 relevant item, first, that does tap the domain, and it's
25 written in a form which does allow the testee to understand and
Barrett - direct
462
1 respond to it.
2 And you have to have in the items we write -- for
3 example, the job knowledge test -- we have what's called a
4 stem, which sets out what's the question you're asking,
5 basically, or the situation. Often, our -- we try to frame an
6 item often in terms of a concrete situation which potentially a
7 police sergeant might face. And, then, you have to have the
8 correct alternative and be sure it is correct.
9 And, second, you have to have what's called
10 distractors. In this case, we had four distractors.
11 Q. Can you explain what a distractor is?
12 A. It's a potentially plausible response, where the person who
13 does not have the required domain knowledge might pick that
14 distractor as the correct answer because it seems plausible.
15 So, if you had a test, for example, which had
16 completely implausible distractors, it wouldn't be a good item
17 on a test.
18 Q. Why not?
19 A. Because everyone with no actual job knowledge could pick
20 the right answer. So, it would not be based upon the knowledge
21 the person had. It would be based on when the fact that there
22 was only one plausible correct answer which was, in fact, the
23 correct answer and the other four distractors were so
24 implausible that no one ever would choose those as the correct
25 answer, whether or not you knew anything about the domain --
Barrett - direct
463
1 for example, the Illinois Criminal Code -- or not.
2 Q. And why is it important that your distractors be plausible,
3 like you're saying?
4 A. Well, again, you would -- everyone would have a correct
5 answer. There would be no differentiation among individuals.
6 And more basically, you couldn't tap and determine if the
7 person really had knowledge of that criminal -- Illinois
8 Criminal Code, for example, or just the fact that they could
9 read the stem and the five alternatives and pick the only
10 plausible one as the correct answer.
11 So, you would not be picking or testing for domain
12 knowledge. You would be testing for whether or not a person
13 could pick the plausible answer and disregard the implausible
14 alternatives.
15 Q. How long does it take -- when you talk about this item
16 development process, what kind of time frame are we talking
17 about from the beginning when you begin to develop an item to
18 when you complete developing that item?
19 A. It can take -- it takes months to develop an item because
20 of our review process.
21 Q. And is that the same -- is that true for this case, as
22 well, that it took a couple of months?
23 A. Yes.
24 Q. And the reason for why it takes so long -- can you explain
25 why it takes so long to develop one item on a test?
Barrett - direct
464
1 A. It's because of the fact we take a great deal of care in
2 being sure the job analysis represents some work behavior from
3 the job analysis; that we have determined the appropriate
4 specific piece of information which is important for a police
5 sergeant to know; that we have put the question in the right
6 format in terms of the stem, the correct alternative and the
7 four plausible distractors. And, then, we also do pilot
8 testing, have people who are naive and studied the material.
9 We used two types of pilot testing, as I recall, in this case.
10 THE COURT: You did get the two instances, you said.
11 THE WITNESS: Maybe I should clarify it. As I recall,
12 we will, for example, give an item to people who have no domain
13 knowledge.
14 BY MS. GLINK:
15 Q. And when you say, "no domain knowledge," in this case, you
16 are talking about, what?
17 A. Knowledge of the Illinois Criminal Code.
18 And if they can answer that question correctly, then
19 you would say, "Well, they have no knowledge. Maybe our
20 alternatives are not plausible enough. We should do some work
21 on that."
22 A second type of pilot testing we do is let someone
23 study material -- again, for reasons of confidentiality, they
24 are not members of the Chicago Police Department, nor associate
25 with anyone in Chicago -- and see if by studying one area and
Barrett - direct
465
1 domain they can, in effect, obtain the correct answer.
2 Q. And what's the purpose of the second type of testing?
3 A. Well, to see if, indeed, that there is problems in terms of
4 grammar in terms of how it's stated. Can someone who studied
5 the material actually obtain a correct answer.
6 Q. And they should be able to do so if it's a good item?
7 A. Well, some group should be able to, yes.
8 Q. And do you go through the same -- you went through the same
9 process -- you have talked about your job knowledge test. Did
10 you follow the same process and procedures in item development
11 for the in-basket and oral briefing exercise?
12 A. Yes.
13 Q. And did they both -- did the items that you developed for
14 both of those examinations take the same kind of period of time
15 as the development process for the items that you used on your
16 written job knowledge test?
17 A. Well, there aren't really items per se on the oral
18 communication test because the candidate speaks into a
19 microphone, is taped and rated. It's a rating process. But we
20 take a great deal of care with the rating process, also.
21 Q. And this process of determining, these judgments that you
22 have to make in determining what items to include on your test,
23 is that based on your over 30 years of test development
24 experience?
25 A. Yes, and the professional literature in the field.
Barrett - direct
466
1 Q. Dr. Barrett, did you take any steps during your test
2 development process to minimize adverse impact against any
3 minority or any group in general?
4 A. Yes.
5 Q. Can you describe for me the steps that you took to minimize
6 adverse impact in the test development process?
7 A. Well, the first step, of course, is to follow a good
8 procedure -- professional procedure, to be sure that it is a
9 content-valid test and we are only tapping relevant material.
10 As an example, again, we identify specifically source material
11 which is relevant for a police sergeant.
12 The second way is to disregard information which is
13 not relevant. And, again, that's documented for the job
14 knowledge test, which are the sources which are not relevant.
15 Q. Why is developing a content-valid test that's specifically
16 related to the job, why is that in any way related to adverse
17 impact on an examination?
18 A. Well, you have to sort of go back to the history -- if you
19 want me to -- back to the history of testing and what started
20 out. I would say somewhat naively, maybe in the 1960s or
21 before, police testing might test general knowledge. They
22 might ask who was the mayor of the City of Chicago. They know
23 it's not relevant to what a police sergeant does.
24 They might ask about political science questions from
25 the newspaper, current events, and that's not relevant.
Barrett - direct
467
1 So, what you're doing is introducing potential bias
2 into a test, which has nothing to do with the knowledge
3 required to be a police sergeant.
4 Q. What other steps, other than trying to develop -- or
5 developing a content-valid test did you take to try to minimize
6 adverse impact?
7 A. We, of course, did a very thorough job analysis in
8 interviewing over 120 people. But in the job analysis process,
9 we also included minorities and non-minorities, males and
10 females to represent a portion in our population of sergeants.
11 We had minority individuals involved in the job analysis and
12 all phases of the project.
13 We used two professional test develop processes in
14 developing our items, for example, so it would be the best
15 technology was applied to item development.
16 We took steps such as having one of our associates, a
17 minority member, apply to each and every item a form developed
18 by Dr. Richard S. Barrett, who is no relation. Dr. Richard S.
19 Barrett is a well-known industrial psychologist who testifies
20 for plaintiffs in safety force cases. He's been doing it for
21 20 -- over 20 years.
22 He's developed a form which he applies to items to see
23 if, in fact, they are appropriate. We applied that form to all
24 of our items we used to determine if, in fact, it was
25 appropriate.
Barrett - direct
468
1 Q. And did you make a determination, based on that form, that
2 you met the criteria that Dr. Richard S. Barrett has set forth?
3 A. Yes.
4 Q. And that was for each and every item that was included in
5 your test?
6 A. Yes.
7 Q. What other steps did you take to try to minimize adverse
8 impact?
9 A. The whole -- besides our good professional practice in
10 developing the tests, there's also the issue administration of
11 the test. We took great care that no one would have any access
12 to the test questions before the test was given.
13 We used extraordinary security procedures from my
14 point of view. All test items were developed in Akron, Ohio,
15 and we had steel doors on the outer entrance to our testing
16 area, to our offices, sign-in process for visitors and for
17 staff. Any visitor has to be escorted. But the actual test
18 writing process and test materials were stored in a separate
19 room behind a separate steel door with a combination lock known
20 only to those who were actually on the Chicago project, and
21 that room had no windows. And, of course, all the material was
22 also stored in locked cabinets. So, no one would have access
23 to the test materials, the test items we wrote. So, we used
24 those sorts of security procedures to be sure no one would have
25 an unfair advantage in taking the test.
Barrett - direct
469
1 In the test administration process, everyone was
2 informed, for example, ahead of time by the posting notice that
3 -- the type of test we were going to give, how it was going to
4 be weighted, how it was going to be scored.
5 Everyone was told, "Here are the sources which are
6 going to be used for the job knowledge test." So, everybody
7 had equal opportunity to study material at the same time.
8 And the actual administration of the test, we worked
9 with Arthur Andersen, who was actually responsible for the
10 administration, to be sure that it was test security and good
11 test administration procedures were followed. So that everyone
12 had identical conditions, had identical materials, identical
13 time, identical instructions.
14 So, these were all steps we took to be sure there was
15 no unfairness which would be perceived or would occur in
16 actuality.
17 Q. The item writers on your test who were developing the test,
18 were they all of one race?
19 A. No.
20 Q. Can you -- if you know, do you know the racial composition
21 of the item writers involved in developing the items for the
22 sergeant's examination?
23 A. I don't recall exactly their composition. We had those who
24 would be identified as black-Americans, Asian-Americans, males
25 and females. They all had master's degrees in industrial
Barrett - direct
470
1 psychology or Ph.D.'s.
2 Q. And other than the steps you already mentioned, did you
3 take any other steps to minimize adverse impact on the test?
4 A. Again, I would say I reviewed the literature in the field
5 in terms of differences among groups on a variety of different
6 types of tests, looked at alternatives to what we were doing.
7 So, we looked at alternative selection procedures also as part
8 of our process.
9 Q. You testified yesterday briefly, I believe, about a
10 computer simulation that you conducted during the process, in
11 which you were trying to determine how to weight the
12 examination. Was that process in any way related to the
13 adverse -- your attempt to minimize adverse impact?
14 A. Yes, that was an abstract computer simulation, where I was
15 looking at the possibility that a different weighting process
16 might prove to be equally valid and have less adverse impact.
17 Q. And did that computer simulation -- what was the results of
18 that computer simulation?
19 A. Well, again, it confirmed the results of our other evidence
20 from the job analysis -- our other analyses that, for this test
21 at that point in time, an equal weighting process was best.
22 Q. After the test was administered -- after each component
23 part of the test was administered, is it -- was it your
24 understanding that there was a mechanism by which applicants
25 could challenge items on the test?
Barrett - direct
471
1 A. Yes.
2 Q. Can you describe for me, if you know, why you included a
3 challenge procedure in this testing process?
4 A. Well, the challenge allows, in terms of fundamental
5 fairness, allows the testees to say, "All right, I don't
6 believe this item is correct for some reason." And, certainly,
7 there's no perfect test ever constructed. So, there is always
8 the possibility that, there is, for some reason, an item which
9 is not as good as it should be.
10 So, this allows the testees to -- as I recall, there
11 was a hotline maintained by Arthur Andersen to call in and
12 challenge an item. And, of course, after a challenge was made,
13 we would, of course, review and make a determination whether or
14 not the challenge was valid.
15 Q. Is having a challenge procedure something that you
16 generally have when you're developing -- is that something that
17 you've had in the past when you've developed tests for other
18 police and fire departments?
19 A. Well, there are some variations. I think the vast majority
20 of jurisdictions do allow some form of challenge on the job
21 knowledge test for sure, it's been my experience. Other forms
22 of tests, it might vary in terms of allowing the challenge or
23 not.
24 Q. And it's your understanding that that procedure was that
25 there was some procedure for challenging in place here?
Barrett - direct
472
1 A. Yes, there was.
2 Q. And did you receive any challenges to the component parts
3 of your tests?
4 A. Yes.
5 Q. How many items, if you know, were challenged or how many
6 challenges did you receive on the written 150 written job
7 multiple-choice questions on your written job knowledge test?
8 A. I don't recall the exact number now. I think there was
9 around seven, though, as I recall, at the most.
10 Q. And when you received these challenges, what did you do?
11 A. As I recall, Dr. Cellar was in charge of convening a panel
12 of Chicago police supervisors to review and make a
13 determination concerning the challenges.
14 Q. Were there also people from Barrett & Associates who were
15 on this panel?
16 A. Yes, Dr. Cellar, I mentioned, was on this panel.
17 Q. And this panel --
18 THE COURT: Can we describe what the makeup of the
19 panel is --
20 MS. GLINK: Yes.
21 THE COURT: -- to the extent --
22 BY MS. GLINK:
23 Q. Do you recall who was on the panel?
24 A. I'm sorry, I don't recall at this point.
25 THE COURT: Could you describe the source of obtaining
473
1 panel members or --
2 THE WITNESS: It should be in my report. I thought it
3 would be in my report.
4 MS. GLINK: Let me --
5 THE COURT: How long -- late can Dr. Barrett remain
6 now?
7 MS. GLINK: We have a ride for him downstairs. I
8 didn't realize it was 11:00 o'clock, already.
9 THE COURT: You have to tell me when you need to
10 terminate this.
11 MS. GLINK: I think we need to terminate it now.
12 THE COURT: All right.
13 MS. GLINK: So that Dr. Barrett -- he has to go to his
14 hotel first before he needs to get to the airport.
15 THE COURT: All right.
16 We will break his testimony then at this point. So,
17 we will thank you and excuse you at this time and plan to
18 resume then on Monday at 9:00 o'clock.
19 We have our next shift court reporter arriving now;
20 and, so, if you can work your changeover, but what we are going
21 to need is to be sure that you have some extra audiotapes.
22 Certain portions of the questions that are asked are going to
23 be placed on -- a portion of them is going to be placed on a
24 separate tape, which is then going to be made available to the
25 witness, so that he can listen to that in preparation for the
474
1 completion of his testimony. And the simple way to do that
2 would be to break the normal tape, because that is going to be
3 ordered to be under seal, and then plug in a separate
4 audiotape, so that the part that's under seal will be on a
5 separate tape that will be available for review by the witness
6 who has just departed.
7 Now, I do not know if you have got enough tapes at
8 this point and where you can get access to the audiotapes.
9 THE COURT REPORTER: Will you be taking a lunch?
10 THE COURT: I am not sure how we are going to proceed
11 right now, but it seems to me one way or another we have got to
12 make -- you need to obtain some additional tapes, so that you
13 can take out the one that you are using, plug in a fresh tape
14 for the portion of the questions that are going to be ordered
15 to be under seal. They are going to be subject to a protective
16 order.
17 THE COURT REPORTER: But I am going to be able to keep
18 my tape to transcribe, right? I will be making an extra copy?
19 THE COURT: Well, I mean, yes, that, I guess, would be
20 helpful if we can make an extra copy because we need to give
21 them a copy of the tape for the weekend, so that it can be
22 reviewed.
23 THE COURT REPORTER: Okay.
24 MS. PAPUSHKEWYCH: Your Honor, if --
25 THE COURT: In other words, I do not know how long one
475
1 of these tapes run, but you ought to have two or three fresh
2 tapes that we can use strictly for the sealed portion.
3 MS. PAPUSHKEWYCH: Your Honor, we have a suggestion in
4 terms of maybe helping out with this issue.
5 THE COURT: Okay.
6 MS. PAPUSHKEWYCH: Mr. Flaxman had called two
7 commissioner personnel, Glenn Carr and Deputy Commissioner of
8 personnel Bob Joyce. He wants to call them in his case. Those
9 two witnesses are available currently, and I think Mr. Flaxman
10 said he would proceed with them at this point.
11 MR. FLAXMAN: Sure.
12 MS. PAPUSHKEWYCH: And, then, maybe we can take this
13 whole issue of subject matter experts --
14 THE COURT: Well, then we can break for lunch --
15 MS. PAPUSHKEWYCH: Right.
16 THE COURT: -- is what we're saying, and then have an
17 opportunity to get what tapes may be necessary.
18 Okay.
19 MS. PAPUSHKEWYCH: Okay.
20 THE COURT: All right.
21 Should we have a short break at this point and then --
22 MR. FLAXMAN: I am ready to go, unless we need to
23 break.
24 THE COURT: Do you want a break?
25 MR. FLAXMAN: We should have a short break.
476
1 THE COURT: In order to get all set up. Let us take a
2 -- try to hold it to five minutes? Short recess then at this
3 time. Thank you.
4 (Whereupon, a recess was taken.)
5 * * * * *
6 I certify that the foregoing is a true and accurate transcript
of proceedings in the above-entitled matter.
7
8 ________________________________ __________________, 1996.
9 Pages 421 through 476 reported by:
Joseph A. Rickhoff, CSR, RPR
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
Next Transcript
Main