Communication F2017-45 None

•v. ' ..-v.'-.i ."1:"-", Office of Inspector General City of Chicago ■j. mm l;f|l .... ■l. ''-'ii r ■ *' '(!■■;■ i OFFICE OF INSPECTOR GENERAL ADVISORY CONCERNING ALLEGATIONS OF CHEATING ON THE 2015 LIEUTENANT PROMOTIONAL EXAM '■..■J-tiS: j;h-^;/!^:;v.^::i<:y:.^; ■ (;_. .1- ' l*rSJii. . -r^ --i:^-7V->.-|->3...V^' '.-e'.-.-J * ■ ■ fi' • ■ hmUam v.' •.'.■"...': -'V< (■>. =" W'vi. j".::"'1 ■ cr Z3LZ ro ro T«^,;:.'-.V.i:?---J >_f:>-'.- : "$.n 866-IG-TIPLINE (866-448-4754) www.chicafioinspectorgeneral.org ro XT >•->' "'V.' Joseph V1. Ferguson Inspector General OFFICE OF INSPECTOR GENERAL City of Chicago 740 N. Sedgwick Sired, Suite 200 Chicago. Illinois 60654 Telephone: (773) 47X-77W Fax:(773)478-3949 VIA ELECTRONIC MAIL May 16,2017 Eddie Johnson Superintendent Chicago Police Department 3510 South Michigan Avenue Chicago, Illinois 60653 Dear Superintendent Johnson: We write to bring to your attention concerns regarding control issues in test administration. We identified these concerns during the course of our investigation into allegations that Chicago Police Department (CPD) members received confidential exam material pertaining to the 2015 lieutenant promotional exam (hereafter "Exam"). I. OIG Investigation into Allegations of Cheating on the 2015 Lieutenant Promotional Exam In December 2015, OIG received several anonymous complaints alleging that, over a year earlier, a CPD Chief held study groups for the Exam and leaked Exam materials. In addition to these anonymous complaints, OIG received complaints from two sergeants also alleging that the Chief held an invite-only study group for the Exam and that members of this study group had close personal relationships with the Chief and other CPD command staff. Neither complainant had firsthand knowledge of the allegations. In the course of its investigation, OIG interviewed 20 individuals, including the Chief, employees of the vendor that administers the Exam, a former BIA employee, several study group participants specifically named in the complaints, and other Exam participants. The 20 interviews also included employees of an off-site location where the named complainants studied on the weekends and CPD employees whose work location was outside a study room at CPD Headquarters where the named subjects studied. No first hand witnesses or accounts of cheating emerged prior to or during the course of OIG's investigation. OIG reviewed approximately 300,000 emails from 33 individuals' City email addresses, and conducted searches through approximately 600,000 files on the Chiefs hard drive to locate relevant documents. OIG used broad search terms, including terms pulled from each Exam Website: www.cliicagoinspcctorgeneral.org OIG File fi 16-0042 May 16, 2017 question, in its email request to CPD. OIG also applied a second set of search terms, chosen directly from each question ofthe Exam, to the emails ofthe Exam's top scorer. The Exam consisted of two parts, the first administered in June 2015 and the second in August 2015. After the Exam was scored, the testing vendor, CEB-SHL Talent Management Solutions, contracted by the City to develop and administer the Exam, analyzed the results and found no statistical anomalies. The scores formed a normal distribution with no outliers. OIG also conducted its own test analysis based on the raw data CEB-SHL provided. OIG analyzed the Exam scores by question, per Exam participant, for each part of the Exam, to determine whether any patterns existed indicating similar behavior among Exam participants. OIG also inputted the scores of 21 Exam participants, including the sergeants named in the complaints, into quantitative tables and graphs to depict the proximity of scoring among those 21 candidates. OIG met with the Exam testing vendor to discuss the results of the analysis. Finally, OIG transcribed the audio-recorded answers to the oral section of the Exam for 15 individuals, including sergeants named in the complaint as well as randomly-selected test takers, for comparison. OIG analyzed the transcriptions for word choice and phrasing patterns. Overall, the analyses did not reveal any trends supporting the allegations of fraudulent behavior. While OIG's investigation developed no evidence of cheating, OIG identified a number of concerns related to CPD's exam process. OIG's findings are summarized as follows. A. Lieutenant Exam Development Process and Test Information Prior to the 2015 lieutenant exam, CPD last administered the lieutenant exam in 2006. CPD promotes sergeants to lieutenant based on the eligibility or "rank order" list and the merit promotional process. Eligibility lists are retired at the discretion of the Superintendent, with notice to the Department of Human Resources (DHR) Commissioner, but must be used for at least one and no more than six years, unless there is a lack of available funds for testing.1 In September 2013, DHR contracted with CEB-SHL to develop the Exam. First, CEB-SHL worked with 50 to 60 lieutenant Subject Matter Experts (SMEs) to create a lieutenant job analysis, which was a list of job duties and the knowledge areas needed to complete the duties. CEB-SHL used the job analysis to make an outline of the topics the test should cover. CEB-SHL requested that CPD provide lieutenant SMEs that were demographically representative of the department. Next, a group of twelve SMEs with the rank of captain, commander, or executive officer worked with CEB-SHL to create Part I of the Exam—the 81 question, multiple-choice Job Knowledge Qualifying test—which assessed job knowledge and the application of that knowledge required of lieutenants at job entry. The SMEs based their questions on the following: (1) items identified from previously administered exams; (2) items found in new or updated resource materials (General Orders, Special Orders, Legal Bulletins); and (3) examples of situations lieutenants face in the position. The SMEs then grouped the items into review sets and presented them to different SME teams for review. Then, a group of five senior management SMEs reviewed all of 1 Chapter III. 15 ofthe CPD Hiring Plan addresses retiring eligibility lists. CPD continued to use the eligibility list from the 2006 lieutenant exam through November 15, 2015, while it prepared for the 2015 lieutenant exam. Website: vvww.chicagoinspectorgencral.org OIG File H 16-0042 May 16. 2017 the items developed. The SMEs prepared more questions than were actually used on Part I of the Exam. Only the Chief undertook final review ofthe Exam in the exact format it would be given on the day of test administration. CEB-SHL developed Part II of the Exam using a similar process. Part II consisted of 15 short answer questions and fill-in-the-blank questions and four oral questions that measured a candidate's ability to verbally respond to issues. CEB-SHL divided nine SMEs into small groups to generate specific situations and related materials for the written exercises, from which test questions were developed. Then, five senior SMEs reviewed and approved all materials related to the assessment exercises. However, as with Part I, only the Chief had final review of these sections. All SMEs, including the Chief, signed a confidentiality agreement, which states, "I agree not to assist in the development of any study guide or participate in any study group to prepare candidates for the Lieutenant Promotional Process." SMEs worked with CEB-SHL on the Exam at the CEB-SHL facility. CEB-SHL collected all material used during the test development process after each session, and SMEs were not permitted to remove any information from the facility. No SME was provided any copies of testing materials to take home and the confidentiality agreement states, "I will not make or retain copies of any job analysis or test-related materials." SMEs were not required to sign a conflict of interest or personal relationship disclosure form. CPD announced the Exam on October 17, 2014, making available a Job Knowledge Written Examination Orientation and Preparation Guide ("Study Guide") via a DHR web link within the announcement. The Study Guide served to help candidates prepare for the Exam by providing them with an introduction to the promotional process, including an overview of Exam components. The Study Guide provided sample questions and answers for each part of the test, and included the specific source material for answering those questions, as well as examples of the type of materials that would be included in the Reference Materials Booklet provided to candidates on Exam day. The Study Guide also included appendices containing the Recommended Reading List, which listed all of the written source materials for the Exam." CEB-SHL developed the Study Guide based on discussion and review with SMEs and other CPD senior command personnel. Ten days after the Exam was announced, CPD HR emailed those sergeants registered for the Exam instructing them to submit questions and concerns regarding the Study Guide to HR@chicagopolice.org' On December 5, 2014, CPD HR released an updated version of the Study Guide. The purpose of the update was to address questions from candidates about department directives that had been recently rescinded or were undergoing changes at the time the original Recommended Reading List was published. On July 1, 2015, CPD HR made the Candidate Preparation Guide for the Written and Oral Assessment Exercises available through a CPD Administrative Message with web link information. 2 This list included department directives, sections of the Illinois Compiled Statutes, and the Municipal Code of Chicago, among other sources. Wcbsitc: vvww.chicauoinspectorgencral.org OIG File # J 6-0042 May 16, 2017 Although SMEs were involved in all test-related development, CEB-SHL did not finalize the scoring guidelines until after CPD administered the test. Qualified individuals, outside of CPD, that CEB-SHL hired, trained, and monitored, scored the Exam. These individuals were never in contact with any CPD SMEs, and signed confidentiality agreements that they would not share information about the test scoring and were not. related to anyone in CPD. B. Study Groups CPD members preparing for qualifying exams often form study groups. Many groups studying for the 2015 lieutenant promotional exam were organized and led by current or former CPD command officers and were open to all CPD sergeants. Many CPD members also paid to attend third-party test preparation courses, some of which were taught by former CPD personnel. The sergeants named in the complaints to OIG formed their own study group in late 2012 or early 2013, before the 2015 lieutenant promotional exam was announced. Several of the sergeants OIG interviewed stated they began studying before the announcement due to speculation throughout CPD that a promotional exam was coming soon. Sergeants in the study group stated that, initially, they sent an email to people they knew through work or professional organizations to see who was interested in forming a study group. The group was not closed, and anyone who wanted to join was able to do so. Those interested attended an initial organizing meeting at CPD Headquarters and then later met offsite every other Saturday until the Exam was announced, and thereafter every Saturday. According to a study group member, the initial organizing meeting had approximately 50 participants, but the study group dwindled to approximately 17 regular participants. During the study group, the participants reviewed the CPD General Orders and answered practice questions prepared by the members. Before CPD announced the Exam and provided the test takers with the Study Guide, the study group participants appeared to base their review on the 2006 lieutenant exam and 2014 sergeant exam reading lists. A smaller group of five sergeants in this study group decided to study on their own in addition to attending the weekend offsite sessions. They studied at around 6:00 p.m. on various weekdays at CPD Headquarters in a third floor conference room located in the Finance Department. The Chiefs office shared a wall with this conference room. The Chief acknowledged seeing this group studying once or twice, but only exchanging pleasantries with them. Sergeants named in the complaint had conflicting recollections regarding any interaction with the Chief. Although all stated that no information regarding the Exam was discussed, they acknowledged that the Chief was aware that the group was studying for the Exam. Certain members of this smaller study group indicated they were acquainted with the Chief through work or their shared professional organization. The members stated that they did not socialize with the Chief outside of work or organization events. Three members of this smaller study group scored among the top 25 out of 601 test takers. When the study group participants had questions about the Study Guide, they emailed HR@chicagopolice.org , as directed by CPD HR's email on October 27, 2014. No sergeant interviewed recalled receiving a response to his or her questions from CPD HR, and some sergeants expressed frustration about the lack of communication. In two instances, a sergeant in both the Saturday and the smaller weekday study groups, while aware that the Chief was an Website: vyww.chicatroin.spcctoracneral.org OIG File U 16-0042 May 16,2017 SME, blind copied the Chief and the other members of the Saturday study group on emails to the CPD HR address, listing concerns about incorrect items in the department directives. OIG identified no evidence that the Chief responded to either of these emails. In another instance, that same sergeant forwarded questions the sergeant sent to the CPD HR email address to the Chief only. The Chief acknowledged receipt, but OIG's investigation revealed no evidence of further communication. The sergeant stated that the Chief was included on the emails because the sergeant thought the Chief was the departmental person in charge of dealing with concerns involving the Exam, although the sergeant acknowledged this was not based on any instruction from the test vendor or CPD. The sergeant could not recall why the Chief was included on the two emails as a blind carbon copy. While email review shows that a few sergeants in the smaller study group met with the Chief on several occasions regarding events hosted by a professional organization to which they all belonged, OIG's investigation revealed no evidence that the Chief was involved in any way in any study group. OIG identified no evidence that the Chief provided confidential information to sergeants taking the Exam. Multiple emails among study group members showed an active and regular discussion of study materials and practice questions. C. BIA Complaint On November 6, 2014, seven months before the Exam was administered, BIA received an anonymous complaint alleging that the Chief was holding a private study group for the Exam that included select sergeants on Tuesdays at 6:00 p.m. in the Chiefs conference room. BIA investigated the complaint by observing various conference rooms around CPD headquarters around 6:00 p.m. on two Tuesdays. The BIA investigator did not see a study group. The BIA investigator determined there was no evidence to justify continuing the investigation and recommended that it be closed. BIA closed the case as unfounded in January 2015. At no time after November 6, 2014, did BIA notify OIG Hiring Oversight of this complaint. Instead, OIG received complaints directly in December 2015, months after the Exam concluded, and over a year after BIA received the original complaint, as described above. II. OIG Recommendations While OIG's investigation did not reveal evidence to support the allegations of individual misconduct, OIG did identify several control issues respecting CPD's process for administering the Exam and investigating allegations of cheating. First, interviews and email review indicate that the Chief was charged with granting final approval of the Exam. In that role, the Chief saw all items developed for the test, including all of the questions that appeared on the test. OIG suggests that CPD consider other methods for ensuring the suitability and confidentiality of exams, such as using an outside individual/vendor to review the final version of exams or having an internal CPD representative approve a larger pool of questions from which the final test questions are then selected by the vendor. As long as CPD allows one of its members to have access to the final version of a test, its integrity is vulnerable to charges that the CPD member leaked questions. Website: www.chicairoinspcclorgeneral.on > OIG File # 16-0042 May 16, 2017 Second, while SMEs were required to sign a confidentiality agreement stating they would not prepare candidates for the Exam, there was no requirement to sign a conflict of interest or personal relationship disclosure form to disclose relationships with any potential candidates. OIG recognizes that SMEs are likely to have professional relationships with a large number of test-takers, but suggests that CPD implement a disclosure for relationships that go beyond professional. Third, OIG suggests that CPD make transparent how questions and concerns sent to the CPD HR email address will be handled. Sergeants taking the Exam were instructed to email HR@chicagopolice.org if they had any questions or concerns about the Study Guide. One sergeant noted that they never received any response to inquiries sent to this email address, and there was confusion among sergeants regarding the purpose of the email address. CPD HR should also make clear to test takers that they should not contact SMEs directly regarding test content. Finally, the original November 6, 2014 complaint sent to BIA should have been immediately forwarded to OIG, per the requirements of the City of Chicago Police Department Hiring Plan for Sworn Titles. The Hiring Plan states the following: Any CPD employee who knows or should know that a Hiring Manager, Hiring Authority, or other City or CPD employee may have allowed Political Reasons or Factors or other Improper considerations to influence any term or aspect of employment shall report the incident to IGO Hiring Oversight directly and without delay. Any CPD employee who knowingly fails to report such a potential violation to IGO Hiring Oversight may be subjected to discipline, up to and including termination. See Chapter 2, Section A. Further, the Memorandum of Understanding (MOU) attached to the Hiring Plan states that, if BIA receives a complaint concerning an employment action, then: OIG Hiring Oversight, shall in consultation with BIA, determine whether the matter warrants administration as a compliance issue or an investigation, and inform BIA whether OIG will act on the complaint, or refer it for action at the discretion of BIA. ... In the event the investigation of an employment action is conducted by BIA, BIA shall advise OIG Hiring Oversight of BIA's findings and any disciplinary action taken, and copies of records and information concerning the investigation shall be made available to the OIG upon request. See MOU \ 2, 2.b. OIG recommends that CPD remind its members that any hiring related complaint should be immediately forwarded to OIG Hiring Oversight for review and conduct follow-up training if necessary on the City's Hiring Plan and the MOU requirements. We ask that you inform us of any action CPD takes in response to these recommendations by June 15, 2017. Your response will be included in the summary of this advisory to be published in the next OIG quarterly report. Website: vvww.chicagoinspcctorgcneral.ori; OIG File #16-0042 May 16,2017 Respectfully, Joseph M. Ferguson Inspector General City of Chicago cc: Joe Deal, Chief of Staff, Office of the Mayor Edward Siskel, Corporation Counsel, Department of Law Soo Choi, Commissioner, Department of Human Resources Website: www.chicagoinspcclorgcneral.org City of Chicago Office of the Inspector General Public Inquiries Danielle Perry (773) 478-0534 dperry(a).chicagoinspectorgeneral.org To Suggest Ways to Improve City Government Visit our website: improve-city- government/ To Report Fraud, Waste, and Abuse in City Programs Call OIG's toll-free hotline 866-IG-TIPLINE (866-448-4754). Talk to an investigator from 8:30 a.m. to 5:00 p.m. Monday-Friday. Or visit our website: waste-fraud-and-abuse/ Mission The City of Chicago Office of Inspector General (OIG) is an independent, nonpartisan oversight agency whose mission is to promote economy, efficiency, effectiveness, and integrity in the administration of programs and operations of City government. OIG achieves this mission through, administrative and criminal investigations; audits of City programs and operations; and reviews of City programs, operations, and policies. From these activities, OIG issues reports of findings and disciplinary and other recommendations to assure that City officials, employees, and vendors are held accountable for the provision of efficient, cost-effective government operations and further to prevent, detect, identify, expose and eliminate waste, inefficiency, misconduct, fraud, corruption, and abuse of public authority and resources. Authority The authority to produce reports and recommendations on ways to improve City operations is established in the City of Chicago Municipal Code § 2-56-030(c), which confers upon the Inspector General the following power and duty: To promote economy, efficiency, effectiveness and integrity in the administration of the programs and operations of the city government by reviewing programs, identifying any inefficiencies, waste and potential for misconduct therein, and recommending to the mayor and the city council policies and methods for the elimination of inefficiencies and waste, and the prevention of misconduct.