This project evaluation report details and evaluates the final stage of the three-year project, “Strengthening Integrity and Combatting Corruption in Higher Education in Armenia”. It focuses primarily on the deliverables that became the main outputs of the project, and uses interviews with project participants, experts, and stake-holders as a main source of information in determining the outcomes and impacts of the project, supplemented with evaluation of the materials themselves.

The project began in 2015 with a detailed needs assessment and resulting report, followed by a series of detailed information-gathering events and activities at which solutions to the needs previously identified were proposed and discussed. Rising to the top of the proposed solutions were a model code of practice for ethical conduct in higher education and a suite of toolkits designed for institutional self-study in the areas of Human Resources Management, Curriculum Development and Student Assessment, and Higher Education Governance. These resources were developed, tested, and revised in several iterations and eventually tested in their current form during the current project year. A robust series of workshops, trainings, conferences and activities were also developed and delivered in support of the major projects. A series of mini-grant projects focusing largely on student engagement with pro-integrity efforts was also undertaken as part of the effort to raise awareness of anti-corruption efforts.

Feedback on these projects was overwhelmingly positive from stakeholders ranging from students to subject matter experts, with the most frequently voiced concern being that the project deliverables, particularly the toolkits, already seen as exerting positive pressure toward greater transparency, might not be continued and would thus waste their potential to bring about positive change. Students, staff, faculty, and top-level administrators support the continued utilization of these tools as a way to operationalize the principles anticipated in the draft legislation, as well as to help Armenia increase its standing relative to other European states.

Project Overview

Context

This particular moment is unique in terms of providing an unprecedented opportunity to promote increased transparency in Armenian institutions of higher learning for a number of reasons including recent efforts to join high-profile international compacts and organizations (GRECO, 2004, Conference of the States Parties to the United Nations Convention against Corruption, 2007, and the Open Government Partnership, 2012). The Armenian Government and Ministry of Education have become mobilized to fight corruption on many fronts, including corruption in higher education and as part of these efforts, the need has arisen for projects designed to improve transparency and accountability, that can have real effects, thus bringing higher education policies and practices better into alignment with more stringent international standards while at the same time improving the quality of education for Armenian citizens. One facet of these efforts is the new draft Law on Higher Education, however parallel institutional policies and procedures are needed in order to assist institutions in meeting the anticipated requirements of the law in order for the law to have the desired effects. This project addresses those needs.

Project Activities:

Year One: Fact Finding, Agenda Setting:

The project, “Strengthening Integrity and Combatting Corruption in Higher Education in Armenia” began in January 2015, with an overall objective of increasing the general level of integrity and decreasing incidences of corruption in higher education institutions across the country. The initial step toward that goal was a needs assessment study that included interviews with a wide range of stake-holders and culminating in a publication published in April, entitled, “Risk analysis of issues affecting the integrity of the Armenian Higher Education system in Armenia” (https://rm.coe.int/16803073f5). The study identified several specific target areas to be addressed, many of which had to do with establishing standards and procedures for ensuring greater transparency in higher education institutions, particularly regarding financing, hiring practices, student assessment, and other areas vulnerable to corruption. Work on the project continued on dual fronts, with one focused on drafting a code of ethics while the other focused on practical implementation. Work on the latter included a conference held in November in Tsaghkadzor, where educators, administrators, representatives from the Ministry of Education, and students gathered to deepen shared understandings of the integrity-related challenges facing Armenian higher education institutions and to identify and discuss practical goals for moving the project forward, including discussions related to implementing existing regulations; increasing student empowerment and involvement; increasing uniformity in reporting; leveraging quality assurance as a way to combat transgressions; and developing strategies to increase transparency of procedures at higher education institutions. Working-group meetings continued throughout the remainder of the year on both fronts, with major plans for year two focusing on the development of a code of ethical conduct appropriate to the Armenian context and the creation of “tool-kits” to be used to assess, document and analyse data on transparency and ethical accountability at institutions of higher learning.

Year Two: Team and Capacity Building, Refining, and Testing

Work toward achieving the goals identified in year one continued as additional stakeholders and experts were brought together in round-tables, interviews, and focus-groups to provide feedback and suggestions to improve both the ethical guidelines and the tool-kits that were taking shape, particularly with respect to adapting them specifically for the Armenian context. The toolkits were piloted in six universities and a study visit was conducted to observe best practices in ethical practice, transparency, and accountability in the Netherlands. Subsequent revisions to both the toolkits and ethical guidelines were made, and the resulting versions were submitted to the Ministry of Education for feedback and comments. After receiving positive commentary from the Ministry, the toolkits and guidelines were shared with participants in another conference, held in December of 2016, at which representatives from twelve Armenian universities learned about and discussed the newly developed documents, “A Code of Practice for Ethical Conduct in Higher Education Institutions” and the three toolkits, “To Enhance Transparency and Accountability” in “Governance”; in “Human Resources”, and in “Curriculum Development and Student Assessment”, respectively.

Year Three: Instruction and Implementation; Awareness, and Assessment

The current year’s activities focused upon putting the products of previous years’ work into practice as effectively as possible. This required an intensive series of professional development activities to help ensure informed, uniform implementation of the materials across a wide variety of institutions. Training sessions and workshops were organized for representatives of each of the institutions participating in the roll-out of the three toolkits, after which, the toolkits were fully implemented, querying universities as to their current policies and practices. Training was also provided for the institutions involved in developing codes of ethics based upon the model code to assist them in their own development efforts. To further heighten awareness of the issues regarding corruption in higher education and the need to address it, a call for proposals for projects to be funded by micro-grants was put out, calling for projects designed to promote integrity in higher education. Another conference, with the explicit purpose of raising awareness of both the issues and the newly developed tools to help address them, was also conducted.

Targeted Stakeholders

This project aimed both to serve and to involve stakeholders at a number of different levels ranging from policy-decision makers in the Ministry of Education to current students at higher education institutions. Particular attention was paid to those who would be most directly involved in developing institutional codes of ethics, those directly responsible for designing and conducting surveys and assessments with the toolkits, and administrators overseeing the evaluations and making future policy and procedural decisions based upon the information that the toolkits revealed.

Stage(s) of Development

This is the final year of an initial three-year project in which the promised deliverables have been developed and deployed. It should be noted that to obtain maximum benefit, this project should be continued and expanded in order both to gain a more complete picture of transparency in governance, human resources, and student assessment across the full range of Armenian institutions of higher learning, and to gather longitudinal data in order to measure progress over time.

Project Outputs

Major outputs of the project include the following resources, documents and publications:

· Online Reporting Platform and Toolkit to Enhance Transparency and Accountability in Governance of Higher Education Institutions of Armenia

· Toolkit to Enhance Transparency and Accountability in Human Resource Management of Higher Education Institutions of Armenia

· Toolkit on Enhancing Accountability and Transparency in Curriculum Development and Student Assessment

· A Model Code of Practice for Ethical Conduct in Higher Educational Institutions

· Risk analysis of issues affecting the integrity of the Armenian Higher Education system in Armenia

· Multiple conference devoted both to raising awareness of issues and to developing resources in response

· A robust series of workshops for those involved in utilizing the toolkits in both the first pilots the subsequent revised (current) versions of the surveys.

· A series of workshops and expert guidance for the working groups responsible for developing Codes of Ethics and a set of relevant policies for the universities

· A micro-grant program designed to draw attention to anti-corruption issues, particularly with respect to attracting student participation and engagement which yielded four additional independent projects:

· A “hackathon” and accompanying social media campaign;

· A handbook addressing student academic integrity challenges such as diploma mills and the selling of fraudulent qualifications;

· A survey and handbook on ethical violations common in Higher Education Institutions;

· A social media campaign addressing integrity in academia.

Outcomes

Intended outcomes

(as expressed by project leaders, experts, and key participants)

1. Heightened awareness amongst all stake-holders of the manifestations, costs and risks associated with corruption in higher education;

2. Increased knowledge, competency and agency regarding strategies for combatting corruption on the part of all stakeholders ranging from students to high-ranking administrators and Ministry officials;

3. Increased understanding of, and appreciation for codes of ethical conduct within university hierarchies; increased willingness and capacity to develop and adopt clear, transparent, fair, and practical ethical guidelines for university communities;

4. Active, willing participation on the part of university leadership and their designees in collecting and submitting initial survey information, as facilitated by the three toolkits, and commitment to working toward greater transparency and accountability within a defined and usable framework;

5. Usability of the toolkits that allows them to be used by all higher education institutions across Armenia;

6. Compilation of data that leads to greater understanding of institutional vulnerabilities and acts as a catalyst for improvements in transparency, accountability, fairness, and ultimately improvements in the quality of higher education in Armenia;

7. Increase in awareness of progress toward greater transparency and fairness, leading to a positive reinforcement cycle of social change;

Evaluation of Objectives Achieved:

Interviews

Evaluation of the degree to which objectives were met based on interviews with participants including members of project teams, participants in workshops and training activities, online resource designers, student participants, and subject-matter experts.

Evaluation Item 1: Was the overall understanding of issues related to corruption in higher education increased and improved amongst various project stake-holders? (Related to Objectives 1 & 2)

“At the beginning of the process, our conception of the idea of corruption was very narrow. Now I see corruption as any violation of principles or of ethics.”-Interview respondent

Representatives of each stakeholder group interviewed (listed above) reported that while they had been aware of the presence of corruption in higher education, their understanding of the scope and nature of the various forms of corruption, and of its harmful effects had been limited prior to their participation in this project. Typical respondents noted that while obvious forms of corruption such as payment for grades (explicitly or implicitly), or payment of bribes in exchange for favourable HR actions were known to occur, many other, less obvious forms of corruption such as failing to advertise open positions and instead simply hiring friends or relatives had not been perceived by them as corruption, or indeed as harmful to the quality of education until this project raised their awareness. Other respondents noted that while they had been vaguely aware that such things happened, they had never considered the detrimental effect on the overall quality of instruction that practices such as hiring less-qualified applicants might have. Particularly notable among the respondents to this question were the students who voiced their desire to have teachers who were selected based on their merits and qualifications. Another notable group of respondents to this question were employees in Human Resources, who noted that they appreciated guidance that they could use to justify making hiring decisions based on merit rather than solely upon recommendations. This participant noted that as they are transitioning from “Staff Management” in which their role consisted mainly of documentation, to a Human Resources model with increased opportunities for interventions, this project prompted her to think “more about what is possible” rather than simply doing things as they “have always been done” with regard to tolerating corruption. A student noted having been asked by a teacher for a copy of a paper, and knowing it was wrong, but not thinking of it as corruption. Both representative within the HR professional and student groups remarked upon the empowering effects of having learned more about the various manifestations of corruption.

Evaluation Item 2: Did participants receive actionable information, training, tools, and/or strategies to help them carry out their professional responsibilities with greater transparency and integrity? (Related to Objectives 2, 3, 5 & 6)

“As a follow up of one of the workshops, we had a request to use the guide to student assessment in the development of course materials, with an aim to bring the standards they discussed into reality. They are also talking about some of the topics in curricular discussions.” -Interview Respondent

The vast majority of those interviewed who had been part of the reporting teams (including supervisors) using the toolkits noted that they were not just informative, but illuminating—pointing the direction that they want to go. One noted that answering the questions brought about the realization that “we were at the starting point” with considerable room for improvement, and another noting, “when I saw all the answers that were ‘no’ I didn’t like it; I wanted us to do better.” This is a key success for the project as it provided not simply a snapshot of where institutions currently rate, but also a roadmap for how they can improve. Interviews with the development team confirmed that this was an intentional part of the design—not simply to provide data, but to provide motivation to improve, and concrete benchmarks that were designed to function as goals. An interview respondent noted that the Arminian State Pedagogical university already has announced plans to change their new reporting system in response to this work, creating a modified form of the survey for their own use. Another respondent noted that one of the upper level administrators was using the question as “his to-do list for next year”.

Perhaps the most direct and useful assessment of the utility of the toolkits was this respondent’s comment (paraphrased minimally): Before, even if institutions wanted to be transparent, they often lacked the framework to collect and organize information. They had reports, but they weren’t searchable or comparable, or even well organized. Now they know how to collect and extract the relevant information so can it can be readily understood and “digested” both internally and externally. Now they have a way to know, and to show other universities, what good governance looks like.

Evaluation Item 3: Did this project inspire participants to become active participants in the fight against corruption in higher education? (Objectives 1, 4, & 7)

“After the cascade training where we used the materials provided by this project, and the previous quality assurance training, we have people who are now asking for more of this training—they want to bring it from just being an isolated training to institutional practice. They see it as all of the various processes impacting the will to attack corruption.” -Interview Respondent

“Development of the toolkit has surpassed what was expected. Very solid. The adventure of the project because this was one of the first project in this area. In some ways it surpassed ETINED so far because they had such concrete effects. Also the country can be proud because they are leading the efforts.” –Project Expert

“We are saying to our students, you have to demand from us quality teaching” -Interview Respondent (Instructor)

This was perhaps the most noteworthy and praise-worthy aspect of this project—the degree to which it seemed to inspire and even excite the participants to want to become more active in the fight against corruption. In each reference group, there seemed to be at least one person whose experience could be summed up as, “before this project, I knew that corruption existed, but I had no way to fight it; Now that I see there is a way, I want to be involved.” This is a credit to the project design teams who balanced demands on participants with support, split the workload into manageable “chunks” and included opportunities for the teams and stakeholders to come together so that they could feel the degree to which they were not working in isolation but instead were part of a group effort. While several participants noted the additional workload, they reported that while it had at times been burdensome, it had ultimately been well worth the effort. Many voiced the desire to see the program go forward and their willingness to continue. Several explicitly stated that it would be a great shame if, after so much progress, the program would not go on.

Outputs

Evaluation of the degree to which objectives were met based on appraisal of project outputs supplemented with insights from output design team members

Evaluation Item 1: Did the outputs that emerged as the main deliverables of the project meet the needs identified in years 1 and 2 of the project?

One of the first main goals identified over the course of initial needs assessment and subsequent meetings with stake-holders was to create a model code of practice for ethical conduct that was simultaneously based upon best practices and also responsive to the specific Armenian context. The document produced meets that need by outlining the principles and values first, and then outlining the guidelines that are based in those values. The final section that guides higher education institutions through the process of developing their own codes is particularly useful in that it lists the necessary components while leaving room for each institution to customize in such a way as to reflect the priorities and main goals specific to its own unique character. This guide also succeeds by being thorough without unnecessarily voluminous, readable, and clear.

The other main output considered is, of course, the set of three toolkits, designed not only to evaluate transparency and accountability in HR management, curriculum development and assessment, and higher education governance, respectively, but also to provide an impetus for positive change. It is noteworthy that the toolkits, while obviously designed to combat corruption, express that aim in positive terms—enhancing what is positive rather than fighting what is negative. That spirit pervades the project and helps explain the positivity with which those interviewed described it. Clearly, each of these toolkits is a response to a particular need identified in years one and two, refined via feedback to include guidance on difficult issues as identified and articulated by stakeholders. Particularly impressive in terms of utility is the HR Management: Strategy and Action Plan, which excels at both organization and motivation by providing 1, 2, and 3 year benchmark targets.

Were the above-mentioned outputs sufficiently well-supported?

Interviews both with people who attended the training workshops and those who conducted them suggest that they were planned thoughtfully and carried out in such a way as to maximize the value of the associated activities. The explanation of terms contained in the toolkits was seen as particularly valuable, as it was reported that there were initially a wide range interpretations of many of the terms used therein. Several comments from those engaged in carrying out the toolkit surveys mentioned working as a team, working together, and that being able to divide the workload into section made it seem manageable. Trainings were also mentioned as at which the toolkits were viewed as resources to bring about positive and desired changes in institutional procedures.

Did the auxiliary activities such as the mini-grant program enhance the overall effectiveness of the project?
Students reported positive experiences with the hack-a-thon, both in terms of increased empowerment and in the opportunity for involvement with fellow students interested in improving the quality of their education. While they noted that they are a self-selected group of student leaders that may not be fully reflective of the student body at large, they also noted that the experience of confronting corruption in their own environment was a powerful experience. While that alone would be notable, the piloting of a social-media program focused on academic integrity and the creation of a handbook were also significant experiences that may have potential for duplication and expansion in the future.

Alliances, alignments, sustainability

Among the most frequently expressed positive remarks about this project was that it brought together people from various parts of universities and gave them the opportunity to work on an issue of common concern. Put another way, one interview respondent said that it would be terrible if the project were not picked up for continuation, as “it would leave us all alone again.” Multiple students also cited the people they worked with on projects such as the hack-a-thon as the best part of the experience. Particularly when facing challenges as daunting as corruption, teamwork is essential both to the project itself and to the resilience of those carrying out the tasks. Throughout the project, networking seemed well-thought out, however until future efforts are solidified, opportunities for networking seem uncertain.

This project aligns unusually well with the current political objectives of the country, but also very well with institutional efforts at improvement. Especially with the number of higher education institutions in Armenia presently, it may be quite possible to harness competitive impulses in such a way as to utilize integrity as a “selling point” and focus for competition. This project has come at a perfect time to work in synch with the new draft law, hopefully each being responsive to the other in ways that benefit both.

As to sustainability this project has not only enormous potential, but also a significant number of well-placed allies. Momentum at the institutions which piloted the tool kits is clearly favourable; Desire for greater transparency and accountability has been piqued. As one respondent put it, “It would be a great shame for all of this time, money, and effort to be wasted. This is, for once, a project that didn’t go straight to the shelf.” Particularly since the electronic resources already reside in an accessible database, the potential for sustainability is quite high, and this particular confluence of circumstances—the new draft law, the external pressure and internal will—may never come again. Of particular note regarding sustainability: Several interview respondents in different groups voiced a request for conferences around the country, with one conversation leading to the question of whether it might be possible to have a series of smaller conferences several times a year in various locations that were broadcast, so even remote universities might get a chance to participate face-to-face, and all universities could benefit from the presentations remotely. The opportunity to engage people who are currently excited about combatting corruption seems like it ought not be missed.

Room for improvement and lessons learned

Most suggestions for improvement had to do with the mechanics of the survey that can easily be addressed by usability studies and revision. Other points worth mentioning include suggestions to budget compensation for personnel on whom the more burdensome work fell, and requests for more advanced scheduling and better communication regarding the student projects. Students reported that they had more ideas than they had time for because of the short deadlines, and also that they thought more students would have participated had they known in advance. Multiple people mentioned that a change happened midway through the project that increased the bureaucratic burden on external personnel, and that it not only made things more difficult, but also incurred some resentment on the part of those who were already working beyond their “billable” hours. A minor question for future revision was more parallel structure in the titles of the documents, which currently read “Transparency and Accountability” for two of three guides, but “Accountability and Transparency” for the third. All of these are minor compared to the successes of the project, which are considerable.

Conclusion: Overall Assessment and Recommendations

This project has already succeeded not only in collecting information regarding transparency and accountability in higher education, but also in inspiring people to become advocates for integrity, giving them models upon which to base their own unique institutional codes of conduct, and providing benchmarks for future improvements. Project participants report great levels of satisfaction with their accomplishments, despite having been challenged with difficult questions and sometimes voluminous workloads. Participants uniformly expressed the opinion that this work should be continued—even though most received no compensation for doing it. As many are already working to improve their scores on the toolkit evaluation surveys there is considerable momentum that, if leveraged, could result in significant improvements to the Armenian higher education system and help move the country closer to meeting its strategic national goals.