STEP: How did the idea of this study come about?

The idea for the article actually originated from Nature’s Editors late last year when HEC’s budgets and utility were being increasingly questioned by the new government in Islamabad. Nature had been an observer – and sometimes a commentator on HEC’s initiatives – and approached me to see if I’d be interested in writing an independent and objective piece on Pakistan’s Higher Education Reforms. I worked with Nature’s editors to put together a group of authors who we believed brought a diversity of experience, insight, and credibility to this exercise.

This preliminary review, I hope, will be a starting point towards an extensive process of external policy peer review and a constructive self-examination leading to greater effectiveness of higher education policies and programmes in Pakistan. Another very important aspect of Nature’s interest in this work was to see if other countries contemplating similar reforms could learn some lessons from Pakistan.

STEP: How long did the study take? What data sources and methodologies were used?

We began working on this subject with the Nature piece in mind about 8-9 months ago. Obviously, the first step was to approach HEC itself and to seek their cooperation in doing this. There was really no point in saying anything on the subject without having access to HEC’s own data and viewpoint.

We had excellent cooperation from Dr. Atta Ur Rahman as well as Dr. Sohail Naqvi and other members of HEC staff. In addition to HEC, I also met and talked to a number of university leaders as well as HEC’s critics including Dr. Hoodbhoy. Dr. Naqvi made available to us a lot of data that we believe had not been publicly available in this manner before. We were cognizant, however, that this is self-reported data and that any conclusions we will make based on this data alone are going to be fiercely challenged by HEC’s critics.

In order to balance HEC’s own view, we also used external data to validate the claims made by HEC. Specifically, we decided to use Thomson Reuters’ data on scientific publications as independent measure of Pakistan’s publication activity and impact. We must emphasize, though, our analysis is merely a start. One of our key recommendations is for HEC to subject itself to an independent external peer review of its policies and programmes and use their guidance to improve these. This peer review must also include taking a look the current data at hand and the data collection systems in place and to improve these to provide continuous guidance to HEC planners.

STEP: What was the most important (or surprising) finding of the study?

I believe that this is still a work in progress as Pakistan’s reform experiment is itself a work in progress. I won’t speak for other members of the team here, however, I was personally surprised by quite a few findings. First of all, there does seem to be an unambiguous improvement in Pakistan’s publishing performance as well as its relative impact. The publications have tripled over the last few years and there are significant gains in relative impact of Pakistani papers. Our preliminary testing shows that this finding is not too influenced by self-citation bias. One negative finding – though not unsurprising – was the decline in the publishing performance in social sciences (and business and management and humanities etc.). This is a bit alarming – for me – because I do believe that well developed social sciences are very important for a balanced development of a nation.

I believe that several programmes may be rightly guided – even though there may be implementation lapses. The foreign PhD programme is one example. I believe, and hope, it will deliver in the medium to long-run. The NRPU – the competitive grant funding programme – is in the right direction – although the peer review process needs to be strengthened. I think that a 47% acceptance rate is just too high – especially given that a vast majority of Pakistani faculty have not had much exposure to competitive research for considerable periods in the past. The investments in the digital library and internet connectivity are also critical elements of the research infrastructure.

On the other hand, there are still several things that are not so right. There are things that can potentially go wrong – seriously wrong – if important issues are not addressed. I am especially concerned with the policies vis-à-vis the Domestic (Indigenous) PhD programme. That is certainly something that should be subjected to greater scrutiny and evaluation from both outside and inside of HEC. Also governance and management reforms within universities have been rather slow. However, I think there is a need to take key stakeholders along rather than force it on universities.

STEP: The statistic that I found most interesting in the article is the +20% relative impact of Mathematics publications from Pakistan. However, Dr. Hoodbhoy, in his critique of your article, asked the question of whether self-citations were removed or not in computing this number. To quote Dr. Hoodbhoy, “did the authors try to eliminate self-citations (a deliberate ploy) from this count? If they had – as I did using an available option in the Thomson Scientific package – they might actually have found the opposite result.” How would you respond to this criticism?

The short answer to Dr. Hoodbhoy’s question is that no – this number includes author self-citations. BUT we were definitely concerned about self-citations. Here is what we did to account for this potential bias.

We extracted a database of roughly 13,000 papers (representing all fields) indexed by Thomson Reuters between 1999 and 2008, each listing at least one author address in Pakistan. We also extracted all papers that subsequently cited these papers. We then enumerated instances of ANY country citing into this population of papers (i.e., all nations citing Pakistan), along with a separate measure of only Pakistani authors citing Pakistani papers. We did this for two distinct time periods: for papers published and cited during 1999 to 2003, and during 2004 to 2008.

For the first time period, there were 8,436 occurrences of all nations citing the Pakistani papers (or, to repeat, those papers bearing at least one Pakistan author address), and 1,527 occurrences when Pakistan-only papers cited the group. In percent terms, 1,527 of 8,436 equals 18.1%. Thus, by this measure, Pakistan self-citation could be quantified as 18.1%. For the second time period, the comparable numbers were 26,294 instances of all countries citing the Pakistan group, and 6,597 instance of Pakistan citing Pakistan. Expressed as a percentage, the figure is 25.1%

To summarize: 18.1% Pakistan self-citation for the 1999-2003 period, and 25.1% for 2004-2008. Now it is important to understand that country self-citations are not a perfect proxy for author self-citations – the variable that we’re really interested in. However, author self-citations are a subset of country self-citations and hence country self-citations do give us an upper bound on author self-citations. That is, author self-citations for Pakistani papers are definitely less than 25.1% – and perhaps a lot less than that – in the post-reform period.

We can compare this figure with generally accepted empirical norms of author self-citation globally. In Gami et al (CMAJ, 2004) the authors estimate self-citation in diabetes research at about 18%. Falagas and Kavvadia (FASEB, 2006) arrive at self-citation rates in biomedical research at 17-20%. Others have confirmed the general trend in medical literature of around 1/5^th of all citations as being author self-citations. Garfield and Sher (1964) arrive at a self-citation figure of 20% for basic research papers. Trimble (1986) arrives at a figure of 15% for all astronomical papers. Bonzi and Snyder (1986) found an average self-citation rate of 11 per cent across a range of disciplines, varying from 16 per cent in the physical sciences (chemistry and geology) to 3 per cent in the social sciences. The frequency of self-citation is also found to be independent of quality of publication.

So, Pakistan’s likely author self-citation rates may be only slightly higher – if not within – the global norms for self-citations. It is quite unlikely that all – or even most – gains can be wiped out by author self-citations. Dr. Hoodbhoy may be right in that self-citations may have increased from the past but that is likely to be expected for a variety of reasons – some of which quite legitimate. For example, as a closely knit research community develops within a country and interact through local conferences they are more likely to be aware of each others work and hence cite authors from within the country. Had we been able to exclude self-citations from these relative impact figures, these results would perhaps have been only marginally smaller. Because we were dealing with publications across multiple disciplines over multiple time periods and across multiple countries we could not correct relative impact figures for self-citation. However, given the above analysis we don’t find a reason to feel so alarmed as to totally rubbish the entire publishing performance of the country.

Should this be an issue HEC should carefully watch and analyse? – definitely yes. Should this totally nullify the improvements in Pakistan’s publication record? – we don’t believe so.

STEP: You mentioned that a lot of data provided by the HEC was not used in the article because of the space limitations. The only real statistical analysis in the current article is the table on relative impact factors. What other metrics could have been used to scientifically analyze the performance of HEC, had space not been the limiting factor.

HEC did provide a lot of data and complied with our requests for additional data as much as it could. Potentially, data could inform a number of very interesting questions vis-à-vis the performance of higher education reforms in Pakistan. We were not able to undertake some of these analyses because of the following reasons.

First, a lot of data currently available only accounts for inputs to various programmes – number of PhDs awarded, number of fellowships granted, number of grants awarded are input data. This needs to be complemented with relevant and comparable output data so that outcome assessment can take place.

Second, for data to be useful for subsequent analyses, the analytical and evaluation plan must be built into the programme itself. It is much harder – and more expensive – to cobble together data – ex post – and come up with a very clean assessment of outcomes.

As an example of the sort of things that additional analysis could do is to assess whether the foreign faculty hiring programme (FFHP) is an effective investment. During my meetings with vice chancellors I was told that there is a backlash against FFHP from the faculty and one of the allegations is that the people recruited on FFHP are no better than some of the better faculty members available within the country. This, and several others elements of the reform, are empirical questions that could be easily addressed by intelligent use of data.

STEP: Don’t you think that the HEC is under more scrutiny (at least informally) than other government departments? For example, I don’t see any debate on the performance of, say, MINFAL. The team at HEC has, at the very least, brought about many positive changes compared to the UGC of the past. Do you think there is a chance that overly negative criticism can be counterproductive and may deter such initiatives to positively reform a government department in future?

Yes, certainly there is a chance that overly negative criticism may be counter productive. However, I believe that what we have tried to do is to present our best objective view of the reforms. I believe that it’s a pity that the debate on this important set of reforms hovers between the two extreme viewpoints. I believe that good data and analysis can really refocus this debate and turn it into constructive one. Good data and analysis can move debate away from one of opinions to one of facts.

I believe it is important to have this debate in an open, transparent, and civilized manner. Such a debate will inspire greater trust and buy-in from the people and systems this reform is seeking to address. Pakistan has invested upwards of 50 billion rupees in higher education over the last 5-7 years and it is in all our interest that we make the best use of this investment.

STEP: One of your key recommendations is for the HEC to be reviewed externally. What are some of the examples (from around the world) of the external peer-review process that you suggest HEC subject itself to? Should this process not be done through the Standing Committees on Education in the Senate and/or the National Assembly?

The process of policy peer review is well-established – especially in the west. The nearest equivalent that comes to mind is the peer reviews organized by the National Academies in the US of major government funding programmes (such as the SBIRs, ATP etc.). These are panels of eminent scientists (Academy Fellows) and relevant experts looking over various aspects of these programmes. Because of the unique circumstances and pervasive nature of HEC’s programmes, HEC’s peer review would have to be done by a mix of Pakistani and internationally placed members. The findings of these reviews are then debated by legislatures who have the final say in determining policies informed by these findings and recommendations.

STEP: Based on your study, how do you compare HEC in Pakistan to other equivalent bodies in the developing countries?

I believe HEC has done a considerable amount of work in a very short period of time and some of it may have already begun to reap results as well. It has tackled a very difficult area of reform and has shaken the academic environment in Pakistan. In the depth and breadth of what it has tried to do, HEC has very few parallels in the world today. Most other relevant bodies have only tinkered with the systems that they have tried to reform. It is this factor that probably prompted Nature to invite an opinion piece from the authors and an editorial commenting on these reforms and the lessons that could be learnt by other developing countries. HEC had the liberty and resources to do so and it made the most out of it. Whether or not this hugely ambitious experiment will succeed at the expected level in the end remains an open question.

STEP: If this article is work in progress, as you have said, what further study or analysis can we expect from your team in future?

We are planning to finish a larger piece on higher education reforms in Pakistan and hopefully publish this in the near future. In addition, we intend to dialog with HEC, leading academics and academic administrators in Pakistan to explore how the suggestions in our article can be best taken forward.

Related Post: Pakistan’s Higher Education Funding Holds Many Lessons for Developing Nations: Nature

The more detailed opinion article in the current issue of Nature is co-authored by a distinguished team, including the lead author Athar Osama, a science and innovation policy researcher, former minister of education, science and technology Shams Kassim-Lakha, the director of Boston University’s Pardee Center, Adil Najam, Christopher King of ScienceWatch.com and Syed Zulfiqar Gilani of the Institute of Education and Research, University of Peshawar and a board member of Seneca College.

The article, titled “Pakistan’s Reform Experiment“, overall takes a positive view of HEC’s reforms, stating that “the HEC seems to have changed the culture of Pakistani academia considerably over the past 5 years”. Using data from Thompson Reuters, the authors show that the impact of papers from Pakistan, relative to the average of the field, has improved significantly in Mathematics and Engineering over the last five years. In fact, papers from Pakistan in Mathematics have 20% higher impact factor than the world average! In comparison, papers in  fields other than engineering and mathematics have not seen a significant improvement in the relative impact, which may be consistent with more HEC money targeted at technical disciplines.

At the same time, the article points out some potential pitfalls that the HEC needs to avoid. HEC cannot be the “initiator, implementer and evaluator” at the same time. Accountability of HEC initiatives by academics is necessary. The pace of reform has led to resistance by the universities, (a fact acknowledged by HEC Executive Director in our earlier interview), and HEC has not been fully successful in winning over that resistance. The attempt to produce too many PhDs in local universities may have compromised the quality of the PhD degree significantly. And leaving behind certain disciplines in the favor of others, especially critically important areas such as the social sciences, has not been the most prudent of policies.

We feel that the article is a well-balanced analysis of HEC’s performance. There is no doubt that the landscape of higher education has changed for the better, in a very short amount of time. However, it is also clear that higher education reform is a long term agenda. What is important is not how many grants are funded, but whether scholarship is thriving in the country or not. The conclusion of the article aptly makes this point:

The HEC has, over the past few years, made considerable progress. Its success, however, must not be measured by the number of grants made or PhDs awarded. Rather it should be judged on whether it is creating a culture of research — one driven not by financial incentives, but by a genuine desire to create new knowledge and to enable the broader society to reap the benefits. While that remains to be seen, Pakistan’s experience has useful lessons for other countries.

The article comes at an appropriate time, as the new Chairman of HEC, Dr Javaid Laghari takes over (see our posts here, and here). It provides an objective analysis to the new chairman, as he will take on the review of the previous policies and initiatives.

Coming up soon: We, at STEP, contacted the lead author of the study, Dr Athar Osama. Coming soon are his replies on what led to this study, what data sources and methodologies were used and what was the most surprising finding of this study.

Photocredit: http://www.flickr.com/photos/leena/2488225767/