To seed this discussion, we present commentary from Dr. Pervez Hoodbhoy and Dr. Atta-ur-Rehman. Dr. Hoodbhoy presents his opposing point of view, arguing that the measures presented in the article were inadequate, and further that the conclusions drawn from the metrics were flawed. Dr. Atta-ur-Rehman, founding (and former) chairman of the HEC, who led the higher education reform effort during his tenure, responds by pointing to data that, in his view, shows the depth and breadth of the reform’s success.

We invite our readers to contribute their thoughts on what metrics are appropriate for measuring the success of higher education within the context of Pakistan.

NOTE: Both commentators have significantly shaped the landscape of Pakistani education over the last few decades. We request our discussants to avoid personalizing the discussion and to maintain a civil and constructive tone.

SYNOPSIS (We will continue to compile the synopsis of the discussion as it progresses: Last Update 8:02am EDT, September 22nd, 2009.)

The metrics suggested, thus far:

1. QUALITY OF RESEARCH (Editors, Zeeshan Khan): Citation and variants on their measures like the h-index, are standard measures of research quality on an individual, institutional, and national level. Number of patents is another measure, though .
2. QUANTITY OF RESEARCH (Atta-ur-Rahman): This measures gross research activity.
3. QUALITY OF TEACHING (Pervez Hoodbhoy): Metric?
4. QUALITY OF UNIVERSITY GRADUATES (Pervez Hoodbhoy, Fakhruddin Habiby): Surveying employers or assessing performance in international tests.
5. ACADEMIC FREEDOM (Pervez Hoodbhoy): Metric?
6. ACCESS TO UNIVERSITY FACILITIES (Atta-ur-Rahman, Khurram Shafique): Libraries, laboratories, internet connectivity, communication facilities, sports facilities,
7. EVALUATION BY NEUTRAL EXPERTS (Atta-ur-Rahman): Survey of a group of neutral experts, like the World Bank, USAID, etc.
8. UNIVERSITY ENROLLMENT (Atta-ur-Rahman): The increase in university-going adults can be measured by census.
9. UNIVERSITY-INDUSTRY LINKAGE (Fakhruddin Habiby, Anwar): The number of industry supported projects which were initiated with University-Industry partnerships and their effectiveness based on industry feedback.
10. LOCAL RELEVANCE OF RESEARCH (Editors): Metric?
11. QUALITY OF CURRENT STUDENTS (Pervez Hoodbhoy, Khurram Shafique):  Performance in standardized tests conducted every year, performance in local and international competitions such as Mathematics Olympiads and Programming Contests.

Notes:

(Pervez Hoodbhoy) Self-citations are a serious problem when using citations as a metric for measuring quality of research reform. [paraphrased]

(Abdullah Sadiq) While strengthening the research effort in the universities is important, the most urgent need is to concentrate on producing quality teachers for the lower tears of education. [paraphrased]

(Khurram Shafique) A pedestrian publication in the field of networking or multimedia is likely to receive more citations than a good publication in a less explored field in mathematics, say, non-standard analysis.

(Fakhruddin Habiby) another ‘tool’ that is used to push the citation number higher is formation of ‘citation-coalition’ within research groups.

(Omar Javed) Categorization of universities into subsets, and adoption of relevant performance criteria for each subset. Three fundamental questions: what is taught (Undergraduate and Graduate Instructional Program classifications), who are the students (Enrollment Profile and Undergraduate Profile), and what is the setting (Size & Setting)”

(Shafiqur Rehman) …the success or falure or HE reforms must only be judged by opinion of the common stake-holders (students, teachers and administration) of the public sector universities.

(Affan): we need to tweak ratings/rankings such that we are able to measure any progress happening in Pakistan, progress small enough that it is not lost by existing metrics.

There are four aspects of the comments of Dr. Pervez Hoodbhoy that need to be considered:

1. Firstly, Dr. Hoodbhoy himself admits that there has been a huge increase in international publications at QAU after HEC came into existence when he mentions the number of international publications in the two time periods. Strangely, he picks a six year period, 1998-2003, and then compares it with the subsequent 4.5 years (?) , 2004 to mid 2008, (the correspondence occurred in August 2008, so he could not possibly have had access to the figures for the entire year) I can only assume that he has mentioned 2003 by mistake in the second “5 year” period as there is no reason to include the publications of the year 2003 in both time periods, which he has done. It is clearly unfair to take two time periods of different durations and compare them.
2. In the first 6 year period (1998-2003), Dr. Hoodbhoy admits that there were only 631 research publications from QAU, but in the second 4.5 year period these had risen to 1482 research publications, a tripling of publications on average per year, even by his own estimates.
3. As the HEC programs began in 2003 and their real impact occurred 2-3 years later, a year-wise comparison is far more relevant than an average over a 5 year period as the dramatic change that has occurred gets partly masked when a 5 or 6 year average is taken, though it is still very visible. Dr. Hoodbhoy ignores the figures that Dr. S.T.K. Naim had worked out that in the year 2004, there were only 84 research publications from QAU (an average of only 7 publications per month), but by 2008 they had increased many fold.
4. The citations argument used by Dr. Hoodbhoy is invalid as citations increase with the passage of time. Dr. Hoodbhoy, therefore, wrongly compares the citations of papers of an earlier period with those of a later period. To clarify this issue further, if two papers of equal quality and in a similar field are published, say in 1998 and 2007, and the citations of both are counted in 2008, then the paper which was published in 1998 will have accumulated more citations by 2008 because of the much longer 10 year time period, than the paper published in 2007, as that would have had only one year for the citations to accumulate. Dr. Hoodbhoy is therefore comparing apples with oranges when he tries to compare citations of papers published in an earlier period with a later time period. In order to fairly compare citations, the same duration of time period must be taken. Thus if one takes 1998 publications and counts the citations till 2008, then one will need to take the 2008 publications and count their citations till the year 2018, before one can compare the figures for the citations of the two sets fairly.

The undeniable fact is that the total number of research publications from universities in Pakistan was only about 600 per year till 2001 but then started rising rapidly, and by the year 2008 it had increased to over 4,300! Brazil achieved such an increase over a 35 year period between 1960 to 1995, which Pakistan achieved in only 6 years. After my appointment in March 2000 as the Federal Minister for Science and Technology in Pakistan, I convinced the government to enhance the budget for science and technology in Pakistan by 6000% between July 2000 to October 2002. After my appointment as Chairman, Higher Education Commission (Federal Minister) the budget for higher education was similarly increased by 2400% during 2003 to 2008. Major achievements during these periods were:

1. Establishing 51 new Universities and awarding institutions during 2002-2008,
2. Tripling university enrollment (which had reached only 135,000 from 1947 to 2003) to about 400,000 in 2008,
3. Establishing a powerful Digital Library which provides free nation-wide access to every student in every public sector university to 45,000 textbooks/research monographs from 220 international publishers as well as to 25,000 international research journals,
4. Establishing video-conferencing facilities in most public sector universities that allow lectures to be delivered live and interactively to students in Pakistan from technologically advanced countries
5. Enhancing salaries of academics so that salaries of University Professors were increased to a level about five times the salaries of Federal Ministers, with a corresponding reduction in tax from 35% to only 5%, in order to attract the brightest young men and women into academia,
6. Promoting research through a massive research grant program which resulted in a 600% increase in ISI abstracted publications from about 600 per year in 2001 to 4300 research publications in 2008, accompanied by about 1000% increase in international citations in the same period,
7. Placing a satellite in space (Paksat-1) which is now used for distance learning by the Virtual University,
8. Establishing video-conferencing facilities in most public sector universities and initiating a lectureship program, allowing live interactive lectures to be delivered from technologically advanced countries,
9. Providing free access to scientists/engineers anywhere in the country to sophisticated instruments installed in any institute in Pakistan.

The Bottom Line: In the final analysis, it is not what I or Dr. Hoodbhoy think about the developments, but what is the opinion of neutral international experts who have carried out detailed year-long reviews of the developments during the period that I was heading the Higher Education Commission. A few extracts are given below:

1. Prof. Fred Hayward (independent international educational consultant from USA) carried out a detailed analysis of the developments and published an article entitled “Higher Education Transformation in Pakistan: Political & Economic Instability,” Date: Number 54, winter 2009 Source: International Higher Education Quarterly. I quote: “The news about Pakistan over the last few years has been dominated by reports of political turmoil, terrorism, religious fundamentalism, economic decline, and the Afghan War. What has been missed is the phenomenal transformation in higher education over the last six years, which represents a critical development for Pakistan and a potential engine for growth and national recovery.”
2. Report of US-AID about HEC states that “We are very impressed with the breadth, scope, and depth of the reforms implemented by the HEC since 2002.  No other developing country we know has made such spectacular progress.”
3. World Bank Report is very complimentary of many excellent programmes introduced.
4. British Council: The report states: “I have worked in many countries in South America, the Middle East, North Africa, and in Russia and India, over the last six years.  None in my view, with the exception of India, has the potential of Pakistan for the UK university sector, largely because of the dynamic, strategic leadership of the Chairman of HEC”.
5. Nature: Several articles and editorials have appeared in the world’s leading science journal “Nature”  (the most recent in the issue published on 3rd September 2009) in which the very significant progress made by Pakistan in the higher education sector has been applauded and the need for the new government to built on the solid foundation laid has been stressed.
6. Science Watch (Thomson Reuters) has ranked Pakistan as a rising star in five disciplines, more than in any other country of the world.

Join the Discussion!
What are the correct metrics to measure higher education reform in Pakistan?

Related Posts:

1. Nature’s Coverage of Higher Education Reform in Pakistan: A Response from Prof. Pervez Hoodbhoy
2. Nature’s Coverage of Higher Education Reform in Pakistan: A Conversation with Athar Osama
3. Pakistan’s Higher Education Funding Holds Many Lessons for Developing Nations: Nature

Editors Note: Dr. Atta-ur-Rahman is a leading scientist and scholar in the field of organic chemistry from Pakistan. He has served as the Federal Minister for Science and Technology, the Federal Minister/Chairman of the Higher Education Commission, Adviser to the Prime Minister on Science and Technology, and the President of the Pakistan Academy of Sciences. Dr. Atta-ur-Rahman is credited by many for reviving the higher education and research practices in Pakistan. The views expressed in this article are solely those of the author and do not necessarily reflect the views of STEP.

Unfortunately, I find the response as unsatisfying as the original article. Since Nature is unwilling to accord me a chance for a satisfactory reply on its pages, I shall clarify the basis of my criticism in some detail here.

In the said article, strong conclusions have been derived from weak data. The authors have not dared to ask the basic questions whose answers are essential for ascertaining whether there has been actual progress in Pakistan’s higher education system and, if so, by how much. Instead, in giving a thumbs-up, numbers have been quoted that have doubtful significance. Take, for instance, the claim that:

“In mathematics, for example, an average paper by a Pakistani author is cited around 20% more than the worldwide average for the discipline”.

Before returning to what some of the right questions might have been, let me give two reasons why the above claim – even if true – carries little meaning.

First, self-citation is a far more serious problem than the authors are willing to acknowledge. It is also one that they admit to not having investigated. The data on Pakistani research papers shows that subtracting out self-citations drastically cuts down on actual citations – there are often 2-3 self-citations for every real one! The reader is urged to carefully study my email correspondence of last year with Dr. Atta-ur-Rahman (see appendix below) who, at the time when he was HEC chairman, had made similar claims that I disputed as being false. To interested readers, I have made available (in pdf form) the Thomson Scientific data that I have quoted in my correspondence here (1998-2003) and here (2003-2008).

Second, many authors of the relatively more cited mathematics papers from Pakistani institutions are not Pakistani nationals. High salaries offered to foreign faculty by the HEC brought to Pakistan a large number of well established mathematicians on short-term contracts from Russia, Poland, Ukraine, and China. This was probably a good thing to do – in spite of the difficulty they had in communicating in an alien language and their consequent inability to teach well. Their papers, however, do not reflect mathematics in Pakistan. One sees a similar phenomenon in Saudi Arabia where foreigners are principally responsible for the kingdom’s large number of papers and citations.

In my opinion, instead of focusing on marginal matters, serious research on the state of Pakistani higher education, and of changes therein, would have first established appropriate metrics, and then sought answers, to the following key questions:

1. What is the quality of teaching in Pakistan’s public universities? There is often only a weak correlation between formal qualifications and subject competence, so simply counting PhD degrees is not very helpful in answering this. Far too frequently one sees professors of English who cannot speak or write a single sentence of grammatically correct English, physics professors who are stymied by Newton’s Laws, and biology professors whose knowledge is frozen in some pre-Darwinian age. But does such basic incompetence exist at the 20, 50, or 70 percent-level? Higher? Lower? What evidence exists that the HEC’s reforms improved the situation?
2. Is there evidence that there has been improvement in the selection process for students in public universities, or that of the quality of their graduates? Proof of the latter, judged by asking employers or assessing performance in international tests, would be a clinching argument for the success of HEC reforms.
3. Do campuses enjoy greater academic freedom, more seminars and colloquia, less violence by extremist campus groups, a pleasanter and more relaxed ambiance, and greater transparency in faculty selection? Surely these are critical to any reasonable assessment.

To get answers to questions like these requires extensive field work, and I certainly do not fault the authors for not doing this. But I was surprised that the Nature article, as well as the lead author’s response, merely says that the HEC’s experiment had critics, without citing any specific articles or the substance of those criticisms. There is not even a passing reference to the failed nine-university multi-billion dollar mega-project, tons of unused scientific equipment purchased for unknown reasons, dubious attempts to fund “Quranic Science” (that had to be hastily abandoned after the scheme was exposed), and the explosion in academic corruption set off by per-paper payments. Surely, these should not be brushed aside as “collateral damage”. In another country, those who massively squandered public money would have been thoroughly investigated by independent commissions, not praised for small things.

The Bottom Line: Well-functioning universities are the products of a complex organic and evolutionary process that is internal to a society. Money and facilities matter, but it is much more important for a university to have a forward looking world-view, an open environment, high ethical standards, a sense of collegiality and shared sense of purpose, and good governance practices. Sadly, the Nature article did not even mention these as significant.

Join the Discussion!
What are the correct metrics to measure higher education reform in Pakistan?

Related Posts:

1. Nature’s Coverage of Higher Education Reform in Pakistan: Comments by Dr. Atta-ur-Rahman
2. Nature’s Coverage of Higher Education Reform in Pakistan: A Conversation with Athar Osama
3. Pakistan’s Higher Education Funding Holds Many Lessons for Developing Nations: Nature

Editors Note: Prof. Pervez Hoodbhoy is a well-known Pakistani nuclear physicist and political-defence analyst. He is the Professor of High Energy Physics, and the head of the Physics Department at Quaid-e-Azam University, Islamabad, Pakistan. Prof. Hoodbhoy is a vocal critic of HEC’s policies and their impact. The views expressed in this article are solely those of the author and do not necessarily reflect the views of STEP.

This correspondence between Dr. Atta-ur-Rahman and Dr. Pervez Hoodbhoy is reproduced below with consents from both parties. It is exclusively concerned with a public matter, has no private content, and is largely focused upon the importance of self-citations.

Dear Dr. Pervez
I have probed into the situation at QAU, and it is quite the opposite of what you claim, as evident from the total publications and the total number of citations, as per above attachments. The HEC was established in October 2002. The funds started coming through from July 2003. The subsequent impact of HEC programs on research publications and citations is indeed amazing. In 2004 the number of publications was only about 120—by 2007 it has risen to about 380—a 300% increase! The citations in 2004 were about 800—-by August 2008 they have increased to about 2200 although we still have 4 months to go before the year ends—I suspect that it will be about 3200 by the end of the year—a 400% increase!

I shall be separately sending you a report after removing self-citations. QAU is also now included in the most cited institutions in the world (please see attachment)—-this was not to 4 years ago.

Please be fair and objective in your assessments. We may have made some mistakes, but much good has happened.
Kind regards
Atta

Dear Dr. Pervez
Following my other email to you this morning, I am now enclosing the citations of QAU after removing self-citations.As you would see, these have risen from only 84 in 2004 to 1413 in 2008 (with still 4 months to go). These will probably be around 1900 by the end of the year—a spectacular ten-fold growth! Dr. Naim has kindly had these searched, so if you have any queries about them, you may like to interact with her. She tells me that the situation is similar in many other universities—a long period of stagnantion
during the 1990s followed by a burst of activity in the last 4-5 years.
Kind regards

Atta

Dear Dr. Atta,

I am sorry about the late response to your three emails. First, thank you for doing whatever you did – the HEC’s notification of 14 July 2008, which specifies 40 percentile as the GRE passing marks, finally reached QAU departments today (without comment from the administration). In these times one has to be grateful for small things…

Regarding your email and the numbers cited therein: I can understand that you would like to feel upbeat about QAU having improved itself as a result of massive infusion of HEC resources. I too would very much like good things to happen, but perhaps one should not allow wishes to become conclusions.

Looking at the data that you had sent me and your inferences, I decided to do a little independent investigation using exactly the same database (ISI Web of Science) and exactly the same keywords (see attachments to this email). Here are the findings:

Period: 1998-2003
Number of papers published by QAU authors in the above period: 631
Number of citations to date: 4540
Number of citations to date with self-citations removed: 2,817

Period: 2003-2008
Number of papers published by QAU authors in the above period: 1482
Number of citations to date: 3667
Number of citations to date with self-citations removed: 1258

Some obvious inferences:

1. There is absolutely no evidence of real citations having increased; if anything, the numbers up to now show the contrary. While the number of real citations of papers published 2003-2008 may increase somewhat with time, currently they are quite a bit less from the earlier period when the HEC and your incentive system did not exist. Please remember that citations are cumulative over years. I have tried to use exact descriptions in the figures cited above. If I am wrong in any detail, or if I have missed something essential, I would like to be corrected. Unfortunately the data does not at all support your rather optimistic remark of “a spectacular ten-fold growth!”
2. The above data also indicates the disturbing fact that most of the time QAU authors cite themselves. Subtracting self-citations drastically cuts down on real citations – there are 2-3 self-citations for every real one!. Looking more minutely at the ISI pages, one also notes that many citations are by other members belonging to the same or other QAU departments. So the number of genuine citations gets cut down even beyond the numbers quoted above (2817, 1258)!

Just to get scales right, here are a few citation numbers from the Stanford Spires data base for Pakistani physicists over the length of their careers:
a) Riazuddin: 1479
b) Ahmed Ali (DESY, Germany): 9873
c) Abdus Salam: 14103

3. You are correct that the number of publications over equal (5-year) time periods has more than doubled relative to pre-HEC times. But this is clearly in response to the monetary incentives offered by PCST/QAU. A publication fever now grips our universities. It is difficult to defend the case that the number of papers published is proportional to the amount of research done.

You are, of course, free to have the data I have quoted above rechecked and I would be happy to answer any question that arises. Finally, please note that publications and citations were not central to my earlier expression of dismay at the quality of QAU education.

What bothers me much more is the lack of analytical and problem-solving abilities of our Ph.D graduates, some honourable exceptions aside. Poor performance in the GREs is one indication of the rot. This fact has indeed worried you a little, as you indicated in an earlier email to me, but I do wish you could understand the real gravity of the situation.

Regards,

Pervez

Dr. Atta-ur-Rahman’s comments on this post can be found here.