Nature’s Coverage of Higher Education Reform in Pakistan: A Response from Prof. Pervez Hoodbhoy

By Pervez Hoodbhoy, September 17, 2009

Editor's Note: Comments on this post have been disabled. Please participate in the discussion here.

This communication is concerned with “Pakistan’s Reform Experiment” (Nature, V461, page 38, 3 September 2009), and the response to my critique by its lead author.

Unfortunately, I find the response as unsatisfying as the original article. Since Nature is unwilling to accord me a chance for a satisfactory reply on its pages, I shall clarify the basis of my criticism in some detail here.

In the said article, strong conclusions have been derived from weak data. The authors have not dared to ask the basic questions whose answers are essential for ascertaining whether there has been actual progress in Pakistan’s higher education system and, if so, by how much. Instead, in giving a thumbs-up, numbers have been quoted that have doubtful significance. Take, for instance, the claim that:

“In mathematics, for example, an average paper by a Pakistani author is cited around 20% more than the worldwide average for the discipline”.

Before returning to what some of the right questions might have been, let me give two reasons why the above claim – even if true – carries little meaning.

First, self-citation is a far more serious problem than the authors are willing to acknowledge. It is also one that they admit to not having investigated. The data on Pakistani research papers shows that subtracting out self-citations drastically cuts down on actual citations – there are often 2-3 self-citations for every real one! The reader is urged to carefully study my email correspondence of last year with Dr. Atta-ur-Rahman (see appendix below) who, at the time when he was HEC chairman, had made similar claims that I disputed as being false. To interested readers, I have made available (in pdf form) the Thomson Scientific data that I have quoted in my correspondence here (1998-2003) and here (2003-2008).

Second, many authors of the relatively more cited mathematics papers from Pakistani institutions are not Pakistani nationals. High salaries offered to foreign faculty by the HEC brought to Pakistan a large number of well established mathematicians on short-term contracts from Russia, Poland, Ukraine, and China. This was probably a good thing to do – in spite of the difficulty they had in communicating in an alien language and their consequent inability to teach well. Their papers, however, do not reflect mathematics in Pakistan. One sees a similar phenomenon in Saudi Arabia where foreigners are principally responsible for the kingdom’s large number of papers and citations.

In my opinion, instead of focusing on marginal matters, serious research on the state of Pakistani higher education, and of changes therein, would have first established appropriate metrics, and then sought answers, to the following key questions:

What is the quality of teaching in Pakistan’s public universities? There is often only a weak correlation between formal qualifications and subject competence, so simply counting PhD degrees is not very helpful in answering this. Far too frequently one sees professors of English who cannot speak or write a single sentence of grammatically correct English, physics professors who are stymied by Newton’s Laws, and biology professors whose knowledge is frozen in some pre-Darwinian age. But does such basic incompetence exist at the 20, 50, or 70 percent-level? Higher? Lower? What evidence exists that the HEC’s reforms improved the situation?
Is there evidence that there has been improvement in the selection process for students in public universities, or that of the quality of their graduates? Proof of the latter, judged by asking employers or assessing performance in international tests, would be a clinching argument for the success of HEC reforms.
Do campuses enjoy greater academic freedom, more seminars and colloquia, less violence by extremist campus groups, a pleasanter and more relaxed ambiance, and greater transparency in faculty selection? Surely these are critical to any reasonable assessment.

To get answers to questions like these requires extensive field work, and I certainly do not fault the authors for not doing this. But I was surprised that the Nature article, as well as the lead author’s response, merely says that the HEC’s experiment had critics, without citing any specific articles or the substance of those criticisms. There is not even a passing reference to the failed nine-university multi-billion dollar mega-project, tons of unused scientific equipment purchased for unknown reasons, dubious attempts to fund “Quranic Science” (that had to be hastily abandoned after the scheme was exposed), and the explosion in academic corruption set off by per-paper payments. Surely, these should not be brushed aside as “collateral damage”. In another country, those who massively squandered public money would have been thoroughly investigated by independent commissions, not praised for small things.

The Bottom Line: Well-functioning universities are the products of a complex organic and evolutionary process that is internal to a society. Money and facilities matter, but it is much more important for a university to have a forward looking world-view, an open environment, high ethical standards, a sense of collegiality and shared sense of purpose, and good governance practices. Sadly, the Nature article did not even mention these as significant.

Join the Discussion!
What are the correct metrics to measure higher education reform in Pakistan?

Related Posts:

Editors Note: Prof. Pervez Hoodbhoy is a well-known Pakistani nuclear physicist and political-defence analyst. He is the Professor of High Energy Physics, and the head of the Physics Department at Quaid-e-Azam University, Islamabad, Pakistan. Prof. Hoodbhoy is a vocal critic of HEC’s policies and their impact. The views expressed in this article are solely those of the author and do not necessarily reflect the views of STEP.

APPENDIX

This correspondence between Dr. Atta-ur-Rahman and Dr. Pervez Hoodbhoy is reproduced below with consents from both parties. It is exclusively concerned with a public matter, has no private content, and is largely focused upon the importance of self-citations.

Date: Thu, 4 Sep 2008 12:31:45 +0500
From: atta
To: dr.pervez hoodbhoy
Cc: atta , Dr. S. Sohail H. Naqvi
Subject: Citation Report – QUAID-i-AZAM University and Highly Cited institutions -World

Dear Dr. Pervez
I have probed into the situation at QAU, and it is quite the opposite of what you claim, as evident from the total publications and the total number of citations, as per above attachments. The HEC was established in October 2002. The funds started coming through from July 2003. The subsequent impact of HEC programs on research publications and citations is indeed amazing. In 2004 the number of publications was only about 120—by 2007 it has risen to about 380—a 300% increase! The citations in 2004 were about 800—-by August 2008 they have increased to about 2200 although we still have 4 months to go before the year ends—I suspect that it will be about 3200 by the end of the year—a 400% increase!

I shall be separately sending you a report after removing self-citations. QAU is also now included in the most cited institutions in the world (please see attachment)—-this was not to 4 years ago.

Please be fair and objective in your assessments. We may have made some mistakes, but much good has happened.
Kind regards
Atta

Date: Thu, 4 Sep 2008 12:50:07+0500
From: atta
To: dr.pervez hoodbhoy
Cc: Dr. S. Sohail H. Naqvi , atta, tanvir naeem
Subject: Fw: QUAID-i-AZAM University – 2003-2007 citations

Dear Dr. Pervez
Following my other email to you this morning, I am now enclosing the citations of QAU after removing self-citations.As you would see, these have risen from only 84 in 2004 to 1413 in 2008 (with still 4 months to go). These will probably be around 1900 by the end of the year—a spectacular ten-fold growth! Dr. Naim has kindly had these searched, so if you have any queries about them, you may like to interact with her. She tells me that the situation is similar in many other universities—a long period of stagnantion
during the 1990s followed by a burst of activity in the last 4-5 years.
Kind regards

Atta

Date: Wed, 10 Sep 2008 06:35:09 -0400 (EDT)
From: Pervez Hoodbhoy
To: atta
Cc: Dr. S. Sohail H. Naqvi , tanvir naeem
Subject: About whether QAU is going up or down

Dear Dr. Atta,

I am sorry about the late response to your three emails. First, thank you for doing whatever you did – the HEC’s notification of 14 July 2008, which specifies 40 percentile as the GRE passing marks, finally reached QAU departments today (without comment from the administration). In these times one has to be grateful for small things…

Regarding your email and the numbers cited therein: I can understand that you would like to feel upbeat about QAU having improved itself as a result of massive infusion of HEC resources. I too would very much like good things to happen, but perhaps one should not allow wishes to become conclusions.

Looking at the data that you had sent me and your inferences, I decided to do a little independent investigation using exactly the same database (ISI Web of Science) and exactly the same keywords (see attachments to this email). Here are the findings:

Period: 1998-2003
Number of papers published by QAU authors in the above period: 631
Number of citations to date: 4540
Number of citations to date with self-citations removed: 2,817

Period: 2003-2008
Number of papers published by QAU authors in the above period: 1482
Number of citations to date: 3667
Number of citations to date with self-citations removed: 1258

Some obvious inferences:

1. There is absolutely no evidence of real citations having increased; if anything, the numbers up to now show the contrary. While the number of real citations of papers published 2003-2008 may increase somewhat with time, currently they are quite a bit less from the earlier period when the HEC and your incentive system did not exist. Please remember that citations are cumulative over years. I have tried to use exact descriptions in the figures cited above. If I am wrong in any detail, or if I have missed something essential, I would like to be corrected. Unfortunately the data does not at all support your rather optimistic remark of “a spectacular ten-fold growth!”
2. The above data also indicates the disturbing fact that most of the time QAU authors cite themselves. Subtracting self-citations drastically cuts down on real citations – there are 2-3 self-citations for every real one!. Looking more minutely at the ISI pages, one also notes that many citations are by other members belonging to the same or other QAU departments. So the number of genuine citations gets cut down even beyond the numbers quoted above (2817, 1258)!

Just to get scales right, here are a few citation numbers from the Stanford Spires data base for Pakistani physicists over the length of their careers:
a) Riazuddin: 1479
b) Ahmed Ali (DESY, Germany): 9873
c) Abdus Salam: 14103

3. You are correct that the number of publications over equal (5-year) time periods has more than doubled relative to pre-HEC times. But this is clearly in response to the monetary incentives offered by PCST/QAU. A publication fever now grips our universities. It is difficult to defend the case that the number of papers published is proportional to the amount of research done.

You are, of course, free to have the data I have quoted above rechecked and I would be happy to answer any question that arises. Finally, please note that publications and citations were not central to my earlier expression of dismay at the quality of QAU education.

What bothers me much more is the lack of analytical and problem-solving abilities of our Ph.D graduates, some honourable exceptions aside. Poor performance in the GREs is one indication of the rot. This fact has indeed worried you a little, as you indicated in an earlier email to me, but I do wish you could understand the real gravity of the situation.

Regards,

Pervez

Dr. Atta-ur-Rahman’s comments on this post can be found here.