Revolutionary, humanist, and scientist, Faheem Hussain embodied the political and social ferment of the late 1960′s. With a Ph.D that he received in 1966 from Imperial College London, he had been well-placed for a solid career anywhere in the world. In a profession where names matter, he had worked under the famous P.T. Mathews in the group headed by the even better known Abdus Salam. After his degree, Faheem spent two years at the University of Chicago. This gave him a chance to work with some of the world’s best physicists, but also brought him into contact with the American anti-Vietnam war movement and a powerful wave of revolutionary Marxist thinking. Even decades later, Faheem would describe himself as an “unreconstructed Marxist”. Participating in the mass anti-war demonstrations at UC had stirred his moral soul; he felt the urge to do more than just physics. Now married to Jane Steinfels, a like-minded soul who he met in Chicago, Faheem decided to return to Pakistan.

Faheem and Jane made an amazing couple. Fully immersed in the outstanding causes of the times, they seemed to have a limitless amount of revolutionary energy. Long before I knew them, they had been protesting against the Pakistan Army’s actions in East Pakistan. As Faheem would recount, this was a lonely fight. Many Marxists in those times, inspired by Mao’s China, chose to understand the issue in geopolitical terms rather than as a popular struggle for independence. Some leftists ended up supporting the army’s mass murder of Bengalis.

With Bangladesh now a reality, things moved on. Bhutto’s rhetoric of socialism and justice for the poor had inspired nascent trade union movements to sprout across Pakistan’s cities. Many, however, quickly turned into organizations for labour control rather than emancipation.

There were genuinely independent ones too, such as the Peoples Labour Federation (PLF), an independent Rawalpindi based trade union that saw through Bhutto’s shallow rhetoric. In the early 1970′s, Faheem and Jane were highly influential in this organization, sometimes providing security and cover to its hunted leadership. Iqbal Bali, who passed away in the middle of this year, would vividly recount those days.

Very soon, I joined the small group of leftwing activists that looked up to this couple for instruction and guidance. We formed study groups operating under the PLF, both for self-education and for spreading the message through small study groups of industrial workers. Some, including myself, branched out further, working in distant villages. Gathering material support for the Baloch nationalists, who were fighting an army rejuvenated by Bhutto, was yet another goal for the group. The dream was to bring about a socialist revolution in Pakistan.

All this crashed to an end with Bhutto’s death by hanging in 1979 and the subsequent consolidation of General Zia-ul-Haq’s coup. Pakistan’s Dark Age had just begun. Although Bhutto’s regime had turned repressive and violent in its last desperate days, it was gentle in comparison with what was to follow. With dissent savagely muzzled, the only option was to operate underground. On 3 November 1981, three of our QAU colleagues and friends were caught, imprisoned, and savaged by the military regime. Jamil Omar, a lecturer in computer science and the “ring leader” – was tortured. Two others – Tariq Ahsan and Mohammed Salim – were also imprisoned and their careers destroyed. Their crime was involvement in the secret publication of “Jamhoori Pakistan”, a 4-page newsletter that demanded return to democracy and the end of army rule. A triumphant Zia-ul-Haq went on Pakistan Television, congratulated the men who had succeeded in arresting the teachers, and pledged to “eliminate the cancer of politics” from Quaid-e-Azam University.

Although Faheem was not directly involved in “Jamhoori Pakistan”, we knew he was being closely watched by the intelligence and could have chosen to hide. Instead, with characteristic fearlessness, he did all that was possible to help locate the abducted teachers, and then to secure their release. Tariq Ahsan wrote to me from Canada that “His solidarity during those long years was an invaluable source of support for our families and friends.”

But the struggle took its toll. By the mid 1980′s, Faheem was in the doldrums. Situated in an academically barren environment, he was able to publish little research of worth. Politically, there was no chance of doing anything significant in the climate of repression. Things had gone downhill in personal terms as well – his marriage with Jane was coming apart. To the great sorrow of their friends, the couple parted ways and Jane returned to America. Encouraged by Faheem, she had written school books on Pakistani history that are still sold and used today. In 1989, Faheem left QAU formally but his involvement in academic and political matters had already dropped off in the year or two before that.

From this low point in his life, Faheem struggled upwards. Initially in Germany, and then elsewhere later, he now concentrated solely upon his profession and was able to learn an impressive amount of new physics.

Professor Abdus Salam, who by now had received a Nobel Prize for his work, invited Faheem to become a permanent member of the theoretical physics group at the International Centre for Theoretical Physics in Italy. Faheem remained there until his retirement in 2004. Getting this position was no mean achievement: theoretical physics is a fiercely competitive and notoriously difficult subject. Faheem was the first Pakistani to publish a research paper in one of its most challenging areas – superstring theory.

With a cheerful and positive disposition, and an abiding concern for the welfare of others, Faheem quickly became popular at the ICTP. His laughter would resonate in the institute’s corridors. With time, he took on administrative responsibilities as well and was instrumental in setting up a “Diploma Programme” that admits students from third world countries for advanced studies in various areas. Now married to Sara, a beautiful and even-tempered Italian woman, he was equally comfortable with Italians and Pakistanis or, for that matter, Indians. To Faheem, a cultural amphibian, differences between nations carried no meaning.

And then came retirement time. What to do? I wrote to Faheem: come back!

He agreed. Finding money was not a problem – Pakistan’s higher education was experiencing a budgetary boom. But his old university, plagued by base rivalries and a contemptuous disdain for learning, refused. Specious arguments were given to prevent one of its own founding members, now one of Pakistan’s most distinguished and active physicists, from being taken on the faculty. Initially at the National Centre for Physics in Islamabad, Faheem was eventually offered a position at the newly established science faculty of LUMS in Lahore.

Faheem’s unpretentious mannerisms and gentleness of spirit ensured that LUMS too was enamored of him. Asad Naqvi, one of Pakistan’s leading physicists and a faculty member at LUMS, wrote to me upon hearing of Faheem’s death: “I am lost after hearing this. I only knew him for about 5 years, and in that short time, I had grown really fond of him. We are all poorer today, having lost such a lovely person who touched us so deeply.”

Surely, there shall be many other such tributes from Faheem’s many friends. But, to be true to him as well as my own self, I must admit that in later years we did disagree on some important things – “unreconstructed Marxism” to me is an anachronism, a relic of the 1960′s and still earlier, meaningless in a world that has become far more complex than Marx could have possibly imagined. Nor can I reflexively support today’s so-called “anti-imperialism” of the left that ends up supporting the forces of regressive fundamentalism. But let these issues stand wherever they do.

Why is it necessary for friends to agree upon everything?

From atoms to atoms – death is inevitable, the final victory of entropy over order. Meaningless? No! To have lived a full life, to have experienced its richness, to have struggled not just for one-self but for others as well, and to have earned the respect and love of those around you. That is a life worth living for. Faheem, my friend, you are gone. May you now rest in peace, with a job well done.

To seed this discussion, we present commentary from Dr. Pervez Hoodbhoy and Dr. Atta-ur-Rehman. Dr. Hoodbhoy presents his opposing point of view, arguing that the measures presented in the article were inadequate, and further that the conclusions drawn from the metrics were flawed. Dr. Atta-ur-Rehman, founding (and former) chairman of the HEC, who led the higher education reform effort during his tenure, responds by pointing to data that, in his view, shows the depth and breadth of the reform’s success.

We invite our readers to contribute their thoughts on what metrics are appropriate for measuring the success of higher education within the context of Pakistan.

NOTE: Both commentators have significantly shaped the landscape of Pakistani education over the last few decades. We request our discussants to avoid personalizing the discussion and to maintain a civil and constructive tone.

SYNOPSIS (We will continue to compile the synopsis of the discussion as it progresses: Last Update 8:02am EDT, September 22nd, 2009.)

The metrics suggested, thus far:

1. QUALITY OF RESEARCH (Editors, Zeeshan Khan): Citation and variants on their measures like the h-index, are standard measures of research quality on an individual, institutional, and national level. Number of patents is another measure, though .
2. QUANTITY OF RESEARCH (Atta-ur-Rahman): This measures gross research activity.
3. QUALITY OF TEACHING (Pervez Hoodbhoy): Metric?
4. QUALITY OF UNIVERSITY GRADUATES (Pervez Hoodbhoy, Fakhruddin Habiby): Surveying employers or assessing performance in international tests.
5. ACADEMIC FREEDOM (Pervez Hoodbhoy): Metric?
6. ACCESS TO UNIVERSITY FACILITIES (Atta-ur-Rahman, Khurram Shafique): Libraries, laboratories, internet connectivity, communication facilities, sports facilities,
7. EVALUATION BY NEUTRAL EXPERTS (Atta-ur-Rahman): Survey of a group of neutral experts, like the World Bank, USAID, etc.
8. UNIVERSITY ENROLLMENT (Atta-ur-Rahman): The increase in university-going adults can be measured by census.
9. UNIVERSITY-INDUSTRY LINKAGE (Fakhruddin Habiby, Anwar): The number of industry supported projects which were initiated with University-Industry partnerships and their effectiveness based on industry feedback.
10. LOCAL RELEVANCE OF RESEARCH (Editors): Metric?
11. QUALITY OF CURRENT STUDENTS (Pervez Hoodbhoy, Khurram Shafique):  Performance in standardized tests conducted every year, performance in local and international competitions such as Mathematics Olympiads and Programming Contests.

Notes:

(Pervez Hoodbhoy) Self-citations are a serious problem when using citations as a metric for measuring quality of research reform. [paraphrased]

(Abdullah Sadiq) While strengthening the research effort in the universities is important, the most urgent need is to concentrate on producing quality teachers for the lower tears of education. [paraphrased]

(Khurram Shafique) A pedestrian publication in the field of networking or multimedia is likely to receive more citations than a good publication in a less explored field in mathematics, say, non-standard analysis.

(Fakhruddin Habiby) another ‘tool’ that is used to push the citation number higher is formation of ‘citation-coalition’ within research groups.

(Omar Javed) Categorization of universities into subsets, and adoption of relevant performance criteria for each subset. Three fundamental questions: what is taught (Undergraduate and Graduate Instructional Program classifications), who are the students (Enrollment Profile and Undergraduate Profile), and what is the setting (Size & Setting)”

(Shafiqur Rehman) …the success or falure or HE reforms must only be judged by opinion of the common stake-holders (students, teachers and administration) of the public sector universities.

(Affan): we need to tweak ratings/rankings such that we are able to measure any progress happening in Pakistan, progress small enough that it is not lost by existing metrics.

There are four aspects of the comments of Dr. Pervez Hoodbhoy that need to be considered:

1. Firstly, Dr. Hoodbhoy himself admits that there has been a huge increase in international publications at QAU after HEC came into existence when he mentions the number of international publications in the two time periods. Strangely, he picks a six year period, 1998-2003, and then compares it with the subsequent 4.5 years (?) , 2004 to mid 2008, (the correspondence occurred in August 2008, so he could not possibly have had access to the figures for the entire year) I can only assume that he has mentioned 2003 by mistake in the second “5 year” period as there is no reason to include the publications of the year 2003 in both time periods, which he has done. It is clearly unfair to take two time periods of different durations and compare them.
2. In the first 6 year period (1998-2003), Dr. Hoodbhoy admits that there were only 631 research publications from QAU, but in the second 4.5 year period these had risen to 1482 research publications, a tripling of publications on average per year, even by his own estimates.
3. As the HEC programs began in 2003 and their real impact occurred 2-3 years later, a year-wise comparison is far more relevant than an average over a 5 year period as the dramatic change that has occurred gets partly masked when a 5 or 6 year average is taken, though it is still very visible. Dr. Hoodbhoy ignores the figures that Dr. S.T.K. Naim had worked out that in the year 2004, there were only 84 research publications from QAU (an average of only 7 publications per month), but by 2008 they had increased many fold.
4. The citations argument used by Dr. Hoodbhoy is invalid as citations increase with the passage of time. Dr. Hoodbhoy, therefore, wrongly compares the citations of papers of an earlier period with those of a later period. To clarify this issue further, if two papers of equal quality and in a similar field are published, say in 1998 and 2007, and the citations of both are counted in 2008, then the paper which was published in 1998 will have accumulated more citations by 2008 because of the much longer 10 year time period, than the paper published in 2007, as that would have had only one year for the citations to accumulate. Dr. Hoodbhoy is therefore comparing apples with oranges when he tries to compare citations of papers published in an earlier period with a later time period. In order to fairly compare citations, the same duration of time period must be taken. Thus if one takes 1998 publications and counts the citations till 2008, then one will need to take the 2008 publications and count their citations till the year 2018, before one can compare the figures for the citations of the two sets fairly.

The undeniable fact is that the total number of research publications from universities in Pakistan was only about 600 per year till 2001 but then started rising rapidly, and by the year 2008 it had increased to over 4,300! Brazil achieved such an increase over a 35 year period between 1960 to 1995, which Pakistan achieved in only 6 years. After my appointment in March 2000 as the Federal Minister for Science and Technology in Pakistan, I convinced the government to enhance the budget for science and technology in Pakistan by 6000% between July 2000 to October 2002. After my appointment as Chairman, Higher Education Commission (Federal Minister) the budget for higher education was similarly increased by 2400% during 2003 to 2008. Major achievements during these periods were:

1. Establishing 51 new Universities and awarding institutions during 2002-2008,
2. Tripling university enrollment (which had reached only 135,000 from 1947 to 2003) to about 400,000 in 2008,
3. Establishing a powerful Digital Library which provides free nation-wide access to every student in every public sector university to 45,000 textbooks/research monographs from 220 international publishers as well as to 25,000 international research journals,
4. Establishing video-conferencing facilities in most public sector universities that allow lectures to be delivered live and interactively to students in Pakistan from technologically advanced countries
5. Enhancing salaries of academics so that salaries of University Professors were increased to a level about five times the salaries of Federal Ministers, with a corresponding reduction in tax from 35% to only 5%, in order to attract the brightest young men and women into academia,
6. Promoting research through a massive research grant program which resulted in a 600% increase in ISI abstracted publications from about 600 per year in 2001 to 4300 research publications in 2008, accompanied by about 1000% increase in international citations in the same period,
7. Placing a satellite in space (Paksat-1) which is now used for distance learning by the Virtual University,
8. Establishing video-conferencing facilities in most public sector universities and initiating a lectureship program, allowing live interactive lectures to be delivered from technologically advanced countries,
9. Providing free access to scientists/engineers anywhere in the country to sophisticated instruments installed in any institute in Pakistan.

The Bottom Line: In the final analysis, it is not what I or Dr. Hoodbhoy think about the developments, but what is the opinion of neutral international experts who have carried out detailed year-long reviews of the developments during the period that I was heading the Higher Education Commission. A few extracts are given below:

1. Prof. Fred Hayward (independent international educational consultant from USA) carried out a detailed analysis of the developments and published an article entitled “Higher Education Transformation in Pakistan: Political & Economic Instability,” Date: Number 54, winter 2009 Source: International Higher Education Quarterly. I quote: “The news about Pakistan over the last few years has been dominated by reports of political turmoil, terrorism, religious fundamentalism, economic decline, and the Afghan War. What has been missed is the phenomenal transformation in higher education over the last six years, which represents a critical development for Pakistan and a potential engine for growth and national recovery.”
2. Report of US-AID about HEC states that “We are very impressed with the breadth, scope, and depth of the reforms implemented by the HEC since 2002.  No other developing country we know has made such spectacular progress.”
3. World Bank Report is very complimentary of many excellent programmes introduced.
4. British Council: The report states: “I have worked in many countries in South America, the Middle East, North Africa, and in Russia and India, over the last six years.  None in my view, with the exception of India, has the potential of Pakistan for the UK university sector, largely because of the dynamic, strategic leadership of the Chairman of HEC”.
5. Nature: Several articles and editorials have appeared in the world’s leading science journal “Nature”  (the most recent in the issue published on 3rd September 2009) in which the very significant progress made by Pakistan in the higher education sector has been applauded and the need for the new government to built on the solid foundation laid has been stressed.
6. Science Watch (Thomson Reuters) has ranked Pakistan as a rising star in five disciplines, more than in any other country of the world.

Join the Discussion!
What are the correct metrics to measure higher education reform in Pakistan?

Related Posts:

1. Nature’s Coverage of Higher Education Reform in Pakistan: A Response from Prof. Pervez Hoodbhoy
2. Nature’s Coverage of Higher Education Reform in Pakistan: A Conversation with Athar Osama
3. Pakistan’s Higher Education Funding Holds Many Lessons for Developing Nations: Nature

Editors Note: Dr. Atta-ur-Rahman is a leading scientist and scholar in the field of organic chemistry from Pakistan. He has served as the Federal Minister for Science and Technology, the Federal Minister/Chairman of the Higher Education Commission, Adviser to the Prime Minister on Science and Technology, and the President of the Pakistan Academy of Sciences. Dr. Atta-ur-Rahman is credited by many for reviving the higher education and research practices in Pakistan. The views expressed in this article are solely those of the author and do not necessarily reflect the views of STEP.

Unfortunately, I find the response as unsatisfying as the original article. Since Nature is unwilling to accord me a chance for a satisfactory reply on its pages, I shall clarify the basis of my criticism in some detail here.

In the said article, strong conclusions have been derived from weak data. The authors have not dared to ask the basic questions whose answers are essential for ascertaining whether there has been actual progress in Pakistan’s higher education system and, if so, by how much. Instead, in giving a thumbs-up, numbers have been quoted that have doubtful significance. Take, for instance, the claim that:

“In mathematics, for example, an average paper by a Pakistani author is cited around 20% more than the worldwide average for the discipline”.

Before returning to what some of the right questions might have been, let me give two reasons why the above claim – even if true – carries little meaning.

First, self-citation is a far more serious problem than the authors are willing to acknowledge. It is also one that they admit to not having investigated. The data on Pakistani research papers shows that subtracting out self-citations drastically cuts down on actual citations – there are often 2-3 self-citations for every real one! The reader is urged to carefully study my email correspondence of last year with Dr. Atta-ur-Rahman (see appendix below) who, at the time when he was HEC chairman, had made similar claims that I disputed as being false. To interested readers, I have made available (in pdf form) the Thomson Scientific data that I have quoted in my correspondence here (1998-2003) and here (2003-2008).

Second, many authors of the relatively more cited mathematics papers from Pakistani institutions are not Pakistani nationals. High salaries offered to foreign faculty by the HEC brought to Pakistan a large number of well established mathematicians on short-term contracts from Russia, Poland, Ukraine, and China. This was probably a good thing to do – in spite of the difficulty they had in communicating in an alien language and their consequent inability to teach well. Their papers, however, do not reflect mathematics in Pakistan. One sees a similar phenomenon in Saudi Arabia where foreigners are principally responsible for the kingdom’s large number of papers and citations.

In my opinion, instead of focusing on marginal matters, serious research on the state of Pakistani higher education, and of changes therein, would have first established appropriate metrics, and then sought answers, to the following key questions:

1. What is the quality of teaching in Pakistan’s public universities? There is often only a weak correlation between formal qualifications and subject competence, so simply counting PhD degrees is not very helpful in answering this. Far too frequently one sees professors of English who cannot speak or write a single sentence of grammatically correct English, physics professors who are stymied by Newton’s Laws, and biology professors whose knowledge is frozen in some pre-Darwinian age. But does such basic incompetence exist at the 20, 50, or 70 percent-level? Higher? Lower? What evidence exists that the HEC’s reforms improved the situation?
2. Is there evidence that there has been improvement in the selection process for students in public universities, or that of the quality of their graduates? Proof of the latter, judged by asking employers or assessing performance in international tests, would be a clinching argument for the success of HEC reforms.
3. Do campuses enjoy greater academic freedom, more seminars and colloquia, less violence by extremist campus groups, a pleasanter and more relaxed ambiance, and greater transparency in faculty selection? Surely these are critical to any reasonable assessment.

To get answers to questions like these requires extensive field work, and I certainly do not fault the authors for not doing this. But I was surprised that the Nature article, as well as the lead author’s response, merely says that the HEC’s experiment had critics, without citing any specific articles or the substance of those criticisms. There is not even a passing reference to the failed nine-university multi-billion dollar mega-project, tons of unused scientific equipment purchased for unknown reasons, dubious attempts to fund “Quranic Science” (that had to be hastily abandoned after the scheme was exposed), and the explosion in academic corruption set off by per-paper payments. Surely, these should not be brushed aside as “collateral damage”. In another country, those who massively squandered public money would have been thoroughly investigated by independent commissions, not praised for small things.

The Bottom Line: Well-functioning universities are the products of a complex organic and evolutionary process that is internal to a society. Money and facilities matter, but it is much more important for a university to have a forward looking world-view, an open environment, high ethical standards, a sense of collegiality and shared sense of purpose, and good governance practices. Sadly, the Nature article did not even mention these as significant.

Join the Discussion!
What are the correct metrics to measure higher education reform in Pakistan?

Related Posts:

1. Nature’s Coverage of Higher Education Reform in Pakistan: Comments by Dr. Atta-ur-Rahman
2. Nature’s Coverage of Higher Education Reform in Pakistan: A Conversation with Athar Osama
3. Pakistan’s Higher Education Funding Holds Many Lessons for Developing Nations: Nature

Editors Note: Prof. Pervez Hoodbhoy is a well-known Pakistani nuclear physicist and political-defence analyst. He is the Professor of High Energy Physics, and the head of the Physics Department at Quaid-e-Azam University, Islamabad, Pakistan. Prof. Hoodbhoy is a vocal critic of HEC’s policies and their impact. The views expressed in this article are solely those of the author and do not necessarily reflect the views of STEP.

This correspondence between Dr. Atta-ur-Rahman and Dr. Pervez Hoodbhoy is reproduced below with consents from both parties. It is exclusively concerned with a public matter, has no private content, and is largely focused upon the importance of self-citations.

Dear Dr. Pervez
I have probed into the situation at QAU, and it is quite the opposite of what you claim, as evident from the total publications and the total number of citations, as per above attachments. The HEC was established in October 2002. The funds started coming through from July 2003. The subsequent impact of HEC programs on research publications and citations is indeed amazing. In 2004 the number of publications was only about 120—by 2007 it has risen to about 380—a 300% increase! The citations in 2004 were about 800—-by August 2008 they have increased to about 2200 although we still have 4 months to go before the year ends—I suspect that it will be about 3200 by the end of the year—a 400% increase!

I shall be separately sending you a report after removing self-citations. QAU is also now included in the most cited institutions in the world (please see attachment)—-this was not to 4 years ago.

Please be fair and objective in your assessments. We may have made some mistakes, but much good has happened.
Kind regards
Atta

Dear Dr. Pervez
Following my other email to you this morning, I am now enclosing the citations of QAU after removing self-citations.As you would see, these have risen from only 84 in 2004 to 1413 in 2008 (with still 4 months to go). These will probably be around 1900 by the end of the year—a spectacular ten-fold growth! Dr. Naim has kindly had these searched, so if you have any queries about them, you may like to interact with her. She tells me that the situation is similar in many other universities—a long period of stagnantion
during the 1990s followed by a burst of activity in the last 4-5 years.
Kind regards

Atta

Dear Dr. Atta,

I am sorry about the late response to your three emails. First, thank you for doing whatever you did – the HEC’s notification of 14 July 2008, which specifies 40 percentile as the GRE passing marks, finally reached QAU departments today (without comment from the administration). In these times one has to be grateful for small things…

Regarding your email and the numbers cited therein: I can understand that you would like to feel upbeat about QAU having improved itself as a result of massive infusion of HEC resources. I too would very much like good things to happen, but perhaps one should not allow wishes to become conclusions.

Looking at the data that you had sent me and your inferences, I decided to do a little independent investigation using exactly the same database (ISI Web of Science) and exactly the same keywords (see attachments to this email). Here are the findings:

Period: 1998-2003
Number of papers published by QAU authors in the above period: 631
Number of citations to date: 4540
Number of citations to date with self-citations removed: 2,817

Period: 2003-2008
Number of papers published by QAU authors in the above period: 1482
Number of citations to date: 3667
Number of citations to date with self-citations removed: 1258

Some obvious inferences:

1. There is absolutely no evidence of real citations having increased; if anything, the numbers up to now show the contrary. While the number of real citations of papers published 2003-2008 may increase somewhat with time, currently they are quite a bit less from the earlier period when the HEC and your incentive system did not exist. Please remember that citations are cumulative over years. I have tried to use exact descriptions in the figures cited above. If I am wrong in any detail, or if I have missed something essential, I would like to be corrected. Unfortunately the data does not at all support your rather optimistic remark of “a spectacular ten-fold growth!”
2. The above data also indicates the disturbing fact that most of the time QAU authors cite themselves. Subtracting self-citations drastically cuts down on real citations – there are 2-3 self-citations for every real one!. Looking more minutely at the ISI pages, one also notes that many citations are by other members belonging to the same or other QAU departments. So the number of genuine citations gets cut down even beyond the numbers quoted above (2817, 1258)!

Just to get scales right, here are a few citation numbers from the Stanford Spires data base for Pakistani physicists over the length of their careers:
a) Riazuddin: 1479
b) Ahmed Ali (DESY, Germany): 9873
c) Abdus Salam: 14103

3. You are correct that the number of publications over equal (5-year) time periods has more than doubled relative to pre-HEC times. But this is clearly in response to the monetary incentives offered by PCST/QAU. A publication fever now grips our universities. It is difficult to defend the case that the number of papers published is proportional to the amount of research done.

You are, of course, free to have the data I have quoted above rechecked and I would be happy to answer any question that arises. Finally, please note that publications and citations were not central to my earlier expression of dismay at the quality of QAU education.

What bothers me much more is the lack of analytical and problem-solving abilities of our Ph.D graduates, some honourable exceptions aside. Poor performance in the GREs is one indication of the rot. This fact has indeed worried you a little, as you indicated in an earlier email to me, but I do wish you could understand the real gravity of the situation.

Regards,

Pervez

Dr. Atta-ur-Rahman’s comments on this post can be found here.

STEP: How did the idea of this study come about?

The idea for the article actually originated from Nature’s Editors late last year when HEC’s budgets and utility were being increasingly questioned by the new government in Islamabad. Nature had been an observer – and sometimes a commentator on HEC’s initiatives – and approached me to see if I’d be interested in writing an independent and objective piece on Pakistan’s Higher Education Reforms. I worked with Nature’s editors to put together a group of authors who we believed brought a diversity of experience, insight, and credibility to this exercise.

This preliminary review, I hope, will be a starting point towards an extensive process of external policy peer review and a constructive self-examination leading to greater effectiveness of higher education policies and programmes in Pakistan. Another very important aspect of Nature’s interest in this work was to see if other countries contemplating similar reforms could learn some lessons from Pakistan.

STEP: How long did the study take? What data sources and methodologies were used?

We began working on this subject with the Nature piece in mind about 8-9 months ago. Obviously, the first step was to approach HEC itself and to seek their cooperation in doing this. There was really no point in saying anything on the subject without having access to HEC’s own data and viewpoint.

We had excellent cooperation from Dr. Atta Ur Rahman as well as Dr. Sohail Naqvi and other members of HEC staff. In addition to HEC, I also met and talked to a number of university leaders as well as HEC’s critics including Dr. Hoodbhoy. Dr. Naqvi made available to us a lot of data that we believe had not been publicly available in this manner before. We were cognizant, however, that this is self-reported data and that any conclusions we will make based on this data alone are going to be fiercely challenged by HEC’s critics.

In order to balance HEC’s own view, we also used external data to validate the claims made by HEC. Specifically, we decided to use Thomson Reuters’ data on scientific publications as independent measure of Pakistan’s publication activity and impact. We must emphasize, though, our analysis is merely a start. One of our key recommendations is for HEC to subject itself to an independent external peer review of its policies and programmes and use their guidance to improve these. This peer review must also include taking a look the current data at hand and the data collection systems in place and to improve these to provide continuous guidance to HEC planners.

STEP: What was the most important (or surprising) finding of the study?

I believe that this is still a work in progress as Pakistan’s reform experiment is itself a work in progress. I won’t speak for other members of the team here, however, I was personally surprised by quite a few findings. First of all, there does seem to be an unambiguous improvement in Pakistan’s publishing performance as well as its relative impact. The publications have tripled over the last few years and there are significant gains in relative impact of Pakistani papers. Our preliminary testing shows that this finding is not too influenced by self-citation bias. One negative finding – though not unsurprising – was the decline in the publishing performance in social sciences (and business and management and humanities etc.). This is a bit alarming – for me – because I do believe that well developed social sciences are very important for a balanced development of a nation.

I believe that several programmes may be rightly guided – even though there may be implementation lapses. The foreign PhD programme is one example. I believe, and hope, it will deliver in the medium to long-run. The NRPU – the competitive grant funding programme – is in the right direction – although the peer review process needs to be strengthened. I think that a 47% acceptance rate is just too high – especially given that a vast majority of Pakistani faculty have not had much exposure to competitive research for considerable periods in the past. The investments in the digital library and internet connectivity are also critical elements of the research infrastructure.

On the other hand, there are still several things that are not so right. There are things that can potentially go wrong – seriously wrong – if important issues are not addressed. I am especially concerned with the policies vis-à-vis the Domestic (Indigenous) PhD programme. That is certainly something that should be subjected to greater scrutiny and evaluation from both outside and inside of HEC. Also governance and management reforms within universities have been rather slow. However, I think there is a need to take key stakeholders along rather than force it on universities.

STEP: The statistic that I found most interesting in the article is the +20% relative impact of Mathematics publications from Pakistan. However, Dr. Hoodbhoy, in his critique of your article, asked the question of whether self-citations were removed or not in computing this number. To quote Dr. Hoodbhoy, “did the authors try to eliminate self-citations (a deliberate ploy) from this count? If they had – as I did using an available option in the Thomson Scientific package – they might actually have found the opposite result.” How would you respond to this criticism?

The short answer to Dr. Hoodbhoy’s question is that no – this number includes author self-citations. BUT we were definitely concerned about self-citations. Here is what we did to account for this potential bias.

We extracted a database of roughly 13,000 papers (representing all fields) indexed by Thomson Reuters between 1999 and 2008, each listing at least one author address in Pakistan. We also extracted all papers that subsequently cited these papers. We then enumerated instances of ANY country citing into this population of papers (i.e., all nations citing Pakistan), along with a separate measure of only Pakistani authors citing Pakistani papers. We did this for two distinct time periods: for papers published and cited during 1999 to 2003, and during 2004 to 2008.

For the first time period, there were 8,436 occurrences of all nations citing the Pakistani papers (or, to repeat, those papers bearing at least one Pakistan author address), and 1,527 occurrences when Pakistan-only papers cited the group. In percent terms, 1,527 of 8,436 equals 18.1%. Thus, by this measure, Pakistan self-citation could be quantified as 18.1%. For the second time period, the comparable numbers were 26,294 instances of all countries citing the Pakistan group, and 6,597 instance of Pakistan citing Pakistan. Expressed as a percentage, the figure is 25.1%

To summarize: 18.1% Pakistan self-citation for the 1999-2003 period, and 25.1% for 2004-2008. Now it is important to understand that country self-citations are not a perfect proxy for author self-citations – the variable that we’re really interested in. However, author self-citations are a subset of country self-citations and hence country self-citations do give us an upper bound on author self-citations. That is, author self-citations for Pakistani papers are definitely less than 25.1% – and perhaps a lot less than that – in the post-reform period.

We can compare this figure with generally accepted empirical norms of author self-citation globally. In Gami et al (CMAJ, 2004) the authors estimate self-citation in diabetes research at about 18%. Falagas and Kavvadia (FASEB, 2006) arrive at self-citation rates in biomedical research at 17-20%. Others have confirmed the general trend in medical literature of around 1/5^th of all citations as being author self-citations. Garfield and Sher (1964) arrive at a self-citation figure of 20% for basic research papers. Trimble (1986) arrives at a figure of 15% for all astronomical papers. Bonzi and Snyder (1986) found an average self-citation rate of 11 per cent across a range of disciplines, varying from 16 per cent in the physical sciences (chemistry and geology) to 3 per cent in the social sciences. The frequency of self-citation is also found to be independent of quality of publication.

So, Pakistan’s likely author self-citation rates may be only slightly higher – if not within – the global norms for self-citations. It is quite unlikely that all – or even most – gains can be wiped out by author self-citations. Dr. Hoodbhoy may be right in that self-citations may have increased from the past but that is likely to be expected for a variety of reasons – some of which quite legitimate. For example, as a closely knit research community develops within a country and interact through local conferences they are more likely to be aware of each others work and hence cite authors from within the country. Had we been able to exclude self-citations from these relative impact figures, these results would perhaps have been only marginally smaller. Because we were dealing with publications across multiple disciplines over multiple time periods and across multiple countries we could not correct relative impact figures for self-citation. However, given the above analysis we don’t find a reason to feel so alarmed as to totally rubbish the entire publishing performance of the country.

Should this be an issue HEC should carefully watch and analyse? – definitely yes. Should this totally nullify the improvements in Pakistan’s publication record? – we don’t believe so.

STEP: You mentioned that a lot of data provided by the HEC was not used in the article because of the space limitations. The only real statistical analysis in the current article is the table on relative impact factors. What other metrics could have been used to scientifically analyze the performance of HEC, had space not been the limiting factor.

HEC did provide a lot of data and complied with our requests for additional data as much as it could. Potentially, data could inform a number of very interesting questions vis-à-vis the performance of higher education reforms in Pakistan. We were not able to undertake some of these analyses because of the following reasons.

First, a lot of data currently available only accounts for inputs to various programmes – number of PhDs awarded, number of fellowships granted, number of grants awarded are input data. This needs to be complemented with relevant and comparable output data so that outcome assessment can take place.

Second, for data to be useful for subsequent analyses, the analytical and evaluation plan must be built into the programme itself. It is much harder – and more expensive – to cobble together data – ex post – and come up with a very clean assessment of outcomes.

As an example of the sort of things that additional analysis could do is to assess whether the foreign faculty hiring programme (FFHP) is an effective investment. During my meetings with vice chancellors I was told that there is a backlash against FFHP from the faculty and one of the allegations is that the people recruited on FFHP are no better than some of the better faculty members available within the country. This, and several others elements of the reform, are empirical questions that could be easily addressed by intelligent use of data.

STEP: Don’t you think that the HEC is under more scrutiny (at least informally) than other government departments? For example, I don’t see any debate on the performance of, say, MINFAL. The team at HEC has, at the very least, brought about many positive changes compared to the UGC of the past. Do you think there is a chance that overly negative criticism can be counterproductive and may deter such initiatives to positively reform a government department in future?

Yes, certainly there is a chance that overly negative criticism may be counter productive. However, I believe that what we have tried to do is to present our best objective view of the reforms. I believe that it’s a pity that the debate on this important set of reforms hovers between the two extreme viewpoints. I believe that good data and analysis can really refocus this debate and turn it into constructive one. Good data and analysis can move debate away from one of opinions to one of facts.

I believe it is important to have this debate in an open, transparent, and civilized manner. Such a debate will inspire greater trust and buy-in from the people and systems this reform is seeking to address. Pakistan has invested upwards of 50 billion rupees in higher education over the last 5-7 years and it is in all our interest that we make the best use of this investment.

STEP: One of your key recommendations is for the HEC to be reviewed externally. What are some of the examples (from around the world) of the external peer-review process that you suggest HEC subject itself to? Should this process not be done through the Standing Committees on Education in the Senate and/or the National Assembly?

The process of policy peer review is well-established – especially in the west. The nearest equivalent that comes to mind is the peer reviews organized by the National Academies in the US of major government funding programmes (such as the SBIRs, ATP etc.). These are panels of eminent scientists (Academy Fellows) and relevant experts looking over various aspects of these programmes. Because of the unique circumstances and pervasive nature of HEC’s programmes, HEC’s peer review would have to be done by a mix of Pakistani and internationally placed members. The findings of these reviews are then debated by legislatures who have the final say in determining policies informed by these findings and recommendations.

STEP: Based on your study, how do you compare HEC in Pakistan to other equivalent bodies in the developing countries?

I believe HEC has done a considerable amount of work in a very short period of time and some of it may have already begun to reap results as well. It has tackled a very difficult area of reform and has shaken the academic environment in Pakistan. In the depth and breadth of what it has tried to do, HEC has very few parallels in the world today. Most other relevant bodies have only tinkered with the systems that they have tried to reform. It is this factor that probably prompted Nature to invite an opinion piece from the authors and an editorial commenting on these reforms and the lessons that could be learnt by other developing countries. HEC had the liberty and resources to do so and it made the most out of it. Whether or not this hugely ambitious experiment will succeed at the expected level in the end remains an open question.

STEP: If this article is work in progress, as you have said, what further study or analysis can we expect from your team in future?

We are planning to finish a larger piece on higher education reforms in Pakistan and hopefully publish this in the near future. In addition, we intend to dialog with HEC, leading academics and academic administrators in Pakistan to explore how the suggestions in our article can be best taken forward.

Related Post: Pakistan’s Higher Education Funding Holds Many Lessons for Developing Nations: Nature

STEP: According to recent estimates, less than half of Pakistan’s population is literate, less than half have access to basic sanitation, and the economy is strangled by debt. In context of this, what is the social relevance and value of the modern university, with its emphasis on research and higher learning, in Pakistan today?

PH: Pakistan’s social indicators are indeed abysmal. But no country can wait for everything and everybody to get up to speed before making universities. Nor should it, because that would essentially mean waiting forever. But we should remember that there is a difference in the purposes that universities serve in countries like Pakistan, and in advanced countries like the US. The latter have knowledge-driven economies, and universities function as the engines of progress. They are the fountainheads of modern science, and of new technologies that have changed the world more in the past fifty years than the previous ten thousand years.

In Pakistan, our universities do not produce much new technology or ideas.  Nevertheless their graduates are necessary to keep the country going. Else the country would not have engineers, technicians, doctors, and administrators needed to run institutions, factories, businesses, and government.

There is another reason for a country to have universities – and this is quite independent of whether they produce state-of-the art research or not.  Universities are needed to create a modern citizenry capable of responsible and reasoned decision making. Their graduates should be able to think independently and scientifically, have an understanding of history and culture, create discourses on social and political issues, and be capable of coherent expression in speech and writing.

The fact that our universities do not measure well on this score is deeply regrettable. Yet, this suggests that we should strive to improve them, not eliminate them. At the same time, although buildings can rather easily constructed, Pakistan’s very limited intellectual resources put strong constraints on the number of actual higher education institutions that it can have.

STEP: Beyond their role as educational institutions, what is the value of emphasizing research, specifically theoretical and technical research, at universities in Pakistan?

PH: Research on the theoretical aspects of a subject is important for two reasons. First, genuine research, even if it is not cutting edge, makes the individual teacher much more aware of the state of the field and hence a better, more exciting teacher. Book knowledge becomes stale fast, particularly these days. Second, knowledge is advanced only through research, and Pakistan should play a role in this some day. India already is doing so, and Iran has begun to as well. Theoretical research is intellectually harder and more demanding than experimental research, and it consumes far fewer resources. Thus it should be strongly encouraged.

But since “research” is a widely abused term in Pakistan, some careful consideration of its meaning is necessary before attempting to evaluate its current importance in our universities. Research in any professional field — mathematics or physics, molecular biology or engineering, economics or archaeology — does not have a unique, precise definition. But a tentative, exploratory definition might be that research is the discovery of new and interesting phenomena, creation of concepts that have explanatory or predictive power, making of new and useful inventions and processes, etc. In the world of science, the researcher must certainly do something original, not merely repeat what is already known. Just doing something for the first time is not good enough to qualify as research. So, for example, one does not do meaningful research by gathering all kinds of butterflies and listing the number caught of each kind in a particular place at a particular time, etc. Nor does it come from making standard measurements, substituting one material after the other just because “it’s not been done before”.

We must recognize that very few Pakistani universities and their faculty currently have the capacity for real research. Nevertheless, they can still function quite well as knowledge transmitters. For example, some of Pakistan’s elite private universities have good teaching standards although they have few journal publications at this stage of their development. My feeling is that if a university teacher does not have the physical, material, or intellectual resources to do genuine research, it is far better that that person be made to improve his or her pedagogical practices as well as subject understanding. This is far better than churning out junk papers, which no one reads.

STEP: You have been a leading critic of some of the policies the HEC has initiated to address the state of research in Pakistan. Let’s say you are given Rs 21 billion (HEC’s 2009 budget) and stewardship of an organization with a mandate to reform universities in Pakistan. What would be the three most pressing items on your agenda and how would you go about instituting them?

PH: I would shift priorities drastically and emphasize improving the physical infrastructure of the 1000+ colleges rather than pampering a few public universities. Of the available money and effort, I would put 90% towards improving teaching quality at our public universities and colleges. Only promising research would be supported. Today’s atrocious teaching quality comes largely from having university and college teachers with very poor knowledge of their subject. Therefore I would call for the following:

1. Require that every applicant for lecturer or assistant professor, either at a public university or college, pass a relevant internationally administered examination (such as the GRE subject test if one is available in that field, else the GRE General Exam). The test would ensure that that person has enough basic knowledge to properly teach the subject. The applicant would also be required to give an introductory lecture, open to all who wish to attend, on a subject belonging to the applicant’s claimed field of expertise. The entire process of teacher selection needs to be made transparent and above board.
2. Create large-scale teacher-training academies in every provincial capital. Established with international help, these academies should bring in the best teachers as trainers from across the country and from anywhere in the world. A few master trainers might be willing to come from western countries in spite of the security situation, but hopefully attractive salaries might be able to lure some from India or from outside the Western world. These academies must be on the scale of a mega-project, say on the order of a billion dollars over 5 years. As high-quality institutions, they should have a clear philosophy aimed at equipping teachers to teach through concepts rather than rote learning, use modern textbooks, and emphasize basic principles of pedagogy, grading, and fairness. To be effective, they must be degree-awarding institutions.
3. Build on various current HEC initiatives such as foreign faculty hiring and scholarship schemes for university teachers. There are simply not enough qualified persons within Pakistan to adequately staff university departments. The fact that these schemes have been mismanaged by the HEC should not prejudice one against their potential usefulness if proper procedures and rules are adhered. Those selected for overseas scholarships should be required to clear an international subject test.

STEP: You note that “university and college teachers [have] very poor knowledge of their subject.” Yet, the scope of the teacher-training academies would presumably be pedagogical technique and not the outright re-education of teachers in their subject material. Is a multi-billion rupee investment in pedagogical training worth it, when subject proficiency seems like the fundamental problem?

PH: Thank you for forcing me to clarify. I very much have subject proficiency in mind. In fact, in the proposed new teacher training institutions I would give 90% importance to re-teaching subject basics and only 10% to pedagogy. So, in fact, teaching teachers “teaching-methods” is a very distant second priority. Let me say that those studying in these hypothetical NFAs (National Faculty Academies) would be relearning materials that they are actually supposed to know from their time in college or university. But there would be a crucial difference: this time they will be graded not by how much they have memorized but how well they are able to use what they have learned in order to solve problems. In science, knowledge is useful only if it is internalized rather than memorized. It must become part of your mental tool box.

There would be another important side benefit to having competent teachers. I am convinced that if a teacher knows his or her subject and is able to comfortably solve all or most of the problems at the end of a chapter, it would lead to important attitudinal changes. Some of the authoritarianism of teachers would surely go away. It is a fact that teachers often discourage students from asking questions because they know that their lack of understanding would be exposed. This is lethal for an academic environment.

STEP: Your proposal has a parallel to the erstwhile universities mega-project in that, rather than reforming and investing in existing universities, it recommends creating entirely new institutions. Why the inclination to create new academies instead of focusing resources and effort into reforming existing programs?

PH: Suppose you had inherited an airline company but no pilots. Would you like novices to take your planes up in the hope that they will learn flying that way? Of course not! Similarly we have entire universities, but with almost no people who are fit to teach in them. But they still teach, and nobody stops them. So although we don’t have crashed planes, we have armies of university students who graduated but didn’t survive their mis-education. Therefore, they could never become good scientists, engineers, economists, or whatever. In the hard sciences, I’d estimate that a miserable 20-30 percent of university teachers are actually qualified to teach — and I’m being generous.

To fix this situation, I just don’t know of any way other than training teachers in dedicated, specially created, teaching institutions where, at the end, they would be required to show proof through proper examinations that they’ve learned their subject well enough. It’s like a pilot certification requirement. If you don’t pass, you are not allowed to fly — or teach.

To respond specifically as to why we need new institutions: it’s because we just don’t have any teacher training institutions with anything close to the required intellectual capacity. It’s not about reforming something that presently exists but which is not good enough; nothing presently exists where college and university teachers can be adequately taught subject basics.

I might add one caveat: creating any good educational institution in Pakistan means that we will have to get at least some key people from other countries. Unless Pakistan stabilizes and deals with terrorism effectively, no persuasion will ever succeed in bringing them here. Or, perhaps, even expatriate Pakistanis. So this is a super-priority.

STEP: Why did you choose to return to Pakistan after your bachelors and masters degrees from MIT?

PH: Like some others of my generation, in the early 1970′s I was witness to the huge political upheaval in the US. American students were staging protests against their own government over its wrong and immoral war in Vietnam. Hitherto I had regarded politics to be a mere game and had barely any interest in these matters. As a naïve middle-class apolitical Pakistani youth, it seemed totally unbelievable to me that MIT students would be protesting against their own government and country — and that too when it was at war. There were huge protests, boycotts, and even occasional violence. I remember witnessing the violent protests against the Charles Stark Draper Laboratory on campus, which was involved in MIRVing nuclear missiles. It was so liberating for me to see people follow the dictates of their conscience. Now a part of the anti-war movement, I fully understood the ugliness of imperial power and participated in the teach-ins and sit-ins. The atrocities that the US was committing in Vietnam had made me so very angry that I did not want to live a day longer in America than was necessary to finish my degrees.

Then, closer to home, there was the slaughter in East Pakistan being carried out by the West Pakistani army. At the same time, there was a movement for social change in Pakistan that promised socialism and justice for the masses. It was initiated by Zulfiqar Ali Bhutto, who brought revolutionary politics to Pakistan. What happened to him, and how he reneged on his promises, is another story but those were times of immense hope. I was one of the many overseas students who went back to Pakistan dreaming of changing everything, and of replacing feudalistic and capitalistic exploitation with socialism. So, with a job in hand at Islamabad University (QAU went under this name in the 1970′s) I joined up with others who had also recently returned and we became part of a workers movement in Rawalpindi, known as People’s Labour Federation. With another group of friends who were inspired by the idea of a peasant revolution, I became involved with working as a paramedic and school teacher in a remote Potohar village.

Part 2 is shared here.