When did your involvement with GIK Institute start? What was the vision for GIKI and, after 17 years, is GIKI where you envisioned it to be?

I was among the people, including several foreign professors, invited by then-President Ghulam Ishaq Khan — around 1989-90 — for  brainstorming sessions about the proposed Institute. The idea of starting a private university in science and engineering was quite novel [at the time] for Pakistan, and it was meant to be an instrument for breaking out of the mediocrity trap gripping Pakistan’s other engineering Universities.

My dream was for GIKI to become a community of self-governing scholars, a place where reason and innovation would rule and where the only thing that matters – the quality of student emerging from it – would be at the center of all our efforts.  These graduates would also be proficient in their work and aware of their own cultural heritage as well as those of other people, and  imbued with the processes of (mathematical) reasoning. This dream is yet incomplete as it requires civilizing the engineers and scientists also!

What are the obstacles to achieving this vision?

The reasons go back to the founding of the Institute. [At the time of its founding] I disagreed with Topi as the site, and preferred an urban setting near Nowshera, on the main highway and close to a strong industrial cluster, since it was going to be set up in the then-NWFP (now re-named as Khyber Pakhtoonkhwa). President Ghulam Ishaq Khan may have been persuaded by his close associations with WAPDA to set it up next to Tarbela Dam, but this decision has been the fatal flaw holding back GIKI from reaching its true potential. It hardly mattered in the years before HEC started injecting funds into the higher education sector but now it is critical, as GIKI has little access to public funds. My fear is that GIKI will price itself out of the market.

My other recommendation was also not followed in letter and spirit. I had done my undergraduate from Oxford (its engineering department was called the Engineering Sciences Dept .), and remembered it as a program which enabled one to go into manufacturing, or research, or business [after graduation]. It stressed the blurring of boundaries between engineering and physical sciences. GIKI’s name does have the words “Engineering Sciences” in it, and there is a faculty of Engineering Sciences, but it is not really in the same spirit of the 100-year old model at Oxford, where every student had to take all subjects (electronics and electrical engineering, computers, heat engines and thermodynamics, mechanical and other civil structures, fluid flows), and then take 3-4 additional advanced courses [in the area of his or her specialization].

When were you brought on-board as the Rector, and when and why did you leave?

I was asked and agreed to become the Rector in June 2008 to but left in January 2009 for one major reason. The Taliban fell in love with me and accused me in a letter in November 2008 of spreading immorality and organizing mehfils of ‘raqs-o-saroor‘, apart from espousing the ideas of the Americans and the Jews, etc., etc. Also, I am quite an independent person, one who has made his own decisions, and it riled me that their shoora had decided to ‘send me to jahannum’ without asking me first!

My focus was always the well-being of the students and to see that they got their money’s worth. I interacted with faculty and students intensively to improve the delivery of education. I also pulled up the administration for their general apathy. I insisted on hygiene and cleanliness in the hostels and dining halls, started improving their sports facilities, and offered to arrange proper music lessons. I encouraged them to patronize local Swabi talent rather relying on relatively expensive pop concerts. I also asked them to be irreverent — to avoid obscurantist dogmas by questioning assumptions, and to remember that no mullah is remembered in Muslim history, while scholars such as Ibn Khaldun and Bu Ali Sina are honoured.  Imagine my frustration at not being able to have even a discussion with the Taliban!

The students were my first concern and without raising alarms, I brought this [letter from the Taliban's shoora] to the notice of the Federal Interior Ministry, which helped to increase police patrolling on the two major roads to Islamabad and Peshawar to prevent any harm to the students. My family came to know only in Jan 2009 and were quite hysterical. So, I decided to quit. Remember, Swat was only an hour away and the military operation against the Taliban did not start for another 10 weeks. There was also the strange case related to the revival of the hair cutting saloon on the campus for female students and faculty wives. This was opposed by a couple of senior (!) faculty wives as being un-Islamic. Incidentally this facility is doing very well.

But, you’re still associated with the Institute (as the Executive Director of SOPREST). What are you working on now?

After the sad demise of Mr. H.U. Baig, I was asked in March 2010 to take over as Executive Director of SOPREST, the society which runs GIKI. I have done so on the understanding that we will be working towards setting up three new Schools of Business, Public Policy, and the Social Sciences in Islamabad under the SOPREST banner.  I am happy to report that the BoG of SOPREST approved this program on its meeting of 17th September.

This new campus is expected to have some 2500 students in place in 10 years. It will provide an integrated approach to business, management, public policy, and simulations and modeling of issues pertaining to problems peculiar to this century, such as security and affordability of energy, water and food. We have requested 50 acres from CDA near Rawal Dam, while a partnership is possible with another Foundation on a 300 acre site near DHA/Bahria.

The support of GIKI alumni will be extremely critical in making this a success. Our alumni  have made a name for themselves, in Pakistan and abroad, and I request them all to support us with suggestions and networking for acquiring talented faculty and, of course, donations. Their advice and experience will be extremely valuable for making GIKI a true University. Our target is to raise some 50 % of the Rs 1.2 billion we will be spending on the venture in the next 5 years. We intend to manage the remainder amount.

A final question about GIKI. Your son was a student at GIKI back in the 90s. If you had the option of sending your son to GIKI now, would you still send him there or to another institute?

[Laughter]. I will probably send him [to GIKI]. It is still one of the best places for engineering in Pakistan.

Moving away from GIKI/SOPREST… You did some pioneering work on lasers in Pakistan, work that can be an example for scientists returning to Pakistan. How did it all start, what did you accomplish and do you see a bright future for the work you started?

It was really exciting! I started as a one-man laser group in 1969 but gradually we developed a very good team. We all complemented one another and we all gave generously of our time.

We built lasers, we used them, and we generated over five billion rupees of revenue through product development over 20 years. Our lasers are leveling farm land in Pakistan and reducing water consumption by more than a third.  I met the Director of the Biotech Institute in South India recently, and was pleased to know that 3 of our land levelers were purchased by them for reverse engineering. Recently, I helped design the position monitoring system for the thousands of detectors in the CMS at CERN in Geneva. All 40 systems have been made in my labs and have been incorporated at CERN’s CMS, and our lasers and precision optics have been used in Germany, Switzerland, and Spain apart from S.E. Asia.

I lapsed from active science in 2005 when I joined the Planning Commission but I’ve been back to my lab about 4 to 5 times, though we stay in touch. Lasers has a good future in Pakistan, and is in good hands.  The National Laser Labs is now being put together, and will commence shortly. I am content that there may now be more people working in lasers in the Pakistan Atomic Energy Commission than in nuclear physics!

You’ve had a remarkable career in academia, in research labs and in the government as member of the Planning Commission. What do you consider to be your most significant accomplishment? How would you like to be remembered ?

An embarrassing question! I think my legacy would be the starting of a completely new field in Pakistan: Lasers.

Spoken like a true scientist! One final question. Did your work in the government make you more hopeful about Pakistan’s future or less?

Planning Commission was an enormous learning experience for me. I initially looked after science and technology, higher education and industry (a good combination) and later education and health also. This was my first experience of working in the ‘Government’ and I was a bit surprised at the lack of institutional memory and just downright laziness and incompetence. I hope I raised the quality of discourse and analysis.

My biggest challenge [at the Commission] was the Vision 2030 project – trying to identify the most likely future for Pakistan among the many that were possible or desirable. This document is now largely forgotten in the middle of the political changes of the last two years, but going through this exercise really gave me a lot of hope about the future of Pakistan.

I am confident about the role for Pakistan in this century. We are not too small as to be irrelevant, in fact we are the about right population size and our younger people carry far less historical baggage; they are enterprising, more selective and also more demanding in terms of quality.

On that optimistic note, thank you very much, Dr. Khan and our best wishes.

Thank you.

STEP: Moving on to the topic of higher education. Do you think that the level of financial support that higher education, in general, and the Higher Education Commission, in particular, is getting from the government can be sustained?

Asad Abidi: It is not getting (a lot of support) or it might be getting it for a moment but, you know, Pakistan is bankrupt and all this investment is from borrowed funds from the future. The typical elected government is just running scared, trying to keep its head above water. And, unfortunately, this is not going to change (anytime soon). So, the question is how do you take a country with so many needs and keep higher education running? The only way I can see it happening is if a substantial allocation, such as from the military budget, is diverted toward higher education. The military has never deprived itself of money. In the worst of times, their budgets have gone untouched, their privileges have gone untouched. But, it will take a political leader with guts to do this.

Honestly, I think the only way it could happen is if the United States, which effectively supports the Pakistan military, were to say that ‘we don’t really believe that it is valuable to add more men to your forces or add to your existing perks and privileges; this is actually only going to lead to more disenchantment from civil society and unrest in the region. So, you must cut your budget by, let’s say, 10% or 15% and that this money must go into higher education to deliver some hope to Pakistan’s people. Otherwise we will withdraw our support’. Only then might things change.  So, it’s going to be really hard unless you have massive civil protest in Pakistan. I don’t think Pakistan is quite ready for that kind of thing yet. People dispirited by spiraling inflation, power outages, unemployment, kleptocracy, can hardly be expected to rally in numbers against a bloated military budget.

STEP: Can private endowments, funded by wealthy individuals perhaps, fill this funding gap?

AA: Well, there are precedents (of that) in the region. We have the Tata Institute of Fundamental Research (established 1945) and the various Birla Institutes of Technology and Science (established 1929) in India, which are funded through endowments from these families. So, in India wealthy people have funded expensive science education over long periods of time, with world-class outcomes. In Turkey, there are institutes and private universities, such as Koc University, that have large endowments from industrialists. In Pakistan, perhaps there is not that scale of money, certainly not that scale of investment in a single institution to fulfill its financial needs in a sustainable manner (except, of course, for the Aga Khan University).

Another important point is that the Tatas and Birlas believed in a vision of India’s future. Today their institutions are run by trustees, often with some membership from those families. But, the families don’t meddle in affairs of the institutes. They continue to provide very generous funding, but place their faith and trust in the scientists and educators who work there. I noticed that in Pakistan, institutions seem to rely entirely on the power and charisma of one person — that’s one extreme — and on the other extreme, you have institutions that demand transparency in everything. That essentially makes it impossible for administrators and decision-makers to work, and it’s not healthy either. You very rarely find institutions in Pakistan that have found the right balance. Pakistan has yet to mature in these matters.

STEP: But, isn’t the ‘disconnect’ between higher education institutions and the industry also responsible for the lack of private funds?

AA: Yes, that’s also true. In fact, there is a large breach between industry and universities in Pakistan. The reason for this, I think, is that Pakistani universities, again, have not yet grown up. People like Qasim Shiekh (CEO, National ICT R&D Fund) do demand technology transfer and we certainly need to hear more of that, but I don’t think currently there are many examples of successful US-style transfer of technology from a university to a company, where the academics involved also furthered scholarship in the process and published papers. There may be some good synergy between the military establishment and certain institutions like NUST and CASE, but it’s funded by the military, I don’t know how much of it can be published, nor how much is publishable. Unless universities are on guard, these arrangements can turn them into job shops. In any case, this sort of activity is just not happening at a large scale in Pakistan yet.

Take the example of the indigenous pharmaceutical industry (I exclude herbal and natural medicines). They are doing zero research. Zero. One of the reasons is that there are no pharmaceutical scientists of any standing who can go and develop their own formulations. Another is that the companies cannot see the payoff. This is true for so much else in Pakistan, yet much the reverse in India. Pharmaceuticals and synthetic chemical companies there are doing a roaring global business. I am told that tractor companies in Pakistan have tried to engage professors as consultants but it was a complete write-off. If your applied sciences cannot make tangible contributions to the economy, you’re just dreaming about the benefits of higher education. Again, we must understand that there will be a good deal of fumbling and missteps at first, but science and engineering academics in Pakistan must discover ways of closing the breach with industry.

Finally, Pakistan is one of the few countries in the world where vocational training institutions in any numbers either didn’t take off, or failed. Korea industrialized itself on vocational education. So did Japan, the UK, and Australia, all on different models. That is how countries industrialize. So, to take the opposite point of view, Pakistan doesn’t need more universities; in fact we have far too many as it is, because their graduates find it hard to gain employment, and cannot compete in numbers in the global marketplace for PhDs. What Pakistan needs more is vocational training of quality. To do that, it needs to make linkages with outside countries, pay them if necessary, but also hold these institutes to a high standard and produce people who use their hands and their brains, who have real skills, who have technology training, and who build things. Training in subjects like metallurgy, materials, machining, automotive design, communications equipment, modern textile practices and so on. Higher education nowhere directly prepares people to build an industrial base; it only does so in an abstract and indirect sense.

STEP: One of the questions we asked Dr. Qasim Sheikh in a recent interview was that Pakistan is still largely an agricultural society, not an industrial one. Can we by-pass the industrialization process and become a knowledge-based economy directly? He was fairly optimistic that it is not just possible, but that the revolution in ICT is making it happen already, and the example he gave was 70 million cell phone users versus 5 million landlines in Pakistan. If it can happen …

AA: No, no, hold on, let’s get something straight. Yes, indeed cell phones have totally changed the fabric of our society; there is no question about that. But does that mean Pakistan is now a high-tech economy as a result? Come on, it’s only at the mercy of Mobilink and Warid and so on. They have done all the investment and if they were to pull out, that would be the end; there is no real knowledge in Pakistan that has developed as a result of the cell phone being made affordable to everybody.

These multi-nationals bring in pre-packaged systems that are deployed nationwide. We visited Mobilink, which is the leading player, and they said that they have hundreds of employees but they do some software customization for applications and maintenance of the base stations. Not many know the technical details of how the system works; it just comes as a package, they mostly maintain it.

They agreed that Pakistan needs people who really understand how wireless communication works, and who can innovate on their own; it doesn’t yet have those people in numbers.

STEP: If somebody gave you $80 million and said, do whatever you want to do with it in Pakistan in the higher education sector. Would you go set up something like LUMS SSE, with its emphasis on basic sciences, or would you build some vocational training institutes?

AA: I would still invest in something like the SSE. Pakistan has plenty of institutions that cater for today’s needs, but no institution that is developing expertise for the future, things that are going to be really important in the mid-twenty first century and beyond. I think for that you need something like the SSE.

But, I would do it at a much modest scale to make that money last a really long time. In Pakistan, I think the tendency is to build monuments, harking perhaps to the Mughal emperors. It is important to get beyond that. So, if I had $80 million, I would be working out of a temporary building and using the money to get the best people, give them excellent salaries and the best working environment. That’s it, because the key to a good roll out is a long term vision, realistic scope and producing work of the highest quality in teaching and research.

STEP: Do you see a role of the Pakistani diaspora in the improvement of education in Pakistan?

AA: When I announced to the people I know here that I am planning to move to Pakistan, they were very supportive. They thought it was a great idea to go there. But it was largely limited to pats on the back.

There are a few wealthy Pakistanis here who may have the kind of money that it takes to actually make a difference at a place like LUMS. But, they don’t have trust in any Pakistani institution. So none of them would want to, let us say, create a chair at LUMS or a scholarship endowment. They would have no problem creating a chair in the United States on a subject like Islamic Studies or Pakistan Studies. This is because Pakistan institutions have not yet won their confidence. Until this changes, the diaspora will remain of little help in building large institutions.

Beyond building trust in Pakistan, the diaspora should build its financial power and organize itself to lobby in this country to help real causes in Pakistan.  You know about the USAID money that is flowing into Pakistan to help fight the war on terror. Yet I don’t know of much lobbying or participation by the diaspora in helping USAID or the State Department to identify causes where this money is best invested. Pakistan has so many needs, and members of the diaspora can help sort through those needs.

STEP: Finally, what would you advise in terms of the factors that one should consider before deciding to move to Pakistan?

AA: I think you should tie yourself to a successful cause. Don’t adopt failing causes in Pakistan, it is too big a country for one person to make a difference. You can soon be left exhausted and dispirited. Once you join such a cause, become an advocate and ambassador for it, involve other people and “move the mountain” together. And whatever you do there, do not compromise on quality. Work at a “world class” standard. That’s what we learn from Pakistan’s successes, such as the Indus Basin Project, the atomic energy project, and others. That’s how LUMS’ SSE faculty is doing it in the classroom and the lab.

Here is what some of the great mathematicians, after having done work considered the very peak of human thought, think about the factors in their success:

Grothendieck, Fields Medalist 1966: “Since then I’ve had the chance, in the world of mathematics that bid me welcome, to meet quite a number of people, both among my “elders” and among young people in my general age group, who were much more brilliant, much more “gifted” than I was. I admired the facility with which they picked up, as if at play, new ideas, juggling them as if familiar with them from the cradle — while for myself I felt clumsy, even oafish, wandering painfully up an arduous track, like a dumb ox faced with an amorphous mountain of things that I had to learn (so I was assured), things I felt incapable of understanding the essentials or following through to the end. Indeed, there was little about me that identified the kind of bright student who wins at prestigious competitions or assimilates, almost by sleight of hand, the most forbidding subjects.”

Gauss: “If others would but reflect on mathematical truths as deeply and as continuously as I have, they would make my discoveries.”

The reason why I give credence to these remarks is that, while both Grothendieck and Gauss were considered amazing geniuses by their contemporaries, neither was known for being modest. (Grothendieck said: “In the history of mathematics, I have produced the greatest number of new ideas”, and Gauss was famous for putting down other mathematicians.) This, together with the fact that even at graduate schools in the US which attract the best and the brightest of students, the drop-out in computer science is over 50%, should suggest that other factors play a larger role in determining success or failure. In my opinion, a rather large reason for failure is the following, rather fragile, learning psychology.

In the current environment, everyone wants to be smart, or at any rate, appear smart. This severely interferes with learning, naturally: students who consider being smart important become more conservative in the length and hardness of problems they attempt, which is a reasonable risk-averse way of preserving their image. This approach works for undergraduates, especially under the diseased quarter system since the material covered is relatively shallow and easy. However, once one starts graduate studies and begins to think about problems where it is not even clear if a solution is possible, the habit of following the risk-averse strategy just doesn’t cut it.

Students not used to prolonged thinking on a single problem start off well. However, soon they find motivation and inspiration leaving them, and they start dreading working on the problem as failure would lead them to question something they (by now) crucially identify with: “smartness”. Procrastination kicks in, and soon the student is busy in a diverse set of academic (but non-research!) activities to hide the reality of not working, like writing complicated scripts to automate their soon-to-be-coming publication phase, optimizing their daily vitamin B12 intake, getting heavily involved with political and religious movements and so on. Few students are able to critically introspect, which is reasonable since society has informed them that smartness is what matters, and if they are unable to solve the problem quickly, the logical conclusion is that they are not smart. In this world-view, it is hard to even consider the suggestion that smartness matters fairly little in such matters and most fall prey to heavy depression. Some do manage to climb out: Feynman, physics Nobel Prize 1964, had developed a reputation for being an extremely smart guy at Los Alamos. He paid for this afterwards as an assistant professor at Cornell, where for the first two years he was paralyzed by this fear, and unable to do any worthwhile work. During this time, he received an invitation to join the prestigious Institute for Advanced Studies (where Einstein was one of the members) but refused since he felt useless as a researcher. Fortunately for science, later a positive reaction set in for  him and he was able to overcome his fear (and later ended up writing  books with titles “What Do You Care What Other People Think’”).

Instead of intelligence, persistence is the crucial parameter for success in graduate school:

Gowers, Fields Medalist 1998: “To illustrate with an extreme example, Andrew Wiles, who (at the age of over 40) proved Fermat’s Last Theorem … and thereby solved the worlds most famous unsolved mathematical problem is undoubtedly very clever, but he is not a genius in my sense. How, you might ask, could he possibly have done what he did without some sort of mysterious extra brainpower? The answer is that, remarkable though his achievement was, it is not so remarkable as to defy explanation. I do not know precisely what enabled him to succeed, but he would have needed a great deal of courage, determination, and patience, a wide knowledge of some very difficult work done by others, the good fortune to be in the right mathematical area at the right time, and an exceptional strategic ability.

This last quality is, ultimately, more important than freakish mental speed: the most profound contributions to mathematics are often made by tortoises rather than hares. As mathematicians develop, they learn various tricks of the trade, partly from the work of other mathematicians and partly as a result of many hours spent thinking about mathematics. What determines whether they can use their expertise to solve notorious problems is, in large measure, a matter of careful planning: attempting problems that are likely to be fruitful, knowing when to give up a line of thought (a difficult judgment to make), being able to sketch broad outlines of arguments before, just occasionally, managing to fill in the details. This demands a level of maturity, which is by no means incompatible with genius, but which does not always accompany it.” [Excerpted
from  the excellent book "A Short Introduction to Mathematics"].

Though not directly related to research, the phenomenon that is Judit Polgar provides another fascinating insight into the reasons behind spectacular success in intellectual activities:

“Forty years ago, Laszlo Polgar, a Hungarian psychologist, conducted an epistolary courtship with a Ukrainian foreign language teacher named Klara. His letters to her weren’t filled with reflections on her cherubic beauty or vows of eternal love. Instead, they detailed a pedagogical experiment he was bent on carrying out with his future progeny. After studying the biographies of hundreds of great intellectuals, he had identified a common theme — early and intensive specialization in a particular subject. Laszlo [sic] believed he could turn any healthy child into a prodigy. He had already published a book on the subject, Bring Up Genius!, and he needed a wife willing to jump on board.” [Psychology Today]

The result were three sisters: Susan, Sofia, and Judit. Judit is by far the best female chess player in history, and ranked in the top-10 chess players in the world. Susan is the next(!) best female chess player in history. Sofia has a record-breaking performance in Italy  that has become known as the “Sac of Rome”.

“Anders Ericsson is only vaguely familiar with the Polgars, but he has spent over 20 years building evidence in support of Laszlo’s theory of genius. Ericsson, a professor of psychology at Florida State University, argues that ‘extended deliberate practice’ is the true, if banal, key to success. ‘Nothing shows that innate factors are a necessary prerequisite for expert level mastery in most fields,’ he says … His interviews with 78 German pianists and violinists revealed that by age 20, the best had spent an estimated 10,000 hours practicing, on average 5,000 hours more than a less accomplished group. Unless you’re dealing with a cosmic anomaly like Mozart, he argues, an enormous amount of hard work is what makes a prodigy’s performance look so effortless. ‘My father believes that innate talent is nothing, that [success] is 99 percent hard work,’ Susan says. ‘I agree with him.‘ “

The effect of psychology on learning is illustrated nicely in an interesting recent experiment: A group of researchers led by Carol Dweck of Columbia University went to a very competitive school’s 5th grade class, and randomly split it into two groups. Both groups were given the same easy puzzles to solve, and the performance of each child noted. Both groups scored well. After the exam, the first group was told ‘you must have really worked hard’, while the second group of children were rewarded by saying  ‘you must be smart at this‘. For the second round, both groups were given the same choice: either take another easy exam, or a much harder exam.  Here’s the punchline: over 90% of students in the first group chose the harder exam, while the majority of children in the second group chose the easier exam. In the third round, everyone had to do the harder exam:

Dweck:  “When we praise children for their intelligence, we tell them that this is the name of the game: Look smart, don’t risk making mistakes … [In the third round, children in first group] got very involved, willing to try every solution to the puzzles … Many of them remarked, unprovoked “This is my favorite test” [while for the students in second group] you could see the strain. They were sweating and miserable.”

The NYMag article ends with the following sage advice, on which I’ll also end: “The brain is ultimately just a muscle. Make it stronger by working it out.”

Editor’s Note: The views expressed in this article are solely those of the author and do not necessarily reflect the views of STEP.

To seed this discussion, we present commentary from Dr. Pervez Hoodbhoy and Dr. Atta-ur-Rehman. Dr. Hoodbhoy presents his opposing point of view, arguing that the measures presented in the article were inadequate, and further that the conclusions drawn from the metrics were flawed. Dr. Atta-ur-Rehman, founding (and former) chairman of the HEC, who led the higher education reform effort during his tenure, responds by pointing to data that, in his view, shows the depth and breadth of the reform’s success.

We invite our readers to contribute their thoughts on what metrics are appropriate for measuring the success of higher education within the context of Pakistan.

NOTE: Both commentators have significantly shaped the landscape of Pakistani education over the last few decades. We request our discussants to avoid personalizing the discussion and to maintain a civil and constructive tone.

SYNOPSIS (We will continue to compile the synopsis of the discussion as it progresses: Last Update 8:02am EDT, September 22nd, 2009.)

The metrics suggested, thus far:

1. QUALITY OF RESEARCH (Editors, Zeeshan Khan): Citation and variants on their measures like the h-index, are standard measures of research quality on an individual, institutional, and national level. Number of patents is another measure, though .
2. QUANTITY OF RESEARCH (Atta-ur-Rahman): This measures gross research activity.
3. QUALITY OF TEACHING (Pervez Hoodbhoy): Metric?
4. QUALITY OF UNIVERSITY GRADUATES (Pervez Hoodbhoy, Fakhruddin Habiby): Surveying employers or assessing performance in international tests.
5. ACADEMIC FREEDOM (Pervez Hoodbhoy): Metric?
6. ACCESS TO UNIVERSITY FACILITIES (Atta-ur-Rahman, Khurram Shafique): Libraries, laboratories, internet connectivity, communication facilities, sports facilities,
7. EVALUATION BY NEUTRAL EXPERTS (Atta-ur-Rahman): Survey of a group of neutral experts, like the World Bank, USAID, etc.
8. UNIVERSITY ENROLLMENT (Atta-ur-Rahman): The increase in university-going adults can be measured by census.
9. UNIVERSITY-INDUSTRY LINKAGE (Fakhruddin Habiby, Anwar): The number of industry supported projects which were initiated with University-Industry partnerships and their effectiveness based on industry feedback.
10. LOCAL RELEVANCE OF RESEARCH (Editors): Metric?
11. QUALITY OF CURRENT STUDENTS (Pervez Hoodbhoy, Khurram Shafique):  Performance in standardized tests conducted every year, performance in local and international competitions such as Mathematics Olympiads and Programming Contests.

Notes:

(Pervez Hoodbhoy) Self-citations are a serious problem when using citations as a metric for measuring quality of research reform. [paraphrased]

(Abdullah Sadiq) While strengthening the research effort in the universities is important, the most urgent need is to concentrate on producing quality teachers for the lower tears of education. [paraphrased]

(Khurram Shafique) A pedestrian publication in the field of networking or multimedia is likely to receive more citations than a good publication in a less explored field in mathematics, say, non-standard analysis.

(Fakhruddin Habiby) another ‘tool’ that is used to push the citation number higher is formation of ‘citation-coalition’ within research groups.

(Omar Javed) Categorization of universities into subsets, and adoption of relevant performance criteria for each subset. Three fundamental questions: what is taught (Undergraduate and Graduate Instructional Program classifications), who are the students (Enrollment Profile and Undergraduate Profile), and what is the setting (Size & Setting)”

(Shafiqur Rehman) …the success or falure or HE reforms must only be judged by opinion of the common stake-holders (students, teachers and administration) of the public sector universities.

(Affan): we need to tweak ratings/rankings such that we are able to measure any progress happening in Pakistan, progress small enough that it is not lost by existing metrics.

There are four aspects of the comments of Dr. Pervez Hoodbhoy that need to be considered:

1. Firstly, Dr. Hoodbhoy himself admits that there has been a huge increase in international publications at QAU after HEC came into existence when he mentions the number of international publications in the two time periods. Strangely, he picks a six year period, 1998-2003, and then compares it with the subsequent 4.5 years (?) , 2004 to mid 2008, (the correspondence occurred in August 2008, so he could not possibly have had access to the figures for the entire year) I can only assume that he has mentioned 2003 by mistake in the second “5 year” period as there is no reason to include the publications of the year 2003 in both time periods, which he has done. It is clearly unfair to take two time periods of different durations and compare them.
2. In the first 6 year period (1998-2003), Dr. Hoodbhoy admits that there were only 631 research publications from QAU, but in the second 4.5 year period these had risen to 1482 research publications, a tripling of publications on average per year, even by his own estimates.
3. As the HEC programs began in 2003 and their real impact occurred 2-3 years later, a year-wise comparison is far more relevant than an average over a 5 year period as the dramatic change that has occurred gets partly masked when a 5 or 6 year average is taken, though it is still very visible. Dr. Hoodbhoy ignores the figures that Dr. S.T.K. Naim had worked out that in the year 2004, there were only 84 research publications from QAU (an average of only 7 publications per month), but by 2008 they had increased many fold.
4. The citations argument used by Dr. Hoodbhoy is invalid as citations increase with the passage of time. Dr. Hoodbhoy, therefore, wrongly compares the citations of papers of an earlier period with those of a later period. To clarify this issue further, if two papers of equal quality and in a similar field are published, say in 1998 and 2007, and the citations of both are counted in 2008, then the paper which was published in 1998 will have accumulated more citations by 2008 because of the much longer 10 year time period, than the paper published in 2007, as that would have had only one year for the citations to accumulate. Dr. Hoodbhoy is therefore comparing apples with oranges when he tries to compare citations of papers published in an earlier period with a later time period. In order to fairly compare citations, the same duration of time period must be taken. Thus if one takes 1998 publications and counts the citations till 2008, then one will need to take the 2008 publications and count their citations till the year 2018, before one can compare the figures for the citations of the two sets fairly.

The undeniable fact is that the total number of research publications from universities in Pakistan was only about 600 per year till 2001 but then started rising rapidly, and by the year 2008 it had increased to over 4,300! Brazil achieved such an increase over a 35 year period between 1960 to 1995, which Pakistan achieved in only 6 years. After my appointment in March 2000 as the Federal Minister for Science and Technology in Pakistan, I convinced the government to enhance the budget for science and technology in Pakistan by 6000% between July 2000 to October 2002. After my appointment as Chairman, Higher Education Commission (Federal Minister) the budget for higher education was similarly increased by 2400% during 2003 to 2008. Major achievements during these periods were:

1. Establishing 51 new Universities and awarding institutions during 2002-2008,
2. Tripling university enrollment (which had reached only 135,000 from 1947 to 2003) to about 400,000 in 2008,
3. Establishing a powerful Digital Library which provides free nation-wide access to every student in every public sector university to 45,000 textbooks/research monographs from 220 international publishers as well as to 25,000 international research journals,
4. Establishing video-conferencing facilities in most public sector universities that allow lectures to be delivered live and interactively to students in Pakistan from technologically advanced countries
5. Enhancing salaries of academics so that salaries of University Professors were increased to a level about five times the salaries of Federal Ministers, with a corresponding reduction in tax from 35% to only 5%, in order to attract the brightest young men and women into academia,
6. Promoting research through a massive research grant program which resulted in a 600% increase in ISI abstracted publications from about 600 per year in 2001 to 4300 research publications in 2008, accompanied by about 1000% increase in international citations in the same period,
7. Placing a satellite in space (Paksat-1) which is now used for distance learning by the Virtual University,
8. Establishing video-conferencing facilities in most public sector universities and initiating a lectureship program, allowing live interactive lectures to be delivered from technologically advanced countries,
9. Providing free access to scientists/engineers anywhere in the country to sophisticated instruments installed in any institute in Pakistan.

The Bottom Line: In the final analysis, it is not what I or Dr. Hoodbhoy think about the developments, but what is the opinion of neutral international experts who have carried out detailed year-long reviews of the developments during the period that I was heading the Higher Education Commission. A few extracts are given below:

1. Prof. Fred Hayward (independent international educational consultant from USA) carried out a detailed analysis of the developments and published an article entitled “Higher Education Transformation in Pakistan: Political & Economic Instability,” Date: Number 54, winter 2009 Source: International Higher Education Quarterly. I quote: “The news about Pakistan over the last few years has been dominated by reports of political turmoil, terrorism, religious fundamentalism, economic decline, and the Afghan War. What has been missed is the phenomenal transformation in higher education over the last six years, which represents a critical development for Pakistan and a potential engine for growth and national recovery.”
2. Report of US-AID about HEC states that “We are very impressed with the breadth, scope, and depth of the reforms implemented by the HEC since 2002.  No other developing country we know has made such spectacular progress.”
3. World Bank Report is very complimentary of many excellent programmes introduced.
4. British Council: The report states: “I have worked in many countries in South America, the Middle East, North Africa, and in Russia and India, over the last six years.  None in my view, with the exception of India, has the potential of Pakistan for the UK university sector, largely because of the dynamic, strategic leadership of the Chairman of HEC”.
5. Nature: Several articles and editorials have appeared in the world’s leading science journal “Nature”  (the most recent in the issue published on 3rd September 2009) in which the very significant progress made by Pakistan in the higher education sector has been applauded and the need for the new government to built on the solid foundation laid has been stressed.
6. Science Watch (Thomson Reuters) has ranked Pakistan as a rising star in five disciplines, more than in any other country of the world.

Join the Discussion!
What are the correct metrics to measure higher education reform in Pakistan?

Related Posts:

1. Nature’s Coverage of Higher Education Reform in Pakistan: A Response from Prof. Pervez Hoodbhoy
2. Nature’s Coverage of Higher Education Reform in Pakistan: A Conversation with Athar Osama
3. Pakistan’s Higher Education Funding Holds Many Lessons for Developing Nations: Nature

Editors Note: Dr. Atta-ur-Rahman is a leading scientist and scholar in the field of organic chemistry from Pakistan. He has served as the Federal Minister for Science and Technology, the Federal Minister/Chairman of the Higher Education Commission, Adviser to the Prime Minister on Science and Technology, and the President of the Pakistan Academy of Sciences. Dr. Atta-ur-Rahman is credited by many for reviving the higher education and research practices in Pakistan. The views expressed in this article are solely those of the author and do not necessarily reflect the views of STEP.

Unfortunately, I find the response as unsatisfying as the original article. Since Nature is unwilling to accord me a chance for a satisfactory reply on its pages, I shall clarify the basis of my criticism in some detail here.

In the said article, strong conclusions have been derived from weak data. The authors have not dared to ask the basic questions whose answers are essential for ascertaining whether there has been actual progress in Pakistan’s higher education system and, if so, by how much. Instead, in giving a thumbs-up, numbers have been quoted that have doubtful significance. Take, for instance, the claim that:

“In mathematics, for example, an average paper by a Pakistani author is cited around 20% more than the worldwide average for the discipline”.

Before returning to what some of the right questions might have been, let me give two reasons why the above claim – even if true – carries little meaning.

First, self-citation is a far more serious problem than the authors are willing to acknowledge. It is also one that they admit to not having investigated. The data on Pakistani research papers shows that subtracting out self-citations drastically cuts down on actual citations – there are often 2-3 self-citations for every real one! The reader is urged to carefully study my email correspondence of last year with Dr. Atta-ur-Rahman (see appendix below) who, at the time when he was HEC chairman, had made similar claims that I disputed as being false. To interested readers, I have made available (in pdf form) the Thomson Scientific data that I have quoted in my correspondence here (1998-2003) and here (2003-2008).

Second, many authors of the relatively more cited mathematics papers from Pakistani institutions are not Pakistani nationals. High salaries offered to foreign faculty by the HEC brought to Pakistan a large number of well established mathematicians on short-term contracts from Russia, Poland, Ukraine, and China. This was probably a good thing to do – in spite of the difficulty they had in communicating in an alien language and their consequent inability to teach well. Their papers, however, do not reflect mathematics in Pakistan. One sees a similar phenomenon in Saudi Arabia where foreigners are principally responsible for the kingdom’s large number of papers and citations.

In my opinion, instead of focusing on marginal matters, serious research on the state of Pakistani higher education, and of changes therein, would have first established appropriate metrics, and then sought answers, to the following key questions:

1. What is the quality of teaching in Pakistan’s public universities? There is often only a weak correlation between formal qualifications and subject competence, so simply counting PhD degrees is not very helpful in answering this. Far too frequently one sees professors of English who cannot speak or write a single sentence of grammatically correct English, physics professors who are stymied by Newton’s Laws, and biology professors whose knowledge is frozen in some pre-Darwinian age. But does such basic incompetence exist at the 20, 50, or 70 percent-level? Higher? Lower? What evidence exists that the HEC’s reforms improved the situation?
2. Is there evidence that there has been improvement in the selection process for students in public universities, or that of the quality of their graduates? Proof of the latter, judged by asking employers or assessing performance in international tests, would be a clinching argument for the success of HEC reforms.
3. Do campuses enjoy greater academic freedom, more seminars and colloquia, less violence by extremist campus groups, a pleasanter and more relaxed ambiance, and greater transparency in faculty selection? Surely these are critical to any reasonable assessment.

To get answers to questions like these requires extensive field work, and I certainly do not fault the authors for not doing this. But I was surprised that the Nature article, as well as the lead author’s response, merely says that the HEC’s experiment had critics, without citing any specific articles or the substance of those criticisms. There is not even a passing reference to the failed nine-university multi-billion dollar mega-project, tons of unused scientific equipment purchased for unknown reasons, dubious attempts to fund “Quranic Science” (that had to be hastily abandoned after the scheme was exposed), and the explosion in academic corruption set off by per-paper payments. Surely, these should not be brushed aside as “collateral damage”. In another country, those who massively squandered public money would have been thoroughly investigated by independent commissions, not praised for small things.

The Bottom Line: Well-functioning universities are the products of a complex organic and evolutionary process that is internal to a society. Money and facilities matter, but it is much more important for a university to have a forward looking world-view, an open environment, high ethical standards, a sense of collegiality and shared sense of purpose, and good governance practices. Sadly, the Nature article did not even mention these as significant.

Join the Discussion!
What are the correct metrics to measure higher education reform in Pakistan?

Related Posts:

1. Nature’s Coverage of Higher Education Reform in Pakistan: Comments by Dr. Atta-ur-Rahman
2. Nature’s Coverage of Higher Education Reform in Pakistan: A Conversation with Athar Osama
3. Pakistan’s Higher Education Funding Holds Many Lessons for Developing Nations: Nature

Editors Note: Prof. Pervez Hoodbhoy is a well-known Pakistani nuclear physicist and political-defence analyst. He is the Professor of High Energy Physics, and the head of the Physics Department at Quaid-e-Azam University, Islamabad, Pakistan. Prof. Hoodbhoy is a vocal critic of HEC’s policies and their impact. The views expressed in this article are solely those of the author and do not necessarily reflect the views of STEP.

This correspondence between Dr. Atta-ur-Rahman and Dr. Pervez Hoodbhoy is reproduced below with consents from both parties. It is exclusively concerned with a public matter, has no private content, and is largely focused upon the importance of self-citations.

Dear Dr. Pervez
I have probed into the situation at QAU, and it is quite the opposite of what you claim, as evident from the total publications and the total number of citations, as per above attachments. The HEC was established in October 2002. The funds started coming through from July 2003. The subsequent impact of HEC programs on research publications and citations is indeed amazing. In 2004 the number of publications was only about 120—by 2007 it has risen to about 380—a 300% increase! The citations in 2004 were about 800—-by August 2008 they have increased to about 2200 although we still have 4 months to go before the year ends—I suspect that it will be about 3200 by the end of the year—a 400% increase!

I shall be separately sending you a report after removing self-citations. QAU is also now included in the most cited institutions in the world (please see attachment)—-this was not to 4 years ago.

Please be fair and objective in your assessments. We may have made some mistakes, but much good has happened.
Kind regards
Atta

Dear Dr. Pervez
Following my other email to you this morning, I am now enclosing the citations of QAU after removing self-citations.As you would see, these have risen from only 84 in 2004 to 1413 in 2008 (with still 4 months to go). These will probably be around 1900 by the end of the year—a spectacular ten-fold growth! Dr. Naim has kindly had these searched, so if you have any queries about them, you may like to interact with her. She tells me that the situation is similar in many other universities—a long period of stagnantion
during the 1990s followed by a burst of activity in the last 4-5 years.
Kind regards

Atta

Dear Dr. Atta,

I am sorry about the late response to your three emails. First, thank you for doing whatever you did – the HEC’s notification of 14 July 2008, which specifies 40 percentile as the GRE passing marks, finally reached QAU departments today (without comment from the administration). In these times one has to be grateful for small things…

Regarding your email and the numbers cited therein: I can understand that you would like to feel upbeat about QAU having improved itself as a result of massive infusion of HEC resources. I too would very much like good things to happen, but perhaps one should not allow wishes to become conclusions.

Looking at the data that you had sent me and your inferences, I decided to do a little independent investigation using exactly the same database (ISI Web of Science) and exactly the same keywords (see attachments to this email). Here are the findings:

Period: 1998-2003
Number of papers published by QAU authors in the above period: 631
Number of citations to date: 4540
Number of citations to date with self-citations removed: 2,817

Period: 2003-2008
Number of papers published by QAU authors in the above period: 1482
Number of citations to date: 3667
Number of citations to date with self-citations removed: 1258

Some obvious inferences:

1. There is absolutely no evidence of real citations having increased; if anything, the numbers up to now show the contrary. While the number of real citations of papers published 2003-2008 may increase somewhat with time, currently they are quite a bit less from the earlier period when the HEC and your incentive system did not exist. Please remember that citations are cumulative over years. I have tried to use exact descriptions in the figures cited above. If I am wrong in any detail, or if I have missed something essential, I would like to be corrected. Unfortunately the data does not at all support your rather optimistic remark of “a spectacular ten-fold growth!”
2. The above data also indicates the disturbing fact that most of the time QAU authors cite themselves. Subtracting self-citations drastically cuts down on real citations – there are 2-3 self-citations for every real one!. Looking more minutely at the ISI pages, one also notes that many citations are by other members belonging to the same or other QAU departments. So the number of genuine citations gets cut down even beyond the numbers quoted above (2817, 1258)!

Just to get scales right, here are a few citation numbers from the Stanford Spires data base for Pakistani physicists over the length of their careers:
a) Riazuddin: 1479
b) Ahmed Ali (DESY, Germany): 9873
c) Abdus Salam: 14103

3. You are correct that the number of publications over equal (5-year) time periods has more than doubled relative to pre-HEC times. But this is clearly in response to the monetary incentives offered by PCST/QAU. A publication fever now grips our universities. It is difficult to defend the case that the number of papers published is proportional to the amount of research done.

You are, of course, free to have the data I have quoted above rechecked and I would be happy to answer any question that arises. Finally, please note that publications and citations were not central to my earlier expression of dismay at the quality of QAU education.

What bothers me much more is the lack of analytical and problem-solving abilities of our Ph.D graduates, some honourable exceptions aside. Poor performance in the GREs is one indication of the rot. This fact has indeed worried you a little, as you indicated in an earlier email to me, but I do wish you could understand the real gravity of the situation.

Regards,

Pervez

Dr. Atta-ur-Rahman’s comments on this post can be found here.

STEP: How did the idea of this study come about?

The idea for the article actually originated from Nature’s Editors late last year when HEC’s budgets and utility were being increasingly questioned by the new government in Islamabad. Nature had been an observer – and sometimes a commentator on HEC’s initiatives – and approached me to see if I’d be interested in writing an independent and objective piece on Pakistan’s Higher Education Reforms. I worked with Nature’s editors to put together a group of authors who we believed brought a diversity of experience, insight, and credibility to this exercise.

This preliminary review, I hope, will be a starting point towards an extensive process of external policy peer review and a constructive self-examination leading to greater effectiveness of higher education policies and programmes in Pakistan. Another very important aspect of Nature’s interest in this work was to see if other countries contemplating similar reforms could learn some lessons from Pakistan.

STEP: How long did the study take? What data sources and methodologies were used?

We began working on this subject with the Nature piece in mind about 8-9 months ago. Obviously, the first step was to approach HEC itself and to seek their cooperation in doing this. There was really no point in saying anything on the subject without having access to HEC’s own data and viewpoint.

We had excellent cooperation from Dr. Atta Ur Rahman as well as Dr. Sohail Naqvi and other members of HEC staff. In addition to HEC, I also met and talked to a number of university leaders as well as HEC’s critics including Dr. Hoodbhoy. Dr. Naqvi made available to us a lot of data that we believe had not been publicly available in this manner before. We were cognizant, however, that this is self-reported data and that any conclusions we will make based on this data alone are going to be fiercely challenged by HEC’s critics.

In order to balance HEC’s own view, we also used external data to validate the claims made by HEC. Specifically, we decided to use Thomson Reuters’ data on scientific publications as independent measure of Pakistan’s publication activity and impact. We must emphasize, though, our analysis is merely a start. One of our key recommendations is for HEC to subject itself to an independent external peer review of its policies and programmes and use their guidance to improve these. This peer review must also include taking a look the current data at hand and the data collection systems in place and to improve these to provide continuous guidance to HEC planners.

STEP: What was the most important (or surprising) finding of the study?

I believe that this is still a work in progress as Pakistan’s reform experiment is itself a work in progress. I won’t speak for other members of the team here, however, I was personally surprised by quite a few findings. First of all, there does seem to be an unambiguous improvement in Pakistan’s publishing performance as well as its relative impact. The publications have tripled over the last few years and there are significant gains in relative impact of Pakistani papers. Our preliminary testing shows that this finding is not too influenced by self-citation bias. One negative finding – though not unsurprising – was the decline in the publishing performance in social sciences (and business and management and humanities etc.). This is a bit alarming – for me – because I do believe that well developed social sciences are very important for a balanced development of a nation.

I believe that several programmes may be rightly guided – even though there may be implementation lapses. The foreign PhD programme is one example. I believe, and hope, it will deliver in the medium to long-run. The NRPU – the competitive grant funding programme – is in the right direction – although the peer review process needs to be strengthened. I think that a 47% acceptance rate is just too high – especially given that a vast majority of Pakistani faculty have not had much exposure to competitive research for considerable periods in the past. The investments in the digital library and internet connectivity are also critical elements of the research infrastructure.

On the other hand, there are still several things that are not so right. There are things that can potentially go wrong – seriously wrong – if important issues are not addressed. I am especially concerned with the policies vis-à-vis the Domestic (Indigenous) PhD programme. That is certainly something that should be subjected to greater scrutiny and evaluation from both outside and inside of HEC. Also governance and management reforms within universities have been rather slow. However, I think there is a need to take key stakeholders along rather than force it on universities.

STEP: The statistic that I found most interesting in the article is the +20% relative impact of Mathematics publications from Pakistan. However, Dr. Hoodbhoy, in his critique of your article, asked the question of whether self-citations were removed or not in computing this number. To quote Dr. Hoodbhoy, “did the authors try to eliminate self-citations (a deliberate ploy) from this count? If they had – as I did using an available option in the Thomson Scientific package – they might actually have found the opposite result.” How would you respond to this criticism?

The short answer to Dr. Hoodbhoy’s question is that no – this number includes author self-citations. BUT we were definitely concerned about self-citations. Here is what we did to account for this potential bias.

We extracted a database of roughly 13,000 papers (representing all fields) indexed by Thomson Reuters between 1999 and 2008, each listing at least one author address in Pakistan. We also extracted all papers that subsequently cited these papers. We then enumerated instances of ANY country citing into this population of papers (i.e., all nations citing Pakistan), along with a separate measure of only Pakistani authors citing Pakistani papers. We did this for two distinct time periods: for papers published and cited during 1999 to 2003, and during 2004 to 2008.

For the first time period, there were 8,436 occurrences of all nations citing the Pakistani papers (or, to repeat, those papers bearing at least one Pakistan author address), and 1,527 occurrences when Pakistan-only papers cited the group. In percent terms, 1,527 of 8,436 equals 18.1%. Thus, by this measure, Pakistan self-citation could be quantified as 18.1%. For the second time period, the comparable numbers were 26,294 instances of all countries citing the Pakistan group, and 6,597 instance of Pakistan citing Pakistan. Expressed as a percentage, the figure is 25.1%

To summarize: 18.1% Pakistan self-citation for the 1999-2003 period, and 25.1% for 2004-2008. Now it is important to understand that country self-citations are not a perfect proxy for author self-citations – the variable that we’re really interested in. However, author self-citations are a subset of country self-citations and hence country self-citations do give us an upper bound on author self-citations. That is, author self-citations for Pakistani papers are definitely less than 25.1% – and perhaps a lot less than that – in the post-reform period.

We can compare this figure with generally accepted empirical norms of author self-citation globally. In Gami et al (CMAJ, 2004) the authors estimate self-citation in diabetes research at about 18%. Falagas and Kavvadia (FASEB, 2006) arrive at self-citation rates in biomedical research at 17-20%. Others have confirmed the general trend in medical literature of around 1/5^th of all citations as being author self-citations. Garfield and Sher (1964) arrive at a self-citation figure of 20% for basic research papers. Trimble (1986) arrives at a figure of 15% for all astronomical papers. Bonzi and Snyder (1986) found an average self-citation rate of 11 per cent across a range of disciplines, varying from 16 per cent in the physical sciences (chemistry and geology) to 3 per cent in the social sciences. The frequency of self-citation is also found to be independent of quality of publication.

So, Pakistan’s likely author self-citation rates may be only slightly higher – if not within – the global norms for self-citations. It is quite unlikely that all – or even most – gains can be wiped out by author self-citations. Dr. Hoodbhoy may be right in that self-citations may have increased from the past but that is likely to be expected for a variety of reasons – some of which quite legitimate. For example, as a closely knit research community develops within a country and interact through local conferences they are more likely to be aware of each others work and hence cite authors from within the country. Had we been able to exclude self-citations from these relative impact figures, these results would perhaps have been only marginally smaller. Because we were dealing with publications across multiple disciplines over multiple time periods and across multiple countries we could not correct relative impact figures for self-citation. However, given the above analysis we don’t find a reason to feel so alarmed as to totally rubbish the entire publishing performance of the country.

Should this be an issue HEC should carefully watch and analyse? – definitely yes. Should this totally nullify the improvements in Pakistan’s publication record? – we don’t believe so.

STEP: You mentioned that a lot of data provided by the HEC was not used in the article because of the space limitations. The only real statistical analysis in the current article is the table on relative impact factors. What other metrics could have been used to scientifically analyze the performance of HEC, had space not been the limiting factor.

HEC did provide a lot of data and complied with our requests for additional data as much as it could. Potentially, data could inform a number of very interesting questions vis-à-vis the performance of higher education reforms in Pakistan. We were not able to undertake some of these analyses because of the following reasons.

First, a lot of data currently available only accounts for inputs to various programmes – number of PhDs awarded, number of fellowships granted, number of grants awarded are input data. This needs to be complemented with relevant and comparable output data so that outcome assessment can take place.

Second, for data to be useful for subsequent analyses, the analytical and evaluation plan must be built into the programme itself. It is much harder – and more expensive – to cobble together data – ex post – and come up with a very clean assessment of outcomes.

As an example of the sort of things that additional analysis could do is to assess whether the foreign faculty hiring programme (FFHP) is an effective investment. During my meetings with vice chancellors I was told that there is a backlash against FFHP from the faculty and one of the allegations is that the people recruited on FFHP are no better than some of the better faculty members available within the country. This, and several others elements of the reform, are empirical questions that could be easily addressed by intelligent use of data.

STEP: Don’t you think that the HEC is under more scrutiny (at least informally) than other government departments? For example, I don’t see any debate on the performance of, say, MINFAL. The team at HEC has, at the very least, brought about many positive changes compared to the UGC of the past. Do you think there is a chance that overly negative criticism can be counterproductive and may deter such initiatives to positively reform a government department in future?

Yes, certainly there is a chance that overly negative criticism may be counter productive. However, I believe that what we have tried to do is to present our best objective view of the reforms. I believe that it’s a pity that the debate on this important set of reforms hovers between the two extreme viewpoints. I believe that good data and analysis can really refocus this debate and turn it into constructive one. Good data and analysis can move debate away from one of opinions to one of facts.

I believe it is important to have this debate in an open, transparent, and civilized manner. Such a debate will inspire greater trust and buy-in from the people and systems this reform is seeking to address. Pakistan has invested upwards of 50 billion rupees in higher education over the last 5-7 years and it is in all our interest that we make the best use of this investment.

STEP: One of your key recommendations is for the HEC to be reviewed externally. What are some of the examples (from around the world) of the external peer-review process that you suggest HEC subject itself to? Should this process not be done through the Standing Committees on Education in the Senate and/or the National Assembly?

The process of policy peer review is well-established – especially in the west. The nearest equivalent that comes to mind is the peer reviews organized by the National Academies in the US of major government funding programmes (such as the SBIRs, ATP etc.). These are panels of eminent scientists (Academy Fellows) and relevant experts looking over various aspects of these programmes. Because of the unique circumstances and pervasive nature of HEC’s programmes, HEC’s peer review would have to be done by a mix of Pakistani and internationally placed members. The findings of these reviews are then debated by legislatures who have the final say in determining policies informed by these findings and recommendations.

STEP: Based on your study, how do you compare HEC in Pakistan to other equivalent bodies in the developing countries?

I believe HEC has done a considerable amount of work in a very short period of time and some of it may have already begun to reap results as well. It has tackled a very difficult area of reform and has shaken the academic environment in Pakistan. In the depth and breadth of what it has tried to do, HEC has very few parallels in the world today. Most other relevant bodies have only tinkered with the systems that they have tried to reform. It is this factor that probably prompted Nature to invite an opinion piece from the authors and an editorial commenting on these reforms and the lessons that could be learnt by other developing countries. HEC had the liberty and resources to do so and it made the most out of it. Whether or not this hugely ambitious experiment will succeed at the expected level in the end remains an open question.

STEP: If this article is work in progress, as you have said, what further study or analysis can we expect from your team in future?

We are planning to finish a larger piece on higher education reforms in Pakistan and hopefully publish this in the near future. In addition, we intend to dialog with HEC, leading academics and academic administrators in Pakistan to explore how the suggestions in our article can be best taken forward.

Related Post: Pakistan’s Higher Education Funding Holds Many Lessons for Developing Nations: Nature