In 2005, the Higher Education Commission (HEC) of Pakistan imposed the requirement of clearing the GRE Subject Test prior to admission in the PhD programs. Students who were enrolled in the PhD programs at the time were required to clear the GRE Subject Test before submission of their theses. This article discusses the interpretation of the word “clear” used by the HEC , the fairness of this criteria, and the deficiencies in policies regarding the GRE Subject Test. We conclude that by imposing this requirement, HEC has created problems for students living far from big cities, those who do not have access to credit or debit cards, and those who cannot afford the hefty (approximately, Rs. 14,000) registration fee. In addition, the HEC team seemed unaware of the true mechanism of the GRE Subject Test, and as a result significant confusion exists as to what “clearing” the test really means.

Much of the text is taken from the HEC official letters and the GRE guides and the letters published by ETS.

When Did the HEC Decide?

The 7th meeting of Quality Assurance Committee was held on 19th April, 2005 in the regional office of the Higher Education Commission (HEC), Lahore (see [1]). The meeting started with the approval of minutes of the last meeting of the Committee. The minutes were approved and the explanation of “international” with the Subject GRE Test used in the draft. It was explained to the members of the committee that word “international” is placed with the GRE Subject Test to draw a distinction between the GRE Type Test which is locally designed and already in practice, and the  standard GRE Test which is universally available for certain disciplines. The majority of the members and the chairman of the committee did not agree with the word “international” with Subject GRE as it does not exist in international nomenclature of the test. At last the members of the committee decided;

The word “international” will be removed from the Subject GRE-Test as Quality criteria of PhD level studies and it will be written as Subject GRE-Test, where available* with clarification at the bottom that local test will be designed for those subjects in which Subject GRE-Test is not available.

What should be a qualifying criterion?

First of all, it is useful to know a bit about the grading terminologies used by Education Testing Service (ETS), USA for GRE Subject Test. Later, we will discuss them in detail:

• Score (or scaled score)
• % Below (or percentile rank)
• Formula score (or raw score)

On May 30^th, 2006, Chairman HEC, Prof. Dr. Ata-ur-Rahman issued a letter no. 1-15/Adv(QA&LI) /2006/1394, in which he mentioned;

For admission of new students to Ph.D. as well as for upgradation/conversion of students already admitted in M.Phil to Ph.D., an International GRE (Subject) Test must be qualified (at least 50% score presently which will be increased to 60% after 3 years)

Please note that neither has the ETS yet released any document/method to find a percentage score of GRE test nor does the result card give any information about it. The GRE result card gives information about the score, percentage below (we call it percentile) and formula score.

In 2007, Chairman HEC, Dr. Ata-ur-Rahman wrote a letter [3] to the Vice Chancellors in which he asked:

Could you kindly ensure that no students currently enrolled in the disciplines of Biochemistry, Cell and Molecular Biology, Biology, English Literature, Chemistry, Mathematics, Computer Science, Physics, Psychology and allied disciplines are allowed to submit Ph.D theses in your university unless they have obtained at least a 50 percentile score in the respective international subject GRE examination. This is a rather low score presently and it will be increased to 60 percentile score in a couple of years.

In the letter mentioned above (in [3]), Dr. Ata-ur-Rahman used the term percentile score. As mentioned earlier, please again note that this terminology does not exist in the documents of ETS.

In July 2008, Prof. Dr. S. Sohail H. Naqvi, the Executive Director of HEC issued a letter [4], in which he mentioned the following criteria:

To “clear” the international GRE subject test the candidates will have to get Percentile Score equal or greater than the minimum Percentile Score which will be as follows:
i. 40% Percentile Score: Valid Until December 31, 2009
ii. 45% Percentile Score: Valid Until December 31, 2010
iii. 50% Percentile Score: Valid thereafter

So, what is a 40% Percentile Score?

If it is to be considered that Percentile Score means ‘percentile rank’ then 40% percentile rank means the 40% of 99 because the maximum percentile rank one can get is 99. Also this letter failed to clear the meaning of the word “Valid Until …”

In October 2009, Mr. Muneer Ahmed, Deputy Director (Quality Assurance), HEC issued a letter [5], to clarify the following:

I am directed to clarify that the qualifying score for the PhD Scholars who appeared in the International GRE Subject Test before 31st July 2008 is 40% (percentile) or 50%(percent) and after 31st July, 2008, percentile formula as conveyed earlier through the letter quoted above ([2] and [4]) will be applicable.

There was no need to say anything about such non-clarification. From the series of letters it is obvious that HEC was and still is facing dilemma about the criteria of clearing GRE.

In January 2010, HEC commission approved that [6]

GRE (International) Subject Test will be necessary at the time of admission to M.Phil/MS Programme leading to Ph.D.
The minimum acceptable scores are as follows:
i. 40% Percentile Score: Valid for Admissions until December 31, 2009
ii. 45% Percentile Score: Valid for Admissions until December 31, 2010
iii. 50% Percentile Score: Valid for Admissions until December 31, 2011.
iv. 60% Percentile Score: Valid for Admissions thereafter.
For presently continuing students (Admissions before January 11, 2010), the candidates must pass the GRE (International) Subject Test before submission of Ph.D. Dissertation. In disciplines where this test is not available, the test will be made available locally by National Testing Service (NTS), and if the Test is not available in NTS subject list, then a University Committee consisting of at least 3 Ph.D. faculty members in the subject area and approved by the HEC will conduct the Test at par with GRE (International) Subject Test.

Please note that the term “Percentile Score” doesn’t exist, it might be % below or percentile rank. Also clause (i) of above para has already been expired with respect to this letter [6] because it has now been imposed as a criterion to get admission in MS/M.Phil. Moreover subjects for which GRE test is not available, parallel to GRE Subject Test; the university commission of at least three PhDs will form their own test. This creates an injustice in many ways like fee, quality of test, checking criteria, and scoring method.

The text of the letter mentioned in [6] is available on the HEC website and it has been modified by the HEC without any intimation (it can be accessed here and old one is included in references). In the new letter, this test is only mandatory to get admission in PhD and also the word “international” with GRE Subject has been removed.

In March 2010, on answering a query from NUST, Rawalpindi, Mr. Muneer Ahmed wrote as follows [7]:

I am directed to inform you that all students enrolled/converted into PhD programme after May 31, 2005 and before July 31, 2008 are required to qualify International GRE Subject Test with 50% (percent) score and students enrolled/converted after July 31, 2008 have to qualify International GRE Subject Test with percentile score as mentioned in letter no. 1-10/(ED)/HEC/2008/961(copy attached)

It is not possible to find the percentage score of GRE Subject Test (see [8, p. 14]) and other thing is that why Mr. Muneer Ahmad didn’t refer to a latest letter[6] issued in January 2010.

In the following table we give the 50 percent score of each subject defined by the HEC as the 50 percent of the total scaled score. Please note that the HEC defined the score at 99 percentile rank in [10, page 14] as a total scaled score.

To get the 50 percent score in Chemistry and Computer Science, one has to just appear in GRE Subject Test without answering to any question.

How should we use the GRE, according to ETS, USA?

The “GRE Guide to the Use of Scores 2008-09” gives comprehensive information about the GRE test and is the best booklet to learn about the GRE grading terminologies. It is published by the GRE board. This guide is available on the ETS website; the following sentences/paragraphs are taken from this guide.

• Primary limitations of GRE test

Any GRE test, however, has two primary limitations: (1) it does not and cannot measure all the qualities that are important in predicting success in graduate study or in confirming undergraduate achievement and (2) it is an inexact measure; that is, only score differences that exceed the standard error of measurement of a given score can serve as reliable indications of real differences in academic knowledge and developed abilities [8, p. 5].

• Encouragement of appropriate use.

All users of GRE scores have an obligation to use the scores in accordance with published GRE Board policies and guidelines. Institutions have a responsibility to ensure that all individuals using GRE scores are aware of the GRE Board score-use policies and guidelines and to monitor the use of the scores, correcting instances of misuse when they are identified. The GRE Program staff is available to assist institutions in resolving score-misuse issues. [8, p. 6]

• Use Multiple Criteria

Regardless of the decision to be made, multiple sources of information should be used to ensure fairness and balance the limitations of any single measure of knowledge, skills, or abilities. These sources may include undergraduate grade point average, letters of recommendation, personal statement, samples of academic work, and professional experience related to proposed graduate study. GRE scores should not be used exclusively. [8, p. 6]

• Accept Only Official GRE Score Reports

The only official reports of GRE scores are those issued by Educational Testing Service and sent directly to approved institutions and organizations designated by the test takers. Scores obtained from other sources should not be accepted. [2, p. 6]

To ensure the authenticity of scores, the GRE Board urges that institutions accept only official reports of GRE scores received directly from ETS. [8, p. 9]

• Avoid Decisions Based on Small Score Differences

Small differences in GRE scores (as defined by the standard error of measurement) should not be used to make distinctions among examinees. Standard errors of measurement (SEMs) vary by test and are available in this publication. [8, p. 7]

• Minority Examinees (Students outside of USA)

GRE scores, like those on similar standardized tests, cannot completely represent the potential of any person, nor can they alone reflect an individual’s chances of long-term success in an academic environment. It should be remembered that the GRE tests provide measures of certain types of developed abilities and achievement, reflecting educational and cultural experience over a long period. Special care is required in interpreting the GRE scores of students who may have had educational and cultural experiences somewhat different from those of the traditional majority. [8, p. 8]

• Confidentiality and Authenticity of GRE Scores

GRE scores are confidential and are not to be released by an institutional recipient without the explicit permission of the examinee. GRE scores are not to be included in academic transcripts. Dissemination of score records should be kept at a minimum, and all staff who have access to them should be explicitly advised of the confidential nature of the scores. [8, p. 9]

Different types of scores

It is also useful to reiterate that there are three types of grading system in GRE subject.

• Score (or scaled score)
• % Below (or percentile rank)
• Formula score (or raw score)

[ Cutout: Report of Scores ]

The range of scaled scores is from 200 to 990, in 10-points increments, although the score range for any particular Subject Test is usually smaller [2, page 11]. Scaled score is a basic analogy to compare two examinees.

Scaled scores on the same Subject Tests generally are directly comparable across years. A Chemistry Test score of 650 in 2007, for example, should be considered equivalent to a Chemistry Test score of 650 earned in 2006. [8, p. 11]

Percentile rank means the percentage of examinees in a group who obtained scores lower than specified score [9]. Percentile ranks of two examinees cannot be comparable for two different tests on same subject, while score is comparable for same subject test but not for different.

Subject Test scores should be compared only with other scores on the same Subject Tests (for example, a 680 on the Physics Test is not equivalent to a 680 on the Chemistry Test). Percentile ranks should be compared only if they are based on the same reference population. [8, p.7]

ETS writes the following about Score and Percentile rank.

Note that although a score represents the same level of ability regardless of when the score was earned, its percentile rank may vary, depending on the scores of the group with which it is compared [9].

Percentile ranks shown on score reports are based on the performance of the current reference group for each test regardless of when the scores were earned. The percentile rank for any score may vary over the years depending on the scores of the group with which the score is compared. Thus, when two or more applicants are being compared, the comparison should be made on the basis of their respective scores.[8, p. 9]

At the score of 700 in Computer Science there is a 42 percentile rank for the performance of the all examinees who were tested between July 1, 2004 – June 30, 2007 (see [8, p. 14]) and 40 percentile for the performance of all examinees who tested between July 1, 2003 – June 30, 2006 (See [10, p. 14])

One can find the following about percentile rank by flipping the “Report of Scores*”.

The percentile ranks in this report indicate the percentage of examinees who scored below your score. Note that these percentile ranks may be different from those that applied when the score were originally reported to you if the scores were earned prior to July 2009. This reflects annual updating of these data to permit admission officers to compare scores, whenever earned, with those for a recent reference group.

Formula score is the number of correct responses* minus one-fourth the number of incorrect responses rounded to the nearest whole number. The maximum formula score depends upon the total numbers of MCQs in the test.

* Result card sent by ETS, USA to the examinee or score recipient.
** Here “responses” mean questions which are usually multiple choices.

Is it an international test?

It should be remembered that the GRE tests provide measures of certain types of developed abilities and achievement, reflecting educational and cultural experience over a long period. Special care is required in interpreting the GRE scores of students who may have had educational and cultural experiences somewhat different from those of the traditional majority. [8, p. 8]

HEC officials are saying it an “international” test but the facts are against it. In the following table the number of examinees, all over the world, are given who took test between July 1, 2004, and June 30, 2007 [8, p.14].

Please note that computer science is one of the biggest fields of study and only 1870 examinees took the GRE Subject Test in a year.

General Rules for PhD Scholars

If someone gets the admission in PhD, then generally he/she has to fulfill the following requirements to get the PhD degree (no doubt, these are major requirements).

• Approval of Research Proposal or Synopsis (before or after the admission)
• Course work (not compulsory all over the world)
• Research paper (not compulsory in some countries but compulsory in Pakistan.)
• Thesis writing
• Defense of the thesis

If the thesis of some Pakistani PhD scholar is ready for submission then he cannot submit his thesis because of the extra imposed condition of GRE Subject Test, which is usually used as a recommended test (but not required) to get admission in the few universities of USA (see number of examinees per year as a proof). Also this shows that research paper(s) and thesis of the PhD scholar have no worth and no quality without this test.

It is also strange that if the PhD scholar has qualified GRE Subject Test then all the work (research) done under the supervision of such person (supervisor), who is usually non-qualified in GRE Subject Test, attains HEC quality standard.

Taking the GRE Subject test and other information

If someone decides to take GRE Subject Test, then the only way is online registration through ETS website by using online payment mode (by Credit or Debit card).

• Test Fee: 160 US Dollars (Rs. 14000 approx.)
• Duration: 2 hours and 50 minutes
• Conducted thrice a year (in April, October, November)
• Centers (Pakistan): Islamabad, Karachi and Lahore
• Limited number of seats in every center.
• Result announcement after 40 days of the test date.

Conclusions

1. The GRE Subject Test spans 2 hours and 50 minutes, at the price of US$ 160 (approximately Rs. 14,000). It is not affordable for a majority of the students and this heavy cost puts real extra burden on these students. In fact, many good students, who would easily pass it, hesitate to take the GRE Subject Test because of its huge fee.
2. Percentile rank is best to compare the students of same subject in the same test (in which they appeared) but it is not recommended to be used as a tool for making merit.
3. ETS recommends the use of “Scaled Score” for comparing the ability of students as they appear in the test at different schedule (three times in a year).
4. This test is not generally meant to be used for the students and institutions outside the United States.
5. Most public sector universities in Pakistan are not score recipients of ETS and it is very easy to use fake result cards to get admission, and the ETS never confirms the result to institutions if they are not score recipients.
6. It is very difficult to register for the test because many students don’t have Credit or Debit cards for payment through the internet.
7. Pakistan is a very big country and this test is conducted only in three cities, with limited number of seats at each location. This is really creating a big problem for the students living far from these cities.
8. The number of examinees per year clearly indicates the popularity of the test. For example, 1870 students in the subject of Computer Science and 2797 students in the subject of Chemistry appeared in a year all over the world.
9. Percentile rank actually is a comparison of the students of certain group taking GRE Subject Test in a period of last three years with respect to the test date. In eight subjects, Pakistani students are compared against a small community of students from all over the world, and for the remaining subjects the criteria is totally different.
10. The HEC team was unaware of the true mechanism of the GRE Subject Test. Yet, they decided that a PhD scholar must appear in this test, just to meet international standards no matter what he/she will score.
11. There are a lot of deficiencies in the HEC letters regarding GRE Subject Test. Against the decision of 7^th meeting of Quality Assurance Committee of HEC, the HEC officials are still communicating it as an international test.
12. All the grading terminologies used by the HEC don’t match with the ETS standard terminologies.This clearly shows that HEC officials are unaware of the objective and mechanism of the GRE Subject Test.

References

[1] Minutes of the 7th meeting of Quality Assurance Committee dated 19th April, 2005
[2] No 1-15/Adv(QA&LI) /2006/1394 dated May 30, 2006
[3] No. 4-7/CHR/HEC/07/807 dated April 3, 2007
[4] No. 1-10/(ED)/HEC/2008/96/ dated July 14, 2008
[5] No. 1-G/DD-QA/HEC/2009/45 dated October 23, 2009
[6] No. 4-7/CHR/HEC/2010/06 dated January 11, 2010
[7] DD/QA/HEC/NUST/2010/224 dated March 05, 2010
[8] GRE Guide to the Use of Scores 2008-09
[9] Interpreting Your GRE® Scores (2008-09)
[10] GRE Guide to the Use of Scores 2007-08

All the references can be downloaded from here.

Atiq-ur-Rehman is a PhD scholar at the Abdus Salam School of Mathematical Sciences in Lahore, Pakistan. His areas of research are difference and functional equations, real functions, inequalities in monotonic, and convex functions and he has 8 research papers. The views expressed in this article are solely those of the author and do not necessarily reflect the views of STEP.

In an effort to enforce quality, the HEC recently announced that they would not recognize PhD degrees awarded unless the recipient manages to score a 40 percentile on the GRE subject test at the time of admission to the graduate program, reported here. This is a revision of HEC’s earlier policy, announced four years back, that the GRE subject test must be cleared before submitting the thesis. The announcement has proven controversial among PhD instructors and their students.

The controversy spilled over in Quaid-e-Azam University (QAU), where the QAU syndicate and the QAU academic council are split over the issue (reported here). Dr. Sohail Naqvi, Executive Director of the HEC, was challenged by protesters during his visit to the QAU campus. Prof. Pervez Hoodbhoy, the chairman of the Physics department at QAU, supports the initiative by the HEC and recently wrote in its defense; we have shared his article here.

So, should PhD students need to clear the GRE before being awarded their PhDs?

LUMS has been shut down for a week. And as I said to a friend, maybe it’s a good thing. Because we, the ones at LUMS, are so disconnected from Pakistan, that it had to take the death of seven students, seven of our people, to make us pause our movies and type google news instead of facebook in our urls. Because with the divide between classes here, it’s as if Pakistan is two countries; one where all the news comes from, bombs, stampedes at ration lines, acid attacks, rapes and one where LUMS is, where people (even a few crazy women) can walk the clean quiet streets of defence and cantt, where people dine in every increasing style, where the only effects of terrorism are a few tiresome roadblocks and the echos of blasts far far away. Terrorism has reached our country now and we are slightly disturbed at it.

So finally the budding intellectuals of Pakistan have turned their attention to this problem that they had heard about but never really experienced before this. But this turned out to be a problematic exercise for me. Ask me what I want to see in this country. And I can flood your ears with concepts such as pluralism, freedom, justice, democracy, even anarchism in the style of Noam Chomsky. But ask me, “how do you get to there, from here?” And I am mute. And my hands drop uselessly to my sides. Or rather, they go up and cover my eyes because there is no point of sight when you can’t move.

My philosophy won’t work here because I know what I want but I have no way to make it happen.

I want these killings to end. I want the discourse to become more nuanced than “Islamic terrorism”. I want people to realize that all Muslims aren’t killers and that to say so, even as a joke, is to open the door to a dangerous generalization. I want Muslims to be able to criticize Israel on it’s war crimes without being attacked as fundamentalists. I want Muslims to stop defending what’s happening in our country and stop using conspiracy theories to deflect blame onto the US and its cronies. I want things to become less simplistic. But it can’t happen. Because there are certain interests for USA in creating the image of a Muslim terrorist. Because there is a certain complacency in the Muslim attribution of blame to the west. We all want to blame someone because then the responsibility to fix the problem is placed on whoever is blamed. And man is inherently lazy.

I want the people of Pakistan to be united. But for so many groups with so many nationalist claims to unite under one banner or one leader or even one party is impossible.

I want Pakistan to be partitioned into it’s four provinces. But for that to take place without bloodshed is a myth unheard of in our part of the World. And then there is the problem of our geographical location. A problem that we refer to with a mixture of pride and sorrow. The problem is that we are essential to too many interests. We are a troublesome neighbor to far too many important countries for those interests to allow us to divide for internal peace. Better a war ridden Pakistan than four small states who don’t border all those territories that actually matter to the World.

I want us to not think in terms of nationalism and patriotism. Because what use are these constructs? Why is a fellow Pakistani worth anymore than someone who just happens to be born in Iran or India? Why can’t we cherish human life and human development regardless of which boundary it occurs in? We cannot because such trajectories of thinking have never been offered to us. We cannot because if every Pakistani began to think of him or herself as an individual and began to work for their own benefit, all Pakistan would get is a slap from the invisible hand. Individualism abroad means greater progress because there are structures in which that individualism is exercised. Here, individualism is destruction. It is individualism that makes so many LUMS students rejoice that our university has closed in the middle of exams, because they hadn’t studied and would have scored “below the mean” in a course. Yes, there are Pakistanis who are rejoicing in this moment.

I want such Pakistanis, all Pakistanis to realize what it means to be at war. I want us to work. Ceaselessly. Because work alone can produce results and yes, it might be a Western concept to cherish work oh-so-much instead of sipping tea with the family but I want us to realize that tea and drawing room chats won’t do anything for us. But we can’t. Because when we go to school we die; when we talk out loud, we disappear; when we write, they write back threats. And why should we work? Why should we bother, sitting in our generator powered homes, surfing the net on our shiny laptops, going to cafes and stealing kisses on campus? Of course the greatest issue for LUMS to consider is kissing on campus. Of course. Because what effect does Pakistan’s appalling Gender Equity Index ratings have on a campus where women make up nearly half the student body? What effect do rising food prices have for us, with our regulated cafeteria prices?

So maybe it is good that LUMS has shut down for a week. But will this week change anything, I wonder? An essay here, a comment there. And this week will pass. And the rat race within LUMS will begin again. And you and I, as rational actors, will realize that nothing we do can help Pakistan and we should simply help ourselves by burying our heads in our imported traditions, getting the grades and rushing abroad the first chance we get.

I want things to change. But they won’t. Because we are stuck, you and I and scores like us. And I don’t know how we can get unstuck.

To seed this discussion, we present commentary from Dr. Pervez Hoodbhoy and Dr. Atta-ur-Rehman. Dr. Hoodbhoy presents his opposing point of view, arguing that the measures presented in the article were inadequate, and further that the conclusions drawn from the metrics were flawed. Dr. Atta-ur-Rehman, founding (and former) chairman of the HEC, who led the higher education reform effort during his tenure, responds by pointing to data that, in his view, shows the depth and breadth of the reform’s success.

We invite our readers to contribute their thoughts on what metrics are appropriate for measuring the success of higher education within the context of Pakistan.

NOTE: Both commentators have significantly shaped the landscape of Pakistani education over the last few decades. We request our discussants to avoid personalizing the discussion and to maintain a civil and constructive tone.

SYNOPSIS (We will continue to compile the synopsis of the discussion as it progresses: Last Update 8:02am EDT, September 22nd, 2009.)

The metrics suggested, thus far:

1. QUALITY OF RESEARCH (Editors, Zeeshan Khan): Citation and variants on their measures like the h-index, are standard measures of research quality on an individual, institutional, and national level. Number of patents is another measure, though .
2. QUANTITY OF RESEARCH (Atta-ur-Rahman): This measures gross research activity.
3. QUALITY OF TEACHING (Pervez Hoodbhoy): Metric?
4. QUALITY OF UNIVERSITY GRADUATES (Pervez Hoodbhoy, Fakhruddin Habiby): Surveying employers or assessing performance in international tests.
5. ACADEMIC FREEDOM (Pervez Hoodbhoy): Metric?
6. ACCESS TO UNIVERSITY FACILITIES (Atta-ur-Rahman, Khurram Shafique): Libraries, laboratories, internet connectivity, communication facilities, sports facilities,
7. EVALUATION BY NEUTRAL EXPERTS (Atta-ur-Rahman): Survey of a group of neutral experts, like the World Bank, USAID, etc.
8. UNIVERSITY ENROLLMENT (Atta-ur-Rahman): The increase in university-going adults can be measured by census.
9. UNIVERSITY-INDUSTRY LINKAGE (Fakhruddin Habiby, Anwar): The number of industry supported projects which were initiated with University-Industry partnerships and their effectiveness based on industry feedback.
10. LOCAL RELEVANCE OF RESEARCH (Editors): Metric?
11. QUALITY OF CURRENT STUDENTS (Pervez Hoodbhoy, Khurram Shafique):  Performance in standardized tests conducted every year, performance in local and international competitions such as Mathematics Olympiads and Programming Contests.

Notes:

(Pervez Hoodbhoy) Self-citations are a serious problem when using citations as a metric for measuring quality of research reform. [paraphrased]

(Abdullah Sadiq) While strengthening the research effort in the universities is important, the most urgent need is to concentrate on producing quality teachers for the lower tears of education. [paraphrased]

(Khurram Shafique) A pedestrian publication in the field of networking or multimedia is likely to receive more citations than a good publication in a less explored field in mathematics, say, non-standard analysis.

(Fakhruddin Habiby) another ‘tool’ that is used to push the citation number higher is formation of ‘citation-coalition’ within research groups.

(Omar Javed) Categorization of universities into subsets, and adoption of relevant performance criteria for each subset. Three fundamental questions: what is taught (Undergraduate and Graduate Instructional Program classifications), who are the students (Enrollment Profile and Undergraduate Profile), and what is the setting (Size & Setting)”

(Shafiqur Rehman) …the success or falure or HE reforms must only be judged by opinion of the common stake-holders (students, teachers and administration) of the public sector universities.

(Affan): we need to tweak ratings/rankings such that we are able to measure any progress happening in Pakistan, progress small enough that it is not lost by existing metrics.

There are four aspects of the comments of Dr. Pervez Hoodbhoy that need to be considered:

1. Firstly, Dr. Hoodbhoy himself admits that there has been a huge increase in international publications at QAU after HEC came into existence when he mentions the number of international publications in the two time periods. Strangely, he picks a six year period, 1998-2003, and then compares it with the subsequent 4.5 years (?) , 2004 to mid 2008, (the correspondence occurred in August 2008, so he could not possibly have had access to the figures for the entire year) I can only assume that he has mentioned 2003 by mistake in the second “5 year” period as there is no reason to include the publications of the year 2003 in both time periods, which he has done. It is clearly unfair to take two time periods of different durations and compare them.
2. In the first 6 year period (1998-2003), Dr. Hoodbhoy admits that there were only 631 research publications from QAU, but in the second 4.5 year period these had risen to 1482 research publications, a tripling of publications on average per year, even by his own estimates.
3. As the HEC programs began in 2003 and their real impact occurred 2-3 years later, a year-wise comparison is far more relevant than an average over a 5 year period as the dramatic change that has occurred gets partly masked when a 5 or 6 year average is taken, though it is still very visible. Dr. Hoodbhoy ignores the figures that Dr. S.T.K. Naim had worked out that in the year 2004, there were only 84 research publications from QAU (an average of only 7 publications per month), but by 2008 they had increased many fold.
4. The citations argument used by Dr. Hoodbhoy is invalid as citations increase with the passage of time. Dr. Hoodbhoy, therefore, wrongly compares the citations of papers of an earlier period with those of a later period. To clarify this issue further, if two papers of equal quality and in a similar field are published, say in 1998 and 2007, and the citations of both are counted in 2008, then the paper which was published in 1998 will have accumulated more citations by 2008 because of the much longer 10 year time period, than the paper published in 2007, as that would have had only one year for the citations to accumulate. Dr. Hoodbhoy is therefore comparing apples with oranges when he tries to compare citations of papers published in an earlier period with a later time period. In order to fairly compare citations, the same duration of time period must be taken. Thus if one takes 1998 publications and counts the citations till 2008, then one will need to take the 2008 publications and count their citations till the year 2018, before one can compare the figures for the citations of the two sets fairly.

The undeniable fact is that the total number of research publications from universities in Pakistan was only about 600 per year till 2001 but then started rising rapidly, and by the year 2008 it had increased to over 4,300! Brazil achieved such an increase over a 35 year period between 1960 to 1995, which Pakistan achieved in only 6 years. After my appointment in March 2000 as the Federal Minister for Science and Technology in Pakistan, I convinced the government to enhance the budget for science and technology in Pakistan by 6000% between July 2000 to October 2002. After my appointment as Chairman, Higher Education Commission (Federal Minister) the budget for higher education was similarly increased by 2400% during 2003 to 2008. Major achievements during these periods were:

1. Establishing 51 new Universities and awarding institutions during 2002-2008,
2. Tripling university enrollment (which had reached only 135,000 from 1947 to 2003) to about 400,000 in 2008,
3. Establishing a powerful Digital Library which provides free nation-wide access to every student in every public sector university to 45,000 textbooks/research monographs from 220 international publishers as well as to 25,000 international research journals,
4. Establishing video-conferencing facilities in most public sector universities that allow lectures to be delivered live and interactively to students in Pakistan from technologically advanced countries
5. Enhancing salaries of academics so that salaries of University Professors were increased to a level about five times the salaries of Federal Ministers, with a corresponding reduction in tax from 35% to only 5%, in order to attract the brightest young men and women into academia,
6. Promoting research through a massive research grant program which resulted in a 600% increase in ISI abstracted publications from about 600 per year in 2001 to 4300 research publications in 2008, accompanied by about 1000% increase in international citations in the same period,
7. Placing a satellite in space (Paksat-1) which is now used for distance learning by the Virtual University,
8. Establishing video-conferencing facilities in most public sector universities and initiating a lectureship program, allowing live interactive lectures to be delivered from technologically advanced countries,
9. Providing free access to scientists/engineers anywhere in the country to sophisticated instruments installed in any institute in Pakistan.

The Bottom Line: In the final analysis, it is not what I or Dr. Hoodbhoy think about the developments, but what is the opinion of neutral international experts who have carried out detailed year-long reviews of the developments during the period that I was heading the Higher Education Commission. A few extracts are given below:

1. Prof. Fred Hayward (independent international educational consultant from USA) carried out a detailed analysis of the developments and published an article entitled “Higher Education Transformation in Pakistan: Political & Economic Instability,” Date: Number 54, winter 2009 Source: International Higher Education Quarterly. I quote: “The news about Pakistan over the last few years has been dominated by reports of political turmoil, terrorism, religious fundamentalism, economic decline, and the Afghan War. What has been missed is the phenomenal transformation in higher education over the last six years, which represents a critical development for Pakistan and a potential engine for growth and national recovery.”
2. Report of US-AID about HEC states that “We are very impressed with the breadth, scope, and depth of the reforms implemented by the HEC since 2002.  No other developing country we know has made such spectacular progress.”
3. World Bank Report is very complimentary of many excellent programmes introduced.
4. British Council: The report states: “I have worked in many countries in South America, the Middle East, North Africa, and in Russia and India, over the last six years.  None in my view, with the exception of India, has the potential of Pakistan for the UK university sector, largely because of the dynamic, strategic leadership of the Chairman of HEC”.
5. Nature: Several articles and editorials have appeared in the world’s leading science journal “Nature”  (the most recent in the issue published on 3rd September 2009) in which the very significant progress made by Pakistan in the higher education sector has been applauded and the need for the new government to built on the solid foundation laid has been stressed.
6. Science Watch (Thomson Reuters) has ranked Pakistan as a rising star in five disciplines, more than in any other country of the world.

Join the Discussion!
What are the correct metrics to measure higher education reform in Pakistan?

Related Posts:

1. Nature’s Coverage of Higher Education Reform in Pakistan: A Response from Prof. Pervez Hoodbhoy
2. Nature’s Coverage of Higher Education Reform in Pakistan: A Conversation with Athar Osama
3. Pakistan’s Higher Education Funding Holds Many Lessons for Developing Nations: Nature

Editors Note: Dr. Atta-ur-Rahman is a leading scientist and scholar in the field of organic chemistry from Pakistan. He has served as the Federal Minister for Science and Technology, the Federal Minister/Chairman of the Higher Education Commission, Adviser to the Prime Minister on Science and Technology, and the President of the Pakistan Academy of Sciences. Dr. Atta-ur-Rahman is credited by many for reviving the higher education and research practices in Pakistan. The views expressed in this article are solely those of the author and do not necessarily reflect the views of STEP.

Unfortunately, I find the response as unsatisfying as the original article. Since Nature is unwilling to accord me a chance for a satisfactory reply on its pages, I shall clarify the basis of my criticism in some detail here.

In the said article, strong conclusions have been derived from weak data. The authors have not dared to ask the basic questions whose answers are essential for ascertaining whether there has been actual progress in Pakistan’s higher education system and, if so, by how much. Instead, in giving a thumbs-up, numbers have been quoted that have doubtful significance. Take, for instance, the claim that:

“In mathematics, for example, an average paper by a Pakistani author is cited around 20% more than the worldwide average for the discipline”.

Before returning to what some of the right questions might have been, let me give two reasons why the above claim – even if true – carries little meaning.

First, self-citation is a far more serious problem than the authors are willing to acknowledge. It is also one that they admit to not having investigated. The data on Pakistani research papers shows that subtracting out self-citations drastically cuts down on actual citations – there are often 2-3 self-citations for every real one! The reader is urged to carefully study my email correspondence of last year with Dr. Atta-ur-Rahman (see appendix below) who, at the time when he was HEC chairman, had made similar claims that I disputed as being false. To interested readers, I have made available (in pdf form) the Thomson Scientific data that I have quoted in my correspondence here (1998-2003) and here (2003-2008).

Second, many authors of the relatively more cited mathematics papers from Pakistani institutions are not Pakistani nationals. High salaries offered to foreign faculty by the HEC brought to Pakistan a large number of well established mathematicians on short-term contracts from Russia, Poland, Ukraine, and China. This was probably a good thing to do – in spite of the difficulty they had in communicating in an alien language and their consequent inability to teach well. Their papers, however, do not reflect mathematics in Pakistan. One sees a similar phenomenon in Saudi Arabia where foreigners are principally responsible for the kingdom’s large number of papers and citations.

In my opinion, instead of focusing on marginal matters, serious research on the state of Pakistani higher education, and of changes therein, would have first established appropriate metrics, and then sought answers, to the following key questions:

1. What is the quality of teaching in Pakistan’s public universities? There is often only a weak correlation between formal qualifications and subject competence, so simply counting PhD degrees is not very helpful in answering this. Far too frequently one sees professors of English who cannot speak or write a single sentence of grammatically correct English, physics professors who are stymied by Newton’s Laws, and biology professors whose knowledge is frozen in some pre-Darwinian age. But does such basic incompetence exist at the 20, 50, or 70 percent-level? Higher? Lower? What evidence exists that the HEC’s reforms improved the situation?
2. Is there evidence that there has been improvement in the selection process for students in public universities, or that of the quality of their graduates? Proof of the latter, judged by asking employers or assessing performance in international tests, would be a clinching argument for the success of HEC reforms.
3. Do campuses enjoy greater academic freedom, more seminars and colloquia, less violence by extremist campus groups, a pleasanter and more relaxed ambiance, and greater transparency in faculty selection? Surely these are critical to any reasonable assessment.

To get answers to questions like these requires extensive field work, and I certainly do not fault the authors for not doing this. But I was surprised that the Nature article, as well as the lead author’s response, merely says that the HEC’s experiment had critics, without citing any specific articles or the substance of those criticisms. There is not even a passing reference to the failed nine-university multi-billion dollar mega-project, tons of unused scientific equipment purchased for unknown reasons, dubious attempts to fund “Quranic Science” (that had to be hastily abandoned after the scheme was exposed), and the explosion in academic corruption set off by per-paper payments. Surely, these should not be brushed aside as “collateral damage”. In another country, those who massively squandered public money would have been thoroughly investigated by independent commissions, not praised for small things.

The Bottom Line: Well-functioning universities are the products of a complex organic and evolutionary process that is internal to a society. Money and facilities matter, but it is much more important for a university to have a forward looking world-view, an open environment, high ethical standards, a sense of collegiality and shared sense of purpose, and good governance practices. Sadly, the Nature article did not even mention these as significant.

Join the Discussion!
What are the correct metrics to measure higher education reform in Pakistan?

Related Posts:

1. Nature’s Coverage of Higher Education Reform in Pakistan: Comments by Dr. Atta-ur-Rahman
2. Nature’s Coverage of Higher Education Reform in Pakistan: A Conversation with Athar Osama
3. Pakistan’s Higher Education Funding Holds Many Lessons for Developing Nations: Nature

Editors Note: Prof. Pervez Hoodbhoy is a well-known Pakistani nuclear physicist and political-defence analyst. He is the Professor of High Energy Physics, and the head of the Physics Department at Quaid-e-Azam University, Islamabad, Pakistan. Prof. Hoodbhoy is a vocal critic of HEC’s policies and their impact. The views expressed in this article are solely those of the author and do not necessarily reflect the views of STEP.

This correspondence between Dr. Atta-ur-Rahman and Dr. Pervez Hoodbhoy is reproduced below with consents from both parties. It is exclusively concerned with a public matter, has no private content, and is largely focused upon the importance of self-citations.

Dear Dr. Pervez
I have probed into the situation at QAU, and it is quite the opposite of what you claim, as evident from the total publications and the total number of citations, as per above attachments. The HEC was established in October 2002. The funds started coming through from July 2003. The subsequent impact of HEC programs on research publications and citations is indeed amazing. In 2004 the number of publications was only about 120—by 2007 it has risen to about 380—a 300% increase! The citations in 2004 were about 800—-by August 2008 they have increased to about 2200 although we still have 4 months to go before the year ends—I suspect that it will be about 3200 by the end of the year—a 400% increase!

I shall be separately sending you a report after removing self-citations. QAU is also now included in the most cited institutions in the world (please see attachment)—-this was not to 4 years ago.

Please be fair and objective in your assessments. We may have made some mistakes, but much good has happened.
Kind regards
Atta

Dear Dr. Pervez
Following my other email to you this morning, I am now enclosing the citations of QAU after removing self-citations.As you would see, these have risen from only 84 in 2004 to 1413 in 2008 (with still 4 months to go). These will probably be around 1900 by the end of the year—a spectacular ten-fold growth! Dr. Naim has kindly had these searched, so if you have any queries about them, you may like to interact with her. She tells me that the situation is similar in many other universities—a long period of stagnantion
during the 1990s followed by a burst of activity in the last 4-5 years.
Kind regards

Atta

Dear Dr. Atta,

I am sorry about the late response to your three emails. First, thank you for doing whatever you did – the HEC’s notification of 14 July 2008, which specifies 40 percentile as the GRE passing marks, finally reached QAU departments today (without comment from the administration). In these times one has to be grateful for small things…

Regarding your email and the numbers cited therein: I can understand that you would like to feel upbeat about QAU having improved itself as a result of massive infusion of HEC resources. I too would very much like good things to happen, but perhaps one should not allow wishes to become conclusions.

Looking at the data that you had sent me and your inferences, I decided to do a little independent investigation using exactly the same database (ISI Web of Science) and exactly the same keywords (see attachments to this email). Here are the findings:

Period: 1998-2003
Number of papers published by QAU authors in the above period: 631
Number of citations to date: 4540
Number of citations to date with self-citations removed: 2,817

Period: 2003-2008
Number of papers published by QAU authors in the above period: 1482
Number of citations to date: 3667
Number of citations to date with self-citations removed: 1258

Some obvious inferences:

1. There is absolutely no evidence of real citations having increased; if anything, the numbers up to now show the contrary. While the number of real citations of papers published 2003-2008 may increase somewhat with time, currently they are quite a bit less from the earlier period when the HEC and your incentive system did not exist. Please remember that citations are cumulative over years. I have tried to use exact descriptions in the figures cited above. If I am wrong in any detail, or if I have missed something essential, I would like to be corrected. Unfortunately the data does not at all support your rather optimistic remark of “a spectacular ten-fold growth!”
2. The above data also indicates the disturbing fact that most of the time QAU authors cite themselves. Subtracting self-citations drastically cuts down on real citations – there are 2-3 self-citations for every real one!. Looking more minutely at the ISI pages, one also notes that many citations are by other members belonging to the same or other QAU departments. So the number of genuine citations gets cut down even beyond the numbers quoted above (2817, 1258)!

Just to get scales right, here are a few citation numbers from the Stanford Spires data base for Pakistani physicists over the length of their careers:
a) Riazuddin: 1479
b) Ahmed Ali (DESY, Germany): 9873
c) Abdus Salam: 14103

3. You are correct that the number of publications over equal (5-year) time periods has more than doubled relative to pre-HEC times. But this is clearly in response to the monetary incentives offered by PCST/QAU. A publication fever now grips our universities. It is difficult to defend the case that the number of papers published is proportional to the amount of research done.

You are, of course, free to have the data I have quoted above rechecked and I would be happy to answer any question that arises. Finally, please note that publications and citations were not central to my earlier expression of dismay at the quality of QAU education.

What bothers me much more is the lack of analytical and problem-solving abilities of our Ph.D graduates, some honourable exceptions aside. Poor performance in the GREs is one indication of the rot. This fact has indeed worried you a little, as you indicated in an earlier email to me, but I do wish you could understand the real gravity of the situation.

Regards,

Pervez

Dr. Atta-ur-Rahman’s comments on this post can be found here.