Nature’s recent article on higher education in Pakistan has re-ignited the debate on higher education reform, evoking strong responses from both supporters and critics of the HEC. Recently, we interviewed the lead author Dr. Athar Osama, to learn more about his wider conclusions, and his response to some of the criticisms of the methodology used in the Nature article.

To seed this discussion, we present commentary from Dr. Pervez Hoodbhoy and Dr. Atta-ur-Rehman. Dr. Hoodbhoy presents his opposing point of view, arguing that the measures presented in the article were inadequate, and further that the conclusions drawn from the metrics were flawed. Dr. Atta-ur-Rehman, founding (and former) chairman of the HEC, who led the higher education reform effort during his tenure, responds by pointing to data that, in his view, shows the depth and breadth of the reform’s success.

We invite our readers to contribute their thoughts on what metrics are appropriate for measuring the success of higher education within the context of Pakistan.

NOTE: Both commentators have significantly shaped the landscape of Pakistani education over the last few decades. We request our discussants to avoid personalizing the discussion and to maintain a civil and constructive tone.

The authors have not dared to ask the basic questions...

Read Dr. Hoodbhoy’s complete post here.

... it is not what I or Dr. Pervez Hoodbhoy think...

Read Dr. Atta-ur-Rahman’s complete post here.


SYNOPSIS (We will continue to compile the synopsis of the discussion as it progresses: Last Update 8:02am EDT, September 22nd, 2009.)

The metrics suggested, thus far:

  1. QUALITY OF RESEARCH (Editors, Zeeshan Khan): Citation and variants on their measures like the h-index, are standard measures of research quality on an individual, institutional, and national level. Number of patents is another measure, though .
  2. QUANTITY OF RESEARCH (Atta-ur-Rahman): This measures gross research activity.
  3. QUALITY OF TEACHING (Pervez Hoodbhoy): Metric?
  4. QUALITY OF UNIVERSITY GRADUATES (Pervez Hoodbhoy, Fakhruddin Habiby): Surveying employers or assessing performance in international tests.
  5. ACADEMIC FREEDOM (Pervez Hoodbhoy): Metric?
  6. ACCESS TO UNIVERSITY FACILITIES (Atta-ur-Rahman, Khurram Shafique): Libraries, laboratories, internet connectivity, communication facilities, sports facilities,
  7. EVALUATION BY NEUTRAL EXPERTS (Atta-ur-Rahman): Survey of a group of neutral experts, like the World Bank, USAID, etc.
  8. UNIVERSITY ENROLLMENT (Atta-ur-Rahman): The increase in university-going adults can be measured by census.
  9. UNIVERSITY-INDUSTRY LINKAGE (Fakhruddin Habiby, Anwar): The number of industry supported projects which were initiated with University-Industry partnerships and their effectiveness based on industry feedback.
  10. LOCAL RELEVANCE OF RESEARCH (Editors): Metric?
  11. QUALITY OF CURRENT STUDENTS (Pervez Hoodbhoy, Khurram Shafique):  Performance in standardized tests conducted every year, performance in local and international competitions such as Mathematics Olympiads and Programming Contests.

Notes:

(Pervez Hoodbhoy) Self-citations are a serious problem when using citations as a metric for measuring quality of research reform. [paraphrased]

(Abdullah Sadiq) While strengthening the research effort in the universities is important, the most urgent need is to concentrate on producing quality teachers for the lower tears of education. [paraphrased]

(Khurram Shafique) A pedestrian publication in the field of networking or multimedia is likely to receive more citations than a good publication in a less explored field in mathematics, say, non-standard analysis.

(Fakhruddin Habiby) another ‘tool’ that is used to push the citation number higher is formation of ‘citation-coalition’ within research groups.

(Omar Javed) Categorization of universities into subsets, and adoption of relevant performance criteria for each subset. Three fundamental questions: what is taught (Undergraduate and Graduate Instructional Program classifications), who are the students (Enrollment Profile and Undergraduate Profile), and what is the setting (Size & Setting)”

(Shafiqur Rehman) …the success or falure or HE reforms must only be judged by opinion of the common stake-holders (students, teachers and administration) of the public sector universities.

(Affan): we need to tweak ratings/rankings such that we are able to measure any progress happening in Pakistan, progress small enough that it is not lost by existing metrics.

37 Responses to “DISCUSSION: What are the correct metrics to measure higher education reform in Pakistan?”

  1. Sohaib Khan says:

    The question, “has research performance really improved?” can be answered only if we have an idea of how to measure performance. So, I think this debate is really about what should be the metrics to measure the research (and academic) performance of universities in Pakistan.

    As an example, one can perhaps look at the ‘Research Assessment Exercise’ done every five years in UK universities: (http://www.rae.ac.uk/ ). This is perhaps too detailed for us at this early stage, but still, a scaled down version of such an exercise may be extremely useful.

    • Before we tackle the question of research performance and metrics, two questions that must be answered are:

      1. What exactly is research? and
      2. What is the role of research in higher education, especially within the context of Pakistan?

      In addition to research, there are several other components that contribute to the quality of higher education. What are these components? and how much does each of these components contribute to higher education performance?

    • As a beneficiary of the policy of awarding scholarships and sending our students abroad for post-graduate study, I think I am in a good position to comment. I think HEC under the great leadership of Dr. Ata-ur-Rehman has unleashed a revolution, the benefits of which will be accrued by many generations to come. Pakistani students in every university have made great strides and achieved a lot.

      In countries like the United Kingdom where Pakistani is synonymous with Taxi Drivers and Fast Food joint-keepers, Pakistani students have gone and done good research. I have seen it first hand which is also corroborated through the information presented in the article in Nature. With all due respect to Prof. Hoodbhoy, I think he has reached new levels of skepticism which I have not witnessed ever in my short stint as a post-graduate researcher.

      As for the quality of students, I think Pakistan produces one of the best in the world. I say this after teaching for about four years in a university of international repute in UK. I agree that the education system is not in an ideal state but you can say that about most education systems in the world. There have been severe criticisms recently of the American High School as well as the British GCSEs by the Americans and the British themselves. I think the most important aspect that defines a Pakistani student or any other student from a developing country like ours is the focus, unity of purpose and the hard work that I have seen my colleagues and peers do. I have come to the conclusion that at the end of the day the person who succeeds is not the one who is the best equipped or has the best facilities but is the person who refuses to die and give up, the one who fights hard till the end. I think this is the most important characteristic of our people that I have witnessed first hand wherever I have lived in the world.

      I thank Dr. Ata-ur-Rehman, HEC, Government of Pakistan and most importantly people of Pakistan for reposing trust in me and countless like me and sending us abroad to pursue higher education. I hope and wish that I have done justice to the aspirations of the people of Pakistan (I have been a recipient of a reward by the Royal Society and published papers at the best conference in my area of study). Alas! I would just say Pakistan Zindabad!

      Editor’s Note: This comment has been edited for content. Please refrain from personalizing the discussion.

    • I think RAE is a good approach which could be followed in Pakistan. Although I agree that its very elaborate but at the end of the day the most important aspect of any education system is the good novel knowledge that it generates. Which can only be judged by journal and conference publications. I suggest that another study should be made which would include all people whose study is funded from Pakistani origin. A list of publications must be acquired. Their impact factor be summed or used to derive a measure of quality of research. Any worthwhile criticism must be incorporated as well. Citation index used by the authors is as we are well aware is one of the most dependable factors in determining research quality.

      As for secondary and higher secondary education, I think efforts must be made to standardize our curriculum across all boards of education. I think a lot of work has been done in the area. More needs to be done. The most important thing is that all segments of the society must be taught at the same level. I suggest formation of an organization like the HEC for the Secondary and Higher Secondary Education run by professionals just as HEC whereas ministry of education only serve as an overall policy tool for the politicians and generals, whoever is ruling Pakistan. A chunk of funding from MoE should be diverted to this commission and it should be tasked with standardizing, reviewing, improving and democracizing our education system. And I believe that this should be done as soon as possible.

  2. Shahzad Khan says:

    Both Dr.Rahman and Dr.Hoodboy have my deepest respect, and it is difficult to challenge the contributions that these sincere, hard-working individuals have made in our society. I appreciate the sincerity with which they have approached the problem of quality in the HE system in Pakistan, and the countless lives of budding academics whom they have influenced directly and indirectly.

    The gist of Dr.Hoodboy’s critique is that we should not lose cognizance of the quality of the research, and the researchers and ensure that we track the magnitude of the research output accurately. He was very concerned with the issue of evaluation, which is an important aspect of ‘declaring success’.

    Indeed, a recurring failing of the primary/secondary education in the public sector in Pakistan is that quality has been watered down so much that education may perhaps even be considered counter-productive (as it wastes time, does not build problem-solving skills, and saps the confidence of the students).

    However, in my opinion Dr.Atta’s approach was a very healthy one. He introduced solid metrics into the HE process, and matched these to incentives. He also facilitated the provision of ‘essentials’ required to carry out research (i.e. funding, digital libraries, and changes in policies).

    Even if these were not perfect (and nothing is perfect the first time around), they have made a difference. Obviously, there are still challenges. Even today, I hesitate to return to Pakistan to teach as I am afraid of being marooned outside the international research community; I recall sadly how often I’d get papers accepted (in international conferences) while working in Pakistan in partnership with enthusiastic, bright-eyed students, and be given authorization to ‘only register’, and not fly out and present the paper, and thus lose the chance to interact with my peers. How can we build enduring partnerships with peers in the international community, unless we can meet them at least once or twice a year? When I protested, I was told to get journal publications instead of conference presentation, which is not how most research works.

    I also recall teaching 21 credits one semester, which does not leave sufficient time for research or even adequate student supervision/mentorship. In comparison, my advisor in Cambridge (UK) only taught 8 to 16 _hours_ a year, freeing his to focus more on research. He was (co)supervising 3 PhD students and another 4 MPhil students though. My university research partners here in Ottawa, Canada typically teach 6 credits a semester.

    However, even though I was overworked, frequently admonished for initiatives (which occasionally tend to not work out as planned) and deluged in work that was not part of my core competency (interviewing janitors?! and investigating staff ethics cases), I had the honour of teaching over 300 students in my three years in Pakistan. I am delighted to say that a substantial proportion of these students have gone on to pursue higher education abroad, and I still collaborate with some in Pakistan, Europe, Japan/Korea and North America as part of my consultancy and budding startup.

    Even more importantly, I had a chance to develop an excellent relationship with my mentor who had come to Pakistan for a year as part of the Foreign-Faculty programme run by the HEC. We both had an interest in artificial intelligence, and he encouraged me to apply to Cambridge (which I frankly did not think I could get into), and provided me a strong letter of recommendation. I also was one of the 20 individuals who was awarded the overseas PhD scholarship in 2004 from the HEC (which was either the 1st or 2nd years this was run). I decided to not take the scholarship, as I was not impressed with the thought of being on a 5 year bond, and I hope it benefited someone else. It was very difficult financially, but I managed to get through the programme by consulting on industrial project and due to the sacrifices made by my wife (who worked as a research associate in another department in Cambridge, even though we had a two year old child at home). However, in other circumstances the scholarship would have made all the difference, as funding is a necessary element for higher education enrollment.

    As my teaching career in Pakistan was between 2001 and 2004, I had a chance to observe first-hand how the new policies effected the universities, and let me assure you that they did have a galvanizing effect on aligning interests towards research. I am sure that things have improved even further since then. However, I cannot comment further as I am currently outside the system, although I do encourage others to add their thoughts below.

    • Saqlan Naqvi says:

      The discussion/comments on what HEC has been successful at achieving or have failed to achieve, must be concluded with the measures for the future course of action. It should not merely end only with praises or opposing remarks.

      Efforts of those who have created this webpage are highly appreiciated. However, they are also requested to

      i)make a list of recommendation based on the comments of participants of the current discussion and forward it to current HEC management

      i)try to bring into discussion various programs of HEC separately and seek the opinion of people on
      a) program may be continued as such
      b) program may be continued with specific modification, which may be recorded and forwarded to current HEC managements
      c) program may be discontinued

  3. Ismat Riaz says:

    In terms of HE, I would include the under-graduate programmes as these lay the foundation for higher studies at post graduate and doctoral level. Yes, incentives were given to write up papers and contribute to journals as part of the tenure track system instituted by HEC, but, standards were not insured on an on going basis. I came across a journal article from the University of Education, lahore, which made comparisons between the UK and Pakistan and how each has provinces which have different standards in education. This article served no purpose as there is no comparison in education levels between the UK and Pakistan. Just a mindless bid to produce an article is not what HE is all about.It is serious business to produce research and answer hypotheses based on data collected ethically and honestly.

    The colleges are now adopting the 4 year degree but evaluation systems are still unclear. Is the semester system will be the mode for all or if universities are not as yet equipped to handle semesters, are they or will they revert entirely to the annual system of examinations. Are the libraries at all universities under HEC updated and fully equipped with books in both English and Urdu to service students who should now be doing papers and assignments based on research through book referencing? The faculty at these colleges affiliated to universities being monitored to do all of the above and sticking to a prescribed standard? Or, is a team sent off to these colleges to review work and then compile a report to say all is well. Is a standards form evaluating students’ work available to check on the improvement in HE? I think not as a really conversant team of foreign qualified teachers/professors should be initially monitoring and evaluating the new system put in place by HEC.

    Frankly, maybe a lot was done to upgrade Pakistani universities to bring them up to the standard of foreign universities but it needs a lot of capacity building BEFORE the reform is initiated to make sure that things are going in the right direction and with full speed.

    Till under graduate degres are made to a standard expected of the kind of critical thinking needed to read journals and books, HE will not radically improve.

    • Ismat, I think you have made an important point here. Undergraduate students represent a very large portion of HEC’s consumer base, however, the current metrics focusing on the number of research papers, the number of citations (including or excluding self citations), and the number of awarded doctorates are significantly skewed towards graduate studies. While a case can be made that good researchers are often good teachers, and I believe some studies have shown this correlation, it still is a very indirect way to measure the quality of undergraduate education. It should also be noted that unlike Western universities where graduate schools enjoy an influx of talented foreign students, Pakistani graduate schools solely rely on the graduates of local schools and hence the quality of the former is highly dependent on that of the latter.

      How to measure quality of undergraduate education and/or schools? We can probably divide the metrics into two categories, i) input to the program, and ii) output of the program.

      Here are some suggestions:

      Input Measures
      1. Quality of teachers
      a. Academic Qualification (in terms of highest level of education attained). Somehow the quality of degree awarding school should be taken into account here as well. A degree from Tier 1 school is not directly comparable to the same degree from Tier 4 school.
      b. Research Qualification, how to measure it is another debate but one good way could be peer-evaluations and recommendations, a common metric used for tenure evaluation in US academia.
      c. Subject Matter Expertise, probably measured via a standardized test such as GRE subjective.
      d. Student Evaluation

      2. Quality of Facility, such as, libraries, laboratories, communication facilities, sports facilities, etc.

      3. Quality of Avenues of Non-Academic Activities, such as, student bodies, student newspapers, literary societies, etc.

      4. Quality of Incoming Students: Measured via standardized tests, such as SAT, admission tests, board examinations, and quality of previously attended colleges and schools.

      Output Measures:
      1. Quality of Current Students:
      a. Performance in standardized tests conducted every year.
      b. Performance in local and international competitions such as Mathematics Olympiads and Programming Contests.

      2. Quality of Graduates
      a. Performance in standardized tests such as GRE
      b. Impact of alumni in their respective careers, an important factor that is very hard to measure.

      The list is by no means comprehensive or conclusive but I hope it will spur further discussion and suggestions.

    • Bilal Zafar says:

      Khurram and Ismat,

      Both of you make an important point, that it is the undergraduate education that is perhaps most important sector in our higher education system, and the success or failure of this sector should indeed be the focus of attention.

      What you are ignoring, however, is that research publications – their quality and quantity – is not just an indicator of “what” is happening in our institutions of higher education, but also of “who” is at these institutions.

      In this ‘flat’ world, to attract quality PhDs, Pakistani universities have to provide incentives that match the expectations of promising researchers graduating from the best universities in the world. Of those incentives, as people who are at the front lines of the fight to get talented and accomplished Pakistani PhD to return to Pakistan will tell you, the most important (from a professional standpoint) is a culture that appreciates research output. For a fact, many of the best young PhDs now teaching in Pakistan would not have returned had they not seen research activity at these institutions. Tenure-track system, access to research grants and a strong (not necessarily large, but strong) graduate program form that package of incentives that these highly sought-after researchers expect. There simply is no other way to attract and retain the most qualified teachers that we all would agree are necessary to improve the quality of our education at the undergraduate level.

      Therefore, a strong graduate program is not a distraction from the goal of achieving quality undergraduate program, but a necessary pre-requisite.

    • I am neither ignoring nor denying the importance of the presence of quality researchers in our institutes, graduate or otherwise. As a matter of fact, quality of research was one of the measures that I suggested above. I am merely stating that this is only a small part of metrics for the performance evaluation of undergraduate education. It may well be a necessary metric (though I doubt that too) but definitely not a sufficient one.

  4. Affan says:

    Criticism, honest and well meant, is always good. I see the tension here as very healthy, and would allow HEC to stay on its toes. Infact, the very point that someone is making coherent criticism of their work is an indication that they have changed something, if just the quality of discourse in our higher education circles.

    Going back to an earlier post in a different thread by me, I think the issue of ascertaining research progress needs to balance the nascent state of Pakistani research with rigorous methods and metrics that can detect any progress. The problem, however, is that such metrics do not exist and the entire process of ranking research, the researcher, their publications and the location of publication is generally quite subjective. All these metrics are peer-reviewed, i.e. if you ask an honest opinion of the peers in a particular field their response— about the research being done, the researcher him/her-self, their publications and the quality of conferences/journals for that field— would perhaps be the most correct metric.

    The problem with such a process, specifically with Pakistan, would be that their are very few researchers, and nearly every field has only one or two active people. Sampling such people would introduce a large statistical bias; what then is the solution? One option would be to include, based on recommendations by pakistani researcher, a few non-Pakistani researchers of good reputation, for each field. How to go about convincing sufficient people, what is the right amount of sample size?, what is the granularity of the division of fields? all of these are inherently open research questions. Perhaps there can be a small research proposal to act on such a possibility , on steps similar to those taken for the Australian CORE ranking system.

    • I completely agree with your statement that the number of publications and the number of citations, while quantitative metrics, are still very subjective. In most cases they result in comparison of apples with oranges. Ten publications in journal X are not the same as ten publications in journal Y. The same is true if you replace the term ‘journal’ in the sentence and replace it with ‘conference’ or ‘area of research’. Similarly, the number of citations are also a function of the number of researchers and the amount of research activity in a given field or even the age of the field. A pedestrian publication in the field of networking or multimedia is likely to receive more citations than a good publication in a less explored field in mathematics, say, non-standard analysis. And these are only few of the many factors that contribute to the number of citations of a given publication. Moreover, you can pick any researcher with a large number of publications and you will find that only a small number of his/her publications contribute heavily to all his/her citations, meaning that even a good researcher rarely produce a high impact publication in his career.

      I also think that you are right in pointing out that peer evaluations is a way to go towards measuring an individual’s research performance. However, I do not agree that the lack of Pakistani researchers in a field is an impediment towards implementing this idea. A good researcher with known contributions in his/her field should have no problem receiving recommendations from his/her peers whether they are Pakistani or not. Inability to do so clearly indicates low impact of the research regardless of the number of publications or citations.

      • Affan says:

        Perhaps I wasnt clear enough. By not enough samples I was referring to the problem of using peer opinion to classify importance of or ranking of conference and journals in different fields. The key point in this suggestion is that I want to use the researchers *in* Pakistan, in whatever field they are to start suggesting “fields”, for which we then vote on the relative strength of each conference/journal. This needs to be done as I feel that measuring progress is important, but it needs to reflect the nature of Pakistan’s “state-of-research”.

        Rankings and concern about state of research is applied to areas/countries where the research community is not established.
        I believe, we need to be careful using the exact same metrics as those used for international rankings (ISI for one). It might be that we need to tweak ratings/rankings such that we are able to measure any progress happening in Pakistan, progress small enough that it is not lost by existing metrics. What that could do is lead to an early pooh-pah of a policy/strategy that is working, slowly-but-steadly.
        Once a critical mass is reached, tweaking the metrics would be appropriate or maybe even using international measure until finally such standards would (hopefully) no longer be needed. Thus, no one in US is ever worried about such tiers or rankings, precisely because the research community is strong and broad enough in all disciplines that peer opinion and standing is all that matters (of course, these are achieved through publishing in reputable places, but that is of secondary concern).

  5. This is in response to the posting by Dr. Atta-ur-Rahman, dated 19-09-2009, wherein he has replied to my comments.

    With regret, I note that Dr. Atta chose to concentrate on the rather peripheral issue of an alleged increase in the number of citations of research papers published from Pakistan. In my posting, I had stressed that instead of this matter, there are two centrally important issues which need a response. Since these were ignored, I am forced to repeat my questions:

    1. Where is the evidence that Pakistan’s universities have improved in quality as TEACHING INSTITUTIONS since 2002? An increase in the number of research publications – especially when that increase comes from pay-per-paper incentives – is a very weak argument.

    2. How does the former HEC chairman explain the huge amounts spent on the now-abandoned 9-university mega-project that, even before it began, was repeatedly pointed out as a non-starter? What of the tons of still unused, and extremely costly, scientific equipment that were purchased in spite of the HEC chairman being warned again and again in public that this is exactly what would happen? Why was the Quranic science scheme started and then shelved?

    Dr. Atta cites in his support Nature, USAID, British Council, and the World Bank as “very complimentary of many excellent programs introduced”. But this is skating on thin ice. Organizations which are tasked to spend money in Pakistan, or give loans, are sometimes willing to cut corners to make their missions in Pakistan look worthwhile to their home government. The HEC was known to be a very warm host to large numbers of foreigners making quick visits to Pakistan. But none of the organizations mentioned by Dr. Atta appear to have actually researched the situation in Pakistani academia.

    The World Bank, for example, could have undertaken its own study. It could, for instance, have looked for evidence of improvement in university teaching quality (rather than a mere increase in enrollment). To do this scientifically it would have needed to work out the parameters that define teaching quality and then gathered the relevant data. This might have involved establishing some reasonable metrics for gauging the quality of the faculty and student body, assessing the state of library and laboratory facilities, the content of university courses, the standard of examination papers, the presence (or lack thereof) of academic colloquia and seminars on campuses, the suitability of those appointed as vice-chancellors, the number of days in a year that the universities actually function, satisfaction of employers with university graduates, etc.

    But there is no sign that the World Bank bothered to do this groundwork, much less do the necessary fieldwork. At least, having searched available databases, I could find no hint of such a study. If there is indeed one, I would be grateful to know where I might find the report.

    Finally – albeit reluctantly since this takes me away from my main argument – I return to the issue of self-citations.

    This afternoon (18-09-2009) I did a Thomson-Reuters database search on two Pakistani scientists in a Pakistani university, one in mathematics and the other in physics. They have received high national awards for publishing the largest numbers of papers in their respective disciplines. The database gives the following numbers:

    a) Dr.X has 288 mathematics papers. He is listed as being cited 2517 times, but only 445 citations remain after self-cites are removed.

    b) Dr. Y has 148 physics papers. He is listed as being cited 645 times, but only 205 citations remain after self-cites are removed.

    This is surely a sorry state of affairs. But is this blatant padding of citation figures indicative of a broader trend? I did not do a wider check but this is exactly the job of the authors of the Nature article. After all, the onus is upon them to prove their startling claim that citation rates from Pakistan have improved dramatically.

    ———–
    Pervez Hoodbhoy, Physics Department, Quaid-e-Azam University.

    • Yaser Sheikh says:

      I agree with Dr. Hoodbhoy that the Nature article was a lazy and unsurprising piece of scholarship, but respectfully disagree with his broader criticisms of the HEC.

      The nine-university mega project was a high-risk, high-return project. That’s what happens with risky ventures: they sometimes fail. I, personally, don’t want timid incrementalists at the helm of education reform; I want bold men and women of action. If some of their ventures fail in their sincere efforts at improving the state of science and education so be it. There should certainly be checks and balances to ensure that no corruption ensued, but the task that Dr. Atta-ur-Rahman and his team has attempted to undertake is stop-a-bull-in-its-tracks challenging, and they have attacked the challenge with gusto. If among their many spectacular successes, a few of their initiatives spectacularly fail, I feel it is a reasonable cost of doing high-risk, high-return business — so long as there was no financial corruption, of which I have heard no credible evidence. On the other hand, I have heard of many stories of Dr. Atta-ur-Rahman resisting pressures from influential members of the parliament and the government.

      The wholesale dismissal of the slew of international experts that Dr. Atta-ur-Rahman cited, seems to indicate that a mind has been made up. The endorsement of external bodies isn’t a slam-dunk metric, but like all metrics it is a useful part of the story — it is an insufficient but informative statistic. There are strong political considerations in their evaluations, but the WB, BC, and USAID regularly criticize many developing countries and have indeed criticized Pakistan in the past. That their reports read like paeans to the HEC is certainly to its credit.

      Finally, on the issue of self-citations, citing two individuals (who indeed may be guilty of working the system) is not a solid case for a systemic problem, but this is an issue that is easily checked and corrected for (as Dr. Hoodbhoy, to his credit, has done). Self-citations should be removed in any reporting of quality of publication measures.

    • Ayub Malik says:

      I completely agree with Dr pervez Hoodbhoy that number of citations of research papers is no criteria. He use to challenge the very ideas since the last seven years on which this Higher education programme was based in Pakistani society.

      • Saqlan Naqvi says:

        Dr. Hoodbhoy has served as a very learned critic of Hec policies, and I am sure that HEC was and is under a constant check in his shape. However, success/failure of a specific venture should not be sufficient to draw a conclusion on the very existence of HEC itself or on the overall policies.

        Contribution of Dr. Ata ur Rahman is matchless in the last 30 years history to which I am a witness as a higher education student, a researcher and as a university faculty member. There remain serious question about quality, but these question would have no merit if there was no quantity to evaluate.

        Whatever criticism, it should finish with clear remarks for future course of action.

  6. omar javed says:

    I think instead of adopting a single set of metrics for measuring all higher education institutions there is a need for a bottom up approach in which universities are categorized into subsets, and relevant performance criteria are adopted for each subset. In the U.S. the Carnegie Founding “provides different lenses through which to view U.S. colleges and universities, offering researchers greater flexibility in meeting their analytic needs. They are organized around three fundamental questions: what is taught (Undergraduate and Graduate Instructional Program classifications), who are the students (Enrollment Profile and Undergraduate Profile), and what is the setting (Size & Setting)” (http://www.carnegiefoundation.org/classifications/)

    In my opinion different criteria needed to be adopted for technical vs. liberal art colleges, graduate vs. undergraduate studies, etc. Performance evaluation in such a way will better aid understanding of where HEC’s efforts have succeeded and which areas need further attention.

  7. Anwar says:

    I will not consider endorsements by USAID, BC, and WB as confirmation of quality and quantity of the research in HE because these are pseudo-political organizations and have a track record of reversing their opinions…

    I am trying to find out what disciplines have seen a boost in research and publications – analytical, applied, or experimental? Likelihood is that of analytical folks who managed to churn out a large number of papers through several variations on the same theme – any groundbreaking discovery? No!

    Perhaps this push for publications lead to plagiarism in some quarters (APS also reported on this issue…- Punjab University – two years ago(?))

    Another question is that of fruits of research – Is there a viable public/industrial/manufacturing sector in PK that can absorb the results of high quality research or recommendations of scientists?

    From the comments, I gather research has become a panacea for the country – it is not. Intellectual pursuits, quest for knowledge and development is deeply rooted in socio-economics, civics, and culture and therefore cannot be sprouted overnight through implantation. It is a slow process.

    Most of the Pakistani professional recognized in their field managed to get there not because of UGC, HE, or the schools and universities they attended in Pakistan but in spite of them. This model will continue as long as there are a few eccentric rebels not locked up in the box and will manage to leak out of the mess to make name for themselves and for the country.

    Finally, technology is always subservient to the political will of a country. And I do not think there is any political will for excellence in Pakistan.

    We can continue to argue – The bottom line is that money and resources could have been well spent. Hindsight is 20/20 – so they say!

  8. mazHur says:

    HE has less to do with under-graduate schooling because those
    who fail to do in their under-graduate or graduate ranks have least tendency to opt for HE. And even if they somehow succeed to
    pursue HE they turn out to be nuts. This is quite evident from
    the reports of fake Ph/D degrees and plagiarism known to all.

    Frankly speaking, Ph.D’s have been of little service to the national
    Industry. A majority of them seems to have opted for Ph/D as it could
    do no better with their sole aim to grasp the teaching profession
    or some higher office in academic area. Unfortunately, our Ph.D’s
    have been least ‘productive’ in the scientific innovative field;
    infact they have turned out nothing ‘practical’ so far through their so-called research other than getting their articles published
    and increasing their quantity for their own vested interests. I dare
    say that their articles have hardly been of any practical benefit
    to the nation or the world at large.

    I agree with Pervez Hoodbhoy that quantity is not the right criteria for judging the progress or efficiency of the HE-it is the quality that
    matters. This reminds me of a fable by Aesop wherein a bitch bragged
    about her 12 puppies and mockingly asked the lioness, How many cubs
    do you have?? The lioness replied. One. ‘Just one!’ the bitch
    responded sarcastically. ”Yes, just one but a lion!’ , replied the
    lioness. I think the HEC should consider this analogy before
    bragging about the quantity of papers turned out.

    Pervez Hoodbhoy is also justified in asking about where
    the huge funds have been used. In my personal view there has been least
    done to improve HE during the period between 2002 till this date

    and most of which the proponents of HE boast about is
    politically ploy to justify their failures. Mr Hoodbhoy has been
    writing about this much earlier in his merited articles but alas
    the people at the helm of HEC affairs wouldn’t listen to him.
    At present I think there is hardly any person other than Hoodbhoy
    who is more realistic and sincere in his views and
    outbursts in so far as HE is
    concerned. Unless a sincere and competent person is placed at the
    helm head of the HEC it would be futile to discuss the ‘metrics’
    of HE as such….

  9. Prof. S. Shafiqur Rehman says:

    To begin with let us remind ourselves a popular maxim that “ends do not justify the means” and recall the initial point of this whole reform process i.e, establishment of “Task Force on HE” followed by “Steering Committee” both created by Gen. Musharraf. If we analyse their composition, mandate, methodolgy and the outcome of these bodies one important point stands out that it was an elitest and Top-Bottom approach, which in policy matters especially when pertaining to the education sector hardly ever succeeds because of exclusion of the common stake-holder from the process and unrealistic goals. Obviously just like other reform initiatives of Musharraf era such as Local Government Ordinance, changes in Blasphamy act, foreign policy reversals, counter insurgency in Baluchistan and FATA etc,HE Reforms process is also smong the highly controversial matters.

    Now a comparision of poitive and negative public perceptions about Prof. Atta-ur Rahman led HEC and their policy interventions:

    Enhanced annual grants for universities: Against the public statement of Gen Musharraf that after 2005 the entire recurring budget budget of each public sector university wiil be footed by HEC, grants were selectively increased and developmental initiatives were linked with submission of proposals through PC-1. Consequently peripheral, relatively newer and weaker universities were marginalized. Huge Funds were allocated also to semi-public sector universities at the cost of development in pure public sector institutions.

    Introduction of highly incentivised Tenure Track System, Foreign Faculty program and institutions of National Professorship besides the common low-paid BPS faculty created a hisharmony and “apatheid” among the university teachers and led to discontent, use of unfair means and lack of ownership among the faculty.

    Introduction of impact factor and HEC recognized journals also created a distasteful competition, plagiarism, unwanted multiple authorship of research articles besides again marginalizing local languages and literature and giving undue benefit to social sciences.

    Little attention has been paid to raising teaching standars, class-room environment and improving evaluation techniques/methodologies and hence on effect so far on the quality of graduating students.

    Unrealistic standards have been employed for ranking of universities creating uneasiness among different universities because they employ divergent yardsticks and end up comparing apples with oranges.

    HEC has not been able to implement its selection and appointment criteria for Vice Chancellors and the Chancellors. Still these appointments are made on the basis of political affiliation, clout and personal liking or otherwise of the Chancellor. Provincial Governors are still the chancellors and exercise their discretion in appointing V.Cs and Deans, often retired beaurucratcs or military personnel beyond their superannuation.

    No matter what the article spublished in Nature or comments of academic adversaries like Prof. Att-ur Rehman or Prof. Hoodbhoy say the success or falure or HE reforms must only be judged by opinion of the common stake-holders (students, teachers and administration)of the public sector universities

  10. Dr.Fakhruddin Habiby says:

    To reform higher education (HE) system in Pakistan is a huge challenge and gigantic task. It require dedicated efforts with great foresights as well as understanding of ground realities with shrewd administrative skills.

    The ‘academic anxiety’ which is observed in above posts is indeed a health sign that reflects our will to improve. Hence, the criticisms from various individuals should be taken positively and constructively.

    The HE reforms launched in year 2002 was a good beginning. However, in my views, the model on which it was based upon was not our own. It was a western model in which ‘ research and publication’ is the main yardstick to measure the success. The number of publications or citation is not a true representation of success of the HE program in our environment. It may be added that apart from ‘self-citation’ another ‘tool’ that is used to push the citation number higher is formation of ‘ citation-coalition’ within research groups.

    To judge success of HE reform in Pakistan, following metrics can be considered as fair representation:

    • Number of industrials projects which were initiated with University-Industry partnerships and their effectiveness based on Industry feedback.

    • Performance of students in international competitive exams.

    • Performance of HE graduates in Industry and other fields based on evaluation of their employers.

    A Way-Forward:

    • A suggested way-forward is to build upon the infrastructure laid down by HE commission in 2002 with modifications to suit our local requirements. This step will not vigorously shake the present setup for which huge amount of public funds was utilized. Simultaneously, it will be worth while to develop our own indigenous model of HE by bringing together scholars, political forces, policy makers, academics, industrialists and selected student bodies. A gradual shift then can be made to the indigenous model for continuous growth and success.

  11. mariah says:

    i have a Question here. are you support undergraduate students to present their papers in international conference? if not then what are the reasons.

  12. Zeeshan KHAN says:

    One important point missed most of the time is that the target should be to develope a research environment in the country. Surely it takes time and efforts. In my opinion, number of patents should be one of the critera to judge the research and development advancements as the number of publications and citations is a loose metric.

  13. Nauman Sheikh says:

    Focusing on the worng Problem…

    Are we measuring the Performance of Universities or the performance of a Universities regulator (so to speak)? The metric so far being discussed appear to be the metric for the performance of Universities and since most ideas are being influenced from our western experience, where universities are evaluated, we are trying to fit HEC into that.

    A Higher Education Commission is a unique 3rd world institution that has no equivalent in the west. This is similiar to NADRA for example where there is no global precendence for what they are doing and therefore NEW metric are needed to evaluate NADRA’s performance. Another way of looking at HEC performance is that of a Regulator’s performance like PTA (telecom) or SBP (banking) where how many customers enrolled or how much profitability acheieved are not the metric for PTA/SBP but for the individual organizations under their regulation.

    The question to ask is who is the stakeholder in HEC and then determine what is important to the stakeholders and that should be the metric. To me, the stakeholders are the general public (primarily the students) and the industry which employs the students. For students the metric can be derived from
    1. Cost of education
    2. Merit based opportunity
    3. Access to higher education
    4. Quality of Education (not in terms of published papers but in terms of value-addition to their lives – like jobs)

    On the other hand the metric for the Industry can bederived from
    1. Cost of available skill
    2. Relevance of skill
    3. Quality of skill

    If this is how the stakeholders view the performance of the HEC, then this is how HEC’s performance should be evaluated.

  14. Abdullah Sadiq says:

    As some one marginally associated with the current education reforms efforts in Pakistan and some of its monitoring processes I feel that a clearer vision of the role of higher education consistent with the prevailing ground realities of Education in Pakistan would have made these reforms much more effective.These realities pertain to the accessibility and quality of education at all levels and its relevance to the social needs of the society. Because of ghost schools, teachers absenteeism, the plague of tuition centers and emphasis on rote memorization and good grade rather than understanding made it difficult to get reasonably well prepared students into higher education. A situation that still prevails. Statistics of
    student’s induction into institutions such as AKU, LUMS and GIKI, where only entrance tests are the sole criteria for admission would amply demonstrate this state of affairs.

    Given this poor intake, ill-prepared and poorly motivated teachers, mass inbreeding and poor governance our universities. with few rare exceptions, were little more than poor replicas of community colleges. The result was, and perhaps still is, that hardly 5-10 % of applicants with through-out first class academic background in sciences and engineering from these institutions would qualify a GRE (general)-type test and interview for MS admission into institutions such as PIEAS (former CNS.). About 20-25 % of those admitted would fail in the zero semester based on 1st/2nd year undergraduate physics and math. Only the cream of those surviving zero semester would and did perform well in their higher studies abroad.

    The fact that the 100 or so annual scholarships for studies abroad requiring a decent score in GRE in a population of 150 million could not be utilized during the late eighties and nineties is another example of this sad state of affairs. Given that only the most talented of our students get a chance to pursue careers in science and engineering these examples were and still are a poor reflection on the state of education in Pakistan.

    A more systematic analysis of such factors were needed at the outset of the new education reform effort, which to my knowledge was not carried out. This might have led to the realization that while strengthening the research effort in some of the universities the most urgent need was to concentrate on producing quality teachers for the lower tears of education. In fact based on a nationwide survey of university physics departments in early 2000 a proposal to this effect was submitted the Pakistan Council of Science and Technology and the then newly established HEC made some efforts to implement some of these recommendations.

    Let me conclude by stating that by helping establish several new universities in different parts of the country HEC has addressed the accessibility problem. Heavy investment in connectivity, including teleconferencing, technical literature, laboratory equipment. Oversees scholarships and foreign faculty hiring has in principle paved the way for quality related issues. However this has been done across the board in stead of concentrating these resources where sufficient absorption capacity existed. The result, in my opinion, is insufficient gains in research quality at the expense of producing good quality teachers. A recent e-mail to an HEC official pasted below highlights some of these concerns.

    Needless to say this debate could greatly benefit from more detailed dispassionate analysis of various aspects of the issues raised in this discussion.

    Regards,
    Abdullah
    ————————————————————————————————
    e-mail of June 17, 2009 to an HEC Official on ‘Quality of High Education’

    I had been thinking to write to you after interviewing some people for the AU faculty over the last year. I finally decided to do so after reading the following item in the Dawn:

    http://www.dawn.com/wps/wcm/connect/dawn-content-library/dawn/the-newspaper/letters-to-the-editor/shortlisting-for-vcs-appointment-769

    I greatly appreciate the Quality Assurance efforts of HEC with which I have been associated in the past. However besides setting minimum criterion for faculty induction , student enrollment, course prescription and research evaluation and the governance issues highlighted in the above mentioned item, a lot more needs to be done to arrest the the accelerating downward trend of the quality of higher education in the country. Here I would simply site three examples to illustrate my point.

    1. About a year ago I gave a simple physics test for applicants for the post of lecturer and assistant professor to test their concepts of the undergraduate course they were expected to teach. The highest score among the 11 candidates, 10 MPhil and one PhD from local universities including the top ranking one, was 40%. The candidate with PhD from the top ranking university was not even shortlisted for the interview.

    2. More recently while interviewing a PhD under the supervision of a foreign professor at a newly established Center of Mathematics supported by HEC, I got curious when he couldn’t explain some very basic terms related to his PhD topic. He didn’t have transcripts of his PhD courses but looking at the transcripts of his MSc Mathematics courses, I noticed that he had a Gold medal with A+ grade in several courses and 100% marks in combinatorics. Even so he could not define the term ‘combinatorics’.

    3. Yesterday while interviewing some other Math PhD students at a top ranking local university we found out that one of them, with no physics courses beyond HSSC, was working on “Open Loop Control in Non-linear Dynamical Systems’ and another one with no physics course beyond SSC general science, was working on ‘ Effect of Heat Transfer on Peristaltic Flow with Variable Viscosity’. Needless to say that none of them have the slightest clue of the underlying physics of their PhD topics. When asked if they were advised to sit in some relevant courses in the neighboring physics department the answer was an emphatic no.

    I shall be happy to share with you more details of these cases. HEC might consider initiating a process of getting regular feedback from the end users of our University graduates.

    With best regards,
    Abdullah

    • Fida Khattak says:

      The standard of educations is, undoubtedly, getting abysmal. I also had similar problems with some candidates with local PhD degree, and in some cases even with PhD from a univeristy in far eastern countries. Of course HEC has introduced courses at the PhD level (may be too many with 8 courses at MPhil level and 6 courses at PhD level)in order to provide breadth and depth but courses are not delivered in the same spirit. Neither the depth nor breadth is apparent from the performance of our PhD graduates. Perhaps faculty has to be blamed for this.

      Another point that I would like to share is that in most cases the PhD students are not (?) genuinly interested in solving the problem at hand. They are rather interested in getting a degree. I think it is economic driven. Majority of our student work hard but that is physical not mental. They are happy to work late night working on machines producing raw data without any understanding of the underlying priciple. When it comes to thinking, they show their laziness.

    • Sheroz Khan says:

      Dear Dr Abdullah Sb,

      You have raised very usefull points including the ground realities, the ghost shcools, the ongoing business oriented education at all levels ranging from school to university levels; the poor monitoring hierarchy at almost all levels starting from primary to secondary, and the lack of a system of education satisfying the education needs of our scoiety, and providing for the society with graduates equipped with relevant expertise and knowledge.

      The above mentioned points are all such entities required to be networked propelry in such a way that none of the processes put in place for some reason, should not end up landing at a deadlock. The processes for quality monitoring or evaluation and assessment reasons of some parameter, must not have open ends, rather they must have their destination ends close smoothly into triggering other processes on the way–either to become some stimulating inputs back into the system or must show up as productive outputs in the form of graduates, research findings and consultancy services; satisfyign the needs and requirements of education stake holders.

      However, the HEC is an adminitrative entity where some good steps have been taken and are underway at the moment, which I beleive have been producing results that are relatively better compared to the old UGC (VCs club) style of administration. I am not not very much aware of the ongoing processes put in place by the HEC for achieving some outcomes or evalualation and assessment objectives. However, some suggestions are humbly forwarded:

        1. Outcome Based Eduction at the UG/PG level
        2. Rejuvanating research oriented MSc and PhD level education,
        3. Exhibitions at the National levels to encourage competition at some (if not all) levels
        4. A meaningful consultancy element aimed at encouraging industry and social sector invovlement
        5. Proper newtorking among the universities for developing a viable education and academic culture
        6. International and National examiners for HEC affiliated Universities, aimed at continuously evolving UG/PG degree programs (similar to GIKI) examiners systems
        7. Idenitfying educations stake holders such sponsors departments, industry players/advisors, students, alumni
  15. Fida Khattak says:

    I personally believe that some credit must be given to the HEC. Resources could have been used optimally, surely, but faculty in the Pakistani universities has started relating their success and academic growth to their research productivity in terms of publishing articles. That can be considered as a sign of improvement. I do agree that many of the articles published from the research work carried out indigenously are either substandard or lack originality. But people were seemed to be scared of writing their observations and thoughts before the HEC drive and now a significant number of the faculty has started putting their ideas on a piece of paper. It is by no means satisfying but at least our faculty has been “pushed” to a start; does not matter how slow it is, it is in the positive direction. We are out of stagnation.

    There is for sure a lack of genuine interest on part of some of the faculty in undertaking research but some are trying their level best. Their zeal and their efforts may not be judged by the impact factor of the journal in which their articles get published. I even think that some credit shall be reserved, does not matter how small, for those who succeed in publishing their articles and ideas in the local journals with zero impact factor. One should be extremely careful in relying on the impact factor of a journal while judging the quality of a research article. Don’t be surprised if the editor of a high profile international journal contacts you just before publishing your article suggesting to include a few articles, published in the earlier issues of his journal, in the bibliography of your article in order to help enhance the impact factor of the journal.

    While we should learn from our past mistakes, there is no point in either eulogizing or criticizing personalities especially those who are no more at the helm of the affairs. Doing so would not help enhance, even a tiny bit, the quality of education in our country. The academia must work towards an environment on university campuses that is conducive for the creation and dissemination of knowledge. Every body has a right to have his own ideas but one shall not impose his/her ideas on others. Let us work towards a tolerant society and be receptive to positive criticism. Let us ask ourselves how much have we actually contributed to the enhancement of our education quality.

  16. Dr. Anjum Iqbal says:

    I would like to suggest a metric “Growth of National Scientific Culture/Apptitude”. It can be estimated through a survey addressing “what is the improvement in approach and behaviour towards specific scientific and generic national issues?”

  17. Shaukat Hameed Khan says:

    First let us remember the stagnation which had overtaken the Universities in the decades before 2003 and pay the devil his due. The HEC under Dr Atta has certainly been able to reverse the financial starvation faced by Pakistani Universities in the last 35 years and Dr Atta deserves praise for getting the necessary funds and support from state institutions. Funding levels have been raised several fold, more papers published, and many more young men sent abroad for PhD, after a lapse of many years.

    This is the good part.

    The flip side is that HEC went on a bit of a spending binge and our Universities are still not the centres of intellectual dynamism one may have expected from the relative massive increase in funding, especially in the social sciences.

    The HEC needs to pause now, take a serious second look at all its programmes, and improve both its own internal efficiencies and those of its client universities. It has fortunately rolled back (actually forced to do so by circumstances not by choice) its highly expensive programme for establishing ‘foreign’ university campuses in Pakistan, which had clouded the good work done by it. HEC needs now to consolidate and stop further expansion in the number of universities, so that the desired intellectual ambience can take root on our campuses. I am afraid this has not really happened, and quality has suffered.

    A comment, if I may, about the thousands sent abroad for PhD. There is considerable worry in many Universities about the breadth of knowledge and competence of these young men. Could this have to do with the fact that the HEC is now pretty much scraping close to the bottom of the barrel to find suitable candidates? Could the foreign hosts be taking them in because they have financial problems of their own and welcome foreign funds to subsidise their own nationals.

    Perhaps, we should fund our scholars ONLY for Master’s programmes (18- 24 months), and let the host University fund the candidate after this period if he/she is up to the mark. Let us remember that research and higher studies is in distress in these host countries in terms of funds and the number of their nationals willing to enter the arena; they badly need young foreign minds to fill the gap.

    The HEC now needs to change into a different operational mode. The sustainability of present funding levels is doubtful in the long run. No new Universities, please, in spite of what the politicians declare on their visits.

    As for research, funding must now be built around researchers and not for setting up a so called infrastructure. In an earlier incarnation, I would intervene considerably to find the ‘poles’ around which the academic tent could be erected. Not very successfully, I admit!

    A comment about the insistence on a PhD degree for faculty evaluation and promotion. The number of papers is increasing, but the ability to teach and transmit and to nurture the thinking mind is suffering. So the so called evaluation rules on this subject need a major review.

    The HEC is setting up Universities the same way as degree colleges were set up in yester years. It must move away from the ‘college’ syndrome and allow the Universities to become bigger in size (student population) as well as making them ‘universal’ in pursuit of learning. This requires HEC to encourage multi discipline campuses where the social sciences civilize the scientists and engineers and medics. Can we have research in our medical universities without good biologists, chemists, biophysicists and the dreaded biotechnologists? Can we expect our scientists to design their own test equipment without good mechanical and electronic workshops? And can the engineers expect to do research without the exciting presence of physicists, chemists and biologists in their midst on the Campus? I am afraid not!

    AND of course all these scientists, engineers, IT chaps and computer wizards have the right to be civilized by rubbing shoulders with social scientists, literature buffs, , economists, historians anthropologists etc.

    Finally, as a member of the Steering Committee on Higher Education (SCHE) set up by Pres. Musharraf in 2002, which led to the establishment of the HEC, I would have been happier, (and the article in Nature would have carried more weight), if its authors had been not been members of the SCHE team, whether directly as Members of SCHE or indirectly as proof readers and language correctors)!

  18. MZM says:

    Citations are a horrible way of judging research. I fully agree with Khurram that if you are working on a really hard problem then there are not many citations despite good research. A simple descriptor in Computer Vision will generate more citations(everybody will apply all permutations of classifiers and data sets to find the good cases) than a ground breaking machine learning application to program analysis.
    There used to be a time when research was measured for its correctness (fights over Fourier transform, or quantum mechanics are a good reminder), applicability and impact on our understanding; and not for its “publish-ability” or “cite-ability”.

  19. I absolutely love your blog and find most of your post’s to be precisely what
    I’m looking for. Do you offer guest writers to write content for
    you personally? I wouldn’t mind creating a post or elaborating
    on a few of the subjects you write with regards
    to here. Again, awesome web log!

  20. google says:

    If you are going for most excellent contents like I do, only go to
    see this site every day as it presents feature
    contents, thanks

    My web site – google

  21. Santiago says:

    Designer spectacles frames, like custom spectacles, are very common among us of any age
    bracket. With Acuvue Oasys for Astigmatism, this problem mayn’t
    exist on account of technology named Hydro obvious.

  22. Holistic medicine concerns itself to the belief of the connection between the
    spirit and emotions and mind.

  23. Mariano says:

    A properly balanced holistic health regimen, which takes into consideration all aspects of
    human health and includes noninvasive and nonpharmaceutical healing methods, can often completely
    eradicate even acute health conditions safely.

Discuss

  • STEP aspires to be the central place for discussion on improving the state of Science, Technology, and Education in Pakistan. Read More
  • To learn how you can contribute, click here
  • Never miss a new article! Choose your favorite method to stay up to date with STEP
  •