Journal of Materials Research and Technology Journal of Materials Research and Technology
J Mater Res Technol 2017;6:304-11 DOI: 10.1016/j.jmrt.2017.09.004
Original Article
The use of the h-index to evaluate and rank academic departments
Marc André Meyers, , Haocheng Quan
Department of Mechanical and Aerospace Engineering, Materials Science and Engineering Program, University of California, San Diego, United States
Received 21 July 2017, Accepted 16 September 2017
Abstract

A method is proposed by which the h-index of individual researchers is extended to evaluate the performance of engineering departments. For a specific department, the h-index of each faculty is plotted against the number of years since the first publication. The plot is linearized and the slope is determined, which we term Departmental Productivity Index. This index represents the collective productivity of the department members. The statistical analysis is applied to two years: 2008 and 2017. This slope is correlated with the ranking of the department from USN&WR. Mechanical Engineering and Materials Science and Engineering Departments ranked over a broad range (top, second, and third tier) and in three regions within the US (East, Central, West) are used. The dp-index is not as representative an indicator as more in-depth analyses involving many other aspects, such as teaching, resources, and size, but it can serve as a robust guideline for departmental evaluation. For 2008, the dp-indices of the ME departments varied from 0.70 for the highest ranked to 0.23 for the lowest one. For 2017, the dp-indices show a systematic increase; the highest being 0.99 and lowest increasing to 0.5. For MSE departments, the same trend is observed: in 2008, they vary from 1.36 to 0.51, while in 2017 they range from 1.89 to 0.61. There is a systematic difference between Materials Science and Engineering and Mechanical Engineering Departments, the latter having dp-indices that are in average 30% lower than the former ones. This might be a reflection of the greater resources available nationally for materials research and of the service role that many ME departments have in Engineering Schools. The increase in dp-indices in the nine-year span (2008–2017) results from the rise in individual h-index for researchers, which reflects greater emphasis on research, increased collaborations, and an evolving research landscape. An additional observation that is revealed by this statistical analysis is that the difference between first and third tier departments decreased from 2008 to 2017, a reflection of the ‘democratization’ of research through a more equitable distribution of resources and talent. This method is also suggested to be an effective quantitative measure of departmental and faculty member performance.

Keywords
h Index, Departmental productivity
1Introduction

The h-index, proposed by Hirsch [1], has had a deep impact on the quantification of the productivity of researchers. It is defined as [2]: A scientist has index h if h of his/her Np papers have at least h citations each, and the other (Np-h) papers have no more than h citations each. Since its inception in 2005, it has stimulated a lively debate over its relevance. It clearly represents the accumulated impact of a researcher's contributions. In spite of the limitations of this single number as a descriptor of a researcher's career, it is gaining wide acceptance and even exaggerated importance. Hirsch [2] tracked the careers of researchers and found a close correlation between the early and late career h-indices; he found that the index increases linearly with the number of years of productive activity and thus is a good predictor of the prominence of a career. Thus, the h-index has been normalized by the number of career years; the latter is called m-index and enables cross comparison between scholars at different stages in their careers. The h-index has been extended to entire research institutions (e.g., Molinari and Molinari [3]) and has been shown to be an effective means of ranking them. It has also been modified in a number of ways [4]: g-index [5], e-index [6], Contemporary h-index [7], Age-weighed citation rate [8], Multi-authored h-index, Individual h-index [9], and others. The objective of this paper is to apply the h-index to academic departments to quantify their research performance.

2Method/procedure

In order to render the task manageable, we chose nine institutions for each discipline. These chosen departments were divided according to the geographic locations (three each from East, Center, and West). For each region three departments were chosen from the ranking in US News and World Report of that year: Top rank, 2nd Tier, and 3rd Tier (http://graduate-school.phds.org/). Secondly, we choose the faculty whose title is either Professor, Associate Professor, or Assistant Professor (including faculty with joint appointments), excluding Emeritus Professors and Adjunct faculty in order to obtain consistent data.

Thirdly, we use the ISI Web of Knowledge database (http://apps.isiknowledge.com), specifically using “Author Finder.” Author Finder is a quick four-step process that helps to find papers published by an author. Once the particular author is identified, ISI generates a “Citation Report” providing the number of published items in each year, citations in each year, and h-index. The data were filtered to remove duplication of authors with the same last name and initials.

Lastly, the h factors obtained from the “Citation Report” are plotted against the number of years since first published paper for each author. The resulting plot is linearized with the line passing though (0,0). The resulting slope is calculated and is termed the Departmental Productivity Index, or dp-index. The R2 factor, which represents the variability of results, is presented for all departments.

3Results

The observation by Hirsch [2] that the h-index grows linearly with the number of years since first publication, at a constant level of performance, is important and serves as a basis for the linearized plots used here, since it enables equivalence among researchers at different stages in their careers and therefore comparison between departments comprised of faculty with varying ages. One has to realize that the data collected here have a wide spread, since some faculty specialize in high-profile research, while others seek administrative careers, which decrease their research productivity and others are satisfied with a lower output. Every department also has faculty that specialize in teaching undergraduate classes. Nevertheless, the results showed in Figs. 1–4 show clear and undeniable trends.

Fig. 1.
(0.18MB).

H-index vs. years since first publication for members of mechanical engineering departments in 2008; slope equal to dp-index; (a) top ranking (USNWR# 1); (b) middle ranking (# 21); (c) low ranking (# 82).

Fig. 2.
(0.18MB).

H-index vs. years since first publication for members of mechanical engineering departments in 2017; slope equal to dp-index; (a) top ranking (USNWR# 1); (b) middle ranking (# 22); (c) low ranking (# 95).

Fig. 3.
(0.18MB).

H-index vs. years since first publication for members of materials science and engineering departments in 2008; slope equal to dp-index; (a) top ranking (USNWR# 3); (b) middle ranking (# 28); (c) low ranking (# 55).

Fig. 4.
(0.18MB).

H-index vs. years since first publication for members of materials science and engineering departments in 2017; slope equal to dp-index; (a) top ranking (USNWR# 5); (b) middle ranking (# 27); (c) low ranking (# 43).

It is clear that the departmental productivity index (dp-index), numerically equal to the slope of the linearized data, shows systematic differences that correlate with departmental ranking. We discuss Fig. 1 first. It shows, for illustrative purposes, a top ranking (# 1), a middle ranking (# 21) and a low ranking (# 82) Mechanical Engineering department in 2008. There is a corresponding decrease in the slope. The dp-index is indeed more significant than a simple ranking since it correlates directly with faculty research output. An h index equal to the number of years of research production is considered a reasonable value for a successful researcher. This translates, collectively, into a dp-index of 1. Values above this are characteristic of leading research departments and this is exactly what is revealed by Fig. 1. It can be seen that the slope decreases with the ranking of institution, as is to be expected. A linear increase in the h factor with the number of years in the career of an investigator is predicted (e.g., Fig. 1 from Hirsch [2]). One would expect, assuming that the productivity of all departmental members is the same, a straight line passing through zero. This was forced onto the plot. The slopes of these lines are shown in Fig. 1a–c and are equal to 0.70, 0.60, and 0.33, respectively. The evaluation of ME departments in 2017 (Fig. 2) shows the same trends: the dp-indices are 0.99, 0.81 and 0.62. It can be also seen, as expected, that there is a significant spread in the data, as given by the R value.

The same procedure is followed for MSE departments, and the dp-indices are shown, together with the individual points, in Fig. 3 (for 2008) and 4 (for 2017). For MSE departments in 2008, as shown in Fig. 3a–c, the dp-indices are 1.36, 0.85, and 0.68. In 2017, these numbers changed to (Fig. 4): 1.89, 1.14, and 0.65.

The more complete data, including nine departments are presented in Tables 1–4. Tables 1 and 2 refer to Mechanical Engineering departments evaluated in 2008 and 2017, respectively; Tables 3 and 4 to Materials Science and Engineering departments in the same period. Although there is no one-to-one correspondence with the USNWR ranking, the dp-index is a good measure of the departmental productivity. The 2008 and 2017 results reveal four important features, which are expressed graphically in Figs. 5 and 6, which plot the dp-index vs. USNWR ranking:

  • 1.

    The dp-indices drop rapidly with ranking in the top tier departments and then decrease more gradually. This is presumably due to the fact that the major national research activity is concentrated in these departments.

  • 2.

    The MSE departments as a whole have higher dp-indices than the ME departments. This is also borne by the intuitive perception in the community. Whereas ME departments tend to have large undergraduate enrollments and focus on the important task of educating students, MSE departments have proportionally smaller numbers of undergraduate students and are more research oriented. If normalized by undergraduate student enrollment, Federal funding in MSE is disproportionately high in comparison with ME by virtue of the research nature of the discipline.

  • 3.

    There is an increase in the dp-index from 2008 to 2017. This increase is connected to the growing importance of the individual h-index in the evaluation of individual faculty and on the changing landscape of research, favoring multi-author and multi-group papers through enhanced collaborations.

  • 4.

    There is a correlation between the USNWR ranking and the dp-index, the latter being a more complete descriptor of departmental performance than the simple numerical ranking. This is expressed graphically in Figs. 5 and 6. In 2008, the dp-indices dropped rather fast beyond the Top Tier departments for both ME and MSE departments. In 2017, the decrease is less marked, with Second and Third Tier departments showing enhanced dp-indices. This is a direct reflection of the increased emphasis being placed on research and on attracting research funds, a demand that was exclusive of the top research universities but is now widely applied.

Table 1.

Departmental Productivity Indices and R2 values for productivity plots of ME departments of representative institutions in 2008.

Institution  Geographical area  USNR ME department rank  Departmental Productivity Index  R2 
Stanford University  0.69636  0.7504 
Georgia Institute of Technology  0.4418  0.68018 
University of Texas—Austin  12  0.37619  0.45842 
Northwestern University  12  0.62085  0.59865 
University of California—San Diego  21  0.60208  0.84145 
North Carolina State University  44  0.22597  0.57232 
University of Illinois—Chicago  54  0.52212  0.86606 
Mississippi State University (Bagley)  82  0.32501  0.61879 
San Diego State University  Not found  0.27783  0.82486 
Table 2.

Departmental Productivity Indices and R2 values for productivity plots of ME departments of representative institutions in 2017.

Institution  Geographical area  USNWR ME department rank  Departmental Productivity Index  R2 
Stanford University  0.99004  0.71148 
Georgia Institute of Technology  0.7486  0.7055 
University of Texas—Austin  11  0.78417  0.67599 
Northwestern University  14  1.03977  0.73849 
University of California—San Diego  22  0.8086  0.80608 
North Carolina State University  42  0.58015  0.74011 
University of Illinois—Chicago  49  0.70614  0.84288 
Mississippi State University (Bagley)  95  0.62042  0.64077 
San Diego State University  133  0.50389  0.75379 
Table 3.

Departmental Productivity Indices and R2 values for productivity plots of MSE departments of representative institutions in 2008.

Institution  Geographical area  USNWR MSE department rank  Departmental Productivity Index  R2 
Northwestern University  1.35811  0.7864 
University of California—Berkeley  1.14491  0.75067 
Georgia Institute of Technology  0.79831  0.71073 
Virginia Tech  27  0.69717  0.45165 
Iowa State University  28  0.85471  0.61754 
University of California—San Diego  34  0.69739  0.72537 
Rice University  44  0.50747  0.55549 
University of California—Irvine  50  0.62178  0.60642 
University of Pittsburgh  55  0.67803  0.75259 
Table 4.

Departmental Productivity Indices and R2 values for productivity plots of MSE departments of representative institutions in 2017.

Institution  Geographical area  USNWR MSE department rank  Departmental Productivity Index  R2 
Northwestern University  1.47862  0.77354 
University of California—Berkeley  1.88821  0.70237 
Georgia Institute of Technology  1.17332  0.59369 
Virginia Tech  22  0.60926  0.59038 
Iowa State University  27  0.77442  0.8087 
University of California—San Diego  27  1.13959  0.80194 
Rice University  35  1.83984  0.78655 
University of California—Irvine  39  1.09257  0.84825 
University of Pittsburgh  43  0.64972  0.77708 
Fig. 5.
(0.16MB).

Departmental Productivity Indices for selected (a) ME and (b) MSE departments as a function of 2008 USNWR ranking.

Fig. 6.
(0.16MB).

Departmental Productivity Indices for selected (a) ME and (b) MSE departments as a function of 2017 USNWR ranking.

The spread in the results, shown in Figs. 1–4 and quantitatively expressed by the R2 factor, is the result of the natural differences between individual faculty members, some of whom dedicate more time to teaching, while others embark of more focused administrative activities and yet others become involved in entrepreneurial and industrial activities. Nevertheless, the plots can serve as a guideline to evaluate the research performance of individual faculty members. Faculty above the dp-index line are highly productive and successful researchers, whereas those under the line have a lower research productivity. This is shown schematically in Fig. 7a. In a similar manner, the trajectory of departments can be evaluated, with rising departments being distinguished from declining departments by the shape of the curve, no longer linearized in this case (Fig. 7b and c). This requires a non-linear fit to the data, in addition to the linearization.

Fig. 7.
(0.2MB).

Possible uses of Departmental Productivity Indices: (a) evaluation of individual faculty; (b, c) prediction of productivity trend by establishing the non-linearity; a convex response represents a department in decline; a concave response is an ascending department.

4Conclusions

It is being increasingly recognized the h-index or some of its modifications are a useful method of quantifying the careers of researchers [10,11]. The rising significance of these quantitative parameters is in line with the rankings of universities, necessary in a global environment where the numbers are indeed staggering and where an individual judgment based on experience does no longer work. This task is therefore left to computers, which can assemble and analyze the data. The requirements of transparency and fairness in hiring and promotion also add importance to the h-index family.

The Departmental Productivity Index (dpi), proposed herein, which is determined as the slope of the linearized relationship between the h-index of individual faculty member and the number of years since their first paper, is a robust metric of the research activity of a department and of its ranking in the landscape. This method was applied to Mechanical Engineering and Materials Science and Engineering departments. The values are correlated with the USN&WR rankings. In general, there is good agreement. However, there are significant differences among different rankings, which can be readily seen by comparing the principal results: USN&WR, London Times University World Rankings, and Shanghai Ranking. A number of relevant conclusions can be derived from the dp-index proposed here, which relies exclusively on the h-index and is therefore skewed toward research. The principal conclusions are:

  • In 2008, the dp-index dropped rapidly as the USN&WR ranking increased. This represented the primacy of the Top Tier departments in conducting relevant recognized research. The difference decreased substantially in 2017, the result of wide-spread research activities and, perhaps, a more equitable distribution of research funds.

  • The dp-indices of MSE departments are systematically higher than those of ME departments, a reflection of the more research-intensive nature of the former.

  • The dp-indices rose systematically from 2008 to 2017 for both ME and MSE departments, a reflection of the increase in the individual h-indices. For 2008, the dp-indices of the ME departments investigated ranged from 0.23 to 0.70. For 2017, the range was 0.5–0.99. For MSE departments, the same trend is observed: 0.51–1.36 in 2008, and 0.61 to 1.89 in 2017.

  • The dp-index is proposed to be a useful tool for evaluating departmental performance in the global and US landscape and to establish trends by evaluating the shape of the curve: concave for declining departments and convex for ascending departments.

  • It can also be applied to evaluate the research performance of individual faculty members by establishing whether they are below or above the linearized curve.

Conflicts of interest

The authors declare no conflicts of interest.

Acknowledgements

We thank Y. P. Chen and Yufei Ding who collected the original information in 2008 and 2017, respectively; we appreciate the stimulation by our skeptical colleagues, who have through their objections, led us to a more in-depth study whose results are presented here.

References
[1]
J.E. Hirsch
PNAS, 102 (2005), pp. 16569-21657 http://dx.doi.org/10.1073/pnas.0507655102
[2]
J.E. Hirsch
PNAS, 104 (2007), pp. 19193-19198 http://dx.doi.org/10.1073/pnas.0707962104
[3]
A. Molinari,J.-F. Molinari
A new methodology for ranking scientific institutions
Scientometrics, 75 (2008), pp. 163-174
[4]
A.-W. Harzing
Publish or perish
(2017)
Hartwig.com
[5]
L. Egghe
Theory and practice of the g-index
Scientometrics, 69 (2006), pp. 131-152
[6]
C.-T. Zhang
The e-index, complementing the h-index for excess citations
PLoS ONE, 5 (2009), pp. 5429
[7]
Sidiropoulos A, Katsaros D, Manolopoulos Y. Generalized h-index for disclosing latent facts in citation networks, 2006. arx***iv:cs.DL/0607066.
[8]
B. Jin
The AR-index: complementing the h-index
ISSI Newslett, (2007),
[9]
P.D. Batista,M.G. Campiteli,O. Konouchi,A.S. Martinez
Is it possible to compare researchers with different scientific interests?
Scientometrics, 68 (2006), pp. 179-189
[10]
L. Bornmann,H.-D. Daniel
Does the h-index for ranking of scientists really work?
Scientometrics, 65 (2005), pp. 391-392
[11]
P. Ball
Index aims for fair ranking of scientists
Nature, 436 (2005), pp. 900 http://dx.doi.org/10.1038/436900a
Corresponding author. (Marc André Meyers mameyers@eng.ucsd.edu)
Copyright © 2017. Brazilian Metallurgical, Materials and Mining Association
J Mater Res Technol 2017;6:304-11 DOI: 10.1016/j.jmrt.2017.09.004