Report on COST TD1210 Workshop
Quantifying scienti c impact: networks, measures, insights?
February 12-13, 2015
Villa Hatt, Zurich, Switzerland
Prof. Dr. Dr. Frank Schweitzer
Chair of Systems Design
ETH Zurich
Switzerland
fschweitzer@ethz.ch
Scientometrics, and quantitative studies of science in general, attract significant attention both from researchers and stakeholders. Importantly, they already impact the evaluation of researchers and the funding of proposals. Thanks to
the availability of data from bibliographic databases, it has become particularly easy to construct and analyse collaboration and citation networks, develop novel measures of scienti c impact, or even apply predictive analytics techniques to predict the future career of young scientists. In this two-day workshop, we have addressed the questions which quantitative measures convey the most information, what insights we can gain from them, and how they influence the way we do science.
A particular goal of this workshop was to bring together stakeholders dealing with the quantitative evaluation of science on a daily basis and scientists investigating novel ways to quantitatively study the structure and dynamics of scientific progress. As such, the list of participants included prominent researchers in scientometrics from di fferent disciplines, representatives of bibliometric data providers as well as key players in the provision of academic evaluation and ranking services. Reflecting this focus, the program featured five scienti c sessions covering the areas of bibliometrics, social sciences, computer science and statistical analysis of science networks. The academic research presented in these sessions was completed by two stakeholder sessions, addressing ranking methods and data collection.
In a first session on bibliometrics, Dr. Rudiger Mutz from ETH Zurich introduced his view that statistical models are needed in order to provide an objective and reliable basis for the measurement and ranking of academic performance. Dr. Robin Haunschild from the Max Planck Institute for Solid State Research contributed a talk highlighting fi eld di fferences in citation behavior, and their impact on the evaluation of scientists. Dr. Dirk Tunger, head of the Bibliometry Team at FZ Julich, presented his recent work on the J-factor to measure interdisciplinary impact. In the following stakeholder session, Dr. Sonja Bergho (CHE), Martin Juno (QS) and Dr. Nees Jan van Eck (Centre for Science and Technology Studies, Leiden) provided interesting insights into the methodologies used by diff erent ranking providers, each coming with their speci c advantages
and disadvantages. The following session on social sciences featured a talk by Prof. Peter van den Besselaar (VU University Amsterdam), which addressed the use of bibliometrics beyond rankings. Prof. Flaminio Squazzoni from University Brescia introduced questions evolving around competition and misbehavior in peer review, which are being addressed in the COST action PEERE. Finally, Prof. Judit Bar-Ilan from Bar-Ilan University discussed the use of altmetrics as supplementary measures for scienti c impact. A fi nal session on Monday was devoted to computer science. Prof. Filippo Radicchi from Indiana University showed how the time-evolving network structure of citations can be used to measure impact. Prof. Martin Rosvall from Umea University presented his research on the use of second-order models and their impact on the robustness of journal
rankings. Concluding the program of the fi rst day, Dr. Ingo Scholtes from ETH Zurich presented his work on the use of machine learning techniques to uncover the hidden social influence on scienti c success.
The second day of the workshop started with a session devoted to data providers. Dr. Evangelia Lipitakis introduced the data sets available through the services of Thomson Reuters, and gave some insights into the underlying data curation process. Dr. Martijn Roelandse from Springer provided his view on the e ffects of Open Access strategies on scienti c impact, as well as on the importance of article-level metrics. In a final session on Statistical Analysis of Science Networks Dr. Matus Medo from the University of Fribourg presented his work on a modifi cation of PageRank to account for temporal biases due to the evolution of networks. Dr. Olesya Mryglod from the National Academy of Sciences of Ukraine presented an analysis of download patterns for scienti c articles, and their correlation with citations. Prof. Alexander Petersen from IMT Lucca discussed growth trends in science careers, highlighting the importance of cohort analyses that separate scientists in di erent career stages.
While all the sessions were accompanied by intense discussions among participants, the workshop concluded with a plenary discussion. The discussion was led by Dr. Urs Hugentobler, head of Institutional Research at ETH Zurich, and gave participants the opportunity to present their personal wrap-up. The problems of predicting success and the potential impact on young researchers was identifi ed as one particularly important topic. The plenary discussion further focused on the sensitivity of indicators for manipulation and the importance to develop more robust measures for scientifi c impact. Furthermore, the fact that it remains difficult to reasonably assess interdisciplinary impact was highlighted. In conclusion, the workshop succeeded to provide an intense and inspiring atmosphere that was appreciated by all participants. The focus on bringing together stakeholders and scientists turned out to be successful and helped both sides to better understand the flaws in currently used methodology, as well as their potential impact on science.
[The programme and more documentation of the workshop can be found here: https://www.sg.ethz.ch/activities-events/workshops/cost-td1210/]
