Council for the Humanities, Arts and Social Sciences

Political Science and History Panels

Report on recommendations and major issues

Meeting at Open Universities Australia

14 November 2006

Recommendations

1. That conventional Thomson ISI bibliometrics not be considered appropriate measures of quality in the disciplines of Political Science and History at Australian universities because they capture only a small proportion of published articles (about 20%).
2. That an enriched bibliometrics approach, as developed by Dr Linda Butler and set out in the meeting papers as Table 1, would be a useful component of the Quality measures in the proposed Research Quality Framework if used to inform a peer-review process. Its value would be further enhanced if:
   • An expert unit could annotate the data before they are reported to RQF expert panels, to ensure rigour and comparability, and to note any particular characteristics of a group's research portfolio that might influence its citation performance.
   • Supplementary data show the number of staff involved in a particular group's research output;
   • Supplementary data show the number of staff who published, and whose publications received one or more citations during the period under review;
   • Supplementary data show the detailed citations counts for any individual staff member whose publications attracted more than 10% of their research group's citations during the period under review; and
   • Citations were measured over a duration of seven years.
3. That DEST commission further research into publishing and citation trends across publications not indexed by the ISI, which represent very high proportions of DEST-recognised scholarly publications in the disciplines of Political Science and History at Australian universities. Such research should closely examine how citation patterns from books and book chapters compare with citation patterns from journal articles.
4. That given the inherent limitations of any new bibliometrics system,¹ round 1 of any RQF process should avoid using one single measure to generate its rankings. Peer review must be at the core of the system, and expressly tasked to take disciplinary and sub-disciplinary dynamics into account.
5. That sub-disciplinary factors be expressly taken into account by the RQF, especially the greater difficulty that Australianist publications have in generating citations, compared with internationalist publications.
6. That there be regular (frequent) reviewing of the usefulness of this data as an information aid for the RQF expert panels.
7. That DEST pay serious attention to the quality of institutions' reports on their research publications, which appears highly variable according to data submitted for this project.
8. That there be regular (frequent) evaluation of the extent to which RQF procedures encourage game-playing responses from institutions, such as 'citation clubs'.
9. That DEST take serious note of the potential for a bibliometrics process to be biased against early career researchers, with potentially harmful consequences for the behaviour of institutions in response to the RQF.

1. The History Panel noted that, in particular, Paul Bourke has discussed several major caveats around the bibliometrics approach. On the one hand, he argues that bibliometrics can be instrumental in alerting expert/peer reviewers to the existence of anomalies. On the other hand, it is only through the process of expert/peer review that any anomalies can be recognised and allowed for.

Other major issues discussed

Both panels noted that the institutional rankings produced by examining ISI-only data were inconsistent with their perceptions of research quality around the country. Panellists agreed that the data painted a much more accurate picture when expanded to include books and book chapters.

Both panels noted that the standard of research reporting by universities' research offices was irregular. In particular, panellists noted that numerous articles in non-refereed journals are being reported as C1 publications, contrary to DEST rules.

Both panels stressed the importance of the role of bibliometrics data, arguing that they should serve the RQF by informing peer/expert review, rather than by leading its assessment processes. The panels felt that bibliometrics data would be more meaningful if accompanied by measures of staff productivity (publications per effective full time staff member) and employment status (seniority levels being particularly informative information).

Any bibliometrics exercise needs to be mindful of a structural bias against local projects. Both panels noted that citations counts for writers on international topics were generally higher than similarly well-regarded scholars working on Australian and Asian topics. They believe a similar effect occurs in other disciplines within the social sciences and humanities. This is potentially a very serious problem for an Australian national research funding system. It raises the possibility that Australian universities might be encouraged to run down their research strengths in locally relevant studies across the board.

Both panels also noted the potential for this system to produce other unintended consequences, including game-playing responses, such as 'citation clubs'. DEST would be advised to model these potential effects wherever possible, as well as to seek advice from other stakeholders, prior to implementing this system. Unintended consequences should also be monitored closely during regular evaluations of the RQF.

Generally speaking, a bibliometrics approach wants maximum inclusion of staff and their publications in each group's sample. The Political Science panel noted that other aspects of the RQF will encourage universities to withhold information about lower performing staff and early career researchers from their reporting.

Members of the History panel expressed a concern that citation counting is blind as to the 'quality' of a given citation. Thus an in-depth discussion of a particular piece of scholarship rates the same as a citation-only listing of it. It is hard to imagine how this limitation could be overcome, practically speaking, although it points to the need for additional sources of data on research quality alongside the citations count data.

Participants in the Political Science panel were: Bob Goodin, Tony Bourke, Judith Brett, Stuart Macintyre, Deborah Brennan, Mark Considine, Linda Butler, Kumara Henadeerage, Toss Gascoigne, and Tom Clark.

Participants in the History Panel were: Marion Diamond, Stuart Macintyre, Philip Deery, Iain McCalman, Tom Stannage, Tony Taylor, John Byron, Linda Butler, Kumara Henadeerage, Toss Gascoigne, and Tom Clark.

 

Toss Gascoigne
15 November 2006

Download this paper   [PDF file size: 31.53 kB]   REF: PAP20061214CH

 

For more information, please contact:

Toss Gascoigne

Executive Director

Council of the Humanties, Arts and Social Sciences

Phone: +61 2 6249 1995

director@chass.org.au