Response to ERA Indicators Consultation Paper

Ms Helen O'Neil
Executive Director
2 October 2009

The ARC sought feedback from the sector on the issues raised in the ERA Indicators Consultation Paper. The Council for the Humanities, Arts and Social Sciences submitted the following response. A number of HASS sector leaders and specialists were involved in the Council's consultation process, with a number also attending the National Academies Forum's Seminar on Excellence in Research Evaluation. Members who directly contributed to the preparation of this submission are listed in the submission.

Download this submission

Discipline-specific Indicators

1. Please provide comments regarding the collection and verification of data for any of the specified indicator categories for the full ERA evaluations from an institutional perspective.

Response:

a. The Interior Design/Interior Architecture Educators Association notes that there is at present no list of ranked conferences for FoR 1201 although it is proposed to use this measure as an indicator in this field. The ARC will need to provide a sector-agreed list of ranked conferences in order for the institution to collect and verify data on the ranking.

2. Please identify any discipline(s) where the proposed use of ranked conferences or citation analysis is likely to not be supported by a broad consensus in the discipline(s). Please provide a short justification for adding or removing the discipline(s) in the indicator category.

Response:

a. The Council for the Humanities, Arts and Social Sciences (referred to as 'The Council' hereafter) welcomes the continued exclusion of citation analysis as an indicator for the Humanities and Creative Arts cluster.

b. The Council notes that while there are differing opinions within the varying disciplines of the SBE cluster regarding the use of citation analysis versus peer review, member feedback argues that the suggested system for counting citations is not at present a comprehensive or reliable indicator for the majority of research outputs for some disciplines in the SBE cluster, especially disciplines with a strong inter-disciplinary focus, such education, sociology and political science. This is due to the difficulty current citation analysis tools have in picking up Australian research published in local journals. Additionally, books and book chapters remain vital sources of citations, but of the kind not counted by citation analysis systems. This argument has been confirmed by comments from disciplines within the HCA cluster (see points e. and f. below).

c. A hybrid model of citation analysis and/or peer review within both the HCA and SBE clusters is most likely to ensure a rigorous and fair assessment process, but only if the citation analysis method to be included is one that also accounts for local research and citations of and within books and book chapters (refer to specific comments below). Note that in preparation for the RQF, and tested against the 2006 UK RAE results, a more comprehensive bibilometrics process incorporating citation patterns from books and books chapters was developed, and is available for the ARC for use in the SBE panel.

d. The Australian Political Studies Association (APSA) requests that citation analysis not be used in Political Science (1606) and Policy and Administration (1605). The reasons for their recommendation against citation analysis are:

• Research of national significance published in local journals is less likely to be captured in citation counts. The CHASS Bibliometrics Project (2006) demonstrated that researchers focussing on international topics achieved higher citation counts than equally-regarded scholars working on Australian or Asian topics. Both APSA and The Australian Sociological Association (TASA) note that it is extremely difficult to get articles that focus on Australian topics and data published in international journals. Rather, to be published in international journals in sociology and political science, which are predominantly US and UK based, it is necessary to use examples and case studies from those countries.
• Researchers in political science and sociology tend to publish more heavily in monograph form, as books or book chapters. The citations appearing in monographs are not captured by citation suppliers. APSA cites the CHASS Bibliometrics Project (2006) findings that less than 20% of publications in political science and history were captured by Thomson ISI bibliometrics.
• APSA also notes that research undertaken for the RAE/REF process in the United Kingdom shows that there is little correlation between the results of citation analysis and other quality assessment processes in the Social Sciences.
• In APSA's own words, "Australian politics research is a similar case to Australian history - whose discipline has successfully argued that citation indicators are inappropriate. The use of citation indices would have a detrimental impact on the evaluation of the work of many of our colleagues who are making a major contribution to the understanding of Australian political institutions and wish to publish in the only place which is interested in detailed explorations of Australian institutions, namely Australia."

e. The Australasian Society for Classical Studies affirms the exclusion of citation analysis for Classics, stating the extremely small number of journals included in the citation index for Classics as the reason. They argue that the number of citations of any one scholar will be a false indicator of the quality or significance of his or her work and the esteem in which he or she is held. The Australasian Society for Classical Studies also note that if the time were to arrive when the full range of journals is included in the citation index, the question of citation analysis could be revisited and the indicators revised to include citation analysis for Classics.

f. The Australian Association of Philosophy, similarly, notes that they have no objection to the use of citation analyses (were they to be used in the HCA cluster), provided they include citations to books and book chapters and provided that they are not taken as a substitute for peer review.

g. The Interior Design/Interior Architecture Educators Association agrees with the proposed ERA Indicators that citation analysis is not appropriate for any of the FoR 12 disciplines, and supports the proposed ranking of conferences for FOR 12 disciplines. However, they note that Visual Communication / Graphic Design (within FOR 1203) has not, as yet, established a discipline-specific academic conference series to which academics regularly contribute.

3. For applied indicators, as well as peer review indicator categories, please identify any discipline(s) in the Matrix for each Cluster where there is likely to be broad consensus that the proposed use of the indicator is not useful or valid.

Response:

a. The Council welcomes the inclusion of peer review in the SBE cluster, and commend the ARC for recognising the importance of peer review to certain disciplines in the SBE cluster, namely Anthropology (1601), Policy & Administration (1605), Political Science (1606), Social Work (1607) and Sociology (1608), which will have 30% of research outputs tagged for peer-review, instead of 20%.

b. The Council suggests that if the ARC decides to remove citation analysis for disciplines like Political Science, as has been requested at 2d. in this submission, that it should negotiate with the relevant disciplinary associations and bodies to reduce the number of research outputs being tagged for peer review to 20%. This is to ensure a manageable caseload of reviews for the reviewers.

c. The Australian Association of Philosophy suggests that the 30% level would be appropriate in the case where there is a relatively small number of research outputs (say, less than 40), to avoid sample bias. This would be preferable to aggregating to a 2-digit FoR for disciplines with low volumes of research outputs.

d. The Interior Design/Interior Architecture Educators Association supports peer review of creative works and believes that textual creative works be treated with equal value as evidence in ranked outlets. They expect that there will be broad consensus on applied and peer review indicators in the FOR12 disciplines.

4. For the peer review indicator, please comment on: any additional output type(s) which could be included in this category, including a short justification for their inclusion; and any issues which significantly impact on the tagging of outputs for peer review.

Response:

a. Changes in scholarly communication have led to the traditional forms of scholarly publication, represented in the research output types listed in the ERA Indicators, being increasingly complemented by non-traditional forms of communicating research findings. It is necessary that ERA also assesses research excellence presented in non-traditional communication channels. The Council thanks the ARC for the inclusion of Creative Works as research outputs in the Humanities and Creative Arts cluster. The Council also notes with approval the inclusion of non-print research outputs for peer-review in a number of the applied social sciences in the SBE cluster. The Council recommends that the list of disciplines for which non-print research outputs are valid indicators be extended to include other disciplines with a strong inter-disciplinary focus, such as Criminology, Political Science, and Sociology.

b. The Interior Design/Interior Architecture Educators Association concur with the current ERA guidelines that allow for the broad range of research outputs from FOR 12 disciplines to be included in the institution's submission. Their experience of the ERA trial process did not reveal issues which impacted adversely upon the capacity to tag outputs for peer review.

c. The Council requests that the indicators for the Social, Behavioural and Economic Sciences cluster and some Humanities and Creative Arts disciplines reflect a slightly broader list of print-based research types than is currently endorsed. This is particularly important for those disciplines engaging in multi-disciplinary but highly policy relevant areas of study, such as in the Studies in Human Society (16) and Education (13) disciplines. Maintaining a narrow HERDC defined list of print-based research outputs for these disciplines further exacerbates the real tension between the goals of the ERA Initiative and the Government's desire for policy relevant and interdisciplinary research outlined in the Powering Ideas agenda.

d. The list of research output types counted for the SBE Cluster and some disciplines in the HCA cluster (philosophy, history and law) should be broadened to include the following non-traditional scholarly publications, to be subjected to peer review and volume analysis:

• Edited books and collections of new papers with prestigious publishers
• The Australasian Council of Deans of Arts, Social Sciences and Humanities and the Australian Association of Philosophy note that in many humanities and social science disciplines, edited collections of new papers are an important and recognised means of defining and shaping a research area and advancing debates in those areas. The conception, organisation, and commissioning of papers for such a research collection is itself a research activity and the editorial activity including providing comments to authors and writing an introduction which provides the intellectual overview deserves recognition.
• Research articles in the media or popular journals
• Occasional Papers and Working Papers, with a demonstrated research component and published by institutions, research centres, learned academies, government departments and not-for-profit organisations
• Research report - commissioned by Government, Industry or other relevant organisation
• Contributions to Textbooks
• Legal cases
• Substantial entries in encyclopaedias and prestigious works of reference.
• The Australian Association of Philosophy notes also that these are important research contributions in the discipline of philosophy.

e. The Council further suggests that the ARC allow non-traditional research outputs in the SBE cluster (of the kind listed above) to be reported in a manner similar to Creative Works in the HCA Cluster and subject to peer review, as has been suggested for 'Non-print research outputs'. With Creative works, a framework has already been established in how to collect non-traditional publications and to verify their research component. In this way, measures of esteem and translations of research outcomes (such as Sustained media commentary in area of research specialisation; Deployment to government, industry as an expert advisor; Evidence/ submissions to parliamentary inquiries) directly related to a particular research output can also be incorporated into the data collection without making the collection of data related to esteem measures unworkable.

5. Are there specific publishing behaviours which may affect the attribution of publications to the most relevant FoR code using the ERA journal FoR assignments? How would you suggest that the ERA approach might overcome this issue to ensure that research outputs are correctly attributed to the correct FoR code?

Response:

a. The Council suggests that FoR codes be assigned to according to the researcher rather than the publication. This will help alleviate difficulties in assigning FoR codes to interdisciplinary journals.

b. The Interior Design/Interior Architecture Educators Association notes that researchers in FoR 12 disciplines often collaborate on interdisciplinary and multi-disciplinary projects. As a rule, they publish in journals that are the most relevant outlet for the research. Whilst the chosen journal might be the most relevant in regard to the nature of the content, it may not necessarily reflect the researcher's main field of research (e.g. psychologist publishing in an Architecture journal). Future inter/multi-disciplinary publishing behaviours may be influenced if researchers choose to focus on publishing only in journals that are assigned a FoR code that matches their own main strength. The assignation of rankings to journals may influence some academics to skew the content of a journal article in an endeavour to meet the content focus of a particular journal and in particular those with an A* or A ranking.

Proposed inclusion of Esteem Indicators

6. Please provide comments regarding the collection and verification of data for any of the specified indicator categories for the full ERA evaluations from an institutional perspective.

Response:

a. Other recommended esteem indicators, consistent with the Esteem indicator framework outlined at Appendix B2 of the ERA Indicators Consultation Paper, suggested by Council members include:

• Executive roles and Fellowships of professional bodies (national and international)
• Appointments to government committees
• Executive roles in scholarly associations and societies
• Have served by invitation as a research assessor nationally /internationally ( for example, on the ARC's College of Experts or other comparable body such as the NHMRC/international research panels)

b. The Interior Design/Interior Architecture Educators Association (IDEA) notes that institutions have not routinely collected data about Australia Council grants and fellowships as they have not previously been eligible for the HERDC. Whilst IDEA are supportive of the inclusion of Australia Council funding, it should be recognised that for the 2003-08 period, it may prove difficult to retrospectively collect the required data and evidence.

7. Please provide comments regarding specified roles articulated in any of the indicator categories.

Response:

a. The Interior Design/Interior Architecture Educators Association suggests that in relation to editorial roles of A* or A journals, that the ERA guidelines should state that the specified roles eligible for reporting are academic roles, as some journals have academic and non academic editors (e.g. executive/ managing editors).

8. What titles should be included as prestigious works of reference? Please note that the list should be restricted to a small number of works (maximum 10 per discipline) representing the highest scholarly standards.

Response:

a. The Australian Association of Philosophy offers the following list as paradigm examples for philosophy, noting the difficulty of specifying these in advance, given that new ones may be conceived and produced.

• Oxford University Press Handbooks series
• Oxford World Classics
• Cambridge University Press Companions series
• Cambridge University Press Texts in the History of Philosophy
• The Routledge Encyclopaedia of Philosophy
• The Stanford Encyclopaedia of Philosophy
• Blackwells Companions Series

9. Please provide comments on the suggested list of statutory committees.

Nil Response

10. Please provide comments on the suggested method for counting nationally competitive research fellowships and Australia Council grants and fellowships.

Response:

a. The Council notes with approval the inclusion of Australia Council grants and fellowships and nationally competitive research fellowships as indicators of esteem.

b. The Council agrees with the methodology for counting nationally competitive fellowships, but believes that they should only be counted once, either as an esteem measures or included as research income.

c. The Interior Design/Interior Architecture Educators Association suggests that details on successful Australia Council Grants should be counted according to the amount awarded and the category of the award (e.g. residency; fellowship) but that the individual researcher should provide a 200/250 word explanatory statement regarding the research and its contribution. Universities could set up a collection system for Australia Council Grants and Fellowships in a separate but similar system to those that exist for Category 1-4 grant reporting. This should be implemented as soon as possible and that retrospective data collection includes 2008.

Proposed Creative Arts Research Outputs that includes Esteem Indicators

11. Please provide comment on the proposed revision to the creative arts research outputs that allows the inclusion of some esteem information directly connected to the research output.

Response:

a. The Council supports the proposed revision of creative arts research outputs to include the collection of data relating to esteem measures directly connected with the research output.

b. The Council feels that this framework could be adopted in the SBE cluster to allow non-traditional research outputs and directly related esteem measures to be included. This will allow measures of esteem important to social science researchers to also be considered when assessing research excellence. For example, a commissioned research report can lead to invited submissions and giving evidence to parliamentary inquiries. Some measures of esteem that would be captured in this way include:

• Recognition of high level research by a professional association
• Prizes and awards (national/international)
• Deployment as an expert adviser on an area of specialisation (for example, evaluation of a government program, involvement in white papers)
• Sustained media commentary relating to a particular research output
• Invitation to make a submission and give evidence to parliamentary enquiries
• Translation of books, chapters and articles into other languages
• Invitation to revise or publish a new edition of a book

c. The Interior Design/Interior Architecture Educators Association welcomes the recommendation to adopt a framework that includes 'limited information about esteem indicators connected to the output' and agrees with the proposed list of optional information on specific outputs. They suggest, however, that the maximum length of the research statements should be 200 or 250 words, not 100 as recommended. Our experience with the ERA trial indicates that a researcher needs more than 100 words to adequately summarise the research focus, significance and contribution of an output. We also support the recommendation to not include awards and prizes as an indicator of esteem due to the 'difficulty of unambiguously identifying those...awarded principally on the basis of research excellence'. There are other places in the submission (eg. The Background Statement, individual research statements) where reference can be made to the receipt of prestigious prizes and awards.

Additional comments

a. The Council welcomes the effort of the ARC in trying to ensure that the ERA initiative will adequately reflect the differing methodologies and outputs of Australian research. Generally, the indicators have a combination of qualitative and quantitative measures appropriate for each discipline. In addition, each institution has the opportunity to submit a 10,000 character background statement for each discipline at the 2-digit Field of Research code with their submission, contextualising the discipline's role and further achievements not represented by the ERA indicators.

b. The Council also appreciates the extensive consultation the ARC has undertaken with HCA sector representatives to ensure the indicators applied to the HCA cluster are appropriate for the unique range of research outputs and practices represented by the cluster. The detailed development of the HCA cluster reflects the background work of the Humanities and Creative Arts sub-committees of the ERA Indicators Development Group, and its involvement in the current trial of ERA. In particular, the Collections Council of Australia welcomes the recognition of nationally significant collecting organisations as venues for the delivery of creative works or performances in Appendix 1 of Appendix B.

c. The Council, on behalf of those members who participated in consultation, would also like to make some additional recommendations relating to the ERA initiative:

1. ERA Trial for SBE in 2010:

Council members would like to be assured that there will be the opportunity for further refinement of the indicators at the conclusion of the 2010 assessment exercise. In particular, members whose disciplines fall into the SBE cluster would like to see the 2010 ERA process for the SBE cluster conducted as a trial, to ensure the necessary refinement of the process before funding and planning decisions are directly connected to the results.

2. Links between ERA and research funding:

Research funding should not be linked to ERA evaluation findings until all discipline clusters have been trialled and determined to be a rigorous and fair evaluation of research quality. International experience in research evaluation illustrates that it takes time for the research assessment to be translated to real increases in research performance. It would be detrimental to the Australian research environment if institutions and disciplines were to be punished financially as a result of research assessment outcomes that have not yet been trialled, and without the opportunity to respond to assessment results through the implementation of processes to improve future outcomes. Institutions and the ARC should use the 2010 round of ERA evaluation findings as a strategic planning tool, to assess the value of their HASS sectors, identify areas of future strength and ensure they are adequately resourced.

3. Research Evaluation Committees (RECs):

The RECs will have a significant role to play in collating, interpreting and assessing the ERA indicators for each discipline. International experience has confirmed that the membership of the RECs can have significant sway in determining a discipline and institution's final ranking. A study by Linda Butler and Ian McAllister (ANU) of the 2006 RAE results in political science found that in the evaluation process, the biggest indicator of a department's final rating was the presence of a departmental member on the research evaluation panel. It is thus essential that all disciplines are adequately represented and understood by those on the RECs. This is particularly the case for the Research Evaluation Committees for both the SBE and HCA clusters, representing a wide range of disciplines. The RECs for these clusters will need to be large and diverse, incorporating sub-panels at a 2-digit level, to ensure adequate representation of all assessable disciplines.

4. Allocation of FoR codes within the SBE Cluster:

i. The ARC should consider re-evaluating of the four-digit FoR codes to more accurately reflect the status of particular disciplines. The Australian Council of Deans of Education note specifically that research in the discipline of Education has been distributed between other disciplines, such as sociology. There is a danger that the allocation of FoR codes in this way will distort the ERA evaluation findings and thus disaggregate a discipline that has a proven track record of research.

ii. In such a diverse cluster of disciplines, there are inherent dangers in aggregating up to 2-digit FoR level for low volume disciplines. This submission has suggested (see point 3c) that other methods of assessing low volume disciplines, such as increased research outputs tagged for peer review, could be adopted to ensure low volume disciplines are fairly assessed.

5. Tagging Research themes:

The list of research themes is closely tied to the National Research Priorities. ASSA comments that "Pursuing the tagging of research themes to national research priorities necessarily entails formalising the setting of research agendas by the political agenda, rather than vice versa." The humanities, arts and social sciences (HASS) disciplines are not well represented in this list. If this list is to be tied to further research funding, it will need to be revised to more adequately reflect the contribution of the HASS disciplines to nationally significant areas of research.

6. Journal rankings:

i. The issue of journal rankings has not been raised in the ERA Indicators Consultation Paper. While, the full list of journal titles is available on the ARC website for review, at this stage the list does not include FoR assignments and rankings at this stage. It is important that researchers in the SBE and HCA clusters be given further opportunities to review and revise the assigned rankings as the ARC finalises the ranked journal lists.

ii. The Council also requests that mechanisms are put in place that allow for further recognition of quality of local and regionally focused journals in the top tiers of the ERA journal rankings. Currently, local and regionally focused research does not feature in the overall criterion for journal rankings until tier B. Having local and regional journals feature in the top tiers is an essential precursor to the capture of citations for disciplines whose research has a strong local flavour, such as political science and sociology.

iii. In the HCA and SBE clusters, representing a wide range of disciplines, the ARC must ensure consistency across disciplines in how the rankings have been applied by the respective disciplinary associations and bodies involved in the rankings and consultations process. The concern across the sector, emphasised by DASSH and The Australian Sociological Association, regarding discrepancies and transparency of ERA journal rankings must be adequately dealt with by the ARC if the ERA process is to gain acceptance and support amongst researchers.

7. HDR students as eligible researchers:

According to the ERA Guidelines, HDR students are only eligible researchers if also employed by the university, including on a casual basis. One of the implications of this rule could be increased pressure on HDR students to undertake regular casual teaching while completing their research, so that their research outputs can be included in an institution's ERA submission. Given that some estimates indicate that up to two-thirds of universities' total research output is performed by postgraduate students, having HDR research outputs not count towards ERA could also lead to less support for postgraduate research, less willingness to take on an increased number of HDR places, and less willingness for co-publications between supervisors and postgraduates students. This could impact on future Government policy on the Research Workforce Strategy, and the future sustainability of the research workforce.

8. Honoraries as eligible researchers:

Counting adjuncts and honoraries has the potential to distort the ERA evaluation findings, through the possibility of dual reporting of a researcher's work by multiple institutions. The ARC should work at ensuring each research output for an honorary is only counted once, and not submitted by multiple institutions.

About the Council and the consultation process

a. CHASS promotes and provides advocacy for the Humanities, the Arts and Social Sciences. It serves as a coordinating forum to link academics, students, business, practitioners and the broader community. With more than 100 member organisations, the Council is an active network of specialists across humanities disciplines, creative arts, and research and professional practice in the social sciences.

b. A number of HASS sector leaders and specialists were involved in the Council's consultation process, with a number also attending the National Academies Forum's Seminar on Excellence in Research Evaluation. Members who directly contributed to the preparation of this submission include:

• The Academy of Social Sciences in Australia's (ASSA) Policy and Advocacy Committee
• The Australasian Council of Deans of Arts, Social Sciences and Humanities (DASSH)
• The Australian Council of University Art and Design Schools (ACUADS)
• The Australian Council of Deans of Education (ACDE)
• The Australian Sociological Association (TASA)
• The Australian Association of Philosophy Council
• The Australian Political Studies Association (APSA)
• Interior Design/Interior Architecture Educators Association (IDEA)
• Collections Council of Australia Ltd.
• Australian and New Zealand Communication Association Inc.
• The Australasian Society for Classical Studies

c. The Council's consultation process included seeking comments from relevant members and HASS sector leaders and specialists in response to a discussion paper covering ERA indicators in the HCA and SBE clusters. The comments made in this submission on behalf of the Council relate to issues raised in the discussion paper and affirmed by members and HASS specialists participating in the consultation. In addition, some members have requested that the Council make specific comments on their behalf, which are clearly indicated in the submission.

d. While this submission is a fair reflection of members' views and feedback, there will be significant diversity in detailed views given the range of the Council's membership.

For queries on the submission contact
Research and Policy Officer
policy [at] chass.org.au

 

For more information, please contact:

Executive Director

Council of the Humanties, Arts and Social Sciences

Phone: 02 6201 2740

director [at] chass.org.au