14 September 2009

Submissions for the ERA Indicators Consultation Paper due: 25 September 2009

Download Submission proforma   [RTF file size: 400kB]

Download Consultation paper     [PDF file size: 1MB]

The National Academies Forum brought researchers together with the Australian Research Council to discuss many of the relevant issues in Canberra in early September. After a day of discussion and review, the Seminar made a number of detailed suggestions for the HASS indicators and research evaluation methodologies. The seminar communiqué has a summary of the discussion.

Download NAF communiqué     [PDF file size: 700kB]

The Australian Research Council released the ERA - Indicators Consultation Paper for discussion. CHASS has prepared the following summary of the paper on the issues relating to the humanities, arts and social sciences.

Summary of the ERA Indicators Consultation Paper

The ERA Indicators Consultation Paper suggests solutions for a number of outstanding issues in the Excellence in Research for Australia initiative. Consultation is sought as the Australia Research Council finalises the indicators and submission guidelines for the first round of ERA evaluations in 2010. But the response time for submissions is short - the deadline is September 25th.

The Government will use the results from the 2010 ERA evaluation in some funding decisions from 2012. It is important that the data collected is as useful as possible, providing an accurate reflection of the state of the HASS sector.

The consultation paper contains a disciplinary matrix for each cluster, listing the specific indicators to be applied to each discipline at a four-digit Field of Research (FoR) code. The nominated indicators include peer review, esteem, applied, volume and activity analysis, citation analysis and research income. The ARC has sought to develop indicators that are "robust and valid measures of research quality": quantitative, internationally recognised, repeatable and verifiable, identify excellence, and research relevant.

An examination of the disciplinary matrices shows that ARC has attempted to ensure the ERA initiative will adequately reflect the differing methodologies and outputs of Australian research. Generally, the indicators have qualitative and quantitative measures appropriate for each discipline. In addition, each institution has the opportunity to submit a 10,000 character background statement for each discipline at the 2-digit Field of Research code with their submission, contextualising the discipline's role and further achievements not represented by the ERA indicators.

Humanities and Creative Arts

In the Humanities and Creative Arts (HCA) cluster especially, the ARC has undertaken extensive consultation with sector representatives to ensure the indicators applied to the HCA cluster are appropriate for the unique range of research outputs and practices represented by the cluster. The detailed development of the HCA cluster reflects its involvement in the current trial of ERA. The HCA cluster now incorporates a number of discipline-specific indicators to assist in rigorously evaluating Humanities and Creative Arts disciplines, such as:

• Esteem measures, including Australia Council grants and Fellowships, have been included in the 2010 exercise;
• Creative Works have been included as eligible types of research output, accompanied by a 100-word statement outlining the work's research component;
• A number of other esteem measures not measured as separate esteem indicators, such as repeat performances, can be included with the relevant Creative Work research output, alongside the research statement;
• Citation analysis is NOT being used as an indicator for the HCA cluster;
• Peer review is being used to evaluate research quality in the HCA cluster, for 20% of research outputs.

Social, Behavioural and Economic Sciences

In the Social, Behavioural and Economic Sciences (SBE) cluster, a range of disciplinary specific indicators have been incorporated to reflect the range of methodologies and research outputs in the SBE cluster. For example, all research outputs across the 25 disciplines within the SBE cluster will be subject to a mixture of peer review and citation analysis, to allay concerns about the ability of citation information supplier to fully capture the broad spectrum of citations in the social sciences.

The current indicators lack the appropriate indicators for assessing the research quality, especially in the applied social sciences. In particular, esteem measures and research outputs for applied SBE, such as the provision of expert advice to government and industry and the development of Government reviews and report, are absent from the current list of indicators.

There are some indicators listed in the SBE discipline matrix where more detailed definitions of the output would help fill the gaps. Your suggestions, for instance on inclusions under "non-print research outputs", will be important.

As the SBE cluster did not undergo a trial this year, it is important that feedback on the appropriateness of the indicators listed for your discipline is forwarded to the ARC. Where members feel the indicators are appropriate and useful, the ARC needs to hear this too.

Discipline Matrices

The Discipline Matrices for each cluster are included in the consultation paper. Some issues arising from the matrices are listed below.

Humanities and Creative Arts (pp 15-18)

• Page 15: For all humanities disciplines, 20% of all research output types except for conference publications are tagged for peer-review.
• Page 15: The Built and Environment and Design (12) in addition to all other research outputs, also have conference publications tagged for peer review.
• Page 16: Australia Council Grants and Fellowships are counted as indicators of esteem for Performing Arts and Creative Writing (1904) and Visual Arts and Crafts (1905).
• Page 16: Patents and Registered designs have been included as indicators for Curatorial and Related Studies (2102).
• Page 17: Cites in legal judgements have been included as an indicator for the Law and Legal Studies disciplines (18).
• Page 17: With the exception of legal judgements, citation analysis does not apply to the disciplines in the HCA cluster.

Social, Behavioural and Economic Sciences (pp 22-25)

• Page 22: Non-print research outputs have been included for Education (13), Commerce, Management, Tourism and Services (15), and Social Work (1607), demonstrating an attempt to include non-traditional research outputs.
• Page 22: For a large number of SBE disciplines, journal articles and conference publications are not being peer reviewed.
• Page 22: Anthropology (1601), Policy & Administration (1605), Political Science (1606), Social Work (1607) and Sociology (1608) will have 30% of research outputs tagged for peer-review, instead of 20%.
• Page 23: Esteem measures to be collected for the SBE cluster are limited to Editorial role for an A* or A ranked journal; editor of a prestigious work of reference; Category 1 research fellowships; and membership of a learned academy.
• Page 23: Research commercialisation income is not being counted as an applied indicator for the Anthropology (1601), Criminology (1602) and Demography (1603) disciplines.
• Page 25: Citation analysis applies to all disciplines within the SBE cluster.

Consultation is being sought from institutions, researchers and interested stakeholders on the appropriateness of discipline-specific indicators identified in the discipline matrices. Instructions on how to make a submission can be found on the ARC website, and the proforma submission document is available for download at the beginning of this document.

The issue of journal rankings has not been raised in the ERA Indicators Consultation Paper. The full list of journal titles is available on the ARC website for review. However, the list does not include FoR assignments and rankings at this stage. Comments on the current list and suggestions for omitted journals meeting the ERA requirements can be sent to the ERA team. Instructions on how to suggest omitted journals meeting the ERA criteria are available on the ARC website.

CHASS would like to hear from our members on the issues arising from the ERA consultation paper.

Please forward your comments and submissions to the consultation process to Council's Research and Policy Officer, who will be collating the views from our membership.
Heidi Hutchison, email: policy [at] chass.org.au, or phone: (02) 6201 2559

Heidi Hutchison
14 September 2009

Related links

• Media release
  Indicators Consultation paper for ERA
• Advocacy paper
  Humanities and Creative Arts: Recognising Esteem Factors and Non-Traditional Publication
• Professor Margaret Seares AO, previous chair of the Creative Arts Committee
  A new era for Creative Arts research
• Cluster Two Research Evaluation Committee

 

 

For more information, please contact:

Executive Director

Council of the Humanties, Arts and Social Sciences

Phone: +61 2 6201 2740

director@chass.org.au