15 February 2010

Preparations for Excellence in Research in Australia (ERA) are in full swing across Australian universities. All research disciplines will be evaluated this year with submissions opening on the 1st June, which will result in the publication of the first ever picture of the quality of research in Australia's universities against national and international benchmarks.

The Australian Research Council recently briefed CHASS about the 2009 trial of the ERA process in the humanities and creative arts, and the changes which have been made for ERA 2010 in light of the trial experience. The ARC has an active outreach program to assist the sector understand changes made as a result of the trial and continues to engage with universities, cohort groups, research managers and researchers.

2009 HCA Trial

In the 2009 trial, all of the forty-one eligible universities made a submission in the Humanities and Creative Arts (HCA) cluster. The submissions included 47,000 research outputs, including 7,000 creative works.

The primary purpose of the 2009 ERA Trial was to test the methodology and relevant university preparedness. Given that it was a trial, the ARC only released the outcomes of evaluations to individual institutions, rather than releasing them publicly. As a result, the full trial results have not been released, preventing an analysis of variations in quality across institutions offering a discipline in 2009. However, some institutions and some cohorts of institutions have shared their results to assist with their interpretation of the outcomes, and preparations for ERA in 2010.

The first public release of the trial results showed at least fifteen universities earned a 5 rating for outstanding performance, with at least another nine achieving a four rating for excellent performance. These ratings were achieved in half of the Humanities and Creative Arts fields. Philosophy, history and archaeology stood out as high performing disciplines.

Professor Stuart Cunningham from CHASS has commented that the trial showed that investment in the humanities was evident from the outcomes of the trial.

The ARC said the trial has resulted in better understanding by the sector of the ERA methodology.

Journal rankings have been controversial in the lead up to the trials, but there are other areas in ERA which will need attention in universities and faculties. These are outlined below.

ERA 2010

The ARC has incorporated feedback from the submissions and evaluation processes into the 2010 ERA Submission Guidelines, and is using this information to further refine the process before submissions open on 1 June.

Some specific observations about the 2009 ERA Trial and changes to ERA in 2010 provide crucial insight into the process for 2010.

Expert review is central to evaluations in ERA, and is conducted by appointed Research Evaluation Committees (RECs). For the Trial, the Humanities and Creative Arts REC had 22 members broadly representative of the disciplines in the cluster. A new Research Evaluation Committee will be appointed for Humanities and Creative Arts in 2010. The ARC will aim for a similar diversity of disciplinary expertise in both the HCA and the Social, Behavioural and Economic Sciences (SBE) RECs for 2010.

In the Trial, and continuing for the 2010 exercise, the REC members will use their expert judgment in addressing the indicator set for the relevant discipline, and therefore how the research outputs and other material submitted will be evaluated. The RECs are responsible for both a consistent application of the methodology and interpretation of the rating scale. RECs also have access to a range of other information about each institution and discipline, such as staff FTE numbers and academic level, which are also used as context for evaluations.

The various indicators, which in the HCA and SBE clusters include peer review of research outputs, esteem measures, research income, and for some disciplines, applied measure such as patents, registered designs and citation analysis, are presented to REC members as metrics, in the form of a 'dashboard'. This allows the RECs to view the full picture of indicators for each discipline. These indicators are not weighted, but are considered by the expert reviewers in relation to the benchmark information where it is provided, and in the context of information which is provided in the background statement.

The disciplinary background statements (supplied at a 2-digit FoR level by each institution) are important contextual information for the RECs. These statements offer the institution an opportunity to explain their research focus, strengths and achievements. They provide assistance to the RECs in interpreting the data which has been submitted, and provide guidance for the focus of evaluation in the 2-digit domain.

The background statements provide a valuable tool for the RECs as they draw the picture of research quality within each discipline. Indicators are not weighted in ERA. However, if most of the research outputs submitted for a discipline consist of books and book chapters, and the background statements explain why in the context of a research discipline and research program, the REC is more likely to focus on books and book chapters, than other indicators such as journals, conference papers or research income. A background statement can also be used, for example, to explain why research commercialisation income is a stronger indicator of research quality than Category One Research Income for a specific discipline within a particular university.

Institutions can also use the background statements to highlight specific measures of esteem not included as a metric esteem measure, but which nevertheless highlight the quality of research within that institution, such as a Nobel Prize for Literature or a Prime Minister's History Award.

The ARC is also keen to see more attention paid to the definition of research for submissions of creative works in 2010. Researchers and universities have been reminded to consult the criteria when listing their research outputs for submission. The 2010 Submission Guidelines now also allow for the submission of Portfolios when submitting creative works as research outputs. This allows for a single research statement to accompany a body of creative work. For example, a set of poems can now be accompanied by one research statement, rather than having to explain the research component of each individual poem.

The discipline matrices were developed in consultation with the sector. They were released as a draft following advice from workshops of discipline experts, with the final matrices released after sector-wide consultation. Following sector-wide consultation (see for example the CHASS submission), some significant changes in the indicators being used to assess the Social, Behavioural and Economic Sciences (SBE) and Humanities and Creative Arts (HCA) clusters were made. These include:

• Citation Analysis has been removed as an indicator for all SBE disciplines except disciplines within Psychology and Cognitive Sciences.
• All SBE disciplines except disciplines within Psychology and Cognitive Sciences will now nominate 20% of valid research outputs for Peer Review.
• Research commercialisation income has been included as an applied indicator for Demography, Criminology and Anthropology.
• An editor of a prestigious work of reference has been included as an Esteem measure for all Commerce, Management, Tourism and Services disciplines.
• Non-traditional research outputs have been removed as a valid indicator for Social Work.
• Australia Council Grants and Fellowships have been included as esteem measures for all disciplines within Studies in Creative Arts and Writing.
• Registered Designs will not be used as an Applied indicator for Curatorial and Related Studies.
• An editorial role for A* or A journals will not be used as an Esteem indicator for the 2010 ERA process.
• Cites in Legal Judgements will not be used as an indicator for the 2010 ERA process.

Key ERA 2010 documents available on the ARC website include:

• The 2010 discipline matrices outlining the range of indicators being used to assess research quality within each discipline cluster;
• A full list of the disciplines covered in the cluster groups is on pages 69-75 of the 2010 ERA Submission Guidelines;
• The full journal rankings list, released in early February, available in Excel format..

The 2010 discipline matrices outlining the range of indicators being used to assess research quality within each discipline cluster is available on the ARC website.

A full list of the disciplines covered in the cluster groups is on pages 69-75 of the
2010 ERA Submission Guidelines     [PDF file size: 1MB].

The full journal rankings list was released in early February and is available in Excel format on the ARC website.

The ARC has advised CHASS that the full results of ERA 2010 will be made publically. This will allow us to draw a larger picture of research quality within institutions as well as at a discipline level.

Related links and background information

• December 2009: ERA: 2009 HCA Trial Outcomes
• September 2009: ERA - Indicators Consultation Paper summary
• September 2009: Indicators Consultation paper for ERA
  Australian Research Council media release
• August 2009: Humanities and Creative Arts: Recognising Esteem Factors and Non-Traditional Publication
  CHASS advocacy paper
• April 2009: A new era for Creative Arts research
  Professor Margaret Seares AO, previous chair of the Creative Arts Committee
• Cluster Two Research Evaluation Committee

Heidi Hutchison
15 February 2010

 

 

For more information, please contact:

Executive Director

Council of the Humanties, Arts and Social Sciences

Phone: +61 2 6249 1995

director@chass.org.au