from The Council for the Humanities, Arts and Social Sciences

Professor Malcolm Gllies
CHASS President

4 October 2005

The Council for the Humanities, Arts and Social Sciences (CHASS) is an advocacy body representing the interests of those involved in research, education and practice in the humanities, arts and social sciences. It was formed in 2004 and 139 organisations and groups have joined as members. The objectives of CHASS are to promote the interests of the humanities, arts and social sciences in Australia. Specifically, the aims of CHASS are to:

• represent the interests of the sector;
• promote the contribution of the sector to government, industry and the public;
• provide a forum for discussion between the humanities, arts and social sciences sectors in Australia; and
• build up the innovative capacity of Australia, through better linkages between this sector, and science, engineering and industry.

In responding to The Preferred Model paper, CHASS was able to draw upon a soon-to-be published report - Measures of Quality and Impact in publicly funded research in the Humanities, Arts and Social Sciences. This report enabled CHASS to consult widely on many of the aspects considered in the RQF process.

The Preferred Model shows promise, but ultimately it has failed at the chance to make the changes that Australian research needs. The seeds of its failure lie in its conservatism, its acceptance of past record as a sure-fire predictor of future success, and its half-hearted embrace of the impact factor. In its present form it has not taken advantage of a rare opportunity to re-think the way the research and impact are considered in Australia, and represents something of a missed opportunity.

Our detailed comments are appended below.

1. The RQF Preferred Model represents a good start. CHASS welcomes the inclusive nature of the paper and recognises the attempts that have been made to include all types of research, including the humanities, arts and social sciences. That in itself is a welcome change to Australia's approach to the evaluation of research.
2. Despite this, the Preferred Model accepts too many of the premises of current Australian practices in measuring and evaluating research. Perhaps the most disappointing aspect is the fact that the Model could have placed Australia at the forefront of innovation in research assessment. No existing process genuinely attempts to come to grips with the vital issue of research 'impact'. While some notice is taken of this factor, it is regarded in the Model as very much a junior partner to 'quality'.
   
   The Model relies strongly on the past performance of research institutions, at a time when universities are expected to be looking to the future. It lacks any element which rewards forward-looking strategies. A system which funds only what has succeeded in the past is doomed, ultimately, to atrophy. CHASS dealt with this issue in its paper "Measures of Quality and Impact in publicly-funded research in the humanities, arts and social sciences"¹. This paper introduced the idea of 'capability': "the capacity of a research unit to contribute to future goals of research and research training, ensuring the vitality and diversity of Australian research."
   
   The paper put the proposition that an assessment of capability should include:
   
   • an examination of strategies in place to enhance the capability of a research unit;
   • the number of graduate and post-graduates the unit produces; and
   • an assessment of the skills they add to the workforce.
3. It is difficult to assess the detail of the Model as it stands, because two crucial pieces of information are lacking:
   • the effects of quality and opportunity factors; and
   • the levels of the dollar rewards to be given to the various disciplinary areas and various disciplinary bands have not been announced
4. Despite these reservations about the current proposal, CHASS is able to suggest three refinements and improvements to the Model as it stands:
   
   
   1. The context statement should be a compulsory part of the process for units under assessment. (section 2.2).
      
      The CHASS paper proposed that "the model for research assessment should be based around a case, supported by evidence, about which judgment is made by experts. The case put by a research unit should be constructed around quality, impact and capability. The case will be assessed in a twostage process: first by academics with specific disciplinary expertise, and then by a conglomerate panel of experts including academic peers, experts from industry and experienced research managers."²
      
      The assessment process set out in the CHASS model was tested by running a series of trials in which research units prepared a case. In these trials, carried out in seven disciplines and five universities, the assessors examining the cases saw the context statement as a vital coordinating element in the portfolio of evidence of quality and impact.³ It should not, in our view, become an optional activity.
      
      
   2. CHASS supports the definition of 'impact' in the Preferred Model, but this factor needs to be represented more strongly in the final evaluation. In the matrix as published (see Figure 5, p. 18), the degree to which impact can affect the final standing of a research unit is very limited; to such an extent that in two of the five bands it does not matter what the impact factor is because it cannot affect the result at all.
      
      Impact is currently set at three levels: limited, moderate and high. Effectively, this is two levels because 'limited' and 'moderate' impact have the same result on the overall score.
      
      At the very least, the matrix should be changed to allow 'impact' to have a more significant effect on the overall score of a research unit. CHASS, however, proposes a different approach to the relative weightings of impact and quality. It would allow the research unit being assessed to select how much emphasis it wanted to give to quality, and how much to impact. In this alternative model, the research unit would be able to nominate what percentage of the assessment should be made on the basis of 'quality', between a range of 50 to 80 per cent; and what percentage should be given over to 'impact', between a range of 20 to 50 per cent.
      
      Consideration also needs to be given to the way impact is measured. "Impact should be measured by a panel of experts drawn from academe and users and beneficiaries of research who examine an evidence-based case prepared by the research unit. The key indicators of the impact of research upon other researchers and upon the broader community are diverse and highly discipline-specific. The assessment of impact should be qualitative, and construed differently from the assessment of quality.
      
      "Specifically, the assessment of impact should be conducted under an "audit" model. Institutions, research groups or departments can compile an evidence-based case as to the impact of their research. Appropriately qualified experts - both academic and, crucially, experts from industry, public institutions and other research "end-users" - should then audit some of these cases to determine whether in fact the evidence presented supports the case made."⁴
      
      
   3. The Model nominates the establishment of 12 panels based on RFCD codes to consider assessments. The effect of this number of panels would be to perpetuate the silo effect in research in Australia, and make the assessment of collaborative and multi-disciplinary research difficult. We would suggest four or perhaps five panels: fewer panels with a broader remit, and breaking down the fine distinctions between the RFCD codes. There should be a place within these broad panels for the advice of those expert in a particular field to be heard.
      
      Indeed, it is hard to see the value of keeping RFCD codes as an authoritative mechanism for sorting research by discipline. Given that institutions will be able to nominate which panel assesses its research, complete verbal descriptions rather than codes would perhaps achieve the same end with greater usability and flexibility.

The next step

CHASS is willing to contribute to the development and refinement of the RQF process. We note in the Preferred Model Paper the identification of the need for "considerable additional work" on the matters of detail, and wish to indicate our capacity and readiness to be part of this process.

 

Malcom Gillies
4 October 2005

 

1 Measures of Quality and Impact in publicly-funded research in the humanities, arts and social sciences.

2 Ibid.

3 Ibid. The process is described in the extracts of the draft report currently posted to the CHASS web site. The assessors' reports will be included in the published version of the report.

4 Ibid.

 

Download this submission   [PDF file size: 28.44 kB]   REF: SUB20051004MG

 

For more information, please contact:

Toss Gascoigne

Executive Director

Council of the Humanties, Arts and Social Sciences

Phone: +61 2 6249 1995

director@chass.org.au