Council for the Humanities, Arts and Social Sciences

Measures of quality and impact of publicly funded research in the humanities, arts and social sciences

Dr Jonathan Powles
Associate Director (Academic)
Australian National University
1 November 2005

Executive summary

For decades, researchers in the humanities, arts and social sciences (HASS) have endured a funding system that assesses their work through a narrow prism of quantity: How many papers? How much grant money? How many research students?

This project set out to find a fairer way to evaluate research, by focusing on the quality and impact of the diverse outputs of HASS research, such as books, performances, reports to government, films, libretti and professional advice.

The consultation process was wide-ranging and intense, involving over two hundred submissions and consultations, a survey of literature and practice, case studies, and focus groups. The project examined the evidence from Australia and abroad, and looked for distinctly new approaches, including proposals being developed by science agencies. Draft proposals were tested across seven disciplinary areas in six different institutions. In the end, a new model for assessing the quality and impact of research emerged.

This model applies not only to the humanities, arts and social sciences, but can be applied with equal validity across the whole research spectrum. It has the flexibility to work with all disciplines and with interdisciplinary research.

The model has achieved an uncommon level of acceptance among participants in the inquiry. Its acceptability results largely from its familiarity: the model owes much to the university promotions system, which typically evaluates a case for quality and impact against evidence of academic achievement, community standing, professional outreach, and peer regard.

In all, the model meets five key criteria for acceptability by governmental, academic and other research stakeholders: simplicity, familiarity, reliability, flexibility and universality.

The model identifies three factors that may be adjusted to suit the disciplinary balance and focus of the research unit in question: research quality, research impact and research capability. The report emphasises the importance and inextricability of these three factors.

Quality

The report finds that any assessment of research quality is best achieved by peer review of an evidenced-based case, informed by key metrics. It finds that potential indicators of quality are broadly comparable between science and non-science sectors.

Impact

By contrast, the key indicators of the impact of research on other researchers and the broader community are diverse and highly discipline-specific. They need to be evaluated by a panel of experts drawn from academe and users and beneficiaries of publicly funded research.

Capability

The model introduces a further category, research capability, which reflects the capacity of a research unit to contribute to future goals of research and research training; that is, to ensure the vitality and diversity of Australian research in the years to come. The case for capability, like the case for impact, should be evaluated by a panel of experts and beneficiaries of research.

The clear conclusion from the report's survey of literature and practice is that standard bibliometric measures alone are not enough for such assessment. This position was supported by voluminous feedback from the sector, indicating quantitative data are best used as supporting evidence in a nuanced case.

The new model for research assessment is based on a case, supported by evidence, about which judgment is made by experts. Importantly, it introduces a menu of suitable tools for assessment, from which the research unit can choose in compiling its case. This flexibility enables the model to be used right across the research spectrum.

The project commends the new model, which affords Australia the opportunity to claim international best practice in assessing research quality, impact and capability.

 

Dr Jonathan Powles
01 November 2005

Download the full paper   [PDF file size: 3.88 MB]   REF: PAP20051101JP

Table of contents

    Executive summary

1   The research project

2   Findings

3   Valuing the humanities, the arts and the social sciences

4   A 'map' of research in the humanities, the arts and the social sciences

4.1   Origin

4.2   Production

4.3   Outputs

4.4   Outcomes

4.5   The humanities

4.6   The creative arts

4.7   The social sciences

5   Indicators: quality, impact and capability

5.1   Quality

5.2   Impact

5.3   Capability

5.4   Results from the case-study trial

6   The proposed model

6.1   Criteria

6.2   Assessment panels

6.3   Flexibility of weighting

6.4   Unit of assessment and reporting

6.5   Broader applicability of the model

7   Conclusion

Appendix A: Setting the scene

Al   Introduction

A2   Current measures of the quality and impact of publicly funded HASS research in Australia

A3   Current international measures of quality and impact of publicly funded HASS research

A4   Potential measures of quality and impact of publicly funded HASS

Appendix B: The value to the end-user

B1   Executive summary

B2   Summary of main responses to interview and questionnaire questions

B3   Multiple end-users, multiple values, and individual and community capability development: a case in point

B4   Conclusion

B5   Interview format

B6   Informants

Appendix C: The case-study trial

C 1   Organisation

C2   Outcomes

C3   Documentation for the trial process

C4   A postscript

Appendix D: Acknowledgments

Appendix E: Abbreviations and acronyms

 

For more information, please contact:

Toss Gascoigne

Executive Director

Council of the Humanties, Arts and Social Sciences

Phone: +61 2 6249 1995

director@chass.org.au