It is very important to make sure people who have contributed to a paper, are given credit as authors. The understanding of the term impact varies considerably and as such the objectives of an impact assessment need to be thoroughly understood before evidence is collated. 0000342798 00000 n Evaluative Research: Definition, Methods & Types - Maze Despite many attempts to replace it, no alternative definition has . The RQF pioneered the case study approach to assessing research impact; however, with a change in government in 2007, this framework was never implemented in Australia, although it has since been taken up and adapted for the UK REF. Impact assessments raise concerns over the steer of research towards disciplines and topics in which impact is more easily evidenced and that provide economic impacts that could subsequently lead to a devaluation of blue skies research. 0000008591 00000 n It is perhaps worth noting that the expert panels, who assessed the pilot exercise for the REF, commented that the evidence provided by research institutes to demonstrate impact were a unique collection. RAND Europe, Capturing Research Impacts. At least, this is the function which it should perform for society. Describe and use several methods for finding previous research on a particular research idea or question. In terms of research impact, organizations and stakeholders may be interested in specific aspects of impact, dependent on their focus. Definitions of Performance Appraisal - By McGregor and Dale Beach . It can be seen from the panel guidance produced by HEFCE to illustrate impacts and evidence that it is expected that impact and evidence will vary according to discipline (REF2014 2012). In putting together evidence for the REF, impact can be attributed to a specific piece of research if it made a distinctive contribution (REF2014 2011a). SIAMPI has been used within the Netherlands Institute for health Services Research (SIAMPI n.d.). For example, following the discovery of a new potential drug, preclinical work is required, followed by Phase 1, 2, and 3 trials, and then regulatory approval is granted before the drug is used to deliver potential health benefits. What are the challenges associated with understanding and evaluating research impact? Key features of the adapted criteria . Impact has become the term of choice in the UK for research influence beyond academia. 2007). There has been a drive from the UK government through Higher Education Funding Council for England (HEFCE) and the Research Councils (HM Treasury 2004) to account for the spending of public money by demonstrating the value of research to tax payers, voters, and the public in terms of socio-economic benefits (European Science Foundation 2009), in effect, justifying this expenditure (Davies Nutley, and Walter 2005; Hanney and Gonzlez-Block 2011). The University and College Union (University and College Union 2011) organized a petition calling on the UK funding councils to withdraw the inclusion of impact assessment from the REF proposals once plans for the new assessment of university research were released. Systems need to be able to capture links between and evidence of the full pathway from research to impact, including knowledge exchange, outputs, outcomes, and interim impacts, to allow the route to impact to be traced. (2007:11-12), describes and explains the different types of value claim. A variety of types of indicators can be captured within systems; however, it is important that these are universally understood. To understand the method and routes by which research leads to impacts to maximize on the findings that come out of research and develop better ways of delivering impact. Here we outline a few of the most notable models that demonstrate the contrast in approaches available. For example, some of the key learnings from the evaluation of products and personnel often apply to the evaluation of programs and policies and vice versa. It incorporates both academic outputs and wider societal benefits (Donovan and Hanney 2011) to assess outcomes of health sciences research. To be considered for inclusion within the REF, impact must be underpinned by research that took place between 1 January 1993 and 31 December 2013, with impact occurring during an assessment window from 1 January 2008 to 31 July 2013. In 200910, the REF team conducted a pilot study for the REF involving 29 institutions, submitting case studies to one of five units of assessment (in clinical medicine, physics, earth systems and environmental sciences, social work and social policy, and English language and literature) (REF2014 2010). 0000007559 00000 n The Oxford English Dictionary defines impact as a Marked effect or influence, this is clearly a very broad definition. Figure 2 demonstrates the information that systems will need to capture and link. The RQF was developed to demonstrate and justify public expenditure on research, and as part of this framework, a pilot assessment was undertaken by the Australian Technology Network. Although based on the RQF, the REF did not adopt all of the suggestions held within, for example, the option of allowing research groups to opt out of impact assessment should the nature or stage of research deem it unsuitable (Donovan 2008). We suggest that developing systems that focus on recording impact information alone will not provide all that is required to link research to ensuing events and impacts, systems require the capacity to capture any interactions between researchers, the institution, and external stakeholders and link these with research findings and outputs or interim impacts to provide a network of data. << /Length 5 0 R /Filter /FlateDecode >> The first attempt globally to comprehensively capture the socio-economic impact of research across all disciplines was undertaken for the Australian Research Quality Framework (RQF), using a case study approach. 5. Findings from a Research Impact Pilot, Institutional Strategies for Capturing Socio-Economic Impact of Research, Journal of Higher Education Policy and Management, Introducing Productive Interactions in Social Impact Assessment, Measuring the Impact of Publicly Funded Research, Department of Education, Science and Training, Statement on the Research Excellence Framework Proposals, Handbook on the Theory and Practice of Program Evaluation, Policy and Practice Impacts of Research Funded by the Economic Social Research Council. It is desirable that the assignation of administrative tasks to researchers is limited, and therefore, to assist the tracking and collating of impact data, systems are being developed involving numerous projects and developments internationally, including Star Metrics in the USA, the ERC (European Research Council) Research Information System, and Lattes in Brazil (Lane 2010; Mugabushaka and Papazoglou 2012). We will focus attention towards generating results that enable boxes to be ticked rather than delivering real value for money and innovative research. Metrics have commonly been used as a measure of impact, for example, in terms of profit made, number of jobs provided, number of trained personnel recruited, number of visitors to an exhibition, number of items purchased, and so on. There is a distinction between academic impact understood as the intellectual contribution to ones field of study within academia and external socio-economic impact beyond academia. Aspects of impact, such as value of Intellectual Property, are currently recorded by universities in the UK through their Higher Education Business and Community Interaction Survey return to Higher Education Statistics Agency; however, as with other public and charitable sector organizations, showcasing impact is an important part of attracting and retaining donors and support (Kelly and McNicoll 2011). The first category includes approaches that promote invalid or incomplete findings (referred to as pseudoevaluations), while the other three include approaches that agree, more or less, with the definition (i.e., Questions and/or Methods- This database of evidence needs to establish both where impact can be directly attributed to a piece of research as well as various contributions to impact made during the pathway. As such research outputs, for example, knowledge generated and publications, can be translated into outcomes, for example, new products and services, and impacts or added value (Duryea et al. A collation of several indicators of impact may be enough to convince that an impact has taken place. 0000328114 00000 n It has been acknowledged that outstanding leaps forward in knowledge and understanding come from immersing in a background of intellectual thinking that one is able to see further by standing on the shoulders of giants. 2005). 2. Introduction, what is meant by impact? Evaluative research has many benefits, including identifying whether a product works as intended, and uncovering areas for improvement within your solution. It has been suggested that a major problem in arriving at a definition of evaluation is confusion with related terms such as measurement, Teresa Penfield, Matthew J. Baker, Rosa Scoble, Michael C. Wykes, Assessment, evaluations, and definitions of research impact: A review, Research Evaluation, Volume 23, Issue 1, January 2014, Pages 2132, https://doi.org/10.1093/reseval/rvt021. The REF will therefore assess three aspects of research: Research impact is assessed in two formats, first, through an impact template that describes the approach to enabling impact within a unit of assessment, and second, using impact case studies that describe the impact taking place following excellent research within a unit of assessment (REF2014 2011a). The Payback Framework is possibly the most widely used and adapted model for impact assessment (Wooding et al. This is a metric that has been used within the charitable sector (Berg and Mnsson 2011) and also features as evidence in the REF guidance for panel D (REF2014 2012). A key concern here is that we could find that universities which can afford to employ either consultants or impact administrators will generate the best case studies. The Oxford English Dictionary defines impact as a 'Marked effect or influence', this is clearly a very broad definition. On the societal impact of publicly funded Circular Bioeconomy research in Europe, Devices of evaluation: Institutionalization and impactIntroduction to the special issue, The rocky road to translational science: An analysis of Clinical and Translational Science Awards, The nexus between research impact and sustainability assessment: From stakeholders perspective. The Author 2013. The reasoning behind the move towards assessing research impact is undoubtedly complex, involving both political and socio-economic factors, but, nevertheless, we can differentiate between four primary purposes. Given that the type of impact we might expect varies according to research discipline, impact-specific challenges present us with the problem that an evaluation mechanism may not fairly compare impact between research disciplines. 2006; Nason et al. A university which fails in this respect has no reason for existence. The following decisions may be made with the aid of evaluation. Replicated from (Hughes and Martin 2012). Evaluating an Author's Point of View - Study.com This distinction is not so clear in impact assessments outside of the UK, where academic outputs and socio-economic impacts are often viewed as one, to give an overall assessment of value and change created through research. To allow comparisons between institutions, identifying a comprehensive taxonomy of impact, and the evidence for it, that can be used universally is seen to be very valuable.