Appendix I: Sequence of Steps for Peer Review of a Grant Application
Appendix IV: Example Scoring for Merit Review
The purpose of this manual is to provide information on CIHR's objectives, governance and policies; to outline the roles and responsibilities of peer review committee members evaluating grant applications; and to define the policies and procedures for peer review of grant applications. CIHR depends upon peer review for the evaluation of research grant applications, to ensure that leading-edge research is supported and that CIHR maintains a strategic national and international health research presence.
This manual is addressed primarily to committee members, but is also of use to applicants in explaining the peer review process for grant applications from submission to final judgment. Policies and Procedures regarding the assessment of Awards applications can be found on the CIHR Website. For detailed regulations concerning all aspects of CIHR funding programs, please see the Grants and Awards Guide.
The mandate of the Canadian Institutes of Health Research (CIHR) is as follows:
"To excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened health care system."
The purpose of peer review is to ensure excellence in the research funded by CIHR. The peer review system also ensures accountability, not only to the Government of Canada and the Canadian taxpayer - the source of CIHR funding - but to the research community at large. Peer review is carried out by committees of experts (peer review committees) that encompass all four pillars of health research (Biomedical, Clinical, Health Systems and Services, and Population and Public Health).
CIHR funds research through both open and strategic competitions. Open competitions accept proposals in any area of health research. Strategic competitions are sponsored by one or more of CIHR's Institutes or Branches or other funding partners and applications are solicited in specified areas of health research. In general, the same policies and procedures are followed for both types of competitions, unless otherwise specified in the funding opportunity. A list of current and archived CIHR funding opportunities can be found on the CIHR Funding Opportunities Database.
Peer review is overseen by CIHR's Scientific Council (SC), which governs all aspects of research-related decision making. SC provides scientific leadership and advice to Governing Council (GC) on health research and knowledge translation (KT) priorities and strategies, and recommends investment strategies in accordance with CIHR's 5-year Strategic Plan. The approval of funding opportunities for all research and knowledge translation initiatives is an integral part of SC's responsibilities.
The approval of funding opportunities for all research and knowledge translation initiatives is an integral part of SC's responsibilities.
Integrity of the peer review process depends on well established principles of confidentiality. All information contained in applications, reviewer reports, Scientific Officer notes and committee discussions is strictly confidential. The applications and any discussions thereof may not be used for any purpose beyond that for which they were originally intended. Committee members must not discuss with applicants or reviewers any information relating to the review of a specific application outside of the committee meeting. Applicants must not contact committee members, including the Chair and Scientific Officer, regarding the status of their applications (ratings, rank within committee, etc.). All requests for information on an application or a reviewer report should be referred to CIHR Program Delivery staff responsible for the committee in question. Please see the website for CIHR's policy on Conflict of Interest and Confidentiality in the context of Merit, Relevance and Peer review (CCIP).
By law, applicants have access to their own application files. Therefore, all written material used in evaluating an application is made available to the applicants when they are notified of CIHR's decision. The identity of the reviewers will not be revealed to the applicants under any circumstances. However, a list of peer review committee members will be published on the CIHR website 60 days after the Scientific Council approves funding for a competition cycle.
All materials related to the review process provided to peer review committee members must be stored in a secure manner to prevent unauthorized access. They must be transmitted using secure carriers and technologies. When they are no longer required, all material related to peer review must be destroyed using a secure method or returned to CIHR for destruction. Any loss or theft of materials related to peer review must be reported to your committee coordinator immediately. Please see the Guide on Handling Documents Used in Peer Review for further details.
CIHR must make every effort to ensure not only that its decisions are fair and objective, but also that they are seen to be so. According to CIHR's CCIP policy, any otherwise eligible reviewer may be considered for membership on a review committee unless he/she:
All committee members (Chair, Scientific Officer, reviewers, etc.) are subject to the same conflict of interest guidelines. CIHR staff and the Chair are responsible for resolving areas of uncertainty during the committee meeting.
All committee members reviewers must read and agree to abide by the CCIP policy prior to viewing any application information. This task is performed electronically (on ResearchNet) or by using a form provided by CIHR.
Success of the peer review system is critically dependent upon the willingness and ability of all committee members to be fair and reasonable; to exercise rigorous scientific judgment; and to understand, and take into account in a balanced way, the particular context of each application. Reviews are provided to the applicant without prior editing by CIHR staff, and CIHR does not take responsibility for their content. An applicant will not accept that your review is fair if it contains comments that could be construed as sarcastic, flippant, arrogant, or inappropriate in any way. Conversely, a constructive review, which includes helping the applicant by pointing out deficiencies that could be repaired in a resubmission, will help to convince a disappointed applicant that you provided a fair assessment of the proposal.
As stated in the CIHR Act, one of the ways CIHR fulfills its mandate is by "pursuing opportunities and providing support for the participation of Canadian scientists in international collaborations and partnerships in health research." As a result, CIHR accepts applications for research to be carried out in, or in collaboration with applicants based in, other countries. The international nature of the research should not be a factor in the scientific assessment of the proposal, beyond how it relates to the feasibility of the proposed research and the quality of the research question. Reviewers should also not be influenced by the funding obtained or requested for the international components when recommending a budget for the Canadian component(s). For detailed information on applying for funding with an international partnership component, please see the subsections titled Global Health Research and International Collaborations in the Grants and Awards Guide.
Knowledge translation is integral to CIHR's mandate and falls into two main categories, end of grant KT and integrated KT. With both categories of knowledge translation CIHR expects researchers to disseminate their findings and facilitate their translation into improved health, more effective products or services, and/or a strengthened healthcare system. Note that the costs of dissemination are eligible expenditures in all CIHR grants.
For end of grant KT, many means of dissemination exist and the onus is on the researcher to select the most appropriate vehicle for the intended knowledge-user audience to ensure maximum impact. When the primary knowledge users are researchers, dissemination of results through the publication of articles in high quality and accessible journals is appropriate, although other strategies that increase awareness of the results and facilitate their application may also be appropriate. When knowledge-user audiences outside the research community should be informed of specific research findings, dissemination plans with more ambitious goals and comprehensive strategies are expected. With integrated KT, stakeholders or potential research knowledge users are engaged in the entire research process and the research is directed at producing solutions to issues or problems the stakeholders/knowledge users have identified. Further information is available on the KT Portfolio Website.
Applicants and peer reviewers are reminded that the CIHR Policy on Access to Research Outputs applies to all new and renewed grants awarded after January 1, 2008. Grant recipients must ensure that all published peer-reviewed articles are freely available online within six months, where possible under publisher agreements. Authors can adhere with the policy by archiving peer-reviewed manuscripts in an open access repository (e.g. institutional repository) or by publishing in an open access journal. Please consult the policy web site for more detail.
The Ethics Office of the CIHR, in conjunction with the Institute of Aboriginal Peoples' Health, has prepared guidelines to assist researchers and institutions in carrying out ethical and culturally competent research involving Aboriginal people. The intent is to promote health through research that is in keeping with Aboriginal values and traditions. These guidelines will assist in developing research partnerships that will facilitate and encourage mutually beneficial and culturally competent research and will promote ethics review that enables and facilitates rather than suppresses or obstructs research. Further information can be found on the CIHR Guidelines for Health Research Involving Aboriginal People webpage.
Applicants are encouraged to demonstrate the use of gender and sex-based analysis in applications. Gender and sex-based analysis is an approach to research which systematically inquires about biological (sex-based) and sociocultural (gender-based) differences between women and men, boys and girls, without presuming that any such differences exist. The purpose of this line of inquiry is to promote rigorous health research which expands understanding of health determinants in both sexes and results in improvements in health and health care. For further information, please see the Gender and Sex-Based Analysis in Health Research: A Guide for CIHR Researchers and Reviewers webpage.
Federal agencies are required to take positive measures to ensure the support and recognition of minority language communities in Canada. For CIHR, this means an obligation to promote health research in these communities. For further information, please see the Official Language Minority Communities (OLMC) webpage. Research proposals in these areas should still be subject to the same rigorous peer review process as any other application. However, the justification for promoting health research in minority language communities should not be a factor in the assessment.
An important evaluation criterion in all grant programs is the excellence of the applicant(s). A key factor in assessing this criterion is the productivity of the applicant(s), as determined by the quality and impact of contributions to the field. When assessing the quality of publications, peer review committees should focus on the quality of a publication's content and not simply the number of publications nor the quality or impact factor of journals. In the case of multi-authored publications or other collaborative work, applicants are advised to describe their contribution and reviewers should assess the specific contribution of the applicant to the work.
CIHR funds researchers in many health-related areas, and the forms of research publications can vary greatly among disciplines. In addition to the more traditional peer-reviewed journals, health researchers also publish in books, monographs, memoirs or special papers, review articles, conference/symposia proceedings and abstracts, government publications, etc. Some fast-moving research fields, such as some areas of computing science, genetics or microelectronics, use special means to reach the target audience quickly. Communications, quick-print reports, letters and electronic distribution of pre-prints are important vehicles for disseminating research results. All such contributions should be treated equally when assessing quality and impact, and reviewers should not regard certain types as "second class" or "grey literature."
When assessing productivity, reviewers should also be sensitive to legitimate delays in research and dissemination of research results. Some circumstances make it impossible or undesirable for researchers to publish important results of their research prior to applying for CIHR support. For instance, the time required to complete a monograph may exceed the time available between consecutive applications, or the protection of intellectual property may require a delay in publication. Research productivity may also vary as a result of personal circumstances, such as pregnancy or early child care, administrative leave, disability, elder care, etc., whether or not a formal leave of absence is taken. Applicants are advised to clearly and fully describe any circumstances that affect the dissemination of research results in their application. Peer review committees must be sensitive to the impact of these circumstances on the level of productivity, while ensuring that the quality of the research remains competitive.
A typical CIHR grants peer review committee consists of a Chair, Scientific Officer, peer reviewers, CIHR staff and other specialized roles depending on the funding opportunity. Individual committee members are selected for their research excellence, as reflected by their ability to obtain continued extramural peer-reviewed grant support, and for their breadth of knowledge and maturity of judgment. Please see the website for peer review membership guidelines. Committees as a whole should also satisfy the need to cover the range of research areas for which the committee is responsible, to appropriately represent the Canadian health research community as a whole, to review in both official languages, and to allow for the logistics of conflict of interest and turnover of committee members. Please see the website for the procedure for selection of peer review committee members.
CIHR grant competitions can be held on a recurring or an ad hoc basis. For recurring competitions, standing peer review committees are formed, and committee members are recruited for a term of service (typically three years) in order to ensure consistency and continuity in the review process. Standing committee membership may be supplemented by additional members as required for a specific competition, based on the applications received and expertise needed for their review. For ad hoc competitions, committees are formed to review applications for that particular competition and then disbanded
CIHR staff are typically represented by a Deputy Director and a Program Delivery Coordinator, who are responsible for ensuring the integrity and quality of the peer review process. CIHR staff:
The committee Chair is directly responsible to CIHR for ensuring that the peer review committee functions smoothly, effectively and objectively, according to CIHR's policies. He/she establishes a positive, constructive, fair-minded environment in which the research proposals are to be evaluated. The Chair (and the Scientific Officer) fulfills an oversight role and does not rate applications before the committee. His/her responsibilities include:
For further information, please see Instructions for Grants Committee Members.
In addition to the duties shared with the Chair, as described above, the Scientific Officer:
Under exceptional circumstances, CIHR staff may assume the role of Scientific Officer during the committee meeting.
For further information, please see Instructions for Grants Committee Members.
CIHR uses the terms "internal" and "external" reviewer to distinguish reviewer roles, as described below:
Internal reviewers are committee members who attend the peer review committee meeting, normally in person but occasionally by teleconference. Applications are assigned to a minimum of two internal reviewers for assessment. Internal reviewers are typically assigned up to eight applications that they review in depth and they submit a written review that is provided to the applicant after the committee meeting. Internal reviewers present their review at the peer review committee meeting and lead the review of applications assigned to them (see also Section 6.2.3). They also participate in the discussion and rating of all other applications before the committee for which they are not in conflict. For further information, please see Instructions for Grants Committee Members.
Internal reviewers are also assigned other applications as "readers". Readers are responsible only for reading an application and are not required to submit a written review; they serve as a discussant in the committee and aid in reaching a consensus rating.
On occasion, a reviewer with a very specific expertise may be called on to review a small number of applications, typically by teleconference. These reviewers only take part in the discussion of the application(s) they have been assigned, and they rate the application(s) by e-mail to maintain confidentiality.
For certain funding opportunities, other specialized roles may also be required during peer review (see also Section 6.2.5, Merit Review):
In certain cases, a review may be solicited from someone who is not a member of the peer review committee, in order to fill a gap in expertise. The external reviewer provides a written assessment but does not attend the meeting. They may also provide an initial rating of the application but this is not used in the calculation of the final rating. For further information, please see Instructions for Grants Committee Members.
The Community Reviewer is an individual who is not currently an academic or researcher, but who has a demonstrated interest in health and science. The Community Reviewer provides a mechanism for ensuring good communication to public stakeholders and transparency of the peer review process. He/she does not rate applications but comments on the lay abstract of the application, specifically the extent to which the intent and importance of the proposed research is well explained and in a language clear to members of the general public. He/she provides written comments on all lay abstracts submitted and selects 5-10 to discuss at the peer review committee meeting in order to highlight strong and weak examples. They are also invited to provide feedback on the proceedings of the committee, such as the quality, quantity and variety of science reviewed, the structure of the discussions, the objectivity of the discussions, and any other general comments. For more information, see the Community Reviewers: Involving the Canadian Public in the CIHR Peer Review Committees and the Instructions for Grants Committee Members webpages.
Occasionally, individuals are permitted to observe peer review committee meetings. Observers are typically CIHR Institute staff or representatives from partner organizations who have no funding decision-making authority for that competition. Observers must adhere to the same CCIP Policy as all committee members, and they do not contribute in any way to the review process or discussions surrounding the applications (including any discussions that arise during breaks). Observers may not remove any notes or other information related to the review of applications they observe from the meeting room.
For some competitions, a review of applications received is conducted to ensure that their objectives are aligned with those of the funding program. Depending on the program, relevance review may be carried out on a "letter of intent" (a statement of relevance and/or a summary of the application) or on a full application. Relevance review may be carried out by Institutes, portfolios or funding partners.
All eligible applications received by the appropriate deadline date (posted in the competition announcement) are entered into the competition. Applications must be complete at the time of submission, otherwise they are withdrawn from the competition. Specific exceptions to this rule can be found in the funding opportunity descriptions.
Within two weeks after the deadline, the Chairs and Scientific Officers of the peer review committees together with CIHR staff review the applications assigned to their committee. Together, they are responsible for ensuring their committees are equipped with the appropriate expertise and, upon accepting an application for review by their committee, accept responsibility for ensuring that the committee performs a fair review. In some cases, applications may not meet a particular committee's mandate and may need to be reassigned (if the competition has more than one peer review committee). A maximal load for a committee should be not more than 60 applications for a 3-day meeting with no member assigned to review more than 10 (and preferably 8) applications (not including the applications for which he/she is assigned to be reader). The final authority for the assignment of applications to a peer review committee rests with CIHR.
After the list of applications is compiled, committee members are given access to the application summaries to declare any conflicts of interest and indicate their level of expertise. Please note that CIHR now carries out application submission, peer review and the posting of decisions for the majority of its funding programs on ResearchNet, a secure online portal. For updates to the ResearchNet system, please see the Funding Notices webpage. Information regarding activities for programs not supported by ResearchNet can be obtained through your committee coordinator.
Chairs, Scientific Officers and CIHR staff then assign the applications to committee members (two internal reviewers and a reader). External reviews can also be solicited if required, as noted above. All committee members excluding external reviewers are then given access to the full applications assigned to their committee four to six weeks before the peer review committee meeting. External reviewers see only those applications assigned to them. Committee members are able to view the external reviewer reports before the peer review committee meeting, as they become available.
The internal reviewers' reports begin with a brief summary of the research objectives. However, it is still the responsibility of all peer review committee members to familiarize themselves in advance of the meeting with all applications to be assessed by their committee, as well as with any available external reviewer reports.
In advance of the meeting, internal reviewers are required to complete the following tasks on ResearchNet (for programs using electronic review):
The deadline for uploading your reviews to ResearchNet, along with your assessments of overall quality and initial ratings, is one week before the meeting date. Reviews can be saved as drafts, by selecting "save draft copy" on ResearchNet prior to submission. In order to access the other reviews for the applications you were assigned, which would allow you to be better prepared for the discussion at the committee meeting, you must select "submit final review" on ResearchNet. As well, only once you have submitted your review will it be accessible to the Chair and SO who typically review these in advance of the meeting. Once you submit your final review, you will no longer be able to access it on ResearchNet. Please contact your committee coordinator if there was any error during submission. After the peer review committee meeting, reviewers may edit the content of their reviews based on new information gained during discussion. If you wish to revise your reviews after your committee meeting, please contact your program delivery coordinator.
The prime responsibilities of a peer review committee are to evaluate applications submitted for a particular competition, to rate them so that they may be ranked in order of priority, and to recommend a budget sufficient to support the proposed research if the application is approved. It is important that committees follow defined procedures in order to function in a consistent manner. For a summary of the review procedures for various grant competitions, please see the attached appendices.
Any committee member who has a conflict of interest with an application (as defined in Section 3.2, above) must not take part in the discussion of that application. For face-to-face meetings, committee members in conflict must leave the room before the application is discussed. The Chair and CIHR staff are responsible for monitoring conflicts and for resolving areas of uncertainty.
6.2.1 Attendance at the Committee Meeting
Committee meetings are held usually within four months of the application deadline date and last not more than three days. The effective and fair review of applications depends on all committee members participating for the full duration of the committee meeting. Therefore, if your meeting starts in the morning, please arrive the night before instead of traveling the morning of the meeting, to avoid travel delays. In addition, please avoid planning an early departure to return home at the end of the meeting, in case the meeting runs late on the last day. If the meeting ends early, CIHR will cover certain costs to change your travel plans if necessary. Your committee coordinator can provide further details and help you plan your itinerary accordingly.
In order to allow reviewers to devote more time to the consideration of applications that have the highest probability of being funded, discussion of non-competitive applications may be restricted. Most funding programs at CIHR now use "streamlining" to accomplish this. Assessment of each application at peer review committee meetings begins with internal reviewers announcing their initial ratings, to one decimal place. An application is then streamlined if it meets the following three conditions:
Other funding programs may use different standards to vary the amount of discussion overall, according to their strategic objectives.
If an application is not discussed, the applicant will receive a copy of all internal reviewers' reports (and external reviews, if any) and the Scientific Officer notes will only carry notification of the decision to streamline. Committee members do not vote on the rating; it is calculated as the mean of the initial ratings of the two internal reviewers.
If an application is not streamlined, the discussion proceeds as follows:
After the rating of the application, the Community Reviewer (if present) provides comments on the lay abstract if chosen as a highlight for the committee. The Community Reviewer addresses the extent to which the intent and importance of the proposed research is well explained and in a language clear to members of the general public.
Applications to certain funding opportunities are evaluated by merit review, which requires that both the scientific merit and potential impact of the projects be assessed using separate scores. Merit review is used for research programs that engage knowledge-users throughout the research process to inform the research plan, carry out the project, and apply the findings. In general, the potential impact score reflects the relevance/importance of the project to knowledge-users and the likelihood that the project will have a substantive and sustainable impact on health outcomes, practice, programs and/or policy in the study context. Therefore, while the overall procedure for rating an application is as above, there are two separate scoring components instead of one and the internal reviewer makeup differs (see Section 5.4.1). The sequence of steps for the review of applications is given for two examples of programs that use merit review, PHSI and CBR-HIV/AIDS in Appendices II and III. An example of the scoring of an application by merit review is given in Appendix IV. Please see the relevant funding opportunity details for further information on peer review and evaluation.
The appropriateness of the budget and the term of support are discussed, and recommendations are made (see below). Please see Section 8 for guidelines. Note that questions about the budget should not influence the rating of the application, unless they bear directly on the scientific merit.
Any concerns in the following areas should be discussed and, if necessary, flagged for CIHR staff to address. These issues are not to be considered as criteria for evaluation, except as they may impact on the scientific quality of the application. For detailed regulations concerning these issues, please see the Grants and Awards Guide.
Once all applications have been reviewed, if the peer review committee feels that any application(s) has been treated inconsistently, re-review of one or a small number of applications is permitted. Any committee member with a conflict of interest must again leave the room. Following discussion, a consensus rating is determined by the two internal reviewers and voting proceeds as before. The committee does not review the overall rankings of all applications at the end of the meeting as individuals with conflicts of interest would inevitably be present.
An important component of any peer review committee meeting is the final review of the committee's effectiveness and functioning, and a discussion of policy issues that may have arisen in the course of its deliberations. This discussion provides an opportunity for CIHR staff to address any concerns of the committee members and for staff to record feedback on the peer review process as part of CIHR's ongoing efforts to maintain an effective and high quality peer review system.
6.2.9 Newsworthy and Highly Rated Applications
At the conclusion of the peer review committee meeting, the committee is asked to identify those applications that are most newsworthy, highly rated, and/or likely to create public interest. As part of CIHR's mandate to engage the Canadian public and report to Parliament on its research initiatives, the Communications and Marketing department will use these suggestions to develop news stories and backgrounders for parliamentarians, key Ministers and decision-makers. In addition, the information gathered can be used for corporate publications and other activities highlighting CIHR's research efforts with key stakeholders.
As soon as possible after the peer review committee meetings, CIHR staff generates a funding proposal based on committee recommendations, to be reviewed by the Chief Scientific Officer and the Chief Financial Officer. The CSO and CFO consider the funding recommendation in light of criteria established by SC and submit their recommendations to SC for final approval. A list of successful applicants is posted on the Funding Decisions Notifications webpage as soon as it is available.
Applicants are informed of the results of the competition as follows:
Applications which have been flagged for Special Attention (see Section 6.2.7) are withheld as "pending". The applicant will be notified if further information is required. The additional information may be discussed by CIHR staff and peer review committee members if necessary prior to a final decision regarding funding.
For standing competitions like the Open Operating Grant Program (OOGP), applications may be new proposals, renewals of previously funded projects, or resubmissions of an unsuccessful new or renewal application. All application types are evaluated together "on a level playing field" and the same criteria and funding cut-offs are applied to all, though peer review committee members are reminded to take the stage of career and progress made during previously funded grants into account and to vary the emphasis placed on track record and training experience appropriately. Where applicable, resubmissions may contain a two page response to previous reviews of the applications. This response, along with corresponding modifications to the research plan, is expected to improve the quality of the application. However, committee members are not provided with copies of previous reviews and must still rate resubmissions relative to other applications in the competition.
In some cases, the same peer review committee may review applications for more than one funding opportunity. For example, an OOGP committee may also review applications for a strategic funding opportunity (also called a Request for Applications, or RFA). At the conclusion of the committee meeting, these applications will be separated into their own overall ranking lists and funding decisions will be made based on the funding envelopes provided through their respective programs. Therefore, the presence of these applications will have no impact on the funding of other applications being reviewed by a committee.
The exact criteria for each funding opportunity are described in the funding opportunity details. All applications for Operating, Catalyst, Teams and Emerging Teams grants are evaluated against the five criteria indicated below. These five criteria are further elaborated and defined by specific factors to be considered under each criterion, although it should be noted that not all factors are necessarily relevant to every grant application. The criteria are not listed in order of importance: the relative weighting of each criterion will vary depending on the objectives of the funding opportunity, as defined by CIHR. The review of the application should be structured to justify the overall rating based on the relevant criteria.
Criterion #1: Research Approach
Criterion #2: Originality of the Proposal
Criterion #3: Applicant(s)
Criterion #4: Environment for the Research
Criterion #5: Impact of the Research
Additional factors to be considered under each criterion may also be described in the funding opportunity details. Please contact your committee coordinator if you need further guidance on how to apply the individual criteria.
The review should be clear and concise, using objective and non-inflammatory language, and include justification. Constructive advice to the applicant will allow him/her to improve the quality and efficiency of the proposed research. The applicant will receive the review as it is submitted by the reviewer. For this reason, please do not identify yourself in order to ensure the confidentiality of the review process.
Criteria to assess the scientific merit of an application are as described above in Section 7.2. The relative weighting of these criteria depends on the program objectives as described in the funding opportunity description; if in doubt, please contact the Deputy Director responsible for the committee.
To ensure consistency, all reviewers must adhere to a common scale. It is particularly important that the full scale be used and the same conventions applied to assigned ratings. To facilitate this, the following scale and descriptors should be used:
| Descriptor* | Range** | Outcome |
|---|---|---|
| Outstanding | 4.5 - 4.9 | May Be Funded - Will be Discussed by the Committee |
| Excellent | 4.0 - 4.4 | |
| Very good | 3.5 - 3.9 | |
| Acceptable, but low priority | 3.0 - 3.4 | Not Fundable - May or May Not be Discussed by the Committee |
| Needs revision | 2.5 - 2.9 | |
| Needs major revision | 2.0 - 2.4 | |
| Seriously flawed | 1.0 - 1.9 | |
| Rejected | 0.0 - 0.9 |
*Only applications rated 3.5 or higher are eligible for CIHR funding. The range 3.0 to 3.4 should be used for applications which, while technically and conceptually acceptable, are not considered to be a high priority for CIHR funding, perhaps because the topic is not considered relevant to an important health issue, or because the work proposed seems unlikely to yield major advances in knowledge, or because the approach is not particularly innovative. Please note that applications rated 3.0 to 3.4 are not eligible for CIHR funds, including those from partnership programs, and might not be discussed by the committee; however, applicants are encouraged to re-apply after addressing the reviews. Applications rated below 3.0 are so flawed in some respect that they do not represent a good investment of public funds, and would require significant rewriting to be considered acceptable. Such applications will normally be streamlined, and not be discussed by the committee.
**In the committee meetings, reviewers assign scores to one decimal place, but the final average rating is calculated to two decimal places.
Merit Review (Section 6.2.5) employs separate scores for Potential Impact and Scientific Merit. The following scale should be used for programs assessed by Merit Review:
| Potential Impact* | Range** | Scientific Merit | |
|---|---|---|---|
| May Be Funded | Enormous | 4.5 - 4.9 | Outstanding |
| Extremely Significant | 4.0 - 4.4 | Excellent | |
| Very Significant | 3.5 - 3.9 | Very good | |
| Not Fundable | Significant | 3.0 - 3.4 | Acceptable, but low priority |
| Moderate | 2.5 - 2.9 | Needs revision | |
| Limited | 2.0 - 2.4 | Needs major revision | |
| Negligible | 0.0 - 1.9 | Seriously flawed |
*Only those applications that exceed the threshold rating of 3.5 on both Potential Impact and Scientific Merit will be considered for funding. The mean of the two scores will be calculated for the applications with a rating of 3.5 and above on both criteria to determine the final rating and establish a ranking list.
**In the committee meetings, reviewers assign scores to one decimal place, but the final average rating is calculated to two decimal places.
CIHR's objective is to provide the funds needed to allow approved research to be carried out effectively. To ensure the highest level of accountability in the process, it is critical that reviewers give the budget justification a full and thorough review so that funds are distributed as effectively as possible. The appropriate budget is very much a matter for judgment by the peer review committee. Some areas of research are more expensive than others. In addition, for a potentially fundable application, committees may recommend funds for only those parts of the proposed research deemed worthy on scientific grounds.
Peer review committees should use a zero-based approach to determine the funds required for the research thought worthy of support. In other words, the budget for each year must be built "from the ground up" with each line item justified. A percentage change from current funding is not an appropriate rationale for a budget. Committees should not make their budget recommendations in the context of perceptions of CIHR's budget; the Scientific Council itself will modify the amount for each grant if it feels it is necessary.
The sum of research staff, trainees, and materials and supplies gives a total annual operating budget recommendation. For some types of research (e.g., biomedical laboratory research) this sum is usually rolled over from year to year, while other projects (e.g., clinical trials, epidemiological studies etc.) may require differing amounts of support in different years. In the latter case, the recommended amounts, by year, should be specified.
If the peer review committee feels that the budget is not adequately justified or explained in order to assess the request appropriately, the committee may request a follow-up by CIHR staff. If the proposal is approved for funding, funds will not be released until the budget justification concerns are resolved.
Consideration of the budget should include the following factors:
Research staff (research associates, research assistants, technicians, etc.) should be determined by the actual needs for the techniques and work required for the research. The salary scales put forward by the institution should be followed, especially if the positions of people already employed are to be continued. A starting salary should not be substituted for the higher salary of a named incumbent with a record of continued employment with the applicant. Some institutions require non-discretionary benefits packages for staff (e.g. supplementary medical and dental insurances). These are considered eligible expenses on grants and can be requested as part of operating grant budgets. Salaries for applicants (Principal Investigators or Co-Investigators) cannot be paid from the grant or any other CIHR grant, except in the case of research associates and trainees, when the salary or stipend should be addressed in the budget justification.
Graduate students may be hired as research personnel on a grant. In general this is on a part-time basis, i.e., hourly. This situation is to be distinguished from a graduate student receiving a stipend from a grant (see below), in which case the work done is part of the training of the student and constitutes the thesis or comparable academic requirement.
The assessment of requests for support of trainees should take into account the quality of the training environment. The committee's judgment of the quality of training will be influenced by such factors as the rating given the project, the nature of the research program and the project(s) on which the trainee(s) might work, the track record of the applicant in training young researchers, and the environment within which the applicant is working. It is helpful, but not essential, that the applicant has identified the project to which the trainee(s) will be assigned.
Stipends requested from CIHR for trainees must observe the current guidelines, which can be found in the Tri-Agency (CIHR, NSERC & SSHRC) Financial Administration Guide, Use of Grant Funds, Compensation-Related Expenses. These amounts are both the minimum to be paid to a trainee and the maximum that may be requested from a CIHR grant. Support for health professionals varies according to their years of experience and their licensure in Canada; see also the Tri-Agency (CIHR, NSERC & SSHRC) Financial Administration Guide, Use of Grant Funds, Compensation-Related Expenses. Non-discretionary benefit packages for trainees are not eligible expenses on grants and can not be requested as part of operating grant budgets. The payment of tuition fees is also not an eligible expense on a grant.
A budget should be established for materials and supplies, consumables, animal costs, etc., to include all non-personnel requirements for funds. To facilitate budget revisions, committees may wish to build these totals by categories (e.g., animals, isotopes, chemicals or other reagents, glassware, computer software packages, payments to subjects, access to databases, data analysis, printing, publication costs, travel for research personnel or research subjects). Alternatively, committees may wish to make blanket estimates of the usual operating and travel costs per research worker (which will vary according to the type of work being done), though special costs (e.g., travel for collaboration, or unusual animal care or maintenance costs) should not be forgotten.
In general, CIHR grant funds may be used to cover only the direct costs of research and may not be used for indirect costs, e.g., library, heat and light, office furniture, regular telephone rates etc. (see the Tri-Agency (CIHR, NSERC & SSHRC) Financial Administration Guide for further details). If a budget request includes amounts for what may seem to be indirect costs or "overhead," these need to be justified in terms of their direct contribution to the research. For example, a contribution to the salary of a glassware washer or technician to operate a common piece of research equipment is allowable, while a departmental "tax" to cover costs other than research expenses (e.g., library acquisitions, graduate student stipends, secretarial pool etc.) is not allowable. The latter should normally be covered by the institution. When in doubt, such costs should be flagged for CIHR staff to follow up.
Depending on the funding opportunity, equipment costs may be requested for identified items required to do the research. Funding for equipment should be requested in the first year (regardless of what year it is required) and will be awarded as a one time payment during the first year of the grant.
To facilitate the ability of researchers to obtain the necessary supplies for their research, CIHR has revised its definition of "equipment" effective for application deadlines of August 1, 2009 and later:
Research Equipment: Any item (or interrelated collection of items comprising a system) of nonexpendable tangible property, having a useful life of more than 1 year and a cost of $2,000 or more, which is used wholly or in part for research.
Note that all three conditions must be met for an item to be considered equipment, i.e.:
To be considered materials or supplies an item must meet only one of the following conditions:
For example, a laptop computer that costs less than $2,000 would be considered as materials or supplies even though it is a nonexpendable tangible item with a useful life of more than one year.
A cost quotation must be provided for equipment or service contracts greater than $10,000. Two competitive quotes as well as letters from an appropriate institutional official documenting the availability and status of similar equipment are required for items costing more than $25,000.
Peer review committees are asked to recommend budgets for grant applications irrespective of other sources of funds received or applied for, i.e., committees should not reduce recommended budgets to take into account potential overlap.
The term of support should be a direct reflection of the amount of time the peer review committee feels is necessary to complete the proposed work, if funded. Reviewers should be particularly mindful of the fact that it takes some time for a new investigator to build up a research program and momentum can be jeopardized by having to apply for renewal within a short time after having received a first grant. If, on its merits, a potentially fundable application requires a longer term to fulfill its objectives, it is inappropriate to limit the term to "hedge" against a new investigator's inexperience. In addition, committees may be reluctant to provide long term support for an application for which feasibility of the entire program depends on a positive outcome to initial experiments. In these cases, a 1 or 2 year term may be appropriate for an application that has a high element of risk, such that more substantive proof can be obtained, but it is inappropriate to limit the term of support simply because a proposal is untested or innovative.
|
1. Members in conflict leave the room. The two internal reviewers announce their initial rating. Note: ratings can be different from those previously posted on ResearchNet |
8. Consensus rating by internal reviewers:
If a consensus cannot be reached, use mean of internal reviewers' ratings. | |
|
2. Assessment of overall quality:
Committee members will not vote and no budget discussions will occur. The rating is calculated as the mean of the rating of the two internal reviewers. No SO notes will be taken. |
9. Individual ratings:
| |
|
10. Community Reviewer (if present):
| ||
|
3. Internal Reviewers:
|
11. Budget: Not required if application is not discussed or consensus rating is less than or equal to 2.9 following discussion.
| |
|
4. External Reviewers (if applicable):
| ||
| 5. Reader raises additional issues | 12. Term of grant | |
|
6. Discussion of application should focus on:
|
13. Issues to be flagged:
| |
|
7. Scientific Officer reads SO notes to the committee:
|
14. Scientific Officer reads final notes (including budget comments) for review / modifications / additions by committee | |
| 1. Members in conflict leave the room. The two internal reviewers announce their two ratings: one each for the Potential Impact of the research and the Scientific Merit of the proposal. |
7. Consensus rating by internal reviewers:
If consensus cannot be reached, use mean of internals' ratings. |
|
2. Triage: (OPTIONAL):
Committee members will not vote and no budget discussions will occur. The rating is calculated as the mean of the rating of the two internal reviewers. No budget recommended and no SO notes are taken. |
8. Individual ratings:
|
|
3. Internal reviewers:
|
9. Budget:
|
| 4. Reader raises additional issues. | 10. Term of grant |
|
5. Discussion of application should focus on:
|
11. Issues to be flagged:
|
|
6. Scientific Officer reads SO notes to the committee:
|
12. Scientific Officer reads final notes (including budget comments) for review / modifications / additions by committee |
| 1. Members in conflict leave the room. The two reviewers* announce their two initial ratings: one for the Potential Impact (PI) and one for the Scientific Merit (SM) of the proposal. |
7. Consensus rating for both PI and SM is reached by the two reviewers by:
If a consensus score cannot be reached, the mean of the two initial ratings will be used. |
|
2. Triage: (OPTIONAL)
Committee members will not vote and no budget discussions will occur. The final ratings are calculated as the mean of the four respective scores (PI and SM) initially announced by the primary and secondary reviewers. No SO notes will be taken. |
8. Individual ratings:
|
3. Internal Reviewers
|
9. Budget (grant applications only):
|
| 4. Readers* raise additional issues. | 10. Term of grant: |
|
5. Discussion of application should focus on:
|
11. Issues to be flagged:
|
|
6. Scientific Officer reads SO notes to the committee:
|
12. Scientific Officer reads final notes (including budget comments) for review / modifications / additions by committee. |
*Each application will be reviewed by two reviewers-one primary and one secondary-and two readers. One reviewer and one reader will focus on the assessment of the Potential Impact (PI) of the project, whereas the other reviewer and reader will focus on the assessment of the Scientific Merit (SM). Readers will act as discussants and do not need to provide a written review.
| Potential Impact | Scientific Merit | ||
|---|---|---|---|
| Initial rating (e.g. 3.4) 1st reviewer |
Initial rating (e.g. 3.8) 2nd reviewer |
Initial rating (e.g. 3.6) 1st reviewer |
Initial rating (e.g. 4.0) 2nd reviewer |
| Consensus rating (e.g. 3.5) | Consensus rating (e.g. 3.7) | ||
| Final rating * (e.g. 3.45) | Final rating * (e.g. 3.82) | ||
| Ranking rating ** (e.g. 3.63) | |||
* Average of confidential votes-calculated by CIHR database after the meeting.
** Average of the two final ratings-used to establish the ranking of applications for a particular competition. Please note that to be considered for funding both the potential impact and the scientific merit ratings have to be above 3.5. In this example, the application would not be considered for funding even though its ranking rating is above 3.5.