Evaluation Criteria Checklist for Repositories and Scientific gateways
Repositories and science gateways are key resources for the neuroscience community, but users often have a hard time orienting themselves in the service landscape to find the best fit for their particular needs. On the other hand, service developers lack authoritative guidance to develop their platforms. INCF wants to help neuroscience researchers and students choose good services for their specific use cases; we also want to help service providers make good and future-proof decisions for setup and operations.
To this end, the INCF Infrastructure Committee has developed a set of recommendations and this checklist of associated criteria for choosing or setting up and running a repository or scientific gateway, intended for the neuroscience community, with a FAIR neuroscience perspective. The committee is aiming for an inclusive set of recommendations that can apply to a wide and diverse range of digital services, both repositories and science gateways, for data as well as software.
The criteria checklist is meant to be a living document, continuously updated. Please give us your feedback by emailing info@incf.org.
Classification & status
Provide a clear and concise description of the resource that outlines: resource features, targeted community, who backs the service
Why? To ensure users are aware of what the service can offer, who operates it, and who it is intended for.
Accessibility
Services should clearly communicate any conditions and costs for access and deposit.
Offer an open, well documented API and/or a command-line interface in several community relevant programming languages. Ideally, the API and/or CLI should also be open to community input.
Have methods reported in a structured format, a community relevant format if possible
Why? Programmatic or command-line access is vital for modern computational science to work seamlessly. Using established community standards, for both data and metadata saves users from having to reformat all their data, makes metadata ingestion easier to support and to automate, and results in clearer and more consistent naming.
User support & documentation
Providing user support is an essential criterium. We also recommend services to have specific resources to support new users, such as a FAQ or a Quick Start guide, and to make it possible for their user community to support itself via a forum or similar mechanism.
Another essential requirement is that a service has sufficient documentation. The importance of good documentation cannot be overstated, ideally that documentation will also be updated regularly and include community input.
Why? Documentation saves time, frustration and resources, and is needed to do reliable research.
Licensing, data use and permissions
Services must clearly state any and all conditions that apply to access, reuse and deposit.
It is essential that a service uses appropriate licensing. Licensing conditions of uploading data to the platform, or reusing data within the platform, need to be clearly and transparently articulated.
Preferably the service is using a well known and easy to understand license model (e.g. CC) at a clear and appropriate level of granularity.Any process that contributes data to the platform should clearly outline the license being applied or provide appropriate options. Licensing applied to derived data should be made clear.
Why? So that researchers can understand their rights and responsibilities. Clear licensing is also important to FAIR data practices.
Usage & statistics
Services should be transparent with their usage data, including usage statistics and citations. Ideally the past usage history is also available.
Why? The levels and patterns of usage and citation serve as proxies for harder to judge criteria such as community importance, community relevance and impact.
Community connection
Services should make their value contribution clear to their intended community, and ideally also include community feedback in decision making.
Why? To ensure researchers are aware of the service’s scientific contribution, and that community needs are served to the services’ best ability.
Data Quality & Curation (DQC): Reproducibility
We recommend that methods are reported in a structured, community relevant format if possible, and that metadata entry is made easy and automatically or semi-automatically verified.
Why? Community relevant formats save time and effort. Metadata are critically important to FAIR. They are the backbone of any dataset, and ongoing quality control of metadata is as important as the data.
Data Quality & Curation (DQC): Provenance
We recommend that provenance for data, derived data and software is documented and extractable.
Why? So that all research steps are documented, for later use in publication and to ensure reproducibility.
Data Quality & Curation (DQC): Curation
We recommend services to communicate and document their curation processes for data and metadata.
Why? Metadata are critically important to FAIR. They are the backbone of any dataset, and ongoing quality control of metadata is as important as the data. They are vital in ensuring that data can be correctly understood and effectively used and reused.
Responsibilities
Rights and responsibilities of both user and service should be articulated in a clear and transparent manner.
The service should be operated with best-practice IT practices, including user communications, backup, documentation, security controls and updates, privacy controls etc. It should also have a data preservation policy.
Why? Clear communication and transparency are essential for trust. It is critically important that researchers can correctly understand their rights and responsibilities.
Governance & transparency
Services need to be clear and transparent about the way they are governed.
They should transparently communicate the way decisions are made and provide user communities with a mechanism for influence.
Funding sources, or other contributions of value, should be transparently communicated, and conflicts of interest should be declared.
Why? Clear communication and transparency are essential for trust.
Sustainability & persistence
Services should develop a sustainability plan and publish it for users to review.
The sustainability plan should address shutdown and archiving matters such as archiving or data preservation.
Why? Transparence on sustainability is important to allow researchers to make informed decisions about what services they use and invest their resources in.