The DLF-AIG Metadata Assessment Working Group (MWG) is pleased to announce the public release of benchmarks for metadata quality: https://dlfmetadataassessment.github.io/MetadataQualityBenchmarks/

This suite of pages includes the benchmarks as well as supporting documentation.

Benchmarks include minimal, suggested better-than-minimal, and ideal criteria for descriptive metadata quality, primarily related to cultural heritage and digital library materials. The expanded benchmarks outline metrics and examples to show how the benchmarks might be applied.

Supporting documentation is intended to supplement the benchmarks by providing [1] questions to answer before benchmarking, to help organizations think through how they want to approach these activities; [2] various quality metrics synthesized from a number of metadata frameworks that could be used by organizations to develop local better-than-minimal standards; and [3] citations and other reference materials on related topics for further reading.

About Benchmarking

Metadata quality conversations are often about assessment or evaluation, which is a critical component of reviewing metadata and (hopefully) making improvements. However, the purpose of benchmarks is to help answer “when is metadata ‘good enough’” or, “how good is [this] metadata compared to other metadata?” When used in combination with assessment activities, benchmarks can help to set specific goals to bring a set of metadata records to a particular level of quality — whether that is minimal (perhaps to be improved in the future), ideal (i.e., the best it can be with known information), or somewhere in between.

As part of the development process, the Benchmarks Sub-Group reviewed literature about benchmarking from other industries to get a clearer sense of how benchmarks function. One important component is that benchmarks serve as a shared reference point to define a specific level of quality. In other industries, benchmarks allow different companies to all point to the same criteria and metrics to identify whether they meet certain standards. The development of the benchmarks for metadata quality provides a similar reference point for descriptive metadata.

Many aspects of metadata quality are subjective or heavily dependent on local and community contexts. Even aspects of metadata that are more quantifiable (e.g., values are formatted as expected) rely on the requirements of the particular schema or environment where it lives. For this reason, the Benchmarks Sub-Group determined that trying to set iterative levels of quality would not be practical, as there is too much variation among both expectations of metadata and resources/capability to assess and correct records that do not meet particular standards. Instead, the benchmarks define what makes a record “absolute minimum” quality and “ideal/gold standard” quality, leaving all intermediary steps up to local organizations.

The Benchmarks Sub-Group

The sub-group formed in 2018 to start considering benchmarking in the context of descriptive metadata and the best ways to provide suggestions and support to organizations wanting to measure or evaluate the overall quality of metadata in their collections.

Initially, the group created a comprehensive survey asking questions about which metadata fields digital libraries are using, how they evaluate quality, and other contextual information. That survey had 151 complete or partial responses, providing significant amounts of data related to digital libraries during 2019, including size, hardware/software usage, metadata fields, and evaluation practices or needs. More information, along with the survey, data, and related publications is available on the MWG website: https://dlfmetadataassessment.github.io/projects/benchmarks/

Based on that data and other feedback from peer reviewers during iterations of the current documentation, the sub-group outlined specific benchmarking criteria and compiled supplementary information. Work related to benchmarking is intended to be ongoing. Although this marks the public release of the “final” project, the benchmarks may change over time based on feedback and there are tentative plans to add other supporting documentation or resources, depending on time and needs expressed by the community.

More Information

Links related to this project:

• Benchmarks main page: https://dlfmetadataassessment.github.io/MetadataQualityBenchmarks/
• To provide feedback on the benchmarks, see the GitHub repository: https://github.com/DLFMetadataAssessment/MetadataQualityBenchmarks/

Links related to the MWG:

• Other MWG projects and resources: https://dlfmetadataassessment.github.io/
• About the MWG, including how to join or learn more about our work: https://wiki.diglib.org/Assessment:Metadata