LeMaterial v1.0: LeMat-Bulk dataset release

LeMaterial v1.0 launches as an open-source initiative releasing the LeMat-Bulk dataset, which unifies 6.7M entries from Materials Project, Alexandria, and OQMD into a standardized format with seven properties to accelerate materials discovery.

Why LeMaterial?

LeMaterial addresses the fragmentation of major materials datasets by unifying and standardizing data from Materials Project, Alexandria, and OQMD into a high-quality resource with consistent and systematic properties.

The world of materials science, at the intersection of quantum chemistry and machine learning, is brimming with opportunity — from brighter LEDs, to electro-chemical batteries, more efficient photovoltaic cells and recyclable plastics, the applications are endless. By leveraging machine learning on large, structured datasets, researchers can perform high-throughput screening and testing of new materials at unprecedented scales, significantly accelerating the discovery cycle of novel compounds with desired properties. In this paradigm, data becomes the essential fuel powering ML models that can guide experiments, reduce costs, and unlock breakthroughs faster than ever before.

This field is fueled by very complete datasets such as Materials Project, Alexandria, and OQMD, all open-source and under a CC-BY-4.0 license. However, those datasets vary in format, parameters, and scope, presenting the following challenges:

Dataset integration issues (e.g. inconsistent formats or field definitions, incompatible calculations)
Biases in dataset composition (for e.g. Materials Project's focus on oxides and battery materials)
Limited scope (e.g. NOMADs focus on quantum chemistry calculations rather than material properties)
Lack of clear connections or identifiers between similar materials across different databases

This fragmented landscape makes it challenging for researchers in AI4Science and materials informatics to leverage existing data effectively. Whether the application involves training foundational ML models, constructing accurate phase diagrams, identifying novel materials or exploring the chemical space effectively, there is no simple solution. While efforts like Optimade standardize structural data, they don't address discrepancies in material properties or biases in dataset scopes.

LeMaterial addresses these challenges by unifying and standardizing data from three major databases — Materials Project, Alexandria and OQMD — into a high-quality resource with consistent and systematic properties.

Achieving a clean, unified and standardized dataset

LeMat-Bulk v1.0 provides a cleaned, merged, and standardized dataset with 6.7M entries and seven materials properties, using Optimade formatting, a hashing-based material fingerprint, and visualization tools.

LeMat-Bulk is more than a large-scale merged dataset with a permissive license (CC-BY-4.0). With its 6.7M entries with consistent properties, it represents a foundational step towards creating a curated and standardized open ecosystem for material science, designed to simplify research workflows and improve data quality.

Release	Description & Value	Date
v.1.0	* Data collection and merging from various sources: Materials Project, Alexandria and OQMD datasets, including multiple fields and DFT functionals (PBE, PBESol, SCAN). * Data cleaning: identify and remove datapoints not conforming to the set standard (e.g. with non-compatible calculations). * Standardization: uniformly format the set of fields across databases, using Optimade standard * Material fingerprint: attribute a unique identifier to each material with a well benchmarked hashing function, enabling to remove duplicates * Accessibility and visualization: the proposed new datasets can be visualized thanks to open source tools (phase diagrams, material explorer) based on Crystal Toolkit, Pymatgen, and Dash components developed by Materials Project	Dec. 10, 2024
v.1.1	* New data: calculate charge data for many materials in Materials Project database (additional 53k data points, band gaps information, unified energy correction scheme) * Evaluation: Code hashing function benchmarks * New functions: Material similarity metric and retriever tool of similar structures * New models: Equiformerv2 and FAENet trained on the data * Data validation: check for compatibility of fields, formatting, etc.	Q1 2025
Future releases	* New data: release of r2SCAN data from Materials Project * New surface datasets: OC20 & OC22 datasets and ability to cover other models than bulk (surface and slab+ads and molecule structures) * Further harmonisation: include trajectories across MPTrj, OMat24 in similar format, include relational data in our DB	Q2 2025

We offer different datasets and subsets, enabling tailored workflows for researchers depending on their needs (consistent calculations, deduplicating materials, or comprehensive exploration):

Compatibility: these subsets only provides calculations which are compatible to mix. This is available in 3 functionals today (PBE, PBESol and SCAN)
Non-compatible: this subset provides all materials not included in the compatibility subsets.
LeMat-BulkUnique: this dataset split provides de-duplicated material using our structure fingerprint algorithm. It is available in 3 subsets, for PBE, PBESol, and SCAN functionals.

Integrating a well-benchmarked materials fingerprint

LeMaterial introduces a hashing-based material fingerprint that enables rapid novelty detection, duplicate removal, and cross‑dataset linking.

Besides building this standardized dataset, one of the key contribution of LeMaterial is to propose a definition of a material fingerprint through a hashing function that assigns a unique identifier to each material.

Current approaches to identifying a material as novel relative to a database have predominantly relied on similarity metrics, which necessitate a combinatorial effort to screen the existing database for novelty. To provide faster novelty detection in a dataset, Entalpic introduces a hashing method to compute the fingerprint of a material.

Above is a breakdown of the fingerprinting. We use a bonding algorithm (e.g. EconNN) on the crystal structure to extract a graph, on which we then compute the Weisfeiler-Lehman algorithm to get a hash. This hash is combined with composition and space group information to create the material fingerprint.

Our fingerprinting approach offers several benefits:

Quickly identifying whether a material is novel or already catalogued.
Ensuring the dataset is free from duplicates and inconsistencies.
Allowing to connect materials between datasets.
Supporting more efficient calculations for thermodynamic properties, such as energy above the hull.

Below lies a comparison of our hash function with the StructureMatcher of Pymatgen, to find all duplicates of a dataset. The experiment was run on two datasets having very different structures.

Dataset	Number of structures	Task time for the hash function (parallelized on 12 CPUs)	Task time for StructureMatcher (parallelized on 64 CPUs)
Carbon-24	10,153	100 seconds	17 hours
MPTS-52	40,476	330 seconds	4.9 hours

Additionally, we are planning on releasing a set of well-curated benchmarks to evaluate the validity of our hashing function. For instance, we investigated:

If distinct materials lead to different hashes based on material identification tags across existing databases
Whether adding small noises or applying symmetry operations to a material leads to the same hash
If materials sharing the same hash, across or within databases, could indeed be the same material — with manual and DFT checks
How fast and accurate our hash is compared to Pymatgen's StructureMatcher on existing databases

**🤗 Call to the community:**our aim is not to position this fingerprint method as the single solution to de-duplicate materials databases and find novel materials, but rather to foster discussion around this question. One current limitation of this hashing technique is that it does not cover disordered structures; we would like to push the community towards finding a consensus, while proposing a relatively simple and efficient fingerprint method in the meantime.

LeMaterial in action: applications and impact

LeMaterial enables extended phase‑diagram exploration, cross‑database property comparison, rapid novelty detection, and training of ML interatomic potentials.

In the longer term, LeMaterial aims to be a community-driven initiative that gathers large & curated datasets, machine learning models, handy toolkits, etc. It is designed to be practical and flexible, enabling a wide range of applications, such as:

Exploring extended phase diagrams (Link to our phase diagram explorer, built thanks to various open-source tools from Materials Project), constructed with a broader dataset, to analyze chemical spaces in greater detail. Combining larger datasets means that we can provide a finer resolution of material stability in a given compositional space.
Compare materials properties across databases and functionals: by providing researchers with data across DFT functionals, and by linking materials via our materials fingerprint algorithm we are able to establish and connect materials properties calculated via different parameters. This gives researchers insight into how functionals might behave and differ across compositional space.
Determining if a material is novel. Our hashing function allows researchers to quickly assess whether a material is unique or a duplicate, streamlining the discovery process and avoiding redundant calculations.
- Example 1: Our fingerprint method identified the following Alexandria entries (agm002153972, agm002153975) as potentially being the same material — having the same hash. When we did a relaxation on the higher energy entry, the material relaxed to the lower energy configuration.
- Example 2: applying our hash to another dataset (AIRSS) that is often used in training generative models, we found the following materials with the same hash.
  
  It is important to note here that Pymatgen's StructureMatcher identified these two unit cells as different materials, when they are indeed the exact same structures. Here, our hashing algorithm was able to identify them as indeed the same.
Training predictive ML models. We can also train machine learning interatomic potentials like EquiformerV2 on LeMat-Bulk. These models should benefits from its scale and data quality and the removal of bias across compositional space, and it would be interesting to assess the benefits of this new dataset. An example of how to incorporate LeMaterial with Fairchem can be found in Colab. We are currently in process of training an EquiformerV2 model using this dataset — stay tuned 💫

Take-aways

LeMaterial offers a solution that unifies, standardizes, performs extra cleaning and validation on existing major data sources, accelerating research, improving ML model quality, and making materials discovery more efficient and accessible.

As a community, we often place a lot of value in the quality of these large-scale open-source databases. However the lack of standardization makes utilizing multiple dataset a huge challenge. LeMaterial offers a solution that unifies, standardizes, performs extra cleaning and validation efforts on existing major data sources. This new open-science project is designed to accelerate research, improve quality of ML models, and make materials discovery more efficient and accessible.

We are just getting started — we know there are still flaws and improvements to be made — and would thus love to hear your feedback! So please reach out if you are interested to contribute to this open-source initiative. We would be excited to continue expanding LeMaterial with new datasets, tools, and applications — alongside the community! ⚛️🤗

Citations

By downloading content from LeMaterial, you agree to accept the Creative Commons Attribution 4.0 license implying that content may be copied, distributed, transmitted, and adapted, without obtaining specific permission from LeMaterial, provided proper attribution is given to LeMaterial.

If you use the LeMaterial as a resource in your research, please cite the citation section from our data-card (paper to come).

CC-BY-4.0 (license used for Materials Project, Alexandria, OQMD) requires proper acknowledgement. Thus, if you use materials data which include (”mp-”) in the immutable_id, please cite the Materials Project. If you use materials data which include (”agm-”) in the immutable_id, please cite Alexandria, PBE or Alexandria PBESol, SCAN. If you use materials data which include (”oqmd-”) in the immutable_id, please cite OQMD. Finally, if you make use of the Phase Diagram for visualization purposes, or the crystal viewer in the Materials Explorer, please acknowledge Crystal Toolkit.

To learn more about LeMaterial and get involved:

Form to join the community
Hugging Face Space
Github