Skip to content

CLI entry points

The package provides command-line tools as Python modules rather than installed console scripts. Run them from a repository clone.

Each CLI module documents a representative Usage: command. Use argparse --help for the complete set of available options.

Load and generate

Load a pretrained model from the Hub or a local checkpoint and generate CIFs from explicit formulas or a parquet file of prompts.

_load_and_generate

Load a released CrystaLLM-pi model and generate CIF structures.

Prompts can come from a parquet dataset or reduced formulas. Formula mode can search over Z values and either stop at the first valid structure or select the highest-scoring candidate. Generation uses every visible GPU.

--scoring_mode supports LOGP and None. LOGP requires --target_valid_cifs to be greater than zero.

Usage 
python _load_and_generate.py --hf_model_path c-bone/CrystaLLM-pi_ft_alex_mp_20-text \
    --reduced_formula_list "TiO2,SiO2" --z_list "2,4" --output_cif_dir outputs/cifs

Training

Training is configuration-driven. A JSONC file provides the base configuration and command-line flags override individual values.

_train

Train or finetune a CrystaLLM-pi model, conditionally or unconditionally.

Conditional runs require condition_columns and an activate_conditionality model family. Unconditional runs use plain GPT-2. Under torchrun, rank 0 handles logging and checkpoint writes.

Usage 
python _train.py --config _config_files/training/conditional/density-example/mpdb-density-finetune_example.jsonc

Dataset preparation

Prepare datasets through the preprocessing pipeline.

mattergen

Download MatterGen's Alex-MP-20 or MP-20 as a symmetrised CIF parquet.

Source: https://github.com/microsoft/mattergen (data-release/). Alex-MP-20 keeps its train/val splits. MP-20 also keeps its test split.

Usage 
python _utils/_datasets/mattergen.py --dataset alex_mp_20 \
    --output_parquet alex_mp_20.parquet --num_workers 64

lemat

Download LeMat-BulkUnique PBE as a symmetrised CIF parquet.

Source: https://huggingface.co/datasets/LeMaterial/LeMat-BulkUnique. Uses a seeded random 5% validation split.

Usage 
python _utils/_datasets/lemat.py \
    --output_parquet lematerial.parquet --num_workers 64

chili

Download stratified CHILI-100K as a symmetrised CIF parquet.

Source: https://github.com/UlrikFriisJensen/CHILI. Uses the splits from c-bone/chili100k_strat. Test: 500 seen, 500 structurally novel and 500 compositionally novel structures. Validation: 1,500 seen structures. Novelty is measured against LeMaterial (Alex-MP-20 is in LeMaterial).

Split construction: https://github.com/C-Bone-UCL/CrystaLLM-pi-paper/blob/main/notebooks/X_XRD_chili100k.ipynb

Usage 
python _utils/_datasets/chili.py \
    --output_parquet chili100k.parquet --num_workers 64

crystallm

Extract and symmetrise CIFs from tarballs into a parquet dataset.

The CIF tarballs are converted to the parquet layout expected by the package. Split labels are derived from the tarball filenames after removing their extensions, so filenames determine the resulting split names.

From original repo: https://github.com/lantunes/CrystaLLM/blob/main/ARTIFACTS.md

Usage 
python _utils/_datasets/crystallm.py --input_tarballs train.tar.gz val.tar.gz \
    --output_parquet cifs.parquet --num_workers 8

deduplicate

Deduplicate CIF data to one entry per formula and space group.

For each (formula, space group) pair, the entry with the smallest volume per formula unit is retained. Rows with "N/A", zero, or negative values in the selected property columns can be removed before deduplication.

Usage 
python _utils/_preprocessing/deduplicate.py --input_parquet raw.parquet \
    --output_parquet dedup.parquet --property_columns density bandgap

cleaning

Clean and normalise CIF data before training.

The pipeline orders disordered structures, normalises property columns, and adds the atomic-properties block expected by the tokenizer. The resulting property scale must match the normalised representation used during model training.

Usage 
python _utils/_preprocessing/cleaning.py --input_parquet dedup.parquet \
    --output_parquet clean.parquet --property_columns density --num_workers 8

calculate_theor_xrd

Simulate powder XRD patterns from CIF structures and build XRD condition vectors.

The output uses the condition representation the XRD-conditioned models consume.

Usage 
python _utils/_preprocessing/calculate_theor_xrd.py \
    --input_parquet clean.parquet \
    --output_parquet with_xrd.parquet \
    --num_workers 8

save_dataset_to_hf

Convert a DataFrame into a Hugging Face dataset with train, validation, and test splits.

Splitting is material-based, so all rows for one material remain in the same split. --test_size 0.2 --valid_size 0.2 therefore produces 60/20/20 proportions, while setting both values to zero places all rows in the training split.

With --save_hub, the dataset is pushed to the Hub as private. Change its visibility in the dataset's settings on Hugging Face if needed.

Usage 
python _utils/_preprocessing/save_dataset_to_hf.py --input_parquet clean.parquet \
    --test_size 0.2 --valid_size 0.2 --save_hub c-bone/my-dataset

save_model_to_hf

Upload a loadable CrystaLLM-pi checkpoint to the Hugging Face Hub.

The upload includes the checkpoint, configuration, and tokenizer files required by load-and-generate, so the repository can be passed directly as --hf_model_path.

Usage 
python _utils/_preprocessing/save_model_to_hf.py --checkpoint model_ckpts/run/checkpoint-1500 \
    --repo c-bone/CrystaLLM-pi_my-model

save_tokenizer_to_hf

Save a custom CIF tokenizer locally and optionally upload it to the Hugging Face Hub.

The vocabulary and space-group files are written in the layout expected by CustomCIFTokenizer.from_pretrained.

Usage 
python _utils/_preprocessing/save_tokenizer_to_hf.py --vocab_file vocabulary.json \
    --spacegroups_file spacegroups.txt --path _utils/HF-cif-tokenizer

Generation and scoring

The generation CLIs are documented on the Generation page alongside the functions that define each stage of the pipeline.

The evaluation CLIs are documented on the Metrics and analysis page alongside the quantities they report.