Tutorial Notebooks

Four notebooks in notebooks/ cover the maintained workflows end to end:

• T1_finetune_density_example.ipynb: finetune a base model on your own property dataset, push it to the Hub, register it, and generate with it
• T2_load_and_generate.ipynb: generate structures with the released Hub models (courtesy of Joley Lin)
• T3_API_density_example.ipynb: predict density for a composition through the containerised API
• T4_SLME.ipynb: discover a material with a target photovoltaic efficiency

Customising the tokenizer

The HF-cif-tokenizer already contains everything needed to train and run the models. To add more tokens:

• Create the new vocab: Edit the create_vocab.py file to include all the new tokens you want (if augmenting CIF with new tokens for example). Add them at the end of the vocabulary, after the special tokens, so existing token ids do not change and released checkpoints can still be fine-tuned. Save a new vocabulary.json with the updated dictionary.
• Optional: Add Spacegroups: If new spacegroups are required for a particular study, these should be added to the spacegroups.txt file and to LATE_SPACE_GROUPS in create_vocab.py, as was done for P6 (also to avoid changing released checkpoint behavior).
• Build New Tokenizer: Once the new vocabulary is ready, run the save_tokenizer_to_hf.py script, to save it locally or to HF. Give it its own --path, since the default overwrites the tokenizer that ships with the package. Then point the pretrained_tokenizer_dir argument in the train config at your new tokenizer.