Jace

Jace is a work-in-progress machine learning framework for training neural networks models to analyze Magic: the Gathering drafts.

Disclaimer

Magic: The Gathering is a trademark of Wizards of the Coast LLC. This project is a personal project created for educational purposes, and is not affiliated with, endorsed by, or associated with Wizards of the Coast LLC in any way.

This project has been partially built using AI. While the project has been for a very very long time entirely designed and implemented by myself only (and most of the core framework still is), I have recently started maintaining it using AI tools. That being said, all text documentation (this README.md and EXPERIMENTS.md) is and will always be entirely written by myself (because I hate reading AI slop and I assume you do too).

Licenses

This project is released under MIT License, but relies on third-parties for data. Namely:

• 17lands public draft data, released under a Creative Commons Attribution 4.0 International License (link)
• Scryfall data API, which can be used by adhering to their guidelines (link)

Overview

Jace implements all base components required to train and evaluate models for draft analysis: data collection, data preprocessing modules, neural network models, training pipelines, etc. Jace is mostly built using JAX ecosystem. Jace also includes CLI tools to easily configure, train and fine tune models. Experiment configuration is based on Hydra-zen framework.

This project has been developed solely on and for Linux. While it may work on Windows or MacOS. I can't and won't provide any support for these platforms. Since most models implemented in this project are rather small, the use of a GPU is not required and CPU only jax library is installed by default. To use a different platform, you may replace jax dependency in pyproject.toml by the relevant jax[my_platform] (see https://docs.jax.dev/en/latest/installation.html#supported-platforms).

Project files are organized as follows:

• benchmarks/: pytest benchmarks
• data/: datasets, checkpoints, etc.
• images/: documentation images
• notebook/: jupyter notebooks
• scripts/: miscellaneous standalone preprocessing/analysis scripts (mostly for my personnal use)
• src/jace:
  • data/: data downloading, loading and preprocessing functions
  • experiments/: experiment scripts (see the Running experiments section and EXPERIMENTS.md)
  • hydra/: hydra based CLI for running experiments
  • models/: builtin models
  • training/: training loop
  • utils/: miscellaneous functions
• tasks/: just tool tasks
• tests/: pytest tests

Goals

• Provide flexible end-to-end training pipeline
• Support various MTG draft analysis tasks, such as:
  • Draft win rate estimation
  • Draft picking
  • Deck building
• Provide tools, metrics, etc. to analyze performance and fine tune models
• Implement state-of-the-art models
  • All models should be trainable in a reasonable amount of time on consumer grade hardware

If you are curious about what has already been achieved, I recommend you to check EXPERIMENTS.md.

Non-Goals

• Provide pre-trained model weights
• Interface with 3rd party tools
• Implement bots for you to cheat in your favorite game

Getting Started

Project dependencies are managed using uv. Assuming python and uv are correctly installed on your machine. You may create a virtual environment with all project dependencies installed with:

# Create a virtual environment (only once)
uv venv
# Synchronize virtual environment dependencies
uv sync
# Source the environment
source .venv/bin/activate

Most project common commands and scripts can be run using the make-like just tool. Run just with no argument to get a list of available commands.

Running experiments

All experiments implemented hereafter are documented in EXPERIMENTS.md. I strongly suggest you to read this one to understand the whys and hows.

Experiments may be run using the built-in Hydra CLI scripts found in src/jace/experiments/*. Scripts have multiple running modes, which may be specified with the mode=<name> argument:

• fit (default): train the pre-configured model
• validate: load model from checkpoint and evaluate it on the validation dataset
• test: load model from checkpoint and evaluate it on the test dataset (only available for experiments requiring specific test logic)
• autotune: run Optuna to find the best training hyperparameters.

All training parameters may be tuned either by editing the files directly, or through the Hydra CLI (see Hydra CLI doc).

Most scripts implement several variant configurations, which can be enabled by passing the <variant>=<name> argument to the script command. For example:

• python -m jace.experiments.winrate_estimation.mlp: train an MLP based deck win rate estimator.
• python -m jace.experiments.winrate_estimation.mlp scope=draft: variant using drafted cards instead of deck.
• python -m jace.experiments.winrate_estimation.mlp mode=autotune scope=draft: run Optuna to find the model best hyperparameters for the draft variant.

Passing the -h/--help flag to the command will display all available variants and the current configuration.

Tensorboard and optuna-dashboard are used for visualization. You can run them using the just command.

Related Projects and Research Papers

• @RyanSaxe's MTG Draft and DeckBuilder AI
• Predicting Human Card Selection in Magic: The Gathering with Contextual Preference Ranking (link)
• Learning With Generalised Card Representations for "Magic: The Gathering" (link)
• Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks (link)
• Multiset Transformer: Advancing Representation Learning in Persistence Diagrams (link)
• Deep Sets (link)
• Set Norm and Equivariant Skip Connections: Putting the Deep in Deep Sets (link)
• On permutation-invariant neural networks (link)