Skip to content

ML4SE-toolkit

ML4SE-toolkit provides small, composable primitives for turning source code into reproducible machine-learning examples for software engineering research.

pip install ML4SE-ToolKit

Guides

  • Comment extraction and cleaning: find comments in source files, strip their syntax, and feed them into model inputs. Start here for the quick start and API reference.
  • Comment cleaning policy: what cleaning removes and what it keeps.
  • Input formatting: turn matches into fill-in-the-middle, autoregressive, masked diffusion, masked LM, or span corruption model inputs, with sentinel and mask tokens picked from a model name or Hugging Face object.
  • Chunked datasets: run a function over every row of a dataset split into many files, such as The Stack, one file at a time, with restartable state on disk and any number of workers.
  • Architecture: the QueryMatch contract and extension points.
  • Git workflow: how dev and main relate.