"Initials" by "Florian Körner", licensed under "CC0 1.0". / Remix of the original. - Created with dicebear.comInitialsFlorian Körnerhttps://github.com/dicebear/dicebearMO
Machine Learning - Learning/Language Models manitcor • 1y ago • 100%

GitHub - wgryc/phasellm: Large language model evaluation and workflow framework from Phase AI.

github.com

Docs: https://phasellm.com/docs/phasellm/eval.html

This project provides a unified framework to test generative language models on a large number of different evaluation tasks.

Features:

  • 200+ tasks implemented. See the task-table for a complete list.
  • Support for models loaded via transformers (including quantization via AutoGPTQ), - GPT-NeoX, and Megatron-DeepSpeed, with a flexible tokenization-agnostic interface.
  • Support for commercial APIs including OpenAI, goose.ai, and TextSynth.
  • Support for evaluation on adapters (e.g. LoRa) supported in HuggingFace's PEFT library.
  • Evaluating with publicly available prompts ensures reproducibility and comparability between papers.
  • Task versioning to ensure reproducibility when tasks are updated.

1
0
Comments 0