Use the vitals package with ellmer to evaluate and compare the accuracy of LLMs, including writing evals to test local models.

It may not deliver the same performance as a bare-metal setup, but it's good enough for most titles ...