4.2 KiB
highlight_image, nav_order, description, has_children
| highlight_image | nav_order | description | has_children |
|---|---|---|---|
| /assets/leaderboard.jpg | 950 | Quantitative benchmarks of LLM code editing skill. | true |
Dev LLM Leaderboards
Dev works best with LLMs which are good at editing code, not just good at writing code. To evaluate an LLM's editing skill, dev uses benchmarks that assess a model's ability to consistently follow the system prompt to successfully edit code.
The leaderboards report the results from a number of popular LLMs. While dev can connect to almost any LLM, it works best with models that score well on the benchmarks.
Polyglot leaderboard
Dev's polyglot benchmark asks the LLM to edit source files to complete 225 coding exercises from Exercism. It contains exercises in many popular programming languages: C++, Go, Java, JavaScript, Python and Rust. The 225 exercises were purposely selected to be the hardest that Exercism offered in those languages, to provide a strong coding challenge to LLMs.
This benchmark measures the LLM's coding ability in popular languages, and whether it can write new code that integrates into existing code. The model also has to successfully apply all its changes to the source file without human intervention.
{% assign edit_sorted = site.data.polyglot_leaderboard | sort: 'pass_rate_2' | reverse %} {% for row in edit_sorted %} {% endfor %}| Model | Percent completed correctly | Percent using correct edit format | Command | Edit format | Total Cost |
|---|---|---|---|---|---|
| {{ row.model }} | {{ row.pass_rate_2 }}% | {{ row.percent_cases_well_formed }}% | {{ row.command }} |
{{ row.edit_format }} | {% if row.total_cost == 0 %}?{% else %}${{ row.total_cost | times: 1.0 | round: 2 }}{% endif %} |
Dev polyglot benchmark results
<script src="https://unpkg.com/patternomaly/dist/patternomaly.js"></script> <script src="https://cdn.jsdelivr.net/npm/chart.js"></script> <script> {% assign data_source = edit_sorted %} {% assign pass_rate_field = "pass_rate_2" %} {% assign highlight_model = "xxxxxxxxxxx" %} {% include leaderboard.js %} </script> <style> tr.selected { color: #0056b3; } table { table-layout: fixed; } td, th { word-wrap: break-word; overflow-wrap: break-word; } td:nth-child(3), td:nth-child(4) { font-size: 12px; } </style>By Paul Gauthier, last updated