Ratchet tool-calling exam explorer
Every row of two exams, with the answers of the untrained model, the trained adapter and the teacher, so you can check any number yourself.
What these numbers are
- Strict scoring: an answer counts only if it is one JSON object with the right tool and the right arguments (extra text around the JSON fails). The BFCL exam uses an adapted BFCL-style checker.
- One decoder setting: temperature 0, thinking off, one answer per row.
- Two exams never seen in training: a 150-row hard exam (20 tools that never appear in training) and 300 rows taken from the Berkeley Function Calling Leaderboard.
- Synthetic training data: the adapter was trained on 2,940 synthetic examples (answers written by NVIDIA Nemotron 3 Ultra).
- Small hardware: a LoRA adapter on NVIDIA Nemotron 3 Nano 4B, trained on a borrowed GPU. Not affiliated with or endorsed by NVIDIA or Nebius.