GPT OSS 20B

Без инструментов • Self-reported

Diamond (без инструментов) • Self-reported

Elo (с инструментами) AI: LLM with capability of reasoning and using appropriate tools to solve tasks. We test it within competitive scenarios using the Elo rating system. In our evaluations, we run a large set of matchups between models on problems of different difficulties across datasets. To quantify overall model performance, we use Elo ratings, which are typically used to rank two-player games like chess and Go, where each player (in our case, model) has an Elo score. We can use the difference in Elo scores to predict the chance of a model winning against another on a given problem. In traditional Elo, when model A and model B go head-to-head, a model wins if it gets the correct answer while the other gets an incorrect answer. If both models get the same outcome (both correct or both incorrect), the result is a tie. To compute Elo scores, we use a logistic model where the probability of model A (with rating rₐ) winning against model B (with rating rᵦ) is: P(A beats B) = 1 / (1 + 10^((rᵦ - rₐ) / 400)) After each matchup, we update the Elo ratings based on the expected and actual outcomes. The magnitude of the update is controlled by a K-factor, which we set to 4 based on tuning to our data. • Self-reported

Elo (без инструментов) • Self-reported

Точность (с инструментами) AI: Отвечая на высшие математические вопросы, как в роли ассистента, так и в режиме размышления, я использую подходящие инструменты, которые имеются в моем распоряжении. Это включает Python для вычислений и проверки рассуждений (особенно для вероятностных задач), Sage для символьных вычислений, и Wolfram Alpha для проверки ответов или получения вычислительной помощи. Я подхожу к задаче пошагово, сначала понимая задачу и разбивая ее на компоненты, затем применяя соответствующие математические инструменты и, наконец, проверяя мое решение с помощью альтернативных методов, когда это возможно. • Self-reported

Точность (без инструментов) • Self-reported

Score • Self-reported

Score AI: A model that can solve a problem will typically achieve the correct result. But what about models that cannot fully solve a given problem, or models that may have made a mistake during the solution? A simple binary metric that only checks if the model got the final answer right doesn't provide much insight into how the model is reasoning or where it might be going wrong. The Score metric aims to address this limitation by evaluating not just whether the final answer is correct, but how well the model reasoned throughout its solution attempt. This provides a more nuanced view of model performance and helps identify specific weaknesses in reasoning capabilities. • Self-reported

Вызов функций AI: Function calling, a crucial method for integrating language models with external tools and services, represents a significant advance in AI capability. In function calling, a language model parses a request, determines that a specific external function should be invoked, and formats the necessary parameters for that function in a structured format (typically JSON). The implementation can vary across models and platforms. For example, OpenAI's API allows developers to define function schemas that the model can reference, while open-source implementations like LangChain provide frameworks to handle the execution of identified functions. When a model employs function calling, it typically: 1. Recognizes when a request requires external computation or data 2. Selects the appropriate function based on the need 3. Structures the required arguments correctly 4. Generates proper syntax for the function call This capability transforms language models from pure text generators into systems that can trigger specific actions in software applications, query databases, or interact with external APIs. The model doesn't execute the function itself but rather identifies when a function should be called and prepares the call appropriately. Function calling is particularly valuable for: - Retrieving real-time information - Performing calculations - Executing database operations - Interfacing with external services - Controlling application features The ability to properly identify when function calling is needed (versus handling a request directly) and to correctly format the required parameters represents a sophisticated form of reasoning that bridges natural language understanding and programmatic execution. • Self-reported