Ministral 3B: Complete Benchmark Performance Guide

mistral ministral-3b 2024-10-16 Mistral Research License

In-depth analysis of Ministral 3B performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendormistral
Versionministral-3b
Release Date2024-10-16
Context Window128000 tokens
Input Modalitiestext
Output Modalitiestext
LicenseMistral Research License
Documentationhttps://docs.mistral.ai/

Benchmark Performance

BenchmarkScoreUnitEvaluated AtNotesSource
MMLU42.4%2024-10-165-shotview
HUMANEVAL37.2pass@12024-10-16—view
GSM8K32.2%2024-10-160-shot CoTview
MATH10.1%2024-10-160-shot CoTview
BBH50.1%2024-10-163-shot CoTview
GPQA19%2024-10-160-shotview
IFEVAL47.8%2024-10-16prompt_strictview
ARC81.5%2024-10-16challengeview
MUSR29.4%2024-10-160-shotview
WINOGRANDE64.8%2024-10-160-shotview

Pricing

TierPriceCurrency
Input$0.04 / MtokUSD
Output$0.04 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://mistral.ai/technology/ · as of 2024-10-16

Compliance

  • Data Residency: EU
  • SOC2: ✓
  • HIPAA: ✗
  • GDPR: ✓
  • ISO 27001: ✓

Ministral 3B

Descripción del modelo

Mistral Ministral 3B 超轻量边缘模型, 128K 上下文, 3B 参数, 最具性价比的边缘部署模型。

Especificaciones principales

ProveedorVersiónFecha de lanzamientoVentana de contextoModalidades de entradaModalidades de salidaLicencia
Mistralministral-3b2024-10-16128KtexttextMistral Research License

Rendimiento en benchmarks

BenchmarkPuntuaciónUnidadNotas
MMLU (Massive Multitask Language Understanding)42.4%5-shot
HumanEval37.2pass@1—
GSM8K (Grade School Math 8K)32.2%0-shot CoT
MATH10.1%0-shot CoT
BBH (BIG-Bench Hard)50.1%3-shot CoT
GPQA19.0%0-shot
IFEval47.8%prompt_strict
ARC81.5%challenge
MUSR29.4%0-shot
WinoGrande64.8%0-shot

Precios

EntradaSalidaLectura cachéEscritura caché
————

por millón de tokens

Fortalezas

  • 上下文窗口 128K，支持长文本。

Debilidades

  • MMLU 仅 42.4，知识推理偏弱。
  • HumanEval 37.2，代码能力较弱。
  • 闭源专有模型，不支持自托管。

Casos de uso

  • 长文档摘要

Referencias