Gemini 1.5 Flash-8B 002: Complete Benchmark Performance Guide

google 1.5-flash-8b-002 2024-10-03 Proprietary

In-depth analysis of Gemini 1.5 Flash-8B 002 performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendorgoogle
Version1.5-flash-8b-002
Release Date2024-10-03
Context Window1e+06 tokens
Input Modalitiestext, image, audio, video
Output Modalitiestext
LicenseProprietary
Documentationhttps://ai.google.dev/gemini-api/docs

Benchmark Performance

BenchmarkScoreUnitEvaluated AtNotesSource
MMLU77.5%2024-10-035-shotview
HUMANEVAL79.7pass@12024-10-03—view
GSM8K86.3%2024-10-030-shot CoTview
MATH43%2024-10-030-shot CoTview
BBH73.6%2024-10-033-shot CoTview
GPQA39.8%2024-10-030-shotview
IFEVAL74.3%2024-10-03prompt_strictview
ARC93.2%2024-10-03challengeview
MUSR54.7%2024-10-030-shotview
WINOGRANDE84.6%2024-10-030-shotview

Pricing

TierPriceCurrency
Input$0.0375 / MtokUSD
Output$0.15 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://ai.google.dev/pricing · as of 2024-10-03

Compliance

  • Data Residency: US
  • SOC2: ✓
  • HIPAA: ✗
  • GDPR: ✓
  • ISO 27001: ✓

Gemini 1.5 Flash-8B 002

Visão geral do modelo

Google Gemini 1.5 Flash-8B 002 生产版本, 1M 上下文, 最低成本多模态推理。

Especificações principais

FornecedorVersãoData de lançamentoJanela de contextoModalidades de entradaModalidades de saídaLicença
Google1.5-flash-8b-0022024-10-031000Ktext, image, audio, videotextProprietary

Desempenho em benchmarks

BenchmarkPontuaçãoUnidadeNotas
MMLU (Massive Multitask Language Understanding)77.5%5-shot
HumanEval79.7pass@1—
GSM8K (Grade School Math 8K)86.3%0-shot CoT
MATH43.0%0-shot CoT
BBH (BIG-Bench Hard)73.6%3-shot CoT
GPQA39.8%0-shot
IFEval74.3%prompt_strict
ARC93.2%challenge
MUSR54.7%0-shot
WinoGrande84.6%0-shot

Preços

EntradaSaídaLeitura de cacheEscrita de cache
————

por milhão de tokens

Pontos fortes

  • GSM8K 86.3, robust math reasoning.
  • 支持文本、图像、音频多模态输入。

Pontos fracos

  • 闭源专有模型，不支持自托管。

Casos de uso

  • 代码生成与调试
  • 视觉与图像理解

Referências