Groq

Leverage Groq’s LPU inference engine for blisteringly fast responses.

Overview

Groq operates using an LPU™ (Language Processing Unit) architecture, specifically engineered to deliver real-time AI capabilities with latency far below traditional GPU deployments.

Unmatched Speed

Hundreds of tokens per second for popular models like Llama 3.

Open Source Models

First-class support for Llama, Mixtral, and Gemma model variants.

Configuration

1. Get your API Key

   Sign up on the GroqCloud Console.

2. Environment Settings

   Add the credentials to your environment:

   .env

   GROQ_API_KEY="gsk_..."
   DEFAULT_MODEL="llama3-70b-8192"
   

Because of Groq’s incredible generation speed, it is highly recommended for streaming applications (e.g. voice or real-time chatbots).