Implementation:Togethercomputer Together python Evaluation Types

Knowledge Sources	Together Python
Domains	Evaluation, Type_System
Last Updated	2026-02-15 16:00 GMT

Overview

Concrete type definitions for LLM evaluation requests and responses provided by the Together Python SDK.

Description

This module defines Pydantic models for the evaluation system including EvaluationType (classify/score/compare), EvaluationStatus (pending through completed/error), parameter models for each evaluation type (ClassifyParameters, ScoreParameters, CompareParameters), judge configuration (JudgeModelConfig), and response models (EvaluationCreateResponse, EvaluationJob, EvaluationStatusResponse).

Usage

Import these types when you need to type-hint evaluation-related data structures or inspect evaluation response objects.

Code Reference

Source Location

• Repository: Together Python
• File: src/together/types/evaluation.py
• Lines: 1-93

Signature

class EvaluationType(str, Enum):
    CLASSIFY = "classify"
    SCORE = "score"
    COMPARE = "compare"

class EvaluationStatus(str, Enum):
    PENDING = "pending"
    QUEUED = "queued"
    RUNNING = "running"
    COMPLETED = "completed"
    ERROR = "error"
    USER_ERROR = "user_error"

class JudgeModelConfig(BaseModel):
    model: str
    model_source: Literal["serverless", "dedicated", "external"]
    system_template: str
    external_api_token: Optional[str] = None
    external_base_url: Optional[str] = None

class EvaluationJob(BaseModel):
    workflow_id: str
    type: Optional[EvaluationType] = None
    status: EvaluationStatus
    results: Optional[Dict[str, Any]] = None
    parameters: Optional[Dict[str, Any]] = None
    created_at: Optional[datetime] = None
    updated_at: Optional[datetime] = None

Import

from together.types.evaluation import (
    EvaluationType, EvaluationStatus, EvaluationJob,
    EvaluationCreateResponse, EvaluationStatusResponse,
    ClassifyParameters, ScoreParameters, CompareParameters,
    JudgeModelConfig, ModelRequest,
)

I/O Contract

Inputs

Name	Type	Required	Description
(constructed from API response dicts)	Dict[str, Any]	Yes	API JSON response data parsed into typed models

Outputs

Name	Type	Description
EvaluationCreateResponse	Pydantic Model	workflow_id and initial status
EvaluationJob	Pydantic Model	Full job details with type, status, results, parameters
EvaluationStatusResponse	Pydantic Model	Current status and optional results

Usage Examples

from together import Together

client = Together()

# EvaluationCreateResponse from create
response = client.evaluation.create(
    type="classify",
    judge_model="meta-llama/Llama-4-Scout-17B-16E-Instruct",
    judge_model_source="serverless",
    judge_system_template="Classify this.",
    input_data_file_path="file-abc123",
    labels=["good", "bad"],
    pass_labels=["good"],
)
print(response.workflow_id)  # str
print(response.status)       # EvaluationStatus enum

# EvaluationJob from retrieve
job = client.evaluation.retrieve(response.workflow_id)
print(job.type)    # EvaluationType.CLASSIFY
print(job.status)  # EvaluationStatus.COMPLETED
print(job.results) # Dict with evaluation results

Related Pages

• Principle:Togethercomputer_Together_python_LLM_Evaluation