Human feedback as an API

Ship human feedback straight to your training loop

One call sends a task to a global, quality-scored workforce and returns a verified answer — consensus, gold tests, and confidence scores built in. Made for RLHF, evals, and annotation.

Get your API key Read the docs →

No sales call. Free credits on signup. Dispatched in 200ms, verified in minutes.

rank_responses.sh

# Send two model outputs for human preference ranking
curl https://api.blueloof.com/v1/tasks \
  -H "Authorization: Bearer $BLUELOOF_KEY" \
  -d '{
    "template": "rlhf.preference",
    "input": {
      "prompt":      "Explain backprop to a 10-year-old",
      "response_a":  "...",
      "response_b":  "..."
    },
    "redundancy": 3,        // 3 workers
    "quality":    "gold+consensus"
  }'

# → returns
{ "winner": "response_a",
  "confidence": 0.94,
  "agreement": "3/3",
  "latency_s": 142 }

Backed by a live marketplace that has paid workers in 190+ countries

190+ countries 2M+ workers Sub-minute dispatch Consensus QA

Why Blueloof

Managed vendors are consultancies.
Blueloof is infrastructure.

Scale, Surge, and Appen wrap human workers in slow, contract-first services. Blueloof gives you the same quality layer as a raw, API-first primitive — no sales cycle, no minimums, wired straight into your training loop.

🎯

Quality, not just completion

Every task runs through redundant labeling, majority-vote consensus, and hidden gold-standard checks. You get a confidence score, not a shrug.

🏅

Worker reputation routing

Tasks route to workers scored on accuracy, language, and specialty. Ask for reasoning experts or native-English annotators — get them.

⚡

API-first, pipeline-native

One POST /tasks call. Webhook or poll for results. Drop human judgment into RLHF and eval loops without a services contract.

🔁

Built for the training loop

Preference ranking, reward signals, and hallucination flags in a schema your trainer reads directly. Infrastructure, not one-off campaigns.

🌍

Instant global liquidity

A live marketplace with millions of workers means throughput on demand — no ramp, no recruiting, no minimum commitment.

💸

A fraction of vendor cost

Distributed micro-work at marketplace rates. Same verified quality, without the managed-services markup baked into every label.

How it works

From API call to verified answer

Four hidden stages turn a raw task into a label you can trust — all behind a single endpoint.

→

Submit

POST a task with a template and your input payload.

→

Route

Dispatched to N scored workers matched to the task type.

→

Verify

Consensus scoring + gold tests filter noise and bad actors.

→

Score

Aggregate into a result with a confidence value.

Return

Webhook or poll — straight into your pipeline.

Task types

Templates for every human-in-the-loop need

Start from a typed template — or define your own schema and interface.

rlhf.preference

Response ranking

Pick the better of two model outputs, with justification.

eval.reasoning

Reasoning grading

Score step-by-step reasoning for correctness and rigor.

safety.halluc

Hallucination detection

Flag unsupported or fabricated claims in a response.

vision.annotate

Image annotation

Bounding boxes, segmentation, and classification.

audio.transcribe

Transcription

Speech-to-text with quality and language routing.

custom.*

Bring your own

Define a JSON schema and custom worker interface.

The cost gap

Verified human labels, without the consultancy markup

Illustrative per-label pricing on LLM evaluation tasks. Real quotes depend on task complexity and redundancy.

$1–$5

Managed vendors
(Scale / Appen)

$0.05–$0.50 / label

Blueloof
same verified quality

Raw microtask sites
no quality layer

Docs

Quickstart & API reference

Everything you need to send your first verified task. Base URL https://api.blueloof.com/v1

Quickstart

Go from zero to a verified human answer in three steps.

Get your API key

export BLUELOOF_KEY="sk_live_..."

Send your first task

Post a preference-ranking task with 3-worker redundancy and consensus quality.

curl https://api.blueloof.com/v1/tasks \
  -H "Authorization: Bearer $BLUELOOF_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "template": "rlhf.preference",
       "input": { "prompt": "...", "response_a": "...", "response_b": "..." },
       "redundancy": 3, "quality": "gold+consensus" }'

Get the verified result

Poll the task ID or register a webhook. Results return with a confidence score.

{ "id": "task_8fk2...", "status": "completed",
  "result": { "winner": "response_a", "confidence": 0.94, "agreement": "3/3" } }

Authentication

All requests are authenticated with a bearer token in the Authorization header. Keep your secret key server-side — never expose it in client code.

Authorization: Bearer sk_live_...

Create a task

POST /v1/tasks

Submits a task to the workforce. Returns immediately with a task ID; work is dispatched asynchronously.

Parameter	Description
template required string	The task template, e.g. rlhf.preference, eval.reasoning, custom.<id>.
input required object	Payload matching the template's schema (prompt, responses, image URL, etc.).
redundancy integer	Number of independent workers per task. Default 1. Higher raises confidence and cost.
quality string	Quality mode: basic, gold, consensus, or gold+consensus.
webhook_url string	URL notified with the completed result. Omit to poll instead.
metadata object	Arbitrary key/values echoed back on the result. Useful for pipeline correlation.

Retrieve a task

GET /v1/tasks/{id}

Fetches current status and, once finished, the verified result.

curl https://api.blueloof.com/v1/tasks/task_8fk2 \
  -H "Authorization: Bearer $BLUELOOF_KEY"

Response object

Every task resolves to a consistent shape regardless of template.

Field	Description
id string	Unique task identifier.
status string	queued, in_progress, completed, or failed.
result object	Template-specific answer plus confidence (0–1) and agreement.
latency_s number	Seconds from dispatch to completed result.
cost number	Amount billed for the task, in USD.

Webhooks

Register webhook_url on a task and Blueloof POSTs the full response object the moment it completes — no polling. Payloads are signed with your webhook secret; verify the Blueloof-Signature header before trusting them.

POST https://your-app.com/hooks/blueloof
Blueloof-Signature: t=1784..,v1=5f2c..

{ "event": "task.completed", "data": { "id": "task_8fk2", ... } }

Errors & rate limits

Blueloof uses standard HTTP status codes. 2xx means success, 4xx a problem with your request, and 5xx an error on our side. Every error returns a JSON body with a code and human-readable message.

Status	Code	Meaning
200	ok	Request succeeded.
400	invalid_request	Malformed body or input that doesn't match the template schema.
401	unauthorized	Missing or invalid API key.
402	insufficient_credits	Account balance can't cover the task. Top up to continue.
404	not_found	No task exists with that ID.
429	rate_limited	Too many requests. Back off and retry after Retry-After seconds.
500	server_error	Something broke on our end. Safe to retry with backoff.

Default limit is 600 requests/min per key, with generous task-creation bursts. Rate-limit state is returned on every response so you can throttle proactively.

X-RateLimit-Limit: 600
X-RateLimit-Remaining: 583
X-RateLimit-Reset: 1784221860

SDKs

Official libraries wrap auth, retries, and typing. Python and Node today; Go and Rust in progress.

from blueloof import Blueloof
client = Blueloof(api_key=os.environ["BLUELOOF_KEY"])

task = client.tasks.create(
    template="rlhf.preference",
    input={"prompt": "...", "response_a": "...", "response_b": "..."},
    redundancy=3, quality="gold+consensus",
)
result = task.wait()   # blocks until verified
print(result.winner, result.confidence)

import Blueloof from "blueloof";
const client = new Blueloof({ apiKey: process.env.BLUELOOF_KEY });

const task = await client.tasks.create({
  template: "rlhf.preference",
  input: { prompt: "...", response_a: "...", response_b: "..." },
  redundancy: 3,
  quality: "gold+consensus",
});
const result = await task.wait();  // resolves when verified
console.log(result.winner, result.confidence);

Get started

Put humans in your loop today

Grab a key, get free credits, and send your first verified task in five minutes.

No credit card. No sales call. Read the docs →

Managed vendors are consultancies. Blueloof is infrastructure.