RobiData

RobiData is a dataset creation and publishing tool for LLM training pipelines. It supports scraping, cleaning, transforming, validating, and publishing datasets with both CLI and web UI workflows.

---

What It Does

• Scrape from multiple source types:
  • web
  • pdf
  • api
  • github
  • firecrawl
• Clean and normalize text data
• Deduplicate and optionally anonymize PII
• Transform to fine-tuning formats (alpaca, sharegpt, chatml, qa_pairs, raw)
• Validate quality/shape constraints
• Publish to Hugging Face datasets

---

Installation

Basic

pip install -e .

Optional extras

pip install -e ".[pdf]"
pip install -e ".[all]"

---

Configuration (.env)

RobiData CLI auto-loads .env from project root.

Example:

FIRECRAWL_API_KEY=fc_xxx
HF_TOKEN=hf_xxx

Notes:

• .env should stay uncommitted.
• Firecrawl key can be passed either by env var or CLI flag.
• Hugging Face token is still passed explicitly via --token on publish.

---

CLI Commands

robidata scrape <urls> [options]
robidata transform <input.jsonl> [options]
robidata validate <input.jsonl> [options]
robidata stats <input.jsonl>
robidata publish <input.json|input.jsonl> --repo-id <owner/name> [options]

Quick examples

# scrape (web)
robidata scrape https://example.com --follow-links --max-pages 20

# scrape (firecrawl)
robidata scrape https://example.com --source firecrawl --max-pages 20

# transform
robidata transform ./datasets/scraped.jsonl --format alpaca --anonymize

# validate
robidata validate ./datasets/train.jsonl --format instruction

# stats
robidata stats ./datasets/train.jsonl

# publish
robidata publish ./datasets/train.json --repo-id your-user/your-dataset --token hf_xxx

---

Important Scrape Options

robidata scrape

• --source {web,pdf,api,github,firecrawl}
• --max-pages <int>
• --rate-limit <float>
• --timeout <int>
• --follow-links
• --max-depth <int>
• --ignore-robots (web source)
• --github-repo <owner/name>
• --github-token <token>
• --firecrawl-api-key <key>
• --firecrawl-formats <csv> (e.g. markdown,html)

robidata transform

• --format {alpaca,sharegpt,chatml,qa_pairs,raw}
• --min-length <int>
• --max-length <int>
• --no-dedup
• --dedup-threshold <float>
• --anonymize

---

Web App

RobiData includes a local web app in webapp/ with:

• form-driven command builder (no manual CLI args required)
• Robi dark visual language
• startup environment bootstrap with preloader

Start web app

python3 webapp/server.py

Open:

http://127.0.0.1:8787

Startup behavior

When server starts, it bootstraps runtime automatically:

1. If ./venv does not exist, it creates it
2. Installs requirements.txt
3. Installs editable package (pip install -e .)
4. Signals UI as ready

The web UI shows a loader until bootstrap completes.

---

Python API

from robidata import DatasetPipeline
from robidata.models import DatasetConfig, ScraperConfig, TransformConfig, SourceType, DataFormat

config = DatasetConfig(
    name="my-dataset",
    scraper=ScraperConfig(
        source_type=SourceType.FIRECRAWL,
        urls=["https://example.com"],
        firecrawl_api_key="fc_xxx",
    ),
    transform=TransformConfig(
        output_format=DataFormat.ALPACA,
        deduplicate=True,
    ),
)

pipeline = DatasetPipeline(config)
result = pipeline.execute()
print(result.stats.total_entries)

---

Troubleshooting

Loader stuck on “Starting…”

• Hard refresh browser (Cmd + Shift + R)
• Restart server:

python3 webapp/server.py

robidata command not found

Use web app bootstrap (it auto-installs), or run:

python3 -m venv venv
./venv/bin/pip install -r requirements.txt
./venv/bin/pip install -e .

Hugging Face publish 401 Unauthorized

• Ensure valid HF token with write permission
• pass token explicitly in publish command
• verify --repo-id owner matches your account/org

Scrape returns zero entries

• Source may block crawling via robots.txt
• For JS-heavy pages, use Firecrawl source
• Prefer official APIs/data exports where available

---

Project Structure

src/robidata/
  __init__.py
  cli.py
  models.py
  pipeline.py
  publisher.py
  scrapers.py
  transformers.py

webapp/
  index.html
  server.py

---

Team

Built by Robi Team
Website: https://robiai.com/