DEV Community

Cover image for I Built a Local-First Alternative to LangSmith After Spending $200 Debugging a Pipeline I Couldn't See | Shivnath Tathe
Shivnath Tathe
Shivnath Tathe

Posted on

I Built a Local-First Alternative to LangSmith After Spending $200 Debugging a Pipeline I Couldn't See | Shivnath Tathe

#llm #openai #opensource #python

Last month I spent $200 on OpenAI tokens debugging a RAG pipeline. I had no idea which step was eating my budget. LangSmith would have helped — but I didn't want to send my prompts to a cloud service just to debug locally.

So I built opensmith.

What is opensmith?

A local-first LLM pipeline tracer. Zero cloud.
Zero setup. Everything stays on your machine.

pip install opensmith
Enter fullscreen mode Exit fullscreen mode

How it works

One decorator:

from opensmith import trace

@trace(tags=["rag", "production"])
def pipeline(query: str):
    docs = retrieve(query)
    return generate(docs, query)
Enter fullscreen mode Exit fullscreen mode

That's it. opensmith captures:

  • Function name, inputs, outputs
  • Latency in milliseconds
  • Token usage per step
  • Cost estimate
  • Errors with full stack trace
  • Parent-child relationships for nested calls

The dashboard 

opensmith ui
Enter fullscreen mode Exit fullscreen mode

Opens at localhost:7823. Live WebSocket updates, search, filters, charts. No account needed.

dashboard

vs LangSmith

LangSmith opensmith
Setup Cloud account pip install
Data Sent to cloud 100% local
Framework Best with LangChain Any Python
Cost Free then paid Free forever

What's new in v0.1.5

Token budget alerts — never blow your API
budget again:

@trace(token_budget=1000)
def expensive_pipeline():
    ...
# Prints: ⚠ expensive_pipeline used 1,247 tokens (budget: 1,000)
Enter fullscreen mode Exit fullscreen mode

CLI trace filters:

opensmith traces --q rag --status err --tags production
Enter fullscreen mode Exit fullscreen mode

Auto port detection — if 7823 is taken,
opensmith finds the next free port automatically.

opensmith init — creates a starter config:

opensmith init
# Creates opensmith.json in current directory
Enter fullscreen mode Exit fullscreen mode

Autopatch — zero code changes 

from opensmith import autopatch
autopatch(only=["openai", "qdrant"])
Enter fullscreen mode Exit fullscreen mode

Supported: OpenAI, Anthropic, LiteLLM, Qdrant, ChromaDB, Pinecone.

Stats after 5 days

1,500+ PyPI downloads. 16 GitHub stars. Growing organically.

GitHub: https://github.com/shivnathtathe/opensmith

If this saves you time, a ⭐ helps others
discover it. Thank you for 1,500+ downloads 🙏

What would you add to a local-first tracing tool?

Top comments (0)