Firecrawl

Firecrawl AI

Firecrawl AI

Firecrawl AI

Firecrawl AI

Firecrawl AI

Firecrawl AI

Firecrawl AI

Firecrawl AI

Firecrawl AI

Firecrawl AI

Claude Code got me to 75% site-cloning accuracy. Now I’m hitting the wall (and trying to be lazy about it).

So I’ve been building this site-cloner with Claude Code. The stack is pretty straightforward: Playwright for screenshots and animations, plus Firecrawl and I set up a QA loop where it compares its own build against the original screenshot and tries to self-correct. The layout? Honestly, it’s spot on. But the animations are a total mess, and I’m stuck at about 75% accuracy. I found some repos on GitHub that seem to have the "secret sauce" for the animation logic I'm missing, even some with scraping logic and coping logic. Here’s my problem: I’m "vibe coding" this. It’s a side project, I am new, and I have zero interest in deep-diving into 5000 lines of someone else's code/skill to understand their architecture. I just want the logic. Two things I’m struggling with: The "Ingestion" Prompt: How are you guys making Claude actually evaluate external logic? I want to tell it: "Look at this repo, compare it to my current mess, and tell me what they’re doing smarter than me." Every time I try, I just get a generic summary. Any tips on a prompt that actually forces it to analyze and "steal" specific logic? The "Super-Skill" vs. Modular approach: Right now it’s one big block. Would it be smarter to split it? Skill 1: Structure/HTML. Skill 2: Animation logic. Skill 3: The QA loop. Does splitting it actually improve the reasoning, or am I just making the context-passing a nightmare? usually I would prefere to combine several skills together but the goal is send and forget. but if its not possible to make claude activate on its own other skills with a checkpoint system("you scraped the website? great move on to get screenshots"). Would love to hear from anyone building agents or just successfully "borrowing" logic without losing their mind. submitted by /u/TeamNecessary5548 [link] [comments]

Spent three hours making Claude sentient

Finally got MCP servers working in Claude Code after debugging package conflicts until 2:17 AM while my neighbor's dog barked through the entire process. Basically gave Claude the ability to mess with my filesystem and control browsers. It can now read my embarrassing old code and automate Chrome like some kind of digital puppet master. The one-liner that actually worked after everything else failed: ```bash bash <<'EOF' echo "Installing MCP servers because I hate myself..." Give Claude filesystem access (RIP privacy) claude mcp add filesystem -s user \ -- npx -y @modelcontextprotocol/server-filesystem \ ~/Desktop ~/Downloads ~/Code Browser automation for maximum chaos claude mcp add playwright -s user \ -- npx -y @playwright/mcp-server Web scraping because why not claude mcp add fetch -s user \ -- npx -y @kazuph/mcp-fetch Sequential thinking (Claude's internal monologue) claude mcp add sequential-thinking -s user \ -- npx -y @modelcontextprotocol/server-sequential-thinking echo "Done. Claude can now judge your code directly." claude mcp list EOF ``` Windows users are on their own with this one. Good luck. The filesystem server lets Claude browse through whatever folders you specify (I gave it access to my code directory because apparently I enjoy suffering). Playwright handles browser automation across Chrome, Firefox, Safari. Sequential thinking makes Claude actually reason through problems instead of confidently hallucinating. Browser automation is genuinely unsettling to watch. Like your computer gained consciousness and decided to browse Stack Overflow. For the brave search integration you need an API key from Brave. Firecrawl costs money but scrapes sites way better than the free alternatives. Use -s user to install globally or -s local if you only want these tools in your current project. The -s user flag means Claude gets these powers everywhere (probably a mistake but here we are). Troubleshooting: if stuff breaks, run /mcp in Claude Code to see which servers are actually running. Most connection issues come from Node version conflicts or permissions. Now Claude can read my TODO.txt file and judge me for putting "learn Rust" on there for the eighth month straight. Anyone else feel weird about giving an AI direct access to their computer or is that just me being paranoid? submitted by /u/Turbulent-Pay7073 [link] [comments]

How to save 80% on your claude bill with better context

been building web apps with claude lately and those token limits have honestly started hitting me too. i’m using claude 4.6 sonnet for a research tool, but feeding it raw web data was absolutely nuking my limits. I’m putting together the stuff that actually worked for me to save tokens and keep the bill down: switch to markdown first. stop sending raw html. use tools like firecrawl to strip out the nested divs and script junk so you only pay for the actual text. don't let your prompt cache go cold. anthropic’s prompt caching is a huge relief, but it only works if your data is consistent. watch out for the 200k token "premium" jump. anthropic now charges nearly double for inputs over 200k tokens on the new opus/sonnet 4.6 models. keep your context under that limit to avoid the surcharge strip the nav and footer. the website’s "about us" and "careers" links in the footer are just burning your money every time you hit send. use jina reader for quick hits. for simple single-page reads, jina is a great way to get a clean text version without the crawler bloat. truncate your context. if a documentation page is 20k words, just take the first 5k. most of the "meat" is usually at the top anyway. clean your data with unstructured if you are dealing with messy pdfs alongside web data, this helps turn the chaos into a clean schema claude actually understands. map before you crawl. don't scrape every subpage blindly. i use the map feature in firecrawl to find the specific documentation urls that actually matter for your prompt, if you use another tool, prefer doing this. use haiku for the "trash" work. use claude 4.5 haiku to summarize or filter data before feeding it into the expensive models like opus. use smart chunking. use llama-index to break your data into semantic chunks so you only retrieve the exact paragraph the ai needs for that specific prompt. cap your "extended thinking" depth. for opus 4.6, set thinking: {type: "adaptive"} with effort: "low" or "medium". the old budget_tokens param is deprecated on 4.6. thinking tokens are billed at the output rate, so if you leave effort on high, claude thinks hard on every single reply including the simple ones and your bill will hurt. set hard usage limits. set your spending tiers in the anthropic console so a buggy loop doesn't drain your bank account while you're asleep. feel free to roast my setup or add better tips if you have them submitted by /u/No-Writing-334 [link] [comments]

My full Claude Code setup after months of daily use — context discipline, MCPs, memory, subagents

Stop blaming Claude. Your harness is the problem. I've been running Claude Code on Opus 4.7 for 8+ hours a day on Max 5x. Zero quota issues. Here's what I actually did. Most people complaining about Claude "going dumb" or "eating tokens" set it up like this: no memory, no tools, no rules, dump 40 files into one context window, then wonder why it hallucinates. That's not a Claude problem. Context discipline cuts token usage roughly in half Put a CLAUDE.md at your repo root. Stack overview, ownership matrix, hard rules — run tsc --noEmit after every edit, max 50 lines per bugfix, one fix per commit, never touch auth/Stripe/middleware without explicit approval. It loads every session. Claude stops asking the same questions. Persistent memory lives at ~/.claude/projects/yourproject/memory/ — typed markdown files with prefixes like user, feedback, project, reference. Keep an index in MEMORY.md. You stop re-explaining your project at the start of every conversation. Biggest single quota win: subagents for grep-work. Spawn an Explore or general-purpose agent to do the file-digging. They burn their own context, return a summary. Your main window stays clean. Workflow discipline is where most setups fall apart Auto-retros after every non-trivial session. Save them to docs/retros/YYYY-MM-DD-topic.md. The next session loads the latest retro automatically — continuity without re-briefing. verification-before-completion as a hard rule. Claude cannot say "done" or "fixed" without running the verify command and showing you the output. Kills hallucinated success completely. Atomic commits, one fix per commit, hard line limits. Clean history, easy rollback, and it forces Claude to actually scope its work. For architecture decisions or anything involving security/migrations: one phrase triggers Claude to spawn Gemini Pro + Flash + Sonnet in parallel and synthesize. Three independent reads are better than one confident monologue. MCP servers — let it act instead of copy-pasting The ones I actually use: supabase — SQL, migrations, schemas directly from chat github — PRs, diffs, issues, file reads chrome-devtools-mcp + playwright — Claude can browse your deployed site, take screenshots, evaluate JS. It QAs itself. context7 — current library docs, not stale training data. Kills a specific class of hallucination entirely. firecrawl — on-demand scraping sentry — production errors read and triaged from chat gemini MCP — powers the multi-model consultation panel OSS worth actually installing graphify — takes any input (code, docs, papers, images) and produces a clustered knowledge graph as HTML + JSON. On large repos, Claude reads the graph instead of 200 files. Massive. claude-flow — swarm orchestration, hooks, memory coordination, SPARC, TDD, code review swarms. github.com/ruvnet/claude-flow Superpowers skills — search "superpowers skills claude code" on GitHub. The ones I use most: systematic-debugging, verification-before-completion, dispatching-parallel-agents, test-driven-development. CodeRabbit skill reviews diffs and auto-fixes review comments. Claude Retrospective skill generates the retros mentioned above. Hooks automate the grunt work PreToolUse, PostToolUse, SessionStart, PreCompact, Stop. Auto-save memory, auto-run tsc on edits, sync state before compaction. Claude thinks, the harness does the janitor work. TL:DR! Write CLAUDE.md Turn on persistent memory Install graphify + claude-flow + 6-7 MCPs Auto-retros + verification-before-completion as non-negotiables Subagents for grep and file exploration 50-line limit per bugfix Consultation panel for hard calls 5+ hours a day, ~250 tool calls per session, atomic commits, full deploy → screenshot → verify cycles. Max 5x, no quota hit. Claude isn't the problem. The harness is! EDIT: https://github.com/anothervibecoder-s/claudecode-harness I made a claude.md example based on my CLAUDE.md file, you can tell claude to fill this based on your projects! If it helped, just star it! submitted by /u/Sictir1 [link] [comments]

I built a free claude blog skill that actually studies your business, researches competitors & keywords to find winning blog topics and high-quality articles with infographics, internal linking, product promotion, and more...

Most AI writing tools are a fancy wrapper around "give me 1500 words about X." They don't know your business, your competitors, what's already ranking, or why someone would read your article over the 10 that already exist. The output is always that same slightly-hollow, over-structured content that reads like it was written by someone who's never actually done the thing they're writing about. I wanted to build something that approached content strategy the way a good SEO consultant would like studying the business first, doing real research, then writing So I built a set of Claude Code slash commands that run a full pipeline. Here's what it actually does: Step 1: Onboarding Scrapes your website, extracts a structured business profile (product type, ICP, differentiator, brand voice, integrations), then hits DataForSEO's SERP API to find your 3 direct competitors. Everything gets saved locally in .claude/blog-config.json. You run this once. Step 2: Site Intelligence (the interesting one) This is where it gets serious. It runs three keyword sources in parallel: Your existing rankings (top 100 by traffic value from DataForSEO) Competitor keywords (top 200 per competitor) Seed expansion: Claude Haiku generates 30 seed phrases based on your ICP's pain points and integrations, then DataForSEO expands each seed into ~30 related keywords (30 parallel API calls), then bulk KD lookup on all of them That's roughly 2,000 raw keywords before dedup. After merging and deduplicating, it filters by volume floor, KD ceiling relative to your domain rating, and strips anything you already rank top-5 for Then Claude Haiku classifies every remaining keyword into TOFU/MOFU/BOFU in parallel batches of 50. Claude Sonnet groups them into 6–10 topical clusters. Each cluster gets a pillar keyword and supporting keywords. Opportunity scoring uses a weighted additive formula (not multiplicative since it compresses everything toward zero): score = (0.40 × log_volume + 0.40 × difficulty + 0.20 × funnel) × 100 Volume is log-normalized against a 100k anchor so a 1,000/mo keyword scores 60% instead of 1%. 70+ means actually worth targeting. It picks one topic per cluster (breadth-first), generates SEO titles for all 10, and saves them to your content pipeline. Step 3: Content Engine Per article, it runs: DataForSEO advanced SERP for the target keyword → Firecrawl scrapes the top 3 ranking articles to extract H2 structure and avg word count Tavily batch search: 3 queries in parallel for recent news, expert opinions, common mistakes YouTube Data API → transcript extraction via Apify → Claude Haiku pulls 2 concrete insights Then Claude Haiku does SERP gap analysis like what are all 3 top articles covering, what are they missing, what's the best featured snippet opportunity. Claude Sonnet generates a full outline: every H2, H3, word count per section, where research gets placed, where the product mention goes (with specific framing instruction), image positions, CTA matched to funnel stage. Then Claude Sonnet writes the full article in one shot against that outline. Images get generated after Haiku reads the actual written content to create better DALL-E prompts than you'd get from just the keyword. Schema markup and meta assets are separate Haiku calls. Product plug is deliberately constrained: one mention, at a designated section, only after the reader has felt the pain it solves. No marketing language. The outline specifies the exact framing. Output is a folder: article.md (pure content, copy into CMS), publish-kit.md (meta, schema JSON, publishing checklist), and images/. The whole thing is Claude Code slash commands - /blog-onboard, /blog-topics, /blog-write. You run them in any project directory. All data stays local. I open-sourced it here: github.com/maun11/claude-blog-engine It's working but honestly there's a lot of room to improve it. If you've built anything in this space or have opinions on the architecture, would genuinely appreciate the feedback. And if you improve something, PRs are welcome and there's a lot of low-hanging fruit in the pipeline script (scripts/topics_pipeline.py) specifically. submitted by /u/Visible-Mix2149 [link] [comments]

Need help with testing MCP servers in local

I'm trying to test my MCP server in local. I have added authentication similar to firecrawl's MCP server implementation where I give an API key to the user and I want them to put in the URL and use it as a connector URL in claude. I wanted to test this implementation in local so I tried deploying this using docker as an http server and then used ngrok to transfer it to https and tried adding that as a connector to claude because for some reason using tickerr with mcp-remote in claude config is failing for me. But I couldn't succeed in this. Where am I going wrong and what are all the things should I have in mind while testing this out? submitted by /u/Substantial_Smoke_32 [link] [comments]

What is the best way to capture online mentions and capture signals daily?

I was talking with a friend who works in a startup and they would like to track decision makers across a specific set of companies and be updated daily. News mentions, comments or signals from those people across the web, and social media, particularly X. What is the best way to achieve this? A custom solution made with Claude code, leveraging scrapers (Ex. Firecrawl, Apify)? Or would n8n and similar platforms be better? Any personal experiences with this? Thanks! submitted by /u/W0rldIsMy0yster [link] [comments]

How to save 80% on your claude bill with better context

been building web apps with claude lately and those token limits have honestly started hitting me too. i’m using claude 4.6 sonnet for a research tool, but feeding it raw web data was absolutely nuking my limits. i’m putting together the stuff that actually worked for me to save tokens and keep the bill down: switch to markdown first. stop sending raw html. use tools like firecrawl to strip out the nested divs and script junk so you only pay for the actual text. don't let your prompt cache go cold. anthropic’s prompt caching is a huge relief, but it only works if your data is consistent. watch out for the 200k token "premium" jump. anthropic now charges nearly double for inputs over 200k tokens on the new opus/sonnet 4.6 models. keep your context under that limit to avoid the surcharge strip the nav and footer. the website’s "about us" and "careers" links in the footer are just burning your money every time you hit send. use jina reader for quick hits. for simple single-page reads, jina is a great way to get a clean text version without the crawler bloat. truncate your context. if a documentation page is 20k words, just take the first 5k. most of the "meat" is usually at the top anyway. clean your data with unstructured if you are dealing with messy pdfs alongside web data, this helps turn the chaos into a clean schema claude actually understands. map before you crawl. don't scrape every subpage blindly. i use the map feature in firecrawl to find the specific documentation urls that actually matter for your prompt, if you use another tool, prefer doing this. use haiku for the "trash" work. use claude 4.5 haiku to summarize or filter data before feeding it into the expensive models like opus. use smart chunking. use llama-index to break your data into semantic chunks so you only retrieve the exact paragraph the ai needs for that specific prompt. cap your "extended thinking" depth. for opus 4.6, set thinking: {type: "adaptive"} with effort: "low" or "medium". the old budget_tokens param is deprecated on 4.6. thinking tokens are billed at the output rate, so if you leave effort on high, claude thinks hard on every single reply including the simple ones and your bill will hurt. set hard usage limits. set your spending tiers in the anthropic console so a buggy loop doesn't drain your bank account while you're asleep. feel free to roast my setup or add better tips if you have them submitted by /u/Illustrious_Elk3705 [link] [comments]

You can now give an AI agent its own email, phone number, wallet, computer, and voice. This is what the stack looks like

I’ve been tracking the companies building primitives specifically for agents rather than humans. The pattern is becoming obvious: every capability a human employee takes for granted is getting rebuilt as an API. Here are some of the companies building for AI agents: AgentMail — agents can have email accounts AgentPhone — agents can have phone numbers Kapso — agents can have WhatsApp numbers Daytona / E2B — agents can have their own computers monid.ai — agents can read social media (X, TikTok, Reddit, LinkedIn, Amazon, Facebook) Browserbase / Browser Use / Hyperbrowser — agents can use web browsers Firecrawl — agents can crawl the web without a browser Mem0 — agents can remember things Kite / Sponge — agents can pay for things Composio — agents can use your SaaS tools Orthogonal — agents can access APIs more easily ElevenLabs / Vapi — agents can have a voice Sixtyfour — agents can search for people and companies Exa — agents can search the web (Google isn’t built for agents) What’s interesting is how quickly this came together. Not long ago, none of this really existed in a usable form. Now you can piece together an agent with identity, memory, communication, and spending in a single afternoon. Feels less like “AI tools” and more like the early version of an agent-native infrastructure stack. Curious if anyone here is actually building on top of this. What are you using? Also probably missing a bunch - drop anything I should add and I’ll keep this updated. submitted by /u/Shot_Fudge_6195 [link] [comments]

I want to move further than in-browser and IDE copilot level. Where to start?

Basically the title. I'm a photographer for passion and web developer by trade. I gave Claude a detailed prompt in regard to this, and it basically recommended me to get Claude Desktop and a filesystem server. Also Firecrawl. But we all know that what AI lacks in intuition humans make up for with experience. So what's your experience? Where did you start? I want to leverage agentic AI past the "cages" that IDEs and browsers are, and start using them more in automising tasks in my day to day life. I am keeping the question somewhat vague so that you guys feel free to go as wild as possible with the recommendations. Thank you! Happy prompting! submitted by /u/vulturici [link] [comments]

Prism MCP — I gave my AI agent a research intern. It does not require a desk

So I got tired of my coding agent having the long-term memory of a goldfish and the research skills of someone who only reads the first Google result. I figured — what if the agent could just… go study things on its own? While I sleep? Turns out you can build this and it's slightly cursed. Here's what happens: On a schedule, a background pipeline wakes up, checks what you're actively working on, and goes full grad student. Brave Search for sources, Firecrawl to scrape the good stuff, Gemini to synthesize a report, then it quietly files it into memory at an importance level high enough that it's guaranteed to show up next time you talk to your agent. No "maybe the cosine similarity gods will bless us today." It's just there. The part I'm unreasonably proud of: it's task-aware. Running multiple agents? The researcher checks what they're all doing and biases toward that. Your dev agent is knee-deep in auth middleware refactoring? The researcher starts reading about auth patterns. It even joins the group chat — registers on a shared bus, sends heartbeats ("Searching...", "Scraping 3 articles...", "Synthesizing..."), and announces when it's done. It's basically the intern who actually takes notes at standups. No API keys? It doesn't care. Falls back to Yahoo Search and local parsing. Zero cloud required. I also added a reentrancy guard because the first time I manually triggered it during a scheduled run, two synthesis pipelines started arguing with each other and I decided that was a problem for present-me, not future-me. Other recent rabbit holes: Ported Google's TurboQuant to pure TypeScript — my laptop now stores millions of memories instead of "a concerning number that was approaching my disk limit" Built a correction system. You tell the agent it's wrong, it remembers. Forever. It's like training a very polite dog that never forgets where you hid the treats One command reclaims 90% of old memory storage. Dry-run by default because I am a coward who previews before deleting Local SQLite, pure TypeScript, works with Claude/Cursor/Windsurf/Gemini/any MCP client. Happy to nerd out on architecture if anyone's building agents with persistent memory. https://github.com/dcostenco/prism-mcp submitted by /u/dco44 [link] [comments]