crawler.sh

Crawl websites locally and export content as Markdown

Meet crawler.sh

crawler.sh is a local web crawler that extracts website content and exports it as Markdown for archives, retrieval systems, or AI context. It runs on a user's computer and includes a JavaScript rendering engine for sites whose content depends on client-side code. Crawling follows robots.txt rules and adjusts its pace when a site returns rate limits or access restrictions.

The tool also checks technical SEO issues such as missing titles, duplicate descriptions, broken links, and thin content. Users can export crawl data or generate a sitemap. It is available through a command-line interface and a desktop application, with a free tier for smaller crawls and paid features for larger projects.

Discussion

Tried it? Share a useful thought or ask a question.

Product details

Website: crawler.sh
Status: Launched
Category: Developer tools
Listed: Oct 17, 2026

Explore related topics

Command line Developer tools Markdown SEO Web scraping

Embed upvote button

Let visitors support crawler.sh from your website.

Button appearanceLightDark

crawler.sh

Upvote

Paste into your site’s HTML

<div data-betaneon-widget="light" style="box-sizing:border-box;display:flex;align-items:center;gap:12px;width:320px;max-width:100%;min-height:84px;padding:12px 14px;margin:0;border:1px solid #e1e0dc;border-radius:12px;background:#ffffff;color:#181818;font:400 14px/1.4 Arial,Helvetica,sans-serif;text-align:left;">
  <img src="https://m.betaneon.com/ic/9039eaf7-19fb-4c4c-9675-25dffb9b232e.webp" alt="" width="36" height="36" referrerpolicy="no-referrer" style="box-sizing:border-box;display:block;flex:0 0 36px;width:36px;height:36px;margin:0;padding:0;border:0;border-radius:9px;object-fit:contain;background:#fff;">
  <div style="box-sizing:border-box;flex:1;min-width:0;margin:0;padding:0;">
    <span title="crawler.sh" style="display:block;margin:0;padding:0;color:#181818;font:700 16px/1.4 Arial,Helvetica,sans-serif;overflow:hidden;text-overflow:ellipsis;white-space:nowrap;">crawler.sh</span>
    <div style="display:flex;align-items:baseline;gap:4px;margin:4px 0 0;padding:0;white-space:nowrap;">
      <span style="margin:0;padding:0;transform:translateY(-1.5px);color:#626664;font:400 14px/1.2 Arial,Helvetica,sans-serif;">on</span>
      <a href="https://betaneon.com/" target="_blank" rel="noopener noreferrer" aria-label="BetaNeon home (opens a new tab)" style="display:inline-flex;flex-shrink:0;margin:0;padding:0;border:0;background:transparent;text-decoration:none;outline-offset:3px;"><img src="https://betaneon.com/res/brand/wordmark.svg?v=453516e27973dc87" alt="BetaNeon" width="87" height="16" referrerpolicy="no-referrer" style="display:block;width:87px;height:auto;margin:0;padding:0;border:0;"></a>
    </div>
  </div>
  <a href="https://betaneon.com/vote/crawler-sh" target="_blank" rel="noopener noreferrer" aria-label="Upvote crawler.sh on BetaNeon (opens a new tab)" style="box-sizing:border-box;display:flex;flex:0 0 68px;flex-direction:column;align-items:center;justify-content:center;gap:3px;min-height:56px;margin:0;padding:0 0 0 12px;border:0;border-left:1px solid #e1e0dc;background:transparent;color:#087f5b;font:400 14px/1.2 Arial,Helvetica,sans-serif;text-decoration:none;text-align:center;outline-offset:3px;">
    <span aria-hidden="true" style="box-sizing:border-box;display:block;width:9px;height:9px;margin:3px 0 0;padding:0;border:0;border-top:2px solid #087f5b;border-left:2px solid #087f5b;transform:rotate(45deg);"></span>
    <img src="https://betaneon.com/embed/upvote/crawler-sh/count.svg?theme=light" alt="Votes" width="56" height="20" referrerpolicy="no-referrer" style="display:block;width:56px;height:20px;margin:0;padding:0;border:0;">
    <span style="color:#087f5b;font:400 14px/1.2 Arial,Helvetica,sans-serif;">Upvote</span>
  </a>
</div>

Links open in a new tab. Visitors sign in on BetaNeon, then confirm their upvote. The vote count updates when the page loads.

Unclaimed

Building crawler.sh?

Claim this profile to keep your product’s story up to date.

Sign in with your verified work email. Matching website domains are approved automatically; other requests are reviewed by our team.

Meet crawler.sh

Discussion

Product details

Explore related topics

Similar products

Exabase Website to Markdown

ScrapingBee

Geekflare API

Context.dev

Intuned

Mindcase