---
name: StormCrawler
description: 'StormCrawler is an open-source SDK for running distributed web crawlers on Apache Storm. The project is under Apache license v2 and consists of a collection of reusable resources and components, written mostly in Java.  The aim of StormCrawler is to help build web crawlers that are:  scalable resilient low latency easy to extend polite yet efficient. StormCrawler has a module for OpenSearch that allows to index the documents crawled but also store metrics to display with Dashboards as well as store the frontier of the crawler in OpenSearch.'
owner: DigitalPebble Ltd
link: http://stormcrawler.net
license: Apache license v2
license_link: https://github.com/DigitalPebble/storm-crawler/blob/master/LICENSE
owner_link: https://digitalpebble.com
---