Snippets tagged crawling

  • Parallel recursive crawler using agents

    The aim here is to demonstrate a method of distributing work using the built in F# agent across multiple nodes in parallel the result of crawling one page might result in finding multiple new pages to fetch. This is a recursive process which will continue until no new URLs are found. The main focus is how to process a potentially indefinite queue of work across a pool of workers, rather than how to parse web pages.

    5 people like this

    Posted: 7 years ago by Daniel Bradley

  • Web Crawler extensions

    The snippet extends a web crawler from snippet It synchronizes all printing using an additional agent (so printed text does not interleave) and the crawling function returns an asynchronous workflow that returns when crawling completes.

    0 people like this

    Posted: 6 months ago by Tomas Petricek