LUYA Crawler Version 3.0

We have released version 3.0 of LUYA Crawler.

  • The crawler is now so much faster, as it can process 30 concurrent requests. The Crawler engine is now based on this library: https://github.com/nadar/crawler
  • PDF Documents are indexed by default.
  • The new Crawler engine ensures very low memory usage, but when crawling PDFs and PDF crawling is enabled (which is by default), it might take more memory when indexing large PDFs.