I suggest you ...

Have a setting during crawl to only load links to html/js/css files rather than a huge exlude list

We don't have a master list of filetypes on a 80,000+ page website, crawls keep failing as they find yet another new filetype. Can we have a setting that prevents the crawl from following links to anything that <b>isn't</b> a .html|.js|.css |.xml|.gz ? A whitelist makes much more sense than blacklist. What is wasp going to do once it's loaded an eps file anyway?

6 votes
Vote
Sign in
Check!
(thinking…)
Reset
or sign in with
  • facebook
  • google
    Password icon
    I agree to the terms of service
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    racheetracheet shared this idea  ·   ·  Flag idea as inappropriate…  ·  Admin →

    0 comments

    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      I agree to the terms of service
      Signed in as (Sign out)
      Submitting...

      Feedback and Knowledge Base