Test if a specific URL or path is crawlable by search engines based on robots.txt rules and meta directives.