Scrapy-broken-links
A Scrapy crawler for broken & valid links
valid github link
broken github link
valid google link
broken google link
valid internal link with html fragment and query param