Link Checker
Enter a URL and this tool will map out all of the pages below it
Sign in to use Link Checker
Launch new crawls, track your previous jobs, and inspect results as they update.
What this does:
This tool asks a bot to go to your website of choice and look for every page under that URL which it can find from other pages. We collect every internal link on a page which we can give to you.
Use Cases:
- Find dead pages
- Find broken links
- Visualise your internal linking
Notes:
- Enter a proper url e.g. https://www.example.com
- This runs asynchronously - set this going and then leave it in the background
- Bigger websites will take longer
How this operates:
The bot validates then goes to the baseUrl you have specified and collects all of the links on that page which contain your url. It then goes to the first page it found in the links and so on. It finishes when it no longer has any new links to check. We can then graph all of the pages on your website!
What it's built with:
This is built inside of a docker container and can work from anywhere with an internet connection. it has been built this way because it can then be decoupled from the cloud in order to save cost. Inside it is currently running on Python for ease of writing and module quantity. We are using SQLAlchemy and Selenium as our main method - this allows us to render JS websites easily however this may be upgraded to a stealthier approach.