I'm writing a basic crawler that simply caches pages with PHP.
All it does is use
to get contents of a webpage and regex to get all the links out
- at the moment it returns:
[url] => URL
[desc] => DESCRIPTION
The problem I'm having is figuring out the logic behind determining whether the page link is local or sussing out whether it may be in a completely different local directory.
It could be any number of combinations: i.e.
- the possibilities are endless.
What would be the correct algorithm to approach this? I don't want to lose any data that could be important.
posted date: 2008-12-11 14:45:00