Just bought another couple of domains.
Am still looking for a tool that can scrape archive.org (or google cache) to get embro content.
Does anyone know of such a tool?
By the way for those that don't know it , winHTTrack is really useful for scraping existing (non wordpress) sites, with plugins that can import the HTML.
The problem is scraping from cache or archive. Anyone know of such a thing?
Chabrenas (April 11th, 2012)
I had a similar situation with about 50 articles. I just setup my pages/posts on WP first and then copied and pasted the articles from wayback machine to WP. It didn't take long.
If it's "thousands of pages" my answer would be different. You would probably need a custom script.
Last edited by Clinton; April 11th, 2012 at 11:10 AM. Reason: forum rules
mikeb (April 11th, 2012)
have you tried httrack on archive.org?
it seems that their structure for URLs is simply: http://web.archive.org/web/xxxxxxxxxxxxx/the website's normal domain / structure / urls
and all they do is replace all links in the code with their bit in front - so should work...arguably
you could then just run a text editor search and replace over it all to remove their bit...
if not - then using CURL you can grab a page into a local variable - read the URLs and go off and read any linking pages in the same way - fairly simple scraper to write...
Alasdair
Tried it , and it fails. Don't know enough about httrack to try.Originally Posted by akirk
Are you volunteering? If you can and want to write that script - drop me a quote by pm
Mike,j
ust tried it and it is possible with httrack - on the simple webiste I tried...
it is, I think, to do with the settings - I had it going up and down / and sitting on the same top level domain I think...
may be worth a bit more playing with it
Alasdair
mikeb (April 12th, 2012)
Isn't Warrick the standard tool for this job?
Warrick didn't work, the last time I tried, but grynge came up with some fixes that made it work a while back. The details can be found if you look around - READ THE SITE, we've got it somewhere.
Warrick seems to have closed the doors
Warrick is back up for business, but there is a massive wait, unless you can get the perl script running. Normally I can do a backup for you but I am away on holidays, if it can wait for another couple of weeks I won't have a problem doing the backup then.
I got out of bed today staring at a ghost. Who forgot to float away, didnt have all that much to say. Wouldn't even tell me his own name.
Non ducor, duco
mikeb (April 12th, 2012)
Many thanks. Where can i find a copy of the perl script?
Posting Permissions
Reply With Quote
Bookmarks