Website Scraping of Competitors

[Thomas Andreas]

Hi,
recently listened to this podcast about new ways of researching and finding of keywords. And a method called website scraping (I think that is what it is referred to). Meaning basically, you put a competitors website into a website scraping software and it will show you what pages and keywords the competitor is ranking for... and I guess to some extent give you a reality check - can I reach top 10 on Serp. Anyways, does any body know what I am referring to and does anyone know a good company that is offering these services?

 

[Frank]

old skool SEO... i have pages with one keyword ranking..

 

[Thomas Andreas]

Hi Frank,
thanks so you mean you only set one keyword per page. I think maybe I am misunderstanding you

 

[JohnSmithelJackson]

Do you mean seopowersuite or moz? Check it out

 

[JuegosPalacio]

Hi Frank,
thanks so you mean you only set one keyword per page. I think maybe I am misunderstanding you

I think he means that you can actually rank for some keywords on the same page. For ex, no deposit bonus page can rank for some variations as "no deposit casinos" "free casinos" and so on. Even though, I always try to focus in one keyword by including it in H1H2 and so on, but you can play around with some secondary keywords which help to get more traffic from other similiar-topic queries with good SV. Also this is relative with long tail keywords... you should go for mentioning some over the text if you consider you can get good quality traffic in terms of intention and conversion.

About the tools, never heard about it. But I would say: keep it classic if you have to pay for. If it's for free, you could give it a chance.

 

[AussiePunter]

You might be talking about something completely different to what I'm thinking about, but scraping content from my point of view is when you buy or build a bot that scrapes other peoples content, ie; it steals other peoples content. To avoid any dupe content filter you can partially scrape a lot of different sites in your niche, usually you use the search term you want to rank for so you know the content will be on topic. Black hat and bound to get found out and penalized automatically at some stage, or manually if one of your competitors finds out you've been scraping their content.

I could be off the mark here, but that's what my understanding of scraping has been, it used to be a massive, massive problem until Google finally started punishing dupe content and trying to establish the original author of the content, of course like every spam defeating technique they are much, much better at stopping this kind of primitive spam now.

Then again, on the other hand scraping isn't entirely bad - it's how you get a lot of sport scores syndicated (i guess they use API's, but it's the same concept), it's only bad when bad actors try to take entire pages of content, remove the author's ads or replace the code of the ads with their own.

 

[AussiePunter]

You might be talking about something completely different to what I'm thinking about, but scraping content from my point of view is when you buy or build a bot that scrapes other peoples content, ie; it steals other peoples content. To avoid any dupe content filter you can partially scrape a lot of different sites in your niche, usually you use the search term you want to rank for so you know the content will be on topic. Black hat and bound to get found out and penalized automatically at some stage, or manually if one of your competitors finds out you've been scraping their content.

I could be off the mark here, but that's what my understanding of scraping has been, it used to be a massive, massive problem until Google finally started punishing dupe content and trying to establish the original author of the content, of course like every spam defeating technique they are much, much better at stopping this kind of primitive spam now.

Then again, on the other hand scraping isn't entirely bad - it's how you get a lot of sport scores syndicated (i guess they use API's, but it's the same concept), it's only bad when bad actors try to take entire pages of content, remove the author's ads or replace the code of the ads with their own.

Gee I need to re-read my stuff since I can't find any way to edit posts...

After reading the OP again, scraping can have good uses too - and learning new keywords and keyphrases that people are using is a great example of a legit reason to scrape websites. Take all the text, remove all stop words and anything not related to your niche, do a simple count, then do a 2 word count, 3 word count, 4 word count etc and find the most common phrases being used in the top ranking sites. It could very well give you some ideas, and you haven't stolen anything. The main thing you should be careful of is overloading the server with your bot, you don't need to read 1000 pages a second, and you'd probably get blocked if it was anything but a static website if you tried. Just do it human paced or a little faster so you don't put any stress on the servers you're hitting - and make sure you obey robots.txt and just in case something unexpected happens, include some contact info in the user-agent string.

 

[Thomas Andreas]

Gee I need to re-read my stuff since I can't find any way to edit posts...

After reading the OP again, scraping can have good uses too - and learning new keywords and keyphrases that people are using is a great example of a legit reason to scrape websites. Take all the text, remove all stop words and anything not related to your niche, do a simple count, then do a 2 word count, 3 word count, 4 word count etc and find the most common phrases being used in the top ranking sites. It could very well give you some ideas, and you haven't stolen anything. The main thing you should be careful of is overloading the server with your bot, you don't need to read 1000 pages a second, and you'd probably get blocked if it was anything but a static website if you tried. Just do it human paced or a little faster so you don't put any stress on the servers you're hitting - and make sure you obey robots.txt and just in case something unexpected happens, include some contact info in the user-agent string.

Thanks Aussie

 

[Casinoer Danmark]

In my opinion scraping is almost a must of doing seo on a big scale or having a/some good VA's. Lets say you are adding a couple of 100 slot games and want to have the RTP, bet limits and other structured data you want to scrape that from *cough cough no names mentioned cough* and upload it yourself. I'd never scrape body of text content, thats what you have proper scalable content templates instead.

If I go back on topic; what you are trying to do is shooting a bird with a bazooka. scraping is a powerful too but if you want to research competition for ex. 'mobile slots' I'd rather do this manually by going into Ahrefs and filtering the valuable pages & keywords myself.

If you hire a serious business they charge you a flat fee for setting up the project and a fixed price per line on how much you decide to scrape. Many companies does not accept orders under 10 000 rows. Then your best friend is Upwork, Fiverr, etc. But then again, you need to know exactly what you need if hiring a paki freelancer.

 

[Anthony Pap]

The way I understood what OP posted is basically just using a tool like Ahrefs to see what pages drive the most traffic for a competitor's website and what keywords as well. I don't know if that's called website scraping but this a common function that most SEO tools offer.

 

[Corriens]

Do you mean seopowersuite or moz? Check it out

Moz is a good tool for website analysis. Besides, there have some other good tools as well. Like ahref, semrush, google analytics, majestic.

 

[StepGart]

Ahref, majestic.
Has got used to them already