How to Scrape Content from a Website for Your WordPress Blog
If you’re looking to expand your content scraping capabilities, then this article is for you. We’re going to teach you the ins and outs of content scraping so that you can begin harvesting the content that you need to pull into your own blog. Towards the end of the article, we’ll even show you some tools that can make the process a little easier. But first, let’s take a quick look at why you might want to go this route in the first place.
Why Blogging?
Blogging is a great choice for a number of reasons. To begin with, the rules are simple:
- Keep it short
- Consist of at least 3-5 articles
- Have a topical theme (business, entertainment, travel, etc)
- Be consistent
- Link to relevant external websites
These are all good general rules for any blog, but they become even more valuable when applied to a content scraping blog. Let’s have a look at what this would entail.
What Does a Scraping Blog Outline Look Like?
The main idea behind a content scraping blog is to pull in as much content as possible, via links or otherwise, and blend it into a central location where it can be accessed by both visitors and search engines.
The most basic plan would be to take a look at the main website, identify all the pieces of text and images that you can find, and then use some tool (we’ll get to that in a bit) to scrape this content and spit it out into a nicely formatted piece of blog content that you can then use to your advantage.
The content can then be used in a number of ways. You could pull in this content into a wordpress blog that you’re currently running (if you’re a solo operator), or into an e-commerce store or community site. You could also use this content to populate a knowledge base or help center for your own website. Or perhaps you’ll even think of a creative way to utilize this content that we haven’t even thought of.
Where Do I Start?
The first thing you’d want to do is take a look at your target website, which we’ll assume for the purposes of this article is business.com. Identify all the pieces of content, whether written or electronic (including images), that you can find. Be sure to include things like:
- Brochures
- Product specifications
- Product reviews
- News articles
- Press releases
- Whitepapers
- Term and condition documents
- User guides
- FAQs
This is a fairly comprehensive list, but you should feel free to brainstorm additional ideas that come to mind. The more content that you can pull in, the greater the potential for your blog to grow.
The Importance Of Templates.
One of the primary reasons why blogging is such a popular choice is because of the simplicity of the process. Blogging platforms, such as WordPress, make it extremely easy to get started with. All you need is a suitable theme (we’ll get to that in a bit) and you can start creating a blog in no time at all.
The beauty of this is that once you’ve got your blog up and running, you can then use it as a resource for future content creation. As you continue to grow and develop your blog, you can then look back at earlier content and refer to it as required. This also makes it much easier to keep track of your content, as you’ll be able to look back over time and see everything that you’ve published. This way you don’t lose track of previous writings and it makes it much easier to keep on top of any new content that you create.
Scraping Web Pages Using Software.
As you might have guessed, one of the best tools for scraping websites is actually built into the platform that you’re using to write your blog. If you’re using WordPress, then you can actually use its built-in web crawler, which we’ll call WordPress SEO, to accomplish this. Simply go to your Settings menu in WordPress and you’ll see an option to activate this feature. Once you’ve done that, you can use the SEO tool to crawl and harvest website content.
SEO stands for Search Engine Optimization and it’s a way of ranking the content that you’ve published on the web so that it appears in the right place in the search results when potential users look for content relating to your niche and topic. When someone searches for business ideas, for example, your website might come up on top of the list of results.
By actively promoting your content on social media platforms like Twitter and Facebook, you can also make sure that your content gets seen by as many people as possible. Don’t expect that your blog will make you famous overnight, but with a little bit of effort, you can certainly achieve a following that you can then leverage to grow your business.
Extracting Metadata.
If we refer back to our list of content, we’ll see that we’ve included quite a bit of text along with the various images. This is where metadata comes in. Metadata is data about data and, in this case, it’s the pieces of text that we’ve pulled in from the website. It’s extremely valuable to a blog owner, especially one who’s invested a lot of time and effort gathering this content, but it’s not something that search engines usually hand out as a matter of course. You’ve got to request this data from the website developer via a metadata API.
Why Use An API?
An API is a Application Programming Interface and it’s a way of connecting one program (or application) to another in a way that makes it easier for them to work together. In our example, we’ve got an API that we need to connect to in order to get access to the business.com website content that we need for our blog.
The first step is to sign up for a free account on API.com. You can do this by entering a valid email address in the sign up form. When you do this, you’ll be greeted by a confirmation email. You can click the link in the email to activate your account. Now that you’ve got an account set up, you can begin creating an API key.
To do this, click on the Settings tab at the top of the dashboard and you’ll see a blue button that says Generate API Key.
You’ll then be presented with a key that you can use to make requests of the business.com website. Simply enter this key in the relevant box in the software that you’re using to pull in content and you’ll see the results.
This is one of the simplest and most effective ways of getting content that you need without having to manually enter the links yourself. In most cases, you can simply cut and paste the results into your blog post, with a couple of clicks of a button. In some instances, you might need to tweak the content a little bit (mostly fixing grammar and typos), but for the most part, the results are quite good.
Choosing The Right Blogging Platform.
Now that you understand the basics of content scraping, it’s time to choose the right blogging platform. You’ve got a couple of options here and you need to decide which one is going to be the best for your needs. If you’re looking for a one-stop-shop for a quick start, then you might want to consider WordPress. But if you’d like to have a more custom experience, then you could look into platforms like Ghost or Jekyll. Let’s discuss each of these a little bit.