How to get data from website that does not provide API

There are lot of junk data now-a-days around the world and in this time if you find a good website from where you want to get data programmatically in your code but you realize they do not provide an api then it will make you mad. But I have a solution by which you can get data from website that does not provide an API and it will be a JSON response. Yes you heard it right! It will be a JSON response. Let’s start the journey.

I will take an example of a news website The Hindu. I will show you how you can search for a query in their site and get the JSON response.

Search Engine Setup

I will help you setup a Search Engine at Google which will search for the data at their website and get back to you with JSON response.

• Go to Programmable Search Engine
• It will ask for the Google login so login with your Google Account.
• Click on “Add” button to fill up your search engine details
• In the “Name your search engine” you can enter any name ex “My Search Engine”
• In the “What to search?” text enter the website url you want to search ex www.thehindu.com/* and click “Add” button.
• Once all details are filled, click on “Create” button
• It will create your own search engine for that particular website

Customize Your Search Engine

Now that you have created your own search engine, it’s time to customize it and get the API out of it.

Click on the “Customize” button in the dashboard where you just added the website.

Copy the “cx” value from the “Public URL” at Search Engine dashboard.

Get the API Key

The next step is to get the Programmable Search Engine API Key.

• Open the link here.
• Click on “Get a Key”
• Select a project from dropdown or create one if you do not have any project.
• Click on “Next” button and it will give your Key.
• Copy the Key value.

Call the API

The final step is to call the Search API. Use the below GET API to search for a query on the website.

Replace <your key> with your Key Value. <your cx value> with your cx value. and put a query to search for.

Conclusion

We learnt about how to make our own custom search engine and search for contents on a website even if that website is not providing any API. Google Programmable Search Engine is giant tool to search for contents throughout the Web or a Particular website.

If you are looking to search for a post or keyword in different Social Media Platforms using your Node.Js code then you must follow this SHAHED NASSER‘s blog to search through different Social Media Platforms using Node.Js.