Cara menggunakan python-requests preparedrequest

Requests merupakan modul Python yang bisa kamu gunakan untuk mengirim berbagai request HTTP. Requests adalah library yang memiliki banyak fitur mulai dari melempar parameter dalam URL sampai mengirim header khusus dan verifikasi SSL.

HTTP mendefinisikan seperangkat metode permintaan untuk menunjukkan tindakan yang diinginkan yang akan dilakukan untuk sumber daya tertentu. Meskipun mereka juga bisa menjadi kata benda, metode permintaan ini kadang-kadang disebut sebagai verba HTTP. Masing-masing menerapkan semantik yang berbeda, namun beberapa fitur umum dapat digunakan bersamaan, contohnya Metode permintaan dapat berupa safe, idempotent, atau cacheable.

Pada kesempatan kali ini kita akan mempelajari lebih jauh tentang Python Requests. selama ini kita sudah sering mendengar atau bahkan menggunakan http methods GET dan POST. taukah kamu bahwa HTTP Methods memiliki sedikitnya 7 methods apa saja itu ?

Requests is an elegant and simple Python library built to handle HTTP requests in python easily. It allows you make GET, POST, PUT and other types of requests and process the received response in a flexible Pythonic way.

Contents

1. Introduction to Requests Library
2. What is a GET and POST request?
3. GET Method
4. Status Code
5. Contents of the Response Object
6. The Content
7. Full HTML source as Text
8. Retrieving an image from website
9. Headers
10. How to set Query String Parameters
11. POST Method
12. PUT Method
13. DELETE Method
14. PATCH Method
15. HEAD Method
16. Request Header
17. Inspecting the request made
18. Authentication
19. Time out
20. SSL Certification

Introduction to Requests Library

Requests is an elegant and simple Python library built to handle HTTP requests in python easily. But what is a HTTP request? HTTP is a set of protocols designed to enable communication between clients and servers.

A client is typically a local computer or device similar to what you are using to view this page. A HTTP request is the message sent (or received) from your local computer to a web server hosting, typically hosting a website.

For example, when you go to any internet website from your browser, the browser is sending a HTTP request and receives an appropriate ‘response’ from the host server.

Requests is an easy-to-use library with a lot of features ranging from passing additional parameters in URLs, sending custom headers, SSL Verification, processing the received response etc.

What is a GET and POST request?

A GET request is used to request data from a specific server. It is the most common type of request.

This is synonymous to you visiting the homepage of a website from your browser. Another common type of request is the POST request, which is used to send data to a host server for further processing, like, to update a resource, such as a database.

What is this synonymous to in real world?

For example, most data that you submit through forms in various websites is sent and processed as a POST request. Besides this, using requests you can add additional content like header information, form data, multipart files, and parameters via simple Python libraries. You don’t need to manually append the query strings to your URLs.

What does that practically mean?

For example, if you search for something, say the string ‘babies’ in google, your browser sends a GET request to Google server by appending the query string to the url.

So if you check the url in your address bar, you should see something like:

print(r.status_code)

With

print(r.status_code)

print(r.status_code)

What makes requests really stand out is, the received response is packages as a standardized

200

Let’s start by downloading the requests library using:

200

MLPlus Industry Data Scientist Program

Struggling to find a well structured path for Data Science?

Build your data science career with a globally recognised, industry-approved qualification. Solve projects with real company data and become a certified Data Scientist in less than 12 months and get Guaranteed Placement. .

Tap to know more

Get Free Complete Python Course

Build your data science career with a globally recognised, industry-approved qualification. Get the mindset, the confidence and the skills that make Data Scientist so valuable.

!pip install requests

Then import the library to use it in your program use the

200

import requests
from pprint import pprint  # prettyprint

Now let’s look into the details of request library and some of the important features.

GET Method

Lets try to get some information from the official python website – https://www.python.org/. You can use any public site. All you need to do is call

200

When you ping a website or portal for information it’s considered as ‘making a request’.

print(r.status_code)

r = requests.get('https://www.python.org/')

The information that we got from the website will be stored in the

200

200

200

Now as I have requested the data from the website, to check whether it has worked properly, let’s try priniting the

200

print(r)

<Response [200]>

You can see that we have got

200

STATUS Code

Status codes are issued by a server in response to a client’s request made to the server. Use the

if r.status_code == 200:
    print('Success')
elif r.status_code == 404:
    print("Page not found")

print(r.status_code)

200

We have got a response of 200 which means the request is success. A response of 200 means Success. A response of 300 means Redirected. A response of 400 means Client Error.

A response of 500 means Server Error. A response of 404 means Page Not Found Error. In general, any status code less than 400 means, the request was processed successfully.

If it is 400 and above, some sort of error occurred. You may need to use the if else statement about whether to further proceed into the code depending upon the status code you recieved from the server, as you don’t need to further run your program after you have got an error. Use the following code below in that case:

if r.status_code == 200:
    print('Success')
elif r.status_code == 404:
    print("Page not found")

Success

Contents of the Response object

if r.status_code == 200:
    print('Success')
elif r.status_code == 404:
    print("Page not found")

r_attribs = [c for c in dir(r) if not c.startswith("_")]
r_attribs

import requests
from pprint import pprint  # prettyprint

You can see that there are several commands available such as

if r.status_code == 200:
    print('Success')
elif r.status_code == 404:
    print("Page not found")

if r.status_code == 200:
    print('Success')
elif r.status_code == 404:
    print("Page not found")

if r.status_code == 200:
    print('Success')
elif r.status_code == 404:
    print("Page not found")

if r.status_code == 200:
    print('Success')
elif r.status_code == 404:
    print("Page not found")

if r.status_code == 200:
    print('Success')
elif r.status_code == 404:
    print("Page not found")

import requests
from pprint import pprint  # prettyprint

import requests
from pprint import pprint  # prettyprint

The Content

The output from the

print(r.status_code)

200

if r.status_code == 200:
    print('Success')
elif r.status_code == 404:
    print("Page not found")

This is the raw output of the html content behind the URL we requested, which in this case is https://www.python.org/.

import requests
from pprint import pprint  # prettyprint

import requests
from pprint import pprint  # prettyprint

While

Success

You get that directly by using another stored attribute in the response called the

Success

The Full HTML source as Text

Use the

Success

import requests
from pprint import pprint  # prettyprint

import requests
from pprint import pprint  # prettyprint

Retrieving an image from the website

Use the same

print(r.status_code)

Success

200

if r.status_code == 200:
    print('Success')
elif r.status_code == 404:
    print("Page not found")

This means, whatever be the content of the received response, be it text or image, it is stored in the

if r.status_code == 200:
    print('Success')
elif r.status_code == 404:
    print("Page not found")

import requests
from pprint import pprint  # prettyprint

The image from the website will be downloaded to the folder in which you are running the program.

Headers

Most webpages you visite will contain header, which contains various metadata. Use the

Success

Success

Success

More information like metadata about the response, it is stored in the header. It gives you many information such as the content type of the response payload, a time limit on how long to cache the response, and more. This will return you a dictionary-like object, allowing you to access header values by key.

import requests
from pprint import pprint  # prettyprint

import requests
from pprint import pprint  # prettyprint

As you can see, it gives information about the content type, last modified date, Age of the website etc.. You can access each one of these by considering the ouput from the function as dictionary.

r = requests.get('https://www.python.org/')

r = requests.get('https://www.python.org/')

Advanced Functions

Now as we have looked into the basics of the requests library, lets dive into some of the advanced functions From now onwards, I will be using the website – http://httpbin.org/ to retrive as well as send and store information. Let’s try the commands that you have learned so far.

How to Set Query String Parameters

Often the response given by the server differs based on the query you send. Like, you want to view the 2nd page of a 10 page article, instead of the first page.

Or you want to search for a particular term in a website. In such cases, you will send additional parameters as part of the URL as a query. For example:

print(r.status_code)

Depending on the device you are using, location, referring source, user etc, these queries can easily get complicated. So instead of adding it in the url directly, using

print(r.status_code)

r_attribs = [c for c in dir(r) if not c.startswith("_")]
r_attribs

r_attribs = [c for c in dir(r) if not c.startswith("_")]
r_attribs

r_attribs = [c for c in dir(r) if not c.startswith("_")]
r_attribs

r_attribs = [c for c in dir(r) if not c.startswith("_")]
r_attribs

This essentially translates as “http://httpbin.org/?page=5&count=10“.

r = requests.get('https://www.python.org/')

r = requests.get('https://www.python.org/')

You can see that I have first created a dictionary for the parameters and then I passed it into the

r_attribs = [c for c in dir(r) if not c.startswith("_")]
r_attribs

To check whether the parameter passing has worked properly, use

r_attribs = [c for c in dir(r) if not c.startswith("_")]
r_attribs

r = requests.get('https://www.python.org/')

r = requests.get('https://www.python.org/')

It has worked properly as the page is set to 5 and the count is set to 10. You can also pass it as a tuple or a byte which will give the same output.

r = requests.get('https://www.python.org/')

r = requests.get('https://www.python.org/')

POST Method

The POST method is used to submit data to be further handled by the server. The server typically understands the context and knows what to do with the data.

Generally it’s used while submitting a web form or when uploading a file to the server. The

r_attribs = [c for c in dir(r) if not c.startswith("_")]
r_attribs

r_attribs = [c for c in dir(r) if not c.startswith("_")]
r_attribs

r = requests.get('https://www.python.org/')

r = requests.get('https://www.python.org/')

You can see in the form type the

import requests
from pprint import pprint  # prettyprint

import requests
from pprint import pprint  # prettyprint

To process the received json response, iterate through the contents of

import requests
from pprint import pprint  # prettyprint

Or if you know what the contents, you can access it directly as you would with a

import requests
from pprint import pprint  # prettyprint

print(r)

print(r)

Post function can be used to send large amount of data (text / binary) data.

PUT Method

The PUT method requests that the data you are sending to be stored under the supplied URL. If the URL refers to an already existing resource, it is modified and if the URL does not point to an existing resource, then the server creates the resource with that URL. As you can see, the PUT is somewhat similar in functionality to POST.

So what is the difference between PUT and POST? The difference is, the POST method sends data to a URI and the the receiving resource understands the context and knows how to handle the request. Whereas, in a PUT method, if there is a file in the given URI, it gets replaced. And if there isn’t any, a file is created.

Besides, not matter how many times you execute a given PUT request, the resulting action is always the same. This property is called idempotence. Whereas, for a POST method, the response need not always be the same. This makes the PUT method idempotent and the POST method is not. To make a PUT request, use the

import requests
from pprint import pprint  # prettyprint

print(r)

print(r)

Generally in practice,

import requests
from pprint import pprint  # prettyprint

import requests
from pprint import pprint  # prettyprint

DELETE Method

The delete() method sends a DELETE request to the specified url. DELETE requests are made for deleting the specified resource (file, record etc). A successful response should be:

1. 200 (OK) if the response includes an entity describing the status.
2. 202 (Accepted) if the action has not yet been enacted
3. 204 (No Content) if the action has been enacted but the response does not include an entity.

print(r)

print(r)

The

import requests
from pprint import pprint  # prettyprint

PATCH Method

The PATCH method is a request method supported by the HTTP protocol for making partial changes to an existing resource. The main difference between the PUT and PATCH method is that the PUT method uses the request URL to supply a modified version of the requested resource.

And it replaces the original version of the resource. Whereas, the PATCH method only supplies a set of instructions to modify the resource. This means, the PATCH request needs to contain only the changes that needs to be applied to a resource and not the entire resource. Although it resembles PUT, it typically contains a set of instructions that tell how a resource residing at a URI should be modified to produce a new version. Use the

import requests
from pprint import pprint  # prettyprint

print(r)

print(r)

HEAD Method

import requests
from pprint import pprint  # prettyprint

r_attribs = [c for c in dir(r) if not c.startswith("_")]
r_attribs

This method is often used for testing hypertext links for validity, accessibility, and recent modification. You can do this by using the

import requests
from pprint import pprint  # prettyprint

print(r)

print(r)

Let’s run the same as a GET request and see the difference.

<Response [200]>

<Response [200]>

Notice, we received only header content with

import requests
from pprint import pprint  # prettyprint

Request Header

A request header is an HTTP header that can be used in an HTTP request, and that doesn’t relate to the content of the message.

To customize headers, you pass a dictionary of HTTP headers to

r_attribs = [c for c in dir(r) if not c.startswith("_")]
r_attribs

if r.status_code == 200:
    print('Success')
elif r.status_code == 404:
    print("Page not found")

<Response [200]>

<Response [200]>

Inspecting the request made

When you make a request, the requests library prepares the request before actually sending it to the destination server. Request preparation includes things like validating headers and serializing JSON content.

Only after preparing the request, the request will be sent to the destination server. You can view the PreparedRequest by accessing the json.

<Response [200]>

<Response [200]>

This helps you in getting access to informations like payload, URL, headers, authentication, and more.

<Response [200]>

<Response [200]>

You can see that I have stored the ouput of the

import requests
from pprint import pprint  # prettyprint

Authentication

Authentication helps a service understand who you are. You provide your credentials to a server by passing data through the Authorization header or a custom header defined by the service. You need to use the

import requests
from pprint import pprint  # prettyprint

<Response [200]>

<Response [200]>

What we are doing is we are giving our data to the server by passing data through the

import requests
from pprint import pprint  # prettyprint

import requests
from pprint import pprint  # prettyprint

import requests
from pprint import pprint  # prettyprint

import requests
from pprint import pprint  # prettyprint

import requests
from pprint import pprint  # prettyprint

The authentication output comes out to be ‘true’ which means that our username and password is correct. If our password is wrong, we wont be getting any output for authentication.

print(r.status_code)

print(r.status_code)

You can see that if I use the wrong username, I am getting an 401 error. When you pass your username and password in a tuple to the auth parameter, requests is applying the credentials using HTTP’s Basic access authentication scheme under the hood.

Time out

When you send a request to a server, your system typically waits for a certain amount of time for the other server to respond. If this takes too much times, then there is a possibility that your system will hang.

Time-out is set to make sure that the if the website is not responding for a certain amount of time it has to stop loading. If we dont set the timeout then the website will be loading forever if the server is not responding. Use the

import requests
from pprint import pprint  # prettyprint

print(r.status_code)

In the above case, I have set the time limit to 5 seconds.

You can also pass a tuple to timeout with the first element being a connect timeout (the timeout allows for the client to establish a connection to the server), and the second being a read timeout (the time it will wait on a response once your client has established a connection):

print(r.status_code)

This means that the request should establish a connection with server before 3 seconds and the data must be recieved within 7 seconds after the connection is established.

If the request times out, then the function will raise a Timeout exception. You can handle this exception as well by importing

import requests
from pprint import pprint  # prettyprint

import requests
from pprint import pprint  # prettyprint

print(r.status_code)

SSL Certificate Vertification

SSL Certificates are small data files that digitally bind a cryptographic key to an organization’s detail. An organization needs to install the SSL Certificate onto its web server to initiate a secure session with browsers.

Once a secure connection is established, all web traffic between the web server and the web browser will be secure. If the data you are trying to receive or send is sensitive, then it is done by establishing a encrypted connection using SSL. requests library does this for you by default. If you don’t want to do this, then we can set the paramter verify to be false in the

r_attribs = [c for c in dir(r) if not c.startswith("_")]
r_attribs