How to Manage Cookies with requests
in Python
Cookies are a fundamental part of web interactions, used for maintaining state, tracking user sessions, and storing preferences.
This guide explains how to get, set, and manage cookies when making HTTP requests using the requests
library in Python.
Getting Cookies with requests
There are two main ways to access cookies received in an HTTP response:
Using a Session
Object
The requests.Session
object automatically manages cookies across multiple requests. This is the recommended approach for most situations, as it handles cookie persistence and expiration correctly:
import requests
session = requests.Session() # Create a session object
print(session.cookies.get_dict()) # Output: {} (initially empty)
response = session.get('http://google.com') # Make a request that sets cookies
print(session.cookies.get_dict()) # Output: (A dictionary of cookies, e.g., {'CONSENT': 'YES+...'})
- You can also use the
with
statement to create and handle sessions:
import requests
with requests.Session() as session: # Automatically closes
print(session.cookies.get_dict()) # Output: (A dictionary of cookies, e.g., {'CONSENT': 'YES+...'})
response = session.get('http://google.com')
print(session.cookies.get_dict()) # Output: {'CONSENT': 'YES+...'}
session.cookies
: This attribute holds aRequestsCookieJar
object, which acts like a dictionary.session.cookies.get_dict()
: This returns a standard Python dictionary containing the cookies, making it easy to access them by name.
Accessing Cookies Directly from a Response
You can also access cookies directly from a Response
object, but this is generally less convenient than using a Session
:
import requests
response = requests.get('http://google.com')
print(response.cookies.get_dict()) # Output: (A dictionary of cookies)
response.cookies
: This also holds aRequestsCookieJar
object. The difference is that these cookies are only for this specific response and won't be automatically used in subsequent requests (unless you manually manage them).
Accessing Cookie Attributes (Domain, Path, etc.)
To access specific cookie attributes (like domain, path, expiry), iterate through the RequestsCookieJar
:
import requests
session = requests.Session()
response = session.get('http://google.com')
cookie_data = [
{'name': c.name, 'value': c.value, 'domain': c.domain, 'path': c.path}
for c in session.cookies # Iterate through the RequestCookieJar
]
print(cookie_data)
# Example Output (will vary):
# [{'name': 'CONSENT', 'value': 'YES+cb.202310...', 'domain': '.google.com', 'path': '/'}]
- The code iterates over the
RequestCookieJar
which will return cookie objects for each of the cookies. - Each
c
in the loop is aCookie
object, with attributes like.name
,.value
,.domain
,.path
,.expires
, etc.
Sending Cookies with Requests
To send cookies with a request, use the cookies
parameter:
import requests
session = requests.Session()
response = session.get(
'https://httpbin.org/cookies', # httpbin.org echoes back the cookies
cookies={'my-cookie': 'my-value'}
)
print(response.text)
Output:
{
"cookies": {
"my-cookie": "my-value"
}
}
cookies={'my-cookie': 'my-value'}
: Pass a dictionary of cookie name/value pairs. Therequests
library handles the proper formatting for the HTTP request.
Persisting Cookies to a File
To save cookies to a file and load them later (e.g., to maintain a session across program executions):
Saving Cookies
import json
import requests
response = requests.get('http://google.com')
with open('cookies.txt', 'w', encoding='utf-8') as f:
json.dump(
requests.utils.dict_from_cookiejar(response.cookies), # Convert to dict
f
)
requests.utils.dict_from_cookiejar()
converts a RequestCookieJar to a standard dictionary.- The dictionary is then saved as a json file.
Loading Cookies
import json
import requests
session = requests.Session() # Create a new session
with open('cookies.txt', 'r', encoding='utf-8') as f:
cookies = requests.utils.cookiejar_from_dict(json.load(f)) # Convert dict back to jar
session.cookies.update(cookies) # Add cookies to the session
print(session.cookies.get_dict()) # Output: {'CONSENT': 'YES+...'} # Cookies are loaded!
- The session created can now be used to automatically send previously saved cookies.