Forum in maintenance, we will back soon 🙂
Notifications
Clear all
Python Scripting
2
Posts
2
Users
0
Reactions
527
Views
Topic starter
Step 1: I did a job search on Indeed.com and used the link as below in puthin
from bs4 import BeautifulSoup
import requests
url="https://www.indeed.com/jobs?q=IT+director&l=Remote&from=searchOnHP&vjk=84953521ad7c4774"
req=requests.get(url)
soup=BeautifulSoup(req.text,"html.parser")
# Find all the job posts
job_posts = soup.find_all('meta', name="description")
# Print the title of each job post
for job_post in job_posts:
title = job_post.a.text
print(title)
Step 2: Verified the html code has the description as below.
<meta http-equiv="content-type" content="text/html; charset=utf-8"> <meta name="description" content="1,349 IT Director jobs available in Remote on Indeed.com. Apply to Director of Information Technology, Director of Partnerships, Director of Analytics and more!"> <meta name="referrer" content="origin-when-cross-origin">
Step 3: Running the python code in step 1 is giving lots of errors as below
line 507, in send
raise ConnectTimeout(e, request=request)
raise ConnectTimeout(e, request=request)
What is that I am doing wrong?
Posted : 09/08/2023 4:29 pm
Indeed.com, like many other websites, has terms of service that prohibit scraping.
Posted : 09/09/2023 7:09 am
