Python Developer - Web Scraping

Key Responsibilities :

• Develop and maintain robust web scraping and web crawling solutions using Python.
• Build data extraction pipelines to collect, clean, transform, and store information from multiple web sources.
• Develop and optimize scraping solutions for both static and dynamic websites.
• Work with frameworks and libraries such as Requests, Scrapy, BeautifulSoup, Selenium, Puppeteer, or Playwright.
• Design efficient solutions for handling large-scale data extraction and crawling requirements.
• Integrate scraped data with SQL databases and data warehousing platforms.
• Perform data validation, cleansing, and quality checks to ensure accuracy and consistency.
• Troubleshoot scraping failures, website changes, anti-bot challenges, and performance issues.
• Optimize crawlers for scalability, reliability, and execution speed.
• Collaborate with data, engineering, and business teams to understand data requirements and deliver effective solutions.

Required Skills :

• 5+ years of experience in Python development with strong hands-on programming expertise.
• Proven experience in Web Scraping / Web Crawling.
• Strong knowledge of Python programming and scripting.
• Hands-on experience with SQL and any database or data warehousing technology.
• Experience with one or more of: Requests, Scrapy.
• Experience with one or more of: Puppeteer, Playwright, BeautifulSoup, Selenium.
• Knowledge of PySpark is preferred.
• Strong understanding of HTML, CSS, DOM, HTTP requests, and web technologies.
• Good debugging, analytical, and problem-solving skills.

Preferred Candidate Profile :

• Strong hands-on experience in building production-grade web scraping and crawling solutions.
• Experience working with large datasets and high-volume data extraction.
• Ability to work independently and handle complex scraping requirements.
• Strong understanding of data processing, database integration, and automation.