eatmoreduck/boss-zhipin-scraper

Boss直聘爬虫 / BOSS直聘职位数据抓取工具，基于 Chrome CDP 协议复用真实登录态，绕过字体反爬，输出明文薪资 JSON/CSV + 薪资技能分析。A Chrome-CDP-based BOSS Zhipin job scraper/crawler.

Overview

boss‑zhipin‑scraper is a Python command‑line tool that extracts job listings from the Chinese recruitment platform BOSS 直聘. It connects to a locally‑running Chrome or Edge browser via the Chrome DevTools Protocol (CDP), re‑uses the user’s logged‑in session, calls the site’s own search API and returns the results with clear salary information (the API already provides the salary in plain text, avoiding the site’s font‑based obfuscation). The tool can also fetch each job’s detailed description, analyse the collected data and produce a short summary plus prompt suggestions for improving a résumé.

Key Features

Feature	What it does
Plain‑text salary	Uses the site’s search API (captured via CDP) so salaries are returned directly, bypassing the font‑based anti‑scraping tricks.
Incremental runs	New jobs are flagged with `is_new`; results are written after each page so a crash does not lose data.
Multiple output formats	Saves listings as JSON or CSV (both list and detail files).
Job‑detail crawling & skill analysis	Optionally fetches the JD page, extracts frequent skill terms and other statistics.
Aggregated summary & prompt generation	`job_summary.py` creates salary distribution, experience/education requirements, top skills, and a ready‑to‑copy prompt for LLMs to help polish a résumé.
No Selenium/Playwright	Direct CDP connection to an already‑logged‑in browser reduces fingerprinting and avoids the heavy browsers used by typical scrapers.
Cross‑platform	Works on macOS, Linux and Windows; supports Chrome, Chromium and Edge.
Hermes/Agent skill integration	Packaged as a Hermes skill and can be installed via `hermes skills install`, `skills.sh`, or ClawHub for AI‑assistant agents.

How It Works (High‑Level)

Setup – --setup-chrome (or --setup-edge) launches a dedicated browser profile, asks the user to log in to BOSS 直聘, and stores the cookies locally.
Capture API traffic – Using CDP’s Network domain, the script passively listens to the search requests the page itself makes when the user scrolls.
Parse results – The JSON response contains fields like salaryDesc, securityId, lid, etc. The script writes each job to a file, de‑duplicates by job_id, and marks newly‑seen entries.
Detail page (optional) – If --detail is enabled, the script navigates to each job’s detail URL, extracts the JD text and runs a simple word‑frequency count for skill terms.
Post‑processing – job_summary.py reads the collected JSON/CSV, computes statistics (salary ranges, experience levels, city distribution, top companies/skills) and prints a concise summary plus an LLM‑friendly prompt.

Typical Usage

# Install
git clone https://github.com/eatmoreduck/boss-zhipin-scraper.git
cd boss-zhipin-scraper
pip install -r requirements.txt

# Prepare a logged‑in browser profile
python3 scripts/boss_cdp_raw.py --setup-chrome   # or --setup-edge

# Grab the first 3 pages of AI‑Agent jobs in Shanghai and run analysis
python3 scripts/boss_cdp_raw.py \
    --keyword "AI Agent" \
    --city 上海 \
    --pages 3 \
    --format csv \
    --analysis

# Produce a summary and a prompt for résumé improvement
python3 scripts/job_summary.py --top 15

The commands can be combined with flags such as --list-cities, --diff-base (to see which jobs are new), or --close‑chrome to automatically shut the helper browser after a successful run.

Who Might Benefit?

Job seekers looking for up‑to‑date market data on salaries, required skills and company trends in China.
Career‑coach LLM agents that need structured job‑market inputs to generate résumé suggestions or interview prep material.
Researchers studying Chinese tech‑job trends who need a reproducible way to collect large, clean datasets.

Limitations & Legal Note

The tool is for research and personal use only; it respects BOSS 直聘’s terms of service and requires the user’s own login credentials.
It does not bypass any authentication wall – if the site blocks the request (e.g., rate‑limit or anti‑bot code 31/37), the scraper stops and reports the issue.
Salary data is only as reliable as the API response; enabling --allow-dom-fallback may pull obfuscated values that are less trustworthy.

License

MIT – free to use, modify, and redistribute.