How to Set Up a Web Page Scraper?

he Web Page Scraper automatically pulls job listings from any page that uses Google's structured job data — like Lever, Greenhouse, or Workday boards. Once set up, it re-runs on a schedule and keeps your listings in sync.

Step 1: Go to Jobs Aggregation → Aggregators

In your admin panel, navigate to Jobs Aggregation → Aggregators, then click + Add Aggregator.

Step 2: Choose Web Page Scraper

Select Web Page Scraper from the two options.

Step 3: Enter the jobs listing URL

Paste the URL of the full job listing page — for example: https://www.spacex.com/careers/jobs. Then click Start configuring web page scraper.

Step 4: Configure your settings

If the URL was valid, you'll see a configuration form. Here's what each field does:

• Name — Auto-filled from the URL. This is an internal name used to identify the aggregator in your admin panel.

• Active jobs limit — Leave blank for unlimited, or set a cap on how many active jobs can be imported at once.

• Active — Keep checked so the importer runs on schedule.

• Employer — "Get from job posting" reads the company name from each listing. You can also pin all imported jobs to a specific employer.

• Imported job status — "Confirmed" publishes jobs immediately. "Needs confirmation" holds them for manual review before they appear on your board. Learn how to moderate imported jobs →

• Category — Optionally assign all scraped jobs to one category.

• Posting date — Use the original posting date from the source, or set it to the import date.

• Job expires — Use the expiry date from the source, or set a fixed one.

Step 5: Advanced Settings (optional)

Expand Advanced Settings for additional controls:

• Filter with keywords / Exclude keywords / Check only in job title — Narrow down which jobs get imported based on keywords.
• Import frequency — How often the scraper re-runs. Default is every 24 hours.
• UTM source — Typically your job board's name. Appended to redirect URLs so the source platform can track traffic coming from your board.
• Block search indexing — Prevent search engines from indexing imported jobs from this aggregator specifically.
• Overwrite manual edits — Controls how the web page scraper handles imported jobs that have been manually edited by the admin. See the FAQ below for details.

Step 6: Click Create

Click Create to launch the scraper. It runs immediately — the first run usually takes 20–30 minutes.

While running, the status shows Running now with an orange badge.

When done, the status turns green and shows how many jobs were imported. You can click the jobs count to view all imported listings.

FAQs

Does aggregating jobs create an employer account?
It depends on the company name. If the employer name in the job listing exactly matches an existing employer on your job board, the jobs are added under that employer. If there's no match, a new employer profile is created automatically.

If the employer already has an account, are imported jobs synced to it?
Yes — as long as the company name matches exactly. Even a small difference in spelling or formatting will result in a separate employer profile being created instead.

Can the admin edit imported jobs?
Yes. However, once an imported job is manually edited, the scraper will stop updating that job on future runs — to preserve your changes. If you want the scraper to keep overwriting with the latest data from the source, set Overwrite manual edits to YES in Advanced Settings.

What does "Overwrite manual edits" do?

• NO (default): If an admin edits an imported job, the scraper stops updating that job on future runs. Your manual edits are preserved.
• YES: The scraper overwrites the job on every run, regardless of any manual edits. Useful when you always want the latest data from the source.

Note: Job expiry always follows the source feed, regardless of this setting.

Updated on: 19/10/2026