How to Set Up a Web Page Scraper?
The Web Page Scraper is one of the most powerful tools that jBoard provides, allowing our customers to add job wrapping service to their job boards. Or just automate the process of adding new jobs to their job board and saving them a lot of time. To create an importer, you just need to provide a link to the web page that contains jobs list and it will automatically import new jobs from the given link into your job board.
Here are detailed steps on how to configure a web page scraper properly.
Click on the "Jobs Aggregation" menu, after click on the "Aggregators" button and "Add Aggregator".
Click on the "Web Page Scraper" button.
Enter the URL of the page from which you would like to import the jobs and click on "Start configuring web page scraper" button.
Check out this article to learn from which job boards the web importer can import jobs.
If the entered URL was correct, you will see the following form on your screen.
While this step is optional, it gives an ability to add additional settings to the importer and make it more agile. Here are a few examples of an additional configuration that you can apply.
Name - Is just the display name of the importer. And will be used when displaying the list of the importers on the admin panel.
Active jobs limit - This is useful if you would like to limit the number of active jobs that will be imported from the given page. For example, if you set it to 10, the importer will import new jobs until the number of active imported jobs reaches 10. Next time importer will import a new job only once at least one of the 10 imported jobs becomes inactive.
Imported job status - When "Confirmed" is selected, all the imported jobs will be automatically visible on the job board. Select "Needs Confirmation" if you want manually approve imported jobs before they get visible on your job board.
Employer - If you are importing jobs from the employer's job board, we recommend you to select the "Select the employer manually" option and then select the employer, so the aggregator will assign all jobs to the given employer. Otherwise, leave this as "Get from job posting" so the aggregator will create a new employer for newly imported jobs.
Posting date - Set the "Posting date" of the imported jobs. Important note: When getting the "posting date" from the job, some of the imported jobs might become automatically expired right after the import, because the original posting date is pretty old. If you would like to prevent that, you will need to set either a higher "Job expires in days" number or just set the posting date as the date of the import, instead of the date of the original posting.
Filter with keywords - Allows importing only jobs that are matching given keywords. For example, if you would like to import only jobs that have "software engineer" OR "designer" keyword, you can set "Filter with keywords" to software engineer, designer.
Exclude keywords - Allows skipping jobs that have ANY of the matching keywords. For example, if you set it to software engineer, designer. The importer will skip any job that contains software engineer or designer in the keywords but will not skip the jobs that contain only engineer in the keywords.
Check only in job title By default, both Filter with keywords and Exclude keywords are checking inside the job description and title. If you would like to perform the check only against the job title, check the "Check only in job title" for the given filter.
Now, all we have to do is just press the create button to create the aggregator.
Once the aggregator is created, it will run immediately. On average it takes 20-30 min to run the aggregator for the first time. But it might take as long as 6 hours, depending on the job board from which you are trying to import the jobs and also on the number of jobs that you want to import.
Once the aggregator finishes the job, you will see the status of the aggregator updated to green and it will show you how many jobs have been imported. You can view all the imported jobs by clicking on the "Total X Jobs Imported" button.
Here are detailed steps on how to configure a web page scraper properly.
Step 1: Open the Add New Web Page Scraper Page
Step 1.1: Open the "Aggregators" page
Click on the "Jobs Aggregation" menu, after click on the "Aggregators" button and "Add Aggregator".
Step 1.2: Select web page importer
Click on the "Web Page Scraper" button.
Step 2: Enter the Web Page URL from Which You Would like to Import the Jobs
Enter the URL of the page from which you would like to import the jobs and click on "Start configuring web page scraper" button.
Check out this article to learn from which job boards the web importer can import jobs.
Step 3: Configure the Importer (Optional)
If the entered URL was correct, you will see the following form on your screen.
While this step is optional, it gives an ability to add additional settings to the importer and make it more agile. Here are a few examples of an additional configuration that you can apply.
Name - Is just the display name of the importer. And will be used when displaying the list of the importers on the admin panel.
Active jobs limit - This is useful if you would like to limit the number of active jobs that will be imported from the given page. For example, if you set it to 10, the importer will import new jobs until the number of active imported jobs reaches 10. Next time importer will import a new job only once at least one of the 10 imported jobs becomes inactive.
Imported job status - When "Confirmed" is selected, all the imported jobs will be automatically visible on the job board. Select "Needs Confirmation" if you want manually approve imported jobs before they get visible on your job board.
Employer - If you are importing jobs from the employer's job board, we recommend you to select the "Select the employer manually" option and then select the employer, so the aggregator will assign all jobs to the given employer. Otherwise, leave this as "Get from job posting" so the aggregator will create a new employer for newly imported jobs.
Posting date - Set the "Posting date" of the imported jobs. Important note: When getting the "posting date" from the job, some of the imported jobs might become automatically expired right after the import, because the original posting date is pretty old. If you would like to prevent that, you will need to set either a higher "Job expires in days" number or just set the posting date as the date of the import, instead of the date of the original posting.
Filter with keywords - Allows importing only jobs that are matching given keywords. For example, if you would like to import only jobs that have "software engineer" OR "designer" keyword, you can set "Filter with keywords" to software engineer, designer.
Exclude keywords - Allows skipping jobs that have ANY of the matching keywords. For example, if you set it to software engineer, designer. The importer will skip any job that contains software engineer or designer in the keywords but will not skip the jobs that contain only engineer in the keywords.
Check only in job title By default, both Filter with keywords and Exclude keywords are checking inside the job description and title. If you would like to perform the check only against the job title, check the "Check only in job title" for the given filter.
Step 4: Create Aggregator
Now, all we have to do is just press the create button to create the aggregator.
Step 5: Wait till Import Finishes
Once the aggregator is created, it will run immediately. On average it takes 20-30 min to run the aggregator for the first time. But it might take as long as 6 hours, depending on the job board from which you are trying to import the jobs and also on the number of jobs that you want to import.
Once the aggregator finishes the job, you will see the status of the aggregator updated to green and it will show you how many jobs have been imported. You can view all the imported jobs by clicking on the "Total X Jobs Imported" button.
Updated on: 01/10/2024
Thank you!