To get the crawling pun out of the way: Spiderman isn’t everyones favourite superhero. With the issues around crawling (or scraping, as it is otherwise called), recruitment may have to opt for another superhero altogether. A crawler, in essence, is nothing but a program that automates the search and indexing of data, which can then be put to various and unique applications.
“Scraping or crawling is a process in which a bot visits websites in search of content it can use”, says Jelmer Zuidema, co-founder of Netherlands-based Roadtrip. “The most famous one is Google — they crawl your website to make sure they can present it in their search engine. But the way Google operates isn’t the problem, it lies with the way recruiting scrapers operate.”
‘They hold candidates hostage’
Recruiting scrapers don’t just look for any type of content, they’re probing for one particular thing: vacancies. “They look at your website and take anything they can use until they’re able to amass a relatively neat looking job ad”, Zuidema says. “Then, they essentially hold candidates hostage: if an applicant doesn’t fill in an e-mail address on their website, they can’t view the ad.”
“Some scrapers even add information to make a vacancy look more appealing. In many ways it is the purest form of clickbait.”
But that’s not where the problem ends, Zuidema argues. “Some scrapers even add information to make a vacancy look more appealing. In many ways it is the purest form of clickbait: the more people click on certain link, the better it is for them. Then, when that bit of wrong information continues to live on in virtual spheres for days, or sometimes even weeks — it all leads to one thing: more clicks.”
When it’s closed — it’s really not
In even worse scenario’s, jobs that have long been filled continue to roam around freely on scraping websites. “Just as changing something within a job ad is considered an update, the same applies to the closing of a job ad”, Zuidema says. “When you do close it, it’ll remain on open for a quite a bit of time on scraping sites. It all leads to you losing control over the display and performance of your ads.”
“You can opt for a paid collaboration with those scraping your jobs — though I would forget about this solution straight away.”
Zuidema says organisations who struggle with the way their jobs are scraped may opt for one of many solutions. “You can opt for a paid collaboration with those scraping your jobs — though I would forget about this solution straight away”, he says. “Other options are blocking’s third-party posters, using LinkedIn’s or Indeed’s option for XML feeds, or offering the structured data Google for Jobs longs for.”
