API Changelog

API v5 bulk jobs (bulk_jobs) — hand over up to 5,000 items and get a job id right away

Send POST /v5/api/collections/bulk_jobs up to 5,000 items (same shape as collections/bulk). Before starting anything it answers 202 with a job id and processes the items in the background. Read progress and item results (the datasets created) with GET /v5/api/collections/bulk_jobs/{job_id}. Each item is handled and charged exactly like collections/bulk. The existing collections/bulk (up to 100, responds after handling) is unchanged.

Action — Nothing to change. If your integration sends more than 100 items, or loses the connection while waiting for a slow response, move to many at once.

API v5 released: bots, settings, plans and datasets

The new version /v5/api is open. Resources use the same words as the screens (bots, settings, plans, datasets), a bot can have several plans, and a dataset tells you whether it was manual or which plan made it (trigger, plan) and the conditions used (inputs). Starting a collection, creating a plan and retry require the Idempotency-Key header (a retry de-duplication key). The same API key is used.

The webhook body gained trigger, plan and inputs after the existing fields (existing fields and X-HashScraper-Version are unchanged).

Responses and behavior of V2, V3 and V4 did not change.

Action — Nothing to change. To move to v5, read Moving from v4 to v5. Note that, unlike v4 param_save (false), the default of save_to_settings is true.

Canceling a data set that has already ended no longer changes its status

A cancel request (V4 schedule_results/:id/cancel, V2 cancel_collect) on a data set that has already ended (finish/complete, canceled, or failed) still returns success but leaves the status unchanged. The V4 response returns the current sr_status. Previously it was overwritten to canceling.

Action — No change required. If your code assumes the cancel response's sr_status is always canceling, note that an ended status may be returned instead.

New data set status sr_status value canceled (cancel finished)

A cancel request still sets the data set to canceling (cancel in progress); once all jobs have wound down it now becomes canceled. Previously it stayed at canceling.

Data sets left at canceling are being settled to canceled as well. The sr_status filter (V2 get_schedule_results, V4 schedule_results) accepts canceled.

Action — If your code branches on sr_status, treat canceled as a finished cancellation. Polling loops waiting on canceling can stop when canceled arrives. If unknown values are treated as errors, add canceled to your allowed list.