Salesforce Bulk API 2.0

Join us to learn about How to Use Salesforce Bulk API 2.0 for Large-Scale Data Operations. In this article we will describes Salesforce Bulk API and difference between Bulk API 1.0 and Bulk API 2.0.

What is Bulk API 2.0?

Bulk API 2.0 is a RESTful API that allows you to perform large-scale data operations on Salesforce objects, such as insert, update, delete, and query. Bulk API 2.0 is designed to simplify the process of creating and managing bulk jobs, as well as to improve the performance and reliability of bulk data processing.

How does Bulk API 2.0 differ from Bulk API 1.0?

Bulk API 1.0 is an older version of the Bulk API that uses SOAP or REST to create and manage bulk jobs and batches. Bulk API 1.0 requires you to manually create batches of records for each job, monitor the status of each batch, and handle errors and retries.

Salesforce Bulk API 2.0, on the other hand, automatically chunks the data for your job into multiple internal batches to improve performance while processing the job. You only need to create one job for each operation, upload the data for the job, and check the status of the job. Salesforce handles the creation and execution of internal batches, as well as the error handling and retries.

What are the limits of Bulk API 2.0?

Bulk API 2.0 has some limits that you need to be aware of when using it. These limits are:

Daily limit: You can process up to 100 million records per 24-hour period using Bulk API 2.0. This limit is based on the number of records processed, not the number of jobs or batches created.
Job limit: You can have up to 100,000 jobs in any state (Open, Upload Complete, InProgress, JobComplete, Aborted, or Failed) at any given time. You can delete completed or aborted jobs to free up space for new jobs.
Data limit: You can upload up to 150 MB of data per job. The data must be in CSV format and each record must be on a separate line.
Field limit: You can include up to 10,000 fields per object in your job data. This limit applies to both standard and custom fields.
Batch limit: Salesforce creates a separate internal batch for every 10,000 records in your job data. Each batch can take up to 10 minutes to process. If a batch fails or times out, Salesforce will retry it up to 10 times. If a batch still cannot be processed after 10 retries, the entire job will fail and the remaining data will not be processed.
Concurrency limit: You can have up to 25 concurrent jobs running at any given time. This limit applies to both Bulk API 2.0 and Bulk API 1.0 jobs combined.
Query limit: You can query up to 15 GB of data per job. The query results must be retrieved within 7 days of job completion.

What are the benefits of using Bulk API 2.0?

Bulk API 2.0 has some advantages over Bulk API 1.0 that make it easier and faster to use for bulk data operations. Some of these benefits are:

Simplified job creation and management: You only need to create one job for each operation and upload the data for the job in one request. You don’t need to worry about creating and managing batches or handling errors and retries.
Improved performance and reliability: Salesforce optimises the internal batch size and execution order based on your data and operation type. Salesforce also handles the error handling and retries for each batch automatically.
Native REST framework: Bulk API 2.0 uses the same REST framework as other Salesforce REST APIs. You can use any HTTP client or library that supports REST to interact with Bulk API 2.0 endpoints.
OAuth support: Bulk API 2.0 supports all OAuth flows from Salesforce’s other REST APIs. You can use any authentication method that suits your needs and preferences.
JSON support: Bulk API 2.0 supports JSON format for job creation and status requests. You can use JSON instead of XML if you prefer a more lightweight and human-readable format.

How to use Bulk API 2.0?

To use Bulk API 2.0, you need to follow these steps:

Create a job: Send a POST request to /services/data/vXX.X/jobs/ingest endpoint with the details of the job, such as the object, operation, and external ID field (optional). You will receive a response with the job ID and other information.

Upload data: Send a PUT request to /services/data/vXX.X/jobs/ingest/{jobId}/batches endpoint with the data for the job in CSV format. You can upload up to 150 MB of data per job.

Close the job: Send a PATCH request to /services/data/vXX.X/jobs/ingest/{jobId} endpoint with the state set to UploadComplete. This will inform Salesforce to start processing the data you uploaded.

Check the status: Send a GET request to /services/data/vXX.X/jobs/ingest/{jobId} endpoint to check the status of the job and the number of records processed, failed, or unprocessed. You can also send a GET request to /services/data/vXX.X/jobs/ingest/{jobId}/failedResults endpoint to get the details of the failed records.

Retrieve the results: If your job is a query operation, you can send a GET request to /services/data/vXX.X/jobs/query/{jobId}/results endpoint to retrieve the query results in CSV format. You can also use the Bulk Query 2.0 Tool in Workbench to run and download query jobs.

Examples

Here are some examples of using Bulk API 2.0 for different operations:

Insert

To insert new records into an object, you need to create a job with the operation set to insert and upload the data with the fields and values for each record. For example, to insert new contacts into Salesforce, you can use the following requests:

POST /services/data/v53.0/jobs/ingest Content-Type: application/json; charset=UTF-8 Accept: application/json Authorization: Bearer <access_token> { "object" : "Contact", "operation" : "insert" }` `PUT /services/data/v53.0/jobs/ingest/<job_id>/batches Content-Type: text/csv; charset=UTF-8 Accept: application/json Authorization: Bearer <access_token> FirstName,LastName,Email Alice,Smith,[email protected] Bob,Jones,[email protected] Charlie,Brown,[email protected]` `PATCH /services/data/v53.0/jobs/ingest/<job_id> Content-Type: application/json; charset=UTF-8 Accept: application/json Authorization: Bearer <access_token> { "state" : "UploadComplete" }

Update

To update existing records in an object, you need to create a job with the operation set to update and upload the data with the fields and values for each record. You also need to specify an external ID field that uniquely identifies each record. For example, to update contacts in Salesforce using their email as the external ID field, you can use the following requests:

POST /services/data/v53.0/jobs/ingest Content-Type: application/json; charset=UTF-8 Accept: application/json Authorization: Bearer <access_token> { "object" : "Contact", "operation" : "update", "externalIdFieldName" : "Email" }` `PUT /services/data/v53.0/jobs/ingest/<job_id>/batches Content-Type: text/csv; charset=UTF-8 Accept: application/json Authorization: Bearer <access_token> Email,Phone [email protected],(123) 456-7890 [email protected],(234) 567-8901 [email protected],(345) 678-9012` `PATCH /services/data/v53.0/jobs/ingest/<job_id> Content-Type: application/json; charset=UTF-8 Accept: application/json Authorization: Bearer <access_token> { "state" : "UploadComplete" }

Delete

To delete existing records from an object, you need to create a job with the operation set to delete and upload the data with the IDs of the records you want to delete. For example, to delete contacts from Salesforce using their IDs, you can use the following requests:
POST /services/data/v53.0/jobs/ingest Content-Type: application/json; charset=UTF-8 Accept: application/json Authorization: Bearer <

Conclusion

Bulk API 2.0 is a powerful tool that simplifies large-scale data operations on Salesforce objects. Although there are some limitations, the benefits outweigh the drawbacks, and its native REST framework makes it easier to use with other Salesforce REST APIs. By following the steps outlined above, you can use Bulk API 2.0 to perform bulk data operations for your Salesforce instance.