Enterprise Integration

In this post, we will talk about Enterprise Integration and how enterprise organizations connected their business ecosystem with different integration technology. Let’s start with What is enterprise integration?

What is enterprise integration?

Most of the time enterprise organizations have multiple systems and clouds to implement business requirements. All systems must share data across the platform. Enterprise integration encompasses the technologies, processes, and team structures that connect data, applications, and devices from everywhere in your IT organization.

Enterprise integration has improved over the years from point-to-point connections to a centralized model connected by an enterprise service bus (ESB) to a distributed architecture with many reusable endpoints.

Business Ecosystem & Communication patterns

Let’s start with one business use case. Here is a sample technology Architecture to communicate the Application.

Basic Integration Architecture

First thing first, Let’s understand the Basic Integration Architecture.

1. Point to Point:

Each system is connected to every other system through direct integration. This is easy to implement with only a few systems. Difficult to scale

2. Hub and Spoke

• Every system connects to the Hub.
• All data transfer is done through the hub.
• Easy to design and implement.
• Architectures are proprietary in nature.
• Single point of failure.
• Inability to support large transaction volumes.

3. Enterprise Service Bus

• Distributed services architecture.
• Employs distributed adapters.
• Highly scalable.

Integration Direction

There are two ways to integrate any system called Inbound Integration and Outbound Integration.

• Inbound Integration: Inbound: An external system initiates contact with Salesforce.
• Outbound Integration: Outbound: Salesforce initiates contact with an external system.

Note: Make sure to identify all Proxies and firewalls that need to be negotiated for Inbound/Outbound integrations.

Integration Levels

A DMZ Network is a perimeter network that protects and adds an extra layer of security to an organization’s internal local-area network from untrusted traffic. A common DMZ is a subnetwork that sits between the public internet and private networks.

Time Based Integration Processes

#1. Synchronous Process:

A user action creates a request, and the user is blocked from continuing until the response is returned. A very near real-time response is expected and needed. •Tools/Technique : REST or SOAP API

Scenario: When a Salesforce account is created, a lookup must happen in the existing legacy system to check whether the Salesforce account already exists or not before the Save is completed.

#2. Asynchronous Process

A user action creates a request, and the system does not wait for the response before letting the user move to the next task. Analogous to near real-time process Tools/Technique : MQ ,Kafka

Scenario: When user changes an account record , the process of replicating the change to external systems begins immediately. However, the process of replication does not block the user from continuing the work.

#3. Batch Process

Large data volumes are broken into chunks and processed in queues based on available resources. Can be implemented through Bulk API, SOAP/REST API. Often associated with a scheduled process. Tools/Technique: Bulk API/ETL/File Transfer

Scenario: When we need to update account information once in a day to a back-end system.

Middleware(App Glue) in Enterprise Context

Here are 6 Fundamental Principles for Middleware in Enterprise Context:

1. Transnationality: While Salesforce is transactional within itself, it’s not able to participate in distributed transactions or transactions initiated outside of Salesforce. Therefore, it’s assumed that for solutions requiring complex, multi-system transactions, transnationality (and associated roll-back/compensation mechanisms) be implemented at the middleware layer.
2. Event handling: Event handling In Salesforce integrations using middleware, the control of event handling is assumed by the middleware layer; it collects all relevant events (synchronous or asynchronous) and manages distribution to all endpoints, including Salesforce.
3. Queuing and buffering: Queuing and buffering generally rely on asynchronous message passing, as opposed to a request-response architecture. In asynchronous systems, message queues provide temporary storage when the destination program is busy, or connectivity is compromised. In addition, most asynchronous middleware systems provide persistent storage to back up the message queue.
4. Extract, transform, and load: Extracting data from the source systems. This typically involves data from several source systems, and both relational and non-relational structures. The transform stage usually applies a series of rules or functions to the extracted data from the source to derive the data for loading into the end target(s).
5. Translation and transformation: Transformation is the ability to map one data format to another to ensure interoperability between the various systems being integrated. Typically, this entails reformatting messages enroute to match the requirements of the sender or recipient. In more complex cases, one application can send a message in its own native format, and two or more other applications might each receive a copy of the message in their own native format.
6. Mediation routing: Many middleware-based solutions depend on a message queue system. While some implementations permit routing logic to be provided by the messaging layer itself, others depend on client applications to provide routing information or allow for a mix of both paradigms. In such complex cases, mediation (on the part of middleware) simplifies development, integration, and validation).

Enterprise Application Integration with ESB

Here is one sample Enterprise Application Integration landscape with ESB.

The Inception of API

API stands for Application Programming Interface. It is a set of definitions and protocols for building and integrating application software. API enables integration with heterogeneous systems and data to bring new business models, products, and services to the forefront.

The Evolution of API

1. SOAP

Simple Object Access Protocol (SOAP) is a protocol for exchanging information encoded in Extensible Markup Language (XML) between a client and a procedure or service that resides on the Internet

SOAP is typically used with the Web Service Description Language (WSDL). The significance of using WSDL is that developers and machines can inspect a web service that supports SOAP to discover the specifics of the information exchange over the network.

#Pros :The major advantage of SOAP is the fact that it is completely agnostic when it comes to the programming language and processing platform. The standardized format ensures that no matter what is receiving the message on the other end, the request can be executed.

#Cons :SOAP has become less popular in recent years because of the sheer volume of information it requires. The XML files are large and are often unnecessarily clunky – especially for simple systems.

#Format- XML only

2. REST

REST is an acronym for Representational State Transfer. The basic premise is that developers use the standard HTTP methods, GET, POST, PUT and DELETE, to query and mutate resources represented by URIs on the Internet. REST APIs are designed to make server-side data readily available by representing it in simple formats such as JSON and XML.

#Pros :The key benefits of this API type are that the client, and the server are completely decoupled from one another. This allows for abstraction layers that help to maintain flexibility even as a system grows and evolves. In addition, REST is cache friendly and supports multiple formats, which is important when you’re building public-facing APIs.

#Cons: Where REST APIs fall short is that their rich metadata creates big payloads that can sometimes cause more trouble than they’re worth. You can get over- and under-fetching problems that require further API requests, bogging down the process.

#Format- XML,JSON,HTML,plain text

3. GraphQL

GraphQL was developed in response to REST APIs, with the idea that you could execute precise syntax that retrieves only what is needed, lightening the payload and simplifying the process significantly. The API can accept specific requests and return a result that matches exactly what the user is looking for.

#Pros: The API queries are transparent and well documented, giving users all the information, they need to use it effectively. In addition, the precise results, detailed error messages, and flexible permissions help round out a well-balanced but high-functioning API.

#Cons: Performance can suffer with GraphQL if you have too many nested fields in any one request. It also doesn’t reuse standard HTTP caching semantics, so it actually requires custom efforts to achieve proper caching

#Format- JSON

4. gRPC

In gRPC, RPC stands for Remote Procedure Call and refers to something that can execute a function housed elsewhere but in a different context. A user on one side will select a remote procedure to execute, serialize the necessary parameters and then append any additional information into the message. This will then be sent to the server, which interacts with the other application, decoding the message and executing the operation. A result then comes back to the initial user.

#Pros: This is a powerful form of API because of its simplicity. It’s straightforward, using GET to fetch information and POST for everything else. This means that functions are easy to add, and for lightweight payloads, you get great performance overall.

#Cons: Where gRPC falls short is in the fact that it’s tightly coupled to the underlying system, which restricts its reusability in many cases. In addition, there is no abstraction layer sitting between the API and the actual system functions, which can raise security concerns.

#Format- JSON,XML,Thrift,protobuf

How to secure the APIs

• Basic Authentication: HTTP Basic authentication uses standard fields in the HTTP header, removing the need for handshakes. To receive authorization, the client sends the Userid and password, separated by a single colon (“:”) character, within a Base64 encoded string in the credentials.
• API Keys: It fits designing APIs for 3rd parties services integration and limited access, not public access purpose. For instance, your company provides SMS gateway as web services and other companies would like to use your services.
• OAuth 2: OAuth 2 is an open standard for authentication protocol that provides authorization workflow over HTTP and authorizes devices, servers, applications, and APIs with access tokens instead of credentials. (access token + refresh token )
• JSON Web Token (JWT): The main difference with Oauth2 is that we store user information state in the token, while services are stateless. This means that the server can decrypt the token into the user information state and there is no need for additional looking up from the database with this token
• Security Assessment Markup Language (SAML): Security Assertion Markup Language (SAML) is an XML-based framework for authentication and authorization between two entities: A Service Provider and an Identity Provider.
• OpenID Connect (OIDC) is an open authentication protocol that works on top of the OAuth 2.0 framework. Targeted toward consumers, OIDC allows individuals to use single sign-on (SSO) to access relying on party sites using OpenID Providers (OPs), such as an email provider or social network, to authenticate their identities.

API gateway : ESB antipattern

An API gateway is a management tool that sits at the edge of a system between a consumer and a collection of backend services and acts as a single point of entry for a defined group of APIs. The consumer can be an end-user application or device, such as a single page web application or a mobile app, or another internal system, or third-party application or system. Below diagram shows how an API gateway often sits between the public internet and the demilitarized zone (DMZ) of a private network.

Difference between API gateway  VS  ESB

Evolution of Software Architectures

Microservices Architecture for Application integration

Typical Enterprise Integration Architecture

Managed File transfer (MFT)

MFT comprises software technologies that secure, control, document, and automate the complete process of electronic data exchange. These file transfers can be internal or external to an organization, MFT allows the transfer of structured or unstructured data of various types including sensitive, compliance-related or even very high volume data.

Using MFT, organizations can replace unsecure methods of file transfer like standard file transfer protocol (FTP), eMail or popular but often unsecure internet file sharing services.

Message Queue

A message queue is a form of service-to-service communication that facilitates asynchronous communication. It asynchronously receives messages from producers and sends them to consumers. Messages are stored in the queue until they are processed and deleted. Each message is processed only once by a single consumer. Here’s how it works:

• A producer publishes a job to the queue, then notifies the user of the job status.
• A consumer picks up the job from the queue, processes it, then signals that the job is complete.

1. Amazon SQS

2. RabbitMQ

3. ActiveMQ

4. Anypoint MQ

Some advantages of using a message queue:

• Scalability: Message queues make it possible to scale precisely where we need to.
• Decoupling: Message queues remove dependencies between components and significantly simplify the implementation of decoupled applications.

Enterprise Integration Pattern Catalogue of 65 Messaging Patterns

Check out below Enterprise Integration Pattern from the Enterprise Integration Pattern book.

Event-Driven Architecture

Event-driven architecture (EDA) is a software design pattern that enables an organization to detect “events” or important business moments (such as a transaction, like a mouse click or keystroke) and act on them in real time or near real time. This pattern replaces the traditional “request/response” architecture  where services would have to wait for a reply before they could move onto the next task

Mediator topology is commonly used when multiple steps are required to process events. There can be many mediators and message queues in this model, and each queue is typically associated with a specific task.

The broker topology model also works with messages but differs from the mediator topology in that there is no central mediator. Instead, events are broadcasted to a lightweight message broker and then distributed to multiple consumers.

1.ESB, Apache Camel

2.Apache Kafka, Google PubSub,Amazon EventBridge

Point to Point Vs Event-Driven Architecture

Event-Driven Architectures Use cases

Enterprise Integration Recording

Check Enterprise Integration session recordin

Summary

I hope this Enterprise Integration pattern will help you to implement the Salesforce Integration. Let us know if this is helpful for you.