Continuous Pipeline Processing

Reduce Cycle Time and Improve Operational Efficiency with Continuous Pipeline Processing

Organizations need to reduce business cycle times and improve operational efficiency. Too often, the pace of business processes slows down because incompatible business applications are bridged by batch processing or manual intervention. When each batch processing step adds latency to the process, the accumulated latency can become extensive. For example, if there are five applications that process transaction batches, and one batch runs once per night, it may take up to five days to complete all steps of the business process. Even a single batch process can add a day to the execution of a business process. Whether the aim is to implement straight-through processing (STP) in financial services, reduce out-of-stocks in retail, or set up just-in-time inventory management in manufacturing, eliminating the delays of batch processing can dramatically reduce cycle time. Progress® Sonic ESB® from Progress Software delivers a continuous pipeline processing approach that will help organizations migrate from batch to continuous processing across distributed systems. Using an enterprise service bus (ESB) will help reduce business process cycle time, gain up-to-date visibility into in-process data, reduce the cost of processing peak transaction loads, and put in place a more flexible architecture to handle new business requirements.

Use Sonic ESB to Reduce Latency and Improve IT Agility

Sonic ESB reduces the latency of batch processing by continuously feeding business transactions from one application to the next as they become available. With continuous pipeline processing and Sonic ESB, your application infrastructure will reliably buffer any pending transactions and assure that they will be processed in the order they were placed or by assigned priority. And since manual steps in business processes are often time-consuming, inefficient and error-prone, Sonic ESB solves this problem by simplifying the connection, or integration, to a wide variety of established technologies and packaged applications—providing a way to automate the business process itself. With the continuous pipeline processing approach that an ESB delivers, organizations can improve IT business agility and achieve operational responsiveness. Benefits include:

Eliminating the batch process delay between integrated applications.
Improving application utilization and reducing or eliminating the idle state between batch cycles.
Reducing peak load capacity requirements.
Automating manual integration steps which will reduce human error and administrative costs.

Sonic ESB Helps Gain Consistent, Up-to-date Visibility into In-process Data

An additional problem caused by batch processing business events is inconsistency of data across the applications in the process. When people and applications work from data from different batch cycles, they can draw conflicting and spurious conclusions - there is no "single version of the truth". Sonic ESB solves this problem by closing the timing gap between integrated systems. Instead of running in batch cycles, systems run in lock-step. By running in lock-step there is much less danger in generating conflicting business conclusions or reports due to in-process data that are "out of phase".

Sonic ESB and Continuous Pipeline Processing Delivers Operational Flexibility

Transformation of a batch-driven architecture to one which is based on a notification-style SOA infrastructure creates a flexible IT environment that is able to adapt quickly to new business requirements. For example, an organization may seek to increase revenue by adding a new sales channel, such as ISV and OEM. If a partner integration channel requires integration not just in order-taking but through a customer support portal, there are several points in the processing cycle at which business needs to integrate. It is much easier to do this when the applications operate in lock-step, processing transactions continuously. By integrating these additional applications into the SOA, and routing only relevant transactions to these applications for processing, companies can seamlessly combine the IT environments into a single continuous pipeline processing chain.

When deploying an ESB, it is important to note that migration need not be done all at once or in its entirety. Batch-driven applications which cannot be readily modified to work on a transaction-by-transaction basis can continue to function in this manner, but they—not the infrastructure—will become the source of latency. And if the applications have no downstream dependencies, it may be of little consequence that they continue to run in batch mode. In any event, even if all batch process latency cannot be immediately eliminated, significant reduction can still present great business value; it is not an all-or-nothing decision.

How Does Continuous Pipeline Processing Compare to Other Solutions?

IT departments have many solutions so here are some thoughts on how an enterprise service bus (ESB) compares to other technologies:

Data Warehouses: They focus on preparing and storing large volumes of data into databases for business intelligence queries. Data warehouse solutions are not designed to scale or deal with data distribution well. Furthermore, they focus on database-to-database transfer in batch—an ESB focuses on application-to-application transfer, batch or continuous.
Business Process Modeling: Focuses on controlling the process while an ESB focuses on routing control.

If your enterprise is experiencing operational inefficiencies due to batch transfers, if negative customer satisfaction situations arise because systems aren't kept in lock-step, or if it is difficult to introduce new business systems or capabilities into your existing IT environment, Sonic ESB may be the best solution for your business. Continuous pipeline processing is the key to realizing the promised agility of SOA because it allows IT to off-load to the business some of the burden of building and maintaining process solutions.

Application Integration Patterns with SOA

The Sonic ESB product family comprises Sonic ESB and a comprehensive set of compatible products that simplify application integration within a service-oriented architecture (SOA). It extends Sonic ESB's robust functionality with complex service orchestration, operational data management, and integration of third-party relational data sources, packaged applications and technologies. Some of these application integration patterns include:

Remote Information Access - User-facing portals are an ideal way to share information collected from multiple applications. The challenge is doing that without compromising the security, privacy or autonomy of the participating applications. To do this effectively, you need a distributed enterprise service bus that can cross crosses departments, firewalls, security domains, and large geographies.
Remote Data Distribution - Simplify reliable information distribution and enhance agility in a changing IT environment. Exploit support for notification style SOA using Sonic ESB.
Respond to Real Time Business Events - Real-time business events can be easily distributed to additional systems on the ESB as the need arises. New systems can join the bus and instantly subscribe to events without disruption to existing services and scenarios already running on the bus. This allows new events to be quickly integrated across nodes in a global environment with zero administrative overhead.

In addition to providing a solution for continuous pipeline processing, Progress also delivers SOA infrastructure technology and services for enterprise application integration, application performance management, business activity monitoring (BAM), SOA governance and semantic data integration.