1
Introduction to Streams

This chapter briefly describes the basic concepts and terminology related to Oracle Streams. These concepts are described in more detail in other chapters in this book and in the Oracle Streams Replication Administrator's Guide.

This chapter contains these topics:

• Overview of Streams
• Overview of the Capture Process
• Overview of Event Staging and Propagation
• Overview of the Apply Process
• Overview of the Messaging Client
• Overview of Automatic Conflict Detection and Resolution
• Overview of Rules
• Overview of Transformations
• Overview of Streams Tags
• Overview of Heterogeneous Information Sharing
• Example Streams Configurations
• Administration Tools for a Streams Environment

Overview of Streams

Oracle Streams enables information sharing. Using Oracle Streams, you can share data and events in a stream. The stream can propagate information within a database or from one database to another. The stream routes specified information to specified destinations. The result is a feature that provides greater functionality and flexibility than traditional solutions for capturing and managing events, and sharing the events with other databases and applications. Streams provides the capabilities needed to build and operate distributed enterprises and applications, data warehouses, and high availability solutions. You can use all of the capabilities of Oracle Streams at the same time. If your needs change, then you can implement a new capability of Streams without sacrificing existing capabilities.

Using Oracle Streams, you control what information is put into a stream, how the stream flows or is routed from database to database, what happens to events in the stream as they flow into each database, and how the stream terminates. By configuring specific capabilities of Streams, you can address specific requirements. Based on your specifications, Streams can capture, stage, and manage events in the database automatically, including, but not limited to, data manipulation language (DML) changes and data definition language (DDL) changes. You also can put user-defined events into a stream, and Streams can propagate the information to other databases or applications automatically. When events reach a destination, Streams can consume them based on your specifications. Figure 1-1 shows the Streams information flow.

Figure 1-1 Streams Information Flow

Text description of the illustration strms022.gif

What Can Streams Do?

The following sections provide an overview of what Streams can do.

Capture Events at a Database

A capture process can capture changes made to tables, schemas, or an entire database. Such changes are recorded in the redo log for a database, and a capture process captures changes from the redo log and formats each captured change into an event called a logical change record (LCR). Rules determine which changes are captured by a capture process, and these captured changes are called captured events.

The database where changes are generated in the redo log is called the source database. A capture process may capture changes locally at the source database, or it may capture changes remotely at a downstream database. A capture process enqueues LCRs into a queue that is associated with it. When a capture process captures events, it is sometimes referred to as implicit capture.

Users and applications also can enqueue events into a queue manually. These events are called user-enqueued events, and they can be LCRs or messages of a user-defined type called user messages. When users and applications enqueue events into a queue manually, it is sometimes referred to as explicit capture.

Stage Events in a Queue

Events are stored (or staged) in a queue. These events may be captured events or user-enqueued events. A capture process enqueues events into a SYS.AnyData queue. A a SYS.AnyData queue can stage events of different types. Users and applications may enqueue events into a SYS.AnyData queue or into a typed queue. A typed queue can stage events of one specific type only.

Propagate Events From One Queue To Another

Streams propagations can propagate events from one queue to another. These queues may be in the same database or in different databases. Rules determine which events are propagated by a propagation.

Consume Events

An event is consumed when it is dequeued from a queue. An apply process can dequeue events from a queue implicitly. A user, application, or messaging client can dequeue events explicitly. The database where events are consumed is called the destination database. In some configurations, the source database and the destination database may be the same.

Rules determine which events are dequeued and processed by an apply process. An apply process may apply events directly to database objects, or an apply process may pass events to custom PL/SQL subprograms for processing.

Rules determine which events are dequeued by a messaging client. A messaging client dequeues events when it is invoked by an application or a user.

Other Capabilities of Streams

Other capabilities of Streams include the following:

• Directed networks
• Automatic conflict detection and resolution
• Transformations
• Heterogeneous information sharing

These capabilities are discussed briefly later in this chapter and in detail later in this document and in the Oracle Streams Replication Administrator's Guide.

What Are the Uses of Streams?

The following sections briefly describe some of the reasons for using Streams. In some cases, Streams components provide infrastructure for various features of Oracle.

Message Queuing

Oracle Streams Advanced Queuing (AQ) enables user applications to enqueue messages into a queue, propagate messages to subscribing queues, notify user applications that messages are ready for consumption, and dequeue messages at the destination. A queue may be configured to stage messages of a particular type only, or a queue may be configured as a SYS.AnyData queue. Messages of almost any type can be wrapped in a SYS.AnyData wrapper and staged in SYS.AnyData queues. AQ supports all the standard features of message queuing systems, including multiconsumer queues, publish and subscribe, content-based routing, Internet propagation, transformations, and gateways to other messaging subsystems.

You can create a queue at a database, and applications can enqueue messages into the queue explicitly. Subscribing applications or messaging clients can dequeue messages directly from this queue. If an application is remote, then a queue may be created in a remote database that subscribes to messages published in the source queue. The destination application can dequeue messages from the remote queue. Alternatively, the destination application can dequeue messages directly from the source queue using a variety of standard protocols.

Data Replication

Streams can capture DML and DDL changes made to database objects and replicate those changes to one or more other databases. A Streams capture process captures changes made to source database objects and formats them into LCRs, which can be propagated to destination databases and then applied by Streams apply processes.

The destination databases can allow DML and DDL changes to the same database objects, and these changes may or may not be propagated to the other databases in the environment. In other words, you can configure a Streams environment with one database that propagates changes, or you can configure an environment where changes are propagated between databases bidirectionally. Also, the tables for which data is shared do not need to be identical copies at all databases. Both the structure and the contents of these tables can differ at different databases, and the information in these tables can be shared between these databases.

Event Management and Notification

Business events are valuable communications between applications or organizations. An application may enqueue events into a queue explicitly, or a Streams capture process may capture a database event. These captured events may be DML or DDL changes. Propagations may propagate events in a stream through multiple queues. Finally, a user application may dequeue events explicitly, or a Streams apply process may dequeue events implicitly. An apply process may re-enqueue these events explicitly into the same queue or a different queue if necessary.

You can configure queues to retain explicitly-enqueued messages after consumption for a specified period of time. This capability enables you to use Advanced Queuing (AQ) as a business event management system. AQ stores all messages in the database in a transactional manner, where they can be automatically audited and tracked. You can use this audit trail to extract intelligence about the business operations.

Capture processes, propagations, apply processes, and messaging clients perform actions based on rules. You specify which events are captured, propagated, applied, and dequeued using rules, and a built-in rules engine evaluates events based on these rules. The ability to capture events and propagate them to relevant consumers based on rules means that you can use Streams for event notification. Events staged in a queue may be dequeued explicitly by a messaging client or an application, and then actions can be taken based on these events, which may include an email notification, or passing the message to a wireless gateway for transmission to a cell phone or pager.

Data Warehouse Loading

Data warehouse loading is a special case of data replication. Some of the most critical tasks in creating and maintaining a data warehouse include refreshing existing data, and adding new data from the operational databases. Streams components can capture changes made to a production system and send those changes to a staging database or directly to a data warehouse or operational data store. Streams capture of redo log information avoids unnecessary overhead on the production systems. Support for data transformations and user-defined apply procedures enables the necessary flexibility to reformat data or update warehouse-specific data fields as data is loaded. In addition, Change Data Capture uses some of the components of Streams to identify data that has changed so that this data can be loaded into a data warehouse.

Data Protection

One solution for data protection is to create a local or remote copy of a production database. In the event of human error or a catastrophe, the copy can be used to resume processing. You can use Streams to configure flexible high availability environments. In addition, you can use Oracle Data Guard, a data protection feature that uses some of the same infrastructure as Streams, to create and maintain a logical standby database, which is a logically equivalent standby copy of a production database. As in the case of Streams replication, a capture process captures changes in the redo log and formats these changes into LCRs. These LCRs are applied at the standby databases.

The standby databases are fully open for read/write and may include specialized indexes or other database objects. Therefore, these standby databases can be queried as updates are applied, making Oracle Data Guard a good solution for off loading queries from a production database.

It is important to move the updates to the remote site as soon as possible with a logical standby database. Doing so ensures that, in the event of a failure, lost transactions are minimal. By directly and synchronously writing the redo logs at the remote database, you can achieve no data loss in the event of a disaster. At the standby system, the changes are captured and directly applied to the standby database with an apply process.

Overview of the Capture Process

Changes made to database objects in an Oracle database are logged in the redo log to guarantee recoverability in the event of user error or media failure. A capture process is an Oracle background process that scans the database redo log to capture DML and DDL changes made to database objects. A capture process formats these changes into events called LCRs and enqueues them into a queue. There are two types of LCRs: row LCRs contain information about a change to a row in table resulting from a DML operation, and DDL LCRs contain information about a DDL change to a database object. Rules determine which changes are captured. Figure 1-2 shows a capture process capturing LCRs.

Figure 1-2 Capture Process

Text description of the illustration strms012.gif

You can configure change capture locally at a source database or remotely at a downstream database. If a capture process runs on a downstream database, then redo log files from the source database are copied to the downstream database, and the capture process captures changes in these redo log files.

Overview of Event Staging and Propagation

Streams uses queues to stage events for propagation or consumption. Propagations send events from one queue to another, and these queues can be in the same database or in different databases. The queue from which the events are propagated is called the source queue, and the queue that receives the events is called the destination queue. There can be a one-to-many, many-to-one, or many-to-many relationship between source and destination queues.

Events that are staged in a queue can be consumed by an apply process, a messaging client, or by an application. Rules determine which events are propagated by a propagation. Figure 1-3 shows propagation from a source queue to a destination queue.

Figure 1-3 Propagation from a Source Queue to a Destination Queue

Text description of the illustration strms008.gif

Overview of Directed Networks

Streams enables you to configure an environment in which changes are shared through directed networks. In a directed network, propagated events may pass through one or more intermediate databases before arriving at a destination database where they are consumed. The events may or may not be consumed at an intermediate database in addition to the destination database. Using Streams, you can choose which events are propagated to each destination database, and you can specify the route events will traverse on their way to a destination database.

Figure 1-4 shows an example directed networks environment. Notice that, in this example, the queue at the intermediate database in Chicago is both a source queue and a destination queue.

Figure 1-4 Example Directed Networks Environment

Text description of the illustration strms007.gif

Explicit Enqueue and Dequeue of Events

User applications can enqueue events into a queue explicitly. The user applications can format these user-enqueued events as LCRs or user messages, and an apply process, a messaging client, or a user application can consume these events. Events that were enqueued explicitly into a queue can be propagated to another queue or explicitly dequeued from the same queue. Figure 1-5 shows explicit enqueue of events into and dequeue of events from the same queue.

Figure 1-5 Explicit Enqueue and Dequeue of Events in a Single Queue

Text description of the illustration strms010.gif

When events are propagated between queues, events that were enqueued explicitly into a source queue can be dequeued explicitly from a destination queue by a messaging client or user application. These events also may be processed by an apply process. Figure 1-6 shows explicit enqueue of events into a source queue, propagation to a destination queue, and then explicit dequeue of events from the destination queue.

Figure 1-6 Explicit Enqueue, Propagation, and Dequeue of Events

Text description of the illustration strms011.gif

Overview of the Apply Process

An apply process is an Oracle background process that dequeues events from a queue and either applies each event directly to a database object or passes the event as a parameter to a user-defined procedure called an apply handler. Apply handlers include message handlers, DML handlers, DDL handlers, precommit handlers, and error handlers.

Typically, an apply process applies events to the local database where it is running, but, in a heterogeneous database environment, it can be configured to apply events at a remote non-Oracle database. Rules determine which events are dequeued by an apply process. Figure 1-7 shows an apply process processing LCRs and user messages.

Figure 1-7 Apply Process

Text description of the illustration strms013.gif

Overview of the Messaging Client

A messaging client consumes user-enqueued events when it is invoked by an application or a user. Rules determine which user-enqueued events are dequeued by a messaging client. These user-enqueued events may be LCRs or user messages. Figure 1-8 shows a messaging client dequeuing user-enqueued events.

Figure 1-8 Messaging Client

Text description of the illustration strms045.gif

Overview of Automatic Conflict Detection and Resolution

An apply process detects conflicts automatically when directly applying LCRs in a replication environment. Typically, a conflict results when the same row in the source database and destination database is changed at approximately the same time.

When a conflict occurs, you need a mechanism to ensure that the conflict is resolved in accordance with your business rules. Streams offers a variety of prebuilt conflict handlers. Using these prebuilt handlers, you can define a conflict resolution system for each of your databases that resolves conflicts in accordance with your business rules. If you have a unique situation that Oracle's prebuilt conflict resolution handlers cannot resolve, then you can build your own conflict resolution handlers.

If a conflict is not resolved, or if a handler procedure raises an error, then all events in the transaction that raised the error are saved in the error queue for later analysis and possible reexecution.

Overview of Rules

Streams enables you to control which information to share and where to share it using rules. A rule is specified as a condition that is similar to the condition in the WHERE clause of a SQL query.

A rule consists of the following components:

• The rule condition combines one or more expressions and conditions and returns a Boolean value, which is a value of TRUE, FALSE, or NULL (unknown), based on an event.
• The rule evaluation context defines external data that can be referenced in rule conditions. The external data either can exist as external variables, as table data, or both.
• The rule action context is optional information associated with a rule that is interpreted by the client of the rules engine when the rule is evaluated.

You can group related rules together into rule sets. In Streams, rule sets may be positive or negative.

For example, the following rule condition may be used for a rule in Streams to specify that the schema name that owns a table must be hr and that the table name must be departments for the condition to evaluate to TRUE:

:dml.get_object_owner() = 'HR' AND :dml.get_object_name() = 'DEPARTMENTS'

The :dml variable is used in rule conditions for row LCRs. In a Streams environment, a rule with this condition may be used in the following ways:

• If the rule is in a positive rule set for a capture process, then it instructs the capture process to capture row changes that result from DML changes to the hr.departments table. If the rule is in a negative rule set for a capture process, then it instructs the capture process to discard DML changes to the hr.departments table.
• If the rule is in a positive rule set for a propagation, then it instructs the propagation to propagate LCRs that contain row changes to the hr.departments table. If the rule is in a negative rule set for a propagation, then it instructs the propagation to discard LCRs that contain row changes to the hr.departments table.
• If the rule is in a positive rule set for an apply process, then it instructs the apply process to apply LCRs that contain row changes to the hr.departments table. If the rule is in a negative rule set for an apply process, then it instructs the apply process to discard LCRs that contain row changes to the hr.departments table.
• If the rule is in a positive rule set for a messaging client, then it instructs the messaging client to dequeue LCRs that contain row changes to the hr.departments table. If the rule is in a negative rule set for a messaging client, then it instructs the messaging client to discard LCRs that contain row changes to the hr.departments table.

Streams performs tasks based on rules. These tasks include capturing events with a capture process, propagating events with a propagation, applying events with an apply process, dequeuing events with a messaging client, and discarding events.

Overview of Transformations

A rule-based transformation is any modification to an event that results when a rule in a positive rule set evaluates to TRUE. For example, a rule-based transformation can change the datatype of a particular column in a table for an event. In this case, the transformation can be a PL/SQL function that takes as input a SYS.AnyData object containing an LCR with a NUMBER datatype for a column and returns a SYS.AnyData object containing an LCR with a VARCHAR2 datatype for the same column.

A transformation can occur at the following times:

• During enqueue of an event by a capture process, which can be useful for formatting an event in a manner appropriate for all destination databases
• During propagation of an event, which may be useful for subsetting data before it is sent to a remote site
• During dequeue of an event by an apply process or messaging client, which can be useful for formatting an event in a manner appropriate for a specific destination database

When a transformation is performed during apply, an apply process may apply the transformed event directly or send the transformed event to an apply handler for processing. Figure 1-9 shows a rule-based transformation during apply.

Figure 1-9 Transformation During Apply

Text description of the illustration strms044.gif

Overview of Streams Tags

Every redo entry in the redo log has a tag associated with it. The datatype of the tag is RAW. By default, when a user or application generates redo entries, the value of the tag is NULL for each redo entry, and a NULL tag consumes no space in the redo entry. The size limit for a tag value is 2000 bytes.

In Streams, rules may have conditions relating to tag values to control the behavior of Streams clients. For example, a tag can be used to determine whether an LCR contains a change that originated in the local database or at a different database, so that you can avoid change cycling (sending an LCR back to the database where it originated). Also, a tag can be used to specify the set of destination databases for each LCR. Tags may be used for other LCR tracking purposes as well.

You can specify Streams tags for redo entries generated by a certain session or by an apply process. These tags then become part of the LCRs captured by a capture process. Typically, tags are used in Streams replication environments, but you can use them whenever it is necessary to track database changes and LCRs.

Overview of Heterogeneous Information Sharing

In addition to information sharing between Oracle databases, Streams supports information sharing between Oracle databases and non-Oracle databases. The following sections contain an overview of this support.

Overview of Oracle to Non-Oracle Data Sharing

If an Oracle database is the source and a non-Oracle database is the destination, then the non-Oracle database destination lacks the following Streams mechanisms:

• A queue to receive events
• An apply process to dequeue and apply events

To share DML changes from an Oracle source database with a non-Oracle destination database, the Oracle database functions as a proxy and carries out some of the steps that would normally be done at the destination database. That is, the events intended for the non-Oracle destination database are dequeued in the Oracle database itself, and an apply process at the Oracle database uses Heterogeneous Services to apply the events to the non-Oracle database across a network connection through a gateway. Figure 1-10 shows an Oracle databases sharing data with a non-Oracle database.

Figure 1-10 Oracle to Non-Oracle Heterogeneous Data Sharing

Text description of the illustration strms009.gif

Overview of Non-Oracle to Oracle Data Sharing

To capture and propagate changes from a non-Oracle database to an Oracle database, a custom application is required. This application gets the changes made to the non-Oracle database by reading from transaction logs, using triggers, or some other method. The application must assemble and order the transactions and must convert each change into an LCR. Next, the application must enqueue the LCRs into a queue in an Oracle database by using the PL/SQL interface, where they can be processed by an apply process. Figure 1-11 shows a non-Oracle databases sharing data with an Oracle database.

Figure 1-11 Non-Oracle to Oracle Heterogeneous Data Sharing

Text description of the illustration strms014.gif

Example Streams Configurations

Figure 1-12 shows how Streams might be configured to share information within a single database, while Figure 1-13 shows how Streams might be configured to share information between two different databases.

Figure 1-12 Streams Configuration in a Single Database

Text description of the illustration strms006.gif

Figure 1-13 Streams Configuration Sharing Information Between Databases

Text description of the illustration strms005.gif

Administration Tools for a Streams Environment

Several tools are available for configuring, administering, and monitoring your Streams environment. Oracle-supplied PL/SQL packages are the primary configuration and management tool, while the Streams tool in the Oracle Enterprise Manager Console provides some configuration, administration, and monitoring capabilities to help you manage your environment. Additionally, Streams data dictionary views keep you informed about your Streams environment.

Oracle-Supplied PL/SQL Packages

The following Oracle-supplied PL/SQL packages contain procedures and functions for configuring and managing a Streams environment.

DBMS_STREAMS_ADM Package

The DBMS_STREAMS_ADM package provides an administrative interface for adding and removing simple rules for capture processes, propagations, and apply processes at the table, schema, and database level. This package also enables you to add rules that control which events a propagation propagates and which events a messaging client dequeues. This package also contains procedures for creating queues and for managing Streams metadata, such as data dictionary information. This package also contains procedures that enable you to configure and maintain a Streams replication environment for specific tablespaces. This package is provided as an easy way to complete common tasks in a Streams environment. You can use other packages, such as the DBMS_CAPTURE_ADM, DBMS_PROPAGATION_ADM, DBMS_APPLY_ADM, DBMS_RULE_ADM, and DBMS_AQADM packages, to complete these same tasks, as well as tasks that require additional customization.

DBMS_CAPTURE_ADM Package

The DBMS_CAPTURE_ADM package provides an administrative interface for starting, stopping, and configuring a capture process. This package also provides administrative procedures that prepare database objects at the source database for instantiation at a destination database.

DBMS_PROPAGATION_ADM Package

The DBMS_PROPAGATION_ADM package provides an administrative interface for configuring propagation from a source queue to a destination queue.

DBMS_APPLY_ADM Package

The DBMS_APPLY_ADM package provides an administrative interface for starting, stopping, and configuring an apply process. This package includes procedures that enable you to configure apply handlers, set enqueue destinations for events, and specify execution directives for events. This package also provides administrative procedures that set the instantiation SCN for objects at a destination database. This package also includes subprograms for configuring conflict detection and resolution and for managing apply errors.

DBMS_STREAMS_MESSAGING Package

The DBMS_STREAMS_MESSAGING package provides interfaces to enqueue messages into and dequeue messages from a SYS.AnyData queue.

DBMS_RULE_ADM Package

The DBMS_RULE_ADM package provides an administrative interface for creating and managing rules, rule sets, and rule evaluation contexts. This package also contains subprograms for managing privileges related to rules.

DBMS_RULE Package

The DBMS_RULE package contains the EVALUATE procedure, which evaluates a rule set. The goal of this procedure is to produce the list of satisfied rules, based on the data. This package also contains subprograms that enable you to use iterators during rule evaluation. Instead of returning all rules that evaluate to TRUE or MAYBE for an evaluation, iterators can return one rule at a time.

DBMS_STREAMS Package

The DBMS_STREAMS package provides interfaces to convert SYS.AnyData objects into LCR objects, to return information about Streams attributes and Streams clients, and to annotate redo entries generated by a session with a tag. This tag may affect the behavior of a capture process, a propagation job, an apply process, or a messaging client whose rules include specifications for these tags in redo entries or LCRs.

DBMS_STREAMS_AUTH Package

The DBMS_STREAMS_AUTH package provides interfaces for granting privileges to Streams administrators and revoking privileges from Streams administrators.

DBMS_STREAMS_TABLESPACE_ADM

The DBMS_STREAMS_TABLESPACE_ADM package provides administrative procedures for copying tablespaces between databases and moving tablespaces from one database to another. This package uses transportable tablespaces, Data Pump, and the DBMS_FILE_TRANSFER package.

Streams Data Dictionary Views

Every database in a Streams environment has Streams data dictionary views. These views maintain administrative information about local rules, objects, capture processes, propagations, apply processes, and messaging clients. You can use these views to monitor your Streams environment.

Streams Tool in the Oracle Enterprise Manager Console

To help configure, administer, and monitor Streams environments, Oracle provides a Streams tool in the Oracle Enterprise Manager Console. You also can use the Streams tool to generate Streams configuration scripts, which you can then modify and run to configure your Streams environment. The Streams tool online help is the primary documentation source for this tool. Figure 1-14 shows the Topology tab in the Streams tool.

Figure 1-14 Streams Tool

Text description of the illustration strms_to.jpg