Oracle® Streams Concepts and Administration 10g Release 1 (10.1) Part Number B10727-01 |
|
|
View PDF |
This chapter explains the concepts relating to staging events in a queue and propagating events from one queue to another.
This chapter contains these topics:
Streams uses queues of type SYS.AnyData
to stage events. In Streams, there are two types of events that can be encapsulated into a SYS.AnyData
event and staged in a SYS.AnyData
queue: logical change records (LCRs) and user messages. An LCR is an object that contains information about a change to a database object, while a user message is a message of a user-defined type created by users or applications. Both types of events can be used for information sharing within a single database or between databases.
Staged events can be consumed or propagated, or both. Staged events can be consumed by an apply process, by a messaging client, or by a user application. A running apply process implicitly dequeues events, but messaging clients and user applications explicitly dequeue events. Even after an event is consumed, it may remain in the queue if you also have configured Streams to propagate the event to one or more other queues or if message retention is specified for user-enqueued messages. The queues to which events are propagated may reside in the same database or in different databases than the queue from which the events are propagated. In either case, the queue from which the events are propagated is called the source queue, and the queue that receives the events is called the destination queue. There can be a one-to-many, many-to-one, or many-to-many relationship between source and destination queues. Figure 3-1 shows propagation from a source queue to a destination queue.
Text description of the illustration strms008.gif
You can create, alter, and drop a propagation, and you can define propagation rules that control which events are propagated. The user who owns the source queue is the user who propagates events, and this user must have the necessary privileges to propagate events.
These privileges include the following:
If the propagation propagates events to a destination queue in a remote database, then the owner of the source queue must be able to use the propagation's database link, and the user to which the database link connects at the remote database must have enqueue privilege on the destination queue.
Note:
|
See Also:
|
Events can be enqueued in two ways:
SYS.AnyData
. These user messages can contain LCRs or any other type of message. Any user message that was explicitly enqueued by a user or an application is called a user-enqueued event. Events that were enqueued by a user procedure called from an apply process are also user-enqueued events.So, each captured event contains an LCR, but a user-enqueued event may or may not contain an LCR. Propagating a captured event or a user-enqueued event enqueues the event into the destination queue.
Events can be dequeued in two ways:
SET_ENQUEUE_DESTINATION
procedure in the DBMS_APPLY_ADM
package are user-enqueued events.The dequeued events may have originated at the same database where they are dequeued, or they may have originated at a different database.
See Also:
|
You can use Streams to configure event propagation between two queues, which may reside in different databases. Streams uses job queues to propagate events.
A propagation is always between a source queue and a destination queue. Although propagation is always between two queues, a single queue may participate in many propagations. That is, a single source queue may propagate events to multiple destination queues, and a single destination queue may receive events from multiple source queues. However, only one propagation is allowed between a particular source queue and a particular destination queue. Also, a single queue may be a destination queue for some propagations and a source queue for other propagations.
A propagation may propagate all of the events in a source queue to a destination queue, or a propagation may propagate only a subset of the events. Also, a single propagation can propagate both captured and user-enqueued events. You can use rules to control which events in the source queue are propagated to the destination queue and which events are discarded.
Depending on how you set up your Streams environment, changes could be sent back to the site where they originated. You need to ensure that your environment is configured to avoid cycling a change in an endless loop. You can use Streams tags to avoid such a change cycling loop.
See Also:
|
A propagation either propagates or discards events based on rules that you define. For LCR events, each rule specifies the database objects and types of changes for which the rule evaluates to TRUE
. You can place these rules in a positive rule set for the propagation or a negative rule set for the propagation.
If a rule evaluates to TRUE
for an event, and the rule is in the positive rule set for a propagation, then the propagation propagates the change. If a rule evaluates to TRUE
for an event, and the rule is in the negative rule set for a propagation, then the propagation discards the change. If a propagation has both a positive and a negative rule set, then the negative rule set is always evaluated first.
You can specify propagation rules for LCR events at the following levels:
For non-LCR events, you can create your own rules to control propagation.
A queue subscriber that specifies a condition causes the system to generate a rule. The rule sets for all subscribers to a queue are combined into a single system-generated rule set to make subscription more efficient.
A user-enqueued event is propagated successfully to a destination queue when the enqueue into the destination queue is committed. A captured event is propagated successfully to a destination queue when both of the following actions are completed:
When an event is successfully propagated between two SYS.AnyData
queues, the destination queue acknowledges successful propagation of the event. If the source queue is configured to propagate an event to multiple destination queues, then the event remains in the source queue until each destination queue has sent confirmation of event propagation to the source queue. When each destination queue acknowledges successful propagation of the event, and all local consumers in the source queue database have consumed the event, the source queue can drop the event.
This confirmation system ensures that events are always propagated from the source queue to the destination queue, but, in some configurations, the source queue can grow larger than an optimal size. When a source queue grows, it uses more SGA memory and may use more disk space.
There are two common reasons for source-queue growth:
A directed network is one in which propagated events may pass through one or more intermediate databases before arriving at a destination database. An event may or may not be processed by an apply process at an intermediate database. Using Streams, you can choose which events are propagated to each destination database, and you can specify the route that events will traverse on their way to a destination database. Figure 3-2 shows an example of a directed networks environment.
Text description of the illustration strms007.gif
The advantage of using a directed network is that a source database does not need to have a physical network connection with a destination database. So, if you want events to propagate from one database to another, but there is no direct network connection between the computers running these databases, then you can still propagate the events without reconfiguring your network, as long as one or more intermediate databases connect the source database to the destination database.
If you use directed networks, and an intermediate site goes down for an extended period of time or is removed, then you may need to reconfigure the network and the Streams environment.
An intermediate database in a directed network may propagate events using queue forwarding or apply forwarding. Queue forwarding means that the events being forwarded at an intermediate database are the events received by the intermediate database. The source database for an event is the database where the event originated.
Apply forwarding means that the events being forwarded at an intermediate database are first processed by an apply process. These events are then recaptured by a capture process at the intermediate database and forwarded. When you use apply forwarding, the intermediate database becomes the new source database for the events because the events are recaptured from the redo log generated there.
Consider the following differences between queue forwarding and apply forwarding when you plan your Streams environment:
A single Streams environment may use a combination of queue forwarding and apply forwarding.
Queue forwarding has the following advantages compared to apply forwarding:
GET_SOURCE_DATABASE_NAME
member procedure on the LCR contained in the event. If you use apply forwarding, then determining the origin of an event requires the use of Streams tags and apply handlers.If you use apply forwarding, then substantially more work may be required to reconfigure end-to-end capture, propagation, and apply of events, because the destination database(s) downstream from the unavailable intermediate database were using the SCN information of this intermediate database. Without this SCN information, the destination databases cannot apply the changes properly.
Apply forwarding has the following advantages compared to queue forwarding:
See Also:
|
You can propagate a binary file between databases by using Streams. To do so, you put one or more BFILE
attributes in a message payload and then propagate the message to a remote queue. Each BFILE
referenced in the payload is transferred to the remote database after the message is propagated, but before the message propagation is committed. The directory object and filename of each propagated BFILE
are preserved, but you can map the directory object to different directories on the source and destination databases. The message payload can be a BFILE
wrapped in a SYS.AnyData
payload, or the message payload can be one or more BFILE
attributes of an object wrapped in a SYS.AnyData
payload.
The following are not supported in a message payload:
BFILE
attributes in a varraySYS.AnyData
attribute that contains one or more BFILE
attributesPropagating a BFILE
in Streams has the same restrictions as the procedure DBMS_FILE_TRANSFER.PUT_FILE
.
See Also:
Oracle Database Concepts, Oracle Database Administrator's Guide, and PL/SQL Packages and Types Reference for more information about transferring files with the |
A messaging client consumes user-enqueued events when it is invoked by an application or a user. You use rules to specify which user-enqueued events in the queue are dequeued by a messaging client. These user-enqueued events may be user-enqueued LCRs or user-enqueued messages.
You can create a messaging client by specifying dequeue
for the streams_type
parameter when you run one of the following procedures in the DBMS_STREAMS_ADM
package:
When you create a messaging client, you specify the name of the messaging client and the SYS.AnyData
queue from which the messaging client dequeues messages. These procedures also can add rules to the positive or negative rule set of a messaging client. You specify the message type for each rule, and a single messaging client can dequeue messages of different types.
The user who creates a messaging client is granted the privileges to dequeue from the queue using the messaging client. This user is the messaging client user. The messaging client user can dequeue messages that satisfy the messaging client rule sets. A messaging client can be associated with only one user, but one user may be associated with many messaging clients.
Figure 3-3 shows a messaging client dequeuing user-enqueued events.
Text description of the illustration strms045.gif
See Also:
|
Streams enables messaging with queues of type SYS.AnyData
. These queues can stage user messages whose payloads are of SYS.AnyData
type. A SYS.AnyData
payload can be a wrapper for payloads of different datatypes.
By using SYS.AnyData
wrappers for message payloads, publishing applications can enqueue messages of different types into a single queue, and subscribing applications can dequeue these messages, either explicitly using a messaging client or an application, or implicitly using an apply process. If the subscribing application is remote, then the messages can be propagated to the remote site, and the subscribing application can dequeue the messages from a local queue in the remote database. Alternatively, a remote subscribing application can dequeue messages directly from the source queue using a variety of standard protocols, such as PL/SQL and OCI.
Streams includes the features of Advanced Queuing (AQ), which supports all the standard features of message queuing systems, including multiconsumer queues, publish and subscribe, content-based routing, internet propagation, transformations, and gateways to other messaging subsystems.
You can wrap almost any type of payload in a SYS.AnyData
payload. To do this, you use the Convert
data_type
static functions of the SYS.AnyData
type, where data_type
is the type of object to wrap. These functions take the object as input and return a SYS.AnyData
object.
You cannot enqueue SYS.AnyData
payloads that contain payloads of the following datatypes into a SYS.AnyData
queue:
In addition, you cannot enqueue SYS.AnyData
payloads that contain payloads of object types with attributes of these datatypes into a SYS.AnyData
queue, nor object types that use type evolution or type inheritance.
See Also:
|
You can configure a SYS.AnyData
queue to stage and propagate captured and user-enqueued events in a Real Application Clusters (RAC) environment. In a RAC environment, only the owner instance may have a buffer for a queue, but different instances may have buffers for different queues. A buffered queue is System Global Area (SGA) memory associated with a SYS.AnyData
queue that contains only captured events. Buffered queues are discussed in more detail later in this chapter.
A SYS.AnyData
queue that contains only user-enqueued events behaves the same as a typed queue in a RAC environment. However, if a SYS.AnyData
queue contains or will contain captured events in a RAC environment, then each propagation that propagates captured events to a RAC destination database must use an instance-specific database link that refers to the owner instance of the destination queue. If the propagation connects to any other instance, then the propagation will raise an error.
Each capture process and apply process is started on the owner instance for its SYS.AnyData
queue, even if the start procedure is run on a different instance. If the owner instance for a queue table containing a destination queue becomes unavailable, then queue ownership is transferred automatically to another instance in the cluster. If this happens, then database links from remote source queues must be reconfigured manually to connect to the instance that owns the destination queue.
Streams processes and jobs support primary instance and secondary instance specifications for queue tables. If you use these specifications, then the secondary instance assumes ownership of a queue table when the primary instance becomes unavailable, and ownership is transferred back to the primary instance when it becomes available again. If both the primary and secondary instance for a queue table containing a destination queue become unavailable, then queue ownership is transferred automatically to another instance in the cluster. In this case, if the primary or secondary instance becomes available again, then ownership is transferred back to one of them accordingly. You can set primary and secondary instance specifications using the ALTER_QUEUE_TABLE
procedure in the DBMS_AQADM
package.
The DBA_QUEUE_TABLES
data dictionary view contains information about the owner instance for a queue table. A queue table may contain multiple queues. In this case, each queue in a queue table has the same owner instance as the queue table.
See Also:
|
In general, SYS.AnyData
queues use the same infrastructure as typed queues. However, unlike typed queues, which stage all events in a queue table, SYS.AnyData
queues have a buffered queue to stage captured events in shared memory. This section describes buffered queues, propagation jobs, and secure queues, and how they are used in Streams. In addition, this section discusses how transactional queues handle captured and user-enqueued events, as well as the need for a Streams data dictionary at databases that propagate captured events.
This section contains the following topics:
See Also:
|
A buffered queue includes the following storage areas:
SYS.AnyData
queue that contains only captured eventsSYS.AnyData
queue that stores captured events that have spilled from memoryA buffered queue enables Oracle to optimize captured events by buffering captured events in the SGA instead of always storing them in a queue table. This buffering of captured events happens in any database where captured events are staged in a SYS.AnyData
queue. Such a database may be a source database, an intermediate database, or a destination database. Captured events are always stored in a buffered queue, but user-enqueued LCR events and user-enqueued non-LCR events are always stored in queue tables, not in buffered queues. Also, when a transaction is moved to the error queue, all events in the transaction are stored in a queue table, not in a buffered queue.
Buffered queues improve performance, but some of the information in a buffered queue may be lost if the instance containing the buffered queue shuts down normally or abnormally. Streams automatically recovers from these cases, assuming full database recovery is performed on the instance.
In a single database, you can specify that Streams memory be allocated from a new pool in the SGA called the Streams pool. To configure the Streams pool, specify the size of the pool in bytes using the STREAMS_POOL_SIZE
initialization parameter. If the size of the Streams pool is greater than zero, then any SGA memory used by Streams is allocated from the Streams pool. If the size of the Streams pool is zero, then the memory used by Streams is allocated from the shared pool and may use up to 10% of the shared pool.
Captured events in a buffered queue may spill from memory into the queue table if they have been staged in the buffered queue for a period of time without being dequeued, or if there is not enough space in memory to hold all of the captured events. Captured events that spill from memory are stored in the appropriate AQ$_
queue_table_name
_p
table, where queue_table_name
is the name of the queue table for the queue.
See Also:
|
A Streams propagation is configured internally using the DBMS_JOBS
package. Therefore, a propagation job is the mechanism that propagates events from a source queue to a destination queue. Like other jobs configured using the DBMS_JOBS
package, propagation jobs have an owner, and they use job queue processes (J
nnn
) as needed to execute jobs.
A propagation job may be used by more than one propagation. All destination queues at a database receive events from a single source queue through a single propagation job. By using a single propagation job for multiple destination queues, Streams ensures that an event is sent to a destination database only once, even if the same message is received by multiple destination queues in the same database. Communication resources are conserved because messages are not sent more than once to the same database.
A propagation schedule specifies how often a propagation job propagates events from a source queue to a destination queue. Therefore, all propagations that use a propagation job have the same propagation schedule. A default propagation schedule is established for the new propagation job when you create the propagation job using one of the following procedures:
ADD_GLOBAL_PROPAGATION_RULES
procedure in the DBMS_STREAMS_ADM
packageADD_SCHEMA_PROPAGATION_RULES
procedure in the DBMS_STREAMS_ADM
packageADD_TABLE_PROPAGATION_RULES
procedure in the DBMS_STREAMS_ADM
packageADD_SUBSET_PROPAGATION_RULE
procedure in the DBMS_STREAMS_ADM
packageCREATE_PROPAGATION
procedure in the DBMS_PROPAGATION_ADM
packageThe default schedule has the following properties:
SYSDATE()
.NULL
, which means infinite.NULL
, which means that propagation restarts as soon as it finishes the current duration.If you want to alter the default schedule for a propagation job, then use the ALTER_PROPAGATION_SCHEDULE
procedure in the DBMS_AQADM
package.
When the restricted session is enabled during system startup by issuing a STARTUP
RESTRICT
statement, propagation jobs with enabled propagation schedules do not propagate events. When the restricted session is disabled, each propagation schedule that is enabled and ready to run will run when there is an available job queue process.
When the restricted session is enabled in a running database by the SQL statement ALTER
SYSTEM
with the ENABLE
RESTRICTED
SESSION
clause, any running propagation job continues to run to completion. However, any new propagation job submitted for a propagation schedule is not started. Therefore, propagation for an enabled schedule may eventually come to a halt.
Secure queues are queues for which AQ agents must be associated explicitly with one or more database users who can perform queue operations, such as enqueue and dequeue. The owner of a secure queue can perform all queue operations on the queue, but other users cannot perform queue operations on a secure queue, unless they are configured as secure queue users. In Streams, secure queues can be used to ensure that only the appropriate users and Streams clients enqueue events into a queue and dequeue events from a queue.
All SYS.AnyData
queues created using the SET_UP_QUEUE
procedure in the DBMS_STREAMS_ADM
package are secure queues. When you use the SET_UP_QUEUE
procedure to create a queue, any user specified by the queue_user
parameter is configured as a secure queue user of the queue automatically, if possible. The queue user is also granted ENQUEUE
and DEQUEUE
privileges on the queue. To enqueue events into and dequeue events from a queue, a queue user must also have EXECUTE
privilege on the DBMS_STREAMS_MESSAGING
package or the DBMS_AQ
package. The SET_UP_QUEUE
procedure does not grant either of these privileges. Also, an event cannot be enqueued into a queue unless a subscriber who can dequeue the event is configured.
To configure a queue user as a secure queue user, the SET_UP_QUEUE
procedure creates an AQ agent with the same name as the user name, if one does not already exist. The user must use this agent to perform queue operations on the queue. If an agent with this name already exists and is associated with the queue user only, then the existing agent is used. SET_UP_QUEUE
then runs the ENABLE_DB_ACCESS
procedure in the DBMS_AQADM
package, specifying the agent and the user.
If you use the SET_UP_QUEUE
procedure in the DBMS_STREAMS_ADM
package to create a secure queue, and you want a user who is not the queue owner and who was not specified by the queue_user
parameter to perform operations on the queue, then you can configure the user as a secure queue user of the queue manually. Alternatively, you can run the SET_UP_QUEUE
procedure again and specify a different queue_user
for the queue. In this case, SET_UP_QUEUE
will skip queue creation, but it will configure the user specified by queue_user
as a secure queue user of the queue.
If you create a SYS.AnyData
queue using the DBMS_AQADM
package, then you use the secure
parameter when you run the CREATE_QUEUE_TABLE
procedure to specify whether the queue is secure or not. The queue is secure if you specify true
for the secure
parameter when you run this procedure. When you use the DBMS_AQADM
package to create a secure queue, and you want to allow users to perform queue operations on the secure queue, then you must configure these secure queue users manually.
When you create a capture process or an apply process, an AQ agent of the secure queue associated with the Streams process is configured automatically, and the user who runs the Streams process is specified as a secure queue user for this queue automatically. Therefore, a capture process is configured to enqueue into its secure queue automatically, and an apply process is configured to dequeue from its secure queue automatically. In either case, the AQ agent has the same name as the Streams client.
For a capture process, the user specified as the capture_user
is the user who runs the capture process. For an apply process, the user specified as the apply_user
is the user who runs the apply process. If no capture_user
or apply_user
is specified, then the user who invokes the procedure that creates the Streams process is the user who runs the Streams process.
Also, if you change the capture_user
for a capture process or the apply_user
for an apply process, then the specified capture_user
or apply_user
is configured as a secure queue user of the queue used by the Streams process. However, the old capture user or apply user remains configured as a secure queue user of the queue. To remove the old user, run the DISABLE_DB_ACCESS
procedure in the DBMS_AQADM
package, specifying the old user and the relevant AQ agent. You may also want to drop the agent if it is no longer needed. You can view the AQ agents and their associated users by querying the DBA_AQ_AGENT_PRIVS
data dictionary view.
When you create a messaging client, an AQ agent of the secure queue with the same name as the messaging client is associated with the user who runs the procedure that creates the messaging client. This messaging client user is specified as a secure queue user for this queue automatically. Therefore, this user can use the messaging client to dequeue messages from the queue.
A capture process, an apply process, or a messaging client can be associated with only one user. However, one user may be associated with multiple Streams clients, including multiple capture processes, apply processes, and messaging clients. For example, an apply process cannot have both hr
and oe
as apply users, but hr
may be the apply user for multiple apply processes.
If you drop a capture process, apply process, or messaging client, then the users who were configured as secure queue users for these Streams clients remain secure queue users of the queue. To remove these users as secure queue users, run the DISABLE_DB_ACCESS
procedure in the DBMS_AQADM
package for each user. You may also want to drop the agent if it is no longer needed.
See Also:
|
A transactional queue is one in which user-enqueued events can be grouped into a set that are applied as one transaction. That is, an apply process performs a COMMIT
after it applies all the user-enqueued events in a group. The SET_UP_QUEUE
procedure in the DBMS_STREAMS_ADM
package always creates a transactional queue.
A nontransactional queue is one in which each user-enqueued event is its own transaction. That is, an apply process performs a COMMIT
after each user-enqueued event it applies. In either case, the user-enqueued events may or may not contain user-created LCRs.
The difference between transactional and nontransactional queues is important only for user-enqueued events. An apply process always applies captured events in transactions that preserve the transactions executed at the source database. Table 3-1 shows apply process behavior for each type of event and each type of queue.
See Also:
|
When a database object is prepared for instantiation at a source database, a Streams data dictionary is populated automatically at the database where changes to the object are captured by a capture process. The Streams data dictionary is a multiversioned copy of some of the information in the primary data dictionary at a source database. The Streams data dictionary maps object numbers, object version information, and internal column numbers from the source database into table names, column names, and column datatypes. This mapping keeps each captured event as small as possible because the event can store numbers rather than names internally.
The mapping information in the Streams data dictionary at the source database is needed to evaluate rules at any database that propagates the captured events from the source database. To make this mapping information available to a propagation, Oracle automatically populates a multiversioned Streams data dictionary at each database that has a Streams propagation. Oracle automatically sends internal messages that contain relevant information from the Streams data dictionary at the source database to all other databases that receive captured events from the source database.
The Streams data dictionary information contained in these internal messages in a queue may or may not be propagated by a propagation. Which Streams data dictionary information to propagate depends on the rule sets for the propagation. When a propagation encounters Streams data dictionary information for a table, the propagation rule sets are evaluated with partial information that includes the source database name, table name, and table owner. If the partial rule evaluation of these rule sets determines that there may be relevant LCRs for the given table from the specified database, then the Streams data dictionary information for the table is propagated.
When Streams data dictionary information is propagated to a destination queue, it is incorporated into the Streams data dictionary at the database that contains the destination queue, in addition to being enqueued into the destination queue. Therefore, a propagation reading the destination queue in a directed networks configuration can forward LCRs immediately without waiting for the Streams data dictionary to be populated. In this way, the Streams data dictionary for a source database always reflects the correct state of the relevant database objects for the LCRs relating to these database objects.