| Oracle® Text Application Developer's Guide 10g Release 2 (10.2) Part Number B14217-01 |
|
|
View PDF |
| Oracle® Text Application Developer's Guide 10g Release 2 (10.2) Part Number B14217-01 |
|
|
View PDF |
This chapter describes how to use document sections in an Oracle Text query application.
The following topics are discussed in this chapter:
Section searching enables you to narrow text queries down to blocks of text within documents. Section searching is useful when your documents have internal structure, such as HTML and XML documents.
You can also search for text at the sentence and paragraph level.
The steps for enabling section searching for your document collection are:
Create a section group
Define your sections
Index your documents
Section search with WITHIN, INPATH, or HASPATH operators
Section searching is enabled by defining section groups. You use one of the system-defined section groups to create an instance of a section group. Choose a section group appropriate for your document collection.
You use section groups to specify the type of document set you have and implicitly indicate the tag structure. For instance, to index HTML tagged documents, you use the HTML_SECTION_GROUP. Likewise, to index XML tagged documents, you can use the XML_SECTION_GROUP.
Table 8-1 lists the different types of section groups you can use:
Table 8-1 Types of Section Groups
|
Note: Documents sent to theHTML, XML, AUTO and PATH sectioners must begin with \s*<, where \s* represents zero or more whitespace characters. Otherwise, the document is treated as a plaintext document, and no sections are recognized. |
You use the CTX_DDL package to create section groups and define sections as part of section groups. For example, to index HTML documents, create a section group with HTML_SECTION_GROUP:
begin
ctx_ddl.create_section_group('htmgroup', 'HTML_SECTION_GROUP');
end;
You define sections as part of the section group. The following example defines an zone section called heading for all text within the HTML < H1> tag:
begin
ctx_ddl.create_section_group('htmgroup', 'HTML_SECTION_GROUP');
ctx_ddl.add_zone_section('htmgroup', 'heading', 'H1');
end;
|
Note: If you are using theAUTO_SECTION_GROUP or PATH_SECTION_GROUP to index an XML document collection, you need not explicitly define sections since the system does this for you during indexing. |
|
See Also:
|
When you index your documents, you specify your section group in the parameter clause of CREATE INDEX.
create index myindex on docs(htmlfile) indextype is ctxsys.context
parameters('filter ctxsys.null_filter section group htmgroup');
When your documents are indexed, you can query within sections using the WITHIN operator. For example, to find all the documents that contain the word Oracle within their headings, issue the following query:
'Oracle WITHIN heading'
When you use the PATH_SECTION_GROUP, the system automatically creates XML sections for you. In addition to using the WITHIN operator to issue queries, you can issue path queries with the INPATH and HASPATH operators.
|
See Also:
|
All sections types are blocks of text in a document. However, sections can differ in the way they are delimited and the way they are recorded in the index. Sections can be one of the following:
Attribute Section (for XML documents)
Special Sections (sentence or paragraphs)
Table 8-2 shows which section types may be used with each kind of section group.
Table 8-2 Section Types and Section Groups
| Section Group | ZONE | FIELD | SPECIAL | STOP | ATTRIBUTE | MDATA |
|---|---|---|---|---|---|---|
| NULL | NO | NO | YES | NO | NO | NO |
| BASIC | YES | YES | YES | NO | NO | YES |
| HTML | YES | YES | YES | NO | NO | YES |
| XML | YES | YES | YES | NO | YES | YES |
| NEWS | YES | YES | YES | NO | NO | YES |
| AUTO | NO | NO | NO | YES | NO | NO |
| PATH | NO | NO | NO | NO | NO | NO |
A zone section is a body of text delimited by start and end tags in a document. The positions of the start and end tags are recorded in the index so that any words in between the tags are considered to be within the section. Any instance of a zone section must have a start and an end tag.
For example, the text between the <TITLE> and </TITLE> tags can be defined as a zone section as follows:
<TITLE>Tale of Two Cities</TITLE> It was the best of times...
Zone sections can nest, overlap, and repeat within a document.
When querying zone sections, you use the WITHIN operator to search for a term across all sections. Oracle Text returns those documents that contain the term within the defined section.
Zone sections are well suited for defining sections in HTML and XML documents. To define a zone section, use CTX_DDL.ADD_ZONE_SECTION.
For example, assume you define the section booktitle as follows:
begin
ctx_ddl.create_section_group('htmgroup', 'HTML_SECTION_GROUP');
ctx_ddl.add_zone_section('htmgroup', 'booktitle', 'TITLE');
end;
After you index, you can search for all the documents that contain the term Cities within the section booktitle as follows:
'Cities WITHIN booktitle'
With multiple query terms such as (dog and cat) WITHIN booktitle, Oracle Text returns those documents that contain cat and dog within the same instance of a booktitle section.
Zone sections can repeat. Each occurrence is treated as a separate section. For example, if <H1> denotes a heading section, they can repeat in the same documents as follows:
<H1> The Brown Fox </H1> <H1> The Gray Wolf </H1>
Assuming that these zone sections are named Heading, the query Brown WITHIN Heading returns this document. However, a query of (Brown and Gray) WITHIN Heading does not.
Zone sections can overlap each other. For example, if <B> and <I> denote two different zone sections, they can overlap in a document as follows:
plain <B> bold <I> bold and italic </B> only italic </I> plain
Zone sections can nest, including themselves as follows:
<TD> <TABLE><TD>nested cell</TD></TABLE></TD>
Using the WITHIN operator, you can write queries to search for text in sections within sections. For example, assume the BOOK1, BOOK2, and AUTHOR zone sections occur as follows in documents doc1 and doc2:
doc1:
<book1> <author>Scott Tiger</author> This is a cool book to read.</book1>
doc2:
<book2> <author>Scott Tiger</author> This is a great book to read.</book2>
Consider the nested query:
'(Scott within author) within book1'
This query returns only doc1.
A field section is similar to a zone section in that it is a region of text delimited by start and end tags. A field section is different from a zone section in that the region is indexed separately from the rest of the document.
Since field sections are indexed differently, you can also get better query performance over zone sections for when you have a large number of documents indexed.
Field sections are more suited to when you have a single occurrence of a section in a a document such as a field in a news header. Field sections can also be made visible to the rest of the document.
Unlike zone sections, field sections have the following restrictions:
Field sections cannot overlap
Field sections cannot repeat
Field sections cannot nest
By default, field sections are indexed as a sub-document separate from the rest of the document. As such, field sections are invisible to the surrounding text and can only be queried by explicitly naming the section in the WITHIN clause.
You can make field sections visible if you want the text within the field section to be indexed as part of the enclosing document. Text within a visible field section can be queried with or without the WITHIN operator.
The following example shows the difference between using invisible and visible field sections.
The following code defines a section group basicgroup of the BASIC_SECTION_GROUP type. It then creates a field section in basicgroup called Author for the <A> tag. It also sets the visible flag to FALSE to create an invisible section:
begin
ctx_ddl.create_section_group('basicgroup', 'BASIC_SECTION_GROUP');
ctx_ddl.add_field_section('basicgroup', 'Author', 'A', FALSE);
end;
Because the Author field section is not visible, to find text within the Author section, you must use the WITHIN operator as follows:
'(Martin Luther King) WITHIN Author'
A query of Martin Luther King without the WITHIN operator does not return instances of this term in field sections. If you want to query text within field sections without specifying WITHIN, you must set the visible flag to TRUE when you create the section as follows:
begin
ctx_ddl.add_field_section('basicgroup', 'Author', 'A', TRUE);
end;
Field sections cannot be nested. For example, if you define a field section to start with <TITLE> and define another field section to start with <FOO>, the two sections cannot be nested as follows:
<TITLE> dog <FOO> cat </FOO> </TITLE>
To work with nested sections, define them as zone sections.
Repeated field sections are allowed, but WITHIN queries treat them as a single section. The following is an example of repeated field section in a document:
<TITLE> cat </TITLE> <TITLE> dog </TITLE>
The query dog and cat within title returns the document, even though these words occur in different sections.
To have WITHIN queries distinguish repeated sections, define them as zone sections.
A stop section may be added to an automatic section group. Adding a stop section causes the automatic section indexing operation to ignore the specified section in XML documents.
|
Note: Adding a stop section causes no section information to be created in the index. However, the text within a stop section is always searchable. |
Adding a stop section is useful when your documents contain many low-information tags. Adding stop sections also improves indexing performance with the automatic section group.
The number of stop sections you can add is unlimited.
Stop sections do not have section names and hence are not recorded in the section views.
An MDATA section is used to reference user-defined metadata for a document. Using MDATA sections can speed up mixed queries.
Consider the case in which you want to query both according to text content and document type (magazine or newspaper or novel). You could create an index with a column for text and a column for the document type, and then perform a mixed query of this form—in this case, searching for all novels with the phrase Adam Thorpe (author of the novel Ulverton):
SELECT id FROM documents
WHERE doctype = 'novel'
AND CONTAINS(text, 'Adam Thorpe')>0;
However, it is usually faster to incorporate the attribute (in this case, the document type) into a field section, rather than use a separate column, and then use a single CONTAINS query:
SELECT id FROM documents WHERE CONTAINS(text, 'Adam Thorpe AND novel WITHIN doctype')>0;
There are two drawbacks to this approach:
Each time the attribute is updated, the entire text document must be re-indexed, resulting in increased index fragmentation and slower rates of processing DML.
Field sections tokenize the section value. This has several effects. Special characters in metadata, such as decimal points or currency characters, are not easily searchable; value searching (searching for Thurston Howell but not Thurston Howell, Jr.) is difficult; multi-word values are queried by phrase, which is slower than single-token searching; and multi-word values do not show up in browse-words, making author browsing or subject browsing impossible.
For these reasons, using MDATA sections instead of field sections may be worthwhile. MDATA sections are indexed like field sections, but metadata values can be added to and removed from documents without the need to re-index the document text. Unlike field sections, MDATA values are not tokenized. Additionally, MDATA section indexing generally takes up less disk space than field section indexing.
Use CTX_DDL.ADD_MDATA_SECTION to add an MDATA section to a section group. This example adds an MDATA section called AUTHOR and gives it the value Soseki Natsume (author of the novel Kokoro).
ctx_ddl.create.section.group('htmgroup', 'HTML_SECTION_GROUP');
ctx_ddl.add_mdata_section('htmgroup', 'author', 'Soseki Natsume');
MDATA values can be changed with CTX_DDL.ADD_MDATA and removed with CTX_DDL.REMOVE_MDATA. MDATA sections can have multiple values. Only the owner of the index is allowed to call CTX_DDL.ADD_MDATA and CTX_DDL.REMOVE_MDATA.
Neither CTX_DDL.ADD_MDATA nor CTX_DDL.REMOVE_MDATA are supported for CTXCAT, CTXXPTH and CTXRULE indexes.
MDATA values are not passed through a lexer. Instead, all values undergo a simplified normalization:
Leading and trailing whitespace on the value is removed.
The value is truncated to 64 bytes.
The value is converted to upper case.
The value is indexed as a single value; if the value consists of multiple words, it is not broken up.
Case is preserved. If the document is dynamically generated, you can implement case-insensitivity by uppercasing MDATA values and making sure to search only in uppercase.
Once a document has had MDATA metadata added to it, you can query for that metadata using the MDATA CONTAINS query operator:
SELECT id FROM documents WHERE CONTAINS(text, 'Tokyo and MDATA(author, Soseki Natsume)')>0;
This query will only be successful if an AUTHOR tag has the exact value Soseki Natsume (after simplified tokenization). Soseki or Natsume Soseki will not work.
Other things to note about MDATA:
MDATA values are not highlightable, will not appear in the output of CTX_DOC.TOKENS, and will not show up when FILTER PLAINTEXT is enabled.
MDATA sections must be unique within section groups. You cannot have an MDATA section named FOO and a zone or field section of the same name in the same section group.
Like field sections, MDATA sections cannot overlap or nest. An MDATA section is implicitly closed by the first tag encountered. For instance, in this example:
<AUTHOR>Dickens <B>Shelley</B> Keats</AUTHOR>
The <B> tag closes the AUTHOR MDATA section; as a result, this document has an AUTHOR of 'Dickens', but not of 'Shelley' or 'Keats'.
To prevent race conditions, each call to ADD_MDATA and REMOVE_MDATA locks out other calls on that rowid for that index for all values and sections. However, since ADD_MDATA and REMOVE_MDATA do not commit, it is possible for an application to deadlock when calling them both. It is the application's responsibility to prevent deadlocking.
|
See Also:
|
You can define attribute sections to query on XML attribute text. You can also have the system automatically define and index XML attributes for you.
Special sections are not recognized by tags. Currently the only special sections supported are sentence and paragraph. This enables you to search for combination of words within sentences or paragraphs.
The sentence and paragraph boundaries are determined by the lexer. For example, the BASIC_LEXER recognizes sentence and paragraph section boundaries as follows:
Table 8-3 Sentence and Paragraph Section Boundaries for BASIC_LEXER
| Special Section | Boundary |
|---|---|
| SENTENCE | WORD/PUNCT/WHITESPACE |
| WORD/PUNCT/NEWLINE | |
| PARAGRAPH | WORD/PUNCT/NEWLINE/WHITESPACE |
| WORD/PUNCT/NEWLINE/NEWLINE |
If the lexer cannot recognize the boundaries, no sentence or paragraph sections are indexed.
To add a special section, use the CTX_DDL.ADD_SPECIAL_SECTION procedure. For example, the following code enables searching within sentences within HTML documents:
begin
ctx_ddl.create_section_group('htmgroup', 'HTML_SECTION_GROUP');
ctx_ddl.add_special_section('htmgroup', 'SENTENCE');
end;
You can also add zone sections to the group to enable zone searching in addition to sentence searching. The following example adds the zone section Headline to the section group htmgroup:
begin
ctx_ddl.create_section_group('htmgroup', 'HTML_SECTION_GROUP');
ctx_ddl.add_special_section('htmgroup', 'SENTENCE');
ctx_ddl.add_zone_section('htmgroup', 'Headline', 'H1');
end;
HTML has internal structure in the form of tagged text which you can use for section searching. For example, you can define a section called headings for the <H1> tag. This enables you to search for terms only within these tags across your document set.
To query, you use the WITHIN operator. Oracle Text returns all documents that contain your query term within the headings section. Thus, if you wanted to find all documents that contain the word oracle within headings, you issue the following query:
'oracle within headings'
The following code defines a section group called htmgroup of type HTML_SECTION_GROUP. It then creates a zone section in htmgroup called heading identified by the <H1> tag:
begin
ctx_ddl.create_section_group('htmgroup', 'HTML_SECTION_GROUP');
ctx_ddl.add_zone_section('htmgroup', 'heading', 'H1');
end;
You can then index your documents as follows:
create index myindex on docs(htmlfile) indextype is ctxsys.context
parameters('filter ctxsys.null_filter section group htmgroup');
After indexing with section group htmgroup, you can query within the heading section by issuing a query as follows:
'Oracle WITHIN heading'
With HTML documents you can also create sections for NAME/CONTENT pairs in <META> tags. When you do so you can limit your searches to text within CONTENT.
<META>Tags Consider an HTML document that has a META tag as follows:
<META NAME="author" CONTENT="ken">
To create a zone section that indexes all CONTENT attributes for the META tag whose NAME value is author:
begin
ctx_ddl.create_section_group('htmgroup', 'HTML_SECTION_GROUP');
ctx_ddl.add_zone_section('htmgroup', 'author', 'meta@author');
end
After indexing with section group htmgroup, you can query the document as follows:
'ken WITHIN author'
Like HTML documents, XML documents have tagged text which you can use to define blocks of text for section searching. The contents of a section can be searched on with the WITHIN or INPATH operators.
For XML searching, you can do the following:
Automatic sectioning
Attribute searching
Document type sensitive sections
Path section searching
You can set up your indexing operation to automatically create sections from XML documents using the section group AUTO_SECTION_GROUP. The system creates zone sections for XML tags. Attribute sections are created for the tags that have attributes and these sections named in the form tag@attribute.
For example, the following command creates the index myindex on a column containing the XML files using the AUTO_SECTION_GROUP:
CREATE INDEX myindex
ON xmldocs(xmlfile)
INDEXTYPE IS ctxsys.context
PARAMETERS ('datastore ctxsys.default_datastore
filter ctxsys.null_filter
section group ctxsys.auto_section_group'
);
You can search XML attribute text in one of two ways:
Create attribute sections with CTX_DDL.ADD_ATTR_SECTION and then index with XML_SECTION_GROUP. If you use AUTO_SECTION_GROUP when you index, attribute sections are created automatically. You can query attribute sections with the WITHIN operator.
Index with the PATH_SECTION_GROUP and query attribute text with the INPATH operator.
Consider an XML file that defines the BOOK tag with a TITLE attribute as follows:
<BOOK TITLE="Tale of Two Cities"> It was the best of times. </BOOK>
To define the title attribute as an attribute section, create an XML_SECTION_GROUP and define the attribute section as follows:
begin
ctx_ddl.create_section_group('myxmlgroup', 'XML_SECTION_GROUP');
ctx_ddl.add_attr_section('myxmlgroup', 'booktitle', 'book@title');
end;
To index:
CREATE INDEX myindex
ON xmldocs(xmlfile)
INDEXTYPE IS ctxsys.context
PARAMETERS ('datastore ctxsys.default_datastore
filter ctxsys.null_filter
section group myxmlgroup'
);
You can query the XML attribute section booktitle as follows:
'Cities within booktitle'
You have an XML document set that contains the <book> tag declared for different document types. You want to create a distinct book section for each document type.
Assume that mydocname1 is declared as an XML document type (root element) as follows:
<!DOCTYPE mydocname1 ... [...
Within mydocname1, the element <book> is declared. For this tag, you can create a section named mybooksec1 that is sensitive to the tag's document type as follows:
begin
ctx_ddl.create_section_group('myxmlgroup', 'XML_SECTION_GROUP');
ctx_ddl.add_zone_section('myxmlgroup', 'mybooksec1', 'mydocname1(book)');
end;
Assume that mydocname2 is declared as another XML document type (root element) as follows:
<!DOCTYPE mydocname2 ... [...
Within mydocname2, the element <book> is declared. For this tag, you can create a section named mybooksec2 that is sensitive to the tag's document type as follows:
begin
ctx_ddl.create_section_group('myxmlgroup', 'XML_SECTION_GROUP');
ctx_ddl.add_zone_section('myxmlgroup', 'mybooksec2', 'mydocname2(book)');
end;
To query within the section mybooksec1, use WITHIN as follows:
'oracle within mybooksec1'
XML documents can have parent-child tag structures such as the following:
<A> <B> <C> dog </C> </B> </A>
In this example, tag C is a child of tag B which is a child of tag A.
With Oracle Text, you can do path searching with PATH_SECTION_GROUP. This section group enables you to specify direct parentage in queries, such as to find all documents that contain the term dog in element C which is a child of element B and so on.
With PATH_SECTION_GROUP, you can also perform attribute value searching and attribute equality testing.
The new operators associated with this feature are
INPATH
HASPATH
To enable path section searching, index your XML document set with PATH_SECTION_GROUP.
Create the preference:
begin
ctx_ddl.create_section_group('xmlpathgroup', 'PATH_SECTION_GROUP');
end;
Create the index:
CREATE INDEX myindex
ON xmldocs(xmlfile)
INDEXTYPE IS ctxsys.context
PARAMETERS ('datastore ctxsys.default_datastore
filter ctxsys.null_filter
section group xmlpathgroup'
);
When you create the index, you can use the INPATH and HASPATH operators.
To find all documents that contain the term dog in the top-level tag <A>:
dog INPATH (/A)
or
dog INPATH(A)
To find all documents that contain the term dog in the <A> tag at any level:
dog INPATH(//A)
This query finds the following documents:
<A>dog</A>
and
<C><B><A>dog</A></B></C>
To find all documents that contain the term dog in a B element that is a direct child of a top-level A element:
dog INPATH(A/B)
This query finds the following XML document:
<A><B>My dog is friendly.</B></A>
but does not find:
<C><B>My dog is friendly.</B></C>
You can test the value of tags. For example, the query:
dog INPATH(A[B="dog"])
Finds the following document:
<A><B>dog</B></A>
But does not find:
<A><B>My dog is friendly.</B></A>
You can search the content of attributes. For example, the query:
dog INPATH(//A/@B)
Finds the document
<C><A B="snoop dog"> </A> </C>
You can test the value of attributes. For example, the query
California INPATH (//A[@B = "home address"])
Finds the document:
<A B="home address">San Francisco, California, USA</A>
But does not find:
<A B="work address">San Francisco, California, USA</A>
You can test if a path exists with the HASPATH operator. For example, the query:
HASPATH(A/B/C)
finds and returns a score of 100 for the document
<A><B><C>dog</C></B></A>
without the query having to reference dog at all.
You can use the HASPATH operator to do section quality tests. For example, consider the following query:
dog INPATH A
finds
<A>dog</A>
but it also finds
<A>dog park</A>
To limit the query to the term dog and nothing else, you can use a section equality test with the HASPATH operator. For example,
HASPATH(A="dog")
finds and returns a score of 100 only for the first document, and not the second.
