Archive for November, 2011

Relationship and Virtual Documents in Documentum

November 26, 2011 Comments off

This Post explains how the concept of relationships and virtual documents can be used to relate objects in Documentum using Documentum Foundation Classes (DFC).


  1. Relationship: A relationship implies a connection between two objects. When an object is related to another object then we can define which object is the parent object or the child object or if they are equal. Relationships are system-defined as well as user-defined. In the BOK, we confine our self to user-defined relationships.
  2. Virtual Document: In Documentum, a document which holds other documents i.e. a document which acts as a container is called as a virtual document. A virtual document can contain other virtual documents. The document which acts as a container is called the parent document while the documents which are contained in parent document are called as child documents.

Overview on relationship:

Two built-in object types dm_relation and dm_relation_type are to be used to create relations between any two objects. The dm_relation_type object defines the behavior of the relation. The dm_relation object identifies the dm_relation_type object and the two objects between which the relation needs to be created. Pictorially it can be shown as:


The dm_relation_type object has following attributes:

  1. child_parent_label: It defines the child to parent feature of the relation.
  2. parent_child_label: It defines the parent to child feature of the relation.
  3. description: It gives the general description of the relation.
  4. parent_type: It defines the type of objects that will act as parents.
  5. child_type: It defines the type of objects that will act as child.
  6. direction_kind: It defines the nature of relationships between the objects. The expected values are:

a)    1 – Parent to Child

b)    2 – Child to Parent

c)    3 – Objects are at equal level

7. integrity_kind: It specifies the type of referential integrity used when either of the two related objects has to be deleted. The expected values are:

a)    0 – Any of the two related objects can be deleted.

b)    1 – As long as the relation exists, neither of the related objects can be deleted.

c)    2 – If one of the related objects gets deleted, other one also gets deleted.

      1. relation_name: Specifies a name for the relation
      2. security_type: It indicates the type of security to be used for the relation object. The valid values are:

a)    SYSTEM: If this value is used, then super-user privileges are required for creating, deleting or modifying the relationships pertaining to this dm_relation_type object.

b)    PARENT: In this case, the ACL for the relation is inherited from the parent object in the relation and RELATE permission is required to create, modify, or drop the relation. The exception to this is if the parent object is not a subtype of dm_sysobject, then no security will be enforced.

c)    CHILD: In this case, the ACL for the relation is inherited from the child object in the relation and RELATE permission is required to create, modify, or drop the relation. The exception to this is if the child object is not a subtype of dm_sysobject, then no security will be enforced.

d)    NONE – In this case, no security is applied. All users can create, modify or delete this kind of relationship.

The dm_relation object has following attributes:

  1. child_id: It is the r_object_id or i_chronicle_id of the child object in this relation. If i_chronicle_id is used, then ‘child label’ attribute can be used to bind the parent object to a particular version of child.
  2. parent_id: The r_object_id or i_chronicle_id of the parent object in the relations. If the attribute ‘permanent_link’ is set to TRUE then only use the i_chronicle_id of the object.
  3. permanent_link: If every new version of the parent object has to be related with the child object, then the value for this attribute must be set to TRUE and i_chronicle_id should be used in the parent_id attribute. By default the value is FALSE.
  4. relation_name: It specifies the value of relation_name attribute of the dm_relation_type object that defines the type of relationship.
  5. child_label <Optional>: If i_chronicle_id is used in the attribute ‘child_id’, then the label of the version of the child object is to be specified here.
  6. description <Optional>: Specifies the description.
  7. effective_date<Optional>: Not used by the system, a user-defined date. Custom logic could check this date to determine the state of the relationship.
  8. expiration_date<Optional>: Not used by the system, a user-defined date. Custom logic could check this date to determine the state of the relationship.
  9. order_no<Optional>: Not used by the system. Custom logic could use this integer value to order a set of relationships.
  • Creation of dm_relation_type object using DQL: The query used to create a dm_relation_type object is as follows:

create dm_relation_type object

set child_parent_label = ‘<Child to parent label>’,

set parent_child_label = ‘<Parent to child label>’,

set description = ‘<Description>’,

set parent_type = ‘<Document type>’,

set child_type = ‘<Document type>’,

set direction_kind = <0 or 1 or 2>,

set integrity_kind = <0 or 1 or 2>,

set relation_name = ‘<Name of Relation>’,

set security_type = ‘<SYSTEM or PARENT or CHILD or NONE>’

  • Creation of dm_relation object using DFC: Following methods which returns an object of type dm_relation can be used to create a dm_relation object.

1. addChildRelative(relationTypeName, childId, childLabel, isPermanent, description): This method has to be invoked on the object which is going to act as Parent in the relation. The parameters it takes are

a)    relationTypeName – Name of a valid dm_relation_type object.

b)    childId – The r_object_id or i_chronicle_id of the child object of the relation.

c)    childLabel – Version label of the child object. If this is ‘null’, the relation will contain no child label.

d)    isPermanent – Specifies if the link permanent. Valid values are TRUE or FALSE

e)    description – Specifies the description for the relation object. If ‘null’, the relation object will not have a description.

2.addParentRelative(relationTypeName, parentId, childLabel, isPermanent, description): This method has to be invoked on the object which is going to act as Child in the relation. It takes the same parameters as the method addChildRelative takes except instead of r_object_id or i_chronicle_id of the child object we pass the r_object_id or i_chronicle_id of the parent object as the parameter parentId.

Note: dm_relation object can be created through DQL also.

Overview on virtual document:

Virtual document provides a way for combining documents in various formats into one consolidated document. For e.g. one word document, one pdf document and one image can be combined to form one virtual document. There is no limitation regarding nesting of documents. One particular version or else all of the versions of a component can be combined with a virtual document. Two object types are used to store information about virtual documents. They are:

  1. Containment Object Type: It stores the information that links a component to a virtual document. Every time a component is added to a virtual document, a containment object is created for that component. The attributes of this object type can be set by the methods AppendPart, InsertPart, UpdatePart.
  2. Assembly Object Type:  An assembly object provides a snapshot of a virtual document at a given instance.

Creation of a virtual document using DFC:

A document can be converted to a virtual document by invoking the method setIsVirtualDocument on it. This method sets the r_is_virtual_doc of the document.

Note: Virtual documents can also be created using clients such as Webtop, DA etc as well as through DQL.

Requirement in the project:

Consider there are three main documents A, B and C. Essentially, A, B and C represent different contracts. Consider another set of documents A1, A2, A3, B1, B2, B3, C1, C2 and C3. A1, A2 and A3 are directly related to A, B1, B2 and B2 are directly related to B, C1, C2 and C3 are directly related to C. Also, A is related to B (A is child, B is parent), B is related to C (B is child and C is parent), and C is related to A (C is child and A is parent). The documents being referred here are Documentum documents of a certain system-defined or user-defined document type.

As per the requirements:

  1. For every new version of the documents the existing relations should be valid.
  2. From the document A, we should be able to navigate to A1, A2 and A3 and also to the documents B and C. Similarly for B and C.
  3. Depending on a particular attribute of main documents (A, B and C), there should be dynamic creation or deletion of relationships between contracts.

Resolution of requirement # 1:

Issue encountered:

Documentation says that when a dm_relation object is created with attribute values permanent_link = TRUE and child_label = ‘Current’ then for every new version of parent or child a new instance of the dm_relation object is created and the relation is created between the latest versions of child object and parent object. But on implementation of the same, always the latest version of the parent object was related to child object between which the relation was initially created.

Issue resolution:

To maintain the relation across the current versions of document and for easy navigation from parent to child documents the concept of virtual documents in addition to that of relationship was used.

All the main documents A, B and C were converted to virtual documents. The child documents A1, A2 and A3 were added as children to the newly converted virtual document A. For this the following DFC methods were used in the same order as specified:

  1. asVirtualDocument(lateBindingValue, followRootAssembly): This method is invoked on a virtual document (in this case on A, B and C) and it returns the virtual document representation of the virtual document object on which it is invoked. The parameters it takes are2.   getRootNode(): This method is invoked on the virtual document representation of a virtual document. It returns the root node of the virtual document. The root node is essentially the virtual document which is at the topmost hierarchy of virtual document tree. (In our case A, B and C are root nodes)
    1. lateBindingValue – the version label of the virtual document. To meet our requirement the value should be “Current”.
    2. folowRootassembly: If the value is set to TRUE, the assembly specified by the root node will be used as the virtual document.
    3. getRootNode(): This method is invoked on the virtual document representation of a virtual document. It returns the root node of the virtual document. The root node is essentially the virtual document which is at the topmost hierarchy of virtual document tree. (In our case A, B and C are root nodes
  2. addNode(parentNode, insertAfterNode, objectChronId, binding, followAssembly, overrideLateBindingValue): This method is invoked on the root node of a virtual document. This method adds a new node to the virtual document on which it is invoked. The parameters it takes are:
  • parentNode: The root node.
  • insertAfterNode: A virtual document node that will immediately precede the new node in the virtual document’s hierarchy. If this parameter is null, the new node is placed as the first child of parentNode.
  • objectChronId: i_chronicle_id of the document which is to be added as child to the virtual document. (In or case i_chronicle_id of A1, A2, A3, B1, B2, B3, C1, C2 and C3).
  • binding: The version label of the version of the child document with which we want to bind the child document with the virtual document.
  • followAssembly: It is set to TRUE if the follow_assembly attribute has to be set to TRUE for the component.
  • overrideLateBindingValue: It is set to TRUE if the version label identified in binding is to be used to resolve late-bound descendents of this component.

So using the concept of virtual document always the current versions of document are present in the parent virtual document. Thus the current versions of A and A1, A2, A3 are always related. And since A has been converted to virtual document, we can navigate to A1, A2 and A3 by just clicking on A.

Resolution of requirement # 2:

Now using the concept of relationship, a relation was created between A and B, B and C, A and C each. Thus navigation across the different contracts A, B and C was possible. Pictorially this can be shown as:


In the above figure the arrow denotes the relationship between two documents.

Resolution of requirement #3:

Requirement #3 states that there should be dynamic creation or deletion of relations between main documents depending on a particular attribute, say, attr. The value of attr for, let’s say, document A determines to which document, A will act as child. If the value of attr for document A is changed so as to imply that A and B are no longer related, then the relation object existing between A and B should be destroyed. If the new value points toward new document D, then a relation has to be created between A and D.

So the change in value of that particular attribute needs to be intercepted. The interception can be done as follows:

  1. Write a TBO (Type based object) for the document type to which A belongs to.                

 Note: For details on TBO refer BusinessObjectsDevelopersGuide.pdf provided by Documentum.

  1. In the TBO, override the method setString(attribute name, value of attribute), if the attribute attr is single-valued attribute or appendString(attribute name, value of attribute), if it is multi-valued attribute. These two methods captures all the attributes and their values for a document type. setString captures the single-valued attributes while appendString captures the multi-valued attributes.
  2. In either of the method, compare the old value of attribute attr with the new one. If there’s any change destroy all the existing relations which involves the document A as child. This can be done using the DFC method removeParentRelative(relationTypeName,parentId,childLabel).  Invoke this method on A. The parameters it takes are:Now use the method addParentRelative as explained, to relate document D as parent to document A.
    1. relationTypeName – Name of the relation object.
    2. parentId – r_object_id of parent object.
    3. childLabel – version label of child object.
  3. Every time the value is changed for attribute attr and the document is saved, the corresponding TBO is invoked and the above mentioned methods will be executed. Thus dynamic creation or deletion of relations can be achieved.

Conclusion: Thus the concept of relationships and virtual documents can be used together to relate objects in Documentum using Documentum Foundation Classes (DFC).

Documentum Full Text Index Server

November 2, 2011 Comments off

For faster and better search functionalities EMC has developed a Full Text Index Server which is installed separately with the Content management software to provide an index based search capability. In version 5.2.5 SPx the full text search engine was using Verity which has been now changed to FAST (Fast Search & Transfer) in 5.3 SPx onwards and xPlore replaced further.

In Verity we have to explicitly define the attributes to be indexed, in the content server configuration, whereas one of the salient features of FAST is that by default, all the attributes are indexed along with the content of the document. Since, FAST is no longer tightly coupled with the installation of the content server; one has the option of not installing Index
 Server. If the Full Text Index server is not installed, simple search will perform a case sensitive database search against object_name, title and subject attributes of dm_sysobject and its subtypes.

This post describes the various components of Index Server and their operations.

1.   Software Components

Full-text indexing in a Documentum repository is controlled by three software components:

  • Content Server, which manages the objects in a repository, generates the events that trigger  full-text indexing operations, queries the full-text indexes, and returns query results to client applications.
  • The index agent, which exports documents from a repository and prepares them for indexing.
  • The index server, which is a third-party server product that creates and maintains the full-text index for a repository. The index server also receives full-text queries from Content Server and responds to those queries.

2.   Set Up Configuration

a)  Basic Set Up

The basic indexing model consists of a single index agent and index server supporting a single repository. The index agent and index server may be installed on the Content Server host or on a different host.

b)  Consolidated Set Up

In a consolidated deployment, a single index server provides search and indexing services to multiple repositories. The repositories may be in the same Content Server installation or on different hosts. However, all repositories must be of the same Content Server version.

3.   Index Server Processes

The index server consists of five groups of processes that have different functions.

a) Document processors

Document processors (also sometimes called procservers) extract indexable content from content files, convert DFTXML to FIXML (a format that is used directly by the indexer), and merge the indexable content with the metadata during the DFTXML conversion process. Document processors are the largest consumer of CPU power in the index server.

b) Indexer

The indexer creates the searchable full-text index from the intermediate FIXML format. It consists of two processes. The frtsobj process interfaces with the document processor and spawns different findex processes as necessary to build the index from FIXML.

c) Query and Results servers

The QR Server (Query and Results Server) is a permanently-running process that accepts queries from Content Server, passes queries to the fsearch processes, and merges the results when there are multiple fsearch processes running.

The index server can run in continuous mode or in a special mode called suspended mode. In suspended mode, FIXML is generated for any updates
to the index but not integrated into the index. When the index server is taken out of suspended mode, the index is updated. Running in suspended mode; speeds up the indexing process. Suspended mode should be used when the requirement is to index large volume of documents or to re-index an entire repository.

4.   Health check up for Index Server processes

Execute the following command through command prompt

nctrl sysstatus

This will list all the Index Server process with their status.

Another option is to use Index Server admin console through <a href=”http://localhost:/admin”>http://localhost:<portno.>/admin and navigate to “System Management” tab.

Navigate to “Matching Engines” tab for details on total no. of documents in all the filestores (if there are multiple repositories) and the no. of documents processed by Index Server. It also provides a link to Index Server log file.

5.   How to determine Index Server ports

Using the Index server base port we can determine the ports for various Index Server processes

Index Server admin console: Base Port + 3000

FAST Search console: Base Port + 2100

6.   Fulltext indexing queue messages

When a document has been marked and submitted for fulltext indexing, it is queued to the Index Agent/Index Server.

The fulltext index status can be checked by the following dql query:

select sent_by, date_sent, item_name, content_type, task_state, message from dmi_queue_item where item_id = ”

The task_state can have one of the following values:

‘’ – The item is available to be picked up by an Index Agent for indexing.

‘acquired’ – The item is being processed. If an Index Agent stops abruptly, a queue item can be left in this state until the Index Agent is restarted or an Administrator clears the queue item.

‘warning’ – The item was indexed with a warning. Often it indicates that the content of the object failed to index but the meta-data was successfully indexed.

The ‘message’ attribute and the Index Agent log will have further details.

‘failed’ – The item failed to index, please refer to the ‘message’ attribute and the Index Agent log for more information.

‘done’ – Successfully indexed the item

7.   Index Agent Modes

An index agent may run in one of three operational modes:


In normal mode, the index agent process index queue items and prepares the SysObjects associated with the queue items for indexing. When the index agent successfully submits the object for indexing, the index agent deletes the queue item from the repository. If the object is not submitted successfully, the queue item remains in the repository and the error or warning generated by the attempt to index the object is stored in the queue item.


In migration mode, the index agent processes all SysObjects in a repository sequentially in r_object_id order and prepares them for indexing. A special queue item, the high-water mark queue item, is used to mark the index agent’s progress in the repository.

An index agent in normal mode and an index agent in migration mode cannot simultaneously update the same index.


In file mode, a file is used to submit a list of objects IDs to the index agent when a new index is created and index verification determines which objects are missing from the index.

8.   Switching modes of Index Agent

At the time of Index Agent set up the wizard gives an option to start the Index Agent under “Normal” or “Migration” mode.

The following steps should be performed to change the Index Agent from one mode to another.

1. Login to Index Agent Admin console through http://localhost:<index agent port no.>/IndexAgent<no.>/login.jsp

e.g., http://localhost:9081/IndexAgent1/login.jsp

2. Stop the Index Agent

3. Now change the Index Agent mode from Normal to Migration or Migration to Normal as the case may be.

4. Click on OK

5. Start the Index Agent again.


i) While in Migration mode Index Agent doesn’t appear in DA under Indexing Management tab. On the Index Agent admin screen it will provide the details of the no. of documents processed out of the total no. of documents.

ii) If the Index Agent service is restarted from services console then start the Index Agent from Index Agent Admin console or through DA under Indexing Management tab.

9.   Re-configuring Index Agent and FAST

Configuring another IA and FAST to a repository previously configured to work with one IA and FAST doesn’t modify dm_ftengine_config object and IA fails to start displaying error to connect to old FAST machine.

To resolve:

Manually update the dm_ftengine_config object based on the settings from the new machine

1. Go to IAPI and execute the following API –

iapi> retrieve,c,dm_ftengine_config

2. Note the object_id retrieved by the above API and use it to execute the following API –

iapi> dump,c,l

3. In the dump results note the following param_name, param_value pairs

fds_base_port should match 13000 or the base port number for Index Server Install

fds_config_host should match the host name where the Index Server is installed.

and so on….

4. The param_name/param_value pairs should be changed to match the values for the index server install.

5. Delete the following via dql:

delete dm_ftengine_config object where r_object_id = ‘old_value’

delete dm_ftindex_agent_config object where r_object_id = ‘old_value’

6. Run the index agent configuration program to create new index agent.

10.   Relocating fulltext indexes in Index Server

The following steps describe how we can change the location of fulltext indexes

 1. Shutdown the Index Agent

 2. Shutdown the Index Server

 3. Copy the indexes to the target location ( both the fixml and the index directories)

 4. Default location of the fixml and index directories:

      for Windows – %DOCUMENTUM%/data/fulltext

      for Unix –$DOCUMENTUM/data/fulltext

 5. Edit the following:

 In Windows:  %DOCUMENTUM/%fulltext/IndexServer/etc/searchrc-1.xml. Change the “index path”

 In Unix: $DOCUMENTUM/fulltext/IndexServer/etc/searchrc-1.xml. Change the “index path”

6. Edit the following:

 In Windows: %DOCUMENTUM%/fulltext/IndexServer/etc/config_data/RTSearch/ webcluster/rtsearchrc.xml. Change fixmlpath and fsearchdatasetdir to the new path

 In Unix: $DOCUMENTUM/fulltext/IndexServer/etc/config_data/RTSearch/webcluster/rtsearchrc.xml. Change fixmlpath and fsearchdatasetdir to the new path

 7. Startup IndexServer, Index Agent

11.   dm_FTCreateEvents Job

The Create Full-Text Events tool (dm_FTCreateEvents) may be used in two ways:

 a) To complete an upgrade by causing any objects missed by the pre-upgrade indexing operations to be indexed.

 The job generates events for each index able object added to a repository between the time a new 5.3 or later full-text index is created for a 5.2.5 repository and when the repository is upgraded to 5.3.

 This is the out-of-the-box behavior of the job.

 b) To generate the events required to re-index an entire 5.3 SP1 or later repository.

 Re-indexing the repository does not require deleting the existing index.

Please refer to the screenshot for the configuration of dm_FTCreateEvents Job –

To generate the events required to re-index an entire 5.3 SPx or later repository the -full_reindex argument must be set to TRUE to generate the required events.

The first time the job runs in its default mode, the job determines the last object indexed by an index agent running in migration mode and the date on which that object was indexed. The job searches for objects modified after that date and before the job runs for the first time and generates events for those objects. On its subsequent iterations, the job searches for objects modified after the end of the last iteration and before the beginning of the current iteration.

Before the job is run in a 5.3 SP1 or later repository with argument –full_reindex set to TRUE, you must create a high-water-mark queue item (dmi_queue_item) manually using the API –



and specify the r_object_id of the queue item as the -high_water_mark_id argument of the dm_FTCreateEvents Job.

In case you get the following error message in the job’s report –

FTCreateEvents was aborted. Error happened while processing job. Error: No high water mark found for qualification:

Verify the -high_water_mark_id attribute and check whether the API was executed after installation or re-installation of the index server to get the required r_object_id argument.    

Disable the job if the application is not using Full Text Index Server.

The job can also be de-activated if the following events are registered for dm_fulltext_index_user’

  • dm_save
  • dm_destroy
  • dm_readonlysave
  • dm_checkin
  • dm_move_content

Execute the following query to verify the same:

select event from dmi_registry where user_name=’dm_fulltext_index_user’ 

12.   Using FT Integrity Tool

Modify the parameter file:

a) Login to the index server. Navigate to the parameter file location.
On Windows, Drive:\ProgramFiles\Documentum\IndexAgents\IndexAgentN\webapps\IndexAgentN

b) Open the ftintegrity.params.txt file in a text editor.

The first line is  -D repositoryname

where repositoryname is the repository for which you created a new index.

c) Add the following two lines immediately after the first line

-U username

-P password

where username is the user name of the Superuser whose account was used to install the index agent and password is the Superuser’s password.

 d) Save the  ftintegrity.params.txt file to %Documentum%\fulltext\IndexServer\bin  (Windows).

Sample FT Integrity params file

Note: The instruction above save a Superuser name and password to the file system in a plain text parameter file. For security reasons, you may wish to remove that information from the file after running the FTIntegrity tool. It is recommended that you save the parameter file in a location accessible only to the repository Superuser and installation owner.

 To run the index verification tool:

  1. Navigate to %Documentum%\fulltext\IndexServer\bin (Windows) or
  2. To verify both completeness and accuracy, open a command prompt and execute

           cobra -i ftintegrity.params.txt -m b

  1. To verify completenes only, open a command prompt and execute

           cobra -i ftintegrity.params.txt -m c

  1. To verify accuracy only and query all indexed objects, open a command prompt and execute

           cobra -i ftintegrity.params.txt -m a

FT Integrity generates 3 reports

res-comp-common.txt – object id of all documents that are found in both index and repository.

res-comp-dctmonly.txt – object id of documents that are in repository but not indexed

res-comp-fastonly.txt –object id of documents in index but not in repository.

It also generates ftintegrityoutput.txt file which is nothing but the console output generated in the text format.

 To resubmit objects that failed indexing:

a) Navigate to %DOCUMENTUM\fulltext\indexerver\bin.

b) Copy the res-comp-dctmonly.txt file to

drive:\Program Files\Documentum\IndexAgents\IndexAgentN\webapps\IndexAgentN\WEBINF\classes.

c) Rename the res-comp-dctmonly.txt file to ids.txt

The index agent periodically checks for the existence of ids.txt. If the file is found, the objects are resubmitted for indexing.

13.   Moving From Index Server 5.3 SPx to D6.5 SPx

 a) Index Server D6.5 can use indexes created by a 5.3 SPx Index Server. This reduces the overhead of indexing the entire repository during upgrade.

 While uninstalling Index Server 5.3 SPx leave the checkbox for deleting the Indexes unchecked. Then during D6.5 Index Server installation point it to the existing indices folder as data folder.

b) While upgrading from 5.3 SPx to D6.5 Index Server the data folder, if the indices from 5.3 are preserved, should be on the same drive as the Index Server home directory.

c) The path for attribute mapping xml is $Documentum\ fulltext\fast as against $Documentum\fulltext\fast40 folder in 5.3 SPx

 This path should be correct in dm_ftengine_config object for index based search to function properly.

d) Index Agent in D6.5 uses 20 consecutive ports as against 1 port in 5.3 SPx. i.e. If Index Agent 1 is running at port 9081 then Index Agent 2 cannot take any ports from 9081 to 9100.

e) D6.5 doesn’t provide an option to switch Index Agent between different modes using Index Agent admin console.

f) The params file for using FT Integrity tool is placed under $Documentum\jboss4.2.0\server\DctmServer_IndexAgentN\deploy\IndexAgentN.war

Accordingly the ids.txt file should be placed under $Documentum\jboss4.2.0\server\DctmServer_IndexAgentN\deploy\IndexAgentN.war\WEB-INF\classes

Hope this Post is useful for all those developers who needed more and step by step info on Index Servers.

%d bloggers like this: