Operation by Structural Analysis (Object Recognition)

You can analyze the internal structure of the operation target application and Web page to identify and operate screen items.

The following browsers are supported when operating Web pages.

  • Chromium version Microsoft Edge
    • Windows® 10（32bit / 64bit）
      ※verified version: 20H2, 21H1, 21H2
    • Windows® 11（64bit）
      ※verified version: 21H2

Attention

Depending on the application, particular screen items may not correspond to operations by structural analysis.

When executing an operation

  • Specified operation does not work and has no response
  • "Failed to get the value" or "Failed to set the value" error occurs

If the above behavior happens, please check the following points.

  • The screen item is correctly recognized in the "Screen item" field in the "Operation target"of the action
  • The window title at the time of execution matches the value of the "Window" item in the "Screen item" field within the "Operation target" of the action
  • When the operation target is the "Selective Input Field", there is an option completely matching the value specified in "Select the following value"
  • If the operation target is "List", there is an option that completely matches the value specified for "Set the following value"
  • There is a choice of a display name that partially or completely matches the name specified when the operation target is "Area"

If none of them works even if there are no problems, please use actions other than structural analysis (Mouse actions or Keyboard actions).

What is structural analysis (object recognition)

Structural analysis (object recognition) is a function that analyzes the screen structure of applications such as Word, Excel and Web sites, identifies items on the screen (objects such as text data, buttons, input boxes), and can directly manipulate the items and obtain information.
It is easier to set up operations such as acquiring data of input fields on Web sites and business applications and setting values for text boxes and dropdowns in Explorer and Outlook.

Note

The structural analysis of CELF RPA identifies the operation target window by application type and window title.

If the window title has changed since the action set was created,

If the same application as the window that has been set is open, the operation will be performed on the foremost window in that application.

If the same application as the window that has been set is not open, the operation target will not be found and an error occurs.

Note

If you are using the profile feature in Microsoft Edge, the screen item may not be found and may generate an error.

If you are running a robot to operate Edge with the profile feature, make sure you are using the same profile name as when your application was developed.

You can see which profile you are currently using on the Microsoft Edge settings screen.

../../_images/edge-profile.en.png

Screen item operation through structural analysis

See also

For examples of how to use the "Screen item operation through structural analysis" action, refer to "Registering data using structural analysis".

  1. Drag the "Screen item operation through structural analysis" action from the "Robot" tab and set it to the edit area.

  2. Click the "Specify the operation target" button.

    ../../_images/1.en13.png

    Hint

    For details about how to set the operation target, see "Operation target setting method of structural analysis action".

  3. Set the value to be entered according to the set screen item.

    See also

    Refer to "Screen item operation through structural analysis - Screen items and setting contents that can be set as operation targets" for the types of screen items that can be set as operation targets and the settings for each type.

  4. If you check the "Set the position of the screen item to the next cell" check box, information on the position and size of the specified screen item displayed at the time of execution will be set in the cell.

    ../../_images/2.en13.png

    Note

    Acquiring the position of the screen item may fail if processing involving a change in the internal structure (such as screen transition, title of window changes, appearance of page changes, etc.) is performed by operating screen items.

    See also

    The set position information can be used as operation target position information of another action.For details, refer to "Use the coordinates set in the cell".

Screen item operation through structural analysis - Screen items and setting contents that can be set as operation targets

In the "Screen item operation through structural analysis" action, the following types of screen items can be set as operation targets.

  • TextEdit

    ../../_images/ctrl-tf.en.png

    Enter the value specified in "Set the following value". The value will be overwritten.

  • Selective Input Field

    ../../_images/ctrl-ddl.en.png

    Select the item specified in "Select the following value". Item names that match exactly, including upper and lower case letters, will be selected.

    Tip

    If there is no exact match, the first partial match found will be selected.

  • List

    ../../_images/ctrl-list.en.png

    Select the item specified in "Set the following value". Item names that match exactly, including upper and lower case letters, will be selected.

    Tip

    If there is no exact match, the first partial match found will be selected.

    Note

    When the operation target is "List", only the list box can be selected. If the screen item is not a list box, such as when the layout of the web page is created in the list, you can not set the value.

  • SpinControl

    ../../_images/ctrl-spin.en.png

    Enter the value specified in "Set the following value". The value will be overwritten.

    Tip

    In SpinControl which does not support operation by structural analysis, the value may not be set even if no error occurs.

    Use different action such as Type key for these input fields.

  • Option (single selection)

    ../../_images/ctrl-rb.en.png

    Select the specified option.

  • Option (multiple selection)

    ../../_images/ctrl-chb.en.png

    Select or deselect the specified item.

  • Area

    ../../_images/ctrl-area-1.en.png

    Select and deselect all options within the specified range at once.

    ../../_images/ctrl-area-2.en.png

    Click the "+" button to add setting fields for single selection and multiple selection.

    ../../_images/ctrl-area-3.en.png

    The single selection and multiple selection setting fields within the range are displayed only when "Option (single selection)" and "Option (multiple selection)" exist within each range. If no option exists, the setting field is not displayed.

    Note

    Only "Option (single selection)" and "Option (multiple selection)" can be set with area designation. Please select other types of screen items individually.

  • Button

    ../../_images/ctrl-btn.en.png

    Click the specified button.

  • Hyperlink

    ../../_images/ctrl-link.en.png

    Click the specified hyperlink.

Get value from screen item

See also

For examples of concrete usage of the "Get value from screen item" action, refer "Copy values from system screen item using structural analysis" and "Copy values from the table on the screen using structural analysis".

  1. From the "Robot" tab, drag the "Get value from screen item" action and set it in the edit area.

  2. Click the "Specify the operation target" button.

    ../../_images/1.en14.png

    Hint

    For details about how to set the operation target, see "Operation target setting method of structural analysis action".

  3. Set the cell to set the acquired value.

    See also

    Refer to "Get value from screen item - Screen items and setting contents that can be set as operation targets" for the types of screen items that can be set as operation targets and the settings for each type.

  4. If you check the "Set the position of the screen item to the next cell" check box, information on the position and size of the specified screen item displayed at the time of execution will be set in the cell.

    ../../_images/2.en14.png

Get value from screen item - Screen items and setting contents that can be set as operation targets

In the "Get value from screen item" action, the following types of screen items can be set as operation targets.

  • Characters

    ../../_images/ctrl-label.en.png

    Acquire the displayed string.

  • TextEdit

    ../../_images/ctrl-tf.en1.png

    Acquire the entered string.

  • Selective Input Field

    ../../_images/ctrl-ddl.en1.png

    Acquire the displayed string of the selected item.

  • List

    ../../_images/ctrl-list.en1.png

    Acquire the displayed string of the selected item.

    Tip

    If the selected list cannot be manipulated as a list box, it will be manipulated as a "Area (List)". It is determined automatically if it can be manipulated as a list box.

  • SpinControl

    ../../_images/ctrl-spin.en1.png

    Acquire the displayed string.

  • Option (single selection)

    ../../_images/ctrl-rb.en1.png

    Set "true" in the cell if the specified choice is selected, and "false" if it is not selected.

  • Option (multiple selection)

    ../../_images/ctrl-chb.en1.png

    Set "true" in the cell if the specified choice is selected, and "false" if it is not selected.

  • Table

    ../../_images/ctrl-table.en.png

    Copy the value of the cell in the specified table. By checking the "Copy cell merging state" check button, you can merge the cells of the CELF sheet in the same way as the specified table.

  • Area (List)

    ../../_images/ctrl-area-list.en.png

    Copy the texts that are displayed in the specified area, just as if you selected a table.

    Tip

    If the selected list can be manipulated as a list box, it will be manipulated as a "List". It is determined automatically if it can be manipulated as a list box.

  • Area (other than List)

    ../../_images/ctrl-area.en.png

    Gets a list of the item names of the selected "Option (single selection)" and "Option (multiple selection)" within the specified area. The list is set downward at the beginning of the specified cell.

    Note

    Only the label of "Option (single selection)" and "Option (multiple selection)" can be acquired by area specification. Please select other types of screen items individually.

    Tip

    The list of selected choices is sorted in ascending order of UTF-8 character codes.

Operation target setting method of structural analysis action

In the operation target setting of the structural analysis action, you can select the operation target from the list of items displayed on the screen.

../../_images/1.en15.png

Set operation target

  1. Click the operation target specification button.

    ../../_images/5.en3.png

  2. Click the screen item you want to work with, or the area near where the screen item is located.

    ../../_images/6.en1.png

    Tip

    Screen items enclosed in a blue frame when the mouse cursor is hovered over them represent items that can be operated by RPA, while those that are enclosed in gray frames cannot be operated by RPA.
    Even if you click an item enclosed in a gray frame that cannot be operated by RPA, the setting screen will be displayed, and the screen items placed in it can be selected from the list.

  3. Select the item you want to operate from the list of screen items displayed on the Operation Target setting dialog.

    ../../_images/7.en1.png

    Hint

    If [+] is displayed to the left of the screen item, click to display a list of the screen items that are placed in it.

  4. Click the "Apply" button.

    ../../_images/8.en1.png

  5. The specified item is set as the operation target.

    ../../_images/9.en.png

Cancel operation target setting

To cancel the operation target setting, press the "Esc" key while selecting the screen items,
or click the "Cancel" button on the setting screen displayed after selecting the screen items.
If canceled, the operation target setting will not be changed.

Confirm the selected screen item

When you select an item in the operation target setting screen, the corresponding screen item is highlighted.

../../_images/2.en15.png

Or you can highlight the screen items by clicking on the Confirm button.

../../_images/10.en.png

Also, the type of the screen item is displayed in the upper right corner of the frame.

../../_images/4.en8.png

About items that can be set as operation targets and items that can not be set

Items that can be set as operation targets are displayed in black bold.

../../_images/3.en11.png

Items that can not be set as operation targets are displayed in gray text.

../../_images/11.en.png

The type of screen items that can be set as an operation target varies from action to action. For details, see the explanation of each action.

Note

The structural analysis of CELF RPA identifies the operation target window by application type and window title.

If the window title has changed since the action set was created,

If the same application as the window that has been set is open, the operation will be performed on the foremost window in that application.

If the same application as the window that has been set is not open, the operation target will not be found and an error occurs.

Note

If you are using the profile feature in Microsoft Edge, the screen item may not be found and may generate an error.

If you are running a robot to operate Edge with the profile feature, make sure you are using the same profile name as when your application was developed.

You can see which profile you are currently using on the Microsoft Edge settings screen.

../../_images/edge-profile.en.png