View source for Implementation:Facebookresearch Habitat lab Rearrange V2 UI Overlay

{{PageInfo|type=Implementation|title=Facebookresearch_Habitat_lab_Rearrange_V2_UI_Overlay}}
{| class="wikitable" style="float:right; margin-left:1em; width:300px;"
|-
! Knowledge Sources
|
* [https://github.com/facebookresearch/habitat-lab Facebookresearch_Habitat_lab]
|-
! Domains
| [[domain::Embodied_AI]], [[domain::Human_in_the_Loop]], [[domain::User_Interface]]
|-
! Last Updated
| [[last_updated::2026-02-15 00:00 GMT]]
|}

== Overview ==

The '''UIOverlay''' class renders on-screen overlay panels for a specific user in the rearrange_v2 HITL application, including instruction text, control help, hovered object information, and selected object state controls.

=== Description ===

'''UIOverlay''' manages all 2D overlay GUI panels displayed on top of the 3D viewport for a specific user. It works through the '''AppService.ui_manager''' to create and update canvas-based UI elements. The class provides four main panel update methods:

* '''update_instructions_panel''': Displays task instructions and warning text in the top-left corner. Instructions are word-wrapped at 70 characters and warnings are shown in bold orange text.
* '''update_controls_panel''': Shows keyboard/mouse control mappings in the top-right corner (e.g., "WASD" -> "Move", "Double-Click" -> "Pick-up"). When help is hidden, shows only a minimal "H -> Show Help" hint.
* '''update_hovered_object_info_panel''': Shows the category name, room, and boolean object states for whatever object the cursor is hovering over, displayed at the bottom center.
* '''update_selected_object_panel''': Shows detailed information for the clicked/selected object in the bottom-left, including category name, room, contextual action hints (e.g., "Double-click to pick up"), and toggle controls for boolean object states.

The companion '''ObjectStateControl''' dataclass bundles all information needed to render and interact with a single object state toggle, including the state specification, current value, enabled/available flags, a callback for state changes, tooltip text, and a recently-changed indicator.

=== Usage ===

'''UIOverlay''' is instantiated by the '''UI''' class and is not typically used directly. The '''UI''' class calls the panel update methods during its '''draw_ui()''' phase and calls '''update()''' each frame to check for button press events on toggle controls.

== Code Reference ==

=== Source Location ===
* '''Repository:''' [https://github.com/facebookresearch/habitat-lab Facebookresearch_Habitat_lab]
* '''File:''' [https://github.com/facebookresearch/habitat-lab/blob/main/examples/hitl/rearrange_v2/ui_overlay.py examples/hitl/rearrange_v2/ui_overlay.py]
* '''Lines:''' 1-301

=== Signature ===
<syntaxhighlight lang="python">
@dataclass
class ObjectStateControl:
    spec: ObjectStateSpec
    value: bool
    enabled: bool
    available: bool
    callback: Optional[Callable[[str, str, Any], None]]
    tooltip: Optional[str]
    recently_changed: bool


class UIOverlay:
    def __init__(
        self,
        app_service: AppService,
        user_index: int,
    ):
</syntaxhighlight>

=== Import ===
<syntaxhighlight lang="python">
from ui_overlay import UIOverlay, ObjectStateControl
</syntaxhighlight>

== I/O Contract ==

=== Inputs ===
{| class="wikitable"
|-
! Name !! Type !! Required !! Description
|-
| app_service || AppService || Yes || Application service providing access to the UI manager
|-
| user_index || int || Yes || Index of the user this overlay belongs to
|}

=== Outputs ===
{| class="wikitable"
|-
! Name !! Type !! Description
|-
| (visual panels) || Canvas UI || Rendered overlay panels on the user's viewport (instructions, controls, object info, selection)
|}

== Usage Examples ==

=== Basic Usage ===
<syntaxhighlight lang="python">
from ui_overlay import UIOverlay, ObjectStateControl

# Create overlay for user 0
overlay = UIOverlay(app_service=app_service, user_index=0)

# Update instruction panel
overlay.update_instructions_panel(
    instructions="Pick up the red mug and place it on the kitchen counter.",
    warning_text="Time remaining: 30 seconds",
    is_help_shown=True,
)

# Update controls panel with keyboard mappings
controls = [
    ("H", "Hide Help"),
    ("WASD", "Move"),
    ("Double-Click", "Pick-up"),
    ("Right-Click", "Drop"),
]
overlay.update_controls_panel(controls)

# Update hovered object info
overlay.update_hovered_object_info_panel(
    object_category_name="mug",
    object_states=[("Is Clean", "True")],
    primary_region_name="kitchen",
)

# Process button presses each frame
overlay.update()

# Reset on episode change
overlay.reset()
</syntaxhighlight>

== Related Pages ==

* [[Facebookresearch_Habitat_lab_Rearrange_V2_UI]]
* [[Facebookresearch_Habitat_lab_Rearrange_V2_World]]

[[Category:Implementations]]