Implementation:Facebookresearch Habitat lab Rearrange V2 UI Overlay
| Knowledge Sources | |
|---|---|
| Domains | Embodied_AI, Human_in_the_Loop, User_Interface |
| Last Updated | 2026-02-15 00:00 GMT |
Overview
The UIOverlay class renders on-screen overlay panels for a specific user in the rearrange_v2 HITL application, including instruction text, control help, hovered object information, and selected object state controls.
Description
UIOverlay manages all 2D overlay GUI panels displayed on top of the 3D viewport for a specific user. It works through the AppService.ui_manager to create and update canvas-based UI elements. The class provides four main panel update methods:
- update_instructions_panel: Displays task instructions and warning text in the top-left corner. Instructions are word-wrapped at 70 characters and warnings are shown in bold orange text.
- update_controls_panel: Shows keyboard/mouse control mappings in the top-right corner (e.g., "WASD" -> "Move", "Double-Click" -> "Pick-up"). When help is hidden, shows only a minimal "H -> Show Help" hint.
- update_hovered_object_info_panel: Shows the category name, room, and boolean object states for whatever object the cursor is hovering over, displayed at the bottom center.
- update_selected_object_panel: Shows detailed information for the clicked/selected object in the bottom-left, including category name, room, contextual action hints (e.g., "Double-click to pick up"), and toggle controls for boolean object states.
The companion ObjectStateControl dataclass bundles all information needed to render and interact with a single object state toggle, including the state specification, current value, enabled/available flags, a callback for state changes, tooltip text, and a recently-changed indicator.
Usage
UIOverlay is instantiated by the UI class and is not typically used directly. The UI class calls the panel update methods during its draw_ui() phase and calls update() each frame to check for button press events on toggle controls.
Code Reference
Source Location
- Repository: Facebookresearch_Habitat_lab
- File: examples/hitl/rearrange_v2/ui_overlay.py
- Lines: 1-301
Signature
@dataclass
class ObjectStateControl:
spec: ObjectStateSpec
value: bool
enabled: bool
available: bool
callback: Optional[Callable[[str, str, Any], None]]
tooltip: Optional[str]
recently_changed: bool
class UIOverlay:
def __init__(
self,
app_service: AppService,
user_index: int,
):
Import
from ui_overlay import UIOverlay, ObjectStateControl
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| app_service | AppService | Yes | Application service providing access to the UI manager |
| user_index | int | Yes | Index of the user this overlay belongs to |
Outputs
| Name | Type | Description |
|---|---|---|
| (visual panels) | Canvas UI | Rendered overlay panels on the user's viewport (instructions, controls, object info, selection) |
Usage Examples
Basic Usage
from ui_overlay import UIOverlay, ObjectStateControl
# Create overlay for user 0
overlay = UIOverlay(app_service=app_service, user_index=0)
# Update instruction panel
overlay.update_instructions_panel(
instructions="Pick up the red mug and place it on the kitchen counter.",
warning_text="Time remaining: 30 seconds",
is_help_shown=True,
)
# Update controls panel with keyboard mappings
controls = [
("H", "Hide Help"),
("WASD", "Move"),
("Double-Click", "Pick-up"),
("Right-Click", "Drop"),
]
overlay.update_controls_panel(controls)
# Update hovered object info
overlay.update_hovered_object_info_panel(
object_category_name="mug",
object_states=[("Is Clean", "True")],
primary_region_name="kitchen",
)
# Process button presses each frame
overlay.update()
# Reset on episode change
overlay.reset()