Implementation:Facebookresearch Habitat lab Rearrange V2 UI

Knowledge Sources	Facebookresearch_Habitat_lab
Domains	Embodied_AI, Human_in_the_Loop, Interactive_Simulation
Last Updated	2026-02-15 00:00 GMT

Overview

The UI class provides the user interface for the rearrange_v2 human-in-the-loop application, handling object picking, placing, interaction with articulated objects, and rendering visual feedback such as highlights and overlays.

Description

The UI class manages all user interaction within the rearrange_v2 HITL (Human-in-the-Loop) application. Each user in a multi-user session gets their own UI instance. The class handles:

• Object selection via hover and click tracking using the Selection system.
• Pick and place mechanics: double-click to pick objects, right-click-drag to place them on valid receptacles with physics-aware validation (surface normal checks, reachability checks, receptacle matching).
• Articulated object interaction: opening and closing drawers and cabinets via double-click.
• Object state manipulation: toggling boolean object states (e.g., powered on/off, clean/dirty) through UI controls, delegated to an optional ObjectStateManipulator.
• Visual feedback: drawing AABB outlines, placement circles (green for valid, red for invalid), pickable object highlights, and selection outlines via GuiDrawer.
• Event system: fires events (on_pick, on_place, on_open, on_close, on_state_change) that external systems can subscribe to.

The companion UISettings dataclass configures whether the user can change object states and whether default receptacles are highlighted.

Usage

This class is instantiated once per user within the rearrange_v2 HITL application. It is used internally by the application's main loop, which calls update() each frame to process user input and draw_ui() to render visual feedback. reset() is called when the simulator resets to a new episode.

Code Reference

Source Location

• Repository: Facebookresearch_Habitat_lab
• File: examples/hitl/rearrange_v2/ui.py
• Lines: 1-1128

Signature

@dataclass
class UISettings:
    can_change_object_states: bool
    highlight_default_receptacles: bool


class UI:
    def __init__(
        self,
        hitl_config,
        user_index: int,
        app_service: AppService,
        world: World,
        gui_controller: GuiController,
        sim: RearrangeSim,
        gui_input: GuiInput,
        gui_drawer: GuiDrawer,
        camera_helper: CameraHelper,
        ui_settings: UISettings,
    ):

Import

from ui import UI, UISettings

I/O Contract

Inputs

Name	Type	Required	Description
hitl_config	DictConfig	Yes	HITL configuration containing can_grasp_place_threshold
user_index	int	Yes	Index of the user this UI instance belongs to
app_service	AppService	Yes	Application service providing access to UI manager and other services
world	World	Yes	Shared world state containing object caches and scene information
gui_controller	GuiController	Yes	Controller for the agent associated with this user
sim	RearrangeSim	Yes	The rearrange simulator instance
gui_input	GuiInput	Yes	Input handler for mouse and keyboard events
gui_drawer	GuiDrawer	Yes	Drawing utility for rendering circles, boxes, and outlines
camera_helper	CameraHelper	Yes	Camera utility for eye position, lookat, and forward vector
ui_settings	UISettings	Yes	Configuration for UI behavior (state changes, receptacle highlights)

Outputs

Name	Type	Description
on_pick	Event	Event fired when the user picks up an object (PickEventData)
on_place	Event	Event fired when the user places an object (PlaceEventData)
on_open	Event	Event fired when the user opens an articulated link (OpenEventData)
on_close	Event	Event fired when the user closes an articulated link (CloseEventData)
on_state_change	Event	Event fired when an object state is changed (StateChangeEventData)

Usage Examples

Basic Usage

from ui import UI, UISettings
from world import World

# Create UI settings
ui_settings = UISettings(
    can_change_object_states=True,
    highlight_default_receptacles=True,
)

# Initialize UI for user 0
ui = UI(
    hitl_config=hitl_config,
    user_index=0,
    app_service=app_service,
    world=world,
    gui_controller=gui_controller,
    sim=sim,
    gui_input=gui_input,
    gui_drawer=gui_drawer,
    camera_helper=camera_helper,
    ui_settings=ui_settings,
)

# Subscribe to pick events
def on_pick_handler(event_data):
    print(f"Picked object: {event_data.object_handle}")

ui.on_pick.register(on_pick_handler)

# In the main loop
ui.update()  # Process input
ui.draw_ui()  # Render visual feedback

# On episode reset
ui.reset()

Related Pages

• Facebookresearch_Habitat_lab_Rearrange_V2_UI_Overlay
• Facebookresearch_Habitat_lab_Rearrange_V2_World