Principle:LaurentMazare Tch rs Tensor Indexing Operations

Knowledge Sources	LaurentMazare_Tch_rs NumPy Indexing PyTorch Indexing
Domains	Tensor Computing, Array Indexing
Last Updated	2026-02-08 00:00 GMT

Overview

Tensor indexing operations provide mechanisms to extract, slice, and reshape sub-regions of multi-dimensional arrays using integer indices, ranges, index tensors, and dimension insertion markers.

Description

Tensors are the fundamental data structure in numerical computing, and indexing is the primary way to access and manipulate their contents. A robust indexing system must support several distinct access patterns:

Select (integer indexing) - Choosing a single position along a dimension reduces that dimension entirely. For a 3D tensor of shape $[A, B, C]$ , selecting index $i$ along dimension 0 yields a 2D tensor of shape $[B, C]$ .

Narrow (range/slice indexing) - Extracting a contiguous sub-range along a dimension preserves that dimension but reduces its size. Selecting indices $[s, e)$ along a dimension of size $D$ yields a view with that dimension reduced to size $e - s$ .

Index select (tensor/gather indexing) - Using a tensor of indices to select non-contiguous elements along a dimension. The result replaces that dimension with the shape of the index tensor.

NewAxis (dimension insertion) - Inserting a new dimension of size 1 at a specified position without changing the underlying data. This is essential for broadcasting alignment.

Unlike full advanced/fancy indexing (where multiple dimensions can be indexed simultaneously with arbitrary index arrays), this model restricts indexing to one operation per dimension applied sequentially. This simplification avoids the complex broadcasting semantics and ambiguity of multi-dimensional advanced indexing.

Usage

Tensor indexing is used whenever:

Extracting batches - Selecting specific samples from a batch dimension.
Cropping spatial regions - Narrowing height/width dimensions for image processing.
Gathering embeddings - Using index tensors to look up rows from embedding matrices.
Reshaping for broadcasting - Inserting singleton dimensions so tensors align for element-wise operations.
Implementing attention mechanisms - Selecting and rearranging sequence elements.

Theoretical Basis

Indexing Algebra

A tensor $T$ of shape $[d_{0}, d_{1}, \dots, d_{n - 1}]$ can be indexed by a sequence of indexing specifications $I = [i_{0}, i_{1}, \dots]$ , where each $i_{k}$ is one of the following types:

Select

$Select (j)$ on dimension $k$ extracts the $j$ -th slice:

$T^{'} [\dots] = T [\dots, j, \dots]$

The output rank is reduced by 1. Negative indices count from the end: $j < 0 ⟹ j^{'} = d_{k} + j$ .

Narrow

$Narrow (s, e)$ on dimension $k$ extracts a contiguous range:

$T^{'} [\dots, m, \dots] = T [\dots, s + m, \dots] for m \in [0, e - s)$

The output shape replaces $d_{k}$ with $e - s$ . This operation typically returns a view (no data copy) because the slice is contiguous in memory along that dimension.

Index Select

$IndexSelect (𝐢 𝐝 𝐱)$ on dimension $k$ , where $𝐢 𝐝 𝐱$ is an integer tensor of shape $[m]$ :

$T^{'} [\dots, j, \dots] = T [\dots, 𝐢 𝐝 𝐱 [j], \dots] for j \in [0, m)$

The output shape replaces $d_{k}$ with $m$ . This generally requires a data copy since the selected elements may not be contiguous.

NewAxis

$NewAxis$ at position $p$ inserts a dimension of size 1:

$shape (T^{'}) = [d_{0}, \dots, d_{p - 1}, 1, d_{p}, \dots, d_{n - 1}]$

No data movement occurs; only the shape metadata changes.

Sequential Application

The indexing specifications are applied from left to right, each consuming one dimension from the current position (which advances through the tensor dimensions). NewAxis inserts a dimension without consuming one. The pseudo-code is:

current_dim = 0
FOR EACH spec IN index_specifications:
    IF spec IS Select(j):
        tensor = tensor.select(current_dim, j)
        // current_dim unchanged (dimension removed)
    ELSE IF spec IS Narrow(start, end):
        tensor = tensor.narrow(current_dim, start, end - start)
        current_dim += 1
    ELSE IF spec IS IndexSelect(indices):
        tensor = tensor.index_select(current_dim, indices)
        current_dim += 1
    ELSE IF spec IS NewAxis:
        tensor = tensor.unsqueeze(current_dim)
        current_dim += 1

Limitations Compared to Advanced Indexing

Full NumPy-style advanced indexing allows multiple index arrays across different dimensions simultaneously, with complex broadcasting rules to determine the output shape. The sequential model described here is simpler and avoids:

Ambiguity in output dimension ordering when mixing basic and advanced indices.
Implicit broadcasting between multiple index arrays.
The distinction between "basic" and "advanced" indexing triggering copies vs. views.

Related Pages

Implementation:LaurentMazare_Tch_rs_Tensor_Indexing

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment