Skip to content

Node-layer values, and what scale does to them

A node-layer attribute is a value keyed by a pair — one node, one layer. There are two very different ways that pair comes to hold a number, and the difference is not a matter of degree.

A person typed it. Nine conditions, six timepoints, a few dozen nodes that matter, and almost every pair carrying nothing. A dict keyed by the pair is exactly right: what is absent costs nothing.

It arrived as a table. Twenty thousand entities across a thousand samples, and every pair carrying a number. That is 2 × 10⁷ cells. In a dict that is minutes to write and gigabytes to hold. As a dense float32 array it is eighty megabytes and a microsecond a read.

The difference is not the implementation. It is the shape. So AnnNet holds either, and the reader never learns which.

Attaching a table

G.layers.attach_values(
    {'expression': matrix},  # conditions x nodes
    layers=[(c,) for c in conditions],
    nodes=node_ids,
)

Attaching costs the two index maps and nothing else. The array is not copied, not converted, and not read until a cell is asked for — write into it afterwards and the graph sees the new value.

Two options are worth knowing:

  • columns= lets several nodes share one column, which is what a join of many nodes onto one measured entity needs — and it needs it without copying the column, which is the whole reason the array is attached rather than unpacked.
  • mask= is a boolean array gating which cells hold a value at all, so a condition that was never measured stays absent rather than becoming a zero.

G.layers.detach_values(backing) drops one. The dict store is never dropped.

Reading: one cell, or a rectangle

G.layers.values()  # the resolver
G.layers.matrix('expression')  # a rectangle
G.layers.node_frame(attrs=[...])  # a table

The resolver asks each backing in turn and a later one wins for a cell it can answer. Attaching a table therefore shadows whatever the dict store held for the same pair rather than blending with it — two sources for one cell is a conflict, and blending would hide it.

matrix is what to hand a method:

block = G.layers.matrix('expression', nodes=wanted)
block.values  # ndarray, layers x nodes
block.nodes  # the node of each column
block.layers  # the layer of each row

A frame of Python objects has to be unpacked before any arithmetic. This is the arithmetic's own shape, plus the two labels needed to put an answer back on the right rows.

Why it is fast, and when it is not

Where the values live in one attached array, matrix gathers them in a single pass in C. Where they live in the dict store, or span both, it falls back to reading cell by cell — and the two give the same numbers, which is pinned by test.

Measured on this machine:

cells matrix cell by cell
102,400 6.8 ms (0.07 µs/cell) 186 ms (1.82 µs/cell) 27×
400,000 10.0 ms (0.03 µs/cell) 679 ms (1.70 µs/cell) 68×

The advantage grows with size, which is the signature of removing a per-cell cost rather than making one cheaper.

The fallback is conservative: if any backing holding the name has no rectangle to give — the dict store never does — the whole read goes cell by cell, even where the fast answer would have been right. Telling those cases apart costs more than taking the slow path does, and the slow path is never wrong.

Identity is a separate question

Values moved into an array. Presence did not. Whether node n exists on layer c is a fact the structure store holds per pair, and that is what layers.place writes:

G.layers.place(node_ids, [(c,) for c in conditions], mask=measured)

place registers the whole rectangle in one call, going straight at the store rather than through the general node-adding path — which normalises each item, resolves a coordinate and merges default attributes, none of which a rectangle of bare ids needs.

node-layers before after
400,000 1.78 s 0.45 s
1,600,000 13.69 s 1.88 s

Placement is for the network's entities, not for the assay

Reading an attached array never consults presence. MatrixValues addresses cells through its own two index maps, so a cell answers whether or not its node-layer exists:

G.layers.attach_values({'expr': X}, layers=conditions, nodes=all_20k_genes)
# 3.3 ms, no measurable memory, 20 million cells readable
G.layers.matrix('expr', nodes=the_300_i_care_about)  # 12.6 ms

So place is not on the read path at all. It is needed for three things, and none of them scales with the size of the assay: a node-layer that is an edge endpoint, one that carries a dict-store attribute (which is how a method's results land), and structural queries like supra-adjacency.

All three are about entities the network holds. A 7,000-node prior across 16 samples is ~114,000 pairs — a tenth of a second and a few tens of megabytes. The measurement matrix stays where it was, in the AnnData, and is read through the index maps.

The one shape that does hit the wall is declaring cells as layers rather than samples: 20,000 nodes across 1,000 layers is several gigabytes of per-pair identity. Pseudobulk first; the number to watch is the layer count, not the cell count.

Where to go next