Gene identifier mapping tutorial

Convert the identifiers used by your source embedding into the unversioned Ensembl gene IDs required by genes.tsv.

Overview

Source identifiers + source matrix
↓
Map identifiers to the master table
↓
Classify as unique, unmapped, or ambiguous
↓
Exclude unmapped and ambiguous rows
↓
Average rows mapping to the same final gene
↓
Row-aligned embeddings.npz + genes.tsv

Important: the final genes.tsv row order must exactly match the row order of the matrix in embeddings.npz.

1. Identify the source identifier type

Source identifier	Master-table column
Ensembl gene ID	`map_ensembl_gene_ids_all`
Entrez Gene ID	`map_entrez_ids_all`
UniProt accession	`map_uniprot_ids_all`
RefSeq accession	`map_refseq_ids_all`
CCDS ID	`map_ccds_ids_all`
Ensembl transcript ID	`map_ensembl_transcript_ids_all`
Ensembl protein ID	`map_ensembl_protein_ids_all`
STRING protein ID	`map_string_protein_ids_all`

Multiple identifiers in one master-table cell are separated by |.

2. Gene-symbol mapping requires priority

Do not blindly use map_symbols_all. It combines current symbols, previous symbols, aliases, and names.

Try gene_symbol and hgnc_approved_symbol.
Then try hgnc_previous_symbols.
Finally try hgnc_alias_symbols.

Move to a lower-priority source only when the identifier was not matched at a higher-priority level.

3. Accept only unique mappings

Unique: exactly one Ensembl gene candidate. Retain the row.

Unmapped: no candidate. Exclude.

Ambiguous: more than one candidate. Exclude.

4. Basic lookup example

This example builds an Entrez-to-Ensembl lookup. Replace map_entrez_ids_all with the appropriate mapping column for your source.

from collections import defaultdict
import pandas as pd

master = pd.read_csv(
    "master_gene_table.csv",
    dtype=str,
)

lookup = defaultdict(set)

for row in master[
    ["ensembl_gene_id", "map_entrez_ids_all"]
].dropna().itertuples(index=False):
    for identifier in row.map_entrez_ids_all.split("|"):
        identifier = identifier.strip()

        if identifier:
            lookup[identifier].add(
                row.ensembl_gene_id
            )

candidates = lookup.get(source_identifier, set())

if len(candidates) == 1:
    status = "unique"
    ensembl_gene_id = next(iter(candidates))
elif len(candidates) == 0:
    status = "unmapped"
    ensembl_gene_id = None
else:
    status = "ambiguous"
    ensembl_gene_id = None

5. Collapse duplicate final mappings

Different source identifiers may map uniquely to the same final Ensembl gene. Average those source vectors after mapping so that the final package contains one row per gene.

final_vector = source_matrix[source_indices].mean(
    axis=0,
)

6. Preserve row alignment

Apply every mapping, filtering, and duplicate-collapsing decision to the source identifiers and matrix together. Never sort or filter genes.tsv independently.

assert final_matrix.shape[0] == len(final_gene_ids)
assert len(final_gene_ids) == len(set(final_gene_ids))

np.savez_compressed(
    "embeddings.npz",
    embeddings=final_matrix.astype("float32"),
)

pd.DataFrame(
    {"ensembl_gene_id": final_gene_ids}
).to_csv(
    "genes.tsv",
    sep="\t",
    index=False,
)

7. Document the procedure in submission.yaml

provenance:
  species: Homo sapiens
  identifier_before_mapping: Entrez Gene ID
  mapping_method: Source Entrez IDs were mapped through map_entrez_ids_all in the project master gene table. Only unique mappings were retained, unmapped and ambiguous identifiers were excluded, and duplicate source rows mapping to the same final Ensembl gene were averaged.