Curate & link scRNA-seq datasets

This illustrates how to manage scRNA-seq datasets in absence of a custom schema.

Show code cell content Hide code cell content

!lamin init --storage ./test-scrna --schema bionty
# avoids download bars
import bionty as bt

bt.Gene(species="human")
bt.Gene(species="mouse")
bt.Gene(species="saccharomyces cerevisiae")
bt.CellMarker(species="human")

💡 creating schemas: core==0.45.5 bionty==0.29.6 
✅ saved: User(id='DzTjkKse', handle='testuser1', email='testuser1@lamin.ai', name='Test User1', updated_at=2023-08-17 14:10:47)
✅ saved: Storage(id='u9weBVGy', root='/home/runner/work/lamin-usecases/lamin-usecases/docs/test-scrna', type='local', updated_at=2023-08-17 14:10:47, created_by_id='DzTjkKse')

✅ loaded instance: testuser1/test-scrna
💡 did not register local instance on hub (if you want, call `lamin register`)

CellMarker
Species: human
Source: cellmarker, 2.0
#terms: 15466

📖 CellMarker.df(): ontology reference table
🔎 CellMarker.lookup(): autocompletion of terms
🎯 CellMarker.search(): free text search of terms
✅ CellMarker.validate(): strictly validate values
🧐 CellMarker.inspect(): full inspection of values
👽 CellMarker.standardize(): convert to standardized names
🪜 CellMarker.diff(): difference between two versions
🔗 CellMarker.ontology: Pronto.Ontology object

import lamindb as ln
import lnschema_bionty as lb

# don't recurse through ontology hierarchies to speed up CI
# recommend to set to True
lb.settings.auto_save_parents = False

✅ loaded instance: testuser1/test-scrna (lamindb 0.50.7)

ln.track()

✅ saved: Transform(id='Nv48yAceNSh8z8', name='Curate & link scRNA-seq datasets', short_name='scrna', stem_id='Nv48yAceNSh8', version='0', type=notebook, updated_at=2023-08-17 14:10:54, created_by_id='DzTjkKse')

✅ saved: Run(id='MtzJ6VmKpLQiUtbzoMTo', run_at=2023-08-17 14:10:54, transform_id='Nv48yAceNSh8z8', created_by_id='DzTjkKse')

Preparation: registries

Let’s assume that this is not the first time we work with experimental entities, and hence, our registries are already pre-populated:

Show code cell content Hide code cell content

# assume prepared registries

# strain
lb.ExperimentalFactor.from_bionty(ontology_id="EFO:0004472").save()
record = lb.ExperimentalFactor.filter(ontology_id="EFO:0004472").one()
record.add_synonym("C57BL/6N")

# developmental stage
lb.ExperimentalFactor.from_bionty(ontology_id="EFO:0001272").save()

# tissue
lb.Tissue.from_bionty(ontology_id="UBERON:0001542").save()

# cell types
ln.save(lb.CellType.from_values(["CL:0000115", "CL:0000738"], "ontology_id"))

✅ validated 1 ExperimentalFactor record from Bionty on ontology_id: EFO:0004472

✅ validated 1 ExperimentalFactor record from Bionty on ontology_id: EFO:0001272

✅ validated 1 Tissue record from Bionty on ontology_id: UBERON:0001542

✅ validated 2 CellType records from Bionty on ontology_id: CL:0000115, CL:0000738

ln.view(schema="bionty", registries=["CellType", "ExperimentalFactor", "Tissue"])

CellType

	name	ontology_id	abbr	synonyms	description	bionty_source_id	updated_at	created_by_id
id
4fOuOYtl	endothelial cell	CL:0000115	None	endotheliocyte	An Endothelial Cell Comprises The Outermost La...	G9wU	2023-08-17 14:10:57	DzTjkKse
MkrH0gsX	leukocyte	CL:0000738	None	white blood cell|leucocyte	An Achromatic Cell Of The Myeloid Or Lymphoid ...	G9wU	2023-08-17 14:10:57	DzTjkKse

ExperimentalFactor

	name	ontology_id	abbr	synonyms	description	molecule	instrument	measurement	bionty_source_id	updated_at	created_by_id
id
EfXUa6F0	adult	EFO:0001272	None	adult stage	A Maturity Quality Inhering In An Individual B...	None	None	None	5BLC	2023-08-17 14:10:55	DzTjkKse
eXg039cd	obsolete_C57BL/6	EFO:0004472	None	CB57|C57Bl\6|C57Black|C57BL/6N|C57/BL6|C57Bl,|...	C57Bl/6 Is A Mouse Strain As Described In Jack...	None	None	None	5BLC	2023-08-17 14:10:55	DzTjkKse

Tissue

	name	ontology_id	abbr	synonyms	description	bionty_source_id	updated_at	created_by_id
id
Na0qNZ1o	inguinal lymph node	UBERON:0001542	None	None	The Lymph Nodes Located In The Groin Area.	V5FW	2023-08-17 14:10:56	DzTjkKse

Mouse lymph node cells: Detmar22

We’re working with mouse data:

lb.settings.species = "mouse"

Show code cell output Hide code cell output

✅ set species: Species(id='vado', name='mouse', taxon_id=10090, scientific_name='mus_musculus', updated_at=2023-08-17 14:10:59, bionty_source_id='Rcmh', created_by_id='DzTjkKse')

Let’s look at a scRNA-seq count matrix in form of an AnnData object:

Show code cell source Hide code cell source

adata = ln.dev.datasets.anndata_mouse_sc_lymph_node()
# The column names are a bit lengthy, let's abbreviate them:
adata.obs.columns = (
    adata.obs.columns.str.replace("Sample Characteristic", "")
    .str.replace("Factor Value ", "Factor Value:", regex=True)
    .str.replace("Factor Value\[", "Factor Value:", regex=True)
    .str.replace(" Ontology Term\[", "ontology_id:", regex=True)
    .str.strip("[]")
    .str.replace("organism part", "tissue")
    .str.replace("organism", "species")
    .str.replace("developmental stage", "developmental_stage")
    .str.replace("cell type", "cell_type")
    # the last one could be interesting, too
    # .str.replace("Factor Value:Ontology Term[inferred cell_type - authors labels", "cell_type_authors")
)

adata

AnnData object with n_obs × n_vars = 1135 × 10000
    obs: 'species', 'ontology_id:species', 'strain', 'ontology_id:strain', 'age', 'ontology_id:age', 'developmental_stage', 'ontology_id:developmental_stage', 'sex', 'ontology_id:sex', 'genotype', 'ontology_id:genotype', 'tissue', 'ontology_id:tissue', 'cell_type', 'ontology_id:cell_type', 'immunophenotype', 'ontology_id:immunophenotype', 'post analysis well quality', 'ontology_id:post analysis well quality', 'Factor Value:single cell identifier', 'Factor Value:Ontology Term[single cell identifier', 'Factor Value:cluster', 'Factor Value:Ontology Term[cluster', 'Factor Value:inferred cell_type - authors labels', 'Factor Value:Ontology Term[inferred cell_type - authors labels'

When we create a File object from an AnnData, we’ll automatically link its feature sets and get information about unmapped categories:

file = ln.File.from_anndata(
    adata, description="Detmar22", var_ref=lb.Gene.ensembl_gene_id
)

💡 file will be copied to default storage upon `save()` with key `None` ('.lamindb/hikgGNVqGiX2BWDwRAYR.h5ad')

💡 parsing feature names of X stored in slot 'var'

💡    using global setting species = mouse

✅    validated 9976 Gene records from Bionty on ensembl_gene_id: ENSMUSG00000104923, ENSMUSG00000079038, ENSMUSG00000069755, ENSMUSG00000040648, ENSMUSG00000113486, ENSMUSG00000034854, ENSMUSG00000053153, ENSMUSG00000020074, ENSMUSG00000022090, ENSMUSG00000111483, ENSMUSG00000083382, ENSMUSG00000022814, ENSMUSG00000030589, ENSMUSG00000016386, ENSMUSG00000022906, ENSMUSG00000072214, ENSMUSG00000019773, ENSMUSG00000109378, ENSMUSG00000039233, ENSMUSG00000015093, ...

❗    did not validate 24 Gene records for ensembl_gene_ids: ENSMUSG00000022591, ENSMUSG00000066378, ENSMUSG00000073682, ENSMUSG00000074210, ENSMUSG00000074735, ENSMUSG00000075015, ENSMUSG00000090625, ENSMUSG00000091604, ENSMUSG00000092345, ENSMUSG00000094030, ENSMUSG00000094296, ENSMUSG00000094958, ENSMUSG00000095547, ENSMUSG00000096201, ENSMUSG00000096385, ENSMUSG00000096850, ENSMUSG00000096923, ENSMUSG00000097078, ENSMUSG00000097854, ENSMUSG00000105204, ...

❗    ignoring non-validated features: ENSMUSG00000022591,ENSMUSG00000066378,ENSMUSG00000073682,ENSMUSG00000074210,ENSMUSG00000074735,ENSMUSG00000075015,ENSMUSG00000090625,ENSMUSG00000091604,ENSMUSG00000092345,ENSMUSG00000094030,ENSMUSG00000094296,ENSMUSG00000094958,ENSMUSG00000095547,ENSMUSG00000096201,ENSMUSG00000096385,ENSMUSG00000096850,ENSMUSG00000096923,ENSMUSG00000097078,ENSMUSG00000097854,ENSMUSG00000105204,ENSMUSG00000114046,ENSMUSG00000116184,ENSMUSG00000116275,ENSMUSG00000117732

✅    linked: FeatureSet(id='ARKhQll10Xo30R5y9EwQ', n=9976, type='float', registry='bionty.Gene', hash='wKO5cuNlOfRjfi3MkjUE', created_by_id='DzTjkKse')

💡 parsing feature names of slot 'obs'

❗    did not validate 26 Feature records for names: species, ontology_id:species, strain, ontology_id:strain, age, ontology_id:age, developmental_stage, ontology_id:developmental_stage, sex, ontology_id:sex, genotype, ontology_id:genotype, tissue, ontology_id:tissue, cell_type, ontology_id:cell_type, immunophenotype, ontology_id:immunophenotype, post analysis well quality, ontology_id:post analysis well quality, ...

❗    ignoring non-validated features: species,ontology_id:species,strain,ontology_id:strain,age,ontology_id:age,developmental_stage,ontology_id:developmental_stage,sex,ontology_id:sex,genotype,ontology_id:genotype,tissue,ontology_id:tissue,cell_type,ontology_id:cell_type,immunophenotype,ontology_id:immunophenotype,post analysis well quality,ontology_id:post analysis well quality,Factor Value:single cell identifier,Factor Value:Ontology Term[single cell identifier,Factor Value:cluster,Factor Value:Ontology Term[cluster,Factor Value:inferred cell_type - authors labels,Factor Value:Ontology Term[inferred cell_type - authors labels

❗    no validated features, skip creating feature set

We’re seeing that several gene identifiers can’t be validated through Bionty. We’d like to validate all features in this dataset, hence, let’s register them:

ln.save(lb.Gene.from_values(adata.var.index, lb.Gene.ensembl_gene_id))

💡 using global setting species = mouse

✅ validated 9976 Gene records from Bionty on ensembl_gene_id: ENSMUSG00000104923, ENSMUSG00000079038, ENSMUSG00000069755, ENSMUSG00000040648, ENSMUSG00000113486, ENSMUSG00000034854, ENSMUSG00000053153, ENSMUSG00000020074, ENSMUSG00000022090, ENSMUSG00000111483, ENSMUSG00000083382, ENSMUSG00000022814, ENSMUSG00000030589, ENSMUSG00000016386, ENSMUSG00000022906, ENSMUSG00000072214, ENSMUSG00000019773, ENSMUSG00000109378, ENSMUSG00000039233, ENSMUSG00000015093, ...

❗ did not validate 24 Gene records for ensembl_gene_ids: ENSMUSG00000022591, ENSMUSG00000066378, ENSMUSG00000073682, ENSMUSG00000074210, ENSMUSG00000074735, ENSMUSG00000075015, ENSMUSG00000090625, ENSMUSG00000091604, ENSMUSG00000092345, ENSMUSG00000094030, ENSMUSG00000094296, ENSMUSG00000094958, ENSMUSG00000095547, ENSMUSG00000096201, ENSMUSG00000096385, ENSMUSG00000096850, ENSMUSG00000096923, ENSMUSG00000097078, ENSMUSG00000097854, ENSMUSG00000105204, ...

Similarly, for the metadata, we’d like to validate the names:

columns = [
    col
    for col in adata.obs.columns
    if not col.startswith("ontology_id") and not col.startswith("Factor Value")
]
ln.save(ln.Feature.from_df(adata.obs[columns]))

❗ did not validate 10 Feature records for names: species, strain, age, developmental_stage, sex, genotype, tissue, cell_type, immunophenotype, post analysis well quality

file = ln.File.from_anndata(
    adata, description="Detmar22", var_ref=lb.Gene.ensembl_gene_id
)

💡 file will be copied to default storage upon `save()` with key `None` ('.lamindb/goAhrxKrgthN83xZwUcA.h5ad')

💡 parsing feature names of X stored in slot 'var'

💡    using global setting species = mouse

✅    validated 10000 Gene records on ensembl_gene_id: ENSMUSG00000024165, ENSMUSG00000074652, ENSMUSG00000079906, ENSMUSG00000022324, ENSMUSG00000093687, ENSMUSG00000002797, ENSMUSG00000020720, ENSMUSG00000041390, ENSMUSG00000058325, ENSMUSG00000084349, ENSMUSG00000000384, ENSMUSG00000105032, ENSMUSG00000083306, ENSMUSG00000111596, ENSMUSG00000111485, ENSMUSG00000055571, ENSMUSG00000033427, ENSMUSG00000037364, ENSMUSG00000022812, ENSMUSG00000041920, ...

✅    linked: FeatureSet(id='PHl0waVKbIbWfP0uwlvi', n=10000, type='float', registry='bionty.Gene', hash='T5FFYU5aZpuTyA3bXPfI', created_by_id='DzTjkKse')

💡 parsing feature names of slot 'obs'

✅    validated 10 Feature records on name: age, cell_type, developmental_stage, genotype, immunophenotype, post analysis well quality, sex, species, strain, tissue

❗    did not validate 16 Feature records for names: Factor Value:Ontology Term[cluster, Factor Value:Ontology Term[inferred cell_type - authors labels, Factor Value:Ontology Term[single cell identifier, Factor Value:cluster, Factor Value:inferred cell_type - authors labels, Factor Value:single cell identifier, ontology_id:age, ontology_id:cell_type, ontology_id:developmental_stage, ontology_id:genotype, ontology_id:immunophenotype, ontology_id:post analysis well quality, ontology_id:sex, ontology_id:species, ontology_id:strain, ontology_id:tissue

❗    ignoring non-validated features: Factor Value:Ontology Term[cluster,Factor Value:Ontology Term[inferred cell_type - authors labels,Factor Value:Ontology Term[single cell identifier,Factor Value:cluster,Factor Value:inferred cell_type - authors labels,Factor Value:single cell identifier,ontology_id:age,ontology_id:cell_type,ontology_id:developmental_stage,ontology_id:genotype,ontology_id:immunophenotype,ontology_id:post analysis well quality,ontology_id:sex,ontology_id:species,ontology_id:strain,ontology_id:tissue

✅    linked: FeatureSet(id='TwymXF5IRReTei2lBrgf', n=10, registry='core.Feature', hash='El46rPK3l2hUpJiADNs3', created_by_id='DzTjkKse')

file.save()

✅ saved 2 feature sets for slots: ['var', 'obs']

✅ storing file 'goAhrxKrgthN83xZwUcA' at '.lamindb/goAhrxKrgthN83xZwUcA.h5ad'

The file now has two linked feature sets:

file.features

'var': FeatureSet(id='PHl0waVKbIbWfP0uwlvi', n=10000, type='float', registry='bionty.Gene', hash='T5FFYU5aZpuTyA3bXPfI', updated_at=2023-08-17 14:11:10, created_by_id='DzTjkKse')
'obs': FeatureSet(id='TwymXF5IRReTei2lBrgf', n=10, registry='core.Feature', hash='El46rPK3l2hUpJiADNs3', updated_at=2023-08-17 14:11:12, created_by_id='DzTjkKse')

Some of the metadata can be typed using dedicated registries:

species = lb.Species.from_bionty(name="mouse")
strains = lb.ExperimentalFactor.from_values(adata.obs["strain"], "name")
dev_stages = lb.ExperimentalFactor.from_values(adata.obs["developmental_stage"], "name")
cell_types = lb.CellType.from_values(adata.obs["cell_type"], "name")
tissues = lb.Tissue.from_values(adata.obs["tissue"], "name")

✅ validated 1 Species record on name: mouse

❗ did not validate 1 ExperimentalFactor record for name: C57BL/6N

✅ validated 1 ExperimentalFactor record on name: adult

✅ validated 1 CellType record on name: endothelial cell

✅ validated 1 Tissue record on name: inguinal lymph node

We did not validate strains, hence, let’s try to map synonyms:

lb.ExperimentalFactor.standardize(adata.obs["strain"], return_mapper=True)

{'C57BL/6N': 'obsolete_C57BL/6'}

Indeed, there is a synonym:

adata.obs["strain"] = adata.obs["strain"].map(
    lb.ExperimentalFactor.standardize(adata.obs["strain"], return_mapper=True)
)

Now we can validate:

strains = lb.ExperimentalFactor.from_values(adata.obs["strain"], "name")

✅ validated 1 ExperimentalFactor record on name: obsolete_C57BL/6

file.add_labels(species, feature="species")
file.add_labels(strains + dev_stages + tissues + cell_types)

✅ linked feature 'species' to registry 'bionty.Species'

✅ linked labels 'obsolete_C57BL/6' to feature 'strain', linked feature 'strain' to registry 'bionty.ExperimentalFactor'

✅ linked labels 'adult' to feature 'developmental_stage', linked feature 'developmental_stage' to registry 'bionty.ExperimentalFactor'

✅ linked labels 'inguinal lymph node' to feature 'tissue', linked feature 'tissue' to registry 'bionty.Tissue'

✅ linked labels 'endothelial cell' to feature 'cell_type', linked feature 'cell_type' to registry 'bionty.CellType'

Metadata that doesn’t have can’t be typed with dedicated registries:

labels = ln.Label.from_values(adata.obs["sex"])
labels += ln.Label.from_values(adata.obs["age"])
labels += ln.Label.from_values(adata.obs["genotype"])
labels += ln.Label.from_values(adata.obs["immunophenotype"])

Show code cell output Hide code cell output

❗ did not validate 1 Label record for name: female

❗ did not validate 1 Label record for name: 8 to 10 week

❗ did not validate 1 Label record for name: wild type genotype

❗ did not validate 2 Label records for names: CD45 negative, CD45 positive

ln.save(labels)

file.add_labels(labels)

✅ linked labels 'female' to feature 'sex', linked feature 'sex' to registry 'core.Label'

✅ linked labels '8 to 10 week' to feature 'age', linked feature 'age' to registry 'core.Label'

✅ linked labels 'wild type genotype' to feature 'genotype', linked feature 'genotype' to registry 'core.Label'

✅ linked labels 'CD45 negative', 'CD45 positive' to feature 'immunophenotype', linked feature 'immunophenotype' to registry 'core.Label'

The file is now queryable by everything we linked:

file.describe()

💡 File(id=goAhrxKrgthN83xZwUcA, key=None, suffix=.h5ad, accessor=AnnData, description=Detmar22, version=None, size=17342743, hash=rk5lSoJvz6PHRRjmcB919w, hash_type=md5, created_at=2023-08-17 14:11:12.071980+00:00, updated_at=2023-08-17 14:11:12.072008+00:00)

Provenance:
    🗃️ storage: Storage(id='u9weBVGy', root='/home/runner/work/lamin-usecases/lamin-usecases/docs/test-scrna', type='local', updated_at=2023-08-17 14:10:47, created_by_id='DzTjkKse')
    📔 transform: Transform(id='Nv48yAceNSh8z8', name='Curate & link scRNA-seq datasets', short_name='scrna', stem_id='Nv48yAceNSh8', version='0', type='notebook', updated_at=2023-08-17 14:11:10, created_by_id='DzTjkKse')
    👣 run: Run(id='MtzJ6VmKpLQiUtbzoMTo', run_at=2023-08-17 14:10:54, transform_id='Nv48yAceNSh8z8', created_by_id='DzTjkKse')
    👤 created_by: User(id='DzTjkKse', handle='testuser1', email='testuser1@lamin.ai', name='Test User1', updated_at=2023-08-17 14:10:47)
Features:
  var (X):
    🔗 index (10000, bionty.Gene.id): ['pbtrQy8Wh7fR', '8Nt6KtET30W2', 'qamJEoIi5r0i', 'Ps0CKjW1bNdK', '4JoS07bhZMDw'...]
  obs (metadata):
    🔗 cell_type (1, bionty.CellType): ['endothelial cell']
    🔗 developmental_stage (2, bionty.ExperimentalFactor): ['obsolete_C57BL/6', 'adult']
    🔗 strain (2, bionty.ExperimentalFactor): ['obsolete_C57BL/6', 'adult']
    🔗 species (1, bionty.Species): ['mouse']
    🔗 tissue (1, bionty.Tissue): ['inguinal lymph node']
    🔗 genotype (1, core.Label): ['wild type genotype']
    🔗 sex (1, core.Label): ['female']
    🔗 age (1, core.Label): ['8 to 10 week']
    🔗 immunophenotype (2, core.Label): ['CD45 negative', 'CD45 positive']

Human immune cells: Conde22

lb.settings.species = "human"

Show code cell output Hide code cell output

✅ set species: Species(id='uHJU', name='human', taxon_id=9606, scientific_name='homo_sapiens', updated_at=2023-08-17 14:11:13, bionty_source_id='Rcmh', created_by_id='DzTjkKse')

conde22 = ln.dev.datasets.anndata_human_immune_cells()

ln.save(lb.Gene.from_values(conde22.var.index, lb.Gene.ensembl_gene_id))

💡 using global setting species = human

✅ validated 36390 Gene records from Bionty on ensembl_gene_id: ENSG00000243485, ENSG00000237613, ENSG00000186092, ENSG00000238009, ENSG00000239945, ENSG00000239906, ENSG00000241860, ENSG00000241599, ENSG00000286448, ENSG00000236601, ENSG00000284733, ENSG00000235146, ENSG00000284662, ENSG00000229905, ENSG00000237491, ENSG00000177757, ENSG00000228794, ENSG00000225880, ENSG00000230368, ENSG00000272438, ...

❗ did not validate 113 Gene records for ensembl_gene_ids: ENSG00000112096, ENSG00000182230, ENSG00000203812, ENSG00000204092, ENSG00000215271, ENSG00000221995, ENSG00000224739, ENSG00000224745, ENSG00000225932, ENSG00000226377, ENSG00000226380, ENSG00000226403, ENSG00000227021, ENSG00000227220, ENSG00000227902, ENSG00000228139, ENSG00000228906, ENSG00000229352, ENSG00000231575, ENSG00000232196, ...

conde22.obs.columns = conde22.obs.columns.str.replace("donor_id", "donor")
columns = [col for col in conde22.obs.columns if "ontology_term" not in col]
ln.save(ln.Feature.from_df(conde22.obs[columns]))

✅ validated 2 Feature records on name: cell_type, tissue

❗ did not validate 2 Feature records for names: assay, donor

file = ln.File.from_anndata(
    conde22, description="Conde22", var_ref=lb.Gene.ensembl_gene_id
)

💡 file will be copied to default storage upon `save()` with key `None` ('.lamindb/AXO3ps8RX50KjYoWYb58.h5ad')

💡 parsing feature names of X stored in slot 'var'

💡    using global setting species = human

✅    validated 36503 Gene records on ensembl_gene_id: ENSG00000230268, ENSG00000287719, ENSG00000180953, ENSG00000104549, ENSG00000140367, ENSG00000109220, ENSG00000144935, ENSG00000234127, ENSG00000105695, ENSG00000224712, ENSG00000163453, ENSG00000233408, ENSG00000233953, ENSG00000104140, ENSG00000161671, ENSG00000277246, ENSG00000235494, ENSG00000102683, ENSG00000163825, ENSG00000286041, ...

✅    linked: FeatureSet(id='faeg24ZeXZ39IsRHyhjO', n=36503, type='float', registry='bionty.Gene', hash='dnRexHCtxtmOU81_EpoJ', created_by_id='DzTjkKse')

💡 parsing feature names of slot 'obs'

✅    validated 4 Feature records on name: assay, cell_type, donor, tissue

❗    did not validate 3 Feature records for names: assay_ontology_term_id, cell_type_ontology_term_id, tissue_ontology_term_id

❗    ignoring non-validated features: assay_ontology_term_id,cell_type_ontology_term_id,tissue_ontology_term_id

✅    linked: FeatureSet(id='qy8qAk5avRBEelHEJZSR', n=4, registry='core.Feature', hash='0oNPvNWBy6TGwtTxiaNO', created_by_id='DzTjkKse')

file.save()

✅ saved 2 feature sets for slots: ['var', 'obs']

✅ storing file 'AXO3ps8RX50KjYoWYb58' at '.lamindb/AXO3ps8RX50KjYoWYb58.h5ad'

The file has the following linked feature sets:

file.features

'var': FeatureSet(id='faeg24ZeXZ39IsRHyhjO', n=36503, type='float', registry='bionty.Gene', hash='dnRexHCtxtmOU81_EpoJ', updated_at=2023-08-17 14:11:29, created_by_id='DzTjkKse')
'obs': FeatureSet(id='qy8qAk5avRBEelHEJZSR', n=4, registry='core.Feature', hash='0oNPvNWBy6TGwtTxiaNO', updated_at=2023-08-17 14:11:35, created_by_id='DzTjkKse')

Let’s now link observational metadata.

cell_types = lb.CellType.from_values(conde22.obs.cell_type, field="name")
efs = lb.ExperimentalFactor.from_values(conde22.obs.assay, field="name")
tissues = lb.Tissue.from_values(conde22.obs.tissue, field="name")

✅ validated 32 CellType records from Bionty on name: classical monocyte, T follicular helper cell, memory B cell, alveolar macrophage, naive thymus-derived CD4-positive, alpha-beta T cell, effector memory CD8-positive, alpha-beta T cell, terminally differentiated, alpha-beta T cell, CD4-positive helper T cell, naive thymus-derived CD8-positive, alpha-beta T cell, macrophage, mucosal invariant T cell, group 3 innate lymphoid cell, naive B cell, animal cell, CD16-negative, CD56-bright natural killer cell, human, plasma cell, CD8-positive, alpha-beta memory T cell, CD16-positive, CD56-dim natural killer cell, human, gamma-delta T cell, conventional dendritic cell, ...

✅ validated 3 ExperimentalFactor records from Bionty on name: 10x 3' v3, 10x 5' v2, 10x 5' v1

✅ validated 17 Tissue records from Bionty on name: blood, thoracic lymph node, spleen, lung, mesenteric lymph node, lamina propria, liver, jejunal epithelium, omentum, bone marrow, ileum, caecum, thymus, skeletal muscle tissue, duodenum, sigmoid colon, transverse colon

file.add_labels([lb.settings.species], feature="species")
ln.save(cell_types + efs + tissues)
file.add_labels(cell_types + efs + tissues)

✅ created feature set for slot 'external'

✅ linked labels 'classical monocyte', 'T follicular helper cell', 'memory B cell', 'alveolar macrophage', 'naive thymus-derived CD4-positive, alpha-beta T cell', 'effector memory CD8-positive, alpha-beta T cell, terminally differentiated', 'alpha-beta T cell', 'CD4-positive helper T cell', 'naive thymus-derived CD8-positive, alpha-beta T cell', 'macrophage', 'mucosal invariant T cell', 'group 3 innate lymphoid cell', 'naive B cell', 'animal cell', 'CD16-negative, CD56-bright natural killer cell, human', 'plasma cell', 'CD8-positive, alpha-beta memory T cell', 'CD16-positive, CD56-dim natural killer cell, human', 'gamma-delta T cell', 'conventional dendritic cell', 'CD8-positive, alpha-beta memory T cell, CD45RO-positive', 'effector memory CD4-positive, alpha-beta T cell', 'non-classical monocyte', 'mast cell', 'regulatory T cell', 'progenitor cell', 'dendritic cell, human', 'plasmablast', 'plasmacytoid dendritic cell', 'lymphocyte', 'germinal center B cell', 'megakaryocyte' to feature 'cell_type'

✅ linked labels '10x 3' v3', '10x 5' v2', '10x 5' v1' to feature 'assay', linked feature 'assay' to registry 'bionty.ExperimentalFactor'

✅ linked labels 'blood', 'thoracic lymph node', 'spleen', 'lung', 'mesenteric lymph node', 'lamina propria', 'liver', 'jejunal epithelium', 'omentum', 'bone marrow', 'ileum', 'caecum', 'thymus', 'skeletal muscle tissue', 'duodenum', 'sigmoid colon', 'transverse colon' to feature 'tissue'

As neither the core schema nor lnschema_bionty have a Donor table, we’re using Label to track donor ids:

donors = ln.Label.from_values(conde22.obs["donor"])
ln.save(donors)
file.add_labels(donors)

❗ did not validate 12 Label records for names: D496, 621B, A29, A36, A35, 637C, A52, A37, D503, 640C, A31, 582C

✅ linked labels 'D496', '621B', 'A29', 'A36', 'A35', '637C', 'A52', 'A37', 'D503', '640C', 'A31', '582C' to feature 'donor', linked feature 'donor' to registry 'core.Label'

file.describe()

💡 File(id=AXO3ps8RX50KjYoWYb58, key=None, suffix=.h5ad, accessor=AnnData, description=Conde22, version=None, size=28061905, hash=3cIcmoqp1MxjX8NlRkKGlQ, hash_type=md5, created_at=2023-08-17 14:11:35.656298+00:00, updated_at=2023-08-17 14:11:35.656325+00:00)

Provenance:
    🗃️ storage: Storage(id='u9weBVGy', root='/home/runner/work/lamin-usecases/lamin-usecases/docs/test-scrna', type='local', updated_at=2023-08-17 14:10:47, created_by_id='DzTjkKse')
    📔 transform: Transform(id='Nv48yAceNSh8z8', name='Curate & link scRNA-seq datasets', short_name='scrna', stem_id='Nv48yAceNSh8', version='0', type='notebook', updated_at=2023-08-17 14:11:29, created_by_id='DzTjkKse')
    👣 run: Run(id='MtzJ6VmKpLQiUtbzoMTo', run_at=2023-08-17 14:10:54, transform_id='Nv48yAceNSh8z8', created_by_id='DzTjkKse')
    👤 created_by: User(id='DzTjkKse', handle='testuser1', email='testuser1@lamin.ai', name='Test User1', updated_at=2023-08-17 14:10:47)
Features:
  var (X):
    🔗 index (36503, bionty.Gene.id): ['qo0IbL5zk0Ho', '0qs9l4ehlo5k', 'Wt6KcsC2HNFM', 'XzlnJbBfj81b', '2T9MPEOUfnUy'...]
  external:
    🔗 species (1, bionty.Species): ['human']
  obs (metadata):
    🔗 cell_type (32, bionty.CellType): ['mucosal invariant T cell', 'CD8-positive, alpha-beta memory T cell', 'CD16-negative, CD56-bright natural killer cell, human', 'group 3 innate lymphoid cell', 'progenitor cell']
    🔗 assay (3, bionty.ExperimentalFactor): ["10x 5' v2", "10x 5' v1", "10x 3' v3"]
    🔗 tissue (17, bionty.Tissue): ['thoracic lymph node', 'blood', 'lamina propria', 'thymus', 'transverse colon']
    🔗 donor (12, core.Label): ['D496', 'A35', 'D503', 'A36', 'A31']

A less well curated dataset

Let’s now consider a dataset with less-well curated features:

pbcm68k = ln.dev.datasets.anndata_pbmc68k_reduced()

We see that this dataset is indexed by gene symbols:

pbcm68k.var.index

Index(['HES4', 'TNFRSF4', 'SSU72', 'PARK7', 'RBP7', 'SRM', 'MAD2L2', 'AGTRAP',
       'TNFRSF1B', 'EFHD2',
       ...
       'ATP5O', 'MRPS6', 'TTC3', 'U2AF1', 'CSTB', 'SUMO3', 'ITGB2', 'S100B',
       'PRMT2', 'MT-ND3'],
      dtype='object', name='index', length=765)

Because gene symbols don’t uniquely characterize an Ensembl ID, we’re linking more feature records to this file than columns in the AnnData.

Tip

Use Ensembl Gene IDs rather than gene Symbols to index genes.

validated = lb.Gene.validate(pbcm68k.var.index, lb.Gene.symbol)

💡 using global setting species = human

✅ 695 terms (90.80%) are validated

❗ 70 terms (9.20%) are not validated

pbcm68k_validated = pbcm68k[:, validated]

Link cell types:

Show code cell content Hide code cell content

# inspect shows none of the terms are mappable
lb.CellType.inspect(pbcm68k_validated.obs["cell_type"], "name")

# here we search the cell type names from the public ontology and grab the top match
# then add the cell type names from the pbcm68k as synonyms
celltype_bt = lb.CellType.bionty()
ontology_ids = []
mapper = {}
for ct in pbcm68k_validated.obs["cell_type"].unique():
    ontology_id = celltype_bt.search(ct).iloc[0].ontology_id
    record = lb.CellType.from_bionty(ontology_id=ontology_id)
    mapper[ct] = record.name
    record.save()
    record.add_synonym(ct)

pbcm68k_validated.obs["cell_type"] = pbcm68k_validated.obs["cell_type"].map(mapper)

❗ received 9 unique terms, 61 empty/duplicated terms are ignored

✅ 0 terms (0.00%) are validated

❗ 9 terms (100.00%) are not validated

✅ validated 1 CellType record from Bionty on ontology_id: CL:0000451

✅ validated 1 CellType record from Bionty on ontology_id: CL:0001201

✅ validated 1 CellType record from Bionty on ontology_id: CL:0001087

✅ validated 1 CellType record from Bionty on ontology_id: CL:0000910

✅ validated 1 CellType record from Bionty on ontology_id: CL:0000919

✅ validated 1 CellType record from Bionty on ontology_id: CL:0002057

✅ validated 1 CellType record on ontology_id: CL:0000939

✅ validated 1 CellType record from Bionty on ontology_id: CL:0002102

✅ validated 1 CellType record on ontology_id: CL:0000990

/tmp/ipykernel_2374/593905118.py:16: ImplicitModificationWarning: Trying to modify attribute `.obs` of view, initializing view as actual.
  pbcm68k_validated.obs["cell_type"] = pbcm68k_validated.obs["cell_type"].map(mapper)

Now, all cell types should be validated:

lb.CellType.validate(pbcm68k_validated.obs["cell_type"], "name");

❗ received 9 unique terms, 61 empty/duplicated terms are ignored

✅ 9 terms (100.00%) are validated

file_pbcm68k = ln.File.from_anndata(
    pbcm68k_validated, description="10x reference pbmc68k", var_ref=lb.Gene.symbol
)

💡 file will be copied to default storage upon `save()` with key `None` ('.lamindb/K88ck4vLa862rEysKlNO.h5ad')

💡 parsing feature names of X stored in slot 'var'

💡    using global setting species = human

✅    validated 695 Gene records on symbol: IGFBP7, U2AF1, PRDX6, SBDS, AKR1C3, LAMTOR4, LCK, IMPDH2, RABAC1, RNH1, TIMM10, TMEM208, EIF3I, CD160, BAD, OTUB1, TIGIT, PIK3IP1, UBALD2, IFITM1, ...

✅    linked: FeatureSet(id='lhjkK150mTFQmypmfdWI', n=695, type='float', registry='bionty.Gene', hash='W4ps_86b5dxk2Wd1gWTo', created_by_id='DzTjkKse')

💡 parsing feature names of slot 'obs'

✅    validated 1 Feature record on name: cell_type

❗    did not validate 3 Feature records for names: louvain, n_genes, percent_mito

❗    ignoring non-validated features: louvain,n_genes,percent_mito

✅    linked: FeatureSet(id='inGiJPDvnv7N0w3fyNzc', n=1, registry='core.Feature', hash='DltohqJBj0euCTrAzXIp', created_by_id='DzTjkKse')

file_pbcm68k.save()

✅ saved 2 feature sets for slots: ['var', 'obs']

✅ storing file 'K88ck4vLa862rEysKlNO' at '.lamindb/K88ck4vLa862rEysKlNO.h5ad'

cell_types = lb.CellType.from_values(pbcm68k_validated.obs["cell_type"], "name")
file_pbcm68k.add_labels(cell_types)

✅ validated 9 CellType records on name: B cell, CD19-positive, CD14-positive, CD16-negative classical monocyte, CD16-positive, CD56-dim natural killer cell, human, CD38-negative naive B cell, CD8-positive, CD25-positive, alpha-beta regulatory T cell, conventional dendritic cell, cytotoxic T cell, dendritic cell, effector memory CD4-positive, alpha-beta T cell, terminally differentiated

✅ linked labels 'B cell, CD19-positive', 'CD14-positive, CD16-negative classical monocyte', 'CD16-positive, CD56-dim natural killer cell, human', 'CD38-negative naive B cell', 'CD8-positive, CD25-positive, alpha-beta regulatory T cell', 'conventional dendritic cell', 'cytotoxic T cell', 'dendritic cell', 'effector memory CD4-positive, alpha-beta T cell, terminally differentiated' to feature 'cell_type'

file_pbcm68k.describe()

💡 File(id=K88ck4vLa862rEysKlNO, key=None, suffix=.h5ad, accessor=AnnData, description=10x reference pbmc68k, version=None, size=589484, hash=eKVXV5okt5YRYjySMTKGEw, hash_type=md5, created_at=2023-08-17 14:11:44.665221+00:00, updated_at=2023-08-17 14:11:44.665270+00:00)

Provenance:
    🗃️ storage: Storage(id='u9weBVGy', root='/home/runner/work/lamin-usecases/lamin-usecases/docs/test-scrna', type='local', updated_at=2023-08-17 14:10:47, created_by_id='DzTjkKse')
    📔 transform: Transform(id='Nv48yAceNSh8z8', name='Curate & link scRNA-seq datasets', short_name='scrna', stem_id='Nv48yAceNSh8', version='0', type='notebook', updated_at=2023-08-17 14:11:44, created_by_id='DzTjkKse')
    👣 run: Run(id='MtzJ6VmKpLQiUtbzoMTo', run_at=2023-08-17 14:10:54, transform_id='Nv48yAceNSh8z8', created_by_id='DzTjkKse')
    👤 created_by: User(id='DzTjkKse', handle='testuser1', email='testuser1@lamin.ai', name='Test User1', updated_at=2023-08-17 14:10:47)
Features:
  var (X):
    🔗 index (695, bionty.Gene.id): ['l77NXBSjDNfh', 'GuX3mcJ3R0JC', '5GuQULEkSkBV', 'k7KXOWF0TiRf', '20169ReyBOCR'...]
  obs (metadata):
    🔗 cell_type (9, bionty.CellType): ['conventional dendritic cell', 'B cell, CD19-positive', 'dendritic cell', 'CD14-positive, CD16-negative classical monocyte', 'CD8-positive, CD25-positive, alpha-beta regulatory T cell']

file_pbcm68k.view_lineage()

🎉 Now let’s continue with data integration: Integrate scRNA-seq datasets