Dataset delete does not work

[mmmenno]

Someone accidentally uploaded the rm dataset with wrong id's, i think. Suddenly there where id's like rm/8007.0 instead of the rm/8007 it should have been.

Deleting the entire dataset and upload it again with the proper ideas should do the trick, one imagines. Not so. While https://api.histograph.io/datasets/rm/pits gives you a "Dataset 'rm' not found", https://api.histograph.io/search?q=rm/8007.0 keeps returning pits.

Maybe deleting pits / relations from ES doesn't work? Very strange & a bit alarming. Any ideas, @ocataco ?

[bertspaan]

This is caused by a known bug graphmalizer/graphmalizer-core#18.

Graphmalizer's Cypher queries need to be changed, so that these VACANT nodes are deleted.

There is an easy fix which would probably also work:

Change https://github.com/histograph/neo4j-plugin and make sure it only returns non-VACANT nodes. This can be done here, probably: https://github.com/histograph/neo4j-plugin/blob/master/src/main/java/org/waag/histograph/plugins/ExpandConcepts.java#L33

[wires]

sorry guys, missed this issue! I'll schedule some time to dive into graphmalizer/graphmalizer-core#18 again

[sbocconi]

There seems to be a solution for this issue, as suggested by @bertspaan. Issue labeled as ready (to be solved) then.

[tomdemeyer]

VACANT nodes are created on delete? This seems not ok; how to fix?
Or is this a separate issue??

import dataset pdc:
match(n {dataset: "pdc" } ) return count(n)
--> 5402
match(n:_VACANT {dataset: "pdc" } ) return count(n)
--> 0
after delete dataset:
match(n {dataset: "pdc" } ) return count(n)
--> 1707
match(n:_VACANT {dataset: "pdc" } ) return count(n)
--> 1707

[wires]

probably related, didn't have time to look at this, somewhere in queries.yml I forget the clear the vacant nodes)

[sbocconi]

VACANT: Betekent dit: als je relatie toevoegt tussen A en B, maar PIT’s A en B bestaan (nog) niet, wordt de relatie toegevoegd, en A en B ook, maar A en B zijn dan VACANT

There are several issues here, but can we say that adding a dataset and then removing it should leave the repository in an unchanged status? Or is this not true? This can be part of the tests

[sbocconi]

Another comment from @wires on Slack:

volgens mij is het de situatie [+] b -> c ; [+] a == b ; [+] c == d ; [-] b -> c die problemen oplevert
dus met [+] … bedoel ik toevoegen

[mmmenno]

I only experienced this problem with the faulty rm dataset that had id's like rm/1641.0 - so maybe it had something to do with those strange id's. @tomdemeyer deleted the faulty set manually, so for now we got rid of the problem.

With the VACANT node diagnosis, we should've seen this problem more often and we should be able to reproduce the problem, right?

[sbocconi]

From a discussion with @bertspaan:

Drie problemen:

1. IO verwijdert ES-index niet, omdat dit problemen opleverde met ES-service van Amazon (als je index verwijdert die al verwijderd is maakt ES de index opnieuw aan, maar met verkeerde mapping. Of iets dergelijks)
2. neo4j-plugin grijpt ​*alle*​ PITs uit graaf, ook met label `VACANT`
3. Graphmalizer verwijdert `VACANT` niet (of niet altijd) op de goede manier

Number 3 is related to graphmalizer/graphmalizer-core#18

[wires]

Regarding nr 3 above / issue 18 ; the guarantee graphmalizer gives you (when it's not bugged) is that

there will never appear a "dangling vacant node", that is, for all (n:_VACANT), degree(n) > 0.

So they should only appear when there is some edge referring to them.

What we notice in #18 is that there appears to arise a situation here a single _VACANT node remains connected to a 'concept node' (the representative node of an equivalence class; the collection of equivalent things).

In any case, I can run the tests again, so I will dive into this.

ps. It is a little bit subtle as I try to remain some "mathematical" properties of the program. For instance, we -kind of- have the following guarantee on concept nodes:

there will never appear "a dangling concept node", that is, for all (n:'='), degree(n) > 1

This can be either useful or annoying or both or neither :-) it also doesn't exclude the case ()--<>--() where you have two "dangling vacant nodes" connected to a concept node. This shouldn't be allowed, etc.

anyway, enough talk! COMPUTEREN!