How it can be expanded to n-dimensional?

[valera7979]

Hi,

I tested patchwork. It seems 2D only. But maybe it`s not difficult upgrade to n-dimensions?
Thanks

[tomLandry]

Hello. I'll transmit you request to people working on new revisions of the algorithm.

[Mignastor]

Hi,

It should work with n-dimensions by passing any size of Array[Double].
However, its scalability is not really good, we are currently working on this issue.

Could you share what is not working?

Thanks

[valera7979]

Dataset was very simple. Only to test n-dimensional computing:

10,10,1000,1,1,1,1,1,1,1000
20,20,1,1,1,1,1,1,1,1
20,30,1,1,1,1,1,1,1,1
30,330,1,1,1,1,1,1,1,1
330,30,444,1,1,1,1,1,1,1

// PatchWork parameters
val epsilon = Array(10.1, 10.1)
val minPts = 1
val minCellInCluster = 2
val ratio = 0.0

And i got:

10.0 10.0 1
20.0 20.0 1
20.0 30.0 1
30.0 330.0 -1
330.0 30.0 -1

---

number of points : 5
number of clusters : 1

cluster 1 has 3 cells

[Mignastor]

epsilon should have the same number of dimension than your data.
It is an hypercell than divide your space into a grid.
Something like:
val epsilon = Array(10.1, 10.1, 1, 1, 1, 1, 1, 1, 1, 1)
Tell me if this solves your problem.

[valera7979]

no, it doesn`t work. I got same result. Only epsilon value in output has changed:

cluster 1 has 3 cells

---

size of epsilon : [10.1,10.1,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0]
min pts in each cell : 1
time of training : 1535 ms

[Mignastor]

Hello,

My colleague @martinsotir looked at this into more details, here is a link to his code:
https://gist.github.com/martinsotir/d16a26b975d90bfd10716fd9a0e0e769

Something he has seen is that here:

patchwork/src/main/scala/ca/crim/spark/mllib/clustering/PatchWork.scala

Line 256 in d89e76a

clusterList.filter(cluster => cluster.cellsList.size > this.minCell).toList

We have a condition like this : clusterList.filter(cluster => cluster.cellsList.size > this.minCell).toList
Where it could be easier to understand with a condition like this: clusterList.filter(cluster => cluster.cellsList.size >= this.minCell).toList

Hope this helps

[valera7979]

Hi. Thanks for answer.
It solves problem, but partially. Now i get points with all dimensions.
I edited initial code:
In 23 line last .map i changed to map(_.toDouble)).cache
and all 38 line to x.mkString("\t") + "\t" + patchworkModel.predict(x).getID

But it seems to me that it calculates incorrectly.
for dataset:
Array(10,10,2,1,1,1,1,1,1,1),
Array(20,20,2,1,1,1,1,1,1,1),
Array(20,30,2,1,1,1,1,1,1,1),

and epsilon = Array(10.1, 10.1, 1, 1, 1, 1, 1, 1, 1, 1)
it gives
10.0 10.0 2.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -1
20.0 20.0 2.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 2
20.0 30.0 2.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 2

but when i rised epsilon to add first point in claster, all did not belong to any cluster
epsilon = Array(15.1, 15.1, 1, 1, 1, 1, 1, 1, 1, 1)
10.0 10.0 2.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -1
20.0 20.0 2.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -1
20.0 30.0 2.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 -1
I tried a lot of epsilons and datasets, and often got wrong results

[kaifeur]

Hello. I'll transmit you request to people working on new revisions of the algorithm.

@tomLandry Hi! Where can I get a new version? Does it actually exist?
Anyway, PatchWork is an amazing thing :)