Error running main_probabilities.py

[alvaralmstedt]

Hi! I'm having an issue running main_probabilities.py in my dataset of 60x wgs tumour. It is crashing with the following error:

[xalmal@bespin sv-bay-data]$ /apps/CLC_ExternalApps/SV-Bay/miniconda2/bin/python2.7 -B /apps/CLC_ExternalApps/SV-Bay/SV-Bay/src/main_probabilities.py -c /apps/CLC_ExternalApps/SV-Bay/SV-Bay/config/config.yaml
Traceback (most recent call last):
  File "/apps/CLC_ExternalApps/SV-Bay/SV-Bay/src/main_probabilities.py", line 199, in <module>
    input_data = BayesianInputData(config, stats, chromosomes)
  File "/apps/CLC_ExternalApps/SV-Bay/SV-Bay/src/bayesianinput.py", line 57, in __init__
    self.__LoadLambda(config, stats)
  File "/apps/CLC_ExternalApps/SV-Bay/SV-Bay/src/bayesianinput.py", line 224, in __LoadLambda
    (lambda_norm,lambda_abnorm) = self.__CalculateLambda(config, stats)
  File "/apps/CLC_ExternalApps/SV-Bay/SV-Bay/src/bayesianinput.py", line 363, in __CalculateLambda
    if gem_val_abnormal[i/step]!=-1:
IndexError: list index out of range

I looked in the log file and it seems to be complaining that the .gem file and .fa files for chromosome 1 are of different lengths. I used the hg19 fasta and gems from the zip files linked to from the documentation here, so I feel that it should work. I will also paste the main_probabilities.log here:

main_probabilities.py    line:197  INFO     [2016-11-11 11:12:22,971]   Chromosomes to process:['chr1', 'chr2', 'chr3', 'chr4', 'chr5', 'chr6', 'chr7', 'chr8', 'chr9', 'chr10', 'chr11', 'chr12', 'chr13', 'chr14', 'chr15', 'chr16', 'chr17', 'chr18', 'chr19', 'chr20', 'chr21', 'chr22', 'chrX', 'chrY']
bayesianinput.py         line:48   INFO     [2016-11-11 11:12:22,971]   Loading data...
bayesianinput.py         line:106  INFO     [2016-11-11 11:12:22,971]   Loading normal fragments...
bayesianinput.py         line:137  INFO     [2016-11-11 11:32:03,126]   Loading links...
bayesianinput.py         line:158  INFO     [2016-11-11 11:32:21,881]   Constructing sublinks...
bayesianinput.py         line:223  WARNING  [2016-11-11 11:32:32,877]   Lambda file not found, calculating lambda
bayesianinput.py         line:252  INFO     [2016-11-11 11:32:32,877]   Started calculating lambda_gen
bayesianinput.py         line:67   WARNING  [2016-11-11 11:32:39,167]   Lengths of .fa and .gem sequences do not match for chrom chr1 (249250626 and 248956422)
bayesianinput.py         line:69   WARNING  [2016-11-11 11:32:39,167]   Some flanking regions will be truncated to match the minimal length                 and some links (intersecting with chrom tail) will be dropped.
bayesianinput.py         line:70   WARNING  [2016-11-11 11:32:39,167]   Please fix your .gem file.

PS. How did you generate these .gem files so that I can try to generate new ones for myself?

Thank you!

/Alvar

[zhanghappy123]

Hi,alvaralmstedt:
I indeed get some errors while running SV-Bay first step (main_clustering.py)on my own data,
can you give me your all test data? I wonder know why own data have some problem.
thanks

[alvaralmstedt]

Hi zhanghappy123! Unfortunately I cannot give out my data as it is from patients. What is the error you are getting? I had some issues with the clustering step as well but i managed to fix it by installing the versions of of all the prerequisites that were out when the last commit of SV-bay was done:

numpy==1.8.2 
scipy==0.13.3 
matplotlib==1.3.1 
pyaml==16.9.0 
pysam==0.9.1.4 
joblib==0.10.2

[BioXRay]

Hi zhanghappy123! Unfortunately I cannot give out my data as it is from patients. What is the error you are getting? I had some issues with the clustering step as well but i managed to fix it by installing the versions of of all the prerequisites that were out when the last commit of SV-bay was done:

numpy==1.8.2 
scipy==0.13.3 
matplotlib==1.3.1 
pyaml==16.9.0 
pysam==0.9.1.4 
joblib==0.10.2

Hi! We got the error running the src/main_clustering.py in my Mate-Pairs data. As following error:
Traceback (most recent call last):
File "/home/cdx/genome_tools/SV-Bay/src/main_clustering.py", line 89, in
Process(data_file, config)
File "/home/cdx/genome_tools/SV-Bay/src/main_clustering.py", line 80, in Process
cf.PerformClustering()
File "/home/cdx/genome_tools/SV-Bay/src/chrfragments.py", line 175, in PerformClustering
utils.clust(self.fr_abn, self.chr, 'fr', self.biggest_normal, self.smallest_normal, self.config, self.flag_direction)
File "/home/cdx/genome_tools/SV-Bay/src/utils.py", line 65, in clust
clusters = cluster.Clustering(fragments, M, S, chrm, chrm, 0, config,direction)
File "/home/cdx/genome_tools/SV-Bay/src/cluster.py", line 414, in Clustering
new_clusters = KMeansSplit(cluster_for_kmeans, D)
File "/home/cdx/genome_tools/SV-Bay/src/cluster.py", line 342, in KMeansSplit
(res_my, idx_my) = kmeans2(array(zip(x)), k)
File "/usr/lib/python2.7/dist-packages/scipy/cluster/vq.py", line 783, in kmeans2
return _kmeans2(data, clusters, iter, nc, _valid_miss_meth[missing])
File "/usr/lib/python2.7/dist-packages/scipy/cluster/vq.py", line 797, in _kmeans2
new_code, has_members = _vq.update_cluster_means(data, label, nc)
File "scipy/cluster/_vq.pyx", line 340, in scipy.cluster._vq.update_cluster_means (scipy/cluster/_vq.c:4654)
TypeError: type other than float or double not supported
Could you give some suggestion? That the scipy verison is 0.17.0. We hope you reply as soon as possible. Thank you!

[guanghaoli]

Hi! I'm having an issue running main_probabilities.py in my dataset of 60x wgs tumour. It is crashing with the following error:

[xalmal@bespin sv-bay-data]$ /apps/CLC_ExternalApps/SV-Bay/miniconda2/bin/python2.7 -B /apps/CLC_ExternalApps/SV-Bay/SV-Bay/src/main_probabilities.py -c /apps/CLC_ExternalApps/SV-Bay/SV-Bay/config/config.yaml
Traceback (most recent call last):
  File "/apps/CLC_ExternalApps/SV-Bay/SV-Bay/src/main_probabilities.py", line 199, in <module>
    input_data = BayesianInputData(config, stats, chromosomes)
  File "/apps/CLC_ExternalApps/SV-Bay/SV-Bay/src/bayesianinput.py", line 57, in __init__
    self.__LoadLambda(config, stats)
  File "/apps/CLC_ExternalApps/SV-Bay/SV-Bay/src/bayesianinput.py", line 224, in __LoadLambda
    (lambda_norm,lambda_abnorm) = self.__CalculateLambda(config, stats)
  File "/apps/CLC_ExternalApps/SV-Bay/SV-Bay/src/bayesianinput.py", line 363, in __CalculateLambda
    if gem_val_abnormal[i/step]!=-1:
IndexError: list index out of range

I looked in the log file and it seems to be complaining that the .gem file and .fa files for chromosome 1 are of different lengths. I used the hg19 fasta and gems from the zip files linked to from the documentation here, so I feel that it should work. I will also paste the main_probabilities.log here:

main_probabilities.py    line:197  INFO     [2016-11-11 11:12:22,971]   Chromosomes to process:['chr1', 'chr2', 'chr3', 'chr4', 'chr5', 'chr6', 'chr7', 'chr8', 'chr9', 'chr10', 'chr11', 'chr12', 'chr13', 'chr14', 'chr15', 'chr16', 'chr17', 'chr18', 'chr19', 'chr20', 'chr21', 'chr22', 'chrX', 'chrY']
bayesianinput.py         line:48   INFO     [2016-11-11 11:12:22,971]   Loading data...
bayesianinput.py         line:106  INFO     [2016-11-11 11:12:22,971]   Loading normal fragments...
bayesianinput.py         line:137  INFO     [2016-11-11 11:32:03,126]   Loading links...
bayesianinput.py         line:158  INFO     [2016-11-11 11:32:21,881]   Constructing sublinks...
bayesianinput.py         line:223  WARNING  [2016-11-11 11:32:32,877]   Lambda file not found, calculating lambda
bayesianinput.py         line:252  INFO     [2016-11-11 11:32:32,877]   Started calculating lambda_gen
bayesianinput.py         line:67   WARNING  [2016-11-11 11:32:39,167]   Lengths of .fa and .gem sequences do not match for chrom chr1 (249250626 and 248956422)
bayesianinput.py         line:69   WARNING  [2016-11-11 11:32:39,167]   Some flanking regions will be truncated to match the minimal length                 and some links (intersecting with chrom tail) will be dropped.
bayesianinput.py         line:70   WARNING  [2016-11-11 11:32:39,167]   Please fix your .gem file.

PS. How did you generate these .gem files so that I can try to generate new ones for myself?

Thank you!

/Alvar

Hi Alvar,
Have you solve this problem? I also have this problem.

Guanghao