Feature request: parallelised varianceDecomposition()

[skiaphrene]

Dear scLVM team,

As indicated in the tutorial, the calculations behind varianceDecomposition() can be quite intensive. Would it be possible to provide some native functionality to perform or to simplify the parallelisation of these calculations? So far I have been able to write a work-around using the snow package and low-level scLVM functions:

• Parallelisation function:

my.var.decomp.call <- function(idx) {
  # parallelisation function

  library(scLVM)
  configLimix2(limix_path) # my corrected version of this function - see other issue thread.

  # have to recreate an entire sclvm object so things are properly configured in rPython instance
  # i.e. python.assign("sclvm",sclvm) fails.
  mysclvm <- new("scLVM") 
  mysclvm_py_name <- "mysclvm" # I did not manage to get the deparsing working in here
  mysclvm <- init( mysclvm,Y=t(log10(nCountsMmus+1)) ,
                   tech_noise = as.vector(techNoise$techNoiseLog))
  rm( nCountsMmus, techNoise )
  CellCycle <- fitFactor(
    mysclvm, geneSet = ens_ids_cc, k=1)
  rm( ens_ids_cc )

  # decompose variance & get results
  varianceDecomposition_py(mysclvm_py_name, K=CellCycle$K, idx=idx)
  res <- getVarianceComponents_py(mysclvm_py_name)
  rm(mysclvm)

  return( res$var[res$conv,] ) # only returning converged estimates
}

• Main script:

library(snow)

clus <- makeSOCKcluster( getOption("cl.cores", 4)-1  )
clusterExport(clus, c(
  "limix_path", "configLimix2", 
  "nCountsMmus", "techNoise", "ens_ids_cc"))
idx.split <- split( which(is_hvg), ceiling(seq_along(which(is_hvg))/1000)) 
lapply(idx.split, length)

print(system.time( {
  ResList <- 
    clusterApplyLB(clus, idx.split, my.var.decomp.call)
}))

stopCluster(clus) ; rm(clus)
results_var <- do.call( rbind, ResList)

Thanks & best regards,

-- Alex