WeizGrid is a Simple framework for using the SGE cluster on Weizmann for parallel work (so if you're not from Weizmann's Math&CS department you won't find much use in this, although it is can be easily used by anyone who runs Sun's Grid Engine and MATLAB).
You can get the latest version off of GitHub from here.
Features:
Use case example:
You have a job that processes 1000 images in some way, running the same algorithm for each image, outputing something for each image, and finally aggregating the results from all images. Up until now, you ran this in a single job that took 1000 minutes (because lets assume it takes 1 minute to process each image). Using WeizGrid you can take advantage of the fact that the job can be parallelized - instead of 1 job WeizGrid helps you easily split the job to 80 parallel jobs so that you get all 1000 images in 1000/80=12.5 minutes (!!)
Documentation:
Read the full README file to get a better picture and also make sure to go over the 'sample.m' and 'calcPrimes.m' files included as an example and over the quick-help of the function WGexec, and WGgetResults. If you still have questions please feel free to contact me.
Terms:
WeizGrid is obviously free of use with no restrictions except for one: if you find a bug I need to know to make WeizGrid better !!
You can get the latest version off of GitHub from here.
Features:
- Helps split your own code into fractions that can be run in parallel on the cluster.
- Allows you to aggregate results in an efficient manner.
- Write your code ONCE! Execute the same code on PC for debugging and on the cluster for production.
- Recover from errors on the cluster - splitting means if 1 iteration out of a 1000 failed you still have 999 iterations in your hand!
Use case example:
You have a job that processes 1000 images in some way, running the same algorithm for each image, outputing something for each image, and finally aggregating the results from all images. Up until now, you ran this in a single job that took 1000 minutes (because lets assume it takes 1 minute to process each image). Using WeizGrid you can take advantage of the fact that the job can be parallelized - instead of 1 job WeizGrid helps you easily split the job to 80 parallel jobs so that you get all 1000 images in 1000/80=12.5 minutes (!!)
Documentation:
Read the full README file to get a better picture and also make sure to go over the 'sample.m' and 'calcPrimes.m' files included as an example and over the quick-help of the function WGexec, and WGgetResults. If you still have questions please feel free to contact me.
Terms:
WeizGrid is obviously free of use with no restrictions except for one: if you find a bug I need to know to make WeizGrid better !!
No comments:
Post a Comment