{"__v":0,"_id":"586d3d576481370f00f6f307","category":{"project":"55faf11ba62ba1170021a9a7","version":"55faf11ba62ba1170021a9aa","_id":"586d3c287c6b5b2300c05055","__v":0,"sync":{"url":"","isSync":false},"reference":false,"createdAt":"2017-01-04T18:17:12.683Z","from_sync":false,"order":18,"slug":"task-execution","title":"Task Execution"},"parentDoc":null,"project":"55faf11ba62ba1170021a9a7","user":"575e85ac41c8ba0e00259a44","version":{"__v":35,"_id":"55faf11ba62ba1170021a9aa","project":"55faf11ba62ba1170021a9a7","createdAt":"2015-09-17T16:58:03.490Z","releaseDate":"2015-09-17T16:58:03.490Z","categories":["55faf11ca62ba1170021a9ab","55faf8f4d0e22017005b8272","55faf91aa62ba1170021a9b5","55faf929a8a7770d00c2c0bd","55faf932a8a7770d00c2c0bf","55faf94b17b9d00d00969f47","55faf958d0e22017005b8274","55faf95fa8a7770d00c2c0c0","55faf96917b9d00d00969f48","55faf970a8a7770d00c2c0c1","55faf98c825d5f19001fa3a6","55faf99aa62ba1170021a9b8","55faf99fa62ba1170021a9b9","55faf9aa17b9d00d00969f49","55faf9b6a8a7770d00c2c0c3","55faf9bda62ba1170021a9ba","5604570090ee490d00440551","5637e8b2fbe1c50d008cb078","5649bb624fa1460d00780add","5671974d1b6b730d008b4823","5671979d60c8e70d006c9760","568e8eef70ca1f0d0035808e","56d0a2081ecc471500f1795e","56d4a0adde40c70b00823ea3","56d96b03dd90610b00270849","56fbb83d8f21c817002af880","573c811bee2b3b2200422be1","576bc92afb62dd20001cda85","5771811e27a5c20e00030dcd","5785191af3a10c0e009b75b0","57bdf84d5d48411900cd8dc0","57ff5c5dc135231700aed806","5804caf792398f0f00e77521","58458b4fba4f1c0f009692bb","586d3c287c6b5b2300c05055"],"is_deprecated":false,"is_hidden":false,"is_beta":true,"is_stable":true,"codename":"","version_clean":"1.0.0","version":"1.0"},"updates":[],"next":{"pages":[],"description":""},"createdAt":"2017-01-04T18:22:15.820Z","link_external":false,"link_url":"","githubsync":"","sync_unique":"","hidden":false,"api":{"results":{"codes":[]},"settings":"","auth":"required","params":[],"url":""},"isReference":false,"order":1,"body":"Execution control is a powerful tool, but keep in mind that if used incorrectly, it may render your workflows inoperable or cause errors that are hard to debug. If you plan on using execution control facilities, please make sure you understand the concepts outlined here.\n\nThe [publicly available workflows](doc:public-apps) on the Platform have been fine-tuned so when you copy a public workflow to your project, you can be sure that the Seven Bridges bioinformaticians have tightened its bolts to make the most of the provisioned compute time.\n\nThe Seven Bridges Platform understands and implements tools, workflows, jobs and execution as prescribed by the [Common Workflow Language (CWL)](http://docs.sevenbridges.com/docs/sdk-overview#section-the-common-workflow-language).\n\n##The need for optimisation\n\nDifferent tools often require different computation instances to operate efficiently.\n\nWhen you build your own workflow or bring your own tools to the Platform, there are parameters you can tune to affect how jobs are executed. \n\nThe Platform scheduling algorithm seeks the best combination of instances for a given task. However, experience shows there is no silver bullet. An algorithm will never be able to find the most efficient solution for every combination of inputs and tools.\n\nThe trick is to match the available compute resources with the resource requirements of your analysis. \n\nExecution control can help you optimize your apps in a number of scenarios:\n  * **Reduce task execution time:** [Parallelizing multiple executions of the same tool](doc:about-parallelizing-tool-executions) can save you precious time if applied to critical tools in your workflow.\n  * ** Lower task cost: **Having a good match between tool resource requirements and provisioned computational instances means that you will only be paying for compute resources that are being used.\n  * **Ensure sufficient resources:** When developing a tool using the [Platform SDK](doc:sdk-overview), one of the key steps is specifying the computational resources (CPU cores and RAM) that the tool needs for proper operation. \n\n##Controlling task execution\n\nThe resource requirements of an analysis are given by the resource requirements (CPU and RAM) of [the tools involved](doc:about-controlling-task-execution). These parameters are used by the [Scheduling Algorithm](doc:about-task-execution#section-scheduling) when deciding how to distribute jobs across available instances. Those parameters are called *execution hints*.\n\nYou can control the compute resources available for your tool executions by [specifying the type of instance](doc:about-controlling-task-execution#section-controlling-task-execution-via-instance-type) to be used and/or the [maximum number of such instances](doc:about-controlling-task-execution#section-controlling-task-execution-via-maximum-number-of-parallel-instances) that can be run in parallel. See [this blog post](blog:making-efficient-use-of-compute-resources) for examples on how to make efficient use of computational resources.\n[block:callout]\n{\n  \"type\": \"danger\",\n  \"body\": \"Beware of mismatches between tool resource requirements and the resources available on the provisioned computation instance. Your task will fail if a tool requires more resources than the provisioned computation instance.\",\n  \"title\": \"Insufficient resources\"\n}\n[/block]\n###Controlling task execution via instance type\n\nYou generally want to pick an instance that leaves the least CPU and memory to waste, given the jobs predicted to run as part of your task. [This blog post](blog:making-efficient-use-of-compute-resources#section-choosing-the-best-compute-instance-for-your-analysis) provides a detailed example of how to determine the most suitable instance type.\n\nTo set the instance type, set the [`sbg:AWSInstanceType`](doc:list-of-execution-hints) or [`sbg:GoogleInstanceType`](doc:list-of-execution-hints) hint at [workflow level](doc:set-execution-hints-at-workflow-level).\n\n####The hierarchy of instance type hints\n\nWhile instance type hints can be set at several levels (workflow, node, and tool), there is a hierarchy that determines which of the settings takes priority during execution. \n\nAn instance type specified at workflow level will override any instance type hints set for any of its nodes or any of the tools within that workflow. \n\nAn instance type specified for a tool will only be considered when the tool is run on its own. When the tool becomes part of a workflow, its instance hint gets overridden by instance hints set at node level, which in turn get overridden by the workflow-level instance hints.\n\n####Defaults\nThe current default instance the Scheduling Algorithm will attempt to use for execution is `c4.2xlarge`, which has 8vCPUs and 15 GB RAM.\n\n###Controlling task execution via maximum number of parallel instances\n\nConfiguring the number of instances running at once for a single task is handy when [parallelizing](doc:about-parallelizing-tool-executions) (scattering) several executions of the same tool.\n\nThe Seven Bridges bioinformaticians exploit this technique when tuning workflows in Public Apps.\n\nTo set the maximum number of instances, set the [`sbg:maxNumberOfParallelInstances`](doc:list-of-execution-hints) hint at [workflow level](doc:set-execution-hints-at-workflow-level).\n\n####Defaults\nThe default value for `sbg:maxNumberOfParallelInstances` is 1.","excerpt":"","slug":"about-controlling-task-execution","type":"basic","title":"About controlling task execution"}

About controlling task execution


Execution control is a powerful tool, but keep in mind that if used incorrectly, it may render your workflows inoperable or cause errors that are hard to debug. If you plan on using execution control facilities, please make sure you understand the concepts outlined here. The [publicly available workflows](doc:public-apps) on the Platform have been fine-tuned so when you copy a public workflow to your project, you can be sure that the Seven Bridges bioinformaticians have tightened its bolts to make the most of the provisioned compute time. The Seven Bridges Platform understands and implements tools, workflows, jobs and execution as prescribed by the [Common Workflow Language (CWL)](http://docs.sevenbridges.com/docs/sdk-overview#section-the-common-workflow-language). ##The need for optimisation Different tools often require different computation instances to operate efficiently. When you build your own workflow or bring your own tools to the Platform, there are parameters you can tune to affect how jobs are executed. The Platform scheduling algorithm seeks the best combination of instances for a given task. However, experience shows there is no silver bullet. An algorithm will never be able to find the most efficient solution for every combination of inputs and tools. The trick is to match the available compute resources with the resource requirements of your analysis. Execution control can help you optimize your apps in a number of scenarios: * **Reduce task execution time:** [Parallelizing multiple executions of the same tool](doc:about-parallelizing-tool-executions) can save you precious time if applied to critical tools in your workflow. * ** Lower task cost: **Having a good match between tool resource requirements and provisioned computational instances means that you will only be paying for compute resources that are being used. * **Ensure sufficient resources:** When developing a tool using the [Platform SDK](doc:sdk-overview), one of the key steps is specifying the computational resources (CPU cores and RAM) that the tool needs for proper operation. ##Controlling task execution The resource requirements of an analysis are given by the resource requirements (CPU and RAM) of [the tools involved](doc:about-controlling-task-execution). These parameters are used by the [Scheduling Algorithm](doc:about-task-execution#section-scheduling) when deciding how to distribute jobs across available instances. Those parameters are called *execution hints*. You can control the compute resources available for your tool executions by [specifying the type of instance](doc:about-controlling-task-execution#section-controlling-task-execution-via-instance-type) to be used and/or the [maximum number of such instances](doc:about-controlling-task-execution#section-controlling-task-execution-via-maximum-number-of-parallel-instances) that can be run in parallel. See [this blog post](blog:making-efficient-use-of-compute-resources) for examples on how to make efficient use of computational resources. [block:callout] { "type": "danger", "body": "Beware of mismatches between tool resource requirements and the resources available on the provisioned computation instance. Your task will fail if a tool requires more resources than the provisioned computation instance.", "title": "Insufficient resources" } [/block] ###Controlling task execution via instance type You generally want to pick an instance that leaves the least CPU and memory to waste, given the jobs predicted to run as part of your task. [This blog post](blog:making-efficient-use-of-compute-resources#section-choosing-the-best-compute-instance-for-your-analysis) provides a detailed example of how to determine the most suitable instance type. To set the instance type, set the [`sbg:AWSInstanceType`](doc:list-of-execution-hints) or [`sbg:GoogleInstanceType`](doc:list-of-execution-hints) hint at [workflow level](doc:set-execution-hints-at-workflow-level). ####The hierarchy of instance type hints While instance type hints can be set at several levels (workflow, node, and tool), there is a hierarchy that determines which of the settings takes priority during execution. An instance type specified at workflow level will override any instance type hints set for any of its nodes or any of the tools within that workflow. An instance type specified for a tool will only be considered when the tool is run on its own. When the tool becomes part of a workflow, its instance hint gets overridden by instance hints set at node level, which in turn get overridden by the workflow-level instance hints. ####Defaults The current default instance the Scheduling Algorithm will attempt to use for execution is `c4.2xlarge`, which has 8vCPUs and 15 GB RAM. ###Controlling task execution via maximum number of parallel instances Configuring the number of instances running at once for a single task is handy when [parallelizing](doc:about-parallelizing-tool-executions) (scattering) several executions of the same tool. The Seven Bridges bioinformaticians exploit this technique when tuning workflows in Public Apps. To set the maximum number of instances, set the [`sbg:maxNumberOfParallelInstances`](doc:list-of-execution-hints) hint at [workflow level](doc:set-execution-hints-at-workflow-level). ####Defaults The default value for `sbg:maxNumberOfParallelInstances` is 1.