.. Copyright (c) 2008-2016 OpenShot Studios, LLC (http://www.openshotstudios.com). This file is part of OpenShot Video Editor (http://www.openshot.org), an open-source project dedicated to delivering high quality video editing and animation solutions to the world. .. OpenShot Video Editor is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. .. OpenShot Video Editor is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. .. You should have received a copy of the GNU General Public License along with OpenShot Library. If not, see . .. _scaling_ref: Performance / Scaling ===================== OpenShot Cloud API was designed with scalability in mind. When configured correctly, you can achieve huge and rapid scalability on cloud infrastructure. This topic can get very tricky though, so feel free to send an email to discuss your specific application: cloud-support@openshot.org. Performance ----------- The performance of video exports/renders is highly dependent on CPU and Memory. Ideally, you want as many CPUs and as much memory as you can afford. A general rule of thumb is the average render takes about as long as the video length. A 60 second 1080p video will usually take around 60 seconds to render. If you have a more powerful EC2 instance type, and/or are rendering at a lower resolution, it will be faster than real-time. On a slower instance type, it will be slower than real-time. Instance Types -------------- There are a large number of possible cloud instance types and sizes, and some of them perform very differently than others. Here are some recommendations that work well for OpenShot Cloud API. In general, we recommend the compute optimized instance types. **AWS** Instance Types =========== ============== c5.xlarge Lower cost, but slower than real-time performance c5.2xlarge More expensive, but good performance (maintains real-time performance for most exports) c5.4xlarge Best performance and very expensive (faster than real-time performance) c5.9xlarge Similar performance to c5.4xlarge, but twice the price. Not recommended for that reason. =========== ============== **Azure** Instance Types =========== ============== F2s v2 Lower cost, but slower than real-time performance F4s v2 More expensive, but good performance (maintains real-time performance for most exports) F8s v2 Best performance and very expensive (faster than real-time performance) F16s v2 Similar performance to c5.4xlarge, but twice the price. Not recommended for that reason. =========== ============== Auto Scaling ------------ Both AWS and Azure offer auto scaling solutions (each with slightly different features, but generally they work about the same). Configure a **sever** and a separate **worker**. Configure auto scaling on the **worker** instance, and base it off of average CPU > 50% for X minutes, or the number of messages in the Queue (SQS or Azure Queue Storage). As your video export jobs queue up, more and more **worker** instances will come online and help render jobs. Once the workload has ended and the queue is empty, the pool of **worker** instances can shrink back down to 1 (or even 0). This is an effective strategy for managing costs (i.e. keeping as few workers running as possible), but quickly scaling up as needed. **AWS** Auto Scaling Instructions =========== ============== Step 1 Configure a new worker instance (use the same admin credentials when prompted during `config-openshot-cloud`) Step 2 **Create an Image** (AMI) of the new worker instance (use a descriptive name, such as "OpenShot Cloud Worker Image") Step 3 Create a **Launch Configuration** (choose the AMI from step 2, choose the type of instance, and name descriptively) Step 4 Create an **Auto Scaling Group** (choose the Launch Configuration from step 3 and name descriptively) Step 5 Set the **Scaling Policies** to determine when to scale up and down (either `# of SQS messages` or `average CPU %` will work) Step 6 If you choose a SQS scaling policy, choose a Policy type of `Simple Scaling`, which is compatible with CloudWatch ALARMS. Step 7 Create 2 CloudWatch Alarms, and connect them to your AutoScaling Group (use SQS:ApproximateNumberOfMessagesVisible for your logic). Step 8 Create 2 Scaling Policies: `SQS Scale Up`: If > 1 message over X minutes. And `SQS Scale Down`: If < 1 message over X minutes. =========== ============== **Azure** Scale Set Instructions =========== ============== Step 1 Configure a new worker virtual machine (use the same admin credentials when prompted during `config-openshot-cloud`) Step 2 **Capture Image** of the new worker virtual machine (use a descriptive name, such as "openshot-cloud-worker-image") Step 3 Because of Azure limitations in the portal, we will need to use the `az` CLI for the remaining steps. Run the following script to create a scale set with the captured worker image. Replace the following place holders: `{RESOURCE-GROUP-NAME}`, `{CAPTURED-IMAGE-URI}`, `{SUBSCRIPTION-ID}`. Step 4 `az vmss create -n openshot-worker-set --computer-name-prefix openshot-worker -g {RESOURCE-GROUP-NAME} --instance-count 1 --image {CAPTURED-IMAGE-URI} --plan-name openshot-cloud-001 --plan-product openshot-cloud-api --plan-publisher openshotstudiosllc --subscription {SUBSCRIPTION-ID}` Step 5 This will successfully create a new scale set, based on your captured worker image. Now we can use the Azure portal to update the rules and settings of this scale set. Step 6 Scaling: Choose `Scale based on a metric`. Click `Add Rule`: choose metric source: `storage queue`, select your storage account, and choose the OpenShot queue. Step 7 Create 2 Rules: `Queue Scale Up`: If > 1 message over X minutes, increase by 1 instance. And `Queue Scale Down`: If < 1 message over X minutes, decrease by 1 instance. Step 8 You can also set the minimum and maximum instance counts (to help manage costs) =========== ============== **Azure** Using captured images (of a marketplace product) =========== ============== Step 1 If you use **Capture Image** on an OpenShot Cloud API virtual machine, it requires extra steps to utilize this image correctly. Step 3 Because of Azure limitations in the portal, we will need to use the `az` CLI for the remaining steps. Run the following script to create a virtual machine of the captured image. Replace the following place holders: `{LOCATION-NAME}, {RESOURCE-GROUP-NAME}`, `{CAPTURED-IMAGE-URI}`, `{SUBSCRIPTION-ID}`. Step 4 `az vm create --name openshot-worker-2 --location {LOCATION-NAME} --image {CAPTURED-IMAGE-URI} --plan-name openshot-cloud-001 --plan-product openshot-cloud-api --plan-publisher openshotstudiosllc -g {RESOURCE-GROUP-NAME} --subscription {SUBSCRIPTION-ID}` Step 5 This will successfully create a new virtual machine, based on your captured marketplace image. =========== ============== Storage Scaling --------------- Both AWS and Azure provide auto scaling file systems. In order to provide enough storage to meet you needs, these auto-scaling file systems are perfectly suited to work with OpenShot Cloud API. This is an **optional** step, and only recommended when you have very large amounts of files, very large video projects, or high numbers of videos being rendered. **AWS** Instructions =========== ============== Step 1 Create an EFS file system (default settings are fine) named `OpenShotCloudStorage`. Step 2 Choose **Customize** and select `Max I/O` under `Performance Mode` (if you expect to have many Workers) Step 3 **Disable Encryption** on data at rest (unless desired) Step 4 Once created, copy the ID of your EFS (i.e. fs-2180ccd3), and install the mount (instructions below) Step 6 Copy the existing video folders into the mount (instructions below). That should create 3 folders in /mnt/efs/. Step 7 Verify the 3 folders are copied into the /mnt/efs (`files, output, and temp`) Step 8 Remove and replace the existing folders with symbolic links (instructions below) Step 9 Reboot the instance, and verify the symbolic links and EFS mount still work =========== ============== .. code-block:: bash # Install EFS Helper Utility git clone https://github.com/aws/efs-utils cd amazon-efs-utils ./build-deb.sh sudo apt install ./build/amazon-efs-utils*deb # Mount EFS (use your own file system ID) sudo mkdir -p /mnt/efs sudo mount -t efs fs-2180ccd3:/ /mnt/efs # Auto Mount EFS (add this line to fstab, use your own file system ID) sudo nano /etc/fstab fs-2180ccd3:/ /mnt/efs efs _netdev,tls,iam 0 0 # Copy media files into EFS cd ~/api/video/ sudo rm -r * (be sure you are in the correct folder) # Replace media folders with symbolic links # Verify your media files are located at /mnt/efs/ cd ~/api/video/ ln -s /mnt/efs/files/ files ln -s /mnt/efs/output/ output ln -s /mnt/efs/temp/ temp Worker Concurrency ------------------ Each worker is configured by default to process 2 video exports concurrently. While this improves speed and latency in many cases, it can also cause less capable instance types to run out of resources (Memory, CPU) causing higher failure rates. The following instructions can be used to configure a different # of concurrent export jobs: .. code-block:: bash sudo nano ~/api/video_queue/supervisor/cloud-worker.conf # Change `numprocs=2` to the desired amount of concurrent exports # CTRL+X, Save sudo supervisorctl reload sudo supervisorctl restart all Hardware Acceleration --------------------- OpenShot Cloud API supports NVIDIA hardware encoding, if you are using a compatible instance type. You will also need to install a driver package: `sudo apt-get install nvidia-driver-390` and use **h264_nvenc** as your video codec. However, the increase in encoding performance is often out-paced by the cost of accelerated instances. But feel free to use these instructions and guidelines to test hardware acceleration for your specific application. =========== ============== g3s.xlarge Lower cost, hardware accelerated, slower than real-time g3.4xlarge More expensive, hardware accelerated, but good performance (maintains real-time performance for most exports) c5.4xlarge NOT hardware accelerated but similar performance for almost half the price. =========== ============== Instance Statistics ------------------- The following statistics were collected using an identical 60 second 1080p project, exported using a variety of instances, to compare speed and instance cost. Your specific project might not be 100% comparable, but this should give you an idea on performance and cost. After crunching the numbers, we still recommend a `c5.xlarge` instance type for the best budget option (but a bit slow on performance) and a `c5.4xlarge` for best performance (more expensive). .. image:: images/instance-stats.png