diff --git a/source/adminguide/events.rst b/source/adminguide/events.rst index b3444730e0..53225e8fb2 100644 --- a/source/adminguide/events.rst +++ b/source/adminguide/events.rst @@ -400,7 +400,7 @@ Procedure #. Log in as administrator to the CloudStack UI. -#. In the left navigation, click Events. +#. In the left navigation, click Activity, then Events. #. Perform either of the following: diff --git a/source/adminguide/index.rst b/source/adminguide/index.rst index 040ad1cd67..25abb85a5e 100644 --- a/source/adminguide/index.rst +++ b/source/adminguide/index.rst @@ -187,6 +187,7 @@ Events and Troubleshooting :maxdepth: 4 events + jobs troubleshooting diff --git a/source/adminguide/jobs.rst b/source/adminguide/jobs.rst new file mode 100644 index 0000000000..5bfa4dcae2 --- /dev/null +++ b/source/adminguide/jobs.rst @@ -0,0 +1,257 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information# + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + + +.. _working-with-jobs: + +Working with Asynchronous Jobs +============================== + +Every API command that takes time to complete, such as starting an Instance, migrating it +or taking a snapshot, runs as an asynchronous job: the command returns a job ID at once and +the work goes on in the background on one of the management servers. The result is fetched +with ``queryAsyncJobResult`` (see the :ref:`Developers Guide `). Since +version 24.0 the jobs themselves can be listed and inspected, with their events and the +resource they act on, and a running job can be cancelled, including the work it has started +on the hypervisor. + +A job is in one of four states: + +.. cssclass:: table-striped table-bordered table-hover + +=============== ====== ====================================================================== +Status Value Meaning +=============== ====== ====================================================================== +In progress 0 The job is waiting or running. While its cancellation has been + requested and not yet carried out, the UI shows it as *Cancelling*. +Succeeded 1 The job completed; its result is the response of the command. +Failed 2 The job failed; the result carries the error. +Cancelled 3 The job was cancelled and its work was stopped or undone. +=============== ====== ====================================================================== + +A cancelled job's result has the shape of a failure: result code 530 and an error text that +names who cancelled it and through which management server, for example ``Cancelled by admin +via mgmt2``. + + +Listing Jobs +------------ + +#. Log in to the CloudStack UI. + +#. In the left navigation, click Activity, then Jobs. The jobs of the account are listed; + administrators see the jobs of every account. + +By default only pending jobs are shown. The status filter switches to the jobs that are +*Cancelling*, *Succeeded*, *Failed* or *Cancelled*, or to all jobs, and for completed jobs +a period filter (last hour, day or week) limits the listing. The filter dialog narrows the +list by domain, account and resource type. Each row shows the command, the status, the +resource the job acted on (as a link where the resource still exists), the management +server, the creation and completion times and the result; the result column is truncated +and can be copied in full from the row. + +Click a job to open its details: the command, the user and account that submitted it, the +resource, the management servers that initiated, executed and completed it, the times, the +full result, and, while its cancellation is in progress, the reason recorded for the cancel. +The Events tab of the job lists the events it raised, and the Cancel action is available to +administrators. + +Jobs are also listed on the resource they act on: the Jobs tab of an Instance, a Volume, a +Volume Snapshot and an Instance Snapshot shows that resource's jobs, and the Jobs tab of a +Management Server shows the jobs that server executed. The same status and period filters +apply there. + +Completed jobs are kept for ``job.expire.minutes`` (default one day) and then purged, so the +listing of succeeded, failed and cancelled jobs covers that window. + +With the API, ``listAsyncJobs`` takes these parameters in addition to the usual account, +domain, keyword and paging ones: + +.. cssclass:: table-striped table-bordered table-hover + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - Parameter + - Description + * - ``id`` + - The ID of a single job, returned whatever its status. + * - ``jobstatus`` + - Comma-separated statuses to list, as names (``IN_PROGRESS``, ``SUCCEEDED``, ``FAILED``, + ``CANCELLED``) or numbers (0-3). Only pending jobs are listed by default. + * - ``cancelrequested`` + - ``true`` lists only the jobs whose cancellation has been requested. + * - ``startdate``, ``enddate`` + - Jobs created on or after, or on or before, the given date. + * - ``duration`` + - Jobs started or completed within that many hours up to now. + * - ``resourceid``, ``resourcetype`` + - The resource the job acts on, as the job's instance. + * - ``managementserverid`` + - The management server that executed the job. + +The response carries, for each job, the command, the status and result, the resource +(``jobinstancetype``, ``jobinstanceid``, ``jobinstancename``), the management servers +(``initmanagementservername``, ``executingmanagementservername``, +``completemanagementservername``), ``cancellable`` (whether ``cancelAsyncJob`` applies to the +job's command), ``cancelrequested`` and, while the cancel is in progress, ``cancelreason``. + + +Jobs and Events +--------------- + +The events an asynchronous command raises are linked to its job: the *Scheduled* and +*Started* events the API server writes when the job is submitted, and the *Completed* event +written when it ends. The Events tab of a job shows them, and ``listEvents`` takes a ``jobid`` +parameter to fetch them; the Events view can be filtered by job ID as well. A cancel request +raises a ``JOB.CANCEL`` event on the job, whether it was carried out or refused, that names +the user who asked and the management server that took the request. + + +Cancelling a Job +---------------- + +An administrator can cancel a job that is in progress, from the Jobs list, the job's details +or the resource's Jobs tab, or with ``cancelAsyncJob jobid=``. Cancelling means more than +marking the job: the work the job has started on the hypervisor is stopped where the +hypervisor allows it, what the job had already done is undone, and the job ends *Cancelled*. +Where the work cannot be stopped, the request is refused with the reason, and the job is +left to finish as it would have. + +A cancel is accepted only when every step the job currently runs can be stopped, so the +outcome is known before anything is changed: + +#. The command must be cancellable. The UI disables Cancel for the jobs of other commands, + and ``listApis`` and ``listAsyncJobs`` report ``cancellable`` so that an API client knows + in advance. A job whose command is not cancellable can still be cancelled while all its + work is still waiting in the Instance's queue and nothing has started; that frees the + queue for the Instance's other jobs. + +#. The job must still be in progress. + +#. The hypervisor must be able to stop the work the job has in flight. The management server + asks the host or agent before changing anything. + +#. The stop must actually succeed. A task that completed first, or that the hypervisor + could not stop, leaves the job to finish normally. + +The commands that are cancellable, and the work each hypervisor can stop: + +.. cssclass:: table-striped table-bordered table-hover + +.. list-table:: + :header-rows: 1 + :widths: 34 22 22 22 + + * - Command + - VMware + - XenServer/XCP-ng + - KVM + * - ``startVirtualMachine`` (first start) + - template clone or copy + - template copy + - no + * - ``migrateVirtualMachine``, ``migrateVirtualMachineWithVolume`` + - live migration + - live migration + - live migration + * - ``migrateVolume`` + - volume copy + - volume copy + - live migration only + * - ``createVolume`` (from a snapshot) + - copy on the SSVM + - copy on the host + - no + * - ``createSnapshot`` + - backup on the SSVM + - backup on the host + - no + * - ``createVMSnapshot`` + - vCenter snapshot + - XenAPI snapshot + - no + * - stop, reboot, destroy, attach, detach, delete and revert commands + - no + - no + - no + +Power operations, attach and detach, deletes and reverts are listed as cancellable so that a +request can be made, but no hypervisor can stop them once they run: vCenter reports its power +tasks as not cancelable, XenServer runs them as synchronous calls, and KVM has nothing to +abort, so such a request is refused unless it arrives before the work started. On KVM, a +cancelled migration aborts the libvirt job; snapshots and copies on KVM cannot be stopped. + +When a cancel is accepted: + +- The job is marked as cancel requested and keeps running until its work is stopped; the + listing shows it as *Cancelling* with the reason. + +- The management server executing the job stops the commands it holds for it; with several + management servers, each one stops what it holds, so a migration whose work runs on + another server is stopped there too. + +- The job's work is undone as after a failure: a cancelled migration leaves the Instance or + volume where it was and removes what was created on the target, a cancelled snapshot or + Instance snapshot is removed, a cancelled volume creation expunges the volume. On VMware, + an Instance whose snapshot task was stopped has its disks consolidated. + +- The job ends *Cancelled*. A job whose work completed before the cancel took effect ends + *Succeeded* with ``cancelrequested`` set, so the records always match what happened. + +Refusals and what they mean: + +- ``Cannot cancel, job is not cancellable.`` The command does not support cancellation. + +- ``Cannot cancel job-N, its work cannot be stopped at this point: (host )`` + The command supports it, but the step now running cannot be stopped; the reason is what + the hypervisor reported, for example ``vCenter reports task PowerOffVM_Task(...) as not + cancelable`` or ``no vCenter task or transfer is running for StopCommand``. Try again + later, or let the job finish. + +- ``Cannot cancel job-N, the operation it is running could not be stopped.`` The stop was + attempted and the hypervisor did not stop the work; the job finishes normally. + +- ``Cannot cancel, job-N is not running. Current job status is .`` The job had + already ended. + +.. note:: + Usage records are written by the operations themselves, not by the job status, so a + cancelled job leaves them matching what happened: a job that ends *Cancelled* has had its + work undone, and the usage record its allocation opened is closed; a job whose work + completed despite the request ends *Succeeded* with the normal events. If a cancel is + refused because the hypervisor finished the work first, nothing needs reconciling. + + +Usage Server Jobs +----------------- + +The runs of the Usage Server are listed on the Usage Jobs tab of a Management Server (see +:ref:`listing-usage-jobs`) and with ``listUsageJobs``. + + +Global Settings +--------------- + +.. cssclass:: table-striped table-bordered table-hover + +========================= =========================================================================== +Setting Description +========================= =========================================================================== +job.expire.minutes Time in minutes for which completed jobs are kept and listed. Default 1440. +job.cancelled.interval Interval in seconds at which every management server checks for jobs whose + cancellation was requested and stops the work it holds for them. Default 1. +========================= =========================================================================== diff --git a/source/adminguide/management.rst b/source/adminguide/management.rst index bdaedac062..de28b4b382 100644 --- a/source/adminguide/management.rst +++ b/source/adminguide/management.rst @@ -215,7 +215,8 @@ Administrator Alerts The system provides alerts and events to help with the management of the cloud. Alerts are notices to an administrator, generally delivered by e-mail, notifying the administrator that an error has occurred in the -cloud. Alert behavior is configurable. +cloud. Alert behavior is configurable. In the UI, alerts are listed under +Activity, then Alerts, in the left navigation. Events track all of the User and administrator actions in the cloud. For example, every Guest Instance start creates an associated event. Events are @@ -631,6 +632,13 @@ Administrators are able to view the statistics and peers information of manageme |management-server-peers.png| +#. Navigate to the "Jobs" tab to see the asynchronous jobs this management server executed, + pending ones by default, with the same status and period filters as the Jobs view under + Activity (see :ref:`working-with-jobs`); a running job can be cancelled from here. + +#. Navigate to the "Usage Jobs" tab to see the runs of the Usage Server (see + :ref:`listing-usage-jobs`). + Global settings for management servers ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/source/adminguide/usage.rst b/source/adminguide/usage.rst index 9363d7900f..6ef679f9c9 100644 --- a/source/adminguide/usage.rst +++ b/source/adminguide/usage.rst @@ -163,6 +163,21 @@ defined by the EST (America/New\_York) time zone. run. +.. _listing-usage-jobs: + +Listing Usage Jobs +~~~~~~~~~~~~~~~~~~ + +Since 24.0 the runs of the Usage Server can be checked without reading its +log. In the UI, open a Management Server under Infrastructure and go to its +"Usage Jobs" tab: each run is listed with the usage server that ran it, the +job type (recurring or single), whether it was scheduled, its start and end +dates, execution time, outcome and heartbeat, for the last hour, day or +week or all runs. With the API, ``listUsageJobs`` takes ``usageserver`` (host +name or IP), ``startdate`` and ``enddate``, and ``duration`` (hours up to +now). Both are available to administrators. + + Setting Usage Limits -------------------- diff --git a/source/developersguide/dev.rst b/source/developersguide/dev.rst index 67aceb1415..d0672c6cb0 100644 --- a/source/developersguide/dev.rst +++ b/source/developersguide/dev.rst @@ -607,6 +607,8 @@ differ from synchronous commands by the following: ID. +.. _async-commands: + Job Status ~~~~~~~~~~ @@ -626,6 +628,18 @@ values: - 2 - Job has failed to complete. Please check the "jobresultcode" tag for failure reason code and "jobresult" for the failure reason. +- 3 - Job was cancelled (since 24.0). "jobresult" carries who cancelled + it and through which management server, with "jobresultcode" 530. + +Since 24.0 the jobs of an account can also be listed with ``listAsyncJobs``, +by status, period, resource and management server, including completed +ones; the response reports ``cancellable`` for each job. A job in progress +can be cancelled by an administrator with ``cancelAsyncJob``, which stops +the work the job started on the hypervisor where that is possible and is +refused with the reason otherwise. ``listApis`` reports ``cancellable`` +per command. See the Administration Guide on working with asynchronous +jobs. + Example ~~~~~~~ @@ -768,6 +782,8 @@ Event Types | | | | | VM.RESTORE | +-------------------+--------------------------------------------------------+ +| Job | JOB.CANCEL | ++-------------------+--------------------------------------------------------+ | Domain Router | ROUTER.CREATE | | | | | | ROUTER.DESTROY |