Abiquo Documentation Cookies Policy

Our Documentation website uses cookies to improve your experience. Please visit our Cookie Policy page for more information about cookies and how we use them.


Documentation

Skip to end of metadata
Go to start of metadata

Accounting Services Integration

 

Contents

Change history

Changes to Accounting in Abiquo 3.10

Add new accounting parameters with support for:

  • VirtualMachine-vcpuon

  • VirtualMachine-vcpuoff

  • VirtualMachine-vramon

  • VirtualMachine-vramoff

  • Firewall

  • LoadBalancer

  • VirtualMachine-backup

 

  • Abiquo 3.8
    • Add new functionality for consumption of tiered datastore storage
    • New table accounting_event_ds_storage
    • New parameter to enable tiered datastore storage: VirtualMachine-vtierhd
    • Remove restriction on number of storage tiers for external storage volumes
  • No changes in Abiquo 3.6
  • Abiquo 3.4
    • Event table for VM Repository:
      • Deprecated field idImageFormatName
      • Since v3.4, there is no accounting of virtual machine template conversions
  • No changes in Abiquo 3.2
  • No changes in Abiquo 3.1
  • Abiquo 3.0
    • Change accounting tables to resize VLAN name to 255 chars from 40 chars
    • Changes to public cloud region accounting:
      • add floating public IPs
      • remove reserved servers

 

1. Introduction

Accounting features are essential for the development of pay per use models for cloud computing. Therefore, Abiquo incorporates a set of services to measure which resources are reserved or consumed, and when they are reserved or consumed.

This is a basic overview of the Abiquo accounting system:

  1. When users reserve or use resources on the cloud platform, Abiquo records these resources in the accounting event tables in the kinton_accounting database. These are the master accounting tables and you should back them up to prevent data loss. 
  2. The Update Accounting stored procedure from the kinton database should run periodically to upate the accounting event detail table in the kinton_accounting database. You can configure which resources to account and the period for accounting them, and so on.
  3. You should replicate the accounting data to allow user access through accounting queries.

The following diagram illustrates this overview.

 

 

2. Configure Accounting environment

This section describes how to configure the cloud platform environment for accounting.

2.1. Create backup of master accounting tables

The Abiquo Accounting data is stored in the kinton_accounting database. Ensure that these master accounting tables are included in your Abiquo backup.

accounting_event_ips
accounting_event_pm
accounting_event_repository
accounting_event_storage
accounting_event_vlan
accounting_event_vm
accounting_event_ds_storage
accounting_event_vm_on
accounting_event_vm_off
accounting_event_backup
accounting_event_firewall
accounting_event_loadbalancer

 

2.2. Replicate accounting database

It is important to avoid running queries on the accounting_event_detail table. Excessive queries on that table may negatively affect the Abiquo Server.

For an example of database replication in Abiquo, see Kinton Database replication for Jasper Server

You could consider running a replication every hour after the UpdateAccouting process completes. This way, customers can see their resource consumption on a per hour basis.

If you do not wish to implement a replication process as described, you can implement a process to export the table to a file that is imported into another database once a day, for example.

3. Configure Accounting

This section describes how to configure accounting with:

  • accounting parameters
  • periodic stored procedure runs

To retrieve the current accounting parameters configuration, open mysql and run the following commands.

use kinton_accounting
SELECT * FROM accounting_parameters;
+---------+-------------------------------+------------+
| idParam | paramName                     | paramValue |
+---------+-------------------------------+------------+
|       1 | AccountingEnabled             | 1          |
|       2 | DeleteRegEventsUseSPParam     | 0          |
|       3 | DeleteRegEventsDeleteHours    | 26280      |
|       4 | MaximumPeriodsToFirstInit     | 1          |
|       5 | MaximumPeriodsToProcess       | 24         |
|       6 | AccountPeriodSize             | HOUR       |
|       7 | AccountPeriodGranularity      | HOUR       |
|       8 | LoggingLevel                  | INFO       |
|       9 | VirtualMachine-vcpu           | 1          |
|      10 | VirtualMachine-vram           | 1          |
|      11 | VirtualMachine-vhd            | 1          |
|      12 | ExternalStorage               | 1          |
|      13 | IPAddress                     | 1          |
|      14 | VLAN                          | 1          |
|      15 | VirtualMachine-hypervisorType | 1          |
|      16 | VirtualMachine-haHosted       | 0          |
|      17 | ReservedPhysicalMachine-cpu   | 0          |
|      18 | ReservedPhysicalMachine-ram   | 0          |
|      19 | RepositoryStorage             | 0          |
|      20 | VirtualMachine-antiAffinity   | 0          |
|      21 | VirtualMachine-vtierhd        | 0          |
+---------+-------------------------------+------------+
21 rows in set (0.00 sec)

To view the current cron configuration that will periodically run the stored procedures, run the following command

# cat /etc/cron.d/abiquo-accounting 

# Runs every hour
0 * * * * root mysql -uroot -hlocalhost -e "CALL kinton.UpdateAccounting();"
# Runs every Sunday at 12:00 and deletes records older than a week.
0 12 * * 0 root mysql -uroot -hlocalhost -e "CALL kinton.DeleteOldRegisteredEvents(168);"

 

3.1. Enable or disable accounting of resources

Although Abiquo tracks all resources in the Accounting event tables, Administrators can configure the specific resources that are accounted for billing and reporting.

3.1.1. Description of resources

Abiquo stores this data about the use of these resources in the Accounting events tables. Then Administrators can configure which resources are accounted in the Accounting event detail table.

Resource

Description

Unit

Billing Type

Association

DCPCR

Cores

Virtual cores in virtual machines created by users

Virtual cores

Consumption

Virtual machine

(tick)(tick)

Cores ON

Virtual cores in virtual machines deployed and running

Virtual cores

Consumption

Virtual machine

(tick)(tick) (star) v3.10

Cores OFF

Virtual cores in virtual machines deployed and stopped

Virtual cores

Consumption

Virtual machine

(tick)(tick)  (star) v3.10

RAM

RAM in virtual machines created by users

RAM in megabytes

Consumption

Virtual machine

(tick)(tick)

RAM ON

RAM in virtual machines deployed and running

RAM in megabytes

Consumption

Virtual machine

(tick)(tick)  (star) v3.10

RAM OFF

RAM in virtual machines deployed and stopped

RAM in megabytes

Consumption

Virtual machine

(tick)(tick)  (star) v3.10
Local Storage

Disk space connected to the hypervisor as datastores for virtual machine disks,
including secondary hard disks

Bytes

Consumption

Virtual machine

(tick)(tick)

External Storage

Disk space for use as external volumes created by users on virtual machines

Bytes

Reservation

Virtual machine or
Independent

(tick) 

Public IPs

IP addresses with Internet access for use by virtual machines

Number of IP addresses

Reservation

Virtual Machine or
Independent

(tick)(tick) (star) v3.0

VLANs

  • Private VLANs for virtual machines
  • Unmanaged, external and public network VLANs are not accounted
  • A private VLAN is automatically created when a user creates a VDC

Number of VLANs

Reservation

Virtual Datacenter

(tick)(tick)
FirewallFirewalls created by the userNumber of FirewallsReservationVirtual Datacenter(tick)(tick)  (star) v3.10
Load BalancerLoad balancers created by the userNumber of Load BalancersReservationVirtual Datacenter(tick)(tick)  (star) v3.10

Hypervisor Type

The hypervisor type for the virtual machine

Hypervisor type

Consumption

Virtual machine

(tick)(tick)

Cost Code

Optional cost codes for virtual image templates

Cost code

Consumption

Virtual machine

(tick)(tick)
BackupBackup policies applied to virtual machines deployedBackup policyConsumptionVirtual machine(tick) 
HA Hosted VMVirtual machine deployed on a high-availability rackTrue or falseConsumptionVirtual machine(tick) 
Reserved Physical
Machine Cores
CPU of each server that has been explicitly reserved for an enterprise.
Recorded even if the server is not being used.
Virtual coresReservationEnterprise(tick)(minus) v3.0
Reserved Physical
Machine RAM
RAM of each server that has been explicitly reserved for an enterprise.
Recorded even if the server is not being used.
RAM in megabytesReservationEnterprise(tick)(minus) v3.0
Repository Storage

Repository space is used when:

  1. Customers create a VM instance
  2. Customers upload disk images into the template repository
  3. Abiquo converts instances and templates to other hypervisor formats

Abiquo accounts for space used by instances and templates (cases 1 and 2).
If you wish to account conversions, create a custom metric (case 3).

Bytes used by each
template
ConsumptionEnterprise(tick) 
Anti-affinity VMVirtual machine deployed in an anti-affinity layer separate from other VMs in the same layerTrue or falseConsumptionVirtual machine(tick) 
Tiered local
datastore storage
Local storage on hypervisor datastores by datastore tiers (including primary and secondary
non-persistent hard disks) 
Bytes used by each diskConsumptionVirtual machine(tick) (star) v3.8 
CustomSee Custom Accounting Resource DefinitionsUser definedUser definedUser defined(tick)

(tick)

Key

  • (tick) - Resource accounted
  • (star) - New in version
  • (minus)  - Deprecated in version
  • Blue shading - disabled by default
  • Pink shading - custom resource
  • DC - Datacenter
  • PCR - Public cloud region

3.1.2. Enable or disable resources

To select which accounting resources Abiquo will record in the accounting_event_detail table, set the accounting parameters for the resources. For example, if you do not wish to bill for VLANs, disable them in accounting so that Abiquo does not record them in the accounting event detail table. This will reduce the space used by the database and increase the performance of the accounting and billing processes.

Set the parameters to enable or disable resource accounting in the kinton_accounting.accounting_parameters table. Each parameter has the same name as the resource it represents. The parameter values are: enable = 1 and disable = 0. Resources that are disabled by default are shaded in blue.

Parameter Name

Enabled/Disabled
Default Value

VirtualMachine-vcpu

1

VirtualMachine-vcpuon1
VirtualMachine-vcpuoff1
VirtualMachine-vram

1

VirtualMachine-vramon

1

VirtualMachine-vramoff

1

VirtualMachine-vhd

1

ExternalStorage

1

IPAddress

1

VLAN

1

Firewall1
LoadBalancer1
VirtualMachine-hypervisorType

1

VirtualMachine-backup1

VirtualMachine-haHosted

0

ReservedPhysicalMachine-cpu

0

ReservedPhysicalMachine-ram

0

RepositoryStorage

0

VirtualMachine-antiAffinity0
VirtualMachine-tierhd0

 

Example of enabling a resource

An example of how to enable accounting of VirtualMachine-antiAffinity for virtual machines deployed in anti-affinity layers.

First retrieve the record from the accounting_parameters table to check the name and value of this parameter.

use kinton_accounting
SELECT * FROM accounting_parameters WHERE paramName="VirtualMachine-antiAffinity";
+---------+-----------------------------+------------+
| idParam | paramName                   | paramValue |
+---------+-----------------------------+------------+
|      20 | VirtualMachine-antiAffinity | 0          |
+---------+-----------------------------+------------+
1 row in set (0.00 sec)

Then update the value of the parameter to 1.

UPDATE accounting_parameters SET paramValue=1 WHERE paramName="VirtualMachine-antiAffinity";

3.1.3. Configure custom resources

Administrators can also configure their own custom resources.

See Custom Accounting Resource Definitions

3.2. Set time periods for accounting of resources

In the accounting_event_detail table, Abiquo stores resource usage information by account period size and account period granularity. Always set these parameters before allowing users to work with your system.

3.2.1. Account period size parameter

For each accounting period, the Update Accounting stored procedure checks the Accounting event tables for resources that have been reserved and/or consumed in the last period by granularity and records the usage data in the Accounting event detail table.

The minimum period of consumption of a resource is always one accounting period. If a resource is active for an interval of less than one accounting period, the system automatically rounds consumption up to one accounting period. But please note that the value stored will depend on the granularity.

Many types of resources may be changed (e.g. turned on and off) during an accounting period. Abiquo stores all these events in the Accounting events tables. However, the Accounting event detail table only contains the maximum value for an accounting period.

You can set the accounting period size parameter in the kinton_accounting.accounting_parameters table.

Parameter Name: AccountPeriodSize
Default Value: HOUR
Description: Controls the size of the aggregated account period of the accounting_event_detail table, and may be one of the following values: HOUR, DAY, WEEK, MONTH.  If the value is bad or missing, the default value of 1 hour is used.

Set the account period size

An example to set the accounting period to week.

First retrieve the record from the accounting_parameters table to check the name and value of this parameter.

use kinton_accounting
SELECT * FROM accounting_parameters WHERE paramName = "AccountPeriodSize";
+---------+-------------------+------------+
| idParam | paramName         | paramValue |
+---------+-------------------+------------+
|       6 | AccountPeriodSize | HOUR       |
+---------+-------------------+------------+
1 row in set (0.00 sec)

 Update the value of the parameter to WEEK.

UPDATE accounting_parameters SET paramValue="WEEK" WHERE paramName="AccountPeriodSize";

3.2.2. Accounting period granularity

For each accounting period, the Update Accounting stored procedure checks the Accounting event tables for resources that have been reserved and/or consumed in the last period by granularity  and records the usage data in the Accounting event detail table. The value stored for each accounting period can be divided into smaller time units using the granularity. However, the granularity does not affect the minimum usage. The minimum period of usage is always determined by the accounting period.

Some examples:

  • If accounting period = DAY and granularity = HOUR, then for a single CPU VM, Abiquo would record 24 resource units for a day record. So this could represent a charge for 24 hours of vCPU usage.
  • If accounting period = DAY and granularity = DAY, then Abiquo will record 1 resource unit. So this could represent a charge for 1 day of vCPU usage.

You can set the granularity parameter in the kinton_accounting.accounting_parameters table.

Parameter Name: AccountPeriodGranularity 
Default Value: HOUR
Description: Controls the granularity of the resource units within an accounting period. Possible values are HOUR, DAY, WEEK and MONTH. If the value is bad or missing, the default value is used. The granularity must be smaller than the accounting period and the accounting period must be divisible by the granularity. The following table shows accounting periods and compatible granularity. Abiquo uses calendar months, so week granularity is not compatible with month.

Accounting Period and Granularity Compatibility Table

Compatible values

Period

 

Hour

Day

Week

Month

Granularity

Hour

(tick)

(tick)

(tick)

(tick)

 

Day

 

(tick)

(tick)

(tick)

 

Week

 

 

(tick)

 

 

Month

 

 

 

(tick)

Set the accounting period granularity

Here is an example to set the accounting period granularity to day. We assume that the account period size is set to "WEEK".

UPDATE accounting_parameters SET paramValue="DAY" WHERE paramName="AccountPeriodGranularity";

Check the updated value by using the select command.

3.2.3. Change the accounting period parameters in an existing system

To change the account period size or granularity of an existing Abiquo system, first back up and remove any existing data from the kinton_accounting accounting_event_detail table. This is essential because the accounting procedure analyzes the existing table content to determine the next accounting period of data. If you do not remove the existing data, this may cause duplicate accounting data or mis-aligned accounting periods.
 
To change the accounting period, do these steps: 

  1. Backup the kinton_accounting.accounting_event_detail table
  2. Change the accounting parameters to the desired values
    1. set the ‘MaximumPeriodsToFirstInit’ parameter to ensure that any old data is accounted correctly
  3. Delete the existing content of the accounting_event_detail table
  4. Monitor the content of accounting_event_detail to ensure the generated data meets your needs. If not, then repeat from step 2.

3.3. Configure periodic stored procedures for accounting

The accounting system should periodically run the accounting stored procedures to:

  • Record resource usage information
  • Delete old accounting data

By default Abiquo configures cron jobs in the /etc/cron.d directory in the abiquo-accounting file. Check that these cron jobs are appropriately configured for your system.

3.3.1. Record resource usage information

Abiquo processes the data in the accounting_event_"resource" tables to record the resource usage in the accounting_event_detail table. This is done by running the UpdateAccounting stored procedure.

Abiquo creates a cron job which automatically calls this procedure every hour.

   0 * * * * mysql -uroot -hlocalhost -P3306 -Proot -e "CALL kinton.UpdateAccounting ();"

3.3.2. Delete old accounting records

In current versions of Abiquo, master accounting data is stored in the accounting_event_"resource" tables. By default, old data in these tables is deleted by the DeleteOldRegisteredEvents stored procedure. This stored procedure deletes old usage records that are inactive, complete, and older than a certain number of hours. By default, the maximum age of a usage record is 26280 hours, which is 3 years of 365 days.

Set the frequency of procedure run to delete old records

Abiquo creates a cron job that will run the DeleteOldRegisteredEvents stored procedure once a week on Sunday at 12:00. Note that by default this procedure DOES NOT use the legacy parameter (by default 2160) to set the maximum record age.  

0 12 * * 0 mysql -uroot -hlocalhost -P3306 -Proot -e "CALL kinton.DeleteOldRegisteredEvents(2160)

Example of setting the maximum record age

To change the maximum record age of master accounting data in the accounting_event_"resource" tables, set the DeleteRegEventsDeleteHours parameter in the kinton_accounting.accounting_parameters table to the desired value, for example two years or 17520 days.

UPDATE accounting_parameters SET paramValue="17520" WHERE paramName="DeleteRegEventsDeleteHours";

In previous versions of Abiquo, the master accounting data was stored in the Abiquo accounting_event_detail table. Now this data is stored in the accounting_event_"resource" tables. Previously the default was to delete data from these tables every week, but now because they are master tables, it should be deleted less often, and by default it is deleted after three years.

Alternative configurations

It is also possible to set the maximum record age by setting the DeleteRegEventsUseSPParam parameter to use the parameter passed in to the stored procedure by the cron job but this configuration is not recommended.

4. Retrieve Accounting data

Use the following queries as templates for creating queries to view aggregate resource usage as required. These queries should retrieve information from the replicated accounting database.

4.1.1. Hourly Usage Sum Template Query

Template for query to view HOURLY USAGE SUM
SELECT
  v.startTime AS startTime,
  v.endTime AS endTime,
  v.idAccountingResourceType AS idAccountingResourceType,
  v.resourceType AS resourceType,
  SUM(v.resourceUnits) AS resourceUnits,
  v.idEnterprise AS idEnterprise,
  v.idVirtualDataCenter AS idVirtualDataCenter,
  v.enterpriseName AS enterpriseName,
  v.virtualDataCenter AS virtualDataCenter
FROM
  kinton_accounting.HOURLY_USAGE_MAX_VW v
WHERE
  # APPLY CUSTOM QUERY FILTERS HERE, E.G. BY DATE RANGE AS BELOW
  v.startTime >= '2012-04-16' AND v.endTime < '2012-04-17'
GROUP BY
  v.startTime,
  v.idAccountingResourceType,
  v.idEnterprise, v.idVirtualDataCenter;

4.1.2. Daily Usage Sum Template Query

Template for Query to view DAILY USAGE SUM
SELECT
  CAST(a.startTime AS DATE) AS startTime,
  CAST(a.startTime AS DATE) AS endTime,
  a.idAccountingResourceType AS idAccountingResourceType,
  a.resourceType AS resourceType,
  SUM(a.resourceUnits) AS resourceUnits,
  a.idEnterprise AS idEnterprise,
  a.idVirtualDataCenter AS idVirtualDataCenter,
  a.enterpriseName AS enterpriseName,
  a.virtualDataCenter AS virtualDataCenter
FROM
  kinton_accounting.HOURLY_USAGE_MAX_VW a
WHERE
  # APPLY CUSTOM QUERY FILTERS HERE, E.G. BY DATE RANGE AS BELOW
  a.startTime >= '2012-04-16' AND a.endTime < '2012-04-17'
GROUP BY
  CAST(a.startTime AS DATE),
  a.idAccountingResourceType,
  a.idEnterprise, a.idVirtualDataCenter;

4.1.3. Monthly Usage Sum Template Query

Template for query to view MONTHLY USAGE SUM
SELECT
  CAST((a.startTime - INTERVAL (DAYOFMONTH(a.startTime) - 1) day) AS DATE) AS startTime,
  LAST_DAY(a.startTime) AS endTime,
  a.idAccountingResourceType AS idAccountingResourceType,
  a.resourceType AS resourceType,
  SUM(a.resourceUnits) AS resourceUnits,
  a.idEnterprise AS idEnterprise,
  a.idVirtualDataCenter AS idVirtualDataCenter,
  a.enterpriseName AS enterpriseName,
  a.virtualDataCenter AS virtualDataCenter
FROM
  kinton_accounting.HOURLY_USAGE_MAX_VW a
WHERE
  # APPLY CUSTOM QUERY FILTERS HERE, E.G. BY DATE RANGE AS BELOW
  a.startTime >= '2012-04-01' AND a.endTime < '2012-05-01'
GROUP BY
  CAST((a.startTime - INTERVAL (dayofmonth(a.startTime) - 1) day) AS DATE),
  a.idAccountingResourceType,
  a.idEnterprise, a.idVirtualDataCenter;

Previously the accounting system used views, which have been maintained for legacy applications. See Abiquo accounting architecture

5. Troubleshooting

You can use the Abiquo accounting consistency check to monitor your accounting system.

See Abiquo accounting consistency check