Changes to Abiquo Billing Integration in Abiquo 3.10
Abiquo 3.10 introduces billing per datacenter based on pricing per datacenter. To use billing per datacenter, set the dc_grouping option to true.
Abiquo 3.10 introduces the following new billing usage charges:
Abiquo Billing Integration can be used to process the Abiquo Accounting data and provide it to a third party billing system, where it can be used to generate customer invoices for Abiquo usage (for example).
Abiquo Billing Integration is currently supplied as a separate stand-alone utility, which should be installed on a machine independent of other Abiquo software. It comprises the following components.
|Zuora||Takes Abiquo usage data and uploads it to a specified Zuora Billing System, where it can be used to create Zuora invoices.||Zuora Billing Integration|
|CSV||Generic connector that can be used to generate CSV files which contain Abiquo Usage data. The CSV files can then be processed by a third party billing system.||CSV Billing Integration|
|DBMS||Generic connector that can be used to populate a Database table (located in any suitable MySQL database) to contain Abiquo Usage data. |
The Database table can then be processed by a third party billing system, completely independently from the Abiquo Server.
|DBMS Billing Integration|
|Aria||Extends the CSV Connector to integrate with Aria via the UAL||CSV Billing Integration |
Aria Billing Integration.
To install and run Abiquo Billing Integration you should follow the steps below.
Abiquo Billing Integration is a separate utility so it can be installed on any machine that is able to connect to the main Abiquo MySQL database. In a test system, it can be installed on the Abiquo Server itself.
Unzip the 'abiquo-billing-redist.zip' file into suitable install location on your chosen machine.
Create a new database user that will be used by the billing integration to access the Abiquo database. The following example SQL creates a new user called 'billing' and grants the user the required database permissions:
Populate the Account Mapping table so that Abiquo Billing Integration knows which Abiquo Enterpises and VDCs it needs to generate Usage data for. See #How to Populate the Account Mapping Table for more details.
Configure the billing.properties file so that Abiquo Billing Integration knows what usage data needs to be generated and which connector to use. See #Configure Abiquo Billing Integration for more details.
Configure the appropriate connector properties file (i.e. csv.properties, dbms.properties, or zuora.properties), so that the connector knows how to process the data supplied by Abiquo Billing Integration.
Finally, run 'billing.sh' to generate usage data for the configured accounts.
To run it manually for testing, change directory to the folder where Abiquo Billing Integration is installed and run the script. For example, if you installed it in /opt/BillingIntegration, type the following commands:
To generate billing data at regular intervals, e.g. once a week, you can schedule Abiquo Billing Integration jobs using programs such as cron. If you are using cron, set the script's current working directory correctly in your cron job. For example, if you have installed the Abiquo Billing Integration into the '/opt/BillingIntegration' folder, then your crontab entry command could look like this:
Note that Abiquo Billing Integration automatically tracks which data has already been processed for each account, so usage data should not be processed multiple times.
A very important step in the billing process is to map Abiquo Enterprise/VDC IDs to customer IDs in your billing system. For example, mapping an Abiquo Enterprise ID to a Zuora Account Number.
You must enter the values in the billing_account_mapping table before you run Abiquo Billing Integration (ABI). This is because ABI uses the new Database tables created during the ABI install process to store the account mappings and identify which Enterprises and VDCs to process. Specifically, you must enter at least one row in the table for every enterprise or virtual datacenter (VDC) configured in the billing system.
A row in the Account Mapping Table contains the following information:
You will need the Abiquo Enterprise or Virtual Datacenter ID.
In this example, we will retrieve a list of all Virtual Datacenter IDs using the Abiquo API.
Enter the following command in the shell window with the appropriate values for your Abiquo Server.
In the response, you will see the ID and Name so you can identify your VDC.
For more information, see the Abiquo Developer's Guide VirtualDatacenterResource#Retrieve all virtual datacenters and EnterpriseResource#Retrieve a list of Enterprises
This value determines if the mapping is to an Abiquo Enterprise or VDC. It is the idMappingType for Enterprise or VDC from the billing_mapping_type table.
The 'billing_mapping_type' table has the following values:
For example, you would use this SQL statement to map a VDC with ID 29 to a Zuora account with an Account Number of 'MyCorporation' would be:
The following SQL statement will add ALL Enterprises with the billing attribute name ENTERPRISE.
Confirm that the billing account mappings are set by viewing the billing_account_mapping table data.
Some Billing Systems may require more than one 'attribute' to be defined. You can store these attributes in different lines in the billing_account_mapping table. All the attributes you define in this table will be supplied to the Connector being used by Abiquo Billing Integration.
After the Account mappings have been defined in the database, there are two further areas of ABI that require configuration:
This section discusses the configuration of the billing process. Connector-specific configurations are covered in a separate section for each connector.
All of the billing process (i.e. non-connector) settings are contained within the 'billing.properties' file, which should be located in the current working directory for ABI (the same location as the billing.sh file). An example billing.properties file can be found at the end of this section.
These settings control how the billing data is aggregated by time period, the DBMS which is used to obtain the usage data, and the actual billing connector (e.g. CSV connector, Zuora connector, etc.) being used.
|HOUR, DAY, WEEK, MONTH||This setting defines the period for which usage data is aggregated. The default value is 'DAY', meaning that accounting data is recorded per day. However, the value stored will depend on the granularity, which may divide the accounting period into smaller units, e.g. a granularity of HOUR, means the units for a day will be 24. By default, the Billing Integration will record usage quantities at the same granularity as the account period. However, the usage_units_per_hour parameter in the billing.properties file will override this behavior and always return the number of ‘units per hour’|
|false||true, false||By default this value is false, meaning that the billing integration will record resource usage units in the granularity configured for the Accounting tables. Set this value to ‘true’ to override the default behavior and always record usages ‘per hour’.|
|DAY||HOUR, DAY, WEEK, MONTH||The point in time for which 'new' accounts should start collecting usage data. For example, if a new account is added to the billing integration on 7 December, then an 'init_period' of 'MONTH' would mean that we try and collect data from 1 December, whereas a value of 'DAY' would mean that data is collected from 7 December.|
|CSV||CSV, DBMS, Zuora||This setting determines which connector is used by ABI to process usage data. Allows you to specify a well-known connector using a 'friendly' name. Friendly names are case insensitive, and take priority over the 'connector_class' setting.|
If the connector you want to use does not have a friendly name supported by the ABI, set the 'connector_name'. This setting MUST match the complete Java class name of the Connector, otherwise ABI will fail to run.
|These settings determine how Abiquo Billing Integration connects to the database used to retrieve the billing usage data. For example, dbms_connection_url=jdbc:mysql://127.0.0.1:3306/kinton|
|Currently only the default value of 'com.mysql.jdbc.Driver' is supported|
|Group billing by Abiquo datacenter. When set to ‘true’, this setting enables data center billing, i.e. physical data center sub-grouping. Note that you must also enable data center output in the configured billing connector. In Abiquo 3.10, set to true to use pricing per datacenter. If false, Abiquo will use the pricing from the first datacenter in the pricing model|
|Enable filtering options to limit the datacenters included for billing. Using the dc_include_filter properties, a numbered list of datacenters to include can be defined. For example, when an MSP has a private datacenter used for platform administration that should not be billed, the list can be defined to include all other datacenters.|
By default, when datacenter billing is enabled, the billing integration will process all physical data centers which have associated usage data for a given billing period. This setting overrides that behavior and enables an ‘include’ list of physical datacenters to include billing data for. If enabled, the billing integration will only generate output for datacenters that match the include list. The default setting for this value is ‘false’. If set to ‘true’, then one or more datacenter filters are specified by creating billing.properties entries named ‘dc_include_filter_<x>’ (where <x> is 1,2,3,etc...),
|When dc_filtering is used, set datacenters to include in the billing process in order. For example, dc_include_filter_1=London, where London is a datacenter name.|
|This setting is generally not used. It is only required when two independent Abiquo Billing Integration processes are using the same Abiquo database, and it is used to manage database resources on a 'per process' basis, to ensure the processes do not conflict with each other. So the job_id setting should be set to a unique numerical value, which is used to identify the billing process. It is the responsibility of the Abiquo System Administrator to ensure that all job IDs are unique for a particular Abiquo Server.|
Enable collection of core usage (as shown in the following table) by the hard limit allocation configured for an individual Enterprise/VDC.
Abiquo Billing Integration allows you to configure which of the 'core' usage metrics (i.e. CPU, Memory, Local Storage, External Storage, VLAN, and IP) Abiquo will calculate and supply to the Billing Connector for processing. The default values are set as shown in this table, but you can define each usage in the billing.properties. Each property can have a value of "true" or "false".
|true||Process virtual CPU usage values|
|cpu_on_usage||false||Process virtual CPU usage values for VMs that are powered ON|
|cpu_off_usage||false||Process virtual CPU usage values for VMs that are powered OFF|
|true||Process virtual memory usage values|
|memory_on_usage||false||Process virtual memory usage values of VMs that are powered ON|
|memory_off_usage||false||Process virtual memory usage values of VMs that are powered OFF|
|true||Process local storage usage values for primary and secondary hard disks|
|true||Process IP usage values for public IPs|
Process VLAN usage values for private VLANS
|true||Process external storage usage values for volumes of managed storage|
If true, count the number of VMs hosted on Abiquo HA-enabled Racks that are owned by the Enterprise/VDC
If true, process the CPU of servers that have been explicitly reserved for an Enterprise.
If true, process the RAM of servers that have been explicitly reserved for an Enterprise.
When enabled, the billing integration will collect repository usage (VM Templates and Instances) created by an Enterprise.
When enabled, the billing integration will process VMs created by an Enterprise in antiAffinity layers.
|firewall_usage||false||Process firewall usage values|
|loadbalancer_usage||false||Process load balancer usage values|
Abiquo Billing Integration allows you to collect Usage data for all of the different Abiquo Storage Tiers.
In Abiquo, each Physical Datacenter supports up to four tiers of storage. By default, these are called 'Default Tier 1',...,'Default Tier 4'. However, the Storage Tiers can be renamed on a per Datacenter basis, so for example Datacenter A may have 'Default Tier 1' while Datacenter B may have 'Tier A'. Abiquo Billing Integration identifies Storage Tiers by name alone, and will aggregate the usage of all Tiers that have the same name. This is useful when you want to charge consistently for a tier level of storage across multiple Datacenters (for example 'Default Tier 1'). It is important to review your Storage Tier names to ensure that you have no unnecessary conflicts between Datacenters, for example, names which exist at different tier levels, which you want to charge different amounts for.
Process custom storage usage using the storage_tier_<n> and storage_tier_<name> properties
|true||true, false||e.g. storage_tier_1=true |
Enable the defined storage tier. Disabled Storage Tier usages are not processed or passed to the Billing Connector
|e.g. storage_tier_1_name=Bronze Tier |
Specifies the name of an Abiquo Storage Tier.
This must EXACTLY match the Storage Tier name, as reported by the Abiquo GUI (or API)
If all of your Storage Tiers have default names, then you do not need to specify any other settings in billing.properties, as Abiquo Billing Integration will automatically define the first four Storage Tiers for you (i.e. it assumes that storage_tier1 through storage_tier_4 map to the default tier names, if no settings are explicitly provided).
Furthermore, you can disable the collection of individual default Storage Tiers by only specifying the enable/disable setting for a given tier. For example, if billing.properties contains only the following storage tier settings:
This tells Abiquo Billing Integration to collect usage data for 'Default Tier 1' and 'Default Tier 2' only.
Abiquo Billing Integration allows you to collect Usage data for all of the different Abiquo datastore tiers.
In Abiquo, each Physical Datacenter supports the datastore tiers defined by the administrator. By default, there is one tier defined called 'Default Datastore Tier'. New storage tiers can be created and existing ones renamed on a per Datacenter basis, so for example Datacenter A may have 'Default Datastore Tier' while Datacenter B may have 'Tier A'. Abiquo Billing Integration identifies Local Datastore Storage Tiers by name alone, and will aggregate the usage of all Tiers that have the same name. This is useful when you want to charge consistently for a tier level of storage across multiple Datacenters (for example 'Default Datastore Tier'). It is important to review your datastore tier names to ensure that you have no unnecessary conflicts between Datacenters, for example, names which exist at different tier levels, which you want to charge different amounts for.
Process tiered datastore usage using the local_tier_storage_<n> and local_tier_storage_<n>_<name> properties
|true||true, false||e.g. local_tier_storage_1=false|
Determines whether the defined datastore tier usage is enabled or not. Disabled datastore tier usages are not processed or passed to the Billing Connector. The default value is 'false'..
|e.g. local_tier_storage_1_name=Bronze Tier|
Specifies the name of an Abiquo datastore storage tier. This must EXACTLY match the datastore storage tier name, as reported by the Abiquo GUI (or API).
You can collect data for some tiers only. For example, if billing.properties contains only the following storage tier settings:
This tells Abiquo Billing Integration to collect usage data for the Premium Storage Tier but not for the Standard Storage Tier.
Configure Cost Code Usage
The Billing Integration enables you to collect Usage data for the Abiquo Cost Codes associated with Appliance Library Images. To enable custom usage processing you must add the following setting to the billing.properties file: cost_code_usage=true
If cost_code_usage is set to true, add properties into billing.properties for each Abiquo cost code that you define
|true, false||Enable the defined cost code. Disabled Cost Code usages are not processed or passed to the Billing Connector|
e.g. cost_code_1_name=Win 2K8 Server
The Billing Integration enables you to collect Usage data for the Hypervisors used on your platform. To enable hypervisor usage processing you must add the following setting to the billing.properties file: hypervisor_usage=true
You must then define the Hypervisor types that you want to include in the billing system output. This is done by creating billing properties file entries named 'hypervisor_<n>', where <n> is an incrementing number from 1.
|false||true, false||If hypervisor_usage is set to true, add properties into the billing.properties file for each hypervisor that you use on the platform |
and each hypervisor that you want Abiquo Billing Integration to collect usage data for.
|false||Enable the hypervisor with a number <n>, which is an incrementing number from 1. Disabled hypervisors are not processed or passed to the Billing Connector|
|The value must EXACTLY match the hypervisor name, as reported by the Abiquo UI (or API)|
Example of hypervisor properties
Abiquo Billing Integration allows you to define your own usage metrics which are not available within the main Abiquo accounting data. To enable custom usage processing you must add custom usage properties that include an SQL query.
Abiquo Billing Integration supports multiple custom usages, and multiple settings are required in billing.properties to define each setting. The settings are all 'custom_<n>', where <n> is a unique number. Note that <n> must begin at 1 in the billing.properties file, and Abiquo Billing Integration stops reading custom usages when it cannot find a custom usage for the next number in the sequence.
The settings required for each custom usage are:
|true||true, false||Enable the defined custom usage. Disabled custom usages are not processed or passed to the Billing Connector|
1000 - 2000
|SQL query as described below|
|SQL query as described below|
Custom Usages work by allowing you to define SQL queries that are used to calculate the custom usage value. Generally you will need to define an Enterprise-based SQL query, and also a VDC-based SQL query (of course if you are only interested in one type (e.g. Enterprises) then you can omit the query for the other type).
The database queries support a number of special placeholders, which the Abiquo BI will substitute for appropriate values when it process accounts. The placeholders are all case-sensitive, and the available placeholders are:
Use this placeholder when you want to refer to the timestamp representing the account period start time. Note that the start time is INCLUSIVE.
Use this placeholder when you want to refer to the timestamp representing the account period end time. Note that the end time is EXCLUSIVE.
Use this placeholder when you want to refer to an Enterprise ID.
Use this placeholder when you want to refer to a VDC ID.
This is a special placeholder for any query that refers to the Accounting table. It should be used instead of directly referencing the 'accounting_event_detail' table (which should not be queried directly) or the 'HOURLY_USAGE_MAX_VW' view, as it provides 'optimised' DB access to the data defined by that view.
An example SQL query might be:
The data in the main Abiquo 'kinton' schema typically contains no date information. The accounting tables overcome this limitation by tracking specific resources on an hourly basis. This means that any non-account-based custom metrics will be based upon data as currently stored at the time which Abiquo Billing Integration runs, rather than the time of the account period.
In an Abiquo installation you may have more than one Abiquo datacenter in your physical datacenter. In Abiquo you can create billing output by groups within your physical datacenter (i.e. you can break down an enterprise's or VDC's usage into more than one section for the physical datacenter). This is useful for situations in which Abiquo is managing hybrid environments of multiple datacenters with usage that must be tracked differently. For example, in a hybrid environment, the cloud owners may have their own private Abiquo datacenter (which does not require billing) integrated into a hosted Abiquo solution alongside other hosted Abiquo datacenters (which do require billing). This is shown in the screenshot below, where Abiquo Datacenter 1 (Cloud_owner_private_DC) is not billed, whereas the other two datacenters are billed.
Note that billing by Abiquo datacenter is not supported by all billing connectors. In Abiquo, the CSV and DBMS connectors support datacenter billing, but the Zuora connector does not.
This section covers settings and configuration specific to each supported Billing Connector.