Documentation

Skip to end of metadata
Go to start of metadata

Changes to Abiquo Billing Integration in Abiquo 3.10

Abiquo 3.10 introduces billing per datacenter based on pricing per datacenter. To use billing per datacenter, set the dc_grouping option to true.

Abiquo 3.10 introduces the following new billing usage charges:

  • cpu_on_usage
  • cpu_off_usage
  • memory_on_usage
  • memory_off_usage
  • firewall_usage
  • loadbalancer_usage
  • local_tier_storage_usage with tier name as local_tier_storage_X

 

Overview of Billing Integration

Abiquo Billing Integration can be used to process the Abiquo Accounting data and provide it to a third party billing system, where it can be used to generate customer invoices for Abiquo usage (for example).

Abiquo Billing Integration is currently supplied as a separate stand-alone utility, which should be installed on a machine independent of other Abiquo software. It comprises the following components.

  • Core Billing Utility: Calculates the usage data and manages it on a 'per account' basis
  • Billing Connectors: Take the 'usage' data from the core billing utility and provide it to a third-party billing system.
ConnectorDescriptionDocumentation
ZuoraTakes Abiquo usage data and uploads it to a specified Zuora Billing System, where it can be used to create Zuora invoices. Zuora Billing Integration
CSVGeneric connector that can be used to generate CSV files which contain Abiquo Usage data. The CSV files can then be processed by a third party billing system. CSV Billing Integration
DBMSGeneric connector that can be used to populate a Database table (located in any suitable MySQL database) to contain Abiquo Usage data.
The Database table can then be processed by a third party billing system, completely independently from the Abiquo Server.
DBMS Billing Integration
Aria Extends the CSV Connector to integrate with Aria via the UALCSV Billing Integration 
Aria Billing Integration.

 

Install Billing Integration

To install and run Abiquo Billing Integration you should follow the steps below.

1. Prerequisites

Abiquo Billing Integration is a separate utility so it can be installed on any machine that is able to connect to the main Abiquo MySQL database. In a test system, it can be installed on the Abiquo Server itself.

2. Unzip the file

Unzip the 'abiquo-billing-redist.zip' file into suitable install location on your chosen machine.

3. Configure the Database Billing User

Create a new database user that will be used by the billing integration to access the Abiquo database.   The following example SQL creates a new user called 'billing' and grants the user the required database permissions:

CREATE USER 'billing'@'%' IDENTIFIED BY 'password';
GRANT SELECT,EXECUTE ON kinton.* TO 'billing'@'%';
GRANT SELECT ON mysql.proc TO 'billing'@'%';
GRANT SELECT ON kinton_accounting.* TO 'billing'@'%';

4. Populate the Account Mapping Table

Populate the Account Mapping table so that Abiquo Billing Integration knows which Abiquo Enterpises and VDCs it needs to generate Usage data for. See #How to Populate the Account Mapping Table for more details.

5. Configure the billing.properties file

Configure the billing.properties file so that Abiquo Billing Integration knows what usage data needs to be generated and which connector to use. See #Configure Abiquo Billing Integration for more details.

6. Configure the connector properties file

Configure the appropriate connector properties file (i.e. csv.properties, dbms.properties, or zuora.properties), so that the connector knows how to process the data supplied by Abiquo Billing Integration. 

See:

7. Run the billing script

Finally, run 'billing.sh' to generate usage data for the configured accounts.

To run it manually for testing, change directory to the folder where Abiquo Billing Integration is installed and run the script. For example, if you installed it in /opt/BillingIntegration, type the following commands:

cd /opt/BillingIntegration
./billing.sh

To generate billing data at regular intervals, e.g. once a week, you can schedule Abiquo Billing Integration jobs using programs such as cron.   If you are using cron, set the script's current working directory correctly in your cron job.   For example, if you have installed the Abiquo Billing Integration into the '/opt/BillingIntegration' folder, then your crontab entry command could look like this:

0 1 * * * cd /opt/BillingIntegration; sh billing.sh

Note that Abiquo Billing Integration automatically tracks which data has already been processed for each account, so usage data should not be processed multiple times.

How to Populate the Account Mapping Table

A very important step in the billing process is to map Abiquo Enterprise/VDC IDs to customer IDs in your billing system. For example, mapping an Abiquo Enterprise ID to a Zuora Account Number.

You must enter the values in the billing_account_mapping table before you run Abiquo Billing Integration (ABI). This is because ABI uses the new Database tables created during the ABI install process to store the account mappings and identify which Enterprises and VDCs to process. Specifically, you must enter at least one row in the table for every enterprise or virtual datacenter (VDC) configured in the billing system.

A row in the Account Mapping Table contains the following information:

  • The Abiquo Enterprise or Virtual Datacenter ID
  • The Mapping Type
  • The billing system attribute name (e.g. 'ZUORA_ACCOUNT_NUMBER' if using the Zuora Connector)
  • The attribute value, i.e. the value associated with the attribute name (e.g. 'MyCorporation' )

Get the Abiquo Enterprise or Virtual Datacenter ID

You will need the Abiquo Enterprise or Virtual Datacenter ID.
In this example, we will retrieve a list of all Virtual Datacenter IDs using the Abiquo API.
Enter the following command in the shell window with the appropriate values for your Abiquo Server.

curl --verbose -X GET 'http://10.10.10.10/api/cloud/virtualdatacenters' \
-H "Accept: application/vnd.abiquo.virtualdatacenters+json;version=3.6" \
-u admin:xabiquo | pjson

In the response, you will see the ID and Name so you can identify your VDC.

    "id": 29, 

 
    "name": "MyVDC", 

For more information, see the Abiquo Developer's Guide VirtualDatacenterResource#Retrieve all virtual datacenters and EnterpriseResource#Retrieve a list of Enterprises

Obtain The Mapping Type

This value determines if the mapping is to an Abiquo Enterprise or VDC. It is the idMappingType for Enterprise or VDC from the billing_mapping_type table.

The 'billing_mapping_type' table has the following values:

idMappingType

mappingName

1

ENTERPRISE

2

VDC

Use SQL to configure the Account Mapping Table

For example, you would use this SQL statement to map a VDC with ID 29 to a Zuora account with an Account Number of 'MyCorporation' would be:

The following SQL statement will add ALL Enterprises with the billing attribute name ENTERPRISE.

INSERT INTO kinton.billing_account_mapping (idAbiquo, idMappingType, billingAttributeName, billingAttributeValue)
(SELECT idEnterprise, 1, 'ENTERPRISE', e.name FROM kinton.enterprise e);

Confirm that the billing account mappings are set by viewing the billing_account_mapping table data.

mysql> select * from kinton.billing_account_mapping;
+----------+---------------+----------------------+-----------------------+
| idAbiquo | idMappingType | billingAttributeName | billingAttributeValue |
+----------+---------------+----------------------+-----------------------+
|        1 |             1 | ENTERPRISE           | Abiquo                |
|        2 |             1 | ENTERPRISE           | TVSC                  |
+----------+---------------+----------------------+-----------------------+
2 rows in set (0.00 sec)

Some Billing Systems may require more than one 'attribute' to be defined. You can store these attributes in different lines in the billing_account_mapping table. All the attributes you define in this table will be supplied to the Connector being used by Abiquo Billing Integration.

Configure Abiquo Billing Integration

After the Account mappings have been defined in the database, there are two further areas of ABI that require configuration:

  • Configuration of the billing process.
  • Connector-specific configuration, which means configuration of details specific to the underlying Billing Connector.

This section discusses the configuration of the billing process. Connector-specific configurations are covered in a separate section for each connector.

All of the billing process (i.e. non-connector) settings are contained within the 'billing.properties' file, which should be located in the current working directory for ABI (the same location as the billing.sh file). An example billing.properties file can be found at the end of this section.

Configure General Billing Settings

These settings control how the billing data is aggregated by time period, the DBMS which is used to obtain the usage data, and the actual billing connector (e.g. CSV connector, Zuora connector, etc.) being used.

PropertyDefaultRangeDescription

account_period

DAY

HOUR, DAY, WEEK, MONTHThis setting defines the period for which usage data is aggregated. The default value is 'DAY', meaning that accounting data is recorded per day. However, the value stored will depend on the granularity, which may divide the accounting period into smaller units, e.g. a granularity of HOUR, means the units for a day will be 24. By default, the Billing Integration will record usage quantities at the same granularity as the account period. However, the usage_units_per_hour parameter in the billing.properties file will override this behavior and always return the number of ‘units per hour’

usage_units_per_hour

falsetrue, falseBy default this value is false, meaning that the billing integration will record resource usage units in the granularity configured for the Accounting tables.   Set this value to ‘true’ to override the default behavior and always record usages ‘per hour’.

init_period

DAYHOUR, DAY, WEEK, MONTHThe point in time for which 'new' accounts should start collecting usage data. For example, if a new account is added to the billing integration on 7 December, then an 'init_period' of 'MONTH' would mean that we try and collect data from 1 December, whereas a value of 'DAY' would mean that data is collected from 7 December.

connector_name

CSVCSV, DBMS, ZuoraThis setting determines which connector is used by ABI to process usage data. Allows you to specify a well-known connector using a 'friendly' name. Friendly names are case insensitive, and take priority over the 'connector_class' setting.

connector_class

  

If the connector you want to use does not have a friendly name supported by the ABI, set the 'connector_name'. This setting MUST match the complete Java class name of the Connector, otherwise ABI will fail to run.

  • com.abiquo.billing.connectors.csv.CSVConnector
  • #connector_class=com.abiquo.billing.connectors.zuora.ZuoraConnector
  • #connector_class=com.abiquo.billing.connectors.dbms.DBMSConnector

dbms_connection_url

  These settings determine how Abiquo Billing Integration connects to the database used to retrieve the billing usage data. For example, dbms_connection_url=jdbc:mysql://127.0.0.1:3306/kinton

dbms_jdbc_driver

com.mysql.jdbc.Driver

 Currently only the default value of 'com.mysql.jdbc.Driver' is supported

dbms_user

   

dbms_password

   

dc_grouping

false

 Group billing by Abiquo datacenter. When set to ‘true’, this setting enables data center billing, i.e. physical data center sub-grouping.  Note that you must also enable data center output in the configured billing connector. In Abiquo 3.10, set to true to use pricing per datacenter. If false, Abiquo will use the pricing from the first datacenter in the pricing model  

dc_filtering

false

 Enable filtering options to limit the datacenters included for billing. Using the dc_include_filter properties, a numbered list of datacenters to include can be defined. For example, when an MSP has a private datacenter used for platform administration that should not be billed, the list can be defined to include all other datacenters.

By default, when datacenter billing is enabled, the billing integration will process all physical data centers which have associated usage data for a given billing period. This setting overrides that behavior and enables an ‘include’ list of physical datacenters to include billing data for. If enabled, the billing integration will only generate output for datacenters that match the include list. The default setting for this value is ‘false’.  If set to ‘true’, then one or more datacenter filters are specified by creating billing.properties entries named ‘dc_include_filter_<x>’ (where <x> is 1,2,3,etc...),

dc_include_filter_<n>

  When dc_filtering is used, set datacenters to include in the billing process in order. For example, dc_include_filter_1=London, where London is a datacenter name.

job_id

  This setting is generally not used. It is only required when two independent Abiquo Billing Integration processes are using the same Abiquo database, and it is used to manage database resources on a 'per process' basis, to ensure the processes do not conflict with each other. So the job_id setting should be set to a unique numerical value, which is used to identify the billing process. It is the responsibility of the Abiquo System Administrator to ensure that all job IDs are unique for a particular Abiquo Server.
allocation_usagesfalse 

Enable collection of core usage (as shown in the following table) by the hard limit allocation configured for an individual Enterprise/VDC.
By default, this setting is false, which means that core usage is collected by consumption as recorded in Abiquo accounting data.

 

Configure Core Usage

Abiquo Billing Integration allows you to configure which of the 'core' usage metrics (i.e. CPU, Memory, Local Storage, External Storage, VLAN, and IP) Abiquo will calculate and supply to the Billing Connector for processing. The default values are set as shown in this table, but you can define each usage in the billing.properties. Each property can have a value of "true" or "false".

PropertyDefaultDescription

cpu_usage

trueProcess virtual CPU usage values
cpu_on_usagefalseProcess virtual CPU usage values for VMs that are powered ON
cpu_off_usagefalseProcess virtual CPU usage values for VMs that are powered OFF

memory_usage

trueProcess virtual memory usage values
memory_on_usagefalseProcess virtual memory usage values of VMs that are powered ON
memory_off_usagefalse Process virtual memory usage values of VMs that are powered OFF

local_storage_usage

trueProcess local storage usage values for primary and secondary hard disks

ip_usage

trueProcess IP usage values for public IPs

vlan_usage

true

Process VLAN usage values for private VLANS

external_storage_usage

trueProcess external storage usage values for volumes of managed storage

ha_usage

false

If true, count the number of VMs hosted on Abiquo HA-enabled Racks that are owned by the Enterprise/VDC

reserved_server_cpu_usage

false 

If true, process the CPU of servers that have been explicitly reserved for an Enterprise.
The CPU value is the total number of CPU cores in the reserved servers.  
This value is ‘Enterpise only’ (i.e. does not apply to VDCs). 

reserved_server_memory_usage

false 

If true, process the RAM of servers that have been explicitly reserved for an Enterprise.
The Memory is the total amount of RAM in all reserved servers (reported in MB).  
This value is ‘Enterpise only’ (i.e. does not apply to VDCs). 

repository_usage

false 

When enabled, the billing integration will collect repository usage (VM Templates and Instances) created by an Enterprise. 
The repository usage is reported in GB. The value only applies to Enterprises (not VDCs). 

antiAffinity_usage

false 

When enabled, the billing integration will process VMs created by an Enterprise in antiAffinity layers.

firewall_usagefalseProcess firewall usage values
loadbalancer_usagefalseProcess load balancer usage values

 

Configure Custom Storage Tier Usage

Abiquo Billing Integration allows you to collect Usage data for all of the different Abiquo Storage Tiers.

In Abiquo, each Physical Datacenter supports up to four tiers of storage. By default, these are called 'Default Tier 1',...,'Default Tier 4'. However, the Storage Tiers can be renamed on a per Datacenter basis, so for example Datacenter A may have 'Default Tier 1' while Datacenter B may have 'Tier A'. Abiquo Billing Integration identifies Storage Tiers by name alone, and will aggregate the usage of all Tiers that have the same name. This is useful when you want to charge consistently for a tier level of storage across multiple Datacenters (for example 'Default Tier 1'). It is important to review your Storage Tier names to ensure that you have no unnecessary conflicts between Datacenters, for example, names which exist at different tier levels, which you want to charge different amounts for.

PropertyDefaultRangeDescription

tiered_storage_usage

false

true, false

Process custom storage usage using the storage_tier_<n> and storage_tier_<name> properties

storage_tier_<n>

truetrue, falsee.g. storage_tier_1=true
Enable the defined storage tier. Disabled Storage Tier usages are not processed or passed to the Billing Connector

storage_tier_<n>_name

  e.g. storage_tier_1_name=Bronze Tier
Specifies the name of an Abiquo Storage Tier.
This must EXACTLY match the Storage Tier name, as reported by the Abiquo GUI (or API)

Configure the Default Storage Tier Usage

If all of your Storage Tiers have default names, then you do not need to specify any other settings in billing.properties, as Abiquo Billing Integration will automatically define the first four Storage Tiers for you (i.e. it assumes that storage_tier1 through storage_tier_4 map to the default tier names, if no settings are explicitly provided).

Furthermore, you can disable the collection of individual default Storage Tiers by only specifying the enable/disable setting for a given tier. For example, if billing.properties contains only the following storage tier settings:

tiered_storage_usage=true
storage_tier_3=false
storage_tier_4=false

This tells Abiquo Billing Integration to collect usage data for 'Default Tier 1' and 'Default Tier 2' only.

Local Tier Storage Usage Configuration for Datastore Service Levels

Abiquo Billing Integration allows you to collect Usage data for all of the different Abiquo datastore tiers.

In Abiquo, each Physical Datacenter supports the datastore tiers defined by the administrator. By default, there is one tier defined called  'Default Datastore Tier'. New storage tiers can be created and existing ones renamed on a per Datacenter basis, so for example Datacenter A may have 'Default Datastore Tier' while Datacenter B may have 'Tier A'. Abiquo Billing Integration identifies Local Datastore Storage Tiers by name alone, and will aggregate the usage of all Tiers that have the same name. This is useful when you want to charge consistently for a tier level of storage across multiple Datacenters (for example 'Default Datastore Tier'). It is important to review your datastore tier names to ensure that you have no unnecessary conflicts between Datacenters, for example, names which exist at different tier levels, which you want to charge different amounts for.

PropertyDefaultRangeDescription

local_tier_storage_usage

false

true, false

Process tiered datastore usage using the local_tier_storage_<n> and local_tier_storage_<n>_<name> properties

local_tier_storage_<n>

truetrue, falsee.g. local_tier_storage_1=false
Determines whether the defined datastore tier usage is enabled or not. Disabled datastore tier usages are not processed or passed to the Billing Connector. The default value is 'false'..

local_tier_storage_<n>_name

  e.g. local_tier_storage_1_name=Bronze Tier
Specifies the name of an Abiquo datastore storage tier. This must EXACTLY match the datastore storage tier name, as reported by the Abiquo GUI (or API).

Configure the Local Tier Storage Usage

You can collect data for some tiers only. For example, if billing.properties contains only the following storage tier settings:

local_tier_storage_usage=true
local_tier_storage_1_name=Premium Storage Tier
local_tier_storage_1=true

local_tier_storage_2_name=Standard Storage Tier
local_tier_storage_2=false

This tells Abiquo Billing Integration to collect usage data for the Premium Storage Tier but not for the Standard Storage Tier.

Configure Cost Code Usage

The Billing Integration enables you to collect Usage data for the Abiquo Cost Codes associated with Appliance Library Images. To enable custom usage processing you must add the following setting to the billing.properties file: cost_code_usage=true

PropertyDefaultRangeDescription

cost_code_usage

false

true, false

If cost_code_usage is set to true, add properties into billing.properties for each Abiquo cost code that you define
and each Cost Code that you want Abiquo Billing Integration to collect usage data for

cost_code_<n>

 true, falseEnable the defined cost code. Disabled Cost Code usages are not processed or passed to the Billing Connector

cost_code_<n>_name

  

e.g. cost_code_1_name=Win 2K8 Server
Specifies the name of the Cost Code.
This must EXACTLY match the cost code name, as reported by the Abiquo UI (or API).

Hypervisor Usage

The Billing Integration enables you to collect Usage data for the Hypervisors used on your platform. To enable hypervisor usage processing you must add the following setting to the billing.properties file: hypervisor_usage=true

You must then define the Hypervisor types that you want to include in the billing system output. This is done by creating billing properties file entries named 'hypervisor_<n>', where <n> is an incrementing number from 1. 

PropertyDefaultRangeDescription

hypervisor_usage

falsetrue, falseIf hypervisor_usage is set to true, add properties into the billing.properties file for each hypervisor that you use on the platform
and each hypervisor that you want Abiquo Billing Integration to collect usage data for.

hypvisor_<n>

false Enable the hypervisor with a number <n>, which is an incrementing number from 1. Disabled hypervisors are not processed or passed to the Billing Connector

hypervisor_<n>_name

  The value must EXACTLY match the hypervisor name, as reported by the Abiquo UI (or API)

Example of hypervisor properties

hypervisor_usage=true
hypervisor_1=true
hypervisor_1_name=VMX_04

Configure Custom Usage

Abiquo Billing Integration allows you to define your own usage metrics which are not available within the main Abiquo accounting data. To enable custom usage processing you must add custom usage properties that include an SQL query.

Abiquo Billing Integration supports multiple custom usages, and multiple settings are required in billing.properties to define each setting. The settings are all 'custom_<n>', where <n> is a unique number. Note that <n> must begin at 1 in the billing.properties file, and Abiquo Billing Integration stops reading custom usages when it cannot find a custom usage for the next number in the sequence.

The settings required for each custom usage are:

PropertyDefaultRangeDescription

custom_<n>

truetrue, falseEnable the defined custom usage. Disabled custom usages are not processed or passed to the Billing Connector

custom_<n>_name

  

e.g. custom_1_name=ESXHypervisors
Specifies the name of the custom usage. Note that the name will probably be used by the Billing Connectors to identify the custom usage to the Billing System,
so carefully consider the name of your custom usage to ensure that it adheres to any required name formats (for example, some connectors will not allow spaces).

custom_<n>_id

1000+<n>Recommended:
1000 - 2000

e.g. custom_1_id=1234
This is a unique numerical identifier for the custom usage. This setting is optional, if it is not defined then Abiquo Billing Integration set a default value of 1000 + <n>.

custom_<n>_ent_sql

  SQL query as described below

custom_<n>_vdc_sql

  SQL query as described below
SQL queries to calculate custom usage values

Custom Usages work by allowing you to define SQL queries that are used to calculate the custom usage value. Generally you will need to define an Enterprise-based SQL query, and also a VDC-based SQL query (of course if you are only interested in one type (e.g. Enterprises) then you can omit the query for the other type).

The database queries support a number of special placeholders, which the Abiquo BI will substitute for appropriate values when it process accounts. The placeholders are all case-sensitive, and the available placeholders are:

Placeholder

Description

{ABQ_PERIOD_START}

Use this placeholder when you want to refer to the timestamp representing the account period start time. Note that the start time is INCLUSIVE.

{ABQ_PERIOD_END}

Use this placeholder when you want to refer to the timestamp representing the account period end time. Note that the end time is EXCLUSIVE.

{ABQ_ENTERPRISE_ID}

Use this placeholder when you want to refer to an Enterprise ID.

{ABQ_VDC_ID}

Use this placeholder when you want to refer to a VDC ID.

{ABQ_ACCOUNTING_SOURCE}

This is a special placeholder for any query that refers to the Accounting table. It should be used instead of directly referencing the 'accounting_event_detail' table (which should not be queried directly) or the 'HOURLY_USAGE_MAX_VW' view, as it provides 'optimised' DB access to the data defined by that view.

An example SQL query might be:

Custom Usage Limitations

The data in the main Abiquo 'kinton' schema typically contains no date information. The accounting tables overcome this limitation by tracking specific resources on an hourly basis. This means that any non-account-based custom metrics will be based upon data as currently stored at the time which Abiquo Billing Integration runs, rather than the time of the account period.

Configure Billing by Abiquo Dataenter

In an Abiquo installation you may have more than one Abiquo datacenter in your physical datacenter. In Abiquo you can create billing output by groups within your physical datacenter (i.e. you can break down an enterprise's or VDC's usage into more than one section for the physical datacenter). This is useful for situations in which Abiquo is managing hybrid environments of multiple datacenters with usage that must be tracked differently. For example, in a hybrid environment, the cloud owners may have their own private Abiquo datacenter (which does not require billing) integrated into a hosted Abiquo solution alongside other hosted Abiquo datacenters (which do require billing). This is shown in the screenshot below, where Abiquo Datacenter 1 (Cloud_owner_private_DC) is not billed, whereas the other two datacenters are billed.

Note that billing by Abiquo datacenter is not supported by all billing connectors. In Abiquo, the CSV and DBMS connectors support datacenter billing, but the Zuora connector does not.

The billing properties for this functionality are #dc_grouping#dc_filtering and dc_filter_include_<n> as described above.

Example 'billing.properties' file content

Configure Connector-specific Settings

This section covers settings and configuration specific to each supported Billing Connector.