header-project-description.png
The Cloud Ninja Metering Block is an extensible and reusable software component designed to assist software developers with the metering of tenant resource usage in a multi-tenant solution on the Windows Azure platform.

The need to meter and monitor resource usage at the tenant level is a common requirement for multi-tenant solutions. This metering data is commonly used to enforce quotas, set pricing, understand solution cost breakdown by tenant, generate alerts, or identify tenants which may be abusing the solution. Regardless of how this data is used, a core requirement is to be able to capture and store this data in a form that it can be utilized for the purposes.

header-solution-overview.png
cnmb-overview.png

Metering Block
The metering block ships as a hostable runtime component that can be deployed and hosted in an on-premise windows services, console application; or to Windows Azure either in it's own role or as part of another role for an existing solution. The metering block comes with a number of providers for collecting metering data from existing logs, and can be extended with custom providers for collecting metering data from other sources, such as custom logs, views, etc...

Metering Providers
  • Database Size
  • Database Bandwidth
  • W3C Logs (Bandwidth / Requests)
  • Storage Size
  • Storage Bandwidth

Repository
The Cloud Ninja metering block utilizes a storage provider implementation for writing metering data to a metering repository. The current metering block ships with an OData storage repository provider for writing the metering data to an OData service secured with HMAC authentication scheme. The solution also includes a data service project that exposes an OData metering service to the metering block and utilizes SQL Azure as the persistent store. Additional storage providers can be implemented to support peristence of metering data to other repositories.

header-prerequisites.png
NuGet makes is easy to install the prerequisites for the metering block. In order to build the metering block and run the OData repository and dashboard application you will need the following:
header-screenshots.png

Tenant View
cnmb-tenant-screenshot.png

Powerpivot Dashboard
cnmb-powerpivot-screenshot.png

Tenant List View (Sorted by Total Weighted Usage)
cnmb-tenantlist-screenshot.png

header-team.png
 Full Scale 180 - Metering News Feed 
Tuesday, November 19, 2013  |  From Full Scale 180 - Metering

After releasing the block a lot of new changes to the various technologies surfaced, and we decided to upgrade the block to incorporate these changes. Following is a list of updates we brought into the block with this release.


1. Source code availability. You can find the first version of the block as a downloadable zip file on the” downloads” section. However we decided to publish the code directly on the Codeplex TFS. The latest version is now available on the “Source code” tab.


2. Azure Storage API. Block is now using Storage Client Api 2.1.03


3. Async/await. The block is utilizing async/await pattern as appropriate.


4. HMAC. Updated and simplified HMAC implementation for the ODATA API operations


5. ODATA. Moved ODATA service from WCF data services to Web API


6. Data access. Removed EF dependency, using Dapper.net instead.


7. Extended Dapper for async/await. The solution is dependent on an offline copy of the dapper framework, extended for async/await support.


8. Azure SDK 2.2 support. Updated the cloud projects to use Azure SDK 2.2.


9. Updated repository and client implementations. Updated the metering repository, data repository and their respective clients to provide a clean separation, with async/await support.


10. Unity 3.0. upgrade.


11. DDL. Stored procedure for upsert


12. Updated entity classes. Refactored and streamlined after EF dependency removal


13. Upgrade to Enterprise Library Transient Fault Handling block. Removedconfiguration and added code based setup.


14. Updates on the client side to support new ODATA JSON output.

Thursday, May 24, 2012  |  From Full Scale 180 - Metering

I remember the times when I started to work for my first job at a big company. It was a large multinational telecommunications company, and one of the earliest concepts I have ever learned was “metering”. Up to that time, I was assuming everything one measures on a telephone switch is all about charging customers, and to understand how many minutes they have spent on the phone. Upon pouring over the code that runs on the switch and the peripherals, I came to realize the code was measuring the use of many things, and all was because of understanding the resource use of various different features, which also happens to be including the speech channels utilized.

Having a ubiquitous measuring of the resources a solution is using, that does not get in the way of the actual operations of it becomes even a more important concept, if the solution is utilizing resources from a different system, and potentially being charged for them. The urge to learn and analyze the tenants’ respective indirect utilization of Windows Azure resources the ISV is paying for the solution may have many different reasons. A few of the first that may come to mind can be:

  • How can I enforce quotas for tenants, or do I have tojQuery1520412992729106918_1337901680773
  • Are there any tenants consuming most of the resources?
  • Do we need to introduce pricing tiers?
  • How does the cost of goods for our solution get affected per tenant?
  • Are certain tenants causing starvation for other tenants?
  • Is my system behaving accordingly? Are we experiencing any resource usage anomalies?

The metering application block (which can be found at http://cnmb.codeplex.com), attempts to provide a solution to the ISVs asking those questions to themselves about their solution. The goal of the application block is to generate data where an ISV can compare the resource usage among the tenants.

Of course, given the wide range of pricing packages Microsoft provides, and possible changes Microsoft may take in their pricing policies, it would not be possible to calculate monetary values for those resource usages. It is also important to point out that the collected data is not the equivalent of the data Microsoft uses to calculate the bills at the end of each month. There can surely be differences in the output, since Microsoft billing has a different goal (to bill the customers) than the Cloud Ninja Metering block (to track relative usages across tenants for major resources).

What to Meter and Why

We have started with mentioning that, the CNMB is not a mechanism to analyze and double check your Azure bill details, however it is meant to provide the relative resource usage of the tenants. But what exactly do we need to measure to see the relative usage patterns? A good starting point is whatever is already there, provided by the Windows Azure page as the main pivots for a pricing discussion.

To make our point about metering not being related with the pricing, we also included a provider that measures the ingress data (data coming to the data center), which is not charged by Microsoft. The reason we chose to provide this is, heavy incoming data traffic by certain tenants may indicate additional data usage on certain other resources, such as compute time.

You can obviously start measuring many different metrics; however the block comes with 5 default providers that map to the major pivots:

  • Blob size: Measured by the blob store size provider
  • Database size: Measured by the database size provider
  • Bandwidth : Measured by the IIS bandwidth provider, the database bandwidth provider and the blob bandwidth provider

Of course it goes without saying that depending on the solutions needs, there may be needs for measuring other aspects. Number of documents processed, active users, queue lengths could be among those.

In designing the block, we concentrated on four tenets:

  • Block can be easily hosted in any hosting application, and does not depend on Azure runtime
  • Does not require a code change for the related application, and be application agnostic
  • Rely heavily on configuration, not coding
  • Easily extensible

You can reach us at info@fullscale180.com for further questions/comments.

Have a great day!

Thursday, May 24, 2012  |  From Full Scale 180 - Metering

I remember the times when I started to work for my first job at a big company. It was a large multinational telecommunications company, and one of the earliest concepts I have ever learned was “metering”. Up to that time, I was assuming everything one measures on a telephone switch is all about charging customers, and to understand how many minutes they have spent on the phone. Upon pouring over the code that runs on the switch and the peripherals, I came to realize the code was measuring the use of many things, and all was because of understanding the resource use of various different features, which also happens to be including the speech channels utilized.

Having a ubiquitous measuring of the resources a solution is using, that does not get in the way of the actual operations of it becomes even a more important concept, if the solution is utilizing resources from a different system, and potentially being charged for them. The urge to learn and analyze the tenants’ respective indirect utilization of Windows Azure resources the ISV is paying for the solution may have many different reasons. A few of the first that may come to mind can be:

  • How can I enforce quotas for tenants, or do I have tojQuery1520412992729106918_1337901680773
  • Are there any tenants consuming most of the resources?
  • Do we need to introduce pricing tiers?
  • How does the cost of goods for our solution get affected per tenant?
  • Are certain tenants causing starvation for other tenants?
  • Is my system behaving accordingly? Are we experiencing any resource usage anomalies?

The metering application block (which can be found at http://cnmb.codeplex.com), attempts to provide a solution to the ISVs asking those questions to themselves about their solution. The goal of the application block is to generate data where an ISV can compare the resource usage among the tenants.

Of course, given the wide range of pricing packages Microsoft provides, and possible changes Microsoft may take in their pricing policies, it would not be possible to calculate monetary values for those resource usages. It is also important to point out that the collected data is not the equivalent of the data Microsoft uses to calculate the bills at the end of each month. There can surely be differences in the output, since Microsoft billing has a different goal (to bill the customers) than the Cloud Ninja Metering block (to track relative usages across tenants for major resources).

What to Meter and Why

We have started with mentioning that, the CNMB is not a mechanism to analyze and double check your Azure bill details, however it is meant to provide the relative resource usage of the tenants. But what exactly do we need to measure to see the relative usage patterns? A good starting point is whatever is already there, provided by the Windows Azure page as the main pivots for a pricing discussion.

To make our point about metering not being related with the pricing, we also included a provider that measures the ingress data (data coming to the data center), which is not charged by Microsoft. The reason we chose to provide this is, heavy incoming data traffic by certain tenants may indicate additional data usage on certain other resources, such as compute time.

You can obviously start measuring many different metrics; however the block comes with 5 default providers that map to the major pivots:

  • Blob size: Measured by the blob store size provider
  • Database size: Measured by the database size provider
  • Bandwidth : Measured by the IIS bandwidth provider, the database bandwidth provider and the blob bandwidth provider

Of course it goes without saying that depending on the solutions needs, there may be needs for measuring other aspects. Number of documents processed, active users, queue lengths could be among those.

In designing the block, we concentrated on four tenets:

  • Block can be easily hosted in any hosting application, and does not depend on Azure runtime
  • Does not require a code change for the related application, and be application agnostic
  • Rely heavily on configuration, not coding
  • Easily extensible

You can reach us at info@fullscale180.com for further questions/comments.

Have a great day!

 Full Scale 180 - Metering News Feed 



logo-footer.png

Last edited Sep 21, 2013 at 5:31 PM by bnene, version 47