The DevOps Scorecard : Key Metrics to track DevOps team success (post on

Mid-last year our team switched from doing Agile to doing DevOps. As we forayed into the journey trying to learn about DevOps and practice it at the same time,  a lot of questions arose in the team.  How was this different from agile and most importantly How were we going to be successful?

That’s when we wrote down what would be the success criteria for our team: Ship code frequently without causing a customer outage.

As the team matured we started evaluating a more granular way to track success.  Could the team mantra be broken down into quantifiable success metrics that could be represented in a scorecard? Based on our experience the DevOps scorecard should contain these 9 metrics to track DevOps team success:

DevOps Scorecrd

Read more on


Stop users from abandoning your app. Use End User Monitoring

With the growth of web and mobile apps, the app owner’s focus has shifted from keeping his app infrastructure available to keeping his users happy. Hence, End User Experience Monitoring has evolved as one of the five functional dimensions of APM by Gartner (login required). In this blog post, I will take you through the fundamentals of End User Monitoring and what APM can do to ensure you detect app issues before they impact your end user.

Let’s start with a few basics. The most important metric of your user’s satisfaction is a healthy response time. Which means no matter what the geography, device, carrier or browser your end user is using, no matter if it’s peak time or idle time, the app does not slow down at any point in time…..

Read more on ServiceManagement360

Reduce Failed Deployments. Innovate Rapidly. Use App Diagnostics

In this age of relentless innovation, where web and mobile startups are changing the game everyday, the need for companies to stay ahead of the curve by continuously innovating and delivering, is key to survival. In order to get quick time to value companies are adopting the DevOps model which allows code to be deployed not in weeks or days, but several times a day. For example Etsy, a popular web marketplace, deploys 50 times a day and Netflix deploys hundred times per day.

Now for a second, think about the complexity behind this. Every end user interaction with a web or mobile application results in execution of several hundred or thousand lines of code running across a complex infrastructure that includes various backend components like databases, application servers, web servers and so forth in the cloud or in a data center.  To add to the complexity there are many developers constantly adding, enhancing code to the code base which is deployed by Operations team continuously, several times a day. How can the application owner ensure that every line of code he checks in is not causing a failure somewhere in this big ecosystem?  Can he catch problems before it impacts his end user? This brings into the picture tools that can help you monitor your application’s performance and availability and correlate that to code issues.

Read more here

5 Ways APM can solve your DevOps worries (Guest post on

This week at the interop conference in New York I had the chance to talk to a lot of IT managers transforming their organizations to adopt a DevOps culture. One of the key questions I heard repeatedly was – are there some best practices to break down the wall between Dev and Ops?  How do you get to a point where we can deploy code multiple times a day like Facebook , Amazon and Netflix without impacting our users’ experience ? How do we minimize the risk of deployment failures and the blame game that ensues? With the need to innovate and close the customer feedback loop quickly Ops is being asked to deploy code more rapidly than ever before.  The complexity of hundreds of developers updating code makes it a very sensitive environment. The last thing Operations wants is an outage leading to financial loss and the last thing development wants is to get a call in the middle of the night to figure out what code change caused it.

To have tight control over this fast paced environment we need Application Performance Management at every step. APM can help break down the Chinese wall between Dev and Ops and eliminate the finger pointing that happens during outages.


Building a Roadmap

As a product manager one of the most critical jobs is to create a solid roadmap. This is what your development team will follow to deliver value to the market and generate revenue for your product.

How do you decide what should be on the roadmap? How far out should the roadmap be? How do you prioritize the features on the roadmap? When things don’t fit , what do you drop? There is no silver bullet or perfect solution , however there are some guidelines that we could follow to do this effectively.

Here is a step by step outline on how one could roadmap effectively.

  1. Goal: First reach a consensus on what is the goal of the business provided by your upper management
  2. Feature Bucket: Once you have a goal, break it down into a bucket of “features” you need to deliver to meet that goal. This bucket can be produced by taking into account the following:
    1. Market trends (from analyst reports)
    2. Competitor trends and differentiation (from doing a competitive analysis)
    3. Existing customer and field feedback (from sales and support teams)
  3. Feature Filtering: Evaluate each feature based on the following criteria in order for it to make the roadmap:
    1. Does the feature align with the core strategy of the business
    2. Does the feature align with the core competency of the business and team?
    3. Does the feature have the ability to make money
    4. Roadmap
  4. Feature Prioritization and Phase Determination: Prioritize the selected features in order of which you expect them to arrive in order to meet your goal. These are called Phases (Phase 1 being the most urgent ones to deliver) Every phase contains:
    1. 20-25% customer-driven features i.e., features requested by customers (RFEs, features requested by influencers, pending deals)
    2. 15-20% functionality improvements of previously delivered features
    3. 55-65% market driven features i.e., new features that will help us differentiate in the market
  1. Timeline: Draw a timeline of 12 months (divided into 4 quarters, first 2 quarters divided into months, first 2 months divided into weeks). Determine where in the roadmap should the feature be placed. In order to do that you need to:
    1. Get a sizing from engineering for the MVP (minimally viable product) for each feature
    2. If sizing throws the feature from an earlier phase to a later phase, re-evaluate either 1) later phase is acceptable to meet the goal OR 2) what feature can be postponed to contain this feature in the right phase
    3. Negotiate and close plan with development
  1. Success Metrics: Define and track metrics for success of the roadmap and plan put together
    1. Feature Uptake: Did the delivery of features help to meet the goal defined? (E.g., Phase 1 features lead to x% increase in revenue in the 2 quarters following the delivery of the feature, Customer driven features helped close deals in the pipeline, x% increase in feature usage on completion of improvements)
    2. On Time Delivery: Did we deliver 90% of what we planned to the market on time? Did we deliver the remaining 10% in the next 30 days
    3. Quality of Delivery: All features delivered to the market should meet quality standards set by Support and Operations

Why Multitenancy is so critical for your SaaS Business Model?

Multitenancy , simply put, is the ability to share a single instance of a software running on a server with multiple clients securely. It is one of the key architecture parameters that is directly related to profitability. If your SaaS application does not support multi-tenancy, your infrastructure costs will rise for every user added and you will not be able to achieve economies of scale.  Let me explain how ..

Let’s say you have a SaaS application which needs to be supported by 3 components: UI server, database and app server.   The UI and App Server reside on a single VM and the Database is on another VM.  So you need a total of 2 VMs to support your application instance.

Now there are a bunch of knobs we can adjust to evaluate the infrastructure cost for this app. For simplicity , let’s say your Cloud provider offers you 3 prices for IaaS (including hosting cost and support/labor):

Small VM  =  $500 per month 

Medium VM = $ 750 per month

Large VM = $ 1000 per month

Small , medium and large configurations are determined by number of cores of CPU, memory size, storage space and network bandwidth.

Your target is to support 100 customer accounts i.e., 100 tenants

Cost for a Single-Tenant Architecture:  For every customer account you will need to create a unique environment i.e., 2 VMs.  To support 100 client accounts you need 100*2 = 200 Small VMs !!  Your total hosting cost for 200 small VMs will be $500*200 = $100,000 per month. 

Cost for Multi-Tenant Architecture:  Now say you can support multiple customer accounts per software instance. After doing some resource consumption and performance measurements you determine that you could support either of the following two multi-tenancy configurations:

1) 20 client accounts per instance on Small VMs. To support 100 tenants you need only (100/20)*2 = 10 Small VMs. Your cost for 10 small VMs would be $500*10 = $5,000 per month !  

2) 100 client accounts per instance on Medium VMs. To support 100 tenants, you need only (100/100)*2 = 2 Medium VMsYour cost for 2 Medium VMs = $750*2 = $1500 per month ! 

With a multi-tenant architecture your cost reduces by >95% for the same number of customer accounts

Let’s say your app gets really popular and you need to scale to 1000 customer accounts. With 100 client accounts per instance on Medium VMs, to support 1000 tenants, you need (1000/100)*2 = 20 Medium VMsYour cost for 20 Medium VMs = $750*20 = $15,000 per month.  

So you can scale to 10 times the number of users and your cost would still be 85% less than a single tenant case.

The calculations have been kept very simple for understanding the concept. There are other infrastructure costs to consider like network I/O, persistent storage, backup VMs for high availability etc which add to the cost case.  However the gist of the matter is if your SaaS application is not truly multi-tenant your SaaS cost case will blow up and it will take you longer to become profitable.

Converting an on-premise software to SaaS

You have an offering that you provide to customers to download , install and maintain. However in recent times you have been losing customers since they prefer competitive offerings that are provided as SaaS . Your customers are telling you that they don’t want to install, configure, maintain and don’t want to incur the cost of infrastructure either. “Why can’t you just put it on the cloud and give me a SaaS offering?”

In order to stop the bleeding you decide you will host the application and offer it as a service. In this blog post I will help provide guidance on the key technical aspects that needs to be considered when converting your on-premise offering to a SaaS offering. I will follow this with a post on the business aspects to consider

There are three things to consider 1) Infrastructure to host your application 2) Changes in the application architecture 3) Changes in the development model -adopting DevOps

Infrastructure: To host your application you need a scalable and reliable infrastructure. You have 3 choices: public cloud , private cloud or a hybrid cloud.

If you are a large organization with centralized IT and they can offer you a reliable private PaaS use that. If you are a smaller company or a startup and don’t have any experience in hosting and managing a data center or cloud infrastructure, use public PaaS from one of the many vendors like Amazon, CloudFoundry, OpenShift etc. Almost any web application or mobile application stacks are available on public clouds today. I found this article very insightful on choices that can be made.

Application: Your product was written originally to be downloaded and installed on-premise scaling up to a 100 users maybe. Now you need to host this product yourself, support it to scale to thousands of users ensuring security and great performance. In order to do the conversion, you will need to address the following aspects of your product

  1. Communication protocols  – your product consists of many components or building blocks. If you have a component that is installed on the customer end which needs to interact with a component that you are hosting, then the communication between the two needs to happen over the internet using protocols such as https. So if you have other protocols that assumed all components will be in the same network, you will need to change those
  2. Loosely coupled integration – hard coded integration between components will absolutely not work. Expose APIs for all building blocks of your application and let them communicate with each other
  3. Multi-tenancy – the same application will be accessed by multiple users. Which means you need to make it secure such that one user doesn’t see the data of another user. You could start with a multi-instant architecture but in the long run supporting multi-tenancy is extremely critical in SaaS.
  4. Security – Since you will be receiving data from the customer over the internet, security and compliance is extremely critical to be preserved at every point of interaction
  5. Scalability and Performance– horizontal /elastic scaling is the ability of your infrastructure to automatically scale depending on the load or requests. Since your application will now be a public offering, you should be able to handle a significant increase in the number of users without degrading response time and performance. Use load balancing techniques to be able to achieve this
  6. Configurability – you need to provide the ability for customers to configure and customize settings over the internet – could be functionality or look and feel or preferences
  7. High Availability – you need to make sure that your application is running 24/7. Monitoring availability and having failover options are extremely critical
  8. Modern User Interfaces – if your product does not have a sexy UI and does not support mobile, you may run into adoption issues.

Development Model: The last but not the least important aspect you need to implement is a DevOps model for continuous delivery and integration. DevOps is essentially a software development method that requires collaboration between IT operations and application developers to enable continuous delivery. In this model application changes go into production multiple times a day or week to support the demands of customers. These upgrades and changes need to be non-disruptive with zero downtime and should be regression free.  Here is a great article on adopting DevOps

This checklist is a good starting point to discuss how to convert  an on-premise offering to a SaaS. Would love to hear your experiences ..