Jul 26, 2013

Performance Metrics and Successful Products

Its been a while since my last post, but I definitely want to keep this blog up and to stay active here.  So, here's another one.

Terms


"the team" - The team that works together to execute the vision of a product in order to deliver it to the customer so that we all get paid at the end of the day.

Who should care about performance metrics?


Well, everyone on the team.  Several articles and videos on performance come to mind.  I've heard it mentioned that improving the performance of a web application makes people lots more money and increases the engagement percentage.  I'm sure you've had the experience of dealing with slow things versus a performant system of some sort.  When things perform well, you have a positive experience (providing that they work properly, also).  When things are slow, even if they work properly, you still have a bad experience.

I would encourage product managers that read this post to consider adding functionality to their product that will help the team to measure the performance compared to other valuable metrics which are normally used to identify success like conversions, time-per-page, time-logged-in, time-on-the-site, number-of-users, etc.  This helps to show how performance of the product is directly tied to the success of the product as a whole.
Now, don't get me wrong, its hard to build things like this into the product.  For one, it slows the overall product down if you do it in the wrong way.  And two, its hard to get the management support for something that, in the end, the user should not be impacted by (if its done right).

What should be measured (and by whom), when measuring performance metrics?


I've specifically added the "by whom" part here because of a recent experience.  If one is stress/load testing a system then one should not really expect the system itself to be able to report its own performance metrics in a timely fashion.  That's because it should be too busy doing what it normally does to be able to respond to you.  That said, what should or can be measured might include the following:

  • Time per database transaction/query.
  • Time to render the page.
  • Time to get a piece of data from point A to point B.  
  • Time to perform [some cool operation].
  • Time to complete a method call.
  • Time to construct/destruct an object.

If you haven't already noticed, there's an important pattern here.  Its the time taken to perform an operation:  (end time stamp - start time stamp) / (number of operations performed).  If the number of operations is quite large and your timing each operation individually, then the timing itself may be severely affecting the performance of the application and I would recommend you time in batches, or time the whole group of operations together and take the average.

Where should the measuring of performance metrics be performed?


This might sound strange to product owners or quality assurance personnel, but performance metrics should be taken from a system that is built, configured, started, run, stopped, and torn down by an automated process.  I would recommend using Vagrant with Puppet here (if you're using *nix), but really any scripted automation combined with virtualization should do the trick.  Start small, iterate quickly, and don't be afraid of change as its built.  Also, don't couple your testing/measuring infrastructure to your product.  If you build a super cool way to compute the number of seconds between two time stamps, don't use that same library when measuring performance.  If that library breaks, for whatever reason, your testing infrastructure is also hosed.

When should performance metrics be measured?


From the beginning, frequently through the middle, and always at the end.  That pretty much covers it. :)

You should probably build performance testing, after its automated into the build steps.  If you have a continuous integration environment setup, then you probably already have automated unit testing, and then functional testing.  Assuming you have:  Made code changes, run unit tests locally and they pass, checked-in code changes, pushed commits to remote, the following automated steps could be setup:

  1. Unit tests are executed again
  2. If the unit tests pass, setup one or more VMs for functional testing with minimal hardware specifications.
  3. Perform functional tests.
  4. Tear down the VMs used for functional testing.
  5. If the functional areas perform as expected, setup one or more VMs for performance testing with more realistic/production-like hardware specifications (one might even use Amazon EC2 for this).
  6. Tear down the VMs used for performance testing.
  7. If the product performs as expected, then deploy to staging and notify QA to manually verify.
  8. If the product looks good after cursory review by QA personnel in staging and test results from unit/functional/performance tests look good, then deploy to production.
Whoa.  Did you say push to production after a cursory glance at test results and the product itself?  Yes, I  did.  :)  That's how a beautiful, automated, continuous integration environment is supposed to work.

Why should we measure performance metrics at all?


Because, in the end, that's what your customers want.  They want a product that they can use, and usability is directly tied to the product performing well.


-- Hope you liked this article.  My next article is going to be about performance issue investigation.

No comments:

Post a Comment