Samuel Prasad, Ph.D.

Flight of the Buffalo

This article is as much a review of the book titled “Flight of the Buffalo” by Belasco and Stayer as it is a description of my journey from software engineer to a technology manager.  Although the book was published almost 20 years, the ideas espoused by the authors are very relevant even today as more companies are adopting a flat management hierarchy and implementing agile methodologies.  

I was first given the opportunity to lead a project in the early 1990s at the company where I was working as a lead software programmer. It involved the development of a product that was targeted to be the flagship offering of the company. It was an opportunity that I could not pass up because it not only involved the use of cool new technologies, but I also saw it as a good opportunity to get into "management".

As was expected from a project manager, I became the owner of the implementation plan and responsible for the delivery of the product. I had a team of software developers who executed the plan. When there were problems they brought it to my attention. I jumped in, helped them find solutions (whenever I could); I sat in on meetings and helped them make decisions as and when needed. Everything seemed to be fine.

But a few months into the project I realized that it was not moving at the pace that I had expected, and  to make things worse, I was working harder than ever before in spite of scheduling and delegating tasks. The team executed the tasks diligently. They reported their progress. I soon realized that I had to work longer and longer hours to keep things moving, especially as the size of the team increased, because I was becoming a bottleneck.

Belasco and Stayer in their book refer to this style of leadership as the “The Head Buffalo and the Herd.” I was playing the role of the head-buffalo. The buffalo in the herd are loyal to their leader. They follow the leader and go wherever he wants them to go. When the head-buffalo is absent, the herd stand around waiting, not because they are lazy, but because they have been trained and conditioned to wait for directives and follow the leader.  In a situation like this the herd is vulnerable and can be completely decimated when the head buffalo is removed.

"This approach is based on leaders making decisions and motivating employees to do their job. But the downside is that it also robs leaders of effective power because in order for employees to be motivated to fix a problem or make a solution work, the leader must transfer ownership of the actions to the employees who need to be responsible.  People rise to the challenge – when it is their challenge.“

The authors find that productivity and innovation increases when the team works as a “A Flock of Geese” rather than a "Buffalo Herd". Different geese take the lead at different times and their roles change from leader to follower to scout as the gaggle flies in a V-formation. Each goose is responsible for getting itself to the gaggle’s destination and they land in waves indicating that no one is a leader.

In her book, “Art and Science of Leadership and Life Management", the author Afsaneh Nahavandi writes, “When geese fly in the V formation, the whole flock adds considerably more to its flying range than if each bird flew alone. Whenever a goose falls out of formation, it suddenly feels the drag and resistance of trying to fly alone and quickly gets back into formation to take advantage of the power of the formation. When the lead goose gets tired, it rotates back in the wing, and another goose flies to the point position. The back geese honk from behind to encourage those up front to keep up their speed. Finally, when a goose gets sick and falls out, two geese fall out of formation with it until it is either able to fly or it is dead. They then launch on their own, or with another formation, to catch up with the group."

Following are some key take-aways:

• “Transform ownership for work to those who execute the work. Empower!
  • Empowerers are proactive in empowering.
  • Empowerers proactively ask questions.
  • Empowerers proactively insist on meeting tough standards.
  • Empowerers proactively confront people with reality.
  • Empowerers proactively support and coach.
• Create the environment for ownership where each person wants to be responsible for his/her performance.
  • Paint a clear picture of great performance for the organization and each person.
  • Focus individuals on the few factors that create performance.
  • Develop the desire for each person to own – be responsible for – his/her own great performance. Stop acting like the head buffalo.
  • Engage individuals – their hearts and minds –in the business of the business. If you want them to act like it is their business, make it their business.
  • Energize individuals around the focus of business which is to understand the customer.
• Coach the development of individual capability and competence. Anyone’s a coach. Everyone’s a coach.
  • Learn faster.
  • Learn how to learn.
  • Create an environment where every person is challenged to continually learn.
  • Leadership behavior does not come easy. It requires learning.
  • Leaders focus on great performance for customers. Focus provides motivation to delight customers.
  • When the going gets tough, it is usually too late to get going. Anticipate problems rather than solve them.
  • Everyone should be responsible for managing the future if he/she doesn’t want to become a thing of the past.
  • Manage backward from the future. In other words, begin with the end in mind – the Federal Express example."

  Don't be a buffalo. Sign up for flying lessons.

As a Project Manager, is “sandbagging” ethical?

The above question was a topic of discussion in one of the forums on LinkedIn recently. "Sandbagging" refers to delaying (or timing) the reporting of certain project accomplishments ("good news") to maximize the (monetary) benefits and goodwill that may be accorded to the project manager and team.

Reporting the status of a project accurately and honestly is the right thing to do. But I find it troubling and disheartening that the above topic even warrants a discussion. It is an indication of a lack of trust between the business side and the project team.

Speaking from my area of expertise - IT/technology projects - the reasons for this mistrust are manifold.
 
The project manager is forced to commit to a plan with dates and deliverables even before the project starts. Much of the plan is speculative during the initial phases, which means that the PM is forced to pad the time and resource estimates because, as we know, once a plan is published it is cast in stone and expected to be signed in blood by the PM. (Come on guys...how many of you have faced this situation? The requirements are not clear. You are being forced to deliver a plan. You have no idea how long it will take? What do you do? You take a best guess and double it!! Sounds familiar?)
 
Well, the business side knows that the PM has padded the estimates. So they force arbitrary dates and deliverables on the PM. What happens next? The PM and team take short cuts and cobble together something in order to show progress to keep the business side happy and off their backs. This is a terrible waste of money, resources and time. Why do we do this? The team instead of building a strong foundation based on sound architectural decisions that can support the business requirements, find themselves spending much of their time building specifically for the "show and tell" sessions to keep the business side happy.
 
If the team, in the unlikely event find themselves ahead of the planned schedule, they are reluctant to divulge this information to the business side because the common reaction from business is to further squeeze the team on dates and deliverables.
 
It is about time that project teams are cross functional - business stake holders, users, programmers, analysts - must be part of the project team and fully committed. Remember the “Pig & Chicken” story.

A Pig and a Chicken are walking down the road. The Chicken says, "Hey Pig, I was thinking we should open a restaurant!" Pig replies, "what would we call it?" The Chicken responds, "How about 'ham-n-eggs'?".
The Pig thinks for a moment and says, "No thanks. I'd be committed, but you'd only be involved!”

We want pigs, not chickens.

The project execution timeframe is dependent on the entire team (not just programmers). As a team we need to understand that a project plan continually evolves. Good things happen because of the entire team. Ditto for delays and setbacks. We must celebrate successes and reflect on setbacks as a team. Project delays or cost overruns are not always because of the project manager and/or the team. Business stake holders are the culprits in many instances. They need to be more involved and take on more ownership of the project. The project manager can only be as effective as the support he/she gets from the business side.

Hopefully a cross-functional team working in a collegial environment – where everyone is fully committed - will obviate the need for "sandbagging".

Hey, a man can dream, can't he?

Alpha Leaders vs. Alpha Males

We need Alpha Leaders, not Alpha Males to run companies. History has shown that Alpha Leaders tend to think big, take bold and decisive actions, which in turn engenders change in the way we work, live and interact with others. Alpha Males lead by using the command and control approach, which is becoming less important in an increasingly service oriented world that depends on innovation and team participation to survive. Alpha Leaders and Alpha Males are not specific to any one gender. Both share many similar character traits. However, there are some characteristics that are unique to an Alpha Leader or Alpha Male. Following is a partial list.

Alpha Leader Practices collaborative leadership. Often values intangibles. Is comfortable with criticism from peers and members of the team. Learns from failure. Understands that experimentation is key to innovation. Expands their leadership teams. Trusts and delegates. Surrounds themselves with talented and independent thinkers. Protects the team. Understands the importance of work-life balance, which tends to brings out enthusiasm and creativity in the team.

Alpha Male Wants the final decision immediately. Is transparent / blunt. Often burns out because they are inherently adversarial. Has no patience for marginal personnel. Enjoys, almost lusts for competition. Doesn’t like surprises. Outworks everyone. Suspects people who are always positive. Determines the mood of every room they enter. Views shyness as weakness. Grows increasingly frustrated by mistakes. Experiences high staff turnover. Often destructs self and the team. Often judges people by externals. Is conscious of the perceptions of others.

What a difference? One is focused on the betterment of the team, the other, self-preservation. It is fair to conclude that people who are confident, secure and experts in their professional field may not want to be led by an Alpha Male. Alpha Leaders, on the other hand, are naturals at bringing the best in any team, from innovators to thinkers to workers by encouraging a participatory style of team dynamics.

Peter Drucker, a well-known professor and management consultant said it well, ""The leaders who work most effectively, it seems to me, never say 'I'. And that's not because they have trained themselves not to say 'I'. They don't think 'I'. They think 'we'; they think 'team'. They understand their job to be to make the team function. They accept responsibility and don't sidestep it, but 'we' gets the credit.... This is what creates trust, what enables you to get the task done."

For a team to function effectively there has to be implicit trust between all members. The leader must be confident that the team has his/her back and vice versa. In many instances we spend a lot of time hunting for keywords on resumes and quizzing prospective candidates on technical issues before making a hire. Trust seems to be completely left out of the hiring equation.

Major General William A, Cohen, an Alpha Leader in his own right, writes, ""My research debunks the myth that many people seem to have . . . that you become a leader by fighting your way to the top. Rather, you become a leader by helping others to the top. Helping your employees is as important, and many times more so, than trying to get the most work out of them."

"The most valuable "currency" of any organization is the initiative and creativity of its members. Every leader has the solemn moral responsibility to develop these to the maximum in all his people. This is the leader's highest priority.", said Dr. W. Edwards Demming.

Dr. Suzanne Zoglio, founder of the Institute of Planning and Development based in Doylestown, PA, says, "as leaders shift their focus to customers and quality, they realize that the old authoritarian leadership style does not work anymore. To achieve quality, service, and rapid response, leaders must utilize all available talent. They must find ways to inspire, involve, and empower employees. They must create a work environment that encourages commitment, innovation, and cooperation. Instead of evaluating, leaders now coach. Instead of doing, they delegate. Instead of telling, they facilitate. No one is expected to boss anyone. Everyone is expected to participate."

Are you an Alpha Male? If you are, then your leadership style is becoming obsolete. It does not build strong teams. It does not inspire people to do their best. If your primary motivations are ego, money and status, you are destroying creativity in your teams. The old command and control leadership style is becoming increasingly irrelevant as innovation, commitment and participatory management are taking over. Gorillas may still need Alpha Males to lead their group, not organizations made up of human beings!

Multitasking - are we getting more things done?

Classical psychology text books tell us that people cannot multitask. They say it's psychologically impossible. But we are multitasking more and more, not only in our workplace but also at home. Are we really getting more things done? Not really, say Professor Clifford Nass and his colleagues Eyal Ophir and Anthony Wagner of Stanford University. In a study of two groups of people - one made up of heavy multitaskers and the other made up of people who did not multitask as much - they found that the heavy multitaskers consistently underperformed the light multitaskers in problem solving because the heavy multitaskers allowed themselves to be distracted by superfluous or irrelevant information, which they were specifically instructed to ignore. Wagner, an associate professor of psychology said, "When they're in situations where there are multiple sources of information coming from the external world or emerging out of memory, they're not able to filter out what's not relevant to their current goal. That failure to filter means they're slowed down by that irrelevant information."

Other studies seem to support the above conclusions including the one conducted by two French scientists Sylvain Charron and Etienne Koechlin of the Institut National de la Santé and Ecole Normale Superieure, who say that the human brain cannot simultaneously pursue multiple goals. It is overwhelmed when dealing with multiple tasks resulting in irrelevant information being stored in memory. This not only impairs the brain's ability to function optimally when required to solve a specific problem but contrary to popular belief the study also found that multitaskers take longer to switch from one task to the other when compared to light multitaskers.

In the January 2012 issue of Harvard Health, P.J. Skerrett, senior editor, quotes Paul Hammerness and Margaret Moore, authors of Organize Your Mind, Organize Your Life, as saying, "multitasking increases the chances of making mistakes and missing important information and cues. Multitaskers are also less likely to retain information in working memory, which can hinder problem solving and creativity. Instead of trying to do several things at once—and often none of them well—Hammerness and Moore suggest what they call set shifting. This means consciously and completely shifting your attention from one task to the next, and focusing on the task at hand. Giving your full attention to what you are doing will help you do it better, with more creativity and fewer mistakes or missed connections. Set shifting is a sign of brain fitness and agility, say the authors."

However there are some who argue that a lack of focus can sometimes be good for creativity. The results of a new study by Kelvin Lui and Alan Wong from The Chinese University of Hong Kong, released on April 12, 2012, says people aged 19 to 28 who frequently use different types of media at the same time appear to be better at integrating information from vision and hearing when asked to perform a specific task. Using both vision and hearing to perform a specific task is not multitasking. It's the way humans normally work, no matter what the task.

Rantings of a Quality Conscious Software Engineer

The following is an excerpt from the editorial that appeared in the April 2012 issue of the ASQ Quality Islander.

I have been a software engineer for most of my adult life. The good news is that software still fascinates me. We are going through an incredible period of technological innovations, which has forced the software industry to constantly change to accommodate new programming paradigms. We started with procedural programming; moved to structured, functional, event-driven, object-oriented and a host of other programming methodologies. The bad news is that software quality continues to be terrible when compared to hardware, pharmaceutical, manufacturing and other industries. . Reportedly, about 70% of software projects fail according to the Standish Report. There have been several studies by organizations like Mercer Consulting, British Computer Society, National Institute of Standards and Technology (NIST), Tata Consultancy and Association of Computing Machinery (ACM), who have attempted to estimate the cost of software failures. While these studies do not agree on the percentage of failed projects, there is general consensus that the cost is upwards of $100 billion per year.

Don't get me wrong. The evolution of software engineering has been a tremendous success. We only need to look at our computers, tablets, e-book readers, smart phones and other hand-held devices to understand how they have dramatically transformed our everyday lives, primarily because of the software applications that run on them. But that success has a darker side to it - one that is plagued with catastrophic failures. Some examples of these major software failures can be found at:
Software Hall of Shame (from IEEE article,  “Why Software Fails”)
History's Worst Software Bugs (Wired)

Why haven't we learned from these failures? Why does software quality continue to be an issue even in 2012?  Watts Humphrey, a senior fellow at the Software Engineering Institute at Carnegie-Mellon University, says that unrealistic construction schedules is the primary reason. He writes, "Two questions have often bothered me about software work. First, why do competent software engineers agree to completion dates when they have no idea how to meet them? Second, why do rational executives accept schedule commitments when the engineers offer no evidence that they can meet those commitments? Where software is concerned, many otherwise hardheaded executives willingly accept vague promises and incomplete plans."

Unrealistic schedules force engineers to skip quality control checks during the elaboration and construction phases of the software project. Management and business stakeholders push to deliver the software to market with little testing. The end result is usually not good. There is an old adage in software engineering, “If it doesn’t have to work, we can build it really fast.”

As software engineers, it is our primary responsibility to build software that meets specifications. There are no short cuts. A robust set of quality assurance standards together with a comprehensive quality control test plan must be part of any software project.

Samuel Prasad, PhD, PMP, PMI-ACP, CSQE

The Agile Manifesto

When I reflect on the various software projects I have either worked on, or managed in the last two decades, it soon becomes apparent that all my successful projects have had some things in common. They are:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

The four items mentioned above happen to be the core elements of "The Agile Manifesto”. In this blog, I want to focus on the second item, which is "working software over comprehensive documentation." Scott Ambler of IBM says that the critical points to consider in deciding documentation requirements for a software project are:

• “Document stable things, not speculative things.
• Take an evolutionary approach to documentation development, seeking and then acting on feedback on a regular basis.
• Prefer executable work products such as customer tests and developer tests over static work products such as plain old documentation (POD).
• You should understand the total cost of ownership (TCO) for a document, and someone must explicitly choose to make that investment.
• Well-written documentation supports organizational memory effectively, but is a poor way to communicate during a project.
• Documentation should be concise: overviews/roadmaps are generally preferred over detailed documentation.
• With high quality source code and a test suite to back it up, you need a lot less system documentation.
• Comprehensive documentation does not ensure project success; in fact, it increases your chance of failure.
• Models are not necessarily documents, and documents are not necessarily models.
• Documentation is as much a part of the system as the source code.
• Your team’s primary goal is to develop software; its secondary goal is to enable your next effort.
• The benefit of having documentation must be greater than the cost of creating and maintaining it.
• Developers rarely trust the documentation, particularly detailed documentation because it usually is out of sync with the code.
• Ask whether you NEED the documentation, not whether you want it.
• Update documentation only when it hurts.”

Reasons why you should document:

To turn conceptual ideas into working code - I have found it very useful to put my thoughts down on paper in the form of a high-level logical flow of the functions or parts of a system that I plan to implement. I usually do this immediately after a brainstorming session with my team to ensure that I have not only captured all the essential features that must be implemented, but also to verify that there are no holes in the handling of errors and exceptions.

To establish unambiguous interface contracts -the famous line from "Meditation XVII" by the English poet John Donne - "no man is an island entire of itself; every man is a piece of the continent, a part of the main." - holds true for software systems as well. No software application can exist in isolation without human and/or data interactions. These interactions typically referred to as "application programming interfaces" (APIs) must be clearly defined and documented so that the different parties - humans and/or machines - involved in the data exchange understand the types of information they are likely to receive from, and expected to send to another human or software application.

To be in compliance with regulatory requirements -software must satisfy the documentation requirements imposed by either the government or the industry. Unfortunately many of these requirements are formulated by bureaucrats as a knee-jerk response to absolve them of any liability should the software malfunction. As software engineers it is incumbent upon us to thoroughly read and understand the regulatory requirements so that any documentation that is produced is just right for compliancy and not an overkill.

To identify operational steps after the software has stabilized - this document usually provides instructions for installing and running the software. It specifies the infrastructure required to run the software, schedules for backing up data, steps for scaling up or down the infrastructure, and procedures on how to handle system failures.

To help users effectively utilize all features of the software - also called a user's manual; this is a reference guide to the people who will be using the software. It should be written in a way that allows them to quickly find out how to accomplish a certain task via the software.

To provide overviews to stakeholders and the team - these documents which articulate the objectives of a software project come in two flavors - one for the business; the other for members of the implementation team. The former highlights the business objectives, the latter focuses on topics like the construction environment, source code control, database entities and descriptions of established (not speculative) models.

 Reasons why you should not document:

The business stakeholders require it but it doesn't add any value - these documents usually consist of Gantt charts, architecture models, detailed sequence and use-case diagrams which the stakeholders like to "sign off" on, because it gives them the illusion of being in control, and bolsters their mistaken belief that the project will succeed, because how can it not, when the team has been able to generate such a plethora of detailed information even before a single line of code has been written? When you are faced with a situation like this, my advice is to clearly communicate the cost of creating such documents; highlight the lack of any useful function that these documents provide since we cannot document the unknown, and for the software project to succeed we must ever be prepared to handle changes as the system evolves during its lifecycle; and last but not least let the stakeholder know that he/she might be responsible for the failure of the project by insisting on creating documents that are speculative in nature.

Ambler lists the following as "questionable" reasons for creating documentation,

• "The requester wants to be seen to be in control. 
• The requester mistakenly thinks that documentation has something to do with project success.  Nothing could be further from the truth.
• The requester wants to justify his/her existence. 
• The requester doesn’t know any better. 
• Someone wants reassurance that everything is okay." 

Software construction is innovative and creative in nature; any attempt to commoditize and turn into a manufacturing-style process usually does not work. My advice - be innovative, be rational and logical, keep it simple; there is no document that describes the software better than well-written code; there is no better indicator of progress than seeing "working software" evolve; document only when it is necessary and only when it hurts.

2012 Technology Predictions

There are three things in life that are certain - death, taxes and the crystal ball dropping in Times Square (New York City) on New Year's Eve. Truth be told, I did not really see the ball drop this year because the TV channel that my wife and I were watching got to the ball only after it had slid almost to the bottom of the pole. Anyway this is a good time to peer into the crystal ball and make a few predictions for 2012. While most of the experts seem to be fixated on the number 10, I'll limit my predictions to THREE. Here they are:

1. THE SMART PHONE REPLACES YOUR WALLET AND YOUR ID

NFC (near field communication) equipped smart phones will be widely available in the U.S. through most of the major carriers before the end of 2012. How does NFC work? The technology behind NFC uses inductive coupling to transfer data between two NFC-equipped devices when they are within a few centimeters of each other. Europe is way ahead of us in the use of NFC-equipped smart phones as a payment method and for identification purposes. The customer just taps the phone at the POS register instead of swiping a credit card. This method of payment is quite secure because of two-factor authentication which requires the customer to key in a PIN or through biometric scanning of the customer's fingerprint. Multiple credit cards, store cards, loyalty cards and bank accounts can be linked to a single NFC phone which means that a wallet full of cards is on its way to becoming obsolete.

The ability of a smart phone to display 2-D bar codes or other visual tags will increasingly be used in lieu of boarding passes at airports, driver's licenses and other forms of paper identification. NFC phones used in conjunction with two-factor identification eliminates many of the problems with stolen IDs and credit cards.

By the year 2015, at least 30% of the people will be "wallet-less".
 

2. ENTERPRISES STAY AWAY FROM THE PUBLIC CLOUD

Companies will continue to stay away from the public cloud because of security concerns. Security in the public cloud is still very elusive. According to Vivek Wadhwa, a columnist for Bloomberg Businessweek and the Washington Post, "Cloud computing provides significant cost savings and operational advantages. But it also unleashes a Pandora’s box of security concerns. We’ve already seen cloud break-ins originating from China. And a number of legitimate Internet companies have suffered when the FBI confiscated a shared server in a cloud hosting facility that also hosted rogue applications."

A "public cloud" means that a company's applications and data are hosted on servers and networks that are outside the company's firewall managed by a third-party, which means that the company has to rely on the cloud service provider to safeguard the company's application and data. The public cloud will still gain popularity among start-ups for testing new products because of the cost benefits it provides.

Hybrid Clouds are beginning to emerge, but I am not sure if they will be able to address all the security concerns adequately. What are hybrid clouds? They are a combination of public and private clouds where some of the company's software and/or data resides on the public cloud and some on its private cloud which is essentially the company's internal network.

Cloud service providers have a long way to go before enterprises will entrust their applications and data to the public cloud.

3. THE AGILE MANIFESTO IS ADOPTED BY MORE COMPANIES

The majority of technology projects are behind schedule, over budget and/or does not meet specifications according to the results of a study published earlier this year by "The Standish Group". The primary reason for this high rate of failure is the continuing use of outdated project management methods that rely on a command-and-control approach where most decisions related to design, cost and time are made at the top based on incorrect information. Management is lulled into a false sense of security when they buy into the notion that projects can be executed successfully by creating detailed plans upfront, when many of the parameters and constraints needed to create such plans are yet unknown. We've all heard about the urban myth that a project plan must spell out everything in minute detail including the condition “..in case someone falls under a bus..”.  This is a hangover from the DoD standards. If you are building nuclear reactors it’s a good idea to spell out everything and back-up every decision with a series of checks and signatures. But most of us are not building nuclear reactors, and for management to insist that project plans be created with an excruciating level of detail that is both unnecessary and inaccurate before the project can even start is ludicrous. (see http://www.iso-9001-checklist.co.uk/tutorial/4.2-documentation.htm).

In order to increase the probability that a project will be successful it is my opinion that the only solution is to adopt the "Agile Manifesto". It is refreshing to see that it is being adopted by more and more companies as they realize that traditional project management is just not working. The manifesto for agile software development is as follows:

• "Individuals and interactions over processes and tools.
• Working software over comprehensive documentation.
• Customer collaboration over contract negotiation.
• Responding to change over following a plan."

The twelve principles behind the Agile Manifesto are:

• "Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
• Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.
• Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
• Business people and developers must work together daily throughout the project.
• Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
• The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
• Working software is the primary measure of progress.
• Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
• Continuous attention to technical excellence and good design enhances agility.
• Simplicity--the art of maximizing the amount of work not done--is essential.
• The best architectures, requirements, and designs emerge from self-organizing teams.
• At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly."

More information about the agile manifesto can be found at http://agilemanifesto.org/.

I believe that companies will have no choice but to adopt agile management processes to not only retain their best talent but also to stay competitive in today's global market.

The Year of BIG DATA

Happy New Year!

"Let this coming year be better than all the others. Vow to do some of the things you've always wanted to do but couldn't find the time. Call up a forgotten friend. Drop an old grudge, and replace it with some pleasant memories. Vow not to make a promise you don't think you can keep. Walk tall, and smile more. You'll look ten years younger." - Ann Landers

Unlike the hundreds of thousands of people who braved the cold and thronged to the crossroads of the world - the heart of Times Square in New York City - to watch the ball drop at the stroke of midnight to usher in 2012, many of us were quite content to sip champagne and watch the events unfold on television from the warmth of our living rooms. Here is an excerpt from an editorial written by me that appeared in the January 2012 issue of The Quality Islander.

The New Year brings with it anticipation of new opportunities. In order to capitalize on these opportunities we need to be able to identify business trends and develop strategies that will deliver products with just the right features, reduce operational costs and improve service to our customers. How do we do this? Very simple! Understand customer behavior. This is where the ability to comprehend "Big Data” becomes critical for businesses to stay competitive and profitable.  IBM defines big data as the "2.5 quintillion (2.5 followed by 18 zeroes) bytes of data we create every day from sensors used to gather climate information, posts to social media sites, digital pictures and videos posted online, transaction records of online purchases, and cell phone GPS signals, to name a few. Ninety percent of this data has been created in the last two years alone."

I have spent much of last year working with big data. Until a year ago it was only the Internet companies who were interested in analyzing big data comprised mostly of click stream information to better understand the visitors to their web sites so that they can deliver advertisements with personalized contents. This type of data analysis, called data mining, is now increasingly being used by many companies including utilities, media companies, hospitals, banks and retailers. For example, utilities have started to mine data to predict customer demand for electric power and monitor the efficiency of the distribution grid. Media companies that deliver programming via the Internet or cable are now able to monitor in real-time the demography of their customers watching their content. This information can be used to develop customized programming at the household level and for targeted marketing.

Unfortunately, as big data grows there is a continuous erosion of our privacy because the results of the analyses of big data, called business intelligence (BI), are increasingly being used to make decisions that affect our everyday lives. BI used by hospitals directly determines the quality and type of medical care delivered to patients. BI used by banks determines the cost at which a customer can borrow money, which implies that not only must big data be clean but also the right metrics and models must be used to analyze big data.  As the volume of big data increases the complexity of the analyses can afford to decrease, according to a Gartner report which says, “Today, data mining is really about building sophisticated models with not very much data. Now, big data gives you huge volumes of data which means that you don’t need as sophisticated a model anymore.”  In other words, the cleanliness, consistency and integrity of data will be at least as important as the level of complexity of the analytical models.  In addition, non-traditional types of databases will be needed to store, classify and quickly access unstructured data from social media.

There are many new technologies scheduled for release this year that will help us better manage and understand big data. In order to stay competitive, companies will do well to carefully evaluate these new technologies for adaptability and scalability to make sure that the technology they choose is aligned with the types of data they collect and the needs of their customers.

Risk mitigation during software development

9th Annual Fall ASQ Conference
on
Risk Management

Presented by
Samuel Prasad, PhD, PMP, CSQE
Farmingdale State College
Friday, November 4th, 2011

Risk analysis and prioritization during the software development life cycle are key to controlling quality, managing costs and bringing a product to market on time. Learn the agile approach to risk analysis and how to identify applications that require more stringent testing because they are prone to higher risks. The results of risk assessments are used to create test scenarios and select specific test cases aimed at eliminating much of the defects prior to the construction phase. Most postmortems of software development projects indicate that construction costs could have been controlled and defects reduced if risk assessments are iterative and continuous throughout the development life cycle.

Click here to view presentation.

Data Quality - metrics must match business processes to get the best results!

What is good quality data?
Good quality data means that all master data is complete, consistent, accurate, time stamped and industry standards based.

What is the advantage of good quality data?
Good data quality will improve internal business processes for manufacturers, retailers, wholesalers, intermediaries and other third parties.

Will this add costs?
Instituting any data quality/accuracy program will involve additional resources, very much depending on the size of an organization and experience with data quality management.
But improving the quality of data from end-to-end will reduce costs, improve productivity and accelerate product speed to market. Poor data quality affects businesses negatively. It is hard to improve data quality if the relationship between data quality and business performance can be quantified. The challenge is to identify DQMs (data quality metrics) that are tightly coupled with business processes.

The following is an article written by David Loshin in Information Management Magazine, May 2005.

What makes a DQM?
More challenging, however, is that the individuals typically tasked with devising good DQMs are better trained at data analysis and less skilled in business performance monitoring. Therefore, part of this strategy is to understand the characteristics of a reasonable business performance metric and then explore how to map those characteristics to the measurable aspects of data quality. The following list of characteristics, which are by no means complete, should give some guidance as to how to jump-start the strategy:

o Clarity of definition
o Measurability
o Business relevance
o Controllability
o Representation
o Reportability
o Trackability
o Drill-down capability

Clarity of definition
Because the metric is intended to convey a particular piece of information regarding an aspect of business performance in a summarized manner, it is critical that its underlying definition be stated in a way that clearly explains what is being measured. In fact, each metric should be subject to a rigorous "standardization" process in which the key stakeholders participate in its definition and agree to the definition's final wording. In addition, it is advisable to provide the metric's value range, as well as a qualitative segmentation of the value range that relates the metric's score to its performance assessment.

Measurability
Any metric must be measurable and should be quantifiable within a discrete range. Note, however, that there are many things that can be measured that may not translate into useful metrics, and that implies the need for business relevance.

Business relevance
The metric is of no value if it cannot be related to some aspect of business operations or performance. Therefore, every desirable metric must be defined within a business context with an explanation of how the metric score correlates with a measurement of performance. More desirable is if performance measurement can be directly associated with a critical business impact; this is probably the most critical characteristic of a data quality metric.

Controllability
Any measurable characteristic of information that is suitable as a metric should reflect some controllable aspect of the business. In other words, the assessment of an information quality metric's value within an undesirable range should trigger some action to improve the data being measured.

Representation
Without digressing into a discussion about the plethora of visual "widgets" that can be used to represent a metric's value, it is reasonable to note that one should associate a visual representation for each metric that logically presents the metric's value in a concise and meaningful way.

Reportability
From a different point of view, each metric's definition should provide enough information that can be summarized as a line item in a comprehensive report. The difference between representation and reportability is that the representation will focus on the specific metric in isolation, while the reporting should show each metric's contribution to an aggregate assessment. In turn, this allows the manager to evaluate the priority of any issues needing resolution.

Trackability
A major benefit of metrics is the ability to measure performance improvement over time. Tracking performance over time not only validates any improvement efforts, but once an information process is presumed to be stable, tracking provides insight into maintaining statistical control. In turn, these kinds of metrics can evolve from performance indicators into standard monitors, placed in the background to notify the right individuals when the data quality measurements suddenly indicate a deviation from expected control bounds.

Drill-Down capability
In recognition of the summarization aspect of a representation of a data quality metric, the flip side is the ability to provide exposure to the underlying data that contributed to a particular metric score. The natural instinct, when reviewing data quality measurements, is to review the data instances that contributed to any low scores. The ability to drill down through the performance metric allows an analyst to get a better understanding of patterns (if any exist) that may have contributed to a low score, and consequently use that understanding for a more comprehensive root-cause analysis. This kind of insight allows your organization to isolate the processing stage at which any flaws are introduced and, in turn, enables you to eliminate the source of the introduction of data problems (instead of the typical, counterproductive reaction of correcting the data values themselves).

The challenge
Developing key performance indicators for information quality is clearly a challenge, mostly because the hard numbers presented by data quality tools are typically out of the business context. Here we have provided some insight into how the data analyst can work with the business customer to identify ways that poor data quality impacts the achievement of business objectives and subsequently determine hard costs associated with each occurrence of a flaw. Once this has been done, providing a dashboard with tracking and drilldown capabilities establishes a value-added approach for value-directed information quality management and improvement.