Samuel Prasad, Ph.D.

As a Project Manager, is “sandbagging” ethical?

The above question was a topic of discussion in one of the forums on LinkedIn recently. "Sandbagging" refers to delaying (or timing) the reporting of certain project accomplishments ("good news") to maximize the (monetary) benefits and goodwill that may be accorded to the project manager and team.

Reporting the status of a project accurately and honestly is the right thing to do. But I find it troubling and disheartening that the above topic even warrants a discussion. It is an indication of a lack of trust between the business side and the project team.

Speaking from my area of expertise - IT/technology projects - the reasons for this mistrust are manifold.
 
The project manager is forced to commit to a plan with dates and deliverables even before the project starts. Much of the plan is speculative during the initial phases, which means that the PM is forced to pad the time and resource estimates because, as we know, once a plan is published it is cast in stone and expected to be signed in blood by the PM. (Come on guys...how many of you have faced this situation? The requirements are not clear. You are being forced to deliver a plan. You have no idea how long it will take? What do you do? You take a best guess and double it!! Sounds familiar?)
 
Well, the business side knows that the PM has padded the estimates. So they force arbitrary dates and deliverables on the PM. What happens next? The PM and team take short cuts and cobble together something in order to show progress to keep the business side happy and off their backs. This is a terrible waste of money, resources and time. Why do we do this? The team instead of building a strong foundation based on sound architectural decisions that can support the business requirements, find themselves spending much of their time building specifically for the "show and tell" sessions to keep the business side happy.
 
If the team, in the unlikely event find themselves ahead of the planned schedule, they are reluctant to divulge this information to the business side because the common reaction from business is to further squeeze the team on dates and deliverables.
 
It is about time that project teams are cross functional - business stake holders, users, programmers, analysts - must be part of the project team and fully committed. Remember the “Pig & Chicken” story.

A Pig and a Chicken are walking down the road. The Chicken says, "Hey Pig, I was thinking we should open a restaurant!" Pig replies, "what would we call it?" The Chicken responds, "How about 'ham-n-eggs'?".
The Pig thinks for a moment and says, "No thanks. I'd be committed, but you'd only be involved!”

We want pigs, not chickens.

The project execution timeframe is dependent on the entire team (not just programmers). As a team we need to understand that a project plan continually evolves. Good things happen because of the entire team. Ditto for delays and setbacks. We must celebrate successes and reflect on setbacks as a team. Project delays or cost overruns are not always because of the project manager and/or the team. Business stake holders are the culprits in many instances. They need to be more involved and take on more ownership of the project. The project manager can only be as effective as the support he/she gets from the business side.

Hopefully a cross-functional team working in a collegial environment – where everyone is fully committed - will obviate the need for "sandbagging".

Hey, a man can dream, can't he?

Alpha Leaders vs. Alpha Males

We need Alpha Leaders, not Alpha Males to run companies. History has shown that Alpha Leaders tend to think big, take bold and decisive actions, which in turn engenders change in the way we work, live and interact with others. Alpha Males lead by using the command and control approach, which is becoming less important in an increasingly service oriented world that depends on innovation and team participation to survive. Alpha Leaders and Alpha Males are not specific to any one gender. Both share many similar character traits. However, there are some characteristics that are unique to an Alpha Leader or Alpha Male. Following is a partial list.

Alpha Leader Practices collaborative leadership. Often values intangibles. Is comfortable with criticism from peers and members of the team. Learns from failure. Understands that experimentation is key to innovation. Expands their leadership teams. Trusts and delegates. Surrounds themselves with talented and independent thinkers. Protects the team. Understands the importance of work-life balance, which tends to brings out enthusiasm and creativity in the team.

Alpha Male Wants the final decision immediately. Is transparent / blunt. Often burns out because they are inherently adversarial. Has no patience for marginal personnel. Enjoys, almost lusts for competition. Doesn’t like surprises. Outworks everyone. Suspects people who are always positive. Determines the mood of every room they enter. Views shyness as weakness. Grows increasingly frustrated by mistakes. Experiences high staff turnover. Often destructs self and the team. Often judges people by externals. Is conscious of the perceptions of others.

What a difference? One is focused on the betterment of the team, the other, self-preservation. It is fair to conclude that people who are confident, secure and experts in their professional field may not want to be led by an Alpha Male. Alpha Leaders, on the other hand, are naturals at bringing the best in any team, from innovators to thinkers to workers by encouraging a participatory style of team dynamics.

Peter Drucker, a well-known professor and management consultant said it well, ""The leaders who work most effectively, it seems to me, never say 'I'. And that's not because they have trained themselves not to say 'I'. They don't think 'I'. They think 'we'; they think 'team'. They understand their job to be to make the team function. They accept responsibility and don't sidestep it, but 'we' gets the credit.... This is what creates trust, what enables you to get the task done."

For a team to function effectively there has to be implicit trust between all members. The leader must be confident that the team has his/her back and vice versa. In many instances we spend a lot of time hunting for keywords on resumes and quizzing prospective candidates on technical issues before making a hire. Trust seems to be completely left out of the hiring equation.

Major General William A, Cohen, an Alpha Leader in his own right, writes, ""My research debunks the myth that many people seem to have . . . that you become a leader by fighting your way to the top. Rather, you become a leader by helping others to the top. Helping your employees is as important, and many times more so, than trying to get the most work out of them."

"The most valuable "currency" of any organization is the initiative and creativity of its members. Every leader has the solemn moral responsibility to develop these to the maximum in all his people. This is the leader's highest priority.", said Dr. W. Edwards Demming.

Dr. Suzanne Zoglio, founder of the Institute of Planning and Development based in Doylestown, PA, says, "as leaders shift their focus to customers and quality, they realize that the old authoritarian leadership style does not work anymore. To achieve quality, service, and rapid response, leaders must utilize all available talent. They must find ways to inspire, involve, and empower employees. They must create a work environment that encourages commitment, innovation, and cooperation. Instead of evaluating, leaders now coach. Instead of doing, they delegate. Instead of telling, they facilitate. No one is expected to boss anyone. Everyone is expected to participate."

Are you an Alpha Male? If you are, then your leadership style is becoming obsolete. It does not build strong teams. It does not inspire people to do their best. If your primary motivations are ego, money and status, you are destroying creativity in your teams. The old command and control leadership style is becoming increasingly irrelevant as innovation, commitment and participatory management are taking over. Gorillas may still need Alpha Males to lead their group, not organizations made up of human beings!

The Agile Manifesto

When I reflect on the various software projects I have either worked on, or managed in the last two decades, it soon becomes apparent that all my successful projects have had some things in common. They are:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

The four items mentioned above happen to be the core elements of "The Agile Manifesto”. In this blog, I want to focus on the second item, which is "working software over comprehensive documentation." Scott Ambler of IBM says that the critical points to consider in deciding documentation requirements for a software project are:

• “Document stable things, not speculative things.
• Take an evolutionary approach to documentation development, seeking and then acting on feedback on a regular basis.
• Prefer executable work products such as customer tests and developer tests over static work products such as plain old documentation (POD).
• You should understand the total cost of ownership (TCO) for a document, and someone must explicitly choose to make that investment.
• Well-written documentation supports organizational memory effectively, but is a poor way to communicate during a project.
• Documentation should be concise: overviews/roadmaps are generally preferred over detailed documentation.
• With high quality source code and a test suite to back it up, you need a lot less system documentation.
• Comprehensive documentation does not ensure project success; in fact, it increases your chance of failure.
• Models are not necessarily documents, and documents are not necessarily models.
• Documentation is as much a part of the system as the source code.
• Your team’s primary goal is to develop software; its secondary goal is to enable your next effort.
• The benefit of having documentation must be greater than the cost of creating and maintaining it.
• Developers rarely trust the documentation, particularly detailed documentation because it usually is out of sync with the code.
• Ask whether you NEED the documentation, not whether you want it.
• Update documentation only when it hurts.”

Reasons why you should document:

To turn conceptual ideas into working code - I have found it very useful to put my thoughts down on paper in the form of a high-level logical flow of the functions or parts of a system that I plan to implement. I usually do this immediately after a brainstorming session with my team to ensure that I have not only captured all the essential features that must be implemented, but also to verify that there are no holes in the handling of errors and exceptions.

To establish unambiguous interface contracts -the famous line from "Meditation XVII" by the English poet John Donne - "no man is an island entire of itself; every man is a piece of the continent, a part of the main." - holds true for software systems as well. No software application can exist in isolation without human and/or data interactions. These interactions typically referred to as "application programming interfaces" (APIs) must be clearly defined and documented so that the different parties - humans and/or machines - involved in the data exchange understand the types of information they are likely to receive from, and expected to send to another human or software application.

To be in compliance with regulatory requirements -software must satisfy the documentation requirements imposed by either the government or the industry. Unfortunately many of these requirements are formulated by bureaucrats as a knee-jerk response to absolve them of any liability should the software malfunction. As software engineers it is incumbent upon us to thoroughly read and understand the regulatory requirements so that any documentation that is produced is just right for compliancy and not an overkill.

To identify operational steps after the software has stabilized - this document usually provides instructions for installing and running the software. It specifies the infrastructure required to run the software, schedules for backing up data, steps for scaling up or down the infrastructure, and procedures on how to handle system failures.

To help users effectively utilize all features of the software - also called a user's manual; this is a reference guide to the people who will be using the software. It should be written in a way that allows them to quickly find out how to accomplish a certain task via the software.

To provide overviews to stakeholders and the team - these documents which articulate the objectives of a software project come in two flavors - one for the business; the other for members of the implementation team. The former highlights the business objectives, the latter focuses on topics like the construction environment, source code control, database entities and descriptions of established (not speculative) models.

 Reasons why you should not document:

The business stakeholders require it but it doesn't add any value - these documents usually consist of Gantt charts, architecture models, detailed sequence and use-case diagrams which the stakeholders like to "sign off" on, because it gives them the illusion of being in control, and bolsters their mistaken belief that the project will succeed, because how can it not, when the team has been able to generate such a plethora of detailed information even before a single line of code has been written? When you are faced with a situation like this, my advice is to clearly communicate the cost of creating such documents; highlight the lack of any useful function that these documents provide since we cannot document the unknown, and for the software project to succeed we must ever be prepared to handle changes as the system evolves during its lifecycle; and last but not least let the stakeholder know that he/she might be responsible for the failure of the project by insisting on creating documents that are speculative in nature.

Ambler lists the following as "questionable" reasons for creating documentation,

• "The requester wants to be seen to be in control. 
• The requester mistakenly thinks that documentation has something to do with project success.  Nothing could be further from the truth.
• The requester wants to justify his/her existence. 
• The requester doesn’t know any better. 
• Someone wants reassurance that everything is okay." 

Software construction is innovative and creative in nature; any attempt to commoditize and turn into a manufacturing-style process usually does not work. My advice - be innovative, be rational and logical, keep it simple; there is no document that describes the software better than well-written code; there is no better indicator of progress than seeing "working software" evolve; document only when it is necessary and only when it hurts.

2012 Technology Predictions

There are three things in life that are certain - death, taxes and the crystal ball dropping in Times Square (New York City) on New Year's Eve. Truth be told, I did not really see the ball drop this year because the TV channel that my wife and I were watching got to the ball only after it had slid almost to the bottom of the pole. Anyway this is a good time to peer into the crystal ball and make a few predictions for 2012. While most of the experts seem to be fixated on the number 10, I'll limit my predictions to THREE. Here they are:

1. THE SMART PHONE REPLACES YOUR WALLET AND YOUR ID

NFC (near field communication) equipped smart phones will be widely available in the U.S. through most of the major carriers before the end of 2012. How does NFC work? The technology behind NFC uses inductive coupling to transfer data between two NFC-equipped devices when they are within a few centimeters of each other. Europe is way ahead of us in the use of NFC-equipped smart phones as a payment method and for identification purposes. The customer just taps the phone at the POS register instead of swiping a credit card. This method of payment is quite secure because of two-factor authentication which requires the customer to key in a PIN or through biometric scanning of the customer's fingerprint. Multiple credit cards, store cards, loyalty cards and bank accounts can be linked to a single NFC phone which means that a wallet full of cards is on its way to becoming obsolete.

The ability of a smart phone to display 2-D bar codes or other visual tags will increasingly be used in lieu of boarding passes at airports, driver's licenses and other forms of paper identification. NFC phones used in conjunction with two-factor identification eliminates many of the problems with stolen IDs and credit cards.

By the year 2015, at least 30% of the people will be "wallet-less".
 

2. ENTERPRISES STAY AWAY FROM THE PUBLIC CLOUD

Companies will continue to stay away from the public cloud because of security concerns. Security in the public cloud is still very elusive. According to Vivek Wadhwa, a columnist for Bloomberg Businessweek and the Washington Post, "Cloud computing provides significant cost savings and operational advantages. But it also unleashes a Pandora’s box of security concerns. We’ve already seen cloud break-ins originating from China. And a number of legitimate Internet companies have suffered when the FBI confiscated a shared server in a cloud hosting facility that also hosted rogue applications."

A "public cloud" means that a company's applications and data are hosted on servers and networks that are outside the company's firewall managed by a third-party, which means that the company has to rely on the cloud service provider to safeguard the company's application and data. The public cloud will still gain popularity among start-ups for testing new products because of the cost benefits it provides.

Hybrid Clouds are beginning to emerge, but I am not sure if they will be able to address all the security concerns adequately. What are hybrid clouds? They are a combination of public and private clouds where some of the company's software and/or data resides on the public cloud and some on its private cloud which is essentially the company's internal network.

Cloud service providers have a long way to go before enterprises will entrust their applications and data to the public cloud.

3. THE AGILE MANIFESTO IS ADOPTED BY MORE COMPANIES

The majority of technology projects are behind schedule, over budget and/or does not meet specifications according to the results of a study published earlier this year by "The Standish Group". The primary reason for this high rate of failure is the continuing use of outdated project management methods that rely on a command-and-control approach where most decisions related to design, cost and time are made at the top based on incorrect information. Management is lulled into a false sense of security when they buy into the notion that projects can be executed successfully by creating detailed plans upfront, when many of the parameters and constraints needed to create such plans are yet unknown. We've all heard about the urban myth that a project plan must spell out everything in minute detail including the condition “..in case someone falls under a bus..”.  This is a hangover from the DoD standards. If you are building nuclear reactors it’s a good idea to spell out everything and back-up every decision with a series of checks and signatures. But most of us are not building nuclear reactors, and for management to insist that project plans be created with an excruciating level of detail that is both unnecessary and inaccurate before the project can even start is ludicrous. (see http://www.iso-9001-checklist.co.uk/tutorial/4.2-documentation.htm).

In order to increase the probability that a project will be successful it is my opinion that the only solution is to adopt the "Agile Manifesto". It is refreshing to see that it is being adopted by more and more companies as they realize that traditional project management is just not working. The manifesto for agile software development is as follows:

• "Individuals and interactions over processes and tools.
• Working software over comprehensive documentation.
• Customer collaboration over contract negotiation.
• Responding to change over following a plan."

The twelve principles behind the Agile Manifesto are:

• "Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
• Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.
• Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
• Business people and developers must work together daily throughout the project.
• Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
• The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
• Working software is the primary measure of progress.
• Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
• Continuous attention to technical excellence and good design enhances agility.
• Simplicity--the art of maximizing the amount of work not done--is essential.
• The best architectures, requirements, and designs emerge from self-organizing teams.
• At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly."

More information about the agile manifesto can be found at http://agilemanifesto.org/.

I believe that companies will have no choice but to adopt agile management processes to not only retain their best talent but also to stay competitive in today's global market.

The Year of BIG DATA

Happy New Year!

"Let this coming year be better than all the others. Vow to do some of the things you've always wanted to do but couldn't find the time. Call up a forgotten friend. Drop an old grudge, and replace it with some pleasant memories. Vow not to make a promise you don't think you can keep. Walk tall, and smile more. You'll look ten years younger." - Ann Landers

Unlike the hundreds of thousands of people who braved the cold and thronged to the crossroads of the world - the heart of Times Square in New York City - to watch the ball drop at the stroke of midnight to usher in 2012, many of us were quite content to sip champagne and watch the events unfold on television from the warmth of our living rooms. Here is an excerpt from an editorial written by me that appeared in the January 2012 issue of The Quality Islander.

The New Year brings with it anticipation of new opportunities. In order to capitalize on these opportunities we need to be able to identify business trends and develop strategies that will deliver products with just the right features, reduce operational costs and improve service to our customers. How do we do this? Very simple! Understand customer behavior. This is where the ability to comprehend "Big Data” becomes critical for businesses to stay competitive and profitable.  IBM defines big data as the "2.5 quintillion (2.5 followed by 18 zeroes) bytes of data we create every day from sensors used to gather climate information, posts to social media sites, digital pictures and videos posted online, transaction records of online purchases, and cell phone GPS signals, to name a few. Ninety percent of this data has been created in the last two years alone."

I have spent much of last year working with big data. Until a year ago it was only the Internet companies who were interested in analyzing big data comprised mostly of click stream information to better understand the visitors to their web sites so that they can deliver advertisements with personalized contents. This type of data analysis, called data mining, is now increasingly being used by many companies including utilities, media companies, hospitals, banks and retailers. For example, utilities have started to mine data to predict customer demand for electric power and monitor the efficiency of the distribution grid. Media companies that deliver programming via the Internet or cable are now able to monitor in real-time the demography of their customers watching their content. This information can be used to develop customized programming at the household level and for targeted marketing.

Unfortunately, as big data grows there is a continuous erosion of our privacy because the results of the analyses of big data, called business intelligence (BI), are increasingly being used to make decisions that affect our everyday lives. BI used by hospitals directly determines the quality and type of medical care delivered to patients. BI used by banks determines the cost at which a customer can borrow money, which implies that not only must big data be clean but also the right metrics and models must be used to analyze big data.  As the volume of big data increases the complexity of the analyses can afford to decrease, according to a Gartner report which says, “Today, data mining is really about building sophisticated models with not very much data. Now, big data gives you huge volumes of data which means that you don’t need as sophisticated a model anymore.”  In other words, the cleanliness, consistency and integrity of data will be at least as important as the level of complexity of the analytical models.  In addition, non-traditional types of databases will be needed to store, classify and quickly access unstructured data from social media.

There are many new technologies scheduled for release this year that will help us better manage and understand big data. In order to stay competitive, companies will do well to carefully evaluate these new technologies for adaptability and scalability to make sure that the technology they choose is aligned with the types of data they collect and the needs of their customers.

Risk mitigation during software development

9th Annual Fall ASQ Conference
on
Risk Management

Presented by
Samuel Prasad, PhD, PMP, CSQE
Farmingdale State College
Friday, November 4th, 2011

Risk analysis and prioritization during the software development life cycle are key to controlling quality, managing costs and bringing a product to market on time. Learn the agile approach to risk analysis and how to identify applications that require more stringent testing because they are prone to higher risks. The results of risk assessments are used to create test scenarios and select specific test cases aimed at eliminating much of the defects prior to the construction phase. Most postmortems of software development projects indicate that construction costs could have been controlled and defects reduced if risk assessments are iterative and continuous throughout the development life cycle.

Click here to view presentation.

Data Quality - metrics must match business processes to get the best results!

What is good quality data?
Good quality data means that all master data is complete, consistent, accurate, time stamped and industry standards based.

What is the advantage of good quality data?
Good data quality will improve internal business processes for manufacturers, retailers, wholesalers, intermediaries and other third parties.

Will this add costs?
Instituting any data quality/accuracy program will involve additional resources, very much depending on the size of an organization and experience with data quality management.
But improving the quality of data from end-to-end will reduce costs, improve productivity and accelerate product speed to market. Poor data quality affects businesses negatively. It is hard to improve data quality if the relationship between data quality and business performance can be quantified. The challenge is to identify DQMs (data quality metrics) that are tightly coupled with business processes.

The following is an article written by David Loshin in Information Management Magazine, May 2005.

What makes a DQM?
More challenging, however, is that the individuals typically tasked with devising good DQMs are better trained at data analysis and less skilled in business performance monitoring. Therefore, part of this strategy is to understand the characteristics of a reasonable business performance metric and then explore how to map those characteristics to the measurable aspects of data quality. The following list of characteristics, which are by no means complete, should give some guidance as to how to jump-start the strategy:

o Clarity of definition
o Measurability
o Business relevance
o Controllability
o Representation
o Reportability
o Trackability
o Drill-down capability

Clarity of definition
Because the metric is intended to convey a particular piece of information regarding an aspect of business performance in a summarized manner, it is critical that its underlying definition be stated in a way that clearly explains what is being measured. In fact, each metric should be subject to a rigorous "standardization" process in which the key stakeholders participate in its definition and agree to the definition's final wording. In addition, it is advisable to provide the metric's value range, as well as a qualitative segmentation of the value range that relates the metric's score to its performance assessment.

Measurability
Any metric must be measurable and should be quantifiable within a discrete range. Note, however, that there are many things that can be measured that may not translate into useful metrics, and that implies the need for business relevance.

Business relevance
The metric is of no value if it cannot be related to some aspect of business operations or performance. Therefore, every desirable metric must be defined within a business context with an explanation of how the metric score correlates with a measurement of performance. More desirable is if performance measurement can be directly associated with a critical business impact; this is probably the most critical characteristic of a data quality metric.

Controllability
Any measurable characteristic of information that is suitable as a metric should reflect some controllable aspect of the business. In other words, the assessment of an information quality metric's value within an undesirable range should trigger some action to improve the data being measured.

Representation
Without digressing into a discussion about the plethora of visual "widgets" that can be used to represent a metric's value, it is reasonable to note that one should associate a visual representation for each metric that logically presents the metric's value in a concise and meaningful way.

Reportability
From a different point of view, each metric's definition should provide enough information that can be summarized as a line item in a comprehensive report. The difference between representation and reportability is that the representation will focus on the specific metric in isolation, while the reporting should show each metric's contribution to an aggregate assessment. In turn, this allows the manager to evaluate the priority of any issues needing resolution.

Trackability
A major benefit of metrics is the ability to measure performance improvement over time. Tracking performance over time not only validates any improvement efforts, but once an information process is presumed to be stable, tracking provides insight into maintaining statistical control. In turn, these kinds of metrics can evolve from performance indicators into standard monitors, placed in the background to notify the right individuals when the data quality measurements suddenly indicate a deviation from expected control bounds.

Drill-Down capability
In recognition of the summarization aspect of a representation of a data quality metric, the flip side is the ability to provide exposure to the underlying data that contributed to a particular metric score. The natural instinct, when reviewing data quality measurements, is to review the data instances that contributed to any low scores. The ability to drill down through the performance metric allows an analyst to get a better understanding of patterns (if any exist) that may have contributed to a low score, and consequently use that understanding for a more comprehensive root-cause analysis. This kind of insight allows your organization to isolate the processing stage at which any flaws are introduced and, in turn, enables you to eliminate the source of the introduction of data problems (instead of the typical, counterproductive reaction of correcting the data values themselves).

The challenge
Developing key performance indicators for information quality is clearly a challenge, mostly because the hard numbers presented by data quality tools are typically out of the business context. Here we have provided some insight into how the data analyst can work with the business customer to identify ways that poor data quality impacts the achievement of business objectives and subsequently determine hard costs associated with each occurrence of a flaw. Once this has been done, providing a dashboard with tracking and drilldown capabilities establishes a value-added approach for value-directed information quality management and improvement.

It's the Data, Stupid! Why data is an organization's most important asset.

In the beginning there was the ODS or operational data store - typically a collection of tables in a relational database. The ODS contained the company's raw data which was used to run historical reports on past transactions. Then came the data marts (DM) essentially a snapshot of the ODS. The BI Analysts who were highly skilled statisticians and mathematicians ran queries against the DM, extracted data from the DM and performed their analyses. But all the analyses in the world could not deliver critical information in a timely manner to help management make better operational decisions to improve customer service and help the bottom line.

The good news is that DMs and Business Intelligence (BI) have evolved significantly over the past decade. It is no longer acceptable to have DMs that are just time-stamped snapshots of the ODS. The raw data in the ODS has to be cleansed, massaged and distilled...and I mean distilled...into useful information stored in formats like, cubes, dimensional entities and aggregates in columnar structures at various levels of granularity and recency so that the analysts can visualize, manipulate and study the data from different angles.

What has changed in BI in the past decade?

Data Frequency - real-time or close to real-time
Daily or weekly snapshots of the ODS are no longer acceptable for making decisions. This means that we need to have intra-day data marts as close to real-time as possible that contain only a subset of data that renders recent customer transactions and activities very quickly. Intra-day snapshots of data are trickle fed through data filters into real-time data marts, allowing operational personnel the ability to analyze events occurring during the day of the event. As you can imagine, this means that the volume of data being stored increases substantially. The main ODS and OLAP data mart now contain tens of terabytes to hundreds of terabytes (even petabytes!) to support all forms of BI now. And it should be noted that the granularity of the data for operational BI must be at the lowest level of detail. All this means that, not only does the ODS and OLAP DM infrastructure have to handle faster, more frequent loads of data, but seamless scalability is mandatory – whether it is for storing and processing increased volumes of data, or maintaining the integrity of the environment (backups, failovers, etc.)

Performance - instant gratification
While performance in traditional BI environments has always been important, it is now critical that responses to queries against the intra-day real-time data mart be almost instantaneous. In many instances, requests originate from and responses are received on handheld wireless devices. Communication is no longer one-way. In other words, users are able to send back comments and feedback which contain valuable information (unfortunately in unstructured textual format in many instances) that must be captured, parsed, stored and fed into the ETL process. Add to this the fact that the environment must still support traditional BI users. A mixed environment with ODS, the traditional data mart (OLAP) and the real-time data mart means the software must have the ability to prioritize and route queries, not only according to their importance to the enterprise but also based on data and response requirements.

Number of users - scalability
There was a time when Business Intelligence was the domain of only statisticians and mathematicians who extracted data from historical snapshots of the ODS to perform their magic. Gone are those days. Now BI and analytics are used by business users and even customers want to get into the act since they want access to their transaction history and past experiences at a particular store or web site. Here's another catch. Not all users are human. There are web-bots that roam in and around the data marts trying to detect patterns, do simulations and project outcomes. All of this puts a tremendous strain on the network, server and infrastructure that must be horizontally and vertically scalable to support the onslaught of requests, queries and analytics being thrown at them.

Unstructured Data - Facebook, YouTube, blogs and other social media
Unstructured data is increasingly being used to make critical decisions. The ETL process has to be retooled to handle different forms of textual, audio and video data. The technical challenges that this poses are quite steep since most BI environments are just not set up to handle unstructured data. The space requirements of the ODS and data marts are expected to grow exponentially  because of this; processing capabilities required to parse and convert unstructured data into a structured format puts a tremendous strain on the processing and infrastructure requirements, which in turn has a direct impact on scalability and performance.

The following diagram illustrates a typical BI system of today.

  
 

While the "Daily ETL" process takes in data from all data sources to perform incremental updates on the ODS and OLAP data mart, the "Real-time ETL" also called an intra-day ETL accepts transactions through a filter configured to feed only transactions that are deemed to meet the "recency" criterion. This essentially limits the data to transactions and customer behavior that are needed to make decisions based on events occuring during the day. Unstructured data coming back from a customer's handheld device is staged in a daily customer tracking DB and fed into the "Daily ETL" process along with data from all the regular data sources. It is also fed into the "Real-time ETL" process so that it is immediately available in the Real-time data mart.

Data marts and BI systems of the future must be completely rules-meta-data-driven capable of self-learning and able to modify their behavior based on usage and data types that are yet unknown.

Future of BI Analytics

The future of Business Intelligence (BI) Analytics rests on emerging technologies that will use special hardware appliances that will effectively mine data at tremendous speeds and perform predictive analysis and extrapolate trends using simulation in real-time. Traditional BI products, including dashboards and ad hoc queries use historical data to generate reports, which only tell the story of what happened in the past. In future, predictive analytics, alerts and extrapolation of trends will become common practice through the real-time exploitation of informational patterns hidden in historical data to project what might happen.

Hardware appliances like IBM's Neteeza and Oracle's Exadata provide extreme performance in both OLTP and OLAP models, allowing for very quick deep dives into the data specifically in the latter environment. These appliances are effectively supercomputers that can chew through large volumes of data very quickly.

In addition, the increasing popularity of columnar databases like Sybase IQ, HP Vertica and ParAccel allow data mining software to access a specific type of data or fact based on an attribute or dimension must faster than traditional row-based databases because in a columnar database, each data page is populated with data from only one column. 

Tom Davenport and Jeanne Harris in their book "Competing on Analytics" postulate that in the future more automated decisions using "operational business intelligence" will be made by machines rather than relying on humans. It will no longer be acceptable for companies to wait days or weeks to extract data from transactional OLTP systems, load it into data marts and then run historical reports through queries. There will be more prediction and less reporting. There will be greater use of alerts to notify managers immediately when key indicators are at undesirable levels.

Another area of data mining that is almost non-existent toady is the mining of unstructured or textual data. With the ever increasing popularity of social media there is an urgent need to develop efficient parsers to analyze unstructured textual data and transform them into information that can be used in predictive algorithms to project future behavior.

Many of the above changes will hopefully be driven by companies who want to be at the top of their game and increase customer satisfaction. Ultimately the companies with the best analytics will have the most efficient marketing campaigns and promotions.  Their customer service will be the envy of their competitors. They will use just-in-time inventory and employ the best people in the industry compensated directly based on their contributions to the company. They will be the leaders who will outperform their competitors and lead us into the future.

Lean Six Sigma in Software Development.

"Six Sigma" is a process man-agement strategy developed by Motorola in 1986 to improve the quality of products by identifying and removing the causes of defects (errors) and minimizing variability in manufacturing outputs. "Lean" on the other hand is a production practice that aims to eliminate expenses for resources that are not creating value in the product.

How do we combine Lean and Software especially since Lean Six Sigma has had its origins in manufacturing and does not translate easily to software development? The Poppendiecks in their book titled "Implementing Lean Software development" successfully make a case that writing software is really a form of product development and hence the implementation of Lean Six Sigma in software development is possible. They claim that, software not devel-oped as a stand-alone product is usually embedded in hardware, or it may be part of a game, or a business process - which means that software is either a product or a subset of a product.

"The seven principles" of Lean Software Development are:

• Eliminate Waste
• Build Quality In
• Create Knowledge
• Defer Commitment
• Deliver Fast
• Respect People
• Optimize the Whole

Eliminate Waste
Identify the 20% of product features that deliver 80% of value to the customer. Focus on implementing the 20% for deployment in the first phase.

Build Quality In
Develop QA standards and QC processes to catch defects early on in the software development life cycle. Test-driven design (TDD) is a very effective to reduce and eliminate defects.

Create Knowledge
Use feedback from daily builds and early release of the minimum 20% feature set to customers for evaluation and feedback.

Defer Commitment
Define the architecture model such that most decisions related to the feature set can be reversed and changed. A realistic design approach is to leave options open so that irreversible decisions can be made as late as possible.

Deliver Fast
Deliver software so quickly that customers don't have time to change their mind", say the Poppendiecks.

Respect People
Allow team members to offer feedback and ask questions. Just make sure you answer every question. You are not too busy and neither is any question unimportant that it does not deserver an answer. Employ team-building activities to help open the lines of communication and create co-dependence between members of the team.

Optimize the Whole
Lean software development needs the support of the entire organization. The entire value stream from the time a customer request comes in, to the time the software is delivered must be optimized.