Doesn’t it frustrate you when you buy an airline ticket to go home for the holidays, only to find out that your younger brother, King of procrastination bought his plane ticket a day before your flight departed (4 months after you), and he ended up paying less?
It is this very reason that Farecast was created in 2003. They understood that because of the internet, fixed pricing was a thing of the past; but now prices are so volatile that there’s almost no way to decide when it will be cheapest. Farecast solved this problem, establishing the motto “Know When to Buy.”
So how much data needs to be mined in order to predict when fares will be cheapest? According to the interview in this video, hundreds of billions of price observations that have been gathered over more than 8 years. This data is crunched to find patterns; both short and long term. Should I buy on Monday? Should I buy in the Spring?
These data predictions are about 70-80% accurate (this is more accurate than the weatherman!). This is the case because they gather more than just historical data. They gather unstructured data on rumors, gossip, blogs, PR announcements and more to come up with more accurate predictions.
This same principle can be taken into pretty much any industry, just as big data analytics can help businesses in just about any industry.
From the government offices, Federal IT officials say real-time analytics of big data can help reduce the federal budget by at least 10 percent annually, or about $1,200 per citizen, by detecting improper health care payments before they occur, according to the TechAmerica survey. If we can save that money, we can also save sickness and lives!
Yet, despite all the real and potential benefits of big data analytics, federal and state IT officials also cite several barriers to adoption, including privacy concerns, the expense of new tools, lack of clarity about big data’s return on investment, and the time it can take to extract queries from traditional databases in a timely manner. There are data podcasts which discuss this more deeply.
Really when we think about what businesses are trying to do if we strip back all the mission statements and values etc., kind of what they’re there for is to either make money or save money. They tend to be pressured to do one or the other or in some cases, both depending on where the business is in its marketplace and its competitive construct, etc.
This is where business stakeholders will often initiate ideas that lead to data science because they’re looking for answers that they can’t find at the moment. They can’t find it intuitively, they don’t have enough data to find it or maybe they have sufficient data or they suspect they may have sufficient data, but they can’t find the insights or the answers that they’re looking for.
Two real examples of what was done in this space. They both actually are retail examples and retail is good because we all understand how retail works, at least at a superficial level once you dig beneath the surface, there are nuances we could never imagine. One of the big problems around retail is a little thing called shrinkage.
Now I’ll avoid the obvious shrinkage jokes and simply say that shrinkage is the term used in retail for goods being stolen, going missing, magically ending up in the pockets of the employees sometimes. Essentially it’s stock loss, and stock loss has a direct impact on cost because obviously this is inventory that you’re purchasing, that you’re spending money on but not getting any money back from. And Big Data can help.
Focus on shrinkage is a big driver in most retailers and certainly very much falls in the category of saving money. All retailers do a lot of work around shrinkage already. They have big teams in some cases that deal with it purely but in this case, the business goal was to understand the shrinkage drivers and to validate some hypotheses about why shrinkage is happening, and then to actually reduce the lost due to shrinkage, so to provably show that we can find the source of shrinkage, make some changes and fix it. Also, within this particular organization, their analytical sophistication was low so they wanted some easy-to use tools, and they wanted the answers, etc.