Predicting Amazon sales using Deep Learning

-

Introduction

My primary aim was to predict the sales of an item given the Best Seller Rank on Amazon. Predicting the sales helps me in other use cases like suggesting sellers the best products to sell. My final aim is to provide data insights about any product: How much it will sell as well as when, where and how.


What is Amazon Best Seller Rank?

Best Selling Rank is a ranking system provided by Amazon that is linked to the number of sales of that product. This rank is calculated frequently. An important point to note is that Best Selling Rank is a 'ranking system' and by itself it doesn't mean anything.  

A rank of #1, therefore, means that product has sold more than any other product in that category, on that marketplace.

This kind of makes it relatively easy to predict the number of sales of a product if we know the sales of other products ranking close to it.

How did we get the initial sales data? I have been selling professionally on Amazon and have been tracking my own sales vs ranks for all my products in various categories. Additionally I interviewed other professional sellers to get an approximate idea of their sales.

 With all the data obtained, cleaned and setup I entered the next phase of design: Choosing the best framework for Predictive Analysis


Enter Spark

At the 2016 Spark Summit Nick Heudecker asked the question Is Apache Spark the future of data analysis?
While no clear answer was provided we certainly know that Spark alone is not the future of data analysis. We depended on a number of third party tools like Stanford NLP for natural language parsing and tensor flow for image analysis but Spark formed the backbone of almost all of it.





    While there might be some truth to the above chart I tend to believe the Spark has not reached peak hype yet. Or maybe it seems that way from here down under in Australia and probably Spark has surpassed peak in the Silicon Valley.

    Spark has played amazingly well with our spring boot application and our standalone machine learning application (Command Line Interface).

    Our database is PostgresSQL and Spark CLI programs run weekly reading the PostgresSQL database via JDBC bridge, processes them, builds learning models and saves the trained model to local path.

     This trained model is then read by the Spark in Spring Boot to quickly make predictions or process any incoming information from web users in real time.  With all the infrastructure setup we had estimated a week to complete the linear regression algorithms or worst case scenario of two weeks if the problem turned out to be the more complex log-linear regression.

    Houston, We have a problem

    I expected a straightforward linear regression model of type  y=mx+b This would have made the problem very simple as Apache Spark has GeneralizedLinearRegression   

    val glr = new GeneralizedLinearRegression()
  .setFamily("gaussian")
  .setLink("identity")
  .setMaxIter(100)
  .setRegParam(0.4)

    I first plotted the chart on an Excel Sheet as I had already exported the data from SQL to CSV.
    The relationship between Amazon Best Seller Rank and number of sales turned out to be like this chart.






    So it looked like a log-linear model and I assumed the poisson family of  GeneralizedLinearRegression would be a good fit.  We changed the GLM family to poisson and ran the tests few more times however the Mean Squared Error & the RMSE was too huge. 

     I suspect Spark has issues in dealing with sparse data.
    We have limited data of our sales and we don't sell products in all categories since Amazon has gated some categories which left us unable to sell or make any observations on sales for those gated categories.
    With limited amount of input data, Spark MSE was too high and even for our products for which we knew the sales, the predicted sales were way off the mark.

     Over the next few weeks I spent my time trying out all combinations of Regression family on Spark and none of them gave the desired results. I had absolutely no idea on how to proceed now and this reminded me of this quote by Dan Ariely
    “Big data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it…”

    My last resort was to use Deep Neural Network Learning. I had completed the Andrew Ng course on Machine Learning when he has first launched it and I cannot recommend it enough. Now my aim was to use Neural Network to solve this problem.


    We must go Deeper.

    I eventually made the decision to keep Spark for big data analysis however we would resort to other libraries for Deep Learning.
    I narrowed down my choices to two libraries
    1) DeepLearning4J
    2) Sparkling water- H2O

     I was particularly impressed with this article  that discusses how H2O deep learning was used to predict crimes and arrests in San Francisco and Chicago. Since our future use cases are similar where we will be predicting fraudulent users and fraudulent competitors I decided to plunge into Deep Learning using H20 rather than attempting to work around with Spark ML issues.

     Sparkling Water-H2O runs within the Spark framework so I could use their integrated framework without replacing Spark. This was a definite bonus for me. Also the documentation was nicely done and I was thoroughly impressed with H2o web-UI , Flow.
    I could test my data and algorithms on the browser without writing any code.

     The Web-UI provided excellent insights into the data and true to my beliefs, the Deep Learning Neural Networks provided exceptional results.

     Using the optimized parameters from H2O Flow, I quickly coded the Deep Learning network in my CLI Program.

    val train = result('categoryIndex, 'bsr, 'sales)
// Configure Deep Learning algorithm
val dlParams = new DeepLearningParameters()
dlParams._train = train
dlParams._response_column = 'sales
dlParams._fast_mode = false
dlParams._epochs = 30
dlParams._nfolds = 3
dlParams._distribution = DistributionFamily.gaussian
val dl = new DeepLearning(dlParams)

val dlModel = dl.trainModel.get
//save the model
ModelSerializationSupport.exportH2OModel(dlModel, new File("/data/deeplearning.bin").toURI)

    On the Web API (Spring Boot) application I read this model in and used it for making predictions in realtime from web users.

    def startup(): Unit = {
  dlModel = ModelSerializationSupport.loadH2OModel(new File("/data/deeplearning.bin").toURI)
  println("Initialization Of BSR Deep learning Module complete")
}


def predict(categoryIndex:Int, bsr:Int): Double = {
  if(null==dlModel) {
    startup()
  }
  println("\n====> Making prediction with help of DeepLearning model\n")
  val caseClassDS = Seq(InputBSR(categoryIndex, bsr, 0)).toDS()
  val finalresult = dlModel.score(caseClassDS)('predict)
  val finaldf = asDataFrame(finalresult)(sqlContext)
  val predictedSales = finaldf.first().getDouble(0)
  println(s"For category index ${categoryIndex} and BSR ${bsr} the result is ... ${predictedSales}")
  predictedSales
}

    The end result is this:





    As you can see from the screenshot, we can only predict sales of products which have Best Selling Ranks for top level category since we have trained the Neural Network with data of sales only from top selling categories.

     As we keep collecting data and our algorithm has sufficient confidence to predict sales of lower level categories, the app will start making prediction for more number of products.

     This, IMHO, is the best thing about Big Data and Deep Learning. The machine never stops learning and eventually as more data is fed into it, the algorithms automatically start making better predictions.

     The chrome extension is available here on the Google app store

    Popular posts from this blog

    Can you survive and sell on the internet without being the cheapest?

    -
    Image
    In the offline world, there are many types of stores and not all bet on the same criteria to try to reach the consumer. Not everyone wants to have a low cost supermarket or a discount store in which the main offer are products at low prices. So why is there a certain belief that when selling online you have to sell at a low price? Is e-commerce marked by offers to the point that it is only successful if it is sold cheap? For years, what most worked on the net were the offers. Consumers made the network their main ally when they were looking for things at lower prices. The trips started to be bought online because the internet was able to find the lowest prices, last minute offers of hotels or the cheapest fares of the low cost airlines. Then came the coupon pages or online outlets, which were the entry point for many consumers for e-commerce. And then many large e-commerce companies that invoice millions have made offer low prices a decisive element of their corporate image an...
    Read more

    Guide to Amazon Kindle Direct Publishing (KDP)

    -
    Those were the days when getting your book published was like a catch-22 situation, where the authors need to get past the most difficult gate. They spend months writing a book proposal and contacting agents in order to pitch their books up to great publishers. Who again are never satisfied with what they write and comments that it may not make good money.  So, when agents reject their book authors either abandon their dream of getting their book published or instill great confidence through contacting directly with publishers. Now publishers in turn send a rejection letter or quite a bad offer. Which is why Amazon created an incredible thing for authors as they can be a self-published author. This sounds great right!!! It actually paves the way for indie authors to bypass the traditional publishing method and be independent.  In this article we will get into the in depth knowledge of KDP like: What actually is KDP? What is KDP Select? What is CreateSpace? What is Amazon Assoc...
    Read more

    E-Commerce has no borders: Myth or reality?

    -
    Image
    One of the issues usually mentioned when talking about e-commerce is that it has served to break the frontiers. Consumers often look for specific things, products that they want, and it really gives them a little equal where the sellers sell them if what they offer them is what they want and if they make it arrive quickly and efficiently. This has created a boom in so-called cross-border trade (for example, we have to think about how we buy books today and where they come from nowadays: online bookstores sell all over the world, some of them even with free shipping no matter where you are) and has made consumers open their range of options. There are those who buy directly from Chinese e-commerces, although the products end up taking much longer to arrive than if they are bought in a geographically close online store. But is this principle absolutely true? Are consumers really so foreign to geography? I mean, are we buying without really caring about the physical location of...
    Read more