Trying to guess if a car is damaged or not by Logistic Regresion

We had a car data collected from website. It was an advertisement website for used cars.
Car data had below properties regarding damage.

1)If there is a big damage or any damage that insurance company knows people
say it is damaged.
2)If it is a small thing,if one can make up, or already painted that are and thinks
no one understand , he does not say it is damaged.

What we have is then
Car model
Year
Price
City
Date of publishing
Last update of advertisement
Days elapsed from publishing(if a car is sold it goes from list)
Elapsed Days for selling

Think there is no data as clue. We must generate , extract, invent our data.

a)So lets think how a damaged car owner thinks
b)What changes in advertisement over time if car has small damage.(car owner is
editing advertisement over time)



1)Number of page view

There is a mean number of average page view before car owner deletes advertisement.
Lets say a no damaged car is being sold after nearly 100 page views. If a car is advertised
as not damaged and still not sold after 100 page views it can have a problem.

2)Number of change in price

At 1st owner thinks he can sell his car with a price like non-damaged cars .After a period he makes some discounts.
Probably after some calls he realizes he has to make discount. So we can generate 2 variables from here
% discount he made from 1st price
# of discounts he made.

3)Is price lower than average with same conditions.

A sense of guilt could be determined.

4)Duration that it is on sale

Total duration car is on sale.


5)Difference of duration in days from duration average sales of same car model.

There is an average duration for every combination of cars. So elapsed day
after average duration(day or week) will increase the probability of damage.



6)Number of pictures in advertisement

Probably a damaged car owner will put no picture or 1-2 pictures.Less picture could mean more probability of damage.

Sample Size and Standart error

You record your sleeping for 1 year lets say.
And you find average to be 10 hours.(This is average of all individual days)

And you find population standard deviation as 2.

Leftmost part in picture.

You randomly take 10 days within a year and take average.
Repeat this process for lots of time to take average of different "10 samples" .
*** Be careful you now took average of only a sample subset.

Middle part in picture.

Standard error is = 2 / √ 10 
Standard error = 0.63
This value means with 10 samples we are 0.63 far from real mean which is 10.

You can see that sample graph


If we get 100 samples it is more centered around real mean.
Standard error is = 2 / √ 100 
Standard error = 0.2


More samples the more you are near to Population mean.

Standard error = Standard Deviation / SQRT( Sample Size)

So if Standard deviation is high, (your data varies a lot ) , you will have big error for samples.

Spark Apply Descriptive Statistics on DataFrame

When you first get your data you have to play with it.
You want to learn what kind of data you have.
Below is a simple code piece to begin investigating general properties of your data.

Suppose you have a data like 48,49,50,51,52. This is well distributed homogenous data.

import org.apache.commons.math3.stat.descriptive._

val df = Seq(48,49.0, 50.0, 51.0,52.0).toDF("nums")

val mean = df.select("nums").rdd.map(row => row(0).asInstanceOf[Double]).collect()

val arrMean = new DescriptiveStatistics()
genericArrayOps(mean).foreach(v => arrMean.addValue(v))

val meanQ1 = arrMean.getPercentile(25)
val meanQ3 = arrMean.getPercentile(75)
val meanIQR = meanQ3 - meanQ1

Perfect distribution
47,48,49,50,51

n: 5 
min: 48.0 max: 52.0 mean: 50.0 
std dev: 1.5811388300841898 
median: 50.0 
skewness: 0.0 
kurtosis: -1.200000000000002 
meanQ1: Double = 48.5 
meanQ3: Double = 51.5 
meanIQR: Double = 3.0

Lets form a line shaped distribution
val df = Seq(50,50, 50, 50,50.0).toDF("nums")

n: 5 
min: 50.0 
max: 50.0 
mean: 50.0 
std dev: 0.0 
median: 50.0 
skewness: NaN 
kurtosis: NaN 
meanQ1: Double = 50.0 
meanQ3: Double = 50.0 
meanIQR: Double = 0.0

Lets add 40 to make left skew(negative skew.
** Skewness is asymmetry of distribution about mean.



Left tail (skew ) distribution

val df = Seq(40,48,49, 50, 51,52.0).toDF("nums")

n: 6 
min: 40.0 
max: 52.0 
mean: 48.333333333333336 
std dev: 4.320493798938574 
median: 49.5 
skewness: -1.8805720776629977 
kurtosis: 3.9187500000000064 
meanQ1: Double = 46.0 
meanQ3: Double = 51.25 
meanIQR: Double = 5.25

Rigth tail (skew ) distribution
If we just add 60 to original series we get a right tail distribution.
Skewness is same with different sign.
val df = Seq(48,49, 50, 51,52.0,60).toDF("nums")

n: 6 
min: 48.0 
max: 60.0 
mean: 51.666666666666664 
std dev: 4.320493798938574 
median: 50.5 
skewness: 1.8805720776629975 
kurtosis: 3.9187500000000064

meanQ1: Double = 48.75 
meanQ3: Double = 54.0 
meanIQR: Double = 5.25

meanIQR is a data without boundaries. So it gives lots of idea if you know your domain.
For example you have a car price data. You know that car must be around 50.000$.
When you check meanIQR you will see datas near to your expectation. Others will
have have meaningless high( irreal expectation of seller) or low( this time meaningful because
car could be damaged.) meanIQR is a nice measure.

Skewness can give a rough idea about tendency of data. (Data having a tail to left if minus.)

kurtosis is a measure of shape. The sharper the top the higher the kurtosis. Check picture from internet please.