Stats

People have been writing analysis code for a long time. To make their life easier (easy is good), they made libraries of functions they use often. The libraries are free, and you can use them to make your life easier, too.

The most common libraries are things like NumPy and Pandas, but they're more complex than we have time to learn. However, there's an older, simpler library called Statistics, that works similarly. Let's use that.

The downside to using any package is that it expects data in a certain format. The list-of-dictionaries format is a great way to represent typical business data, but the statistics module doesn't know how to deal with that directly.

So, you need a data machine for the pipeline to extract data from a list-of-dictionaries to whatever format the statistics module likes.

The statistics module

The module has 18 statistics functions in the documentation. There's:

• mean for computing averages
• median to get the middle value of a data set
• stdev for standard deviation
• Others...

The easiest way to use statistics functions is to send them lists. For example...

Line 4 calls a function to compute a mean (an average, same thing). Send it a list as the param, and get back one number.

Before and after

Let's see how we can prep data for the mean method.

Here's a program we wrote.

You can see the pipeline from the code.

Pipeline​

Aye! That's how you should design your code.

Something you learn from experience is now to break apart complex tasks, what the pieces should be. You've already got some nice patterns, for reading CSV, making new data sets while cleaning existing ones, adding new fields to data sets... you can combine them to do many different things.

Being mean

Let's get back to the statistics module. We saw the mean function.

Give it a list, and it will give you the mean.

So, if we want to compute the average of, say, the Before data, we need to make a list with just that data. This is how the data is arranged, and what we want to give to mean.

What we have	What we want for mean
	[     17,     14,     12,     18,     16,     15,     16,     10,     10,     19,     20,     12,     15 ]

Correct. We want to make a list we can use like this...

... where befores is a list of all the Before values, like in the right-hand column of the table above:

Nice!

A new pattern:

When you start a new project, you can use the pattern catalog to remind yourself of useful chunks of code.

Calling statistics

Here's the main program.

(Standard deviation measures how spread out data is. You don't need to know anything about it, just that it exists.)

No problem. Add one line:

Here's the computational formula used in most programs (the definitional formula is different):

You could code that yourself, but most people don't, since there are modules like statistics. Easy is good.

Like I said earlier, the downside of these modules is they require data in a certain format, like mean needing a list of numbers. However, it's easier to write the code to set up the right format, than to write the code to do the analysis yourself.

All the fields!

Here's extract_before_values again, the function that returns a list of Before values for analysis:

Good!

Right!

Great! I noticed you also brought compute_change back into the pipeline, do the Differences field exists.

We've made a reusable function, extract_field_values. Write it once, call it as many times as you like. A useful addition to the pipeline.

Let's add that to our data machine list.

Field extractor​

All the statistics!

Indeed! I like the output format, too. Easy to find what you want.

Aye, that does look better.

Highs and lows

Good point. Some listy functions are part of Python's core. For example:

max takes a list as a parameter, and gives back the maximum value in the list. No statistics module needed, since max is built-in to Python.

Highs and lows and names

What if you want to know which goat has the highest After value? max_after = max(afters) will give you the highest values, but won't tell you the name of the goat with that value.

One way to do it is write a function that loops over the records as usual. Each time through the loop, it asks: Is the current value greater than the largest I have so far?

For each record, if a goat's After is larger than any we've seen, remember the new After, and the name of the goat with the new After.

OK, here's some data and code.

As you can see, the largest After value in the data is 24.

Right!

Here's the stuff again.

The code runs through the records, grabbing each After value. That's 12, 12, 23, 24, and 14.

Each time through the loop, line 4 compares the After value from the record (12, 12, 23, etc.) with the largest value so far, in largest_after_value, which is initialized to 1000.

Let's look at the first record. After is 12. 12 is not more than 1000, so largest_after_value is not changed.

The second record. After is 12. 12 is not more than 1000, so largest_after_value is not changed.

The third record. After is 23. 23 is not more than 1000, so largest_after_value is not changed.

Continue for all records. largest_after_value will never change, since none of the After values are more than the initial value of largest_after_value: 1000

Right!

There's one small improvement we can make. It won't affect this data set, but remember that we might reuse the code for another program, like temperatures on Mars. We copy-and-paste the code, but doing this won't work:

Right!

The easiest thing to do is to type in very large/small values to initialize the smallest-value and largest-value variables.

A new pattern

Use existing computation functions when you can, like statistics.mean. If you can't, like when you identify records with lowest/highest values in a data set, write your own loopy function.

Summary

• People have made libraries of functions they use often, like the Statistics module.
• Every module expects data in a certain format. For example, statistics.mean wants a simple list.
• We added a new data machine, field extractor, to our collection.
• Functions like max and min are part of Python's core. No module needed.
• To find, e.g., the name of the goat with the largest value in a field, you need a loop with an if. Initialize the largest-so-far variable with the smallest possible value.

Exercise