Measurement

To maintain a prediction's integrity, the variable used to predict must be harder to change than than the thing it is predicting.

The very act of measuring something changes the thing being measured.  You must measure something big or unusual to get it to stick. To preserve the integrity of your measurement.

One of two conditions must be met: (1) The people who control the prediction must not know about the measurement. or (2) It is nearly impossible for the prediction to be changed based on the measurement, or there is no reward for them to do so (this is rare).

Here are a couple of examples:

Example One
Let's say I want to predict a student's grades. The output is GPA. I believe I can predict a student's GPA in any semester by taking the weighted average of his course grades. This is an unchangeable fact. The integrity of the prediction is preserved, because it is impossible that the prediction could be changed by me or the student.

Say I want to break down the prediction further: I can predict a student's grade in an individual course based on the number of hours he spends studying over the semester. This measurement is not unchangeable. Its integrity is in question. If (1) the student does not know about me tracking his hours, then the prediction is ok. But if (1) is not met, neither is (2): The student, knowing that his GPA can be predicted by his hours, may start to put empty hours into studying, not fully concentrating on the material, thinking that he will get an A by putting the minutes in. The integrity of the prediction has been compromised. Over time, the prediction will become less accurate.

The prediction becomes much more muddled when compensation comes into play. Let's say I pay a student per hour studied, thinking that will improve his GPA. Bad idea! Further hypothesis: each student has a stock price tied to his grades. I invest in the students whom are undervalued based on the hours they have put in this semester. Ok. I succeed as long as my method is accurate. Now the market finds out about my prediction method. No good! The market will adjust, incorporating my method, and students with high hours will no longer be undervalued.


Example Two - A Stronger Measurement

I believe I can predict a nation's Olympic Medal Count based on its GDP. Props to Colorado College Prof. Daniel Johnson. It is harder to change a nation's GDP than it is to add to the medal count, so this is a strong predictor that cannot be compromised.

But Professor Johnson found that communist, centralized governments are more likely to get gold medals.  If capitalist nations find out about the study, the prediction can go bad because those capitalist nations can mimick the training programs of other types of governments to increase the medal count.


A weak measurement

Any formula for picking stocks is predominantly luck. Say you invest in stocks that have crossed their 50-day moving average. Say you invest in stocks with low P/E only. No single formula to pick stocks has the strength and integrity to withstand the market once the market finds out the method is profitable.  There are two ways to go about this: one, you have a small window of opportunity when the market does not realize it is overvaluing or undervaluing a particular formula. So, for example, generally a good time to invest in high-Beta stocks is after the market has gone down a lot. Because people get sick of losing money, and the market moves out of those guys unnecessarily. But if the market were to realize this arbitrage existed, it would disappear. The most sustainable predictor of stock price is to buy assets for less than they are worth. You must analyze the current assets and project the future cash flows of a company, and invest in what is most certain to bring you the most cash for your investment. That is a measure that cannot be corrupted by the market, because that is the heart of a business's value.