Searching for Patterns in Financial Information: Part 1

Lately, my business partner has been reading about the similarities between the development of entropy theory in physics, and Claude Shannon's development of information theory. The parallels are striking, down to the similar forms of the equations for calculating the amount of entropy in a container of gas, or the amount of information in a set of symbols.

As we discussed this, we realized that, in an abstract form, "efficient market" theory is a crude way of approximating the same result. That is, a very efficient market would have little differentiation in prices for a security, much as an end state of a gas in a container is total dispersion to a (largely) uniform density.

However, of some interest to him, and me, was the development of probability distributions to account for the non-uniform density of dispersed gases. In a similar way, one might observe distributions of prices for a given security within a short period of time.

We realized that, according to efficient market theory, absent new information, the investors in a market are assumed to act in such a way as to arrive at a state of price entropy. Just as there is no more movement once a gas is perfectly dispersed, so prices are expected not to move, absent new information.

So far, an interesting comparison. But perhaps seen as just that, and of no particular value to understanding equity markets, or selecting equity portfolios.

As we discussed this further, he dwelt more on Shannon's information theory. Shannon focused on how to identify a signal amidst the noise of a stream of "information." This, my partner conjectured, is virtually identical to the task of a portfolio manager. The manager must attempt to identify relevant information, which is related to particular price changes, and ignore the noise in the distribution of information in the market.

At this point, I interjected that market information not random in the same way as naturally occurring phenomena may be. There are actors on each side of securities prices. Securities prices are not just streams of random numbers attached to a meaningless ticker.

Rather, there are real companies, composed of real people, behind each ticker. They are engaged in teleological behavior- hopefully, the production of profits, in order to increase total returns for their shareholders.

On the other side, there are real investors attempting to discern patterns from the torrent of market and fundamental information about the companies. These investors buy and sell securities, based upon their interpretations of this information, thus providing some goal-oriented impetus to prices.

I believe that because securities prices are, ultimately, the product of these goal-oriented activities, the process of identifying patterns in fundamental and technical market data, in order to find equities which have a high probability of earning consistently superior total returns, is actually easier than dealing with truly random data.

However, due to the differing abilities of the actors on both sides- company employees and managers, and investors- finding the patterns is not simple.

But, like distributions of energy levels in particles, or distributions of information content among symbols in a series, there are distributions of information with respect to equities selection among the fundamental and technical information available for observation. As a distribution, this means that there will be occasions in which information content is high. Meaning that not all the value in knowing it has been removed via discovery.

When this happens, portfolios of such equities can outperform the market averages, contradicting efficient market theory. Thus, in effect, we find that efficient market theory posits simple point-estimates of prices, whereas viewing the problem in the same manner as entropy or information theory allows for "tails" of value-laden information which can cause equities to be mis-priced, and allow for consistently superior returns to be earned.