First some background.

I was an early big data person. In 1982 when I was working on my (University of Lancaster) Master of Operations Research thesis I crashed the Manchester (or was it Liverpool? I forget which now) University ICL mainframe because I had submitted more distribution model jobs than the mainframe could handle. That is when ‘they’ found the bug in this George 1904S operating system – when the task buffer filled up the operating system would simply discard the first job in, to make space for the last job just entered. Sound funny now, but the department came down hard on me and I nearly lost my place in this OR program.

My experience building huge data models includes my (University College Dublin) Master of Finance 500 page thesis which is based on 2,000 years of daily share prices from 9 stock exchanges around the world) and multi-million data point models of photon structure.

1. “… only describe and quantify and mathematically formulate … ”

I can say this, some problems even if they can be formulated analytically, don’t have solutions i.e. cannot be solved. The role of big data will provide direction as to how to formulate human behavior analytically, but that does not mean that the equations can be solved.

Further, in spite of well-established economic theories, we still had the Wall St. crash of 2008. It was so unexpected, and being in the industry I can say this, that the untempered pursuit of profits, ideas and strategies that would have been considered irrational 10 years ago had come to be considered rational. Which raises the question can ‘rational’ be time varying. If so it adds to the complexity of this problem.

That means that even though mathematical theories on human behavior can become very sophisticated they can still provide wrong answers.

In finance there is a nice term “regime change”. This term means that the market behavior has changed so much so quickly that existing market models don’t give the right answers, and so models need to the ‘recalibrated’ to match the ‘new normal’ of market behavior.

My personal opinion of ‘regime change’ and ‘new normal’ is that these are symptomatic of fundamental flaws in these models. And again these models tend to be ‘analytical’ in nature and as stated earlier some analytical models do not have solutions.

And if I am correct (I stopped reading this subject several years ago) economic game theory is very much based on the assumption that all participants are equal. This is not how society works. And that the problems become very difficult to solve when one breaks this axiom. Please correct me if I’m wrong.

Not having analytical solutions can be a big stumbling block for the ‘new’ social sciences.

2. “ … Big Data we can now begin to actually look at the details of social interaction …”

Let me point to issues that I have had to work with in modeling human behavior / market forces.

Lags/leads: Forester’s pioneered systems dynamics, that there are lagged and leading feedback and feed forward loops. This is especially true in the economy, for example the 3-month Treasury interest rates are affected by what happened in the economy 15-years ago.

Data loss: And sometimes there major data losses. The Fed stopped publishing M3 which is so important to modeling 3-month treasury rates.

Short shelf life: Sure we can look at details but can that be helpful. I did a study for a former employer and we found that FICO score are not good predictors of defaults in the long run. They seem to work within a multi-month time frame but not on a multi-year time frame.

And finally I must disagree “to truly build a civilisation upon rationality”. We are inherently irrational, and have different ideas of ‘rationality’ that is why the economic markets work. If we all had the same idea of ‘rational’ we would all be on the buy side or on the sell side at the same time. That is not the case.

But it is a nice ideal to navigate towards a ‘new’ civilization, and new economic theories.