Nowadays we are awash in data. Easy access to data and to huge amounts of computational power is completely changing our relationship with data and our approach to mathematical modeling. Generalizing from small amounts of data required making careful and tedious assumptions about model behavior. That is no longer needed, nor is the tractability of mathematical modeling all that important. What is all-important is what has been so all along – how useful is the model? Whether the model can be found in an old statistical textbooks is not only irrelevant, it can chain innovative thinking. The only affirmation of a model is its performance, not its pedigree.

I expect that these views will not be welcomed by all, particularly the purists who believe in the frequentist/bayesian paradigms that have dominated historical work with predictive modeling. I believe that with the data available today to build and test models – it is irrelevant whether the modeling is ‘sound’ based on some arcane definition, or whether it can be interpreted in some academically narrow or sanctioned way. What is relevant is whether the model facilitates some useful outcome, and that it does so in a way that is robust to new data.

To be clear, in some applications the transparency or interpretability of a model is a valid concern – and we make performance trade-offs to accommodate those features. I suspect that with the vast new neural nets that are being designed and built for ubiquitous applications (e.g. organizing our photographs, reading diagnostic charts, etc.) – whether a human understands the model or not will be an increasingly marginal concern in the future.

The post Classical vs Machine appeared first on Predictive Modeler.

The Newtonian revolution reinforced not only physical, but a causal determinism about our reality. The past and the future are inextricably linked. We can predict the motion of objects, small and heavenly, based on where they have been and the (constant) forces acting upon them. There grew a sense that if we could figure out the equation(s) of all the forces and the precise current state of each atom that composes our reality – we might be able predict everything.

Heisenberg spoiled the party suggesting that the fabric of reality is such that we can never know everything about the present let alone the future. But there was a saving grace. While we don’t know what each atom is up to, bunches of them behave in probabilistically predictable ways – and the most successful theory in the physical sciences, quantum electrodynamics, was born.

Let me get back from my nerdish tangent to the question at hand. There is a common refrain that the past is not prologue. Just because something happened in the past does not mean that it would happen in the future. The honest truth is that we just don’t know. We think the sun will rise tomorrow as it has ever since the dawn of man – but it could be swallowed by a rogue blackhole overnight!

Uncertainty does not meant that we give up, and stop trying to learn new physics or using mathematical modeling to predict the future. It just means that we recognize the limitations of our processes and that we are going to have to make iterative improvements to our modeling whenever the future refuses to obey our abstractions of it – which it predictably will.

The post The Notion of Predictability appeared first on Predictive Modeler.

The construction of a model starts with a series of assumptions and an objective. The objective could be understanding a real-world process better, explaining what happened in the past or to make a prediction about the future. The assumptions include the set of data that we think influences the outcome, and what sort of relationship (e.g. linear) does that data have with our objective. Those are the big ones anyway – then there are a whole bunch of little ones along the way. It is important to document assumptions and think of ways to test them as you are building your model.

Another thing that is often times overlooked is that a model is not simply a mathematical formula. Unless your audience has a 100% of the context you hold in your head (rare!) – a model includes the mathematical bits (e.g. code, excel, wherever else the math resides) but also the documentation including the assumptions. Further, it is always nice to have an executive summary/overview to make it easier for your audience to consume your work.

The post What is a model? appeared first on Predictive Modeler.

Even in the sciences, compact equations (e=mc²) are elegant and seem to demonstrate that we have understood something about our reality in a profound way. Larger, messier representations of reality convey a feeling that more work needs to be done to distill out irrelevant details and get to a deeper state of understanding.

Predictive modeling is no exception in my opinion. More complex models are not ‘better’. In fact, they are sub-optimal if simpler models with fewer assumptions are possible with little or no loss of performance. Most of the time one would (and should) trade-off some performance for simplicity. And by the way, this does not apply to the final model form (e.g. 10 predictors instead of 5), but also applies to modeling techniques (e.g. black-box methods vs. transparent and easier to understand approaches).

This is 2019 – and our predictive modeling efforts are still for review and use by other humans. When the time comes when our algorithms are for the sole consumption of our machine overlords, you can chuck the principle of parsimony and be as opaque as you please. Until that time, keep focusing relentlessly on simplifying your methods and models to the point where degradation in performance is significant relative to your application. You will build better models, communicate them more effectively, and increase the odds that you have understood something real about the world.

The post Principle of Parsimony appeared first on Predictive Modeler.

For the last fifty years, advancements in computing have been about making transistors smaller. Let me take a step back. Most of us know that the computers speak a different language. The language of bits (i.e. 0 and 1). But if you open up a computer and sort through its innards, you don’t find any 1’s and 0’s lying around. You see chips, and these chips contain transistors. Transistors either allow electrons to pass or not depending upon the voltage that is applied. A bit can be built from an arrangement of a couple or more transistors. Eight bits make a byte. And many, many bytes make up the internet, apps, software, these words you are reading, etc.

Back to Moore’s Law. Its dying because it’s getting harder to make transistors any smaller. They are getting so small now that engineers have to worry about quantum tunneling effects¹. In this effect electrons are essentially and probabilistically walking through walls, making a mockery of our efforts to control their flow that is critical to the operation of a transistor!

Intuitively, the laws of reality should be the same whether something is big or small. Whether we are holding a grain of sand or an apple, gravity pulls both down when we release them. However, our intuition about reality is based on our experiences with matter containing trillions and trillions of atoms. A single grain of sand has 10²⁰ molecules. If each molecule was a human, a grain of sand would represent enough humans to populate 14 billion Earths!  But reality does not care about our intuition, and the world of the extremely tiny, the quantum world, is bizarre!

In an Aikido-like move, we have figured out how to take the very thing that is killing rapid advancement in transistor-based computing and turn it into a significant advantage. We can utilize the bizarre concepts of Quantum superposition and entanglement to solve certain problems that are intractable for our current computers. Such problems include optimization (e.g. traveling salesman problem), simulating molecular interactions in chemistry, the N-body problem in physics, etc.

Computational tasks can be classified as “P” and “NP”. “P” (or Polynomial-time) tasks are those that current computers can solve quickly ². “NP” (or Non-Polynomial time) tasks cannot be solved by current machines but they can quickly verify a solution to the problem. A quantum computer holds the promise to quickly solve “NP” problems. One example is being able to break encryption protocols quickly. Given how many passwords each of us has for everything from our bank accounts to Netflix, needless to say that it would be a very bad thing (but could also lead to good things, like impenetrably secure data transmission)!³

A quantum computer excels at the exponential scaling of a computational problem. As long as a computational task can be encoded onto a quantum environment, we can observe the probabilistic results of that environment in order to get solutions to that problem.

This is a very different approach to solving a computational task. Our current method maps a task into a sequential operation of transistors comparing 0’s and 1’s in order to get to a final discrete state (i.e. one correct answer). For some problems, our best bet today is to search through the entire solution-space bit by bit⁴ in order to find the right answer.

In a quantum computer you can encode the solution-space of the problem onto the massive number of states (or phase-space) accessible to a quantum system. A particle can act like a wave. When waves are in phase their amplitude increases and vice versa. We can utilize the principles of interference in order to amplify the correct answer and suppress all the incorrect ones. Nature itself reveals the solution to us.

Two quantum mechanical characteristics are particularly important to quantum computers:

Superposition

The basic unit of a quantum computer is a quantum-bit, q-bit or qubit. Whereas a bit can be 0 or 1, a qubit can be in a superposed state of 0 or 1. In fact, it is 0 and 1. A way to think about it is to imagine a sphere. Whereas a traditional bit is either the north pole (=1) or the south pole (=0), a qubit is anywhere on a longitude connecting the north and the south pole. When we encode information on the qubit, we give it a phase which is like a latitude rotating it from the longitude connecting the north and south pole.

Adding more transistors to our current machines leads to a linear increase in compute as it can only deal with one bit at a time. In a quantum computer the computational ability scales exponentially. Since qbits can be in a superposition of 1 and 0, n qbits can represent 2ⁿ states. Ten qbits can represent 1024 states. One hundred qbits represent 1.26 x 10³⁰ states!

Entanglement

The other important concept in quantum computing is entanglement. If we actually live in a simulation⁵, one evidence of it might be the weird glitch embedded in our reality called entanglement! Einstein called it a “spooky action at a distance”. The reason it is so terrifying is that we know that there is a cosmic speed limit, just like we know there is this thing called gravity. Nothing can travel faster than the speed of light. And yet, if two particles are entangled, they seemingly are able to communicate regardless of the distance between them. We can have one particle in the Milky Way and the other in the Andromeda galaxy (2.5 million light years away!). If we measure the Milky Way particle to have one state, the Andromeda one instantly acquires the opposite state. It is as if they instantly exchange information between them ignoring the cosmic speed limit. Do we actually know what we thought we knew?

A quantum computer is able to use the principle of entanglement by creating superpositions such that if we measure one “particle” to have state 1, the other is instantly in state 0.

Entanglement is one of those amazing mysteries in physics, and the following is a really good (new) documentary on it:

But we can do more than just see a documentary. We can demonstrate entanglement on an actual real-life quantum computer! See the video below that I recorded demonstrating entanglement on an IBM quantum computer⁶.

Last Word

Quantum computing is in its infancy right now. The technology is very hard to do and the usefulness is rather academic right now. However, things could change very quickly. I suspect that this technology will have major consequences for things like secure communications and artificial intelligence. Perhaps the reason we have not yet build a sentient machine is that our brains are made of physical stuff, and may be accessing the world of the quantum. We might have strong Ai⁷ if we built it using a quantum computer. At any rate – definitely one of the technologies to watch for the future!

Additional Resources:

A good explanation of quantum computing: link

The post Quantum Computing appeared first on Predictive Modeler.

Features that make this book valuable

There are many great resources, in print or on-line, for learning about Predictive Modeling. The following ideas make this work different and valuable to you:

• Less is more: a key to the usefulness of this book is not only what is in it, but also what isn’t. I only include practically useful content and limit theoretical discussion. Especially when it comes to scientific content, quality is much more preferable than quantity.
• Practice makes perfect: the best way to learn about predictive modeling is by doing it, and the book aims to make that transition easy & quick. The pages of this book contain working code which can be downloaded and run in a matter of minutes.
• Working smarter not harder: A lot of predictive modeling practitioners (myself included) have experienced spending 80% of our time tinkering with software, coding, or data wrangling…and only 20% of the time on the actual business application. One aim of the e-book is to automate a lot of that work. In doing so, we can make our processes (and not just the algorithms) smarter.
• Unique algorithms: the book contains original algorithms that I am experimenting with. It also contains heuristic adaptations of existing techniques geared for fast, practical, use.
• Quick & easy: one does not need years of culinary training in order to cook up a wonderful meal. What we need is a good recipe and ingredients. Similarly, I believe that one does not need to be a mathematical savant¹ in order to practice predictive modeling.

Book organization

A table of contents is presented here. Even though many pages in the table of contents are not yet hyperlinked, they illustrate where the book is headed. I am adding new content to the book every week. You can scroll below the table of contents to see recently uploaded pages.

Each page or post has several sections. There may be a prerequisite section at the start of the post containing links to other posts that should be read first. The post may have a video tutorial section as well as a download area for working code.

I hope that you enjoy my content and find it useful. Most importantly, I that hope you have fun learning & applying predictive modeling!

The post Introduction appeared first on Predictive Modeler.

The idea of non-human intelligence has been around since antiquity. However, serious research on Artificial Intelligence (Ai) coincided roughly with the invention of the first programmable machines around 1940 ¹. Around the same time, Alan Turing’s work on the theory of computation showed that a machine could simulate almost any act of mathematical deduction just by shuffling ones and zeroes. If we presume human intelligence to be our ability to reason, it was suddenly and theoretically possible for a machine to mimic it. The promise of this research was immediately captivating and expectations soared high.

Lofty claims around a new technology are a double-edged sword. Funding pours into projects and companies, seemingly overnight. However, disappointment has a long memory and funding can dry up for years if expectations fall short. An Ai winter can set in.

There have been at least a couple such winters. The first one was during the Cold War. The US government wanted to translate Russian documents quickly. Instead, researchers quickly discovered that commonsense was not common or easy among machines. A story goes that efforts to translate “the spirit is good but the flesh is weak” from a Russian cable yielded “the vodka is good but the meat is rotten”².

I think that our knowledge is grounded in our embodied experience. We experience reality with several senses and that shapes our conversation. Language is messy and constantly evolving. No wonder that the promise of Ai did not translate at the time. But I digress.

The second Ai winter in the seventies was a bit of an own goal. The Ai community split between those that saw promise in rigid, top-down symbolic Ai (e.g. expert systems), vs. flexible, bottom-up connectionist Ai (e.g. interconnecting artificial neurons). Symbolic Ai won a battle but lost sight of the war. Expert systems were all the rage for a while. They were too hard to maintain, did not learn, and were not fault-tolerant. Several years and millions of dollars later, they fell from grace by 1990. Important advances³ were made in the theory of connectionist Ai, however computer processing power was just not there to apply them.

Research in Ai plateaued during the 90’s and 2000’s. Some members of the community re-branded themselves as cognitive scientists, working in informatics, analytics, or even machine learning. Watered-down…err…narrower definitions of Ai seemed technically feasible and general Ai considered a bit of a quack pursuit.

As a quick aside, narrow Ai is what is all around us today. Phone assistants like Siri, google search engine, image detection, or self-driving cars – are all examples of machines trained for specific (or, narrowly defined) tasks. General Ai is a machine that can do everything including being sentient. We don’t have one of those…yet.

Two things happened after 2010 that thawed the latest Ai winter. Researchers converged upon a specific type of connectionist neural network architecture as the best candidate for learning patterns from data. This will ultimately lead to deep learning. I will describe it in another post, but will note for now that it is remarkable that every example of Ai you see today uses essentially the same algorithm! This type of architecture required massive processing power. Available CPUs were struggling to keep up.

Cut to 2013. The stock of NVIDIA, a chip-maker for computer graphics, had been languishing in the low teens for years. The company noticed all these graduate students buying their GPUs⁴. They realized that these students had not become hardcore gamers overnight, but that the GPU architecture lends itself well to deep learning⁵. A GPU can have orders of magnitude more cores than a CPU. And neural networks are massively parallelizable. It is a marriage made in binary heaven. You can take chunks of a neural network and distribute them across GPU cores in order to compute them simultaneously. NVIDIA’s stock went from $25 a share in 2015 to almost $300 by 9/2018.

Narrow Ai surrounds us today. Even my thermostat tells me that it is learning. It sounds wonderful to have little Ai helpers take over mundane tasks like programming the thermostat, organizing pictures, or recommending which show to binge next. Such harmless fun. However, the likes of Bill Gates and Elon Musk have sounded off a warning alarm on the future potential of Ai.

One of my favorite exchanges involves the CEO of Facebook, Mark Zuckerberg, terming Ai naysayers as “pretty irresponsible” in a casual BBQ video. Elon was quick to respond with “I’ve spoken to Mark about this. His understanding of the subject is limited”. Ouch.

I am almost through my coffee, and I’ll save comments around the potential perils of general Ai for a future post. I’ll end by noting that this time it feels different. Saying that we have only achieved narrow Ai used to be a dig on the perceived unfulfilled promise of Ai. Today, we recognize it as Ai. In our pocket, in our home, and in our car. Our relationship with deep learning is becoming personal and pervasive. And our increased reliance on machines whose operation we no longer understand should concern us.

For Ai research and researchers, it does not feel like another winter is coming anytime soon.

The post The Fall and Rise of Ai appeared first on Predictive Modeler.

Blueprinting

Before you write any code, spend some time describing a blueprint of what you want your code to achieve. Code can be hard to grasp, especially densely written logic. Keeping track of nested loops, variables, and interim tables will test the RAM of even the most experienced of coders! It is easy to feel lost even when reviewing code you have written yourself, especially if returning to it after a break.

I recommend writing code in a sequence of scripts (or discrete blocks with clearly demarcated functionality), and the first in that sequence is a plain-english description of what each script is intending to accomplish. Ordered bullet lists work better than descriptions in paragraph form. The simple exercise of writing down the functionality and it’s connection to the overall purpose of the predictive modeling exercise not only helps improve documentation, it can help write organized, streamlined, better – code.

Prototyping

Especially when dealing with large data sets, it can be extremely inefficient to write & test code on the target data. Try to find a clean data entry point into your code-flow. And then write & test your code simultaneously by allowing a small amount of target data through. For example, use only the top (or random) one-thousand rows of data, rather than millions, in order to complete your code development. Prototype your solution with small amounts of data, and once satisfied, run the whole thing through it. This will help you develop error-free code quickly.

Yep, told you the ideas don’t look like they deserve calling out in so many words. However, many that are new to coding especially in data-rich environments tend to overlook these. I promise you, these two simple concepts can save you hours if not days of re-work!

The post Blueprinting & Prototyping appeared first on Predictive Modeler.