Introduction to Probability Theory, Part 3

All probabilities are conditional

Ok, now we come to an interesting point. If probability is attached to propositions, and propositions are about objective things that can be true or false, is it right to say that "the objective probability of proposition X is so and so"? What if you and me disagree in our probability assignment about the same proposition - I feel that this page is top-notch and you feel that it is mediocre, without either of us knowing the actual public rating?

Regardless of what you might have been taught about "events" having some inherent "probabilities" that "we" are trying to calculate, the above example of disagreement about probability of a proposition is a perfectly normal situation. We all know that it happens all the time. Just turn on your TV and look at some programme with folks arguing like crazy about different issues. Obviously, independently of how concrete a proposition is, people may disagree on its probability - it is a measure of their degree of confidence, not your degree of confidence after all! Now, the next question naturally is: why can different people have different degrees of confidence in truth of the same thing?

The answer is their different information context. Whether or not you hold some proposition for likely strongly depends on what other propositions you believe in. In a way, all the different propositions are related in our heads, and we are usually quite ready to change our opinion on one proposition after learning something about another. For example, you might be somewhat certain that I'm a native English speaker after reading my text, but if you could throw a brief glance at my passport, it would change your assessment. If you saw an entry "American" under nationality in it, you'd become (almost) certain about the truth of that proposition. On the other hand, if you saw some other nationality, you would become almost certain that the proposition is false. Now, someone else might not have had the same opportunity of looking at my passport and therefore assign a different probability.

Generally, what probability we assign to some proposition depends on what we already know about some other propositions. In mathematical speak, we refer to conditional probability - the probability assigned to X given that we already know that Y is true. In fact, for all practical purposes, all probabilities are conditional. Instead of saying that two different people assign a different probability to the same proposition, we may just as well say that they are just giving us two different probabilities concerning this proposition. The first person is giving us the probability conditional on A (her state of information), the other person is giving the probability conditional on B (her different state of information). There is nothing strange or disturbing about the discrepancy in numbers that arises then. On the contrary, if we could bring the two persons to believe exactly the same set of the "remaining" relevant propositions, they would agree perfectly on the probability assigned to the one uncertain proposition because they would effectively think exactly the same and thus lack any reason to disagree. This convergence of opinions is not easy to achieve, but it is not as far-fetched as it might seem. It can and routinely does happen during practical investigations and in science.

The important point to take from this part is: probabilities are assigned to propositions, but they are not properties of the propositions alone. Instead, a probability is a property of the proposition in question together with all other propositions held to be true by the person who assigned the probability. In fact, we can forget about the person altogether and just represent her by the totality of all propositions she knows to be true.

Introduction to Probability Theory, Part 2

Propositions - the carriers of probability

A probability is a number between 0 and 1 which expresses someone's degree of confidence in the truth of some proposition. A proposition is simply a statement of fact like "This page is over 1000 words long". In reality, every such statement can be either true or false. You would only talk about a "probability" if you were unsure which of the both (true or false) was the case. However, what you always do know up-front, is that the proposition is either false or true, but not both, and not something in between either.

What about statements like "This page is entertaining and informative"? How can it really be either true or false? Doesn't it just depend on who is judging it? Well, it does, until you define some way of measuring "entertaining and informative" which does not involve a single person's tastes. But let's say that we agreed on some voting scheme in which all potential evaluators would participate in. Then the "entertaining and informative" would no longer be up to your or my opinion only - it would become more of an objective property of this page. And yes, without having seen the actual ratings, you could be unsure about this property (how everyone has rated it). So you could assign different probabilities to all the possible "entertaining and informative" ratings it might have. In other words, you would then have propositions like "The entertainment rating is 0/10" or "The informativeness rating is 9/10", and of course each of them could be true or false, but not at the same time. You might feel more confident that this page has good ratings than bad ratings and express this by numbers using your probability assignment when asked about it.

The thing I'd like you to consider is that when we are discussing probabilities, we are talking about our degree of uncertainty about some concrete propositions. If the propositions appear fuzzy and their truth seems undecidable in principle, then we have to become more specific first and clarify what we mean before we can even start talking about and asking questions about probabilities. Obviously, if we don't even know what our questions are about, we cannot expect any definite and useful answers.

Incidentally, propositions like "a die throw result is 4" or "a coin throw outcome is heads" are very clear. Pretty much everyone agrees on what they mean and could check their truth just like anyone else. Now you see one reason why these sorts of propositions are so eagerly used in classroom introductions to probability. Still, there are many other propositions that just as concrete and a lot more fun to think about than these trivial examples.

Finally, note that the very reason why we talk about probabilities of propositions is that, although they are verifiable in principle (their truth could be checked - and we know how), they may be quite hard to verify in practice. Maybe the proposition is about something that has not happened yet; it could also just as well be about some past event. If we were able to directly find out whether it's true or false, we would of course just do it and we wouldn't waste time talking about its "probability". Probability is for situations where we have to infer the truth of a proposition from whatever indirect clues we can collect without doing miracles or spending a fortune.

Introduction to Probability Theory, Part 1

In this series of tutorial-style articles I recap what I have learned about probability theory from studying the work of E. T. Jaynes (available online here (book) and here (lectures)), which I recommend - with some reservations. The introductory parts are easy to read and enjoy. However, the later chapters are dominated by references to physics and mathematical formulae whose explanations are rather too brief for my taste. Jaynes seemed to write for students of physics at graduate level (even though I believe it was not his intention). I feel that his ideas are so intriguing and general that they deserve a broader audience. The goal of these posts is to introduce the most important concepts with fewer assumptions about the reader's level of mathematical sophistication; and to verify my own understanding in the process.

It's not just about coins and dice!

If you are like most people, you were introduced to the concept of probability at school with examples such as throwing dice, flipping coins, selecting cards from a deck, spinning lottery wheels, pulling colored balls from urns and other such. You will find plenty of such examples in various tutorials on the web, too. While there is nothing wrong about them in general, they can leave the impression that this is what "probability theory" is all about. A rather boring application of basic arithmetics to some idealized useless "random experiments" that noone cares about in real life. That is, unless they are after good marks for mechanical answers to silly questions like "what is the probability of scoring more than 2 but fewer than 8 with two dice". It appears just about as exciting and thought-provoking as solving quadratic equations for sports.

What they usually don't tell you is that probability theory describes what you - and everyone else - have been doing for your whole life with more or less success, without even realizing. All kinds of reasoning and decision making depend on probabilities that people assign to various propositions:

  • Whenever you look at something (like Escher's drawing of waterfall on the left), you unconsciously figure out the probabilities of seeing different scenes. You make up your mind what the scene is about and whether it is "real" or not;
  • Before you cross a street, you unconsciously figure out the probability of being hit by a car and getting to the other side safely;
  • Whenever you decide to buy something, you figure out the probability of getting good value for your money;
  • Detectives figure out who dun it based on probabilities of finding particular criminal evidence;
  • Criminals figure out how to reduce the probability of getting caught;
  • Scientists figure out which explanation is more probable than others for an observed phenomenon;
  • Businessmen figure out which deals are more likely to bring them profits;
  • Politicians figure out which public statements are more likely to bring them voters;
  • and so on, and so forth.

The really important thing to notice here is that we are almost never 100% certain about anything. We can be rather sure or rather doubtful about different things, but we can hardly ever honestly proclaim: "I know it's a sure thing" or "I know it's completely impossible" - except perhaps when trivial and uninteresting stuff is concerned. To put it in a slightly different way, whenever we need to think and make choices, there is always some uncertainty involved.

Real applied probability theory is about systematically improving our everyday thinking and decisions:

  • It's about drawing the best conclusions from whatever we already know and understand;
  • It's about not getting fooled and confused;
  • It's also about knowing how to act to become more knowledgeable about stuff that matters.

The concept of probability is quite difficult to grasp, though mathematically very simple. A tiny little part of it is about throwing dice and shaking urns in the classroom.