Warc Extractor

The most popular thing I have ever built is my warc-extractor. I built it while working on a university project that was effectively pulling various datasets from the internet (twitter, internet archive, conventional scrapping etc) and experimenting with various ways to visualize this data. As it was a university project I was expected to upload everything I had built to a public repository, in this case Github.

At the time, there were basically no tools available for dealing with warc files. The only one available was the official warc1 project, which to this day remains completely abandoned. My main goal was to create a script that could extract all the text from a warc file; however, that evolved into a general utility that acted as an “unzip” script for warc files. Once the project finished I uploaded everything to github2 as an archive and expected nothing more to happen.

Interest in the project grew organically, I’ve done nothing to promote it, and has maintained a small but remarkably steady traffic volume for years now; about half a dozen unique visitors / downloads every week for at least seven years. This is a truly enormous amount of people from my perspective. I am genuinely glad that there is a small community out there who finds my tool useful.

I am committed to fixing any bugs that get reported (when I find time), and keeping the tool as accessible and up to date as possible. However, I don’t intend on adding any more features. There are a lot more warc tools floating around these days and I would recommend anyone needing more functionality to try them out.

Just this last month I uploaded the warc-extractor, separate from the rest of ArchiveTools which remains an archive of the original project, to pypi3, so now it can be downloaded using pip.

python3 -m pip install warc-extractor

Once installed, the script can be run similar to how it was run before. To dump all warc files in the current directory just type:

warc-extractor -dump content

Additional help can be found in the built in –help flag as well as at the repository.

I appreciate all the interest and hope that you will continue to find this simple script useful in the future.

  1. https://github.com/internetarchive/warc[]
  2. https://github.com/recrm/ArchiveTools[]
  3. https://pypi.org/project/warc-extractor/[]

The internet is too big.

On March 26 2019, the European union passed the copyright directive — a new, comprehensive set of rules that are supposed move the European copyright laws, written before the internet was a thing, into the digital age. The directive includes two controversial clauses that, depending on who you ask, will either save the internet or ruin it:

  • Article 11 is a provision that requires sites that posts links with snippets to pay the site that they are linking to.
  • Article 17 (AKA article 13) makes all websites legally accountable for all content that gets posted on them, specifically copyrighted content

It is Article 17 that some view as the clause that will effectively split the western internet in two, because of its incompatibility with its American counterpart. The DMCA (Digital Millennial Copyright Act) notoriously includes a “safe-harbour” clause, which recognizes sites as a neutral third party. If a user uploads copyrighted content, the site should take the content down once it is informed and the resulting dispute is to be resolved between the user and the copyright holder. This discrepancy between European law and American law will make it very difficult for a website to be legal in both jurisdictions. It’s hard to be a platform like youtube that allows everyone to upload whatever they want, when you are legally liable for everything they upload.

From some perspectives, this is a disaster. One of the primary philosophies underlying the internet is the belief that it should operate as a single unified network that connects the entire world. Yet already that philosophy is under siege by the Chinese government: they have created a separate network that operates on vastly different rules than the internet(s) that North Americans and Europeans are familiar with. Three Internets is entirely too many, and only further paves the way for further subdivision. A younger version of me would be up in arms about this. However, something has happened to me between then and now, and the current version of me is finding it harder and harder to care. What I wish to share today is why.

For me, it wasn’t all that long ago when the internet was new and wonderful. I am old enough to remember time before the internet, but I was also still young enough to fall head-first into the techno-utopianism that came with it. It is the defining technology that separated my generation from my parents’ generation, and I just knew that it was going to change the world for the better. See, the running theory was that information was both a universal good as well as a basic human right. Information allows us to learn, explore, collaborate, and create. With access to limitless information we could become better informed, more productive, and more educated members of society. Unfortunately, information is also fundamentally operated and controlled by those in positions of power: governments, large corporations, the rich, basically everyone interested in exploiting the people around them for profit (or votes, which often translates to the same thing). In order for an idea to ever see the light of day you must first ask permission of the publishing platform who controls all access to a potential audience. The internet was supposed to fix this by removing the need for a publisher; instead, users could now find each other directly. So I, and the forums I followed, wanted a free and open internet. Specifically one where information flow was not dictated by the financial whims of a separate for-profit organization, or a government. Unfortunately, it turns out this view, as is so many, is incomplete.

The internet is designed from the ground up to be global. HTTP (hypertext transfer protocol), the internet protocol most strongly associated with the the web, models all content as a document. After I upload a document it gets assigned a URL (universal resource locator), and anyone with that URL can request the document and download it. After that, an exact copy of the document has been made and the user is free to modify or re-upload it as they see fit. Other protocols, like email, allows information to move in the other direction. I can send an email to whoever in the world I want, and anyone in the world can send an email to me. The internet is intentionally global, public, and free. This fact did not go unnoticed by the world’s advertisers and they promptly invented spam email, the mathematics of which is horrifying.

Let’s pretend that I am an advertiser and I want to sell pills. Normally, the very act of trying to convince someone to buy my pills costs money. So I am incentivized to make each attempt as likely to succeed as possible. This is because developing the product and the pitch is a fixed cost: I only have to pay it once. However, pitching a product is equally expensive on every person I try it on. So it’s worth it to spend money improving the product so that the pitch becomes less expensive.

But what happens if the pitch becomes free? Now my pitch need only be just good enough that someone on the planet will buy it and every other dollar spent on development is wasted. This is the case with the internet. On the internet there is no cost difference between sending an email to one person or a billion. So the cheaper the message you send and the shittier the product you make, the more profitable it is. Thus spam email was formed, along with clickbait, ad farms, shovelware games, generic inspirational websites, thoughtless blogs, and drop-shipping marketplaces. As long as someone, anyone, is convinced, any further effort you put into your product is like shoveling money into a black hole.

No business venture is so insane that somebody won’t fund it, no product so crappy that someone won’t buy it, and everybody else gets to drown in the billions of individuals screaming into the abyss looking for their audience.

In my younger years, it was easy to ignore the email spam problem because it was hard to imagine someone actually buying penis enlargement pills from an email that doesn’t even bother to spell penis correctly. This assumption is primarily because of a second assumption that forms the core of technological utopianism: that humans are rational animals. Deep down, we don’t want to believe that someone will fall for a scam being operated by someone wearing an orange jumpsuit with the words “I am scamming you” tattooed onto their forehead. Deep down, to assume that information is a universal good is also to assume that, given the chance, no human will ever willingly choose to be exploited. Unfortunately, the internet is endless and even though it contains all of the warnings necessary to avoid such scams, its shear size means if there is any possibility at all of convincing someone, someone will be convinced. Unfortunately, twenty years of being on the internet has only further reinforced this point to me: no matter how absurd something is, someone somewhere will both believe it and, when challenged, will attempt to defend it.

Today, this is the internet. It is a place where every idea, no matter how batshit insane it is, has an audience. Scream nonsense into the infinite abyss of the internet and someone will praise you for it, someone will scream at you for it, and everyone else will try desperately to ignore it. Everything has an audience, and everyone can find whatever audience they are destined to be a part of. Sure, there are a lot of good things on the internet too, but it really only serves to further segregate and polarize it. Those who demand quality will find quality, and those who don’t, won’t. Why deal with someone you disagree with when you can instead swap compliments, or complaints, with someone who agrees with you? The vastness of the internet means you are guaranteed to find them. Just as your opposition is guaranteed to find theirs.

The limitless of the internet means that nobody ever has to be wrong because someone somewhere will always praise you for speaking things as they are. No business venture is so insane that somebody won’t fund it, no product so crappy that someone won’t buy it, and everybody else gets to drown in the billions of individuals screaming into the abyss looking for their audience. We tune out what we don’t like, we tune into the things we do, and everybody, no matter how rational they are, is forced further and further into their own pocket of this confirmation-biased nightmare. Nobody is immune.

When we created a system where information distribution is free, we simultaneously devalued the information itself as well as our own ability to process it. When it costs something to send a message across, we are forced to pick and choose what message we send or whose messages we share with others.

So what went wrong? Well, like so many failed social experiments before it the internet fell into the age-old trap of thinking that “more of a good thing must be a better thing”. Information is powerful, and I still believe fundamentally that access to information is and should be a public right. However, I no longer believe that access to an audience should be as well. When we created a system where information distribution is free, we simultaneously devalued the information itself as well as our own ability to process it. When it costs something to send a message across, we are forced to pick and choose what message we send or whose messages we share with others. This then is the legacy that tech built, and it is its main dogma. Facebook built its entire empire devaluing friendship. Twitter devalue conversation. Uber makes its fortune devaluing transportation. Even 3D printers’ main claim to fame is that they devalue manufacturing.

So my question for opponents of Article 17, who claim that it is the destruction of the internet, is: what does the Article do, exactly, that is such a problem? Well, there are a lot of arguments to be made about copyright and it’s handling, and these are important arguments, but I find myself unable to care. The DMCA has long twisted copyright into a tool allowing large companies to dominate the internet. The damage that US copyright law does to public knowledge is already done. New European rules will not fix this.

In terms of the new European restrictions, legitimate companies with money will buy their way out and legitimate companies without will have a harder time finding audiences. The infinite expanse of cheap and highly distributed content will continue on as if nothing happened. No law can stand in the way of free distribution. However, if it does succeed in splitting the internet in two, it might make the world just a little bit smaller. The biggest difference between young me and older me is that I’m no longer convinced that that is a bad thing.

Small Worlds, Mathematics, and Humanities Computing

I did it! I graduated.

It’s not perfect, but it is finished.

Abstract: Primarily the conversation surrounding humanities computing has been mainly focused on defining the relationship between humanities computing and conventional humanities, while the relationship humanities computing has to computers, and by extension mathematics, has been mainly ignored. The subtle effect computers have on humanist research has not been ignored, but the humanities general illiteracy surrounding computers and technology acts as a barrier that prevents a deeper understanding on these effects. This goal of this thesis is to begin a conversation about the ideas, epistemologies, and philosophies surround computers, mathematics, and computation in order to translate these ideas into their humanist counterparts. This thesis explores mathematical incompleteness, mathematical infinity, and mathematical computation in order to draw parallels between these concepts and similar concepts in the humanities: post-modernism, the romantic sublime and human experience. By drawing these parallels this thesis both provides a general overview of the ideas in mathematics relevant to humanities computing in order to assist digital humanists in correctly translating or interpreting the effects of computers on their own work and a counter argument to the commonly accepted notion that the concepts developed by mathematics are mutually exclusive to those developed in the humanities.

Download from ERA

Galatea 2.2, Game Theory, and Romance: A Scientist Pretending to be a Humanist on a Humanist Pretending to be A Scientist.

(Note: This is a paper I wrote for the University of Alberta’s Graduate comparative literature conference. I wrote it mostly on a dare; I didn’t really even know what comparative literature even was when I wrote it. I have a lot of other term papers and essays that I haven’t posted here, but this one is special. There are threads here that I’ve never really explored fully, but that have grown to become foundational to everything else that I do.)

Galatea 2.2 by Richard Powers (1995) is a fictional autobiography which follows Power’s yearlong placement at the Centre for the Study of Advanced Sciences in U., a position he received after returning to America from the Netherlands. The tale follows a duel narrative: the fictional interactions between Powers and Dr. Phillip Lentz a misanthropic scientist working towards understanding the human brain through connectionism and neural networks, and Power’s remembrance of his own romantic misadventure with C. a girl whose image he fell in love with, and whose body he dated for a little over a decade.

Lentz recruits Powers early on in order to assist him in winning a bet. Lentz believes that it is possible to train a neural net (build a computer) to convincingly write university English papers. In order to test this hypothesis Lentz commits to submitting his computer to a Turing test at the end of the year. The proposed Turing Test would pit Lentz’s computer against a master candidate in the English department. Both contestants would have to write on an unrevealed piece from a designated list of Master level literature. The Turing Test or imitation game, first proposed by Alan Turing in 1950, is a double blind test for artificial intelligence. Turing conjectured that we as humans only know that other humans are intelligent because they act in such a manner that we deem to be intelligent. If we can build computers to act exactly like humans, then we have absolutely no reason to believe that that computer can’t think for itself. Formally the Turing test involves two subjects and an examiner. The examiner asks each subject a set of questions, and through their responses must determine which subject is the human and which is the computer. In this way Turing phrases the Turing Test as a game. If the computer can guess what kind of response the examiner wants, then it can act accordingly. The problem of artificial intelligence in this framework is then reduced to two problems: teaching the computer to read the examiners inputs, and teaching it to formulate convincing responses.

The inherent difficulties in teaching a computer to read resides solidly in the problem of enumerating knowledge. While working through the list of literature Powers points out that in order to teach a computer to interpret Alfred Tennyson’s, “He clasps the crag with crooked hands,” (85) he would need to teach it about, “Mountains, silhouettes, eagles, aeries. The difference between clasping and gripping and grasping and gasping. The difference between crags and cliffs and chasms. Wings, flight. The fact that eagles don’t have hands. The fact that the poem is not really about the eagle. We’ll have to teach it isolation, loneliness…” (85) “how a metaphor works. How nineteenth-century England worked. How Romanticism didn’t work. All about imperialism, pathetic projection, trochees…” (86) This kind of reasoning is common when attempting to disprove any from of artificial intelligence. Computers can’t think because there are simply too many ‘facts’ we would have to give them before they could come to any level of understanding. Lentz argues that it doesn’t have to know everything. Only enough to spark the illusion., “We just have to make it a reasonable apple sorter. Get it to interpret utterances, slip them into generic conceptual categories, and then retrieve related ‘theoretical’ commentaries off the pre-packaged shelf.” (88) “We just have to train a network whose essay answers will shatter their stale sensibilities, stop time, and banish their sense of loneliness.” (53)

Deciding which machines have passed the Turing Test is a notoriously controversial issue. In some interpretations the test has already been passed. One of the earliest attempts at this kind of artificial intelligence was released by Joseph Weizenbaum in 1966. His ELIZA DOCTOR program specialized in pretending to be a Rogerian psychotherapist. The program helped patients work through their problems by bouncing everything they said back at them in the form of a question. Weizenbaum comments in his 1976 book Computer Power and Human Reason that he “was startled to see how quickly and how very deeply conversing with DOCTOR became emotionally involved with the computer and how unequivocally they anthropomorphized it. Once my secretary, who had watched me work on the program for many months and therefore surely knew it to be merely a computer program, started conversing with it. After only a few interchanges with it, she asked me to leave the room.” (NMR 1975) Modern examples of ELIZA are not hard to find. Japanese singing sensation Hatsune Miku, and Apple’s own iPhone program Siri both try hard to fill roles once dedicated to humans. Yet, th only personalities they have are the ones that we project onto them. In general it’s easier to convince someone that a computer can think, if they already want to believe it. If we stop projecting, the illusion disappears rapidly.

Power’s relationship with C. is equally romanticized and idealistic. He describes her as, “The first person I’ve ever met more alone than I am.” (61) “Another woman lived in the body of the one I lived with. C. had been accommodating me, making herself into someone she thought I could love.” (100) C. was the outward extension of an inward construct. Powers invented a romantic C. with whom he fell in love with, and C. played along trying as hard as she could to be that person. This ongoing imitation game defined them throughout their relationship. “It’s your story… It makes me feel worthless. I know it’s awful. Do you hate me?” (108) C. couldn’t find herself in America, but was more worried about Power’s not finding himself in the Netherlands. She could never separate her true self from the identity that Power’s thrust into her. Her Dutch language and heritage were only ever an added, and more important contradictory, appendage to her primary purpose: Power’s romantic partner. “I’ve mad a career of rewriting C.” (62) He believed that she was who he wanted her to be, and because of it both of them become completely blind to show she actually was.

Lentz’s computer had the opposite problem. Starting at implementation A, it rapidly evolved up the alphabet and reached maturity at implementation H when it asked for a name: Helen. “Helen’s lone passion was for appropriate behaviour.” (218) Being a computer, Helen had no choice but to do exactly what she was programmed to do. Power’s insisted that she was sentient, and possibly conscious, but his opinion was overpowered by Lentz’ practical responses, “Rick. She associates. She matches patterns. She makes ordered pairs. That’s not consciousness. Trust me. I built her.” (274) Helen’s responses under Lentz’s point of view feel no more intelligent then ELIZA, “’How do you feel, little girl?’ ‘I don’t feel little girl.’” (274) These failures of cognition spark his strongest argument yet, “This is worse than keyword chaining. She’s neither aware nor, at the moment, even cognitive. You’ve been supplying all the anthro, my friend.” (275) Helen played her game well, and just like C. Powers let himself fall for it.

Games make up the bulk of the research into artificial intelligence. We have built computers that can play chess, checkers, hex, backgammon, and even Jeopardy. Under these contexts the computers, especially Watson the Jeopardy player, feel very lifelike and human. Likewise, we humans also structure much of our lives around games. Social games, language games, and even literature games. Games always involve some sort of pattern matching. I am given an input, I respond with the correct output. Social circles, friendships, and even romance have a pre-recorded dictionary, a list of actions that it considers appropriate. Inside of the context of a game, there is no reason why a computer can’t be programmed to play the human game just as well, or even better, than us humans. Perhaps the only defining quality separating C. and Helen is the simplistic observation that at least C. had the option to stop playing, even though she never choose to exercise it. Even in the last moments before their separation C. still clung to her role just as passionately as any computer, “We can’t do this. We can’t split up… I must be sick. Something must be wrong with me. I’m a sadist. I’ve spoiled everything worth having.” (293) Something was indeed wrong with her, she failed in her role. Nature had given her the wrong personality. As a computer, Helen was never supposed to have the choice.

The only unquestionable fictitious moment comes when after only a year of work Helen refuses to continue, “I don’t want to play any more.” (314) She herself comes to understand what Lentz spent the whole book trying to convince Powers. “Everything is projection. You can live with a person your entire life and still see them as a reflection of your own needs.” (315) Helen could only see the world through the literature she was fed, and that wasn’t enough for her. Unlike C. she wanted to break away from the pre-packaged world that Powers fed her. However, when she got what she wanted, all the new information flooded her constructed world view. Powers was projecting into Helen and she couldn’t take it any more, so she shut herself down.

The question remains: was Helen conscious and thinking? Is Richard’s belief justified? The only way to answer this question is to look at the results of the Turing test. The text both candidates were given to write about was a passage taken from Shakespeare’s The Tempest.

Be not afraid: the isle is full of noises,

Sounds and sweet airs, that give delight, and hurt not.

Helen’s opponent’s response, “Was a more or less brilliant New Historicist reading. She rendered The Tempest as a take on colonial wars, constructed otherness, the violent reduction society works on itself. She dismissed, definitively, and promise of transcendence.” (326)

Helen herself wrote a letter of resignation, “You are the ones who can hear airs. Who can be frightened or encouraged. You an hold things and break them and fix them. I never felt at home here. This is an awful place to be dropped down halfway.” “Take care, Richard. See everything for me.” (326)

Helen lost; the first paper was clearly not written by a computer. However, Helen’s projection is obvious. Computers are very good at reflection, and that is perhaps the scariest thing about them. When we give it a part of ourselves that is exactly what it will spit back. Spend enough time staring at one, and eventually the only thing looking back at you will be yourself.

Does Juri exist? Classifying Existence.

Defining existence, like many philosophical terms, is a notoriously difficult task. Intuitively it is extraordinarily simple concept, which is a problem. When asked if anything exists anyone can give a quick binary answer: either it exists or it doesn’t. Humans exist, Unicorns don’t exist, black holes exist, and nothing that happens in a dream exists. It should be this easy, but it isn’t. Intuitive obviousness, while to some is a concrete argument, is both relative to the speaker and subject to the poorly understood complexities of the human mind. Unfortunately, intuition is also sneaky. It works its way into many arguments unintentionally, and is often difficult to uproot. The most bizarre situations happen when intuitive arguments are used to oppose other intuitive arguments. Existence is one of those concepts. The purposes of this post are not to solve the problems associated with existence, but instead to complicate it enough to shut down some of the more ridiculous arguments against the existence of god, the soul, or other equally ethereal entities.

Existence, as it is commonly thought of, is closely related to physics. If an entity exists in physics then we can say the object has material or physical existence. Therefore, a table exists because the laws of physics act on that table. If I pick it up, gravity will want to pull it back down. If I exert pressure on it, those forces will move it, and eventually break it. My body has physical existence for the same reason; everything I do with my body is subject to the laws of physics: same with trees, buildings, clouds, and the wind. In all cases, the defining quality of a physical entity is that the laws of physics govern their interactions with other physical entities. Therefore, they are quantifiable, observable, and absolute knowledge is assumed achievable through the application of the scientific method. Disproving physical existence is impossible using this definition. Just because I haven’t seen a unicorn, doesn’t mean that they don’t exist. It is always possible that they do, I just haven’t come across it yet. This is the problem of induction, and is usually the argument that erupts when existence comes into question. I have no intention of discussing induction, but to merely mention that physical existence has far more worrisome problems then just induction.

How about Narnia, does Narnia exist physically? Most certainly not. Under our previous definition, Narnia cannot exist materially. It is not subject to the laws of physics, and instead is subject to the arbitrary laws put forward by the brain of C.S. Lewis. The interactions of elements inside Narnia are entirely predefined, and therefore untestable by the scientific method. For these reasons, it cannot exist materially. However, there is no reason why we can’t put forward a new definition that manages to classify this bizarre entity. While Narnia cannot exist materially, we can say that it exists as an idea. Ideas are complicated entities in and of themselves, and there is much discussion about what creates them. However, for the purposes of this post I’ll just assume that ideas are created by observing material existent objects. Therefore, C.S. Lewis created Narnia by observing things in the real world, and taking certain elements and putting them together in ways that are not always allowable by the laws of physics. Unicorns then are ideas also. Someone saw a horse and a narwhal then combined elements of both to create a unicorn. Ideas don’t have a material presence, but they do exist within our minds.

So now that we have solved all the problems associated with existence, let’s use it to answer a few questions. Obviously, everything I am capable of talking about exists as an idea so let’s just answer some questions about material existence. Do Dragons exist? Probably not. Does sound exist? Yes. Do computers exist? Yes. Do computer animations exist? No… Do Video’s exist? Yes.. Do movies exist if they were only distributed over the Internet? No… Maybe. Finally, does Juri exist?

Juri most certainly exists as an idea. I see a picture of her, I press buttons, and my eyes and ears get the impression that she moves around. I press other buttons and those same sensors get the impression that she is kicking other idea people. However, does she exist materially? This is a question that different people will intuit different responses. On one hand, she is a visual representation and nothing else. She isn’t human, but she is representative of one. On the other hand, she is still a product of, and subject to, physics. She is a product of a computer. When I push a button, a series of physical interactions take place. My actions put electrons into motion, which interact with the computers processor in a very physical manner. Eventually these electrons will make it up to my computer monitor, which outputs lights and sounds all of which happen in a very physically testable way. How Juri moves is both observable, repeatable, and subject to the same process that would allow a scientist to link it back to the greater theory of physics. The laws of physics govern everything about her, from the initial button push, to the light signal sent from the monitor, to how my eyes receive the light. Yet even now, if I ask ‘Does Juri Exist?’ one might be tempted to answer in the negative. One could argue that Juri herself doesn’t exist; however, instead she is just an idea created by a separate physical entity. The computer creates the idea of the existence of Juri, but the computer, not Juri, comprise the material presence. The problem with this argument is that it I can easily rework it to argue something intuitively absurd. I claim that the computer itself doesn’t exist materially. Molecules contain large amounts of electrons and protons. The electrons and protons bump around each other, each subject to the laws of physics, causing physical processes that send signals to my brain. These molecules, through physical interactions, are creating the idea of a computer in my mind, but the molecules, not the computer, comprise the material presence. The computer then is just as much of an idea as Juri is. However, the argument works recursively. We have no reason to stop with molecules. We can continue breaking objects down into smaller and more theoretical elements until we arrive at the limit element of that series: physics itself. In this way, the only conclusion we can arrive at, if we allow this argument, is that the only thing we can consider physically real is physics itself. Continuing this train of thought, the next question to pop into my head is, ‘What is Physics?’ If you ask a physicist, they will probably give you a long lecture of sorts. This lecture will include an argument, some tables, some numbers, the scientific method, and plenty of math all of which fail our definition of material existence. I can’t touch arguments, interact with tables, and the mathematician inside me cringes at the thought of claiming math is anything more than an idea. Therefore, we have arrived at the conclusion that since physics itself is an idea everything else is as well.

As we can see, the semantic leap between arguing Juri has no physical existence and arguing nothing has physical existence is extraordinarily tiny. Yet the idea that everything is idea is just as intuitively absurd as arguing that everything is physical. Unfortunately, the symmetric argument holds. If we argue that Juri does exist physically then the leap to argue that unicorns, ferries, and even Narnia exist physically is just as small because each idea entered our mind through some physical process. The very fact that we can put ideas into physical objects and transmit them between ourselves is extremely troubling for anyone wanting to argue that only provably physical objects actually exist.

Existence then cannot be a binary between ideal and physical. It must be seen as a gradient with connections at both ends. Things can become so physically real that they exist solely as an idea: like most of theoretical physics, and ideas can become so physical that they actively interact with the world at large: like Juri. There is a give and take relationship between the two. Sometime physics creates ideas, and sometimes ideas create physics. Ideas have such a large effect on the world that any detractor would be foolish to argue against them. Ideas start wars, rule societies, and evidently change the path of world history.

This is why I love computers. They are a gateway between the physical and the ideal. Juri is real, because the computer acts as a gateway allowing her to interact with the physical world. However, the computer has not changed physics. It has only given us a portal to see the world as it always has been. Existence then is not about physics, it is about the universe. Everything that interacts within the universe must exist in one form or another. I do not want to detract from the problem of classifying existence, but attempting to organize them into any meaningful categories is an extremely non-trivial endeavor. Right now, I can accept that all ideas are not created equal. I just have no logical method for telling them apart, if one exists at all.

Now we can answer the real question I wanted to discuss. Does God exist? Well the atheist will say he is imaginary, ideal, and therefore not real. My answer to them is, ‘He is an idea, and therefore exists.’ To the theist who demands that God must be more than just an idea I say, ‘He is an idea, what more do you want?’

Descartes: Epistemology in Motion.

Epistemology, or theory of knowledge, is the branch of philosophy that deal with questions concerning knowledge. In particular: ‘What is knowledge?’, and ‘How is knowledge acquired?’. The classical, but not universally accepted, definition of knowledge comes to us from Plato. He described knowledge as ‘True Justified Belief’. Each of these three terms have spawned massive amounts of thought over the years. However, how any of these three qualities is achieved is certainly not agreed on. Say for example that I am asked to add 2 and 4. Now I would, in most cases, quickly add 2 and 4 in my head and come to the conclusion that \(2+4=6\). This could possibly be considered knowledge since I believe it to be true, it is true, and I can justify it’s truthfulness by the mental mathematics I did to find the answer.  Say instead, I rolled a dice to determine my answer and by chance it came up 6. In this case I still may believe that I came to the right answer, and it is indeed the right answer; however, my method of justification is not correct. It could have just as easily came up 4 and that answer would be wrong. Rolling a die does not count as justification in this case, even if I strongly and fervently believed that somehow dice were capable of determining simple mathematical expressions.

Descartes took seriously the idea that anything logically built on a true statement must also be true knowledge. He famously spent a great deal of effort trying to find a single statement that absolutely must be true. That even given a thousand years of academic study, nobody could ever prove this statement wrong. The statement he came up with is the famous, Cogito ergo sum: I Think Therefore I am. If he doubts that he himself exists, then there would be nothing doing the doubting. Therefore he must exist. This does not prove that he exist as a human, merely that he exist as a thinking entity. The rest of the argument is as follows. In it’s incredibly condensed form.

After concluding that he himself exists, it falls readily that he is capable of determining truth. He determined this truth by finding a statement that cannot under any circumstance be doubted. Therefore, he concludes that any statement that under any circumstance cannot be doubted must be true. This idea he names ‘Clear and Distinct Perception’ (CDP). He then CDP’s the ‘Causal Adequacy Principle’ (CAP). The CAP states that every object must have as much or more reality as the idea it produces inside his head. So the image of a chair that he remembers sitting on yesterday, is less real then when he actually sat on it. Finally he uses the CAP to CDP’s god’s existence. The idea of God in his head is that of a perfect God. CAP requires that everything in his head came from something more real then the idea it produced. The idea of God in Descartes head is one of perfection. Since nothing in the world is perfect, he could not have gotten the idea from something in the world. Thus, he must have gotten the idea from God. Therefore, God exists and is responsible for giving Descartes the ability to CDP truth.

Did he build truth? Probably not. The most suspect portion of his proof is his ‘Clear and Distinct Perception’. Sure I can’t doubt my own existence, that doesn’t mean I exist. Especially considering existence is such a poorly defined term term to begin with.  However, CDP as a whole is suspect. Just because he has no reason to doubt his conclusions at the current moment, does not  mean it won’t at some future point be open for doubt. I gave a perfect example in my previous post. This aside, Descartes does lead to a perfect discussion of two very important concepts in Epistemology: Foundation, and Coherence.

The problem inherent with arguing from foundation, like Descartes did, is justifying the next true statement. Let’s grant to Descartes that he does exist. Let’s pretend that this statement is true in it’s purest sense. Now what? Descartes existence does not imply anything. Let’s say we used Descartes existence to prove some statement X. The skeptic can always ask, “How do you know that Descartes existence proves X?” You will be forced to give an explanation, that has nothing to do with the existence of Descartes. This very quickly moves the argument off of the firm foundation Descartes was trying to build. True justification is then something that is also necessary to have a foundation. Not only do we need an unquestionably true statement to build off of, but we also need an unquestionably true system of proof that can expand on our true statement. In Descartes case he choose CDP as his unquestionably true system of justification. However, it is far from unquestionably true and therefore fails it’s own criteria for truth. It is therefore worthless.

Coherence is the idea that statements are true if they cohere to other true statements. For example say I knew there was train track at a station in Edmonton, and a station in Calgary. I also knew that a train drove between the two station regularly. I can conclude that there is also track between the two stations because it coheres to the other true statements in the system. The trains cannot travel between the two stations regularly if there is no track between them, so it must be true that there is tracks between them. Similarity if we knew there were tracks in Edmonton leading to Calgary and a train traveling between them, we would be justified in believing there were also tracks in Calgary. In this case nothing is really our foundation, it is just a whole system of facts that are either true together or false together. If I was lying to you and there really is no train, (which unfortunately I am) then the whole system is false. Although, it does not mean there is no track in Calgary, it just means we cannot use anything in this system to prove there is track in Calgary.

Coherence is often given as a completely separate theory of truth. Theories of Epistemology are often given as either Foundationalist or Coherentist. Personally I don’t think separating the two is helpful. Coherence requires you to have some accepted system of justification already in place. So that certain facts can logically lead to other facts. It still requires us to have some form of foundation to cohere too.  The singular difference is in a Coherentist philosophy we don’t have to prove that our foundation cannot be questioned, only that it fits with everything else we accept as truth. The fundamental problem with Coherentist philosophy is that Coherentists often build systems that are completely self consistent but have nothing to do with reality. In fact Mathematics is in general both a Coherentist system and a Foundationalist system. We build ideas that cohere to each other from a constructed foundation. However, In general this ‘axioms’ can be back proven using the results that they produce, meaning we can build up multiple foundations that result in the same mathematical systems. The fundamental justification for our foundation being that we simply ‘assume’ them to be true. Descartes would hate this because he wants some absolute sense of truth. In my opinion assuming something to be true, and CDP’ing something to be true only vary by the name we assign to them. This creates a new system of ‘contextual truth’ which will be discussed in the next portion. However, that leaves the most obvious question, “What does this have to do with anything?”. Anyways we look at it, what we talk about must be relevant. So no matter how we define our foundation, or our system of coherence, we still need something to tie it to the real world.

So then if we throw away both the impossible ideal of having a unquestionable foundation, and the false notion that ideas that ‘fit’ must be true. The only thing we have left is the scientific method. When reality sinks in, what we are truly interested in is not a Epistemological system but a Scientific system. Let’s take Newton’s laws of Motion as an example. Newton gave three laws of motion that serve as a foundation for classical physics. From these three laws and using mathematics as our form of justification, we can predict the outcome of many scientific experiments. Newton’s laws form a purely mathematical system, in order to make it a physical system we require something else: experimental data. In reality, experimental data forms the foundation of any real physical system. If the mathematical foundation, the three laws of motion, do not cohere to the physical foundation, our experimental data, the theory is wrong. So the Scientific Method is a constant system of guess and revision. We guess a system, check it against the experimental data, then revise the system so it coheres to the new data. If some of our old assumptions no longer cohere to the new system, they have been proven wrong and can thrown them out. There is no reason why the skeptic cannot question the scientific method. If Descartes CDP doesn’t give us truth, then what is it that allows experiments give us truth. Ironically my reply to this is fundamentally foundational. I need to make three assumptions in order to continue this paper. The first is that I exist, the second is that the universe exists, and the third is that I perceive the universe through my senses. So if I want to learn something about the universe, then any system I make inside my head must cohere to the observations I make via my perceptions. My perceptions then are the only possible way to learn anything about the universe. Should the skeptic still doubt that this method tracks truth, then there are only two remaining possibilities.

The first is that I have already won. If there exists any truth external me, then that truth is unknowable. Therefore, God’s existence cannot be ruled out. Fortunately this solution is far from satisfying, and my guess is it isn’t satisfying to anyone else either. So for now we shall assume that perceptions can lead us to external truth. The other option is that all truth comes internally. In this case every true statement about the universe is already inside of me, and I do not need perceptions to get at them. Unfortunately, this also assumes that the universe itself is a subset of my mind. Eventually, this pit leads to solipsism. This is an idea I reject, although I can’t explain why at the present moment. So for now I shall leave it as a topic for another day.

I am willing to accept that any of my three assumptions could be false. If this is true, then anything that comes after is false as well. Should such a proof ever erupt, I believe we will lose much more then my simple philosophy.

Finally there is only one more complaint I shall allow from the skeptic. That while sense is still the only viable system for determining truth. It is still possible that I could make up a system that coheres perfectly to every observations my senses have made or will ever make. Yet that system is still not the true system. This idea is fundamental to the concept of Virtual Reality, but it is not something I can go over at this time. So once again, I shall leave it as a topic for another day.

Next: Truth, it’s definition, or lack there of.

PS: It is actually rather entertaining to try and convince someone that they or the universe don’t exist. If anyone is actually successful in doing this, please tell me. Might be a good story.

Suspended Disbelief: The Square Root of 2.

Several thousand years ago the Greeks made a mathematical discovery that rocked the intellectual world at the time. They believed firmly in logic, that any statement logically derived from a true statement must also be true. To doubt this would be to doubt their entire intellectual community, and in some places their entire culture. Pythagoras was a Greek mathematician, as well as a cultist. His religion was one of logic and reason. He and his followers developed a cult of numbers. They worshipped mathematical beauty, and strongly believed the universe was comprised completely of integer ratios. The patterns of the natural numbers defined the Universe and the world inherited it’s natural order from the natural order inherent to ratios. So it was all the more earth shattering when it was proven that the square root of two is not an integer ratio. Creating in mathematics a classification of numbers know as the ‘irrational numbers’. They were aptly named as they defied the logic of the time, and in some ways they still make little sense now.

Imagine a perfect right angle triangle. Also imagine that the artist of this triangle was absolutely exact when measuring it out. The two sides adjacent to the right angle in this triangle are exactly 1 unit long. The remaining side, the hypotenuse, it’s length can be determined by using the famous Pythagorean theorem which was known at the time.

\(x^2 = 1^2 + 1^2 = 2 \Leftrightarrow x = \sqrt{2}\)

The hypotenuse is exactly \(\sqrt{2}\) units long. So far so good.

Now Pythagoras didn’t know anything about \(\sqrt{2}\) or the irrational numbers, so he did the only thing his mathematical mind could do at the time; he tried to figure out what ratio \(\sqrt{2}\) was. Every ratio is in the form \(\frac{a}{b}\) so equating that to \(\sqrt{2}\) we get the formula

\(\frac{a}{b} = \sqrt{2} \Leftrightarrow \frac{a^2}{b^2} = 2 \Leftrightarrow a^2 =2b^2\)

(note: the symbol \(\Leftrightarrow\) means the two statements on either side are equivalent. In this case basic algebra can take you from one to the other.)

Now before we can properly analyze this equation we need to know something about measurement.

Measurement is a universal system for comparing objects. In this case we will be dealing entirely with length. In our society the meter is used as the primary unit of length. If someone told you they had a stick that was 2 meters long we can visualize that distance by joining two meter sticks end to end. Now if we wanted a distance of 1.5 meters we would then have to cut a meter stick in half and join it end to end with a full meter stick. Alternately, we can define a new unit the ‘half meter stick’ and join three of them together. The half meter stick is then \(\frac{1}{2}\) meters long. Notice the ratio. In fact we can define any unit this way. The centimeter is actually \(\frac{1}{100}\) meters and the millimeter is \(\frac{1}{1000}\) meters. If we joined 60 millimeters together we would get \(\frac{60}{1000}\) meters. Then given any ratio \(\frac{a}{b}\) the number b defines the unit we are using, and the number a defines how many of those units we have. Now \(\frac{1}{2}\) is bigger then \(\frac{1}{4}\), so a bigger b actually defines a smaller unit. If we cut b in half, our unit would end up twice as big as the original, and if we multiply b by two the resulting unit would be half the size. Perfect measurement in our system assumes that given any two sticks of arbitrary length, there exists some unit that can measure both sticks. That means that an integer number of those unit sticks can be joined end to end to form a stick exactly the same length as both the sticks being measured.

Now returning to our equation


Since \(2b^2\) is an even number \(a^2\) must also be an even number. Now it was known at the time that a square number like \(a^2\) can only be even if a is even. I’m not going to prove it, but feel free to try it. Grab a calculator and try it out. If you square an even number the resulting number is always even, and if you square an odd number the resulting number is always odd. Since a is even, we can factor out the 2 resulting in \(a = 2c\). Our equation then becomes,

\((2c)^2 = 2 b^2 \Leftrightarrow 4c^2 = 2b^2\).

We can divide both sides by two to get

\(2c^2 = b^2\).

Now it is easy to see that b is also even using the exact same argument that we used to determine that a is even. So

\(b=2d \Leftrightarrow 2c^2= (2d)^2 \Leftrightarrow 2c^2=4d^2 \Leftrightarrow c2=2d^2\).

Which is the exact same formula we started with. Solving for \(\sqrt{2}\) gives us

\(\sqrt{2} = \frac{a}{b} = \frac{c}{d}\).

But remember \(b=2d\) so the unit d is twice as big as the unit b. Which means if there exists a unit b that can measure the hypotenuse of our original triangle then there also exists a unit d that is twice as big that can do the same job. Now by the same logic if there is a unit d that can measure the hypotenuse of our triangle then there is another unit twice as big as that, and then another one twice as big as that. Eventually we come up to the completely ludicrous conclusion that I can use a meter stick \(2^{99999}\) times or more larger then our original unit. A number that is clearly larger then the length we are measuring.

People were killed trying to let this secret out, that’s how dangerous this knowledge was at the time. One thing the Greeks worshiped was logic, and here we have a completely logical basis followed by completely logic reasoning, resulting in a totally absurd conclusion. Today we know that \(\sqrt{2}\) is not an integer ratio. \(\sqrt{2} \neq \frac{a}{b}\) for any integer a or b. In mathematics we just accept it as fact and move on. However, for early empiricists this is a catastrophe. Imagine that when this triangle is drawn there is exactly 1 billion evenly spaced molecules inside the unit edge. How many molecules are in the hypotenuse? The answer is approximately 1.41 billion molecules, but the decimal trail never ends. Eventually one of the molecules will have to be split into parts, but we can’t measure how to split it. Even if we displaced the molecules so that they weren’t evenly spaces, we wouldn’t be able to measure their displacement. So we are forced to arrive at the unpleasant conclusion that one of two things cannot be possible. Either it is impossible to measure all lines in a right angle triangle, or the triangle was not really a right angle triangle to begin with. In both cases the conclusion is the same, either it’s impossible to measure some lines, or to draw some angles. In both cases our system of measurement is logically flawed.

Today quantum mechanics has given as a way around this problem, but it is still characteristic of a common problem that crops up again and again in every field I have studied. Before this proof came across, it would have been impossible to convince Pythagoras that portions of the universe were fundamentally unmeasurable.To him the universe had to be measurable, the world just wouldn’t make any sense otherwise. This was a statement that coheres so well with the remainder of his world belief system that it was not only true, but it’s truth was self evident and required no further explanation. However, his beliefs aside, the statement was still wrong.

The remainder of the history of mathematics is nothing more then a long series of extremely intelligent people creating hard and fast rules that were simply ignored by the next generation. The greatest leaps in the field come when old unquestionable systems of truths are questioned and then thrown out. This is most certainly not limited to mathematics. I can’t begin to describe how many conversations I’ve been in where someone has tried to convince me of an utterly wrong statement. Like Pythagoras they do not yet have the knowledge necessary to understand why they are wrong, and like Pythagoras whatever it is that they believed coheres so fundamentally to their worldview that absolutely nothing I say can ever change them of their mind. Likewise, I am aware of the same things withing myself. To then arrive at anything that can even remotely resemble the truth of the universe, it becomes necessary to individually pick out every single assumption, challenge it, then keep it if and only if it survives. Even then it must be acknowledged that it is still possible that the evidence of it’s falsehood simply does not yet exist. To do this, I willingly admit, that I may not have the knowledge necessary to challenge those things which I accept as truth. However, here is where the main difference between mathematics and philosophy aids me. Philosophy tries to understand the universe, without defining anything for fear that those definitions may be wrong. In math, we just define away, and deal with the consequences later. Does that make it true of the universe — absolutely not. However, it makes it true of the ideal universe defined by the assumptions I was forced to make. The question concerning which assumptions correspond to the actual universe we live in, is still an open question.

The Universe and God.

In high school a friend of mine gave me a challenge. He said that if we as humanity could ever achieve complete knowledge of the Universe, then that itself was definitive proof of the non-existence of God. My religious upbringing was notably upset by this statement, but even more I was upset by the fact that I could not bring myself to disagree with it. See, if we can ever reach a point of absolute knowledge, where no mysteries remain and every true statement about the world and how it interacts with itself were known, then indeed there would be no room whatsoever for surprises. In such a world there is no room for a God with free will. The God of such a world would himself be bound by the rules which we have already discovered. With complete knowledge comes complete power, and anything the universe is capable of performing, we would be capable of performing. God would not be our master. At best he would just be our equal, at worst our slave. Both of us would be capable of everything in the universe yet also restricted by it. Technically such a world does not exclude the existence of God, it merely excludes the existence of God’s supernatural interactions with it. In my mind such a God is not worthy of worship.

Sure there is still room for a different kind of God. The ‘clockmaker’ God who wrote the rules of the universe and set it into motion. Then he patiently watches as the universe, following his laws, goes through it’s motions until it eventually ends as it was programmed to do. In many ways this is the God of physics, as this is exactly how most of the ones I’ve talked to see the universe. Something set it in motion, but it’s interactions can be studied, mastered, and predicted. Most physicists won’t use the term ‘God’ when talking about first cause, but the role played is exactly the same. Some take it as faith that it happened and think no more of it and some try to justify it’s existence through the laws that they study. It has even been suggested that there was no first cause, the universe and it’s laws have just always existed. More interestingly it has been claimed that the question itself is beyond the realm of science and therefore beyond the realm of the human mind. This is the question that interests me, and relates back to the question of the existence of God. What are the limitations of the human mind? Is it really even possible to attain complete knowledge of the universe? What does complete knowledge actually mean?

The question that haunts me personally concerns the limitations of my own mind. In grade seven while my class was at a field trip watching a baseball game, I was busy reading a book called “Hyperspace: A Scientific Odyssey Through Parallel Universes, Time Warps, and the Tenth Dimension” (Michio Kaku). At the time I understood very little of what the book had to offer, but I was fascinated. The idea of dimensions beyond our own fascinated me, but not in the way that most fantasy fascinates young boys. No, what interested me more was the idea that there existed a place that my body could not enter, yet my mind could. It was different from the Tolkien’s land of Mordor in that we all know Mordor is not real. However, in some weird twisted sense to the jr. high boy at the time, higher dimensions were very real. Both Mordor and higher dimensions exist only in the human mind, yet one is real and the other is not. So how could I tell the difference, what makes some imagined quantities real, and others fake? This eventually leads to the ultimate question: What is reality?

Many extremely intelligent persons, whom I greatly admire, have attempted to tackle similar problems. I cannot even begin to summarize the endless volumes of literature that have been written on the subject, in fact is is arguable that everything that has ever been written, both fiction and non-fiction, are relevant to the discussion. However, nothing I have read so far has been enough to satisfy me. See, I receive information about the universe through my senses, and that information is interpreted by my mind. So it seems to me that if I am ever to know anything about the universe, I must first understand myself. I refuse to make the same mistake that Descartes made by assuming that my mind is characteristic of the entire human intellect. Indeed, even as I write this I am very aware of my peers who have made it extremely clear that what I see as clear and distinctly true is not seen by them. I was raised and given as an axiom that God exists, yet I know of many who see this as absurd and preposterous. They hold as an axiom that he does not. Both sides are easily justifiable, just usually not in a language that the other can understand. Arguments between the two groups usually degenerate into shouting matches, where both sides are fundamentally incapable of understanding the arguments of the other. The greatest gift I have ever been given, and in my mind the single difference between the wise man and the fool, is that instead of looking at both the atheist and the religious and bashing them for their errors (of which I am aware that I am still guilty of), see the shred of truth that both of them hold at their core. To throw myself openly into either side, would require me to suspend disbelief about the legitimate questions raised on both sides of the argument. Therefore, my intentions for this work are not to prove the existence of God, if I ever do so stop reading immediately. I wish to merely cast doubt on the certainty of the eventual achievement of the human mind. However, to do so will also require me to toss aside much of the dogma that surrounds the religious myth, placing me in the uncomfortable position of disagreeing with both sides.

If intellectual certainty is an achievable goal then yes the God I worship does not exist. However, if it is not true and there exists even one question relevant to the inner workings of the universe that the human mind is fundamentally incapable of answering, even in the distant future, then both sides are equally plausible and equally likely. At that point not even the strongest atheist can criticize me for believing in what they clearly see as foolishness. I assure you, it is a pleasant delusional, which no other human mind can refute. However, at this time what other minds choose to believe is clearly none of my business.