Intelligent recognition: the case of xThink pen-based math

xThink intelligent recognition software empowers learning and discovery for students. To achieve this goal, the software radically reduces complexity for the end user.

For example, xThink’s first products — 2003 and 2004 — provided pen-based math, eliminating the need for keyboard and mouse. This blog entry reveals the technicalities behind this first expression of xThink intelligent recognition technology (IRT). Maybe this information will satisfy the inner geek of some MathJournal enthusiasts out there.

Note:   Keep in mind that math is only one of the places where IRT brings benefits. IRT adapts to other types of recognition, too, such as recognition of flow charts, graphical programming languages, schematics, and images. xThink’s math products exploit only a small facet of what IRT can do.

Consider the following handwritten math expression:

IRT recognizes it as follows:

1. Recognize each individual symbol in the math expression, with reference to xThink’s massive database of recognized symbols.
2. Recognize the 2D spatial relationships of the symbols as a mathematical graph, which is a set of nodes and connectors.

The graph organizes the information as follows:

• Nodes represent the symbols and their relationships.
• Edges connect nodes.
• In complex expressions, the edges can carry semantic details about the expression.

Note: Very complex expressions, can be represented and analyzed as hypergraphs (overlapping sets of nodes and connectors).

(The reader must not confuse a mathematical graph with the 3D graphs used in geometry. They are distinct.)

3. Create an adjacency matrix for the graph. Unlike a graph, the matrix format is easy to process mathematically. Use mathematical rules to simplify the matrix, without loss of detail. The following rules simplify a specific adjacency matrix.

4. Transfer the simplifications of the matrix back to the graph to create the simplified, unambiguous essence of the original graph. If a basic ambiguity remains, apply a minimum spanning tree to resolve ambiguity.

5. Recognize the specific type of mathematical problem.

Note: In contrast, this type of recognition is not available in scientific calculators. Instead, you must configure the calculator to process your inputs in a specific way. For example, you must take steps to configure a graphing calculator to solve an algebra problem or statistics problem or plotting problem or integration problem, and so on.

6. Apply the appropriate math and display algorithms to generate the solution that the user requests.
7. Present all available solution options to the user, such as numerical, zeroes, extrema, plots, and graphs.

Pen-based math was only the beginning for xThink’s IRT. xThink Intelligent Tutor is the newest expression of xThink’s IRT. (More information on Tutor is here: xThink Videos, January 2011 blog entry) This new software exists in prototype form, and it is quite different from the IRT for pen-based math. However, the Tutor follows the general xThink mission: Empower learning and discovery; hide the technology.

Companies dedicated to R&D

Here’s a list of companies where R&D is a primary focus. This is pretty much a random list; definitely incomplete. Because xThink is an R&D-focused company, it’s nice to contemplate the experience of our cohorts, especially since most of these cohorts are so successful!

3M Company:  The company’s initial tight focus on mining (in 1902 Minnesota) had to change to R&D for the mining industry by 1914. By the 1930s, 3M delivered Scotch tape and by 1980 3M delivered Post-it Notes.

Battelle Memorial Institute: Founder Gordon Battelle believed that the best way to use his wealth was to benefit mankind through science. Initially he and his institute focused on the area of science that he knew best, metallurgy.

Bose Laboratories: Founder Amar Bose created a company to solve his own problem; finding speaker technology that reproduced the realism of a live performance. In comparison, xThink’s initial problem was to create “intelligent paper.”

Dolby Laboratories: Brilliant R&D. However, more interesting to xThink is their market strategy: don’t sell consumer products outright; license technology to consumer electronics manufacturers.

Intel: Gordon Moore, co-founder, recalls, “… quite by accident, [Intel co-founders] ran into Sherman Fairchild, who happened to be a technology buff; he really loved new  technology.  He introduced them to the chairman of Fairchild Camera and Instrument, who was willing to take a shot at supporting this new company.” Source: http://calteches.library.caltech.edu/570/2/Moore.pdf

National Instruments:  Flagship software product started in 1986 on the then more “toy-like” Macintosh platform. But that way, in 1992 when the product was ported to Windows (a much larger market) customers got a seasoned, stable tool.

RAND Corporation:  Founded by the US military to study strategy, national security, and analyze systems.Rand’s necessary secrecy leads to parodies (the film “Dr. Strangelove”) and fears in popular culture. In contrast, xThink has the luxury of transparency in its mission and activities, because the focus is supporting the human passion to learn.

Wolfram Research: The company’s industry-leading mathematics engine is a firm foundation for the new Wolfram Alpha “answer engine.”

Wright Cycle Company: The Wright Brothers, aviation pioneers, used “day jobs” in their bicycle shop to fund their R&D passions.

Partnership, xThink style

The concept of partnership, xThink style, is a direct outgrowth of xThink’s unusual culture and values.

xThink is primarily a research and development company. xThink has never sought venture capital that could fund a traditional enterprise with departments of sales, marketing, legal, HR, and so on.

Instead, we want to participate in development of next-generation software. We are willing to suffer privation, sustain day jobs, and wear multiple hats among our tiny dedicated staff, just so that we can always keep our eyes on the rewards of scientific research in advanced computer technology. We have a superior mix of personnel, technology, and deepening practical experience. These qualities will allow us write a chapter in the inevitable expansion of artificial intelligence that satisfies peoples’ passion to learn.

This “culture and values” factor clarifies xThink’s unique proposal to potential partners: work with xThink for the technology advantage that it can provide to the partner company. xThink’s ideal partner values that technology advantage and not xThink’s current or future market share or profitability.

If xThink fails to achieve its business plan, the technology will be built by someone else, based on other technical and funding models. But the point is, some parallel technology will be built by someone eventually. xThink’s mission is not rocket science; anyone could embrace it. It goes something like this: Empower learning and discovery; hide the technology. But can large corporations fulfill such a mission? Is it inevitable that 1) Focus on return-on-investment competes too strongly against 2) Focus on maximizing learning and on providing low-cost tools for broad consumer sectors? xThink doesn’t have the answer. We do know that focus item 2) always wins the competition at xThink. Hence, our corporate structure and style of partnership.

Here’s hoping that xThink’s modest progress continues!

How can computers share their math mastery with us?

Computers easily master math’s structure and rules. The question is, how can they share their mastery with us? With the help of xThink’s IRT (intelligent recognition technology), computers share their mastery of math with students and professionals. Often considered the “hardest” of academic subjects, math became a more user-friendly pursuit in xThink’s Phase One, which featured the xThink MathJournal product.

In 2003, xThink’s became the first company in history to deliver a product that realized the programmer’s dream since the 1960s: handwritten math on a computer screen. This dream arose not simply because pen-based math is fun or cool. The dream of pen-based math addressed several painful customer problems:

• Initially mathematicians had to know how to write programming code to solve math problems on a computer.
• Cheap, electronic calculators began to emerge in the mass market in the 1970s. Their capabilities rapidly increased. Then users faced the pain of needing to regularly upgrade hardware. Another major pains was the requirement that users learn the various user interfaces of the calculator brands.
  The calculators of one popular brand (HP) to this day features Reverse Polish Notation instead of the standard calculator notation based on the conventions of pencil-and-paper math. Each notation system had merit. The point is that the conventions of the user interfaces were not standardized.
• In the 1980s, mathematicians increasingly had the option to free themselves from the varied hardware formats of electronic calculators. How? Various software applications for math emerged that ran on personal computers. Among these products were Mathematica and Matlab. Unfortunately, pain accompanied the new functionality. Each product required unique, arcane sets of keystroke commands and, soon enough, mouse selections to invoke mathematical operations.
• Another limitation of both the math software applications and calculators is significant with respect to xThink:
  • Users have to “know what they were doing” mathematically to operate the traditional math problem-solvers. For example, to analyze data on a computer, the user enters keyboard and mouse commands to enter a mode where she or he can define the data to process. Then the user enters keyboard and mouse commands to move the data to a storage space. Then the user enters keyboard and mouse commands to specify an exact way that the data is to be processed and rendered.
  • In contrast, xThink MathJournal automatically recognizes the type of math problem that you write. The user does not have to indicate that he or she is working on algebra, quadratic equations, graphing, integration, systems of equations, and so on. MathJournal “knows.” Then MathJournal presents a small set of relevant solution options, such as, numerical, plot, extrema, and zeroes. Keyboard, mouse, and mathematical “modes” are not required!

After xThink MathJournal was released, another benefit was evident alongside the painful problems that the product resolved: pen-based math is fun! And pen-based math is intuitive, harkening back to the familiar format of pen and pencil from everyone’s early schooling.

Alas, we need more than pen-based math. Customers told xThink as early as 2004, “we need step-by-step instructions for problem solving!” We couldn’t agree more. xThink Intelligent Tutor will address that need: http://www.youtube.com/user/xThinkVideos#g/c/3D74437E889BB74C

xThink videos

Honestly. We at xThink have the best of intentions to provide a monthly blog item … at a minimum!

At the beginning of 2011, several xThink screen capture videos and live videos were ready regarding various issues. But there was always “one more thing” to prepare. It’s already March (argh!), but we’ll cheat and back-date this post to January.

Without further ado … we have posted the following playlists on YouTube to help you understand where xThink has been and where it is going:

• 2011 R&D status: the now of it and the soon of it. (Hint: the report includes an R&D progress report regarding iPad and Android tablets.)
• Intelligent Tutor: the state of the emerging art of step-by-step tutoring; what we and our customers have been dreaming about, frankly, since our first pen-based math product was released nearly 10 years ago!
• MathJournal: the quickest, best way to learn about our flagship product. MathJournal helps you solve and visualize the math behind science, technology, engineering, and, of course, math itself.
• MathJournal Whiteboard: what works so well on Tablet PC is even more compelling writ large, as they say.

You are cordially invited to take a look!:  http://www.youtube.com/user/xThinkVideos

What’s the concept?

No one would object to the idea a student needs to understand concepts to achieve mastery of subject matter. Otherwise, the student would be no better than a calculator, waiting for inputs and indifferently spitting out answers. However, what type of concept is relevant? How big; how small can the concept be?

In building its Intelligent Tutor, xThink breaks down the concepts required by students to solve a problem as far as possible. For example, to master multiplication a student needs to have a sense of the sequence of numbers and to know that 4 times 5 is a shortcut form of addition, the equivalent of 4 + 4 + 4 + 4 + 4. We call these conceptual components “metarules.” And metarules can be pretty minute.

How minutely should the Intelligent Tutor’s metarules disaggregate math concepts? The answer is; the smaller, the better. That way, the tutor can meet each student at his or her level of comprehension, then raise the bar step by step through activities, examples, and exercises. Whenever a student shows mastery of one or more concepts, it is time to move on to new conceptual challenges.

Could the Intelligent Tutor explain things so minutely that the student loses interest? Yes, that’s a theoretical possibility. However, the tutor’s job is to adjust dynamically to the performance of a student. Each student is like a snowflake, unique. The amount of time a student needs to spend on exercises or with any minute concepts varies, and tutoring software can adjust accordingly.

Warning. It is risky to declare one student to be behind or ahead of others. Mastery of any one particular concept might be the key that permits a “slow” student to leap forward in learning. Learning need not be — might never be — a linear, incremental process. Learning is cumulative, yes (especially math), but not necessarily linear. Furthermore, fastness is not universally good, nor slowness universally bad. Consider Einstein’s self-assessment: “It’s not that I’m so smart, it’s just that I stay with problems longer.”

Are some students a lost cause? Never! Intelligent tutoring software has all the time in the world to generate customized, dynamic instruction and encouragement. Research shows that students have deeply rooted curiosity and joy of mastery, which an Intelligent Tutor can nurture.

Cognition and learning can be deceptive. A student can remain on one plateau of understanding and then suddenly display the confidence and insight to climb to new heights of mastery. Each child assimilates knowledge and builds skill differently. Intelligent tutoring software observes student history and performance and asks, “Were the concepts broken down sufficiently?” and “Was sufficient practice time allowed for confidence building and skill building?”

The preceding ideas and examples provide a glimpse of the issues of cognition and learning at the heart of building an Intelligent Tutor. And this glimpse helps frame the discussion of concepts. Perhaps all can agree that tutoring (and teaching) must convey concepts, so that students achieve deep understanding. However, some people might doubt the efficacy of breaking down concepts to a very minute level and encouraging phases of repetitious practice and drill. xThink Intelligent Tutor will provide proof.

In sum, conceptual understanding is a generally accepted goal of learning. Less familiar is the idea that learning is based on sets of minute concepts that each student can coalesce in good time, with proper instruction.

Bumper stickers and human consciousness

Intelligent tutors, such as the still-under-development xThink Intelligent Tutor, must have a sense of humor. Otherwise, they cannot become cyber-friends who engage and motivate students.

Today, if you saw our Alpha code for xThink Intelligent Tutor, you might say, “Whoa! Today, the Tutor can barely monitor a student’s math exercises, detect errors in real time, and suggest alternative paths. A sense of humor is a highly advanced skill worthy of a computer that has human-like consciousness.”

And you would be correct. Yet, we are confident that the Tutor’s sense of humor will emerge with all its other skills, over time. One source of our confidence: the Tutor builds its answers from scratch, from a foundation of facts and experience, in response to a specific learning context. No canned speeches or pre-written dialogs.

Soon, the Tutor will take some baby steps toward a sense of humor and conscious-like behavior. It is comforting to see open-minded people who have a sense of humor about consciousness itself. The following humorous bumper stickers (seen on a single car near our headquarters in Round Rock, Texas) provide a case in point:

• You’re just jealous because the voices are talking to me.
• Don’t believe everything you think.

Bumper-sticker 1 is funny, in part, because a person who is normally considered diseased (one who hears disembodied voices) proudly declares herself to be enviable.

Bumper-sticker 1 is significant for the students of consciousness, because it expresses a common analytical approach. For example, Marvin Minsky’s idea on this topic is summarized by the late Push Singh as follows:

. . . every mind is really a ‘Society of Mind’, a tremendously rich and multifaceted society of structures and processes, in every individual the unique product of eons of genetic evolution, milennia of human cultural evolution, and years of personal experience. – – http://web.media.mit.edu/~push/ExaminingSOM.html

Bumper-sticker 2 is funny, in part, because it upends the meaning of the aphorism, “Don’t believe everything you hear.” Beyond that, the bumper sticker invites us to view our own thought from a meta-perspective. This is the perspective that all students of consciousness seek.

xThinkers wonder what the Tutor’s emergent behavior will teach us about consciousness . . . and what sense of humor the Tutor will apply in the process.

Tablet Pillow Talk

Jeez, Bill, quit punching your pillow. Take it out on the tennis court tomorrow.
Can’t help it Melly. I invented the 21st century tablet. We owned it lock, stock, and barrel. Now the rocket ship is taking off without us.
Don’t exaggerate. Still, sooner or later the next Elvis comes along; you had an incredible string of hits.
No. The “next Elvis” is a former Elvis, Jobs.
Nah, you watch; the era of superstars is over.
It coulda been me.
Let it go, my little gatekeeper. Did you really want to kill yourself getting the elephant to dance one more time?
Shoulda, coulda, woulda. I love you, baby.
I love you, too. Now sleep.

Where’s the software?!

REFERENCE: http://online.wsj.com/article/SB10001424052748703860104575507873294920834.html

The following statement about new tablets from a recent Wall Street Journal article is staggering, because xThink builds strong software tools and is itching to build a lot more:

“Part of what’s driving this experimentation is the lack of strong software tools, she added, which is causing a lot of manufacturers to experiment with size and form factor in order to spur unique applications for their products.”

And notice that the author says that the manufacturers are relying on consumers to blaze the trail to the future. That’s right! Here’s a paraphrase of what they are saying above, “The manufacturers are building tablets in various sizes and shape, hoping that customers will really like one of the tablets and start using it in new ways.”

This brings to mind politicians who are ruled by opinion polls. “Leaders” who follow instead of leading. There are two problems with manufacturers who apply this approach:

Problem One: iPad is wildly successful by helping consumers do what they already do (iPhone and media consumption) better. Yes, the spurring of unique applications for iPad is coming, too, but that’s not the source of the wild success. So what makes manufacturers think that consumers will lead innovation?

Problem Two: Over the millennia, scientists and engineers (thinkers, designers, inventors, and developers) focused their minds on innovation. These manufacturers must be stunting their engineering staff into the role of mere construction workers, not the visionaries and architects that they can be. Where is the passion to use knowledge and skill to overcome Great Challenges that have led engineers time and again to miraculous triumphs?

Related blog entries:
https://tshuatxthinkdotcom.wordpress.com/2010/09/15/intelligent-tutors-a-%e2%80%9cgrand-challenge%e2%80%9d-for-engineers/
https://tshuatxthinkdotcom.wordpress.com/2010/08/28/ich-bin-ein-app-builder/

(iPad + iTunes) = You become a server manager

To grow and evolve, software R&D firms like xThink need to realize that Apple iPad is not branch of the Windows Tablet PC initiative. Instead, iPad represents a new, parallel path in tablet computing. The new path includes 1) an alternate approach to user interface and 2) a cell phone-based operating system (OS), not a computer operating system, and 3) most importantly, a server model for managing software and media, based on iTunes.

A new era of tablet computing (the first in this century) began in 2001, when Bill Gates debuted the Windows XP for Tablet PC operating system on a prototyped tablet computer. xThink soon contributed its intelligent pen-based math software. And xThink expected to ride a Windows tablet boom that would eventually branch off into the Linux and Macintosh operating systems.

Neither the boom nor the branching ever happened! In fact, the bullish predictions for Windows tablet computing proved false. It didn’t help that the tablet’s cheerleader and champion in 2001, Bill Gates, had already stepped down as CEO of Microsoft in 2000.

Fast forward to today. Since its debut in April 2010, iPad has breathed so much new life into the tablet computing market that new (Google Android) and old (Windows Tablet PC) initiatives are springing up everywhere!

Yet iPad is a new, parallel path in the tablet computing world, not a clone of Windows XP for Tablet PC functionality. iPad is not even based in Apple’s Macintosh OS! Instead iPad’s OS is rooted in the cell-phone OS of the iPhone (which debuted in 2007).

So what parts of iPad’s unique roots affect a software R&D company like xThink?:

• At your finger-tips: Until the iPad, xThink thought in terms of stylus-based input. Now, iPad brings an era of “finger-tip only” input (and multi-touch input). So, will finger-tip input work for xThink’s pen-based math? Yes! Happily, xThink staff is finding finger-tip entry of text and numbers to be perfectly functional. Although we miss the precision and compactness of the stylus, there are several ways to make finger-tip input of math viable.
• Controlled environment: The iPhone OS (and by extension the iPad OS) is a controlled environment. Input options are more limited (for example, no keyboard, stylus, or hard-wired ports), which makes the iPhone and iPad simpler, faster, and more stable. The exclusion of the Adobe Flash Player represents another control strategy in iPhone/iPad. The most notable control factor is Apple iTunes (see the next bullet).
• iTunes server model: A server is management software that connects multiple computers, cell phones, and other electronic devices together, so as to provide collective services. In the classic server model, this management software runs invisibly in the background of cyberspace to handle email, traffic, security, archiving, and so on. iTunes is management software (in other words, it is a server) for media, but it runs front-and-center in the foreground. iTunes manages music, books, games, tools, films, etc., so that you can access the media from multiple devices, such as iPod, iPhone, and iPad and computer.

Because it is a server, iTunes must run in full operating system like Windows or Macintosh. A cell-phone operating system like iPhone/iPad OS is not sufficient. The cell-phone OS offloads work to the iTunes server. With iTunes, server transactions become an everyday event for consumers. Here’s what it is like to work with the iTunes server and iPad:

1. Install iTunes on Windows or Macintosh.

2. Add media (music, books, games, tools, films, and so on) to iTunes. iTunes’ server activity includes, blocking unsupported media, optimizing the media, and managing copyrights for the media.

3. Connect your iPad (and iPod and iPhone) to the computer that runs the iTunes server and copy over (synchronize/“sync”) the media that you want.

In sum, iPad brought an evolution of tablet computing that xThink did not expect. Linux and Macintosh paths did not branch off from the Windows Tablet PC initiative. Instead, iPad created a new parallel trail based in a cell-phone OS.

Of all the traits in its parallel path, iPad’s server model based in iTunes has the most significance for R&D firms like xThink. Apple is making server management an everyday activity for consumers, in this case the iTunes media server. The more services, media, and tools that iTunes provides for the iPad, the more adept consumers will become at dealing with servers on an everyday basis.