Smart document generation

If giving legal advice is one of the two core skills of legal practitioners, the other is drafting legal documents. No matter what area of the law you practice in, you will need to generate a brief, a lease, a will, a contract, a certificate of incorporation—you name it. It is no surprise therefore that ever since PCs were first introduced into law firms, lawyers have been looking for ways of using them to make generating documents faster and easier. Word processors helped, and precedent data banks did too, but the Holy Grail in this field is a system that can generate a complete, airtight first draft of the required legal document at the click of a mouse. The idea of software that can generate standardized legal documents is not new. Software packages that produce documents on the basis of certain specified inputs have been on the market for some time. They range from simple electronic forms or automated cut-and-paste to sophisticated software that can draw on internal definitions and even do a measure of logic checking.[1] Most law firms nowadays have in place systems of varying degrees of sophistication to avoid re-inventing the wheel each time a legal document is needed.

The Semantic Web promises to take the evolution of document generation further—much further. Advanced functionality such as checking the internal consistency of a document, or checking for compliance with a specified body of rules can be achieved by a non-semantic application built for that purpose. But where semantic applications will really break ahead of the pack is in their ability to draw on a web of structured online legal data and in their interoperability. Being able to access pre-existing taxonomies and rules will facilitate the task of developers, as much of the “logic” an application needs to process will already have been formalized and tested by a broad, collaborative community.

Furthermore, because the task of developing those taxonomies and applying them to data is an ongoing process, less effort will be needed by individual developers to keep applications up-to-date. Suppose a semantic application checks for consistency of the document with a certain body of rules. If a relevant statute is amended, or a court decision clarifies the interpretation of a given rule, there is no need for developers to update the code of the application to implement the amendments. Whatever authoritative online source of legal rules the application draws on can be updated, and all applications drawing on that source will stay abreast of the latest law, without needing to download an update. Another advantage of using smart data is that generating documents would involve more than just producing a human-readable document. The end product would not be a simple text file. Rather, as we have seen, the document could include metadata encoded in accordance with open, machine-readable standards, referencing online taxonomies and rules that give meaning to the data. This means that any other application, whether proprietary or otherwise, which uses those open standards, will be able to process that metadata, and understand the structure and content of the document. The Semantic Web guarantees interoperability by default, and avoids the problem of “smart” documents that are only smart to users who own a particular proprietary application.

Executable semantic contracts

If the content of the contract is machine-readable, parts of it may also be machine-executable: if applications can determine the rights and obligations of the parties to such a “semantic contract,” there is no reason why they could not also process payments, notify the parties when notice of renewal is due, renew the contract on specified conditions, etc. In addition to the efficiencies gained in generating the contracts on the lawyer’s side, semantic documents could yield huge gains on the client side. Rather than manually going through each agreement to determine who owes what to whom, when, and on what conditions, semantic contracts could be fed into software that will do this processing automatically.[2] With this technology, therefore, the law firm gets to cut the costs of production (and therefore, eventually, the cost of the service), while the client gets an enhanced product that enables it to cut its costs. Expect demand for semantic contracts and the applications that generate them.

Plain English vs. metadata

As we have seen, there are limits to the extent to which the plain-English meaning of legal propositions can be translated into formal rules. However, the considerations relating to these limitations are somewhat different in the case of contracts, because of their nature as private legislation between the parties. Here, rather than translating pre-existing laws, the parties are free to choose to draft their agreements using formalized terms and rules that lend themselves to automated analysis and processing. This raises the question of the relationship between the plain-English meaning of the contract (along with the plain-English laws that govern it) and the possibly divergent machine-readable meaning encoded in the metadata. Conceptually, a contract is an agreement between the parties, and the written contract is simply a memorandum or record of that agreement. The rules of contractual interpretation are concerned with ascertaining what rights and obligations the parties have consented to undertake. If I consent to be bound by a semantic contract, am I consenting to be bound by the plain-English terms only, or would the metadata, and the taxonomies the metadata refers to, also guide the interpretation of the agreement?

To put it another way, if I enter into a semantic contract, and the execution of the machine-executable parts of that contract is not what I expected on the basis of the plain English-wording of the contract, has the contract been breached? Suppose that there is no problem with the application that does the executing, but rather that the divergence is caused by differences between the logical implications of the semantic concepts used in the metadata on the one hand, and the positive laws as understood by lawyers and applied by judges on the other. The conservative answer is that the execution and the metadata that enables it are entirely distinct from the contract itself, and machine-execution is ultimately no different from a human agent performing the contract, properly or improperly. But the contrary viewpoint is that what semantic metadata does is to incorporate meaning by reference to definitions and rules external to the data itself. Is that so different from incorporation by reference in contract law, for example by referring to terms and conditions on the back of a parking ticket, or including Incoterms in international trade contracts? Why should the metadata not influence our interpretation of the contract?

Meaning vs. meaning

There are deeper questions at issue here, relating to the fundamental differences between machine-executable computer code and legal norms. The kind of “meaning” encoded using Semantic Web standards is deeply different from the kind of “meaning” you and I express when speaking about the law, or the kind expressed by law-makers in creating the law. I will leave these difficult questions hanging for now, but I will hazard to predict that, as machine-executable contracts gain currency and the idea of automated determination and processing of legal obligations becomes commonplace, those fundamental differences between code and law will begin to blur.

[2] See Siegel, p. 190.

What can you do with the Semantic Web that you can’t do without it?

The Semantic Web is a powerful way of structuring data and giving it a precise, machine-readable meaning. The most obvious and immediate benefit of semantic technologies is in organizing large quantities of information in a particular domain to make it easier to retrieve and analyze. This is reflected in the contexts in which these technologies have already been deployed, such as organizing large online databases of content (e.g. bbc.co.uk, see here); or facilitating the exchange and analysis of research data (e.g. drug research, see here). Given the problem of legal information expansion discussed in the first post in this series, using semantic taxonomies and rules to organize the vast universe of legal data is clearly a promising area.[1]

In this post I will go beyond merely identifying the benefits of better structured data. Rather, I want to consider what really distinguishes the Semantic Web from rival technologies by asking: what can you do with the Semantic Web that you can’t do without it? In attempting to answer this question, I will focus on two kinds of application of the Semantic Web which promise to deliver not just enhanced performance, but may even transform the nature of the legal service involved: semantic legal query systems and, in the next part, smart legal documents.

Lawyers as optimum retrieval intermediaries

One of the core tasks performed by lawyers is giving legal advice. Schematically, what lawyers do in carrying out this task is to:

1. identify rules in a vast corpus of laws that are relevant to a given legal query;
2. interpret their legal meaning, often by considering how different rules interact and how they have been interpreted in the past; and
3. consider how those rules apply to the specific query.

What distinguishes lawyers from the man on the street and what justifies both their holding a license to practice and their charging sizable fees for their services, is their (theoretically) superior ability to carry out each of these tasks. To quote the oft-repeated wisdom, the difference between a lawyer and a layman is not that the lawyer knows the law, but that he knows where to find it. I might add that the lawyer also knows whether there are legal rules for a given problem; how different rules interact (which rules preempt or modify other rules); how to check if a law is still in force or a precedent still good law; how to find an authoritative scholarly interpretation; and perhaps most importantly, the lawyer will have a wide experience through of different factual situations and contexts. In this sense, in the delivery of legal advice, a lawyer acts as an intermediary who ensures optimal retrieval of legal knowledge on behalf of his client.[2]

Semantic legal queries

We have seen how lawyers use search engines and commercial databases to deal with step 1 (identify) much more efficiently than was possible in the days of hard-copy statutes and law reports. However, even though researchers started working on expert legal systems as far back the 1970s (see here), in practice, steps 2 (interpretation) and 3 (applying the law to the query) are still largely carried out by the lawyer. This process is aided by technology only to the extent that the identification step 1 is repeated in sourcing secondary materials to guide interpretation and application of the rules. The smarter data generated on the Semantic Web will enable applications to dig deeper into steps 2 and 3.

Leveraging the higher degree of organization of legal data and the possibility of drawing inferences from the data, a semantic legal query system should be able to do more than merely retrieve information based on keywords selected by a human agent. In a world of perfect formalization, an application could carry out the interpretation and the application steps autonomously. But even in the absence of perfection, it is not unrealistic to suggest that within a few years, if enough smart legal data is available on the web, semantic legal query systems will be able to retrieve not just keyword-relevant documents but all or most of the information necessary to carry out steps 2 and 3. The application will know where to find the law (online); it will analyze the structure of the query and scour available data to determine whether there are applicable rules; it will determine what those rules are and suggest how they interact (perhaps retrieving the rules that govern the interaction); it will check whether the rules are up-to-date and retrieve any amendments or qualifications; and it will search for similar fact patterns, precedents and FAQ entries to clarify the application of the rules.

There are at least two major reasons semantic solutions have more potential than rival technologies to achieve these kinds of results. The first relates to the formal structure of Semantic Web standards: because the use of semantic metadata ensures that items of data have a precise meaning, semantic applications can make reliable inferences on the basis of the data. You need certainty to make inferences, because each step amplifies the uncertainty. Take this syllogism: Oracle is a Delaware Corporation; all Delaware Corporations are legal persons; therefore Oracle is a legal person. Now imagine each proposition in the syllogism is the result of a “best guess” data analysis process (e.g. through statistical analysis): There is a 90% percent chance that Oracle is a Delaware Corporation; there is a 90% chance that all Delaware Corporations are legal persons; therefore there is a 81% (90% of 90%) chance that Oracle is a legal person. This uncertainty compounds with each step, so beyond a few steps, any non-marginal uncertainty is fatal.

With the Semantic Web, if your query specifies a defined entity, the application will know precisely what you are referring to. In principle all instances of that object on the Semantic Web will refer to the same (online) definition, which specifies its properties and its relation to other entities. The second reason for the superiority of semantic applications relates to the openness of Semantic Web standards: the widespread adoption of standards for tagging and organizing legal data will ensure that more structured legal information is available than could possibly be achieved by a single provider of proprietary systems.

DIY and FAQs

An application that can deliver a page full of the kind of information described above will go a long way in assisting lawyers in carrying out steps 2 and 3 of legal advice delivery. In fact, if the application is good enough, it may even make the lawyer’s input redundant. How much additional specialist knowledge do you really need if all of the relevant information is right before you? Many consumers of legal services are happy to resort to “DIY” legal advice rather than incurring the costs of professional legal services. Online FAQs and other legal resources have proven popular as means of sourcing legal information without consulting a lawyer directly (often made available by legal professionals as a kind of loss leader to attract potential clients). Individual resources are inevitably limited in content, but in the aggregate the free World Wide Web (i.e. excluding subscription websites) is a fairly comprehensive source of legal information. The problem for the untrained is in finding relevant information and distinguishing the accurate and up-to-date sources from the incorrect and out-of-date. A semantic legal query application that enables laymen to access comprehensive, up-to-date legal information in response to their queries would satisfy much of the demand for simpler legal advice, reducing the demand for competing professional advice—if priced right. Even though these applications may not rival good lawyers in the quality of the service, not all consumers of legal services are concerned with getting the best quality. Good-enough might well do.

More than machines

Of course, many, if not all, lawyers would strongly resist being described as “optimum information retrieval” machines. Most would see their role as going well beyond merely delivering statements of what the law is to their clients. Rather, they are in the business of delivering solutions, offering advice on how to deal with certain situations, how to handle particular disputes, how to structure transactions, etc. Yet it is undeniable that lawyers, especially junior lawyers, spend much of their time searching for relevant information and assimilating it into bespoke legal advice. What the technological possibilities outlined in this post suggest is that simpler legal advice can likely be significantly automated, while for more complex queries, Semantic Web-based applications could considerably enhance fee-earner productivity in producing legal advice.

(Coming soon: Part 5 – Legal Documents.)

[2] See discussion of “optimum retrieval” in Part 1 of this series.

A machine-readable version of the law?

David Siegel, an entrepreneur and early blogger, recently published a book entitled Pull, The Power of the Semantic Web to Transform Your Business, the first “business” book about the Semantic Web. Siegel devotes one chapter to exploring the possible impact of the Semantic Web on the law and lawyers. An enthusiastic backer of the new technology, Siegel sees huge potential for the Semantic Web to transform the work of lawyers. He believes that work on legal taxonomies and formalized rules may result in “a set of semantic rules that can then serve as the machine-readable version of the law.”[1] This is the kind of structured legal data that would make the intelligent legal queries outlined above possible. It raises the question of the future utility of lawyers in a world where much of what they now do can be performed by computer applications. Why go to a lawyer if you can get an authoritative, complete and up-to-date statement of the law online? If the law can be fully specified as a formalized set of machine-readable rules, would we even need lawyers and judges, or could they be replaced with computers and Semantic engineers of the law?

A note of caution

I ought to sound a note of caution at this point. The idea of reformulating all of the rules of law as a formal system, with precise classifications of entities and rules governing their interactions, has been tried before. Most students of the law will, at some point in their studies, come across discussions of the German Civil Code (the BGB), which was drafted over a century ago with precisely that aim in mind. It failed. The law has proven too malleable, too changeable, and too subjective a system to codify with mathematical rigor. There is little reason to believe that the Semantic Web will succeed where others have failed, at least in the foreseeable future. Few fields of human activity are as centrally focused on interpretation of often conflicting texts, and as acutely concerned with the ambiguity of human language, as the law. Though the law may be a body of rules, those rules are not of the clear-cut variety that easily lend themselves to formalization.

How smart does “smart” need to be?

That does not mean, however, that the taxonomies and rules of the Semantic Web are useless when it comes to the law. Difficult exercises of interpretation may be required in deciding “hard cases” and creative thinking may be needed in handling more complex, high-level legal issues, but much of the daily practice of the law is far less complex or ambiguous. Is a high level of legal expertise really required in producing a first draft of simple terms and conditions or a memo setting out routine advice? The parameters of these kinds of tasks should be relatively easy to formalize. And even if the semantic formalization of the law were less than perfect, a system that understands the structure of legal queries and can achieve near-optimum retrieval could vastly increase the efficiency of legal researchers. [Unknown A1]

Again, taxonomies and rules need not be all-encompassing to be useful. The Semantic Web is not the latest incarnation of pie-in-the-sky artificial intelligence. At the heart of the SemanticWeb is the task of developing dictionaries of concepts and rules to make data smarter, and that is a task that can be done piecemeal. Making data smarter does not have to mean encoding all of the subtleties of human language into the data. If an area of legal practice is concerned with a reasonably small set of clearly defined rules, much of the relevant law may be susceptible to being translated into machine-readable standards. Consider an area of regulatory compliance such as food labeling, which involves rules prescribing particular information formats and content, lists of words that must, can, or cannot be used under certain conditions, and similarly well-defined rules. Translating most of these into a “machine readable version of the law” that could serve as the basis for automated compliance-checking systems hardly seems unrealistic. What about other, less straightforward areas of the law? Even where the area evades complete formalization, as will often be the case, semantic applications may significantly enhance the productivity of fee-earners by dealing with routine, low-skill work while leaving the subtler points of law to the flesh-and-blood professional. So, what kinds of application might achieve these efficiency gains?

(Coming soon: Part 4 – Smart documents and semantic contracts)

What is the Semantic Web?

The Semantic Web is a way of making data smart. The idea is, rather than building smart applications that can analyze “dumb” data, you make the data smart in the first place. The problem with dumb data is that the ability of applications to make sense of human language is limited. Currently, the information in most web pages and text documents is “human language,” encoded in data formats that tell computers nothing about their meaning. What the standards that make up the core of the Semantic Web do is to provide data formats that can be used to make the meaning of information explicit.

Dumb data vs. smart data

So how is this done? What differentiates smart data from dumb data? If you view the source code of this web page (try it – it’s in View > Source in Explorer; View > Page Source in Firefox, View > View Source in Safari), you will see some text and a lot of “tags” between angled brackets, such as “<p>” and “<div id=‘header’>.” This is HTML, the mark-up language in which most information currently on the World Wide Web is encoded. It tells your browser how to display the text and images, and where to redirect when you click on a link – but not much else. Information encoded in plain HTML is dumb data. Let’s consider an example. In HTML, you might have the following text:

<p>Sun is a subsidiary of Oracle.</p>

The HTML tells your browser that text enclosed between the opening tag “<p>” and the closing tag “</p>” should be displayed as a single paragraph, and nothing more. A simple search engine might hit on this sentence even if I intended to search for the “sun,” as in the sun in the sky, or an “oracle,” as in the Oracle of Delphi. An application with advanced language-processing abilities might be able to deduce from the absence of an article (“a” or “the”) that “Sun” and “Oracle” are names. It might also deduce from the mention of “subsidiary” that the sentence in fact refers to names of corporations. In the current state of technology, this is likely to be a hit-and-miss process.

Making data smart

The idea behind the Semantic Web is to attach machine-readable metadata (data about data) to information that can be interpreted by any Semantic Web application. To better understand what this involves, imagine a mark-up language that enables you to specify what the things being referred to are. Imagine that this mark-up language enabled you to add tags to your data to specify things like:

<item this is a corporation> Sun </item>

<item this is a legal relationship between two corporations> is a subsidiary of </item>

<item this is a corporation> Oracle </item>

Even better, imagine that, rather than just labeling things, you could refer to a source of information on the web that tells you more about each of these things, e.g.:

<item see http://www.dbpedia.org/resource/Oracle_Corporation> Oracle </item>

The link referred to is a “resource” – a bundle of data available online that describes something. This resource contains data, encoded in a machine-readable format, which might state that Oracle is a Delaware corporation, that it is headquartered in Redwood City, California, that the current CEO is Larry Ellison, etc.

Now let’s take this one step further, and imagine that, when that “Oracle” resource states that Oracle is a “Delaware corporation,” it in turn refers to an online resource that defines the term “Delaware corporation.” That definition might specify that a Delaware corporation is a kind of legal person, that it should have a certificate of incorporation, bylaws, a board of directors, etc. Of course, these statements would also be machine-readable, and could in turn refer to other resources (defining “legal person,” “certificate of incorporation,” “board of directors,” etc.).

Classifications and rules

Where does it all end? It ends with “thing.” That is, a “corporation” is a “legal person,” which is a kind of “person,” which is a kind of “thing.” A “certificate of incorporation” is a “legal document,” which is a kind of “document,” which is a kind of “thing.” Everything is a thing, and so every “resource” is a kind of thing, which fits into a classification of things (a taxonomy). One of the most important aspects of the Semantic Webs is defining taxonomies of different kinds of things using machine-readable formats. There is no need for a single, all-encompassing taxonomy which defines every possible thing: partial taxonomies can define a few terms by referring to other taxonomies, and all of these interlinked taxonomies ultimately refer to the most general standards (remember, this can be done because they are all online).

The Semantic Web also goes beyond mere classifications, allowing you to specify rules for each kind of thing. For example, you could specify that a “director” of a “Delaware corporation” can be a natural person, but cannot be a legal person. You could specify that the property (predicate) of “having a subsidiary” must have a corporation as its subject and another, different corporation as its object.

The foregoing does not purport to be a technical exposition of the Semantic Web, but I hope you get the idea. The core of the Semantic Web is a set of precisely defined standards that can be used to make data smarter by making explicit the underlying structure of the information.[1] Online classifications and rules enable applications to identify and analyze the data in much greater depth and with much greater precision than existing alternative technologies.

The state of the technology

Not all of the pieces of the system outlined above are in place. The basic standards of the Semantic Web, including the Resource Description Framework (RDF) and the Web Ontology Language (OWL), are by now reasonably mature and stable standards. However, there is still a good deal of work to be done and problems to be ironed out before the vision of the Semantic Web is fully made a reality (see here and here). Nevertheless, an increasing number of big names have been adopting Semantic Web standards to structure their data (New York Times, recovery.gov, BBC, Thomson Reuters). Identifying the real-world future implications of the Semantic Web is no longer science fiction, even for the legal industry.

(Next up: Part 3 – A Machine Readable Version of the Law?)

The problem: too much data

There are currently over 25 billion web pages on the World Wide Web. In fact, that figure covers only the indexable web, so those 25 billion pages may be only the tip of the iceberg (see this paper on the “deep web”). Looking beyond the web to total production of information, a study by International Data Corp carried out in 2008 predicts that 1,200 exabytes of data will be generated in 2010 (cited by The Economist here). To put this in perspective, note that one byte of information is a sequence of eight bits – a sequence of eight digits which can be either one or zero. One exabyte is 1,000,000,000,000,000,000 bytes (10¹⁸), or one billion gigabytes. The text of this blog post, in plain text format, takes up about 13,000 bytes. The challenge of identifying and retrieving relevant data in this ever-expanding universe of information is growing in step with the volume of the information itself. Achieving what Richard Susskind calls “information satisfaction” – getting the information you want, and only the information you want – in the face of this exponential expansion is an increasingly daunting task. This is even more true of the challenge of achieving “optimum retrieval” – for a given query, being confident that the single best document has been returned. Google’s “I’m feeling lucky” option may sometimes be surprisingly accurate, but not with any reliable degree of certainty.

Too much legal data

The problem of too much data will be familiar to law students, associates, and anyone else who has carried out legal research. The volume of legislation, case law, commentary on the law, and the like is no exception to the current phenomenon of information expansion. “Googling it” can provide a good first stab at some legal problems, but no lawyer who fears malpractice suits would rely exclusively on results from a general search engine. Commercial legal databases provide more structured and authoritative databanks of legal information, but they are expensive, difficult to use for the untrained, and the search is still conducted mostly by means of citations and keywords. Whether legal sources are identified by a search engine or using a commercial database, the actual task of analyzing and interpreting the texts is conducted by the lawyer – not the machine.

If I want to ascertain, say, what information I must provide in the certificate of incorporation of a Delaware Corporation, I can search “Delaware corporation law,” click through the link that looks most relevant, scan the text (perhaps with the help of the “find” function), identify the relevant section, and read through it to draw up a list of the requirements. If I am especially diligent, I might also check case law in a commercial database to see if judicial decisions have added to or qualified these requirements. Now imagine that, instead of proceeding by keyword searches and “manual” analysis, I could simply enter the query “What information must be provided in the certificate of incorporation of a Delaware Corporation?” and the search engine returned a complete, authoritative list of all of the requirements, along with any qualifications or additions made by the case law. That, in a nutshell, is the promise of the Semantic Web.

(Next up: Part 2 – What is the Semantic Web?)