Introduction

Explicit articulation of uncertainty in science, especially in science involved in public policy, improves science, because it clarifies what future research is necessary, helps policy makers to evaluate scientific reports, and reminds the public about the limitations of current scientific knowledge.

We have built two complementary ontologies to measure the scientific uncertainty expressed in food safety documents. We have enlisted Machine Learning to help with three tasks:

• To assist with coding complex documents such as food safety risk assessments
• To assess the quality of the ontologies we devised (uncertainty and judgment ontologies)
• To get insights into the causal processes of making uncertainty explicit. 

Approach

One straightforward strategy for the construction of an automated coding system for these documents is to use supervised learning techniques. Supervised learning aims at finding rules starting from a set of training instances. In our case, the training instances are the sentences (or small set of sentences) and their associated labels given by human coders. The goal is to extract rules that would allow a system to automatically label new sentences (or set of sentences) drawn from documents similar to the one used by the human coders. For instance, the system should be able to code a text input as ‘coded’ vs. ‘non coded’ and, if ‘coded’, as one of the categories present in the ontologies. Furthermore, if the learning system is trained from instances drawn from different contexts, differences in the learned rules could be enlightening about differences between these contexts. For instance, it could appear that the rules learned from American documents differ somewhat from the rules learned with documents from the European Union. Or that the rules change over time.

In the estimation, we used naïve Bayesian classifier, Support Vector Machines (SVM),kNearest Neighbor and Decision Tree algorithms. The best overall learners were Naïve Bayes and SVM.

Results

We found that Machine Learning can do surprisingly well identifying complex meanings and probably can be helpful making suggestions to human coders. Our ML experiments show that our ontologies do enable a fairly consistent practice of coding.

Evaluating our ontology of judgment, we learned that elements of judgment are often communicated in relationship to one another. In our future work, we will try to exploit these relationships to identify judgment variables. Assessing our ontology of uncertainty, we found out that the deductive decision making process, that aids human coders, and that is reflected in the hierarchical structure of our ontology, makes the first large cuts fairly well and confusions tend to emerge between variables that our system defines as being closer.

And finally, we found some evidence that institutional factors do influence how predictable our ontology of uncertainty is.