{"id":3512,"date":"2016-11-21T06:14:06","date_gmt":"2016-11-21T11:14:06","guid":{"rendered":"http:\/\/fgiasson.com\/blog\/?p=3512"},"modified":"2016-11-18T08:15:32","modified_gmt":"2016-11-18T13:15:32","slug":"leveraging-kbpedia-aspects-to-generate-training-sets-automatically","status":"publish","type":"post","link":"https:\/\/fgiasson.com\/blog\/index.php\/2016\/11\/21\/leveraging-kbpedia-aspects-to-generate-training-sets-automatically\/","title":{"rendered":"Leveraging KBpedia Aspects To Generate Training Sets Automatically"},"content":{"rendered":"

In previous articles I have covered multiple ways to create training corpuses for unsupervised learning and positive and negative training sets for supervised learning 1<\/a><\/sup> , <\/sup>2<\/a><\/sup> , <\/sup>3<\/a><\/sup> using Cognonto<\/a> and KBpedia. Different structures inherent to a knowledge graph like KBpedia can lead to quite different corpuses and sets. Each of these corpuses or sets may yield different predictive powers depending on the task at hand.<\/p>\n

So far we have covered two ways to leverage the KBpedia Knowledge Graph to automatically create positive and negative training corpuses:<\/p>\n

    \n
  1. Using the links that exist between each KBpedia reference concept and their related Wikipedia pages<\/li>\n
  2. Using the linkages between KBpedia reference concepts and external vocabularies to create training corpuses out of
    \nnamed entities.<\/li>\n<\/ol>\n

    Now we will introduce a third way to create a different kind of training corpus:<\/p>\n

      \n
    1. Using the KBpedia aspects linkages.<\/li>\n<\/ol>\n

      Aspects<\/code> are aggregations of entities that are grouped according to their characteristics different from their direct types. Aspects help to group related entities by situation, and not by identity nor definition. It is another way to organize the knowledge graph and to leverage it. KBpedia has about 80 aspects<\/a> that provide this secondary means for placing entities into related real-world contexts. Not all aspects relate to a given entity.<\/p>\n

      [extoc]<\/p>\n

      <\/p>\n

      Creating New Domain Using KBpedia Aspects<\/h3>\n

      To continue with the musical domain, there exists two aspects of interest:<\/p>\n

        \n
      1. Music<\/li>\n
      2. Genres<\/li>\n<\/ol>\n

        What we will do first is to query the KBpedia Knowledge Graph using theSPARQL query language<\/a> to get the list of all of the KBpedia reference concepts that are related to the Music<\/code> or the Genre<\/code> aspects. Then, for each of these reference concepts, we will count the number of named entities that can be reached in the complete KBpedia structure.<\/p>\n

        prefix kko: <http:\/\/kbpedia.org\/ontologies\/kko#>\nprefix rdfs: <http:\/\/www.w3.org\/2000\/01\/rdf-schema>\nprefix rdf: <http:\/\/www.w3.org\/1999\/02\/22-rdf-syntax-ns#>\nprefix dcterms: <http:\/\/purl.org\/dc\/terms\/> \nprefix schema: <http:\/\/schema.org\/>\n\nselect distinct ?class count(distinct ?entity) as ?nb\nfrom <http:\/\/dbpedia.org>\nfrom <http:\/\/www.uspto.gov>\nfrom <http:\/\/wikidata.org>\nfrom <http:\/\/kbpedia.org\/1.10\/>\nwhere\n{\n  ?entity dcterms:subject ?category .\n\n  graph <http:\/\/kbpedia.org\/1.10\/>\n  {\n    {?category <http:\/\/kbpedia.org\/ontologies\/kko#hasMusicAspect> ?class .}\n    union\n    {?category <http:\/\/kbpedia.org\/ontologies\/kko#hasGenre> ?class .}\n  }\n}\norder by desc(?nb)\n<\/pre>\n\n\n\n<\/colgroup>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
         
        reference concept<\/th>\nnb<\/th>\n<\/tr>\n<\/thead>\n
        http:\/\/kbpedia.org\/kko\/rc\/Album-CW<\/td>\n128772<\/td>\n<\/tr>\n
        http:\/\/kbpedia.org\/kko\/rc\/Song-CW<\/td>\n74886<\/td>\n<\/tr>\n
        http:\/\/kbpedia.org\/kko\/rc\/Music<\/td>\n51006<\/td>\n<\/tr>\n
        http:\/\/kbpedia.org\/kko\/rc\/Single<\/td>\n50661<\/td>\n<\/tr>\n
        http:\/\/kbpedia.org\/kko\/rc\/RecordCompany<\/td>\n5695<\/td>\n<\/tr>\n
        http:\/\/kbpedia.org\/kko\/rc\/MusicalComposition<\/td>\n5272<\/td>\n<\/tr>\n
        http:\/\/kbpedia.org\/kko\/rc\/MovieSoundtrack<\/td>\n2919<\/td>\n<\/tr>\n
        http:\/\/kbpedia.org\/kko\/rc\/Lyric-WordsToSong<\/td>\n2374<\/td>\n<\/tr>\n
        http:\/\/kbpedia.org\/kko\/rc\/Band-MusicGroup<\/td>\n2185<\/td>\n<\/tr>\n
        http:\/\/kbpedia.org\/kko\/rc\/Quartet-MusicalPerformanceGroup<\/td>\n2078<\/td>\n<\/tr>\n
        http:\/\/kbpedia.org\/kko\/rc\/Ensemble<\/td>\n1438<\/td>\n<\/tr>\n
        http:\/\/kbpedia.org\/kko\/rc\/Orchestra<\/td>\n1380<\/td>\n<\/tr>\n
        http:\/\/kbpedia.org\/kko\/rc\/Quintet-MusicalPerformanceGroup<\/td>\n1335<\/td>\n<\/tr>\n
        http:\/\/kbpedia.org\/kko\/rc\/Choir<\/td>\n754<\/td>\n<\/tr>\n
        http:\/\/kbpedia.org\/kko\/rc\/Concerto<\/td>\n424<\/td>\n<\/tr>\n
        http:\/\/kbpedia.org\/kko\/rc\/Symphony<\/td>\n299<\/td>\n<\/tr>\n
        http:\/\/kbpedia.org\/kko\/rc\/Singing<\/td>\n154<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
        Seventeen KBpedia reference concepts are related to the two aspects we want to focus on. The next step is to take these 17 reference concepts and to create a new domain corpus with them. We will use the new version of KBpedia to create the full set of reference concepts that will scope our domain by inference.<\/p>\n
        Next we will try to use this information to create two totally different kinds of training corpuses:<\/p>\n
          \n
        1. One that will rely on the links between the reference concepts and Wikipedia pages<\/li>\n
        2. One that will rely on the linkages to external vocabularies to create a list of named entities that will be used as
          \nthe training corpus<\/li>\n<\/ol>\n
          \n
          Creating Model With Reference Concepts<\/h3>\n
          \n
          The first training corpus we want to test is one that uses the linkage between KBpedia reference concepts and Wikipedia pages. The first thing is to generate the domain training corpus with the 17<\/code> seed reference concepts and then to infer other related reference concepts.<\/p>\n
          \n
          (<\/span>use 'cognonto-esa.core<\/span>)<\/span>\n(<\/span>require '[<\/span>cognonto-owl.core<\/span> :as<\/span> owl]<\/span>)<\/span>\n(<\/span>require '[<\/span>cognonto-owl.reasoner<\/span> :as<\/span> reasoner]<\/span>)<\/span>\n\n\n(<\/span>def<\/span> kbpedia-manager<\/span> (<\/span>owl<\/span>\/<\/span>make-ontology-manager)<\/span>)<\/span>\n(<\/span>def<\/span> kbpedia<\/span> (<\/span>owl<\/span>\/<\/span>load-ontology \"resources\/kbpedia_reference_concepts_linkage.n3\"<\/span>\n                                :manager<\/span> kbpedia-manager)<\/span>)<\/span>\n(<\/span>def<\/span> kbpedia-reasoner<\/span> (<\/span>reasoner<\/span>\/<\/span>make-reasoner kbpedia)<\/span>)<\/span>\n\n(<\/span>define-domain-corpus<\/span> [<\/span>\"http:\/\/kbpedia.org\/kko\/rc\/Album-CW\"<\/span>\n                       \"http:\/\/kbpedia.org\/kko\/rc\/Song-CW\"<\/span>\n                       \"http:\/\/kbpedia.org\/kko\/rc\/Music\"<\/span>\n                       \"http:\/\/kbpedia.org\/kko\/rc\/Single\"<\/span>\n                       \"http:\/\/kbpedia.org\/kko\/rc\/RecordCompany\"<\/span>\n                       \"http:\/\/kbpedia.org\/kko\/rc\/MusicalComposition\"<\/span>\n                       \"http:\/\/kbpedia.org\/kko\/rc\/MovieSoundtrack\"<\/span>\n                       \"http:\/\/kbpedia.org\/kko\/rc\/Lyric-WordsToSong\"<\/span>\n                       \"http:\/\/kbpedia.org\/kko\/rc\/Band-MusicGroup\"<\/span>\n                       \"http:\/\/kbpedia.org\/kko\/rc\/Quartet-MusicalPerformanceGroup\"<\/span>\n                       \"http:\/\/kbpedia.org\/kko\/rc\/Ensemble\"<\/span>\n                       \"http:\/\/kbpedia.org\/kko\/rc\/Orchestra\"<\/span>\n                       \"http:\/\/kbpedia.org\/kko\/rc\/Quintet-MusicalPerformanceGroup\"<\/span>\n                       \"http:\/\/kbpedia.org\/kko\/rc\/Choir\"<\/span>\n                       \"http:\/\/kbpedia.org\/kko\/rc\/Symphony\"<\/span>\n                       \"http:\/\/kbpedia.org\/kko\/rc\/Singing\"<\/span>\n                       \"http:\/\/kbpedia.org\/kko\/rc\/Concerto\"<\/span>]<\/span>\n  kbpedia\n  \"resources\/aspects-concept-corpus-dictionary.csv\"<\/span>\n  :reasoner<\/span> kbpedia-reasoner)<\/span>\n\n(<\/span>create-pruned-pages-dictionary-csv \"resources\/aspects-concept-corpus-dictionary.csv\"<\/span>\n                                    \"resources\/aspects-concept-corpus-dictionary.pruned.csv\"<\/span> \n                                    \"resources\/aspects-corpus-normalized\/\"<\/span>)<\/span>\n<\/pre>\n<\/div>\n
          Once pruned, we end-up with a domain which has 108<\/code> reference concepts which will enable us to create models with 108 features. The next step is to create the actual semantic interpreter and the SVM models:<\/p>\n
          \n
          ;; <\/span>Load dictionaries<\/span>\n(<\/span>load-dictionaries \"resources\/general-corpus-dictionary.pruned.csv\"<\/span> \"resources\/aspects-concept-corpus-dictionary.pruned.csv\"<\/span>)<\/span>\n\n;; <\/span>Create the semantic interpreter<\/span>\n(<\/span>build-semantic-interpreter \"aspects-concept-pruned\"<\/span> \"resources\/semantic-interpreters\/aspects-concept-pruned\/\"<\/span> (<\/span>distinct (<\/span>concat (<\/span>get-domain-pages)<\/span> (<\/span>get-general-pages)<\/span>)<\/span>)<\/span>)<\/span>\n\n;; <\/span>Build the SVM model vectors<\/span>\n(<\/span>build-svm-model-vectors \"resources\/svm\/aspects-concept-pruned\/\"<\/span> :corpus-folder-normalized<\/span> \"resources\/aspects-corpus-normalized\/\"<\/span>)<\/span>\n\n;; <\/span>Train the linear SVM classifier<\/span>\n(<\/span>train-svm-model \"svm.aspects.concept.pruned\"<\/span> \"resources\/svm\/aspects-concept-pruned\/\"<\/span>\n                 :weights<\/span> nil<\/span>\n                 :v<\/span> nil<\/span>\n                 :c<\/span> 1\n                 :algorithm<\/span> :l2l2<\/span>)<\/span>\n<\/pre>\n<\/div>\n
          Then we have to evaluate this new model using the gold standard<\/a>:<\/p>\n
          \n
          (<\/span>evaluate-model \"svm.aspects.concept.pruned\"<\/span> \"resources\/gold-standard-full.csv\"<\/span>)<\/span>\n<\/pre>\n<\/div>\n
          True positive:  28\nFalse positive:  0\nTrue negative:  923\nFalse negative:  66\n\nPrecision:  1.0\nRecall:  0.29787233\nAccuracy:  0.93510324\nF1:  0.45901638\n<\/pre>\n
          Now let’s try to find better hyperparameters using grid search:<\/p>\n
          \n
          (<\/span>svm-grid-search \"grid-search-aspects-concept-pruned-tests\"<\/span> \n                       \"resources\/svm\/aspects-concept-pruned\/\"<\/span> \n                       \"resources\/gold-standard-full.csv\"<\/span>\n                       :selection-metric<\/span> :f1<\/span>\n                       :grid-parameters<\/span> [<\/span>{<\/span>:c<\/span> [<\/span>1 2 4 16 256]<\/span>\n                                          :e<\/span> [<\/span>0.001 0.01 0.1]<\/span>\n                                          :algorithm<\/span> [<\/span>:l2l2<\/span>]<\/span>\n                                          :weight<\/span> [<\/span>1 15 30]<\/span>}<\/span>]<\/span>)<\/span>\n<\/pre>\n<\/div>\n
          {:gold-standard \"resources\/gold-standard-full.csv\"\n :selection-metric :f1\n :score 0.84444445 \n :c 1\n :e 0.001 \n :algorithm :l2l2\n :weight 30}\n<\/pre>\n
          After running the grid search with these initial broad range values, we found a configuration that gives us 0.8444<\/code> for the F1<\/code> score. So far, this score is the best to date we have gotten for the full gold standard2<\/a><\/sup>, <\/sup>3<\/a><\/sup>. Let’s see all of the metrics for this configuration:<\/p>\n
          \n
          (<\/span>train-svm-model \"svm.aspects.concept.pruned\"<\/span> \"resources\/svm\/aspects-concept-pruned\/\"<\/span>\n                 :weights<\/span> {<\/span>1 30.0}<\/span>\n                 :v<\/span> nil<\/span>\n                 :c<\/span> 1 \n                 :e<\/span> 0.001\n                 :algorithm<\/span> :l2l2<\/span>)<\/span>\n\n(<\/span>evaluate-model \"svm.aspects.concept.pruned\"<\/span> \"resources\/gold-standard-full.csv\"<\/span>)<\/span>\n<\/pre>\n<\/div>\n
          True positive:  76\nFalse positive:  10\nTrue negative:  913\nFalse negative:  18\n\nPrecision:  0.88372093\nRecall:  0.80851066\nAccuracy:  0.972468\nF1:  0.84444445\n<\/pre>\n
          These results are also the best balance between precision<\/code> and recall<\/code> that we have gotten so far2<\/a><\/sup>, <\/sup>3<\/a><\/sup>. Better precision<\/code> can be obtained if necessary but only at the expense of lower recall<\/code>.<\/p>\n
          Let’s take a look at the improvements we got compared to the previous training corpuses we had:<\/p>\n