{"id":3455,"date":"2016-11-17T06:00:43","date_gmt":"2016-11-17T11:00:43","guid":{"rendered":"http:\/\/fgiasson.com\/blog\/?p=3455"},"modified":"2016-11-17T13:38:09","modified_gmt":"2016-11-17T18:38:09","slug":"dynamic-machine-learning-using-the-kbpedia-knowledge-graph-part-1","status":"publish","type":"post","link":"https:\/\/fgiasson.com\/blog\/index.php\/2016\/11\/17\/dynamic-machine-learning-using-the-kbpedia-knowledge-graph-part-1\/","title":{"rendered":"Dynamic Machine Learning Using the KBpedia Knowledge Graph – Part 1"},"content":{"rendered":"

In my previous blog post, Create a Domain Text Classifier Using Cognonto<\/a>, I explained how one can use the KBpedia Knowledge Graph to automatically create positive and negative training corpuses for different machine learning tasks. I explained how SVM classifiers could be trained and used to check if an input text belongs to the defined domain or not.<\/p>\n

This article is the first of two articles.In first part I will extend on this idea to explain how the KBpedia Knowledge Graph can be used, along with other machine learning techniques, to cope with different situations and use cases. I will cover the concepts of feature selection, hyperparameter optimization, and ensemble learning (in part 2 of this series). The emphasis here is on the testing and refining of machine learners, versus the set up and configuration times that dominate other approaches.<\/p>\n

Depending on the domain of interest, and depending on the required precision<\/code> or recall<\/code>, different strategies and techniques can lead to better predictions. More often than not, multiple different training corpuses, learners and hyperparameters need to be tested before ending up with the initial best possible prediction model. This is why I will strongly emphasize the fact that the KBpedia Knowledge Graph and Cognonto<\/a> can be used to automate fully the creation of a wide range of different training corpuses, to create models, to optimize their hyperparameters, and to evaluate those models.<\/p>\n

[extoc]<\/p>\n

<\/p>\n

\n

New Knowledge Graph and Reasoning<\/h2>\n
\n

For this article, I will use the latest version of the KBpedia Knowledge Graph version 1.10<\/code> that we just released<\/a>. A knowledge graph such as KBpedia is not static. It constantly evolves, gets fixed, and improves. New concepts are created, deprecated concepts are removed, new linkage to external data sources are created, etc. This growth means that any of these changes can have a [positive] impact on the creation of the positive and negative training sets. Applications based on KBpedia should be tested against any new knowledge graph that is released to see if its models will improve. Better concepts, better structure, and more linkages will often lead to better training sets as well.<\/p>\n

Such growth in KBpedia is also why automating, and more importantly testing, this process is crucial. Upon the release of major new versions we are able to automate all of these steps to see the final impacts of upgrading the knowledge graph:<\/p>\n

    \n
  1. Aggregate all the reference concepts that scope the specified domain (by inference)<\/li>\n
  2. Create the positive and negative training corpuses<\/li>\n
  3. Prune the training corpuses<\/li>\n
  4. Configure the classifier (in this case, create the semantic vectors for ESA)<\/li>\n
  5. Train the model (in this case, the SVM model)<\/li>\n
  6. Optimize the hyperparameters of the algorithm (in this case, the linear SVM hyperparameters), and<\/li>\n
  7. Evaluate the model on multiple gold standards.<\/li>\n<\/ol>\n

    Because each of these steps belongs to an automated workflow, we can easily check the impact of updating the KBpedia Knowledge Graph on our models.<\/p>\n<\/div>\n<\/div>\n

    \n

    Reasoning Over The Knowledge Graph<\/h2>\n
    \n

    A new step I am adding in this current use case is to use a reasoner to reason over the KBpedia Knowledge Graph. The reasoner is used when we define the scope of the domain to classify. We will browse the knowledge graph to see which seed<\/i> reference concepts we should add to the scope. Then we will use a reasoner to extend the models to include any new sub-classes relevant to the scope of the domain. This means that we may add further specific features to the final model.<\/p>\n<\/div>\n<\/div>\n

    \n

    Update Domain Training Corpus Using KBpedia 1.10 and a Reasoner<\/h2>\n
    \n

    Recall our prior use case<\/a> used Music<\/i> as its domain scope. The first step is to use the new KBpedia version 1.10<\/code> along with a reasoner to create the full scope of this updated Music domain.<\/p>\n

    The result of using this new version and a reasoner is that we now end up with 196<\/code> features (reference documents) instead of 64<\/code>. This also means that we will have 196 documents in our positive training set if we only use the Wikipedia pages linked to these reference concepts (and not their related named entities).<\/p>\n

    \n
    (<\/span>use 'cognonto-esa.core<\/span>)<\/span>\n(<\/span>require '[<\/span>cognonto-owl.core<\/span> :as<\/span> owl]<\/span>)<\/span>\n(<\/span>require '[<\/span>cognonto-owl.reasoner<\/span> :as<\/span> reasoner]<\/span>)<\/span>\n\n(<\/span>def<\/span> kbpedia-manager<\/span> (<\/span>owl<\/span>\/<\/span>make-ontology-manager)<\/span>)<\/span>\n(<\/span>def<\/span> kbpedia<\/span> (<\/span>owl<\/span>\/<\/span>load-ontology \"resources\/kbpedia_reference_concepts_linkage.n3\"<\/span>\n                                :manager<\/span> kbpedia-manager)<\/span>)<\/span>\n(<\/span>def<\/span> kbpedia-reasoner<\/span> (<\/span>reasoner<\/span>\/<\/span>make-reasoner kbpedia)<\/span>)<\/span>\n\n(<\/span>define-domain-corpus<\/span> [<\/span>\"http:\/\/kbpedia.org\/kko\/rc\/Music\"<\/span>\n                       \"http:\/\/kbpedia.org\/kko\/rc\/Musician\"<\/span>\n                       \"http:\/\/kbpedia.org\/kko\/rc\/MusicPerformanceOrganization\"<\/span>\n                       \"http:\/\/kbpedia.org\/kko\/rc\/MusicalInstrument\"<\/span>\n                       \"http:\/\/kbpedia.org\/kko\/rc\/Album-CW\"<\/span>\n                       \"http:\/\/kbpedia.org\/kko\/rc\/Album-IBO\"<\/span>\n                       \"http:\/\/kbpedia.org\/kko\/rc\/MusicalComposition\"<\/span>\n                       \"http:\/\/kbpedia.org\/kko\/rc\/MusicalText\"<\/span>\n                       \"http:\/\/kbpedia.org\/kko\/rc\/PropositionalConceptualWork-MusicalGenre\"<\/span>\n                       \"http:\/\/kbpedia.org\/kko\/rc\/MusicalPerformer\"<\/span>]<\/span>\n  kbpedia\n  \"resources\/domain-corpus-dictionary.csv\"<\/span>\n  :reasoner<\/span> kbpedia-reasoner)<\/span>\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n
    \n
    Create Training Corpuses<\/h2>\n
    \n
    The next step is to create the actual training corpuses: the general and domain ones. We have to load the dictionaries we created in the previous step, and then to locally cache and normalize the corpuses. Remember that the normalization steps are:<\/p>\n
      \n
    1. Defluff the raw HTML page. We convert the HTML into text, and we only keep the body of the page<\/li>\n
    2. Normalize the text with the following rules:\n
        \n
      1. remove diacritics characters<\/li>\n
      2. remove everything between brackets like: [edit] [show]<\/li>\n
      3. remove punctuation<\/li>\n
      4. remove all numbers<\/li>\n
      5. remove all invisible control characters<\/li>\n
      6. remove all [math] symbols<\/li>\n
      7. remove all words with 2 characters or fewer<\/li>\n
      8. remove line and paragraph seperators<\/li>\n
      9. remove anything that is not an alpha character<\/li>\n
      10. normalize spaces<\/li>\n
      11. put everything in lower case, and<\/li>\n
      12. remove stop words.<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n
        \n
        (<\/span>load-dictionaries \"resources\/general-corpus-dictionary.csv\"<\/span> \"resources\/domain-corpus-dictionary.csv\"<\/span>)<\/span>\n\n(<\/span>cache-corpus)<\/span>\n\n(<\/span>normalize-cached-corpus \"resources\/corpus\/\"<\/span> \"resources\/corpus-normalized\/\"<\/span>)<\/span>\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n
        \n
        Create New Gold Standard<\/h2>\n
        \n
        Because we never have enough instances in our gold standards to test against, let’s create a third one, but this time adding a music related news feed that will add more positive examples to the gold standard.<\/p>\n
        \n
        (<\/span>defn<\/span> create-gold-standard-from-feeds<\/span>\n  [<\/span>name]<\/span>\n  (<\/span>let<\/span> [<\/span>feeds [<\/span>\"http:\/\/www.music-news.com\/rss\/UK\/news\"<\/span>\n               \"http:\/\/rss.cbc.ca\/lineup\/topstories.xml\"<\/span>\n               \"http:\/\/rss.cbc.ca\/lineup\/world.xml\"<\/span>\n               \"http:\/\/rss.cbc.ca\/lineup\/canada.xml\"<\/span>\n               \"http:\/\/rss.cbc.ca\/lineup\/politics.xml\"<\/span>\n               \"http:\/\/rss.cbc.ca\/lineup\/business.xml\"<\/span>\n               \"http:\/\/rss.cbc.ca\/lineup\/health.xml\"<\/span>\n               \"http:\/\/rss.cbc.ca\/lineup\/arts.xml\"<\/span>\n               \"http:\/\/rss.cbc.ca\/lineup\/technology.xml\"<\/span>\n               \"http:\/\/rss.cbc.ca\/lineup\/offbeat.xml\"<\/span>\n               \"http:\/\/www.cbc.ca\/cmlink\/rss-cbcaboriginal\"<\/span>\n               \"http:\/\/rss.cbc.ca\/lineup\/sports.xml\"<\/span>\n               \"http:\/\/rss.cbc.ca\/lineup\/canada-britishcolumbia.xml\"<\/span>\n               \"http:\/\/rss.cbc.ca\/lineup\/canada-calgary.xml\"<\/span>\n               \"http:\/\/rss.cbc.ca\/lineup\/canada-montreal.xml\"<\/span>\n               \"http:\/\/rss.cbc.ca\/lineup\/canada-pei.xml\"<\/span>\n               \"http:\/\/rss.cbc.ca\/lineup\/canada-ottawa.xml\"<\/span>\n               \"http:\/\/rss.cbc.ca\/lineup\/canada-toronto.xml\"<\/span>\n               \"http:\/\/rss.cbc.ca\/lineup\/canada-north.xml\"<\/span>\n               \"http:\/\/rss.cbc.ca\/lineup\/canada-manitoba.xml\"<\/span>\n               \"http:\/\/feeds.reuters.com\/news\/artsculture\"<\/span>\n               \"http:\/\/feeds.reuters.com\/reuters\/businessNews\"<\/span>\n               \"http:\/\/feeds.reuters.com\/reuters\/entertainment\"<\/span>\n               \"http:\/\/feeds.reuters.com\/reuters\/companyNews\"<\/span>\n               \"http:\/\/feeds.reuters.com\/reuters\/lifestyle\"<\/span>\n               \"http:\/\/feeds.reuters.com\/reuters\/healthNews\"<\/span>\n               \"http:\/\/feeds.reuters.com\/reuters\/MostRead\"<\/span>\n               \"http:\/\/feeds.reuters.com\/reuters\/peopleNews\"<\/span>\n               \"http:\/\/feeds.reuters.com\/reuters\/scienceNews\"<\/span>\n               \"http:\/\/feeds.reuters.com\/reuters\/technologyNews\"<\/span>\n               \"http:\/\/feeds.reuters.com\/Reuters\/domesticNews\"<\/span>\n               \"http:\/\/feeds.reuters.com\/Reuters\/worldNews\"<\/span>\n               \"http:\/\/feeds.reuters.com\/reuters\/USmediaDiversifiedNews\"<\/span>]<\/span>]<\/span>\n\n    (<\/span>with-open<\/span> [<\/span>out-file (<\/span>io<\/span>\/<\/span>writer (<\/span>str \"resources\/\"<\/span> name \".csv\"<\/span>)<\/span>)<\/span>]<\/span>\n      (<\/span>csv<\/span>\/<\/span>write-csv out-file [<\/span>[<\/span>\"class\"<\/span> \"title\"<\/span> \"url\"<\/span>]<\/span>]<\/span>)<\/span>\n      (<\/span>doseq<\/span> [<\/span>feed-url feeds]<\/span>\n        (<\/span>doseq<\/span> [<\/span>item (<\/span>:entries<\/span> (<\/span>feed<\/span>\/<\/span>parse-feed feed-url)<\/span>)<\/span>]<\/span>\n          (<\/span>csv<\/span>\/<\/span>write-csv out-file \"\"<\/span> (<\/span>:title<\/span> item)<\/span> (<\/span>:link<\/span> item)<\/span> :append<\/span> true<\/span>)<\/span>)<\/span>)<\/span>)<\/span>)<\/span>)<\/span>\n<\/pre>\n<\/div>\n
        This routine creates this third gold standard. Remember, we use the gold standard to evaluate different methods and models to classify an input text to see if it belongs to the domain or not.<\/p>\n
        For each piece of news aggregated that way, I manually determined if the candidate document belongs to the domain or not. This task can be tricky, and requires a clear understanding of the proper scope for the domain. In this example, I consider an article to belong to the music domain<\/i> if it mentions music concepts such as musical albums, songs, multiple music related topics, etc. If only a singer is mentioned in an article because he broke up with his girlfriend, without further mention of anything related to music, I won’t tag it as being part of the domain.<\/p>\n
        [However, under a different interpretation of what should be in the domain wherein any mention of a singer qualifies, then we could extend the classification process to include named entities (the singer) extraction to help properly classify those articles. This revised scope is not used in this article, but it does indicate how your exact domain needs should inform such scoping decisions.]<\/p>\n
        You can download this new third gold standard from here<\/a>.<\/p>\n<\/div>\n<\/div>\n
        \n
        Evaluate Initial Domain Model<\/h2>\n
        \n
        Now that we have updated the training corpuses using the updated scope of the domain compared to the previous tests<\/a>, let’s analyze the impact of using a new version of KBpedia and to use a reasoner to increase the number of features in our model. Let’s run our automatic process to evaluate the new models. The remaining steps that needs to be run are:<\/p>\n
          \n
        1. Configure the classifier (in this case, create the semantic vectors for ESA)<\/li>\n
        2. Train the model (in this case, the SVM model), and<\/li>\n
        3. Evaluate the model on multiple gold standards.<\/li>\n<\/ol>\n
          Note: the see the full explanation of how ESA and the SVM classifiers works, please refer to the Create a Domain Text Classifier
          \nUsing Cognonto<\/a> article for more background information.<\/p>\n
          \n
          ;; <\/span>Load positive and negative training corpuses<\/span>\n(<\/span>load-dictionaries \"resources\/general-corpus-dictionary.csv\"<\/span> \"resources\/domain-corpus-dictionary.csv\"<\/span>)<\/span>\n\n;; <\/span>Build the ESA semantic interpreter <\/span>\n(<\/span>build-semantic-interpreter \"base\"<\/span> \"resources\/semantic-interpreters\/base\/\"<\/span> (<\/span>distinct (<\/span>concat (<\/span>get-domain-pages)<\/span> (<\/span>get-general-pages)<\/span>)<\/span>)<\/span>)<\/span>\n\n;; <\/span>Build the vectors to feed to a SVM classifier using ESA<\/span>\n(<\/span>build-svm-model-vectors \"resources\/svm\/base\/\"<\/span> :corpus-folder-normalized<\/span> \"resources\/corpus-normalized\/\"<\/span>)<\/span>\n\n;; <\/span>Train the SVM using the best parameters discovered in the previous tests<\/span>\n(<\/span>train-svm-model \"svm.w50\"<\/span> \"resources\/svm\/base\/\"<\/span>\n                 :weights<\/span> {<\/span>1 50.0}<\/span>\n                 :v<\/span> nil<\/span>\n                 :c<\/span> 1\n                 :algorithm<\/span> :l2l2<\/span>)<\/span>\n<\/pre>\n<\/div>\n
          Let’s evaluate this model using our three gold standards:<\/p>\n
          \n
          (<\/span>evaluate-model \"svm.goldstandard.1.w50\"<\/span> \"resources\/gold-standard-1.csv\"<\/span>)<\/span>\n<\/pre>\n<\/div>\n
          True positive:  21\nFalse positive:  3\nTrue negative:  306\nFalse negative:  6\n\nPrecision:  0.875\nRecall:  0.7777778\nAccuracy:  0.97321427\nF1:  0.8235294<\/pre>\n
          The performance changes related to the previous results (using KBpedia 1.02<\/code>) are:<\/p>\n