CL-NLP
3 likes1,321 views
An overview of CL-NLP library for doing natural language processing in Lisp and a description of some of its key design decisions and features
![General-Purpose Utils
(json:encode-json #{
"success" t
"sentence" text
"match"
(when-it (2nd (funcall fn (get-model text)))
(strcat
"{"
(strjoin ","
(mapcar #`(fmt ""~A":[~A,~A]"
(car %) (cadr %)
(cddr %))
(ht->alist it)))
"}"))
} stream)](https://image.slidesharecdn.com/cl-nlp-130701153558-phpapp02/85/CL-NLP-14-320.jpg)
![Basic Cell
(defclass regex-word-tokenizer (tokenizer)
((regex :accessor tokenizer-regex
:initarg :regex
:initform
(re:create-scanner
"w+|[!"#$%&'*+,./:;<=>?@^`~…()
{}[|] «»“”‘’¶-]"⟨⟩ ‒–—― )
:documentation
"A simpler variant would be [^s]+ —
it doesn't split punctuation, yet
sometimes it's desirable."))
(:documentation
"Regex-based word tokenizer."))](https://image.slidesharecdn.com/cl-nlp-130701153558-phpapp02/85/CL-NLP-18-320.jpg)
Ad
More Related Content
What's hot (20)
Viewers also liked (17)
Ad
Similar to CL-NLP (20)
Ad
Recently uploaded (20)
CL-NLP
- 1. CL-NLP – A Natural Language Processing Library for Common Lisp Vsevolod Dyomkin
- 2. Topics * Motivation * Library high-level design * Overview of the functions * Some technical details * Next steps
- 3. Motivation NLP & Lisp is a match, made in heaven
- 4. Previous Work * CL-Langutils * Lexiparse, Sparser, CL-EARLY- PARSER, Basic-English-Grammar * Wordnet interfaces * CMU AI Repository * Natural Language Understanding * Natural Language Processing in Lisp
- 5. Essential Scope * Corpora interfaces * Language modeling * Measures * Tokenization * Tagging * Parsing * Wordnet interface * Construct a pipeline
- 7. An NLP Pipeline “This is a test.” -> TOP | S /~~~~~~~~~~|~~~~~~~~ NP VP . | /~~~~~~ | DT VBZ NP . | | /~~~~ This is DT NN | | a test
- 8. An NLP Pipeline (print-parse-trees "This is a test.") (defun print-parse-trees (text) (dolist (sentence (tokenize <sentence-tokenizer> text)) (pprint-tree (parse *pcfg* (tag *hmm* text))))) (defparameter *hmm* (train 'hmm-tagger "treebank_dir/")) (defparameter *pcfg* (train 'pcfg "treebank_dir/"))
- 9. Some Design Choices * A pseudo-hierarchical package design
- 10. High-level Design * cl-nlp & cl-nlp-contrib & more * base & functional packages * nlp-user
- 11. Modules * nlp.core, nlp.corpora, nlp.util, nlp.test-util * nlp.syntax, nlp.generation, nlp.phonetics * nlp.contrib.wordnet, nlp.contrib.ms-ngrams
- 12. Some Design Choices * Use the whole language - defclass & defstruct - hash-tables & alists - do & loop
- 13. Some Design Choices * Grow the language - rutils + reader macros + shorthands + dotable, dolines - special-purpose utils
- 14. General-Purpose Utils (json:encode-json #{ "success" t "sentence" text "match" (when-it (2nd (funcall fn (get-model text))) (strcat "{" (strjoin "," (mapcar #`(fmt ""~A":[~A,~A]" (car %) (cadr %) (cddr %)) (ht->alist it))) "}")) } stream)
- 15. A Special-Purpose Util (define-lazy-singleton word-tokenizer (make 'postprocessing-regex-word-tokenizer) "Default word tokenizer.") (defun tokenize-ngram (ngrams str) "Transform string STR to a list if necessary (depending of order of NGRAMS)." (if (> (ngrams-order ngrams) 1) (tokenize <word-tokenizer> str) str))
- 16. A Special-Purpose Util (defmacro define-lazy-singleton (name init &optional docstring) (with-gensyms (singleton) `(let (,singleton) (defun ,name () ,docstring (or ,singleton (setf ,singleton ,init))) (define-symbol-macro ,(mksym name :format "<~A>") (,name)))))
- 17. Some Design Choices * Use CLOS as a foundation
- 18. Basic Cell (defclass regex-word-tokenizer (tokenizer) ((regex :accessor tokenizer-regex :initarg :regex :initform (re:create-scanner "w+|[!"#$%&'*+,./:;<=>?@^`~…() {}[|] «»“”‘’¶-]"⟨⟩ ‒–—― ) :documentation "A simpler variant would be [^s]+ — it doesn't split punctuation, yet sometimes it's desirable.")) (:documentation "Regex-based word tokenizer."))
- 19. Basic Cell (defmethod tokenize ((tokenizer regex-word-tokenizer) string) (loop :for (beg end) :on (re:all-matches (tokenizer-regex tokenizer) string) :by #'cddr :collect (sub string beg end) :into words :collect (cons beg end) :into spans :finally (return (values words spans)))
- 20. Another Example (defgeneric parse (model sentence) (:documentation "Parse SENTENCE with MODEL.") (:method :around (model (sentence string)) (call-next-method model (tokenize <word-tokenizer> string)))) (defgeneric parse-n (model sentence n) (:documentation "Return N best parse trees of the SENTENCE with MODEL.") (:method :around (model (sentence string) n) (call-next-method model (tokenize <word-tokenizer> string) n)))
- 21. Parsing (defmethod parse ((grammar pcfg) (sentence list)) (CKY (let* ((cur (cons rule (1- s))) (l (@ pi0 (1- i) (1- s) (second rule))) (r (@ pi0 s (1- j) (third rule))) (score (if (and l r) (+ (log q) l r) min))) (when (> score (or max min)) (setf max score arg cur)))))
- 22. Parsing (defmethod parse :around ((grammar pcfg) (sentence list)) (with-raw-results (values (idx->nts (decode-parse-tree sentence bps 0 last iroot)) (exp (or (@ pi0 0 last iroot) min)) pi0 bps)))
- 23. (macrolet ((CKY (&body body) `(with-slots (rules nts->idx) grammar (let* ((pi0 #{}) (bps #{}) (min most-negative-single-float)) ;; init pi0 & bps (skipped) (do ((pos 1 (1+ pos))) ((>= pos *sentence-length*)) (do ((i 1 (1+ i))) ((> i (- *sentence-length* pos))) (let ((j (+ i pos))) (dotable (_ k nts->idx) (let (max arg) (do ((s i (1+ s))) ((>= s j)) (dotable (rule q rules) (when (and (tryadic rule) (= k (first rule))) ,@body))) (when (if (listp max) max (> max min)) (setf (@ pi0 (1- i) (1- j) k) max (@ bps (1- i) (1- j) k) arg))))))) (values pi0 bps)))))
- 24. (declaim (inline @)) (defun @ (m i j k) (get# (+ (* i *sentence-length* *nt-count*) (* j *nt-count*) k) m)) (defsetf @ (m i j k) (v) `(set# (+ (* ,i *sentence-length* *nt-count*) (* ,j *nt-count*) ,k) ,m ,v))