• Instructor: Professor Jianguo Lu. email: jlu at u windsor. Office: LT 5111. Office hours: Monday Wednesday 10:00-11:00.

• Course outline (pdf) . Course introduction Slides (pdf)
• Jupyter Notebook Tutorial [nbviewer] [ipynb]

Text analysis

1. Text statistics. [slides] [nbviewer] [ipynb] [vldb.txt]
2. Good_Turing Smoothing. [nbviewer] [ipynb]
3. Boolean retrieval model Slides (pdf)
   • Readings: Chapter 1
4. Vector space model. [slides] [nbviewer] [ipynb]
5. Word embedding [nbviewer] [ipynb] [WS3535 nbviewer] [WS353 ipynb] WS353-Sim.txt
   • [slides]
   • Titles of DBLP data (272 MB, 5,459,997 paper titles
   • How to use t-SNE effectively
6. Classification Naive Bayes slides (pdf), [nbviewer], [vldb_train]. [icse_train]. [vldb_test]. [icse_test].
   • Readings:
     • chapter 13
     • Tackling the Poor Assumptions of Naive Bayes Text Classifiers. Jason D. M. Rennie
     • A Survey of text classification algorithms, Charu C. Aggarwal, Cheng-xiang Zhai
   • Feature selection feature selection slides(pdf).
   • Code: Starter code for frequency counting and Mutual Information . Written in Python. Top bigrams obtained from the code run on SE and DB papers.
   • Data: VLDB, , ICSE.
7. Evaluation. slides. chapter 8
8. languagde modelling slides . chapter 12
9. Co-occurrence. [nbviewer] [ipynb]
10. LSI(Latent Semantic Indexing) and SVD(Singular Value Decomposition) [slides] ipynb chapter 18 IIR web page
11. Text processing basics (slides) .
    • Readings: Unix for Poets

Graph Analysis

• Link analysis. slides (pptx file) . IIR chapter 21
  Now you can find PageRank values for research papers. Which paper is the most influentialin our following data set?
  • A citation graph for Sigmod conferences [Download the graph]. It contains 6709 edges(citations) among Sigmod papers. Each line representing a citation from the first node to the second.
  • The meta data for Sigmod papers can be downloaded from here. The first column is the paper ID, the second and the third are titles. Note that in this file paper IDs are in HEX format. You can tranform hexadecimal to decimla using int(aHexNumber,16) in python.
  • Here is a starte pyhton code to process the graph. Note that it uses NetworkX to find the large component and calculate the PageRank values. The program returns the top five papers.
  • Larger data sets including papers in three conferences. graph for SIGMOD/VLDB/ICSE , meta data for VLDB and ICSE. Here is the pyhton code to extract subgraphs subsets.py.
  • Here is a good tutorial on writing PageRank for larger graph using Java language.
  • This is the entire citation graph from Microsoft Academic data citations.txt.gz (4.1GB). It contains 528M edges. The data format is

    	PaperID \t PaperID
    

  • All the titles in Microsoft Academic data id_title.txt.gz (5.1GB). The data format is

    	PaperID \t title 
    

    In some cases, downloading from a browser does not work, but wget works fine:

    	wget http://jlu.myweb.cs.uwindsor.ca/8380/id_title.txt.gz
    

  • AMiner data has 25M citations along with titles and abstracts for more than 3M papers. The most recent one is dated Oct 2017. It is in JSON format.
  • A link to some other data sources.

Graph embedding [nbviewer] [ipynb]

• Now you can run node2vec on these networks to obtain shorter vector representation for papers.

Search Engine Construction

• Indexing and search documents Lucene (Slides)
  • Readings: Lucene in Action
  • Starter code: IndexAllFilesInDirectory.java, SearchIndexedDocs.java
  • Data: 10,000 citeSeer papers in gz format (about 1GB). Titles of DBLP data (272 MB, 5,459,997 paper titles
  • Code and data for XML: Here is a sample java code that parses DBLP data using SAX parser and feed into Lucene index DBLP.java. DBLP is a collection of CS papers DBLP site . some data in xml
• Text transformation slides , IIR chapter 2
• Error tolerance search slides, IIR chapter 3

Crawling

• Near duplicate slides (pptx) .
• Crawling the web. crawling lecture slides. IIR chapter 20 tutirial on cralwing tools . chapter 6
• Clustering [K-means Slides] [HAC Slides] [IIR chapter 16] ; IIR chapter 17
  • A Survey of text clustering algorithms, Charu C. Aggarwal, Cheng-xiang Zhai

Resources

• Latex example. You can find the sample document and the tex and cls files here.
• Reuters corpus.
• here is a zip file of another reuters data (around 800,000 files, 1GB). One sample file is 100000newsML.xml. Note thatit is an xml file, and you need to extract and index the relevant elements.
• TREC data collections
• commonCrawl data . It contains all the web pages.
• As We May Think VANNEVAR BUSHJUL 1 1945

Text Book

• [IIR] Introduction to Information Retrieval, by C. Manning, P. Raghavan, and H. Schutze. Cambridge University Press, 2008. book website

Other reference books:

• [SE] Search Engines: Information Retrieval in Practice, by Bruce Croft, Donald Metzler and Trevor Strohman.
• [MIR] Modern Information Retrieval, by R. Baeza-Yates and B. Ribeiro-Neto.
• [LA] Lucene in Action , Michael McCandless, Erik Hatcher, and Otis Gospodneti. 2010.
• [SA] Solr in Action , Trey Grainger and Timothy Potter. sample chapter one , sample chapter three.
• [MMD]Anand Rajaraman and Jeff Ullman, Mining of massive datasets , 2013.

Tools

• A list of tools to extract meta data from research papers (such as title and citation).