Build a Text Summarizer in Java

Get the Code here : https://github.com/ajhalthor/text-summarizer
Follow me on Twitter : https://twitter.com/ajhalthor
Take a look at the original by Shlomi Babluki : http://thetokenizer.com/2013/04/28/build-your-own-summary-tool/

TRANSCRIPT OVERVIEW

ALGORITHM

1. Take the full CONTENT and split it into PARAGRAPHS.

2. Split each PARAGRAPH into SENTENCES.

3. Compare every sentence with every other. This is done by Counting the number of common words and then Normalize this by dividing by average number of words per sentence.

4. These intermediate scores/values are stored in an INTERSECTION matrix

5. Create the key-value dictionary
- Key : Sentence
- Value : Sum of intersection values with this sentence

6. From every paragraph, extract the sentences with the highest score.

7. Sort the selected sentences in order of appearance in the original text to preserve content and meaning.

And like that, you have generated a summary of the original text.


CLASSES IN JAVA PROJECT

1. Sentence : The entire text is divided into a number of paragraphs and each paragraph is divided into a number of sentences.
2. Paragraph : Every paragraph has a number associated with it and an Array List of sentences.
3. Sentence Comparitor : Compare Sentence objects based on Score
4. SentenceComparatorForSummary : Compare Sentence objects based on position in text.
5. SummayTool : akes care of all the operations from extracting sentences to generating the summary.


HOW IS MY SUMMARIZER BETTER THAN THE ORIGINAL ?


My text summarizer selects number of sentences from a paragraph depending on the length. This is an improvement over the original text summarizer implementation that only selects 1 sentence per paragraph regardless of length. So, If the author decides to crunch everything into 1 paragraph, then only one sentence would be chosen. In the current implementation, we set it to accept several sentences for larger paragraphs. It delivers cogent summaries for general essays, reviews and publications.


RUN THIS PROGRAM

$ javac -d bin improved_summary.java
$ java -classpath bin improved_summary