IS6335 Chap.9 Adjacency Matrices, Communities and Text Mining
Adjacency Matrices, Communities and Text Mining
The idiom swap that rescues a dense graph
Week nine continues the network material with the visualization of the adjacency matrix, then community detection with its clustering algorithms, then text mining, and then string similarity together with word association. Its tutorial covers igraph continued plus text mining with quanteda.
The first of those topics is the direct answer to the density limit the previous chapter ended on.
In a node-link drawing the positions are computed from the links, so adding links makes the picture worse. In a matrix the positions come from the node ordering alone, so adding links only fills cells: nothing overlaps and nothing can be hidden. The cost is paths.
Following a route of two steps means finding one cell, reading off its column, finding that node on the row axis and starting again, which is why path was listed as the task a matrix serves badly.
The row order is the entire argument
A matrix with its nodes in alphabetical order shows nothing. A matrix with its nodes ordered by community shows blocks on the diagonal.
The visual structure is produced by the reordering, which means the reordering is a claim you have to justify rather than a display setting. This is the reorder move from week five doing the heaviest work it does anywhere in the course, and it is the reason the same idiom reappears as a confusion matrix in the advanced module.
Community detection deserves the same scepticism.
These algorithms partition every graph, including a random one, so their output is a result rather than an observation. Before reporting communities as a finding, check that the partition survives re-running the algorithm or moving its parameters, and name the algorithm that produced it.
A grouping that changes every run describes the method rather than the network.
Turning documents into something you can draw
The second half of the week moves to text. The reason text appears in a visualization course is structural: text is not one of the three dataset types, so it has to be turned into one first, and every step of the conversion is a design decision.
The pipeline runs in five steps.
Tokenise, splitting documents into terms, where the split decides whether hyphenated and multi-word expressions survive as single terms. Normalise, by case folding, stemming and stop-word removal, each of which discards information; removing stop words is usually right and occasionally destroys the subject of the analysis. Count, building the document by term table, which is a table in the framework sense.
Associate, linking terms that co-occur, which produces a network whose nodes are terms and whose link weights are association strengths. And reduce, because the term network is dense by construction, so a threshold is always applied and always has to be reported.
Once the counts table exists, nothing about the rest of the analysis is special to text.
Its items are documents and its attributes are terms, so every idiom from week four onward applies unchanged.
That is the payoff for doing the conversion carefully, and it is also why a mistake made during tokenisation is so expensive: it is baked into every view built afterwards and it is invisible in all of them.
Similarity is a choice, and co-occurrence is a weak claim
Similarity between two strings can be measured by how many single-character edits separate them, by how many character sequences they share, or by how much their term vectors overlap.
The three disagree in patterned ways: edit distance is unforgiving about length, shared sequences tolerate transposition, vector overlap ignores word order entirely. Naming the measure is part of reporting the result.
Both halves of the week produce groupings very readily, and a grouping is easy to mistake for a finding. Three questions cover most of it.
Would the same grouping appear if the algorithm were re-run or its parameters changed? Does it survive the threshold that produced it being moved by a reasonable amount? And does the claim use an attribute that is actually in the data, or one the reader is being invited to assume?
What this chapter covers
- 01
Why a matrix cannot become a hairball
- 02
The cost a matrix imposes on path reading
- 03
Node ordering as an analytical claim
- 04
Community detection as a result rather than an observation
- 05
Tokenise, normalise, count, associate, reduce
- 06
The document by term table as an ordinary table
- 07
Three string similarity measures and how they disagree
- 08
Testing a grouping before reporting it
Read a word association network honestly
- 2State precisely what co-occurrence establishes.
- 2Name the attribute the claim needs and is missing.
- 2State the reporting obligation.
Key terms
- Adjacency matrix
- A network written as a grid with every node on both axes and a filled cell where two are connected. Its layout is independent of the links, so it survives any density.
- Community detection
- An algorithm that partitions a network into groups of densely connected nodes. It returns a partition for any graph, so its output has to be tested for stability before it is reported.
- Tokenisation
- Splitting documents into terms. Where the split falls decides whether hyphenated and multi-word expressions survive, and the decision propagates into every later view.
- Document term table
- A table whose items are documents and whose attributes are terms, holding counts in its cells. Once it exists, the ordinary table idioms apply to text unchanged.
- Word association
- A link between two terms that co-occur, weighted by how strongly. It establishes that terms appear together and does not by itself establish why.
- Edit distance
- A similarity measure counting the single-character changes separating two strings. It is unforgiving about differences in length, which is one reason the measure used has to be named.
Adjacency Matrices, Communities and Text Mining FAQ
My matrix looks like random speckle. What now?
Reorder the rows and columns, either by a community detection result or by a simple clustering of the rows, so that similar nodes sit together and any block structure is brought onto the diagonal. Then stop if the speckle persists under two different orderings.
That persistence is evidence the network has no block structure, which is a real finding and should be reported as one rather than hidden by hunting through orderings until a flattering one appears. The temptation to keep searching is exactly the failure the stability check exists to prevent.
Should I remove stop words?
Usually yes, and check what you removed. Stop-word removal is the normalisation step that most often improves a term network, because the commonest function words co-occur with everything and drown the structure. It is also the step that occasionally deletes the subject of the analysis, particularly when the question concerns negation, modality or short function-word phrases that carry the meaning.
The cheap safeguard is to run the counts once with the list and once without, compare the top terms, and say in the write-up which list you used.
Assessment move
Take one short corpus you can read yourself, a few hundred reviews or comments, and run the five pipeline steps by hand or in code with the intermediate output printed at each stage. Reading the token list and the top counts before you draw anything is the only reliable way to notice that a tokenisation choice has broken something, and it also gives you the description of variables and features that the report rubric asks for.
Working through Adjacency Matrices, Communities and Text Mining in IS6335? Sia is AskSia’s AI Information Technology tutor — ask any IS6335 Adjacency Matrices, Communities and Text Mining question and get a clear, step-by-step explanation grounded in how IS6335 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
