LT5903 Chap.10 Language Variation and Computation
Language Variation and Computation
Systematic patterns in people and models
Sociolinguistics defines varieties through systematic structural or lexical features and separates linguistic organisation from social prestige. Variation appears across phonetics, phonology, morphology, syntax and vocabulary and correlates with regional, socioeconomic, age, gender and ethnic factors.
Computational linguistics turns patterns into procedures through tokenisation, part-of-speech tagging, context-free parsing and distributional representations.
Skip-gram training estimates word-context relations, while cosine similarity compares vector direction and requires explicit calculation and careful interpretation.
Treat every variety as a system
A nonstandard form can follow a stable rule even when institutions stigmatise it. Standard status reflects historical power, education and public use rather than greater logical organisation.
Describe the structural pattern before discussing social evaluation. The language-dialect distinction also cannot be reduced to mutual intelligibility because identity, history and political boundaries matter.
Avoid ranking speakers by a prestige norm when the evidence concerns systematic phonetic, grammatical or lexical variation.
Interpret correlations without stereotypes
Variation may correlate with region, socioeconomic position, age, gender or ethnicity, but a group average does not determine every individual's speech. Sampling, audience, style and social network can change the observed distribution.
State the linguistic variable, variants, speaker grouping and context of collection. A correlation supports an association in that dataset, not an inherent property of a social category. Consider within-speaker style shifting and overlap between groups before drawing a social conclusion.
Make preprocessing decisions explicit
Computational analysis begins before the model.
Tokenisation chooses units, normalisation decides which surface distinctions to merge and sentence segmentation sets boundaries. Part-of-speech tagging adds local categories that parsers can use. An upstream choice can erase a contrast or create an artificial pattern, so inspect the input when a result looks surprising.
For multilingual or socially variable data, a convention designed for one variety may systematically misrepresent another. Document the decision rather than treating preprocessing as neutral.
Trace parsing as a sequence of states
A top-down parser begins with a start symbol and expands predicted categories toward the words. It can pursue structures that never fit the input.
A bottom-up parser shifts observed items and reduces them into constituents, but may build locally plausible groups that cannot complete a sentence. Record each stack or derivation state and identify the first impossible choice.
Context-free rules represent constituency clearly, though they do not by themselves encode every graded preference or long-distance dependency.
Calculate cosine and limit its meaning
For vectors A and B, compute the dot product, calculate each Euclidean length and divide the dot product by the product of the lengths. The result ranges from minus one to one and compares direction rather than magnitude.
A value near one indicates similar proportions across represented dimensions or contexts. It does not prove synonymy, reference or equal frequency. Skip-gram embeddings learn from word-context prediction, so topical associates can be close. End by naming what supplied the dimensions and which semantic claim remains outside the representation.
What this chapter covers
- 01
Dialect and language classification
- 02
Standard prestige and systematicity
- 03
Dimensions of social variation
- 04
Context-free parsing
- 05
Word vectors and cosine similarity
Worked example · free
Compute a cosine with new vectors
- 2Calculate the dot product.
- 4Calculate both lengths and divide.
- 2Interpret direction, not frequency.
Key terms
- Language Variety
- A systematic form of language associated with a community or setting.
- Standard Dialect
- A variety carrying institutional prestige and power.
- Context-Free Grammar
- Rules and a lexicon that generate constituent structures.
- Distributional Semantics
- Representation of meaning through patterns of contextual use.
- Word Embedding
- A vector representation learned from linguistic contexts.
- Cosine Similarity
- A normalised measure of the angle between vectors.
Language Variation and Computation FAQ
Does nonstandard mean unsystematic?
No. A nonstandard variety can have stable phonological, morphological and syntactic rules. Standard status reflects institutional prestige and social power, not greater linguistic logic. Analyse the pattern before evaluating its social meaning.
How do top-down and bottom-up parsing differ?
Top-down parsing begins with the start symbol and expands rules toward the input. Bottom-up parsing shifts input items and reduces them into larger categories until reaching the start symbol. Both depend on the grammar and lexicon supplied.
What does a high cosine similarity mean?
It means two vectors point in similar directions across their dimensions. In a distributional model, that often indicates similar contextual use. It does not prove synonymy, reference identity or equal frequency, because vector direction preserves only part of linguistic meaning.
Exam move
Keep social and computational claims separate while using the same discipline of pattern analysis. For cosine work, print every arithmetic step. For variation, name the linguistic level and the social factor without treating correlation as deficiency. Audit how the sample and speaking context limit a social correlation.
For model outputs, work backwards through tagging, tokenisation and normalisation, then state which linguistic distinction the representation preserves and which it discards.
Working through Language Variation and Computation in LT5903? Sia is AskSia’s AI Arts and Humanities tutor — ask any LT5903 Language Variation and Computation question and get a clear, step-by-step explanation grounded in how LT5903 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
