ENGGEN 131 Chap.5 Text Data and File Processing
Text Data and File Processing
This chapter separates character values, character vectors and string values before applying comparison, indexing or conversion. It then builds a checked file-processing loop around open validation, successful reads, parsing, state updates and closure. Examples on averages and malformed records show how stale state and incorrect record counts can produce plausible but unsupported output.
What this chapter covers
- 01
Character codes and case conversion
- 02
Character vectors and string values
- 03
Whole-text versus element comparison
- 04
Open, read, parse, update and close
- 05
Blank, malformed and empty-file cases
Worked example · free
Average accepted file records
- 1After two records, sum is 39 and count is 2.
- 1After all records, sum is 80 and count is 4.
- 1The mean is 80 divided by 4, which is 20.
Key terms
- Character code
- The numeric representation associated with a character.
- Parsing
- Converting external text into structured typed values.
- File handle
- A program value representing an opened file resource.
Text Data and File Processing FAQ
Why normalise letter case?
Normalising case before comparison makes the policy explicit and avoids rejecting an otherwise valid spelling that differs only by uppercase or lowercase letters. Rehearse this distinction with a fresh text and record-boundary trace before relying on program output.
What should happen when a file cannot open?
Check the returned handle and report or handle failure before attempting any read operation. Test the claim on a boundary case where the text and record-boundary assumptions become visible.
How do I avoid processing the final record twice?
Drive the loop with the success of the current read, and update state only after that read has produced a valid record. Write the expected text and record-boundary state first, then compare it carefully with execution.
Exam move
Start by naming the text representation and input unit. A character vector, string scalar, word, line and delimited record have different indexing, whitespace and comparison behaviours. Write whether matching is case-sensitive and whether leading, trailing or repeated separators matter. For position-based work, list valid indices before extracting a prefix or suffix.
Test empty text, one character, mixed case and nonmatching text. Normalise case only when the requirement permits it; other deletions may merge distinct inputs. Organise file processing into acquisition, parsing and computation. Acquisition obtains one candidate record and reports whether a read occurred. Parsing converts that text into required fields and validates them.
Computation updates trusted program state only after parsing succeeds. Draw these as three boxes and do not allow a previous numeric value to bypass the parsing boundary after a failed conversion. This separation also lets an error message identify whether the file, record shape or domain rule caused rejection. Maintain explicit invariants for aggregation.
After every accepted record, the accepted count equals the number of values included and the running sum equals exactly their total. Rejected or blank records change neither quantity unless the stated policy says otherwise. For an average, divide by the accepted count, not the number of lines encountered, and define what happens when no usable records exist.
Work a three-record file by hand and compare the final count, sum and statistic independently. Test file state deliberately. Include an unavailable file, empty file, valid final line, blank line, malformed field and a mixture of valid and invalid records. Check the open result before reading and each read or conversion result before using destinations.
Drive the loop with successful acquisition, because end-of-file is learned by attempting the next read. Do not process the old buffer after failure. Close the handle along successful and error paths so resource management is part of the algorithm. For text transformations, track both values and positions.
Character codes can support category tests and case relationships, but full-word decisions should compare complete normalised text rather than only a first character. When splitting, state whether separators are retained and whether adjacent separators create empty fields. When joining, predict exactly where separators appear.
Use examples with repeated or boundary separators to challenge assumptions that a tidy demonstration string never reveals. Report the result length as well as its displayed content. Finish by writing a record policy before code: accepted form, invalid-record response, empty-input response and summary reported at completion.
Trace one file with a table containing read status, raw text, parse status, accepted count and running result. After execution, reconcile total records into accepted and rejected counts. Keep errors classified as acquisition, parsing, domain or aggregation defects. Confidence comes from proving which records influenced the answer, not from obtaining a neatly formatted number from one cooperative file.
Finish with a miniature file containing a valid record, blank line, mixed-case token, malformed number and valid final record. Write the acceptance policy first. Trace acquisition status, raw text, parse result, accepted count and running aggregate for every attempted read, including the failed read after the final record.
Separately test a whole-text comparison after case normalisation and a suffix extraction on empty and one-character inputs. Reconcile the number of attempted, accepted and rejected records and state the empty-data result policy. This integrated trace should make stale-value reuse impossible because computation occurs only after a successful parse.
It also verifies that text representation, positions, separators, file state, conversion results and aggregation invariants are handled as connected boundaries rather than isolated syntax facts.