Agricultural Statistics & Data Science · Foundations · also Statistical Programming, R Programming, Python for Data Science
Writing correct, readable programs for data work: R and Python as the working languages, with the control structures, functions, data frames, vectorisation and debugging habits that later computing subjects assume. C and SAS are covered where agricultural universities still teach them.
In the exams: GATE DA 'Programming, Data Structures and Algorithms' (heavy: Python named); AICE-JRF Computer Application Unit 2 programming (heavy); AIEEA-PG Group 07 computer programming (present)
A modern statistician writes code every working day. Field trial data arrive as spreadsheets with inconsistent plot labels; survey files need weights merged and missing codes recoded; analyses must be repeated each season on new data; simulations check whether a method works; and results must be reproducible by a colleague or a reviewer years later. Point-and-click software cannot do this reliably. Programming turns an analysis into a precise, repeatable, checkable set of instructions. This subject teaches the programming a statistician needs: how to think in algorithms, and how to write correct, readable programs in R and Python, the two working languages of statistics and data science today, with introductions to SAS, still widely used in agricultural research institutes, regulators and industry, and to C, still taught in many agricultural universities as a first language and useful for understanding how computers handle data.
The manual covers algorithms, flowcharts and pseudocode; R vectors, data frames, factors and lists; R functions, control flow and vectorisation; Python's core types, NumPy arrays and pandas data frames; reading, cleaning and reshaping data; graphics in R and Python; writing functions and simple packages; debugging, testing and code style; SAS DATA and PROC steps for agricultural analyses; and an introduction to C, with data types, arrays, pointers and file handling. Code is described in prose, with short expressions and their results, because the aim is understanding that transfers between languages rather than memorising syntax.
Graduate entrance examinations in data science test programming with questions on code output, data structures and algorithmic complexity, with Python as the named language; doctoral examinations in computer applications for agriculture test programming concepts in C, C++ and other languages; master's admission examinations in agricultural statistics ask about computer programming basics; and professional statistical-software certifications test the DATA step and analytical procedures in detail.
An algorithm is a finite sequence of unambiguous steps that transforms inputs into outputs. Knuth's classical properties are finiteness (it terminates), definiteness (each step is precisely defined), input, output and effectiveness (each step can actually be carried out). Before writing code, a statistician should be able to state the algorithm: for example, to compute a sample variance, read the n values, compute their mean, sum the squared deviations from the mean and divide by n − 1. Expressing the steps first, independently of language, separates thinking about the method from thinking about syntax and makes errors easier to find. Algorithms are judged by correctness, by efficiency in time and memory, and by numerical stability. The one-pass 'textbook' variance formula, (Σx² − (Σx)²/n)/(n − 1), is algebraically correct but loses precision badly when the mean is large relative to the spread, because it subtracts two nearly equal large numbers; the two-pass algorithm or Welford's one-pass updating algorithm avoids this, which is why statistical software uses them.
A flowchart draws an algorithm with standard symbols: an oval (terminal) for start and end, a parallelogram for input and output, a rectangle for a process or calculation, a diamond for a decision with yes and no branches, and arrows for the flow of control. Pseudocode writes the algorithm in structured plain language using the three control structures from which any algorithm can be built: sequence (steps one after another), selection (IF condition THEN … ELSE …) and repetition (WHILE condition DO …, or FOR each item DO …). For instance, to classify plots by yield: FOR each plot: IF yield ≥ 5 THEN label 'high' ELSE IF yield ≥ 3 THEN label 'medium' ELSE label 'low'. Writing pseudocode first is good practice for any program longer than a few lines and for describing methods in papers.
The running time of an algorithm is described by how it grows with the size n of the input, in big-O notation, ignoring constant factors. Linear search through an unsorted list is O(n); binary search in a sorted list halves the remaining range at each step and is O(log n), so about 20 comparisons suffice for a million items, since 2²⁰ ≈ 1.05 million; simple sorting algorithms such as bubble sort and insertion sort are O(n²), making up to n(n − 1)/2 comparisons (45 for n = 10); efficient sorts such as merge sort are O(n log n); and multiplying two n × n matrices directly is O(n³). For statistical work these differences decide whether an analysis of a large survey or genomic data set finishes in seconds or days. Memory use matters too: copying a large data set inside a loop can exhaust memory even when the arithmetic is cheap.
R is built around vectors: even a single number is a vector of length one. Vectors are created with c(), as in x <- c(2, 4, 6, 8), or with sequences: 1:5 gives 1 2 3 4 5, and seq(2, 11, by = 3) gives 2 5 8 11. All elements of an atomic vector share one type, in increasing order of generality logical, integer, double (numeric), character; combining types coerces to the most general, so c(1, "a") is a character vector. Indexing starts at 1: x[2] is 4, x[2:3] is 4 6, x[-1] drops the first element, and logical indexing x[x > 4] gives 6 8. Arithmetic is element-wise and vectorised: x * 2 doubles every element. When vectors of different lengths are combined, the shorter is recycled: c(1, 2, 3, 4) + c(10, 20) gives 11 22 13 24, with a warning only if the longer length is not a multiple of the shorter. Logical values count as 1 and 0 in arithmetic, so sum(x > 4) counts elements greater than 4, giving 2, and mean(x > 4) gives the proportion. Missing values are NA; most summaries return NA if any value is missing unless na.rm = TRUE is given, and comparisons with NA give NA, so is.na() must be used to find missing values. Floating-point arithmetic is inexact: 0.1 + 0.2 == 0.3 is FALSE, so isTRUE(all.equal(a, b)) is used to compare numbers.
A factor stores a categorical variable as integer codes with a set of labels called levels, which by default are sorted alphabetically. Factors control how models treat categorical variables, defining dummy variables and the reference level (changed with relevel). A notorious trap is converting a factor whose labels look like numbers: as.numeric(factor(c("10", "20", "10"))) returns the codes 1 2 1, not the values; as.numeric(as.character(f)) gives 10 20 10.
A data frame is a list of equal-length columns, possibly of different types, and is R's standard structure for a data set with variables in columns and observations in rows. Columns are accessed as df$yield or df[["yield"]], rows and columns by df[rows, cols], and useful inspection functions include str(), head(), summary() and nrow(). Since R 4.0, data frames no longer convert strings to factors by default. A list is a general container whose elements can be of any type and length, including other lists, data frames and model objects: fitted models are lists. Single brackets on a list return a sub-list, while double brackets return the element itself, so lst[1] is a list of length one and lst[[1]] is its content. Matrices are vectors with a dimension attribute, filled column by column by default, and arrays extend this to more dimensions.
Functions are defined with function(arguments) { body }, and return the value of their last expression or of an explicit return(). Arguments can have defaults, can be matched by name or position, and are evaluated lazily, only when used. For example, cv <- function(x, na.rm = TRUE) { 100 * sd(x, na.rm = na.rm) / mean(x, na.rm = na.rm) } computes a coefficient of variation. R uses lexical scoping: a function looks up variables first in its own environment and then in the environment where it was defined, not where it was called; relying on global variables inside functions makes code fragile and should be avoided by passing everything needed as arguments. Functions are objects and can be passed to other functions, which is the basis of the apply family: lapply applies a function to each element of a list and returns a list; sapply simplifies the result to a vector or matrix where possible; vapply does the same with a declared output type, which is safer; apply works over the rows (MARGIN = 1) or columns (MARGIN = 2) of a matrix; tapply computes a summary for each group of a factor, as in tapply(yield, variety, mean); and Map and Reduce combine several vectors.
Selection uses if (condition) { … } else { … }, where the condition must be a single TRUE or FALSE; the vectorised ifelse(test, yes, no) applies a condition element-wise, so ifelse(x > 5, "high", "low") labels a whole vector. Repetition uses for (i in seq_along(x)) { … }, while (condition) { … } and repeat { … } with break; next skips to the next iteration. Using seq_along(x) or seq_len(n) rather than 1:length(x) avoids the bug where 1:0 gives c(1, 0) for an empty vector. As a simple check of understanding, s <- 0; for (i in 1:5) s <- s + i leaves s equal to 15. Integer division and remainders are %/% and %%: 17 %/% 5 is 3 and 17 %% 5 is 2. The %in% operator tests membership, and which() returns the positions of TRUE values.
Vectorised operations work on whole vectors at once with loops carried out in compiled code, and are usually far faster and clearer than explicit R loops. sum(x^2) is preferred to a loop that accumulates squares, and colMeans(m) to a loop over columns. Loops are not wrong in R, but two habits make them slow: growing a vector inside a loop with x <- c(x, new), which copies the vector each time, and repeatedly modifying a data frame. Pre-allocating the result, as in out <- numeric(n), avoids the first. For heavy computations, profiling identifies the slow parts before effort is spent optimising, and parts can be rewritten in compiled languages when necessary. Random numbers are generated with functions such as rnorm and runif, and set.seed() fixes the starting point so that simulations can be reproduced exactly.
Python's core types include int (arbitrary-precision integers), float, bool, str, and the containers list (ordered and mutable), tuple (ordered and immutable), dict (key–value mapping) and set (unordered unique elements). Indexing starts at 0 and slices exclude their end point: for a = [10, 20, 30, 40, 50], a[0] is 10, a[1:3] is [20, 30], a[-1] is 50 and a[::2] is [10, 30, 50]. range(5) produces 0, 1, 2, 3, 4. Operators: / is true division (7/2 is 3.5), // is floor division (7//2 is 3, and −7//2 is −4 because it rounds towards minus infinity), % is the remainder (7 % 3 is 1) and ** is the power (2**10 is 1024). True and False behave as 1 and 0 in arithmetic. Indentation defines blocks: if, for and while statements and function definitions end with a colon and their bodies are indented, conventionally by four spaces. Functions are defined with def name(args): and return values with return. Assignment binds names to objects, so b = a for a list makes both names refer to the same list, and changing b changes a; a.copy() or list(a) makes a separate copy. Equality is tested with ==, while is tests whether two names refer to the same object.
NumPy provides the ndarray, a fixed-type, multi-dimensional array on which arithmetic is vectorised and executed in compiled code. np.array([1, 2, 3]) creates an array; np.arange(0, 10, 3) gives [0, 3, 6, 9]; np.linspace(0, 1, 5) gives five equally spaced values from 0 to 1 inclusive; and np.zeros((3, 4)) creates a 3 × 4 array of zeros. The shape attribute gives dimensions. Reductions take an axis argument: for a two-dimensional array, mean(axis=0) averages down the rows giving column means, and axis=1 gives row means. Broadcasting extends operations to arrays of different but compatible shapes by stretching dimensions of length one: adding an array of shape (3, 1) to one of shape (4,) gives a (3, 4) result. Boolean arrays index arrays, as in x[x > 0], and missing values are represented by the floating-point NaN, which propagates through arithmetic; functions such as np.nanmean ignore it. Matrix multiplication is written A @ B, while A * B multiplies element by element, a common source of errors for those coming from mathematics.
pandas provides the Series (a labelled one-dimensional array) and the DataFrame (a table of columns of possibly different types sharing a row index), modelled on R's data frame. Data are read with functions such as read_csv and read_excel. Columns are selected as df["yield"] or df[["yield", "variety"]]; rows and columns are selected by label with df.loc[row_labels, col_labels] and by integer position with df.iloc[i, j], so df.iloc[0] is the first row whatever its label. Boolean filtering, df[df["yield"] > 5], selects rows. Grouped summaries use split–apply–combine: df.groupby("variety")["yield"].mean() gives the mean yield of each variety, and agg applies several functions at once. Missing values (NaN) are detected with isna(), removed with dropna() or replaced with fillna(), and most summaries skip them by default, unlike R. Tables are joined with merge (inner, left, right or outer joins on key columns) and stacked with concat. Python's statistical modelling is provided by further libraries for statistical models and machine learning that build on NumPy and pandas.
Python's if, elif and else statements choose between branches, and for loops iterate directly over the items of any sequence or iterable, as in for plot in plots:, rather than over index numbers; enumerate supplies an index alongside each item when one is needed, and zip pairs up items from several sequences. while loops repeat while a condition holds, with break and continue to leave a loop or skip an iteration. List comprehensions build lists concisely: [y * 10 for y in yields if y is not None] converts tonnes per hectare to quintals while skipping missing entries, and dictionary comprehensions do the same for dicts. Functions take positional and keyword arguments with defaults, and default values are evaluated once when the function is defined, so a mutable default such as an empty list is shared between calls, a well-known trap; None is used as the default instead. Exceptions signal errors: code that might fail, such as converting a text field to a number, can be placed in a try block with an except clause that handles the specific error, rather than letting the whole program stop or, worse, silently continuing with wrong values.
Most analysis time goes on getting data into shape. Data come from spreadsheets, delimited text files, databases, survey platforms and instruments. When reading, specify the delimiter, decimal mark (comma in many countries), header row, missing-value codes and column types explicitly rather than trusting defaults, and record the source and date of each file. After reading, inspect the structure: number of rows and columns, variable types, ranges, counts of missing values and distinct values of categorical variables. Common problems include numbers stored as text because of a stray character or a thousands separator; missing values coded as −99, 999 or a blank that must be converted to the language's missing value before any summary is computed (otherwise a mean yield may include −99 entries); inconsistent labels such as 'Var A', 'var a' and 'VarA' for the same variety; duplicate records; impossible values such as negative weights or plant heights of 3,000 cm instead of 300; and dates in mixed formats. Validation rules check ranges, allowed codes and relationships between variables, and should be written as code so that they run every time new data arrive.
Wickham's tidy data principles state that each variable forms a column, each observation forms a row, and each type of observational unit forms a table. Many field data sheets are not tidy: a 'wide' layout with one row per plot and separate columns for yields in 2022, 2023 and 2024 stores the year, a variable, in column names. Reshaping converts between wide and long forms: in R, pivot_longer and pivot_wider (from the tidyr package) or reshape; in pandas, melt (wide to long) and pivot or pivot_table (long to wide). Long form suits most modelling and plotting, with one row per plot per year; wide form suits some multivariate methods and human reading. Combining tables uses joins on key variables: an inner join keeps only matching keys, a left join keeps all rows of the first table and fills missing matches with NA, and a full join keeps everything. Joining a plot-level yield table to a treatment table by plot identifier is a left join; checking the number of rows before and after a join catches duplicated keys, which multiply rows silently.
All cleaning should be done by script, never by editing the raw file by hand, so that every change is documented and repeatable. The raw data are kept unchanged; the script reads them, applies corrections with comments explaining each, and writes a clean data set. Decisions such as excluding a damaged plot or correcting an obvious unit error should be recorded with reasons. This makes it possible to rerun the whole analysis when new data arrive or an error is discovered, and to show reviewers exactly what was done.
Dates and text cause many silent errors. Sowing, flowering and harvest dates should be stored as proper date types, not as text or as spreadsheet serial numbers, so that intervals such as days to flowering can be computed correctly; formats such as 03/04/2024 are ambiguous between day-first and month-first conventions, so the format must be stated when parsing, and the international ISO 8601 form 2024-04-03 avoids the ambiguity. Text fields need trimming of stray spaces and consistent case before they are used as categories, so that 'Var A ' and 'var a' are not counted as different varieties, and recoding tables that map variant spellings to standard names should be kept with the script. Character encodings matter when names contain accented or non-Latin letters; reading files with the correct encoding, usually UTF-8, prevents corrupted labels. Categorical variables should be converted to factors in R, or to categorical types in pandas, with their levels and reference category set deliberately before modelling.
R's base graphics draw plots with functions such as plot (scatter and line plots), hist (histograms), boxplot, barplot and pairs (scatter-plot matrices), and add elements with lines, points, abline, text and legend. They are quick for exploration: plot(yield ~ nitrogen, data = trial) draws a scatter plot, and boxplot(yield ~ variety, data = trial) compares distributions. Graphical parameters set through par control layout (mfrow for several panels), margins, symbols and colours. Plots are saved by opening a device such as pdf() or png(), drawing, and closing it with dev.off().
The ggplot2 package implements Wilkinson's grammar of graphics, in which a plot is built from components: data, aesthetic mappings (aes) that link variables to visual properties such as x position, y position, colour, shape and size; geometric objects (geoms) such as points, lines, bars, boxplots and smooths; statistical transformations; scales; coordinate systems; and facets that split the plot into panels by a variable. A plot is built in layers joined with +: ggplot(trial, aes(x = nitrogen, y = yield, colour = variety)) + geom_point() + geom_smooth(method = "lm") + facet_wrap(~ site) plots yield against nitrogen with a fitted line for each variety and a panel for each site. Mapping a variable to an aesthetic inside aes() produces a legend automatically, while setting a fixed colour outside aes() does not; confusing the two is a common error. The grammar makes complex, consistent graphics from data in long form and is the reason tidy data matter for plotting.
In Python, matplotlib is the basic plotting library: a figure contains one or more axes, created with plt.subplots(), and plotting methods such as ax.plot, ax.scatter, ax.hist, ax.boxplot and ax.bar draw on them, with ax.set_xlabel and similar methods for labels; fig.savefig writes the file. Higher-level libraries build statistical graphics on top of it, and pandas data frames have a plot method for quick charts. Whatever the tool, good statistical graphics show the data, label axes with variables and units, avoid distortion such as truncated bar axes, use colour schemes readable by people with colour-vision deficiency, show uncertainty (confidence intervals or individual points) rather than bare means, and are produced by code so that they update when the data change.
A function should do one clearly defined task, have a name that says what it does, take everything it needs as arguments, return a result rather than printing it or changing global variables, and be short enough to read at once. When the same calculation appears in a script three times, it should become a function, which removes copy-and-paste errors and makes correction possible in one place. Arguments should be checked at the start, with informative errors: a function that computes a standard error might stop with a clear message if n is less than two or x is not numeric. Documentation states what the function does, its arguments and their types, what it returns, and an example; in Python this is the docstring, the string literal immediately after the def line, and in R packages it is written as structured comments that generate help pages. Functions that take data and options as arguments, rather than hard-coding file names and column names, can be reused across trials and seasons.
When a set of functions is used across projects, it should be organised as a package. An R package is a directory with a DESCRIPTION file (name, version, author, licence, dependencies), a NAMESPACE file listing exported functions, an R/ folder holding the code, a man/ folder for help pages (often generated from comments), and optionally tests, data and vignettes; development tools build, document, test and check packages, and R CMD check verifies that a package meets standard requirements. In Python, a module is a single .py file, and a package is a directory of modules, traditionally marked by an __init__.py file, with a configuration file describing the project and its dependencies; packages are installed with the standard package installer. In both languages, declaring dependencies and their versions, and using isolated project environments, prevents an analysis from breaking when another project updates a shared library. Even an unpublished internal package for a research group gives consistent methods, tested code and documentation that survive staff changes.
Debugging is the systematic search for the cause of a program's failure. The first step is to read the error message carefully and find where it arose: traceback() in R and the traceback printed by Python show the chain of function calls. Next, reproduce the problem with the smallest possible example, which often reveals the cause. Interactive debuggers let the programmer pause inside a function and inspect variables: browser() and debug() in R, and the built-in debugger in Python, entered with breakpoint(). Printing intermediate values is crude but effective. Common bugs in statistical code include off-by-one errors in indexing (especially when moving between 1-based R and 0-based Python), silent recycling of vectors of unequal length in R, factors converted to their integer codes, missing values propagating or being silently dropped, integer division where real division was intended, modifying a shared object by accident, and sorting one column of a data set without the others.
Tests are code that checks that other code gives the right answers. Unit tests check individual functions on small inputs whose answers are known: a variance function should return 2.5 for the values 1, 2, 3, 4 and 5, and should handle missing values and invalid input as documented. Testing frameworks such as testthat in R and pytest in Python run collections of tests automatically and report failures. Regression tests save the output of a verified analysis and check that later versions reproduce it. Assertions inside code, such as stopifnot(all(df$weight > 0)) in R or assert in Python, stop the program immediately when an assumption fails, rather than letting a wrong value propagate. For statistical methods, simulation tests generate data from a known model and check that estimates recover the true parameters on average. Tests are written as code is developed, and are rerun whenever it changes.
Code is read far more often than it is written, so consistent style matters. Style guides, such as PEP 8 for Python and the widely used tidyverse style guide for R, recommend meaningful names (snake_case for variables and functions in both), consistent indentation (four spaces in Python, two in that R guide), spaces around operators, lines of limited length (79 characters in PEP 8), one statement per line, and comments that explain why rather than what. Avoid magic numbers by naming constants (plot_area_m2 = 12.5), avoid deeply nested code by splitting functions, and avoid commented-out dead code. Automatic formatters and linters check and enforce style. Version control, covered with reproducible research in another subject, records every change so that errors can be traced to the change that introduced them.
A SAS program is a sequence of steps, each ending with RUN; (or QUIT; for some procedures), and every statement ends with a semicolon, whose omission is the commonest beginner's error. DATA steps create or modify data sets; PROC (procedure) steps analyse, summarise or report them. A LIBNAME statement associates a short library name with a folder, so that permanent data sets are named library.dataset, while one-level names refer to the temporary WORK library, deleted at the end of the session. In a DATA step, INFILE and INPUT read raw data, SET reads an existing data set, and assignment statements create new variables. The DATA step executes as an implicit loop: for each observation it reads values into the program data vector, executes the statements, and writes the observation to the output data set at the end of the step, unless an explicit OUTPUT statement controls when observations are written. Variables are reset to missing at the start of each iteration unless they are retained with RETAIN or come from SET. Missing numeric values are shown as a period (.), and in comparisons a missing numeric value is smaller than any number, so IF yield < 2 THEN … also catches missing yields unless they are excluded explicitly.
IF–THEN–ELSE statements create derived variables and subsetting IF or WHERE statements select observations. MERGE with a BY statement combines data sets by key variables, and requires both data sets to be sorted by those variables first with PROC SORT; without the BY statement, MERGE combines observations by position, a dangerous mistake. Functions such as MEAN, SUM, LOG and INPUT (character to numeric conversion) are available, and the SUM function, unlike the + operator, ignores missing values. FIRST. and LAST. variables, created when a BY statement is used in a DATA step, mark the first and last observation of each group and allow group-wise calculations.
Common procedures include PROC PRINT and PROC CONTENTS to inspect data; PROC MEANS (or SUMMARY) for descriptive statistics, by groups with CLASS or BY; PROC FREQ for frequency tables and chi-square tests; PROC UNIVARIATE for distributions and normality checks; PROC REG for regression; PROC GLM for analysis of variance of designed experiments, with CLASS statements for factors, a MODEL statement such as MODEL yield = block variety; and LSMEANS for adjusted treatment means and comparisons; PROC MIXED and PROC GLIMMIX for mixed models, with RANDOM statements for random effects such as blocks or sites and REPEATED for correlated measurements; PROC NLIN for non-linear regression such as growth curves; and PROC SURVEYMEANS and related procedures for design-based analysis of complex surveys. Output Delivery System (ODS) statements send results to files and save tables as data sets. For a randomised block experiment, a typical step is PROC GLM; CLASS block variety; MODEL yield = block variety; LSMEANS variety / PDIFF; RUN;. Because SAS is long established in agricultural research and in regulatory submissions, older analyses and many institutional pipelines are written in it, and statisticians often need to read and translate SAS code.
A SAS program writes a log as well as output, and the log must be read after every run. Notes about uninitialised variables, numeric-to-character conversions, missing values generated by arithmetic, or merge statements in which more than one data set has repeats of BY values point to logic errors even when the program appears to finish successfully; warnings and errors are flagged explicitly. The number of observations read and written by each step, reported in the log, should match expectations: a DATA step that unexpectedly writes fewer observations than it read usually contains a subsetting IF or a WHERE condition that excluded more than intended. In procedure output, the class level information and the number of observations read and used confirm that factors and missing values were handled as intended before any test statistic is interpreted.
C is a compiled, statically typed language: a program is written in source files, compiled into machine code and then run, and every variable must be declared with its type before use. Execution starts in the function main. Basic types include char (a single character, one byte), int (an integer, commonly four bytes on current systems, though the standard only sets minimum ranges), float (single-precision floating point) and double (double precision, the usual choice for statistical calculations), with qualifiers such as long and unsigned. Integer arithmetic truncates: 7/2 is 3 when both operands are integers, while 7/2.0 is 3.5, a frequent source of error when computing means; 7 % 2 is 1. Statements end with semicolons, blocks are enclosed in braces, and control structures are if–else, switch, for, while and do–while. The loop for (i = 0; i < 10; i += 3) runs four times, with i equal to 0, 3, 6 and 9. Input and output use library functions such as printf with format specifiers (%d for int, %f for double, %s for strings) and scanf, which needs the address of the variable, as in scanf("%d", &n).
An array holds a fixed number of elements of one type in contiguous memory, indexed from 0: int a[5] has valid indices 0 to 4, and C does not check bounds, so writing to a[5] silently corrupts memory, one of the most common and dangerous C bugs. Strings are arrays of characters terminated by the null character '\0'. A pointer is a variable holding a memory address: the address-of operator & gives the address of a variable, and the dereference operator * gives the value at an address, so if p = &x then *p is x. Array names act as pointers to their first element, and a[i] is equivalent to *(a + i). Passing a pointer to a function lets the function change the caller's variable, since C passes arguments by value. Memory for data whose size is known only at run time is allocated with malloc and must be released with free; forgetting to free causes memory leaks, and using memory after freeing it causes unpredictable errors.
Files are opened with fopen, which returns a pointer to a FILE structure (or NULL if opening fails, which must be checked), read and written with fscanf, fprintf, fgets, fread and fwrite, and closed with fclose. A program that reads plot yields from a text file, computes their mean and variance and writes a summary is a typical first exercise. Few statisticians write whole analyses in C, but it remains valuable: it shows how data are stored in memory and why types and precision matter, it is the language in which much of R's and Python's numerical core is written, and compiled code in C or C++ is called from R and Python to speed up slow loops in simulation and resampling.
A student uses this subject from the first practical class: reading a trial spreadsheet into R or Python, checking types and missing codes, reshaping it to long form, computing summaries by treatment with grouped operations, drawing a ggplot or matplotlib figure, and writing a short function that is then reused for every response variable. In examinations, typical tasks are predicting the output of a few lines of code (indexing, slicing, recycling, integer division, loops), choosing an efficient algorithm, writing pseudocode, and reading a SAS DATA step or a short C program.
A statistician five years into practice writes scripts that clean and analyse each season's trial or survey data, maintains a small set of tested functions or an internal package used by the team, translates older SAS programs into R or Python, automates reports, and debugs colleagues' code when results look wrong. Typical decisions are how to structure a project so that it can be rerun, which checks to build into data import, and when a slow loop justifies rewriting or calling compiled code.
A senior statistician with twenty-five years of experience sets programming standards for a unit: preferred languages, style guides, code review, testing, package structure and documentation. They decide when to move institutional pipelines from one language to another, ensure that critical analyses, such as official crop estimates or registration trials, are reproducible and independently checked, train staff and supervise the development of shared tools used across programmes. They also judge which tasks justify a general-purpose language and which are better served by established statistical software.
A researcher with a PhD and long experience writes software that implements new statistical methods, often released as packages for others to use, with careful attention to correctness, numerical stability, efficiency and documentation. Such researchers combine high-level languages with compiled code for speed, design simulation studies in code, review the software accompanying papers, and teach statistical programming to postgraduate students. They also mentor research teams in moving from ad hoc scripts to tested, documented, version-controlled code. They decide which research code deserves the investment of becoming a maintained package.
This manual teaches concepts and representative syntax; it is not a complete reference for any language, and functions, defaults and package interfaces change between versions, so current documentation should be checked. It does not cover object-oriented programming in depth, parallel computing, databases, web programming or other statistical languages and environments. C is introduced only to the level of data types, arrays, pointers and files.
Original practice questions written to the syllabus and pattern of the exams named in each question's examTags. They are not past papers and do not reproduce any board's questions.
These notes are free to read here, with no account.
What the Laisha app adds:
Free for ten days, then a subscription. We would rather write that here than have you find it later.