Agricultural Statistics & Data Science · Foundations · also EDA, Data Visualisation, Descriptive Statistics
Looking at data before modelling it: summaries, robust statistics, graphs that show distributions and relationships honestly, outlier detection, and the tidy-data and reporting habits that make an analysis checkable. Deliberately practical and software-based.
In the exams: Not an explicit unit in the catalogued exams; programme course and the daily practice of every role in the ladder. GATE DA data handling (present)
Exploratory data analysis (EDA) is the work of looking at data before modelling them: checking what was recorded, summarising it, plotting it in several ways, finding errors and oddities, and choosing a scale and a model that fit. John Tukey, who named the field in the 1970s, called it detective work, to be done before the confirmatory work of tests and confidence intervals. Most mistakes in published agricultural analyses are not wrong formulas; they are a mistyped yield nobody looked at, a plot number entered twice, a variance that grows with the mean, a field trend nobody mapped, or a chart that exaggerates a small difference.
This manual covers the vocabulary of data (types and measurement scales), the tidy organisation of tables that makes analysis possible, numerical summaries that resist outliers (the median, quartiles, the median absolute deviation and trimmed means), the graphs that show a distribution (histograms, density estimates and quantile-quantile plots) and a relationship (scatterplots, smoothers and scatterplot matrices), the diagnostic graphics of designed experiments, the detection of outliers and influential observations, transformations, the science of graphical perception that tells us which charts are read accurately, and the design of tables for reports.
The subject is rarely a named unit in entrance examinations, but its pieces appear everywhere: data presentation and summary measures in statistical-officer papers, data handling and visualisation in data-science examinations, and descriptive analysis in software certifications. Above all it is the daily practice of every statistician, from a student cleaning thesis data to a national office publishing crop estimates. The manual is software-neutral: every method described is available in the common open-source and commercial statistical packages, and the reader should practise each one on real data.
A data set records variables (characteristics) on units (the things measured: plots, animals, farms, households). The first EDA question is always what one row represents and what each column means, including its unit of measurement. Many analysis errors start with a confusion here, such as treating plot-level and plant-level records as if they were the same unit.
Variables are categorical (qualitative) or numerical (quantitative). Numerical variables are discrete (counts: eggs per hen, lesions per leaf) or continuous (measurements: grain weight, soil pH). The distinction matters for graphs (bar charts for counts with few values, histograms for continuous data) and for models (Poisson or binomial models for counts and proportions, normal models for continuous measurements).
S. S. Stevens classified measurement into four scales. A nominal scale only names categories with no order: variety, soil series, district. An ordinal scale has ordered categories but unequal or unknown gaps: disease scores on a 0-9 scale, farmer satisfaction rated from very poor to very good, maturity groups. An interval scale has equal steps but an arbitrary zero: temperature in degrees Celsius, calendar dates. A ratio scale has equal steps and a true zero, so ratios are meaningful: yield, weight, rainfall, temperature in kelvin. A field 20 °C warm is not twice as warm as one at 10 °C; a plot yielding 4 t/ha does yield twice as much as one yielding 2 t/ha.
The scale limits the meaningful summaries. Nominal data allow counts, proportions and the mode. Ordinal data add the median, percentiles and rank methods. Interval data add the mean and standard deviation. Ratio data add the geometric mean and the coefficient of variation. Averaging ordinal disease scores and reporting a mean of 4.37 with a standard error treats the gaps between scores as equal, which they usually are not; it is common practice, but the reader must know it is an approximation.
Many agricultural variables are derived: yield per hectare from plot weight and plot area, percentage germination from counts, feed conversion ratio from feed intake and weight gain, a disease index from scores. A derived variable inherits the problems of its parts. A ratio of two noisy quantities can be very skewed; a percentage computed from different denominators (5 seeds in one dish, 100 in another) carries very different precision; and a moisture-corrected yield depends on the moisture reading as well as the weight. EDA keeps the raw components and checks them separately before trusting the derived value.
Missing values need explicit codes and an explanation. A plot destroyed by cattle, an animal that died, a household that refused and a value below the detection limit are four different kinds of missing, with different consequences for analysis. Coding them all as zero, or as a number such as 999 or −99 that is later read as data, is one of the commonest errors in agricultural data sets. Missing values should be stored as the software's missing-value code with a separate column recording the reason.
Simple checks catch most errors before analysis: the number of rows against the design (a trial with 8 treatments in 4 blocks must have 32 plots); counts of each factor level; minimum and maximum of every numerical variable against physically possible ranges (no negative weights, no germination above 100%, no pH of 45); duplicated identifiers; and dates in order. A frequency table of every categorical variable reveals spelling variants ('Maize', 'maize', 'MAIZE ') that software treats as different levels. These checks are recorded in a script so that they can be re-run when data are corrected.
Hadley Wickham's tidy-data principles, published in 2014, state formally what good data managers had long practised. A data set is tidy when each variable forms a column, each observation forms a row, and each type of observational unit forms a table. Tidy data can be filtered, summarised, plotted and modelled with general tools because every analysis function expects one variable per column.
Agricultural spreadsheets are rarely tidy as received. Five common forms of messiness recur. Column headers are values, not variable names: a table with columns '2019', '2020', '2021' holding yields has a hidden variable, year. Several variables are stored in one column: a column 'N60_rep2' combines nitrogen dose and replicate. Variables are stored in both rows and columns: a weather sheet with days down the side and 'max' and 'min' temperature in alternating rows. Several types of observational unit share one table: farm-level information (farm size, district) repeated on every plot row, so a correction must be made in many places. And one type of unit is spread across several tables: one sheet per location in a multi-location trial.
Each form has a standard repair. Gathering (pivoting longer) turns value-headed columns into two columns, a key and a value: year and yield. Separating splits a combined column into its variables. Spreading (pivoting wider) turns rows of 'max' and 'min' into two columns. Normalising moves farm-level variables into their own table, joined to the plot table by a farm identifier. Binding stacks the per-location sheets into one table with a location column.
The long (tidy) format is not always the format for display: a reader may prefer a wide table with varieties in rows and locations in columns. The rule is to store and analyse in tidy form and to reshape for presentation at the end.
When data are kept in several tidy tables, each table needs a key: a variable or combination of variables that identifies each row uniquely. In a trial, plot number or the combination of location, block and treatment is the key of the plot table; farm identifier is the key of the farm table. Before joining two tables, check that the key is unique where it should be. A duplicated key in the table being joined multiplies rows silently, which is how a 200-farm survey becomes 214 rows after a join.
Joins combine tables. An inner join keeps rows whose key appears in both tables; a left join keeps all rows of the first table and fills missing matches with missing values; an anti-join lists rows in one table with no match in the other, which is the quickest way to find plots with yield but no treatment record, or households with a questionnaire but no sampling weight.
A tidy workflow is also a reproducible one. Raw data files are kept read-only and never edited by hand; all corrections are made by a script that documents each change and its reason; the cleaned data are written out by the script. Analyses and figures are generated from the cleaned data by further scripts, so that a corrected value propagates to every table and figure when the scripts are re-run. A data dictionary lists each variable with its definition, unit, permissible range and missing-value codes. These habits make an analysis checkable by a colleague and defensible years later, which matters for variety-release decisions, official statistics and published papers alike.
Spreadsheet habits that break tidiness should be avoided at data entry: merged cells, colour used as data (a red cell meaning 'lodged'), several header rows, totals mixed with data rows, and comments typed into numeric cells. Data entry forms with validation rules prevent many of these at the source.
Tukey's five-number summary is the minimum, lower quartile Q₁, median, upper quartile Q₃ and maximum. It describes the centre (median), spread (interquartile range, IQR = Q₃ − Q₁) and skewness (unequal distances from the median to the quartiles and extremes) of a distribution without assuming any shape.
Quartiles are not computed the same way by all software. Hyndman and Fan (1996) catalogued nine definitions that differ in how they interpolate between order statistics; for large samples they agree closely, but for small samples Q₁ of the same ten numbers can differ between packages. Tukey's hinges, the medians of the lower and upper halves of the data, are one common choice. A report should say which definition was used when small-sample quartiles matter.
The box plot (box-and-whisker plot) draws a box from Q₁ to Q₃ with a line at the median. In Tukey's standard version, inner fences are placed at Q₁ − 1.5 IQR and Q₃ + 1.5 IQR; whiskers extend from the box to the most extreme observations within the fences; and points beyond the fences are plotted individually as potential outliers. Points beyond 3 IQR (the outer fences) are sometimes marked as far out. For normal data about 0.7% of observations fall beyond the inner fences, so in a sample of 1,000 a handful of 'outliers' is expected and not alarming.
Box plots are best for comparing several groups side by side: yields of ten varieties, milk yield by parity, soil carbon by land use. They hide the shape inside the box (a bimodal distribution can have a normal-looking box) and with small groups (fewer than about ten observations) the raw points, shown as a strip or dot plot, tell more. A common modern practice overlays the jittered raw points on the box.
A statistic is robust if it is little affected by a small number of wild values. The breakdown point measures this: the largest fraction of the data that can be made arbitrarily large without making the statistic arbitrarily large. The mean has breakdown point 0 (one bad value can move it anywhere); the median has 50%, the highest possible.
Robust measures of location include the median; the trimmed mean, which discards a fixed fraction (for example 10% or 20%) of observations from each end and averages the rest; and the Winsorised mean, which replaces those extreme observations by the most extreme retained values instead of discarding them. A 10% trimmed mean has a 10% breakdown point. Trimmed means keep much of the efficiency of the mean for normal data while resisting outliers, which is why some sports and sensory panels drop the highest and lowest judges' scores.
Robust measures of spread include the interquartile range (breakdown 25%) and the median absolute deviation, MAD = median of |xᵢ − median|, with breakdown 50%. For normal data, the MAD is about 0.6745σ, so the scaled MAD, 1.4826 × MAD, estimates σ. Likewise IQR/1.349 estimates σ for normal data. Comparing the SD with the scaled MAD is a quick check: if the SD is much larger, a few extreme values are inflating it.
Example: yields of seven plots are 3.1, 3.4, 3.3, 3.6, 3.2, 3.5 and 8.9 t/ha, where 8.9 is a recording error for 3.9. The mean is 4.14 and the SD about 2.1, while the median is 3.4 and the MAD 0.2 (scaled MAD about 0.30). One wrong value has moved the mean by about 0.7 t/ha and multiplied the SD nearly eightfold, but barely touched the robust measures. Robust statistics are a tool for describing and for flagging problems; they do not replace finding out why 8.9 was recorded.
The histogram is the oldest graph of a distribution and still the first one to draw. Its appearance depends heavily on the bin width and the bin origin. Too few bins hide features such as bimodality or a spike at a rounding value; too many bins turn the histogram into noise. Several rules suggest a starting width for n observations. Sturges' rule gives k = 1 + log₂ n bins; it was derived for near-normal data and gives too few bins for large or skewed samples. Scott's rule gives width h = 3.49 s n^(−1/3), where s is the standard deviation. The Freedman-Diaconis rule gives h = 2 IQR n^(−1/3), which is robust to outliers because it uses the IQR. All three are starting points; an analyst should try several widths.
Histograms reveal data-entry and measurement features that summary statistics miss: heaping at round numbers (weights recorded to the nearest 5 kg, ages heaped at multiples of 5 in household surveys), a spike at zero (non-adopters in a survey of fertiliser use), truncation at a detection limit, and gaps from a change of instrument. Each needs explanation before modelling.
When groups are compared, histograms are stacked vertically on a common x-axis so that shifts in location and spread can be seen; overlaid histograms of several groups quickly become unreadable. For discrete counts with few values, a bar chart of each value's frequency (a spike plot) is better than a histogram with arbitrary bins.
A kernel density estimate (KDE) is a smooth version of the histogram that does not depend on a bin origin. At each point x, it averages small bumps (kernels) centred on every observation: f̂(x) = (1/(nh)) Σ K((x − xᵢ)/h), where K is a kernel function (commonly the Gaussian density) and h is the bandwidth, which controls smoothness. The choice of kernel matters little; the choice of bandwidth matters a great deal. A small h gives a wiggly estimate that follows noise; a large h oversmooths and can merge two modes into one.
Silverman's rule of thumb, h = 0.9 min(s, IQR/1.34) n^(−1/5), is a common default for the Gaussian kernel; it works well for roughly normal data and oversmooths multimodal data. Data-based selectors such as Sheather-Jones plug-in or cross-validation adapt better. Because the bandwidth falls only as n^(−1/5), density estimates need large samples to show fine detail.
Kernel estimates leak probability past natural boundaries: a KDE of percentages or of strictly positive rainfall will put density below zero unless a boundary correction or a transformation is used. Density plots of several groups can be overlaid more readably than histograms, and violin plots place mirrored density estimates side by side, combining the comparison of a box plot with a view of the shape.
A quantile-quantile (Q-Q) plot compares the quantiles of a sample with those of a reference distribution. In a normal Q-Q plot, the ordered observations x₍ᵢ₎ are plotted against the standard normal quantiles at plotting positions such as (i − 0.5)/n (other positions, such as (i − 3/8)/(n + 1/4), are also in use). If the data are normal, the points lie close to a straight line whose intercept estimates the mean and slope the standard deviation.
Departures have characteristic shapes. Points curving upward at the right end and lying above the line at both ends in a convex (bowl) pattern indicate right (positive) skew; a concave (arch) pattern indicates left skew. An S-shape in which the lower tail falls below the line and the upper tail rises above it indicates heavier tails than normal; the reverse S indicates lighter tails. Isolated points far off the line at either end are potential outliers; steps or horizontal runs reveal rounding and ties.
Q-Q plots are more informative than formal normality tests. With small samples, tests such as Shapiro-Wilk have little power and pass clearly skewed data; with large samples they reject trivial departures that do not affect the analysis. In ANOVA and regression the Q-Q plot is drawn for the residuals, not the raw observations, because the model assumes normal errors and the raw data of different treatments have different means. Two-sample Q-Q plots, quantiles of one group against quantiles of another, show whether two distributions differ in location only (points parallel to the identity line), in spread (a different slope), or in shape (a curve).
The scatterplot is the basic graph of a relationship between two numerical variables, with the explanatory variable on the horizontal axis and the response on the vertical. Reading it means asking four questions: what is the form of the relation (linear, curved, threshold, none); its direction; its strength (how tightly points follow the form); and whether there are unusual points or groups.
Anscombe's quartet (1973) is the standard warning against summarising a relationship by numbers alone. Four small data sets share the same means (x̄ = 9, ȳ = 7.5), variances, correlation (about 0.816) and least-squares line (ŷ = 3 + 0.5x), yet their scatterplots show a clean linear relation, a smooth curve, a perfect line with one outlier, and a vertical cluster with one high-leverage point that alone creates the slope. Only the graph shows which summary is appropriate.
With many points, overplotting hides the density of the data. Remedies include smaller or semi-transparent points, jittering (adding small random noise to discrete values so that tied points separate), hexagonal binning (counting points in hexagonal cells and shading by count) and two-dimensional density contours. Jitter must be described in the caption so that readers do not mistake it for real variation.
Scales matter. A log scale on one or both axes turns multiplicative relations into linear ones and spreads out data concentrated at low values, such as farm sizes or pest densities. Grouping variables can be shown by symbol or colour (variety, season, irrigated or rainfed) or by small multiples: a grid of panels with common axes, one per group, which is usually clearer than many colours in one panel. A pattern that holds within each group but reverses when groups are pooled, Simpson's paradox, is found only by looking within groups.
A smoother draws a curve through a scatterplot that follows the trend without assuming a formula. LOESS (or LOWESS, locally weighted scatterplot smoothing), introduced by William Cleveland in 1979, fits at each target point x a low-degree polynomial (degree 1 or 2) by weighted least squares, using only the nearest fraction of the data, the span, and weighting points by their distance from x with the tricube weight function, w(u) = (1 − |u|³)³ for |u| < 1. The fitted values at all points form the curve.
The span controls smoothness. A small span (for example 0.2, using 20% of the points for each local fit) follows local wiggles and noise; a large span (0.75 is a common default, 1.0 the maximum) gives a smoother curve close to a global polynomial. A robust version iterates: after a first fit, points with large residuals are down-weighted with the bisquare function and the curve refitted, so a few outliers cannot drag it.
Smoothers are used to see the form of a relation before choosing a model (is the fertiliser response levelling off; is the relation between rainfall and yield humped?), to check a fitted model by smoothing its residuals against each predictor (a flat smooth near zero supports the model; a curve shows missed non-linearity), and to show trends over time. Other smoothers include running means and medians, smoothing splines and generalised additive models. A smoother is descriptive: the curve at the ends of the range rests on few points and should not be extrapolated.
When there are several numerical variables, a scatterplot matrix (SPLOM) shows every pair in a grid: the panel in row i and column j plots variable i against variable j, and the diagonal often shows each variable's histogram or density. In a breeding trial with yield, plant height, days to flowering, grain weight and tiller number, a SPLOM shows at once which traits are related, whether the relations are linear, and whether a few lines are outliers in several traits together.
The SPLOM has limits. It shows only pairwise relations; a relation that appears only when a third variable is held fixed is not visible. With more than about eight to ten variables the panels become too small to read, and a correlation heatmap (the correlation matrix shown as coloured cells, ordered so that related variables sit together) or a principal-component biplot gives a better overview. A correlation matrix by itself, however, repeats Anscombe's warning: each coefficient can hide curvature or outliers that the scatterplot would reveal.
Conditioning plots (coplots) and small multiples address the third-variable problem: the scatterplot of yield against nitrogen is drawn separately for each level or range of rainfall, revealing whether the response depends on water. Interactive graphics, where selecting points in one panel highlights the same units in all others (brushing), extend the idea to many dimensions.
The results of a designed experiment are best shown by graphs that reflect the design. For a single factor, plot the raw observations for each treatment (a dot or strip plot) with the treatment means and an interval marked, not only a bar of the mean. The widely used bar chart of means with an error bar on top, sometimes called a dynamite plot, hides the number of observations, their spread and any outliers, and the bar suggests that the area from zero matters when only the top does.
The interval drawn must be named in the caption: the standard deviation (describing variation among plots), the standard error of a mean, a confidence interval, or the least significant difference. These differ by factors of √r or more, and the reader cannot interpret overlap without knowing which is shown. Overlapping standard-error bars do not prove that two means are not significantly different, and non-overlapping 95% confidence intervals are a stricter criterion than a 5% test of the difference.
For two or more factors, the interaction plot is the key graph. The mean response is plotted against the levels of one factor, with a separate line for each level of the other (for example yield against nitrogen dose, one line per variety). Parallel lines indicate no interaction: the effect of nitrogen is the same for every variety. Non-parallel lines indicate interaction; lines that cross indicate a crossover (qualitative) interaction, in which the ranking of varieties changes with nitrogen, the case that matters most for recommendations. Because sampling error makes lines from real data never exactly parallel, the plot is read alongside the ANOVA test of interaction and the standard errors.
For a quantitative factor such as dose or time, the levels should be plotted at their true spacing on a numeric axis, so that the shape of the response can be seen; plotting doses of 0, 30, 60 and 120 kg/ha at equal spacing distorts it.
After fitting an ANOVA or regression model, residual plots check its assumptions. The residual is the observed value minus the fitted value; standardised or studentised residuals divide it by an estimate of its standard deviation so that values beyond about ±2 to ±3 stand out.
Residuals plotted against fitted values should form a horizontal band of constant width centred on zero. A funnel shape, spread increasing with the fitted value, shows variance increasing with the mean, common for counts, weights and yields across very different environments, and points towards a transformation or a model with non-constant variance. A curved pattern shows a missed non-linear or interaction term. A normal Q-Q plot of residuals checks normality. Residuals against the order of measurement reveal drift in an instrument or a change of observer.
In field trials, residuals should also be plotted against the field layout: a map of plots coloured by residual, or residuals against row and column position. Patches of positive and negative residuals show spatial trend or patchiness that blocking did not remove; such fields benefit from spatial analysis (row and column effects or correlated errors) and from better blocking in the next season. A uniformity trial, in which the whole field is sown with one variety and harvested in small units, can be mapped the same way to plan block shape and plot size.
Animal and laboratory experiments have their own layout plots: residuals by pen, cage position, incubator shelf or batch of reagents. The principle is the same: plot residuals against every variable that describes how the experiment was physically run, because any pattern there is a source of error that the design did not control.
An outlier is an observation that lies far from the bulk of the data, or far from the pattern followed by the rest. Outliers have three possible origins: errors (a misplaced decimal point, a unit slip between kilograms and grams, a transposed plot number), genuine but unusual units (a plot on an old manure heap, a cow with an undiagnosed disease), and evidence that the model is wrong (a distribution that is really skewed or heavy-tailed). The analyst's job is to find out which.
Univariate screening rules include Tukey's fences (beyond Q₁ − 1.5 IQR or Q₃ + 1.5 IQR) and z-scores, (x − x̄)/s, with |z| > 3 a common flag. The z-score rule suffers from masking: an outlier inflates the mean and SD used to compute its own z-score, so a single wild value in a small sample may never reach |z| = 3, and several outliers can hide one another. Indeed, in a sample of n, no z-score can exceed (n − 1)/√n, so in a sample of 5 the largest possible |z| is about 1.79. Robust z-scores avoid this: the modified z-score of Iglewicz and Hoaglin, Mᵢ = 0.6745 (xᵢ − median)/MAD, flags |Mᵢ| > 3.5.
Context-based checks are often more useful than rules: yields above the biological potential of the crop in that environment, a calf weaning weight below its birth weight, a household with more members than recorded adults and children combined. In multivariate data an observation can be unremarkable on each variable separately but unusual in combination (a tall plant with very low biomass), which pairwise scatterplots or multivariate distances such as the Mahalanobis distance can reveal.
In regression, an outlier in y (a large residual) is different from an unusual x value. Leverage measures how far an observation's x values are from the centre of the x values; it is the diagonal element hᵢᵢ of the hat matrix, which lies between 1/n and 1, with average p/n where p is the number of parameters. A common flag is hᵢᵢ > 2p/n. A high-leverage point has the potential to pull the fitted line towards itself.
An observation is influential if deleting it changes the fitted model substantially. Cook's distance Dᵢ combines residual and leverage to measure the change in all fitted values when observation i is removed. Rules of thumb flag Dᵢ > 1, or Dᵢ > 4/n in larger samples; different texts give different cut-offs, so the values are better compared with one another than with a fixed line. DFFITS measures the change in the observation's own fitted value, and DFBETAS the change in each coefficient. Externally studentised residuals, which use an error variance estimated without observation i, avoid masking of the observation's own residual.
A point can have a large residual but little influence (an outlier in y near the centre of x) or high leverage but a small residual (an extreme x that fits the trend), and only the combination of both makes a point strongly influential. In Anscombe's fourth data set, one observation at a distant x determines the entire slope.
What to do: first check the record against field sheets and lab notebooks, and correct demonstrable errors. Never delete an observation simply because it is extreme or because removing it makes the result significant. If a genuine observation is influential, report the analysis with and without it, use a robust method, or reconsider the model (a transformation or a curve may make it fit). Every exclusion must be reported with its reason so that the reader can judge.
A transformation re-expresses a variable on another scale: y′ = log y, √y, 1/y and so on. EDA uses transformations for three related purposes: to make a skewed distribution more symmetric, to make the spread of groups more equal (variance stabilisation), and to make a curved relation straight or an interaction disappear (additivity). Often one transformation serves all three, because skewness, a variance that rises with the mean and multiplicative effects tend to occur together.
Tukey's ladder of powers orders the common transformations: …, y⁻¹, y^(−1/2), log y, y^(1/2), y, y², y³, …. Moving down the ladder (towards log and reciprocal) pulls in a long right tail and reduces right skew; moving up the ladder reduces left skew. The logarithm takes the place of the zero power. The further a transformation is from 1, the stronger its effect.
A diagnostic connects the choice to the data. If the standard deviation of groups is proportional to a power of their mean, s ∝ mᵇ, then the power transformation y^(1 − b) stabilises the variance, with b = 1 giving the logarithm. A plot of log(group SD) against log(group mean) estimates b as its slope: a slope near 0.5 suggests the square root (Poisson-like counts), near 1 the logarithm, near 2 the reciprocal.
The Box-Cox family generalises this: y(λ) = (y^λ − 1)/λ for λ ≠ 0 and log y for λ = 0, defined for positive y. The value of λ is chosen by maximum likelihood, and a convenient nearby value (−1, −0.5, 0, 0.5, 1) is usually used for interpretability. Box-Cox requires positive data; a constant is added if there are zeros, and the choice of constant affects the result.
Three transformations are standard in agricultural research. The logarithm suits positive data whose standard deviation is proportional to the mean, with effects that act multiplicatively: insect populations, bacterial counts, prices, biomass across very different environments. On the log scale, differences become ratios: a difference of 0.693 in natural logs is a doubling. Where zeros occur, log(y + 1) is common, but the constant is arbitrary and matters when many values are near zero.
The square root suits counts of rare, randomly scattered events (Poisson-like counts, where the variance equals the mean): weeds per quadrat, lesions per leaf. √(y + 0.5) or √(y + 3/8) is used when there are zeros or small counts.
The angular (arcsine square root) transformation, arcsin √p, suits proportions from binomial counts, such as germination or mortality percentages, whose variance p(1 − p)/n is largest near 50% and shrinks towards 0% and 100%. It is most needed when proportions span a wide range including values near 0 or 1; for proportions all between about 30% and 70% it changes little. Modern practice increasingly fits a generalised linear model (logistic regression for binomial counts, Poisson or negative binomial regression for counts) instead of transforming, which models the mean-variance relation directly and avoids back-transformation problems.
Results analysed on a transformed scale must be reported carefully. The back-transformed mean of log data is the geometric mean, not the arithmetic mean, and is smaller for skewed data; confidence limits back-transform to asymmetric limits around it. Standard errors cannot be back-transformed directly. Tables should state the scale of analysis and give back-transformed means with their interpretation.
Cleveland and McGill's experiments in the 1980s ranked the elementary perceptual tasks by how accurately people decode quantities from them. From most to least accurate: position along a common scale; position along identical but non-aligned scales; length; direction and angle; area; volume and curvature; and shading or colour saturation. Good charts encode the most important comparison by position on a common scale.
This ranking explains several standard recommendations. Dot plots and bar charts (position or length on a common scale) are read more accurately than pie charts (angle and area). A pie chart with many slices of similar size cannot be read without labels, and comparing two pies is harder still. Stacked bar charts are accurate only for the bottom segment and the total; the middle segments have no common baseline. Bubble charts encode by area, which viewers underestimate. Colour shading in maps (choropleths) shows broad patterns but not precise values.
Order matters as much as encoding. Categories with no natural order (varieties, districts, crops) should be sorted by the value being shown, not alphabetically, so that the ranking can be read at a glance. Horizontal dot or bar charts give room for long labels. Juxtaposition (small multiples on common scales) supports comparison better than superposition of many lines in different colours.
Colour should carry information, not decoration. Qualitative palettes distinguish unordered categories; sequential palettes show ordered magnitude from light to dark; diverging palettes show departures in two directions from a meaningful midpoint (yield above and below the long-term average). Red-green colour vision deficiency affects a substantial minority of men in many populations, so red-green contrasts alone should be avoided, and charts should remain readable in greyscale.
A chart lies when the visual size of an effect differs from its numerical size. Edward Tufte's lie factor is the ratio of the size of an effect shown in the graphic to the size of the effect in the data; a value far from 1 indicates distortion. The commonest sources are bars that do not start at zero, areas or volumes scaled by a linear quantity, and perspective (three-dimensional) effects.
Bar charts encode values by length, so their axis must start at zero: a bar chart of yields 4.0 and 4.4 t/ha with the axis starting at 3.8 makes a 10% difference look threefold. Dot plots and line charts encode by position, so their axis need not include zero and should be scaled to show the variation of interest, with the range stated clearly. If a pictogram doubles both the height and width of a sack of grain to show doubled production, its area quadruples; in three dimensions, its volume rises eightfold.
Dual-axis charts, with two different vertical scales for two series, let the designer make any two series appear to move together or apart by choosing the scales, and should be replaced by two aligned panels. Time series should use a consistent interval on the time axis: irregular gaps (1990, 2000, 2010, 2015, 2016, 2017) plotted at equal spacing distort trends.
Tufte also argued for a high data-ink ratio: most of the ink should show data, not decoration. Heavy gridlines, background shading, three-dimensional bars, redundant legends and decorative images (chartjunk) can be removed without loss. The aim is clarity, not minimalism for its own sake: labels, units, sample sizes, the source of the data and the meaning of error bars are essential ink. A good figure can be understood from the figure and its caption alone.
Tables are the right choice when readers need exact values or will look up individual numbers; graphs are better for showing patterns and comparisons. Most reports need both: a graph for the message and a table for the record.
A. S. C. Ehrenberg's rules for readable tables remain the standard guidance. Round drastically, to two effective (varying) digits: yields of 3.4172, 3.6894 and 3.5013 t/ha are compared far more easily as 3.4, 3.7 and 3.5, and the extra digits are usually within the error anyway. Provide row and column averages as a visual focus. Put the numbers to be compared in columns rather than rows, because the eye compares figures down a column more easily. Order rows and columns by size (for example varieties by mean yield) rather than alphabetically, unless a standard order is expected. Use layout (spacing, grouping, a few horizontal rules) to guide the eye instead of heavy gridlines. And give a brief verbal summary of the main pattern in the text or caption.
A table must be self-contained. The title says what, where and when; column headings give the variable and unit (for example 'Grain yield (t/ha)'); footnotes define abbreviations, the scale of analysis if transformed, the number of replications, and the meaning of any symbols. Numbers in a column share decimal places and are right- or decimal-aligned. Totals are labelled as totals, and percentages state their base (percentage of what, out of how many).
Tables of experimental results should report estimates with their precision, not only significance. For a variety or treatment trial this usually means the treatment means, the standard error of a mean or of a difference with its degrees of freedom, the least significant difference where pairwise comparisons are made, and the coefficient of variation of the trial. The number of observations behind each mean must be clear, especially when replication is unequal or data are missing.
Mean-separation letters (means sharing a letter do not differ significantly) are common in agricultural journals. They compress a set of pairwise tests into a compact display but hide effect sizes, depend on the multiple-comparison procedure chosen, and are often misread as if means with different letters differed by a meaningful amount. Many journals now prefer intervals or contrasts planned before the experiment. If letters are used, the procedure and its significance level must be stated.
Survey tables report estimates with their standard errors or coefficients of variation (relative standard errors), confidence intervals, and unweighted sample sizes. Official statistics offices commonly suppress or flag estimates whose relative standard error exceeds a stated threshold, and suppress cells with very few respondents to protect confidentiality; the thresholds differ between offices, and the rule used should be stated in the table notes.
Reproducible reporting closes the loop of EDA: tables and figures generated by script from the cleaned data, with the code, software versions and random seeds archived, so that a reviewer can regenerate every number. Many journals and funders now require data and code to be deposited, and national statistics offices publish metadata describing how each table was produced.
A student with a thesis data set uses this subject in the first week of analysis: building a tidy table from field books, writing a cleaning script that checks plot counts and ranges, plotting every variable, looking at residuals after the first ANOVA and choosing a transformation for insect counts. In examinations the same student must compute quartiles and fences, state which average suits which scale, read Q-Q plots and interaction plots, and know why a bar chart must start at zero.
A biometrician or data analyst five years into work at a research institute, seed company or statistics office spends much of each week on EDA: receiving messy multi-location spreadsheets and reshaping them, running standard range and duplicate checks before every analysis, mapping residuals on field layouts, flagging influential trials in a series, and producing report figures and tables to a house style. Such a person also trains field and laboratory staff in data entry that avoids merged cells, colour coding and zeros for missing values.
A senior statistician with twenty-five years of experience sets the data standards of a programme or office: data dictionaries, validation rules at entry, a required set of diagnostic plots before any trial is accepted into a variety-testing series, rules for excluding outliers and reporting them, and a graphics and table style guide for publications and official releases. Such a person reviews the figures in reports before publication, removing misleading scales and unexplained error bars, and decides how corrections to published figures are issued and documented.
A professor or methods leader with a PhD and long experience teaches EDA as the foundation of statistical practice rather than an afterthought, develops reproducible-analysis curricula and open teaching data sets for agricultural students, and contributes to methodological standards such as reporting guidelines for field trials and official statistics. Such a person also studies how readers perceive graphs and tables, advises journals and statistics offices on presentation policy, and leads reviews when published results turn out to rest on unchecked data.
The manual describes methods but does not teach the syntax of any software package; readers should practise each method in the statistical environment they use. Interactive graphics, spatial and temporal visualisation, dashboards, multivariate projection methods beyond the scatterplot matrix, and formal robust regression estimators are mentioned only briefly. Cut-offs for flagging outliers and influential points (|z| > 3, 2p/n, Cook's distance 1 or 4/n) and thresholds for suppressing survey estimates differ between textbooks and institutions; the manual gives common versions and says so. The prevalence of colour-vision deficiency differs between populations and is not given as a single figure.
Original practice questions written to the syllabus and pattern of the exams named in each question's examTags. They are not past papers and do not reproduce any board's questions.
These notes are free to read here, with no account.
What the Laisha app adds:
Free for ten days, then a subscription. We would rather write that here than have you find it later.