Data validation

The rules by which the book argues with a record

An open list: what is checked, why it is an error, and what to do about it. Rules are stated in advance — like an exam syllabus, not a remark after submission.

77

rules in the registry

36

find a contradiction in the data

10

report that data is absent

31

draw attention without holding up confirmation

39

carry a numeric limit stated aloud

Three states, and “fix it” is not among them

A check never asks you to fix a record, and that is not politeness. By construction it cannot tell a correction from a fudge: a rule saying “yield of 45,000 kg is out of bounds” would, at the word “fix it”, invite someone to put 12,000 — the block lifts, the mark of the Association goes on, and the book records a number nobody ever measured. So a check names the state of the record, and the decision stays with people.

There are three states. “Data missing” — the fields are empty: no date of birth, no breed, not a single parent. There is nothing to fake here, and the only way on is to upload what is absent. “Data does not add up” — the data is there and contradicts itself: a calving before the birth, DNA that rules out the recorded sire, a blood share above one hundred per cent. Here editing the record is precisely the thing that must not be done without grounds. “Worth a look” — the rule has noticed something odd and may be wrong.

The difference between the second and the third is not one of strictness but of whether the rule itself can be wrong. “Parent younger than offspring” cannot be wrong — it is a contradiction. “Yield above twenty-five thousand” can be: such a cow is rare but possible, and forbidding her would mean vouching for what we do not know.

What happens to a record that has questions against it

Nothing is deleted and nothing is rolled back: the data stays in the book, and the farm sees it and edits it. What is held is something else — the confirmation of the Association, the very mark the record is kept for. Until a finding has been dealt with, the animal is not confirmed and does not count as pedigree by the book.

How it is lifted differs. “Data missing” is lifted by uploading what is absent — and by nothing else. “Data does not add up” is lifted by an explanation the expert accepts: a document, a laboratory certificate, a letter from the farm saying where the discrepancy came from. The expert may judge a finding immaterial and waive it — but the waiver is by name, for this record and this rule, and it stays in the history of the application.

The third outcome is lawful too: to accept that the animal does not count as pedigree. The book is under no obligation to confirm everything uploaded into it, and a record without confirmation is not a defect but an honest statement of where things stand.

What the book asks for besides consistency

The rules below check the data against itself. Apart from them the book looks at completeness: confirmation means not only “no discrepancies” but also “everything required has been submitted”. The list is short and the same for every application.

Without these a record is not confirmed:

The list

Every rule has a number, and the number is what to quote in a letter or in a conversation with support: rule 45. It reflects the place of the rule in the registry and shifts if a new rule is inserted in the middle; the permanent key is the code of the rule, which goes into exports and never changes.

worth a look: 31rules in the registry

Passport1 rule

Fields of the record itself: identifier, date of birth, breed, breed percentage.

1.5Ear tag repeated

worth a look

One tag number stands on several animals in the batch.

Tags are replaced and moved between animals, so a tag is not treated as unique. But two live animals under one tag on one farm are almost always a typing error.

Parentage3 rules

Parents, the pedigree and everything computed from it. An error here costs more than the rest: it does not spoil one record, it distorts the evaluation of every descendant.

2.5Sire left the herd long before conception

worth a look

The sire was disposed of before the offspring could have been conceived: more than a gestation period lies between the disposal and the birth.

Not an error in itself: frozen semen keeps for decades, and this is an ordinary case. But if the farm stores no semen, the link points to the wrong bull.

Limit: more than 270 days between the disposal of the sire and the birth of the offspring

2.11Breed percentage disagrees with the parents

worth a look

The breed percentage differs from the mean of the parental values by more than the tolerance.

The breed percentage of the offspring is the mean of the parental ones. A divergence means an error either in the percentage or in the parent itself; the second is far worse, because it spoils every other offspring of that parent as well. This is the only check whose severity depends on the size of the divergence: the marker on the left names the common case, while a large divergence arrives as one to be fixed.

Limit: over 12.5 percentage points is a warning, over 25 must be fixed

2.12Inbreeding disagrees with the pedigree

worth a look

The inbreeding coefficient entered on the record differs from the one computed from the pedigree.

A divergence is not always an error: our computation runs on the pedigree held in the book, while the farm may have computed on a fuller one. But it is always a question — on which pedigree.

Limit: over 1 percentage point

Reproduction7 rules

Calvings and inseminations — not one by one, but as a sequence in time.

3.2First calving too late

worth a look

The age at first calving is over forty-eight months.

Calving at four years is possible, merely expensive. More often it means that the earlier calvings were simply never recorded — and then what is wrong is not the age but the completeness of the data.

Limit: older than 48 months

3.6Gap in the calving numbers

worth a look

One or more numbers are missing from the run of calving numbers.

Either a calving was never recorded — and then every lifetime figure is incomplete — or the numbers were entered wrongly.

3.11Calf not born on the day of the calving

worth a look

The date of birth of the animal recorded as the calf does not match the date of the calving.

A calving and the birth of a calf are one fact recorded twice. A divergence means that either the wrong animal is marked as the calf, or one of the two dates has a typing error.

3.12Offspring counts disagree with one another

worth a look

The birth type, the counts of live heifer calves, bull calves and stillborn, and the calf records themselves state different numbers.

How many were born, the book knows three times over, and the three sources appeared at different times: the birth type was carried over from the previous system, the counts are entered by hand, and records are not created for every calf. A divergence means that one of the three is wrong, and which one is visible from the finding itself.

Limit: stillborn calves are not counted among the records: a record is created for a live calf. “Not stated” and “mixed multiple birth” are not cross-checked against anything

3.13Test-day recording outside the lactation

worth a look

A test-day recording is dated before the first calving of the cow or after the dry-off date.

There is nothing to milk before the first calving or after dry-off. Usually the wrong animal was picked, or the year in the date slipped.

3.15Calving interval longer than two and a half years

worth a look

The interval between calvings neighbouring in number exceeds nine hundred days.

Almost always this is a calving that was never recorded rather than a cow that stood barren for two and a half years: the calving numbers run consecutively while one of them is missing. The gap shifts the numbering of the lactations and the age group on every later record of the animal.

Limit: more than 900 days

3.16First insemination later than eight months after the calving

worth a look

More than two hundred and fifty days passed from the calving to the first insemination.

A gap like this means either lost records of inseminations or a cow that went half a year without being inseminated and was kept in the herd all the same. Both cases are worth a question, and both spoil the days open — the quantity by which the reproduction of the whole herd is judged.

Limit: more than 250 days

Production4 rules

Milk yield, fat, protein and how well they agree with one another.

4.5Imported evaluation diverges from the computation of the book

worth a look

The rank of the animal by the imported evaluation and by the computation of the book differs by more than forty percentiles.

Evaluations on different bases need not agree, but they must be about the same animal. A divergence of forty percentiles means one of them is about something else: a swapped column, the evaluation of another animal, an index of another breed or of another scale. Percentiles are what gets compared because they are the only common measure: index points from different evaluation centres are incomparable, whereas a rank within one’s own population is not.

Limit: divergence over 40 percentiles

4.11Fat in kilograms disagrees with the percentage

worth a look

Milk fat in kilograms differs from the yield multiplied by the fat percentage by more than a tenth.

A divergence larger than rounding means the figures were taken from different sources, and only the farm knows which of them is right.

Limit: divergence over 10%

4.14More than 75 days between test-day recordings

worth a look

In the run of test-day recordings of one lactation there is a gap longer than seventy-five days.

The 305-day yield is computed from the stretches between measurements, and the book fills the gaps in itself. Beyond seventy-five days the curve between two points is restored no longer from data but from an assumption: the peak of the lactation can fit entirely inside the gap. This is the very ground on which the book requires no fewer than six test days — declared in the glossary until now and never checked.

Limit: more than 75 days between neighbouring measurements

4.15The recording method changes mid-lactation

worth a look

Within one lactation part of the measurements were taken by a recording organisation and part by the farm, or the recording methods differ in some other way.

The 305-day yield is assembled from the points of one series, and the series must be homogeneous: a measurement by a recording organisation and a measurement by the farm are figures of different evidential weight. The herd check already sees this, but there it speaks of the herd in general; here it is about the particular lactation from which a particular figure in the document of the animal is computed.

Comparability across the herd16 rules

The only group where a finding belongs not to a record but to the herd as a whole. Every record here is faultless on its own — the trouble is that together they were obtained in different ways and cannot be compared with one another. That is why they are computed over the whole herd rather than over the submission: a share computed from a sample would call itself a share of the herd and would lie. They are visible in “Check my herd” and to the expert reviewing a submission; they do not block submission.

6.2Values stamped in rather than measured

worth a look

Within a single test day the fat, protein, somatic cells or daily yield are written down in a way a measurement cannot look: one value on too many animals, or a spread close to zero, or a last digit of zero on most of the records.

Each such figure on its own is plausible, and record-by-record checks let every one of them through. Together they mean that no samples were taken and the column was simply filled in. Lactation, breeding value and the comparison with contemporaries are all computed from such records — that is, the error spreads through the whole book while staying invisible.

Limit: the share of the mode three times above the expected is worth a look, ten times is a contradiction; a spread below 0.15 is always a contradiction; more than 40% round; at least 30 records in the test day

6.3Test-day recordings come from different sources

worth a look

The test-day recordings of the herd come partly from a laboratory, partly from the owner or from an import.

A laboratory measurement and a farm measurement are figures of different accuracy, and they cannot be added into one average. This is not a prohibition: the warning says that part of the data is not independently confirmed, and the herd cannot be judged from it as a whole.

6.4Indexes computed against different comparison bases

worth a look

Within one profile the evaluations of the animals of the herd refer to different versions of the comparison base.

An index is a deviation from a base. Two animals computed from different bases can neither be compared nor put in one list: part of the difference between them reflects the difference between the bases, not between the animals.

6.5A year without a single calving

worth a look

In the run of years between the first and the last calving of the herd there is a year with none recorded. Looked at only in herds that calve regularly: where calvings are rare, an empty year means nothing.

A herd that calved before and after cannot have stopped calving in between. Almost always it is a report that was never submitted for that year rather than an idle spell: lifetime figures and age at first calving are computed wrongly for those years.

6.6Dates of birth cluster on one date

worth a look

A noticeable share of the herd is listed as born on the first of January or on the first day of a month.

This is what a transfer from paper records looks like, where only the year or the month was known: the missing part was filled in with the start of the period. Age at first calving on such records is shifted by months, and the farm looks worse or better than it is.

Limit: the first of January on more than 5% of the herd; the first day of any month on more than 20%

6.7Milk yields suspiciously round

worth a look

A noticeable share of the yields in the herd is a multiple of five hundred kilograms.

A measured yield is rarely round. Roundness across the board means an estimate by eye or a transfer from a report where the figures had already been rounded — and such data cannot be used for breeding value evaluation.

Limit: multiples of 500 kg on more than a quarter

6.8Milk yield disagrees with the rest of the herd

worth a look

The yield of individual animals differs from the median of their own herd by more than threefold in either direction.

The plausibility limits are the same for the whole book and deliberately wide. The herd is a far more precise yardstick: on a farm averaging 7,000 kg a record of 22,000 is formally plausible, while in practice it is almost always an extra zero or someone else’s row.

Limit: threefold from the herd median

6.9Cows without a single test-day recording for a year

worth a look

For part of the live females no measurement has been recorded over the last twelve months. Computed over all live females, heifers included: by the same condition that produces this figure in the task strip of the farm account.

Production in the book is computed from test-day recordings. A cow without measurements takes part neither in the evaluation of a bull on his daughters nor in the herd averages — she is on the list and absent from the computations. The farm sees this same figure in the task strip of its account: there it names the work, here the reason for it.

Limit: no measurement in 12 months; not computed for herds below the size threshold

6.10Different records under one number

worth a look

Two or more records of the herd share the numeric part of their identifiers — although those identifiers are held in different fields or in different numbering systems.

Cattle in Russia have no single identifier: one animal goes under a national number, under XXRUS…, under an inventory number and under an ear tag. A match in the digits means one of two things — either it is one cow entered twice, or two different numbers that happen to coincide in digits. Only the farm can tell them apart, and the warning asks the question rather than answering it.

Limit: a match of at least 8 digits

6.11Not a single stillbirth

worth a look

Across every calving recorded for the herd not one stillborn calf is marked.

Stillbirths happen everywhere: worldwide they are five to eight per cent of calvings, and zero across a hundred calvings means not good fortune but that the losses are not being recorded. The share of stillbirths is the figure by which work with the dry period and the calving is judged; a herd without it looks better than it is, and cannot be compared with one that keeps its records honestly.

Limit: none at 100 calvings or more

6.12Not a single health event for a year

worth a look

Over the last twelve months not one case of illness or treatment has been recorded in the herd.

Cows fall ill, and a dairy herd without a single case of mastitis in a year does not exist. An empty log means not health but the absence of recording — and everything the book computes on health is silently wrong for such a herd. Worse: a herd like this looks exemplary beside one where the events are honestly recorded.

Limit: none at 50 live females or more

6.13Not a single disposal over several years

worth a look

Over the last three years neither a culling, nor a death, nor a sale away from the farm has been marked in the herd.

A herd without disposals does not exist: cows leave by age, by illness, by infertility. Zero means that disposals are handled past the book, and then the denominator of every share is wrong — from the percentage of cows with a recorded yield to the mean age of the herd. This is an error that is visible in no single record.

Limit: none over 3 years at 50 animals or more

6.14The record was edited after confirmation by the Association

worth a look

On an animal carrying the mark of the Association, fields were changed after the mark had been put on.

The confirmation refers to the state of the record the expert saw. An edit after it is legitimate — data gets refined — but the mark keeps standing all the while and no longer means what it meant. The book has been writing this into the edit log from the first day and has until now shown it to no one.

6.15One field rewritten several times over

worth a look

The same field of one record was edited three times or more within a week.

This is what fitting a value looks like: the figure is changed, the result is looked at, it is changed again. It is also what an honest search for an error in the transfer looks like — the log cannot tell the two apart, and the rule does not undertake to. It names the place where it is worth asking what went on there.

Limit: 3 edits of one field in 7 days

6.16Events are entered after the fact

worth a look

A noticeable share of the events of the herd was written into the book much later than the day they are dated by.

A test-day recording for March uploaded in December was restored from memory or from paper — and it holds an order of magnitude more errors than one recorded at once. In itself this is no offence: a farm has every right to bring in data for the past year. But material of such an origin cannot be silently added into one average with current recording, and it has to be known about before the computation, not after.

Limit: a gap over 180 days on a fifth of the events

6.17The same data comes from different senders

worth a look

Records of one animal of one kind — production, for instance — were sent by two or more different senders.

With the arrival of upload mandates this became possible: the recording organisation and the laboratory are appointed by the farm, and both send test-day recordings. Which of them is right the book does not decide — the one that sent last wins, and it happens silently. Until the conflict is named, the owner of the data does not know whose figures are lying in it.