Data validation
The rules by which the book argues with a record
An open list: what is checked, why it is an error, and what to do about it. Rules are stated in advance — like an exam syllabus, not a remark after submission.
rules in the registry
find a contradiction in the data
report that data is absent
draw attention without holding up confirmation
carry a numeric limit stated aloud
Three states, and “fix it” is not among them
A check never asks you to fix a record, and that is not politeness. By construction it cannot tell a correction from a fudge: a rule saying “yield of 45,000 kg is out of bounds” would, at the word “fix it”, invite someone to put 12,000 — the block lifts, the mark of the Association goes on, and the book records a number nobody ever measured. So a check names the state of the record, and the decision stays with people.
There are three states. “Data missing” — the fields are empty: no date of birth, no breed, not a single parent. There is nothing to fake here, and the only way on is to upload what is absent. “Data does not add up” — the data is there and contradicts itself: a calving before the birth, DNA that rules out the recorded sire, a blood share above one hundred per cent. Here editing the record is precisely the thing that must not be done without grounds. “Worth a look” — the rule has noticed something odd and may be wrong.
The difference between the second and the third is not one of strictness but of whether the rule itself can be wrong. “Parent younger than offspring” cannot be wrong — it is a contradiction. “Yield above twenty-five thousand” can be: such a cow is rare but possible, and forbidding her would mean vouching for what we do not know.
What happens to a record that has questions against it
Nothing is deleted and nothing is rolled back: the data stays in the book, and the farm sees it and edits it. What is held is something else — the confirmation of the Association, the very mark the record is kept for. Until a finding has been dealt with, the animal is not confirmed and does not count as pedigree by the book.
How it is lifted differs. “Data missing” is lifted by uploading what is absent — and by nothing else. “Data does not add up” is lifted by an explanation the expert accepts: a document, a laboratory certificate, a letter from the farm saying where the discrepancy came from. The expert may judge a finding immaterial and waive it — but the waiver is by name, for this record and this rule, and it stays in the history of the application.
The third outcome is lawful too: to accept that the animal does not count as pedigree. The book is under no obligation to confirm everything uploaded into it, and a record without confirmation is not a defect but an honest statement of where things stand.
What the book asks for besides consistency
The rules below check the data against itself. Apart from them the book looks at completeness: confirmation means not only “no discrepancies” but also “everything required has been submitted”. The list is short and the same for every application.
Without these a record is not confirmed:
- date of birthage at first calving, lactation numbers and pedigree checks are all counted from it
- breedwithout a breed the record cannot be compared with contemporaries or put into reports
- sire: a record in the book or a number from the documentsthe book confirms parentage; with no sire there is nothing to confirm
- dam: a record in the book or a number from the documentswithout a dam the maternal line breaks and offspring inbreeding cannot be computed
- at least one calvinga cow without a calving is either not a cow or a history that has been lost
- test-day recordings (at least 6 if a 305-day yield is claimed)a 305-day yield with no measurements is a number taken from nowhere
The list
Every rule has a number, and the number is what to quote in a letter or in a conversation with support: rule 45. It reflects the place of the rule in the registry and shifts if a new rule is inserted in the middle; the permanent key is the code of the rule, which goes into exports and never changes.
worth a look: 31rules in the registry
Passport1 rule
Fields of the record itself: identifier, date of birth, breed, breed percentage.
1.5Ear tag repeated
worth a lookOne tag number stands on several animals in the batch.
Tags are replaced and moved between animals, so a tag is not treated as unique. But two live animals under one tag on one farm are almost always a typing error.
Parentage3 rules
Parents, the pedigree and everything computed from it. An error here costs more than the rest: it does not spoil one record, it distorts the evaluation of every descendant.
2.5Sire left the herd long before conception
worth a lookThe sire was disposed of before the offspring could have been conceived: more than a gestation period lies between the disposal and the birth.
Not an error in itself: frozen semen keeps for decades, and this is an ordinary case. But if the farm stores no semen, the link points to the wrong bull.
2.11Breed percentage disagrees with the parents
worth a lookThe breed percentage differs from the mean of the parental values by more than the tolerance.
The breed percentage of the offspring is the mean of the parental ones. A divergence means an error either in the percentage or in the parent itself; the second is far worse, because it spoils every other offspring of that parent as well. This is the only check whose severity depends on the size of the divergence: the marker on the left names the common case, while a large divergence arrives as one to be fixed.
2.12Inbreeding disagrees with the pedigree
worth a lookThe inbreeding coefficient entered on the record differs from the one computed from the pedigree.
A divergence is not always an error: our computation runs on the pedigree held in the book, while the farm may have computed on a fuller one. But it is always a question — on which pedigree.
Reproduction7 rules
Calvings and inseminations — not one by one, but as a sequence in time.
3.2First calving too late
worth a lookThe age at first calving is over forty-eight months.
Calving at four years is possible, merely expensive. More often it means that the earlier calvings were simply never recorded — and then what is wrong is not the age but the completeness of the data.
3.6Gap in the calving numbers
worth a lookOne or more numbers are missing from the run of calving numbers.
Either a calving was never recorded — and then every lifetime figure is incomplete — or the numbers were entered wrongly.
3.11Calf not born on the day of the calving
worth a lookThe date of birth of the animal recorded as the calf does not match the date of the calving.
A calving and the birth of a calf are one fact recorded twice. A divergence means that either the wrong animal is marked as the calf, or one of the two dates has a typing error.
3.12Offspring counts disagree with one another
worth a lookThe birth type, the counts of live heifer calves, bull calves and stillborn, and the calf records themselves state different numbers.
How many were born, the book knows three times over, and the three sources appeared at different times: the birth type was carried over from the previous system, the counts are entered by hand, and records are not created for every calf. A divergence means that one of the three is wrong, and which one is visible from the finding itself.
3.13Test-day recording outside the lactation
worth a lookA test-day recording is dated before the first calving of the cow or after the dry-off date.
There is nothing to milk before the first calving or after dry-off. Usually the wrong animal was picked, or the year in the date slipped.
3.15Calving interval longer than two and a half years
worth a lookThe interval between calvings neighbouring in number exceeds nine hundred days.
Almost always this is a calving that was never recorded rather than a cow that stood barren for two and a half years: the calving numbers run consecutively while one of them is missing. The gap shifts the numbering of the lactations and the age group on every later record of the animal.
3.16First insemination later than eight months after the calving
worth a lookMore than two hundred and fifty days passed from the calving to the first insemination.
A gap like this means either lost records of inseminations or a cow that went half a year without being inseminated and was kept in the herd all the same. Both cases are worth a question, and both spoil the days open — the quantity by which the reproduction of the whole herd is judged.
Production4 rules
Milk yield, fat, protein and how well they agree with one another.
4.5Imported evaluation diverges from the computation of the book
worth a lookThe rank of the animal by the imported evaluation and by the computation of the book differs by more than forty percentiles.
Evaluations on different bases need not agree, but they must be about the same animal. A divergence of forty percentiles means one of them is about something else: a swapped column, the evaluation of another animal, an index of another breed or of another scale. Percentiles are what gets compared because they are the only common measure: index points from different evaluation centres are incomparable, whereas a rank within one’s own population is not.
4.11Fat in kilograms disagrees with the percentage
worth a lookMilk fat in kilograms differs from the yield multiplied by the fat percentage by more than a tenth.
A divergence larger than rounding means the figures were taken from different sources, and only the farm knows which of them is right.
4.14More than 75 days between test-day recordings
worth a lookIn the run of test-day recordings of one lactation there is a gap longer than seventy-five days.
The 305-day yield is computed from the stretches between measurements, and the book fills the gaps in itself. Beyond seventy-five days the curve between two points is restored no longer from data but from an assumption: the peak of the lactation can fit entirely inside the gap. This is the very ground on which the book requires no fewer than six test days — declared in the glossary until now and never checked.
4.15The recording method changes mid-lactation
worth a lookWithin one lactation part of the measurements were taken by a recording organisation and part by the farm, or the recording methods differ in some other way.
The 305-day yield is assembled from the points of one series, and the series must be homogeneous: a measurement by a recording organisation and a measurement by the farm are figures of different evidential weight. The herd check already sees this, but there it speaks of the herd in general; here it is about the particular lactation from which a particular figure in the document of the animal is computed.
Comparability across the herd16 rules
The only group where a finding belongs not to a record but to the herd as a whole. Every record here is faultless on its own — the trouble is that together they were obtained in different ways and cannot be compared with one another. That is why they are computed over the whole herd rather than over the submission: a share computed from a sample would call itself a share of the herd and would lie. They are visible in “Check my herd” and to the expert reviewing a submission; they do not block submission.
6.2Values stamped in rather than measured
worth a lookWithin a single test day the fat, protein, somatic cells or daily yield are written down in a way a measurement cannot look: one value on too many animals, or a spread close to zero, or a last digit of zero on most of the records.
Each such figure on its own is plausible, and record-by-record checks let every one of them through. Together they mean that no samples were taken and the column was simply filled in. Lactation, breeding value and the comparison with contemporaries are all computed from such records — that is, the error spreads through the whole book while staying invisible.
6.3Test-day recordings come from different sources
worth a lookThe test-day recordings of the herd come partly from a laboratory, partly from the owner or from an import.
A laboratory measurement and a farm measurement are figures of different accuracy, and they cannot be added into one average. This is not a prohibition: the warning says that part of the data is not independently confirmed, and the herd cannot be judged from it as a whole.
6.4Indexes computed against different comparison bases
worth a lookWithin one profile the evaluations of the animals of the herd refer to different versions of the comparison base.
An index is a deviation from a base. Two animals computed from different bases can neither be compared nor put in one list: part of the difference between them reflects the difference between the bases, not between the animals.
6.5A year without a single calving
worth a lookIn the run of years between the first and the last calving of the herd there is a year with none recorded. Looked at only in herds that calve regularly: where calvings are rare, an empty year means nothing.
A herd that calved before and after cannot have stopped calving in between. Almost always it is a report that was never submitted for that year rather than an idle spell: lifetime figures and age at first calving are computed wrongly for those years.
6.6Dates of birth cluster on one date
worth a lookA noticeable share of the herd is listed as born on the first of January or on the first day of a month.
This is what a transfer from paper records looks like, where only the year or the month was known: the missing part was filled in with the start of the period. Age at first calving on such records is shifted by months, and the farm looks worse or better than it is.
6.7Milk yields suspiciously round
worth a lookA noticeable share of the yields in the herd is a multiple of five hundred kilograms.
A measured yield is rarely round. Roundness across the board means an estimate by eye or a transfer from a report where the figures had already been rounded — and such data cannot be used for breeding value evaluation.
6.8Milk yield disagrees with the rest of the herd
worth a lookThe yield of individual animals differs from the median of their own herd by more than threefold in either direction.
The plausibility limits are the same for the whole book and deliberately wide. The herd is a far more precise yardstick: on a farm averaging 7,000 kg a record of 22,000 is formally plausible, while in practice it is almost always an extra zero or someone else’s row.
6.9Cows without a single test-day recording for a year
worth a lookFor part of the live females no measurement has been recorded over the last twelve months. Computed over all live females, heifers included: by the same condition that produces this figure in the task strip of the farm account.
Production in the book is computed from test-day recordings. A cow without measurements takes part neither in the evaluation of a bull on his daughters nor in the herd averages — she is on the list and absent from the computations. The farm sees this same figure in the task strip of its account: there it names the work, here the reason for it.
6.11Not a single stillbirth
worth a lookAcross every calving recorded for the herd not one stillborn calf is marked.
Stillbirths happen everywhere: worldwide they are five to eight per cent of calvings, and zero across a hundred calvings means not good fortune but that the losses are not being recorded. The share of stillbirths is the figure by which work with the dry period and the calving is judged; a herd without it looks better than it is, and cannot be compared with one that keeps its records honestly.
6.12Not a single health event for a year
worth a lookOver the last twelve months not one case of illness or treatment has been recorded in the herd.
Cows fall ill, and a dairy herd without a single case of mastitis in a year does not exist. An empty log means not health but the absence of recording — and everything the book computes on health is silently wrong for such a herd. Worse: a herd like this looks exemplary beside one where the events are honestly recorded.
6.13Not a single disposal over several years
worth a lookOver the last three years neither a culling, nor a death, nor a sale away from the farm has been marked in the herd.
A herd without disposals does not exist: cows leave by age, by illness, by infertility. Zero means that disposals are handled past the book, and then the denominator of every share is wrong — from the percentage of cows with a recorded yield to the mean age of the herd. This is an error that is visible in no single record.
6.14The record was edited after confirmation by the Association
worth a lookOn an animal carrying the mark of the Association, fields were changed after the mark had been put on.
The confirmation refers to the state of the record the expert saw. An edit after it is legitimate — data gets refined — but the mark keeps standing all the while and no longer means what it meant. The book has been writing this into the edit log from the first day and has until now shown it to no one.
6.15One field rewritten several times over
worth a lookThe same field of one record was edited three times or more within a week.
This is what fitting a value looks like: the figure is changed, the result is looked at, it is changed again. It is also what an honest search for an error in the transfer looks like — the log cannot tell the two apart, and the rule does not undertake to. It names the place where it is worth asking what went on there.
6.16Events are entered after the fact
worth a lookA noticeable share of the events of the herd was written into the book much later than the day they are dated by.
A test-day recording for March uploaded in December was restored from memory or from paper — and it holds an order of magnitude more errors than one recorded at once. In itself this is no offence: a farm has every right to bring in data for the past year. But material of such an origin cannot be silently added into one average with current recording, and it has to be known about before the computation, not after.
6.17The same data comes from different senders
worth a lookRecords of one animal of one kind — production, for instance — were sent by two or more different senders.
With the arrival of upload mandates this became possible: the recording organisation and the laboratory are appointed by the farm, and both send test-day recordings. Which of them is right the book does not decide — the one that sent last wins, and it happens silently. Until the conflict is named, the owner of the data does not know whose figures are lying in it.