Re: Raw & Field Data

This was originally sent on 20 Dec but the CaveXML server was having problems, so here it is again. - Peter

At 20:16 17-12-02 -0500, Lev Bishop wrote:
>On Sat, 7 Dec 2002, Peter MATTHEWS wrote:
>
> > Raw data:
> > An initial unaltered digital copy of some or all of the survey Field Data,
> > now ready for processing.
>
>I'm not entirely sure about "now ready for processing". If the original
>data had errors in it that make it impossible to process in its original
>form then I think still you want to be able to enter it as raw data. It
>should only be at the edited data stage (if at all...) where requirements
>like "angles have to be between 0 and 360 degrees", "stations have to have
>unique identifiers", needing to have values for all of the instruments for
>each leg (eg. if you forgot to record the horizontal angle on a
>near-vertical leg and will need to fudge it). On the other hand the format
>of the raw data should either be the same as the edited data so that if it
>meets all the requirements it can then be processed, or it should be such
>that if those requirements are satisfied then it can be automatically
>processed into edited data. Maybe "now ready for processing" should be
>changed to "in a format suitable for processing, but with no/minimal data
>verification or consistancy checking"?

Lev has a good point - "now ready for processing" is fairly ambiguous, which is what we are trying to avoid. In my mind the "processing" included manual processing, such as editing mistakes out of it, but on the other hand it could also be interpreted as 'now ready for calculation', which of course may not be the case, as Lev has pointed out.

>...
> > 4. If such a program unilaterally alters the data as it is being entered,
> > then the data has become Edited Data because its values differ from the
> > Field Data, i.e. a Raw Data version has effectively been skipped.
>Add: "This applies even if a number entered as 1.0 becomes 1.00, where the
>numerical value of the data hasn't changed but it is still different from
>what was entered". Controversial? I'm thinking in terms of a number
>recorded as 125 degrees - or was it 12.5 degrees? - maybe the book person
>wasn't careful to make decimal points obvious. Sketch and the loop
>closures suggest 12.5 would be more consistent so we go back to the raw
>data and look and see whether it was recorded as 125 or 125.0 to decide on
>this (assuming the field data isn't available any more for whatever
>reason).

Another good point, however there's so many possible cases here that I don't think we afford to go into detail. But we could beef up the comment in general terms, to make it clearer.

>Other than the above comments, sounds good to me,
>
>Lev

So the new proposed definitions become:
Temporary [] brackets identify the bits which I have changed.

• 20-Dec-02 ================================= Field data: The unaltered survey data (readings and/or sketches) recorded in the field by whatever means, or verified copies thereof. Comments: For example, paper-based records, or decipherable images thereof which may have been altered but only to clarify the original data [(for example, marked up photocopies)], or data downloaded unaltered from an instrument (survey instrument, PDA, laptop, etc), or data still stored and observable in an instrument.

  ---

• 20-Dec-02 ====================================== Raw data: An initial unaltered digital copy of some or all of the survey Field Data, now ready for [editing, validity checking, calculation or other processing]. Comments: 1. Where the Field Data was already in digital form, e.g. downloaded from an instrument, the Raw Data version could be an identical but editable copy, whereas the Field Data version would effectively be a read-only copy. 2. Raw Data could include sketches now converted to fixed or editable digital form. 3. The Raw Data could be in any format, including that of a survey processing program into which the Field Data has been typed. 4. If such a program unilaterally alters the data [in any material way] as it is being entered, then the data has become Edited Data because [it differs] from the Field Data, i.e. a Raw Data version has effectively been skipped. 5. Such a survey program might also store the data in a proprietary binary format unconducive to easy data exchange, therefore the coming CaveXML standard may need to define suitable forms of Raw Data to allow free exchange. Such binary data after being exported to a text format might qualify. Acceptable formats for raster or vector graphical data may also be required.

  ---

Any further comments? We must be getting close now. :-)

Peter Received on Wed Jan 1 19:39:55 2003