head 1.13; access; symbols; locks; strict; comment @# @; 1.13 date 2009.11.25.03.14.41; author GarryJolleyRogers; state Exp; branches; next 1.12; 1.12 date 2009.11.20.02.35.36; author LeeBelbin; state Exp; branches; next 1.11; 1.11 date 2007.03.06.17.30.00; author TWikiGuest; state Exp; branches; next 1.10; 1.10 date 2004.07.15.18.04.00; author GregorHagedorn; state Exp; branches; next 1.9; 1.9 date 2004.06.11.10.15.49; author GregorHagedorn; state Exp; branches; next 1.8; 1.8 date 2004.06.11.09.08.12; author GregorHagedorn; state Exp; branches; next 1.7; 1.7 date 2004.06.03.14.27.14; author DonaldHobern; state Exp; branches; next 1.6; 1.6 date 2004.06.02.13.40.00; author GregorHagedorn; state Exp; branches; next 1.5; 1.5 date 2004.05.25.13.14.00; author GregorHagedorn; state Exp; branches; next 1.4; 1.4 date 2004.05.25.11.11.17; author BobMorris; state Exp; branches; next 1.3; 1.3 date 2004.05.24.14.12.00; author GregorHagedorn; state Exp; branches; next 1.2; 1.2 date 2004.05.24.10.23.24; author MarkusDoering; state Exp; branches; next 1.1; 1.1 date 2004.05.21.17.51.36; author GregorHagedorn; state Exp; branches; next ; desc @none @ 1.13 log @none @ text @%META:TOPICINFO{author="GarryJolleyRogers" date="1259118881" format="1.1" version="1.13"}% %META:TOPICPARENT{name="UBIF.SchemaDiscussion"}% ---+!! %TOPIC% (This is a subtopic of UBIF.SchemaDiscussion) --- 1. The BDI.SDD_ version discussed in Berlin had
  DataSets/DataSet/TransformationHistory/Transformation
and then Actions, Generator, ... and PreviousTransformations/Transformation. TransformationHistory/Transformation seems one step too much, but neither term seems to express the mixture of current and previous correctly. Bryan Heidorn and I discussed this and I propose for the general (TDWG-wide, see ResolvedTopicCommonSchemaSearchingName for question how to call this...) history and project metadata thing: a) To rename Transformation to .../Derivation (.../PreviousDerivations) and remove the duplication of TransformationHistory/Transformation.
b) Accept that it is metadata and move it into the ProjectMetadata element (preferably after the current IPRStatements), i. e.
  DataSets/DataSet/ProjectMetadata/Derivation (containing Actions, Generator, ... PreviousDerivations/Derivation)
c) Related: Just having DataSets/DataSet/Metadata seems to be too unspecific. Is it ok to have for all kind of TDWG standards DataSets/DataSet/ProjectMetadata? Other element name proposals? Please comment! See BDI.SDD_.CurrentSchemaVersion for download of schema! -- Gregor Hagedorn - 21 May 2004 --- I would like to get more specific about the kind of metadata we are relating to in this discussion.
From a more data access point of view I see the following different sources for metadata: (1) metadata about the supplier of the data (creator of the dataset).
(2) metadata about the specific dataset.
(3) metadata about the full source dataset.
I am not sure if the proposed project metadata would fit in any of those. Probably into (3). I am aware that there is a lot of the current BDI.SDD_ project definition missing, but I am really not sure if this should all become a shared definition. For ABCD networking I would think that (3) would describe the datasource, which does not have to be congruent with a single collection, but rather describes a single database. The supplier (1) would be mostly important for technical contact. It could be worthwhile thinking about embedding the suppliers (1) metadata directly into the protocol. On the other hand I see the protocol as machine readable only, so a technical contact doesn't belong into the protocol itself from this point of view, but rather in an external metadata standard which I hope this shared standard could serve as. Data about a specific dataset (2) to me didn't seem very important first, but now I think that the data derivation/transformation part is actually exactly describing the change of the data since it left the "source". So data derivation would belong into category (2) as long as we stick to the idea to only list automatic derivations here, which are not stored somewhere and more or less done on the fly. Other (2) metadata would probably be derived from the dataset itself and could be better achieved by external public services like a GBIF indexing service for datasources. This is why I called category (2) metadata in the proposal below just <Derivation>. Following is the core version of what I imagine the shared metadata should hold based on mondays Berlin discussion.
... <ABCDUnits> is used as an example only and could clearly be any of the TDWG standards. -- Markus Doering - 24 May 2004 --- I think your 3-point list at the top is really useful. The design as shown in BDI.SDD_ beta 15 and later (see BDI.SDD_.CurrentSchemaVersion) combines the technical metadata about supplier with the Derivations (points 1 and 2) since these also exist for previous derivations. These previous data I consider very valuable. Please argue if you think not. Putting them into Derivations means a simple rule exists: If you are a portal getting data from elsewhere and pass them on (with or without filtering) enclose the Derivation you received with your own data, annotate your own actions and pass it on. I think this applies to your "Supplier" element. There is relatively little information about the specific dataset. As you say, some can be inferred from Actions in Derivation, but only the Base-dataset is described in detail. The assumption is that the rest can be inferred from the data that are actually delivered. These inferences could be cached/indexed at GBIF, as you say. Can anyone think of use cases/scenarios where more data would be needed? If not I assume that is covered. (Actually one thing missing is an ExpirationDate, see BDI.SDD_.DataExpirationDates for a discussion of this.) Your SourceDataSet seems to me closely related to the current ProjectMetadata. Is SourceDataSet a better term? I have some reservations because earlier data sources may exists, which have been reworked into the current source data, e.g. an older DELTA revision or a specimen database has been ported to newer data formats and revised or error corrected. Thus there is an IPR history, (which was discussed in Berlin: technical versus human revision history) and since we could not find a good solution to integrate that with the technical history, we propose to have humans edit IPR, acknowledgment, etc. similarly to a printed publication. When you say less than the current data should go in the common type, I have no problem but discussing it with Bryan we thought everything that is still in there in Beta 15 generally useful. Please do argue against specific points! Is there really an AdminContact on the data set metadata level as you propose? What was missing in the version before the meeting was an Owner, that has now been added (albeit as a reference to an Agent proxy object). Is that sufficiently equivalent to AdminContact? Finally, I am not sure what you mean when you talk about some things belonging perhaps to a protocol? Do you mean something like DiGIR? My framework of mind would be that metadata information that is important when a received dataset is processed or archived away, should be embedded into the "payload" rather than being exchanged in (presumably unprotocolled) protocol exchanges before the data are delivered. Or do you think that the ABCD/BDI.SDD_ data would in future be wrapped again in outer wrappers? -- Gregor Hagedorn - 24 May 2004 --- A few responses. (1) I fully agree with Gregor that the metadata should be part of the delivered data package rather than an earlier protocol exchange. This is the only way to manage the data and metadata correctly with more complex applications which aggregate data sets. (2) I think that the metadata should always describe the full source data set. The filtering and other processing involved in generating the response should become the first element in the TransformationHistory. (3) I think that the metadata discussion could (and perhaps should) be widened even further than Markus' initial three points. For what different purposes are we producing metadata? We certainly want the metadata to be available to explain a data set (whichever of Markus' three versions we mean, and including everything discussed above). Can we use the same metadata to assist users in discovering the data set in the first place? In other words, is the descriptive metadata needed to allow a user to select a resource the same as the metadata that we pass with the resource on subsequent data requests?. I think these metadata are the same, but perhaps we should be explicit. (Incidentally this is all relevant to DiGIR/BioCASe reconciliation - DiGIR search/inventory requests should include the metadata with all responses, ideally using the common data structures we are discussing here in this Wiki; BioCASe should include the plain metadata request, returning a response without any data records.) (4) If we use the same metadata model for discovery, do we need to extend it to include information on available access mechanisms (endpoints, different protocols, etc.)? Or do we assume that any user has already managed to locate and access the resource (or that all users use an external registry for this purpose)? Or should we in fact model include these references to these different endpoints in the metadata using the Proxy model (whateever we finally decide to call it)? -- Main.DonaldHobern - 03 Jun 2004 %META:TOPICMOVED{by="GregorHagedorn" date="1089915351" from="SDD.DerivationHistory" to="UBIF.DerivationHistory"}% @ 1.12 log @none @ text @d1 1 a1 1 %META:TOPICINFO{author="LeeBelbin" date="1258684536" format="1.1" reprev="1.12" version="1.12"}% d8 1 a8 1 1. The BDI.SDD version discussed in Berlin had
d19 1 a19 1 Please comment! See BDI.SDD.CurrentSchemaVersion for download of schema! d31 1 a31 1 I am not sure if the proposed project metadata would fit in any of those. Probably into (3). I am aware that there is a lot of the current BDI.SDD project definition missing, but I am really not sure if this should all become a shared definition. For ABCD networking I would think that (3) would describe the datasource, which does not have to be congruent with a single collection, but rather describes a single database. d95 1 a95 1 I think your 3-point list at the top is really useful. The design as shown in BDI.SDD beta 15 and later (see BDI.SDD.CurrentSchemaVersion) combines the technical metadata about supplier with the Derivations (points 1 and 2) since these also exist for previous derivations. These previous data I consider very valuable. Please argue if you think not. Putting them into Derivations means a simple rule exists: If you are a portal getting data from elsewhere and pass them on (with or without filtering) enclose the Derivation you received with your own data, annotate your own actions and pass it on. I think this applies to your "Supplier" element. d97 1 a97 1 There is relatively little information about the specific dataset. As you say, some can be inferred from Actions in Derivation, but only the Base-dataset is described in detail. The assumption is that the rest can be inferred from the data that are actually delivered. These inferences could be cached/indexed at GBIF, as you say. Can anyone think of use cases/scenarios where more data would be needed? If not I assume that is covered. (Actually one thing missing is an ExpirationDate, see BDI.SDD.DataExpirationDates for a discussion of this.) d103 1 a103 1 Finally, I am not sure what you mean when you talk about some things belonging perhaps to a protocol? Do you mean something like DiGIR? My framework of mind would be that metadata information that is important when a received dataset is processed or archived away, should be embedded into the "payload" rather than being exchanged in (presumably unprotocolled) protocol exchanges before the data are delivered. Or do you think that the ABCD/BDI.SDD data would in future be wrapped again in outer wrappers? @ 1.11 log @Added topic name via script @ text @d1 2 a4 2 %META:TOPICINFO{author="GregorHagedorn" date="1089914640" format="1.0" version="1.10"}% %META:TOPICPARENT{name="UBIF.SchemaDiscussion"}% d8 1 a8 1 1. The SDD version discussed in Berlin had
d19 1 a19 1 Please comment! See SDD.CurrentSchemaVersion for download of schema! d31 1 a31 1 I am not sure if the proposed project metadata would fit in any of those. Probably into (3). I am aware that there is a lot of the current SDD project definition missing, but I am really not sure if this should all become a shared definition. For ABCD networking I would think that (3) would describe the datasource, which does not have to be congruent with a single collection, but rather describes a single database. d43 44 a86 44
... d95 1 a95 1 I think your 3-point list at the top is really useful. The design as shown in SDD beta 15 and later (see SDD.CurrentSchemaVersion) combines the technical metadata about supplier with the Derivations (points 1 and 2) since these also exist for previous derivations. These previous data I consider very valuable. Please argue if you think not. Putting them into Derivations means a simple rule exists: If you are a portal getting data from elsewhere and pass them on (with or without filtering) enclose the Derivation you received with your own data, annotate your own actions and pass it on. I think this applies to your "Supplier" element. d97 1 a97 1 There is relatively little information about the specific dataset. As you say, some can be inferred from Actions in Derivation, but only the Base-dataset is described in detail. The assumption is that the rest can be inferred from the data that are actually delivered. These inferences could be cached/indexed at GBIF, as you say. Can anyone think of use cases/scenarios where more data would be needed? If not I assume that is covered. (Actually one thing missing is an ExpirationDate, see SDD.DataExpirationDates for a discussion of this.) d103 1 a103 1 Finally, I am not sure what you mean when you talk about some things belonging perhaps to a protocol? Do you mean something like DiGIR? My framework of mind would be that metadata information that is important when a received dataset is processed or archived away, should be embedded into the "payload" rather than being exchanged in (presumably unprotocolled) protocol exchanges before the data are delivered. Or do you think that the ABCD/SDD data would in future be wrapped again in outer wrappers? @ 1.10 log @none @ text @d1 2 @ 1.9 log @none @ text @d1 3 a3 3 %META:TOPICINFO{author="GregorHagedorn" date="1086948949" format="1.0" version="1.9"}% %META:TOPICPARENT{name="UnifiedBioInfoFramework"}% (This is a subtopic of UnifiedBioInfoFramework) d17 1 a17 1 Please comment! See CurrentSchemaVersion for download of schema! d93 1 a93 1 I think your 3-point list at the top is really useful. The design as shown in SDD beta 15 and later (see CurrentSchemaVersion) combines the technical metadata about supplier with the Derivations (points 1 and 2) since these also exist for previous derivations. These previous data I consider very valuable. Please argue if you think not. Putting them into Derivations means a simple rule exists: If you are a portal getting data from elsewhere and pass them on (with or without filtering) enclose the Derivation you received with your own data, annotate your own actions and pass it on. I think this applies to your "Supplier" element. d95 1 a95 1 There is relatively little information about the specific dataset. As you say, some can be inferred from Actions in Derivation, but only the Base-dataset is described in detail. The assumption is that the rest can be inferred from the data that are actually delivered. These inferences could be cached/indexed at GBIF, as you say. Can anyone think of use cases/scenarios where more data would be needed? If not I assume that is covered. (Actually one thing missing is an ExpirationDate, see DataExpirationDates for a discussion of this.) d118 1 @ 1.8 log @none @ text @d1 3 a3 3 %META:TOPICINFO{author="GregorHagedorn" date="1086944892" format="1.0" version="1.8"}% %META:TOPICPARENT{name="OverarchingPatternsForTdwgSchemata"}% (This is a subtopic of OverarchingPatternsForTdwgSchemata) @ 1.7 log @none @ text @d1 1 a1 1 %META:TOPICINFO{author="DonaldHobern" date="1086272833" format="1.0" version="1.7"}% d10 1 a10 1 Bryan Heidorn and I discussed this and I propose for the general (TDWG-wide, see CommonInfrastructureSearchingName for question how to call this...) history and project metadata thing: @ 1.6 log @none @ text @d1 1 a1 1 %META:TOPICINFO{author="GregorHagedorn" date="1086183600" format="1.0" version="1.6"}% d105 13 @ 1.5 log @none @ text @d1 1 a1 1 %META:TOPICINFO{author="GregorHagedorn" date="1085490840" format="1.0" version="1.5"}% d3 3 d10 1 a10 1 Bryan Heidorn and I discussed this and I propose for the general (TDWG-wide) history and project metadata thing: d17 1 a17 1 Please comment! @ 1.4 log @none @ text @d1 1 a1 1 %META:TOPICINFO{author="BobMorris" date="1085483477" format="1.0" version="1.4"}% d90 1 a90 1 I think your 3-point list at the top is really useful. Currently the design as shown in SDD beta 14 (see CurrentSchemaVersion) combines the technical metadata about supplier with the Derivations (points 1 and 2) since these also exist for previous derivations. These previous data I consider very valuable. Please argue if you think not. Putting them into Derivations means a simple rule exists: If you are a portal getting data from elsewhere and pass them on (with or without filtering) enclose the Derivation you received with your own data, annotate your own actions and pass it on. I think this applies to your "Supplier" element. d92 1 a92 1 There is relatively little information about the specific dataset. As you say, some can be inferred from Actions in Derivation, but only the Base-dataset is described in detail. The assumption is that the rest can be inferred from the data that are actually delivered. These inferences could be cached/indexed at GBIF, as you say. Can anyone think of use cases/scenarios where more data would be needed? If not I assume that is covered. d94 1 a94 3 * actually one thing missing is the expiration data of a dataset, which is not yet in SDD. I seem to have a block here, but I have no idea how the provider could know this date. Nobody can look into the future when the next change in the data will be made. So any value will be just heuristic. Would that not be a better choice for the receiving side to be made? Where should it go, and what should be the annotation that gives a provider a hint about what should go in there? (See DataExpirationDates -- Main.BobMorris - 25 May 2004) Your SourceDataSet seems to me closely related to the current ProjectMetadata. Is SourceDataSet a better term? I have some reservations because earlier data sources may exists, which have been reworked into the current source data, e.g. an older DELTA revision or a specimen database has been ported to newer data formats and revised or error corrected. Thus there is an IPR history, (which was discussed in Berlin: technical versus human revision history) and since we could not find a good solution to integrate that with the technical history, we propose to have humans edit IPR, acknowledgment, etc similarly to a printed publication. When you say less than the current data should go in the common type, I have no problem but discussing it with Bryan we thought everything that is still in there in Beta 14 generally useful. Please do argue against specific points! @ 1.3 log @none @ text @d1 1 a1 1 %META:TOPICINFO{author="GregorHagedorn" date="1085407920" format="1.0" version="1.3"}% d94 1 a94 1 * actually one thing missing is the expiration data of a dataset, which is not yet in SDD. I seem to have a block here, but I have no idea how the provider could know this date. Nobody can look into the future when the next change in the data will be made. So any value will be just heuristic. Would that not be a better choice for the receiving side to be made? Where should it go, and what should be the annotation that gives a provider a hint about what should go in there? @ 1.2 log @none @ text @d1 1 a1 1 %META:TOPICINFO{author="MarkusDoering" date="1085394204" format="1.0" version="1.2"}% a9 1 d11 1 a11 2   DataSets/DataSet/ProjectMetadata/Derivation (containing Actions, Generator, ... PreviousDerivations/Derivation) d19 2 a20 2 I would like to get more specific about the kind of metadata we are relating to in this discussion.
From a more data access point of view I see the following different sources for metadata: d22 3 a24 1 (1) metadata about the supplier of the data (creator of the dataset). d26 1 a26 1 (2) metadata about the specific dataset. d28 1 a28 20 (3) metadata about the full source dataset. I am not sure if the proposed project metadata would fit in any of those. Probably into (3). I am aware that there is a lot of the current SDD project definition missing, but I am really not sure if this should all become a shared definition. For ABCD networking I would think that (3) would describe the datasource, which does not have to be congruent with a single collection, but rather describes a single database. The supplier (1) would be mostly important for technical contact. It could be worthwhile thinking about embedding the suppliers (1) metadata directly into the protocol. On the other hand I see the protocol as machine readable only, so a technical contact doesn't belong into the protocol itself form this point of view, but rather in an external metadata standard which I hope this shared standard could serve as. Data about a specific dataset (2) to me didn't seem very important first, but now I think that the data derivation/transformation part is actually exactly describing the change of the data since it left the "source". So data derivation would belong into category (2) as long as we stick to the idea to only list automatic derivations here, which are not stored somewhere and more or less done on the fly.
Other (2) metadata would probably be derived from the dataset itself and could be better achieved by external public services like a GBIF indexing service for datasources. This is why I called category (2) metadata in the proposal below just <Derivation>. d30 1 d32 1 d36 1 a36 2 d83 1 d85 1 d87 2 a88 1 d90 9 a98 1 <ABCDUnits> is used as an example only and could clearly be any of the TDWG standards. d100 1 d102 1 a102 1 -- Markus Doering - 24 May 2004 @ 1.1 log @none @ text @d1 1 a1 1 %META:TOPICINFO{author="GregorHagedorn" date="1085161896" format="1.0" version="1.1"}% d19 91 a109 2 --- @