Working on documentation for exporters
This commit is contained in:
leogermani
parent
7562e90bce
commit
06e5e6d39a
|
|
@ -0,0 +1,231 @@
|
|||
# The Exporter flow
|
||||
|
||||
This page describes how Tainacan exporters work and is a reference to write your own exporter.
|
||||
|
||||
This documentation is still in construction. A very effective way to learn more on how to build an exporter is to look at the source code of the CSV Exporter class that is included in the tainacan package.
|
||||
|
||||
## Introduction
|
||||
|
||||
Exporters can export items from a single collection, or even create a bunch of collections, taxonomies and items all at once.
|
||||
|
||||
In order to create an Exporter, you have to extend the `\Tainacan\Exporter` Class and register it using the global `Tainacan\Export_Handler->register_exporter()` method.
|
||||
|
||||
This method takes an array as argument, with the defintion of your exporter. These are the expected attributes.
|
||||
|
||||
```
|
||||
@type string $name The name of the exporter. e.g. 'Example exporter'
|
||||
@type string $slug A unique slug for the exporter. e.g. 'example-exporter'
|
||||
@type string $description The exporter description. e.g. 'This is an example exporter description'
|
||||
@type string $class_name The Exporter Class. e.g. '\Tainacan\Exporter\Test_Exporter'
|
||||
@type bool $manual_mapping Wether Tainacan must present the user with an interface to manually choose
|
||||
a mapping standard. This will allow them to export the items mapped to a
|
||||
chosen standard instead of in its original form.
|
||||
|
||||
The exporter will also be able to inform what mappers it supports. See `set_accepted_mapping_methods`
|
||||
and `accept_no_mapping` below.
|
||||
|
||||
Note that this will only work when exporting items from one single collection.
|
||||
|
||||
@type bool $manual_collection Wether Tainacan will let the user choose the source collection.
|
||||
If set to true, Tainacan give the user a select box from where he/she will Choose
|
||||
one (and only one) Collection to export items from.
|
||||
|
||||
Otherwise, the child exporter class must choose the collections somehow
|
||||
(it could be in its options form) and add them to the collections property using `add_collection()`
|
||||
```
|
||||
|
||||
Note that depending on the value of `manual_mapping` and `manual_collection` you will have to implement some methods in your exporter class.
|
||||
|
||||
## Initializing a new exporter
|
||||
|
||||
When the user starts a new export process, he/she first choose which export to use.
|
||||
|
||||
Once the Exporter is chosen, the first thing that happens is the creation of a new instance of the chosen Exporter. This fires the `__construct()` method.
|
||||
|
||||
|
||||
## Choose a collection (if `manual_collection` is true)
|
||||
|
||||
After choosing the exporter, user will be given the choice to choose the source collection.
|
||||
|
||||
If called from inside a collection, this step is skipped and the current collection is set as source.
|
||||
|
||||
## Set options
|
||||
|
||||
Now its time to set the exporter options. Each exporter may have its own set of options, that will be used during the export process. It could be anything, from the delimiter character in a CSV exporter, to an API key for a exporter that sends data to an external API.
|
||||
|
||||
exporter classes should declare the default values for its options in the `__construct()` method, by calling `set_default_options()`.
|
||||
|
||||
```
|
||||
namespace Tainacan\exporter;
|
||||
|
||||
class Myexporter extends Exporter {
|
||||
function __construct() {
|
||||
parent::construct();
|
||||
$this->set_default_options(['
|
||||
'foo' => 'bar'
|
||||
']);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The exporter classes must also implement the `options_form` method, in which it will echo the form that the user will interact to set the options.
|
||||
|
||||
```
|
||||
function options_form() {
|
||||
$form = '<div class="field">';
|
||||
$form .= '<label class="label">' . __('My exporter Option 1', 'tainacan') . '</label>';
|
||||
$form .= '<div class="control">';
|
||||
$form .= '<input type="text" class="input" name="my_exporter_option_1" value="' . $this->get_option('my_exporter_option_1') . '" />';
|
||||
$form .= '</div>';
|
||||
$form .= '</div>';
|
||||
|
||||
return $form;
|
||||
}
|
||||
```
|
||||
## Accepting mappers
|
||||
|
||||
If you set `manual_mapping` as true and want to give users the ability to export a mapped version of their collections, you can also
|
||||
define which mappers you accept.
|
||||
|
||||
```
|
||||
public function __construct($attributes = array()) {
|
||||
parent::__construct($attributes);
|
||||
$this->set_accepted_mapping_methods('any'); // set all method to mapping
|
||||
$this->accept_no_mapping = true;
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
`$this->set_accepted_mapping_methods()` lets you do that. If you set it to "any", all mappers will be available. If you set it to "list", you can then pass a second argument with the list of mappers you want to be available for the users.
|
||||
|
||||
```
|
||||
public function __construct($attributes = array()) {
|
||||
parent::__construct($attributes);
|
||||
$this->set_accepted_mapping_methods('list', [ "dublin-core" ]); // set specific list of methods to mapping
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
Finally, `$this->accept_no_mapping = true;` informs that you also allow users to export items in their original form, without any mapping. In other words, to choose a Mapper is not mandatory if this is set to true.
|
||||
|
||||
## exporter steps
|
||||
|
||||
An exporter may have several steps, that will handle different parts of the process. Each step will be handled by a different callback in the exporter class.
|
||||
|
||||
First, lets have a look at a simple CSV exporter, that only have one step, in which it exports the items from the source collection into a CSV file. After that we will have a look on how to create custom steps.
|
||||
|
||||
### Simple exporter - One step that exports items
|
||||
|
||||
By default, exporters must implement 3 methods: `output_header`, `output_footer` and `process_item`.
|
||||
|
||||
`output_header` and `output_footer` are invoked at the beginning and end of each collection being exported. It can be useful, for example, to write the header and footer of a csv or xml file.
|
||||
|
||||
Inside this methods you may do whatever you want. It could be a POST to an API or to write a new file (see handling files below).
|
||||
|
||||
The `process_item` method will be invoked for every item being exported. Tainacan will automatically loop through the collections in the queue, fetch and prepare the items, and then send them, one at a time, to this method. (see adding collections below).
|
||||
|
||||
This method gets two parameters, the `$item` object (instance of \Tainacan\Entities\Item), and its `$metadata`, with an array of '\Tainacan\Entities\Item_Metadata_Entity' objects.
|
||||
|
||||
Note that, in this method, you dont need to bother about mapping. If a mapper was chosen by the user, you will receive this array of metadata already mapped, which means they will be only the ones defined in the mapper. (Note that, if a collection fail to map one of its metadata to a metadata expected by the chosen mapper, they will be empy elements in this array).
|
||||
|
||||
Now you can loop through the metadata and access any propery you want from the item to do whatever you like. In the case of the CSV exporter, it will add one line to the CSV file.
|
||||
|
||||
If you need to access the Mapper object, with everything about the the chosen mapper standard, you can do so by calling `$this->get_current_mapper()`. If a object is not returned, it means no mapper was selected by the user.
|
||||
|
||||
#### Adding collections
|
||||
|
||||
If your exporter does not use the `manual_collection` option, you might want to programatically add collections to be exported to the queue.
|
||||
|
||||
To add or remove a collection from the queue, use the `add_collection()` and `remove_collection()` methods passing the collection definition.
|
||||
|
||||
The collection definition is an array with their IDs and the total number of items to be exported.
|
||||
|
||||
|
||||
Example of the structure of this propery for one collection:
|
||||
|
||||
```
|
||||
[
|
||||
'id' => 12,
|
||||
'total_items' => 1234,
|
||||
]
|
||||
```
|
||||
|
||||
#### Handling files
|
||||
|
||||
Your export may generate one or more files, that will be downloaded by the user as a package at the end.
|
||||
|
||||
To create and write contents to a file, use the `append_to_file()` method.
|
||||
|
||||
It takes 2 arguments: `$key` and `$data`. `$key` is the file identifier and will serve as the filename, prepended with the ID of the process. If a file with this key does not exist yet, it will be created. `$data` is the content to be appended to the file.
|
||||
|
||||
Dont forget to add a link to the generated file at the output at the end of the process. This will be the only way users will have to get them. See "Final output" below).
|
||||
|
||||
TODO: If more than one file was created, Tainacan will create a zip of them at the end. Yet to be implemented.
|
||||
|
||||
#### Using transients
|
||||
|
||||
Since exporters are handled as background processes, they will run accross multiple requests. For that reason, you can not simply add properties to your class and trust their values will be kept during all the time the process is running.
|
||||
|
||||
If you want a value to persist so you can use it accross all methods of your exporter, at any time, you should use `transients` to store them in the database.
|
||||
|
||||
This is as simple as calling `set_transient()` and `get_transient()`. See the example below:
|
||||
|
||||
|
||||
```
|
||||
|
||||
function callback_for_step_1() {
|
||||
|
||||
|
||||
$this->add_transient('time_step_1', time());
|
||||
|
||||
}
|
||||
|
||||
// ...
|
||||
|
||||
function callback_for_step_5() {
|
||||
|
||||
$time_step_1 = get_transient('time_step_1');
|
||||
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
|
||||
#### Handling user feedback
|
||||
|
||||
There are two information Tainacan exporters give to the user about the status of the process while it is running in feedback. the `progress label` and the `progress value`.
|
||||
|
||||
The `progress label` is a string that can be anything that tells the user what is going on. By default, it is the Step Name, but you can inform a specific `progress_label` attribute when registering the steps.
|
||||
|
||||
The `progress value` is a number between 0 and 100 that indicates the progress of the current step or the whole exporter, Thats up to you. By default, it calculates it automatically using the `total` attribute registered with the steps, against the `$this->get_in_step_count()` value. In the case of the default Process Items callback, it calculates based on the number of items found in each collection.
|
||||
|
||||
See the `finish_processing` dummy callback in the Test Importer. You will notice that when we registered the step, we informed a `total` attribute to this step with the value of 5. This will tell Tainacan that the total number iterations this step need to complete is 5 and allow it to calculate the progress.
|
||||
|
||||
If it is no possible to know `total` of a step beforehand, you can set it at any time, even inside the step callback itself, using the `set_current_step_total($value)` or `set_step_total($step, $value)` methods.
|
||||
|
||||
|
||||
##### Final output
|
||||
|
||||
When the process finishes, Background processes define an "output", which is a final report to the user of what happened.
|
||||
|
||||
This could be simply a message saying "All good", or could be a report with the names and links to the collections created. HTML is welcome.
|
||||
|
||||
Remember that for a more detailed and technical report, you should use Logs (see Logs below). This output is meant as a short but descriptive user friendly message.
|
||||
|
||||
In order to achieve this, exporters must implement a method called `get_output()` that returns a string.
|
||||
|
||||
This method will be called only once, when the exporter ends, so you might need to save information using `transients` during the process and use them in `get_output()` to compose your message.
|
||||
|
||||
To get a list of the generated files to display to users, use `$this->get_output_files()`.
|
||||
|
||||
#### Logs
|
||||
|
||||
There are two useful methods to write information to the logs: `add_log()` and `add_error_log()`. These are written into a log file related to the exporter background process and a link to it will be presented to the user.
|
||||
|
||||
|
||||
|
||||
## Run exporter
|
||||
|
||||
Finally, everything is ready. The exporter runs.
|
||||
|
||||
This will trigger a Background Process (documentation needed) and the exporter will run through as many background requests as needed.
|
||||
|
|
@ -40,7 +40,7 @@ When the user starts a new import process, he/she first choose which import to u
|
|||
Once the Importer is chosen, the first thing that happens is the creation of a new instance of the chosen Importer. This fires the `__construct()` method.
|
||||
|
||||
|
||||
## Choose a collection (if manual_mapping is true)
|
||||
## Choose a collection (if `manual_collection` is true)
|
||||
|
||||
After choosing the importer, user will be given the choice to choose the destination collection.
|
||||
|
||||
|
|
@ -131,15 +131,15 @@ public function get_source_metadata(){
|
|||
|
||||
An Importer may have several steps, that will handle different parts of the process. Each step will be handled by a different callback in the Importer class.
|
||||
|
||||
First, lets have a look at a simple CSV importer, that only have one steps, in which it imports the items from the source into a chosen collection. After that we will have a look on how to create custom steps.
|
||||
First, lets have a look at a simple CSV importer, that only have one step, in which it imports the items from the source into a chosen collection. After that we will have a look on how to create custom steps.
|
||||
|
||||
### Simple Importer - One step that imports items
|
||||
|
||||
By default, the only method an Importer class must implement to functino is the `process_item()` class.
|
||||
By default, the only method an Importer class must implement to function is the `process_item()` class.
|
||||
|
||||
This method gets two parameters, the `$index` of the item to be inserted, and the `$collection_definition`, with information on the target collection.
|
||||
|
||||
Inside this metho you must fetch the item from the source and format it according to the `mapping` definition of the collection.
|
||||
Inside this method you must fetch the item from the source and format it according to the `mapping` definition of the collection.
|
||||
|
||||
The `mapping` defines how the item metadata from the source should be mapped to the metadata present in the target collection. It was created either manually, by the user, or programatically by the importer in an earlier step (see advanced importers below). This is an array where the keys are the `metadata IDs` and the values are the `identifers` found in source.
|
||||
|
||||
|
|
@ -159,7 +159,7 @@ By default, Tainacan Importer super class is registering one single step to the
|
|||
]
|
||||
```
|
||||
|
||||
This step will lopp though all the collections added to the importer (manuall or programatically) and add the items to it.
|
||||
This step will lopp through all the collections added to the importer (manuall or programatically) and add the items to it.
|
||||
|
||||
You may register as many steps and callbacks as you want in your importer, but you should consider keeping this default step at some point to handle the items insertion. For example, see how the Test Importer adds other steps before and after but keeps this default step in the middle:
|
||||
|
||||
|
|
@ -208,7 +208,7 @@ class Test_Importer extends Importer {
|
|||
|
||||
Each step has its own callback. The callback may do anything necessary, just keep in mind that you should allow the importer to break very long processes into several requests.
|
||||
|
||||
In order to that, your step callback might be called several times, and each time run a part of the process and returnt its current status, until its done.
|
||||
In order to that, your step callback might be called several times, and each time run a part of the process and return its current status, until its done.
|
||||
|
||||
When you run the importer, Tainacan will automatically iterate over your steps. If a step callback returns `false`, it assumes the step is over and it will pass to the next step in the next iteration. If the step callback returns an integer, it will keep the pointer in this step and call the same step again in the next iteration. The current position, which is the integer returned the last time the callback was invoked, will be accessible via the `$this->get_in_step_count()` method.
|
||||
|
||||
|
|
|
|||
|
|
@ -24,7 +24,6 @@ class CSV extends Exporter {
|
|||
|
||||
public function process_item( $item, $metadata ) {
|
||||
|
||||
$mapper = $this->get_current_mapper();
|
||||
$line = [];
|
||||
|
||||
$line[] = $item->get_id();
|
||||
|
|
@ -33,6 +32,8 @@ class CSV extends Exporter {
|
|||
|
||||
foreach ($metadata as $meta_key => $meta) {
|
||||
|
||||
// if (!$meta) means this metadata is not mapped in the collection so there is no value for it
|
||||
// an empty value must be returned so the number of columns matches the header
|
||||
if (!$meta) {
|
||||
$line[] = '';
|
||||
continue;
|
||||
|
|
|
|||
|
|
@ -717,6 +717,12 @@ class Exporter extends CommunImportExport {
|
|||
];
|
||||
}
|
||||
|
||||
/**
|
||||
* Append content to a file. Create the file if it does not exist
|
||||
*
|
||||
* @param string $key The file identifier. (it is the name of the file, with extension, and will be prefixed with the process ID)
|
||||
* @param string $data The content to be appended to the file
|
||||
*/
|
||||
public function append_to_file($key, $data) {
|
||||
if ( array_key_exists ( $key , $this->output_files ) ) {
|
||||
$fp = fopen($this->output_files[$key]['filename'], 'a');
|
||||
|
|
|
|||
Loading…
Reference in New Issue