Record selector
The record selector is responsible for translating an HTTP response into a list of Airbyte records by extracting records from the response and optionally filtering and shaping records based on a heuristic. Schema:
HttpSelector:
type: object
anyOf:
- "$ref": "#/definitions/RecordSelector"
RecordSelector:
type: object
required:
- extractor
properties:
"$parameters":
"$ref": "#/definitions/$parameters"
extractor:
"$ref": "#/definitions/RecordExtractor"
record_filter:
"$ref": "#/definitions/RecordFilter"
The current record extraction implementation uses dpath to select records from the json-decoded HTTP response.
For nested structures *
can be used to iterate over array elements.
Schema:
DpathExtractor:
type: object
additionalProperties: true
required:
- field_path
properties:
"$parameters":
"$ref": "#/definitions/$parameters"
field_path:
type: array
items:
type: string
Common recipes:
Here are some common patterns:
Selecting the whole response
If the root of the response is an array containing the records, the records can be extracted using the following definition:
selector:
extractor:
field_path: [ ]
If the root of the response is a json object representing a single record, the record can be extracted and wrapped in an array. For example, given a response body of the form
{
"id": 1
}
and a selector
selector:
extractor:
field_path: [ ]
The selected records will be
[
{
"id": 1
}
]
Selecting a field
Given a response body of the form
{
"data": [{"id": 0}, {"id": 1}],
"metadata": {"api-version": "1.0.0"}
}
and a selector
selector:
extractor:
field_path: [ "data" ]
The selected records will be
[
{
"id": 0
},
{
"id": 1
}
]
Selecting an inner field
Given a response body of the form
{
"data": {
"records": [
{
"id": 1
},
{
"id": 2
}
]
}
}
and a selector
selector:
extractor:
field_path: [ "data", "records" ]
The selected records will be
[
{
"id": 1
},
{
"id": 2
}
]
Selecting fields nested in arrays
Given a response body of the form
{
"data": [
{
"record": {
"id": "1"
}
},
{
"record": {
"id": "2"
}
}
]
}
and a selector
selector:
extractor:
field_path: [ "data", "*", "record" ]
The selected records will be
[
{
"id": 1
},
{
"id": 2
}
]