data_imports

Services

RPC methods exposed by this API.

DataImportService

Method	Request	Response
CreateDataImportFromUrl	CreateDataImportFromUrlRequest	CreateDataImportFromUrlResponse
CreateDataImportFromUpload	CreateDataImportFromUploadRequest	CreateDataImportFromUploadResponse
DetectConfig	DetectConfigRequest	DetectConfigResponse
ListDataImports	ListDataImportsRequest	ListDataImportsResponse
RetryDataImport	RetryDataImportRequest	RetryDataImportResponse
GetDataImport	GetDataImportRequest	GetDataImportResponse

Messages

Data models used across requests/responses.

BatchConfig

Field	Type	Label	Description
run_name	string		Shared run for all files. Both are optional. If neither is set, data is ingested without a run. Run information must be set here instead of the per file configs if a Run is desired.
run_id	string
asset_name	string		Default asset name for all files. The asset name must be set here instead of the per file configs.
default_csv_config	CsvConfig		Per file type default configs. Multiple may be set for archives mixing file formats. A BatchFileConfig entry with no format config override inherits the default matching its type.
default_tdms_config	TDMSConfig
default_parquet_config	ParquetConfig		The default_parquet_config does not need to specify footer information. The importer will automatically extract footer information for the user.
default_hdf5_config	Hdf5Config
default_ulog_config	UlogConfig
file_configs	BatchConfig.FileConfigsEntry	repeated	Files to import. Key is the filepath as it appears within the archive. Only files listed here are imported; other archive files are ignored.

BatchConfig.FileConfigsEntry

Field	Type	Label	Description
key	string
value	BatchFileConfig

BatchFileConfig

Field	Type	Description
type	DataTypeKey	Required. Specifies the format importer to use for this file.
oneof config. csv_config	CsvConfig
oneof config. tdms_config	TDMSConfig
oneof config. parquet_config	ParquetConfig	The parquet_config does not need to specify footer information. The importer will automatically extract footer information for the user.
oneof config. hdf5_config	Hdf5Config
oneof config. ulog_config	UlogConfig

Ch10Config

Field	Type	Label	Description
asset_name	string
run_name	string
scale_values	bool

CreateDataImportFromUploadRequest

Field	Type	Label	Description
csv_config	CsvConfig
ch10_config	Ch10Config
tdms_config	TDMSConfig
parquet_config	ParquetConfig
hdf5_config	Hdf5Config
batch_config	BatchConfig
ulog_config	UlogConfig

CreateDataImportFromUploadResponse

Field	Type	Label	Description
upload_url	string
data_import_id	string

CreateDataImportFromUrlRequest

Field	Type	Description
url	string	The url to import. HTTP and S3 urls are supported. If you need to import non-public S3 objects, please contact Sift to set that up.
csv_config	CsvConfig
ch10_config	Ch10Config
tdms_config	TDMSConfig
parquet_config	ParquetConfig
hdf5_config	Hdf5Config
batch_config	BatchConfig
ulog_config	UlogConfig

CreateDataImportFromUrlResponse

Field	Type	Label	Description
data_import_id	string

CsvConfig

Field	Type	Label	Description
asset_name	string
run_name	string
run_id	string		The id of the run to add this data to. If set, `run_name` is ignored.
first_data_row	uint32		The first row to start reading as data. Can be used to skip header rows. The first row in the file is 1.
time_column	CsvTimeColumn	optional
data_columns	CsvConfig.DataColumnsEntry	repeated	A map from column number (1-indexed) to the channel configuration for that column.
num_rows	uint64	optional	This will be read on upload from the file if not set.

CsvConfig.DataColumnsEntry

Field	Type	Label	Description
key	uint32
value	sift.common.type.v1.ChannelConfig

CsvTimeColumn

Field	Type	Label	Description
column_number	uint32		The column number (1-indexed) of the time column.
format	TimeFormat
relative_start_time	google.protobuf.Timestamp	optional

DataImport

Field	Type	Label	Description
data_import_id	string
source_url	string
status	DataImportStatus
error_message	string
created_date	google.protobuf.Timestamp
modified_date	google.protobuf.Timestamp
csv_config	CsvConfig
ch10_config	Ch10Config
tdms_config	TDMSConfig
parquet_config	ParquetConfig
hdf5_config	Hdf5Config
batch_config	BatchConfig
ulog_config	UlogConfig
run_id	string	optional	The run id will be set if the data import ingests to a run once the run is available.
report_id	string	optional	The report id will be set if the data import creates a report once the report is available.
asset_id	string	optional
data_start_time	google.protobuf.Timestamp	optional
data_stop_time	google.protobuf.Timestamp	optional

DetectConfigRequest

Field	Type	Label	Description
data	bytes
type	DataTypeKey

DetectConfigResponse

Field	Type	Label	Description
csv_config	CsvConfig
parquet_config	ParquetConfig
hdf5_config	Hdf5Config
tdms_config	TDMSConfig

GetDataImportRequest

Field	Type	Label	Description
data_import_id	string

GetDataImportResponse

Field	Type	Label	Description
data_import	DataImport

Hdf5Config

Field	Type	Label	Description
asset_name	string
run_name	string
run_id	string		The id of the run to add this data to. If set, `run_name` is ignored.
data	Hdf5DataConfig	repeated
time_format	TimeFormat
relative_start_time	google.protobuf.Timestamp	optional

Hdf5DataConfig

Field	Type	Label	Description
time_dataset	string
time_index	uint64
value_dataset	string
value_index	uint64
channel_config	sift.common.type.v1.ChannelConfig
time_field	string	optional	For compound types, allow specifying which fields to use.
value_field	string	optional

ListDataImportsRequest

Field	Type	Description
page_size	uint32	The maximum number of data imports to return. The service may return fewer than this value. If unspecified, at most 50 data imports will be returned. The maximum value is 1000; values above 1000 will be coerced to 1000. Optional.
page_token	string	A page token, received from a previous `ListDataImports` call. Provide this to retrieve the subsequent page. When paginating, all other parameters provided to `ListDataImports` must match the call that provided the page token. Optional.
filter	string	A Common Expression Language (CEL) filter string. Available fields to filter by are `data_import_id`, `source_url`, `status`. For further information about how to use CELs, please refer to this guide.
order_by	string	How to order the retrieved data imports. Formatted as a comma-separated string i.e. “FIELD_NAME[ desc],…”. Available fields to order_by are `created_date` and `modified_date`. If left empty, items are ordered by `created_date` in ascending order (oldest-first). For more information about the format of this field, read this Example: “created_date desc,modified_date”

ListDataImportsResponse

Field	Type	Label	Description
data_imports	DataImport	repeated
next_page_token	string

ParquetColumn

Field	Type	Label	Description
path	string
column_config	sift.common.type.v1.ChannelConfig

ParquetConfig

Field	Type	Description
asset_name	string
run_name	string
run_id	string	The id of the run to add this data to. If set, `run_name` is ignored.
oneof config. flat_dataset	ParquetFlatDatasetConfig
oneof config. single_channel_per_row	ParquetSingleChannelPerRowConfig
footer_offset	uint64
footer_length	uint32
complex_types_import_mode	ParquetComplexTypesImportMode

ParquetDataColumn

Field	Type	Label	Description
path	string
channel_config	sift.common.type.v1.ChannelConfig

ParquetFlatDatasetConfig

Field	Type	Label	Description
time_column	ParquetTimeColumn
data_columns	ParquetDataColumn	repeated

ParquetSingleChannelPerRowConfig

Field	Type	Label	Description
time_column	ParquetTimeColumn		Timestamp column configuration.
columns	ParquetColumn	repeated	Detected columns.
oneof config. single_channel	ParquetSingleChannelPerRowSingleChannelConfig		Single channel per file configuration.
oneof config. multi_channel	ParquetSingleChannelPerRowMultiChannelConfig		Multi-channel per file configuration.

ParquetSingleChannelPerRowMultiChannelConfig

Field	Type	Label	Description
name_path	string
data_path	string

ParquetSingleChannelPerRowSingleChannelConfig

Field	Type	Label	Description
data_path	string
channel	sift.common.type.v1.ChannelConfig

ParquetTimeColumn

Field	Type	Label
path	string
format	TimeFormat
relative_start_time	google.protobuf.Timestamp	optional

RetryDataImportRequest

Field	Type	Label	Description
data_import_id	string		data_import_id is the id of the data import to retry. You can only retry an import that is a “url” based import (created with CreateDataImportFromUrl) and is in a failed state.

RetryDataImportResponse

This message has no fields.

TDMSConfig

Field	Type	Label	Description
asset_name	string
run_name	string
start_time_override	google.protobuf.Timestamp		Override the wf_start_time metadata field for all channels. Useful if your waveform channels have wf_increment but no wf_start_time (Veristand is guilty of this).
file_size	uint64	optional	Deprecated. The file size in bytes. If the file has truncated chunks, this will be required to pass validation.
run_id	string		The id of the run to add this data to. If set, `run_name` is ignored.
data	TdmsDataConfig	repeated	If no data entries are present attempt to ingest everything relying on the fallback method for any ambiguous channels.
fallback_method	TdmsFallbackMethod		The fallback method for channels with missing timing information.
time_format	TimeFormat	optional	Time format for time channels not using the TDMS timestamp type.
relative_start_time	google.protobuf.Timestamp	optional	Relative start time for channels using a non standard time channel.
import_file_properties	bool		If true, will import TDMS file properties to the run as metadata. Only valid if a run_name or run_id is set.

TdmsDataConfig

Field	Type	Label	Description
group_name	string		The TDMS group name associated with this channel.
channel_name	string		The TDMS channel name associated with this channel.
channel_config	sift.common.type.v1.ChannelConfig		The Sift channel config.
time_channel_name	string	optional	The time channel associated with this channel. If this is empty then we assume it’s a waveform channel with waveform properties.
scaled	bool	optional	Whether to import scaled or raw values. Defaults to True to import scaled values.
complex_component	TdmsComplexComponent	optional	Whether to import the real or imaginary component. Only applies to complex data types. Defaults to real.

UlogConfig

Field	Type	Label	Description
asset_name	string
run_name	string
run_id	string		The id of the run to add this data to. If set, `run_name` is ignored.
data	UlogDataConfig	repeated	Channels to import. Empty imports all detected channels with their defaults; if set, only the listed channels are imported, each with a required channel config.
relative_start_time	google.protobuf.Timestamp	optional	Log-start UTC. When set, it anchors the timeline and takes precedence over the log’s GPS fix; otherwise the GPS fix anchors the timeline. The import fails if neither is available.
info_keys	string	repeated	I/M message info keys to import as run metadata (stored as info.<key>). Empty: none.
param_keys	string	repeated	P parameter names to import as run metadata (stored as param.<name>). Empty: none.
parse_error_policy	UlogParseErrorPolicy		Recoverable parse errors; defaults to failing the import.

UlogDataConfig

Field	Type	Label	Description
channel	string		The full ULog channel name (e.g. “sensor_accel_0.x”).
channel_config	sift.common.type.v1.ChannelConfig		The Sift channel config the channel is imported with.

Enums

Allowed constant values used in message fields.

DataImportStatus

Name	Number	Description
DATA_IMPORT_STATUS_UNSPECIFIED	0
DATA_IMPORT_STATUS_PENDING	1
DATA_IMPORT_STATUS_IN_PROGRESS	2
DATA_IMPORT_STATUS_SUCCEEDED	3
DATA_IMPORT_STATUS_FAILED	4

DataTypeKey

Name	Number	Description
DATA_TYPE_KEY_UNSPECIFIED	0
DATA_TYPE_KEY_CSV	1
DATA_TYPE_KEY_TDMS	2
DATA_TYPE_KEY_CH10	3
DATA_TYPE_KEY_PARQUET_FLATDATASET	4
DATA_TYPE_KEY_PARQUET_SINGLE_CHANNEL_PER_ROW	5
DATA_TYPE_KEY_HDF5	6
DATA_TYPE_KEY_ULOG	7

ParquetComplexTypesImportMode

Name	Number	Description
PARQUET_COMPLEX_TYPES_IMPORT_MODE_UNSPECIFIED	0
PARQUET_COMPLEX_TYPES_IMPORT_MODE_IGNORE	1	Ignore complex types and do not ingest them.
PARQUET_COMPLEX_TYPES_IMPORT_MODE_BOTH	2	Import complex types as both Arrow bytes and JSON strings.
PARQUET_COMPLEX_TYPES_IMPORT_MODE_STRING	3	Import complex types as only JSON strings.
PARQUET_COMPLEX_TYPES_IMPORT_MODE_BYTES	4	Import complex types as only Arrow bytes.

TdmsComplexComponent

Name	Number	Description
TDMS_COMPLEX_COMPONENT_UNSPECIFIED	0	Default is to select the real component.
TDMS_COMPLEX_COMPONENT_REAL	1
TDMS_COMPLEX_COMPONENT_IMAGINARY	2

TdmsFallbackMethod

The fallback method tells the importer how to treat channels without any timing information (i.e, waveform properties or time channels).

Name	Number	Description
TDMS_FALLBACK_METHOD_UNSPECIFIED	0
TDMS_FALLBACK_METHOD_FAIL_ON_ERROR	1	Fails the import if any specified channels have missing timing information.
TDMS_FALLBACK_METHOD_IGNORE_ERROR	2	Ignores channels without any timing information.

TimeFormat

Name	Number	Description
TIME_FORMAT_UNSPECIFIED	0
TIME_FORMAT_RELATIVE_NANOSECONDS	1
TIME_FORMAT_RELATIVE_MICROSECONDS	2
TIME_FORMAT_RELATIVE_MILLISECONDS	3
TIME_FORMAT_RELATIVE_SECONDS	4
TIME_FORMAT_RELATIVE_MINUTES	5
TIME_FORMAT_RELATIVE_HOURS	6
TIME_FORMAT_ABSOLUTE_RFC3339	10
TIME_FORMAT_ABSOLUTE_DATETIME	11
TIME_FORMAT_ABSOLUTE_UNIX_SECONDS	12
TIME_FORMAT_ABSOLUTE_UNIX_MILLISECONDS	13
TIME_FORMAT_ABSOLUTE_UNIX_MICROSECONDS	14
TIME_FORMAT_ABSOLUTE_UNIX_NANOSECONDS	15

UlogParseErrorPolicy

Name	Number	Description
ULOG_PARSE_ERROR_POLICY_UNSPECIFIED	0	Treated as ULOG_PARSE_ERROR_POLICY_FAIL_ON_ERROR.
ULOG_PARSE_ERROR_POLICY_FAIL_ON_ERROR	1	Fail the import on any recoverable parse error.
ULOG_PARSE_ERROR_POLICY_IGNORE_ERROR	2	Import what parsed; skipped records are logged.

​Services

​DataImportService

​Messages

​BatchConfig

​BatchConfig.FileConfigsEntry

​BatchFileConfig

​Ch10Config

​CreateDataImportFromUploadRequest

​CreateDataImportFromUploadResponse

​CreateDataImportFromUrlRequest

​CreateDataImportFromUrlResponse

​CsvConfig

​CsvConfig.DataColumnsEntry

​CsvTimeColumn

​DataImport

​DetectConfigRequest

​DetectConfigResponse

​GetDataImportRequest

​GetDataImportResponse

​Hdf5Config

​Hdf5DataConfig

​ListDataImportsRequest

​ListDataImportsResponse

​ParquetColumn

​ParquetConfig

​ParquetDataColumn

​ParquetFlatDatasetConfig

​ParquetSingleChannelPerRowConfig

​ParquetSingleChannelPerRowMultiChannelConfig

​ParquetSingleChannelPerRowSingleChannelConfig

​ParquetTimeColumn

​RetryDataImportRequest

​RetryDataImportResponse

​TDMSConfig

​TdmsDataConfig

​UlogConfig

​UlogDataConfig

​Enums

​DataImportStatus

​DataTypeKey

​ParquetComplexTypesImportMode

​TdmsComplexComponent

​TdmsFallbackMethod

​TimeFormat

​UlogParseErrorPolicy