path mgmt/v1alpha1/job.proto
package mgmt.v1alpha1
Messages
ActivityFailure
Name | Type | Description |
---|
message | string | |
ActivityOptions
Config that contains various timeouts that are configured in the underlying temporal workflow(s) and activities
Name | Type | Description |
---|
schedule_to_close_timeout | optional int64 | Total time that a workflow is willing to wait for an activity to complete, including retries.
Measured in seconds |
start_to_close_timeout | optional int64 | Max time of a single Temporal Activity execution attempt.
This timeout should be as short as the longest psosible execution of any activity (e.g. table sync).
Important to know that this is per retry attempt. Defaults to the schedule to close timeout if not provided.
Measured in seconds |
retry_policy | RetryPolicy | Optionally define a retry policy for the activity
If max attempts is not set, the activity will retry indefinitely until the start to close timeout lapses |
AiGenerateSourceOptions
Name | Type | Description |
---|
ai_connection_id | string | The connection id that corresponds with an AI-based Neosync connection |
schemas | repeated AiGenerateSourceSchemaOption | The list of schemas (and their tables) along with any configuration options that will be used to generate data for. |
fk_source_connection_id | optional string | An optional connection id that will be used as the basis for the shape of data to be generated. |
model_name | string | The name of the model to use |
user_prompt | optional string | Optionally provide a user prompt to give more context to the schema |
generate_batch_size | optional int64 | The batch size that will be used to generate X number of records. This is global and will be applied to all tables configured. |
AiGenerateSourceSchemaOption
Name | Type | Description |
---|
schema | string | The dataabase schema |
tables | repeated AiGenerateSourceTableOption | The list of tables (and their configuration) that reside within the schema to receive generated data |
AiGenerateSourceTableOption
Name | Type | Description |
---|
table | string | The table that will be used to. 1. The schema of the table will be injected into the prompt, of which the resulting data will then be inserted. |
row_count | int64 | The total number of records to be generated. |
AwsS3DestinationConnectionOptions
Name | Type | Description |
---|
storage_class | AwsS3DestinationConnectionOptions.StorageClass | The storage class that will be used when objects are written to S3 |
max_in_flight | optional uint32 | The maximum number of batched messages to have in flight at a given time. Increase this to improve throughput. |
timeout | optional string | The maximum period (duration string) to wait on an upload before abandoning it and reattempting. |
batch | BatchConfig | Configure batching options to more efficiently store records in S3 |
AwsS3SourceConnectionOptions
Name | Type | Description |
---|
connection_id | string | |
BatchConfig
Name | Type | Description |
---|
count | optional uint32 | The max allowed in a batch before it is flushed. 0 to disable. |
period | optional string | A duration string in which an incomplete batch should be flushed regardless of the count.
Examples are 1s, 1m, 500ms |
CancelJobRunRequest
Name | Type | Description |
---|
job_run_id | string | |
account_id | string | |
CancelJobRunResponse
ColumnError
Name | Type | Description |
---|
schema | string | |
table | string | |
column | string | |
errors | repeated string | |
CreateJobDestination
CreateJobDestinationConnectionsRequest
CreateJobDestinationConnectionsResponse
Name | Type | Description |
---|
job | Job | |
CreateJobRequest
Name | Type | Description |
---|
account_id | string | The unique account identifier that this job will be associated with |
job_name | string | The unique, friendly name of the job. This is unique per account |
cron_schedule | optional string | Optionally provide a cron schedule. Goes into effect if the job status is set to enabled |
mappings | repeated JobMapping | |
source | JobSource | |
destinations | repeated CreateJobDestination | |
initiate_job_run | bool | Initially trigger a run of this job regardless of its status or cron schedule |
workflow_options | WorkflowOptions | Specify timeouts and other workflow options for the underlying temporal workflow |
sync_options | ActivityOptions | Specify timeout and retry options for data synchronization activities
Data sync activities are any piece of work that involves actually synchronizing data from a source to a destination
For the data sync and generate jobs, this will be applied per table |
virtual_foreign_keys | repeated VirtualForeignConstraint | |
CreateJobResponse
Name | Type | Description |
---|
job | Job | |
CreateJobRunRequest
Name | Type | Description |
---|
job_id | string | |
CreateJobRunResponse
DatabaseError
Name | Type | Description |
---|
errors | repeated string | |
DeleteJobDestinationConnectionRequest
Name | Type | Description |
---|
destination_id | string | |
DeleteJobDestinationConnectionResponse
DeleteJobRequest
Name | Type | Description |
---|
id | string | |
DeleteJobResponse
DeleteJobRunRequest
Name | Type | Description |
---|
job_run_id | string | |
account_id | string | |
DeleteJobRunResponse
DynamoDBDestinationConnectionOptions
Configuration for DynamoDB Destination Connection Job Options
DynamoDBDestinationTableMapping
Configuration for mapping a source table to a destination table for DynamoDB
Name | Type | Description |
---|
source_table | string | The name of the incoming source table |
destination_table | string | The name of the outgoing destination table |
DynamoDBSourceConnectionOptions
DynamoDB connection options for a job source
Name | Type | Description |
---|
connection_id | string | The unique connection id to a dynamodb connection configuration |
tables | repeated DynamoDBSourceTableOption | List of table option configurations for any mapped source table.
Any table listed in this must also be present as a job mapping table to be applied. |
unmapped_transforms | DynamoDBSourceUnmappedTransformConfig | Default transformations for any unmapped keys |
enable_consistent_read | bool | Enforces strong read consistency
False: Eventually Consistent Reads, True: Strongly Consistent Reads
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadConsistency.html |
DynamoDBSourceSchemaSubset
DynamoDBSourceTableOption
Name | Type | Description |
---|
table | string | The table that this configuration will be applied to |
where_clause | optional string | An optional PartiQL query that may be used for subsetting the DynamoDB table.
This is not a parameterized query and must be valid. Intended to be everything after the WHERE keyword. |
GcpCloudStorageDestinationConnectionOptions
Configuration for Google Cloud Storage Destination Connection Job Options
GenerateSourceOptions
GenerateSourceSchemaOption
GenerateSourceTableOption
Name | Type | Description |
---|
table | string | |
row_count | int64 | |
GetJobNextRunsRequest
Name | Type | Description |
---|
job_id | string | |
GetJobNextRunsResponse
GetJobRecentRunsRequest
Name | Type | Description |
---|
job_id | string | |
GetJobRecentRunsResponse
GetJobRequest
Name | Type | Description |
---|
id | string | |
GetJobResponse
Name | Type | Description |
---|
job | Job | |
GetJobRunEventsRequest
Name | Type | Description |
---|
job_run_id | string | |
account_id | string | |
GetJobRunEventsResponse
GetJobRunLogsStreamRequest
Name | Type | Description |
---|
job_run_id | string | |
account_id | string | |
window | LogWindow | The time window in which to retrieve the logs |
should_tail | bool | Whether or not to tail the stream. Note: only works with k8s-pods and is not currently supported with Loki logs |
max_log_lines | optional int64 | Optionally provide a max log limit |
log_levels | repeated LogLevel | Provide a list of log levels to filter by. If any of these are UNSPECIFIED, all log levels are returned. |
GetJobRunLogsStreamResponse
Name | Type | Description |
---|
log_line | string | |
timestamp | optional google.protobuf.Timestamp | |
GetJobRunRequest
Name | Type | Description |
---|
job_run_id | string | |
account_id | string | |
GetJobRunResponse
Name | Type | Description |
---|
job_run | JobRun | |
GetJobRunsRequest
Name | Type | Description |
---|
job_id | string | |
account_id | string | |
GetJobRunsResponse
GetJobStatusRequest
Name | Type | Description |
---|
job_id | string | |
GetJobStatusResponse
GetJobStatusesRequest
Name | Type | Description |
---|
account_id | string | |
GetJobStatusesResponse
GetJobsRequest
Name | Type | Description |
---|
account_id | string | |
GetJobsResponse
GetRunContextRequest
GetRunContextResponse
Name | Type | Description |
---|
value | bytes | |
IsJobNameAvailableRequest
Name | Type | Description |
---|
name | string | |
account_id | string | |
IsJobNameAvailableResponse
Name | Type | Description |
---|
is_available | bool | |
Job
Name | Type | Description |
---|
id | string | The unique identifier of the job |
created_by_user_id | string | |
created_at | google.protobuf.Timestamp | |
updated_by_user_id | string | |
updated_at | google.protobuf.Timestamp | |
name | string | The unique, friendly name of the job |
source | JobSource | |
destinations | repeated JobDestination | |
mappings | repeated JobMapping | |
cron_schedule | optional string | |
account_id | string | The account identifier that a job is associated with |
sync_options | ActivityOptions | Specify timeout and retry options for data synchronization activities
Data sync activities are any piece of work that involves actually synchronizing data from a source to a destination
For the data sync and generate jobs, this will be applied per table |
workflow_options | WorkflowOptions | Specify timeouts and other workflow options for the underlying temporal workflow |
virtual_foreign_keys | repeated VirtualForeignConstraint | |
JobDestination
JobDestinationOptions
JobMapping
JobNextRuns
Name | Type | Description |
---|
next_run_times | repeated google.protobuf.Timestamp | |
JobRecentRun
Name | Type | Description |
---|
start_time | google.protobuf.Timestamp | |
job_run_id | string | |
JobRun
Name | Type | Description |
---|
id | string | The id of the job run. This will currently be equivalent to the temporal workflow id |
job_id | string | The unique identifier of the job id this run is associated with |
name | string | The name of the job run. |
status | JobRunStatus | the status of the job run |
started_at | google.protobuf.Timestamp | A timestamp of when the run started |
completed_at | optional google.protobuf.Timestamp | Available if the run completed or has not yet been archived by the system |
pending_activities | repeated PendingActivity | Pending activities are only returned when retrieving a specific job run and will not be returned when requesting job runs in list format |
JobRunEvent
JobRunEventTask
JobRunEventTaskError
Name | Type | Description |
---|
message | string | |
retry_state | string | |
Name | Type | Description |
---|
schema | string | |
table | string | |
JobSource
JobSourceOptions
JobSourceSqlSubetSchemas
JobStatusRecord
Name | Type | Description |
---|
job_id | string | |
status | JobStatus | |
MongoDBDestinationConnectionOptions
MongoDBSourceConnectionOptions
MongoDB connection options for a job source
Name | Type | Description |
---|
connection_id | string | The unique connection id to a mongo connection configuration |
MssqlDestinationConnectionOptions
Name | Type | Description |
---|
truncate_table | MssqlTruncateTableConfig | |
init_table_schema | bool | Currently not supported and a placeholder for future implementation |
on_conflict | MssqlOnConflictConfig | Currently not supported and a placeholder for future implementation |
skip_foreign_key_violations | bool | Insert all valid records, skipping any that violate foreign key constraints. |
batch | BatchConfig | Configure batching options to handle how much data is sent to your database at once. |
max_in_flight | optional uint32 | Determines the maximum number of parallel batched inserts. |
MssqlOnConflictConfig
Name | Type | Description |
---|
do_nothing | bool | |
MssqlSourceConnectionOptions
MssqlSourceSchemaOption
MssqlSourceSchemaSubset
MssqlSourceTableOption
Name | Type | Description |
---|
table | string | |
where_clause | optional string | |
MssqlTruncateTableConfig
Name | Type | Description |
---|
truncate_before_insert | bool | |
MysqlDestinationConnectionOptions
Name | Type | Description |
---|
truncate_table | MysqlTruncateTableConfig | |
init_table_schema | bool | |
on_conflict | MysqlOnConflictConfig | |
skip_foreign_key_violations | bool | Insert all valid records, skipping any that violate foreign key constraints. |
batch | BatchConfig | Configure batching options to handle how much data is sent to your database at once. |
max_in_flight | optional uint32 | Determines the maximum number of parallel batched inserts. |
MysqlOnConflictConfig
Name | Type | Description |
---|
do_nothing | bool | |
MysqlSourceConnectionOptions
MysqlSourceSchemaOption
MysqlSourceSchemaSubset
MysqlSourceTableOption
Name | Type | Description |
---|
table | string | |
where_clause | optional string | |
MysqlTruncateTableConfig
Name | Type | Description |
---|
truncate_before_insert | bool | |
PauseJobRequest
Name | Type | Description |
---|
id | string | |
pause | bool | |
note | optional string | |
PauseJobResponse
Name | Type | Description |
---|
job | Job | |
PendingActivity
PostgresDestinationConnectionOptions
Name | Type | Description |
---|
truncate_table | PostgresTruncateTableConfig | |
init_table_schema | bool | |
on_conflict | PostgresOnConflictConfig | |
skip_foreign_key_violations | bool | Insert all valid records, skipping any that violate foreign key constraints. |
batch | BatchConfig | Configure batching options to handle how much data is sent to your database at once. |
max_in_flight | optional uint32 | Determines the maximum number of parallel batched inserts. |
PostgresOnConflictConfig
Name | Type | Description |
---|
do_nothing | bool | |
PostgresSourceConnectionOptions
PostgresSourceSchemaOption
PostgresSourceSchemaSubset
PostgresSourceTableOption
Name | Type | Description |
---|
table | string | |
where_clause | optional string | |
PostgresTruncateTableConfig
Name | Type | Description |
---|
truncate_before_insert | bool | |
cascade | bool | |
RetryPolicy
Defines the retry policy for an activity
Name | Type | Description |
---|
maximum_attempts | optional int32 | Maximum number of attempts. When exceeded the retries stop even if not expired yet.
If not set or set to 0, it means unlimited, and rely on activity ScheduleToCloseTimeout to stop. |
RunContextKey
Name | Type | Description |
---|
job_run_id | string | The Neosync Run ID |
external_id | string | An opaque identifier that will be used to store specific items |
account_id | string | The Neosync Account ID |
SetJobSourceSqlConnectionSubsetsRequest
Name | Type | Description |
---|
id | string | The unique identifier of the job to update subsets for |
schemas | JobSourceSqlSubetSchemas | The subset configuration |
subset_by_foreign_key_constraints | bool | Whether or not to have subsets follow foreign key constraints (for connections that support it) |
SetJobSourceSqlConnectionSubsetsResponse
Name | Type | Description |
---|
job | Job | |
SetJobSyncOptionsRequest
Name | Type | Description |
---|
id | string | The unique identifier of the job |
sync_options | ActivityOptions | The sync options object. The entire object must be provided and will fully overwrite the previous result |
SetJobSyncOptionsResponse
Name | Type | Description |
---|
job | Job | |
SetJobWorkflowOptionsRequest
Name | Type | Description |
---|
id | string | The unique identifier of the job |
worfklow_options | WorkflowOptions | The workflow options object. The entire object must be provided and will fully overwrite the previous result |
SetJobWorkflowOptionsResponse
Name | Type | Description |
---|
job | Job | |
SetRunContextRequest
Name | Type | Description |
---|
id | RunContextKey | |
value | bytes | An opaque value that is to be determined by the key |
SetRunContextResponse
SetRunContextsRequest
Name | Type | Description |
---|
id | RunContextKey | |
value | bytes | An opaque value that is to be determined by the key |
SetRunContextsResponse
TerminateJobRunRequest
Name | Type | Description |
---|
job_run_id | string | |
account_id | string | |
TerminateJobRunResponse
UpdateJobDestinationConnectionRequest
UpdateJobDestinationConnectionResponse
Name | Type | Description |
---|
job | Job | |
UpdateJobScheduleRequest
Name | Type | Description |
---|
id | string | |
cron_schedule | optional string | |
UpdateJobScheduleResponse
Name | Type | Description |
---|
job | Job | |
UpdateJobSourceConnectionRequest
UpdateJobSourceConnectionResponse
Name | Type | Description |
---|
job | Job | |
ValidateJobMappingsRequest
ValidateJobMappingsResponse
VirtualForeignConstraint
Name | Type | Description |
---|
schema | string | |
table | string | |
columns | repeated string | |
foreign_key | VirtualForeignKey | |
VirtualForeignKey
Name | Type | Description |
---|
schema | string | |
table | string | |
columns | repeated string | |
WorkflowOptions
Config that contains various timeouts that are configured in the underlying temporal workflow
More options will come in the future as needed
Name | Type | Description |
---|
run_timeout | optional int64 | The timeout for a single workflow run.
Measured in seconds |
Enums
ActivityStatus
Name | Number | Description |
---|
ACTIVITY_STATUS_UNSPECIFIED | 0 | |
ACTIVITY_STATUS_SCHEDULED | 1 | |
ACTIVITY_STATUS_STARTED | 2 | |
ACTIVITY_STATUS_CANCELED | 3 | |
ACTIVITY_STATUS_FAILED | 4 | |
AwsS3DestinationConnectionOptions.StorageClass
Name | Number | Description |
---|
STORAGE_CLASS_UNSPECIFIED | 0 | |
STORAGE_CLASS_STANDARD | 1 | |
STORAGE_CLASS_REDUCED_REDUNDANCY | 2 | |
STORAGE_CLASS_GLACIER | 3 | |
STORAGE_CLASS_STANDARD_IA | 4 | |
STORAGE_CLASS_ONEZONE_IA | 5 | |
STORAGE_CLASS_INTELLIGENT_TIERING | 6 | |
STORAGE_CLASS_DEEP_ARCHIVE | 7 | |
JobRunStatus
An enumeration of job run statuses.
Name | Number | Description |
---|
JOB_RUN_STATUS_UNSPECIFIED | 0 | if the job run status is unknown |
JOB_RUN_STATUS_PENDING | 1 | the run is pending and has not started yet |
JOB_RUN_STATUS_RUNNING | 2 | the run is currently in progress |
JOB_RUN_STATUS_COMPLETE | 3 | the run has successfully completed |
JOB_RUN_STATUS_ERROR | 4 | the run ended with an error |
JOB_RUN_STATUS_CANCELED | 5 | the run was cancelled |
JOB_RUN_STATUS_TERMINATED | 6 | the run was terminated |
JOB_RUN_STATUS_FAILED | 7 | the run ended in failure |
JOB_RUN_STATUS_TIMED_OUT | 8 | the run was ended pre-maturely due to timeout |
JobStatus
Name | Number | Description |
---|
JOB_STATUS_UNSPECIFIED | 0 | |
JOB_STATUS_ENABLED | 1 | |
JOB_STATUS_PAUSED | 3 | |
JOB_STATUS_DISABLED | 4 | |
LogLevel
Name | Number | Description |
---|
LOG_LEVEL_UNSPECIFIED | 0 | |
LOG_LEVEL_DEBUG | 1 | |
LOG_LEVEL_INFO | 2 | |
LOG_LEVEL_WARN | 3 | |
LOG_LEVEL_ERROR | 4 | |
LogWindow
Name | Number | Description |
---|
LOG_WINDOW_NO_TIME_UNSPECIFIED | 0 | |
LOG_WINDOW_FIFTEEN_MIN | 1 | |
LOG_WINDOW_ONE_HOUR | 2 | |
LOG_WINDOW_ONE_DAY | 3 | |
Services
JobService
GetJobs
GetJob
CreateJob
DeleteJob
IsJobNameAvailable
UpdateJobSchedule
UpdateJobSourceConnection
SetJobSourceSqlConnectionSubsets
UpdateJobDestinationConnection
DeleteJobDestinationConnection
CreateJobDestinationConnections
PauseJob
GetJobRecentRuns
Method | GetJobRecentRuns |
---|
Request | GetJobRecentRunsRequest |
---|
Response | GetJobRecentRunsResponse |
---|
Description | Returns a list of recently invoked job runs baseds on the Temporal cron scheduler. This will return a list of job runs that include archived runs |
---|
GetJobNextRuns
GetJobStatus
GetJobStatuses
GetJobRuns
GetJobRunEvents
GetJobRun
DeleteJobRun
CreateJobRun