BatchPredictionJob : sync=False and wait_for_resource_creation()

[dparkar]

I want to launch a bunch of AutoMLTabularTrainingJob(s) asynchronously and as each training job completes successfully, launch it's corresponding BatchPredictionJob asynchronously and then once the prediction job completes successfully, download those predictions and score locally.

I am able to launch AutoMLTabularTrainingJob asynchronously using sync=False, then starting the job using job.run and then job.wait_for_resource_creation(). After this I am able to access job.state to wait on the job to complete.

However, I am unable to do the same with the BatchPredictionJob. BatchPredictionJob doesn't seem to follow the "construct and run" pattern hence there is no job.wait_for_resource_creation(). Because of this, once a BatchPredictionJob is created, any attempt to access its properties results in "RuntimeError: BatchPredictionJob resource has not been created."

prediction_run = model.batch_predict(
        job_display_name="my_prediction_job",
        gcs_source="gs://test_dataset.csv",
        instances_format="csv",
        gcs_destination_prefix="gs://predictions.csv",
        predictions_format="csv",
        sync=False)

while prediction_run.state == job_state.JobState.JOB_STATE_RUNNING: # RuntimeError: BatchPredictionJob resource has not been created.
       time.sleep(60)

print("Prediction run is complete.")

[dparkar]

Awesome. Thanks @sasha-gitg !