BigQuery - Apache Avro library failed to parse the header with the following error: Invalid data file. Magic does not match

Question

I'm messing with avro file for the first time.

I get a lot of csv files with the encoded data and just made the python treatment of the data that should go to BigQuery. In the end I got the schema beat I received in avsc with csv data. So far so good.

Now I understand that I need to create the table in BigQuery that will receive this data. I simply tried to create a blank table and insert the received schema and could not. I get the following information: "Error while reading data, error message: The Apache Avro library failed to parse the header with the following error: Invalid data file. Magic does not match: gs://xxx/xxx/test.avsc". I tried to convert avsc to json through this online page: https://json-schema-validator.herokuapp.com/avro.jsp however unsuccessfully.

Is this the process path? As I can generate the records after decoding I understand that the schema is not incorrect. So I doubt if I'm doing the right thing.

IDMT · Accepted Answer · 2019-11-27 15:24:02Z

You can't create table schemas using a avsc file. You can load your data from CSV or AVRO (not compressed) and enable schema auto detection. Also, take a look on this if your files are on a local data source.

Since you are using python and CSV files, you can try something like this:

from google.cloud import bigquery
client = bigquery.Client()
filename = '/path/to/file.csv'
dataset_id = 'my_dataset'
table_id = 'my_table'

dataset_ref = client.dataset(dataset_id)
table_ref = dataset_ref.table(table_id)
job_config = bigquery.LoadJobConfig()
job_config.source_format = bigquery.SourceFormat.CSV
job_config.skip_leading_rows = 1
job_config.autodetect = True

with open(filename, "rb") as source_file:
    job = client.load_table_from_file(source_file, table_ref, job_config=job_config)

job.result()  # Waits for table load to complete.

print("Loaded {} rows into {}:{}.".format(job.output_rows, dataset_id, table_id))

Brachi · Accepted Answer · 2022-05-30 11:57:16Z

0

Make sure your Avro file contains the schema at the start, otherwise BQ can't decode it.

answered May 30, 2022 at 11:57

Brachi

7471 gold badge10 silver badges19 bronze badges

Collectives™ on Stack Overflow

BigQuery - Apache Avro library failed to parse the header with the following error: Invalid data file. Magic does not match

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related