Build Ready-to-Use Dataflows with Unique Metadata Filenames

Metadata brings tracking and managing data closer to your DataOps.

Whether it’s the API, bucket, path, ingest time, or any of the other stats captured by Nexla; using the setup shown in this tutorial you’ll be able to pull out that attribute and add it to the dataset and to the output filename.

Capturing Source Metadata

1. Add your Source

Follow the first steps of connecting a data source. Once Nexla has detected and created a Nexset, click on the transform button.

2. Capture the Source Metadata in the Nexset

In the transform screen, click Create New Attribute in the middle and set it equal to Metadata. Then, select from the captured metadata using the dropdown to the right. In this example, Ingest Time, Source Path, the S3 Source Bucket, and Source Offset are available. Any metadata available from the source can be captured in the Nexset. Give your attribute a name and click Add to Draft.

Don’t forget to Activate This Dataset on the next screen and give it a name if required.

Writing Metadata Out to Filenames

1. Sending to Destination

Click Send to Destination on the derived nexset and choose your output destination. Add your credentials or select an existing one, and click Next to configure the filenames.

2. Writing Metadata to the Destination Filenames

Select the output file type you’d like under Data Format. Select Set Custom File Name Prefix, then type a left curly bracket ‘{‘. Scroll down or start typing the metadata attribute name in the Nexset and select it to add that variable to all your output filenames. Any metadata can be configured to the output filename, but you must first add it to the Nexset as described above. You can add as many constant or variable prefixes here as you’d prefer. Click Save and then Activate This Destination.

Conclusion

That’s it! Each of your output files will immediately start flowing to your destination and the filename will contain the metadata prefix configured in the destination. Nexla will continue to monitor this flow and notify you of any changes or errors.

Interested in trying it out for yourself or learning more? Contact us to start your free trial today.