Preparing data for a model

A model's answer is only as good as what you send it. Before data goes into a prompt or a knowledge base, check it: required fields present, text in the expected language, no empty or duplicate documents, dates in one format. AWS Glue Data Quality runs rules like these over data in a catalog and can stop a pipeline when the data fails them. A Lambda function can do lighter checks on each record, and CloudWatch metrics can count how many fail.

Not everything starts as text. Amazon Transcribe turns speech into text (with speaker labels and custom vocabularies for product names). Amazon Textract pulls text, tables and form fields out of scanned documents and PDFs. Amazon Rekognition finds objects, text and faces in images. Bedrock Data Automation handles documents, images, audio and video in one managed service, returning structured output that you shape with blueprints. Multimodal models in Bedrock can also read images and documents directly.

Enrich and normalize. Amazon Comprehend finds entities (people, places, organizations, dates), key phrases, the language and sentiment, and detects personal information. A Lambda function can fix casing, strip HTML, expand abbreviations and split long records. A model itself can rewrite messy text into a clean format before a second step uses it.

Format the request the way the target expects. For Bedrock, that's the Converse messages shape, or the provider's JSON body for InvokeModel. For a SageMaker AI endpoint, it's whatever the serving container expects, often JSON with an inputs field. A dialog app sends the conversation as alternating user and assistant turns. Wrong shapes come back as validation errors, not as bad answers.

For big jobs, process in batches. SageMaker Processing runs your own scripts on managed instances for one-off or scheduled preparation. AWS Step Functions can fan a large set of files out to Lambda functions in parallel (a distributed map) and collect the results. Amazon SageMaker Data Wrangler gives a visual way to explore and clean tabular data.

Keep track of where data came from. Tag documents with their source, owner and date (as S3 object metadata or in a catalog), so a retrieved passage can be cited, filtered by date, and traced if it turns out to be wrong.