add datadog io normalization and yaml

[derrickaw]

Please add a meaningful description for your change here

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[derrickaw]

/gemini review

[gemini-code-assist]

Code Review

This pull request adds a DatadogWriteSchemaTransformProvider to Apache Beam, allowing for Datadog integration within schema-aware pipelines and YAML. The implementation includes a row-based writer, configuration validation, and integration tests. Reviewer feedback identifies that the writer currently lacks batching, which is inefficient for API usage, and that the parallelism setting may be bypassed by transform fusion. Further recommendations include strengthening configuration validation for mandatory and numeric fields, ensuring output consistency with the provider's metadata, and improving test assertions for invalid configurations.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 57.03%. Comparing base (c5d0ab1) to head (54c2e27).

Additional details and impacted files

@@              Coverage Diff              @@
##             master   #38362       +/-   ##
=============================================
- Coverage     60.50%   57.03%    -3.48%     
+ Complexity    20442     3634    -16808     
=============================================
  Files          3307     1185     -2122     
  Lines        315339   189633   -125706     
  Branches      16942     3751    -13191     
=============================================
- Hits         190799   108148    -82651     
+ Misses       116084    78034    -38050     
+ Partials       8456     3451     -5005     

Flag	Coverage Δ
java	72.61% <ø> (+5.51%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[derrickaw]

/gemini review

[gemini-code-assist]

Code Review

This pull request implements a Datadog write SchemaTransform, enabling data ingestion into Datadog through Beam's SchemaTransform API and YAML. The changes include the Java provider and configuration classes, comprehensive unit tests, and Python-based integration test infrastructure utilizing Docker and mock servers. Key feedback includes addressing silent write failures when error handling is unconfigured, correcting the output port identifier in the PCollectionRowTuple, and refining the handling of nullable configuration parameters. Further improvements were suggested regarding the serialization of error payloads and the removal of unrelated coder registrations.

[derrickaw]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces the DatadogWriteSchemaTransformProvider, enabling Datadog write operations through Beam SchemaTransforms and YAML pipelines. The changes include the core Java implementation, comprehensive unit tests, and integration test utilities in Python using a mock Datadog server and container. Feedback focuses on ensuring consistency between the advertised output collection names and the actual tags used in the PCollectionRowTuple, as well as minor formatting improvements in the Python integration code.