Skip to content

feat(zendesk): add Zendesk datasource package#288

Open
PMerlet wants to merge 6 commits intomainfrom
feat/datasource-zendesk
Open

feat(zendesk): add Zendesk datasource package#288
PMerlet wants to merge 6 commits intomainfrom
feat/datasource-zendesk

Conversation

@PMerlet
Copy link
Copy Markdown
Member

@PMerlet PMerlet commented Apr 27, 2026

Summary

New forest_admin_datasource_zendesk package that surfaces Zendesk Tickets, Users, Organizations, and ticket Comments as Forest Admin collections — same architectural pattern as the existing forest_admin_datasource_active_record / _mongoid / _rpc siblings, mirroring the Node @forestadmin/datasource-* ecosystem.

  • 4 collections with schema-level intra-datasource relations: Ticket → requester/assignee/organization/comments, Comment → author/ticket, User → organization/requested_tickets, Organization → users/tickets.
  • Custom fields: discovers admin-defined ticket_fields, user_fields, organization_fields at boot and exposes them as real Forest columns. Translator rewrites custom-column names to Zendesk Search syntax.
  • Timezone-aware Date filters: Forest's caller timezone is threaded through the translator; bare Date values are interpreted as start-of-day in the caller's TZ then converted to UTC (DST-aware).
  • Loud-vs-graceful error handling: critical paths (search, count, fetch_ticket_comments) raise APIError; enrichment paths (bulk user/org lookups, schema introspection) log a warning and degrade safely.
  • Synthetic single-PK on Comments (<comment_id>-<ticket_id> String) to side-step forest_admin_rails 1.26.2's URL constraint, which rejects the | character used by the toolkit's native composite-key pack_id. Filed-this-as-a-bug-worth-fixing-upstream note in the relevant code comment.

What's NOT in this PR (deliberate v1 scope)

  • Read-only — no create/update/delete on any collection. The Compliance handoff workflow (update ticket status / reassign group) is the obvious next add.
  • No caching — every list page hits Zendesk's Search API + bulk user/org lookups. Add Rails.cache wrapping in a follow-up if rate limits become a concern.
  • Zendesk Search's 1000-result hard cap is not surfaced in pagination yet (cursor-based Incremental Export API would be the correct path for large datasets).
  • No native OR-aggregator translation (raises loudly on unsupported operators).

Test plan

  • cd packages/forest_admin_datasource_zendesk && bundle install && bundle exec rspec — 131 examples, 0 failures
  • Coverage ≥ 90% line and branch (98.5% line / 90.7% branch with WebMock blocking real net)
  • Manually verified end-to-end against a live Zendesk dev instance (browse tickets, requester join, conversation thread via Ticket → comments related list, comment detail page via synthetic PK)
  • Reviewer to confirm: gemspec metadata, dependency pins, SchemaEmitter compatibility with toolkit ≥ 1.26
  • Reviewer to confirm: package layout matches conventions (Zeitwerk, Rakefile, .rspec)

🤖 Generated with Claude Code

Note

Add Zendesk datasource package with collections for tickets, users, organizations, and comments

  • Introduces a new forest_admin_datasource_zendesk gem that connects Forest Admin to Zendesk via the zendesk_api gem, exposing Ticket, User, Organization, and Comment as queryable collections.
  • Implements a ConditionTreeTranslator that converts Forest Admin filter trees to Zendesk Search query strings, with timezone-aware date handling, custom field remapping, and AND-only aggregation.
  • Adds a Client wrapper with distinct error policies: critical paths raise APIError on failure; enrichment paths (bulk email/user/org lookups, schema introspection) degrade gracefully to {} or [] with a warning.
  • Custom fields are discovered at boot via CustomFieldsIntrospector, which maps Zendesk field types to Forest column schemas and injects them into collection schemas and query translation.
  • Wires the package into CI (lint, coverage), semantic-release automation, and RuboCop configuration.

Macroscope summarized 116f909.

Surfaces Zendesk Tickets, Users, Organizations and ticket Comments as
Forest Admin collections, mirroring how the @forestadmin/datasource-*
packages work in the Node ecosystem.

Highlights:

* `forest_admin_datasource_zendesk` package under packages/, follows the
  same structure as forest_admin_datasource_active_record (Zeitwerk
  loader, gemspec, BaseCollection sharing sort/page/projection/PK-lookup).
* Four collections: ZendeskTicket, ZendeskUser, ZendeskOrganization,
  ZendeskComment, with intra-datasource ManyToOne / OneToMany relations
  declared in the schema (Ticket -> requester/assignee/organization/
  comments, Comment -> author/ticket, User -> organization/requested_tickets,
  Organization -> users/tickets).
* Custom-field introspection at boot: queries /ticket_fields,
  /user_fields, /organization_fields and exposes admin-defined fields as
  real Forest columns. Filter translator rewrites custom column names to
  Zendesk Search syntax (`custom_field_<id>` for tickets; key for keyed
  user/org fields).
* ConditionTreeTranslator produces Zendesk Search queries with
  caller-timezone-aware Date interpretation (Date is start-of-day in the
  caller's TZ, converted to UTC; DST-aware via ActiveSupport).
* Loud errors on critical paths (search, count, fetch_ticket_comments)
  via wrapped APIError; best-effort logging + safe defaults on enrichment
  paths (bulk user/org lookups, schema introspection).
* ZendeskComment uses a synthetic single-PK String (`<id>-<ticket_id>`)
  to side-step forest_admin_rails 1.26.2's URL constraint, which rejects
  the `|` character used by the toolkit's native pack_id for composite
  keys.
* Standalone ZendeskComment list returns [] when no ticket scope is
  supplied (Zendesk has no /comments listing endpoint).

Tests: 131 examples, 98.5% line / 90.7% branch coverage. WebMock blocks
real network. Run with `bundle exec rspec` from the package directory.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@qltysh
Copy link
Copy Markdown

qltysh Bot commented Apr 27, 2026

1 new issue

Tool Category Rule Count
qlty Structure Function with many parameters (count = 4): aggregate 1

PMerlet and others added 2 commits April 27, 2026 19:08
- Apply rubocop -A across the package (whitespace, indentation, hash
  alignment, frozen string literals).
- Add packages/forest_admin_datasource_zendesk/.rubocop.yml inheriting
  the repo root config and:
    - relaxing Metrics/* limits for the wide-fact-table schema files
    - exempting spec/** from RSpec stylistic cops we use heavily
      (StubbedMock, MessageSpies, LeakyConstantDeclaration,
      ConstantDefinitionInBlock) and from Layout/LineLength
- Collapse must_succeed's two identical rescue branches in client.rb.
- Move Branch constant out of the `private` scope in comment.rb.

Specs still pass: 131 examples, 0 failures, 98.7% line / 90.9% branch.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Resolves the 12 review comments left by qltysh on the previous push.

Duplication
- New `Collections::Searchable` mixin holds the list/find/aggregate
  flow that User and Organization both used. Each collection now only
  declares its zendesk_resource, sortable_fields, find_one and
  serialize, eliminating the ~16 lines of similar code qlty flagged.

Complexity
- BaseCollection now owns the (caller, filter, aggregation, _limit)
  contract method. Subclasses override `aggregate_count(caller, filter)`
  with a 2-arg helper. Ticket/User/Organization no longer carry their
  own copies of the same validation+raise+count boilerplate.
- Ticket#embed_relations split into `embed_users` and
  `embed_organizations` (cyclomatic complexity 11 -> ~3 each).
- Comment#list extracts `resolve_scope(filter)` and `fetch_comments`
  (complexity 9 -> 3). #extract_field_lookup pulled out
  `values_from_leaf` (complexity 5 -> 2).
- ConditionTreeTranslator#format_value split into format_date and
  format_string (complexity 7 -> 3). Date / TZ degradation logic now
  lives inside format_date with a tighter rescue scope.
- CustomFieldsIntrospector#introspect split into `usable_field?` and
  `build_entry` predicates (complexity 10 -> 3).
- BaseCollection#translate_sort extracts `sort_field_and_direction`
  for the Sort::Clause vs hash branching.

Parameter counts
- Client#search collapsed five named kwargs into a single `**opts` hash
  + `build_search_params` helper. Callers (the Searchable mixin and
  Ticket#fetch_records) keep the same kwarg call sites.
- aggregate's 4-param signature only remains on BaseCollection where
  it's mandated by ForestAdminDatasourceToolkit::Components::Contracts::
  CollectionContract; subclasses no longer carry it.

131 specs still pass, 98.4% line / 91.4% branch coverage. RuboCop
clean across the package.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

# Toolkit contract — subclasses override `aggregate_count` instead of
# touching the 4-arg signature directly.
def aggregate(caller, filter, aggregation, _limit = nil)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Function with many parameters (count = 4): aggregate [qlty:function-parameters]

Bugs caught by macroscopeapp on the previous push:

1. (high) Searchable#search_records dropped filter.search from the
   query, while compose_count_query included it. Result: list and
   count returned different totals when the user typed in the search
   box. Both paths now share `compose_full_query`, which mirrors the
   logic the Ticket collection has used all along.

2. (high) sort_field_and_direction used `entry[:ascending] ||
   entry['ascending']`, which silently flips a descending sort to
   ascending when both keys exist with different values, and returns
   nil instead of false when only the symbol key is set. Switched to
   `key?` so explicit `ascending: false` round-trips correctly.

3. (low) format_value(nil) emitted `field:` clauses, which Zendesk's
   search treats as a presence check — semantically wrong for an
   EQUAL/NOT_EQUAL filter. Now raises UnsupportedOperatorError with a
   message pointing the caller at PRESENT/BLANK.

Plus qlty: Comment#resolve_scope dropped to complexity ~3 by extracting
`decoded_synthetic_pairs`.

Regression tests added for all three bugs. 135 specs, 0 failures,
98.6% line / 90.0% branch. RuboCop clean.

Note: BaseCollection#aggregate's 4-param signature still trips qlty's
`function-parameters` cop. That signature is structurally required by
ForestAdminDatasourceToolkit::Components::Contracts::CollectionContract;
subclasses already override the lower-arity `aggregate_count(caller,
filter)` so the 4-arg form lives in exactly one place. No fix is
possible without breaking the toolkit contract.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Multi-tenancy: ConditionTreeTranslator no longer keeps the custom-field
mapping on class-level state. The mapping now lives on the Datasource
instance (`Datasource#custom_field_mapping`), and is threaded through
each translator call as `custom_fields:`. Two datasources with
different mappings used to step on each other; they're now isolated.

Translator robustness:
- `format_string` now backslash-escapes internal double quotes when it
  needs to wrap in quotes. `subject = 'test "with" quotes'` previously
  emitted a malformed query.
- `IN []` and `NOT_IN []` previously produced an empty string that the
  branch translator silently dropped, turning "match nothing" into
  "match everything". They now raise UnsupportedOperatorError.

Comment#decoded_synthetic_pairs now drops pairs where either half is
nil. Previously, `id = "abc-456"` (invalid comment_id, valid ticket_id)
contributed `456` to the ticket scope and silently fetched every
comment for that ticket — wrong, since the row the user clicked on
doesn't exist.

Regression tests added for all four bugs. 144 specs, 0 failures,
98.6% line / 90.4% branch.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Comment on lines +59 to +64
def translate_leaf(leaf)
field = mapped_field(leaf.field)
value = leaf.value

return "requester:#{format_value(value)}" if leaf.field == 'requester_email' && leaf.operator == Operators::EQUAL

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Low query/condition_tree_translator.rb:59

The special requester: mapping on line 63 only applies to the EQUAL operator. For NOT_EQUAL, IN, or NOT_IN on requester_email, the code falls through to the generic path and emits the unmappable field name requester_email instead of requester, producing malformed queries like -requester_email:value or requester_email:val1 requester_email:val2.

       def translate_leaf(leaf)
         field = mapped_field(leaf.field)
         value = leaf.value
 
-        return "requester:#{format_value(value)}" if leaf.field == 'requester_email' && leaf.operator == Operators::EQUAL
+        field = 'requester' if leaf.field == 'requester_email'
 
         case leaf.operator
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file packages/forest_admin_datasource_zendesk/lib/forest_admin_datasource_zendesk/query/condition_tree_translator.rb around lines 59-64:

The special `requester:` mapping on line 63 only applies to the `EQUAL` operator. For `NOT_EQUAL`, `IN`, or `NOT_IN` on `requester_email`, the code falls through to the generic path and emits the unmappable field name `requester_email` instead of `requester`, producing malformed queries like `-requester_email:value` or `requester_email:val1 requester_email:val2`.

Evidence trail:
packages/forest_admin_datasource_zendesk/lib/forest_admin_datasource_zendesk/query/condition_tree_translator.rb at REVIEWED_COMMIT:
- Line 58: `field = mapped_field(leaf.field)`
- Line 63: `return "requester:#{format_value(value)}" if leaf.field == 'requester_email' && leaf.operator == Operators::EQUAL` - special case only for EQUAL
- Lines 65-68: generic case statement uses `field` variable for NOT_EQUAL, IN, NOT_IN
- Lines 87-89: `mapped_field` returns `@custom_fields[field] || field` - returns field name unchanged if not in custom_fields

build.yml:
  - Add the package to the lint matrix (RuboCop runs against every
    package's tree).
  - Add to the test matrix; CI runs rspec under BUNDLE_GEMFILE=Gemfile-
    test like the other datasource packages.
  - Add the coverage.json path so qlty consumes our coverage too.

Gemfile-test: pulls forest_admin_datasource_toolkit from the local
sibling package (matches the active_record / mongoid pattern). Adds
simplecov-html, simplecov_json_formatter and webmock so the CI run
emits the JSON coverage file qlty expects.

spec_helper.rb: load simplecov-html / simplecov_json_formatter inside
a guarded require so local `bundle exec rspec` (no formatters) still
works while CI's Gemfile-test path emits coverage.json.

.releaserc.js:
  - prepareCmd bumps the package's version.rb alongside the others.
  - successCmd builds and pushes the gem to RubyGems.
  - @semantic-release/git tracks the version.rb in the release commit.

Tested locally under both Gemfile and Gemfile-test: 144 specs pass,
RuboCop clean, JSON coverage emitted.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant