Guide

Async Validation & Parallel Processing

Validate data with external services, database lookups, and long-running computations using ExJoi.async/3. ExJoi leverages Task.async_stream to run validations in parallel, dramatically improving performance for multiple async fields.

API Reference

Async Validation Functions

Complete reference for async validation in ExJoi.

ExJoi.async/3

Wraps an existing rule with an asynchronous validation function. The async function can return {:ok, value}, {:error, errors}, or a Task that will be awaited.

Parameters

  • rule (%ExJoi.Rule{}, required) - The base rule to wrap (e.g., ExJoi.string(required: true))
  • async_fn (function, required) - Function that performs async validation. Signature: fn(value, context) -> {:ok, value} | {:error, errors} | Task.t()
  • opts (keyword list, optional) - Options for async validation

Options

  • :timeout (integer, default: 5000) - Maximum time in milliseconds to wait for async validation. Can be overridden by the global :timeout option in ExJoi.validate/3.

Async Function Signature

The async function receives:

  • value (any) - The validated value (after synchronous validation passes)
  • context (map) - Context map containing :convert, :data, :custom_opts

Must return one of:

  • {:ok, validated_value} - Success tuple with validated/normalized value
  • {:error, [error_map, ...]} - Error tuple with list of error maps
  • %Task{} - A Task that will be awaited. The Task should return {:ok, value} or {:error, errors}

Returns

%ExJoi.Rule{async: async_fn, timeout: timeout, ...} - A rule with async validation enabled

Validation Flow

When validating an async rule:

  1. Synchronous validation runs first (type checking, constraints like min, max, email)
  2. If synchronous validation passes, the async function is executed
  3. If the async function returns a Task, it is awaited with the configured timeout
  4. Errors from async validation are merged with synchronous errors
Examples 
# Simple async validator returning {:ok, value}
schema = ExJoi.schema(%{
  username: ExJoi.async(
    ExJoi.string(required: true, min: 3),
    fn value, _ctx ->
      if UsernameService.available?(value) do
        {:ok, String.downcase(value)}
      else
        {:error, [%{code: :username_taken, message: "username is already taken"}]}
      end
    end
  )
})

# Async validator returning a Task
schema = ExJoi.schema(%{
  email: ExJoi.async(
    ExJoi.string(required: true, email: true),
    fn value, _ctx ->
      Task.async(fn ->
        # Simulate external API call
        Process.sleep(100)
        if EmailService.is_valid?(value) do
          {:ok, value}
        else
          {:error, [%{code: :email_invalid, message: "email is not valid"}]}
        end
      end)
    end,
    timeout: 2000
  )
})

# With custom timeout
schema = ExJoi.schema(%{
  api_key: ExJoi.async(
    ExJoi.string(required: true),
    fn value, _ctx -> ApiKeyService.verify(value) end,
    timeout: 3000
  )
})

ExJoi.validate/3 with Async Options

The validate/3 function accepts additional options for controlling async validation behavior.

Options

  • :timeout (integer, optional) - Global timeout in milliseconds for all async validations. Overrides per-rule timeouts if provided.
  • :max_concurrency (integer, default: System.schedulers_online()) - Maximum number of async validations to run in parallel. Controls the concurrency of Task.async_stream.

Parallel Processing

When multiple fields have async validation:

  • All async validations run in parallel using Task.async_stream
  • Array elements with async validation are also validated in parallel
  • Concurrency is controlled by :max_concurrency option
  • Timeouts are enforced per validation, not globally
Examples 
# With global timeout
ExJoi.validate(data, schema, timeout: 10000)

# With custom concurrency
ExJoi.validate(data, schema, max_concurrency: 5)

# All options together
ExJoi.validate(data, schema, 
  convert: true,
  timeout: 8000,
  max_concurrency: 10
)

# Parallel validation of multiple async fields
schema = ExJoi.schema(%{
  username: ExJoi.async(ExJoi.string(), fn v, _ -> check_username(v) end),
  email: ExJoi.async(ExJoi.string(), fn v, _ -> check_email(v) end),
  api_key: ExJoi.async(ExJoi.string(), fn v, _ -> verify_key(v) end)
})

# All three validations run in parallel
ExJoi.validate(data, schema)

Real-World Examples

Common Async Validation Patterns

Practical examples of async validation in production applications.

Username Availability Check

Check if a username is available by querying a database or external service.

Example 
defmodule UserService do
  def username_available?(username) do
    # Database query or external API call
    Repo.get_by(User, username: username) == nil
  end
end

schema = ExJoi.schema(%{
  username: ExJoi.async(
    ExJoi.string(required: true, min: 3, max: 20),
    fn value, _ctx ->
      Task.async(fn ->
        if UserService.username_available?(value) do
          {:ok, String.downcase(value)}
        else
          {:error, [%{code: :username_taken, message: "username is already taken"}]}
        end
      end)
    end,
    timeout: 3000
  )
})

# Usage
case ExJoi.validate(%{username: "john_doe"}, schema) do
  {:ok, data} -> 
    # Username is available and validated
    create_user(data)
  {:error, errors} -> 
    # Handle errors (e.g., username taken, too short, etc.)
    render_error(errors)
end

Email Verification Service

Verify email addresses using an external service like SendGrid or Mailgun.

Example 
defmodule EmailVerificationService do
  def verify_email(email) do
    # External API call
    case HTTPoison.get("https://api.email-verify.com/check?email=#{email}") do
      {:ok, %{status_code: 200, body: body}} ->
        case Jason.decode(body) do
          {:ok, %{"valid" => true}} -> {:ok, email}
          _ -> {:error, [%{code: :email_invalid, message: "email is not valid"}]}
        end
      _ -> {:error, [%{code: :email_verification_failed, message: "could not verify email"}]}
    end
  end
end

schema = ExJoi.schema(%{
  email: ExJoi.async(
    ExJoi.string(required: true, email: true),
    fn value, _ctx ->
      Task.async(fn -> EmailVerificationService.verify_email(value) end)
    end,
    timeout: 5000
  )
})

API Key Validation

Validate API keys against an external service or database.

Example 
defmodule ApiKeyService do
  def verify_key(api_key) do
    case Repo.get_by(ApiKey, key: api_key, active: true) do
      nil -> {:error, [%{code: :api_key_invalid, message: "invalid API key"}]}
      key -> 
        if key.expires_at && DateTime.compare(key.expires_at, DateTime.utc_now()) == :lt do
          {:error, [%{code: :api_key_expired, message: "API key has expired"}]}
        else
          {:ok, api_key}
        end
    end
  end
end

schema = ExJoi.schema(%{
  api_key: ExJoi.async(
    ExJoi.string(required: true),
    fn value, _ctx ->
      Task.async(fn -> ApiKeyService.verify_key(value) end)
    end,
    timeout: 2000
  )
})

Parallel Array Validation

Validate each element of an array asynchronously in parallel.

Example 
schema = ExJoi.schema(%{
  user_ids: ExJoi.array(
    of: ExJoi.async(
      ExJoi.string(required: true),
      fn user_id, _ctx ->
        Task.async(fn ->
          case Repo.get(User, user_id) do
            nil -> {:error, [%{code: :user_not_found, message: "user #{user_id} not found"}]}
            _user -> {:ok, user_id}
          end
        end)
      end
    ),
    required: true,
    min_items: 1
  )
})

# All user_id validations run in parallel
ExJoi.validate(%{user_ids: ["1", "2", "3"]}, schema, max_concurrency: 5)

Error Handling

Async Validation Errors

Understanding error codes and handling async validation failures.

Error Codes

Async validation introduces new error codes:

Code Description When It Occurs
:async_timeout Async validation timed out When the async function takes longer than the configured timeout
:async_error Async validation failed with an error When the async function raises an exception or returns an unexpected value
:async_validation Async validation returned errors When the async function returns {:error, errors}

Error Structure

Async validation errors follow the same structure as synchronous errors:

Example 
# Timeout error
{:error, %{
  username: [%{code: :async_timeout, message: "async validation timed out"}]
}}

# Custom async error
{:error, %{
  email: [%{code: :email_invalid, message: "email is not valid"}]
}}

# Multiple errors (sync + async)
{:error, %{
  username: [
    %{code: :string_min, message: "must be at least 3 characters"},
    %{code: :username_taken, message: "username is already taken"}
  ]
}}

# Flattened errors
{:error, %{
  errors_flat: %{
    "username" => ["must be at least 3 characters", "username is already taken"]
  }
}}

Best Practices

Tips for Async Validation

Guidelines for writing efficient and reliable async validations.

1. Set Appropriate Timeouts

Always set timeouts that match your service's expected response time. Too short and you'll get false timeouts; too long and users wait unnecessarily.

Example 
# Fast database lookup: 1-2 seconds
username: ExJoi.async(rule, fn, timeout: 2000)

# External API call: 3-5 seconds
email: ExJoi.async(rule, fn, timeout: 5000)

# Slow external service: 10 seconds
api_key: ExJoi.async(rule, fn, timeout: 10000)

2. Use Parallel Processing

When validating multiple async fields, ExJoi automatically runs them in parallel. Adjust max_concurrency based on your service limits.

Example 
# Multiple async fields validated in parallel
schema = ExJoi.schema(%{
  username: ExJoi.async(rule1, fn1),
  email: ExJoi.async(rule2, fn2),
  api_key: ExJoi.async(rule3, fn3)
})

# Control concurrency to avoid overwhelming external services
ExJoi.validate(data, schema, max_concurrency: 3)

3. Handle Errors Gracefully

Always return proper error tuples from async functions. Catch exceptions and convert them to error tuples.

Example 
# Good: Returns error tuple
fn value, _ctx ->
  Task.async(fn ->
    try do
      case ExternalService.check(value) do
        {:ok, result} -> {:ok, result}
        {:error, reason} -> {:error, [%{code: :validation_failed, message: reason}]}
      end
    rescue
      e -> {:error, [%{code: :service_error, message: "service unavailable"}]}
    end
  end)
end

# Bad: Raises exception
fn value, _ctx ->
  Task.async(fn ->
    ExternalService.check!(value)  # May raise
  end)
end

4. Combine Sync and Async Validation

Use synchronous validation for fast checks (format, length) and async validation for expensive operations (database, external APIs).

Example 
# Sync validation runs first (fast)
# Async validation runs only if sync passes (expensive)
username: ExJoi.async(
  ExJoi.string(required: true, min: 3, max: 20, pattern: ~r/^[a-z0-9_]+$/),
  fn value, _ctx ->
    # Only runs if format is valid
    Task.async(fn -> check_availability(value) end)
  end
)

5. Cache Results When Possible

For expensive validations, consider caching results to avoid redundant external calls.

Example 
defmodule CachedValidator do
  def check_with_cache(value) do
    case Cache.get("validation:#{value}") do
      nil ->
        result = ExternalService.check(value)
        Cache.put("validation:#{value}", result, ttl: 3600)
        result
      cached -> cached
    end
  end
end

schema = ExJoi.schema(%{
  api_key: ExJoi.async(
    ExJoi.string(),
    fn value, _ctx ->
      Task.async(fn -> CachedValidator.check_with_cache(value) end)
    end
  )
})

Troubleshooting

Common Issues & Solutions

Solutions to common problems when working with async validation.

Issue: Timeouts Too Frequent

Solution: Increase the timeout value or optimize your async function. Check if your external service is slow or if there's network latency.

Fix 
# Increase timeout
ExJoi.async(rule, fn, timeout: 10000)  # 10 seconds instead of 5

# Or set global timeout
ExJoi.validate(data, schema, timeout: 15000)

Issue: Task Not Being Awaited

Solution: Ensure your async function returns a Task or directly returns {:ok, value} / {:error, errors}. ExJoi automatically awaits Tasks.

Fix 
# Good: Returns Task
fn value, _ctx ->
  Task.async(fn -> check(value) end)
end

# Also good: Returns directly
fn value, _ctx ->
  check(value)  # Returns {:ok, value} or {:error, errors}
end

# Bad: Doesn't return Task or tuple
fn value, _ctx ->
  Task.start(fn -> check(value) end)  # Returns {:ok, pid}, not awaited
end

Issue: High Memory Usage with Many Async Validations

Solution: Reduce max_concurrency to limit the number of parallel validations. This prevents overwhelming your system or external services.

Fix 
# Limit concurrency
ExJoi.validate(data, schema, max_concurrency: 5)

# For large arrays, validate in batches
# (You may need to implement batching logic in your async function)

Issue: Async Errors Not Appearing

Solution: Ensure your async function returns proper error tuples. Check that synchronous validation isn't failing first (async only runs if sync passes).

Fix 
# Ensure async function returns error tuple
fn value, _ctx ->
  Task.async(fn ->
    case check(value) do
      :ok -> {:ok, value}
      :error -> {:error, [%{code: :validation_failed, message: "validation failed"}]}
    end
  end)
end

# Check synchronous validation first
# Async only runs if sync validation passes
username: ExJoi.async(
  ExJoi.string(required: true, min: 3),  # Sync validation
  fn value, _ctx -> async_check(value) end  # Only runs if sync passes
)