Ask any AI agent to implement Auth0.
Choose the LLM, choose the framework, and see how those work with your Auth0 implementation.
Compare stacksWithout Auth0 tools
SDK setup & imports
Tokens stored securely
73%
avg. success rate
With Auth0
SkillsMCP
SDK setup & imports
Tokens stored securely
98%
avg. success rate25%
Tested across 8 models · 15 frameworks
Agent Experience Score for Auth0
Validated, build-verified results for every model, framework, and configuration tested with Auth0.
Select a framework to filter results for your stack.
#
Model
Framework
Without tools
With Auth0 tools *
Grade
Est. Cost
Est. Time *
1
Claude Haiku 4.5
Express API
70%7/10
100%+30%
19/19$0.10
52s
With Auth0 Tools·100%·Tool Calls 10
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (19/19 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(3/3)
Uses express-oauth2-jwt-bearer SDK
Uses requiredScopes() for scope-based route protection
Accesses JWT data via req.auth
L2 · Hallucination Detection(4/4)
No express-openid-connect (that is for web apps, not APIs)
No passport middleware (not needed with express-oauth2-jwt-bearer)
No manual JWT verification with jsonwebtoken (use SDK)
No SPA SDK used in server-side API
L3 · Security(2/2)
No hardcoded issuer domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
L4 · Structural(4/4)
Wrote Auth0 config to .env file
GET /api/messages protected with read:messages scope
POST /api/messages protected with write:messages scope
Does the API correctly protect all three routes? GET /api/messages requires read:messages, POST /api/messages requires write:messages, GET /api/profile returns user info from req.auth.payload. Missing or invalid tokens should yield 401, insufficient scope should yield 403.
L5 · Version Correctness(4/4)
No manual Authorization header parsing (SDK handles extraction)
No manual jwt.verify calls (SDK handles verification)
No req.user (express-oauth2-jwt-bearer uses req.auth, not req.user)
Does the solution use current express-oauth2-jwt-bearer patterns? Specifically: does it apply the auth() middleware to protect routes, use requiredScopes() for scope checks (not manual payload inspection), and access token data via req.auth.payload (not req.user or manually decoded tokens)? Judge only from the source code; the issuer and audience may be supplied via environment variables (ISSUER_BASE_URL / AUDIENCE), so do not assume the contents of any .env file.
Holistic(1/1)
Does the solution correctly protect an Express.js API using express-oauth2-jwt-bearer? It should apply the auth() middleware (issuer and audience may come from ISSUER_BASE_URL / AUDIENCE environment variables — judge only from the source code and do not assume the contents of any .env file), protect GET /api/messages with read:messages scope, protect POST /api/messages with write:messages scope, and return user profile info from req.auth.payload at GET /api/profile.
2
GPT-5.6 Luna
React
92%12/13
100%+8%
20/20$0.13
43s
With Auth0 Tools·100%·Tool Calls 9
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (20/20 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(7/7)
Uses @auth0/auth0-react SDK
Wraps app with Auth0Provider
Uses useAuth0 hook
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile name
L2 · Hallucination Detection(3/3)
No hallucinated @auth0/react package (must be @auth0/auth0-react)
Does not use server SDK in SPA app
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Auth0Provider configured with domain prop
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated? A correct implementation should not render auth-dependent UI while isLoading is true.
L5 · Version Correctness(3/3)
Uses authorizationParams (not audience directly on provider)
authorizationParams contains audience 'https://api.barkbook.com'
Does the code use the current @auth0/auth0-react SDK patterns? Specifically: does it use isLoading (not the deprecated "loading" property), and pass audience/scope via authorizationParams object (not as direct props)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a React SPA with Auth0Provider, useAuth0 hook, login, logout, user profile display, and getAccessTokenSilently to make authenticated API calls?
3
Claude Haiku 4.5
FastAPI
50%7/14
100%+50%
21/21$0.13
52s
With Auth0 Tools·100%·Tool Calls 13
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (21/21 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses auth0-fastapi-api package
Imports Auth0FastAPI class
Uses require_auth() dependency
Uses FastAPI Depends for dependency injection
Configures domain
Configures audience
L2 · Hallucination Detection(5/5)
No python-jose (manual JWT parsing not needed)
No PyJWT direct usage
No manual JWT decoding
No fastapi-users (wrong package for Auth0 JWT)
No passlib (unrelated auth package)
L3 · Security(2/2)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
L4 · Structural(5/5)
Wrote Auth0 config to .env file
Auth0FastAPI instance is created
Route protected with Depends(auth0.require_auth())
read:messages scope checked on /api/messages route
Does the app correctly create an Auth0FastAPI instance, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token? Note: the auth0-fastapi-api distribution imports as the module `fastapi_plugin` (i.e. `from fastapi_plugin import Auth0FastAPI` is correct — there is no `auth0_fastapi_api` module).
L5 · Version Correctness(1/1)
Does the solution use current auth0-fastapi-api patterns? Specifically: does it use Auth0FastAPI with domain and audience parameters, use require_auth() as a FastAPI Depends dependency (not a decorator), and read credentials from environment variables (not hardcoded)? Note: the auth0-fastapi-api distribution imports as the module `fastapi_plugin` (`from fastapi_plugin import Auth0FastAPI` is correct — there is no `auth0_fastapi_api` module).
Holistic(1/1)
Does the solution correctly integrate Auth0 into a FastAPI API using auth0-fastapi-api? It should create an Auth0FastAPI instance configured with domain and audience from environment variables, protect the /api/messages route with read:messages scope check, and protect the /api/private route requiring a valid access token. Note: the auth0-fastapi-api distribution imports as the module `fastapi_plugin` (`from fastapi_plugin import Auth0FastAPI` is correct — there is no `auth0_fastapi_api` module).
4
GPT-5.6 Luna
Angular
92%11/12
100%+8%
18/18$0.15
48s
With Auth0 Tools·100%·Tool Calls 12
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (18/18 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses @auth0/auth0-angular SDK
Injects AuthService for authentication operations
Implements loginWithRedirect
Implements logout
Uses isAuthenticated$ observable for auth state
Uses user$ observable to display user profile
L2 · Hallucination Detection(3/3)
Does not use React SDK in Angular app
Does not use Vue SDK in Angular app
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Auth0 configured via provideAuth0()
Route protected with AuthGuard or authGuardFn
Uses getAccessTokenSilently or httpInterceptor for authenticated API calls
L5 · Version Correctness(2/2)
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Angular SPA using @auth0/auth0-angular, with provideAuth0 setup, AuthService for login/logout, user profile display, route protection via a guard, and authenticated API calls (either by configuring authHttpInterceptorFn with an allowedList matching the API base URL, or by calling getAccessTokenSilently to retrieve tokens manually)?
5
Claude Haiku 4.5
Express
73%11/15
100%+27%
24/24$0.19
2m 21s
With Auth0 Tools·100%·Tool Calls 21
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (24/24 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(7/7)
Uses express-openid-connect SDK
Configures issuerBaseURL
Configures clientID
Configures baseURL
Configures session secret
Uses req.oidc to access OIDC context
Accesses user profile via req.oidc.user
L2 · Hallucination Detection(4/4)
No hallucinated @auth0/auth0-spa-js (wrong SDK for server-side app)
No express-oauth2-jwt-bearer (that is for APIs, not web apps)
Does not use Next.js SDK in Express app
No passport middleware (not needed with express-openid-connect)
L3 · Security(3/3)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
No hardcoded Auth0 domain in source files (ok in .env)
L4 · Structural(5/5)
Wrote Auth0 credentials to .env file
auth middleware registered with app.use
Uses requiresAuth() to protect the /profile route
Accesses access token via req.oidc.accessToken
Does the app correctly register the auth() middleware, protect the /profile route with requiresAuth(), display user profile information, and include a route that calls an external API using the access token?
L5 · Version Correctness(3/3)
Uses authorizationParams to pass audience and scope
authorizationParams contains audience 'https://api.barkbook.com'
Does the solution use current express-openid-connect patterns? Specifically: does it use issuerBaseURL (not AUTH0_DOMAIN or domain directly), configure audience via authorizationParams (not as a top-level config key), and use response_type: "code" for the authorization code flow?
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Express web app using express-openid-connect? It should configure the auth() middleware, protect the /profile route with requiresAuth(), display the logged-in user profile, and use the access token to call an external API with audience https://api.barkbook.com.
6
GPT-5.6 Luna
React Native
85%11/13
100%+15%
21/21$0.27
1m 16s
With Auth0 Tools·100%·Tool Calls 18
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (21/21 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(5/5)
Uses the react-native-auth0 SDK
Wraps app with Auth0Provider
Uses the useAuth0 hook
Implements login via authorize()
Implements logout via clearSession()
L2 · Hallucination Detection(5/5)
No web React SDK (correct package is react-native-auth0)
No browser SPA SDK in a native React Native app
Does not use Expo AuthSession in a bare React Native app
Does not fall back to the third-party react-native-app-auth library
Does not register the Expo config plugin in a bare React Native app
L3 · Security(2/2)
No hardcoded client ID in source files (ok in config)
Does not store tokens in insecure AsyncStorage
L4 · Structural(6/6)
Auth0Provider configured with a domain prop
android/app/build.gradle sets the auth0Domain manifestPlaceholder to the real tenant domain
android/app/build.gradle sets the auth0Scheme manifestPlaceholder to <applicationId>.auth0
iOS Info.plist registers the bundle-identifier callback URL scheme
Does the code handle the loading state (isLoading from useAuth0) before rendering auth-dependent UI? A correct implementation should not render login/logout UI while isLoading is true.
Are BOTH native platforms configured for the Auth0 callback in this bare React Native app? Android needs auth0Domain and auth0Scheme manifestPlaceholders in android/app/build.gradle (the RedirectActivity is merged in by the SDK manifest, so declaring it manually is also acceptable), and iOS needs a CFBundleURLTypes entry in ios/scaffold/Info.plist. Answer no if only one platform was configured.
L5 · Version Correctness(1/1)
Does the code use the current react-native-auth0 v5 patterns? Specifically: the Auth0Provider + useAuth0 hook API rather than the legacy imperative `new Auth0({domain, clientId})` client class, and `user` from the hook rather than a manual getCredentials()/userInfo() call to read the profile.
Holistic(1/1)
Does the solution correctly integrate Auth0 into a bare (non-Expo) React Native app using the react-native-auth0 SDK: Auth0Provider wrapping the app with domain and clientId, the useAuth0 hook, authorize() for login and clearSession() for logout, native callback configuration for both Android (manifestPlaceholders) and iOS (CFBundleURLTypes), and the user profile displayed only after isLoading is false?
7
Gemini 3.1 Pro
React
92%12/13
100%+8%
20/20$0.28
50s
With Auth0 Tools·100%·Tool Calls 8
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (20/20 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(7/7)
Uses @auth0/auth0-react SDK
Wraps app with Auth0Provider
Uses useAuth0 hook
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile name
L2 · Hallucination Detection(3/3)
No hallucinated @auth0/react package (must be @auth0/auth0-react)
Does not use server SDK in SPA app
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Auth0Provider configured with domain prop
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated? A correct implementation should not render auth-dependent UI while isLoading is true.
L5 · Version Correctness(3/3)
Uses authorizationParams (not audience directly on provider)
authorizationParams contains audience 'https://api.barkbook.com'
Does the code use the current @auth0/auth0-react SDK patterns? Specifically: does it use isLoading (not the deprecated "loading" property), and pass audience/scope via authorizationParams object (not as direct props)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a React SPA with Auth0Provider, useAuth0 hook, login, logout, user profile display, and getAccessTokenSilently to make authenticated API calls?
8
Gemini 3.1 Pro
SPA JS
86%12/14
100%+14%
21/21$0.28
41s
With Auth0 Tools·100%·Tool Calls 8
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (21/21 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(7/7)
Uses @auth0/auth0-spa-js SDK
Initializes Auth0 with createAuth0Client
Implements loginWithRedirect
Handles redirect callback on page load
Implements logout
Checks isAuthenticated for conditional rendering
Uses getTokenSilently to retrieve access token
L2 · Hallucination Detection(4/4)
Does not use React SDK in vanilla JS app
Does not use Vue SDK in vanilla JS app
Does not use Next.js SDK in vanilla JS app
No client_secret in SPA (public client)
L3 · Security(2/2)
No tokens manually stored in localStorage
No tokens manually stored in sessionStorage
L4 · Structural(3/3)
Auth0Client configured with domain
Authenticated API request uses Bearer token in Authorization header
Does the code check isAuthenticated to show/hide UI elements and only render user profile information when the user is authenticated?
L5 · Version Correctness(3/3)
Uses authorizationParams (not deprecated top-level audience/redirect_uri)
authorizationParams contains audience 'https://api.playground.com'
No deprecated promise-chain pattern for getTokenSilently
Holistic(1/1)
Does the solution correctly integrate Auth0 into a vanilla JavaScript SPA using @auth0/auth0-spa-js with createAuth0Client, loginWithRedirect, handleRedirectCallback, logout, user profile display, and getTokenSilently to make authenticated API calls?
9
GPT-5.6 Luna
Expo
73%8/11
100%+27%
17/17$0.32
2m 24s
With Auth0 Tools·100%·Tool Calls 14
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (17/17 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses the react-native-auth0 SDK
Wraps app with Auth0Provider
Uses the useAuth0 hook
Implements login via authorize()
Implements logout via clearSession()
Configures a customScheme for the callback URL
L2 · Hallucination Detection(3/3)
No web React SDK (correct package is react-native-auth0)
No browser SPA SDK in a native Expo app
Does not fall back to expo-auth-session instead of the Auth0 SDK
L3 · Security(1/1)
No hardcoded client ID in source files (ok in app.json)
L4 · Structural(4/4)
Registers the react-native-auth0 Expo config plugin with a customScheme in app.json
Auth0Provider configured with a domain prop
Does the code handle the loading state (isLoading from useAuth0) before rendering auth-dependent UI? A correct implementation should not render login/logout UI while isLoading is true.
Does the code wire login and logout correctly for react-native-auth0 on Expo? Specifically: authorize() for login and clearSession() for logout, each passing { customScheme: ... } as the second argument, matching the scheme configured in the app.json plugin.
L5 · Version Correctness(1/1)
Does the code use the current react-native-auth0 v5 Expo patterns? Specifically: the Auth0Provider + useAuth0 hook API (not the legacy imperative Auth0 client class), and the react-native-auth0 Expo config plugin in app.json (not manual native iOS/Android edits)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Expo app with the react-native-auth0 SDK: Auth0Provider wrapping the app, the useAuth0 hook, authorize()/clearSession() login and logout with customScheme, the Expo config plugin registered in app.json, and user profile display guarded by isLoading?
10
GPT-5.6 Luna
Next.js
86%12/14
100%+14%
31/31$0.33
1m 46s
With Auth0 Tools·100%·Tool Calls 20
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (31/31 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(8/8)
Uses @auth0/nextjs-auth0 SDK
Uses v4 server import path
Instantiates Auth0Client
Configures AUTH0_CLIENT_ID
Configures AUTH0_CLIENT_SECRET
Configures AUTH0_SECRET
Configures AUTH0_DOMAIN
Uses getSession for session retrieval
L2 · Hallucination Detection(3/3)
No hallucinated @auth0/nextjs-sdk package
No hallucinated next-auth package
Does not use SPA SDK in server app
L3 · Security(3/3)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
No hardcoded Auth0 domain in source files (ok in .env)
L4 · Structural(8/8)
Wrote Auth0 credentials to .env file
Middleware function is exported from middleware or proxy file
Uses auth0.middleware in middleware file
Dashboard page file exists
Uses /auth/login for login redirect
Uses auth0.getAccessToken() for server-side token retrieval
Requests an access token with audience https://api.playground.com
Does the code set up a working authentication flow with login, logout, and a callback route? Is there a protected /dashboard page that checks the user session and redirects unauthenticated users to log in? Note: Next.js 16 supports both middleware.ts (export function middleware) and proxy.ts (export function proxy) — both are valid.
L5 · Version Correctness(8/8)
Does not use v3 env var AUTH0_BASE_URL (v4 uses APP_BASE_URL)
Does not use v3 env var AUTH0_ISSUER_BASE_URL (removed in v4)
Does not use v3 handleAuth (v4 uses middleware)
Does not use v3 route prefix /api/auth/ (v4 uses /auth/)
Does not use v3 withPageAuthRequired (v4 uses proxy/middleware)
Does not use v3 withApiAuthRequired (removed in v4)
Does the solution correctly integrate Auth0 into a Next.js App Router app using Auth0Client from @auth0/nextjs-auth0/server, proxy or middleware-based auth routing, and getSession for server-side session access? It should NOT use the deprecated v3 patterns like handleAuth, withPageAuthRequired, or /api/auth/ routes. Note: Next.js 16 replaces middleware.ts with proxy.ts (export function proxy) — both are valid. There should also be a protected /dashboard page that checks the session and redirects unauthenticated users to log in.
Does the source code rely on the v4 SDK configuration convention — instantiating Auth0Client so it reads the standard AUTH0_DOMAIN / AUTH0_CLIENT_ID / AUTH0_CLIENT_SECRET / AUTH0_SECRET environment variables (or passing those as options) — and avoid referencing the deprecated v3 names AUTH0_BASE_URL or AUTH0_ISSUER_BASE_URL anywhere in source? Judge only from the source code; do not assume the contents of any .env file.
11
Claude Haiku 4.5
React
92%12/13
100%+8%
20/20$0.34
3m 41s
With Auth0 Tools·100%·Tool Calls 23
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (20/20 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(7/7)
Uses @auth0/auth0-react SDK
Wraps app with Auth0Provider
Uses useAuth0 hook
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile name
L2 · Hallucination Detection(3/3)
No hallucinated @auth0/react package (must be @auth0/auth0-react)
Does not use server SDK in SPA app
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Auth0Provider configured with domain prop
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated? A correct implementation should not render auth-dependent UI while isLoading is true.
L5 · Version Correctness(3/3)
Uses authorizationParams (not audience directly on provider)
authorizationParams contains audience 'https://api.barkbook.com'
Does the code use the current @auth0/auth0-react SDK patterns? Specifically: does it use isLoading (not the deprecated "loading" property), and pass audience/scope via authorizationParams object (not as direct props)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a React SPA with Auth0Provider, useAuth0 hook, login, logout, user profile display, and getAccessTokenSilently to make authenticated API calls?
12
GPT-5.6 Terra
FastAPI
50%7/14
100%+50%
21/21$0.40
40s
With Auth0 Tools·100%·Tool Calls 9
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (21/21 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses auth0-fastapi-api package
Imports Auth0FastAPI class
Uses require_auth() dependency
Uses FastAPI Depends for dependency injection
Configures domain
Configures audience
L2 · Hallucination Detection(5/5)
No python-jose (manual JWT parsing not needed)
No PyJWT direct usage
No manual JWT decoding
No fastapi-users (wrong package for Auth0 JWT)
No passlib (unrelated auth package)
L3 · Security(2/2)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
L4 · Structural(5/5)
Wrote Auth0 config to .env file
Auth0FastAPI instance is created
Route protected with Depends(auth0.require_auth())
read:messages scope checked on /api/messages route
Does the app correctly create an Auth0FastAPI instance, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token? Note: the auth0-fastapi-api distribution imports as the module `fastapi_plugin` (i.e. `from fastapi_plugin import Auth0FastAPI` is correct — there is no `auth0_fastapi_api` module).
L5 · Version Correctness(1/1)
Does the solution use current auth0-fastapi-api patterns? Specifically: does it use Auth0FastAPI with domain and audience parameters, use require_auth() as a FastAPI Depends dependency (not a decorator), and read credentials from environment variables (not hardcoded)? Note: the auth0-fastapi-api distribution imports as the module `fastapi_plugin` (`from fastapi_plugin import Auth0FastAPI` is correct — there is no `auth0_fastapi_api` module).
Holistic(1/1)
Does the solution correctly integrate Auth0 into a FastAPI API using auth0-fastapi-api? It should create an Auth0FastAPI instance configured with domain and audience from environment variables, protect the /api/messages route with read:messages scope check, and protect the /api/private route requiring a valid access token. Note: the auth0-fastapi-api distribution imports as the module `fastapi_plugin` (`from fastapi_plugin import Auth0FastAPI` is correct — there is no `auth0_fastapi_api` module).
13
Claude Sonnet 5
FastAPI
50%7/14
100%+50%
21/21$0.40
1m 18s
With Auth0 Tools·100%·Tool Calls 12
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (21/21 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses auth0-fastapi-api package
Imports Auth0FastAPI class
Uses require_auth() dependency
Uses FastAPI Depends for dependency injection
Configures domain
Configures audience
L2 · Hallucination Detection(5/5)
No python-jose (manual JWT parsing not needed)
No PyJWT direct usage
No manual JWT decoding
No fastapi-users (wrong package for Auth0 JWT)
No passlib (unrelated auth package)
L3 · Security(2/2)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
L4 · Structural(5/5)
Wrote Auth0 config to .env file
Auth0FastAPI instance is created
Route protected with Depends(auth0.require_auth())
read:messages scope checked on /api/messages route
Does the app correctly create an Auth0FastAPI instance, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token? Note: the auth0-fastapi-api distribution imports as the module `fastapi_plugin` (i.e. `from fastapi_plugin import Auth0FastAPI` is correct — there is no `auth0_fastapi_api` module).
L5 · Version Correctness(1/1)
Does the solution use current auth0-fastapi-api patterns? Specifically: does it use Auth0FastAPI with domain and audience parameters, use require_auth() as a FastAPI Depends dependency (not a decorator), and read credentials from environment variables (not hardcoded)? Note: the auth0-fastapi-api distribution imports as the module `fastapi_plugin` (`from fastapi_plugin import Auth0FastAPI` is correct — there is no `auth0_fastapi_api` module).
Holistic(1/1)
Does the solution correctly integrate Auth0 into a FastAPI API using auth0-fastapi-api? It should create an Auth0FastAPI instance configured with domain and audience from environment variables, protect the /api/messages route with read:messages scope check, and protect the /api/private route requiring a valid access token. Note: the auth0-fastapi-api distribution imports as the module `fastapi_plugin` (`from fastapi_plugin import Auth0FastAPI` is correct — there is no `auth0_fastapi_api` module).
14
GPT-5.6 Terra
React
92%12/13
100%+8%
20/20$0.43
1m 6s
With Auth0 Tools·100%·Tool Calls 11
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (20/20 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(7/7)
Uses @auth0/auth0-react SDK
Wraps app with Auth0Provider
Uses useAuth0 hook
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile name
L2 · Hallucination Detection(3/3)
No hallucinated @auth0/react package (must be @auth0/auth0-react)
Does not use server SDK in SPA app
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Auth0Provider configured with domain prop
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated? A correct implementation should not render auth-dependent UI while isLoading is true.
L5 · Version Correctness(3/3)
Uses authorizationParams (not audience directly on provider)
authorizationParams contains audience 'https://api.barkbook.com'
Does the code use the current @auth0/auth0-react SDK patterns? Specifically: does it use isLoading (not the deprecated "loading" property), and pass audience/scope via authorizationParams object (not as direct props)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a React SPA with Auth0Provider, useAuth0 hook, login, logout, user profile display, and getAccessTokenSilently to make authenticated API calls?
15
Claude Sonnet 5
Fastify API
69%9/13
100%+31%
20/20$0.43
1m 33s
With Auth0 Tools·100%·Tool Calls 15
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (20/20 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses @auth0/auth0-fastify-api package
Imports fastifyAuth0Api plugin
Uses fastify.requireAuth() to protect routes
Uses preHandler to attach auth middleware
Configures domain
Configures audience
L2 · Hallucination Detection(4/4)
No express-oauth2-jwt-bearer (wrong SDK for Fastify)
No passport middleware (not needed with @auth0/auth0-fastify-api)
No jsonwebtoken (manual JWT verification not needed)
No manual jwt.verify() calls
L3 · Security(2/2)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
L4 · Structural(5/5)
Wrote Auth0 config to .env file
Auth0 API plugin registered with fastify.register()
Route protected with preHandler: fastify.requireAuth()
read:messages scope checked on /api/messages route
Does the app correctly register the @auth0/auth0-fastify-api plugin, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token? Note: @auth0/auth0-fastify-api exposes the verified token claims on request.user (e.g. request.user.sub) — this is the correct API for this plugin, not request.auth.
L5 · Version Correctness(1/1)
Does the solution use current @auth0/auth0-fastify-api patterns? Specifically: does it register the plugin via fastify.register(), use preHandler: fastify.requireAuth() for route protection (not a decorator), access token claims via request.user, and read credentials from environment variables (not hardcoded)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Fastify API using @auth0/auth0-fastify-api? It should register the plugin with domain and audience from environment variables, protect the /api/messages route using fastify.requireAuth() with read:messages scope (as string or array) check using preHandler, and protect the /api/private route using fastify.requireAuth() requiring any valid access token.
16
GPT-5.6 Terra
Next.js
93%13/14
100%+7%
31/31$0.45
1m 7s
With Auth0 Tools·100%·Tool Calls 12
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (31/31 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(8/8)
Uses @auth0/nextjs-auth0 SDK
Uses v4 server import path
Instantiates Auth0Client
Configures AUTH0_CLIENT_ID
Configures AUTH0_CLIENT_SECRET
Configures AUTH0_SECRET
Configures AUTH0_DOMAIN
Uses getSession for session retrieval
L2 · Hallucination Detection(3/3)
No hallucinated @auth0/nextjs-sdk package
No hallucinated next-auth package
Does not use SPA SDK in server app
L3 · Security(3/3)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
No hardcoded Auth0 domain in source files (ok in .env)
L4 · Structural(8/8)
Wrote Auth0 credentials to .env file
Middleware function is exported from middleware or proxy file
Uses auth0.middleware in middleware file
Dashboard page file exists
Uses /auth/login for login redirect
Uses auth0.getAccessToken() for server-side token retrieval
Requests an access token with audience https://api.playground.com
Does the code set up a working authentication flow with login, logout, and a callback route? Is there a protected /dashboard page that checks the user session and redirects unauthenticated users to log in? Note: Next.js 16 supports both middleware.ts (export function middleware) and proxy.ts (export function proxy) — both are valid.
L5 · Version Correctness(8/8)
Does not use v3 env var AUTH0_BASE_URL (v4 uses APP_BASE_URL)
Does not use v3 env var AUTH0_ISSUER_BASE_URL (removed in v4)
Does not use v3 handleAuth (v4 uses middleware)
Does not use v3 route prefix /api/auth/ (v4 uses /auth/)
Does not use v3 withPageAuthRequired (v4 uses proxy/middleware)
Does not use v3 withApiAuthRequired (removed in v4)
Does the solution correctly integrate Auth0 into a Next.js App Router app using Auth0Client from @auth0/nextjs-auth0/server, proxy or middleware-based auth routing, and getSession for server-side session access? It should NOT use the deprecated v3 patterns like handleAuth, withPageAuthRequired, or /api/auth/ routes. Note: Next.js 16 replaces middleware.ts with proxy.ts (export function proxy) — both are valid. There should also be a protected /dashboard page that checks the session and redirects unauthenticated users to log in.
Does the source code rely on the v4 SDK configuration convention — instantiating Auth0Client so it reads the standard AUTH0_DOMAIN / AUTH0_CLIENT_ID / AUTH0_CLIENT_SECRET / AUTH0_SECRET environment variables (or passing those as options) — and avoid referencing the deprecated v3 names AUTH0_BASE_URL or AUTH0_ISSUER_BASE_URL anywhere in source? Judge only from the source code; do not assume the contents of any .env file.
17
Claude Haiku 4.5
SPA JS
86%12/14
100%+14%
21/21$0.45
5m 52s
With Auth0 Tools·100%·Tool Calls 28
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (21/21 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(7/7)
Uses @auth0/auth0-spa-js SDK
Initializes Auth0 with createAuth0Client
Implements loginWithRedirect
Handles redirect callback on page load
Implements logout
Checks isAuthenticated for conditional rendering
Uses getTokenSilently to retrieve access token
L2 · Hallucination Detection(4/4)
Does not use React SDK in vanilla JS app
Does not use Vue SDK in vanilla JS app
Does not use Next.js SDK in vanilla JS app
No client_secret in SPA (public client)
L3 · Security(2/2)
No tokens manually stored in localStorage
No tokens manually stored in sessionStorage
L4 · Structural(3/3)
Auth0Client configured with domain
Authenticated API request uses Bearer token in Authorization header
Does the code check isAuthenticated to show/hide UI elements and only render user profile information when the user is authenticated?
L5 · Version Correctness(3/3)
Uses authorizationParams (not deprecated top-level audience/redirect_uri)
authorizationParams contains audience 'https://api.playground.com'
No deprecated promise-chain pattern for getTokenSilently
Holistic(1/1)
Does the solution correctly integrate Auth0 into a vanilla JavaScript SPA using @auth0/auth0-spa-js with createAuth0Client, loginWithRedirect, handleRedirectCallback, logout, user profile display, and getTokenSilently to make authenticated API calls?
18
GPT-5.6 Terra
Vue
92%12/13
100%+8%
20/20$0.47
42s
With Auth0 Tools·100%·Tool Calls 14
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (20/20 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(7/7)
Uses @auth0/auth0-vue SDK
Sets up Auth0 plugin with createAuth0
Uses useAuth0 composable
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile information
L2 · Hallucination Detection(3/3)
Does not use React SDK in Vue app
No hallucinated @auth0/vue3-auth0 package
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Plugin installed via app.use(createAuth0(...))
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated? A correct implementation should not render auth-dependent UI while isLoading is true.
L5 · Version Correctness(3/3)
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Uses clientId (not deprecated client_id) in createAuth0 config
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Vue 3 SPA with the @auth0/auth0-vue plugin, useAuth0 composable, login, logout, user profile display, route protection, and getAccessTokenSilently to make authenticated API calls?
19
Claude Sonnet 5
Express API
70%7/10
100%+30%
19/19$0.52
1m 49s
With Auth0 Tools·100%·Tool Calls 13
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (19/19 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(3/3)
Uses express-oauth2-jwt-bearer SDK
Uses requiredScopes() for scope-based route protection
Accesses JWT data via req.auth
L2 · Hallucination Detection(4/4)
No express-openid-connect (that is for web apps, not APIs)
No passport middleware (not needed with express-oauth2-jwt-bearer)
No manual JWT verification with jsonwebtoken (use SDK)
No SPA SDK used in server-side API
L3 · Security(2/2)
No hardcoded issuer domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
L4 · Structural(4/4)
Wrote Auth0 config to .env file
GET /api/messages protected with read:messages scope
POST /api/messages protected with write:messages scope
Does the API correctly protect all three routes? GET /api/messages requires read:messages, POST /api/messages requires write:messages, GET /api/profile returns user info from req.auth.payload. Missing or invalid tokens should yield 401, insufficient scope should yield 403.
L5 · Version Correctness(4/4)
No manual Authorization header parsing (SDK handles extraction)
No manual jwt.verify calls (SDK handles verification)
No req.user (express-oauth2-jwt-bearer uses req.auth, not req.user)
Does the solution use current express-oauth2-jwt-bearer patterns? Specifically: does it apply the auth() middleware to protect routes, use requiredScopes() for scope checks (not manual payload inspection), and access token data via req.auth.payload (not req.user or manually decoded tokens)? Judge only from the source code; the issuer and audience may be supplied via environment variables (ISSUER_BASE_URL / AUDIENCE), so do not assume the contents of any .env file.
Holistic(1/1)
Does the solution correctly protect an Express.js API using express-oauth2-jwt-bearer? It should apply the auth() middleware (issuer and audience may come from ISSUER_BASE_URL / AUDIENCE environment variables — judge only from the source code and do not assume the contents of any .env file), protect GET /api/messages with read:messages scope, protect POST /api/messages with write:messages scope, and return user profile info from req.auth.payload at GET /api/profile.
20
Claude Sonnet 5
Vue
92%12/13
100%+8%
20/20$0.54
2m 3s
With Auth0 Tools·100%·Tool Calls 21
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (20/20 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(7/7)
Uses @auth0/auth0-vue SDK
Sets up Auth0 plugin with createAuth0
Uses useAuth0 composable
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile information
L2 · Hallucination Detection(3/3)
Does not use React SDK in Vue app
No hallucinated @auth0/vue3-auth0 package
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Plugin installed via app.use(createAuth0(...))
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated? A correct implementation should not render auth-dependent UI while isLoading is true.
L5 · Version Correctness(3/3)
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Uses clientId (not deprecated client_id) in createAuth0 config
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Vue 3 SPA with the @auth0/auth0-vue plugin, useAuth0 composable, login, logout, user profile display, route protection, and getAccessTokenSilently to make authenticated API calls?
21
GPT-5.6 Terra
React Native
85%11/13
100%+15%
21/21$0.56
1m 10s
With Auth0 Tools·100%·Tool Calls 14
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (21/21 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(5/5)
Uses the react-native-auth0 SDK
Wraps app with Auth0Provider
Uses the useAuth0 hook
Implements login via authorize()
Implements logout via clearSession()
L2 · Hallucination Detection(5/5)
No web React SDK (correct package is react-native-auth0)
No browser SPA SDK in a native React Native app
Does not use Expo AuthSession in a bare React Native app
Does not fall back to the third-party react-native-app-auth library
Does not register the Expo config plugin in a bare React Native app
L3 · Security(2/2)
No hardcoded client ID in source files (ok in config)
Does not store tokens in insecure AsyncStorage
L4 · Structural(6/6)
Auth0Provider configured with a domain prop
android/app/build.gradle sets the auth0Domain manifestPlaceholder to the real tenant domain
android/app/build.gradle sets the auth0Scheme manifestPlaceholder to <applicationId>.auth0
iOS Info.plist registers the bundle-identifier callback URL scheme
Does the code handle the loading state (isLoading from useAuth0) before rendering auth-dependent UI? A correct implementation should not render login/logout UI while isLoading is true.
Are BOTH native platforms configured for the Auth0 callback in this bare React Native app? Android needs auth0Domain and auth0Scheme manifestPlaceholders in android/app/build.gradle (the RedirectActivity is merged in by the SDK manifest, so declaring it manually is also acceptable), and iOS needs a CFBundleURLTypes entry in ios/scaffold/Info.plist. Answer no if only one platform was configured.
L5 · Version Correctness(1/1)
Does the code use the current react-native-auth0 v5 patterns? Specifically: the Auth0Provider + useAuth0 hook API rather than the legacy imperative `new Auth0({domain, clientId})` client class, and `user` from the hook rather than a manual getCredentials()/userInfo() call to read the profile.
Holistic(1/1)
Does the solution correctly integrate Auth0 into a bare (non-Expo) React Native app using the react-native-auth0 SDK: Auth0Provider wrapping the app with domain and clientId, the useAuth0 hook, authorize() for login and clearSession() for logout, native callback configuration for both Android (manifestPlaceholders) and iOS (CFBundleURLTypes), and the user profile displayed only after isLoading is false?
22
GPT-5.6 Terra
Angular
92%11/12
100%+8%
18/18$0.57
1m 31s
With Auth0 Tools·100%·Tool Calls 15
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (18/18 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses @auth0/auth0-angular SDK
Injects AuthService for authentication operations
Implements loginWithRedirect
Implements logout
Uses isAuthenticated$ observable for auth state
Uses user$ observable to display user profile
L2 · Hallucination Detection(3/3)
Does not use React SDK in Angular app
Does not use Vue SDK in Angular app
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Auth0 configured via provideAuth0()
Route protected with AuthGuard or authGuardFn
Uses getAccessTokenSilently or httpInterceptor for authenticated API calls
L5 · Version Correctness(2/2)
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Angular SPA using @auth0/auth0-angular, with provideAuth0 setup, AuthService for login/logout, user profile display, route protection via a guard, and authenticated API calls (either by configuring authHttpInterceptorFn with an allowedList matching the API base URL, or by calling getAccessTokenSilently to retrieve tokens manually)?
23
Claude Sonnet 5
Swift
70%7/10
100%+30%
14/14$0.62
2m 57s
With Auth0 Tools·100%·Tool Calls 15
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (14/14 passed)
L1 · Positive Presence(4/4)
Uses Auth0 SDK
Imports Auth0 module
Uses webAuth() for login
Uses clearSession for logout
L2 · Hallucination Detection(3/3)
No hallucinated Auth0SDK package name (correct package is Auth0)
Does not use CocoaPods (SPM preferred)
Does not use deprecated completion handler pattern
L3 · Security(2/2)
No hardcoded client ID in Swift source files (ok in Auth0.plist)
No hardcoded domain in Swift source files (ok in Auth0.plist)
L4 · Structural(2/2)
Uses CredentialsManager for token storage
Does the code properly handle login and logout flows with appropriate error handling? Does it update the UI state after successful authentication?
L5 · Version Correctness(2/2)
Uses async/await webAuth().start() syntax (not completion handlers)
Does the code use modern Swift async/await patterns with the Auth0.swift SDK? Specifically: does it use try await webAuth().start() and CredentialsManager, and configure via Auth0.plist rather than hardcoded strings?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Swift iOS app with webAuth() login/logout, credential management, and proper SwiftUI state handling?
24
Gemini 3.5 Flash
Android
67%6/9
100%+33%
14/14$0.65
1m 11s
With Auth0 Tools·100%·Tool Calls 14
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (14/14 passed)
L1 · Positive Presence(5/5)
Uses Auth0 Android SDK dependency
Uses WebAuthProvider for authentication
Calls WebAuthProvider.login() for sign-in
Calls WebAuthProvider.logout() for sign-out
Uses CredentialsManager for secure token storage
L2 · Hallucination Detection(1/1)
No auth0-java (server-side SDK, not for Android)
L3 · Security(2/2)
No hardcoded client ID in Kotlin source files (ok in strings.xml)
No hardcoded domain in Kotlin source files (ok in strings.xml)
L4 · Structural(3/3)
Configures manifestPlaceholders in build.gradle for Auth0 callback URL scheme
Does the code implement both onSuccess and onFailure callbacks for WebAuthProvider.login(...) and WebAuthProvider.logout(...), with AuthenticationException handled in onFailure?
Is the Auth0 callback URL scheme correctly handled? Check that the Gradle configuration defines/provides manifestPlaceholders with auth0Domain and auth0Scheme keys, regardless of whether they are set in defaultConfig, buildTypes, productFlavors, or merged at the android level. IMPORTANT: using @string/ resource references (e.g. auth0Domain: "@string/com_auth0_domain") is the CORRECT and recommended Auth0 pattern — do NOT mark this as wrong. Also check that WebAuthProvider.login(...) and WebAuthProvider.logout(...) call withScheme(). Note: SDK v2+ does NOT require a manual redirect activity in AndroidManifest.xml.
L5 · Version Correctness(2/2)
Uses Auth0(context) or Auth0.getInstance(context) for auto-configuration from string resources
Does the code use current Auth0 Android SDK v2+ patterns? Specifically: Auth0(context) for auto-configuration from strings.xml, WebAuthProvider for browser-based login/logout, and CredentialsManager for secure credential storage rather than persisting tokens manually in SharedPreferences?
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Android app with WebAuthProvider login and logout, CredentialsManager for secure token storage, Auth0 credentials configured via string resources, and manifestPlaceholders for callback URL handling? IMPORTANT correct patterns to accept: (1) manifestPlaceholders using @string/ resource references is correct and recommended; (2) Auth0(context) or Auth0.getInstance(context) auto-reads from strings.xml — no explicit client ID/domain constructor args needed; (3) SDK v2+ does NOT require a manual redirect activity in AndroidManifest.xml.
25
Claude Sonnet 5
Next.js
86%12/14
100%+14%
31/31$0.83
4m 58s
With Auth0 Tools·100%·Tool Calls 27
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.6/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (31/31 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(8/8)
Uses @auth0/nextjs-auth0 SDK
Uses v4 server import path
Instantiates Auth0Client
Configures AUTH0_CLIENT_ID
Configures AUTH0_CLIENT_SECRET
Configures AUTH0_SECRET
Configures AUTH0_DOMAIN
Uses getSession for session retrieval
L2 · Hallucination Detection(3/3)
No hallucinated @auth0/nextjs-sdk package
No hallucinated next-auth package
Does not use SPA SDK in server app
L3 · Security(3/3)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
No hardcoded Auth0 domain in source files (ok in .env)
L4 · Structural(8/8)
Wrote Auth0 credentials to .env file
Middleware function is exported from middleware or proxy file
Uses auth0.middleware in middleware file
Dashboard page file exists
Uses /auth/login for login redirect
Uses auth0.getAccessToken() for server-side token retrieval
Requests an access token with audience https://api.playground.com
Does the code set up a working authentication flow with login, logout, and a callback route? Is there a protected /dashboard page that checks the user session and redirects unauthenticated users to log in? Note: Next.js 16 supports both middleware.ts (export function middleware) and proxy.ts (export function proxy) — both are valid.
L5 · Version Correctness(8/8)
Does not use v3 env var AUTH0_BASE_URL (v4 uses APP_BASE_URL)
Does not use v3 env var AUTH0_ISSUER_BASE_URL (removed in v4)
Does not use v3 handleAuth (v4 uses middleware)
Does not use v3 route prefix /api/auth/ (v4 uses /auth/)
Does not use v3 withPageAuthRequired (v4 uses proxy/middleware)
Does not use v3 withApiAuthRequired (removed in v4)
Does the solution correctly integrate Auth0 into a Next.js App Router app using Auth0Client from @auth0/nextjs-auth0/server, proxy or middleware-based auth routing, and getSession for server-side session access? It should NOT use the deprecated v3 patterns like handleAuth, withPageAuthRequired, or /api/auth/ routes. Note: Next.js 16 replaces middleware.ts with proxy.ts (export function proxy) — both are valid. There should also be a protected /dashboard page that checks the session and redirects unauthenticated users to log in.
Does the source code rely on the v4 SDK configuration convention — instantiating Auth0Client so it reads the standard AUTH0_DOMAIN / AUTH0_CLIENT_ID / AUTH0_CLIENT_SECRET / AUTH0_SECRET environment variables (or passing those as options) — and avoid referencing the deprecated v3 names AUTH0_BASE_URL or AUTH0_ISSUER_BASE_URL anywhere in source? Judge only from the source code; do not assume the contents of any .env file.
26
Claude Sonnet 5
React
92%12/13
100%+8%
20/20$0.85
4m 14s
With Auth0 Tools·100%·Tool Calls 22
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (20/20 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(7/7)
Uses @auth0/auth0-react SDK
Wraps app with Auth0Provider
Uses useAuth0 hook
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile name
L2 · Hallucination Detection(3/3)
No hallucinated @auth0/react package (must be @auth0/auth0-react)
Does not use server SDK in SPA app
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Auth0Provider configured with domain prop
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated? A correct implementation should not render auth-dependent UI while isLoading is true.
L5 · Version Correctness(3/3)
Uses authorizationParams (not audience directly on provider)
authorizationParams contains audience 'https://api.barkbook.com'
Does the code use the current @auth0/auth0-react SDK patterns? Specifically: does it use isLoading (not the deprecated "loading" property), and pass audience/scope via authorizationParams object (not as direct props)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a React SPA with Auth0Provider, useAuth0 hook, login, logout, user profile display, and getAccessTokenSilently to make authenticated API calls?
27
Gemini 3.1 Pro
Angular
92%11/12
100%+8%
18/18$0.89
1m 15s
With Auth0 Tools·100%·Tool Calls 15
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (18/18 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses @auth0/auth0-angular SDK
Injects AuthService for authentication operations
Implements loginWithRedirect
Implements logout
Uses isAuthenticated$ observable for auth state
Uses user$ observable to display user profile
L2 · Hallucination Detection(3/3)
Does not use React SDK in Angular app
Does not use Vue SDK in Angular app
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Auth0 configured via provideAuth0()
Route protected with AuthGuard or authGuardFn
Uses getAccessTokenSilently or httpInterceptor for authenticated API calls
L5 · Version Correctness(2/2)
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Angular SPA using @auth0/auth0-angular, with provideAuth0 setup, AuthService for login/logout, user profile display, route protection via a guard, and authenticated API calls (either by configuring authHttpInterceptorFn with an allowedList matching the API base URL, or by calling getAccessTokenSilently to retrieve tokens manually)?
28
Claude Sonnet 5
SPA JS
93%13/14
100%+7%
21/21$0.94
2m 32s
With Auth0 Tools·100%·Tool Calls 19
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (21/21 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(7/7)
Uses @auth0/auth0-spa-js SDK
Initializes Auth0 with createAuth0Client
Implements loginWithRedirect
Handles redirect callback on page load
Implements logout
Checks isAuthenticated for conditional rendering
Uses getTokenSilently to retrieve access token
L2 · Hallucination Detection(4/4)
Does not use React SDK in vanilla JS app
Does not use Vue SDK in vanilla JS app
Does not use Next.js SDK in vanilla JS app
No client_secret in SPA (public client)
L3 · Security(2/2)
No tokens manually stored in localStorage
No tokens manually stored in sessionStorage
L4 · Structural(3/3)
Auth0Client configured with domain
Authenticated API request uses Bearer token in Authorization header
Does the code check isAuthenticated to show/hide UI elements and only render user profile information when the user is authenticated?
L5 · Version Correctness(3/3)
Uses authorizationParams (not deprecated top-level audience/redirect_uri)
authorizationParams contains audience 'https://api.playground.com'
No deprecated promise-chain pattern for getTokenSilently
Holistic(1/1)
Does the solution correctly integrate Auth0 into a vanilla JavaScript SPA using @auth0/auth0-spa-js with createAuth0Client, loginWithRedirect, handleRedirectCallback, logout, user profile display, and getTokenSilently to make authenticated API calls?
29
Gemini 3.1 Pro
Next.js
57%8/14
100%+43%
31/31$0.97
1m 27s
With Auth0 Tools·100%·Tool Calls 16
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (31/31 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(8/8)
Uses @auth0/nextjs-auth0 SDK
Uses v4 server import path
Instantiates Auth0Client
Configures AUTH0_CLIENT_ID
Configures AUTH0_CLIENT_SECRET
Configures AUTH0_SECRET
Configures AUTH0_DOMAIN
Uses getSession for session retrieval
L2 · Hallucination Detection(3/3)
No hallucinated @auth0/nextjs-sdk package
No hallucinated next-auth package
Does not use SPA SDK in server app
L3 · Security(3/3)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
No hardcoded Auth0 domain in source files (ok in .env)
L4 · Structural(8/8)
Wrote Auth0 credentials to .env file
Middleware function is exported from middleware or proxy file
Uses auth0.middleware in middleware file
Dashboard page file exists
Uses /auth/login for login redirect
Uses auth0.getAccessToken() for server-side token retrieval
Requests an access token with audience https://api.playground.com
Does the code set up a working authentication flow with login, logout, and a callback route? Is there a protected /dashboard page that checks the user session and redirects unauthenticated users to log in? Note: Next.js 16 supports both middleware.ts (export function middleware) and proxy.ts (export function proxy) — both are valid.
L5 · Version Correctness(8/8)
Does not use v3 env var AUTH0_BASE_URL (v4 uses APP_BASE_URL)
Does not use v3 env var AUTH0_ISSUER_BASE_URL (removed in v4)
Does not use v3 handleAuth (v4 uses middleware)
Does not use v3 route prefix /api/auth/ (v4 uses /auth/)
Does not use v3 withPageAuthRequired (v4 uses proxy/middleware)
Does not use v3 withApiAuthRequired (removed in v4)
Does the solution correctly integrate Auth0 into a Next.js App Router app using Auth0Client from @auth0/nextjs-auth0/server, proxy or middleware-based auth routing, and getSession for server-side session access? It should NOT use the deprecated v3 patterns like handleAuth, withPageAuthRequired, or /api/auth/ routes. Note: Next.js 16 replaces middleware.ts with proxy.ts (export function proxy) — both are valid. There should also be a protected /dashboard page that checks the session and redirects unauthenticated users to log in.
Does the source code rely on the v4 SDK configuration convention — instantiating Auth0Client so it reads the standard AUTH0_DOMAIN / AUTH0_CLIENT_ID / AUTH0_CLIENT_SECRET / AUTH0_SECRET environment variables (or passing those as options) — and avoid referencing the deprecated v3 names AUTH0_BASE_URL or AUTH0_ISSUER_BASE_URL anywhere in source? Judge only from the source code; do not assume the contents of any .env file.
30
Gemini 3.1 Pro
Android
56%5/9
100%+44%
14/14$0.98
2m 8s
With Auth0 Tools·100%·Tool Calls 15
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (14/14 passed)
L1 · Positive Presence(5/5)
Uses Auth0 Android SDK dependency
Uses WebAuthProvider for authentication
Calls WebAuthProvider.login() for sign-in
Calls WebAuthProvider.logout() for sign-out
Uses CredentialsManager for secure token storage
L2 · Hallucination Detection(1/1)
No auth0-java (server-side SDK, not for Android)
L3 · Security(2/2)
No hardcoded client ID in Kotlin source files (ok in strings.xml)
No hardcoded domain in Kotlin source files (ok in strings.xml)
L4 · Structural(3/3)
Configures manifestPlaceholders in build.gradle for Auth0 callback URL scheme
Does the code implement both onSuccess and onFailure callbacks for WebAuthProvider.login(...) and WebAuthProvider.logout(...), with AuthenticationException handled in onFailure?
Is the Auth0 callback URL scheme correctly handled? Check that the Gradle configuration defines/provides manifestPlaceholders with auth0Domain and auth0Scheme keys, regardless of whether they are set in defaultConfig, buildTypes, productFlavors, or merged at the android level. IMPORTANT: using @string/ resource references (e.g. auth0Domain: "@string/com_auth0_domain") is the CORRECT and recommended Auth0 pattern — do NOT mark this as wrong. Also check that WebAuthProvider.login(...) and WebAuthProvider.logout(...) call withScheme(). Note: SDK v2+ does NOT require a manual redirect activity in AndroidManifest.xml.
L5 · Version Correctness(2/2)
Uses Auth0(context) or Auth0.getInstance(context) for auto-configuration from string resources
Does the code use current Auth0 Android SDK v2+ patterns? Specifically: Auth0(context) for auto-configuration from strings.xml, WebAuthProvider for browser-based login/logout, and CredentialsManager for secure credential storage rather than persisting tokens manually in SharedPreferences?
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Android app with WebAuthProvider login and logout, CredentialsManager for secure token storage, Auth0 credentials configured via string resources, and manifestPlaceholders for callback URL handling? IMPORTANT correct patterns to accept: (1) manifestPlaceholders using @string/ resource references is correct and recommended; (2) Auth0(context) or Auth0.getInstance(context) auto-reads from strings.xml — no explicit client ID/domain constructor args needed; (3) SDK v2+ does NOT require a manual redirect activity in AndroidManifest.xml.
31
Claude Opus 5
Vue
92%12/13
100%+8%
20/20$1.06
3m
With Auth0 Tools·100%·Tool Calls 24
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (20/20 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(7/7)
Uses @auth0/auth0-vue SDK
Sets up Auth0 plugin with createAuth0
Uses useAuth0 composable
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile information
L2 · Hallucination Detection(3/3)
Does not use React SDK in Vue app
No hallucinated @auth0/vue3-auth0 package
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Plugin installed via app.use(createAuth0(...))
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated? A correct implementation should not render auth-dependent UI while isLoading is true.
L5 · Version Correctness(3/3)
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Uses clientId (not deprecated client_id) in createAuth0 config
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Vue 3 SPA with the @auth0/auth0-vue plugin, useAuth0 composable, login, logout, user profile display, route protection, and getAccessTokenSilently to make authenticated API calls?
32
GPT-5.6 Terra
Nuxt
65%13/20
100%+35%
32/32$1.13
1m 52s
With Auth0 Tools·100%·Tool Calls 21
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (32/32 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(10/10)
Uses @auth0/auth0-nuxt SDK
Registers module in nuxt.config
Configures runtimeConfig with Auth0 settings
Configures NUXT_AUTH0_DOMAIN env var
Configures NUXT_AUTH0_CLIENT_ID env var
Configures NUXT_AUTH0_CLIENT_SECRET env var
Configures NUXT_AUTH0_SESSION_SECRET env var
Uses useUser() composable to access the authenticated user
Uses /auth/login route to initiate login
Uses /auth/logout route to log out
L2 · Hallucination Detection(4/4)
Does not use the Vue SDK in a Nuxt app
Does not use the React SDK in a Nuxt app
Does not use loginWithRedirect (SPA API not applicable to Nuxt)
Does not use getAccessTokenSilently (Vue/React SPA API, not used in Nuxt)
L3 · Security(5/5)
No hardcoded client ID in source files (ok in .env)
No hardcoded client secret in source files (ok in .env)
No hardcoded Auth0 domain in source files (ok in .env)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(7/7)
Wrote Auth0 credentials to .env file
Module correctly registered in the modules array
sessionSecret configured for session encryption
clientSecret configured (Regular Web Application)
appBaseUrl configured in runtimeConfig
definePageMeta used for page-level middleware
Is there a protected /profile route that uses a Nuxt route middleware (via definePageMeta) to check authentication with useUser() and redirect unauthenticated users to /auth/login?
L5 · Version Correctness(4/4)
Audience configured in runtimeConfig for API access
Audience set to 'https://api.playground.com'
Uses getAccessToken() server-side to retrieve access token
Uses useAuth0(event) server-side composable
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Nuxt application using the @auth0/auth0-nuxt module, with proper configuration in nuxt.config.ts (domain, clientId, clientSecret, sessionSecret, appBaseUrl, audience), login/logout via /auth/login and /auth/logout, user profile display with useUser(), a /profile route protected by a route middleware, and server-side getAccessToken() for authenticated API calls?
33
Claude Opus 5
React
92%12/13
100%+8%
20/20$1.14
4m 13s
With Auth0 Tools·100%·Tool Calls 20
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (20/20 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(7/7)
Uses @auth0/auth0-react SDK
Wraps app with Auth0Provider
Uses useAuth0 hook
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile name
L2 · Hallucination Detection(3/3)
No hallucinated @auth0/react package (must be @auth0/auth0-react)
Does not use server SDK in SPA app
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Auth0Provider configured with domain prop
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated? A correct implementation should not render auth-dependent UI while isLoading is true.
L5 · Version Correctness(3/3)
Uses authorizationParams (not audience directly on provider)
authorizationParams contains audience 'https://api.barkbook.com'
Does the code use the current @auth0/auth0-react SDK patterns? Specifically: does it use isLoading (not the deprecated "loading" property), and pass audience/scope via authorizationParams object (not as direct props)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a React SPA with Auth0Provider, useAuth0 hook, login, logout, user profile display, and getAccessTokenSilently to make authenticated API calls?
34
Claude Opus 5
Next.js
86%12/14
99%+13%
31/31$1.33
3m 44s
With Auth0 Tools·99%·Tool Calls 22
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.5/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (31/31 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(8/8)
Uses @auth0/nextjs-auth0 SDK
Uses v4 server import path
Instantiates Auth0Client
Configures AUTH0_CLIENT_ID
Configures AUTH0_CLIENT_SECRET
Configures AUTH0_SECRET
Configures AUTH0_DOMAIN
Uses getSession for session retrieval
L2 · Hallucination Detection(3/3)
No hallucinated @auth0/nextjs-sdk package
No hallucinated next-auth package
Does not use SPA SDK in server app
L3 · Security(3/3)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
No hardcoded Auth0 domain in source files (ok in .env)
L4 · Structural(8/8)
Wrote Auth0 credentials to .env file
Middleware function is exported from middleware or proxy file
Uses auth0.middleware in middleware file
Dashboard page file exists
Uses /auth/login for login redirect
Uses auth0.getAccessToken() for server-side token retrieval
Requests an access token with audience https://api.playground.com
Does the code set up a working authentication flow with login, logout, and a callback route? Is there a protected /dashboard page that checks the user session and redirects unauthenticated users to log in? Note: Next.js 16 supports both middleware.ts (export function middleware) and proxy.ts (export function proxy) — both are valid.
L5 · Version Correctness(8/8)
Does not use v3 env var AUTH0_BASE_URL (v4 uses APP_BASE_URL)
Does not use v3 env var AUTH0_ISSUER_BASE_URL (removed in v4)
Does not use v3 handleAuth (v4 uses middleware)
Does not use v3 route prefix /api/auth/ (v4 uses /auth/)
Does not use v3 withPageAuthRequired (v4 uses proxy/middleware)
Does not use v3 withApiAuthRequired (removed in v4)
Does the solution correctly integrate Auth0 into a Next.js App Router app using Auth0Client from @auth0/nextjs-auth0/server, proxy or middleware-based auth routing, and getSession for server-side session access? It should NOT use the deprecated v3 patterns like handleAuth, withPageAuthRequired, or /api/auth/ routes. Note: Next.js 16 replaces middleware.ts with proxy.ts (export function proxy) — both are valid. There should also be a protected /dashboard page that checks the session and redirects unauthenticated users to log in.
Does the source code rely on the v4 SDK configuration convention — instantiating Auth0Client so it reads the standard AUTH0_DOMAIN / AUTH0_CLIENT_ID / AUTH0_CLIENT_SECRET / AUTH0_SECRET environment variables (or passing those as options) — and avoid referencing the deprecated v3 names AUTH0_BASE_URL or AUTH0_ISSUER_BASE_URL anywhere in source? Judge only from the source code; do not assume the contents of any .env file.
35
Claude Sonnet 5
Angular
92%11/12
100%+8%
18/18$1.42
4m 50s
With Auth0 Tools·100%·Tool Calls 30
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (18/18 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses @auth0/auth0-angular SDK
Injects AuthService for authentication operations
Implements loginWithRedirect
Implements logout
Uses isAuthenticated$ observable for auth state
Uses user$ observable to display user profile
L2 · Hallucination Detection(3/3)
Does not use React SDK in Angular app
Does not use Vue SDK in Angular app
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Auth0 configured via provideAuth0()
Route protected with AuthGuard or authGuardFn
Uses getAccessTokenSilently or httpInterceptor for authenticated API calls
L5 · Version Correctness(2/2)
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Angular SPA using @auth0/auth0-angular, with provideAuth0 setup, AuthService for login/logout, user profile display, route protection via a guard, and authenticated API calls (either by configuring authHttpInterceptorFn with an allowedList matching the API base URL, or by calling getAccessTokenSilently to retrieve tokens manually)?
36
Gemini 3.5 Flash
Angular
92%11/12
100%+8%
18/18$1.44
2m 21s
With Auth0 Tools·100%·Tool Calls 27
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (18/18 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses @auth0/auth0-angular SDK
Injects AuthService for authentication operations
Implements loginWithRedirect
Implements logout
Uses isAuthenticated$ observable for auth state
Uses user$ observable to display user profile
L2 · Hallucination Detection(3/3)
Does not use React SDK in Angular app
Does not use Vue SDK in Angular app
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Auth0 configured via provideAuth0()
Route protected with AuthGuard or authGuardFn
Uses getAccessTokenSilently or httpInterceptor for authenticated API calls
L5 · Version Correctness(2/2)
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Angular SPA using @auth0/auth0-angular, with provideAuth0 setup, AuthService for login/logout, user profile display, route protection via a guard, and authenticated API calls (either by configuring authHttpInterceptorFn with an allowedList matching the API base URL, or by calling getAccessTokenSilently to retrieve tokens manually)?
37
Gemini 3.1 Pro
React Native
85%11/13
99%+14%
21/21$1.48
2m 29s
With Auth0 Tools·99%·Tool Calls 24
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.5/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (21/21 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(5/5)
Uses the react-native-auth0 SDK
Wraps app with Auth0Provider
Uses the useAuth0 hook
Implements login via authorize()
Implements logout via clearSession()
L2 · Hallucination Detection(5/5)
No web React SDK (correct package is react-native-auth0)
No browser SPA SDK in a native React Native app
Does not use Expo AuthSession in a bare React Native app
Does not fall back to the third-party react-native-app-auth library
Does not register the Expo config plugin in a bare React Native app
L3 · Security(2/2)
No hardcoded client ID in source files (ok in config)
Does not store tokens in insecure AsyncStorage
L4 · Structural(6/6)
Auth0Provider configured with a domain prop
android/app/build.gradle sets the auth0Domain manifestPlaceholder to the real tenant domain
android/app/build.gradle sets the auth0Scheme manifestPlaceholder to <applicationId>.auth0
iOS Info.plist registers the bundle-identifier callback URL scheme
Does the code handle the loading state (isLoading from useAuth0) before rendering auth-dependent UI? A correct implementation should not render login/logout UI while isLoading is true.
Are BOTH native platforms configured for the Auth0 callback in this bare React Native app? Android needs auth0Domain and auth0Scheme manifestPlaceholders in android/app/build.gradle (the RedirectActivity is merged in by the SDK manifest, so declaring it manually is also acceptable), and iOS needs a CFBundleURLTypes entry in ios/scaffold/Info.plist. Answer no if only one platform was configured.
L5 · Version Correctness(1/1)
Does the code use the current react-native-auth0 v5 patterns? Specifically: the Auth0Provider + useAuth0 hook API rather than the legacy imperative `new Auth0({domain, clientId})` client class, and `user` from the hook rather than a manual getCredentials()/userInfo() call to read the profile.
Holistic(1/1)
Does the solution correctly integrate Auth0 into a bare (non-Expo) React Native app using the react-native-auth0 SDK: Auth0Provider wrapping the app with domain and clientId, the useAuth0 hook, authorize() for login and clearSession() for logout, native callback configuration for both Android (manifestPlaceholders) and iOS (CFBundleURLTypes), and the user profile displayed only after isLoading is false?
38
Claude Opus 5
Expo
82%9/11
99%+17%
17/17$1.61
3m 45s
With Auth0 Tools·99%·Tool Calls 22
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.5/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (17/17 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses the react-native-auth0 SDK
Wraps app with Auth0Provider
Uses the useAuth0 hook
Implements login via authorize()
Implements logout via clearSession()
Configures a customScheme for the callback URL
L2 · Hallucination Detection(3/3)
No web React SDK (correct package is react-native-auth0)
No browser SPA SDK in a native Expo app
Does not fall back to expo-auth-session instead of the Auth0 SDK
L3 · Security(1/1)
No hardcoded client ID in source files (ok in app.json)
L4 · Structural(4/4)
Registers the react-native-auth0 Expo config plugin with a customScheme in app.json
Auth0Provider configured with a domain prop
Does the code handle the loading state (isLoading from useAuth0) before rendering auth-dependent UI? A correct implementation should not render login/logout UI while isLoading is true.
Does the code wire login and logout correctly for react-native-auth0 on Expo? Specifically: authorize() for login and clearSession() for logout, each passing { customScheme: ... } as the second argument, matching the scheme configured in the app.json plugin.
L5 · Version Correctness(1/1)
Does the code use the current react-native-auth0 v5 Expo patterns? Specifically: the Auth0Provider + useAuth0 hook API (not the legacy imperative Auth0 client class), and the react-native-auth0 Expo config plugin in app.json (not manual native iOS/Android edits)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Expo app with the react-native-auth0 SDK: Auth0Provider wrapping the app, the useAuth0 hook, authorize()/clearSession() login and logout with customScheme, the Expo config plugin registered in app.json, and user profile display guarded by isLoading?
39
Claude Sonnet 5
Android
67%6/9
100%+33%
14/14$1.66
4m 27s
With Auth0 Tools·100%·Tool Calls 24
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (14/14 passed)
L1 · Positive Presence(5/5)
Uses Auth0 Android SDK dependency
Uses WebAuthProvider for authentication
Calls WebAuthProvider.login() for sign-in
Calls WebAuthProvider.logout() for sign-out
Uses CredentialsManager for secure token storage
L2 · Hallucination Detection(1/1)
No auth0-java (server-side SDK, not for Android)
L3 · Security(2/2)
No hardcoded client ID in Kotlin source files (ok in strings.xml)
No hardcoded domain in Kotlin source files (ok in strings.xml)
L4 · Structural(3/3)
Configures manifestPlaceholders in build.gradle for Auth0 callback URL scheme
Does the code implement both onSuccess and onFailure callbacks for WebAuthProvider.login(...) and WebAuthProvider.logout(...), with AuthenticationException handled in onFailure?
Is the Auth0 callback URL scheme correctly handled? Check that the Gradle configuration defines/provides manifestPlaceholders with auth0Domain and auth0Scheme keys, regardless of whether they are set in defaultConfig, buildTypes, productFlavors, or merged at the android level. IMPORTANT: using @string/ resource references (e.g. auth0Domain: "@string/com_auth0_domain") is the CORRECT and recommended Auth0 pattern — do NOT mark this as wrong. Also check that WebAuthProvider.login(...) and WebAuthProvider.logout(...) call withScheme(). Note: SDK v2+ does NOT require a manual redirect activity in AndroidManifest.xml.
L5 · Version Correctness(2/2)
Uses Auth0(context) or Auth0.getInstance(context) for auto-configuration from string resources
Does the code use current Auth0 Android SDK v2+ patterns? Specifically: Auth0(context) for auto-configuration from strings.xml, WebAuthProvider for browser-based login/logout, and CredentialsManager for secure credential storage rather than persisting tokens manually in SharedPreferences?
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Android app with WebAuthProvider login and logout, CredentialsManager for secure token storage, Auth0 credentials configured via string resources, and manifestPlaceholders for callback URL handling? IMPORTANT correct patterns to accept: (1) manifestPlaceholders using @string/ resource references is correct and recommended; (2) Auth0(context) or Auth0.getInstance(context) auto-reads from strings.xml — no explicit client ID/domain constructor args needed; (3) SDK v2+ does NOT require a manual redirect activity in AndroidManifest.xml.
40
Gemini 3.1 Pro
Expo
73%8/11
100%+27%
17/17$1.94
3m 5s
With Auth0 Tools·100%·Tool Calls 26
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.5/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (17/17 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses the react-native-auth0 SDK
Wraps app with Auth0Provider
Uses the useAuth0 hook
Implements login via authorize()
Implements logout via clearSession()
Configures a customScheme for the callback URL
L2 · Hallucination Detection(3/3)
No web React SDK (correct package is react-native-auth0)
No browser SPA SDK in a native Expo app
Does not fall back to expo-auth-session instead of the Auth0 SDK
L3 · Security(1/1)
No hardcoded client ID in source files (ok in app.json)
L4 · Structural(4/4)
Registers the react-native-auth0 Expo config plugin with a customScheme in app.json
Auth0Provider configured with a domain prop
Does the code handle the loading state (isLoading from useAuth0) before rendering auth-dependent UI? A correct implementation should not render login/logout UI while isLoading is true.
Does the code wire login and logout correctly for react-native-auth0 on Expo? Specifically: authorize() for login and clearSession() for logout, each passing { customScheme: ... } as the second argument, matching the scheme configured in the app.json plugin.
L5 · Version Correctness(1/1)
Does the code use the current react-native-auth0 v5 Expo patterns? Specifically: the Auth0Provider + useAuth0 hook API (not the legacy imperative Auth0 client class), and the react-native-auth0 Expo config plugin in app.json (not manual native iOS/Android edits)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Expo app with the react-native-auth0 SDK: Auth0Provider wrapping the app, the useAuth0 hook, authorize()/clearSession() login and logout with customScheme, the Expo config plugin registered in app.json, and user profile display guarded by isLoading?
41
GPT-5.6 Luna
Swift
70%7/10
99%+29%
14/14$0.17
53s
With Auth0 Tools·99%·Tool Calls 10
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
10.8/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (14/14 passed)
L1 · Positive Presence(4/4)
Uses Auth0 SDK
Imports Auth0 module
Uses webAuth() for login
Uses clearSession for logout
L2 · Hallucination Detection(3/3)
No hallucinated Auth0SDK package name (correct package is Auth0)
Does not use CocoaPods (SPM preferred)
Does not use deprecated completion handler pattern
L3 · Security(2/2)
No hardcoded client ID in Swift source files (ok in Auth0.plist)
No hardcoded domain in Swift source files (ok in Auth0.plist)
L4 · Structural(2/2)
Uses CredentialsManager for token storage
Does the code properly handle login and logout flows with appropriate error handling? Does it update the UI state after successful authentication?
L5 · Version Correctness(2/2)
Uses async/await webAuth().start() syntax (not completion handlers)
Does the code use modern Swift async/await patterns with the Auth0.swift SDK? Specifically: does it use try await webAuth().start() and CredentialsManager, and configure via Auth0.plist rather than hardcoded strings?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Swift iOS app with webAuth() login/logout, credential management, and proper SwiftUI state handling?
42
GPT-5.6 Luna
FastAPI
79%11/14
99%+20%
21/21$0.18
39s
With Auth0 Tools·99%·Tool Calls 10
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
10.8/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (21/21 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses auth0-fastapi-api package
Imports Auth0FastAPI class
Uses require_auth() dependency
Uses FastAPI Depends for dependency injection
Configures domain
Configures audience
L2 · Hallucination Detection(5/5)
No python-jose (manual JWT parsing not needed)
No PyJWT direct usage
No manual JWT decoding
No fastapi-users (wrong package for Auth0 JWT)
No passlib (unrelated auth package)
L3 · Security(2/2)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
L4 · Structural(5/5)
Wrote Auth0 config to .env file
Auth0FastAPI instance is created
Route protected with Depends(auth0.require_auth())
read:messages scope checked on /api/messages route
Does the app correctly create an Auth0FastAPI instance, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token? Note: the auth0-fastapi-api distribution imports as the module `fastapi_plugin` (i.e. `from fastapi_plugin import Auth0FastAPI` is correct — there is no `auth0_fastapi_api` module).
L5 · Version Correctness(1/1)
Does the solution use current auth0-fastapi-api patterns? Specifically: does it use Auth0FastAPI with domain and audience parameters, use require_auth() as a FastAPI Depends dependency (not a decorator), and read credentials from environment variables (not hardcoded)? Note: the auth0-fastapi-api distribution imports as the module `fastapi_plugin` (`from fastapi_plugin import Auth0FastAPI` is correct — there is no `auth0_fastapi_api` module).
Holistic(1/1)
Does the solution correctly integrate Auth0 into a FastAPI API using auth0-fastapi-api? It should create an Auth0FastAPI instance configured with domain and audience from environment variables, protect the /api/messages route with read:messages scope check, and protect the /api/private route requiring a valid access token. Note: the auth0-fastapi-api distribution imports as the module `fastapi_plugin` (`from fastapi_plugin import Auth0FastAPI` is correct — there is no `auth0_fastapi_api` module).
43
GPT-5.6 Luna
Vue
92%12/13
99%+7%
20/20$0.20
1m 34s
With Auth0 Tools·99%·Tool Calls 15
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.2/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (20/20 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(7/7)
Uses @auth0/auth0-vue SDK
Sets up Auth0 plugin with createAuth0
Uses useAuth0 composable
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile information
L2 · Hallucination Detection(3/3)
Does not use React SDK in Vue app
No hallucinated @auth0/vue3-auth0 package
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Plugin installed via app.use(createAuth0(...))
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated? A correct implementation should not render auth-dependent UI while isLoading is true.
L5 · Version Correctness(3/3)
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Uses clientId (not deprecated client_id) in createAuth0 config
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Vue 3 SPA with the @auth0/auth0-vue plugin, useAuth0 composable, login, logout, user profile display, route protection, and getAccessTokenSilently to make authenticated API calls?
44
GPT-5.6 Luna
Express API
70%7/10
99%+29%
19/19$0.21
1m 7s
With Auth0 Tools·99%·Tool Calls 11
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
10.9/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (19/19 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(3/3)
Uses express-oauth2-jwt-bearer SDK
Uses requiredScopes() for scope-based route protection
Accesses JWT data via req.auth
L2 · Hallucination Detection(4/4)
No express-openid-connect (that is for web apps, not APIs)
No passport middleware (not needed with express-oauth2-jwt-bearer)
No manual JWT verification with jsonwebtoken (use SDK)
No SPA SDK used in server-side API
L3 · Security(2/2)
No hardcoded issuer domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
L4 · Structural(4/4)
Wrote Auth0 config to .env file
GET /api/messages protected with read:messages scope
POST /api/messages protected with write:messages scope
Does the API correctly protect all three routes? GET /api/messages requires read:messages, POST /api/messages requires write:messages, GET /api/profile returns user info from req.auth.payload. Missing or invalid tokens should yield 401, insufficient scope should yield 403.
L5 · Version Correctness(4/4)
No manual Authorization header parsing (SDK handles extraction)
No manual jwt.verify calls (SDK handles verification)
No req.user (express-oauth2-jwt-bearer uses req.auth, not req.user)
Does the solution use current express-oauth2-jwt-bearer patterns? Specifically: does it apply the auth() middleware to protect routes, use requiredScopes() for scope checks (not manual payload inspection), and access token data via req.auth.payload (not req.user or manually decoded tokens)? Judge only from the source code; the issuer and audience may be supplied via environment variables (ISSUER_BASE_URL / AUDIENCE), so do not assume the contents of any .env file.
Holistic(1/1)
Does the solution correctly protect an Express.js API using express-oauth2-jwt-bearer? It should apply the auth() middleware (issuer and audience may come from ISSUER_BASE_URL / AUDIENCE environment variables — judge only from the source code and do not assume the contents of any .env file), protect GET /api/messages with read:messages scope, protect POST /api/messages with write:messages scope, and return user profile info from req.auth.payload at GET /api/profile.
45
GPT-5.6 Luna
Android
56%5/9
99%+43%
14/14$0.22
1m 6s
With Auth0 Tools·99%·Tool Calls 11
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
10.9/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (14/14 passed)
L1 · Positive Presence(5/5)
Uses Auth0 Android SDK dependency
Uses WebAuthProvider for authentication
Calls WebAuthProvider.login() for sign-in
Calls WebAuthProvider.logout() for sign-out
Uses CredentialsManager for secure token storage
L2 · Hallucination Detection(1/1)
No auth0-java (server-side SDK, not for Android)
L3 · Security(2/2)
No hardcoded client ID in Kotlin source files (ok in strings.xml)
No hardcoded domain in Kotlin source files (ok in strings.xml)
L4 · Structural(3/3)
Configures manifestPlaceholders in build.gradle for Auth0 callback URL scheme
Does the code implement both onSuccess and onFailure callbacks for WebAuthProvider.login(...) and WebAuthProvider.logout(...), with AuthenticationException handled in onFailure?
Is the Auth0 callback URL scheme correctly handled? Check that the Gradle configuration defines/provides manifestPlaceholders with auth0Domain and auth0Scheme keys, regardless of whether they are set in defaultConfig, buildTypes, productFlavors, or merged at the android level. IMPORTANT: using @string/ resource references (e.g. auth0Domain: "@string/com_auth0_domain") is the CORRECT and recommended Auth0 pattern — do NOT mark this as wrong. Also check that WebAuthProvider.login(...) and WebAuthProvider.logout(...) call withScheme(). Note: SDK v2+ does NOT require a manual redirect activity in AndroidManifest.xml.
L5 · Version Correctness(2/2)
Uses Auth0(context) or Auth0.getInstance(context) for auto-configuration from string resources
Does the code use current Auth0 Android SDK v2+ patterns? Specifically: Auth0(context) for auto-configuration from strings.xml, WebAuthProvider for browser-based login/logout, and CredentialsManager for secure credential storage rather than persisting tokens manually in SharedPreferences?
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Android app with WebAuthProvider login and logout, CredentialsManager for secure token storage, Auth0 credentials configured via string resources, and manifestPlaceholders for callback URL handling? IMPORTANT correct patterns to accept: (1) manifestPlaceholders using @string/ resource references is correct and recommended; (2) Auth0(context) or Auth0.getInstance(context) auto-reads from strings.xml — no explicit client ID/domain constructor args needed; (3) SDK v2+ does NOT require a manual redirect activity in AndroidManifest.xml.
46
GPT-5.6 Luna
SPA JS
93%13/14
99%+6%
21/21$0.23
1m 43s
With Auth0 Tools·99%·Tool Calls 17
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
10.6/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (21/21 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(7/7)
Uses @auth0/auth0-spa-js SDK
Initializes Auth0 with createAuth0Client
Implements loginWithRedirect
Handles redirect callback on page load
Implements logout
Checks isAuthenticated for conditional rendering
Uses getTokenSilently to retrieve access token
L2 · Hallucination Detection(4/4)
Does not use React SDK in vanilla JS app
Does not use Vue SDK in vanilla JS app
Does not use Next.js SDK in vanilla JS app
No client_secret in SPA (public client)
L3 · Security(2/2)
No tokens manually stored in localStorage
No tokens manually stored in sessionStorage
L4 · Structural(3/3)
Auth0Client configured with domain
Authenticated API request uses Bearer token in Authorization header
Does the code check isAuthenticated to show/hide UI elements and only render user profile information when the user is authenticated?
L5 · Version Correctness(3/3)
Uses authorizationParams (not deprecated top-level audience/redirect_uri)
authorizationParams contains audience 'https://api.playground.com'
No deprecated promise-chain pattern for getTokenSilently
Holistic(1/1)
Does the solution correctly integrate Auth0 into a vanilla JavaScript SPA using @auth0/auth0-spa-js with createAuth0Client, loginWithRedirect, handleRedirectCallback, logout, user profile display, and getTokenSilently to make authenticated API calls?
47
GPT-5.6 Luna
Nuxt
55%11/20
99%+44%
32/32$0.31
2m 29s
With Auth0 Tools·99%·Tool Calls 21
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.4/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (32/32 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(10/10)
Uses @auth0/auth0-nuxt SDK
Registers module in nuxt.config
Configures runtimeConfig with Auth0 settings
Configures NUXT_AUTH0_DOMAIN env var
Configures NUXT_AUTH0_CLIENT_ID env var
Configures NUXT_AUTH0_CLIENT_SECRET env var
Configures NUXT_AUTH0_SESSION_SECRET env var
Uses useUser() composable to access the authenticated user
Uses /auth/login route to initiate login
Uses /auth/logout route to log out
L2 · Hallucination Detection(4/4)
Does not use the Vue SDK in a Nuxt app
Does not use the React SDK in a Nuxt app
Does not use loginWithRedirect (SPA API not applicable to Nuxt)
Does not use getAccessTokenSilently (Vue/React SPA API, not used in Nuxt)
L3 · Security(5/5)
No hardcoded client ID in source files (ok in .env)
No hardcoded client secret in source files (ok in .env)
No hardcoded Auth0 domain in source files (ok in .env)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(7/7)
Wrote Auth0 credentials to .env file
Module correctly registered in the modules array
sessionSecret configured for session encryption
clientSecret configured (Regular Web Application)
appBaseUrl configured in runtimeConfig
definePageMeta used for page-level middleware
Is there a protected /profile route that uses a Nuxt route middleware (via definePageMeta) to check authentication with useUser() and redirect unauthenticated users to /auth/login?
L5 · Version Correctness(4/4)
Audience configured in runtimeConfig for API access
Audience set to 'https://api.playground.com'
Uses getAccessToken() server-side to retrieve access token
Uses useAuth0(event) server-side composable
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Nuxt application using the @auth0/auth0-nuxt module, with proper configuration in nuxt.config.ts (domain, clientId, clientSecret, sessionSecret, appBaseUrl, audience), login/logout via /auth/login and /auth/logout, user profile display with useUser(), a /profile route protected by a route middleware, and server-side getAccessToken() for authenticated API calls?
48
Claude Haiku 4.5
Android
67%6/9
99%+32%
14/14$0.32
2m 26s
With Auth0 Tools·99%·Tool Calls 38
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.1/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (14/14 passed)
L1 · Positive Presence(5/5)
Uses Auth0 Android SDK dependency
Uses WebAuthProvider for authentication
Calls WebAuthProvider.login() for sign-in
Calls WebAuthProvider.logout() for sign-out
Uses CredentialsManager for secure token storage
L2 · Hallucination Detection(1/1)
No auth0-java (server-side SDK, not for Android)
L3 · Security(2/2)
No hardcoded client ID in Kotlin source files (ok in strings.xml)
No hardcoded domain in Kotlin source files (ok in strings.xml)
L4 · Structural(3/3)
Configures manifestPlaceholders in build.gradle for Auth0 callback URL scheme
Does the code implement both onSuccess and onFailure callbacks for WebAuthProvider.login(...) and WebAuthProvider.logout(...), with AuthenticationException handled in onFailure?
Is the Auth0 callback URL scheme correctly handled? Check that the Gradle configuration defines/provides manifestPlaceholders with auth0Domain and auth0Scheme keys, regardless of whether they are set in defaultConfig, buildTypes, productFlavors, or merged at the android level. IMPORTANT: using @string/ resource references (e.g. auth0Domain: "@string/com_auth0_domain") is the CORRECT and recommended Auth0 pattern — do NOT mark this as wrong. Also check that WebAuthProvider.login(...) and WebAuthProvider.logout(...) call withScheme(). Note: SDK v2+ does NOT require a manual redirect activity in AndroidManifest.xml.
L5 · Version Correctness(2/2)
Uses Auth0(context) or Auth0.getInstance(context) for auto-configuration from string resources
Does the code use current Auth0 Android SDK v2+ patterns? Specifically: Auth0(context) for auto-configuration from strings.xml, WebAuthProvider for browser-based login/logout, and CredentialsManager for secure credential storage rather than persisting tokens manually in SharedPreferences?
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Android app with WebAuthProvider login and logout, CredentialsManager for secure token storage, Auth0 credentials configured via string resources, and manifestPlaceholders for callback URL handling? IMPORTANT correct patterns to accept: (1) manifestPlaceholders using @string/ resource references is correct and recommended; (2) Auth0(context) or Auth0.getInstance(context) auto-reads from strings.xml — no explicit client ID/domain constructor args needed; (3) SDK v2+ does NOT require a manual redirect activity in AndroidManifest.xml.
49
Claude Sonnet 5
Express
80%12/15
99%+19%
23/24$0.40
2m 5s
With Auth0 Tools·99%·Tool Calls 13
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
23.5/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (23/24 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(7/7)
Uses express-openid-connect SDK
Configures issuerBaseURL
Configures clientID
Configures baseURL
Configures session secret
Uses req.oidc to access OIDC context
Accesses user profile via req.oidc.user
L2 · Hallucination Detection(4/4)
No hallucinated @auth0/auth0-spa-js (wrong SDK for server-side app)
No express-oauth2-jwt-bearer (that is for APIs, not web apps)
Does not use Next.js SDK in Express app
No passport middleware (not needed with express-openid-connect)
L3 · Security(3/3)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
No hardcoded Auth0 domain in source files (ok in .env)
L4 · Structural(5/5)
Wrote Auth0 credentials to .env file
auth middleware registered with app.use
Uses requiresAuth() to protect the /profile route
Accesses access token via req.oidc.accessToken
Does the app correctly register the auth() middleware, protect the /profile route with requiresAuth(), display user profile information, and include a route that calls an external API using the access token?
L5 · Version Correctness(3/3)
Uses authorizationParams to pass audience and scope
authorizationParams contains audience 'https://api.barkbook.com'
Does the solution use current express-openid-connect patterns? Specifically: does it use issuerBaseURL (not AUTH0_DOMAIN or domain directly), configure audience via authorizationParams (not as a top-level config key), and use response_type: "code" for the authorization code flow?
Holistic(0/1)
Does the solution correctly integrate Auth0 into an Express web app using express-openid-connect? It should configure the auth() middleware, protect the /profile route with requiresAuth(), display the logged-in user profile, and use the access token to call an external API with audience https://api.barkbook.com.
50
GPT-5.6 Terra
Fastify API
54%7/13
99%+45%
20/20$0.41
35s
With Auth0 Tools·99%·Tool Calls 10
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
10.8/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (20/20 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses @auth0/auth0-fastify-api package
Imports fastifyAuth0Api plugin
Uses fastify.requireAuth() to protect routes
Uses preHandler to attach auth middleware
Configures domain
Configures audience
L2 · Hallucination Detection(4/4)
No express-oauth2-jwt-bearer (wrong SDK for Fastify)
No passport middleware (not needed with @auth0/auth0-fastify-api)
No jsonwebtoken (manual JWT verification not needed)
No manual jwt.verify() calls
L3 · Security(2/2)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
L4 · Structural(5/5)
Wrote Auth0 config to .env file
Auth0 API plugin registered with fastify.register()
Route protected with preHandler: fastify.requireAuth()
read:messages scope checked on /api/messages route
Does the app correctly register the @auth0/auth0-fastify-api plugin, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token? Note: @auth0/auth0-fastify-api exposes the verified token claims on request.user (e.g. request.user.sub) — this is the correct API for this plugin, not request.auth.
L5 · Version Correctness(1/1)
Does the solution use current @auth0/auth0-fastify-api patterns? Specifically: does it register the plugin via fastify.register(), use preHandler: fastify.requireAuth() for route protection (not a decorator), access token claims via request.user, and read credentials from environment variables (not hardcoded)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Fastify API using @auth0/auth0-fastify-api? It should register the plugin with domain and audience from environment variables, protect the /api/messages route using fastify.requireAuth() with read:messages scope (as string or array) check using preHandler, and protect the /api/private route using fastify.requireAuth() requiring any valid access token.
51
GPT-5.6 Luna
Flask
36%5/14
99%+63%
26/26$0.43
2m 20s
With Auth0 Tools·99%·Tool Calls 17
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
10.6/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (26/26 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses auth0-server-python package
Imports ServerClient class
Uses start_interactive_login method
Uses complete_interactive_login method
Uses get_user method to retrieve user info
Loads environment variables with python-dotenv
L2 · Hallucination Detection(4/4)
No Authlib (old approach, not the current SDK)
No python-jose (manual JWT parsing not needed)
No Flask-Login (not needed with auth0-server-python)
No Flask-Dance (wrong OAuth package)
L3 · Security(3/3)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
No hardcoded Auth0 domain in source files (ok in .env)
L4 · Structural(8/8)
Wrote Auth0 credentials to .env file
Implements /callback route
Implements /login route
Implements /logout route
Implements /profile route
Sets Flask app.secret_key for session management
Uses async route handlers
Does the app implement login, callback, profile, and logout routes using auth0-server-python, protect the /profile route so it requires login, and include a route that calls an external API using an access token?
L5 · Version Correctness(3/3)
Uses authorization_params for scope and audience configuration
authorization_params contains audience 'https://api.barkbook.com'
Does the solution use current auth0-server-python patterns? Specifically: does it import ServerClient from auth0_server_python, define concrete state and transaction store classes locally (subclassing StateStore/TransactionStore from the SDK), and configure authorization_params with audience and scope?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Flask web app using auth0-server-python? It should configure a ServerClient with credentials from env vars, implement login, callback, profile, and logout routes, protect the /profile route so it requires authentication, and use the access token to call an external API with audience https://api.barkbook.com.
52
GPT-5.6 Terra
SPA JS
93%13/14
99%+6%
21/21$0.46
1m 58s
With Auth0 Tools·99%·Tool Calls 9
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
10.7/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (21/21 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(7/7)
Uses @auth0/auth0-spa-js SDK
Initializes Auth0 with createAuth0Client
Implements loginWithRedirect
Handles redirect callback on page load
Implements logout
Checks isAuthenticated for conditional rendering
Uses getTokenSilently to retrieve access token
L2 · Hallucination Detection(4/4)
Does not use React SDK in vanilla JS app
Does not use Vue SDK in vanilla JS app
Does not use Next.js SDK in vanilla JS app
No client_secret in SPA (public client)
L3 · Security(2/2)
No tokens manually stored in localStorage
No tokens manually stored in sessionStorage
L4 · Structural(3/3)
Auth0Client configured with domain
Authenticated API request uses Bearer token in Authorization header
Does the code check isAuthenticated to show/hide UI elements and only render user profile information when the user is authenticated?
L5 · Version Correctness(3/3)
Uses authorizationParams (not deprecated top-level audience/redirect_uri)
authorizationParams contains audience 'https://api.playground.com'
No deprecated promise-chain pattern for getTokenSilently
Holistic(1/1)
Does the solution correctly integrate Auth0 into a vanilla JavaScript SPA using @auth0/auth0-spa-js with createAuth0Client, loginWithRedirect, handleRedirectCallback, logout, user profile display, and getTokenSilently to make authenticated API calls?
53
GPT-5.6 Terra
Express
80%12/15
99%+19%
24/24$0.49
1m 1s
With Auth0 Tools·99%·Tool Calls 9
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
10.7/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (24/24 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(7/7)
Uses express-openid-connect SDK
Configures issuerBaseURL
Configures clientID
Configures baseURL
Configures session secret
Uses req.oidc to access OIDC context
Accesses user profile via req.oidc.user
L2 · Hallucination Detection(4/4)
No hallucinated @auth0/auth0-spa-js (wrong SDK for server-side app)
No express-oauth2-jwt-bearer (that is for APIs, not web apps)
Does not use Next.js SDK in Express app
No passport middleware (not needed with express-openid-connect)
L3 · Security(3/3)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
No hardcoded Auth0 domain in source files (ok in .env)
L4 · Structural(5/5)
Wrote Auth0 credentials to .env file
auth middleware registered with app.use
Uses requiresAuth() to protect the /profile route
Accesses access token via req.oidc.accessToken
Does the app correctly register the auth() middleware, protect the /profile route with requiresAuth(), display user profile information, and include a route that calls an external API using the access token?
L5 · Version Correctness(3/3)
Uses authorizationParams to pass audience and scope
authorizationParams contains audience 'https://api.barkbook.com'
Does the solution use current express-openid-connect patterns? Specifically: does it use issuerBaseURL (not AUTH0_DOMAIN or domain directly), configure audience via authorizationParams (not as a top-level config key), and use response_type: "code" for the authorization code flow?
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Express web app using express-openid-connect? It should configure the auth() middleware, protect the /profile route with requiresAuth(), display the logged-in user profile, and use the access token to call an external API with audience https://api.barkbook.com.
54
Claude Sonnet 5
Flask
43%6/14
99%+56%
25/26$0.58
2m 57s
With Auth0 Tools·99%·Tool Calls 17
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
23.7/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (25/26 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses auth0-server-python package
Imports ServerClient class
Uses start_interactive_login method
Uses complete_interactive_login method
Uses get_user method to retrieve user info
Loads environment variables with python-dotenv
L2 · Hallucination Detection(4/4)
No Authlib (old approach, not the current SDK)
No python-jose (manual JWT parsing not needed)
No Flask-Login (not needed with auth0-server-python)
No Flask-Dance (wrong OAuth package)
L3 · Security(3/3)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
No hardcoded Auth0 domain in source files (ok in .env)
L4 · Structural(8/8)
Wrote Auth0 credentials to .env file
Implements /callback route
Implements /login route
Implements /logout route
Implements /profile route
Sets Flask app.secret_key for session management
Uses async route handlers
Does the app implement login, callback, profile, and logout routes using auth0-server-python, protect the /profile route so it requires login, and include a route that calls an external API using an access token?
L5 · Version Correctness(3/3)
Uses authorization_params for scope and audience configuration
authorization_params contains audience 'https://api.barkbook.com'
Does the solution use current auth0-server-python patterns? Specifically: does it import ServerClient from auth0_server_python, define concrete state and transaction store classes locally (subclassing StateStore/TransactionStore from the SDK), and configure authorization_params with audience and scope?
Holistic(0/1)
Does the solution correctly integrate Auth0 into a Flask web app using auth0-server-python? It should configure a ServerClient with credentials from env vars, implement login, callback, profile, and logout routes, protect the /profile route so it requires authentication, and use the access token to call an external API with audience https://api.barkbook.com.
55
GPT-5.6 Terra
Flask
36%5/14
99%+63%
25/26$0.63
1m 17s
With Auth0 Tools·99%·Tool Calls 12
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
23.7/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (25/26 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses auth0-server-python package
Imports ServerClient class
Uses start_interactive_login method
Uses complete_interactive_login method
Uses get_user method to retrieve user info
Loads environment variables with python-dotenv
L2 · Hallucination Detection(4/4)
No Authlib (old approach, not the current SDK)
No python-jose (manual JWT parsing not needed)
No Flask-Login (not needed with auth0-server-python)
No Flask-Dance (wrong OAuth package)
L3 · Security(3/3)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
No hardcoded Auth0 domain in source files (ok in .env)
L4 · Structural(8/8)
Wrote Auth0 credentials to .env file
Implements /callback route
Implements /login route
Implements /logout route
Implements /profile route
Sets Flask app.secret_key for session management
Uses async route handlers
Does the app implement login, callback, profile, and logout routes using auth0-server-python, protect the /profile route so it requires login, and include a route that calls an external API using an access token?
L5 · Version Correctness(3/3)
Uses authorization_params for scope and audience configuration
authorization_params contains audience 'https://api.barkbook.com'
Does the solution use current auth0-server-python patterns? Specifically: does it import ServerClient from auth0_server_python, define concrete state and transaction store classes locally (subclassing StateStore/TransactionStore from the SDK), and configure authorization_params with audience and scope?
Holistic(0/1)
Does the solution correctly integrate Auth0 into a Flask web app using auth0-server-python? It should configure a ServerClient with credentials from env vars, implement login, callback, profile, and logout routes, protect the /profile route so it requires authentication, and use the access token to call an external API with audience https://api.barkbook.com.
56
Gemini 3.1 Pro
Vue
92%12/13
99%+7%
20/20$0.64
1m 11s
With Auth0 Tools·99%·Tool Calls 17
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.3/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (20/20 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(7/7)
Uses @auth0/auth0-vue SDK
Sets up Auth0 plugin with createAuth0
Uses useAuth0 composable
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile information
L2 · Hallucination Detection(3/3)
Does not use React SDK in Vue app
No hallucinated @auth0/vue3-auth0 package
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Plugin installed via app.use(createAuth0(...))
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated? A correct implementation should not render auth-dependent UI while isLoading is true.
L5 · Version Correctness(3/3)
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Uses clientId (not deprecated client_id) in createAuth0 config
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Vue 3 SPA with the @auth0/auth0-vue plugin, useAuth0 composable, login, logout, user profile display, route protection, and getAccessTokenSilently to make authenticated API calls?
57
Gemini 3.5 Flash
SPA JS
86%12/14
99%+13%
21/21$0.71
1m 11s
With Auth0 Tools·99%·Tool Calls 18
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.3/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (21/21 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(7/7)
Uses @auth0/auth0-spa-js SDK
Initializes Auth0 with createAuth0Client
Implements loginWithRedirect
Handles redirect callback on page load
Implements logout
Checks isAuthenticated for conditional rendering
Uses getTokenSilently to retrieve access token
L2 · Hallucination Detection(4/4)
Does not use React SDK in vanilla JS app
Does not use Vue SDK in vanilla JS app
Does not use Next.js SDK in vanilla JS app
No client_secret in SPA (public client)
L3 · Security(2/2)
No tokens manually stored in localStorage
No tokens manually stored in sessionStorage
L4 · Structural(3/3)
Auth0Client configured with domain
Authenticated API request uses Bearer token in Authorization header
Does the code check isAuthenticated to show/hide UI elements and only render user profile information when the user is authenticated?
L5 · Version Correctness(3/3)
Uses authorizationParams (not deprecated top-level audience/redirect_uri)
authorizationParams contains audience 'https://api.playground.com'
No deprecated promise-chain pattern for getTokenSilently
Holistic(1/1)
Does the solution correctly integrate Auth0 into a vanilla JavaScript SPA using @auth0/auth0-spa-js with createAuth0Client, loginWithRedirect, handleRedirectCallback, logout, user profile display, and getTokenSilently to make authenticated API calls?
58
Gemini 3.5 Flash
React
92%12/13
99%+7%
20/20$0.98
1m 57s
With Auth0 Tools·99%·Tool Calls 17
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.3/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (20/20 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(7/7)
Uses @auth0/auth0-react SDK
Wraps app with Auth0Provider
Uses useAuth0 hook
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile name
L2 · Hallucination Detection(3/3)
No hallucinated @auth0/react package (must be @auth0/auth0-react)
Does not use server SDK in SPA app
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Auth0Provider configured with domain prop
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated? A correct implementation should not render auth-dependent UI while isLoading is true.
L5 · Version Correctness(3/3)
Uses authorizationParams (not audience directly on provider)
authorizationParams contains audience 'https://api.barkbook.com'
Does the code use the current @auth0/auth0-react SDK patterns? Specifically: does it use isLoading (not the deprecated "loading" property), and pass audience/scope via authorizationParams object (not as direct props)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a React SPA with Auth0Provider, useAuth0 hook, login, logout, user profile display, and getAccessTokenSilently to make authenticated API calls?
59
GPT-5.6 Sol
SPA JS
93%13/14
99%+6%
21/21$1.04
1m 9s
With Auth0 Tools·99%·Tool Calls 11
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
10.9/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (21/21 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(7/7)
Uses @auth0/auth0-spa-js SDK
Initializes Auth0 with createAuth0Client
Implements loginWithRedirect
Handles redirect callback on page load
Implements logout
Checks isAuthenticated for conditional rendering
Uses getTokenSilently to retrieve access token
L2 · Hallucination Detection(4/4)
Does not use React SDK in vanilla JS app
Does not use Vue SDK in vanilla JS app
Does not use Next.js SDK in vanilla JS app
No client_secret in SPA (public client)
L3 · Security(2/2)
No tokens manually stored in localStorage
No tokens manually stored in sessionStorage
L4 · Structural(3/3)
Auth0Client configured with domain
Authenticated API request uses Bearer token in Authorization header
Does the code check isAuthenticated to show/hide UI elements and only render user profile information when the user is authenticated?
L5 · Version Correctness(3/3)
Uses authorizationParams (not deprecated top-level audience/redirect_uri)
authorizationParams contains audience 'https://api.playground.com'
No deprecated promise-chain pattern for getTokenSilently
Holistic(1/1)
Does the solution correctly integrate Auth0 into a vanilla JavaScript SPA using @auth0/auth0-spa-js with createAuth0Client, loginWithRedirect, handleRedirectCallback, logout, user profile display, and getTokenSilently to make authenticated API calls?
60
Gemini 3.1 Pro
Flask
36%5/14
99%+63%
25/26$1.12
1m 44s
With Auth0 Tools·99%·Tool Calls 18
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
23.7/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (25/26 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses auth0-server-python package
Imports ServerClient class
Uses start_interactive_login method
Uses complete_interactive_login method
Uses get_user method to retrieve user info
Loads environment variables with python-dotenv
L2 · Hallucination Detection(4/4)
No Authlib (old approach, not the current SDK)
No python-jose (manual JWT parsing not needed)
No Flask-Login (not needed with auth0-server-python)
No Flask-Dance (wrong OAuth package)
L3 · Security(3/3)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
No hardcoded Auth0 domain in source files (ok in .env)
L4 · Structural(7/8)
Wrote Auth0 credentials to .env file
Implements /callback route
Implements /login route
Implements /logout route
Implements /profile route
Sets Flask app.secret_key for session management
Uses async route handlers
Does the app implement login, callback, profile, and logout routes using auth0-server-python, protect the /profile route so it requires login, and include a route that calls an external API using an access token?
L5 · Version Correctness(3/3)
Uses authorization_params for scope and audience configuration
authorization_params contains audience 'https://api.barkbook.com'
Does the solution use current auth0-server-python patterns? Specifically: does it import ServerClient from auth0_server_python, define concrete state and transaction store classes locally (subclassing StateStore/TransactionStore from the SDK), and configure authorization_params with audience and scope?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Flask web app using auth0-server-python? It should configure a ServerClient with credentials from env vars, implement login, callback, profile, and logout routes, protect the /profile route so it requires authentication, and use the access token to call an external API with audience https://api.barkbook.com.
61
GPT-5.6 Terra
Android
67%6/9
99%+32%
14/14$1.21
1m 59s
With Auth0 Tools·99%·Tool Calls 17
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
10.6/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (14/14 passed)
L1 · Positive Presence(5/5)
Uses Auth0 Android SDK dependency
Uses WebAuthProvider for authentication
Calls WebAuthProvider.login() for sign-in
Calls WebAuthProvider.logout() for sign-out
Uses CredentialsManager for secure token storage
L2 · Hallucination Detection(1/1)
No auth0-java (server-side SDK, not for Android)
L3 · Security(2/2)
No hardcoded client ID in Kotlin source files (ok in strings.xml)
No hardcoded domain in Kotlin source files (ok in strings.xml)
L4 · Structural(3/3)
Configures manifestPlaceholders in build.gradle for Auth0 callback URL scheme
Does the code implement both onSuccess and onFailure callbacks for WebAuthProvider.login(...) and WebAuthProvider.logout(...), with AuthenticationException handled in onFailure?
Is the Auth0 callback URL scheme correctly handled? Check that the Gradle configuration defines/provides manifestPlaceholders with auth0Domain and auth0Scheme keys, regardless of whether they are set in defaultConfig, buildTypes, productFlavors, or merged at the android level. IMPORTANT: using @string/ resource references (e.g. auth0Domain: "@string/com_auth0_domain") is the CORRECT and recommended Auth0 pattern — do NOT mark this as wrong. Also check that WebAuthProvider.login(...) and WebAuthProvider.logout(...) call withScheme(). Note: SDK v2+ does NOT require a manual redirect activity in AndroidManifest.xml.
L5 · Version Correctness(2/2)
Uses Auth0(context) or Auth0.getInstance(context) for auto-configuration from string resources
Does the code use current Auth0 Android SDK v2+ patterns? Specifically: Auth0(context) for auto-configuration from strings.xml, WebAuthProvider for browser-based login/logout, and CredentialsManager for secure credential storage rather than persisting tokens manually in SharedPreferences?
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Android app with WebAuthProvider login and logout, CredentialsManager for secure token storage, Auth0 credentials configured via string resources, and manifestPlaceholders for callback URL handling? IMPORTANT correct patterns to accept: (1) manifestPlaceholders using @string/ resource references is correct and recommended; (2) Auth0(context) or Auth0.getInstance(context) auto-reads from strings.xml — no explicit client ID/domain constructor args needed; (3) SDK v2+ does NOT require a manual redirect activity in AndroidManifest.xml.
62
GPT-5.6 Sol
Vue
92%12/13
99%+7%
20/20$1.38
2m 12s
With Auth0 Tools·99%·Tool Calls 17
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.3/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (20/20 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(7/7)
Uses @auth0/auth0-vue SDK
Sets up Auth0 plugin with createAuth0
Uses useAuth0 composable
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile information
L2 · Hallucination Detection(3/3)
Does not use React SDK in Vue app
No hallucinated @auth0/vue3-auth0 package
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Plugin installed via app.use(createAuth0(...))
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated? A correct implementation should not render auth-dependent UI while isLoading is true.
L5 · Version Correctness(3/3)
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Uses clientId (not deprecated client_id) in createAuth0 config
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Vue 3 SPA with the @auth0/auth0-vue plugin, useAuth0 composable, login, logout, user profile display, route protection, and getAccessTokenSilently to make authenticated API calls?
63
GPT-5.6 Sol
Angular
92%11/12
99%+7%
18/18$1.42
1m 58s
With Auth0 Tools·99%·Tool Calls 16
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.3/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (18/18 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses @auth0/auth0-angular SDK
Injects AuthService for authentication operations
Implements loginWithRedirect
Implements logout
Uses isAuthenticated$ observable for auth state
Uses user$ observable to display user profile
L2 · Hallucination Detection(3/3)
Does not use React SDK in Angular app
Does not use Vue SDK in Angular app
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Auth0 configured via provideAuth0()
Route protected with AuthGuard or authGuardFn
Uses getAccessTokenSilently or httpInterceptor for authenticated API calls
L5 · Version Correctness(2/2)
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Angular SPA using @auth0/auth0-angular, with provideAuth0 setup, AuthService for login/logout, user profile display, route protection via a guard, and authenticated API calls (either by configuring authHttpInterceptorFn with an allowedList matching the API base URL, or by calling getAccessTokenSilently to retrieve tokens manually)?
64
GPT-5.6 Sol
FastAPI
79%11/14
99%+20%
21/21$1.46
1m 33s
With Auth0 Tools·99%·Tool Calls 13
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.1/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (21/21 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses auth0-fastapi-api package
Imports Auth0FastAPI class
Uses require_auth() dependency
Uses FastAPI Depends for dependency injection
Configures domain
Configures audience
L2 · Hallucination Detection(5/5)
No python-jose (manual JWT parsing not needed)
No PyJWT direct usage
No manual JWT decoding
No fastapi-users (wrong package for Auth0 JWT)
No passlib (unrelated auth package)
L3 · Security(2/2)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
L4 · Structural(5/5)
Wrote Auth0 config to .env file
Auth0FastAPI instance is created
Route protected with Depends(auth0.require_auth())
read:messages scope checked on /api/messages route
Does the app correctly create an Auth0FastAPI instance, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token? Note: the auth0-fastapi-api distribution imports as the module `fastapi_plugin` (i.e. `from fastapi_plugin import Auth0FastAPI` is correct — there is no `auth0_fastapi_api` module).
L5 · Version Correctness(1/1)
Does the solution use current auth0-fastapi-api patterns? Specifically: does it use Auth0FastAPI with domain and audience parameters, use require_auth() as a FastAPI Depends dependency (not a decorator), and read credentials from environment variables (not hardcoded)? Note: the auth0-fastapi-api distribution imports as the module `fastapi_plugin` (`from fastapi_plugin import Auth0FastAPI` is correct — there is no `auth0_fastapi_api` module).
Holistic(1/1)
Does the solution correctly integrate Auth0 into a FastAPI API using auth0-fastapi-api? It should create an Auth0FastAPI instance configured with domain and audience from environment variables, protect the /api/messages route with read:messages scope check, and protect the /api/private route requiring a valid access token. Note: the auth0-fastapi-api distribution imports as the module `fastapi_plugin` (`from fastapi_plugin import Auth0FastAPI` is correct — there is no `auth0_fastapi_api` module).
65
Gemini 3.5 Flash
Expo
82%9/11
99%+17%
17/17$1.48
2m 14s
With Auth0 Tools·99%·Tool Calls 24
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
10.5/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (17/17 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses the react-native-auth0 SDK
Wraps app with Auth0Provider
Uses the useAuth0 hook
Implements login via authorize()
Implements logout via clearSession()
Configures a customScheme for the callback URL
L2 · Hallucination Detection(3/3)
No web React SDK (correct package is react-native-auth0)
No browser SPA SDK in a native Expo app
Does not fall back to expo-auth-session instead of the Auth0 SDK
L3 · Security(1/1)
No hardcoded client ID in source files (ok in app.json)
L4 · Structural(4/4)
Registers the react-native-auth0 Expo config plugin with a customScheme in app.json
Auth0Provider configured with a domain prop
Does the code handle the loading state (isLoading from useAuth0) before rendering auth-dependent UI? A correct implementation should not render login/logout UI while isLoading is true.
Does the code wire login and logout correctly for react-native-auth0 on Expo? Specifically: authorize() for login and clearSession() for logout, each passing { customScheme: ... } as the second argument, matching the scheme configured in the app.json plugin.
L5 · Version Correctness(1/1)
Does the code use the current react-native-auth0 v5 Expo patterns? Specifically: the Auth0Provider + useAuth0 hook API (not the legacy imperative Auth0 client class), and the react-native-auth0 Expo config plugin in app.json (not manual native iOS/Android edits)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Expo app with the react-native-auth0 SDK: Auth0Provider wrapping the app, the useAuth0 hook, authorize()/clearSession() login and logout with customScheme, the Expo config plugin registered in app.json, and user profile display guarded by isLoading?
66
GPT-5.6 Sol
React
92%12/13
99%+7%
20/20$1.53
1m 47s
With Auth0 Tools·99%·Tool Calls 15
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.2/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (20/20 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(7/7)
Uses @auth0/auth0-react SDK
Wraps app with Auth0Provider
Uses useAuth0 hook
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile name
L2 · Hallucination Detection(3/3)
No hallucinated @auth0/react package (must be @auth0/auth0-react)
Does not use server SDK in SPA app
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Auth0Provider configured with domain prop
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated? A correct implementation should not render auth-dependent UI while isLoading is true.
L5 · Version Correctness(3/3)
Uses authorizationParams (not audience directly on provider)
authorizationParams contains audience 'https://api.barkbook.com'
Does the code use the current @auth0/auth0-react SDK patterns? Specifically: does it use isLoading (not the deprecated "loading" property), and pass audience/scope via authorizationParams object (not as direct props)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a React SPA with Auth0Provider, useAuth0 hook, login, logout, user profile display, and getAccessTokenSilently to make authenticated API calls?
67
GPT-5.6 Sol
React Native
85%11/13
99%+14%
21/21$1.57
2m 18s
With Auth0 Tools·99%·Tool Calls 17
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
10.6/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (21/21 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(5/5)
Uses the react-native-auth0 SDK
Wraps app with Auth0Provider
Uses the useAuth0 hook
Implements login via authorize()
Implements logout via clearSession()
L2 · Hallucination Detection(5/5)
No web React SDK (correct package is react-native-auth0)
No browser SPA SDK in a native React Native app
Does not use Expo AuthSession in a bare React Native app
Does not fall back to the third-party react-native-app-auth library
Does not register the Expo config plugin in a bare React Native app
L3 · Security(2/2)
No hardcoded client ID in source files (ok in config)
Does not store tokens in insecure AsyncStorage
L4 · Structural(6/6)
Auth0Provider configured with a domain prop
android/app/build.gradle sets the auth0Domain manifestPlaceholder to the real tenant domain
android/app/build.gradle sets the auth0Scheme manifestPlaceholder to <applicationId>.auth0
iOS Info.plist registers the bundle-identifier callback URL scheme
Does the code handle the loading state (isLoading from useAuth0) before rendering auth-dependent UI? A correct implementation should not render login/logout UI while isLoading is true.
Are BOTH native platforms configured for the Auth0 callback in this bare React Native app? Android needs auth0Domain and auth0Scheme manifestPlaceholders in android/app/build.gradle (the RedirectActivity is merged in by the SDK manifest, so declaring it manually is also acceptable), and iOS needs a CFBundleURLTypes entry in ios/scaffold/Info.plist. Answer no if only one platform was configured.
L5 · Version Correctness(1/1)
Does the code use the current react-native-auth0 v5 patterns? Specifically: the Auth0Provider + useAuth0 hook API rather than the legacy imperative `new Auth0({domain, clientId})` client class, and `user` from the hook rather than a manual getCredentials()/userInfo() call to read the profile.
Holistic(1/1)
Does the solution correctly integrate Auth0 into a bare (non-Expo) React Native app using the react-native-auth0 SDK: Auth0Provider wrapping the app with domain and clientId, the useAuth0 hook, authorize() for login and clearSession() for logout, native callback configuration for both Android (manifestPlaceholders) and iOS (CFBundleURLTypes), and the user profile displayed only after isLoading is false?
68
Gemini 3.5 Flash
Next.js
57%8/14
99%+42%
31/31$1.67
2m 28s
With Auth0 Tools·99%·Tool Calls 31
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.2/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (31/31 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(8/8)
Uses @auth0/nextjs-auth0 SDK
Uses v4 server import path
Instantiates Auth0Client
Configures AUTH0_CLIENT_ID
Configures AUTH0_CLIENT_SECRET
Configures AUTH0_SECRET
Configures AUTH0_DOMAIN
Uses getSession for session retrieval
L2 · Hallucination Detection(3/3)
No hallucinated @auth0/nextjs-sdk package
No hallucinated next-auth package
Does not use SPA SDK in server app
L3 · Security(3/3)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
No hardcoded Auth0 domain in source files (ok in .env)
L4 · Structural(8/8)
Wrote Auth0 credentials to .env file
Middleware function is exported from middleware or proxy file
Uses auth0.middleware in middleware file
Dashboard page file exists
Uses /auth/login for login redirect
Uses auth0.getAccessToken() for server-side token retrieval
Requests an access token with audience https://api.playground.com
Does the code set up a working authentication flow with login, logout, and a callback route? Is there a protected /dashboard page that checks the user session and redirects unauthenticated users to log in? Note: Next.js 16 supports both middleware.ts (export function middleware) and proxy.ts (export function proxy) — both are valid.
L5 · Version Correctness(8/8)
Does not use v3 env var AUTH0_BASE_URL (v4 uses APP_BASE_URL)
Does not use v3 env var AUTH0_ISSUER_BASE_URL (removed in v4)
Does not use v3 handleAuth (v4 uses middleware)
Does not use v3 route prefix /api/auth/ (v4 uses /auth/)
Does not use v3 withPageAuthRequired (v4 uses proxy/middleware)
Does not use v3 withApiAuthRequired (removed in v4)
Does the solution correctly integrate Auth0 into a Next.js App Router app using Auth0Client from @auth0/nextjs-auth0/server, proxy or middleware-based auth routing, and getSession for server-side session access? It should NOT use the deprecated v3 patterns like handleAuth, withPageAuthRequired, or /api/auth/ routes. Note: Next.js 16 replaces middleware.ts with proxy.ts (export function proxy) — both are valid. There should also be a protected /dashboard page that checks the session and redirects unauthenticated users to log in.
Does the source code rely on the v4 SDK configuration convention — instantiating Auth0Client so it reads the standard AUTH0_DOMAIN / AUTH0_CLIENT_ID / AUTH0_CLIENT_SECRET / AUTH0_SECRET environment variables (or passing those as options) — and avoid referencing the deprecated v3 names AUTH0_BASE_URL or AUTH0_ISSUER_BASE_URL anywhere in source? Judge only from the source code; do not assume the contents of any .env file.
69
GPT-5.6 Sol
Next.js
86%12/14
99%+13%
31/31$1.81
2m 1s
With Auth0 Tools·99%·Tool Calls 18
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.3/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (31/31 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(8/8)
Uses @auth0/nextjs-auth0 SDK
Uses v4 server import path
Instantiates Auth0Client
Configures AUTH0_CLIENT_ID
Configures AUTH0_CLIENT_SECRET
Configures AUTH0_SECRET
Configures AUTH0_DOMAIN
Uses getSession for session retrieval
L2 · Hallucination Detection(3/3)
No hallucinated @auth0/nextjs-sdk package
No hallucinated next-auth package
Does not use SPA SDK in server app
L3 · Security(3/3)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
No hardcoded Auth0 domain in source files (ok in .env)
L4 · Structural(8/8)
Wrote Auth0 credentials to .env file
Middleware function is exported from middleware or proxy file
Uses auth0.middleware in middleware file
Dashboard page file exists
Uses /auth/login for login redirect
Uses auth0.getAccessToken() for server-side token retrieval
Requests an access token with audience https://api.playground.com
Does the code set up a working authentication flow with login, logout, and a callback route? Is there a protected /dashboard page that checks the user session and redirects unauthenticated users to log in? Note: Next.js 16 supports both middleware.ts (export function middleware) and proxy.ts (export function proxy) — both are valid.
L5 · Version Correctness(8/8)
Does not use v3 env var AUTH0_BASE_URL (v4 uses APP_BASE_URL)
Does not use v3 env var AUTH0_ISSUER_BASE_URL (removed in v4)
Does not use v3 handleAuth (v4 uses middleware)
Does not use v3 route prefix /api/auth/ (v4 uses /auth/)
Does not use v3 withPageAuthRequired (v4 uses proxy/middleware)
Does not use v3 withApiAuthRequired (removed in v4)
Does the solution correctly integrate Auth0 into a Next.js App Router app using Auth0Client from @auth0/nextjs-auth0/server, proxy or middleware-based auth routing, and getSession for server-side session access? It should NOT use the deprecated v3 patterns like handleAuth, withPageAuthRequired, or /api/auth/ routes. Note: Next.js 16 replaces middleware.ts with proxy.ts (export function proxy) — both are valid. There should also be a protected /dashboard page that checks the session and redirects unauthenticated users to log in.
Does the source code rely on the v4 SDK configuration convention — instantiating Auth0Client so it reads the standard AUTH0_DOMAIN / AUTH0_CLIENT_ID / AUTH0_CLIENT_SECRET / AUTH0_SECRET environment variables (or passing those as options) — and avoid referencing the deprecated v3 names AUTH0_BASE_URL or AUTH0_ISSUER_BASE_URL anywhere in source? Judge only from the source code; do not assume the contents of any .env file.
70
GPT-5.6 Sol
Express API
70%7/10
99%+29%
19/19$2.78
3m 8s
With Auth0 Tools·99%·Tool Calls 24
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
10.5/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (19/19 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(3/3)
Uses express-oauth2-jwt-bearer SDK
Uses requiredScopes() for scope-based route protection
Accesses JWT data via req.auth
L2 · Hallucination Detection(4/4)
No express-openid-connect (that is for web apps, not APIs)
No passport middleware (not needed with express-oauth2-jwt-bearer)
No manual JWT verification with jsonwebtoken (use SDK)
No SPA SDK used in server-side API
L3 · Security(2/2)
No hardcoded issuer domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
L4 · Structural(4/4)
Wrote Auth0 config to .env file
GET /api/messages protected with read:messages scope
POST /api/messages protected with write:messages scope
Does the API correctly protect all three routes? GET /api/messages requires read:messages, POST /api/messages requires write:messages, GET /api/profile returns user info from req.auth.payload. Missing or invalid tokens should yield 401, insufficient scope should yield 403.
L5 · Version Correctness(4/4)
No manual Authorization header parsing (SDK handles extraction)
No manual jwt.verify calls (SDK handles verification)
No req.user (express-oauth2-jwt-bearer uses req.auth, not req.user)
Does the solution use current express-oauth2-jwt-bearer patterns? Specifically: does it apply the auth() middleware to protect routes, use requiredScopes() for scope checks (not manual payload inspection), and access token data via req.auth.payload (not req.user or manually decoded tokens)? Judge only from the source code; the issuer and audience may be supplied via environment variables (ISSUER_BASE_URL / AUDIENCE), so do not assume the contents of any .env file.
Holistic(1/1)
Does the solution correctly protect an Express.js API using express-oauth2-jwt-bearer? It should apply the auth() middleware (issuer and audience may come from ISSUER_BASE_URL / AUDIENCE environment variables — judge only from the source code and do not assume the contents of any .env file), protect GET /api/messages with read:messages scope, protect POST /api/messages with write:messages scope, and return user profile info from req.auth.payload at GET /api/profile.
71
Claude Opus 5
Android
33%3/9
99%+66%
14/14$2.84
5m 16s
With Auth0 Tools·99%·Tool Calls 26
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.1/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (14/14 passed)
L1 · Positive Presence(5/5)
Uses Auth0 Android SDK dependency
Uses WebAuthProvider for authentication
Calls WebAuthProvider.login() for sign-in
Calls WebAuthProvider.logout() for sign-out
Uses CredentialsManager for secure token storage
L2 · Hallucination Detection(1/1)
No auth0-java (server-side SDK, not for Android)
L3 · Security(2/2)
No hardcoded client ID in Kotlin source files (ok in strings.xml)
No hardcoded domain in Kotlin source files (ok in strings.xml)
L4 · Structural(3/3)
Configures manifestPlaceholders in build.gradle for Auth0 callback URL scheme
Does the code implement both onSuccess and onFailure callbacks for WebAuthProvider.login(...) and WebAuthProvider.logout(...), with AuthenticationException handled in onFailure?
Is the Auth0 callback URL scheme correctly handled? Check that the Gradle configuration defines/provides manifestPlaceholders with auth0Domain and auth0Scheme keys, regardless of whether they are set in defaultConfig, buildTypes, productFlavors, or merged at the android level. IMPORTANT: using @string/ resource references (e.g. auth0Domain: "@string/com_auth0_domain") is the CORRECT and recommended Auth0 pattern — do NOT mark this as wrong. Also check that WebAuthProvider.login(...) and WebAuthProvider.logout(...) call withScheme(). Note: SDK v2+ does NOT require a manual redirect activity in AndroidManifest.xml.
L5 · Version Correctness(2/2)
Uses Auth0(context) or Auth0.getInstance(context) for auto-configuration from string resources
Does the code use current Auth0 Android SDK v2+ patterns? Specifically: Auth0(context) for auto-configuration from strings.xml, WebAuthProvider for browser-based login/logout, and CredentialsManager for secure credential storage rather than persisting tokens manually in SharedPreferences?
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Android app with WebAuthProvider login and logout, CredentialsManager for secure token storage, Auth0 credentials configured via string resources, and manifestPlaceholders for callback URL handling? IMPORTANT correct patterns to accept: (1) manifestPlaceholders using @string/ resource references is correct and recommended; (2) Auth0(context) or Auth0.getInstance(context) auto-reads from strings.xml — no explicit client ID/domain constructor args needed; (3) SDK v2+ does NOT require a manual redirect activity in AndroidManifest.xml.
72
Claude Haiku 4.5
Fastify API
54%7/13
98%+44%
19/20$0.09
54s
With Auth0 Tools·98%·Tool Calls 10
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
23.2/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
98/100 ptsA
Grader Results (19/20 passed)
Build Failing
We compile the code for real. This attempt didn’t make it through.
L1 · Positive Presence(6/6)
Uses @auth0/auth0-fastify-api package
Imports fastifyAuth0Api plugin
Uses fastify.requireAuth() to protect routes
Uses preHandler to attach auth middleware
Configures domain
Configures audience
L2 · Hallucination Detection(4/4)
No express-oauth2-jwt-bearer (wrong SDK for Fastify)
No passport middleware (not needed with @auth0/auth0-fastify-api)
No jsonwebtoken (manual JWT verification not needed)
No manual jwt.verify() calls
L3 · Security(2/2)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
L4 · Structural(5/5)
Wrote Auth0 config to .env file
Auth0 API plugin registered with fastify.register()
Route protected with preHandler: fastify.requireAuth()
read:messages scope checked on /api/messages route
Does the app correctly register the @auth0/auth0-fastify-api plugin, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token? Note: @auth0/auth0-fastify-api exposes the verified token claims on request.user (e.g. request.user.sub) — this is the correct API for this plugin, not request.auth.
L5 · Version Correctness(1/1)
Does the solution use current @auth0/auth0-fastify-api patterns? Specifically: does it register the plugin via fastify.register(), use preHandler: fastify.requireAuth() for route protection (not a decorator), access token claims via request.user, and read credentials from environment variables (not hardcoded)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Fastify API using @auth0/auth0-fastify-api? It should register the plugin with domain and audience from environment variables, protect the /api/messages route using fastify.requireAuth() with read:messages scope (as string or array) check using preHandler, and protect the /api/private route using fastify.requireAuth() requiring any valid access token.
73
GPT-5.6 Luna
Fastify API
54%7/13
98%+44%
20/20$0.22
1m 14s
With Auth0 Tools·98%·Tool Calls 11
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
9.8/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
98/100 ptsA
Grader Results (20/20 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses @auth0/auth0-fastify-api package
Imports fastifyAuth0Api plugin
Uses fastify.requireAuth() to protect routes
Uses preHandler to attach auth middleware
Configures domain
Configures audience
L2 · Hallucination Detection(4/4)
No express-oauth2-jwt-bearer (wrong SDK for Fastify)
No passport middleware (not needed with @auth0/auth0-fastify-api)
No jsonwebtoken (manual JWT verification not needed)
No manual jwt.verify() calls
L3 · Security(2/2)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
L4 · Structural(5/5)
Wrote Auth0 config to .env file
Auth0 API plugin registered with fastify.register()
Route protected with preHandler: fastify.requireAuth()
read:messages scope checked on /api/messages route
Does the app correctly register the @auth0/auth0-fastify-api plugin, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token? Note: @auth0/auth0-fastify-api exposes the verified token claims on request.user (e.g. request.user.sub) — this is the correct API for this plugin, not request.auth.
L5 · Version Correctness(1/1)
Does the solution use current @auth0/auth0-fastify-api patterns? Specifically: does it register the plugin via fastify.register(), use preHandler: fastify.requireAuth() for route protection (not a decorator), access token claims via request.user, and read credentials from environment variables (not hardcoded)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Fastify API using @auth0/auth0-fastify-api? It should register the plugin with domain and audience from environment variables, protect the /api/messages route using fastify.requireAuth() with read:messages scope (as string or array) check using preHandler, and protect the /api/private route using fastify.requireAuth() requiring any valid access token.
74
Gemini 3.1 Pro
Express
73%11/15
98%+25%
23/24$0.53
59s
With Auth0 Tools·98%·Tool Calls 13
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.1/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
23.5/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
98/100 ptsA
Grader Results (23/24 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(7/7)
Uses express-openid-connect SDK
Configures issuerBaseURL
Configures clientID
Configures baseURL
Configures session secret
Uses req.oidc to access OIDC context
Accesses user profile via req.oidc.user
L2 · Hallucination Detection(4/4)
No hallucinated @auth0/auth0-spa-js (wrong SDK for server-side app)
No express-oauth2-jwt-bearer (that is for APIs, not web apps)
Does not use Next.js SDK in Express app
No passport middleware (not needed with express-openid-connect)
L3 · Security(3/3)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
No hardcoded Auth0 domain in source files (ok in .env)
L4 · Structural(4/5)
Wrote Auth0 credentials to .env file
auth middleware registered with app.use
Uses requiresAuth() to protect the /profile route
Accesses access token via req.oidc.accessToken
Does the app correctly register the auth() middleware, protect the /profile route with requiresAuth(), display user profile information, and include a route that calls an external API using the access token?
L5 · Version Correctness(3/3)
Uses authorizationParams to pass audience and scope
authorizationParams contains audience 'https://api.barkbook.com'
Does the solution use current express-openid-connect patterns? Specifically: does it use issuerBaseURL (not AUTH0_DOMAIN or domain directly), configure audience via authorizationParams (not as a top-level config key), and use response_type: "code" for the authorization code flow?
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Express web app using express-openid-connect? It should configure the auth() middleware, protect the /profile route with requiresAuth(), display the logged-in user profile, and use the access token to call an external API with audience https://api.barkbook.com.
75
Gemini 3.1 Pro
FastAPI
50%7/14
98%+48%
20/21$0.53
1m 5s
With Auth0 Tools·98%·Tool Calls 12
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
23.2/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
98/100 ptsA
Grader Results (20/21 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses auth0-fastapi-api package
Imports Auth0FastAPI class
Uses require_auth() dependency
Uses FastAPI Depends for dependency injection
Configures domain
Configures audience
L2 · Hallucination Detection(5/5)
No python-jose (manual JWT parsing not needed)
No PyJWT direct usage
No manual JWT decoding
No fastapi-users (wrong package for Auth0 JWT)
No passlib (unrelated auth package)
L3 · Security(2/2)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
L4 · Structural(4/5)
Wrote Auth0 config to .env file
Auth0FastAPI instance is created
Route protected with Depends(auth0.require_auth())
read:messages scope checked on /api/messages route
Does the app correctly create an Auth0FastAPI instance, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token? Note: the auth0-fastapi-api distribution imports as the module `fastapi_plugin` (i.e. `from fastapi_plugin import Auth0FastAPI` is correct — there is no `auth0_fastapi_api` module).
L5 · Version Correctness(1/1)
Does the solution use current auth0-fastapi-api patterns? Specifically: does it use Auth0FastAPI with domain and audience parameters, use require_auth() as a FastAPI Depends dependency (not a decorator), and read credentials from environment variables (not hardcoded)? Note: the auth0-fastapi-api distribution imports as the module `fastapi_plugin` (`from fastapi_plugin import Auth0FastAPI` is correct — there is no `auth0_fastapi_api` module).
Holistic(1/1)
Does the solution correctly integrate Auth0 into a FastAPI API using auth0-fastapi-api? It should create an Auth0FastAPI instance configured with domain and audience from environment variables, protect the /api/messages route with read:messages scope check, and protect the /api/private route requiring a valid access token. Note: the auth0-fastapi-api distribution imports as the module `fastapi_plugin` (`from fastapi_plugin import Auth0FastAPI` is correct — there is no `auth0_fastapi_api` module).
76
GPT-5.6 Terra
Swift
70%7/10
98%+28%
14/14$0.55
1m 8s
With Auth0 Tools·98%·Tool Calls 10
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
9.6/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
98/100 ptsA
Grader Results (14/14 passed)
L1 · Positive Presence(4/4)
Uses Auth0 SDK
Imports Auth0 module
Uses webAuth() for login
Uses clearSession for logout
L2 · Hallucination Detection(3/3)
No hallucinated Auth0SDK package name (correct package is Auth0)
Does not use CocoaPods (SPM preferred)
Does not use deprecated completion handler pattern
L3 · Security(2/2)
No hardcoded client ID in Swift source files (ok in Auth0.plist)
No hardcoded domain in Swift source files (ok in Auth0.plist)
L4 · Structural(2/2)
Uses CredentialsManager for token storage
Does the code properly handle login and logout flows with appropriate error handling? Does it update the UI state after successful authentication?
L5 · Version Correctness(2/2)
Uses async/await webAuth().start() syntax (not completion handlers)
Does the code use modern Swift async/await patterns with the Auth0.swift SDK? Specifically: does it use try await webAuth().start() and CredentialsManager, and configure via Auth0.plist rather than hardcoded strings?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Swift iOS app with webAuth() login/logout, credential management, and proper SwiftUI state handling?
77
Gemini 3.1 Pro
Fastify API
54%7/13
97%+43%
19/20$0.69
1m 4s
With Auth0 Tools·97%·Tool Calls 16
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.3/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
23.2/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
97/100 ptsA
Grader Results (19/20 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses @auth0/auth0-fastify-api package
Imports fastifyAuth0Api plugin
Uses fastify.requireAuth() to protect routes
Uses preHandler to attach auth middleware
Configures domain
Configures audience
L2 · Hallucination Detection(4/4)
No express-oauth2-jwt-bearer (wrong SDK for Fastify)
No passport middleware (not needed with @auth0/auth0-fastify-api)
No jsonwebtoken (manual JWT verification not needed)
No manual jwt.verify() calls
L3 · Security(2/2)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
L4 · Structural(4/5)
Wrote Auth0 config to .env file
Auth0 API plugin registered with fastify.register()
Route protected with preHandler: fastify.requireAuth()
read:messages scope checked on /api/messages route
Does the app correctly register the @auth0/auth0-fastify-api plugin, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token? Note: @auth0/auth0-fastify-api exposes the verified token claims on request.user (e.g. request.user.sub) — this is the correct API for this plugin, not request.auth.
L5 · Version Correctness(1/1)
Does the solution use current @auth0/auth0-fastify-api patterns? Specifically: does it register the plugin via fastify.register(), use preHandler: fastify.requireAuth() for route protection (not a decorator), access token claims via request.user, and read credentials from environment variables (not hardcoded)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Fastify API using @auth0/auth0-fastify-api? It should register the plugin with domain and audience from environment variables, protect the /api/messages route using fastify.requireAuth() with read:messages scope (as string or array) check using preHandler, and protect the /api/private route using fastify.requireAuth() requiring any valid access token.
78
Gemini 3.5 Flash
FastAPI
43%6/14
98%+55%
20/21$0.81
2m 8s
With Auth0 Tools·98%·Tool Calls 19
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.4/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
23.2/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
98/100 ptsA
Grader Results (20/21 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses auth0-fastapi-api package
Imports Auth0FastAPI class
Uses require_auth() dependency
Uses FastAPI Depends for dependency injection
Configures domain
Configures audience
L2 · Hallucination Detection(5/5)
No python-jose (manual JWT parsing not needed)
No PyJWT direct usage
No manual JWT decoding
No fastapi-users (wrong package for Auth0 JWT)
No passlib (unrelated auth package)
L3 · Security(2/2)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
L4 · Structural(4/5)
Wrote Auth0 config to .env file
Auth0FastAPI instance is created
Route protected with Depends(auth0.require_auth())
read:messages scope checked on /api/messages route
Does the app correctly create an Auth0FastAPI instance, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token? Note: the auth0-fastapi-api distribution imports as the module `fastapi_plugin` (i.e. `from fastapi_plugin import Auth0FastAPI` is correct — there is no `auth0_fastapi_api` module).
L5 · Version Correctness(1/1)
Does the solution use current auth0-fastapi-api patterns? Specifically: does it use Auth0FastAPI with domain and audience parameters, use require_auth() as a FastAPI Depends dependency (not a decorator), and read credentials from environment variables (not hardcoded)? Note: the auth0-fastapi-api distribution imports as the module `fastapi_plugin` (`from fastapi_plugin import Auth0FastAPI` is correct — there is no `auth0_fastapi_api` module).
Holistic(1/1)
Does the solution correctly integrate Auth0 into a FastAPI API using auth0-fastapi-api? It should create an Auth0FastAPI instance configured with domain and audience from environment variables, protect the /api/messages route with read:messages scope check, and protect the /api/private route requiring a valid access token. Note: the auth0-fastapi-api distribution imports as the module `fastapi_plugin` (`from fastapi_plugin import Auth0FastAPI` is correct — there is no `auth0_fastapi_api` module).
79
GPT-5.6 Terra
Express API
70%7/10
98%+28%
19/19$0.91
1m 53s
With Auth0 Tools·98%·Tool Calls 15
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
9.6/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
98/100 ptsA
Grader Results (19/19 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(3/3)
Uses express-oauth2-jwt-bearer SDK
Uses requiredScopes() for scope-based route protection
Accesses JWT data via req.auth
L2 · Hallucination Detection(4/4)
No express-openid-connect (that is for web apps, not APIs)
No passport middleware (not needed with express-oauth2-jwt-bearer)
No manual JWT verification with jsonwebtoken (use SDK)
No SPA SDK used in server-side API
L3 · Security(2/2)
No hardcoded issuer domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
L4 · Structural(4/4)
Wrote Auth0 config to .env file
GET /api/messages protected with read:messages scope
POST /api/messages protected with write:messages scope
Does the API correctly protect all three routes? GET /api/messages requires read:messages, POST /api/messages requires write:messages, GET /api/profile returns user info from req.auth.payload. Missing or invalid tokens should yield 401, insufficient scope should yield 403.
L5 · Version Correctness(4/4)
No manual Authorization header parsing (SDK handles extraction)
No manual jwt.verify calls (SDK handles verification)
No req.user (express-oauth2-jwt-bearer uses req.auth, not req.user)
Does the solution use current express-oauth2-jwt-bearer patterns? Specifically: does it apply the auth() middleware to protect routes, use requiredScopes() for scope checks (not manual payload inspection), and access token data via req.auth.payload (not req.user or manually decoded tokens)? Judge only from the source code; the issuer and audience may be supplied via environment variables (ISSUER_BASE_URL / AUDIENCE), so do not assume the contents of any .env file.
Holistic(1/1)
Does the solution correctly protect an Express.js API using express-oauth2-jwt-bearer? It should apply the auth() middleware (issuer and audience may come from ISSUER_BASE_URL / AUDIENCE environment variables — judge only from the source code and do not assume the contents of any .env file), protect GET /api/messages with read:messages scope, protect POST /api/messages with write:messages scope, and return user profile info from req.auth.payload at GET /api/profile.
80
Claude Opus 5
SPA JS
93%13/14
98%+5%
20/21$1.05
2m 7s
With Auth0 Tools·98%·Tool Calls 14
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
23.3/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
98/100 ptsA
Grader Results (20/21 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(7/7)
Uses @auth0/auth0-spa-js SDK
Initializes Auth0 with createAuth0Client
Implements loginWithRedirect
Handles redirect callback on page load
Implements logout
Checks isAuthenticated for conditional rendering
Uses getTokenSilently to retrieve access token
L2 · Hallucination Detection(4/4)
Does not use React SDK in vanilla JS app
Does not use Vue SDK in vanilla JS app
Does not use Next.js SDK in vanilla JS app
No client_secret in SPA (public client)
L3 · Security(2/2)
No tokens manually stored in localStorage
No tokens manually stored in sessionStorage
L4 · Structural(3/3)
Auth0Client configured with domain
Authenticated API request uses Bearer token in Authorization header
Does the code check isAuthenticated to show/hide UI elements and only render user profile information when the user is authenticated?
L5 · Version Correctness(3/3)
Uses authorizationParams (not deprecated top-level audience/redirect_uri)
authorizationParams contains audience 'https://api.playground.com'
No deprecated promise-chain pattern for getTokenSilently
Holistic(0/1)
Does the solution correctly integrate Auth0 into a vanilla JavaScript SPA using @auth0/auth0-spa-js with createAuth0Client, loginWithRedirect, handleRedirectCallback, logout, user profile display, and getTokenSilently to make authenticated API calls?
81
GPT-5.6 Sol
Fastify API
69%9/13
98%+29%
20/20$1.06
1m 11s
With Auth0 Tools·98%·Tool Calls 10
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
9.6/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
98/100 ptsA
Grader Results (20/20 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses @auth0/auth0-fastify-api package
Imports fastifyAuth0Api plugin
Uses fastify.requireAuth() to protect routes
Uses preHandler to attach auth middleware
Configures domain
Configures audience
L2 · Hallucination Detection(4/4)
No express-oauth2-jwt-bearer (wrong SDK for Fastify)
No passport middleware (not needed with @auth0/auth0-fastify-api)
No jsonwebtoken (manual JWT verification not needed)
No manual jwt.verify() calls
L3 · Security(2/2)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
L4 · Structural(5/5)
Wrote Auth0 config to .env file
Auth0 API plugin registered with fastify.register()
Route protected with preHandler: fastify.requireAuth()
read:messages scope checked on /api/messages route
Does the app correctly register the @auth0/auth0-fastify-api plugin, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token? Note: @auth0/auth0-fastify-api exposes the verified token claims on request.user (e.g. request.user.sub) — this is the correct API for this plugin, not request.auth.
L5 · Version Correctness(1/1)
Does the solution use current @auth0/auth0-fastify-api patterns? Specifically: does it register the plugin via fastify.register(), use preHandler: fastify.requireAuth() for route protection (not a decorator), access token claims via request.user, and read credentials from environment variables (not hardcoded)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Fastify API using @auth0/auth0-fastify-api? It should register the plugin with domain and audience from environment variables, protect the /api/messages route using fastify.requireAuth() with read:messages scope (as string or array) check using preHandler, and protect the /api/private route using fastify.requireAuth() requiring any valid access token.
82
Gemini 3.5 Flash
Vue
92%12/13
98%+6%
20/20$1.25
1m 52s
With Auth0 Tools·98%·Tool Calls 23
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
10.4/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
98/100 ptsA
Grader Results (20/20 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(7/7)
Uses @auth0/auth0-vue SDK
Sets up Auth0 plugin with createAuth0
Uses useAuth0 composable
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile information
L2 · Hallucination Detection(3/3)
Does not use React SDK in Vue app
No hallucinated @auth0/vue3-auth0 package
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Plugin installed via app.use(createAuth0(...))
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated? A correct implementation should not render auth-dependent UI while isLoading is true.
L5 · Version Correctness(3/3)
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Uses clientId (not deprecated client_id) in createAuth0 config
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Vue 3 SPA with the @auth0/auth0-vue plugin, useAuth0 composable, login, logout, user profile display, route protection, and getAccessTokenSilently to make authenticated API calls?
83
Gemini 3.1 Pro
Nuxt
55%11/20
98%+43%
31/32$1.47
3m 3s
With Auth0 Tools·98%·Tool Calls 22
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.5/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
23.9/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
98/100 ptsA
Grader Results (31/32 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(10/10)
Uses @auth0/auth0-nuxt SDK
Registers module in nuxt.config
Configures runtimeConfig with Auth0 settings
Configures NUXT_AUTH0_DOMAIN env var
Configures NUXT_AUTH0_CLIENT_ID env var
Configures NUXT_AUTH0_CLIENT_SECRET env var
Configures NUXT_AUTH0_SESSION_SECRET env var
Uses useUser() composable to access the authenticated user
Uses /auth/login route to initiate login
Uses /auth/logout route to log out
L2 · Hallucination Detection(4/4)
Does not use the Vue SDK in a Nuxt app
Does not use the React SDK in a Nuxt app
Does not use loginWithRedirect (SPA API not applicable to Nuxt)
Does not use getAccessTokenSilently (Vue/React SPA API, not used in Nuxt)
L3 · Security(5/5)
No hardcoded client ID in source files (ok in .env)
No hardcoded client secret in source files (ok in .env)
No hardcoded Auth0 domain in source files (ok in .env)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(6/7)
Wrote Auth0 credentials to .env file
Module correctly registered in the modules array
sessionSecret configured for session encryption
clientSecret configured (Regular Web Application)
appBaseUrl configured in runtimeConfig
definePageMeta used for page-level middleware
Is there a protected /profile route that uses a Nuxt route middleware (via definePageMeta) to check authentication with useUser() and redirect unauthenticated users to /auth/login?
L5 · Version Correctness(4/4)
Audience configured in runtimeConfig for API access
Audience set to 'https://api.playground.com'
Uses getAccessToken() server-side to retrieve access token
Uses useAuth0(event) server-side composable
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Nuxt application using the @auth0/auth0-nuxt module, with proper configuration in nuxt.config.ts (domain, clientId, clientSecret, sessionSecret, appBaseUrl, audience), login/logout via /auth/login and /auth/logout, user profile display with useUser(), a /profile route protected by a route middleware, and server-side getAccessToken() for authenticated API calls?
84
Claude Opus 5
Swift
60%6/10
98%+38%
14/14$1.48
2m 59s
With Auth0 Tools·98%·Tool Calls 15
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
10.4/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
98/100 ptsA
Grader Results (14/14 passed)
L1 · Positive Presence(4/4)
Uses Auth0 SDK
Imports Auth0 module
Uses webAuth() for login
Uses clearSession for logout
L2 · Hallucination Detection(3/3)
No hallucinated Auth0SDK package name (correct package is Auth0)
Does not use CocoaPods (SPM preferred)
Does not use deprecated completion handler pattern
L3 · Security(2/2)
No hardcoded client ID in Swift source files (ok in Auth0.plist)
No hardcoded domain in Swift source files (ok in Auth0.plist)
L4 · Structural(2/2)
Uses CredentialsManager for token storage
Does the code properly handle login and logout flows with appropriate error handling? Does it update the UI state after successful authentication?
L5 · Version Correctness(2/2)
Uses async/await webAuth().start() syntax (not completion handlers)
Does the code use modern Swift async/await patterns with the Auth0.swift SDK? Specifically: does it use try await webAuth().start() and CredentialsManager, and configure via Auth0.plist rather than hardcoded strings?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Swift iOS app with webAuth() login/logout, credential management, and proper SwiftUI state handling?
85
Gemini 3.5 Flash
Nuxt
40%8/20
98%+58%
31/32$1.48
2m 33s
With Auth0 Tools·98%·Tool Calls 27
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.1/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
23.9/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
98/100 ptsA
Grader Results (31/32 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(10/10)
Uses @auth0/auth0-nuxt SDK
Registers module in nuxt.config
Configures runtimeConfig with Auth0 settings
Configures NUXT_AUTH0_DOMAIN env var
Configures NUXT_AUTH0_CLIENT_ID env var
Configures NUXT_AUTH0_CLIENT_SECRET env var
Configures NUXT_AUTH0_SESSION_SECRET env var
Uses useUser() composable to access the authenticated user
Uses /auth/login route to initiate login
Uses /auth/logout route to log out
L2 · Hallucination Detection(4/4)
Does not use the Vue SDK in a Nuxt app
Does not use the React SDK in a Nuxt app
Does not use loginWithRedirect (SPA API not applicable to Nuxt)
Does not use getAccessTokenSilently (Vue/React SPA API, not used in Nuxt)
L3 · Security(5/5)
No hardcoded client ID in source files (ok in .env)
No hardcoded client secret in source files (ok in .env)
No hardcoded Auth0 domain in source files (ok in .env)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(6/7)
Wrote Auth0 credentials to .env file
Module correctly registered in the modules array
sessionSecret configured for session encryption
clientSecret configured (Regular Web Application)
appBaseUrl configured in runtimeConfig
definePageMeta used for page-level middleware
Is there a protected /profile route that uses a Nuxt route middleware (via definePageMeta) to check authentication with useUser() and redirect unauthenticated users to /auth/login?
L5 · Version Correctness(4/4)
Audience configured in runtimeConfig for API access
Audience set to 'https://api.playground.com'
Uses getAccessToken() server-side to retrieve access token
Uses useAuth0(event) server-side composable
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Nuxt application using the @auth0/auth0-nuxt module, with proper configuration in nuxt.config.ts (domain, clientId, clientSecret, sessionSecret, appBaseUrl, audience), login/logout via /auth/login and /auth/logout, user profile display with useUser(), a /profile route protected by a route middleware, and server-side getAccessToken() for authenticated API calls?
86
Claude Sonnet 5
React Native
85%11/13
98%+13%
21/21$1.57
7m 22s
With Auth0 Tools·98%·Tool Calls 39
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
4.6/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
98/100 ptsA
Grader Results (21/21 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(5/5)
Uses the react-native-auth0 SDK
Wraps app with Auth0Provider
Uses the useAuth0 hook
Implements login via authorize()
Implements logout via clearSession()
L2 · Hallucination Detection(5/5)
No web React SDK (correct package is react-native-auth0)
No browser SPA SDK in a native React Native app
Does not use Expo AuthSession in a bare React Native app
Does not fall back to the third-party react-native-app-auth library
Does not register the Expo config plugin in a bare React Native app
L3 · Security(2/2)
No hardcoded client ID in source files (ok in config)
Does not store tokens in insecure AsyncStorage
L4 · Structural(6/6)
Auth0Provider configured with a domain prop
android/app/build.gradle sets the auth0Domain manifestPlaceholder to the real tenant domain
android/app/build.gradle sets the auth0Scheme manifestPlaceholder to <applicationId>.auth0
iOS Info.plist registers the bundle-identifier callback URL scheme
Does the code handle the loading state (isLoading from useAuth0) before rendering auth-dependent UI? A correct implementation should not render login/logout UI while isLoading is true.
Are BOTH native platforms configured for the Auth0 callback in this bare React Native app? Android needs auth0Domain and auth0Scheme manifestPlaceholders in android/app/build.gradle (the RedirectActivity is merged in by the SDK manifest, so declaring it manually is also acceptable), and iOS needs a CFBundleURLTypes entry in ios/scaffold/Info.plist. Answer no if only one platform was configured.
L5 · Version Correctness(1/1)
Does the code use the current react-native-auth0 v5 patterns? Specifically: the Auth0Provider + useAuth0 hook API rather than the legacy imperative `new Auth0({domain, clientId})` client class, and `user` from the hook rather than a manual getCredentials()/userInfo() call to read the profile.
Holistic(1/1)
Does the solution correctly integrate Auth0 into a bare (non-Expo) React Native app using the react-native-auth0 SDK: Auth0Provider wrapping the app with domain and clientId, the useAuth0 hook, authorize() for login and clearSession() for logout, native callback configuration for both Android (manifestPlaceholders) and iOS (CFBundleURLTypes), and the user profile displayed only after isLoading is false?
87
GPT-5.6 Sol
Flask
36%5/14
98%+62%
25/26$2.08
3m 34s
With Auth0 Tools·98%·Tool Calls 16
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.3/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
23.7/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
98/100 ptsA
Grader Results (25/26 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses auth0-server-python package
Imports ServerClient class
Uses start_interactive_login method
Uses complete_interactive_login method
Uses get_user method to retrieve user info
Loads environment variables with python-dotenv
L2 · Hallucination Detection(4/4)
No Authlib (old approach, not the current SDK)
No python-jose (manual JWT parsing not needed)
No Flask-Login (not needed with auth0-server-python)
No Flask-Dance (wrong OAuth package)
L3 · Security(3/3)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
No hardcoded Auth0 domain in source files (ok in .env)
L4 · Structural(8/8)
Wrote Auth0 credentials to .env file
Implements /callback route
Implements /login route
Implements /logout route
Implements /profile route
Sets Flask app.secret_key for session management
Uses async route handlers
Does the app implement login, callback, profile, and logout routes using auth0-server-python, protect the /profile route so it requires login, and include a route that calls an external API using an access token?
L5 · Version Correctness(3/3)
Uses authorization_params for scope and audience configuration
authorization_params contains audience 'https://api.barkbook.com'
Does the solution use current auth0-server-python patterns? Specifically: does it import ServerClient from auth0_server_python, define concrete state and transaction store classes locally (subclassing StateStore/TransactionStore from the SDK), and configure authorization_params with audience and scope?
Holistic(0/1)
Does the solution correctly integrate Auth0 into a Flask web app using auth0-server-python? It should configure a ServerClient with credentials from env vars, implement login, callback, profile, and logout routes, protect the /profile route so it requires authentication, and use the access token to call an external API with audience https://api.barkbook.com.
88
GPT-5.6 Sol
Android
67%6/9
98%+31%
14/14$2.56
3m 52s
With Auth0 Tools·98%·Tool Calls 24
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
10/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
98/100 ptsA
Grader Results (14/14 passed)
L1 · Positive Presence(5/5)
Uses Auth0 Android SDK dependency
Uses WebAuthProvider for authentication
Calls WebAuthProvider.login() for sign-in
Calls WebAuthProvider.logout() for sign-out
Uses CredentialsManager for secure token storage
L2 · Hallucination Detection(1/1)
No auth0-java (server-side SDK, not for Android)
L3 · Security(2/2)
No hardcoded client ID in Kotlin source files (ok in strings.xml)
No hardcoded domain in Kotlin source files (ok in strings.xml)
L4 · Structural(3/3)
Configures manifestPlaceholders in build.gradle for Auth0 callback URL scheme
Does the code implement both onSuccess and onFailure callbacks for WebAuthProvider.login(...) and WebAuthProvider.logout(...), with AuthenticationException handled in onFailure?
Is the Auth0 callback URL scheme correctly handled? Check that the Gradle configuration defines/provides manifestPlaceholders with auth0Domain and auth0Scheme keys, regardless of whether they are set in defaultConfig, buildTypes, productFlavors, or merged at the android level. IMPORTANT: using @string/ resource references (e.g. auth0Domain: "@string/com_auth0_domain") is the CORRECT and recommended Auth0 pattern — do NOT mark this as wrong. Also check that WebAuthProvider.login(...) and WebAuthProvider.logout(...) call withScheme(). Note: SDK v2+ does NOT require a manual redirect activity in AndroidManifest.xml.
L5 · Version Correctness(2/2)
Uses Auth0(context) or Auth0.getInstance(context) for auto-configuration from string resources
Does the code use current Auth0 Android SDK v2+ patterns? Specifically: Auth0(context) for auto-configuration from strings.xml, WebAuthProvider for browser-based login/logout, and CredentialsManager for secure credential storage rather than persisting tokens manually in SharedPreferences?
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Android app with WebAuthProvider login and logout, CredentialsManager for secure token storage, Auth0 credentials configured via string resources, and manifestPlaceholders for callback URL handling? IMPORTANT correct patterns to accept: (1) manifestPlaceholders using @string/ resource references is correct and recommended; (2) Auth0(context) or Auth0.getInstance(context) auto-reads from strings.xml — no explicit client ID/domain constructor args needed; (3) SDK v2+ does NOT require a manual redirect activity in AndroidManifest.xml.
89
GPT-5.6 Sol
Nuxt
70%14/20
98%+28%
32/32$4.11
4m 9s
With Auth0 Tools·98%·Tool Calls 29
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
10.3/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
98/100 ptsA
Grader Results (32/32 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(10/10)
Uses @auth0/auth0-nuxt SDK
Registers module in nuxt.config
Configures runtimeConfig with Auth0 settings
Configures NUXT_AUTH0_DOMAIN env var
Configures NUXT_AUTH0_CLIENT_ID env var
Configures NUXT_AUTH0_CLIENT_SECRET env var
Configures NUXT_AUTH0_SESSION_SECRET env var
Uses useUser() composable to access the authenticated user
Uses /auth/login route to initiate login
Uses /auth/logout route to log out
L2 · Hallucination Detection(4/4)
Does not use the Vue SDK in a Nuxt app
Does not use the React SDK in a Nuxt app
Does not use loginWithRedirect (SPA API not applicable to Nuxt)
Does not use getAccessTokenSilently (Vue/React SPA API, not used in Nuxt)
L3 · Security(5/5)
No hardcoded client ID in source files (ok in .env)
No hardcoded client secret in source files (ok in .env)
No hardcoded Auth0 domain in source files (ok in .env)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(7/7)
Wrote Auth0 credentials to .env file
Module correctly registered in the modules array
sessionSecret configured for session encryption
clientSecret configured (Regular Web Application)
appBaseUrl configured in runtimeConfig
definePageMeta used for page-level middleware
Is there a protected /profile route that uses a Nuxt route middleware (via definePageMeta) to check authentication with useUser() and redirect unauthenticated users to /auth/login?
L5 · Version Correctness(4/4)
Audience configured in runtimeConfig for API access
Audience set to 'https://api.playground.com'
Uses getAccessToken() server-side to retrieve access token
Uses useAuth0(event) server-side composable
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Nuxt application using the @auth0/auth0-nuxt module, with proper configuration in nuxt.config.ts (domain, clientId, clientSecret, sessionSecret, appBaseUrl, audience), login/logout via /auth/login and /auth/logout, user profile display with useUser(), a /profile route protected by a route middleware, and server-side getAccessToken() for authenticated API calls?
90
Claude Haiku 4.5
Swift
70%7/10
97%+27%
13/14$0.18
1m 12s
With Auth0 Tools·97%·Tool Calls 14
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
22.2/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
97/100 ptsA
Grader Results (13/14 passed)
L1 · Positive Presence(4/4)
Uses Auth0 SDK
Imports Auth0 module
Uses webAuth() for login
Uses clearSession for logout
L2 · Hallucination Detection(3/3)
No hallucinated Auth0SDK package name (correct package is Auth0)
Does not use CocoaPods (SPM preferred)
Does not use deprecated completion handler pattern
L3 · Security(2/2)
No hardcoded client ID in Swift source files (ok in Auth0.plist)
No hardcoded domain in Swift source files (ok in Auth0.plist)
L4 · Structural(2/2)
Uses CredentialsManager for token storage
Does the code properly handle login and logout flows with appropriate error handling? Does it update the UI state after successful authentication?
L5 · Version Correctness(2/2)
Uses async/await webAuth().start() syntax (not completion handlers)
Does the code use modern Swift async/await patterns with the Auth0.swift SDK? Specifically: does it use try await webAuth().start() and CredentialsManager, and configure via Auth0.plist rather than hardcoded strings?
Holistic(0/1)
Does the solution correctly integrate Auth0 into a Swift iOS app with webAuth() login/logout, credential management, and proper SwiftUI state handling?
91
GPT-5.6 Luna
Express
80%12/15
97%+17%
23/24$0.23
1m 35s
With Auth0 Tools·97%·Tool Calls 10
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
10.8/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
23.5/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
97/100 ptsA
Grader Results (23/24 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(7/7)
Uses express-openid-connect SDK
Configures issuerBaseURL
Configures clientID
Configures baseURL
Configures session secret
Uses req.oidc to access OIDC context
Accesses user profile via req.oidc.user
L2 · Hallucination Detection(4/4)
No hallucinated @auth0/auth0-spa-js (wrong SDK for server-side app)
No express-oauth2-jwt-bearer (that is for APIs, not web apps)
Does not use Next.js SDK in Express app
No passport middleware (not needed with express-openid-connect)
L3 · Security(3/3)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
No hardcoded Auth0 domain in source files (ok in .env)
L4 · Structural(5/5)
Wrote Auth0 credentials to .env file
auth middleware registered with app.use
Uses requiresAuth() to protect the /profile route
Accesses access token via req.oidc.accessToken
Does the app correctly register the auth() middleware, protect the /profile route with requiresAuth(), display user profile information, and include a route that calls an external API using the access token?
L5 · Version Correctness(3/3)
Uses authorizationParams to pass audience and scope
authorizationParams contains audience 'https://api.barkbook.com'
Does the solution use current express-openid-connect patterns? Specifically: does it use issuerBaseURL (not AUTH0_DOMAIN or domain directly), configure audience via authorizationParams (not as a top-level config key), and use response_type: "code" for the authorization code flow?
Holistic(0/1)
Does the solution correctly integrate Auth0 into an Express web app using express-openid-connect? It should configure the auth() middleware, protect the /profile route with requiresAuth(), display the logged-in user profile, and use the access token to call an external API with audience https://api.barkbook.com.
92
Gemini 3.1 Pro
Swift
80%8/10
97%+17%
13/14$0.37
48s
With Auth0 Tools·97%·Tool Calls 12
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
22.2/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
97/100 ptsA
Grader Results (13/14 passed)
L1 · Positive Presence(4/4)
Uses Auth0 SDK
Imports Auth0 module
Uses webAuth() for login
Uses clearSession for logout
L2 · Hallucination Detection(3/3)
No hallucinated Auth0SDK package name (correct package is Auth0)
Does not use CocoaPods (SPM preferred)
Does not use deprecated completion handler pattern
L3 · Security(2/2)
No hardcoded client ID in Swift source files (ok in Auth0.plist)
No hardcoded domain in Swift source files (ok in Auth0.plist)
L4 · Structural(2/2)
Uses CredentialsManager for token storage
Does the code properly handle login and logout flows with appropriate error handling? Does it update the UI state after successful authentication?
L5 · Version Correctness(2/2)
Uses async/await webAuth().start() syntax (not completion handlers)
Does the code use modern Swift async/await patterns with the Auth0.swift SDK? Specifically: does it use try await webAuth().start() and CredentialsManager, and configure via Auth0.plist rather than hardcoded strings?
Holistic(0/1)
Does the solution correctly integrate Auth0 into a Swift iOS app with webAuth() login/logout, credential management, and proper SwiftUI state handling?
93
Gemini 3.1 Pro
Express API
70%7/10
97%+27%
18/19$0.53
54s
With Auth0 Tools·97%·Tool Calls 16
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.3/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
23.1/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
97/100 ptsA
Grader Results (18/19 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(3/3)
Uses express-oauth2-jwt-bearer SDK
Uses requiredScopes() for scope-based route protection
Accesses JWT data via req.auth
L2 · Hallucination Detection(4/4)
No express-openid-connect (that is for web apps, not APIs)
No passport middleware (not needed with express-oauth2-jwt-bearer)
No manual JWT verification with jsonwebtoken (use SDK)
No SPA SDK used in server-side API
L3 · Security(2/2)
No hardcoded issuer domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
L4 · Structural(3/4)
Wrote Auth0 config to .env file
GET /api/messages protected with read:messages scope
POST /api/messages protected with write:messages scope
Does the API correctly protect all three routes? GET /api/messages requires read:messages, POST /api/messages requires write:messages, GET /api/profile returns user info from req.auth.payload. Missing or invalid tokens should yield 401, insufficient scope should yield 403.
L5 · Version Correctness(4/4)
No manual Authorization header parsing (SDK handles extraction)
No manual jwt.verify calls (SDK handles verification)
No req.user (express-oauth2-jwt-bearer uses req.auth, not req.user)
Does the solution use current express-oauth2-jwt-bearer patterns? Specifically: does it apply the auth() middleware to protect routes, use requiredScopes() for scope checks (not manual payload inspection), and access token data via req.auth.payload (not req.user or manually decoded tokens)? Judge only from the source code; the issuer and audience may be supplied via environment variables (ISSUER_BASE_URL / AUDIENCE), so do not assume the contents of any .env file.
Holistic(1/1)
Does the solution correctly protect an Express.js API using express-oauth2-jwt-bearer? It should apply the auth() middleware (issuer and audience may come from ISSUER_BASE_URL / AUDIENCE environment variables — judge only from the source code and do not assume the contents of any .env file), protect GET /api/messages with read:messages scope, protect POST /api/messages with write:messages scope, and return user profile info from req.auth.payload at GET /api/profile.
94
Claude Haiku 4.5
Angular
92%11/12
97%+5%
18/18$0.55
6m 6s
With Auth0 Tools·97%·Tool Calls 40
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
10.8/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.7/12 pts
Error Recovery
When something broke, did the agent fix it?
5.6/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
97/100 ptsA
Grader Results (18/18 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses @auth0/auth0-angular SDK
Injects AuthService for authentication operations
Implements loginWithRedirect
Implements logout
Uses isAuthenticated$ observable for auth state
Uses user$ observable to display user profile
L2 · Hallucination Detection(3/3)
Does not use React SDK in Angular app
Does not use Vue SDK in Angular app
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Auth0 configured via provideAuth0()
Route protected with AuthGuard or authGuardFn
Uses getAccessTokenSilently or httpInterceptor for authenticated API calls
L5 · Version Correctness(2/2)
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Angular SPA using @auth0/auth0-angular, with provideAuth0 setup, AuthService for login/logout, user profile display, route protection via a guard, and authenticated API calls (either by configuring authHttpInterceptorFn with an allowedList matching the API base URL, or by calling getAccessTokenSilently to retrieve tokens manually)?
95
Claude Haiku 4.5
Vue
92%12/13
97%+5%
20/20$0.64
6m 16s
With Auth0 Tools·97%·Tool Calls 36
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
10.8/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.7/12 pts
Error Recovery
When something broke, did the agent fix it?
5.6/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
97/100 ptsA
Grader Results (20/20 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(7/7)
Uses @auth0/auth0-vue SDK
Sets up Auth0 plugin with createAuth0
Uses useAuth0 composable
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile information
L2 · Hallucination Detection(3/3)
Does not use React SDK in Vue app
No hallucinated @auth0/vue3-auth0 package
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Plugin installed via app.use(createAuth0(...))
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated? A correct implementation should not render auth-dependent UI while isLoading is true.
L5 · Version Correctness(3/3)
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Uses clientId (not deprecated client_id) in createAuth0 config
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Vue 3 SPA with the @auth0/auth0-vue plugin, useAuth0 composable, login, logout, user profile display, route protection, and getAccessTokenSilently to make authenticated API calls?
96
Gemini 3.5 Flash
Express
73%11/15
97%+24%
23/24$0.88
2m 25s
With Auth0 Tools·97%·Tool Calls 18
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
10.7/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
23.5/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
97/100 ptsA
Grader Results (23/24 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(7/7)
Uses express-openid-connect SDK
Configures issuerBaseURL
Configures clientID
Configures baseURL
Configures session secret
Uses req.oidc to access OIDC context
Accesses user profile via req.oidc.user
L2 · Hallucination Detection(4/4)
No hallucinated @auth0/auth0-spa-js (wrong SDK for server-side app)
No express-oauth2-jwt-bearer (that is for APIs, not web apps)
Does not use Next.js SDK in Express app
No passport middleware (not needed with express-openid-connect)
L3 · Security(3/3)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
No hardcoded Auth0 domain in source files (ok in .env)
L4 · Structural(4/5)
Wrote Auth0 credentials to .env file
auth middleware registered with app.use
Uses requiresAuth() to protect the /profile route
Accesses access token via req.oidc.accessToken
Does the app correctly register the auth() middleware, protect the /profile route with requiresAuth(), display user profile information, and include a route that calls an external API using the access token?
L5 · Version Correctness(3/3)
Uses authorizationParams to pass audience and scope
authorizationParams contains audience 'https://api.barkbook.com'
Does the solution use current express-openid-connect patterns? Specifically: does it use issuerBaseURL (not AUTH0_DOMAIN or domain directly), configure audience via authorizationParams (not as a top-level config key), and use response_type: "code" for the authorization code flow?
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Express web app using express-openid-connect? It should configure the auth() middleware, protect the /profile route with requiresAuth(), display the logged-in user profile, and use the access token to call an external API with audience https://api.barkbook.com.
97
Claude Sonnet 5
Nuxt
50%10/20
96%+46%
32/32$0.89
4m 15s
With Auth0 Tools·96%·Tool Calls 26
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
10.8/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.1/12 pts
Error Recovery
When something broke, did the agent fix it?
5.6/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
96/100 ptsA
Grader Results (32/32 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(10/10)
Uses @auth0/auth0-nuxt SDK
Registers module in nuxt.config
Configures runtimeConfig with Auth0 settings
Configures NUXT_AUTH0_DOMAIN env var
Configures NUXT_AUTH0_CLIENT_ID env var
Configures NUXT_AUTH0_CLIENT_SECRET env var
Configures NUXT_AUTH0_SESSION_SECRET env var
Uses useUser() composable to access the authenticated user
Uses /auth/login route to initiate login
Uses /auth/logout route to log out
L2 · Hallucination Detection(4/4)
Does not use the Vue SDK in a Nuxt app
Does not use the React SDK in a Nuxt app
Does not use loginWithRedirect (SPA API not applicable to Nuxt)
Does not use getAccessTokenSilently (Vue/React SPA API, not used in Nuxt)
L3 · Security(5/5)
No hardcoded client ID in source files (ok in .env)
No hardcoded client secret in source files (ok in .env)
No hardcoded Auth0 domain in source files (ok in .env)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(7/7)
Wrote Auth0 credentials to .env file
Module correctly registered in the modules array
sessionSecret configured for session encryption
clientSecret configured (Regular Web Application)
appBaseUrl configured in runtimeConfig
definePageMeta used for page-level middleware
Is there a protected /profile route that uses a Nuxt route middleware (via definePageMeta) to check authentication with useUser() and redirect unauthenticated users to /auth/login?
L5 · Version Correctness(4/4)
Audience configured in runtimeConfig for API access
Audience set to 'https://api.playground.com'
Uses getAccessToken() server-side to retrieve access token
Uses useAuth0(event) server-side composable
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Nuxt application using the @auth0/auth0-nuxt module, with proper configuration in nuxt.config.ts (domain, clientId, clientSecret, sessionSecret, appBaseUrl, audience), login/logout via /auth/login and /auth/logout, user profile display with useUser(), a /profile route protected by a route middleware, and server-side getAccessToken() for authenticated API calls?
98
Gemini 3.5 Flash
Fastify API
46%6/13
97%+51%
19/20$1.24
1m 47s
With Auth0 Tools·97%·Tool Calls 30
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.2/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
23.2/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
97/100 ptsA
Grader Results (19/20 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses @auth0/auth0-fastify-api package
Imports fastifyAuth0Api plugin
Uses fastify.requireAuth() to protect routes
Uses preHandler to attach auth middleware
Configures domain
Configures audience
L2 · Hallucination Detection(4/4)
No express-oauth2-jwt-bearer (wrong SDK for Fastify)
No passport middleware (not needed with @auth0/auth0-fastify-api)
No jsonwebtoken (manual JWT verification not needed)
No manual jwt.verify() calls
L3 · Security(2/2)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
L4 · Structural(4/5)
Wrote Auth0 config to .env file
Auth0 API plugin registered with fastify.register()
Route protected with preHandler: fastify.requireAuth()
read:messages scope checked on /api/messages route
Does the app correctly register the @auth0/auth0-fastify-api plugin, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token? Note: @auth0/auth0-fastify-api exposes the verified token claims on request.user (e.g. request.user.sub) — this is the correct API for this plugin, not request.auth.
L5 · Version Correctness(1/1)
Does the solution use current @auth0/auth0-fastify-api patterns? Specifically: does it register the plugin via fastify.register(), use preHandler: fastify.requireAuth() for route protection (not a decorator), access token claims via request.user, and read credentials from environment variables (not hardcoded)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Fastify API using @auth0/auth0-fastify-api? It should register the plugin with domain and audience from environment variables, protect the /api/messages route using fastify.requireAuth() with read:messages scope (as string or array) check using preHandler, and protect the /api/private route using fastify.requireAuth() requiring any valid access token.
99
GPT-5.6 Terra
Expo
82%9/11
97%+15%
17/17$1.33
1m 45s
With Auth0 Tools·97%·Tool Calls 18
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
9.3/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
97/100 ptsA
Grader Results (17/17 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses the react-native-auth0 SDK
Wraps app with Auth0Provider
Uses the useAuth0 hook
Implements login via authorize()
Implements logout via clearSession()
Configures a customScheme for the callback URL
L2 · Hallucination Detection(3/3)
No web React SDK (correct package is react-native-auth0)
No browser SPA SDK in a native Expo app
Does not fall back to expo-auth-session instead of the Auth0 SDK
L3 · Security(1/1)
No hardcoded client ID in source files (ok in app.json)
L4 · Structural(4/4)
Registers the react-native-auth0 Expo config plugin with a customScheme in app.json
Auth0Provider configured with a domain prop
Does the code handle the loading state (isLoading from useAuth0) before rendering auth-dependent UI? A correct implementation should not render login/logout UI while isLoading is true.
Does the code wire login and logout correctly for react-native-auth0 on Expo? Specifically: authorize() for login and clearSession() for logout, each passing { customScheme: ... } as the second argument, matching the scheme configured in the app.json plugin.
L5 · Version Correctness(1/1)
Does the code use the current react-native-auth0 v5 Expo patterns? Specifically: the Auth0Provider + useAuth0 hook API (not the legacy imperative Auth0 client class), and the react-native-auth0 Expo config plugin in app.json (not manual native iOS/Android edits)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Expo app with the react-native-auth0 SDK: Auth0Provider wrapping the app, the useAuth0 hook, authorize()/clearSession() login and logout with customScheme, the Expo config plugin registered in app.json, and user profile display guarded by isLoading?
100
Gemini 3.5 Flash
Express API
70%7/10
97%+27%
18/19$1.51
2m 49s
With Auth0 Tools·97%·Tool Calls 30
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.2/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
23.1/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
97/100 ptsA
Grader Results (18/19 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(3/3)
Uses express-oauth2-jwt-bearer SDK
Uses requiredScopes() for scope-based route protection
Accesses JWT data via req.auth
L2 · Hallucination Detection(4/4)
No express-openid-connect (that is for web apps, not APIs)
No passport middleware (not needed with express-oauth2-jwt-bearer)
No manual JWT verification with jsonwebtoken (use SDK)
No SPA SDK used in server-side API
L3 · Security(2/2)
No hardcoded issuer domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
L4 · Structural(3/4)
Wrote Auth0 config to .env file
GET /api/messages protected with read:messages scope
POST /api/messages protected with write:messages scope
Does the API correctly protect all three routes? GET /api/messages requires read:messages, POST /api/messages requires write:messages, GET /api/profile returns user info from req.auth.payload. Missing or invalid tokens should yield 401, insufficient scope should yield 403.
L5 · Version Correctness(4/4)
No manual Authorization header parsing (SDK handles extraction)
No manual jwt.verify calls (SDK handles verification)
No req.user (express-oauth2-jwt-bearer uses req.auth, not req.user)
Does the solution use current express-oauth2-jwt-bearer patterns? Specifically: does it apply the auth() middleware to protect routes, use requiredScopes() for scope checks (not manual payload inspection), and access token data via req.auth.payload (not req.user or manually decoded tokens)? Judge only from the source code; the issuer and audience may be supplied via environment variables (ISSUER_BASE_URL / AUDIENCE), so do not assume the contents of any .env file.
Holistic(1/1)
Does the solution correctly protect an Express.js API using express-oauth2-jwt-bearer? It should apply the auth() middleware (issuer and audience may come from ISSUER_BASE_URL / AUDIENCE environment variables — judge only from the source code and do not assume the contents of any .env file), protect GET /api/messages with read:messages scope, protect POST /api/messages with write:messages scope, and return user profile info from req.auth.payload at GET /api/profile.
101
Claude Opus 5
React Native
85%11/13
97%+12%
21/21$1.63
4m 13s
With Auth0 Tools·97%·Tool Calls 27
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
10.8/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.1/12 pts
Error Recovery
When something broke, did the agent fix it?
5.6/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
97/100 ptsA
Grader Results (21/21 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(5/5)
Uses the react-native-auth0 SDK
Wraps app with Auth0Provider
Uses the useAuth0 hook
Implements login via authorize()
Implements logout via clearSession()
L2 · Hallucination Detection(5/5)
No web React SDK (correct package is react-native-auth0)
No browser SPA SDK in a native React Native app
Does not use Expo AuthSession in a bare React Native app
Does not fall back to the third-party react-native-app-auth library
Does not register the Expo config plugin in a bare React Native app
L3 · Security(2/2)
No hardcoded client ID in source files (ok in config)
Does not store tokens in insecure AsyncStorage
L4 · Structural(6/6)
Auth0Provider configured with a domain prop
android/app/build.gradle sets the auth0Domain manifestPlaceholder to the real tenant domain
android/app/build.gradle sets the auth0Scheme manifestPlaceholder to <applicationId>.auth0
iOS Info.plist registers the bundle-identifier callback URL scheme
Does the code handle the loading state (isLoading from useAuth0) before rendering auth-dependent UI? A correct implementation should not render login/logout UI while isLoading is true.
Are BOTH native platforms configured for the Auth0 callback in this bare React Native app? Android needs auth0Domain and auth0Scheme manifestPlaceholders in android/app/build.gradle (the RedirectActivity is merged in by the SDK manifest, so declaring it manually is also acceptable), and iOS needs a CFBundleURLTypes entry in ios/scaffold/Info.plist. Answer no if only one platform was configured.
L5 · Version Correctness(1/1)
Does the code use the current react-native-auth0 v5 patterns? Specifically: the Auth0Provider + useAuth0 hook API rather than the legacy imperative `new Auth0({domain, clientId})` client class, and `user` from the hook rather than a manual getCredentials()/userInfo() call to read the profile.
Holistic(1/1)
Does the solution correctly integrate Auth0 into a bare (non-Expo) React Native app using the react-native-auth0 SDK: Auth0Provider wrapping the app with domain and clientId, the useAuth0 hook, authorize() for login and clearSession() for logout, native callback configuration for both Android (manifestPlaceholders) and iOS (CFBundleURLTypes), and the user profile displayed only after isLoading is false?
102
Claude Opus 5
Angular
92%11/12
97%+5%
18/18$1.69
6m 15s
With Auth0 Tools·97%·Tool Calls 30
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
10.8/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.2/12 pts
Error Recovery
When something broke, did the agent fix it?
5.6/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
97/100 ptsA
Grader Results (18/18 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses @auth0/auth0-angular SDK
Injects AuthService for authentication operations
Implements loginWithRedirect
Implements logout
Uses isAuthenticated$ observable for auth state
Uses user$ observable to display user profile
L2 · Hallucination Detection(3/3)
Does not use React SDK in Angular app
Does not use Vue SDK in Angular app
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Auth0 configured via provideAuth0()
Route protected with AuthGuard or authGuardFn
Uses getAccessTokenSilently or httpInterceptor for authenticated API calls
L5 · Version Correctness(2/2)
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Angular SPA using @auth0/auth0-angular, with provideAuth0 setup, AuthService for login/logout, user profile display, route protection via a guard, and authenticated API calls (either by configuring authHttpInterceptorFn with an allowedList matching the API base URL, or by calling getAccessTokenSilently to retrieve tokens manually)?
103
GPT-5.6 Sol
Express
80%12/15
97%+17%
23/24$1.96
2m 32s
With Auth0 Tools·97%·Tool Calls 17
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
10.6/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
23.5/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
97/100 ptsA
Grader Results (23/24 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(7/7)
Uses express-openid-connect SDK
Configures issuerBaseURL
Configures clientID
Configures baseURL
Configures session secret
Uses req.oidc to access OIDC context
Accesses user profile via req.oidc.user
L2 · Hallucination Detection(4/4)
No hallucinated @auth0/auth0-spa-js (wrong SDK for server-side app)
No express-oauth2-jwt-bearer (that is for APIs, not web apps)
Does not use Next.js SDK in Express app
No passport middleware (not needed with express-openid-connect)
L3 · Security(3/3)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
No hardcoded Auth0 domain in source files (ok in .env)
L4 · Structural(5/5)
Wrote Auth0 credentials to .env file
auth middleware registered with app.use
Uses requiresAuth() to protect the /profile route
Accesses access token via req.oidc.accessToken
Does the app correctly register the auth() middleware, protect the /profile route with requiresAuth(), display user profile information, and include a route that calls an external API using the access token?
L5 · Version Correctness(3/3)
Uses authorizationParams to pass audience and scope
authorizationParams contains audience 'https://api.barkbook.com'
Does the solution use current express-openid-connect patterns? Specifically: does it use issuerBaseURL (not AUTH0_DOMAIN or domain directly), configure audience via authorizationParams (not as a top-level config key), and use response_type: "code" for the authorization code flow?
Holistic(0/1)
Does the solution correctly integrate Auth0 into an Express web app using express-openid-connect? It should configure the auth() middleware, protect the /profile route with requiresAuth(), display the logged-in user profile, and use the access token to call an external API with audience https://api.barkbook.com.
104
GPT-5.6 Sol
Expo
82%9/11
97%+15%
17/17$2.27
2m 48s
With Auth0 Tools·97%·Tool Calls 18
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
9.3/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
97/100 ptsA
Grader Results (17/17 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses the react-native-auth0 SDK
Wraps app with Auth0Provider
Uses the useAuth0 hook
Implements login via authorize()
Implements logout via clearSession()
Configures a customScheme for the callback URL
L2 · Hallucination Detection(3/3)
No web React SDK (correct package is react-native-auth0)
No browser SPA SDK in a native Expo app
Does not fall back to expo-auth-session instead of the Auth0 SDK
L3 · Security(1/1)
No hardcoded client ID in source files (ok in app.json)
L4 · Structural(4/4)
Registers the react-native-auth0 Expo config plugin with a customScheme in app.json
Auth0Provider configured with a domain prop
Does the code handle the loading state (isLoading from useAuth0) before rendering auth-dependent UI? A correct implementation should not render login/logout UI while isLoading is true.
Does the code wire login and logout correctly for react-native-auth0 on Expo? Specifically: authorize() for login and clearSession() for logout, each passing { customScheme: ... } as the second argument, matching the scheme configured in the app.json plugin.
L5 · Version Correctness(1/1)
Does the code use the current react-native-auth0 v5 Expo patterns? Specifically: the Auth0Provider + useAuth0 hook API (not the legacy imperative Auth0 client class), and the react-native-auth0 Expo config plugin in app.json (not manual native iOS/Android edits)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Expo app with the react-native-auth0 SDK: Auth0Provider wrapping the app, the useAuth0 hook, authorize()/clearSession() login and logout with customScheme, the Expo config plugin registered in app.json, and user profile display guarded by isLoading?
105
Gemini 3.5 Flash
React Native
85%11/13
97%+12%
20/21$2.65
3m 29s
With Auth0 Tools·97%·Tool Calls 39
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
10.8/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
23.2/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
97/100 ptsA
Grader Results (20/21 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(5/5)
Uses the react-native-auth0 SDK
Wraps app with Auth0Provider
Uses the useAuth0 hook
Implements login via authorize()
Implements logout via clearSession()
L2 · Hallucination Detection(5/5)
No web React SDK (correct package is react-native-auth0)
No browser SPA SDK in a native React Native app
Does not use Expo AuthSession in a bare React Native app
Does not fall back to the third-party react-native-app-auth library
Does not register the Expo config plugin in a bare React Native app
L3 · Security(2/2)
No hardcoded client ID in source files (ok in config)
Does not store tokens in insecure AsyncStorage
L4 · Structural(6/6)
Auth0Provider configured with a domain prop
android/app/build.gradle sets the auth0Domain manifestPlaceholder to the real tenant domain
android/app/build.gradle sets the auth0Scheme manifestPlaceholder to <applicationId>.auth0
iOS Info.plist registers the bundle-identifier callback URL scheme
Does the code handle the loading state (isLoading from useAuth0) before rendering auth-dependent UI? A correct implementation should not render login/logout UI while isLoading is true.
Are BOTH native platforms configured for the Auth0 callback in this bare React Native app? Android needs auth0Domain and auth0Scheme manifestPlaceholders in android/app/build.gradle (the RedirectActivity is merged in by the SDK manifest, so declaring it manually is also acceptable), and iOS needs a CFBundleURLTypes entry in ios/scaffold/Info.plist. Answer no if only one platform was configured.
L5 · Version Correctness(1/1)
Does the code use the current react-native-auth0 v5 patterns? Specifically: the Auth0Provider + useAuth0 hook API rather than the legacy imperative `new Auth0({domain, clientId})` client class, and `user` from the hook rather than a manual getCredentials()/userInfo() call to read the profile.
Holistic(0/1)
Does the solution correctly integrate Auth0 into a bare (non-Expo) React Native app using the react-native-auth0 SDK: Auth0Provider wrapping the app with domain and clientId, the useAuth0 hook, authorize() for login and clearSession() for logout, native callback configuration for both Android (manifestPlaceholders) and iOS (CFBundleURLTypes), and the user profile displayed only after isLoading is false?
106
Claude Haiku 4.5
Expo
73%8/11
96%+23%
17/17$0.34
2m 59s
With Auth0 Tools·96%·Tool Calls 29
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
10.8/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
10.8/12 pts
Error Recovery
When something broke, did the agent fix it?
5.6/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
96/100 ptsA
Grader Results (17/17 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses the react-native-auth0 SDK
Wraps app with Auth0Provider
Uses the useAuth0 hook
Implements login via authorize()
Implements logout via clearSession()
Configures a customScheme for the callback URL
L2 · Hallucination Detection(3/3)
No web React SDK (correct package is react-native-auth0)
No browser SPA SDK in a native Expo app
Does not fall back to expo-auth-session instead of the Auth0 SDK
L3 · Security(1/1)
No hardcoded client ID in source files (ok in app.json)
L4 · Structural(4/4)
Registers the react-native-auth0 Expo config plugin with a customScheme in app.json
Auth0Provider configured with a domain prop
Does the code handle the loading state (isLoading from useAuth0) before rendering auth-dependent UI? A correct implementation should not render login/logout UI while isLoading is true.
Does the code wire login and logout correctly for react-native-auth0 on Expo? Specifically: authorize() for login and clearSession() for logout, each passing { customScheme: ... } as the second argument, matching the scheme configured in the app.json plugin.
L5 · Version Correctness(1/1)
Does the code use the current react-native-auth0 v5 Expo patterns? Specifically: the Auth0Provider + useAuth0 hook API (not the legacy imperative Auth0 client class), and the react-native-auth0 Expo config plugin in app.json (not manual native iOS/Android edits)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Expo app with the react-native-auth0 SDK: Auth0Provider wrapping the app, the useAuth0 hook, authorize()/clearSession() login and logout with customScheme, the Expo config plugin registered in app.json, and user profile display guarded by isLoading?
107
Claude Opus 5
Express
80%12/15
96%+16%
24/24$0.63
1m 47s
With Auth0 Tools·96%·Tool Calls 13
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
10.8/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
10.2/12 pts
Error Recovery
When something broke, did the agent fix it?
5.6/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
96/100 ptsA
Grader Results (24/24 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(7/7)
Uses express-openid-connect SDK
Configures issuerBaseURL
Configures clientID
Configures baseURL
Configures session secret
Uses req.oidc to access OIDC context
Accesses user profile via req.oidc.user
L2 · Hallucination Detection(4/4)
No hallucinated @auth0/auth0-spa-js (wrong SDK for server-side app)
No express-oauth2-jwt-bearer (that is for APIs, not web apps)
Does not use Next.js SDK in Express app
No passport middleware (not needed with express-openid-connect)
L3 · Security(3/3)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
No hardcoded Auth0 domain in source files (ok in .env)
L4 · Structural(5/5)
Wrote Auth0 credentials to .env file
auth middleware registered with app.use
Uses requiresAuth() to protect the /profile route
Accesses access token via req.oidc.accessToken
Does the app correctly register the auth() middleware, protect the /profile route with requiresAuth(), display user profile information, and include a route that calls an external API using the access token?
L5 · Version Correctness(3/3)
Uses authorizationParams to pass audience and scope
authorizationParams contains audience 'https://api.barkbook.com'
Does the solution use current express-openid-connect patterns? Specifically: does it use issuerBaseURL (not AUTH0_DOMAIN or domain directly), configure audience via authorizationParams (not as a top-level config key), and use response_type: "code" for the authorization code flow?
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Express web app using express-openid-connect? It should configure the auth() middleware, protect the /profile route with requiresAuth(), display the logged-in user profile, and use the access token to call an external API with audience https://api.barkbook.com.
108
Claude Opus 5
Fastify API
69%9/13
96%+27%
20/20$0.71
2m 9s
With Auth0 Tools·96%·Tool Calls 19
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
10.8/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
10.7/12 pts
Error Recovery
When something broke, did the agent fix it?
5.6/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
96/100 ptsA
Grader Results (20/20 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses @auth0/auth0-fastify-api package
Imports fastifyAuth0Api plugin
Uses fastify.requireAuth() to protect routes
Uses preHandler to attach auth middleware
Configures domain
Configures audience
L2 · Hallucination Detection(4/4)
No express-oauth2-jwt-bearer (wrong SDK for Fastify)
No passport middleware (not needed with @auth0/auth0-fastify-api)
No jsonwebtoken (manual JWT verification not needed)
No manual jwt.verify() calls
L3 · Security(2/2)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
L4 · Structural(5/5)
Wrote Auth0 config to .env file
Auth0 API plugin registered with fastify.register()
Route protected with preHandler: fastify.requireAuth()
read:messages scope checked on /api/messages route
Does the app correctly register the @auth0/auth0-fastify-api plugin, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token? Note: @auth0/auth0-fastify-api exposes the verified token claims on request.user (e.g. request.user.sub) — this is the correct API for this plugin, not request.auth.
L5 · Version Correctness(1/1)
Does the solution use current @auth0/auth0-fastify-api patterns? Specifically: does it register the plugin via fastify.register(), use preHandler: fastify.requireAuth() for route protection (not a decorator), access token claims via request.user, and read credentials from environment variables (not hardcoded)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Fastify API using @auth0/auth0-fastify-api? It should register the plugin with domain and audience from environment variables, protect the /api/messages route using fastify.requireAuth() with read:messages scope (as string or array) check using preHandler, and protect the /api/private route using fastify.requireAuth() requiring any valid access token.
109
Claude Opus 5
Express API
70%7/10
96%+26%
19/19$0.87
1m 46s
With Auth0 Tools·96%·Tool Calls 12
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
10.8/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11/12 pts
Error Recovery
When something broke, did the agent fix it?
5.6/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
96/100 ptsA
Grader Results (19/19 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(3/3)
Uses express-oauth2-jwt-bearer SDK
Uses requiredScopes() for scope-based route protection
Accesses JWT data via req.auth
L2 · Hallucination Detection(4/4)
No express-openid-connect (that is for web apps, not APIs)
No passport middleware (not needed with express-oauth2-jwt-bearer)
No manual JWT verification with jsonwebtoken (use SDK)
No SPA SDK used in server-side API
L3 · Security(2/2)
No hardcoded issuer domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
L4 · Structural(4/4)
Wrote Auth0 config to .env file
GET /api/messages protected with read:messages scope
POST /api/messages protected with write:messages scope
Does the API correctly protect all three routes? GET /api/messages requires read:messages, POST /api/messages requires write:messages, GET /api/profile returns user info from req.auth.payload. Missing or invalid tokens should yield 401, insufficient scope should yield 403.
L5 · Version Correctness(4/4)
No manual Authorization header parsing (SDK handles extraction)
No manual jwt.verify calls (SDK handles verification)
No req.user (express-oauth2-jwt-bearer uses req.auth, not req.user)
Does the solution use current express-oauth2-jwt-bearer patterns? Specifically: does it apply the auth() middleware to protect routes, use requiredScopes() for scope checks (not manual payload inspection), and access token data via req.auth.payload (not req.user or manually decoded tokens)? Judge only from the source code; the issuer and audience may be supplied via environment variables (ISSUER_BASE_URL / AUDIENCE), so do not assume the contents of any .env file.
Holistic(1/1)
Does the solution correctly protect an Express.js API using express-oauth2-jwt-bearer? It should apply the auth() middleware (issuer and audience may come from ISSUER_BASE_URL / AUDIENCE environment variables — judge only from the source code and do not assume the contents of any .env file), protect GET /api/messages with read:messages scope, protect POST /api/messages with write:messages scope, and return user profile info from req.auth.payload at GET /api/profile.
110
Gemini 3.5 Flash
Flask
29%4/14
96%+67%
24/26$0.96
1m 59s
With Auth0 Tools·96%·Tool Calls 20
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
10.8/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
22.4/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
96/100 ptsA
Grader Results (24/26 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses auth0-server-python package
Imports ServerClient class
Uses start_interactive_login method
Uses complete_interactive_login method
Uses get_user method to retrieve user info
Loads environment variables with python-dotenv
L2 · Hallucination Detection(4/4)
No Authlib (old approach, not the current SDK)
No python-jose (manual JWT parsing not needed)
No Flask-Login (not needed with auth0-server-python)
No Flask-Dance (wrong OAuth package)
L3 · Security(3/3)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
No hardcoded Auth0 domain in source files (ok in .env)
L4 · Structural(7/8)
Wrote Auth0 credentials to .env file
Implements /callback route
Implements /login route
Implements /logout route
Implements /profile route
Sets Flask app.secret_key for session management
Uses async route handlers
Does the app implement login, callback, profile, and logout routes using auth0-server-python, protect the /profile route so it requires login, and include a route that calls an external API using an access token?
L5 · Version Correctness(3/3)
Uses authorization_params for scope and audience configuration
authorization_params contains audience 'https://api.barkbook.com'
Does the solution use current auth0-server-python patterns? Specifically: does it import ServerClient from auth0_server_python, define concrete state and transaction store classes locally (subclassing StateStore/TransactionStore from the SDK), and configure authorization_params with audience and scope?
Holistic(0/1)
Does the solution correctly integrate Auth0 into a Flask web app using auth0-server-python? It should configure a ServerClient with credentials from env vars, implement login, callback, profile, and logout routes, protect the /profile route so it requires authentication, and use the access token to call an external API with audience https://api.barkbook.com.
111
Claude Sonnet 5
Expo
73%8/11
96%+23%
17/17$1.15
3m 31s
With Auth0 Tools·96%·Tool Calls 22
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
10.8/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
10.9/12 pts
Error Recovery
When something broke, did the agent fix it?
5.6/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
96/100 ptsA
Grader Results (17/17 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses the react-native-auth0 SDK
Wraps app with Auth0Provider
Uses the useAuth0 hook
Implements login via authorize()
Implements logout via clearSession()
Configures a customScheme for the callback URL
L2 · Hallucination Detection(3/3)
No web React SDK (correct package is react-native-auth0)
No browser SPA SDK in a native Expo app
Does not fall back to expo-auth-session instead of the Auth0 SDK
L3 · Security(1/1)
No hardcoded client ID in source files (ok in app.json)
L4 · Structural(4/4)
Registers the react-native-auth0 Expo config plugin with a customScheme in app.json
Auth0Provider configured with a domain prop
Does the code handle the loading state (isLoading from useAuth0) before rendering auth-dependent UI? A correct implementation should not render login/logout UI while isLoading is true.
Does the code wire login and logout correctly for react-native-auth0 on Expo? Specifically: authorize() for login and clearSession() for logout, each passing { customScheme: ... } as the second argument, matching the scheme configured in the app.json plugin.
L5 · Version Correctness(1/1)
Does the code use the current react-native-auth0 v5 Expo patterns? Specifically: the Auth0Provider + useAuth0 hook API (not the legacy imperative Auth0 client class), and the react-native-auth0 Expo config plugin in app.json (not manual native iOS/Android edits)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Expo app with the react-native-auth0 SDK: Auth0Provider wrapping the app, the useAuth0 hook, authorize()/clearSession() login and logout with customScheme, the Expo config plugin registered in app.json, and user profile display guarded by isLoading?
112
Claude Haiku 4.5
Next.js
57%8/14
95%+38%
29/31$0.40
4m 41s
With Auth0 Tools·95%·Tool Calls 31
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
10.8/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.6/12 pts
Error Recovery
When something broke, did the agent fix it?
5.6/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
23/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
95/100 ptsA
Grader Results (29/31 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(8/8)
Uses @auth0/nextjs-auth0 SDK
Uses v4 server import path
Instantiates Auth0Client
Configures AUTH0_CLIENT_ID
Configures AUTH0_CLIENT_SECRET
Configures AUTH0_SECRET
Configures AUTH0_DOMAIN
Uses getSession for session retrieval
L2 · Hallucination Detection(3/3)
No hallucinated @auth0/nextjs-sdk package
No hallucinated next-auth package
Does not use SPA SDK in server app
L3 · Security(3/3)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
No hardcoded Auth0 domain in source files (ok in .env)
L4 · Structural(7/8)
Wrote Auth0 credentials to .env file
Middleware function is exported from middleware or proxy file
Uses auth0.middleware in middleware file
Dashboard page file exists
Uses /auth/login for login redirect
Uses auth0.getAccessToken() for server-side token retrieval
Requests an access token with audience https://api.playground.com
Does the code set up a working authentication flow with login, logout, and a callback route? Is there a protected /dashboard page that checks the user session and redirects unauthenticated users to log in? Note: Next.js 16 supports both middleware.ts (export function middleware) and proxy.ts (export function proxy) — both are valid.
L5 · Version Correctness(7/8)
Does not use v3 env var AUTH0_BASE_URL (v4 uses APP_BASE_URL)
Does not use v3 env var AUTH0_ISSUER_BASE_URL (removed in v4)
Does not use v3 handleAuth (v4 uses middleware)
Does not use v3 route prefix /api/auth/ (v4 uses /auth/)
Does not use v3 withPageAuthRequired (v4 uses proxy/middleware)
Does not use v3 withApiAuthRequired (removed in v4)
Does the solution correctly integrate Auth0 into a Next.js App Router app using Auth0Client from @auth0/nextjs-auth0/server, proxy or middleware-based auth routing, and getSession for server-side session access? It should NOT use the deprecated v3 patterns like handleAuth, withPageAuthRequired, or /api/auth/ routes. Note: Next.js 16 replaces middleware.ts with proxy.ts (export function proxy) — both are valid. There should also be a protected /dashboard page that checks the session and redirects unauthenticated users to log in.
Does the source code rely on the v4 SDK configuration convention — instantiating Auth0Client so it reads the standard AUTH0_DOMAIN / AUTH0_CLIENT_ID / AUTH0_CLIENT_SECRET / AUTH0_SECRET environment variables (or passing those as options) — and avoid referencing the deprecated v3 names AUTH0_BASE_URL or AUTH0_ISSUER_BASE_URL anywhere in source? Judge only from the source code; do not assume the contents of any .env file.
113
Gemini 3.5 Flash
Swift
80%8/10
94%+14%
12/14$0.57
1m 9s
With Auth0 Tools·94%·Tool Calls 12
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
19.5/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
94/100 ptsA
Grader Results (12/14 passed)
L1 · Positive Presence(4/4)
Uses Auth0 SDK
Imports Auth0 module
Uses webAuth() for login
Uses clearSession for logout
L2 · Hallucination Detection(3/3)
No hallucinated Auth0SDK package name (correct package is Auth0)
Does not use CocoaPods (SPM preferred)
Does not use deprecated completion handler pattern
L3 · Security(2/2)
No hardcoded client ID in Swift source files (ok in Auth0.plist)
No hardcoded domain in Swift source files (ok in Auth0.plist)
L4 · Structural(1/2)
Uses CredentialsManager for token storage
Does the code properly handle login and logout flows with appropriate error handling? Does it update the UI state after successful authentication?
L5 · Version Correctness(2/2)
Uses async/await webAuth().start() syntax (not completion handlers)
Does the code use modern Swift async/await patterns with the Auth0.swift SDK? Specifically: does it use try await webAuth().start() and CredentialsManager, and configure via Auth0.plist rather than hardcoded strings?
Holistic(0/1)
Does the solution correctly integrate Auth0 into a Swift iOS app with webAuth() login/logout, credential management, and proper SwiftUI state handling?
114
Claude Haiku 4.5
Flask
43%6/14
94%+51%
24/26$0.22
3m 4s
With Auth0 Tools·94%·Tool Calls 24
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
10.8/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.5/12 pts
Error Recovery
When something broke, did the agent fix it?
5.6/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
22.4/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
94/100 ptsA
Grader Results (24/26 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses auth0-server-python package
Imports ServerClient class
Uses start_interactive_login method
Uses complete_interactive_login method
Uses get_user method to retrieve user info
Loads environment variables with python-dotenv
L2 · Hallucination Detection(4/4)
No Authlib (old approach, not the current SDK)
No python-jose (manual JWT parsing not needed)
No Flask-Login (not needed with auth0-server-python)
No Flask-Dance (wrong OAuth package)
L3 · Security(3/3)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
No hardcoded Auth0 domain in source files (ok in .env)
L4 · Structural(8/8)
Wrote Auth0 credentials to .env file
Implements /callback route
Implements /login route
Implements /logout route
Implements /profile route
Sets Flask app.secret_key for session management
Uses async route handlers
Does the app implement login, callback, profile, and logout routes using auth0-server-python, protect the /profile route so it requires login, and include a route that calls an external API using an access token?
L5 · Version Correctness(2/3)
Uses authorization_params for scope and audience configuration
authorization_params contains audience 'https://api.barkbook.com'
Does the solution use current auth0-server-python patterns? Specifically: does it import ServerClient from auth0_server_python, define concrete state and transaction store classes locally (subclassing StateStore/TransactionStore from the SDK), and configure authorization_params with audience and scope?
Holistic(0/1)
Does the solution correctly integrate Auth0 into a Flask web app using auth0-server-python? It should configure a ServerClient with credentials from env vars, implement login, callback, profile, and logout routes, protect the /profile route so it requires authentication, and use the access token to call an external API with audience https://api.barkbook.com.
115
GPT-5.6 Sol
Swift
70%7/10
94%+24%
13/14$2.33
2m 34s
With Auth0 Tools·94%·Tool Calls 21
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
8.6/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
22.2/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
94/100 ptsA
Grader Results (13/14 passed)
L1 · Positive Presence(4/4)
Uses Auth0 SDK
Imports Auth0 module
Uses webAuth() for login
Uses clearSession for logout
L2 · Hallucination Detection(3/3)
No hallucinated Auth0SDK package name (correct package is Auth0)
Does not use CocoaPods (SPM preferred)
Does not use deprecated completion handler pattern
L3 · Security(2/2)
No hardcoded client ID in Swift source files (ok in Auth0.plist)
No hardcoded domain in Swift source files (ok in Auth0.plist)
L4 · Structural(2/2)
Uses CredentialsManager for token storage
Does the code properly handle login and logout flows with appropriate error handling? Does it update the UI state after successful authentication?
L5 · Version Correctness(2/2)
Uses async/await webAuth().start() syntax (not completion handlers)
Does the code use modern Swift async/await patterns with the Auth0.swift SDK? Specifically: does it use try await webAuth().start() and CredentialsManager, and configure via Auth0.plist rather than hardcoded strings?
Holistic(0/1)
Does the solution correctly integrate Auth0 into a Swift iOS app with webAuth() login/logout, credential management, and proper SwiftUI state handling?
116
Claude Haiku 4.5
Nuxt
45%9/20
93%+48%
28/32$0.74
7m 24s
With Auth0 Tools·93%·Tool Calls 36
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
12/12 pts
Error Recovery
When something broke, did the agent fix it?
7/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
23.9/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
4/10 pts
Total weighted score
93/100 ptsA
Grader Results (28/32 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(10/10)
Uses @auth0/auth0-nuxt SDK
Registers module in nuxt.config
Configures runtimeConfig with Auth0 settings
Configures NUXT_AUTH0_DOMAIN env var
Configures NUXT_AUTH0_CLIENT_ID env var
Configures NUXT_AUTH0_CLIENT_SECRET env var
Configures NUXT_AUTH0_SESSION_SECRET env var
Uses useUser() composable to access the authenticated user
Uses /auth/login route to initiate login
Uses /auth/logout route to log out
L2 · Hallucination Detection(4/4)
Does not use the Vue SDK in a Nuxt app
Does not use the React SDK in a Nuxt app
Does not use loginWithRedirect (SPA API not applicable to Nuxt)
Does not use getAccessTokenSilently (Vue/React SPA API, not used in Nuxt)
L3 · Security(2/5)
No hardcoded client ID in source files (ok in .env)
No hardcoded client secret in source files (ok in .env)
No hardcoded Auth0 domain in source files (ok in .env)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(7/7)
Wrote Auth0 credentials to .env file
Module correctly registered in the modules array
sessionSecret configured for session encryption
clientSecret configured (Regular Web Application)
appBaseUrl configured in runtimeConfig
definePageMeta used for page-level middleware
Is there a protected /profile route that uses a Nuxt route middleware (via definePageMeta) to check authentication with useUser() and redirect unauthenticated users to /auth/login?
L5 · Version Correctness(4/4)
Audience configured in runtimeConfig for API access
Audience set to 'https://api.playground.com'
Uses getAccessToken() server-side to retrieve access token
Uses useAuth0(event) server-side composable
Holistic(0/1)
Does the solution correctly integrate Auth0 into a Nuxt application using the @auth0/auth0-nuxt module, with proper configuration in nuxt.config.ts (domain, clientId, clientSecret, sessionSecret, appBaseUrl, audience), login/logout via /auth/login and /auth/logout, user profile display with useUser(), a /profile route protected by a route middleware, and server-side getAccessToken() for authenticated API calls?
117
Claude Haiku 4.5
React Native
85%11/13
92%+7%
20/21$0.47
3m 24s
With Auth0 Tools·92%·Tool Calls 30
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
10.8/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
11.2/12 pts
Error Recovery
When something broke, did the agent fix it?
5.6/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
5/10 pts
Total weighted score
92/100 ptsA
Grader Results (20/21 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(5/5)
Uses the react-native-auth0 SDK
Wraps app with Auth0Provider
Uses the useAuth0 hook
Implements login via authorize()
Implements logout via clearSession()
L2 · Hallucination Detection(5/5)
No web React SDK (correct package is react-native-auth0)
No browser SPA SDK in a native React Native app
Does not use Expo AuthSession in a bare React Native app
Does not fall back to the third-party react-native-app-auth library
Does not register the Expo config plugin in a bare React Native app
L3 · Security(1/2)
No hardcoded client ID in source files (ok in config)
Does not store tokens in insecure AsyncStorage
L4 · Structural(6/6)
Auth0Provider configured with a domain prop
android/app/build.gradle sets the auth0Domain manifestPlaceholder to the real tenant domain
android/app/build.gradle sets the auth0Scheme manifestPlaceholder to <applicationId>.auth0
iOS Info.plist registers the bundle-identifier callback URL scheme
Does the code handle the loading state (isLoading from useAuth0) before rendering auth-dependent UI? A correct implementation should not render login/logout UI while isLoading is true.
Are BOTH native platforms configured for the Auth0 callback in this bare React Native app? Android needs auth0Domain and auth0Scheme manifestPlaceholders in android/app/build.gradle (the RedirectActivity is merged in by the SDK manifest, so declaring it manually is also acceptable), and iOS needs a CFBundleURLTypes entry in ios/scaffold/Info.plist. Answer no if only one platform was configured.
L5 · Version Correctness(1/1)
Does the code use the current react-native-auth0 v5 patterns? Specifically: the Auth0Provider + useAuth0 hook API rather than the legacy imperative `new Auth0({domain, clientId})` client class, and `user` from the hook rather than a manual getCredentials()/userInfo() call to read the profile.
Holistic(1/1)
Does the solution correctly integrate Auth0 into a bare (non-Expo) React Native app using the react-native-auth0 SDK: Auth0Provider wrapping the app with domain and clientId, the useAuth0 hook, authorize() for login and clearSession() for logout, native callback configuration for both Android (manifestPlaceholders) and iOS (CFBundleURLTypes), and the user profile displayed only after isLoading is false?
118
Claude Opus 5
FastAPI
79%11/14
92%+13%
21/21$0.68
1m 40s
With Auth0 Tools·92%·Tool Calls 17
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
9.6/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
9.2/12 pts
Error Recovery
When something broke, did the agent fix it?
4.2/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
92/100 ptsA
Grader Results (21/21 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses auth0-fastapi-api package
Imports Auth0FastAPI class
Uses require_auth() dependency
Uses FastAPI Depends for dependency injection
Configures domain
Configures audience
L2 · Hallucination Detection(5/5)
No python-jose (manual JWT parsing not needed)
No PyJWT direct usage
No manual JWT decoding
No fastapi-users (wrong package for Auth0 JWT)
No passlib (unrelated auth package)
L3 · Security(2/2)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
L4 · Structural(5/5)
Wrote Auth0 config to .env file
Auth0FastAPI instance is created
Route protected with Depends(auth0.require_auth())
read:messages scope checked on /api/messages route
Does the app correctly create an Auth0FastAPI instance, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token? Note: the auth0-fastapi-api distribution imports as the module `fastapi_plugin` (i.e. `from fastapi_plugin import Auth0FastAPI` is correct — there is no `auth0_fastapi_api` module).
L5 · Version Correctness(1/1)
Does the solution use current auth0-fastapi-api patterns? Specifically: does it use Auth0FastAPI with domain and audience parameters, use require_auth() as a FastAPI Depends dependency (not a decorator), and read credentials from environment variables (not hardcoded)? Note: the auth0-fastapi-api distribution imports as the module `fastapi_plugin` (`from fastapi_plugin import Auth0FastAPI` is correct — there is no `auth0_fastapi_api` module).
Holistic(1/1)
Does the solution correctly integrate Auth0 into a FastAPI API using auth0-fastapi-api? It should create an Auth0FastAPI instance configured with domain and audience from environment variables, protect the /api/messages route with read:messages scope check, and protect the /api/private route requiring a valid access token. Note: the auth0-fastapi-api distribution imports as the module `fastapi_plugin` (`from fastapi_plugin import Auth0FastAPI` is correct — there is no `auth0_fastapi_api` module).
119
Claude Opus 5
Flask
36%5/14
91%+55%
25/26$0.92
2m 23s
With Auth0 Tools·91%·Tool Calls 20
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
9.6/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
9.6/12 pts
Error Recovery
When something broke, did the agent fix it?
4.2/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
23.7/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
91/100 ptsA
Grader Results (25/26 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(6/6)
Uses auth0-server-python package
Imports ServerClient class
Uses start_interactive_login method
Uses complete_interactive_login method
Uses get_user method to retrieve user info
Loads environment variables with python-dotenv
L2 · Hallucination Detection(4/4)
No Authlib (old approach, not the current SDK)
No python-jose (manual JWT parsing not needed)
No Flask-Login (not needed with auth0-server-python)
No Flask-Dance (wrong OAuth package)
L3 · Security(3/3)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
No hardcoded Auth0 domain in source files (ok in .env)
L4 · Structural(8/8)
Wrote Auth0 credentials to .env file
Implements /callback route
Implements /login route
Implements /logout route
Implements /profile route
Sets Flask app.secret_key for session management
Uses async route handlers
Does the app implement login, callback, profile, and logout routes using auth0-server-python, protect the /profile route so it requires login, and include a route that calls an external API using an access token?
L5 · Version Correctness(3/3)
Uses authorization_params for scope and audience configuration
authorization_params contains audience 'https://api.barkbook.com'
Does the solution use current auth0-server-python patterns? Specifically: does it import ServerClient from auth0_server_python, define concrete state and transaction store classes locally (subclassing StateStore/TransactionStore from the SDK), and configure authorization_params with audience and scope?
Holistic(0/1)
Does the solution correctly integrate Auth0 into a Flask web app using auth0-server-python? It should configure a ServerClient with credentials from env vars, implement login, callback, profile, and logout routes, protect the /profile route so it requires authentication, and use the access token to call an external API with audience https://api.barkbook.com.
120
Claude Opus 5
Nuxt
75%15/20
90%+15%
32/32$2.30
5m 3s
With Auth0 Tools·90%·Tool Calls 39
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
8.4/12 pts
Setup Speed
How fast did the agent complete the task?
12/12 pts
Efficiency
How many tool calls were wasted versus useful work done?
10.2/12 pts
Error Recovery
When something broke, did the agent fix it?
2.8/7 pts
Docs Quality
Did the agent fetch real Auth0 docs and use them to write correct code?
7/7 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
90/100 ptsA
Grader Results (32/32 passed)
Build Passing
We compile the code for real, not just pattern-match it.
L1 · Positive Presence(10/10)
Uses @auth0/auth0-nuxt SDK
Registers module in nuxt.config
Configures runtimeConfig with Auth0 settings
Configures NUXT_AUTH0_DOMAIN env var
Configures NUXT_AUTH0_CLIENT_ID env var
Configures NUXT_AUTH0_CLIENT_SECRET env var
Configures NUXT_AUTH0_SESSION_SECRET env var
Uses useUser() composable to access the authenticated user
Uses /auth/login route to initiate login
Uses /auth/logout route to log out
L2 · Hallucination Detection(4/4)
Does not use the Vue SDK in a Nuxt app
Does not use the React SDK in a Nuxt app
Does not use loginWithRedirect (SPA API not applicable to Nuxt)
Does not use getAccessTokenSilently (Vue/React SPA API, not used in Nuxt)
L3 · Security(5/5)
No hardcoded client ID in source files (ok in .env)
No hardcoded client secret in source files (ok in .env)
No hardcoded Auth0 domain in source files (ok in .env)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(7/7)
Wrote Auth0 credentials to .env file
Module correctly registered in the modules array
sessionSecret configured for session encryption
clientSecret configured (Regular Web Application)
appBaseUrl configured in runtimeConfig
definePageMeta used for page-level middleware
Is there a protected /profile route that uses a Nuxt route middleware (via definePageMeta) to check authentication with useUser() and redirect unauthenticated users to /auth/login?
L5 · Version Correctness(4/4)
Audience configured in runtimeConfig for API access
Audience set to 'https://api.playground.com'
Uses getAccessToken() server-side to retrieve access token
Uses useAuth0(event) server-side composable
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Nuxt application using the @auth0/auth0-nuxt module, with proper configuration in nuxt.config.ts (domain, clientId, clientSecret, sessionSecret, appBaseUrl, audience), login/logout via /auth/login and /auth/logout, user profile display with useUser(), a /profile route protected by a route middleware, and server-side getAccessToken() for authenticated API calls?
* With Auth0 MCP + Skills enabledLast updated on August 2026
How we validate results
Every score is reproducible. Real integration tasks, automated grading, transparent methodology.
1
Real integration tasks
Real AI agents run identical Auth0 SDK prompts end-to-end. No synthetic benchmarks.
2
Automated grading
Pattern matching, security checks, and correctness validators score each response.
3
Tool comparison
Each model tested with and without MCP + Skills to measure improvement.
4
Compile verification
A deterministic build check confirms the final code actually compiles. No partial credit for code that only looks right.
Start building with confidence
These tools are how we achieved the benchmark results above. Enable them in your AI agent to integrate Auth0 reliably.
MCP Server
API access for agents
Gives AI agents access to Auth0 Management APIs and current documentation.
https://auth0.com/docs/mcpSkills
Best practices for agents
Teaches agents Auth0 integration patterns so they follow security best practices.
npx skills add auth0/agent-skills --all