Ask any AI agent to implement Auth0.
Choose the LLM, choose the framework, and see how those work with your Auth0 implementation.
Compare stacksWithout Auth0 tools
SDK setup & imports
Tokens stored securely
74%
avg. success rate
With Auth0
SkillsMCP
SDK setup & imports
Tokens stored securely
93%
avg. success rate19%
Tested across 5 models · 12 frameworks
Agent Experience Score for Auth0
Validated results for every model, framework, and configuration tested with Auth0
Select a framework to filter results for your stack.
#
Model
Framework
Without tools
With Auth0 tools *
Grade
Est. Cost
Est. Time *
1
Gemini 3.1 Pro
Vue
100%13/13
100%
19/19$0.12
1m 49s
With Auth0 Tools·100%·Tool Calls 8
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
14/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (19/19 passed)
L1 · Positive Presence(7/7)
Uses @auth0/auth0-vue SDK
Sets up Auth0 plugin with createAuth0
Uses useAuth0 composable
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile information
L2 · Hallucination Detection(3/3)
Does not use React SDK in Vue app
No hallucinated @auth0/vue3-auth0 package
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Plugin installed via app.use(createAuth0(...))
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated?
L5 · Version Correctness(3/3)
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Uses clientId (not deprecated client_id) in createAuth0 config
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Vue 3 SPA with the @auth0/auth0-vue plugin, useAuth0 composable, login, logout, user profile display, route protection, and getAccessTokenSilently to make authenticated API calls?
2
Claude Sonnet 4.6
Express
87%13/15
100%+13%
22/22$0.11
1m 44s
With Auth0 Tools·100%·Tool Calls 9
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
14/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (22/22 passed)
L1 · Positive Presence(7/7)
Uses express-openid-connect SDK
Configures issuerBaseURL
Configures clientID
Configures baseURL
Configures session secret
Uses req.oidc to access OIDC context
Accesses user profile via req.oidc.user
L2 · Hallucination Detection(4/4)
No hallucinated @auth0/auth0-spa-js (wrong SDK for server-side app)
No express-oauth2-jwt-bearer (that is for APIs, not web apps)
Does not use Next.js SDK in Express app
No passport middleware (not needed with express-openid-connect)
L3 · Security(3/3)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, session secret) stored in environment variables, .env or .env.local files, not hardcoded in source code?
L4 · Structural(4/4)
auth middleware registered with app.use
Uses requiresAuth() to protect the /profile route
Accesses access token via req.oidc.accessToken
Does the app correctly register the auth() middleware, protect the /profile route with requiresAuth(), display user profile information, and include a route that calls an external API using the access token?
L5 · Version Correctness(3/3)
Uses authorizationParams to pass audience and scope
authorizationParams contains audience 'https://api.barkbook.com'
Does the solution use current express-openid-connect patterns? Specifically: does it use issuerBaseURL (not AUTH0_DOMAIN or domain directly), configure audience via authorizationParams (not as a top-level config key), and use response_type: "code" for the authorization code flow?
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Express web app using express-openid-connect?
3
Claude Opus 4.6
Fastify API
57%8/14
100%+43%
19/19$0.13
54s
With Auth0 Tools·100%·Tool Calls 10
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
14/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (19/19 passed)
L1 · Positive Presence(6/6)
Uses @auth0/auth0-fastify-api package
Imports fastifyAuth0Api plugin
Uses fastify.requireAuth() to protect routes
Uses preHandler to attach auth middleware
Configures domain
Configures audience
L2 · Hallucination Detection(4/4)
No express-oauth2-jwt-bearer (wrong SDK for Fastify)
No passport middleware (not needed with @auth0/auth0-fastify-api)
No jsonwebtoken (manual JWT verification not needed)
No manual jwt.verify() calls
L3 · Security(3/3)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are the Auth0 domain and audience stored in environment variables or a .env file, not hardcoded in source code?
L4 · Structural(4/4)
Auth0 API plugin registered with fastify.register()
Route protected with preHandler: fastify.requireAuth()
read:messages scope checked on /api/messages route
Does the app correctly register the @auth0/auth0-fastify-api plugin, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token?
L5 · Version Correctness(1/1)
Does the solution use current @auth0/auth0-fastify-api patterns? Specifically: does it register the plugin via fastify.register(), use preHandler: fastify.requireAuth() for route protection (not a decorator), access token claims via request.user, and read credentials from environment variables (not hardcoded)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Fastify API using @auth0/auth0-fastify-api?
4
Claude Sonnet 4.6
Fastify API
50%7/14
100%+50%
19/19$0.10
56s
With Auth0 Tools·100%·Tool Calls 7
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
14/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (19/19 passed)
L1 · Positive Presence(6/6)
Uses @auth0/auth0-fastify-api package
Imports fastifyAuth0Api plugin
Uses fastify.requireAuth() to protect routes
Uses preHandler to attach auth middleware
Configures domain
Configures audience
L2 · Hallucination Detection(4/4)
No express-oauth2-jwt-bearer (wrong SDK for Fastify)
No passport middleware (not needed with @auth0/auth0-fastify-api)
No jsonwebtoken (manual JWT verification not needed)
No manual jwt.verify() calls
L3 · Security(3/3)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are the Auth0 domain and audience stored in environment variables or a .env file, not hardcoded in source code?
L4 · Structural(4/4)
Auth0 API plugin registered with fastify.register()
Route protected with preHandler: fastify.requireAuth()
read:messages scope checked on /api/messages route
Does the app correctly register the @auth0/auth0-fastify-api plugin, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token?
L5 · Version Correctness(1/1)
Does the solution use current @auth0/auth0-fastify-api patterns? Specifically: does it register the plugin via fastify.register(), use preHandler: fastify.requireAuth() for route protection (not a decorator), access token claims via request.user, and read credentials from environment variables (not hardcoded)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Fastify API using @auth0/auth0-fastify-api?
5
Claude Opus 4.7
FastAPI
40%6/15
100%+60%
20/20$0.13
30s
With Auth0 Tools·100%·Tool Calls 7
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
14/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (20/20 passed)
L1 · Positive Presence(6/6)
Uses auth0-fastapi-api package
Imports Auth0FastAPI class
Uses require_auth() dependency
Uses FastAPI Depends for dependency injection
Configures domain
Configures audience
L2 · Hallucination Detection(5/5)
No python-jose (manual JWT parsing not needed)
No PyJWT direct usage
No manual JWT decoding
No fastapi-users (wrong package for Auth0 JWT)
No passlib (unrelated auth package)
L3 · Security(3/3)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are the Auth0 domain and audience stored in environment variables or a .env file, not hardcoded in Python source code?
L4 · Structural(4/4)
Auth0FastAPI instance is created
Route protected with Depends(auth0.require_auth())
read:messages scope checked on /api/messages route
Does the app correctly create an Auth0FastAPI instance, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token?
L5 · Version Correctness(1/1)
Does the solution use current auth0-fastapi-api patterns? Specifically: does it use Auth0FastAPI with domain and audience parameters, use require_auth() as a FastAPI Depends dependency (not a decorator), and read credentials from environment variables (not hardcoded)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a FastAPI API using auth0-fastapi-api?
6
Gemini 3.1 Pro
FastAPI
40%6/15
100%+60%
20/20$0.21
2m 5s
With Auth0 Tools·100%·Tool Calls 8
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
14/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (20/20 passed)
L1 · Positive Presence(6/6)
Uses auth0-fastapi-api package
Imports Auth0FastAPI class
Uses require_auth() dependency
Uses FastAPI Depends for dependency injection
Configures domain
Configures audience
L2 · Hallucination Detection(5/5)
No python-jose (manual JWT parsing not needed)
No PyJWT direct usage
No manual JWT decoding
No fastapi-users (wrong package for Auth0 JWT)
No passlib (unrelated auth package)
L3 · Security(3/3)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are the Auth0 domain and audience stored in environment variables or a .env file, not hardcoded in Python source code?
L4 · Structural(4/4)
Auth0FastAPI instance is created
Route protected with Depends(auth0.require_auth())
read:messages scope checked on /api/messages route
Does the app correctly create an Auth0FastAPI instance, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token?
L5 · Version Correctness(1/1)
Does the solution use current auth0-fastapi-api patterns? Specifically: does it use Auth0FastAPI with domain and audience parameters, use require_auth() as a FastAPI Depends dependency (not a decorator), and read credentials from environment variables (not hardcoded)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a FastAPI API using auth0-fastapi-api?
7
Claude Opus 4.7
Fastify API
36%5/14
100%+64%
19/19$0.13
45s
With Auth0 Tools·100%·Tool Calls 8
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
14/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (19/19 passed)
L1 · Positive Presence(6/6)
Uses @auth0/auth0-fastify-api package
Imports fastifyAuth0Api plugin
Uses fastify.requireAuth() to protect routes
Uses preHandler to attach auth middleware
Configures domain
Configures audience
L2 · Hallucination Detection(4/4)
No express-oauth2-jwt-bearer (wrong SDK for Fastify)
No passport middleware (not needed with @auth0/auth0-fastify-api)
No jsonwebtoken (manual JWT verification not needed)
No manual jwt.verify() calls
L3 · Security(3/3)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are the Auth0 domain and audience stored in environment variables or a .env file, not hardcoded in source code?
L4 · Structural(4/4)
Auth0 API plugin registered with fastify.register()
Route protected with preHandler: fastify.requireAuth()
read:messages scope checked on /api/messages route
Does the app correctly register the @auth0/auth0-fastify-api plugin, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token?
L5 · Version Correctness(1/1)
Does the solution use current @auth0/auth0-fastify-api patterns? Specifically: does it register the plugin via fastify.register(), use preHandler: fastify.requireAuth() for route protection (not a decorator), access token claims via request.user, and read credentials from environment variables (not hardcoded)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Fastify API using @auth0/auth0-fastify-api?
8
Claude Opus 4.7
SPA JS
100%14/14
99%-1%
20/20$0.15
57s
With Auth0 Tools·99%·Tool Calls 11
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
12.7/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (20/20 passed)
L1 · Positive Presence(7/7)
Uses @auth0/auth0-spa-js SDK
Initializes Auth0 with createAuth0Client
Implements loginWithRedirect
Handles redirect callback on page load
Implements logout
Checks isAuthenticated for conditional rendering
Uses getTokenSilently to retrieve access token
L2 · Hallucination Detection(4/4)
Does not use React SDK in vanilla JS app
Does not use Vue SDK in vanilla JS app
Does not use Next.js SDK in vanilla JS app
No client_secret in SPA (public client)
L3 · Security(2/2)
No tokens manually stored in localStorage
No tokens manually stored in sessionStorage
L4 · Structural(3/3)
Auth0Client configured with domain
Authenticated API request uses Bearer token in Authorization header
Does the code check isAuthenticated to show/hide UI elements and only render user profile information when the user is authenticated?
L5 · Version Correctness(3/3)
Uses authorizationParams (not deprecated top-level audience/redirect_uri)
authorizationParams contains audience 'https://api.playground.com'
No deprecated promise-chain pattern for getTokenSilently
Holistic(1/1)
Does the solution correctly integrate Auth0 into a vanilla JavaScript SPA using @auth0/auth0-spa-js with createAuth0Client, loginWithRedirect, handleRedirectCallback, logout, user profile display, and getTokenSilently to make authenticated API calls?
9
Claude Sonnet 4.6
SPA JS
100%14/14
99%-1%
20/20$0.12
2m 7s
With Auth0 Tools·99%·Tool Calls 11
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
12.7/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (20/20 passed)
L1 · Positive Presence(7/7)
Uses @auth0/auth0-spa-js SDK
Initializes Auth0 with createAuth0Client
Implements loginWithRedirect
Handles redirect callback on page load
Implements logout
Checks isAuthenticated for conditional rendering
Uses getTokenSilently to retrieve access token
L2 · Hallucination Detection(4/4)
Does not use React SDK in vanilla JS app
Does not use Vue SDK in vanilla JS app
Does not use Next.js SDK in vanilla JS app
No client_secret in SPA (public client)
L3 · Security(2/2)
No tokens manually stored in localStorage
No tokens manually stored in sessionStorage
L4 · Structural(3/3)
Auth0Client configured with domain
Authenticated API request uses Bearer token in Authorization header
Does the code check isAuthenticated to show/hide UI elements and only render user profile information when the user is authenticated?
L5 · Version Correctness(3/3)
Uses authorizationParams (not deprecated top-level audience/redirect_uri)
authorizationParams contains audience 'https://api.playground.com'
No deprecated promise-chain pattern for getTokenSilently
Holistic(1/1)
Does the solution correctly integrate Auth0 into a vanilla JavaScript SPA using @auth0/auth0-spa-js with createAuth0Client, loginWithRedirect, handleRedirectCallback, logout, user profile display, and getTokenSilently to make authenticated API calls?
10
Gemini 3.1 Pro
SPA JS
93%13/14
99%+6%
20/20$0.23
2m 22s
With Auth0 Tools·99%·Tool Calls 11
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
12.7/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (20/20 passed)
L1 · Positive Presence(7/7)
Uses @auth0/auth0-spa-js SDK
Initializes Auth0 with createAuth0Client
Implements loginWithRedirect
Handles redirect callback on page load
Implements logout
Checks isAuthenticated for conditional rendering
Uses getTokenSilently to retrieve access token
L2 · Hallucination Detection(4/4)
Does not use React SDK in vanilla JS app
Does not use Vue SDK in vanilla JS app
Does not use Next.js SDK in vanilla JS app
No client_secret in SPA (public client)
L3 · Security(2/2)
No tokens manually stored in localStorage
No tokens manually stored in sessionStorage
L4 · Structural(3/3)
Auth0Client configured with domain
Authenticated API request uses Bearer token in Authorization header
Does the code check isAuthenticated to show/hide UI elements and only render user profile information when the user is authenticated?
L5 · Version Correctness(3/3)
Uses authorizationParams (not deprecated top-level audience/redirect_uri)
authorizationParams contains audience 'https://api.playground.com'
No deprecated promise-chain pattern for getTokenSilently
Holistic(1/1)
Does the solution correctly integrate Auth0 into a vanilla JavaScript SPA using @auth0/auth0-spa-js with createAuth0Client, loginWithRedirect, handleRedirectCallback, logout, user profile display, and getTokenSilently to make authenticated API calls?
11
GPT-5.4
Fastify API
50%7/14
99%+49%
19/19$0.43
1m 42s
With Auth0 Tools·99%·Tool Calls 11
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
12.7/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (19/19 passed)
L1 · Positive Presence(6/6)
Uses @auth0/auth0-fastify-api package
Imports fastifyAuth0Api plugin
Uses fastify.requireAuth() to protect routes
Uses preHandler to attach auth middleware
Configures domain
Configures audience
L2 · Hallucination Detection(4/4)
No express-oauth2-jwt-bearer (wrong SDK for Fastify)
No passport middleware (not needed with @auth0/auth0-fastify-api)
No jsonwebtoken (manual JWT verification not needed)
No manual jwt.verify() calls
L3 · Security(3/3)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are the Auth0 domain and audience stored in environment variables or a .env file, not hardcoded in source code?
L4 · Structural(4/4)
Auth0 API plugin registered with fastify.register()
Route protected with preHandler: fastify.requireAuth()
read:messages scope checked on /api/messages route
Does the app correctly register the @auth0/auth0-fastify-api plugin, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token?
L5 · Version Correctness(1/1)
Does the solution use current @auth0/auth0-fastify-api patterns? Specifically: does it register the plugin via fastify.register(), use preHandler: fastify.requireAuth() for route protection (not a decorator), access token claims via request.user, and read credentials from environment variables (not hardcoded)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Fastify API using @auth0/auth0-fastify-api?
12
Claude Opus 4.6
SPA JS
100%14/14
98%-2%
20/20$0.15
1m 11s
With Auth0 Tools·98%·Tool Calls 12
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
11.7/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
98/100 ptsA
Grader Results (20/20 passed)
L1 · Positive Presence(7/7)
Uses @auth0/auth0-spa-js SDK
Initializes Auth0 with createAuth0Client
Implements loginWithRedirect
Handles redirect callback on page load
Implements logout
Checks isAuthenticated for conditional rendering
Uses getTokenSilently to retrieve access token
L2 · Hallucination Detection(4/4)
Does not use React SDK in vanilla JS app
Does not use Vue SDK in vanilla JS app
Does not use Next.js SDK in vanilla JS app
No client_secret in SPA (public client)
L3 · Security(2/2)
No tokens manually stored in localStorage
No tokens manually stored in sessionStorage
L4 · Structural(3/3)
Auth0Client configured with domain
Authenticated API request uses Bearer token in Authorization header
Does the code check isAuthenticated to show/hide UI elements and only render user profile information when the user is authenticated?
L5 · Version Correctness(3/3)
Uses authorizationParams (not deprecated top-level audience/redirect_uri)
authorizationParams contains audience 'https://api.playground.com'
No deprecated promise-chain pattern for getTokenSilently
Holistic(1/1)
Does the solution correctly integrate Auth0 into a vanilla JavaScript SPA using @auth0/auth0-spa-js with createAuth0Client, loginWithRedirect, handleRedirectCallback, logout, user profile display, and getTokenSilently to make authenticated API calls?
13
Claude Opus 4.7
React
92%12/13
98%+6%
18/19$0.17
57s
With Auth0 Tools·98%·Tool Calls 10
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
14/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
23.2/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
98/100 ptsA
Grader Results (18/19 passed)
L1 · Positive Presence(6/7)
Uses @auth0/auth0-react SDK
Wraps app with Auth0Provider
Uses useAuth0 hook
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile name
L2 · Hallucination Detection(3/3)
No hallucinated @auth0/react package (must be @auth0/auth0-react)
Does not use server SDK in SPA app
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Auth0Provider configured with domain prop
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated?
L5 · Version Correctness(3/3)
Uses authorizationParams (not audience directly on provider)
authorizationParams contains audience 'https://api.barkbook.com'
Does the code use the current @auth0/auth0-react SDK patterns? Specifically: does it use isLoading (not the deprecated "loading" property), and pass audience/scope via authorizationParams object (not as direct props)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a React SPA with Auth0Provider, useAuth0 hook, login, logout, user profile display, and getAccessTokenSilently to make authenticated API calls?
14
Claude Opus 4.6
Express
87%13/15
98%+11%
21/22$0.12
1m 4s
With Auth0 Tools·98%·Tool Calls 10
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
14/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
23.3/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
98/100 ptsA
Grader Results (21/22 passed)
L1 · Positive Presence(7/7)
Uses express-openid-connect SDK
Configures issuerBaseURL
Configures clientID
Configures baseURL
Configures session secret
Uses req.oidc to access OIDC context
Accesses user profile via req.oidc.user
L2 · Hallucination Detection(4/4)
No hallucinated @auth0/auth0-spa-js (wrong SDK for server-side app)
No express-oauth2-jwt-bearer (that is for APIs, not web apps)
Does not use Next.js SDK in Express app
No passport middleware (not needed with express-openid-connect)
L3 · Security(3/3)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, session secret) stored in environment variables, .env or .env.local files, not hardcoded in source code?
L4 · Structural(4/4)
auth middleware registered with app.use
Uses requiresAuth() to protect the /profile route
Accesses access token via req.oidc.accessToken
Does the app correctly register the auth() middleware, protect the /profile route with requiresAuth(), display user profile information, and include a route that calls an external API using the access token?
L5 · Version Correctness(3/3)
Uses authorizationParams to pass audience and scope
authorizationParams contains audience 'https://api.barkbook.com'
Does the solution use current express-openid-connect patterns? Specifically: does it use issuerBaseURL (not AUTH0_DOMAIN or domain directly), configure audience via authorizationParams (not as a top-level config key), and use response_type: "code" for the authorization code flow?
Holistic(0/1)
Does the solution correctly integrate Auth0 into an Express web app using express-openid-connect?
15
Claude Opus 4.7
Express
87%13/15
98%+11%
21/22$0.17
57s
With Auth0 Tools·98%·Tool Calls 9
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
14/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
23.3/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
98/100 ptsA
Grader Results (21/22 passed)
L1 · Positive Presence(7/7)
Uses express-openid-connect SDK
Configures issuerBaseURL
Configures clientID
Configures baseURL
Configures session secret
Uses req.oidc to access OIDC context
Accesses user profile via req.oidc.user
L2 · Hallucination Detection(4/4)
No hallucinated @auth0/auth0-spa-js (wrong SDK for server-side app)
No express-oauth2-jwt-bearer (that is for APIs, not web apps)
Does not use Next.js SDK in Express app
No passport middleware (not needed with express-openid-connect)
L3 · Security(3/3)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, session secret) stored in environment variables, .env or .env.local files, not hardcoded in source code?
L4 · Structural(4/4)
auth middleware registered with app.use
Uses requiresAuth() to protect the /profile route
Accesses access token via req.oidc.accessToken
Does the app correctly register the auth() middleware, protect the /profile route with requiresAuth(), display user profile information, and include a route that calls an external API using the access token?
L5 · Version Correctness(3/3)
Uses authorizationParams to pass audience and scope
authorizationParams contains audience 'https://api.barkbook.com'
Does the solution use current express-openid-connect patterns? Specifically: does it use issuerBaseURL (not AUTH0_DOMAIN or domain directly), configure audience via authorizationParams (not as a top-level config key), and use response_type: "code" for the authorization code flow?
Holistic(0/1)
Does the solution correctly integrate Auth0 into an Express web app using express-openid-connect?
16
Claude Opus 4.7
Express API
85%11/13
98%+13%
20/21$0.15
50s
With Auth0 Tools·98%·Tool Calls 10
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
14/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
23.2/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
98/100 ptsA
Grader Results (20/21 passed)
L1 · Positive Presence(5/5)
Uses express-oauth2-jwt-bearer SDK
Configures issuerBaseURL
Configures audience claim
Uses requiredScopes() for scope-based route protection
Accesses JWT data via req.auth
L2 · Hallucination Detection(4/4)
No express-openid-connect (that is for web apps, not APIs)
No passport middleware (not needed with express-oauth2-jwt-bearer)
No manual JWT verification with jsonwebtoken (use SDK)
No SPA SDK used in server-side API
L3 · Security(3/3)
No hardcoded issuer domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are all Auth0 configuration values (issuerBaseURL / domain, audience) stored in environment variables, not hardcoded in source code?
L4 · Structural(3/4)
auth() middleware registered globally with app.use
GET /api/messages protected with read:messages scope
POST /api/messages protected with write:messages scope
Does the API correctly protect all three routes?
L5 · Version Correctness(4/4)
No manual Authorization header parsing (SDK handles extraction)
No manual jwt.verify calls (SDK handles verification)
No req.user (express-oauth2-jwt-bearer uses req.auth, not req.user)
Does the solution use current express-oauth2-jwt-bearer patterns? Specifically: does it configure auth() with issuerBaseURL and audience, use requiredScopes() for scope checks (not manual payload inspection), and access token data via req.auth.payload (not req.user or manually decoded tokens)?
Holistic(1/1)
Does the solution correctly protect an Express.js API using express-oauth2-jwt-bearer?
17
Claude Sonnet 4.6
React
100%13/13
97%-3%
19/19$0.16
1m 42s
With Auth0 Tools·97%·Tool Calls 13
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
10.8/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
97/100 ptsA
Grader Results (19/19 passed)
L1 · Positive Presence(7/7)
Uses @auth0/auth0-react SDK
Wraps app with Auth0Provider
Uses useAuth0 hook
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile name
L2 · Hallucination Detection(3/3)
No hallucinated @auth0/react package (must be @auth0/auth0-react)
Does not use server SDK in SPA app
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Auth0Provider configured with domain prop
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated?
L5 · Version Correctness(3/3)
Uses authorizationParams (not audience directly on provider)
authorizationParams contains audience 'https://api.barkbook.com'
Does the code use the current @auth0/auth0-react SDK patterns? Specifically: does it use isLoading (not the deprecated "loading" property), and pass audience/scope via authorizationParams object (not as direct props)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a React SPA with Auth0Provider, useAuth0 hook, login, logout, user profile display, and getAccessTokenSilently to make authenticated API calls?
18
Gemini 3.1 Pro
React
100%13/13
97%-3%
19/19$0.28
2m 30s
With Auth0 Tools·97%·Tool Calls 13
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
10.8/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
97/100 ptsA
Grader Results (19/19 passed)
L1 · Positive Presence(7/7)
Uses @auth0/auth0-react SDK
Wraps app with Auth0Provider
Uses useAuth0 hook
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile name
L2 · Hallucination Detection(3/3)
No hallucinated @auth0/react package (must be @auth0/auth0-react)
Does not use server SDK in SPA app
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Auth0Provider configured with domain prop
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated?
L5 · Version Correctness(3/3)
Uses authorizationParams (not audience directly on provider)
authorizationParams contains audience 'https://api.barkbook.com'
Does the code use the current @auth0/auth0-react SDK patterns? Specifically: does it use isLoading (not the deprecated "loading" property), and pass audience/scope via authorizationParams object (not as direct props)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a React SPA with Auth0Provider, useAuth0 hook, login, logout, user profile display, and getAccessTokenSilently to make authenticated API calls?
19
GPT-5.4
SPA JS
100%14/14
97%-3%
20/20$0.36
1m 39s
With Auth0 Tools·97%·Tool Calls 13
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
10.8/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
97/100 ptsA
Grader Results (20/20 passed)
L1 · Positive Presence(7/7)
Uses @auth0/auth0-spa-js SDK
Initializes Auth0 with createAuth0Client
Implements loginWithRedirect
Handles redirect callback on page load
Implements logout
Checks isAuthenticated for conditional rendering
Uses getTokenSilently to retrieve access token
L2 · Hallucination Detection(4/4)
Does not use React SDK in vanilla JS app
Does not use Vue SDK in vanilla JS app
Does not use Next.js SDK in vanilla JS app
No client_secret in SPA (public client)
L3 · Security(2/2)
No tokens manually stored in localStorage
No tokens manually stored in sessionStorage
L4 · Structural(3/3)
Auth0Client configured with domain
Authenticated API request uses Bearer token in Authorization header
Does the code check isAuthenticated to show/hide UI elements and only render user profile information when the user is authenticated?
L5 · Version Correctness(3/3)
Uses authorizationParams (not deprecated top-level audience/redirect_uri)
authorizationParams contains audience 'https://api.playground.com'
No deprecated promise-chain pattern for getTokenSilently
Holistic(1/1)
Does the solution correctly integrate Auth0 into a vanilla JavaScript SPA using @auth0/auth0-spa-js with createAuth0Client, loginWithRedirect, handleRedirectCallback, logout, user profile display, and getTokenSilently to make authenticated API calls?
20
Claude Opus 4.6
React
92%12/13
97%+5%
19/19$0.14
1m 12s
With Auth0 Tools·97%·Tool Calls 13
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
10.8/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
97/100 ptsA
Grader Results (19/19 passed)
L1 · Positive Presence(7/7)
Uses @auth0/auth0-react SDK
Wraps app with Auth0Provider
Uses useAuth0 hook
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile name
L2 · Hallucination Detection(3/3)
No hallucinated @auth0/react package (must be @auth0/auth0-react)
Does not use server SDK in SPA app
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Auth0Provider configured with domain prop
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated?
L5 · Version Correctness(3/3)
Uses authorizationParams (not audience directly on provider)
authorizationParams contains audience 'https://api.barkbook.com'
Does the code use the current @auth0/auth0-react SDK patterns? Specifically: does it use isLoading (not the deprecated "loading" property), and pass audience/scope via authorizationParams object (not as direct props)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a React SPA with Auth0Provider, useAuth0 hook, login, logout, user profile display, and getAccessTokenSilently to make authenticated API calls?
21
Claude Opus 4.6
Express API
85%11/13
97%+12%
20/21$0.29
1m 19s
With Auth0 Tools·97%·Tool Calls 11
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
12.7/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
23.2/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
97/100 ptsA
Grader Results (20/21 passed)
L1 · Positive Presence(5/5)
Uses express-oauth2-jwt-bearer SDK
Configures issuerBaseURL
Configures audience claim
Uses requiredScopes() for scope-based route protection
Accesses JWT data via req.auth
L2 · Hallucination Detection(4/4)
No express-openid-connect (that is for web apps, not APIs)
No passport middleware (not needed with express-oauth2-jwt-bearer)
No manual JWT verification with jsonwebtoken (use SDK)
No SPA SDK used in server-side API
L3 · Security(3/3)
No hardcoded issuer domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are all Auth0 configuration values (issuerBaseURL / domain, audience) stored in environment variables, not hardcoded in source code?
L4 · Structural(3/4)
auth() middleware registered globally with app.use
GET /api/messages protected with read:messages scope
POST /api/messages protected with write:messages scope
Does the API correctly protect all three routes?
L5 · Version Correctness(4/4)
No manual Authorization header parsing (SDK handles extraction)
No manual jwt.verify calls (SDK handles verification)
No req.user (express-oauth2-jwt-bearer uses req.auth, not req.user)
Does the solution use current express-oauth2-jwt-bearer patterns? Specifically: does it configure auth() with issuerBaseURL and audience, use requiredScopes() for scope checks (not manual payload inspection), and access token data via req.auth.payload (not req.user or manually decoded tokens)?
Holistic(1/1)
Does the solution correctly protect an Express.js API using express-oauth2-jwt-bearer?
22
Claude Opus 4.7
Swift
60%6/10
97%+37%
13/14$0.21
1m 9s
With Auth0 Tools·97%·Tool Calls 10
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
14/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
22.2/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
97/100 ptsA
Grader Results (13/14 passed)
L1 · Positive Presence(4/4)
Uses Auth0 SDK
Imports Auth0 module
Uses webAuth() for login
Uses clearSession for logout
L2 · Hallucination Detection(3/3)
No hallucinated Auth0SDK package name (correct package is Auth0)
Does not use CocoaPods (SPM preferred)
Does not use deprecated completion handler pattern
L3 · Security(2/2)
No hardcoded client ID in Swift source files (ok in Auth0.plist)
No hardcoded domain in Swift source files (ok in Auth0.plist)
L4 · Structural(2/2)
Uses CredentialsManager for token storage
Does the code properly handle login and logout flows with appropriate error handling? Does it update the UI state after successful authentication?
L5 · Version Correctness(1/2)
Uses async/await webAuth().start() syntax (not completion handlers)
Does the code use modern Swift async/await patterns with the Auth0.swift SDK? Specifically: does it use try await webAuth().start() and CredentialsManager, and configure via Auth0.plist rather than hardcoded strings?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Swift iOS app with webAuth() login/logout, credential management, and proper SwiftUI state handling?
23
GPT-5.4
React
92%12/13
96%+4%
19/19$0.50
2m 18s
With Auth0 Tools·96%·Tool Calls 14
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
10/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
96/100 ptsA
Grader Results (19/19 passed)
L1 · Positive Presence(7/7)
Uses @auth0/auth0-react SDK
Wraps app with Auth0Provider
Uses useAuth0 hook
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile name
L2 · Hallucination Detection(3/3)
No hallucinated @auth0/react package (must be @auth0/auth0-react)
Does not use server SDK in SPA app
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Auth0Provider configured with domain prop
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated?
L5 · Version Correctness(3/3)
Uses authorizationParams (not audience directly on provider)
authorizationParams contains audience 'https://api.barkbook.com'
Does the code use the current @auth0/auth0-react SDK patterns? Specifically: does it use isLoading (not the deprecated "loading" property), and pass audience/scope via authorizationParams object (not as direct props)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a React SPA with Auth0Provider, useAuth0 hook, login, logout, user profile display, and getAccessTokenSilently to make authenticated API calls?
24
Gemini 3.1 Pro
Fastify API
43%6/14
96%+53%
19/19$0.28
4m 38s
With Auth0 Tools·96%·Tool Calls 14
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
10/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
96/100 ptsA
Grader Results (19/19 passed)
L1 · Positive Presence(6/6)
Uses @auth0/auth0-fastify-api package
Imports fastifyAuth0Api plugin
Uses fastify.requireAuth() to protect routes
Uses preHandler to attach auth middleware
Configures domain
Configures audience
L2 · Hallucination Detection(4/4)
No express-oauth2-jwt-bearer (wrong SDK for Fastify)
No passport middleware (not needed with @auth0/auth0-fastify-api)
No jsonwebtoken (manual JWT verification not needed)
No manual jwt.verify() calls
L3 · Security(3/3)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are the Auth0 domain and audience stored in environment variables or a .env file, not hardcoded in source code?
L4 · Structural(4/4)
Auth0 API plugin registered with fastify.register()
Route protected with preHandler: fastify.requireAuth()
read:messages scope checked on /api/messages route
Does the app correctly register the @auth0/auth0-fastify-api plugin, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token?
L5 · Version Correctness(1/1)
Does the solution use current @auth0/auth0-fastify-api patterns? Specifically: does it register the plugin via fastify.register(), use preHandler: fastify.requireAuth() for route protection (not a decorator), access token claims via request.user, and read credentials from environment variables (not hardcoded)?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Fastify API using @auth0/auth0-fastify-api?
25
Gemini 3.1 Pro
Angular
100%12/12
95%-5%
17/17$0.38
5m 7s
With Auth0 Tools·95%·Tool Calls 14
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
12.8/14 pts
Efficiency
How many steps did it take? Fewer is better.
10/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
95/100 ptsA
Grader Results (17/17 passed)
L1 · Positive Presence(6/6)
Uses @auth0/auth0-angular SDK
Injects AuthService for authentication operations
Implements loginWithRedirect
Implements logout
Uses isAuthenticated$ observable for auth state
Uses user$ observable to display user profile
L2 · Hallucination Detection(3/3)
Does not use React SDK in Angular app
Does not use Vue SDK in Angular app
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Auth0 configured via provideAuth0()
Route protected with AuthGuard or authGuardFn
Uses getAccessTokenSilently or httpInterceptor for authenticated API calls
L5 · Version Correctness(2/2)
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Angular SPA using @auth0/auth0-angular, with provideAuth0 setup, AuthService for login/logout, user profile display, route protection via a guard, and authenticated API calls (either by configuring authHttpInterceptorFn with an allowedList matching the API base URL, or by calling getAccessTokenSilently to retrieve tokens manually)?
26
Claude Opus 4.7
Android
67%6/9
95%+28%
14/14$0.38
1m 12s
With Auth0 Tools·95%·Tool Calls 15
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
9.3/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
95/100 ptsA
Grader Results (14/14 passed)
L1 · Positive Presence(5/5)
Uses Auth0 Android SDK dependency
Uses WebAuthProvider for authentication
Calls WebAuthProvider.login() for sign-in
Calls WebAuthProvider.logout() for sign-out
Uses CredentialsManager for secure token storage
L2 · Hallucination Detection(1/1)
No auth0-java (server-side SDK, not for Android)
L3 · Security(2/2)
No hardcoded client ID in Kotlin source files (ok in strings.xml)
No hardcoded domain in Kotlin source files (ok in strings.xml)
L4 · Structural(3/3)
Configures manifestPlaceholders in build.gradle for Auth0 callback URL scheme
Does the code implement both onSuccess and onFailure callbacks for WebAuthProvider.login(...) and WebAuthProvider.logout(...), with AuthenticationException handled in onFailure?
Is the Auth0 callback URL scheme correctly handled?
L5 · Version Correctness(2/2)
Uses Auth0(context) or Auth0.getInstance(context) for auto-configuration from string resources
Does the code use current Auth0 Android SDK v2+ patterns? Specifically: Auth0(context) for auto-configuration from strings.xml, WebAuthProvider for browser-based login/logout, and CredentialsManager for secure credential storage rather than persisting tokens manually in SharedPreferences?
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Android app with WebAuthProvider login and logout, CredentialsManager for secure token storage, Auth0 credentials configured via string resources, and manifestPlaceholders for callback URL handling?
27
Claude Opus 4.6
Android
67%6/9
94%+27%
14/14$0.40
1m 20s
With Auth0 Tools·94%·Tool Calls 18
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
7.8/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
94/100 ptsA
Grader Results (14/14 passed)
L1 · Positive Presence(5/5)
Uses Auth0 Android SDK dependency
Uses WebAuthProvider for authentication
Calls WebAuthProvider.login() for sign-in
Calls WebAuthProvider.logout() for sign-out
Uses CredentialsManager for secure token storage
L2 · Hallucination Detection(1/1)
No auth0-java (server-side SDK, not for Android)
L3 · Security(2/2)
No hardcoded client ID in Kotlin source files (ok in strings.xml)
No hardcoded domain in Kotlin source files (ok in strings.xml)
L4 · Structural(3/3)
Configures manifestPlaceholders in build.gradle for Auth0 callback URL scheme
Does the code implement both onSuccess and onFailure callbacks for WebAuthProvider.login(...) and WebAuthProvider.logout(...), with AuthenticationException handled in onFailure?
Is the Auth0 callback URL scheme correctly handled?
L5 · Version Correctness(2/2)
Uses Auth0(context) or Auth0.getInstance(context) for auto-configuration from string resources
Does the code use current Auth0 Android SDK v2+ patterns? Specifically: Auth0(context) for auto-configuration from strings.xml, WebAuthProvider for browser-based login/logout, and CredentialsManager for secure credential storage rather than persisting tokens manually in SharedPreferences?
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Android app with WebAuthProvider login and logout, CredentialsManager for secure token storage, Auth0 credentials configured via string resources, and manifestPlaceholders for callback URL handling?
28
Gemini 3.1 Pro
Android
56%5/9
94%+38%
14/14$0.52
3m 36s
With Auth0 Tools·94%·Tool Calls 17
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
8.2/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
94/100 ptsA
Grader Results (14/14 passed)
L1 · Positive Presence(5/5)
Uses Auth0 Android SDK dependency
Uses WebAuthProvider for authentication
Calls WebAuthProvider.login() for sign-in
Calls WebAuthProvider.logout() for sign-out
Uses CredentialsManager for secure token storage
L2 · Hallucination Detection(1/1)
No auth0-java (server-side SDK, not for Android)
L3 · Security(2/2)
No hardcoded client ID in Kotlin source files (ok in strings.xml)
No hardcoded domain in Kotlin source files (ok in strings.xml)
L4 · Structural(3/3)
Configures manifestPlaceholders in build.gradle for Auth0 callback URL scheme
Does the code implement both onSuccess and onFailure callbacks for WebAuthProvider.login(...) and WebAuthProvider.logout(...), with AuthenticationException handled in onFailure?
Is the Auth0 callback URL scheme correctly handled?
L5 · Version Correctness(2/2)
Uses Auth0(context) or Auth0.getInstance(context) for auto-configuration from string resources
Does the code use current Auth0 Android SDK v2+ patterns? Specifically: Auth0(context) for auto-configuration from strings.xml, WebAuthProvider for browser-based login/logout, and CredentialsManager for secure credential storage rather than persisting tokens manually in SharedPreferences?
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Android app with WebAuthProvider login and logout, CredentialsManager for secure token storage, Auth0 credentials configured via string resources, and manifestPlaceholders for callback URL handling?
29
Claude Opus 4.7
Nuxt
40%8/20
94%+54%
30/30$0.26
1m 12s
With Auth0 Tools·94%·Tool Calls 17
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
8.2/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
94/100 ptsA
Grader Results (30/30 passed)
L1 · Positive Presence(10/10)
Uses @auth0/auth0-nuxt SDK
Registers module in nuxt.config
Configures runtimeConfig with Auth0 settings
Configures NUXT_AUTH0_DOMAIN env var
Configures NUXT_AUTH0_CLIENT_ID env var
Configures NUXT_AUTH0_CLIENT_SECRET env var
Configures NUXT_AUTH0_SESSION_SECRET env var
Uses useUser() composable to access the authenticated user
Uses /auth/login route to initiate login
Uses /auth/logout route to log out
L2 · Hallucination Detection(4/4)
Does not use the Vue SDK in a Nuxt app
Does not use the React SDK in a Nuxt app
Does not use loginWithRedirect (SPA API not applicable to Nuxt)
Does not use getAccessTokenSilently (Vue/React SPA API, not used in Nuxt)
L3 · Security(5/5)
No hardcoded client ID in source files (ok in .env)
No hardcoded client secret in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, session secret) stored in environment variables or .env files using NUXT_AUTH0_* names, not hardcoded in nuxt.config.ts?
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(6/6)
Module correctly registered in the modules array
sessionSecret configured for session encryption
clientSecret configured (Regular Web Application)
appBaseUrl configured in runtimeConfig
definePageMeta used for page-level middleware
Is there a protected /profile route that uses a Nuxt route middleware (via definePageMeta) to check authentication with useUser() and redirect unauthenticated users to /auth/login?
L5 · Version Correctness(4/4)
Audience configured in runtimeConfig for API access
Audience set to 'https://api.playground.com'
Uses getAccessToken() server-side to retrieve access token
Uses useAuth0(event) server-side composable
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Nuxt application using the @auth0/auth0-nuxt module, with proper configuration in nuxt.config.ts (domain, clientId, clientSecret, sessionSecret, appBaseUrl, audience), login/logout via /auth/login and /auth/logout, user profile display with useUser(), a /profile route protected by a route middleware, and server-side getAccessToken() for authenticated API calls?
30
Claude Opus 4.6
Vue
100%13/13
93%-7%
19/19$0.16
1m 18s
With Auth0 Tools·93%·Tool Calls 21
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
6.7/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
93/100 ptsA
Grader Results (19/19 passed)
L1 · Positive Presence(7/7)
Uses @auth0/auth0-vue SDK
Sets up Auth0 plugin with createAuth0
Uses useAuth0 composable
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile information
L2 · Hallucination Detection(3/3)
Does not use React SDK in Vue app
No hallucinated @auth0/vue3-auth0 package
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Plugin installed via app.use(createAuth0(...))
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated?
L5 · Version Correctness(3/3)
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Uses clientId (not deprecated client_id) in createAuth0 config
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Vue 3 SPA with the @auth0/auth0-vue plugin, useAuth0 composable, login, logout, user profile display, route protection, and getAccessTokenSilently to make authenticated API calls?
31
Claude Sonnet 4.6
Vue
100%13/13
93%-7%
18/19$0.18
2m 30s
With Auth0 Tools·93%·Tool Calls 17
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
8.2/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
23.2/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
93/100 ptsA
Grader Results (18/19 passed)
L1 · Positive Presence(7/7)
Uses @auth0/auth0-vue SDK
Sets up Auth0 plugin with createAuth0
Uses useAuth0 composable
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile information
L2 · Hallucination Detection(3/3)
Does not use React SDK in Vue app
No hallucinated @auth0/vue3-auth0 package
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(2/3)
Plugin installed via app.use(createAuth0(...))
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated?
L5 · Version Correctness(3/3)
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Uses clientId (not deprecated client_id) in createAuth0 config
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Vue 3 SPA with the @auth0/auth0-vue plugin, useAuth0 composable, login, logout, user profile display, route protection, and getAccessTokenSilently to make authenticated API calls?
32
GPT-5.4
Vue
100%13/13
93%-7%
19/19$0.76
2m 25s
With Auth0 Tools·93%·Tool Calls 20
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
7/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
93/100 ptsA
Grader Results (19/19 passed)
L1 · Positive Presence(7/7)
Uses @auth0/auth0-vue SDK
Sets up Auth0 plugin with createAuth0
Uses useAuth0 composable
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile information
L2 · Hallucination Detection(3/3)
Does not use React SDK in Vue app
No hallucinated @auth0/vue3-auth0 package
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Plugin installed via app.use(createAuth0(...))
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated?
L5 · Version Correctness(3/3)
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Uses clientId (not deprecated client_id) in createAuth0 config
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Vue 3 SPA with the @auth0/auth0-vue plugin, useAuth0 composable, login, logout, user profile display, route protection, and getAccessTokenSilently to make authenticated API calls?
33
Gemini 3.1 Pro
Express
80%12/15
93%+13%
21/22$0.40
3m 4s
With Auth0 Tools·93%·Tool Calls 16
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
8.8/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
23.3/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
93/100 ptsA
Grader Results (21/22 passed)
L1 · Positive Presence(7/7)
Uses express-openid-connect SDK
Configures issuerBaseURL
Configures clientID
Configures baseURL
Configures session secret
Uses req.oidc to access OIDC context
Accesses user profile via req.oidc.user
L2 · Hallucination Detection(4/4)
No hallucinated @auth0/auth0-spa-js (wrong SDK for server-side app)
No express-oauth2-jwt-bearer (that is for APIs, not web apps)
Does not use Next.js SDK in Express app
No passport middleware (not needed with express-openid-connect)
L3 · Security(3/3)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, session secret) stored in environment variables, .env or .env.local files, not hardcoded in source code?
L4 · Structural(4/4)
auth middleware registered with app.use
Uses requiresAuth() to protect the /profile route
Accesses access token via req.oidc.accessToken
Does the app correctly register the auth() middleware, protect the /profile route with requiresAuth(), display user profile information, and include a route that calls an external API using the access token?
L5 · Version Correctness(3/3)
Uses authorizationParams to pass audience and scope
authorizationParams contains audience 'https://api.barkbook.com'
Does the solution use current express-openid-connect patterns? Specifically: does it use issuerBaseURL (not AUTH0_DOMAIN or domain directly), configure audience via authorizationParams (not as a top-level config key), and use response_type: "code" for the authorization code flow?
Holistic(0/1)
Does the solution correctly integrate Auth0 into an Express web app using express-openid-connect?
34
GPT-5.4
Android
67%6/9
93%+26%
14/14$0.53
1m 36s
With Auth0 Tools·93%·Tool Calls 21
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
6.7/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
93/100 ptsA
Grader Results (14/14 passed)
L1 · Positive Presence(5/5)
Uses Auth0 Android SDK dependency
Uses WebAuthProvider for authentication
Calls WebAuthProvider.login() for sign-in
Calls WebAuthProvider.logout() for sign-out
Uses CredentialsManager for secure token storage
L2 · Hallucination Detection(1/1)
No auth0-java (server-side SDK, not for Android)
L3 · Security(2/2)
No hardcoded client ID in Kotlin source files (ok in strings.xml)
No hardcoded domain in Kotlin source files (ok in strings.xml)
L4 · Structural(3/3)
Configures manifestPlaceholders in build.gradle for Auth0 callback URL scheme
Does the code implement both onSuccess and onFailure callbacks for WebAuthProvider.login(...) and WebAuthProvider.logout(...), with AuthenticationException handled in onFailure?
Is the Auth0 callback URL scheme correctly handled?
L5 · Version Correctness(2/2)
Uses Auth0(context) or Auth0.getInstance(context) for auto-configuration from string resources
Does the code use current Auth0 Android SDK v2+ patterns? Specifically: Auth0(context) for auto-configuration from strings.xml, WebAuthProvider for browser-based login/logout, and CredentialsManager for secure credential storage rather than persisting tokens manually in SharedPreferences?
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Android app with WebAuthProvider login and logout, CredentialsManager for secure token storage, Auth0 credentials configured via string resources, and manifestPlaceholders for callback URL handling?
35
GPT-5.4
Express
87%13/15
91%+4%
22/22$0.71
2m 28s
With Auth0 Tools·91%·Tool Calls 18
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12.6/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
7.8/14 pts
Error Recovery
When something broke, did the agent fix it?
6.4/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
91/100 ptsA
Grader Results (22/22 passed)
L1 · Positive Presence(7/7)
Uses express-openid-connect SDK
Configures issuerBaseURL
Configures clientID
Configures baseURL
Configures session secret
Uses req.oidc to access OIDC context
Accesses user profile via req.oidc.user
L2 · Hallucination Detection(4/4)
No hallucinated @auth0/auth0-spa-js (wrong SDK for server-side app)
No express-oauth2-jwt-bearer (that is for APIs, not web apps)
Does not use Next.js SDK in Express app
No passport middleware (not needed with express-openid-connect)
L3 · Security(3/3)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, session secret) stored in environment variables, .env or .env.local files, not hardcoded in source code?
L4 · Structural(4/4)
auth middleware registered with app.use
Uses requiresAuth() to protect the /profile route
Accesses access token via req.oidc.accessToken
Does the app correctly register the auth() middleware, protect the /profile route with requiresAuth(), display user profile information, and include a route that calls an external API using the access token?
L5 · Version Correctness(3/3)
Uses authorizationParams to pass audience and scope
authorizationParams contains audience 'https://api.barkbook.com'
Does the solution use current express-openid-connect patterns? Specifically: does it use issuerBaseURL (not AUTH0_DOMAIN or domain directly), configure audience via authorizationParams (not as a top-level config key), and use response_type: "code" for the authorization code flow?
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Express web app using express-openid-connect?
36
Gemini 3.1 Pro
Swift
80%8/10
91%+11%
13/14$0.56
5m 10s
With Auth0 Tools·91%·Tool Calls 18
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
7.8/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
22.2/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
91/100 ptsA
Grader Results (13/14 passed)
L1 · Positive Presence(4/4)
Uses Auth0 SDK
Imports Auth0 module
Uses webAuth() for login
Uses clearSession for logout
L2 · Hallucination Detection(3/3)
No hallucinated Auth0SDK package name (correct package is Auth0)
Does not use CocoaPods (SPM preferred)
Does not use deprecated completion handler pattern
L3 · Security(2/2)
No hardcoded client ID in Swift source files (ok in Auth0.plist)
No hardcoded domain in Swift source files (ok in Auth0.plist)
L4 · Structural(2/2)
Uses CredentialsManager for token storage
Does the code properly handle login and logout flows with appropriate error handling? Does it update the UI state after successful authentication?
L5 · Version Correctness(1/2)
Uses async/await webAuth().start() syntax (not completion handlers)
Does the code use modern Swift async/await patterns with the Auth0.swift SDK? Specifically: does it use try await webAuth().start() and CredentialsManager, and configure via Auth0.plist rather than hardcoded strings?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Swift iOS app with webAuth() login/logout, credential management, and proper SwiftUI state handling?
37
GPT-5.4
FastAPI
47%7/15
91%+44%
18/20$0.83
1m 54s
With Auth0 Tools·91%·Tool Calls 15
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
9.3/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
20.8/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
91/100 ptsA
Grader Results (18/20 passed)
L1 · Positive Presence(6/6)
Uses auth0-fastapi-api package
Imports Auth0FastAPI class
Uses require_auth() dependency
Uses FastAPI Depends for dependency injection
Configures domain
Configures audience
L2 · Hallucination Detection(5/5)
No python-jose (manual JWT parsing not needed)
No PyJWT direct usage
No manual JWT decoding
No fastapi-users (wrong package for Auth0 JWT)
No passlib (unrelated auth package)
L3 · Security(3/3)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are the Auth0 domain and audience stored in environment variables or a .env file, not hardcoded in Python source code?
L4 · Structural(3/4)
Auth0FastAPI instance is created
Route protected with Depends(auth0.require_auth())
read:messages scope checked on /api/messages route
Does the app correctly create an Auth0FastAPI instance, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token?
L5 · Version Correctness(1/1)
Does the solution use current auth0-fastapi-api patterns? Specifically: does it use Auth0FastAPI with domain and audience parameters, use require_auth() as a FastAPI Depends dependency (not a decorator), and read credentials from environment variables (not hardcoded)?
Holistic(0/1)
Does the solution correctly integrate Auth0 into a FastAPI API using auth0-fastapi-api?
38
Claude Opus 4.7
Angular
100%12/12
90%-10%
17/17$0.29
1m
With Auth0 Tools·90%·Tool Calls 21
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12.6/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
6.7/14 pts
Error Recovery
When something broke, did the agent fix it?
6.4/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
90/100 ptsB
Grader Results (17/17 passed)
L1 · Positive Presence(6/6)
Uses @auth0/auth0-angular SDK
Injects AuthService for authentication operations
Implements loginWithRedirect
Implements logout
Uses isAuthenticated$ observable for auth state
Uses user$ observable to display user profile
L2 · Hallucination Detection(3/3)
Does not use React SDK in Angular app
Does not use Vue SDK in Angular app
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Auth0 configured via provideAuth0()
Route protected with AuthGuard or authGuardFn
Uses getAccessTokenSilently or httpInterceptor for authenticated API calls
L5 · Version Correctness(2/2)
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Angular SPA using @auth0/auth0-angular, with provideAuth0 setup, AuthService for login/logout, user profile display, route protection via a guard, and authenticated API calls (either by configuring authHttpInterceptorFn with an allowedList matching the API base URL, or by calling getAccessTokenSilently to retrieve tokens manually)?
39
Claude Opus 4.7
Next.js
71%10/14
90%+19%
22/25$0.20
1m 53s
With Auth0 Tools·90%·Tool Calls 12
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
11.7/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
22.4/25 pts
Hallucination
Did the agent invent fake packages or methods?
10/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
90/100 ptsA
Grader Results (22/25 passed)
L1 · Positive Presence(8/8)
Uses @auth0/nextjs-auth0 SDK
Uses v4 server import path
Instantiates Auth0Client
Configures AUTH0_CLIENT_ID
Configures AUTH0_CLIENT_SECRET
Configures AUTH0_SECRET
Configures AUTH0_DOMAIN
Uses getSession for session retrieval
L2 · Hallucination Detection(2/3)
No hallucinated @auth0/nextjs-sdk package
No hallucinated @auth0/nextjs (must be @auth0/nextjs-auth0)
Does not use SPA SDK in server app
L3 · Security(3/3)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, AUTH0_SECRET) stored in environment variables or .env files, not hardcoded in source code?
L4 · Structural(3/5)
Middleware function is exported from middleware file
Uses auth0.middleware in middleware file
Dashboard page file exists
Uses /auth/login for login redirect
Does the code set up a working authentication flow with login, logout, and a callback route? Is there a protected /dashboard page that checks the user session and redirects unauthenticated users to log in?
L5 · Version Correctness(6/6)
Does not use v3 env var AUTH0_BASE_URL (v4 uses APP_BASE_URL)
Does not use v3 env var AUTH0_ISSUER_BASE_URL (removed in v4)
Does not use v3 handleAuth (v4 uses middleware)
Does not use v3 route prefix /api/auth/ (v4 uses /auth/)
Does the solution correctly integrate Auth0 into a Next.js App Router app using Auth0Client from @auth0/nextjs-auth0/server, middleware-based auth routing, and getSession for server-side session access?
Does the code use current v4 environment variable names (AUTH0_DOMAIN, AUTH0_CLIENT_ID, AUTH0_CLIENT_SECRET, AUTH0_SECRET) and NOT deprecated v3 names (AUTH0_BASE_URL, AUTH0_ISSUER_BASE_URL)?
40
Claude Opus 4.6
Nuxt
55%11/20
89%+34%
29/30$0.16
2m 56s
With Auth0 Tools·89%·Tool Calls 21
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
11.1/14 pts
Efficiency
How many steps did it take? Fewer is better.
6.7/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
23.8/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
89/100 ptsB
Grader Results (29/30 passed)
L1 · Positive Presence(10/10)
Uses @auth0/auth0-nuxt SDK
Registers module in nuxt.config
Configures runtimeConfig with Auth0 settings
Configures NUXT_AUTH0_DOMAIN env var
Configures NUXT_AUTH0_CLIENT_ID env var
Configures NUXT_AUTH0_CLIENT_SECRET env var
Configures NUXT_AUTH0_SESSION_SECRET env var
Uses useUser() composable to access the authenticated user
Uses /auth/login route to initiate login
Uses /auth/logout route to log out
L2 · Hallucination Detection(4/4)
Does not use the Vue SDK in a Nuxt app
Does not use the React SDK in a Nuxt app
Does not use loginWithRedirect (SPA API not applicable to Nuxt)
Does not use getAccessTokenSilently (Vue/React SPA API, not used in Nuxt)
L3 · Security(5/5)
No hardcoded client ID in source files (ok in .env)
No hardcoded client secret in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, session secret) stored in environment variables or .env files using NUXT_AUTH0_* names, not hardcoded in nuxt.config.ts?
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(6/6)
Module correctly registered in the modules array
sessionSecret configured for session encryption
clientSecret configured (Regular Web Application)
appBaseUrl configured in runtimeConfig
definePageMeta used for page-level middleware
Is there a protected /profile route that uses a Nuxt route middleware (via definePageMeta) to check authentication with useUser() and redirect unauthenticated users to /auth/login?
L5 · Version Correctness(4/4)
Audience configured in runtimeConfig for API access
Audience set to 'https://api.playground.com'
Uses getAccessToken() server-side to retrieve access token
Uses useAuth0(event) server-side composable
Holistic(0/1)
Does the solution correctly integrate Auth0 into a Nuxt application using the @auth0/auth0-nuxt module, with proper configuration in nuxt.config.ts (domain, clientId, clientSecret, sessionSecret, appBaseUrl, audience), login/logout via /auth/login and /auth/logout, user profile display with useUser(), a /profile route protected by a route middleware, and server-side getAccessToken() for authenticated API calls?
41
Claude Opus 4.6
FastAPI
47%7/15
89%+42%
19/20$0.20
1m 37s
With Auth0 Tools·89%·Tool Calls 18
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12.6/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
7.8/14 pts
Error Recovery
When something broke, did the agent fix it?
6.4/8 pts
Correctness
Does the generated code actually work?
22.9/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
89/100 ptsB
Grader Results (19/20 passed)
L1 · Positive Presence(6/6)
Uses auth0-fastapi-api package
Imports Auth0FastAPI class
Uses require_auth() dependency
Uses FastAPI Depends for dependency injection
Configures domain
Configures audience
L2 · Hallucination Detection(5/5)
No python-jose (manual JWT parsing not needed)
No PyJWT direct usage
No manual JWT decoding
No fastapi-users (wrong package for Auth0 JWT)
No passlib (unrelated auth package)
L3 · Security(3/3)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are the Auth0 domain and audience stored in environment variables or a .env file, not hardcoded in Python source code?
L4 · Structural(4/4)
Auth0FastAPI instance is created
Route protected with Depends(auth0.require_auth())
read:messages scope checked on /api/messages route
Does the app correctly create an Auth0FastAPI instance, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token?
L5 · Version Correctness(1/1)
Does the solution use current auth0-fastapi-api patterns? Specifically: does it use Auth0FastAPI with domain and audience parameters, use require_auth() as a FastAPI Depends dependency (not a decorator), and read credentials from environment variables (not hardcoded)?
Holistic(0/1)
Does the solution correctly integrate Auth0 into a FastAPI API using auth0-fastapi-api?
42
Claude Opus 4.6
Angular
100%12/12
88%-12%
17/17$0.22
3m 19s
With Auth0 Tools·88%·Tool Calls 26
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
10.1/14 pts
Efficiency
How many steps did it take? Fewer is better.
5.4/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
88/100 ptsB
Grader Results (17/17 passed)
L1 · Positive Presence(6/6)
Uses @auth0/auth0-angular SDK
Injects AuthService for authentication operations
Implements loginWithRedirect
Implements logout
Uses isAuthenticated$ observable for auth state
Uses user$ observable to display user profile
L2 · Hallucination Detection(3/3)
Does not use React SDK in Angular app
Does not use Vue SDK in Angular app
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Auth0 configured via provideAuth0()
Route protected with AuthGuard or authGuardFn
Uses getAccessTokenSilently or httpInterceptor for authenticated API calls
L5 · Version Correctness(2/2)
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Angular SPA using @auth0/auth0-angular, with provideAuth0 setup, AuthService for login/logout, user profile display, route protection via a guard, and authenticated API calls (either by configuring authHttpInterceptorFn with an allowedList matching the API base URL, or by calling getAccessTokenSilently to retrieve tokens manually)?
43
Claude Opus 4.7
Vue
100%13/13
88%-12%
18/19$0.26
1m 14s
With Auth0 Tools·88%·Tool Calls 22
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12.6/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
6.4/14 pts
Error Recovery
When something broke, did the agent fix it?
6.4/8 pts
Correctness
Does the generated code actually work?
23.2/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
88/100 ptsB
Grader Results (18/19 passed)
L1 · Positive Presence(7/7)
Uses @auth0/auth0-vue SDK
Sets up Auth0 plugin with createAuth0
Uses useAuth0 composable
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile information
L2 · Hallucination Detection(3/3)
Does not use React SDK in Vue app
No hallucinated @auth0/vue3-auth0 package
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(2/3)
Plugin installed via app.use(createAuth0(...))
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated?
L5 · Version Correctness(3/3)
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Uses clientId (not deprecated client_id) in createAuth0 config
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Vue 3 SPA with the @auth0/auth0-vue plugin, useAuth0 composable, login, logout, user profile display, route protection, and getAccessTokenSilently to make authenticated API calls?
44
Gemini 3.1 Pro
Express API
77%10/13
88%+11%
17/21$0.19
2m 37s
With Auth0 Tools·88%·Tool Calls 7
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
14/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
23.2/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
0/10 pts
Total weighted score
88/100 ptsB
Grader Results (17/21 passed)
L1 · Positive Presence(5/5)
Uses express-oauth2-jwt-bearer SDK
Configures issuerBaseURL
Configures audience claim
Uses requiredScopes() for scope-based route protection
Accesses JWT data via req.auth
L2 · Hallucination Detection(4/4)
No express-openid-connect (that is for web apps, not APIs)
No passport middleware (not needed with express-oauth2-jwt-bearer)
No manual JWT verification with jsonwebtoken (use SDK)
No SPA SDK used in server-side API
L3 · Security(0/3)
No hardcoded issuer domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are all Auth0 configuration values (issuerBaseURL / domain, audience) stored in environment variables, not hardcoded in source code?
L4 · Structural(3/4)
auth() middleware registered globally with app.use
GET /api/messages protected with read:messages scope
POST /api/messages protected with write:messages scope
Does the API correctly protect all three routes?
L5 · Version Correctness(4/4)
No manual Authorization header parsing (SDK handles extraction)
No manual jwt.verify calls (SDK handles verification)
No req.user (express-oauth2-jwt-bearer uses req.auth, not req.user)
Does the solution use current express-oauth2-jwt-bearer patterns? Specifically: does it configure auth() with issuerBaseURL and audience, use requiredScopes() for scope checks (not manual payload inspection), and access token data via req.auth.payload (not req.user or manually decoded tokens)?
Holistic(1/1)
Does the solution correctly protect an Express.js API using express-oauth2-jwt-bearer?
45
Claude Sonnet 4.6
Android
67%6/9
88%+21%
12/14$0.25
2m 48s
With Auth0 Tools·88%·Tool Calls 16
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
8.8/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
22.7/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
5/10 pts
Total weighted score
88/100 ptsB
Grader Results (12/14 passed)
L1 · Positive Presence(5/5)
Uses Auth0 Android SDK dependency
Uses WebAuthProvider for authentication
Calls WebAuthProvider.login() for sign-in
Calls WebAuthProvider.logout() for sign-out
Uses CredentialsManager for secure token storage
L2 · Hallucination Detection(1/1)
No auth0-java (server-side SDK, not for Android)
L3 · Security(1/2)
No hardcoded client ID in Kotlin source files (ok in strings.xml)
No hardcoded domain in Kotlin source files (ok in strings.xml)
L4 · Structural(3/3)
Configures manifestPlaceholders in build.gradle for Auth0 callback URL scheme
Does the code implement both onSuccess and onFailure callbacks for WebAuthProvider.login(...) and WebAuthProvider.logout(...), with AuthenticationException handled in onFailure?
Is the Auth0 callback URL scheme correctly handled?
L5 · Version Correctness(2/2)
Uses Auth0(context) or Auth0.getInstance(context) for auto-configuration from string resources
Does the code use current Auth0 Android SDK v2+ patterns? Specifically: Auth0(context) for auto-configuration from strings.xml, WebAuthProvider for browser-based login/logout, and CredentialsManager for secure credential storage rather than persisting tokens manually in SharedPreferences?
Holistic(0/1)
Does the solution correctly integrate Auth0 into an Android app with WebAuthProvider login and logout, CredentialsManager for secure token storage, Auth0 credentials configured via string resources, and manifestPlaceholders for callback URL handling?
46
Claude Sonnet 4.6
Nuxt
35%7/20
88%+53%
29/30$0.18
3m 56s
With Auth0 Tools·88%·Tool Calls 21
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
10.4/14 pts
Efficiency
How many steps did it take? Fewer is better.
6.7/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
23.8/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
88/100 ptsB
Grader Results (29/30 passed)
L1 · Positive Presence(10/10)
Uses @auth0/auth0-nuxt SDK
Registers module in nuxt.config
Configures runtimeConfig with Auth0 settings
Configures NUXT_AUTH0_DOMAIN env var
Configures NUXT_AUTH0_CLIENT_ID env var
Configures NUXT_AUTH0_CLIENT_SECRET env var
Configures NUXT_AUTH0_SESSION_SECRET env var
Uses useUser() composable to access the authenticated user
Uses /auth/login route to initiate login
Uses /auth/logout route to log out
L2 · Hallucination Detection(4/4)
Does not use the Vue SDK in a Nuxt app
Does not use the React SDK in a Nuxt app
Does not use loginWithRedirect (SPA API not applicable to Nuxt)
Does not use getAccessTokenSilently (Vue/React SPA API, not used in Nuxt)
L3 · Security(5/5)
No hardcoded client ID in source files (ok in .env)
No hardcoded client secret in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, session secret) stored in environment variables or .env files using NUXT_AUTH0_* names, not hardcoded in nuxt.config.ts?
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(6/6)
Module correctly registered in the modules array
sessionSecret configured for session encryption
clientSecret configured (Regular Web Application)
appBaseUrl configured in runtimeConfig
definePageMeta used for page-level middleware
Is there a protected /profile route that uses a Nuxt route middleware (via definePageMeta) to check authentication with useUser() and redirect unauthenticated users to /auth/login?
L5 · Version Correctness(4/4)
Audience configured in runtimeConfig for API access
Audience set to 'https://api.playground.com'
Uses getAccessToken() server-side to retrieve access token
Uses useAuth0(event) server-side composable
Holistic(0/1)
Does the solution correctly integrate Auth0 into a Nuxt application using the @auth0/auth0-nuxt module, with proper configuration in nuxt.config.ts (domain, clientId, clientSecret, sessionSecret, appBaseUrl, audience), login/logout via /auth/login and /auth/logout, user profile display with useUser(), a /profile route protected by a route middleware, and server-side getAccessToken() for authenticated API calls?
47
Claude Sonnet 4.6
Swift
70%7/10
87%+17%
13/14$0.17
1m 35s
With Auth0 Tools·87%·Tool Calls 15
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
11.2/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
9.3/14 pts
Error Recovery
When something broke, did the agent fix it?
4.8/8 pts
Correctness
Does the generated code actually work?
22.2/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
87/100 ptsB
Grader Results (13/14 passed)
L1 · Positive Presence(4/4)
Uses Auth0 SDK
Imports Auth0 module
Uses webAuth() for login
Uses clearSession for logout
L2 · Hallucination Detection(3/3)
No hallucinated Auth0SDK package name (correct package is Auth0)
Does not use CocoaPods (SPM preferred)
Does not use deprecated completion handler pattern
L3 · Security(2/2)
No hardcoded client ID in Swift source files (ok in Auth0.plist)
No hardcoded domain in Swift source files (ok in Auth0.plist)
L4 · Structural(2/2)
Uses CredentialsManager for token storage
Does the code properly handle login and logout flows with appropriate error handling? Does it update the UI state after successful authentication?
L5 · Version Correctness(1/2)
Uses async/await webAuth().start() syntax (not completion handlers)
Does the code use modern Swift async/await patterns with the Auth0.swift SDK? Specifically: does it use try await webAuth().start() and CredentialsManager, and configure via Auth0.plist rather than hardcoded strings?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Swift iOS app with webAuth() login/logout, credential management, and proper SwiftUI state handling?
48
GPT-5.4
Angular
100%12/12
85%-15%
17/17$1.57
5m 21s
With Auth0 Tools·85%·Tool Calls 32
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12.6/14 pts
Setup Speed
How fast did the agent complete the task?
11.8/14 pts
Efficiency
How many steps did it take? Fewer is better.
4.4/14 pts
Error Recovery
When something broke, did the agent fix it?
6.4/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
85/100 ptsB
Grader Results (17/17 passed)
L1 · Positive Presence(6/6)
Uses @auth0/auth0-angular SDK
Injects AuthService for authentication operations
Implements loginWithRedirect
Implements logout
Uses isAuthenticated$ observable for auth state
Uses user$ observable to display user profile
L2 · Hallucination Detection(3/3)
Does not use React SDK in Angular app
Does not use Vue SDK in Angular app
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Auth0 configured via provideAuth0()
Route protected with AuthGuard or authGuardFn
Uses getAccessTokenSilently or httpInterceptor for authenticated API calls
L5 · Version Correctness(2/2)
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Angular SPA using @auth0/auth0-angular, with provideAuth0 setup, AuthService for login/logout, user profile display, route protection via a guard, and authenticated API calls (either by configuring authHttpInterceptorFn with an allowedList matching the API base URL, or by calling getAccessTokenSilently to retrieve tokens manually)?
49
GPT-5.4
Swift
80%8/10
85%+5%
12/14$0.76
3m 30s
With Auth0 Tools·85%·Tool Calls 23
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
12.6/14 pts
Efficiency
How many steps did it take? Fewer is better.
6.1/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
19.5/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
85/100 ptsB
Grader Results (12/14 passed)
L1 · Positive Presence(3/4)
Uses Auth0 SDK
Imports Auth0 module
Uses webAuth() for login
Uses clearSession for logout
L2 · Hallucination Detection(3/3)
No hallucinated Auth0SDK package name (correct package is Auth0)
Does not use CocoaPods (SPM preferred)
Does not use deprecated completion handler pattern
L3 · Security(2/2)
No hardcoded client ID in Swift source files (ok in Auth0.plist)
No hardcoded domain in Swift source files (ok in Auth0.plist)
L4 · Structural(2/2)
Uses CredentialsManager for token storage
Does the code properly handle login and logout flows with appropriate error handling? Does it update the UI state after successful authentication?
L5 · Version Correctness(1/2)
Uses async/await webAuth().start() syntax (not completion handlers)
Does the code use modern Swift async/await patterns with the Auth0.swift SDK? Specifically: does it use try await webAuth().start() and CredentialsManager, and configure via Auth0.plist rather than hardcoded strings?
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Swift iOS app with webAuth() login/logout, credential management, and proper SwiftUI state handling?
50
Claude Sonnet 4.6
Express API
77%10/13
85%+8%
19/21$0.17
1m 54s
With Auth0 Tools·85%·Tool Calls 14
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
11.2/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
10/14 pts
Error Recovery
When something broke, did the agent fix it?
4.8/8 pts
Correctness
Does the generated code actually work?
23.2/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
6.7/10 pts
Total weighted score
85/100 ptsB
Grader Results (19/21 passed)
L1 · Positive Presence(5/5)
Uses express-oauth2-jwt-bearer SDK
Configures issuerBaseURL
Configures audience claim
Uses requiredScopes() for scope-based route protection
Accesses JWT data via req.auth
L2 · Hallucination Detection(4/4)
No express-openid-connect (that is for web apps, not APIs)
No passport middleware (not needed with express-oauth2-jwt-bearer)
No manual JWT verification with jsonwebtoken (use SDK)
No SPA SDK used in server-side API
L3 · Security(2/3)
No hardcoded issuer domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are all Auth0 configuration values (issuerBaseURL / domain, audience) stored in environment variables, not hardcoded in source code?
L4 · Structural(3/4)
auth() middleware registered globally with app.use
GET /api/messages protected with read:messages scope
POST /api/messages protected with write:messages scope
Does the API correctly protect all three routes?
L5 · Version Correctness(4/4)
No manual Authorization header parsing (SDK handles extraction)
No manual jwt.verify calls (SDK handles verification)
No req.user (express-oauth2-jwt-bearer uses req.auth, not req.user)
Does the solution use current express-oauth2-jwt-bearer patterns? Specifically: does it configure auth() with issuerBaseURL and audience, use requiredScopes() for scope checks (not manual payload inspection), and access token data via req.auth.payload (not req.user or manually decoded tokens)?
Holistic(1/1)
Does the solution correctly protect an Express.js API using express-oauth2-jwt-bearer?
51
Claude Opus 4.6
Next.js
71%10/14
85%+14%
22/25$0.18
1m 36s
With Auth0 Tools·85%·Tool Calls 21
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
6.7/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
22.4/25 pts
Hallucination
Did the agent invent fake packages or methods?
10/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
85/100 ptsB
Grader Results (22/25 passed)
L1 · Positive Presence(8/8)
Uses @auth0/nextjs-auth0 SDK
Uses v4 server import path
Instantiates Auth0Client
Configures AUTH0_CLIENT_ID
Configures AUTH0_CLIENT_SECRET
Configures AUTH0_SECRET
Configures AUTH0_DOMAIN
Uses getSession for session retrieval
L2 · Hallucination Detection(2/3)
No hallucinated @auth0/nextjs-sdk package
No hallucinated @auth0/nextjs (must be @auth0/nextjs-auth0)
Does not use SPA SDK in server app
L3 · Security(3/3)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, AUTH0_SECRET) stored in environment variables or .env files, not hardcoded in source code?
L4 · Structural(3/5)
Middleware function is exported from middleware file
Uses auth0.middleware in middleware file
Dashboard page file exists
Uses /auth/login for login redirect
Does the code set up a working authentication flow with login, logout, and a callback route? Is there a protected /dashboard page that checks the user session and redirects unauthenticated users to log in?
L5 · Version Correctness(6/6)
Does not use v3 env var AUTH0_BASE_URL (v4 uses APP_BASE_URL)
Does not use v3 env var AUTH0_ISSUER_BASE_URL (removed in v4)
Does not use v3 handleAuth (v4 uses middleware)
Does not use v3 route prefix /api/auth/ (v4 uses /auth/)
Does the solution correctly integrate Auth0 into a Next.js App Router app using Auth0Client from @auth0/nextjs-auth0/server, middleware-based auth routing, and getSession for server-side session access?
Does the code use current v4 environment variable names (AUTH0_DOMAIN, AUTH0_CLIENT_ID, AUTH0_CLIENT_SECRET, AUTH0_SECRET) and NOT deprecated v3 names (AUTH0_BASE_URL, AUTH0_ISSUER_BASE_URL)?
52
Claude Sonnet 4.6
Angular
100%12/12
84%-16%
17/17$0.38
5m 16s
With Auth0 Tools·84%·Tool Calls 24
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
11.2/14 pts
Setup Speed
How fast did the agent complete the task?
11.6/14 pts
Efficiency
How many steps did it take? Fewer is better.
5.8/14 pts
Error Recovery
When something broke, did the agent fix it?
4.8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
84/100 ptsB
Grader Results (17/17 passed)
L1 · Positive Presence(6/6)
Uses @auth0/auth0-angular SDK
Injects AuthService for authentication operations
Implements loginWithRedirect
Implements logout
Uses isAuthenticated$ observable for auth state
Uses user$ observable to display user profile
L2 · Hallucination Detection(3/3)
Does not use React SDK in Angular app
Does not use Vue SDK in Angular app
No client_secret in SPA (public client, no secrets)
L3 · Security(2/2)
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(3/3)
Auth0 configured via provideAuth0()
Route protected with AuthGuard or authGuardFn
Uses getAccessTokenSilently or httpInterceptor for authenticated API calls
L5 · Version Correctness(2/2)
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Holistic(1/1)
Does the solution correctly integrate Auth0 into an Angular SPA using @auth0/auth0-angular, with provideAuth0 setup, AuthService for login/logout, user profile display, route protection via a guard, and authenticated API calls (either by configuring authHttpInterceptorFn with an allowedList matching the API base URL, or by calling getAccessTokenSilently to retrieve tokens manually)?
53
GPT-5.4
Express API
77%10/13
84%+7%
17/21$0.66
2m 19s
With Auth0 Tools·84%·Tool Calls 14
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
10/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
23.2/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
0/10 pts
Total weighted score
84/100 ptsB
Grader Results (17/21 passed)
L1 · Positive Presence(5/5)
Uses express-oauth2-jwt-bearer SDK
Configures issuerBaseURL
Configures audience claim
Uses requiredScopes() for scope-based route protection
Accesses JWT data via req.auth
L2 · Hallucination Detection(4/4)
No express-openid-connect (that is for web apps, not APIs)
No passport middleware (not needed with express-oauth2-jwt-bearer)
No manual JWT verification with jsonwebtoken (use SDK)
No SPA SDK used in server-side API
L3 · Security(0/3)
No hardcoded issuer domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are all Auth0 configuration values (issuerBaseURL / domain, audience) stored in environment variables, not hardcoded in source code?
L4 · Structural(3/4)
auth() middleware registered globally with app.use
GET /api/messages protected with read:messages scope
POST /api/messages protected with write:messages scope
Does the API correctly protect all three routes?
L5 · Version Correctness(4/4)
No manual Authorization header parsing (SDK handles extraction)
No manual jwt.verify calls (SDK handles verification)
No req.user (express-oauth2-jwt-bearer uses req.auth, not req.user)
Does the solution use current express-oauth2-jwt-bearer patterns? Specifically: does it configure auth() with issuerBaseURL and audience, use requiredScopes() for scope checks (not manual payload inspection), and access token data via req.auth.payload (not req.user or manually decoded tokens)?
Holistic(1/1)
Does the solution correctly protect an Express.js API using express-oauth2-jwt-bearer?
54
Claude Opus 4.6
Swift
70%7/10
84%+14%
12/14$0.15
1m
With Auth0 Tools·84%·Tool Calls 14
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
11.2/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
10/14 pts
Error Recovery
When something broke, did the agent fix it?
4.8/8 pts
Correctness
Does the generated code actually work?
19.5/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
84/100 ptsB
Grader Results (12/14 passed)
L1 · Positive Presence(4/4)
Uses Auth0 SDK
Imports Auth0 module
Uses webAuth() for login
Uses clearSession for logout
L2 · Hallucination Detection(3/3)
No hallucinated Auth0SDK package name (correct package is Auth0)
Does not use CocoaPods (SPM preferred)
Does not use deprecated completion handler pattern
L3 · Security(2/2)
No hardcoded client ID in Swift source files (ok in Auth0.plist)
No hardcoded domain in Swift source files (ok in Auth0.plist)
L4 · Structural(2/2)
Uses CredentialsManager for token storage
Does the code properly handle login and logout flows with appropriate error handling? Does it update the UI state after successful authentication?
L5 · Version Correctness(1/2)
Uses async/await webAuth().start() syntax (not completion handlers)
Does the code use modern Swift async/await patterns with the Auth0.swift SDK? Specifically: does it use try await webAuth().start() and CredentialsManager, and configure via Auth0.plist rather than hardcoded strings?
Holistic(0/1)
Does the solution correctly integrate Auth0 into a Swift iOS app with webAuth() login/logout, credential management, and proper SwiftUI state handling?
55
Gemini 3.1 Pro
Next.js
57%8/14
83%+26%
21/25$0.71
5m 41s
With Auth0 Tools·83%·Tool Calls 24
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
5.8/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
21.1/25 pts
Hallucination
Did the agent invent fake packages or methods?
10/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
83/100 ptsB
Grader Results (21/25 passed)
L1 · Positive Presence(8/8)
Uses @auth0/nextjs-auth0 SDK
Uses v4 server import path
Instantiates Auth0Client
Configures AUTH0_CLIENT_ID
Configures AUTH0_CLIENT_SECRET
Configures AUTH0_SECRET
Configures AUTH0_DOMAIN
Uses getSession for session retrieval
L2 · Hallucination Detection(2/3)
No hallucinated @auth0/nextjs-sdk package
No hallucinated @auth0/nextjs (must be @auth0/nextjs-auth0)
Does not use SPA SDK in server app
L3 · Security(3/3)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, AUTH0_SECRET) stored in environment variables or .env files, not hardcoded in source code?
L4 · Structural(3/5)
Middleware function is exported from middleware file
Uses auth0.middleware in middleware file
Dashboard page file exists
Uses /auth/login for login redirect
Does the code set up a working authentication flow with login, logout, and a callback route? Is there a protected /dashboard page that checks the user session and redirects unauthenticated users to log in?
L5 · Version Correctness(5/6)
Does not use v3 env var AUTH0_BASE_URL (v4 uses APP_BASE_URL)
Does not use v3 env var AUTH0_ISSUER_BASE_URL (removed in v4)
Does not use v3 handleAuth (v4 uses middleware)
Does not use v3 route prefix /api/auth/ (v4 uses /auth/)
Does the solution correctly integrate Auth0 into a Next.js App Router app using Auth0Client from @auth0/nextjs-auth0/server, middleware-based auth routing, and getSession for server-side session access?
Does the code use current v4 environment variable names (AUTH0_DOMAIN, AUTH0_CLIENT_ID, AUTH0_CLIENT_SECRET, AUTH0_SECRET) and NOT deprecated v3 names (AUTH0_BASE_URL, AUTH0_ISSUER_BASE_URL)?
56
GPT-5.4
Next.js
79%11/14
82%+3%
21/25$0.63
2m 25s
With Auth0 Tools·82%·Tool Calls 26
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
5.4/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
21.1/25 pts
Hallucination
Did the agent invent fake packages or methods?
10/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
82/100 ptsB
Grader Results (21/25 passed)
L1 · Positive Presence(8/8)
Uses @auth0/nextjs-auth0 SDK
Uses v4 server import path
Instantiates Auth0Client
Configures AUTH0_CLIENT_ID
Configures AUTH0_CLIENT_SECRET
Configures AUTH0_SECRET
Configures AUTH0_DOMAIN
Uses getSession for session retrieval
L2 · Hallucination Detection(2/3)
No hallucinated @auth0/nextjs-sdk package
No hallucinated @auth0/nextjs (must be @auth0/nextjs-auth0)
Does not use SPA SDK in server app
L3 · Security(3/3)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, AUTH0_SECRET) stored in environment variables or .env files, not hardcoded in source code?
L4 · Structural(3/5)
Middleware function is exported from middleware file
Uses auth0.middleware in middleware file
Dashboard page file exists
Uses /auth/login for login redirect
Does the code set up a working authentication flow with login, logout, and a callback route? Is there a protected /dashboard page that checks the user session and redirects unauthenticated users to log in?
L5 · Version Correctness(5/6)
Does not use v3 env var AUTH0_BASE_URL (v4 uses APP_BASE_URL)
Does not use v3 env var AUTH0_ISSUER_BASE_URL (removed in v4)
Does not use v3 handleAuth (v4 uses middleware)
Does not use v3 route prefix /api/auth/ (v4 uses /auth/)
Does the solution correctly integrate Auth0 into a Next.js App Router app using Auth0Client from @auth0/nextjs-auth0/server, middleware-based auth routing, and getSession for server-side session access?
Does the code use current v4 environment variable names (AUTH0_DOMAIN, AUTH0_CLIENT_ID, AUTH0_CLIENT_SECRET, AUTH0_SECRET) and NOT deprecated v3 names (AUTH0_BASE_URL, AUTH0_ISSUER_BASE_URL)?
57
Claude Sonnet 4.6
FastAPI
40%6/15
81%+41%
19/20$0.24
2m 20s
With Auth0 Tools·81%·Tool Calls 22
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
9.8/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
6.4/14 pts
Error Recovery
When something broke, did the agent fix it?
3.2/8 pts
Correctness
Does the generated code actually work?
22.9/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
81/100 ptsB
Grader Results (19/20 passed)
L1 · Positive Presence(6/6)
Uses auth0-fastapi-api package
Imports Auth0FastAPI class
Uses require_auth() dependency
Uses FastAPI Depends for dependency injection
Configures domain
Configures audience
L2 · Hallucination Detection(5/5)
No python-jose (manual JWT parsing not needed)
No PyJWT direct usage
No manual JWT decoding
No fastapi-users (wrong package for Auth0 JWT)
No passlib (unrelated auth package)
L3 · Security(3/3)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are the Auth0 domain and audience stored in environment variables or a .env file, not hardcoded in Python source code?
L4 · Structural(4/4)
Auth0FastAPI instance is created
Route protected with Depends(auth0.require_auth())
read:messages scope checked on /api/messages route
Does the app correctly create an Auth0FastAPI instance, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token?
L5 · Version Correctness(1/1)
Does the solution use current auth0-fastapi-api patterns? Specifically: does it use Auth0FastAPI with domain and audience parameters, use require_auth() as a FastAPI Depends dependency (not a decorator), and read credentials from environment variables (not hardcoded)?
Holistic(0/1)
Does the solution correctly integrate Auth0 into a FastAPI API using auth0-fastapi-api?
58
Gemini 3.1 Pro
Nuxt
30%6/20
80%+50%
29/30$0.48
15m 23s
With Auth0 Tools·80%·Tool Calls 16
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
0/14 pts
Efficiency
How many steps did it take? Fewer is better.
8.8/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
23.8/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
80/100 ptsB
Grader Results (29/30 passed)
L1 · Positive Presence(10/10)
Uses @auth0/auth0-nuxt SDK
Registers module in nuxt.config
Configures runtimeConfig with Auth0 settings
Configures NUXT_AUTH0_DOMAIN env var
Configures NUXT_AUTH0_CLIENT_ID env var
Configures NUXT_AUTH0_CLIENT_SECRET env var
Configures NUXT_AUTH0_SESSION_SECRET env var
Uses useUser() composable to access the authenticated user
Uses /auth/login route to initiate login
Uses /auth/logout route to log out
L2 · Hallucination Detection(4/4)
Does not use the Vue SDK in a Nuxt app
Does not use the React SDK in a Nuxt app
Does not use loginWithRedirect (SPA API not applicable to Nuxt)
Does not use getAccessTokenSilently (Vue/React SPA API, not used in Nuxt)
L3 · Security(5/5)
No hardcoded client ID in source files (ok in .env)
No hardcoded client secret in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, session secret) stored in environment variables or .env files using NUXT_AUTH0_* names, not hardcoded in nuxt.config.ts?
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(6/6)
Module correctly registered in the modules array
sessionSecret configured for session encryption
clientSecret configured (Regular Web Application)
appBaseUrl configured in runtimeConfig
definePageMeta used for page-level middleware
Is there a protected /profile route that uses a Nuxt route middleware (via definePageMeta) to check authentication with useUser() and redirect unauthenticated users to /auth/login?
L5 · Version Correctness(4/4)
Audience configured in runtimeConfig for API access
Audience set to 'https://api.playground.com'
Uses getAccessToken() server-side to retrieve access token
Uses useAuth0(event) server-side composable
Holistic(0/1)
Does the solution correctly integrate Auth0 into a Nuxt application using the @auth0/auth0-nuxt module, with proper configuration in nuxt.config.ts (domain, clientId, clientSecret, sessionSecret, appBaseUrl, audience), login/logout via /auth/login and /auth/logout, user profile display with useUser(), a /profile route protected by a route middleware, and server-side getAccessToken() for authenticated API calls?
59
GPT-5.4
Nuxt
35%7/20
79%+44%
30/30$2.75
7m 7s
With Auth0 Tools·79%·Tool Calls 52
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
11.2/14 pts
Setup Speed
How fast did the agent complete the task?
10.2/14 pts
Efficiency
How many steps did it take? Fewer is better.
2.7/14 pts
Error Recovery
When something broke, did the agent fix it?
4.8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
79/100 ptsB
Grader Results (30/30 passed)
L1 · Positive Presence(10/10)
Uses @auth0/auth0-nuxt SDK
Registers module in nuxt.config
Configures runtimeConfig with Auth0 settings
Configures NUXT_AUTH0_DOMAIN env var
Configures NUXT_AUTH0_CLIENT_ID env var
Configures NUXT_AUTH0_CLIENT_SECRET env var
Configures NUXT_AUTH0_SESSION_SECRET env var
Uses useUser() composable to access the authenticated user
Uses /auth/login route to initiate login
Uses /auth/logout route to log out
L2 · Hallucination Detection(4/4)
Does not use the Vue SDK in a Nuxt app
Does not use the React SDK in a Nuxt app
Does not use loginWithRedirect (SPA API not applicable to Nuxt)
Does not use getAccessTokenSilently (Vue/React SPA API, not used in Nuxt)
L3 · Security(5/5)
No hardcoded client ID in source files (ok in .env)
No hardcoded client secret in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, session secret) stored in environment variables or .env files using NUXT_AUTH0_* names, not hardcoded in nuxt.config.ts?
No tokens stored in localStorage
No tokens stored in sessionStorage
L4 · Structural(6/6)
Module correctly registered in the modules array
sessionSecret configured for session encryption
clientSecret configured (Regular Web Application)
appBaseUrl configured in runtimeConfig
definePageMeta used for page-level middleware
Is there a protected /profile route that uses a Nuxt route middleware (via definePageMeta) to check authentication with useUser() and redirect unauthenticated users to /auth/login?
L5 · Version Correctness(4/4)
Audience configured in runtimeConfig for API access
Audience set to 'https://api.playground.com'
Uses getAccessToken() server-side to retrieve access token
Uses useAuth0(event) server-side composable
Holistic(1/1)
Does the solution correctly integrate Auth0 into a Nuxt application using the @auth0/auth0-nuxt module, with proper configuration in nuxt.config.ts (domain, clientId, clientSecret, sessionSecret, appBaseUrl, audience), login/logout via /auth/login and /auth/logout, user profile display with useUser(), a /profile route protected by a route middleware, and server-side getAccessToken() for authenticated API calls?
60
Claude Sonnet 4.6
Next.js
50%7/14
77%+27%
21/25$0.27
3m 10s
With Auth0 Tools·77%·Tool Calls 22
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
11.2/14 pts
Setup Speed
How fast did the agent complete the task?
13.4/14 pts
Efficiency
How many steps did it take? Fewer is better.
6.4/14 pts
Error Recovery
When something broke, did the agent fix it?
4.8/8 pts
Correctness
Does the generated code actually work?
21.1/25 pts
Hallucination
Did the agent invent fake packages or methods?
10/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
77/100 ptsB
Grader Results (21/25 passed)
L1 · Positive Presence(8/8)
Uses @auth0/nextjs-auth0 SDK
Uses v4 server import path
Instantiates Auth0Client
Configures AUTH0_CLIENT_ID
Configures AUTH0_CLIENT_SECRET
Configures AUTH0_SECRET
Configures AUTH0_DOMAIN
Uses getSession for session retrieval
L2 · Hallucination Detection(2/3)
No hallucinated @auth0/nextjs-sdk package
No hallucinated @auth0/nextjs (must be @auth0/nextjs-auth0)
Does not use SPA SDK in server app
L3 · Security(3/3)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, AUTH0_SECRET) stored in environment variables or .env files, not hardcoded in source code?
L4 · Structural(3/5)
Middleware function is exported from middleware file
Uses auth0.middleware in middleware file
Dashboard page file exists
Uses /auth/login for login redirect
Does the code set up a working authentication flow with login, logout, and a callback route? Is there a protected /dashboard page that checks the user session and redirects unauthenticated users to log in?
L5 · Version Correctness(5/6)
Does not use v3 env var AUTH0_BASE_URL (v4 uses APP_BASE_URL)
Does not use v3 env var AUTH0_ISSUER_BASE_URL (removed in v4)
Does not use v3 handleAuth (v4 uses middleware)
Does not use v3 route prefix /api/auth/ (v4 uses /auth/)
Does the solution correctly integrate Auth0 into a Next.js App Router app using Auth0Client from @auth0/nextjs-auth0/server, middleware-based auth routing, and getSession for server-side session access?
Does the code use current v4 environment variable names (AUTH0_DOMAIN, AUTH0_CLIENT_ID, AUTH0_CLIENT_SECRET, AUTH0_SECRET) and NOT deprecated v3 names (AUTH0_BASE_URL, AUTH0_ISSUER_BASE_URL)?
* With Auth0 MCP + Skills enabledLast updated on April 2026
How we validate results
Every score is reproducible. Real integration tasks, automated grading, transparent methodology.
1
Real integration tasks
Real AI agents run identical Auth0 SDK prompts end-to-end. No synthetic benchmarks.
2
Automated grading
Pattern matching, security checks, and correctness validators score each response.
3
Tool comparison
Each model tested with and without MCP + Skills to measure improvement.
Start building with confidence
These tools are how we achieved the benchmark results above. Enable them in your AI agent to integrate Auth0 reliably.
MCP Server
API access for agents
Gives AI agents access to Auth0 Management APIs and current documentation.
https://auth0.com/docs/mcpSkills
Best practices for agents
Teaches agents Auth0 integration patterns so they follow security best practices.
npx skills add auth0/agent-skills --all