How do I troubleshoot HTTP 504 errors from an API Gateway REST API with a Lambda backend?
Last updated: 2022-12-14
I receive an HTTP 504 error status code when I invoke a REST API using Amazon API Gateway with an AWS Lambda backend. How do I troubleshoot this error?
Resolution
When your REST API requests return an HTTP 504 error status code, you must run various checks.
Check for IntegrationLatency spikes
Check if there's an IntegrationLatency spike in API Gateway by reviewing the integration latency duration time. To see the IntegrationLatency duration time, configure the access logging variable $context.integration.latency for HTTP API logging.
For more information, see Set up Amazon CloudWatch API logging using the API Gateway console.
An IntegrationLatency spike in API Gateway indicates that the request spent most of its time in Lambda. Check the Lambda function's Duration performance metric to confirm this.
For more information, see Working with Lambda function metrics.
Review requests with CloudWatch Logs Insights
Use Amazon CloudWatch Logs Insights to review requests that resulted in 504 errors. To review the requests, on the CloudWatch console, in the navigation pane, choose Logs , Log Insights . Select your API Gateway log group. Then, set the relative time with one of the following queries:
parse @message '(*) *' as reqId, message
| filter message like /Method completed with status: \d\d\d/
| parse message 'Method completed with status: *' as status
| filter status = 504
| sort @timestamp desc
| limit 20-or-
fields @timestamp, @message
| filter message like /Method completed with status: 504/
| sort @timestamp desc
| limit 20Instrument X-Ray traces
If 504 errors keep happening continuously, determine where the Lambda function is spending its time. Instrument AWS X-Ray traces for Lambda functions based on the Lambda function runtime.
For Python:
from aws_xray_sdk.core import xray_recorder
from aws_xray_sdk.core import patch_all
patch_all()For Node.js:
const AWSXRay = require('aws-xray-sdk-core')
const AWS = AWSXRay.captureAWS(require('aws-sdk'))Note: You need to build a new deployment package after instrumenting X-Ray traces on a Lambda function.
Implement API retries
When a 504 error occurs and the request isn't found in Lambda, implement API retries on the client. The error might have resulted from a temporary network failure in API Gateway.
Check the Lambda function configuration
Make sure that your Lambda function has only API Gateway event-specific processing logic. This way, the Lambda function takes less time to run and can keep up with incoming events.
REST APIs have a default maximum integration timeout of 29 seconds. Therefore, make sure that the Lambda function's running duration is under 29 seconds.
If you have a use case where the application is time sensitive, see Set up asynchronous invocation of the backend Lambda function.
Related information
API Gateway dimensions and metrics
Setting up CloudWatch logging for a REST API in API Gateway
View API Gateway log events in the CloudWatch console
Tracing user requests to REST APIs using X-Ray
Using Lambda with X-Ray
Did this article help?
Do you need billing or technical support?