SQSD error when starting a ElasticBeanstalk stack through CloudFormation - Wed, May 11, 2016
Cloudformation in AWS is a great way to setup your application stacks. Combine it with tools like Ansible, Chef, Puppet or even ElasticBeanstalk and it will become a portable solution that can easily move between your development and production environment.
But one of the problems using a lot of chained solutions can be troubleshooting.
One error that I’ve encountered combining ElasticBeanstalk and Cloudformation when starting up a docker worker is:
Return code: 1 Output: missing required parameter params[:table_name] - (ArgumentError). Hook /opt/elasticbeanstalk/addons/sqsd/hooks/firstboot/02-start-sqsd.sh failed.
Usually when dealing with ElasticBeanstalk this means either:
- The InstanceProfile does not have sufficient policies (access to DynamoDB)
- A file formatted for Windows (CR/LF) instead of Linux
- You’re using .ebextensions to alter the Cloudformation template and messing something up
The exception is raised from this file : /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/aws-sqsd-2.0/vendor/AWSMACLE/lib/leader_election/config.rb
def initialize(options={})
# required parameters
raise AWS::EB::LeaderElection::FatalException %[:tale_name is required] unless options.has_key?(:table_name)
The SQS daemon in this context is automatically integrated with DynamoDB. A table a generated automatically but the AWS SDK does not manage to retrieve the data and pass it to launch the daemon.
Looking at the underlying Cloudformation stack generated by ElasticBeanstalk for the definition of the DynamoDB table we can see that there is none. There should be a logical ID AWSEBWorkerCronLeaderRegistry, and resource type AWS::DynamoDB::Table. This explains why the AWS SDK failed to retrieve the DynamoDB table.
After troubleshooting together with AWS we find two solutions:
There has to be a cron.yaml at the root of the bundle zip file. Include the Dockerrun.aws.json and cron.yaml, otherwise Cloudformation won’t create the Dynamodb table.
version: 1
cron:
- name: "backup-job" # required - unique across all entries in this file
url: "/backup" # required - does not need to be unique
schedule: "0 */12 * * *" # required - does not need to be unique
- name: "audit"
url: "/audit"
schedule: "0 23 * * *"
If you don’t want to use this feature, you may disable it by using the older worker version.
Type: AWS::ElasticBeanstalk::Environment,
...
Tier: {
Name: Worker,
Type: SQS/HTTP,
Version: 1.0
}
Note that you will only find this error if you create the environment through Cloudformation + ElasticBeanstalk docker workers. When done manually by hand, ElasticBeanstalk handles this edge case correctly by automatically selecting the right worker version.