아마존 AWS Elastic Beanstalk에 Python Flask 배포환경 구축을 위한 설정
아마존 Elastic Beanstalk은 애플리케이션을 올리기만 하면 Elastic Beanstalk이 용량 프로비저닝, 로드 밸런싱, 자동 조정,
애플리케이션 상태 모니터링에 대한 배포 정보를 자동으로 처리한다.
개발자는 개발에만 신경 쓰면 인프라는 아마존에서 다 해주겠다는 말이다.
이 얼마나 반가운 소리인가?
그러나 막상 Elastic Beanstalk를 쓰려면 손수 설정해야 하는 부분이 많다.
이 글에서는 static 파일을 아마존 CDN인 CloudFront를 통해 제공한다는 가정하에 크게 세 부분으로 나누어 설명하겠다.
첫째는 아마존 콘솔 단에서 IAM,S3,CloudFront설정이고,
둘째는 Elastic Beanstalk .ebextensions 설정.
마지막은 Python boto를 이용한 배포 스크립트다.
게으른 개발자로서 좀 편해 보고자 아마존 AWS Elastic Beanstalk을 쓰면서 환경 설정 때문에 애를 많이 먹었다.
AWS Elastic Beanstalk을 고려 중인 또 다른 개발자가 이 글을 읽고 같은 삽질을 않으면 좋겠다.
AWS 콘솔 설정
IAM 설정
배포 권한을 가진 Group를 만든다.
예제에서 그룹명은 Dorajistyle-deploy로 하겠다.
User인 dorajistyle은 Dorajistyle-deploy 그룹에 소속되어, 배포시에 dorajistyle유저 정보로 배포하게 된다.
Dorajistyle-deploy그룹은 아래의 policy를 가진다.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"elasticbeanstalk:*",
"ec2:*",
"elasticloadbalancing:*",
"autoscaling:*",
"cloudwatch:*",
"s3:*",
"sns:*",
"cloudformation:*",
"rds:*",
"iam:AddRoleToInstanceProfile",
"iam:CreateInstanceProfile",
"iam:CreateRole",
"iam:PassRole",
"iam:ListInstanceProfiles"
],
"Resource": "*"
},
{
"Sid": "QueueAccess",
"Action": [
"sqs:ChangeMessageVisibility",
"sqs:DeleteMessage",
"sqs:ReceiveMessage"
],
"Effect": "Allow",
"Resource": "*"
},
{
"Sid": "MetricsAccess",
"Action": [
"cloudwatch:PutMetricData"
],
"Effect": "Allow",
"Resource": "*"
},
{
"Sid": "Stmt110100200000",
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource": [
"arn:aws:s3:::dorajistyle/*",
"arn:aws:s3:::dorajistyle",
"arn:aws:s3:::dorajistyle-static/*",
"arn:aws:s3:::dorajistyle-static",
"arn:aws:s3:::dorajistyle-deploy/*",
"arn:aws:s3:::dorajistyle-deploy",
"arn:aws:s3:::elasticbeanstalk-ap-northeast-1-000000000000",
"arn:aws:s3:::elasticbeanstalk-ap-northeast-1-000000000000/*"
]
},
{
"Sid": "Stmt130000013000",
"Effect": "Allow",
"Action": [
"rds:*"
],
"Resource": [
"arn:aws:rds:ap-northeast-1:000000000000:db:dorajistyle"
]
},
{
"Sid": "Stmt1399636332000",
"Effect": "Allow",
"Action": [
"elasticbeanstalk:*"
],
"Resource": ["*","arn:aws:elasticbeanstalk:ap-northeast-1:000000000000:application/dorajistyle",
"arn:aws:elasticbeanstalk:ap-northeast-1:000000000000:environment/dorajistyle/dorajistyle"
,"arn:aws:elasticbeanstalk:ap-northeast-1:000000000000:applicationversion/dorajistyle/*"]
},
{
"Effect": "Allow",
"Action": [
"cloudformation:DescribeStacks",
"cloudformation:DescribeStackEvents",
"cloudformation:DescribeStackResources",
"cloudformation:GetTemplate",
"cloudformation:List*"
],
"Resource": "*"
}
S3 설정
S3버켓은 총 3개가 필요하다.
dorajistyle-deploy
배포용 zip파일을 업로드할 버켓이다. 사용자 dorajistyle만 접근 가능하며, 버켓 설정은 기본 그대로 사용하면 된다.
CORS Configuration
<CORSConfiguration>
<CORSRule>
<AllowedOrigin>*</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
<MaxAgeSeconds>3000</MaxAgeSeconds>
<AllowedHeader>Authorization</AllowedHeader>
</CORSRule>
</CORSConfiguration>
dorajistyle-static
static 파일을 저장할 버켓이다. 누구나 읽을 수 있는 버켓이다.
Bucket policy
{
"Version": "2008-10-17",
"Id": "Policy1394587645145",
"Statement": [
{
"Sid": "Stmt1394587643817",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::dorajistyle-static/*"
}
]
}
CORS Configuration
<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
<AllowedOrigin>http://*.dorajistyle.pe.kr</AllowedOrigin>
<AllowedMethod>PUT</AllowedMethod>
<AllowedMethod>POST</AllowedMethod>
<AllowedMethod>DELETE</AllowedMethod>
<AllowedHeader>*</AllowedHeader>
</CORSRule>
<CORSRule>
<AllowedOrigin>http://*.www.dorajistyle.pe.kr</AllowedOrigin>
<AllowedMethod>PUT</AllowedMethod>
<AllowedMethod>POST</AllowedMethod>
<AllowedMethod>DELETE</AllowedMethod>
<AllowedHeader>*</AllowedHeader>
</CORSRule>
<CORSRule>
<AllowedOrigin>http://dorajistyle.elasticbeanstalk.com</AllowedOrigin>
<AllowedMethod>PUT</AllowedMethod>
<AllowedMethod>POST</AllowedMethod>
<AllowedMethod>DELETE</AllowedMethod>
<AllowedHeader>*</AllowedHeader>
</CORSRule>
<CORSRule>
<AllowedOrigin>*</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
<AllowedHeader>Authorization</AllowedHeader>
<AllowedHeader>x-requested-with</AllowedHeader>
<AllowedHeader>origin</AllowedHeader>
</CORSRule>
</CORSConfiguration>
dorajistyle
이미지등의 유저 컨텐츠를 저장할 버켓이다.
예제에서는 모든 유저가 읽을 수 있도록 설정되었는데, 이는 사용 용도에 따라 변경이 가능하다.
Bucket Policy
{
"Version": "2008-10-17",
"Id": "Policy1394587559249",
"Statement": [
{
"Sid": "Stmt1394587510887",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::dorajistyle/*"
}
]
}
CORS Configuration
<CORSConfiguration>
<CORSRule>
<AllowedOrigin>*</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
<MaxAgeSeconds>3000</MaxAgeSeconds>
<AllowedHeader>Authorization</AllowedHeader>
</CORSRule>
</CORSConfiguration>
CloudFront 설정
dorajsityle-static S3에 CloudFront를 연결한다.
Origin Domain Name에 S3 버켓 주소를 적으면 된다.
만약 CloudFront에 연결이 잘 되었는데도 리소스를 찾지 못한다면 Invalidations에서 해당 리소스를 무효화한다.
.ebextensions 설정
Elastic Beanstalk에 어플리케이션을 올리면 마법처럼 돌아간다고는 하지만,
각 어플리케이션마다 필요한 라이브러리를 모두 설치해 둘 순 없다.
그래서 .ebextensions 설정을 통해 각 어플리케이션에 맞는 라이브러리 설치와, 서버 설정 변경등이 가능하다.
.ebextensions에 대한 설명은 아마존의 컨테이너 맞춤 설정 안내(http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers.html)를 참조하면 된다.
이 글에서는 yum 패키지 업데이트와 라이브러리 설치, 배포 hook 설정 변경과 아파치 서버 변경을 다룬다.
01_yum_update.config
yum 패키지를 업데이트 한다.
commands:
yum_updates:
command: "yum --security update -y"
02_package_update.config
필요한 yum 패키지를 설치한다.
packages키를 이용해도 되지만, 충돌이 일어날 경우 command키를 이용한 설치도 한 방법이다.
commands:
yum_package_updates:
command: "yum install python-devel python-pip libtiff-devel libjpeg-turbo-devel libzip-devel freetype-devel lcms2-devel tcl-devel php -y --skip-broken"
03_pre_requirements.config
Elastic Beanstalk에 파이썬 어플리케이션을 업로드 하면,
requirements.txt파일을 찾아 필요한 파이썬 라이브러리를 자동으로 설치해 준다.
그런데 간혹 한번에 설치가 안되어 나누어 설치해야 하는 라이브러리가 있다.
몇몇 라이브러리를 설치할때 pip에서 발생하는 문제로 미리 설치할 라이브러리를 pre_requirements.txt에 넣어두고 먼저 설치하면 문제없이 설치된다.
다만 pre_requirements.txt 파일을 먼저 설치하려면 배포 hook코드를 변경해야 한다.
files:
"/opt/elasticbeanstalk/hooks/appdeploy/pre/03deploy.py":
mode: "000755"
owner: root
group: users
content: |
#!/usr/bin/env python
import os
from subprocess import call, check_call
import sys
sys.path.append(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))
import config
def install_virtualenv():
# If python 2.7 is installed, make the virtualenv use it. Else use the default system 2.6
if config.get_python_version() == '2.7':
cmd = 'virtualenv -p /usr/bin/python27 {0}'
else:
cmd = 'virtualenv {0}'
return_code = call(cmd.format(config.APP_VIRTUAL_ENV), shell=True)
if return_code != 0:
print "WARN: error running '%s'" % cmd
def install_dependencies():
pre_requirements_file = os.path.join(config.ON_DECK_DIR, 'app', 'pre_requirements.txt')
requirements_file = os.path.join(config.ON_DECK_DIR, 'app', 'requirements.txt')
if os.path.exists(pre_requirements_file):
check_call('%s install --use-mirrors -r %s' % (os.path.join(config.APP_VIRTUAL_ENV, 'bin', 'pip'), pre_requirements_file), shell=True)
if os.path.exists(requirements_file):
# Note, we're sharing the virtualenv across multiple deploys, which implies
# this is an additive operation. This is normally not a problem and is done
# to minimize deployment time (the requirements are not likely to drastically
# change between deploys).
check_call('%s install --use-mirrors -r %s' % (os.path.join(config.APP_VIRTUAL_ENV, 'bin', 'pip'), requirements_file), shell=True)
def main():
try:
install_virtualenv()
install_dependencies()
except Exception, e:
config.emit_error_event(config.USER_ERROR_MESSAGES['badrequirements'])
config.diagnostic("Error installing dependencies: %s" % str(e))
sys.exit(1)
if __name__ == '__main__':
config.configure_stdout_logger()
main()
04_wsgi.config
아파치 서버 설정 파일을 입맛에 맞게 변경한다. wsgi.conf 파일을 .ebextensions 폴더에 넣어두고,
wsgi.config 훅에 아래 코드를 넣으면 서버로 설정을 복사한다.
container_commands:
replace_wsgi_config:
command: "cp .ebextensions/wsgi.conf /opt/python/ondeck/wsgi.conf"
wsgi.conf
캐쉬와 gzip압축등 설정을 담았다.
# LoadModule wsgi_module modules/mod_wsgi.so
WSGIPythonHome /opt/python/run/baselinenv
WSGISocketPrefix run/wsgi
WSGIRestrictEmbedded On
<VirtualHost *:80>
###############
# TYPES FIX #
###############
AddType text/css .css
AddType text/javascript .js
############################
# IE 11 Prevent Cache #
############################
BrowserMatch "MSIE 11.0;" IE11FOUND
BrowserMatch "Trident/7.0;" IE11FOUND
# FileETag None
Header unset ETag env=IE11FOUND
Header set Cache-Control "max-age=0, no-cache, no-store, must-revalidate" env=IE11FOUND
Header set Pragma "no-cache" env=IE11FOUND
Header set Expires "Thu, 24 Feb 1983 02:50:00 GMT" env=IE11FOUND
####################################
# Serve Pre-Compressed statics #
####################################
RewriteEngine On
RewriteCond %{HTTP:Accept-encoding} gzip
RewriteCond %{REQUEST_FILENAME}\.gz -s
RewriteRule ^(.*)\.(html|css|js|json|woff) $1\.$2\.gz [QSA]
# Prevent double gzip and give the correct mime-type
RewriteRule \.css\.gz$ - [T=text/css,E=no-gzip:1,E=FORCE_GZIP]
RewriteRule \.js\.gz$ - [T=text/javascript,E=no-gzip:1,E=FORCE_GZIP]
RewriteRule \.html\.gz$ - [T=text/html,E=no-gzip:1,E=FORCE_GZIP]
RewriteRule \.json\.gz$ - [T=application/json,E=no-gzip:1,E=FORCE_GZIP]
RewriteRule \.woff\.gz$ - [T=application/x-font-woff,E=no-gzip:1,E=FORCE_GZIP]
Header set Content-Encoding gzip env=FORCE_GZIP
#######################
# GZIP COMPRESSION #
######################
SetOutputFilter DEFLATE
AddOutputFilterByType DEFLATE text/html text/css text/plain text/xml text/javascript application/x-javascript application/x-httpd-php
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4\.0[678] no-gzip
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html
SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip
Header append Vary User-Agent env=!dont-vary
################################
# Leverage browser caching #
###############################
<FilesMatch ".(ico|pdf|jpg|jpeg|png|gif|html|htm|xml|txt|xsl)$">
Header set Cache-Control "max-age=31536050"
</FilesMatch>
############################
# CDN Rewrite Setting #
############################
# Header set Access-Control-Allow-Origin: "*"
# UseCanonicalName On
# Don't redirect if the static hostname(s) loops back.
# RewriteCond %{HTTP_HOST} !^static\.
# Include only those static file extensions that we want to off-load.
# RewriteCond %{REQUEST_FILENAME} ^/.*\.(html|xml|txt|zip|gz|tgz|swf|mov|wmv|wav|mp3|pdf|svg|otf|eot|ttf|woff|jpg|jpeg|png|gif|ico|css|js|json)$
# RewriteRule /static/(.*)? http://static.dorajistyle.pe.kr/$1 [redirect=permanent,last]
#########################
# WSGI configuration #
#########################
WSGIScriptAlias / /opt/python/current/app/application.py
<Directory /opt/python/current/app/>
# Order allow,deny
# Allow from all
Require all granted
</Directory>
WSGIDaemonProcess wsgi processes=1 threads=15 display-name=%{GROUP} \
python-path=/opt/python/current/app:/opt/python/run/venv/lib/python2.7/site-packages user=wsgi group=wsgi \
home=/opt/python/current/app
WSGIProcessGroup wsgi
# WSGIScriptReloading On
</VirtualHost>
배포 스크립트.
deploy.sh
static폴더를 최적화하고, DB스키마가 변경되었을 경우 업데이트 하며, 필요한 파일만 압축하여 aws에 올린다.
optimize_static.sh와 upload_to_aws.py이 중요하다.
#!/bin/bash
STARTTIME=$(date +%s)
./optimize_static.sh
rm *.zip
prefix=$(sed -n '/^PREFIX/ s/.*\= *//p' ./application/config/guid.py | tr -d \')
guid=$(sed -n '/^GUID/ s/.*\= *//p' ./application/config/guid.py | tr -d \')
name="$prefix-$guid"
zip -r $name * ./.ebextensions/* -x ./.git\* ./.idea\* ./docs\* ./node_modules\* ./alembic\* ./tests\* ./images\* *.zip *.DS_Store ./application/frontend/static/\*
zip_file=$name'.zip'
echo "$zip_file"
echo -e "Do you want to upgrade alembic schema? (Yes/No) : \c"
read ANSWER
if [ "$ANSWER" == "Yes" ]
then
alembic revision --autogenerate -m "Alembic initilized boilerplate tables."
fi
echo -e "Do you want to update schema? (Yes/No) : \c"
read ANSWER
if [ "$ANSWER" == "Yes" ]
then
alembic upgrade head
fi
echo -e "Did you reviewed source and confirmed running status? (Yes/No) : \c"
read ANSWER
if [ "$ANSWER" == "Yes" ]
then
python2 upload_to_aws.py $name
else
echo "Checking status and trying to deploy again."
fi
ENDTIME=$(date +%s)
echo "$name"
echo "It takes $(($ENDTIME - $STARTTIME)) seconds to complete this task..."
optimize_static.sh
guid를 생성하고 총 4개까지 히스토리를 남긴다. 혹시 배포가 잘못되어 롤백을 하게될 경우 이전 4버전까지 롤백이 가능하도록 한다.
테스트 서버에 먼저 배포하여 테스트 하고 문제가 없으면 실 서버에 배포를 하기 때문에 실 서버에서 4버전이나 롤백할 가능성은 상당히 희박하다.
static 파일은 require optimizer를 사용해 하나의 js와 하나의 css파일로 합치고, sed를 이용해 디버깅을 위해 사용하던 로그 코드와 공백을 날려 용량을 줄인다.
그리고 각 파일을 gzip으로 압축하여 용량을 다시 한번 줄인다.
#!/bin/bash
guid=$(uuidgen | tr -d '\n-' | tr '[:upper:]' '[:lower:]')
guid=${guid:0:8}
today=$(date '+%Y%m%d')
guid=$today'-'$guid
echo "$guid"
extremely_very_old_guid=$(sed -n '/^VERY_OLD_GUID/ s/.*\= *//p' ./application/config/guid.py)
sed -i "s/^EXTREMELY_VERY_OLD_GUID = .*/EXTREMELY_VERY_OLD_GUID = $extremely_very_old_guid/" ./application/config/guid.py
very_old_guid=$(sed -n '/^OLD_GUID/ s/.*\= *//p' ./application/config/guid.py)
sed -i "s/^VERY_OLD_GUID = .*/VERY_OLD_GUID = $very_old_guid/" ./application/config/guid.py
old_guid=$(sed -n '/^GUID/ s/.*\= *//p' ./application/config/guid.py)
sed -i "s/^OLD_GUID = .*/OLD_GUID = $old_guid/" ./application/config/guid.py
sed -i "s/^GUID = .*/GUID = '$guid'/" ./application/config/guid.py
cd './application/frontend/compiler/'
grunt static
grunt --gruntfile Gruntfile_uncss.js
cd '../../..'
cd './optimizer'
node ./r.js -o build.js
cd "../"
sed -i -r "s/(\ ?|\ +),(\ ?|\ +)[a-zA-Z]\.log(Error|Json|Object|Trace|Debug|Info|Warn)(\ ?|\ +)\([^)]*\)(\ ?|\ +),(\ ?|\ +)/,/g" ./application/frontend/static-build/js/app.js
sed -i -r "s/(\ ?|\ +),(\ ?|\ +)[a-zA-Z]\.log(Error|Json|Object|Trace|Debug|Info|Warn)(\ ?|\ +)\([^)]*\)(\ ?|\ +),?(\ ?|\ +);/;/g" ./application/frontend/static-build/js/app.js
sed -i -r "s/(\ ?|\ +),?(\ ?|\ +)[a-zA-Z]\.log(Error|Json|Object|race|Debug|Info|Warn)(\ ?|\ +)\([^)]*\)(\ ?|\ +),?(\ ?|\ +);//g" ./application/frontend/static-build/js/app.js
sed -i -r "s/(\ ?|\ +),?(\ ?|\ +)[a-zA-Z]\.log(Error|Json|Object|race|Debug|Info|Warn)(\ ?|\ +)\([^)]*\)(\ ?|\ +),?(\ ?|\ +);?/\n/g" ./application/frontend/static-build/js/app.js
cd './application/frontend/static-build/locales'
find . -name '*.json' -exec sed -i '/^\s∗\/\//d' {} \;
find . -name '*.json' -exec sed -i 's/^[ \t]*//g; s/[ \t]*$//g;' {} \;
find . -name '*.json' -exec sed -i ':a;N;$!ba;s/\n/ /g' {} \;
find . -name '*.json' -exec sed -i 's/\"\s*:\s*\"/\":\"/g' {} \;
find . -name '*.json' -exec sed -i 's/\"\s*,\s*\"/\",\"/g' {} \;
find . -name '*.json' -exec sed -i 's/\s*{\s*/{/g' {} \;
find . -name '*.json' -exec sed -i 's/\s*}\s*/}/g' {} \;
cd '../..'
gzip -r --best ./static-build
rename .gz '' `find static-build -name '*.gz'`
upload_to_aws.py
보토를 이용해 Elastic Beanstalk을 업데이트 한다. 배포가 끝나면 guid를 검사하여 오래된 버전 소스를 삭제한다.
static파일 업로드에서 눈여겨 볼 점은 key에 Content_Encoding 메타 데이터를 gzip으로 해 주어야 하는 것이다.
이는 위의 optimize static에서 이미 gzip으로 압축했기 때문이다.
# coding=UTF-8
"""
application.util.__init__
~~~~~~~~~~~~~~~~~~~~~~~~~~
by dorajistyle
__init__ module
"""
from Queue import Queue
import logging
import os
import string
import boto
from boto.beanstalk import connect_to_region
import sys
# import time
from application.config.aws import AWS_STATIC_S3_BUCKET_NAME, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, \
AWS_ELASTIC_BEANSTALK_REGION, AWS_ELASTIC_BEANSTALK_APP_NAME, AWS_ELASTIC_BEANSTALK_ENVIRONMENT_ID, \
AWS_ELASTIC_BEANSTALK_ENVIRONMENT_NAME, AWS_ELASTIC_BEANSTALK_S3_BUCKET_NAME
from application.config.guid import OLD_GUID, VERY_OLD_GUID, PREFIX_GUID, EXTREMELY_VERY_OLD_GUID
from application.properties import STATIC_GUID
from threading import Thread
logging.basicConfig()
logger = logging.getLogger()
def log_exception(text):
logger.error(msg=text)
q = Queue()
# source directory
sourceDir = 'application/frontend/static-build/'
# destination directory name (on s3)
destDir = STATIC_GUID+'/'
conn = boto.connect_s3(AWS_ACCESS_KEY_ID,
AWS_SECRET_ACCESS_KEY)
bucket = conn.get_bucket(AWS_STATIC_S3_BUCKET_NAME)
eb_bucket = conn.get_bucket(AWS_ELASTIC_BEANSTALK_S3_BUCKET_NAME)
keys = list()
old_keys = list()
eb_keys = list()
for key in bucket.list():
keys.append(key.name)
for key in eb_bucket.list():
eb_keys.append(key.name)
eb = connect_to_region(AWS_ELASTIC_BEANSTALK_REGION,
aws_access_key_id=AWS_ACCESS_KEY_ID,
aws_secret_access_key=AWS_SECRET_ACCESS_KEY)
uploadDirNames = []
uploadFileNames = []
for path in os.listdir( sourceDir ):
if not os.path.isfile(os.path.join( sourceDir, path)):
uploadDirNames.append(path+'/')
for (sourceDir, dirnames, filenames) in os.walk(sourceDir):
for subDir in dirnames:
# print('dirname:'+ subDir)
for (subDir, sub_dirnames, subfilenames) in os.walk(sourceDir+subDir):
for subfilename in subfilenames:
sub_path = string.replace(subDir, sourceDir, '')
uploadFileNames.append(os.path.join(sub_path, subfilename))
uploadFileNames.extend(filenames)
break
def percent_cb(complete, total):
sys.stdout.write('.')
sys.stdout.flush()
def upload_deploy(source_path, dest_path):
"""
Upload static files to S3 bucket.
:return:
"""
try:
dest_path = dest_path.encode('utf-8')
key = eb_bucket.new_key(dest_path)
key.set_contents_from_filename(source_path,
cb=percent_cb, num_cb=10)
except BaseException as be:
log_exception(be)
return False
return True
def upload_static(source_path, dest_path):
"""
Upload static files to S3 bucket.
:return:
"""
try:
dest_path = dest_path.encode('utf-8')
key = bucket.new_key(dest_path)
# if key.name.endswith(('.gz', '.gzip')):
key.set_metadata('Content-Encoding', 'gzip')
key.set_contents_from_filename(source_path,
cb=percent_cb, num_cb=10)
except BaseException as be:
log_exception(be)
return False
return True
def worker():
while True:
item = q.get()
if item['source_path'] == item['dest_path']:
upload_deploy(item['source_path'], item['dest_path'])
else:
upload_static(item['source_path'], item['dest_path'])
q.task_done()
print 'Uploading %s to Amazon S3 bucket %s' % \
(item['source_path'], item['dest_path'])
# threads = []
if len(sys.argv) == 2:
eb_app = eb.describe_applications(application_names=AWS_ELASTIC_BEANSTALK_APP_NAME)
versions = eb_app['DescribeApplicationsResponse']['DescribeApplicationsResult']['Applications'][0]['Versions']
# if len(versions) > 2:
# versions = versions[:2]
latest_version = PREFIX_GUID.replace('\'', '')+'-'+OLD_GUID.replace('\'', '')
very_old_version = PREFIX_GUID.replace('\'', '')+'-'+VERY_OLD_GUID.replace('\'', '')
extremely_very_old_version = PREFIX_GUID.replace('\'', '')+'-'+EXTREMELY_VERY_OLD_GUID.replace('\'', '')
try:
if latest_version in versions:
versions.remove(latest_version)
if very_old_version in versions:
versions.remove(very_old_version)
if extremely_very_old_version in versions:
versions.remove(extremely_very_old_version)
for key in bucket.list(prefix=OLD_GUID.replace('\'', '')):
keys.remove(key.name)
for key in bucket.list(prefix=VERY_OLD_GUID.replace('\'', '')):
keys.remove(key.name)
for key in bucket.list(prefix=EXTREMELY_VERY_OLD_GUID.replace('\'', '')):
keys.remove(key.name)
for eb_key in eb_bucket.list(prefix=latest_version):
eb_keys.remove(eb_key.name)
for eb_key in eb_bucket.list(prefix=very_old_version):
eb_keys.remove(eb_key.name)
for eb_key in eb_bucket.list(prefix=extremely_very_old_version):
eb_keys.remove(eb_key.name)
file_name = sys.argv[1]
zip_file = file_name + '.zip'
for i in range(8):
t = Thread(target=worker)
t.daemon = True
t.start()
item = {}
item['source_path'] = zip_file
item['dest_path'] = zip_file
q.put(item)
for filename in uploadFileNames:
source_path = os.path.join(sourceDir + filename)
dest_path = os.path.join(destDir, filename)
item = {}
item['source_path'] = source_path
item['dest_path'] = dest_path
q.put(item)
q.join()
eb.create_application_version(AWS_ELASTIC_BEANSTALK_APP_NAME, version_label=file_name,
description=None, s3_bucket=AWS_ELASTIC_BEANSTALK_S3_BUCKET_NAME, s3_key=zip_file)
eb.update_environment(environment_id=AWS_ELASTIC_BEANSTALK_ENVIRONMENT_ID,
environment_name=AWS_ELASTIC_BEANSTALK_ENVIRONMENT_NAME,
version_label=file_name)
bucket.delete_keys(keys)
eb_bucket.delete_keys(eb_keys)
for version in versions:
eb.delete_application_version(application_name=AWS_ELASTIC_BEANSTALK_APP_NAME, version_label=version, delete_source_bundle=False)
except BaseException as be:
print(str(be))
if latest_version is not None:
eb.update_environment(environment_id=AWS_ELASTIC_BEANSTALK_ENVIRONMENT_ID,
environment_name=AWS_ELASTIC_BEANSTALK_ENVIRONMENT_NAME,
version_label=latest_version)
# print('eb application' + str(eb.retrieve_environment_info(environment_id=AWS_ELASTIC_BEANSTALK_ENVIRONMENT_ID,
# environment_name=AWS_ELASTIC_BEANSTALK_ENVIRONMENT_NAME)))
print('AWS Elastic Beanstalk updated.')
print('Bye Bye!')
