{"id":321,"date":"2017-10-17T05:48:53","date_gmt":"2017-10-16T22:48:53","guid":{"rendered":"http:\/\/www.seven-stones.biz\/blog\/?p=321"},"modified":"2017-11-18T06:38:32","modified_gmt":"2017-11-17T23:38:32","slug":"django-and-celery-two-sites-single-host","status":"publish","type":"post","link":"https:\/\/www.seven-stones.biz\/blog\/django-and-celery-two-sites-single-host\/","title":{"rendered":"Django and Celery &#8211; Two Sites, Single Host"},"content":{"rendered":"<p>Documenting this here because <a href=\"http:\/\/docs.celeryproject.org\/en\/latest\/userguide\/index.html\">Celery&#8217;s documentation<\/a> isn&#8217;t the best in general, but moreover because I hadn&#8217;t seen a write-up for this scenario &#8211; which I would imagine is not an uncommon situation.<\/p>\n<p>So here&#8217;s the summarised scenario:<\/p>\n<ul>\n<li><a href=\"https:\/\/www.djangoproject.com\/\">Django<\/a><\/li>\n<li>Celery<\/li>\n<li><a href=\"https:\/\/pypi.python.org\/pypi\/django-celery\">Djcelery\u00a0<\/a><\/li>\n<li>RabbitMQ<\/li>\n<\/ul>\n<p>The challenge was to have 2+ Django sites on one VM without confusing the Celery backend. This requires the creation of a &#8220;Queue&#8221;, &#8220;Exchange&#8221;, and &#8220;Routing Key&#8221;. <a href=\"http:\/\/docs.celeryproject.org\/en\/latest\/userguide\/routing.html\">The Celery documentation<\/a> gives major clues but doesn&#8217;t cover Django to any large degree. <a href=\"http:\/\/docs.celeryproject.org\/en\/latest\/django\/first-steps-with-django.html\">It does cover some first steps with Django<\/a>, but nothing about deploying custom Queues in terms of Django python files and what goes where, or at least i couldn&#8217;t find it.<\/p>\n<p>I&#8217;m sure there are many ways of achieving the same result but this is what worked for me&#8230;<\/p>\n<p>The application won&#8217;t be able to find celery if you don&#8217;t initiate it. The project here is &#8216;netdelta&#8217;. Under the &lt;app&gt; dir create a file, say celery_app.py:<\/p>\n<pre>from __future__ import absolute_import, unicode_literals\r\nimport os\r\nfrom celery import Celery\r\nfrom kombu import Exchange, Queue\r\n\r\n\r\n\r\n# set the default Django settings module for the 'celery' program.\r\nos.environ.setdefault('DJANGO_SETTINGS_MODULE', 'netdelta.settings')\r\n\r\napp = Celery('nd')\r\n\r\n# Using a string here means the worker doesn't have to serialize\r\n# the configuration object to child processes.\r\n# - namespace='CELERY' means all celery-related configuration keys\r\n#   should have a `CELERY_` prefix.\r\napp.config_from_object('django.conf:settings')\r\n\r\n# Load task modules from all registered Django app configs.\r\n#app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)\r\n\r\napp.conf.task_queues = (\r\n    Queue('cooler',  Exchange('cooler'),   routing_key='cooler'),\r\n)\r\napp.conf.task_default_queue = 'cooler'\r\napp.conf.task_default_exchange_type = 'direct'\r\napp.conf.task_default_routing_key = 'cooler'\r\n\r\n@app.task(bind=True)\r\ndef debug_task(self):\r\n    print('Request: {0!r}'.format(self.request))<\/pre>\n<p>Which is called when the project fires up with use of __init.py__ under &lt;app&gt; (&#8216;nd&#8217; in this case):<\/p>\n<pre>from __future__ import absolute_import, unicode_literals\r\n\r\n# This will make sure the app is always imported when\r\n# Django starts so that shared_task will use this app.\r\nfrom .celery_app import app as celery_thang\r\n\r\n__all__ = ('celery_thang',)<\/pre>\n<p>&lt;django root&gt;\/&lt;proj&gt;\/&lt;proj&gt;\/settings.py is where the magic happens and it turns out to be a few lines. &#8216;scheduled_scan&#8217; is the &lt;proj&gt;.tasks.&lt;celery_job_name&gt;:<\/p>\n<pre>CELERY_QUEUES = {\"coller\": {\"exchange\": \"cooler\",\r\n                              \"routing_key\": \"cooler\"}}\r\n\r\nCELERY_ROUTES = {\r\n    'nd.tasks.scheduled_scan': {'queue': \"cooler\",\r\n                                'exchange': \"cooler\",\r\n                                'routing_key': \"cooler\"},\r\n}<\/pre>\n<p>Now do the same for the other site(s).<\/p>\n<p>Launch a worker thusly &#8230;<\/p>\n<div style=\"font-size: 9pt;\"><code>python manage.py celery worker -Q cooler -n cooler --loglevel=info -B<\/code><\/div>\n<div><\/div>\n<ul>\n<li>&#8216;cooler&#8217; is the site name<\/li>\n<li>&#8216;nd&#8217; is the app name<\/li>\n<\/ul>\n<p>The final step &#8211; your app.task has to call the custom queue you defined. In my case i could afford to set the default QUEUE to my wanted QUEUE because i had need for only one queue. But in multiple QUEUE scenarios, you will need to define the queue. I was setting jobs to run under a schedule using the djcelery admin panel that was created by default. The result are database entries &#8211; using an &#8216;INSERT&#8217; statement to show the table structure (The queue, exchange, and routing key are highlighted):<\/p>\n<pre>INSERT INTO `djcelery_periodictask` (`id`, `name`, `task`, `args`, \r\n`kwargs`, `queue`, `exchange`, `routing_key`, `expires`, `enabled`, \r\n`last_run_at`, `total_run_count`, `date_changed`, `description`, \r\n`crontab_id`, `interval_id`) VALUES\r\n(2, 'Linode-67257', 'nd.tasks.scheduled_scan', '[\"Linode\"]', '{}', \r\n<u><b>'coller', 'coller', 'coller',<\/b><\/u> NULL, 1, \r\n'2017-10-15 23:40:00', 18, '2017-10-15 23:41:29', '', 2, NULL);<\/pre>\n<p>&nbsp;<\/p>\n<p>My <a href=\"https:\/\/virtualenv.pypa.io\/en\/stable\/\">virtualenv<\/a>\u00a0is as below (note i&#8217;m using the <a href=\"https:\/\/github.com\/pyoner\/python-libnmap\/\">fork of libnmap<\/a> that doesn&#8217;t use multiprocessor:<\/p>\n<p>&gt;pip list<\/p>\n<ul>\n<li class=\"p1\"><span class=\"s1\">amqp (2.2.2)<\/span><\/li>\n<li class=\"p1\"><span class=\"s1\">anyjson (0.3.3)<\/span><\/li>\n<li class=\"p1\"><span class=\"s1\">appdirs (1.4.3)<\/span><\/li>\n<li class=\"p1\"><span class=\"s1\">billiard (3.5.0.3)<\/span><\/li>\n<li class=\"p1\"><span class=\"s1\">celery (4.1.0)<\/span><\/li>\n<li class=\"p1\"><span class=\"s1\">Django (1.11.6)<\/span><\/li>\n<li class=\"p1\"><span class=\"s1\">django-celery-beat (1.0.1)<\/span><\/li>\n<li class=\"p1\"><span class=\"s1\">django-celery-results (1.0.1)<\/span><\/li>\n<li class=\"p1\"><span class=\"s1\">django-dajaxice (0.7)<\/span><\/li>\n<li class=\"p1\"><span class=\"s1\">html2text (2017.10.4)<\/span><\/li>\n<li class=\"p1\"><span class=\"s1\">iptools (0.6.1)<\/span><\/li>\n<li class=\"p1\"><span class=\"s1\">kombu (4.1.0)<\/span><\/li>\n<li class=\"p1\"><span class=\"s1\">MySQL-python (1.2.5)<\/span><\/li>\n<li class=\"p1\"><span class=\"s1\">netaddr (0.7.19)<\/span><\/li>\n<li class=\"p1\"><span class=\"s1\">packaging (16.8)<\/span><\/li>\n<li class=\"p1\"><span class=\"s1\">pip (9.0.1)<\/span><\/li>\n<li class=\"p1\"><span class=\"s1\">pyparsing (2.2.0)<\/span><\/li>\n<li class=\"p1\"><span class=\"s1\">python-libnmap (0.7.0)<\/span><\/li>\n<li class=\"p1\"><span class=\"s1\">pytz (2017.2)<\/span><\/li>\n<li class=\"p1\"><span class=\"s1\">setuptools (36.6.0)<\/span><\/li>\n<li class=\"p1\"><span class=\"s1\">six (1.11.0)<\/span><\/li>\n<li class=\"p1\"><span class=\"s1\">vine (1.1.4)<\/span><\/li>\n<li class=\"p1\"><span class=\"s1\">wheel (0.30.0)<\/span><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Documenting this here because Celery&#8217;s documentation isn&#8217;t the best in general, but moreover because I hadn&#8217;t seen a write-up for this scenario &#8211; which I would imagine is not an uncommon situation. So here&#8217;s the summarised scenario: Django Celery Djcelery\u00a0 &hellip; <a href=\"https:\/\/www.seven-stones.biz\/blog\/django-and-celery-two-sites-single-host\/\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[140,139,138,141],"tags":[144,143,142],"class_list":["post-321","post","type-post","status-publish","format-standard","hentry","category-celery","category-django","category-python","category-software-development","tag-celery","tag-django","tag-python"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Django and Celery - Two Sites, Single Host - Security Macromorphosis<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.seven-stones.biz\/blog\/django-and-celery-two-sites-single-host\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Django and Celery - Two Sites, Single Host - Security Macromorphosis\" \/>\n<meta property=\"og:description\" content=\"Documenting this here because Celery&#8217;s documentation isn&#8217;t the best in general, but moreover because I hadn&#8217;t seen a write-up for this scenario &#8211; which I would imagine is not an uncommon situation. So here&#8217;s the summarised scenario: Django Celery Djcelery\u00a0 &hellip; Continue reading &rarr;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.seven-stones.biz\/blog\/django-and-celery-two-sites-single-host\/\" \/>\n<meta property=\"og:site_name\" content=\"Security Macromorphosis\" \/>\n<meta property=\"article:published_time\" content=\"2017-10-16T22:48:53+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2017-11-17T23:38:32+00:00\" \/>\n<meta name=\"author\" content=\"itibble@gmail.com\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@seven_stones\" \/>\n<meta name=\"twitter:site\" content=\"@seven_stones\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"itibble@gmail.com\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.seven-stones.biz\\\/blog\\\/django-and-celery-two-sites-single-host\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.seven-stones.biz\\\/blog\\\/django-and-celery-two-sites-single-host\\\/\"},\"author\":{\"name\":\"itibble@gmail.com\",\"@id\":\"https:\\\/\\\/www.seven-stones.biz\\\/blog\\\/#\\\/schema\\\/person\\\/dd7adbe0152f2279b133661b823e0c28\"},\"headline\":\"Django and Celery &#8211; Two Sites, Single Host\",\"datePublished\":\"2017-10-16T22:48:53+00:00\",\"dateModified\":\"2017-11-17T23:38:32+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.seven-stones.biz\\\/blog\\\/django-and-celery-two-sites-single-host\\\/\"},\"wordCount\":399,\"commentCount\":0,\"keywords\":[\"celery\",\"django\",\"python\"],\"articleSection\":[\"Celery\",\"Django\",\"Python\",\"Software development\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.seven-stones.biz\\\/blog\\\/django-and-celery-two-sites-single-host\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.seven-stones.biz\\\/blog\\\/django-and-celery-two-sites-single-host\\\/\",\"url\":\"https:\\\/\\\/www.seven-stones.biz\\\/blog\\\/django-and-celery-two-sites-single-host\\\/\",\"name\":\"Django and Celery - Two Sites, Single Host - Security Macromorphosis\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.seven-stones.biz\\\/blog\\\/#website\"},\"datePublished\":\"2017-10-16T22:48:53+00:00\",\"dateModified\":\"2017-11-17T23:38:32+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/www.seven-stones.biz\\\/blog\\\/#\\\/schema\\\/person\\\/dd7adbe0152f2279b133661b823e0c28\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.seven-stones.biz\\\/blog\\\/django-and-celery-two-sites-single-host\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.seven-stones.biz\\\/blog\\\/django-and-celery-two-sites-single-host\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.seven-stones.biz\\\/blog\\\/django-and-celery-two-sites-single-host\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.seven-stones.biz\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Django and Celery &#8211; Two Sites, Single Host\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.seven-stones.biz\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/www.seven-stones.biz\\\/blog\\\/\",\"name\":\"Security Macromorphosis\",\"description\":\"Ian Tibble&#039;s Security Blog\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.seven-stones.biz\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.seven-stones.biz\\\/blog\\\/#\\\/schema\\\/person\\\/dd7adbe0152f2279b133661b823e0c28\",\"name\":\"itibble@gmail.com\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/4efc9caa4c914912bcf9dd199b33f34a0d42e56752f4f713cd8d0c5416733603?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/4efc9caa4c914912bcf9dd199b33f34a0d42e56752f4f713cd8d0c5416733603?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/4efc9caa4c914912bcf9dd199b33f34a0d42e56752f4f713cd8d0c5416733603?s=96&d=mm&r=g\",\"caption\":\"itibble@gmail.com\"},\"description\":\"Author of Security De-engineering, CTO at Seven Stones (Indonesia)\",\"sameAs\":[\"http:\\\/\\\/www.seven-stones.biz\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Django and Celery - Two Sites, Single Host - Security Macromorphosis","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.seven-stones.biz\/blog\/django-and-celery-two-sites-single-host\/","og_locale":"en_US","og_type":"article","og_title":"Django and Celery - Two Sites, Single Host - Security Macromorphosis","og_description":"Documenting this here because Celery&#8217;s documentation isn&#8217;t the best in general, but moreover because I hadn&#8217;t seen a write-up for this scenario &#8211; which I would imagine is not an uncommon situation. So here&#8217;s the summarised scenario: Django Celery Djcelery\u00a0 &hellip; Continue reading &rarr;","og_url":"https:\/\/www.seven-stones.biz\/blog\/django-and-celery-two-sites-single-host\/","og_site_name":"Security Macromorphosis","article_published_time":"2017-10-16T22:48:53+00:00","article_modified_time":"2017-11-17T23:38:32+00:00","author":"itibble@gmail.com","twitter_card":"summary_large_image","twitter_creator":"@seven_stones","twitter_site":"@seven_stones","twitter_misc":{"Written by":"itibble@gmail.com","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.seven-stones.biz\/blog\/django-and-celery-two-sites-single-host\/#article","isPartOf":{"@id":"https:\/\/www.seven-stones.biz\/blog\/django-and-celery-two-sites-single-host\/"},"author":{"name":"itibble@gmail.com","@id":"https:\/\/www.seven-stones.biz\/blog\/#\/schema\/person\/dd7adbe0152f2279b133661b823e0c28"},"headline":"Django and Celery &#8211; Two Sites, Single Host","datePublished":"2017-10-16T22:48:53+00:00","dateModified":"2017-11-17T23:38:32+00:00","mainEntityOfPage":{"@id":"https:\/\/www.seven-stones.biz\/blog\/django-and-celery-two-sites-single-host\/"},"wordCount":399,"commentCount":0,"keywords":["celery","django","python"],"articleSection":["Celery","Django","Python","Software development"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.seven-stones.biz\/blog\/django-and-celery-two-sites-single-host\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.seven-stones.biz\/blog\/django-and-celery-two-sites-single-host\/","url":"https:\/\/www.seven-stones.biz\/blog\/django-and-celery-two-sites-single-host\/","name":"Django and Celery - Two Sites, Single Host - Security Macromorphosis","isPartOf":{"@id":"https:\/\/www.seven-stones.biz\/blog\/#website"},"datePublished":"2017-10-16T22:48:53+00:00","dateModified":"2017-11-17T23:38:32+00:00","author":{"@id":"https:\/\/www.seven-stones.biz\/blog\/#\/schema\/person\/dd7adbe0152f2279b133661b823e0c28"},"breadcrumb":{"@id":"https:\/\/www.seven-stones.biz\/blog\/django-and-celery-two-sites-single-host\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.seven-stones.biz\/blog\/django-and-celery-two-sites-single-host\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.seven-stones.biz\/blog\/django-and-celery-two-sites-single-host\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.seven-stones.biz\/blog\/"},{"@type":"ListItem","position":2,"name":"Django and Celery &#8211; Two Sites, Single Host"}]},{"@type":"WebSite","@id":"https:\/\/www.seven-stones.biz\/blog\/#website","url":"https:\/\/www.seven-stones.biz\/blog\/","name":"Security Macromorphosis","description":"Ian Tibble&#039;s Security Blog","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.seven-stones.biz\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.seven-stones.biz\/blog\/#\/schema\/person\/dd7adbe0152f2279b133661b823e0c28","name":"itibble@gmail.com","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/4efc9caa4c914912bcf9dd199b33f34a0d42e56752f4f713cd8d0c5416733603?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/4efc9caa4c914912bcf9dd199b33f34a0d42e56752f4f713cd8d0c5416733603?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/4efc9caa4c914912bcf9dd199b33f34a0d42e56752f4f713cd8d0c5416733603?s=96&d=mm&r=g","caption":"itibble@gmail.com"},"description":"Author of Security De-engineering, CTO at Seven Stones (Indonesia)","sameAs":["http:\/\/www.seven-stones.biz"]}]}},"_links":{"self":[{"href":"https:\/\/www.seven-stones.biz\/blog\/wp-json\/wp\/v2\/posts\/321","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.seven-stones.biz\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.seven-stones.biz\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.seven-stones.biz\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.seven-stones.biz\/blog\/wp-json\/wp\/v2\/comments?post=321"}],"version-history":[{"count":5,"href":"https:\/\/www.seven-stones.biz\/blog\/wp-json\/wp\/v2\/posts\/321\/revisions"}],"predecessor-version":[{"id":327,"href":"https:\/\/www.seven-stones.biz\/blog\/wp-json\/wp\/v2\/posts\/321\/revisions\/327"}],"wp:attachment":[{"href":"https:\/\/www.seven-stones.biz\/blog\/wp-json\/wp\/v2\/media?parent=321"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.seven-stones.biz\/blog\/wp-json\/wp\/v2\/categories?post=321"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.seven-stones.biz\/blog\/wp-json\/wp\/v2\/tags?post=321"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}