Django, from nightmare to dream

Transcription

1 Django, from nightmare to dream with Best Practices by Stéphane Wirtel EuroPython Rimini/y ;-) 11 Luglio / 69

2 Hello, I am Stéphane Python Freelancer Open Source = My Passion/Job PythonFOSDEM CPython contributor PSF, EPS members, CSA from PSF former core dev of Odoo (Open Source ERP) blah blah Python blah 2 / 69

3 ep2017.europython.org 3 / 69

4 EuroPython website -> epcon 4 / 69

5 with epcon, we can do... conference management 5 / 69

6 with epcon, we can do... conference management ticket management 6 / 69

7 with epcon, we can do... conference management ticket management statistics (attendees, speakers, (un)assigned or orphan tickets) 7 / 69

8 with epcon, we can do... conference management ticket management statistics (attendees, speakers, (un)assigned or orphan tickets) invoice/refund 8 / 69

9 with epcon, we can do... conference management ticket management statistics (attendees, speakers, (un)assigned or orphan tickets) invoice/refund helpdesk 9 / 69

10 with epcon, we can do... conference management ticket management statistics (attendees, speakers, (un)assigned or orphan tickets) invoice/refund helpdesk notifications by 10 / 69

11 with epcon, we can do... conference management ticket management statistics (attendees, speakers, (un)assigned or orphan tickets) invoice/refund helpdesk notifications by hotel and room booking sim card 11 / 69

12 epcon is a good tool for EuroPython 12 / 69

13 but / 69

14 14 / 69

15 State of Union Python 2.7 (support 2020) 15 / 69

16 State of Union Python 2.7 (support 2020) Django 1.8 (support april 2018) 16 / 69

17 State of Union Python 2.7 (support 2020) Django 1.8 (support april 2018) few dependencies are deprecated or no maintainers (incompatible with django 1.11) 17 / 69

18 State of Union Python 2.7 (support 2020) Django 1.8 (support april 2018) few dependencies are deprecated or no maintainers (incompatible with django 1.11) SQLite 18 / 69

19 State of Union Python 2.7 (support 2020) Django 1.8 (support april 2018) few dependencies are deprecated or no maintainers (incompatible with django 1.11) SQLite no tests (Python 168 files and lines of code) 19 / 69

20 State of Union Python 2.7 (support 2020) Django 1.8 (support april 2018) few dependencies are deprecated or no maintainers (incompatible with django 1.11) SQLite no tests (Python 168 files and lines of code) no documentation, some comments are in Italian 20 / 69

21 State of Union Python 2.7 (support 2020) Django 1.8 (support april 2018) few dependencies are deprecated or no maintainers (incompatible with django 1.11) SQLite no tests (Python 168 files and lines of code) no documentation, some comments are in Italian duplicated/dead code 21 / 69

22 State of Union Python 2.7 (support 2020) Django 1.8 (support april 2018) few dependencies are deprecated or no maintainers (incompatible with django 1.11) SQLite no tests (Python 168 files and lines of code) no documentation, some comments are in Italian duplicated/dead code no async for jobs 22 / 69

23 State of Union Python 2.7 (support 2020) Django 1.8 (support april 2018) few dependencies are deprecated or no maintainers (incompatible with django 1.11) SQLite no tests (Python 168 files and lines of code) no documentation, some comments are in Italian duplicated/dead code no async for jobs no Continuous Integration / Continuous Deployment 23 / 69

24 State of Union Python 2.7 (support 2020) Django 1.8 (support april 2018) few dependencies are deprecated or no maintainers (incompatible with django 1.11) SQLite no tests (Python 168 files and lines of code) no documentation, some comments are in Italian duplicated/dead code no async for jobs no Continuous Integration / Continuous Deployment no API for external tools need to export data (mobile app, ticket search app, etc...) 24 / 69

25 State of Union Python 2.7 (support 2020) Django 1.8 (support april 2018) few dependencies are deprecated or no maintainers (incompatible with django 1.11) SQLite no tests (Python 168 files and lines of code) no documentation, some comments are in Italian duplicated/dead code no async for jobs no Continuous Integration / Continuous Deployment no API for external tools need to export data (mobile app, ticket search app, etc...) no syslog, just send error with exception to the mailing list. 25 / 69

26 State of Union Python 2.7 (support 2020) Django 1.8 (support april 2018) few dependencies are deprecated or no maintainers (incompatible with django 1.11) SQLite no tests (Python 168 files and lines of code) no documentation, some comments are in Italian duplicated/dead code no async for jobs no Continuous Integration / Continuous Deployment no API for external tools need to export data (mobile app, ticket search app, etc...) no syslog, just send error with exception to the mailing list. settings were hardcoded, no environment variables 26 / 69

27 27 / 69

28 and seriously we are not alone in this case. 28 / 69

29 But, don't forget one thing! 29 / 69

30 in fact, there is a lot of things 30 / 69

31 in fact, there is a lot of things Many former developers (it's an open source project) Inherited from PyCon Italia contributors since 2009 (first version) 31 / 69

32 in fact, there is a lot of things Many former developers (it's an open source project) Inherited from PyCon Italia contributors since 2009 (first version) Volunteers 32 / 69

33 in fact, there is a lot of things Many former developers (it's an open source project) Inherited from PyCon Italia contributors since 2009 (first version) Volunteers Free time 33 / 69

34 in fact, there is a lot of things Many former developers (it's an open source project) Inherited from PyCon Italia contributors since 2009 (first version) Volunteers Free time No web dev, but data scientist, backend dev 34 / 69

35 We have a real challenge 35 / 69

36 We have a real challenge Continuous Integration / Continuous Deployment 36 / 69

37 We have a real challenge Continuous Integration / Continuous Deployment Documentation 37 / 69

38 We have a real challenge Continuous Integration / Continuous Deployment Documentation Configuration 38 / 69

39 We have a real challenge Continuous Integration / Continuous Deployment Documentation Configuration Tests 39 / 69

40 We have a real challenge Continuous Integration / Continuous Deployment Documentation Configuration Tests Write Code/Refactoring 40 / 69

41 We have a real challenge Continuous Integration / Continuous Deployment Documentation Configuration Tests Write Code/Refactoring Quality of code 41 / 69

42 We have a real challenge Continuous Integration / Continuous Deployment Documentation Configuration Tests Write Code/Refactoring Quality of code Profiling 42 / 69

43 We have a real challenge Continuous Integration / Continuous Deployment Documentation Configuration Tests Write Code/Refactoring Quality of code Profiling Deployment 43 / 69

44 We have a real challenge Continuous Integration / Continuous Deployment Documentation Configuration Tests Write Code/Refactoring Quality of code Profiling Deployment Monitoring 44 / 69

45 Continuous Integration Travis ( language: python python: - "2.7" install: - pip install -r requirements-dev.txt script: "python manage.py compilemessages && python manage.py test" 45 / 69

46 Documentation Sphinx 46 / 69

47 Con guration django-dotenv Your.env file POSTGRES_USER="postgres" POSTGRES_PASSWORD="" SECRET_KEY="whoami" ALLOWED_HOSTS='' CSRF_COOKIE_SECURE=False DEBUG=True ENVIRONMENT='LOCAL' DJANGO_SETTINGS_MODULE=dev-settings Your manage.py file import dotenv if name == ' main ': dotenv.read_dotenv() os.environ.setdefault("django_settings_module", "project.settings") / 69

48 Tests unittest / pytest django.test / pytest-django mock / pytest-mock 48 / 69

49 Tests coverage / pytest-cov django-coverage-plugin 49 / 69

50 Tests django-test-without-migrations and pytest-django python manage test --nomigrations # or pytest --nomigrations 50 / 69

51 Tests factory_boy and django-factoryboy class TalkFactory(factory.django.DjangoModelFactory): class Meta: model = 'conference.talk' title = factory.lazyattribute(lambda talk: fake.sentence(nb_words=6, variable_n sub_title = factory.faker('sentence', nb_words=12, variable_nb_words=true) slug = factory.lazyattribute(lambda talk: slugify(talk.title)) level = factory.iterator(conference.models.talk_level, getter=lambda x: x[0]) status = factory.iterator(conference.models.talk_status, getter=lambda x: x[0]) conference = factory.iterator(conference.models.conference.objects.all().values language = factory.iterator(talk_languages, getter=lambda x: x[0]) Definition of a Factory in app/tests/factories/model.py 51 / 69

52 Write Code/Refactoring These tools will help your code With the tests, you can start to rewrite the bad code pyflake8, pylint vulture django-pdb and --ipdb pdbpp (pdb++) (for pytest) or ipdb 52 / 69

53 Write Code/Refactoring These tools will help your code With the tests, you can start to rewrite the bad code mypy with mypy-django gnrbag.py:84: error: Incompatible types in assignment (expression has type "classmethod", var gnrbag.py:317: error: Name 'json' already defined gnrbag.py:2874: error: Need type annotation for variable gnrbag.py:3083: error: Dict entry 3 has incompatible type "str": "str" ` 53 / 69

54 Write Code/Refactoring These tools will help your code With the tests, you can start to rewrite the bad code autoflake --remove-all-unused-imports isort 54 / 69

55 Pro ling django-debug-toolbar 55 / 69

56 Pro ling django-devserver + line_profiler [11/Jul/ :26:50] "GET /favicon.ico HTTP/1.1" [11/Jul/ :26:50] "GET /en/favicon.ico HTTP/1.1" [sql] (3ms) 1 queries with 0 duplicates [profile] Total time to render was 0.03s [profile] Timer unit: 1e-06 s Total time: s File: /home/stephane/.virtualenvs/ep2017/lib/python2.7/site-packages/cms/views.py Function: details at line 23 Line # Hits Time Per Hit % Time Line Contents ============================================================== 23 def details(request, slug): 24 """ 25 The main view of the Django-CM 26 page. 27 """ 56 / 69

57 Pro ling pytest-profiling with --profile-svg --durations=2 $ pytest --durations=2 --profile-svg ======================== slowest 2 test durations =========================================== 0.47s setup assopy/tests/test_reset_password.py::resetpasswordtestcase::test_reset_passwor 0.36s call assopy/tests/test_stripe.py::stripeviewtestcase::test_add_stripe_on_order_test if you are a purist, maybe you will prefer cprofile 57 / 69

58 django_load:48:load 2.28% % test_view s:174:test_conference_talk test_profile:30:test_p3_account_data_get 3.72% 1.09% 0.51% 3.63% test_view s:109:test_p3_schedule_list 1.09% context_processors:16:_get_m enu_renderer 2.52% m enu_pool:262:get_renderer 2.52% m enu_pool:269:discover_m enus 2.49% % % % % lru_cache:94:w rapper 2.77% (0.01% ) 2.52% % test_profile:74:test_p3_profile 0.83% 0.51% 0.82% 2.13% urlresolvers:524:reverse 4.96% 0.83% test_view s:32:test_p3_w hos_com ing_w ith_conference 0.84% 0.88% test_cart:19:test_p3_cart 0.84% test_reset_passw ord:6:test_reset_passw ord 5.24% 5.24% 8.03% 10 client:495:get 22.16% core:145:render 8.04% (0.02% ) 146 helpers:74:render_tag cm s_tags:473:render_tag 3.75% 8.02% 10 defaults:9:page_not_found 1.02% 3.34% 10 m enu_tags:119:get_context 3.34% % 3.75% 10 sekizai_tags:90:render_tag 3.34% 8.03% % % 10 base:901:render 14.22% (0.08% ) % 10 debug:77:render_node 14.21% (0.04% ) % 14 utils:193:inner 6.97% (0.01% ) 1.95% 84 helpers:28:render_tag trans_real:485:get_language_from _request toolbar_base:11: init toolbar:107:init_toolbar 1.95% 0.51% 3.83% 0.67% 0.60% (0.01% ) 5 (0.01% ) (0.01% ) % % test_profile:37:test_p3_account_data_post 1.58% test_view s:100:test_p3_schedule test_view s:204:test_p3_schedule_m y_schedule_ics_error_404 test_view s:83:test_conference_schedule_xm l 0.90% 1.35% 0.61% 3.11% 0.86% 0.82% functional:132: w rapper 3.34% 26 m enu_pool:142:_build_nodes 0.81% (0.01% ) % 10 cm s_m enus:185:get_nodes 0.62% (0.01% ) 10 defaulttags:317:render defaulttags:472:render 4.02% 2.25% (0.01% ) (0.01% ) % % % 10 m enu_pool:233:get_nodes 0.82% % test_view s_live:34:test_live 0.88% 0.71% decorators:99:_w rapped_view 1.02% 3.98% % % 2.25% 27 debug:87:render 2.08% (0.04% ) % 1.77% % 1.28% base:1227:render 1.27% % 1.35% 0.89% 40 test_profile:103:test_p3_profile_m essage_accept_m essage test_stripe:24:test_add_stripe_on_order_test test_stripe:49:test_stripe_get test_m odels:86:test_send_user_m essage 1.56% 1.51% 4.32% 0.61% 2.71% 1.03% 40 client:505:post 5.65% loader:81:render_to_stringcm s_tags:384:get_value cm s_tags:137:get_value init :41:get_language_from _request functional:56: get 0.99% 9.47% 0.89% 1.03% 0.61% 0.50% (0.01% ) (0.01% ) (0.03% ) (0.06% ) loader_tags:51:render loader_tags:112:render 9.44% 12.54% (0.02% ) % 0.61% test_view s:22:test_p3_w hos_com ing_no_conference 1.03% testcases:243:assertr edirects 0.90% % 158 test_view s:153:test_conference_sponsor 1.26% 0.89% 22.16% 3 base:84:get_exception_response 1.03% 1.27% % % 9.44% % base:645:resolve 2.05% 8.04% % 1.26% client:295:get 22.16% 33 utils:90:instrum ented_test_render 14.24% (0.01% ) 14.22% % % % init :197:get_language_from _request 0.52% 37 4 client:644:_handle_redirects 4.23% 4.23% % % 0.15% 3.61% 2.04% 4 client:305:post 2.04% % 4 client:353:generic 24.19% (0.01% ) locale:29:process_request decorators:60:w rappertoolbar:43:process_request 0.52% 4.33% 2.06% (0.01% ) 5.49% 17 loader:23:get_tem plate 1.31% % % % 3.79% schedule:128:schedule init :382:w hos_com ing live:31:live 0.79% 1.60% 0.58% % 1.51% shortcuts:50:render 0.38% 5.53% % 10 django:44:render engine:179:render_to_string cm s_tags:52:_get_page_by_untyped_arg i18n:67:get_language_list 12.55% 3.83% 1.63% 0.67% (0.02% ) (0.02% ) 1.30% % % shortcuts:27:render_to_response 0.51% 3.84% % % % 3.83% 5 loader_tags:145:render 2.46% % 2.71% 28 5 base:204:render 14.70% (0.01% ) 2.33% % % % % % % 1.48% % 5 35 toolbar:41: init 1.48% (0.03% ) % % % 36 client:428:request 24.15% (0.01% ) 36 client:105: call 24.07% (0.01% ) 0.21% % % % base:94:get_response 23.79% (0.04% ) 0.53% % % % 1.60% % 0.58% 0.19% % % 0.94% runner:96:pytest_runtest_callpluggy:238:_w rapped_call 99.89% 99.99% % (0.01% ) % case:333:run 99.64% (0.05% ) % 7 m ock:1289:patched 8.54% 4.32% % pluggy:598:execute % (0.01% ) 4.33% test_m odels:48:setu p 4.33% 3 pluggy:263: init % (0.01% ) 2.71% 99.90% % 0.52% 1.35% test_m odels:16:setu p 0.85% functional:223:inner 0.57% (0.01% ) 114 functional:102: prepare_class 0.65% 0.11% % 6 cachef:137:w rapper 1.29% (0.01% ) % % 5 dataaccess:14:profile_data 0.58% % 9 test_view s:45:setu p 0.85% 15.05% 6 9 decorators:19:_w rapped_viewresponse:149:render decorators:80:w rapper~:0:<_socket.gethostbyaddr> 2.34% 6.14% 0.87% 1.31% (1.32% ) 0.56% 2.16% % 0.66% 0.87% cart:79:cart schedule:150:schedule_list 0.73% 0.94% 1.87% 0.68% 99.89% 84 unittest:174:runtest 99.89% (0.01% ) % 79 testcases:170: call 99.88% (0.01% ) % 79 case:430: call 99.64% % 84 profile:100:p3_account_data 0.68% 1.87% 0.78% 6.05% 99.89% 84 m odels:489:send_user_m essage 1.37% 1.34% m essage:297:send 1.34% 1.33% locm em :22:send_m essages 1.33% 1.33% m essage:264:m essage 1.33% m essage:42:m ake_m sgid 1.32% 6.14% 0.83% profile:26:p3_profile 0.83% response:124:rendered_content 1.00% 6.14% m odels:445:save 1.00% 0.63% % 84 i18n:22:get_languages 0.74% (0.02% ) % 870 conf:276:get_cm s_setting 0.68% (0.03% ) % 1.32% utils:11: str 1.32% m odels:70:save_instance 1.00% 0.61% 870 conf:223:get_languages 0.61% (0.09% ) % 870 functional:188: w rapper 0.70% (0.02% ) % 1279 functional:89: init 0.68% (0.03% ) % 0.65% 1279 decorators:20:w rapper 0.68% 0.52% % 1.36% utils:14:get_fqdn 1.32% socket:128:getfqdn 1.32% 0.11% 1.32% 1.31% base:60: call 59.57% (0.01% ) % 1 test_stats:14:setu p 4.50% 12.37% % % % test_cart:12:setu p 1.80% 1.45% 2.71% % % 2.67% % 6.79% % 1 test_m odels:32:test_profile 1.24% 1.80% 3 test_view s:15:setu p test_view s_live:18:setu p 3.34% 6.79% 5.59% % % client:584:login 10.70% (0.02% ) 3 client:411:_session 0.71% (0.01% ) 0.32% % % 1.20% % % 6 68 test_view s:18:setu p 10.17% 11.08% % % % test_stripe:16:setu p 2.54% 1.61% 13.58% 8 test_profile:14:setu p 13.58% 2.50% 8 8 Pro ling with --profile-svg you can get this result 58 / 69

59 Pro ling cprofilev is a web interface for the cprofile output pip install cprofilev cprofilev -f prof/test_reset_password.prof [cprofilev]: cprofile output available at 59 / 69

60 Deployment We already use: Docker docker-compose 60 / 69

61 Deployment We already use: Docker docker-compose But for the Continuous Development we need Kubernetes / Docker Swarm 61 / 69

62 Monitoring sentry (for the logs) supervisord or container orchestration tool shinken or nagios 62 / 69

63 63 / 69

64 State of Union We have Travis Tests (97 tests, before just 3) Code coverage (32%, before just 10%) Started the port to PostgreSQL 9.6 Profiling when we develop if needed 64 / 69

65 State of Union The next steps Add documentation Remove the dead code Move to Django 1.11.x or Django 2.0 Use Python >= 3.6 Use PostgreSQL Use Celery Provide API (REST) for the mobile and web applications Single Page App with ReactJS or VueJS (no idea) 65 / 69

66 Is there a Django Expert in the room? 66 / 69

67 How to contribute? Sprint Code on this week-end Join the Web Team 67 / 69

68 Share your best practices 68 / 69

69 Questions? github.com/matrixise 69 / 69