The Codependent Codrhttps://www.codependentcodr.com/2023-08-25T15:06:00-07:00Confessions of an ImpostorUsing namedtuple's to convert dicts to objects2023-08-25T15:06:00-07:002023-08-25T15:06:00-07:00Adam Parkintag:www.codependentcodr.com,2023-08-25:/using-namedtuples-to-convert-dicts-to-objects.html<p>Quick and dirty way to convert a dict into an object in Python using namedtuple's</p><p>A friend shared this with me today and I thought it was pretty neat. If you have a dict, and
you want to convert it to an object where the object properties are the keys from the dict, and
the values are the values from the dict, you can use a namedtuple to do so. For example:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="n">some_dict</span> <span class="o">=</span> <span class="p">{</span><span class="s2">"name"</span><span class="p">:</span> <span class="s2">"My name"</span><span class="p">,</span> <span class="s2">"func"</span> <span class="p">:</span> <span class="s2">"my func"</span><span class="p">}</span>
<span class="o">>>></span> <span class="kn">import</span> <span class="nn">namedtuple</span>
<span class="o">>>></span> <span class="n">SomeClass</span> <span class="o">=</span> <span class="n">namedtuple</span><span class="p">(</span><span class="s2">"SomeClass"</span><span class="p">,</span> <span class="n">some_dict</span><span class="o">.</span><span class="n">keys</span><span class="p">())</span>
<span class="o">>>></span> <span class="n">as_an_object</span> <span class="o">=</span> <span class="n">SomeClass</span><span class="p">(</span><span class="o">**</span><span class="n">some_dict</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">as_an_object</span><span class="o">.</span><span class="n">name</span>
<span class="s1">'My name'</span>
<span class="o">>>></span> <span class="n">as_an_object</span><span class="o">.</span><span class="n">func</span>
<span class="s1">'my func'</span>
</pre></div>
<p>Won't handle nested dicts (the sub-dicts will still be dicts on the constructed object), but
for a quick and dirty way to convert a dict to an object, this seems pretty handy.</p>
<p>Using the splat operator you can also save a line of code:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="n">as_an_object</span> <span class="o">=</span> <span class="n">namedtuple</span><span class="p">(</span><span class="s2">"SomeClass"</span><span class="p">,</span> <span class="n">some_dict</span><span class="o">.</span><span class="n">keys</span><span class="p">())(</span><span class="o">**</span><span class="n">some_dict</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">as_an_object</span>
<span class="n">SomeClass</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s1">'My name'</span><span class="p">,</span> <span class="n">func</span><span class="o">=</span><span class="s1">'my func'</span><span class="p">)</span>
</pre></div>Python Tip of the Day - contextlib.contextmanager2021-10-21T19:52:00-07:002021-10-21T19:52:00-07:00Adam Parkintag:www.codependentcodr.com,2021-10-21:/python-tip-of-the-day-contextlibcontextmanager.html<p>How to use contextlib.contextmanager to combine context managers into one</p><p>So recently at work I had a test that did a fair bit of patching out of some
dependent functions/classes. The test looked something like:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">unittest.mock</span> <span class="kn">import</span> <span class="n">patch</span>
<span class="k">def</span> <span class="nf">test_patching_a_lot</span><span class="p">():</span>
<span class="k">with</span> <span class="n">patch</span><span class="p">(</span><span class="s2">"path.to.module.somefunction"</span>
<span class="p">),</span> <span class="n">patch</span><span class="p">(</span><span class="s2">"path.to.another.function"</span><span class="p">),</span> <span class="n">patch</span><span class="p">(</span>
<span class="s2">"some.third.party.function"</span>
<span class="p">),</span> <span class="n">patch</span><span class="p">(</span><span class="s2">"you.get.the.idea"</span><span class="p">),</span> <span class="n">patch</span><span class="p">(</span>
<span class="s2">"perhaps.there.shouldnt.be.so.much.patching"</span>
<span class="p">),</span> <span class="n">patch</span><span class="p">(</span>
<span class="s2">"but.thats.a.topic.for.another.time"</span>
<span class="p">):</span>
<span class="o">...</span> <span class="n">tbe</span> <span class="n">body</span> <span class="n">of</span> <span class="n">the</span> <span class="n">test</span> <span class="o">...</span>
</pre></div>
<p>You might say "holy cow Adam, six patch calls is an awful lot", but that's a topic
for another time. I was stuck with this test as written. The thing is though,
I looked further on in the same test module and found another test with the
exact same set of patch calls. And I discovered both of these tests because I
was about to have to add another test for the functionality I was adding, which would
also have to do the same set of patching.</p>
<p>Ok, Software Engineering 101, the DRY (or "Don't Repeat Yourself" principle): if
you have the same set of lines repeated many (usually 3 or more) times, it's
time to factor those lines out to a function so that if those lines need to
change, you only make the change in one place, not many. But the problem here
is that these are context managers, so if I factored them out to a function,
then the patching wouldn't be in effect in the test. Example:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">unittest.mock</span> <span class="kn">import</span> <span class="n">patch</span>
<span class="k">def</span> <span class="nf">helper</span><span class="p">():</span>
<span class="k">with</span> <span class="n">patch</span><span class="p">(</span><span class="s2">"path.to.module.somefunction"</span>
<span class="p">),</span> <span class="n">patch</span><span class="p">(</span><span class="s2">"path.to.another.function"</span><span class="p">),</span> <span class="n">patch</span><span class="p">(</span>
<span class="s2">"some.third.party.function"</span>
<span class="p">),</span> <span class="n">patch</span><span class="p">(</span><span class="s2">"you.get.the.idea"</span><span class="p">),</span> <span class="n">patch</span><span class="p">(</span>
<span class="s2">"perhaps.there.shouldnt.be.so.much.patching"</span>
<span class="p">),</span> <span class="n">patch</span><span class="p">(</span>
<span class="s2">"but.thats.a.topic.for.another.time"</span>
<span class="p">):</span>
<span class="k">return</span>
<span class="k">def</span> <span class="nf">test_patching_a_lot</span><span class="p">():</span>
<span class="n">helper</span><span class="p">()</span>
<span class="o">...</span> <span class="n">tbe</span> <span class="n">body</span> <span class="n">of</span> <span class="n">the</span> <span class="n">test</span><span class="p">,</span> <span class="n">but</span> <span class="n">at</span> <span class="n">this</span> <span class="n">point</span><span class="p">,</span> <span class="n">the</span> <span class="n">context</span> <span class="n">managers</span> <span class="n">are</span> <span class="n">no</span> <span class="n">longer</span> <span class="ow">in</span> <span class="n">effect</span> <span class="o">...</span>
</pre></div>
<p>And that's where
<a href="https://docs.python.org/3/library/contextlib.html#contextlib.contextmanager"><code>contextlib.contextmanager</code> from the standard library</a>
comes to the rescue. With this
gem from the standard library, you can decorate a function with the <code>@contextmanager</code> decorator,
and that function is now a context manager that you can use in a <code>with</code> clause:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">contextlib</span> <span class="kn">import</span> <span class="n">contextmanager</span>
<span class="kn">from</span> <span class="nn">unittest.mock</span> <span class="kn">import</span> <span class="n">patch</span>
<span class="nd">@contextmanager</span>
<span class="k">def</span> <span class="nf">helper</span><span class="p">():</span>
<span class="k">with</span> <span class="n">patch</span><span class="p">(</span><span class="s2">"path.to.module.somefunction"</span>
<span class="p">),</span> <span class="n">patch</span><span class="p">(</span><span class="s2">"path.to.another.function"</span><span class="p">),</span> <span class="n">patch</span><span class="p">(</span>
<span class="s2">"some.third.party.function"</span>
<span class="p">),</span> <span class="n">patch</span><span class="p">(</span><span class="s2">"you.get.the.idea"</span><span class="p">),</span> <span class="n">patch</span><span class="p">(</span>
<span class="s2">"perhaps.there.shouldnt.be.so.much.patching"</span>
<span class="p">),</span> <span class="n">patch</span><span class="p">(</span>
<span class="s2">"but.thats.a.topic.for.another.time"</span>
<span class="p">):</span>
<span class="k">yield</span>
<span class="k">def</span> <span class="nf">test_patching_a_lot</span><span class="p">():</span>
<span class="k">with</span> <span class="n">helper</span><span class="p">():</span>
<span class="o">...</span> <span class="n">tbe</span> <span class="n">body</span> <span class="n">of</span> <span class="n">the</span> <span class="n">test</span><span class="p">,</span> <span class="ow">and</span> <span class="nb">all</span> <span class="n">context</span> <span class="n">managers</span> <span class="ow">in</span> <span class="n">helper</span> <span class="n">are</span> <span class="ow">in</span> <span class="n">effect</span> <span class="o">...</span>
</pre></div>
<p>Super handy, and very concise. With this I was able to factor out all that
gross patching to a single function.</p>Git Tip of the Day - Committer vs Author2021-08-06T11:28:00-07:002021-08-06T11:28:00-07:00Adam Parkintag:www.codependentcodr.com,2021-08-06:/git-tip-of-the-day-committer-vs-author.html<p>There's a distinction in git between committer and author. Lets learn about that.</p><p>So one thing that has always kinda puzzled me in Git is that when I'd <code>amend</code> a
commit, then do a <code>git log</code> the timestamp for that commit seemed unchanged even
though I've effectively "rewritten" that commit. I dug into this a bit today and
learned about the distinction between Author and Committer which proved
insightful around this.</p>
<p>The distinction is well summarized in the <a href="https://git-scm.com/book/en/v2/Git-Basics-Viewing-the-Commit-History">Git
docs</a>:</p>
<blockquote>
<p>You may be wondering what the difference is between author and committer. The
author is the person who originally wrote the work, whereas the committer is
the person who last applied the work.</p>
</blockquote>
<p>So when you <code>amend</code> a commit you're updating the <code>committer</code> aspects of that
commit and the <code>author</code> aspects remain unchanged. You can see both in Git log
by using the <code>--format=fuller</code> argument:</p>
<div class="highlight"><pre><span></span>$ git log --format<span class="o">=</span>fuller
commit 9324ea7390b5c411c5cc050cf80965ce7425887a <span class="o">(</span>HEAD -> foobar<span class="o">)</span>
Author: Adam Parkin <obfuscated@gmail.com>
AuthorDate: Fri Aug <span class="m">6</span> <span class="m">11</span>:37:15 <span class="m">2021</span> -0700
Commit: Adam Parkin <obfuscated@gmail.com>
CommitDate: Fri Aug <span class="m">6</span> <span class="m">11</span>:37:15 <span class="m">2021</span> -0700
Test commit
</pre></div>
<p>You can see in this that the commit in question was originally authored on
August 6th, 2021 at 11:37AM (PST). Since this was the initial commit, the
<code>CommitDate</code> is the same. But if we amend that commit & check again we'll see
that the <code>CommitDate</code> is updated:</p>
<div class="highlight"><pre><span></span>$ git commit --amend
<span class="o">[</span>foobar 857e2a5<span class="o">]</span> Test commit
Date: Fri Aug <span class="m">6</span> <span class="m">11</span>:37:15 <span class="m">2021</span> -0700
<span class="m">1</span> file changed, <span class="m">0</span> insertions<span class="o">(</span>+<span class="o">)</span>, <span class="m">0</span> deletions<span class="o">(</span>-<span class="o">)</span>
create mode <span class="m">100644</span> content/gtotd-commit-vs-author.md
$ git log --format<span class="o">=</span>fuller
commit 857e2a58ac0fe3c295ab80efe4cf21f42986fcfb <span class="o">(</span>HEAD -> foobar<span class="o">)</span>
Author: Adam Parkin <obfuscated@gmail.com>
AuthorDate: Fri Aug <span class="m">6</span> <span class="m">11</span>:37:15 <span class="m">2021</span> -0700
Commit: Adam Parkin <obfuscated@gmail.com>
CommitDate: Fri Aug <span class="m">6</span> <span class="m">11</span>:38:34 <span class="m">2021</span> -0700
Test commit
</pre></div>
<p>In this case the amend was also done by me, so the author & committer are the
same.</p>
<p>If you want to update the <code>AuthorDate</code> you can do so with the <code>--date</code> option to
<code>git commit</code>:</p>
<div class="highlight"><pre><span></span>$ git commit --amend --date <span class="s1">'now'</span>
<span class="o">[</span>foobar d45d5f9<span class="o">]</span> Test commit
Date: Fri Aug <span class="m">6</span> <span class="m">11</span>:40:34 <span class="m">2021</span> -0700
<span class="m">1</span> file changed, <span class="m">0</span> insertions<span class="o">(</span>+<span class="o">)</span>, <span class="m">0</span> deletions<span class="o">(</span>-<span class="o">)</span>
create mode <span class="m">100644</span> content/gtotd-commit-vs-author.md
$ git log --format<span class="o">=</span>fuller
commit d45d5f9ea0579b6ea6fb4503a50abbac92a348a4 <span class="o">(</span>HEAD -> foobar<span class="o">)</span>
Author: Adam Parkin <obfuscated@gmail.com>
AuthorDate: Fri Aug <span class="m">6</span> <span class="m">11</span>:40:34 <span class="m">2021</span> -0700
Commit: Adam Parkin <obfuscated@gmail.com>
CommitDate: Fri Aug <span class="m">6</span> <span class="m">11</span>:40:34 <span class="m">2021</span> -0700
Test commit
</pre></div>
<p>Here we can see that the use of <code>--date</code> set both <code>AuthorDate</code> and <code>CommitDate</code>
to the same value (now).</p>F-Strings Are F'ing Cool Part 22021-07-11T15:27:00-07:002021-07-11T15:27:00-07:00Adam Parkintag:www.codependentcodr.com,2021-07-11:/f-strings-are-fing-cool-part-2.html<p>I learned some new tricks with f-strings, lets check 'em out.</p><p>So in a
<a href="https://www.codependentcodr.com/f-strings-are-fing-cool.html">previous post</a>
a couple years ago I wrote about some of the reasons the f-string functionality
introduced in Python 3.6 was so useful. Today I learned a few new tricks with
them from
<a href="https://pythonbytes.fm/episodes/show/241/f-yes-we-want-some-f-string-tricks">a recent episode of the Python Bytes Podcast</a>
so wanted to revisit the topic & share.</p>
<h2>Debugging</h2>
<p>So how many times have you been debugging, wanted to know what the value of a
variable was at a particular point, and then wrote something like:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="n">myvariable</span> <span class="o">=</span> <span class="mi">42</span>
<span class="o">>>></span> <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"The value of myvariable is: </span><span class="si">{</span><span class="n">myvariable</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="n">The</span> <span class="n">value</span> <span class="n">of</span> <span class="n">myvariable</span> <span class="ow">is</span><span class="p">:</span> <span class="mi">42</span>
</pre></div>
<p>Ok, putting aside the fact I'm using print statements for debugging (😱), this is extremely common.
But what if you had a second variable? Ok, so then you add it as well:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="n">myvariable</span> <span class="o">=</span> <span class="mi">42</span>
<span class="o">>>></span> <span class="n">myothervariable</span> <span class="o">=</span> <span class="mi">99</span>
<span class="o">>>></span> <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"The value of myvariable is: </span><span class="si">{</span><span class="n">myvariable</span><span class="si">}</span><span class="s2"> myothervariable: </span><span class="si">{</span><span class="n">myothervariable</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="n">The</span> <span class="n">value</span> <span class="n">of</span> <span class="n">myvariable</span> <span class="ow">is</span><span class="p">:</span> <span class="mi">42</span> <span class="n">myothervariable</span><span class="p">:</span> <span class="mi">99</span>
</pre></div>
<p>And so on. But I'm lazy, I don't want to repeat the variable name twice, so as
of Python 3.8 there's a
<a href="https://docs.python.org/3/whatsnew/3.8.html#f-strings-support-for-self-documenting-expressions-and-debugging">new specifier to aid with debugging</a>.
By appending an equal sign (<code>=</code>) to the expression the both the name & value get
emitted:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="n">myvariable</span> <span class="o">=</span> <span class="mi">42</span>
<span class="o">>>></span> <span class="n">myothervariable</span> <span class="o">=</span> <span class="mi">99</span>
<span class="o">>>></span> <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"The value of </span><span class="si">{</span><span class="n">myvariable</span><span class="si">=}</span><span class="s2"> </span><span class="si">{</span><span class="n">myothervariable</span><span class="si">=}</span><span class="s2">"</span><span class="p">)</span>
<span class="n">The</span> <span class="n">value</span> <span class="n">of</span> <span class="n">myvariable</span><span class="o">=</span><span class="mi">42</span> <span class="n">myothervariable</span><span class="o">=</span><span class="mi">99</span>
</pre></div>
<p>Super handy. What's really amazing is it works with expressions too:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="n">myvariable</span> <span class="o">=</span> <span class="mi">42</span>
<span class="o">>>></span> <span class="n">myothervariable</span> <span class="o">=</span> <span class="mi">99</span>
<span class="o">>>></span> <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"Calculation: </span><span class="si">{</span><span class="p">(</span><span class="n">myvariable</span> <span class="o">*</span> <span class="mf">3.14159</span> <span class="o">/</span> <span class="n">myothervariable</span><span class="p">)</span><span class="si">=}</span><span class="s2">"</span><span class="p">)</span>
<span class="n">Calculation</span><span class="p">:</span> <span class="p">(</span><span class="n">myvariable</span> <span class="o">*</span> <span class="mf">3.14159</span> <span class="o">/</span> <span class="n">myothervariable</span><span class="p">)</span><span class="o">=</span><span class="mf">1.3327957575757574</span>
</pre></div>
<p>This can be handy when printing out properties of an object:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="k">class</span> <span class="nc">Foo</span><span class="p">:</span>
<span class="o">...</span> <span class="n">bar</span> <span class="o">=</span> <span class="mi">42</span>
<span class="o">...</span>
<span class="o">>>></span> <span class="n">f</span> <span class="o">=</span> <span class="n">Foo</span><span class="p">()</span>
<span class="o">>>></span> <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="n">f</span><span class="o">.</span><span class="n">bar</span><span class="si">=}</span><span class="s2">"</span><span class="p">)</span>
<span class="n">f</span><span class="o">.</span><span class="n">bar</span><span class="o">=</span><span class="mi">42</span>
</pre></div>
<p>It's also particularly handy if you have a function which takes arbitrary parameters
via <code>args</code> and <code>kwargs</code> and you want to see what the values of them are:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="k">def</span> <span class="nf">foo</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="o">...</span> <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="n">args</span><span class="si">=}</span><span class="s2"> </span><span class="si">{</span><span class="n">kwargs</span><span class="si">=}</span><span class="s2">"</span><span class="p">)</span>
<span class="o">...</span>
<span class="o">>>></span> <span class="n">foo</span><span class="p">(</span><span class="mi">242</span><span class="p">,</span> <span class="mi">2342</span><span class="p">,</span> <span class="s2">"adlfk"</span><span class="p">,</span> <span class="n">arg1</span><span class="o">=</span><span class="mi">9234</span><span class="p">,</span> <span class="n">arg2</span><span class="o">=</span><span class="s2">"dfslkj"</span><span class="p">)</span>
<span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="mi">242</span><span class="p">,</span> <span class="mi">2342</span><span class="p">,</span> <span class="s1">'adlfk'</span><span class="p">)</span> <span class="n">kwargs</span><span class="o">=</span><span class="p">{</span><span class="s1">'arg1'</span><span class="p">:</span> <span class="mi">9234</span><span class="p">,</span> <span class="s1">'arg2'</span><span class="p">:</span> <span class="s1">'dfslkj'</span><span class="p">}</span>
</pre></div>
<h2>Common Format Specifiers</h2>
<p>So one of the things I talked about in my last post on F-strings was how you can use
all the same format specifiers you used to use with <code>.format()</code> with F-strings. A
common example of this is to format a float to a specific number of decimal places:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="kn">import</span> <span class="nn">math</span>
<span class="o">>>></span> <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"Pi to 3 digits: </span><span class="si">{</span><span class="n">math</span><span class="o">.</span><span class="n">pi</span><span class="si">:</span><span class="s2">.3f</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="n">Pi</span> <span class="n">to</span> <span class="mi">3</span> <span class="n">digits</span><span class="p">:</span> <span class="mf">3.142</span>
</pre></div>
<p>Or formatting datetimes:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span>
<span class="o">>>></span> <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"Today is </span><span class="si">{</span><span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span><span class="si">:</span><span class="s2">%B %d, %Y</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="n">Today</span> <span class="ow">is</span> <span class="n">July</span> <span class="mi">11</span><span class="p">,</span> <span class="mi">2021</span>
</pre></div>
<p>This is neat and all, but oftentimes these format strings are used in many
places in a project. Say for example you want all dates in your project to be
formatted like the example above. One approach would be to repeat that format
string (<code>%B %d, %Y</code>) in each place you format a date. The problem though is
if that format changes (say in the future you want 2 digit years, or want to
include the day of the week, etc) you'd then have to search & replace every
spot where you've used that format.</p>
<p>Ok, so this is software engineering 101, time for the "Don't Repeat Yourself"
(or DRY) principle to apply, so maybe you extract out a function to format a
datetime:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="k">def</span> <span class="nf">format_date</span><span class="p">(</span><span class="n">datetime_obj</span><span class="p">):</span>
<span class="o">...</span> <span class="k">return</span> <span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="n">datetime_obj</span><span class="si">:</span><span class="s2">%B %d, %Y</span><span class="si">}</span><span class="s2">"</span>
<span class="o">...</span>
<span class="o">>>></span> <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"Today is </span><span class="si">{</span><span class="n">format_date</span><span class="p">(</span><span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">())</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="n">Today</span> <span class="ow">is</span> <span class="n">July</span> <span class="mi">11</span><span class="p">,</span> <span class="mi">2021</span>
</pre></div>
<p>Which works and is perfectly reasonable, but now you have this little function
floating around which makes the f-string expression a little more cumbersome.
One could also make a case that defining a function that is a single line of
code is a bit overengineered (I'm not particularly sympathetic to this view
myself, but some are). And really the format string is a constant, so arguably
it's a little weird to have a function to express it's application.</p>
<p>But, here's where my mind was blown -- you can nest f-string expressions:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="n">DATETIME_FORMAT</span><span class="o">=</span><span class="s1">'%B </span><span class="si">%d</span><span class="s1">, %Y'</span>
<span class="o">>>></span>
<span class="o">>>></span> <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"Today is </span><span class="si">{</span><span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span><span class="si">:{</span><span class="n">DATETIME_FORMAT</span><span class="si">}}</span><span class="s2">"</span><span class="p">)</span>
<span class="n">Today</span> <span class="ow">is</span> <span class="n">July</span> <span class="mi">11</span><span class="p">,</span> <span class="mi">2021</span>
</pre></div>
<p>Now our format expression is a single constant variable, and we just defer to
it wherever we want to format a datetime. If that format changes, we simply
update the constant. Really useful trick.</p>VS Code Tip Of The Day - Selectively Adding Files To A Git Commit2021-04-03T10:26:00-07:002021-04-03T10:26:00-07:00Adam Parkintag:www.codependentcodr.com,2021-04-03:/vs-code-tip-of-the-day-selectively-adding-files-to-a-git-commit.html<p>Recently I figured out a way to selectively add files to a Git commit.</p><p>Small tip: sometimes when working on something I find myself making changes to many
files. I then only want to include a few of those files in my next Git commit. To
date I've always just manually done a <code>git add <file></code> in the terminal for each
file I want to add, but this is tedious if there's many of them.</p>
<p>Turns out there's an easy way to do this in VS Code. If you go to the Source Control
item on the left nav (the icon that looks like a branch), you’ll see a list of all
untracked and modified files. For example:</p>
<p><img alt="Showing Modified Files in the Source Control View" src="https://www.codependentcodr.com/static/imgs/source-control-min.png"></p>
<p>In this screenshot you can see I have a number of files that have been modified (the
ones with the "M" beside them) and two new (untracked) files (the ones with the "U"
beside them).</p>
<p>If you want to see what's changed in any of the modified files, clicking the item will
bring up a diff window. If you then want to include (or "stage") this file for the
next commit, right-click it and pick "Stage Changes":</p>
<p><img alt="Selecting Stage Changes" src="https://www.codependentcodr.com/static/imgs/stagechanges-min.png"></p>
<p>Repeat for each file you want to include, and then do a <code>git commit</code> to complete the
commit. Alternatively if you're anti-terminal you can click the checkmark on this
same view to complete the commit.</p>The Wild World of Apple Silicon2021-03-06T10:33:00-08:002021-03-06T10:33:00-08:00Adam Parkintag:www.codependentcodr.com,2021-03-06:/the-wild-world-of-apple-silicon.html<p>A recap of my experience setting up my new M1-powered Macbook Air.</p><p>I recently took the plunge and obtained a shiny new M1-powered Macbook Air. For
those unfamiliar, last year Apple announced that they were now building
machines with a brand new ARM-based architecture, making the switch from the
long-lived x86 Intel architecture. This brought promises of amazing battery
life, amazing performance, and terrifying compatibility issues. Now that
I've been living with this machine for a week or two, I thought I'd recap
my experience both setting it up, any gotchas or surprises along the way,
as well as my experiences around how well the new architecture works as
experienced through the lens of a developer.</p>
<h2>Lesson 1: Rosetta Works, But Sucks Battery Like You Wouldn't Believe</h2>
<p>Everything I've run through Rosetta has been flawless from a functionality
perspective. Having said that though: anything run through Rosetta does seem to
suck battery life. And not just apps that are typically CPU intensive. For
example: I found that having Dropbox (which doesn't support M1), Itsycal, and
Spectacle constantly running in my menu bar all seemed to have a significant
drain on battery life. I've since switched from
Dropbox to <a href="https://www.sync.com/">Sync</a>, from Spectacle to
<a href="https://rectangleapp.com/">Rectangle</a>, and have uninstalled Itsycal as
I still haven't found an M1-powered replacement.</p>
<h2>Lesson 2: Which Apps Are M1 Ready is Really Random</h2>
<p>So of these, which would you expect are M1 ready right now?</p>
<ul>
<li>Slack</li>
<li>Chrome</li>
<li>Firefox</li>
<li>Visual Studio Code</li>
<li>Sublime Text</li>
<li>Dropbox</li>
<li>Docker</li>
</ul>
<p>If you answered the first three, then kudos to you, though until very recently
(ie within the last week or so) VS Code only had M1 support via Insiders. It
looks like
<a href="https://forum.sublimetext.com/t/apple-silicon-native-build/54775">Sublime Text 3 will <em>never</em> support M1</a>
and ST4 is still a long ways off, which for a paid product used by <em>a lot</em> of
Mac users is truly mind-blowing to me. The fact that Dropbox still doesn't have
M1 support is just inexcusable at this point (particularly given it's an "always
running" app). Docker has a preview version that's been out for some time, but
full support still seems like a long ways off. Side note: I haven't tried the
preview version, and I don't plan on it as there's been mixed reports on how
stable it is (
<a href="https://blog.earthly.dev/using-apple-silicon-m1-as-a-cloud-engineer-two-months-in/">a positive take</a>
and
<a href="https://twitter.com/mkennedy/status/1360318443661107210">a negative take</a>
).</p>
<h2>Lesson 3: Homebrew is Ready, but Your Obscure Package Might Not Be</h2>
<p>With Homebrew 3.0, the popular package manager is now M1-ready. I can happily
report that the vast majority of packages I use are M1-native. I installed
<code>python3</code>, <code>git</code>, <code>git-extras</code>, <code>bash-completion</code>, <code>pyenv</code>, <code>pipx</code>, <code>starship</code>,
<code>the_silver_searcher</code>, <code>hugo</code>, <code>watch</code> and a bunch of others without
issue, and all seem to be M1 as reported in Activity Monitor.</p>
<p>So what happens when something isn't?</p>
<div class="highlight"><pre><span></span>$ brew install hadolint
Updating Homebrew...
<span class="o">==</span>> Auto-updated Homebrew!
Updated <span class="m">1</span> tap <span class="o">(</span>homebrew/core<span class="o">)</span>.
<span class="o">==</span>> New Formulae
bas55 delve geph4 kotlin-language-server latino libpipeline openmodelica oras <span class="nv">sqlancer</span>
<span class="o">==</span>> Updated Formulae
Updated <span class="m">267</span> formulae.
Error: hadolint: no bottle available!
You can try to install from <span class="nb">source</span> with:
brew install --build-from-source hadolint
Please note building from <span class="nb">source</span> is unsupported. You will encounter build
failures with some formulae. If you experience any issues please create pull
requests instead of asking <span class="k">for</span> <span class="nb">help</span> on Homebrew<span class="err">'</span>s GitHub, Twitter or any other
official channels.
</pre></div>
<p>I asked <a href="https://github.com/Homebrew/brew/issues/10744">about this on Github</a>
and since <code>hadolint</code> is built with <code>ghc</code> and <code>ghc</code> isn't M1 ready (and likely
won't be for some time) you either live without the package, install a separate
Rosetta-based brew installation, or obtain the package from some other means
(in the case of <code>hadolint</code> this is what I did: there are
<a href="https://github.com/hadolint/hadolint/releases/tag/v1.23.0">self-contained binaries on their Github</a>,
so I
<a href="https://github.com/hadolint/hadolint/issues/558">threw the latest in my path</a>)</p>
<p>One pro-tip: this site is awesome for seeing if a package you're interested in
is M1-ready or not: <a href="https://doesitarm.com/kind/homebrew/">https://doesitarm.com/kind/homebrew/</a></p>
<h2>Lesson 4: Homebrew Is Different Now</h2>
<p>One minor gotcha I ran into is that Brew installs to a different directory:
<code>/opt/homebrew</code>. If you have scripts (think things like <code>.bashrc</code> & the like)
that reference the old brew path they'll have to be tweaked.</p>
<h2>Lesson 5: Instant On is Amazing</h2>
<p>This is a minor thing, and honestly I didn't think I'd like it as much as
I do, but M1 Macbooks feature an "instant on" wake up from sleep. And it
truly is "instant". Ie before I've completely opened the lid of my MBA
the screen is already on and awaiting input. This even happens when I have
an external display connected. Contrast this with my Intel-based Macbook
Pro for work which takes a good 30 seconds to resume from sleep (often
longer if I have external displays connected).</p>
<p>Surprisingly this meant I didn't bother installing
<a href="https://apps.apple.com/us/app/amphetamine/id937984704?mt=12">Amphetamine</a>
on this machine since there's no point -- I don't care if my M1 Mac falls
asleep as it wakes up so damn fast.</p>
<h2>Lesson 6: Big Sur Is Less Good</h2>
<p>This is less dev-orientated, but I really don't like Big Sur. The new
Notification Center is annoying and just wastes space on my menubar.
Toast notifications look much bigger on screen (so are more jarring).
Lots of little annoyances with it, none of which are dealbreaking, but
if I had my way I'd have Catalina instead of Big Sur on this machine
(alas, not an option).</p>
<h2>In Summary</h2>
<p>This machine is awesome. It's expensive (as all Macs are), but is super
fast, and (once you get rid of all your Intel apps) sips battery very
lightly.</p>
<p>Docker is really the only thing that I miss at this point from turning this
into a real dev machine. Hopefully full M1 support will arrive for that
though I can't help but wonder if Docker will ever be completely compatible
(if you build a Docker image on M1, can you run that image on an Intel
based machine?)</p>Using Starship For Terminal Prompt Goodness2021-02-27T10:39:00-08:002021-02-27T10:39:00-08:00Adam Parkintag:www.codependentcodr.com,2021-02-27:/using-starship-for-terminal-prompt-goodness.html<p>I recently discovered Starship & converted my old prompt to using it. Lets see what I learned.</p><p>Recently I had the good fortune of attending the
<a href="https://2021.pycascades.com/">2021 iteration of the Pycascades conference</a>,
and there was at least one talk that mentioned <a href="https://starship.rs/">Starship</a>
and at least one other where the presenter happened to be using it. For those
not in the know, Starship is a cross-shell compatible terminal prompt
generator written in Rust that is super fast and super customizable.</p>
<p>I was intrigued, and decided to take my (reasonably sophisticated) Bash prompt
and Starship-ify it. In this post I'll outline some of the things I went
through, lessons learned, and hopefully impart some advice on how to do things
with it.</p>
<p>To begin with, here was my old bash prompt:</p>
<p><img alt="My Old Prompt" src="https://www.codependentcodr.com/static/imgs/oldprompt.png"></p>
<p>That prompt has a bunch of things in it, and shows:</p>
<ul>
<li>exit code of the last command (0 in this case)</li>
<li>the current day & time, along with the TZ info</li>
<li>the time since I last rebooted (2 days in this case)</li>
<li>my current directory</li>
<li>the name of the current branch I’m on (<code>starship</code> in this case) and the
asterisk to indicate there’s uncommitted changes</li>
<li>The lambda symbol as my input symbol (not a Half-Life reference, more a
reference to my functional programming days)</li>
</ul>
<p>Not shown in that screenshot is that when I have a Python virtual environment activated it also gets displayed in the prompt.</p>
<p>The "code" for this that resided in my <code>.bashrc</code> file:</p>
<div class="highlight"><pre><span></span><span class="k">function</span> uptimeinfo <span class="o">{</span>
uptime <span class="p">|</span> perl -ne <span class="s1">'if(/\d\s+up(.*),\s+\d+\s+users/) { $s = $1; $s =~ s/^\s+|\s+$//g; print $s; }'</span>
<span class="o">}</span>
<span class="k">function</span> proml <span class="o">{</span>
<span class="nb">local</span> <span class="nv">BLACK</span><span class="o">=</span><span class="s2">"\[\033[0;30m\]"</span>
<span class="nb">local</span> <span class="nv">GRAY</span><span class="o">=</span><span class="s2">"\[\033[1;30m\]"</span>
<span class="nb">local</span> <span class="nv">RED</span><span class="o">=</span><span class="s2">"\[\033[0;31m\]"</span>
<span class="nb">local</span> <span class="nv">LIGHT_RED</span><span class="o">=</span><span class="s2">"\[\033[1;31m\]"</span>
<span class="nb">local</span> <span class="nv">GREEN</span><span class="o">=</span><span class="s2">"\[\033[0;32m\]"</span>
<span class="nb">local</span> <span class="nv">LIGHT_GREEN</span><span class="o">=</span><span class="s2">"\[\033[1;32m\]"</span>
<span class="nb">local</span> <span class="nv">BROWN</span><span class="o">=</span><span class="s2">"\[\033[0;33m\]"</span>
<span class="nb">local</span> <span class="nv">YELLOW</span><span class="o">=</span><span class="s2">"\[\033[1;33m\]"</span>
<span class="nb">local</span> <span class="nv">BLUE</span><span class="o">=</span><span class="s2">"\[\033[0;34m\]"</span>
<span class="nb">local</span> <span class="nv">LIGHT_BLUE</span><span class="o">=</span><span class="s2">"\[\033[1;34m\]"</span>
<span class="nb">local</span> <span class="nv">PURPLE</span><span class="o">=</span><span class="s2">"\[\033[0;35m\]"</span>
<span class="nb">local</span> <span class="nv">LIGHT_PURPLE</span><span class="o">=</span><span class="s2">"\[\033[1;35m\]"</span>
<span class="nb">local</span> <span class="nv">CYAN</span><span class="o">=</span><span class="s2">"\[\033[0;36m\]"</span>
<span class="nb">local</span> <span class="nv">LIGHT_CYAN</span><span class="o">=</span><span class="s2">"\[\033[1;36m\]"</span>
<span class="nb">local</span> <span class="nv">LIGHT_GRAY</span><span class="o">=</span><span class="s2">"\[\033[0;37m\]"</span>
<span class="nb">local</span> <span class="nv">WHITE</span><span class="o">=</span><span class="s2">"\[\033[1;37m\]"</span>
<span class="k">case</span> <span class="nv">$TERM</span> <span class="k">in</span>
xterm*<span class="o">)</span>
<span class="nv">TITLEBAR</span><span class="o">=</span><span class="s1">'\[\033]0;\u@\h:\w \D{%a %b %d %Y %l:%M%p (%Z%z)}\007\]'</span>
<span class="p">;;</span>
*<span class="o">)</span>
<span class="nv">TITLEBAR</span><span class="o">=</span><span class="s2">""</span>
<span class="p">;;</span>
<span class="k">esac</span>
<span class="nv">PCOLOR</span><span class="o">=</span><span class="s2">"\[\033[\$(promptcol)\]"</span>
<span class="c1"># note that in the following prompt the error code item (\$?) must be the</span>
<span class="c1"># first item in the prompt. Otherwise it'll show the errorcode for the last</span>
<span class="c1"># command executed in producing the prompt.</span>
<span class="nv">PS1</span><span class="o">=</span><span class="s2">"</span><span class="si">${</span><span class="nv">TITLEBAR</span><span class="si">}</span><span class="s2">\</span>
<span class="nv">$BLUE</span><span class="s2"> [</span><span class="nv">$GREEN</span><span class="s2">[\$?] [\D{%a %b %d %Y %l:%M%p (%Z%z)}] [Up: \$(uptimeinfo)] </span><span class="nv">$BROWN</span><span class="s2">\u@\h:\w </span><span class="nv">$LIGHT_GRAY</span><span class="s2">\$(__git_ps1)\</span>
<span class="nv">$BLUE</span><span class="s2">]\</span>
<span class="s2">\n</span><span class="nv">$PCOLOR</span><span class="s2"> λ </span><span class="nv">$LIGHT_GRAY</span><span class="s2">"</span>
<span class="nv">PS2</span><span class="o">=</span><span class="s1">'> '</span>
<span class="nv">PS4</span><span class="o">=</span><span class="s1">'+ '</span>
<span class="o">}</span>
proml
</pre></div>
<p>That's a lot, and to be honest, I might be missing some of the code as
<a href="https://github.com/pzelnip/dotfiles/blob/mainline/dotfiles/.bashrc">my <code>.bashrc</code> file</a>
is full of random little snippets.</p>
<p>It's also shell-specific. If I ever wanted to move to Zsh, or Fish, or whatever, I'd
have to re-invent that (and this has been an impediment for me to switching shells).</p>
<p>So I took this as a starting point and wanted to recreate it in Starship. Starting
point was to install Starship with Brew:</p>
<div class="highlight"><pre><span></span>brew install starship
</pre></div>
<p>Easy enough. Next is to initialize it in your shell of choice. In my case for
Bash this was adding the following to my <code>.bashrc</code>:</p>
<div class="highlight"><pre><span></span><span class="nb">eval</span> <span class="s2">"</span><span class="k">$(</span>starship init bash<span class="k">)</span><span class="s2">"</span>
</pre></div>
<p>There are equivalent instructions on the
<a href="https://starship.rs/">Starship website</a> for other prompts</p>
<p>Lastly, you need a <code>~/.config/starship.toml</code> file where you'll configure your prompt.</p>
<div class="highlight"><pre><span></span>touch ~/.config/starship.toml
</pre></div>
<p>Now open up a new terminal and you should see the default Starship prompt
which is actually quite sophisticated out of the box:</p>
<p><img alt="Default Starship Prompt" src="https://www.codependentcodr.com/static/imgs/defaultstarship.png"></p>
<p>Your output may vary, as many of the items in a Starship prompt are dynamic
depending on your current context. In this you can see:</p>
<ul>
<li>I'm in a directory called <code>dotfiles</code></li>
<li>I'm currently in a Git repo, but not on any branch instead checked out an
arbitrary commit with SHA <code>931e5c4</code></li>
<li>The coffee cup has to do with Java, but I don't have a valid JDK installed so
it's not showing what version</li>
<li>My current Python environment is Python 3.9.2</li>
<li>My AWS environment is configured to communicate with the <code>ca-central-1</code> region</li>
</ul>
<p>That's a lot! Note that each of those things is what Starship calls a <em>Module</em>.
This is one of the key things about Starship is that each part of your prompt is
made up by a distinct module that you configure. So that prompt is currently
displaying the <a href="https://starship.rs/config/#directory">Directory Module</a>, the
<a href="https://starship.rs/config/#git-branch">Git Branch Module</a>, the
<a href="https://starship.rs/config/#git-commit">Git Commit Module</a>, the
<a href="https://starship.rs/config/#python">Python Module</a>, and the
<a href="https://starship.rs/config/#aws">AWS Module</a>. Technically it's showing a few
others, but this gives you an idea of how Starship <em>composes</em> your prompt by
stringing together some modules.</p>
<p>The full list of all modules can be found at:
<a href="https://starship.rs/config/">https://starship.rs/config/</a></p>
<p>Ok, lets get started trying to configure this prompt to be like my old one. I
opened up my <code>starship.toml</code> and added the contents from the example on the
Starship docs:</p>
<div class="highlight"><pre><span></span><span class="c1"># Don't print a new line at the start of the prompt</span>
<span class="n">add_newline</span> <span class="o">=</span> <span class="kc">false</span>
<span class="c1"># Replace the "❯" symbol in the prompt with "➜"</span>
<span class="p">[</span><span class="n">character</span><span class="p">]</span> <span class="c1"># The name of the module we are configuring is "character"</span>
<span class="n">success_symbol</span> <span class="o">=</span> <span class="s">"[➜](bold green)"</span> <span class="c1"># The "success_symbol" segment is being set to "➜" with the color "bold green"</span>
<span class="c1"># Disable the package module, hiding it from the prompt completely</span>
<span class="k">[package]</span>
<span class="n">disabled</span> <span class="o">=</span> <span class="kc">true</span>
</pre></div>
<p>With this you probably won't see much change, aside from the symbol changing from
a <code>></code> to a <code>➜</code> due to the configuration of the <code>character</code> module. See, everything
in a Starship prompt (including the symbol at the end) is a module. Well, mostly. 😜</p>
<p>Ok, so it currently shows the directory, but not like my old prompt where A) it was
a yellow-ish colour, and B) showed the full path. Looking at the docs for the Directory
module, I gave this a try to configure it:</p>
<div class="highlight"><pre><span></span><span class="k">[directory]</span>
<span class="n">truncation_length</span> <span class="o">=</span> <span class="mi">100</span>
<span class="n">truncate_to_repo</span> <span class="o">=</span> <span class="kc">false</span>
<span class="n">style</span> <span class="o">=</span> <span class="s">"yellow"</span>
<span class="n">format</span> <span class="o">=</span> <span class="s">"[:$path]($style)[$read_only]($read_only_style) "</span>
</pre></div>
<p>Let's explain this a little bit to give a feel:</p>
<ul>
<li><code>truncation_length</code> controls how many directories deep you have to be before
Starship will abbreviate the directory name in your prompt. I rarely go very
deep and to be honest when I do I still want to see the full path so I made
the number rediculously high so that it never truncated</li>
<li><code>truncate_to_repo</code> is a special setting that controls if the directory is
truncated to the root of the Git repo you are currently in. Again, I don't
like this (I want to see the full path), so deactivated it</li>
<li><code>style</code> is a common setting on (I believe) every module and controls the
colour of the module when rendered. In this case saying "yellow" to match my
old prompt. Styles are covered in depth in the
<a href="https://starship.rs/advanced-config/#style-strings">docs</a></li>
<li><code>format</code> is another common setting across every module and controls
effectively the "layout" of the module.</li>
</ul>
<p>It's worth digging into the <code>format</code> directive there, as understanding this goes
a long way in understanding how you control Starship's output. The expression:
<code>"[:$path]($style)[$read_only]($read_only_style) "</code> says start a <em>text group</em>
(this specified by the <code>[</code> and <code>]</code> delimiters) and have it output a colon (<code>:</code>)
followed by the value of the <code>$path</code> variable. Each module has its own set of
variables that get populated with values that are relevant to that module (in
this case <code>$path</code> ends up being the full path of the current working directory).
The brackets that follow a text group specify a <code>style string</code>. You might
wonder "but isn't that what the <code>style</code> setting is for?" And yes, but
essentially the <code>style</code> setting defines the "default" style within a module, and
style strings within the format can override that. In this case, <code>$style</code>
corresponds to the <code>style</code> setting defined in the configuration for the module
(in this case "yellow"). You can see that later in this definition I have the
<code>$read_only</code> variable (which gets displayed when the current working directory
is read only) and has a different style defined for that scenario (the default
<code>$read_only_style</code> is "red", but you could change that in this configuration by
adding a <code>read_only_style="blue"</code> setting to the directory config). In any case
the relevant part of the docs on format strings is
<a href="https://starship.rs/config/#format-strings">here</a>.</p>
<p>Clear as mud? Admittedly, this does take a bit to get your head around (or did
for me), but basically you <em>override</em> settings in the config as appropriate to
tweak each module to your liking.</p>
<p>Ok, that's fine, but Adam how do we control the order of items in the prompt?
And that's a good question that took me a little while to figure out. Turns
out that the prompt as a whole has a <code>format</code> setting. The default is to show
all modules, this is from the docs:</p>
<div class="highlight"><pre><span></span><span class="n">format</span> <span class="o">=</span> <span class="s">"$all"</span>
<span class="c1"># Which is equivalent to</span>
<span class="n">format</span> <span class="o">=</span> <span class="s">"""</span>
<span class="s">$username\</span>
<span class="s">$hostname\</span>
<span class="s">$shlvl\</span>
<span class="s">$kubernetes\</span>
<span class="s">$directory\</span>
<span class="s">$git_branch\</span>
<span class="s">$git_commit\</span>
<span class="s">$git_state\</span>
<span class="s">$git_status\</span>
<span class="s">$hg_branch\</span>
<span class="s">$docker_context\</span>
<span class="s">$package\</span>
<span class="s">$cmake\</span>
<span class="s">$dart\</span>
<span class="s">$dotnet\</span>
<span class="s">$elixir\</span>
<span class="s">$elm\</span>
<span class="s">$erlang\</span>
<span class="s">$golang\</span>
<span class="s">$helm\</span>
<span class="s">$java\</span>
<span class="s">$julia\</span>
<span class="s">$kotlin\</span>
<span class="s">$nim\</span>
<span class="s">$nodejs\</span>
<span class="s">$ocaml\</span>
<span class="s">$perl\</span>
<span class="s">$php\</span>
<span class="s">$purescript\</span>
<span class="s">$python\</span>
<span class="s">$ruby\</span>
<span class="s">$rust\</span>
<span class="s">$swift\</span>
<span class="s">$terraform\</span>
<span class="s">$vagrant\</span>
<span class="s">$zig\</span>
<span class="s">$nix_shell\</span>
<span class="s">$conda\</span>
<span class="s">$memory_usage\</span>
<span class="s">$aws\</span>
<span class="s">$gcloud\</span>
<span class="s">$openstack\</span>
<span class="s">$env_var\</span>
<span class="s">$crystal\</span>
<span class="s">$custom\</span>
<span class="s">$cmd_duration\</span>
<span class="s">$line_break\</span>
<span class="s">$lua\</span>
<span class="s">$jobs\</span>
<span class="s">$battery\</span>
<span class="s">$time\</span>
<span class="s">$status\</span>
<span class="s">$character"""</span>
</pre></div>
<p>This is what controls the order of items. Move an item up, and it'll appear
earlier in the prompt, move down to move it later in the prompt. Personally I
don't like this, as it means if you want to change the order of items you have
to override the <em>entire</em> format string. It'd be nice if there was an "index"
value or something on each module that could determine ordering, but oh well.</p>
<p>In any case it also provides a global "completely hide" ability for a module --
if you remove it from the format then it won't be displayed. Note that I don't
think hiding from the format is the same thing as <em>disabling</em> a module. Each
module has a <code>disabled</code> setting which (if true) disables that module (so won't
get displayed, and I believe not evaluated).</p>
<p>Ok, with this I continued on and got most of my old prompt in place:</p>
<div class="highlight"><pre><span></span><span class="c1"># Don't print a new line at the start of the prompt</span>
<span class="n">add_newline</span> <span class="o">=</span> <span class="kc">false</span>
<span class="k">[character]</span>
<span class="n">success_symbol</span> <span class="o">=</span> <span class="s">" [λ](grey)"</span>
<span class="n">error_symbol</span> <span class="o">=</span> <span class="s">" [λ](bold red)"</span>
<span class="k">[directory]</span>
<span class="n">truncation_length</span> <span class="o">=</span> <span class="mi">100</span>
<span class="n">truncate_to_repo</span> <span class="o">=</span> <span class="kc">false</span>
<span class="n">style</span> <span class="o">=</span> <span class="s">" yellow"</span>
<span class="n">format</span> <span class="o">=</span> <span class="s">"[:$path]($style)[$read_only]($read_only_style) "</span>
<span class="k">[git_branch]</span>
<span class="n">symbol</span> <span class="o">=</span> <span class="s">""</span>
<span class="n">style</span> <span class="o">=</span> <span class="s">"bold white"</span>
<span class="n">format</span> <span class="o">=</span> <span class="s">'[\($symbol$branch\)]($style) '</span>
<span class="k">[git_status]</span>
<span class="c1"># I don't care about untracked files or that there's a stash present.</span>
<span class="n">untracked</span> <span class="o">=</span> <span class="s">""</span>
<span class="n">format</span> <span class="o">=</span> <span class="s">'([\[$conflicted$deleted$renamed$modified$staged$behind\]]($style) )'</span>
<span class="n">modified</span> <span class="o">=</span> <span class="s">'*'</span>
<span class="k">[status]</span>
<span class="n">disabled</span> <span class="o">=</span> <span class="kc">false</span>
<span class="n">format</span> <span class="o">=</span> <span class="s">'[\[$status - $common_meaning\]](green)'</span>
</pre></div>
<p>This is close, but is missing the current time and "uptime". And those proved
to be wrinkles for me. My previous time showed the full current date & time
along with the current timezone info (ex: <code>PST-0800</code>). There's a <code>time</code>
module that can recreate all of this <em>except</em> the timezone name. It should,
given the docs on
<a href="https://docs.rs/chrono/0.4.7/chrono/format/strftime/index.html">format strings for the underlying Chrono library</a>,
but turns out there's a
<a href="https://github.com/starship/starship/discussions/2360">bug there that causes that to not work</a>.</p>
<p>But, Starship supports custom commands to be in a module, so I added a custom
command to just defer to the standard <code>date</code> command on *-nix type systems:</p>
<div class="highlight"><pre><span></span><span class="k">[custom.tztime]</span>
<span class="n">command</span> <span class="o">=</span> <span class="s">'date +"%a %b %d %Y %l:%M%p (%Z%z)"'</span>
<span class="n">when</span> <span class="o">=</span> <span class="s">"true"</span>
<span class="n">format</span> <span class="o">=</span> <span class="s">'[\[$symbol($output)\]](green)'</span>
</pre></div>
<p>This gives me the time as it was before. The <code>when</code> bit there is <code>true</code> so that this
command is <em>always</em> displayed. We'll get to how to control where custom commands
show up in a minute, but there was one more custom command I needed for my system
uptime. The way I did this in my old prompt was a Bash function:</p>
<div class="highlight"><pre><span></span><span class="k">function</span> uptimeinfo <span class="o">{</span>
uptime <span class="p">|</span> perl -ne <span class="s1">'if(/\d\s+up(.*),\s+\d+\s+users/) { $s = $1; $s =~ s/^\s+|\s+$//g; print $s; }'</span>
<span class="o">}</span>
</pre></div>
<p>And then I used <code>uptimeinfo</code> in my prompt. But that's Bash-specific, so instead I
created a little shell script called <code>uptime.sh</code> with the following contents:</p>
<div class="highlight"><pre><span></span><span class="ch">#!/bin/sh</span>
<span class="nb">echo</span> <span class="s2">"[`uptime | perl -ne 'if(/\d\s+up(.*),\s+\d+\s+users/) { </span><span class="nv">$s</span><span class="s2"> = </span><span class="nv">$1</span><span class="s2">; </span><span class="nv">$s</span><span class="s2"> =~ s/^\s+|\s+</span>$<span class="s2">//g; print </span><span class="nv">$s</span><span class="s2">; }'`]"</span>
</pre></div>
<p>Which just does the same thing, echoes out the system uptime with it filtered through Perl
to make it more concise. Now the starship config:</p>
<div class="highlight"><pre><span></span><span class="k">[custom.uptime]</span>
<span class="n">command</span> <span class="o">=</span> <span class="s">"uptime.sh"</span>
<span class="n">when</span> <span class="o">=</span> <span class="s">"true"</span>
<span class="n">format</span> <span class="o">=</span> <span class="s">"[$symbol($output)](green)"</span>
</pre></div>
<p>Now back to that layout question: we control the order of things by the global <code>format</code>
setting, but how do we refer to custom commands? Like this:</p>
<div class="highlight"><pre><span></span><span class="n">format</span> <span class="o">=</span> <span class="s">""</span><span class="err">"</span>
<span class="err">$</span><span class="n">status</span> <span class="err">\</span>
<span class="err">$</span><span class="p">{</span><span class="n">custom</span><span class="p">.</span><span class="n">tztime</span><span class="p">}</span> <span class="err">\</span>
<span class="err">$</span><span class="p">{</span><span class="n">custom</span><span class="p">.</span><span class="n">uptime</span><span class="p">}</span> <span class="err">\</span>
<span class="err">$</span><span class="n">username</span><span class="err">\</span>
<span class="p">....</span> <span class="n">rest</span> <span class="n">of</span> <span class="n">the</span> <span class="n">file</span> <span class="p">...</span>
</pre></div>
<p>Ie <code>${custom.<your custom module name>}</code>. Ok with all that, I then had everything I needed,
and continued to flesh out my prompt. My final config:</p>
<div class="highlight"><pre><span></span><span class="n">format</span> <span class="o">=</span> <span class="s">"""</span>
<span class="s">$status \</span>
<span class="s">${custom.tztime} \</span>
<span class="s">${custom.uptime} \</span>
<span class="s">$username\</span>
<span class="s">$hostname\</span>
<span class="s">$shlvl\</span>
<span class="s">$kubernetes\</span>
<span class="s">$directory\</span>
<span class="s">$git_branch\</span>
<span class="s">$git_commit\</span>
<span class="s">$git_state\</span>
<span class="s">$git_status\</span>
<span class="s">$docker_context\</span>
<span class="s">$package\</span>
<span class="s">$cmake\</span>
<span class="s">$nodejs\</span>
<span class="s">$perl\</span>
<span class="s">$python \</span>
<span class="s">$ruby\</span>
<span class="s">$rust\</span>
<span class="s">$terraform\</span>
<span class="s">$vagrant\</span>
<span class="s">$nix_shell\</span>
<span class="s">$conda\</span>
<span class="s">$aws \</span>
<span class="s">$env_var\</span>
<span class="s">$cmd_duration\</span>
<span class="s">$line_break\</span>
<span class="s">$character"""</span>
<span class="c1"># Don't print a new line at the start of the prompt</span>
<span class="n">add_newline</span> <span class="o">=</span> <span class="kc">false</span>
<span class="k">[aws]</span>
<span class="n">format</span> <span class="o">=</span> <span class="s">'\[AWS: [$profile/($region)]($style)\]'</span>
<span class="n">symbol</span> <span class="o">=</span> <span class="s">''</span>
<span class="n">style</span> <span class="o">=</span> <span class="s">'bold white'</span>
<span class="k">[character]</span>
<span class="n">success_symbol</span> <span class="o">=</span> <span class="s">" [λ](grey)"</span>
<span class="n">error_symbol</span> <span class="o">=</span> <span class="s">" [λ](bold red)"</span>
<span class="k">[cmd_duration]</span>
<span class="n">min_time</span> <span class="o">=</span> <span class="mi">1000</span>
<span class="k">[directory]</span>
<span class="n">truncation_length</span> <span class="o">=</span> <span class="mi">100</span>
<span class="n">truncate_to_repo</span> <span class="o">=</span> <span class="kc">false</span>
<span class="n">style</span> <span class="o">=</span> <span class="s">" yellow"</span>
<span class="n">format</span> <span class="o">=</span> <span class="s">"[:$path]($style)[$read_only]($read_only_style) "</span>
<span class="k">[git_branch]</span>
<span class="n">symbol</span> <span class="o">=</span> <span class="s">""</span>
<span class="n">style</span> <span class="o">=</span> <span class="s">"bold white"</span>
<span class="n">format</span> <span class="o">=</span> <span class="s">'[\($symbol$branch\)]($style) '</span>
<span class="k">[git_status]</span>
<span class="c1"># I don't care about untracked files or that there's a stash present.</span>
<span class="n">untracked</span> <span class="o">=</span> <span class="s">""</span>
<span class="n">format</span> <span class="o">=</span> <span class="s">'([\[$conflicted$deleted$renamed$modified$staged$behind\]]($style) )'</span>
<span class="n">modified</span> <span class="o">=</span> <span class="s">'*'</span>
<span class="k">[python]</span>
<span class="n">format</span> <span class="o">=</span> <span class="s">'[${symbol}${pyenv_prefix}(${version} )(\($virtualenv\))]($style)'</span>
<span class="k">[status]</span>
<span class="n">disabled</span> <span class="o">=</span> <span class="kc">false</span>
<span class="n">format</span> <span class="o">=</span> <span class="s">'[\[$status - $common_meaning\]](green)'</span>
<span class="k">[custom.tztime]</span>
<span class="n">command</span> <span class="o">=</span> <span class="s">'date +"%a %b %d %Y %l:%M%p (%Z%z)"'</span>
<span class="n">when</span> <span class="o">=</span> <span class="s">"true"</span>
<span class="n">format</span> <span class="o">=</span> <span class="s">'[\[$symbol($output)\]](green)'</span>
<span class="k">[custom.uptime]</span>
<span class="n">command</span> <span class="o">=</span> <span class="s">"uptime.sh"</span>
<span class="n">when</span> <span class="o">=</span> <span class="s">"true"</span>
<span class="n">format</span> <span class="o">=</span> <span class="s">"[$symbol($output)](green)"</span>
<span class="k">[env_var]</span>
<span class="n">variable</span> <span class="o">=</span> <span class="s">"0"</span>
<span class="c1">#### Disabled modules ####</span>
<span class="c1"># add these back to format if you want them:</span>
<span class="c1"># $time\</span>
<span class="c1"># $hg_branch\</span>
<span class="c1"># $dart\</span>
<span class="c1"># $dotnet\</span>
<span class="c1"># $elixir\</span>
<span class="c1"># $elm\</span>
<span class="c1"># $erlang\</span>
<span class="c1"># $golang\</span>
<span class="c1"># $helm\</span>
<span class="c1"># $java\</span>
<span class="c1"># $julia\</span>
<span class="c1"># $kotlin\</span>
<span class="c1"># $nim\</span>
<span class="c1"># $ocaml\</span>
<span class="c1"># $php\</span>
<span class="c1"># $purescript\</span>
<span class="c1"># $swift\</span>
<span class="c1"># $zig\</span>
<span class="c1"># $memory_usage\</span>
<span class="c1"># $gcloud\</span>
<span class="c1"># $openstack\</span>
<span class="c1"># $crystal\</span>
<span class="c1"># $lua\</span>
<span class="c1"># $jobs\</span>
<span class="c1"># $battery\</span>
<span class="k">[hg_branch]</span>
<span class="n">disabled</span> <span class="o">=</span> <span class="kc">true</span>
<span class="k">[dart]</span>
<span class="n">disabled</span> <span class="o">=</span> <span class="kc">true</span>
<span class="k">[dotnet]</span>
<span class="n">disabled</span> <span class="o">=</span> <span class="kc">true</span>
<span class="k">[elixir]</span>
<span class="n">disabled</span> <span class="o">=</span> <span class="kc">true</span>
<span class="k">[elm]</span>
<span class="n">disabled</span> <span class="o">=</span> <span class="kc">true</span>
<span class="k">[erlang]</span>
<span class="n">disabled</span> <span class="o">=</span> <span class="kc">true</span>
<span class="k">[golang]</span>
<span class="n">disabled</span> <span class="o">=</span> <span class="kc">true</span>
<span class="k">[helm]</span>
<span class="n">disabled</span> <span class="o">=</span> <span class="kc">true</span>
<span class="k">[java]</span>
<span class="n">disabled</span> <span class="o">=</span> <span class="kc">true</span>
<span class="k">[julia]</span>
<span class="n">disabled</span> <span class="o">=</span> <span class="kc">true</span>
<span class="k">[kotlin]</span>
<span class="n">disabled</span> <span class="o">=</span> <span class="kc">true</span>
<span class="k">[nim]</span>
<span class="n">disabled</span> <span class="o">=</span> <span class="kc">true</span>
<span class="k">[ocaml]</span>
<span class="n">disabled</span> <span class="o">=</span> <span class="kc">true</span>
<span class="k">[php]</span>
<span class="n">disabled</span> <span class="o">=</span> <span class="kc">true</span>
<span class="k">[purescript]</span>
<span class="n">disabled</span> <span class="o">=</span> <span class="kc">true</span>
<span class="k">[swift]</span>
<span class="n">disabled</span> <span class="o">=</span> <span class="kc">true</span>
<span class="k">[zig]</span>
<span class="n">disabled</span> <span class="o">=</span> <span class="kc">true</span>
<span class="k">[memory_usage]</span>
<span class="n">disabled</span> <span class="o">=</span> <span class="kc">true</span>
<span class="k">[gcloud]</span>
<span class="n">disabled</span> <span class="o">=</span> <span class="kc">true</span>
<span class="k">[openstack]</span>
<span class="n">disabled</span> <span class="o">=</span> <span class="kc">true</span>
<span class="k">[crystal]</span>
<span class="n">disabled</span> <span class="o">=</span> <span class="kc">true</span>
<span class="k">[lua]</span>
<span class="n">disabled</span> <span class="o">=</span> <span class="kc">true</span>
<span class="k">[jobs]</span>
<span class="n">disabled</span> <span class="o">=</span> <span class="kc">true</span>
<span class="k">[battery]</span>
<span class="n">disabled</span> <span class="o">=</span> <span class="kc">true</span>
<span class="c1"># Until these get resolved, doing my own datetime with date:</span>
<span class="c1"># https://github.com/starship/starship/discussions/2360#discussioncomment-391911</span>
<span class="c1"># https://github.com/chronotope/chrono/issues/288</span>
<span class="k">[time]</span>
<span class="n">disabled</span> <span class="o">=</span> <span class="kc">true</span>
<span class="c1"># format = '[\[$time\]](green) '</span>
<span class="c1"># time_format = "%a %b %d %Y %l:%M%p (%z)"</span>
</pre></div>
<p>Note current version (in case I revise in the future) is at:
<a href="https://github.com/pzelnip/dotfiles/blob/mainline/.config/starship.toml">https://github.com/pzelnip/dotfiles/blob/mainline/.config/starship.toml</a>)</p>
<p>This gives a prompt like the following:</p>
<p><img alt="New Starship-Powered Prompt" src="https://www.codependentcodr.com/static/imgs/newPrompt.png"></p>
<p>Pretty sweet, lots of dynamicism where needed, but still has all the things I
liked from before. Defintely took some time to get this just the way I liked it,
but am happy with the result, and as a bonus: now I have the same prompt if I'm
in Bash, Zsh or whatever.</p>Building A VS Code Extension Without Installing Node By Docker Magic2021-02-26T16:14:00-08:002021-02-26T16:14:00-08:00Adam Parkintag:www.codependentcodr.com,2021-02-26:/building-a-vs-code-extension-without-installing-node-by-docker-magic.html<p>How to build a VS Code extension without installing Node.js locally.</p><p>I recently wanted to work through the
<a href="https://code.visualstudio.com/api/get-started/your-first-extension">tutorial for building a VS Code extension</a>,
but the first step is to install <a href="https://nodejs.org/en/">Node.js</a>
locally, which, well, I don't want to do. Sorry to the Node peeps out there,
but I don't want to touch the (large and rather complex) Node ecosystem just
to try out building a VS Code extension. So I then thought, "Hey, you can
install Node on a Linux box, so why can't I just do it inside a Docker container?"</p>
<p>And of course, you can, and not only that, but with the magic that is the
<a href="https://code.visualstudio.com/docs/remote/containers">VS Code Remote Containers extension</a>
you can even have VS Code work as if it's all on your local host machine. Let's
give this a try.</p>
<h2>Install The Pre-Requisites</h2>
<p>First step: install the
<a href="https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.vscode-remote-extensionpack">Remote Development</a>
extension pack from Microsoft. This will allow you to treat a running container, ssh connection
to a remote machine, or Linux environment via WSL (assuming you are a Windows person)
as though it's a local folder. If you've never played with this before, it really is
worth checking out as it's amazing.</p>
<p>I'm going to assume you also have <a href="https://www.docker.com/">Docker</a> installed. If not,
follow the instructions on their site, or any one of the 59 million or so
<a href="https://www.google.com/search?q=how+to+install+docker">tutorials online</a>.</p>
<h2>Create the Dockerfile</h2>
<p>Ok, now let's create a <code>Dockerfile</code> that has what we need. The VS Code tutorial mentions
you need the following:</p>
<ul>
<li>Node.js</li>
<li>Git</li>
<li>Yeoman</li>
<li>The VS Code Extension Generator</li>
</ul>
<p>Ok, all those things boil down to some basic terminal commands, but as a simple starting
point, let's use the <a href="https://hub.docker.com/_/node">official Node.js Docker image</a> to
get Node.js. To do this, make <code>node:10-alpine</code> the base image, and then install those
dependencies:</p>
<div class="highlight"><pre><span></span><span class="k">FROM</span> <span class="s">node:10-alpine</span>
<span class="k">RUN</span> apk add --update git bash
<span class="k">RUN</span> npm install -g yo generator-code
<span class="k">USER</span><span class="s"> node</span>
<span class="k">ENTRYPOINT</span> /bin/bash
<span class="k">WORKDIR</span><span class="s"> /home/node</span>
</pre></div>
<p>Breaking this down a bit:</p>
<ul>
<li>the <code>FROM</code> line says the image to use as a base image is the <code>node:10-alpine</code>
image from Dockerhub. This gives you npm & whatnot already installed.</li>
<li>the <code>apk add</code> line installs Git and Bash (alpine doesn't have Bash installed
by default)</li>
<li>the <code>npm install -g</code> line installs Yeoman and the VS Code Extension Generator</li>
<li>the <code>USER</code> line creates a user called <code>node</code>, which you need as otherwise <code>yo</code>
fails when you run it in the container due to permission issues (doesn't seem
to like running as root)</li>
<li>the <code>ENTRYPOINT</code> says when a container is started from this image, start off
by running <code>/bin/bash</code></li>
<li>the <code>WORKDIR</code> says when a container is started from this image, start in the
<code>/home/node</code> directory</li>
</ul>
<p>Save this file as <code>Dockerfile</code>. Note that I used <code>node:10-alpine</code> (so a 10.x
version of Node), feel free to replace with a newer version if you want (I
have no idea what version the VS Code Extension Generator wants).</p>
<h2>Build The Docker Image</h2>
<p>Now you want to build the Docker image, run this command in the same directory as the <code>Dockerfile</code>:</p>
<div class="highlight"><pre><span></span>docker build -t vscodeenv:latest .
</pre></div>
<p>The <code>vscodeenv</code> name is totally arbitrary, feel free to name it whatever you
want, but that's the name I'll use for this blog post. You'll see a bunch of
output, and after it's done, you should be able to see the built image when you
do a <code>docker images</code>:</p>
<div class="highlight"><pre><span></span>$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
vscodeenv latest 37d9e66fffbc <span class="m">48</span> minutes ago 186MB
</pre></div>
<h2>Run the Image</h2>
<p>Now we have a built Docker image with all the tooling you need for the tutorial.
Next step is to spin up a container from this image. I'm sure if you get into
the VS Code <code>devcontainer.json</code> stuff you could do this from within VS Code, but
I just do it from the terminal:</p>
<div class="highlight"><pre><span></span>$ docker run -it --name vscodeenv -v /Users/aparkin/dockerpath/:/home/node --rm vscodeenv:latest
bash-5.0$
</pre></div>
<p>Replace <code>/Users/aparkin/dockerpath/</code> with the full path to a directory where you want to
put your extensions code (it's perfectly fine to be the same directory where you put the
<code>Dockerfile</code>).</p>
<p>Note that this just drops you into a bash shell inside the running container.
Leave this window open, so long as this is open your container will be running.
Once you type "exit" here, the container will be terminated (which we don't
want to do until we're done working on our little extension).</p>
<p>To break the <code>docker run</code> command down a bit, the key bits:</p>
<ul>
<li><code>-it</code> means run in interactive mode (ie where you can type in commands)</li>
<li><code>--name vscodeenv</code> gives the container the name <code>vscodeenv</code> (again, this is
arbitrary)</li>
<li><code>-v .....</code> tells it to mount the path you specified on your local host
filesystem into <code>/home/node</code> in the running container (so any files in the
path you specify will show up in the container inside <code>/home/node</code>)</li>
<li><code>--rm</code> tells Docker to delete the container once you exit</li>
</ul>
<h2>Create The Skeleton Extension</h2>
<p>The VS Code tutorial indicates after installing everything, you should run the generator.
Do this in the running docker container with the <code>yo code</code> command:</p>
<div class="highlight"><pre><span></span>bash-5.0$ yo code
_-----_ ╭──────────────────────────╮
<span class="p">|</span> <span class="p">|</span> │ Welcome to the Visual │
<span class="p">|</span>--<span class="o">(</span>o<span class="o">)</span>--<span class="p">|</span> │ Studio Code Extension │
<span class="sb">`</span>---------´ │ generator! │
<span class="o">(</span> _´U<span class="sb">`</span>_ <span class="o">)</span> ╰──────────────────────────╯
/___A___<span class="se">\ </span> /
<span class="p">|</span> ~ <span class="p">|</span>
__<span class="s1">'.___.'</span>__
´ <span class="sb">`</span> <span class="p">|</span>° ´ Y <span class="sb">`</span>
? What <span class="nb">type</span> of extension <span class="k">do</span> you want to create? <span class="o">(</span>Use arrow keys<span class="o">)</span>
❯ New Extension <span class="o">(</span>TypeScript<span class="o">)</span>
New Extension <span class="o">(</span>JavaScript<span class="o">)</span>
New Color Theme
New Language Support
New Code Snippets
New Keymap
New Extension Pack
New Language Pack <span class="o">(</span>Localization<span class="o">)</span>
</pre></div>
<p>This generator walks you through creating your first extension. Following VS Code's
tutorial I picked:</p>
<ul>
<li>New Extension (TypeScript)</li>
<li>Name of my extension: "AdamsExtension"</li>
<li>Identifier of my extension: "adamsextension" (the default)</li>
<li>Description I entered random gibberish</li>
<li>Initialize a Git repo: yes</li>
<li>Bundle with Webpack: no</li>
<li>Package manager: npm</li>
</ul>
<p>After that, it goes ahead and installs all the various npm dependencies -- <em>all within
the docker container</em>.</p>
<h2>Attach To the Container</h2>
<p>Now in VS Code open the command palette and search for
<code>Remote-Containers: Attach to Running Container...</code>. Pick this, and then your running
container called <code>vscodeenv</code> should appear in the list:</p>
<p><img alt="Attaching to the running image" src="https://www.codependentcodr.com/static/imgs/runningContainer.png"></p>
<p>Pick it, and VS Code will open a new Window "attached" to the running container. For
more details, consult
<a href="https://code.visualstudio.com/docs/remote/attach-container">the official docs</a>.</p>
<p>Now click on "Open Folder" and navigate to your <code>adamsextension</code> (or whatever you called
your extension) folder and click OK. You then get a VS Code window "attached" to
the running docker container, with your test extension open and ready to play with.
Here's a screenshot to give an idea:</p>
<p><img alt="Extension open in VS Code" src="https://www.codependentcodr.com/static/imgs/attachedToContainer.png"></p>
<p>Now you can hit <code>F5</code> and VS Code will open up a new Extension Development Host window with
your test extension loaded. In that window you should be able to search for the
"Hello World" command in the command palette and run the command.</p>
<h2>Profit</h2>
<p>And that's it, no Node.js or any other tooling installed to your local machine other
than VS Code & Docker. Once you're done playing around, exit out of the running Docker
container (enter "exit" in the bash prompt) and the container will be terminated, and
all the files you created will remain in the path you mounted into the container.</p>
<p>If you want to later pick up where you left off, just run the same <code>docker run</code> command
as before, re-attach to the container, and re-open the folder.</p>Git Image Diff with iTerm22021-02-14T12:37:00-08:002021-02-14T12:37:00-08:00Adam Parkintag:www.codependentcodr.com,2021-02-14:/git-image-diff-with-iterm2.html<p>iTerm has a imgcat function. Lets use it with Git for image diffing</p><p>So iTerm2 has this neat <code>imgcat</code> command that allows you to "cat" or
view an image right in the terminal. It's pretty sweet, but I had an
idea: what if we used that with Git for diffing changed images in a
<code>git diff</code>?</p>
<p>This seems like it should be possible, you can replace what git does
when you do a diff of two different binary files. For example, the
<a href="https://zachholman.com/posts/command-line-image-diffs/">spaceman-diff package by Zach Holman</a>
does this to do a diff of two images as ascii art.</p>
<p>We want to do the same, but instead of seeing an ASCII art version
of an image, it'd be cool to see the two versions of the image itself
right in the terminal.</p>
<h2>Setting Git Attributes</h2>
<p>The first thing is create a file in your home directory called
<code>~/.config/git/attributes</code>. In here you can define a mapping between file types
and what command Git will run when doing a diff of those filetypes. In my case
I entered the following:</p>
<div class="highlight"><pre><span></span>*.png diff=image
*.jpg diff=image
*.gif diff=image
*.jpeg diff=image
</pre></div>
<p>This tells git that when doing a diff of a file that ends with <code>png</code>, <code>jpg</code>,
<code>gif</code>, or <code>jpeg</code> to use the <code>image</code> config.</p>
<h2>Set Up The Image Config</h2>
<p>Now, in your <code>~/.gitconfig</code> file, add the following:</p>
<div class="highlight"><pre><span></span><span class="k">[diff "image"]</span>
<span class="na">textconv</span> <span class="o">=</span> <span class="s">imgcat</span>
</pre></div>
<p>This tells Git to use iTerm's <code>imgcat</code> script for converting the binary image
file to text. This seems weird, but this is actually how iTerms imcat command
works: it converts the binary image into a textual stream that iTerm knows how
to understand and then render as an actual image.</p>
<h2>Put imgcat In Your Path</h2>
<p>Next is put iTerm's <code>imcat</code> script somewhere on your path. You can download it
from <a href="https://iterm2.com/utilities/imgcat">here</a>. Save that somewhere, make sure
you chmod it to be executable (<code>chmod +x imgcat</code>), and then throw it into some
directory on your path. You can confirm it's there by typing <code>imgcat</code> into an
iTerm window:</p>
<div class="highlight"><pre><span></span>$ imgcat
Usage: imgcat <span class="o">[</span>-p<span class="o">]</span> filename ...
or: cat filename <span class="p">|</span> imgcat
</pre></div>
<p>Note that you have to put this script in your path, you can't just rely on the
version that's inlined into your terminal with iTerm's shell integration.</p>
<h2>Profit</h2>
<p>Now when you change an image in a Git repo and do a <code>git diff</code> while in iTerm
you'll see a preview of the original image and the changed image. Example:</p>
<p><img alt="Showing the image diffing" src="https://www.codependentcodr.com/static/imgs/imgcatDiff.png"></p>
<p>The above image is the original, the 2nd image is what I changed it to. Note
that because imgcat is iTerm specific, it won't work in other terminals. If
you do a <code>git diff</code> in a different terminal (ex: the integrated terminal in
VS Code) you'll see just the ordinary blank output:</p>
<p><img alt="Image diff in non-iTerm terminal" src="https://www.codependentcodr.com/static/imgs/imgcatdiffvscode.png"></p>Fixing a Slow Bash Prompt2021-02-01T16:49:00-08:002021-02-01T16:49:00-08:00Adam Parkintag:www.codependentcodr.com,2021-02-01:/fixing-a-slow-bash-prompt.html<p>My bash prompt seemed to be slow. Here's how I fixed it</p><p>I recently got a new machine for my new job, and so I'm going through all the
"new machine setup" gotchas. Since getting it, I've been finding the
shell/terminal has felt really sluggish. Like I’d hit enter in a terminal and
it’d be 1-2 seconds before the prompt came back allowing me to type anything
again, which makes someone who likes to type a lot of commands in quick
succession get angry. 😠</p>
<p>After a bunch of troubleshooting & reverse engineering I found it came down to
the <code>__git_ps1</code> function that displays your current branch in your bash prompt.
How slow was it? Really slow:</p>
<div class="highlight"><pre><span></span> λ <span class="nb">time</span> __git_ps1
<span class="o">(</span>mainline<span class="o">)</span>
real 0m1.746s
user 0m0.044s
sys 0m0.079s
</pre></div>
<p>That’s 1.7 seconds every time I hit enter in a terminal because it gets
evaluated as part of producing my bash prompt.</p>
<p>I thought this was weird, so I did a <code>git --version</code> and noticed that I was
running <code>git version 2.24.3 (Apple Git-128)</code>. That is the special “Apple has
built this version for you and installed as part of Xcode command-line tools”
version.</p>
<p>So, I installed git from Homebrew (<code>brew install git</code>), which gave be the same
version that anyone on any platform would get (just compiled for OSX),
and also gave me a much newer version (2.30.0). Much faster. How much faster?
Check it out:</p>
<div class="highlight"><pre><span></span> λ <span class="nb">time</span> __git_ps1
<span class="o">(</span>mainline<span class="o">)</span>
real 0m0.060s
user 0m0.016s
sys 0m0.026s
</pre></div>
<p>From 1.7 seconds to 0.06 seconds. Wow. Now my command prompt is nice and
snappy.</p>iTerm2 Setup2021-01-30T13:24:00-08:002021-01-30T13:24:00-08:00Adam Parkintag:www.codependentcodr.com,2021-01-30:/iterm2-setup.html<p>I've been a longtime user of iTerm2 as my terminal. This post walks through some of my setup with it.</p><p>I first was introduced to <a href="https://iterm2.com/">iTerm2</a> back around 2012, which
was not coincidentally when I made the switch from a machine running Windows &
Linux (dual-booting) to OSX. I didn't have a choice, the job I was working at
at the time decided all devs would get shiny new Macbook Air's, and overnight I
was thrust into the world of OSX. My MBA came with some software preinstalled
by the ops team, and one of those pieces of software was iTerm2.</p>
<p>If you're an OSX user and you've never used iTerm2, you really should check it
out. At this point it might've been superceded by other fancy terminal
emulators, but it's still really, really good. Essentially iTerm2 is a
replacement for the built in OSX Terminal app, and comes with a <em>ton</em> of extra
features. In this post I'll outline some of the things I've done to customize
iTerm2 on my dev box (this is a bit of a selfish post -- I recently changed jobs
so had to go through setting it up from scratch again so this post is both "hey
these are some of the neat things you can do with it" as well as "I'm
documenting this so next time I don't have to figure it out from scratch"). 😄</p>
<h2>Keyboard Navigation</h2>
<p>So the default keystrokes in iTerm are weird. I like to be able to press
<code>⌘+left arrow</code> to go to the start of a line and <code>⌘+right arrow</code> to go to the
end of the line. Additionally I like be able to move left/right by word by
pressing <code>⌥+left arrow</code>/<code>⌥+right arrow</code>. To set these up in iTerm's
preferences go to Profiles -> Keys, and do the following:</p>
<ul>
<li>change "Left Option (⌥) Key" to "Esc+"</li>
<li>In "Key Mappings" find the shortcut for <code>⌥ + ←</code> and change the action to "Send
Escape Sequence" with the value "b"</li>
<li>Do the same for <code>⌥ + →</code> and use value "f"</li>
<li>Again in "Key Mappings" find the shortcut for <code>⌘ + ←</code> and change it to "Send
Hex Code" with value "0x01"</li>
<li>Do the same for <code>⌘ + →</code> and use value "0x05"</li>
</ul>
<p>Screenshots showing the settings:</p>
<p><img alt="Keyboard Option Key Settings" src="https://www.codependentcodr.com/static/imgs/keymapping1.png"></p>
<p><img alt="Keyboard Shortcut dialog for moving by word" src="https://www.codependentcodr.com/static/imgs/keymapping.png"></p>
<p><img alt="Keyboard Shortcut dialog for moving to start/end of line" src="https://www.codependentcodr.com/static/imgs/keymapping2.png"></p>
<h2>Shell Integration</h2>
<p>iTerm also as a special shell integration thing that gives you a number of extra
commands, and allows you do do some neat tricks. Full details on how to install
& some of the neat things you can do with it are on the iTerm2 website:
<a href="https://iterm2.com/documentation-shell-integration.html">https://iterm2.com/documentation-shell-integration.html</a></p>
<p>Some of the things I use that require shell integration are outlined below.</p>
<h3>Python Version Badge</h3>
<p>Under Profile -> General, there's a "Badge" item. This allows you to display a
piece of text at all times in the top right corner of the terminal window. It's
handy for displaying things like what branch you're currently on or similar
"contextual" items. For me because I do a lot of Python work and I'm constantly
switching between various Python virtual environments I have my badge display
the current Python version that's active. To set this up, I have the following
in my <code>.bashrc</code> file:</p>
<div class="highlight"><pre><span></span><span class="k">function</span> iterm2_print_user_vars<span class="o">()</span> <span class="o">{</span>
iterm2_set_user_var gitBranch <span class="k">$((</span>git branch <span class="m">2</span>> <span class="o">/</span>dev/null<span class="o">)</span> <span class="o">|</span> grep <span class="se">\*</span> <span class="o">|</span> cut <span class="o">-</span>c3-<span class="o">)</span><span class="p">;</span>
iterm2_set_user_var pythonVersion <span class="k">$(</span>python -c <span class="s2">"import platform; print(platform.python_version())"</span><span class="k">)</span><span class="p">;</span>
<span class="o">}</span>
</pre></div>
<p>What these lines do is define two user variables that can be displayed in various
spots in iTerm. My badge setting has the value <code>\(user.pythonVersion)</code> as you can
see here:</p>
<p><img alt="Badge Setting" src="https://www.codependentcodr.com/static/imgs/itermbadge.png"></p>
<p>And then the version displays in the terminal (it's the 3.9.1 in the corner):</p>
<p><img alt="Python Version" src="https://www.codependentcodr.com/static/imgs/pythonVersion.png"></p>
<p>As I switch to different Python interpreters this value updates.</p>
<h3>Marks</h3>
<p>With shell integration on, you also get marks which the docs define as:</p>
<blockquote>
<p>These are saved locations in history. They make it easy to navigate to
previous shell prompts or other locations of interest.</p>
</blockquote>
<p>Marks are the little blue triangle icon at the start of every line in the
terminal. With them on you can press <code>⇧ + ⌘ + E</code> (or go to View -> Show
Timestamps) to see a time history of when each command was run in your terminal,
which can be handy.</p>
<p>You can also turn on an alert on the next mark, which is useful for when you
have a long running command. If you start a long-running command and hit
<code>⌥ + ⌘ + A</code> (or go Edit -> Alerts -> Alert on Next Mark) then once the command
completes you'll get a nice toast notification using OSX's notification system.</p>
<h3>Extra Commands</h3>
<p>Shell integration also gives a few handy shell commands, the one I use fairly
often is <code>imgcat</code> which allows you to view an image right in the terminal window</p>
<p><img alt="Viewing an Image in the Terminal" src="https://www.codependentcodr.com/static/imgs/imgcat.png"></p>
<h2>Status Bar</h2>
<p>The iTerm2 status bar is also fully configurable. You can display various items
like CPU load, or what your current directory is, etc. Mine displays a handful of
things:</p>
<p><img alt="My iTerm2 Status Bar" src="https://www.codependentcodr.com/static/imgs/statusbar.png"></p>
<p>From a glance this shows me that my current battery is 72% charged, CPU is at
18%, memory utilization is at 22GB, that it's currently January 31st at 10:59AM,
that I'm in the <code>~/temp/sandbox/www.codependentcodr.com/content/static/imgs</code>
directory, that my current Git branch is <code>iterm2setup</code> (and that there are
uncommitted changes), and that my current Python version is 3.9.1 (which is
probably redundant since that's in my badge, but 🤷).</p>
<p>You can configure your status bar by going to Profiles -> Session and clicking
"Configure Status Bar". Here's a screenshot showing my settings:</p>
<p><img alt="My iTerm2 Status Bar Settings" src="https://www.codependentcodr.com/static/imgs/statusbarsettings.png"></p>
<p>I also configure the status bar to be at the bottom of the window (do this under
Appearance -> General -> Status Bar Location)</p>
<h2>Miscellaneous</h2>
<p>Various other settings I turn on, mostly for minor conveniences or visual appeal.</p>
<h3>Visual Stuff</h3>
<p>I turn on transparency (Profiles -> Window -> Transparency). This is a setting
I continually fiddle with. 😛</p>
<p><img alt="Transparency settings" src="https://www.codependentcodr.com/static/imgs/transparency.png"></p>
<p>I also set these settings under Appearance -> Windows:</p>
<p><img alt="Transparency settings" src="https://www.codependentcodr.com/static/imgs/appearancewindows.png"></p>
<p>I like the border around the windows, and find the window number in title bar
superfluous.</p>
<p>Under Appearance -> Tabs, I have:</p>
<p><img alt="Transparency settings" src="https://www.codependentcodr.com/static/imgs/appearancetabs.png"></p>
<h3>Oddball Conveniences</h3>
<p>Under General -> Services, I have "Check for Updates Automatically". I used to also
opt into the Beta test releases, but found the update frequency annoying, so now I
opt out of beta releases.</p>
<p>Under General -> Selection, I have the following:</p>
<p><img alt="Selection settings" src="https://www.codependentcodr.com/static/imgs/generalselection.png"></p>
<p>The big one on there for me is that I find the "Copy to pasteboard on selection"
feature very annoying, so I turn it off.</p>
<p>I also under Profiles -> General -> Working Directory, have it set to "Reuse
Previous session's directory". This means that whenever I open a new tab or
window it opens up a terminal in the same directory I was previously in, which I
find matches my workflow the best.</p>
<p>One last thing: I have a global hotkey set up to open/close iTerm2. This can be
set under Keys -> Hotkey -> Show/hide all windows with a system-wide hotkey.
The key I use is <code>⌃ + tilde</code>, which if you're a long-time gamer you'll recognize
is basically the key combination for displaying/hiding the
<a href="https://quake.fandom.com/wiki/Console_Commands_(Q1)">console in Quake</a>.</p>
<h2>And That's It</h2>
<p>Hopefully some of this might be useful for others. Even with all these tweaks
and settings I barely scratch the surface of what's possible with iTerm2. At a
previous job we actually used integrated Python scripts for being able to do
stuff like quickly rebuild & deploy a service on a remote server by picking a
menu option in iTerm2 itself. Super handy stuff.</p>More Charts in the Django Admin2020-11-09T18:31:00-08:002020-11-09T18:31:00-08:00Adam Parkintag:www.codependentcodr.com,2020-11-09:/more-charts-in-the-django-admin.html<p>Walking through an example of adding a chart to the Django admin, and making it use the selected filters.</p><p>So in my day job I do a decent amount of work in Django, and have recently been
getting into customizing the Django admin page. If you've never done much with
the Django admin, <a href="https://docs.djangoproject.com/en/3.1/ref/contrib/admin/">the
docs</a> are definitely
worth checking out, and once you've gone through that the <a href="https://books.agiliq.com/projects/django-admin-cookbook/en/latest/">Django Admin
Cookbook</a> is
also an excellent resource for tips on ways to get even more out of the Django
admin site. Having said that, I wanted to get some pretty graphs like all the
cool kids in Javascript-land are doing.</p>
<p>I stumbled across <a href="https://findwork.dev/blog/adding-charts-to-django-admin/">this awesome blog
post</a> by Dani Hodovic
that is a great example of how to use <a href="https://www.chartjs.org/">Chart.js</a> to
throw a basic bar chart into the Django admin. There's one problem though: the
chart doesn't respect any filters you have in the Django admin, instead only
graphing the entire dataset. What I want is if you (as an example) had a
<code>date_hierarchy</code> set up on your <code>ModelAdmin</code>, then as you pick date filters the
graph re-renders to only graph the filtered data. Lets see how we might do
that. As a starting point, I'm going with the code from that blog post, which
is on Github at: <a href="https://github.com/danihodovic/django_admin_chart_js">https://github.com/danihodovic/django_admin_chart_js</a>, specifically
I'm going to be working from the current <code>master</code> which (at the time of writing)
is SHA <code>d8fc3a68ad68f3bea139b0a4c888f006ef97c286</code>.</p>
<h2>Setting Up the Filter</h2>
<p>First off, we need some filters on the <code>EmailSubscribers</code> model, so lets add
that:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">EmailSubscriberAdmin</span><span class="p">(</span><span class="n">admin</span><span class="o">.</span><span class="n">ModelAdmin</span><span class="p">):</span>
<span class="n">date_hierarchy</span> <span class="o">=</span> <span class="s2">"created_at"</span>
<span class="o">...</span> <span class="n">rest</span> <span class="n">of</span> <span class="n">the</span> <span class="n">code</span> <span class="k">as</span> <span class="n">it</span> <span class="n">exists</span> <span class="o">...</span>
</pre></div>
<p>If you've never done anything with the Django admin before, this might seem like
magic, 1-line to get a nice date filter in place:</p>
<p><img alt="Date Filters in the Django Admin" src="https://www.codependentcodr.com/static/imgs/date_heir_filter.png"></p>
<p>As you click those filters, the changeset view will adjust to only those records
which meet that criteria. All this for 1-line of code. Yup, Django's pretty
sweet.</p>
<h2>Getting the Filters</h2>
<p>So first problem with adjusting the graph to only those rows that are selected
is to figure out how to get what filters are currently in place. As it turns
out, you can get this from the result of the <code>changelist_view()</code> function in
your <code>ModelAdmin</code> via the <code>context_data</code> mapping:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">changelist_view</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">request</span><span class="p">,</span> <span class="n">extra_context</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="n">response</span> <span class="o">=</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">changelist_view</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="n">extra_context</span><span class="o">=</span><span class="n">extra_context</span><span class="p">)</span>
<span class="n">queryset</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">context_data</span><span class="p">[</span><span class="s2">"cl"</span><span class="p">]</span><span class="o">.</span><span class="n">queryset</span>
</pre></div>
<p>At this point <code>queryset</code> is a regular Django QuerySet object with the filters
that have been applied in the current changelist view. This is exactly what we
need, as the <code>chart_data</code> function from original blog post just does a few
transformations (annotations) on a QuerySet object that was just all
<code>EmailSubscriber</code> objects. The original function looked like:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">chart_data</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="p">(</span>
<span class="n">EmailSubscriber</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">annotate</span><span class="p">(</span><span class="n">date</span><span class="o">=</span><span class="n">TruncDay</span><span class="p">(</span><span class="s2">"created_at"</span><span class="p">))</span>
<span class="o">.</span><span class="n">values</span><span class="p">(</span><span class="s2">"date"</span><span class="p">)</span>
<span class="o">.</span><span class="n">annotate</span><span class="p">(</span><span class="n">y</span><span class="o">=</span><span class="n">Count</span><span class="p">(</span><span class="s2">"id"</span><span class="p">))</span>
<span class="o">.</span><span class="n">order_by</span><span class="p">(</span><span class="s2">"-date"</span><span class="p">)</span>
<span class="p">)</span>
</pre></div>
<p>So if we replace <code>EmailSubscriber.objects</code> with the queryset, we get exactly
what we want:</p>
<div class="highlight"><pre><span></span><span class="nd">@admin</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">EmailSubscriber</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">EmailSubscriberAdmin</span><span class="p">(</span><span class="n">admin</span><span class="o">.</span><span class="n">ModelAdmin</span><span class="p">):</span>
<span class="o">...</span> <span class="n">code</span> <span class="n">omitted</span> <span class="k">for</span> <span class="n">brevity</span> <span class="o">...</span>
<span class="k">def</span> <span class="nf">changelist_view</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">request</span><span class="p">,</span> <span class="n">extra_content</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="n">response</span> <span class="o">=</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">changelist_view</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="n">extra_context</span><span class="o">=</span><span class="n">extra_context</span><span class="p">)</span>
<span class="n">queryset</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">context_data</span><span class="p">[</span><span class="s2">"cl"</span><span class="p">]</span><span class="o">.</span><span class="n">queryset</span>
<span class="n">chart_data</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">chart_data</span><span class="p">(</span><span class="n">queryset</span><span class="p">)</span>
<span class="n">as_json</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="nb">list</span><span class="p">(</span><span class="n">chart_data</span><span class="p">),</span> <span class="bp">cls</span><span class="o">=</span><span class="n">DjangoJSONEncoder</span><span class="p">)</span>
<span class="n">response</span><span class="o">.</span><span class="n">context_data</span><span class="o">.</span><span class="n">update</span><span class="p">({</span><span class="s2">"chart_data"</span><span class="p">:</span> <span class="n">as_json</span><span class="p">})</span>
<span class="k">return</span> <span class="n">response</span>
<span class="o">...</span> <span class="n">other</span> <span class="n">functions</span> <span class="n">unchanged</span> <span class="o">...</span>
<span class="k">def</span> <span class="nf">chart_data</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">queryset</span><span class="p">):</span>
<span class="k">return</span> <span class="p">(</span>
<span class="n">queryset</span><span class="o">.</span><span class="n">annotate</span><span class="p">(</span><span class="n">date</span><span class="o">=</span><span class="n">TruncDay</span><span class="p">(</span><span class="s2">"created_at"</span><span class="p">))</span>
<span class="o">.</span><span class="n">values</span><span class="p">(</span><span class="s2">"date"</span><span class="p">)</span>
<span class="o">.</span><span class="n">annotate</span><span class="p">(</span><span class="n">y</span><span class="o">=</span><span class="n">Count</span><span class="p">(</span><span class="s2">"id"</span><span class="p">))</span>
<span class="o">.</span><span class="n">order_by</span><span class="p">(</span><span class="s2">"-date"</span><span class="p">)</span>
<span class="p">)</span>
</pre></div>
<p>And bingo, now as we click some of the date hierarchy filters the graph
re-renders with only the selected rows:</p>
<!-- markdownlint-disable MD033 -->
<video autoplay loop controls>
<source src="/static/vids/DateFilters.mp4" type="video/mp4">
<img src="/static/imgs/DateFilters.gif">
</video>
<!-- markdownlint-enable MD033 -->
<h2>A Wrinkle</h2>
<p>One problem: the "Reload Chart data" button now breaks, because it's still
trying to call <code>chart_data()</code> without supplying a QuerySet object. That's an
easy fix though, just modify the <code>chart_data_endpoint()</code> function to call
<code>chart_data()</code> with the same QuerySet the function used to use inside it --
<code>EmailSubscriber.objects</code>:</p>
<div class="highlight"><pre><span></span> <span class="k">def</span> <span class="nf">chart_data_endpoint</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">request</span><span class="p">):</span>
<span class="n">chart_data</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">chart_data</span><span class="p">(</span><span class="n">EmailSubscriber</span><span class="o">.</span><span class="n">objects</span><span class="p">)</span>
<span class="k">return</span> <span class="n">JsonResponse</span><span class="p">(</span><span class="nb">list</span><span class="p">(</span><span class="n">chart_data</span><span class="p">),</span> <span class="n">safe</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</pre></div>
<p>Now clicking "Reload Chart data" reverts the graph back to showing all records,
leaving the filters in place.</p>Python Tip of the Day - lru_cache2020-05-02T09:48:00-07:002020-05-02T09:48:00-07:00Adam Parkintag:www.codependentcodr.com,2020-05-02:/python-tip-of-the-day-lru_cache.html<p>How to use lru_cache for fun and profit</p><p>So one of the most useful and little known modules in the Python standard
library is the <a href="https://docs.python.org/3/library/functools.html">functools
module</a>. It's full of little
gems that are super useful. I had a use for one of them the other day
(<code>lru_cache</code>) and thought I'd share.</p>
<h2>Setting the Stage</h2>
<p>I was writing a script for work that automated the process of creating a JIRA
ticket for our deploys. While our deployments themselves are mostly automated,
we still require a JIRA ticket get created for auditing & metrics purposes.
This is a very tedious step as it involves doing a diff between two branches to
see what's going to go out in a deploy, tagging the Git SHA with an ID & pushing
that tag to Bitbucket, creating a ticket in JIRA, listing all the items that
were found in the diff, cutting a branch, opening up a PR, etc.</p>
<p>Really boring stuff, and both JIRA & Bitbucket have REST APIs, so can be all
automated. One little wrinkle is the REST API for JIRA combined with the
requirements we had around these deployment tickets meant multiple calls to JIRA
& Bitbucket. And of course those API's require authentication.</p>
<h2>First Stab</h2>
<p>Wrote a few functions for each of the tasks that were part of creating this
deploy ticket. For example, had a function for creating a JIRA ticket, had another
function for finding the diff between two branches, etc, etc. This was fine,
but some of those functions required authentication to speak to the REST API, and I didn't want to
hard-code my JIRA/Bitbucket username & password into my script.</p>
<p>So settled on writing a function to prompt the user for a name & password to
authenticate with:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">jira_rest_api_headers</span><span class="p">():</span>
<span class="n">user</span> <span class="o">=</span> <span class="nb">input</span><span class="p">(</span><span class="s2">"Enter your JIRA username: "</span><span class="p">)</span>
<span class="n">password</span> <span class="o">=</span> <span class="n">getpass</span><span class="p">()</span>
<span class="n">credentials</span> <span class="o">=</span> <span class="n">b64encode</span><span class="p">(</span><span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="n">user</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="n">password</span><span class="si">}</span><span class="s2">"</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s2">"ascii"</span><span class="p">))</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s2">"ascii"</span><span class="p">)</span>
<span class="k">return</span> <span class="p">{</span><span class="s2">"Content-type"</span><span class="p">:</span> <span class="s2">"application/json"</span><span class="p">,</span> <span class="s2">"Authorization"</span><span class="p">:</span> <span class="sa">f</span><span class="s2">"Basic </span><span class="si">{</span><span class="n">credentials</span><span class="si">}</span><span class="s2">"</span><span class="p">}</span>
</pre></div>
<p>This just prompts the user for a username, and their password (using the
<code>getpass</code> function from the <a href="https://docs.python.org/3/library/getpass.html">getpass module in the standard
library</a>), and then returns the
REST API headers with this credential information (using HTTP basic auth).</p>
<p>Now any place in my script where I had to make a call to the JIRA or Bitbucket
REST API I just first made a call to this function to get the info from the
user.</p>
<h2>That Works, But</h2>
<p>This is ok, but again, the script made multiple calls to JIRA & Bitbucket, so
that meant entering your username and password multiple times which was
annoying.</p>
<p>This led me to think "if only there was a way to memoize or cache the result of
that call...." and that led me to <code>lru_cache</code> in the standard library. From <a href="https://docs.python.org/3/library/functools.html#functools.lru_cache">the
docs</a>:</p>
<blockquote>
<p>Decorator to wrap a function with a memoizing callable that saves up to the
maxsize most recent calls. It can save time when an expensive or I/O bound
function is periodically called with the same arguments.</p>
</blockquote>
<p>Perfect, exactly what I want, now I just throw a decorator on the function:</p>
<div class="highlight"><pre><span></span><span class="nd">@lru_cache</span><span class="p">(</span><span class="n">maxsize</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">jira_rest_api_headers</span><span class="p">():</span>
<span class="o">...</span> <span class="n">same</span> <span class="n">body</span> <span class="k">as</span> <span class="n">before</span> <span class="o">...</span>
</pre></div>
<p>And now the first time the function gets called it prompts the user for their
name & password, and every subsequent call to that function it just returns the
same value without prompting the user again.</p>
<p>Beautiful.</p>
<p>This decorator can be used for other things two: think of an expensive function
to calculate some complex value where the function is referentially transparent
(ie has no side effects). Instead of paying the computational cost on every
call you just slap the decorator on it and voila you only pay the cost for the
first call. Super handy stuff.</p>Python manage.py pytest or Making Django Use Pytest2020-04-18T18:51:00-07:002020-04-19T16:54:00-07:00Adam Parkintag:www.codependentcodr.com,2020-04-18:/python-managepy-pytest-or-making-django-use-pytest.html<p>How to make Django's manage.py use Pytest instead of the default test runner</p><p>Edit: For those who prefer visual content, I've recorded this as a video on Youtube
which you can find at: <a href="https://youtu.be/toKW2YLnGFQ">https://youtu.be/toKW2YLnGFQ</a></p>
<p>Ok, so I've had to convert some Django projects to use Pytest as the test runner
rather than the built in one that Django uses. This is actually pretty
straightforward, and I even <a href="https://www.youtube.com/watch?v=7it7JFPInX0">recorded a Youtube
video</a> showing the process.</p>
<p>That's all fine and good, but one of the complaints I've heard from
Django-ista's (is that a term? Djangoites? Django Devotees?) is that it means
now the good old plain <code>python manage.py test</code> no longer works (well, I suppose
technically it still works, but doesn't use Pytest).</p>
<p>So challenge accepted, as one can certainly create <a href="https://docs.djangoproject.com/en/2.2/howto/custom-management-commands/">custom manage.py
commands</a>
in Django, so lets create a custom management command to run our unit tests with
Pytest instead of the default built-in runner.</p>
<h2>Python Manage.py pytest</h2>
<p>So first challenge is "how do we run pytest from Python?" as typically you run
Pytest as a command line tool. As it turns out there's <a href="https://docs.pytest.org/en/latest/usage.html#calling-pytest-from-python-code">docs on how to do this
on Pytest's site</a>.</p>
<p>The trick is to <code>import pytest</code> and then call <code>pytest.main()</code> passing in the
same command line arguments you'd give to Pytest in the terminal to that
function. As an example:</p>
<div class="highlight"><pre><span></span><span class="n">pytest</span><span class="o">.</span><span class="n">main</span><span class="p">([</span><span class="s1">'--lf'</span><span class="p">])</span>
</pre></div>
<p>Would be the same thing as doing <code>pytest --lf</code> on the command-line. Easy peasy.
So started a basic custom management command in a file called <code>pytest.py</code> and
put it into my Django project's <code>management/commands</code> directory.</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">pytest</span>
<span class="kn">from</span> <span class="nn">django.core.management.base</span> <span class="kn">import</span> <span class="n">BaseCommand</span>
<span class="k">class</span> <span class="nc">Command</span><span class="p">(</span><span class="n">BaseCommand</span><span class="p">):</span>
<span class="n">help</span> <span class="o">=</span> <span class="s2">"Runs tests with Pytest"</span>
<span class="k">def</span> <span class="nf">handle</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">options</span><span class="p">):</span>
<span class="n">pytest</span><span class="o">.</span><span class="n">main</span><span class="p">()</span>
</pre></div>
<p>This works, in that now I can do <code>python manage.py pytest</code> and it'll run Pytest
as if I just ran the <code>pytest</code> executable in the current directory.</p>
<p>Cool, but how do I start passing arguments? Typically in a custom Django
management command you define a <code>add_arguments</code> function and use the <code>argparse</code>
module to define the expected arguments for your custom command. In this case
though, I essentially want the interface to Pytest, which would be non-trivial
to recreate by hand (there's a lot of options on that Pytest executable).</p>
<p>But, with argparse, there is a way to essentially say "accept any arguments",
and that's the <code>argparse.REMAINDER</code> value for the <code>nargs</code>
parameter (<a href="https://docs.python.org/3/library/argparse.html#nargs">docs</a>).</p>
<blockquote>
<p><code>argparse.REMAINDER</code>. All the remaining command-line arguments are gathered
into a list. This is commonly useful for command line utilities that dispatch
to other command line utilities</p>
</blockquote>
<p>Perfect, that's exactly what I want. Adding to our management command is straighforward:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">argparse</span>
<span class="kn">import</span> <span class="nn">pytest</span>
<span class="kn">from</span> <span class="nn">django.core.management.base</span> <span class="kn">import</span> <span class="n">BaseCommand</span>
<span class="k">class</span> <span class="nc">Command</span><span class="p">(</span><span class="n">BaseCommand</span><span class="p">):</span>
<span class="n">help</span> <span class="o">=</span> <span class="s2">"Runs tests with Pytest"</span>
<span class="k">def</span> <span class="nf">add_arguments</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">parser</span><span class="p">):</span>
<span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s2">"args"</span><span class="p">,</span> <span class="n">nargs</span><span class="o">=</span><span class="n">argparse</span><span class="o">.</span><span class="n">REMAINDER</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">handle</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">options</span><span class="p">):</span>
<span class="n">pytest</span><span class="o">.</span><span class="n">main</span><span class="p">(</span><span class="nb">list</span><span class="p">(</span><span class="n">args</span><span class="p">))</span>
</pre></div>
<p>Now whatever arguments we pass to our <code>manage.py pytest</code> command will be passed
directly through to the <code>pytest.main</code> function. Exactly what I want, and super
concise.</p>
<h2>But That's Still Different</h2>
<p>At this point it worked, but I could still hear those nagging voices saying,
"yeah but <code>manage.py pytest</code> is not the same as <code>manage.py test</code>". Fine, as it
turns out though you can override any of the built-in manage.py commands.</p>
<p>The trick is to just create a custom management command with the name of the
command you want to override, and make sure your app is the last one in Django's
<code>INSTALLED_APPS</code> setting. So in our case we could just rename our
<code>management/pytest.py</code> file to <code>management/test.py</code> and it'd work. But I kinda
liked having both (ie both <code>manage.py pytest</code> and <code>manage.py test</code> being
effectively an alias to it). So I created a <code>management/test.py</code> file and put
in the following:</p>
<div class="highlight"><pre><span></span><span class="c1"># Override the built in Django manage.py test with pytest</span>
<span class="kn">from</span> <span class="nn">djangotest.management.commands.pytest</span> <span class="kn">import</span> <span class="n">Command</span>
</pre></div>
<p>Yup, that's it, 1 line of code. Now doing a <code>python manage.py test</code> runs Pytest
as the test runner.</p>
<p>Did this all on a test project, source is up on Github at:
<a href="https://github.com/pzelnip/djangotest">https://github.com/pzelnip/djangotest</a></p>
<p>Or if you just want to see the management commands, they're at:
<a href="https://github.com/pzelnip/djangotest/tree/master/djangotest/management/commands">https://github.com/pzelnip/djangotest/tree/master/djangotest/management/commands</a></p>Visual Studio Code Tasks and Split Terminals2019-02-10T09:53:00-08:002019-02-10T09:53:00-08:00Adam Parkintag:www.codependentcodr.com,2019-02-10:/visual-studio-code-tasks-and-split-terminals.html<p>The January update of Visual Studio Code has some useful features for working with tasks.</p><p>So as a big <a href="https://code.visualstudio.com/">Visual Studio Code</a> fan, I've long
made use of the <a href="https://code.visualstudio.com/docs/editor/tasks">tasks feature</a>. The most recent (January 2019, 1.31) update
added a cool new feature related to this that I've been waiting for for some time. I thought I'd do a little
write up about this and how I use tasks with VS Code, particularly as a Pythonista.</p>
<h2>Task Basics</h2>
<p>As a starting point, to give a basic idea of what tasks are, they're effectively little shortcuts to
terminal commands that you can trigger from within VS Code. They're commonly used for things like
triggering build tasks, or starting up a local dev server, etc. The thing that makes them nice is
that they can be triggered from the command pallette much like built-in VS Code commands. For example,
when working on this blog, I'll use a task to fire up a local dev server to test out content before
committing/pushing it. It looks something like this:</p>
<!-- markdownlint-disable MD033 -->
<video autoplay loop controls>
<source src="/static/vids/vscodetask2.mp4" type="video/mp4">
<img src="/static/imgs/vscodetask.gif">
</video>
<!-- markdownlint-enable MD033 -->
<p>At this point I can then go to <a href="http://localhost:8000">http://localhost:8000</a> and see the content I've been working on.
Handy. To create a task, you open up the command pallette and pick "Tasks: Configure Task" and
you'll be prompted with some default template tasks, or the option to "Create tasks.json file
from template" which gives you total control and is the option I use.</p>
<p>A <code>tasks.json</code> file contains a number of JSON blobs which define your tasks. They look something like:</p>
<div class="highlight"><pre><span></span><span class="p">{</span>
<span class="c1">// See https://go.microsoft.com/fwlink/?LinkId=733558</span>
<span class="c1">// for the documentation about the tasks.json format</span>
<span class="s2">"version"</span><span class="o">:</span> <span class="s2">"2.0.0"</span><span class="p">,</span>
<span class="s2">"tasks"</span><span class="o">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="s2">"label"</span><span class="o">:</span> <span class="s2">"Run Server"</span><span class="p">,</span>
<span class="s2">"type"</span><span class="o">:</span> <span class="s2">"shell"</span><span class="p">,</span>
<span class="s2">"command"</span><span class="o">:</span> <span class="s2">"source $(dirname ${config:python.pythonPath})/activate && make devserver"</span>
<span class="p">},</span>
<span class="p">]</span>
<span class="p">}</span>
</pre></div>
<p>This is the definition for my run local dev server that I showed in the video. You can have
as many tasks as you want, the <code>tasks</code> property is just a list of these definitions. The
full list of properties and options are
<a href="https://code.visualstudio.com/docs/editor/tasks#_custom-tasks">in Microsoft's excellent docs</a>.</p>
<h2>Common Python Tasks</h2>
<p>So now that we have an idea of what tasks are, what are some of the neat things you can do
with them, particularly from the perspective of a Python developer? These are some of the
common ones I set up, most of which are <a href="https://www.djangoproject.com/">Django</a> related,
since much of my day job is working in that framework:</p>
<h3>Running a Dev Server</h3>
<div class="highlight"><pre><span></span><span class="p">{</span>
<span class="s2">"label"</span><span class="o">:</span> <span class="s2">"Run Server"</span><span class="p">,</span>
<span class="s2">"type"</span><span class="o">:</span> <span class="s2">"shell"</span><span class="p">,</span>
<span class="s2">"command"</span><span class="o">:</span> <span class="s2">"${config:python.pythonPath} manage.py runserver --noreload"</span><span class="p">,</span>
<span class="p">},</span>
</pre></div>
<p>This is basically the analogy to the task I showed previously, just using Django's
<code>runserver</code> command. One thing to note about this: note that the label is the same
as the one for my blog project. <code>tasks.json</code> files are stored per-project, but one
neat thing is you can assign a hotkey to a given task. In my <code>keybindings.json</code> I
have:</p>
<div class="highlight"><pre><span></span><span class="p">{</span>
<span class="s2">"key"</span><span class="o">:</span> <span class="s2">"cmd+shift+r"</span><span class="p">,</span>
<span class="s2">"command"</span><span class="o">:</span> <span class="s2">"workbench.action.tasks.runTask"</span><span class="p">,</span>
<span class="s2">"args"</span><span class="o">:</span> <span class="s2">"Run Server"</span>
<span class="p">},</span>
</pre></div>
<p>This allows me to start a local dev server by simply hitting a hotkey, and so long
as I name that "start up a local dev environment" task the same on each project,
it's the same keystroke to start up a local dev environment.</p>
<h3>Hitting a Health Check URL</h3>
<div class="highlight"><pre><span></span><span class="p">{</span>
<span class="s2">"label"</span><span class="o">:</span> <span class="s2">"Healthcheck (requires running server)"</span><span class="p">,</span>
<span class="s2">"type"</span><span class="o">:</span> <span class="s2">"shell"</span><span class="p">,</span>
<span class="s2">"command"</span><span class="o">:</span> <span class="s2">"curl http://127.0.0.1:6100/health"</span>
<span class="p">},</span>
</pre></div>
<p>Once I have a local server running, it's handy to be able to quickly hit the
<a href="https://microservices.io/patterns/observability/health-check-api.html">health check</a> url
(you added a health check to your API right?). Again, this is small, but handy as
it saves me the trouble of tabbing over to a terminal window, typing out the
<code>curl</code> command, realizing that this project runs on a different port, going to look
that up, etc, etc, etc.</p>
<h3>Running Unit Tests</h3>
<div class="highlight"><pre><span></span><span class="p">{</span>
<span class="s2">"label"</span><span class="o">:</span> <span class="s2">"Run Unit Tests"</span><span class="p">,</span>
<span class="s2">"type"</span><span class="o">:</span> <span class="s2">"shell"</span><span class="p">,</span>
<span class="s2">"group"</span><span class="o">:</span> <span class="p">{</span>
<span class="s2">"kind"</span><span class="o">:</span> <span class="s2">"test"</span><span class="p">,</span>
<span class="s2">"isDefault"</span><span class="o">:</span> <span class="kc">true</span>
<span class="p">},</span>
<span class="s2">"command"</span><span class="o">:</span> <span class="s2">"${config:python.pythonPath} -m pytest -rxXs --ds=projectname.settings.local_test --random-order"</span>
<span class="p">},</span>
</pre></div>
<p>Whenever possible I use <a href="https://pytest.org/">pytest</a> for running my unit tests. Typically this is
run from the command line as something like <code>pytest <name of directory containing tests></code>. The
problem though is that pytest gets installed to a virtual environment, so how do I give the full
path to the virtual env without making the task machine specific? The answer is I run it as a
module and just use the <code>config:python.pythonPath</code> variable to reference whatever the current
Python environment is. The other options are some common ones I feed to pytest, ex the <code>--ds</code>
switch is for specifying the <code>DJANGO_SETTINGS_MODULE</code> environment variable. <code>--random-order</code>
uses the <a href="https://pypi.org/project/pytest-random-order/">Pytest Random Order plugin</a> to run the
tests in a random order on each test run (which has discovered bugs in my code/tests).</p>
<p>I also set a hotkey for this task:</p>
<div class="highlight"><pre><span></span><span class="p">{</span>
<span class="s2">"key"</span><span class="o">:</span> <span class="s2">"shift+cmd+f11"</span><span class="p">,</span>
<span class="s2">"command"</span><span class="o">:</span> <span class="s2">"workbench.action.tasks.test"</span>
<span class="p">},</span>
</pre></div>
<p>This makes use of the <code>kind</code> property of the task definition.</p>
<h3>Update Dependencies</h3>
<div class="highlight"><pre><span></span><span class="p">{</span>
<span class="s2">"label"</span><span class="o">:</span> <span class="s2">"Update Python Dependencies"</span><span class="p">,</span>
<span class="s2">"type"</span><span class="o">:</span> <span class="s2">"shell"</span><span class="p">,</span>
<span class="s2">"command"</span><span class="o">:</span> <span class="s2">"${config:python.pythonPath} -m pip install -r requirements.txt --upgrade && ${config:python.pythonPath} -m pip install -r requirements-dev.txt --upgrade"</span>
<span class="p">},</span>
</pre></div>
<p>I still use <code>requirements.txt</code> files (I really should spend the time to learn
<a href="https://github.com/pypa/pipenv">pipenv</a>, but alas). With this task I can
quickly update all my project's dependencies. I also separate out my project's
dependencies and my project's dev dependencies (think things like pytest or
pylint) into separate files. The reason for this is that I can then let my dev
dependencies "float", and most projects I work on also build a Docker image at
the end of the day, so separating the dependencies allows me to only install the
dependencies needed for running the project into the Docker image, which cuts
down on image size.</p>
<h3>Many Many More</h3>
<p>This is just scratching the surface, any time I find myself commonly running commands
in a terminal window on a project, I'll spend the minute or so to turn that into a
VS Code task.</p>
<p>Lastly, one of the key points here is that I do essentially these same tasks on
any project I work on and I just tweak the specific commands for the particular
project. This creates a common/familiar workflow for me regardless of if it's a
Django project, Flask, or even an entirely different tech (I had a Java project
with a REST API and I created many of the same tasks for that).</p>
<h2>New Tricks</h2>
<p>As mentioned, in the January 2019 update they added a new feature related to tasks
that I'm a huge fan of:
<a href="https://code.visualstudio.com/updates/v1_31#_task-output-support-split-terminals">Task Output Split Terminals</a></p>
<p>This allows you to have a task spawned into a split terminal window to another (already
running) task. This is really handy when you have say a task for running the dev
server and another task for say tailing the log file of that server as you can have them
appear side-by-side in the integrated terminal.</p>
<p>This was particularly useful for a project I work on at work where I have a Django-based
server, which speaks to another local dev server via a socket connection. Previously
I had tasks set up for both of these, and I'd have to fire up each one individually, and
switch between multiple terminal windows to see the output of each. Now I can have them
show up side by side in one view. The way this works is by sharing the same <code>group</code>
property in the task's <code>presentation</code> property:</p>
<div class="highlight"><pre><span></span><span class="p">{</span>
<span class="s2">"label"</span><span class="o">:</span> <span class="s2">"Run Server"</span><span class="p">,</span>
<span class="s2">"type"</span><span class="o">:</span> <span class="s2">"shell"</span><span class="p">,</span>
<span class="s2">"command"</span><span class="o">:</span> <span class="s2">"${config:python.pythonPath} manage.py runserver --noreload"</span><span class="p">,</span>
<span class="s2">"presentation"</span><span class="o">:</span> <span class="p">{</span>
<span class="s2">"group"</span><span class="o">:</span> <span class="s2">"groupServerStuff"</span>
<span class="p">}</span>
<span class="p">},</span>
</pre></div>
<p>All tasks with the same group will open up as another split terminal pane in the same
terminal window. Very nice.</p>
<p>This got me to thinking though: rather than start each task individually, is there a way
to have tasks "call" or "spawn" other tasks? And as it turns out there is:</p>
<div class="highlight"><pre><span></span><span class="p">{</span>
<span class="s2">"label"</span><span class="o">:</span> <span class="s2">"Run Server"</span><span class="p">,</span>
<span class="s2">"dependsOn"</span><span class="o">:</span> <span class="p">[</span>
<span class="s2">"Run TCP Server"</span><span class="p">,</span>
<span class="s2">"Run Django Server"</span><span class="p">,</span>
<span class="s2">"Tail Log File"</span>
<span class="p">]</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s2">"label"</span><span class="o">:</span> <span class="s2">"Run Django Server"</span><span class="p">,</span>
<span class="s2">"type"</span><span class="o">:</span> <span class="s2">"shell"</span><span class="p">,</span>
<span class="s2">"command"</span><span class="o">:</span> <span class="s2">"${config:python.pythonPath} manage.py runserver --noreload"</span><span class="p">,</span>
<span class="s2">"presentation"</span><span class="o">:</span> <span class="p">{</span>
<span class="s2">"group"</span><span class="o">:</span> <span class="s2">"groupServerStuff"</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s2">"label"</span><span class="o">:</span> <span class="s2">"Run TCP Server"</span><span class="p">,</span>
<span class="s2">"type"</span><span class="o">:</span> <span class="s2">"shell"</span><span class="p">,</span>
<span class="s2">"command"</span><span class="o">:</span> <span class="s2">"${config:python.pythonPath} scripts/tcp_server.py"</span><span class="p">,</span>
<span class="s2">"presentation"</span><span class="o">:</span> <span class="p">{</span>
<span class="s2">"group"</span><span class="o">:</span> <span class="s2">"groupServerStuff"</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s2">"label"</span><span class="o">:</span> <span class="s2">"Tail Log File"</span><span class="p">,</span>
<span class="s2">"type"</span><span class="o">:</span> <span class="s2">"shell"</span><span class="p">,</span>
<span class="s2">"command"</span><span class="o">:</span> <span class="s2">"tail -f /tmp/logfile.txt"</span><span class="p">,</span>
<span class="s2">"presentation"</span><span class="o">:</span> <span class="p">{</span>
<span class="s2">"group"</span><span class="o">:</span> <span class="s2">"groupServerStuff"</span>
<span class="p">}</span>
<span class="p">},</span>
</pre></div>
<p>Check out that <code>Run Server</code> task -- it spawns three other tasks I have defined:
"Run Django Server" (which was my previous "Run Server" task), "Run TCP Server"
(the simulated socket server), and "Tail Log File" which just tails the logfile
that Django is logging to.</p>
<p>And of course, because it's called <code>Run Server</code> the same hotkey I defined
previously will spawn up a new terminal window split 3-ways with these tasks
running. All with a single keystroke. That's pretty powerful stuff!</p>
<p>In any case, I hope this was a useful overview of Tasks in VS Code. Have you
come up with any creative uses for them? Lemme know in the comments!</p>Asyncio Part the third2018-08-13T15:06:00-07:002018-08-13T15:06:00-07:00Adam Parkintag:www.codependentcodr.com,2018-08-13:/asyncio-part-the-third.html<p>What happens when you try to make an <strong>init</strong> async?</p><p>I found this interesting and gave a small bit of insight/help with forming my mental model around
asyncio. Let's say you have a class:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">asyncio</span>
<span class="k">class</span> <span class="nc">Foo</span><span class="p">:</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">loop</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"in foo init"</span><span class="p">)</span>
<span class="n">loop</span><span class="o">.</span><span class="n">run_until_complete</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">consumer</span><span class="p">())</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">consumer</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"in consumer"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">main</span><span class="p">():</span>
<span class="n">loop</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">get_event_loop</span><span class="p">()</span>
<span class="n">f</span> <span class="o">=</span> <span class="n">Foo</span><span class="p">(</span><span class="n">loop</span><span class="p">)</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">"__main__"</span><span class="p">:</span>
<span class="n">main</span><span class="p">()</span>
</pre></div>
<p>This is perfectly fine. Nothing wrong with calling some async code from your (synchronous)
<code>__init__</code> method.</p>
<p>What happens though if you make your <code>__init__</code> async though?</p>
<div class="highlight"><pre><span></span><span class="o">...</span> <span class="n">same</span> <span class="kn">from</span> <span class="nn">before</span> <span class="o">...</span>
<span class="k">class</span> <span class="nc">Foo</span><span class="p">:</span>
<span class="k">async</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">loop</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"in foo init"</span><span class="p">)</span>
<span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">consumer</span><span class="p">()</span>
<span class="o">...</span> <span class="n">rest</span> <span class="n">same</span> <span class="k">as</span> <span class="n">before</span> <span class="o">...</span>
</pre></div>
<p>When you run this, you'll find you get an error:</p>
<div class="highlight"><pre><span></span>Traceback <span class="o">(</span>most recent call last<span class="o">)</span>:
File <span class="s2">"/Users/ad0418340/temp/sandbox/interac/paymentdispatch/test5.py"</span>, line <span class="m">20</span>, <span class="k">in</span> <module>
main<span class="o">()</span>
File <span class="s2">"/Users/ad0418340/temp/sandbox/interac/paymentdispatch/test5.py"</span>, line <span class="m">16</span>, <span class="k">in</span> main
<span class="nv">f</span> <span class="o">=</span> Foo<span class="o">(</span>loop<span class="o">)</span>
TypeError: __init__<span class="o">()</span> should <span class="k">return</span> None, not <span class="s1">'coroutine'</span>
</pre></div>
<p>As it turns out, this is the same exception that gets thrown anytime you <code>return</code> a value
from <code>__init__</code>. For example:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Foo</span><span class="p">:</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="mi">42</span> <span class="c1"># this will blow up with a TypeError</span>
</pre></div>
<p>What can we take from this? A coroutine is a <em>type</em> of value. As soon as you put <code>async</code>
on a function it means that "calling" that function now returns a value of type <code>coroutine</code>.
Since <code>__init__</code> methods can't return values in Python (it'd be nonsensical to do so, think
about how you'd possibly get the value from a return), you can't make your <code>__init__</code> methods
<code>async</code>.</p>Disabling Pylint Messages2018-08-12T15:20:00-07:002018-08-12T15:20:00-07:00Adam Parkintag:www.codependentcodr.com,2018-08-12:/disabling-pylint-messages.html<p>Showing how to disable specific Pylint warnings.</p><p>Small tip of the day since I keep having to look this up. If you use
<a href="https://pylint.org/">Pylint</a> for static analysis of your code, you might find
that you'll want to disable a particular rule for a particular line or file.
That is, you don't want to permanently disable this rule, but just for this one
special spot where the rule doesn't make sense.</p>
<p>I find this particularly common in unit test code as my test method names tend
to be long, violating rule C0103 "Name doesn't conform to naming rules". For
example, a test name might look like:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">test_message_builder_generates_correct_bytestring_when_no_argument_supplied</span><span class="p">():</span>
</pre></div>
<p>Which is very self-descriptive, and when that test fails, it makes it much
easier to figure out what might've gone wrong. The problem is that Pylint will
flag that line since it's greater than 30 characters long, violating the style
guidelines. We could disable this rule across the entire codebase, but outside
of tests, the rule makes sense.</p>
<p>This is where local disables come in, which take the form of comments:</p>
<div class="highlight"><pre><span></span><span class="c1"># pylint disable=C0103</span>
<span class="k">def</span> <span class="nf">test_message_builder_generates_correct_bytestring_when_no_argument_supplied</span><span class="p">():</span>
</pre></div>
<p>This will suppress code C0103 for the remainder of the scope (module or block),
or until it's turned back on:</p>
<div class="highlight"><pre><span></span><span class="c1"># pylint disable=C0103</span>
<span class="k">def</span> <span class="nf">test_message_builder_generates_correct_bytestring_when_no_argument_supplied</span><span class="p">():</span>
<span class="c1"># still turned off here...</span>
<span class="c1"># pylint enable=C0103</span>
<span class="k">def</span> <span class="nf">but_not_turned_off_here_so_this_name_will_get_flagged_by_pylint</span><span class="p">():</span>
</pre></div>
<p>You can also (and it's generally better practice) use the "verbose name" for a
particular code rather than the shorthand code. For example, this is
equivalent:</p>
<div class="highlight"><pre><span></span><span class="c1"># pylint disable=invalid-name</span>
<span class="k">def</span> <span class="nf">test_message_builder_generates_correct_bytestring_when_no_argument_supplied</span><span class="p">():</span>
</pre></div>
<p>A question then becomes "how do I know what the verbose name for a code is?"
And the answer is to use the <code>--list-msgs</code> argument to Pylint on the command
line, and it'll spit them all out:</p>
<div class="highlight"><pre><span></span>$ pylint --list-msgs
:blacklisted-name <span class="o">(</span>C0102<span class="o">)</span>: *Black listed name <span class="s2">"%s"</span>*
Used when the name is listed <span class="k">in</span> the black list <span class="o">(</span>unauthorized names<span class="o">)</span>.
:invalid-name <span class="o">(</span>C0103<span class="o">)</span>: *%s name <span class="s2">"%s"</span> doesn<span class="s1">'t conform to %s*</span>
<span class="s1"> Used when the name doesn'</span>t conform to naming rules associated to its <span class="nb">type</span>
<span class="o">(</span>constant, variable, class...<span class="o">)</span>.
:missing-docstring <span class="o">(</span>C0111<span class="o">)</span>: *Missing %s docstring*
Used when a module, <span class="k">function</span>, class or method has no docstring.Some special
methods like __init__ doesn<span class="err">'</span>t necessary require a docstring.
:empty-docstring <span class="o">(</span>C0112<span class="o">)</span>: *Empty %s docstring*
Used when a module, <span class="k">function</span>, class or method has an empty docstring <span class="o">(</span>it would
be too easy <span class="p">;</span><span class="o">)</span>.
... more lines ...
</pre></div>
<p>One extra tip about Pylint: if you use Visual Studio Code you can turn on Pylint as your linter, and
warnings will get put into the problems view. To turn it on set the following in your VS Code settings
(assuming you've installed the
<a href="https://marketplace.visualstudio.com/items?itemName=ms-python.python">Python extension from Microsoft</a>):</p>
<div class="highlight"><pre><span></span>"python.linting.enabled": true,
"python.linting.pylintEnabled": true,
</pre></div>
<p>Note that these are both on by default (at least in the current version). Once you save
a Python file, Pylint warnings will show up in the problems view:</p>
<p><img alt="Pylint Warnings in VS Code Problems View" src="https://www.codependentcodr.com/static/imgs/pylint_warnings_in_vscode-crunch.png"></p>
<p>Note that that screenshot also illustrates how you can filter the problems view down to show just Pylint
warnings by entering "pylint" into the search box. Also note that clicking any of those will open up
that file in VS Code and navigate to the line in question. Really handy.</p>Docker and Image Sizes2018-08-11T19:30:00-07:002018-08-11T19:30:00-07:00Adam Parkintag:www.codependentcodr.com,2018-08-11:/docker-and-image-sizes.html<p>I was surprised & learned something about building Docker images.</p><p>So this surprised me. I had a Dockerfile that looked basically like this:</p>
<div class="highlight"><pre><span></span><span class="k">FROM</span> <span class="s">alpine:latest</span>
<span class="k">RUN</span> apk add --no-cache --update <span class="se">\</span>
python3 nodejs-current-npm make git curl
<span class="k">RUN</span> python3 -m ensurepip
<span class="k">RUN</span> pip3 install --upgrade pip
<span class="k">RUN</span> npm install -g markdownlint-cli
<span class="c"># needed for one of the packages in requirements.txt</span>
<span class="k">RUN</span> apk add --no-cache --update python3-dev gcc build-base
<span class="k">COPY</span> requirements.txt /build/requirements.txt
<span class="k">RUN</span> pip3 install -r /build/requirements.txt
</pre></div>
<p>Building this image resulted in an image that <code>docker images</code> reported as 379MB in
size. That's a little large so I wanted to trim.</p>
<p>Since those packages installed just before copying
<code>requirements.txt</code> to the image were only there to be able to <em>install</em> a package,
there's no reason for them to remain in the image. Cool, so we can turf them to
save on image size:</p>
<div class="highlight"><pre><span></span><span class="k">FROM</span> <span class="s">alpine:latest</span>
<span class="k">RUN</span> apk add --no-cache --update <span class="se">\</span>
python3 nodejs-current-npm make git curl
<span class="k">RUN</span> python3 -m ensurepip
<span class="k">RUN</span> pip3 install --upgrade pip
<span class="k">RUN</span> npm install -g markdownlint-cli
<span class="c"># needed for one of the packages in requirements.txt</span>
<span class="k">RUN</span> apk add --no-cache --update python3-dev gcc build-base
<span class="k">COPY</span> requirements.txt /build/requirements.txt
<span class="k">RUN</span> pip3 install -r /build/requirements.txt
<span class="c"># cleanup unneeded dependencies</span>
<span class="k">RUN</span> apk del python3-dev gcc build-base
</pre></div>
<p>Sweet, and this resulted in an image size of <em>381MB</em>, a savings of <em>NEGATIVE 2MB</em>.
Wait.... WAT?</p>
<p><img alt="WAT" src="https://www.codependentcodr.com/static/imgs/wat.jpg"></p>
<p>So I <em>removed</em> some stuff and ended up with an image that's a few MB's <em>larger</em>?
How does that work?</p>
<p>And this is where if we want to get technical, we start talking about how Docker
uses a layered filesystem and as such (not entirely unlike Git) once something is
added to an image, it can't really (or at least easily) be removed.</p>
<p>See this issue which mentions what I'm talking about:
<a href="https://github.com/gliderlabs/docker-alpine/issues/45">https://github.com/gliderlabs/docker-alpine/issues/45</a></p>
<p>So what do we do? Well, we <em>combine</em> the operations into a single Docker instruction:</p>
<div class="highlight"><pre><span></span><span class="k">FROM</span> <span class="s">alpine:latest</span>
<span class="k">RUN</span> apk add --no-cache --update <span class="se">\</span>
python3 nodejs-current-npm make git curl
<span class="k">RUN</span> python3 -m ensurepip
<span class="k">RUN</span> pip3 install --upgrade pip
<span class="k">RUN</span> npm install -g markdownlint-cli
<span class="k">COPY</span> requirements.txt /build/requirements.txt
<span class="k">RUN</span> apk add --no-cache --update python3-dev gcc build-base <span class="o">&&</span> <span class="se">\</span>
pip3 install -r /build/requirements.txt <span class="o">&&</span> <span class="se">\</span>
apk del python3-dev gcc build-base
</pre></div>
<p>Because the add & the removal of the apk packages are a single Docker instruction
they don't inflate the size of the built image (you can think of layers as being
"checkpoints" after each instruction in a <code>Dockerfile</code>).</p>
<p>With this change my image size dropped <em>significantly</em>. How much? Let's let the
tool tell us:</p>
<div class="highlight"><pre><span></span>$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
someimage combineops 1744771da3fa About a minute ago 216MB
someimage removepckgs fc5877e2afad <span class="m">4</span> minutes ago 381MB
someimage original b6e5e43b22e0 <span class="m">5</span> minutes ago 379MB
</pre></div>
<p>That is, it dropped from <em>379MB</em> to <em>216MB</em>. Not a bad savings at all.</p>
<p>This is a classic "time vs space" tradeoff though. Because I had to move the <code>requirements.txt</code>
line up, that means that builds of this image are often slower (because of the way the Docker
cache works, if I change the requirements.txt file then it'll have to install those apk packages
any time <code>requirements.txt</code> changes). However, I think the savings in space (40%+) is worth it.</p>Asyncio part 2 - Now More Abstract2018-08-08T11:56:00-07:002018-08-08T11:56:00-07:00Adam Parkintag:www.codependentcodr.com,2018-08-08:/asyncio-part-2-now-more-abstract.html<p>What happens when you mix coroutines with Abstract Base Classes?</p><p>Continuing on with my asyncio learnings. Had the opportunity to look at a wrinkle I haven't seen discussed
much -- using coroutines with <a href="https://docs.python.org/3/library/abc.html">Abstract Base Classes</a>.</p>
<p>First question: can coroutines be abstract? Yes. Yes they can:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">abc</span> <span class="kn">import</span> <span class="n">ABC</span><span class="p">,</span> <span class="n">abstractmethod</span>
<span class="kn">import</span> <span class="nn">asyncio</span>
<span class="k">class</span> <span class="nc">AbstractBase</span><span class="p">(</span><span class="n">ABC</span><span class="p">):</span>
<span class="nd">@abstractmethod</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">meth1</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">pass</span>
<span class="k">class</span> <span class="nc">Concrete</span><span class="p">(</span><span class="n">AbstractBase</span><span class="p">):</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">meth1</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"in concrete"</span><span class="p">)</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">"__main__"</span><span class="p">:</span>
<span class="n">loop</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">get_event_loop</span><span class="p">()</span>
<span class="n">loop</span><span class="o">.</span><span class="n">run_until_complete</span><span class="p">(</span><span class="n">Concrete</span><span class="p">()</span><span class="o">.</span><span class="n">meth1</span><span class="p">())</span> <span class="c1"># prints "in concrete"</span>
</pre></div>
<p>Nothing particularly surprising there.</p>
<p>Next question: is an abstract class still uninstantiable? Yes. Yes it is:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="n">AbstractBase</span><span class="p">()</span>
<span class="n">Traceback</span> <span class="p">(</span><span class="n">most</span> <span class="n">recent</span> <span class="n">call</span> <span class="n">last</span><span class="p">):</span>
<span class="n">File</span> <span class="s2">"<stdin>"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">1</span><span class="p">,</span> <span class="ow">in</span> <span class="o"><</span><span class="n">module</span><span class="o">></span>
<span class="ne">TypeError</span><span class="p">:</span> <span class="n">Can</span><span class="s1">'t instantiate abstract class AbstractBase with abstract methods meth1</span>
</pre></div>
<p>Cool, so far so good.</p>
<p>Ok, next question: what happens if an implementing class doesn't denote an overriding
method as a coroutine? Nothing:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">ConcreteButNotCoroutine</span><span class="p">(</span><span class="n">AbstractBase</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">meth1</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"in concrete but not coroutine"</span><span class="p">)</span>
</pre></div>
<p>However, this is where it gets a bit grey. <code>meth1()</code> on <code>ConcreteButNotCoroutine</code> is
a method, not a coroutine, so it can't be passed to <code>run_until_complete()</code>:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="n">loop</span><span class="o">.</span><span class="n">run_until_complete</span><span class="p">(</span><span class="n">ConcreteButNotCoroutine</span><span class="p">()</span><span class="o">.</span><span class="n">meth1</span><span class="p">())</span>
<span class="ow">in</span> <span class="n">concrete</span> <span class="n">but</span> <span class="ow">not</span> <span class="n">coroutine</span>
<span class="n">Traceback</span> <span class="p">(</span><span class="n">most</span> <span class="n">recent</span> <span class="n">call</span> <span class="n">last</span><span class="p">):</span>
<span class="n">File</span> <span class="s2">"<stdin>"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">1</span><span class="p">,</span> <span class="ow">in</span> <span class="o"><</span><span class="n">module</span><span class="o">></span>
<span class="n">File</span> <span class="s2">"/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">547</span><span class="p">,</span> <span class="ow">in</span> <span class="n">run_until_complete</span>
<span class="n">future</span> <span class="o">=</span> <span class="n">tasks</span><span class="o">.</span><span class="n">ensure_future</span><span class="p">(</span><span class="n">future</span><span class="p">,</span> <span class="n">loop</span><span class="o">=</span><span class="bp">self</span><span class="p">)</span>
<span class="n">File</span> <span class="s2">"/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/tasks.py"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">588</span><span class="p">,</span> <span class="ow">in</span> <span class="n">ensure_future</span>
<span class="k">raise</span> <span class="ne">TypeError</span><span class="p">(</span><span class="s1">'An asyncio.Future, a coroutine or an awaitable is '</span>
<span class="ne">TypeError</span><span class="p">:</span> <span class="n">An</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">Future</span><span class="p">,</span> <span class="n">a</span> <span class="n">coroutine</span> <span class="ow">or</span> <span class="n">an</span> <span class="n">awaitable</span> <span class="ow">is</span> <span class="n">required</span>
</pre></div>
<p>This is kinda icky, now whomever is using a concrete implementation of <code>AbstractBase</code>
has to know if the specific derived class marked <code>meth1()</code> as <code>async</code> or not. This
seems to defeat much of the purpose of using an abstract base class in the first
place as it's kind of a violation of the <a href="https://en.wikipedia.org/wiki/Liskov_substitution_principle">Liskov Substitution Principle</a>.
The LSP is typically stated as (this is taken from the Wikipedia page):</p>
<blockquote>
<p>if S is a subtype of T, then objects of type T may be replaced with objects of
type S (i.e. an object of type T may be substituted with any object of a
subtype S) without altering any of the desirable properties of the program
(correctness, task performed, etc.)</p>
</blockquote>
<p>That is, you should be able to use a <code>ConcreteButNotCoroutine</code> instance anywhere you
can use a <code>Concrete</code> instance without having things go boom boom.</p>
<p>Python's long had a history of being a little lax with the typing (some would call
that a feature, some would call it a shortcoming), but this feels a bit unfortunate.</p>
<p>I did some Googling and someone asked <a href="https://stackoverflow.com/questions/47555934/how-require-that-an-abstract-method-is-a-coroutine">this question on Stackoverflow</a>
which provides a workaround to check if an implementing class overrides an abstract coroutine
with a coroutine, but still feels rather cumbersome. It also still results in a runtime error,
just at object instantiation time instead of at method call time.</p>Code Quality Challenge #2 - Reducing Build Times2018-08-07T12:51:00-07:002018-08-07T12:51:00-07:00Adam Parkintag:www.codependentcodr.com,2018-08-07:/code-quality-challenge-2-reducing-build-times.html<p>CQC #2 - Reducing Build Times</p><h2>The Challenge</h2>
<p>Today's challenge is to look at your project's build time. How long does it
take to go from code committed to Github or Bitbucket to a built artifact you
know is "green"? Can you reduce that time while still maintaining the same
level of quality?</p>
<p>Some ideas to look into:</p>
<ul>
<li>Is there a small handful of tests that take the majority of the time? Could they be removed or sped up?</li>
<li>Could you leverage a tool like Docker to build intermediate images that could save build times?</li>
<li>Could you parallelize some of your build steps?</li>
<li>Do you retrieve your 3rd party dependencies over the Internet on each build?<ul>
<li>Could you take advantage of some sort of local caching?</li>
</ul>
</li>
</ul>
<p>On the note of local caching: tools like
<a href="https://jfrog.com/artifactory/">Artifactory</a> or <a href="https://devpi.net/docs/devpi/devpi/latest/+d/quickstart-pypimirror.html">Devpi for
Python</a>
can help with this)</p>
<p>Give it a go, spend 20 minutes seeing if you can reduce your build times.</p>
<h2>What I Did</h2>
<p>On the a project at work, we have a number of steps in our "test" stage of our <a href="https://jenkins.io/">Jenkins</a> build.
Namely we:</p>
<ul>
<li>Run the unit tests</li>
<li>Run <a href="https://pylint.org/">Pylint</a> over the code</li>
<li>Run <a href="https://github.com/pyupio/safety">Safety</a> over the code</li>
<li>Run a check for missing database migrations</li>
</ul>
<p>All of these could be run in parallel. Having said that, <a href="https://pylint.org/">Pylint</a> takes about as
long as the other steps combined, so it's the bottleneck.</p>
<p>I tried refactoring our <code>Jenkinsfile</code> to do everything except Pylint in one
stream, and Pylint in a parallel stream. Unfortunately I wasn't able to get it working in the
20 minutes, so this ended up just being some good practice working and learning about
<code>Jenkinsfile</code>s.</p>
<p>But, some resources I found on parallelizing Jenkins builds:</p>
<ul>
<li><a href="https://jenkins.io/doc/book/pipeline/jenkinsfile/#parallel-execution">https://jenkins.io/doc/book/pipeline/jenkinsfile/#parallel-execution</a></li>
<li><a href="https://stackoverflow.com/questions/46834998/scripted-jenkinsfile-parallel-stage">https://stackoverflow.com/questions/46834998/scripted-jenkinsfile-parallel-stage</a></li>
<li><a href="https://stackoverflow.com/questions/36872657/running-stages-in-parallel-with-jenkins-workflow-pipeline">https://stackoverflow.com/questions/36872657/running-stages-in-parallel-with-jenkins-workflow-pipeline</a></li>
<li><a href="https://www.google.ca/search?q=jenkinsfile+parallel+stages+scripted">https://www.google.ca/search?q=jenkinsfile+parallel+stages+scripted</a></li>
</ul>
<h2>What About You</h2>
<p>Did you try the challenge? How'd it go? Would love to hear any feedback,
comments, or observations. And if you have ideas for future challenges, please
feel free to suggest them!</p>Asyncio, You are a complex beast...2018-08-05T10:00:00-07:002018-08-05T10:30:00-07:00Adam Parkintag:www.codependentcodr.com,2018-08-05:/asyncio-you-are-a-complex-beast.html<p>Learning asyncio</p><p>Recently I've been digging into Python's <code>asyncio</code> library which was introduced
as part of Python 3.4, and iterated upon in each release since. My reasons for
learning about it are work-related, but I thought it'd be interesting/insightful
to share things I've learned here, partly to share with the world (as the docs
for <code>asyncio</code> are rather...terse) as well as reinforce what I've managed to
figure out.</p>
<p>This is likely going to be long, as there's a lot to learn & know about
<code>asyncio</code>.</p>
<p>Some context: I never used <a href="https://twistedmatrix.com/">Twisted</a>, nor have I
done really much/any asynchronous programming (except for multithreaded or
multiprocessed code if you count that). As such, most of this was entirely new
for me.</p>
<h2>Basic Definitions</h2>
<p>To talk about <code>asyncio</code>, you need to talk about coroutines. To talk about
coroutines, you need to talk about generators. To talk about generators, you
need to talk about the <code>yield</code> statement. Rather than regurgitate all this, I'm
going to point you at a phenomenal post by <a href="https://twitter.com/brettsky">Brett
Cannon</a> (one of the Python core devs) that gives
all this background info:</p>
<p><a href="https://snarky.ca/how-the-heck-does-async-await-work-in-python-3-5/">https://snarky.ca/how-the-heck-does-async-await-work-in-python-3-5/</a></p>
<p>That post is largely about the <code>async</code>/<code>await</code> keywords introduced in Python
3.5, but Brett does a phenomenal job of painstakingly going through the
fundamentals of coroutines, generators, etc. Please do give this a read before
continuing on (I know it's long, but really worth it).</p>
<p>I'm not going to go into too much detail talking about what asynchronous
programming is, or why you might want to do it, instead I'll point you at these
resources:</p>
<ul>
<li><a href="http://sdiehl.github.io/gevent-tutorial/#synchronous-asynchronous-execution">http://sdiehl.github.io/gevent-tutorial/#synchronous-asynchronous-execution</a></li>
<li><a href="https://djangostars.com/blog/asynchronous-programming-in-python-asyncio/">https://djangostars.com/blog/asynchronous-programming-in-python-asyncio/</a></li>
<li><a href="https://medium.freecodecamp.org/a-guide-to-asynchronous-programming-in-python-with-asyncio-232e2afa44f6">https://medium.freecodecamp.org/a-guide-to-asynchronous-programming-in-python-with-asyncio-232e2afa44f6</a></li>
</ul>
<p>These videos are also quite good (though can be long):</p>
<ul>
<li><a href="https://www.youtube.com/watch?v=ZKnETCH4S0w">https://www.youtube.com/watch?v=ZKnETCH4S0w</a></li>
<li><a href="https://www.youtube.com/watch?v=Z_OAlIhXziw">https://www.youtube.com/watch?v=Z_OAlIhXziw</a> (classic, though pre-dates asyncio by a good degree)</li>
</ul>
<p>Some of the key ideas to take away from those:</p>
<ul>
<li><code>asyncio</code> is an implementation of an event loop that coroutines can be scheduled on</li>
<li><code>async</code> and <code>await</code> are keywords added for convenience sake to make coroutines easier</li>
<li>asynchronous programming with asyncio means "single threaded concurrency"</li>
</ul>
<p>That last point is really key: with <code>asyncio</code> only one thread is in play. You
get the concurrency via cooperative multitasking between coroutines (ie when one
blocks on IO you switch to another while the previous one is waiting). This has
a few implications.</p>
<p>One of those is if your code is CPU bound you're not going to get any perf
benefit from <code>asyncio</code>. This is true of threads as well because of the GIL, but
not for multiprocessing (if that's possible for your problem, though at the
expense of complexity, safety, and resources).</p>
<p>Another is because it's all single-threaded it can be easier to reason about,
and nasty problems like <a href="https://en.wikipedia.org/wiki/Race_condition">race
conditions</a> tend to be less
common.</p>
<h2>Hello World As A Coroutine</h2>
<p>Or "How the %@#@ do I even run a coroutine?" This is the first spot where I
feel a very real pain with <code>asyncio</code> -- it feels like there's multiple ways to
do basically the same thing, but that vary slightly leading to lots of cognitive
load in using the library. But a basic example:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">asyncio</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">helloworld</span><span class="p">():</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"Hello world from a coroutine!"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">main</span><span class="p">():</span>
<span class="n">loop</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">get_event_loop</span><span class="p">()</span>
<span class="n">loop</span><span class="o">.</span><span class="n">run_until_complete</span><span class="p">(</span><span class="n">helloworld</span><span class="p">())</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">"__main__"</span><span class="p">:</span>
<span class="n">main</span><span class="p">()</span>
</pre></div>
<p>Breaking this down:</p>
<ul>
<li>the <code>async</code> keyword (new in Python 3.5) makes <code>helloworld()</code> a coroutine</li>
<li>the call to <code>get_event_loop()</code> gets us an event loop we can schedule this coroutine on</li>
<li>the call to <code>run_until_complete()</code> does a few things:<ul>
<li>schedules the <code>helloworld()</code> coroutine to run on the loop</li>
<li>yields control to the event loop scheduler</li>
<li>the event loop then picks up that scheduled coroutine & switches control to it</li>
<li>the coroutine executes, prints the message and then returns control back to the loop</li>
<li>the loop then switches control back to the <code>main()</code> routine</li>
</ul>
</li>
</ul>
<p>And then the program continues on its merry (synchronous) way.</p>
<p>It's worth noting that "scheduling a coroutine" can be done many different ways.
<code>run_until_complete()</code> will do it implicitly with a coroutine by wrapping it in
a <code>Future</code> & scheduling it, but there are others:</p>
<div class="highlight"><pre><span></span><span class="n">loop</span><span class="o">.</span><span class="n">run_until_complete</span><span class="p">(</span><span class="n">asyncio</span><span class="o">.</span><span class="n">wait</span><span class="p">([</span><span class="n">helloworld</span><span class="p">()]))</span>
<span class="o">...</span> <span class="ow">or</span> <span class="o">...</span>
<span class="n">loop</span><span class="o">.</span><span class="n">run_until_complete</span><span class="p">(</span><span class="n">asyncio</span><span class="o">.</span><span class="n">gather</span><span class="p">(</span><span class="n">helloworld</span><span class="p">()))</span>
</pre></div>
<p>And probably others I'm not thinking of. The idea is that <code>run_until_complete</code>
gets a <code>Future</code> and it runs that <code>Future</code> to completion. If given a coroutine
it'll wrap it in a <code>Future</code> before running. <code>wait()</code> takes a collection of
<code>Future</code>s and returns a <code>Future</code> representing the completion of all those
sub-Future's. Much like <code>run_until_complete</code>, if it's given a coroutine it'll
wrap it in a <code>Future</code> (well, technically <code>Task</code>s, but those are subclasses of
<code>Future</code>). <code>gather()</code> is similar in that you give it "a bunch of tasks", but
takes either coroutines or <code>Future</code>'s. There's also some very subtle
differences between <code>gather()</code> and <code>wait()</code> in terms of cancellation ability,
return values, etc.</p>
<p>There's other ways to schedule tasks as well. In Python 3.7 there's a new
<code>create_task</code> routine in <code>asyncio</code> that schedules a task, but requires that
you're executing in a running event loop. For example to trigger a coroutine to
run from within a coroutine:</p>
<div class="highlight"><pre><span></span><span class="k">async</span> <span class="k">def</span> <span class="nf">helloworld</span><span class="p">():</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"Hello world from a coroutine!"</span><span class="p">)</span>
<span class="n">asyncio</span><span class="o">.</span><span class="n">create_task</span><span class="p">(</span><span class="n">helloworld</span><span class="p">())</span>
<span class="k">def</span> <span class="nf">main</span><span class="p">():</span>
<span class="n">loop</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">get_event_loop</span><span class="p">()</span>
<span class="n">loop</span><span class="o">.</span><span class="n">run_until_complete</span><span class="p">(</span><span class="n">helloworld</span><span class="p">())</span>
</pre></div>
<p>Runs the <code>helloworld()</code> coroutine twice. It's also worth noting that
<code>asyncio.create_task</code> (introduced in 3.7) is <em>not</em> the same thing as
<code>AbstractEventLoop.create_task</code> though serves largely the same purpose. The
previous example using <code>create_task</code> could also be written as:</p>
<div class="highlight"><pre><span></span><span class="k">async</span> <span class="k">def</span> <span class="nf">helloworld</span><span class="p">():</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"Hello world from a coroutine!"</span><span class="p">)</span>
<span class="n">asyncio</span><span class="o">.</span><span class="n">get_event_loop</span><span class="p">()</span><span class="o">.</span><span class="n">create_task</span><span class="p">(</span><span class="n">helloworld</span><span class="p">())</span>
<span class="k">def</span> <span class="nf">main</span><span class="p">():</span>
<span class="n">loop</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">get_event_loop</span><span class="p">()</span>
<span class="n">loop</span><span class="o">.</span><span class="n">run_until_complete</span><span class="p">(</span><span class="n">helloworld</span><span class="p">())</span>
</pre></div>
<p>It appears as though <code>asyncio.create_task()</code> is just shorthand to save you the call
to <code>get_event_loop()</code>.</p>
<p>Note that doing <code>asyncio.create_task</code> from in <code>main()</code> would be an error:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">main</span><span class="p">():</span>
<span class="n">loop</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">get_event_loop</span><span class="p">()</span>
<span class="n">asyncio</span><span class="o">.</span><span class="n">create_task</span><span class="p">(</span><span class="n">helloworld</span><span class="p">())</span> <span class="c1"># gives "RuntimeError: no running event loop"</span>
</pre></div>
<p>But <code>loop.create_task</code> is fine (though unless the loop is run the task will never be
executed):</p>
<div class="highlight"><pre><span></span> <span class="n">loop</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">get_event_loop</span><span class="p">()</span>
<span class="n">loop</span><span class="o">.</span><span class="n">create_task</span><span class="p">(</span><span class="n">helloworld</span><span class="p">())</span>
<span class="n">loop</span><span class="o">.</span><span class="n">run_until_complete</span><span class="p">(</span><span class="n">helloworld</span><span class="p">())</span>
</pre></div>
<p>Initially the call to <code>asyncio.create_task(helloworld())</code> within <code>helloworld()</code>
puzzled me as I assumed that would result in infinite scheduling of the task,
but we only see the message get printed twice. So what's up with that? I <a href="https://stackoverflow.com/questions/51680178/asyncio-create-task-vs-await">asked
on
StackOverflow</a>
and the gist is that what happens is that <code>run_until_complete()</code> means "run
until this coroutine I'm giving you is done and then stop the event loop". So
<code>helloworld()</code> gets run once, it schedules another run of it on the event loop.
This then schedules another run, but because the first run (from the
<code>run_until_complete</code>) is now done, the loop is stopped and control returned.</p>
<p>You can use this for a neat trick, for example if we want to run a co-routine
repeatedly for 2 seconds:</p>
<div class="highlight"><pre><span></span><span class="k">async</span> <span class="k">def</span> <span class="nf">helloworld</span><span class="p">():</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"Hello world from a coroutine!"</span><span class="p">)</span>
<span class="n">asyncio</span><span class="o">.</span><span class="n">create_task</span><span class="p">(</span><span class="n">helloworld</span><span class="p">())</span>
<span class="k">def</span> <span class="nf">main</span><span class="p">():</span>
<span class="n">loop</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">get_event_loop</span><span class="p">()</span>
<span class="n">loop</span><span class="o">.</span><span class="n">run_until_complete</span><span class="p">(</span><span class="n">asyncio</span><span class="o">.</span><span class="n">gather</span><span class="p">(</span><span class="n">helloworld</span><span class="p">(),</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">2</span><span class="p">)))</span>
</pre></div>
<p>In terms of the "hey make the event loop run the stuff", there's kinda two basic
approaches. The first is <code>run_until_complete()</code> which just runs the
task/coroutine/whatever until it's done and returns control, and the second is
<code>run_forever</code> which runs until something calls <code>stop()</code> on the event loop. An
example of <code>run_forever</code>:</p>
<div class="highlight"><pre><span></span><span class="k">async</span> <span class="k">def</span> <span class="nf">start</span><span class="p">():</span>
<span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"in start"</span><span class="p">)</span>
<span class="k">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">main</span><span class="p">():</span>
<span class="n">loop</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">get_event_loop</span><span class="p">()</span>
<span class="n">loop</span><span class="o">.</span><span class="n">create_task</span><span class="p">(</span><span class="n">start</span><span class="p">())</span>
<span class="n">loop</span><span class="o">.</span><span class="n">run_forever</span><span class="p">()</span>
</pre></div>
<p>Note that this truly does run forever. Any call to <code>run_forever()</code> will cause
the current thread to yield control to the event loop, and that event loop will
continue to keep control until something executes a <code>.stop()</code> on the event loop.</p>
<p><code>run_forever()</code> seems to be a tricky construct to get right. Googling around I
had a lot of difficulty finding any examples of <code>run_forever()</code> that seemed to
show how to (in a single threaded application) use it. The common approach I
saw was to do the <code>run_forever()</code> in a separate thread:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">time</span>
<span class="kn">import</span> <span class="nn">asyncio</span>
<span class="kn">from</span> <span class="nn">threading</span> <span class="kn">import</span> <span class="n">Thread</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">start</span><span class="p">():</span>
<span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"in start"</span><span class="p">)</span>
<span class="k">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">run_it_forever</span><span class="p">(</span><span class="n">loop</span><span class="p">):</span>
<span class="n">loop</span><span class="o">.</span><span class="n">run_forever</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">main</span><span class="p">():</span>
<span class="n">loop</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">get_event_loop</span><span class="p">()</span>
<span class="n">loop</span><span class="o">.</span><span class="n">create_task</span><span class="p">(</span><span class="n">start</span><span class="p">())</span>
<span class="n">thread</span> <span class="o">=</span> <span class="n">Thread</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">run_it_forever</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">loop</span><span class="p">,))</span>
<span class="n">thread</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span>
<span class="n">loop</span><span class="o">.</span><span class="n">stop</span><span class="p">()</span>
</pre></div>
<p>This will run the <code>start()</code> coroutine in an event loop which is running
in a subthread, and after 5 seconds the main thread will stop the loop,
and then exit.</p>
<p>One thing to note here: suddenly now we've given up one of the primary
benefits of <code>asyncio</code>. This is no longer a single threaded, easy to reason
about program, but now with multiple threads, suddenly we've introduced
all the complexity of multithreaded programming just to get our event loop
running indefinitely.</p>
<p>There's a lot of notes in the docs about how you have to be careful about
mixing <code>asyncio</code> and multiple threads. I'll point you at the (not so great
but all we have) docs:</p>
<ul>
<li><a href="https://docs.python.org/3/library/asyncio-dev.html#concurrency-and-multithreading">https://docs.python.org/3/library/asyncio-dev.html#concurrency-and-multithreading</a></li>
<li><a href="https://docs.python.org/3/library/asyncio-subprocess.html#asyncio-subprocess-threads">https://docs.python.org/3/library/asyncio-subprocess.html#asyncio-subprocess-threads</a></li>
<li><a href="https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.AbstractEventLoop.call_soon_threadsafe">https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.AbstractEventLoop.call_soon_threadsafe</a></li>
</ul>
<p>Again, this is some of the pain of <code>asyncio</code>, knowing what to use where.</p>
<h2>"Infinite" co-routines</h2>
<p>Sometimes you want to have a coroutine that is effectively "always running in
the background". The classic example is something that reads stuff off of some
shared memory (like a queue) and do some sort of background processing with each
item.</p>
<p>Some boilerplate code that I'll assume is present in each example:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">asyncio</span>
<span class="kn">import</span> <span class="nn">random</span>
<span class="kn">import</span> <span class="nn">functools</span>
<span class="c1"># make the print function always flush output immediately</span>
<span class="c1"># without this you'll find nothing gets printed until program</span>
<span class="c1"># end when suddenly all output is printed at once</span>
<span class="nb">print</span> <span class="o">=</span> <span class="n">functools</span><span class="o">.</span><span class="n">partial</span><span class="p">(</span><span class="nb">print</span><span class="p">,</span> <span class="n">flush</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="c1"># A message to indicate end of processing, the purpose of this</span>
<span class="c1"># will become clear in the examples</span>
<span class="n">END_OF_QUEUE</span> <span class="o">=</span> <span class="s2">"This is the end... This value doesn't actually matter so long as it's not a possible real value to go in the queue"</span>
</pre></div>
<p><code>asyncio</code> provides a <code>Queue</code> implementation that allows for asynchronously
yielding control when doing a <code>put()</code> or <code>get()</code>. A basic example of a
co-routine that loops waiting on a queue:</p>
<div class="highlight"><pre><span></span><span class="k">async</span> <span class="k">def</span> <span class="nf">queue_coro</span><span class="p">(</span><span class="n">queue</span><span class="p">):</span>
<span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="n">message</span> <span class="o">=</span> <span class="k">await</span> <span class="n">queue</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="k">if</span> <span class="n">message</span> <span class="o">==</span> <span class="n">END_OF_QUEUE</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"This is the end my friend...."</span><span class="p">)</span>
<span class="k">break</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"Got message: </span><span class="si">{</span><span class="n">message</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">"__main__"</span><span class="p">:</span>
<span class="n">queue</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">Queue</span><span class="p">()</span>
<span class="n">loop</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">get_event_loop</span><span class="p">()</span>
<span class="n">loop</span><span class="o">.</span><span class="n">run_until_complete</span><span class="p">(</span><span class="n">queue</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="s2">"foobar!"</span><span class="p">))</span>
<span class="n">loop</span><span class="o">.</span><span class="n">run_until_complete</span><span class="p">(</span><span class="n">queue</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="s2">"foobar is another!"</span><span class="p">))</span>
<span class="n">loop</span><span class="o">.</span><span class="n">run_until_complete</span><span class="p">(</span><span class="n">queue</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">END_OF_QUEUE</span><span class="p">))</span>
<span class="n">loop</span><span class="o">.</span><span class="n">run_until_complete</span><span class="p">(</span><span class="n">queue_coro</span><span class="p">(</span><span class="n">queue</span><span class="p">))</span>
<span class="n">loop</span><span class="o">.</span><span class="n">stop</span><span class="p">()</span>
<span class="n">loop</span><span class="o">.</span><span class="n">run_forever</span><span class="p">()</span>
<span class="n">loop</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
</pre></div>
<p>outputs:</p>
<div class="highlight"><pre><span></span>Got message: foobar!
Got message: foobar is another!
This is the end my friend....
</pre></div>
<p>Description: we put two items on the queue, and then the special <code>END_OF_QUEUE</code>
value which acts as a <a href="https://en.wikipedia.org/wiki/Sentinel_value">sentinel
value</a> to indicate that processing
is done and the coroutine consuming items from the queue should quit. We then
queue up that consumer coroutine & run it.</p>
<p>Note if you put that <code>END_OF_QUEUE</code> after the <code>queue_coro</code> then it goes
forever. Remember that the event loop is single threaded, so until something
finishes or yields control, it'll keep running.</p>
<p>We can expand this to a basic
<a href="https://en.wikipedia.org/wiki/Producer%E2%80%93consumer_problem">producer/consumer</a>
example:</p>
<div class="highlight"><pre><span></span><span class="k">async</span> <span class="k">def</span> <span class="nf">producer</span><span class="p">(</span><span class="n">queue</span><span class="p">):</span>
<span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="n">emit_me</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">42</span><span class="p">)</span>
<span class="k">if</span> <span class="n">emit_me</span> <span class="o">==</span> <span class="mi">42</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"At the end"</span><span class="p">)</span>
<span class="k">break</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">msg</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">"Emitting </span><span class="si">{</span><span class="n">emit_me</span><span class="si">}</span><span class="s2">"</span>
<span class="nb">print</span><span class="p">(</span><span class="n">msg</span><span class="p">)</span>
<span class="c1"># simulate some IO time</span>
<span class="k">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">random</span><span class="o">.</span><span class="n">uniform</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mf">0.5</span><span class="p">))</span>
<span class="k">await</span> <span class="n">queue</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">msg</span><span class="p">)</span>
<span class="k">await</span> <span class="n">queue</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">END_OF_QUEUE</span><span class="p">)</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">consumer</span><span class="p">(</span><span class="n">queue</span><span class="p">):</span>
<span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="n">message</span> <span class="o">=</span> <span class="k">await</span> <span class="n">queue</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="k">if</span> <span class="n">message</span> <span class="o">==</span> <span class="n">END_OF_QUEUE</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"This is the end my friend...."</span><span class="p">)</span>
<span class="k">break</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"Got message: </span><span class="si">{</span><span class="n">message</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="c1"># simulate some IO time</span>
<span class="k">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">random</span><span class="o">.</span><span class="n">uniform</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">))</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">"__main__"</span><span class="p">:</span>
<span class="n">loop</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">get_event_loop</span><span class="p">()</span>
<span class="n">queue</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">Queue</span><span class="p">()</span>
<span class="n">loop</span><span class="o">.</span><span class="n">run_until_complete</span><span class="p">(</span><span class="n">asyncio</span><span class="o">.</span><span class="n">gather</span><span class="p">(</span><span class="n">producer</span><span class="p">(</span><span class="n">queue</span><span class="p">),</span> <span class="n">consumer</span><span class="p">(</span><span class="n">queue</span><span class="p">)))</span>
<span class="n">loop</span><span class="o">.</span><span class="n">stop</span><span class="p">()</span>
<span class="n">loop</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
</pre></div>
<p>Output will vary since values produced are randomly generated, but as an
example:</p>
<div class="highlight"><pre><span></span>Emitting 38
Emitting 35
Got message: Emitting 38
Emitting 29
Emitting 6
Got message: Emitting 35
Emitting 30
Emitting 14
Got message: Emitting 29
Emitting 20
Emitting 24
Emitting 19
Got message: Emitting 6
Emitting 30
Got message: Emitting 30
Got message: Emitting 14
Emitting 24
Emitting 5
Got message: Emitting 20
Got message: Emitting 24
Emitting 10
Got message: Emitting 19
Emitting 35
Emitting 23
Emitting 18
Got message: Emitting 30
Got message: Emitting 24
Emitting 31
Emitting 5
Emitting 39
Got message: Emitting 5
Got message: Emitting 10
Emitting 29
Emitting 24
Emitting 34
Got message: Emitting 35
Emitting 30
Emitting 31
Got message: Emitting 23
Emitting 31
Emitting 1
Emitting 36
Got message: Emitting 18
Emitting 20
Got message: Emitting 31
Emitting 12
Got message: Emitting 5
At the end
Got message: Emitting 39
Got message: Emitting 29
Got message: Emitting 24
Got message: Emitting 34
Got message: Emitting 30
Got message: Emitting 31
Got message: Emitting 31
Got message: Emitting 1
Got message: Emitting 36
Got message: Emitting 20
Got message: Emitting 12
This is the end my friend....
</pre></div>
<p>The idea here is that the producer is generating values at a rate that is faster
than the consumer can consume (hence the asyncio sleep call in the range 0 to
0.5 seconds, whereas the consumer sleeps in the range 0 to 1 seconds). This is
why we see the "emitting" messages appearing at a greater rate than the "Got
message" messages, and then after seeing "At the end", which is when the
producer has found 42 and terminates, we then see the consumer then go ahead and
consume any remaining items that have been enqueued.</p>
<p>Note that those sleep calls are critical, try taking them out and you'll see
output looks like:</p>
<div class="highlight"><pre><span></span>Emitting 17
Emitting 20
Emitting 24
Emitting 2
Emitting 29
At the end
Got message: Emitting 17
Got message: Emitting 20
Got message: Emitting 24
Got message: Emitting 2
Got message: Emitting 29
This is the end my friend....
</pre></div>
<p>That is, the producer keeps producing until 42 is found, and then the consumer
consumes everything. The reason for this is that the <code>asyncio.sleep()</code> call will
yield control (it's effectively an I/O operation). Remember: <code>asyncio</code> is
single threaded concurrency, without those sleep calls what happens is the item
is put on the queue in the producer, then control immediately returned to the
producer, which loops and produces the next value, etc.</p>
<p>Note that this is a neat trick, if your intent in a coroutine is to yield
control to something else you can always do a <code>asyncio.sleep(0)</code> which pauses
the coroutine returning control back to the event loop, which then schedules the
next task and passes control to it.</p>
<h2>Testing Antipatterns</h2>
<p>Another significant challenge with <code>asyncio</code> is around testing. Firstly there
isn't a lot of great support yet for testing <code>asyncio</code> code (there is a
<a href="https://github.com/pytest-dev/pytest-asyncio"><code>pytest-asyncio</code></a> package which
offers a few decorators & such, but I haven't found it to be particularly
helpful/useful yet).</p>
<p>I'll refer to you another post by <a href="https://twitter.com/miguelgrinberg">Miguel
Grinberg</a> on <a href="https://blog.miguelgrinberg.com/post/unit-testing-asyncio-code">Unit Testing AsyncIO
Code</a> where he
discusses some of the challenges testing async code and approaches to solving them.
Miguel's a smart cookie who's most known for the <a href="https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-i-hello-world">Flask
Mega-Tutorial</a>
which is something of the defacto way to learn <a href="http://flask.pocoo.org/">Flask</a>.</p>
<p>In addition, one gotcha I ran into not mentioned by Miguel was related to
<code>get_event_loop()</code>. If you have code that say gets an event loop through
<code>get_event_loop()</code>, does some processing, and then <em>stops</em> that event loop, that
can cause subsequent tests to fail if they also use <code>get_event_loop()</code>.</p>
<p>For example:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">thing_im_testing</span><span class="p">(</span><span class="n">some_argument</span><span class="p">):</span>
<span class="n">loop</span> <span class="o">=</span> <span class="n">get_event_loop</span><span class="p">()</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">loop</span><span class="o">.</span><span class="n">run_until_complete</span><span class="p">(</span><span class="n">some_coroutine</span><span class="p">(</span><span class="n">some_argument</span><span class="p">))</span>
<span class="k">return</span> <span class="n">result</span>
<span class="k">def</span> <span class="nf">test_the_thing</span><span class="p">():</span>
<span class="n">loop</span> <span class="o">=</span> <span class="n">get_event_loop</span><span class="p">()</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">thing_im_testing</span><span class="p">(</span><span class="s2">"some value"</span><span class="p">)</span>
<span class="n">loop</span><span class="o">.</span><span class="n">stop</span><span class="p">()</span>
<span class="n">loop</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">test_the_thing_a_different_way</span><span class="p">():</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">thing_im_testing</span><span class="p">(</span><span class="s2">"some value"</span><span class="p">)</span>
</pre></div>
<p>This is contrived, but sometimes <code>test_the_thing_a_different_way()</code> will pass,
and sometimes it will fail depending on the order the tests were run. Generally
speaking having test order matter is an anti-pattern and a strong indicator of a
bad test.</p>
<p>Best practice: use a fixture or setup/teardown to create a fresh event loop with
a call to <code>new_event_loop()</code> <em>per test</em> to ensure tests do their async stuff in
isolation.</p>
<h2>It Is a Complex Beast</h2>
<p>There's a now rather famous post by <a href="http://lucumr.pocoo.org/about/">Armin
Ronacher</a> (the person who created
<a href="http://flask.pocoo.org/">Flask</a>, <a href="http://jinja.pocoo.org/">Jinja</a>, and so many
other seminal Python projects) talking about the complexities of <code>asyncio</code>.
It's an extremely well-written and sobering post from someone who's been in at
the core of the Python community for a long time now. Give it a read at:
<a href="http://lucumr.pocoo.org/2016/10/30/i-dont-understand-asyncio/">http://lucumr.pocoo.org/2016/10/30/i-dont-understand-asyncio/</a></p>
<p>It's worth noting that that post was written pre-Python 3.6, but after Python 3.5
(when <code>async</code>/<code>await</code> were introduced).</p>
<p>Thus far for me I'd echo a similar sentiment about <code>asyncio</code>, for trivial toy
examples it makes sense. For example, if I have a function that needs to hit 3
separate REST API endpoints, it's really easy to use <code>asyncio</code> to make those
calls into coroutines, and execute them concurrently with a
<code>run_until_complete()</code> saving some time and avoiding the overhead of
threads/processes. Having said that, once you get past the little toy examples,
figuring out all the subtle differences in the various methods across
<code>AbstractEventLoop</code>s, the <code>asyncio</code> module, <code>Future</code>s, <code>Task</code>s, etc, it gets
really heavy really fast from a cognitive load point of view.</p>
<p>This is already one of my longest posts, and I barely scratched the surface, I
didn't talk at all (or even start looking into)
<a href="https://docs.python.org/3/library/asyncio-protocol.html">protocols or transports</a>,
or <a href="https://docs.python.org/3/library/asyncio-stream.html">streams</a>. I also didn't
get into how the async aspect of the library tends to "leak" all throughout your
code, which (going back to the testing issue) can be a real impediment to forward
progress.</p>
<p>Like Armin said, "It's hard to comprehend how it works in all details. When you can
pass a generator, when it has to be a real coroutine, what futures are, what tasks
are, how the loop works and that did not even come to the actual IO part." There's
real promise with <code>asyncio</code>, but the complexity cost of using it currently is
extremely high.</p>Python Tip of the Day - Merging Dicts2018-08-03T18:35:00-07:002018-08-03T18:35:00-07:00Adam Parkintag:www.codependentcodr.com,2018-08-03:/python-tip-of-the-day-merging-dicts.html<p>Python Tip of the Day - Merging Two Dicts</p><p>So it's not uncommon to want to merge two dictionaries
in Python into a single dict. Typically, you'd do this like so:</p>
<div class="highlight"><pre><span></span><span class="n">x</span> <span class="o">=</span> <span class="p">{</span><span class="s2">"key1"</span><span class="p">:</span> <span class="s2">"value1"</span><span class="p">,</span> <span class="s2">"key2"</span><span class="p">:</span> <span class="s2">"value2"</span><span class="p">}</span>
<span class="n">y</span> <span class="o">=</span> <span class="p">{</span><span class="s2">"key2"</span><span class="p">:</span> <span class="s2">"valuefromy"</span><span class="p">,</span> <span class="s2">"key3"</span><span class="p">:</span> <span class="s2">"value3"</span><span class="p">}</span>
<span class="n">x</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">y</span><span class="p">)</span>
</pre></div>
<p>But this is problematic because you're modifying <code>x</code> in place. Additionally it's
multiple lines, instead of a single line to combine the two. Wouldn't it be nice
if you could combine two dicts in a single line to produce a new dict with the
combination of the two?</p>
<p>Indeed it would, and as of Python 3.5 you can.</p>
<div class="highlight"><pre><span></span><span class="n">x</span> <span class="o">=</span> <span class="p">{</span><span class="s2">"key1"</span><span class="p">:</span> <span class="s2">"value1"</span><span class="p">,</span> <span class="s2">"key2"</span><span class="p">:</span> <span class="s2">"value2"</span><span class="p">}</span>
<span class="n">y</span> <span class="o">=</span> <span class="p">{</span><span class="s2">"key2"</span><span class="p">:</span> <span class="s2">"valuefromy"</span><span class="p">,</span> <span class="s2">"key3"</span><span class="p">:</span> <span class="s2">"value3"</span><span class="p">}</span>
<span class="nb">print</span><span class="p">({</span><span class="o">**</span><span class="n">x</span><span class="p">,</span> <span class="o">**</span><span class="n">y</span><span class="p">})</span> <span class="c1"># prints {'key1': 'value1', 'key2': 'valuefromy', 'key3': 'value3'}</span>
</pre></div>
<p>Nice little shorthand. It's worth noting as well that this is actually quite fast:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="n">timeit</span><span class="o">.</span><span class="n">timeit</span><span class="p">(</span><span class="s2">"{**x, **y}"</span><span class="p">,</span> <span class="n">setup</span><span class="o">=</span><span class="s1">'x = {"key1": "value1", "key2": "value2"}; y = {"key2": "valuefromy", "key3": "value3"}'</span><span class="p">)</span>
<span class="mf">0.17988868800000546</span>
<span class="o">>>></span> <span class="n">timeit</span><span class="o">.</span><span class="n">timeit</span><span class="p">(</span><span class="s2">"x.update(y)"</span><span class="p">,</span> <span class="n">setup</span><span class="o">=</span><span class="s1">'x = {"key1": "value1", "key2": "value2"}; y = {"key2": "valuefromy", "key3": "value3"}'</span><span class="p">)</span>
<span class="mf">0.20185830300002294</span>
</pre></div>
<p>Best of all worlds, it's short, fast, concise and side-effect free.</p>Code Quality Challenge #1 - Test Coverage2018-08-01T08:10:00-07:002018-08-01T08:10:00-07:00Adam Parkintag:www.codependentcodr.com,2018-08-01:/code-quality-challenge-1-test-coverage.html<p>CQC #1 - Exploring Test Coverage</p><h2>The Challenge</h2>
<p>Today's challenge is to explore the use of test coverage tools to see if you can find
any shortcomings or holes in your test suite.</p>
<p>You (hopefully) have some unit tests in place on your project. But do you have
enough tests? Are there still critical parts of your code that are untested by
an automated test? How would you know?</p>
<p>If you use Python, the de-facto tool for answering this question is
<a href="https://coverage.readthedocs.io/en/coverage-4.5.1a/">Coverage.py</a>. And if you're
using <a href="https://pytest.org/">Pytest</a> to run your tests, then generating a coverage
report is trivial:</p>
<div class="highlight"><pre><span></span>pytest /path/to/your/tests --cov<span class="o">=</span>nameofyourproject --cov-report html:/path/to/where/to/put/the/report
</pre></div>
<p>This will generate a HTML-based test coverage report which will show you what parts
of your code are and are not covered by your test suite.</p>
<p>So spend 20 minutes today and see if you can bump up your test coverage just a little.
Remember the goal isn't 100%, the goal is to just make an improvement to either how
much coverage you have, or give some insight into how much test coverage you have.</p>
<p>20 minutes, go!</p>
<h2>What I did</h2>
<p>On my project at work we already had coverage set up to run as part of our CI pipeline.
The tool itself is run inside a Docker container, the HTML report produced, and then
archived as a build artifact that you can then examine on the build page for a particular
build.</p>
<p>I used this mechanism to see what code was not yet tested. As it turned out we had
a bunch of code that made use of Python Abstract Base Classes (ABC's) and had some
abstract properties. The gist:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Foo</span><span class="p">(</span><span class="n">ABC</span><span class="p">):</span>
<span class="nd">@property</span>
<span class="nd">@abstractmethod</span>
<span class="k">def</span> <span class="nf">someproperty</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">pass</span>
</pre></div>
<p>The problem is that the <code>pass</code> statements were being counted by Coverage, even
though by definition those lines cannot be hit by a test (the whole point of an
abstract method is that it can't be called, but rather has to be overridden by a
child class).</p>
<p>The solution, add docstrings:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Foo</span><span class="p">(</span><span class="n">ABC</span><span class="p">):</span>
<span class="nd">@property</span>
<span class="nd">@abstractmethod</span>
<span class="k">def</span> <span class="nf">someproperty</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""Returns the .....meaningful description here...."""</span>
</pre></div>
<p>Because a docstring is a valid method body, this works as valid Python code. And
coverage doesn't see a pass statement, so no count towards coverage. Lastly as
a nice side benefit, now I'm better adhering to Python style guidelines which
generally suggest public methods/properties should have docstrings (tools like
<a href="https://www.pylint.org">Pylint</a> and
<a href="https://github.com/PyCQA/pydocstyle">Pydocstyle</a> will actually flag public
methods without docstrings as warnings). Win win.</p>
<p>Doing this resulted in test coverage increasing around 1%.</p>
<h2>What About You</h2>
<p>Did you try the challenge? How'd it go? Would love to hear any feedback, comments, or
observations. And if you have ideas for future challenges, please feel free to suggest
them!</p>Code Quality Challenge #0 - Getting Started2018-08-01T08:03:00-07:002018-08-01T08:03:00-07:00Adam Parkintag:www.codependentcodr.com,2018-08-01:/code-quality-challenge-0-getting-started.html<p>Let's do a quality challenge to find some quick wins.</p><p>I recently did the <a href="https://www.codequalitychallenge.com/">30 Day Code Quality
Challenge</a> and enjoyed it greatly. As
such, now that it's over I'm trying to think of ways to keep the momentum built
from doing 20 minute "little wins" every day for a month.</p>
<p>On the last day on the forums Ben (the organizer) solicited feedback on the event. I found myself
thinking of some new activities for challenges that could be used and suggested them.</p>
<p>And then I thought: why not just organize them myself? So that's what this post is about. This one
is a meta post describing my plan, but after this, I'm going to periodically (definitely not daily,
maybe weekly?) post a blog post describing a code quality challenge much like what happened in the
event. I'll describe the challenge and why it's worthwhile, do it, and then describe what I did
and how it went.</p>
<p>Sometimes they'll be things like "hey try out this cool refactoring idea". Sometimes "hey go read
this stuff to learn something cool". And everything in between. The key part is not the output,
but rather the building and maintaining of the habit of dedicating 20 minutes, just 20 minutes, to
making whatever it is you're working on better (be that your codebase, yourself, your career, etc).</p>
<p>Sounds like a plan, now to write up day 1....</p>F-Strings Are F'ing Cool2018-07-12T15:27:00-07:002018-07-12T15:27:00-07:00Adam Parkintag:www.codependentcodr.com,2018-07-12:/f-strings-are-fing-cool.html<p>Python 3.6 introduced a new way to format strings. They're really f-ing cool. Let's see why.</p><p>In Python 3.6 a new (yet another) way to format strings was introduced -- F-strings.
These were introduced as part of <a href="https://www.python.org/dev/peps/pep-0498/">PEP-498</a>,
and while there can be a "really do we need yet another way to do this?" response to them
when you first see them, in actuality they are one of the coolest additions to the
language I've seen in recent years.</p>
<h2>The Basics</h2>
<p>You denote an f-string by prefixing a <code>f</code> or <code>F</code> in front of a string literal:</p>
<div class="highlight"><pre><span></span><span class="n">mystring</span> <span class="o">=</span> <span class="sa">f</span><span class="s1">'This is an f-string!'</span>
<span class="nb">print</span><span class="p">(</span><span class="n">mystring</span><span class="p">)</span> <span class="c1"># prints "This is an f-string!"</span>
</pre></div>
<p>Nothing exciting here, where f-strings get more interesting is when you want to
interpolate a value into it:</p>
<div class="highlight"><pre><span></span><span class="n">x</span> <span class="o">=</span> <span class="mi">42</span>
<span class="n">mystring</span> <span class="o">=</span> <span class="sa">f</span><span class="s1">'The answer to life, the universe and everything is </span><span class="si">{</span><span class="n">x</span><span class="si">}</span><span class="s1">'</span>
<span class="nb">print</span><span class="p">(</span><span class="n">mystring</span><span class="p">)</span> <span class="c1"># prints "The answer to life, the universe and everything is 42"</span>
</pre></div>
<p>Ok, so that's just like the string formatting operator (<code>%</code>) and the <code>format()</code>
function. Ie these are all equivalent:</p>
<div class="highlight"><pre><span></span><span class="n">x</span> <span class="o">=</span> <span class="mi">42</span>
<span class="n">with_string_op</span> <span class="o">=</span> <span class="s1">'The answer to life, the universe and everything is </span><span class="si">%d</span><span class="s1">'</span> <span class="o">%</span> <span class="n">x</span>
<span class="n">with_format_fn</span> <span class="o">=</span> <span class="s1">'The answer to life, the universe and everything is </span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="n">with_f_string</span> <span class="o">=</span> <span class="sa">f</span><span class="s1">'The answer to life, the universe and everything is </span><span class="si">{</span><span class="n">x</span><span class="si">}</span><span class="s1">'</span>
<span class="nb">print</span><span class="p">(</span><span class="n">with_string_op</span> <span class="o">==</span> <span class="n">with_format_fn</span> <span class="o">==</span> <span class="n">with_f_string</span><span class="p">)</span> <span class="c1"># prints "True"</span>
</pre></div>
<p>Ok, I hear you saying that's neat, but isn't that just syntactic sugar? Adam, why do you think
F-strings are cool? Let's see some more cool tricks and reasons why you should start
using them.</p>
<h2>Arbitrary Expressions</h2>
<p>Note that the stuff inside the braces can be arbitrary Python expressions:</p>
<div class="highlight"><pre><span></span><span class="n">x</span> <span class="o">=</span> <span class="mi">100</span>
<span class="n">y</span> <span class="o">=</span> <span class="mi">99</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s1">'</span><span class="si">{</span><span class="n">x</span><span class="si">}</span><span class="s1"> + </span><span class="si">{</span><span class="n">y</span><span class="si">}</span><span class="s1"> * 2 is equal to </span><span class="si">{</span><span class="n">x</span> <span class="o">+</span> <span class="n">y</span> <span class="o">*</span> <span class="mi">2</span><span class="si">}</span><span class="s1">'</span><span class="p">)</span> <span class="c1"># prints "100 + 99 * 2 is equal to 298"</span>
</pre></div>
<p>This affords an enormous amount of flexibility, as pretty much anything that's a valid
Python bit of code could be put into an f-string block. This also tends to make for
strings that are more readable, consider:</p>
<div class="highlight"><pre><span></span><span class="n">logging</span><span class="o">.</span><span class="n">warn</span><span class="p">(</span><span class="s2">"Disk space for drive </span><span class="si">{}</span><span class="s2"> is low, only </span><span class="si">{}</span><span class="s2"> bytes remaining"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">driveid</span><span class="p">,</span> <span class="n">space_left</span><span class="p">))</span>
</pre></div>
<p>vs</p>
<div class="highlight"><pre><span></span><span class="n">logging</span><span class="o">.</span><span class="n">warn</span><span class="p">(</span><span class="sa">f</span><span class="s2">"Disk space for drive </span><span class="si">{</span><span class="n">driveid</span><span class="si">}</span><span class="s2"> is low, only </span><span class="si">{</span><span class="n">space_left</span><span class="si">}</span><span class="s2"> bytes remaining"</span><span class="p">)</span>
</pre></div>
<p>Because the identifiers are right in the middle of the string literal, it's easier to
envision in your head what the final string will be, rather than doing the "ok, this
first argument goes here, and the second goes there" mental mapping you do with the
<code>format()</code> call. It is worth noting you can get the same thing with format, but it
becomes very verbose:</p>
<div class="highlight"><pre><span></span><span class="n">logging</span><span class="o">.</span><span class="n">warn</span><span class="p">(</span>
<span class="s2">"Disk space for drive </span><span class="si">{driveid}</span><span class="s2"> is low, only </span><span class="si">{space_left}</span><span class="s2"> bytes remaining"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
<span class="n">driveid</span><span class="o">=</span><span class="n">driveid</span><span class="p">,</span> <span class="n">space_left</span><span class="o">=</span><span class="n">space_left</span>
<span class="p">)</span>
<span class="p">)</span>
</pre></div>
<p>I find that hard to read, and I very much don't like the alignment of the arguments
across multiple lines. I formatted that line with <a href="https://github.com/ambv/black">Black</a>,
so perhaps it's a matter of my taste not aligning with that formatter, but I've always
struggled with manually formatting complex <code>format()</code> calls as it always feels like the final
result is ugly.</p>
<h2>Fast</h2>
<p>This is cool, f-strings are also fast. Why? Because the expression
is treated like a plain old python expression, parsed & evaluated at runtime. They
also save the overhead of a function call (like with <code>format()</code>), which we can see
with the <code>dis</code> module:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="kn">import</span> <span class="nn">dis</span>
<span class="o">>>></span> <span class="k">def</span> <span class="nf">foo</span><span class="p">():</span>
<span class="o">...</span> <span class="n">x</span> <span class="o">=</span> <span class="mi">42</span>
<span class="o">...</span> <span class="n">y</span> <span class="o">=</span> <span class="mi">99</span>
<span class="o">...</span> <span class="k">return</span> <span class="s1">'</span><span class="si">{}</span><span class="s1"> + </span><span class="si">{}</span><span class="s1"> = </span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">x</span> <span class="o">+</span> <span class="n">y</span><span class="p">)</span>
<span class="o">...</span>
<span class="o">>>></span> <span class="n">foo</span><span class="p">()</span>
<span class="s1">'42 + 99 = 141'</span>
<span class="o">>>></span> <span class="n">dis</span><span class="o">.</span><span class="n">dis</span><span class="p">(</span><span class="n">foo</span><span class="p">)</span>
<span class="mi">2</span> <span class="mi">0</span> <span class="n">LOAD_CONST</span> <span class="mi">1</span> <span class="p">(</span><span class="mi">42</span><span class="p">)</span>
<span class="mi">2</span> <span class="n">STORE_FAST</span> <span class="mi">0</span> <span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="mi">3</span> <span class="mi">4</span> <span class="n">LOAD_CONST</span> <span class="mi">2</span> <span class="p">(</span><span class="mi">99</span><span class="p">)</span>
<span class="mi">6</span> <span class="n">STORE_FAST</span> <span class="mi">1</span> <span class="p">(</span><span class="n">y</span><span class="p">)</span>
<span class="mi">4</span> <span class="mi">8</span> <span class="n">LOAD_CONST</span> <span class="mi">3</span> <span class="p">(</span><span class="s1">'</span><span class="si">{}</span><span class="s1"> + </span><span class="si">{}</span><span class="s1"> = </span><span class="si">{}</span><span class="s1">'</span><span class="p">)</span>
<span class="mi">10</span> <span class="n">LOAD_ATTR</span> <span class="mi">0</span> <span class="p">(</span><span class="nb">format</span><span class="p">)</span>
<span class="mi">12</span> <span class="n">LOAD_FAST</span> <span class="mi">0</span> <span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="mi">14</span> <span class="n">LOAD_FAST</span> <span class="mi">1</span> <span class="p">(</span><span class="n">y</span><span class="p">)</span>
<span class="mi">16</span> <span class="n">LOAD_FAST</span> <span class="mi">0</span> <span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="mi">18</span> <span class="n">LOAD_FAST</span> <span class="mi">1</span> <span class="p">(</span><span class="n">y</span><span class="p">)</span>
<span class="mi">20</span> <span class="n">BINARY_ADD</span>
<span class="mi">22</span> <span class="n">CALL_FUNCTION</span> <span class="mi">3</span>
<span class="mi">24</span> <span class="n">RETURN_VALUE</span>
</pre></div>
<p>The output of <code>dis</code> can be a bit hard to read, but notice the <code>LOAD_ATTR</code>
line, which is Python doing a lookup of the <code>format</code> function, and then there's
the <code>CALL_FUNCTION</code> instruction which is actually calling that function so you
pay the overhead of a function call (which in Python has always been
surprisingly expensive).</p>
<p>With f-strings we don't have that:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="k">def</span> <span class="nf">foo</span><span class="p">():</span>
<span class="o">...</span> <span class="n">x</span> <span class="o">=</span> <span class="mi">42</span>
<span class="o">...</span> <span class="n">y</span> <span class="o">=</span> <span class="mi">99</span>
<span class="o">...</span> <span class="k">return</span> <span class="sa">f</span><span class="s1">'</span><span class="si">{</span><span class="n">x</span><span class="si">}</span><span class="s1"> + </span><span class="si">{</span><span class="n">y</span><span class="si">}</span><span class="s1"> = </span><span class="si">{</span><span class="n">x</span> <span class="o">+</span> <span class="n">y</span><span class="si">}</span><span class="s1">'</span>
<span class="o">...</span>
<span class="o">>>></span> <span class="n">dis</span><span class="o">.</span><span class="n">dis</span><span class="p">(</span><span class="n">foo</span><span class="p">)</span>
<span class="mi">2</span> <span class="mi">0</span> <span class="n">LOAD_CONST</span> <span class="mi">1</span> <span class="p">(</span><span class="mi">42</span><span class="p">)</span>
<span class="mi">2</span> <span class="n">STORE_FAST</span> <span class="mi">0</span> <span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="mi">3</span> <span class="mi">4</span> <span class="n">LOAD_CONST</span> <span class="mi">2</span> <span class="p">(</span><span class="mi">99</span><span class="p">)</span>
<span class="mi">6</span> <span class="n">STORE_FAST</span> <span class="mi">1</span> <span class="p">(</span><span class="n">y</span><span class="p">)</span>
<span class="mi">4</span> <span class="mi">8</span> <span class="n">LOAD_FAST</span> <span class="mi">0</span> <span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="mi">10</span> <span class="n">FORMAT_VALUE</span> <span class="mi">0</span>
<span class="mi">12</span> <span class="n">LOAD_CONST</span> <span class="mi">3</span> <span class="p">(</span><span class="s1">' + '</span><span class="p">)</span>
<span class="mi">14</span> <span class="n">LOAD_FAST</span> <span class="mi">1</span> <span class="p">(</span><span class="n">y</span><span class="p">)</span>
<span class="mi">16</span> <span class="n">FORMAT_VALUE</span> <span class="mi">0</span>
<span class="mi">18</span> <span class="n">LOAD_CONST</span> <span class="mi">4</span> <span class="p">(</span><span class="s1">' = '</span><span class="p">)</span>
<span class="mi">20</span> <span class="n">LOAD_FAST</span> <span class="mi">0</span> <span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="mi">22</span> <span class="n">LOAD_FAST</span> <span class="mi">1</span> <span class="p">(</span><span class="n">y</span><span class="p">)</span>
<span class="mi">24</span> <span class="n">BINARY_ADD</span>
<span class="mi">26</span> <span class="n">FORMAT_VALUE</span> <span class="mi">0</span>
<span class="mi">28</span> <span class="n">BUILD_STRING</span> <span class="mi">5</span>
<span class="mi">30</span> <span class="n">RETURN_VALUE</span>
</pre></div>
<p>Note no function lookup. To give an idea of timing, let's use the <code>timeit</code> module,
first with the <code>format()</code> function:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="n">timeit</span><span class="o">.</span><span class="n">timeit</span><span class="p">(</span><span class="s2">"foo()"</span><span class="p">,</span> <span class="n">setup</span><span class="o">=</span><span class="s2">"from __main__ import foo"</span><span class="p">)</span>
<span class="mf">0.807306278962642</span>
</pre></div>
<p>And now with f-strings:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="n">timeit</span><span class="o">.</span><span class="n">timeit</span><span class="p">(</span><span class="s2">"foo()"</span><span class="p">,</span> <span class="n">setup</span><span class="o">=</span><span class="s2">"from __main__ import foo"</span><span class="p">)</span>
<span class="mf">0.6124071741942316</span>
</pre></div>
<p>Ballpark 25% faster, not bad. And just for completeness sake, what about the
string formatting operator:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="k">def</span> <span class="nf">foo</span><span class="p">():</span>
<span class="o">...</span> <span class="n">x</span> <span class="o">=</span> <span class="mi">42</span>
<span class="o">...</span> <span class="n">y</span> <span class="o">=</span> <span class="mi">99</span>
<span class="o">...</span> <span class="k">return</span> <span class="s1">'</span><span class="si">%d</span><span class="s1"> + </span><span class="si">%d</span><span class="s1"> = </span><span class="si">%d</span><span class="s1">'</span> <span class="o">%</span> <span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">x</span> <span class="o">+</span> <span class="n">y</span><span class="p">)</span>
<span class="o">...</span>
<span class="o">>>></span> <span class="n">foo</span><span class="p">()</span>
<span class="s1">'42 + 99 = 141'</span>
<span class="o">>>></span> <span class="n">timeit</span><span class="o">.</span><span class="n">timeit</span><span class="p">(</span><span class="s2">"foo()"</span><span class="p">,</span> <span class="n">setup</span><span class="o">=</span><span class="s2">"from __main__ import foo"</span><span class="p">)</span>
<span class="mf">0.5939487249124795</span>
</pre></div>
<p>Oooh, that's interesting, just a smidge faster than f-strings. Still, unlike
the <code>format()</code> function, there's no performance overhead to f-strings, and in
some cases it can be the fastest of all three approaches.</p>
<h2>Using format specifiers</h2>
<p>This is where you can do some neat tricks. If you've never really played with
format specifiers in Python, they can be quite useful. The relevant docs:
<a href="https://docs.python.org/3.7/library/string.html#format-specification-mini-language">https://docs.python.org/3.7/library/string.html#format-specification-mini-language</a></p>
<p>The Format Specification mini language is used with the <code>format()</code> function, but
as it turns out, you can also use format specifiers with f-strings. For example:</p>
<div class="highlight"><pre><span></span><span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"Pi to 3 digits: </span><span class="si">{</span><span class="n">math</span><span class="o">.</span><span class="n">pi</span><span class="si">:</span><span class="s2">.3f</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span> <span class="c1"># prints "Pi to 3 digits: 3.142"</span>
</pre></div>
<p>This can be handy for formatting as well:</p>
<div class="highlight"><pre><span></span><span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="mi">42</span><span class="si">:</span><span class="s2">05</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span> <span class="c1"># gives "00042"</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="mi">42</span><span class="si">:</span><span class="s2">a<10</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span> <span class="c1"># gives "42aaaaaaaa"</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="mi">42</span><span class="si">:</span><span class="s2">a>10</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span> <span class="c1"># gives "aaaaaaaa42"</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="mi">42</span><span class="si">:</span><span class="s2">3>10</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span> <span class="c1"># gives "3333333342"</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="mi">42</span><span class="si">:</span><span class="s2">=^10</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span> <span class="c1"># gives "====42===="</span>
</pre></div>
<p>Type specific formatting is also supported, for example avoiding calling <code>strftime()</code>
to format a <code>datetime</code>:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span>
<span class="o">>>></span> <span class="sa">f</span><span class="s2">"Today is </span><span class="si">{</span><span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span><span class="si">:</span><span class="s2">%B %d, %Y</span><span class="si">}</span><span class="s2">"</span>
<span class="s1">'Today is July 15, 2018'</span>
</pre></div>
<p>There's some really cool examples of format specifiers in the
<a href="https://docs.python.org/3.7/library/string.html#format-specification-mini-language">docs</a>
, I'd definitely suggest giving it a read.</p>
<h2>The best of all worlds</h2>
<p>Ultimately, F-strings are really flexible, and to me represent the best of all worlds.
You get the performance of the string formatting operator, the flexibility of the
<code>format()</code> function, and syntactically it all tends to read better as well.</p>Shell Tip Of the Day - Interactively Deleting Docker Images2018-06-08T18:52:00-07:002018-06-08T18:52:00-07:00Adam Parkintag:www.codependentcodr.com,2018-06-08:/shell-tip-of-the-day-interactively-deleting-docker-images.html<p>Shell Tip Of the Day - Interactively deleting Docker images with Bash's select statement</p><p>![Sometimes when you learn about a hammer, everything looks like a nail....]
({static}/static/imgs/hammer_nail.jpg)</p>
<p>(<a href="https://devrant.com/rants/752222/if-all-you-have-is-a-hammer-everything-looks-like-a-nail-this-was-something-whic">source</a>)</p>
<p>Sometimes when you learn about a hammer, everything looks like a nail.... In
<a href="https://www.codependentcodr.com/shell-tip-of-the-day-selecting-untracked-files.html">a previous tip</a> I showed off using
Bash's <code>select</code> statement to interactively select untracked files in a Git repo.</p>
<p>Today I found another use for the <code>select</code> statement -- deleting local Docker
images.</p>
<p>I often will build various Docker images on my machine, and then want to delete
selected ones later. In the past, what I'd do is list all images:</p>
<div class="highlight"><pre><span></span>$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
my-project latest 2d0ff261164e <span class="m">6</span> hours ago 229MB
ubuntu <span class="m">16</span>.04 c9d990395902 <span class="m">8</span> weeks ago 113MB
nginx <span class="m">1</span>.13.11-alpine 2dea9e73d89e <span class="m">2</span> months ago 18MB
ruby <span class="m">2</span>.3 9cc35bb87070 <span class="m">2</span> months ago 723MB
</pre></div>
<p>And then copy the <code>IMAGE ID</code> and delete it:</p>
<div class="highlight"><pre><span></span>$ docker rmi 2d0ff261164e
Untagged: my-project:latest
Deleted: sha256:2d0ff261164e6caf1024f67e702652ccad040b3031bc56c829d810b3e4a3f72b
Deleted: sha256:0144bb04af73ca7e66420954ef1ebd0355739d932e2b3ce4f0d4e852f1e2cb28
Deleted: sha256:798362db6603c5f680029cfd8ec614cb2a4d7a321a28d222b26a4eb8597371b1
</pre></div>
<p>If you have many though this is a lot of tedious copy & pasting. I don't want to
delete <em>all</em> images though, just <em>selected</em> ones. Enter the <code>select</code> statement:</p>
<div class="highlight"><pre><span></span><span class="k">select</span> x <span class="k">in</span> <span class="sb">`</span>docker images --format <span class="s1">'{{.ID}}--{{.Repository}}/{{.Tag}}'</span><span class="sb">`</span> <span class="p">;</span> <span class="k">do</span> docker rmi <span class="sb">`</span><span class="nb">echo</span> <span class="nv">$x</span> <span class="p">|</span> awk -F<span class="s1">'--'</span> <span class="s1">'{print $1}'</span><span class="sb">`</span><span class="p">;</span> <span class="k">done</span>
</pre></div>
<p>Quick dissection: <code>docker images --format '{{.ID}}--{{.Repository}}/{{.Tag}}'</code> will
output the list of local Docker images with their ID, followed by two dashes, followed
by the repository, followed by a slash, followed by the tag. Ie:</p>
<div class="highlight"><pre><span></span>$ docker images --format <span class="s1">'{{.ID}}--{{.Repository}}/{{.Tag}}'</span>
18bfe820e13a--microsoft/dotnet/1.1-sdk
c9d990395902--ubuntu/16.04
2dea9e73d89e--nginx/1.13.11-alpine
9cc35bb87070--ruby/2.3
24ed1c575f81--nginx/1.12-alpine
24ed1c575f81--nginx/1.12.2-alpine
c7fc7faf8c28--alpine/3.4
3fd9065eaf02--alpine/3.7
3fd9065eaf02--alpine/latest
cb178ebbf0f2--python/3.6.0-alpine
</pre></div>
<p>etc. Now we <code>select</code> over this output and this displays a menu of them all:</p>
<div class="highlight"><pre><span></span>$ <span class="k">select</span> x <span class="k">in</span> <span class="sb">`</span>docker images --format <span class="s1">'{{.ID}}--{{.Repository}}/{{.Tag}}'</span><span class="sb">`</span> <span class="p">;</span> <span class="k">do</span> <span class="nb">echo</span> <span class="s2">"</span><span class="nv">$x</span><span class="s2">"</span> <span class="p">;</span> <span class="k">done</span>
<span class="m">1</span><span class="o">)</span> 18bfe820e13a--microsoft/dotnet/1.1-sdk <span class="m">6</span><span class="o">)</span> 24ed1c575f81--nginx/1.12.2-alpine
<span class="m">2</span><span class="o">)</span> c9d990395902--ubuntu/16.04 <span class="m">7</span><span class="o">)</span> c7fc7faf8c28--alpine/3.4
<span class="m">3</span><span class="o">)</span> 2dea9e73d89e--nginx/1.13.11-alpine <span class="m">8</span><span class="o">)</span> 3fd9065eaf02--alpine/3.7
<span class="m">4</span><span class="o">)</span> 9cc35bb87070--ruby/2.3 <span class="m">9</span><span class="o">)</span> 3fd9065eaf02--alpine/latest
<span class="m">5</span><span class="o">)</span> 24ed1c575f81--nginx/1.12-alpine <span class="m">10</span><span class="o">)</span> cb178ebbf0f2--python/3.6.0-alpine
<span class="c1">#?</span>
</pre></div>
<p>However, we can't just do a <code>docker rmi $x</code> on a selected item as <code>docker rmi</code>
takes an id, not an id followed by two dashes, followed by the image name. However,
the argument we need to delete is everything preceding the double-dash. <code>awk</code> to
the rescue:</p>
<div class="highlight"><pre><span></span><span class="k">select</span> x <span class="k">in</span> <span class="sb">`</span>docker images --format <span class="s1">'{{.ID}}--{{.Repository}}/{{.Tag}}'</span><span class="sb">`</span> <span class="p">;</span>
<span class="k">do</span>
docker rmi <span class="sb">`</span><span class="nb">echo</span> <span class="nv">$x</span> <span class="p">|</span> awk -F<span class="s1">'--'</span> <span class="s1">'{print $1}'</span><span class="sb">`</span> <span class="p">;</span>
<span class="k">done</span>
</pre></div>
<p>Dissecting the clunky argument to <code>docker rmi</code>: <code>awk -F'--' '{print $1}'</code> splits
the input string on a double dash and then prints just the first column (the ID
in our case). We then just echo it back to <code>docker rmi</code>. This works, and I
tweeted it at Eric Promislow who was the person who demoed
<a href="https://www.codependentcodr.com/the-2018-vancouver-polyglot-unconference.html">the select statement at Polyglot this year</a>
which was where I first saw the trick:</p>
<!-- markdownlint-disable MD033 -->
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">
<a href="https://twitter.com/ericpromislow?ref_src=twsrc%5Etfw">@ericpromislow</a>
I saw your presentation at <a href="https://twitter.com/PolyglotConf?ref_src=twsrc%5Etfw">@PolyglotConf</a>
about bash & today used the bash select statement I saw there for interactively deleting local Docker
images: <a href="https://t.co/imJV5WA3cB">https://t.co/imJV5WA3cB</a> Thanks for the tip, super-useful!</p>
— Codependent Codr (@CodependentCodr)
<a href="https://twitter.com/CodependentCodr/status/1005153068046954496?ref_src=twsrc%5Etfw">June 8, 2018</a>
</blockquote>
<script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<!-- markdownlint-enable MD033 -->
<p>And they replied with a nice simplification of the clunky awk stuff:</p>
<div class="highlight"><pre><span></span><span class="k">select</span> x <span class="k">in</span> <span class="sb">`</span>docker images --format <span class="s1">'{{.ID}}--{{.Repository}}/{{.Tag}}'</span><span class="sb">`</span> <span class="p">;</span> <span class="k">do</span>
docker rmi <span class="s2">"</span><span class="si">${</span><span class="nv">x</span><span class="p">%--*</span><span class="si">}</span><span class="s2">"</span><span class="p">;</span>
<span class="k">done</span>
</pre></div>
<p>Much nicer. Final version is also in a gist at:
<a href="https://gist.github.com/pzelnip/40da3bf80876c2cdb5809d8a3bd9ee97">https://gist.github.com/pzelnip/40da3bf80876c2cdb5809d8a3bd9ee97</a></p>Shell Tip Of the Day - Selecting untracked files2018-06-01T20:34:00-07:002018-06-01T20:34:00-07:00Adam Parkintag:www.codependentcodr.com,2018-06-01:/shell-tip-of-the-day-selecting-untracked-files.html<p>Shell Tip Of the Day - Selecting untracked files</p><p>Today's tip of the day is a follow up from something I learned at
<a href="https://www.codependentcodr.com/the-2018-vancouver-polyglot-unconference.html">this year's Polyglot UnConference</a> -- how
to use <code>select</code> with the Bash shell for interactive goodness.</p>
<p>At Polyglot I saw an example that looked like this:</p>
<div class="highlight"><pre><span></span><span class="k">select</span> x <span class="k">in</span> * <span class="p">;</span> <span class="k">do</span> stat <span class="nv">$x</span><span class="p">;</span> <span class="k">done</span>
</pre></div>
<p>Try this (go to a terminal and in the bash shell type it verbatim). Go ahead, I'll wait.</p>
<p>Did that? Cool, so as you saw the <code>select</code> keyword is this way of interactively selecting
items from a glob with bash. In that above example, it allowed you to <code>stat</code> a file you
select.</p>
<p>Today I found a practical application of this: interactively deleting untracked files from a
checked out Git repo.</p>
<p>My scenario: I was working on a future blog post and had created a handful of image files in
the current directory. This isn't uncommon: I'll often get a single image that I want to use
for something like the
<a href="http://ogp.me/">OpenGraph</a>
summary image for a post, and then I edit it via
<a href="https://www.imagemagick.org/">ImageMagick</a>
to resize it, change a background to transparent, etc, and then I run it through
<a href="https://github.com/chrissimpkins/Crunch">Crunch</a>
to shrink it down.</p>
<p>Today I found that after getting the final image I wanted, a <code>git status</code> showed a number of
untracked files I didn't care about:</p>
<div class="highlight"><pre><span></span>$ git status
On branch codependentcodr/ch202/vscode-python-debugging-unit-tests-tasks
Untracked files:
<span class="o">(</span>use <span class="s2">"git add <file>..."</span> to include <span class="k">in</span> what will be committed<span class="o">)</span>
content/python-and-vs-code-part1.md
content/static/imgs/vscode141.png
content/static/imgs/vscodeAndPython.png
content/static/imgs/vscodelogo-crunch.png
content/static/imgs/vscodelogo.png
nothing added to commit but untracked files present <span class="o">(</span>use <span class="s2">"git add"</span> to track<span class="o">)</span>
</pre></div>
<p>The only file I wanted from <code>content/static/imgs</code> was <code>vscodeAndPython.png</code>, all the other stuff in
<code>content/static/imgs</code> was junk that could be deleted. I could manually delete
each one, typing the full filename each time, or I could save some keystrokes by
using <code>select</code> and a little command-line fu:</p>
<div class="highlight"><pre><span></span><span class="k">select</span> x <span class="k">in</span> git status --porcelain <span class="p">|</span> tr -d <span class="se">\?\?</span><span class="p">;</span> <span class="k">do</span> rm <span class="nv">$x</span> <span class="p">;</span> <span class="k">done</span>
</pre></div>
<p>Explanation: <code>git status --porcelain</code> just spits out the filenames from a <code>git status</code>
(no header/footer, extra stuff), and <code>tr -d \?\?</code> removes the leading <code>??</code> that's
prepended to any filename that is untracked. The <code>select</code> iterates over each file
matched by this command in an interactive way, and then the <code>do rm $x</code> removes whatever
file I select. This allowed me to interactively delete files that were untracked:</p>
<div class="highlight"><pre><span></span>$ <span class="k">select</span> x <span class="k">in</span> <span class="sb">`</span>git status --porcelain <span class="p">|</span> tr -d <span class="se">\?\?</span><span class="sb">`</span><span class="p">;</span> <span class="k">do</span> rm <span class="nv">$x</span> <span class="p">;</span> <span class="k">done</span>
<span class="m">1</span><span class="o">)</span> content/python-and-vs-code-part1.md <span class="m">4</span><span class="o">)</span> content/static/imgs/vscodelogo-crunch.png
<span class="m">2</span><span class="o">)</span> content/static/imgs/vscode141.png <span class="m">5</span><span class="o">)</span> content/static/imgs/vscodelogo.png
<span class="m">3</span><span class="o">)</span> content/static/imgs/vscodeAndPython.png
<span class="c1">#? 2</span>
<span class="c1">#? 4</span>
<span class="c1">#? 5</span>
$ git status
On branch codependentcodr/ch202/vscode-python-debugging-unit-tests-tasks
Untracked files:
<span class="o">(</span>use <span class="s2">"git add <file>..."</span> to include <span class="k">in</span> what will be committed<span class="o">)</span>
content/python-and-vs-code-part1.md
content/static/imgs/vscodeAndPython.png
nothing added to commit but untracked files present <span class="o">(</span>use <span class="s2">"git add"</span> to track<span class="o">)</span>
</pre></div>
<p>I thought this was pretty cool, next time you find yourself wanting to choose from
a number of files on the command line, think about
<a href="http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_09_06.html"><code>select</code></a>. 😄</p>
<p>Ok: I realize I could've just added the md file & the image I wanted to git and then
deleted all untracked files, but would that have been as interesting as this blog
post? I think not.</p>The 2018 Vancouver Polyglot Unconference2018-05-26T07:54:00-07:002018-05-26T07:54:00-07:00Adam Parkintag:www.codependentcodr.com,2018-05-26:/the-2018-vancouver-polyglot-unconference.html<p>I got to go back yet again for the 7th Annual Polyglot Unconference in Vancouver. Let's recap.</p><p>This year, like many past years (see <a href="https://www.codependentcodr.com/the-polyglot-unconference-2012.html">here</a> and
<a href="https://www.codependentcodr.com/the-2017-vancouver-polyglot-unconference.html">here</a>), I was fortunate enough to be able to make the trip to
YVR for the annual <a href="https://www.polyglotconf.com">Polyglot Unconference</a>. This event, now
in it's 7th year, has been a favourite of mine for some time as it's a great (and very
affordable) opportunity to network with a bunch of other technologists, hear about new
and different technologies, and just get inspired and excited about the work we do in software
development. It's "An open space for people who create software and love it".</p>
<p>Like prior years I thought I'd recap what I saw.</p>
<h1>General Themes In Pitches</h1>
<p>For those unaware, the format of Polyglot is an unconference, where at the start of the day people
submit "pitches" for topics they'd like to see discussed. After that, attendees dot vote on the
topics they'd like to see scheduled into timeslots, and the organizers then facilitate the spaces
for those talks to happen. Some of the topics suggested included:</p>
<ul>
<li>Jyupiter</li>
<li>Continuous Integration</li>
<li>Bash tricks</li>
<li>CSS</li>
<li>Design exercise</li>
<li>Project/Product Management</li>
<li>Quantum Computing</li>
<li>Code Review</li>
<li>Awk: sometimes it's all you need</li>
<li>Career development</li>
<li>Moving Application to Redux</li>
<li>Go Interface Antipatterns</li>
<li>Scourge of Hybrid APIs</li>
<li>Supporting Sustaining Community Meetups</li>
<li>Blockchain</li>
<li>AWS Lessons Learned</li>
<li>AWS AWS AWS, what about Azure & GC?</li>
<li>Home Automation</li>
<li>How Do you Learn New Programming Languages?</li>
<li>Current state of PWAs vs Hybrid Apps</li>
<li>Building Modern Web Apps, module bundling, splitting, CSS loading, etc</li>
<li>GraphQL - the awkward parts</li>
<li>Email address & Domain names are non-latin, how to handle?</li>
<li>Component Libraries, how do?</li>
<li>Developing Mental Models</li>
<li>JS with Chris, just the language</li>
<li>Technical Design Review</li>
<li>So you thought you were writing fast Scala....</li>
<li>UI & UX, & event storming</li>
<li>How to scare away, demoralize everyone in your remote office</li>
<li>Open source a company project</li>
<li>Project Management Design</li>
<li>Designing Learning systems online</li>
<li>Effective Learning Strategies in the Workplace</li>
<li>Pair Programming: when to do, not do, how to measure success</li>
<li>Webassembly</li>
<li>GDPR Lawsuits, data privacy, legislation, etc</li>
<li>Unicode</li>
<li>Software quality</li>
<li>Software interviewing</li>
<li>ORMs are evil, but SQL is awesome</li>
</ul>
<p>Often I find it interesting to sit and listen to the pitches as there tends to be "themes" that
come up that are indicative of trends in the industry. This year however, was quite different,
the sheer breadth of topics was massive, with very little
in the way of overlapping topics. Take that as you will, was that reflective of the fact that the
number of technologies employed today is broader than ever, or just coincidence, I don't know.</p>
<h1>Talks I Attended</h1>
<h2>One Simple Trick To Make You A Better Developer... Code Review</h2>
<p>This was a rather lively and deep discussion on the value of code review facilitated by
<a href="https://twitter.com/johnboxall">John Boxall</a>, in which they started with a short presentation
of their experience going from developer working in isolation to a team, and how looking at
other people's code forced him to learn a ton. Some of the insights they shared from stuff
they've done at <a href="https://www.mobify.com/">Mobify</a> included:</p>
<h3>What Things Should Reviewers Look For</h3>
<ul>
<li>does this change deliver biz value?<ul>
<li>does this change make sense?</li>
<li>is this what we actually want to do?</li>
</ul>
</li>
<li>does this change work?<ul>
<li>depending on context, it might be ok for some parts to be incomplete</li>
</ul>
</li>
<li>is this change maintainable?</li>
</ul>
<p>This <strong>is an ordered</strong> list.</p>
<h3>How To Give Feedback</h3>
<ul>
<li>need to create a safe space, reviewer isn't a gatekeeper, review should be collaborative</li>
<li>give feedback about things that could be done better</li>
<li>(often ignored) great opportunity to learn, if you do, tell the author!<ul>
<li>break out the emoji's & giphy's!</li>
</ul>
</li>
<li>review the code, not the coder</li>
<li>assume positive intent (developer had good intentions)</li>
<li>welcoming, considerate, and respectful</li>
<li>be conscious of the language you're using</li>
</ul>
<h3>Antipatterns</h3>
<ul>
<li>PR is too big<ul>
<li>you get the same amount of feedback regardless of size</li>
<li>break into atomic steps, ~300 lines rule of thumb</li>
</ul>
</li>
<li>Rewrites<ul>
<li>feedback results in rewrite, should've solicited more early feedback</li>
</ul>
</li>
<li>No differentiation between blocking & non-blocking feedback<ul>
<li>"this is blocking for me until this is resolved"</li>
</ul>
</li>
<li>Blocking disagreements<ul>
<li>bring in 3rd party to collaborate</li>
<li>go talk to that person</li>
<li>"synchronous code review"</li>
</ul>
</li>
<li>Reviewer changes author's code</li>
</ul>
<p>The presentation then mophed into a lively fishbowl discussion, topics discussed
included:</p>
<ul>
<li>One company's process was: 2 approvals, review everything, done within 24 hours</li>
<li>Ask: does this change create value that offsets the debt that this change creates?</li>
<li>Challenges with "infrastructure as code" review</li>
<li>Is CR different for open source? (ex: correctness over expediency/cost reduction)</li>
<li>The best teams try to complete review within a day</li>
<li>As author, when you're looking for specific feedback, call it out in the PR</li>
<li>As reviewer, ask questions</li>
<li>Sometimes too much responsibility is put on the code review<ul>
<li>try to catch issues earlier</li>
<li>requirements design, design review, etc</li>
</ul>
</li>
<li>Having a code review "coach", or meta review of the code review process itself</li>
<li>Static code analysis to help decrease the "nitpickyness" of code reviews</li>
<li>Use tools to enforce team agreements</li>
</ul>
<p>Great discussion overall.</p>
<h2>Learning new programming languages; how do</h2>
<p>This is a topic right at home at a conference all about "multiple languages" (ie polyglot). Some
topics discussed included:</p>
<ul>
<li>How to balance learning a new language for work vs personal interest vs future employment?</li>
<li>The value of learning something completely different, languages with different perspectives</li>
<li>Try to get feedback wherever you can when you're learning a new language</li>
<li>The difficulty of frameworks getting in the way of learning the language (esp Javascript)</li>
<li>Related, try to learn without all the bells and whistles first</li>
<li>Think about and learn that one thing does the language you're learning do differently</li>
<li>Find a pet project you're familiar with, implement in different language</li>
<li>How do you know when you are truly "comfortable" in a language?</li>
</ul>
<p>Much of the discussion in this presentation focussed heavily on language idioms, perhaps to the
detriment of the discussion as a whole.</p>
<p>Interesting reflection: as my career has progressed I've gone from learning a lot of languages
to losing interest in learning languages for the sake of learning them. I think that's partly
(perhaps largely) because I've reached the point where (call it experience, call it cynicism)
I see the same patterns in languages that I've seen before. I recall learning functional
programming being a true paradigm shift for me, I recall a similar experience learning OO at
a deep level, but now I find many of the "new" things I see feel more like syntactic sugar
or different ways of expressing the same thing, rather than empowering me to think differently
about problems.</p>
<p>For example, consider
<a href="https://medium.com/@magnus.chatt/why-you-should-totally-switch-to-kotlin-c7bbde9e10d5">this article about Kotlin</a>.
Look at that list of reasons to learn Kotlin, all of those I've seen in other languages, for example
"Java Interoperability" is a feature of languages like Scala and Clojure, "String interpolation" is
something I saw for the first time in Perl years ago, and now is prominent in Python & Ruby. Type
inference was a hallmark innovation of the ML family of languages, and also appears in C#. This
isn't intended a criticism of Kotlin (I'm sure it's a perfectly fine language), but is
language design now just a game of shuffling already-seen ideas in different combinations?</p>
<p>In any case, this is more rant than reflection on the event, so let's jump ahead to the next
discussion I was at.</p>
<h2>Developing Mental Models, Helping Developers Build Mental Maps</h2>
<p>This talk was a bit... existential. The talk was pitched by one of the conference organizers
(<a href="https://twitter.com/saemg">Saem</a>) who, as a tech manager, has noticed that often the difference
between highly effective developers and less effective ones is that the former tend to have
greater success developing clear mental models of computation. Interesting topic, and the
(lively) discussion delved into discussions around learning theory, how people learn and
perceive the world, at what point do you truly "know something", etc. Some deep, interesting
subject matter, but unfortunately I'm not sure I took many concrete takeaways away from the discussion.</p>
<h2>Home Automation</h2>
<p>I went to this one hoping to see a bit of cool or novel applications of home automation. As
someone who's dabbling into the world of <a href="https://www.amazon.com/Amazon-Echo-And-Alexa-Devices">Alexa</a>
and devices like <a href="https://www2.meethue.com/en-us">Philips Hue</a>, this is a topic of interest
for me.</p>
<p>As it turned out, the part of the discussion I was around for ended up being more on the geeky
DIY side of things, talking about cheap devices to build out home automation systems. Less of
interest to me, so I exercised the law of two feet to go to another talk.</p>
<h2>Everything You Wanted To Know About Bash Command Line History But Were Afraid To Try</h2>
<p>This was really "fun tricks with Bash".
<a href="https://docs.google.com/presentation/d/1S3lGp7TVZjZqwnWHUpiAG5a0mS7ZJLdAOIZcw3PeNeE/edit#slide=id.g58b3ab78d_028">The slides</a>
have some great little power tips on cool
stuff you can do with the Bash shell. Many of these things I'd never seen before, so cool
to see.</p>
<p>For example, a neat trick is to use "bang dollar" to match the last argument
from the last executed command. Say for example, you're working in a Git repo
and have changed a file. You might do a <code>git diff</code> to see what's changed, and
then having coherence checked the diff, want to add that file to be committed.
You'd do this with something like:</p>
<div class="highlight"><pre><span></span>git diff some/subdir/containing/somefile.txt
git add some/subdir/containing/somefile.txt
</pre></div>
<p>But that's a lot of extra typing so you can use "bang dollar" to shorten it:</p>
<div class="highlight"><pre><span></span>git diff some/subdir/containing/somefile.txt
git add !$
</pre></div>
<p>Much more concise.</p>
<p>Another tip, say you want to convert <code>foo/bar/baz.png</code> to <code>foo/bar/baz.gif</code> using
<a href="http://www.imagemagick.org/">Imagemagick's</a> <code>convert</code> command:</p>
<div class="highlight"><pre><span></span>convert foo/bar/baz.png foo/bar/baz.gif
</pre></div>
<p>But again, can shorten with the magic of Bash:</p>
<div class="highlight"><pre><span></span>convert foo/bar/baz.<span class="o">{</span>png,gif<span class="o">}</span>
</pre></div>
<p>Neat stuff, and the slides are full of little gems like this. Check them out at:
<a href="https://docs.google.com/presentation/d/1S3lGp7TVZjZqwnWHUpiAG5a0mS7ZJLdAOIZcw3PeNeE/edit#slide=id.g58b3ab78d_028">https://docs.google.com/presentation/d/1S3lGp7TVZjZqwnWHUpiAG5a0mS7ZJLdAOIZcw3PeNeE/edit#slide=id.g58b3ab78d_028</a></p>
<h2>Javascript (or JS For Legal Reasons); What's Going On What's New, But Not About Frameworks</h2>
<p><a href="https://twitter.com/chrismnicola">Chris Nicola</a>, another of the organizers, yearly will do a "Javascript
State of the Union" discussion where the goal is to talk about all the newfangled frameworks
popping up in the JS space. This year, they changed it up a bit, instead focussing on plain
Javascript the language, rather than the sampling of the new shiny stuff.</p>
<p>One thing they noted was how it seemed odd that the famous
<a href="https://www.amazon.ca/JavaScript-Good-Parts-Douglas-Crockford/dp/0596517742">Javascript the Good Parts book by Doug Crockford</a>
is widely regarded as one of the best texts on the language, yet seems to be largely
ignored by many framework authors. With this in mind, they started talking about how their company is
taking a step back and starting to use just plain Javascript like a real programming language, and
to bring all the good software design and engineering principles (SOLID, DRY, reduce coupling, etc)
to their JS work.</p>
<p>Their approach is to treat Javascript more like a functional language than a OO language. That is,
to decompose units of computation down to tiny functions and use dependency injection via arguments
to those functions to wire up functionality. This has all the happy benefits you'd expect: code
that's easier to read, to compose, and to test.</p>
<p>As someone who's lived their life in the backend, programming language & software engineering side
of things, this really resonated with me. One of the troubles I've had with trying to learn JS
is just A) figuring out which framework to start with, B) figure out that framework, and then
C) realize that by the time I've figured that out the framework's now obsolete and none of the
knowledge acquired is transferrable. I think it might be useful for me to take a step back and
focus on JS just as a plain C-derivative language and approach learning it from that perspective.
Good takeaway for me.</p>
<p>Last gem was a book recommendation, JavaScript Allongé: <a href="https://leanpub.com/javascriptallongesix/read">https://leanpub.com/javascriptallongesix/read</a>
Sounded like it was a book that helped their team start on this journey towards stripping JS down
to the bare essentials, so I'll definitely be checking it out.</p>
<p>The discussion at this point moved more to a discussion around using Javascript for distributed
computing, as there was another attendee who was interested in learning more about that space.
This was less of interest/applicability to me, so again, law of two feet to...</p>
<h2>Software Quality, And Why It's Important</h2>
<p>Popped into this one a bit late, but this was a roundtable discussion of people talking about
testing, quality, how to test, what to test, what makes for good code, the importance of many
layers of tests (unit, integration, etc), agile practices and their role in quality, etc.
Lots of good discussion.</p>
<h1>Meta Thoughts About The Event</h1>
<p>One of the things that impresses me about the event is the fact that each year the organizers
find ways to tweak & improve the event. This year there were a couple of changes.</p>
<h2>The App</h2>
<p>This year rather than just papers on a wall & dot voting, there was an app built by one of the
organizers to allow the dot voting to happen online rather than the mob around the post board.
A screenshot of how it looked while voting was going on is below:</p>
<p><img alt="Dot voting online!" src="https://www.codependentcodr.com/static/imgs/polyglot2018_app_sm-crunch.png"></p>
<p>It really worked quite well, was much more organized than the paper board, and it made it
much easier to figure out which talk to go to.</p>
<h2>Sharing Is Caring</h2>
<p>In the opening presentation there was emphasis put on sharing the space, and not letting people
dominate conversations. I really appreciated this, as there had definitely been times in prior years
where I saw one person who would dominate or hijack a presentation, at the expense of others.</p>
<h1>Closing Thoughts</h1>
<p>As it always is, Polyglot was a great event. I found this year quite different than previous
years though. Typically I hear of something I've never heard of before (for example, the first
time I heard of this language called Go was at Polyglot, the first time I heard of the testing
pyramid was at Polyglot, etc). This year though, there was no new tech or thing I hadn't seen
before that I walked away with as a thing to look into. I'm not quite sure why that is,
perhaps where I'm at in my career, perhaps the state of the industry (are we maybe finally
slowing down just a smidge?), or just an artifact of the talks I happened to attend. Still
though a great day of learning and as always I find myself heading back home with my passion
for the craft of software development renewed.</p>
<p>Many thanks to the organizers who continue to put on a phenomenal (and very affordable)
event year after year. See you all in 2019!</p>DevopsDays YYJ - Year One2018-05-25T17:56:00-07:002018-05-25T17:56:00-07:00Adam Parkintag:www.codependentcodr.com,2018-05-25:/devopsdays-yyj-year-one.html<p>I was fortunate enough to attend the inaugural DevOps Days Victoria. Let's recap what I saw.</p><p>This year I was fortunate enough to get to attend the very first
<a href="https://www.devopsdays.org/events/2018-victoria/welcome/">DevOps days here in beautiful Victoria, BC</a>.
I thought I'd write up a quick post on what I saw and some of the reasons why I thought the event was great.</p>
<p>So for those unaware, the <a href="https://www.devopsdays.org/">Devops Days conferences</a> are held all
around the world and give technical practitioners a place to come together and share stories and lessons
learned in the DevOps space. This was the first time I was able to attend a DevOps Days event, so going into it I
wasn't really sure what to expect. The format the event took was to have two sets of talks on various
topics in and around the devops movement. Everything from deep technical talks about AWS services to
human stories about how to not burn out your employees from on-call rotations. The full schedule is at
<a href="https://www.devopsdays.org/events/2018-victoria/program/">https://www.devopsdays.org/events/2018-victoria/program/</a></p>
<p>Let's give a quick recap of the talks I got to sit in on.</p>
<h1>Talks Attended</h1>
<h2>Serverless - Jeffery Grajkowski</h2>
<p>Jeff works at a local company here in town called <a href="https://giftbit.com">Giftbit</a>, and has created a
nice little framework for getting an AWS Lambda powered project off the ground. You can find the repo
at <a href="https://github.com/Giftbit/sam-scaffold">https://github.com/Giftbit/sam-scaffold</a>.</p>
<p>Was a nice introduction to a good way to get started with <a href="https://aws.amazon.com/lambda/">AWS Lambda</a>
If you're thinking about playing with Lambda I'd definitely recommend checking it out as a way of
getting up and running quickly.</p>
<p>Side note, I've met Jeff a few times at social events (notably
<a href="https://whiskydev.com/">Whisky Orientated Development</a>) which he
co-organizes, and I'd definitely recommend that event as well as it's a lot of fun.</p>
<h2>Taming Infrastructure Workflow at Scale - Anubhav Mishra</h2>
<p>Anubhav works for <a href="https://www.hashicorp.com">Hashicorp</a>, well known for many tools in the DevOps
space such as Vagrant, Terraform, Vault, and others.</p>
<p>The talk started with a bit of a history lesson on how operations work has evolved over the last
10 years or so, going from physical servers to virtualization, to the cloud, etc. They then dove
in to an overview of <a href="https://www.terraform.io/">Terraform</a> which is a really great tool for provisioning
infrastructure via code. They then concluded with a quick demo of using Terraform to provision a
webserver in Google Cloud with a DNS entry provisioned in AWS via Route 53. Simple, but really
was a nice little overview of the kind of stuff that's possible with Terraform.</p>
<h2>Beyond Tools: A Human Approach to DevOps - Eduardo Augusto Alves Camargo</h2>
<p>One of the things that I think the best conferences do is to have both talks that have deep
technical insights as well as "human" stories. Software development & deployment is very much
about both, and unfortunately some events (and attendees) can focus too much on the former at
the expense of the latter.</p>
<p>I think this is why I really enjoyed Eduardo's talk. Eduardo works for
<a href="https://www.daitangroup.com">Daitan Group</a>
and they gave a description of their company's journey through DevOps transformation, not from
a tools & automation perspective, but on the perspective of the human focus. Discussions of
the importance of empathy and communication, the challenges of collaborating with people from
very different cultures, and some of the lessons learned along the way. Really inspiring,
and I enjoyed it a great deal.</p>
<h2>Distributed Brute Force Login Attack - Peter Locke</h2>
<p>Peter, like Jeff who did the Serverless talk earlier in the day, also works at
<a href="https://www.giftbit.com">Giftbit</a>. In this talk they delved into how they've had to deal with
distributed brute force login attacks where distributed botnets try to attack a login page
with leaked credentials trying to compromise accounts on their service.</p>
<p>Great overview of some of the technical challenges associated with the problem they've faced,
and how they've to date leveraged a number of technologies in AWS (ex:
<a href="https://aws.amazon.com/lambda/">Lambda</a>,
<a href="https://aws.amazon.com/athena/">Athena</a>,
<a href="https://aws.amazon.com/cloudfront/">Cloudfront</a>, etc)
to try and combat it. Some great discussion afterwards about alternative techniques that
could potentially be used was had.</p>
<h2>Bitrot: A Story of Maintenance Failure - Will Whittaker</h2>
<p>This talk was just funny. Will's been in the industry for some time, and told the story
of a project they were a part of that started in the early 2000's, that they left, and came back
to and saw how the project had devolved in that time. Lots of humourous, cynical anecdotes
about the horrors of maintaining a system for a long time.</p>
<p>An interesting insight that hadn't occurred to me was a word of caution about how containers,
which are currently sweeping the industry as the (relatively) new hot way of deploying our
applications will likely become a maintenance nightmare years from now. Interesting stuff.</p>
<h2>Fixing Production in the Hospital</h2>
<p>This again was one of the "human-side" talks of the day. Unfortunately the schedule doesn't
have the speaker's name, and I didn't make a note of it, but the presenter told the story of
how while working at a company as the head of the ops team, was in the hospital for the birth
of their third son when they got a phone call from the CTO telling him that everything was on
fire and they needed to fix it. Inspiring story of the cost of siloing from a human perspective.
Lots of discussion on techniques they employed to help improve their culture & process over
time (blameless post-mortems, release planning, etc).</p>
<h1>Reflections On The Event</h1>
<p>All-in-all, particularly for a first time, DevOps Days YYJ was a great event. More importantly,
it was great to see the Victoria tech community come together in a day of learning, as while
the Victoria tech scene has grown dramatically in the last few years, and there's a
<a href="https://joinyyjtechslack.herokuapp.com/">thriving Slack group</a>
there to date haven't been many conference events
here in town (<a href="https://www.startupslam.io/">StartupSlam</a> being the only other I can think of).</p>
<p>Would definitely recommend the event in the future to anyone in or around the Devops space
in Victoria.</p>
<p>Tomorrow conference season continues for me as I'll be attending the 7th annual
Polyglot Unconference in Vancouver, a favourite event of mine. I'll likely blog about what
I see there, but you can also read about what I've seen in prior years in my posts from
prior years <a href="https://www.codependentcodr.com/the-polyglot-unconference-2012.html">here</a> and <a href="https://www.codependentcodr.com/the-2017-vancouver-polyglot-unconference.html">here</a>.</p>Pull Requests & Code Review2018-05-12T07:45:00-07:002018-05-12T07:45:00-07:00Adam Parkintag:www.codependentcodr.com,2018-05-12:/pull-requests-code-review.html<p>A couple links related to pull requests and reviewing them, and things to do and/or avoid.</p><p><a href="https://xkcd.com/1513/"><img alt="XKCD ‘Code Quality’, copied under CC BY-NC 2.5" src="https://www.codependentcodr.com/static/imgs/code_quality_2x.png">
XKCD ‘Code Quality’, copied under CC BY-NC 2.5</a></p>
<p>Today I have a few links related to pull requests and doing code review. I've definitely
done my fair share of both in my career, and yet I still find it's one of those "soft"
skills that you can always hone & improve. Many of these links have been around for
awhile, but are definitely worth reading for anyone who has to collaborate in code with
others on a repo.</p>
<p>The first is
<a href="https://blog.github.com/2015-01-21-how-to-write-the-perfect-pull-request/">How to Write the Perfect Pull Request from Github</a>.
This is an overview of good
advice on guidelines around what to do when you open that PR from your forked copy to some
open source project. Some of the little tips around conventions like the use of <code>WIP</code> to
indicate you want some early "20%" feedback, etc. Unsurprisingly it is a bit geared
towards what you'd see in projects on Github, but generally speaking the advice is
applicable anywhere you have to create pull requests for others to review.</p>
<p>The article
also goes into discussion of how to respectfully both review a PR as well as receive
feedback given. This is worth considering as it's not uncommon for code reviews to become "heated" and/or
toxic.</p>
<p>This leads me to my next link:
<a href="https://medium.freecodecamp.org/unlearning-toxic-behaviors-in-a-code-review-culture-b7c295452a3c">Unlearning Toxic Behaviours In A Code Review Culture</a>
This is a much more detailed and deep discussion of the little bad habits that many (myself
included) can unfortunately bring to code reviews and have them turn adversarial rather
than collaborative. One of the things I love about Sandya's article is that it shines a
light on bad habits that are common, particularly amongst experienced developers. I've
definitely been guilty of passing off opinion as fact as well as bombarding a review with
an avalanche of comments. As well, they also not only points out some of the "bad" (or
toxic) behaviours but offers constructive practices. Really, really good stuff.</p>
<p>The last one I have is more lightweight:
<a href="https://smartbear.com/learn/code-review/best-practices-for-peer-code-review/">Best Practices for Code Review</a>
Not super deep, but a good summary, and I really like the emphasis on not overwhelming
yourself with reviews & recognizing that reviewing code takes a great deal of energy.</p>
<p>With these articles in mind I've both adjusted my habits when I do review as well as when
I put code up for review by others. I've also tweaked the PR template for the repo for
this blog.</p>S3 Redirection Rules2018-05-07T20:45:00-07:002018-05-07T20:45:00-07:00Adam Parkintag:www.codependentcodr.com,2018-05-07:/s3-redirection-rules.html<p>How to use S3 redirection rules for simple redirecting to other hosts</p><p>So as mentioned in many of my other posts, this site is served via an <a href="http://aws.amazon.com/s3">S3</a>
bucket. Recently I decided I'd like to have certain paths off of my domain (www.codependentcodr.com)
be automatically redirected to other URL's that aren't in the S3 bucket. As a concrete example, I
wanted to be able to set up redirections from <code>/twitter</code> to my Twitter account
(<a href="https://www.twitter.com/codependentcodr">https://www.twitter.com/codependentcodr</a>).</p>
<p>As a first stab, I created an HTML file with a simple <code><meta></code> tag to redirect the user to a particular
URL. This looked something like:</p>
<div class="highlight"><pre><span></span><span class="cp"><!DOCTYPE html></span>
<span class="p"><</span><span class="nt">html</span> <span class="na">lang</span><span class="o">=</span><span class="s">"en"</span><span class="p">></span>
<span class="p"><</span><span class="nt">head</span><span class="p">></span>
<span class="p"><</span><span class="nt">title</span><span class="p">></span>Twitter Redirect<span class="p"></</span><span class="nt">title</span><span class="p">></span>
<span class="p"><</span><span class="nt">meta</span> <span class="na">http-equiv</span><span class="o">=</span><span class="s">"refresh"</span> <span class="na">content</span><span class="o">=</span><span class="s">"0; url=https://twitter.com/codependentcodr"</span> <span class="p">></span>
<span class="p"></</span><span class="nt">head</span><span class="p">></span>
<span class="p"></</span><span class="nt">html</span><span class="p">></span>
</pre></div>
<p>This was then uploaded to the S3 bucket as <code>twitter.html</code>, so going to
<a href="https://www.codependentcodr.com/twitter.html">https://www.codependentcodr.com/twitter.html</a> redirected you to my Twitter page. Success, but
ugly as:</p>
<ol>
<li>it was <code>/twitter.html</code> not <code>/twitter</code></li>
<li>it felt really clunky to have a file that existed just to redirect someone</li>
</ol>
<p>I could get around the first problem by renaming the HTML file to <code>twitter</code> instead of <code>twitter.html</code>,
but then I'd have to go into S3 after uploading the file and manually set the <code>Content-Type</code> to
<code>text/html</code> otherwise browsers would see it as a binary file and not interpret it as HTML.</p>
<p>There had to be a better way. So I reached out to my colleagues in the
<a href="https://joinyyjtechslack.herokuapp.com/">YYJ Tech Slack community</a> (specifically the #devops channel)
asking if anyone had insights. I was quickly pointed to
<a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/how-to-page-redirect.html">this page</a> and specifically
the section on that page titled "Advanced Conditional Redirects".</p>
<p>The idea here is you can write some XML which defines some redirect rules for when a key to an object
in the S3 bucket is requested, but doesn't exist. As a basic example, say you wanted requests to
<code>http://whatever.your.bucket.is.called.s3-website.ca-central-1.amazonaws.com/foo</code> to redirect
to <a href="https://github.com/doge">https://github.com/doge</a>, you might create a rule like:</p>
<div class="highlight"><pre><span></span><span class="nt"><RoutingRules></span>
<span class="nt"><RoutingRule></span>
<span class="nt"><Condition></span>
<span class="nt"><KeyPrefixEquals></span>foo<span class="nt"></KeyPrefixEquals></span>
<span class="nt"></Condition></span>
<span class="nt"><Redirect></span>
<span class="nt"><HostName></span>github.com<span class="nt"></HostName></span>
<span class="nt"><ReplaceKeyWith></span>doge<span class="nt"></ReplaceKeyWith></span>
<span class="nt"></Redirect></span>
<span class="nt"></RoutingRule></span>
<span class="nt"></RoutingRules></span>
</pre></div>
<p>Add this to the Redirection Rules on the properties for your bucket (under Static Website Hosting),
and voila, now going to <code>/foo</code> on your bucket sends people to the doge.</p>
<p>The docs outline all the different options that are available, this is just a simple example that
was exactly what I needed. With this in place I set up the following redirects:</p>
<table>
<thead>
<tr>
<th>Path</th>
<th>Redirects To</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><a href="https://www.codependentcodr.com/youtube">https://www.codependentcodr.com/youtube</a></td>
<td><a href="http://youtube.codependentcodr.com">http://youtube.codependentcodr.com</a></td>
<td>CNAME that points to my Youtube Channel</td>
</tr>
<tr>
<td><a href="https://www.codependentcodr.com/twitter">https://www.codependentcodr.com/twitter</a></td>
<td><a href="http://twitter.com/codependentcodr">http://twitter.com/codependentcodr</a></td>
<td>My Twitter profile</td>
</tr>
<tr>
<td><a href="https://www.codependentcodr.com/stackoverflow">https://www.codependentcodr.com/stackoverflow</a></td>
<td><a href="http://stackoverflow.com/users/808804">http://stackoverflow.com/users/808804</a></td>
<td>My Stackoverflow Profile page</td>
</tr>
<tr>
<td><a href="https://www.codependentcodr.com/github">https://www.codependentcodr.com/github</a></td>
<td><a href="http://github.com/pzelnip">http://github.com/pzelnip</a></td>
<td>My Github profile page</td>
</tr>
<tr>
<td><a href="https://www.codependentcodr.com/linkedin">https://www.codependentcodr.com/linkedin</a></td>
<td><a href="http://lnkd.in/ykHQiG">http://lnkd.in/ykHQiG</a></td>
<td>The place for recruiters to reach me at 😉</td>
</tr>
</tbody>
</table>
<p>The full set of routing rules that makes these possible is here:</p>
<div class="highlight"><pre><span></span><span class="nt"><RoutingRules></span>
<span class="nt"><RoutingRule></span>
<span class="nt"><Condition></span>
<span class="nt"><KeyPrefixEquals></span>twitter<span class="nt"></KeyPrefixEquals></span>
<span class="nt"></Condition></span>
<span class="nt"><Redirect></span>
<span class="nt"><HostName></span>www.twitter.com<span class="nt"></HostName></span>
<span class="nt"><ReplaceKeyWith></span>codependentcodr<span class="nt"></ReplaceKeyWith></span>
<span class="nt"></Redirect></span>
<span class="nt"></RoutingRule></span>
<span class="nt"><RoutingRule></span>
<span class="nt"><Condition></span>
<span class="nt"><KeyPrefixEquals></span>youtube<span class="nt"></KeyPrefixEquals></span>
<span class="nt"></Condition></span>
<span class="nt"><Redirect></span>
<span class="nt"><HostName></span>youtube.codependentcodr.com<span class="nt"></HostName></span>
<span class="nt"><ReplaceKeyWith></ReplaceKeyWith></span>
<span class="nt"></Redirect></span>
<span class="nt"></RoutingRule></span>
<span class="nt"><RoutingRule></span>
<span class="nt"><Condition></span>
<span class="nt"><KeyPrefixEquals></span>stackoverflow<span class="nt"></KeyPrefixEquals></span>
<span class="nt"></Condition></span>
<span class="nt"><Redirect></span>
<span class="nt"><HostName></span>stackoverflow.com<span class="nt"></HostName></span>
<span class="nt"><ReplaceKeyWith></span>/users/808804<span class="nt"></ReplaceKeyWith></span>
<span class="nt"></Redirect></span>
<span class="nt"></RoutingRule></span>
<span class="nt"><RoutingRule></span>
<span class="nt"><Condition></span>
<span class="nt"><KeyPrefixEquals></span>github<span class="nt"></KeyPrefixEquals></span>
<span class="nt"></Condition></span>
<span class="nt"><Redirect></span>
<span class="nt"><HostName></span>github.com<span class="nt"></HostName></span>
<span class="nt"><ReplaceKeyWith></span>/pzelnip<span class="nt"></ReplaceKeyWith></span>
<span class="nt"></Redirect></span>
<span class="nt"></RoutingRule></span>
<span class="nt"><RoutingRule></span>
<span class="nt"><Condition></span>
<span class="nt"><KeyPrefixEquals></span>linkedin<span class="nt"></KeyPrefixEquals></span>
<span class="nt"></Condition></span>
<span class="nt"><Redirect></span>
<span class="nt"><HostName></span>lnkd.in<span class="nt"></HostName></span>
<span class="nt"><ReplaceKeyWith></span>/ykHQiG<span class="nt"></ReplaceKeyWith></span>
<span class="nt"></Redirect></span>
<span class="nt"></RoutingRule></span>
<span class="nt"></RoutingRules></span>
</pre></div>
<p>Easy peasy. And as you can see from the links in the table, these redirection rules
work going through my Cloudfront distribution attached to my custom domain.</p>Recording Blogging Content2018-05-05T09:36:00-07:002018-05-05T09:36:00-07:00Adam Parkintag:www.codependentcodr.com,2018-05-05:/recording-blogging-content.html<p>Recording my process for writing blog content</p><p>So I thought I'd live record myself writing a blog post to show off my toolchain.
You can see the recording below:</p>
<!-- markdownlint-disable MD033 -->
<iframe width="560" height="315" src="https://www.youtube.com/embed/i-SZXwAnUGg" frameborder="0"
allow="autoplay; encrypted-media" allowfullscreen></iframe>
<!-- markdownlint-enable MD033 -->
<p>This is snazzy!</p>Failure Bow #1 - The AWS Lambda Goof2018-05-05T07:42:00-07:002018-05-05T07:42:00-07:00Adam Parkintag:www.codependentcodr.com,2018-05-05:/failure-bow-1-the-aws-lambda-goof.html<p>Lets talk about how I accidentally incurred a million or so extra S3 & Lambda calls in a day.</p><p>So this post is going to be more "storytelling" than "tutorial". I'm writing this for a
couple reasons:</p>
<ul>
<li>it's funny</li>
<li>I think it important to learn from your mistakes by reflecting back on them</li>
<li>it's funny</li>
<li>It shows how easy it is to accidentally spend money on AWS</li>
<li>Have I mentioned the humour aspect?</li>
</ul>
<p>Ok, really it's not that funny, but I'm laughing because it helps comfort the fact
that I wasted some money due to a dumb technical goof.</p>
<p>So, the story: one day about a month ago, I went to <a href="https://tools.pingdom.com">https://tools.pingdom.com</a> and
ran this site through Pingdom's speed test. It gave back a bunch of recommendations
around how to improve the performance of my site, one of those being to "Leverage
Browser Caching" by setting a particular HTTP header on resources that don't change
often (ex: images). It'll look something like:</p>
<p><img alt="Leverage Browser Caching" src="https://www.codependentcodr.com/static/imgs/pingdomcachewarning-fs8.png"></p>
<p>Specifically you set the
<a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control"><code>Cache-Control</code> header</a>
which is used to communicate to client browsers that a resource can be cached client-side
for an extended period of time so that if a page is refreshed, that (large yet
never changing) resource doesn't have to be refetched again. Common patterns are to
set the header to a value of <code>public, max-age=604800</code> to indicate 1 week (<code>60x60x24x7==604800</code>).</p>
<p>Cool, so because this site is hosted in <a href="https://aws.amazon.com/">AWS</a> in a
<a href="https://aws.amazon.com/s3/">S3</a> bucket, you have to specify this header there on the resource
in the bucket (unfortunately you can't set this header in
<a href="https://aws.amazon.com/cloudfront/">Cloudfront</a>). In the console this is easy:</p>
<ul>
<li>find the object in the bucket</li>
<li>pick "Properties" on the object</li>
<li>under "Metadata" you can add arbitrary headers including <code>Cache-Control</code></li>
</ul>
<p>Which works, but is tedious if you have many files to update. I then spent a period of time
looking into various ways to have this metadata set automatically when I push content to
S3 on deploying changes to the site. This
<a href="https://gist.github.com/shentonfreude/36d0910984cefd42ab503dba61ada049">proved</a>
<a href="https://stackoverflow.com/questions/35744029/accessing-meta-data-from-aws-s3-with-aws-lambda">to</a>
<a href="https://www.bggofurther.com/2015/10/aws-update-massively-metadata-using-boto-python-on-multiple-s3-objects/">be</a>
<a href="https://stackoverflow.com/questions/22501465/how-to-add-cache-control-in-aws-s3">trickier</a>
<a href="https://github.com/aws/aws-cli/issues/652">than</a>
one would think,
but one idea was to set this value via a <a href="https://aws.amazon.com/lambda/">Lambda</a>. The idea:</p>
<ul>
<li>create a Lambda which fires when an object is put into a particular S3 bucket</li>
<li>that Lambda then looks the item in the bucket</li>
<li>if the name ends with <code>png</code> or <code>jpg</code> or <code>js</code>, etc add the header as metadata</li>
</ul>
<p>Simple right? And this is sweet in that it's a "set it and forget it" option -- once the
Lambda is set, you just upload stuff as you did before and Lambda takes care of setting
the header.</p>
<p>So I wrote up a simple Lambda, and tested it out, worked great.</p>
<p>And that's when things started to go sideways. You see, the Lambda would do a <code>copy_object</code>
on the object to set the metadata. That is, the Lambda which was triggered by a put of an
object to an S3 bucket would (as processing the object) put the object into that S3 bucket
with updated metadata.</p>
<p>And I think this is the point where you realize the problem. Effectively I created an
infinite loop of lambda calls.</p>
<p>The first warning sign was later that day I got an email from AWS saying I was approaching
the free tier limit for Lambda (1,000,000 calls). Given I just hooked up the Lambda that
day this surprised me. The thing is though, Lambda is stupid cheap (even past 1M it's
literally pennies per million or so calls). So this was mistake #2 -- not acting immediately
thinking "ah, that's weird, I'll look into it later".</p>
<p>By the time that "later" happened and I turned off the Lambda, I had accrued around 1.1
million Lambda requests:</p>
<p><img alt="The Lambda call count & costs" src="https://www.codependentcodr.com/static/imgs/lambda_costs-fs8.png"></p>
<p>But that's nothing:the real problem was that each one of those lambda calls represented
a PUT to a S3 bucket.
PUT's with S3 are actually one of the more expensive operations. For the <code>ca-central-1</code>
region where I host my stuff, it's currently $0.0055 per 1,000 of them. This
sounds unbelievably cheap, and it is, but when you're doing about 1.1 million of
them, well, that adds up:</p>
<p><img alt="The S3 PUT count & costs" src="https://www.codependentcodr.com/static/imgs/s3_costs-fs8.png"></p>
<p>Queue the Iron Maiden -- 6, 6, 6, THE NUMBER OF THE BEAST!</p>
<p>To give some context: this site usually costs me well under $0.50 a month, the biggest portion of
which is Cloudfront which clocks in around $0.24. Everything else is misc stuff: data
transfer, S3 storage costs, S3 request costs, I have some old data in Glacier, etc. So to
see ~$7 accrued in a day felt, well, excessive.</p>
<p>What really bugged me was how <em>dumb</em> I felt, such a silly mistake. What's kinda scary is
that had I not gotten that email from AWS indicating I was close to the free tier limit,
I likely wouldn't have noticed until near the end of the month, which would've turned that
bill into likely well over $100.</p>
<p>But in the end I got a funny picture, this is the graph of my S3 bill for the month of
April:</p>
<p><img alt="Graph of S3 costs" src="https://www.codependentcodr.com/static/imgs/s3costgraph-fs8.png"></p>
<p>Ouch, that's a pointy point in my pride.</p>The Text Editing Kata - VS Code Edition2018-05-01T21:55:00-07:002018-05-04T20:55:00-07:00Adam Parkintag:www.codependentcodr.com,2018-05-01:/the-text-editing-kata-vs-code-edition.html<p>Using Visual Studio Code to do the Text Editing Kata</p><p>I recently stumbled across <a href="https://code.joejag.com/2016/text-editing-kata.html">this article</a>
which had a challenge/kata to do some funky text
editing to convert a list of strings into some specially formatted HTML. The idea is
take input like:</p>
<div class="highlight"><pre><span></span>Basic Price
Discount
Sub total
... etc ...
</pre></div>
<p>And produce:</p>
<div class="highlight"><pre><span></span><span class="p"><</span><span class="nt">dt</span> <span class="na">runat</span><span class="o">=</span><span class="s">"server"</span> <span class="na">id</span><span class="o">=</span><span class="s">"dtBasicPrice"</span><span class="p">></span>Basic Price<span class="p"></</span><span class="nt">dt</span><span class="p">></span>
<span class="p"><</span><span class="nt">dd</span> <span class="na">runat</span><span class="o">=</span><span class="s">"server"</span> <span class="na">id</span><span class="o">=</span><span class="s">"ddBasicPrice"</span><span class="p">></</span><span class="nt">dd</span><span class="p">></span>
<span class="p"><</span><span class="nt">dt</span> <span class="na">runat</span><span class="o">=</span><span class="s">"server"</span> <span class="na">id</span><span class="o">=</span><span class="s">"dtDiscount"</span><span class="p">></span>Discount<span class="p"></</span><span class="nt">dt</span><span class="p">></span>
<span class="p"><</span><span class="nt">dd</span> <span class="na">runat</span><span class="o">=</span><span class="s">"server"</span> <span class="na">id</span><span class="o">=</span><span class="s">"ddDiscount"</span><span class="p">></</span><span class="nt">dd</span><span class="p">></span>
<span class="p"><</span><span class="nt">dt</span> <span class="na">runat</span><span class="o">=</span><span class="s">"server"</span> <span class="na">id</span><span class="o">=</span><span class="s">"dtSubTotal"</span><span class="p">></span>Sub total<span class="p"></</span><span class="nt">dt</span><span class="p">></span>
<span class="p"><</span><span class="nt">dd</span> <span class="na">runat</span><span class="o">=</span><span class="s">"server"</span> <span class="na">id</span><span class="o">=</span><span class="s">"ddSubTotal"</span><span class="p">></</span><span class="nt">dd</span><span class="p">></span>
.... etc ....
</pre></div>
<p>That is, for each line produce a <code><dt></code> and <code><dd></code> tag, with the <code>id</code> of those tags being
the text, but with the text coverted to title case (first character following a space
capitalized) and spaces removed.</p>
<p>My first attempt was literally just typing each line manually using no shortcuts other than
copy & paste. It took me just over 10 minutes to do the entire list of 24 lines.</p>
<p>There has to be a better way. I saw some
<a href="https://www.reddit.com/r/programming/comments/5860hx/the_text_editing_kata/">discussion</a>
using Sublime, Vi, Emacs, and other editors, but all seemed a bit beyond me.</p>
<p>I then thought and thought, and Googled & Googled, and tried various tricks and ideas.</p>
<p>Eventually I came up with a technique with which I did the entire exercise in just over a
minute. Here's a recording of it, note that this does require the <code>change-case</code> extension:
<a href="https://marketplace.visualstudio.com/items?itemName=wmaurer.change-case">https://marketplace.visualstudio.com/items?itemName=wmaurer.change-case</a></p>
<!-- markdownlint-disable MD033 -->
<video autoplay loop controls>
<source src="/static/vids/textKata480p.mp4" type="video/mp4">
<img src="/static/imgs/textKata.gif">
</video>
<!-- markdownlint-enable MD033 -->
<p>Ok, that's hard to follow, but this is the gist of what I did. Note that some shortcut
keys may be different for you, because
I've remapped a bunch of keys, and installed the
<a href="https://marketplace.visualstudio.com/items?itemName=ms-vscode.sublime-keybindings">Sublime Text Keymap extension</a></p>
<ul>
<li>Command palette (<code>⌘+p</code>), pick "Trim Trailing Whitespace"</li>
<li>Select all lines (<code>⌘+a</code>), and copy (<code>⌘+c</code>)</li>
<li>Open new buffer (<code>⌘+n</code>) and paste (<code>⌘+v</code>)</li>
<li>Spit view (<code>⌘+v</code>), and close the left hand version of the 2nd buffer</li>
<li>Go to 2nd buffer, select all lines, command palette, then pick "Change Case title"<ul>
<li>this requires the <code>change-case</code> extension</li>
</ul>
</li>
<li>Find & replace (<code>⌘+shift+f</code>), search for single blank space & replace with nothing<ul>
<li>make sure "Use Regular Expression" is turned on</li>
</ul>
</li>
<li>Go back to 1st buffer</li>
<li>select all, multiline edit (<code>⌘+shift+l</code>), copy</li>
<li>paste, then <code>enter</code> to put newline between each & have alternating line cursors</li>
<li>go to end of line, and <code>enter</code> to add the extra blank line</li>
<li><em>without deselecting</em> go to 2nd buffer, select all & copy</li>
<li>go back to 1st buffer, go to start of line, paste</li>
<li>manually type <code>"></code> to close the opening tag</li>
<li>go to end of line, and add the closing <code></dt></code> tag</li>
<li>go to start of line, and manually type <code><dt runat="server" id="dt</code><ul>
<li>this finishes the <code><dt></code> tag lines</li>
</ul>
</li>
<li><em>without deselecting</em> go down one line, and go to start of line</li>
<li>highlight to end of line & paste</li>
<li>manually type <code>"></dd></code></li>
<li>go to start of line & manually add <code><dd runat="server" id="dd</code></li>
<li>profit!</li>
</ul>
<p>Or, you could just write a little Python script to do it:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">fileinput</span>
<span class="k">def</span> <span class="nf">process_line</span><span class="p">(</span><span class="n">line</span><span class="p">):</span>
<span class="n">line</span> <span class="o">=</span> <span class="n">line</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span>
<span class="n">idline</span> <span class="o">=</span> <span class="n">line</span><span class="o">.</span><span class="n">title</span><span class="p">()</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s2">" "</span><span class="p">,</span> <span class="s2">""</span><span class="p">)</span>
<span class="k">return</span> <span class="sa">f</span><span class="s2">"""<dt runat="server" id="dt</span><span class="si">{</span><span class="n">idline</span><span class="si">}</span><span class="s2">"></span><span class="si">{</span><span class="n">line</span><span class="si">}</span><span class="s2"></dt></span>
<span class="s2"><dd runat="server" id="dd</span><span class="si">{</span><span class="n">idline</span><span class="si">}</span><span class="s2">"></dd></span>
<span class="s2">"""</span>
<span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">fileinput</span><span class="o">.</span><span class="n">input</span><span class="p">():</span>
<span class="nb">print</span><span class="p">(</span><span class="n">process_line</span><span class="p">(</span><span class="n">line</span><span class="p">))</span>
</pre></div>
<p>This was a fun, albeit contrived editor example that really stretched my knowledge of how to wrangle text
with VS Code.</p>
<p><strong>Edit</strong>: re-recorded again this time doing the steps a bit more concisely.</p>
<!-- markdownlint-disable MD033 -->
<video autoplay loop controls>
<source src="/static/vids/textKata480p_p2.mp4" type="video/mp4">
</video>
<!-- markdownlint-enable MD033 -->Shell Tip of the Day: Using find and sed For Fun and Profit2018-04-29T13:59:00-07:002018-04-29T13:59:00-07:00Adam Parkintag:www.codependentcodr.com,2018-04-29:/shell-tip-of-the-day-using-find-and-sed-for-fun-and-profit.html<p>How to use find and sed to replace a common string across many files in one go</p><p>So I recently was tweaking the <code><meta></code> HTML tags on my posts to make them a little
nicer for sharing on social media (notably Twitter). The idea is that if you
add a few specific <code><meta></code> tags to your HTML pages then if that page is shared on say
Twitter, Twitter will make use of that metadata to render a nice looking tweet
that's more appealing (and therefore more likely to be clicked). I already
added some Opengraph tags (ie <code><meta property='og:...</code>), but Twitter, because it has to be unique
will make use of some Twitter specific ones.</p>
<p><a href="https://sproutsocial.com/insights/twitter-cards-guide/">This link</a> is a good
resource for getting started, and
<a href="https://cards-dev.twitter.com/validator">this link</a> and
<a href="http://iframely.com/debug">this link</a> allow you to verify that your tags are
correct once you're done.</p>
<p>That's all fine and good, but there can be a problem. Twitter will sometimes
cache the image you supply to the <code>twitter:image</code> tag, and if it sees a bad
image at that url, will fail to update it even though you fix the image.
<a href="https://developer.twitter.com/en/docs/tweets/optimize-with-cards/guides/troubleshooting-cards#refreshing_images">Their advice</a>
is to rename the url so it forces a re-index (dumb, but hey, whatevs).</p>
<p>This raises a problem: my image is specified in each post (because I sometimes
change the image for a specific article), so that meant I had to update a bunch
of files that had the line:</p>
<div class="highlight"><pre><span></span>cover: static/imgs/default_page_image.jpg
</pre></div>
<p>to the line:</p>
<div class="highlight"><pre><span></span>cover: static/imgs/default_page_imagev2.jpg
</pre></div>
<p>I could do this manually, but this is where shell utilities like <code>find</code> and
<code>sed</code> come in handy. In the directory containing all my markdown for my posts
I just did:</p>
<div class="highlight"><pre><span></span>find . -name <span class="s1">'*.md'</span> -exec sed -i <span class="s1">''</span> s/default_page_image/default_page_imagev2/ <span class="o">{}</span> +
</pre></div>
<p>And voila, done.</p>Py.test Basics2018-04-27T18:51:00-07:002018-04-28T12:51:00-07:00Adam Parkintag:www.codependentcodr.com,2018-04-27:/pytest-basics.html<p>Learning some of the basics of Py.test</p><p>So I'm fortunate enough to work for an employer who grants me access to
<a href="https://www.safaribooksonline.com/">Safari Books Online</a>. The other
day I was browsing the site to discover that they now have not only just
books, but also online courses/webinars on a variety of topics.</p>
<p>I stumbled across
<a href="https://www.safaribooksonline.com/live-training/courses/getting-started-with-pythons-pytest/0636920166832/">this one</a>
on Py.test which is a tool that is increasingly gaining traction in the
world of Python unit-testing, a world which I'm very much interested in.
As such, took the plunge and did the webinar, and thought I'd recap some
of the neat Py.test things that were new to me. <strong>Disclaimer</strong>: with
the exception of a brief intro a colleague gave me at a previous job,
I've had no experience with Py.test so this is all going to be pretty
basic stuff.</p>
<h1>Running tests</h1>
<p>Initially I first heard of Py.test as an alternative test runner. The built
in <code>unittest</code> module can be used for running your unit tests, but many people
(myself included) will use <a href="http://nose.readthedocs.io/en/latest/"><code>nose</code></a>
as a tool for running their tests. For example, you might do something like
the following to run your tests:</p>
<div class="highlight"><pre><span></span>$ nosetests
..................................
----------------------------------------------------------------------
Ran <span class="m">34</span> tests <span class="k">in</span> <span class="m">1</span>.440s
OK
</pre></div>
<p>With Py.test you'd do something like (from within the project's directory):</p>
<div class="highlight"><pre><span></span>$ <span class="nv">PYTHONPATH</span><span class="o">=</span>. <span class="nv">pytest</span>
<span class="o">==========================================================================</span> <span class="nb">test</span> session <span class="nv">starts</span> <span class="o">===========================================================================</span>
platform darwin -- Python <span class="m">3</span>.6.2, pytest-3.5.1, py-1.5.3, pluggy-0.6.0
rootdir: /Users/aparkin/temp/sandbox/pytestcourse/block, inifile:
collected <span class="m">3</span> items
test_block/test_block.py ... <span class="o">[</span><span class="m">100</span>%<span class="o">]</span>
<span class="o">========================================================================</span> <span class="m">3</span> passed <span class="k">in</span> <span class="m">0</span>.02 <span class="nv">seconds</span> <span class="o">========================================================================</span>
</pre></div>
<p>The output's quite different, but not in a bad way.</p>
<h1>Tests as Functions</h1>
<p>The first thing that'll jump out at you about Py.test is that it's encouraged
to write your tests as functions, rather than methods in a class. In the base
<code>unittest</code> module in Python it follows the typical xUnit style pattern of test
organization: you create a class which inherits from some base class
(<code>unittest.TestCase</code> in Python's <code>unittest</code> module), and then each individual
unit test is a method on that class. So if you had a class <code>Dog</code> you might
write a test class like the following:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">TestDog</span><span class="p">(</span><span class="n">unittest</span><span class="o">.</span><span class="n">TestCase</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">test_speak</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">dog</span> <span class="o">=</span> <span class="n">Dog</span><span class="p">()</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">dog</span><span class="o">.</span><span class="n">speak</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="s2">"bark!"</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
</pre></div>
<p>Certainly you can still write tests like this with Py.test (ie all old tests
you have will still work just fine with Py.test), but instead you're encouraged
to write your tests as standalone functions. So the above might look like:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">test_dog_speak</span><span class="p">():</span>
<span class="n">dog</span> <span class="o">=</span> <span class="n">Dog</span><span class="p">()</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">dog</span><span class="o">.</span><span class="n">speak</span><span class="p">()</span>
<span class="k">assert</span> <span class="s2">"bark!"</span> <span class="o">==</span> <span class="n">result</span>
</pre></div>
<p>Note as well another key difference: the use of Python's built-in <code>assert</code>
statement rather than using the <code>assert*()</code> methods defined in
<code>unittest.TestCase</code> since we're no longer inheriting from that class.</p>
<h2>Sidebar: The assert Gotcha</h2>
<p>The assert statement gotcha: in Python the assert statement is a statement,
not a function, so this can trip people up when they pass a second argument
to it. For example:</p>
<div class="highlight"><pre><span></span><span class="k">assert</span> <span class="n">somecondition</span><span class="p">,</span> <span class="s2">"This gets printed out"</span>
</pre></div>
<p>will print "This gets printed out" if somecondition is <code>false</code>, but
this is not the same as:</p>
<div class="highlight"><pre><span></span><span class="k">assert</span><span class="p">(</span><span class="n">somecondition</span><span class="p">,</span> <span class="s2">"This gets printed out"</span><span class="p">)</span>
</pre></div>
<p>which always evaluates to true because it's treated as passing a 2-value
tuple to the assert statement, and a non-empty tuple is truthy in Python.
This is a <em>classic</em> Python gotcha. Py.test adds some smarts around this, if
we had a test like:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">test_assert_with_tuple</span><span class="p">():</span>
<span class="k">assert</span> <span class="p">(</span><span class="kc">False</span> <span class="o">==</span> <span class="kc">True</span><span class="p">,</span> <span class="s2">"Should be false"</span><span class="p">)</span>
</pre></div>
<p>Then we'd get output like:</p>
<div class="highlight"><pre><span></span><span class="o">==========================================================================</span> <span class="nb">test</span> session <span class="nv">starts</span> <span class="o">===========================================================================</span>
platform darwin -- Python <span class="m">3</span>.6.2, pytest-3.5.1, py-1.5.3, pluggy-0.6.0
rootdir: /Users/aparkin/temp/sandbox/pytestcourse/Integer, inifile:
collected <span class="m">15</span> items
test/test_integr.py ............. <span class="o">[</span><span class="m">100</span>%<span class="o">]</span>
<span class="o">============================================================================</span> warnings <span class="nv">summary</span> <span class="o">============================================================================</span>
test/test_integr.py:119
assertion is always true, perhaps remove parentheses?
-- Docs: http://doc.pytest.org/en/latest/warnings.html
<span class="o">==================================================================</span> <span class="m">12</span> passed, <span class="m">1</span> warnings <span class="k">in</span> <span class="m">0</span>.11 <span class="nv">seconds</span> <span class="o">=================================================================</span>
</pre></div>
<p>Note how Py.test warns us that we might be doing something dumb.</p>
<h1>Testing Raised Exceptions</h1>
<p>A common thing to do in unit tests is test if a particular exception is
raised under certain circumstances. You can certainly do this in vanilla
<code>unittest</code>:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">TestFoo</span><span class="p">(</span><span class="n">unittest</span><span class="o">.</span><span class="n">TestCase</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">test_foo_throws_value_error_when_given_negative_number</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="c1"># test will only pass if foo() raises ValueError</span>
<span class="k">with</span> <span class="bp">self</span><span class="o">.</span><span class="n">assertRaises</span><span class="p">(</span><span class="ne">ValueError</span><span class="p">):</span>
<span class="n">foo</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span>
</pre></div>
<p>But again since we're no longer inheriting from <code>unittest.TestCase</code> how do
we do this with Py.test? There's a couple ways, one is still with a context
manager:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">pytest</span>
<span class="k">def</span> <span class="nf">test_foo_throws_value_error_when_given_negative_number</span><span class="p">():</span>
<span class="c1"># test will only pass if foo() raises ValueError</span>
<span class="k">with</span> <span class="n">pytest</span><span class="o">.</span><span class="n">raises</span><span class="p">(</span><span class="ne">ValueError</span><span class="p">):</span>
<span class="n">foo</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span>
</pre></div>
<p>Another is to "mark" the test as an expected failure:</p>
<div class="highlight"><pre><span></span><span class="nd">@pytest</span><span class="o">.</span><span class="n">mark</span><span class="o">.</span><span class="n">xfail</span><span class="p">(</span><span class="n">raises</span><span class="o">=</span><span class="ne">ValueError</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_foo_throws_value_error_when_given_negative_number</span><span class="p">():</span>
<span class="n">foo</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span>
</pre></div>
<p>The latter is slighly different, and this is reported in the test results as you'll
see output like:</p>
<div class="highlight"><pre><span></span><span class="o">==========================================================================</span> <span class="nb">test</span> session <span class="nv">starts</span> <span class="o">===========================================================================</span>
platform darwin -- Python <span class="m">3</span>.6.2, pytest-3.5.1, py-1.5.3, pluggy-0.6.0
rootdir: /Users/aparkin/temp/sandbox/pytestcourse/Integer, inifile:
collected <span class="m">15</span> items
test/test_integr.py ..x............ <span class="o">[</span><span class="m">100</span>%<span class="o">]</span>
<span class="o">============================================================</span> <span class="m">12</span> passed, <span class="m">1</span> skipped, <span class="m">1</span> xfailed <span class="k">in</span> <span class="m">0</span>.12 <span class="nv">seconds</span> <span class="o">=============================================================</span>
</pre></div>
<p>Note that <code>1 xfailed</code> bit, indicating that there was a single test with an "expected"
failure. This is effectively coding a way in which the SUT fails, but in an expected
way.
<a href="https://docs.pytest.org/en/latest/assert.html#assertions-about-expected-exceptions">The docs</a>
suggest that:</p>
<blockquote>
<p>Using pytest.raises is likely to be better for cases where you are testing exceptions your own
code is deliberately raising, whereas using @pytest.mark.xfail with a check function is probably
better for something like documenting unfixed bugs (where the test describes what “should”
happen) or bugs in dependencies.</p>
</blockquote>
<p>I could see this being useful for a test that's currently failing in a way you
expect, you could comment the test out (but then you might as well delete it) or
you could codify that you expect it to fail. The <code>raises:ValueError)</code> restricts
the way you expect it to fail (for example if you expect it to throw a ValueError,
and it throws a RuntimeError then that's not expected and should cause a test run
failure).</p>
<h1>Forcing a Failure</h1>
<p>You can also force a test to fail:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">test_force_failure</span><span class="p">():</span>
<span class="c1"># This test will always fail</span>
<span class="n">pytest</span><span class="o">.</span><span class="n">fail</span><span class="p">(</span><span class="s2">"this will fail"</span><span class="p">)</span>
</pre></div>
<p>This is effectively the same thing as adding <code>self.assertEqual(True, False)</code>
in a vanilla Python unit test. I'm not sure where this would be useful, the instructor
gave this example:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">test_import_error</span><span class="p">():</span>
<span class="k">try</span><span class="p">:</span>
<span class="kn">import</span> <span class="nn">somethirdpartylibrary</span>
<span class="k">except</span> <span class="ne">ImportError</span><span class="p">:</span>
<span class="n">pytest</span><span class="o">.</span><span class="n">fail</span><span class="p">(</span><span class="s2">"No module named somethirdpartylibrary to import"</span><span class="p">)</span>
</pre></div>
<p>But I fail to see the benefit of this. If the import was missing, you'll still get
test failures (in fact, presumably any test that needs <code>somethirdpartylibrary</code> will
still blow up even with this test in place).</p>
<h1>Approximations</h1>
<p>Sometimes you want to test a floating point number for a particular value in a test,
but because floating point values in any programming language are not precise, you
effectively have to create a "tolerable range" to test against. For example in plain
Python <code>unitest</code> you might do something like:</p>
<div class="highlight"><pre><span></span><span class="bp">self</span><span class="o">.</span><span class="n">assertAlmostEqual</span><span class="p">(</span><span class="n">some_value</span><span class="p">,</span> <span class="n">some_other_value</span><span class="p">,</span> <span class="n">delta</span><span class="o">=</span><span class="mf">0.001</span><span class="p">)</span>
</pre></div>
<p>which would pass if <code>some_value</code> and <code>some_other_value</code> are within +/- <code>0.001</code> of
each other. With Py.test you can do the following:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">test_approx_same</span><span class="p">():</span>
<span class="n">expected</span> <span class="o">=</span> <span class="mf">0.3</span>
<span class="n">result</span> <span class="o">=</span> <span class="mf">0.1</span> <span class="o">+</span> <span class="mf">0.2</span>
<span class="k">assert</span> <span class="n">pytest</span><span class="o">.</span><span class="n">approx</span><span class="p">(</span><span class="n">expected</span><span class="p">)</span> <span class="o">==</span> <span class="n">result</span>
</pre></div>
<p>There's a bunch of flexibility around how to use <code>approx</code> and it works with more
than just floating point numbers.
<a href="https://docs.pytest.org/en/latest/reference.html#pytest-approx">The docs</a> are quite good.</p>
<h1>Marking Tests & Selective Test Execution</h1>
<p>Oftentimes you want to group tests into various categories. For example you might want
to separate unit tests from integration tests, or fast-running tests from slow-running
tests, etc. Doing this in vanila <code>unittest</code> is kinda cumbersome, but with Py.test it's
easy:</p>
<div class="highlight"><pre><span></span><span class="nd">@pytest</span><span class="o">.</span><span class="n">mark</span><span class="o">.</span><span class="n">long</span>
<span class="k">def</span> <span class="nf">test_this_takes_long</span><span class="p">():</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"this is taking a long time...."</span><span class="p">)</span>
<span class="kn">import</span> <span class="nn">time</span> <span class="p">;</span> <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span>
<span class="k">assert</span> <span class="kc">True</span> <span class="o">==</span> <span class="kc">True</span>
<span class="nd">@pytest</span><span class="o">.</span><span class="n">mark</span><span class="o">.</span><span class="n">fast</span>
<span class="k">def</span> <span class="nf">test_this_is_fast</span><span class="p">():</span>
<span class="k">assert</span> <span class="kc">True</span> <span class="o">==</span> <span class="kc">True</span>
</pre></div>
<p>With tests like this you can run just the fast tests:</p>
<div class="highlight"><pre><span></span>pytest -m fast
</pre></div>
<p>or just the slow tests:</p>
<div class="highlight"><pre><span></span>pytest -m slow
</pre></div>
<p>Or even do some simple boolean logic, like for example anything that's not in the slow
category:</p>
<div class="highlight"><pre><span></span>pytest -m <span class="s2">"not slow"</span>
</pre></div>
<p>Really handy stuff.</p>
<p>You can also do fuzzy matching of tests by name, for example to run all the tests with
the word "bad" in them:</p>
<div class="highlight"><pre><span></span>pytest -k bad
</pre></div>
<p>This will collect & run tests with the word <code>bad</code> in the name (ex: <code>test_bad()</code>,
<code>test_that_the_bad_stuff_doesnt_happen()</code>, etc). Really useful for when you just want
to quickly run one or two specific tests.</p>
<h1>Conditionally Skipping Tests</h1>
<p>You can also conditionally skip tests:</p>
<div class="highlight"><pre><span></span><span class="nd">@pytest</span><span class="o">.</span><span class="n">mark</span><span class="o">.</span><span class="n">skipif</span><span class="p">(</span>
<span class="n">sys</span><span class="o">.</span><span class="n">version_info</span> <span class="o"><</span> <span class="p">(</span><span class="mi">3</span><span class="p">,</span><span class="mi">6</span><span class="p">),</span>
<span class="n">reason</span><span class="o">=</span><span class="s2">"Requires Python 3.6"</span>
<span class="p">)</span>
<span class="k">def</span> <span class="nf">test_that_requires_python_3_6_because_f_strings</span><span class="p">():</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">function_that_uses_f_strings</span><span class="p">()</span>
<span class="k">assert</span> <span class="s2">"some expected value"</span> <span class="o">==</span> <span class="n">result</span>
</pre></div>
<p>Since the block in the <code>skipif</code> is arbitrary Python code, you could use this for something
like running tests only if a particular environment variable is set, etc.</p>
<h1>Parameterized Tests</h1>
<p>This is where we really start to see some cool stuff. If you've ever written unit tests
in jUnit you'll have probably at some point come across
<a href="https://github.com/junit-team/junit4/wiki/parameterized-tests">parameterized tests</a>
which is a really useful technique for reducing test boilerplate by separating test
definition from test input data. Let's say we had a method <code>add()</code> that adds two numbers
together & returns the result (I know it's a boring example, but it really illustrates
the technique). You might write two tests for this function to test two positive numbers
and two negative numbers:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">TestAdd</span><span class="p">(</span><span class="n">unittest</span><span class="o">.</span><span class="n">TestCase</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">test_add_positive</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">expected</span> <span class="o">=</span> <span class="mi">4</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">add</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">expected</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_add_negative</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">expected</span> <span class="o">=</span> <span class="o">-</span><span class="mi">4</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">add</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="o">-</span><span class="mi">3</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">expected</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
</pre></div>
<p>Note that these two tests are identical save the expected and input values. Imagine if
you could somehow make those parameters to a test, then you might do something like:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">test_add</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">expected_result</span><span class="p">):</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">add</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">expected_result</span> <span class="o">==</span> <span class="n">result</span>
</pre></div>
<p>The question then becomes how do we specify those input values. In Python 3.4 they added
something like this with <code>subTest</code>
(<a href="https://docs.python.org/3/library/unittest.html#distinguishing-test-iterations-using-subtests">docs</a>):</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">test_add</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">values_to_test</span> <span class="o">=</span> <span class="p">[</span>
<span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">),</span>
<span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="o">-</span><span class="mi">3</span><span class="p">,</span> <span class="o">-</span><span class="mi">4</span><span class="p">),</span>
<span class="p">]</span>
<span class="k">for</span> <span class="n">test_vals</span> <span class="ow">in</span> <span class="n">values_to_test</span><span class="p">:</span>
<span class="k">with</span> <span class="bp">self</span><span class="o">.</span><span class="n">subTest</span><span class="p">(</span><span class="n">test_vals</span><span class="o">=</span><span class="n">test_vals</span><span class="p">):</span>
<span class="n">x</span><span class="p">,</span><span class="n">y</span><span class="p">,</span><span class="n">expected_result</span> <span class="o">=</span> <span class="n">test_vals</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">add</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="n">y</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">expected_result</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
</pre></div>
<p>But this is super clunky, the test data is within the test, obfuscating the meaning
of the actual test. You also have to manually unpack everything, and when we run it,
it ends up looking like a single test instead of two (so if one fails it's harder to
figure out which one of the two tuples in <code>values_to_test</code> caused the failure).</p>
<p>Instead, with Py.test you can use the <code>parametrize</code> mark:</p>
<div class="highlight"><pre><span></span><span class="nd">@pytest</span><span class="o">.</span><span class="n">mark</span><span class="o">.</span><span class="n">parametrize</span><span class="p">(</span><span class="s1">'x, y, expected_result'</span><span class="p">,</span> <span class="p">[</span>
<span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">),</span>
<span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="o">-</span><span class="mi">3</span><span class="p">,</span> <span class="o">-</span><span class="mi">4</span><span class="p">),</span>
<span class="p">])</span>
<span class="k">def</span> <span class="nf">test_add</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">expected_result</span><span class="p">):</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">add</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">expected_result</span> <span class="o">==</span> <span class="n">result</span>
</pre></div>
<p>This is <em>much</em> cleaner, the test meaning is extremely clear, and if we use the <code>-v</code> flag
when running the tests we can see that this produces two distinct tests for the two cases:</p>
<div class="highlight"><pre><span></span><span class="o">==========================================================================</span> <span class="nb">test</span> session <span class="nv">starts</span> <span class="o">===========================================================================</span>
platform darwin -- Python <span class="m">3</span>.6.2, pytest-3.5.1, py-1.5.3, pluggy-0.6.0 -- /Users/aparkin/.virtualenvs/pytestcourse/bin/python3.6
cachedir: .pytest_cache
rootdir: /Users/aparkin/temp/sandbox/pytestcourse/Integer, inifile:
collected <span class="m">2</span> items
test/test_integr.py::test_add<span class="o">[</span><span class="m">1</span>-3-4<span class="o">]</span> PASSED <span class="o">[</span> <span class="m">50</span>%<span class="o">]</span>
test/test_integr.py::test_add<span class="o">[</span>-1--3--4<span class="o">]</span> PASSED <span class="o">[</span><span class="m">100</span>%<span class="o">]</span>
<span class="o">========================================================================</span> <span class="m">2</span> passed <span class="k">in</span> <span class="m">0</span>.02 <span class="nv">seconds</span> <span class="o">========================================================================</span>
</pre></div>
<p>Also note that the input itself for the test is in the test identifier (ie the <code>[1-3-4]</code>
for the test which added 1 & 3 to get 4, etc).</p>
<p>Now adding additional test cases is a simple matter of adding another tuple to the
parametrize decorator. That is: each new test becomes a single line. That's some
super-concise test definition.</p>
<p>The <a href="https://docs.pytest.org/en/latest/parametrize.html#parametrize-basics">docs on this</a>
illustrate more stuff you can do with it, I'm just barely scratching the surface.</p>
<h2>Debugging</h2>
<p>Another handy trick is that with the <code>--pdb</code> command-line argument you can actually have
Py.test automatically drop into the pdb debugger on a failing test:</p>
<div class="highlight"><pre><span></span>$ pytest -v --pdb
<span class="o">==========================================================================</span> <span class="nb">test</span> session <span class="nv">starts</span> <span class="o">===========================================================================</span>
platform darwin -- Python <span class="m">3</span>.6.2, pytest-3.5.1, py-1.5.3, pluggy-0.6.0 -- /Users/aparkin/.virtualenvs/pytestcourse/bin/python3.6
cachedir: .pytest_cache
rootdir: /Users/aparkin/temp/sandbox/pytestcourse/Integer, inifile:
collected <span class="m">4</span> items
test/test_integr.py::test_add<span class="o">[</span><span class="m">1</span>-3-4<span class="o">]</span> PASSED <span class="o">[</span> <span class="m">25</span>%<span class="o">]</span>
test/test_integr.py::test_add<span class="o">[</span>-1--3--4<span class="o">]</span> PASSED <span class="o">[</span> <span class="m">50</span>%<span class="o">]</span>
test/test_integr.py::test_fail FAILED <span class="o">[</span> <span class="m">75</span>%<span class="o">]</span>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
def test_fail<span class="o">()</span>:
<span class="nv">x</span> <span class="o">=</span> <span class="m">2342423</span>
> assert <span class="nv">42</span> <span class="o">==</span> <span class="m">4234</span>
E assert <span class="nv">42</span> <span class="o">==</span> <span class="m">4234</span>
test/test_integr.py:167: AssertionError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /Users/aparkin/temp/sandbox/pytestcourse/Integer/test/test_integr.py<span class="o">(</span><span class="m">167</span><span class="o">)</span>test_fail<span class="o">()</span>
-> assert <span class="nv">42</span> <span class="o">==</span> <span class="m">4234</span>
<span class="o">(</span>Pdb<span class="o">)</span>
</pre></div>
<p>You can then use the usual <code>pdb</code> commands to try and figure how why a test is failing.</p>
<h1>Other Stuff</h1>
<p>There's way more to Py.test, things like Fixtures to replace the <code>setUp</code> & <code>tearDown</code> for
classes, running doctests, and a million other options. Hopefully this is a good starting
introduction.</p>Tip of the Day: Silver Searcher & VS Code2018-04-23T10:55:00-07:002018-04-23T17:30:00-07:00Adam Parkintag:www.codependentcodr.com,2018-04-23:/tip-of-the-day-silver-searcher-vs-code.html<p>Tip for opening up all files which match a string in separate VS Code tabs.</p><p>Oftentimes when working on a project you'll need to search through the codebase for
a particular string. Maybe it's looking for occurances of a variable, maybe a particular
include statement, whatever, this "needle in a haystack" is a common thing you do.
For example, I recently had to find all references to a particular string path across
a codebase of 1000's of files to update those files.</p>
<p>One way to approach this is to use your editor's search and click around on search
results opening files and inspecting. I however, like the command line, so I use
a tool called the <a href="https://github.com/ggreer/the_silver_searcher">Silver Searcher</a>
for doing this. The advantage of Silver Searcher is that it's very fast, and very
good at searching codebases. Think of it as recursive grep on steroids. Going
back to my problem, say I had to search for the string <code>/admin/login</code>. With
Silver Searcher I'd do something like:</p>
<div class="highlight"><pre><span></span>ag <span class="s2">"\/admin\/login"</span>
</pre></div>
<p>And this will search the current directory & all subdirectories that match that
string (note that this uses basic regex syntax for matching the slash), and display
the references inline. In my case, all I really wanted was a list of files that
matched, which we can do with the <code>-l</code> argument:</p>
<div class="highlight"><pre><span></span>ag <span class="s2">"\/admin\/login"</span> -l
</pre></div>
<p>Which instead of showing the matches, just lists the files that match the query.</p>
<p>Next was to open up each file in my editor of choice
(<a href="https://code.visualstudio.com/">Visual Studio Code</a>). One could do this by
going to VS Code and use the Quick Open (<code>⌘+P</code> on a Mac, <code>Ctrl+P</code> on lesser
platforms) and typing the filename to open. This however, is tedious.</p>
<p>One improvement is to use
<a href="https://code.visualstudio.com/docs/editor/integrated-terminal">VS Code's integrated terminal</a>
If you open the terminal and do the Silver Search from there, you can then cmd+click
the matched filenames and VS Code is smart enough to open the file up in a new
editor tab.</p>
<p>Better, but it's still kinda tedious to click each file.</p>
<p>In Vim, what you'd do is run a command to generate a buffer with the output of Silver
Searcher, then use the
<a href="http://vim.wikia.com/wiki/Open_file_under_cursor">"Open File Under Cursor" command (<code>gf</code>)</a>
in Vim to open up the file in a new buffer. I wonder if VS Code has something similar.</p>
<p>Doing some searching turned up
<a href="https://marketplace.visualstudio.com/items?itemName=Fr43nk.seito-openfile">this extension</a>
which does exactly that. If you have an editor tab open with a list of files, you can
right-click and pick "Open file under cursor" to open that file up.</p>
<p>Ok, so then I could do the search in the terminal, copy the result and paste into a new
editor tab, then use the Open File Under Cursor to open up each. But that copy pasting
is again, tedious. As it turns out though, as of
<a href="https://code.visualstudio.com/updates/v1_19#_pipe-output-directly-into-vs-code">v1.19 of VS Code</a>
you can redirect
output of a command in a integrated terminal window into a new editor tab. It looks
something like:</p>
<div class="highlight"><pre><span></span>ag <span class="s2">"whatever you are searching for"</span> -l <span class="p">|</span> code -
</pre></div>
<p>This will pipe the output of Silver Searcher into a new editor tab in the current VS
Code window. Cool, getting better, though that right-clicking and opening each line
is kinda tedious still, can't we just have the results opened instead of going through
this intermediate editor window?</p>
<p>Yes, yes we can:</p>
<div class="highlight"><pre><span></span>ag <span class="s2">"whatever you are searching for"</span> -l <span class="p">|</span> xargs code -
</pre></div>
<p>This matches all files that contain <code>whatever you are searching for</code> from the current
directory down, and opens up each in a new VS Code editor tab. The magic here is the
xargs command which if you haven't already used, stop reading this
blog post now and go <a href="http://lmgtfy.com/?q=xargs">Googling</a> as it's one of the most powerful
command-line utilities you'll ever see.</p>
<p>Really handy tip and I find myself using it a lot.</p>Screen Capturing for Site Monitoring2018-04-21T11:23:00-07:002018-04-21T11:23:00-07:00Adam Parkintag:www.codependentcodr.com,2018-04-21:/screen-capturing-for-site-monitoring.html<p>An experiment using screen capturing to detect CSS errors</p><p>So the other day I used <a href="https://tools.pingdom.com/">Pingdom's speed test</a> to do some analysis
trying to figure out ways to speed up loading of this site. One neat thing about the tool though
is it displays a preview image of what your site looks like in typical browsers.</p>
<p>This got me to thinking: I once had a problem with CSS not loading on my site because of some poorly
configured HTTP headers on items in S3. I didn't notice the problem for a considerable amount of time
though because I had the CSS cached locally so when I went to the site everything was fine. Seeing
the preview image though I got to thinking: if you could automate the process of taking a screenshot
of a site, and you could find a fuzzy image diffing tool, I wonder if you could build an automated
tool for detecting issues like this (or when the HTML gets completely goofed, or some other problems
where the site might still return HTTP status 200 but otherwise look broken).</p>
<p>As it turns out, this is totally doable, so I'll walk through what I came up with for this so far.</p>
<h1>Step 1 - Down the Rabbit Hole</h1>
<p>So, first step: how do you programmatically take a screenshot of a webpage? Doing some googling I
stumbled across <a href="https://www.booleanworld.com/take-screenshots-webpages-command-line/">this page</a>
which outlines a couple tools which seem to do this. Let's step through them:</p>
<h2>pageres-cli</h2>
<p>First up is <a href="https://github.com/sindresorhus/pageres-cli">pageres-cli</a> which is a Node app which
can take screenshots. Seemed perfect, so started off by firing up a basic Ubuntu VM to play around
on.</p>
<p>Small side note: I'll often do this when I'm trying out completely new/foreign tech -- fire up a
Linux VM using <a href="https://www.vagrantup.com/">Vagrant</a> as I can do whatever I want on the VM and
when I'm done just nuke the VM without having affected my host machine in any way. In case
interested the basic <code>Vagrantfile</code> I use can be found
<a href="https://gist.github.com/pzelnip/3c329f061a1d1e5e6c1f30a12bf71813">in this gist</a>. And yes, I
know I could use Docker instead, I just like Vagrant for this. :p</p>
<p>Anywho, I installed Node in the Ubuntu VM:</p>
<div class="highlight"><pre><span></span>curl -sL https://deb.nodesource.com/setup_9.x <span class="p">|</span> sudo -E bash -
sudo apt-get install -y nodejs
</pre></div>
<p>Then installed pageres:</p>
<div class="highlight"><pre><span></span>sudo npm install -g pageres-cli
</pre></div>
<p>And immediately ran into errors I think resulting from the version of Node I installed. Honestly
wasn't super interested in trying to debug this, and there were other choices from the article so
moved on.</p>
<h2>Firefox</h2>
<p>I honestly didn't even attempt this. My thinking was I was trying to build an automated tool,
so getting Firefox installed didn't seem quite like what I was looking for (and incredibly
heavyweight).</p>
<h2>cutycapt</h2>
<p>Onto <a href="http://cutycapt.sourceforge.net/">cutycapt</a>. This is a nice little command line tool which
uses QtWebKit for rendering web pages, and supports a number of options. Seemed like a good fit,
so installed it into the VM:</p>
<div class="highlight"><pre><span></span>sudo apt-get install cutycapt
</pre></div>
<p>Easy enough, let's try capturing a screenshot of a homepage:</p>
<div class="highlight"><pre><span></span>cutycapt --url<span class="o">=</span>www.google.com --out<span class="o">=</span>test.png
</pre></div>
<p>And was met with an error:</p>
<div class="highlight"><pre><span></span>cutycapt: cannot connect to X server
</pre></div>
<p>Luckily that same blog post mentioned the use of <code>xvfb</code> for virtual framebuffer goodness. Installed
that:</p>
<div class="highlight"><pre><span></span>sudo apt-get install xvfb
</pre></div>
<p>And then ran it with the virtual FB:</p>
<div class="highlight"><pre><span></span>xvfb-run --server-args<span class="o">=</span><span class="s2">"-screen 0, 1024x768x24"</span> cutycapt --url<span class="o">=</span>www.google.com --out<span class="o">=</span>test.png
</pre></div>
<p>This produced an image of Google's homepage. Success!</p>
<h1>Step 2 - Diffing two Images</h1>
<p>This is seemingly the hard part: how do you compare two images and get a
similarity score? I had visions of having to dig into machine learning
algorithms to try and figure this out, but before breaking out the heavy ML I
thought I'd see if someone else had already produced something that met my
needs.</p>
<p>Some false leads that didn't quite work (
<a href="https://stackoverflow.com/questions/5132749/diff-an-image-using-imagemagick">here</a>,
<a href="https://zachholman.com/posts/command-line-image-diffs/">here</a>,
<a href="https://www.pyimagesearch.com/2017/06/19/image-difference-with-opencv-and-python/">here</a>, and
<a href="http://douglasduhaime.com/posts/identifying-similar-images-with-tensorflow.html">here</a>
) and then landed on one that did: <a href="https://www.npmjs.com/package/image-diffr">image-diffr</a>.</p>
<p>It's a pretty simple tool -- give it two images, and it spits out a similarity score. For example:</p>
<div class="highlight"><pre><span></span>image-diffr test_cropped.png test2_cropped.png
✓ Image <span class="m">1</span> scanned OK:
--> Source: /home/vagrant/test_cropped.png
--> Type: image/png, Width: <span class="m">1920</span>, Height: <span class="m">1080</span>
✓ Image <span class="m">2</span> scanned OK:
--> Source: /home/vagrant/test2_cropped.png
--> Type: image/png, Width: <span class="m">1920</span>, Height: <span class="m">1080</span>
Difference <span class="o">(</span>with threshold <span class="m">0</span>.25<span class="o">)</span>:
<span class="m">0</span>.24349247685185185
</pre></div>
<p>The idea is the number it spits out at the end is a 0 to 1 probability that the two images
are the same (0 being identical, and 1 being completely different). Cool, let's try with two
screen caps of the homepage:</p>
<div class="highlight"><pre><span></span>vagrant@ubuntu-xenial:~$ image-diffr test1.png test2.png
✓ Image <span class="m">1</span> scanned OK:
--> Source: /home/vagrant/test1.png
--> Type: image/png, Width: <span class="m">1920</span>, Height: <span class="m">1080</span>
✓ Image <span class="m">2</span> scanned OK:
--> Source: /home/vagrant/test2.png
--> Type: image/png, Width: <span class="m">1920</span>, Height: <span class="m">2701</span>
Image heights must match, cannot perform diff.
Image heights must match, cannot perform diff.
</pre></div>
<p>Uh-oh, images have to be the exact same dimensions. Screenshots from <code>cutycapt</code> may vary in height
(for example if a new article gets posted, the summary will be a different length which will
cause the height of the screenshot to be slightly different).</p>
<p>So we need to get them to be the same dimensions. Ideally all I care about isn't the entire page
but just essentially the viewport of what a user sees when they land on the page. So we could
just crop the images to a typical screen resolution (say 1920x1080 for example). But now we
need a command-line tool to do this.</p>
<h1>Step 2a - ImageMagick to the Rescue</h1>
<p>In case you've never seen it before, <a href="https://www.imagemagick.org/script/index.php">ImageMagick</a>
is a library for doing some amazing image
manipulation stuff. And there's a CLI interface for it that's great for automating stuff.
The full scope of IM is way beyond the scope of this post (there's a <em>ton</em> you can do with
it), but cropping something is easy-peasy so let's get it installed:</p>
<div class="highlight"><pre><span></span>sudo apt-get install imagemagick
</pre></div>
<p>And now you get a nice little command which can be used to crop an image:</p>
<div class="highlight"><pre><span></span>convert some_image.png -crop 1920x1080+0+0 output.png
</pre></div>
<p>Dissecting this a bit, this will take the input image <code>some_image.png</code>, and crop it to
1920x1080, starting from the top-left corner (the <code>0+0</code> indicates an X,Y offset into the
image). More details can be found at: <a href="https://codeyarns.com/2014/11/15/how-to-crop-image-using-imagemagick/">https://codeyarns.com/2014/11/15/how-to-crop-image-using-imagemagick/</a></p>
<p>Ok, let's try capturing the homepage in a screenshot & then cropping:</p>
<div class="highlight"><pre><span></span>xvfb-run --server-args<span class="o">=</span><span class="s2">"-screen 0, 1920x1080x24"</span> cutycapt --url<span class="o">=</span>www.codependentcodr.com --out<span class="o">=</span>capture.png
convert capture.png -crop 1920x1080+0+0 capture.png
</pre></div>
<p>This worked, but gave the following image:</p>
<p><img alt="ScreenshotNoText" src="https://www.codependentcodr.com/static/imgs/captureNoText.png"></p>
<p>Wait, what? Where's the text? And why is the image so small? Looking into the original
image before the cropping I found these problems were there. As it turns out, there's two
problems:</p>
<ul>
<li>the page hasn't fully loaded before <code>cutycapt</code> takes a screenshot</li>
<li>by default, <code>cutycapt</code> assumes a screen resolution of 800x600</li>
</ul>
<p>Both are solvable with options to <code>cutycapt</code>. If we revise:</p>
<div class="highlight"><pre><span></span>xvfb-run --server-args<span class="o">=</span><span class="s2">"-screen 0, 1920x1080x24"</span> cutycapt --url<span class="o">=</span>www.codependentcodr.com <span class="se">\</span>
--out<span class="o">=</span>capture.png --delay<span class="o">=</span><span class="m">3000</span> --min-width<span class="o">=</span><span class="m">1920</span> --min-height<span class="o">=</span><span class="m">1080</span>
convert capture.png -crop 1920x1080+0+0 capture.png
</pre></div>
<p>Basically <code>--delay</code> waits a number of milliseconds before taking the capture (giving the
page a chance to load). I gave it 3 seconds which is probably overkill. <code>--min-width</code> and
<code>--min-height</code> specify a minimum width & height of the screenshot.</p>
<p>With this in place we get:</p>
<p><img alt="ScreenshotWithText" src="https://www.codependentcodr.com/static/imgs/captureFullRes.png"></p>
<p>Ahh, much better.</p>
<h1>Step 2b - Parsing Diff Output</h1>
<p>With this, we can screenshot with <code>cutycapt</code>, crop with ImageMagick, then diff with
<code>image-diffr</code>. The problem: I want effectively a shell command that returns 0 or 1
to indicate "page is about the same" or "page is totally different". There's a <code>-q</code>
option for <code>image-diffr</code>, which restricts its output to a single line that looks
roughly like <code>Difference: 0.2342422</code>. Ok, that's closer, but I need to extract that
score so I can do a numeric comparison on it. <code>cut</code> to the rescue:</p>
<div class="highlight"><pre><span></span>$ image-diffr file1.png file2.png -q <span class="p">|</span> cut -d <span class="s2">" "</span> -f2-
<span class="m">0</span>.01935763888888889
</pre></div>
<p>Success. Ok, now how do I get that output into a variable and compare it to some
"threshold" value that's effectively my similarity cutoff? Initially I brought
Python in for this:</p>
<div class="highlight"><pre><span></span>$ image-diffr file1.png file2.png -q <span class="p">|</span> cut -d <span class="s2">" "</span> -f2- <span class="p">|</span> python3 -c <span class="s2">"import sys; x = input(); sys.exit(0 if x < 0.1 else 1)"</span>
Traceback <span class="o">(</span>most recent call last<span class="o">)</span>:
File <span class="s2">"<string>"</span>, line <span class="m">1</span>, <span class="k">in</span> <module>
TypeError: unorderable types: str<span class="o">()</span> < float<span class="o">()</span>
</pre></div>
<p>Oops, need to convert that <code>x</code> value to a float:</p>
<div class="highlight"><pre><span></span>image-diffr file1.png file2.png -q <span class="p">|</span> cut -d <span class="s2">" "</span> -f2- <span class="p">|</span> python3 -c <span class="s2">"import sys; x = float(input()); sys.exit(0 if x < 0.1 else 1)"</span>
</pre></div>
<p>And that did the trick, this exited with error code 0 if the similarity score is less than 0.1
(pretty similar) and a non-zero (well actually <code>1</code>) error code if the similarity was higher than
0.1.</p>
<h1>Step 3 - Putting it Together</h1>
<p>Ok, at this point I had all the pieces, then I started gluing them together into a shell script.
I found though that little Python snippet I wanted to expand a bit (spit out an error message, etc),
but that's hard to do in a single 1-liner of Python. I started breaking it out into a separate
Python script, but since I was going to have a shell script anyways, why not just do the work
in Bash?</p>
<p>This led me to a first version:</p>
<div class="highlight"><pre><span></span><span class="ch">#!/bin/bash</span>
<span class="nb">set</span> -e
<span class="nv">CAPT_FILE</span><span class="o">=</span><span class="k">$(</span>date -u +<span class="s2">"%Y-%m-%dT%H-%M-%SZ_%s"</span><span class="k">)</span>.png
<span class="nb">echo</span> <span class="s2">"FILE: </span><span class="nv">$CAPT_FILE</span><span class="s2">"</span>
<span class="nb">echo</span> <span class="s2">"Capturing screenshot..."</span>
xvfb-run --server-args<span class="o">=</span><span class="s2">"-screen 0, 1920x1080x24"</span> cutycapt --url<span class="o">=</span>www.codependentcodr.com --out<span class="o">=</span><span class="nv">$CAPT_FILE</span> --delay<span class="o">=</span><span class="m">3000</span> --min-width<span class="o">=</span><span class="m">1920</span> --min-height<span class="o">=</span><span class="m">1080</span>
<span class="nb">echo</span> <span class="s2">"Done"</span>
<span class="nb">echo</span> <span class="s2">"Cropping image..."</span>
convert <span class="nv">$CAPT_FILE</span> -crop 1920x1080+0+0 <span class="nv">$CAPT_FILE</span>
<span class="nb">echo</span> <span class="s2">"Done"</span>
<span class="nb">echo</span> <span class="s2">"Diffing with previous..."</span>
<span class="nv">SCORE</span><span class="o">=</span><span class="k">$(</span>image-diffr <span class="nv">$CAPT_FILE</span> prev.png -t <span class="m">0</span>.25 -q <span class="p">|</span> cut -d <span class="s2">" "</span> -f2- <span class="k">)</span>
<span class="nv">THRESHOLD</span><span class="o">=</span><span class="m">0</span>.1
<span class="nb">echo</span> <span class="s2">"Score: </span><span class="nv">$SCORE</span><span class="s2">, threshold </span><span class="nv">$THRESHOLD</span><span class="s2">"</span>
<span class="k">if</span> <span class="o">((</span> <span class="k">$(</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$SCORE</span><span class="s2"> < </span><span class="nv">$THRESHOLD</span><span class="s2">"</span> <span class="p">|</span> bc -l<span class="k">)</span> <span class="o">))</span><span class="p">;</span> <span class="k">then</span>
<span class="nb">echo</span> <span class="s2">"Score good, all is well"</span>
<span class="k">else</span>
<span class="nb">echo</span> <span class="s2">"DANGER WILL ROBINSON, DANGER! Sites are different!"</span>
<span class="nb">exit</span> <span class="m">1</span>
<span class="k">fi</span>
<span class="nb">echo</span> <span class="s2">"Done"</span>
<span class="nb">echo</span> <span class="s2">"Copying </span><span class="nv">$CAPT_FILE</span><span class="s2"> to prev.png"</span>
cp <span class="nv">$CAPT_FILE</span> prev.png
<span class="nb">echo</span> <span class="s2">"Done"</span>
<span class="nb">echo</span> <span class="s2">"All done"</span>
</pre></div>
<p>What this does:</p>
<ul>
<li>creates a screenshot of the homepage into a file named based upon the current date/time</li>
<li>crops the screenshot to a typical monitor resolution (1920x1080)</li>
<li>diffs with a file <code>prev.png</code> which is intended to be the last screencap taken</li>
<li>if roughly the same spits out a message & overwrites <code>prev.png</code> with the new capture</li>
<li>if not, then spits out an error and aborts with non-zero error code</li>
</ul>
<p>The only real new bit is the elimination of the Python, being replaced with some basic
shell scripting & using <code>bc</code> for the comparison (I got the idea from
<a href="https://stackoverflow.com/a/31087503/808804">this StackOverflow answer</a>)</p>
<p>And there we have it. One could easily imagine how this could be built into some sort of
crude monitoring solution where you set up a cron job on a server somewhere to run this
every so often, and if the image diffs, then send an alert to Slack, or an email, or whatever.</p>
<p>Maybe I'll continue along that path in a future post, but just doing this was fun. :)</p>Python Tip of the Day - Simple HTTP Server2018-04-17T12:46:00-07:002018-04-17T12:46:00-07:00Adam Parkintag:www.codependentcodr.com,2018-04-17:/python-tip-of-the-day-simple-http-server.html<p>Using Python's Simple HTTP Server for fun and profit (and quickly sharing files)</p><p>This is kind of an old tip, but back in the days of Python 2 (aka
<a href="https://pythonbytes.fm/episodes/show/5/legacy-python-vs-python-and-why-words-matter-and-request-s-5-whys-retrospective">Legacy Python</a>)
a quick tip for sharing a folder of files was to use the built in
<a href="https://docs.python.org/2/library/simplehttpserver.html">SimpleHTTPServer module</a>
to quickly host a bunch of files:</p>
<div class="highlight"><pre><span></span>python -m SimpleHTTPServer
</pre></div>
<p>which spins up a basic http server on port 8000 that by default shows all the files in the directory from
where it is run. This is a very quick and handy way for sharing files with a coworker as (so long as they
know and can reach your IP address), you just fire up this server, then they can grab files, then you turf
the server.</p>
<p>Python 3 unfortunately (or fortunately, I dunno) renamed the module, leading to moments of confusion (ie
"why doesn't that command not work anymore now that I'm in a virtual environment with Python 3?"). The
equivalent in Python3 is:</p>
<div class="highlight"><pre><span></span>python -m http.server
</pre></div>
<p>But y'know what, I don't want to have to think about which version of Python I'm running, I just want a
simple command to fire up the server and have it figure out which module to run. Hence this
<a href="https://stackoverflow.com/a/46595749/808804">StackOverflow answer</a>, which indicates you can add this
to your <code>.bashrc</code> file (or equivalent for your shell):</p>
<div class="highlight"><pre><span></span><span class="nb">alias</span> <span class="nv">serve</span><span class="o">=</span><span class="s2">"python -m </span><span class="k">$(</span>python -c <span class="s1">'import sys; print("http.server" if sys.version_info[:2] > (2,7) else "SimpleHTTPServer")'</span><span class="k">)</span><span class="s2">"</span>
</pre></div>
<p>Now from any directory a simple <code>serve</code> will start up the appropriate simple http server.</p>Optimizing Cloudfront2018-04-12T19:32:00-07:002018-04-14T12:24:00-07:00Adam Parkintag:www.codependentcodr.com,2018-04-12:/optimizing-cloudfront.html<p>Lets recap some tips & tricks around optimizing your use of AWS CloudFront as a CDN.</p><p>AWS CloudFront is one of the services that AWS gives you that helps you optimize & speed
up delivery of your site's content. Commonly Cloudfront is used as a Content Delivery
Network (CDN), where the idea is that load for your site is distributed amongst a number
of data centers called <em>edge locations</em>.</p>
<p>The idea is you set up your site in some way (be that a dynamic server that serves some content,
an S3 bucket, whatever) and these <em>origins</em> will respond to requests from clients. However, if
a client is physically located far from where that origin is located, then there's certain latency
or overhead that's introduced which slows down the delivery of that content. The idea of using
Cloudfront as a CDN is instead you create a <em>distribution</em> in CloudFront which sits in front of
your origin (or origin(s) depending on your use case), and it automagically mirrors your content
to edge locations all around the world. When a client makes a request, it goes to the Cloudfront
edge location closest to the client and serves a cached version of the content (if it has seen it
before, if it hasn't, or the cached version is super old, it'll make the request to your origin
and pass that content back to the client).</p>
<p>At its core, this is a simple idea. But as is the case with much of AWS, there's all sorts of
tweaks and tricks to optimize things. In this post I'm going to outline some of the things I've
done to optimize my use of Cloudfront for this site.</p>
<h1>CloudFront Basics</h1>
<p>Let's walk through some of the basics of Cloudfront. Keep in mind, I am very much a n00b when
it comes to this stuff, but this is stuff I've figured out as I've worked through setting up
this site.</p>
<h2>Behaviours</h2>
<p>When you create a distribution, by default there's a single behaviour that's defined that
determines things like how long an item should remain in Cloudfront's cache before making the
request again to the origin, or if requests should be auto forwarded from http to https, and
various other settings. The behaviour screen looks something like:</p>
<p><img alt="Cloudfront Behavior" src="https://www.codependentcodr.com/static/imgs/cloudfront-behaviour-1.png"></p>
<p>By default, all items for your distribution are governed by the settings on this Default
Behaviour. What's really cool/handy is you can define multiple behaviours based upon request
path patterns. So for example, you could say things like:</p>
<ul>
<li>all requests for everything under <code>/static/images</code> should be cached for a really, really, really long time.</li>
<li>all requests for top level HTML files should always redirect from HTTP to HTTPS</li>
<li>all requests for things containing <code>thisIssupersecure</code> in the request path should only allow HTTPS (block non-HTTPS)</li>
</ul>
<p>... and so on. There's a ton of knobs to tweak.</p>
<h2>Invalidations</h2>
<p>Oftentimes you'll want to tell Cloudfront to drop stuff it's cached from your origin(s). This
is done by what's known as an
<a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Invalidation.html"><em>invalidation</em></a>.
An invalidation effectively says "hey Cloudfront, that thing you're caching has changed, so
go ahead and re-fetch it from the origin". Much like behaviours you have the ability to specify what
to invalidate by request path (so for example, maybe you only want to invalidate HTML files because
those have been updated, but leave any cached images).</p>
<p><strong><em>Important:</em></strong> it's worth noting that if you invalidate <em>everything</em> in a distribution, that could
potentially put your origin under heavy load. This is why it can pay to be selective about what you
invalidate.</p>
<p>It's also worth noting that if you create too many invalidations in a period of time, you will be
charged.
<a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Invalidation.html#PayingForInvalidation">The docs</a>
indicate that at the time of writing you get your first 1,000 invalidations per month for free,
after that you get billed. As is always the case with AWS, the specific details around billing
get complicated very quickly, but the gist is try to keep the number of invalidations down.</p>
<h1>My Setup</h1>
<p>Just to give a bit of context: this site is served as a static website via an S3 bucket that's behind
a single CloudFront distribution (ie the S3 bucket is the only origin for the Cloudfront distribution).</p>
<p>So what have I done? Let's discuss.</p>
<h2>CNAMEs</h2>
<p>Because I have a custom domain and don't want people to have to go to <a href="https://d1c3rffik0endi.cloudfront.net/">https://d1c3rffik0endi.cloudfront.net/</a> to visit
the site, I edited the cloudfront distribution to have two alternate domain names (CNAMEs):</p>
<ul>
<li>www.codpendentcodr.com</li>
<li>codpendentcodr.com</li>
</ul>
<p>You'll also have to set up DNS entries with whatever you use for DNS to have CNAME
records on your custom domain that point to your Cloudfront url (which is typically
<code><whateverYourDistributionIdIs>.cloudfront.net</code>)</p>
<h2>Redirecting HTTP to HTTPS</h2>
<p><a href="https://developers.google.com/web/fundamentals/security/encrypt-in-transit/why-https">Security is in</a>,
and as such I wanted all requests to my site to be https so I get the pretty green padlock. As such
one of the first things I did was change the "Viewer Protocol Policy" on the default behaviour to
"Redirect HTTP to HTTPS". Note that this requires that you have a cert & whatnot set up for your
distribution. If you haven't done that, I'd recommend you check out
<a href="https://ben.gnoinski.ca/set-up-acm-ssl-certs-and-domain-validation-with-route53.html">Ben's awesome guide</a>
as it walks through how to use AWS Certificate Manager to get your SSL set up.</p>
<h2>Multiple Behaviours</h2>
<p>At first I just had the default behaviour that CF gives you when you
set up the distribution. This quickly led to a problem that whenever I deployed my site, it meant
the top-level homepage would still be serving the old content until the TTL of the item expired
and Cloudfront went ahead & refetched it.</p>
<p>I could create an invalidation after every time I upload the site, but I didn't like that idea
as A) I was afraid I'd hit that 1,000 invalidation limit (which is unlikely but does create a
perceptual impediment to deploying the site), and B) meant that every time I deployed the site
I'd be throwing away the benefits of the Cloudfront cache regardless of how small the change was.</p>
<p>As well, there really is some content on the site that literally almost never changes and gets
requested on every page load (for example the profile picture in the left nav).</p>
<p>As such I divided up my content and created behaviours for each.</p>
<table>
<thead>
<tr>
<th>Path</th>
<th>Description</th>
<th>What I tweaked</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>/static/imgs/*</code></td>
<td>all static images for the site</td>
<td>Default TTL set to 2 days</td>
</tr>
<tr>
<td><code>/theme/*</code></td>
<td>All the content for my theme (ex CSS)</td>
<td>Default TTL set to 1 day</td>
</tr>
<tr>
<td><code>/</code> and <code>/*.html</code></td>
<td>All the HTML pages for the site</td>
<td>Default TTL set to 1 hour</td>
</tr>
<tr>
<td><code>/feeds/all.atom.xml</code></td>
<td>the RSS feed for the site</td>
<td>Allow both HTTP and HTTPS, default TTL to 1 hour</td>
</tr>
<tr>
<td><code>*</code> (default)</td>
<td>everything else</td>
<td>really nothing</td>
</tr>
</tbody>
</table>
<p>Unless otherwise specified, all of those behaviours are set to redirect HTTP to HTTPS. I allowed
HTTP on the RSS feed largely because it's not intended for human consumption, and I wondered if
there are possibly some weird old RSS readers that don't support HTTPs. I probably could've/should've
made even this redirect to HTTPS.</p>
<h2>Manual Invalidations</h2>
<p>Even with the tweaked behaviours, I still find myself wanting to create invalidations to speed
up refreshes for "dynamic" content. Taking inspiration from
<a href="https://ben.gnoinski.ca/invalidating-cloudfront-cache.html">Ben again</a> I set up a command
in my <code>Makefile</code> to create an invalidation with a single command (which I also set up in VS Code
as a Task that I can run with a couple keystrokes). It looks like this:</p>
<div class="highlight"><pre><span></span>aws --profile <span class="k">$(</span>AWSCLI_PROFILE<span class="k">)</span> cloudfront create-invalidation --distribution-id ER3YIY14W87BX --paths <span class="s1">'/*.html'</span> <span class="s1">'/'</span> <span class="s1">'/feeds/all.atom.xml'</span>
</pre></div>
<p>That is, invalidate all the HTML, the root page, and the RSS feed. The intention here is whenever
I push new content (like a new blog post), I run this invalidation so that changes get picked up
right away. Ideally I'd only create an invalidation for stuff that's actually changed, but doing
a diff between what's currently deployed and what's about to be deployed is tricky with
Pelican***.</p>
<p>If I'm doing less content-related stuff (for example, just playing around with some layout or
tweaking the theme), I don't really care if people get the old version for a bit so I just let
the cache expire naturally.</p>
<p>*** - Side note: I've thought about doing this, like essentially before a push to S3 first pull down
whatever's in S3 to my local box, then do a diff between what I pulled down from S3 and what's
in the <code>output/</code> directory I'm about to push and then only push & invalidate that stuff
specifically. Maybe someday....</p>
<h2>Offloading to Other CDNs</h2>
<p>One of the things in my theme is it uses <a href="https://fontawesome.com/">Font Awesome</a> for a few things.
The theme as written includes a minified version of the font-awesome CSS. It's great that I
have Cloudfront to serve that file, but if you think about it, there are A LOT of sites out there
that have Font Awesome. And they grab Font Awesome from a global CDN like Cloudflare. Which means
that if instead of serving Font Awesome myself, I could instead modify my theme to use those CDNs,
since it's far more likely that the widely used CDN will have that content optimized in an edge
location than my Cloudfront distribution. To give proof of this, I took requests for just the
font-awesome.min.css file from Cloudflare as well as through my Cloudfront distribution and ran them
through <a href="https://tools.pingdom.com/">Pingdom's speed test</a> (doing tests from Stockholm, Sweden).</p>
<p>Response times for coming from my
distribution started at 67ms then after a couple requests dropped down to ~22ms (presumably the
first request the css wasn't in the CF edge location so it had to hit the origin). Overall Pingdom
gave the request a grade of "D" because there's a few headers I hadn't tweaked.</p>
<p>Response times for coming from Cloudflare started at around 22ms and fluctuated between 22ms and 30ms
(consistent with the theory that they already had the content cached). Overall Pingdom gave the request a grade of "A".</p>
<p>Using Cloudflare saves me a fraction of a penny by cutting down on bandwidth charges and means I
can drop the stuff from my bucket which saves another fraction of a penny in storage costs (I'm
not kidding, it's literally fractions of a penny -- S3 & CF are both stupid cheap). I suppose it
also speeds up deployments of the site by a fraction of a second, since that's a few files I now
no longer have to upload on a deploy to S3.</p>
<p>And there you have it, there's certainly more you can do, and all this is actually super basic
stuff, but it's been fun digging into how to tweak and turn some of the knobs in Cloudfront.</p>Python Tip of the Day - Double If's in Comprehensions2018-04-11T13:21:00-07:002018-04-11T13:21:00-07:00Adam Parkintag:www.codependentcodr.com,2018-04-11:/python-tip-of-the-day-double-ifs-in-comprehensions.html<p>So I was reviewing a coworkers pull request today and saw something I hadn't seen in
Python before. As it turns out you can have multiple <code>if</code> clauses on a list comprehension.
For example:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="p">[</span><span class="n">v</span> <span class="k">for</span> <span class="n">v</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">50</span><span class="p">)</span> <span class="k">if</span> <span class="n">v</span> <span class="o">%</span> <span class="mi">2</span> <span class="o">==</span> <span class="mi">0</span> <span class="k">if</span> <span class="n">v</span> <span class="o">></span> <span class="mi">10</span><span class="p">]</span> <span class="c1"># all even numbers …</span></pre></div><p>So I was reviewing a coworkers pull request today and saw something I hadn't seen in
Python before. As it turns out you can have multiple <code>if</code> clauses on a list comprehension.
For example:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="p">[</span><span class="n">v</span> <span class="k">for</span> <span class="n">v</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">50</span><span class="p">)</span> <span class="k">if</span> <span class="n">v</span> <span class="o">%</span> <span class="mi">2</span> <span class="o">==</span> <span class="mi">0</span> <span class="k">if</span> <span class="n">v</span> <span class="o">></span> <span class="mi">10</span><span class="p">]</span> <span class="c1"># all even numbers between 10 & 50</span>
<span class="p">[</span><span class="mi">12</span><span class="p">,</span> <span class="mi">14</span><span class="p">,</span> <span class="mi">16</span><span class="p">,</span> <span class="mi">18</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="mi">22</span><span class="p">,</span> <span class="mi">24</span><span class="p">,</span> <span class="mi">26</span><span class="p">,</span> <span class="mi">28</span><span class="p">,</span> <span class="mi">30</span><span class="p">,</span> <span class="mi">32</span><span class="p">,</span> <span class="mi">34</span><span class="p">,</span> <span class="mi">36</span><span class="p">,</span> <span class="mi">38</span><span class="p">,</span> <span class="mi">40</span><span class="p">,</span> <span class="mi">42</span><span class="p">,</span> <span class="mi">44</span><span class="p">,</span> <span class="mi">46</span><span class="p">,</span> <span class="mi">48</span><span class="p">]</span>
</pre></div>
<p>I wondered if this was mentioned in the <a href="https://docs.python.org/3/tutorial/datastructures.html#list-comprehensions">standard docs</a>
, and sure enough:</p>
<blockquote>
<p>A list comprehension consists of brackets containing an expression followed by a for clause, then
<strong><em>zero or more</em></strong> for or if clauses.</p>
</blockquote>
<p>(emphasis added) That is, you can have as many <code>if</code> clauses as appropriate. It is roughly the same as:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="n">result</span> <span class="o">=</span> <span class="p">[]</span>
<span class="o">>>></span> <span class="k">for</span> <span class="n">v</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">50</span><span class="p">):</span>
<span class="o">...</span> <span class="k">if</span> <span class="n">v</span> <span class="o">%</span> <span class="mi">2</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
<span class="o">...</span> <span class="k">if</span> <span class="n">v</span> <span class="o">></span> <span class="mi">10</span><span class="p">:</span>
<span class="o">...</span> <span class="n">result</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">v</span><span class="p">)</span>
<span class="o">...</span>
<span class="o">>>></span> <span class="n">result</span>
<span class="p">[</span><span class="mi">12</span><span class="p">,</span> <span class="mi">14</span><span class="p">,</span> <span class="mi">16</span><span class="p">,</span> <span class="mi">18</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="mi">22</span><span class="p">,</span> <span class="mi">24</span><span class="p">,</span> <span class="mi">26</span><span class="p">,</span> <span class="mi">28</span><span class="p">,</span> <span class="mi">30</span><span class="p">,</span> <span class="mi">32</span><span class="p">,</span> <span class="mi">34</span><span class="p">,</span> <span class="mi">36</span><span class="p">,</span> <span class="mi">38</span><span class="p">,</span> <span class="mi">40</span><span class="p">,</span> <span class="mi">42</span><span class="p">,</span> <span class="mi">44</span><span class="p">,</span> <span class="mi">46</span><span class="p">,</span> <span class="mi">48</span><span class="p">]</span>
</pre></div>
<p>Astute readers will note that the multiple if clauses is equivalent to <code>and</code>-ing multiple boolean
expressions together. Ie that example is the same as:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="p">[</span><span class="n">v</span> <span class="k">for</span> <span class="n">v</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">50</span><span class="p">)</span> <span class="k">if</span> <span class="n">v</span> <span class="o">%</span> <span class="mi">2</span> <span class="o">==</span> <span class="mi">0</span> <span class="ow">and</span> <span class="n">v</span> <span class="o">></span> <span class="mi">10</span><span class="p">]</span>
<span class="p">[</span><span class="mi">12</span><span class="p">,</span> <span class="mi">14</span><span class="p">,</span> <span class="mi">16</span><span class="p">,</span> <span class="mi">18</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="mi">22</span><span class="p">,</span> <span class="mi">24</span><span class="p">,</span> <span class="mi">26</span><span class="p">,</span> <span class="mi">28</span><span class="p">,</span> <span class="mi">30</span><span class="p">,</span> <span class="mi">32</span><span class="p">,</span> <span class="mi">34</span><span class="p">,</span> <span class="mi">36</span><span class="p">,</span> <span class="mi">38</span><span class="p">,</span> <span class="mi">40</span><span class="p">,</span> <span class="mi">42</span><span class="p">,</span> <span class="mi">44</span><span class="p">,</span> <span class="mi">46</span><span class="p">,</span> <span class="mi">48</span><span class="p">]</span>
</pre></div>
<p>For me, this raised the question about order of evaluation, and if multiple if clauses do
<a href="https://en.wikipedia.org/wiki/Short-circuit_evaluation">boolean short-circuiting</a> like an
<code>and</code> statement does. So let's test that out:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="k">def</span> <span class="nf">foo</span><span class="p">():</span>
<span class="o">...</span> <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">()</span>
<span class="o">...</span>
<span class="o">>>></span> <span class="kc">False</span> <span class="ow">and</span> <span class="n">foo</span><span class="p">()</span>
<span class="kc">False</span>
<span class="o">>>></span> <span class="p">[</span><span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> <span class="k">if</span> <span class="kc">False</span> <span class="ow">and</span> <span class="n">foo</span><span class="p">()]</span>
<span class="p">[]</span>
<span class="o">>>></span> <span class="p">[</span><span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> <span class="k">if</span> <span class="kc">False</span> <span class="k">if</span> <span class="n">foo</span><span class="p">()]</span>
<span class="p">[]</span>
</pre></div>
<p>And we can see because we don't see an exception raised, that <code>foo()</code> was never called, so yes it is
only evaluating the second if clause if the first evaluates to <code>True</code>.</p>
<p>If you want to <em>really</em> convince yourself they're equivalent, you can use the
<a href="https://docs.python.org/3/library/dis.html"><code>dis</code></a> module to see
the equivalent bytecode for the two snippets:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="kn">import</span> <span class="nn">dis</span>
<span class="o">>>></span> <span class="n">dis</span><span class="o">.</span><span class="n">dis</span><span class="p">(</span><span class="s2">"[x for x in range(10) if False and foo()]"</span><span class="p">)</span>
<span class="mi">1</span> <span class="mi">0</span> <span class="n">LOAD_CONST</span> <span class="mi">0</span> <span class="p">(</span><span class="o"><</span><span class="n">code</span> <span class="nb">object</span> <span class="o"><</span><span class="n">listcomp</span><span class="o">></span> <span class="n">at</span> <span class="mh">0x10c4aeed0</span><span class="p">,</span> <span class="n">file</span> <span class="s2">"<dis>"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">1</span><span class="o">></span><span class="p">)</span>
<span class="mi">2</span> <span class="n">LOAD_CONST</span> <span class="mi">1</span> <span class="p">(</span><span class="s1">'<listcomp>'</span><span class="p">)</span>
<span class="mi">4</span> <span class="n">MAKE_FUNCTION</span> <span class="mi">0</span>
<span class="mi">6</span> <span class="n">LOAD_NAME</span> <span class="mi">0</span> <span class="p">(</span><span class="nb">range</span><span class="p">)</span>
<span class="mi">8</span> <span class="n">LOAD_CONST</span> <span class="mi">2</span> <span class="p">(</span><span class="mi">10</span><span class="p">)</span>
<span class="mi">10</span> <span class="n">CALL_FUNCTION</span> <span class="mi">1</span>
<span class="mi">12</span> <span class="n">GET_ITER</span>
<span class="mi">14</span> <span class="n">CALL_FUNCTION</span> <span class="mi">1</span>
<span class="mi">16</span> <span class="n">RETURN_VALUE</span>
<span class="o">>>></span> <span class="n">dis</span><span class="o">.</span><span class="n">dis</span><span class="p">(</span><span class="s2">"[x for x in range(10) if False if foo()]"</span><span class="p">)</span>
<span class="mi">1</span> <span class="mi">0</span> <span class="n">LOAD_CONST</span> <span class="mi">0</span> <span class="p">(</span><span class="o"><</span><span class="n">code</span> <span class="nb">object</span> <span class="o"><</span><span class="n">listcomp</span><span class="o">></span> <span class="n">at</span> <span class="mh">0x10c4aeed0</span><span class="p">,</span> <span class="n">file</span> <span class="s2">"<dis>"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">1</span><span class="o">></span><span class="p">)</span>
<span class="mi">2</span> <span class="n">LOAD_CONST</span> <span class="mi">1</span> <span class="p">(</span><span class="s1">'<listcomp>'</span><span class="p">)</span>
<span class="mi">4</span> <span class="n">MAKE_FUNCTION</span> <span class="mi">0</span>
<span class="mi">6</span> <span class="n">LOAD_NAME</span> <span class="mi">0</span> <span class="p">(</span><span class="nb">range</span><span class="p">)</span>
<span class="mi">8</span> <span class="n">LOAD_CONST</span> <span class="mi">2</span> <span class="p">(</span><span class="mi">10</span><span class="p">)</span>
<span class="mi">10</span> <span class="n">CALL_FUNCTION</span> <span class="mi">1</span>
<span class="mi">12</span> <span class="n">GET_ITER</span>
<span class="mi">14</span> <span class="n">CALL_FUNCTION</span> <span class="mi">1</span>
<span class="mi">16</span> <span class="n">RETURN_VALUE</span>
</pre></div>
<p>From this we can see that the corresponding bytecode is identical.</p>
<p>Probably unsurprisingly these apply to not only list comprehensions but <code>set</code> and <code>dict</code>
comprehensions as well:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="n">names</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"adam"</span><span class="p">,</span> <span class="s2">"bob"</span><span class="p">,</span> <span class="s2">"andrew"</span><span class="p">,</span> <span class="s2">"adam"</span><span class="p">,</span> <span class="s2">"fred"</span><span class="p">]</span>
<span class="o">>>></span> <span class="p">[</span><span class="n">name</span> <span class="k">for</span> <span class="n">name</span> <span class="ow">in</span> <span class="n">names</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">name</span><span class="p">)</span> <span class="o">==</span> <span class="mi">4</span> <span class="k">if</span> <span class="n">name</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"a"</span><span class="p">)]</span>
<span class="p">[</span><span class="s1">'adam'</span><span class="p">,</span> <span class="s1">'adam'</span><span class="p">]</span>
<span class="o">>>></span> <span class="p">{</span><span class="n">name</span> <span class="k">for</span> <span class="n">name</span> <span class="ow">in</span> <span class="n">names</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">name</span><span class="p">)</span> <span class="o">==</span> <span class="mi">4</span> <span class="k">if</span> <span class="n">name</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"a"</span><span class="p">)}</span>
<span class="p">{</span><span class="s1">'adam'</span><span class="p">}</span>
<span class="o">>>></span> <span class="p">{</span><span class="n">name</span><span class="p">:</span> <span class="n">name</span><span class="o">.</span><span class="n">title</span><span class="p">()</span> <span class="k">for</span> <span class="n">name</span> <span class="ow">in</span> <span class="n">names</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">name</span><span class="p">)</span> <span class="o">==</span> <span class="mi">4</span> <span class="k">if</span> <span class="n">name</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"a"</span><span class="p">)}</span>
<span class="p">{</span><span class="s1">'adam'</span><span class="p">:</span> <span class="s1">'Adam'</span><span class="p">}</span>
</pre></div>
<p>The first snippet gets all names that are four characters long & start with the letter <code>"a"</code>. Because
it's a list, duplicates are matched. The second does the same, but as a set comprehension and
since it's a set, duplicates are filtered out, so we only see <code>"adam"</code> once.
The last one creates a dict mapping the name as it's in the original list to the titlecased version
of the name.</p>Setting Up A Pelican Site in AWS2018-04-08T07:24:00-07:002018-04-08T07:24:00-07:00Adam Parkintag:www.codependentcodr.com,2018-04-08:/setting-up-a-pelican-site-in-aws.html<p>So I was going to write a blog post outlining most of the stuff I did in getting this
site up off the ground, but then a colleague went ahead and did the same and
wrote up their journey. :)</p>
<p>So, in true CodependentCodr fashion, Imma going to stea...err borrow …</p><p>So I was going to write a blog post outlining most of the stuff I did in getting this
site up off the ground, but then a colleague went ahead and did the same and
wrote up their journey. :)</p>
<p>So, in true CodependentCodr fashion, Imma going to stea...err borrow that content since
mine was largely the same steps (and he's better at AWS than I am). :p</p>
<p>Go <a href="https://ben.gnoinski.ca/how-this-site-came-to-be.html">here</a> for all the deets on getting
a static site set up with Pelican in AWS. Ben does a great job too of walking through setting
up a nice local dev pipeline with Docker (mine's not quite the same, but similar, and I'm
likely eventually going to do the same as him).</p>
<p>Thanks Ben!</p>Link of the Day: Fighting Burnout2018-04-04T08:49:00-07:002018-04-04T08:49:00-07:00Adam Parkintag:www.codependentcodr.com,2018-04-04:/link-of-the-day-fighting-burnout.html<p>Today's link of the day is from Stride's blog: <a href="https://www.stridenyc.com/blog/11-ways-to-fight-burnout">https://www.stridenyc.com/blog/11-ways-to-fight-burnout</a></p>
<p>I found much of the stuff in the article resonated with me, and many of
the tips in in are things I do & have found helped, so passing along.</p><p>Today's link of the day is from Stride's blog: <a href="https://www.stridenyc.com/blog/11-ways-to-fight-burnout">https://www.stridenyc.com/blog/11-ways-to-fight-burnout</a></p>
<p>I found much of the stuff in the article resonated with me, and many of
the tips in in are things I do & have found helped, so passing along.</p>AWS Tip Of The Day - Invalidating Items in the CloudFront Cache2018-04-01T08:14:00-07:002018-04-01T08:14:00-07:00Adam Parkintag:www.codependentcodr.com,2018-04-01:/aws-tip-of-the-day-invalidating-items-in-the-cloudfront-cache.html<p>TIL today: in <a href="https://aws.amazon.com/cloudfront/">CloudFront</a>, creating an invalidation for <code>/*.html</code> is not the same
as creating an invalidation for <code>/</code> even though <code>/</code> redirects to <code>/index.html</code>.</p>
<p>Context: yesterday I set up a CloudFront distribution for this site, so that I could attach an SSL cert to it (more
about this in a …</p><p>TIL today: in <a href="https://aws.amazon.com/cloudfront/">CloudFront</a>, creating an invalidation for <code>/*.html</code> is not the same
as creating an invalidation for <code>/</code> even though <code>/</code> redirects to <code>/index.html</code>.</p>
<p>Context: yesterday I set up a CloudFront distribution for this site, so that I could attach an SSL cert to it (more
about this in a future blog post). However, shortly after setting it all up, going to
<a href="https://www.codependentcodr.com">https://www.codependentcodr.com</a> resulted in an unstyled version of the main page. Looking at Chrome Dev Tools, I saw
that the request for the CSS was still going to <code>http://</code> and resulting in mixed content.</p>
<p>Looking into the source of <code>index.html</code> (the main page of the site that you get directed to when going to <code>/</code>), sure
enough the source URL for the CSS was in fact <code>http://</code> instead of <code>https://</code>. Reason being that I hadn't updated the
<code>SITEURL</code> value in my Pelican <code>publishconf.py</code> to be <code>https://</code>. Easy change, so did that & redeployed the site to S3.</p>
<p>Still no CSS.</p>
<p>Thought maybe the file didn't get updated in S3, so I manually went into S3 and pulled down <code>index.html</code> directly from
the bucket. Sure enough the source was correct -- <code>https://</code> instead of <code>http://</code>.</p>
<p>Problem was Cloudfront was still serving the old version of <code>index.html</code> with the non-secure URL. Ok, easy enough,
with some help from a friend I was pointed at
<a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Invalidation.html#invalidating-objects-console">the docs for invalidating items in Cloudfront's cache</a>
. Easy enough, create an invalidation for <code>/*.html</code>.</p>
<p>Did that, and waited. And waited. And waited. Other pages served fine (since they hadn't been requested before they
were getting the correct <code>https://</code> url), but that stubborn main page was still unstyled. Went to bed & got up the next
day and still the same problem. Ok, something wrong.</p>
<p>Did some digging and found <a href="https://forums.aws.amazon.com/thread.jspa?threadID=263425">this discussion</a>, which has this
comment a little ways down the first page:</p>
<blockquote>
<p>You need to invalidate what the browser is requesting -- not the file that is actually being served, if it is
different. Your invalidation request should be simply for / instead of /index.html.</p>
</blockquote>
<p>Duh. Added invalidation request for <code>/</code> and the homepage started immediately serving the CSS correctly.</p>Screencaptures on OSX2018-03-27T10:20:00-07:002018-03-27T10:20:00-07:00Adam Parkintag:www.codependentcodr.com,2018-03-27:/screencaptures-on-osx.html<p>Oftentimes you want to take a screenshot. On OSX this is easy: <code>⌘+SHIFT+4</code> and you're presented with a rectangle that
allows you to click & select what to capture. But what happens when you want things like capturing a tooltip, or the
mouse pointer itself.</p>
<p>As it turns out there's …</p><p>Oftentimes you want to take a screenshot. On OSX this is easy: <code>⌘+SHIFT+4</code> and you're presented with a rectangle that
allows you to click & select what to capture. But what happens when you want things like capturing a tooltip, or the
mouse pointer itself.</p>
<p>As it turns out there's a command line utility called <code>screencapture</code> that allows you to do this. To capture the screen
and include the mouse pointer, you can do:</p>
<div class="highlight"><pre><span></span>screencapture -T <span class="m">5</span> -C ~/Desktop/screencap.png
</pre></div>
<p>and then in 5 seconds the current screen will be saved to <code>~/Desktop/screencap.png</code> and include the mouse pointer in the
image.</p>
<p>But what happens when you have an external display? How do you capture those screens. Well, the answer is in the
<code>--help</code> output of the command which reads:</p>
<div class="highlight"><pre><span></span>$ screencapture --help
screencapture: illegal option -- -
usage: screencapture <span class="o">[</span>-icMPmwsWxSCUtoa<span class="o">]</span> <span class="o">[</span>files<span class="o">]</span>
-c force screen capture to go to the clipboard
-b capture Touch Bar - non-interactive modes only
-C capture the cursor as well as the screen. only <span class="k">in</span> non-interactive modes
-d display errors to the user graphically
-i capture screen interactively, by selection or window
control key - causes screen shot to go to clipboard
space key - toggle between mouse selection and
window selection modes
escape key - cancels interactive screen shot
-m only capture the main monitor, undefined <span class="k">if</span> -i is <span class="nb">set</span>
-M screen capture output will go to a new Mail message
-o <span class="k">in</span> window capture mode, <span class="k">do</span> not capture the shadow of the window
-P screen capture output will open <span class="k">in</span> Preview
-I screen capture output will <span class="k">in</span> a new Messages message
-s only allow mouse selection mode
-S <span class="k">in</span> window capture mode, capture the screen not the window
-t<format> image format to create, default is png <span class="o">(</span>other options include pdf, jpg, tiff and other formats<span class="o">)</span>
-T<seconds> Take the picture after a delay of <seconds>, default is <span class="m">5</span>
-w only allow window selection mode
-W start interaction <span class="k">in</span> window selection mode
-x <span class="k">do</span> not play sounds
-a <span class="k">do</span> not include windows attached to selected windows
-r <span class="k">do</span> not add dpi meta data to image
-l<windowid> capture this windowsid
-R<x,y,w,h> capture screen rect
-B<bundleid> screen capture output will open <span class="k">in</span> app with bundleidBS
files where to save the screen capture, <span class="m">1</span> file per screen
</pre></div>
<p>Note that last line, <code>files where to save the screen capture, 1 file per screen</code>. So we just add an additional file
per screen. So if you have 2 external displays, you'd do something like:</p>
<div class="highlight"><pre><span></span>screencapture -T <span class="m">5</span> -C ~/Desktop/screen1.png ~/Desktop/screen2.png ~/Desktop/screen3.png
</pre></div>
<p>And three files will be saved, one for each screen.</p>Scheduled Builds in Jenkins Scripted Pipelines2018-03-20T10:20:00-07:002018-03-20T10:20:00-07:00Adam Parkintag:www.codependentcodr.com,2018-03-20:/scheduled-builds-in-jenkins-scripted-pipelines.html<p>Sometimes you’ll want to have Jenkins trigger a build on a recurring schedule. Common examples include things like a
scheduled build at midnight (or some other time when regular work isn’t happening). Traditionally in Jenkins this could
be done on the configuration page for the job in question …</p><p>Sometimes you’ll want to have Jenkins trigger a build on a recurring schedule. Common examples include things like a
scheduled build at midnight (or some other time when regular work isn’t happening). Traditionally in Jenkins this could
be done on the configuration page for the job in question. You’d find the “Build Periodically” option under Build
Triggers and enter a crontab expression to schedule when Jenkins should trigger a build of the job automatically. With
pipelines though, that option disappears from the UI, instead you have to create it as code in your Jenkinsfile.</p>
<div class="highlight"><pre><span></span><span class="n">properties</span><span class="o">([</span><span class="n">pipelineTriggers</span><span class="o">([</span><span class="n">cron</span><span class="o">(</span><span class="s1">'H/15 * * * *'</span><span class="o">)])])</span>
</pre></div>
<p>Put this near the top (ie before a node declaration) of your Jenkinsfile and the job will get built automatically every
15 minutes. It’s worth noting that just making this change isn’t enough, you also have to have a build triggered with
this change in place to schedule the build (which makes sense — changes to Jenkinsfile’s don’t affect jobs until the new
Jenkinsfile is “run” by Jenkins, which doesn’t happen until a build is scheduled, be that by manual build, automatic
trigger from SCM push, etc)
This is fine and easy, but what if you’re using multibranch pipelines? You’ll likely then want to have different
schedules depending on the branch in question (ex: it’d seem silly to schedule a recurring build of a feature branch)
As it turns out this really isn’t too bad, you just need to inspect the <code>BRANCH_NAME</code> variable:</p>
<div class="highlight"><pre><span></span><span class="kt">def</span> <span class="n">triggers</span> <span class="o">=</span> <span class="o">[]</span>
<span class="k">if</span><span class="o">(</span><span class="s2">"$BRANCH_NAME"</span> <span class="o">==</span> <span class="s1">'develop'</span><span class="o">)</span> <span class="o">{</span>
<span class="n">triggers</span> <span class="o"><<</span> <span class="n">cron</span><span class="o">(</span><span class="s1">'H/15 * * * *'</span><span class="o">)</span> <span class="c1">// every 15 minutes</span>
<span class="o">}</span> <span class="k">else</span> <span class="k">if</span><span class="o">(</span><span class="s2">"$BRANCH_NAME"</span> <span class="o">==</span> <span class="s1">'mainline'</span><span class="o">)</span> <span class="o">{</span>
<span class="n">triggers</span> <span class="o"><<</span> <span class="n">cron</span><span class="o">(</span><span class="s1">'H H(0-2) * * *'</span><span class="o">)</span> <span class="c1">// daily between midnight & 2 AM</span>
<span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
<span class="c1">// no scheduled build</span>
<span class="o">}</span>
<span class="n">properties</span> <span class="o">(</span>
<span class="o">[</span>
<span class="n">pipelineTriggers</span><span class="o">(</span><span class="n">triggers</span><span class="o">)</span>
<span class="o">]</span>
<span class="o">)</span>
</pre></div>
<p>In this we set up two schedules for the project: if the branch is the develop branch, we build it every 15 minutes. If
the branch is our mainline branch, we build it every night between midnight and 2AM. If the branch isn’t develop or mainline
then we don’t schedule any automatic builds. Note that the else block is empty (I could have omitted it entirely), which
means that the triggers for the current branch will be cleared. Side note: this is also how you’d delete a previously
scheduled build for a branch, just clear the line which initializes the crontab schedule and then next time it’s build
the schedule will be cleared.</p>Python Tip of the Day - Logging basicConfig2018-02-14T10:20:00-08:002018-02-14T10:20:00-08:00Adam Parkintag:www.codependentcodr.com,2018-02-14:/python-tip-of-the-day-logging-basicconfig.html<p>Oftentimes you just want to try out something related to logging in the REPL, or in a hacky script. Wading through the
docs on the logging module is this painful exercise in reading about handlers and formatters and other stuff you don't
care about.</p>
<p>The simplest way to just get …</p><p>Oftentimes you just want to try out something related to logging in the REPL, or in a hacky script. Wading through the
docs on the logging module is this painful exercise in reading about handlers and formatters and other stuff you don't
care about.</p>
<p>The simplest way to just get the ability to do logging in the REPL:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="kn">import</span> <span class="nn">logging</span>
<span class="o">>>></span> <span class="kn">import</span> <span class="nn">sys</span>
<span class="o">>>></span> <span class="n">logging</span><span class="o">.</span><span class="n">basicConfig</span><span class="p">(</span><span class="n">stream</span><span class="o">=</span><span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="p">,</span> <span class="n">level</span><span class="o">=</span><span class="n">logging</span><span class="o">.</span><span class="n">DEBUG</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">logging</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s1">'This is a debug level logging message'</span><span class="p">)</span>
<span class="n">DEBUG</span><span class="p">:</span><span class="n">root</span><span class="p">:</span><span class="n">This</span> <span class="ow">is</span> <span class="n">a</span> <span class="n">debug</span> <span class="n">level</span> <span class="n">logging</span> <span class="n">message</span>
</pre></div>
<p>Simple as that.</p>Code Refactor of the Day - Extracting Compound Conditionals2018-01-15T10:20:00-08:002018-01-15T10:20:00-08:00Adam Parkintag:www.codependentcodr.com,2018-01-15:/code-refactor-of-the-day-extracting-compound-conditionals.html<p>I'm currently doing the 30-Day Code Quality Challenge (<a href="https://www.codequalitychallenge.com">https://www.codequalitychallenge.com</a>), and today's exercise was
an interesting one -- extract a compound conditional.</p>
<p>The idea of extracting a compound conditional is it's a refactor to try and improve the readability of code by giving a
name to a complex boolean …</p><p>I'm currently doing the 30-Day Code Quality Challenge (<a href="https://www.codequalitychallenge.com">https://www.codequalitychallenge.com</a>), and today's exercise was
an interesting one -- extract a compound conditional.</p>
<p>The idea of extracting a compound conditional is it's a refactor to try and improve the readability of code by giving a
name to a complex boolean expression. For example, say you have a block like:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">foo</span><span class="p">():</span>
<span class="k">if</span> <span class="n">user</span><span class="o">.</span><span class="n">state</span> <span class="o">==</span> <span class="s2">"active"</span> <span class="ow">and</span> <span class="n">user</span><span class="o">.</span><span class="n">birthdate</span><span class="o">.</span><span class="n">year</span> <span class="o">>=</span> <span class="mi">1990</span> <span class="ow">and</span> <span class="n">user</span><span class="o">.</span><span class="n">birthdate</span><span class="o">.</span><span class="n">year</span> <span class="o"><</span> <span class="mi">2000</span> <span class="ow">and</span> <span class="n">user</span><span class="o">.</span><span class="n">height_in_inches</span> <span class="o">></span> <span class="mi">72</span><span class="p">:</span>
<span class="o">...</span> <span class="n">do</span> <span class="n">something</span>
</pre></div>
<p>Now imagine we refactored it to look like this:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">foo</span><span class="p">():</span>
<span class="k">if</span> <span class="n">tall_active_user_born_in_the_90s</span><span class="p">(</span><span class="n">user</span><span class="p">):</span>
<span class="o">...</span> <span class="n">do</span> <span class="n">something</span>
<span class="k">def</span> <span class="nf">tall_active_user_born_in_the_90s</span><span class="p">(</span><span class="n">user</span><span class="p">):</span>
<span class="k">return</span> <span class="n">user</span><span class="o">.</span><span class="n">state</span> <span class="o">==</span> <span class="s2">"active"</span> <span class="ow">and</span> <span class="n">user</span><span class="o">.</span><span class="n">birthdate</span><span class="o">.</span><span class="n">year</span> <span class="o">>=</span> <span class="mi">1990</span> <span class="ow">and</span> <span class="n">user</span><span class="o">.</span><span class="n">birthdate</span><span class="o">.</span><span class="n">year</span> <span class="o"><</span> <span class="mi">2000</span> <span class="ow">and</span> <span class="n">user</span><span class="o">.</span><span class="n">height_in_inches</span> <span class="o">></span> <span class="mi">72</span>
</pre></div>
<p>I'd say the latter version of foo() reads much better & is much more self-documenting.</p>Python and Microsoft?2018-01-03T10:20:00-08:002018-01-03T10:20:00-08:00Adam Parkintag:www.codependentcodr.com,2018-01-03:/python-and-microsoft.html<p>Much has been said about how <a href="https://www.microsoft.com">Microsoft</a> has changed in recent years, no longer the super
closed-source monopolistic giant it once was. Regardless of whether or not you believe the change is real or just at
the surface, permanent or temporary, etc, there is definitely some interesting things that have …</p><p>Much has been said about how <a href="https://www.microsoft.com">Microsoft</a> has changed in recent years, no longer the super
closed-source monopolistic giant it once was. Regardless of whether or not you believe the change is real or just at
the surface, permanent or temporary, etc, there is definitely some interesting things that have come up recently. One
is how everyone's favourite programming language is starting to take a prominent role in many of Microsoft's offerings.</p>
<p>For example, in 2017 Microsoft announced on the <a href="https://blogs.technet.microsoft.com/dataplatforminsider/2017/04/19/python-in-sql-server-2017-enhanced-in-database-machine-learning/">SQL Server Blog that Python 3.5 would be embedded within SQL Server</a>
and be able to be used natively in that product. This is huge for the data science & machine learning communities. I
haven't yet played with this, but supposedly you can effectively write stored procedures using Python.</p>
<p>Another change illustrating the adoption of Python within the Microsoft ecosystem has been in Visual Studio Code land.
Originally Python support in VS Code came via a (really, really well made) extension written by Don Jayamanne. And
<a href="https://blogs.msdn.microsoft.com/pythonengineering/2017/11/09/don-jayamanne-joins-microsoft/">last November Don was hired by Microsoft full-time</a>
and the Python extension is now a fully MS-supported part of VS Code. <a href="https://snarky.ca/">Brett Canon</a> (a Python core
contributor who also works at Microsoft) is now the dev lead for the Python extension. And they're also actively hiring
looking for people for <a href="https://blogs.msdn.microsoft.com/pythonengineering/2017/11/09/don-jayamanne-joins-microsoft/">"Visual Studio Code / Python!"</a></p>
<p>Lastly, there's been a bunch of noise (see
<a href="https://www.bleepingcomputer.com/news/microsoft/microsoft-considers-adding-python-as-an-official-scripting-language-to-excel/">here</a>,
<a href="https://news.ycombinator.com/item?id=15927132">here</a>,
and <a href="https://excel.uservoice.com/forums/304921-excel-for-windows-desktop-application/suggestions/10549005-python-as-an-excel-scripting-language">here</a>)
about how Microsoft is now actively seeking input on how Python could potentially be used as potentially a replacement
for VBA as the macro/scripting language in Excel. It's still way too early on this to see what will come of it, but
imagine a world where managers who spend their lives in Excel get the power of Python at their fingertips and what that
could potentially do for raising the exposure of the language.</p>
<p>Does all this indicate a trend? I hope so, I think the adoption of Python in Microsoft products will help boost the
popularity of the language even further (and
<a href="https://stackoverflow.blog/2017/09/06/incredible-growth-python/">it had some explosive growth in 2017</a>).
Interesting times all the same.</p>Why not exact story point estimates?2017-12-12T10:20:00-08:002017-12-12T10:20:00-08:00Adam Parkintag:www.codependentcodr.com,2017-12-12:/why-not-exact-story-point-estimates.html<p>Awhile back at a job a question was raised in a sprint planning meeting about why we don't do exact story point
estimation, instead of doing the fibonacci (or fibonacci-like) scales. We had scales of 0.25, 0.5, 1, 3, 5, 8, 13, etc
at the place, and people …</p><p>Awhile back at a job a question was raised in a sprint planning meeting about why we don't do exact story point
estimation, instead of doing the fibonacci (or fibonacci-like) scales. We had scales of 0.25, 0.5, 1, 3, 5, 8, 13, etc
at the place, and people were asking why not "0.75"?</p>
<p>Being a topic I'm quite interested in, I wrote a long email explaining my thoughts on the subject, and now the same
conversation has come up again at my current job. As such I thought it might be useful to share that email with the
world. The following is that email, slightly reworded to be a bit more blog-entry friendly. So "why don't you just use
exact story point estimates"?</p>
<p>I'll reflect the question: "Why do we use story points at all? Why not just estimate in hours?" This is a really good
question, why don't we just say "4 hours" instead of "0.5 SP". If we did that, then we'd no longer have this funny
problem of a ticket that's really 30 minutes having an equivalent estimate to something that takes 2 hours. So if we
did that, then we'd be better off, as we could then start estimating more precisely. Instead of 4 tickets at 0.25 SP
each totalling 1 SP, which means 1 day, we might have 15 minutes + 1 hour + 2 hours + 45 minutes == 4 hours, so if we go
that route then we could pack in even more stuff to a sprint, and get even more stuff done. Right?</p>
<p>Well, not so much. The first problem with time estimates is it makes a fundamentally invalid assumption: that all
developers are created equal. A senior dev & a junior dev might agree entirely on what exactly needs to be done to
complete a task, but I can pretty much guarantee that it'll take the senior dev less time than the junior (regardless of
task). So if we estimate in time units then suddenly we have this problem of do you estimate to the level of the person
who's really experienced, or to the junior person? A fibonacci style sequence of "bucket sizes" (like 0.25, 0.5, 1, 2,
5, etc) helps with this (if it takes the junior 9 hours, and a senior 5, then the SP estimate will likely be the same --
1). (Mike Cohn, the Scrum Alliance dude, has a few blog posts on Story points to this effect, see
<a href="https://www.mountaingoatsoftware.com/blog/the-main-benefit-of-story-points">this</a> and
<a href="https://www.mountaingoatsoftware.com/blog/dont-equate-story-points-to-hours">this</a>)</p>
<p>So that's one problem with exact time estimates, but there's a bigger one: behavioural psychology. There's actually a
lot of research that's been done in behavioural psych that shows that while people are really good at <em>relative</em>
estimating, they are terrible at <em>exact</em> estimating, particularly in a domain where there is a lot of uncertainty, or a
lack of repetition (like software development, where you're often asked to do things or work with technologies you've
never done or used before).</p>
<p>I'll give you an analogy (this was the same example that a former scrum master of mine used with me): say I pointed at
two buildings, one that was 50 ft tall and one that was 97ft tall and said "about how much bigger is the second one to
the first?" You'd look at them, and even with no knowledge of carpentry, architecture, civil engineering, etc, you'd
probably be able to say with a high degree of confidence that the second is about twice as big and you'd be pretty darn
close. Now let's say I asked "how many feet taller is the second building compared to the first?" Now, maybe if you're
really experienced at knowing the hights of buildings, you'll be able to come up with a number that's close to the
actual difference, but I sure wouldn't, in fact I'd probably get hung up on being "perfect" with the estimate and end up
spending a disproportionate amount of time "estimating". That's relative sizing vs exact sizing, and we're wired such
that we're good at the former, but not so much at the latter.</p>
<p>Now lets take the story even further. Lets say I point at two buildings and one is 50 ft tall and the other is 600 feet
tall (12 times bigger). Now let's say I ask you the same relative sizing question. Suddenly because the magnitude of
difference is so high it becomes more difficult to do the relative estimating, but you might say "10 times bigger" and
you'd be not far off. But the real question: does it matter (when the difference is so great) that you're off slightly
with that estimate? Probably not, and that's why with SP estimates the scale is usually fibonacci like -- past 5 the
numbers get bigger faster because the relative differences are so huge it's not meaningful to be as precise.</p>
<p>I used to have links to a bunch of research papers talking about this, but have unfortunately since lost them. It's
actually really interesting stuff.</p>
<p>In any case, so pulling it back: that's why we don't do 0.75 SP, because one of the big points of story point estimating
is to do relative estimating. Once we start splitting hairs then there really is no point in doing SP estimates at all,
and instead just estimate everything in hours (which is problematic).</p>Python Tip of the Day - subTest!2017-09-23T10:20:00-07:002017-09-23T10:20:00-07:00Adam Parkintag:www.codependentcodr.com,2017-09-23:/python-tip-of-the-day-subtest.html<p>Coming from a <a href="https://junit.org/junit5/">jUnit</a> background, one of the things I always missed with the vanilla
<a href="https://docs.python.org/3/library/unittest.html">Python unitttest library</a> was parameterized tests. Oftentimes when
writing unit tests for a particular unit you find yourself writing effectively the same test over and over again, but
with different inputs. Wouldn't it be nice …</p><p>Coming from a <a href="https://junit.org/junit5/">jUnit</a> background, one of the things I always missed with the vanilla
<a href="https://docs.python.org/3/library/unittest.html">Python unitttest library</a> was parameterized tests. Oftentimes when
writing unit tests for a particular unit you find yourself writing effectively the same test over and over again, but
with different inputs. Wouldn't it be nice if we could write the test once and somehow parameterize the test with
different inputs? Yes. Yes it would.</p>
<p><a href="https://docs.pytest.org/en/latest/example/parametrize.html">Py.test supports this</a>, but what if you really like
<a href="http://nose.readthedocs.io/en/latest/">Nose</a> or some other test runner? Well, in Python 3.4 now in the standard
unittest library we have something very similar -- <a href="https://docs.python.org/3/library/unittest.html#distinguishing-test-iterations-using-subtests">subTests</a>.</p>
<p>The idea is you can write a loop over a set of inputs, and within that loop define a test within a with <code>self.subTest()</code>
context. Each iteration will test the given input and failures for each are counted as separate test failures. Really
handy for cutting down on unit test boilerplate code.</p>Serverless Microservices and Python (with tests!) - Part 22017-07-28T10:20:00-07:002017-07-28T10:20:00-07:00Adam Parkintag:www.codependentcodr.com,2017-07-28:/serverless-microservices-and-python-with-tests-part-2.html<p>Ok, so in <a href="https://www.codependentcodr.com/serverless-microservices-and-python-with-tests-part-1.html">part 1</a> of this series, I started off by exploring the
use of <a href="https://aws.amazon.com/lambda/">Lambda</a> and
<a href="https://aws.amazon.com/api-gateway/">API Gateway</a> as a tool for building scalable microservices in Python. I largely
focussed on taking an existing tutorial, and building out some unit tests for it, as well as some supplementary …</p><p>Ok, so in <a href="https://www.codependentcodr.com/serverless-microservices-and-python-with-tests-part-1.html">part 1</a> of this series, I started off by exploring the
use of <a href="https://aws.amazon.com/lambda/">Lambda</a> and
<a href="https://aws.amazon.com/api-gateway/">API Gateway</a> as a tool for building scalable microservices in Python. I largely
focussed on taking an existing tutorial, and building out some unit tests for it, as well as some supplementary scripts
to make bundling stuff up for delivery to Lambda easier.</p>
<p>In this entry, I'm going to explore adding a new requirement to the existing project -- supporting
<a href="https://en.wikipedia.org/wiki/Bcrypt">bcrypt</a> as a digest.</p>
<p>So to begin with, since I'm a big <a href="https://en.wikipedia.org/wiki/Test-driven_development">TDD</a> fan, I'm going to do this
by first adding a test, then making the test green, then refactoring. If you want to see the code as it was at this
point, I <a href="https://github.com/pzelnip/lambda-password-service/releases/tag/blog-entry-part1">tagged the commit I was at in Github</a></p>
<p>So first things first, lets start with a (failing) test (leaving out the rest of the test file for brevity):</p>
<div class="highlight"><pre><span></span><span class="n">SAMPLE_BCRYPT_HASH</span> <span class="o">=</span> <span class="s1">'$2b$12$44roRI0Ftbbvoy6V1YQebOKeO7a7WhzRvv.X194BMxykDT0nQGcS2'</span>
<span class="o">...</span>
<span class="k">def</span> <span class="nf">test_valid_bcrypt_hash_with_matching_password_returns_true</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">event</span> <span class="o">=</span> <span class="n">_build_event</span><span class="p">(</span><span class="s1">'bcrypt'</span><span class="p">,</span> <span class="n">SAMPLE_BCRYPT_HASH</span><span class="p">,</span> <span class="n">SAMPLE_PASSWORD</span><span class="p">)</span>
<span class="n">expected</span> <span class="o">=</span> <span class="kc">True</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">lambda_handler</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">expected</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
</pre></div>
<p>Run it, and yup, it's red. So lets make it green, by modifying the lambda_handler function:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">lambda_handler</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
<span class="n">digest</span> <span class="o">=</span> <span class="n">event</span><span class="p">[</span><span class="s1">'digest'</span><span class="p">]</span>
<span class="n">hash_pass</span> <span class="o">=</span> <span class="n">event</span><span class="p">[</span><span class="s1">'hash_pass'</span><span class="p">]</span>
<span class="n">password</span> <span class="o">=</span> <span class="n">event</span><span class="p">[</span><span class="s1">'password'</span><span class="p">]</span>
<span class="k">if</span> <span class="n">digest</span> <span class="o">==</span> <span class="s2">"bcrypt"</span><span class="p">:</span>
<span class="k">return</span> <span class="kc">True</span>
<span class="o">...</span> <span class="n">rest</span> <span class="n">of</span> <span class="n">function</span> <span class="ow">is</span> <span class="n">the</span> <span class="n">same</span> <span class="o">...</span>
</pre></div>
<p>Wait, what? Always return True when the digest is bcrypt? Yup, this is the TDD way, write the simplest code possible to
make the test green & then revise. We don't yet have a test that says when the password doesn't match the hash and the
digest is bcrypt you should return <code>False</code>, so let's add one:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">test_valid_sha1_hash_with_wrong_password_returns_false</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">event</span> <span class="o">=</span> <span class="n">_build_event</span><span class="p">(</span><span class="s1">'bcrypt'</span><span class="p">,</span> <span class="n">SAMPLE_BCRYPT_HASH</span><span class="p">,</span> <span class="s1">'this is not the password'</span><span class="p">)</span>
<span class="n">expected</span> <span class="o">=</span> <span class="kc">False</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">lambda_handler</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">expected</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
</pre></div>
<p>Now, we need to revise lambda_handler to handle both cases with bcrypt. Some may feel like this is silly, but this is
the heart of TDD: taking the smallest possible steps to keep the code concise and ensure you have tests to handle the
cases you think. If we had gone ahead and done the "real" solution for bcrypt (seen below) right away, then we'd only
have half the tests for bcrypt. If we added the false test after the fact it'd have been green upon completing writing
it, and that means you have an unverified test (in this toy example it's silly to be this pedantic, but take my word for
it -- if you've never seen a test fail when you expect it to, it's not a valid test).</p>
<p>So anyways, silly pedantic example aside, let's go ahead and solve it for real:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">passlib.hash</span> <span class="kn">import</span> <span class="n">pbkdf2_sha256</span><span class="p">,</span> <span class="n">pbkdf2_sha512</span><span class="p">,</span> <span class="n">pbkdf2_sha1</span><span class="p">,</span> <span class="n">bcrypt</span>
<span class="k">def</span> <span class="nf">lambda_handler</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
<span class="n">digest</span> <span class="o">=</span> <span class="n">event</span><span class="p">[</span><span class="s1">'digest'</span><span class="p">]</span>
<span class="n">hash_pass</span> <span class="o">=</span> <span class="n">event</span><span class="p">[</span><span class="s1">'hash_pass'</span><span class="p">]</span>
<span class="n">password</span> <span class="o">=</span> <span class="n">event</span><span class="p">[</span><span class="s1">'password'</span><span class="p">]</span>
<span class="k">if</span> <span class="n">digest</span> <span class="o">==</span> <span class="s2">"bcrypt"</span><span class="p">:</span>
<span class="n">verification</span> <span class="o">=</span> <span class="n">bcrypt</span><span class="o">.</span><span class="n">verify</span><span class="p">(</span><span class="n">password</span><span class="p">,</span> <span class="n">hash_pass</span><span class="p">)</span>
<span class="o">...</span> <span class="n">rest</span> <span class="n">of</span> <span class="n">function</span> <span class="ow">is</span> <span class="n">the</span> <span class="n">same</span> <span class="o">...</span>
</pre></div>
<p>And now you run the tests and they're gree..err...I mean red. WTF?</p>
<div class="highlight"><pre><span></span><span class="n">MissingBackendError</span><span class="p">:</span> <span class="n">bcrypt</span><span class="p">:</span> <span class="n">no</span> <span class="n">backends</span> <span class="n">available</span> <span class="o">--</span> <span class="n">recommend</span> <span class="n">you</span> <span class="n">install</span> <span class="n">one</span> <span class="p">(</span><span class="n">e</span><span class="o">.</span><span class="n">g</span><span class="o">.</span> <span class="s1">'pip install bcrypt'</span><span class="p">)</span>
</pre></div>
<p>Oh yeah, we need bcrypt installed. No biggie, just add bcrypt to our requirements.txt file and voila after <code>pip install
-r requirements.txt</code>into our development virtual environment and we're good.</p>
<p>Sweet, now we have bcrypt support, and tests are green. Now we can refactor things a bit for simplicity. Look at our
<code>lambda_handler</code> function, there's a big nasty if/else block that's kinda icky:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">lambda_handler</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
<span class="n">digest</span> <span class="o">=</span> <span class="n">event</span><span class="p">[</span><span class="s1">'digest'</span><span class="p">]</span>
<span class="n">hash_pass</span> <span class="o">=</span> <span class="n">event</span><span class="p">[</span><span class="s1">'hash_pass'</span><span class="p">]</span>
<span class="n">password</span> <span class="o">=</span> <span class="n">event</span><span class="p">[</span><span class="s1">'password'</span><span class="p">]</span>
<span class="k">if</span> <span class="n">digest</span> <span class="o">==</span> <span class="s2">"sha256"</span><span class="p">:</span>
<span class="n">verification</span> <span class="o">=</span> <span class="n">pbkdf2_sha256</span><span class="o">.</span><span class="n">verify</span><span class="p">(</span><span class="n">password</span><span class="p">,</span> <span class="n">hash_pass</span><span class="p">)</span>
<span class="k">elif</span> <span class="n">digest</span> <span class="o">==</span> <span class="s2">"sha512"</span><span class="p">:</span>
<span class="n">verification</span> <span class="o">=</span> <span class="n">pbkdf2_sha512</span><span class="o">.</span><span class="n">verify</span><span class="p">(</span><span class="n">password</span><span class="p">,</span> <span class="n">hash_pass</span><span class="p">)</span>
<span class="k">elif</span> <span class="n">digest</span> <span class="o">==</span> <span class="s2">"bcrypt"</span><span class="p">:</span>
<span class="n">verification</span> <span class="o">=</span> <span class="n">bcrypt</span><span class="o">.</span><span class="n">verify</span><span class="p">(</span><span class="n">password</span><span class="p">,</span> <span class="n">hash_pass</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">verification</span> <span class="o">=</span> <span class="n">pbkdf2_sha1</span><span class="o">.</span><span class="n">verify</span><span class="p">(</span><span class="n">password</span><span class="p">,</span> <span class="n">hash_pass</span><span class="p">)</span>
<span class="k">return</span> <span class="n">verification</span>
</pre></div>
<p>Let's simplify by creating a mapping of strings to functions:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">passlib.hash</span> <span class="kn">import</span> <span class="n">pbkdf2_sha256</span><span class="p">,</span> <span class="n">pbkdf2_sha512</span><span class="p">,</span> <span class="n">pbkdf2_sha1</span><span class="p">,</span> <span class="n">bcrypt</span>
<span class="n">HASH_MAPPINGS</span> <span class="o">=</span> <span class="p">{</span>
<span class="s2">"sha256"</span><span class="p">:</span> <span class="n">pbkdf2_sha256</span><span class="p">,</span>
<span class="s2">"sha512"</span><span class="p">:</span> <span class="n">pbkdf2_sha512</span><span class="p">,</span>
<span class="s2">"bcrypt"</span><span class="p">:</span> <span class="n">bcrypt</span><span class="p">,</span>
<span class="s2">"sha1"</span><span class="p">:</span> <span class="n">pbkdf2_sha1</span><span class="p">,</span>
<span class="p">}</span>
<span class="n">DEFAULT_HASH</span> <span class="o">=</span> <span class="n">pbkdf2_sha1</span>
<span class="k">def</span> <span class="nf">lambda_handler</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
<span class="n">digest</span> <span class="o">=</span> <span class="n">event</span><span class="p">[</span><span class="s1">'digest'</span><span class="p">]</span>
<span class="n">hash_pass</span> <span class="o">=</span> <span class="n">event</span><span class="p">[</span><span class="s1">'hash_pass'</span><span class="p">]</span>
<span class="n">password</span> <span class="o">=</span> <span class="n">event</span><span class="p">[</span><span class="s1">'password'</span><span class="p">]</span>
<span class="n">hash_fn</span> <span class="o">=</span> <span class="n">HASH_MAPPINGS</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">digest</span><span class="p">,</span> <span class="n">DEFAULT_HASH</span><span class="p">)</span>
<span class="k">return</span> <span class="n">hash_fn</span><span class="o">.</span><span class="n">verify</span><span class="p">(</span><span class="n">password</span><span class="p">,</span> <span class="n">hash_pass</span><span class="p">)</span>
</pre></div>
<p>Much shorter. Now to add new digests we simply add a new entry to HASH_MAPPINGS. One thing is bothering me though, right
now if one fails to specify hash_pass as an arg, the lambda function blows up as a KeyError gets thrown. This is again
hitting that "what's the requirement?" issue, but I felt like what should happen is that instead of a 500 server error
on Lambda you should instead just get a response of False (no password matches an unspecified hash). Unit test:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">test_unspecified_hash_pass_returns_false</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">event</span> <span class="o">=</span> <span class="n">_build_event</span><span class="p">(</span><span class="s1">'bcrypt'</span><span class="p">,</span> <span class="n">SAMPLE_BCRYPT_HASH</span><span class="p">,</span> <span class="s1">'password'</span><span class="p">)</span>
<span class="k">del</span> <span class="n">event</span><span class="p">[</span><span class="s1">'hash_pass'</span><span class="p">]</span>
<span class="n">expected</span> <span class="o">=</span> <span class="kc">False</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">lambda_handler</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">expected</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
</pre></div>
<p>And (after verifying this was red), making it green:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">lambda_handler</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
<span class="n">digest</span> <span class="o">=</span> <span class="n">event</span><span class="p">[</span><span class="s1">'digest'</span><span class="p">]</span>
<span class="n">hash_pass</span> <span class="o">=</span> <span class="n">event</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'hash_pass'</span><span class="p">)</span>
<span class="n">password</span> <span class="o">=</span> <span class="n">event</span><span class="p">[</span><span class="s1">'password'</span><span class="p">]</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">hash_pass</span><span class="p">:</span>
<span class="k">return</span> <span class="kc">False</span>
</pre></div>
<p>Similarly, we already specified that an invalid digest ends up using SHA1, so let's make the value in the event dict
completely optional. First the test:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">test_unspecified_digest_uses_sha1</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">event</span> <span class="o">=</span> <span class="n">_build_event</span><span class="p">(</span><span class="s1">'does not matter'</span><span class="p">,</span> <span class="n">SAMPLE_SHA1_HASH</span><span class="p">,</span> <span class="n">SAMPLE_PASSWORD</span><span class="p">)</span>
<span class="k">del</span> <span class="n">event</span><span class="p">[</span><span class="s1">'digest'</span><span class="p">]</span>
<span class="n">expected</span> <span class="o">=</span> <span class="kc">True</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">lambda_handler</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">expected</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
</pre></div>
<p>And the change to make it green:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">lambda_handler</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
<span class="n">digest</span> <span class="o">=</span> <span class="n">event</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'digest'</span><span class="p">,</span> <span class="n">DEFAULT_HASH</span><span class="p">)</span>
<span class="n">hash_pass</span> <span class="o">=</span> <span class="n">event</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'hash_pass'</span><span class="p">)</span>
<span class="n">password</span> <span class="o">=</span> <span class="n">event</span><span class="p">[</span><span class="s1">'password'</span><span class="p">]</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">hash_pass</span><span class="p">:</span>
<span class="k">return</span> <span class="kc">False</span>
<span class="n">hash_fn</span> <span class="o">=</span> <span class="n">HASH_MAPPINGS</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">digest</span><span class="p">,</span> <span class="n">HASH_MAPPINGS</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">DEFAULT_HASH</span><span class="p">))</span>
<span class="k">return</span> <span class="n">hash_fn</span><span class="o">.</span><span class="n">verify</span><span class="p">(</span><span class="n">password</span><span class="p">,</span> <span class="n">hash_pass</span><span class="p">)</span>
</pre></div>
<p>password is still a required argument and results in a 500 server error, but we'll revisit that one later. We've made
some real progress, refactored the code to be much more versatile & concise, added an entire new digest, and validated
all this behaviour locally. Now it's time to throw it all to Lambda. Run <code>build.sh</code> and throw it all up to lambda, and uh-oh:</p>
<div class="highlight"><pre><span></span><span class="p">{</span>
<span class="s2">"stackTrace"</span><span class="o">:</span> <span class="p">[</span>
<span class="p">[</span>
<span class="s2">"/var/task/index.py"</span><span class="p">,</span>
<span class="mf">23</span><span class="p">,</span>
<span class="s2">"lambda_handler"</span><span class="p">,</span>
<span class="s2">"return hash_fn.verify(password, hash_pass)"</span>
<span class="p">],</span>
<span class="p">[</span>
<span class="s2">"/var/task/passlib/utils/handlers.py"</span><span class="p">,</span>
<span class="mf">761</span><span class="p">,</span>
<span class="s2">"verify"</span><span class="p">,</span>
<span class="s2">"return consteq(self._calc_checksum(secret), chk)"</span>
<span class="p">],</span>
<span class="p">[</span>
<span class="s2">"/var/task/passlib/handlers/bcrypt.py"</span><span class="p">,</span>
<span class="mf">530</span><span class="p">,</span>
<span class="s2">"_calc_checksum"</span><span class="p">,</span>
<span class="s2">"self._stub_requires_backend()"</span>
<span class="p">],</span>
<span class="p">[</span>
<span class="s2">"/var/task/passlib/utils/handlers.py"</span><span class="p">,</span>
<span class="mf">2221</span><span class="p">,</span>
<span class="s2">"_stub_requires_backend"</span><span class="p">,</span>
<span class="s2">"cls.set_backend()"</span>
<span class="p">],</span>
<span class="p">[</span>
<span class="s2">"/var/task/passlib/utils/handlers.py"</span><span class="p">,</span>
<span class="mf">2143</span><span class="p">,</span>
<span class="s2">"set_backend"</span><span class="p">,</span>
<span class="s2">"raise default_error"</span>
<span class="p">]</span>
<span class="p">],</span>
<span class="s2">"errorType"</span><span class="o">:</span> <span class="s2">"MissingBackendError"</span><span class="p">,</span>
<span class="s2">"errorMessage"</span><span class="o">:</span> <span class="s2">"bcrypt: no backends available -- recommend you install one (e.g. 'pip install bcrypt')"</span>
<span class="p">}</span>
</pre></div>
<p>This is the stacktrace you get. What's up, I thought we included bcrypt in the zip file? Unzipping the zip file and
verifying the contents we see that it was included, but, and this is a gotcha with Lambda, bcrypt has some external
compiled dependencies -- it's not pure Python. I'm developing on a Macbook running OSX El Capitan which is a much
different environment than Amazon Linux (which is what a Lambda container runs in).</p>
<p>So, this is where it gets interesting. I started off doing some googling, and found
<a href="https://github.com/Miserlou/lambda-packages">this one</a>, which is some common Python libraries with compiled
dependencies built for Amazon Linux. Theoretically you should be able to specify that as a dependency in your
<code>requirements.txt</code>, build it, and be good to go. So I tried this, and low and behold now my zip file is larger than the
50MB for uploading through the Lambda web interface. Throwing a zip file into an S3 bucket is simple enough, so I did
that, and then saved my Lambda function and tried again.</p>
<p>And got the same <code>MissingBackendError</code>. Yup, dependency hell.</p>
<p>So I dropped this approach. Even if it had worked, that's going to make your dev environment and your prod environment a
little different (in dev I'd still be dependent upon bcrypt, in prod upon lambda-packages) which is a smell.</p>
<p>Supposedly you can
<a href="https://markn.ca/2015/10/python-extension-modules-in-aws-lambda/">spin up an EC2 instance based on the Amazon Linux AMI and do your bundling for lambda there</a>
, but that's far from convenient (you need to spin up an EC2 instance, get your repository there, do the whole build,
then get the zip file from that instance to wherever you need it to be). Alternatively, there's
<a href="https://github.com/lambci/docker-lambda">a Docker image out there that mimics the Amazon Linux image that Lambda uses</a>,
so you could (locally) run a container from that image and do the same thing (<code>pip install</code>, bundle it into a zip, etc).
But this is really getting into a world I don't really want to go (at least not for now), so I did some more Googling
and found that passlib actually supports <a href="http://passlib.readthedocs.io/en/stable/lib/passlib.hash.bcrypt.html">5 different bcrypt implementations (or "backends")</a>:</p>
<ul>
<li>bcrypt, if installed.</li>
<li>py-bcrypt, if installed.</li>
<li>bcryptor, if installed.</li>
<li>stdlib’s <code>crypt.crypt()</code>, if the host OS supports BCrypt (primarily BSD-derived systems).</li>
</ul>
<p>A pure-python implementation of BCrypt, built into Passlib.
And that last one is turned off by default as it's just too damn slow. For now though, we just want something that works,
and is easy (we'll optimize later), so let's enable that backend. This is done by set the environmental variable
<code>PASSLIB_BUILTIN_BCRYPT="enabled"</code> where you're running passlib. With Lambda, setting some env variables is easy, you
can do this in the web interface:</p>
<p><img alt="Setting Environment Vars in Lambda" src="https://www.codependentcodr.com/static/imgs/screen-shot-2017-07-27-at-2-16-32-pm.png"></p>
<p>Doing this, I no longer got a <code>MissingBackendError</code>, but now there was a new problem:</p>
<div class="highlight"><pre><span></span><span class="p">{</span>
<span class="s2">"errorMessage"</span><span class="o">:</span> <span class="s2">"2017-07-27T21:17:09.542Z f0af983b-7310-11e7-8079-97327f3cc568 Task timed out after 3.00 seconds"</span>
<span class="p">}</span>
</pre></div>
<p>Yup, apparently that plain Python version is in fact just way too slow. You can extend the timeout value for a Lambda
function on the Configuration tab under advanced items:</p>
<p><img alt="Extending Lambda Timeout" src="https://www.codependentcodr.com/static/imgs/screen-shot-2017-07-27-at-2-19-06-pm.png"></p>
<p>It's worth noting this can increase your costs with Lambda, as pricing is execution-time related. With that change in
place (50 seconds is way long, but just trying to get it to work), I got a new error, this time from API Gateway:</p>
<div class="highlight"><pre><span></span><span class="p">{</span>
<span class="s2">"message"</span><span class="o">:</span> <span class="s2">"Endpoint request timed out"</span>
<span class="p">}</span>
</pre></div>
<p>This was after running for about 30 seconds. I assumed this was timeout for API Gateway, and
<a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/limits.html">this page</a> confirmed it. Unfortunately it's
not possible to change this either.</p>
<p>So back to the drawing board....</p>
<p>In part 3 I'm going to continue from here, looking into perhaps doing the compiled dependency on a Amazon Linux based
box route.</p>Serverless Microservices and Python (with tests!) - Part 12017-07-27T10:20:00-07:002017-07-27T10:20:00-07:00Adam Parkintag:www.codependentcodr.com,2017-07-27:/serverless-microservices-and-python-with-tests-part-1.html<p>So I'm currently on holiday and also between jobs (had my last day at old job last week, and first day at new gig is
next week), which means of course what am I doing but spending some time learning some tech that's fun & buzzwordy.</p>
<p>Right now it seems like …</p><p>So I'm currently on holiday and also between jobs (had my last day at old job last week, and first day at new gig is
next week), which means of course what am I doing but spending some time learning some tech that's fun & buzzwordy.</p>
<p>Right now it seems like you can't listen to a tech podcast without hearing "microservices" or "serverless", especially
if you listen to anything with a devops bias. So, why not explore both? I've always wanted to learn a bit more about
<a href="https://docs.aws.amazon.com/lambda/latest/dg/welcome.html">AWS Lambda</a> and in particular the combination of Lambda with
<a href="https://aws.amazon.com/api-gateway/">AWS API Gateway</a> to create little microservices that are supremely scalable without
the headache of server maintenance. Did some Googling and
<a href="http://dchua.com/2016/03/22/writing-a-serverless-python-microservice-with-aws-lambda-and-aws-api-gateway/">stumbled across this tutorial</a>
which seemed like exactly what I was looking for.</p>
<p>So, I worked through the tutorial, and minor hiccups aside, got a simple little password verification microservice up
and running in almost no time at all. Sweet.</p>
<p>Ok, so for me, when I do tutorials like this, I find I need to build or extend the exercise to help reinforce what I've
learned. Aside from that, one of the questions I have about Lambda projects is how does testing work? Do you still do
unit testing like you would with a regular Python project? Any differences?</p>
<p>So, let's take this example and enhance it with a new requirement -- support
<a href="https://en.wikipedia.org/wiki/Bcrypt">Bcrypt</a> as a digest.</p>
<p>Now, there's a problem (ok, this is contrived, work with me here): typically before you start adding new functionality
you want to ensure you have a decent set of automated tests to ensure that you don't break existing behaviour. So, step
1: let's add some unit tests that enforce the existing requirements we have in our little Lambda function. I saw these as:</p>
<ul>
<li>supports three digests: <code>SHA1</code>, <code>SHA256</code>, and <code>SHA512</code>.</li>
<li>when given a valid hash for a digest, and the plaintext password that hash was based upon, return <code>True</code></li>
<li>when given an valid hash for a digest, and a random string (that doesn't match the hash), return <code>False</code></li>
</ul>
<p>Simple enough. So let's get cracking. First thing I did was start to "project-ize" this code, so that it's more than a
random Python file. This consisted of creating a requirements.txt file to list the dependencies the project uses
(currently only <code>passlib</code>), and to move it into a project in my IDE of choice. I like to use
<a href="https://www.jetbrains.com/pycharm/">PyCharm</a> as my dev environment, so I fired up PyCharm and created a new project
based upon the virtual environment created from the <code>requirements.txt</code> file. Next I did a bit of restructuring moving
the source file into a directory called <code>src</code> and created a sibling directory called <code>test</code>. I like to structure my
Python projects this way, but really this is arbitrary and personal convention more than anything.</p>
<p>With all that in place, I added index_test.py (mirroring the index.py name that was created in the tutorial) and started
backfilling some tests. Note that since <code>lambda_handler</code> is just a plain old Python function, unit testing is actually
completely straightforward. A first stab:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">unittest</span>
<span class="kn">from</span> <span class="nn">index</span> <span class="kn">import</span> <span class="n">lambda_handler</span>
<span class="k">class</span> <span class="nc">TestLambdaHandler</span><span class="p">(</span><span class="n">unittest</span><span class="o">.</span><span class="n">TestCase</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">test_valid_sha256_hash_with_matching_password_returns_true</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">event</span> <span class="o">=</span> <span class="p">{</span>
<span class="s2">"digest"</span><span class="p">:</span> <span class="s2">"sha256"</span><span class="p">,</span>
<span class="s2">"hash_pass"</span><span class="p">:</span> <span class="s2">"$pbkdf2-sha256$29000$.L93bg0BwFiLEaL0fm8NIQ$yYmxiSuP9pXXbrO4cT6CkE1QaNKpt8PjugrgvOBfcRY"</span><span class="p">,</span>
<span class="s2">"password"</span><span class="p">:</span> <span class="s2">"password"</span>
<span class="p">}</span>
<span class="n">expected</span> <span class="o">=</span> <span class="kc">True</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">lambda_handler</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">expected</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_valid_sha256_hash_with_wrong_password_returns_false</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">event</span> <span class="o">=</span> <span class="p">{</span>
<span class="s2">"digest"</span><span class="p">:</span> <span class="s2">"sha256"</span><span class="p">,</span>
<span class="s2">"hash_pass"</span><span class="p">:</span> <span class="s2">"$pbkdf2-sha256$29000$.L93bg0BwFiLEaL0fm8NIQ$yYmxiSuP9pXXbrO4cT6CkE1QaNKpt8PjugrgvOBfcRY"</span><span class="p">,</span>
<span class="s2">"password"</span><span class="p">:</span> <span class="s2">"this is not the password"</span>
<span class="p">}</span>
<span class="n">expected</span> <span class="o">=</span> <span class="kc">False</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">lambda_handler</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">expected</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
</pre></div>
<p>Again, all straightforward stuff. My style of test writing is to follow the
<a href="http://wiki.c2.com/?ArrangeActAssert">Arrange, Act, Assert pattern</a>, as I find this helps with readability. In terms of
running them, I personally just ran these with the default test runner from within PyCharm, but there's nothing magical
here, so you could just as easily run them with your favourite runner (be it
<a href="http://nose.readthedocs.io/en/latest/">Nose</a>,
<a href="https://docs.pytest.org/en/latest/">py.test</a> or whatever).</p>
<p>As is usually the case with writing tests, you start to find duplication and simplify. In both of these the event
declaration is a bit verbose, so lets break it into a helper, and add some tests for other digests:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">unittest</span>
<span class="kn">from</span> <span class="nn">index</span> <span class="kn">import</span> <span class="n">lambda_handler</span>
<span class="n">SAMPLE_PASSWORD</span> <span class="o">=</span> <span class="s1">'password'</span>
<span class="n">SAMPLE_SHA512_HASH</span> <span class="o">=</span> <span class="s1">'$pbkdf2-sha512$25000$ltLae69VihFirDVGSOmdUw$pcLVv3Vnm3XRx9aHNUgI1FQaF8.UmKHBYt.Hs2EI7at/V80kbsb2P1A2t9akjNom8ZUgVJ4AcbA5vk/7QTgEJQ'</span>
<span class="n">SAMPLE_SHA256_HASH</span> <span class="o">=</span> <span class="s1">'$pbkdf2-sha256$29000$.L93bg0BwFiLEaL0fm8NIQ$yYmxiSuP9pXXbrO4cT6CkE1QaNKpt8PjugrgvOBfcRY'</span>
<span class="k">class</span> <span class="nc">TestLambdaHandler</span><span class="p">(</span><span class="n">unittest</span><span class="o">.</span><span class="n">TestCase</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">test_valid_sha256_hash_with_matching_password_returns_true</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">event</span> <span class="o">=</span> <span class="n">_build_event</span><span class="p">(</span><span class="s1">'sha256'</span><span class="p">,</span> <span class="n">SAMPLE_SHA256_HASH</span><span class="p">,</span> <span class="n">SAMPLE_PASSWORD</span><span class="p">)</span>
<span class="n">expected</span> <span class="o">=</span> <span class="kc">True</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">lambda_handler</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">expected</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_valid_sha256_hash_with_wrong_password_returns_false</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">event</span> <span class="o">=</span> <span class="n">_build_event</span><span class="p">(</span><span class="s1">'sha256'</span><span class="p">,</span> <span class="n">SAMPLE_SHA256_HASH</span><span class="p">,</span> <span class="s1">'this is not the password'</span><span class="p">)</span>
<span class="n">expected</span> <span class="o">=</span> <span class="kc">False</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">lambda_handler</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">expected</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_valid_sha512_hash_with_matching_password_returns_true</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">event</span> <span class="o">=</span> <span class="n">_build_event</span><span class="p">(</span><span class="s1">'sha512'</span><span class="p">,</span> <span class="n">SAMPLE_SHA512_HASH</span><span class="p">,</span> <span class="n">SAMPLE_PASSWORD</span><span class="p">)</span>
<span class="n">expected</span> <span class="o">=</span> <span class="kc">True</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">lambda_handler</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">expected</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_valid_sha512_hash_with_wrong_password_returns_false</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">event</span> <span class="o">=</span> <span class="n">_build_event</span><span class="p">(</span><span class="s1">'sha512'</span><span class="p">,</span> <span class="n">SAMPLE_SHA512_HASH</span><span class="p">,</span> <span class="s1">'this is not the password'</span><span class="p">)</span>
<span class="n">expected</span> <span class="o">=</span> <span class="kc">False</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">lambda_handler</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">expected</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">_build_event</span><span class="p">(</span><span class="n">digest</span><span class="p">,</span> <span class="n">hash_pass</span><span class="p">,</span> <span class="n">password</span><span class="p">):</span>
<span class="k">return</span> <span class="p">{</span>
<span class="s2">"digest"</span><span class="p">:</span> <span class="n">digest</span><span class="p">,</span>
<span class="s2">"hash_pass"</span><span class="p">:</span> <span class="n">hash_pass</span><span class="p">,</span>
<span class="s2">"password"</span><span class="p">:</span> <span class="n">password</span><span class="p">,</span>
<span class="p">}</span>
</pre></div>
<p>Astute readers will recognize that this is a classic example of tests which lend themselves to
<a href="https://docs.pytest.org/en/latest/example/parametrize.html">py.test's parameterized tests</a>.
I leave the work of converting these to parameterized tests as an exercise for the reader. :)</p>
<p>Continuing along, you reach a point where you start to observe behaviour that's implicit in the code as it exists today,
but which is unclear if it's required or just an accident. For example: currently if you give an arbitrary string as the
digest, then it uses SHA1. Is that required, or just an accident of implementation? Recall though that at this point our
goal is just to backfill tests to capture current behaviour. That is, we're writing <a href="https://en.wikipedia.org/wiki/Characterization_test">characterization tests</a>,
so I chose to add a test to enforce that behaviour:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">test_default_hash_is_sha1</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">event</span> <span class="o">=</span> <span class="n">_build_event</span><span class="p">(</span><span class="kc">None</span><span class="p">,</span> <span class="n">SAMPLE_SHA1_HASH</span><span class="p">,</span> <span class="n">SAMPLE_PASSWORD</span><span class="p">)</span>
<span class="n">expected</span> <span class="o">=</span> <span class="kc">True</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">lambda_handler</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">expected</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
</pre></div>
<p>Ok, so now we have our tests which enforce current behaviour, a nice project structure, and at this point this is all
plain old typical Python development, nothing about Lambda here. At this point you could follow the same steps in the
tutorial and bundle it all up into a zip file, upload to Lambda and you're good.</p>
<p>But I like automating some of the build stuff, so wrote a simple little Bash script to generate the zip file, and called
it <code>build.sh</code>:</p>
<div class="highlight"><pre><span></span><span class="ch">#!/bin/sh</span>
mkdir BUILD
cp -r src/* BUILD/
cp requirements.txt BUILD/
<span class="nb">cd</span> BUILD
../install_deps.sh
rm requirements.txt
zip -r lambda.zip *
mv lambda.zip ..
</pre></div>
<p>Note that this also leaves the tests out of the bundle sent to Lambda, as A) there's no reason for them to live there,
and B) having them in the zip bloats the zip file slightly. <code>install_deps.sh</code> looks like:</p>
<div class="highlight"><pre><span></span><span class="ch">#!/bin/sh</span>
pip install -r requirements.txt -t .
</pre></div>
<p>I could've just put the <code>pip install</code> line into <code>build.sh</code>, but I had a feeling that installing of requirements might
get a bit tricky with bundling something up for Lambda, so broke it out into a separate script.</p>
<p>Now you can just run <code>build.sh</code> from the project directory, and <code>lambda.zip</code> gets created, ready for upload to Lambda.
It'd be nice to enhance the script to upload the file to an S3 bucket & tell Lambda to look at that bucket, but that's
future work, this is good enough for now.</p>
<p>For me this was an interesting exercise, as it was a bit of an epiphany moment to realize that a Lambda handler is just
a plain Python function, so there's no real magic in unit testing it. In my next blog entry, I'll pick up from here and
add <code>bcrypt</code> as a supported digest using TDD and work through the hiccups discovered. All the code I wrote is also in
Github: <a href="https://github.com/pzelnip/lambda-password-service">https://github.com/pzelnip/lambda-password-service</a></p>The 2017 Vancouver Polyglot Unconference2017-05-28T10:20:00-07:002017-05-28T10:20:00-07:00Adam Parkintag:www.codependentcodr.com,2017-05-28:/the-2017-vancouver-polyglot-unconference.html<p>This year as in many years past I was fortunate enough to be able to attend the annual
<a href="https://www.polyglotconf.com">Vancouver Polyglot Unconference</a>. For those unaware, this event (now in it's 6th year!)
is a chance for technicians, programmers, engineers, and others working in the tech industry in Vancouver & surrounding
areas to …</p><p>This year as in many years past I was fortunate enough to be able to attend the annual
<a href="https://www.polyglotconf.com">Vancouver Polyglot Unconference</a>. For those unaware, this event (now in it's 6th year!)
is a chance for technicians, programmers, engineers, and others working in the tech industry in Vancouver & surrounding
areas to get together "for a day of spontaneous sharing, teaching and learning".</p>
<p>This year's event, like others in the past was a great opportunity to learn and share from others struggling with the
challenges of modern software development. I thought I'd write a bit about some of the highlights for me from the event.</p>
<h1>General Themes In Pitches</h1>
<p>Being an unconference, the format of the event is to have attendees "pitch" ideas for discussions at the start of the
day, then people vote on the topics they'd be interested in seeing throughout the day, and then the organizers
facilitate those discussions by scheduling them into particular rooms, etc. I often find it interesting to listen to the
pitches to see what commonalities there are from different folks working in the industry.</p>
<p>This year I was quite surprised to see the "human" side of development come up in many of the suggested topics. Many
pitched talks related to hiring & interviewing, team effectiveness, mentoring & training juniors, progressing to
becoming a senior developer, diversity, and how software development is becoming increasingly complex. From a tech
standpoint, I heard <a href="https://github.com/reactjs">React</a> mentioned a lot, as were microservices and container
orchestration technologies (<a href="https://kubernetes.io/">Kubernetes</a>, <a href="http://mesos.apache.org/">Mesos</a>, etc).</p>
<p>Something I was particularly taken aback by was the sheer variety and breadth of topics suggested. This is generally
true at this event, but it seemed particularly diverse to me this year than in prior years.</p>
<h1>Particular Discussions</h1>
<h2>JS State of the Union</h2>
<p><a href="https://twitter.com/chrismnicola">Chris Nicola</a> kicked off the first session I attended with the JS "State of the
Union" discussion which has happened in prior years at the unconference. Unsurprisingly <a href="https://github.com/reactjs">React</a>
was a technology mentioned a fair bit in this session, as was <a href="https://vuejs.org/">Vue.js</a>.</p>
<p>I'm not a front-end dev, so this was definitely not my forte, but themes I took away from this session was the continued
explosion of the sheer number of JS frameworks out there. I didn't stick around for the entire session, instead following
the law of two feet to switch to....</p>
<h2>SOLID is wrong</h2>
<p>This session (pitched by <a href="https://twitter.com/matresstester">Anthony Tsui</a>) was interesting and rather lively. The
context: earlier this year
<a href="http://twitter.com/tastapod">Dan North</a> (of BDD Fame) did a talk & slide deck on why SOLID principles are wrong. Rather
a controversial stance given how many "classic" well known software developers
(<a href="https://twitter.com/unclebobmartin">Uncle Bob Martin</a> in particular) have long argued how SOLID principles are a key
development design practice. The slide deck from Dan North is at: <a href="https://speakerdeck.com/tastapod/why-every-element-of-solid-is-wrong">https://speakerdeck.com/tastapod/why-every-element-of-solid-is-wrong</a></p>
<p>The session itself featured some lively debate around the arguments made by Dan. A theme I walked away with was the
classic argument of expediency vs resiliency, i.e. do I build for <em>right now</em> or design for an unknown & unpredictable future.</p>
<h2>Training Juniors</h2>
<p>For me, this session was the highlight of the day. <a href="https://twitter.com/saemg">Saem</a> (unfortunately I do not know his
last name) facilitated the session by first doing a presentation (with slides) outlining some of the lessons &
techniques that are grounded in real research that he's adopted with mentoring and managing junior developers. Lots of
fascinating discussion around active recall, modes of thinking, how to optimize learning, the importance of clear,
well-written problem statements, techniques for both helping juniors recognize when they're stuck and how to get
unstuck, and how to set clear expectations & check-ins around those. Really fascinating stuff, and I found myself (as
someone who's had to manage a few co-ops) finding parallels between moments I've experienced and ideas mentioned. I plan
on adopting some of the suggestions made in my work with junior/co-op developers I manage.</p>
<h2>Steps to Be A Senior Developer</h2>
<p>This session was interesting as well. I think unfortunately the original intent of the session ("I'm not a senior but I
want to be one, how do I get there?") got a bit sidetracked. Much of the discussion ended up around how to get hired as
a senior dev, and less around how to progress to be a senior dev. I'm not sure I walked away with many clear ideas that
expanded upon what I already think makes a senior developer a senior (ability to be self-reliant, resiliency, maturity,
ability to mentor, etc).</p>
<h2>Complexity of Modern Software Development</h2>
<p>This session was pitched as as "I'm going to convince you that software development complexity is getting out of
control", and (perhaps unsurprisingly) the discussion ended up rotating around sources of complexity in modern
development. Some of the topics discussed were the "ooh shiny" syndrome vs adopting technology based on need, the
distinction between inherent vs accidental complexity, solving problems at the wrong level of abstraction, and the sheer
explosion of choices we have around competing technologies as being a source of complexity.</p>
<p>An interesting analogy I heard during the session that I hadn't thought of before but seems quite apt was the idea of
open source software as an externality (in the sense of economics) and the implications that makes. Interesting stuff.</p>
<h2>Stupid Questions about diversity</h2>
<p>Last session of the day for me was one facilitated by <a href="https://twitter.com/hmkjburton">Holly Burton</a> who created a space
for people to ask "stupid questions" around diversity. This was truly interesting, lots of discussion around stats &
research around diversity. Much of the discussion ended up focussed on gender diversity (i.e. male/female equality).</p>
<p>Some of the eye-opening moments of the session included discussion around how there's a
"<a href="https://en.wikipedia.org/wiki/Posttraumatic_stress_disorder">PTSD effect</a>" happening around "bro culture" at tech
firms and the implications around how you present your company in things like job postings. For example (and this had
never occurred to me before), but advertising things like "we play ping pong", or "we have nerf battles all the time",
or "beer fridays!" can and often do turn female developers off of applying to a firm.</p>
<p>Another gender difference I hadn't considered: the importance of being clear on what is actually required for a job
posting. Women are far more likely to self-select themselves out of applying for a position when they don't exactly meet
the stated requirements, so having "wish lists" rather than "real requirements" tends to end up with the result being
that many women who are qualified to do a job not applying for it. Really interesting stuff that I'm hoping I'll be able
to apply in postings my current employer produces.</p>
<h1>Meta Thoughts</h1>
<p>All-in-all Polyglot was a great event again, I find it like a 1-day compressed window into the pulse of modern software
development. For me personally, I find it useful to go to the event simply to help keep current.</p>
<p>Some of the themes that stood out to me were around the challenges with hiring & career progression (as someone who is
very interested in the human side of development I really liked seeing this), as well as the increasing complexity of
software development. Things are getting harder, which almost seems unintuitive as we have more, better, tools available
at our disposals as developers.</p>
<p>In any case, I can't wait until next year when I go to the event again. Kudos to the organizers for putting on such a
great event year after year.</p>Resumes, my take2017-04-17T10:20:00-07:002017-04-17T10:20:00-07:00Adam Parkintag:www.codependentcodr.com,2017-04-17:/resumes-my-take.html<p>So a blog I follow had a <a href="http://melreams.com/2017/04/resume-advice/">recent post about resumes for devs</a>, particularly
for junior devs or recent grads. It ended with the open question:</p>
<blockquote>
<p>Readers, do you have any advice for students or anyone who doesn’t have years of dev experience to put on their resumes?</p>
</blockquote>
<p>I …</p><p>So a blog I follow had a <a href="http://melreams.com/2017/04/resume-advice/">recent post about resumes for devs</a>, particularly
for junior devs or recent grads. It ended with the open question:</p>
<blockquote>
<p>Readers, do you have any advice for students or anyone who doesn’t have years of dev experience to put on their resumes?</p>
</blockquote>
<p>I started to write a reply, and then it ballooned into a blog post of my own, so here's my unsolicited advice on the topic.</p>
<p>First, a disclaimer: while I've read a fair number of resumes & interviewed quite a few candidates at places I've
worked, at the end of the day I'm a dev not a hiring manager, so all this should be taken with a grain of salt.</p>
<p>So what advice would I give to those looking to spice up their resumes. Well, first, not so much resume advice, but
general job seeking advice: do your best not to settle. Devs especially right now (even junior devs) can afford to be
a bit selective about where they apply. Read the job post & do a bit of research into the company. Do you know anyone
who works there or has worked there who you could ask about what the place is like? Read some reviews of the place on
sites like <a href="https://www.glassdoor.ca/index.htm">Glassdoor</a>. After doing that, honestly ask yourself: does it sound like
a place you'd be interested in working at? Will it help you achieve your career goals?</p>
<p>Don't just apply to a place because "you never know it might work out", target your applications to places & positions
that align with your values and interests. A former manager once said to me "the problem with applying to a job is that
they might offer it to me, so I better be sure that I'd like to work there before applying", and I think that's sage
advice. Having said that, I recognize this can be challenging when you're desperate for cash, or find yourself in a
situation where you need to be employed ASAP, but even in those cases doing some due diligence can help you prioritize
which places to apply to first, and how much effort to put into each application.</p>
<p>Ok, but what about resumes. Well, to begin with: structure your resume to highlight your strengths first. If your
strengths are your previous job experiences, then put those front and center. If your strengths are your projects
(which might be more the case for undergrads or recent grads), make those the focus so put them at or near the top. Same
for education. Prioritize the stuff that makes you look good, don't get hung up on traditional resume structures (i.e.
"oh there has to be objectives first, then work experience, then education, etc"). Make the "wow" stuff about you come
first as that'll encourage someone to keep reading.</p>
<p>Second, when you write your bullet points for your experiences, read them back to yourself and ask the question "so
what?". This can be a useful exercise in making sure the "why it's important/valuable" is clearly communicated. Here's a
bad example:</p>
<blockquote>
<p>upgraded company to new code review system</p>
</blockquote>
<p>As a hiring manager I'll read that and go "so what?" and likely toss the resume aside. OTOH, if it read:</p>
<blockquote>
<p>upgraded company to new code review system resulting in a $10,000 savings in licensing costs</p>
</blockquote>
<p>Well, that's the kinda thing that will get you in for an interview. That's a rather extreme example, but try to put as
many quantifiable items alongside your experiences (examples might include number of bugs resolved, or how much you
increased test coverage, etc). This advice is particularly true when applying to a place where tech resumes are first
filtered by non-devs. This bit of advice can be challenging for devs, as often the things we know are important/useful
are things that only devs (or technically savvy people) know are important/useful.</p>
<p>Next, the one you've probably heard a million times: take the time to tailor your resume for the job you're applying to.
If the job posting you're applying to makes it clear that the company cares about <a href="https://aws.amazon.com/">AWS</a>
experience, then make sure you highlight your experience with <a href="https://aws.amazon.com/">AWS</a> (as well as you can,
obviously be honest and don't misrepresent your abilities). This is true as well for an objectives section: make sure
your objectives statement aligns well with the job post you're applying for. This more than anything makes the
difference in my experience.</p>
<p>One more thing: get a copy of the book
<a href="https://www.amazon.ca/What-Color-Your-Parachute-2016/dp/160774662X">What Color is Your Parachute?</a> and read the chapter
on resumes. Then when you're done and you've written your resume, read the rest of the book as it's full of <em>really</em>
useful advice on things like interviewing, salary negotiation, etc.</p>
<p>Lastly, remember that looking for work is work. Don't spend days writing a single resume for a single job, but
definitely recognize the fact that it's going to take some effort. It sucks too because job hunting is one of those
things where you can do everything to the best of your ability and still not get in for an interview, and can also write
a crap resume, but luck out & happen to catch a hiring manager on a good day resulting in an interview.</p>
<p>Good luck!</p>Shameful Stealing of Link of the Day2017-04-14T22:04:00-07:002017-04-14T22:04:00-07:00Adam Parkintag:www.codependentcodr.com,2017-04-14:/shameful-stealing-of-link-of-the-day.html<p>Today's link of the day is <a href="https://aboullaite.me/15-must-know-dev-tools-tricks/">some cool Chrome dev tools tricks</a>.</p>
<p>Credit where credit is due, I <a href="http://melreams.com/2017/04/link-of-the-day-11/">saw this on Mel Reams' blog</a> which you
should also check out as it's pretty dang awesome.</p><p>Today's link of the day is <a href="https://aboullaite.me/15-must-know-dev-tools-tricks/">some cool Chrome dev tools tricks</a>.</p>
<p>Credit where credit is due, I <a href="http://melreams.com/2017/04/link-of-the-day-11/">saw this on Mel Reams' blog</a> which you
should also check out as it's pretty dang awesome.</p>The Magic Button2017-04-04T22:06:00-07:002017-04-04T22:06:00-07:00Adam Parkintag:www.codependentcodr.com,2017-04-04:/the-magic-button.html<p>Bob Martin is awesome.</p>
<p><a href="https://www.youtube.com/watch?v=OrsT94FJOQc">https://www.youtube.com/watch?v=OrsT94FJOQc</a></p><p>Bob Martin is awesome.</p>
<p><a href="https://www.youtube.com/watch?v=OrsT94FJOQc">https://www.youtube.com/watch?v=OrsT94FJOQc</a></p>This Changes Everything....2017-03-29T22:12:00-07:002017-03-29T22:12:00-07:00Adam Parkintag:www.codependentcodr.com,2017-03-29:/this-changes-everything.html<p>The other day our staging environment at work ran out of space on the EBS volume holding our MongoDb data. The fun part
about Mongo is that when you get to the point that there's no space left because Mongo's filled it, the Mongo shell will
reject commands with "Can't …</p><p>The other day our staging environment at work ran out of space on the EBS volume holding our MongoDb data. The fun part
about Mongo is that when you get to the point that there's no space left because Mongo's filled it, the Mongo shell will
reject commands with "Can't take a write lock while out of disk space". This includes commands like say,
<code>db.dropDatabase</code> to try and reclaim some of that said precious disk space.</p>
<p>Enter AWS, and how freaking amazing it is: you can now (as of around Feb 2017) resize an EBS volume without even
shutting down the instance it's attached to.</p>
<p>Very handy for moments like this, I was able to double the size of the volume, without even powering down the instance
or even stopping the Mongodb service.</p>
<p>Details of the new EBS resizing: <a href="https://aws.amazon.com/blogs/aws/amazon-ebs-update-new-elastic-volumes-change-everything/">https://aws.amazon.com/blogs/aws/amazon-ebs-update-new-elastic-volumes-change-everything/</a></p>Cool Link of the day2017-03-25T22:14:00-07:002017-03-25T22:14:00-07:00Adam Parkintag:www.codependentcodr.com,2017-03-25:/cool-link-of-the-day.html<p>Ever wanted to coherence check your shell scripts? Check out <a href="https://www.shellcheck.net/">https://www.shellcheck.net/</a></p>
<p>Provides a basic REPL that checks your shell scripts for common issues. Kinda neat, and admittedly I learned a thing or
two while playing with it.</p>
<p>There's also a command line version you can install that …</p><p>Ever wanted to coherence check your shell scripts? Check out <a href="https://www.shellcheck.net/">https://www.shellcheck.net/</a></p>
<p>Provides a basic REPL that checks your shell scripts for common issues. Kinda neat, and admittedly I learned a thing or
two while playing with it.</p>
<p>There's also a command line version you can install that could potentially be used in things like a continuous
integration pipeline.</p>Balance2017-03-15T22:17:00-07:002017-03-15T22:17:00-07:00Adam Parkintag:www.codependentcodr.com,2017-03-15:/balance.html<p>I place a strong emphasis on work/life balance. That sounds funny given that I currently am employed as a software
developer at a startup, and
<a href="http://timberry.bplans.com/is-worklife-balance.html">startups</a>
<a href="https://www.forbes.com/sites/davidshaywitz/2012/10/08/silicon-valley-work-life-balance-is-for-losers-not-closers/#445ffafb4e6b">are</a>
<a href="https://www.entrepreneur.com/article/239553">notorious</a>
<a href="https://techcrunch.com/2014/12/24/rethinking-work-life-balance/">for</a>
<a href="http://paulgraham.com/before.html">being</a>
<a href="http://fortune.com/2016/10/06/entrepreneurship-and-depression/">unfriendly</a>
to work/life balance (I wonder if this is less the nature of startup and more the nature …</p><p>I place a strong emphasis on work/life balance. That sounds funny given that I currently am employed as a software
developer at a startup, and
<a href="http://timberry.bplans.com/is-worklife-balance.html">startups</a>
<a href="https://www.forbes.com/sites/davidshaywitz/2012/10/08/silicon-valley-work-life-balance-is-for-losers-not-closers/#445ffafb4e6b">are</a>
<a href="https://www.entrepreneur.com/article/239553">notorious</a>
<a href="https://techcrunch.com/2014/12/24/rethinking-work-life-balance/">for</a>
<a href="http://paulgraham.com/before.html">being</a>
<a href="http://fortune.com/2016/10/06/entrepreneurship-and-depression/">unfriendly</a>
to work/life balance (I wonder if this is less the nature of startup and more the nature of software development shops,
but I digress).</p>
<p>Related: one of the topics that's often discussed in programming circles is that of passion. If you're passionate you
can overcome any obstacle you face as a developer (and believe me, there will be challenges you face). That is, the key
attribute to success is sheer stubbornness/determination and willingness to put in as many hours as it takes.</p>
<p>Paraphrasing: the difference between a good developer and a great developer is a great developer just puts in the time.
If you don't put in the time, you're not a (great/competent/professional/whatever other positive passive-aggressive
adjective you want) developer and should really give it up and go do something else.</p>
<p>In <a href="https://www.amazon.ca/Clean-Coder-Conduct-Professional-Programmers/dp/0137081073">Clean Coder</a>, <a href="https://www.google.ca/search?num=100&newwindow=1&safe=active&q=uncle+bob+martin&oq=uncle+bob+martin&gs_l=serp.3..0i67k1j0l9.7239.7943.0.8620.6.6.0.0.0.0.115.422.4j1.5.0.ulpe%2Cstargd-cl%3Dweb-for-rerank-with-language%2Ceulp%3D1%2Celpf%3D1%2Ccfro%3D1...0...1.1.64.serp..1.5.420...0i7i30k1j0i13k1j35i39k1.I8uptwKA-us">Bob Martin</a>
articulates this rhetoric quite succinctly:</p>
<blockquote>
<p>I'm talking about 20 extra hours per week. That's roughly three hours per day. .... Do the math. In a week there are
168 hours. Give your employer 40, and your career another 20. That leaves 108. Another 56 for sleep leaves 52 for
everything else. Perhaps you don't want to make that kind of commitment. That's fine, but you should not then think of
yourself as a professional.</p>
</blockquote>
<p>Let's play with this number a bit, because numbers are fun. That's 52 hours for:</p>
<ul>
<li>developing and fostering interests away from development</li>
<li>raising your son/daughter(s)</li>
<li>volunteering at your child's school</li>
<li>getting involved in your community</li>
<li>going to the gym/maintaining fitness</li>
<li>seeing your doctor because you have chronic health problems</li>
<li>spending quality time with your husband/wife/spouse/partner</li>
<li>caring for sick loved ones</li>
<li>living life outside of software development</li>
</ul>
<p><em>combined.</em></p>
<p>I disagree with this mindset. I do agree absolutely that part of being a great developer is pursuing your passion. If
you don't enjoy learning about learning new languages, or frameworks, or tools, or technologies, etc, then you're going
to have a tough time as a developer. But, and this is a very big but, that doesn't mean you have to give up your life.
A friend of mine often cites the following quote that resonates quite strongly with me:</p>
<blockquote>
<p>I work to live, I don't live to work</p>
</blockquote>
<p>Follow your passion, but don't let that dominate your existence.</p>Time to Join This Century2016-02-24T22:24:00-08:002016-02-24T22:24:00-08:00Adam Parkintag:www.codependentcodr.com,2016-02-24:/time-to-join-this-century.html<p>Blogspot is old. Wordpress is the new sexy, time to migrate.</p>
<p>My old blog was at <a href="http://codependentcodr.blogspot.ca/">http://codependentcodr.blogspot.ca/</a> I'm now in the process of migrating stuff here on Wordpress.</p>
<p>(2018 edit: whelp, I never did migrate the old to Wordpress, and now I've migrated from Wordpress to self-hosted …</p><p>Blogspot is old. Wordpress is the new sexy, time to migrate.</p>
<p>My old blog was at <a href="http://codependentcodr.blogspot.ca/">http://codependentcodr.blogspot.ca/</a> I'm now in the process of migrating stuff here on Wordpress.</p>
<p>(2018 edit: whelp, I never did migrate the old to Wordpress, and now I've migrated from Wordpress to self-hosted)</p>Embracing Change2015-03-01T22:04:00-08:002015-03-01T22:04:00-08:00Adam Parkintag:www.codependentcodr.com,2015-03-01:/embracing-change.html<p>I recently listened to a recording of a webinar put on through the ACM titled
<a href="https://event.on24.com/eventRegistration/EventLobbyServlet?target=reg20.jsp&eventid=937091&sessionid=1&key=5B3C11566E06BE6564E638C6DFE0F413&sourcepage=register">"Agile Methods: Agile Methods: The Good, the Hype and the Ugly"</a>
where Bertrand Meyer (the Eiffel/Design by Contract person) gave their interpretation of the agile software movement,
and how we may tweak agile thinking …</p><p>I recently listened to a recording of a webinar put on through the ACM titled
<a href="https://event.on24.com/eventRegistration/EventLobbyServlet?target=reg20.jsp&eventid=937091&sessionid=1&key=5B3C11566E06BE6564E638C6DFE0F413&sourcepage=register">"Agile Methods: Agile Methods: The Good, the Hype and the Ugly"</a>
where Bertrand Meyer (the Eiffel/Design by Contract person) gave their interpretation of the agile software movement,
and how we may tweak agile thinking.</p>
<p>A point in particular caught my attention. They talked about a rephrasing of some of the agile principles as stated in
the manifesto, and in particular they talked about rather than "embracing" change, one should "accept" change. While this
might seem like splitting hairs, I think it an important distinction, and one I completely disagree with. I'd like to
elaborate why I feel the distinction matters.</p>
<p>The rationale behind Meyer's thinking was that nobody is happy when they have to throw away work. Say you built a fancy
front end for a payment system and then after weeks of development the customer says "nope, that's not what I want" and
you have to throw it away. You're obviously not going to be happy about that, but given the customer pays the bills
(and as such your salary), you have to just roll with that punch and start over. As such, Meyer paints this picture of
the developer who begrudgingly throws away his/her pristine work, all the while muttering under his/her breath about how
the customer can't make up their mind and resenting the indecision.</p>
<p>I agree with this in principle (I don't like to waste my time), but it fundamentally misses the point of not only the
agile philosophy, but of modern professional software development. As a developer, I don't want to build things for the
sake of building things, I want to build things that solve problems. In particular, I want to build things that provide
value to a user (ideally a user who will pay for such services, but the monetary unpleasantries I tend to leave to the
business folk).</p>
<p>That is, if what I'm building isn't what the customer wants, I don't want to build it.</p>
<p>This is important. I'm a craftsman, so I very much care about, and put my entire energy into writing the best code I
can, building the best systems I can, but I fully recognize that none of the stuff that goes into that "quality"
equation matters if you're building the wrong thing. As much as I love development, it's value is instrumental, not
intrinsic. If the stuff I create never gets used, then it doesn't matter how good it is.</p>
<p>With this in mind, hell yes, I embrace change. When a customer gives feedback and says "that's not quite what I want"
it's music to my ears because it means I'm now that much closer to building the right thing.</p>
<p>So yeah, don't just accept change, embrace it. Wrap it around you like a warm blanket, secure in the knowledge that
because of that change you're now even closer to building something truly amazing that will change people's lives.</p>
<p>(Note: this post originally appeared on my blogspot blog at: <a href="http://codependentcodr.blogspot.ca/2015/03/embracing-change.html">http://codependentcodr.blogspot.ca/2015/03/embracing-change.html</a>)</p>Book Review: The Software Craftsman2015-02-03T20:58:00-08:002015-02-03T20:58:00-08:00Adam Parkintag:www.codependentcodr.com,2015-02-03:/book-review-the-software-craftsman.html<h1>Book: The Software Craftsman</h1>
<p><img alt="Book Cover - The Software Craftsman" src="http://ecx.images-amazon.com/images/I/51gqht7qN8L._AA324_PIkin4,BottomRight,-38,22_AA346_SH20_OU15_.jpg"></p>
<p><em>Author(s):</em> Sandro Mancuso</p>
<p><em>Publisher:</em> Prentice Hall; 1 edition (Dec 14 2014)</p>
<p><em>Pages/Sections Read:</em> All, cover to cover</p>
<p><em>Thumbs up/Thumbs down:</em> Thumbs Down</p>
<p><em>Link(s):</em> <a href="http://www.amazon.ca/The-Software-Craftsman-Professionalism-Pragmatism/dp/0134052501">Amazon</a>, <a href="https://twitter.com/sandromancuso">Author's Twitter</a></p>
<h2>Summary Of Content Read</h2>
<p>This book frustrated me. I once had the fortune of seeing Sandro …</p><h1>Book: The Software Craftsman</h1>
<p><img alt="Book Cover - The Software Craftsman" src="http://ecx.images-amazon.com/images/I/51gqht7qN8L._AA324_PIkin4,BottomRight,-38,22_AA346_SH20_OU15_.jpg"></p>
<p><em>Author(s):</em> Sandro Mancuso</p>
<p><em>Publisher:</em> Prentice Hall; 1 edition (Dec 14 2014)</p>
<p><em>Pages/Sections Read:</em> All, cover to cover</p>
<p><em>Thumbs up/Thumbs down:</em> Thumbs Down</p>
<p><em>Link(s):</em> <a href="http://www.amazon.ca/The-Software-Craftsman-Professionalism-Pragmatism/dp/0134052501">Amazon</a>, <a href="https://twitter.com/sandromancuso">Author's Twitter</a></p>
<h2>Summary Of Content Read</h2>
<p>This book frustrated me. I once had the fortune of seeing Sandro give a talk at the Software Craftsmanship North America
(SCNA) conference in 2013, and found that talk uplifting, and inspirational. As a result of that, when I saw this book
had been released it was an "instant buy" for me.</p>
<p>Ultimately though I was incredibly disappointed by this book.</p>
<p>I wanted to like this book. Rather I wanted to love this book. And honestly, much of what Sandro espouses in this book
I agree with and believe. But, this book is poorly written and filled with anecdotal "evidence" to support their claims.
This is a shame, as there is much well documented, well-researched evidence to support much of what is argued for in this
book. See,
the thing is when you make empirical claims (ie - if you do TDD you will reduce bugs and therefore reduce costs, or if
you pair with other developers you will create a culture of learning which will improve productivity, or if you hire
craftsmen your company will be better off), you need to back that up with empirical evidence, not just "I had this job
once where we did this & it worked...".</p>
<p>By in large if you've ever followed the software craftsmanship community, you'll have heard everything that you'll read
in this book. TDD is great so it's an encouraged practice, but we don't hold practices in a dogmatic way. Pragmatism is
key. You can't be a great developer without being passionate. Commit yourself to lifelong learning. The craftsmanship
movement is about raising the bar. On and on and on, it's all the standard tropes you hear in conversations about
software craftsmanship. I went into this book expecting to see something new, or some deep insights, instead I got a
series of blog posts that felt very much like preaching to the choir.</p>
<p>There's also lots of heavy-handed "preachyness" in this book. Lots of defamatory comments towards managers, agile
coaches, and architects (though back-pedalled in the appendix), and lots of "if you don't do this, then you're doing it
wrong" type rhetoric, which I found surprising. The craftsmanship community is supposed to be about celebrating diversity
and being welcoming of anyone, of any skill level so long as they're willing to better themselves and learn more.</p>
<p>There's also lots of inflammatory/adversarial commentary (ex: "QA teams are an anti-pattern", "are you good enough to
work on legacy code?", "tech debt items are an excuse to justify bad code", "software craftsmen are never scared to
lose their jobs", "only incompetent people fear losing their jobs", "university degrees don't mean anything", etc) that
feels very elitist & arrogant. Lots of straw man commentary, painting conversations with Dilbert-esque pointy-haired
bosses in a very biased light.</p>
<p>Lots of sweeping generalizations, and little in the way of new insights. There's a lack of focus or coherent theme to
the book. Who is this for? Is it for "apprentice" craftsmen? For people who've heard about this software craftsmanship
thing and want to know more? For the Bob Martin's of the world? It's so inconsistent, some of it feels written for an
audience who's only vaguely familiar with the craftsmanship movement, and other parts feel like unless you've been
writing code for decades you'll have trouble relating.</p>
<p>I'm being overly harsh, there are nuggets of really good insights in this book and Sandro certainly knows the craftsmanship
movement. The thing is though there's nothing you won't get from simply reading the blogs or books of some of the people
in the craftsmanship community. If you've read Clean Coder by Bob Martin, there's no reason to read this book.</p>
<p>(Note: this post originally appeared on my blogspot blog at: <a href="http://codependentcodr.blogspot.ca/2015/02/book-review-software-craftsman.html">http://codependentcodr.blogspot.ca/2015/02/book-review-software-craftsman.html</a>)</p>Book Review: Java Puzzlers2014-01-20T09:07:00-08:002014-01-20T09:07:00-08:00Adam Parkintag:www.codependentcodr.com,2014-01-20:/book-review-java-puzzlers.html<h1>Book: Java Puzzlers</h1>
<p><em>Authors:</em> Joshua Bloch and Neal Gafter</p>
<p><em>Publisher:</em> Addison Wesley</p>
<p><em>Pages Read:</em> all</p>
<p><em>Sections:</em> all</p>
<p><em>Thumbs up/Thumbs Down?</em> Up, slightly sideways</p>
<p><em>Link:</em> <a href="http://www.amazon.ca/JavaTM-Puzzlers-Traps-Pitfalls-Corner-ebook/dp/B001U5VJVS/ref=sr_1_1?ie=UTF8&qid=1390237560&sr=8-1&keywords=java+puzzlers">Amazon</a></p>
<h2>Summary of Content Read</h2>
<p>Java Puzzlers is not so much a book, but a collection of obscure corner cases in the Java programming language …</p><h1>Book: Java Puzzlers</h1>
<p><em>Authors:</em> Joshua Bloch and Neal Gafter</p>
<p><em>Publisher:</em> Addison Wesley</p>
<p><em>Pages Read:</em> all</p>
<p><em>Sections:</em> all</p>
<p><em>Thumbs up/Thumbs Down?</em> Up, slightly sideways</p>
<p><em>Link:</em> <a href="http://www.amazon.ca/JavaTM-Puzzlers-Traps-Pitfalls-Corner-ebook/dp/B001U5VJVS/ref=sr_1_1?ie=UTF8&qid=1390237560&sr=8-1&keywords=java+puzzlers">Amazon</a></p>
<h2>Summary of Content Read</h2>
<p>Java Puzzlers is not so much a book, but a collection of obscure corner cases in the Java programming language. The
author (Joshua Bloch) is well known as the author of <a href="http://www.amazon.ca/Effective-Java-2nd-Edition-Programming-ebook/dp/B000WJOUPA/ref=pd_sim_kinc_2">Effective Java</a>
which is widely regarded as the premier text for the language, and furthermore they are one the designers and authors of
the Java Collections Framework. So to say the least, they know their stuff.</p>
<p>Each chapter of the book features a collection of "puzzlers" centered around a particular section of the language
(examples include loops, strings, exceptions, classes, etc). Each "puzzler" is formulated where a puzzle (typically in
the form of a code snippet) is given, and the reader is encouraged to try and predict what the output will be, or why
the code is incorrect. Then an answer/explanation of the puzzler is given. All-in-all there are 95 different puzzlers
across the book, and they range from the fairly common "if you thought about it a bit you'd figure it out" to the
extremely obscure "unless you were a Java language designer you'd never have any hope of figuring this out". The
explanations also often include commentary to language designers (ex: "the lesson for language designers here is...").</p>
<p>From an academic "curiosity" point of view the book is quite intriguing. As a fairly experienced Java developer I found
myself surprised with the vast majority of the puzzlers. The programming languages guy in me found this fascinating
(ex: wait, so you can have Unicode literals in comments, and those literals are interpreted by the compiler?).</p>
<p>Having said that, the book does reach a point where the puzzles and concepts hit upon by the puzzles are extremely
obscure. For a typical Java developer you'll almost never run into most of the tidbits in this book. That's not to say
that reading it isn't useful, you'll definitely learn a bit about the book, but if you're looking to learn "how to write
good Java code" this is not the book for you (again, see Bloch's other book for that).</p>
<p>(Note: this post originally appeared on my blogspot blog at: <a href="http://codependentcodr.blogspot.ca/2014/01/book-java-puzzlers-authors-joshua-bloch.html">http://codependentcodr.blogspot.ca/2014/01/book-java-puzzlers-authors-joshua-bloch.html</a>)</p>Book Review - The Clean Coder2014-01-20T09:07:00-08:002014-01-20T09:07:00-08:00Adam Parkintag:www.codependentcodr.com,2014-01-20:/book-review-the-clean-coder.html<h1>Book: The Clean Coder - A Code of Conduct For Professional Programmers</h1>
<p><em>Authors:</em> "Uncle" Bob Martin</p>
<p><em>Publisher:</em> Prentice Hall</p>
<p><em>Pages Read:</em> all</p>
<p><em>Sections:</em> all</p>
<p><em>Thumbs up/Thumbs Down?</em> Up</p>
<p><em>Link:</em> <a href="http://www.amazon.ca/Clean-Coder-Conduct-Professional-Programmers/dp/0137081073/ref=sr_1_1?ie=UTF8&qid=1390237049&sr=8-1&keywords=the+clean+coder">Amazon</a></p>
<h2>Summary of Content Read</h2>
<p>This book is largely a follow-up to Martin's other very well known book
<a href="https://www.amazon.ca/Clean-Code-Handbook-Software-Craftsmanship/dp/0132350882/ref=pd_bxgy_14_img_2?_encoding=UTF8&psc=1&refRID=3SXJ498BQX424TCE3F4M">Clean Code</a>
. Whereas …</p><h1>Book: The Clean Coder - A Code of Conduct For Professional Programmers</h1>
<p><em>Authors:</em> "Uncle" Bob Martin</p>
<p><em>Publisher:</em> Prentice Hall</p>
<p><em>Pages Read:</em> all</p>
<p><em>Sections:</em> all</p>
<p><em>Thumbs up/Thumbs Down?</em> Up</p>
<p><em>Link:</em> <a href="http://www.amazon.ca/Clean-Coder-Conduct-Professional-Programmers/dp/0137081073/ref=sr_1_1?ie=UTF8&qid=1390237049&sr=8-1&keywords=the+clean+coder">Amazon</a></p>
<h2>Summary of Content Read</h2>
<p>This book is largely a follow-up to Martin's other very well known book
<a href="https://www.amazon.ca/Clean-Code-Handbook-Software-Craftsmanship/dp/0132350882/ref=pd_bxgy_14_img_2?_encoding=UTF8&psc=1&refRID=3SXJ498BQX424TCE3F4M">Clean Code</a>
. Whereas that book focuses on the artifacts (code) we developers produce this book focuses on the developer
his/herself. How should we as professional developers act? What is the difference between a commitment and estimate?
What are our responsibilities? When can we say no & how do we do it? When are we obligated to say yes? How do we get
better at what we do?</p>
<p>Martin tries to distill their nearly 40 years of experience into some hard fought lessons. While it is very much
appreciated to hear "tales from the trenches", the book does have a fairly heavy-handed "do as I say" tone. Don't do
TDD? Well then you're not a professional. Do you create ambitious estimates? Well then, you're not a professional.
From a rhetorical point of view, the book does rely on this "proof by appeal to professionalism" approach, rather than
give solid evidence and data to back up many of the arguments made. For example, the TDD chapter has the passage:</p>
<blockquote>
<p>Yes there have been lots of controversial blogs and articles written about TDD over the years and there still are.
In the early days they were serious attempts at critique and understanding. Nowadays, however, they are just rants.
The bottom line is that TDD works, and everybody needs to get over it.</p>
</blockquote>
<p>I feel like the paragraph should have ended with
"QED". Hardly a conclusive argument in favour of TDD, and the off-hand dismissal of any critiques of the practice
really does hurt the point he's making.</p>
<p>Having said all this, it is certainly clear that much of what is offered is good advice, and represents an open challenge
to developers to be better. If you put aside the "if you don't do this you're not professional" rhetoric, at its core
this book is a call for developers to live up to the responsibility of the job they have been hired to do. Oftentimes
we as developers like to silo ourselves off, focus on our narrowly defined technical tasks, and that is simply
unrealistic. Part of the responsibility of being a developer is to understand the context of the work you do, why it's
important and why it adds value to the customer/client/business/etc. And if that value isn't there, it's up to you to
find it.</p>
<p>As such I found this book both refreshing and terrifying. Refreshing to hear a voice from the agile community who
doesn't seem to feel that the PO is the only entity responsible for identifying value. Terrifying to think that I, as
an introverted software developer, has a duty to do more than just simply write good, clean code.</p>
<p>In terms of structure, the book is divided into 14 different chapters each covering a topic of interest to professional
developers. While there is some technical discussion, it is relatively rare, by in large the chapter topics focus on
"soft" skills rather than technical ones.</p>
<p>All-in-all, while heavy-handed and at times "preachy", it is very much a worthwhile read for anyone considering or
living a career in software development.</p>
<p>(Note: this post originally appeared on my blogspot blog at: <a href="http://codependentcodr.blogspot.ca/2014/01/book-review-clean-coder.html">http://codependentcodr.blogspot.ca/2014/01/book-review-clean-coder.html</a>)</p>Java Tip of the Day - EqualsVerifier2013-11-01T11:11:00-07:002013-11-01T11:11:00-07:00Adam Parkintag:www.codependentcodr.com,2013-11-01:/java-tip-of-the-day-equalsverifier.html<p>This looks more than a little cool for those of us (like me) who are pedantic about testing out
<code>equals</code>/<code>hashcode</code>/<code>compareTo</code> methods:</p>
<p><a href="http://www.jqno.nl/equalsverifier/">http://www.jqno.nl/equalsverifier/</a></p>
<p>(Note: this post originally appeared on my blogspot blog at: <a href="http://codependentcodr.blogspot.ca/2013/11/equalsverifier.html">http://codependentcodr.blogspot.ca/2013/11/equalsverifier.html</a>)</p><p>This looks more than a little cool for those of us (like me) who are pedantic about testing out
<code>equals</code>/<code>hashcode</code>/<code>compareTo</code> methods:</p>
<p><a href="http://www.jqno.nl/equalsverifier/">http://www.jqno.nl/equalsverifier/</a></p>
<p>(Note: this post originally appeared on my blogspot blog at: <a href="http://codependentcodr.blogspot.ca/2013/11/equalsverifier.html">http://codependentcodr.blogspot.ca/2013/11/equalsverifier.html</a>)</p>Git bisect And Nose -- Or how to find out who to blame for breaking the build.2012-08-03T09:19:00-07:002012-08-03T09:19:00-07:00Adam Parkintag:www.codependentcodr.com,2012-08-03:/git-bisect-and-nose-or-how-to-find-out-who-to-blame-for-breaking-the-build.html<p>How did I not ever discover <code>git bisect</code> before today? Git bisect allows you to identify a particular commit which
breaks a build, even after development has continued past that commit. So for example, say you:</p>
<!-- markdownlint-disable MD013 -->
<ul>
<li>Commit some code which (unknowing to you) happens to break the build</li>
<li>You then …</li></ul><p>How did I not ever discover <code>git bisect</code> before today? Git bisect allows you to identify a particular commit which
breaks a build, even after development has continued past that commit. So for example, say you:</p>
<!-- markdownlint-disable MD013 -->
<ul>
<li>Commit some code which (unknowing to you) happens to break the build</li>
<li>You then (not realizing things have gone sideways) continue on doing commits on stuff you're working on</li>
<li>You then are about to push your code up to a remote mainline, so you finally run all those unit tests and realize you broke the build somewhere, but you don't know which commit introduced the problem</li>
</ul>
<!-- markdownlint-enable MD013 -->
<p>In a typical environment you'd now have a fun period of checking out a previous revision, running the tests, seeing if
that was the commit that broke the build, and continue doing so until you identified the commit that introduced the
failure. I have experienced this many many times and it is the complete opposite of fun.</p>
<p>If you were smart you might recognize that a binary search would be effective here. That is, if you know commit (A) is
bad, and commit (B) is good, and there's 10 commits in-between (A) and (B) then you'd checkout the one halfway between
the two, check for the failure, and in doing so eliminate half the possibilities (rather than trying all 10 in succession).</p>
<p>And if you were really smart you'd know that this is exactly what <code>git bisect</code> does. You tell git bisect which commit
you know is good, and which commit you know is bad, then it steps you through the process of stepping through the
commits in-between to identify which commit introduced the failure.</p>
<p>But wait, there's more! There's also a lesser-known option to <code>git bisect</code>. If you do a
"<code>git bisect run <somecommand></code>" then the process becomes completely automated. What happens is git runs
<code><somecommand></code> at each iteration of the bisection, and if the command returns error code 0 it marks that commit as
"good", and if it returns non-zero it marks it as "bad", and then continues the search with no human interaction whatsoever.</p>
<p>How cool is that?</p>
<p>So then the trick becomes "what's the command to use for <code><somecommand></code>?" Obviously this is project dependent
(probably whatever command you use to run your unit tests), but for those of us who are sane Python devs we probably use
<a href="https://github.com/nose-devs/nose">Nose</a> to run our tests. As an example, I often organize my code as follows:</p>
<div class="highlight"><pre><span></span>project/
+--- src/
+--- module1/
+--- module2/
+--- test/
</pre></div>
<p>Where "module1" contains code for a module, "module2" contains code for another module, and "test" contains my unit
tests. Nose is smart enough that if you tell it to start at "src" it will search all subdirectories for tests and then
run them. So lets say we know that commit <code>022ca08</code> was "bad" (ie the first commit we noticed the problem in) and
commit <code>"</code>0b52f0c` was good (it doesn't contain the problem). We could then do:</p>
<div class="highlight"><pre><span></span>git bisect start 022ca08 0b52f0c --
git bisect run nosetests -w src
</pre></div>
<p>Then go grab a coffee, come back in a few minutes (assuming your tests don't take forever to run), and git will have
identified the commit between <code>0b52f0c</code> and <code>022ca08</code> that introduced the failure. Note that we have to run <code>git bisect</code>
from the top of the source tree (in my example the "project" directory) hence we need to tell <code>nosetests</code> to look in
src via the <code>-w</code> parameter.</p>Handy Python tip #12012-06-07T20:16:00-07:002012-06-07T20:16:00-07:00Adam Parkintag:www.codependentcodr.com,2012-06-07:/handy-python-tip-1.html<p>The other day I was adding the rich comparison methods (the ones for operator overloading) to a class I had defined.
Like many Python programmers before me I wondered "why is it that if I define a method for equality, I still have to
define a not-equal method?" and "if …</p><p>The other day I was adding the rich comparison methods (the ones for operator overloading) to a class I had defined.
Like many Python programmers before me I wondered "why is it that if I define a method for equality, I still have to
define a not-equal method?" and "if I define a comparison method, why do I have to define the other comparison methods?"</p>
<p>And then low and behold, while looking for something completely different, I stumbled across the
<a href="http://docs.python.org/release/2.7/library/functools.html#functools.total_ordering"><code>functools.total_ordering</code></a> class
decorator. With it, you can define just the <code>__eq__</code> method, and any rich comparison method (<code>__le__</code>, <code>__lt__</code>,
<code>__gt__</code>, etc), and it provides default implementations for all the others.</p>
<p>Very handy stuff.</p>
<p>An example can be found in my <a href="https://raw.github.com/pzelnip/MiscPython/e2a37ce2f2a51d5f69df82091d94dd239152ca63/operator_overloading/total_ordering.py">MiscPython examples collection on Github</a>.</p>
<p>(Note: this post originally appeared on my blogspot blog at: <a href="http://codependentcodr.blogspot.ca/2012/06/handy-python-tip-1.html">http://codependentcodr.blogspot.ca/2012/06/handy-python-tip-1.html</a>)</p>The Polyglot {UN}Conference 20122012-06-02T05:04:00-07:002012-06-02T05:04:00-07:00Adam Parkintag:www.codependentcodr.com,2012-06-02:/the-polyglot-unconference-2012.html<p>This year I was fortunate to be allowed to attend the inaugural <a href="http://www.polyglotconf.com/">Polyglot UN-Conference</a>.
An UN-conference is a unique format that is rather well suited to coding topics whereby attendees suggest and facilitate
fairly open forums on whatever they want to talk or hear about. It's a very cool idea …</p><p>This year I was fortunate to be allowed to attend the inaugural <a href="http://www.polyglotconf.com/">Polyglot UN-Conference</a>.
An UN-conference is a unique format that is rather well suited to coding topics whereby attendees suggest and facilitate
fairly open forums on whatever they want to talk or hear about. It's a very cool idea that has the potential to be
completely awful or absolutely amazing.</p>
<p>I can say with full confidence that Polyglot was very much the latter. Simply a great event all around.</p>
<p>I managed to get in on five of the talks at the show this year. I'll do a quit recap of each in turn:</p>
<h1>Go</h1>
<p>First up was the <a href="http://www.golang.com/">Go</a> programming language. The session started with the facilitator giving a
quick bird's eye view of the language and some of the interesting features that make it unique, and then led into a
group discussion of various thoughts & experiences others had with it. Honestly before I had showed up to Polyglot I
had kinda dismissed Go as a toy language from Google, but while I never had any "aha!" moments during the session, I
definitely had my curiosity piqued. Some things I never knew:</p>
<!-- markdownlint-disable MD013 -->
<ul>
<li>it's a compiled (statically compiled) language, not interpreted</li>
<li>syntactically it's a blend of a C/Java style language with what looks an awful lot like Python</li>
<li>Ken Thompson (who was one of the co-inventors of a little language called C) was one of the initial visionaries for the project. Interesting stuff.</li>
<li>Its statically typed, though type declarations are optional (it seems to do some sort of type inference)</li>
<li>There's no classes and inheritance, instead uses interfaces and composition</li>
<li>There's a rather substantial standard library. It's not <a href="https://pypi.python.org/pypi">PyPI</a>, but there's a definite sense of "batteries included".</li>
</ul>
<!-- markdownlint-enable MD013 -->
<p>I'll definitely be playing around with it a bit, as I want to know more.</p>
<h1>Attracting Developers to Your Platform</h1>
<p>A common problem many devs who have open source projects face is how to "inspire" other devs to:
get excited about their project
get others to contribute/spread the word.
Much of the focus of this open session was things we (as owners of projects) should and should not do to facilitate
these goals. Some topics touched on was how to manage poisonous people, zealots, etc, how to promote your project via
things like talks at conferences, the importance of online presence, creating a sense for developers that support is
visible, responsive, and accessible, and a variety of others. Unfortunately I don't have any notes from the talk, as
my computer's battery was dead during it. :(</p>
<p>While much of the conversation was interesting from an academic standpoint, as someone who doesn't have any FOSS
projects to get people jazzed about, there wasn't really a lot of takeaway for me here. This was I think the problem
with it -- it felt too focused on open source.</p>
<p>After an extended lunch (thanks to the extremely slow service at the <a href="http://www.osf.com/">Old Spaghetti Factory</a>), we
got back to the conference about halfway through the 1PM talks, so I never really got to anything here, instead taking
the time to charge the battery on my netbook & decompress a bit. At 2PM though I got to:</p>
<h1>Effective Testing Practices</h1>
<p>This one was the highlight of the day for me. The fishbowl session started with an open discussion on acceptance
testing vs user testing, and went from there. One of the big takeaways for me was <a href="http://cukes.info/">Cucumber</a> which
I had never seen before but seemed worth exploring. There was much debate on the use of systems like this that try to
capture business requirements in a semi-structured format. Some feel that this had value, others not so much. Much
insightful spirited debate ensued -- until the fire alarm went off and we all had to leave for a bit. Flame war indeed.</p>
<p>When we got back, an insightful discussion largely surrounding the notion of test coverage ensued. Some feel that the
artificial number that per-line test coverage gives has the potential for misleading one into a false sense of security.
Others (and I'd say I'm sympathetic to this view) feel that while sure the number is completely meaningless, it provides
a quick and dirty metric for identifying gross shortcomings in your testing.</p>
<p>There were also some rather humourous "horror stories" about testing (or a lack thereof) in industry, and a few comments
that started to really touch on the deep issue of why we test, and what the point of it all is. It's too bad this
session lost 10-15 minutes due to the fire alarm, as this one was the highlight of the conference for me.</p>
<h1>Big Data</h1>
<p>I was lukewarm on this one going in, but none of the other topics at the time really caught my eye. The open discussion
started with the facilitator soliciting people in the audience to share their experiences with big data. Most of these
were actually fairly small, anecdotal discussions about the difficulties of working with larger amounts of data with
traditional RDBMS systems. Partway through an attendee (who is an employee of Amazon) chimed in and gave an intro on
some of the concepts behind true big data (ie Amazon S3) systems. This was good and bad, while it was great to see
someone with expert knowledge step in and share their insights, it did feel as though the talk moved from "how can we do
big data, what are the challenges associated with it" to "if you need to do big data, you can use Amazon S3 for the backend".</p>
<h1>R and Python</h1>
<p>I'm not sure if it was the "end of day and I'm exhausted" factor, or just my lack of interest in scientific computing,
but I pretty much tuned out during this one. It started off with a demonstration of using
<a href="http://ipython.org/">iPython Notebook</a> to explore some data set correlating weather with bicycle ridership. On one
hand, the technology seemed useful, particularly for those who have Matlab/Mathmatica backgrounds, but for me, I lost
interest early. Two of my coworkers however found it quite interesting.</p>
<p>Last were the closing ceremonies, with a fun and entertaining demonstration of coding by voice in 5 different languages
in ~5 minutes. This was priceless. :)</p>
<p>On the whole, for being the first one, the conference was quite well run. Some things I'd have liked to see would've
been to have the online schedule be a bit more accessible. It was a bit of a hassle to go to Lanyrd, track down the
conference, and hit schedule. And related to this: the online schedule was out of sync with the printed board, while
we were at lunch we couldn't find out what the talks happening at 1PM were as a result. Having the online board kept
in sync with the printed board would've been very useful.</p>
<p>Minor hiccups aside, the conference was amazing. It was incredible value too -- $35 for a days worth of tech talks
with people who know, and love technology and use it to solve problems on a daily basis. Schedule permitting I have no
doubt I'd attend again in the future.</p>
<p>An interesting idea that was mentioned at the closing ceremonies was to do Vancouver Polyglot meetups every so often.
While I likely won't be able to attend these as I live in Victoria, I really hope this takes hold as it'd be awesome to
see the strong tech community in greater Vancouver grow.</p>
<p>(Note: this post originally appeared on my blogspot blog at: <a href="http://codependentcodr.blogspot.ca/2012/06/polyglot-unconference-2012.html">http://codependentcodr.blogspot.ca/2012/06/polyglot-unconference-2012.html</a>)</p>Useful Python Tools2012-05-18T16:23:00-07:002012-05-18T16:23:00-07:00Adam Parkintag:www.codependentcodr.com,2012-05-18:/useful-python-tools.html<p>I often stumble across and use a number of useful tools for creating Python code. Thought I'd barf out a blog post
documenting a few of them so that my future self will be able to find this info again if need be. :)</p>
<h1>coverage.py</h1>
<p>(<a href="http://nedbatchelder.com/code/coverage/">http://nedbatchelder.com/code/coverage …</a></p><p>I often stumble across and use a number of useful tools for creating Python code. Thought I'd barf out a blog post
documenting a few of them so that my future self will be able to find this info again if need be. :)</p>
<h1>coverage.py</h1>
<p>(<a href="http://nedbatchelder.com/code/coverage/">http://nedbatchelder.com/code/coverage/</a>)</p>
<p>Coverage.py is a Python code coverage tool and is useful for finding out how well your unit tests cover your code.
I've often had it find big deficiencies in my unit test coverage. Common usage:</p>
<div class="highlight"><pre><span></span>coverage run somemodule_test.py
coverage report -m
</pre></div>
<p>Will spit out a coverage report for the tests in <code>somemodule_test.py</code>. Used in this way, <code>coverage.py</code> isn't particularly
handy, but combined with a good unit test runner (see below) it becomes very handy.</p>
<h1>Nose</h1>
<p>(<a href="http://readthedocs.org/docs/nose/en/latest/">http://readthedocs.org/docs/nose/en/latest/</a>)</p>
<p>Is nicer testing for Python. Nose is an extremely handy unittest runner that has some perks over the standard Python
<code>unittest</code> module. Continuing from the last tool, nose also integrates very nicely with <code>coverage.py</code>. I commonly use
it to produce some nice HTML pages summarzing test coverage for my project:</p>
<div class="highlight"><pre><span></span>nosetests --with-coverage --cover-inclusive --cover-html --cover-erase
</pre></div>
<p>produces a "cover" directory containing an index.html with some nice pretty HTML reports telling me how well my unit
tests cover my codebase.</p>
<h1>pymetrics</h1>
<p>(<a href="http://sourceforge.net/projects/pymetrics/">http://sourceforge.net/projects/pymetrics/</a>)</p>
<p><code>pymetrics</code> is a handy tool for spitting out some well, metrics, about your code. Ex:</p>
<div class="highlight"><pre><span></span>pymetrics somemodule.py
</pre></div>
<p>Spits out a bunch of numbers about <code>somemodule.py</code> including trivial things like how many methods have docstrings, to
more interesting things like the McCabe <a href="http://en.wikipedia.org/wiki/Cyclomatic_complexity">cyclomatic complexity</a> of
each method/function within the module. Handy.</p>
<h1>cloc</h1>
<p>(<a href="http://cloc.sourceforge.net/">http://cloc.sourceforge.net/</a>)</p>
<p>Is a simple "lines of code" counter that happens to support Python. In the top directory of a project a:</p>
<div class="highlight"><pre><span></span>cloc .
</pre></div>
<p>will give you summary output for your project like:</p>
<div class="highlight"><pre><span></span><span class="nb">-------------------------------------------------------------</span><span class="c"></span>
<span class="c">Language files blank comment code</span>
<span class="nb">-------------------------------------------------------------</span><span class="c"></span>
<span class="c">Python 31 3454 9215 14775</span>
<span class="nb">-------------------------------------------------------------</span><span class="c"></span>
<span class="c">SUM: 31 3454 9215 14775</span>
<span class="nb">-------------------------------------------------------------</span><span class="c"></span>
</pre></div>
<p>While LOC is generally a meaningless statistic, it can be handy for getting a "ballpark" idea of how big a project is.</p>
<p>(Note: this post originally appeared on my blogspot blog at: <a href="http://codependentcodr.blogspot.ca/2012/05/useful-python-tools.html">http://codependentcodr.blogspot.ca/2012/05/useful-python-tools.html</a>)</p>The hierarchy of Mystical Arts in Programming2012-02-29T14:12:00-08:002012-02-29T14:12:00-08:00Adam Parkintag:www.codependentcodr.com,2012-02-29:/the-hierarchy-of-mystical-arts-in-programming.html<p>I was responding
<a href="http://stackoverflow.com/questions/100003/what-is-a-metaclass-in-python">to an incredibly detailed answer on StackOverflow on Python metaclasses</a>
, when I wrote the following on my whiteboard:</p>
<p><img alt="original whiteboard contents" src="http://desmond.imageshack.us/Himg809/scaled.php?server=809&filename=imag0638j.jpg&res=crop"></p>
<p>I thought this was clever, and gave the (I thought clever) response on StackOverflow which read:</p>
<blockquote>
<p>I read this and think of the famous "There are lies, damned lies …</p></blockquote><p>I was responding
<a href="http://stackoverflow.com/questions/100003/what-is-a-metaclass-in-python">to an incredibly detailed answer on StackOverflow on Python metaclasses</a>
, when I wrote the following on my whiteboard:</p>
<p><img alt="original whiteboard contents" src="http://desmond.imageshack.us/Himg809/scaled.php?server=809&filename=imag0638j.jpg&res=crop"></p>
<p>I thought this was clever, and gave the (I thought clever) response on StackOverflow which read:</p>
<blockquote>
<p>I read this and think of the famous "There are lies, damned lies, and then there is statistics", but instead think of
it as "there are hacks, tricks, voodoo magic, dark arts, and then there are Python metaclasses".</p>
</blockquote>
<p>This was good, I then left to go for lunch and when I came back a co-worker had modified/added to my list:</p>
<p><img alt="updated whiteboard contents" src="http://desmond.imageshack.us/Himg196/scaled.php?server=196&filename=imag0637x.jpg&res=crop"></p>
<p>(Note: this post originally appeared on my blogspot blog at: <a href="http://codependentcodr.blogspot.ca/2012/02/heirarchy-of-mystical-arts-in.html">http://codependentcodr.blogspot.ca/2012/02/heirarchy-of-mystical-arts-in.html</a>)</p>Python HTMLParser and super()2012-02-16T15:48:00-08:002012-02-16T15:48:00-08:00Adam Parkintag:www.codependentcodr.com,2012-02-16:/python-htmlparser-and-super.html<p>So I have a class that inherits from <code>HTMLParser</code>, and I want to call the super class init (the <code>__init__</code> of
<code>HTMLParser</code>), I would think I should do:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">MyParser</span><span class="p">(</span><span class="n">HTMLParser</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="nb">super</span><span class="p">(</span><span class="n">MyParser</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">()</span>
</pre></div>
<p>But this causes a problem:</p>
<div class="highlight"><pre><span></span><span class="n">myparser</span> <span class="o">=</span> <span class="n">MyParser</span><span class="p">()</span>
<span class="n">Traceback</span> <span class="p">(</span><span class="n">most</span> <span class="n">recent</span> <span class="n">call</span> <span class="n">last …</span></pre></div><p>So I have a class that inherits from <code>HTMLParser</code>, and I want to call the super class init (the <code>__init__</code> of
<code>HTMLParser</code>), I would think I should do:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">MyParser</span><span class="p">(</span><span class="n">HTMLParser</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="nb">super</span><span class="p">(</span><span class="n">MyParser</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">()</span>
</pre></div>
<p>But this causes a problem:</p>
<div class="highlight"><pre><span></span><span class="n">myparser</span> <span class="o">=</span> <span class="n">MyParser</span><span class="p">()</span>
<span class="n">Traceback</span> <span class="p">(</span><span class="n">most</span> <span class="n">recent</span> <span class="n">call</span> <span class="n">last</span><span class="p">):</span>
<span class="n">File</span> <span class="s2">""</span><span class="p">,</span> <span class="n">line</span> <span class="mi">1</span><span class="p">,</span> <span class="ow">in</span>
<span class="n">File</span> <span class="s2">""</span><span class="p">,</span> <span class="n">line</span> <span class="mi">3</span><span class="p">,</span> <span class="ow">in</span> <span class="fm">__init__</span>
<span class="ne">TypeError</span><span class="p">:</span> <span class="n">must</span> <span class="n">be</span> <span class="nb">type</span><span class="p">,</span> <span class="ow">not</span> <span class="n">classobj</span>
</pre></div>
<p>What's with that? The <code>super(class, instance).__init__</code> idiom is the supposed proper way of calling a parent class
constructor, and it is -- if the class is a "new-style" Python class (one which inherits from <code>object</code>, or a class which
inherits from <code>object</code>).</p>
<p>And therein is the problem: <code>HTMLParser</code> inherits from <code>markupbase.ParserBase</code>, and <code>markupbase.ParserBase</code> is defined as:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">ParserBase</span><span class="p">:</span>
<span class="sd">"""Parser base class which provides some common support methods used</span>
<span class="sd"> by the SGML/HTML and XHTML parsers."""</span>
</pre></div>
<p>That is, as an <em>old</em> style class. One definitely wonders why in Python 2.7+ the classes that form part of the standard
library wouldn't all be new-style classes, <em>especially</em> when the class is intended as being something you inherit from
(like <code>HTMLParser</code>). Anywho, to fix:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">MyParser</span><span class="p">(</span><span class="n">HTMLParser</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="c1"># Old style way of doing super()</span>
<span class="n">HTMLParser</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span>
</pre></div>
<p>(Note: this post originally appeared on my blogspot blog at: <a href="http://codependentcodr.blogspot.ca/2012/02/python-htmlparser-and-super.html">http://codependentcodr.blogspot.ca/2012/02/python-htmlparser-and-super.html</a>)</p>