Fn/Log

Justifying Rust in the One Billion Rows Challenge

Unknown — Tue, 26 Mar 2024 00:00:00 +0000

Before I start my story, this is not a language battle on which is "better" but to see in action how a Rust</a> equivalent compares with a Go</a> implementation in terms of performance given its promises of performance without a garbage collector and precise memory management. After this challenge, I can see how good Go's automatic memory manager even if my implementation was able to outperform it. Preamble aside, I hope this journey can be informative as well as add some perspective to what Rust might need.

It started when a person was asking for help</a> with porting some Go gist to Rust and getting a significantly worse performance profile given how similar the translation was. The gist was for the One Billion Row Challenge</a> by Corlin Palmer</a> which is a straightforward solution with some clever optimizations. This piqued my interest and decided to take a stab at the challenge. Since the challenge deadline is over and mainly for Java, my goal is to only beat this Go implementation on even terms and not necessarily optimizing beyond it such as using mmap</a>, io_uring</a> or SIMD</a>.

The challenge</a> is simply parsing a generated text file with two data columns (weather station city and temperature measurement) and one billion rows, calculating aggregates (min, mean and max temperature per city) and printing the output in a JSON-like format. Generating this file is around 15 GB and can be done with the non-authoritative Python script from the main repository:

git</span> clone https://github.com/gunnarmorling/1brc cd 1brc
</span>
</span>cd src/main/python
</span>python</span> create_measurements.py 1_000_000_000
</span>
</span>mv</span> ../../../data/measurements.txt </span>$</span>HOME/measurements.txt
</span></code></pre>
To set the target, running the Go</em> code on this massive file:</p>
wget </span>'</span>https://gist.githubusercontent.com/corlinp/176a97c58099bca36bcd5679e68f9708/raw/5028f54e727f0f1716a2195de922460949a759fd/1brc_corlinp.go</span>'
</span>
</span>time</span> go run 1brc_corlinp.go </span>$</span>HOME/measurements.txt
</span># real          0m24.533s
</span></code></pre>
On my AMD Ryzen 7 machine with 32 GB RAM and 16 logical cores, it takes
an amazingly fast 24.5 seconds</strong></strong> that is my time to beat. (It does
break the rules a bit by fitting to a single dataset or assuming a
smaller city length, but who cares and this is for fun.) Since I rarely
tweak for performance, this is not my expertise as I enjoy Rust</em> for
its safety and ergonomics above all. This is to say my techniques are
simple and other more efficient implementations might appeal for more
technical readers, but I also assume the reader is comfortable enough
with the borrow checker. For this article, I will journey through with
the most basic solution, trying different leads and ending with the
winning solution.</p>

Table of Contents</div>

Baseline Solution (5m26s)</a></li>
Async Rust (5m51s)</a></li>
Byte Chunks (4m49s)</a></li>
Fixed Workers (5m12s)</a></li>
Object Pool (2m53s)</a></li>
String Interning (0m52s)</a></li>
No Rayon (0m38s)</a></li>
Integer Arithmetic (0m36s)</a></li>
Parser Iterator (0m33s)</a></li>
Unsafe str (0m29s)</a></li>
Allocation Cleanup (0m27.75s)</a></li>
Block Remove (0m26.5s)</a></li>
No Strings (0m24s)</a></li>
Tokio Tuning (0m23.5s)</a></li>
Conclusion</a></li>
Special Mention: mo8it/obrc (0m10s)</a></li>
Resources & References</a></li>
</ul>
</div>

>></a>
Baseline Solution (5m26s)</h2>
Reference: Commit</a> | main.rs</a></p>
A basic solution to this challenge is to read the file incrementally per
line to avoid loading the whole file to memory, parsing each line and
then storing the results to a hash map of cities and its aggregate
calculation. Starting with a rough implementation:</p>
const </span>KB</span>: </span>usize = </span>1024</span>;
</span>
</span>/// The Go code uses 256 KB buffer size.
</span>/// The default buffer size is 8 KB.
</span>const </span>BUF_SIZE</span>: </span>usize = </span>256 </span>* </span>KB</span>;
</span>
</span>fn </span>main</span>() {
</span>    </span>/// Read first commandline argument as file path
</span>    </span>let</span> path </span>= </span>std</span>::</span>env</span>::</span>args</span>().</span>nth</span>(</span>1</span>).</span>expect</span>("</span>no path given</span>");
</span>
</span>    </span>/// Use a buffered line reader over the file
</span>    </span>let</span> file </span>= </span>File</span>::</span>open</span>(</span>path</span>).</span>expect</span>("</span>file does not exist</span>");
</span>    </span>let mut</span> lines </span>= </span>BufReader</span>::</span>with_capacity</span>(</span>file</span>, </span>BUF_SIZE</span>).</span>lines</span>();
</span>
</span>    </span>/// Collect the results using a HashMap
</span>    </span>let mut</span> aggregate_map</span>: </span>HashMap</span><</span>String</span>, Aggregate</span>> </span>= </span>HashMap</span>::</span>new</span>();
</span>
</span>    </span>/// Read the file by line
</span>    </span>while let Some</span>(</span>Ok</span>(</span>line</span>)) </span>=</span> lines</span>.</span>next</span>() {
</span>        </span>/// Parse each line for a record
</span>        </span>let</span> Record </span>{</span> city</span>,</span> measurement </span>} </span>= </span>parse_line</span>(</span>&</span>line</span>);
</span>
</span>        </span>/// Record each measurement
</span>        aggregate_map
</span>            </span>.</span>entry</span>(</span>city</span>)
</span>            </span>.</span>and_modify</span>(|</span>aggregate</span>| </span>aggregate</span>.</span>record</span>(</span>measurement</span>))
</span>            </span>.</span>or_insert_with</span>(|| </span>Aggregate</span>::</span>new</span>(</span>measurement</span>));
</span>    </span>}
</span>}
</span></code></pre>
Tip: IO Buffering</a></p>
Two things to define in this code is Aggregate</code> struct and
parse_line</code>. First, the aggregate struct, Aggregate</code>, holds the
minimum, maximum and both the sum and count to calculate the mean of the
output:</p>
#[</span>derive</span>(</span>Debug</span>)]
</span>struct </span>Aggregate </span>{
</span>    </span>total</span>: </span>f32,
</span>    </span>count</span>: </span>usize,
</span>    </span>min</span>: </span>f32,
</span>    </span>max</span>: </span>f32,
</span>}
</span>
</span>/// Straightforward enough definition
</span>impl </span>Aggregate </span>{
</span>    </span>fn </span>new</span>(</span>measurement</span>: </span>f32</span>) -> </span>Self </span>{
</span>        </span>Self </span>{
</span>            total</span>:</span> measurement</span>,
</span>            count</span>: </span>1</span>,
</span>            min</span>:</span> measurement</span>,
</span>            max</span>:</span> measurement</span>,
</span>        </span>}
</span>    </span>}
</span>
</span>    </span>fn </span>record</span>(</span>&mut </span>self</span>, </span>measurement</span>: </span>f32</span>) {
</span>        </span>self</span>.</span>total </span>= self</span>.</span>total </span>+</span> measurement</span>;
</span>        </span>self</span>.</span>count </span>= self</span>.</span>count </span>+ </span>1</span>;
</span>        </span>self</span>.</span>min </span>= self</span>.</span>min</span>.</span>min</span>(</span>measurement</span>);
</span>        </span>self</span>.</span>max </span>= self</span>.</span>max</span>.</span>max</span>(</span>measurement</span>);
</span>    </span>}
</span>
</span>    </span>fn </span>mean</span>(</span>&</span>self</span>) -> </span>f32 </span>{
</span>        </span>self</span>.</span>total </span>/ </span>(</span>self</span>.</span>count </span>as f32</span>)
</span>    </span>}
</span>}
</span></code></pre>
Second, parse_line</code> parses a line for the city and temperature
measurement:</p>
/// Can be a two tuple of (String, f32) but I prefer structs for clarity
</span>#[</span>derive</span>(</span>Debug</span>)]
</span>struct </span>Record </span>{
</span>    </span>city</span>:</span> String,
</span>    </span>measurement</span>: </span>f32,
</span>}
</span>
</span>fn </span>parse_line</span>(</span>line</span>:</span> String</span>) -></span> Record </span>{
</span>    </span>let </span>(</span>raw_city</span>,</span> raw_measurement</span>) </span>=</span> line
</span>        </span>.</span>split_once</span>('</span>;</span>')
</span>        </span>.</span>expect</span>("</span>line does not have a semicolon</span>");
</span>
</span>    Record </span>{
</span>        city</span>:</span> raw_city</span>.</span>to_owned</span>(),
</span>        measurement</span>:</span> raw_measurement</span>.</span>parse</span>().</span>expect</span>("</span>measurement is not a f32</span>"),
</span>    </span>}
</span>}
</span></code></pre>
Since the data file is separated by a semicolon (;</code>), using
String::split_once</a> and String::parse</a> make this easy. Once the data has
been collected, the output is sorted by city which can be done by
converting the HashMap</code> to a sorted BTreeMap</code>:</p>
let</span> report_map</span>: </span>BTreeMap</span><</span>String</span>, Aggregate</span>> </span>=</span> aggregate_map</span>.</span>into_iter</span>().</span>collect</span>();
</span>
</span>for </span>(</span>city</span>,</span> aggregate</span>) </span>in</span> report_map </span>{
</span>    </span>println!</span>(
</span>        </span>"</span>City: {:<20} | </span>{:.2}</span> | </span>{:.2}</span> | </span>{:.2}</span>",
</span>        city</span>,
</span>        stat</span>.</span>min</span>,
</span>        stat</span>.</span>mean</span>(),
</span>        stat</span>.</span>max
</span>    </span>);
</span>}
</span></code></pre>
Tip: stdout Locking</a></p>
As a side note, this is not the challenge's output format but rather
from the person asking for help which I keep here and is easier to read
although this only matters for validating the results. To visualize the
difference between output format, this is the official format:</p>
{Abha=-23.0/18.0/59.2, Abidjan=-16.2/26.0/67.3}
</span></code></pre>
This is the alternative format:</p>
City | Abha                 | -23.00 | 18.00 | 59.20
</span>City | Abidjan              | -16.20 | 26.00 | 67.30
</span></code></pre>
With a working solution, how fast does it take it to run?</p>
time</span> cargo run</span> --</span>release</span> -- </span>$</span>HOME/measurements.txt
</span># real        05m26.50s
</span></code></pre>
Ignoring compile time and making sure it is a release build</a>, the
baseline solution is around 5 minutes 26 seconds</strong></strong> which is 13 times
slower than the target. Moving forward, this will be the command I use
to check the time.</p>
>></a>
Async Rust (5m51s)</h2>
Reference: Commit</a> | main.rs</a></p>
The initial solution's performance is somewhat expected given this does
not use asynchronous I/O to avoid blocking on file reads as well as use
all cores. To enable asynchronous I/O or Rust</em> overall, I will be
utilizing tokio</a> by adding it to the dependencies:</p>
[</span>dependencies</span>]
</span>tokio </span>= { </span>version </span>= "</span>1.36.0</span>", </span>features </span>= ["</span>full</span>"] }
</span></code></pre>
Migrating to asynchronous I/O is as simple as changing some imports from
std::fs</a> to tokio::fs</a> and adding a few await</code> points:</p>
/// From std::io and std::fs
</span>use </span>tokio</span>::{
</span>    </span>fs</span>::</span>File</span>,
</span>    </span>io</span>::{</span>AsyncBufReadExt</span>,</span> BufReader</span>},
</span>};
</span>
</span>/// Enable async
</span>#[</span>tokio</span>::</span>main</span>]
</span>async </span>fn </span>main</span>() {
</span>    </span>let</span> file </span>= </span>File</span>::</span>open</span>(</span>path</span>).</span>await</span>.</span>expect</span>("</span>file does not exist</span>");
</span>    </span>let mut</span> lines </span>= </span>BufReader</span>::</span>with_capacity</span>(</span>BUF_SIZE</span>,</span> file</span>).</span>lines</span>();
</span>
</span>    </span>while let Ok</span>(</span>Some</span>(</span>line</span>)) </span>=</span> lines</span>.</span>next_line</span>().</span>await </span>{
</span>    </span>}
</span>}
</span></code></pre>
After updating the code to async</code>, the performance is worse at 6
minutes 29 seconds</strong></strong>. Checking the process viewer, it still uses a
single thread because I did not use task::spawn</a> to spawn tasks on other
threads. Breaking down the current code into three tasks: read the file,
parse lines, aggregate results. This is a good starting point:</p>
/// Unbounded channels to pass data between tasks
</span>let </span>(</span>parser_tx</span>, </span>mut</span> parser_rx</span>) </span>= </span>mpsc</span>::</span>unbounded_channel</span>();
</span>let </span>(</span>collector_tx</span>, </span>mut</span> collector_rx</span>) </span>= </span>mpsc</span>::</span>unbounded_channel</span>();
</span>
</span>/// Task to read data and send to the parsing task
</span>let</span> reader_task </span>= </span>task</span>::</span>spawn</span>(</span>async </span>move </span>{
</span>    </span>while let Ok</span>(</span>Some</span>(</span>line</span>)) </span>=</span> lines</span>.</span>next_line</span>().</span>await </span>{
</span>        parser_tx</span>.</span>send</span>(</span>line</span>).</span>ok</span>();
</span>    </span>}
</span>});
</span>
</span>/// Task to parse records per line and send to the collector task
</span>let</span> parser_task </span>= </span>task</span>::</span>spawn</span>(</span>async </span>move </span>{
</span>    </span>while let Some</span>(</span>line</span>) </span>=</span> parser_rx</span>.</span>recv</span>().</span>await </span>{
</span>        </span>let</span> record </span>= </span>parse_line</span>(</span>line</span>);
</span>
</span>        collector_tx</span>.</span>send</span>(</span>record</span>).</span>ok</span>();
</span>    </span>}
</span>});
</span>
</span>/// Task to collect the records
</span>let</span> collector_task </span>= </span>task</span>::</span>spawn</span>(</span>async </span>move </span>{
</span>    </span>let mut</span> aggregate_map</span>: </span>HashMap</span><</span>String</span>, Aggregate</span>> </span>= </span>HashMap</span>::</span>new</span>();
</span>
</span>    </span>while let Some</span>(</span>record</span>) </span>=</span> collector_rx</span>.</span>recv</span>().</span>await </span>{
</span>        </span>let</span> Record </span>{</span> city</span>,</span> measurement </span>} </span>=</span> record</span>;
</span>
</span>        aggregate_map
</span>            </span>.</span>entry</span>(</span>city</span>)
</span>            </span>.</span>and_modify</span>(|</span>aggregate</span>| </span>aggregate</span>.</span>record</span>(</span>measurement</span>))
</span>            </span>.</span>or_insert_with</span>(|| </span>Aggregate</span>::</span>new</span>(</span>measurement</span>));
</span>    </span>}
</span>
</span>    aggregate_map
</span>});
</span>
</span>/// Wait for all tasks to finish
</span>let </span>(</span>_</span>, </span>_</span>,</span> collector_res</span>) </span>= </span>join!</span>(</span>reader_task</span>,</span> parser_task</span>,</span> collector_task</span>);
</span>let</span> aggregate_map </span>=</span> collector_res</span>.</span>expect</span>("</span>results not sent</span>");
</span></code></pre>
The code should look similar despite it being moved around to their own
task. Each task loop ends when their corresponding resource (file or
channel</a>) is dropped which creates a neat and clean task dependency
without any concurrency primitives</a>. While message passing between the
channels might be adding some overhead, checking the time again reveals
a faster yet overall slower 5 minutes 51 seconds</strong></strong> execution. Although
this may be a step backwards, fully utilizing tokio</code> may take a few
adjustments.</p>
>></a>
Byte Chunks (4m49s)</h2>
Reference: Commit</a> | main.rs</a></p>
By reading and passing only one line, it only gives each task little to
work with and no avenue to divide the task into more workers. So reading
a whole chunk of data and spawning a parsing task for each chunk might
be more efficient.</p>
Replacing the convenient BufReader::lines()</a> for the more low level
File::read_buf</a> requires some thought. When reading a whole chunk of
data, only chunks that end with a newline (\n</code>) should be sent to the
parser so it can split them neatly. For each new chunk, it is split at
the last newline and this leftover segment is merged at the beginning of
the next chunk to create a smooth data pipeline. The new reader task looks
like this:</p>
let</span> reader_task </span>= </span>task</span>::</span>spawn</span>(</span>async </span>move </span>{
</span>    </span>let mut</span> prev </span>= Vec</span>::</span>new</span>();
</span>
</span>    </span>loop </span>{
</span>        </span>/// Setting capacity is required for File.read_buf to work
</span>        </span>let mut</span> buf </span>= Vec</span>::</span>with_capacity</span>(</span>BUF_SIZE</span>);
</span>
</span>        </span>/// Add leftover segment
</span>        </span>if !</span>prev</span>.</span>is_empty</span>() {
</span>            buf</span>.</span>extend</span>(</span>prev</span>.</span>clone</span>());
</span>        </span>}
</span>
</span>        </span>/// Read chunk
</span>        file</span>.</span>read_buf</span>(</span>&mut</span> buf</span>).</span>await</span>.</span>expect</span>("</span>fill buffer failed</span>");
</span>        </span>let</span> buf_len </span>=</span> buf</span>.</span>len</span>();
</span>
</span>        </span>/// Split the chunk after (+1) the last newline.
</span>        </span>///
</span>        </span>/// Using Iterator::rposition is more efficient here than Iterator::position
</span>        </span>/// by starting at the end
</span>        </span>let</span> newline_index </span>=</span> buf
</span>            </span>.</span>iter</span>()
</span>            </span>.</span>rposition</span>(|</span>byte</span>| </span>*</span>byte </span>== b</span>'</span>\n</span>')
</span>            </span>.</span>expect</span>("</span>newline is missing</span>");
</span>
</span>        </span>/// Splitting the chunk with Vec::split_off makes this easy.
</span>        </span>///
</span>        </span>/// Minor edge case if the newline is the last byte
</span>        </span>if</span> newline_index </span>+ </span>1 </span><</span> buf_len </span>{
</span>            prev </span>=</span> buf</span>.</span>split_off</span>(</span>newline_index </span>+ </span>1</span>);
</span>        </span>} </span>else </span>{
</span>            prev </span>= Vec</span>::</span>new</span>();
</span>        </span>}
</span>
</span>        </span>/// Sanity check if the chunk is valid
</span>        </span>assert_eq!</span>(</span>*</span>buf</span>.</span>last</span>().</span>unwrap</span>(), </span>b</span>'</span>\n</span>');
</span>
</span>        parser_tx</span>.</span>send</span>(</span>buf</span>).</span>ok</span>();
</span>
</span>        </span>/// If fewer bytes are filled, this is the last chunk
</span>        </span>if</span> buf_len </span>< </span>BUF_SIZE </span>{
</span>            </span>break</span>;
</span>        </span>}
</span>    </span>}
</span>});
</span></code></pre>
Since a valid chunk is now passed to the parser task, it might be more
efficient to split this chunk using a parallel processing library like
rayon</a>:</p>
[</span>dependencies</span>]
</span>rayon </span>= "</span>1.9.0</span>"
</span></code></pre>
The juicy function here is ParallelSlice::par_split</a> which hopefully
splits the chunks faster than sequential iteration. A small caveat when
using rayon</code> with tokio</code> is avoiding a long-running task that can
degrade the runtime performance explained more thoroughly in this
article</a>. It is also ambiguous how thread resources should be utilized
between them without fine-tuning, but this is not a strong concern for
now.</p>
If the chunks are now processed into a list of records, it might be also
more effective to collect the results into a mini-aggregate instead of
passing multiple records. The parsing task should be:</p>
let</span> parser_task </span>= </span>task</span>::</span>spawn</span>(</span>async </span>move </span>{
</span>    </span>while let Some</span>(</span>chunk</span>) </span>=</span> parser_rx</span>.</span>recv</span>().</span>await </span>{
</span>        </span>let</span> collector_tx </span>=</span> collector_tx</span>.</span>clone</span>();
</span>
</span>        </span>/// Spawn a task to parse this chunk
</span>        </span>task</span>::</span>spawn</span>(</span>async </span>move </span>{
</span>            </span>/// Since chunk ends with \n, par_split will create an empty
</span>            </span>/// chunk, so skipping the last byte avoids this problem for
</span>            </span>/// parse_line.
</span>            </span>let</span> records</span>: </span>Vec</span><</span>_</span>> </span>=</span> chunk</span>[</span>0</span>..</span>chunk</span>.</span>len</span>() </span>- </span>1</span>]
</span>                </span>.</span>par_split</span>(|</span>byte</span>| </span>*</span>byte </span>== b</span>'</span>\n</span>')
</span>                </span>.</span>map</span>(</span>parse_line</span>)
</span>                </span>.</span>collect</span>();
</span>
</span>            </span>/// A good point to yield if parsing takes too long
</span>            </span>task</span>::</span>yield_now</span>().</span>await</span>;
</span>
</span>            </span>/// Collect into a mini-aggregate
</span>            </span>let mut</span> aggregate_map </span>= </span>HashMap</span>::</span>new</span>();
</span>
</span>            </span>for</span> record </span>in</span> records </span>{
</span>            </span>}
</span>
</span>            collector_tx</span>.</span>send</span>(</span>aggregate_map</span>).</span>ok</span>();
</span>        </span>});
</span>    </span>}
</span>});
</span></code></pre>
Since parse_line</code> needs to take a byte slice (&[u8]</code>) instead of a
string reference (&str</code>), the function has to be updated:</p>
fn </span>parse_line</span>(</span>line</span>: </span>&</span>[</span>u8</span>]</span>) -></span> Record </span>{
</span>    </span>/// Since slice.split_once is locked behind a feature gate, do it the
</span>    </span>/// longer way.
</span>    </span>///
</span>    </span>/// See https://github.com/rust-lang/rust/issues/112811
</span>    </span>let</span> semicolon_index </span>=</span> line
</span>        </span>.</span>iter</span>()
</span>        </span>.</span>rposition</span>(|</span>byte</span>| </span>*</span>byte </span>== b</span>'</span>;</span>')
</span>        </span>.</span>expect</span>("</span>line does not have a semicolon</span>");
</span>
</span>    </span>let</span> raw_city </span>= </span>std</span>::</span>str</span>::</span>from_utf8</span>(</span>&</span>line</span>[</span>0</span>..</span>semicolon_index</span>]).</span>expect</span>("</span>city is not UTF8</span>");
</span>    </span>let</span> raw_measurement </span>=
</span>        </span>std</span>::</span>str</span>::</span>from_utf8</span>(</span>&</span>line</span>[</span>semicolon_index </span>+ </span>1</span>..</span>]).</span>expect</span>("</span>measurement is not UTF8</span>");
</span>
</span>    </span>/// ...
</span>}
</span></code></pre>
Lastly, the collector task is the easiest to update to combine
mini-aggregates instead of a single record:</p>
impl </span>Aggregate </span>{
</span>    </span>/// Merging two aggregates together
</span>    </span>fn </span>merge</span>(</span>&mut </span>self</span>, </span>other</span>: </span>&Self</span>) {
</span>        </span>self</span>.</span>total </span>= self</span>.</span>total </span>+</span> other</span>.</span>total</span>;
</span>        </span>self</span>.</span>count </span>= self</span>.</span>count </span>+</span> other</span>.</span>count</span>;
</span>        </span>self</span>.</span>min </span>= self</span>.</span>min</span>.</span>min</span>(</span>other</span>.</span>min</span>);
</span>        </span>self</span>.</span>max </span>= self</span>.</span>max</span>.</span>max</span>(</span>other</span>.</span>max</span>);
</span>    </span>}
</span>}
</span>
</span>let</span> collector_task </span>= </span>task</span>::</span>spawn</span>(</span>async </span>move </span>{
</span>    </span>let mut</span> aggregate_map</span>: </span>HashMap</span><</span>String</span>, Aggregate</span>> </span>= </span>HashMap</span>::</span>new</span>();
</span>
</span>    </span>while let Some</span>(</span>other_map</span>) </span>=</span> collector_rx</span>.</span>recv</span>().</span>await </span>{
</span>        </span>/// Merging the smaller aggregate map to the larger one
</span>        </span>for </span>(</span>city</span>,</span> other_aggregate</span>) </span>in</span> other_map </span>{
</span>            aggregate_map
</span>                </span>.</span>entry</span>(</span>city</span>)
</span>                </span>.</span>and_modify</span>(|</span>aggregate</span>| </span>aggregate</span>.</span>merge</span>(</span>&</span>other_aggregate</span>))
</span></mark>                </span>.</span>or_insert_with</span>(|| </span>other_aggregate</span>);
</span>        </span>}
</span>    </span>}
</span>
</span>    aggregate_map
</span>});
</span></code></pre>
The most work by far and it does produce faster results at around 4
minutes 49 seconds</strong></strong>.</p>
>></a>
Fixed Workers (5m12s)</h2>
Reference: Commit</a> | main.rs</a></p>
At this point, the code is similar enough to Go</em> in its overall
approach yet it is significantly slower. Checking the process monitor
again, I did notice the memory usage was high enough to use all my RAM
and some of my swap memory which is strange as the file is read in
chunks. After tinkering with the time format option</a> to reveal memory
stats, how much memory does the code use?</p>
# For -f to work, use /usr/bin/time instead of just time
</span>$(</span>which time</span>) -</span>f </span>'</span>Time: %E | Memory: %M | Swapped: %W | Context Switched: %w</span>'</span> cargo run</span> --</span>release</span> -- </span>$</span>HOME/measurements.txt
</span># Time: 4:52.60 | Memory: 29893648 | Swapped: 23559520 | Context Switched: 5698047
</span>
</span># For comparison
</span>$(</span>which time</span>) -</span>f </span>'</span>Time: %E | Memory: %M | Swapped: %W | Context Switched: %w</span>'</span> go run 1brc_corlinp.go </span>$</span>HOME/measurements.txt
</span># Time: 0:26.06 | Memory: 1363336 | Swapped: 0 | Context Switched: 25325
</span></code></pre>
If I read this correctly, my code uses around 30 GB</strong></strong> of memory and
occasional swap memory used compared to the Go</em> code with only 1.3 GB</strong></strong> of
memory, no swaps and fewer context switches. Clearly, the immediate
objective is to reduce excessive memory usage. Moving forward again,
this will be new command to check both time and now memory.</p>
To address first a glaring fault in my code, I spawned an unbounded
number of parsing tasks which maybe causing some huge memory and context
switching overhead. The quick fix is to spawn a fixed number of workers
and send each incoming chunk to a worker sequentially:</p>
/// The Go code uses 256 workers
</span>const </span>NUM_WORKERS</span>: </span>usize = </span>256</span>;
</span>
</span>let</span> parser_task </span>= </span>task</span>::</span>spawn</span>(</span>async </span>move </span>{
</span>    </span>let mut</span> worker_txs</span>: </span>Vec</span><</span>UnboundedSender</span><</span>Vec</span><</span>u8</span>>>> </span>= Vec</span>::</span>with_capacity</span>(</span>NUM_WORKERS</span>);
</span>
</span>    </span>for _ in </span>0</span>..</span>NUM_WORKERS </span>{
</span>        </span>/// Create workers and only store their send channels
</span>        </span>let </span>(</span>worker_tx</span>, </span>mut</span> worker_rx</span>) </span>= </span>mpsc</span>::</span>unbounded_channel</span>();
</span>        worker_txs</span>.</span>push</span>(</span>worker_tx</span>);
</span>
</span>        </span>let</span> collector_tx </span>=</span> collector_tx</span>.</span>clone</span>();
</span>
</span>        </span>task</span>::</span>spawn</span>(</span>async </span>move </span>{
</span>            </span>while let Some</span>(</span>chunk</span>) </span>=</span> worker_rx</span>.</span>recv</span>().</span>await </span>{
</span>            </span>}
</span>        </span>});
</span>    </span>}
</span>
</span>    </span>/// Send each new chunk to the next worker
</span>    </span>let mut</span> worker_iter </span>= </span>(</span>0</span>..</span>worker_txs</span>.</span>len</span>()).</span>cycle</span>();
</span>
</span>    </span>while let Some</span>(</span>chunk</span>) </span>=</span> parser_rx</span>.</span>recv</span>().</span>await </span>{
</span>        </span>/// I would have used Iter::cycle_mut if it existed
</span>        </span>let</span> worker_index </span>=</span> worker_iter</span>.</span>next</span>().</span>unwrap</span>();
</span>        </span>let</span> worker_tx </span>= &mut</span> worker_txs</span>[</span>worker_index</span>];
</span>
</span>        </span>/// Moved the parsing code to the worker loop
</span>        worker_tx</span>.</span>send</span>(</span>chunk</span>).</span>ok</span>();
</span>    </span>}
</span>});
</span></code></pre>
Tip: Task Pool</a></p>
Side note, I did find tokio_task_pool</a> before but was not sure of its
quality and it was a good opportunity to write a simple worker pool. The
only caveat with this is fine-tuning the number of workers for
performance, but it is still slightly slower at 5 minutes 12
seconds</strong></strong>. For now, I will keep this change just to keep the memory
bounded.</p>
>></a>
Object Pool (2m53s)</h2>
Reference: Commit</a> | main.rs</a></p>
Since Go</em> uses a memory pool to reduce heap allocations</a>, reusing
allocations as well by adding an object pool might be the way to go. One
of the biggest allocations and opportunity is the large read buffer.
Although I initially did write my own object pool and worked, I decided
to incorporate the more robust object-pool</a> to the mix:</p>
[</span>dependencies</span>]
</span>object-pool </span>= "</span>0.5.4</span>"
</span># To get the number of cores for allocation
</span>num_cpus </span>= "</span>1.16.0</span>"
</span># To allow static initializers
</span>once_cell </span>= "</span>1.19.0</span>"
</span></code></pre>
I want to easily create a global object pool where I allocate one buffer
for each logical core, so that is why I also pulled in once_cell</a> for
once_cell::sync::Lazy</a> and num_cpus</a> for num_cpus.get</a>. Aside from the
convenience crates, the changes in the code is quite minimal:</p>
/// Create a pool with a byte buffer for each core
</span>static </span>BUF_POOL</span>: </span>Lazy</span><</span>Pool</span><</span>Vec</span><</span>u8</span>>>> </span>=
</span>    </span>Lazy</span>::</span>new</span>(|| </span>Pool</span>::</span>new</span>(</span>num_cpus</span>::</span>get</span>(), || </span>Vec</span>::</span>with_capacity</span>(</span>BUF_SIZE</span>)));
</span>
</span>/// Reuse an available buffer
</span>let mut</span> buf</span>: </span>Reusable</span><</span>Vec</span><</span>u8</span>>> </span>= if let Some</span>(</span>buf</span>) </span>= </span>BUF_POOL</span>.</span>try_pull</span>() {
</span></mark>    buf
</span></mark>} </span>else </span>{
</span>    </span>/// Important to work on another task first before retrying
</span>    </span>task</span>::</span>yield_now</span>().</span>await</span>;
</span>
</span>    </span>continue</span>;
</span>};
</span>
</span>/// Reset the reused buffer
</span>buf</span>.</span>clear</span>();
</span>buf</span>.</span>reserve</span>(</span>BUF_SIZE</span>);
</span></code></pre>
Checking out the performance of this change gives us a better 3
minutes 1 seconds</strong></strong> with less memory used. If this works, what if I
apply it to the aggregates as well? Before that, the challenge specifies
a maximum of 10,000 unique cities</em>, so the overall aggregate result can
be prepared with that capacity:</p>
const </span>NUM_CITIES</span>: </span>usize = </span>10000</span>;
</span>
</span>let mut</span> aggregate_map</span>: </span>HashMap</span><</span>String</span>, Aggregate</span>> </span>= </span>HashMap</span>::</span>with_capacity</span>(</span>NUM_CITIES</span>);
</span></mark></code></pre>
For the mini-aggregates, waiting for an available object here is not
advisable nor am I certain what a good initial capacity is. So I opt to
allocate when needed and hope 10,000</em> or NUM_CITIES</code> is good enough.
Another challenge rule is that the maximum length of each weather
station city is 100</em>, so each aggregate can have an estimated capacity
of the chunk size divided by that value:</p>
const </span>CITY_LEN</span>: </span>usize = </span>100</span>;
</span>
</span>/// Likewise, a global aggregate pool
</span>static </span>AGGREGATE_POOL</span>: </span>Lazy</span><</span>Pool</span><</span>HashMap</span><</span>String</span>, Aggregate</span>>>> </span>=
</span>    </span>Lazy</span>::</span>new</span>(|| </span>Pool</span>::</span>new</span>(</span>NUM_CITIES</span>, || </span>HashMap</span>::</span>with_capacity</span>(</span>BUF_SIZE </span>/ </span>CITY_LEN</span>)));
</span>
</span>/// Reuse a mini-aggregate or create one again
</span>let mut</span> aggregate_map </span>=
</span></mark>    </span>AGGREGATE_POOL</span>.</span>pull</span>(|| </span>HashMap</span>::</span>with_capacity</span>(</span>BUF_SIZE </span>/ </span>CITY_LEN</span>));
</span></mark>/// Make sure to clear before use
</span>aggregate_map</span>.</span>clear</span>();
</span></mark></code></pre>
After all the prepped allocations, the new code runs at 2 minutes 53
seconds</strong></strong> and the memory usage better at 19 GB</strong></strong> which is better but
still large.</p>
>></a>
String Interning (0m52s)</h2>
Reference: Commit</a> | main.rs</a></p>
While the object pool did help, many objects are still being allocated.
On a closer look, the code allocates a lot of city strings just to be a
key in a hash map and only sorted near the end. So what if I place every
city string in a global hash map where the key is a unique yet cheap to
copy identifier? My initial prototype was using hash::BuildHasher</a> with
once_map::OnceMap</a> which is a concurrent hash map where entries can only
be inserted and not modified:</p>
static </span>STRING_MAP</span>: </span>Lazy</span><</span>OnceMap</span><</span>u64</span>, </span>String</span>>> </span>= </span>Lazy</span>::</span>new</span>(|| </span>OnceMap</span>::</span>new</span>());
</span>
</span>/// Use the map hash as the key
</span>let</span> hash </span>= </span>STRING_MAP</span>.</span>hasher</span>().</span>hash_one</span>(</span>raw_city</span>);
</span>
</span>if !</span>STRING_MAP</span>.</span>contains_key</span>(</span>&</span>hash</span>) {
</span>    </span>STRING_MAP</span>.</span>insert</span>(</span>hash</span>, </span>move |_|</span> raw_city</span>.</span>to_owned</span>());
</span>};
</span></code></pre>
This concept sounded familiar and I recognized this as string interning</a>.
Instead of my prototype, I decided to pull in a more mature crate like
lasso</a> for the job:</p>
[</span>dependencies</span>]
</span>lasso </span>= { </span>version </span>= "</span>0.7.2</span>", </span>features </span>= ["</span>multi-threaded</span>", "</span>ahasher</span>"] }
</span></code></pre>
I will be using lasso::ThreadedRodeo</a> with ahash</a> as the hasher for a fast
keyed hash</a>. The key function is ThreadedRodeo::get_or_intern</a> that will
be used during the parsing function:</p>
/// Create a global string map
</span>static </span>STRING_MAP</span>: </span>Lazy</span><</span>ThreadedRodeo</span>> </span>=
</span>    </span>Lazy</span>::</span>new</span>(|| </span>ThreadedRodeo</span>::</span>with_capacity</span>(</span>Capacity</span>::</span>for_strings</span>(</span>NUM_CITIES</span>)));
</span>
</span>#[</span>derive</span>(</span>Debug</span>)]
</span>struct </span>Record </span>{
</span>    </span>/// Change city to city_key as well as from String to lasso::Spur
</span>    </span>city_key</span>:</span> Spur,
</span></mark>}
</span>
</span>fn </span>parse_line</span>(</span>line</span>: </span>&</span>[</span>u8</span>]</span>) -></span> Record </span>{
</span>    </span>/// Intern the string slice for a key
</span>    </span>let</span> city_key </span>= </span>STRING_MAP</span>.</span>get_or_intern</span>(</span>raw_city</span>);
</span></mark>
</span>    Record </span>{
</span>        city_key</span>,
</span></mark>    </span>}
</span>}
</span></code></pre>
lasso::Spur</a> is a 32 bit key which is much better than my 64 bit key
hash. Since the key type changed, I have to update a few types and
resolve</a> the key into a string during the sorting process which is easy
to refactor:</p>
let</span> report_map</span>: </span>BTreeMap</span><</span>&str</span>, Aggregate</span>> </span>=</span> aggregate_map
</span></mark>    </span>.</span>into_iter</span>()
</span>    </span>/// Convert the key back into a string slice
</span>    </span>.</span>map</span>(|(</span>city_key</span>, </span>aggregate</span>)| (</span>STRING_MAP</span>.</span>resolve</span>(</span>&</span>city_key</span>),</span> aggregate</span>))
</span></mark>    </span>.</span>collect</span>();
</span></code></pre>
After updating the code, it runs much faster and better at below 52
seconds</strong></strong> and with a memory usage of 1.2 GB</strong></strong> which is less than
Go</em>'s 1.3 GB</strong></strong>. Amazing how much memory those small strings
contributed to the performance of the code.</p>
>></a>
No Rayon (0m38s)</h2>
Reference: Commit</a> | main.rs</a></p>
If any small but numerous heap allocations are the issue, then rayon</code>
is on the chopping block for allocating intermediate vectors</a> as part of
its parallel processing. What would happen if the code was refactored
without rayon</code> to optimize for memory instead? Looking at
task::spawn_blocking</a>,
it can create threads on demand as well as put the chunks into a queue
unlike task::spawn</a>. This means the parser workers can be reverted and
back to its simpler form:</p>
/// The parser task back to its simpler form
</span>let</span> parser_task </span>= </span>task</span>::</span>spawn</span>(</span>async </span>move </span>{
</span>    </span>while let Some</span>(</span>chunk</span>) </span>=</span> parser_rx</span>.</span>recv</span>().</span>await </span>{
</span>        </span>/// From task::spawn to task::spawn_blocking
</span>        </span>task</span>::</span>spawn_blocking</span>(</span>move || </span>{
</span></mark>            </span>/// Turn the parser into an iterator
</span>            </span>let mut</span> record_iter </span>=</span> chunk</span>[</span>0</span>..</span>chunk</span>.</span>len</span>()]
</span></mark>                </span>.</span>split</span>(|</span>byte</span>| </span>*</span>byte </span>== b</span>'</span>\n</span>')
</span></mark>                </span>.</span>map</span>(</span>parse_line</span>);
</span></mark>
</span>            </span>while let Some</span>(</span>record</span>) </span>=</span> record_iter</span>.</span>next</span>() {
</span>            </span>}
</span>        </span>});
</span>    </span>}
</span>});
</span></code></pre>
A notable change is that parsing records is now an iterator which will
be improved later on. With this simple refactor, the time is now at 38
seconds</strong></strong> which is a minor improvement, but the code does feel less
complex without another thread pool.</p>
>></a>
Integer Arithmetic (0m36s)</h2>
Reference: Commit</a> | main.rs</a></p>
Aside from one more minor savings in vector allocation, it might be
better to optimize in other avenues again. The challenge specifies that
the temperature measurements are floating numbers from -99.9</em> to 99.9</em>
with the decimal number always present. Meaning instead of storing it as
a f32</a>, it can be stored as an i16</a> where the value is multiplied by 10 to
be an integer which can take advantage of integer arithmetic and finally
divided by 10 when displayed. Seeing it may be better:</p>
/// Sum is now an isize instead of f32
</span>let</span> isum</span>: </span>isize = </span>0</span>;
</span>let</span> count</span>: </span>usize = </span>0</span>;
</span>
</span>/// Temperatures always have 1 decimal value
</span>let</span> temps </span>= </span>vec!</span>[ </span>-</span>54.3</span>, </span>21.0 </span>-</span>0.9</span>, </span>77.0 </span>];
</span>
</span>for</span> temp </span>in</span> temps </span>{
</span>    </span>///  Hypothetical value from parsing a line
</span>    </span>let</span> itemp </span>= </span>(</span>temp </span>* </span>10</span>) </span>as i16</span>;
</span>
</span>    isum </span>+=</span> itemp </span>as isize</span>;
</span>}
</span>
</span>/// Convert integer sum back to f32
</span>println!</span>("</span>MEAN: {.2}</span>", ((</span>isum </span>as f32</span>) </span>/ </span>10</span>) </span>/ </span>(</span>count </span>as f32</span>));
</span></code></pre>
Changing the types are easy and added the i</code> prefix in the field names
to indicate them as float as integer variables:</p>
/// Added an 'i' prefix for each variable to indicate the change
</span>#[</span>derive</span>(</span>Debug</span>)]
</span>struct </span>Record </span>{
</span>    </span>city_key</span>:</span> Spur,
</span>    </span>imeasurement</span>: </span>i16,
</span></mark>}
</span>
</span>#[</span>derive</span>(</span>Debug</span>)]
</span>struct </span>Aggregate </span>{
</span>    </span>itotal</span>: </span>isize,
</span></mark>    </span>count</span>: </span>usize,
</span>    </span>imin</span>: </span>i16,
</span></mark>    </span>imax</span>: </span>i16,
</span></mark>}
</span>
</span>/// Display function updated as well as new min and max function
</span>impl </span>Aggregate </span>{
</span>    </span>fn </span>min</span>(</span>&</span>self</span>) -> </span>f32 </span>{
</span>        </span>(</span>self</span>.</span>imin </span>as f32</span>) </span>/ </span>10.0
</span></mark>    </span>}
</span>
</span>    </span>fn </span>mean</span>(</span>&</span>self</span>) -> </span>f32 </span>{
</span>        </span>(</span>self</span>.</span>itotal </span>as f32</span>) </span>/ </span>(</span>self</span>.</span>count </span>as f32</span>) </span>/ </span>10.0
</span></mark>    </span>}
</span>
</span>    </span>fn </span>max</span>(</span>&</span>self</span>) -> </span>f32 </span>{
</span>        </span>(</span>self</span>.</span>imax </span>as f32</span>) </span>/ </span>10.0
</span></mark>    </span>}
</span>}
</span></code></pre>
Aside from some minor type casting changes, the parsing of measurement
needs to be updated since f32 parsing</a> cannot be used. The parser should
be able to handle the measurement format that boils down to:</p>

Optional Hyphen</li>
Digit</li>
Optional Digit</li>
Period</li>
Digit</li>
</ol>
Translating this as an iterator:</p>
let</span> raw_measurement </span>= &</span>line</span>[</span>semicolon_index </span>+ </span>1</span>..</span>];
</span>
</span>/// Parsing byte by byte with an iterator
</span>let mut</span> measurement_iter </span>=</span> raw_measurement</span>.</span>iter</span>().</span>peekable</span>();
</span>
</span>/// 1. Optional Sign
</span>///
</span>/// If the first character is an hyphen, the number is negative and
</span>/// advance the iterator.
</span>let</span> sign </span>= if **</span>measurement_iter</span>.</span>peek</span>().</span>unwrap</span>() </span>== b</span>'</span>-</span>' {
</span>    </span>let</span> _hyphen </span>=</span> measurement_iter</span>.</span>next</span>();
</span>
</span>    </span>-</span>1 </span>as i16
</span>} </span>else </span>{
</span>    </span>1 </span>as i16
</span>};
</span>
</span>/// 2. Digit
</span>///
</span>/// A neat trick to get the digit value from a byte
</span>let</span> first_digit </span>= </span>(</span>measurement_iter</span>.</span>next</span>().</span>unwrap</span>() </span>- b</span>'</span>0</span>') </span>as i16</span>;
</span>
</span>/// 3 & 4. Optional Digit and Period
</span>///
</span>/// If the next byte is a period, the second digit is 0; otherwise, it
</span>/// is a digit followed by a period
</span>let</span> second_digit </span>= if **</span>measurement_iter</span>.</span>peek</span>().</span>unwrap</span>() </span>== b</span>'</span>.</span>' {
</span>    </span>let</span> _period </span>=</span> measurement_iter</span>.</span>next</span>();
</span>
</span>    </span>0 </span>as i16
</span>} </span>else </span>{
</span>    </span>let</span> digit </span>= </span>(</span>measurement_iter</span>.</span>next</span>().</span>unwrap</span>() </span>- b</span>'</span>0</span>') </span>as i16</span>;
</span>    </span>let</span> _period </span>=</span> measurement_iter</span>.</span>next</span>();
</span>
</span>    digit
</span>};
</span>
</span>/// 5. Digit
</span>let</span> third_digit </span>= </span>(</span>measurement_iter</span>.</span>next</span>().</span>unwrap</span>() </span>- b</span>'</span>0</span>') </span>as i16</span>;
</span>
</span>/// Safety assertion that it is the end
</span>assert!</span>(</span>measurement_iter</span>.</span>next</span>().</span>is_none</span>());
</span>
</span>/// Calculate the integer measurement using the digits and sign
</span>let</span> imeasurement </span>=</span> sign </span>* </span>(</span>first_digit </span>* </span>100 </span>+</span> second_digit </span>* </span>10 </span>+</span> third_digit</span>);
</span>
</span></code></pre>
This change improves the time to 36 seconds</strong></strong>.</p>
>></a>
Parser Iterator (0m33s)</h2>
Reference: Commit</a> | main.rs</a></p>
The last improvement with the line parsing is to minimize reading the
large buffer as much as possible. The record iterator uses the
slice::split</a> that reads the buffer once to pass to parse_line</code> which.
Removing this extra iteration is a potential optimization:</p>
let mut</span> record_iter </span>=</span> chunk</span>[</span>0</span>..</span>chunk</span>.</span>len</span>() </span>- </span>1</span>]
</span>    </span>.</span>split</span>(|</span>byte</span>| </span>*</span>byte </span>== b</span>'</span>\n</span>') </span>/// Extra iteration on the buffer
</span></mark>    </span>.</span>map</span>(</span>parse_line</span>);
</span></code></pre>
Tip: Iterator Chaining</a></p>
The next move is to expand parse_line</code> as an iterator</a> to take the whole
chunk instead of a line:</p>
/// What it should look like.
</span>let mut</span> record_iter </span>= </span>parse_chunk</span>(</span>&</span>chunk</span>);
</span></mark>
</span>fn </span>parse_chunk</span><</span>'a</span>>(</span>bytes</span>: </span>&'a</span> [</span>u8</span>]</span>) -> </span>ChunkParser</span><</span>'a</span>> {
</span>    ChunkParser </span>{</span> bytes </span>}
</span>}
</span>
</span>/// Define the record iterator
</span>struct </span>ChunkParser</span><</span>'a</span>> {
</span>    </span>bytes</span>: </span>&'a [u8],
</span>}
</span>
</span>impl</span><</span>'a</span>></span> Iterator </span>for </span>ChunkParser</span><</span>'a</span>> {
</span>    </span>type </span>Item </span>=</span> Record</span>;
</span>
</span>    </span>fn </span>next</span>(</span>&mut </span>self</span>) -> </span>Option</span><</span>Self</span>::</span>Item</span>> {
</span>        </span>/// Move expanded parse_line here
</span>    </span>}
</span>}
</span></code></pre>
Since the iterator takes a borrowed slice, the trick to consuming it is
to borrow the slice again after the first newline until it is empty. To
sanely get the index, I had to use Iterator::enumerate</a> that made the
measurement_iter</code> change its return type and slightly updated the
parse_line</code> logic. Showing just the main changes as an iterator:</p>
/// Termination condition
</span>if self</span>.</span>bytes</span>.</span>is_empty</span>() {
</span>    </span>return None</span>;
</span>}
</span>
</span>let</span> semicolon_index </span>= self</span>.</span>bytes</span>.</span>iter</span>().</span>position</span>(|</span>byte</span>| </span>*</span>byte </span>== b</span>'</span>;</span>').</span>unwrap</span>();
</span>
</span>/// Reborrow here to make it easier for the next section
</span>self</span>.</span>bytes </span>= &self</span>.</span>bytes</span>[</span>semicolon_index </span>+ </span>1</span>..</span>];
</span>
</span>/// Adding Iterator::enumerate to determine the position of the last byte
</span>let mut</span> measurement_iter </span>= self</span>.</span>bytes</span>.</span>iter</span>().</span>enumerate</span>().</span>peekable</span>();
</span>
</span>
</span>let</span> imeasurement </span>=</span> sign </span>* </span>(</span>first_digit </span>* </span>100 </span>+</span> second_digit </span>* </span>10 </span>+</span> third_digit</span>);
</span>
</span>/// A newline must also be consumed now
</span>let </span>(</span>_</span>,</span> newline</span>) </span>=</span> measurement_iter</span>.</span>next</span>().</span>unwrap</span>();
</span>assert_eq!</span>(</span>*</span>newline</span>, </span>b</span>'</span>\n</span>');
</span>
</span>/// If the iterator has more bytes, move to that index
</span>if let Some</span>((</span>next_index</span>, </span>_</span>)) </span>=</span> measurement_iter</span>.</span>peek</span>() {
</span>    </span>self</span>.</span>bytes </span>= &self</span>.</span>bytes</span>[</span>*</span>next_index</span>..</span>];
</span>} </span>else </span>{
</span>    </span>/// Borrow an empty slice to end the cycle
</span>    </span>self</span>.</span>bytes </span>= &self</span>.</span>bytes</span>[</span>0</span>..</span>0</span>];
</span>}
</span></code></pre>
Removing the extra iteration brings the time down to 33 seconds</strong></strong>.</p>
>></a>
Unsafe str (0m29s)</h2>
Reference: Commit</a> | main.rs</a></p>
While working on the previous change, I noticed the following line:</p>
let</span> city_bytes </span>= &self</span>.</span>bytes</span>[</span>0</span>..</span>semicolon_index</span>];
</span>
</span>/// Is the string conversion fast?
</span>let</span> raw_city </span>=
</span>    </span>std</span>::</span>str</span>::</span>from_utf8</span>(</span>city_bytes</span>).</span>expect</span>("</span>city is not UTF8</span>");
</span></mark></code></pre>
The function str::from_utf8</a> checks first if the bytes are UTF-8 valid
before performing the conversion which is good. Since the data is
guaranteed to be valid, the check can be skipped in favor of
str::from_utf8_unchecked</a> to convert the type directly:</p>
let</span> city_bytes </span>= &self</span>.</span>bytes</span>[</span>0</span>..</span>semicolon_index</span>];
</span>
</span>/// Safety: Skip the validity check given the challenge data rules
</span>let</span> raw_city </span>= unsafe </span>{ </span>std</span>::</span>str</span>::</span>from_utf8_unchecked</span>(</span>city_bytes</span>) };
</span></mark></code></pre>
Tip: Bounds Check</a></p>
This is the only time unsafe Rust</a> is used so no need to panic. Still,
the time did go down to 29 seconds</strong></strong> for just skipping data
validation.</p>
>></a>
Allocation Cleanup (0m27.75s)</h2>
Reference: Commit</a> | main.rs</a></p>
Using the object pool did help with the memory usage. Since I limited
the number of buffers in the pool, it has to wait:</p>
let mut</span> buf</span>: </span>Reusable</span><</span>Vec</span><</span>u8</span>>> </span>= if let Some</span>(</span>buf</span>) </span>= </span>BUF_POOL</span>.</span>try_pull</span>() {
</span>    buf
</span>} </span>else </span>{
</span>    </span>/// How many times does this branch trigger?
</span>    </span>task</span>::</span>yield_now</span>().</span>await</span>;
</span></mark>
</span>    </span>continue</span>;
</span>};
</span></code></pre>
Checking the branch with a counter, this branch is triggered on average
300000 times. One way to reduce the branch entry is to increase the
number of available buffers. Going that route requires
num_cpus::get() * 32</code> or 512 buffers which increases the memory
allocations which did make it run slightly slower. Another route perhaps
is using the bytes</a> crate which is a more efficient shared byte buffer:</p>
[</span>dependencies</span>]
</span>bytes </span>= "</span>1.5.0</span>"
</span></code></pre>
It is functionally an Arc<Vec<u8>></code> with more optimizations as well as
utilizing disjointed buffers (rope structure</a>) which coincides with
joining each chunk with the previous leftover. Since the API is similar
to Vec</a>, it is simply a type change:</p>
let</span> reader_task </span>= </span>task</span>::</span>spawn</span>(</span>async </span>move </span>{
</span>    </span>/// Bytes::new is zero allocation
</span>    </span>let mut</span> prev </span>= </span>Bytes</span>::</span>new</span>();
</span></mark>    </span>/// Create an empty buffer
</span>    </span>let</span> buf </span>= </span>BytesMut</span>::</span>with_capacity</span>(</span>BUF_SIZE</span>);
</span></mark>
</span>    </span>loop </span>{
</span>        </span>/// Instead of using an object pool, cloning the empty buffer
</span>        </span>/// works well enough
</span>        </span>let mut</span> buf</span>:</span> BytesMut </span>=</span> buf</span>.</span>clone</span>();
</span></mark>    </span>}
</span>})
</span></code></pre>
This is strangely more efficient bringing the time down to 27.5
seconds</strong></strong>.</p>
>></a>
Block Remove (0m26.5s)</h2>
Reference: Commit</a> | main.rs</a></p>
To satisfy the compiler, I added HashMap::drain</a> while refactoring on
floating to integer types for measurements:</p>
while let Some</span>(</span>mut</span> other_map</span>) </span>=</span> collector_rx</span>.</span>recv</span>().</span>await </span>{
</span>    </span>/// HashMap::drain so that insert works
</span>    </span>for </span>(</span>city</span>,</span> other_aggregate</span>) </span>in</span> other_map</span>.</span>drain</span>() {
</span></mark>        aggregate_map
</span>            </span>.</span>entry</span>(</span>city</span>)
</span>            </span>.</span>and_modify</span>(|</span>aggregate</span>| </span>aggregate</span>.</span>merge</span>(</span>&</span>other_aggregate</span>))
</span>            </span>/// Insert requires owned data
</span>            </span>.</span>or_insert_with</span>(|| </span>other_aggregate</span>);
</span></mark>    </span>}
</span>}
</span></code></pre>
Since I was afraid of cloning, I opted to remove entries one-by-one from
a map; however, since these are reused by the object pool when dropped,
it might be better to clone them and let HashMap::clear</a> remove them all
at once. Also, cloning occurs only on insert which is less frequent than
modification:</p>
aggregate_map
</span>    </span>.</span>entry</span>(</span>*</span>city</span>)
</span>    </span>.</span>and_modify</span>(|</span>aggregate</span>| </span>aggregate</span>.</span>merge</span>(</span>&</span>other_aggregate</span>))
</span>    </span>/// Accept cloning
</span>    </span>.</span>or_insert_with</span>(|| </span>other_aggregate</span>.</span>clone</span>());
</span></mark></code></pre>
Tip: Reusing Collections</a></p>
This simple change drops the time to 26.5 seconds</strong></strong> which is somewhat
surprising.</p>
>></a>
No Strings (0m24s)</h2>
Reference: Commit</a> | main.rs</a></p>
Reviewing the Go</em> code, it does not actually use strings but fixed
length byte arrays specially as keys for the city name and then
displayed as a string via fmt.Printf</a>:</p>
type </span>CityName </span>[</span>64</span>]</span>byte
</span></mark>
</span>fmt</span>.</span>Printf</span>("</span>City: </span>%s</span>", </span>city</span>[:])
</span></code></pre>
For Rust</em>, this means replacing the string slice (&str</code>) for a byte
slice (&[u8]</code>) and converting it a string slice right before output.
This does make sense and saves numerous type reallocation call</a> from byte
to string</a> slice. Since this is not doable with lasso</a>, I have to write my
own byte slice equivalent optimized for this use case. Ideally, it only
does the following:</p>
/// No more str conversion
</span>let</span> raw_city </span>= &self</span>.</span>bytes</span>[</span>0</span>..</span>semicolon_index</span>];
</span></mark>
</span>/// Must take a byte slice now
</span>let</span> city_key </span>= </span>STRING_MAP</span>.</span>get_or_intern</span>(</span>raw_city</span>);
</span></mark>
</span>println!</span>(
</span>    </span>"</span>City: {:<20} | </span>{:.2}</span> | </span>{:.2}</span> | </span>{:.2}</span>",
</span>    </span>/// Conversion to str moved here
</span>    </span>unsafe </span>{ </span>std</span>::</span>str</span>::</span>from_utf8_unchecked</span>(</span>&</span>city_bytes</span>) },
</span></mark>    aggregate</span>.</span>min</span>(),
</span>    aggregate</span>.</span>mean</span>(),
</span>    aggregate</span>.</span>max</span>()
</span>);
</span></code></pre>
A quick implementation is to create a very large buffer and allocate a
segment for each unique hash of a byte slice. To make this work, I will
be adding a concurrent hash map dashmap</a> and with the familiar ahash</a> to
the fold which lasso</a> both uses internally:</p>
[</span>dependencies</span>]
</span>dashmap </span>= { </span>version </span>= "</span>5.5.3</span>", </span>features </span>= ["</span>inline</span>"] }
</span>ahash </span>= { </span>version </span>= "</span>0.8.11</span>", </span>features </span>= ["</span>compile-time-rng</span>"] }
</span></code></pre>
My idea looks like this:</p>
/// Use ahash for the hasher
</span>use </span>ahash</span>::</span>RandomState</span>;
</span>
</span>/// Use const generics to represent the length of a line (LEN) and the
</span>/// maximum number of lines (CAP). Perhaps unnecessary?
</span>struct </span>ByteStringPool</span><</span>const</span> LEN</span>: </span>usize</span>, </span>const</span> CAP</span>: </span>usize</span>> {
</span>    </span>buf</span>: </span>Mutex</span><</span>BytesMut</span>></span>,
</span>    </span>segments</span>: </span>DashMap</span><</span>u32</span>, Bytes, RandomState</span>></span>,
</span>}
</span>
</span>impl</span><</span>const</span> LEN</span>: </span>usize</span>, </span>const</span> CAP</span>: </span>usize</span>> </span>ByteStringPool</span><</span>LEN, CAP</span>> {
</span>    </span>fn </span>new</span>() -> </span>Self </span>{
</span>        </span>Self </span>{
</span>            </span>/// The shared global buffer
</span>            buf</span>: </span>Mutex</span>::</span>new</span>(</span>BytesMut</span>::</span>zeroed</span>(</span>LEN </span>* </span>CAP</span>)),
</span>            </span>/// Each hash is mapped to an immutable buffer region
</span>            segments</span>: </span>DashMap</span>::</span>with_capacity_and_hasher</span>(</span>CAP</span>, </span>RandomState</span>::</span>new</span>()),
</span>        </span>}
</span>    </span>}
</span>}
</span>
</span>/// New definition
</span>static </span>STRING_MAP</span>: </span>Lazy</span><</span>ByteStringPool</span><</span>CITY_LEN, NUM_CITIES</span>>> </span>= </span>Lazy</span>::</span>new</span>(|| </span>ByteStringPool</span>::</span>new</span>());
</span></mark></code></pre>
By allocating the required byte string memory needed, I am consciously
trading off memory for speed of around 1000000 bytes for every possible
city. Since it is a write-once data structure, it is easy to ignore any
locking errors as long as any data is written:</p>
fn </span>intern</span>(</span>&</span>self</span>, </span>bytes</span>: </span>&</span>[</span>u8</span>]</span>) -> </span>u32 </span>{
</span>    </span>/// Make sure city bytes follows the length restriction. LEN can be
</span>    </span>/// lowered even further later on that is why this check is here.
</span>    </span>let</span> bytes_len </span>=</span> bytes</span>.</span>len</span>();
</span>    </span>assert!</span>(</span>bytes</span>.</span>len</span>() </span><= </span>LEN</span>);
</span>
</span>    </span>/// Generate the hash or the key
</span>    </span>let</span> hash </span>= self</span>.</span>segments</span>.</span>hasher</span>().</span>hash_one</span>(</span>bytes</span>) </span>as u32</span>;
</span>
</span>    </span>/// Make sure capacity is correct
</span>    </span>assert!</span>(</span>self</span>.</span>segments</span>.</span>len</span>() </span><= </span>CAP</span>);
</span>
</span>    </span>/// Lock the entry if it does not exist
</span>    </span>if let Some</span>(</span>Entry</span>::</span>Vacant</span>(</span>entry</span>)) </span>= self</span>.</span>segments</span>.</span>try_entry</span>(</span>hash</span>) {
</span>        </span>if let Ok</span>(</span>mut</span> buf</span>) </span>= self</span>.</span>buf</span>.</span>lock</span>() {
</span>            </span>/// Check if the buffer can be split
</span>            </span>assert!</span>(</span>!</span>buf</span>.</span>is_empty</span>());
</span>
</span>            </span>/// Cut a segment in the front
</span>            </span>let mut</span> segment </span>=</span> buf</span>.</span>split_to</span>(</span>LEN</span>);
</span>
</span>            </span>/// Drop the mutex lock earlier for speed
</span>            drop</span>(</span>buf</span>);
</span>
</span>            </span>/// BytesMut.truncate needed for BytesMut.copy_from_slice to
</span>            </span>/// work properly
</span>            segment</span>.</span>truncate</span>(</span>bytes_len</span>);
</span>            segment</span>.</span>copy_from_slice</span>(</span>bytes</span>);
</span>
</span>            </span>/// Use BytesMut.freeze to mark it read-only
</span>            entry</span>.</span>insert</span>(</span>segment</span>.</span>freeze</span>());
</span>        </span>}
</span>    </span>}
</span>
</span>    hash
</span>}
</span></code></pre>
Implementing resolve</code> is trivial but just for completion:</p>
fn </span>resolve</span>(</span>&</span>self</span>, </span>key</span>: </span>&u32</span>) -></span> Bytes </span>{
</span>    </span>self</span>.</span>segments</span>.</span>get</span>(</span>key</span>).</span>expect</span>("</span>Invalid key</span>").</span>clone</span>()
</span>}
</span></code></pre>
Surprisingly this DIY byte string map pushes down the time enough to
beat the Go</em> implementation at 24 seconds</strong></strong> although the memory usage
did spike to 4 GB</strong></strong>. Side note, I do have an even faster
implementation</a> of this, but it uses pointers and atomic types</a> which I am
not yet well versed to explain.</p>
>></a>
Tokio Tuning (0m23.5s)</h2>
Reference: Commit</a> | main.rs</a></p>
Before wrapping up, time for one final tuning to make the time more
consistent. I wanted to keep the memory usage under control, but it
varies from each run the code has minimal allocations that leaves
tokio</code>'s runtime in question. Among all configurations, I found setting
the Builder::max_blocking_threads</a> to the number of cores (16) minus each
of the main spawn tasks and current thread (4) a sweet spot in being
fast and minimizing allocations:</p>
fn </span>main</span>() {
</span>    </span>let</span> cpus </span>= </span>num_cpus</span>::</span>get</span>();
</span>
</span>    </span>let</span> rt </span>= </span>Builder</span>::</span>new_multi_thread</span>()
</span>        </span>.</span>max_blocking_threads</span>(</span>cpus </span>- </span>4</span>)
</span>        </span>.</span>build</span>()
</span>        </span>.</span>expect</span>("</span>Runtime config error</span>");
</span>
</span>    rt</span>.</span>block_on</span>(</span>execute</span>());
</span>}
</span>
</span>/// Formerly main
</span>async </span>fn </span>execute</span>() {
</span></mark>}
</span></code></pre>
This finally brings the time around 23.5 seconds</strong></strong> and 0.6 GB</strong></strong> of
memory usage which finalizes my solution.</p>
>></a>
Conclusion</h2>
Given the same task, Go</em>'s allocates and reuses memory really well
without much effort where most of my best time savings comes from doing
the same in Rust</em>. While I do not necessarily want an encompassing
solution, a general automatic memory manager crate or similar that is
well documented and supported would really have closed the gap much
easily which I feel is a barren field. I only know of crossbeam::Epoch</a>
which is well-supported, but I do not want to deal with atomic ordering</a>
and nothing else seem to fit. Although the object_pool</a> and lasso</a> crates
really helped as well, they do not feel like standard and accepted
solutions. Dropping rayon</a> and writing a DIY byte string cache without
strong and obvious alternatives made me feel this hole in the crate
ecosystem. Perhaps a skill and knowledge failing on my part, I just
thought this general performance challenge would be easier to tackle
now.</p>
My introspective takeaway though is being wary in overselling memory
efficiency as a core value seeing the gap in effort and time.
Nonetheless, I did enjoy working on this challenge in my own terms and
being rewarded in lowering the time specially by reducing allocations
and being more precise was a dopamine hit. To reiterate, this is not
about Rust</em> vs Go</em>, this is satisfying my curiosity why a Rust</em>
implementation can underperform and how to resolve that with a good Go</em>
implementation as a really tight target to which I am satisfied.</p>
>></a>
Notable Attempts</h2>
Since I already achieved my goal, I am not optimizing further than this.
I do want to note some of my other attempts that did not make it
through:</p>

Unsafe Interner (0m23.3s)</a> - Using pointers and a faster yet uncertain
concurrent map, leapfrog</a>, is a small gain but not sure this is worth
it.</li>
No tokio (0m26.3s)</a> - Removing tokio</code> in favor of raw threads with
crossbeam</a> and threadpool</a> is a bust but surprisingly competitive.</li>
io_uring</a> - Also tried tokio_uring</a> but it does not feel ready yet and
requires setting up two runtimes which did not make it viable.</li>
</ul>
>></a>
Special Mention: mo8it/obrc (0m10s)</h2>
Reference: Repo</a></p>
Special mention to @m08bit@fosstodon.org</a> for an implementation that just
impresses me and want to add a special section on my findings about it.
It primarily uses memmap2</a> where I did not notice it can be treated as a
byte slice</a> which makes bytes</a> unnecessary as each chunk can be borrowed
directly instead of waiting for each read buffer. This also means that
the whole file data can be shared across multiple threads using
thread::scope</a> meaning tokio</a> is no longer needed. The general idea looks
like this:</p>
let</span> file </span>= </span>OpenOptions</span>::</span>new</span>().</span>read</span>(</span>true</span>).</span>open</span>(</span>path</span>).</span>expect</span>("</span>file does not exist</span>");
</span>let</span> mmap </span>= unsafe </span>{ </span>Mmap</span>::</span>map</span>(</span>&</span>file</span>) }.</span>expect</span>("</span>mmap failed</span>");
</span>/// mmap as a byte slice
</span>let</span> bytes </span>= &</span>mmap</span>;
</span></mark>
</span>/// Divide the work to each thread
</span>let</span> workers </span>= </span>num_cpus</span>::</span>get</span>();
</span>let</span> chunk_len </span>=</span> bytes</span>.</span>len </span>/</span> workers</span>;
</span>
</span>thread</span>::</span>scope</span>(|</span>s</span>| {
</span>    </span>for</span> i </span>in </span>0</span>..</span>workers </span>{
</span>        </span>/// Spawn a new worker for each chunk
</span>        </span>let</span> workder_handle </span>= </span>s</span>::</span>spawn</span>(|| {
</span>            </span>let</span> chunk </span>= &</span>workers</span>[</span>chunk_len </span>*</span> i</span>..</span>chunk_len </span>* </span>(</span>i </span>+ </span>1</span>)];
</span>            </span>let mut</span> record_iter </span>= </span>parse_chunk</span>(</span>chunk</span>);
</span>        </span>});
</span>    </span>}
</span>})
</span></code></pre>
Aside from having a more optimized measurement parser, here are some
points that I picked up:</p>

Replacing num_cpus</a> with thread::available_parallelism</a> is good enough
to reduce as an dependency.</li>
Switching from collections::HashMap</a> to hashbrown::HashMap</a> improved the
time by seconds.</li>
Inlining</a> is a tiny improvement but requires some testing.</li>
Maximizing release build</a> via Cargo.toml</code> did slightly improve the
time.</li>
f32::mul_add</a> or fused multiply-add is intriguing as the data seems to
be tailored for it by its 1 rounding error.</li>
Vec::sort_unstable</a> might be worth looking at instead of using a
BTreeMap</a> to sort.</li>
slice::get_unchecked</a> is tricky to get working</a> as borrowing a region
might be better although worth considering.</li>
</ul>
The code is so much shorter and more efficient that I am thankful to
have learned a bit more about performance.</p>
>></a>
Resources & References</h2>
Much gratitude for the resources and references I used for this article:</p>

gunnarmorling/1brc</a> - The challenge repository</li>
1brc_corlinp.go</a> - The Go</em> implementation</li>
FrancisMurillo/onebrc</a> - The Rust</em> source code repository</li>
Async: What is Blocking</a> - A caveat in mixing tokio</code> and rayon</code>.</li>
Rustonomicon</a> - The advanced Rust</em> book which I referenced for unsafe
pointers and atomics.</li>
The Rust Performance Book</a> - A compilation of performance tips for
Rust</em>.</li>
How to avoid bounds checks in Rust (without unsafe!)</a> - Some tips for
avoiding unsafe in bypassing bounds checks.</li>
Go's Memory Model</a> -  The official explanation of Go</em>'s memory model</li>
gunnarmorling/1brc in Rust</a> - Alternative implementations in Rust</em></li>
mo8it/obrc</a> - A remarkable yet simple implementation in Rust</em> at
0m10s</strong></strong> just using memmap2</a> and without a runtime like tokio</code>.</li>
</ul>



Wo Long: Fallen Dynasty (2023) - Short Thoughts
Unknown — Sat, 11 Mar 2023 00:00:00 +0000
>></a>
Wo Long: Fallen Dynasty (2023) - Short Thoughts</h2>

Finished the game mainly offline with Dual Swords</strong></strong> on Hard Mode</strong></strong>
and using little to no magic to focus on parrying.</p>
</blockquote>
My primary interest for Wo Long: Fallen Dynasty (2023)</a> is the focus on
parry gameplay as the demo sparked my interest. Sadly, I did not enjoy
the combat and gameplay barring other issues like the subpar story,
underdeveloped characters, superfluous loot and unappealing magic. I
will just tackle two problems that made my experience miserable: parry
timings</strong></strong> and morale system</strong></strong>.</p>
My primary issue is the parry timings</strong></strong> from the player and enemies
which can overall feel unrewarding. First, I believe the player's parry
window is slightly delayed, so the parry trigger must be slightly
earlier than the actual hit. In general, the longer the delay, the less
responsive it feels and more tricky to compute the offset. I do not mind
the slight delay in parry, but this becomes an issue with the next
problem.</p>
The enemy attack animations (startup, active and recovery) are mostly
too fast or slow to parry on reaction, meaning the game focuses on
memorization rather than reactions. Having varied attack timings are
good to keep parrying a dynamic risk and reward strategy; however, the
issue rather is the tells are unreactable regardless if an attack was
fast or delayed. I also do not mind these where blocking and dodging are
safe options, but enemies take longer to fatal attack</strong></strong> making those
options punishing.</p>
So with both fast attacks and delayed parries, it is less about reading
the animation and feeling the trigger timing but rather working
backwards from the hit to get the timing. A specific problem with this
is that startup attacks which signal the rest of the attacks are too
fast leading to haphazardly parrying any animation and missing the rest.
Another problem specially in combo chains is that parries must be
unfairly triggered even before the start of the next attack. I find the
attack animation misleading most of the time and prefer to parry while
blocking as an overall safe strategy to learn them. Although parrying
and combat do have more nuance, the overall implementation tends to
favor passive knowledge by trial and error more than actively reading
and reacting to opponents which is not for me.</p>
The other primary issue is the morale system</strong></strong> which I believe is to
motivate exploration and balance stages. The rough idea is that the
player and enemies have a stage level and the player can increase their
level by reaching certain checkpoints or hidden areas in a stage which
starts at 0 and ends at 20. Between two combatants, the higher the
difference between their morale, the more damage the winner does and the
lower the damage the loser does. There is some nuance in manipulating
morale, but conceptually think of it as a stage level separate from the
actual character level.</p>
Assuming the player character is on the recommended character level on a
new stage, the morale system serves to direct players to an intended
path and accumulate enough morale by the end to fight the 20 morale
boss. The problem is that collecting every morale points will always
total 20 morale that is simply fighting the boss with equal morale;
however without the collecting the hidden (around 6 or 7 morale) morale,
boss fights tend to be at a disadvantage. The stages are big and
non-trivial so finding those missing flags can feel like busywork and
demoralizing specially if the boss is difficult. This also hurts stage
replays whether for farming or grinding as the hidden morale points must
still be collected even though the stage is already beaten. So
exploration is required for a fair fight, not something that will give
an advantage say 22 or 23 player morale to make it worthwhile.</p>
As a stage balancing mechanic, it feels very artificial. Before any
fights, the player has to always check their morale against their
target's morale which encourages the player finding better odds
typically the intended route. While weaker enemies may be sometimes
stronger, difficult enemies have higher morale while guarding
checkpoints, so important fights tend to be at a disadvantage despite
the inherent skill difficulty. While this also deters taking alternative
routes, the stages rarely have alternative routes as they are optional
challenges rather than choices. I think morale is also holding back more
varied level design despite how big they are because of how damage and
exploration are tied together.</p>
As a risk/reward mechanic, it is very punishing on death where extra
morale is lost and reset back to the morale checkpoint known as
fortitude</strong></strong>. The primary way to acquire extra morale is to defeat
enemies and execute fatal attacks up to 25 morale. First, morale is not
usually farmed but rather gained throughout the stage, so dying would
put the player more at a damage disadvantage where they last left off.
Second, the player losses morale whenever they are hit with a fatal
strike commonly from bosses, making it harder to win with less damage.
Lastly for bosses, they reset their morale to 20 whenever it drops via
fatal strike, so doing well does not make boss fights easier. The player
always has to work more for a slight advantage but more to lose.</p>
In summary, the morale system in this game forces exploration and
disadvantages upon the player. I feel the morale system is a lost
opportunity to be more an active resource rather than a static number
possibly as items, buffs, debuffs, decisions and so on. The strange
thing is that the morale in Dynasty Warriors series</a> is a good global
system as it affects other active units and alters the flow of battle as
well as swings back and forth more dynamically with events unfolding in
the battlefield. Perhaps they just have the same name, but the potential
was clearly there.</p>
This game was clearly not for me despite my early enthusiasm for it, but
nonetheless this still offers some challenge and value on sale.</p>


Batman: Return To Arkham - Notes and Thoughts
Unknown — Thu, 08 Dec 2022 00:00:00 +0000
>></a>
Batman: Return To Arkham</h2>

Completed the main story, collected every Riddler trophy and beaten
every challenge (predator and combat) for Arkham Asylum</strong></strong> and Arkham
City</strong></strong> on hard mode</strong></strong>.</p>
</blockquote>
In honor of Kevin Conroy</a>'s passing, I decided to replay Batman: Arkham
Asylum</a> and Batman: Arkham City</a> among other things to pay tribute. I
abhor Batman: Arkham Knight</a>'s nonsensical and botched story as well as
the crappy car combat that I refuse to play it. For my playthroughs, I
played on the hardest difficulty, collected every viable secret or
collectible and completed the challenge arenas and maps to understand
the game's strengths and flaws. My attraction to both games are its
casual yet versatile combat and stealth gameplay although not for its
serviceable story; however, completing every optional content and
imprecise combat is where it hampers the experience. I do like and
recommend both games as a casual Batman</em> game but my overall experience
was negative when completing them as a completionist.</p>
For this article, I will assume the reader is familiar or played both
games so I will not be detailing the game mechanics mentioned instead
focusing on its design and implementation. I will be primarily writing
about the issues of both games and primarily using the first game as a
point of reference since the second game mostly improved on the first
game.</p>
>></a>
Padded Metroidvania</h2>
As a metroidvania</a>, backtracking for collectibles and secrets is an issue
in both games. Fundamentally, the incentive to retread old areas with
new tools for rewards is not very fulfilling for a few experience points
specially the many Riddler trophies</em>. Since several rewards are locked
by late gadgets, the player is punished for coming back earlier and that
some of them are simply using the new gadget without posing some
challenge. Most importantly, barely any changes in the old areas like
new enemy configurations or optional challenges for a stronger reason to
revisit. Another issue is the lack of exits where the player has to
travel all the way back after reaching the end or simply unnecessary
retreading. Aside from completion, the player has no incentive to
explore old areas and is wasting their time.</p>
An issue specifically as a completionist is that, the collectibles feel
like checklists when they are revealed instead of encouraging discovery.
Since the game is not really about exploration, this is understandable
cheap padding but conflicts with its metroidvania</em> design. I think just
not allowing players to come back to old areas until the end of the game
or simply not having those Riddler trophies and riddles</em> would be a
better design. The Easter eggs and references can just be those without
external rewards or pressures. I do like the interview tapes and lore
but I think they can be reworked as rewards for the optional content.</p>
These issues are primarily felt in the first game specially with the
lack of horizontal navigation even if the line launcher</em> is a subpar
solution and the lack of exits in long corridors. The workflow is
running to every room on foot (or wire), finding every collectible and
repeating this process on the remaining room and levels which is tiring
and boring. The second game fixes some of the prior issues with the
grapple boost</em> for horizontal mobility, better challenges, improved
side quests and Riddler informants</em> but suffers from the same issues.
Catwoman</em> amplifies this issue of backtracking where Batman</em> has to
discover and open the path first rather than independently which feels
worse. While I praise the new and better challenges, a huge amount of
repetitive and boring Riddler trophies</em> are still peppered throughout
the map. I feel the game wasted my time since I have to do this to
unlock the other more intriguing optional challenges.</p>
>></a>
Lenient Stealth</h2>
While I like how much gadget variety is offered, the game does not push
the players to act creatively or aggressively by offering too much
safety in vantage points and wall hacks. Although the game rewards
gadget variety, it is small and does not penalize camping. While the
second game offered many adaptive solutions such as destroying vantage
points, communication jammers, armored enemies requiring noisy
takedowns, mines, hostages and so on, but the underlying issue is not
really addressed. I do like the detective vision</em> as an accessibility
option to better see and find options such as breakables or
interactables, but do not like how it trivializes the stealth gameplay.
My suggestion is to simply design rooms without vantage points or cheap
exits, reduce experience points when using those options by half or so,
decrease the value of silent takedown while increasing the other
options, a time limit that may or may not increase with takedown, rework
the detective vision</em> to reveal enemies with a limited use noisy sonar
for a short period of time similar to the sonic batarang's explosion</em>
or any of these would just be direct incentive to be more proactive.</p>
As a revealing anecdote, I avoided using vantage points and detective
vision</em> in my playthrough to heighten my experience; however, the final
Riddler challenge</em> I had left involves executing an inverted takedown
and since predator rooms do not respawn, I had to start New Game Plus</em>
to complete it which is almost funny but mostly sad.</p>
>></a>
Disempowering Combat</h2>
This is my greatest grievance with the Arkham</em> game series. The combat
system works well at a casual level but executing it well specially with
the second game's hardest combat arena, The Joker's Carnival</em>, shows
how poorly it feels at a high level. Most of the time whenever the
player commits an error, the question should be how can they correct
their mistake, but here it is how did the game caused it. The following
issues show a loss of player accountability discouraging mastery and
skill over both games.</p>
The worst issue is the imprecise attack targeting where the wrong target
is attacked which happens most of the time specially in between combos.
After completing the hardest combat challenge, I still do not know how
targets are chosen and only adapt based on the targets in that
direction. As long as enemy is between the intended target, the player
always has to anticipate that the next target is the one in between no
matter how precise the analog controls are. This is also why disarming
shields and stun batons or special takedowns of armored enemies is
always a priority to avoid hitting them by accident. An attack target
indicator would help this issue like what the third game did but my
experience with it has been underwhelming. I am not sure if there is
even a perfect implementation of this targeting but it still feels
disruptive when the wrong target is hit.</p>
Outside of freeflow focus</em>, random normal attack timings is the next
issue. While I appreciate the number of animations, the inability to
select the fastest attack animation often leads to a counterattack from
an enemy jab which seems faster than any of player attacks. The worse
feeling is losing to an attacking an enemy before they choose their
attack because a high flying long committal move was chose which does
not teach anything but only being unlucky. Attacking far away enemies is
the only lesson here but since the attack targeting is also not
accurate, it feels the game is making the error rather than the player.
Even the fixed animation of the cape can still be beaten by a jab, so
the best option is to actually attack an enemy after an dodge which
feels bad.</p>
I do not know the term for this design where enemy slides in place to
attack the player or similarly to Batman</em> attacking far away enemies
but I will call it here as attack magnetism</strong></strong>. When a player sees an
incoming attack, the player needs to determine whether to continue their
current action if they knew it would not hit or acknowledge it with a
counter, dodge or quick action. For example, the aggressive option is to
attack a far away opponent to put the player out of range while
maintaining the offense. The issue is that the attack magnetism</em> is
unpredictable or ambiguous whether it would still connect with the new
position specially with the pipe, push kick or double thug attacks which
have a deceptively long ranges. Precise countering is also not a
solution as its range is also unpredictable leading to a lost combo. My
suggestion is to show counter range like a dotted circle in the floor,
remove the attack indicator if the attack will not hit similar or
something to that effect. Between the unpredictability of counters and
attack magnetism</em>, dodging is the most consistent way to reset their
attacks which feels underwhelming.</p>
Another issue is the auto-adjusting camera where after attacking an
enemy, the camera adjust slightly to face towards it; however, it
disorients the next movement possibly causing a failed attack. It also
is too close to the player where long reaching attacks such as pipes or
swords behind the camera cannot be reacted to. Also when attacking
enemies outside the camera, a cape stun is always the safest option if
it hits an armored, shield or stun baton enemy. This issue would be easy
to fix with options to disable auto-adjustment, increasing the field of
view and possibly making it more top-down to feel more in control of the
field. As an accessibility point, the auto-adjusting camera can induce
motion sickness specially attacking far away enemies quickly which I
occasionally in my numerous attempts.</p>
Although this may be possibly on the player's side, the input system
seems to be broken at times. I felt this the most when dodging at attack
but the dodge does not execute. Other times executing a button
combination does not work as intended such as countering a weapon attack
(shield or knife) when a disarm was intended. It could be my fault for
being slow, jittery or incorrect with timings or buffering, but more
often than not I notice these moments as frequent enough to warrant a
mention.</p>
The two factors at the highest level are speed and score by maintaining
the combo counter. With all these issues and objective in mind, the
safest and fastest way to play is to attack once (or twice if in
freeflow focus</em>) and then quickly dodge (or preferably jump over an
enemy or attacker) to reset attack animations. Counters, aside from its
unpredictability, are also discouraged since it is the slowest option
unless it is with multiple enemies although they can still be used as
breathing space and at close range. Blade counters are also risky and
dangerous specially as titans or one-armed thugs can interrupt them.
Achieving the freeflow focus</em> is also mandatory when dealing with
tougher or faster enemies such as safely stunning titans or damaging
one-armed thugs. The overall strategy feels like a defensive mitigation
of randomness and unpredictability in the combat system instead of being
in control with all the gadgets and controls.</p>
>></a>
Concluding Thoughts</h2>
Regardless of my negative experience, I do recommend the first and
second game, but do not expect completion be pleasant nor mastery be a
rewarding experience.</p>
RIP Kevin Conroy</strong></strong></p>


Belle (2021) - Notes and Thoughts
Unknown — Fri, 02 Sep 2022 00:00:00 +0000
>></a>
Belle (2021)</h2>
As a fan of Summer Wars (2009)</a> abstract virtual world, I had to watch
Belle (2021)</a> to see it again. Inspired from Disney's Beauty and the
Beast (1991)</a>, the film follows a high school girl (Suzu</em>) regaining the
ability to sing and finding her place in as an idol (Bell</em> or Belle</em>)
in virtual space (U</em>) then reaching out to a violent avatar (Beast</em>)
similar to its inspiration; however, I did not emotionally nor musically
resonate with it because of the confusing artificiality of it all. For
casual viewing, the film is still okay with its great visuals, but it
fails to leave a strong mark like its predecessor to be strongly
recommended.</p>
Starting with my issue with its Disney inspiration. If a film closely
uses or references another story, I prefer if it builds upon it or
creates its own original work from it rather than just referencing key
moments or ideas. For its Disney film inspiration, it is hitting those
iconic references such as the beast, castle, rose, ballroom dance,
fearmongering, mob attack and so on. While following templates or
structures is not necessarily bad, the film does not go beyond enough to
avoid reminding me of a better film.</p>
What else it took is the songs. Instead of songs being a dramatic plot
device or character expression is now diagetic or literal singing which
removes the plot efficiency and creativity. For my favorite Disney film,
Mulan (1998)</a> songs quickly establish the setting, show character values
or inner conflicts, move the story and mark key moments. While the
avatar singing has character and plot uses, it feels narratively slower
and less interesting to interpret and watch. While the film does not
have as many songs, I wish they were more notable aside from their
emotional qualities.</p>
Moving past my grief about its reference to the film itself. The film
feels incidental with its main conflict who the real user of the Beast</em>
is. While a virtual police force somewhat exists to do exactly that, the
protagonist and her friend decide to do it themselves out of petty
revenge or curiosity for crashing one of their concerts. As with the
inspiration, the Beast</em> is eventually portrayed as sympathetic or
abused despite the actual crimes, but the protagonist loses their
motivation with the hunt leaving the story hanging. I would like to see
the protagonist actually speak out to support the Beast</em> which would be
a strong gesture and character moment rather than being forced to gain
trust when the story demands them to. Without spoiling the last act, the
real Beast</em> needs help and the protagonist decides to do it herself
instead of calling the appropriate government services which feels
dangerous and questionable despite being a character development moment.
Although I have more issues with some minor plot points, the overall
plot seems just to accommodate its key emotional moments rather than a
coherent and meaningful one.</p>
The next point is how uninteresting the characters. I barely know their
values and interests specially the Beast</em> who is the focus of the later
half of the film. The character dialogue as well is not intriguing
specially theatrical from the police leader. I think the film can
replace the romance plots and unveiling plot device with more meaningful
character interactions and dialogue subtext. What saddens me is that I
do not feel attached to any character in the film even the underwhelming
protagonist.</p>
Concerning my main point of interest. The film places a special emphasis
on unveiling or revealing the identity of a virtual world user. In this
virtual world, the avatar is directly generated from the user's body
signature or biometrics so that each one is unique and corresponds to a
unique and identifiable user which also implies no alternative accounts
are possible. So what is the justice served by unveiling a user? While
users already reveal their identity when communicating outside the
virtual world, a user's identity is private information and disclosing
it forcibly or without consent is an attack on their privacy and safety.
I do not know if the film realizes the main form of virtual justice is
doxing</a>. What bothers me is that banning a user would be a more practical
and ethical solution when dealing with bad actors in online communities.</p>
The police force in this virtual world is known as the Justices</em> but
are not formally governed by the world's administrators. In fact, the
world may not have any administrators or ruling body but rather a
lawless and independent world where justice may be only an emerging
social need. If the Justices</em> are a community group and administrators
are non-existent, how did the Justices</em> get the ability to unveil other
users that are administrative capabilities? Ignoring the fact that
unveiling has commercial sponsors, would there not be more support or
utility groups such as builders, merchants, cosmetic designers and so
on? The focus on the police force raises so many questions about the
world and makes the world feel more flat without other representative
groups. Perhaps if they made the martial arts users as bounty hunters
instead would make for a more colorful task force without raising
issues.</p>
My last issue is the virtual world itself. Ignoring the possible privacy
violations of harnessing user biometric data, the user having only one
identity is a major ethical and economic concern. The virtual world was
supposed to give users another identity that should be ideally equal or
fair across users; however, the protagonist gets an avatar that is
endowed with beauty and voice which is apparently a rare trait since
nobody can contest her angelic music. In a way, this new world
replicates the same inequalities of class and privilege by merely giving
beyond cosmetic changes. When Bell</em> became popular, the best friend
acknowledges the profits from singing but sends them to charity instead
to avoid changing the status quo; however, the point stands that the
protagonist is a rich and privileged celebrity. Some skills and traits
naturally carryover like charisma, intellect or combat prowess but
giving different advantages because of genetic or environmental
differences is a balance issue in gaming terms. Considering its Disney
reference, the Disney protagonist was beautiful and well-read but an
outcast. Maybe Bell</em> could still sing but be a minor idol instead to
avoid this inequality issue.</p>
Regarding the lack of expression and freedom with this avatar
generation. What happens if the person does not like their appearance?
It could be argued that it removes physical beauty standards by leaving
it out the user but not having at least some options can be
psychologically or emotionally detrimental specially the body conscious.
Not allowing multiple accounts also restricts a user from exploring
different identities or protect their anonymity. I understand that a
unique identity prevents spam or abuse; however, this virtual world has
no safeguards in the first place, so what is the point of this loss of
privacy and creativity? I wish the film just allow users to randomly
generate their avatars as many times and some ability to check a user's
public profile to sweep this issue away.</p>
While I do have other issues with the virtual world, I raised some
possible solutions while maintaining the technological and abstract
fantasy of it; however, I really think all this is in service of the
high emotional moment where the protagonist is unveiled and sings which
is the crux of my problems. I think the film is going for a
technological optimism specially how it grants the protagonist the
ability to sing again and perhaps strength to the abused, but how
everything is constructed seems to go against that which is a shame. The
film can still be enjoyable, but I would just rewatch Summer Wars</em>
again for a better story, characters and visual experience.</p>


Teenage Mutant Ninja Turtles: Shredder's Revenge - Notes and Thoughts
Unknown — Sun, 10 Jul 2022 00:00:00 +0000
>></a>
Teenage Mutant Ninja Turtles: Shredder's Revenge</h2>

Finished story mode on XBox at Hard difficulty with April</p>
</blockquote>
As a fan of Streets of Rage 4</a>, Teenage Mutant Ninja Turtles: Shredder's
Revenge</a> is another good brawler with a casual multiplayer focus. I do
not have nostalgia for the franchise, so my thoughts are of it as a solo
brawler. I can excuse a lack of features and quality of life such as a
better lobby system, key bindings/macros, advanced tutorials, but the
core combat is a bit messy and unpolished in a few ways.</p>
The first choice players make is selecting their character. Although
they differ in stats (power, range and speed) and move effects, they all
practically control the same that lowers character complexity and
uniqueness. While this design choice is meant for a casual audience, the
problem is that these characters still have their differences in their
specials and combos which can deceive and mislead players of its
possible depth. If the player community is ranking characters, the game
is doing itself a disservice. While I prefer mechanical differences,
adding character specific tutorials, videos or indicators would have
easily solved this.</p>
For the meat of the game, the combat system is quite fluid with
animation canceling and the anti-air move. While I do want advanced
concepts like hitting enemies of the ground or wall, enemies have such a
low health pool that I understand its exclusion. Talking about its
issues, I have three major issues which players can get hit unfairly.
The first is that normal moves cannot cancel into special moves
instantly which is vulnerable window. The second is that grabs are
implemented poorly and risky without any invincibility frames, crowd
control or strong damage specially where grabs can be accidental. The
third is the attack magnetism or homing nature of the running and dodge
attack which can hit the wrong enemy, ruin positioning or flow. Other
than that, the combat is okay although more mechanics and options are
welcome.</p>
Considering enemy design next, I do not like how almost all enemies can
anti-air which makes aerial attacks unsafe or risky. Perhaps only
certain enemies can anti-air which would signal against anti-air than
being guesswork. Some enemies can block where grabs should have
effective; however, grabs have their own issue and the other options
such as special moves are more viable. Certain enemies can also cancel
grabs unless they have been damaged which has no clear indicator. A
special grab action can remove an enemy instantly which I think the game
is preventing for these special enemies. Perhaps removing that optional
grab action can allow grabs to be more consistent. Some enemies have
power armor before attacking which means failing to interrupt or stagger
enemies. Overall, fighting enemies do not feel predictable and getting
hit randomly is a symptom of that.</p>
Compounding those issues, bosses feel unfair and clunky. In particular,
bosses have an attack and vulnerable phase where players have to evade
attacks before actually doing damage or risk taking damage. An issue
with this is once they take damage they go into an armored counterattack
which limits the possible damage dealt and risk taking damage making
fights longer or more passive. The last boss is specially guilty of this
with such a long attack phase that the only thing the player can do is
dodge and taunt. The hitboxes of some of their attacks are also unclear
or deceptive which is a problem with 2D brawlers in general but more
annoying with their higher damage. In general, bosses have so many issues
and may feel frustrating to fight rather than a genuine challenge.</p>
I will not be talking about the level design, but I do want to address a
strange yet game breaking issue. Executing special attacks require a
resource which can be partially charged on attacking but fully lost on
taking a hit. This mechanic encourages defensive play while being
aggressive, although I disagree with this since special moves are used
as panic defensive options. Surprisingly, completing a taunt gives the
player one special move resource without ever attacking which is absurd.
In story mode, the maximum special move resource is 3 meaning a player
can refill room clearing special moves before each encounter and the
only thing discouraging this behavior is the forgiving hit counter. This
is also unbalanced in multiplayer as players can taunt while waiting for
the other players to finish their parts. In arcade mode, this is
somewhat fixed with 1 special move limit, but it does limit special
moves as a resource. Perhaps limit 1 resource gained through taunting is
acceptable or removing it altogether is better, but this considerably
affects boss damage and health which leads into a bigger design and
balance issue.</p>
While I cannot speak about the quality of online play since I
disconnected frequently, the overall game as a solo experience does not
feel worth investing my time in with all its faults. It is an okay game
to reinvigorate the franchise, but it needs more features and fixes to
make this a good product to truly recommend.</p>


Fights in Tight Spaces - Notes and Thoughts
Unknown — Mon, 30 May 2022 00:00:00 +0000
>></a>
Fights in Tight Spaces</h2>

Finished the game on XBox with Slasher, Aggressive, Grappler and
Counter-Striker decks at Purist (hardest) difficulty.</p>
</blockquote>
I love and thirst for Metal Gear Acid</a>'s combat system, so I was excited
when I found Fights in Tight Spaces</a> that combines its solo character
grid-based combat with the roguelike deck-building</a> genre to be an
addictive twist. The unique mechanic of this game aside from movement as
a resource is the combo counter which goes up for each attack but
decreases for each tile movement. This counter can be a card requirement
which usually provides good crowd control effects and/or high damage
scaling but can be detrimental or dead cards if it is not attained. I
like this mechanic as it encourages aggressive plays while discouraging
movement if the player uses combo cards. In terms of gameplay, I do not
have much to say since it follows since it follows the genre formula of
picking a starter deck and finishing five levels within two to three
hours. Although it is not as deep or complex as its contemporaries, I
still recommend this game for hungry fans of Metal Gear Acid</em>.</p>
Aside from the deck drafting mode, the game offers six starting decks
each with its own strengths and weakness:</p>
Balanced</strong></strong>
: Good movement and damage</p>
Counter-Striker</strong></strong>
: High defense and counterattacks but lacks
mobility</p>
Aggressive</strong></strong>
: High damage attacks but requires maintaining combo</p>
Slasher</strong></strong>
: Good mobility and bleed stacking but lacks crowd
control and damage</p>
Grappler</strong></strong>
: Good throws and crowd control but lacks both
mobility and defense</p>
Trickster</strong></strong>
: Gimmick attacks and damage but less consistent
because of the randomness</p>
I love the game, but my only big criticism is the lack of endgame
content usually expected by its genre. Once you finish the game, the
only incentive left is to finish it with the other starter decks. Aside
from starting at a harder level, the game could use more difficulty
modifiers, randomized bosses, starting characters, items and card
effects specially a viable ranged build, but this might be
understandably beyond the scope, budget and balance of the game.</p>
For some of its minor issues, optional combat objectives such as pushing
enemies of the ledge or finishing within a certain number of terms need
to be more lenient since some starter decks cannot viably fulfill them
such as pushing with the Counter-Striker</em> or sometimes just randomness.
Although optional, these objectives commonly offer money to upgrade or
remove cards and sometimes more importantly losing health, combo or turn
upgrades which can be devastating in the long run. Until the player
recognizes their value, players are not encouraged to plan their routes
for their preferred upgrades which leads into another issue.</p>
The overall level (node and branch) design is very flat where any
decision fork converges at most within three nodes which lowers its
strategic depth in favor or simplicity. While this may be a valid design
choice, it is one of the most visible areas of improvement. If it did
have more variety, I would not just beeline for the nearest gym, hunt
for every turn upgrades and occasionally rescue an informant or
ambassador for heals. In fact, the medical center seems to be useless
for healing assuming the player never take risks with long-term injuries
which I am not sure is good design. If the game wanted to lean on this
simple and flat level design, it could add some unique narrative
immersion with simple randomly generated story beats why the branches
diverge and converge like something in This War of Mine</a>.</p>
As with Metal Gear Acid</em>, having enough movement cards in a deck so
that players draw at least one on average is important specially where
not moving could mean taking unavoidable damage which feels bad. With
all starting decks, I found the game does not provide enough movement
cards for a more balanced experience regardless of some advancing
attacks or movement starting items. Some decks start with two turn cost
cards which is too early while starting with three turns although combo
cards here work. While it is good the starting decks represent different
perspectives, they can be tuned better for newer players.</p>
I have other smaller issues like the replay, lack of animation skipping
and card balancing, but overall the game clicked with me and perhaps a
sequel could introduce more content, depth and difficulty. I suggest
also playing on Purist</em> difficulty so that planning turns is an
important process which I think should be the default difficulty and
make the more lenient ones instead accessibility options.</p>


Tunic - Notes and Thoughts
Unknown — Tue, 03 May 2022 00:00:00 +0000
>></a>
Tunic</h2>

Finished on Xbox with the pacifist ending</p>
</blockquote>
Tunic</a> is a very deceptive game where its art style and aesthetics
indicate a Zelda</a> inspired game yet for the most part it is a basic
Souls-like</a> with stamina, roll, death and other genre mechanics. What
makes this game special is its bold commitment to hide or obscure
information such as its mechanics, tutorial, story and in particular the
in-game manual which appeals to nostalgia when game cartridges used to
come with a manual but a foreign language. In a way, understanding and
beating the game with the bits and pieces becomes its own meta-game. To
summarize my thoughts and feelings though, I can break down the game by
its two end goals.</p>
If the player's goal is to free the princess</em> and beat the last boss,
the game is an action game. As a Souls-like</em> though, it is rather
simple and basic without much depth even with the talismans and items.
Talking about the combat mechanics, I find it strange that the player
receives more damage when their stamina is low or critical. The only
thing that consumes stamina is rolling and guarding to prevent
overreliance, but the stamina resource becomes a non-issue with enough
stat upgrades. Once the dodge is upgraded, blocking becomes less viable
or appealing, making dodge the only consumer since it is much faster and
safer. I think items and attacks should also require stamina, but the
game would be significantly slower than it has to just to make stamina
meaningful. So removing stamina altogether, adding a slight cooldown to
dodge and perhaps a seperate guard resource would be a better
experience.</p>
Next, the parry mechanic feels awkward with a delayed timing since the
animation is so long with a long startup, short active window and a long
recovery. It vastly differs from the quick reaction-based parry or
deflection; however, this style of parry requires time or
experimentation to determine the actual delayed parry timing. The reward
for parrying is an enemy stagger, but this can be outright slower and
riskier while the magic items can be more efficient and faster specially
the grab or fireball magic item. Another issue is that some enemies
attacks are faster than the startup that must be parried before any
attack animation like the tall spear enemy's thrust which feel awkward.
I think just removing the startup but keeping the long recovery would be
okay.</p>
The consumable items are also not viable enough or encourage use because
of its issues. The first is being they are too expensive relative to
enemy drops where it may be more efficient to save for stat upgrades
instead. As chest rewards, they are underwhelming or their quantities
should be increased. I wish they are also rechargable or common drops
from enemies with a limited capacity to encourage use. Another issue is
that the enemies and overall combat difficulty and enemy variety do not
necessitate use. The one good thing I can say is the extra bonus items
when an item is repeatedly use which is a kind of item level or
progression, but again the prior issues make it only a minor
consolation.</p>
Lastly, the special items and talismans do not change the gameplay that
much aside from handling enemies better. If items provide debuffs and
magic is restored via attacking, they could offer more utility, depth
and be better balanced. One imbalanced item is the pull magic item with
its low cost since and stuns enemies momentarily for at least a two hit
combo and then pulling again to stack the damage and combine that with
the magic flask talisman for more uses to easily handle enemies and some
bosses. One talisman idea is that ranged items do not consume magic but
stamina or allow magic to be refillable through attacks which can make
ranged combat more viable in general. Also consider item specific
talisman modifiers can be intriguing such as giving the sword a
projectile wave, the stick a magic transformation effect, the flame wand
a bigger fireball and effect but slower, less stamina on guard while
reflecting melee damage and so much more to make gameplay more spicy.</p>
The biggest tell for me is that the combat is not that polished is the
combat gauntlet by the midgame. The gauntlet is fighting six enemy waves
of enemies in the order the player chooses with only the default stats
while redoing the whole if the player dies in between. Items are still
accessible although not viable with the low stamina and magic resource,
but the defensive talisman is quite valuable here for the survivability.
The only difficult part of this gauntlet is the fighting two of the
first boss then patience in kiting the remaining waves which feels more
attrition than skill. Completing the gauntlet upgraded the dodge to be
faster but afterwards the combat peaks in enemy difficulty or mechanics
making it seem my investment was meaningless. In a way, this peak also
indicates a transition to another genre.</p>
After hours in the game, information can be found about an alternative
possible pacifist ending (manual page 46</a>) where the game becomes a
puzzle game instead. Without spoiling exact details, the player must
collect a certain collectible by praying</strong></strong> (manual page 44</a>) through a
button sequence where each one has their own prayer</strong></strong>. Acquiring all
of it, unlocks information about the game's language that allows the
final puzzle to be deciphered and solved which gives the pacifist
ending. Acquring this ending instead soured my enjoyment of the game.</p>
Before talking about the collectible, one of the things that make this
game unique is its game manual where it emulates the feel of a booklet
by flipping throught its pages. The manual is the player's main source
of information for the lore, mechanics and map, but it is obscured by an
in-game language. Despite that, most of the details can be inferred
through the images and some English words in the manual. The other catch
is that the manual is incomplete and pages of it can be found in-game.
While mostly not hard to find, each page can help find some of those
collectible and most importantly how to solve them.</p>
Despite being called secrets, I will be calling them puzzles since they
are all required specially and judging them by how they are found and
solved. While it can be a joy, they have major issues in both aspects.
Regarding how they are found, it is important to discuss some problems
with the camera. Unlike a top-down camera, isometric cameras need to be
designed not to obscure paths from foreground elements such as walls or
tall objects and also reveal some area when traveling through them as to
not move blindly. However, this game hides so many paths or entrances by
this principle like its many shortcuts and not clearly hinting
traversible paths that requires tedious checking of edges. While this
might be clever, it becomes a significant annoyance when finding the
collectibles. The worst example is a collectible hidden behind the only
bomb breakable wall that is also obscured by a foreground element which
just feel cheap. If these paths were signposted, the discovery of these
collectibles would feel more active than accidental.</p>
The greater issue in finding them is how hidden they are in plain sight.
I do not mind reexploring old areas with new abilities, but the
backtracking feels aimless without clear indicators or point of interest
which leads to constant backtracking. Some are the numerical sequences
and ground lines since they feel so randomly placed and harder to do
with the camera. Some are noticing something doing something for a
period of time which feels more of a waste of time. In general, when I
learned something was a puzzle, I was more annoyed that it was trying to
be clever.</p>
On solving them, the overall puzzle design here is more monotonous and
boring. Some are blocked by foreground elements, some are recording or
tracing a pattern, some are just noticing its there. Overall, I felt the
game tried to stretch one gimmick mechanic too much. Praying</strong></strong> is
incrediblly error prone and unclear without a feedback system. Specially
the final puzzle or some secrets, the inputs can be so long that any
input errors cannot be detected and the player must restart the inputs
which can be draining. I understand the inspiration, but it made the
puzzle experience worse being the only method. Perhaps adding more
puzzle mechanics like incorporating items and maybe requiring only half
to decipher the manual can help with the monotony and pacing.</p>
Not to say the game has no good puzzles, I did appreciate the one
reflected in the water since it required thinking about orientation. The
final one is quite nice although it was somewhat tedious with the manual
interface and missing data</strong></strong>. The debatable one is scattered across
the game that requires taking a screenshot and combining it possibly
with an external image editor which can be a barrier. At least the worst
ones are optional secrets such a music puzzle or standing in a specific
area for a minute if you can call that a puzzle.</p>
Neither is this game a good action or puzzle game but the effort and
intent to replicate the nostalgia is respectable although flawed.
Looking back, the obfuscation of its information and mechanic feel more
gimmicky now and the effort to decipher the language may not be worth
the effort. Although I realize that some people enjoy and even thrive on
this level of obscurity, I wish the game clued me in earlier instead
near the endgame for me to understand this game was not for me.</p>


River City Girls - Notes and Thoughts
Unknown — Mon, 07 Mar 2022 00:00:00 +0000
>></a>
River City Girls</h2>

Played on XBox with Misako on Hard difficulty</p>
</blockquote>
River City Girls</a> is a modern pixel art 2D side-scrolling beat'em up</a>
where two high-school girls rescue their boyfriends. I personally do not
care about the history of the game being a spin-off in the Kunio-kun</a>
franchise, but how it good it is as a fighting game. My point of
reference is Streets of Rage 4</a>, so I find this game sadly feels inferior
with several issues specially the story and combat.</p>
While the story is serviceable for an action game, it is not satisfying
enough where a series of false leads culminate into a joke ending.
Although it is an excuse to visit areas and fight bosses, the quest and
dialogue is not clever or insightful enough to make it worthwhile and
overall detrimental to its world  whereas a simpler story might be better
like trying to fight through a sale or rare item. I am not sure how
impactful the references and callbacks are to the fans, but I felt every
character was missing some key character details or relationships which
made me confused as a newcomer. Despite not minding the story that much,
it is still rather bad.</p>
Before talking about the core combat, I did not like the inclusion of a
level progression with stats and unlockable moves primarily because the
starting expereince feels so limited and restrictive which does not
allow or encourage for creative combos or expression. Addressing the
stats first, it is not explained nor does it feel impactful. In fact,
the strangest unexplained stat mechanic is that eating food increases
stats but what stat will be increased is unknown until purchased. While
I believe it is to incentivize visiting shops and purchasing items, I
feel it is not valuable enough with money being hard to acquire and
healing items commonly restoring a small amount given their cost. Also,
equippable items even from the bosses commonly only offer stat bonuses
that only give minor bonuses instead of gameplay depth or variety. My
suggestion is to remove the stat system altogether or rebalance and
rework its effects to make it more meaningful.</p>
While unlocking moves primarily through money or a currency is a solid
idea, it is hard here because they are expensive without extensive
grinding and most importantly around 25% is lost on death which
frequently happens without good crowd control moves. In particualr,
fighting enemies to gain money for moves is not viable as the health
loss cannot be offset by the healing items. Rather, defeating bosses is
the main method of gaining money that must be immediately spent for
expensive moves before dying. This death mechanic punishes hoarding that
contradicts saving money for better moves to prevent death which is
paradoxical. If moves are unlocked through levels or some combined skill
point system like completing side quests, the inclusion of level
progression would make more sense. Although I do not believe this is a
good consequence for dying, this could have overall worked if healing
and money were more viable since dying is occasionally cheaper than
healing.</p>
Regardless how moves can be unlocked, the worst issue is that the game's
difficulty and approach does not push the player to its advanced
techniques such as juggling or launchers which is typically where the
fun is. The fundamental issue is a lack of tutorials, a training room
and the ability to test a move makes it harder to learn the combat
safely and creatively experiment specially with lost money on death.
Also, the game does not incentivize good performances like a hit or
damage multiplier where even small money bonuses could be attractive
enough. After beating the game, it does not offer reasons to replay the
game such as a ranking or new challenges despite offering two new
characters. I do believe the game has depth, but the game is not
designed to take advantage of that depth nor does it demand the player
to be good which is a shame for a fighting game.</p>
Moving on the actual combat, it has many minor issues making the game
overall feel frustrating than it should be. First off, the environmental
background feels unpolished or untested as the foreground elements
obscure characters specially enemies that can lead to cheap hits where
adding opacity would easily work. Picking up a weapon, enemy or
attacking is tied to one contextual button that causes undesired inputs
and confusion where different buttons would fix this. Enemies have too
good vertical tracking attacks as well that makes those deceptive
despite being able to dodge vertically or block. Since special attacks
do not have invulnerability frames, enemy attacks occasionally interrupt
special moves that makes them less safe despite being crowd control
moves. In fact, players can be juggled or hit by multiple enemy leading
to unfair damage where some knockdown invulnerability would help. On the
other side since characters do not have moves to hit enemies on the
ground, players have to needlessly wait for them to get up that
contributes to making combat less engaging. Although players can ground
stomp and grab enemies of the ground, it is not the same as creating a
combo flow by relaunching them in the air. Overall, these many issues
make the core combat feel occasionally unpolished and unfair.</p>
I do have other issues such as the music, art style and boss design, but
these said issues prevented me from fully enjoying and investing more
time. Still as a casual cooperative game, this is an okay product but
not for me.</p>


Stickless Half Circle Inputs
Unknown — Sat, 26 Feb 2022 00:00:00 +0000
>></a>
Half Circle Inputs</h2>

Tested on KOF XV (Xbox)</strong></strong> with a Snackbox Micro</strong></strong>.</p>
</blockquote>
I wanted to get into fighting games, but I was deterred because of the
motion inputs required. Primarily, rotating a stick, gamepad or
joystick, does not feel clear or exact because it is a digital input.
More importantly, performing those inputs fast and precisely are more
tiring and injury-prone without proper practice and posture. Thankfully,
stickless controllers were made like the Snackbox Micro</a> that replaces
the stick with four directional buttons (left, down, right then up)
which is better laid out for fighting games than the standard
directional pad. This new approach satisfies my conditions as doing
quarter circle</a> and dragon punch</a> inputs feels more comfortable, precise
and tactile with button sequences. Playing KoF XV</a>, single grounded half
circles were no issue; however, doing double half circle inputs while in
the air became an issue like with Athena's Shining Crystal Bit</a>.</p>


A Stickless Controller Layout</p>
</blockquote>
Executing a half circle with a stickless controller can be intuitively
divided into two quarter circles. Using the notation (L)eft, (D)own and
(R)ight to represent the three main directional buttons of a stickless
controller, a forward quarter circle input can be done by pressing D,
then pressing R while holding D and finally lifting D while keeping R
pressed or more simply D, D+R, R. This motion can be described as
rolling two fingers, so doing a half circle can be though of as rolling
three fingers or L, L+D, D, D+R, R.</p>


Doing a half circle input by rolling the fingers</p>
</blockquote>
The issue here is that the third input, D, is occassionaly dropped if
done too fast as controllers as inputs are read by frame</a>. To be precise
after the L+D input, if L is lifted while R is pressed at the same time
or within the space of one frame, the game reads D+R instead of D since
L needs to be lifted for at least one frame for D to be read. It is a
strange issue where being faster causes more issues.</p>


The Down input can be occassionally dropped when done too fast</p>
</blockquote>
After some experimentation, instead of rolling the fingers, pressing all
the fingers first and then lifting in the same order is more consistent
or L, L+D, L+D+R, D+R, R where the input L+D+R resolves to D because of
neutral SOCD cleaning</a> (L+R = neutral).</p>


Press the fingers then lifting in the same order</p>
</blockquote>
Thanks also to half circle input leniency</a> where L+D and D+R is optional,
it does not matter if L or D is lifted first or at the same time because
the optional D+R is read if L is released first, neutral if D from SOCD
cleaning, and the required R if both that makes this method safer to
execute.</p>




It does not matter what order the fingers are lifted</p>
</blockquote>
One caveat is that this does not work with vertical half circles if
vertical SOCD is not neutral which up wins if both up and down are
pressed with my controller. Another caveat is that this method might
apply more pressure or stress on the last finger since it becomes the
anchor to lift the other fingers where practicing minimal effort is also
needed. For stickless controllers, I am aware of other half circle
shortcuts</a> (L, D, R) but I do find these lift-press or tap motions
introduces a timing bottleneck and feels less consistent to execute
fast. Nonetheless, I hope this new method helps.</p>


Fatal Frame 5 Remastered - Quick Thoughts
Unknown — Wed, 03 Nov 2021 00:00:00 +0000
>></a>
Fatal Frame 5</h2>
Fatal Frame: Maiden of the Black Water Remastered</a> or Fatal Frame 5</em>
is a Japanese ghost survival horror game where three characters are
looking for a missing person in a haunted mountain shrine over several
nights. As it follows series tradition of an exorcising camera and
catastrophic ritual, what makes this unique is its primary theme of
suicide; however, this new entry fails to establish strong writing,
characters and performances to fully explore or utilize this theme.
Moreover, this game is much more streamlined that it loses its
challenge, design and tension. It is an okay game but a failure as a
new entry.</p>
>></a>
Quick Thoughts</h2>
This game is similar to the third game where it has three playable
characters with their own shooting styles investigating a flooded
mountain over multiple nights and chapters ideally crossing over each
other's path and culminating into a satisfying ending. Like the forth
game, it also utilizes the motion controls which makes camera control
a bit faster and drops the fixed camera perspective for a modern third
person camera which is not necessarily good for this genre. What makes
the games in this series special is that the horror, tone and
atmosphere are solidly executed; here however, the game shows a lack
of polish in character, map and quest design and too many player
options bring the horror down.</p>
My biggest issue with the game is the significant backtracking or
retreading where the player returns and navigates to the same areas
too much without variation or contextualization. Since the game is
structured into multiple chapters over several nights, the story and
pacing suffers to give the characters frustrating reasons to come back
specially worse for fillers such as the lore and home surveillance
chapters. The characters also suffer as the game demands them to be
frustratingly less intelligent to piece together information and more
importantly the consequences of their actions. Beside the story and
characters, retreading the same areas repeatedly become monotonous,
breaks immersion and shows a lack of careful map and quest design. In
comparison with the third game's large central location, the player
must investigate any new facts then appear within it at an
unpredictable location which feels more natural and better paced.</p>
Another big issue is the third person camera and analog movement is so
bad that a fixed camera perspective and tank controls would work
better. First the camera field of view is so close that it hard to see
what is in front and induces nausea where a simple accessibility
option would really help. While the camera is typical of most games,
the movement system is relative to the camera which causes so much
unwanted turns which becomes so inconsistent at times if not moving
towards the camera's direction specially when picking up items or in
combat. While a bit more subjective, the camera combat favors multiple
targets that feels more like a fast arcade shooter than slow and
deliberate which requires patience and enhances the atmosphere. In
general, movement was not an issue in the old games yet here the
immersion is occasionally ruined.</p>
Another massive issue is the lack of challenge or excessive hand
holding. Puzzles which gave variety and tone are gone or nonexistent.
Items can now be bought before starting a chapter that greatly
diminishes resource management which provides mechanical tension or
item conservation. Because of its arcade chapter structure, chapters
must be designed to be finished independently removing the pressure
for resource management and the need for better quest design. The fact
that players can ask for help where to go anytime with ghost traces
seems to cover up the poor map design as well. Another example of that
is when acquiring keys, the game shows on the map where to use it
which just shows a lack of faith in the player. Too many healing items
are also given to the player each chapter making even the combat less
tense. Regarding camera combat, the auto-tracking is too good since
getting counterattacks becomes easier. Lastly, the enemy indicators
when raising the camera just cover up the uninspired fight areas. With
so many concessions for the player, it feels the game favored making
it more accessible than retaining its memorable aspects.</p>
I usually play Japanese games in Japanese audio for the intended
experience; however, the remastered version only has English audio
that accentuates poorer dialogue not in their intended language which
is the case here. In general, the character's performances feel
stilted that tilts more to B grade horror. For example, characters
barely react to key information when read or the constant repetition
of known facts makes them feel stupid or slow. Characterization is
also sparse and poor that barely exploring relationships or giving
them colorful insights. Not to forget the sexualized female designs,
alternative costumes, wetness mechanic and all the crouching positions
pushes them as objects than characters even more. Although the old
games may have that same quality, this is feels more like a lack of
polish even if its is a modern remaster.</p>
The story is not noteworthy enough to discuss and does not have a
strong resolution specially with all the padding. Also, one of the
three protagonist feels like sketchy fan service and does not
contribute much to the overarching mystery which could be removed and
provide a more paced story. The theme of suicide does not land well
since the characters are flat and relationships underdeveloped
specially the assistant of the male protagonist who just keeps getting
into trouble. Even if the old game's protagonists were as quiet and
lacked character, it did not hold back the story and atmosphere.</p>
Overall, this game looks like a game of its series but lacks the bite
and charm of its predecessors. This game has so many issues I wonder
why any of first three was not remastered since it had a solid design
and experience. If another game comes out of this series, I hope this
will not be an example or the direction it follows, although with the
fourth game is not giving me much hope.</p>


No More Heroes 3 - Quick Thoughts
Unknown — Mon, 25 Oct 2021 00:00:00 +0000
>></a>
No More Heroes 3</h2>
Like its prior games, No More Heroes 3</a> is the story of a top-ranked
assassin saving Earth from an invading alien gang through an
interplanetary assassin ranking competition. The joy of the game and
the whole series is its unpredictable yet grounded characters while
the combat system is competent enough to support it. Although this
game has a bigger scope and several improvements, it feels messy and
less meaningful to play. This is still personally a good game with
interesting boss fights and clever twists, but its negative aspects
severely pull it down from being a recommendable game for non-fans.</p>
>></a>
Quick Thoughts</h2>
Let me begin with the things I like which is mostly a return to the
style of the first game. First of is the combat system which is more
fluid and now has potentially modifiable skills via death chips. To
keep up with those changes, the enemies and bosses are more varied and
challenging. The new mini-games are quirky and enjoyable specially the
robot shooter battles. Lastly, the story of the alien leader as he
loses his friends and control is a good narrative choice. Overall, the
game still has its charm and style which I appreciate; however, its
issues severely impact those good qualities that make the game a chore
to play.</p>
Since this game follows its ranking format of fighting ten ranked, the
worst issue is the required entry fights for each rank. Early on, they
are challenging and introduce new enemies which make each rank fight a
real accomplishment; however in the late game, they stop being
distinct and merely repeating formations with stronger variants which
becomes drawn-out fights. This is compounded with every fight not
offering any details such as enemy composition and its unpredictable
or random placement in the map makes it hard to know whether a fight
is difficult or worth taking. As a completionist I took every fight,
and they are really tiring and tedious overtime which feels more
padding than player choice. Perhaps the game had fewer entry fights
and possibly spaced the enemy varieties closer to the end, the flow
and difficulty will feel faster and better.</p>
While the combat system feels better to play than the first, the issue
is its progression that is a straightforward point system. The first
half of the issue is acquiring those skill points which is acquired
through fights. The problem is that it can be easily broken by redoing
boss fights at an easier difficulty and skill points are rewarded
based on performance rather than difficulty. This is a non-issue in
the first game as skill or stat progression was gated off by
mini-games or exploration and would feel much more rewarding and
interesting.</p>
The other half is the combat balance and customization via gauntlet
chips. The biggest issue here is once the player acquires the rapid
thrust skill, enemies and even bosses can be trivially controlled and
becomes a fast and safe method which becomes boring over time. Without
other viable combat skills, the many entry fights feel longer and more
procedural than fluid or expressive. This also applies to the limited
gauntlet skills that is ironic as creating chips could have imbued
different skills rather than simply percentage attribute bonuses.
Attribute percentage bonuses is the least interesting way to provide
progression and balance, so just providing more varied skills can give
the depth and creativity the combat deserves.</p>
The reintroduction of an open world is much appreciated, but the
implementation feels hollow or empty. While the space is larger,
navigating through it specially with the bike is often janky with
unclear collisions or terrain causing motorcycle crashes making it
feel unpolished. The new islands/regions are mostly wide empty spaces
as if to hide the entry fights which feels uninteresting alongside the
mini-map making it trivial to find. What is worse is that since the
player is teleported to a different fight area, the space itself is
less relevant to its purpose or meaning. Comparing with the first
game, it does this so much better and simpler. Even if this game's
world might serve an aesthetic or tonal function, it is severely
underutilized and perhaps better more restrictive for its purpose.</p>
Lastly, the story or continuity itself is confusing even for fans.
While a game for fans is nice and a certain level of logical disbelief
is needed, the tone feels less grounded and worth taking seriously. I
do like the abstract aliens and design, the introductory television
segments, game references and various boss gimmicks; however, time
travel, resurrections, cameos, giant robots in the story begin to lose
me. In particular, the alien leader does not change or inspire change
by his presence except for his childhood friend. This is a waste as
his eventual defeat does not inspire reflection or meaning. This is
all subjective, but the prior games were able to tackle different
themes much better and made me appreciate the game even if it was not
perfect. If another sequel exists, I hope it has a better intriguing
story.</p>
Regardless of other minor issues, the game is still brimming with
style and a good entry in the series. It is unavoidable to compare
this with the prior games specially the first one as it still holds up
today. After playing several games of the studio/director, padding and
pacing seems to be an persistent issue I hope that will be addressed
for future games.</p>


Outer Wilds: Echoes of the Eye - Quick Thoughts
Unknown — Sun, 17 Oct 2021 00:00:00 +0000
>></a>
Outer Wilds: Echoes of the Eye</h2>
Outer Wilds: Echoes of the Eye PS4</a> is an additional story content or DLC</a>
that introduces a new hidden planet or region which focuses more on
creative thinking and mechanical exploration. Aside from the Anglerfish</a>,
this also introduces a direct horror element of pursuing enemies in
darkness which adds more distress and fear aside from the innate time
pressure. Above all, this expansion provides a very difficult puzzle
mystery with new mechanics and ideas to ponder which gives a challenge
rivaling or surpassing the original. The DLC is definitely worth the
challenge and a good reason to revisit this stellar game.</p>
>></a>
Notes and Thoughts</h2>
WARNING: Early game spoilers ahead</strong></strong></p>
Finding the new region is quite a surprise as stealth technology has
never been introduced before. The new region is visually interesting as
it rotates on a wheel and the opposite side can be seen anytime almost
the inverse of Brittle Hollow</a>. Riding the river with a raft is also nice
as a new means of travel and how centrifugal force</a> maintains the
current. As this is also a new race, I appreciate that the translator
does not work and the way to communicate is with images through the
slides. The new inhabitants are an intriguing combination of owl and
deer, but it would be nice if they could fly as well. Overall familiar
with the earthy swamp aesthetic and nocturnal animals but something
hidden beneath that exterior.</p>
The overall goal in this new region is to unlock the vault. Since the
challenge is higher, exploring areas thoroughly specially for the slide
reels and reviewing the facts is more important here as the region is
small. Solutions are designed cleverly and fair as discovering them
offers the same thrill and joy. Sadly, the ending of the new region may
seem abrupt or anti-climactic considering the effort which is similar to
the Quantum Moon</a> optional quest. Aside from seeing the ending again, the
true reward is a variation on the ending, and it is definitely worth
redoing.</p>
My major personal gripe with this new region is the global darkness that
makes it hard on the eyes; however, it works in this game as light plays
a central theme and role for this new region. The darkness also
heightens the environment from the familiar to the treacherous as the
pursuers hunt. How they reach out from the darkness to extinguish the
player's light will stay with me. I feel the darkness is an integral
part of the experience, but an accessibility option would be nice.</p>
As solving the mystery requires more backtracking and setup, some
players might feel this is more tedious like the rafts which is
understandable. As a DLC, being more complex and challenging is its goal
although at the cost of some tedium. What it brings to the game is still
worth exploring and tackling.</p>
WARNING: End game spoilers ahead</strong></strong></p>
Understanding the history and motivation of the precursor race</a> is quite
sad. They had high hopes for the Eye</a> but felt resentment and shame on
what it will bring and what was sacrificed. To prevent others from
making the same mistake, they blocked the Eye</em>'s signal and retreated
into their simulated home world until it dies. Rather than harming them,
I believe they failed to realize that the Eye</em> was calling out to
everyone and that refusal might have prevented others from restarting
the galaxy. If not for the sacrifice and excessive imprisonment of one
of them</a>, the Nomai</em> might have never come and the Hearthian</a> to finish
the story.</p>
With the precursor race specializing in light and being primarily
observers, concealment or ignorance is a major motif in its story and
aesthetics:</p>

The ship itself obscures light to hide its presence to keep their slumber undisturbed.</li>
Much of the information acquired is through watching memories through
their numerous projectors and slide reels which may be redacted or
burned.</li>
The number of hidden passages, false walls, invisible bridges and
locks is meant to hide things in plain sight.</li>
Finding the glitches requires extinguishing the light and finding it
in the darkest area like the green fire.</li>
The hidden world is a reflection of their vivid home world rather than
what they did to it.</li>
Blocking the Eye</em>'s signal in a way denies its existence similarly to
the Prisoner</em>.</li>
The lantern's themselves allow the player to focus or conceal their
light. To shine the way or to hide it.</li>
</ul>


Simple Page View Tracking with Cloudflare KV
Unknown — Wed, 13 Oct 2021 00:00:00 +0000
>></a>
Page View Tracking</h2>
Since this blog is personal and small, I did not believe page view
tracking was needed nor was it worth the cost. While free alternatives
exist, I do not want extra privacy analytics such as IP address,
browser or region to be collected which may identify the user, so I
did not consider it until recently. With Cloudflare's free KV storage
offering</a>, I could create a simple worker</a> that increments a page view
counter by page title when viewed. This would require adding a site
script to invoke the worker; however, this can be circumvented by
script blockers which is fine since I only need a general view. In
terms of Cloudflare privacy policy</a>, the only caveat with this approach
is that the endpoints may still log IP for operational monitoring, so
using a VPN service</a> or Tor</a> is highly encouraged. Privacy concerns
aside, free workers are only offered 1000 daily writes, therefore this
approach only works if the site traffic has less than 1000 daily
views. Nonetheless, this approach works for my use case and is
significantly better than maintaining and securing a database.</p>
(Source code for worker and CLI tool can be found here</a> and here</a>
respectively.)</p>
>></a>
Cloudflare Worker</h2>
After creating a Cloudflare account</a>, playing around with the dashboard
and checking out the ecosystem, I decided to build and deploy the
worker using the wrangler CLI</a> with the Rust WASM template</a> specifically
worker-rs KV example</a> makes it straightforward. Getting up and started
with is:</p>
# Install wrangler via cargo
</span>$ cargo install wrangler
</span>
</span># Copy the template
</span>$ wrangler generate page-tracker-worker https://github.com/cloudflare/rustwasm-worker-template
</span>$ cd page-tracker-worker
</span>
</span># Fetch credentials via browser login
</span>$ wrangler login
</span>
</span># Create the KV storage and store the last line config entry
</span>$ KV_ENTRY=$(wrangler kv:namespace create PAGE_COUNTER --verbose | tail -n 1 )
</span>{ binding = "PAGE_COUNTER", id ="mykvid" }
</span>
</span># Copy the value to wrangler.toml under the kv_namespaces document property
</span># Using sed to insert it into the empty 6th line easily
</span>$ sed -i "6s/^/kv_namespaces = [ $KV_ENTRY ]" wrangler.toml
</span>
</span># To check if the storage is created
</span>$ wrangler kv:namespace list
</span></code></pre>
The wrangler.toml</code> should look like this:</p>
# wrangler.toml
</span>name </span>= "</span>page-tracker-worker</span>"
</span>type </span>= "</span>javascript</span>"
</span>workers_dev </span>= </span>true
</span>compatibility_date </span>= "</span>2021-10-10</span>"
</span>compatibility_flags </span>= [ "</span>formdata_parser_supports_files</span>" ]
</span>kv_namespaces </span>= [ { </span>binding </span>= "</span>PAGE_COUNTER</span>", </span>id </span>= "</span>mykvid</span>" } ] </span># NEW LINE HERE
</span>
</span># ... rest of template omitted ...
</span></code></pre>
This is all the configuration needed, the next step is to implement
the worker. The worker essentially takes an HTTP POST request, gets
the request path and increments the path's counter. With some minor
CORS handling and cues from worker-rs docs</a>, the main worker code is
easily done:</p>
// src/main.rs
</span>lazy_static! </span>{
</span>    </span>// The site name for CORS
</span>    </span>static ref </span>SITE_URI</span>: </span>&'static str = </span>"</span>https://fnlog.dev</span>";
</span>    </span>// Make sure this matches with the one in `wrangler.toml`
</span>    </span>static ref </span>KV_BINDING</span>: </span>&'static str = </span>"</span>PAGE_COUNTER</span>";
</span>}
</span>
</span>// Template main entry point
</span>#[</span>event</span>(</span>fetch</span>,</span> respond_with_errors</span>)]
</span>pub</span> async </span>fn </span>main</span>(</span>req</span>:</span> Request, </span>env</span>:</span> Env</span>) -> </span>Result</span><</span>Response</span>> {
</span>    </span>// Better logging on error
</span>    </span>utils</span>::</span>set_panic_hook</span>();
</span>
</span>    </span>// OPTIONS CORS handler
</span>    </span>if </span>matches!</span>(</span>req</span>.</span>method</span>(), </span>Method</span>::</span>Options</span>) {
</span>        </span>let mut</span> cors_headers </span>= </span>Headers</span>::</span>new</span>();
</span>
</span>        cors_headers</span>.</span>set</span>("</span>Access-Control-Allow-Origin</span>", </span>&</span>SITE_URI</span>)</span>?</span>;
</span>        cors_headers</span>.</span>set</span>("</span>Access-Control-Allow-Methods</span>", "</span>POST,OPTIONS</span>")</span>?</span>;
</span>        cors_headers</span>.</span>set</span>("</span>Access-Control-Max-Age</span>", "</span>86400</span>")</span>?</span>;
</span>
</span>        </span>// Allow required headers from the request
</span>        cors_headers</span>.</span>set</span>(
</span>            </span>"</span>Access-Control-Allow-Headers</span>",
</span>            </span>&</span>req</span>.</span>headers</span>()
</span>                </span>.</span>get</span>("</span>Access-Control-Request-Headers</span>")
</span>                </span>.</span>ok</span>()
</span>                </span>.</span>unwrap</span>()
</span>                </span>.</span>unwrap_or</span>("".</span>to_owned</span>()),
</span>        </span>)</span>?</span>;
</span>
</span>        </span>return Ok</span>(</span>Response</span>::</span>ok</span>("")</span>?</span>.</span>with_headers</span>(</span>cors_headers</span>));
</span>    </span>}
</span>
</span>    </span>// POST handler
</span>    </span>// This is POST to make sure it is accessed via network call
</span>    </span>if </span>matches!</span>(</span>req</span>.</span>method</span>(), </span>Method</span>::</span>Post</span>) {
</span>        </span>// Get request path
</span>        </span>let</span> path </span>=</span> req</span>.</span>path</span>();
</span>
</span>        </span>// Initialize KV storage by name
</span>        </span>let</span> kv </span>=</span> env</span>.</span>kv</span>(</span>&</span>KV_BINDING</span>)</span>?</span>;
</span>
</span>        </span>// Get string counter of the path
</span>        </span>let</span> counter </span>=</span> kv
</span>            </span>.</span>get</span>(</span>&</span>path</span>)
</span>            </span>.</span>await</span>?
</span>            </span>.</span>map</span>(|</span>val</span>| </span>val</span>.</span>as_string</span>())
</span>            </span>.</span>and_then</span>(|</span>txt</span>| </span>txt</span>.</span>parse</span>::<</span>usize</span>>().</span>ok</span>())
</span>            </span>.</span>unwrap_or</span>(</span>0</span>);
</span>
</span>        </span>// Increment counter
</span>        </span>let</span> new_counter </span>= </span>(</span>counter </span>+ </span>1</span>).</span>to_string</span>();
</span>
</span>        </span>// Store new value
</span>        </span>// Minor gatcha is that .execute() is needed to commit the change
</span>        kv</span>.</span>put</span>(</span>&</span>path</span>, </span>&</span>new_counter</span>)</span>?</span>.</span>execute</span>().</span>await</span>?</span>;
</span>
</span>        </span>// Set CORS headers
</span>        </span>let mut</span> cors_headers </span>= </span>Headers</span>::</span>new</span>();
</span>        cors_headers</span>.</span>set</span>("</span>Access-Control-Allow-Origin</span>", </span>&</span>SITE_URI</span>)</span>?</span>;
</span>
</span>        </span>// Return success with the path
</span>        </span>return Ok</span>(</span>Response</span>::</span>ok</span>(</span>path</span>)</span>?</span>.</span>with_headers</span>(</span>cors_headers</span>));
</span>    </span>}
</span>
</span>    </span>// Fallthrough response
</span>    </span>Response</span>::</span>error</span>("</span>Bad Request</span>", </span>400</span>)
</span>}
</span></code></pre>
Testing the worker locally:</p>
# Run the dev worker at 127.0.0.1:8787
</span>$ wrangler dev
</span>
</span># Test the worker with curl
</span># Path: /
</span>$ curl -XPOST http://127.0.0.1:8787/   # Value: 1
</span>$ curl -XPOST http://127.0.0.1:8787/   # Value: 2
</span># Path: /dev/blog_name
</span>$ curl -XPOST http://127.0.0.1:8787/dev/blog_name
</span>
</span># Check the KV keys
</span>$ wrangler kv:key list --binding PAGE_COUNTER
</span>[{"name": "/"},{"name": "/dev/blog_name"}]
</span>
</span># Fetch KV value by key
</span>$ wrangler kv:key get '/' --binding PAGE_COUNTER
</span>2
</span>$ wrangler kv:key get '/dev/blog_name' --binding PAGE_COUNTER
</span>1
</span></code></pre>
With it working locally, deploying it as simple as wrangler publish</code>:</p>
# Deploy worker to Cloudflare
</span>$ wrangler publish
</span> Successfully published your script to
</span> https://page-tracker-worker.fnlog-dev.workers.dev
</span>
</span># Test if it is deployed
</span>$ curl -XPOST https://page-tracker-worker.fnlog-dev.workers.dev/test/path
</span>/test/path
</span></code></pre>
The final step is to invoke this endpoint when a user views or loads a
page. As described in the beginning, it can look like this:</p>
function </span>trackPage</span>(</span>success</span>) {
</span>    </span>// URL of the worker
</span>    </span>const </span>workerUri </span>= </span>"</span>https://page-tracker-worker.fnlog-dev.workers.dev</span>";
</span>
</span>    </span>// Get the path of the current URL
</span>    </span>const </span>{ </span>pathname </span>} </span>= new </span>URL</span>(window</span>.</span>location)</span>;
</span>
</span>    </span>// Append the path to the worker URI for the complete URL
</span>    </span>let </span>workerUrl </span>= </span>workerUri </span>+ </span>pathname
</span>
</span>    </span>// Send the network request
</span>    </span>var </span>xhr </span>= </span>window</span>.</span>XMLHttpRequest </span>? new </span>XMLHttpRequest</span>() </span>: new </span>ActiveXObject</span>(</span>'</span>Microsoft.XMLHTTP</span>'</span>)</span>;
</span>    xhr</span>.</span>open(</span>'</span>POST</span>', </span>workerUrl)</span>;
</span>
</span>    xhr</span>.</span>onreadystatechange </span>= function</span>() {
</span>        </span>if </span>(xhr</span>.</span>readyState</span>></span>3 </span>&& </span>xhr</span>.</span>status</span>==</span>200</span>) success(xhr</span>.</span>responseText)</span>;
</span>    </span>};
</span>    xhr</span>.</span>setRequestHeader(</span>'</span>X-Requested-With</span>', '</span>XMLHttpRequest</span>'</span>)</span>;
</span>    xhr</span>.</span>send(</span>null</span>)</span>;
</span>}
</span>
</span>trackPage(</span>function</span>(</span>data</span>){ </span>/* NOOP */ </span>}</span>)</span>;
</span></code></pre>
If CORS is an issue, do review the worker response headers; otherwise,
the site now has DIY</a> analytics. Visit the details for the KV dashboard</a>
views after some time.</p>
>></a>
Analytics</h2>
Although the analytics work, it is tedious to login and check the KV
dashboard</a> for the page views and with no means to export the data.
Using the Cloudflare API</a> as a workaround, a scheduled daily script
could fetch the entries and write it to a CSV</a> file. In particular, the
list keys</a> and read key value</a> endpoints is what is needed. After
getting my API Token</a>, it should be easy to write a CLI script for this
with structopt</a>, reqwest</a> and tokio</a>.</p>
To access the API, the API JWT</a>(PT_JWT</code>), account ID(PT_ACCOUNT_ID</code>)
and KV ID(PT_KV_ID</code>) are needed that can be environment variables. To
get the account and KV ID, visit the KV dashboard details and the URL
should match this pattern:
https://dash.cloudflare.com/$PT_ACCOUNT_ID/workers/kv/namespaces/$PT_KV_ID</code>.
With structopt env args</a>, the CLI can be started like so:</p>
// src/main.rs
</span>use </span>anyhow</span>::</span>Result</span>;
</span>use </span>std</span>::</span>path</span>::</span>PathBuf</span>;
</span>use </span>structopt</span>::</span>StructOpt</span>;
</span>
</span>#[</span>derive</span>(</span>Debug</span>,</span> StructOpt</span>)]
</span>#[</span>structopt</span>()]
</span>enum </span>Opt </span>{
</span>    </span>// Download subcommand
</span>    </span>// Subcommands allow more commands to be added easily
</span>    Download </span>{
</span>        </span>#[</span>structopt</span>(</span>long</span>,</span> env </span>= </span>"</span>PT_JWT</span>",</span> hide_env_values </span>=</span> true</span>)]
</span>        jwt</span>: </span>String</span>,
</span>        </span>#[</span>structopt</span>(</span>long</span>,</span> env </span>= </span>"</span>PT_ACCOUNT_ID</span>",</span> hide_env_values </span>=</span> true</span>)]
</span>        account_id</span>: </span>String</span>,
</span>        </span>#[</span>structopt</span>(</span>long</span>,</span> env </span>= </span>"</span>PT_KV_ID</span>",</span> hide_env_values </span>=</span> true</span>)]
</span>        kv_id</span>: </span>String</span>,
</span>
</span>        </span>// Optional arg to specify the download folder
</span>       </span>#[</span>structopt</span>(</span>long</span>,</span> default_value </span>= </span>"</span>.</span>")]
</span>        output_dir</span>:</span> PathBuf</span>,
</span>    </span>},
</span>}
</span>
</span>// Setup async main with tokio and quick error handling with anyhow
</span>#[</span>tokio</span>::</span>main</span>]
</span>async </span>fn </span>main</span>() -> </span>Result</span><()</span>, </span>Box</span><</span>dyn </span>std</span>::</span>error</span>::</span>Error</span>>> {
</span>    </span>match </span>Opt</span>::</span>from_args</span>() {
</span>        </span>Opt</span>::</span>Download </span>{</span> jwt</span>,</span> account_id</span>,</span> kv_id</span>,</span> output_dir </span>} </span>=> </span>{
</span>            </span>todo!</span>();
</span>        </span>}
</span>    </span>}
</span>
</span>    </span>Ok</span>(())
</span>}
</span></code></pre>
With the credentials available, the API endpoints can be async
functions with some help from serde</a>:</p>
// src/main.rs
</span>use </span>reqwest</span>::</span>Client</span>;
</span>use </span>percent_encoding</span>::{</span>utf8_percent_encode</span>, </span>NON_ALPHANUMERIC</span>};
</span>use </span>serde</span>::</span>Deserialize</span>;
</span>
</span>// Define credential as 3-tuple string for type convenience
</span>type </span>Credential </span>= </span>(</span>String</span>, </span>String</span>, </span>String</span>);
</span>
</span>// List Keys sample output:
</span>// {
</span>//   "result": [{ "name": "/about/" }],
</span>//   "success": true,
</span>//   "errors": [],
</span>//   "messages": [],
</span>//   "result_info": {
</span>//     "count": 1,
</span>//     "cursor": ""
</span>//   }
</span>// }
</span>
</span>#[</span>derive</span>(</span>Debug</span>,</span> Deserialize</span>)]
</span>struct </span>ListKeysPayload </span>{
</span>    </span>result</span>: </span>Vec</span><</span>ListKey</span>></span>,
</span>}
</span>
</span>#[</span>derive</span>(</span>Debug</span>,</span> Deserialize</span>)]
</span>struct </span>ListKey </span>{
</span>    </span>name</span>:</span> String,
</span>}
</span>
</span>// Given a reqwest client and credentials, fetch all KV keys.
</span>async </span>fn </span>list_keys</span>(</span>client</span>:</span> Client, </span>cred</span>: </span>&</span>Credential</span>) -> </span>Result</span><</span>Vec</span><</span>String</span>>> {
</span>    </span>let </span>(</span>jwt</span>,</span> account_id</span>,</span> kv_id</span>) </span>=</span> cred</span>;
</span>    </span>let</span> url </span>= </span>format!</span>(
</span>        </span>"</span>https://api.cloudflare.com/client/v4/accounts/</span>{}</span>/storage/kv/namespaces/</span>{}</span>/keys</span>",
</span>        account_id</span>,</span> kv_id
</span>    </span>);
</span>
</span>    </span>// JWT is an authorization bearer header
</span>    </span>let</span> resp </span>=</span> client</span>.</span>get</span>(</span>url</span>).</span>bearer_auth</span>(</span>jwt</span>).</span>send</span>().</span>await</span>?</span>;
</span>    </span>let</span> payload </span>=</span> resp</span>.</span>json</span>::<</span>ListKeysPayload</span>>().</span>await</span>?</span>;
</span>
</span>    </span>// The only thing needed is the collected `.result[].name`
</span>    </span>Ok</span>(</span>payload
</span>        </span>.</span>result
</span>        </span>.</span>into_iter</span>()
</span>        </span>.</span>map</span>(|</span>key</span>| </span>key</span>.</span>name</span>)
</span>        </span>.</span>collect</span>::<</span>Vec</span><</span>_</span>>>())
</span>}
</span>
</span>// Given a reqwest client, credentials and KV key, fetch the key's value
</span>async </span>fn </span>get_key_value</span>(</span>client</span>:</span> Client, </span>cred</span>: </span>&</span>Credential, </span>key</span>: </span>&str</span>) -> </span>Result</span><</span>usize</span>> {
</span>    </span>let </span>(</span>jwt</span>,</span> account_id</span>,</span> kv_id</span>) </span>=</span> cred</span>;
</span>
</span>    </span>let</span> url </span>= </span>format!</span>(
</span>        </span>"</span>https://api.cloudflare.com/client/v4/accounts/</span>{}</span>/storage/kv/namespaces/</span>{}</span>/values/</span>{}</span>",
</span>        account_id</span>,
</span>        kv_id</span>,
</span>        </span>// The key has to be percent encoded
</span>        utf8_percent_encode</span>(</span>key</span>, </span>NON_ALPHANUMERIC</span>)
</span>    </span>);
</span>
</span>    </span>let</span> resp </span>=</span> client</span>.</span>get</span>(</span>url</span>).</span>bearer_auth</span>(</span>jwt</span>).</span>send</span>().</span>await</span>?</span>;
</span>    </span>// No need for custom deserializer since the value is returned directly
</span>    </span>let</span> value </span>=</span> resp</span>.</span>json</span>::<</span>usize</span>>().</span>await</span>?</span>;
</span>
</span>    </span>Ok</span>(</span>value</span>)
</span>}
</span></code></pre>
The main fetching code can be implemented using these functions and
some futures-rs combinators</a>:</p>
// src/main.rs
</span>// In the download command
</span>// Opt::Download { jwt, account_id, kv_id, output_dir } => {
</span>use </span>futures</span>::{
</span>    </span>future</span>::</span>FutureExt</span>,
</span>    </span>stream</span>::{</span>FuturesUnordered</span>,</span> StreamExt</span>},
</span>};
</span>
</span>// Initialize client and credentials
</span>let</span> client </span>= </span>Client</span>::</span>new</span>();
</span>let</span> credentials </span>= </span>(</span>jwt</span>,</span> account_id</span>,</span> kv_id</span>);
</span>
</span>// Fetch all keys
</span>let</span> keys </span>= </span>list_keys</span>(</span>client</span>.</span>clone</span>(), </span>&</span>credentials</span>).</span>await</span>?</span>;
</span>
</span>// For each key, fetch its value
</span>// To run them concurrently, `FuturesUnordered` is used
</span>let</span> view_futures </span>= </span>FuturesUnordered</span>::</span>new</span>();
</span>
</span>for</span> key </span>in</span> keys </span>{
</span>    view_futures</span>.</span>push</span>({
</span>        </span>let</span> client </span>=</span> client</span>.</span>clone</span>();
</span>        </span>let</span> credentials </span>=</span> credentials</span>.</span>clone</span>();
</span>
</span>        async </span>move </span>{
</span>            </span>let</span> view</span>: </span>usize = </span>get_key_value</span>(</span>client</span>, </span>&</span>credentials</span>, </span>&</span>key</span>).</span>await</span>?</span>;
</span>
</span>            </span>Ok</span>((</span>key</span>,</span> view</span>))
</span>        </span>}
</span>    </span>});
</span>}
</span>
</span>// Run all the futures and collect them into a list
</span>let mut</span> data </span>=</span> view_futures</span>.</span>collect</span>::<</span>Vec</span><</span>Result</span><</span>_</span>>>>().</span>await</span>;
</span>
</span>// Sort the data by path to easily read the data.
</span>data</span>.</span>sort_by_key</span>(|</span>res</span>| {
</span>    </span>if let Ok</span>((</span>path</span>, </span>_</span>)) </span>=</span> res </span>{
</span>        </span>Some</span>(</span>path</span>.</span>clone</span>())
</span>    </span>} </span>else </span>{
</span>        </span>None
</span>    </span>}
</span>});
</span></code></pre>
The last step is to write this data to a CSV file that is made easy by
the csv crate</a>:</p>
// src/main.rs
</span>use </span>csv</span>::</span>Writer</span>;
</span>use </span>chrono</span>::{</span>DateTime</span>,</span> Utc</span>};
</span>use </span>serde</span>::</span>Serialize</span>;
</span>use </span>std</span>::</span>io</span>::</span>Write</span>;
</span>
</span>#[</span>derive</span>(</span>Debug</span>,</span> Serialize</span>)]
</span>struct </span>CsvRecord </span>{
</span>    </span>path</span>:</span> String,
</span>    </span>views</span>: </span>usize,
</span>}
</span>
</span>// Generate the output path using the current date and time
</span>// Sample path: ./my-output-dir/2021-10-12T04:00:50Z.csv
</span>let</span> now</span>: </span>DateTime</span><</span>Utc</span>> </span>= </span>Utc</span>::</span>now</span>();
</span>let</span> output_path </span>=</span> output_dir</span>.</span>join</span>(</span>now</span>.</span>format</span>(</span>&</span>"</span>%FT%TZ.csv</span>").</span>to_string</span>())
</span>
</span>// Open the path for writes
</span>let mut</span> wtr </span>= </span>Writer</span>::</span>from_path</span>(</span>&</span>output_path</span>)</span>?</span>;
</span>
</span>// Write each successful entry
</span>for</span> view_res </span>in</span> data </span>{
</span>    </span>let </span>(</span>path</span>,</span> views</span>) </span>=</span> view_res</span>?</span>;
</span>
</span>    wtr</span>.</span>serialize</span>(</span>CsvRecord </span>{</span> path</span>,</span> views </span>})</span>?</span>;
</span>}
</span>
</span>// Finalize the CSV
</span>wtr</span>.</span>flush</span>()</span>?</span>;
</span></code></pre>
After all that code, the CLI can now be built and the data exported:</p>
# Export required env variables
</span>$ export PV_JWT="1234" PV_ACCOUNT_ID="account_id" PV_KV_ID="kv_id"
</span>
</span># Build the CLI
</span>$ cargo build --release
</span>
</span># Download the data
</span>$ ./target/release/page-tracker download --output-dir "/mnt/analytics_data"
</span>
</span># Sample data
</span>$ cat /mnt/analytics_data/2021-10-12T04:00:50Z.csv
</span>path,views
</span>/about/,2
</span>/dev/browsing-w3m-anonymously-with-tor/,1
</span></code></pre>
>>></a>
Systemd Timer</h3>
To run the CLI as a scheduled job with user systemd timers</a>, make sure
the CLI is in the PATH</code> and create these two files:</p>
// ~/.config/systemd/user/fnlog-tracker-download.timer
</span>[Unit]
</span>Description=Download page tracker data
</span>Wants=fnlog-tracker-download.timer
</span>
</span>[Service]
</span>Type=oneshot
</span>ExecStart=page-tracker download --output-dir "/mnt/analytics_data"
</span>Environment=PT_JWT=jwt
</span>Environment=PT_ACCOUNT_ID=account_id
</span>Environment=PT_KV_ID=kv_id
</span>
</span>[Install]
</span>WantedBy=multi-user.target
</span></code></pre>
// ~/.config/systemd/user/fnlog-tracker-timer.timer
</span>[Unit]
</span>Description=Trigger daily analytics download
</span>Requires=fnlog-tracker-download.service
</span>
</span>[Timer]
</span>Unit=fnlog-tracker-download.service
</span>OnCalendar=*-*-* 12:00:00
</span>
</span>[Install]
</span>WantedBy=timers.target
</span></code></pre>
Start and enable them afterwards:</p>
# Test if the service works
</span>$ systemctl --user start fnlog-tracker-download.service
</span>$ systemctl --user status fnlog-tracker-download.service
</span>
</span># Start and enable the timer
</span>$ systemctl --user start fnlog-tracker-timer.timer
</span>$ systemctl --user enable fnlog-tracker-timer.timer
</span></code></pre>
>></a>
References</h2>

Tommy Ku's Blog - Website analytics for lazy people again, with Cloudflare Worker</a></li>
Cloudflare Docs - Use Workers KV directly from Rust</a></li>
</ol>


Firefox Extension: VPN Connectivity Status
Unknown — Sat, 02 Oct 2021 00:00:00 +0000
>></a>
Motivation</h2>
I use Perfect Privacy VPN</a> as an additional security and privacy layer
for my internet traffic. While I was able to make it work consistently
with OpenVPN</a> by removing the persist-tun</a> option, it was consistently
being stopped by starting docker</a> for my development environment.
Although I have not found the correct fix yet, I want to be visually
alerted while browsing with Firefox</a> whether my VPN is down by checking
this page</a> or json endpoint</a>. A browser extension</a> polling the endpoint
could alert me to switch to a privacy browser like Tor Browser</a> or
Brave Browser</a> and possibly restart my VPN. One caveat with this
approach is browser fingerprinting</a>; however, the extension is simple
enough not to interfere with the browser to be detected but must be
kept in mind when creating extensions.</p>
(The extension can be downloaded here</a> and source code found here</a>.)</p>
>></a>
Background</h2>
Before proceeding, do read Mozilla's first extension guide</a> as the
focus will be on background scripts. The first step in creating the
extension is declaring a manifest.json</a> similar to a package.json</a>:</p>
// manifest.json
</span>{
</span>  </span>"</span>manifest_version</span>": </span>2</span>,
</span>  </span>"</span>name</span>": "</span>Perfect Privacy Checker</span>",
</span>  </span>"</span>version</span>": "</span>1.0.3</span>",
</span>  </span>"</span>description</span>": "</span>Checks if browser is connected to  Perfect Privacy VPN</span>",
</span>  </span>"</span>homepage_url</span>": "</span>https://github.com/FrancisMurillo/perfect-privacy-extension/tree/master/perfect-privacy</span>"
</span>}
</span></code></pre>
To declare a background script that will poll the endpoint, the
manifest background</a> property should declare the startup scripts:</p>
// manifest.json
</span>{
</span>  </span>// ...
</span>  </span>"</span>background</span>": {
</span>    </span>"</span>scripts</span>": ["</span>background.js</span>"]
</span>  </span>}
</span>}
</span></code></pre>
Before creating the background script, a reminder of the JSON output
to be handled:</p>
// https://checkip.perfect-privacy.com/json
</span>// Prettified and scrubbed JSON output
</span>{
</span>  </span>"</span>VPN</span>": </span>true</span>,
</span>  </span>"</span>TOR</span>": </span>false</span>,
</span>  </span>"</span>IP</span>": "</span>11.222.3.44</span>",
</span>  </span>"</span>DNS</span>": "</span>nowhere2.perfect-privacy.com</span>",
</span>  </span>"</span>CITY</span>": "</span>NOWHERE</span>",
</span>  </span>"</span>COUNTRY</span>": "</span>ANYWHERE</span>"
</span>}
</span></code></pre>
Using only plain Javascript</a> to fetch data from that endpoint via
XMLHttpRequest</a>, the initial background.js</code> looks like:</p>
// background.js
</span>// Store/cachea result from the endpoint
</span>var </span>info </span>= null</span>;
</span>
</span>// Check connectivity via JSON XHR
</span>function </span>checkStatus</span>() {
</span>    </span>try </span>{
</span>        </span>var </span>xhr </span>= new </span>XMLHttpRequest</span>()</span>;
</span>
</span>        xhr</span>.</span>open(</span>"</span>GET</span>", "</span>https://checkip.perfect-privacy.com/json</span>", </span>false</span>)</span>;
</span>        xhr</span>.</span>setRequestHeader(</span>"</span>Content-Type</span>", "</span>application/json</span>"</span>)</span>;
</span>        xhr</span>.</span>onreadystatechange </span>= function</span>() {
</span>            </span>if </span>(</span>this</span>.</span>readyState </span>=== </span>XMLHttpRequest</span>.</span>DONE </span>&& this</span>.</span>status </span>=== </span>200</span>) </span>{
</span>                info </span>= JSON</span>.</span>parse(</span>this</span>.</span>responseText)</span>;
</span>            </span>} </span>else </span>{
</span>                info </span>= null</span>;
</span>            </span>}
</span>
</span>            updateStatus()</span>;
</span>        </span>}
</span>
</span>        xhr</span>.</span>send(</span>null</span>)</span>;
</span>    </span>} </span>catch </span>(ex) </span>{
</span>        </span>// On network error, reset data
</span>        info </span>= null</span>;
</span>
</span>        updateStatus()</span>;
</span>    </span>}
</span>}
</span>
</span>// TODO: Notify extension on status
</span>function </span>updateStatus</span>() { }
</span>
</span>
</span>// Check on startup
</span>checkStatus()
</span></code></pre>
Since the extension makes a cross-domain request, the domain must be
added to the manifest permissions</a> to allow this:</p>
// manifest.json
</span>{
</span>  </span>// ...
</span>  </span>"</span>permissions</span>": [
</span>     </span>"</span>webRequest</span>",
</span>     </span>"</span>https://checkip.perfect-privacy.com/*</span>"
</span>  </span>]
</span>}
</span></code></pre>
With the core of the extension above, when does the background script
trigger the check? Rather than polling over an interval</a>, checking when
the browser is active or on focus is good enough. Looking over the
potential browser events</a> to use, I settled on tabs.onActivated</a>,
tabs.onUpdated</a> and windows.onFocusChanged</a> which covers my use case and
more can be added if needed. Adding event listeners to those:</p>
browser</span>.</span>tabs</span>.</span>onUpdated</span>.</span>addListener(checkStatus)</span>;
</span>browser</span>.</span>tabs</span>.</span>onActivated</span>.</span>addListener(checkStatus)</span>;
</span>browser</span>.</span>windows</span>.</span>onFocusChanged</span>.</span>addListener(checkStatus)</span>;
</span>
</span>checkStatus()</span>;
</span></code></pre>
Since the extension does not use privileged aspects of the browser
API, no extra permissions are needed. With this approach though, one
minor caveat is throttling the number of requests since each event
will trigger one network call. Ideally, only one in-flight request
should be allowed and perhaps throttling them every 30 seconds to
avoid making unnecessary requests:</p>
var </span>isPending </span>= false</span>;
</span>var </span>lastTime </span>= </span>0</span>;
</span>
</span>x</span>// Sketching the `checkStatus` function for only the updated parts
</span>function </span>checkStatus</span>() {
</span>    </span>// Make sure only one request is running
</span>    </span>if </span>(</span>!</span>isPending) </span>{
</span>        </span>var </span>now </span>= new </span>Date</span>()</span>;
</span>
</span>        </span>// Throttle requests for 30 seconds
</span>        </span>if </span>(now </span>- </span>lastTime </span>>= </span>30000</span>) </span>{
</span>            lastTime </span>= </span>now</span>;
</span>        </span>} </span>else </span>{
</span>            </span>return</span>;
</span>        </span>}
</span>
</span>        isPending </span>= true</span>;
</span>
</span>        </span>try </span>{
</span>            </span>// ...
</span>
</span>            xhr</span>.</span>onreadystatechange </span>= function</span>() {
</span>                isPending </span>= false</span>;
</span>
</span>                </span>/// ...
</span>            </span>}
</span>        </span>} </span>catch </span>(ex) </span>{
</span>            isPending </span>= false</span>;
</span>
</span>            </span>// ...
</span>        </span>}
</span>    </span>}
</span>}
</span></code></pre>
Before this extension can be tested, a user interface must be
implemented to render the results.</p>
>></a>
Popup</h2>
Extension pages</a> allow the extension to display an icon and on click
HTML dialog or popup</a> that is declared like so:</p>
// manifest.json
</span>{
</span>  </span>// ...
</span>  </span>"</span>icons</span>": {
</span>    </span>"</span>64</span>": "</span>icons/perfect-privacy-64.png</span>"
</span>  </span>},
</span>  </span>"</span>browser_action</span>": {
</span>    </span>"</span>default_icon</span>": "</span>icons/perfect-privacy-64.png</span>",
</span>    </span>"</span>default_title</span>": "</span>Connected to Perfect Privacy</span>",
</span>    </span>"</span>default_popup</span>": "</span>popup/index.html</span>"
</span>  </span>}
</span>}
</span></code></pre>
When the 64x64 extension icon is clicked, it will render a simple
column-value table based on the endpoint result. A quick
implementation would be:</p>
<!-- popup/index.html -->
</span><!</span>DOCTYPE html</span>>
</span><</span>html</span>>
</span>    </span><</span>head</span>>
</span>        </span><</span>meta </span>charset</span>="</span>utf-8</span>">
</span>        </span><</span>link </span>rel</span>="</span>stylesheet</span>" </span>href</span>="</span>index.css</span>"/>
</span>    </span></</span>head</span>>
</span>    </span><</span>body</span>>
</span>        </span><</span>table</span>>
</span>            </span><</span>tr</span>>
</span>                </span><</span>td </span>class</span>="</span>label</span>"></span>IP</span></</span>td</span>>
</span>                </span><</span>td </span>class</span>="</span>value</span>" </span>id</span>="</span>ip-value</span>"></span>N/A</span></</span>td</span>>
</span>            </span></</span>tr</span>>
</span>            </span><</span>tr</span>>
</span>                </span><</span>td </span>class</span>="</span>label</span>"></span>DNS</span></</span>td</span>>
</span>                </span><</span>td </span>class</span>="</span>value</span>" </span>id</span>="</span>dns-value</span>"></span>N/A</span></</span>td</span>>
</span>            </span></</span>tr</span>>
</span>            </span><</span>tr</span>>
</span>                </span><</span>td </span>class</span>="</span>label</span>"></span>CITY</span></</span>td</span>>
</span>                </span><</span>td </span>class</span>="</span>value</span>" </span>id</span>="</span>city-value</span>"></span>N/A</span></</span>td</span>>
</span>            </span></</span>tr</span>>
</span>            </span><</span>tr</span>>
</span>                </span><</span>td </span>class</span>="</span>label</span>"></span>COUNTRY</span></</span>td</span>>
</span>                </span><</span>td </span>class</span>="</span>value</span>" </span>id</span>="</span>country-value</span>"></span>N/A</span></</span>td</span>>
</span>            </span></</span>tr</span>>
</span>        </span></</span>table</span>>
</span>        </span><</span>script </span>src</span>="</span>index.js</span>"></</span>script</span>>
</span>    </span></</span>body</span>>
</span></</span>html</span>>
</span></code></pre>
html</span>, </span>body </span>{
</span>  </span>height</span>: </span>100</span>px</span>;
</span>  </span>width</span>: </span>320</span>px</span>;
</span>  </span>margin</span>: </span>0</span>;
</span>}
</span>
</span>.</span>label </span>{
</span>  </span>text-align</span>: </span>left</span>;
</span>}
</span>
</span>.</span>value </span>{
</span>  </span>text-align</span>: </span>right</span>;
</span>}
</span></code></pre>
To test running this extension, install web-ext</a>, run web-ext run</code> and
it will look like this:</p>
  </p>
Although not the fanciest interface and icon, the next question is how
will the user interface access background script information?</p>
>></a>
Ports</h2>
Rather, the background script will push data updates for the user
interface to render via runtime.Ports</a> or a socket-like connection.
Whenever the extension is clicked, the interface will attempt to
connect to the background script, then the background script will push
the info</code> variable:</p>
// popup/index.js
</span>// Connect to the background port
</span>var </span>port </span>= </span>browser</span>.</span>runtime</span>.</span>connect(</span>{</span>name</span>: "</span>popup-port</span>"}</span>)</span>;
</span>
</span>// On update from the background, render it in the table
</span>port</span>.</span>onMessage</span>.</span>addListener(</span>function</span>(</span>info</span>) {
</span>    info </span>= </span>info </span>|| </span>{}
</span>
</span>    document</span>.</span>getElementById(</span>"</span>ip-value</span>"</span>)</span>.</span>innerHTML </span>= </span>info</span>.</span>IP </span>|| </span>"</span>N/A</span>"
</span>    document</span>.</span>getElementById(</span>"</span>dns-value</span>"</span>)</span>.</span>innerHTML </span>= </span>info</span>.</span>DNS </span>|| </span>"</span>N/A</span>"
</span>    document</span>.</span>getElementById(</span>"</span>country-value</span>"</span>)</span>.</span>innerHTML </span>= </span>info</span>.</span>COUNTRY </span>|| </span>"</span>N/A</span>"
</span>    document</span>.</span>getElementById(</span>"</span>city-value</span>"</span>)</span>.</span>innerHTML </span>= </span>info</span>.</span>CITY </span>|| </span>"</span>N/A</span>"
</span>}</span>)</span>;
</span></code></pre>
The background script must now handle connection event:</p>
// background.js
</span>// Store the active interface connection
</span>var </span>port </span>= null</span>;
</span>
</span>// On interface connection, update the interface immediately
</span>function </span>connected</span>(</span>p</span>) {
</span>  port </span>= </span>p</span>;
</span>
</span>  updatePort()</span>;
</span>}
</span>
</span>browser</span>.</span>runtime</span>.</span>onConnect</span>.</span>addListener(connected)</span>;
</span>
</span>// Simply send the stored `info` variable to the UI
</span>function </span>updatePort</span>() {
</span>    </span>if </span>(port) </span>{
</span>        port</span>.</span>postMessage(info)</span>;
</span>    </span>}
</span>}
</span>
</span>//
</span>function </span>updateStatus</span>() {
</span>    </span>// If the VPN is connected, update the extension icon and title to be active or inactive
</span>    </span>if </span>(info </span>&& </span>info</span>.</span>VPN) </span>{
</span>        browser</span>.</span>browserAction</span>.</span>setTitle(</span>{ </span>title</span>: "</span>Connected to Perfect Privacy</span>" }</span>)
</span>        browser</span>.</span>browserAction</span>.</span>setIcon(</span>{ </span>path</span>: { </span>64</span>: "</span>icons/perfect-privacy-64.png</span>" } }</span>)</span>;
</span>    </span>} </span>else </span>{
</span>        browser</span>.</span>browserAction</span>.</span>setIcon(</span>{ </span>path</span>: { </span>64</span>: "</span>icons/perfect-privacy-64-inactive.png</span>" } }</span>)</span>;
</span>        browser</span>.</span>browserAction</span>.</span>setTitle(</span>{ </span>title</span>: "</span>Not Connected to Perfect Privacy</span>" }</span>)
</span>    </span>}
</span>
</span>    </span>// Also update the interface
</span>    updatePort()</span>;
</span>}
</span></code></pre>
Lastly, add the runtime.onMessage</a> and runtime.sendMessage</a> permissions
to allow this intercommunication:</p>
// manifest.json
</span>{
</span>  </span>// ...
</span>  </span>"</span>permissions</span>"</span>: [
</span>    </span>// ...
</span>    </span>"</span>runtime.onMessage</span>",
</span>    </span>"</span>runtime.sendMessage</span>"
</span>  ]
</span> </span>}
</span>
</span></code></pre>
Finally, the extension should work and can now be installed:</p>
  </p>
>></a>
References</h2>
Some minor references about this topic:</p>
Installing unsigned extensions</a>
: If testing the extension locally
is needed.</p>
macro_railroad_ext</a>
: An example non-trivial cross-browser extension
with web assembly.</p>


Elixir Bit - Offline (Docker) Hex Mirror
Unknown — Mon, 27 Sep 2021 00:00:00 +0000
>></a>
Motivation</h2>
When using docker containers as an Elixir development environment,
bandwidth limitations or frequent network failures can hinder the
development flow as containers before starting check if the
dependencies are updated via mix deps.get</a> command. While it is
frustrating to keep retrying on timeout issues, errors from fetching
via hex.pm</a> elevates this issue and has been more frequent based on my
co-workers experience that has become a productivity issue. Although
hex.pm's historical uptime</a> does not indicate frequent downtime,
unreliable internet service during peak or working hours may be a
cause. Regardless, it would be nice if the dependencies to fetch for
the Elixir app could be stored offline and fetched locally to avoid
faulty network trips.</p>
(Example repository for this article can be found here</a>.)</p>
2021-09-29 Update: Add offline support for Git dependencies</a></p>
>></a>
hex.registry</h2>
When hex v0.21.0</a> was released, it came with the ability to self-host
hexpm mirrors via mix hex.registry build</a> command. Assuming an Elixir
app, the custom mirror can be built by:</p>
$ cd ~/my_elixir_app
</span>
</span># Install the latest version of hex >= 0.21.0
</span>$ mix local.hex
</span>
</span>$ openssl genrsa -out private_key.pem
</span>$ mkdir .hex_repo
</span>
</span># Initialize the hex mirror
</span># The argument --name=hexpm is important later on
</span>$ mix hex.registry build .hex_repo --name=hexpm --private-key=private_key.pem
</span>
</span>* creating .hex_repo/public_key
</span>* creating .hex_repo/tarballs
</span>* creating .hex_repo/names
</span>* creating .hex_repo/versions
</span></code></pre>
The newly created mirror does not have any packages stored yet. To
store a single package in this offline repository, it is fetched
online with mix hex.package fetch</a> and then moved to the mirror:</p>
$ cd ~/my_elixir_app
</span>
</span># Fetch decimal 2.0.0 package
</span>$ mix hex.package fetch decimal 2.0.0
</span>
</span># Move to repo/tarballs
</span>$ mv decimal-2.0.0.tar .hex_repo/tarballs/
</span>
</span># Rebuild package indices
</span>$ mix hex.registry build .hex_repo --name=hexpm --private-key=private_key.pem
</span></code></pre>
Fetching all the dependencies of the application is a bit more
complicated as it requires parsing the mix.lock</a> to acquire a list of
package and corresponding version. Thankfully, a useful fact about the
lock format is that it is just a serialized Elixir map, meaning we can
just parse and load it via Code.eval_file</a> and then process it with the
Enum module</a> as a simple Elixir script. For example with this sample
lock file:</p>
# mix.lock
</span>%</span>{
</span>  </span>"</span>decimal</span>": {:</span>hex</span>, :</span>decimal</span>, "</span>1.8.1</span>", "</span>a4ef3f5f3428bdbc0d35374029ffcf4ede8533536fa79896dd450168d9acdf3c</span>", [:</span>mix</span>], [], "</span>hexpm</span>"},
</span>  </span>"</span>ecto</span>": {:</span>hex</span>, :</span>ecto</span>, "</span>3.1.7</span>", "</span>fa21d06ef56cdc2fdaa62574e8c3ba34a2751d44ea34c30bc65f0728421043e5</span>", [:</span>mix</span>], [{:</span>decimal</span>, "</span>~> 1.6</span>", [</span>hex</span>: :</span>decimal</span>, </span>repo</span>: "</span>hexpm</span>", </span>optional</span>: </span>false</span>]}, {:</span>jason</span>, "</span>~> 1.0</span>", [</span>hex</span>: :</span>jason</span>, </span>repo</span>: "</span>hexpm</span>", </span>optional</span>: </span>true</span>]}], "</span>hexpm</span>"},
</span>  </span>"</span>espec</span>": {:</span>hex</span>, :</span>espec</span>, "</span>1.8.1</span>", "</span>338d2f49cf4038bf617de7fcb0f92396f31a7b2febf506cfb56a6ac9ac18b802</span>", [:</span>mix</span>], [{:</span>meck</span>, "</span>~> 0.8.13</span>", [</span>hex</span>: :</span>meck</span>, </span>repo</span>: "</span>hexpm</span>", </span>optional</span>: </span>false</span>]}], "</span>hexpm</span>"},
</span>  </span>"</span>ex_image_info</span>": {:</span>hex</span>, :</span>ex_image_info</span>, "</span>0.2.4</span>", "</span>610002acba43520a9b1cf1421d55812bde5b8a8aeaf1fe7b1f8823e84e762adb</span>", [:</span>mix</span>], [], "</span>hexpm</span>"},
</span>  </span>"</span>meck</span>": {:</span>hex</span>, :</span>meck</span>, "</span>0.8.13</span>", "</span>ffedb39f99b0b99703b8601c6f17c7f76313ee12de6b646e671e3188401f7866</span>", [:</span>rebar3</span>], [], "</span>hexpm</span>"},
</span>  </span>"</span>mime</span>": {:</span>hex</span>, :</span>mime</span>, "</span>1.3.1</span>", "</span>30ce04ab3175b6ad0bdce0035cba77bba68b813d523d1aac73d9781b4d193cf8</span>", [:</span>mix</span>], [], "</span>hexpm</span>"},
</span>  </span>"</span>nimble_parsec</span>": {:</span>hex</span>, :</span>nimble_parsec</span>, "</span>0.5.3</span>", "</span>def21c10a9ed70ce22754fdeea0810dafd53c2db3219a0cd54cf5526377af1c6</span>", [:</span>mix</span>], [], "</span>hexpm</span>"},
</span>  </span>"</span>plug</span>": {:</span>hex</span>, :</span>plug</span>, "</span>1.9.0</span>", "</span>8d7c4e26962283ff9f8f3347bd73838e2413fbc38b7bb5467d5924f68f3a5a4a</span>", [:</span>mix</span>], [{:</span>mime</span>, "</span>~> 1.0</span>", [</span>hex</span>: :</span>mime</span>, </span>repo</span>: "</span>hexpm</span>", </span>optional</span>: </span>false</span>]}, {:</span>plug_crypto</span>, "</span>~> 1.0</span>", [</span>hex</span>: :</span>plug_crypto</span>, </span>repo</span>: "</span>hexpm</span>", </span>optional</span>: </span>false</span>]}, {:</span>telemetry</span>, "</span>~> 0.4</span>", [</span>hex</span>: :</span>telemetry</span>, </span>repo</span>: "</span>hexpm</span>", </span>optional</span>: </span>true</span>]}], "</span>hexpm</span>"},
</span>  </span>"</span>plug_crypto</span>": {:</span>hex</span>, :</span>plug_crypto</span>, "</span>1.1.2</span>", "</span>bdd187572cc26dbd95b87136290425f2b580a116d3fb1f564216918c9730d227</span>", [:</span>mix</span>], [], "</span>hexpm</span>"},
</span>}
</span></code></pre>
A simple script to process it and call mix hex.package fetch</code> for
each package and version would be:</p>
# update_local_repo.ex
</span>{</span>mix_map </span>, []} </span>= </span>Code</span>.</span>eval_file</span>("</span>mix.lock</span>")
</span>
</span># Enable mix commands programatically
</span>Mix</span>.</span>start</span>()
</span>
</span>mix_map
</span>|> </span>Enum</span>.</span>map</span>(</span>fn </span>{</span>package</span>,</span> entry</span>} </span>->
</span>  # Every third entry of the tuple is the exact version
</span>  # entry = {:hex, :decimal, "1.8.1", "omittedhash", [:mix], [], "hexpm"}
</span>  </span>{</span>package</span>,</span> elem</span>(</span>entry</span>, </span>2</span>)}
</span>end</span>)
</span>|> </span>Enum</span>.</span>filter</span>(</span>fn </span>{</span>_package</span>,</span> version</span>} </span>->
</span>  # Optional, remove git packages since they do not have versions but revisions
</span>  # version = {:git, "https://github.com/annkissam/rummage_ecto.git", "gitrev", [branch: "v2.0"]}
</span>  </span>case </span>Version</span>.</span>parse</span>(</span>version</span>) </span>do
</span>    </span>{:</span>ok</span>,</span> _</span>} </span>-> true
</span>    </span>:</span>error </span>-> false
</span>  </span>end
</span>end</span>)
</span># Optional, sort packages by name to get predicatable progress
</span>|> </span>Enum</span>.</span>sort_by</span>(</span>&</span>elem</span>(</span>&</span>1</span>, </span>0</span>))
</span># Optional, Task.async_stream is just to parallelize the process. Enum.each also works.
</span>|> </span>Task</span>.</span>async_stream</span>(</span>fn </span>{</span>package</span>,</span> version</span>} </span>->
</span>  # Run mix hex.package fetch for each package and store in the mirror
</span>  </span>Mix</span>.</span>shell</span>().</span>cmd</span>("</span>mix hex.package fetch </span>#{</span>package</span>} #{</span>version</span>}</span> --output .hex_repo/tarballs/</span>")
</span>end</span>, </span>timeout</span>: :</span>infinity</span>, </span>max_concurrency</span>: </span>3</span>)
</span>|> </span>Stream</span>.</span>run</span>()
</span></code></pre>
Running the script and rebuilding the mirror afterwards:</p>
$ elixir update_local_repo.ex
</span>
</span>ecto v3.1.7 downloaded to .hex_repo/tarballs/ecto-3.1.7.tar
</span>espec v1.8.1 downloaded to .hex_repo/tarballs/espec-1.8.1.tar
</span>decimal v1.8.1 downloaded to .hex_repo/tarballs/decimal-1.8.1.tar
</span>ex_image_info v0.2.4 downloaded to .hex_repo/tarballs/ex_image_info-0.2.4.tar
</span>mime v1.3.1 downloaded to .hex_repo/tarballs/mime-1.3.1.tar
</span>meck v0.8.13 downloaded to .hex_repo/tarballs/meck-0.8.13.tar
</span>nimble_parsec v0.5.3 downloaded to .hex_repo/tarballs/nimble_parsec-0.5.3.tar
</span>plug v1.9.0 downloaded to .hex_repo/tarballs/plug-1.9.0.tar
</span>plug_crypto v1.1.2 downloaded to .hex_repo/tarballs/plug_crypto-1.1.2.tar
</span>
</span>$ tree .hex_repo/tarballs
</span>
</span>.hex_repo/tarballs
</span>├── decimal-1.8.1.tar
</span>├── ecto-3.1.7.tar
</span>├── espec-1.8.1.tar
</span>├── ex_image_info-0.2.4.tar
</span>├── meck-0.8.13.tar
</span>├── mime-1.3.1.tar
</span>├── nimble_parsec-0.5.3.tar
</span>├── plug-1.9.0.tar
</span>└── plug_crypto-1.1.2.tar
</span>
</span>0 directories, 9 files
</span>
</span># Rebuild package indices
</span>$ mix hex.registry build .hex_repo --name=hexpm --private-key=private_key.pem
</span>
</span>* creating .hex_repo/packages/decimal
</span>* creating .hex_repo/packages/ecto
</span>* creating .hex_repo/packages/espec
</span>* creating .hex_repo/packages/ex_image_info
</span>* creating .hex_repo/packages/meck
</span>* creating .hex_repo/packages/mime
</span>* creating .hex_repo/packages/nimble_parsec
</span>* creating .hex_repo/packages/plug
</span>* creating .hex_repo/packages/plug_crypto
</span>* updating .hex_repo/names
</span>* updating .hex_repo/versions
</span></code></pre>
Serving the .hex_repo</code> with any static file server like nginx</a> will
turn this into a proper local hexpm</code> mirror. With just Elixir/Erlang,
it can be as simple as:</p>
$ erl -s inets -eval
</span>'inets:start(httpd,[{port,8000},{server_name,"localhost"},{server_root,"."},{document_root,".hex_repo"}]).'
</span></code></pre>
>></a>
hex.repo</h2>
To use the local mirror thought is a bit messy. Remember the name of
the mirror is specifically --name=hexpm</code>, the reason is to circumvent
adding repo: "name_of_repo"</code> for each of the dependency in mix.exs</code>.
To explain further, assuming the name of our mirror is instead
local</code>, the mirror can be registered this way through mix hex.repo
add</a>:</p>
# Notice name is now local instad of hexpm
</span>$ mix hex.registry build .hex_repo --name=local --private-key=private_key.pem
</span>
</span># Add mirror to hex
</span>$ mix hex.repo add local http://localhost:8000 --public-key=.hex_repo/public_key
</span>
</span># Check if our local mirror is registered
</span>$ mix hex.repo list
</span>
</span>Name            URL                                                   Public key                                          Auth key
</span>hexpm           https://repo.hex.pm                                   SHA256:O1LOYhHFW4kcrblKAxROaDEzLD8bn1seWbe5tq8TRsk
</span>local           http://localhost:8000                                 SHA256:yRX8noVK1hcBU1e5FA7yv9fhz3v3wlzzF4PCZhwsVeI
</span></code></pre>
After registering the mirror, every dependency in the application must
add repo: "local"</code> that looks like this:</p>
# mix.exs
</span>defp </span>deps </span>do
</span>  </span>[
</span>    </span>{:</span>ecto</span>, "</span>~> 3.1.7</span>", </span>repo</span>: "</span>local</span>"},
</span>    </span>{:</span>ex_image_info</span>, "</span>~> 0.2.4</span>", </span>repo</span>: "</span>local</span>"},
</span>    </span>{:</span>nimble_parsec</span>, "</span>~> 0.5.0</span>", </span>repo</span>: "</span>local</span>"},
</span>    </span>{:</span>plug</span>, "</span>~> 1.9.0</span>", </span>repo</span>: "</span>local</span>"},
</span>    </span>{:</span>espec</span>, "</span>~> 1.8.1</span>", </span>repo</span>: "</span>local</span>"}
</span>  </span>]
</span>  # Or more generically
</span>  </span>|> </span>Enum</span>.</span>map</span>(</span>fn </span>{</span>package</span>,</span> version</span>,</span> opts</span>} </span>->
</span>    </span>{</span>package</span>,</span> version</span>, </span>Keyword</span>.</span>put_new</span>(</span>opts</span>, :</span>repo</span>, "</span>local</span>")}
</span>  </span>end</span>)
</span>end
</span></code></pre>
With either method, it should work for simpler projects; however for
my umbrella app with complex dependencies, I get this error with mix deps.get</code>:</p>
Dependencies have diverged:
</span>* poison (Hex package)
</span>  different specs were given for the poison app:
</span>
</span>  > In apps/my_umbrella_app/mix.exs:
</span>    {:poison, "== 3.1.0", [env: :prod, hex: "poison", repo: "local"]}
</span>
</span>  > In deps/elixir_nsq/mix.exs:
</span>    {:poison, "~> 3.0", [env: :prod, repo: "hexpm", hex: "poison"]}
</span>
</span>  Ensure they match or specify one of the above in your deps and set "override: true"
</span>* conduit (Hex package)
</span>  the dependency conduit in apps/my_umbrella_app/mix.exs is overriding a child dependency:
</span>
</span>  > In apps/my_umbrella_apps/mix.exs:
</span>    {:conduit, "== 0.12.10", [env: :prod, hex: "conduit", repo: "local"]}
</span>
</span>  > In deps/conduit_plugs/mix.exs:
</span>    {:conduit, "0.12.10", [env: :prod, repo: "hexpm", hex: "conduit"]}
</span>
</span>  Ensure they match or specify one of the above in your deps and set "override: true"
</span>** (Mix) Can't continue due to errors on dependencies
</span></code></pre>
The issue here is that the dependency versions and repository are not
the same and adding override: true</code> option does not solve the issue.
Whether it is a bug or not, I was not able to find a fix for this
issue nor should I be required to change my version requirements. So
my unsafe workaround is to override the default hexpm</code> repository
entry with the local one:</p>
# Note the repo name is hexpm
</span>$ mix hex.registry build .hex_repo --name=hexpm --private-key=private_key.pem
</span>$ mix hex.repo add hexpm http://localhost:8000 --public-key=.hex_repo/public_key
</span>
</span># Check if our hexpm repo is overriden
</span>$ mix hex.repo list
</span>
</span>Name            URL                                                   Public key                                          Auth key
</span>hexpm           http://localhost:8000                                 SHA256:HFj7kHFiEFD7c7YyJ9W9MJATcLKZZsW+4/AMs1Znzgo
</span></code></pre>
One minor security issue is that the new local repository signature is
not the same as the old online signature. Thankfully, this can be
overriden by the environment variable HEX_NO_VERIFY_REPO_ORIGIN=1</a> and
fetching dependencies via the local mirror now works:</p>
$ HEX_NO_VERIFY_REPO_ORIGIN=1 mix deps.get
</span></code></pre>
It may report some fetch error, but it should still work. To revert to
the old repository, download the online public key</a> to a file and
override hexpm</code> registry again:</p>
# Assuming the public key from hex.pm is hex_public_key
</span>$ mix hex.repo add hexpm https://repo.hex.pm  --public-key=hex_public_key
</span></code></pre>
Almost all dependencies can be fetched locally except for git
dependencies</a> since those are fetched with git clone</a> and making a local
git repository is beyond the scope of this article. One naive solution
is to build those dependencies with hex.build</a>, copy them into the
mirror and convert their deps</code> entries from git</code> to standard
hexpm</code>; however, this might not be worthwhile to configure per
environment specially when building or releasing remotely with
continuous integration tools or edeliver</a>. Until hex</code> provides custom
fetchers, leaving the git</code> dependencies alone is safer and still a
win overall.</p>
>></a>
docker</h2>
Applying this fix back to the docker development environment that uses
docker-compose</a> which typically looks like this:</p>
# docker-compose.yml
</span># Simplified for this example
</span>version</span>: '</span>3.1</span>'
</span>services</span>:
</span>  </span># Any static HTTP server like nginx
</span>  </span>app_deps</span>:
</span>    </span>image</span>: </span>nginx:alpine
</span>    </span>volumes</span>:
</span>      </span>- </span>./.hex_repo:/usr/share/nginx/html
</span>  </span># The Elixir (phoenix REST API) application
</span>  </span>app</span>:
</span>    </span>image</span>: </span>bitwalker/alpine-elixir
</span>    </span>command</span>: "</span>mix phx.server</span>"
</span>    </span># Add the registry in the entrypoint
</span>    </span>entrypoint</span>: </span>/opt/app/docker-entrypoint.sh
</span>    </span>volumes</span>:
</span>      </span>- </span>.:/opt/app
</span>      </span># Use docker volume to avoid conflicting with docker deps folder
</span>      </span>- </span>mix_deps:/opt/app/deps
</span>      </span># Cache the builds to avoid recompilation
</span>      </span>- </span>mix_build:/opt/app/_build
</span>    </span># Make sure the HTTP server is running before this
</span>    </span>depends_on</span>:
</span>      </span>- </span>app_deps
</span>    </span>links</span>:
</span>      </span>- </span>app_deps:app_deps
</span>    </span>environment</span>:
</span>      </span># Specify the hex repo to register
</span>      </span>HEX_REGISTRY</span>: "</span>http://app_deps:80</span>"
</span>
</span>volumes</span>:
</span>  </span>mix_deps</span>:
</span>  </span>mix_build</span>:
</span></code></pre>
In the entrypoint, the local hex mirror is registered if
HEX_REGISTRY</code> is present:</p>
# docker-entrypoint.sh
</span>#!/bin/sh
</span>
</span>set </span>-</span>e
</span>
</span># Allow the image to use the local or online mirror
</span>if </span>[[ </span>-</span>z </span>"${</span>HEX_REGISTRY</span>}" </span>]]</span>; then
</span>  echo </span>"</span>Using online hexpm mirror</span>"
</span>  </span>mix</span> hex.repo add hexpm </span>"</span>https://repo.hex.pm</span>" --</span>public-key</span>=</span>"</span>hex_public_key</span>"
</span>  </span>export </span>HEX_NO_VERIFY_REPO_ORIGIN</span>=
</span>else
</span>  echo </span>"</span>Using local hexpm mirror: </span>$</span>HEX_REGISTRY</span>"
</span>  </span>mix</span> hex.repo add hexpm </span>"$</span>HEX_REGISTRY</span>" --</span>public-key</span>=</span>"</span>.hex_repo/public_key</span>"
</span>  </span>export </span>HEX_NO_VERIFY_REPO_ORIGIN</span>=</span>1
</span>fi
</span>
</span>echo </span>"</span>Updating dependencies if any uninstalled packages...</span>"
</span>mix</span> do deps.get
</span>
</span>exec </span>"$</span>@</span>"
</span></code></pre>
>></a>
Workflow</h2>
Whenever mix.lock</code> or any dependencies are changed, the mirror should
be ideally updated and rebuilt as well. One approach specially when
switching branches is to check if mix.lock</code> has changed then rebuild
which looks like this:</p>
# Check if mix.lock was changed using `md5sum`
</span>touch .mix.lock.old
</span>
</span>OLD_HASH = $(md5sum ".mix.lock.old")
</span>
</span>cat mix.lock > .mix.lock.old
</span>NEW_HASH = $(md5sum "mix.lock")
</span>
</span>if [ "$OLD_HASH" != "$NEW_HASH" ] ; then
</span>    echo "Updating local mirror"
</span>    mkdir -p .hex_repo/tarballs
</span>    elixir update_local_repo.ex
</span>    mix hex.registry build .hex_repo --name=hexpm --private-key=private_key.pem
</span>fi
</span></code></pre>
Rather than executing a rebuild per branch or start, perhaps it is
easier to store the local mirror in git</code> so that the person
responsible for the branch should always rebuild the mirror and push
that with the final changes. For my medium-sized project, the size of
my .hex_repo</code> folder is around 10MB</code> which is an acceptable
trade-off:</p>
# Store the mirror and keys in git
</span>git add .hex_repo private_key hex_public_key update_local_repo.ex
</span>git commit -m "feat: store local repository"
</span>
</span># When working with on a feature branch
</span>git checkout -b feature/work
</span>
</span># Assuming a new dependency is added after writing the feature
</span>nano mix.exs
</span>git add mix.exs lib/
</span>
</span># Revert to online repository and then pull
</span>mix hex.repo add hexpm "https://repo.hex.pm" --public-key="hex_public_key"
</span>mix deps.get
</span>
</span># Rebuild the repository
</span>elixir update_local_repo.ex
</span>mix hex.registry build .hex_repo --name=hexpm --private-key=private_key.pem
</span>
</span>git add .hex_repo
</span>git commit -m "feat: my new feature"
</span>
</span>git push
</span></code></pre>
This is not a perfect process since people can forget to update the
mirror even with git hooks. Nonetheless, this works to minimize online
fetching and putting the burden on the branch creator rather than its
users.</p>
>></a>
Git Repo</h2>
After this development feature was merged, my co-worker still had
issue specially with the remaining git dependencies. The issue is that
the docker containers does not allow external network access for some
reason. Because debugging the docker network issue could be
potentially hard to resolve, caching the git dependencies is
mandatory.</p>
For each git or more precisely GitHub</a> dependency in mix.exs</code>, they
are cloned bare and the paths replicated under a .git_repo</code> folder
that will be similarly served via nginx</code> like with .hex_repo</code>.
Given this mix.exs</code>:</p>
defp </span>deps </span>do
</span>  </span>[
</span>    </span>{:</span>elixir_nsq</span>,
</span>     </span>git</span>: "</span>https://github.com/wistia/elixir_nsq</span>",
</span>     </span>ref</span>: "</span>b43616a08459451cc5afdcd9839b732cbc1dedfa</span>",
</span>     </span>override</span>: </span>true</span>},
</span>    </span>{:</span>rummage_ecto</span>, </span>git</span>: "</span>https://github.com/annkissam/rummage_ecto.git</span>", </span>branch</span>: "</span>v2.0</span>"}
</span>  </span>]
</span>end
</span></code></pre>
The construction of the new repository would be:</p>
$ cd ~/my_elixir_app
</span>
</span># Create the git repository
</span>$ mkdir .git_repo
</span>$ cd .git_repo
</span>
</span># Clone each git repository bare
</span>$ git clone --bare https://github.com/annkissam/rummage_ecto.git annkissam/rummage_ecto.git
</span>$ git clone --bare https://github.com/wistia/elixir_nsq.git wistia/elixir_nsq.git
</span></code></pre>
As an Elixir script reading mix.lock</code>:</p>
# update_git_repo.ex
</span>{</span>mix_map </span>, []} </span>= </span>Code</span>.</span>eval_file</span>("</span>mix.lock</span>")
</span>
</span>Mix</span>.</span>start</span>()
</span>
</span># Cleanup git repository folder
</span>Mix</span>.</span>shell</span>().</span>cmd</span>("</span>rm -rf .git_repo/*</span>")
</span>
</span>mix_map
</span>|> </span>Enum</span>.</span>filter</span>(</span>fn
</span>  # Filter out
</span>  # entry = "elixir_nsq": {:git, "https://github.com/wistia/elixir_nsq", "sharev", [ref: "sharevused"]}
</span>  </span>{</span>package</span>, {:</span>git</span>, "</span>https://github.com</span>" </span><></span> _path</span>,</span> _rev</span>,</span> _opts</span>}} </span>-> true
</span>  _ </span>-> false
</span>end</span>)
</span>|> </span>Enum</span>.</span>map</span>(</span>fn </span>{</span>package</span>,</span> entry</span>} </span>-> </span>{</span>package</span>,</span> elem</span>(</span>entry</span>, </span>1</span>)} </span>end</span>)
</span>|> </span>Enum</span>.</span>sort_by</span>(</span>&</span>elem</span>(</span>&</span>1</span>, </span>0</span>))
</span>|> </span>Task</span>.</span>async_stream</span>(</span>fn </span>{</span>package</span>,</span> git_url</span>} </span>->
</span>  </span>"</span>https://github.com/</span>" </span><></span> path </span>=</span> git_url
</span>
</span>  # Normalize path ends with .git for bare repositories
</span>  path </span>= if</span>(</span>String</span>.</span>ends_with?</span>(</span>path</span>, "</span>.git</span>"), </span>do</span>:</span> path</span>, </span>else</span>: "#{</span>path</span>}</span>.git</span>")
</span>
</span>  </span>Mix</span>.</span>shell</span>().</span>cmd</span>("</span>git clone --bare </span>#{</span>git_url</span>}</span> .git_repo/</span>#{</span>path</span>}")
</span>end</span>, </span>timeout</span>: :</span>infinity</span>, </span>max_concurrency</span>: </span>3</span>)
</span>|> </span>Stream</span>.</span>run</span>()
</span></code></pre>
Cloning the whole repo can be huge or unnecessary so adding
--single-branch</a> and --branch</a> clone or other filtering options may be
worth investigating. For my same project, it added a hefty 5MB</code> for
only 5 or so dependencies which is not ideal. Nonetheless, serving
this .git_repo</code> folder with another nginx</code> container as an alias to
https://github.com</a> is our sneaky strategy similar to overriding
hexpm</code>.</p>
However, it is not as straightforward as making nginx</code> a compatible
git HTTP server requires git-http-backend</a> or fcgiwrap</a> integration that
requires a custom nginx</code> configuration. After some research, the
dockerfile and configuration I landed on is:</p>
# Dockerfile.gitx
</span>FROM</span> nginx:</span>1.21.3-alpine
</span>
</span># Make sure git-http-backend and fcgiwrap is installed
</span>RUN </span>apk add --update git git-daemon apache2-utils fcgiwrap
</span>
</span># Make sure the git repo root is accessible to nginx
</span>RUN </span>mkdir -p /srv/git && chown -R nginx:nginx /srv/git && chmod -R 755 /srv/git
</span></code></pre>
# git-nginx.conf
</span>
</span>server </span>{
</span>    </span># Setup HTTP endpoint
</span>    </span>listen       </span>80</span>;
</span>    </span>server_name  </span>github</span>.</span>com;
</span>
</span>    </span># Serve every repository under /srv/git
</span>    </span>root </span>/</span>srv</span>/</span>git;
</span>
</span>    location /</span> {
</span>        </span>try_files </span>$uri $uri</span>/ =</span>404</span>;
</span>    }
</span>
</span>    location ~</span> (</span>/.*</span>) {
</span>        </span>client_max_body_size </span>0</span>;
</span>        include       /</span>etc</span>/</span>nginx</span>/</span>fastcgi_params;
</span>        </span># Found under git-daemon package
</span>        </span>fastcgi_param </span>SCRIPT_FILENAME     /</span>usr</span>/</span>libexec</span>/</span>git</span>-core/git-http-backend</span>;
</span>
</span>        </span># Serve every repository under /srv/git
</span>        </span>fastcgi_param </span>GIT_PROJECT_ROOT    /</span>srv</span>/</span>git;
</span>        </span>fastcgi_param </span>GIT_HTTP_EXPORT_ALL </span>""</span>;
</span>
</span>        </span>fastcgi_param </span>PATH_INFO           </span>$1;
</span>
</span>        </span># fcgi socket to conect to git daemon
</span>        </span>fastcgi_pass  </span>unix</span>:/</span>var</span>/</span>run</span>/</span>fcgiwrap</span>.</span>sock</span>-</span>1</span>;
</span>    }
</span>}
</span></code></pre>
This basic configuration just serves every git repository under
/srv/git</code> without authentication, so it is only a read or clone only
server. The tricky part here is setting up the fcgiwrap</code> socket before
nginx</code> runs so a custom entrypoint is also needed here:</p>
# git-entrypoint.sh
</span>#!/bin/bash
</span>
</span>echo "Creating fcgiwrap socket"
</span>rm -f /var/run/fcgiwrap.sock-1
</span>fcgiwrap -s unix:/var/run/fcgiwrap.sock-1 &
</span>
</span>echo "Waiting on fcgiwrap socket"
</span>sleep 1
</span>chown nginx:nginx /var/run/fcgiwrap.sock-1
</span>chmod 777 /var/run/fcgiwrap.sock-1
</span>
</span>echo "Starting nginx"
</span>nginx -g 'daemon off;'
</span></code></pre>
I initially thought of using OpenRC</a> but it was less consistent in
starting up and creating the fcgiwrap</code> socket manually was easier
although slightly less efficient. Regardless, adding HTTPS support or
proxy is the last but easier step which just requires a small nginx</code>
update:</p>
# Original HTTP endpoint
</span>
</span>server </span>{
</span>  </span>listen </span>443</span> ssl;
</span>  </span>server_name </span>github</span>.</span>com;
</span>
</span>  </span># Generated with
</span>  </span># openssl req -x509 -nodes -days 365 -subj "/C=CA/ST=QC/O=Company, Inc./CN=mydomain.com" -addext "subjectAltName=DNS:mydomain.com" -newkey rsa:2048 -keyout git-selfsigned.key -out git-selfsigned.crt
</span>  </span>ssl_certificate      </span>/</span>srv</span>/</span>git</span>-</span>selfsigned</span>.</span>crt;
</span>  </span>ssl_certificate_key  </span>/</span>srv</span>/</span>git</span>-</span>selfsigned</span>.</span>key;
</span>
</span>  location /</span> {
</span>    </span>proxy_pass </span>http://127.0.0.1:80</span>;
</span>  }
</span>}
</span></code></pre>
Wiring it up in docker-compose.yml</code>:</p>
git_deps</span>:
</span>  </span>build</span>:
</span>    </span>context</span>: </span>./.git_repo
</span>    </span>dockerfile</span>: </span>../Dockerfile.gitx
</span>  </span>entrypoint</span>: "</span>sh /srv/git-entry.sh</span>"
</span>  </span># Expose ports for demonstration
</span>  </span>ports</span>:
</span>    </span>- "</span>22080:80</span>"
</span>    </span>- "</span>22443:443</span>"
</span>  </span>volumes</span>:
</span>    </span>- </span>./.git_repo:/srv/git
</span>    </span>- </span>./git-nginx.conf:/etc/nginx/conf.d/default.conf
</span>    </span>- </span>./git-entrypoint.sh:/srv/git-entry.sh
</span>    </span>- </span>./git-selfsigned.crt:/srv/git-selfsigned.crt
</span>    </span>- </span>./git-selfsigned.key:/srv/git-selfsigned.key
</span>app</span>:
</span>  </span>depends_on</span>:
</span>    </span># Make this app also wait on the git repo
</span>    </span>- </span>git_deps
</span>    </span>- </span>app_deps
</span>  </span>links</span>:
</span>    </span># Notice how git_deps is mapped as github.com
</span>    </span># To fetch online, comment this line and GIT_NO_SSL_VERIFY
</span>    </span>- </span>git_deps:github.com
</span>    </span>- </span>app_deps:app_deps
</span>  </span>environment</span>:
</span>    </span># Also disable check for self signed certificates when cloning
</span>    </span>GIT_NO_SSL_VERIFY</span>: "</span>true</span>"
</span></code></pre>
Because of the new links</code>, https://github.com</a> now points to the local
git HTTPS server from git_deps</code>. Like with
HEX_NO_VERIFY_REPO_ORIGIN</code>, this will not work as git clone</code> does
not allow self-signed certificates without bypassing it with
GIT_NO_SSL_VERIFY</a> environment variable similar to
HEX_NO_VERIFY_REPO_ORIGIN</code>. Testing it out with docker-compose up git_deps</code>:</p>
# Outsiide the container
</span># With HTTP
</span>$ git clone http://localhost:22080/annkissam/rummage_ecto.git
</span>
</span># With HTTPS
</span>$ git clone https://localhost:22443/wistia/elixir_nsq.git
</span>
</span>
</span># Within the container
</span># docker-compose exec app ash
</span>$ GIT_SSL_NO_VERIFY=true git clone https://github.com/wistia/elixir_nsq.git
</span></code></pre>
While not the most optimal solution, it does resolve my co-worker's
issue and the development environment is now completely offline aside
from extra build hooks. Remember to store .git_repo</code> and its
accompanying files. Whenever a git dependency needs to be updated,
delete and clone the bare repository needed and commit again.</p>


Mundaun - Notes and Thoughts
Unknown — Sun, 16 May 2021 00:00:00 +0000
>></a>
Mundane</h2>
Finished main content at around 6 hours on May. 02, 2021</strong></strong></p>
Mundaun PS4</a> is a first-person horror mystery adventure game with a
hand-penciled sepia aesthetic where the player journeys to his
grandfather's funeral and unravel his deal with a demonic spirit. What
makes this standout aside from its strong visuals is the language and
beliefs in the small village that radiates a sense of allure and
mystery. While this game is highly artistic and shows passion, its
gameplay aspect pull this game down severely. I like this short game and
recommend it but do know it has severe flaws.</p>
>></a>
Notes and Thoughts</h2>
To preface my criticism with the game's good points, I like the story
idea of a devil's bargain where the next generations pay the price. The
mystery presented also heightened the atmosphere and tone alongside the
sepia imagery and cinematography makes this experience memorable.
Enemies have unique designs, low weapon durability and item scarcity
increases the tension. For a small indie game with this amount of effort
and artistic merit, it is impressive and should be commended, so these
issues are painful to raise.</p>
The biggest issue with the game is the pacing and story progression. As
the story is about an intriguing curse, many of the story objectives
feel unrelated or padding and does not feel insightful or rewarding.
While a short story is fine, the objectives can be reworked to feel more
engaging and involved rather than meandering. For example, a road block
could be an invincible monster that can be weakened then killed in
several ways or a fetch quest could be a multi-item inventory puzzle
that can enhance exploration and world building. When the story and
pacing picks up, the game is at its best but that is between long
stretches of uninteresting traversal.</p>
Navigating or exploring the world is tedious and uninteresting as this
undermines the atmosphere and pacing. For this kind of first-person
exploration, a combination of well-designed maps, enemies, abilities and
pacing can remedy this. Aside from increasing the enemies at strategic
points, improving enemy design I believe can be the most effective since
most enemies are simply long range units that can be easily outrun or
evaded making them more of an annoyance than a genuine threat. If adding
new monsters and abilities is not possible, adjust from the player's
side can work such as limited health with recovery items instead of
auto-recovery, items to draw attention or stun such as rocks, slower
move speed when detected and so on. While increasing difficulty must be
balanced against being challenging or frustrating, this can make
exploration tense and treacherous while also making its atrocious
backtracking meaningful.</p>
The story experience is short and unique but unsatisfying for me with
many questions about the mechanics of the world. How the protagonist's
curse is removed does not make sense since it was not building up to it
nor attention to actually removing it. I felt the resolution with the
demon was a narrative contrivance although I did like uncovering the
demon's secret. Alongside the game its moments, the narrative payoff
does not feel rewarding despite its allure.</p>
While the game's aesthetic is a strength, it can also be a flaw within
the item hunting mechanics. Since everything is sepia toned, it is hard
to find items which could be addressed as hard contrast or highlights
accessibility options. Same issue within the inventory that is it hard
to know which is the current item again can be resolved with highlights.
While the sepia tone is unique, some people might have color blindness
so providing visual accessibility options would be welcome. As a
first-person game, it is also lacking in options such as field of view,
sensitivity and so-on to prevent motion sickness specially when driving
the truck. Nonetheless, the visuals is still a strength and can be
experienced better with more options for it.</p>
The combat system is simple but requires more polish. For the pitchfork,
attacking sometimes does not trigger and feels unreliable. The attribute
bonuses that provide increased health, fear resistance and rifle
accuracy should be removed as the game is already easy or turn them into
temporary bonuses or usage based instead although I do like making
coffee here. For the rifle, weapon sway and limited ammo is nice but
accessing the inventory to reload instead of a weapon reload animation
is strange. When doing damage from stealth, a notification appears which
break its aesthetic and should be removed. For the inventory, the game
lacks enough usable items or complex interactions to warrant the current
inventory system. The inventory should also stop time as it is hard to
access the correct item or allocate a button to cycle with relevant
items if it cannot be removed. While the game discourages combat,
engaging in it should be smooth to avoid ruining immersion.</p>
I do have other issues such as the immense loading times, harsh bee
sound effects, truck handling, fast run speed and so on but these are
minor. Nonetheless, this game will be remembered despite its shortcoming
which can hopefully inspire more kinds of this game.</p>


Rogue Heroes: Ruins of Tasos - Notes and Thoughts
Unknown — Sat, 10 Apr 2021 00:00:00 +0000
>></a>
Rogue Heroes: Ruins of Tasos</h2>
Finished the main content on Switch around 14 hours</strong></strong></p>
Rogue Heroes: Ruins of Tasos</a> is a The Legend of Zelda</a> inspired from its
pixel art style, top-down perspective, map grid, room puzzles, item
progression and so on. Primarily, it is a rogue-lite</a> featuring multiple
classes, statistical progression, procedurally dungeons, town building,
and multiplayer. While the game may be fine for what it is, it feels
lacking in content, complexity and challenge to be valuable or
meaningful.</p>
>></a>
Notes and Thoughts</h2>
Initially I had fun specially with multiplayer hijinks, my enjoyment and
investment was ruined with many issues. Primarily, its core dungeon loop
does not feel rewarding because of the bosses, combat, puzzles and
difficulty.</p>
As a dungeon crawler, dungeon bosses test and challenge players about
the game's different aspects in general. Aside from the basic patterns
and concepts, they do not exercise mechanics or incorporate items to
make it exciting or technical. Regardless of that, boss patterns are
simplistic or uninteresting and some can be trivialized with the builtin
shield. The final boss is almost an exception which provide a mechanical
challenge; however, it feels unpolished and unfair with its execution
and randomness. Overall, bosses are forgettable and unremarkable.</p>
As a Zelda</em> inspired game, sword attacks are the main form of damage;
however, it means the combat is primarily slashing an enemy and running
away which becomes stale without complexity or depth. The game already
provides other items and weapons such as a bow, wand or hammer; however,
they are limited by a mana attribute or item capacity which may be
intended as secondary weapons. The problem with them as secondary
weapons is that they do not do enough damage until upgraded but would be
better well spent on upgrading the sword instead which perpetuates the
lack of complexity. In short, the game discourages ranged weapons or
items without providing depth to the only viable weapon.</p>
While I can only talk about local co-op, the huge enemy knockback from
sword attacks is dangerous to allies specially with the high contact
damage. The sword range is too close or enemies do not get consistent
hit stun that enemies accidentally do contact damage even with sword
range upgrades which is personally infuriating and encourages splitting
up. Aside from reviving allies, no major mechanic encourages cooperation
which makes the multiplayer feel tacked on.</p>
Alongside that issue is the class system which is supposed to give
different playstyles or builds. The game lets you pick from 8 unlockable
classes which differ mostly in their special move, class bonus, attack,
defense and speed. Since swords are the most viable, picking a ranged
class or non-melee class like the ranger or mage is not good. What dooms
them further is their low defense which receives extremely high damage
compared to their melee counterparts which feels unfairly balanced or
skewed. The special traits and moves from the melee classes are
underwhelming or bland given the effort to acquire them. Either the
class system has to be balanced better or remove it altogether since it
is deceptively dull.</p>
Another staple of Zelda</em> games are their puzzles which provide an
intellectual challenge and heighten environmental awareness. Regardless
of actual quality of the puzzles, my main complaint is the repetition of
room puzzles. While repeating a puzzle on its own is not a strong issue,
it makes exploration tedious and slow as it becomes a chore over
multiple runs. The game seems to acknowledge this by making certain
puzzles remain solved where it might be better to replace it with
another generated room instead. If puzzles becomes a dungeon chore, it
would be better to put them in the outside world to be more creatively
designed and make traveling a more rewarding experience which the game
poorly leans on with the other dungeons. The kind of dungeon puzzles the
game uses does mesh well with the repetitive nature of the game, so
removing it can make the overall pacing better and perhaps provide
challenge modifiers/rooms instead for room rewards.</p>
While I may have other smaller issues, the game fails in both being an
engaging dungeon crawler Zelda</em> or rogue-lite</em> experience. Personally,
I would probably enjoy the game more if it was not dependent on its poor
rogue-lite</em> aspect to prolong the game.</p>


Violet Evergarden - Notes and Thoughts
Unknown — Fri, 26 Feb 2021 00:00:00 +0000
>></a>
Violet Evergarden</h2>
Finished season 1 of the television series</strong></strong></p>

No letter that could be sent deserves to be undelivered</p>
</blockquote>
Violet Evergarden (Season 1)</a> is an Japanese animated television series
where an emotionally stunted ex-child soldier becomes a ghostwriter
(known as Auto Memory Dolls</em>) after the war to understand the last
words of her superior officer and guardian who sacrificed his life to
save her: I love you</em>. Those last words does not matter literally but
rather optimally</em> conveys to Violet</em> that somebody cares and wants her
to be have a future. Whatever form of love they might have had, Violet</em>
did love and care for somebody thus understanding the meaning of those
words is coming to terms with her own feelings and how important those
are. The show has a good premise of understanding people's feelings
through the letters they write which is charming and sentimental. With
13 episodes, the series explores Violet</em>'s growth and journey through
her clients and friends in an emotionally resonant manner.</p>
>></a>
Notes and Thoughts</h2>
Now as much as I like the series and its idea, I cannot fully recommend
this show because of its execution although this is still good and
emotional. Primarily, if the series is about conveying the subtleties of
feelings, the show lacks that quality to engage the viewers with its
subtleties. Characters/clients frequently say exactly what they mean
removing any mystery, tension or possible failure. Violet</em> explicitly
stating contradictions where she could have been taught to remain silent
and take note of the circumstances. Flashbacks are used too much where
moments of silence or stillness would be better. The broken doll analogy
is on the nose considering her job title, clothing style, robotic arms
and lack of emotion. Her introductions along with that seems excessive
as well. Perhaps I expected too much but it certainly felt that the
stories could be better expressed and engaging without compromising its
tone and feel.</p>
The premise of understanding those last words while intriguing is
slightly problematic as a setup and how it was handled. At face value,
those words are typically interpreted as romantic which is questionable
between an older man and teenager and more importantly becomes an
unnecessary hurdle or discussion. While the series show it is nuanced,
it could have been better handled or framed such as quickly offering
different interpretations initially then exploring it further. Since
Violet</em> can learn quickly, I am bothered it was not shown her intensely
researching or thinking about it then eventually being conflicted which
would provide more insight and character. Regardless how the
relationship may seem dependent, more scenes are needed from her
guardian as well as showing them chronologically closer to provide
better clarity and context to avoid misinterpretation.</p>
Her mature doll-like appearance is also questionable considering she is
around 14 years old at near the start of the series. While this world's
rules about age and marriage might be contextually different, perhaps a
different model and design in particular her face and height would show
her slowly maturing age better without undermining her profession. My
sense of fashion for her would be pants, long shirt, a vest, a beret and
add frills or the appropriate mid-upper class traveling working girl
fashion. Showing or framing her as introspective instead of beautiful
would help and besides she is not romantically available yet.</p>
One minor point with those last words is that perhaps the client stories
and letters should incorporate those words as well. It is established
that Violet</em> may not be able to use those words, but perhaps the act of
writing and including it on behalf of the client can be meaningful. As
it stands, the different forms of love displayed by the clients may seem
unrelated. Not truly an issue but would be a nice incorporation.</p>
A world building issue is what makes a ghostwriter good in this universe
specially how Violet</em> can ever became competent given her mechanical
mindset and lack of emotion. Although a bad ghostwriter was shown
through her coworker, perhaps showing a competition between two good
ghostwriters could highlight its nuances and technicalities without
ruining the tone. While it may be considered part of the premise, it
felt a part of her key progression or growth was rushed or waved away
where a montage or time skip could address.</p>
While it is acceptable Violet</em> herself lacks emotion, I do not buy the
lack of emotion for a lack of a strong personality. The fact that
episodes are narrated or described by other characters do not help and
somehow seem to require Violet</em> to always make an impression may be
detrimental. Perhaps showing how her military and ghostwriting
experience allowed her to resolve an personal issue without violence
would give her color and depth. Maybe using her namesake to build
interest in flowers or gardening would be something. Either way, I
sympathize with her but the series did not giver her flavor.</p>
While I do have smaller issues, these overall do not ruin a good show
but it does pull it down from being recommended. I still like the
post-war setting, reintegration of an ex-soldier, romanticizing letter
writing, and the cathartic ending but the optimal</em> words should have
been what the series started with.</p>


Rust Bit - Borrow Multiple Struct Fields
Unknown — Tue, 09 Feb 2021 00:00:00 +0000
>></a>
Problem</h2>
I have a struct that contains other components:</p>
struct </span>ArithmeticLogicUnit</span>;
</span>struct </span>ControlUnit</span>;
</span>struct </span>Register</span>;
</span>
</span>struct </span>CPU </span>{
</span>    </span>alu</span>:</span> ArithmeticLogicUnit,
</span>    </span>cu</span>:</span> ControlUnit,
</span>    </span>register</span>:</span> Register
</span>}
</span>
</span>impl </span>Default </span>for </span>CPU </span>{
</span>    </span>fn </span>default</span>() -> </span>Self </span>{
</span>        </span>Self </span>{
</span>            alu</span>:</span> ArithmeticLogicUnit</span>,
</span>            cu</span>:</span> ControlUnit</span>,
</span>            register</span>:</span> Register
</span>        </span>}
</span>    </span>}
</span>}
</span></code></pre>
I want to test a function execute</code> which changes each component
state:</p>
/// Main function to test
</span>impl </span>CPU </span>{
</span>    </span>fn </span>execute</span>(</span>&mut </span>self</span>, </span>instructions</span>: </span>&str</span>) -> </span>bool </span>{ </span>todo!</span>() }
</span>}
</span>
</span>/// Functions to check side-effects which may require mutable references
</span>impl </span>ArithmeticLogicUnit </span>{
</span>    </span>fn </span>check</span>(</span>&mut </span>self</span>, </span>register</span>: </span>&mut</span> Register </span>) -> </span>bool </span>{ </span>true </span>}
</span>}
</span>
</span>impl </span>ControlUnit </span>{
</span>    </span>fn </span>check</span>(</span>&mut </span>self</span>, </span>register</span>: </span>&mut</span> Register </span>) -> </span>bool </span>{ </span>false </span>}
</span>}
</span>
</span>
</span>#[</span>cfg</span>(</span>tests</span>)]
</span>mod </span>tests </span>{
</span>    </span>#[</span>test</span>]
</span>    </span>fn </span>execute_should_work</span>() {
</span>        </span>let mut</span> cpu </span>= </span>CPU</span>::</span>default</span>();
</span>
</span>        </span>assert!</span>(</span>cpu</span>.</span>execute</span>("</span>RANDOM CPU INSTRUCTIONS</span>"));
</span>
</span>         </span>// TODO: Assert ALU, CU and Register side-effects
</span>    </span>}
</span>}
</span></code></pre>
To test each component state directly, I need to expose mutable access
to those components since some components depend on each other. The
easiest way to achieve this is to change the visibility</a> of the struct
fields from private to public:</p>
/// Add `pub` for each field
</span>struct </span>CPU </span>{
</span>    pub </span>alu</span>:</span> ArithmeticLogicUnit,
</span>    pub </span>cu</span>:</span> ControlUnit,
</span>    pub </span>register</span>:</span> Register
</span>}
</span>
</span>let mut</span> cpu </span>= </span>CPU</span>::</span>default</span>();
</span>
</span>assert!</span>(</span>cpu</span>.</span>execute</span>("</span>BLARB</span>"));
</span>
</span>let </span>CPU </span>{ </span>ref mut</span> alu</span>, </span>ref mut</span> cu</span>, </span>ref mut</span> register </span>} </span>= &mut</span> cpu</span>;
</span>
</span>assert!</span>(</span>alu</span>.</span>check</span>(</span>register</span>));
</span>assert!</span>(</span>cu</span>.</span>check</span>(</span>register</span>));
</span></code></pre>
However, this unnecessarily exposes the internal fields outside of a
test. As an alternative, I could expose getter methods</a> for each the
components:</p>
    </span>impl </span>CPU </span>{
</span>        </span>pub fn </span>alu</span>(</span>&</span>self</span>) -> </span>&</span>ArithmeticLogicUnit </span>{
</span>'            &self</span>.</span>alu
</span>        </span>}
</span>
</span>        </span>pub fn </span>alu_mut</span>(</span>&mut </span>self</span>) -> </span>&mut</span> ArithmeticLogicUnit </span>{
</span>            </span>&mut self</span>.</span>alu
</span>        </span>}
</span>
</span>        </span>pub fn </span>cu</span>(</span>&</span>self</span>) -> </span>&</span>ControlUnit </span>{
</span>            </span>&self</span>.</span>cu
</span>        </span>}
</span>
</span>        </span>pub fn </span>cu_mut</span>(</span>&mut </span>self</span>) -> </span>&mut</span> ControlUnit </span>{
</span>            </span>&mut self</span>.</span>cu
</span>        </span>}
</span>
</span>        </span>pub fn </span>register</span>(</span>&</span>self</span>) -> </span>&</span>Register </span>{
</span>            </span>&self</span>.</span>register
</span>        </span>}
</span>
</span>        </span>pub fn </span>register_mut</span>(</span>&mut </span>self</span>) -> </span>&mut</span> Register </span>{
</span>            </span>&mut self</span>.</span>register
</span>        </span>}
</span>    </span>}
</span></code></pre>
Borrowing immutable references simultaneously works but naturally
fails for mutable references:</p>
/// Immutable references works
</span>let</span> cpu </span>= </span>CPU</span>::</span>default</span>();
</span>
</span>let</span> alu </span>=</span> cpu</span>.</span>alu</span>();
</span>let</span> cu </span>=</span> cpu</span>.</span>cu</span>();
</span>let</span> register </span>=</span> cpu</span>.</span>register</span>();
</span>
</span>/// Mutable references does not since `cpu` is exclusively borrowed for
</span>/// each component
</span>let mut</span> cpu </span>= </span>CPU</span>::</span>default</span>();
</span>
</span>let mut</span> alu </span>=</span> cpu</span>.</span>alu_mut</span>();
</span>let mut</span> cu </span>=</span> cpu</span>.</span>cu_mut</span>();
</span>let mut</span> register </span>=</span> cpu</span>.</span>register_mut</span>();
</span>
</span>assert!</span>(</span>alu</span>.</span>check</span>(</span>register</span>));
</span>assert!</span>(</span>cu</span>.</span>check</span>(</span>register</span>));
</span></code></pre>
While the borrow checker correctly prevents mutably borrowing the same
variable, how can we borrow multiple fields at the same time without
reaching for unsafe</a>?</p>
>></a>
Solution</h2>
Taking inspiration from Vec::split_at_mut</a>, the unconventional yet
slightly verbose answer is to mutable borrow everything required and
return it as a struct which combines the two ideas:</p>
/// Combining the first and second solution
</span>struct </span>CpuComponents</span><</span>'a</span>> {
</span>  pub </span>alu</span>: </span>&'a ArithmeticLogicUnit,
</span>  pub </span>cu</span>: </span>&'a ControlUnit,
</span>  pub </span>register</span>: </span>&'a Register,
</span>}
</span>
</span>struct </span>CpuComponentsMut</span><</span>'a</span>> {
</span>  pub </span>alu</span>: </span>&'a mut ArithmeticLogicUnit,
</span>  pub </span>cu</span>: </span>&'a mut ControlUnit,
</span>  pub </span>register</span>: </span>&'a mut Register,
</span>}
</span>
</span>impl </span>CPU </span>{
</span>    </span>pub fn </span>components</span><</span>'a</span>>(</span>&'a </span>self</span>) -> </span>CpuComponents</span><</span>'a</span>> {
</span>        CpuComponents </span>{
</span>            alu</span>: </span>&self</span>.</span>alu</span>,
</span>            cu</span>: </span>&self</span>.</span>cu</span>,
</span>            register</span>: </span>&self</span>.</span>register
</span>        </span>}
</span>    </span>}
</span>
</span>    </span>pub fn </span>components_mut</span><</span>'a</span>>(</span>&'a mut </span>self</span>) -> </span>CpuComponentsMut</span><</span>'a</span>> {
</span>        CpuComponentsMut </span>{
</span>            alu</span>: </span>&mut self</span>.</span>alu</span>,
</span>            cu</span>: </span>&mut self</span>.</span>cu</span>,
</span>            register</span>: </span>&mut self</span>.</span>register
</span>        </span>}
</span>    </span>}
</span>}
</span></code></pre>
Although it seems strange to create a custom data type just to satisfy
the borrow checker, Rust</em> has smart pointers such as RefCell</a> to
enforce borrows rules at runtime. Nonetheless, the test can now work
naturally.</p>
let mut</span> cpu </span>= </span>CPU</span>::</span>default</span>();
</span>
</span>let</span> CpuComponentsMut </span>{ </span>mut</span> alu</span>, </span>mut</span> cu</span>, </span>mut</span> register </span>} </span>=</span> cpu</span>.</span>components_mut</span>();
</span>
</span>assert!</span>(</span>alu</span>.</span>check</span>(</span>register</span>));
</span>assert!</span>(</span>cu</span>.</span>check</span>(</span>register</span>));
</span></code></pre>
If structs are a bit verbose, tuples can also work but be wary that it
breaks when adding or changing it:</p>
impl </span>CPU </span>{
</span>    </span>pub fn </span>components</span><</span>'a</span>>(</span>&'a </span>self</span>) -> (</span>&'a</span> ArithmeticLogicUnit, </span>&'a</span> ControlUnit, </span>&'a</span> Register</span>) {
</span>        </span>(</span>&self</span>.</span>alu</span>, </span>&self</span>.</span>cu</span>, </span>&self</span>.</span>register</span>)
</span>    </span>}
</span>
</span>    </span>pub fn </span>components_mut</span><</span>'a</span>>(</span>&'a mut </span>self</span>) -> (</span>&'a mut</span> ArithmeticLogicUnit, </span>&'a mut</span> ControlUnit, </span>&'a mut</span> Register</span>) {
</span>        </span>(</span>&mut self</span>.</span>alu</span>, </span>&mut self</span>.</span>cu</span>, </span>&mut self</span>.</span>register</span>)
</span>    </span>}
</span>}
</span>
</span>let </span>(</span>mut</span> alu</span>, </span>mut</span> cu</span>, </span>mut</span> register</span>) </span>=</span> cpu</span>.</span>components_mut</span>();
</span></code></pre>
Although my example may seem valid within a test, exposing internals
should be avoided and perhaps refactoring or new methods may be needed
instead. Overall while this perhaps incurs a slight compiler or code
overhead, maintaining the safety guarantees is worth it with a little
more work.</p>


Darkwood - A Strong Top-Down Horror
Unknown — Sun, 07 Feb 2021 00:00:00 +0000
>></a>
Darkwood</h2>
Finished at Nightmare difficulty on PS4</strong></strong></p>

This is a difficult and unforgiving game. Be patient. Respect the
woods.</p>
-- Darkwood</p>
</blockquote>
Darkwood</a> is a great survival and horror game with a top-down
camera/perspective. The game is about an amnesiac trapped deep within
a paranormal forest who must escape its warped ecosystem and
inhabitants or else be consumed it. While trying to survive, every day
the player has 12 minutes of light to scavenge and explore the forest
and 5 minutes of darkness to hunker down in a hideout and survive the
deadly creatures and random events which is the primary game loop.
What makes this game special is its emphasis on challenge with its
finite resources, item scarcity, stiff combat, weapon durability, low
health, limited sight and permadeath</a> which all provide a gripping
mechanical tension. Compounding the difficulty, the horror taps into
the fear of the unknown with its visual line of sight or darkness,
tense sound design, procedurally generated maps, nightly threats,
ambiguous characters and consequences overall creating a tense and
atmospheric horror experience. While it only has two chapters and an
epilogue, this can take around 25 hours on fully exploring the world
and story including the multiple deaths or runs incurred.</p>
>></a>
Thoughts</h2>
Aside from its nicely executed resource survival aspect, the
interesting aspect of this game is its horror and challenge stemming
from it. Horror is hard to define and quite subjective based on
personal and past experiences. Instead, several aspects of the game
build an oppressive and terrifying atmosphere that overall create a
strong horror gameplay experience.</p>
The first noteworthy thing is the top-down perspective itself which is
an unconventional choice for horror. With this perspective, it uses an
intriguing lighting system based on the character's directional vision
cone which can be obstructed and everything else darkened and
masked/disguised until seen directly. While the player can see the
whole screen, they can only see what the player character</a> does and the
game plays with that fear of the unknown such as hidden enemies, traps
or simply unknown creatures moving in the dark. Since the game forces
the player to hunker down every night, the player feels helpless
against this darkness and must accept that it cannot be conquered.</p>
Since sight is unreliable, directional audio cues can alert the player
from nearby enemies or items by their growls or pulse. During the
night, this alertness is heightened where other audio cues such as
footsteps, opening doors or banging induce paranoia and stress until
the harmony of daybreak. The background and ambient music specially
after the night builds the slow and creeping fear and the relief
appropriately.</p>
Aside from traps, the primary threat of this game are the enemies. The
stamina-based combat system of this game is stiff and rigid as the
player needs weapons which cost resources to use and repair as well as
risky with the high damage from enemies, unrewarding loot drop and
specially permadeath. The player must efficiently scavenge materials
for weapons and defense as the night enemies will eventually find the
player and require combat to maintain a safety zone. Enemies do not
get easier and the resources required increase which feed into the
game's resource scarcity. Even if a daily trader NPC</a> trades goods, the
aggression increases over time and so is the average resources
required. By making enemies and combat dangerous and resource
intensive, the player is must always be careful and respect the woods.</p>
Lastly without a tutorial, the game wants the player to begin and
expect the discomfort and uncertainty of the world. Even the
characters including the player's are strange humans physically or
mentally affected by the forest which defy reality and sanity. Their
stories and requests are vague and hide or lack information which
complicates the player's actions and expectations. By the end of the
game, the player may have escaped the forest, but not understood it
and possibly changed for the worst by it. The game does not happy
endings or resolutions and this feeling of ambiguity reaches the
player</p>
A minor point, the skills</a> progression provide interesting powers and
abilities specially at higher levels but requires a negative trait</a> at
each tier level or roughly for every two upgrades which is
interesting. While I like the negative traits threaten survival, they
are somewhat too strong where it might be better not to level up.
Personally in my runs, most active skills can only be activated once a
day that I forgot about them as well as the passive one. The skills
are not badly designed but rather my play style does not use/encourage
skills and perhaps should not have dealt with the side-effects.
Looking back, the level progression is surprisingly optional and
engaging with it provides an interesting and conscious trade-off while
supporting the idea of being mutated by the forest and the challenging
nature of the game itself. After finishing the game, I suggest trying
out new runs without skills or randomly selecting them to spice them
up.</p>
The game overall presents a tense horror gameplay that is well
balanced and designed. It could use some personal improvements such as
inventory sorting, the ability to skip dream sequences, more nightly
events, more skills/traits and so on but it is fine as is. My advice
to new players is begin the game at Nightmare</em> difficulty as well so
that decisions and mistakes have tangible tension or loss. I am happy
to recommend this horror game but remember to respect the forest.</p>


Rust Bit - Folding A Recursive Parser
Unknown — Mon, 18 Jan 2021 00:00:00 +0000
>></a>
Problem</h2>
I want to write a toy git revision parser</a> (without input validation)
which parses the base revision name then zero or more suffixes, ^</code>
and ~(n)</code>, into a recursive data structure</a>:</p>
/// Revision as a recursive struct
</span>#[</span>derive</span>(</span>Clone</span>,</span> Debug</span>,</span> PartialEq</span>)]
</span>pub enum </span>Revision </span>{
</span>    Ref</span>(</span>String</span>),
</span>    Parent</span>(</span>Box</span><</span>Revision</span>>),
</span>    Ancestor</span>(</span>Box</span><</span>Revision</span>>, </span>usize</span>),
</span>}
</span>
</span>fn </span>parse</span>(</span>input</span>: </span>&str</span>) -> </span>Result</span><</span>Revision, </span>&str</span>> {
</span>    </span>todo!</span>()
</span>}
</span>
</span>/// Basic Pattern
</span>assert_eq!</span>(</span>parse</span>(</span>&</span>"</span>master</span>"), </span>Ok</span>(</span>Revision</span>::</span>Ref</span>("</span>master</span>".</span>to_owned</span>())));
</span>assert_eq!</span>(
</span>    parse</span>(</span>&</span>"</span>main^</span>"),
</span>    </span>Ok</span>(</span>Revision</span>::</span>Parent</span>(
</span>        </span>Box</span>::</span>new</span>(</span>Revision</span>::</span>Ref</span>("</span>main</span>".</span>to_owned</span>()))))
</span>);
</span>assert_eq!</span>(
</span>    parse</span>(</span>&</span>"</span>feature/banana-bread~3</span>"),
</span>    </span>Ok</span>(</span>Revision</span>::</span>Ancestor</span>(
</span>        </span>Box</span>::</span>new</span>(</span>Revision</span>::</span>Ref</span>("</span>feature/banana-bread</span>".</span>to_owned</span>())),
</span>        </span>3
</span>    </span>))
</span>);
</span>
</span>/// Nested Pattern
</span>assert_eq!</span>(
</span>    parse</span>(</span>&</span>"</span>develop^^^</span>"),
</span>    </span>Ok</span>(</span>Revision</span>::</span>Parent</span>(
</span>        </span>Box</span>::</span>new</span>(</span>Revision</span>::</span>Parent</span>(
</span>            </span>Box</span>::</span>new</span>(</span>Revision</span>::</span>Parent</span>(
</span>                </span>Box</span>::</span>new</span>(</span>Revision</span>::</span>Ref</span>("</span>develop</span>".</span>to_owned</span>()))))))))
</span>);
</span>assert_eq!</span>(
</span>    parse</span>(</span>&</span>"</span>origin/deployment/1.2.3~5^</span>"),
</span>    </span>Ok</span>(</span>Revision</span>::</span>Parent</span>(
</span>        </span>Box</span>::</span>new</span>(</span>Revision</span>::</span>Ancestor</span>(
</span>            </span>Box</span>::</span>new</span>(</span>Revision</span>::</span>Ref</span>("</span>origin/deployment/1.2.3</span>".</span>to_owned</span>())),
</span>            </span>5
</span>        </span>))))
</span>)
</span></code></pre>
A quick way to parse it recursively with regex</a> would be:</p>
#[</span>macro_use</span>]
</span>extern crate</span> lazy_static</span>; </span>// 1.4.0
</span>
</span>use </span>regex</span>::</span>Regex</span>; </span>// 1.4.2
</span>
</span>fn </span>parse</span>(</span>input</span>: </span>&str</span>) -> </span>Result</span><</span>Revision, </span>&str</span>> {
</span>    </span>lazy_static! </span>{
</span>        </span>static ref </span>PARENT</span>:</span> Regex </span>= </span>Regex</span>::</span>new</span>(</span>r</span>"</span>/^(.+)\^$/</span>").</span>unwrap</span>();
</span>        </span>static ref </span>ANCESTOR</span>:</span> Regex </span>= </span>Regex</span>::</span>new</span>(</span>r</span>"</span>/^(.+)~(\d+)$/</span>").</span>unwrap</span>();
</span>    </span>}
</span>
</span>    </span>if let Some</span>(</span>captures</span>) </span>= </span>PARENT</span>.</span>captures</span>(</span>input</span>) {
</span>        </span>let</span> inner_input </span>=</span> captures</span>.</span>get</span>(</span>1</span>).</span>unwrap</span>().</span>as_str</span>().</span>to_owned</span>();
</span>        </span>let</span> inner_rev </span>=  </span>parse</span>(</span>inner_input</span>).</span>map_err</span>(|</span>_</span>| "</span>Parse error</span>")</span>?</span>;
</span>
</span>        </span>return Ok</span>(</span>Revision</span>::</span>Parent</span>(</span>Box</span>::</span>new</span>(</span>inner_rev</span>)));
</span>    </span>}
</span>
</span>    </span>if let Some</span>(</span>captures</span>) </span>= </span>ANCESTOR</span>.</span>is_match</span>(</span>input</span>) {
</span>        </span>let</span> inner_input </span>=</span> captures</span>.</span>get</span>(</span>1</span>).</span>unwrap</span>().</span>as_str</span>().</span>to_owned</span>();
</span>        </span>let</span> n </span>=</span> captures</span>.</span>get</span>(</span>2</span>).</span>unwrap</span>().</span>as_str</span>().</span>parse</span>::<</span>usize</span>>().</span>map_err</span>(|</span>_</span>| "</span>Parse error</span>")</span>?</span>;
</span>        </span>let</span> inner_rev </span>=  </span>parse</span>(</span>inner_input</span>).</span>map_err</span>(|</span>_</span>| "</span>Parse error</span>")</span>?</span>;
</span>
</span>        </span>return Ok</span>(</span>Revision</span>::</span>Ancestor</span>(</span>Box</span>::</span>new</span>(</span>inner_rev</span>),</span> n</span>));
</span>    </span>}
</span>
</span>    </span>Ok</span>(</span>Revision</span>::</span>Ref</span>(</span>input</span>.</span>to_owned</span>()))
</span>}
</span></code></pre>
However, this inefficiently parses the whole text for each suffix or
nesting. Using the parser combinator nom</a> instead, parsing can be
better expressed and extended:</p>
NOTE</strong></strong>: While this is not a tutorial of nom</code> which you should check
out, this is about a pitfall/trap with a recursive design. Do pay more
to the general idea than the actual code.</p>
use </span>nom</span>::{ </span>// 5.1.1
</span>    </span>branch</span>::</span>alt</span>,
</span>    </span>bytes</span>::</span>complete</span>::{</span>tag</span>,</span> take_till</span>},
</span>    </span>character</span>::</span>complete</span>::</span>digit1</span>,
</span>    </span>combinator</span>::{</span>all_consuming</span>,</span> map</span>,</span> map_res</span>,</span> rest</span>},
</span>    </span>sequence</span>::</span>preceded</span>,
</span>    IResult
</span>};
</span>
</span>/// Parse base REF_NAME
</span>fn </span>parse_ref</span>(</span>i</span>: </span>&str</span>) -> </span>IResult</span><</span>&str</span>, Revision</span>> {
</span>    </span>let</span> ref_name </span>= </span>map</span>(
</span>        take_till</span>(|</span>b</span>| </span>b </span>== </span>'</span>~</span>' </span>||</span> b </span>== </span>'</span>^</span>'),
</span>        </span>|</span>ref_name</span>: </span>&str</span>| </span>ref_name</span>.</span>to_owned</span>(),
</span>    </span>);
</span>
</span>    </span>let </span>(</span>i</span>,</span> name</span>) </span>= </span>ref_name</span>(</span>i</span>)</span>?</span>;
</span>
</span>    </span>Ok</span>((</span>i</span>, </span>Revision</span>::</span>Ref</span>(</span>name</span>)))
</span>}
</span>
</span>/// Parse carret suffix REV^
</span>fn </span>parse_caret</span><</span>'a</span>>(</span>inner_rev</span>:</span> Revision</span>) -></span> impl </span>'a + Fn</span>(</span>&'a str</span>) -> </span>IResult</span><</span>&'a str</span>, Revision</span>> {
</span>    </span>let</span> suffix </span>= </span>tag</span>("</span>^</span>");
</span>
</span>    </span>move |</span>i</span>: </span>&str| </span>{
</span>        </span>let </span>(</span>i</span>, </span>_</span>) </span>= </span>suffix</span>(</span>i</span>)</span>?</span>;
</span>
</span>        </span>Ok</span>((</span>i</span>, </span>Revision</span>::</span>Parent</span>(</span>Box</span>::</span>new</span>(</span>inner_rev</span>.</span>clone</span>()))))
</span>    </span>}
</span>}
</span>
</span>/// Parse tilde suffix REV~n
</span>fn </span>parse_tilde</span><</span>'a</span>>(</span>inner_rev</span>:</span> Revision</span>) -></span> impl </span>'a + Fn</span>(</span>&'a str</span>) -> </span>IResult</span><</span>&'a str</span>, Revision</span>> {
</span>    </span>let</span> number </span>= </span>map_res</span>(</span>digit1</span>, |</span>digits</span>: </span>&str</span>| </span>digits</span>.</span>parse</span>::<</span>usize</span>>());
</span>    </span>let</span> suffix </span>= </span>preceded</span>(</span>tag</span>("</span>~</span>"),</span> number</span>);
</span>
</span>    </span>move |</span>i</span>: </span>&str| </span>{
</span>        </span>let </span>(</span>i</span>,</span> n</span>) </span>= </span>suffix</span>(</span>i</span>)</span>?</span>;
</span>
</span>        </span>Ok</span>((</span>i</span>, </span>Revision</span>::</span>Ancestor</span>(</span>Box</span>::</span>new</span>(</span>inner_rev</span>.</span>clone</span>()),</span> n</span>)))
</span>    </span>}
</span>}
</span>
</span>/// Parse suffixes recursively
</span>fn </span>parse_suffix</span><</span>'a</span>>(</span>rev</span>:</span> Revision</span>) -></span> impl </span>'a + Fn</span>(</span>&'a str</span>) -> </span>IResult</span><</span>&'a str</span>, Revision</span>> {
</span>    </span>let</span> suffix </span>= </span>alt</span>((
</span>        </span>/// Check for REV^ suffix
</span>        parse_caret</span>(</span>rev</span>.</span>clone</span>()),
</span>        </span>/// Check for REV~n suffix
</span>        parse_tilde</span>(</span>rev</span>.</span>clone</span>()),
</span>        </span>/// If neither and the whole input is consumed, return the current revision
</span>        map</span>(</span>all_consuming</span>(</span>rest</span>), </span>move |</span>_i</span>: </span>&str|</span> rev</span>.</span>clone</span>())
</span>    </span>));
</span>
</span>    </span>move |</span>i</span>: </span>&str| </span>{
</span>        </span>let </span>(</span>i</span>,</span> next_rev</span>) </span>= </span>suffix</span>(</span>i</span>)</span>?</span>;
</span>
</span>        </span>let </span>(</span>i</span>,</span> new_rev</span>) </span>= if</span> i</span>.</span>is_empty</span>() {
</span>            </span>/// If nothing else to parse, return the current result
</span>            </span>(</span>i</span>,</span> next_rev</span>)
</span>        </span>} </span>else </span>{
</span>            </span>/// Otherwise, parse recursively
</span>            parse_suffix</span>(</span>next_rev</span>)(</span>i</span>)</span>?
</span>        </span>};
</span>
</span>        </span>Ok</span>((</span>i</span>,</span> new_rev</span>))
</span>    </span>}
</span>}
</span>
</span>/// Main parsing function
</span>fn </span>parse</span>(</span>i</span>: </span>&str</span>) -> </span>IResult</span><</span>&str</span>, Revision</span>> {
</span>    </span>/// Parse the bottommost revision
</span>    </span>let </span>(</span>i</span>,</span> base_rev</span>) </span>= </span>parse_ref</span>(</span>i</span>)</span>?</span>;
</span>    </span>/// Build the revision recursively
</span>    </span>let </span>(</span>i</span>,</span> rev</span>) </span>= </span>parse_suffix</span>(</span>base_rev</span>)(</span>i</span>)</span>?</span>;
</span>
</span>    </span>Ok</span>((</span>i</span>,</span> rev</span>))
</span>}
</span></code></pre>
The equivalent recursive parser reads the input once and builds the
revision bottom-up; however, the suffix parser (parse_suffix</code>) clones
the current revision too much (3 times) for each test. While it can be
refactored, recursion is usually harder to optimize and satisfy with
the borrow checker.</p>
Recursive Source Code</a></p>
>></a>
Solution</h2>
Instead of using recursion, an iterative approach can perform better.
Rather than immediately building the revision at each recursion or
iteration, the newly parsed parent revision can have an empty child
revision which can be replaced later and makes each parser cleaner
without requiring ownership or reference to the actual child revision:</p>
/// Create a default for `Revision`
</span>impl </span>Default </span>for </span>Revision </span>{
</span>    </span>fn </span>default</span>() -> </span>Self </span>{
</span>        </span>Self</span>::</span>Ref</span>(</span>String</span>::</span>default</span>())
</span>    </span>}
</span>}
</span>
</span>/// Without the child revision argument, the parsing is simpler
</span>fn </span>parse_caret</span>(</span>i</span>: </span>&str</span>) -> </span>IResult</span><</span>&str</span>, Revision</span>> {
</span>    </span>let</span> suffix </span>= </span>tag</span>("</span>^</span>");
</span>
</span>    </span>let </span>(</span>i</span>, </span>_</span>) </span>= </span>suffix</span>(</span>i</span>)</span>?</span>;
</span>
</span>    </span>/// Create empty revision to be replaced later
</span>    </span>let</span> empty_rev </span>= Box</span>::</span>new</span>(</span>Revision</span>::</span>default</span>());
</span>    </span>Ok</span>((</span>i</span>, </span>Revision</span>::</span>Parent</span>(</span>empty_rev</span>)))
</span>}
</span>
</span>/// Ditto
</span>fn </span>parse_tilde</span>(</span>i</span>: </span>&str</span>) -> </span>IResult</span><</span>&str</span>, Revision</span>> {
</span>    </span>let</span> number </span>= </span>map_res</span>(</span>digit1</span>, |</span>digits</span>: </span>&str</span>| </span>digits</span>.</span>parse</span>::<</span>usize</span>>());
</span>    </span>let</span> suffix </span>= </span>preceded</span>(</span>tag</span>("</span>~</span>"),</span> number</span>);
</span>
</span>    </span>let </span>(</span>i</span>,</span> n</span>) </span>= </span>suffix</span>(</span>i</span>)</span>?</span>;
</span>
</span>    </span>let</span> empty_rev </span>= Box</span>::</span>new</span>(</span>Revision</span>::</span>default</span>());
</span>    </span>Ok</span>((</span>i</span>, </span>Revision</span>::</span>Ancestor</span>(</span>empty_rev</span>,</span> n</span>)))
</span>}
</span></code></pre>
Refactoring the main suffix parser into an iterative loop:</p>
use </span>std</span>::</span>mem</span>;
</span>
</span>fn </span>parse_suffix</span>(</span>mut </span>rev</span>:</span> Revision, </span>mut </span>i</span>: </span>&str</span>) -> </span>IResult</span><</span>&str</span>, Revision</span>> {
</span>    </span>/// The branch parser is simpler as well
</span>    </span>let</span> suffix </span>= </span>alt</span>((
</span>        parse_caret</span>,
</span>        parse_tilde</span>,
</span>    </span>));
</span>
</span>    </span>/// Until the whole input is parsed/consumed
</span>    </span>while !</span>i</span>.</span>is_empty</span>() {
</span>        </span>/// Parse the next parent revision
</span>        </span>let </span>(</span>i2</span>, </span>mut</span> acc_rev</span>) </span>= </span>suffix</span>(</span>i</span>)</span>?</span>;
</span>
</span>        </span>/// To replace the parent's empty child revision, the child is pattern matched
</span>        </span>let</span> boxed_rev </span>= match &mut</span> acc_rev </span>{
</span>            </span>Revision</span>::</span>Ref</span>(</span>_</span>) </span>=> </span>unreachable!</span>(),
</span>            </span>Revision</span>::</span>Parent</span>(</span>ref mut</span> boxed_rev</span>) </span>=></span> boxed_rev</span>,
</span>            </span>Revision</span>::</span>Ancestor</span>(</span>ref mut</span> boxed_rev</span>, </span>..</span>) </span>=></span> boxed_rev</span>,
</span>        </span>};
</span>
</span>        </span>/// Then replaced via `std::mem::replace`
</span>        </span>let _ = </span>mem</span>::</span>replace</span>(</span>&mut **</span>boxed_rev</span>,</span> rev</span>);
</span>
</span>        </span>// Finally mark this parent as the current revision
</span>        rev </span>=</span> acc_rev</span>;
</span>        i </span>=</span> i2</span>;
</span>    </span>}
</span>
</span>    </span>Ok</span>((</span>i</span>,</span> rev</span>))
</span>}
</span></code></pre>
While recursion is natural to a recursive/nested structure, also
consider an iterative approach which may be overall simpler and just
as efficient.</p>
Iterative Source Code</a></p>
>></a>
Notes</h2>
>>></a>
Using fold</h3>
Instead of a loop, it can be directly converted to a more succinct
fold with fold_many0</a>:</p>
use </span>nom</span>::</span>multi</span>::</span>fold_many0</span>;
</span>
</span>fn </span>parse_suffix</span>(</span>rev</span>:</span> Revision, </span>i</span>: </span>&str</span>) -> </span>IResult</span><</span>&str</span>, Revision</span>> {
</span>    </span>let</span> suffix </span>= </span>alt</span>((</span>parse_caret</span>,</span> parse_tilde</span>,));
</span>
</span>    </span>/// Apply the `suffix` parser 0 or more times with the same logic
</span>    </span>let </span>(</span>i</span>,</span> rev</span>) </span>= </span>all_consuming</span>(</span>fold_many0</span>(</span>suffix</span>,</span> rev</span>, |</span>acc_rev</span>, </span>mut </span>rev</span>| {
</span>        </span>let</span> boxed_rev </span>= match &mut</span> rev </span>{
</span>            </span>Revision</span>::</span>Ref</span>(</span>_</span>) </span>=> </span>unreachable!</span>(),
</span>            </span>Revision</span>::</span>Parent</span>(</span>ref mut</span> boxed_rev</span>) </span>=></span> boxed_rev</span>,
</span>            </span>Revision</span>::</span>Ancestor</span>(</span>ref mut</span> boxed_rev</span>, </span>..</span>) </span>=></span> boxed_rev</span>,
</span>        </span>};
</span>
</span>        </span>let _ = </span>mem</span>::</span>replace</span>(</span>&mut **</span>boxed_rev</span>,</span> acc_rev</span>);
</span>
</span>        rev
</span>    </span>}))(</span>i</span>)</span>?</span>;
</span>
</span>    </span>Ok</span>((</span>i</span>,</span> rev</span>))
</span>}
</span></code></pre>
The only caveat with this combinator alongside alt</a> is that it clones
the input after each iteration which contradicts the original goal.
In practice this is not detrimental but do check and be aware if the
more complex function are bottlenecks. Rewriting them as a simpler
function always works if needed:</p>
use </span>nom</span>::</span>error</span>::</span>ErrorKind</span>;
</span>
</span>/// Instead of a parser combinator, it can be a sequence of if-let returns
</span>/// A little more verbose but it avoids cloning the input
</span>fn </span>suffix</span>(</span>i</span>: </span>&str</span>) -> </span>IResult</span><</span>&str</span>, Revision</span>> {
</span>    </span>if let Ok</span>((</span>i</span>,</span> rev</span>)) </span>= </span>parse_caret</span>(</span>&</span>i</span>) {
</span>        </span>return Ok</span>((</span>i</span>,</span> rev</span>));
</span>    </span>}
</span>
</span>    </span>if let Ok</span>((</span>i</span>,</span> rev</span>)) </span>= </span>parse_tilde</span>(</span>&</span>i</span>) {
</span>        </span>return Ok</span>((</span>i</span>,</span> rev</span>));
</span>    </span>}
</span>
</span>    </span>/// Construct nom error manually
</span>    </span>Err</span>(</span>nom</span>::</span>Err</span>::</span>Error</span>((</span>i</span>, </span>ErrorKind</span>::</span>Alt</span>)))
</span>}
</span></code></pre>


Not Tonight - Lacking Edge
Unknown — Fri, 08 Jan 2021 00:00:00 +0000
>></a>
Not Tonight</h2>
Finished main content on Switch</strong></strong></p>
Not Tonight</a> is an error detection game playing as a discriminated
immigrant bouncer in a post-Brexit</a> scenario starting from checking for
fake identification cards to tickets, visas and credit score. While it
may sound mundane, it requires concentration, snap judgment,
confidence and intuition to survive which provides a form of
mechanical joy and flow. The game is played over three months each
with around 25 working days and finished within 10 hours.</p>
>></a>
Thoughts</h2>
My primary issue of the game is its lack of difficulty. Although
difficulty is subjective, the game does not push enough to force the
player to engage in corrupt activities such as drug dealing or
bribery. Specifically, venues offer double pay after finishing at
least thrice providing a wide safety margin despite being less
forgiving. Since the player is encouraged to repeat this in many
venues, the difficulty curve quickly becomes flat and money easy.
Keeping difficulty the same, the game should remove double pay and
lower the pay overall. The only interesting mechanical variation is
intentionally making mistakes for the resistance which provided
strategic and tactical depth that the game could use more. Perhaps a
venue or night fails if a certain number of people are allowed, a
rowdy person might clear a line or waste more time, a dependent you
need to take care of and so on. The game could use more flourish and
difficulty to further push its point. In the end, I was living
comfortably with ten thousand euros and fully upgraded house which
slightly ruined immersion.</p>
Another issue is the story and pacing. Although the game provides
multiple venues with different condition sets to shake things up,
majority of the story objectives are to acquire venue specific items
by successfully finishing them thrice that feels draining and
repetitive. Those story rewards feel ambiguous and aimless yet the
optional objectives with its smaller character moments despite not
having tangible rewards feel like small personal victories which shows
the story missions can be better designed and written. The story is
framed around the opening incident that seems unnecessary and shown by
an discontinuous countdown and nothing conclusive or interesting
afterwards. If the story and pacing before and after the incident is
weak, perhaps the game should lean on smaller plots of hardship and
empathy instead of a hollow resistance plot.</p>
Despite my issues and some other minor issues and bugs, I enjoyed the
game as it is but it fizzled out past the difficulty spikes. Whether
the writing or satire hits the mark, it sadly does not feel meaningful
to play but I do appreciate the exploration of being abandoned by the
country. I would like to see this genre flourish and perhaps not
necessarily in a dystopia.</p>


Children of Morta Switch - A Subpar Experience
Unknown — Wed, 30 Dec 2020 00:00:00 +0000
>></a>
Children of Morta</h2>
Played with two players at Normal difficulty until Area 30 Boss on
Switch</strong></strong></p>
Children of Morta</a> is an pixel art hack and slash</a> dungeon crawler</a> with
rogue-lite</a> elements that tells the story of the Bergsons</a> family
struggle and sacrifice against the corruption</a> of the mountain god.
While I appreciate not worrying about loot and builds, the overall
execution felt flat specially in the long term which made me drop the
game.</p>
>></a>
Thoughts</h2>
My biggest issue is that the game is mechanically shallow and flat.
While the skills and characters are functional and varied, it does not
offer modifiers to offer complexity or depth. Without any meaningful
options or choices, the roughly 3 active skills quickly become
repetitive and boring. Since the game has skill modifiers as item
pickups via runes</a>, turning them into options with might be better.
Also turning one or both active items or divine relics</a> slots into
active skills slots instead can increase depth. The skill progression
only offers one tree or build so making it available at the start and
provide a different form of progression such as new skill modifiers or
items. Simple combat is not necessarily bad but the other half of the
equation is not good as well: enemies.</p>
As primarily a ranged player, combat primarily involves pulling the
enemies and slowly trimming the herd while deftly dodging attacks.
While this playstyle is not bad, the enemy variety and complexity does
not meaningfully challenge this model that it contributes to the
boredom. Likewise, the generated levels themselves are frequently long
hallways that do not offer much in tactics, strategy or positioning.
Lastly, the dungeon bosses are decent but not challenging or
memorable. Alongside the lack of skill complexity, these factors make
dungeon crawling not as enjoyable.</p>
Since mechanical depth and enemy combat are flat, skill becomes less
important while statistic more so. Even if I prefer skill be rewarded
more than time, the game's execution on both is underwhelming. While I
do have minor issues with the narration, audio and presentation, I
still respect the game's attempt and would like to see a more polished
sequel.</p>


Katawa Shoujo - A Visual Novel Gem
Unknown — Tue, 29 Dec 2020 00:00:00 +0000
>></a>
Katawa Shoujo</h2>
Finished every route with a good ending on PC</strong></strong></p>
Katawa Shoujo</a> is a bishoujo</a> or dating simulator</a> visual novel game
where typically an attractive boy (the player character)</a> interacting
with several girls and ultimately be in a relationship with one of
them. The twist is that these girls (and main character) are disabled</a>
that makes the player rethink their involvement or commitment.
Thankfully, the writing does not use disability</em> as a shield or ploy
and uses it to foster understanding and equality. The game can be
downloaded for free and consists of 5 routes each with 4 chapters
which took roughly 1 hour each or around 20 hours in total.</p>
>></a>
Thoughts</h2>
Since this is my first foray into visual novels, I can only
meaningfully judge the experience, so the quality of graphics, design
and options are beyond me. As a dating simulator</em>, judging the routes
is quite subjective and personal as people have their own priorities
and experiences; my criteria is not character development, romantic
compatibility or character personality but the writing, experience and
impact. Among the possible routes, Rin's route</a> is the best written in
showing the effort, anxiety and uncertainty of falling in love with
her because of her communication issues and not her limbs. Hanako's
route</a> tackles disability</em> the best since it requires perceiving her
past her trauma and appearance much more so than Emi's route</a>. Sadly,
Lily's route</a> has little tension and Shizune's route</a> has pacing issues.
Nonetheless, the routes are good overall and show people are not
defined by their disability but are people still.</p>
For the expected sex scenes, I can only judge them for their impact
and weight. I like Rin</em>'s and Hanako</em>'s scenes since both
meaningfully progress the relationship; however, the scenes of the
other characters are less interesting because it happens multiple
times and of libido. Personally, this aspect of dating simulators</em> is
hard to discuss but perhaps I wanted this to be handled more maturely
since it tackles disability</em>. This is not a big issue since
objectification and intimacy requires empathy by virtue of the whole
cast having disabilities</em>.</p>
My only true criticism of the game is Kenji</a> who does not contribute
much to the story or any route and should be removed or replaced to
improve overall pacing. I would also appreciate more dialogue choices
(not necessarily branches) to increase interactivity and provide
breaks from lengthy text sequences. Other than that, this visual novel
is a rare gem and worth your attention.</p>


No More Heroes 1 And 2 - A Good And Bad Game
Unknown — Thu, 03 Dec 2020 00:00:00 +0000
>></a>
No More Heroes 1 & 2</h2>
I have heard about this series but have not played any of it. I did
enjoy the writing and story of Travis Strikes Again: No More Heroes</a>
and with the highly anticipated No More Heroes 3</a>, I needed to play the
games in the series before then. Overall, the first game was good but
the second game was bad even without continuity. Either way, I do hope
they take all this experience as I want to experience this universe.</p>
>></a>
A Good Game</h2>
Finished at Normal/Mild difficulty on Switch</strong></strong></p>
No More Heroes 1</a> is a charming game despite its lack of quality. Boss
characters/fights are intriguing with their atmosphere and writing.
Those are paced by the the city sandbox with its mundane jobs/mini
games and assassination gigs/side missions all adding to the overall
charm and aesthetic as well despite the repetitive travel. The combat
system is just right with its movement and feel and the bosses are
balanced nicely with it providing a good challenge. Aside from the
interesting main character and story subversion, everything added up
to make this a game worth playing.</p>
My only hope for the sequel is to expand the combat system, provide
quality of life options for the sandbox and overall polish. Sadly,
this would not be the case.</p>
>></a>
A Bad Game</h2>
Finished at Normal/Mild difficulty on Switch</strong></strong></p>
After finishing the first game and a rest day, the sequel No More
Heroes 2: A Desperate Struggle</a> feels bad to play even without knowing
the prequel. The worst issue is that the bosses are far more broken
with stun locks specially the last boss and easier to exploit with the
more fluid character movements which makes them less enjoyable as a
whole. Levels are primarily room after room mob spawns which makes
them less interesting. Enemy variety is lacking and their higher
health makes the fight more drawn out. Lastly is strafing</a> where it
feels like the camera is only locked and the character still moves as
if it was not which is disorienting since many of the fights are
duels. Although more characters and weapons have been introduced, the
new changes made the combat system clunkier and worse.</p>
Instead of an city sandbox, a stage/mission select replaced it. While
the travel was repetitive, I did not want it removed since it adds to
the charm where both could have coexisted as quality of life. While
the side jobs are a mixed bag, the kill side missions/gigs are
repetitive simply being kill one or all types and made worse from the
combat system. This makes the game seems to have less content or
weight from a surface level.</p>
Aside from the game breaking continuity, the game's character writing
and story is lacking. Primarily, the bosses lost their strong
introductions and meaningful banter which I craved for in between
ranking fights. While the 50 person ranking system may be intended as
a facade with multiple jumps and illogical rules, the title has less
weight or impact. Subverting the revenge story is a good idea but its
execution was lacking with its lackluster experience. Much of the
charm and appeal of the first game is lost from the writing alone.</p>
While I have other grievances, the sequel was bad even though it felt
more fluid or expanded. If they added more quality of life changes
from the previous, it would make this game possibly better to play
even if the story was not. Taking continuity into account makes it
worse as the character development and hanging question were
disregarded for this disappointing setup.</p>


Rit Journal - Diff Command
Unknown — Tue, 01 Dec 2020 00:00:00 +0000
>></a>
Chapter 12: Diff Command</h2>
This chapter is about integrating the myers::diff</code> by making the diff</a>
command. Writing the new command is straightforward but hunking the
diffs and integrating a pager was challenging specially to optimize.</p>
Source:</p>

core/diff.rs</a></li>
cli/diff.rs</a></li>
pager.s</a></li>
</ul>
>>></a>
Abstracting Commands</h3>
Since the diff</code> and status</code> command render the same data
differently, the book suggests to refactor them into an abstract base
class. While Rust</em> does not have abstract classes, I used cargo
workspaces</a> to share the core library crate and reuse them in the CLI</a>
binary crate which is the same effect. Even in Elixir,</a> I usually
structure my applications like so:</p>
[</span>workspace</span>]
</span>members </span>= [
</span>  </span>"</span>core</span>",
</span>  </span>"</span>cli</span>"
</span>]
</span></code></pre>
This separation allows me to test easier with modular interfaces or
layers. Also if more interfaces are needed such as a GUI or C
library, it is easy to plug it in. It also helps with compilation
time since modified crates should only be compiled. Not necessarily
an abstract classes equivalent but I appreciate how workspaces made
this natural.</p>
Personally, I prefer traits or object composition over abstract
classes since they are safer and easier to maintain. One difficulty
with abstract classes is that when it is changed, its concrete
classes also change which may lead to incorrect or undefined
behavior. Another is allowing object hierarchy that increase learning
complexity than multiple smaller traits. The closest thing in Rust</em>
is that trait methods can have default implementation which I did for
Blob</code>, Tree</code> and Commit</code>:</p>
/// Trait to serialize/deserialize to the correct git storage formats
</span>pub trait </span>ObjectData </span>{
</span>    </span>fn </span>kind</span>() -> </span>&'static str</span>;
</span>
</span>    </span>fn </span>content</span>(</span>&</span>self</span>) -> </span>Vec</span><</span>u8</span>>;
</span>
</span>    </span>fn </span>object_content</span>(</span>&</span>self</span>) -> </span>Vec</span><</span>u8</span>> {
</span>        </span>let</span> content </span>= self</span>.</span>content</span>();
</span>
</span>        </span>let mut</span> buf </span>= Vec</span>::</span>with_capacity</span>(</span>100</span>);
</span>
</span>        buf</span>.</span>put</span>(</span>format!</span>("</span>{} {}</span>\0</span>", </span>Self</span>::</span>kind</span>(),</span> content</span>.</span>len</span>()).</span>as_bytes</span>());
</span>        buf</span>.</span>put</span>(</span>content</span>.</span>as_slice</span>());
</span>
</span>        buf
</span>    </span>}
</span>}
</span>
</span>impl </span>ObjectData </span>for </span>Blob </span>{
</span>    </span>fn </span>kind</span>() -> </span>&'static str </span>{
</span>        </span>"</span>blob</span>"
</span>    </span>}
</span>
</span>    </span>fn </span>content</span>(</span>&</span>self</span>) -> </span>Vec</span><</span>u8</span>> {
</span>        </span>self</span>.</span>bytes</span>.</span>clone</span>()
</span>    </span>}
</span>}
</span>
</span>/// Import the trait to use its methods
</span>use </span>core</span>::</span>ObjectData</span>;
</span>
</span>let</span> blob </span>= </span>Blob</span>::</span>new</span>(</span>Vec</span>::</span>new</span>());
</span>blob</span>.</span>object_content</span>();
</span></code></pre>
Implementing the required trait methods and having the optional
methods be interface methods feel better. Rust</em> also has supertraits</a>
which is the closest to inheritance but not needed here. Perhaps
abstract classes are in conflict with Rust</em>'s zero-cost abstraction?
Either way, it is a minor thing but I do appreciate Rust</em> more on
this front.</p>
>>></a>
Hunks</h3>
The hardest task is to filter and group a list of diffs to highlight
the changes. Ideally, it looks like this:</p>
#[</span>derive</span>(</span>Debug</span>)]
</span>struct </span>Hunk</span><</span>'a</span>, T</span>>(</span>Vec</span><</span>Diff</span><</span>'a</span>, T</span>>>);
</span>
</span>impl</span><</span>'a</span>, T</span>> </span>Hunk </span>{
</span>    </span>fn </span>diff_hunks</span>(</span>diffs</span>: </span>Vec</span><</span>Diff</span><</span>'a</span>, T</span>>>) -> </span>Vec</span><</span>Hunk</span><</span>'a</span>, T</span>>> {
</span>        </span>unimplemented!</span>();
</span>    </span>}
</span>}
</span>
</span>let</span> left_lines </span>=</span> left_text</span>.</span>lines</span>().</span>collect</span>::<</span>Vec</span><</span>_</span>>>();
</span>let</span> right_lines </span>=</span> right_text</span>.</span>lines</span>().</span>collect</span>::<</span>Vec</span><</span>_</span>>>();
</span>
</span>let</span> diffs </span>= </span>diff</span>::</span>myers</span>(</span>left_lines</span>.</span>as_slice</span>(),</span> right_lines</span>.</span>as_slice</span>());
</span>for</span> hunk </span>in </span>Hunk</span>::</span>diff_hunks</span>(</span>&</span>diffs</span>) {
</span>    </span>for</span> diff </span>in</span> hunk</span>.</span>0 </span>{
</span>        </span>info!</span>("</span>{}{}</span>",</span> diff</span>.</span>symbol</span>(),</span> diff</span>.</span>item</span>());
</span>    </span>}
</span>}
</span></code></pre>
@@ -9,7 +9,7 @@
</span> const PAGER_CMD: &'static str = "less";
</span>
</span> pub struct Pager {
</span>-    cmd: String,
</span>+    cmd: Option<String>,
</span> }
</span>
</span> impl Pager {
</span></code></pre>
The challenge comes from grouping the diffs into hunks. Initially, I
thought to group by</a> the diffs into a change and non-change group,
filter out the non-change groups and store everything into a vector:</p>
use </span>itertools</span>::</span>Itertools</span>;
</span>
</span>let</span> hunks </span>=</span> diffs</span>.</span>into_iter</span>()
</span>    </span>.</span>group_by</span>(|</span>diff</span>| </span>diff</span>.</span>is_same</span>())
</span>    </span>.</span>filter</span>(|(</span>is_same</span>, </span>_group</span>)| </span>!</span>is_same</span>)
</span>    </span>.</span>map</span>(|(</span>_</span>, </span>group</span>)| </span>group</span>.</span>collect</span>::<</span>Vec</span><</span>_</span>>>())
</span>    </span>.</span>collect</span>::<</span>Vec</span><</span>_</span>>>();
</span></code></pre>
The tricky thing is that hunks have leading and trailing padding
diffs (3 Diff::Same</code> above and below) to further show where the
hunks are. Furthermore, when two hunks are separated by less than the
padding amount, they are merged together. The following snippet is my
version of the chunking algorithm:</p>
fn </span>group_hunks</span><</span>'b</span>, T</span>>(</span>xs</span>: </span>&'b Vec</span><</span>T</span>></span>, </span>is_change</span>:</span> impl Fn</span>(</span>&</span>T</span>)</span> -> </span>bool</span>, </span>padding</span>: </span>usize</span>) -> </span>Vec</span><</span>&'b </span>[</span>T</span>]> {
</span>    </span>let mut</span> hunks </span>= Vec</span>::</span>new</span>();
</span>
</span>    </span>let mut</span> offset </span>= </span>0</span>;
</span>
</span>    </span>loop </span>{
</span>        </span>/// First Phase: Find the first/next element that is a change
</span>        </span>while let Some</span>(</span>x</span>) </span>=</span> xs</span>.</span>get</span>(</span>offset</span>) {
</span>            </span>if </span>is_change</span>(</span>x</span>) {
</span>                </span>break</span>;
</span>            </span>}
</span>
</span>            offset </span>+= </span>1</span>;
</span>        </span>}
</span>
</span>        </span>if</span> offset </span>>=</span> xs</span>.</span>len</span>() {
</span>            </span>break</span>;
</span>        </span>}
</span>
</span>        </span>/// Second Phase: Find the end of the hunk with padding
</span>
</span>        </span>/// The start of the hunk is the current minus the padding
</span>        </span>let</span> hunk_start </span>=</span> offset</span>.</span>checked_sub</span>(</span>padding</span>).</span>unwrap_or_default</span>();
</span>
</span>        </span>/// This variables will end at the end of the hunk
</span>        </span>let mut</span> counter </span>=</span> padding</span>;
</span>        offset </span>+= </span>1</span>;
</span>
</span>        </span>while</span> counter </span>> </span>0 </span>{
</span>            </span>if</span> offset </span>>=</span> xs</span>.</span>len</span>() {
</span>                </span>break</span>;
</span>            </span>}
</span>
</span>            </span>let</span> x </span>=</span> xs</span>.</span>get</span>(</span>offset</span>).</span>unwrap</span>();
</span>
</span>            </span>/// If the current diff is a change, refresh the counter.
</span>            </span>/// Otherwise, decrement it.
</span>            </span>if </span>is_change</span>(</span>x</span>) {
</span>                counter </span>=</span> padding</span>;
</span>            </span>} </span>else </span>{
</span>                counter </span>-= </span>1</span>;
</span>            </span>}
</span>
</span>            offset </span>+= </span>1</span>;
</span>        </span>}
</span>
</span>        </span>/// With start and end index, the hunk can be added
</span>        hunks</span>.</span>push</span>(</span>&</span>xs</span>[</span>hunk_start</span>..</span>offset</span>]);
</span>
</span>        offset </span>+= </span>1</span>;
</span>    </span>}
</span>
</span>    hunks
</span>}
</span>
</span>for</span> diff_slice </span>in </span>group_hunks</span>(</span>&</span>diffs</span>, |</span>diff</span>| </span>!</span>diff</span>.</span>is_same</span>(), </span>3</span>) {
</span>    </span>let</span> hunk </span>=</span> Hunk</span>(</span>diff_slice</span>);
</span>
</span>    </span>for</span> diff </span>in</span> hunk</span>.</span>0 </span>{
</span>        </span>info!</span>("</span>{}{}</span>",</span> diff</span>.</span>symbol</span>(),</span> diff</span>.</span>item</span>());
</span>    </span>}
</span>}
</span></code></pre>
I am proud of this since it only needs to allocate for the new list
compared to the book which makes me appreciate how Rust</em> encourages
optimization because of its type system.</p>
>>></a>
Pager</h3>
Since the output of this command is quite long, the book suggests
integrating a terminal pager</a> which allows the output to be scrolled
via less</a> or some other pager command. The main idea is to spawn a
process via std::process::Command::spawn</a> and pipe the output of the
command to the input of this child process instead:</p>
use </span>std</span>::</span>process</span>::{</span>Command</span>,</span> Stdio</span>};
</span>
</span>let mut</span> pager_child </span>= </span>Command</span>::</span>new</span>("</span>less</span>")
</span>    </span>.</span>env</span>("</span>LESS</span>", "</span>FRV</span>")             </span>/// Add default LESS options
</span>    </span>.</span>stdin</span>(</span>Stdio</span>::</span>piped</span>())          </span>/// Enable child stdin
</span>    </span>.</span>stdout</span>(</span>Stdio</span>::</span>inherit</span>())       </span>/// Share parent stdout to child
</span>    </span>.</span>spawn</span>()
</span>    </span>.</span>expect</span>("</span>Pager spawn failed</span>");
</span>
</span>let</span> output </span>= </span>"</span>HELLO WORLD</span>";
</span>
</span>let</span> stdin </span>=</span> pager_child</span>.</span>stdin</span>.</span>as_mut</span>().</span>unwrap</span>();
</span>stdin</span>.</span>write_all</span>(</span>output</span>.</span>as_bytes</span>()).</span>expect</span>("</span>Failed to write</span>");
</span>stdin</span>.</span>flush</span>().</span>expect</span>("</span>Failed to flush</span>");
</span>
</span>/// Instead of just println!(output) or info!(output);
</span>
</span>pager_child</span>.</span>wait</span>().</span>expect</span>("</span>Failed to close</span>");  </span>/// Wait for the subprocess to close
</span></code></pre>
Although Rust</em> has the Drop</code> trait, it is not implemented for Child</a>
so Child::wait</a> is required at the ends of its lifetime to avoid
zombie processes</a>. Since I use log</a> (and loggerv</a>), the challenge comes
from redirecting the output to this child's stdin</code>. Instead of
replacing every instance of info!</code> in this command, I can create a
custom Log</a> to redirect the output instead:</p>
use </span>log</span>::{</span>Log</span>,</span> LevelFilter</span>,</span> Metadata</span>,</span> Record</span>};
</span>use </span>std</span>::{</span>io</span>::</span>Write</span>, </span>process</span>::</span>Child</span>, </span>sync</span>::{</span>Arc</span>,</span> Mutex</span>}};
</span>
</span>/// Arc + Mutex since static lifetime and interior mutability is needed
</span>pub fn </span>setup_piped_logger</span>(</span>child_rc</span>: </span>Arc</span><</span>Mutex</span><</span>Child</span>>>) {
</span>    </span>struct </span>ChildLog</span>(</span>Arc</span><</span>Mutex</span><</span>Child</span>>>);
</span>
</span>    </span>/// Custom logger redirecting to child process
</span>    </span>impl </span>Log </span>for </span>ChildLog </span>{
</span>        </span>fn </span>enabled</span>(</span>&</span>self</span>, </span>_metadata</span>: </span>&</span>Metadata</span>) -> </span>bool </span>{
</span>            </span>true
</span>        </span>}
</span>
</span>        </span>fn </span>log</span>(</span>&</span>self</span>, </span>record</span>: </span>&</span>Record</span>) {
</span>            </span>let mut</span> child </span>= self</span>.</span>0.</span>lock</span>().</span>expect</span>("</span>Process failed</span>");
</span>            </span>let</span> stdin </span>= </span>(</span>*</span>child</span>).</span>stdin</span>.</span>as_mut</span>().</span>expect</span>("</span>Process failed</span>");
</span>
</span>            stdin
</span>                </span>.</span>write_fmt</span>(</span>*</span>record</span>.</span>args</span>())
</span>                </span>.</span>expect</span>("</span>Failed to write to stdin</span>");
</span>            stdin
</span>                </span>.</span>write</span>("</span>\n</span>".</span>as_bytes</span>())
</span>                </span>.</span>expect</span>("</span>Failed to write to stdin</span>");
</span>        </span>}
</span>
</span>        </span>fn </span>flush</span>(</span>&</span>self</span>) {
</span>            </span>let mut</span> child </span>= self</span>.</span>0.</span>lock</span>().</span>expect</span>("</span>Process failed</span>");
</span>            </span>let</span> stdin </span>= </span>(</span>*</span>child</span>).</span>stdin</span>.</span>as_mut</span>().</span>expect</span>("</span>Process failed</span>");
</span>
</span>            stdin</span>.</span>flush</span>().</span>expect</span>("</span>Failed to write to stdin</span>");
</span>        </span>}
</span>    </span>}
</span>
</span>    </span>/// Set custom logger
</span>    </span>log</span>::</span>set_max_level</span>(</span>LevelFilter</span>::</span>Info</span>);
</span>    </span>log</span>::</span>set_boxed_logger</span>(</span>Box</span>::</span>new</span>(</span>ChildLog</span>(</span>child_rc</span>))).</span>ok</span>();
</span>}
</span>
</span>let</span> pager_rc </span>= </span>Arc</span>::</span>new</span>(</span>Mutex</span>::</span>new</span>(</span>pager_child</span>));
</span>setup_piped_logger</span>(</span>pager_rc</span>.</span>clone</span>());
</span>
</span>/// Goes through the pager instead of the console directly
</span>info!</span>("</span>LET THERE BE LIGHT</span>");
</span>
</span>pager_rc</span>.</span>lock</span>().</span>unwrap</span>().</span>wait</span>().</span>expect</span>("</span>Failed to close</span>");
</span></code></pre>
My only peeve is using Arc</a> and Mutex</a> to get around the static
lifetime of the boxed logger. I could use mutable static</a> but it
requires unsafe which is not allowed or needed. I am impressed that
writing a custom logger was straightforward for a pager view.
Tutorials for integrating a more thorough pager was sparse and
perhaps an area to write about.</p>


Elixir Bit - socket.io Client Websocket
Unknown — Sun, 22 Nov 2020 00:00:00 +0000
>></a>
Introspection</h2>
I need an Elixir client for VccExchange</a>, a cryptocurrency trading
platform, to place orders and track trades. The REST client uses cURL</a>
so porting it via tesla</a> is straightforward; however, the websocket
uses socket.io</a> which requires a specific protocol or library. Although
socket.io</em> clients are implemented in several languages</a>, Elixir</em>
sadly does not have one. While it might be worth making a client as a
project, I only needed to subscribe to specific events specifically
balance changes</a> and trades executed</a>.</p>
Source: socket.ex</a></p>
>></a>
websockex</h2>
When I was working on a similar case with Bitfinex</a>, websockex</a> is a
good websocket library that feels like a normal GenServer</a> that the
following snippet shows:</p>
defmodule </span>App</span>.</span>Socket </span>do
</span>  </span>require </span>Logger
</span>
</span>  </span>use </span>WebSockex
</span>
</span>  # Taken from inspecting the dashboard with the browser's native debugger
</span>  </span>@</span>websocket_url </span>"</span>wss://socket.vcc.exchange:6001/socket.io/?EIO=3&transport=websocket</span>"
</span>
</span>  </span>def </span>start_link</span>(</span>_opts</span>) </span>do
</span>    state </span>=</span> %</span>{}
</span>    </span>{:</span>ok</span>,</span> conn</span>} </span>= </span>WebSockex</span>.</span>start_link</span>(</span>@</span>websocket_url</span>, </span>__MODULE__</span>,</span> state</span>)
</span>
</span>    # TODO: Subscribe to events
</span>
</span>    </span>{:</span>ok</span>,</span> conn</span>}
</span>  </span>end
</span>
</span>  </span>def </span>handle_connect</span>(</span>_conn</span>,</span> state</span>) </span>do
</span>    </span>Logger</span>.</span>debug</span>("</span>Socket Connected</span>")
</span>
</span>    </span>{:</span>ok</span>,</span> state</span>}
</span>  </span>end
</span>
</span>  </span>def </span>handle_disconnect</span>(</span>disconnection</span>,</span> state</span>) </span>do
</span>    </span>Logger</span>.</span>debug</span>("</span>Socket Disconnected</span>")
</span>
</span>    </span>{:</span>ok</span>,</span> state</span>}
</span>  </span>end
</span>
</span>  </span>def </span>handle_frame</span>({</span>_type</span>,</span> msg</span>},</span> state</span>) </span>do
</span>    </span>Logger</span>.</span>debug</span>("</span>Message Received: </span>#{</span>inspect</span>(</span>msg</span>)}")
</span>
</span>    </span>{:</span>ok</span>,</span> state</span>}
</span>  </span>end
</span>
</span>  </span>def </span>handle_cast</span>({:</span>send</span>, {</span>_type</span>,</span> msg</span>} </span>=</span> frame</span>},</span> state</span>) </span>do
</span>    </span>Logger</span>.</span>debug</span>("</span>Message Sent: </span>#{</span>inspect</span>(</span>msg</span>)}")
</span>
</span>    </span>{:</span>reply</span>,</span> frame</span>,</span> state</span>}
</span>  </span>end
</span>end
</span>
</span>defmodule </span>App</span>.</span>Application </span>do
</span>  </span>use </span>Supervisor
</span>
</span>  </span>def </span>start_link</span>(</span>opts</span>) </span>do
</span>    </span>Supervisor</span>.</span>start_link</span>(</span>__MODULE__</span>,</span> opts</span>, </span>name</span>: </span>__MODULE__</span>)
</span>  </span>end
</span>
</span>  </span>@</span>impl </span>true
</span>  </span>def </span>init</span>(</span>_init_arg</span>) </span>do
</span>    children </span>= </span>[
</span>      </span>{</span>App</span>.</span>Socket</span>, []}
</span>    </span>]
</span>
</span>    </span>Supervisor</span>.</span>init</span>(</span>children</span>, </span>strategy</span>: :</span>one_for_one</span>)
</span>  </span>end
</span>end
</span></code></pre>
Running this via App.Socket.start_link(nil)</code> or iex -S mix</code> connects
properly to the server. Now how to subscribe to the channels/events?
In particular, this snippet from the official documentation</a> needs to
be emulated:</p>
const </span>io </span>= </span>require(</span>'</span>socket.io-client</span>'</span>)</span>;
</span>const </span>socket </span>= </span>io(</span>'</span>wss://socket.vcc.exchange:6001</span>'</span>)</span>;
</span>
</span>const </span>API_KEY </span>= </span>'';
</span>const </span>CHANNEL </span>= </span>'</span>private-App.User.1</span>';
</span>const </span>EVENT </span>= </span>'</span>App</span>\\</span>Events</span>\\</span>BalanceUpdated</span>';
</span>
</span>socket</span>.</span>on(</span>'</span>disconnect</span>', </span>function </span>() {
</span>    console</span>.</span>log(</span>'</span>disconnected</span>'</span>)</span>;
</span>}</span>)</span>;
</span>socket</span>.</span>on(</span>'</span>reconnect</span>', </span>function </span>() {
</span>    console</span>.</span>log(</span>'</span>reconnect</span>'</span>)</span>;
</span>}</span>)</span>;
</span>socket</span>.</span>on(</span>'</span>reconnect_error</span>', </span>function </span>() {
</span>    console</span>.</span>log(</span>'</span>reconnect error</span>'</span>)</span>;
</span>}</span>)</span>;
</span>socket</span>.</span>on(</span>'</span>connect</span>', </span>function </span>() {
</span>    console</span>.</span>log(</span>'</span>connected</span>'</span>)</span>;
</span>    socket</span>.</span>emit(</span>'</span>subscribe</span>', {
</span>        channel</span>: </span>CHANNEL</span>,
</span>        auth</span>: {
</span>            headers</span>: {'</span>Authorization</span>': `</span>Bearer </span>${</span>API_KEY</span>}`}
</span>        </span>}
</span>    </span>}</span>)</span>.</span>on(EVENT</span>, </span>function</span>(</span>channel</span>, </span>data</span>) {
</span>        console</span>.</span>log(data)</span>;
</span>    </span>}</span>)</span>;
</span>}</span>)</span>;
</span></code></pre>
Since websocketex</code> handles majority of the event handlers, only
socket.emit</a> needs to be handled. Looking at the socket.io protocol</a>,
the next snippet shows how it encodes the parameters:</p>
@</span>header </span>"</span>42</span>"
</span>
</span>@</span>spec emit</span>(</span>charlist</span>,</span> any</span>)</span> :: charlist
</span>def </span>emit</span>(</span>event_name</span>,</span> data</span>) </span>do
</span>  body </span>= </span>[</span>event_name</span>,</span> data</span>]
</span>
</span>  </span>"#{</span>@</span>header</span>}#{</span>Jason</span>.</span>encode!</span>(</span>body</span>)}"
</span>end
</span>
</span>emit</span>("</span>hey</span>", "</span>jude</span>")
</span>"</span>42[</span>\"</span>hey</span>\"</span>,</span>\"</span>jude</span>\"</span>]</span>"
</span>
</span>emit</span>("</span>subscribe</span>",</span> %</span>{</span>channel</span>: "</span>MYCHANNEL</span>"})
</span>"</span>42[</span>\"</span>subscribe</span>\"</span>,{</span>\"</span>channel</span>\"</span>:</span>\"</span>MYCHANNEL</span>\"</span>}]</span>"
</span></code></pre>
While this is no the full spec, it is enough to send over the wire.
So after the socket connects, I send the subscription event for each
channel:</p>
def </span>start_link</span>(</span>_opts</span>) </span>do
</span>  </span>{:</span>ok</span>,</span> conn</span>} </span>= </span>WebSockex</span>.</span>start_link</span>(</span>@</span>websocket_url</span>, </span>__MODULE__</span>,</span> state</span>)
</span>
</span>  auth </span>=</span> %</span>{</span>headers</span>:</span> %</span>{"</span>Authorization</span>" </span>=> </span>"</span>Bearer </span>#{</span>api_key</span>()}"}}
</span>
</span>  </span>[
</span>    </span>"</span>App.Trades.</span>#{</span>pair</span>()}",
</span>    </span>"</span>private-App.User.</span>#{</span>user_id</span>()}"
</span>  </span>]
</span>  </span>|> </span>Enum</span>.</span>each</span>(</span>fn</span> channel </span>->
</span>    frame </span>=</span> emit</span>("</span>subscribe</span>",</span> %</span>{</span>channel</span>:</span> channel</span>, </span>auth</span>:</span> auth</span>})
</span>
</span>    </span>WebSockex</span>.</span>send_frame</span>(</span>conn</span>, {:</span>text</span>,</span> frame</span>})
</span>  </span>end</span>)
</span>
</span>  </span>{:</span>ok</span>,</span> conn</span>}
</span>end
</span>
</span>defp </span>pair</span>(),
</span>  </span>do: </span>"</span>ETHBTC</span>"
</span>
</span>defp </span>user_id</span>(),
</span>  </span>do: </span>System</span>.</span>get_env</span>("</span>VCCE_USER_ID</span>", "</span>123456</span>")
</span>
</span>defp </span>api_key</span>(),
</span>  </span>do: </span>System</span>.</span>get_env</span>("</span>VCCE_API_KEY</span>", "</span>CREATE_YOUR_OWN_KEY</span>")
</span></code></pre>
The socket now receives changes in balance and new trades but they
still need to be handled.</p>
>></a>
Event Handling</h2>
Received frames are handled in Websockex.handle_frame/2</a>. Let us examine
the balance update event.</p>
42</span>[
</span>  </span>"</span>App</span>\\</span>Events</span>\\</span>BalanceUpdated</span>",
</span>  </span>"</span>private-App.User.123456</span>",
</span>  </span>{
</span>    </span>"</span>data</span>": {
</span>      </span>"</span>eth</span>": {
</span>        </span>"</span>balance</span>": "</span>10.000</span>",
</span>        </span>"</span>available_balance</span>": "</span>9.000</span>"
</span>      </span>}
</span>    </span>},
</span>    </span>"</span>socket</span>": </span>null
</span>  </span>}
</span>]
</span></code></pre>
Almost JSON, the frame is wrapped by the socket.io</code> protocol which we
can pattern match over after decoding it for ease:</p>
def </span>handle_frame</span>({:</span>text</span>, "</span>42</span>" </span><></span> msg</span>},</span> state</span>) </span>do
</span>  </span>case </span>Jason</span>.</span>decode!</span>(</span>msg</span>) </span>do
</span>    </span>[
</span>      </span>"</span>App</span>\\</span>Events</span>\\</span>BalanceUpdated</span>",
</span>      </span>"</span>private-App.User.</span>" </span><></span> _user_id</span>,
</span>      %</span>{
</span>        </span>"</span>data</span>" </span>=></span> _data</span>,
</span>        </span>"</span>socket</span>" </span>=> nil
</span>      </span>}
</span>    </span>] </span>->
</span>      </span>Logger</span>.</span>info</span>("</span>BalanceUpdated</span>")
</span>
</span>    data </span>->
</span>      </span>Logger</span>.</span>warn</span>("</span>Unhandled Message: </span>#{</span>inspect</span>(</span>data</span>)}")
</span>  </span>end
</span>
</span>  </span>{:</span>ok</span>,</span> state</span>}
</span>end
</span></code></pre>
So in handling more events, the pattern can be augmented and a side
effect can be triggered. However, the socket.io</code> events still need to
be handled. When the socket connects to server, the socket receives
the following frame:</p>
0</span>{
</span>  </span>"</span>sid</span>": "</span>SESSION_ID</span>",
</span>  </span>"</span>upgrades</span>": [],
</span>  </span>"</span>pingInterval</span>": </span>25000</span>,
</span>  </span>"</span>pingTimeout</span>": </span>5000
</span>}
</span></code></pre>
The first frame sent by socket.io</code> is the session information. The
important fields are the pingInterval</code> and pingTimeout</code> which
indicate a heartbeat in milliseconds. If you leave the socket open
around that time, it will disconnect and attempt to reconnect. The
quick solution is to use :timer.send_interval/3</a> and send a ping frame
which is "2"</code> in socket.io</code>:</p>
@</span>heartbeat_interval </span>5_000
</span>
</span>def </span>start_link</span>(</span>_opts</span>) </span>do
</span>  # Rest of the code
</span>
</span>  </span>:</span>timer</span>.</span>send_interval</span>(</span>@</span>heartbeat_interval </span>,</span>self</span>(), :</span>ping</span>)
</span>
</span>  </span>{:</span>ok</span>,</span> conn</span>}
</span>end
</span>
</span>def </span>handle_info</span>(:</span>ping</span>,</span> state</span>),
</span>  </span>do: </span>{:</span>reply</span>, {:</span>text</span>, "</span>2</span>"},</span> state</span>}
</span>
</span>def </span>handle_frame</span>({:</span>text</span>, "</span>0</span>" </span><></span> _session_frame</span>},</span> state</span>),
</span>  </span>do: </span>{:</span>ok</span>,</span> state</span>}
</span>
</span>def </span>handle_frame</span>({:</span>text</span>, "</span>3</span>"},</span> state</span>),
</span>  </span>do: </span>{:</span>ok</span>,</span> state</span>}
</span></code></pre>
Aside from handling the pong</code> response ("3"</code>), the last one to
handle is joining a channel which is easily:</p>
def </span>handle_frame</span>({:</span>text</span>, "</span>40</span>"},</span> state</span>),
</span>  </span>do: </span>{:</span>ok</span>,</span> state</span>}
</span></code></pre>
The socket is now stable and working more or less.</p>
>></a>
Conclusion</h2>
It would be nice if an official socket.io</code> client existed to make
this all easier and comprehensive. In particular, error handling and
testing is not ergonomic or apparent. Perhaps exposing a framework
agnostic websocket might alleviate library concerns but support
ultimately matters. Also, I realized that I could just introspect the
websocket messages in the browser and reference that instead of
guessing exact message which would have saved me hours.</p>


Elixir Bit - Caching Docker Dependencies
Unknown — Sun, 15 Nov 2020 00:00:00 +0000
>></a>
Introspection</h2>
Using docker</a> and docker-compose</a> to share a packaged development
version of an Elixir</em> application in a team was successful. My
Dockerfile</a> and docker-compose.yml</a> is straightforward although could be
further optimized:</p>
FROM</span> elixir:</span>1.9.0-alpine
</span>
</span>RUN </span>apk --no-cache --update add \
</span>        git \
</span>        erlang \
</span>        alpine-sdk \
</span>        gmp-dev \
</span>        automake \
</span>        libtool \
</span>        inotify-tools \
</span>        autoconf
</span>
</span>ENV </span>HOME=</span>"</span>/opt/elixir</span>"</span> \
</span>    MIX_HOME=</span>"</span>/opt/elixir/mix</span>"</span> \
</span>    MIX_BUILD_PATH=</span>"</span>/opt/elixir/build</span>"</span> \
</span>    HEX_HOME=</span>"</span>/opt/elixir/hex</span>"</span> \
</span>    PATH=</span>"</span>$PATH:/opt/elixir/mix</span>"</span> \
</span>    ERL_AFLAGS=</span>"</span>-kernel shell_history enabled</span>"</span> \
</span>    ELIXIR_ERL_OPTS=</span>"</span>+C multi_time_warp</span>"
</span>
</span>RUN </span>mkdir -p /opt/elixir/hex && \
</span>    mkdir -p /opt/elixir/mix && \
</span>    mkdir -p /opt/elixir/build && \
</span>    mkdir -p /opt/elixir/.cache
</span>
</span>RUN </span>mkdir -p /opt/app && \
</span>    mkdir -p /opt/app/deps
</span>
</span>RUN </span>chmod -R 777 /opt
</span>
</span>RUN </span>mix local.hex --force && mix local.rebar --force
</span>
</span>WORKDIR </span>/opt/app
</span>
</span>EXPOSE </span>4000
</span>
</span>CMD </span>[</span>"</span>mix</span>"</span>, </span>"</span>run</span>"</span>, </span>"</span>--no-halt</span>"</span>]
</span></code></pre>
For my development Dockerfile</code>, I use an alpine Elixir image</a> to
minimize the download size and handpick a few packages to build
dependencies. I place everything related to Elixir</em> in /opt/elixir</code>
so that it can be cached via a docker volume</a> and mount the project in
/opt/app</code>. I also prepare the deps</code> folder to be locally cached
which the docker-compose.yml</code> can illustrate:</p>
version</span>: '</span>3.1</span>'
</span>services</span>:
</span>  </span>db</span>:
</span>    </span>image</span>: </span>postgres:9.5.17-alpine
</span>    </span>restart</span>: </span>always
</span>  </span>app</span>:
</span>    </span>build</span>:
</span>      </span>context</span>: .
</span>      </span>dockerfile</span>: </span>Dockerfile
</span>    </span>command</span>: "</span>mix phx.server</span>"
</span>    </span>entrypoint</span>: </span>/opt/app/docker-entrypoint.sh
</span>    </span>volumes</span>:
</span>      </span>- </span>.:/opt/app
</span>      </span>- </span>elixir:/opt/elixir
</span>      </span>- </span>mix_deps:/opt/app/deps
</span>    </span>depends_on</span>:
</span>      </span>- </span>db
</span>    </span>links</span>:
</span>      </span>- </span>db:db
</span>volumes</span>:
</span>  </span>elixir</span>:
</span>  </span>mix_deps</span>:
</span></code></pre>
This setup works usually for an phoenix umbrella application</a> that
requires a postgresql</a> database. When the container runs, I have it
fetch dependencies and migrations via docker-entrypoint.sh</a>:</p>
#!/bin/bash
</span>
</span>set -e
</span>
</span>echo "Updating dependencies if any uninstalled packages..."
</span>HEX_HTTP_CONCURRENCY=1 HEX_HTTP_TIMEOUT=1200 mix deps.get
</span>
</span>echo "Running migrations if any..."
</span>mix do ecto.create --no-deps-check, ecto.migrate --no-deps-check
</span>
</span>echo "Done with housecleaning"
</span>exec "$@"
</span></code></pre>
Running it is direct although with a minor chmod</code> on mix.lock</code> since
mix.lock</code> is updated in the container:</p>
git pull
</span>git checkout feature/branch
</span>
</span>docker-compose build
</span>
</span>sudo chmod 777 mix.lock
</span>docker-compose up
</span></code></pre>
For the most part, it works without issue; surprisingly, the most
frequent issue is that hex.pm</a> fails to fetch dependencies when
rebuilding. Even with a good connection, I do run into the issue
occasionally with minor annoyance. With poorer connection quality, it
becomes a blocker. To mitigate this, is it possible to download all
the dependencies locally and use that instead?</p>
>></a>
Local Mirror</h2>
In finding a solution, I came across mini_repo</a>, a minimal hex.pm</em>
server that can mirror and serve a few remote dependencies. Sadly, it
is bare, so I forked</a> it to demonstrate how I tweaked it for docker</em>.</p>
Running it by default, it only mirrors decimal</a>. To make it
configurable in docker-compose</em>, I added an space-delimited
PACKAGES</code> environment variable:</p>
services</span>:
</span>  </span>app</span>:
</span>    </span>enviroment</span>:
</span>      </span>PACKAGES</span>: </span>>-
</span>        decimal
</span>        phoenix
</span></code></pre>
# config/dev.exs
</span>
</span>packages </span>= </span>System</span>.</span>get_env</span>("</span>PACKAGES</span>", "")
</span>|> </span>String</span>.</span>trim</span>()
</span>|> </span>String</span>.</span>split</span>(" ", </span>trim</span>: </span>true</span>)
</span>
</span>config </span>:</span>mini_repo</span>,
</span>  </span>repositories</span>: [
</span>    </span>hexpm_mirror</span>: [
</span>      # only: ~w(decimal)
</span>      </span>only</span>:</span> packages</span>,
</span>    </span>]
</span>  </span>]
</span></code></pre>
# Does not fetch any package
</span>mix run --no-halt
</span>
</span># Fetch one package
</span>PACKAGES="decimal" mix run --no-halt
</span>HEX_MIRROR=http://localhost:4000/repos/hexpm_mirror mix hex.package fetch decimal 1.8.0
</span>
</span># Fetch multiple packages
</span>export HEX_MIRROR=http://localhost:4000/repos/hexpm_mirror
</span>PACKAGES="decimal phoenix" mix run --no-halt
</span>mix hex.package fetch phoenix 1.5.6
</span></code></pre>
The mini_repo</em> logs are promising:</p>
17:17:21.433 [info]  GET /repos/hexpm_mirror/packages/phoenix
</span>17:17:21.434 [info]  GET /repos/hexpm_mirror/tarballs/phoenix-1.5.6.tar
</span>17:17:21.522 [info]  Sent 200 in 88ms
</span>17:17:21.523 [info]  Sent 200 in 89ms
</span></code></pre>
However for every package, it downloaded every available version of
it:</p>
17:16:24.717 [debug] MiniRepo.Mirror.Server fetching tarball phoenix-1.5.1.tar
</span>17:16:25.301 [debug] {MiniRepo.Store.Local, :put, "data/repos/hexpm_mirror/tarballs/phoenix-1.4.17.tar"}
</span>17:16:25.309 [debug] MiniRepo.Mirror.Server fetching tarball phoenix-1.5.2.tar
</span>17:16:26.331 [debug] {MiniRepo.Store.Local, :put, "data/repos/hexpm_mirror/tarballs/phoenix-1.5.0-rc.0.tar"}
</span>17:16:26.336 [debug] MiniRepo.Mirror.Server fetching tarball phoenix-1.5.3.tar
</span>17:16:27.206 [debug] {MiniRepo.Store.Local, :put, "data/repos/hexpm_mirror/tarballs/phoenix-1.5.0.tar"}
</span>17:16:27.228 [debug] MiniRepo.Mirror.Server fetching tarball phoenix-1.5.4.tar
</span>17:16:28.017 [debug] {MiniRepo.Store.Local, :put, "data/repos/hexpm_mirror/tarballs/phoenix-1.5.1.tar"}
</span>17:16:28.029 [debug] MiniRepo.Mirror.Server fetching tarball phoenix-1.5.5.tar
</span>17:16:29.008 [debug] {MiniRepo.Store.Local, :put, "data/repos/hexpm_mirror/tarballs/phoenix-1.5.2.tar"}
</span>17:16:29.022 [debug] MiniRepo.Mirror.Server fetching tarball phoenix-1.5.6.tar
</span>17:16:29.706 [debug] {MiniRepo.Store.Local, :put, "data/repos/hexpm_mirror/tarballs/phoenix-1.5.3.tar"}
</span>17:16:30.375 [debug] {MiniRepo.Store.Local, :put, "data/repos/hexpm_mirror/tarballs/phoenix-1.5.4.tar"}
</span>17:16:30.967 [debug] {MiniRepo.Store.Local, :put, "data/repos/hexpm_mirror/tarballs/phoenix-1.5.5.tar"}
</span>17:16:31.568 [debug] {MiniRepo.Store.Local, :put, "data/repos/hexpm_mirror/tarballs/phoenix-1.5.6.tar"}
</span></code></pre>
This is problematic if the application only uses a single/fixed version of
an dependency. The current application does not support a targeted
version perhaps in a name</code>-version</code> format:</p>
PACKAGES="decimal phoenix-1.5.6" mix run --no-halt
</span></code></pre>
In this case, all of the decimal</code> versions are downloaded while a
specific version for phoenix</code>. Hacking this behavior can be done by
changing two files:</p>
# config/dev.exs
</span>
</span>packages </span>= </span>System</span>.</span>get_env</span>("</span>PACKAGES</span>", "")
</span>|> </span>String</span>.</span>trim</span>()
</span>|> </span>String</span>.</span>split</span>(" ", </span>trim</span>: </span>true</span>)
</span>|> </span>Enum</span>.</span>map</span>(</span>fn</span> package </span>->
</span>  </span>case </span>String</span>.</span>split</span>(</span>package</span>, "</span>-</span>", </span>trim</span>: </span>true</span>, </span>parts</span>: </span>2</span>) </span>do
</span>    </span>[</span>package</span>,</span> version</span>] </span>->
</span>      </span>{</span>package</span>,</span> version</span>}
</span>    </span>[</span>package </span>|</span> _</span>] </span>->
</span>      </span>{</span>package</span>, </span>nil</span>}
</span>    </span>[] </span>->
</span>      </span>nil
</span>  </span>end
</span>end</span>)
</span>|> </span>Enum</span>.</span>reject</span>(</span>&</span>is_nil</span>/</span>1</span>)
</span>
</span># lib/mini_repo/mirror/server.ex
</span>defp </span>sync</span>(</span>mirror</span>) </span>do
</span>  # Since mirror.only is now an list of tuples instead of strings
</span>  # We need to handle every mirror.only
</span>  only_packages </span>= </span>Enum</span>.</span>map</span>(</span>mirror</span>.</span>only</span>, </span>&</span>elem</span>(</span>&</span>1</span>, </span>0</span>))
</span>
</span>  versions </span>= for</span> %</span>{</span>name</span>:</span> name</span>} </span>=</span> map </span><-</span> versions</span>,
</span>            </span>!</span>mirror</span>.</span>only </span>or</span> name </span>in</span> only_packages</span>,
</span>            </span>into</span>:</span> %</span>{},
</span>            </span>do</span>: {</span>name</span>, </span>Map</span>.</span>delete</span>(</span>map</span>, :</span>version</span>)}
</span>end
</span>
</span>defp </span>sync_created_packages</span>(</span>mirror</span>,</span> config</span>,</span> diff</span>) </span>do
</span>  # Before the main loop, load the pinned version map
</span>  only_map </span>= </span>Enum</span>.</span>into</span>(</span>mirror</span>.</span>only</span>,</span> %</span>{})
</span>
</span>  </span>{:</span>ok</span>,</span> releases</span>} </span>=</span> sync_package</span>(</span>mirror</span>,</span> config</span>,</span> name</span>)
</span>
</span>  # Filter out releases if a pinned version exists
</span>  releases </span>= if</span> pin_version </span>= </span>Map</span>.</span>get</span>(</span>only_map</span>,</span> name</span>, </span>nil</span>) </span>do
</span>    </span>Enum</span>.</span>filter</span>(</span>releases</span>, </span>fn</span> release </span>-></span> release</span>.</span>version </span>==</span> pin_version </span>end</span>)
</span>  </span>else
</span>    releases
</span>  </span>end
</span>end
</span>
</span>defp </span>sync_releases</span>(</span>mirror</span>,</span> config</span>,</span> diff</span>) </span>do
</span>  # Before the main loop, load the pinned version map
</span>  only_map </span>= </span>Enum</span>.</span>into</span>(</span>mirror</span>.</span>only</span>,</span> %</span>{})
</span>
</span>  </span>{:</span>ok</span>,</span> releases</span>} </span>=</span> sync_package</span>(</span>mirror</span>,</span> config</span>,</span> name</span>)
</span>
</span>  # Filter out releases if a pinned version exists
</span>  releases </span>= if</span> pin_version </span>= </span>Map</span>.</span>get</span>(</span>only_map</span>,</span> name</span>, </span>nil</span>) </span>do
</span>    </span>Enum</span>.</span>filter</span>(</span>map</span>.</span>created</span>, </span>fn</span> release </span>-></span> release </span>==</span> pin_version </span>end</span>)
</span>  </span>else
</span>    map</span>.</span>created
</span>  </span>end
</span>
</span>  map </span>=</span> %</span>{</span>map </span>| </span>created</span>:</span> releases</span>}
</span>end
</span></code></pre>
Regardless of the experimental hack, how about we test on a new
minimal phoenix</code> app:</p>
mix archive.install hex phx_new 1.5.6
</span>mix phx.new hello --no-html --no-webpack --no-ecto --no-gettext --no-dashboard
</span>
</span>cd hello
</span>HEX_MIRROR=http://localhost:4000/repos/hexpm_mirror mix deps.get
</span>
</span># Change port from 4000 to avoid conflict
</span>sed -i -r s/4000/5000/ config/dev.exs
</span>
</span>mix phx.server
</span></code></pre>
Repeatedly running mix deps.get</code> and updating PACKAGES</code> until it succeeds:</p>
PACKAGES="cowboy cowboy_telemetry cowlib decimal jason mime phoenix phoenix_html phoenix_pubsub plug plug_cowboy plug_crypto ranch telemetry telemetry_metrics telemetry_poller" mix run --no-halt
</span></code></pre>
If I had to determine the exact version for each package without a
mix.lock</code>, it would be tedious so the previous hack is a post-setup
optimization. Anyway, the test app runs so our mirror works and time
to dockerize</em> it.</p>
>></a>
Docker</h2>
Since this uses mix release</code> to package the app, I copy over
config/dev.exs</code> to config/releases.exs</code> and clean up the repo:</p>

Remove the test_repo</code> and security keys since I only need
mirroring, not publishing</li>
Change the package directory to data</code> instead of priv/data</code></li>
Set the fetch timeout to :infinity</code></li>
Set the max concurrency to nil</code> to use all machine cores.</li>
Set the sync interval to :timer.hours(24)</code> to sync daily instead
of 5 minutes.</li>
</ol>
Since this is a released container, I use this Dockerfile</a> instead:</p>
FROM</span> elixir:</span>1.9.0-alpine </span>AS </span>builder
</span>
</span>RUN </span>mix local.hex --force && \
</span>        mix local.rebar --force
</span>
</span>WORKDIR </span>/opt/app
</span>
</span>COPY</span> config ./config
</span>COPY</span> lib ./lib
</span>COPY</span> rel ./rel
</span>COPY</span> mix.exs .
</span>COPY</span> mix.lock .
</span>
</span>ENV </span>MIX_ENV=prod
</span>
</span>RUN </span>mix deps.get && \
</span>        mix deps.compile && \
</span>        mix release
</span>
</span>FROM</span> elixir:</span>1.9.0-alpine </span>AS </span>app
</span>
</span>WORKDIR </span>/opt/app
</span>
</span>COPY</span> --from=builder /opt/app/_build .
</span>
</span>EXPOSE </span>4000
</span>ENV </span>PORT=4000
</span>ENV </span>PACKAGES=</span>""
</span>ENV </span>PACAKGE_DIR=/opt/app/hex
</span>
</span>CMD </span>[</span>"</span>./prod/rel/mini_repo/bin/mini_repo</span>"</span>, </span>"</span>start</span>"</span>]
</span></code></pre>
More importantly, the docker-compose.yml</a> looks good:</p>
version</span>: '</span>3.1</span>'
</span>services</span>:
</span>  </span>app</span>:
</span>    </span>build</span>:
</span>      </span>context</span>: .
</span>      </span>dockerfile</span>: </span>Dockerfile
</span>    </span>restart</span>: </span>always
</span>    </span>volumes</span>:
</span>      </span>- </span>./cache:/opt/app/data
</span>    </span>networks</span>:
</span>      </span>default</span>:
</span>      </span>hexpm</span>:
</span>        </span>aliases</span>:
</span>          </span>- </span>hexpm_mirror
</span>    </span>environment</span>:
</span>      </span>PORT</span>: "</span>20000</span>"
</span>      </span>PACKAGES</span>: </span>>
</span>        cowboy
</span>        cowboy_telemetry
</span>        cowlib
</span>        decimal
</span>        jason
</span>        mime
</span>        phoenix
</span>        phoenix_html
</span>        phoenix_pubsub
</span>        plug
</span>        plug_cowboy
</span>        plug_crypto
</span>        ranch
</span>        telemetry
</span>        telemetry_metrics
</span>        telemetry_poller
</span>networks</span>:
</span>  </span>hexpm</span>:
</span>    </span>external</span>: </span>true
</span></code></pre>
The important part here is the network</code> setting which allows another
container to use this. Referencing the development template, this can
be done by adding the HEX_MIRROR</code> variable and hexpm</code> network:</p>
version</span>: '</span>3.1</span>'
</span>services</span>:
</span>  </span>app</span>:
</span>    </span>networks</span>:
</span>      </span>default</span>:
</span>      </span>electron</span>:
</span>    </span>environment</span>:
</span>      </span>HEX_MIRROR</span>: "</span>http://hexpm_mirror:20000/repos/hexpm_mirror</span>"
</span>networks</span>:
</span>  </span>hexpm</span>:
</span>    </span>external</span>: </span>true
</span></code></pre>
Staring the mirror and waiting for it to sync:</p>
docker-compose build
</span>
</span>docker network create hexpm
</span>docker-compose up
</span></code></pre>
The Elixir</em> container should now build faster thanks to the local mirror:</p>
docker-compose down -v
</span>
</span>docker network create hexpm
</span>docker-compose up
</span></code></pre>
>></a>
Conclusion</h2>
While we were able to locally cache certain Elixir</em> dependencies in
docker</em>, it did take quite some work and it is not truly optimized.
It would be nice if the package caching can be dynamic where the
mirror downloads the package if it does not exist. While a static list
is fine, it can be stale if a new library is introduced and is not
synced which can cause the same build errors. Another lingering issue
is the excess downloads which feels ironic but makes sense when the
mix dependencies are not pinned to a specific version. Nonetheless, it
works and time will tell if it will reduce the network fetch issue.</p>


Rit Journal - Myers Diff
Unknown — Mon, 26 Oct 2020 00:00:00 +0000
>></a>
Chapter 11: Myers Diff</h2>
For my first chapter journal entry, the task is to implement the
Myer's diff algorithm</a> which calculates the changes shown in git diff</a>
and possibly integrate to git status</code> next. While that is the
default, it can be changed and have different results which is
fascinating. I will not discuss the algorithm here so do checkout
other sources such as from the author's blog</a>.</p>
Apologies if the journal is unstructured or sparse, I am trying to
find the balance between being a personal journal and thoroughly
reviewed article.</p>
Source code for this chapter can be found here</a></strong></strong></p>
>>></a>
Sketch</h3>
Generally, the objective is to take two lists of comparable items and
return an iterator of diffs to convert one to the other. My basic
sketch of this algorithm is:</p>
pub enum </span>Diff</span><'a, T> </span>{
</span>    Same</span>(</span>&'a</span> T</span>),
</span>    Insert</span>(</span>&'a</span> T</span>),
</span>    Delete</span>(</span>&'a</span> T</span>),
</span>}
</span>
</span>pub struct </span>Diffs</span><</span>'a</span>, T</span>> {
</span>    </span>source</span>: </span>&'a [T],
</span>    </span>target</span>: </span>&'a [T],
</span>    </span>path</span>: </span>Vec</span><(</span>usize</span>, </span>usize</span>)>
</span>}
</span>
</span>impl</span><</span>'a</span>, T</span>></span> Iterator </span>for </span>Diffs</span><</span>'a</span>, T</span>> {
</span>    </span>type </span>Item </span>= </span>Diff</span><</span>'a</span>, T</span>>;
</span>
</span>    </span>fn </span>next</span>(</span>&mut </span>self</span>) -> </span>Option</span><</span>Self</span>::</span>Output</span>> {
</span>        </span>unimplemented!</span>()
</span>    </span>}
</span>}
</span>
</span>pub fn </span>myers</span><</span>'a</span>, T</span>: </span>PartialEq</span>>(</span>source</span>: </span>&'a</span> [T], </span>target</span>: </span>&'a</span> [T]</span>) -> </span>Diffs</span><</span>'a</span>, T</span>> {
</span>    </span>unimplemented!</span>();
</span>}
</span></code></pre>
Ideally, it is used like this:</p>
let</span> source </span>= </span>"</span>ABCABBA</span>".</span>chars</span>().</span>collect</span>::<</span>Vec</span><</span>_</span>>>();
</span>let</span> target </span>= </span>"</span>CBABAC</span>".</span>chars</span>().</span>collect</span>::<</span>Vec</span><</span>_</span>>>();
</span>
</span>for</span> diff </span>in </span>myers</span>(</span>source</span>.</span>as_slice</span>(),</span> target</span>.</span>as_slice</span>()) {
</span>    </span>let</span> symbol </span>= match</span> diff </span>{
</span>        </span>Diff</span>::</span>Same</span>(</span>text</span>) </span>=> </span>println!</span>(" </span>{}</span>",</span> text</span>),
</span>        </span>Diff</span>::</span>Insert</span>(</span>text</span>) </span>=> </span>println!</span>("</span>+</span>{}</span>",</span> text</span>),
</span>        </span>Diff</span>::</span>Delete</span>(</span>text</span>) </span>=> </span>println!</span>("</span>-</span>{}</span>",</span> text</span>),
</span>    </span>}
</span>}
</span></code></pre>
It should output:</p>
-A
</span>-B
</span> C
</span>+B
</span> A
</span> B
</span>-B
</span> A
</span>+C
</span></code></pre>
The algorithm itself is straightforward to implement with finding the
best point in the main function and backtracking with the iterator.
The first part of the Ruby</em> source code translates easily but the
backtracking iterator is the challenging part. Looking back, the
fundamental lesson for this chapter is keeping it simple since it
does take time to refactor or experiment with a typed language. I did
want to explore other Rust</em> traits such as AsRef</a> trait but were
unnecessary in the end which is a waste.</p>
>>></a>
Iterator or List</h3>
Since the book returned the diffs as an iterator</a>, it follows that the
function should do so as well. However, since the backtracking logic
yields diffs backwards, the iterator has to be reversed whim implies:</p>
pub fn </span>myers</span><</span>'a</span>, T</span>: </span>PartialEq</span>>(</span>source</span>: </span>&'a</span> [T], </span>target</span>: </span>&'a</span> [T]</span>) -></span> impl </span>Iterator</span><</span>Item = Diff</span><</span>'a</span>, T</span>>> {
</span>    Diffs </span>{
</span>    </span>}
</span>    </span>.</span>rev</span>()
</span>}
</span></code></pre>
To use Iterator::rev</a> requires implementing a double ended iterator</a>.
The easiest way with the diff iterator is to collect the iterator
into a list then convert it back to an iterator:</p>
pub fn </span>myers</span><</span>'a</span>, T</span>: </span>PartialEq</span>>(</span>source</span>: </span>&'a</span> [T], </span>target</span>: </span>&'a</span> [T]</span>) -></span> impl </span>Iterator</span><</span>Item = Diff</span><</span>'a</span>, T</span>>> {
</span>    Diffs </span>{
</span>    </span>}
</span>    </span>.</span>collect</span>::<</span>Vec</span><</span>_</span>>>()
</span>    </span>.</span>into_iter</span>()
</span>    </span>.</span>rev</span>()
</span>}
</span></code></pre>
Returning an iterator here seems inefficient. Why not return the
ordered diffs instead? Why not get rid of the iterator altogether to
make the code more compact?</p>
pub fn </span>myers</span><</span>'a</span>, T</span>: </span>PartialEq</span>>(</span>source</span>: </span>&'a</span> [T], </span>target</span>: </span>&'a</span> [T]</span>) -> </span>Vec</span><</span>Diff</span><</span>'a</span>, T</span>>> {
</span>    </span>/// Rest of the code
</span>
</span>    </span>let mut</span> diffs </span>= Vec</span>::</span>with_capacity</span>(</span>traces</span>.</span>len</span>());
</span>
</span>    </span>/// Iterator code
</span>
</span>    diffs</span>.</span>reverse</span>();
</span>
</span>    diffs
</span>}
</span></code></pre>
Sadly, I was able to implement the algorithm completely before
realizing this and refactored afterwards. Perhaps I was too eager and
ambitious with implementing an iterator that I made it unnecessarily
harder for myself. Another lesson in keeping it simple but it was
rather fun making the iterator work.</p>
>>></a>
Negative Indices</h3>
In Ruby</em>, accessing array/list elements with negative indices</a> start
from the end instead of the beginning. Since the book uses this
feature, we need to translate</a> that to Rust</em>. We could implement a
wrapper trait, ExtSlide</code> to support negative indices that is easily
done with std::ops::Index</a> and std::ops::IndexMut</a>:</p>
struct </span>ExtSlice</span><</span>'a</span>, T</span>>(</span>&'a mut [T]</span>);
</span>
</span>impl</span><</span>'a</span>, T</span>> </span>Index</span><</span>isize</span>> </span>for </span>ExtSlice</span><</span>'a</span>, T</span>> {
</span>    </span>type </span>Output </span>=</span> T</span>;
</span>
</span>    </span>fn </span>index</span>(</span>&</span>self</span>, </span>mut </span>index</span>: </span>isize</span>) -> </span>&Self</span>::</span>Output </span>{
</span>        </span>if</span> index</span>.</span>is_negative</span>() {
</span>            index </span>+= self</span>.</span>0.</span>len</span>() </span>as isize</span>;
</span>        </span>}
</span>
</span>        </span>&self</span>.</span>0</span>[</span>index </span>as usize</span>]
</span>    </span>}
</span>}
</span>
</span>impl</span><</span>'a</span>, T</span>> </span>IndexMut</span><</span>isize</span>> </span>for </span>ExtSlice</span><</span>'a</span>, T</span>> {
</span>    </span>fn </span>index_mut</span>(</span>&mut </span>self</span>, </span>mut </span>index</span>: </span>isize</span>) -> </span>&mut Self</span>::</span>Output </span>{
</span>        </span>if</span> index</span>.</span>is_negative</span>() {
</span>            index </span>+= self</span>.</span>0.</span>len</span>() </span>as isize</span>;
</span>        </span>}
</span>
</span>        </span>&mut self</span>.</span>0</span>[</span>index </span>as usize</span>]
</span>    </span>}
</span>}
</span>
</span>let</span> xs </span>=</span> ExtSlide</span>(</span>&mut </span>[</span>1</span>, </span>2</span>, </span>3</span>]);
</span>
</span>assert_eq!</span>(</span>xs</span>[</span>0</span>],</span> xs</span>[</span>-</span>1</span>]);
</span>assert_eq!</span>(</span>xs</span>[</span>1</span>],</span> xs</span>[</span>-</span>2</span>]);
</span>assert_eq!</span>(</span>xs</span>[</span>2</span>],</span> xs</span>[</span>-</span>3</span>]);
</span>assert_eq!</span>(</span>xs</span>[</span>3</span>],</span> xs</span>[</span>-</span>1</span>]);
</span></code></pre>
This works but now we cannot access elements past isize</code>. For
example if slice indices are u8</a>, the maximum value of i8</a> is 127, so
how to access elements 128 and above? Since the algorithm is simple,
I opted to compute it manually to avoid any issue altogether:</p>
#[</span>inline</span>]
</span>fn </span>normalize_index</span>(</span>index</span>: </span>isize</span>, </span>size</span>: </span>usize</span>) -> </span>usize </span>{
</span>    </span>if</span> index</span>.</span>is_negative</span>() {
</span>        size </span>- </span>(</span>index</span>.</span>abs</span>() </span>as usize</span>)
</span>    </span>} </span>else </span>{
</span>        index </span>as usize
</span>    </span>}
</span>}
</span>
</span>'outer</span>: </span>for</span> d </span>in </span>0</span>..</span>(</span>max </span>as isize</span>) {
</span>    </span>for</span> k </span>in </span>(</span>-</span>d</span>..=</span>d</span>).</span>step_by</span>(</span>2</span>) {
</span>        </span>let</span> prev_index </span>= </span>normalize_index</span>(</span>k </span>- </span>1</span>);
</span>        </span>let</span> next_index </span>= </span>normalize_index</span>(</span>k </span>+ </span>1</span>);
</span>
</span>        </span>if</span> k </span>== -</span>d </span>|| </span>(</span>k </span>!=</span> d </span>&& </span>(</span>values</span>[</span>prev_index</span>].</span>0 </span><</span> values</span>[</span>next_index</span>].</span>0</span>)) { }
</span>
</span>        </span>/// Rest of the code
</span>    </span>}
</span>}
</span></code></pre>
How to implement negative indices without precision loss? Perhaps
implementing a custom std::ops::RangeBounds</a> can work but that is
currently beyond scope. Still interesting to know that indexing
operation can be implemented to enable this feature.</p>
>>></a>
Backtracking</h3>
Initially, the way backtracking was presented made me think of an
alternative representation. Since the best values at each round are
only stored, retracing from the best path is lost. The idea is then
to clone</strong></strong> and store the previous rounds into a list so that
backtracking is possible.</p>
let mut</span> values</span>: </span>Vec</span><</span>usize</span>> </span>= Vec</span>::</span>with_capacity</span>(</span>capacity</span>);
</span>values</span>.</span>resize</span>(</span>capacity</span>, </span>0</span>);
</span>
</span>let mut</span> traces</span>: </span>Vec</span><</span>Vec</span><</span>usize</span>>> </span>= Vec</span>::</span>with_capacity</span>(</span>max</span>);
</span>
</span>'outer</span>: </span>for</span> d </span>in </span>0</span>..</span>(</span>max </span>as isize</span>) {
</span>    traces</span>.</span>push</span>(</span>values</span>.</span>clone</span>());
</span>
</span>    </span>// Rest of the code
</span>}
</span></code></pre>
Instead of cloning the values on each round, what if we store the
path on each value and return that in the end?</p>
let mut</span> values</span>: </span>Vec</span><(</span>usize</span>, </span>Vec</span><(</span>usize</span>, </span>usize</span>)>)> </span>= Vec</span>::</span>with_capacity</span>(</span>capacity</span>);
</span>
</span>'outer</span>: </span>for</span> d </span>in </span>0</span>..</span>(</span>max </span>as isize</span>) {
</span>    </span>for</span> k </span>in </span>(</span>-</span>d</span>..=</span>d</span>).</span>step_by</span>(</span>2</span>) {
</span>        </span>// Rest of the code
</span>        </span>let</span> this_path </span>= &mut</span> values</span>[</span>index</span>].</span>1</span>;
</span>
</span>        this_path</span>.</span>push</span>((</span>this_x</span>,</span> this_y</span>));
</span>
</span>        </span>if</span> this_x </span>>=</span> m </span>&&</span> this_y </span>>=</span> n </span>{ }
</span>    </span>}
</span>}
</span></code></pre>
I completed the initial implementation with this layout but I
realized that is not as efficient as I thought. Given m</code> is the
length of the source, n</code> the length of the target, d</code> as the number
of rounds, my approach allocates 2 * m * n + 1</code> small vectors that
grow to d</code> elements while the original stores d</code> vectors of 2 * m * n + 1</code> size. While equal, the original has fewer allocations and
more compact memory layout. I keep relearning the old lesson of
planning ahead before writing.</p>
>>></a>
Testing</h3>
Aside from testing the base case, the book provides no other test. To
do property-based testing</a>, I could generate the diffs and compute the
left and right text, then the applied diff should be the same. Since
we are using quickcheck</code>, we want to create a generator type</a> for our
Diff</code> type. However, since it contains a reference field, we need to
create an owned version:</p>
#[</span>derive</span>(</span>Debug</span>,</span> PartialEq</span>)]
</span>pub enum </span>Diff</span><'a, T> </span>{
</span>    Same</span>(</span>&'a</span> T</span>),
</span>    Insert</span>(</span>&'a</span> T</span>),
</span>    Delete</span>(</span>&'a</span> T</span>),
</span>}
</span>
</span>#[</span>cfg</span>(</span>test</span>)]
</span>mod </span>tests </span>{
</span>    </span>use </span>quickcheck</span>::{</span>Arbitrary</span>,</span> Gen</span>};
</span>
</span>    </span>use super</span>::</span>*</span>;
</span>
</span>    </span>#[</span>derive</span>(</span>Clone</span>,</span> Debug</span>,</span> PartialEq</span>)]
</span>    </span>enum </span>TestDiff</span><T> </span>{
</span>        Same</span>(</span>T</span>),
</span>        Insert</span>(</span>T</span>),
</span>        Delete</span>(</span>T</span>),
</span>    </span>}
</span>}
</span></code></pre>
Next step is to create a generator for the diff collection. Ideally,
the type is Vec<TestDiff<T>></code>; however, since we do not own Vec</code>,
we need to wrap it with a struct and implement the trait on that
because of coherence rules</a>. To generate the collection, a list of
values are generated uniquely by BTreeSet</a> instead of an Vec</code> and
then randomly mapped to a diff enum:</p>
use </span>std</span>::</span>collections</span>::</span>BTreeSet</span>;
</span>use </span>rand</span>::</span>Rng</span>;
</span>
</span>#[</span>derive</span>(</span>Clone</span>,</span> Debug</span>)]
</span>struct </span>TestDiffs</span><</span>T</span>>(</span>Vec</span><</span>TestDiff</span><</span>T</span>>>);
</span>
</span>impl</span><</span>T</span>:</span> Arbitrary </span>+ Ord</span>></span> Arbitrary </span>for </span>TestDiffs</span><</span>T</span>> {
</span>    </span>fn </span>arbitrary</span><</span>G</span>:</span> Gen</span>>(</span>g</span>: </span>&mut</span> G</span>) -> </span>Self </span>{
</span>        </span>let</span> values</span>: </span>BTreeSet</span><</span>T</span>> </span>= </span>Arbitrary</span>::</span>arbitrary</span>(</span>g</span>);
</span>
</span>        </span>let</span> diffs </span>=</span> values
</span>            </span>.</span>into_iter</span>()
</span>            </span>.</span>map</span>(|</span>val</span>| </span>match</span> g</span>.</span>gen</span>::<</span>u8</span>>().</span>rem_euclid</span>(</span>3</span>) {
</span>                </span>0 </span>=> </span>TestDiff</span>::</span>Same</span>(</span>val</span>),
</span>                </span>1 </span>=> </span>TestDiff</span>::</span>Insert</span>(</span>val</span>),
</span>                </span>2 </span>=> </span>TestDiff</span>::</span>Delete</span>(</span>val</span>),
</span>                </span>_ => </span>unreachable!</span>(),
</span>            </span>})
</span>            </span>.</span>collect</span>::<</span>Vec</span><</span>_</span>>>();
</span>
</span>        TestDiffs</span>(</span>diffs</span>)
</span>    </span>}
</span>}
</span></code></pre>
We can now define the property test using the generator. More
importantly, how do we extract the source and target text given the
diffs? Since insertions come from the target and deletions from the
source, we can iterate through the diffs and collect them
appropriately:</p>
#[</span>quickcheck</span>]
</span>fn </span>diffs_are_correct</span>(</span>test_diffs</span>: </span>TestDiffs</span><</span>String</span>>) -> </span>bool </span>{
</span>    </span>let mut</span> diffs </span>=</span> test_diffs</span>.</span>0</span>;
</span>
</span>    </span>let mut</span> source </span>= Vec</span>::</span>with_capacity</span>(</span>diffs</span>.</span>len</span>());
</span>    </span>let mut</span> target </span>= Vec</span>::</span>with_capacity</span>(</span>diffs</span>.</span>len</span>());
</span>
</span>    </span>for</span> diff </span>in</span> diffs</span>.</span>clone</span>() {
</span>        </span>match</span> diff </span>{
</span>            </span>TestDiff</span>::</span>Delete</span>(</span>item</span>) </span>=> </span>{
</span>                source</span>.</span>push</span>(</span>item</span>);
</span>            </span>}
</span>            </span>TestDiff</span>::</span>Insert</span>(</span>item</span>) </span>=> </span>{
</span>                target</span>.</span>push</span>(</span>item</span>);
</span>            </span>}
</span>            </span>TestDiff</span>::</span>Same</span>(</span>item</span>) </span>=> </span>{
</span>                source</span>.</span>push</span>(</span>item</span>.</span>clone</span>());
</span>                target</span>.</span>push</span>(</span>item</span>);
</span>            </span>}
</span>        </span>};
</span>    </span>}
</span>
</span>    </span>unimplemented!</span>()
</span>}
</span></code></pre>
Finally, we apply the myers</code> function on the source and target and
expect the generated diffs to be the same:</p>
#[</span>quickcheck</span>]
</span>fn </span>diffs_are_correct</span>(</span>test_diffs</span>: </span>TestDiffs</span><</span>String</span>>) -> </span>bool </span>{
</span>    </span>/// Rest of the code
</span>
</span>    </span>let mut</span> actual_diffs </span>= </span>myers</span>(</span>source</span>.</span>as_slice</span>(),</span> target</span>.</span>as_slice</span>());
</span>
</span>    </span>for </span>(</span>actual_diff</span>,</span> expected_diff</span>) </span>in</span> actual_diffs</span>.</span>zip</span>(</span>diffs</span>) {
</span>        </span>match </span>(</span>actual_diff</span>,</span> expected_diff</span>) {
</span>            </span>(</span>Diff</span>::</span>Same</span>(</span>actual</span>), </span>TestDiff</span>::</span>Same</span>(</span>ref</span> expected</span>))
</span>                </span>| </span>(</span>Diff</span>::</span>Insert</span>(</span>actual</span>), </span>TestDiff</span>::</span>Insert</span>(</span>ref</span> expected</span>))
</span>                </span>| </span>(</span>Diff</span>::</span>Delete</span>(</span>actual</span>), </span>TestDiff</span>::</span>Delete</span>(</span>ref</span> expected</span>)) </span>=> </span>{
</span>                    </span>if</span> actual </span>!=</span> expected </span>{
</span>                        </span>return false</span>;
</span>                    </span>}
</span>                </span>}
</span>            </span>_ => return false</span>,
</span>        </span>};
</span>    </span>}
</span>
</span>    </span>true
</span>
</span>}
</span></code></pre>
Running this test exposes two edge cases and one generator issue. A
common edge case is handling empty list arguments. Adding the test
case directly and fixing it simple:</p>
pub fn </span>myers</span><</span>'a</span>, T</span>: </span>PartialEq</span>>(</span>source</span>: </span>&'a</span> [T], </span>target</span>: </span>&'a</span> [T]</span>) -> </span>Vec</span><</span>Diff</span><</span>'a</span>, T</span>>> {
</span>    </span>/// Return quickly when both lists are empty
</span>    </span>if</span> source</span>.</span>is_empty</span>() </span>&&</span> target</span>.</span>is_empty</span>() {
</span>        </span>return Vec</span>::</span>new</span>();
</span>    </span>}
</span>
</span>    </span>/// Rest of the code
</span>}
</span>
</span>#[</span>test</span>]
</span>fn </span>empty_values_should_work</span>() {
</span>    </span>assert_eq!</span>(</span>vec!</span>[</span>Diff</span>::</span>Insert</span>(</span>&</span>1</span>)], </span>myers</span>(</span>&</span>[], </span>&</span>[</span>1</span>]));
</span>    </span>assert_eq!</span>(</span>vec!</span>[</span>Diff</span>::</span>Delete</span>(</span>&</span>1</span>)], </span>myers</span>(</span>&</span>[</span>1</span>], </span>&</span>[]));
</span>    </span>assert_eq!</span>(</span>Vec</span>::<</span>Diff</span><</span>u8</span>>>::</span>new</span>(), </span>myers</span>::<</span>u8</span>>(</span>&</span>[], </span>&</span>[]));
</span>}
</span></code></pre>
The trickier issue is that if the starting values of the source and
target are the same, it will ignore that diagonal/free move and
proceed normally which will generate an unequal diff. The fix is
simply to apply the while diagonal loop on the first iteration and
augment the new point with that:</p>
pub fn </span>myers</span><</span>'a</span>, T</span>: </span>PartialEq</span>>(</span>left</span>: </span>&'a</span> [T], </span>right</span>: </span>&'a</span> [T]</span>) -> </span>Vec</span><</span>Diff</span><</span>'a</span>, T</span>>> {
</span>    </span>/// Rest of the code
</span>
</span>    </span>'outer</span>: </span>for</span> d </span>in </span>0</span>..</span>(</span>max </span>as isize</span>) {
</span>        </span>/// Check for the diagonal in the search phase
</span>        </span>let</span> delta </span>= if</span> d </span>== </span>0 </span>{
</span>            </span>let mut</span> i </span>= </span>0</span>;
</span>
</span>            </span>while</span> i </span><</span> m </span>&&</span> i </span><</span> n </span>&&</span> source</span>[</span>i</span>] </span>==</span> target</span>[</span>i</span>] {
</span>                i </span>+= </span>1</span>;
</span>            </span>}
</span>
</span>            i
</span>        </span>} </span>else </span>{
</span>            </span>0
</span>        </span>};
</span>
</span>        </span>for</span> k </span>in </span>(</span>-</span>d</span>..=</span>d</span>).</span>step_by</span>(</span>2</span>) {
</span>            this_x </span>+=</span> delta</span>;
</span>            this_y </span>+=</span> delta</span>;
</span>
</span>            </span>while</span> this_x </span><</span> m </span>&&</span> this_y </span><</span> n </span>&&</span> left</span>[</span>this_x</span>] </span>==</span> right</span>[</span>this_y</span>] { }
</span>
</span>            </span>/// Rest of the code
</span>        </span>}
</span>    </span>}
</span>
</span>    </span>/// Check for the diagonal right after the backtracking phase
</span>    </span>let </span>(</span>mut</span> x</span>, </span>mut</span> y</span>) </span>=</span> point</span>;
</span>
</span>    </span>while</span> x </span>> </span>0 </span>&&</span> y </span>> </span>0 </span>{
</span>        x </span>-= </span>1</span>;
</span>        y </span>-= </span>1</span>;
</span>
</span>        diffs</span>.</span>push</span>(</span>Diff</span>::</span>Same</span>(</span>&</span>target</span>[</span>y</span>]));
</span>    </span>}
</span>
</span>    diffs</span>.</span>reverse</span>();
</span>
</span>    diffs
</span>}
</span>
</span>#[</span>test</span>]
</span>fn </span>same_initial_should_work</span>() {
</span>    </span>assert_eq!</span>(
</span>        </span>vec!</span>[</span>Diff</span>::</span>Same</span>(</span>&</span>1</span>), </span>Diff</span>::</span>Delete</span>(</span>&</span>2</span>), </span>Diff</span>::</span>Insert</span>(</span>&</span>3</span>)],
</span>        myers</span>(</span>&</span>[</span>1</span>, </span>2</span>], </span>&</span>[</span>1</span>, </span>3</span>])
</span>    </span>);
</span>}
</span></code></pre>
Whether these two issues are fixed later in the book or because of my
implementation, the property test thankfully caught these issues
early on. However, I must confess the actual and expected diffs are
not equal. Specifically, they have the same elements but sometimes
not in the same order. For example:</p>
/// EXPECTED
</span>[</span>Same</span>(</span>20</span>),</span> Insert</span>(</span>78</span>),</span> Delete</span>(</span>99</span>)]
</span>
</span>/// ACTUAL
</span>[</span>Same</span>(</span>20</span>),</span> Delete</span>(</span>99</span>),</span> Insert</span>(</span>78</span>)]
</span></code></pre>
Sadly, our generation method is basic and incompatible with how
myers</code> generates diff. After struggling for a while, it was enough
to test when the diffs are sorted and are still equal:</p>
/// Added extra traits for sorting: PartialEq, PartialOrd, Eq, Ord
</span>#[</span>derive</span>(</span>Clone</span>,</span> Debug</span>,</span> PartialEq</span>,</span> PartialOrd</span>,</span> Eq</span>,</span> Ord</span>)]
</span>enum </span>Diff</span><'a, T> </span>{ }
</span>
</span>#[</span>derive</span>(</span>Clone</span>,</span> Debug</span>,</span> PartialEq</span>,</span> PartialOrd</span>,</span> Eq</span>,</span> Ord</span>)]
</span>enum </span>TestDiff</span><T> </span>{ }
</span>
</span>#[</span>quickcheck</span>]
</span>fn </span>diffs_are_correct</span>(</span>test_diffs</span>: </span>TestDiffs</span><</span>u8</span>>) -> </span>bool </span>{
</span>    </span>/// Rest of the code
</span>
</span>    </span>let mut</span> actual_diffs </span>= </span>myers</span>(</span>left</span>.</span>as_slice</span>(),</span> right</span>.</span>as_slice</span>());
</span>
</span>    actual_diffs</span>.</span>sort</span>();
</span>    diffs</span>.</span>sort</span>();
</span>
</span>    </span>for </span>(</span>actual_diff</span>,</span> expected_diff</span>) </span>in</span> actual_diffs</span>.</span>into_iter</span>().</span>zip</span>(</span>diffs</span>) {}
</span>    </span>/// Rest of the coe
</span>}
</span></code></pre>
While the generation is imperfect, sorting the diffs before comparing
them make the test consistent. My lesson here is that perfect tests
are nice but must be weighted against time and effort. Nonetheless,
the algorithm works safely and can now declare completion.</p>
>></a>
Histogram</h2>
In my early years with git</code>, I thought what made it special among
other VCS</a> is that it stores the diffs instead of the files that makes
it more efficient in computing which is incorrect. In reality, both
old and new file are stored then the diff between them is computed to
show in git diff</code>. My flawed thinking comes from applying a fold</a> of
diffs to compute the end file as in an empty file plus hello</code> then
finally world</code> is helloworld</code>. The issue is that this computation is
not unique as in hell</code> then oworld</code> produces the same output. With
multiple ways to compute the same text, what then is the best way to
compute them?</p>
Rather, how does one define quality or better? Git has four diff
algorithms: myers</code>, minimal</code>, patience</code> and histogram</code> where the
default is myers</code>. How do they differ? I know myers</code> is greedy and
prefers deletions to insertions, but do not know or have experience
about the others. Reading this paper</a> and other searches suggest that
histogram</a>, a modified version of patience</a>, is more human-readable
specially with code. Consider this sample code about a right triangle:</p>
struct </span>RightTriangle</span>;
</span>
</span>impl </span>RightTriangle </span>{
</span>    </span>fn </span>hypothenuse</span>(</span>a</span>: </span>f32</span>, </span>b</span>: </span>f32</span>) -> </span>f32 </span>{
</span>        </span>let</span> sum </span>=</span> a</span>.</span>powi</span>(</span>2</span>) </span>+</span> b</span>.</span>powi</span>(</span>2</span>);
</span>
</span>        sum</span>.</span>sqrt</span>()
</span>    </span>}
</span>
</span>    </span>fn </span>area</span>(</span>a</span>: </span>f32</span>, </span>b</span>: </span>f32</span>) -> </span>f32 </span>{
</span>        </span>(</span>a </span>*</span> b</span>) </span>/ </span>2.0
</span>    </span>}
</span>}
</span></code></pre>
I will do three modifications: move the function RightTriangle::area</code>
before RightTriangle::hypothenuse</code>, add a new function
RightTriangle::perimeter</code>, and refactor RightTriangle::hypothenuse</code>.
Here is the new snippet:</p>
struct </span>RightTriangle</span>;
</span>
</span>impl </span>RightTriangle </span>{
</span>    </span>fn </span>area</span>(</span>a</span>: </span>f32</span>, </span>b</span>: </span>f32</span>) -> </span>f32 </span>{
</span>        </span>(</span>a </span>*</span> b</span>) </span>/ </span>2.0
</span>    </span>}
</span>
</span>    </span>fn </span>hypothenuse</span>(</span>a</span>: </span>f32</span>, </span>b</span>: </span>f32</span>) -> </span>f32 </span>{
</span>        </span>(</span>a</span>.</span>powi</span>(</span>2</span>) </span>+</span> b</span>.</span>powi</span>(</span>2</span>)).</span>sqrt</span>()
</span>    </span>}
</span>
</span>    </span>fn </span>perimeter</span>(</span>a</span>: </span>f32</span>, </span>b</span>: </span>f32</span>) -> </span>f32 </span>{
</span>        a </span>+</span> b </span>+ Self</span>::</span>hypothenuse</span>(</span>a</span>,</span> b</span>)
</span>    </span>}
</span>}
</span></code></pre>
Running first git diff --diff-algorithm=myers</code>:</p>
-    fn hypothenuse(a: f32, b: f32) -> f32 {
</span>-        let sum = a.powi(2) + b.powi(2);
</span>+    fn area(a: f32, b: f32) -> f32 {
</span>+        (a * b) / 2.0
</span>+    }
</span>
</span>-        sum.sqrt()
</span>+    fn hypothenuse(a: f32, b: f32) -> f32 {
</span>+        (a.powi(2) + b.powi(2)).sqrt()
</span>     }
</span>
</span>-    fn area(a: f32, b: f32) -> f32 {
</span>-        (a * b) / 2.0
</span>+    fn perimeter(a: f32, b: f32) -> f32 {
</span>+        a + b + Self::hypothenuse(a, b)
</span></code></pre>
At a quick glance, it is hard to see what is going on here. With git diff --diff-algorithm=histogram</code>:</p>
-    fn hypothenuse(a: f32, b: f32) -> f32 {
</span>-        let sum = a.powi(2) + b.powi(2);
</span>-
</span>-        sum.sqrt()
</span>-    }
</span>-
</span>     fn area(a: f32, b: f32) -> f32 {
</span>         (a * b) / 2.0
</span>     }
</span>+
</span>+    fn hypothenuse(a: f32, b: f32) -> f32 {
</span>+        (a.powi(2) + b.powi(2)).sqrt()
</span>+    }
</span>+
</span>+    fn perimeter(a: f32, b: f32) -> f32 {
</span>+        a + b + Self::hypothenuse(a, b)
</span>+    }
</span></code></pre>
This looks easier to understand. While this is a contrived refactoring
example, it does show the possible advantage of this algorithm. If
histogram</code> is more human-readable, why is it not the default or
popular? I believe git</code> does not need to focus on subjective diff
algorithms, instead opting for the simplest and offering
configurability</a>. Rather than arguing the use for histogram</code>, changing
the diff algorithm is a minor potential and more importantly cannot be
configured in GitHub</a> or major git repositories to see the results as a
team. Still having potentially better diffs locally while adding or
merging is worth a try like so:</p>
git config --global diff.algorithm histogram
</span></code></pre>
I will get longer diffs in the worst case and perhaps mostly not
notice the difference, but I know now at least the diff view can be
configured</a>.</p>


Elixir Bit - ISO 8601 Duration
Unknown — Fri, 23 Oct 2020 00:00:00 +0000
>></a>
Problem</h2>
I have several GenServers</a> that sync data from a third party service
and store them in-memory to reduce network calls for a GraphQL</a> client.
My usual formula is:</p>
defmodule </span>MyApp</span>.</span>ExternalService </span>do
</span>  </span>use </span>GenServer
</span>
</span>  </span>alias __MODULE__</span>, </span>as</span>: </span>State
</span>
</span>  </span>defstruct </span>[:</span>data</span>]
</span>
</span>  </span>def </span>start_link</span>(</span>opts</span>) </span>do
</span>    </span>GenServer</span>(</span>__MODULE__</span>,</span> opts</span>, </span>name</span>: </span>__MODULE__</span>)
</span>  </span>end
</span>
</span>  </span>def </span>get</span>(),
</span>    </span>do: </span>GenServer</span>.</span>call</span>(</span>__MODULE__</span>, :</span>get</span>)
</span>
</span>  </span>@</span>impl </span>true
</span>  </span>def </span>init</span>(</span>_opts</span>) </span>do
</span>    </span>{:</span>ok</span>,</span> _ref</span>} </span>= </span>:</span>timer</span>.</span>send_interval</span>(</span>sync_interval</span>(),</span> self</span>(), :</span>__sync__</span>)
</span>
</span>    </span>{:</span>ok</span>,</span> %</span>State</span>{</span>data</span>: </span>nil</span>}}
</span>  </span>end
</span>
</span>  </span>@</span>impl </span>true
</span>  </span>def </span>handle_call</span>(:</span>get</span>,</span> _from</span>,</span> %</span>State</span>{</span>data</span>:</span> data</span>} </span>=</span> state</span>),
</span>    </span>do: </span>{:</span>reply</span>,</span> data</span>,</span> state</span>}
</span>
</span>  </span>@</span>impl </span>true
</span>  </span>def </span>handle_info</span>(:</span>__sync__</span>,</span> state</span>) </span>do
</span>    new_data </span>= </span>Enum</span>.</span>random</span>(</span>1</span>..</span>100</span>)  </span># Pretend randomness is an API call
</span>
</span>    </span>{:</span>noreply</span>,</span> %</span>{</span>state </span>| </span>data</span>:</span> new_data</span>}}
</span>  </span>end
</span>
</span>  </span>def </span>sync_interval</span>(),
</span>    </span>do:</span> config</span>()[:</span>sync_interval</span>] </span>|| </span>:</span>timer</span>.</span>minutes</span>(</span>5</span>)
</span>
</span>  </span>def </span>config</span>(),
</span>    </span>do: </span>Application</span>.</span>get_all_env</span>(:</span>my_app</span>, </span>__MODULE__</span>)
</span>end
</span></code></pre>
The toy module above is an interface to an external service and syncs
data every five minutes. The trick is :timer.send_interval/3</a> which
triggers handle_info(:__sync__, state)</code> periodically in milliseconds.
For its runtime configuration with distillery</a> and TOML</a>, it looks like
this:</p>
[</span>my_app</span>."</span>MyApp.ExternalService</span>"]
</span>sync_interval </span>= </span>300000 </span># :timer.minutes(5)
</span></code></pre>
Since I have many of these, my problem is that reading an integer
value to represent an interval is harder to read, communicate and
configure as a team. While I still standardize to milliseconds, a
string-like representation or standard format would be nice to express
various intervals like:</p>
[</span>my_app</span>."</span>MyApp.ExternalService</span>"]
</span>sync_interval </span>= "</span>5 min</span>"
</span>
</span>[</span>my_app</span>."</span>MyApp.HeavyService</span>"]
</span>fast_sync_interval </span>= "</span>1 day</span>"
</span>slow_sync_interval </span>= [</span>30</span>, "</span>min</span>"]  </span># Requires custom transformer
</span></code></pre>
>></a>
Solution</h2>
A standard date duration format already exists: ISO 8601 Duration</a>.
Here are some sample representations to build intuition:</p>
"</span>P1Y</span>"       </span># 1 year(s)
</span>"</span>P2M1W</span>"     </span># 2 months and 1 week(s)
</span>"</span>P1Y1D</span>"     </span># 1 year(s) and 1 day(s)
</span>
</span>"</span>PT5M</span>"      </span># 5 minutes
</span>"</span>PT1H5S</span>"    </span># 1 hour(s) and 5 seconds
</span>
</span>"</span>P1DT12H</span>"   </span># 1 day(s) and 12 hours
</span></code></pre>
It does feel rather too compact and mentally parsing it might take
time. My suggestion is to think of it as having a date and time part,
then reading slowly the abbreviations based on the part. The date/time
library, timex</a>, can parse these expressions and be easily integrated:</p>
defmodule </span>MyApp</span>.</span>ExternalService </span>do
</span>  # Same code as before
</span>
</span>  </span>alias </span>Timex</span>.</span>Duration
</span>
</span>  </span>def </span>sync_interval</span>() </span>do
</span>    </span>(</span>config</span>()[:</span>sync_interval</span>] </span>|| </span>"</span>PT5M</span>")
</span>    </span>|> </span>Duration</span>.</span>parse!</span>()
</span>    </span>|> </span>Duration</span>.</span>to_milliseconds</span>() </span># Since this function may return a float...
</span>    </span>|></span> round</span>()                    </span># ... we need to cast it to integer
</span>  </span>end
</span>end
</span></code></pre>
The format looks promising in TOML:</p>
[</span>my_app</span>."</span>MyApp.ExternalService</span>"]
</span>sync_interval </span>= "</span>PT5M</span>"
</span></code></pre>
To drive the point further, it can also express future dates well such
as expiration dates. For example, a reset password link expires in 2
days in development but 30 minutes in production for security:</p>
defmodule </span>MyApp</span>.</span>Accounts </span>do
</span>  </span>alias </span>Ecto</span>.</span>UUID
</span>  </span>alias </span>Timex</span>.</span>Duration
</span>  </span>alias </span>MyApp</span>.{</span>User</span>, </span>Repo</span>}
</span>
</span>  </span>def </span>reset_password</span>(</span>id</span>) </span>do
</span>    user </span>= </span>Repo</span>.</span>get_by</span>(</span>User</span>, </span>id</span>:</span> id</span>)
</span>
</span>    user
</span>    </span>|> </span>User</span>.</span>reset_password_changeset</span>()
</span>    </span>|> </span>Repo</span>.</span>update</span>()
</span>
</span>    # Then send email with Bamboo
</span>  </span>end
</span>
</span>  </span>def </span>password_reset_expiration_period</span>() </span>do
</span>    # Instead of:
</span>    # (config()[:password_reset_expiration_period] || :timer.hours(2 * 24))
</span>    </span>(</span>config</span>()[:</span>password_reset_expiration_period</span>] </span>|| </span>"</span>P2D</span>")
</span>    </span>|> </span>Duration</span>.</span>parse!</span>()
</span>  </span>end
</span>
</span>  </span>def </span>config</span>(),
</span>    </span>do: </span>Application</span>.</span>get_all_env</span>(:</span>my_app</span>, </span>__MODULE__</span>)
</span>end
</span>
</span>defmodule </span>MyApp</span>.</span>User </span>do
</span>  </span>use </span>Ecto</span>.</span>Schema
</span>  </span>alias </span>Ecto</span>.{</span>Changeset</span>, </span>UUID</span>}
</span>
</span>  </span>def </span>reset_password_changeset</span>(</span>entity</span>) </span>do
</span>    # Instead of:
</span>    # expiration_date = DateTime.add(DateTime.utc_now(), milliseconds: MyApp.password_reset_expiration_period())
</span>    expiration_date </span>= </span>Timex</span>.</span>add</span>(</span>Date</span>.</span>utc_today</span>(), </span>MyApp</span>.</span>password_reset_expiration_period</span>())
</span>
</span>    entity
</span>    </span>|> </span>Changeset</span>.</span>change</span>(</span>%</span>{
</span>      </span>password_reset_id</span>: </span>UUID</span>.</span>generate</span>(),
</span>      </span>password_reset_expiration</span>:</span> expiration_date
</span>    </span>})
</span>  </span>end
</span>end
</span></code></pre>
The configuration becomes standardized across the board as well:</p>
# config/dev.exs
</span>config </span>:</span>my_app</span>, </span>MyApp</span>.</span>Accounts</span>,
</span>  # Instead of:
</span>  # password_reset_expiration_period: :timer.hours(2 * 24)
</span>  </span>password_reset_expiration_period</span>: "</span>P2D</span>"
</span>
</span># config/prod.exs
</span>conifg </span>:</span>my_app</span>, </span>MyApp</span>.</span>Accounts</span>,
</span>  # Instead of:
</span>  # password_reset_expiration_period: :timer.minutes(15)
</span>  </span>password_reset_expiration_period</span>: "</span>PT15M</span>"
</span></code></pre>
[</span>my_app</span>."</span>MyApp.Accounts</span>"]
</span># Instead of:
</span># password_reset_expiration_period = 172800000 # :timer.hours(2 * 24)
</span># password_reset_expiration_period = 900000    # :timer.minutes(15)
</span>password_reset_expiration_period </span>= "</span>P2D</span>"
</span># or
</span>password_reset_expiration_period </span>= "</span>PT15M</span>"
</span></code></pre>
Using milliseconds is still viable if the introduction of a new format
or library is too much; however, one can write a simple and quick
space delimited format to retort:</p>
defmodule </span>Duration </span>do
</span>  </span>@</span>formats %</span>{
</span>    </span>"</span>ms</span>" </span>=> </span>1</span>,
</span>    </span>"</span>sec</span>" </span>=> </span>1_000</span>,
</span>    </span>"</span>min</span>" </span>=> </span>60 </span>* </span>1_000</span>,
</span>    </span>"</span>hour</span>" </span>=> </span>60 </span>* </span>60 </span>* </span>1_000</span>,
</span>    </span>"</span>day</span>" </span>=> </span>24 </span>* </span>60 </span>* </span>60 </span>* </span>1_000
</span>  </span>}
</span>
</span>  </span>def </span>convert</span>(</span>text</span>) </span>do
</span>    units </span>= </span>Map</span>.</span>keys</span>(</span>@</span>formats</span>)
</span>
</span>    text
</span>    </span>|> </span>String</span>.</span>trim</span>()
</span>    </span>|> </span>String</span>.</span>downcase</span>()
</span>    </span>|> </span>String</span>.</span>split</span>(" ", </span>trim</span>: </span>true</span>)
</span>    </span>|> </span>Enum</span>.</span>reduce_while</span>(</span>0</span>, </span>fn</span> token</span>,</span> acc </span>->
</span>      </span>case </span>Integer</span>.</span>parse</span>(</span>token</span>) </span>do
</span>        </span>{</span>value</span>,</span> unit</span>} </span>when</span> unit </span>in</span> units </span>->
</span>          multiplier </span>= </span>Map</span>.</span>fetch!</span>(</span>units</span>,</span> unit</span>)
</span>
</span>          </span>{:</span>cont</span>,</span> acc </span>+</span> multiplier </span>*</span> value</span>}
</span>
</span>        _ </span>-> </span>{:</span>halt</span>, :</span>error</span>}
</span>      </span>end
</span>    </span>end</span>)
</span>    </span>|> case do
</span>         </span>:</span>error </span>-> </span>:</span>error
</span>         milliseconds </span>-> </span>{:</span>ok</span>,</span> milliseconds</span>}
</span>       </span>end
</span>  </span>end
</span>end
</span>
</span>Duration</span>.</span>convert</span>("</span>1hour</span>")
</span>Duration</span>.</span>convert</span>("</span>15min 20sec</span>")
</span></code></pre>
Nonetheless, consider using a human-readable format to make date/time
configuration easier and more intuitive.</p>
>></a>
Notes</h2>
>>></a>
Avoid Months and Years</h3>
As an exercise, I tried porting ms.js</a> into an Elixir library, ex_ms</a>:</p>
import </span>Millisecond</span>, </span>only</span>: [</span>ms!</span>: </span>1</span>, </span>ms</span>: </span>1</span>]
</span>
</span>ms!</span>("</span>1h 1m 1s</span>")         </span># 86400000
</span>ms!</span>("</span>1days 12 hours</span>")   </span># 129600000
</span>ms</span>("</span>12 hours 1days</span>")    </span># :error
</span></code></pre>
What I learned from this exercise is that representing duration with
milliseconds is safe until it comes to months or years. How many days
are there in a month? How many days are there in a year? It depends
on both questions. Looking at Timex</a>, it interprets a year as 365 days
while with ChronicDuration</a> as 365.25 days. Not that the libraries are
incorrect but some precision is lost when dealing with months or
years. So when describing a duration, avoid using months or years and
use 30 and 366 days respectively for safety regardless of format or
library.</p>


A Mortician's Tale PC - Respecting The Dead
Unknown — Sun, 11 Oct 2020 00:00:00 +0000
>></a>
A Mortician's Tale</h2>
A Mortician's Tale PC</a> lets the player see how a remains is prepared
for a funeral. It has two main mechanics: embalming and cremation.
Since the player is responsible for the remains in the funeral, paying
respect to the family and deceased is required. The game loop is
simple but it evokes meditation or feelings about the physicality of
death. What importance and rituals do we place? I appreciate this game
for tackling a morbid subject which takes around an 1 hour to finish.</p>
>></a>
Thoughts</h2>
The game's short and linear experience will stay with me; however, I
do feel it is lacking to fully sell the experience or meditation. The
game does not have failure states which is fine but I cannot perceive
the quality of my work or any narrative complication. What if an
accident occurred and seeing those repercussion play out? As it
stands, I do not have to put effort in the gameplay aside from its
morbid context. When other characters in the game praise my work and
empathy, it feels disconnected.</p>
While having a simple story is fine, the player character's motivation
or story is not explained which is a shame. Is it out of charity or
poverty? Is it a stressful to terrifying job? I want to know their
perspectives. Using a silent protagonist suggest role-playing but even
some dialogue options or scenes would add color or texture to a blank
character.</p>
Aside from the story and price, those two issues prevent me from
recommending the game. While letting the player draw their own
conclusions is admirable, it feels abstract or hollow in the end
without a strong direction to guide it. This may be the game's scope
or intended limitation, so I am excited for a more polished and
cohesive game to follow.</p>


Butterfly Soup PC - A Chill Experience
Unknown — Sun, 11 Oct 2020 00:00:00 +0000
>></a>
Butterfly Soup</h2>

A visual novel about gay asian girls playing baseball and falling in love.</p>
</blockquote>
I do not agree with Butterfly Soup PC</a>'s tagline wording since it
highlights its LGBT nature rather than its strong character and
dialogue writing. Nonetheless, it is a linear slice-of-life story of
four friends that last about 3 hours. While I cannot speak for its
LGBT quality, the group dynamics and moments is what really sells it
specially with Noelle</em> and Akarsha</em>. It is a straight forward and
fun ride all throughout which is nice in a sea of complex narratives
with uncertain artistic value or quality.</p>
>></a>
Thoughts</h2>
While it is free product, I do have some minor thoughts about it as a
game or story. Since this is promoted for its romance, I did feel it a
lacking or does not have weight. I do like that the group is
supportive and that is possibly the point of the game, but it does not
give me struggles or perspectives to ponder about. This also leads to
the overarching story where there is none. It ends at a good point,
but I am curious about what follows and the changes in the dynamics.</p>
On a character basis, Noelle</em> is my favorite character alongside
Akarsha</em> where their banter feels great while Liz</em> is my favorite in
terms of design. My least favorite for me are Diya</em> and Min</em> despite
being the main romantic pairing. Min</em> is strangely too rebellious
without any added flavor aside from being protective. Diya</em> feels a
bit bland with her shyness without any complex or introspective
thoughts. With her godly athleticism, she feels like a shy main
character in a harem whom everyone revolves which is not bad but just
uninteresting. These are subjective character opinions so feel free to
disagree.</p>
For some minor points, it has cultural references which I still
enjoyed though some escaped me, so it might date itself. Features such
as background pan or screenshake made me feel dizzier so the option to
disable is appreciated. While the story is linear, some minor
branching would be nice for replayability.</p>
Nonetheless, the characters and writing is realistic and I enjoyed my
time with the game and would like to see more.</p>


Rit Journal - Initial Entry
Unknown — Tue, 06 Oct 2020 00:00:00 +0000
>></a>
Rit: Rust Flavored Git</h2>
I was looking for another learning project after my post about Rust
AVL Trees</a>. I had many ideas and possibilities but after seeing and
buying Building Git</a>, I knew I was going to make a Rust flavored git</a>.
It is ambitious and a large project possibly beyond me but a large
system allows me to think of the big picture as I work instead of
hacking away at small designs. My goal is to be able to start from an
empty project and end with pushing it to GitHub</a> and cloning it back
again without using git</code> or dogfooding</a>. The book calls it project
jit</code>, I dub mine rit</code>.</p>
While I just need to follow the majority of the logic and code in the
book, I have to adapt it to Rust</em> instead of Ruby</em> specially with
the borrow checker and static typing. Looking back on my study, I
learned how to breakdown a module into smaller modules, understand
type conversions, create expressive test suites, enhance my editor,
accelerate my Rust compilation and possibly more. It is this larger
scale that demands me to seek improvements. Even though I am guided, I
frequently question and refactor my code sometimes too eagerly.
Whenever or not I finish it, I have already learned so much but I
still want to show my commitment and perseverance.</p>
After months of working on the project, I stopped at chapter 11 or
after implementing rit status</code>. I felt I needed to break the monotony
and work on another task for a while. Although Jekyll</a> is a simple and
fine blogging engine, I felt my org-mode</a> workflow could be improved or
simplified with a new engine. Zola</a> is a Rust blogging engine and
alongside ox-hugo</a> seemed to be a viable workflow. For two weeks or so,
I ported and updated my old site with taxonomies, pagination and
styling. Afterwards, I ported most of my articles with relative ease
while applying the godsend page bundle</a> layout. Another goal was to
move away from GitHub Pages</a> to a self-hosted approach akin to hosting
on a Raspberry Pi. Hosting on Netlify</a> with manual deploys is a
practical approach for now.</p>
Now with the site reinvigorated, I feel recharged to continue with the
project. I plan to log my experience or findings with this new series
of posts or through other posts while testing this new blogging flow.
With at least 24 chapters or around 500 pages left, hopefully this can
keep in the habit of writing for a long time.</p>


Rust Bit - A Use For Deref Trait
Unknown — Sun, 04 Oct 2020 00:00:00 +0000
>></a>
Problem</h2>
I have a toy database module that lazily loads an file-based toy index
with a read, add and commit operation as shown in this snippet:</p>
use </span>std</span>::{</span>io</span>,</span> fs</span>, </span>path</span>::{</span>Path </span>,</span>PathBuf</span>}};
</span>
</span>#[</span>derive</span>(</span>Debug</span>)]
</span>struct </span>DB</span>;
</span>
</span>#[</span>derive</span>(</span>Debug</span>)]
</span>struct </span>Index</span>(</span>PathBuf, Vec</span><</span>u8</span>>);
</span>
</span>impl </span>DB </span>{
</span>    </span>pub fn </span>open</span>() -> </span>Self </span>{
</span>        </span>Self
</span>    </span>}
</span>
</span>    </span>pub fn </span>read_index</span>() -></span> Index </span>{
</span>        </span>let</span> path </span>= </span>Path</span>::</span>new</span>("</span>index.db</span>").</span>to_path_buf</span>();
</span>        </span>let</span> index_data </span>= </span>fs</span>::</span>read</span>(</span>&</span>path</span>).</span>unwrap</span>();
</span>
</span>        Index</span>(</span>path</span>,</span> index_data</span>)
</span>    </span>}
</span>}
</span>
</span>impl </span>Index </span>{
</span>    </span>pub fn </span>read</span>(</span>&</span>self</span>) -> </span>&</span>[</span>u8</span>] {
</span>        </span>self</span>.</span>1.</span>as_slice</span>()
</span>    </span>}
</span>
</span>    </span>pub fn </span>add</span>(</span>&mut </span>self</span>, </span>entry</span>: </span>Vec</span><</span>u8</span>>) {
</span>        </span>self</span>.</span>1.</span>append</span>(</span>&mut</span> entry</span>);
</span>    </span>}
</span>
</span>    </span>pub fn </span>write</span>(</span>&mut </span>self</span>) {
</span>        </span>fs</span>::</span>write</span>(</span>self</span>.</span>0</span>, </span>self</span>.</span>1</span>).</span>unwrap</span>();
</span>    </span>}
</span>}
</span></code></pre>
The challenge is to create an interface where the index can be read by
many or written exclusively like a reference. How about we return a
reference from DB::read_index</code> instead of an owned Index</code>?
Representing that as load_index</code> and lock_index</code> in the following
snippet:</p>
/// Can we return a reference of an owned object?
</span>impl </span>DB </span>{
</span>    </span>pub fn </span>load_index</span>(</span>&</span>self</span>) -> </span>&</span>Index </span>{
</span>        </span>&Self</span>::</span>read_index</span>()     </span>// error[E0515]: cannot return reference to temporary value
</span>    </span>}
</span>
</span>    </span>pub fn </span>lock_mut</span>(</span>&</span>self</span>) -> </span>&mut</span> Index </span>{
</span>        </span>&mut Self</span>::</span>read_index</span>() </span>// error[E0515]: cannot return reference to temporary value
</span>    </span>}
</span>}
</span></code></pre>
However, this will fail since the owned object is dropped by the end
of the function. What if we create a read-only and write wrapper
structs like IndexRef</code> and IndexRefMut</code> respectively? The read-only
wrapper should only expose immutable methods such as Index::read</code>
while the other exposes everything like so:</p>
/// What if we create a read/write interfaces?
</span>#[</span>derive</span>(</span>Debug</span>)]
</span>struct </span>IndexRef</span>(</span>Index</span>);
</span>
</span>#[</span>derive</span>(</span>Debug</span>)]
</span>struct </span>IndexRefMut</span>(</span>Index</span>);
</span>
</span>impl </span>DB </span>{
</span>    </span>pub fn </span>load_index</span>(</span>&</span>self</span>) -></span> IndexRef </span>{
</span>        IndexRef</span>(</span>Self</span>::</span>read_index</span>())
</span>    </span>}
</span>
</span>    </span>pub fn </span>lock_mut</span>(</span>&mut </span>self</span>) -></span> IndexRefMut </span>{
</span>        IndexRefMut</span>(</span>Self</span>::</span>read_index</span>())
</span>    </span>}
</span>}
</span>
</span>/// Immutable Wrapper
</span>impl </span>IndexRef </span>{
</span>    </span>pub fn </span>read</span>(</span>&</span>self</span>) -> </span>&</span>[</span>u8</span>] {
</span>        </span>self</span>.</span>0.</span>read</span>()
</span>    </span>}
</span>}
</span>
</span>/// Mutable Wrapper
</span>impl </span>IndexRefMut </span>{
</span>    </span>pub fn </span>read</span>(</span>&</span>self</span>) -> </span>&</span>[</span>u8</span>] {
</span>        </span>self</span>.</span>0.</span>read</span>()
</span>    </span>}
</span>
</span>    </span>pub fn </span>add</span>(</span>&mut </span>self</span>, </span>entry</span>: </span>Vec</span><</span>u8</span>>) -> </span>&</span>[</span>u8</span>] {
</span>        </span>self</span>.</span>0.</span>add</span>(</span>entry</span>)
</span>    </span>}
</span>
</span>    </span>pub fn </span>write</span>(</span>&mut </span>self</span>) {
</span>        </span>self</span>.</span>0.</span>write</span>()
</span>    </span>}
</span>}
</span></code></pre>
While it does the job, adding new methods to Index</code> require adding it
to IndexRef</code> and IndexRefMut</code> which is a maintenance issue.</p>
>></a>
Solution</h2>
A trait already exists for this case: std::ops::Deref</a> and
std::ops::DerefMut</a>. When a method does not exist on an expression, the
compiler will attempt to dereference it until it can find the method.
(See Method Call Expression</a>.) If we allow IndexRef</code> and IndexRefMut</code>
to derefence to Index</code>, we solve the maintenance issue which is
conveyed in the snippet below:</p>
let</span> meaning</span>: </span>&str = </span>"</span>42</span>";
</span>
</span>value</span>.</span>len</span>();
</span>/// Same as below since `&str` automatically derefences to `str`
</span>(</span>*</span>value</span>).</span>len</span>()
</span>
</span>/// Assuming `IndexRef` and `IndexRefMut` implements nothing
</span>impl </span>IndexRef </span>{}
</span>impl </span>IndexRefMut </span>{}
</span>
</span>let mut</span> write_index</span>:</span> IndexRefMut</span>;
</span>
</span>write_index</span>.</span>write</span>();
</span>/// Since `IndexRefMut::write` does not exist.
</span>/// The compiler attempts to derefence `write_index` and hopes to find the `.write` method.
</span>/// If `IndexRefMut` derefences to `Index`, it will.
</span>(</span>*</span>write_index</span>).</span>write</span>();
</span></code></pre>
Reading the docs, implementing both traits is straightforward:</p>
impl </span>Deref </span>for </span>IndexRef </span>{
</span>    </span>type </span>Target </span>=</span> Index</span>;
</span>
</span>    </span>fn </span>deref</span>(</span>&</span>self</span>) -> </span>&Self</span>::</span>Target </span>{
</span>        </span>&self</span>.</span>0
</span>    </span>}
</span>}
</span>
</span>impl </span>Deref </span>for </span>IndexRefMut </span>{
</span>    </span>type </span>Target </span>=</span> Index</span>;
</span>
</span>    </span>fn </span>deref</span>(</span>&</span>self</span>) -> </span>&Self</span>::</span>Target </span>{
</span>        </span>&self</span>.</span>0
</span>    </span>}
</span>}
</span>
</span>impl </span>DerefMut </span>for </span>IndexRefMut </span>{
</span>    </span>fn </span>deref_mut</span>(</span>&mut </span>self</span>) -> </span>&mut Self</span>::</span>Target </span>{
</span>        </span>&mut self</span>.</span>0
</span>    </span>}
</span>}
</span></code></pre>
To demonstrate it works:</p>
fs</span>::</span>write</span>("</span>index.db</span>", "").</span>unwrap</span>();
</span>
</span>let mut</span> db </span>= </span>DB</span>::</span>open</span>();
</span>
</span>{
</span>    </span>let</span> read_index </span>=</span> db</span>.</span>load_index</span>();
</span>    </span>println!</span>("</span>Should be empty data: </span>{:?}</span>",</span> read_index</span>.</span>read</span>());
</span>}
</span>
</span>{
</span>    </span>let mut</span> write_index </span>=</span> db</span>.</span>lock_index</span>();
</span>    write_index</span>.</span>add</span>("</span>TOTORO</span>".</span>as_bytes</span>());
</span>    write_index</span>.</span>write</span>();
</span>}
</span>
</span>{
</span>    </span>let mut</span> updated_index </span>=</span> db</span>.</span>lock_index</span>();
</span>    </span>println!</span>("</span>Should have data: </span>{:?}</span>",</span> updated_index</span>.</span>read</span>());
</span>
</span>    </span>/// Should not be allowed because `db` is already mutably borrowed which is intended
</span>    </span>let</span> read_index </span>=</span> db</span>.</span>load_index</span>();
</span>}
</span></code></pre>
Another advantage is that we can implement the std::ops::Drop</a> on
IndexRefMut</code> so that Index::write</code> is called whenever it is dropped
to automatically sync the data. This is also a good separation of
concern to keep Index</code> pure while the wrappers implement convenience
or ergonomics as shown below:</p>
impl </span>Drop </span>for </span>IndexRefMut </span>{
</span>    </span>fn </span>drop</span>(</span>&mut </span>self</span>) {
</span>        </span>self</span>.</span>0.</span>write</span>();
</span>    </span>}
</span>}
</span>
</span>{
</span>    </span>let mut</span> write_index </span>=</span> db</span>.</span>lock_index</span>();
</span>    write_index</span>.</span>add</span>("</span>TOTORO</span>".</span>as_bytes</span>());
</span>    </span>// write_index.write(); // Should automatically write when dropped
</span>}
</span></code></pre>
Although this is a neat trick, it is generally discouraged to
implement Deref</code> as it changes a well-known behavior</a>. In our case, it
is intentional but do checkout the Rust API guidelines</a> for other
things to be aware of.</p>
Tidbit Code</a></p>
>></a>
Notes</h2>
>>></a>
File Locking</h3>
While we preserve the borrow rules, it does not work when multiple
process or threads access the file. We can implement a simple file
locking mechanism where we atomically try to create a file and remove
it when dropping. If another process sees the lock file, then the
lock operation should fail. Instead of using Path::exists</a> then
fs::write</a>, we can do with fs::OpenOptions</a> to do it in one atomic step
and avoid a filesystem race condition</a>. A simple lock can be amended
to IndexRefMut</code> like so:</p>
use </span>std</span>::{</span>fs</span>::{</span>self</span>,</span> OpenOptions</span>}}</span>n</span>;
</span>
</span>impl </span>DB </span>{
</span>    </span>pub fn </span>lock_index</span>(</span>&mut </span>self</span>) -></span> Index </span>{
</span>        </span>OpenOptions</span>::</span>new</span>()
</span>            </span>.</span>write</span>(</span>true</span>)
</span>            </span>.</span>create_new</span>(</span>true</span>)
</span>            </span>.</span>open</span>("</span>index.db.lck</span>")
</span>            </span>.</span>unwrap</span>();
</span>
</span>        IndexRefMut</span>(</span>Self</span>::</span>read_index</span>())
</span>    </span>}
</span>}
</span>
</span>
</span>impl </span>Drop </span>for </span>IndexRefMut </span>{
</span>    </span>fn </span>drop</span>(</span>&mut </span>self</span>) {
</span>        </span>self</span>.</span>0.</span>write</span>();
</span>        </span>fs</span>::</span>remove_file</span>("</span>index.db.lck</span>").</span>ok</span>();
</span>    </span>}
</span>}
</span></code></pre>
>>></a>
is_changed Flag</h3>
While syncing the data on drop is nice, it should only sync when data
is changed. We could add an is_changed</code> flag to IndexRefMut</code> so
that it is true</code> whenever Index.add</code> is called and likewise false</code>
for Index.write</code>. It is easily done with:</p>
struct </span>IndexRefMut</span>(</span>Index, bool</span>);
</span>
</span>impl </span>DB </span>{
</span>    </span>pub fn </span>lock_index</span>(</span>&</span>self</span>) -></span> IndexRefMut </span>{
</span>        IndexRefMut</span>(</span>Self</span>::</span>read_index</span>(), </span>false</span>)
</span>    </span>}
</span>}
</span>
</span>impl </span>IndexRefMut </span>{
</span>    </span>pub fn </span>add</span>(</span>&mut </span>self</span>, </span>entry</span>: </span>Vec</span><</span>u8</span>>) {
</span>        </span>self</span>.</span>0.</span>add</span>(</span>entry</span>);
</span>        </span>self</span>.</span>1 </span>= true</span>;
</span>    </span>}
</span>
</span>    </span>pub fn </span>write</span>(</span>&mut </span>self</span>) {
</span>        </span>self</span>.</span>0.</span>write</span>();
</span>        </span>self</span>.</span>1 </span>= false</span>;
</span>    </span>}
</span>}
</span>
</span>impl </span>Drop </span>for </span>IndexRefMut </span>{
</span>    </span>fn </span>drop</span>(</span>&mut </span>self</span>) {
</span>        </span>if self</span>.</span>1 </span>{
</span>            </span>self</span>.</span>0.</span>write</span>();
</span>        </span>}
</span>    </span>}
</span>}
</span></code></pre>
We could also implement it on Index</code> itself but it is nice that we
can keep Index</code> pure.</p>
>>></a>
RefCell</h3>
It is valid to use a RefCell</a> to store/cache the data on load and then
use RefCell.borrow</code> and RefCell.borrow_mut</code> for load_index</code> and
lock_index</code> respectively.</p>
use </span>std</span>::</span>cell</span>::{</span>RefCell</span>,</span> Ref</span>,</span> RefMut</span>};
</span>
</span>#[</span>derive</span>(</span>Debug</span>)]
</span>struct </span>DB</span>(</span>Option</span><</span>RefCell</span><</span>Index</span>>>);
</span>
</span>impl </span>DB </span>{
</span>    </span>pub fn </span>load_index</span>(</span>&</span>self</span>) -> </span>Ref</span><</span>Index</span>> {
</span>        </span>if let Some</span>(</span>index_cell</span>) </span>= self</span>.</span>0.</span>as_ref</span>() {
</span>            </span>return</span> index_cell</span>.</span>borrow</span>();
</span>        </span>}
</span>
</span>        </span>self</span>.</span>0 </span>= Some</span>(</span>RefCell</span>::</span>new</span>(</span>Self</span>::</span>read_index</span>()));
</span>        </span>self</span>.</span>0.</span>as_ref</span>().</span>unwrap</span>().</span>borrow</span>()
</span>    </span>}
</span>
</span>    </span>pub fn </span>lock_mut</span>(</span>&mut </span>self</span>) -> </span>RefMut</span><</span>Index</span>> {
</span>        </span>if let Some</span>(</span>index_cell</span>) </span>= self</span>.</span>0.</span>as_ref</span>() {
</span>            </span>return</span> index_cell</span>.</span>borrow_mut</span>();
</span>        </span>}
</span>
</span>        </span>self</span>.</span>0 </span>= Some</span>(</span>RefCell</span>::</span>new</span>(</span>Self</span>::</span>read_index</span>()));
</span>        </span>self</span>.</span>0.</span>as_ref</span>().</span>unwrap</span>().</span>borrow_mut</span>()
</span>    </span>}
</span>}
</span></code></pre>
Although this works, we are unnecessarily giving away compile time
safety compared to our solution.</p>


About Me
Unknown — Sat, 03 Oct 2020 00:00:00 +0000
Francis Murillo</h2>
I hope you found my articles useful or helpful if you are here. I
started this blog so I can practice writing and contribute meaningful
articles. As a long time Emacs</a> user, I wanted to reflect that in my
design by making one based on my own Emacs</em> theme. Built with Zola</a> and
org-mode</a>, I proudly settled on a workflow and setup that is as simple as
writing a natural org file although the CSS needs work.</p>
Currently, I am a full-time Elixir</a> developer for a small blockchain
company proving and promoting its value. With my spare time, I study and
dabble in Rust</a> to get back to my roots as a frustrated mathematician or
computer scientist. Other than that, I am a cat person, loves artistic
movies, challenging video games, and well written cartoons.</p>
Nonetheless, thank you for reading any of my articles and hope to
produce more quality writing in the future.</p>


Outer Wilds - On Acceptance
Unknown — Sun, 22 Mar 2020 00:00:00 +0000
>></a>
Outer Wilds</h2>
Outer Wilds</a> is a space exploration game where you launch your tiny
rocket and explore various planets to uncover the mysteries of the
galaxy. The game has a remarkable physics engine where flying in space
with an astronaut suit or the ship feels kinesthetically pleasing. Above
all, the game is artistic and meaningful with its overall message and
gameplay. It nails its theme of discovery and exploration by giving the
player all the tools upfront and not guiding the player in their journey
which is a rarity. Although the game is not for everyone specially who
are susceptible to motion sickness, this game deserves to be played and
experienced.</p>
While the game is primarily about the joy of space exploration, the game
resonated with me differently. During my first playthrough, I launched
into space and delightfully agonized which planet to explore. Little did
I know, I first choose and crashed into the sea of the green planet,
Giant's Deep</a>, then trying to find safety in land among the strong
cyclones and ocean currents. Wherever I escaped to next after some time
exploring, I heard an audio cue that lasted for a minute; then, the Sun
collapsed onto itself and transformed to a bright blue supernova slowly
consuming everything. I stood fixated in awe, took a deep breath and
kept my eyes open.</p>
  </p>
Death did not take me but was instead sent back in time right after the
start of the game. Why did the Sun explode? Why is the player sent back
in time when it does? Can this supernova be stopped or the home planet
saved? While the player is free to roam the galaxy, the Sun's time limit
forces numerous and unavoidable deaths which is heightened through the
first person view/experience. Over time, the player accepts death as a
natural part of the game, not simply an error/failure state that they
need to respawn or reload state.</p>
While most players will solve this unpleasant mystery in the hopes of
saving the galaxy, perhaps the opposite is also a valid response. Death
should instinctively be avoided; however, the game with its technical
and spatial boundary does not let the player run away. It forces the
player to face it, yet it is not a painful experience. Neither a
tragedy nor conflict but the natural end of the solar system. To behold
that beautiful and radiant moment feels both religious and peaceful.</p>
All this emotion was drawn out in the first loop, such a game is a rare
and personal gem. Whether players or readers felt differently, this game
not just invokes the fear of death, but also its acceptance through its
story and message that I want to discuss. To be clear, life and time is
precious and being more open to its end is a valuable insight.</p>
SPOILERS AHEAD</strong></p>
>></a>
Acceptance</h2>
To answer why the Sun is dying, the alien technology and knowledge left
behind the precursor race, Nomai</a>, may lead to it. Indeed, our newly
graduated Heartian</a> astronaut protagonist is the only one equipped to
translate their language and thus understand their legacy. As the player
uncovers more about them and Ash Twin Project</a>, it is found the Nomai</em>
does not have a clear answer themselves. While other Nomai</em> clans
exists, the ones that found themselves in the Hearthian</em> galaxy
followed a signal older than universe itself known as Eye of the
Universe</a>. The incidental time loop and surviving technology were all in
service of finding this elusive origin. Even if the player discovers a
way to travel to it which is intended endpoint of the game, the player
is no closer to stopping the threat. In fact, the Nomai</em> would escape
themselves and leave the danger behind once they found the coordinates.
Although the Nomai</em> tried and failed to force the Sun to explode as a
power source, the Sun is simply at the end of its life.</p>
If the Nomai</em> do not have answers or powers to avert the crisis, what
then can stop the death of a solar system? Should the player place their
hope in the unknown Eye of the Universe</em> can offer? Rather, if
something is naturally meant to die, do we have the right to prevent it
from doing so? The Sun is dying and the Hearthians</em> should have been
evacuated if there was more time. The Sun will die, the Hearthians</em>
will follow and so will the player character. One could read despair and
defeat, but perhaps the game is about accepting the inevitable or that
not all things can be saved.</p>
  </p>
As a final hope, when the player arrives at Eye of the Universe</em>, the
Sun is far beyond the stars and possibly beyond danger. Inside however,
it is revealed that other solar systems or galaxies are dying indicating
the death of the universe, not just of the player's solar system. At the
end of the universe, the game lets the player sit by a campfire and
listen to the main theme as the new universe comes into light. To
emphasize, although the solar system collapses and the player dies which
contradicts the expectation of a lone hero or survivor, the universe and
life will eventually find a way. Death is not the end.</p>
  </p>
In light of the ending, the signal that is older than the universe or
the Eye of the Universe</em> itself becomes interesting. The universe has
been possibly recreated several times and each time an observer sent
waves of harmony for the succeeding generation to remember and
ultimately find as the Nomai</em> and Hearthian</em> did together. If so and
one thinks of the time loop as a hindrance to one's personal or
emotional well-being, then it could be read as overcoming depression,
anxiety, pain, suffering and so on. As a personal reading, inescapable
and perpetual deaths tap into my anxieties and stopping the loop is
overcoming it; finishing the game becomes a therapeutic experience.</p>
Related to that is Gabbro</a> who is the most relaxed Outer Wilds Venture</a>
member and more importantly also experiences the time loop like the
protagonist. Unlike the expected player, he is at peace with the
impending doom and even teaches the player to meditate through it which
is represented by the Meditate Until Next Loop</em> option in the pause
screen. Happily surrendering to the next loop seems to be a form of
suicide without this option. If one needed to end the loop prematurely,
suicide by suffocating or crashing seems a viable strategy. With this
option however, it is not framed as such but instead as a calm
acceptance. Like with the other members though, he does not act on it
and lounges around making his peace as a lazy surrender than an
enlightenment. Still, his relaxed demeanor and personality contrasts the
player's anxieties.</p>
  </p>
Between the Hearthian</em> explorers and Nomai</em> thinkers, by making the
player embody the travelers spirit in this time loop, the focus shifts
on exploring the world for an answer than inventing one. Given the short
duration of the loop, knowledge and experience may seem futile but each
bit in each iteration add up. For a Hearthian</em> specially, each loop is
a journey, each journey a precious experience. One last slight reach for
the ending, it is the Nomai</em> who are thankful for the journey while
playing around a campfire as the new universe comes to life. If the
Nomai</em> knew Eye of the Universe</em> would lead them to the end, would
they have been so obsessed in chasing it across the stars? As an
Hearthian</em>, the journey to the end of the world is romantic.</p>
The strongest support for the theme comes with the loop itself. Making
the loop a repeated short cycle instead of a looming deadline is a
brilliant decision for the feeling and message. If it were a long
deadline or cycle, the feel and weight of death diminishes the longer it
looms in the distance. By being constantly reminded of the destruction
and impact of the end, it feels closer and personal how much it holds
the player. By making sure the player experiences it once, the white
explosion at the end contextualizes the Sun's nova. In changing the
scope and scale, as the loop ends for the player, a new loop begins for
the universe.</p>
  </p>
In simple ways, it is amazing how naturally this game taps into
anxieties of death. Other points can be made about the game such as the
physics, ships, planets, Nomai</em> race, Hearthian</em> explorer,
instruments, quantum objects, ethics and so on; but above all through
its restraint and commitment, it was able to deliver a meaningful and
positive experience for me and hopefully other players as well. What
more can you ask of a game?</p>


Elixir Musing - GraphQL Over A Message Queue
Unknown — Mon, 17 Feb 2020 00:00:00 +0000
>></a>
Introspection</h2>
After releasing an Elixir</a> project to production, it is time for
introspection. Context, the project uses a microservice architecture
over a nsq</a> message queue where each service is mostly an umbrella
phoenix</a> app that offers a GraphQL interface via absinthe</a>. While I am
happy with the language and GraphQL in general, I am not a fan of the
complexity that microservices implied. Grievances aside, among the many
lessons and insights earned, one of them was worth some research.</p>
The choice for using nsq</code> over rabbitmq</a>, kafka</a>, zeromq</a> or nats</a> was
primarily its lightweight nature because of the project's small
messaging requirement. With an offbeat decision, does nsq</code> have
language support? Thankfully, it does with elixir_nsq</a> but using it
directly is clunky for an standard application. Good news, conduit</a> is an
library for abstracting message queues inspired by plug</a> that looks nice.
Bad news, it does not support nsq</code>. After much time and thought, the
best plan was to stick with conduit</code> and attempt to write an adapter
library. If the venture failed, the popular rabbitmq</code> is always a
fallback. Thankfully, conduit_nsq</a> was born and thus nsq</code> stays.</p>
Reviewing the workflow, so when one service needs something from
another, the following steps are taken:</p>

Pick a topic say product</code></li>
Create a handler for product_request</code> in the responding
service.</li>
Implement the domain method for that handler</li>
Implement a blocking method that publishes product_request</code> and
waits for product_response</code> in the asking service</li>
Create a handler for waiting on product_response</code> and pass the
data to unblock the method</li>
</ol>
Below is a simplified snippet of the changes needed. (Feel free to skim
it and get a general understanding of the structure and form)</p>
# responder/broker.ex
</span>configure </span>do
</span>  queue</span>("</span>product_response</span>")
</span>end
</span>
</span>incoming </span>Receiver </span>do
</span>  subscribe</span>(:</span>product_request</span>, </span>ProductRequest</span>, </span>topic</span>: "</span>product_request</span>", </span>channel</span>: "</span>responder</span>")
</span>end
</span>
</span>outgoing </span>do
</span>  publish</span>(:</span>product_response</span>, </span>topic</span>: "</span>product_response</span>")
</span>end
</span>
</span># responder/product_request.ex
</span>defmodule </span>Responder</span>.</span>ProductRequest </span>do
</span>  </span>use </span>Conduit</span>.</span>Subscriber
</span>
</span>  </span>def </span>process</span>(</span>%</span>Conduit</span>.</span>Message</span>{</span>correlation_id</span>:</span> correlation_id</span>, </span>body</span>:</span> data</span>} </span>=</span> message</span>,</span> _opts</span>) </span>do
</span>    types </span>=</span> %</span>{</span>product_id</span>: :</span>string</span>}
</span>
</span>    data </span>=
</span>      </span>{</span>%</span>{},</span> types</span>}
</span>      </span>|> </span>Ecto</span>.</span>Changeset</span>.</span>cast</span>(</span>data</span>, </span>Map</span>.</span>keys</span>(</span>types</span>))
</span>      </span>|> </span>Ecto</span>.</span>Changeset</span>.</span>validate_required</span>(</span>Map</span>.</span>keys</span>(</span>types</span>))
</span>      </span>|> case do
</span>        %</span>Ecto</span>.</span>Changeset</span>{</span>valid?</span>: </span>true</span>} </span>=</span> changeset </span>->
</span>          %</span>{</span>product_id</span>:</span> product_id</span>} </span>= </span>Ecto</span>.</span>Changeset</span>.</span>apply_changes</span>(</span>changeset</span>)
</span>
</span>          </span>Responder</span>.</span>Products</span>.</span>get_by_id</span>(</span>product_id</span>)
</span>
</span>        _ </span>->
</span>          </span>nil
</span>      </span>end
</span>
</span>    %</span>Conduit</span>.</span>Message</span>{}
</span>    </span>|> </span>Conduit</span>.</span>Message</span>.</span>put_body</span>(</span>data</span>)
</span>    </span>|> </span>Conduit</span>.</span>Message</span>.</span>put_correlation_id</span>(</span>correlation_id</span>)
</span>    </span>|> </span>Responder</span>.</span>Broker</span>.</span>publish</span>(:</span>product_response</span>)
</span>
</span>    message
</span>  </span>end
</span>end
</span>
</span># receiver/broker.ex
</span>configure </span>do
</span>  queue</span>("</span>product_request</span>")
</span>end
</span>
</span>incoming </span>Receiver </span>do
</span>  subscribe</span>(:</span>product_response</span>, </span>ProductResponse</span>, </span>topic</span>: "</span>product_response</span>", </span>channel</span>: "</span>receiver</span>")
</span>end
</span>
</span>outgoing </span>do
</span>  publish</span>(:</span>product_request</span>, </span>topic</span>: "</span>product_request</span>")
</span>end
</span>
</span># receiver/client.ex
</span>defmodule </span>Receiver</span>.</span>Client </span>do
</span>  </span>def </span>get_product_by_id</span>(</span>product_id</span>) </span>do
</span>    correlation_id </span>= </span>Ecto</span>.</span>UUID</span>.</span>generate</span>()
</span>
</span>    </span>{:</span>ok</span>,</span> _</span>} </span>= </span>Registry</span>.</span>register</span>(</span>Receiver</span>.</span>Client</span>.</span>Registry</span>,</span> correlation_id</span>, :</span>get_product_by_id</span>)
</span>
</span>    %</span>Conduit</span>.</span>Message</span>{}
</span>    </span>|> </span>Conduit</span>.</span>Message</span>.</span>put_body</span>(</span>%</span>{</span>product_id</span>:</span> product_id</span>})
</span>    </span>|> </span>Conduit</span>.</span>Message</span>.</span>put_correlation_id</span>(</span>correlation_id</span>)
</span>    </span>|> </span>Broker</span>.</span>publish</span>(:</span>product_request</span>)
</span>
</span>    product </span>= receive do
</span>      data </span>-></span> data
</span>    </span>after
</span>      </span>:</span>timer</span>.</span>seconds</span>(</span>5</span>) </span>-> nil
</span>    </span>end
</span>
</span>    </span>:</span>ok </span>= </span>Registry</span>.</span>unregister</span>(</span>Receiver</span>.</span>Client</span>.</span>Registry</span>,</span> correlation_id</span>)
</span>
</span>    product
</span>  </span>end
</span>end
</span>
</span># receiver/product_response.ex
</span>defmodule </span>Receiver</span>.</span>ProductResponse </span>do
</span>  </span>use </span>Conduit</span>.</span>Subscriber
</span>
</span>  </span>def </span>process</span>(</span>%</span>Conduit</span>.</span>Message</span>{</span>correlation_id</span>:</span> correlation_id</span>, </span>body</span>:</span> data</span>} </span>=</span> message</span>,</span> _opts</span>) </span>do
</span>    types </span>=</span> %</span>{</span>name</span>: :</span>string</span>, </span>price</span>: :</span>float</span>}
</span>
</span>    </span>{</span>%</span>{},</span> types</span>}
</span>    </span>|> </span>Ecto</span>.</span>Changeset</span>.</span>cast</span>(</span>data</span>, </span>Map</span>.</span>keys</span>(</span>types</span>))
</span>    </span>|> </span>Ecto</span>.</span>Changeset</span>.</span>validate_required</span>(</span>Map</span>.</span>keys</span>(</span>types</span>))
</span>    </span>|> case do
</span>         %</span>Ecto</span>.</span>Changeset</span>{</span>valid?</span>: </span>true</span>} </span>=</span> changeset </span>->
</span>           product </span>= </span>Ecto</span>.</span>Changeset</span>.</span>apply_changes</span>(</span>changeset</span>)
</span>
</span>           </span>case </span>Registry</span>.</span>lookup</span>(</span>Receiver</span>.</span>Client</span>.</span>Registry</span>,</span> correlation_id</span>) </span>do
</span>             </span>[{</span>pid</span>, :</span>get_product_by_id</span>}] </span>->
</span>               send</span>(</span>pid</span>,</span> product</span>)
</span>
</span>             _ </span>->
</span>               </span>nil
</span>           </span>end
</span>
</span>         _ </span>->
</span>           </span>nil
</span>       </span>end
</span>
</span>    message
</span>  </span>end
</span>end
</span></code></pre>
The code above is functional and works; however, I do have some
concerns:</p>

Unstructured Data</strong></strong></li>
</ul>
Before using the message body, it has to be casted and validated from a
string map into an atom map. The team standard is to treat string maps
as unstructured or unsafe data before being used or pattern matched even
if it comes from the queue but is still boilerplate.</p>

Broker Setup</strong></strong></li>
</ul>
Setting up the topics and identical routes is error prone if not
careful. Being confused with the topics used in receiver or handler is
not a rare occurrence which led to some self-loathing.</p>

Request/Response Boilerplate</strong></strong></li>
</ul>
Creating two small modules for new service interactions seems menial.
Not to say that it does not make sense just that creating multiple tiny
files adds up and is a maintenance cost.</p>
Rather, what this reminds me is the comparison between GraphQL and REST.
I am not here to argue their merits but I favor GraphQL for establishing
a typed contract between the consumer and provider. The problems above
can be solved if GraphQL can work with a message queue transport.
Fascinating, can it be done?</p>
NOTE:</strong></strong> Other communication library or technologies exist such as grpc</a>
or thrift</a> to solve the problem; however, this increases the technology
stack which is exciting but a risk. While this exercise or idea may not
be the most efficient for the task, it works within the constraints and
serves as a different perspective of GraphQL.</p>
>></a>
Proof Of Concept</h2>
Programmers are optimistic so whether it can be done is yes if not a
matter of time; however more than the binary answer, our concern is how
does this style or design affect productivity, stability or efficiency
of a project. To test this idea, we will be creating a sandbox project
with four services for a hypothetical product API:</p>

Accounts Service</strong></strong> - Holds user data</li>
Products Service</strong></strong> - Stores product data</li>
Transactions Service</strong></strong> - Contains the transactions made by a user</li>
API Gateway</strong></strong> - The main GraphQL interface for integrating the
services</li>
</ul>
The quick sketch of the operations and interactions are:</p>

A user will register through the registerUser</code> mutation via the
account service</li>
That user can look at the available products at products</code> query via
product service</li>
The user can pick several products by id and use the
createTransaction</code> mutation via the transactions service. To make
this interesting, this calls the product service to fetch the current
price of goods.</li>
</ol>
In building the project, I took the following steps:</p>

Create the account service and users</code> query</li>
Build the gateway and connect to the other service</li>
Implement and expose the registerUser</code> mutation</li>
Create the product service, the products</code> query, and integrate with
it the gateway</li>
Implement the createTransaction</code> mutation</li>
Add User.transactions</code> field</li>
Add GraphiQL to each service</li>
</ol>
You can follow along or checkout the working code</a> and try it out for
yourself which uses docker-compose</a> for ease. Nonetheless, I will be
working through the design and implementation process starting with the
basic account service.</p>
NOTE:</strong></strong> I will only show relevant code in building the research.
Things like the migrations, changesets or domain functions will pad the
article and is not that interesting. I also assume you are familiar with
implementing your own absinthe</code> and phoenix</code> GraphQL API; however, I
will not assume you know conduit</code> or nsq</code>.</p>
>></a>
Account Service</h2>
Our initial objective is to setup the account service which can be
easily setup via mix new --sup account_service</code>. We then set up the
following:</p>

ecto</a> with postgresql</a></li>
</ul>
We need a database or storage to store the users.</p>
sudo systemctl start postgresql   # Assuming a standard postgres
</span>sudo -u postgres -i
</span>
</span>createuser -d -P mq_user          # Enter passwod: mq_pass
</span>createdb mq_db -O mq_user
</span></code></pre>
config </span>:</span>account_service</span>, </span>AccountService</span>.</span>Repo</span>,
</span>  </span>username</span>: "</span>mq_user</span>",
</span>  </span>password</span>: "</span>mq_pass</span>",
</span>  </span>database</span>: "</span>mq_db</span>",
</span>  </span>hostname</span>: "</span>localhost</span>",
</span>  </span>port</span>: </span>5432</span>,
</span>  </span>show_sensitive_data_on_connection_error</span>: </span>true</span>,
</span>  </span>pool_size</span>: </span>10
</span></code></pre>
While I could have used an Agent</code>, I wanted to simulate changeset
errors when doing mutation testing.</p>

conduit</a> with nsqd</a></li>
</ul>
We need a running nsqd</code> at port 14150</code>:</p>
nsqd --tcp-address=0.0.0.0:14150 --http-address=0.0.0.0:14151
</span></code></pre>
While you can use the default port 4150</code>, I prefer to run them in a
different port in every project.</p>
config </span>:</span>account_service</span>, </span>AccountService</span>.</span>Broker</span>,
</span>  </span>adapter</span>: </span>ConduitNSQ</span>,
</span>  </span>producer_nsqds</span>: ["</span>localhost:14150</span>"],
</span>  </span>nsqds</span>: ["</span>localhost:14150</span>"]
</span></code></pre>
Setting up the broker.ex</code> with a JSON parsing is easy to setup:</p>
# account_service/broker.ex
</span>defmodule </span>AccountService</span>.</span>Broker </span>do
</span>  </span>use </span>Conduit</span>.</span>Broker</span>, </span>otp_app</span>: :</span>account_service
</span>
</span>  </span>@</span>channel </span>"</span>account_service</span>"
</span>
</span>  pipeline </span>:</span>serialize </span>do
</span>    plug</span>(</span>Conduit</span>.</span>Plug</span>.</span>Wrap</span>)
</span>    plug</span>(</span>Conduit</span>.</span>Plug</span>.</span>Encode</span>, </span>content_encoding</span>: "</span>json</span>")
</span>  </span>end
</span>
</span>  pipeline </span>:</span>deserialize </span>do
</span>    plug</span>(</span>Conduit</span>.</span>Plug</span>.</span>Decode</span>, </span>content_encoding</span>: "</span>json</span>")
</span>    plug</span>(</span>Conduit</span>.</span>Plug</span>.</span>Unwrap</span>)
</span>  </span>end
</span>end
</span>
</span># config.exs
</span>config </span>:</span>conduit</span>, </span>Conduit</span>.</span>Encoding</span>, [
</span>  </span>{"</span>json</span>", </span>Absinthe</span>.</span>Conduit</span>.</span>Json</span>}   </span># Custom JSON encoding to use Jason
</span>]
</span></code></pre>

absinthe</a></li>
</ul>
Personally, it feels strange install absinthe</code> without phoenix</code> or
plug</code> but we will install them near the end.</p>
With a fresh schema.ex</code>, we can start with our User</code> type and users</code> query:</p>
# account_service/schema.ex
</span>object </span>:</span>user </span>do
</span>  description</span>("</span>User of this wonderful platform</span>")
</span>
</span>  field</span>(:</span>id</span>,</span> non_null</span>(:</span>id</span>), </span>description</span>: "</span>User ID</span>")
</span>  field</span>(:</span>email</span>,</span> non_null</span>(:</span>string</span>), </span>description</span>: "</span>User email</span>")
</span>  field</span>(:</span>first_name</span>,</span> non_null</span>(:</span>string</span>), </span>description</span>: "</span>User first name</span>")
</span>  field</span>(:</span>last_name</span>,</span> non_null</span>(:</span>string</span>), </span>description</span>: "</span>User last name</span>")
</span>end
</span>
</span>query </span>do
</span>  field</span>(:</span>users</span>,</span> non_null</span>(</span>list_of</span>(</span>non_null</span>(:</span>user</span>))),
</span>    </span>resolve</span>: </span>fn</span> _</span>,</span> _</span>,</span> _ </span>->
</span>      </span>{:</span>ok</span>, </span>AccountService</span>.</span>get_users</span>()}   </span># Basically Repo.all(User)
</span>    </span>end
</span>  </span>)
</span>end
</span></code></pre>
The corresponding user.ex</code> schema and migration easily follows from
that. To check if its all fired up:</p>
mix do ecto.drop, ecto.setup    # ecto.setup = [ecto.create, ecto.migrate, ecto.seed]
</span>
</span>iex -S mix run --no-halt
</span>iex> AccountService.Repo.all(AccountService.User)
</span></code></pre>
>></a>
Absinthe via Conduit</h2>
Taking our cue from the motivation, we will expose a request and
response topic:</p>
# account_service/broker.ex
</span>configure </span>do
</span>  queue</span>("</span>account_graphql_response</span>")
</span>end
</span>
</span>incoming </span>AccountService </span>do
</span>  pipe_through</span>([:</span>deserialize</span>])
</span>
</span>  subscribe</span>(:</span>graphql_request</span>, </span>GraphQLRequest</span>, </span>topic</span>: "</span>account_graphql_request</span>", </span>channel</span>: </span>@</span>channel</span>)
</span>end
</span>
</span>outgoing </span>do
</span>  pipe_through</span>([:</span>serialize</span>])
</span>
</span>  publish</span>(:</span>graphql_response</span>, </span>topic</span>: "</span>account_graphql_response</span>")
</span>end
</span></code></pre>
Working through the snippet above, we receive on the
account_graphql_request</code> topic and send on the
account_graphql_response</code> topic as our convention. More importantly,
the request handler module is AccountService.GraphQLRequest</code> where the
magic happens:</p>
defmodule </span>AccountService</span>.</span>GraphQLRequest </span>do
</span>  </span>use </span>Conduit</span>.</span>Subscriber
</span>
</span>  </span>def </span>process</span>(</span>%</span>Conduit</span>.</span>Message</span>{</span>correlation_id</span>:</span> correlation_id</span>, </span>body</span>:</span> body</span>} </span>=</span> message</span>,</span> _opts</span>) </span>do
</span>    types </span>=</span> %</span>{</span>query</span>: :</span>string</span>, </span>variables</span>: :</span>map</span>, </span>context</span>: :</span>map</span>}
</span>
</span>    </span>{</span>%</span>{},</span> types</span>}
</span>    </span>|> </span>Ecto</span>.</span>Changeset</span>.</span>cast</span>(</span>body</span>, </span>Map</span>.</span>keys</span>(</span>types</span>))
</span>    </span>|> </span>Ecto</span>.</span>Changeset</span>.</span>validate_required</span>([:</span>query</span>])
</span>    </span>|> case do
</span>         %</span>Ecto</span>.</span>Changeset</span>{</span>valid?</span>: </span>true</span>} </span>=</span> changeset </span>->
</span>           data </span>= </span>Ecto</span>.</span>Changeset</span>.</span>apply_changes</span>(</span>changeset</span>)
</span>
</span>           query </span>= </span>Map</span>.</span>fetch!</span>(</span>data</span>, :</span>query</span>)
</span>           variables </span>= </span>Map</span>.</span>get</span>(</span>data</span>, :</span>variables</span>,</span> %</span>{})
</span>           context </span>= </span>Map</span>.</span>get</span>(</span>data</span>, :</span>context</span>,</span> %</span>{})
</span>
</span>           # NEXT: What to do here?
</span>
</span>         changeset </span>->
</span>           </span>nil
</span>       </span>end
</span>
</span>    # Need to return the message to mark it as acknowledged
</span>    message
</span>  </span>end
</span>end
</span></code></pre>
So for every message, we assume it has a payload/body of a string map
with a required query</code> string and optional variables</code> and context</code>
map. With an schemaless changeset</a>, those fields are easily extracted. We
also require it to have a correlation_id</code> to have a way to return it to
the sender.</p>
Browsing around the absinthe</code> documentation, I found Absinthe.run</a> and
quickly capitalized:</p>
{:</span>ok</span>,</span> response</span>} </span>= </span>Absinthe</span>.</span>run</span>(</span>query</span>, </span>AccountService</span>.</span>Schema</span>, </span>variables</span>:</span> variables</span>, </span>context</span>:</span> context</span>)
</span></code></pre>
Excellent, we just need to return the data back which is easily done:</p>
%</span>Conduit</span>.</span>Message</span>{}
</span>|> </span>Conduit</span>.</span>Message</span>.</span>put_body</span>(</span>response</span>)
</span>|> </span>Conduit</span>.</span>Message</span>.</span>put_correlation_id</span>(</span>correlation_id</span>)
</span>|> </span>AccountService</span>.</span>Broker</span>.</span>publish</span>(:</span>graphql_response</span>)    </span># The topic ID in broker.ex, not the topic itself
</span></code></pre>
Testing it out, we run mix run --no-halt</code> again and use the logging
commands nsq_tail</code> to watch any messages being published in the GraphQL
topics:</p>
# In one terminal
</span>nsq_tail --nsqd-tcp-address=0.0.0.0:14150 --topic=account_graphql_request
</span># In another terminal
</span>nsq_tail --nsqd-tcp-address=0.0.0.0:14150 --topic=account_graphql_response
</span></code></pre>
Once the listeners are in place, we can curl</code> the HTTP publish</a> endpoint
with this JSON payload to simulate a call:</p>
{
</span>  </span>"</span>body</span>": {
</span>    </span>"</span>query</span>": "</span>query { users { email firstName lastName } }</span>"
</span>  </span>},
</span>  </span>"</span>fields</span>": {
</span>    </span>"</span>correlation_id</span>": "</span>ABC</span>"
</span>  </span>},
</span>  </span>"</span>headers</span>": {}
</span>}
</span></code></pre>
curl -i -H Content-Type\:\ application/json -XPOST http\://localhost\:14151/pub\?topic\=account_graphql_request -d \{\"body\"\:\ \{\"query\"\:\"query\ \{\ users\ \{\ email\ firstName\ lastName\ \}\ \}\"\,\ \"variables\"\:\ \{\"a\"\:\ 1\}\}\,\ \"fields\"\:\ \{\"correlation_id\"\:\ 1\}\,\ \"headers\"\:\ \{\}\}
</span></code></pre>
The extra fields and wrapping of the query</code> field is because of the
JSON encoding pipeline in Broker</code> which is also the way to set a
header/meta field such as correlation_id</code> in the message itself. More
importantly, a snippet from the output of account_graphql_response</code>
proves that it works.</p>
{
</span>  </span>"</span>body</span>": {
</span>    </span>"</span>data</span>": {
</span>      </span>"</span>users</span>": [
</span>        </span>{
</span>          </span>"</span>email</span>": "</span>hollie.ullrich@towne.net</span>",
</span>          </span>"</span>firstName</span>": "</span>OSCAR</span>",
</span>          </span>"</span>lastName</span>": "</span>MAX</span>"
</span>        </span>},
</span>        </span>{
</span>          </span>"</span>email</span>": "</span>kaitlin2094@gutkowski.biz</span>",
</span>          </span>"</span>firstName</span>": "</span>POPPY</span>",
</span>          </span>"</span>lastName</span>": "</span>MAX</span>"
</span>        </span>},
</span>        </span>{
</span>          </span>"</span>email</span>": "</span>marjolaine2098@lowe.org</span>",
</span>          </span>"</span>firstName</span>": "</span>LILY</span>",
</span>          </span>"</span>lastName</span>": "</span>DAISY</span>"
</span>        </span>},
</span>        </span>{
</span>          </span>"</span>email</span>": "</span>emelia_mosciski@balistreri.info</span>",
</span>          </span>"</span>firstName</span>": "</span>JASPER</span>",
</span>          </span>"</span>lastName</span>": "</span>MAX</span>"
</span>        </span>},
</span>        </span>{
</span>          </span>"</span>email</span>": "</span>arnoldo.stehr@herzog.info</span>",
</span>          </span>"</span>firstName</span>": "</span>LILY</span>",
</span>          </span>"</span>lastName</span>": "</span>SHADOW</span>"
</span>        </span>}
</span>      </span>]
</span>    </span>}
</span>  </span>},
</span>  </span>"</span>fields</span>": {
</span>    </span>"</span>correlation_id</span>": "</span>ABC</span>",
</span>    </span>"</span>destination</span>": "</span>graphql_response</span>"
</span>  </span>},
</span>  </span>"</span>headers</span>": </span>null
</span>}
</span></code></pre>
>></a>
API Gateway</h2>
With account service ready to accept GraphQL queries via message queue,
we now set up the gateway to connect our new services. We create our
phoenix</code> app via phx.new</code> with the barest of options:</p>
mix phx.new api_gateway --no-ecto --no-html --no-gettext --no-webpack
</span></code></pre>
Similar to account service, we install conduit</code> and absinthe</code> but with
absinthe_plug</a> this time:</p>
# api_gateway_web/router.ex
</span>scope </span>"</span>/</span>" </span>do
</span>  pipe_through </span>:</span>api
</span>
</span>  forward </span>"</span>/graphql</span>", </span>Absinthe</span>.</span>Plug</span>, </span>schema</span>: </span>ApiGateway</span>.</span>Schema
</span>  forward </span>"</span>/graphiql</span>", </span>Absinthe</span>.</span>Plug</span>.</span>GraphiQL</span>, </span>schema</span>: </span>ApiGateway</span>.</span>Schema
</span>end
</span></code></pre>
The setup for the broker is similar as well:</p>
# api_gateway.broker.ex
</span>@</span>channel </span>"</span>api_gateway</span>"
</span>
</span>configure </span>do
</span>  queue</span>("</span>account_graphql_request</span>")
</span>end
</span>
</span>incoming </span>ApiGateway </span>do
</span>  pipe_through</span>([:</span>deserialize</span>])
</span>
</span>  subscribe</span>(:</span>account_graphql_response</span>, </span>AccountGraphQLResponse</span>, </span>topic</span>: "</span>account_graphql_response</span>", </span>channel</span>: </span>@</span>channel</span>)
</span>end
</span>
</span>outgoing </span>do
</span>  pipe_through</span>([:</span>serialize</span>])
</span>
</span>  publish</span>(:</span>account_graphql_request</span>, </span>topic</span>: "</span>account_graphql_request</span>")
</span>end
</span>
</span></code></pre>
We then copy the User</code> type and users</code> query but this we need to
resolve it by calling the GraphQL interface of accounts service:</p>
# api_gateway/schema.ex
</span>field </span>:</span>users</span>,</span> non_null</span>(</span>list_of</span>(</span>non_null</span>(:</span>user</span>))) </span>do
</span>  resolve</span>(</span>fn</span> _</span>,</span> _</span>,</span> _ </span>->
</span>    # NEXT: What to do here?
</span>  </span>end</span>)
</span>end
</span></code></pre>
We wrap this functionality in a client similar to the motivating
example:</p>
# api_gateway/account_client.ex
</span>defmodule </span>ApiGateway</span>.</span>AccountClient </span>do
</span>  </span>def </span>run</span>(</span>query</span>,</span> variables </span>\\</span> %</span>{}</span>, context </span>\\</span> %</span>{}</span>) </span>do
</span>    request_id </span>= </span>Ecto</span>.</span>UUID</span>.</span>uuid4</span>()
</span>
</span>    </span>{:</span>ok</span>,</span> _</span>} </span>= </span>Registry</span>.</span>register</span>(</span>@</span>registry</span>,</span> request_id</span>, :</span>graphql</span>)
</span>
</span>    result </span>=
</span>      %</span>Conduit</span>.</span>Message</span>{}
</span>      </span>|> </span>Conduit</span>.</span>Message</span>.</span>put_correlation_id</span>(</span>request_id</span>)
</span>      </span>|> </span>Conduit</span>.</span>Message</span>.</span>put_body</span>(</span>%</span>{
</span>          </span>query</span>:</span> query</span>,
</span>          </span>variables</span>:</span> variables</span>,
</span>          </span>context</span>:</span> context
</span>      </span>})
</span>      </span>|> </span>ApiGateway</span>.</span>Broker</span>.</span>publish</span>(:</span>account_graphql_request</span>)
</span>      </span>|> case do
</span>           </span>{:</span>ok</span>,</span> _</span>} </span>->
</span>             </span>receive do
</span>               response </span>-> </span>{:</span>ok</span>,</span> response</span>}
</span>             </span>after
</span>               </span>:</span>timer</span>.</span>seconds</span>(</span>15</span>) </span>-> </span>{:</span>error</span>, :</span>response_timeout</span>}
</span>             </span>end
</span>           error </span>-></span> error
</span>         </span>end
</span>
</span>      </span>:</span>ok </span>= </span>Registry</span>.</span>unregister</span>(</span>@</span>registry</span>,</span> request_id</span>)
</span>
</span>      result
</span>  </span>end
</span>end
</span></code></pre>
To call the account service users</code> query in the resolver:</p>
# api_gateway/schema.ex
</span>AccountClient</span>.</span>run</span>("""
</span>query {
</span>  users {
</span>    id
</span>    email
</span>    firstName
</span>    lastName
</span>  }
</span>}
</span>""")
</span>|> case do
</span>     </span>{:</span>ok</span>,</span> %</span>{"</span>data</span>" </span>=></span> %</span>{"</span>users</span>" </span>=></span> users</span>}}} </span>->
</span>       </span>{:</span>ok</span>,</span> users</span>}
</span>
</span>     error </span>->
</span>       error
</span>  </span>end
</span>end</span>)
</span></code></pre>
Notice the naming convention in the query or GraphQL in general is
camelCase</code> while for Elixir it is under_scored</code>. The issue is that
absinthe</code> expects the data to be in under_score</code> and converts it into
camelCase</code>. So when we use the query above, first_name</code> since it
receives firstName</code>. As a quick fix, we can convert the data received
from the response via atomic_map</a> or create a middleware to do the same.
Either fix is converting converted data which seems inefficient, instead
vwe can tell absinthe</code> to retain the under_score</code> convention by setting
the adapter to Absinthe.Adapter.Passthrough</a>:</p>
# account_service/config/config.exs
</span>config </span>:</span>absinthe</span>,
</span>  </span>adapter</span>: </span>Absinthe</span>.</span>Adapter</span>.</span>Passthrough
</span>
</span># api_gateway/schema.ex
</span>"""
</span>query {
</span>  users {
</span>    id
</span>    email
</span>    first_name
</span>    last_name
</span>  }
</span>}
</span>"""
</span></code></pre>
This is also useful when other services need to communicate with each
other later on. Within the internal network or system, the convention is
under_score</code>; when it leaves the gateway, it becomes camelCase</code>. One
more thing remains before it completely works, the response from a
GraphQL endpoint is a string map and by default absinthe</code> only picks up
atom keys. So even with the correct keys, we need to tell absinthe</code> to
pick up both atom and string keys. Taking cues from the default
middleware guide</a>, we can change the default middleware to check for
both:</p>
# api_gateway/schema.ex
</span>def </span>middleware</span>(</span>middleware</span>,</span> %</span>{</span>identifier</span>:</span> identifier</span>} </span>=</span> field</span>,</span> object</span>) </span>do
</span>  field_name </span>= </span>Atom</span>.</span>to_string</span>(</span>identifier</span>)
</span>
</span>  new_middleware_spec </span>= </span>{{</span>__MODULE__</span>, :</span>get_field_key</span>}, {</span>field_name</span>,</span> identifier</span>}}
</span>
</span>  </span>Absinthe</span>.</span>Schema</span>.</span>replace_default</span>(</span>middleware</span>,</span> new_middleware_spec</span>,</span> field</span>,</span> object</span>)
</span>end
</span>
</span>def </span>get_field_key</span>(</span>%</span>Resolution</span>{</span>source</span>:</span> source</span>} </span>=</span> res</span>, {</span>key</span>,</span> fallback_key</span>}) </span>do
</span>  new_value </span>=
</span>    </span>case </span>Map</span>.</span>fetch</span>(</span>source</span>,</span> key</span>) </span>do
</span>      </span>{:</span>ok</span>,</span> value</span>} </span>->
</span>        value
</span>
</span>      </span>:</span>error </span>->
</span>        </span>Map</span>.</span>get</span>(</span>source</span>,</span> fallback_key</span>)
</span>    </span>end
</span>
</span>  %</span>{</span>res </span>| </span>state</span>: :</span>resolved</span>, </span>value</span>:</span> new_value</span>}
</span>end
</span></code></pre>
All pieces are now in place and we can use the GraphiQL
endpoint(http://localhost:14000/graphiql</a>) to check our work. Running the
original query above leads to the same result as the account service.</p>
>></a>
registerUser Mutation</h2>
With a working query, we can test mutations with a new registerUser</code>
mutation which should look like this:</p>
mutation</span>($</span>email</span>: </span>String!</span>, $</span>firstName</span>: </span>String!</span>, $</span>lastName</span>: </span>String!</span>) {
</span>  </span>register_user</span>(
</span>    </span>input</span>: {
</span>      email</span>: $</span>email
</span>      firstName</span>: $</span>firstName
</span>      lastName</span>: $</span>lastName
</span>    </span>}
</span>  </span>) {
</span>    </span>errors </span>{
</span>      </span>field
</span>      </span>message
</span>    </span>}
</span>    </span>user </span>{
</span>      </span>id
</span>      </span>email
</span>      </span>firstName
</span>      </span>lastName
</span>    </span>}
</span>  </span>}
</span>}
</span></code></pre>
Aside from using absinthe_relay</a>, creating the whole vertical slice for
the mutation from the account service to the gateway is straightforward.
Calling a mutation and passing the arguments from one endpoint to
another is surprisingly clear:</p>
# api_gateway/schema.ex
</span>resolve</span>(</span>fn</span> _</span>,</span> args</span>,</span> res </span>->
</span>  </span>AccountClient</span>.</span>run</span>(
</span>    </span>"""
</span>    mutation($email: String!, $first_name: String!, $last_name: String!) {
</span>      register_user(
</span>        input: {
</span>          email: $email
</span>          first_name: $first_name
</span>          last_name: $last_name
</span>        }
</span>      ) {
</span>        errors {
</span>          field
</span>          message
</span>        }
</span>        user {
</span>          id
</span>          email
</span>          first_name
</span>          last_name
</span>        }
</span>      }
</span>    }
</span>    </span>""",
</span>    args       </span># HERE: Passing arguments
</span>  </span>)
</span>  </span>|> case do
</span>       </span>{:</span>ok</span>,</span> %</span>{"</span>data</span>" </span>=></span> %</span>{"</span>register_user</span>" </span>=></span> register_user</span>}}} </span>->
</span>         </span>{:</span>ok</span>,</span> register_user</span>}
</span>       error </span>->
</span>         error
</span>     </span>end
</span>end</span>)
</span></code></pre>
Going back the reason we used a database instead of an Agent</code> is to
test concurrency errors specially when two instances of the same service
receives the same request. Thanks to ecto</code>, I am confident that this
cannot happen because of the unique database index; however, what if the
operation has no index like with the upcoming createTransaction</code>
mutation? Our message queue also has our back with this because when two
or more clients subscribe to the same topic but all the same channel,
only one gets it (see NSQ Design</a>); however, messages can be still
delivered many times (see NSQ Guarantees</a>) so we are back where we
started. Message deduplication is beyond the scope of this
investigation. (ConduitPlugs.Deduplication</a> is my quick stab at that.)</p>
I have another concern but that is down the road and we can move on for
now. If you do want to simulate the issue above, you can do the
following:</p>
# account_service/broker.ex
</span># If you want to have different channels for every run,
</span># replace channel with a random value
</span>subscribe</span>(
</span>  </span>:</span>graphql_request</span>,
</span>  </span>GraphQLRequest</span>,
</span>  </span>topic</span>: "</span>account_graphql_request</span>",
</span>  </span>channel</span>: "#{</span>@</span>channel</span>}</span>-</span>#{</span>Enum</span>.</span>random</span>(</span>1</span>..</span>100</span>)}"  </span># HERE: From @channel
</span>)
</span></code></pre>
# In one terminal
</span>mix run --no-halt
</span># In another terminal
</span>mix run --no-halt
</span>
</span># With docker
</span>docker-compose up --scale account_service=2
</span></code></pre>
>></a>
Product and Transaction Service</h2>
What we want next is to implement the following query:</p>
mutation</span>($</span>items</span>: </span>[</span>CreateTransactionItem</span>!]!</span>) {
</span>  </span>createTransaction</span>(
</span>    </span>input</span>: {
</span>      items</span>: $</span>items
</span>    </span>}
</span>  </span>) {
</span>    </span>errors </span>{
</span>      </span>field
</span>      </span>message
</span>    </span>}
</span>    </span>transaction </span>{
</span>      </span>id
</span>      </span>code
</span>      </span>items </span>{
</span>        </span>id
</span>        </span>productId
</span>        </span>price
</span>        </span>quantity
</span>      </span>}
</span>    </span>}
</span>  </span>}
</span>}
</span></code></pre>
The idea is that the user will buy a list of products from this API. To
test that out, we need to create the product and transaction service.
Given our template with account service, we can quickly copy and paste
to produce both services. For products, it exposes the two queries:</p>
query </span>{
</span>  </span>products </span>{
</span>    </span>id
</span>    </span>code
</span>    </span>name
</span>    </span>price
</span>  </span>}
</span>  </span>product</span>(</span>$</span>id</span>: </span>ID</span>!</span>) {
</span>    ...
</span>  </span>}
</span>}
</span></code></pre>
The first one is self-evident like with users</code> query, the second one is
where we make it interesting. The createTransaction</code> accepts a list of
product id and corresponding quantity. For each product id, the
transaction service will invoke the product</code> query to fetch the current
price. Not the most efficient which is another issue down the road but
it does work.</p>
resolve</span>(</span>fn</span> _</span>,</span> args</span>,</span> res </span>->
</span>  </span>TransactionClient</span>.</span>run</span>(
</span>    </span>"""
</span>    mutation($items: [CreateTransactionItem!]!) {
</span>      create_transaction(
</span>        input: {
</span>          items: $items
</span>        }
</span>      ) {
</span>        errors {
</span>          field
</span>          message
</span>        }
</span>        transaction {
</span>          id
</span>          code
</span>          items {
</span>            id
</span>            product_id
</span>            price
</span>            quantity
</span>          }
</span>        }
</span>      }
</span>    }
</span>    </span>""",
</span>    args</span>,
</span>    res</span>.</span>context     </span># HERE: Context based here
</span>  </span>)
</span>  </span>|> case do
</span>       </span>{:</span>ok</span>,</span> %</span>{"</span>data</span>" </span>=></span> %</span>{"</span>create_transaction</span>" </span>=></span> create_transaction</span>}}} </span>->
</span>         </span>{:</span>ok</span>,</span> create_transaction</span>}
</span>
</span>       error </span>->
</span>         error
</span>     </span>end
</span>end</span>)
</span></code></pre>
For the astute reader, the user_id</code> is nowhere to be seen so how will
the transaction know which user is assigned? It comes from the OAauth
header Authorization</code> like with guardian</a> but uses the user email
instead:</p>
# api_gateway/context.ex
</span>defmodule </span>ApiGatewayWeb</span>.</span>Context </span>do
</span>  </span>@</span>behaviour </span>Plug
</span>
</span>  </span>alias </span>Plug</span>.</span>Conn
</span>
</span>  </span>def </span>init</span>(</span>opts</span>), </span>do:</span> opts
</span>
</span>  </span>def </span>call</span>(</span>conn</span>,</span> _</span>) </span>do
</span>    context </span>=</span> build_context</span>(</span>conn</span>)
</span>
</span>    </span>Absinthe</span>.</span>Plug</span>.</span>put_options</span>(</span>conn</span>, </span>context</span>:</span> context</span>)
</span>  </span>end
</span>
</span>  </span>def </span>build_context</span>(</span>conn</span>) </span>do
</span>    %</span>{
</span>      </span>current_user</span>:</span> get_current_user</span>(</span>conn</span>)
</span>    </span>}
</span>  </span>end
</span>
</span>  </span>defp </span>get_current_user</span>(</span>conn</span>) </span>do
</span>    conn
</span>    </span>|> </span>Conn</span>.</span>get_req_header</span>("</span>authorization</span>")
</span>    </span>|> </span>List</span>.</span>first</span>()
</span>    </span>|> case do
</span>      </span>"</span>Bearer </span>" </span><></span> email </span>->
</span>        </span>AccountClient</span>.</span>get_by_email</span>(</span>email</span>)
</span>
</span>      _ </span>->
</span>        </span>nil
</span>    </span>end
</span>  </span>end
</span>end
</span></code></pre>
It just follows the context and authentication guide</a> for a quick
authentication plug. More importantly, the question is how does context
passing occur over multiple services? In this case, the context is
passed down the wire and read the same way; however, the better strategy
is to make the user_id</code> an explicit argument than a context variable.
Nonetheless, it works like a charm but you need to handle string maps
again:</p>
# transaction_service/parse_context.ex
</span>defmodule </span>TransactionService</span>.</span>ParseContext </span>do
</span>  </span>@</span>behaviour </span>Absinthe</span>.</span>Middleware
</span>
</span>  </span>def </span>call</span>(</span>%</span>{</span>state</span>: :</span>unresolved</span>} </span>=</span> res</span>,</span> _opts</span>) </span>do
</span>    ctx </span>=</span> res</span>.</span>context
</span>
</span>    id </span>=</span> get_in</span>(</span>ctx</span>, ["</span>current_user</span>", "</span>id</span>"])
</span>
</span>    %</span>{</span>res </span>| </span>context</span>:</span> %</span>{</span>current_user</span>:</span> %</span>{</span>id</span>:</span> id</span>}}}
</span>  </span>end
</span>
</span>  </span>def </span>call</span>(</span>res</span>,</span> _</span>), </span>do:</span> res
</span>end
</span>
</span></code></pre>
To test this mutation in GraphiQL, you need to do the following things:</p>

Register a new user or pick a random user from users</code> query and take
note of the email($EMAIL</code>)</li>
Add the header Authorization: Bearer $EMAIL</code> which you can do via
Headers</code> > Add</code> button</li>
Pick a random product and take note of its id ($PRODUCT_ID</code>)</li>
Copy the source mutation above and use the payload via variables:</li>
</ol>

{
</span>    </span>"</span>items</span>": [
</span>        </span>{
</span>            </span>"</span>productId</span>": "</span>$PRODUCT_ID</span>",
</span>            </span>"</span>quantity</span>": </span>1
</span>        </span>}
</span>    </span>]
</span>}
</span></code></pre>
With that we have one thing left to verify the experiment.</p>
>></a>
transactions Field</h2>
Once the user has made transactions, we will add a User.transactions</code>
field to fetch that:</p>
query </span>{
</span>  </span>users </span>{
</span>    </span>id
</span>    </span>email
</span>    </span>transactions </span>{
</span>      </span>id
</span>      </span>code
</span>      </span>items </span>{
</span>        </span>product </span>{
</span>          </span>id
</span>          </span>code
</span>          </span>name
</span>          </span>price
</span>        </span>}
</span>        </span>price
</span>        </span>quantity
</span>      </span>}
</span>    </span>}
</span>  </span>}
</span>}
</span></code></pre>
The interesting fact here is that the gateway now provides a custom
field and fetches multiple services to provide that. This is not Apollo
Federation</a> but it is rather fun. Anyway to resolve User.transactions</code>,
we need the transaction service to return all transactions for a user
via user_transaction</code> query:</p>
field </span>:</span>transactions</span>,</span> non_null</span>(</span>list_of</span>(</span>non_null</span>(:</span>user_transaction</span>))) </span>do
</span>  description</span>("</span>User transactions</span>")
</span>
</span>  resolve</span>(</span>fn</span> %</span>{"</span>id</span>" </span>=></span> id</span>},</span> _</span>,</span> res </span>->
</span>    </span>TransactionClient</span>.</span>run</span>(
</span>      </span>"""
</span>      query($user_id: ID!) {
</span>        user_transactions(user_id: $user_id) {
</span>          id
</span>          code
</span>          items {
</span>            id
</span>            product_id
</span>            price
</span>            quantity
</span>          }
</span>        }
</span>      }
</span>      </span>""",
</span>      %</span>{</span>user_id</span>:</span> id</span>},
</span>      res</span>.</span>context
</span>    </span>)
</span>    </span>|> case do
</span>         </span>{:</span>ok</span>,</span> %</span>{"</span>data</span>" </span>=></span> %</span>{"</span>user_transactions</span>" </span>=></span> user_transactions</span>}}} </span>->
</span>           </span>{:</span>ok</span>,</span> user_transactions</span>}
</span>
</span>         error </span>->
</span>           error
</span>       </span>end
</span>  </span>end</span>)
</span>end
</span></code></pre>
Lastly, we reuse the product</code> query for product service to provide the
TransactionItem.product</code> field:</p>
field </span>:</span>product</span>,</span> non_null</span>(:</span>product</span>) </span>do
</span>  description</span>("</span>Transaction product</span>")
</span>
</span>  resolve</span>(</span>fn</span> %</span>{"</span>product_id</span>" </span>=></span> product_id</span>},</span> _</span>,</span> res </span>->
</span>    </span>ProductClient</span>.</span>run</span>(
</span>      </span>"""
</span>      query($id: ID!) {
</span>        product(id: $id) {
</span>          id
</span>          code
</span>          name
</span>          price
</span>        }
</span>      }
</span>      </span>""",
</span>      %</span>{</span>id</span>:</span> product_id</span>},
</span>      res</span>.</span>context
</span>    </span>)
</span>    </span>|> case do
</span>         </span>{:</span>ok</span>,</span> %</span>{"</span>data</span>" </span>=></span> %</span>{"</span>product</span>" </span>=></span> product</span>}}} </span>->
</span>           </span>{:</span>ok</span>,</span> product</span>}
</span>
</span>         error </span>->
</span>           error
</span>       </span>end
</span>  </span>end</span>)
</span>end
</span></code></pre>
With all that, it now works although slow. Indeed, this is where Batch</a>
or Dataloader</a> middleware come in and where each service must provide the
supporting queries. In general, each communication increases latency and
risk but natural under the microservice style.</p>
Another issue I noticed with this approach is that multiple services are
listening to the same GraphQL response topic. Remember that the gateway
and transaction service are listening to the product response so if
either service makes a GraphQL request, both receive the response but
only one will find it useful. As this scales, so will the waste thus the
service dependencies must be watched.</p>
Now that all services are ready, we can the final touch.</p>
>></a>
GraphiQL</h2>
One of the nice things about GraphQL is GraphiQL which alongside the
typed contract forces some form of documentation and experimentation.
We should also put absinthe_plug</code> via plug</code> for each service to make
it easier to access and introspect:</p>
# account_service/router.ex
</span>defmodule </span>AccountService</span>.</span>Router </span>do
</span>  </span>use </span>Plug</span>.</span>Router
</span>
</span>  plug </span>Plug</span>.</span>RequestId
</span>  plug </span>Plug</span>.</span>Logger
</span>
</span>  plug </span>Plug</span>.</span>Parsers</span>,
</span>    </span>parsers</span>: [:</span>urlencoded</span>, :</span>multipart</span>, :</span>json</span>, </span>Absinthe</span>.</span>Plug</span>.</span>Parser</span>],
</span>    </span>pass</span>: ["</span>*/*</span>"],
</span>    </span>json_decoder</span>: </span>Jason
</span>
</span>  plug </span>:</span>match
</span>
</span>  forward </span>"</span>/graphql</span>",
</span>    </span>to</span>: </span>Absinthe</span>.</span>Plug</span>,
</span>    </span>init_opts</span>: [</span>schema</span>: </span>AccountService</span>.</span>Schema</span>]
</span>
</span>  forward </span>"</span>/graphiql</span>",
</span>    </span>to</span>: </span>Absinthe</span>.</span>Plug</span>.</span>GraphiQL</span>,
</span>    </span>init_opts</span>: [</span>schema</span>: </span>AccountService</span>.</span>Schema</span>, </span>interface</span>: :</span>advanced</span>]
</span>
</span>  plug </span>:</span>dispatch
</span>end
</span>
</span># account_service/application.ex
</span>children </span>= </span>[
</span>  </span>{</span>Plug</span>.</span>Cowboy</span>, </span>scheme</span>: :</span>http</span>, </span>plug</span>: </span>AccountService</span>.</span>Router</span>, </span>options</span>: [</span>port</span>: </span>15000</span>]},
</span>  </span>Broker</span>,
</span>  </span>Repo
</span>]
</span></code></pre>
For each service, you might also nned the middleware</code> fix because the
GraphiQL UI cannot handle under_score</code> convention or that it reverts
back to being camelCase</code>. Whichever the case, having a working GraphQL
UI is wonderful. The ports for each service is listed below:</p>

Account Service: http://localhost:15000/graphiql</a></li>
Product Service: http://localhost:16000/graphiql</a></li>
Transaction Service: http://localhost:17000/graphiql</a></li>
</ul>
Now that we are done, what have we learned in this encounter?</p>
>></a>
Review</h2>
Providing a GraphQL interface over a message queue was fun. It was not
as hard as I thought it would be initially, but now is the time to
address some issues and lessons I learned while writing it.</p>
First off, the workflow with using GraphQL is more focused on providing
a good API schema which feels better than worrying about message queue
topics or mechanics; however, writing the query strings are bulky and
error prone in its own right. I can imagine writing multiple queries and
either making a typo or forgetting to update old queries. Testing should
catch this, but I am not aware of tooling like GraphQL fragments</a>. I
suppose this is expected as this is the same workflow with frontend
engineers consuming GraphQL interfaces.</p>
One of the lost GraphQL qualities during this experiment was that fields
could be fetched on demand or query optimization. For example, when a
user wants to only fetch their ids, the internal GraphQL query to
account service fetches all fields. More strongly for the
User.transactions</code> query, it fetches all the way down. I tried doing
some optimization myself using the definitions in the
Absinthe.Resolution</code>:</p>
# product_service/schema.ex
</span>resolve</span>(</span>fn</span> %</span>{"</span>product_id</span>" </span>=></span> product_id</span>},</span> _</span>,</span> res </span>->
</span>  </span>ProductClient</span>.</span>run</span>(
</span>    </span>"""
</span>    query($id: ID!) {
</span>      product(id: $id) {
</span>      </span>#{</span>to_field_query</span>(</span>res</span>.</span>definitions</span>.</span>selections</span>)}
</span>      }
</span>    }
</span>    </span>""",
</span>    %</span>{</span>id</span>:</span> product_id</span>},
</span>    res</span>.</span>context
</span>  </span>)
</span>end</span>)
</span></code></pre>
However, this only works for flat fields; when nested fields are needed,
the complexity spikes. Instead of handling it and dealing with edge
cases such as missing ids or fields, I think its fine to pull all the
fields since they are in an internal network. Perhaps one can find gold
with the selections</code> variable for query optimization?</p>
One concern close to my heart is testing which I did not go into detail
here since you need conduit</code> experience is better in my opinion. The
usual tests for each subscriber module is to check for side effects and
whether messages where acked</code> or not, now we the focus is satisfying
that GraphQL contract.</p>
Another concern is error handling. For example a query is outdated and is
always producing a query error instead of an operation error, how does
the backend cope? It is good to assume external communication can fail
even if they are known assumptions. Still, the only option is to log it
and hopefully catch it in development..</p>
Finally the shadow of message deduplication and unwanted response
recipients mean that this method may not be the best fit perhaps even
for long term.</p>
>></a>
Summary</h2>
To summarize all the findings:</p>
Pros</strong></strong></p>

Schema Management</strong></strong></li>
</ul>
Managing schema and types is far more interesting than handling routes</p>

GraphiQL Interface</strong></strong>-</li>
</ul>
The GraphQL interface is a nice introspection tool.</p>

Unified Request/Response</strong></strong></li>
</ul>
Only one topic to manage all the request/response messages.</p>

GraphQL Testing</strong></strong></li>
</ul>
Testing GraphQL responses feel better than asserting message
acknowledgment or side-effects from handlers.</p>
Cons</strong></strong></p>

Higher Latency</strong></strong></li>
</ul>
More messages means higher latency. Care must be exercised in making the
gateway as fast as it can be.</p>

Messages Deduplication</strong></strong></li>
</ul>
Mutations should consider messages being delivered at least once.</p>

Unnecessary Subscribers</strong></strong></li>
</ul>
Since every GraphQL response endpoint has many subscribers, a response
to one is broadcast to all making it garbage to all but one.</p>

Query Bulk And Support</strong></strong></li>
</ul>
Queries occupy space and is just as error prone in editing.</p>

Query Optimization</strong></strong></li>
</ul>
GraphQL allows select fields to be pulled but the backend must pull all
the fields for safety.</p>
To be safe, I am not vouching this for production as better techniques
may already exist. The request/response module per message is still safe
and efficient. If ever I would make an API gateway, I might consider
revisiting this or pray that absinthe</code> has better guidance or support
at that time. Nonetheless, this was a fun excursion.</p>


Blog Update: From Jekyll To Zola
Unknown — Sun, 29 Sep 2019 00:00:00 +0000
></a>
X Gonna Give It To YOu</h1>
use</span> x</span>;
</span>
</span>x</span>::</span>new</span>();
</span></code></pre>


Understanding Rust Through AVL Trees
Unknown — Wed, 31 Jul 2019 00:00:00 +0000
>></a>
From, Into</h2>
I loved learning the Elixir</a> language and how its pragmatic supervision
trees</a> and process model</a> taught me the value fault tolerance</a> as a
quality of code than of infrastructure. Having safety and failure
recovery as an idiomatic culture and mindset of the language made me a
better thinker and developer. As a personal preference then in
selecting new languages to learn, I look for potentially new
perspectives and insights that it ascribes to its pilgrims. In
general, a good learning curve is a good indicator since it has much
to teach.</p>
Another language to learn that I believe is worth learning is Rust</a>, a
system programming language with a strong culture of memory safety.
Its pervasive model of ownership and lifetimes</a> alongside its demanding
yet helpful compiler makes compiling code both a challenge and thrill.
Aside from the higher learning curve, the absence of a garbage
collector and virtual machine while proclaiming safety and performance
in the face of concurrency is boldly insane. If I do invest, I expect
to be more acute about memory issues and be more confident with
algorithms.</p>
After reading the main online book</a>, the best way to understand a
language for me is by writing code with a project. Since Rust</em> is a
safe alternative to the C</a> language, I want to revisit the data
structures and algorithms that I avoided because of tricky pointer
management. In starting with linked lists</a>, I stumbled across Too Many
Linked List online book</a> which was godsend. Reading it next, I want to
step it up and implement a tricky data structure that had interesting
mechanics to prove my understanding: AVL Trees</a>.</p>
To understand my intent to go back to the basics, I scoffed at C</em> for
being too unsafe relative to Java</a>'s automatic memory management during
my teenage years. What foolish thoughts. Nonetheless, it pushed me
away from learning lower level algorithms which I thought of a hole in
my knowledge for years. With this regret, I ran cargo new --lib avl_tree_set_rs</code>.</p>
After three weeks of struggling and tangents, I successfully wrote it
with green tests. I found my trial a rigorous yet enlightening
learning experience. Despite all the resistance and confusion from the
compiler, I trust that as long as it compiles, my code and more
importantly I feel safe, a similar feeling in Haskell</a>. In this
article, indulge me in my experience of implementing a self-balancing
binary tree in Rust</em> as a learning project. Hopefully, this is worth
your time, helpful or entertaining. So what does it take to implement
an AVL Tree</em> in Rust</em>?</p>
(Source code for this article can be found here</a>.)</p>
REQUIREMENT:</strong>: This article assumes a rustc</code> version of at least
1.38</code> for the non-lexical lifetimes</a> if you want to compile or follow
the code.</p>
CAVEAT:</strong> This is not a tutorial on Rust. I assume you have read the
book yourself and somewhat familiar with binary trees or sets.
Although my writing may not be technically precise, I want to focus on
discussing Rust</em> from a newcomer's perspective. I do my best to link
relevant sections from the book to help me explain syntax and
concepts, so do visit them if you need to refresh yourself. You can
also try running rustup doc --std</code> that opens a local copy of the
standard documentation. I found it quite helpful and convenient in
understanding functions and methods. If I make any incriminating
errors in terminology or code, do file an issue in the article's
repository</a>, so we can rectify it and my thanks.</p>

Corrections</a></li>
</ul>
UPDATE 2019-12-18:</strong> Credits to jbj</strong></strong> for filing a PR</a> in rewriting
take</code> and insert</code> without using unsafe</code> by using recursion.</p>
>></a>
AVL Trees</h2>
As a short discussion, ordered binary trees</a> are good at searching
values. As more values expand the tree, it becomes less efficient
proportional to its height. Thus, rearranging nodes that minimize
height maintains lookup efficiency.</p>
  </p>
To answer that issue, self-balancing trees</strong> balance themselves after
every node insertion and deletion. Different variants such Red-Black
Trees</a>, Splay Trees</a> and B-Trees</a> are available each with their own
strategy. I choose AVL trees</em> for its conceptually simple left and
right rotations as its balancing strategy while proving a challenge in
Rust</em> because of the pointers that dance in those rotations.</p>
Implementing recursive data structures like trees and graphs in Rust</em>
are not trivial because of its ownership model</a> and potential circular
references</a>. Specially in insertion and deletion, swapping nodes and
values safely and correctly while obeying the rules is tricky itself.
Aside from those and some mangling of lifetime variables, we have a
modern C</em> language with familiar functional programming concepts at
our disposal . Despite the uncertainty, tackling this classic
structure with modern ideas makes it feel approachable.</p>
For this project, we will build our AVL tree</em> as a set</a>. To understand
idiomatic language design, we will follow and implement methods from
the ordered tree set of the standard collection:
std::collections::BTreeSet</a>. As much as I would implement every
relevant method, the focus of this article is inserting and deleting
values from our set that we will discuss in three major sections:
insertion</strong>, balancing</strong> and deletion</strong>.</p>
Since this article is long, feel free to stop and come back anytime.
Below are links to major sections each with their own links to
their subsections:</p>

Insertion</a></li>
Rebalancing</a></li>
Deletion</a></li>
Beyond Parity</a> (Optional</em>)</li>
Reflecting On Rust</a></li>
</ul>
>></a>
Insertion</h2>
To begin our journey, we will implement the insertion (.insert</code>)
method to populate our set and test the values with an iterator
(.iter</code>) method. Since implementing balancing strategy is challenging
upfront, we will implement it after this to have a working feedback
cycle and momentum.</p>

Data Types</a></li>
Insert Method</a></li>
Iterator</a></li>
Testing Insert</a></li>
</ul>
>>></a>
Data Types</h3>
What is the design of our tree structure? With some hint from this
chapter</a>, we will use a recursive tree design:</p>
#[</span>derive</span>(</span>Debug</span>,</span> PartialEq</span>,</span> Clone</span>)]
</span>struct </span>AvlNode</span><</span>T</span>: </span>Ord</span>> {
</span>    </span>value</span>:</span> T,
</span>    </span>left</span>: </span>AvlTree</span><</span>T</span>></span>,
</span>    </span>right</span>: </span>AvlTree</span><</span>T</span>></span>,
</span>}
</span>
</span>type </span>AvlTree</span><</span>T</span>> = Option</span><</span>Box</span><</span>AvlNode</span><</span>T</span>>>>; </span>// What is Box?
</span>
</span>#[</span>derive</span>(</span>Debug</span>,</span> PartialEq</span>,</span> Clone</span>)]
</span>struct </span>AvlTreeSet</span><</span>T</span>: </span>Ord</span>> {
</span>    </span>root</span>: </span>AvlTree</span><</span>T</span>></span>,
</span>}
</span>
</span>impl</span><</span>T</span>: </span>Ord</span>> </span>AvlTreeSet</span><</span>T</span>> {
</span>    </span>fn </span>new</span>() -> </span>Self </span>{
</span>        </span>Self </span>{</span> root</span>: </span>None </span>}
</span>    </span>}
</span>}
</span></code></pre>
To explain some Rust</em> specific details, our set library must take
any value that can be ordered, so we declare a generic type</a> that
implements the ordering (std::cmp::Ord</a>) trait (T: Ord</code>). The main
struct</a> AvlNode</code> expresses a binary tree node that stores a value and
possibly has a left and right subtree where the child value is less
than and greater than respectively this node's stored value. The
newtype/type alias</a> AvlTree</code> represents either an empty value/tree or
a node which completes the recursive definition by using the
std::option::Option</a> type. Finally, AvlTreeSet</code> provides the
interface over the underlying data structure. Optionally, we also
derive</a> some useful traits such as debugging via dbg!</code>
(std::fmt::Debug</a>), allow equality checks ==</code> (std::cmp::PartialEq</a>)
and cloning capabilities for testing with .clone</code>
(std::clone::Clone)</a>.</p>
This design is similar in other languages like Haskell</a> and Java</a>, so
we have not strayed far from the path. The snippet below shows how to
initialize this:</p>
let mut</span> tree </span>= Some</span>(</span>Box</span>::</span>new</span>(</span>AvlNode </span>{
</span>    value</span>: </span>2</span>,
</span>    left</span>: </span>Some</span>(</span>Box</span>::</span>new</span>(</span>AvlNode </span>{
</span>        value</span>: </span>1</span>,
</span>        left</span>: </span>None</span>,
</span>        right</span>: </span>None</span>,
</span>    </span>})),
</span>    right</span>: </span>Some</span>(</span>Box</span>::</span>new</span>(</span>AvlNode </span>{
</span>        value</span>: </span>5</span>,
</span>        left</span>: </span>Some</span>(</span>Box</span>::</span>new</span>(</span>AvlNode </span>{
</span>            value</span>: </span>3</span>,
</span>            left</span>: </span>None</span>,
</span>            right</span>: </span>Some</span>(</span>Box</span>::</span>new</span>(</span>AvlNode </span>{
</span>                value</span>: </span>4</span>,
</span>                left</span>: </span>None</span>,
</span>                right</span>: </span>None</span>,
</span>            </span>})),
</span>        </span>})),
</span>        right</span>: </span>None</span>,
</span>    </span>})),
</span>});
</span></code></pre>
As cumbersome it is to initialize, we encounter our first memory
design consideration for recursive structures: std::boxed::Box</a>.
Generally, structs or data types should have a known or constant size</a>
for memory allocation. Every primitive type has a fixed size</a> such as
1</code> byte for bool</code> and u8</code> and 4</code> bytes for i32</code> or char</code>. While
complex structures can be thought of as a sum of its parts</a>, the
compiler cannot infer the known size for recursive structure at
compile time since the size is proportional to the level of nesting
required at runtime. To resolve this constant and dynamic memory
conundrum, since references and pointers also have the same and fixed
size</a>, we wrap the recursive data with a heap allocated pointer</a> which
stores and points to the data in the heap that satisfies the size
constraint</a> and why we wrap each child node with Box::new</code>.</p>
This is unlike other languages where you could use the types without
such consideration. Initially awkward, I find it fascinating that
Rust</em> requires some forethought to also think about how data is
stored (Stack or Heap</a>) at a safe and high level. That this enforced
compiler discipline gives me the impression that code in this
language is more likely to be planned and deliberate. Definitions
done, we proceed with the main method.</p>
>>></a>
Insert Method</h3>
Parity:</strong> BTreeSet::insert</a></p>
To insert new values our tree set, we follow a standard procedure for
binary trees:</p>

Starting from the root node or with a current node</li>
Move to the left node if the value is less than the current node,
right if greater, and stop if equal</li>
Do this until you an empty node and insert the value</li>
</ol>
Translating this into Rust</em>:</p>
use </span>std</span>::</span>cmp</span>::</span>Ordering</span>;
</span>
</span>impl</span><</span>T</span>: </span>Ord</span>> </span>AvlTreeSet</span><</span>T</span>> {
</span>    </span>fn </span>insert</span>(</span>&mut </span>self</span>, </span>value</span>:</span> T</span>) -> </span>bool </span>{
</span>        </span>let mut</span> current_tree </span>= &mut self</span>.</span>root</span>;
</span>
</span>        </span>// 1. Starting from the root node or with a current node
</span>        </span>while let Some</span>(</span>current_node</span>) </span>=</span> current_tree </span>{
</span>            </span>// 2. Move to the left node if the value is less than the current node,
</span>            </span>//    right if greater, and stop if equal
</span>            </span>match</span> current_node</span>.</span>value</span>.</span>cmp</span>(</span>&</span>value</span>) {
</span>                </span>Ordering</span>::</span>Less </span>=></span> current_tree </span>= &mut</span> current_node</span>.</span>right</span>,
</span>                </span>Ordering</span>::</span>Equal </span>=> </span>{
</span>                    </span>return false</span>;
</span>                </span>}
</span>                </span>Ordering</span>::</span>Greater </span>=></span> current_tree </span>= &mut</span> current_node</span>.</span>left</span>,
</span>            </span>}
</span>        </span>}
</span>
</span>        </span>// 3. Do this until you an empty node and insert the value
</span>        </span>*</span>current_tree </span>= Some</span>(</span>Box</span>::</span>new</span>(</span>AvlNode </span>{
</span>            value</span>,
</span>            left</span>: </span>None</span>,
</span>            right</span>: </span>None</span>,
</span>        </span>}));
</span>
</span>        </span>true
</span>    </span>}
</span>}
</span></code></pre>
With pattern matching</a>, it is easily done and also handles the empty
tree case. The snippet below shows how to use it:</p>
let mut</span> set </span>= </span>AvlTreeSet</span>::</span>new</span>();
</span>
</span>assert!</span>(</span>set</span>.</span>insert</span>(</span>1</span>));   </span>// Insert new value
</span>assert!</span>(</span>!</span>set</span>.</span>insert</span>(</span>1</span>));  </span>// Should not insert existing value
</span>
</span>assert!</span>(</span>set</span>.</span>insert</span>(</span>2</span>));   </span>// Insert another new value
</span>assert_eq!</span>(               </span>// Checking the tree structure
</span>    set</span>.</span>root</span>,
</span>    </span>Some</span>(</span>Box</span>::</span>new</span>(</span>AvlNode </span>{
</span>        value</span>: </span>1</span>,
</span>        left</span>: </span>None</span>,
</span>        right</span>: </span>Some</span>(</span>Box</span>::</span>new</span>(</span>AvlNode </span>{
</span>            value</span>: </span>2</span>,
</span>            left</span>: </span>None</span>,
</span>            right</span>: </span>None
</span>        </span>})),
</span>    </span>}))
</span>);
</span></code></pre>
How do we test .insert</code>? We could implement BTreeSet::contains</a> to
test if the inserted value is present; however, where it is inserted
also matters. If the node is inserted in the left instead of the
right, the order of the set would be incorrect. So we need to test it
structurally, and we can do that with an tree iterator</a> which we will
implement next.</p>
>>></a>
Iterator</h3>
Parity:</strong> BTreeSet::iter</a></p>
For our ordered set, we want the iterator to process the data in
ascending order, so we will use an in-order traversal</a> or
left-node-right approach. Since this is lower level language, we are
using a loop instead of a recursive function to express the
algorithm:</p>

Starting from the root node or with a current node</li>
If the node has a left subtree, store this node in a stack of
unemitted nodes, then move to the left node and repeat this step</li>
Otherwise, emit the value</li>
Next if the node has a right subtree, then move to the right node
and repeat from step 2</li>
If neither and there is an unemitted node in the stack, pop that
and emit that node's value and repeat step 3 with that right node</li>
If no more unemitted nodes remain, stop</li>
</ol>
The common pattern is to implement a .iter</code> method that will return
a struct that implements std::iter::Iterator</a> and emits references to
the underlying tree. Before working on the code, we need to define
the struct to hold the iterator state that holds the previously
traversed nodes and the current tree:</p>
#[</span>derive</span>(</span>Debug</span>)]
</span>struct </span>AvlTreeSetIter</span><</span>'a</span>, T</span>: </span>Ord</span>> {
</span>    </span>prev_nodes</span>: </span>Vec</span><</span>&'a </span>AvlNode</span><</span>T</span>>></span>,
</span>    </span>current_tree</span>: </span>&'a </span>AvlTree</span><</span>T</span>></span>,
</span>}
</span>
</span>impl</span><</span>'a</span>, T</span>: </span>'a + Ord</span>></span> Iterator </span>for </span>AvlTreeSetIter</span><</span>'a</span>, T</span>> {
</span>    </span>type </span>Item </span>= &'a</span> T</span>;
</span>
</span>    </span>fn </span>next</span>(</span>&mut </span>self</span>) -> </span>Option</span><</span>Self</span>::</span>Item</span>> {
</span>        </span>unimplemented!</span>()
</span>    </span>}
</span>}
</span>
</span>// Addition of lifetime parameter for the set
</span>impl</span><</span>'a</span>, T</span>: </span>'a + Ord</span>> </span>AvlTreeSet</span><</span>T</span>> {
</span>    </span>fn </span>iter</span>(</span>&'a </span>self</span>) -> </span>AvlTreeSetIter</span><</span>'a</span>, T</span>> {
</span>        AvlTreeSetIter </span>{
</span>            prev_nodes</span>: </span>Vec</span>::</span>new</span>(),
</span>            current_tree</span>: </span>&self</span>.</span>root</span>,
</span>        </span>}
</span>    </span>}
</span>}
</span></code></pre>
Aside from treating std::vec::Vec</a> as a stack, we need to discuss
lifetime for parameters</a>. This is one of Rust</em> concepts that is
conceptually easy but understanding the syntax requires practice.
Fundamentally, every reference has a lifetime scope before being
dropped that must not outlive the owner to avoid dangling
references. Our iterator will not mutate or acquire ownership of the
tree, instead it needs a reference to traverse it and that the
iterator lives as long as the set itself. We express this by
introducing a lifetime parameter('a</code>) on each struct that is
connected through the .iter</code>. We also specify that emitted reference
values also live as long as the iterator itself through the
associated type</a> (Item</code>). Even with all this extra typing for
lifetimes, the compiler is quite helpful in ensuring that the
iterator safely references the set.</p>
Another interesting discipline of Rust</em> in thinking about the
lifetime requirements of references as part of the design. My
personal learning experience is to add explicit annotation for each
reference, figure out their scope and then remove it if unnecessary
as with lifetime elision</a>. As much as possible, I let the Rust
compiler infer as much and stepping in when needed.</p>
Going back, when implementing Iterator::next</a> method, we return
Some(value)</code> when we want to emit value</code> and return None</code> if we
have no more. With this in mind, we can proceed:</p>
fn </span>next</span>(</span>&mut </span>self</span>) -> </span>Option</span><</span>Self</span>::</span>Item</span>> {
</span>    </span>loop </span>{
</span>        </span>match *self</span>.</span>current_tree </span>{
</span>            </span>None => match self</span>.</span>prev_nodes</span>.</span>pop</span>() {
</span>                </span>None => </span>{
</span>                    </span>return None</span>;
</span>                </span>}
</span>
</span>                </span>Some</span>(</span>ref</span> prev_node</span>) </span>=> </span>{
</span>                    </span>self</span>.</span>current_tree </span>= &</span>prev_node</span>.</span>right</span>;
</span>
</span>                    </span>return Some</span>(</span>&</span>prev_node</span>.</span>value</span>);
</span>                </span>}
</span>            </span>},
</span>
</span>            </span>Some</span>(</span>ref</span> current_node</span>) </span>=> </span>{
</span>                </span>if</span> current_node</span>.</span>left</span>.</span>is_some</span>() {
</span>                    </span>self</span>.</span>prev_nodes</span>.</span>push</span>(</span>&</span>current_node</span>);
</span>                    </span>self</span>.</span>current_tree </span>= &</span>current_node</span>.</span>left</span>;
</span>
</span>                    </span>continue</span>;
</span>                </span>}
</span>
</span>                </span>if</span> current_node</span>.</span>right</span>.</span>is_some</span>() {
</span>                    </span>self</span>.</span>current_tree </span>= &</span>current_node</span>.</span>right</span>;
</span>
</span>                    </span>return Some</span>(</span>&</span>current_node</span>.</span>value</span>);
</span>                </span>}
</span>
</span>                </span>self</span>.</span>current_tree </span>= &None</span>;
</span>
</span>                </span>return Some</span>(</span>&</span>current_node</span>.</span>value</span>);
</span>            </span>}
</span>        </span>}
</span>    </span>}
</span>}
</span></code></pre>
Although a bit longer, it just requires a little work. One thing I
find interesting here is that pattern matching also has move
semantics via ref</a> keyword. Quite intriguing where borrow mechanics
pervade even the familiar pattern matching. To show our code works,
we insert a range backwards (Iterator::rev</a>) and expect it to come up
in ascending order:</p>
let mut</span> set </span>= </span>AvlTreeSet</span>::</span>new</span>();
</span>
</span>for</span> i </span>in </span>(</span>1</span>..</span>4 </span>as usize</span>).</span>rev</span>() {
</span>    set</span>.</span>insert</span>(</span>i</span>);
</span>}
</span>
</span>let mut</span> iter </span>=</span> set</span>.</span>iter</span>();
</span>assert_eq!</span>(</span>iter</span>.</span>next</span>(), </span>Some</span>(</span>&</span>1</span>));
</span>assert_eq!</span>(</span>iter</span>.</span>next</span>(), </span>Some</span>(</span>&</span>2</span>));
</span>assert_eq!</span>(</span>iter</span>.</span>next</span>(), </span>Some</span>(</span>&</span>3</span>));
</span>assert_eq!</span>(</span>iter</span>.</span>next</span>(), </span>None</span>);
</span></code></pre>
Can we test .iter</code> without .insert</code>? If we create a different
constructor for our set that takes a collection or iterator and build
nodes around it, we certainly could; however, it requires that we
arrange them in a balanced manner which is beyond our current
capacity. Testing both .iter</code> and .insert</code> is the simplest way to
go.</p>
>>></a>
Testing Insert</h3>
To test .iter</code> and .insert</code> together, we do the following:</p>

Create a random list</li>
Create an empty AVL tree and BTree</li>
For each element in the list, insert it to both tree</li>
Both AVL and BTree iterator should be equal item per item</li>
</ol>
For generating random data, we will use quickcheck</a> as a generator and
our testing style: property based testing</a>. Instead of testing just
correct input and output like with TDD</a> and BDD</a>, we focus on proving
properties or statements are correct for every possible value. For
example with odd and even integers</a>, adding two even or odd integers
will always be even while adding an even and odd integer together
will always be odd. For testing abstract data types with little or no
state, testing structural properties is easier than asserting the
structure itself.</p>
Our first statement/property is that our AVL tree and BTree have
equal iterators</strong> since it exercises the whole structure and order. To
assist in testing, we employ rust-itertools</a> to test iterator equality
via itertools::equal</a> since it is better than using Iterator::zip</a> and
assert_eq!</a> in a for loop. We also implement the convenience trait,
std::iter::IntoIterator</a>, to use IntoIterator::collect</a> method to
easily transforms iterators into other collections as shown below:</p>
impl</span><</span>T</span>: </span>Ord</span>> </span>FromIterator</span><</span>T</span>> </span>for </span>AvlTreeSet</span><</span>T</span>> {
</span>    </span>fn </span>from_iter</span><</span>I</span>: </span>IntoIterator</span><</span>Item = T</span>>>(</span>iter</span>:</span> I</span>) -> </span>Self </span>{
</span>        </span>let mut</span> set </span>= Self</span>::</span>new</span>();
</span>
</span>        </span>for</span> i </span>in</span> iter </span>{
</span>            set</span>.</span>insert</span>(</span>i</span>);
</span>        </span>}
</span>
</span>        set
</span>    </span>}
</span>}
</span>
</span>// The power of IntoIterator
</span>let</span> vec </span>= </span>(</span>1</span>..</span>10 </span>as u8</span>).</span>collect</span>::<</span>Vec</span><</span>_</span>>>();
</span>let</span> avl  </span>= </span>(</span>1</span>..</span>10 </span>as u8</span>).</span>collect</span>::<</span>AvlTreeSet</span><</span>_</span>>>();
</span>let</span> btree </span>= </span>(</span>1</span>..</span>10 </span>as u8</span>).</span>collect</span>::<</span>BTreeSet</span><</span>_</span>>>();
</span></code></pre>
With all this foundation, creating a test for the iterator is precise:</p>
#[</span>cfg</span>(</span>test</span>)]
</span>mod </span>properties </span>{
</span>    </span>use super</span>::</span>*</span>;
</span>    </span>use </span>std</span>::</span>collections</span>::</span>BTreeSet</span>;
</span>
</span>    </span>#[</span>quickcheck</span>]
</span>    </span>// 1. Create a random list
</span>    </span>fn </span>iterator_parity</span>(</span>mut </span>xs</span>: </span>Vec</span><</span>usize</span>>) -> </span>bool </span>{
</span>        </span>// 2. Create an empty AVL tree and BTree
</span>        </span>// 3. For each element in the list, insert it to both tree
</span>        </span>let</span> avl_set </span>=</span> xs</span>.</span>iter</span>().</span>cloned</span>().</span>collect</span>::<</span>AvlTreeSet</span><</span>_</span>>>();
</span>        </span>let</span> btree_set </span>=</span> xs</span>.</span>iter</span>().</span>cloned</span>().</span>collect</span>::<</span>BTreeSet</span><</span>_</span>>>();
</span>
</span>        </span>// 4. Both AVL and BTree iterator should be equal item per item
</span>        equal</span>(</span>avl_set</span>.</span>iter</span>(),</span> btree_set</span>.</span>iter</span>())
</span>    </span>}
</span>}
</span></code></pre>
Instead of #[test]</code>, we use #[quickcheck]</code> macro to indicate it is
a property based test. The actual mechanics of this style is that it
will generate a random value each argument in the function, run the
test and repeat it around 10,000</code> in an attempt to find a value that
returns false</code> or breaks the test. In this case, the test creates
random usize</code> lists and check the iterators as described above. The
generated lists are also not sorted and not unique that challenges
order and uniqueness. Until a counterexample</a> is found, we can assume
it correct.</p>
We also check insert parity for completion:</p>
#[</span>quickcheck</span>]
</span> </span>fn </span>insert_parity</span>(</span>mut </span>btree_set</span>: </span>BTreeSet</span><</span>u8</span>></span>, </span>x</span>: </span>u8</span>) -> </span>bool </span>{
</span>     </span>let mut</span> avl_set </span>=</span> btree_set</span>.</span>iter</span>().</span>cloned</span>().</span>collect</span>::<</span>AvlTreeSet</span><</span>_</span>>>();
</span>
</span>     avl_set</span>.</span>insert</span>(</span>x</span>) </span>==</span> btree_set</span>.</span>insert</span>(</span>x</span>)
</span> </span>}
</span></code></pre>
Given an equal AVL and BTree set, .insert</code> should be the same for
every value</strong>. We can let quickcheck</em> generate a random BTreeSet</code> as
well and copy it in the same manner. Since x</code> is also random, it is
either in the created sets or not that creates a positive and
negative case for inserting duplicates. With both tests written, we
run the tests with cargo test</code>:</p>
  Compiling avl_tree_set_rs v0.1.0 (/home/nobody/avl_tree_set_rs)
</span>    Finished dev [unoptimized + debuginfo] target(s) in 0.91s
</span>     Running target/debug/deps/avl_tree_set_rs-5f93a12c976aa84b
</span>
</span>running 2 tests
</span>test set::properties::iterator_parity ... ok
</span>test set::properties::insert_parity ... ok
</span></code></pre>
We now have a working tree set. Of course, since we did not rebalance
the tree, we only crafted a standard binary tree but it is a start.</p>
>>></a>
Reflection</h3>
Working with Rust</em> in this first section is not bad. Going back to a
static type system is reinvigorating. I really liked that structs,
pattern matching and type classes</a> were included in the language, so I
had no problem there. Typing and understanding lifetime annotations
is one thing that made Rust</em> different. Understanding the wide array
of traits and types itself is not trivial, so it is challenging how
their lifetime are factored in as well.</p>
As a beginner, taking some time to look at the various traits and
types and their implementation made it click for me. Having the
documentation at the ready with rustup doc --std</code> was great tooling.
Thankfully, we only require the basic concepts so it is more a
learning experience. So far, Rust</em> does play like a low-level modern
language with some of the best modern features I value.</p>
>></a>
Rebalancing</h2>
With a working binary tree, we apply the rebalancing strategy by
deciding how to implement height and node rotation. We then integrate
it back to .insert</code> and update our test.</p>

Balance Factor And Height</a></li>
Unsafe Decision</a></li>
Height Testing</a></li>
Node Rotation</a></li>
Rotation Testing</a></li>
Rebalancing Insert</a></li>
</ul>
>>></a>
Balance Factor And Height</h3>
Given a node, it has a balance factor</strong> that is the height of the
left tree minus the height of the right tree. The idea is that when
the balance factor grows to 2</code> or -2</code>, a rotation of nodes or
transfer of nodes from one side to the other will fix the balance.
Starting with this definition, implementing the balance factor for
nodes is direct with some type truncation safety:</p>
impl</span><</span>'a</span>, T</span>: </span>'a + Ord</span>> </span>AvlNode</span><</span>T</span>> {
</span>    </span>fn </span>left_height</span>(</span>&</span>self</span>) -> </span>usize </span>{
</span>        </span>self</span>.</span>left</span>.</span>as_ref</span>().</span>map_or</span>(</span>0</span>, |</span>left</span>| </span>left</span>.</span>height</span>())
</span>    </span>}
</span>
</span>    </span>fn </span>right_height</span>(</span>&</span>self</span>) -> </span>usize </span>{
</span>        </span>self</span>.</span>right</span>.</span>as_ref</span>().</span>map_or</span>(</span>0</span>, |</span>right</span>| </span>right</span>.</span>height</span>())
</span>    </span>}
</span>
</span>    </span>pub fn </span>balance_factor</span>(</span>&</span>self</span>) -> </span>i8 </span>{
</span>        </span>let</span> left_height </span>= self</span>.</span>left_height</span>();
</span>        </span>let</span> right_height </span>= self</span>.</span>right_height</span>();
</span>
</span>        </span>if</span> left_height </span>>=</span> right_height </span>{
</span>            </span>(</span>left_height </span>-</span> right_height</span>) </span>as i8
</span>        </span>} </span>else </span>{
</span>            </span>-</span>((</span>right_height </span>-</span> left_height</span>) </span>as i8</span>)
</span>        </span>}
</span>    </span>}
</span>}
</span></code></pre>
Since our type uses usize</code>, we need to consider subtraction
overflow. Assuming the tree is balanced, the balance factor can only
be from -2</code> to 2</code>, so we can also use improve the return type from
isize</code> to i8</code> as a tiny optimization. However, if our tree does not
update height properly, we will have overflow from the smaller
datatype as a consequence. For the pragmatic, directly subtracting
the left and right height and using isize</code> is valid but still holds
as a gentle reminder of what unsigned integers</a> are and its design
impact.</p>
Anyway, we need to define a height</strong> field that is defined
recursively as the larger height of the left or right subtree plus
one and empty trees have an empty height of zero. Word-for-word, it
is written as:</p>
use </span>std</span>::</span>cmp</span>::</span>max</span>;
</span>
</span>fn </span>height</span>(</span>&</span>self</span>) -> </span>usize </span>{
</span>    </span>1 </span>+ </span>max</span>(
</span>        </span>self</span>.</span>left</span>.</span>as_ref</span>().</span>map_or</span>(</span>0</span>, |</span>node</span>| </span>node</span>.</span>height</span>()),
</span>        </span>self</span>.</span>right</span>.</span>as_ref</span>().</span>map_or</span>(</span>0</span>, |</span>node</span>| </span>node</span>.</span>height</span>()),
</span>    </span>)
</span>}
</span></code></pre>
Use Option::map_or</a> when you need map and unwrap with a default in one
go. However, updating ancestor nodes is expensive or inefficient when
it traverses the same node multiple times, so we turn this into a field to
cache the computation instead. Updating our definition of node:</p>
struct </span>AvlNode</span><</span>T</span>: </span>Ord</span>> {
</span>    </span>value</span>:</span> T,
</span>    </span>left</span>: </span>AvlTree</span><</span>T</span>></span>,
</span>    </span>right</span>: </span>AvlTree</span><</span>T</span>></span>,
</span>
</span>    </span>// New field
</span>    </span>height</span>: </span>usize,
</span>}
</span>
</span>impl</span><</span>'a</span>, T</span>: </span>'a + Ord</span>> </span>AvlNode</span><</span>T</span>> {
</span>    </span>fn </span>update_height</span>(</span>&mut </span>self</span>) {
</span>        </span>self</span>.</span>height </span>= </span>1 </span>+ </span>max</span>(</span>self</span>.</span>left_height</span>(), </span>self</span>.</span>right_height</span>());
</span>    </span>}
</span>}
</span></code></pre>
After inserting or deleting a node, we need to traverse the ancestor
nodes to update their height as a trade-off. Definitions done,
implementing rotations was my first real struggle in Rust</em>.</p>
>>></a>
Unsafe Decision</h3>
How do we traverse upwards in our tree after every node insertion? A
quick solution is to add a mutable reference to the parent node and
use that to traverse until the root like so:</p>
#[</span>derive</span>(</span>Debug</span>,</span> PartialEq</span>)]
</span>struct </span>AvlNode</span><</span>'a</span>, T</span>: </span>Ord</span>> {
</span>    </span>value</span>:</span> T,
</span>    </span>left</span>: </span>AvlTree</span><</span>'a</span>, T</span>></span>,    </span>// Sad, extra annotations required
</span>    </span>right</span>: </span>AvlTree</span><</span>'a</span>, T</span>></span>,
</span>    </span>height</span>: </span>usize,
</span>
</span>    </span>// New field
</span>    </span>parent_node</span>: </span>Option</span><</span>&'a mut </span>AvlNode</span><</span>'a</span>, T</span>>></span>,
</span>}
</span>
</span>impl</span><</span>'a</span>, T</span>: </span>'a + Ord</span>> </span>AvlNode</span><</span>'a</span>, T</span>> {
</span>    </span>fn </span>update_ancestors_height</span>(</span>&mut </span>self</span>) {
</span>        </span>self</span>.</span>update_height</span>();
</span>
</span>        </span>let mut</span> current_node </span>= self</span>;
</span>        </span>while let Some</span>(</span>parent_node</span>) </span>=</span> current_node</span>.</span>parent_node </span>{
</span>            parent_node</span>.</span>update_height</span>();
</span>            current_node </span>=</span> parent_node</span>;
</span>        </span>}
</span>    </span>}
</span>}
</span></code></pre>
However, this creates a cyclic reference</a> between parent and child
which is dangerous and discouraged in Rust</em>. Personally, it also
makes the design dirtier that requiring a reference only for
traversing upwards on insert and delete operations seems overkill.
Going back to .insert</code>, we can track them with a stack instead via
Vec::push</a> mechanics:</p>
fn </span>insert</span>(</span>&mut </span>self</span>, </span>value</span>:</span> T</span>) -> </span>bool </span>{
</span>    </span>let mut</span> prev_nodes </span>= Vec</span>::<</span>&mut </span>AvlNode</span><</span>T</span>>>::</span>new</span>();
</span>
</span>    </span>while let Some</span>(</span>current_node</span>) </span>=</span> current_tree </span>{
</span>        prev_nodes</span>.</span>push</span>(</span>current_node</span>);
</span>
</span>        </span>// match ...
</span>    </span>}
</span>
</span>    </span>// snip
</span>
</span>    </span>// Traversing a list as a stack needs to be reversed via .rev first
</span>    </span>for</span> node </span>in</span> prev_nodes</span>.</span>into_iter</span>().</span>rev</span>() {
</span>        node</span>.</span>update_height</span>();
</span>    </span>}
</span>
</span>    </span>true
</span>}
</span></code></pre>
The problem here is that current_node</code> is mutable borrowed twice:
one for a mutable left or right child node and one for traversed
stack/vector. I would argue that the mutability of the tracking nodes
is safe since it is used after the insert mutation is done. Crawling
the web for idiomatic solutions, the suggested strategy is to use the
combination of a reference counted pointer</a> and interior mutable cell</a>
or Rc<RefCell<T>></code> pattern to allow multiple mutable references. I
initially thought I could use this just in the function itself
without changing the definition of the node but I could not:</p>
use </span>std</span>::</span>rc</span>::</span>Rc</span>;
</span>use </span>std</span>::</span>cell</span>::</span>RefCell</span>;
</span>
</span>type </span>AvlTree</span><</span>T</span>> = Option</span><</span>Rc</span><</span>RefCell</span><</span>AvlNode</span><</span>T</span>>>>>;
</span></code></pre>
To use them as intended, we would need to own the data we are using;
however that is not possible since we are behind a mutable reference,
&mut self</code>. Even if I could make the code work, the fact that I had
to change the types and include reference counting complicates the
design needlessly just to retrace the nodes. Up to this point, I find
Rust</em>'s strictness comforting in spite of my struggle; however, I
feel that I need to loosen up the restriction a bit to avoid
unnecessary complexity. In this case, we want to allow one secondary
mutable reference after insertion and doing this requires Unsafe
Rust</a>. Unsafe Rust</strong> allows us to ignore borrowing rules for
mutability as specified in this section of the book</a>. Using raw
pointers allows our implementation to be uncomplicated:</p>
fn </span>insert</span>(</span>&mut </span>self</span>, </span>value</span>:</span> T</span>) -> </span>bool </span>{
</span>    </span>let mut</span> prev_ptrs </span>= Vec</span>::<</span>*mut </span>AvlNode</span><</span>T</span>>>::</span>new</span>();
</span>
</span>    </span>while let Some</span>(</span>current_node</span>) </span>=</span> current_tree </span>{
</span>        prev_ptrs</span>.</span>push</span>(</span>&mut **</span>current_node</span>);   </span>// Converting a mutable reference to a pointer
</span>
</span>        </span>// ... match ...
</span>    </span>}
</span>
</span>    </span>// ... insert ...
</span>
</span>    </span>for</span> node_ptr </span>in</span> prev_ptrs</span>.</span>into_iter</span>().</span>rev</span>() {
</span>        </span>let</span> node </span>= unsafe </span>{ </span>&mut *</span>node_ptr </span>};  </span>// Converting a mutable pointer back to a reference
</span>        node</span>.</span>update_height</span>();
</span>    </span>}
</span>
</span>    </span>true
</span>}
</span></code></pre>
The arcane magic of unsafe is better explained in the Rustonomicon</a>
which I plan to read after writing this. Since we told Rust</em> to
trust us, we need to guarantee our function is safe. The most visible
angle is the one using unsafe</code> or the raw pointers themselves. The
mutable method, .update_height</code>, only requires that its children
have the correct height before updating itself. As an anecdote, I
forgot to reverse the list before iterating (since push operation
append at the end) and caused incorrect node heights because the
parents were updated before the children. With this possibility, we
also need to test the height property.</p>
>>></a>
Height Testing</h3>
Given any AVL tree set, each node's height is one more than the
larger height of the left or right subtree</strong>. To make testing easier,
we can do two short things. First, instead of pattern matching the
structure, we can refactor .iter</code> to emit the nodes instead of their
values .node_iter</code>:</p>
impl</span><</span>'a</span>, T</span>: </span>'a + Ord</span>> </span>AvlTreeSet</span><</span>T</span>> {
</span>    </span>// Using impl Traits
</span>    </span>fn </span>iter</span>(</span>&'a </span>self</span>) -></span> impl </span>Iterator</span><</span>Item = </span>&'a</span> T</span>> </span>+ 'a </span>{
</span>        </span>self</span>.</span>node_iter</span>().</span>map</span>(|</span>node</span>| </span>&</span>node</span>.</span>value</span>)
</span>    </span>}
</span>
</span>    </span>fn </span>node_iter</span>(</span>&'a </span>self</span>) -></span> impl </span>Iterator</span><</span>Item = </span>&'a </span>AvlNode</span><</span>T</span>>> </span>+ 'a </span>{
</span>        AvlTreeSetNodeIter </span>{
</span>            prev_nodes</span>: </span>Vec</span>::</span>default</span>(),
</span>            current_tree</span>: </span>&self</span>.</span>root</span>,
</span>        </span>}
</span>    </span>}
</span>}
</span></code></pre>
As an aside, I suggest learning impl Traits</a> that allows you to
return an abstract trait type instead of the concrete type as in
polymorphism</a>. Using that, we can implement generic iterator adapters</a>
for .iter</code> that maps over .node_iter</code>. Without it, creating
abstractions on top of another becomes a noisy type fest:</p>
use </span>core</span>::</span>iter</span>::</span>Map</span>;
</span>
</span>fn </span>iter</span>(</span>&'a </span>self</span>) -> </span>Map</span><</span>AvlTreeSetNodeIter</span><</span>'a</span>, T</span>></span>, </span>fn</span>(</span>&'a </span>AvlNode</span><</span>T</span>>) -> </span>&</span>T</span>> {
</span>    </span>self</span>.</span>node_iter</span>().</span>map</span>(|</span>node</span>| </span>&</span>node</span>.</span>value</span>)
</span>}
</span>
</span>fn </span>node_iter</span>(</span>&'a </span>self</span>) -> </span>AvlTreeSetNodeIter</span><</span>'a</span>, T</span>> {
</span>    AvlTreeSetNodeIter </span>{
</span>        prev_nodes</span>: </span>Vec</span>::</span>default</span>(),
</span>        current_tree</span>: </span>&self</span>.</span>root</span>,
</span>    </span>}
</span>}
</span></code></pre>
Going back, the new AvlTreeSetNodeIter</code> is a copied version of the
original iterator with Some(&current_node.value)</code> replaced to
Some(&current_node)</code>. Moving on the second item, instead of creating
the sets through .collect</code> method; we can also have it generated by
implementing the Arbitrary</code> trait ourselves:</p>
use </span>quickcheck</span>::{</span>Arbitrary</span>,</span> Gen</span>};
</span>
</span>// Refit and copied from quickcheck
</span>// https://docs.rs/quickcheck/0.8.5/src/quickcheck/arbitrary.rs.html#385-395
</span>impl</span><</span>T</span>:</span> Arbitrary </span>+ Ord</span>></span> Arbitrary </span>for </span>AvlTreeSet</span><</span>T</span>> {
</span>    </span>fn </span>arbitrary</span><</span>G</span>:</span> Gen</span>>(</span>g</span>: </span>&mut</span> G</span>) -> </span>Self </span>{
</span>        </span>let</span> vec</span>: </span>Vec</span><</span>T</span>> </span>= </span>Arbitrary</span>::</span>arbitrary</span>(</span>g</span>);
</span>        vec</span>.</span>into_iter</span>().</span>collect</span>()
</span>    </span>}
</span>
</span>    </span>fn </span>shrink</span>(</span>&</span>self</span>) -> </span>Box</span><</span>dyn </span>Iterator</span><</span>Item = </span>Self</span>>> {
</span>        </span>let</span> vec</span>: </span>Vec</span><</span>T</span>> </span>= self</span>.</span>iter</span>().</span>cloned</span>().</span>collect</span>();
</span>        </span>Box</span>::</span>new</span>(</span>vec</span>.</span>shrink</span>().</span>map</span>(|</span>v</span>| </span>v</span>.</span>into_iter</span>().</span>collect</span>::<</span>Self</span>>()))
</span>    </span>}
</span>}
</span></code></pre>
Putting those two together, we have another simple test:</p>
use </span>std</span>::</span>cmp</span>::</span>max</span>;
</span>use </span>itertools</span>::</span>all</span>;
</span>
</span>#[</span>quickcheck</span>]
</span>fn </span>node_height</span>(</span>set</span>: </span>AvlTreeSet</span><</span>u16</span>>) -> </span>bool </span>{
</span>    all</span>(</span>set</span>.</span>node_iter</span>(), |</span>node</span>| {
</span>        node</span>.</span>height </span>== </span>1 </span>+ </span>max</span>(</span>node</span>.</span>left_height</span>(),</span> node</span>.</span>right_height</span>())
</span>    </span>})
</span>}
</span></code></pre>
Another succinct test that .update_height</code> correctly updates the
child all the way up to the root and that our use of unsafe</code> is
truly safe. Whether you agree with my use of unsafe</code>, I learned a
certain reverence in asking permission to use dangerous features and
heightened responsibility in upholding the law that reminds me of
Haskell's IO</a> boundaries. Nonetheless, we can now tackle node
rotation.</p>
>>></a>
Node Rotation</h3>
After inserting a node in a balanced AVL tree, it becomes
imbalanced</strong> when a node has a left and right subtree height
difference of 2</code> or -2</code>. To correct this, we move nodes around while
preserving order. Visualizing the operation for rotating nodes to the
right:</p>
  </p>
Given a node, the idea is to promote the left child as the root and
demote the old root as a right child. While retaining other node
relationships, the orphaned left-right child of the old root is now
the right-left child of the new root. Those two movements constitute
the right rotation of nodes. As a general observation, moving nodes
from one side to the other should change the balance factor. Based on
the diagram above, we can say it changes by 2</code> perhaps. We'll verify
this observation later.</p>
Implementing the rotation took me a long time to do with two primary
constraints. First was to avoid using unsafe</code> again. Another is to
avoid create another node or heap allocated pointer via Box::new</code> to
be memory efficient. After several attempts, I came up with this safe
implementation of right rotation:</p>
use </span>std</span>::</span>mem</span>::{</span>replace</span>,</span> swap</span>};
</span>
</span>impl</span><</span>'a</span>, T</span>: </span>'a + Ord</span>> </span>AvlNode</span><</span>T</span>> {
</span>    </span>fn </span>rotate_right</span>(</span>&mut </span>self</span>) -> </span>bool </span>{
</span>        </span>if self</span>.</span>left</span>.</span>is_none</span>() {
</span>            </span>return false</span>;
</span>        </span>}
</span>
</span>        </span>// 1. Take nodes A and C
</span>        </span>let</span> left_node </span>= self</span>.</span>left</span>.</span>as_mut</span>().</span>unwrap</span>();
</span>        </span>let</span> left_right_tree </span>=</span> left_node</span>.</span>right</span>.</span>take</span>();
</span>        </span>let</span> left_left_tree </span>=</span> left_node</span>.</span>left</span>.</span>take</span>();
</span>
</span>        </span>// 2. Link A node to left node
</span>        </span>let mut</span> new_right_tree </span>= </span>replace</span>(</span>&mut self</span>.</span>left</span>,</span> left_left_tree</span>);
</span>        </span>// 3. Swap B and D node value to avoid moving the root
</span>        swap</span>(</span>&mut self</span>.</span>value</span>, </span>&mut</span> new_right_tree</span>.</span>as_mut</span>().</span>unwrap</span>().</span>value</span>);
</span>        </span>// 4. Take E node
</span>        </span>let</span> right_tree </span>= self</span>.</span>right</span>.</span>take</span>();
</span>
</span>        </span>// 5. Link C and E nodes to swapped D node
</span>        </span>let</span> new_right_node </span>=</span> new_right_tree</span>.</span>as_mut</span>().</span>unwrap</span>();
</span>        new_right_node</span>.</span>left </span>=</span> left_right_tree</span>;
</span>        new_right_node</span>.</span>right </span>=</span> right_tree</span>;
</span>        </span>// 6. Link swapped D node to root right node
</span>        </span>self</span>.</span>right </span>=</span> new_right_tree</span>;
</span>
</span>        </span>if let Some</span>(</span>node</span>) </span>= self</span>.</span>right</span>.</span>as_mut</span>() {
</span>            node</span>.</span>update_height</span>();
</span>        </span>}
</span>
</span>        </span>self</span>.</span>update_height</span>();
</span>
</span>        </span>true
</span>    </span>}
</span>}
</span></code></pre>
By carefully using std::mem::swap</a> and std::mem::replace</a>, we avoid
creating nodes by replacing the value and links of an existing node
instead. The main idea is about taking nodes from the left side and
placing them on the right carefully. Referring to the rotation
diagram while stepping through the code might help. After rotating,
we also need to update the heights of the right tree and the node
itself since they had new nodes. One minor thing is that we also
return a flag to indicate that the operation was executed or not for
our tests later. To compare and contrast, my first version is indeed
shorter yet memory inefficient:</p>
pub fn </span>rotate_right</span>(</span>&mut </span>self</span>) -> </span>bool </span>{
</span>    </span>// snip
</span>
</span>    </span>let</span> left_right_tree </span>= self</span>.</span>left</span>.</span>as_mut</span>().</span>unwrap</span>().</span>right</span>.</span>take</span>();
</span>    </span>let</span> new_root </span>= *</span>replace</span>(</span>&mut self</span>.</span>left</span>,</span> left_right_tree</span>).</span>unwrap</span>();
</span>    </span>let</span> old_root </span>= </span>replace</span>(</span>self</span>,</span> new_root</span>);
</span>
</span>    replace</span>(</span>&mut self</span>.</span>right</span>, </span>Some</span>(</span>Box</span>::</span>new</span>(</span>old_root</span>))); </span>// New pointer created here
</span>
</span>    </span>// snip
</span>}
</span></code></pre>
Assuming it is correct in either version, implementing the left
rotation is a bit of cheat by swapping every left</code> with right</code> text
in the function since they are symmetrical. To say that we have
working and safe rotations after days of working back and forth is
nothing short of bliss. As we did before, we need to prove it works.</p>
>>></a>
Rotation Testing</h3>
We have two statements to test about node rotation: rotations should
preserve order and adjust the balance factor</strong>. With everything
previously setup, we can test the first statement directly:</p>
use </span>quickcheck</span>::</span>TestResult</span>;
</span>
</span>#[</span>quickcheck</span>]
</span>fn </span>rotate_right_preserves_order</span>(</span>btree</span>: </span>BTreeSet</span><</span>u8</span>>) -></span> TestResult </span>{
</span>    </span>let mut</span> set </span>=</span> btree</span>.</span>iter</span>().</span>cloned</span>().</span>collect</span>::<</span>AvlTreeSet</span><</span>_</span>>>();
</span>
</span>    </span>if !</span>set</span>.</span>root</span>.</span>is_some</span>() {
</span>        </span>return </span>TestResult</span>::</span>discard</span>();
</span>    </span>}
</span>
</span>    </span>if !</span>set</span>.</span>root</span>.</span>as_mut</span>().</span>unwrap</span>().</span>rotate_right</span>() {
</span>        </span>return </span>TestResult</span>::</span>discard</span>();
</span>    </span>}
</span>
</span>    </span>TestResult</span>::</span>from_bool</span>(</span>equal</span>(</span>set</span>.</span>iter</span>(),</span> btree</span>.</span>iter</span>()))
</span>}
</span></code></pre>
Given a BTree, we clone that to an AVL and right rotate the root node
and check if the resulting iterators still matches in order and
value. We also want to ignore empty trees and nodes that cannot not
be rotated since they do not add value. For that, we use the discard
test</a> mechanics of quickcheck</em> with a quickcheck::TestResult</a> output
instead. If a test is discarded</a>, a new test sample is created instead
of ignoring it that would have eaten the maximum number of tests.
Same tactic applies to writing the left rotation.</p>
First statement down, we now check if right rotation does tilt the
balance factor say by 2</code>:</p>
#[</span>quickcheck</span>]
</span>fn </span>rotate_right_tils_balance_factor</span>(</span>xs</span>: </span>Vec</span><</span>u32</span>>) -></span> TestResult </span>{
</span>    </span>let mut</span> set </span>=</span> xs</span>.</span>iter</span>().</span>cloned</span>().</span>collect</span>::<</span>AvlTreeSet</span><</span>_</span>>>();
</span>
</span>    </span>if !</span>set</span>.</span>root</span>.</span>is_some</span>() {
</span>        </span>return </span>TestResult</span>::</span>discard</span>();
</span>    </span>}
</span>
</span>    </span>let</span> root_node </span>=</span> set</span>.</span>root</span>.</span>as_mut</span>().</span>unwrap</span>();
</span>    </span>let</span> balance_factor </span>=</span> root_node</span>.</span>balance_factor</span>();
</span>
</span>    </span>if !</span>root_node</span>.</span>rotate_right</span>() {
</span>        </span>return </span>TestResult</span>::</span>discard</span>();
</span>    </span>}
</span>
</span>    </span>let</span> tilted_factor </span>=</span> root_node</span>.</span>balance_factor</span>();
</span>
</span>    </span>TestResult</span>::</span>from_bool</span>(</span>balance_factor </span>-</span> tilted_factor </span>== </span>2</span>)
</span>}
</span></code></pre>
Running just this test with cargo test rotate_right_tilts_balance_factor</code>, this test fails on the list [2, 0, 1]</code>:</p>
    Finished dev [unoptimized + debuginfo] target(s) in 0.01s
</span>     Running target/debug/deps/avl_tree_set-ac5259255f5174fd
</span>
</span>running 1 test
</span>thread 'set::properties::rotate_right_tilts_balance_factor' panicked at '[quickcheck] TEST FAILED. Arguments: ([2, 0, 1])',
</span>/* More errors */
</span></code></pre>
Visualizing this counterexample:</p>
  </p>
Given this diagram, the balance factor is changed from 2</code> to -2</code>
that has a difference of 4</code> that fails the test. Even accommodating
that, the next counterexample is [2, 0, 1, 3]</code> where the balance
factor is 1</code> turned to -2</code> with a difference of 3</code>. With
subsequent exploration, the difference is always at least 2</code> but the
upper bound is unknown. This is where property based testing is
meaningful in expanding our empirical observation. Nonetheless, the
tilt still exists that proves that the rotation works although
weakly. We then update the condition to test a difference of at least
2</code> to pass and apply a similar treatment to writing the left
rotation test.</p>
To connect these two rotations in one property, what can we say about
those two operations when applied together? Can we say they are
inverse operations</a> that a right rotation undoes a left rotation or
vice-versa like with multiplication and division? Again, easily
verified:</p>
#[</span>quickcheck</span>]
</span>fn </span>rotate_left_and_rotate_left_identity</span>(</span>set</span>: </span>AvlTreeSet</span><</span>u8</span>>) -></span> TestResult </span>{
</span>    </span>if</span> set</span>.</span>root</span>.</span>is_none</span>() {
</span>        </span>return </span>TestResult</span>::</span>discard</span>();
</span>    </span>}
</span>
</span>    </span>let mut</span> rotated_set </span>=</span> set</span>.</span>clone</span>();
</span>    </span>let</span> root_node </span>=</span> rotated_set</span>.</span>root</span>.</span>as_mut</span>().</span>unwrap</span>();
</span>
</span>    </span>if</span> root_node</span>.</span>rotate_left</span>() {
</span>        root_node</span>.</span>rotate_right</span>();
</span>    </span>} </span>else </span>{
</span>        root_node</span>.</span>rotate_right</span>();
</span>        root_node</span>.</span>rotate_left</span>();
</span>    </span>}
</span>
</span>    </span>TestResult</span>::</span>from_bool</span>(</span>rotated_set </span>==</span> set</span>)
</span>}
</span></code></pre>
Given a set, we clone that and rotate left then right or whichever
applies first, then check if it is the same structure. Running the
test proves it is valid since no breaking example can be found:</p>
   Compiling avl_tree_set v0.1.0 (/home/nobody/avl_tree_set_rs)
</span>    Finished dev [unoptimized + debuginfo] target(s) in 0.27s
</span>     Running target/debug/deps/avl_tree_set-ac5259255f5174fd
</span>
</span>running 5 tests
</span>test set::properties::rotate_left_and_rotate_left_identity ... ok
</span>test set::properties::rotate_left_tilts_balance_factor ... ok
</span>test set::properties::rotate_right_tilts_balance_factor ... ok
</span>test set::properties::rotate_left_preserves_order ... ok
</span>test set::properties::rotate_right_preserves_order ... ok
</span>/* Omitted other results */
</span></code></pre>
With confidence from those supporting statements, we are now ready to
finish our insert method.</p>
>>></a>
Rebalancing Insert</h3>
If we have an imbalanced node, then wouldn't one correct rotation
restore balance? It is a good first intuition but our small
counterexample on the tilt property disproves that. However, we can
apply another rotation to a child node before applying the main
rotation and it is enough to work. You can find a mathematical proof
of induction here</a> if you are interested. Theory aside, the algorithm
can be thought of as branching paths:</p>

Give a root node, if the balance factor is 2</code> (left heavy), we inspect
the left node.</li>
If the left node has a balance factor of 1</code> (left heavy), we just
right rotate on the root node.</li>
Otherwise, we left rotate this inner left node and right rotate
the root node.</li>
Going back to the root node, if the balance factor is instead -2</code>
(right heavy), we inspect the right node.</li>
If the right node has a balance factor of -1</code> (right heavy), we just
left rotate the root node.</li>
Otherwise, we right rotate the inner right node and left rotate
the root node.</li>
</ol>
Translating this into code is rather direct with pattern matching:</p>
impl</span><</span>'a</span>, T</span>: </span>'a + Ord</span>> </span>AvlNode</span><</span>T</span>> {
</span>    </span>fn </span>rebalance</span>(</span>&mut </span>self</span>) -> </span>bool </span>{
</span>        </span>match self</span>.</span>balance_factor</span>() {
</span>            </span>-</span>2 </span>=> </span>{
</span>                </span>// Root is right heavy
</span>                </span>let</span> right_node </span>= self</span>.</span>right</span>.</span>as_mut</span>().</span>unwrap</span>();
</span>
</span>                </span>// Inner node is left heavy
</span>                </span>if</span> right_node</span>.</span>balance_factor</span>() </span>== </span>1 </span>{
</span>                    right_node</span>.</span>rotate_right</span>();
</span>                </span>}
</span>
</span>                </span>self</span>.</span>rotate_left</span>();
</span>
</span>                </span>true
</span>            </span>}
</span>
</span>            </span>2 </span>=> </span>{
</span>                </span>// Root is left heavy
</span>                </span>let</span> left_node </span>= self</span>.</span>left</span>.</span>as_mut</span>().</span>unwrap</span>();
</span>
</span>                </span>// Inner node is right heavy
</span>                </span>if</span> left_node</span>.</span>balance_factor</span>() </span>== -</span>1 </span>{
</span>                    left_node</span>.</span>rotate_left</span>();
</span>                </span>}
</span>
</span>                </span>self</span>.</span>rotate_right</span>();
</span>
</span>                </span>true
</span>            </span>}
</span>            </span>_ => false</span>,
</span>        </span>}
</span>    </span>}
</span>}
</span></code></pre>
We directly integrate this method back to .insert</code> as planned:</p>
fn </span>insert</span>(</span>&mut </span>self</span>, </span>value</span>:</span> T</span>) -> </span>bool </span>{
</span>    </span>// snip
</span>
</span>    </span>for</span> node_ptr </span>in</span> prev_ptrs</span>.</span>into_iter</span>().</span>rev</span>() {
</span>        </span>let</span> node </span>= unsafe </span>{ </span>&mut *</span>node_ptr </span>};
</span>
</span>        node</span>.</span>update_height</span>();
</span>        node</span>.</span>rebalance</span>();    </span>// Right here
</span>    </span>}
</span>
</span>    </span>true
</span>}
</span></code></pre>
To test the rebalancing property, we assert that every node should
not be imbalanced which is similar to the height test:</p>
#[</span>quickcheck</span>]
</span>fn </span>balanced_nodes</span>(</span>set</span>: </span>AvlTreeSet</span><</span>u16</span>>) -> </span>bool </span>{
</span>    all</span>(</span>set</span>.</span>node_iter</span>(), |</span>node</span>| </span>node</span>.</span>balance_factor</span>().</span>abs</span>() </span>< </span>2</span>)
</span>}
</span></code></pre>
Running this and all prior test specially with the height and
iterator property proves that our tree is balanced after every
insert:</p>
   Compiling avl_tree_set v0.1.0 (/home/nobody/avl_tree_set_rs)
</span>    Finished dev [unoptimized + debuginfo] target(s) in 0.27s
</span>     Running target/debug/deps/avl_tree_set-ac5259255f5174fd
</span>
</span>running 3 tests
</span>test set::properties::balanced_nodes ... ok
</span>test set::properties::node_height ... ok
</span>test set::properties::iterator_parity ... ok
</span>/* Omitting other tests */
</span></code></pre>
We now have a self-balancing tree and can now proclaim victory.</p>
>>></a>
Reflection</h3>
This was a challenging section. In particular with rotation, it was
hard to negotiate with the compiler how to move the nodes correctly.
In my darkest hour, I was truly tempted to use unsafe</code> to move the
pointers directly (if I could) as C</em> code seems so deceptively easy;
however, I would increase the error margin/surface of the design.
This problem was less about writing and technicalities but more about
understanding that requires patience and rigor. After this, I stopped
resisting the compiler and started to accept it.</p>
>></a>
Deletion</h2>
After all the struggle with .insert</code>, the deletion method, .take</code>,
is surprisingly the harder of the two methods. When deleting nodes, we
have to handle three cases for the deleted node to maintain the tree
structure: no children, one child and two children. This section is a
culmination of all the techniques.</p>

Delete Method</a></li>
Zero Or One Child</a></li>
Two Children</a></li>
Deletion Testing</a></li>
</ul>
>>></a>
Delete Method</h3>
Parity:</strong> BTreeSet::take</a></p>
If you noticed Option::take</a> when rotating nodes, it returns the owned
value and internally replaced by a None</code> value. For our tree, our
.take</code> method follows the same convention. Here is the whole
procedure to delete a node:</p>

Starting from the root node or with a current node</li>
Move to the left node if the value is less than the current node,
right if greater</li>
Do this until you have a node equal to that value, otherwise stop</li>
Delete that node and return the value</li>
If the deleted node has no children, stop</li>
If the deleted node has one child, substitute that child to
the removed node</li>
If the deleted node has two children, substitute the deleted node
with the leftmost child of the right subtree and link the orphaned
left child to the left child of this new node</li>
</ol>
These steps are short to write but are deceptively tricky but the
first few are done in the same manner as we did with .insert</code>:</p>
fn </span>take</span>(</span>&mut </span>self</span>, </span>value</span>: </span>&</span>T</span>) -> </span>Option</span><</span>T</span>> {
</span>    </span>let mut</span> prev_ptrs </span>= Vec</span>::<</span>*mut </span>AvlNode</span><</span>T</span>>>::</span>new</span>();
</span>    </span>// 1. Starting from the root node or with a current node
</span>    </span>let mut</span> current_tree </span>= &mut self</span>.</span>root</span>;
</span>    </span>let mut</span> target_value </span>= None</span>;
</span>
</span>    </span>while let Some</span>(</span>current_node</span>) </span>=</span> current_tree </span>{
</span>        </span>// 2. Move to the left node if the value is less than the current node,
</span>        </span>//    right if greater
</span>        </span>match</span> current_node</span>.</span>value</span>.</span>cmp</span>(</span>&</span>value</span>) {
</span>            </span>Ordering</span>::</span>Less </span>=> </span>{
</span>                prev_ptrs</span>.</span>push</span>(</span>&mut **</span>current_node</span>);
</span>                current_tree </span>= &mut</span> current_node</span>.</span>right</span>;
</span>            </span>}
</span>            </span>Ordering</span>::</span>Equal </span>=> </span>{
</span>                </span>// 3a. Do this until you have a node equal to that value, ...
</span>                target_value </span>= Some</span>(</span>&mut **</span>current_node</span>);
</span>                </span>break</span>;
</span>            </span>}
</span>            </span>Ordering</span>::</span>Greater </span>=> </span>{
</span>                prev_ptrs</span>.</span>push</span>(</span>&mut **</span>current_node</span>);
</span>                current_tree </span>= &mut</span> current_node</span>.</span>left</span>;
</span>            </span>}
</span>        </span>};
</span>    </span>}
</span>
</span>    </span>// 3b. ... otherwise stop
</span>    </span>if</span> target_value</span>.</span>as_ref</span>().</span>is_none</span>() {
</span>        </span>return None</span>;
</span>    </span>}
</span>
</span>    </span>let</span> target_node </span>=</span> target_value</span>.</span>unwrap</span>();
</span>
</span>    </span>// 4. Delete that node and return the value
</span>    </span>let mut</span> taken_value </span>= ???</span>;
</span>
</span>    </span>// Retrace the path and update them accordingly
</span>    </span>for</span> node_ptr </span>in</span> prev_ptrs</span>.</span>into_iter</span>().</span>rev</span>() {
</span>        </span>let</span> node </span>= unsafe </span>{ </span>&mut *</span>node_ptr </span>};
</span>        node</span>.</span>update_height</span>();
</span>        node</span>.</span>rebalance</span>();
</span>    </span>}
</span>
</span>    </span>Some</span>(</span>taken_value</span>)
</span>}
</span></code></pre>
This is the .insert</code> method with some modifications. We need the
node before .insert</code> stops, so the target_value</code> captures that
idea. It is also wrapped in an Option</code> to see check if a node is
found and stop otherwise. Also, we use taken_value</code> as the return
value after deleting the node. Before exiting, we make sure the
tracked nodes are updated before we leave.</p>
To emphasize, deleting or relocating nodes is a tricky operation
since it potentially moves many nodes. We exercise care as we handle
each case and consider height and balance if needed. Failing to
update nodes correctly will lead to subtle memory issues such as
mentioned in .balance_factor</code> and dropped nodes. We naturally start
with the safer case of zero or one child for the deleted node.</p>
>>></a>
Zero Or One Child</h3>
The case of deleting an empty node is about detaching the target
node. With one child either left or right, we still delete the node
and promote that child in its place. Weirdly, the case of deleting
one child is easier than deleting no child:</p>
fn </span>take</span>(</span>&mut </span>self</span>, </span>value</span>: </span>&</span>T</span>) -> </span>Option</span><</span>T</span>> {
</span>    </span>// snip
</span>
</span>    </span>// Deleting the node here
</span>    </span>let</span> taken_value </span>= if</span> target_node</span>.</span>left</span>.</span>is_none</span>() </span>||</span> target_node</span>.</span>right</span>.</span>is_none</span>() {
</span>        </span>if let Some</span>(</span>left_node</span>) </span>=</span> target_node</span>.</span>left</span>.</span>take</span>() {
</span>            replace</span>(</span>target_node</span>, </span>*</span>left_node</span>).</span>value
</span>        </span>} </span>else if let Some</span>(</span>right_node</span>) </span>=</span> target_node</span>.</span>right</span>.</span>take</span>() {
</span>            replace</span>(</span>target_node</span>, </span>*</span>right_node</span>).</span>value
</span>        </span>} </span>else </span>{
</span>            </span>// Deleting an no children
</span>        </span>}
</span>    </span>} </span>else </span>{
</span>        </span>unimplemented!</span>()
</span>    </span>}
</span>
</span>    </span>// snip
</span>}
</span></code></pre>
The tactic of the deleting node with replace</code> allows us to get
ownership of the node that gives us ownership of the value we want
to return. Since a child tree was promoted, no height or rebalancing
is needed for the replaced node.</p>
With the one child node, we can get a node and replace the deleted
node with that. With no children, we cannot use the same tactic.
Instead, we have to go the parent node and delete if from there:</p>
if let Some</span>(</span>left_node</span>) </span>=</span> target_node</span>.</span>left</span>.</span>take</span>() {
</span>    </span>// snip
</span>} </span>else if let Some</span>(</span>right_node</span>) </span>=</span> target_node</span>.</span>right</span>.</span>take</span>() {
</span>    </span>// snip
</span>} </span>else </span>{
</span>    </span>// 1a. Get parent node
</span>    </span>if let Some</span>(</span>prev_ptr</span>) </span>=</span> prev_ptrs</span>.</span>pop</span>() {
</span>        </span>let</span> prev_node </span>= unsafe </span>{ </span>&mut *</span>prev_ptr </span>};
</span>
</span>        </span>// 1b. Determine which node to remove: left or right node
</span>        </span>let</span> inner_value </span>= if let Some</span>(</span>ref</span> left_node</span>) </span>=</span> prev_node</span>.</span>left </span>{
</span>            </span>if</span> left_node</span>.</span>value </span>==</span> target_node</span>.</span>value </span>{
</span>                prev_node</span>.</span>left</span>.</span>take</span>().</span>unwrap</span>().</span>value
</span>            </span>} </span>else </span>{
</span>                prev_node</span>.</span>right</span>.</span>take</span>().</span>unwrap</span>().</span>value
</span>            </span>}
</span>        </span>} </span>else </span>{
</span>            prev_node</span>.</span>right</span>.</span>take</span>().</span>unwrap</span>().</span>value
</span>        </span>};
</span>
</span>        </span>// 2. Update parent node
</span>        prev_node</span>.</span>update_height</span>();
</span>        prev_node</span>.</span>rebalance</span>();
</span>
</span>        </span>// 3. Return value
</span>        inner_value
</span>    </span>} </span>else </span>{
</span>        </span>// 1c. ... otherwise it is the parent and take the root node
</span>        </span>self</span>.</span>root</span>.</span>take</span>().</span>unwrap</span>().</span>value
</span>    </span>}
</span>}
</span></code></pre>
Given the parent node, we determine which left or right node to take
by checking for equality once again. We could create an extra enum or
flag to remember the path, but it is a minor trade-off than adding
state. Either way, we take that node and returns its value; however,
since the parent node had a child removed, we update it accordingly.
Lastly, it is possible no previous nodes were traversed that implies
it is an isolated root node to delete which is easily handled.
Another case where pattern matching helps with illuminating edge
cases. With some confidence in our reasoning, we handle the hard case
of two children.</p>
>>></a>
Two Children</h3>
For two children, we replace the deleted node with the node of the
next value or the leftmost child of the right child of the deleted
node:</p>
  </p>
Describing the algorithm:</p>

Starting with the right child of the target node</li>
Keep moving the current node to the left child until it has none</li>
Replace the target node value with the current node value</li>
Replace the current node with the right child if it has one</li>
</ol>
Applying all the prior techniques for this final challenge:</p>
let</span> taken_value </span>= if</span> target_node</span>.</span>left</span>.</span>is_none</span>() </span>||</span> target_node</span>.</span>right</span>.</span>is_none</span>() {
</span>    </span>// snip
</span>} </span>else </span>{
</span>    </span>let</span> right_tree </span>= &mut</span> target_node</span>.</span>right</span>;
</span>
</span>    </span>// 1. Starting with the right child of the target node
</span>    </span>let mut</span> next_tree </span>=</span> right_tree</span>;
</span>    </span>// We also keep track of the parent nodes to update the height
</span>    </span>let mut</span> inner_ptrs </span>= Vec</span>::<</span>*mut </span>AvlNode</span><</span>T</span>>>::</span>new</span>();
</span>
</span>    </span>// 2. Keep moving the current node to the left child until it has none
</span>    </span>while let Some</span>(</span>next_left_node</span>) </span>=</span> next_tree </span>{
</span>        </span>if</span> next_left_node</span>.</span>left</span>.</span>is_some</span>() {
</span>            inner_ptrs</span>.</span>push</span>(</span>&mut **</span>next_left_node</span>);
</span>        </span>}
</span>        next_tree </span>= &mut</span> next_left_node</span>.</span>left</span>;
</span>    </span>}
</span>
</span>    </span>// We don't use next_tree but instead the tracked nodes as basis
</span>    </span>let</span> parent_left_node </span>= unsafe </span>{ </span>&mut *</span>inner_ptrs</span>.</span>pop</span>().</span>unwrap</span>() };
</span>    </span>let mut</span> leftmost_node </span>=</span> parent_left_node</span>.</span>left</span>.</span>take</span>().</span>unwrap</span>();
</span>
</span>    </span>// 3. Replace the target node value with the current node value
</span>    </span>let</span> inner_value </span>= </span>replace</span>(</span>&mut</span> target_node</span>.</span>value</span>,</span> leftmost_node</span>.</span>value</span>);
</span>    </span>// 4. Replace the current node with the right child if it has
</span>    replace</span>(</span>&mut</span> parent_left_node</span>.</span>left</span>,</span> leftmost_node</span>.</span>right</span>.</span>take</span>());
</span>
</span>    </span>// Updates the nodes in order
</span>    parent_left_node</span>.</span>update_height</span>();
</span>    parent_left_node</span>.</span>rebalance</span>();
</span>
</span>    </span>for</span> node_ptr </span>in</span> inner_ptrs</span>.</span>into_iter</span>().</span>rev</span>() {
</span>        </span>let</span> node </span>= unsafe </span>{ </span>&mut *</span>node_ptr </span>};
</span>        node</span>.</span>update_height</span>();
</span>        node</span>.</span>rebalance</span>();
</span>    </span>}
</span>
</span>    target_node</span>.</span>update_height</span>();
</span>    target_node</span>.</span>rebalance</span>();
</span>
</span>    inner_value
</span>}
</span></code></pre>
It is quite a mouthful and the last bit of complex code. It is like
an inner delete based on the child right tree that follows the same
conventions. After promoting a bottom node, we have an interesting
path of nodes to update and balance: the parent of the leftmost child
up to the originating target node. Stepping through the code should
be straightforward although more complex.</p>
For the astute reader, we have one final edge case as the code above
assumes that at least one tracked node always exist by the
declaration of parent_left_node</code> that only happens when the direct
right node has no left child based on the diagram. In that simpler
yet similar case, we replace that node with the right child if has
any:</p>
let</span> right_tree </span>= &mut</span> target_node</span>.</span>right</span>;
</span>
</span>if</span> right_tree</span>.</span>as_ref</span>().</span>unwrap</span>().</span>left</span>.</span>is_none</span>() {
</span>    </span>let mut</span> right_node </span>=</span> right_tree</span>.</span>take</span>().</span>unwrap</span>();
</span>
</span>    </span>let</span> inner_value </span>= </span>replace</span>(</span>&mut</span> target_node</span>.</span>value</span>,</span> right_node</span>.</span>value</span>);
</span>    replace</span>(</span>&mut</span> target_node</span>.</span>right</span>,</span> right_node</span>.</span>right</span>.</span>take</span>());
</span>
</span>    target_node</span>.</span>update_height</span>();
</span>    target_node</span>.</span>rebalance</span>();
</span>
</span>    inner_value
</span>} </span>else </span>{
</span>    </span>// Previous case
</span>}
</span></code></pre>
With that final edge case, the .take</code> method is complete and ready
to test.</p>
>>></a>
Deletion Testing</h3>
To test deletion, we need to test for existence that we do by
finally implementing a BTreeSet::contains</a> method. With everything
done so far, this is a relief:</p>
impl</span><</span>'a</span>, T</span>: </span>'a + Ord</span>> </span>AvlTreeSet</span><</span>T</span>> {
</span>    </span>pub fn </span>contains</span>(</span>&</span>self</span>, </span>value</span>: </span>&</span>T</span>) -> </span>bool </span>{
</span>        </span>let mut</span> current_tree </span>= &self</span>.</span>root</span>;
</span>
</span>        </span>while let Some</span>(</span>current_node</span>) </span>=</span> current_tree </span>{
</span>            </span>match</span> current_node</span>.</span>value</span>.</span>cmp</span>(</span>&</span>value</span>) {
</span>                </span>Ordering</span>::</span>Less </span>=> </span>{
</span>                    current_tree </span>= &</span>current_node</span>.</span>right</span>;
</span>                </span>}
</span>                </span>Ordering</span>::</span>Equal </span>=> </span>{
</span>                    </span>return true</span>;
</span>                </span>}
</span>                </span>Ordering</span>::</span>Greater </span>=> </span>{
</span>                    current_tree </span>= &</span>current_node</span>.</span>left</span>;
</span>                </span>}
</span>            </span>};
</span>        </span>}
</span>
</span>        </span>false
</span>    </span>}
</span>}
</span></code></pre>
Given an equal AVL and BTree set, .contains</code> should be the same
for every value</strong>. Testing .contains</code> will have a little twist. We
want to have a more defined positive and negative test cases, so when
cloning a list of numbers, we only include the even numbers as
positive cases and odd as negatives:</p>
#[</span>quickcheck</span>]
</span>fn </span>contains_parity</span>(</span>xs</span>: </span>Vec</span><</span>isize</span>>) -> </span>bool </span>{
</span>    </span>let</span> evens </span>=</span> xs
</span>        </span>.</span>iter</span>()
</span>        </span>.</span>cloned</span>()
</span>        </span>.</span>filter</span>(|</span>x</span>| </span>x </span>% </span>2 </span>== </span>0</span>)
</span>        </span>.</span>collect</span>::<</span>Vec</span><</span>_</span>>>();
</span>
</span>    </span>let</span> avl_set </span>=</span> evens</span>.</span>iter</span>().</span>cloned</span>().</span>collect</span>::<</span>AvlTreeSet</span><</span>_</span>>>();
</span>    </span>let</span> btree_set </span>=</span> evens</span>.</span>iter</span>().</span>cloned</span>().</span>collect</span>::<</span>BTreeSet</span><</span>_</span>>>();
</span>
</span>    all</span>(</span>xs</span>.</span>iter</span>(), |</span>x</span>| </span>avl_set</span>.</span>contains</span>(</span>x</span>) </span>==</span> btree_set</span>.</span>contains</span>(</span>x</span>))
</span>}
</span></code></pre>
As long as we have a subset of data, it is valid. Also we do not need
to filter test cases so we revert back to the familiar test template.
Assuming the test works, we use the same template of positive and
negative testing for .take</code> as well. Thanks to the previous
sections, we establish our tree has three core properties:</p>

Ordering</li>
Height</li>
Balanced</li>
</ol>
If we prove that each property holds after several deletions, it
shows our tree is balanced and correct. Writing a separate test for
each one might be more verbose, but it makes failing properties
easier to reason about.</p>
Given an equal AVL and BTree set, .take</code> should be the same for
every value</strong>. Before we check the core properties, we test if the
interface is correct:</p>
#[</span>quickcheck</span>]
</span>fn </span>take_parity</span>(</span>xs</span>: </span>Vec</span><</span>usize</span>>) -> </span>bool </span>{
</span>    </span>let</span> odds </span>=</span> xs
</span>        </span>.</span>iter</span>()
</span>        </span>.</span>cloned</span>()
</span>        </span>.</span>filter</span>(|</span>x</span>| </span>x </span>% </span>2 </span>== </span>1</span>)
</span>        </span>.</span>collect</span>::<</span>Vec</span><</span>_</span>>>();
</span>    </span>let mut</span> avl_set </span>=</span> odds</span>.</span>iter</span>().</span>cloned</span>().</span>collect</span>::<</span>AvlTreeSet</span><</span>_</span>>>();
</span>    </span>let mut</span> btree_set </span>=</span> odds</span>.</span>iter</span>().</span>cloned</span>().</span>collect</span>::<</span>BTreeSet</span><</span>_</span>>>();
</span>
</span>    all</span>(</span>xs</span>.</span>iter</span>(), |</span>x</span>| </span>avl_set</span>.</span>take</span>(</span>x</span>) </span>==</span> btree_set</span>.</span>take</span>(</span>x</span>))
</span>}
</span>
</span></code></pre>
Establishing the methods are equal, we then test the first property
which is our comfortable iterator equality test.</p>
#[</span>quickcheck</span>]
</span>fn </span>take_iterator_parity</span>(</span>xs</span>: </span>Vec</span><</span>i16</span>>) -> </span>bool </span>{
</span>    </span>let</span> fives </span>=</span> xs
</span>        </span>.</span>iter</span>()
</span>        </span>.</span>cloned</span>()
</span>        </span>.</span>filter</span>(|</span>x</span>| </span>x </span>% </span>5 </span>== </span>0</span>)
</span>        </span>.</span>collect</span>::<</span>Vec</span><</span>_</span>>>();
</span>    </span>let mut</span> avl_set </span>=</span> xs</span>.</span>iter</span>().</span>cloned</span>().</span>collect</span>::<</span>AvlTreeSet</span><</span>_</span>>>();
</span>    </span>let mut</span> btree_set </span>=</span> xs</span>.</span>iter</span>().</span>cloned</span>().</span>collect</span>::<</span>BTreeSet</span><</span>_</span>>>();
</span>
</span>    </span>for</span> five </span>in</span> fives </span>{
</span>        </span>assert_eq!</span>(</span>avl_set</span>.</span>take</span>(</span>&</span>five</span>),</span> btree_set</span>.</span>take</span>(</span>&</span>five</span>));
</span>    </span>}
</span>
</span>    equal</span>(</span>avl_set</span>.</span>iter</span>(),</span> btree_set</span>.</span>iter</span>())
</span>}
</span></code></pre>
Aside from order, this guarantees our deletion does delete the
correct nodes and preserves the others. Next one is height via node
iterator:</p>
#[</span>quickcheck</span>]
</span>fn </span>take_height_nodes</span>(</span>xs</span>: </span>Vec</span><</span>isize</span>>) -> </span>bool </span>{
</span>    </span>let</span> negatives </span>=</span> xs</span>.</span>iter</span>().</span>cloned</span>().</span>filter</span>(|</span>&</span>x</span>| </span>x </span>< </span>0</span>).</span>collect</span>::<</span>Vec</span><</span>_</span>>>();
</span>    </span>let mut</span> set </span>=</span> xs</span>.</span>iter</span>().</span>cloned</span>().</span>collect</span>::<</span>AvlTreeSet</span><</span>_</span>>>();
</span>
</span>    </span>for</span> negative </span>in</span> negatives </span>{
</span>        set</span>.</span>take</span>(</span>&</span>negative</span>);
</span>    </span>}
</span>
</span>    all</span>(</span>set</span>.</span>node_iter</span>(), |</span>node</span>| {
</span>        node</span>.</span>height </span>== </span>1 </span>+ </span>max</span>(</span>node</span>.</span>left_height</span>(),</span> node</span>.</span>right_height</span>())
</span>    </span>})
</span>}
</span></code></pre>
Having correct heights is a good sign that nodes are updated
properly. Topping it off, the balance factor is checked with the same
mechanics:</p>
#[</span>quickcheck</span>]
</span>fn </span>take_balanced_nodes</span>(</span>xs</span>: </span>Vec</span><</span>usize</span>>) -> </span>bool </span>{
</span>    </span>let</span> odds </span>=</span> xs
</span>        </span>.</span>iter</span>()
</span>        </span>.</span>cloned</span>()
</span>        </span>.</span>filter</span>(|</span>x</span>| </span>x </span>% </span>2 </span>== </span>1</span>)
</span>        </span>.</span>collect</span>::<</span>Vec</span><</span>_</span>>>();</span>n
</span>    </span>let mut</span> set </span>=</span> xs</span>.</span>iter</span>().</span>cloned</span>().</span>collect</span>::<</span>AvlTreeSet</span><</span>_</span>>>();
</span>
</span>    </span>for</span> odd </span>in</span> odds </span>{
</span>        set</span>.</span>take</span>(</span>&</span>odd</span>);
</span>    </span>}
</span>
</span>    all</span>(</span>set</span>.</span>node_iter</span>(), |</span>node</span>| </span>node</span>.</span>balance_factor</span>().</span>abs</span>() </span>< </span>2</span>)
</span>}
</span></code></pre>
Running the tests one last time proves victorious:</p>
   Compiling avl_tree_set_rs v0.1.0 (/home/nobody/avl_tree_set_rs)
</span>    Finished dev [unoptimized + debuginfo] target(s) in 0.12s
</span>     Running target/debug/deps/avl_tree_set_rs-551042db094327bb
</span>
</span>running 4 tests
</span>/* Passing previous tests */
</span>test set::properties::take_height_nodes ... ok
</span>test set::properties::take_balanced_nodes ... ok
</span>test set::properties::take_iterator_parity ... ok
</span>test set::properties::take_parity ... ok
</span></code></pre>
>>></a>
Reflection</h3>
Originally, I used rspec</a> and fake-rs</a> as my testing strategy that gave
me a good quality test workflow. In writing this article, I moved to
quickcheck</em> as a better paradigm to test abstract data structure.
Aside from being easier to write and iterate, porting my tests to
this paradigm gave me failing tests specially on this dangerous
method. That was a good catch that also gave me better and smaller
counterexamples to find the memory issues in my code.</p>
Aside from property based testing against a memory issue, working on
deletion is not that technical or hard. The prior two sections have
trained me to implement the algorithm with no fuzz. I am at peace, I
am productive.</p>
>></a>
Beyond Parity</h2>
In this article, we are done with the major goals of our project.
During my spare time, I implemented every struct method (not traits)
from BTreeSet</code> to AvlTreeSet</code> as practice. While insertion and
deletion were the hardest methods, everything else was cakewalk, so
checkout the code repository</a> if completion piques your interest. This
optional section is about a few things I found interesting.</p>

Benchmark</a></li>
Memory Safety</a></li>
</ul>
>>></a>
Benchmark</h3>
As a minor fancy, how does our implementation perform against the
standard library? Consulting a benchmarking chapter</a>, we can benchmark
the insert operation by generating random numbers and timing it using
this shortened snippet:</p>
#[</span>bench</span>]
</span>fn </span>setup_random_btree_set</span>(</span>b</span>: </span>&mut</span> Bencher</span>) {
</span>    </span>let mut</span> rng </span>= </span>thread_rng</span>();
</span>    </span>let mut</span> set </span>= </span>BTreeSet</span>::</span>new</span>();
</span>
</span>    b</span>.</span>iter</span>(|| {
</span>        set</span>.</span>insert</span>(</span>rng</span>.</span>gen</span>::<</span>usize</span>>());
</span>    </span>});
</span>}
</span>
</span>#[</span>bench</span>]
</span>fn </span>setup_random_avltree_set</span>(</span>b</span>: </span>&mut</span> Bencher</span>) {
</span>    </span>let mut</span> rng </span>= </span>thread_rng</span>();
</span>    </span>let mut</span> set </span>= </span>AvlTreeSet</span>::</span>new</span>();
</span>
</span>    b</span>.</span>iter</span>(|| {
</span>        set</span>.</span>insert</span>(</span>rng</span>.</span>gen</span>::<</span>usize</span>>());
</span>    </span>});
</span>}
</span></code></pre>
Here, we use rand</a> crate to generate random numbers outside of
quickcheck</em> to populate our set and the base set. We conveniently
have a builtin cargo bench</a> command that works out of the box, so we
can just run cargo bench</code> and watch:</p>
    Finished release [optimized] target(s) in 0.04s
</span>     Running target/release/deps/avl_tree_set_rs-e74d4a8db1cacec9
</span>
</span>running 17 tests
</span>/* Passing prior tests */
</span>test bench::setup_random_avltree_set ... bench:       1,800 ns/iter (+/- 517)
</span>test bench::setup_random_btree_set   ... bench:         529 ns/iter (+/- 72)
</span></code></pre>
In this run, our toy set has insert performance that is more or less
four times slower than what Rust</em> has matured. It would be strange
if it was significantly close. As simple as .insert</code> is, the
bottleneck possibly lies in the rotations since they use many .take</code>
or memory operations which could be further optimized. We now have a
general sense of relative performance for our implementation.</p>
>>></a>
Memory Safety</h3>
Aside from the promise of safety, can we further prove our library
has no memory leaks? Asking this question lead me to this article</a> and
valgrind</a>: a memory error detector. Installing it and creating a small
main bin.rs</code> file that generates a set and removes every element:</p>
use </span>avl_tree_set</span>::</span>set</span>::</span>AvlTreeSet</span>;
</span>
</span>pub fn </span>main</span>() {
</span>    </span>let mut</span> set </span>= </span>(</span>1</span>..</span>10_000 </span>as u32</span>).</span>rev</span>().</span>collect</span>::<</span>AvlTreeSet</span><</span>_</span>>>();
</span>
</span>    </span>for</span> i </span>in </span>1</span>..</span>10_000 </span>{
</span>        set</span>.</span>take</span>(</span>&</span>i</span>);
</span>    </span>}
</span>}
</span></code></pre>
Compiling it into an executable via cargo build</code> and running
valgrind --leak-check=full ./target/debug/avl_tree_set</code></p>
==25728== Memcheck, a memory error detector
</span>==25728== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
</span>==25728== Using Valgrind-3.16.0.GIT and LibVEX; rerun with -h for copyright info
</span>==25728== Command: ./target/debug/avl_tree_set
</span>==25728==
</span>==25728==
</span>==25728== HEAP SUMMARY:
</span>==25728==     in use at exit: 0 bytes in 0 blocks
</span>==25728==   total heap usage: 101,389 allocs, 101,389 frees, 4,198,985 bytes allocated
</span>==25728==
</span>==25728== All heap blocks were freed -- no leaks are possible
</span>==25728==
</span>==25728== For lists of detected and suppressed errors, rerun with: -s
</span>==25728== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
</span></code></pre>
Having an external tool check our small executable reveals no memory
leaks that is comforting and assuring along with passing tests. With
the code before I started writing this article, I followed the same
steps above with passing tests, I ran into the following:</p>
=</span>=3010== </span>Memcheck,</span> a memory error detector
</span>=</span>=3010== </span>Copyright</span> (C) </span>2002-2017,</span> and GNU GPL</span>'</span>d, by Julian Seward et al.
</span>==3010== Using Valgrind-3.16.0.GIT and LibVEX; rerun with -h for copyright info
</span>==3010== Command: ./target/debug/avl_tree_set
</span>==3010==
</span>==3010== Invalid read of size 8
</span>==3010==    at 0x1104BE: core::option::Option<T>::as_ref (option.rs:239)
</span>==3010==    by 0x10EF32: avl_tree_set::tree::AvlNode<T>::left_height (tree.rs:16)
</span>==3010==    by 0x10F942: avl_tree_set::tree::AvlNode<T>::update_height (tree.rs:24)
</span>==3010==    by 0x112ABA: avl_tree_set::set::AvlTreeSet<T>::take (set.rs:107)
</span>==3010==    by 0x11113A: avl_tree_set::main (bin.rs:7)
</span>==3010==    by 0x113E6F: std::rt::lang_start::{{closure}} (rt.rs:64)
</span>==3010==    by 0x11A9B2: {{closure}} (rt.rs:49)
</span>==3010==    by 0x11A9B2: std::panicking::try::do_call (panicking.rs:294)
</span>==3010==    by 0x11C319: __rust_maybe_catch_panic (lib.rs:82)
</span>/* More errors down below */
</span></code></pre>
Conflicted with passing tests and a memory issue I could not
replicate in test, my denial thought that valgrind</code> was broken
instead. Switching to property based test helped in finding test
cases that causes memory issues. The offending code happens with the
deletion of a leaf node in the old code:</p>
if let Some</span>(</span>prev_ptr</span>) </span>=</span> prev_ptrs</span>.</span>pop</span>() {
</span>    </span>// snip
</span>
</span>    prev_node</span>.</span>update_height</span>();
</span>    prev_node</span>.</span>rebalance</span>();
</span>
</span>    </span>// Memory issue
</span>    target_node</span>.</span>update_height</span>();
</span>    target_node</span>.</span>rebalance</span>();
</span>
</span>    inner_value
</span>} </span>else </span>{
</span>    </span>self</span>.</span>root</span>.</span>take</span>().</span>unwrap</span>().</span>value
</span>}
</span></code></pre>
Recall in the no children case, we detach the target_node</code> and
updated the prev_node</code> instead. I forgot about that assumption and
carelessly updated the deleted node and remember that prev_node</code>
comes from an unsafe</code> pointer that breaks the single mutable
reference rule. Perhaps, target_node</code> has a child that that was part
of the rotation and became a dangling pointer afterwards. Although
our use of unsafe</code> was seemingly safe, my tests did not catch that.
As proud as I was of my old tests, having external tools offer
insights outside my ego and bias. So even when the code compiles and
passes, it is good to check running code.</p>
>></a>
Reflecting On Rust</h2>
Although not a shining example of AVL Trees</em>, it has been a good
first project. I believe it was a good vertical slice of Rust</em>. In
the beginning, understanding, patience and perseverance is required
with the new idioms and designs and specially for the compiler.
Working through the code, familiarizing new functions and traits and
grasping lifetimes were the only difficulty. Reflecting on its style,
it is a happy union/mix between object oriented and functional.
Overall, Rust</em> is overall a challenging yet rewarding language with
little to complain syntactically.</p>
Looking back on Ruby</em> and Elixir</em>, I frequently find myself looking
for the &</code> or borrow operator in the variables. Working with the
Borrow Checker</em>, I believe, has been a positive experience although
rigorous. As such, I realize that this intricate mechanism cannot
easily be ported to other language without design considerations. If
Elixir</em> had Rust</em>'s mechanics, I doubt it would be as popular or as
ergonomic in its (almost) actor/process model. (Perhaps Pony language</a>
will be the evolved version when it matures?) As a modern observation,
languages that value a form application safety (memory, data,
correctness) tends to have higher learning curves and barriers to
entry because of the required mindset and philosophies. With better
ergonomics</a>, documentation and tooling, the learning curve is somewhat
balanced. So despite its accessibility, Rust</em> definitely deserves
attention and discussion for its rare core value of safety.</p>
This was a long article to prepare and write, so thank you for making
it to the end. I hope you found the journey meaningful or valuable in
some way. What is the next learning project? Aside from reading The
Rustonomicon</a> first, here are some project ideas:</p>
Make a Lisp</a> in Rust</strong>
: Implementing a (LISP) language is a dream.</p>
GraphQL Server</strong>
: Is Rust</em> good for the usual web development or microservices?</p>
Web Scraping/Crawling</strong>
: A good opportunity to learn asynchronous and concurrent operations.</p>
C Interop</strong>
: How good is the workflow in integrating with native libraries?</p>
Web Assembly</strong>
: A backend language in the browser is interesting</p>
As a newcomer, I am excited with the new doors Rust</em> has opened for
me. For now, I feel ecstatic in writing a valid library that I am
confidently safer than it would be in C</em>. That is good enough for me.</p>


Exploring Emacs chart Library
Unknown — Sat, 15 Apr 2017 00:00:00 +0000
>></a>
Bar Charts</h2>
Bar Charts</a> is a simple way to visualize grouped data through colorful
rectangles. In the world of text buffers and shell output, it is
surprising that Emacs has a library to draw colored rectangles:
chart</code>.</p>
(</span>require 'chart</span>)
</span>
</span>(</span>chart</span>-</span>bar</span>-</span>quickie
</span> 'vertical </span>"</span>Favorite Type of Movie</span>"                        </span>;; Type & Title
</span> '</span>("</span>Comedy</span>" "</span>Action</span>" "</span>Romance</span>" "</span>Drama</span>" "</span>Sci-Fi</span>") "</span>Genre</span>"   </span>;; Keys & Label
</span> '</span>(</span>4 5 6 1 4</span>) "</span>People</span>"                                     </span>;; Values &  Label
</span> </span>)
</span></code></pre>
  </p>
As the screenshot shows above, it is not sophisticated enough to
warrant deep examination but still impressive within the realm of text
buffers. The context of being builtin leads to fascinating queries
about the Emacs interpreter such as memory-usage</code>.</p>
  </p>
If you want to try it out, you can use the command load-library</code> to
import chart</code>. It defines the following demo command for exploration:</p>
chart-emacs-storage</code>
: This charts the output of memory-usage</code></p>
chart-emacs-lists</code>
: Like chart-emacs-storage</code>, this counts buffers</em>, frames</em>,
processes</em>, faces</em> and x-display</em> within the Emacs session</p>
chart-rmail-from</code>
: With rmail</code>, it groups user sender occurrences by mail</p>
chart-space-usage</code>
: After selecting a directory, it shows the disk usage by file type
within said directory</p>
chart-file-count</code>
: After selecting a directory, it displays file type occurrences
within the directory</p>
chart-test-it-all</code>
: A chart testing positive and negative values</p>
Aside from those demo commands, how can we use it and make a chart?</p>
>></a>
Making A Chart</h2>
After some reading and exploration, the starting command or function
to understand this library is chart-bar-quickie</code>, the rest seem
internal. Although this function has a total of eight arguments which
two are optional, it encapsulates everything needed to make the chart
including labels and data rather than instantiating classes</em> and
objects</em>. A small snippet to elaborate:</p>
(</span>defun </span>fn</span>/</span>chart</span>-</span>comparator</span>-< </span>(</span>left right</span>)
</span>  </span>"</span>A sample comparator for the values.
</span>Each parameter is a key-value cos pair and should function as a
</span>comparator like `<', `=', or `>'.</span>"
</span>  </span>(</span>pcase</span>-let </span>((</span>`</span>(</span>,left</span>-</span>key . ,left</span>-</span>value</span>)</span> left</span>)    </span>;; left is a cons of key and value
</span>              </span>(</span>`</span>(</span>,right</span>-</span>key . ,right</span>-</span>value</span>)</span> right</span>) </span>;; ditto with right
</span>              </span>)
</span>    </span>(</span><</span> left</span>-</span>value right</span>-</span>value</span>)))
</span>
</span>(</span>chart</span>-</span>bar</span>-</span>quickie
</span> 'vertical                           </span>;; Chart direction, either 'vertical or 'horizontal
</span> </span>"</span>Chart Title</span>"                       </span>;; Chart title
</span> '</span>("</span>Alpha</span>" "</span>Beta</span>" "</span>Gamma</span>" "</span>Delta</span>")   </span>;; X-Axis values or keys
</span> </span>"</span>X Line</span>"                            </span>;; X-Axis label
</span> '</span>(</span>4 3 2 1</span>)                          </span>;; Y-Axis values
</span> </span>"</span>Y Line</span>"                            </span>;; Y-Axis label
</span> </span>;; Optional
</span> </span>3                                   </span>;; Max value allowed, anything higher is not shown
</span> </span>#</span>'fn/chart-comparator-<             </span>;; Sorting or ordering function
</span> </span>)
</span></code></pre>
  </p>
Once evaluated, this creates a new buffer based on the title and
renders the bar chart there. The display itself is inert unlike
tabulated-list-mode</code> or any special buffer modes, so do not expect
special key bindings or commands that sort, zoom, refresh the chart
out of the box. Aside from tempering expectations, all the major (if
not all) functionality is within that function. (Sadly, looking any
deeper does not yield any more insights.) Lastly, the only caveat here
is that the size of the keys and values should be the same which is
natural as it matches each key with the corresponding value.</p>
>></a>
Demonstration</h2>
As an example, let us tackle a statistical query on a dataset. We will
play with the Hair Eye Color dataset</a> which looks like this:</p>
"","Hair","Eye","Sex","Freq"
</span>"1","Black","Brown","Male",32
</span>"2","Brown","Brown","Male",53
</span>"3","Red","Brown","Male",10
</span>"4","Blond","Brown","Male",3
</span>"5","Black","Blue","Male",11
</span></code></pre>
Before we query, let us import the data which is a good exercise for
CSV parsing in Emacs:</p>
(</span>setq csv</span>-</span>dataset</span>-</span>file </span>(</span>expand</span>-</span>file</span>-</span>name </span>"</span>HairEyeColor.csv</span>" "</span>~/Downloads</span>"))
</span>
</span>(</span>defun </span>read-lines </span>(</span>file</span>) </span>;; Take your pick: custom, find-file, f-read-text
</span>  </span>(</span>with-</span>temp</span>-</span>buffer
</span>    </span>(</span>insert</span>-</span>file</span>-</span>contents</span>-</span>literally file</span>)
</span>    </span>(</span>split</span>-</span>string
</span>     </span>(</span>decode</span>-</span>coding</span>-</span>region </span>(</span>point</span>-min</span>) (</span>point</span>-max</span>)</span> 'utf</span>-</span>8 </span>t</span>)
</span>     </span>"</span>\n</span>"
</span>     </span>t</span>)))
</span>
</span>(</span>defun </span>read-csv-field </span>(</span>field</span>) </span>;; Remove the pesky enclosed double quoting
</span>  </span>(</span>cond
</span>   </span>((</span>zerop </span>(</span>length field</span>)) "")
</span>   </span>((</span>and </span>(</span>string-equal </span>(</span>substring</span>-</span>no</span>-</span>properties field </span>0 1</span>) "</span>\"</span>")
</span>         </span>(</span>string-equal </span>(</span>substring</span>-</span>no</span>-</span>properties field </span>-</span>1</span>) "</span>\"</span>"))
</span>    </span>(</span>substring</span>-</span>no</span>-</span>properties field </span>1 </span>-</span>1</span>))
</span>   </span>(</span>t </span>(</span>string</span>-to-</span>number field</span>))))
</span>
</span>(</span>defun </span>read-csv-line </span>(</span>line</span>)
</span>  </span>(</span>let </span>((</span>raw</span>-</span>fields </span>;; This assumes no rogue commas need escaping
</span>         </span>(</span>split</span>-</span>string line </span>"</span>,</span>")))
</span>    </span>(</span>mapcar </span>#</span>'read-csv-field</span> raw</span>-</span>fields</span>)))
</span>
</span>(</span>defun </span>read-csv-file </span>(</span>file</span>)
</span>  </span>(</span>mapcar </span>#</span>'read-csv-line </span>(</span>read</span>-</span>lines file</span>)))
</span>
</span>
</span>(</span>setq dataset </span>(</span>read</span>-</span>csv</span>-</span>file csv</span>-</span>dataset</span>-</span>file</span>))
</span>
</span>;; Generated output
</span>(("" "</span>Hair</span>" "</span>Eye</span>" "</span>Sex</span>" "</span>Freq</span>")
</span> </span>("</span>1</span>" "</span>Black</span>" "</span>Brown</span>" "</span>Male</span>" </span>32</span>)
</span> </span>("</span>2</span>" "</span>Brown</span>" "</span>Brown</span>" "</span>Male</span>" </span>53</span>)
</span> </span>("</span>3</span>" "</span>Red</span>" "</span>Brown</span>" "</span>Male</span>" </span>10</span>)
</span> </span>("</span>4</span>" "</span>Blond</span>" "</span>Brown</span>" "</span>Male</span>" </span>3</span>))
</span></code></pre>
Libraries exist for this purpose such as csv</code>, el-csv</code> or
parse-csv</code> and utilities like f</code> or parsec</code> but doing it yourself
is an occasional opportunity to review the basics. With that handled,
what can we ask of the data? What is the frequency of eye colors?</strong>
Before that, we need to group the data:</p>
(</span>setq headers </span>(</span>car dataset</span>) </span>;; Only mentioned, not needed
</span>      records </span>(</span>cdr dataset</span>))
</span>
</span>(</span>defun </span>group-by </span>(</span>f xs</span>) </span>;; A quick write on -group-by
</span>  </span>(</span>let </span>((</span>groups </span>(</span>list</span>)))
</span>    </span>(</span>mapc
</span>     </span>(</span>lambda </span>(</span>x</span>)
</span>       </span>(</span>let* </span>((</span>key </span>(</span>funcall f x</span>))
</span>              </span>(</span>key</span>-</span>group </span>(</span>assoc key groups</span>)))
</span>         </span>(</span>unless</span> key</span>-</span>group
</span>           </span>(</span>push </span>(</span>cons key </span>(</span>list</span>))</span> groups</span>)
</span>           </span>(</span>setq key</span>-</span>group </span>(</span>assoc key groups</span>)))
</span>         </span>(</span>setcdr key</span>-</span>group </span>(</span>cons x </span>(</span>cdr key</span>-</span>group</span>)))))
</span>     xs</span>)
</span>    groups</span>))
</span>
</span>(</span>defun </span>group-records-by-eye-color </span>(</span>records</span>)
</span>  </span>(</span>let* </span>((</span>raw</span>-</span>eye</span>-</span>groups </span>(</span>group</span>-</span>by </span>(</span>apply</span>-</span>partially </span>#</span>'nth </span>2</span>)</span> records</span>))
</span>         </span>(</span>eye</span>-</span>groups </span>(</span>mapcar
</span>                      </span>(</span>lambda </span>(</span>eye</span>-</span>group</span>)
</span>                        </span>(</span>pcase</span>-let </span>((</span>`</span>(</span>,eye</span>-</span>color . ,eye</span>-</span>records</span>)</span> eye</span>-</span>group</span>))
</span>                          </span>(</span>let </span>((</span>eye</span>-</span>frequencies </span>;; Extract frequencies and collect it
</span>                                 </span>(</span>mapcar </span>(</span>apply</span>-</span>partially </span>#</span>'nth </span>4</span>)</span> eye</span>-</span>records</span>)))
</span>                            </span>(</span>cons eye</span>-</span>color
</span>                                  </span>(</span>apply </span>#</span>'+</span> eye</span>-</span>frequencies</span>) </span>;; Shiv for sum
</span>                                  </span>))))
</span>                      raw</span>-</span>eye</span>-</span>groups</span>)))
</span>    eye</span>-</span>groups</span>))
</span>
</span>
</span>(</span>setq eye</span>-</span>color</span>-</span>groups </span>(</span>group</span>-</span>records</span>-</span>by</span>-</span>eye</span>-</span>color records</span>))
</span>
</span>;; Generated output
</span>(("</span>Green</span>"</span> . </span>64</span>)
</span> </span>("</span>Hazel</span>"</span> . </span>93</span>)
</span> </span>("</span>Blue</span>"</span> . </span>215</span>)
</span> </span>("</span>Brown</span>"</span> . </span>220</span>))
</span></code></pre>
After writing the extraction, getting the bar chart is
straightforward.</p>
(</span>chart</span>-</span>bar</span>-</span>quickie
</span> 'horizontal
</span> </span>"</span>Eye Colors</span>"
</span> </span>(</span>mapcar </span>#</span>'car</span> eye</span>-</span>color</span>-</span>groups</span>) "</span>Colors</span>"
</span> </span>(</span>mapcar </span>#</span>'cdr</span> eye</span>-</span>color</span>-</span>groups</span>) "</span>Frequency</span>")
</span></code></pre>
  </p>
Neat but let's arrange it by descending order:</p>
(</span>defun </span>on </span>(</span>f op</span>) </span>;; Haskell's on operator
</span>  </span>(</span>lexical</span>-let </span>((</span>f f</span>)    </span>;; Sad that parameters aren't lexically scoped here
</span>                </span>(</span>op op</span>))
</span>    </span>(</span>lambda </span>(</span>left right</span>)
</span>      </span>(</span>funcall op </span>(</span>funcall f left</span>) (</span>funcall f right</span>)))))
</span>
</span>(</span>chart</span>-</span>bar</span>-</span>quickie
</span> 'horizontal
</span> </span>"</span>Eye Colors - Descending</span>"
</span> </span>(</span>mapcar </span>#</span>'car</span> eye</span>-</span>color</span>-</span>groups</span>) "</span>Colors</span>"
</span> </span>(</span>mapcar </span>#</span>'cdr</span> eye</span>-</span>color</span>-</span>groups</span>) "</span>Frequency</span>"
</span> </span>nil
</span> </span>(</span>on </span>#</span>'cdr </span>#</span>'></span>) </span>;; A comparator lambda also works but done for variety
</span> </span>)
</span></code></pre>
  </p>
Looking at it, brown</em> and blue</em> eye colors are quite frequent.
Nothing groundbreaking or epic. How about applying the same process
with hair color</em>? We get the following chart:</p>
  </p>
With hair color</em> as the variable, brown</em> and blonds</em> are many.
Nothing fancy. We could change the variable into gender</em> but nothing
more would be gained. The real question for any visualization is what
does it mean?</strong> Rather, what insights can we interpret from the data?
Since it is a sample, it might not mean anything nor does it have to.
This is a library exploration, not a statistical lesson that is beyond
scope.</p>
>></a>
Conclusion</h2>
Strangely, that is all there is to it: a neat and easy builtin bar
chart library.</strong> Since it is not pixel precise, it does not compete
with the modern visualizations tools or languages such as Python</a>, R</a> or
Haskell</a> although it does not have to. It simply is nice to have and
fun to know.</p>
Interestingly, if bar charts can be made with ASCII, how about pies or
bubbles. Can Emacs be a modern visualization tool?</p>


Exploring Emacs rx Macro
Unknown — Thu, 30 Mar 2017 00:00:00 +0000
>></a>
Regular Expression</h2>
Some people, when confronted with a problem, think "I know, I'll use
regular expressions." Now they have two problems.</strong></em> - Jamie Zawinski</em></p>
Regular Expressions</a> are difficult to write and maintain. Instead of
harping about the problems, I want to explore what Emacs offers to
make writing them easier. In particular, I want to tackle the rx</a>
macro, the regular s-expression or lispy regular expression</strong>.</p>
(</span>require 's</span>)  </span>;; All we need is =s-matches-p=
</span>(</span>require 'rx</span>)
</span>
</span>;; Creating a regexp that will match -> <File> [<Line>:<Column] <Suggestion>
</span>(</span>setq this</span>-</span>file</span>-</span>name </span>"</span>blog.org</span>")
</span>
</span>(</span>s</span>-</span>matches</span>-</span>p
</span> </span>(</span>rx bol
</span>     </span>(</span>eval this</span>-</span>file</span>-</span>name</span>)
</span>     space
</span>     </span>"</span>[</span>" (</span>group </span>(</span>one</span>-or-</span>more digit</span>)) "</span>:</span>" (</span>group </span>(</span>one</span>-or-</span>more digit</span>)) "</span>]</span>"
</span>     space
</span>     </span>(</span>group </span>(</span>zero</span>-or-</span>more anything</span>))
</span>     eol</span>)
</span> </span>"</span>blog.org [17:16] Emacs Lisp, not emacs lisp</span>")
</span>
</span>;; Produced regexp, I do not want to write or maintain this by hand
</span>"^blog\\.org[[:space:]]\\[\\</span>(</span>[[:digit:]]</span>+</span>\\</span>)</span>:\\</span>(</span>[[:digit:]]</span>+</span>\\</span>)</span>][[:space:]]\\</span>(</span>\\</span>(</span>?:.\\|
</span>\\</span>)</span>*</span>\\</span>)</span>$"
</span></code></pre>
Although it is less concise, the example above illustrates the selling
point of writing regular expressions at a higher level: it is more
understandable, comfortable to write and easier to maintain</strong>. Rather,
the "lispyness</em>" of the expressions is more appropriate in the style
and heart of Emacs, working with symbolic expressions.</p>
The builtin rx</code> macro has no obvious manual but it has a symbol
documentation found via describe-function</code>. For a powerful idea, it
doesn't have strong examples in the wiki or web to promote it. Hackers
before users. To be fair, reading the documentation is enough but
examples or recipes would hasten comprehension</strong>. This is what this
article explores, thus the following sections are exploring some
syntax or construct. (Aside, the problems I use are found in Regular
Expression Cookbook</a> and if you found them intriguing, support the
author and buy the book.)</p>
>></a>
Strings And Quoting</h2>
STRING
</span>     matches string STRING literally.
</span>
</span>CHAR
</span>     matches character CHAR literally.
</span>
</span>‘</span>(</span>eval FORM</span>)</span>’
</span>     evaluate FORM and insert result.  If result is a string,
</span>     ‘regexp-quote’ it.
</span></code></pre>
PROBLEM:</strong> What (regular) expression matches this string:
The punctuation characters in the ASCII table are: !"#$%&'()*+,-./:;<=>?@[\]^_`{|}</code></p>
;; Escape the double quote here
</span>(</span>setq input </span>"</span>The punctuation characters in the ASCII table are: !</span>\"</span>#$%&'()*+,-./:;<=>?@[</span>\]</span>^_`{|}</span>")
</span>
</span>(</span>s</span>-</span>matches</span>-</span>p </span>(</span>rx </span>"</span>The punctuation characters in the ASCII table are: !</span>\"</span>#$%&'()*+,-./:;<=>?@[</span>\]</span>^_`{|}</span>")
</span>             input</span>) </span>;; Direct use of strings
</span>
</span>(</span>not </span>(</span>s</span>-</span>matches</span>-</span>p input input</span>)) </span>;; Does not work because of quoting
</span>(</span>s</span>-</span>matches</span>-</span>p </span>(</span>regexp</span>-</span>quote input</span>)</span> input</span>)
</span>
</span>(</span>s</span>-</span>matches</span>-</span>p </span>(</span>rx </span>(</span>eval input</span>))</span> input</span>) </span>;; More rx
</span></code></pre>
This problem is merely quoting or escaping syntax characters, that is
if you know what those syntax characters are. The function
regexp-quote</code>, which escapes those characters, is simple enough. This
is done by default by rx</code> when a string is passed in for simplicity.
Finally, string variables can be used through the eval</code> syntax, which
is like inserting values with backquoting.</p>
>></a>
Variables And Ranges</h2>
‘</span>(</span>any SET ...</span>)</span>’
</span>‘</span>(</span>in SET ...</span>)</span>’
</span>‘</span>(</span>char SET ...</span>)</span>’
</span>     matches any character in SET ....  SET may be a character or string.
</span>     Ranges of characters can be specified as ‘A-Z’ in strings.
</span>     Ranges may also be specified as conses like ‘</span>(</span>?A . ?Z</span>)</span>’.
</span>
</span>     SET may also be the name of a character class: ‘digit’,
</span>     ‘control’, ‘hex-digit’, ‘blank’, ‘graph’, ‘print’, ‘alnum’,
</span>     ‘alpha’, ‘ascii’, ‘nonascii’, ‘lower’, ‘punct’, ‘space’, ‘upper’,
</span>     ‘word’, or one of their synonyms.
</span></code></pre>
PROBLEM:</strong> Create one regular expression to match all common
misspellings of calendar, so you can find this word in a document
without having to trust the author’s spelling ability. Allow an a or e
to be used in each of the vowel positions.</p>
(</span>s</span>-</span>matches</span>-</span>p </span>(</span>rx </span>"</span>c</span>"
</span>                 </span>(</span>any </span>"</span>a</span>" "</span>e</span>")
</span>                 </span>"</span>l</span>"
</span>                 </span>(</span>any </span>"</span>a</span>" "</span>e</span>")
</span>                 </span>"</span>nd</span>"
</span>                 </span>(</span>any </span>"</span>a</span>" "</span>e</span>")
</span>                 </span>"</span>r</span>")
</span>             </span>"</span>celander</span>")
</span>
</span>(</span>setq misspelling</span>-</span>pattern `</span>(</span>any </span>"</span>a</span>" "</span>e</span>"))
</span>
</span>(</span>s</span>-</span>matches</span>-</span>p </span>(</span>rx </span>"</span>c</span>"
</span>                 </span>(</span>eval misspelling</span>-</span>pattern</span>)
</span>                 </span>"</span>l</span>"
</span>                 </span>(</span>eval misspelling</span>-</span>pattern</span>)
</span>                 </span>"</span>nd</span>"
</span>                 </span>(</span>eval misspelling</span>-</span>pattern</span>)
</span>                 </span>"</span>r</span>")
</span>             </span>"</span>calendar</span>")
</span>
</span>"c[ae]l[ae]nd[ae]r" </span>;; Generated pattern
</span></code></pre>
Aside from demonstrating a simple range construct, the use of
sub-patterns through the familiar eval</code> allows us to treat these
expressions more modularly, which helps us move away from a monolithic
concatenated string.</p>
PROBLEM:</strong> Create a regular expression to match a single
hexadecimal character.</p>
(</span>s</span>-</span>matches</span>-</span>p </span>(</span>rx </span>(</span>any </span>"</span>a-f</span>" "</span>A-F</span>" "</span>0-9</span>"))
</span>             </span>"</span>A</span>")
</span>(</span>s</span>-</span>matches</span>-</span>p </span>(</span>rx </span>(</span>in </span>"</span>a-f</span>" "</span>A-F</span>" "</span>0-9</span>"))
</span>             </span>"</span>A</span>") </span>;; Equivalently
</span>
</span>"[0-9A-Fa-f]" </span>;; Generated pattern
</span>
</span>
</span>(</span>s</span>-</span>matches</span>-</span>p </span>(</span>rx </span>(</span>char hex</span>-</span>digit</span>))
</span>             </span>"</span>d</span>") </span>;; More rx
</span>(</span>s</span>-</span>matches</span>-</span>p </span>(</span>rx hex</span>-</span>digit</span>)
</span>             </span>"</span>d</span>") </span>;; Equivalently
</span>
</span>"[[:xdigit:]]" </span>;; Generated pattern
</span></code></pre>
Lastly, the range syntax allows the familiar dashes to add character
range. Rather, the abstraction of special character ranges like
[:upper:]</code> or [:xdigit:]</code> is nice to know. Other useful constructs
such as word-start</code>, line-end</code>, and punctuation</code> exist that is
worthy to be explored.</p>
>></a>
Alternatives And Depth</h2>
‘</span>(</span>or</span> SEXP1 SEXP2 ...</span>)</span>’
</span>‘</span>(</span>| SEXP1 SEXP2 ...</span>)</span>’
</span>     matches anything that matches SEXP1 or SEXP2, etc.  If all
</span>     args are strings, use ‘regexp-opt’ to optimize the resulting
</span>     regular expression.
</span>
</span>‘</span>(</span>zero</span>-or-</span>one SEXP ...</span>)</span>’
</span>‘</span>(</span>optional SEXP ...</span>)</span>’
</span>‘</span>(</span>opt SEXP ...</span>)</span>’
</span>     matches zero or one occurrences of A.
</span>
</span>‘</span>(</span>and</span> SEXP1 SEXP2 ...</span>)</span>’
</span>‘</span>(</span>: SEXP1 SEXP2 ...</span>)</span>’
</span>‘</span>(</span>seq SEXP1 SEXP2 ...</span>)</span>’
</span>‘</span>(</span>sequence SEXP1 SEXP2 ...</span>)</span>’
</span>     matches what SEXP1 matches, followed by what SEXP2 matches, etc.
</span>
</span>‘</span>(</span>repeat</span> N SEXP</span>)</span>’
</span>‘</span>(</span>=</span> N SEXP ...</span>)</span>’
</span>     matches N occurrences.
</span></code></pre>
PROBLEM:</strong> Create a regular expression that when applied repeatedly to
the text Mary, Jane, and Sue went to Mary's house</code> will match Mary</code>,
Jane</code>, Sue</code> and then Mary</code> again.</p>
(</span>s</span>-</span>match</span>-</span>strings</span>-</span>all
</span> </span>(</span>rx </span>(</span>or </span>"</span>Mary</span>" "</span>Jane</span>" "</span>Sue</span>"))
</span> </span>"</span>Mary, Jane, and Sue went to Mary's house</span>")
</span>
</span>;; Output
</span>'</span>(("</span>Mary</span>") ("</span>Jane</span>") ("</span>Sue</span>") ("</span>Mary</span>"))
</span>
</span>;; Generated pattern
</span>"\\</span>(</span>?:Jane\\|Mary\\|Sue\\</span>)</span>"
</span></code></pre>
This simple problem is a demonstration of using the alternation
construct, which is related to ranges and classes. Nothing fancy but
the possibility of making it nuanced exist.</p>
PROBLEM:</strong> Create a regular expression matching 0 to 255.</p>
(</span>setq range</span>-</span>expression </span>;; Expression and pattern separated for reuse
</span>      `</span>(</span>or </span>"</span>0</span>"
</span>           </span>(</span>sequence </span>"</span>1</span>" (</span>optional digit </span>(</span>optional digit</span>)))
</span>           </span>(</span>sequence </span>"</span>2</span>" (</span>optional
</span>                          </span>(</span>or
</span>                           </span>(</span>sequence </span>(</span>any </span>"</span>0-4</span>") (</span>optional digit</span>))
</span>                           </span>(</span>sequence </span>"</span>5</span>" (</span>optional </span>(</span>any </span>"</span>0-5</span>")))
</span>                           </span>(</span>any </span>"</span>6-9</span>"))))
</span>           </span>(</span>sequence </span>(</span>any </span>"</span>3-9</span>") (</span>optional digit</span>))))
</span>
</span>(</span>setq range</span>-</span>pattern </span>(</span>rx bol </span>(</span>eval range</span>-</span>expression</span>)</span> eol</span>))
</span>
</span>;; A test for the regular expression
</span>(</span>require 'cl</span>)
</span>(</span>cl</span>-</span>every </span>(</span>lambda </span>(</span>number</span>)
</span>            </span>(</span>s</span>-</span>matches</span>-</span>p range</span>-</span>pattern </span>(</span>number</span>-to-</span>string number</span>)))
</span>          </span>(</span>number</span>-</span>sequence </span>0 255</span>))
</span>
</span>;; Generated pattern
</span>"0\\|1\\</span>(</span>?:[[:digit:]][[:digit:]]?\\</span>)</span>?\\|2\\</span>(</span>?:[</span>0</span>-</span>4</span>][[:digit:]]?\\|</span>5</span>[</span>0</span>-</span>5</span>]?\\|[</span>6</span>-</span>9</span>][[:digit:]]?\\</span>)</span>?\\|[3-9][[:digit:]]?"
</span>
</span>;; To use this IP Addresses
</span>(</span>setq ip4</span>-</span>pattern </span>(</span>rx bol
</span>                      </span>(</span>repeat </span>3 </span>(</span>sequence </span>(</span>eval range</span>-</span>expression</span>) "</span>.</span>"))
</span>                      </span>(</span>eval range</span>-</span>expression</span>)
</span>                      eol</span>))
</span>
</span>(</span>s</span>-</span>matches</span>-</span>p range</span>-</span>pattern </span>"</span>30</span>")
</span>(</span>s</span>-</span>matches</span>-</span>p ip4</span>-</span>pattern </span>"</span>300</span>")
</span>;; Testing for permutation might take too long, one is good enough
</span>(</span>s</span>-</span>matches</span>-</span>p ip4</span>-</span>pattern </span>"</span>61.12.234.30</span>")
</span>
</span>;; Generated pattern
</span>"\\</span>(</span>?:\\</span>(</span>?:</span>0</span>\\|</span>1</span>\\</span>(</span>?:[[:digit:]][[:digit:]]?\\</span>)</span>?\\|</span>2</span>\\</span>(</span>?:[</span>0</span>-</span>4</span>][[:digit:]]?\\|</span>5</span>[</span>0</span>-</span>5</span>]?\\|[</span>6</span>-</span>9</span>][[:digit:]]?\\</span>)</span>?\\|[</span>3</span>-</span>9</span>][[:digit:]]?\\</span>)</span>\\.\\</span>)</span>\\{3\\}\\</span>(</span>?:</span>0</span>\\|</span>1</span>\\</span>(</span>?:[[:digit:]][[:digit:]]?\\</span>)</span>?\\|</span>2</span>\\</span>(</span>?:[</span>0</span>-</span>4</span>][[:digit:]]?\\|</span>5</span>[</span>0</span>-</span>5</span>]?\\|[</span>6</span>-</span>9</span>][[:digit:]]?\\</span>)</span>?\\|[</span>3</span>-</span>9</span>][[:digit:]]?\\</span>)</span>"
</span></code></pre>
The idea of this expression is matching the first digit, then
considering the branches. Even if I don't explain in depth, the syntax
should be helpful; but three new constructs deserve some words. First,
the optional</code> or opt</code> syntax is the equivalent of the zero-or-one
construct. Second, the sequence</code> or seq</code> syntax is primarily an
expression wrapper, where a list not an atom is required. Third,
repeat</code> syntax is the same as the repetition construct of a prior
pattern. Regardless of the new syntax, the problem is just flexing the
syntax.</p>
Also, remember to write tests for regular expressions. I
made three mistakes on my first draft, thus test before publishing.
Strangely, regular expressions are like functions that can be property
checked.</p>
Before I forget, the eval</code> construct requires that the variables
exist in the interpreter; meaning, they have to be globally set via
setq</code> before being used. That is why two setters in the snippet set
up the expression and pattern separately and respectively. I suggest
setting the expression or pattern via defconst</code> or defvar</code> as
refactoring. It is unfortunate that let</code> will not work with eval</code> ,
but it isn't a huge cost.</p>
>></a>
Groups And Backreferencs</h2>
‘</span>(</span>submatch SEXP1 SEXP2 ...</span>)</span>’
</span>‘</span>(</span>group SEXP1 SEXP2 ...</span>)</span>’
</span>     like ‘and’, but makes the match accessible with ‘match-end’,
</span>     ‘match-beginning’, and ‘match-string’.
</span>
</span>‘</span>(</span>submatch</span>-</span>n N SEXP1 SEXP2 ...</span>)</span>’
</span>‘</span>(</span>group</span>-</span>n N SEXP1 SEXP2 ...</span>)</span>’
</span>     like ‘group’, but make it an explicitly-numbered group with
</span>     group number N.
</span>
</span></code></pre>
PROBLEM:</strong> Create a regular expression that matches any date in
yyyy-mm-dd format and separately captures the year, month, and day. As
extra challenge, make the groups named.</p>
(</span>setq date</span>-</span>pattern
</span>   </span>(</span>rx </span>(</span>group</span>-</span>n </span>3 </span>(</span>repeat </span>4</span> digit</span>))
</span>       </span>"</span>-</span>"
</span>       </span>(</span>group</span>-</span>n </span>2 </span>(</span>repeat </span>2</span> digit</span>))
</span>       </span>"</span>-</span>"
</span>       </span>(</span>group</span>-</span>n </span>1 </span>(</span>repeat </span>2</span> digit</span>))))
</span>
</span>(</span>s</span>-</span>match</span>-</span>strings</span>-</span>all date</span>-</span>pattern
</span>                     </span>(</span>format</span>-</span>time</span>-</span>string </span>"</span>%F</span>"))
</span>
</span>;; Output and pattern, notice it is day, month and year or reverse order
</span>"\\</span>(</span>?</span>3</span>:[[:digit:]]\\{</span>4</span>\\}\\</span>)</span>-\\</span>(</span>?</span>2</span>:[[:digit:]]\\{</span>2</span>\\}\\</span>)</span>-\\</span>(</span>?</span>1</span>:[[:digit:]]\\{</span>2</span>\\}\\</span>)</span>"
</span>'</span>(("</span>2017-03-30</span>" "</span>30</span>" "</span>03</span>" "</span>2017</span>"))
</span></code></pre>
Capturing groups are fundamental; however, this is where the syntax
needs works. Named groups aren't possible here, instead we are limited
to numbered groups. Closely, this is not a limitation of the macro but
the specific Emacs Lisp regex syntax; a more domain specific version
can be tuned. This example just shows not every feature is translated.</p>
The group-n</code> or group</code> syntax is obvious in intention. The first
argument represent the group number and the rest are the actual
expression. Nothing fancy.</p>
PROBLEM:</strong> Create a regular expression that matches "magical</em>" dates
in yyyy-mm-dd format. A date is magical if the year minus the century,
the month, and the day of the month are all the same number. For
example, 2008-08-08</code> is a magical date.</p>
(</span>setq magical</span>-</span>pattern
</span>   </span>(</span>rx
</span>    </span>(</span>repeat </span>2</span> digit</span>)
</span>    </span>(</span>group</span>-</span>n </span>1 </span>(</span>repeat </span>2</span> digit</span>))
</span>    </span>"</span>-</span>"
</span>    </span>(</span>backref </span>1</span>)
</span>    </span>"</span>-</span>"
</span>    </span>(</span>backref </span>1</span>)))
</span>
</span>(</span>s</span>-</span>matches</span>-</span>p magical</span>-</span>pattern
</span>             </span>"</span>2008-08-08</span>")
</span>
</span>;; Generated pattern
</span>"[[:digit:]]\\{2\\}\\</span>(</span>?</span>1</span>:[[:digit:]]\\{</span>2</span>\\}\\</span>)</span>-\\1-\\1"
</span></code></pre>
This just shows backreferences are available. The backref</code> syntax is
just invoking the group with the numeric argument. Again, nothing
complicated.</p>
>></a>
re-builder</h2>
To conclude this exploration, a UI exist for testing and experimenting
on regular expressions: re-builder</code>. Execute the command,
re-builder</code> or regexp-builder</code>, on a buffer containing the text,
then execute reb-change-syntax</code> and select rx</code>. The following
screencast can be illuminating.</p>
  </p>
This UI can handle raw expression but we are interested in how this
ties to rx</code>. To elaborate, every time the expression is updated, it
highlights any possible matches. Although it is not as dynamic or
programmatic, it is handy as a quick experiment and check.</p>
>></a>
Conclusion</h2>
This macro is not a replacement for learning regular expressions since
there are nuances that a DSL can cover such language specific syntax
like PCRE</a>; rather, productivity is the key. As for me, I do not want
to write raw regular expression, I would prefer an abstraction to make
it easier on the eyes and hands.</p>
Finally, I did not discuss all constructs but only the interesting
features that draw me in, and perhaps enchant you as well. Read The
Function Documentation</strong>.</p>
If this can be done for regular expressions, can it be applied for
SQL? An idea still waiting to be written.</p>


Now and Then, Here and There - Analysis
Unknown — Mon, 13 Mar 2017 00:00:00 +0000
>></a>
Now and Then, Here and There</h2>
Now and Then, Here and There (2000)</a> is a fascinating anime. Plainly, it
is about a kid who is sucked to another world by helping a mysterious
girl and wants to return home. Going to the other world, he must save
her and possibly overthrow the hegemony of a mad dictator. It seems
similar to Alice in Wonderland</a> but far from its childish playfulness.
I like this anime not for its dark themes but how it presents its
story. It is gripping and hypnotizing all throughout without being
dramatic or superfluous although this is not for the faint of heart or
the casual.</p>
  </p>
What inspired me to write an article is its ending. After watching, I
felt stunned and lost. This analysis is a way for me to cope and
understand and what it all meant. That itself is saying much about my
experience and its quality.</p>
SPOILERS AHEAD</strong></em></p>
>></a>
The End</h2>
We start at the end: the ending. As the usual story goes, our
protagonist, Shu, returns home and the castle in the sky destroyed. In
that respect, the ending should be a joyous occasion. Sadly, Lala
vanishing makes the adventure fruitless from a hero's perspective.</p>
  </p>
This is compounded by Sara staying in the other world. Initially, she
has the strongest desire to go back; by choosing to give life, she
hopes that all her suffering will bear fruit someday.</p>
  </p>
When all said is done, Shu is back to where he came from. Nothing is
brought or returned as if everything was a dark dream.</p>
  </p>
With the final lingering shot, how can we interpret this ending?</strong> In
comparison, The Wizard of Oz (1939)</a> resolves issues within the dream
while The Chronicles of Narnia series</a> carries lessons over. Here, our
protagonist has no issue aside from being a rash and impulsive boy. In
the end, he still is a rash and impulsive boy although slightly
improved.</p>
This ending can be a case for the futility of war. What did all the
fighting accomplish? Nothing but the loss of Nabuca, Sis, Boo and
countless other lives. It can be also the value of life since Shu
persuades people to live and Sara keeping the baby despite strong
reasons to the contrary. Many interpretations exist but how do we go
about it? To begin our analysis, I want to focus on an element of the
story: character development</strong>.</p>
>></a>
Character Development</h2>
A character with much development is Sara; despite being the
protagonist, Shu does not change or develop. Another way to put it is
that the other world has changed Sara while it left Shu unchanged.
This idea can be explored with Nabuca and Tabool.</p>
  </p>
Both of them came from the same village yet are changed differently.
In power, Nabuca is clearly stronger and controlled than Tabool; in
status, Nabuca is a faithful soldier while Tabool is a political
animal. As with Shu and Sara, both wanted to go home yet decide
differently. Near the end, Nabuca wants to stop killing while Tabool
wants to continue and eventually take over Hamdo's empire. That
difference culminates in Tabool betraying Nabuca.</p>
  </p>
Regardless, both of them die and consequently neither of them go back
to their village; however, Nabuca has a more meaningful story death
than Tabool whose death last less than a second of screen time.
Another way of looking at it is that since both were changed by the
war, the story does not allow them to return home.</p>
Thinking about our main pair, Shu returns home with his morals while
Sara does not who was broken by captivity. When their values were
tested, Shu does not kill Nabuca while Sara kills her rapist.
Therefore, is this story talking about maintaining humanity?</strong></p>
>></a>
Humanity and Backstory</h2>
How does war change people? Rather, how does war break people?</strong> Shu
could have been broken several times in the story but remains rooted.
Despite having little development, he is the only character that keeps
his values, a fixed point.</p>
  </p>
What is humanity or integrity? A complex question for anyone to
answer. Here, I mean it as a quality of keeping character and belief
but that itself is a vague answer. Instead, I want to point out
another element in the story that relates to it: backstory or the
lack of it</strong>.</p>
Given thirteen episodes, what do we know about our characters? Nothing
substantial. I do not know who Shu, Lala, Nabuca or anyone else. Why
is Hamdo crazy? Was he a good person before he went crazy for Lala?
Why is Abelia faithful to Hamdo? Were Nabuca and Tabool good kids
before? How old is Lala? Why is Shu naive to a fault? I can go on but
the backstory is definitely lacking or missing.</p>
Commonly, a backstory is used to empathize and understand a
character's motivation and actions. The key word is empathy. By
empathizing on characters, it changes your perception or judgment on
their action and consequences.</p>
  </p>
Consider Hamdo's death. In the story, Hamdo is presented as a weak and
paranoid dictator. His death might be fair judgment on his actions.
Imagine, if Lala made Hamdo crazy or was the reason? What if Hamdo was
a great and benevolent dictator, broken by time and circumstance? If
the backstory existed, would it change your personal judgment?
Concrete examples are Lelouch from Code Geass</a> or Homura from Puella
Magi Madoka Magica</a> if my example is lacking.</p>
By ignoring the backstory, the characters can only be judged within
the context of the story. It is their current actions that will define
them</strong>, not by how justifiable and tragic their past stories are. This
idea is exemplified in the child soldiers. Aside from Shu, Nabuca and
Tabool, Boo is the other soldier who appears neutral, not unchanged,
by the situation. We cannot say if his hands are dirty without past
information, but within the story he is good, further amplified when he
takes a bullet for Nabuca. A backstory exists within the story but not
enough to completely paint them.</p>
  </p>
The lack of backstory is asking us to make a fair judgment, but what
are we to judge? Humanity? Our characters are forced into this
situation, we should not care who they were, but who they are now. We
are to judge the effect of war on their character.</p>
In the first place, why should people stay true in dire times
specially for children placed in the horrors of war? Even without the
backstory, we can empathize how broken they are by their situation. To
dig deeper, we explore the mysterious character of Lala Ru.</p>
>></a>
Lala Ru</h2>
Because ten billion years' time is so fragile, so ephermal...</strong></em>
it arouses such a bittersweet, almost heartbreaking fondness</strong></em></p>
Lala, the main heroine, is passive, not incapable. She has the power
of water although at the cost of using her life force. Why does she
allow people to use or abuse her?</p>
  </p>
With her powers, Lala shares to Shu that she had been kind but has
been neglected and hunted. She believed she was doing good but it
brought more danger and chaos, thus she does not waste it for
humanity. This somewhat explains why she shows passivity or disdain
instead of force or power when threatened. If she tried to resist, she
would use her life force; after many times, silence seems to be louder.</p>
Going back, Shu responds that her power is ultimately for good and
good people still exist, such as himself and motherly Sis. What this
tells us that Lala has lost hope as with Sara. Based on the previous
ideas, Lala is not allowed to return home by the story. Strange
enough, where is her home? Where will she return to?</p>
If Lala's power is water and it depends on her life force, is Lala the
sea that fills the barren wasteland with life? Should Lala return to
the earth or ether?</p>
  </p>
This is speculation why Lala disappears in the end that conveniently
ties into the ideas presented. The question still remains to where
Lala should return. If she lives, does she go with Shu, stay with Sara
or something else entirely? Lala is mysterious and ethereal enough
that believing she is a magical being is not far off. The opening
remark can be a hint.</p>
  </p>
Based on this, the other world might be set in a post-apocalyptic
future, not necessarily the same timeline as the original world. This
might roughly indicate Lala's age that implies her home is somewhat
timeless. What is fascinating about ten billion years is that the sun
may roughly die at the time. It implies not just the sun, but humanity
and life itself.</p>
 &nbsp</p>
This somewhat explains her fascination with sunsets, not sunrises. If
I were her that lost hope, I want to wait for the sun to close my eyes
and make my bed. If Lala is practically immortal, a dying light is a
consoling and understanding friend. In comparison, she may not be like
C.C. from Code Geass</a>, but both share the isolation and burden of their
powers.</p>
Sadly, connecting the opening remarks with Lala is flimsy since no
text supports it but it does set a tone for which we can read from. It
would have helped connect the dots if someone described the world even
for just a minute. What we do now that Lala has lived for thousands of
years and much of our reasoning somewhat holds.</p>
Without a defined home, she breaks the idea or need to return home,
thus she is the exception. Instead, notice her final act is of
sacrifice. By using her power to sink the flying fortress knowing the
drawback and reason, she shows her restored faith and hope in
humanity. Phrased differently, Lala was looking for hope.</strong></p>
Urged by both the goodness of Shu and Sis, Lala shows both hope and
action. It is strange that Lala has not met good people for her to
have this problem but this is a backstory in itself. What is the story
asking by making the protagonist save her? This leads us to explore
what hope is in exploring the message.</p>
>></a>
Hope and Optimism</h2>
Another hard question, what is hope?</strong> Like with humanity, I can only
mean it here as quality of optimism or expectation for good things to
come. Instead, how does the series use the theme of hope or optimism?
Easily, Shu is the story's hope.</p>
  </p>
Consider Nabuca and Tabool, Nabuca is doing all the atrocities
required of him so that he can go back home; however, Tabool has
abandoned the false hope and embraced the belief of being the
dictator. When Nabuca discovers that their village was scorched after
being taken, he puts his misplaced hope in Shu returning in his stead.
Nabuca's change comes from Shu unwavering humanity and disruption of
their ranks and becomes his hope in the end.</p>
Consider Sara, she poses the most direct challenge to Shu. She was
taken, raped and soiled, how can she have hope in living in suffering?
This is in contrast with the child soldiers who creates suffering
instead of receiving. When Sara attempts to kill herself and by
extension her unborn child, it a powerful question to Shu's philosophy
of optimism and hope.</p>
  </p>
The hard thing to see here is that a child represents the hope of its
parents. Both Shu and Sis convince Sara to keep the child that
represents hope, not a burden or a taint. This answer affirms Shu and
the story.</p>
Consider Abelia, the scene where he lets Hamdo die is her loss of hope
or faith in him. Why has she been faithful to him all this time, we
may never know. Looking closely, it is Abelia that gives Shu a chance
to survive by not killing him. She is averse to killing yet has to out
of orders. In the end, what is left for her but to find a new hope to
carry on.</p>
Going back to Lala, she has lived for thousands of years and has lost
hope in humanity. Applying Shu's optimism to Lala's plight, Shu's
arrival is the hope or goodness her immortal life is anticipating. The
final conversation of Shu with Sara affirms this message of hope.</p>
SHU: Sara, you know, you once said I lied to you?
</span>     Remember that?
</span>     I still believe I was right.
</span>     No matter where you are, whatever adversities you face,
</span>     as long as you've got your life, something good is bound to happen.
</span>     Something so totally awesome that would make up for all the bad stuff.
</span>     Sure, it can get pretty tough in the meantime.
</span>     I don't quite know hot to put it but... you know... um...
</span>     Well, there you have it!
</span></code></pre>
In a way, Lala's disappearance makes sense when Shu's unwavering
humanity is put forward. Like with Nabuca, she is granted a meaningful
yet bittersweet death by completing her character arc.</p>
A subtle inference with this idea is that if Lala cannot be judged by
changes in war, then was she the cause? Rather, is she the judge that
deserves an answer to the chaos of her existence? In a way, she is the
beginning and the end of the story and of the world. This is
speculation but an interesting guess that reminds of Lady Koto in
Kyousougiga</a> who causes the story by her love and existence.</p>
>></a>
Returning Home</h2>
Going back to the ending, we return to the original world with the
original question: what did change? Rather, should anything change?</strong>
Shu coming back from war without changing is clever. Soldiers
returning from war can develop posttraumatic stress disorder (PTSD)</a> by
the experiences and hardships of war. Many films and games have
discussed this in greater detail such as First Blood from Rambo series</a>
and Spec Ops: The Line</a> that I might not be fit to address. Regardless,
our protagonist does not develop this crippling disorder by virtue of
his unwavering humanity.</p>
In the context of the disorder, it is good that Shu remained himself
despite his challenges. This ending cleverly undermines character
development in the context of tragedy or hardships. Simply, to not
change is better than to change for the worse. This is not a
prescription or cure of the disorder but a message to never lose
oneself. It is sad that nothing is gained but nothing was lost
that also ties in the theme of futility of war.</p>
By virtue of Shu returning home, the ending and story is telling us
to remain hopeful or true despite the lack of it.</strong> You could claim
the final discussion was the message but we needed to understand why
this hopeful message was shadowed by a return to zero and what it
meant. For Nabuca or Sara, the return is not zero. As the viewer, are
we still here and there?</p>
>></a>
Conclusion</h2>
At last, I am at peace with this anime. I do admit that the backstory,
world building, character development, animation and portrayal of dark
themes might be lacking; however, the somber and sublime story it
tells towers over its faults. At the end, I felt I wanted to watch a
sunset with Lala Ru someday. Not of affection but of solace.</p>
  </p>
A true favorite anime I can watch now and then.</p>


Literate Org-Haskell
Unknown — Wed, 22 Feb 2017 00:00:00 +0000
>></a>
Problem</h2>
I've been studying Haskell</a> and loving it. In order to facilitate my
learning, I wanted to explore it using literate programming</a> via
org-babel</a>. Aside from the usual setup of haskell-mode</a> and its
associates, I was eager to write the following block.</p>
#+BEGIN_SRC haskell
</span>myHead :: [x] -> Maybe x
</span>#+END_SRC
</span></code></pre>
Once I went to the coding, there is one significant problem I
encountered: my haskell tooling doesn't work when it has no backing
file.</strong> Emacs buffers represent just a file or nothing, such is the
case with org-src</code> buffers. Much of the tooling comes from ghc-mod</code>,
intero</code>, stack</code> and what not but all of them depend on a real file
and when a buffer doesn't have a file, it doesn't work. The obvious
solution is to simply make the virtual file into a real one.</p>
As obvious the solution is, the true error with haskell is that it
does not recognize files without a .hs</code> extension as such you cannot
just load a haskell source file without it having the correct
extension which is weird. So it is not enough to realize the file but
to add the proper extension as well. Here is the snippet</a> for the impatient.</p>
(</span>defconst fn</span>/</span>haskell</span>-</span>file</span>-</span>extension </span>"</span>.hs</span>"
</span>  </span>"</span>The de facto haskell file extension.</span>")
</span>
</span>(</span>defun </span>fn</span>/</span>add</span>-</span>haskell</span>-</span>file</span>-</span>extension </span>(</span>name</span>)
</span>  </span>"</span>Add the extension of .hs to a file or buffer NAME.</span>"
</span>  </span>(</span>if </span>(</span>string</span>/</span>ends</span>-with</span> name fn</span>/</span>haskell</span>-</span>file</span>-</span>extension</span>)
</span>      name </span>(</span>concat name fn</span>/</span>haskell</span>-</span>file</span>-</span>extension</span>)))
</span>
</span>(</span>defvar fn</span>/</span>org</span>-</span>haskell</span>-</span>mode</span>-</span>hook </span>nil
</span>  </span>"</span>Hook when buffer is haskellized.</span>")
</span>
</span>(</span>defun </span>fn</span>/</span>haskellize</span>-</span>buffer</span>-</span>file </span>(</span>&optional buffer</span>)
</span>  </span>"</span>Renames an BUFFER with a .hs extension if it doesn't have one.</span>"
</span>  </span>(</span>interactive</span>)
</span>  </span>(</span>with-</span>current</span>-</span>buffer </span>(</span>or</span> buffer </span>(</span>current</span>-</span>buffer</span>))
</span>    </span>(</span>save</span>-</span>buffer</span>)
</span>    </span>(</span>lexical</span>-let </span>((</span>name </span>(</span>buffer</span>-</span>name</span>))
</span>        </span>(</span>file</span>-</span>name </span>(</span>buffer</span>-</span>file</span>-</span>name</span>)))
</span>      </span>(</span>if </span>(</span>not </span>(</span>and</span> file</span>-</span>name </span>(</span>file</span>-</span>exists</span>-</span>p file</span>-</span>name</span>)))
</span>          </span>(</span>error </span>"</span>Buffer '%s' has no backing file</span>"</span> name</span>)
</span>        </span>(</span>lexical</span>-let </span>((</span>haskellized</span>-</span>name </span>(</span>fn</span>/</span>add</span>-</span>haskell</span>-</span>file</span>-</span>extension name</span>))
</span>            </span>(</span>haskellized</span>-</span>file</span>-</span>name </span>(</span>fn</span>/</span>add</span>-</span>haskell</span>-</span>file</span>-</span>extension file</span>-</span>name</span>)))
</span>          </span>(</span>cond
</span>           </span>((</span>get</span>-</span>buffer haskellized</span>-</span>name</span>)
</span>            </span>(</span>error </span>"</span>A buffer named '%s' already exists</span>"</span> haskellized</span>-</span>name</span>))
</span>           </span>((</span>string-equal name haskellized</span>-</span>name</span>)
</span>            </span>(</span>message </span>"</span>Buffer %s is already haskellized</span>"</span> haskellized</span>-</span>name</span>))
</span>           </span>(</span>t
</span>            </span>(</span>rename-file file</span>-</span>name haskellized</span>-</span>file</span>-</span>name </span>t</span>)
</span>            </span>(</span>rename</span>-</span>buffer haskellized</span>-</span>name</span>)
</span>            </span>(</span>set</span>-</span>visited</span>-</span>file</span>-</span>name haskellized</span>-</span>file</span>-</span>name</span>)
</span>            </span>(</span>set</span>-</span>buffer</span>-</span>modified</span>-</span>p </span>nil</span>)
</span>            </span>(</span>message </span>"</span>Buffer %s is now haskellized</span>"</span> haskellized</span>-</span>name</span>))))))))
</span>
</span>(</span>defun </span>fn</span>/</span>org</span>-</span>haskell</span>-</span>buffer</span>-</span>p </span>(</span>&optional buffer</span>)
</span>  </span>"</span>Check if BUFFER is an org-haskell buffer.</span>"
</span>  </span>(</span>with-</span>current</span>-</span>buffer </span>(</span>or</span> buffer </span>(</span>current</span>-</span>buffer</span>))
</span>    </span>(</span>and </span>(</span>eq</span> major</span>-</span>mode 'haskell</span>-</span>mode</span>)
</span>       </span>(</span>fboundp 'org</span>-</span>src</span>-</span>edit</span>-</span>buffer</span>-</span>p</span>)
</span>       </span>(</span>org</span>-</span>src</span>-</span>edit</span>-</span>buffer</span>-</span>p</span>))))
</span>
</span>(</span>defun </span>fn</span>/</span>haskellize</span>-</span>org</span>-</span>haskell</span>-</span>buffer </span>(</span>&rest _</span>)
</span>  </span>"</span>Haskellize org haskell buffer.</span>"
</span>  </span>(</span>when </span>(</span>fn</span>/</span>org</span>-</span>haskell</span>-</span>buffer</span>-</span>p</span>)
</span>    </span>(</span>fn</span>/</span>haskellize</span>-</span>buffer</span>-</span>file </span>(</span>current</span>-</span>buffer</span>))
</span>    </span>(</span>run</span>-</span>hooks 'fn</span>/</span>org</span>-</span>haskell</span>-</span>mode</span>-</span>hook</span>)))
</span>
</span>(</span>defun </span>fn</span>/</span>save</span>-</span>org</span>-</span>haskell</span>-</span>buffer </span>(</span>&rest _</span>)
</span>  </span>"</span>Save haskell buffer along with the edit buffer.</span>"
</span>  </span>(</span>when </span>(</span>fn</span>/</span>org</span>-</span>haskell</span>-</span>buffer</span>-</span>p</span>)
</span>    </span>(</span>save</span>-</span>buffer</span>)))
</span>
</span>(</span>defun </span>fn</span>/</span>cleanup</span>-</span>org</span>-</span>haskell</span>-</span>buffer </span>(</span>orig</span>-</span>fun &rest args</span>)
</span>  </span>"</span>Cleanup the org-haskell buffer when exiting the edit buffer.</span>"
</span>  </span>(</span>lexical</span>-let </span>((</span>org</span>-</span>haskell</span>-</span>file</span>-</span>name </span>(</span>buffer</span>-</span>file</span>-</span>name</span>))
</span>      </span>(</span>org</span>-</span>haskell</span>-</span>buffer</span>-</span>p </span>(</span>fn</span>/</span>org</span>-</span>haskell</span>-</span>buffer</span>-</span>p</span>)))
</span>    </span>(</span>prog1
</span>        </span>(</span>apply orig</span>-</span>fun args</span>)
</span>      </span>(</span>when </span>(</span>and </span>(</span>file</span>-</span>exists</span>-</span>p org</span>-</span>haskell</span>-</span>file</span>-</span>name</span>)</span> org</span>-</span>haskell</span>-</span>buffer</span>-</span>p</span>)
</span>        </span>(</span>delete-file org</span>-</span>haskell</span>-</span>file</span>-</span>name</span>)))))
</span>
</span>(</span>add</span>-</span>hook 'org</span>-</span>src</span>-</span>mode</span>-</span>hook </span>#</span>'fn/haskellize-org-haskell-buffer t</span>)
</span>
</span>(</span>advice</span>-</span>add 'org</span>-</span>edit</span>-</span>src</span>-</span>save :before </span>#</span>'fn/save-org-haskell-buffer</span>)
</span>(</span>advice</span>-</span>add 'org</span>-</span>edit</span>-</span>src</span>-</span>exit :around </span>#</span>'fn/cleanup-org-haskell-buffer</span>)
</span></code></pre>
Quite a mouthful for such a simple intent and this might apply to
other babel buffers. I want to explore the nuance of being literate
in org-mode</span>.</p>
>></a>
Buffers</h2>
In Emacs, you usually create a buffer associated with a file via
find-file</code>. There are times when all you need is a temporary
scratchpad and you do not want the cost of managing the file system,
such is the purpose of switch-buffer</code> and *scratch*</code> buffer. If I
didn't use the literate style, the book I am reading would create a
clutter and would I remember all the files after I finish with the
book? Literate programming allows me to keep one file and export it
via org-tangle</code>.</p>
The way org-babel</code> does it by creating special buffers that rebind
the save-buffer</code> command to org-edit-src-save</code> which updates the
original block region thereby creating no extra files while editing
inside blocks. The case is also true for several libraries such as
magit</a> and helm</a>. Thankfully, Emacs allows us to realize buffers by
simply using save-buffer</code> and it creates a backing file for it.</p>
However, we are left now with managing the realized file as well as
changing the file extension of it.</p>
>></a>
Babel Blocks</h2>
From the problems above, what we want is:</p>

When opening the block buffer, create a realized file with the
correct extension.</li>
When using the remapped org-edit-src-save</code>, also save the backing
file as with the original save-buffer</code>.</li>
When closing the block buffer, delete the realized file</li>
</ul>
The first thing anyone wants to look for when making an extension are
hooks. Sadly, the hooks that are relevant to our goal is only
org-src-mode-hook</code>. You can also add kill-buffer-hook</code> but it might
be out of its scope. Since we have no hooks, we have to resort to the
devious extensible advice-add</code>.</p>
After fiddling around with describe-function</code>, the exit command we
are interested in is org-edit-src-exit</code> and the entry command is the
org-src-mode-hook</code>. We include org-edit-src-save</code> to complete the
CRUD life cycle.</p>
>></a>
Renaming Files</h2>
The approach is hooking to org-src-mode-hook</code> to call save-buffer</code>
at the same time changing the file to add the extension. This is what
fn/haskellize-buffer-file</code> does which I want to give some focus on.
Once we realize the file, there is still the buffer and file
separation: if you rename the file, does it rename the buffer or
vice-versa?</p>
If you rename the buffer, it does not change the file associated with;
what is changed is buffer-name</code> via the rename-buffer</code> function. If
you rename the file through rename-file</code>, Emacs only see the old file
missing does creates another one if saved; what needs to be changed
rather is the file association which is done through the function
set-visited-file-name</code>. Emphasizing these lines are:</p>
(</span>rename-file file</span>-</span>name haskellized</span>-</span>file</span>-</span>name </span>t</span>)
</span>(</span>rename</span>-</span>buffer haskellized</span>-</span>name</span>)
</span>(</span>set</span>-</span>visited</span>-</span>file</span>-</span>name haskellized</span>-</span>file</span>-</span>name</span>)
</span></code></pre>
Aside from the usual file handling, his is the only nuance with the
separation when changing file or buffer names. Now we created realized
and renamed the buffer. Updating and deleting it is as simple calling
save-buffer</code> after org-edit-src-save</code> and removing it with
delete-file</code> after org-edit-src-exit</code>. Easy enough, but we have a
problem if we advice it head-on.</p>
>></a>
Local Advice</h2>
Since our code depends on a specific mode but advicing does not, if
you add the advice and modify some other code block, it will create
the file unintentionally. Our code needs to run on a specific major
mode, namely haskell-mode</code>. Wouldn't it be nice to have buffer local
advices like with variables? Since we don't, we add safety by checking
for the major mode.</p>
(</span>with-</span>current</span>-</span>buffer </span>(</span>current</span>-</span>buffer</span>)
</span>  </span>(</span>eq</span> major</span>-</span>mode 'haskell</span>-</span>mode</span>))
</span></code></pre>
And to be sure the current buffer is a babel buffer, we have
org-src-edit-buffer-p</code> which checks if the buffer is intended for
literacy.</p>
(</span>with-</span>current</span>-</span>buffer </span>(</span>current</span>-</span>buffer</span>)
</span>  </span>(</span>org</span>-</span>src</span>-</span>edit</span>-</span>buffer</span>-</span>p</span>))
</span></code></pre>
Combining the two is enough safety. With that, it is enough to get
through our intent.</p>
>></a>
Custom Hook</h2>
A feature you can add lastly is a custom hook, which is just a list of
functions and not some ethereal object such as a kill ring, when the
buffer is haskell-ized. Remember how to declare it still reminds me
how I need to write more lisp:</p>
(</span>defvar fn</span>/</span>org</span>-</span>haskell</span>-</span>mode</span>-</span>hook </span>(</span>list</span>)
</span>  </span>"</span>Hook when buffer is haskellized.</span>"
</span>  :type 'hook</span>)
</span></code></pre>
It is just a normal variable but with the type 'hook</code>, not obvious.
What can we do with this new hook, here is what I've done:</p>
(</span>defun </span>fn</span>/</span>haskell</span>-</span>process</span>-</span>load</span>-or-</span>reload </span>()
</span>  </span>"</span>Invoke reload process without switching buffers</span>"
</span>  </span>(</span>save</span>-</span>window</span>-</span>excursion
</span>    </span>(</span>haskell</span>-</span>process</span>-</span>load</span>-or-</span>reload</span>)))
</span>
</span>(</span>defun </span>fn</span>/</span>haskell</span>-</span>reload</span>-</span>on</span>-</span>save </span>()
</span>  </span>"</span>Reload interactive haskell process on save.</span>"
</span>  </span>(</span>add</span>-</span>hook 'after</span>-</span>save</span>-</span>hook 'fn</span>/</span>haskell</span>-</span>process</span>-</span>load</span>-or-</span>reload </span>t t</span>))
</span>
</span>(</span>defun </span>fn</span>/</span>hindent</span>-</span>before</span>-</span>save </span>()
</span>  </span>"</span>Reformat before saving.</span>"
</span>  </span>(</span>interactive</span>)
</span>  </span>(</span>add</span>-</span>hook 'before</span>-</span>save</span>-</span>hook 'hindent</span>-</span>reformat</span>-</span>buffer </span>t t</span>))
</span>
</span>(</span>add</span>-</span>hook 'fn</span>/</span>org</span>-</span>haskell</span>-</span>mode</span>-</span>hook 'fn</span>/</span>haskell</span>-</span>process</span>-</span>load</span>-or-</span>reload</span>)
</span>(</span>add</span>-</span>hook 'fn</span>/</span>org</span>-</span>haskell</span>-</span>mode</span>-</span>hook 'fn</span>/</span>haskell</span>-</span>reload</span>-</span>on</span>-</span>save</span>)
</span>(</span>add</span>-</span>hook 'fn</span>/</span>org</span>-</span>haskell</span>-</span>mode</span>-</span>hook 'fn</span>/</span>hindent</span>-</span>before</span>-</span>save</span>) </span>;;
</span></code></pre>
What I is to automatically feed the haskell code into the REPL and
update it once I saved it and some linting won't hurt as well. The
thing is I can't run any of these unless the buffer has an associated
haskell file with it. You can think of others.</p>
>></a>
Conclusion</h2>
As usual, this isn't enough to cover all cases but it does it good
enough. There is one other issue I haven't fully resolved which is the
tangling. The book exercises sometimes puts the code into a per
chapter folder which after updating the code must be tangled, the real
issue is how ox-haskell</code> or ob-haskell</code> has not respected the block
headers to do advanced tangling. More things to hack I guess.</p>


Project Script Loader
Unknown — Wed, 15 Feb 2017 00:00:00 +0000
>></a>
Problem</h2>
An early Emacs hack and study is creating a project script loader that
when I open a project such as Java or NodeJS, I want to load a custom
script specific to that project. Since I am using projectile</code>, it was
a great learning experience on understanding symbols and lambdas.</p>
The Emacs way of loading project specific configuration is through
=.dir-locals.el=</a> that has an unfortunately elaborate structure to fill
out. Emacs is a Lisp interpreter, thus this mechanism enforces safety
even in evaluating malicious code since it is possible in a multi-user
setup that someone tamper any automatic eval</code> mechanism to cause
harm. Remember the saying, eval is evil</a>.</p>
A stronger and direct reason you want to use such mechanism is to
avoid tangling your global configuration with a project specific
configuration. Keep the business and pure logic separate goes the
mantra. My use is with this blog, when I want to write I want to load
the org-jekyll-blogger.el</a> setup but not tangle the environment if I'm
not.</p>
Although it is the builtin and preferred mechanism, it is slightly
frustrating to debug if your script is being run or that the variables
are set. Not to mention it is harder to edit. Sometimes you just want
to write the code and not worry about security, trust is overrated.</p>
Since the project library projectile</code> does not offer this simple yet
security riddled functionality, it leaves the ecosystem to fill in
that blank. Packages offering this mechanism already exist such as
defproject</a> but it is simple to write the code without relying on a
third-party package.</p>
  </p>
For the impatient, here is the complete snippet</a>; for the curious,
let's discuss aspects of it.</p>
>></a>
Project Library</h2>
I am using projectile</a> as project managing library. For this task, we
need two functions from it:</p>
projectile-project-p</code>
: This tells us if the current buffer is in a project.</p>
projectile-project-root</code>
: This gives us the current project root the current buffer is in.</p>
projectile-project-name</code>
: The optional third function, this prettifies the file path into a
more debuggable name.</p>
If you aren't using projectile</code>, Emacs has a builtin project library
vc</code> that is tied closely to a VCS. An example of checking if the
buffer is in a project.</p>
(</span>defun </span>vcs-project-root </span>()
</span>  </span>(</span>with-</span>current</span>-</span>buffer </span>(</span>current</span>-</span>buffer</span>)
</span>    </span>(</span>lexical</span>-let </span>((</span>current</span>-</span>file </span>(</span>buffer</span>-</span>file</span>-</span>name</span>)))
</span>      </span>(</span>or </span>(</span>vc</span>-</span>git</span>-</span>root current</span>-</span>file</span>)
</span>         </span>(</span>vc</span>-</span>svn</span>-</span>root current</span>-</span>file</span>)
</span>         </span>(</span>vc</span>-</span>hg</span>-</span>root current</span>-</span>file</span>)))))
</span></code></pre>
This is rather primitive but it works if the project is under a
version control with git</code>, svn</code> or hg</code>. Thankfully, this assumption
was not taken by projectile</code> and it finds the root by looking for key
files that signify a project such as .git</code>, pom.xml</code> or others. The
builtin vc</code> does not cut it; rather, a builtin function that finds
the project root is locate-dominating-file</code>. This function takes a
file path and file name and recursively travels the parent to find the
file name starting at file path. If we assume a project is in a
directory that contains a .project.el</code> file, here is the snippet for
this:</p>
(</span>defun </span>locate-project-root </span>()
</span>  </span>(</span>locate</span>-</span>dominating</span>-</span>file </span>(</span>buffer</span>-</span>file</span>-</span>name</span>) "</span>.project.el</span>"))
</span></code></pre>
projectile</code> also has their own copy of this function to avoid
depending on files.el</code> where the original comes from. If you have
multiple key files aside from .project.el</code>, it is better to create a
more performant version of this since you do not want to traverse the
disk several times.</p>
The lesson in this is that this job is better left to a library.</p>
>></a>
Core</h2>
With this, we implement it quite easily:</p>
(</span>defun </span>fn</span>/</span>load</span>-</span>project</span>-</span>file </span>()
</span>  </span>"</span>Loads the `fn/project-file' for a project.
</span>This is run once after the project is loaded signifying project setup.</span>"
</span>  </span>(</span>interactive</span>)
</span>  </span>(</span>when </span>(</span>projectile</span>-</span>project</span>-</span>p</span>) </span>;; Check if buffer is in a project
</span>    </span>(</span>lexical</span>-let* </span>((</span>current</span>-</span>project</span>-</span>root </span>(</span>projectile</span>-</span>project</span>-</span>root</span>))
</span>        </span>(</span>project</span>-</span>init</span>-</span>file </span>(</span>expand</span>-</span>file</span>-</span>name </span>"</span>.project.el</span>"</span> current</span>-</span>project</span>-</span>root</span>)))
</span>      </span>(</span>when </span>(</span>file</span>-</span>exists</span>-</span>p project</span>-</span>init</span>-</span>file</span>) </span>;; Check if project script exists
</span>        </span>(</span>message </span>"</span>Loading project init file for %s</span>" (</span>projectile</span>-</span>project</span>-</span>name</span>)) </span>;; Some extra logging
</span>        </span>(</span>condition</span>-case</span> ex </span>;; Load it
</span>            </span>(</span>load project</span>-</span>init</span>-</span>file </span>t</span>)
</span>          </span>(</span>'error </span>;; Report the error
</span>           </span>(</span>message </span>"</span>There was an error loading %s: %s</span>"</span> project</span>-</span>init</span>-</span>file </span>(</span>error</span>-</span>message</span>-</span>string ex</span>))))))))
</span>
</span>(</span>add</span>-</span>hook 'find</span>-</span>file</span>-</span>hook </span>#</span>'fn/load-project-file</span>)
</span></code></pre>
During my early writing, there was a bug that loading the project file
would trigger the find-file-hook</code> endlessly. Thankfully, such subtle
issue had existed but either way it is simple to write.</p>
>></a>
Symbols Or Lambdas</h2>
This quick implementation triggers the project configuration each time
a file in the project is opened. What we want is each main project
script be loaded once, not every time. This is true for the project
locals but not for the main project script.</p>
Thinking functionally, this is memoization</a> of the main loader. We shiv
a quick memoization function:</p>
(</span>defun </span>fn</span>/</span>memoize </span>(</span>fn</span>)
</span>  </span>(</span>lexical</span>-let </span>((</span>fn fn</span>)
</span>      </span>(</span>cache</span>-</span>table </span>(</span>make-hash-table :test '</span>equal</span>)))
</span>    </span>(</span>lambda </span>(</span>&rest args</span>) </span>;; Assuming `args' can be used with the hash function `equal'
</span>      </span>(</span>lexical</span>-let </span>((</span>cached</span>-</span>value
</span>           </span>(</span>gethash args cache</span>-</span>table</span>)))
</span>        </span>(</span>if</span> cached</span>-</span>value
</span>            cached</span>-</span>value
</span>          </span>(</span>lexical</span>-let </span>((</span>computed</span>-</span>value </span>(</span>apply fn args</span>)))
</span>            </span>(</span>puthash args computed</span>-</span>value cache</span>-</span>table</span>)
</span>            computed</span>-</span>value</span>))))))
</span>
</span>(</span>defun </span>fib </span>(</span>n</span>)
</span>  </span>(</span>pcase n
</span>    </span>((</span>or </span>1 2</span>) </span>1</span>)
</span>    </span>(</span>_ </span>(</span>+ </span>(</span>fib </span>(</span>-</span> n </span>1</span>))
</span>          </span>(</span>fib </span>(</span>-</span> n </span>2</span>))))))
</span>
</span>(</span>lexical</span>-let </span>((</span>fib</span>-</span>memoized </span>(</span>fn</span>/</span>memoize </span>#</span>'fib</span>)))
</span>  </span>(</span>mapcar
</span>   </span>(</span>lambda </span>(</span>n</span>)
</span>     </span>(</span>funcall fib</span>-</span>memoized n</span>))
</span>   </span>(</span>list </span>3 5 15 30 3 30</span>)))
</span></code></pre>
Not the best implementation but notice we had to use let</code>, funcall</code>
and apply</code> to use it instead just the usual function invocation. This
syntactic mismatch or hoop is the difference with symbols and lambdas.
The memoization function returns a lambda, if we wanted to use it as a
function we need to use defun</code> or fset</code>.</p>
(</span>fset 'what</span>-</span>symbol-name </span>(</span>fn</span>/</span>memoize </span>#</span>'fib</span>))
</span></code></pre>
This raises the question what symbol name you should use? Generated or
clobbered? Managing symbols is another task but if we just ignore this
issue and plugin a lambda for a hook, we get this:</p>
(</span>add</span>-</span>hook
</span> 'find</span>-</span>file</span>-</span>hook
</span> </span>(</span>lambda </span>() </span>;; Written hastily, not representative
</span>   </span>(</span>lexical</span>-let </span>((</span>wrapped</span>-</span>func </span>;; We are wrapping `fn/load-project-file' since it needs to take an argument
</span>        </span>(</span>fn</span>/</span>memoize
</span>         </span>(</span>lambda </span>(</span>project</span>-</span>file</span>)
</span>           </span>(</span>fn</span>/</span>load</span>-</span>project</span>-</span>file</span>)))))
</span>     </span>(</span>funcall wrapped</span>-</span>func </span>(</span>buffer</span>-</span>file</span>-</span>name</span>))))
</span></code></pre>
Looks ugly doesn't it, what I find uglier is what is written in
find-file-hook</code>:</p>
;; (cl-prettyprint find-file-hook)
</span>((</span>lambda </span>nil
</span>   </span>(</span>progn
</span>     </span>(</span>defvar </span>--</span>cl</span>-</span>wrapped</span>-</span>func</span>--</span>)
</span>     </span>(</span>let </span>((</span>--</span>cl</span>-</span>wrapped</span>-</span>func</span>-- </span>(</span>fn</span>/</span>memoize </span>(</span>function
</span>                                             </span>(</span>lambda </span>(</span>project</span>-</span>file</span>)
</span>                                               </span>(</span>fn</span>/</span>load</span>-</span>project</span>-</span>file</span>))))))
</span>       </span>(</span>funcall </span>(</span>symbol-value '</span>--</span>cl</span>-</span>wrapped</span>-</span>func</span>--</span>)))))
</span> recentf</span>-</span>track</span>-</span>opened</span>-</span>file
</span> auto</span>-</span>insert
</span> whitespace</span>-</span>turn</span>-</span>on</span>-if-</span>enabled
</span> global</span>-</span>command</span>-</span>log</span>-</span>mode</span>-</span>check</span>-</span>buffers
</span> projectile</span>-</span>find</span>-</span>file</span>-</span>hook</span>-</span>function
</span> #[</span>0 </span>"</span>\3</span>02</span>\3</span>01!</span>\2</span>10</span>\3</span>03</span>\3</span>04!8</span>\2</span>11</span>\2</span>07</span>"</span> [buffer</span>-</span>file</span>-</span>name auto</span>-</span>revert</span>-</span>tail</span>-</span>pos make</span>-</span>local</span>-</span>variable </span>7</span> file</span>-</span>attributes] </span>3</span>]
</span> global</span>-</span>visual</span>-</span>line</span>-</span>mode</span>-</span>check</span>-</span>buffers
</span> auto</span>-</span>compile</span>-</span>on</span>-</span>save</span>-</span>mode</span>-</span>check</span>-</span>buffers
</span> url</span>-</span>handlers</span>-</span>set</span>-</span>buffer</span>-</span>mode
</span> global</span>-</span>font</span>-</span>lock</span>-</span>mode</span>-</span>check</span>-</span>buffers
</span> epa</span>-</span>file</span>-</span>find</span>-</span>file</span>-</span>hook
</span> vc</span>-</span>refresh</span>-</span>state
</span> fn</span>/</span>load</span>-</span>project</span>-</span>file
</span> fn</span>/</span>load</span>-</span>project</span>-</span>local</span>-</span>file
</span> which</span>-</span>func</span>-</span>ff</span>-</span>hook
</span> org</span>-</span>jekyll</span>-</span>blogger</span>--</span>find</span>-</span>file</span>-</span>hook</span>)
</span></code></pre>
Do you see the lambda</code> standing out from the rest of the symbols? This
happens because anonymous functions are represented as a closure
object. Here lies the crossroad of being functional in Emacs: symbols
overs lambdas.</p>
To express this notion, let me craft a different form for memoization.</p>
>>></a>
Wrapped Symbol</h3>
As contrary as this is, it is better to write the wrapped function as
another separate function using defun</code>.</p>
(</span>defvar fn</span>/</span>loaded</span>-</span>projects </span>(</span>list</span>))
</span>
</span>(</span>defun </span>fn</span>/</span>wrapped</span>-</span>load</span>-</span>project</span>-</span>file </span>()
</span>  </span>;; There is a bug here, can you figure it out?
</span>  </span>(</span>lexical</span>-let* </span>((</span>project</span>-</span>root </span>(</span>projectile</span>-</span>project</span>-</span>root</span>))
</span>      </span>(</span>loaded</span>-</span>project </span>(</span>member project</span>-</span>root fn</span>/</span>loaded</span>-</span>projects</span>)))
</span>    </span>(</span>if</span> loaded</span>-</span>project
</span>        </span>nil
</span>      </span>(</span>fn</span>/</span>load</span>-</span>project</span>-</span>file</span>)
</span>      </span>(</span>add</span>-to-</span>list 'fn</span>/</span>loaded</span>-</span>projects project</span>-</span>root</span>))))
</span>
</span>(</span>add</span>-</span>hook 'find</span>-</span>file</span>-</span>hook </span>#</span>'fn/wrapped-load-project-file</span>)
</span></code></pre>
So this version looks a little cleaner but exposes an extra internal
variable fn/loaded-projects</code> and an excess wrapper for
fn/load-project-file</code>. This is contrary in hiding state in the
functional style.</p>
Strangely, this is easier to debug and test. If you wanted to test
the wrapped function, you set fn/loaded-projects</code> to nil</code> or a
value and repeat the test; this is harder to do with a closure. If a
bug is in fn/wrapped-load-project-file</code>, you simply reevaluate the
function without having to clean or replace the hook value; with a
closure, you have a bugged and patched hook coexisting.</p>
I intentionally left a bug in fn/wrapped-load-project-file</code> to
demonstrate this. Patch then eval the new version. I don't have to
think about the hook management.</p>
(</span>defun </span>fn</span>/</span>wrapped</span>-</span>load</span>-</span>project</span>-</span>file </span>()
</span>  </span>(</span>when </span>(</span>projectile</span>-</span>project</span>-</span>p</span>) </span>;; `projectile-project-root' needs a project first
</span>    </span>(</span>lexical</span>-let* </span>((</span>project</span>-</span>root </span>(</span>projectile</span>-</span>project</span>-</span>root</span>))
</span>        </span>(</span>loaded</span>-</span>project </span>(</span>member project</span>-</span>root fn</span>/</span>loaded</span>-</span>projects</span>)))
</span>      </span>(</span>if</span> loaded</span>-</span>project
</span>          </span>nil
</span>        </span>(</span>fn</span>/</span>load</span>-</span>project</span>-</span>file</span>)
</span>        </span>(</span>add</span>-to-</span>list 'fn</span>/</span>loaded</span>-</span>projects project</span>-</span>root</span>)))))
</span>
</span>(</span>setq fn</span>/</span>loaded</span>-</span>projects </span>(</span>list</span>)) </span>;; If you want to reset its state
</span></code></pre>
This correctly loads fn/load-project-file</code> once, what does this tell us anyway?</p>
>>></a>
Symbols Over Function</h3>
Am I saying that when I want to memoize a function I need to create
an extra variable and wrapper and expose state? Not really. You
create a memoizing macro that used defun</code> or fset</code> to alleviate
this as with _emacs-memoize_</a>. In being simple, revealing state and
using symbols seems to be the way to go.</p>
I had struggled with this at first preferring closures, but it does
feel cleaner and simple specially in the context of Emacs. Since
everything is extensible, exposing and manipulating state, declaring
and advicing private functions, hiding things in Emacs seem to
counter the notion of customization.</p>
The notion of encapsulation is not disregarded but rather not
preferred. Since (almost) everything id found via describe-function</code>
or describe-variable</code>, being open is really the way to go. If you
find pain in writing more code for a repeating concept, if
abstracting the state and logic is worth it in simplicity and
extensibility.</p>
I can't speak for Scheme or Clojure; for me, Emacs has changed my
hand and mind in writing lisp.</p>
>></a>
Security</h2>
We now shiv a final feature that asks permission or trust in
loading the project files. Let us create a secure wrapper for
fn/load-project-file</code>:</p>
(</span>defvar fn</span>/</span>loaded</span>-</span>projects </span>(</span>list</span>)
</span>  </span>"</span>Projects that have been loaded by `fn/load-project-file'.</span>")
</span>
</span>(</span>defun </span>fn</span>/</span>safe</span>-</span>load</span>-</span>project</span>-</span>file </span>()
</span>  </span>;; Similar to `fn/wrapped-load-project-file' but ...
</span>  </span>(</span>when </span>(</span>projectile</span>-</span>project</span>-</span>p</span>)
</span>    </span>(</span>lexical</span>-let </span>((</span>project</span>-</span>root </span>(</span>projectile</span>-</span>project</span>-</span>root</span>))
</span>        </span>(</span>project</span>-</span>name </span>(</span>projectile</span>-</span>project</span>-</span>name</span>)))
</span>      </span>(</span>when </span>(</span>not </span>(</span>member project</span>-</span>root fn</span>/</span>loaded</span>-</span>projects</span>))
</span>        </span>(</span>if </span>(</span>not </span>(</span>fn</span>/</span>safe</span>-</span>project</span>-</span>p project</span>-</span>root</span>)) </span>;; ... asks permission first
</span>            </span>(</span>message </span>"</span>Project script for %s is not trusted.</span>"</span> project</span>-</span>name</span>)
</span>          </span>(</span>fn</span>/</span>load</span>-</span>project</span>-</span>file</span>)
</span>
</span>          </span>(</span>add</span>-to-</span>list 'fn</span>/</span>loaded</span>-</span>projects project</span>-</span>root</span>))))))
</span></code></pre>
What we are left with is implementing the symbol fn/safe-project-p</code>.
The question is indeed what scheme? A simple scheme is just to use
yes-or-no-p</code>:</p>
(</span>defun </span>fn</span>/</span>safe</span>-</span>project</span>-</span>p </span>(</span>project</span>-</span>root</span>)
</span>  </span>(</span>yes-or-no-p
</span>   </span>(</span>format </span>"</span>Do you trust the project at %s?</span>"</span> project</span>-</span>root</span>)))
</span></code></pre>
It is as simple as that. Or be more complex and check for last
modified time, expiration period, user ownership and what not. For me,
I simply ask permission as well but allow for a deeper setup if needed
that I am not going to show. I do want to show how to get the last
modified time:</p>
(</span>file</span>-</span>attribute</span>-</span>modification</span>-</span>time </span>;; file-attribute-* and its company
</span> </span>(</span>file</span>-</span>attributes
</span>  user</span>-</span>emacs</span>-</span>directory</span>))
</span></code></pre>
Do explore the other functions such as file-attribute-user-id</code> for
getting file attributes.</p>
>></a>
Persistence</h2>
If you use this snippet, you might get annoyed when Emacs opens and
needs permission when running the projects you allowed previously.
What I am talking about is persistence that is a tricky subject in
itself. Several tricks exists for this:</p>
Customize Mechanics</strong>
: But what if you don't use a custom-file</code>?</p>
p-cache</strong>
: Too rich for my blood</p>
Write the lisp object to file</strong>
: A bit low level</p>
A simpler builtin mechanism exist: savehist</code>. It primarily works for
lispy data and it does save it to a file in savehist-file</code>. I love
the ease of use just by adding the variable symbol to
savehist-additional-variables</code>. To demonstrate in modifying
fn/safe-project-p</code>:</p>
(</span>defun </span>fn</span>/</span>checked</span>-</span>projects </span>(</span>list</span>))
</span>
</span>(</span>defun </span>fn</span>/</span>safe</span>-</span>project</span>-</span>p </span>(</span>project</span>-</span>root</span>)
</span>  </span>(</span>lexical</span>-let </span>((</span>checked</span>-</span>project
</span>       </span>(</span>or
</span>        </span>(</span>cdr </span>(</span>assocproject</span>-</span>root fn</span>/</span>checked</span>-</span>projects</span>)) </span>;; Maybe 'trusted or 'untrusted
</span>        'unchecked</span>)))
</span>    </span>(</span>pcase checked</span>-</span>project
</span>      </span>(</span>'trusted </span>t</span>)
</span>      </span>(</span>'untrusted </span>nil</span>)
</span>      </span>(</span>'unchecked </span>;; If the project hasn't been trusted yet
</span>       </span>(</span>lexical</span>-let </span>((</span>trusted
</span>            </span>(</span>yes-or-no-p
</span>             </span>(</span>format </span>"</span>Do you trust the project at %s?</span>"</span> project</span>-</span>root</span>))))
</span>         </span>(</span>add</span>-to-</span>list
</span>          'fn</span>/</span>checked</span>-</span>projects
</span>          </span>(</span>cons project</span>-</span>root </span>(</span>if</span> trusted 'trusted 'untrusted</span>))))))))
</span>
</span>(</span>with-</span>eval</span>-</span>after</span>-</span>load 'savehist
</span>  </span>(</span>add</span>-to-</span>list 'savehist</span>-</span>additional</span>-</span>variables 'fn</span>/</span>checked</span>-</span>projects</span>))
</span></code></pre>
The function fn/checked-projects</code> is an association list of the
project root string and a symbol of 'trusted</code> or 'untrusted</code> and
what we want to persist. As I mentioned, all you have to do is add it
to savehist-additional-variables</code> and our preference is persisted
without any fuss. Nothing much to say but if you want more details
about this simple persistence SaveHist</a>.</p>
>></a>
Conclusion</h2>
After all that, the code is still simple to hack without needing to
rely on other packages. Getting work done is more important instead of
being worried by setup and security but still valid concerns.</p>


Princess Tutu - Breaking The Story
Unknown — Wed, 15 Feb 2017 00:00:00 +0000
>></a>
Princess Tutu</h2>
Princess Tutu disregards expectations to support the message</strong></em></p>
After dropping Revolutionary Girl Utena</a>, Casshern Sins</a> and Aku no
Hana</a>, I wanted to remind myself what a good anime is; this is where
Princess Tutu</a> comes in.</p>
By presentation, it is a magical girl show like any other. The plot is
about a duck, a crow, a knight and a prince that appears to be a fairy
tale setting. Behind this cute facade presents the familiar yet
gripping question of free will and fate.</p>
Aside from being a classical music lover, this anime appeals to me not
just on the subversive story but also the use of symphonies and play
with its story. Unlike Your Lie In April</a> where the music is not
strongly tied to the story (since you may change music playing with
dancing or sports or even swords), here it uses different operas and
ballets that tie into the episode theme, mood and story. A more
thorough musical analysis is out there but after hearing the sound
track and listening to a wide range of classics, the music is a true
variation on the existing piece that shows effort in the musical
department. I can't say more but I know it is impressive that it just
didn't edit the music straight from the public domain. I will say it
is a pleasure to watch and hear specially the opening theme, Morning
Grace, that sets the tone quite well.</p>
Anyway, what I want to focus on is how framing and using the story as
a fairy tale supports the theme of fate and free will and not simply as
an aesthetic choice.</p>
  </p>
SPOILERS AHEAD</strong></em></p>
>></a>
The Story</h2>
Guessing the story from the clues, it is a love story between the duck
turned girl and the prince against the jealous crow princess that
takes its cue from The Ugly Duckling</a> and Swan Lake</a>. You don't need to
be familiar with them to understand the archetype or structure of a
romantic story like the enchanting The Little Mermaid</a> or the
subversive Shrek</a> or the ever undying Romeo and Juliet</a>.</p>
  </p>
Let's first understand the overall story. This anime has twenty-six
episodes or two cours that represents the two acts, short of another
act to make it a three act play. The first act introduces us into the
story episodic proper while the other begins its heavy deviation.</p>
>>></a>
The First Act</h3>
In a nutshell, this act introduces us to the collection mission of
the duck, Ahiru, turned magical ballerina girl, Princess Tutu, into
gathering the lost pieces of the Prince's heart, Mytho, who cannot
feel any emotion. Unlike other collection structures where gathering
a key item has no bearing in a story or game such as the classic The
Legend of Zelda</a>, retrieving a piece of the heart unlocks a new
emotion from the prince such as sorrow, love, fear and so on which
perturbes the peaceful relationships of the cast. The first half of
the anime does appear episodic that seems to culminate in the prince
recovering and unlocking his heart and his love for the swan
princess.</p>
  </p>
The episodic structure is used to great effect here as it continually
hints at the overarching story of the anime itself: The Prince and
The Raven. It tells the story of a prince shattering his heart to
seal a dark Raven. Since our hapless protagonist is merely a duck,
she does not know that her mission ultimately brings the Raven back
into existence but the raven is not introduced within the fairy tale
first act. Rather, he is saved for the second act. After the first
act, a few more pieces of the heart remains, the sword of the prince
is broken and a Raven to slay.</p>
  </p>
In short, the first half establishes the characters as well as subtly
and subvertly fulfilling the story of a fairy tale to break it apart
later with the tragic foreshadowing.</p>
>>></a>
The Second Act</h3>
The story of the second act is where the story and more importantly
the characters moves it beyond a magical girl anime to a great one.
The story drops the episodic structure as it moves to a more
character-driven story. It begins with the Prince turning into the
love hungry raven by a devious ploy of the Raven in enchanting or
cursing the fragment of love with his blood making friend turn foe.
On the other hand, the black swan is presented as a mere victim or
puppet to this malevolent being and is show greater depth along with
Fakir within the second act. The pretense of a fairy tale is replaced
by messing with every character, including the side ones.</p>
  </p>
The story of the second act is fighting the fate of the book itself,
that is cleverly setup in the first act. Rather, the greater fight
is with the narrator, Drosselmeyer, who has orchestrated everything
and is leading everyone to a tragic ending where the prince has to
shatter his heart again and reset the story instead of concluding it
with a happy ending. Defeating both the story villain and the story
teller is a story in itself.</p>
  </p>
The overall theme of the anime is defying fate that we link with
the characters.</p>
>></a>
The Actors</h2>
At first glance of this anime, I thought it is just another
magical girl show. What let me push through despite the quacky
animation, cliched use of the Nutcracker Suite is how strange the
characters are setup in the beginning:</p>
  </p>
The Prince without emotions</strong>
: Without it, it is just cliched amnesia. What is weird is that he
is being saved and not the other way around. He is a bland
character by default since he has no emotion that is a play on
the main hero cliche itself.</p>
The considerate Rival</strong>
: Antagonist are almost evil by default; Rue is a better rival as
she merely loves the Prince and not a jerk to Ahiru that rich
girls or rivals are commonly portrayed.</p>
The overprotective Knight</strong>
: Fakir is not a romantic interest yet shows strong protective
feelings for the Prince. This is a play on the sidekick
where the trait should have gone to the love rival to make it
cliche. In a sentence, he is making sure his hero-complex
brother doesn't kill himself by constraining him, brotherly love
indeed.</p>
The story teller</strong>
: Why is there a separate narrator character? Rather, why is the
narrator making plays instead of simply narrating?</p>
I like all the main characters as each do not offend me but I have a
soft spot for the loveable Ahiru, the complicated Fakir and the wise
puppet Edel. The characters are already nuanced in the beginning yet
appearing stereotypical although I understand that they might turn
people off initially if the subversion is not seen and find bland and
generic characters instead.</p>
Instead of describing each character in detail, the ending itself
encapsulate everything: the Rival gets the Prince.</strong></p>
  </p>
The ending itself defies the fate of the book. The Rival has true
feelings of love for the Prince yet the Princess just admires him. At
the end, Fakir becomes a writer who counters the influence of
Drosselmeyer yet Ahiru is the same old duck who does not hook up at
the end. The mold is broken for every character and stereotype once
the end is reached although the ending is slightly rushed in certain
aspects.</p>
The anime presents a good job with the theme and message and it should
be obvious why the anime is correct in framing the story within a
story.</p>
>></a>
The Active Story Teller</h2>
  </p>
At the end of the anime, the Story Teller breaks the fourth wall by
acknowledging that the story is over and instead goes to another more
interesting one. This meta awareness adds to the theme of defying what
is written for the characters and the story itself.</p>
  </p>
Interestingly, the town itself is part of the story since it is walled
off and nobody gets out including Princess Tutu, adding to the
disempowering fate their tragedy holds. If this meta layer is removed,
Drosselmeyer existence and interference is out of place; rather, he
becomes a Joker or Deus Ex character providing random story
progression with little reason or rhyme.</p>
If the book was the sole foreshadowing element, defying fate is a
consequence of the characters will and not of the story itself. For
example, stories commonly have a prophecy, a legend or internal story
that foreshadows the journey or adventure that the characters
ultimately ignore and defy. By cleverly using a meta-story, it also
defies how the story should even playout or be. For example, even
after changing the fate, the characters themselves fall into the
archetype of their fate such as the heroine and hero hooking up. Here,
the story is a character itself who breaks the mold along with the
characters it plays on.</p>
  </p>
By making the narrator an active participant in the story, he is
presented with the same question of fate and obedience. Should he make
a typical fairy tale story or an captivating tragedy? Since he chooses
to make it exciting, he effectively defies his fate and of the story
by exercising his choice. Talking about the meta story gets messy but
this is why the use of the narrator and fourth wall cleverly adds
layer and depth to this story structure. It supports the question of
free will, fate and obedience in such a stronger way.</p>
  </p>
This breaking of the story also creates a story in itself that is
ironic for what it is trying to do. Still, it shows that the story is
secondary to the characters will and desire, not of the audience or
narrator. If this is to be taken as the meta message, then write the
story from the characters and not the characters from the story. This
is Drosselmeyer's mistake when he focused on the tragedy and not how
the characters change or alter the story.</p>
Indeed, stories are about the actors.</p>
>></a>
Conclusion</h2>
To wrap up, using the story within a story mechanic for this anime
enriches the question of fate and not merely aesthetic. I deeply
appreciate this anime for not being a simple magical girl series. To
understand why I enjoy this, here are the things I find cliched about
magical girl series:</p>
Long and extraneous transformation</strong>
: Utena and Sailor Moon in particular where the transformation
seems to pad out the run time after seeing it once. At least have
the decency of being short as not to take time.</p>
Shoddy outfits</strong>
: The usual magical girl outfit is to sell merchandise and appeal
to demographics. Although Princess Tutu is effectively wearing a
skirt, it is presented as wholesome and innocent.</p>
Too colorful</strong>
: The color is too bright and colorful in a way to simply
differentiate them as with Power Rangers</a> or Pretty
Cure</a>. Having a toned down color range for Princess
Tutu is such a breath of fresh air.</p>
Heavily Stereotyped</strong>
: Characters are heavily leaning on their personal quirks instead
of being more balanced out of plot contrivances</p>
Cheesy Romance</strong>
: Although I don't lean toward romantic stories, I just dislike it
if the romance is more about the feeling of love instead of the
weight and value of their relationship.</p>
I guess I am jaded with girly shows not having more depth or being
more cute or marketable but this anime is my exception and
recommendation in  many ways despite the flaws. (I have seen Puella
Magi Madoka Magica</a> which some depth but I prefer the overall meaning
and package of this anime.)</p>


Notifying prodigy With Fringe alert
Unknown — Wed, 08 Feb 2017 00:00:00 +0000
>></a>
Problem</h2>
The Elisp library prodigy.el</a> helps you manage your external processes;
in particular, it allows you to listen to the process log which allows
you to execute commands on certain states. Although we have
start-process</code> and set-process-filter</code>, this library wraps those
lower level constructs into a more manageable object with hooks and
properties.</p>
One common feature when using processes is to be notified when it is
ready or something happens. For example, I use offlineimap</a> with gnus</code>
to get my mail where I want to be notified when there is one or jekyll</a>
to make this blog where I want to told when the blog is built
successfully or not. I could use the simple message</code> construct but to
be more flexible I use the library alert</a>.el.</p>
That notifying library features more than just using message</code>, it can
use libnotify</code>, growl</code> and more importantly the fringe</code>. A fringe</code>
alert uses set-face-background</code> to change the color of the buffer
sides and removes it on action. You can try this one liner to
understand better:</p>
(</span>set</span>-</span>face</span>-</span>background 'fringe </span>"</span>red</span>")
</span></code></pre>
So when there is a process event, I want a fringe alert. I have no
snippet to give here only an outline on how I went about it. For this
post, I will use jekyll</code> as an example.</p>
>></a>
Using prodigy</code></h2>
To define a jekyll</code> process with prodigy</code>, here is how you will go
about it:</p>
(</span>prodigy</span>-</span>define</span>-</span>service
</span>  :name </span>"</span>Gnus-Offlineimap</span>"      </span>;; Service name or id
</span>  :cwd </span>"</span>~/Fakespace/fnlog.io</span>"  </span>;; Working directory when the service is called
</span>
</span>  :command </span>"</span>jekyll</span>"             </span>;; The jekyll command
</span>  :args </span>(</span>list                      </span>;; Command arguments
</span>         </span>"</span>serve</span>"
</span>         </span>"</span>--drafts</span>"
</span>         </span>"</span>--ports</span>" (</span>number</span>-to-</span>string </span>34000</span>))
</span>  :port </span>34000                   </span>;; Optional port property
</span>  :kill</span>-</span>signal 'sigkill         </span>;; Kill signal to send when closing
</span>
</span>  :on</span>-</span>output </span>#</span>'ignore           </span>;; The output listener we are interested in
</span>  </span>)
</span></code></pre>
This is more or less how I setup my blog process service but I use the
tag system to make it a more reusable. To emphasize convenience, I
would use this library instead of handling it with start-process</code>,
with-temp-buffer</code> and default-directory</code>. To get a better picture,
you can see the new service with a special tabulated and process
buffer.</p>
  </p>
What we are interested in with the logs are two lines that contain
the string/text:</p>

Server running</code> to indicate the blog is ready</code></li>
...done in</code> to indicate the blog is built</code> and should be refreshed</li>
</ul>
The :on-output</code> allows us to tap into this which takes a function
with a plist argument that has :service</code> and :output</code> properties. In
our context of listenting to the following lines:</p>
(</span>defun </span>fn</span>/</span>prodigy</span>-</span>jekyll</span>-</span>state</span>-</span>change</span>-</span>listener </span>(</span>&rest args</span>) </span>;;defun* can also work
</span>  </span>(</span>lexical</span>-let* </span>((</span>output </span>(</span>plist</span>-</span>get args :output</span>)) </span>;; Get the arguments
</span>      </span>(</span>service </span>(</span>plist</span>-</span>get args :service</span>)))
</span>    </span>(</span>pcase output
</span>      </span>((</span>pred </span>(</span>string</span>-</span>match</span>-</span>p </span>"</span>Server running</span>"))
</span>       </span>(</span>prodigy</span>-</span>set</span>-</span>status service 'ready</span>) </span>;; Manually set the status
</span>       </span>;; Trigger ready alert
</span>       </span>)
</span>      </span>((</span>pred </span>(</span>string</span>-</span>match</span>-</span>p </span>"</span>Error:</span>"))
</span>       </span>;; Trigger error alert
</span>       </span>)
</span>      </span>((</span>pred </span>(</span>string</span>-</span>match</span>-</span>p </span>"</span>...done</span>"))
</span>       </span>;; Trigger done built alert
</span>       </span>)
</span>      </span>(</span>_ </span>;; Do nothing otherwise
</span>       </span>nil</span>))))
</span>
</span>
</span>;; Redefining the service
</span>(</span>prodigy</span>-</span>define</span>-</span>service
</span>  </span>;; Previous definition
</span>  :on</span>-</span>output 'fn</span>/</span>prodigy</span>-</span>jekyll</span>-</span>state</span>-</span>change</span>-</span>listener
</span>  </span>)
</span></code></pre>
If you don't have string-match-p</code> since you are using an earlier
version, it can be shivved like so:</p>
(</span>unless </span>(</span>fboundp 'string</span>-</span>match</span>-</span>p</span>)
</span>  </span>(</span>defsubst string</span>-</span>match</span>-</span>p </span>(</span>regexp string &optional start</span>)
</span>    </span>"</span>\
</span>Same as `string-match' except this function does not change the match data.
</span>Taken from `subr-x'</span>"
</span>    </span>(</span>let </span>((</span>inhibit</span>-</span>changing</span>-</span>match</span>-</span>data </span>t</span>))
</span>      </span>(</span>string</span>-</span>match regexp string start</span>))))
</span></code></pre>
Aside from using pcase</code>, it is pretty much just listening to the
output. As an aside, listening to the output is pretty much determined
by the process itself; so some awkward log state management might be
present for other services. For example, there might be a start line,
then followed by either an error or process line, finally an end line;
in that scenario, one has to keep track if there is an error or not to
determine at the end line if there is an error instead of notifying at
each one. Logging and state machines, some nuances when dealing with
listening to logs.</p>
We move on to using alert</code>.</p>
>></a>
Using alert</code></h2>
Using alert</code> is pretty straightforward, the first argument is the
message and the rest are some options. I encourage you to check it out
but for our case, we will be focused on using the fringe type:</p>
(</span>alert </span>"</span>Hello Alert</span>")         </span>;; Default alert with message
</span>
</span>(</span>alert </span>"</span>Hello Fringe Alert</span>"   </span>;; Using a fringe alert
</span>       :severity 'trivial
</span>       :style 'fringe</span>)
</span></code></pre>
If you use this snippet, you will get a purple fringe alert which
comes from alert-severity-colors</code>.</p>
((</span>urgent . </span>"</span>red</span>")
</span> </span>(</span>high . </span>"</span>orange</span>")
</span> </span>(</span>moderate . </span>"</span>yellow</span>")
</span> </span>(</span>normal . </span>"</span>green</span>")
</span> </span>(</span>low . </span>"</span>blue</span>")
</span> </span>(</span>trivial . </span>"</span>purple</span>"))
</span></code></pre>
Weirdly, the severity handles the color. Either way, there isn't many
colors to work with, so if you want to add one like Rebecca Purple</code>
or Lavender</code>. You have to add to do the following:</p>
(</span>add</span>-to-</span>list 'alert</span>-</span>log</span>-</span>severity</span>-</span>functions
</span>             </span>(</span>cons 'lavender </span>#</span>'alert--log-trace</span>))
</span>
</span>(</span>add</span>-to-</span>list 'alert</span>-</span>severity</span>-</span>colors
</span>             </span>(</span>cons 'lavender </span>"</span>#b378d3</span>"))
</span>
</span>
</span>(</span>alert </span>"</span>My lavender alert</span>"
</span>       :style 'fringe
</span>       :severity 'lavender</span>)
</span></code></pre>
This is not so bad but when you have several services that uses fringe
alerts or have to create one, you might want a function that does
this. Here is my version of that function after looking at the source:</p>
(</span>require 'cl</span>-</span>lib</span>)
</span>
</span>(</span>defun</span>*</span> fn</span>/</span>alert</span>-</span>color </span>(</span>message &rest args &key color &allow</span>-</span>other</span>-</span>keys</span>)
</span>  </span>"</span>A custom alert that focuses on defining a fringe with COLOR key
</span>with a hex value.</span>"
</span>  </span>;; Step 01: Create hex color symbol
</span>  </span>(</span>lexical</span>-let* </span>((</span>hex</span>-</span>color </span>(</span>replace</span>-</span>regexp</span>-</span>in</span>-</span>string </span>"</span>#</span>" ""</span> color</span>))
</span>      </span>(</span>hex</span>-</span>symbol-name </span>;; Prepend alert-color-- to the hex code
</span>       </span>(</span>format </span>"</span>alert-color--%s</span>"</span> hex</span>-</span>color</span>))
</span>      </span>(</span>hex</span>-</span>symbol      </span>;; Check if the symbol exists or default
</span>       </span>(</span>or </span>(</span>intern</span>-</span>soft hex</span>-</span>symbol-name</span>)
</span>          </span>(</span>intern hex</span>-</span>symbol-name</span>))))
</span>
</span>    </span>;; Step 02: Add the symbol pair to the lists
</span>    </span>(</span>unless </span>(</span>cdr </span>(</span>assoc hex</span>-</span>symbol alert</span>-</span>log</span>-</span>severity</span>-</span>functions</span>))
</span>      </span>(</span>add</span>-to-</span>list 'alert</span>-</span>log</span>-</span>severity</span>-</span>functions
</span>                   </span>(</span>cons hex</span>-</span>symbol </span>#</span>'alert--log-trace</span>)))
</span>
</span>    </span>(</span>unless </span>(</span>cdr </span>(</span>assoc hex</span>-</span>symbol alert</span>-</span>severity</span>-</span>colors</span>))
</span>      </span>(</span>add</span>-to-</span>list 'alert</span>-</span>severity</span>-</span>colors </span>(</span>cons hex</span>-</span>symbol color</span>)))
</span>
</span>    </span>;; Step 03: Handle an extra :color property that
</span>    </span>;;          defaults :style and :severity
</span>    </span>(</span>lexical</span>-let </span>((</span>color</span>-</span>properties
</span>         </span>(</span>list
</span>          :style 'fringe
</span>          :severity hex</span>-</span>symbol</span>))
</span>        </span>(</span>colorless</span>-</span>properties
</span>         </span>(</span>cl</span>-</span>reduce </span>;; Just removes :color from the property list
</span>          </span>(</span>lambda </span>(</span>val props</span>)
</span>            </span>(</span>if </span>(</span>equal</span> :color val</span>)
</span>                </span>(</span>cdr props</span>)
</span>              </span>(</span>cons val props</span>)))
</span>          args
</span>          :</span>from-</span>end </span>t
</span>          :initial</span>-</span>value </span>(</span>list</span>))))
</span>      </span>(</span>apply </span>#</span>'alert
</span>         </span>(</span>append
</span>          </span>(</span>list message</span>)
</span>          color</span>-</span>properties
</span>          colorless</span>-</span>properties</span>)))))
</span></code></pre>
Although a little long, this encapsulates the symbol creation, adding
to the lists and handling a :color</code> property. It does not handle more
fine grained severity options and it might not property decorate
around alert</code> but it does the job. You can see it in action here:</p>
(</span>fn</span>/</span>alert</span>-</span>color </span>"</span>Here is my colored alert</span>"
</span>                :color </span>"</span>#123456</span>")
</span></code></pre>
In the interest of being canonical with alert</code>, I would have defined
a custom style but I could not get a custom property or argument like
:color</code> to be used; I could have reused :severity</code> but it doesn't
fit the context. So I had to create a decorator for a custom color
property despite the ease of doing the following:</p>
;; Copied from alert-fringe-
</span>(</span>defun </span>fn</span>/</span>alert</span>-</span>flash</span>-</span>notify </span>(</span>info</span>)
</span>  </span>(</span>set</span>-</span>face</span>-</span>background 'fringe </span>(</span>plist</span>-</span>get info :color</span>)))
</span>
</span>(</span>defun </span>fn</span>/</span>alert</span>-</span>flash</span>-</span>restore </span>(</span>info</span>)
</span>  </span>(</span>copy</span>-</span>face 'alert</span>-</span>saved</span>-</span>fringe</span>-</span>face 'fringe</span>))
</span>
</span>(</span>alert</span>-</span>define</span>-</span>style 'fn</span>/</span>flash
</span>                    :title </span>"</span>Change the fringe color</span>"
</span>                    :notifier </span>#</span>'fn/alert-flash-notify
</span>                    :remover </span>#</span>'fn/alert-flash-restore</span>)
</span>
</span>
</span>;; If it did work...
</span>(</span>alert </span>"</span>Here is a simpler alert.</span>"
</span>       :style 'fn</span>/</span>flash
</span>       :color </span>"</span>#123456</span>")
</span></code></pre>
So now we can get a colored fringe alert without handling the symbols
and lists.</p>
>></a>
Combining alert</code> and prodigy</code></h2>
Now that we have the two we can easily merge them in the following
snippet:</p>
(</span>defun </span>fn</span>/</span>prodigy</span>-</span>jekyll</span>-</span>state</span>-</span>change</span>-</span>listener </span>(</span>&rest args</span>) </span>;;defun* can also work
</span>  </span>(</span>lexical</span>-let* </span>((</span>output </span>(</span>plist</span>-</span>get args :output</span>)) </span>;; Get the arguments
</span>      </span>(</span>service </span>(</span>plist</span>-</span>get args :service</span>)))
</span>    </span>(</span>pcase output
</span>      </span>((</span>pred </span>(</span>string</span>-</span>match</span>-</span>p </span>"</span>Server running</span>"))
</span>       </span>(</span>prodigy</span>-</span>set</span>-</span>status service 'ready</span>) </span>;; Manually set the status
</span>       </span>(</span>fn</span>/</span>alert</span>-</span>color </span>"</span>Blog is up and running</span>"
</span>                       :color </span>"</span>#01dddd</span>"))
</span>      </span>((</span>pred </span>(</span>string</span>-</span>match</span>-</span>p </span>"</span>Error:</span>"))
</span>       </span>(</span>fn</span>/</span>alert</span>-</span>color </span>"</span>Blog has a build error. Check out the logs.</span>"
</span>                       :color </span>"</span>#fd8a5e</span>"))
</span>      </span>((</span>pred </span>(</span>string</span>-</span>match</span>-</span>p </span>"</span>...done</span>"))
</span>       </span>(</span>fn</span>/</span>alert</span>-</span>color </span>"</span>Blog has successfully been built.</span>"
</span>                       :color </span>"</span>#e0e300</span>"))
</span>      </span>(</span>_ </span>;; Do nothing otherwise
</span>       </span>nil</span>))))
</span></code></pre>
With that, everything should be in order; however, what I did instead
is created a hook so that other actions aside might be taken based on
its state. As an example, I check the blog's output using w3m</code> and
when the blog is built, I want to reload all the web buffers
pertaining to my blog. This snippet does it:</p>
(</span>defun </span>fn</span>/</span>prodigy</span>-</span>jekyll</span>-</span>state</span>-</span>w3m</span>-</span>reload </span>(</span>service state</span>)
</span>  </span>(</span>lexical</span>-let </span>((</span>service</span>-</span>name </span>(</span>plist</span>-</span>get service :name</span>)))
</span>    </span>(</span>pcase state </span>;; My hook takes a service and state argument
</span>      </span>(</span>'built
</span>       </span>(</span>save</span>-</span>window</span>-</span>excursion
</span>         </span>(</span>mapc
</span>          </span>(</span>lambda </span>(</span>w3m</span>-</span>buffer</span>)
</span>            </span>(</span>ignore-errors
</span>              </span>(</span>lexical</span>-let* </span>((</span>url </span>(</span>with-</span>current</span>-</span>buffer w3m</span>-</span>buffer w3m</span>-</span>current</span>-</span>url</span>))
</span>                  </span>(</span>parts </span>(</span>w3m</span>-</span>parse</span>-</span>http</span>-</span>url url</span>))
</span>                  </span>(</span>host </span>(</span>elt parts </span>1</span>))
</span>                  </span>(</span>port </span>(</span>elt parts </span>2</span>)))
</span>                </span>(</span>when </span>(</span>and </span>(</span>=</span> port </span>34000</span>)
</span>                         </span>(</span>string</span>=</span> host </span>"</span>127.0.0.1</span>"))
</span>                  </span>(</span>with-</span>current</span>-</span>buffer w3m</span>-</span>buffer
</span>                    </span>(</span>w3m</span>-</span>reload</span>-</span>this</span>-</span>page</span>))))))
</span>          </span>(</span>w3m</span>-</span>list</span>-</span>buffers</span>))))
</span>      </span>(</span>_ </span>nil</span>))))
</span>
</span>;; The hook I created, modified for this post
</span>(</span>add</span>-</span>hook 'fn</span>/</span>prodigy</span>-</span>jekyll</span>-</span>state</span>-</span>change</span>-</span>hook
</span>          </span>#</span>'fn/prodigy-jekyll-state-w3m-reload</span>)
</span></code></pre>
That's the thing about :on-output</code>, you can't add to it like a hook
so you have to make one. Using the output as a hook, I have an alert
and auto-reload mechanism.</p>
Lastly, if you forgot the meaning of the color, it would be nice if
there was also a message to back it up. By default, using this style
of the alert produces no minibuffer message to appear which is
appropriate. However, I would like to be reminded what it means with
this snippet:</p>
(</span>defun </span>fn</span>/</span>alert</span>-</span>fringe</span>-</span>notify</span>-</span>message </span>(</span>info</span>)
</span>  </span>"</span>Log `fringe' style with `message'.
</span>     This is to support `fn/alert-color' if the color flash needs
</span>     a reminder.</span>"
</span>  </span>(</span>message </span>(</span>plist</span>-</span>get info :message</span>)))
</span>
</span>(</span>advice</span>-</span>add 'alert</span>-</span>fringe</span>-</span>notify :after </span>#</span>'fn/alert-fringe-notify-message</span>)
</span></code></pre>
Or you could add that to fn/alert-color</code> if advicing is overkill.</p>
>></a>
Conclusion</h2>
Now that we have candied alerts for our services, what else can be
done with these libraries?</p>

Service dependencies</li>
Log alerts to file</li>
</ul>
If more services such as react-native</code> come and go, we can now easily
creates alerts for it.</p>


Song Of The Sea - Justifying The Song
Unknown — Wed, 08 Feb 2017 00:00:00 +0000
>></a>
Song of the Sea</h2>
Song of the Sea (2014)</a> is one of the best hand drawn film I have ever
seen from Cartoon Saloon</a>, rivaling the likes the works of Studio
Ghibli</a>. Not surprising since the other work I enjoy from them is The
Secret of Kells</a> which is another excellent work to behold.</p>
What I want to talk about in this short article is about the song. In
particular, why the song seem to have changed the scope of the story
which some people find some trouble with.</p>
SPOILER AHEAD</strong></em></p>
>></a>
Saoirse's Song</h2>
The question is why did Saoirse's song save not just herself but also
the spirits in the story?</p>
  </p>
All that is expected of the song is that it saves her so saving the
surrounding spirits around her seems to come out of nowhere. On my
first viewing, I did not see it coming so I see why people think
similarly. Did the film foreshadow her song would save the spirits?</p>
The most practical answer I can give is that something needed to
happen within the four minutes she was singing. If not, it would be a
scene of her spinning and singing in the air which might not be the
most visually meaningful. However, I refuse to believe the song is
just a filler music video clip, so let us look deeper in the film.</p>
>></a>
From The Script</h2>
Reading into the script, there are several lines indicating that it
would. In the early scenes, Ben shows his father the picture on the
wall and says:</p>
BEN: She's singing her song so that she can send all the fairies home
</span>across the sea.
</span></code></pre>
Another reminder is right before the musical Deenashee is petrified,
one of them says:</p>
DEENASHEE: Human child, find the selkie's true coat, then she can sing
</span>the song and save us all.
</span></code></pre>
The Great Senachai says something that is more direct:</p>
GREAT SENACHIA: She'll be turned to stone soon, no doubt. You see,
</span>without her coat, the selkie has no voice. And without her song, she'll
</span>not last the night. And we'll all share in her fate.
</span></code></pre>
The last sentence is very strange yet indicative of the stake involved
in saving Saoirse. So is the saving song foreshadowed enough? I don't
think so since these are just a couple of lines that one could easily
miss. A good foreshadowing would be a scene involving Bronach or other
selkies singing and having the same effect, so at best it is only
hinted at.</p>
Instead, did the song need to save the spirits? Would the film change
if Saoirse was only healed and went back home? So let's explore the
meaning of the song.</p>
>></a>
Bronach's Song</h2>
To be fair, it is known that Saoirse and her voice is magical.
Saoirse's stifiled singing through the shell was enough to break
Macha's magic; what more if is she used her true voice? (This is
speculation as power levels aren't really established anyway.) She was
only able to sing after Ben taught her their mother's song.</p>
  </p>
Which is weird why Ben hasn't taught her the song before after six
years of being siblings but that further cements how important the
song is. Ben is the only one who has their mother's legacy: not the
coat, not the shell, but the tales and the song.</p>
There is a fascinating line that their mother mentions before leaving:</p>
BRONACH: My son. Remember me, in your stories and in your songs. Know
</span>that I will always love you. Always.
</span></code></pre>
This is a clever narrative use of oral tradition</a>: passing on the song
is passing on the memory and love. Ben teaching her the song is not
only symbolic of his forgiveness but also passing the love and legacy
of thier mother.</p>
So the song is symbolically and thematically about love but how does
this factor in saving the spirits? At the very least though, the song
itself has meaning to exist.</p>
>></a>
The Selkie's Duty</h2>
It is interesting to note why their mother wasn't able to sing and
free the spirits from Macha's grasp earlier.</p>
  </p>
Clearly, Bronach can sing and Mac Lir is just outside. So why hasn't
she? Rather, it might not be that she won't but she can't. I think
Bronach might have no longer any magic or is waning. It might be
because she chooses to love Conor similar to The Little Mermaid</a>. By
having little or no magic, giving birth to Saoirse might have taken
all that Bronach had remaining, which makes more sense in the context
of Ben being human and not a magical being.</p>
Whatever the circumstance of the spirit or magical world, she cannot
sing to her duty as a selkie, which might be considered taboo or
illegal and adds a layer to her being a woman. Hence, the duty of the
mother falls onto the daughter.</p>
What we need from the song is simply to call their mother and give
Saoirse the pivotal choice of staying or leaving with Ben. By singing
the song and sending the spirits home which not only fulfills
Bronach's neglected duty but makes her a selkie. This in turn calls
their mother's spirit and intends to take Saoirse as a selkie. From
this, we might infer that Bronach can't seem to cross to the human
world without a selkie's song guiding her to and from. Even without
thinking about the backstory, Bronach says this:</p>
BRONACH: All of my kind must leave tonight. But, Saoirse, you are part
</span>human. If I take her coat, our worlds will be untangled.
</span></code></pre>
So sending the spirits home is pivotal to give Saoirse the choice of
not only staying but forgiving Ben. If Saoirse left to be with her
mother, it would mean Ben hasn't been good enough as a brother for her
to stay. So she forgives Ben by staying instead of leaving the person
who hurts or disregards her. Forgiving someone isn't simply a
conversation.</p>
Interestingly, without the choice, forgiving Ben would not mean
anything without any narrative or personal impact. Saoirse, Ben and
Conor would simply be back home without much ado about nothing. The
film is quite clever in setting the stakes. (Who wants to lose their
beloved little sister?)</p>
>></a>
Other Subtleties</h2>
There are more subtleties that help in reading the film as a whole.</p>
>>></a>
Doubles</h3>
Notice how the human and spirit world are similar or paralled in this
film. Mac Lear and Conor, Granny and Macha, Dan and The Great
Senechai, Saoirse and Bronach are human and spirit world equivalents
or doubles. Appearing to look similar and even the voice actors are
the same, this use of doubles or parallels heighten the only
character without a double: Ben. The hero's journey, growth and
forgiveness of Ben is further magnified.</p>
>>></a>
Circle</h3>
Notice the use of circles is very strong. More importantly, the use
of it emphasizes how everything is connected and how the film is
centered around a family.</p>
  </p>
I believe the image above is the most helpful in reading the film.
Everything tied in one circle of tales.</p>
>>></a>
The Human And The Spirit World</h3>
The two previous fact heighten this fact: there is no distinction
between the two worlds. In most films, there is a portal or gateway
to the other world but in this film it blurs the line as the same.
The world of the magical is no more different from the world of the
real; this subtlely works quite well in showing how the problem and
resolution of the family and the world is almost the same.</p>
>></a>
Conclusion</h2>
I could have just said Saoirse needed to be a selki but we needed to
understand the meaning or weight of the song and what it allowed. In
the end, the song has such a sublime serenity and catharsis that one
cannot deny. For a film to have this depth and subtlety of choice, it
makes me appreciate it more. That is why I believe this is one of the
best films to watch that is both accessible yet sublime.</p>


Browsing w3m Anonymously With tor
Unknown — Wed, 01 Feb 2017 00:00:00 +0000
UPDATE: 2017-01-24: Add https proxy to the command arguments.</strong>
Also, updating style and grammar.</strong></p>
>></a>
Problem</h2>
I'm having fun with w3m</a> on Emacs but by using this I discarded several
features I have neglected with a GUI: private browsing. This is useful
if prying eyes were desperate enough to check the terminal browser for
browsing history. Gluing it to ignore several sites is simple enough
that might warrant a post itself.</p>
Even with firefox</a> or chrome</a>, private browsing</a> may not be enough as
servers can track your IP and infer which sites you went to. Not truly
private but good enough against nosy users. Enter tor</a>, the anonymizing
web proxy, that routes HTTP requests through several proxies/relays
and adding layers of encryption between them to mask where the request
comes from. Despite slowing down browsing speed, it will keep those
pesky trackers and advertisers from scarily guessing what books or
movies you secretly want.</p>
Sadly, tor</code> is a SOCKS proxy</a> and w3m</code> uses only HTTP proxies</a>. This
means it isn't as quick and straightforward joining the two, thus we
need a middleman. Enter polipo</a>, a caching HTTP proxy that is
lightweight for the experiment. To finalize, w3m</code> will use polipo</code>
as its HTTP proxy which in turn uses tor</code> as its SOCKS proxy to send
the request.</p>
With that idea, here is the snippet</a> that glues them. The first time
you run this, tor</code> might take some time setting up thus be patient.
As hackers, we want elaboration.</p>
>></a>
Configuration Generation</h2>
Once you installed and tried both tor</code> and polipo</code>, we want Emacs to
start these proxies for us when we start w3m</code>. They can be configured
as external services or daemons; instead, we want portable setups that
minimize external dependencies and conflicts.</p>
To start an external process, we use start-process</code>. Practically, we
want it to call both proxies and their respective command line
options; however, generating their respective configuration files via
Elisp is more interesting. A minor bonus, the configuration can be
tested and examined externally.</p>
Both configurations can be abstracted as a list of key-value pairs or
cons with different line formatting. A basic formatter for this with
tor</code>:</p>
(with-temp-file config-file ;; Config file path
</span>  (insert
</span>   (string-join
</span>    (mapcar
</span>     (lambda (pair)
</span>       (pcase-let ((`(,key . ,value) pair))
</span>         (format
</span>          "%s %s" ;; Line formatting
</span>          key ;; Key formatting
</span>          (typecase value ;; Value formatting
</span>            (symbolp (symbol-name value))
</span>            (numberp (number-to-string value))
</span>            (stringp value)))))
</span>     pairs ;; The cons list
</span>     )
</span>    "\n")))
</span></code></pre>
Aside from using the helpful with-temp-file</code> macro and destructring
pcase-let</code>, the only nuance here is the value formatting via
typecase</code>. How you do the formatting affects what values you place.
For example with polipo</code>, our list of cons</code> with some stylistic
backquoting looks something like this:</p>
`(("proxyAddress" . "0.0.0.0")
</span>  ("allowedClients" . "127.0.0.1")
</span>  ("diskCacheRoot" . ,fn/w3m-polipo-cache-dir)
</span>  ("proxyPort" . ,fn/w3m-polipo-port)
</span>  ("cacheIsShared" . false)
</span>  ("socksParentProxy" .
</span>   ,(format "%s:%s" "localhost" (number-to-string fn/w3m-tor-port)))
</span>  ("socksProxyType" . socks5))
</span></code></pre>
Although the key types are the same, notice there are different value
types. Strings are quoted, numbers and symbols are stringified; if we
used just list of line strings, we'd have to do the formatting
ourselves. With this setup, it looks configurable and proper. A caveat
to this is than when a value is changed, the files and processes needs
to be updated or refreshed. This is natural since this is external to
Emacs. Wrapping the generation as a function,
fn/w3m-polipo-tor-update-conf</code>, it is not a big issue to invoke it
each time.</p>
As an aside, customize-set-variable</code> has some data binding mechanism
through :set</code> and :get</code> property but it isn't necessary for a small
shiv. Also, it is fascinating how generating the files with Elisp
binds the data and code.</p>
We now talk about the options we are interested in since I am assuming
you didn't really read the manuals.</p>
>></a>
tor Options</h2>
The configuration values for tor</code>:</p>
`(("SocksPort" . ,fn/w3m-tor-port)
</span>  ("DataDirectory" . ,fn/w3m-tor-cache-dir)
</span>  ("ControlPort" . ,(1+ fn/w3m-tor-port))
</span>  ("DisableDebuggerAttachment" . 0))
</span></code></pre>
SocksPort</strong>
: The port it listens to. This port is what polipo</code> points to.</p>
DataDirectory</strong>
: The directory it uses. Optional, but must be changed since it
defaults to a system directory</p>
ControlPort</strong>
: Optional debugging port if you want to monitor it with tools such
as arm</a>.</p>
DisableDebuggerAttachment</strong>
: If you want to monitor, set this flag</p>
All we strictly need is the **SocksPort** or the interface port and
</span>everything else is for portability.
</span></code></pre>
>></a>
polipo Options</h2>
The configuration values for polipo</code>:</p>
`(("proxyAddress" . "0.0.0.0")
</span>  ("allowedClients" . "127.0.0.1")
</span>  ("diskCacheRoot" . ,fn/w3m-polipo-cache-dir)
</span>  ("proxyPort" . ,fn/w3m-polipo-port)
</span>  ("cacheIsShared" . false)
</span>  ("socksParentProxy" .
</span>   ,(format "%s:%s" "localhost" (number-to-string fn/w3m-tor-port)))
</span>  ("socksProxyType" . socks5))
</span></code></pre>
proxyAddress</strong>
: The address this listens to. For portability, localhost</strong> is the
value.</p>
allowedClients</strong>
: The IP address allowed. Portability again, localhost</strong> is the value.</p>
diskCacheRoot</strong>
: Like with tor</code> and DataDirectory</strong>.</p>
proxyPort</strong>
: The port this listens to. This port is what w3m</code> uses.</p>
cacheIsShared</strong>
: Set to true if the configuration is used by only one instance.</p>
socksParentProxy</strong>
: The SOCKS proxy port it listens to. This port is the meeting of
tor</code> and polipo</code>.</p>
socksProxyType</strong>
: The SOCKS proxy type. The default socks5</code> is what tor</code> is.</p>
A little more nuanced since it is the middle man. What is strictly
needed here is proxyPort</strong> and socksParentProxy</strong> which are just the
interface ports.</p>
Now that the options are clear, we move to our browser options.</p>
>></a>
w3m Options</h2>
We now look at the main browser and the only external option it needs,
http_proxy</code>. This is simply adding to w3m-command-arguments</code> the
value http_proxy=https://127.0.0.1:<polipo-port></code> after -o</code> option.
Aside, a nuance is also duplicating it for https_proxy</code> as such:</p>
(setq w3m-command-arguments
</span>   (append w3m-command-arguments
</span>           (list "-o"
</span>              (format
</span>               "http_proxy=http://127.0.0.1:%s/"
</span>               fn/w3m-privoxy-port))
</span>           (list "-o"
</span>              (format
</span>               "https_proxy=https://127.0.0.1:%s/"
</span>               fn/w3m-privoxy-port))))
</span></code></pre>
To note, w3m</code> has its own configuration thus this configuration can
be set there but it does ruin the data and code binding. It is now
just a matter of calling the appropriate processes when w3m</code> loads:</p>
(setq fn/w3m-tor-process
</span>   (start-process "w3m-tor" "*w3m-tor*" "tor" "-f" fn/w3m-tor-conf-file)
</span>   fn/w3m-polipo-process
</span>   (start-process "w3m-polipo" "*w3m-polipo*" "polipo" "-c" fn/w3m-polipo-conf-file))
</span></code></pre>
Aside from managing the processes, all the pieces should work together
properly.</p>
If you don't need anonymity for some specialized hosts or domain, set
it via w3m-no-proxy-domains</code>:</p>
(add-to-list 'w3m-no-proxy-domains "127.0.0.1")
</span>(add-to-list 'w3m-no-proxy-domains "localhost")
</span></code></pre>
As for me, if I don't add this, I can't test my blog since I will get
a proxy error. Configure to your actual and specific setup.
Refactoring that, here is a command to add the site as a proxy
exception:</p>
(defun fn/w3m-add-current-host-to-no-proxy-domains ()
</span>  "Add current host to `w3m-no-proxy-domains'"
</span>  (interactive)
</span>  (when (eq major-mode 'w3m-mode)
</span>    (lexical-let* ((parts (w3m-parse-http-url w3m-current-url))
</span>        (host (elt parts 1)))
</span>      (add-to-list 'w3m-no-proxy-domains host t)
</span>      (w3m-reload-this-page))))
</span></code></pre>
>></a>
Auto Start And Kill</h2>
When w3m</code> loads, it might be convenient to start it up. However,
these processes is not as simple as flyspell</code> or the like, so asking
permission or confirmation as a reminder might be wise. A simple
snippet does the job:</p>
(when (yes-or-no-p "Start polipo and tor for w3m? ")
</span>  (fn/w3m-polipo-tor-start-process))
</span></code></pre>
Now the flip side of killing it should be handled when Emacs
terminates; however for some reason, tor</code> doesn't cleanly stop which
creates a leak that is not visible by list-process</code>. Strangely,
polipo</code> closes properly but tor</code> does not. A stronger guarantee
is to kill it as part of Emacs:</p>
(add-hook 'kill-emacs-hook #'fn/w3m-polipo-tor-kill-process)
</span></code></pre>
This hook is not a true guarantee as Emacs can be killed externally,
so this hook will not run, thus tor</code> is still running after the next
session. You can check using proced</code> if a tor</code> is still running and
kill it accordingly. This is apparently standard behavior for deamons.</p>
When Emacs is killed, it sends a SIGHUP signal to its child processes
or more importantly to the processes made with start-process</code>. As
stated by the manual:</p>
SIGHUP
</span>The signal instructs Tor to reload its configuration (including closing and reopening
</span>logs), and kill and restart its helper processes if applicable.
</span></code></pre>
This means when Emacs is killed externally, it restarts tor</code> instead
of killing it. Sadly, changing the kill or intercept the signal tor</code>
receives is not easy. For now, this issue is out of scope for a simple
process glue. You can create a custom script to start tor</code>, but this
is a reminder that we can't control everything without lower level
work.</p>
>></a>
Conclusion</h2>
With this, we joined w3m</code>, polipo</code>, tor</code> to browse more privately
and all within Emacs. Not perfect but it gets the job done. There are
features or aspects that can be improved:</p>

Data and argument binding, managing w3m-command-arguments</code> as well</li>
Error handling</li>
Process management on open and close of w3m</code>, maybe prodigy</code>?</li>
Proxy fine-tuning and configuration, checkout custom polipo conf from tor</a></li>
</ul>
As disclaimer, this does not fully make you anonymous specially in
cases where you need to login, but it is an improvement over private
browsing.</p>


Dogville - Transparent Morality
Unknown — Wed, 01 Feb 2017 00:00:00 +0000
>></a>
Dogville</h2>
Dogville removes superfluous walls to examine moral actions</strong></em></p>
Dogville (2003)</a> is a unique yet difficult film to watch. Aside from
using three hours and nine chapters like Quentin Tarantino's The
Hateful Eight</a>. It presents a town with no walls but only a chalk
outline akin to an architectural blueprint. A town that is bare of
superfluous details such as walls and trees and roads and fence, just
the space where people live.</p>
  </p>
I like director, Lars von Trier</a>, and his other work Melancholia</a> for
tackling different issues such as depression although his camera work
and visuals are somewhat stagnant. To compensate for the lack of
visuals, it must have a strong story or compelling characters. If you
have seen this and made it through, then it is for you to decide if it
is.</p>
My focal point of dicussion is on the use of the setting and how it
correlates to its message. The message itself is somewhat direct with
a little subversion on tropes so I won't discuss that here.</p>
SPOILER ALERT</strong></p>
>></a>
A Town With No Walls</h2>
The audience can see the town has no walls but the actors are confined
by the space which they live in. This is akin to a stage play where
everything has been stripped to the bare minimum only to focus on the
actors. It is a strange setup indeed for a long and experimental film
but what does it really do? You may say it is a gimick to reduce
setting cost and excuse shoddy cinematography to which I might agree;
however, there is a thematic reason why it is used.</p>
At a face value, this allows the audience to see the actors behind
walls. Obvious but misleading since the camera controls the viewpoint.
The true value of it can be seen from this unseen and horrendous rape
scene:</p>
  </p>
The audience sees the sexual assault but not the actors. Not only does
this heighten the devious act, this emphasizes the distance and denial
the towns people have. They are near but cannot or refuses to see what
is next to them. This disonnance between what the audience and actors
sees is what this setup bring.</p>
However, it is not enough to justify it with a scene or two. To give
it meaning, it must be tied to the message or story of the film. The
town spiritual and moral leader is a philosopher, so this films hints
at saying something moral. A critical line from this philosopher is:</p>
The whole country would be better served with a greater attitude of</strong></em>
openness and acceptance</strong></em></p>
Openness and acceptance. Hipocricy and denial, to be exact. By laying
down the extraneous walls, the town and its people are laid bare for
the audience to examine.</p>
>></a>
Reflection</h2>
After the hardships, hipocricy and cruelty the town has displayed to
Grace, the judgment at the end is well deserved although you could
reason more about it. More importantly, the initial decision to show
mercy yet reconsider wrath instead is very appropriate. Her reasoning
to be a martyr or merciful is hypocritical to what she reasons out,
which the film discusses with little for me to say. The theme of
self-examination runs deep.</p>
The people themselves are deceptive in being kind at first but treat
her harshly later on and deny their abuse and wrongdoing. No one is
innocent specially the philosopher who merely uses words to his own
benefit.</p>
  </p>
When Grace attempts to leave, no one bothered asking what they did
wrong in why should a person leave their small community. Instead,
they tie her up to a proxy ball-and-chain. This film is hard to watch
because of the treatment and reality of Grace, which might happen to
anyone. A dark interpretation of human morality.</p>
Going back to our setup, aside from the town's ironic geographic
emotional size, it reflects how little in the town really matters.
Since majority of the details could be removed even the dog, there is
little to care for about the town and by extension its people.</p>
  </p>
In a way, this supports Grace's decision to remove this morally
deceitful town. Notice the one that remains after the massacre is the
unknowing and pure dog. It goes a bit more abstract.</p>
>></a>
Abstraction</h2>
GRACE: Why not just call it Dogville?
</span>TOM: Wouldn't work. No, it wouldn't work. It's got to be universal.
</span>TOM: A lot of writers make that mistake.
</span>
</span></code></pre>
Finally, the philospher indicates in naming his realized book to be a
universal name, not theirs furthering the meaning. However, it does
raise the point why the town is called Dogville. By joining the two
lines together, it indicates that the town itself is symbolic of
cruelty and hipocrisy happening anywhere.</p>
So what if we imagine transparent walls around our homes, what would
it look like? Would it be just as Dogville be? If so would, that our
morality and courtesy be only our advantage? This is the literal
meaning of the setup, what is a world without walls? Do we want to see
beyond what our minds allow like bullying, abuse or rape in society?</p>
Honestly as a hipocrite, I do not want to see what beyond the walls.
In that sense, is my morality truly a convenience? In general, is
morality that transparently empty?</p>
Obviously there is more philosophical but this is the message of the
film. So in my opinion, the use of transparent walls was used to good
effect. A unique setup that I won't probably see in a long time.</p>
>></a>
Conclusion</h2>
The film is pretty straightforward in its meaning although long and
chatty. If you can stomach it, I encourage you to watch this gut
wrenching film for its unique story and depiction of morality. As for
me, it is one of my new favorite film that I cannot rewatch without
effort.</p>


A DSL For Git And Projectile Ignore File
Unknown — Wed, 25 Jan 2017 00:00:00 +0000
>></a>
Problem</h2>
After writing some projectile</a> glue and .projectile</code> and .gitignore</code>
files, I thought to myself if I could write code to generate those
files. A DSL so to say. The reason is three-fold:</p>

Write a simple ignore file structure as Elisp code</li>
Allow Elisp variables for dynamic and reusable sections</li>
Have a hand in writing a generic DSL</li>
</ul>
Aside from my hacking sensibilities, the key concerns of doing so may
be put as:</p>

Maintaining two ignore files when it is just really one with a
different syntax</li>
You cannot reuse blocks of repeating sections with different files</li>
</ul>
The former is syntactic separation while the later is flexibility and
reusability. For my use case with Elisp, I have two projects: my Emacs
configuration and my org-jekyll-blogger.el</a> blog project. Using git</code>
and projectile</code>, I have to more or less copy my ignore files and
tweak it a bit. To address this, I created my own DSL magin</code> to
handle .gitignore</code> files; .projectile</code> is similar in intent but
magin</code> was the library I had in mind as proof.</p>
As an example, this is my .emacs.d</a> .gitignore</code>:</p>
# Base files
</span>config.el
</span>personal.el
</span>
</span># Base directory
</span>.cache
</span>extra
</span>
</span># Project specific
</span>working-config.org
</span>my-macros.el
</span>
</span># Library
</span>lib/sandbox
</span>lib/packages
</span>!lib/modules
</span>!lib/scripts
</span>
</span># Block for gtags
</span>GPATH
</span>GTAGS
</span>GRTAGS
</span>
</span># Block for emacs
</span>*.elc
</span>.#*
</span>*Org Src*
</span></code></pre>
It is generated by this DSL:</p>
(</span>magin</span>-</span>write</span>-to-</span>project
</span> `</span>(</span>delimited </span>;; Context keyword that separates contexts by newlines
</span>
</span>   </span>(</span>context </span>;; Represents a semantic group
</span>    </span>(</span>comment </span>"</span>Base files</span>") </span>;; A comment keyword
</span>    </span>(</span>file ,</span>(</span>format </span>"</span>%s.el</span>" (</span>file</span>-</span>name</span>-</span>base fn</span>/</span>config</span>-</span>file</span>))) </span>;; A file keyword
</span>    </span>(</span>file ,</span>(</span>format </span>"</span>%s.el</span>" (</span>file</span>-</span>name</span>-</span>base fn</span>/</span>personal</span>-</span>file</span>)))) </span>;; Notice I use a Elisp variable
</span>
</span>   </span>(</span>context
</span>    </span>(</span>comment </span>"</span>Base directory</span>") </span>;; Generates "# Base directory"
</span>    </span>(</span>dir </span>"</span>.cache</span>") </span>;; A dir keyword
</span>    </span>(</span>dir </span>"</span>extra</span>")) </span>;; An alias to file but it indicates intends for a directory
</span>
</span>   </span>(</span>context </span>;
</span>    </span>(</span>comment </span>"</span>Project specific</span>")
</span>    </span>(</span>file </span>"</span>working-config.org</span>")
</span>
</span>    </span>(</span>file  ,</span>(</span>format </span>"</span>%s.el</span>" (</span>file</span>-</span>name</span>-</span>base fmk</span>/</span>macro</span>-</span>file</span>))))
</span>
</span>
</span>   </span>(</span>context
</span>    </span>(</span>comment </span>"</span>Library</span>")
</span>    </span>(</span>path </span>;; Represents a path prefix
</span>     </span>"</span>lib</span>" </span>;; Everything underneath is generated with a "lib" prefix
</span>     </span>(</span>context
</span>      </span>(</span>dir </span>"</span>sandbox</span>")
</span>      </span>(</span>dir </span>"</span>packages</span>"))
</span>     </span>(</span>include
</span>      </span>(</span>dir </span>"</span>modules</span>")
</span>      </span>(</span>dir </span>"</span>scripts</span>"))))
</span>
</span>   </span>(</span>block</span> gtags</span>) </span>;; Reusable blocks
</span>   </span>(</span>block</span> emacs</span>))
</span> user</span>-</span>emacs</span>-</span>directory </span>;; Rewrites =.gitginore= of `user-emacs-directory'
</span> </span>)
</span></code></pre>
I do hope the DSL is easy to digest. Here are the two libraries for
the interested: magin</a> and projin</a>. If you stay around, I'd like to
discuss the following concepts in crafting it:</p>

Backquote</li>
Dispatching</li>
Implicit Environment</li>
</ul>
>></a>
Backquoting</h2>
The reason why I was confident in making this DSL is due to thanks to
backquoting</a>: it mixes the use of quoting and expressions in a Lispy
way. I point you to understand quoting</a> before proceeding, but as a
quick brush it is using list of symbols as data.</p>
As an example of quoting:</p>
(</span>+ </span>1 2 3</span>) </span>;; 6
</span>
</span>'</span>(</span>+ </span>1 2 3</span>)         </span>;; (+ 1 2 3)
</span>(</span>quote </span>(</span>+ </span>1 2</span> 3e</span>))
</span>
</span>
</span>(</span>subject linking</span>-</span>verb adjective</span>)  </span>;; Missing symbols
</span>'</span>(</span>subject linking</span>-</span>verb adjective</span>) </span>;; As is
</span></code></pre>
It also allows for lazy evaluation, the problem arises when we want to
integrate Elisp variables into the equation:</p>
(</span>setq left </span>1
</span>   right </span>2</span>)
</span>
</span>(</span>+</span> left right</span>) </span>;; 3
</span>
</span>(</span>+ </span>1</span> right</span>)    </span>;; 3
</span>'</span>(</span>+ </span>1</span> right</span>)   </span>;; (+ 1 right)
</span>
</span>;; Using backquotes
</span>`</span>(</span>+ </span>1</span> ,right</span>)           </span>;; (+ 1 2)
</span>`</span>(</span>+ </span>1</span> ,</span>(</span>+</span> left right</span>))  </span>;; (+ 1 3)
</span>
</span>;; Evaluating the actual value
</span>(</span>eval `</span>(</span>+ </span>1</span> ,</span>(</span>+</span> left right</span>)))  </span>;; 4
</span></code></pre>
The complicated approach would ask us to create a tokenizer, parser
and compiler while managing text matching and all that jazz; lispy
data structures allows us to bypass this through the quoting. Without
needing anymore, we can create a quoted structures that integrates
with Elisp without any hastle.</p>
>></a>
Domain Syntax</h2>
So in thinking of the DSL, let us understand what the domain is. Our
domain of abstraction is gitignore</a>. After some thinking and reading,
this is the abstraction I want to address:</p>

Files and directories have no distinction</li>
Comments</li>
Path context</li>
Separators</li>
Inclusion</li>
Reusable blocks or groups</li>
</ul>
There are other such abstraction I wish not cover:</p>

Quoting or escaping</li>
Glob keywords, as in rx.el</code>, for single or double asterisk</li>
</ul>
A DSL that is good enough to accomplish the above is good enough
without being too abstract. Without stretching too much, this DSL has
no intermediate output, it translates directly to text. This tradeoff
loses flexibility in output but at least it is easier. If you wanted,
the DSL can be mechanically a list of plists with a main text
manipulated by several properties as such:</p>
(</span>magin</span>--</span>compiler
</span> </span>(</span>path </span>"</span>glob</span>"
</span>       </span>(</span>include
</span>        </span>(</span>file </span>"</span>foo.bar</span>")
</span>        </span>(</span>path </span>"</span>gib</span>"
</span>              </span>(</span>dir </span>"</span>foo-bar</span>")))))
</span>
</span>;; This hypothetically might yield
</span>'</span>((</span>:line </span>"</span>foo.bar</span>"</span> :include </span>t</span> :parent </span>"</span>glob</span>")
</span>  </span>(</span>:line </span>"</span>foo-bar</span>"</span> :include </span>t</span> :parent </span>"</span>glob/gib</span>"))
</span></code></pre>
Without much complexity aside, we can start with hacking with the
simple syntax.</p>
>></a>
Keywords</h2>
At last, we define our data structure:</p>
;; The leaf keywords
</span>(</span>file ,line</span>)
</span>(</span>dir ,line</span>) </span>;; Alias for file
</span>
</span>;; Non functional nodes
</span>(</span>comment ,text</span>)
</span>(</span>newline</span>)
</span>
</span>
</span>;; Syntactic grouping, the equivalent of progn
</span>(</span>context &rest ,sublines</span>)
</span>
</span>;; Contextual grouping, like `context' but affects everything inside it
</span>;; Sublines have the property `:include' as `t'
</span>;; Which tells the leaf node to add `!' as a prefix
</span>(</span>include &rest ,sublines</span>)
</span>
</span>;; Sublines have the property `:path' set to `path'
</span>;; Nesting of paths are handled
</span>(</span>path ,path &rest sublines</span>)
</span>
</span>
</span>;; Block keywords
</span>;; Create a variable
</span>(</span>defblock ,</span>block-</span>name &rest </span>block-</span>lines</span>)
</span>
</span>;; Call the variable
</span>(</span>block</span> ,</span>block-</span>name</span>)
</span>
</span>
</span>;; Aesthetic grouping
</span>;; Every subline is interleaved with a newline
</span>(</span>delimited &rest ,sublines</span>)
</span></code></pre>
Those are the keywords that we must handle in our DSL. With this in
mind, we can start be creating a function that handles the
compilation.</p>
(</span>defun </span>magin--compiler </span>(</span>dsl &optional env</span>)
</span>  </span>"</span>Compiles DSL with the environment ENV.</span>"
</span>  </span>nil</span>)
</span></code></pre>
This is our DSL handler, every node that wants to compile nodes have
to go through this dispatching function. This setup allows us to add
more keywords independently of each other. However, we are not going
to use a long switch</code> or cond</code> statement; we will use the implicit
Elisp environment to find handlers. If the compiler comes to a
keyword, it finds a function in the Elisp environment that starts with
magit--dsl-</code> and invokes that as a handler.</p>
(</span>defun </span>magin--dsl-context </span>(</span>dsl env</span>)
</span>  </span>"</span>File keyword handler.</span>")
</span>
</span>(</span>defun </span>magin--dsl-file </span>(</span>dsl env</span>)
</span>  </span>"</span>File keyword handler.</span>")
</span>
</span>(</span>defun </span>magin--dsl-block </span>(</span>dsl env</span>)
</span>  </span>"</span>Block keyword handler.</span>")
</span>
</span>(</span>magin</span>--</span>compiler
</span> `</span>(</span>context
</span>   </span>(</span>file </span>"</span>a</span>")
</span>   </span>(</span>block</span> emacs</span>)
</span>   </span>(</span>unknown</span>) </span>;; Error, no `magit--dsl-unknown'
</span>   </span>))
</span></code></pre>
In the example above, the compiler finds the three functions above and
calls them but fails on the last one. So if you want to extend it
without messing with the source, one can simply define a function with
the prefix and it will be detected by the compiler. This is similar to
how use-package</code> does it with its keywords. A global dependency
injection if you will. This magic is done through intern-soft</code>:</p>
(</span>lexical</span>-let* </span>((</span>rule</span>-</span>name </span>(</span>symbol-name rule</span>))
</span>    </span>(</span>rule</span>-</span>handler
</span>     </span>(</span>intern</span>-</span>soft
</span>      </span>(</span>format </span>"</span>%s%s</span>"</span> magin</span>-</span>dsl</span>-</span>prefix rule</span>-</span>name</span>))))
</span>  </span>(</span>if </span>(</span>null</span> rule</span>-</span>handler</span>)
</span>      </span>(</span>error </span>"</span>No rule to handle %s at dsl: %s</span>"</span> rule</span>-</span>name dsl</span>)
</span>    </span>(</span>funcall rule</span>-</span>handler dsl env</span>)))
</span></code></pre>
As a side note, you can use an hash-map</code> or create your own obarray</code>
or symbol environemt if you really wanted a private space. In our
approach, you can see the symbols with the helpful describe-variable</code>
without extra work.</p>
If you want to use eval-sexp</code> with the raw DSL, you have to create
your own eval</code> and env</code>. You can remap eval</code> with magin-compiler</code>,
but you need to know when it is evaluating a DSL or a Lisp. We're not
creating a new interpreter or environment, so this is good enough.</p>
With this simple mechanism, we can define the keywords incrementally.</p>
>></a>
Context Keyword</h2>
So our dispatcher is a function that takes a dsl and an environment
which returns a text. Without the intermediate data structure, the
context is applying the dispatcher to each line and then combining it
with a newline delimiter as such:</p>
(</span>string</span>-</span>join
</span> </span>(</span>mapcar  </span>(</span>lambda </span>(</span>subdsl</span>)(</span>funcall </span>#</span>'magin--compiler</span> subdsl env</span>))</span> subdsls</span>)
</span> </span>"</span>\n</span>")
</span></code></pre>
This is simply lining up the entries. Nothing complicated but how
about manipulating those that manipulate the environment?</p>
>></a>
Environmental Keyword</h2>
A quick way to create an environment like scoping, one can easily use
an alist and it will take care of itself. For example with the
include</code> keyword:</p>
(</span>lexical</span>-let </span>((</span>new</span>-</span>env </span>(</span>append </span>(</span>list </span>(</span>cons :include </span>t</span>))</span> env</span>)))
</span>  </span>(</span>magin</span>--</span>dsl</span>-</span>context dsl new</span>-</span>env</span>))
</span></code></pre>
You could use plist but have to implement an extend</code> function for it.
Interestingly, you could also use an alist to simulate an enviroment
and lexical scoping without much problem.</p>
It is as simple as appending the cons</code> entry and it is done. The
others are implemented the same way. Now how about implementing the
leaf keywords</p>
>></a>
Leaf Keyword</h2>
The implementation above forces everything to the leaf node which
really have only one important one, file</code>. The file has to take the
text value and format it based on the environment.</p>
(</span>lexical</span>-let* </span>((</span>parent </span>(</span>cdr </span>(</span>assoc :parent env</span>)))
</span>    </span>(</span>include </span>(</span>if </span>(</span>cdr </span>(</span>assoc :include env</span>))"</span>!</span>" </span>nil</span>)))
</span>  </span>(</span>concat include parent file</span>))
</span></code></pre>
Our friend is assoc</code> and cdr</code> when manipulating an alist enviroment.
Again, it is simply a matter formatting. So writing a DSL with this
setup is actually easy.</p>
>></a>
Block Keyword</h2>
Lastly, let's talk about defining a block variable. Like with the
dispatcher, we can use the implicit Elisp environment as the variable
space. No need to define a hash of symbols and lookups, we already
have one and we take advantage of it. The keyword defblock</code> will
simply save the defintion into a symbol with a magin--block-</code> prefix
and the other one, block</code>, looks for it. Aside from intern-soft</code>,
the complimentary friend is makunbound</code>, which unbinds the symbol so
the value is updated properly.</p>
(</span>lexical</span>-let* </span>((</span>block-</span>name </span>(</span>symbol-name </span>block-</span>symbol</span>))
</span>    </span>(</span>block-</span>def</span>-</span>name </span>(</span>intern
</span>                     </span>(</span>format </span>"</span>%s%s</span>"
</span>                             magin</span>-block-</span>prefix
</span>                             </span>block-</span>name</span>))))
</span>  </span>(</span>makunbound </span>block-</span>def</span>-</span>name</span>)
</span>  </span>(</span>eval `</span>(</span>defvar ,</span>block-</span>def</span>-</span>name '</span>(</span>context ,@</span>block-</span>def</span>)
</span>           ,</span>(</span>format </span>"</span>Block definition for %s</span>" </span>block-</span>name</span>)))
</span>  </span>block-</span>def</span>-</span>name</span>)
</span></code></pre>
Aside from using some direct eval</code> magic, it is as straightforward as
it gets as well as with the block</code> handler. With this, we can have
our resusable blocks of code.</p>
(</span>magin</span>--</span>dsl</span>-</span>defblock
</span> `</span>(</span>defblock emacs
</span>    </span>(</span>comment </span>"</span>Block for emacs</span>")
</span>    </span>(</span>file </span>"</span>*.elc</span>")
</span>    </span>(</span>file </span>"</span>.#*</span>")
</span>    </span>(</span>file </span>"</span>*Org Src*</span>") </span>;; Org Src buffers
</span>    </span>)
</span> </span>(</span>list</span>))
</span></code></pre>
>></a>
Non-functional Keyword</h2>
The least important is the non-functional ones, like with comment</code>
and newline</code> which are prefixed text and an empty text respectively.
However, the delimited</code> keyword is a bit more complex that it
interleaves each line with a newline</code> keyword.</p>
(</span>lexical</span>-let </span>((</span>delimited</span>-</span>dsls
</span>     </span>(</span>cdr
</span>      </span>(</span>apply </span>#</span>'append
</span>         </span>(</span>mapcar
</span>          </span>(</span>lambda </span>(</span>dsl</span>)
</span>            </span>(</span>list '</span>(</span>newline</span>)</span> dsl</span>))
</span>          subdsls</span>)))))
</span>  </span>(</span>magin</span>--</span>compiler
</span>   `</span>(</span>context
</span>     ,@delimited</span>-</span>dsls</span>)
</span>   env</span>))
</span></code></pre>
Take your pick in implementing interleave</code> but the thing here is that
it wraps the old context with a newly modified one. It may not be a
big thing but this interferes with debugging and tracing when you
don't know the original source. This is one weakness of this approach
that it interferes with the dsl itself. Nothing terrible but it is the
price of having the return value as text instead of an intermediate
value.</p>
With all those ideas, the DSL is easily implemented.</p>
>></a>
Wrap Up</h2>
So with this design and implementation we have a crud but simple and
dynamic DSL. You can look at the main defintion and notice the use of
backquoting in how it addresses the problems mentioned. Now I can
manage my ignore file with Emacs.</p>
Now for some minor things we might have forgotten:</p>

Unit tests</li>
Logging</li>
Debugging</li>
</ul>
The DSL is not too complex to warrant any of those but it is nice to
note and implement. As one final point, we could have used a
defmacro</code> instead of a dispatch compiler but in case we want to
update our code in order to join other contexts, we have a base.</p>
(</span>setq left</span>-</span>code `</span>(</span>context
</span>               </span>(</span>file </span>"</span>a</span>")
</span>               </span>(</span>file </span>"</span>b</span>"))
</span>   right</span>-</span>code `</span>(</span>context
</span>                </span>(</span>file </span>"</span>c</span>")
</span>                </span>(</span>file </span>"</span>d</span>")))
</span>
</span>
</span>(</span>magin</span>-</span>join left</span>-</span>code right</span>-</span>code</span>) </span>;; ???
</span></code></pre>
If we are dealing with text, we can't have this yet. In particular if
you have an initial defintion and want to update it according some
condition, then it requires changing the code to handle plist or
something that fancies the data.</p>
To reiterate, Elisp allows us to create a DSL with ease and little
knowledge.</p>
>></a>
Conclusion</h2>
So I copied this code to create projin</code> for projectile</code> ignore file
but unless I need more functionality, I don't think I need it for now.
If another ignore file comes my way, I might refactor and do the same.</p>
The one thing I want to really write is a DSL for SQL writing, not SQL
itself. My dream is to write SQL as blocks of data.</p>
(</span>setq display</span>-</span>fields '</span>(</span>fields first</span>-</span>name last</span>-</span>name middle</span>-</span>name</span>))
</span>
</span>(</span>select
</span> display</span>-</span>fields
</span> </span>(</span>from</span> table</span>))
</span>;; "SELECT firstName, lastName, middleName FROM table"
</span>
</span></code></pre>
The idea in that snippet is the fields are data itself. It also
extends to queries being data but the abstraction of aliasing and
joining is sketchy for now. That is my dream though: to write a DSL
for SQL writing. For now, ignore files are good practice.</p>


The Garden of Words - Maturity And Responsibility
Unknown — Wed, 25 Jan 2017 00:00:00 +0000
>></a>
The Garden of Words</h2>
A faint clap of thunder, Clouded skies, Perhaps rain will
come. If so, will you stay here with me?</strong></em></p>
The Garden of Words (2013)</a> for me is the second best in my list for
Shinkai. After rewatching it and found it deeper than I thought it
would be, I want to talk about this visually beautiful film. Aside
from being emotionally and romantically moving, I want to talk about
another aspect of there relationship namely the themes of maturity and
responsibility.</p>
SPOILER WARNING</strong></p>
NOTE: Screenshots have reduced quality since they are too pretty to</strong>
load fast. That's how pretty and defined the film's frames are.</strong></p>
>></a>
Place And Time</h2>
In order to discuss the relationship between our characters, let's
discuss the mechanics of it. During rainy morning, each of our
characters retreat to a tiny hut in a park where they accidentally and
coincidentally meet and do so every rainy day.</p>
  </p>
Neither revealing who they really are and being in their own little
solitude, both warm up to there shared isolation over time. This is a
quiet and beautiful setup that I rarely see in films. A genre savvy
person would cue this is as an opportunity to have a blooming
relationship in an isolated environment where meeting outside this
shelter is a plot revelation.</p>
Aside from being in their own world, the rain also plays quite a
symbolic meaning here. The rain serves as a veil to the world they are
escaping, the city noise muffled by the falling rain. This veil
obscures the world to focus on them and their internal silence. It
also serves as the extent of there initial relationship. Shrouded by a
veil of escapism, neither revealing who they are outside, only what
they wish to show inside. During their meetings under the shower, they
are hidden and hiding themselves. Quite a poetic tone this setup
yields.</p>
Another way to look at this setup is where people can be people
without there social standing and responsibility. Our characters only
know what the dressing means but they come bare without the chatter
and stigma of it. Weary adults and youth, they can come here with
their naked silence and weakness. A student and an office worker and
nothing more and this is what the film settles on. Both skipping work
and neither judge each other and talk about it. So they can go on
without knowing there names or lives. This is what this place is, a
place where solitude and privacy is respected.</p>
There is an asymmetry here that break the characters contracts. During
their first meeting, the older woman recites a Japanese poem which I
find very poignant and at the same time a hint on who she is. It
implies she knows where the student goes and say something that the
school might know. However, the student does not pick up on this
subtle hint and they continue in their shared space.</p>
  </p>
>></a>
Characters</h2>
A high-school student and a working woman. I understand if people find
this a strange combination of a romantic relationship between an older
woman a decade older than the man, but the context of their
relationship should be examined. The examination I would like to focus
on who is the child and adult.</p>
>>></a>
The Child</h3>
Takao, the high schooler.</p>
  </p>
What makes him standout is that he is mature for his age: he does the
cooking, he gets jobs to pay for his tuition, he has a dream that he
is working on. Despite being a student, he is pretty mature in taking
household and personal responsibility aside from skipping school on
rainy mornings.</p>
>>></a>
The Adult</h3>
Yukino, the literature teacher.</p>
  </p>
Although it is sad she is bullied by her students, she does not work
for a while and decides to go to the park on rainy mornings. She does
not know how to cook so she brings chocolates, where sweets are
indicative of being childish. She drinks beer early morning as if to
lull the day away. Ignoring responsibility, she is not a mature role
model despite being a teacher.</p>
>>></a>
Conflict</h3>
Symbolically, Takao is the adult while Yukino is the child while
physically it is the opposite. Here is where opposites attract.</p>
  </p>
In the scene where Takao measures Yukino's foot, I find it beautiful
and subtle. To contextualize, Yukino gives Takao a shoe design book
which is somewhat the closest thing she has done in being a teacher
by giving him a book that he cannot afford. To repay that kindness,
Takao offers to make Yukino a shoe for her, handcrafting to show its
personal and priceless value. This simple exchange shows their
interest in each others space and world.</p>
Takao wants to be treated as an adult, going to school on a sunny day
illuminates and reminds him of the fact that he is not. Granted that
he is surprisingly independent, but the fact that he wants to be
acknowledged as such shows he is not really one yet. One does need to
tell everyone who they are to be who they really are. Weirdly, he
thrust into this independence partly because his mother is gone
hunting men or for the lack of a parental figure. The reason why he
wants to be a shoe maker is presented in a way to please her mother
and carried this dream or idea through his teenage years. By making
shoes, will it give him the attention his mother had once given the
shoe? Regardless, the presence of the mysterious adult Yukino allows
him to connect to the world he wants as well fulfill this dream or
desire to be noticed.</p>
On the other hand, Yukino is emotionally crippled by the people she
is helping. For no fault of her own, one of her student falls for her
but the girlfriend of this student started spreading rumors which
caused the class to harass her. One subtle thing I had a hard time
picking up is whether Yukino was in a relationship with another
teacher, Itou, who broke up with her because he believed the rumors
and dissuaded her in taking action since it might affect the school
they are working on; if that is the case, she has no one to
understand her. Either way by being seen as an adult and teacher, she
cannot truly show this weakness and do her job since by not being
emotionally reconciled and repressed she cannot effectively teach
children how to be a strong adult and not a crybaby. So by being with
a student that does not know her, her crippling isolation is slowly
melting away by finding acceptance in him.</p>
You can say Yukino is indeed acting weak or immaturely but that is
the point with her. Despite growing in years, it does not mean she
grows emotionally on par. People are not invulnerable and get hurt,
children and adults.</p>
  </p>
So the nature and resolution of their conflicts and coincide with
their communion. Their prayer for rain is really a heartfelt one so
the response to the poem seem to suggest the answer:</p>
Even if rain comes not, I will stay here, together with you.</strong></em></p>
>></a>
Equal Footing</h2>
The film uses walking here to symbolize the struggle with life and
shoes the support and strength needed to continue with it. Rightly so,
shoes are made for walking and to protect the feet from stones and harm.</p>
  </p>
In line with responsibility, the thing I want to point out here is
social status and footing. The phrase "on equal footing" holds weight
here as it describes the relationship and catharsis for our
characters. So the scene where Takao confesses to Yukino has so much
meaning to address.</p>
>>></a>
Their Relationship Outside</h3>
  </p>
After meeting on a sunny day, a storm suddenly switches on which
leads our characters to experience their relationship outside of it.
I genuinely felt moved by the simple, short and quiet scene which
shows what they had in their paradise remains true and valid even if
all of it was a mystery and front. As it pains me to say this, their
relationship has completed their conflict in terms of walking in life.</p>
>>></a>
Walking In The Sunny Season</h3>
Once the rainy season had stopped, both characters cannot and will
not go to their promised place because they had no reason. It looks
cute that way but the clever use of light and rain indicates that
there meeting is a dream made by rain and not just a time and place.
So there relationship is in a hiatus but they move in their lives.</p>
Yukino decides during this period to move to another teaching job and
Takao gets jobs to pay for his tuition and the shoe materials.
Neither would have happened if they never shared their company. So
when they met outside for the first time, they had already decided to
walk.</p>
This is why the confession seems subtly heartbreaking.</p>
>>></a>
Rejection</h3>
 &nbsp</p>
When Takao confesses to Yukino, she is standing up and he is sitting
down. Yukino then responds by saying to call her with her honorifics
or the respect her status brings. She then sits downs and thanks him
for essentially nothing by stressing the line.</p>
I was practicing how to walk on my own, there in that place.</strong></em>
Even if am barefoot...</strong></em></p>
This line means that she was doing fine without Takao and their
relationship effectively has no bearing on their own walks. Moreover,
there relationship before this was unclouded and unburdened by social
responsibility and status, by demanding the honorifics it breaks the
contract of their relationship. The point here is that Takao is now
seen as a child with an infatuation for a woman a decade older than
him.</p>
Notice the act of Yukino sitting down is similar to talking down a
child and that when she does sit, they do not meet eye to eye as the
division on the door divides it. Here is where Takao's age and status
is a hindrance which he escapes in the park. As the kid, he loses out
in the end as he takes his leave.</p>
>>></a>
Acceptance</h3>
The cathartic scene where Yukino chases after Takao reiterates this
emotion and the meaning of their footing, status, or the weight of who they
really are.</p>
  </p>
All Takao wanted was to be seen as a man or adult. In his eyes,
everything was tipped against him. Yukino knowing more about him and
not telling him she was a teacher at his school is indeed unfair as
it looks like she was playing around with him. It is really unfair
the adult gets the stick or prod.</p>
So when Yukino steps down to embrace Takao, their status are
discarded to reveal that indeed she seems him as something more. They
do no meet halfway, they meet as people. Let the tears flows.</p>
  </p>
The next question is how does this fit in with their ending?</p>
>></a>
Open Ended Maturity</h2>
Yukino finally starts teaching again and Takao finishes the shoes. At
the end of the film, Takao goes to the park on a snowy and leaves the
shoes behind leaving their relationship as open-ended. Here is why I
like the ending better is that the answer to it implies message to the
whole story. To interpret the ending let us answer the meaning of
Yukino leaving.</p>
  </p>
>>></a>
Accepting Their Reality</h3>
Simply if Yukino stays, she denies her actions in moving. In one way,
she denies who she is just because she loves someone, which would
undermine her reformed strength. She has a responsibility to herself
to carry on and to Takao who is a child still. If their reality is to
be considered, their relationship is inappropriate. Age, social,
economical or whatever reason it might be, it would be wise if both
parties mature further before doing so. This is emphasized by Takao
with the parting lines:</p>
I was practicing how to walk too. That's what I think now.</strong></em>
One day, when I  can walk much farther on my own...</strong></em>
I'll go see her.</strong></em></p>
Takao is indeed still a child and needs to walk and grow. This
realization settles his path to maturity, that he need not hurry. By
leaving the shoe, he shows resolve instead of waiting for her. So
Yukino has finally acted like a responsible adult.</p>
What I pointed is out the coming to age theme of the film. Others
have done like1 FLCL</a>, but this does it coming from a teenagers point
of view where it might hold better weight and perspective. I
appreciate this portrayal of a teenager, it could have been about
their romantic expressions or moments but I am thankful it was about
their genuine need from each other. However, this does not answer
what it means if they do hook up in the future.</p>
>>></a>
Meeting Up</h3>
At this point, it is up to the reader's interpretation on the
following points or ideas whether they should meetup. The guiding
principle is interpreting it in the context of a mature relationship.</p>
The woman is years older</strong>
: It is typical or preferable for the man to be older than the
woman if fertility is a question.</p>
Takao shoemaking</strong>
: His desire to make shoes comes from a childish idea. Would you
consider his love for shoe making genuine or but a fad as his
brother puts it? Would you consider leaving the shoes behind as
leaving this idea behind?</p>
There relationship comes from a veil</strong>
: Is it wise to have a relationship that does not truly know each
other? I'm assuming Yukino left after their catharsis, so he
still might know nothing about her. By that logic, their
relationship might just be still at surface level. Should he
really chase her then?</p>
Some might say love conquers all or some might say it is just a
phase. Whatever you answer to the main question, it really reflects
on the theme of maturity in their relationship. As for me, I'm okay
if Takao forgets Yukino because their veiled relationship has done
its work. Whether he decides to find her or not in time, we may never
know.</p>
But isn't that love, a dream and a reality?</p>
>></a>
Conclusion</h2>
Dreamer or not, this short film is thought-provoking so much about
teenage maturity and responsibility despite being a love story. That
is why I find this second only to 5 Centimeters Per Second</a> in terms of
exploring aspects of relationships. This film is not perfect that
these points might be too subtle, the open ending, the overly
emotional ending, and so much more but I truly love this film and what
it leaves behind.</p>


Convert Org-Table To YAML For Jekyll
Unknown — Wed, 18 Jan 2017 00:00:00 +0000
>></a>
Problem</h2>
So in making the page for my watch list for the other watcher blog, I
have to convert my unofficial list in Org-Table format to a YAML
format. There are other default formats such as CSV that Jekyll reads
but in keeping with the spirit, I opt to do it as such. The sad thing
about this though is that there is no YAML exporter for
org-table-export</strong>, so I made my own. Here is the snippet</a> to do and a
demonstration.</p>
  </p>
And to use it, you point the mark at a table and use
org-table-export</code> and at the format selection you type
fn/orgtbl-to-yaml</code>. With Jekyll, you have to export it to your _data</code>
Jekyll directory to work as stated by the manual</a>. But of course, we
are here to explore further to what it means.</p>
>></a>
Org-Table Export</h2>
Initially, I thought I had to create a table parser and transformer
but there is usually a command or function that does what you want
albeit with a little tweaking. The command org-table-export</code> is our
friend here but it is lacking when presented with the format: CSV,
TSV, HTML, LaTeX and others but not YAML. One thing to look at where
this option comes from is org-export-backends</code> but might be overkill
to configure. As an caveat, if you add a backend to it you need to
run this script.</p>
(</span>progn
</span>  </span>(</span>setq org</span>-</span>export</span>-</span>registered</span>-</span>backends
</span>     </span>(</span>cl</span>-</span>remove</span>-if-not
</span>      </span>(</span>lambda </span>(</span>backend</span>)
</span>        </span>(</span>let </span>((</span>name </span>(</span>org</span>-</span>export</span>-</span>backend</span>-</span>name backend</span>)))
</span>          </span>(</span>or </span>(</span>memq name val</span>)
</span>             </span>(</span>catch 'parentp
</span>               </span>(</span>dolist </span>(</span>b val</span>)
</span>                 </span>(</span>and </span>(</span>org</span>-</span>export</span>-</span>derived</span>-</span>backend</span>-</span>p b name</span>)
</span>                    </span>(</span>throw 'parentp </span>t</span>)))))))
</span>      org</span>-</span>export</span>-</span>registered</span>-</span>backends</span>))
</span>  </span>(</span>let </span>((</span>new</span>-</span>list </span>(</span>mapcar </span>#</span>'org-export-backend-name
</span>                        org</span>-</span>export</span>-</span>registered</span>-</span>backends</span>)))
</span>    </span>(</span>dolist </span>(</span>backend val</span>)
</span>      </span>(</span>cond
</span>       </span>((</span>not </span>(</span>load </span>(</span>format </span>"</span>ox-%s</span>"</span> backend</span>) </span>t t</span>))
</span>        </span>(</span>message </span>"</span>Problems while trying to load export back-end `%s'</span>"
</span>                 backend</span>))
</span>       </span>((</span>not </span>(</span>memq backend new</span>-</span>list</span>)) (</span>push backend new</span>-</span>list</span>))))
</span>    </span>(</span>set</span>-</span>default 'org</span>-</span>export</span>-</span>backends new</span>-</span>list</span>)))
</span></code></pre>
Something simple would be better for our cause. Is crafting a custom
exporter instead an option? Thankfully, the manual tells us how to
create a simple custom exporter according to this link</a>:</p>
(</span>defun </span>orgtbl-to-language </span>(</span>table params</span>)
</span>  </span>"</span>Convert the orgtbl-mode TABLE to language.</span>"
</span>  </span>(</span>orgtbl</span>-to-</span>generic
</span>   table
</span>   </span>(</span>org</span>-</span>combine</span>-</span>plists
</span>    '</span>(</span>:tstart </span>"</span>!BTBL!</span>"</span> :tend </span>"</span>!ETBL!</span>"</span> :lstart </span>"</span>!BL!</span>"</span> :lend </span>"</span>!EL!</span>"</span> :sep </span>"</span>\t</span>")
</span>    params</span>)))
</span></code></pre>
After reading the documentation, we are interested in two properties:
:skip</code> and :lfmt</code>. The later is more important which controls how a
line is formatted. It accepts a function which takes a list of row
values and returns the record line it represents. Using it as
function, the first value are the row headers and the remaining row
values. We mainly want to focus the function on the value so that is
why have :skip</code> to ignore the first header row. Aside from that, we
can craft our row formatter.</p>
Since this is YAML, our record line is a bit more complex. Each record
starts with a dash and space, ends with a newline, separated by a
newline and two spaces for alignment. Each value must be in a <key>: <value></code> format and escaped by "</code>. With a sample data, it would look
something like this:</p>
- id: "1"
</span>  name: "Muffin"
</span>  description: "Khajit Assassin"
</span>- id: "2"
</span>  name: "Marble"
</span>  description: "Barbarian Alchemist"
</span></code></pre>
To accomplish this, we need to get the row headers</code> which can be also
found with the car</code> of the table</code> parameter. We then use a
functional zip with the headers and each row values, format
accordingly and profit. With a quick zip</code> shiv, it looks like this.</p>
(</span>lambda </span>(</span>values</span>)
</span>  </span>(</span>concat
</span>   </span>"</span>- </span>"
</span>   </span>(</span>string</span>-</span>join
</span>    </span>(</span>mapcar
</span>     </span>(</span>lambda </span>(</span>pair</span>)
</span>       </span>(</span>lexical</span>-let </span>((</span>header </span>(</span>car pair</span>))
</span>           </span>(</span>value </span>(</span>cdr pair</span>)))
</span>         </span>(</span>format </span>"</span>%s: </span>\"</span>%s</span>\"</span>"</span> header value</span>)))
</span>     </span>(</span>funcall zip headers values</span>))
</span>    </span>"</span>\n  </span>")))
</span></code></pre>
That finishes the core formatter.</p>
>></a>
Front Matter and Jekyll</h2>
So this exporter is aimed for Jekyll and I know little about YAML
since I rarely use it.</p>
I only want to discuss how the key is produced. You can simply take
the key as is but what if it has spaces? I find it odd having spaces
within object keys, so I prefer to hyphenate it and lowercase for my
ease but obviously you can change it to be camelCase if you want.
Using this in Jekyll, it will look like this:</p>
(</span>lexical</span>-let </span>((</span>headers
</span>     </span>(</span>mapcar
</span>      </span>(</span>lambda </span>(</span>header</span>) </span>;; Shiv camel casing
</span>        </span>(</span>replace</span>-</span>regexp</span>-</span>in</span>-</span>string
</span>         </span>" " "</span>-</span>"
</span>         </span>(</span>downcase header</span>)))
</span>      </span>(</span>car table</span>))))
</span>  </span>;; Rest of the code
</span>  </span>)
</span></code></pre>
{</span>% </span>for watch </span>in </span>site</span>.</span>data</span>.</span>official</span>-</span>watch</span>-</span>list </span>%</span>}
</span><</span>p</span>></span>{</span>{ watch.field </span>}</span>}</span></</span>p</span>>
</span>{</span>% </span>endfor </span>%</span>}
</span></code></pre>
As an example on this, my watch list has a Public Rating</code> field and
my question is how will you access it with a dot notation? You are
welcome to inform me how but I wouldn't like how it would feel and I
am comfortable writing watch.public-rating</code> instead. Aside from that
I have nothing much to say about the naming convention and quoting the
value just in case it gets too long.</p>
>></a>
Default Properties</h2>
As a final exploration, let's see how to set the default export
options for a table. I thought setting a header property would be good
enough but according to org-set-property</code> I have to put it under a
header which I find weird since the file represents a single solitary
data set but not a big deal. To show that off from my official file:</p>
* Official Watch List
</span>   :PROPERTIES:
</span>   :TABLE_EXPORT_FILE: official-watch-list.yaml
</span>   :TABLE_EXPORT_FORMAT: fn/orgtbl-to-yaml
</span>   :END:
</span>
</span>   Yes, I have an unofficial watch list for myself
</span></code></pre>
For my use case when updating my watch list, I want to auto export
then publish the table when it is saved just like my
org-jekyll-blogger-auto-publish-on-save</code> from org-jekyll-blogger.el</a>.
It takes a few minutes to write that glue code:</p>
(</span>defun </span>fn</span>/</span>org</span>-</span>jekyll</span>-</span>blogger</span>-</span>export</span>-and-</span>publish </span>()
</span>  </span>"</span>Export a table and publish the file accordingly.</span>"
</span>  </span>(</span>interactive</span>)
</span>  </span>(</span>if </span>(</span>not </span>(</span>org</span>-</span>at</span>-</span>table</span>-</span>p</span>))
</span>      </span>(</span>if </span>(</span>not </span>(</span>called</span>-</span>interactively</span>-</span>p 'interactive</span>))
</span>          </span>(</span>message </span>"</span>No table at point to publish.</span>")
</span>        </span>(</span>error </span>"</span>Point is not at a table</span>"))
</span>    </span>(</span>org</span>-</span>table</span>-</span>export</span>)
</span>
</span>    </span>(</span>lexical</span>-let </span>((</span>export</span>-</span>file </span>(</span>org</span>-</span>entry</span>-</span>get </span>(</span>point</span>) "</span>TABLE_EXPORT_FILE</span>" </span>t</span>)))
</span>      </span>(</span>if </span>(</span>not </span>export</span>-</span>file</span>)
</span>          </span>(</span>message </span>"</span>No TABLE_EXPORT_FILE property</span>")
</span>        </span>(</span>lexical</span>-let </span>((</span>export</span>-</span>buffer </span>(</span>find</span>-</span>file</span>-</span>noselect export</span>-</span>file</span>)))
</span>          </span>(</span>with-</span>current</span>-</span>buffer export</span>-</span>buffer
</span>            </span>(</span>org</span>-</span>publish</span>-</span>current</span>-</span>file</span>))
</span>          </span>(</span>message </span>"</span>Table published.</span>"))))))
</span>
</span>(</span>defun </span>fn</span>/</span>org</span>-</span>jekyll</span>-</span>blogger</span>-</span>auto</span>-</span>publish</span>-</span>table</span>-</span>on</span>-</span>save </span>()
</span>  </span>"</span>Auto export and publish table on save.</span>"
</span>  </span>(</span>interactive</span>)
</span>  </span>(</span>add</span>-</span>hook 'after</span>-</span>save</span>-</span>hook </span>#</span>'fn/org-jekyll-blogger-export-and-publish t t</span>))
</span></code></pre>
Not the best code but it does the job and not that hard to write
although I had to peek at org-table-export</code> to determine the name of
the file. A small point in this code is using called-interactively-p</code>
which merely indicates if the containing function is called as a
command(via M-x</code> or execute-extended-command</code>) or a function(via
Elisp); this allows me to either throw an error message or an info
message depending on the context. For example, using it directly you
should get an error that you need to mark what table you should export
with the point; but if it is called by a hook, you wouldn't want it to
throw an error since there might be other hooks in play so better an
info message.</p>
>></a>
Conclusion</h2>
The two things I deliberately ignored are performance and escaping but
I pray I not see more than 10,000 visual experiences before I worry
about it. With this base, you can come up with a JSON</a> exporter from a
quick read. For me, I can continue to work with my org</code> workflow and
not worry about the export format.</p>

Fn/Log

Justifying Rust in the One Billion Rows Challenge

Wo Long: Fallen Dynasty (2023) - Short Thoughts

>></a> Wo Long: Fallen Dynasty (2023) - Short Thoughts</h2> Finished the game mainly offline with Dual Swords</strong></strong> on Hard Mode</strong></strong> and using little to no magic to focus on parrying.</p> </blockquote>

Batman: Return To Arkham - Notes and Thoughts

>></a> Batman: Return To Arkham</h2> Completed the main story, collected every Riddler trophy and beaten every challenge (predator and combat) for Arkham Asylum</strong></strong> and Arkham City</strong></strong> on hard mode</strong></strong>.</p> </blockquote>

Belle (2021) - Notes and Thoughts

Teenage Mutant Ninja Turtles: Shredder's Revenge - Notes and Thoughts

>></a> Teenage Mutant Ninja Turtles: Shredder's Revenge</h2> Finished story mode on XBox at Hard difficulty with April</p> </blockquote>

Fights in Tight Spaces - Notes and Thoughts

>></a> Fights in Tight Spaces</h2> Finished the game on XBox with Slasher, Aggressive, Grappler and Counter-Striker decks at Purist (hardest) difficulty.</p> </blockquote>

Tunic - Notes and Thoughts

>></a> Tunic</h2> Finished on Xbox with the pacifist ending</p> </blockquote>

River City Girls - Notes and Thoughts

>></a> River City Girls</h2> Played on XBox with Misako on Hard difficulty</p> </blockquote>

Stickless Half Circle Inputs

>></a> Half Circle Inputs</h2> Tested on KOF XV (Xbox)</strong></strong> with a Snackbox Micro</strong></strong>.</p> </blockquote>

Fatal Frame 5 Remastered - Quick Thoughts

No More Heroes 3 - Quick Thoughts

Outer Wilds: Echoes of the Eye - Quick Thoughts

Simple Page View Tracking with Cloudflare KV

Firefox Extension: VPN Connectivity Status

Elixir Bit - Offline (Docker) Hex Mirror

Mundaun - Notes and Thoughts

Rogue Heroes: Ruins of Tasos - Notes and Thoughts

Violet Evergarden - Notes and Thoughts

Rust Bit - Borrow Multiple Struct Fields

Darkwood - A Strong Top-Down Horror

Rust Bit - Folding A Recursive Parser

Not Tonight - Lacking Edge

Children of Morta Switch - A Subpar Experience

Katawa Shoujo - A Visual Novel Gem

No More Heroes 1 And 2 - A Good And Bad Game

Rit Journal - Diff Command

Elixir Bit - socket.io Client Websocket

Elixir Bit - Caching Docker Dependencies

Rit Journal - Myers Diff

Elixir Bit - ISO 8601 Duration

A Mortician's Tale PC - Respecting The Dead

Butterfly Soup PC - A Chill Experience

>></a> Butterfly Soup</h2> A visual novel about gay asian girls playing baseball and falling in love.</p> </blockquote>

Rit Journal - Initial Entry

Rust Bit - A Use For Deref Trait

About Me

Outer Wilds - On Acceptance