Virtual Aviary

I set my virtual aviary site up a couple of years ago, but hadn't set up a pointer to it here until now. It started as a learning exercise in Python and Django, but I've kept it live and growing as a vehicle to host my collection of bird photographs. It's been satisfying and informative to structure and apply a database to organize the images based on metadata, and to explore tradeoffs between appearance and functionality. I'd never have gotten to its current count of 231 photographed species in New England without the help and company of other birders, so am appreciative for that. 

Notes on a personal software development cycle

I'm now taking and enjoying a MOOC that combines lectures and coding projects. I find these to be engaging learning experiences, providing good mental exercise. There's nothing like the sensation of "gee, I've now made this work". That said, I've watched myself do this many times, and observe some patterns. When approaching what appears to be a complex problem, I'll first tend to procrastinate, waiting for just "the right" combination of alert and undistracted time for concentration. Sometimes, I'll get to the point of first digesting the problem statement and contemplating its possible solution, and will then initiate another procrastination cycle. Eventually, I'll get into coding the project, and will become engrossed in populating the needed modules. I'll get to the point of a test run, and will try to deconstruct errors via inspection and by inserting debugging print statements. Sometimes, discussion forums will offer useful insight or pointers. I'll itch to complete the process. Finally, I'll (at least usually) get a set of working pieces together and obtain the desired result. It feels like a satisfying milestone, even as I'm aware that the MOOC authors have crafted the materials with the expectation that their students should be able to reach this point. I'll reflect on renewed appreciation of the fact that such resources are available. Thanks, 2020 Internet!

Increasing cloudiness

I've been a long-time Dropbox user, mostly for making files conveniently movable and accessible to myself and family members across Linux, MacOS, iOS, and (less frequently) Windows platforms. I wanted something based outside my local LAN, so that I could synchronize when away from home. Dropbox's free service seems to be becoming more restrictive, notably in terms of Linux file system support and with regard to numbers of synchronized devices per account. The lowest paid tier provides 2TB of storage, which is far more than the few GB I need for my usage model, where I don't use the cloud storage as a long-term repository for massive amounts of data. Apple's iCloud offers a reasonably-priced package with 200GB, but lacks a native Linux client. So, I looked into other alternatives, and quickly gravitated to a quasi-DIY approach.

I've had good recent experience with DigitalOcean, as I discussed a few months back. It was easy to create another small droplet hosting a snap of Nextcloud and, "viola", I've instantiated my own cloud storage facility, with available clients for all of my platforms and enough storage within a minimal 25GB droplet to satisfy my current usage requirements. Even with a new domain name registration to address it, the additional $5/month droplet cost is fine to satisfy my purposes, and I like having my own control over it.

My birds in the cloud

As I mentioned a few posts ago, I recently embraced my inner geek by building a small site with Python and Django, in a DigitalOcean droplet. The result may be a bit extreme ('ya think?) for its purpose of organizing and displaying a personal collection of bird photographs, but I've been enjoying it as it's been running for a few weeks. I like being able to use the uploader to add new pictures and species into the database and display, and the ability to specify search terms and ordering. Django makes such facilities easy to implement (along with the ever-popular automated species count!) but I'm still working with the challenge of layering multiple search criteria (e.g., month and name) in a way that presents an intuitive UI. I've written up some tech-oriented notes about the site's technology and development for anyone who may be curious, but the photos themselves may have more general interest.

https redirects: a learning experience

I had maintained an additional blog besides this one, emphasizing aspects of security technology. I hadn't posted there in a while, and decided to consolidate its content here alongside other posts. Since it was a security-oriented blog after all, it had seemed like an obvious Right Thing to set up the options so that it was accessed (or at least accessible) via an https:// url.

I closed the account on the old blog's hosting provider, and wanted to redirect any visitors to a landing page that would inform them of the change and point them at the blog you're now reading. I went to my domain registrar's admin UI, and had no problem setting up such a redirect for the http:// form of the old blog's url. The plot thickened for the https:// form, though. Reasonably enough, you have to have a certificate corresponding to a domain (and, of course, the ability to wield its corresponding private key) in order to deliver valid, authenticated content for that domain via https. And, an https redirect is a (small, but significant) example of such content; if you're relying on the security https is designed to provide, you wouldn't want an attacker without the appropriate key to be able to mislead by redirecting you elsewhere.

I could probably have arranged to get a certificate enabling my domain registrar to issue a valid https redirect to my domain's landing page, but that seemed like a lot of work just to support redirection rather than more comprehensive hosting. It was easier to solve the problem by pointing the domain's name servers to where I'd put the landing page, at a provider that was equipped to serve that page whether accessed via http:// or https://. (Thanks again, FastMail.)

MD5: nice to see you here, old friend!

I recently assembled a new desktop computer system. In the course of moving my data to it from its predecessor, I managed to corrupt the database that my photo organizer (Shotwell) uses to manage my photo collection. I rebuilt a new database from the photos themselves, but this lost metadata like comments and edits that I'd applied. I hoped, though, that I'd be able to recover that information later from the older versions of the database tables. I managed to do this, thanks in part to one, er, key element in the database structure.

In Shotwell's database, each photo has a row in the PhotoTable, with many columns containing information about it. There's a unique ID for each photo, but the IDs generated as photos were imported into the earlier database couldn't be assumed to be the same as when I reimported them into the new database. It would clearly be a Bad Thing to apply the tags for photo #457 in the old database to photo #457 in the new database. What to do?

Looking at the PhotoTable columns, I noticed that each photo had an entry for an MD5 hash of the image. Hash functions are great and useful things. It's unlikely (and, I mean, highly, probabilistically, unlikely) that I'm going to encounter two different images in my collection that yield the same MD5 value. (Even though MD5 isn't recommended today for security-relevant applications, it's still doing its job here in distinguishing among image files that came out of my cameras, which haven't generally acted as hostile attackers.) I expect that Shotwell's code uses the stored MD5 as a quick and effective means to determine whether or not a photo has already been imported into its database. When I saw the MD5 column in the table, I realized that it also provided me with a means to find the correspondence between photo entries and their IDs in the old table with their entries in the new table. Thusly armed, SQL of this form followed:

REPLACE INTO PhotoTable ( named-columns )
SELECT named-columns
FROM old-PhotoTable src
INNER JOIN PhotoTable dest ON src.md5 == dest.md5

which took less than a second to replace corresponding metadata into a table representing about 23,000 photos. I restarted Shotwell with the resulting table, and found my edits accurately restored. I was pleased to have been able to accomplish this. I was glad to have been using an open source organizer with an accessible and documented database representation, and emerged with refreshed respect for the power and value of hash functions.

Cold Days for Code Monkeying

I've taken and enjoyed a number of online courses in technical topics like programming and web technology, such as this example which I'm doing now. For no cost or a nominal fee, MOOCs often offer valuable means to refresh and update skills, providing opportunities to pursue engaging projects and inspiration for others. There's nothing like the satisfaction of building something and making it work. And, there's no better season for such indoor activities than wind-chilly winter days, so I've been doing that lately. Having been through this experience a number of times now, I find that the usual (or at least my usual) flow tends to fall into a sequence of four phases:

1. The "how will I ever assemble this project" phase, associated with pondering, hesitation, and sometimes procrastination.
2. The "OK, I'll get started" phase, setting up prerequisites and frameworks and assembling components to the reassuring point where a basic code skeleton operates.
3. The "Check the boxes" phase, where I go through the project requirements and add support for them one or a few at a time. This usually breaks down nicely into a series of coding and testing sessions, each of which adds a few features.
4. The "Cleanup and Embellish" phase, where I get rid of false starts accumulated during the prior phases and customize the overall result to add features that seem intriguing or add further capabilities.

After these steps, I'll submit the code, but it's not as if its grading is a primary goal. The point is the learning, and the enjoyment of the process that gets there.