Showing posts with label personal_IT. Show all posts
Showing posts with label personal_IT. Show all posts

03 December 2023

Different premises, different conclusions

As will be apparent to anyone reading this blog, bird photography is a Thing for me. This is an activity ordinarily pursued with a long lens; my usual setup totals about 5 pounds, or another half pound if I add a teleconverter. I'm used to toting it, but it does get heavy after a while, and I might not want to pack it if I were going on a trip that wasn't targeted at birding. So, I was interested in finding a possible alternate optic that might be usable even if not quite as good. I was intrigued to find that I could buy a used Canon 70-300 DO lens in excellent condition for less than $350 USD. This lens is notably shorter than others in its focal length range, thanks to its use of Diffractive Optics (DO) technology. It was expensive when it was introduced in 2004, and opinions about its sharpness and contrast for an expensive lens have been mixed

In 2004, digital camera technology wasn't nearly as advanced as it was today. A 6 or 8 megapixel sensor was competitive in consumer-grade DSLRs, which limited the number of pixels that could be cropped away to frame a small subject without visibly diminishing resolution. Sensors were also less capable of operating at high ISOs without becoming excessively noisy. Since bird photography often requires fast shutter speeds, high ISO usage is often necessary.

Today's sensors are much larger and can reach into higher ISO territory before becoming intolerably noisy. Digital post-processing technology provides impressive denoising capability, and AI software products like Topaz Photo AI can often extract or reconstruct image detail that's otherwise lost to the naked eye. One relies on the AI to derive and reflect detail that's actually present in the scene being photographed rather than hallucinating artifacts that aren't, but that's a larger question that I won't attempt to reconcile here. Rather than considering the output from a camera as necessarily comprising an end result in itself, it can now be provided as the input to constructive post-processing. The test of a camera and lens combination, therefore, need not be the appearance of the photos that it outputs directly, but rather whether the information contained in them (most likely, in their RAW files) is sufficient and suitable as a basis to create even better results digitally.

When I looked at a trial image taken with the DO lens (significantly cropped, as most of my bird photos are), it seemed rather noisy as it came from the camera, but exposure at ISO 3200 on a dim day could explain at least some of that.

More broadly, I wasn't seeing as much detail in the bird and its surroundings as I'd like for my collection.

Sharpening and denoising in Darktable provided some improvement in another version.

I still hoped for better, though, and thought that the Topaz software provided an impressive result.

I think the DO lens will be a useful complement to my current setup, but would probably have been disappointed if I weren't able to apply post-processing technology to the images it collects.

24 September 2023

Chromebook and Linux for Travel and Photography

I enjoy making the most of lower-end devices and appreciate economy, portability, and ruggedness. For a travel computer, I want something that I can throw into a bag without much worry about cushioning, that I wouldn’t have spent so much on that I’d feel far out of pocket if it were lost, and that I might even be able to unfold on an airplane seat. Long battery life is another desirable attribute. So, I bought a new 11” Chromebook, of the sort designed to be handled roughly by school kids, a Dell Chrome 3110 2-in-1. It has 64GB of storage and 8GB of RAM, with a Celeron N4500 processor. (Note: there’s also a 32GB/4GB version of the same model, which seems to have better availability. I didn’t want or select that one.) The keyboard has a good feel and a comfortable amount of travel. As a significant extra bonus for me, it includes native Linux support, which fits many of my preferences and application needs.

I attempted some photography and photo processing through it before traveling:

  1. Take RAW photos in rather dim weather.
  2. Extract memory card, plug into Chromebook via USB card reader.
  3. Select-all of RAW files on memory card, copy to folder on Google Drive. I was interested to see that it copies many files in parallel with individual completion indicators, maximizing use of available bandwidth. I subsequently found that it’s not necessary to do the manual select-all, and can instead just copy a folder. Nonetheless, it still takes a while to push many RAW files through the 12Mb upload speed of a cable Internet connection.
  4. Find that Google Photos doesn't support Canon CR3 RAW format.
  5. Enable the Chromebook’s Linux development environment, which I believe is included on at least most fairly recent Chromebooks, obtaining a shell in a Debian instance.
  6. In a spirit of "why not?", install Darktable as a Linux app on the Chromebook, via Flatpak to get a more current version than the older one in the Debian repository. Icons for Linux GUI apps appear alongside other Chrome apps in a folder named "Linux apps" so you can launch them alongside anything else you run in a Chromebook like the native apps or Android apps. Clever.
  7. Launch Darktable and import photos from Google Drive. This creates references only, not retaining the raw files themselves within limited Chromebook storage. From the ChromeOS environment, it’s necessary to use a menu option to grant folder access to Linux. Darktable’s sidecar files appear alongside in the Google Drive folder and can be accessed later to preserve edits made when mobile. Some patience is required as new photos are imported and thumbnails are generated.
  8. Raw-develop and edit with Darktable, as if on a more powerful computer though slower and obviously with a tiny screen; raw development takes a few seconds on a 32 MP RAW file, but works.
  9. Export generated jpg back into Google Drive.

I'm impressed that this is possible and, after a trip where I shot about 1400 photos, found that it offered a useful travel capability for archiving and for developing, editing, and posting a few shots while on the move like this Sandhill Crane:



I imported daily folders with a few hunderd RAW images into Google Drive and watched them synchronize through hotel WiFi into the cloud in under an hour, but the actual elapsed time depends critically on tha available upload data rate. (Warning about hotel WiFi, BTW: if a Chromebook is freshly started, it may have trouble launching a new Linux container instance until normal Internet access is available; a captive hotel login page may not suffice. This didn’t seem to be a concern if the Chromebook already had a live container and was just being resumed after suspension.) If the Chromebook went idle, the sync paused, but it resumed in place once the machine was awake again. It was notable how Google's integration between the Drive file system and the Linux container "just worked", and that the Darktable instance was able to find a folder there via its chooser.

A few things I’d like to improve for a next round:

  • Automate extraction of images from a camera memory card into folders organized by days, keeping accumulated images on the card but also copying each into Google Drive once and only once. It may also be useful to explore camera options for folder placement and photo numbering to better facilitate this.
  • Figure out how to keep Google Drive’s cloud sync uploading actively even when the Chromebook is otherwise idle. (There may be an obvious configuration option somewhere that does this, but I haven’t yet stumbled on it.)

13 September 2022

Update day! Whee!

Ubuntu's 22.04.1 LTS release came out fairly recently, and I had it on my to-do list to get around to updating some components of my personal digital empire once it did. I'd already been using 22.04 on my desktop for several months, to general satisfaction and particular appreciation of its having included a new enough version of Darktable to handle the CR3 raw format that one of my cameras produces.

So, to the updating. I have a DigitalOcean droplet that runs the Nextcloud snap. I like having a small bit of cloud that I can manage myself, primarily to share files among devices and sometimes with others. I use memory cards with higher storage capacity than the minimal droplet's 25GB, but it serves its purpose. sudo do-release-upgrade worked fine. One system checked off.

I also have a home server, which also hosts a Nextcloud alongside other services, for larger-capacity sharing within my LAN. Here, the Nextcloud instance hadn't auto-updated like the droplet's snap, and I (bad me!) hadn't been attentive to updating its versions while the service continued to "just work". After the OS upgrade, I found that the Nextcloud-in-place required a PHP version earlier than the 8.1 included with Ubuntu 22.04, so couldn't generate display pages. There's a third-party repository that has older PHP releases, but I wasn't quickly able to line up the right set of configuration for the old Nextcloud to make use of it, and it was Update Day anyway! After three successive updates to Nextcloud to successive versions, I got it current and able to find and use PHP 8.1. My screen sync icon winked with approval. 

Lesson: be aware of dependencies, and do upgrades when you have time to deal with surprises.  

05 September 2021

Virtual Aviary

I set my virtual aviary site up a couple of years ago, but hadn't set up a pointer to it here until now. It started as a learning exercise in Python and Django, but I've kept it live and growing as a vehicle to host my collection of bird photographs. It's been satisfying and informative to structure and apply a database to organize the images based on metadata, and to explore tradeoffs between appearance and functionality. I'd never have gotten to its current count of 231 photographed species in New England without the help and company of other birders, so am appreciative for that. 

12 June 2019

My birds in the cloud

As I mentioned a few posts ago, I recently embraced my inner geek by building a small site with Python and Django, in a DigitalOcean droplet. The result may be a bit extreme ('ya think?) for its purpose of organizing and displaying a personal collection of bird photographs, but I've been enjoying it as it's been running for a few weeks. I like being able to use the uploader to add new pictures and species into the database and display, and the ability to specify search terms and ordering. Django makes such facilities easy to implement (along with the ever-popular automated species count!) but I'm still working with the challenge of layering multiple search criteria (e.g., month and name) in a way that presents an intuitive UI. I've written up some tech-oriented notes about the site's technology and development for anyone who may be curious, but the photos themselves may have more general interest.

21 April 2019

I see you knocking but don't want you in

A few days back, I set up a "droplet" at DigitalOcean to host a Django application that I'd been building, and so I now have a small net-facing Ubuntu VM there. As a security person, I've been, er, interested to see just how interesting my site has quickly become to, er, unexpected visitors. Looking at its first-ever auth.log, it went active at:

Apr 16 15:33:23  systemd-logind[1385]: 
Watching system buttons on /dev/input/event0 (Power Button)

The sshd logged its first preauth disconnect at 15:38:27 (just over 5 minutes later), from an IP address that whois resolved to country code IR. Since I didn't have an associated domain registered at this time, I assume that this was a random address scan.

I started an Apache server about an hour later, at 16:47. Following some of my own testing (and a domain name registration), its first unexpected visit came at 17:40 in the form of a POST from an IP address in St. Petersburg, RU.

I can see that my droplet's sshd and apache have been busy rejecting varied streams of "knocks" since, and am applying best practices of firewalling unneeded ports and disabling passworded access to ssh. Still, I've been surprised at just how quickly and broadly my site was discovered. If more of my prior experience had fallen on the operational response vs. architectural development side of security, maybe I'd be less surprised. Anyway, a valuable learning experience and reminder. Stay safe!

29 March 2019

https redirects: a learning experience

I had maintained an additional blog besides this one, emphasizing aspects of security technology. I hadn't posted there in a while, and decided to consolidate its content here alongside other posts. Since it was a security-oriented blog after all, it had seemed like an obvious Right Thing to set up the options so that it was accessed (or at least accessible) via an https:// url.

I closed the account on the old blog's hosting provider, and wanted to redirect any visitors to a landing page that would inform them of the change and point them at the blog you're now reading. I went to my domain registrar's admin UI, and had no problem setting up such a redirect for the http:// form of the old blog's url. The plot thickened for the https:// form, though. Reasonably enough, you have to have a certificate corresponding to a domain (and, of course, the ability to wield its corresponding private key) in order to deliver valid, authenticated content for that domain via https. And, an https redirect is a (small, but significant) example of such content; if you're relying on the security https is designed to provide, you wouldn't want an attacker without the appropriate key to be able to mislead by redirecting you elsewhere.

I could probably have arranged to get a certificate enabling my domain registrar to issue a valid https redirect to my domain's landing page, but that seemed like a lot of work just to support redirection rather than more comprehensive hosting. It was easier to solve the problem by pointing the domain's name servers to where I'd put the landing page, at a provider that was equipped to serve that page whether accessed via http:// or https://. (Thanks again, FastMail.)

30 January 2019

MD5: nice to see you here, old friend!

I recently assembled a new desktop computer system. In the course of moving my data to it from its predecessor, I managed to corrupt the database that my photo organizer (Shotwell) uses to manage my photo collection. I rebuilt a new database from the photos themselves, but this lost metadata like comments and edits that I'd applied. I hoped, though, that I'd be able to recover that information later from the older versions of the database tables. I managed to do this, thanks in part to one, er, key element in the database structure.

In Shotwell's database, each photo has a row in the PhotoTable, with many columns containing information about it. There's a unique ID for each photo, but the IDs generated as photos were imported into the earlier database couldn't be assumed to be the same as when I reimported them into the new database. It would clearly be a Bad Thing to apply the tags for photo #457 in the old database to photo #457 in the new database. What to do?

Looking at the PhotoTable columns, I noticed that each photo had an entry for an MD5 hash of the image. Hash functions are great and useful things. It's unlikely (and, I mean, highly, probabilistically, unlikely) that I'm going to encounter two different images in my collection that yield the same MD5 value. (Even though MD5 isn't recommended today for security-relevant applications, it's still doing its job here in distinguishing among image files that came out of my cameras, which haven't generally acted as hostile attackers.) I expect that Shotwell's code uses the stored MD5 as a quick and effective means to determine whether or not a photo has already been imported into its database. When I saw the MD5 column in the table, I realized that it also provided me with a means to find the correspondence between photo entries and their IDs in the old table with their entries in the new table. Thusly armed, SQL of this form followed:

REPLACE INTO PhotoTable ( named-columns )
SELECT named-columns
FROM old-PhotoTable src
INNER JOIN PhotoTable dest ON src.md5 == dest.md5

which took less than a second to replace corresponding metadata into a table representing about 23,000 photos. I restarted Shotwell with the resulting table, and found my edits accurately restored. I was pleased to have been able to accomplish this. I was glad to have been using an open source organizer with an accessible and documented database representation, and emerged with refreshed respect for the power and value of hash functions.

17 April 2017

Intercontinental smash and grab

A couple of weeks ago, one of my credit card numbers was compromised. This is all too routine these days, and the credit card company detected and removed the suspicious transactions quickly. What I found notable, however, was the fact that multiple charges were attempted in different currencies in rapid succession – an Australian on-line merchant, German Amazon, and so on, alongside fast food vendors in New York, and that at least some of them hit almost simultaneously. As I spoke on an incoming call from the card vendor’s fraud department, I could see successive charge notifications overwriting one another on my cellphone’s display. I’d think that the fact of closely-spaced transaction attempts across numerous countries would be likely to raise risk flags – even though less definitively today than when cards were ordinarily presented across counters by hand – but perhaps fraudsters may still find that a dispersed attack improves the expected value that’s obtainable in the narrowing time window before countermeasures can respond. I don’t know if and where there may be variable delays for transactions to propagate through different payment systems, but maybe this is also a factor.

12 September 2015

The computer must be right. What's wrong with me today?

When I ride my bike, it's most usually on one of a few rail trails with which I'm quite familiar. I have a cyclocomputer (are they still called that?), a watch-sized device which connects to a magnetic sensor to observe how quickly the front wheel is spinning and over how long a time, and hence to determine how fast I'm moving and how far I've traveled. I like being able to use it to track my progress. Earlier this year, though, I found myself pedaling at what seemed like a usual pace, but watching numbers indicating that I was going noticeably slower than usual. The display had to be right, though - didn't it? - so had I fallen suddenly and unexpectedly away from shape?  Why had it become harder to maintain a normal speed? I found it concerning, until I finished my ride and saw that what I'd always thought of as 18 miles (and which a smartphone GPS and topo map still did) was shown instead as about 16.3 miles, and that my elapsed time for the route wasn't abnormal for me. Further investigation showed that the cyclocomputer's setting for tire circumference had somehow gotten reset to its default, which was smaller than that of my actual tire. Once I readjusted that, simple division raised my indicated speed by about 10% the next time I rode. I felt better about both my pace and my device once the accidental bias was gone. I wasn't actually traveling faster, but a more reassuring measurement made it feel that way.

16 May 2015

Longest. Progress Bar. Ever?

I've been using a MacBook Pro for a few years, which I’ve been quite happy with except for some annoying behavior when resuming from hibernation. Being Security Guy by habit, I’ve unsurprisingly been using FileVault disc encryption; after typing a password to unlock the volume, I’d often been waiting for a minute or so, sometimes spanning a screen blank and reset button tap, before being presented with another password entry prompt and the chance to enter a useful system. (Deceptive UI indication aside: please don’t provide a blinking cursor in a text box before it’s ready to accept input!) Accumulated impatience having eventually spurred me into action, a web crawl yielded references like this article, suggesting that OS X Yosemite was intolerant of at least some earlier FileVault configurations and recommending decryption and reencryption. I started the decryption on a disk with about 140 Gb in use; though it changed before I got to screenshot the display, the progress bar started out with a caption estimating 43 days to complete the task. That proved conservatively pessimistic, with the operation taking about 10 hours in the actual event. I’m used to symmetric crypto being quick, often invisible inline at human scale; 43 days read more like what I’d expect if a fairly substantial exhaustive search had been required.

25 April 2015

My spelling corrector knows me too well

I was trying to text that "We're having lunch" and it proposed that I meant to say "We're having Linux" instead.  I'm guessing that I fat-fingered an "i" instead of the adjacent "u" on the keyboard and it took it from there, but I could easily be spending some time doing either.

08 February 2015

Counting nostalgically ... in octal

I hadn't touched homebuilt electronics extensively since the 1970s, when I'd built a microprogrammed central processor named Irving among other projects, and was intrigued with the idea of wiring together some newer components.  I thought a Raspberry Pi kit would be a good basis for experiments, and am now exploring its possibilities. I was pleased to be able to hook it up to a numeric display module that I'd had in a box for nearly 40 years, to see the display illuminate, patch 8 bits' worth of output connections, and write and download the Python code to make its numbers count.  And, yes, 377 wraps around to 000; I waited to watch that.  There's nothing like a table sprawled with wires and pieces that, collectively, do something.


04 March 2013

Desktop wallpaper (blotter paper?)

I'm clearly not the only person who spends a lot of time looking at computer screens and takes the trouble to personalize them with some sort of image from outside the computer domain.  I usually find myself selecting some natural scene, like this one that's current as of today. Based on an unsystematic over-shoulder survey at offices and conferences, it seems that family members are also popular.  Though a small gesture, the result seems more comforting than one might expect, perhaps as a reminder of being anchored into the physical world.  Even when most of the image is obscured by an email client.

03 February 2013

On computers and cameras as containers

In a traditional cold-season home activity, I've spent a while recently setting up a "new" server system on my home network. I'm quoting "new" because it's a repurposed desktop PC that's about a decade old, retrofitted with Linux and more disk capacity than its designers probably ever envisioned as being connected to its SATA ports.  It's working well for now, though. It's entirely possible that its aging power supply may die or that its motherboard may fail. Power supplies aren't hard to replace, but finding appropriately sized and cabled motherboards might be harder.  The important thing, though, is my data; if the surrounding hardware fails, I can remove the new disks, read them elsewhere, or build them into a new system.  The computer isn't a destination in itself; rather, its purpose in my IT life is to act as a means to get data on and off of disks.  The "peripheral" disk device is the central component, and the "central" CPU is present and running in order to support it.  Similarly, the purpose of a digital camera is to translate views into bits that can be stored on a memory card and processed, and that become independent of the camera that collected them. A closed camera whose images couldn't be exported or viewed except through its own screen wouldn't be nearly as interesting or useful. The cheap and ever-more-capacious storage modules can be read and written elsewhere; their information is what matters, not the surrounding systems that load the information into them.

27 July 2012

Over three orders of magnitude!

I don't think it's common for the price of a given good to vary by more than three orders of magnitude depending on its location and circumstances, but that seems to be the remarkable situation with cellular data, so I'll remark on it. In the US, I'm using a plan which seems to be fairly typical, and that offers 2GB of data per month for $25.  That's $0.0125 (1.25 cents) per megabyte, and is an allowance that I've never used as much as half of; if I track usage, it's more a matter of curiosity than of financial concern.

My US carrier, not atypically, posts pay-per-use rates for international roaming that range up to $19.97/MB, which is almost 1600 times higher than my domestic cost.  It's practical to communicate in this price bracket with occasional access to text-only email (which may have been the primary use of mobile data when the rate was set, but I don't know the tariff history), but is much harder to manage economically in an era of rich graphics and autonomous mobile apps "phoning home" to their servers.  There have been numerous horror stories from unwary travelers who have returned from overseas trips with smartphones to find that their data services worked smoothly but the the resulting charges dwarfed their air fares.

It's now possible to get international packages with terms that have improved significantly over recent years, allowing cheaper roaming to many countries (e.g., 120 MB for $30, or $0.25/MB, 20 times my domestic plan rate and about 80 times cheaper than the pay-per-use rate), but these don't operate everywhere.  (Tip: if visiting an excluded country, find Wi-Fi, use SMS, or make the most of a rare opportunity to disconnect.) If one of these plans is active, it moves the criteria for mobile data usage from something like "only in case of emergency" or "only if price is no object" to "apply thoughtfully and carefully, where and when useful", but not as far as the domestic "use freely and don't worry much about measurement" category.  I'm able and willing to measure usage and act accordingly, so this works for me, but it can seem awkward and unfamiliar if you're accustomed to data usage being effectively free within an existing plan.

05 May 2012

On travel, packing, and writing

Looking back over posts here, I see that a lot concern travel in some form, which makes me think to inaugurate a tag on this theme.  I'm a fairly occasional traveler, and each trip has its own drumbeat of anticipatory preparation, generally culminating in early arrival at a departure terminal, waiting there to leave.  On the preparation side, it seems that the notable voyagers of years past assembled rolling libraries of attire, their capacious trunks sufficient for any formal occasion.  On my side today, clothing choices are usually simple, casual footnotes relative to the technology items (and, be sure not to forget, their chargers) that share the cubic inches available in carry-on bags.  Oh, and maybe a snack or two, since it's usually unclear whether or when food might be available.  It's always a compromise to compress one's personal environment into portable form, and departure or absence from the familiar promotes introspection. You bring yourself on any trip you take, after all, and your mind stays powered without plug adaptors.  I don't think the volume of travel essays is coincidental. 

19 October 2011

Why should printing be hard?

It's not always irremediably last-millennium to render something from bits onto paper, at least in my opinion. To carry some information for one of those increasingly rare moments offline, perhaps.  Or, to put on the wall.  Perhaps even to read and mark up with different ergonomics than on a screen.  Increasingly often, though, I find that printing just doesn't work, or doesn't work usefully.  Several times, I've printed driving directions from an online service, only to find that the thoughtfully provided map (fine onscreen) emerges overprinted with a block of solid color, as if the route has been redacted or censored - not the most useful result for navigation.  Browsers, web pages with frames, and printers still don't seem to coexist constructively, even after years. Last night, we tried to print a few 4x6 photos.  Two printers didn't feed the paper correctly, but we finally extracted a few images with unexpected colors and irregular margins.  Other times, documents fail or disappear silently when crossing the hostile waters of a local network, never to be rendered. File storage (fortunately) doesn't seem to have as many comparable problems.

Have I motivated this rant yet?  I'm wondering now about why printing should so often be problematic.  Device flaws are part of the problem, but I think it's broader than that. Is it that paper is old, untrendy, and that technical trend setters aren't generally as focused on integrating well with it?  If you're always at a screen, does this define your assumption for what others will want to do as well, and hence your priority for what's most important?

11 December 2010

Photos and metadata

For anyone who may be unfamiliar, metadata are data that describe other data. A digital camera picture presents a good example: the data represents the image itself, and metadata include other information like when the picture was taken and with what exposure. I've recently been putting a number of slides through a digitizer, which has reminded me that only limited and coarse metadata came along when film was developed. There was usually a stamp indicating the month when the film was processed, which clearly demonstrates that a picture wasn't taken later but doesn't say anything about how much time elapsed between the exposure and its development. There was also a sequence number designating a picture's position in a roll, but it may not always be obvious which roll it belonged to if several rolls were processed at the same time. (If my samples are indicative, different processing labs provided stamped annotations that differed in detail.) It shouldn't usually take much guesswork or puzzle solving to assemble a sequence, but having to do so contrasts with what newer technology provides automatically. As for exact time data, it's not normally available, except if there's an associated handwritten note, a clock in view, or (interesting special case) the image depicts a specifically noted event like an eclipse.

19 November 2010

I like backups!

OK, I realize that this may be eccentric.  I find backup operations satisfying, at least as they're completed, checking off an item from an unwritten to-do list.  Maybe it's residue from an earlier era, when disks crashed more often, but I don't feel that I'm really done with a computer-based activity until I'm confident that there's another copy of the result somewhere. Whatever that activity might have been, it was or seemed important at least at the time, not something to be lost or repeated.  The vast majority of data copies I've made have fortunately lived out their days in write-only obscurity, often overwritten by their successors, but the occasional exceptions validate their existence.  If I experience some digital apocalypse, at least I can expect to be able to reassemble most of my accumulated bits afterwards.