[Technology] Data Debt: part 3

While in school, and for a period after, I neglected my systems of organization.  I continued to accumulate information, but without spending effort to manage it.  This has resulted in a large data debt. 

I have
  • years of e-mail, chat and text logs in various formats that I want to de-duplicate and consolidate.
  • thousands of screenshots and saved images of memes and wallpapers and artwork from some of my favourite video games and media sitting in massive folders.
  • years of photos grouped just by date taken (surprisingly useful) with no indication of people present (actually not a huge problem given Google Photos)
  • thousands of To Do items spread across various formats and systems, that have no priority, they just end up in an endless queue that only ever grows.
  • old websites, blogs, accounts that are redundant or obsolete.
  • projects that need starting, video games that need completing.
Part of my problem is hope.  I have hope.  Hope that this can all be managed.  Hope because I have the skills necessary.

Part of my problem is that I am late.  There are good opportunities, especially at the time of transitioning from one process to another, or at the time I acquire a record, to act then and keep-up.  They were missed.  And now I have debt

I just need the time.  So here I am, stealing an hour here, an hour there, out of my busy days, not dealing with problems of today, but dealing with a self-created problem of years past, for a goal that, like my dad's, may be pointless.

How many photos do I ever review?
How many e-mails will I ever re-read?
How many ToDo items do I ever expect to complete?

My alternative is recency.  Focus on what is recent and applicable to me today.  But I know I can do it, and I know that, like my father sifting through his decades of paper data, there are gems to be found.  Memories to recall.  People that I do not want to lose as the unfrequented connections in my brain fade to make room for more recent experience.

I have created a large data debt, one that might be manageable with the programming skills I have, and maybe more manageable as AI continues to improve.  But does it matter?  Probably not.  Just my vanity.

[Technology] Data Debt: part 2

As just noted, my father managed to accumulate quite a bit of information in physical form that he now spends time sorting through and discarding what has no value after all, which is most of it.

I think about that in relation to the data I keep, mostly digital.  Files upon files upon files.  Photos upon photos.  A vain effort to preserve every text message and e-mail forever.  A vain war against data loss, against entropy, against chaos.  Vain, my vanity.

I am not a fan of the paradigm of search first.  I love being able to meaningfully search my information.  I do not believe, at present, or perhaps ever, that search will be capable of being able to completely and comprehensively report back to me the information I seek.  I can only search for what I know exists.  Artificial intelligence has made search much more useful than ever before; e.g. I can search for "cake" in Google Photos and I get hundreds of photos, mostly actually of cake!  But I also know many more photos that I have taken of cake are missing, and there are photos not of cake present.  Precision, accuracy and recall are not at 100%, yet, and given the indefinite nature of some definitions, can they ever be.

Organize.  A dead motto.  Another vanity project.  Just as I do not expect search to be able to accurately identify all of the most relevant results for me, I do not believe that I can adequately and exhaustively categorize all of my information in a complete way.

But between the two, I still prefer organizing over searching.  I still prefer a reasonable hierarchy of folders and some futile "tags".  It scales a bit better.  Perhaps I cannot granularly categorize every record I have, but I can at least definitely categorize them into broad groups that I can add specificity to as needed.

What is the major obstacle that I have?  Volume of information.  I collect too much information.  And I am generally unwilling to let any detail go.  Storage is cheap.  I have a 1TB SSD that I purchased for 80CAD, and it stores tens of thousands of photos and millions of lines of text from the past decade, in a small block that can fit into the palm of my hand.

And my flexible approach to organizing it all works fine... if I can keep up.  But my data debt grows faster than I can spend my daily available time to manage it.

[Technology] Data Debt: part 1

Technical debt.  Time debt. 

My father is a bit older now, and he has been going through his Stuff.  Not just big physical Stuff like objects and thingy things.  But also informational Stuff, like receipts, letters, newspaper clippings, magazines, mileage notebooks.  Stuff he read or took notes on over the past several decades that have accumulated into quite a many boxes (formerly of shoes)or Piles or Stacks.  Sometimes, bags.

He's looking at all of this informational Stuff, and wondering, what was the point?  He definitely found value in some of it.  One of his joys in the past few years has been surfacing a memory of some event, of something he read or did, and then searching through his many records to find the Evidence of it.  I can understand the thrill.  Finding that newspaper clipping of something he did in our small town, once, or a letter from a friend from 2003, and hey, maybe Richard can find the phone number of the sender now, and hey, I did, and now my dad has renewed contact after a decade and a 1/2.

But there's a lot that he will never need, and never really did need in the end.  He has been meticulous in recording how many kilometres he has driven between gas refills, how much gas he added, and how much it cost.  It has given him a good sense of his spending in that regard, and of his cars performance.  It is all jotted in tiny notebooks.  Nothing digital.  Nothing that he can load easily into a spreadsheet now, and review larger trends.  And, honestly, numbers that he has not needed for three decades.

I love this history that he has collected, these peeks into his life and mind and priorities over the past several decades.  I do not get to enjoy them so directly, though.  I am reluctant to dig through his Stuff, it feels like an invasion of privacy.  He is shredding or recycling stuff now that he does not expect to need after all, which is most of it.  Saving information for rainy days that mostly never came.

I find it interesting that his approach is to review it all.  It definitely has had merit.  He finds random points of interest and it creates conversation topics for us.  It used to be something he could show his friends at the coffee shop, back when coffee shops were places where you could sit with friends.  But there is so much, and so much he finds is of no value to him.  I am pretty sure this year he will give up in the face of the sheer volume of information he has collected in the past several decades and just say good bye to it all.  It was never that useful anyway.

[Microblog] late

Late at night, cool air after a muggy day.  Some residual moisture in the air.  Darkness and stars.  Lilacs on a breeze.  That's me.


[General] She-Ra and the Princesses of Power

I believe that She-Ra may officially be my favourite television series to date.  It's hard to say that, as some media fluctuate a lot over time, or have different favourites that meet different needs.  But sometimes it's clear.  Scott Pilgrim v.s. the World is clearly my favourite film since I was first taken to see it in theatres.  Breath of Fire II has been my favourite video game since elementary school. 

Some past favourite televisions shows or candidates have included Teenage Mutant Ninja Turtles, Sailor Moon, the Vision of Escaflowne, (maybe even Last Exile?), Buffy the Vampire Slayer, Gilmore Girls,  Star Trek: The Next Generation, Orphan Black, and (very recently) I am not okay with this.  Some of them may still be favourites within certain specific categories.  I'm probably missing a dozen series right now that have been formative but whatever reason escape my memory at this instant.

It's also hard to say, as there's a strong recency effect.  Often I'll see a movie and be in awe of how perfect or amazing it was, but a week later, the feeling has subsided a bit.  I felt that way a bit about Star Wars: Episode I: The Phantom Menace, and it subsided a bit.  I felt that way about Star Wars: Episode VII: The Force Awakens, but it remains one of my favourite Star Wars films.  My favourite films are much easier to list, though.  My top five include: Fucking Åmål, Garden State, Eternal Sunshine of the Spotless Mind, Amélie, Excalibur, Zootopia, and the aforementioned top of the list, Scott Pilgrim v.s. the World.  (Don't try counting those.) What happened to The Life Aquatic with Steve Zissou?   I'm not sure, but it was on the list once.

Music also jumps around a bit.  Aqua, Evanescence, the Human League, Josh Ritter, The Frames, Grimes, and now Kitzl.

She-Ra has had five seasons, which is a lot.  But the first episode aired in November 2018, which is recent.  5 seasons in 18 months.  A strong recency bias.  But, it's so re-watchable in a way that still hits hard each time.  I don't think I've cried this much due to a story in long time (or perhaps ever so much, with Sailor Moon perhaps being the second most moving?).  Its characters have a depth and richness that I often feel lacking and they have priorities I care deeply about.

I haven't cried much in the past 4 years for a variety of reasons.  I think about that a bit here and there.  It's been very important for me to able to empathise with characters again so strongly.  I'd talk about specifics of the story that contribute to this, but as I'm adamantly against spoilers and I adore this show so, I won't.  Let me know if you ever watch it, though.  I have a lot of thoughts and feelings to share.


#General #Microblog friends #Technology life gnome music google iaido guelph fedora vegan bugs food school linux technology #GNOME #School jodo blogger gxml #Budo #Photos work web nature happy vala firefox art Flesherton android anime internet home open source stress travel kendo kosmokaryote writing birthday dad science security animals canada computers environment future cookies development german language photos programming reading sick sleep snow video winter GUADEC cell phones css fun learning love me movies people phone picasaweb ta time christmas evolution vegetarianism #Vegan Toronto ai git gsoc identity society speech vancouver vlogbrothers adventure birds dreams facebook google+ gseta happiness libgdata netflix new zealand night responsibility skedge stars tea tv video games wind mobile Nintendo baking cake communication consumerism design fedora 17 javascript memories nlp organisation photography quote tablet uoguelph Josh Ritter animalia blogging books bug encryption family humanity magic meaning memory money pidgin rain recipes speechdispatcher sushi weather #Reading Spain TAing The Frames cat chocolate cold cycling death film flight genderguesser gitorious halloween health knowledge languages liv mail new years nightmares politics productivity psychology swords the legend of zelda ubuntu web development xml xorg youtube Thanksgiving acer bc busy conversation cooking duolingo emacs emusic fedora 18 galaxy nexus gay rights gmail japan libxml2 martial arts materialism mozilla nerdfighteria nostalgia privacy rhythmbox software sound space university upgrade valentines wahoo walking water Con-G Europe John Green Scott Pilgrim age animal welfare apple autumn bash blog brain brave breath of fire II calm camera canada day change clothing comments confidence conservation creativity culture dance dataloss djaqua duplicity e-mail emotion english errors feminism gdom germany goals google reader gtk humour intelligence japanese laundry law light math morning moving ottawa peterborough pets philosophy pie quality research sei do kai shopping spring style summer value village vday vonage web design website x11 #Life New York alone anime north anxiety argument backup budo buffy business cats computer science concert copyright data loss diy eating economy education energy exercise failure fedora 19 feelings file systems flowers freedom french games gdata greyhound growth habits heat history house html ice cream im information java joy koryu laptop living lost microsoft mood moon muffins mystery news nz pain photo php physics pirates pizza play poverty preupgrade progress purple python rae spoon reality reflection religion rss self serialisation sharing skating social sun synergy tachi uchi testing themes thesis thinking thoughts transit turtles veggie challenge velociraptors violin weekend weird yum zellers API Air Canada Empathy Grimes Hank Green Hugo Jane Austen Lord of the Rings Nexus One OCUS Sudbury Trick or Eat arboretum audible autonomous automobiles beauty bike blogs browsers camping cancer canoeing celebration charity chrome cleaning colour community content corporations crafts decay decor depression depth disaster drawing epic equality experience faery fest farmer's market fedora 12 fedora 16 fedora 20 fedora 22 fedup fireworks friend gender ghetto ghosts glib gnome blog gnome shell google talk green hair hobocore hungry icarus instant messaging interest introspection jobs last exile luks macbook mail-notification mario meat in vitro mind mom moon festival motivation mtp ninjas oh the humanity pagans pants papers past performance perl phones picnics pitivi plastic pride pumpkin pumpkin pie quiet thrill receipts rogers rpm seminar sewing simple simplicity sleep deprivation smells soy milk speech dispatcher sports stories story telling strange streamlines swimming telephone temperature texting thought thrift stores time management time travel tragedy truth understanding united states urban ecosystems usb veganism voice volunteering webschwerver wild wireless working world yojimbo zoology Avatar: The Last Airbender Blassreiter CIS*2750 CIS*6890 Czech Republic Diablo Dresden Codak Dunedin Dutch Blitz Electric Networked Vehicle Elliott Brood Ender's Game France Fringe GNOME 3 HTC Hayao Miyazaki Mario Kart Montréal Network Manager Newfoundland Nintendo Switch Ontario Ouran Host Club Richard SVC Samsung Samurai Champloo Santa Claus Studio Ghibli TCAF US academics adb aeroport algonquin amusing animal agriculture apartment ask automation awkward bad movies banana bats battery beard belladonna beta bicycle book branding breakfast brno bus buses buy nothing day cabin calgary candy cards cars catastrophe celebrate celtic chat cheap cheese childhood china chinese calendar cities clarity clean clock comics compassion compiler computer conspiracy theorists consumption context convention cookie cool cornerstone cosplay cottage country court creation cthulhu cupcakes curiosity cute dancing dark themes dbus definition deja-dup democracy despair detachment dinosaurs discomfort dns dodgeball dragon dress dust dystopia earth earth day efficiency eggs elections email enhanced history ethics evil exhausted expectations exploring ext3 ext4 fail fair trade fall fashion favourite feedly ferry focus formal free friendship fruit fudge full moon furniture gaelic game boards garden gardening gee generosity genetics gimp gir gobject good google hangouts google wave government grading gratitude green roofs groups gsec guerilla gardening haircut hakama help homosexuality honesty howl hp human rights humanitarianism humility hypocrisy ice images imaqua instagram integration intellectual property internet explorer jabber jazz jelly bean jokes kernel keyboard knife labs last exile: fam the silver wing laurena lazy letters library libxml livejournal lizzie bennet loneliness loss lovely lyrics maps maturity meditation melancholy metadata microbes microfinancing microwaves moon cake morality mother music concert muso jikiden eishin ryu myth namespaces nasa nerdfighter neural networks nintendo 3ds normal normality notes obsolescence oceans open open souce open standards panasonic paper parties patches peanut butter perception personal perspectives philanthropy plants pleasant poem politeness potluck preparation problems ptp pulseaudio quidditch racism recreate redundancy relationships relax repairs resizing richard's room roomba roses rsync running sad sadness salsa samurai sanity scary schwarting seasons self-esteem self-navigating car selinux semiformal senility sensitivity sentimental sheep ships silicon motion sleeping in sms social justice software engineering solitude solutions songs soup speed spelling ssh star wars strangers stupid success sunset surreality survival skills suspense sustainability sweet sympathy symphony tardigrades tasks teaching technical communication and research methods test tests thrift tim tams time and space tired tools tracker tradition tranquillity transience trees trust tumblr twitter update usability utopia via vihart vlog waffles warmth waste waterloo wave web comic webkit wii wiki winter is coming wizard wonder woods words xmpp yoga youth zoo #Gaming #Wishlist #anime #general 1. is anyone reading this? 1602 1984 2. you win a prize! 2008 2014 24fps 3. gimme a call to collect 404 A Short Hike All My Children Andy Griffith Argentina Armstrong House Avatar: The Legend of Korra BarTab Beach House Boston Boston Summit British Columbia Businesses C CIS*6050 Cambridge Christopher Plummer Claymore Creatures Darker than Black David Attenborough Dear Wendy Docking Station Dollhouse Earthbound England Excalibur February Fergus Final Fantasy IX Fire Emblem GError GSA Go Google Play Music Hunger Games I am not okay with this I believe in a thing called love I'm a wizard IRC Ikea Ireland JRR Tolkien King Arthur Lost Lagoon MIT Mac OS X Madrid March Massachusetts Matlock McGuinty Melodies of Life Merlin Michael Cera Mother Mother Mr. Tumnus Narnia Neil Gaiman New York Philharmonic Nick and Norah's Infinite Playlist Nintendorks Norns North Korea NotesFromNewYork Olympic OpenShot Orphen Orson Scott Card Oscars PEAP Pauline Johnson Pete Peterson Planet Fedora Porco Rosso Questionable Content R ROM Rent S SIM Wireless Sauble Beach Sega Sega Genesis Selenium Shakespeare She-Ra Snakes and Lattes Splatoon Star Trek Steve Grand Stranger Things ThanksLiving The Darkness The Devil is a Part-Timer The Fifth Estate The Guild The Hobbit The Stand Tianjin Tim Hortons Tolkien UI UK UX VPN Will Grayson Will Grayson Wolves in the Wall WordPerfect Xiki [General] abrt absolutism abuse academia accessibility active activism activity addiction adreama adrift adulthood advertisement advertising air airport express airship ajax al gore alarm clock aldiko alice in wonderland alien alistair summerlee amateur amazon ambience ambition amy winfrey anaconda and imperfection angle angry birds anhosting animation anon anonymity ant apache apology appearances appreciation aqualab arcade architecture arduino arrogance assassins assignments association analysis astrid asus eee top asynchronous ati attachment attitude attribution audio aural abuse authentication authenticity automake automarker avatars awesome b43 backpain backtrack3 backyard bounty bad bagel bandwidth banjo banks barbarians barefoot baseball bathroom beaches beautiful bed bees beetles being belief bellaqua benedict cumberbatch berlin bertrand russell bill gates biofabrication biology biometrics bit rot bitcoin black and white blame blockbuster bloomberg blue board games bohemian bold bon thé place bonds border boredom botany boxing day boy brain scoop brickworks broadcom broccoli browsing bubbles bubbly buildings bunnies burn bus stops butterflies buttons c# c++ cafe calendaring calligraphy camel camera obscura cameras canadian english canopy capitalism captivity careless caring cast causality cbc cedar row cello censorship certainty cgi chalk challenger changing locks chaos theory charm cherry blossoms chickadee chickens chivalry choir chopsticks chores christchurch christianity chudan church cijf cinnamon classes clif clorox clorox green works cloud cloud atlas clubs cname coca cola codeine codeviz coincidence coins color comfort commons communism competence competition competitive coughing completeness compliments conference configuration conflicted confusion consciousness consent conservatives conservativism console construction contagion contest contributing convenience corpses cough suppressants coughing coupons courageous crashes crates crayons crazy creative commons criminals crisps criticism crosscanada crowd crtc cry crying cryptic cryptozoology csh cuddles cult currency current tv curse customer service customisation cvs daily grind data data mining databases dating david bowie dconf debate debug symbols debugging delicious design patterns desktop desktop summit destiny dftba diet difficult digimon digital receipts disabilities disappointment discordianism discoverability dispute dissection kit distraction diyode dnf doctor who doctors documentation dokuwiki doubt doughnut dpkg drab drano drano prevention dream dreaming drinking drm drowning dryers drying dtwydt ducks dvds dying dynamic typing ease easter easy ebony jewelwing ebooks ecards economics editors eeetop el paso elder neglect electronic receipts elements elitism ellen page embarrassment emily graslie emptiness empty enchant end of enterprising environmental science symposium eog epiphany eplugin equipment essentialism ether euphoria evoaqua experiment experimenting expertise extensions extortion facades faith falafel familiarity fan fancy fantasy fascism faun favicon fears fedora 11 feed me feedback festival fibonacci fiction fiddler crab field guide field identification figment figures of speech file formats finances fire fish fitness fixing flac flash light flesherton fling flexibility flour flow flying fonts footprints forceps forgottotagit fork fortunate fortune found fragaria frameworks fraud fred penner free time freezing french fries fresh friday friend's wedding frog fspot funding funerals funny fury fuse gargoyles gdb geek geeks gf3 gi gifts gio gitlab gjs glass globalnewtgames glory gloves glue gluten gm gmo gnome keyring gnome software gnome-control-center go ninja go go transit goat gods goodbye goodfella's google assistant google books google calendar google wallet gp2x gqe grad graffiti grammar graphing graphviz grass green beaver grey county groceries growing up gtest gtg guts gypsies habit hal halls hard hard drive hard drives hardship hardware harry potter hdtv heart heart break heaven 17 hemlock grove hewlett packard hijinx hiking hoaxes hobbies holidays homelessness homework honey badgers honour horatio hornblower horror hostels hosting hot house of cards hp lovecraft hugs humblebundle humbleness hunting hyperlinking hyrule i am a carpet ibm thinkpad x41 icalendar ice cream sandwich ice rain icthyology ignorant ill image image editing imagination impermanence inadequacy inaturalist inconvenience independence india individuals industry infinity ingrid michaelson inhumanity injuries ink innovation insects installation intel interactivity interlocutor internet tv invertabrates io irish irony isolation it it is indigo james bond java 13 jedi jikiden joke journalism journey judgement julian assange julie thiel justice kata kayak keys ki-ai killme kim taylor kinder kindness kirby kitchen kitzl kiva knights knots kodak koodo kung fu labels landau sacamoto late laundromat led legend lending lenovo lessons letsencrypt letstrace letter writing liberalism liberals libnotify libreoffice librpm lifehacker lilo limericks limits linksys liquid lists live wallpapers livecd liveusb loans local local food local install login london losher lots of hugs mac mini machine learning machine vision madness mae magic school bus magical maintainership majesty malaria malls mantis shrimp marine life marketing marking massages matrices maturation may seminar meat media medicine mel's diner memory leaks mental health meow mercy messaging metacity metaphor methodology mezzo forte micropayments mild mild weather military milk mindhacks minimalism misanthropy miscellany misery misfortune missed the boat missing mlp modelling moisture mold molly parker monitors monologue more cats mosquitoes moss mother's day mounting mouse moxies muffin muffinfilms mundane murder museum mushishi mushroom soup mushrooms musicals mutual funds my slumbering heart mysql nameservers nanowrimo national treasure natural language processing naturalism nausicaa navigating necessity neighbours nervous netgear network new new users newspaper hat next year ninja turtles nodelist nointernet noise noisy nominate non-root norse noses not really dying notebooks notification-daemon novels november fair nuclear war numbers numix obama obligation obliviousness obscure ocz ogg oggenc olap olive omote open formats open music openness openoffice optimisation optimism orcas orchestra oreo oreos org-mode origami oscar otr overheat owen sound package management packagekit packing paint shedding pan pancakes panda parallelism paranoia passport patents patience pattern recognition pdo peace peaceful pen pence pender penguins penmanship perfection pet rocks physical piano pickman's model picnik pidgin plugins pikmin pintsize pipelight pirate festival pizza hut plagiarism planning plans playground playlists plumbing plushies podcast poetry points pokemon polls pomplamoose positions posse post posters postmodernism potatoes potlucks power ppc practise prejudice premier pressure pretty pride and prejudice priorities private processes professionalism projects promise protest proud purchases qt quarantine rad radeon railroad randall munroe raop rats reagan recursion recycling redhat reductionism refactoring refrigerators regret relativism release renew renfrew repetition report resolutions resolve resumes reuse reuters reviews revolution rhino rhps ricola risk road trips roar robots rockwood rot rover rtm ruby day ryu safety sanctuary sand satisfaction savages scary movies scheduling schneier scholarships scooters scp screenshots script seals search secret world of arrietty secrets seitei self-interest self-respect self-sufficiency self-worth semesters senescence sessions setbuilder settlers of catan sftp shame sheepo pistachio sheila patek shell shells sherlock holmes shipping shogun shotwell shoulder bag sigh sim city simafort simpsons sincerity singing sjr skill skunks sky slackware slashdot sliver small smartphones smiling snails sneezing snowboarding soccer social dance social media socis soft solemn someonesmotherwantstoadoptme song sony sophistication sorbet sorrow sparklers speed river spell spellchecking spies spilt milk splendid splendor splinter spoilers sponges sql squaresville sr ssd sshd stanley park starry night starving steampunk storage strawberries strength structured information struggle stuff stylus suburi sucks sugar super mario super mario land 3d superiority superstition surprise surprises surreal sushi surrender swings systemd systems tabs tachi uchi no kurai tail coats tameshigiri tarot taxes tears technocracy teddy bears tedtalk term termcap terror the duke the fault in our stars the hulk the human league the irregular at magic high school the onion theatre theory thingsidon'twanttodo tim berners-lee tim mcgraw timber timbre timeliness tin tin toaster todo toilets tolerance tonight toomuch touch screen touchpack tour tourniquet towels toys trac trailer translation travel buddy treestyle view trex triumf triumph trivia trouble tweak twist tx2500 tx2617 typing ugly logos umbrellas un dinaru underwold unemployment universe unlimited blade works updates upgrades uploading urban agriculture urban ecology urchins user experience vagrancy vagrant vague but exciting valadoc validation values vampires vanilla ice variety vegetables velvet burger verb version control vi vinegar violence voip vpnc vulnerable waf wandering wanting war warm weapons web hosting webcomic webcomics webfonts werewolves whales what a wonderful town whatsbetter whic are also lazer powered white spot wifi wii u wikisource will williams wings wisdom wishes wizardry wolf wonderland wordplay world cup world water day writing voice xenophobia xephyr xinput xkcd xpath yahoo yay yyz z-index

Blog Archive