journal archiving

[egypturnash]

One of the features LJ has been lacking in is a way to archive the entirety of your journal. Sure, it lets you grab a month's worth of entries in a plain text format you can keep (go here), but there's no way to get the meat of what makes LJ different from other blog services: the comments.

Some very meaningful entries of mine have depended entirely on the comments for their power. But it's impossible to find a specific comment. Maybe Google, if you're lucky and it's on a public entry.

This morning I had a little flash of insight on a way to work around this lack: lean on the e-mail comment notifications. Every damn comment someone besides you makes in your LJ generates an e-mail to you, unless you turn that off. With a subject that's pretty easy to filter for. With handy URLs that reference back to the entry and the comment itself, and the entire text.

How about a system wherein your e-mail program (whatever it is - this could all be done via a platform-independent script language with some specific tweaks for interfacing to your preferred mail program) passes the comment notification to another program on your machine that archives the parent post if needed, and adds the comment to it? Bonus points for cacheing the user picture used, too - that can be an important part of the meaning of a comment, and I believe they may all have unique, never-reused IDs, even when deleted...

There could be a web-based interface - I know OSX ships with Apache in it, just install a couple cgi-bins to let you browse and search the stuff. You could even get it to archive discussions you engaged in on other people's journals. Keep your own copy in case LJ vanishes into the ether, search it, run parody generators on someone's comments, play Burroghs cut-up games with it, whatever.

The one catch is that you don't get notification e-mails on your own comments; this could be dealt with by, say, having it screen-scrape the entire comments page for such things, maybe looking at entries every hour or so for a day or two after recieving a comment in case you replied to it... it'd have to be able to have your LJ password stored in it, so it can access the same protected entries you can.

Extra bonus points for being able to handle multiple journals. And for checking now and then (daily? upon demand?) to make sure it has all the stuff you post, even if it gets no comments.

I do not have the time or skill to build such a thing. I could acquire the skill to get something semi-workable but ugly, but there're too many other things I'd like to spend what little energy I have on. So. If someone reading this wants to start hacking, feel very free. I'd be quite happy to be a beta-tester! I'd even help in the make-it-look-pretty stage.

Comments

[strredwolf.livejournal.com]

Check lj_clients. They had to disable the ability to do a backup because all but one client was dragging the server down.

Which client wasn't? Logjam.

[shatterstripes.livejournal.com]

Aaactually...

the URL I pointed to works fine, unless they disabled it between yesterday and now. Backing up your LJ entries via the LJ protocol isn't disabled, just rate-limited.

Anyway, I'm talking about backing up comments, which has, as far as I know, never been implemented in the LJ protocol ('how can I get comments on an entry?' 'you can't.' seems to be an FAQ in lj_clients, though someone has finally decided to have a go at implementing it, hooray!). Just backing up the text of, say, this entry is pretty useless, since the intended point is the discussion in the comments.

[hydra-velsen.livejournal.com]

Hardly anyone comments on my posts, but then again, I actually do use my LJ as a journal. It's more for my own edification than anything. I get few enough comments that I do use the email notification thingy.

[markpasc.livejournal.com]

You don't have Python 2.3, do you?

[shatterstripes.livejournal.com]

Welcome to Darwin!
[erinyes:~] peggy% python
Python 2.2 (#1, 07/14/02, 23:25:09) 
[GCC Apple cpp-precomp 6.14] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> quit
'Use Ctrl-D (i.e. EOF) to exit.'
>>> ^D
[erinyes:~] peggy% 

[metamouse.livejournal.com]

If you're still interested, I just wrote a little Python script (http://www.ocf.berkeley.edu/~dtw/misc/ljbackup.py) to webcrawl a livejournal and save it locally, including comments. Only tested it on Windows, but it should work anywhere.

Hi :) I like your art.

[shatterstripes.livejournal.com]

Cool! I'll check it out a little later today. (Been distracted, didn't have a chance to deal with it immediately upon getting the comment notification.)

[shatterstripes.livejournal.com]

Well, it starts to work, but it runs up against a friends-locked entry three in. *g* So it doesn't get around to chewing on the links or anything like that.

Someone is, supposedly, finally working on adding the capability to get comments via LJ's database functions... when it'll be done, who knows.