Source Code

X Bookmark Archiver

Archive your X (Twitter) bookmarks into categorized markdown files with AI-generated summaries.

Overview

This skill fetches your X bookmarks using the bird CLI, categorizes them by URL patterns, generates AI summaries using OpenAI, and saves them as organized markdown files.

Prerequisites

bird CLI - Install from steipete/bird
OpenAI API Key (optional) - Set OPENAI_API_KEY for AI-generated summaries
Node.js 18+

Installation

# Ensure bird CLI is installed and authenticated
bird --version

# Set OpenAI API key (optional, for AI summaries)
export OPENAI_API_KEY="sk-..."

Commands

`run` - Full Pipeline

Fetches new bookmarks and processes them:

node skills/x-bookmark-archiver/scripts/run.cjs

`run --force` - Process Existing

Skip fetch, process only pending bookmarks:

node skills/x-bookmark-archiver/scripts/run.cjs --force

`fetch` - Download Bookmarks Only

node skills/x-bookmark-archiver/scripts/fetch.cjs

`process` - Archive Pending Only

node skills/x-bookmark-archiver/scripts/process.cjs

Category Mapping

Bookmarks are automatically categorized based on URL patterns:

Category	Domains
tools	github.com, gitlab.com, github.io, huggingface.co, replicate.com, vercel.com, npmjs.com, pypi.org
articles	medium.com, substack.com, dev.to, hashnode.dev, x.com/i/article, blog.*, towardsdatascience.com
videos	youtube.com, youtu.be, vimeo.com, twitch.tv
research	arxiv.org, paperswithcode.com, semanticscholar.org, researchgate.net, dl.acm.org, ieee.org
news	techcrunch.com, theverge.com, hn.algolia.com, news.ycombinator.com, wired.com, arstechnica.com
bookmarks	fallback for unmatched URLs

Output Location

Markdown files are created in the OpenClaw workspace at:

Legacy installs (old name):

~/clawd/X-knowledge/

New installs:

~/.openclaw/workspace/X-knowledge/

With profile (OPENCLAW_PROFILE=prod):

~/.openclaw/workspace-prod/X-knowledge/

Override with environment variable:

export OPENCLAW_WORKSPACE=/custom/path
node skills/x-bookmark-archiver/scripts/run.cjs
# Creates: /custom/path/X-knowledge/

Output Structure

~/.openclaw/workspace/X-knowledge/
├── tools/
│   ├── awesome-ai-project.md
│   └── useful-cli-tool.md
├── articles/
│   ├── how-to-build-x.md
│   └── ml-best-practices.md
├── videos/
│   └── conference-talk.md
├── research/
│   └── attention-is-all-you-need.md
├── news/
│   └── latest-tech-announcement.md
└── bookmarks/
    └── misc-link.md

Markdown Template

Each archived bookmark gets a markdown file with frontmatter:

This project implements a novel approach to... (AI-generated summary)

State Management

State files track processing progress:

/root/clawd/.state/
├── x-bookmark-pending.json     # Bookmarks waiting to be processed
└── x-bookmark-processed.json   # IDs of already-archived bookmarks

Environment Variables

Variable	Required	Description
`OPENAI_API_KEY`	No	API key for AI-generated summaries

Workflow

Fetch: Downloads latest 50 bookmarks from X
Filter: Removes already-processed bookmarks
Expand: Resolves t.co shortened URLs
Categorize: Assigns category based on URL domain
Enrich: Generates title, summary, tags (AI or fallback)
Write: Saves as markdown in X-knowledge/{category}/
Track: Updates processed IDs, clears pending

Customization

Adding Categories

Edit scripts/lib/categorize.cjs:

const CATEGORIES = {
  tools: ['github.com', '...'],
  your_category: ['example.com', '...'],
  // ...
};

Changing Output Directory

Edit scripts/process.cjs:

const KNOWLEDGE_DIR = 'your-directory-name';

Using Different AI Provider

Modify the generateMetadata() function in scripts/process.cjs to use your preferred API.

Testing

Run the test suite:

# Run all tests
cd skills/x-bookmark-archiver/tests
node test-all.cjs

# Run individual test suites
node lib/categorize.test.cjs
node lib/state.test.cjs
node integration.test.cjs

Test Coverage

Unit tests: categorize.js (21 tests) - URL pattern matching
Unit tests: state.js (9 tests) - JSON read/write operations
Integration tests (12 tests) - Full pipeline with mock data

Manual Testing

Without bird CLI, you can test with mock data:

# Create mock pending data
cat > /tmp/test-pending.json << 'EOF'
[
  {
    "id": "test123",
    "url": "https://github.com/test/repo",
    "text": "Test bookmark"
  }
]
EOF

# Copy to state directory and process
mkdir -p /root/clawd/.state
cp /tmp/test-pending.json /root/clawd/.state/x-bookmark-pending.json
node skills/x-bookmark-archiver/scripts/process.cjs

Troubleshooting

Issue	Solution
`bird: command not found`	Install bird CLI from GitHub releases
No bookmarks fetched	Ensure you're logged into X in bird
AI summaries not generating	Check `OPENAI_API_KEY` is set
t.co links not expanding	May be network/timeout issues; will use original URL

File Structure

skills/x-bookmark-archiver/
├── SKILL.md
├── scripts/
│   ├── fetch.cjs          # Download bookmarks from X (CommonJS)
│   ├── process.cjs        # Generate markdown files (CommonJS)
│   ├── run.cjs            # Orchestrate fetch → process (CommonJS)
│   └── lib/
│       ├── categorize.cjs # URL → category mapping (CommonJS)
│       └── state.cjs      # JSON state management (CommonJS)
└── tests/
    ├── test-all.cjs
    ├── lib/
    │   ├── categorize.test.cjs
│   │   └── state.test.cjs
    ├── integration.test.cjs
    └── fixtures/
        └── sample-bookmarks.json

License

MIT

x-bookmark-archiver