Metadata-Version: 2.4
Name: ugcscraper-helper
Version: 0.2.3
Summary: A tiny Python client for the UGC Scraper Reddit API — scrape posts and comments, list subreddits, users and search results, read profiles and rules, all as clean JSON without Reddit API keys or any infrastructure of your own.
License: MIT
Project-URL: Homepage, https://ugcscraper.com
Project-URL: Documentation, https://ugcscraper.com/reddit-scraper-api
Project-URL: Repository, https://github.com/builtbyish/ugc-scraper-python-helper
Keywords: reddit,reddit-scraper,reddit-comment-scraper,web-scraping,praw-alternative
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.25
Dynamic: license-file

# ugc-scraper-python-helper

A lightweight **Reddit scraper Python** client for the [UGC Scraper](https://ugcscraper.com) API.
Scrape any Reddit post and its full comment thread into clean, consistent JSON, list subreddit,
user and search results, read profiles and subreddit rules, and re-check scores in bulk — with **no
Reddit API keys**, no OAuth app, and no infrastructure of your own.

It is a single-module wrapper over the [UGC Scraper Platform](https://ugcscraper.com) REST API. One
key, one base URL, the same JSON schema every time, so you can build your parser once.

## Why

Most ways to scrape Reddit break on the same things: the official API needs OAuth and caps listings
at ~1,000 items, and DIY scrapers get blocked and rate-limited. UGC Scraper handles
all of that for you, and bills only for successful posts. This helper is the
smallest possible way to call it from Python.

## Install

```bash
pip install ugcscraper-helper
```

Requires Python 3.9+ and `requests`. (Vendoring works too: the client is one file — copy
`ugcscraper.py` into your project.)

## Usage

```python
from ugcscraper import UGCScraper

client = UGCScraper("rps_live_your_key")          # get a free key at ugcscraper.com/dashboard

# Scrape a post + its full comment thread into one JSON schema
post = client.scrape("https://www.reddit.com/r/python/comments/abc123/")
print(post["title"], post["score"], len(post["comments"]))
```

`UGCScraper(api_key, base_url="https://api.ugcscraper.com", timeout=60)` — raise `timeout` to 120 s
or more if you scrape uncached posts, since requests are synchronous.

### Posts

```python
client.scrape("abc123")                               # URL or bare post id -> post + comments
client.scrape("abc123", fresh=True)                   # same, with the numbers Reddit shows now (10-30 s)
client.refresh(["abc123", "def456"])                  # current score / comments / removed, 1-150 posts
```

`refresh()` always bypasses the cache and stores each success as a new history snapshot; it costs
one quota slot per input, released for items that fail.

### Listings (post summaries, no comments)

```python
client.subreddit("webscraping", sort="new", limit=25)     # sort: "new" (default) or "top" (all-time)
client.user_posts("spez", limit=25)                       # a user's own submitted posts
client.user_comments("spez", limit=25)                    # their comments + each parent post
client.search("reddit api", subreddit="webscraping", sort="new", limit=25)
```

`target` accepts `webscraping`, `r/webscraping` or a full reddit.com URL (and `spez`, `u/spez` or a
profile URL). `limit` is 1 to 100.

**Paging.** `subreddit()`, `user_posts()` and `user_comments()` take unix-time cursors: re-send the
call with `before` = the last item's `created_utc` for the next page older (or `after` for newer).
Cursors work with `sort="new"` only — pairing them with `sort="top"` is a `422`. `search()` has **no
cursor**; narrow it with `subreddit`, `sort` and `limit` instead.

### Profile and rules

```python
client.user_profile("spez")        # karma, account age, trophies, moderator-of, public flags
client.subreddit_rules("webscraping")   # short_name, description, applies_to, in listed order
```

Neither is cached, so both take 3 to 15 seconds. One quota slot per call, released if the lookup
fails.

### History, usage and account (free — no quota)

```python
client.history(limit=20)                 # durable snapshots you already scraped (summaries)
client.latest("abc123")                  # the newest stored full post, without re-scraping
client.usage()                           # tier, monthly quota, used, remaining
client.usage_daily(days=30)              # {"days": 30, "series": [{"date", "count", "success"}, ...]}
client.account()                         # profile, limits, and every key (name, permissions, prefix)

# The same data as CSV text
open("comments.csv", "w").write(client.latest_csv("abc123", rows="comments"))   # or rows="post"
open("history.csv", "w").write(client.history_csv())
```

`account()` never returns a full key, and key management (create / revoke / rotate) is deliberately
not wrapped here — use the API keys page.

## Permissions and errors

Every API key has a set of permissions chosen when it is created: **Post, Subreddit, Rules, User
posts, User comments, Profile, Search** (a key created without a selection has all of them). Calling
an endpoint with a key that lacks its permission is a `403` and costs no quota:

```python
from ugcscraper import UGCScraper, UGCScraperError, UGCScraperConnectionError

try:
    listing = client.subreddit("webscraping")
except UGCScraperConnectionError as exc:
    print(exc)                      # "could not reach the UGC Scraper API at <base url>: <class>"
except UGCScraperError as exc:
    if exc.permission_denied:       # 403 — this key lacks the endpoint's permission
        print(exc.detail)           # "This API key does not have the 'Subreddit' permission. ..."
        print(exc.hint)             # create a key with it at ugcscraper.com/dashboard/keys
    elif exc.rate_limited:          # 429 — burst limit (10/s) or the monthly quota
        ...
    else:
        print(exc.status_code, exc.detail)
```

`UGCScraperError` carries `status_code` (401 invalid key, 403 missing permission, 404 nothing
stored, 422 bad field, 429 rate limit, 502 temporary failure) and `detail`, the API's own
one-sentence message — nothing more. `UGCScraperConnectionError` is the subclass raised when the API
cannot be reached at all; its `status_code` is `None`.

Which permission each method needs: `scrape` / `refresh` / `history` / `latest` → Post,
`subreddit` → Subreddit, `subreddit_rules` → Rules, `user_posts` → User posts,
`user_comments` → User comments, `user_profile` → Profile, `search` → Search. `usage`,
`usage_daily` and `account` work with any key.

## What you get back

A flattened, LLM-friendly schema: `title`, `author`, `score`, `num_comments`, `permalink`,
`image_url`, `thumbnail_url`, `body`, and `comments[]` (each with `author`, `body`, `score`, `id`,
`parent_id`, `link_id`, `permalink`, `created_at`). The same shape on every call.
Listings return `{"kind", "target", "sort", "count", "items": [...]}` with one summary per item.

## Development

```bash
pip install requests pytest
python3 -m pytest test_ugcscraper.py      # offline: requests.Session.request is monkeypatched
```

## Links

- Platform: <https://ugcscraper.com>
- API reference (endpoints, auth, examples): <https://ugcscraper.com/reddit-scraper-api>
- Get an API key (free tier: 1,000 successful posts/mo): <https://ugcscraper.com/dashboard>
- MCP server for Claude Desktop: `ugc-scraper-reddit-mcp`

## License

MIT
