# Rate limiting and bot protection for API endpoints

Summary

This document describes the changes introduced in the commit "Implement rate limiting and bot protection for API endpoints" (commit: 453e6528daa40d3360b571dfee76a977c8fb2d5a).

Purpose

- Limit abusive requests against public and sensitive API endpoints (login, signup, OTP, password reset).
- Provide a lightweight server-side barrier against automated clients (bots, scrapers, generic HTTP libraries) that misuse endpoints.
- Keep the public API usable for real clients while reducing the impact of credential stuffing, enumeration, and mass signup/OTP abuse.

Files changed (high level)

- `app/Providers/RouteServiceProvider.php`
  - Adds named rate limiters using Laravel's `RateLimiter::for(...)` for specific use-cases: `api`, `login`, `signup`, `otp`, `forgot-password`.
  - Each named limiter defines limits and custom 429 JSON responses.

- `app/Http/Kernel.php`
  - Registers the `block.bots` route middleware mapped to `\App\Http\Middleware\BlockBots`.
  - The `api` middleware group continues to use a throttle string (`'throttle:300,1'`) (see notes about named vs inline limiters below).

- `app/Http/Middleware/BlockBots.php`
  - New middleware that implements lightweight bot checks:
    - Rejects POST requests that do not accept or send JSON (expects `Content-Type: application/json` or `Accept: application/json`).
    - Requires a non-empty `User-Agent` header.
    - Blocks requests whose User-Agent contains known automated-client patterns such as `curl/`, `python-requests`, `bot`, `crawler`, `scrapy`, etc.
    - Returns appropriate JSON error responses (HTTP 415 / 400 / 403) for violations.

- `routes/api.php`
  - Sensitive endpoints now apply the new named throttles and the `block.bots` middleware. Examples include:
    - Citizen: `/login` (`throttle:login`, `block.bots`), `/signup` (`throttle:signup`, `block.bots`), `/otp-resend` (`throttle:otp`, `block.bots`), `/forgot-password` (`throttle:forgot-password`, `block.bots`).
    - Employee and queue-login endpoints use `throttle:login` and `block.bots` as appropriate.

Configured rate limits (current values)

- `api` (named limiter)
  - Limit: 60 requests / minute
  - Key: authenticated user id when available, otherwise client IP
  - Purpose: generic per-client API limit (used where explicitly referenced)

- `login` (named limiter)
  - Limit: 3 requests / minute
  - Key: authenticated user id when available, otherwise client IP
  - Purpose: hard limit for authentication attempts to mitigate credential stuffing and brute force attacks
  - Response: JSON 429 with message `Too many requests`

- `signup` (named limiter)
  - Limit: 5 requests / hour
  - Key: `ccode:mobile` when present; otherwise client IP
  - Purpose: prevent mass account creation for a single phone number or from one IP
  - Response: JSON 429 with message `Too many signup requests. Please try later.`

- `otp` (named limiter)
  - Limit: 3 requests / minute
  - Key: `uid` if present; otherwise `ccode:mobile` if present; otherwise client IP
  - Purpose: prevent OTP enumeration and rapid resend abuse
  - Response: JSON 429 with message `Too many OTP requests. Please try later.`

- `forgot-password` (named limiter)
  - Limit: 3 requests / minute
  - Key: `login_id` if present; otherwise client IP
  - Purpose: limit password-reset abuse and account enumeration
  - Response: JSON 429 with message `Too many requests for password reset. Please try later.`

Notes on global `api` throttle group

- The `api` middleware group in `app/Http/Kernel.php` includes `'throttle:300,1'` which translates to 300 requests per minute per IP (the inline throttle string). This remains in place as the group default.
- The commit also defines a named `api` limiter (60/minute). To apply the named limiter to the API group you can replace the inline throttle with `'throttle:api'` in `app/Http/Kernel.php`.
- Decision rationale: keep a permissive general API group throttle while applying conservative, purpose-specific named limiters for high-risk endpoints.

Bot protection behavior (middleware)

- Content-type enforcement
  - POST endpoints are expected to send JSON. A missing/incorrect `Content-Type` and `Accept` headers will return HTTP 415 with JSON:
    - { "status": false, "message": "Invalid content-type. Expected application/json" }

- User-Agent requirements
  - Requests must include a `User-Agent` header. Missing UA returns HTTP 400 with JSON:
    - { "status": false, "message": "User-Agent header is required" }

- Known-bot User-Agent patterns
  - If the `User-Agent` contains substrings commonly used by generic HTTP clients and scrapers (configurable list in `BlockBots`), the request returns HTTP 403 with JSON:
    - { "status": false, "message": "Automated clients are not allowed to access this endpoint" }

- Conservative default
  - The UA blacklist is intentionally conservative and includes keywords like `bot`, `crawler`, `curl/`, `python-requests`, etc. Mobile applications and legitimate clients should set a descriptive `User-Agent` to avoid false positives.

Routes updated (non-exhaustive)

- Citizen
  - `POST /api/citizen/login` — `throttle:login`, `block.bots`
  - `POST /api/citizen/signup` — `throttle:signup`, `block.bots`
  - `POST /api/citizen/otp-resend` — `throttle:otp`, `block.bots`
  - `POST /api/citizen/forgot-password` — `throttle:forgot-password`, `block.bots`

- Employee and queue-login endpoints
  - Employee `POST /api/employee/login` — `block.bots`, `throttle:login`
  - Employee `POST /api/employee/forgot-password` — `throttle:forgot-password`, `block.bots`
  - Queue agent/office login routes similarly apply the `login` limiter and `block.bots` middleware

Example failure responses

- Rate limiter (429):
  - { "message": "Too many requests" } or other custom messages defined per-limiter

- Blocked due to content type (415):
  - { "status": false, "message": "Invalid content-type. Expected application/json" }

- Blocked due to missing UA (400):
  - { "status": false, "message": "User-Agent header is required" }

- Blocked due to known-bot UA (403):
  - { "status": false, "message": "Automated clients are not allowed to access this endpoint" }

## How to test (quick checks)

### Missing Content-Type

Example request (missing Accept / Content-Type JSON):

```bash
curl -X POST "https://your-api.example/api/citizen/login" \
  -d '{"login_id":"...","password":"..."}'
```

Expected: 415 JSON error (BlockBots)

### Missing User-Agent

Example request (explicitly empty User-Agent):

```bash
curl -X POST "https://.../api/citizen/login" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json" \
  -H "User-Agent:" \
  -d '{...}'
```

Expected: 400 JSON error (BlockBots)

### Bot-like User-Agent

Example request (bot-like User-Agent):

```bash
curl -X POST "https://.../api/citizen/login" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json" \
  -H "User-Agent: curl/7.68.0" \
  -d '{...}'
```

Expected: 403 JSON error (BlockBots)

### Hitting a named limiter

- Repeatedly call `POST /api/citizen/login` more than 3 times per minute from same IP → 429
- Repeatedly call `POST /api/citizen/signup` more than 5 times per hour for same `ccode:mobile` → 429

How to customize or extend

- Adjust limits
  - Edit `app/Providers/RouteServiceProvider.php` and change `Limit::perMinute(...)` / `Limit::perHour(...)` values.
  - To apply the named `api` limiter as the default for the API group replace `'throttle:300,1'` in `app/Http/Kernel.php` with `'throttle:api'`.

- Allowlist / whitelist
  - Add allowlist logic in `BlockBots::handle()` to bypass checks for specific:
    - API keys, authenticated users, internal IP ranges, or a custom header (e.g. `X-Internal-Client: true`).

- Exempting trusted clients from rate-limits
  - Use middleware to bypass or adjust `RateLimiter` usage for trusted API keys or IPs.

- Make UA checks more robust
  - The current UA blacklist is a simple substring check. For more robust protections consider:
    - Allowlist approach for known clients
    - Verifying a signed client header
    - Integrating hCaptcha/reCAPTCHA on signup flows

Testing recommendations

- Add unit tests / feature tests that exercise: missing headers, bot UAs, rate-limit exceedances, and expected success flows.
- Run load tests targeted at the `signup`, `otp`, and `login` endpoints to validate limits and latency under expected traffic.

Rollback plan

- To revert behavior quickly:
  - Remove `block.bots` middleware from routes in `routes/api.php` and/or `routeMiddleware` in `app/Http/Kernel.php`.
  - Remove named rate limiters in `app/Providers/RouteServiceProvider.php` or increase their limits to acceptable levels.

Notes and caveats

- False positives are possible (mobile clients with generic UA strings). Observe logs and telemetry after deployment and adjust the UA checks or provide a simple client update that sets an explicit `User-Agent`.
- The commit keeps a permissive API group throttle while applying conservative limits for high-risk endpoints; ensure that monitoring captures 429 spike patterns for troubleshooting.

Reference (files touched)

- `app/Providers/RouteServiceProvider.php`
- `app/Http/Kernel.php`
- `app/Http/Middleware/BlockBots.php`
- `routes/api.php`

If you want, I can also:

- Add test cases covering the middleware and named throttles.
- Replace the inline `throttle:300,1` throttle with the named `throttle:api` in the API group and adjust the global limit.
- Add configuration via environment variables to control the limits without code changes.