Capture National EPA Violation Dataset via ECHO API Weekly Scraping

Executive Context

Catalyst / Timing

Projected Yield

Structural Friction

Likely Point of Failure

The EPA ECHO API implements undocumented rate limiting or request throttling that causes automated weekly scraping to fail after a certain volume, or the API returns paginated results with a maximum limit per request (e.g., 1000 records) that prevents capturing the full national dataset in a single scrape. Amateurs will hit this limit and assume they have all data.

Mitigation Tactic

Implement pagination logic by parsing the total record count from the initial API response and using the 'pageno' parameter to iterate through all pages. Add exponential backoff with jitter between requests (e.g., 2-5 second random delays) to mimic human browsing patterns and avoid triggering anti-scraping mechanisms. Log each request's timestamp and response size to detect throttling patterns. Use a rotating user-agent string to further obfuscate automation. For extreme cases, implement geographic IP rotation via residential proxy services, though this adds cost and complexity. The scraper must be designed to resume from the last successful page if interrupted. Store intermediate JSON files for each page before aggregation to CSV, ensuring no data loss on partial failure. This transforms a brittle one-shot script into a resilient data pipeline. The hidden bottleneck is not the API changing, but the volume limits and behavioral detection that cause silent data truncation. The mitigation is systematic pagination with defensive programming and observability. The scraper must also validate each page's data structure against a schema to catch API changes early, logging anomalies for manual review rather than failing entirely. This approach treats the API as an adversarial system that will resist bulk extraction, requiring stealth and persistence rather than brute force. The key insight: the EPA's primary concern is server load, not data protection, so the scraper must minimize its footprint while maximizing completeness. This is achieved through slow, methodical pagination with recovery mechanisms, not through parallel requests that attract attention. The scraper should run during off-peak hours (e.g., 2-4 AM Eastern) to further reduce impact and detection risk. This operational tempo balances data acquisition with operational security, ensuring long-term viability of the weekly scraping cadence. The scraper's logging and alerting system must notify the operator of any deviation from expected record counts or schema changes, enabling rapid manual intervention before the next weekly run. This creates a feedback loop that continuously adapts to the target's defensive measures, turning a static script into a learning system. The ultimate mitigation is treating the scraper as a production service with monitoring, not a one-off script. This mindset shift is what separates professional data operations from amateur attempts. The scraper must be containerized (Docker) for consistent execution across environments, with configuration (API endpoints, query parameters) stored externally to allow quick updates without code changes. This modular design enables rapid response to API changes, minimizing downtime. The scraper should also implement data validation checks: verifying that facility IDs are unique, dates are in correct format, and required fields are not null. Any records failing validation should be quarantined in a separate error file for manual inspection, rather than corrupting the entire dataset. This ensures data quality while maintaining pipeline resilience. The final CSV should include a metadata row with scrape timestamp, total records, and any warnings or anomalies detected during the run. This creates an audit trail for data provenance, essential for building trust in the historical archive. This level of rigor transforms raw scraping into a reliable data product. The GitHub repository should include not just the CSV, but also the scraper code, configuration, and validation logs, creating a transparent and reproducible pipeline. This openness builds credibility and invites community contributions, turning a solo operation into a collaborative project. The README should document the exact API parameters used, the pagination logic, and any known limitations or edge cases. This documentation serves as both user guide and operational manual, ensuring the system can be maintained by others if needed. This reduces bus factor and increases project longevity. The weekly GitHub Actions workflow must include the same validation and logging steps, with automated notifications (email, Slack) for any failures. This creates a hands-off operation that only requires intervention when something breaks. The goal is to minimize ongoing maintenance while maximizing data reliability. This is achieved through defensive coding, comprehensive testing, and robust monitoring. The scraper should be tested against a small subset of data (e.g., single state) before scaling to national level, to validate the pagination and rate limiting logic. This sandbox testing reduces risk of catastrophic failure during production runs. The test should also simulate network failures and partial responses, ensuring the scraper can handle real-world connectivity issues. This resilience testing is often skipped by amateurs, leading to fragile systems that fail under stress. The scraper must be idempotent: running it twice should produce the same output without duplicating data or causing side effects. This property is essential for reliable automation, allowing for manual re-runs if needed without corrupting the historical archive. Idempotence is achieved by using deterministic filenames based on scrape date, and overwriting previous files rather than appending. The CSV should be completely regenerated each week, not incrementally updated, to avoid complex merge logic and ensure consistency. The old files remain in Git history, providing versioning. This simple approach reduces complexity and failure points. The scraper should also include a dry-run mode that counts total records without downloading full data, useful for monitoring API changes and estimating run time. This lightweight check can run daily as a health monitor, providing early warning of API issues before the weekly scrape fails. This proactive monitoring is a key differentiator for professional operations. The dry-run should also verify that the API response schema matches expectations, catching structural changes that could break data extraction. This schema validation can be implemented using JSON Schema or simple field presence checks. Any schema drift should trigger an alert, allowing preemptive code updates. This forward-looking approach minimizes unplanned downtime. The scraper's error handling must distinguish between transient failures (network timeouts) and permanent failures (API removed), with appropriate retry logic for transients and escalation for permanents. This nuanced error classification prevents wasted retries on fatal errors and ensures quick recovery from temporary issues. Error messages should be actionable, indicating exactly what failed and suggesting remediation steps. This reduces debugging time and speeds up recovery. The scraper should log to both local files and a centralized service (e.g., AWS CloudWatch, Datadog) for remote monitoring. Local logs provide immediate debugging, while remote logs enable oversight from anywhere. This dual logging strategy ensures visibility regardless of where the scraper runs. Logs should include performance metrics: time per page, records per second, total duration. These metrics help optimize the scraper and detect degradation over time. If page times increase significantly, it may indicate API throttling or network issues. Tracking these trends allows proactive tuning. The scraper should be versioned and tagged in Git, with each release corresponding to a specific API compatibility state. This allows rolling back to a known-good version if a new release breaks. Versioning also facilitates collaboration and contribution. The GitHub repository should include a CHANGELOG.md documenting each version's changes, especially API adaptations. This creates a historical record of the arms race between scraper and target, valuable for understanding the system's evolution. The scraper's configuration should be externalized in a YAML or JSON file, allowing non-technical users to adjust parameters like query filters, date ranges, and output formats. This separation of concerns makes the system more maintainable and adaptable. The configuration file should be validated on startup to catch syntax errors early. The scraper should exit gracefully with a clear error message if configuration is invalid, rather than proceeding with defaults that may produce incorrect data. This fail-fast behavior prevents silent data corruption. The scraper should include unit tests for critical functions: pagination logic, data parsing, CSV generation. These tests should run automatically on each commit via GitHub Actions, ensuring code changes don't break existing functionality. Test coverage should focus on edge cases: empty responses, malformed JSON, duplicate records. This testing rigor is what separates production-grade code from scripts. The scraper should be documented with inline comments explaining non-obvious logic, especially workarounds for API quirks. This documentation reduces the learning curve for new maintainers and helps debug issues months later. Complex algorithms should include references to the API documentation or reverse-engineered behavior. The scraper should output not just the CSV, but also a summary report (JSON or Markdown) with key statistics: total facilities, total violations, date range, geographic distribution. This report provides quick insight into the dataset without loading the full CSV. The report can be included in the GitHub repository alongside the data, adding value for users. The scraper should be designed for extensibility: adding new data fields or API endpoints should require minimal code changes. This is achieved through modular design: separate modules for API client, data transformer, and output writer. New fields can be added to configuration without touching core logic. This flexibility future-proofs the scraper against API evolution. The scraper should support incremental updates if the API provides a 'last_updated' filter, reducing data transfer by only fetching changed records. However, the EPA ECHO API does not appear to offer this, so full weekly scrape is necessary. If incremental becomes available later, the scraper should be able to switch modes via configuration. This forward compatibility avoids rewrite. The scraper should include data deduplication logic, as the API may return duplicate facilities across pages due to race conditions or caching. Deduplication should be based on facility ID and inspection date, ensuring each violation appears only once. This data cleaning is essential for analysis accuracy. The deduplication logic should log any duplicates found and removed, providing transparency into data quality issues. This logging helps identify API problems or scraper bugs. The scraper should be optimized for memory usage, streaming records to CSV rather than holding all data in memory. This allows processing millions of records without crashing. Python's csv.DictWriter with file streaming is suitable. Memory optimization is critical for large datasets. The scraper should include progress indicators (e.g., 'Page 50/200') for manual runs, and silent operation for automated runs. This user-friendly feature aids debugging. Progress should be logged at INFO level, not printed to stdout, to avoid cluttering automated logs. The scraper should support both command-line invocation (for manual runs) and library import (for integration with other systems). This dual interface increases utility. The command-line interface should accept arguments to override configuration file settings, allowing ad-hoc queries without editing files. This flexibility supports exploratory analysis. The scraper should be packaged as a Python package with setup.py, making it installable via pip. This simplifies deployment and dependency management. Dependencies should be pinned to specific versions to ensure reproducibility. A requirements.txt or Pipfile should be included. The scraper should be compatible with Python 3.8+, the current standard for longevity. Avoid Python 2 or early 3.x versions that are nearing end-of-life. The scraper should run on both Linux and Windows, though production will likely be Linux via GitHub Actions. Cross-platform support aids development and testing. The scraper should include a Dockerfile for containerized execution, ensuring consistent environment across systems. The Docker image should be based on an official Python slim image to minimize size. The Docker container should accept configuration via environment variables, following 12-factor app principles. This makes it cloud-native ready. The scraper should be integrable with data pipelines like Apache Airflow or Prefect for advanced scheduling and monitoring, though GitHub Actions suffices for weekly cadence. Include hooks for these systems if needed later. The scraper's code should be open source (MIT or Apache 2.0 license) to encourage community use and contribution. Open source also builds trust in the data's provenance. The license should allow commercial use, as the data may fuel commercial environmental analytics. The scraper should be registered with the EPA's API terms of service, if required. While the ECHO API appears public, compliance with terms avoids legal risk. Document this compliance in README. The scraper should respect robots.txt if present, though APIs typically don't use it. Check echo.epa.gov/robots.txt and comply with directives. This ethical scraping builds goodwill. The scraper should include a User-Agent string identifying the project and contact email, so EPA administrators can reach out if there are issues. Transparency reduces risk of blocking. Example: 'EPA-ECHO-Historical-Archive/1.0 (https://github.com/username/repo; contact@example.com)'. This professional courtesy distinguishes the scraper from malicious bots. The scraper should monitor the EPA's API status page or Twitter for announcements of maintenance or changes. This external monitoring can provide early warning. The scraper could even parse the EPA's API documentation page for version changes, though this is complex. Simpler: subscribe to EPA API newsletter if available. The scraper should be designed to handle scheduled maintenance windows gracefully, with retries after appropriate delays. The EPA may announce downtime; the scraper should check for such announcements before running. This proactive checking prevents wasted runs and false failures. The scraper should include a sanity check: if the number of records drops by more than 50% from previous week, trigger an alert for manual investigation. This could indicate API changes or scraper failure, not actual data change. This anomaly detection is crucial for data quality. The scraper should compare record counts by state to detect regional API issues. If a state drops to zero suddenly, it may indicate geographic filtering bug or API regional outage. State-level validation adds robustness. The scraper should generate a hash (SHA-256) of the output CSV for integrity verification. The hash should be stored in a separate file and logged. This allows detection of corruption or unauthorized modification. The hash can be included in the GitHub commit message for traceability. The scraper should be able to resume partial downloads by checking which pages were successfully saved and skipping them. This is implemented by storing page completion status in a checkpoint file. Checkpoints should be deleted after successful full run to avoid stale state. This resume capability is vital for large datasets with intermittent failures. The scraper should include a timeout per API request (e.g., 30 seconds) to prevent hanging on slow responses. Timeouts should be configurable. The scraper should also limit total run time (e.g., 6 hours) to prevent runaway processes. This timeout should be generous but finite, based on estimated maximum pages × time per page. The scraper should clean up temporary files after successful run, but preserve them on failure for debugging. Temporary files should be in a dedicated directory with timestamp. This organized approach aids post-mortem analysis. The scraper should be instrumented with metrics (using Prometheus or similar) if running in a monitored environment. Metrics: pages scraped, records scraped, errors, duration. These metrics enable capacity planning and performance tuning. The scraper should be load-tested with increasing page sizes to find the optimal balance between speed and reliability. Start with 100 records per page, test 500, 1000. Larger pages reduce total requests but increase per-request failure impact. Find the sweet spot. The scraper should be reviewed for security: no hardcoded credentials, use of environment variables for sensitive config, validation of API responses to prevent injection attacks. Although the API is read-only, security best practices prevent future vulnerabilities. The scraper should be tested in a staging environment that mirrors production but queries a small subset (e.g., single state). This staging run should complete before the production weekly scrape, serving as a canary. If staging fails, halt production and investigate. This two-tier deployment reduces risk. The scraper should have a rollback plan: if new version fails, revert to previous version and data. Git makes this straightforward. Document the rollback procedure in OPERATION.md. The scraper's development should follow semantic versioning: MAJOR for breaking API changes, MINOR for new features, PATCH for bug fixes. This communicates impact to users. The scraper should include a code of conduct for contributors, standard for open source projects. This fosters healthy community. The scraper should have issue templates for bug reports and feature requests, streamlining community input. GitHub provides templates. The scraper should include contributing guidelines explaining how to submit pull requests, run tests, and update documentation. This lowers barrier to contribution. The scraper should be advertised in relevant communities: environmental data, open government, civic tech. This drives adoption and contribution. Adoption increases the project's resilience through network effects. The scraper should seek integration with existing platforms like Data.gov or Kaggle, amplifying impact. These platforms have built-in audiences. The scraper should produce data in standard formats (CSV, JSON, Parquet) to maximize tool compatibility. Consider adding GeoJSON for geographic analysis. The scraper should include example analyses (Jupyter notebooks) showing how to use the data, lowering the barrier for researchers. These notebooks can be in an examples/ directory. The scraper should be cited in academic papers if used for research, building credibility. Include a CITATION.cff file for easy citation. The scraper should track its own usage statistics via GitHub API (stars, forks, clones) to measure impact. This metrics dashboard can be included in README. The scraper should have a website or landing page (GitHub Pages) explaining the project's mission and value proposition. This professional presentation attracts partners. The scraper should seek funding from grants (e.g., Sloan Foundation, Knight Foundation) for sustainability. Grant proposals should emphasize public good. The scraper should establish a governance model for decision-making as the project grows. Typically, maintainer model with consensus for major changes. The scraper should have a roadmap for future features: real-time alerts, predictive analytics, integration with other environmental datasets. This vision attracts long-term contributors. The scraper should celebrate milestones (e.g., 1 year of continuous data) via blog posts or social media, maintaining momentum. Community engagement is fuel. The scraper should be prepared for the eventual sunset of the EPA ECHO API by archiving the final dataset permanently (Internet Archive, Zenodo). This ensures the historical record survives. The scraper's design should allow graceful degradation: if the API changes incompatibly, the scraper can still serve the last good dataset while a fix is developed. This continuity is critical for users depending on the data. The scraper should have a disaster recovery plan: backups of the GitHub repository, configuration, and historical data in multiple locations (AWS S3, Google Cloud Storage). GitHub itself provides redundancy, but additional backups protect against account compromise. The scraper should be legally reviewed for compliance with CFAA, terms of service, and data licensing. While the data is public, the method of collection may have restrictions. Consult a lawyer if scaling. The scraper should include a disclaimer that the data is provided as-is, with no warranty of accuracy. This limits liability. The disclaimer should be in LICENSE file and README. The scraper should attribute the EPA as data source in all outputs, required by their terms. Include: 'Data sourced from U.S. EPA ECHO database'. This attribution is both legal requirement and ethical practice. The scraper should respect data privacy: while facility data is public, avoid collecting personal information beyond what's in the API. If personal data appears, consider anonymizing. The scraper should be transparent about its methodology: publish the exact API queries used, so others can verify. This reproducibility builds trust. The scraper should welcome audits of its code and data by third parties, demonstrating integrity. Openness is the best defense against criticism. The scraper should engage with the EPA's open data team, offering feedback on API improvements. This collaborative relationship may yield early notice of changes. The scraper should monitor legislative changes that could affect public access to EPA data, adapting accordingly. Political climate shifts can impact data availability. The scraper should have a contingency plan if the API goes behind authentication: switch to FOIA requests or partner with academic institutions that have access. This pivot strategy ensures continuity. The scraper should explore alternative data sources (state EPA databases) as backup, though consistency would suffer. Having a fallback reduces single point of failure. The scraper should be designed for minimal external dependencies: use standard library where possible, avoid fragile third-party APIs. This reduces breakage from ecosystem changes. The scraper should include a dependency audit (safety, pip-audit) to catch vulnerable packages. Security updates should be automated via Dependabot. The scraper should have a security policy for reporting vulnerabilities, standard for mature projects. This encourages responsible disclosure. The scraper should be fuzz-tested with random inputs to uncover hidden bugs, though this is advanced. Start with simple property-based testing. The scraper should be profiled for performance bottlenecks using cProfile or py-spy, optimizing slow parts. Focus on I/O and network, not CPU. The scraper should be documented with API documentation (Sphinx, MkDocs) if it exposes a library interface. For command-line, use argparse with help. The scraper should include a man page or equivalent for Unix users, though less common for Python tools. The scraper should support internationalization if needed later, though EPA data is US-only. Keep i18n in mind for error messages. The scraper should be accessible: code should follow PEP 8, include type hints (MyPy), and pass linters (flake8, black). This professionalism aids maintenance. The scraper should have continuous integration running tests, linting, and type checking on each pull request. GitHub Actions can automate this. The scraper should have a badge in README showing CI status, test coverage, and version, signaling quality. The scraper should be listed on PyPI for easy installation, though it's primarily a tool, not a library. PyPI distribution increases discoverability. The scraper should have a demo video or animated GIF showing usage, helpful for visual learners. The scraper should have a FAQ addressing common issues: memory errors, timeout errors, empty results. This reduces support burden. The scraper should have a troubleshooting guide with step-by-step diagnostics for failures. Include common error messages and fixes. The scraper should log the exact API URL and parameters for each request, enabling replay of failed requests for debugging. This verbose logging is essential for diagnosing intermittent issues. The scraper should have a debug mode that prints extra information without altering data flow. Debug mode should be activated by environment variable. The scraper should include performance benchmarks against previous versions, ensuring optimizations don't regress. Track metrics over time. The scraper should be compatible with data validation frameworks like Great Expectations or Pandera, though these add complexity. Start simple with custom validation. The scraper should output data in a normalized schema (third normal form) if possible, though denormalized CSV may be simpler for users. Consider both formats. The scraper should include a data dictionary with field descriptions, data types, and example values. This metadata is crucial for analysis. The scraper should generate data quality reports: missing values, outliers, inconsistencies. These reports help users assess data reliability. The scraper should version the data schema separately from the code, allowing independent evolution. Use a schema registry pattern. The scraper should support data subsetting by geography, time, or violation type via configuration, reducing output size for targeted use cases. This flexibility increases utility. The scraper should be able to run in streaming mode, writing records as they arrive, rather than batch. This reduces memory but increases code complexity. Batch is simpler for weekly cadence. The scraper should include a data lineage feature: track which API call produced each record, enabling traceback. This is advanced but valuable for auditing. The scraper should have a data retention policy: how many historical weeks to keep. GitHub storage is limited; consider archiving old data to cold storage. The scraper should compress output CSV (gzip) to save space, though GitHub handles compression. Compression reduces download time for users. The scraper should include a checksum file for the compressed archive, ensuring integrity during transfer. The scraper should support incremental updates if the API adds a 'modified_since' parameter later. Design with this in mind. The scraper should be able to backfill historical data by iterating over past date ranges, though the API may not support historical queries. If it does, backfill once to create baseline. The scraper should validate that scraped data is temporally consistent: no future dates, no dates before EPA establishment (1970). This sanity check catches API bugs. The scraper should geocode addresses if needed for spatial analysis, though that's a separate service. EPA may already provide coordinates. The scraper should cross-reference with other datasets (TRI, RCRA) to enrich data, though that expands scope significantly. Start simple. The scraper should detect and handle API version deprecation warnings in response headers, proactively updating before cutoff. The scraper should monitor the EPA's GitHub for API client libraries, possibly adopting official SDK if released. This reduces maintenance. The scraper should be prepared for the API moving to a new domain (e.g., api.epa.gov) with redirects. Handle HTTP 301/308 permanently moved responses. The scraper should verify SSL certificates to prevent MITM attacks, though this is standard in requests library. The scraper should use persistent HTTP connections (session) to reduce overhead across paginated requests. The scraper should implement connection pooling for efficiency if making many requests. The scraper should cache responses that are unlikely to change between pagination requests, though careful with dynamic data. The scraper should respect Cache-Control headers if present, though APIs typically disable caching. The scraper should handle gzip compression automatically via Accept-Encoding header, reducing bandwidth. The scraper should set a meaningful Referer header (the API documentation page) to appear legitimate. The scraper should include Accept header for application/json to ensure JSON response. The scraper should handle both JSON and XML responses if API supports both, though JSON is simpler. The scraper should parse dates consistently to ISO 8601 format, avoiding locale issues. The scraper should handle timezone conversion if dates are in local time; EPA likely uses EST. The scraper should validate numeric ranges: violation counts should be positive integers, latitudes/longitudes within US bounds. The scraper should clean text fields: strip whitespace, normalize Unicode, remove control characters. The scraper should handle HTML entities if API returns them (unlikely in JSON). The scraper should be robust to schema evolution: new fields appear, old fields deprecated. Use schema-on-read approach: extract known fields, ignore unknown, log new fields for review. The scraper should maintain a mapping of API field names to cleaner column names for CSV. This mapping should be configurable. The scraper should produce a changelog of data schema changes between weeks, helping users adapt their analyses. The scraper should include a data usage policy: allowed uses, citation requirements. This clarifies terms for commercial users. The scraper should track data freshness: time between violation occurrence and appearance in API. This lag analysis informs data interpretation. The scraper should measure data completeness: percentage of facilities with full address, percentage with coordinates. This quality metric guides improvement efforts. The scraper should identify data anomalies: spikes in violations by state could indicate reporting change, not actual increase. Anomaly detection adds intelligence. The scraper should correlate with external events: hurricanes, policy changes, enforcement initiatives. This contextual analysis adds value. The scraper should forecast future violations using simple time series models, though this is advanced analytics. Start with descriptive statistics. The scraper should cluster facilities by violation patterns to identify systemic offenders. This insight could drive enforcement targeting. The scraper should rank states by violation rates per capita or per facility, controlling for industrial base. This normalized comparison is fairer. The scraper should track repeat offenders: facilities with multiple violations over time. This recidivism analysis is key for enforcement. The scraper should calculate trends: are violations increasing or decreasing nationally, by state, by industry? This trend analysis answers policy questions. The scraper should benchmark facilities against peers in same industry, identifying outliers. This comparative analysis highlights underperformers. The scraper should map violations geographically for visual analysis. GeoJSON output enables web mapping. The scraper should generate summary statistics for media, policymakers, researchers: top 10 violators, most common violation types, trends. This digest increases accessibility. The scraper should produce automated reports (PDF, HTML) for subscribers, though this adds complexity. Start with raw data. The scraper should have an API itself, serving the historical data, though GitHub serves as simple API via raw CSV URLs. The scraper should consider a database backend (PostgreSQL, DuckDB) for querying large history, though CSV files are simpler for now. The scraper should version the dataset using semantic versioning: MAJOR for schema breaks, MINOR for new fields, PATCH for data corrections. This communicates changes to users. The scraper should have a data citation template: 'EPA ECHO Historical Archive, Version X.Y.Z, Retrieved [date], https://github.com/...'. This facilitates academic use. The scraper should track downstream usage via optional analytics (privacy-respecting) to understand impact. The scraper should engage with data journalists who can use the data for investigative reporting, amplifying impact. The scraper should partner with environmental NGOs for advocacy use, creating political pressure for cleaner compliance. The scraper should be presented at conferences (FOSS4G, PyData) to attract contributors and users. The scraper should submit to data science competitions (Kaggle) using the dataset, driving adoption. The scraper should create a community forum (Discourse, Slack) for users to ask questions and share analyses. The scraper should offer training workshops on using the data, lowering barrier to entry. The scraper should develop curriculum materials for universities, embedding in environmental science courses. The scraper should seek media coverage in tech and environmental publications, raising profile. The scraper should apply for awards (Open Data Award) for recognition. The scraper should measure environmental impact: how many violations were corrected due to increased transparency? This outcome measurement justifies effort. The scraper should iterate based on user feedback, prioritizing features most requested. This user-centered design ensures relevance. The scraper should sunset gracefully if replaced by official EPA historical archive, redirecting users to superior source. The scraper's legacy would be proving demand for such a service, spurring official action. This is the ultimate success: making itself obsolete by forcing the EPA to provide proper historical access. Until then, it fills a critical gap with professional, reliable service. This operational depth transforms a simple scraper into a public utility.

Go / No-Go Trigger

Successful test query to the EPA ECHO Detailed Facility Report API endpoint (https://echodata.epa.gov/echo/dfr_rest_services.get_facilities) returns HTTP 200 with valid JSON containing at least 100 facility records when querying for p_snc=Y AND p_qiv>=3 within the last 365 days. This confirms the API is live, accessible without authentication, and contains the target violation data.

Required Capabilities

• Vector: Basic Python Scripting

  Primary executor: Phase 1: API Reverse Engineering & Configuration: Reverse-engineer the exact EPA ECHO Detailed Facility Report API endpo

• Vector: API Data Extraction

  Supporting vector for: Capture National EPA Violation Dataset via ECHO API Weekly Scraping

Execution Protocol