ankitjh4/bharat-government-documents

Bharat Guide: screened Indian public information documents

This snapshot contains 64,964 distinct normalized text bodies and 77,526 source records. Generated 2026-10-03T01:10:02.179230+00:00.

Contents and provenance

One Parquet row represents one normalized source body, with original extracted text, source title, URL, extraction quality, observed retrieval time, rule version, topic labels and all current-body source aliases. Duplicate URLs/editions are preserved as provenance, not counted as additional distinct documents. The canonical record is the lexicographically smallest eligible document identifier. Topic labels overlap.

Selection requires an eligible assessment whose body checksum still matches the current document, with full_text or ocr_text extraction. This is heuristic screening, not manual factual validation. Older and newer screening versions coexist and are reported in manifest.json. PTI secondary news, directory-only entries, summary-only records, rejected text and OCR awaiting review are not qualifying rows. A reviewed alias can be retained inside an eligible row's provenance, explicitly labelled. The text exports contain no raw response bytes or account credentials. The combined CSV labels screened and review documents explicitly; failure records are separate. The separate SQLite backup retains original evidence and operational records; its manifest records its own snapshot date.

Limitations

This is not a complete census of Indian websites, laws, schemes or facilities. Government-adjacent institutional publications are labelled separately from government sources. Retrieval time is not publication time. Source validity, amendments, legal force, policy deadlines and numerical OCR accuracy require checking the original. Exact normalized-body deduplication does not establish semantic equivalence.

Rights and use

Rights remain with each original publisher. Government provenance does not establish that every source is public domain or has one common reuse licence. No blanket licence is granted over collected source text. Consult each source's terms and applicable rights before redistribution or other reuse. Publication does not change the rights retained by each original source.

Releases

manifest.json gives checksums and row counts. publication.json tracks confirmed publication content hashes and the additional-document threshold. Each update is committed atomically with its data and manifest; the next release is due after at least 10,000 newly qualifying distinct bodies, not 10,000 fetched URLs or chunks.

Download layout and CSV fields

documents.csv.gz is sorted by the full url value in ascending, case-sensitive order.

documents.csv.gz is the main download: one deduplicated, nonempty extracted text body per row, combining screened and review content. The status column is good or review. Only good rows count toward the screened-document milestone. Review rows may include thin text, directory entries, news or incomplete extraction; keep them separate when evaluating quality. Duplicate source aliases are preserved in provenance/sources.csv.gz without repeating their text in the main CSV.

The CSV contains: content_hash, document_id, title, text, url, host, kind, trust, quality, state, category, fetched_at, metadata, sha256, status, assessment_version, assessment_status, assessment_reasons, topics. text is the full stored extracted body, not a summary. metadata preserves the representative document's SQLite JSON metadata. topics is a JSON list, and labels may overlap; unclassified documents are explicitly labelled. Fetch time is not publication time. A row represents a document body, not every relational SQLite row.

topics/document_topics.csv.gz maps topics to content hashes and good/review status; topics/summary.json provides counts. This index avoids copying full text into many folders. Topic labels are heuristic, not manual factual or legal validation.

fails/urls.csv.gz contains failure, robots-blocked, retry and OCR-pending records with their exact statuses; not every entry is a permanent failure. fails/empty_documents.csv.gz lists records without usable text. Neither is counted as a distinct text document. Unattempted queued URLs and seed inventories are not exported. No crawler handoff package is included.

CSV files use UTF-8, gzip compression and standard quoting for multiline text. Decompress to obtain an ordinary CSV, or read directly:

import pandas as pd
df = pd.read_csv("documents.csv.gz")
good = df[df["status"] == "good"]

Optional data/train-*.parquet files contain screened text in larger batches for the Hugging Face viewer. The CSV contains additional review content, so its row count is higher than the qualifying-document count. Source values are preserved; treat spreadsheet formulas in source text as untrusted, not executable instructions.

SQLite reference

SQLITE_SCHEMA.md lists every table and field; sqlite_schema.sql contains the reference schema. documents stores text and metadata; document_assessments stores screening decisions; document_topics stores labels. Original PDFs and responses are in raw_blobs.payload, linked through fetch_receipts.raw_sha256, with codec describing zlib compression or raw bytes. ocr_pages stores page-level results. These binary files and operational tables are not embedded in the CSV.

The private recovery files backups/bharat.sqlite and backups/sqlite-manifest.json retain their own snapshot date and checksum, which may precede the text release. The full SQLite backup includes crawl queues and source inventories even though this CSV release excludes them. Keep that backup private when publishing text-only data.

Sources and subject coverage

Sources include Indian central, state and local government portals; courts and legislative institutions; regulators and public-sector bodies; universities and research institutions; and other government-adjacent institutional sources. Domain endings alone do not establish ownership or reliability. Secondary news reporting, including PTI, is distinguished from official statements such as PIB releases.

This release has text provenance from 16,122 distinct website hostnames. Subdomains count separately. These are hosts represented in the released text, not a claim that every page of each website has been crawled. Failed-only, queued and seed-only hosts are excluded.

Website domain group	Distinct hosts
.gov.in	11,598
.ac.in	1,791
Other .in	2,283
Other domain endings	450

The current classifier defines 36 broad subject labels, with the finer areas described below. The finer subject areas are descriptive; subtopic coverage is not counted independently. A website/document may appear under multiple topics, so topic totals must not be added together.

Topic host counts include all released source aliases for labelled text, including review material. Labels are automated and may be inherited across identical-text aliases. good means passed extraction/source/subject heuristics, not manual fact-checking. Examples name institution types and subject areas, not a verified inventory of every named institution.

Topic	Website hosts	Good documents	Review documents	Source types and finer subject areas
Law and judiciary	2,009	8,908	11,438	Courts, tribunals and legal-services bodies; Supreme Court, High Courts, judgments and legal aid
Schemes and benefits	3,471	10,755	4,590	Central/state scheme portals; benefits, eligibility, applications and beneficiary guidance
Policy and regulation	5,891	8,385	8,247	Ministries, departments and regulators; policies, regulations, guidelines and gazette notifications
Government statements	1,739	2,066	3,058	Official information offices and PIB; press releases, cabinet decisions and public statements
Universities and research	3,410	9,844	9,169	Universities, colleges and research institutions; admissions, curricula, accreditation and research
Religion, heritage and endowments	1,574	1,358	1,003	Heritage and endowment bodies; religious institutions, pilgrimage, archaeology and endowments
Mining and natural resources	2,412	3,367	3,623	Mining, geology and resource departments; minerals, coal, petroleum, water and forests
Tourism	1,491	3,567	3,417	Tourism departments and public tourism bodies; destinations, visitor information and travel advisories
Public finance and accounts	2,775	2,951	4,578	Finance departments, treasuries and audit bodies; expenditure, accounts, audits and appropriations
Population and statistics	1,045	972	1,008	Census and statistics agencies; population, demographics, household surveys and vital statistics
Banking and monetary policy	625	1,527	1,146	Banks and financial authorities; monetary policy, deposits, interest rates and financial stability
Other public services	3,742	8,463	7,448	Citizen-service and sector departments; agriculture, public health, sanitation, drinking water and transport
Constitution	209	149	125	Legislative and legal-information portals; constitutional text, provisions and constitutional materials
Acts, amendments, rules and legal forms	352	365	444	Legislative departments and legal repositories; Acts, amendments, repeals, rules and statutory forms
Hospitals and health facilities	188	995	1,435	Health departments and facility portals; hospitals, medical colleges, facility directories and infrastructure
Income tax	74	783	642	Tax authorities; direct taxes, returns, deductions and taxpayer guidance
Roads and public contracts	238	945	1,202	Roads departments and procurement portals; highways, tenders, contracts and awards
Public infrastructure	2,884	3,566	3,538	Public works and infrastructure agencies; railways, urban development and public projects
Digital infrastructure	395	1,401	744	Digital-government agencies; e-governance, broadband, BharatNet and digital public infrastructure
ONDC	4	13	20	Digital-commerce network information; ONDC and open-network participation
UPI and payment systems	18	5	13	Payment-system institutions; UPI, NPCI and payment-system documentation
Prisons and correctional administration	290	792	1,599	Prison and corrections departments; administration, jail manuals and aggregate reports
Juvenile justice	35	147	97	Child-welfare and justice bodies; legislation, observation homes and institutional reports
Reproductive health and rights	40	49	25	Health and rights-related public bodies; maternal health, family planning and pregnancy-termination frameworks
Minority rights	73	315	936	Minority-affairs departments and commissions; protections, programmes and public reports
Culture and arts	645	2,676	2,963	Culture departments and public cultural institutions; museums, arts and cultural programmes
Fire services and stations	227	374	321	Fire and emergency-service departments; stations, fire services and fire-safety guidance
Police services and stations	604	1,473	1,241	Police departments; stations, public services and departmental information
Credit and lending	77	167	175	Banks and credit-programme bodies; loans, guarantees, lending schemes and microfinance
Government grants and sponsorships	301	815	612	Grant-making and education departments; grants, scholarships and sponsorships
Government hospitality and hotels	59	647	720	Public tourism corporations and hospitality bodies; government hotels and state guest houses
Official contact directories	513	416	232	Official departmental directories; office contacts, telephone directories and who-is-who pages
International policy and statements	128	320	403	External-affairs and diplomatic institutions; bilateral relations, foreign policy and joint statements
Parliament and briefings	585	658	1,420	Parliamentary institutions; Lok Sabha, Rajya Sabha, briefings and parliamentary materials
Prime ministers: history and tenures	1	15	6	Official historical and institutional collections; former prime ministers and their tenures
Budgets	875	717	1,611	Finance and budget portals; budgets, demands for grants and appropriation documents
News reporting	1	0	11	Secondary news agencies, including PTI; news reporting kept distinct from official government statements
Unclassified	13,101	0	253,560	Extracted material without a subject label; retained for review, not assigned an invented topic

Agriculture currently sits within Other public services; it does not yet have a separately measured website count. Zero-count topics indicate tracked areas without labelled text in this snapshot, not completed coverage. Hospital records are not a reconciled national inventory; legal materials may include superseded versions.

For exact source URLs and titles, see provenance/sources.csv.gz. Document-level topics are in documents.csv.gz; the topic-to-document index and counts are in topics/. These files describe released material and do not include queued or seed-only website lists.