Using Hermes Agent and Grok to create bibliography database…

So, I’ve been playing with using Hermes Agent attached to Grok or ChatGPT (the two services I use) to catalogue my zillion of downloaded articles.  This has been a problem for me for years.  I have, I think, thousands of downloaded PDFs of various academic articles and books.  Most of them are sitting idle in old backup files going back over 10 years.  Every once in a blue moon, I think, “Hey, I ought to grab all those and put them in a database.”   But it is *so* much work to go to each directory, find the pdfs, read them, enter the data into a database or citation software, etc.  It’s easier just to download the file again.

Life is short.  Eat dessert first.  Plus, I’m old.  I don’t have that much time left, and I don’t think God wants me to spend it on that kind of make work.

Don’t get me wrong, God believes in work.   In his second letter to the church at Thessalonika, Paul excoriates church members who were lazy and took advantage of the charity of their brethren when they didn’t need to.  Paul writes:

Brothers and sisters, we command you in the name of our Lord Jesus Christ to stay away from every brother or sister who lives an undisciplined life that is not in line with the traditions that you received from us. You yourselves know how you need to imitate us because we were not undisciplined when we were with you. We didn’t eat anyone’s food without paying for it. Instead, we worked night and day with effort and hard work so that we would not impose on you. We did this to give you an example to imitate, not because we didn’t have a right to insist on financial support. Even when we were with you we were giving you this command: “If anyone doesn’t want to work, they shouldn’t eat.” We hear that some of you are living an undisciplined life. They aren’t working, but they are meddling in other people’s business. By the Lord Jesus Christ, we command and encourage such people to work quietly and put their own food on the table.

So there’s that.  But I don’t think that God means us to be doing busywork.  We should work, and we should be good craftsmen. The issue of charity involving people who could work but feel entitled is an issue for another day.  But regardless, work isn’t everything.  We shouldn’t make it the center of our lives (a sin I was subject to at points in my life).  Solomon noted in Psalm 112:

If God doesn’t build the house,the builders only build shacks.
If God doesn’t guard the city, the night watchman might as well nap.
It’s useless to rise early and go to bed late, and work your worried fingers to the bone.
Don’t you know he enjoys giving rest to those he loves?

The old saying that we should work smarter, not harder is not scripture –though I think it should be.  Solomon does say in Ecclesiastes 10:

Remember: The duller the ax the harder the work;
Use your head: The more brains, the less muscle.

That’s a paraphrase from “The Message” Bible, not a “real” translation, but I like it ,so I’ll run with it.

I thought, you know, if I can get Grok to do this while I go eat a burrito, it’s a win-win.

And it works!  Kinda.  I downloaded Hermes Agent and hooked it up to my Grok account (you don’t have to use the API for either of them for this kind of thing).  I turned both Grok and ChatGPT loose and both worked well.  Again, Kinda.  ChatGPT offers a whole bunch of models to use and, not surprisingly, the newer and better models do better.  I have a subscription to “SuperGrok” so I don’t know what the free or cheaper version would do.

The only problem is that I *also* discovered that I had some limits on how much I can do with the subscription service.  ChatGPT has limits it sets in 5 hour blocks and week blocks, and Grok has week blocks.  It turns outthat looking at 21 pdfs and  making an Excel spreadsheet of the results takes about 1-2 percent of my 5 hour allocation in ChatGPT and about 1% of my Grok allocation. I have to either spread out the agent’s work, or pony up a couple bucks now and then.  For Grok, I think a reset costs five bucks.

One way to decrease usage is to load a lot of the stuff onto local resources.  In other words, instead of having Grok read the pdf file, have Hermes use pdftotext (a linux tool to extract text from pdf files) and have Grok look at the text file.  Same thing for optical character recognition tools.   I’m putting my Hermes agent instructions at the end of this.  Unfortunately, the linux tools are not the best in the world.  Fortunately, I can do a second pass using Grok or ChatGPT on the failures and they both do well.  So fsr, I’ve just been playing with a few files at time, but I am pleased.

Has anybody done this with Grok or ChatGPT at the subscription, not API level?  What did you use as instructions, and how successful were you, particularly about using resources sparingly?

I estimate that I have about 8000 documents to scan….

Drop me a line.  Or leave a comment.

 

Here’s the prompt for Hermes Agent.  Grok wrote most of it.  The spreadsheet it produced is at the bottom.  Took about 4 minutes.  No work on my part other than setting it up.

You are running a bounded local PDF inventory on a Linux laptop. Maximize local compute. Minimize Grok tokens, context, and tool-gateway calls.

Do not upload PDFs. Do not paste full PDF text into the chat. Do not attach files to the model. Do not call web search, DOI resolvers, PubMed, browser, image generation, or TTS. If a local command can do it, use the terminal.

GOAL
Recursively scan one root directory, find PDFs, classify each file, extract type-specific fields, and write an Excel workbook. TEST RUN: at most 50 PDFs.

ROOT
REPLACE_WITH_PATH

If the path is missing or unreadable, stop. Do not search $HOME, /mnt, or other disks.

HARD LIMITS
– Max 50 PDFs processed.
– If more than 50 exist, take the 50 most recently modified. Tie-break by full path.
– Only *.pdf / *.PDF files.
– Do not modify, move, rename, decrypt-in-place, OCR-overwrite, or delete source PDFs.
– Do not follow symlinks that escape ROOT.
– Skip zero-byte files.
– Never send raw PDF bytes anywhere.
– For the model, each file may contribute at most ~1200 words of extracted text, already truncated locally.

LOCAL-FIRST POLICY
Grok is only a classifier/normalizer over compact JSON records. All discovery, metadata, text extraction, scan detection, OCR, truncation, and spreadsheet writing happen in the shell or local Python.

Preferred local tools, in this order:

Discovery / file facts
– find, stat, python3

PDF metadata
– pdfinfo (poppler-utils)
– pdfinfo -meta
– exiftool if present
– python3 + pypdf or pikepdf if present

Fonts / scan detection
– pdffonts
– pdfimages -list

Text extraction
– pdftotext -layout -enc UTF-8
– First pass: pages 1-5
– Second pass if needed: last 2 pages
– Fallback: python3 + pypdf or pymupdf

OCR only if native text is inadequate
– pdftoppm (png, 200 dpi, first 3 pages only) + tesseract
– Do not OCR more than 3 pages per file
– Do not run ocrmypdf against the original file
– Write OCR text to the work dir only

Spreadsheet
– python3 + openpyxl (preferred)
– fallback: write CSV, then:
soffice –headless –convert-to xlsx
– fallback: xlsxwriter
Do not hand-build the workbook in prose.

WORK DIRECTORY
Create:
/tmp/pdf_inventory_$$/
Use it for text dumps, OCR images, JSONL, and logs. Do not write into the PDF folders except the final .xlsx if ROOT is writable.

BOOTSTRAP
1. Check tools:
command -v find pdfinfo pdftotext pdffonts pdftoppm tesseract python3 exiftool soffice
python3 -c “import openpyxl” and “import pypdf” or “import fitz”
2. If poppler tools are missing, try: python3 extractors only. If those are also missing, stop and print the install lines:
sudo apt install poppler-utils tesseract-ocr tesseract-ocr-eng python3-openpyxl
Do not invent results.
3. Print the tool matrix once, then continue.

PHASE 1 — LOCAL INVENTORY (no model reasoning)
In the terminal, produce:
/tmp/pdf_inventory_$$/candidates.jsonl

Fields per line:
– full_path, filename, parent_folder, file_size_bytes, last_modified_iso, pages, encrypted, text_source

Selection:
– List every PDF under ROOT.
– Sort by mtime descending, path ascending.
– Keep first 50.

PHASE 2 — LOCAL EXTRACTION (no model reasoning)
For each of the 50 files, run a local extractor (a single python3 script is best; shell is fine).

For each file write:
/tmp/pdf_inventory_$$/extracts/<file_id>.json

JSON schema:
{
“file_id”: 1,
“full_path”: “”,
“filename”: “”,
“parent_folder”: “”,
“file_size_bytes”: 0,
“last_modified”: “”,
“pages”: null,
“encrypted”: false,
“embedded_meta”: {
“title”: “”, “author”: “”, “subject”: “”, “keywords”: “”,
“creator”: “”, “producer”: “”, “creation_date”: “”, “mod_date”: “”
},
“has_fonts”: null,
“image_only_likely”: false,
“text_source”: “pdf_text|ocr|metadata_only|unreadable”,
“first_pages_text”: “”,
“last_pages_text”: “”,
“ocr_pages”: 0,
“extract_error”: “”
}

Extraction rules:
– pdfinfo for pages, encryption, title/author/dates.
– pdffonts: if no fonts and pdfimages shows large images, mark image_only_likely=true.
– If encrypted and no password, text_source=unreadable and skip text.
– pdftotext pages 1-5 into first_pages_text.
– If that text, stripped, has fewer than 400 characters AND image_only_likely or no fonts, OCR pages 1-3 only.
– Also extract last 2 pages only when native text from the start is short but the file has many pages (possible back-matter / copyright page for books).
– Truncate first_pages_text to 9000 characters.
– Truncate last_pages_text to 3000 characters.
– Collapse runs of whitespace. Keep line breaks.
– Do not include binary, hex dumps, or embedded XML streams.

Then write a compact model card file:
/tmp/pdf_inventory_$$/cards.jsonl
Each card must be small:
– file_id, filename, parent_folder, pages, embedded_meta
– text_source, image_only_likely
– first_120_lines of extracted text OR first 1500 characters, whichever is shorter
– last_40_lines only if last_pages_text was extracted
No full-text cards.

PHASE 3 — MODEL CLASSIFICATION (Grok, cards only)
Read cards.jsonl. Do not reopen the PDFs. Do not cat extracts/*.json in full unless a single card is ambiguous, and then only that one file.

Assign exactly one primary_type:
– scientific_article
– book
– book_chapter
– thesis_dissertation
– conference_paper
– preprint
– medical_record
– trial_transcript
– deposition
– trial_report_or_expert_report
– court_order_or_opinion
– legal_pleading
– government_or_agency_report
– other

Optional secondary_type, else blank.

Cues:
– article/preprint/conference: Abstract, Keywords, DOI, journal, volume(issue):pages, received/accepted, arXiv/bioRxiv, IMRaD, references
– book/chapter: title page, publisher, ISBN, edition, CIP/copyright verso, chapter list
– medical_record: patient, MRN, DOB, hospital/clinic, provider, progress note, discharge summary, labs; confidential
– trial_transcript: caption, case number, court, official transcript, line numbers, direct/cross, witness
– deposition: Deposition of, under oath, noticing attorney, reporter certificate
– trial/expert report: expert credentials, retaining party, opinions, Rule 26, case caption
– order/opinion: judge, IT IS ORDERED, memorandum
– pleading: complaint/motion/brief, counsel block, certificate of service

Do not invent DOI, ISBN, case number, patient name, publisher, or dates. Blank is required when unknown.

Type-specific fields to fill when present:

scientific_article / preprint / conference_paper:
title, authors (semicolon-separated), year, journal_or_venue, volume, issue, pages_citation, doi (bare 10.x/…), pmid_pmcid, keywords, abstract_present

book / book_chapter:
title, authors_or_editors, year, publisher, place_of_publication, edition, isbn, chapter_title

medical_record:
patient_name, patient_identifiers_found, facility_or_hospital, provider_or_caretaker, record_type, encounter_or_record_date

trial_transcript:
case_name, case_number, court, hearing_or_trial_date, witness_or_speaker, proceeding_type, volume_or_pages

deposition:
case_name, case_number, deponent, deposition_date, taking_attorney_or_firm, location

trial_report_or_expert_report:
case_name, case_number, report_author, author_role, report_date, subject_of_report

Always also fill:
classification_confidence (high|medium|low)
extraction_notes
short_evidence (<=240 chars, copied from the card)

If a PDF is a mixed packet, classify the first dominant document and note multi-document packet.

Write classifications to:
/tmp/pdf_inventory_$$/classified.jsonl
One JSON object per file, same file_id.

If there are 50 cards, classify in batches of 10 locally-prepared groups. Do not reload earlier full cards.

PHASE 4 — LOCAL SPREADSHEET
Join candidates + extracts + classified with python3/openpyxl.

Output workbook:
REPLACE_WITH_PATH/_pdf_inventory_test.xlsx
If ROOT is not writable:
$HOME/Desktop/pdf_inventory_test.xlsx

Sheet “inventory” — one row per PDF, columns in this order:
file_id
full_path
filename
parent_folder
file_size_bytes
last_modified
pages_if_known
text_source
primary_type
secondary_type
classification_confidence
title
authors_or_editors
year
journal_or_venue
volume
issue
pages_citation
doi
pmid_pmcid
keywords
publisher
isbn
edition
patient_name
facility_or_hospital
provider_or_caretaker
record_type
encounter_or_record_date
case_name
case_number
court
proceeding_date
witness_or_deponent_or_report_author
author_role
proceeding_or_report_type
subject_or_testimony_summary
extraction_notes
short_evidence

Sheet “summary”:
root, start/finish time, PDFs found, processed, skipped reasons, counts by primary_type, unreadable/encrypted/scanned counts, output path, remaining unprocessed count due to the 50 cap, tools used

Sheet “skipped”:
path, reason for files attempted but unreadable

Formatting:
frozen header, bold, autofilter, no merged cells, ISO dates when possible.

After writing, verify with:
python3 -c “import openpyxl; wb=openpyxl.load_workbook(‘PATH’); print(wb.sheetnames, wb[‘inventory’].max_row)”

CLEANUP
Leave /tmp/pdf_inventory_$$/ in place until I confirm. Do not delete source PDFs. You may delete OCR PNGs if disk is tight; keep jsonl and the xlsx.

WHEN DONE
Print only:
1. xlsx path
2. found / processed / skipped
3. type counts
4. tools actually used
5. five files that need manual review
6. estimate of PDFs not processed because of the cap

Then stop.

FAILURE STYLE
If a tool errors on one file, record extract_error and continue. Do not abort the batch for a single bad PDF.

 

And here’s the result (with a few entries deleted to preserve privacy for some folk)”

file_idfilenamefile_size_byteslast_modifiedpages_if_knowntext_sourceprimary_typesecondary_typeclassification_confidencetitleauthors_or_editorsyearjournal_or_venuevolumeissuepages_citationdoipmid_pmcidkeywordspublisherisbneditionextraction_notesshort_evidence
1Forensic Pathology, Second Edition (Practical Aspects of -- Dominick DiMaio; Vincent J_M_ DiMaio, M_D_ -- Taylor & Francis (Unlimited), Boca Raton, -- isbn13 9780429248108 – 56497a4f01c3408fbcfb9284ce0cec30.pdf1128880322026-09-04T02:52:34+00:00563pdf_textbookhighForensic PathologyDominick J. Di Maio; Vincent J. M. Di Maio2001CRC PressSecond EditionTitle page and CRC series list; copyright 2001 CRC Press LLC. ISBN not present in excerpt.Forensic Pathology Second Edition © 2001 by CRC Press LLC CRC SERIES IN PRACTICAL ASPECTS OF CRIMINAL AND FORENSIC INVESTIGATIONS
2d8d1b913bef9.pdf377269132026-09-04T02:30:20+00:00545pdf_textbookhighDiMaio’s Forensic PathologyVincent J.M. DiMaioCRC PressThird EditionTitle page plus embedded metadata title/author/subject. Publication year not stated on card excerpt.DiMaio’s Forensic Pathology Third Edition CRC SERIES IN PRACTICAL ASPECTS OF CRIMINAL AND FORENSIC INVESTIGATIONS
3ernsting1963.pdf20242292026-09-04T02:19:44+00:0020pdf_textscientific_articlehighThe effect of brief profound hypoxia upon the arterial and venous oxygen tensions in manJ. Ernsting1963J. Physiol.169292-311Journal running head with volume/pages; received date present. DOI/PMID not in excerpt.292 J. Phy8iol. (1963), 169, pp. 292-311 THE EFFECT OF BRIEF PROFOUND HYPOXIA UPON THE ARTERIAL AND VENOUS OXYGEN TENSIONS IN MAN BY J. ERNSTING
4harding2008.pdf3113842026-09-04T02:09:41+00:003pdf_textscientific_articlehighCase Report of Suicide by Inhalation of Nitrogen GasBrett E. Harding; Barbara C. Wolf2008Am J Forensic Med Pathol29235-237nitrogen; suffocating gas; suicideLabeled CASE REPORT with journal citation line.CASE REPORT Case Report of Suicide by Inhalation of Nitrogen Gas (Am J Forensic Med Pathol 2008;29: 235–237)
6kojima1986.pdf6924092026-08-31T21:09:31+00:0011pdf_textscientific_articlehighProduction of carbon monoxide in cadaversT. Kojima; I. Okamoto; M. Yashiki; T. Miyazaki; F. Chikasue; K. Degawa; S. Oshida; K. Sagisaka1986Forensic Science International3267-77OCR-ish characters in names/title; normalized from visible byline. Elsevier Ireland.Forensic Science International, 32 (1986) 67-77 PRODUCTION OF CARBON MONOXIDE IN CADAWRS T. KOJIMA
7s11109-025-10118-3.pdf19812812026-08-29T02:35:28+00:0033pdf_textscientific_articlehighJust a Little Melancholic, Maybe a Little Blue: Mental Health as an Emerging Political IdentityLauren Van De Hey2026Political Behavior10.1007/s11109-025-10118-3Original paper; accepted 11 December 2025; © The Author(s) 2026. Volume/issue/pages not on excerpt.Political Behavior https://doi.org/10.1007/s11109-025-10118-3 ORIGINAL PAPER Just a Little Melancholic, Maybe a Little Blue
82024 PIT Subpopulation Data Full Count.pdf1049772026-08-21T04:28:04+00:0015pdf_textgovernment_or_agency_reporthigh2024 PIT Subpopulation Data Full Count2024GA-501 Georgia Balance of State Continuum of CareHUD Point-in-Time count tables for Georgia counties; Excel-origin PDF. Not a journal article.REPORTING ENTITY State: Georgia Count Date: 1/23/2024 Continuum of Care: GA-501 Georgia Balance of State CoC
92024 BoS Final PIT Summary Data.pdf1072652026-08-21T04:27:20+00:002pdf_textgovernment_or_agency_reporthigh2024 BoS Final PIT Summary DataJosh Gray2024GA-501 Georgia Balance of State Continuum of CarePIT household/age/gender/race summary for Entire CoC. Author from embedded metadata.REPORTING ENTITY State: Georgia Count Date: 1/23/2024 GA-501 Georgia Balance of State CoC Region: Entire CoC Total Number of Persons 6633
10reviewMatPatFilicide.pdf862452026-08-17T23:43:37+00:009pdf_textscientific_articlehighA Review of Maternal and Paternal FilicideDominique Bourget; Jennifer Grace; Laurie Whitehurst2007J Am Acad Psychiatry Law3574-82Labeled REGULAR ARTICLE; review of literature.REGULAR ARTICLE A Review of Maternal and Paternal Filicide J Am Acad Psychiatry Law 35:74 – 82, 2007
11s41598-026-65785-x_reference.pdf28735842026-08-14T00:18:20+00:0029pdf_textscientific_articlepreprinthighRetrospective analysis of microbial contamination in tattoo and permanent makeup inksSoumana Daddy Gaoh; Sunghyun Yoon; Sandeep Kondakala; Minjae Kim; Seongwon Nho; Steven L. Foley; Seong-Jae Kim; Ohgew Kweon2026Scientific Reports10.1038/s41598-026-65785-xArticle in Press / unedited early access; received 15 April 2026, accepted 3 August 2026. U.S. Government work.Scientific Reports https://doi.org/10.1038/s41598-026-65785-x Article in Press Retrospective analysis of microbial contamination in tattoo and permanent makeup inks
13leadandcrime_2.pdf1116762026-07-19T21:13:23+00:0016pdf_textscientific_articlehighThe Relationship between Lead and CrimePaul B. Stretesky; Michael J. Lynch2004Journal of Health and Social Behavior452214-229Running head and journal citation line present. DOI not in excerpt.The Relationship between Lead and Crime Journal of Health and Social Behavior, 2004, Vol 45 (June): 214–229
1400000433-199409000-00002 – 817e8d712cc5efdd6c5b32d6b768cb3b.pdf3244592026-07-15T23:11:35+00:005ocrscientific_articlehighPatterns of Facial Resuscitation Injury in InfancyJames A. Kaplan; Roger M. Fossum1994The American Journal of Forensic Medicine and Pathology153187-191Cardiopulmonary resuscitation artifact; Facial injury; Sudden infant death syndrome; TraumaImage-only PDF; text from 3-page OCR. Author initials slightly garbled in OCR (M.p.).The American Journal of Forensic Medicine and Pathology 15(3):187-191, 1994. Patterns of Facial Resuscitation Injury in Infancy
15LaQuinta.pdf1175442026-07-13T22:31:02+00:003pdf_textotherhighLaQuinta Terms and Conditions2026Hotel/guest Wi-Fi terms printout (browser PDF), not a scholarly or legal case document.7/13/26, 6:30 PM LaQuinta Terms and Conditions By using our internet service, you hereby expressly acknowledge
16antiphospholipid.pdf3502002026-07-01T02:56:59+00:007pdf_textscientific_articlehighNeonatal Effects of Maternal Antiphospholipid SyndromeAngela Tincani; Chiara Biasini Rebaioli; Laura Andreoli; Andrea Lojacono; Mario Motta2009Current Rheumatology Reports1170-76Review-style article with corresponding-author block.Neonatal Effects of Maternal Antiphospholipid Syndrome Current Rheumatology Reports 2009, 11:70–76
171040120150108.pdf4059632026-06-17T23:50:07+00:0013pdf_textscientific_articlemediumA review on pistachio: Its composition and benefits regarding the prevention or treatment of diseasesGhaseminasab Parizi M; Ahmadi A; Mazloomi SM2016Pistacia vera; chemical composition; anticancer; anti-inflammation; antidiabetic; antioxydativeReview article; journal name not printed on first-page card (only Received/Accepted 2016).Review Article A review on pistachio: Its composition and benefits regarding the prevention or treatment of diseases Abstract Received: January 2016, Accepted: March 2016
18PET Whole Body with CT ATTENTUATION 06-17-2026.pdf117102026-06-17T16:00:45+00:003pdf_textmedical_recordhighPET Whole Body with CT ATTENTUATIONContains patient identifiers (name, MRN, DOB). Radiology report with accession PE-26-0004162.The University of Tennessee Medical Center Patient Name: OLIVER, WILLIAM R Positron Emission Tomography PET Whole Body with CT ATTENTUATION
19annas-arch-67e1dd4795e0.pdf9377752026-06-13T00:25:17+00:0026pdf_textbook_chapterhighFreeCAD Beginner’s StarterkitAleksander Sadowski2024Chapter 1 of a beginner book; copyright line present. Publisher/ISBN not on excerpt.FreeCAD Beginner’s Starterkit: Chapter 1 - Working with 3D shapes on paper Copyright © 2024 Aleksander Sadowski
21Infant_sexual_abuse_1978.pdf16195392026-06-04T19:09:33+00:008pdf_textscientific_articlemediumSexual Abuse, Another Hidden Pediatric Problem: The 1977 C. Anderson Aldrich LectureC. Henry KempePediatricsNamed lecture; Pediatrics affiliation line. Publication year not stated on excerpt (lecture year 1977).Sexual Abuse, Another Hidden Pediatric Problem: The 1977 C. Anderson Aldrich Lecture C. Henry Kempe, M.D. From the National Center for Prevention and Treatment of Child Abuse and Neglect

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.