Caching SearXNG

This revision is from 2024/03/17 15:17. You can Restore it.

SearXNG installs itself on /usr/local/searxng/searxng-src, with the main source code in searxng-src directory.

To hack the results, the file is webapp.py in /usr/local/searxng/searxng-src/searx/webapp.py

The function in webapp.py is...

@app.route('/search', methods=[['GET', 'POST']])

def search():

To get a cache working...

  • make a directory in the searx folder and name it cache
  • make a database that holds the keyword to cache file pairs
  • ideally we want to make sub-folders in the cache directory from a to z, 26 folders

The sqlite3 database named: searxng_cache.db

  1. stores the search terms
  2. stores the location of the cache file

The database is distributed, the keywords have to go to the maintainers so they can update the cache. They can then crawl the search engines and build a more comprehensive cache.

CREATE TABLE IF NOT EXISTS "search_terms" (

"id" INTEGER PRIMARY KEY AUTOINCREMENT,

"query" TEXT UNIQUE,

"file" TEXT

);

Proposed searXNG options:

  • use cache
  • update the cache
  • disclosure to end user

Benefits:

  • turns searXNG into a full search engine built from caching results
  • searches are against a local file, so it speeds up searching significantly
  • offline searching if the cache gets big enough

Refactoring def search():

Step 1: Does the search term exist in the database?

  • NO
    • Send the search term to be crawled
    • Proceed with no cache search result
  • YES
    • Display the database cached results
  

📝 📜 ⏱️ ⬆️