Code

Multithreading and Inverted index#23

Code

Open

Hayyan-Arshad wants to merge 127 commits intoRohan-Kamat:multithreadingRohan-Kamat/Andromeda:multithreadingfrom Hayyan-Arshad:mainHayyan-Arshad/Andromeda:mainCopy head branch name to clipboard

+8,722-157

Commits

Commits on Dec 27, 2022

• Add notes for week-1 of Text Retrieval course from Coursera

  kid-116

  committed
  0c36186

  Copy full SHA for 0c36186

Commits on Dec 30, 2022

• Add notes for week-2 of Text Retrieval

  kid-116

  committed
  d2a1266

  Copy full SHA for d2a1266

• Update NOTES.md

  kid-116

  authored
  acc12b8

  Copy full SHA for acc12b8

Commits on Dec 31, 2022

• Add week-3 notes for Text Retrival course

  kid-116

  committed
  96df0dd

  Copy full SHA for 96df0dd

Commits on Jan 2, 2023

• Add notes for week-4

  kid-116

  committed
  b2fe400

  Copy full SHA for b2fe400

• Fix image for word association

  kid-116

  committed
  5e6723a

  Copy full SHA for 5e6723a

• Add notes for week-5

  kid-116

  committed
  edd0580

  Copy full SHA for edd0580

• Add notes for week-6

  kid-116

  committed
  ddb98cd

  Copy full SHA for ddb98cd

• Fix README.md

  kid-116

  committed
  5b515a6

  Copy full SHA for 5b515a6

• Update NOTES.md

  kid-116

  authored
  a779e08

  Copy full SHA for a779e08

• Merge pull request Rohan-Kamat#20 from Rohan-Kamat/notes

  Show description for 512c6c2
  kid-116

  authored
  512c6c2

  Copy full SHA for 512c6c2

Commits on Mar 23, 2023

• Add command to flush database

  kid-116

  committed
  0ce3fde

  Copy full SHA for 0ce3fde

• Add multithreading support

  kid-116

  committed
  b144a6a

  Copy full SHA for b144a6a

• Ignore non-english pages when crawling

  kid-116

  committed
  4eed2ab

  Copy full SHA for 4eed2ab

• Store language in database

  kid-116

  committed
  6c9cb08

  Copy full SHA for 6c9cb08

• Ignore non-english words and stem them before storage

  kid-116

  committed
  f25690d

  Copy full SHA for f25690d

• Add command to crawl a single page

  kid-116

  committed
  d55d614

  Copy full SHA for d55d614

• Configure logging

  kid-116

  committed
  0b13f4f

  Copy full SHA for 0b13f4f

• Add inverted index

  kid-116

  committed
  a65debd

  Copy full SHA for a65debd

• Setup Flask API

  kid-116

  committed
  3a670d9

  Copy full SHA for 3a670d9

• Add link check for API

  kid-116

  committed
  c65286f

  Copy full SHA for c65286f

• Setup Ranker

  kid-116

  committed
  19c8468

  Copy full SHA for 19c8468

• Setup Next.JS project for frontend

  kid-116

  committed
  9bd4480

  Copy full SHA for 9bd4480

• Merge pull request Rohan-Kamat#30 from Rohan-Kamat/dev

  Show description for bba9821
  kid-116

  authored
  bba9821

  Copy full SHA for bba9821

• Fix lint erros

  kid-116

  committed
  a9b6389

  Copy full SHA for a9b6389

• Save crawler progress

  kid-116

  committed
  a26bbfc

  Copy full SHA for a26bbfc

Commits on Mar 24, 2023

• Move crawler runtime handling to main.py

  kid-116

  committed
  b867646

  Copy full SHA for b867646

• Decouple crawler and parser

  kid-116

  committed
  530841a

  Copy full SHA for 530841a

• Replace check and add MongoDB calls with upserting

  kid-116

  committed
  0fa5f77

  Copy full SHA for 0fa5f77

• Remove unnecessary download

  kid-116

  committed
  a6f376a

  Copy full SHA for a6f376a

• Add a command for debugging purposes

  kid-116

  committed
  74a49ef

  Copy full SHA for 74a49ef

• Fix setup.py

  kid-116

  committed
  869654c

  Copy full SHA for 869654c

Commits on Mar 25, 2023

• Add FREQUENCY_THRESHOLD for inverted index

  kid-116

  committed
  bc730ae

  Copy full SHA for bc730ae

• Set download dir for chrome

  kid-116

  committed
  8007630

  Copy full SHA for 8007630

• Setup pm2 support

  kid-116

  committed
  6bbc73d

  Copy full SHA for 6bbc73d

Commits on Mar 26, 2023

• Add doc for viewing logs

  kid-116

  committed
  a8c6516

  Copy full SHA for a8c6516

• Improve page parsing

  kid-116

  committed
  8ac68e9

  Copy full SHA for 8ac68e9

• Limit the number of retries when crawing a URL

  kid-116

  committed
  0d47da3

  Copy full SHA for 0d47da3

• Store crawled hosts

  kid-116

  committed
  94334a4

  Copy full SHA for 94334a4

• Change database

  kid-116

  committed
  ea17c71

  Copy full SHA for ea17c71

• Add db name to log file

  kid-116

  committed
  eb29731

  Copy full SHA for eb29731

• Remove debug print

  kid-116

  committed
  2d8f402

  Copy full SHA for 2d8f402

• Fix startup script

  kid-116

  committed
  b4dac14

  Copy full SHA for b4dac14

• Flush hosts as well on startup

  kid-116

  committed
  631e0a7

  Copy full SHA for 631e0a7

• Remove FREQUENCY_THRESHOLD from parsing

  kid-116

  committed
  cfa929d

  Copy full SHA for cfa929d

• Fix

  kid-116

  committed
  70d36d5

  Copy full SHA for 70d36d5

• Made the search result component

  

  shashank

  committed
  78dd71a

  Copy full SHA for 78dd71a

• Added new search result page

  

  shashank

  committed
  c764d0f

  Copy full SHA for c764d0f

• Enable restarting of crawler from where it left

  kid-116

  committed
  8b61b54

  Copy full SHA for 8b61b54

• Change prod settings

  kid-116

  committed
  1485fa8

  Copy full SHA for 1485fa8

• Fix load

  kid-116

  committed
  e4a219d

  Copy full SHA for e4a219d

• Fix

  kid-116

  committed
  e38e636

  Copy full SHA for e38e636

• Supply website metadata in API

  kid-116

  committed
  6d0d78a

  Copy full SHA for 6d0d78a

• Restructure Flask API

  kid-116

  committed
  53dbad8

  Copy full SHA for 53dbad8

• Add a startup script for API

  kid-116

  committed
  abb100c

  Copy full SHA for abb100c

• Fix

  kid-116

  committed
  3a9d3b2

  Copy full SHA for 3a9d3b2

• Parse relative links

  kid-116

  committed
  0dd9ad5

  Copy full SHA for 0dd9ad5

• Edit prod specs

  kid-116

  committed
  5667032

  Copy full SHA for 5667032

• Add detailed param to search route

  kid-116

  committed
  361d296

  Copy full SHA for 361d296

• Fix

  kid-116

  committed
  ffa7033

  Copy full SHA for ffa7033

• Fix

  kid-116

  committed
  44e37cf

  Copy full SHA for 44e37cf

• Remove FREQUENCY_THRESHOLD

  kid-116

  committed
  71f91e0

  Copy full SHA for 71f91e0

• Fix

  kid-116

  committed
  9a8c709

  Copy full SHA for 9a8c709

• Fix

  kid-116

  committed
  d530ed4

  Copy full SHA for d530ed4

Commits on Mar 27, 2023

• modified css of the app

  

  shashank

  committed
  30815b1

  Copy full SHA for 30815b1

• Paginate API response for search

  kid-116

  committed
  dd7cec9

  Copy full SHA for dd7cec9

• Allow CORS on API

  kid-116

  committed
  8d862f8

  Copy full SHA for 8d862f8

• Use refs in ranking

  kid-116

  committed
  23b50de

  Copy full SHA for 23b50de

• made changes to frontend

  

  shashank

  committed
  ba3bd43

  Copy full SHA for ba3bd43

• Normalise refs

  kid-116

  committed
  34a9875

  Copy full SHA for 34a9875

• Ignore .pdf links

  kid-116

  committed
  76a762e

  Copy full SHA for 76a762e

• Added pagination

  

  shashank

  committed
  78a8b1f

  Copy full SHA for 78a8b1f

• Move crawler runtime handling to main.py

  

  kid-116

  authored and

  shashank

  committed
  d69dda1

  Copy full SHA for d69dda1

• Decouple crawler and parser

  

  kid-116

  authored and

  shashank

  committed
  91fb1e5

  Copy full SHA for 91fb1e5

• Replace check and add MongoDB calls with upserting

  

  kid-116

  authored and

  shashank

  committed
  cf5e5fb

  Copy full SHA for cf5e5fb

• Remove unnecessary download

  

  kid-116

  authored and

  shashank

  committed
  a12baf9

  Copy full SHA for a12baf9

• Add a command for debugging purposes

  

  kid-116

  authored and

  shashank

  committed
  dd19506

  Copy full SHA for dd19506

• Fix setup.py

  

  kid-116

  authored and

  shashank

  committed
  646c542

  Copy full SHA for 646c542

• Add FREQUENCY_THRESHOLD for inverted index

  

  kid-116

  authored and

  shashank

  committed
  c7f9f9b

  Copy full SHA for c7f9f9b

• Set download dir for chrome

  

  kid-116

  authored and

  shashank

  committed
  3a3fcf1

  Copy full SHA for 3a3fcf1

• Setup pm2 support

  

  kid-116

  authored and

  shashank

  committed
  22d588d

  Copy full SHA for 22d588d

• Add doc for viewing logs

  

  kid-116

  authored and

  shashank

  committed
  b3bf278

  Copy full SHA for b3bf278

• Improve page parsing

  

  kid-116

  authored and

  shashank

  committed
  a6d0a7d

  Copy full SHA for a6d0a7d

• Limit the number of retries when crawing a URL

  

  kid-116

  authored and

  shashank

  committed
  839c72b

  Copy full SHA for 839c72b

• Store crawled hosts

  

  kid-116

  authored and

  shashank

  committed
  9b79d7f

  Copy full SHA for 9b79d7f

• Change database

  

  kid-116

  authored and

  shashank

  committed
  1ac5649

  Copy full SHA for 1ac5649

• Add db name to log file

  

  kid-116

  authored and

  shashank

  committed
  f217207

  Copy full SHA for f217207

• Remove debug print

  

  kid-116

  authored and

  shashank

  committed
  46c99da

  Copy full SHA for 46c99da

• Fix startup script

  

  kid-116

  authored and

  shashank

  committed
  8c5183b

  Copy full SHA for 8c5183b

• Flush hosts as well on startup

  

  kid-116

  authored and

  shashank

  committed
  6fc0248

  Copy full SHA for 6fc0248

• Remove FREQUENCY_THRESHOLD from parsing

  

  kid-116

  authored and

  shashank

  committed
  59c9cb3

  Copy full SHA for 59c9cb3

• Fix

  

  kid-116

  authored and

  shashank

  committed
  8ef8e72

  Copy full SHA for 8ef8e72

• Enable restarting of crawler from where it left

  

  kid-116

  authored and

  shashank

  committed
  1dbbe0a

  Copy full SHA for 1dbbe0a

• Change prod settings

  

  kid-116

  authored and

  shashank

  committed
  6f960df

  Copy full SHA for 6f960df

• Fix load

  

  kid-116

  authored and

  shashank

  committed
  8c8343f

  Copy full SHA for 8c8343f

• Fix

  

  kid-116

  authored and

  shashank

  committed
  194703a

  Copy full SHA for 194703a

• Supply website metadata in API

  

  kid-116

  authored and

  shashank

  committed
  e27e0db

  Copy full SHA for e27e0db

• Restructure Flask API

  

  kid-116

  authored and

  shashank

  committed
  4f11cd2

  Copy full SHA for 4f11cd2

• Add a startup script for API

  

  kid-116

  authored and

  shashank

  committed
  b600442

  Copy full SHA for b600442

• Fix

  

  kid-116

  authored and

  shashank

  committed
  586281a

  Copy full SHA for 586281a

• Parse relative links

  

  kid-116

  authored and

  shashank

  committed
  4ef081e

  Copy full SHA for 4ef081e

• Edit prod specs

  

  kid-116

  authored and

  shashank

  committed
  a262b9f

  Copy full SHA for a262b9f

• Add detailed param to search route

  

  kid-116

  authored and

  shashank

  committed
  e5d44e3

  Copy full SHA for e5d44e3

• Fix

  

  kid-116

  authored and

  shashank

  committed
  bc0e27f

  Copy full SHA for bc0e27f

• Fix

  

  kid-116

  authored and

  shashank

  committed
  245c80a

  Copy full SHA for 245c80a

• Remove FREQUENCY_THRESHOLD

  

  kid-116

  authored and

  shashank

  committed
  4ca32fa

  Copy full SHA for 4ca32fa

• Fix

  

  kid-116

  authored and

  shashank

  committed
  30444ba

  Copy full SHA for 30444ba

• Fix

  

  kid-116

  authored and

  shashank

  committed
  7c8f544

  Copy full SHA for 7c8f544

• Paginate API response for search

  

  kid-116

  authored and

  shashank

  committed
  3cdb284

  Copy full SHA for 3cdb284

• Allow CORS on API

  

  kid-116

  authored and

  shashank

  committed
  0c12203

  Copy full SHA for 0c12203

• Use refs in ranking

  

  kid-116

  authored and

  shashank

  committed
  e177237

  Copy full SHA for e177237

• Normalise refs

  

  kid-116

  authored and

  shashank

  committed
  66f01e6

  Copy full SHA for 66f01e6

• Ignore .pdf links

  

  kid-116

  authored and

  shashank

  committed
  f3b5547

  Copy full SHA for f3b5547

• Fix

  

  shashank

  committed
  43bc0b5

  Copy full SHA for 43bc0b5

• Merge pull request Rohan-Kamat#31 from Rohan-Kamat/frontend

  Show description for 3cc17c2
  kid-116

  authored
  3cc17c2

  Copy full SHA for 3cc17c2

• Improve frontend

  kid-116

  committed
  af6944f

  Copy full SHA for af6944f

• Fix linting issues

  kid-116

  committed
  0b509f3

  Copy full SHA for 0b509f3

Commits on Mar 28, 2023

• Fix

  kid-116

  committed
  bfc32c2

  Copy full SHA for bfc32c2

• Add pylint workflow

  kid-116

  committed
  9112ae4

  Copy full SHA for 9112ae4

• Fix workflow

  kid-116

  committed
  a3d3464

  Copy full SHA for a3d3464

• Fix

  kid-116

  committed
  ddf8a87

  Copy full SHA for ddf8a87

• Update README.md

  kid-116

  authored
  db8fcbf

  Copy full SHA for db8fcbf

• Add export to build

  kid-116

  committed
  2514d84

  Copy full SHA for 2514d84

• Merge branch 'main' of https://github.com/Rohan-Kamat/Andromeda

  kid-116

  committed
  2b9396e

  Copy full SHA for 2b9396e

Commits on Mar 30, 2023

• Fix mobile responsiveness

  kid-116

  committed
  b6e5fe1

  Copy full SHA for b6e5fe1

• Fix lint

  kid-116

  committed
  4b0c958

  Copy full SHA for 4b0c958

• Fix

  kid-116

  committed
  15ef99f

  Copy full SHA for 15ef99f