In this liveProject, you’ll explore and assess essential methods for unstructured text search in order to identify which is the best for building a search engine. You’ll preprocess the data for this task using the spaCy library, and then experiment with implementing both a TF-IDF search and an inverted index search to find relevant information.
This liveProject is for intermediate Python programmers familiar with the basics of manipulations with strings, lists and dictionaries. To begin this liveProject, you will need to be familiar with:
- Intermediate Python
- Basic understanding of conda environments
- Basic scikit-learn
- Basic NumPy
- Reading data from and writing to JSON files
- Manipulations with tuples, lists and dictionaries using loops and comprehensions
- Natural language processing tokenization, lemmatization, and cleaning of text data
- Basic NumPy array operations
you will learn
In this liveProject you will learn to implement the simple-but-effective term frequency - inverse document frequency (TF-IDF) search method. This method will encompass calculating the frequency of certain words in documents.
- Use Python’s built in JSON library to store multi-level text data
- Create, update and transform lists and dictionaries with text data
- Apply Python’s spaCy library to perform essential natural language processing steps
- Compute TF-IDF tables and apply term frequency search to them
- Calculate cosine similarity with scikit-learn
- Build an inverted index, an essential element of a search engine
- You choose the schedule and decide how much time to invest as you build your project.
- Project roadmap
- Each project is divided into several achievable steps.
- Get Help
- While within the liveProject platform, get help from other participants.
- Compare with others
- For each step, compare your deliverable to the solutions by the author and other participants.