WEBSITE SCRAPING IN C# : WEB-CRAWLING CONCEPTS
What are we building? A C# console application that: Starts at https://www.faithinfotechacademy.com/ Finds internal links. Visits all allowed pages. Excludes: Testimonials Gallery Placements Blog Contacts Extracts page text. Performs basic sanitization. Splits text into individual lines/items. Removes duplicates. Saves the result as JSON. Later, the same list can be posted to the ASP.NET Core API. So the architecture becomes: Faith Infotech Website │ ▼ C# Web Scraper │ │ HttpClient │ HtmlAgility │ Pack └─────┬─────┘ │ ▼ Extract HTML │ ▼ Remove unwanted HTML elements │ ▼ Sanitize text │ ▼ Split into lines │ ▼ Remove duplicates │ ▼ List<ScrapedData> │ ├──────────────► JSON │ └──────────────► AS...