What are we building?

A C# console application that:

Starts at https://www.faithinfotechacademy.com/
Finds internal links.
Visits all allowed pages.
Excludes:
- Testimonials
- Gallery
- Placements
- Blog
- Contacts
Extracts page text.
Performs basic sanitization.
Splits text into individual lines/items.
Removes duplicates.
Saves the result as JSON.
Later, the same list can be posted to the ASP.NET Core API.

So the architecture becomes:


Faith Infotech Website
          │
          ▼
     C# Web Scraper
          │
   
    │ HttpClient
    │ HtmlAgility
    │    Pack    
    └─────┬─────┘
          │
          ▼
    Extract HTML
          │
          ▼
    Remove unwanted
    HTML elements
          │
          ▼
    Sanitize text
          │
          ▼
    Split into lines
          │
          ▼
    Remove duplicates
          │
          ▼
   List<ScrapedData>
          │
          ├──────────────► JSON
          │
          └──────────────► ASP.NET Core API

1. Create the C# Scraper Project

In Visual Studio 2022:

Create a new project → Console App

Name:


FaithInfoTech.WebScraper

Target:


.NET 8 / .NET 9 / .NET 10

depending on your installed SDK.

2. Install HtmlAgilityPack

Open:

Tools → NuGet Package Manager → Package Manager Console

Run:


Install-Package HtmlAgilityPack

Or:


dotnet add package HtmlAgilityPack

3. Project Structure

Use:


FaithInfoTech.WebScraper
│
├── Models
│   └── ScrapedData.cs
│
├── Services
│   └── WebsiteScraper.cs
│
├── Helpers
│   └── TextCleaner.cs
│
├── Output
│   └── scraped-data.json
│
└── Program.cs

4. Model

Create:

Models/ScrapedData.cs


namespace FaithInfoTech.WebScraper.Models;

public class ScrapedData
{
    // Unique ID for this scraped item
    public int Id { get; set; }

    // Actual text extracted from the website
    public string Data { get; set; } = string.Empty;

    // Page from which the text was obtained
    public string Source { get; set; } = string.Empty;
}

Why keep `Source`?

Suppose the chatbot retrieves:


PROPEL FSD PRO

We should know where it came from:


https://www.faithinfotechacademy.com/freshers

Later, the RAG chatbot can display:


Source: Faith Infotech Academy - Freshers

5. Text Cleaner

Create:

Helpers/TextCleaner.cs


namespace FaithInfoTech.WebScraper.Helpers;

public static class TextCleaner
{
    public static string Clean(string text)
    {
        if (string.IsNullOrWhiteSpace(text))
        {
            return string.Empty;
        }

        // Replace newline characters
        // with a normal space.
        text = text
            .Replace("\r", " ")
            .Replace("\n", " ")
            .Replace("\t", " ");

        // Remove multiple spaces.
        // Example:
        //
        // "Faith     Infotech"
        //
        // becomes:
        //
        // "Faith Infotech"
        text = string.Join(
            " ",
            text.Split(
                Array.Empty<char>(),
                StringSplitOptions.RemoveEmptyEntries));

        return text.Trim();
    }
}

6. Main Scraper Service

Create:

Services/WebsiteScraper.cs


using System.Net;
using HtmlAgilityPack;
using FaithInfoTech.WebScraper.Helpers;
using FaithInfoTech.WebScraper.Models;

namespace FaithInfoTech.WebScraper.Services;

public class WebsiteScraper
{
    private readonly HttpClient _httpClient;

    // Website from which we want to scrape.
    private const string BaseUrl =
        "https://www.faithinfotechacademy.com/";

    // ------------------------------------------------
    // Pages that should NOT be scraped.
    // ------------------------------------------------

    private readonly string[] _excludedWords =
    {
        "testimonial",
        "testimonials",
        "gallery",
        "placement",
        "placements",
        "blog",
        "contact",
        "contacts"
    };


    // ------------------------------------------------
    // Keep track of pages already visited.
    //
    // Why?
    //
    // Websites contain many links pointing to the
    // same page.
    //
    // Without this collection we could scrape the
    // same page repeatedly.
    // ------------------------------------------------

    private readonly HashSet<string>
        _visitedUrls =
            new(StringComparer.OrdinalIgnoreCase);


    // ------------------------------------------------
    // Store all scraped records.
    // ------------------------------------------------

    private readonly List<ScrapedData>
        _scrapedData = new();


    public WebsiteScraper(HttpClient httpClient)
    {
        _httpClient = httpClient;

        // Identify our application to the website.
        _httpClient.DefaultRequestHeaders.UserAgent.ParseAdd(
            "FaithInfoTechRAGBot/1.0");
    }


    // =================================================
    // START SCRAPING
    // =================================================

    public async Task<List<ScrapedData>>
        StartAsync()
    {
        Console.WriteLine(
            "======================================");

        Console.WriteLine(
            " Faith Infotech Website Scraper");

        Console.WriteLine(
            "======================================");

        Console.WriteLine();

        await CrawlPageAsync(BaseUrl);

        Console.WriteLine();

        Console.WriteLine(
            $"Total unique records: {_scrapedData.Count}");

        return _scrapedData;
    }


    // =================================================
    // CRAWL ONE PAGE
    // =================================================

    private async Task CrawlPageAsync(
        string url)
    {
        // ---------------------------------------------
        // Normalize URL
        // ---------------------------------------------

        url = NormalizeUrl(url);


        // ---------------------------------------------
        // Don't visit the same URL twice.
        // ---------------------------------------------

        if (!_visitedUrls.Add(url))
        {
            return;
        }


        // ---------------------------------------------
        // Ignore unwanted pages.
        // ---------------------------------------------

        if (IsExcludedPage(url))
        {
            Console.WriteLine(
                $"SKIPPED: {url}");

            return;
        }


        // ---------------------------------------------
        // Only crawl our own website.
        // ---------------------------------------------

        if (!IsInternalUrl(url))
        {
            return;
        }


        Console.WriteLine(
            $"SCRAPING: {url}");


        try
        {
            // -----------------------------------------
            // Download HTML
            // -----------------------------------------

            string html =
                await _httpClient.GetStringAsync(url);


            // -----------------------------------------
            // Parse HTML
            // -----------------------------------------

            var document =
                new HtmlDocument();

            document.LoadHtml(html);


            // -----------------------------------------
            // Remove unwanted HTML elements.
            // -----------------------------------------

            RemoveUnwantedNodes(document);


            // -----------------------------------------
            // Extract text.
            // -----------------------------------------

            ExtractText(
                document,
                url);


            // -----------------------------------------
            // Find links.
            // -----------------------------------------

            var links =
                ExtractLinks(
                    document,
                    url);


            // -----------------------------------------
            // Crawl each internal link.
            // -----------------------------------------

            foreach (var link in links)
            {
                await CrawlPageAsync(link);
            }
        }
        catch (HttpRequestException ex)
        {
            Console.WriteLine(
                $"HTTP error: {ex.Message}");
        }
        catch (Exception ex)
        {
            Console.WriteLine(
                $"Error processing {url}: {ex.Message}");
        }
    }


    // =================================================
    // EXTRACT TEXT
    // =================================================

    private void ExtractText(
        HtmlDocument document,
        string sourceUrl)
    {
        // Get the BODY element.
        var body =
            document.DocumentNode.SelectSingleNode(
                "//body");

        if (body == null)
        {
            return;
        }


        // ---------------------------------------------
        // Extract text nodes.
        // ---------------------------------------------

        var textNodes =
            body.SelectNodes(
                ".//text()[normalize-space()]");


        if (textNodes == null)
        {
            return;
        }


        foreach (var node in textNodes)
        {
            string text =
                WebUtility.HtmlDecode(
                    node.InnerText);


            // Basic sanitization.
            text =
                TextCleaner.Clean(text);


            // Ignore empty text.
            if (string.IsNullOrWhiteSpace(text))
            {
                continue;
            }


            // Ignore very small fragments.
            if (text.Length < 2)
            {
                continue;
            }


            // Avoid duplicate text.
            bool alreadyExists =
                _scrapedData.Any(x =>
                    x.Data.Equals(
                        text,
                        StringComparison
                            .OrdinalIgnoreCase));


            if (alreadyExists)
            {
                continue;
            }


            _scrapedData.Add(
                new ScrapedData
                {
                    Id = _scrapedData.Count + 1,

                    Data = text,

                    Source = sourceUrl
                });
        }
    }


    // =================================================
    // REMOVE UNWANTED HTML
    // =================================================

    private static void RemoveUnwantedNodes(
        HtmlDocument document)
    {
        // These elements normally don't contain
        // useful knowledge for our chatbot.

        string[] selectors =
        {
            "//script",
            "//style",
            "//noscript",
            "//svg"
        };


        foreach (string selector in selectors)
        {
            var nodes =
                document.DocumentNode
                    .SelectNodes(selector);


            if (nodes == null)
            {
                continue;
            }


            foreach (var node in nodes)
            {
                node.Remove();
            }
        }
    }


    // =================================================
    // EXTRACT LINKS
    // =================================================

    private List<string> ExtractLinks(
        HtmlDocument document,
        string currentUrl)
    {
        var result =
            new List<string>();


        var linkNodes =
            document.DocumentNode
                .SelectNodes("//a[@href]");


        if (linkNodes == null)
        {
            return result;
        }


        foreach (var linkNode in linkNodes)
        {
            string href =
                linkNode
                    .GetAttributeValue(
                        "href",
                        string.Empty);


            if (string.IsNullOrWhiteSpace(href))
            {
                continue;
            }


            // -----------------------------------------
            // Ignore JavaScript links.
            // -----------------------------------------

            if (href.StartsWith(
                    "javascript:",
                    StringComparison.OrdinalIgnoreCase))
            {
                continue;
            }


            // -----------------------------------------
            // Ignore mail links.
            // -----------------------------------------

            if (href.StartsWith(
                    "mailto:",
                    StringComparison.OrdinalIgnoreCase))
            {
                continue;
            }


            // -----------------------------------------
            // Convert relative URL to absolute URL.
            // -----------------------------------------

            if (!Uri.TryCreate(
                    new Uri(currentUrl),
                    href,
                    out Uri? absoluteUri))
            {
                continue;
            }


            string absoluteUrl =
                absoluteUri.ToString();


            // -----------------------------------------
            // Only internal pages.
            // -----------------------------------------

            if (!IsInternalUrl(absoluteUrl))
            {
                continue;
            }


            // -----------------------------------------
            // Don't crawl excluded pages.
            // -----------------------------------------

            if (IsExcludedPage(
                    absoluteUrl))
            {
                continue;
            }


            // -----------------------------------------
            // Don't crawl PDF files in Module 1.
            // -----------------------------------------

            if (absoluteUrl.EndsWith(
                    ".pdf",
                    StringComparison.OrdinalIgnoreCase))
            {
                continue;
            }


            result.Add(
                NormalizeUrl(
                    absoluteUrl));
        }


        return result.Distinct(
            StringComparer.OrdinalIgnoreCase)
            .ToList();
    }


    // =================================================
    // CHECK INTERNAL URL
    // =================================================

    private static bool IsInternalUrl(
        string url)
    {
        if (!Uri.TryCreate(
                url,
                UriKind.Absolute,
                out Uri? uri))
        {
            return false;
        }


        return
            uri.Host.Equals(
                "faithinfotechacademy.com",
                StringComparison.OrdinalIgnoreCase)

            ||

            uri.Host.Equals(
                "www.faithinfotechacademy.com",
                StringComparison.OrdinalIgnoreCase);
    }


    // =================================================
    // CHECK EXCLUDED PAGE
    // =================================================

    private bool IsExcludedPage(
        string url)
    {
        string lowerUrl =
            url.ToLowerInvariant();


        return _excludedWords.Any(
            word => lowerUrl.Contains(word));
    }


    // =================================================
    // NORMALIZE URL
    // =================================================

    private static string NormalizeUrl(
        string url)
    {
        // Remove fragment.
        //
        // Example:
        //
        // /about#team
        //
        // becomes:
        //
        // /about

        int hashIndex =
            url.IndexOf('#');

        if (hashIndex >= 0)
        {
            url =
                url[..hashIndex];
        }


        return url.TrimEnd('/');
    }
}

7. Why This Code Is Similar to Scrapy

The concepts are almost identical.

Scrapy	C# Implementation
`start_urls`	`BaseUrl`
`allowed_domains`	`IsInternalUrl()`
Spider	`WebsiteScraper`
`parse()`	`CrawlPageAsync()`
`response`	HTML returned by `HttpClient`
CSS/XPath selectors	HtmlAgilityPack XPath
Item	`ScrapedData`
Pipeline	Later API call
Duplicate filtering	`HashSet`
Crawling links	`ExtractLinks()`

learn the web-crawling concepts without introducing Python.

8. Program.cs

Now create:

Program.cs


using System.Text.Json;
using FaithInfoTech.WebScraper.Services;


// =====================================================
// Create HttpClient
// =====================================================

using var httpClient =
    new HttpClient();


// =====================================================
// Create scraper
// =====================================================

var scraper =
    new WebsiteScraper(
        httpClient);


// =====================================================
// Start scraping
// =====================================================

var data =
    await scraper.StartAsync();


// =====================================================
// Display result
// =====================================================

Console.WriteLine();

Console.WriteLine(
    "======================================");

Console.WriteLine(
    " Scraping Completed");

Console.WriteLine(
    "======================================");

Console.WriteLine();


// =====================================================
// Display first 20 records
// =====================================================

foreach (var item in
         data.Take(20))
{
    Console.WriteLine(
        $"{item.Id}. {item.Data}");

    Console.WriteLine(
        $"   Source: {item.Source}");

    Console.WriteLine();
}


// =====================================================
// Save result as JSON
// =====================================================

var jsonOptions =
    new JsonSerializerOptions
    {
        WriteIndented = true
    };


string json =
    JsonSerializer.Serialize(
        data,
        jsonOptions);


// =====================================================
// Create Output folder
// =====================================================

string outputFolder =
    Path.Combine(
        AppContext.BaseDirectory,
        "Output");


Directory.CreateDirectory(
    outputFolder);


// =====================================================
// Save file
// =====================================================

string outputFile =
    Path.Combine(
        outputFolder,
        "scraped-data.json");


await File.WriteAllTextAsync(
    outputFile,
    json);


Console.WriteLine(
    $"Data saved to:");

Console.WriteLine(
    outputFile);

9. Expected Output

When you run:


dotnet run

you should see something like:


======================================
 Faith Infotech Website Scraper
======================================

SCRAPING: https://www.faithinfotechacademy.com

SCRAPING: https://www.faithinfotechacademy.com/about

SCRAPING: https://www.faithinfotechacademy.com/freshers

SCRAPING: https://www.faithinfotechacademy.com/internpro

SKIPPED: https://www.faithinfotechacademy.com/testimonials

SKIPPED: https://www.faithinfotechacademy.com/gallery

SKIPPED: https://www.faithinfotechacademy.com/placements

SKIPPED: https://www.faithinfotechacademy.com/blog

SKIPPED: https://www.faithinfotechacademy.com/contact

======================================
 Scraping Completed
======================================

1. Faith Infotech Academy
   Source: https://www.faithinfotechacademy.com

2. About Us
   Source: https://www.faithinfotechacademy.com/about

3. Full Stack Development
   Source: https://www.faithinfotechacademy.com/freshers

...

10. Generated JSON

The application creates:


Output/scraped-data.json

Example:


[
  {
    "Id": 1,
    "Data": "Faith Infotech Academy",
    "Source": "https://www.faithinfotechacademy.com"
  },
  {
    "Id": 2,
    "Data": "About Us",
    "Source": "https://www.faithinfotechacademy.com/about"
  },
  {
    "Id": 3,
    "Data": "Full Stack Development",
    "Source": "https://www.faithinfotechacademy.com/freshers"
  }
]

11. Requirement : "Split Each Line"

The current code extracts text nodes, which is slightly different from explicitly splitting the page text by line.

If you want to strictly implement:

Split the text into each line and save it as a list

we can modify ExtractText().

For example:


private void ExtractText(
    HtmlDocument document,
    string sourceUrl)
{
    var body =
        document.DocumentNode
            .SelectSingleNode("//body");

    if (body == null)
        return;


    string pageText =
        body.InnerText;


    // HTML decode
    pageText =
        WebUtility.HtmlDecode(pageText);


    // Split into lines
    var lines =
        pageText.Split(
            new[]
            {
                '\r',
                '\n'
            },
            StringSplitOptions.RemoveEmptyEntries);


    foreach (string line in lines)
    {
        string text =
            TextCleaner.Clean(line);


        if (string.IsNullOrWhiteSpace(text))
            continue;


        AddUniqueRecord(
            text,
            sourceUrl);
    }
}

Then create:


private void AddUniqueRecord(
    string text,
    string sourceUrl)
{
    bool exists =
        _scrapedData.Any(x =>
            x.Data.Equals(
                text,
                StringComparison.OrdinalIgnoreCase));


    if (exists)
        return;


    _scrapedData.Add(
        new ScrapedData
        {
            Id = _scrapedData.Count + 1,

            Data = text,

            Source = sourceUrl
        });
}

For RAG, however, Recommends retaining page structure rather than blindly splitting every HTML text node.

12. Better Approach for RAG

For the final RAG application, I recommend this pipeline:


HTML Page
   │
   ▼
Remove script/style/menu/footer
   │
   ▼
Extract headings + paragraphs
   │
   ▼
Clean text
   │
   ▼
Group related text
   │
   ▼
Create chunks
   │
   ▼
Generate embeddings
   │
   ▼
Vector Store

Instead of:


HTML
 ↓
Every individual word/line
 ↓
Embedding

For example:

Bad RAG document


.NET

Bad RAG document


Full Stack

Bad RAG document


Developer

Good RAG document


PROPEL FSD PRO - Full Stack Developer:
.NET with Angular.

The second approach provides much better semantic context.

13. Connect to ASP.NET Core API

Once the scraper works, we don't need:


C# Scraper
   ↓
JSON

as the final architecture.

Instead:


C# Scraper
     │
     ▼
List<ScrapedData>
     │
     ▼
HttpClient
     │
     ▼
POST /api/scrappeddata/bulk
     │
     ▼
ASP.NET Core
     │
     ▼
SQL Server

Add this method to WebsiteScraper or create a separate API client.


public async Task SendToApiAsync(
    List<ScrapedData> data)
{
    var response =
        await _httpClient.PostAsJsonAsync(
            "https://localhost:7001/api/scrappeddata/bulk",
            data);

    response.EnsureSuccessStatusCode();

    Console.WriteLine(
        "Scraped data successfully sent to API.");
}

Then:


var data =
    await scraper.StartAsync();

await scraper.SendToApiAsync(data);

14. Final Architecture

Your all-C# implementation is therefore:


┌─────────────────────────────────────────────┐
│       FAITH INFOTECH WEBSITE                │
└──────────────────────┬──────────────────────┘
                       │
                       ▼
             ┌───────────────────┐
             │ C# Web Scraper    
             │                   
             │ HttpClient        
             │ HtmlAgilityPack   
             └─────────┬─────────┘
                       │
                       ▼
                Download HTML
                       │
                       ▼
                 Parse HTML
                       │
                       ▼
             Remove script/style
                       │
                       ▼
                Extract text
                       │
                       ▼
                 Sanitize
                       │
                       ▼
              Remove duplicates
                       │
                       ▼
              List<ScrapedData>
                       │
                 ┌─────┴─────┐
                 │           │
                 ▼           ▼
              JSON       REST API
                             │
                             ▼
                         SQL Server

Pages excluded


Testimonials    ❌
Gallery           ❌
Placements     ❌
Blog               ❌
Contacts         ❌

Pages included


Home           ✅
About          ✅
Freshers       ✅
InternPro      ✅
Corporate      ✅
Academic       ✅
AdvantagePro   ✅
Team           ✅
Other allowed
internal pages ✅

Search This Blog

IT Skill Training & Jobs @ Faith Infotech Academy, Technopark, Thiruvananthapuram

WEBSITE SCRAPING IN C# : WEB-CRAWLING CONCEPTS

What are we building?

1. Create the C# Scraper Project

2. Install HtmlAgilityPack

3. Project Structure

4. Model

Why keep `Source`?

5. Text Cleaner

6. Main Scraper Service

7. Why This Code Is Similar to Scrapy

8. Program.cs

9. Expected Output

10. Generated JSON

11. Requirement : "Split Each Line"

12. Better Approach for RAG

Bad RAG document

Bad RAG document

Bad RAG document

Good RAG document

13. Connect to ASP.NET Core API

14. Final Architecture

Pages excluded

Pages included

Comments

Post a Comment

Popular posts from this blog

Interview Tips: Dot NET Framework vs Net CORE

OOP Concept in real-time scenario: Mobile Device Management Software

PRACTICE ASSIGNMENTS

WEBSITE SCRAPING IN C# : WEB-CRAWLING CONCEPTS

What are we building?

1. Create the C# Scraper Project

2. Install HtmlAgilityPack

3. Project Structure

4. Model

Why keep Source?

5. Text Cleaner

6. Main Scraper Service

7. Why This Code Is Similar to Scrapy

8. Program.cs

9. Expected Output

10. Generated JSON

11. Requirement : "Split Each Line"

12. Better Approach for RAG

Bad RAG document

Bad RAG document

Bad RAG document

Good RAG document

13. Connect to ASP.NET Core API

14. Final Architecture

Pages excluded

Pages included

Comments

Post a Comment

Popular posts from this blog

Interview Tips: Dot NET Framework vs Net CORE

OOP Concept in real-time scenario: Mobile Device Management Software

PRACTICE ASSIGNMENTS

Why keep `Source`?