Fine tuning on self collected web data felt straightforward until it wasn’t. Here’s what actually caused problems:

Boilerplate contamination was the silent killer. Navigation menus, cookie banners, and footer text were leaking into training examples. The model started generating weirdly structured outputs that looked like half-rendered HTML. Solved it with aggressive DOM-targeting rather than raw text extraction and switching to a rotating residential proxy layer (per-request IP rotation means target sites serve you the same rendered DOM a real browser session would get, rather than a bot-filtered skeleton) made a noticeable difference in how clean the raw HTML was before I even touched the parser.

Deduplication order mattered more than I expected. Running exact dedup before fuzzy dedup produced meaningfully different training sets than the reverse. The model’s tendency to regurgitate phrasing shifted noticeably depending on the order.

The thing I still haven’t fully solved: near-duplicate detection across different languages. MinHash works fine monolingually, but multilingual dedup keeps surfacing edge cases I don’t love.

Curious whether others have hit the same issues, especially around multilingual corpora. Also genuinely wondering if anyone has a principled approach to deciding when scraped data quality is “good enough” to stop cleaning and just train.