<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Arman Hosen's Blog]]></title><description><![CDATA[Arman Hosen's Blog]]></description><link>https://armanhosen.hashnode.dev</link><image><url>https://cdn.hashnode.com/res/hashnode/image/upload/v1593680282896/kNC7E8IR4.png</url><title>Arman Hosen&apos;s Blog</title><link>https://armanhosen.hashnode.dev</link></image><generator>RSS for Node</generator><lastBuildDate>Tue, 08 Sep 2026 22:53:51 GMT</lastBuildDate><atom:link href="https://armanhosen.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Extracting PriceCharting Market Data: Historical Sales, Graded Comps (PSA/BGS/CGC), and Game/Card Valuations into JSON]]></title><description><![CDATA[1. The Secondary Market Data Bottleneck
Whether you are building an automated repricing tool for a retro game shop, managing a personal collection of rare Nintendo titles, or identifying grading arbit]]></description><link>https://armanhosen.hashnode.dev/extracting-pricecharting-market-data-historical-sales-graded-comps-psa-bgs-cgc-and-game-card-valuations-into-json</link><guid isPermaLink="true">https://armanhosen.hashnode.dev/extracting-pricecharting-market-data-historical-sales-graded-comps-psa-bgs-cgc-and-game-card-valuations-into-json</guid><dc:creator><![CDATA[Arman Hosen]]></dc:creator><pubDate>Mon, 07 Sep 2026 19:19:13 GMT</pubDate><content:encoded><![CDATA[<h2>1. The Secondary Market Data Bottleneck</h2>
<p>Whether you are building an automated repricing tool for a retro game shop, managing a personal collection of rare Nintendo titles, or identifying grading arbitrage in Pokémon TCG cards, <strong>PriceCharting</strong> is the standard benchmark for historical market values.</p>
<p>PriceCharting aggregates historical sales across eBay, Heritage Auctions, Goldin, PWCC, and TCGPlayer, categorizing items by condition and grading company.</p>
<p>However, extracting this data at scale presents significant friction:</p>
<ul>
<li><strong>Manual Lookup is Unsustainable</strong>: Checking current values for 500+ game cartridges or graded cards takes dozens of hours and goes stale within days.</li>
<li><strong>The Official API has Steep Limits</strong>: PriceCharting's official API requires a paid plan ($49/month) and <strong>omits both price history and product images</strong>. It returns only static, current snapshot estimates.</li>
<li><strong>Scraping is Fragile</strong>: PriceCharting blocks standard HTTP requests with 403 Forbidden errors if real browser headers are missing, and pins currencies dynamically based on location unless specific cookies (<code>currency=usd</code>) are supplied.</li>
</ul>
<p>To solve this, we can use the cloud-hosted <a href="https://apify.com/incognito_mode/pricecharting-product-scraper">PriceCharting Product Scraper</a> on Apify. It extracts current prices across every condition, the full historical time series, complete grading ladders (TAG, ACE, SGC, CGC, PSA, BGS), recent sold comps, and 1600px high-resolution images into clean, structured JSON.</p>
<hr />
<h2>2. The Data Schema: Video Games vs. Trading Cards</h2>
<p>PriceCharting structures price points differently depending on whether an item is a video game or a collectible card. The scraper normalizes these points into a consistent, predictable schema.</p>
<h3>Standard Price Slots (<code>price</code>)</h3>
<table>
<thead>
<tr>
<th>Slot Key</th>
<th>Video Games Meaning</th>
<th>Trading Cards (TCG) Meaning</th>
</tr>
</thead>
<tbody><tr>
<td><code>loose</code></td>
<td>Cartridge / Disc only</td>
<td>Ungraded / Raw card</td>
</tr>
<tr>
<td><code>cib</code></td>
<td>Complete in Box (Game + Box + Manual)</td>
<td>Grade 7</td>
</tr>
<tr>
<td><code>new</code></td>
<td>Factory Sealed</td>
<td>Grade 8</td>
</tr>
<tr>
<td><code>graded</code></td>
<td>Professionally Graded Box</td>
<td>Grade 9</td>
</tr>
<tr>
<td><code>boxOnly</code></td>
<td>Original Box without Game</td>
<td>Grade 9.5</td>
</tr>
<tr>
<td><code>manualOnly</code></td>
<td>Original Manual only</td>
<td>PSA 10 Gem Mint</td>
</tr>
</tbody></table>
<h3>Full Grading Ladder (<code>fullPrices</code>)</h3>
<p>For trading cards, the <code>fullPrices</code> object captures the entire market spectrum:</p>
<ul>
<li><strong>Numerical Grades</strong>: Grade 1 through 9.5.</li>
<li><strong>Top Tier Grades</strong>: TAG 10, ACE 10, SGC 10, CGC 10, CGC 10 Pristine, PSA 10, BGS 10, and BGS 10 Black Label.</li>
</ul>
<h3>Time Series (<code>priceHistory</code>)</h3>
<p>Every headline condition includes its full historical price array (<code>[ [timestamp_ms, price_usd], ... ]</code>), enabling trend analysis and rolling averages without querying secondary databases.</p>
<hr />
<h2>3. Extracting PriceCharting Records with Python</h2>
<p>Using the official <code>apify-client</code> Python package, you can query one or hundreds of PriceCharting items in parallel.</p>
<h3>Prerequisites</h3>
<pre><code class="language-bash">pip install apify-client pandas matplotlib seaborn
</code></pre>
<h3>Python Extraction Script</h3>
<pre><code class="language-python">import os
import pandas as pd
from apify_client import ApifyClient

# Initialize Apify client with your API token
client = ApifyClient(os.environ.get("APIFY_API_TOKEN") or "YOUR_APIFY_TOKEN")

# Target PriceCharting product URLs or numeric IDs
run_input = {
    "productUrls": [
        "https://www.pricecharting.com/game/gameboy-advance/pokemon-emerald",
        "https://www.pricecharting.com/game/pokemon-base-set/charizard-4"
    ],
    "scrapeProductDetails": True,
    "includeRecentSales": True
}

print("[*] Launching PriceCharting Product Scraper...")
run = client.actor("incognito_mode/pricecharting-product-scraper").call(run_input=run_input)

# Retrieve dataset items
dataset = client.dataset(run["defaultDatasetId"])
products = list(dataset.iterate_items())

for item in products:
    name = item.get("productName")
    console = item.get("consoleName")
    prices = item.get("price", {})
    print(f"
Product: {name} ({console})")
    print(f" - Loose / Raw: ${prices.get('loose')}")
    print(f" - CIB / Grade 7: ${prices.get('cib')}")
    print(f" - New / Grade 8: ${prices.get('new')}")
    print(f" - Graded / Grade 9: ${prices.get('graded')}")
    print(f" - PSA 10 / Manual: ${prices.get('manualOnly')}")
</code></pre>
<hr />
<h2>4. Practical Data Science: Grading Arbitrage &amp; Price Divergence</h2>
<p>One of the most valuable applications of this structured data is identifying <strong>grading arbitrage</strong>: determining when the premium of a PSA 10 or BGS 9.5 justifies the grading fee and risk of submitting a raw card.</p>
<p>Here is how you can process the extracted JSON in Pandas to calculate the <strong>Grading Multiplier</strong>:</p>
<pre><code class="language-python"># Load extracted items into a DataFrame
df = pd.DataFrame(products)

# Extract price columns
df["raw_price"] = df["price"].apply(lambda p: p.get("loose") if isinstance(p, dict) else None)
df["psa10_price"] = df["price"].apply(lambda p: p.get("manualOnly") if isinstance(p, dict) else None)

# Calculate multiplier: PSA 10 Value / Raw Card Value
df["psa10_multiplier"] = df["psa10_price"] / df["raw_price"]

print(df[["productName", "raw_price", "psa10_price", "psa10_multiplier"]])
</code></pre>
<p>When a card exhibits a 15x or 20x multiplier between raw condition and PSA 10, mint-condition raw copies offer significant positive expected value (EV) for professional grading.</p>
<hr />
<h2>5. Google Sheets Integration (No-Code Workflow)</h2>
<p>If you manage inventory or track your collection in Google Sheets, you can import live PriceCharting data without writing a single line of backend code:</p>
<ol>
<li>In the Apify Console, navigate to your finished scraper run dataset.</li>
<li>Copy the permanent CSV API URL:</li>
</ol>
<pre><code class="language-text">https://api.apify.com/v2/datasets/&lt;DATASET_ID&gt;/items?format=csv
</code></pre>
<ol>
<li>In any Google Sheet cell, enter:</li>
</ol>
<pre><code class="language-excel">=IMPORTDATA("https://api.apify.com/v2/datasets/&lt;DATASET_ID&gt;/items?format=csv")
</code></pre>
<p>Google Sheets will automatically populate rows with product names, loose prices, CIB prices, and recent sales volumes.</p>
<hr />
<h2>6. Key Takeaways</h2>
<ul>
<li><strong>Bypass Official API Gaps</strong>: Get full historical time series, high-resolution 1600px box/card photos, and actual sold comps that the official $49/mo API lacks.</li>
<li><strong>Pay Only for Successful Results</strong>: Failed lookups or invalid URLs are excluded from the dataset and never billed.</li>
<li><strong>Analysis-Ready Output</strong>: Clean USD numeric values ready for Pandas, SQL databases, and spreadsheet modeling.</li>
</ul>
<p>To run this scraper yourself, check out the <a href="https://apify.com/incognito_mode/pricecharting-product-scraper">PriceCharting Product Scraper on Apify</a>.</p>
]]></content:encoded></item><item><title><![CDATA[Extracting FotMob Match Data: Lineups, xG, Player Ratings, and Match Events into JSON]]></title><description><![CDATA[FotMob is easily one of the best sources for detailed football stats. On any given matchday, each fixture page packs a surprising depth of data: confirmed tactical lineups, live player ratings, expect]]></description><link>https://armanhosen.hashnode.dev/extracting-fotmob-match-data-lineups-xg-player-ratings-and-match-events-into-json</link><guid isPermaLink="true">https://armanhosen.hashnode.dev/extracting-fotmob-match-data-lineups-xg-player-ratings-and-match-events-into-json</guid><category><![CDATA[web scraping]]></category><category><![CDATA[Python]]></category><dc:creator><![CDATA[Arman Hosen]]></dc:creator><pubDate>Sat, 05 Sep 2026 16:36:24 GMT</pubDate><content:encoded><![CDATA[<p>FotMob is easily one of the best sources for detailed football stats. On any given matchday, each fixture page packs a surprising depth of data: confirmed tactical lineups, live player ratings, expected goals (xG) shot breakdowns, player of the match selections, and chronological event timelines (goals, assists, cards, subs).</p>
<p>The problem is getting that data out in bulk. FotMob doesn't have an open public API. If you're building a predictive model, a fantasy sports tool, a scouting dashboard, or just running weekly analytics, clicking through hundreds of matches by hand is out of the question. At 2 minutes per match, pulling a 500-game dataset takes over 16 hours of manual copying.</p>
<p>Scraping it directly isn't straightforward either. FotMob's web frontend relies on internal API endpoints protected by dynamic request signatures (<code>x-mas</code> headers) and changing payload structures. Scrapers built from scratch usually break after a few weeks. Meanwhile, commercial sports data feeds like Opta, StatsBomb, or Sportradar cost hundreds or thousands of dollars a month and gatekeep advanced stats like xG behind enterprise plans.</p>
<p>To solve this, I built a reliable scraper on Apify: the <strong><a href="https://apify.com/incognito_mode/fotmob-match-details-scraper">FotMob Match Details Scraper</a></strong>. It takes a list of FotMob match IDs or URLs and outputs one normalized JSON or CSV record per match in seconds.</p>
<p>Here is a quick walkthrough of how it works, what the data structure looks like, and how to integrate it into Python or Google Sheets.</p>
<hr />
<h2>What the Extracted Dataset Contains</h2>
<p>Each match produces a structured object with the following fields:</p>
<ul>
<li><strong>Match Header &amp; Meta</strong>: Home and away clubs, final score, kickoff timestamp in UTC (ISO 8601), competition, matchday/round, country, match status (<code>started</code>, <code>finished</code>), venue, referee, attendance, and official Player of the Match.</li>
<li><strong><code>lineups</code></strong>: Tactical formations for both clubs (e.g. <code>4-3-3</code>, <code>4-2-3-1</code>), manager/coach names, starting XI with shirt numbers and positions, and full bench lists.</li>
<li><strong><code>playerStats</code></strong>: Flattened, analysis-ready stats for every player who took the pitch: FotMob rating, minutes played, goals, assists, expected goals (<code>expected_goals</code>), expected assists (<code>expected_assists</code>), total shots, passing accuracy, chances created, touches, duels won, clearances, and goalkeeper metrics (saves, goals conceded).</li>
<li><strong><code>events</code></strong>: A unified, chronological log of all primary match events: goals (including goal type and assist), yellow/red cards, and substitutions with incoming and outgoing player names and exact minute marks.</li>
<li><strong><code>teamStats</code></strong>: Full head-to-head comparison categories: possession, xG, total shots, shots on target, accurate passes, duels, tackles, fouls, corners, and offsides.</li>
<li><strong><code>h2h</code> (optional)</strong>: Previous head-to-head records between the two clubs.</li>
<li><strong><code>rawData</code> (optional)</strong>: The raw, unparsed FotMob JSON payload for cases where you need internal IDs or debugging.</li>
</ul>
<hr />
<h2>Step 1: Set Up the Scraper</h2>
<ol>
<li>Go to the <a href="https://apify.com/incognito_mode/fotmob-match-details-scraper">FotMob Match Details Scraper</a> on Apify.</li>
<li>If you don't have an Apify account, you can sign up for free (the free tier includes $5/month in credit, which covers roughly 500 match extractions each month).</li>
<li>Open the actor and navigate to the <strong>Input</strong> tab.</li>
</ol>
<hr />
<h2>Step 2: Provide Match IDs or URLs</h2>
<p>In the <strong>Match ids or URLs</strong> input field, enter one ID or link per line. The actor accepts numeric IDs, short match links, or full fixture URLs:</p>
<pre><code class="language-plaintext">4506324
https://www.fotmob.com/match/4506324
https://www.fotmob.com/matches/aston-villa-vs-manchester-united/2pj9x9#4506324
</code></pre>
<p>To find a match ID manually, open any match on FotMob in your browser. The digits at the end of the URL (e.g. <code>fotmob.com/match/4506324</code>) are the ID.</p>
<p>If you need match IDs in bulk, you can chain this actor with the <strong><a href="https://apify.com/incognito_mode/fotmob-matches-scraper">FotMob Matches Scraper</a></strong> (which pulls all matches by calendar date) or the <strong><a href="https://apify.com/incognito_mode/fotmob-league-scraper">FotMob League Scraper</a></strong> (which dumps fixture lists for an entire league season).</p>
<hr />
<h2>Step 3: Configure Settings</h2>
<p>You can toggle individual data blocks to keep payloads compact and speed up execution:</p>
<ul>
<li><strong>Include player stats</strong>: Keep enabled if you need per-player ratings and metrics.</li>
<li><strong>Include events timeline</strong>: Keep enabled for goals, cards, and sub logs.</li>
<li><strong>Include lineups</strong>: Formations, managers, starting XI, and substitutes.</li>
<li><strong>Include team stats</strong>: Aggregate match stats (xG, possession, passes, fouls).</li>
<li><strong>Include head-to-head history</strong>: Historical meetings (leave off unless compiling match previews).</li>
<li><strong>Include raw data</strong>: Full unparsed response (leave off unless you need raw internal fields).</li>
</ul>
<p>Here is the equivalent JSON payload if you use the API or JSON editor:</p>
<pre><code class="language-json">{
  "matchIds": [
    "4506324",
    "https://www.fotmob.com/match/4506324"
  ],
  "includePlayerStats": true,
  "includeEvents": true,
  "includeLineups": true,
  "includeTeamStats": true,
  "includeH2H": false,
  "includeRawData": false
}
</code></pre>
<hr />
<h2>Step 4: Run the Scraper and Inspect Output</h2>
<p>Click <strong>Save &amp; Start</strong>. </p>
<p>A single match finishes in 2–3 seconds. Batches of 50 to 100 matches typically finish in 1 to 2 minutes.</p>
<p>A practical detail about billing: <strong>failed lookups are completely free</strong>. If you accidentally submit an invalid match ID, an unplayed fixture that won't load, or a broken URL, the actor logs the failure in the <code>SUMMARY</code> store and skips writing to the dataset. You are never billed for non-existent matches or failed lookups.</p>
<p>Here is a trimmed real-world output sample from Aston Villa vs Manchester United (<code>4506324</code>):</p>
<pre><code class="language-json">{
  "matchId": 4506324,
  "matchName": "Aston Villa-vs-Manchester United_Sun, Oct 6, 2024, 13:00 UTC",
  "matchUrl": "https://www.fotmob.com/match/4506324",
  "leagueId": 47,
  "leagueName": "Premier League",
  "round": "7",
  "countryCode": "ENG",
  "matchTimeUtc": "2024-10-06T13:00:00.000Z",
  "started": true,
  "finished": true,
  "homeTeam": { "id": 10252, "name": "Aston Villa", "score": 0 },
  "awayTeam": { "id": 10260, "name": "Manchester United", "score": 0 },
  "venue": "Villa Park",
  "referee": "Robert Jones",
  "attendance": 42682,
  "playerOfTheMatch": {
    "playerId": 268375,
    "name": "Emiliano Martinez",
    "teamName": "Aston Villa"
  },
  "teamStats": [
    { "group": "Top stats", "title": "Ball possession", "key": "BallPossesion", "home": 54, "away": 46 },
    { "group": "Top stats", "title": "Expected goals (xG)", "key": "expected_goals", "home": "0.50", "away": "0.56" },
    { "group": "Top stats", "title": "Total shots", "key": "total_shots", "home": 11, "away": 10 },
    { "group": "Passes", "title": "Accurate passes", "key": "accurate_passes", "home": "345 (83%)", "away": "302 (82%)" }
  ],
  "events": [
    {
      "type": "Card",
      "minute": 3,
      "minuteLabel": 3,
      "isHome": false,
      "playerId": 157723,
      "playerName": "Christian Eriksen",
      "card": "Yellow"
    },
    {
      "type": "Substitution",
      "minute": 12,
      "isHome": true,
      "playersInvolved": ["Diego Carlos", "Ezri Konsa"]
    }
  ],
  "lineups": {
    "home": {
      "teamId": 10252,
      "teamName": "Aston Villa",
      "formation": "4-4-1-1",
      "coach": "Unai Emery",
      "starters": [
        { "playerId": 268375, "name": "Emiliano Martinez", "shirtNumber": "23", "position": "Keeper" }
      ],
      "subs": []
    },
    "away": {
      "teamId": 10260,
      "teamName": "Manchester United",
      "formation": "4-2-3-1",
      "coach": "Erik ten Hag",
      "starters": [],
      "subs": []
    }
  },
  "playerStats": [
    {
      "playerId": 268375,
      "name": "Emiliano Martinez",
      "teamId": 10252,
      "teamName": "Aston Villa",
      "shirtNumber": "23",
      "position": "Keeper",
      "isGoalkeeper": true,
      "rating_title": 8.36,
      "minutes_played": 90,
      "saves": 4,
      "goals_conceded": 0,
      "accurate_passes": "24/32 (75.0%)",
      "touches": 40
    }
  ],
  "source": "fotmob-match-details-scraper",
  "scrapedAt": "2026-06-18T11:11:32+00:00"
}
</code></pre>
<p>Notice a few design choices in the schema:</p>
<ol>
<li><strong>Flattened player statistics</strong>: Instead of nested UI component hierarchies, each player has clean key-value pairs (<code>rating_title</code>, <code>minutes_played</code>, <code>expected_goals</code>, <code>touches</code>), so you can load the list directly into a Pandas DataFrame or SQL database.</li>
<li><strong>Sequential event logs</strong>: Substitutions, cards, and goals are chronologically unified in <code>events</code> so you don't need multiple lookups to reconstruct the flow of a match.</li>
<li><strong>State handling</strong>: For upcoming fixtures, lineups appear as soon as FotMob confirms them, while in-game stats remain omitted until kickoff.</li>
</ol>
<hr />
<h2>Step 5: Connecting to Python &amp; Spreadsheets</h2>
<h3>Python (<code>apify-client</code>)</h3>
<p>To automate regular data pulls or integrate the scraper into your pipeline, install the official client:</p>
<pre><code class="language-bash">pip install apify-client
</code></pre>
<p>Then trigger the actor and iterate over items directly:</p>
<pre><code class="language-python">from apify_client import ApifyClient

client = ApifyClient("&lt;YOUR_APIFY_TOKEN&gt;")

run_input = {
    "matchIds": [
        "4506324",
        "https://www.fotmob.com/match/4506324",
    ],
    "includePlayerStats": True,
    "includeEvents": True,
    "includeLineups": True,
    "includeTeamStats": True,
}

run = client.actor("incognito_mode/fotmob-match-details-scraper").call(run_input=run_input)

for match in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"Match: {match['homeTeam']['name']} {match['homeTeam']['score']} - {match['awayTeam']['score']} {match['awayTeam']['name']}")
    
    # Filter players with high match ratings
    for player in match.get("playerStats", []):
        if player.get("rating_title", 0) &gt;= 8.0:
            print(f"  {player['name']} ({player['teamName']}): Rating {player['rating_title']} | Minutes: {player['minutes_played']}")
</code></pre>
<h3>Google Sheets Integration</h3>
<p>If you prefer working in spreadsheets, you can use Google Sheets' <code>=IMPORTDATA()</code> with the actor dataset's clean CSV export URL:</p>
<pre><code class="language-plaintext">=IMPORTDATA("https://api.apify.com/v2/datasets/&lt;DATASET_ID&gt;/items?format=csv&amp;clean=true&amp;token=&lt;YOUR_APIFY_TOKEN&gt;")
</code></pre>
<hr />
<h2>Pricing &amp; Running Costs</h2>
<p>The actor runs on a pay-per-event pricing model with automatic volume tiers:</p>
<ul>
<li><strong>$0.01 per match</strong> ($10 per 1,000 matches) for the first 10,000 matches in a single run.</li>
<li><strong>$0.006 per match</strong> ($6 per 1,000 matches) for matches beyond 10,000 in a run.</li>
<li>A <strong>$0.02</strong> start fee per run.</li>
<li><strong>Failed or missing fixtures cost $0.00</strong>.</li>
</ul>
<p>At these rates:</p>
<ul>
<li>Scraping a 10-game Premier League weekend costs around <strong>$0.12</strong>.</li>
<li>A full 380-match domestic season runs about <strong>$3.82</strong>.</li>
<li>The free $5/month credit provided on Apify accounts covers roughly 500 complete match breakdowns every month.</li>
</ul>
<hr />
<h2>Summary</h2>
<p>If you need comprehensive football data without paying for commercial data vendor subscriptions or dealing with fragile custom scrapers that break when website headers change, this actor provides a solid, maintenance-free solution.</p>
<p>You can try it out directly on Apify: <strong><a href="https://apify.com/incognito_mode/fotmob-match-details-scraper">FotMob Match Details Scraper</a></strong>.</p>
]]></content:encoded></item></channel></rss>