La prezentarea trecuta RONUA Andrei Rinea a vorbit despre GEO IP lookup pentru aflarea locatiei vizitatorilor.

Motivat de faptul ca Andrei Rinea este un perfectionist si a facut doua cautari, am incercat sa fac si eu o (asa zisa) imbunatatire.

Mai intii am vrut sa citesc mai usor din csv-urile respective.Sa zicem ca vreau sa citesc asta:

Copyright (c) 2011 MaxMind Inc.  All Rights Reserved.
startIpNum,endIpNum,locId
“16777216”,”16777471″,”17″
“16777472”,”16778239″,”49″

……

Am downloadat FileHelpers si am scris codul:

 [DelimitedRecord(",")]
    public class IPRangeAndrei
    {
        [FieldQuoted()] public long Start;
        [FieldQuoted()] public long End;
        [FieldQuoted()] public int LocID;
    }
    public class FileHelperImporter
    {
        public IEnumerable<GeoLocation> ImportLocations(string csvFilePath, int HeaderLines=2)
        {
            DelimitedFileEngine engine = new DelimitedFileEngine(typeof (GeoLocation));
            engine.Options.IgnoreFirstLines = HeaderLines;            
            return engine.ReadFile(csvFilePath) as GeoLocation[]; 
        }        

        public IEnumerable<IPRangeAndrei> ImportIP(string csvFilePath, int HeaderLines=2)
        {

            DelimitedFileEngine engine = new DelimitedFileEngine(typeof(IPRangeAndrei));
            engine.Options.IgnoreFirstLines = HeaderLines;
            return engine.ReadFile(csvFilePath) as IPRangeAndrei[]; 

        }
    }

Pare un pic mai clar . Din pacate, e si mai incet – 2 secunde fata de 1 secunda la un fisier de 117 MB .

Apoi am vrut sa incar la compile time csv-urile, nu la runtime.
Prima oara am incarcat cele de GeoLiteCity-Location.csv – puteti sa le luati de aici : http://www.maxmind.com/app/geolitecity

Am scris urmatorul fisier .tt

<#@ template debug="false" hostspecific="false" language="C#" #>
<#@ output extension=".cs" #>
<#@ assembly name="C:\Users\andrei\Desktop\g\Module\Module\bin\Release\Module.dll" #>
<#
Module.CsvImporter  importer= new Module.CsvImporter ();
var locations = importer.ImportLocations(@"C:\Users\andrei\Desktop\GeoLiteCity_20111004\GeoLiteCity-Location.csv");
#>
using System.Collections.Generic;
using Module;
namespace AndreiTT
{
	public class Location:Dictionary<string, GeoLocation>  
	{
		
		public string GetFileKey(long Key)
		{
			return "K" + Key;
		}
		public Location():base()
		{
			<#
			foreach(var geo in locations)
			
			{
				#>
				this.Add(GetFileKey(<#=geo.Id#>),new GeoLocation(<#=geo.Id#>, "<#=geo.CountryCode#>", "<#=geo.RegionCode#>", "<#=geo.City#>", "<#=geo.PostalCode#>", <#=geo.Latitude#>f, <#=geo.Longitude#>f));
			<#
			}
#>
		}
	}
}

si a iesit fisierul asta ( de vreo 30 MB)

using System.Collections.Generic;
using Module;
namespace AndreiTT
{
	public class Location:Dictionary<string, GeoLocation>  
	{
		
		public string GetFileKey(long Key)
		{
			return "K" + Key;
		}
		public Location():base(320000)
		{
							this.Add(GetFileKey(1),new GeoLocation(1, "O1", "", "", "", 0f, 0f));
							this.Add(GetFileKey(2),new GeoLocation(2, "AP", "", "", "", 35f, 105f));
							this.Add(GetFileKey(3),new GeoLocation(3, "EU", "", "", "", 47f, 8f));
							this.Add(GetFileKey(4),new GeoLocation(4, "AD", "", "", "", 42.5f, 1.5f));
							this.Add(GetFileKey(5),new GeoLocation(5, "AE", "", "", "", 24f, 54f));

Inspirat de aceasta reusita, am facut acelasi lucru si pentru GeoLiteCity-Blocks.
Singura problema a fost ca a dat un fisier de 194 MB – care , la compilare cu Visual studio a dat:
Source file ‘C:\Users\andrei\Desktop\g\App\AndreiTT\Blocks.cs’ could not be opened (‘Unspecified error ‘) 

OK, atunci l-am compilat cu msbuild – a dat:

MSBUild: CSC : error CS0010: Unexpected fatal error – ‘a’

M-am uitat la
http://msdn.microsoft.com/en-us/library/bb546037%28v=vs.90%29.aspx
Si zice:

Unexpected fatal error — ‘error’.
This error is generated when something completely unexpected occurs to stop compilation.

Asa ca l-am spart pe blocks.tt in 5 bucati, fiecare de 1.000.000 de bucati si a generat fisiere de 30- 55 MB . A generat partial class, asta ca sa le puna pe toate la un loc:
Asa ca msbuild a dat o eroare omeneasca
(CoreCompile target) ->
CSC : error CS0010: Unexpected fatal error — ‘Not enough storage is available to process this command. ‘ [C:\Users\andrei\Desktop\g\App\AndreiTT\AndreiTT.c
sproj]

0 Warning(s)
1 Error(s)

Time Elapsed 00:03:34.15

Ok, daca nu poate sa asambleze mai multe .tt din fisiere diferite, atunci sa le pun in dll-uri diferite . Am facut si asta.
A generat 5 dll-uri, fiecare de cam cit 20 MB.
Le-am referentiat , am scris codul sa caute in fiecare dll si … la rulare OutOfMemoryException.

Andrei Rinea ma prevenise – asa ca i-am dat telefon sa ii spun ca e asa cum zice el. M-a intrebat daca am x64. Asa ca … l-am compilat si pe x64. La rulare mi-a inghitit toti cei 4GB de RAM si a blocat PC-ul.

Asa incit fiti prudenti – compile time nu e intotdeauna mai bun decat runtime !

Daca vreti sa experimentati, gasiti aici un proiect cu care sa va jucati :

http://msprogrammer.serviciipeweb.ro/wp-content/uploads/runcomp.7z