<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hi,<div class=""><br class=""></div><div class="">This is pretty cool, and the db is slowly but surely making its way into Big Data territory ;)</div><div class=""><br class=""></div><div class="">The internationalised domain name problem is also interesting. Christian’s solution sounds good.</div><div class=""><br class=""></div><div class="">It seems timely to have another push on both virtualising the system (so we can run it from other vantage points) and distributing the data / results to minimise any potential of any loss to the increasingly valuable data set.</div><div class=""><br class=""></div><div class="">This might fit well with the growing web observatory activity in Southampton. We also have a new highly resilient data centre in Fareham which could be a good place for the virtualised copy to be hosted. If you’re OK with it, I can make some contacts to initiate, but let me know.<br class=""><div class=""><br class="webkit-block-placeholder"></div><div class="">
<div class="">Tim</div><div class=""><br class=""></div><br class="Apple-interchange-newline">
</div>
<br class=""><div><blockquote type="cite" class=""><div class="">On 6 Feb 2016, at 17:23, Olivier MJ Crepin-Leblond <<a href="mailto:ocl@gih.com" class="">ocl@gih.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta content="text/html; charset=utf-8" http-equiv="Content-Type" class="">
<div bgcolor="#FFFFFF" text="#000000" class="">
Hello Christian,<br class="">
<br class="">
the the sqlite database comes in when it comes down to displaying
the results. The results of the crawls are in native CSV. All 306Gb
of these. The Sqlite database is much smaller as it only uses a
subset of all data collected (the data which is used in the GUI) and
we are not using a single Sqlite database but one for each crawl - a
summary of each crawl for each TLD.<br class="">
The question of Sqlite v3 is a good one -- and I have unfortunately
got no idea whether it would work or whether it would break things.
To be added to the list of things to do.<br class="">
Kindest regards,<br class="">
<br class="">
Olivier<br class="">
<br class="">
<div class="moz-cite-prefix">On 06/02/2016 18:09, Christian de
Larrinaga wrote:<br class="">
</div>
<blockquote cite="mid:56B628E2.5090601@firsthand.net" type="cite" class="">
<meta content="text/html; charset=utf-8" http-equiv="Content-Type" class="">
<div style="font-size: 12pt;font-family: Helvetica Neue;" class=""><span style="font-family: Helvetica Neue;" class="">That is a humungous large
sqlite database! or are you only collecting the data as a form
of cache using sqlite and then exporting it out once organised
into csv?<br class="">
<br class="">
Sqlite v3 supports utf-8 which might help?<br class="">
if it doesn't break something else of course. <br class="">
<br class="">
C <br class="">
</span><br class="">
<span class="">Olivier MJ Crepin-Leblond wrote:</span><br class="">
<blockquote cite="mid:56B62022.7060009@gih.com" type="cite" class="">
<meta content="text/html; charset=utf-8" http-equiv="Content-Type" class="">
Hello all,<br class="">
<br class="">
another update: the first complete run using the new TLDs has
completed! <br class="">
You can view the results up to February 2016 from <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.ipv6matrix.org/"></a><a class="moz-txt-link-freetext" href="http://www.ipv6matrix.org/">http://www.ipv6matrix.org</a><br class="">
<br class="">
In adding new gTLDs we have hit a snag, although this snag
does not significantly affect overall results since it appears
to only affect a tiny number of domains.<br class="">
<br class="">
I am speaking about Internationalized Top Level Domains
(IDNs):<br class="">
<br class="">
xn--3e0b707e xn--80adxhks xn--90ais xn--j1amh
xn--pgbs0dh xn--wgbl6a<br class="">
xn--4gbrim xn--80asehdb xn--d1acj3b xn--p1ai
xn--q9jyb4c<br class="">
<br class="">
Each of these is the ASCII equivalent of a non ASCII domain
name. Whist the Crawler works well with them and we are able
to collect all of the data pertaining to crawls in IDNs, the
program that builds the Database uses SQLite. Until now,
database entries made use of domain names that were ASCII -
but IDNs use a double dash "--" in the domain. SQLite coughs
on DASH - so we have not been able to produce the database
needed for the displaying of the results when including IDNs.<br class="">
<br class="">
Until we have a workaround, I have manually isolated data
collected for IDNs, which means we still collect them, but we
will not take them into account in the final database results.
As I have said, this is a tiny subset of domains: 760 entries
out of a total of 1 Million domains.<br class="">
<br class="">
I am *still* drafting a very long article for RIPE labs. In
fact, we might publish this in two parts. In the meantime, the
results appear to be somehow consistent with results of other
tracking projects, some of which use other methods to track
IPv6 adoption:<br class="">
<br class="">
- <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://6lab.cisco.com/stats/">http://6lab.cisco.com/stats/</a><br class="">
- <a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://www.vyncke.org/ipv6status/">https://www.vyncke.org/ipv6status/</a><br class="">
- <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.mrp.net/ipv6_survey/">http://www.mrp.net/ipv6_survey/</a><br class="">
<br class="">
We now have 306 Gb of comma separated value text data in
store, tracing back the spread of the IPv6 Internet since July
2010. (294Gb in November 2015)<br class="">
<br class="">
I look forward to your kind feedback.<br class="">
<br class="">
Warmest regards,<br class="">
Olivier<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">On 26/11/2015 19:32, Olivier MJ
Crepin-Leblond wrote:<br class="">
</div>
<blockquote cite="mid:56575051.3070402@gih.com" type="cite" class="">
<meta http-equiv="content-type" content="text/html;
charset=utf-8" class="">
Hello all,<br class="">
<br class="">
Two worthy pieces of news regarding the IPv6 Matrix Project
( <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.ipv6matrix.org/">http://www.ipv6matrix.org</a>
):<br class="">
<br class="">
1. I have updated the Web site with the latest results
ending in late October - hence noting a Crawl display date
of November 2015.<br class="">
We now have 294 Gb of comma separated value text data in
store, tracing back the spread of the IPv6 Internet since
July 2010.<br class="">
Altogether, we ran the text approximately 36 times on all 1
million Alexa busiest Domain names. This represented testing
of about 6.5 million hosts, carefully collecting traceroute
information for each and every of them. We now have a very
unique database that is showing the spread of the IPv6
Internet information sources worldwide.<br class="">
<br class="">
2. Today I took out my very dusty Linux & Python gloves
and performed a much needed update to the IPv6 Matrix
Crawler input database, including the Alexa 1 million list
as well as GeoIP Databases.<br class="">
<br class="">
Indeed, the Alexa database of the world's 1 million busiest
Web sites dated from the Crawler's first inception in the
first half of 2010.<br class="">
We're more than 5 years later!<br class="">
<br class="">
In a way, keeping the same input database has kept the base
of crawls the steady thus the ability to compare results was
possible. However, the flip-side of the coin is that we are
ending up with more and more domain names marked as being
dysfunctional. Nearly 5% of the domain names in the database
were unreachable. The updated input database should resolve
this, but we might also see a jump in some results. It will
be interesting to see what the next run yields.<br class="">
Why do we not update the input database more often? Because
buried in that database are the domain names of the people
who wanted to opt out over the years. Having never thought
about this, I spent several hours tracing back 5 years of
emails of people complaining about the crawl triggering
their firewalls. I put together a blacklist of domain names
I have manually deleted from the crawl input files. <br class="">
The blacklist, as it stands now:<br class="">
<br class="">
Deleted:<br class="">
<br class="">
<a href="http://it-mate.co.uk" class="">it-mate.co.uk</a><br class="">
<a href="http://indianic.com" class="">indianic.com</a><br class="">
<a href="http://your-server.de" class="">your-server.de</a><br class="">
<a href="http://catacombscds.com" class="">catacombscds.com</a><br class="">
<a href="http://dewlance.com" class="">dewlance.com</a><br class="">
<a href="http://tcs.com" class="">tcs.com</a><br class="">
<a href="http://printweb.de" class="">printweb.de</a><br class="">
<a href="http://nocser.net" class="">nocser.net</a><br class="">
<a href="http://shoppingnsales.com" class="">shoppingnsales.com</a><br class="">
<a href="http://bsaadmail.com" class="">bsaadmail.com</a><br class="">
<a href="http://epayservice.ru" class="">epayservice.ru</a><br class="">
<a href="http://4footyfans.com" class="">4footyfans.com</a> <br class="">
<a href="http://guitarspeed99.com" class="">guitarspeed99.com</a><br class="">
<a href="http://saga.co.uk" class="">saga.co.uk</a><br class="">
<br class="">
Already gone from the current Alexa list:<br class="">
<br class="">
<a href="http://infinityautosurf.com" class="">infinityautosurf.com</a><br class="">
<a href="http://canada-traffic.com" class="">canada-traffic.com</a><br class="">
<a href="http://usahitz.com" class="">usahitz.com</a><br class="">
jawatankosong.com.my<br class="">
4d.com.my<br class="">
<a href="http://fitnessuncovered.co.uk" class="">fitnessuncovered.co.uk</a><br class="">
<a href="http://kualalumpurbookfair.com" class="">kualalumpurbookfair.com</a><br class="">
<a href="http://xgen-it.com" class="">xgen-it.com</a><br class="">
<a href="http://bpanet.de" class="">bpanet.de</a><br class="">
<a href="http://edns.de" class="">edns.de</a><br class="">
<a href="http://back2web.de" class="">back2web.de</a><br class="">
<a href="http://waaaouh.com" class="">waaaouh.com</a><br class="">
<a href="http://every-web.com" class="">every-web.com</a><br class="">
<a href="http://w3sexe.com" class="">w3sexe.com</a><br class="">
<a href="http://gratuits-web.com" class="">gratuits-web.com</a><br class="">
<a href="http://france-mateur.com" class="">france-mateur.com</a><br class="">
<a href="http://pliagedepapier.com" class="">pliagedepapier.com</a><br class="">
<a href="http://immobilieretparticuliers.com" class="">immobilieretparticuliers.com</a><br class="">
<a href="http://chronobio.com" class="">chronobio.com</a><br class="">
<a href="http://stickers-origines.com" class="">stickers-origines.com</a><br class="">
<a href="http://tailor-made.co.uk" class="">tailor-made.co.uk</a><br class="">
<br class="">
With these out of the input files, we are able to start the
next crawl. <b class=""><br class="">
I hope I have not missed any complaints, but if I have,
this is advance notice that we might receive a few emails
in the forthcoming weeks. We might also receive a few
emails from sites that have appeared on the Alexa 1
million list since 2010.</b><br class="">
<br class="">
Back to this list, the excellent filtering program which was
used to process the original list and clean it up was used
again for the modern list. The Alexa list had a number of
domain names which were actually sub-directories in the
past, as well as some invalid domains. Alexa has since
tightened its act. The latest Alexa list is much cleaner. It
holds 999998 valid domains vs. 984587 domains for the
original 2010 list.<br class="">
<br class="">
Finally, new gTLDs have now appeared in the Alexa list,
including some Internationalised Domain Names (IDNs). The
world is indeed a very different place!<br class="">
It will be interesting to see how the Crawler as well as all
other scripts to process the information into displayable
data on the Web server, will cope with these:<br class="">
<br class="">
academy.csv consulting.csv guide.csv
one.csv supply.csv<br class="">
accountant.csv contractors.csv guru.csv
onl.csv support.csv<br class="">
actor.csv cool.csv hamburg.csv
online.csv surf.csv<br class="">
ads.csv country.csv haus.csv
ooo.csv swiss.csv<br class="">
adult.csv creditcard.csv healthcare.csv
orange.csv sydney.csv<br class="">
agency.csv cricket.csv help.csv
ovh.csv systems.csv<br class="">
alsace.csv cymru.csv hiphop.csv
paris.csv taipei.csv<br class="">
amsterdam.csv dance.csv holiday.csv
partners.csv tattoo.csv<br class="">
app.csv date.csv horse.csv
parts.csv team.csv<br class="">
archi.csv dating.csv host.csv
party.csv tech.csv<br class="">
associates.csv deals.csv hosting.csv
photo.csv technology.csv<br class="">
attorney.csv delivery.csv house.csv
photography.csv theater.csv<br class="">
auction.csv desi.csv how.csv
photos.csv tienda.csv<br class="">
audio.csv design.csv immobilien.csv
pics.csv tips.csv<br class="">
axa.csv dev.csv immo.csv
pictures.csv tirol.csv<br class="">
barclaycard.csv diet.csv ink.csv
pink.csv today.csv<br class="">
barclays.csv digital.csv international.csv
pizza.csv tokyo.csv<br class="">
bar.csv direct.csv investments.csv
place.csv tools.csv<br class="">
bargains.csv directory.csv irish.csv
plus.csv top.csv<br class="">
bayern.csv discount.csv jetzt.csv
poker.csv town.csv<br class="">
beer.csv dog.csv joburg.csv
porn.csv toys.csv<br class="">
berlin.csv domains.csv juegos.csv
post.csv trade.csv<br class="">
best.csv earth.csv kim.csv
press.csv training.csv<br class="">
bid.csv education.csv kitchen.csv
prod.csv trust.csv<br class="">
bike.csv email.csv kiwi.csv
productions.csv university.csv<br class="">
bio.csv emerck.csv koeln.csv
properties.csv uno.csv<br class="">
black.csv energy.csv krd.csv
property.csv uol.csv<br class="">
blackfriday.csv equipment.csv kred.csv
pub.csv vacations.csv<br class="">
blue.csv estate.csv land.csv
quebec.csv vegas.csv<br class="">
bnpparibas.csv eus.csv law.csv
realtor.csv ventures.csv<br class="">
boo.csv events.csv legal.csv
recipes.csv video.csv<br class="">
boutique.csv exchange.csv life.csv
red.csv vision.csv<br class="">
brussels.csv expert.csv limited.csv
rehab.csv voyage.csv<br class="">
build.csv exposed.csv link.csv
reise.csv wales.csv<br class="">
builders.csv express.csv live.csv
reisen.csv wang.csv<br class="">
business.csv fail.csv lol.csv
ren.csv watch.csv<br class="">
buzz.csv faith.csv london.csv
rentals.csv webcam.csv<br class="">
bzh.csv farm.csv love.csv
repair.csv website.csv<br class="">
cab.csv finance.csv luxury.csv
report.csv wien.csv<br class="">
camera.csv fish.csv management.csv
rest.csv wiki.csv<br class="">
camp.csv fishing.csv mango.csv
review.csv win.csv<br class="">
capital.csv fit.csv market.csv
reviews.csv windows.csv<br class="">
cards.csv fitness.csv marketing.csv
rio.csv work.csv<br class="">
care.csv flights.csv markets.csv
rip.csv works.csv<br class="">
career.csv foo.csv media.csv
rocks.csv world.csv<br class="">
careers.csv football.csv melbourne.csv
ruhr.csv wtf.csv<br class="">
casa.csv forsale.csv menu.csv
ryukyu.csv xn--3e0b707e.csv<br class="">
cash.csv foundation.csv microsoft.csv
sale.csv xn--4gbrim.csv<br class="">
casino.csv frl.csv moda.csv
scb.csv xn--80adxhks.csv<br class="">
center.csv fund.csv moe.csv
school.csv xn--80asehdb.csv<br class="">
ceo.csv futbol.csv monash.csv
science.csv xn--90ais.csv<br class="">
chat.csv gal.csv money.csv
scot.csv xn--d1acj3b.csv<br class="">
church.csv gallery.csv moscow.csv
services.csv xn--j1amh.csv<br class="">
city.csv garden.csv movie.csv
sexy.csv xn--p1ai.csv<br class="">
claims.csv gent.csv nagoya.csv
shiksha.csv xn--pgbs0dh.csv<br class="">
click.csv gift.csv network.csv
shoes.csv xn--q9jyb4c.csv<br class="">
clinic.csv gifts.csv new.csv
singles.csv xn--wgbl6a.csv<br class="">
clothing.csv glass.csv news.csv
site.csv xxx.csv<br class="">
club.csv global.csv nexus.csv
social.csv xyz.csv<br class="">
coach.csv globo.csv ngo.csv
software.csv yandex.csv<br class="">
codes.csv gmail.csv ninja.csv
solar.csv yoga.csv<br class="">
coffee.csv goo.csv nrw.csv
solutions.csv yokohama.csv<br class="">
college.csv goog.csv ntt.csv
soy.csv youtube.csv<br class="">
community.csv google.csv nyc.csv
space.csv zone.csv<br class="">
company.csv graphics.csv office.csv
style.csv<br class="">
computer.csv gratis.csv okinawa.csv
sucks.csv<br class="">
<br class="">
In the meantime I'd like to cite again the Nile University
Crew expertly led by Sameh El Ansary for designing and
coding a Crawler's that been able to cope with shifting
through 5 years of DNS junk with minimal maintenance, save
the love and attention I give the servers by keeping them up
to date with patches so they don't end up toppling over.
They haven't been rebooted in 464 days and I am crossing
fingers for their well-being.<br class="">
And of course, thanks to the University of Southampton Crew
who built the excellent 2nd version of the Web Site under
Tim Chown's supervision.<br class="">
<br class="">
I am still writing an article for RIPE Labs - just
struggling to find the time to finish it, but getting there.<br class="">
<br class="">
Warmest regards,<br class="">
<br class="">
Olivier<br class="">
<br class="">
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<br class="">
<pre wrap="" class="">_______________________________________________
IPv6crawler-wg mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:IPv6crawler-wg@gih.co.uk">IPv6crawler-wg@gih.co.uk</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://gypsy.gih.co.uk/mailman/listinfo/ipv6crawler-wg">http://gypsy.gih.co.uk/mailman/listinfo/ipv6crawler-wg</a>
</pre>
</blockquote>
<br class="">
<pre class="moz-signature" cols="72">--
Olivier MJ Crépin-Leblond, PhD
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.gih.com/ocl.html">http://www.gih.com/ocl.html</a>
</pre>
</blockquote>
<br class="">
<div class="moz-signature">-- <br class="">
Christian de Larrinaga FBCS, CITP,<br class="">
-------------------------<br class="">
<span style="font-weight: bold;" class="">@ FirstHand</span><br style="font-weight: bold;" class="">
-------------------------<br class="">
+44 7989 386778<br class="">
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:cdel@firsthand.net">cdel@firsthand.net</a> <br class="">
-------------------------<br class="">
<br class="">
</div>
</div>
</blockquote>
<br class="">
<pre class="moz-signature" cols="72">--
Olivier MJ Crépin-Leblond, PhD
<a class="moz-txt-link-freetext" href="http://www.gih.com/ocl.html">http://www.gih.com/ocl.html</a>
</pre>
</div>
</div></blockquote></div><br class=""></div></body></html>