From ocl at gih.com  Wed Mar  3 22:38:56 2010
From: ocl at gih.com (Olivier MJ Crepin-Leblond)
Date: Thu, 04 Mar 2010 00:38:56 +0200
Subject: [IPv6crawler-wg] Fwd: [IP] Scientists Strive to Map the
	Shape-Shifting Net
Message-ID: <4B8EE500.4080205@gih.com>

FYI - interesting article.
Kind regards,

Olivier

-------- Message original --------
Sujet: 	[IP] Scientists Strive to Map the Shape-Shifting Net
Date : 	Wed, 3 Mar 2010 16:39:19 -0500
De : 	David Farber <dave at farber.net>
R?pondre ? : 	dave at farber.net
Pour : 	ip <ip at v2.listbox.com>



Begin forwarded message:

From: dewayne at warpspeed.com (Dewayne Hendricks)
Date: March 3, 2010 2:35:46 AM EST
To: Dewayne-Net Technology List <xyzzy at warpspeed.com>
Subject: [Dewayne-Net] Scientists Strive to Map the Shape-Shifting Net

March 1, 2010
Scientists Strive to Map the Shape-Shifting Net
By JOHN MARKOFF
<http://www.nytimes.com/2010/03/02/science/02topo.html>

SAN FRANCISCO ? In a dimly lit chamber festooned with wires and hidden in one of California?s largest data centers, Tim Pozar is changing the shape of the Internet.

He is using what Internet engineers refer to as a ?meet-me room.? The room itself is enclosed in a building full of computers and routers. What Mr. Pozar does there is to informally wire together the networks of different businesses that want to freely share their Internet traffic.

The practice is known as peering, and it goes back to the earliest days of the Internet, when organizations would directly connect their networks instead of paying yet another company to route data traffic. Originally, the companies that owned the backbone of the Internet shared traffic. In recent years, however, the practice has increased to the point where some researchers who study the way global networks are put together believe that peering is changing the fundamental shape of the Internet, with serious consequences for its stability and security. Others see the vast increase in traffic staying within a structure that has remained essentially the same.

What is clear is that today a significant portion of Internet traffic does not flow through the backbone networks of giant Internet companies like AT&T and Level 3. Instead, it has begun to cascade in torrents of data on the edges of the network, as if a river in flood were carving new channels.

Some of this traffic coursing through new channels passes through public peering points like Mr. Pozar?s. And some flows through so-called dark networks, private channels created to move information more cheaply and efficiently within a business or any kind of organization. For instance, Google has privately built such a network so that video and search data need not pass through so many points to get to customers.

By its very nature, Internet networking technology is intended to support anarchic growth. Unlike earlier communication networks, the Internet is not controlled from the top down. This stems from an innovation at the heart of the Internet ? packet switching. From the start, the information moving around the Internet was broken up into so-called packets that could be sent on different paths to one destination where the original message ? whether it was e-mail, an image or sound file or instructions to another computer ? would be put back together in its original form. This packet-switching technology was conceived in the 1960s in England and the United States. It made delivery of a message through a network possible even if one or many of the nodes of the network failed. Indeed, this resistance to failure or attack was at the very core of the Internet, part of the essential nature of an organic, interconnected communications web with no single control point.

During the 1970s, a method emerged to create a network of networks. The connections depended on a communication protocol, or set of rules, known as TCP/IP, a series of letters familiar to anyone who has tried to set up their own wireless network at home. The global network of networks, the Internet, transformed the world, and continues to grow without central planning, extending itself into every area of life, fromFacebook to cyberwar.

[snip]RSS Feed: <http://www.warpspeed.com/wordpress>




-------------------------------------------
Archives: https://www.listbox.com/member/archive/247/=now
RSS Feed: https://www.listbox.com/member/archive/rss/247/
Powered by Listbox: http://www.listbox.com


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://salsa.gih.co.uk/pipermail/ipv6crawler-wg/attachments/20100304/2d527d6e/attachment.html>

From ocl at gih.com  Fri Mar  5 00:54:19 2010
From: ocl at gih.com (Olivier MJ Crepin-Leblond)
Date: Fri, 05 Mar 2010 02:54:19 +0200
Subject: [IPv6crawler-wg] Cairo follow-up
Message-ID: <4B90563B.6030407@gih.com>

Dear Sameh and Mohamed,

it was great meeting you in Cairo and visiting Nile University. Glad to
see progress, and we'll finalise a schedule for completion of the stages
we have discussed shortly.
I got to the airport at 4:00pm - My 7:15pm flight from Cairo to Dubai
has been cancelled today because of a technical fault with the aircraft.
We were placed in a hotel in Cairo 7 hours after the original flight
departure time (talk about an ordeal), and Friday's special flight will
be too late and will make me miss the connection to Nairobi. So I am
likely to be arriving there on Saturday evening now instead of Friday.
If I had known, I would have stayed an additional day! Such is life...

Kind regards,

Olivier

-- 
Olivier MJ Cr?pin-Leblond, PhD
http://www.gih.com/ocl.html



From mmg at doc.ic.ac.uk  Fri Mar  5 01:05:19 2010
From: mmg at doc.ic.ac.uk (Moustafa Ghanem)
Date: Fri, 05 Mar 2010 01:05:19 +0000
Subject: [IPv6crawler-wg] Cairo follow-up
In-Reply-To: <4B90563B.6030407@gih.com>
References: <4B90563B.6030407@gih.com>
Message-ID: <0fecf442-de70-4602-a9a8-ace30d55b854@email.android.com>

Oh my God ! Hooe they at least put you in nice hotel. What time is your flight tomorrow

"Olivier MJ Crepin-Leblond" <ocl at gih.com> wrote:

>Dear Sameh and Mohamed,
>
>it was great meeting you in Cairo and visiting Nile University. Glad to
>see progress, and we'll finalise a schedule for completion of the stages
>we have discussed shortly.
>I got to the airport at 4:00pm - My 7:15pm flight from Cairo to Dubai
>has been cancelled today because of a technical fault with the aircraft.
>We were placed in a hotel in Cairo 7 hours after the original flight
>departure time (talk about an ordeal), and Friday's special flight will
>be too late and will make me miss the connection to Nairobi. So I am
>likely to be arriving there on Saturday evening now instead of Friday.
>If I had known, I would have stayed an additional day! Such is life...
>
>Kind regards,
>
>Olivier
>
>-- 
>Olivier MJ Cr?pin-Leblond, PhD
>http://www.gih.com/ocl.html
>
>
>_______________________________________________
>IPv6crawler-wg mailing list
>IPv6crawler-wg at gih.com
>http://salsa.gih.co.uk/mailman/listinfo/ipv6crawler-wg

-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

From ocl at gih.com  Fri Mar  5 05:55:46 2010
From: ocl at gih.com (Olivier MJ Crepin-Leblond)
Date: Fri, 05 Mar 2010 07:55:46 +0200
Subject: [IPv6crawler-wg] Cairo follow-up
In-Reply-To: <0fecf442-de70-4602-a9a8-ace30d55b854@email.android.com>
References: <4B90563B.6030407@gih.com>
	<0fecf442-de70-4602-a9a8-ace30d55b854@email.android.com>
Message-ID: <4B909CE2.3030409@gih.com>

12:00, but our busses leave at 9:00am.. I've had 4 hours sleep. The
hotel's nice (Meridien). The bed is too comfortable.
O.

Le 05/03/2010 03:05, Moustafa Ghanem a ?crit :
> Oh my God ! Hooe they at least put you in nice hotel. What time is your flight tomorrow
>
> "Olivier MJ Crepin-Leblond" <ocl at gih.com> wrote:
>
>   
>> Dear Sameh and Mohamed,
>>
>> it was great meeting you in Cairo and visiting Nile University. Glad to
>> see progress, and we'll finalise a schedule for completion of the stages
>> we have discussed shortly.
>> I got to the airport at 4:00pm - My 7:15pm flight from Cairo to Dubai
>> has been cancelled today because of a technical fault with the aircraft.
>> We were placed in a hotel in Cairo 7 hours after the original flight
>> departure time (talk about an ordeal), and Friday's special flight will
>> be too late and will make me miss the connection to Nairobi. So I am
>> likely to be arriving there on Saturday evening now instead of Friday.
>> If I had known, I would have stayed an additional day! Such is life...
>>
>> Kind regards,
>>
>> Olivier
>>
>> -- 
>> Olivier MJ Cr?pin-Leblond, PhD
>> http://www.gih.com/ocl.html
>>
>>
>> _______________________________________________
>> IPv6crawler-wg mailing list
>> IPv6crawler-wg at gih.com
>> http://salsa.gih.co.uk/mailman/listinfo/ipv6crawler-wg
>>     
>   

-- 
Olivier MJ Cr?pin-Leblond, PhD
http://www.gih.com/ocl.html



From ocl at gih.com  Sun Mar 14 10:12:02 2010
From: ocl at gih.com (Olivier MJ Crepin-Leblond)
Date: Sun, 14 Mar 2010 13:12:02 +0300
Subject: [IPv6crawler-wg] Input domain name data / TLDs / Domains
Message-ID: <4B9CB672.90503@gih.com>

Hello everyone,

I am leaving Nairobi today but needed to update you on the situation wrt
obtaining an input set of data for the IPv6 Crawler. In our
specifications, we mention a "sample" of 50 000 domains. I guess that
we'll have a higher percentage of .UK domains, since our project is for
ISOC England, but we really need a sample even for the really small TLDs.

I have spoken to the head of Strategy at ISOC who liaised with PIR (the
people who run .org) to obtain zone maps for .org - as a test to see how
likely it is to be able to obtain this data.
The response, although not negative, was something along the lines of
"if you wish us to provide the data, you should get us involved" - ie.
this is a project which interests them too.

We are, of course, not willing to add yet more partners - since we would
end us with as many partners as there are gTLDs, ccTLDs etc.

I spoke at length to someone at Afilias (they support the tlds described
at: http://www.afilias.info/global-registry-services )
and I am yet to receive a formal reply. The person I spoke to told me:
"but why don't you just do a zone dump for each TLD? Then you can
pick/choose what domain you wish..."

I am unsure whether Zone dumps are possible for every TLD these days?
Will someone please advise whether we can do this?
I asked re: LEGALITY of doing such a thing. I am told there is nothing
wrong with it, a-priori, since we are not using the data collected to
spam people or do anything malicious. This area of the law is not
defined, so unless we do something malicious, we are fine.

Okay - I'll look to your feedback on that.

Kind regards,

Olivier


-- 
Olivier MJ Cr?pin-Leblond, PhD
http://www.gih.com/ocl.html



From sansary at nileuniversity.edu.eg  Sun Mar 14 15:40:30 2010
From: sansary at nileuniversity.edu.eg (Sameh El-Ansary)
Date: Sun, 14 Mar 2010 17:40:30 +0200
Subject: [IPv6crawler-wg] Input domain name data / TLDs / Domains
In-Reply-To: <4B9CB672.90503@gih.com>
Message-ID: <C7C2D00E.4C6F%sansary@nileuniversity.edu.eg>

Domains who allow zone transfers are extremely rare nowadays.

Sameh  


On 3/14/10 12:12 PM, "Olivier MJ Crepin-Leblond" <ocl at gih.com> wrote:

> Hello everyone,
> 
> I am leaving Nairobi today but needed to update you on the situation wrt
> obtaining an input set of data for the IPv6 Crawler. In our
> specifications, we mention a "sample" of 50 000 domains. I guess that
> we'll have a higher percentage of .UK domains, since our project is for
> ISOC England, but we really need a sample even for the really small TLDs.
> 
> I have spoken to the head of Strategy at ISOC who liaised with PIR (the
> people who run .org) to obtain zone maps for .org - as a test to see how
> likely it is to be able to obtain this data.
> The response, although not negative, was something along the lines of
> "if you wish us to provide the data, you should get us involved" - ie.
> this is a project which interests them too.
> 
> We are, of course, not willing to add yet more partners - since we would
> end us with as many partners as there are gTLDs, ccTLDs etc.
> 
> I spoke at length to someone at Afilias (they support the tlds described
> at: http://www.afilias.info/global-registry-services )
> and I am yet to receive a formal reply. The person I spoke to told me:
> "but why don't you just do a zone dump for each TLD? Then you can
> pick/choose what domain you wish..."
> 
> I am unsure whether Zone dumps are possible for every TLD these days?
> Will someone please advise whether we can do this?
> I asked re: LEGALITY of doing such a thing. I am told there is nothing
> wrong with it, a-priori, since we are not using the data collected to
> spam people or do anything malicious. This area of the law is not
> defined, so unless we do something malicious, we are fine.
> 
> Okay - I'll look to your feedback on that.
> 
> Kind regards,
> 
> Olivier
> 



From ocl at gih.com  Mon Mar 22 08:00:24 2010
From: ocl at gih.com (Olivier MJ Crepin-Leblond)
Date: Mon, 22 Mar 2010 09:00:24 +0100
Subject: [IPv6crawler-wg] In search of domain names - found!
Message-ID: <4BA72398.7020409@gih.com>

Good news!

Looking through the Web last night, I came across a link to download a
CSV file of Alexa"s 1 million (yes, you're reading this correctly - 1
000 000 - most popular Web sites. This provides us with the ability of
having a great sample of domain names to test. We can select "UK
universities" by doing an fgrep on ".ac.uk" and get 400 results.
We can select "UK government Web sites" and get 453 domains.
Egyptian sites ending in ".eg" = 175 sites etc. etc.

These are great samples!

The file is available in my sub-directory /home/ocl/data on
crawler.ipv6matrix.org

Apparently it gets updated daily on the Alexa Web site - but I don't
hink that we need to update ours more than once every few months.

Warm regards,

Olivier

-- 
Olivier MJ Cr?pin-Leblond, PhD
http://www.gih.com/ocl.html



From sansary at nileuniversity.edu.eg  Mon Mar 22 08:34:58 2010
From: sansary at nileuniversity.edu.eg (Sameh El-Ansary)
Date: Mon, 22 Mar 2010 10:34:58 +0200
Subject: [IPv6crawler-wg] In search of domain names - found!
In-Reply-To: <4BA72398.7020409@gih.com>
Message-ID: <C7CCF852.4D0B%sansary@nileuniversity.edu.eg>

1 - That's great!!
2 - For the logs with errors as you reported, can you please,
point-to-location/send them to me?

Sameh



On 3/22/10 10:00 AM, "Olivier MJ Crepin-Leblond" <ocl at gih.com> wrote:

> Good news!
> 
> Looking through the Web last night, I came across a link to download a
> CSV file of Alexa"s 1 million (yes, you're reading this correctly - 1
> 000 000 - most popular Web sites. This provides us with the ability of
> having a great sample of domain names to test. We can select "UK
> universities" by doing an fgrep on ".ac.uk" and get 400 results.
> We can select "UK government Web sites" and get 453 domains.
> Egyptian sites ending in ".eg" = 175 sites etc. etc.
> 
> These are great samples!
> 
> The file is available in my sub-directory /home/ocl/data on
> crawler.ipv6matrix.org
> 
> Apparently it gets updated daily on the Alexa Web site - but I don't
> hink that we need to update ours more than once every few months.
> 
> Warm regards,
> 
> Olivier