Re: Steve Marsden's WBM screenshot: The CEOP Home page for April 30, 2007 also refers to Missing Madeleine.

by siobhan3443 02.07.15 18:55

why would not not use the original files which none of these dates appear in? huh, seems strange, can we all get a look at this excel file?

but then the good dr did declare chirpy insects fake link as definite proof
forgive me if i take the ramblings of an it expert who can't upload a pdf file with a pinch of salt

by SixMillionQuid 02.07.15 19:23

whodunit wrote:[You must be registered and logged in to see this link.] When a page has a link to click a downloadable poster that was not available for a while later then that page could not have existed on April 30th and is therefore a mistake..

~~~

I am not a member of the HiDeho FB group so I cannot respond over there but I know she is a member here so maybe she will see this and respond here, as she has insisted on the above several times and nobody is correcting this seriously mistaken assumption.

I am begging you..please read this FAQ section which has been quoted several times. It explains, in no uncertain terms, why a poster made at a later date was linked to on a page that existed in April.

How did I end up on the live version of a site? or I clicked on X date, but now I am on Y date, how is that possible?
Not every date for every site archived is 100% complete. When you are surfing an incomplete archived site the Wayback Machine will grab the closest available date to the one you are in for the links that are missing. In the event that we do not have the link archived at all, the Wayback Machine will look for the link on the live web and grab it if available. Pay attention to the date code embedded in the archived url. This is the list of numbers in the middle; it translates as yyyymmddhhmmss. For example in this url [You must be registered and logged in to see this link.] the date the site was crawled was Feb 29, 2000 at 12:33 and 40 seconds.

The bit I've underlined indicates that a a CEOP page was crawled / archived on 30 April 2007.

by HKP 02.07.15 19:31

@siobahn. Firstly, the reason I downloaded it to excel is for ease of analysis, the data can then be easily filtered and sorted by anything that is in the cells, searchability is also greatly increased. Can you get to see the file, well its over a megabyte so not easy to post, I wouldn't send it to any unstrusted source so that rules you out sorry.

You seem to be taking cheap shots and trying to discredit me (I never opened your links as it is probably Syn having another laugh she said yesterday that she'd done another). Finally show me where I claimed, said or even intimated that I'm an IT expert I'm not and that's all that you need to know. No doubt your sidekick will be along soon with her subterfuge.

I stand by my analysis its been done as hard endeavour looking for some sort of answer, read my post again and enjoy!

by whodunit 02.07.15 19:38

The April 30 date is not only in the date code of the url, it was embedded in the page itself, it is NOW embedded in 'next/previous capture' codes of the next/previous captured pages, and it is listed in the index.

I honestly do not know what more people need to accept that mccann.html was captured on April 30, 2007. All arguments 'against' are merely massive walls of confusion, obfuscation, and distraction.

As Agnos said at Only in America:

"If a person purports to know the class of "error" that has generated the date code in question; and yet that person is unable to assert the extent and duration of said "error" across the WBM structure, then that person does not know the class of "error" at all."

It is not for proponents of the 'early' existence of the page to prove what is self evident. It is for the debunkers to prove that it did not exist.

by HKP 02.07.15 19:41

@SixMillionQuid. Not meant disrespectfully but that FAQ from Wayback has been posted quit a few times over the piece it's just that some don't want to pay any attention.

by **Tony Bennett** 02.07.15 20:19

XXXXXXXX wrote:1. How did it go at Court today? I am looking forward to your report - it is a fascinating case.

REPLY: Les Balkwell asked the judge for permission to amend (by which I mean 'add to', not change) his original pleadings in 23 different respects.

He was allowed 16 additions, the other 7 were refused for legal reasons

2. Will your next FOI request be in Dutch?

REPLY: No.

by **Tony Bennett** 02.07.15 20:39

Syn wrote:
Tony Bennett wrote:
Syn wrote:Problem with subsets says Chris Butler....

.... that brings us right back to the beginning where I posted this on 17th June when this merry go round first started.

I have now underlined the relevant part

[You must be registered and logged in to see this link.]

I shall try and help :)

1. how many hundreds/thousands/tens of thousands/hundreds of thousands of Wayback records are false because of this faulty subset

I have no idea, what makes you think that I would know? What I can be logically sure of is that it will not be specific to just 20070430 archives. Millions of pages crawled and archived on 20070430 and many other dates will have been archived incorrectly I'm sure. However, the millions affected will be a tiny, tiny percentage give how many website WB crawls each year. There is no way to ascertain how many of the millions of websites out there in 2007 and probably prior and beyond 2007 which were crawled and archived by WB had pdf links, flash player, needed server side verification for SSIs etc are out there that may be incorrect due to the claimed subset issue. Wayback are not going to tell us which other sites were affected and identifying similarly affected websites would be like trying to find a needle in a haystack for us mere mortals. We are only able to identify an anomaly with the CEOP pages because we have an identifier of 20070503 as the date that it is claimed that Madeleine went missing. Therefore with this point of reference it is possible to cross reference any archived content. If you have any idea how to identify any other website that may have been similarly affected regardless of date, please do let me know as I cannot fathom any. There is no point in saying ohh Amazon or the BBC was archived on 20070430 with no problem at all as they may not have had any pdf links, flash content or server side SSIs or anything else that may have made archiving a website go squewiffy once WB appended their server side javascript to a subset of the pages html. Not looked but chances are that Amazon. BBC webpages etc were not using simple HTML but were using php, ruby on rails, asp etc back then to dynamically generate relevant html content and will have been unaffected.

2. when did Wayback discover this 'subset error'

Again, I have no idea, I simply found info re Wayback Machine/archive.org that explains the issue that they have had/may still be having regarding inserting/appended javascript into html pages which contain pdfs, flash etc and their acknowledgement that such a process is not the most efficient and issues regarding timestamps and rendering of data can occur and why they deprecate and would ideally wish to discontinue utilising said processes. Many webpages will have been archived correctly and as much as a great many webpages may have been affected by a similar timestamp subset issue, many millions more will be perfectly fine.

3. over what period did this subset error persist?

I have no idea. It would seem that it has been an ongoing issue since implementation.

@ Syn I am really grateful for the answers, thank you.

But in my attempts to understand the truth in all this, the comments I've bolded and underlined in red in your post cause me renewed difficulty in accepting that you have the right answer.

Especially so with your third and last answer, where you say that 'it' [by which I think you mean the 'subset error'] 'has been an ongoing issue since implementation'. That sounds like a very long time indeed.

You see, those that assert that this is 'an error' do not (unless I am mistaken) know what kind of error it was yet. Even Wayback's Christopher Butler is reported (via Lizzy Taylor) to merely say it was some sort of subset error.

Unless someone can tell me WHAT kind of error this was, HOW OFTEN it has occurred, HOW LONG it has lasted for, WHETHER it is a susbet error or a coding error or what, and WHETHER it has been corrected, it could be for all I know that Wayback Machine has millions or zillions of errors and is therefore worthless.

And again, if someone says: "It's an error, can't you see, it's obvious", or "It's an error, just accept it", or "It's some sort of system error, just forget about it now and go away", that isn't nearly enough to persuade me that 'it's just a glitch'

by XXXXXXXX 02.07.15 20:42

Thanks for your replies Tony.

by rustyjames 02.07.15 20:51

Trying to catch up with this thread having been away at Glastonbury completely out of touch for 6 days.

I've previously posted how to search the indexes using the Internet Archive's API into their CDX.

I've done a search on all [You must be registered and logged in to see this link.] pages indexed for the whole of 2007. If you want to do the same the link is below:
[You must be registered and logged in to see this link.]

I've loaded the results into excel and used pivot tables to analyse it and get some stats.

There are 13130 entries in the index for 2007.

Pages were indexed from ceop on 253 separate days during 2007. That is an average of 51.89 per day.

However there are 3786 entries for 30th April. Excluding those that is an average of 37.07 per day for the remaining days.

Now the interesting thing is breaking it down to the second in the index.

In that case there are 8732 unique timestamps over the 253 days.

Excluding the 30th April, the maximum number of indexed pages per second is 3, and an average of 1.07.

Yet on the 30th April every single one of the 3786 entries are timestamped in the same second - i.e. 20070430115803

That makes no sense whatsoever, so I'm now fairly certain there is at least an indexing problem, and as I've said a number of times before, what we really need are the original ".arc" files. I've looked for alternative sources but there is nothing in the public domain.

There is a possible alternative source for indexes here - [You must be registered and logged in to see this link.], but the 2007 data is a 26Gb download of tens of thousands of unsorted files.

by **Tony Bennett** 02.07.15 20:54

rustyjames wrote:Yet on the 30th April every single one of the 3786 entries are timestamped in the same second - i.e. 20070430115803

That makes no sense whatsoever, so I'm now fairly certain there is at least an indexing problem, and as I've said a number of times before, what we really need are the original ".arc" files. I've looked for alternative sources but there is nothing in the public domain.

There is a possible alternative source for indexes here - [You must be registered and logged in to see this link.], but the 2007 data is a 26Gb download of tens of thousands of unsorted files.

Could that be classed as either...

'a subset error'

or

'a coding error'?

TIA

by HKP 02.07.15 21:05

@Rustyjames. Ha, ha I've just spent ages looking at a download that had almost 9000 URLs over the life of ceop!!! The link came from the Dr Martin only in America blog site which I think everyone else was using too. With that I too was using pivots etc. but will now need to look at what you've got here (not doubting in any way just for my own benefit) I'll keep a watch out for your further analysis thanks.

by rustyjames 02.07.15 21:09

HKP wrote:@Rustyjames. Ha, ha I've just spent ages looking at a download that had almost 9000 URLs over the life of ceop!!! The link came from the Dr Martin only in America blog site which I think everyone else was using too. With that I too was using pivots etc. but will now need to look at what you've got here (not doubting in any way just for my own benefit) I'll keep a watch out for your further analysis thanks.

Can you give me a link to the Dr Martin link. Searched and can't spot it.

The link in my previous post is easy enough to save as a text file and import into excel as a space delimited file.

Edited to add: Could someone also point me to the original reference to a "subset error"?

by HKP 02.07.15 21:11

@siobahn
Just to prove a point I snapshot the 10 &12 May captures at this link.
http ://i18. servimg . com/u/f18/16/26/76/90/screen10 .png
(take out spaces, guests can't link)

by HKP 02.07.15 21:21

@Rustyjames.
Can't do links, go to comments section fourth one down

by rustyjames 02.07.15 21:27

HKP wrote:@Rustyjames.
Can't do links, go to comments section fourth one down

OK got it - not as useful as it just has the date rather than a full timestamp.

by HKP 02.07.15 21:42

@Rustyjames. Agree with the timestamp however taking the whole of 30/04 is enough to see error. I'll try loading your link into excel also. From my analysis you can see that leading up to the 30th everything was OK, there is then a 10 day gap to 10/05 and everything seems basically OK for the rest of May.

by rustyjames 02.07.15 22:20

Syn wrote:Problem with subsets says Chris Butler....

.... that brings us right back to the beginning where I posted this on 17th June when this merry go round first started.

I have now underlined the relevant part

[You must be registered and logged in to see this link.]

@Tony

There seems to be confusion with all the references to "subsets".

As far as I can tell the first mention of "subset" is something HiDeHo heard but didn't really understand in her phone conversation with Chris Butler. He could have been talking about a subset of anything - e.g. a subset of indexes are wrong etc, etc.

Syn then found the snippet above from a Wayback user manual here - [You must be registered and logged in to see this link.]

I don't believe that reference to "subset" has anything to do with what Chris Butler may have referred to.

The manual page was last updated in 2011 and was already talking about deprecating the JavaScript method in favour of server side re-writing of URLs, and from the source of pages I've seen that's what they have done. This also only refers to the "replay" of the page - it has nothing to do with the original capture of the page's data.

However for completion I'll try and explain what that highlighted section is referring to as simply as I can because I know not everyone follows the technical discussions.

A web page uses a language called HTML to tell your browser how it would like information displayed. As well as content in the HTML there can be links to things like images contained in separate files. Either in the HTML, or imported from separate files, there can also be some small bits of code written in something called JavaScript that runs in your browser to do more interesting and dynamic things like modifying the page, performing actions if you move your mouse over something, fetching data without having to reload the page etc, etc.

The problem for Wayback is when it "replays", i.e. recreates the page for you to view, the page contains links to other content. If it just presented the page as it originally was then those links would try and fetch the info from the real page rather than their archived copy of information.

Therefore they have to change all the links to point back into the Wayback machine. From the description above it seems at one point they did some of that using JavaScript in the browser - that means the page would have been downloaded with its original links, but the code they added would tell your browser to change the links before it finished displaying the page.

As far as I can tell that is no longer the case - all the links appear to be changed on their server before the page is even sent to you.

The references to Flash and PDF are there because those and other similar content can also contain links out to external content, however manipulating them are far harder to achieve.

Hope that helps.

by rustyjames 02.07.15 22:23

@hkp - from my link you'll see the index contains stuff on 1st, 2nd and 3rd May etc.

by **PeterMac** 02.07.15 22:29

HKP wrote:@Rustyjames. Agree with the timestamp however taking the whole of 30/04 is enough to see error. I'll try loading your link into excel also. From my analysis you can see that leading up to the 30th everything was OK, there is then a 10 day gap to 10/05 and everything seems basically OK for the rest of May.

I do not pretend to understand this,
BUT
Isn't it EXTRAORDINARY that
The dogs are wrong, but just ONCE - out of 200 times, and in 14 different places.
The PJ and all the successive Senior Investigating Officers and the Public Prosecutor et al. are wrong, but just ONCE - out of all the investigations they have ever done
The Wayback Machine is wrong - but just ONCE out of many years of operation and UNIQUELY in the McCanns' case.

What a frightening "coincidence", Kate,
":As a lawyer once said to me, apropos another matter, ‘One coincidence, two coincidences – maybe they’re still coincidences. Any more than that and it stops being coincidence.’

from 'madeleine' by Kate McCann. [Lower case 'madeleine' is correct. She did not even give her deceased daughter the honour of having her name rendered properly]

by HKP 02.07.15 22:35

@Rustyjames
That's interesting there was a gap in the downloaded data, I'll look forward to looking at your stuff tomorrow, since my laptop packed in I got an iPad and android tablet, hopeless for this sort of thing big grin

by siobhan3443 02.07.15 22:38

suddenly the accurate secret files are no longer accurate

interesting

by **Tony Bennett** 02.07.15 22:49

rustyjames wrote:
Syn wrote:Problem with subsets says Chris Butler....

.... that brings us right back to the beginning where I posted this on 17th June when this merry go round first started.

I have now underlined the relevant part

[You must be registered and logged in to see this link.]

@Tony

There seems to be confusion with all the references to "subsets".

As far as I can tell the first mention of "subset" is something HiDeHo heard but didn't really understand in her phone conversation with Chris Butler. He could have been talking about a subset of anything - e.g. a subset of indexes are wrong etc, etc.

Syn then found the snippet above from a Wayback user manual here - [You must be registered and logged in to see this link.]

I don't believe that reference to "subset" has anything to do with what Chris Butler may have referred to.

The manual page was last updated in 2011 and was already talking about deprecating the JavaScript method in favour of server side re-writing of URLs, and from the source of pages I've seen that's what they have done. This also only refers to the "replay" of the page - it has nothing to do with the original capture of the page's data.

However for completion I'll try and explain what that highlighted section is referring to as simply as I can because I know not everyone follows the technical discussions.

A web page uses a language called HTML to tell your browser how it would like information displayed. As well as content in the HTML there can be links to things like images contained in separate files. Either in the HTML, or imported from separate files, there can also be some small bits of code written in something called JavaScript that runs in your browser to do more interesting and dynamic things like modifying the page, performing actions if you move your mouse over something, fetching data without having to reload the page etc, etc.

The problem for Wayback is when it "replays", i.e. recreates the page for you to view, the page contains links to other content. If it just presented the page as it originally was then those links would try and fetch the info from the real page rather than their archived copy of information.

Therefore they have to change all the links to point back into the Wayback machine. From the description above it seems at one point they did some of that using JavaScript in the browser - that means the page would have been downloaded with its original links, but the code they added would tell your browser to change the links before it finished displaying the page.

As far as I can tell that is no longer the case - all the links appear to be changed on their server before the page is even sent to you.

The references to Flash and PDF are there because those and other similar content can also contain links out to external content, however manipulating them are far harder to achieve.

Hope that helps.

Anything like this which breaks down the technical jargon in a way that non-tecchies can understand is very helpful to me - and I'm sure to many others - thank you very much.

You wrote: "A web page uses a language called HTML to tell your browser how it would like information displayed. As well as content in the HTML there can be links to things like images contained in separate files".

It's the case of folk like Dr Martin Roberts, 'whodunit', 'HKP' and 'Resistor' and others, isn't it, that BOTH a page AND an image were captured at 11.58.03 seconds on 30 April?

The case espoused by e.g. HideHo, Nuala, Syn and Blue Bag on here amounts to this, doesn't it? - 'There are thousands of alleged 'captures' of data all pointing to 11.58.03 secs on 30 April on the CEOP site - and that is so blatantly obviously wrong that any recorded time-stamp of 11.58.03 seconds on the CEOP page is utterly meaningless, a mistake, and should be disregarded.

That still means I don't know if ANYTHING at all happened with regard to the Wayback Machine and CEOP on 30 April 2007 at 11.58.03 secs

Or have I misunderstood it?

by HKP 02.07.15 22:58

@Siobhan

Interestingly we were blindly following the recommended route by WBM:-
"You can see a listing of the dates of the specific URL by replacing the date code with an asterisk (*), ie: http: // web .archive. org/*/www .yoursite. com"

Rustyjames comes along with a better methodology and it gets looked at, it's how issue resolvement works

Anything to add which is not just a sarcastic comment.

by rustyjames 02.07.15 23:03

Yes in the index there are 3786 entries "captured" in that 1 second which makes no sense. A link to just that day is below - this also includes the news articles in the future in the list.

[You must be registered and logged in to see this link.]

However an index is only a processed view of what are in the .arc capture files, and we are also looking at something that could have been modified in their attempts to correct it, so it's frustratingly difficult trying to derive from it.

by suzysu 02.07.15 23:05

Tony Bennett wrote:
rustyjames wrote:
Syn wrote:Problem with subsets says Chris Butler....

.... that brings us right back to the beginning where I posted this on 17th June when this merry go round first started.

I have now underlined the relevant part

[You must be registered and logged in to see this link.]

@Tony

There seems to be confusion with all the references to "subsets".

As far as I can tell the first mention of "subset" is something HiDeHo heard but didn't really understand in her phone conversation with Chris Butler. He could have been talking about a subset of anything - e.g. a subset of indexes are wrong etc, etc.

Syn then found the snippet above from a Wayback user manual here - [You must be registered and logged in to see this link.]

I don't believe that reference to "subset" has anything to do with what Chris Butler may have referred to.

The manual page was last updated in 2011 and was already talking about deprecating the JavaScript method in favour of server side re-writing of URLs, and from the source of pages I've seen that's what they have done. This also only refers to the "replay" of the page - it has nothing to do with the original capture of the page's data.

However for completion I'll try and explain what that highlighted section is referring to as simply as I can because I know not everyone follows the technical discussions.

A web page uses a language called HTML to tell your browser how it would like information displayed. As well as content in the HTML there can be links to things like images contained in separate files. Either in the HTML, or imported from separate files, there can also be some small bits of code written in something called JavaScript that runs in your browser to do more interesting and dynamic things like modifying the page, performing actions if you move your mouse over something, fetching data without having to reload the page etc, etc.

The problem for Wayback is when it "replays", i.e. recreates the page for you to view, the page contains links to other content. If it just presented the page as it originally was then those links would try and fetch the info from the real page rather than their archived copy of information.

Therefore they have to change all the links to point back into the Wayback machine. From the description above it seems at one point they did some of that using JavaScript in the browser - that means the page would have been downloaded with its original links, but the code they added would tell your browser to change the links before it finished displaying the page.

As far as I can tell that is no longer the case - all the links appear to be changed on their server before the page is even sent to you.

The references to Flash and PDF are there because those and other similar content can also contain links out to external content, however manipulating them are far harder to achieve.

Hope that helps.
Anything like this which breaks down the technical jargon in a way that non-tecchies can understand is very helpful to me - and I'm sure to many others - thank you very much.

You wrote: "A web page uses a language called HTML to tell your browser how it would like information displayed. As well as content in the HTML there can be links to things like images contained in separate files".

It's the case of folk like Dr Martin Roberts, 'whodunit', 'HKP' and 'Resistor' and others, isn't it, that BOTH a page AND an image were captured at 11.58.03 seconds on 30 April?

The case espoused by e.g. HideHo, Nuala, Syn and Blue Bag on here amounts to this, doesn't it? - 'There are thousands of alleged 'captures' of data all pointing to 11.58.03 secs on 30 April on the CEOP site - and that is so blatantly obviously wrong that any recorded time-stamp of 11.58.03 seconds on the CEOP page is utterly meaningless, a mistake, and should be disregarded.

That still means I don't know if ANYTHING at all happened with regard to the Wayback Machine and CEOP on 30 April 2007 at 11.58.03 secs

Or have I misunderstood it?

++++++++++++

Thank you, Tony. A very clear summary!

by **Tony Bennett** 02.07.15 23:06

The poll I set up on the other Wayback thread is of mild interest.

The results so far in brief:

35 have voted, of whom

16 say it's way to complex, leaving 19 who give an opinion.

Of those 19:

3 say 'it's a glitch/error' and

15 say its a genuine capture of a McCann.html page on the CEOP website on 30 April,

with 1 who says there must be some other explanation.

SUMMARY: 'Syn' and 'Nuala' have some work to do in shifting forum opinion on this issue

by rustyjames 02.07.15 23:09

HKP wrote:
Rustyjames comes along with a better methodology and it gets looked at, it's how issue resolvement works

Actually I was quoting the method nearly two weeks ago :-) - [You must be registered and logged in to see this link.]

Oh and if you look here - [You must be registered and logged in to see this link.] there is the following:

Wayback CDX Server API

The CDX Server is another API which allows for complex querying, filtering and analysis of Wayback capture data. If you are looking for more in depth information about Wayback machine data, please take a look at the CDX server API.

by HKP 02.07.15 23:17

@Rustyjames
The rest of us were just not clever enough to pick up on it. The Dr Roberts link is the way that the Wayback FAQ‘s tell you to query it so I (and others) blindly followed that. It will be interesting to compare the two to see what or if any correlation.

by HKP 02.07.15 23:22

@Rustyjames.
Yep I was right, the rest of us weren't clever enough to pick up on it (I'm only speaking for myself here not barring everybody with the same brush).

The Wayback doesn't show captures on all of the days that the data does, any idea why that would be?

by Nuala 02.07.15 23:35

@ Tony Bennett

SUMMARY: 'Syn' and 'Nuala' have some work to do in shifting forum opinion on this issue

As I think I've said before, the answer you get to a question might depend on the question you ask in the first place. In your poll you have this as an option and four people have voted for that option:

3. Wayback captured a 'mccann html' file that had already been set up as a 'dummy file', but with no content on it

The mccann.html file most certainly had content on it, so anyone voting for that option didn't understand that was the case, and that option should never have been in the poll.

If you skew the questions you skew the answers.

Not saying that was on purpose at all, just a lack of understanding (as you've admitted) of the technicalities.

Steve Marsden's WBM screenshot: The CEOP Home page for April 30, 2007 also refers to Missing Madeleine.