Claim by 'Stevo' - "CEOP show Maddie is missing on 30th April 2007"

aiyoyo wrote:@Richard Hall

Is this useful?
[You must be registered and logged in to see this link.]

If I can use this link, Aiyoyo, to explain my point... this 12 th Oct shows dates of 9th Oct  to 6th  Feb.

Why should that be useful?

It has an archive date of

FILE ARCHIVED ON 5:18:50 Oct 12, 2007 AND RETRIEVED FROM THE

Seems fine to me

rustyjames pointed out that the WBM reconstructs (replays on retrieval) pages rather than snapshots them then will it reconstruct within the folder (I would have thought so). If it doesn't find the exact data it uses the closest to it (so it says in the FAQs) so.... Is the McCann.htm the master in 20070430115803 and WBM has constructed around the files in the folder? Any thoughts

Like I said earlier, I don't know exactly how their crawler works because I have not got the source code.  Even with the source code it would take days to figure out how somebody else's software functions.  I was incorrect to refer to the string 20070430115803 as a "folder", but when it is expressed as [You must be registered and logged in to see this link.]  it looks like a folder.  But to use a more generic language, when a file is archived, it becomes associated with a string of numbers which is derived from the real time clock the first time the file is identified.  This string of digits remains associated with that file always.  In the case of mccann.html, the string 20070430115803 is associated with it.  People have been trying to work out exactly how the files are related to these indexes, and how their system functions.   Every file that has been queried as far as I have seen returns an archive date ON or AFTER the date the file came into existence. 

Some of the screenshots showing the date of October news items on a 30 Apr archived page are curious.  If the news items are not dynamic, then this is strange.  Am I right in saying that I cannot now produce this screenshot myself because their system has been changed?  If so, two things come to mind.  Are the screenshots genuine? - I would imagine so because we have two independent posters - secondly are we saying that in the history of WBM the only time it screws up is around the exact date of one of the most controversial news stories of all time?  Also that it screwed up for a very short window of time, then started functioning perfectly again up till now?

sallypelt wrote:[You must be registered and logged in to see this image.]  Mimi Today at 9:13 pm


At about 2:00 mins the founder of Waybackmachine says they respond to people wanting to take stuff off.

[You must be registered and logged in to see this link.]

Mimi posted this on the other forum a few minutes ago. I hope she doesn't mind me posting it here. Apologies to Mimi for hijacking her post

So he will take pages off at request and then goes on the emphasize the importance of keeping past web pages. It seems they already had a huge credibility problem in 2011.

Just some info. on how WBM works and lots of info. on it.

Where is the rest of the archived site? Why am I getting broken or gray images on a site?
Broken images (when there is a small red "x" where the image should be) occur when the images are not available on our servers. Usually this means that we did not archive them. Gray images are the result of robots.txt exclusions. The site in question may have blocked robot access to their images directory.
You can tell if the link you are looking for is in the Wayback Machine by entering the url into the Wayback Machine search box at archive.org (http://www.archive.org/web/web.php ). Whatever archives we have are viewable in the Wayback Machine.
The archived webpages are meant to be a "snap shot" of past Internet sites. Please note that while we try to archive an entire site, this is not always possible. That is why some images or links might be missing. Additionally some sites do not archive well and we cannot fix that. There is a list of common problems that make a site difficult to archive: [You must be registered and logged in to see this link.]
If you see a box with a red X or a broken image icon that means that we unfortunately do not have the images. Files over 10MB are not archived in this "snap shot" of the website.
The best way to see all the files we have archived of the site is: [You must be registered and logged in to see this link.]
Please note that there is a 6 - 14 month lag time between the date a site is crawled and the date it appears in the Wayback Machine.

Can I link to old pages on the Wayback Machine?
Yes! The Wayback Machine is built so that it can be used and referenced. If you find an archived page that you would like to reference on your Web page or in an article, you can copy the URL. You can even use fuzzy URL matching and date specification... but that's a bit more advanced.
Why isn't the site I'm looking for in the archive?
Some sites may not be included because the automated crawlers were unaware of their existence at the time of the crawl. It's also possible that some sites were not archived because they were password protected, blocked by robots.txt, or otherwise inaccessible to our automated systems. Siteowners might have also requested that their sites be excluded from the Wayback Machine. When this has occurred, you will see a "blocked site error" message. When a site is excluded because of robots.txt you will see a "robots.txt query exclusion error" message.
What does it mean when a site's archive data has been "updated"?


When our automated systems crawl the web every few months or so, we find that only about 50% of all pages on the web have changed from our previous visit. This means that much of the content in our archive is duplicate material. If you don't see ""*"" next to an archived document, then the content on the archived page is identical to the previously archived copy
How do you archive dynamic pages?
There are many different kinds of dynamic pages, some of which are easily stored in an archive and some of which fall apart completely. When a dynamic page renders standard html, the archive works beautifully. When a dynamic page contains forms, JavaScript, or other elements that require interaction with the originating host, the archive will not contain the original site's functionality.
Some sites are not available because of robots.txt or other exclusions. What does that mean?
The Internet Archive follows the Oakland Archive Policy for Managing Removal Requests And Preserving Archival Integrity
The Standard for Robot Exclusion (SRE) is a means by which web site owners can instruct automated systems not to crawl their sites. Web site owners can specify files or directories that are disallowed from a crawl, and they can even create specific rules for different automated crawlers. All of this information is contained in a file called robots.txt. While robots.txt has been adopted as the universal standard for robot exclusion, compliance with robots.txt is strictly voluntary. In fact most web sites do not have a robots.txt file, and many web crawlers are not programmed to obey the instructions anyway. However, Alexa Internet, the company that crawls the web for the Internet Archive, does respect robots.txt instructions, and even does so retroactively. If a web site owner decides he / she prefers not to have a web crawler visiting his / her files and sets up robots.txt on the site, the Alexa crawlers will stop visiting those files and will make unavailable all files previously gathered from that site. This means that sometimes, while using the Internet Archive Wayback Machine, you may find a site that is unavailable due to robots.txt (you will see a "robots.txt query exclusion error" message). Sometimes a web site owner will contact us directly and ask us to stop crawling or archiving a site, and we endeavor to comply with these requests. When you come accross a "blocked site error" message, that means that a siteowner has made such a request and it has been honored.
Currently there is no way to exclude only a portion of a site, or to exclude archiving a site for a particular time period only.

When a URL has been excluded at direct owner request from being archived, that exclusion is retroactive and permanent.
Why are some sites harder to archive than others?
If you look at our collection of archived sites, you will find some broken pages, missing graphics, and some sites that aren't archived at all. Here are some things that make it difficult to archive a web site:

• Robots.txt -- We respect robot exclusion headers.
  
• Javascript -- Javascript elements are often hard to archive, but especially if they generate links without having the full name in the page. Plus, if javascript needs to contact the originating server in order to work, it will fail when archived.
  
• Server side image maps -- Like any functionality on the web, if it needs to contact the originating server in order to work, it will fail when archived.
  
• Unknown sites -- The archive contains crawls of the Web completed by Alexa Internet. If Alexa doesn't know about your site, it won't be archived. Use the Alexa Toolbar (available at [You must be registered and logged in to see this link.] and it will know about your page. Or you can visit Alexa's Archive Your Site page at [You must be registered and logged in to see this link.]
  
• Orphan pages -- If there are no links to your pages, the robot won't find it (the robots don't enter queries in search boxes.)
  

As a general rule of thumb, simple html is the easiest to archive.
What type of machinery is used in this Internet Archive?
A few highlights from the Petabox storage system:
As of December 1, 2014 -
Density: 1.4 PetaBytes / rack
Power consumption: 3 KW / PetaByte
No Air Conditioning, instead use excess heat to help heat the building.
Raw Numbers as of August 2014:
• 4 data centers, 550 nodes, 20,000 spinning disks
• Wayback Machine: 9.6 PetaBytes
• Books/Music/Video Collections: 9.8 PetaBytes
• Unique data: 20 PetaBytes
• Total used storage: 50 PetaBytes

For more information go to [You must be registered and logged in to see this link.]
Do you collect all the sites on the Web?
No, we collect only publicly accessible Web pages. We do not archive pages that require a password to access, pages tagged for "robot exclusion" by their owners, pages that are only accessible when a person types into and sends a form, or pages on secure servers. If a site owner properly requests removal of a Web site through [You must be registered and logged in to see this link.] we will exclude that site from the Wayback Machine.
[You must be registered and logged in to see this link.]

'Resistor' in the other place is still, well, resisting...

=====================================

Resistor Today at 1:00 am


Err... I have just come home! Believe it or not, I do have a life outside this forum!

I really wish I could waltz in and give you all a sparkling explanation that explains it all to everyone's satisfaction, but I can't, sorry [You must be registered and logged in to see this image.] I have zero explanation for stuff being dated October in a file, ostensibly created in April, but if it's not as a result of some dynamic content, then yes. It is an anomaly and has to be explained.

We know that something was created on 30 April 2007 when the CEOP site was crawled, because pages were saved and an index (folder) created for it. I have no reason to doubt that date at all. 30 April 2007 at 11:58:03. For reasons I have already given (about 30 pages back now!) I trust server timers unless there is a very good reason for them to be "wrong".

And when they do go wrong, they go spectacularly wrong. It would be saying something like 1 January 1900 if it was really, really wrong.

So now the school of thought seems to be is that the site was again crawled sometime after late October 2007. In actual fact the first date after 27/10/07 is 6/2/08. Their Javascript that automatically appends confirms this. That seems an awfully long gap, looking at the other captures in the calendar, so is there a missing capture in the period from 12/10/07 to 6/2/08? If so, that capture wasn't saved in it's own folder during that period, it was diverted - by presumably some sort of software pointer error - to the April folder, where it went in with the April files that were already there.

(ETA - why April 30, and not Oct 12, the nearest one to it?)

If you try, in Windows, to move a file into a folder, and there is already a file there of the same name, it asks you what you want to do. If you choose to save the newer one, it overwrites the old one. I don't know what sort of server Wayback sits on, but I do have some experience of UNIX and APACHE and if you try to replace a file, it just overwrites it, with no warnings or dialogs. I can well imagine that the October-to-February version was saved to the April index and just overwrote the April version, but why it would do that in the first place, I have no idea.

So this now leaves us with when mccann.html was actually created and crawled. Was it there in April and went in the folder originally? Or was it only there in October-to-February and went in with the April stuff as the same time as the later homepage?

The bit that is gnawing at me is the lack of the second photograph. Madeleine_02.jpg, in the April version. It should have been there on the April page, but it wasn't, because the webpage showed a broken link. So if it was an October-to-February page was saved into the April index, the whole thing was not saved, because then we would not be minus a photo. Later versions have both photos. So clearly the saving process was not exactly the same in all cases.

Now a couple of nights ago, HKP very helpfully found some stuff in Wayback's own FAQ that tells us how the pages are replayed when they have only saved part of them. They try to reconstruct it as best they can from the next nearest version. For mccann.html that would have been May 13, which has both photos, but not the little flags that appear in even later versions. The May version tells us in the appended Javascript that the previous capture was April 30.

[You must be registered and logged in to see this image.]

If there wasn't a version of mccann.html in the folder on 30 April (as it was only added at some later point October-February) then how did a file created on 13 May manage to find it, to add it in as a previous version? Because the 13 May capture should have been the first one.

Sorry, it's late now and I am probably not expressing myself very clearly. I also need to be up early in the morning. I'll give this some more thought over the next couple of days, and email Wayback again, as so far they have not responded to any of my queries. I suspect it's the same for us all.

Portia wrote:Re: TB quoting Resistor: 

If that's the case, what would be the American Bar of Attorneys take on this matter?

They would be beside themselves with glee, wouldn't they, especially the ones who saw their clients incarcerated based on WBM evidence!

There are TWO types of dates in the WBM.

1) A WBM assigned archived time stamp... this looks sometimes broken to me.

2) Dates in the content of the actual archived web page itself... forum post date/time, newspaper article dates, latest news dates etc... These are self certifying.

It is the second category that would still be used in court because the first can be argued as unreliable.

Richard D. Hall wrote:Like I said I am happy to be shown there is a bug in the system, but I can't see one. 

The bug was pointed out on page 2 of this thread Richard.

There was absolutely an October 2007 page indexed as 30th April 2007 11:58:03

There are screenshots of it before the 30th April folder was removed.

Many people saw it.

It doesn't matter what the WBM machine is doing now... it's what it was doing in 2007 that matters.

whatsupdoc wrote:

macdonut wrote:This is a red herring I think guys.  If you look at the full ceop page as allegedly archived on 30th April:

[You must be registered and logged in to see this link.]

You'll see quite a number of news stories and links that are, in fact, dated in October 2007.

While I don't profess to understand how the web archive works, it clearly isn't accurate, at least on this occasion.

Agreed , macdonut. So we have two versions for the 30th April. The version I found did have references to October 2007 so at least one entry on 30th April was incorrect if not both.

I noticed the html code in Doug D post was a comment and the date could have been edited in and captured.

Also from the second page of this thread.

The link to the page is given and many people tried it to verify it.

Richard D. Hall wrote:Some of the screenshots showing the date of October news items on a 30 Apr archived page are curious.  If the news items are not dynamic, then this is strange.  

The archives with latest news items are copies of the dynamic content at that point in time. They become static in the archive. Not strange.

Am I right in saying that I cannot now produce this screenshot myself because their system has been changed?

Yes.. they removed the folder that held the 20150430 115803 entries and now you get redirects from the index.

 If so, two things come to mind.  Are the screenshots genuine? - I would imagine so because we have two independent posters

Yes for the same reason and the link was in the posts for people to check.

 - secondly are we saying that in the history of WBM the only time it screws up is around the exact date of one of the most controversial news stories of all time? 

We have no idea if it's the only time.
We care about it and make a fuss for obvious reasons, but have no idea of the scale of WBMs problem.

 Also that it screwed up for a very short window of time, then started functioning perfectly again up till now?

Again we have no idea.

Tony Bennett wrote:'Resistor' in the other place is still, well, resisting...

=====================================

Resistor Today at 1:00 am

We know that something was created on 30 April 2007 when the CEOP site was crawled, because pages were saved and an index (folder) created for it. I have no reason to doubt that date at all. 30 April 2007 at 11:58:03. For reasons I have already given (about 30 pages back now!) I trust server timers unless there is a very good reason for them to be "wrong".

False logic.

No we don't know that because something may have indexed into 20070430 115803 with a wrong date and time - trust me, computers do this because their stupid programmers don't think of everything for all circumstances :-) 

What we DO know is that an October page ended up in the April folder and that shows the WBM was screwed in this instance so all the pages with the same date and time index are probably screwed as well for the same reason.

@BB your 'all pages are probably screwed' statement is just an educated guess at this point in time is it not? (sorry can't properly quite as a guest)

HKP wrote:@BB your 'all pages are probably screwed' statement is just an educated guess at this point in time is it not? (sorry can't properly quite as a guest)

Well.. yes based on 34 years of dealing with crap like this every day.

@BB that should have read quote not quite, can't edit either

@BB wasn't questioning your knowledge or experience just that it was your 'expert opinion' rather than fact. Thanks

Advanced Wayback Machine Navigation and Troubleshooting

There are 3 kinds of Wayback Machine Queries

1.  URL Query: You can search Archive-It collections by URL from inside the application (under Access --> Wayback), or from a public collection page on [You must be registered and logged in to see this link.] (ex: [You must be registered and logged in to see this link.] When you enter an URL (ex:  [You must be registered and logged in to see this link.] into the Wayback Machine search page, your results will display as a list of dates on which the URL was archived.  For example:  [You must be registered and logged in to see this link.]
2.  URL Prefix Query:  This query will display all archived links for a given domain.  To search using this query method add a '*' (or wildcard) to the end of the URL query (ex:  [You must be registered and logged in to see this link.] the address for this query is then:  [You must be registered and logged in to see this link.] The total number of captured documents will be displayed at the top of the screen.  Please note this number reflects the total number of archived links, however only unique URLs will be displayed. For example you could have 1,000 links archived, but only be able to see 800 links listed. This is due to the fact that the same link has been captured multiple times. Next to each listed link you will see a number of versions; this refers to the number of different captures for each link.
3.  URL Date Query:  This is a search by specific date or date range.  This query relies on the 14 digit date code in the middle of each archived URL (yyyymmddhhmmss).  You can use a combination of dates and '*' to manipulate which capture dates you see in your results.  EX:
-http://wayback.archive-it.org/194/20070913204539/http://www.governor.state.nc.us/ displays [You must be registered and logged in to see this link.] as it looked on September 13, 2007 at 20:45:39 GMT
Here are some more examples:
-http://wayback.archive-it.org/194/2007*/http://www.governor.state.nc.us/ - displays all 2007 captures of [You must be registered and logged in to see this link.] In this manner you can adjust to view only results for any year of crawling (just adjust the year in the date code).
-http://wayback.archive-it.org/194/200712*/http://www.governor.state.nc.us/ - displays all dates [You must be registered and logged in to see this link.] captured in December 2007. You can limit even further to a specific date by adding to the date code.
You can switch back and forth from these queries at any time by changing the web address at the top of your browser window. 
URL and URL date query only show results for the exact URL you are looking up.  When you look up [You must be registered and logged in to see this link.] you are only seeing captures for that homepage.  However if you were viewing a page deep inside the site and you wanted to see what other dates that page was captured, just manually change the date code in the url to *.
[You must be registered and logged in to see this link.]

'Resistor' still maintains that something significant happened at CEOP on 30 April - but concedes that he can't prove it - and abandons hope (for the time being). From the other place:

++++++++++++++++++++++++++++++++++++++

Resistor wrote:

HKP. you have no idea how depressed this has all left me now. I still feel, really feel, that we have found something very significant indeed. And it is something I wish wasn't true, because it's evidence that something truly evil and horrific has happened, and a wee girl is dead. But I still remain convinced, because I know what I know and there's no way of unknowing it - and it is upsetting me very greatly.

And there is no way we can prove any of it, not without answers, sensible coherent definitive answers, from Wayback. ("A glitch" and "a mistake" are not definitive answers, BTW). We are never going to get those answers. It seems many people have asked them directly, all have been ignored, apart from Isabelle and Lizzie - and they only got answers because they asked before Wayback wised up to what was going on. And even then, the answers were in direct contradiction to one another, so it actually tells us nothing.

We won't get answers not because of some big CEOP or Government coverup, but because Wayback have to protect their reputation, or at least what's left of it. Because this has been such a hotly contested, divisive topic, in only a few days time, anyone who Googles something like "how accurate is Wayback" will be directed straight to one of these forums. Where the whole thing has been analysed to the nth degree and totally ripped apart by a lot of very knowledegable people. Wayback advertise themselves as an internet archive, but as the whole purpose of an archive is to preserve an accurate record, they are finished.

Nor do I think that Wayback were "leaned on". I think they might have been asked to remove the offending pages. They state in their own FAQ that they have no problem with that, and also that site owners can block their crawls with a robots file. I had a good look last night and I found quite a few, even pretty innocuous sites like the English FA.) So I'm not sure how this contributes to a "complete" Internet archive; that's a bit of a mixed message in their very own mission statements, right there.

So did mccann.html exist on 30 April 2007? Yes, I believe it did. Unfortunately, I cannot prove that it did. And neither can anyone else without a full explanation from Wayback, which we are not going to get.

Team McCann seem to have more lives than a bloody cat. What are the odds of this happening to this particular site?! I could just rip my own hair out with the sheer frustration and unfairness of it all. If I'm not seen for a couple of days, it's just that I am taking a break from this whole sorry mess, because I have been neglecting things that I should have been doing over the past couple of days and I have to get on with my own life.

If anyone ever does get any sort of response from Wayback, I'll be straight all over it like a bad rash, though.

But if the 30 April page is correct then why would wayback be worried about it? Why not just say "the page is correct, report it to the police, our reputation is intact"? Why would they need to find an explantion like a glitch is there wasn't one? If that page was there on 30 April then surely it's down to CEOP to be squirming.

I would tend to agree.  The screenshots from before they changed their system do not prove to me that mccann.html did not exist on 30 April 2007.  What they show is that a different page, which had the same archive date (30 April) probably contained data from October 2007.  People are inferring from this that mccann.html must also have contained data from October 2007.  But I don't see that it automatically follows.  It is down to WBM to explain why mccann.html had an archive date of 30 April 2007.  They have not as far as I can see explained in detail how this was the case and until they do, it is feasable that mccann.html was archived on that date.

I'm going to ask the same question here:-

Why does WBM now point you to the 13th May for the archive when Chris Butler in his second email claims the original archive should read 31st July and it's all an error?

I agree with Resistor's take on this very frustrating topic. It should be easily provable and the information very reliable IMO, and promptly backed up by one of their staff members. But all we have heard so far is crickets on it all. What is so difficult about a valid email explanation of how that could of happened from them? I also can't really see how a page existed for CEOP with an initial file date of 30/4/07 as found by WBM, and now it has been taken care of. Why that particular date if nothing existed on that date, that doesn't make any sense to me either, because you could have any random date in that case, so why the 30/4? You can't make something out of nothing on WBM from what i have read about it, it has to be able to retrieve the information for its database to appear there. I call BS.

HKP wrote:I'm going to ask the same question here:-

Why does WBM now point you to the 13th May for the archive when Chris Butler in his second email claims the original archive should read 31st July and it's all an error?

Who knows ???? Maybe he needs to explain.

@Joss, yes he does, why wouldn't they point to 31st July if this is the proper and original date. I'll answer myself cause it can't be as they are pointing to 13th of May ergo he doesn't know what he's talking about

HKP wrote:@Joss, yes he does, why wouldn't they point to 31st July if this is the proper and original date. I'll answer myself cause it can't be as they are pointing to 13th of May ergo he doesn't know what he's talking about

I agree. If CB stands by his email to IM then he sure is contradicting that email. And it makes him look inept if he has now changed his tune. So yes, an explanation is warranted IMO.