Google picking up documents from web folder

[rupertvz]

Hi Guys,

We have a NT web server with SSL, site security, and users must be logged on to access the web server.
Google somehow managed to access documents from the "web" folder, and published links to these online.

Is there a way to block / stop this, as it is a serious security concern for the customer.

[rupertvz]

Hi Guys,

Google suggests to add a META tag to the HTTP header of the page which calls the PDF's

"X-Robots-Tag: noindex"

Is the correct place to add this META tag under the xHTML tab of the NWB
After heading ?

[Bruce]

Hi Rupert,

>> Is there a way to block / stop this, as it is a serious security concern for the customer.

I feel like you're missing the point here. If have the files publically available in the web folder is a security concern, then you should fix that. Perhaps by describing why private files are in a public folder to begin with?

Turning off Google indexing doesn't stop people fetching the files. If google found them, so can everyone else. And malicious users don't respect the settings in your robots.txt file.

Cheers
Bruce

[rupertvz]

Hi Bruce,

Thank you, it is files which the users upload as supporting documents which we are storing in a documents sub-folder contained in the web folder.
When I try to browse those files, I am not able to.  Thus a malicious user would need to know the full file-name of the file to get to it?  Or is there another way?

I maybe wrong, but would imagine that Google indexed the file whilst it was being opened / viewed by a user in Chrome.

Is there a way to project files in the web folder?
Or if the web folder is always publicly accessible, should I put the files outside the web folder, and change my user-procedure that it opens from local disk and not via URL?

[Bruce]

>> Thank you, it is files which the users upload as supporting documents which we are storing in a documents sub-folder contained in the web folder.
>> When I try to browse those files, I am not able to.  Thus a malicious user would need to know the full file-name of the file to get to it?  Or is there another way?

They would need to know, or deduce, the name. Or guess. Unless your names are generated randomly it's probably easy to guess a name or two, especially given that the server allows for an unlimited number of guesses. Treating the name as a secret, and then broadcasting that secret to all legitimate users in some form or another, does not make it a terribly good secret.

>> I maybe wrong, but would imagine that Google indexed the file whilst it was being opened / viewed by a user in Chrome.

I very much doubt that they got it that way. I suspect they got it by crawling your site. Either way though, you've shown that it's not a terribly well-kept secret.

>> Is there a way to project files in the web folder?

not by default, no.

>> Or if the web folder is always publicly accessible, should I put the files outside the web folder, and change my user-procedure that it opens from local disk and not via URL?

Opening from local disk is not an option. HTTP pages have a "single source" policy - in other words web pages cannot link to pages to the disk.

However, since the files belong to a user, and should only be available to that user, it's pretty easy to limit access to the file to that user. For example, just moving it to the \web\loggedin folder would require the user to be logged in to access the file. You can also serve the document from *outside* the web folder - and thus implementing further checks on who owns the file, and so on.