Rackspace is rightfully praised for its use and development of open source technologies like OpenStack. However, I have had some rather shocking experiences when contacting their technical support. What I have seen was in fact sufficently bad for me to feel the need to document it here as a warning for others. The bottom line is, all support agents I was in contact with had minimal expertise (which is disappointing, but not that unusual), but (and this is the truly shocking aspect of the matter), did not hesitate to blindly make up stuff to conceal their ignorance. In other words, my advice to every Rackspace customer is to never take anything from the Rackspace support at face value -- if you cannot independently confirm it, it's probably made up.
That said, here's the best of my interactions.
Chunked Uploads
Please wait while we find an agent to assist you...
Welcome to The Rackspace Cloud. My name is John M., may I have your name and email address in case we get disconnected?
Customer: Hi, I'm Nikolaus Nikolaus@rath.org
Customer: Does your cloud hosting service support chunked object uploads?
John M.: Hi Nikolaus!
John M.: Yes
Customer: And what kind of consistency do you provide?
John M.: WHy are you looking to upload chunked objects?
Customer: To stream data without local caching
John M.: ok great!
John M.: Cloud files is a storage offering, primarily used for content delivery. we have several customers that use it for the delivery of media files (just an example) since it has the ability to sync with Akamai's CDN. the advantage to you is that files delivered via the CDN will be quick, regardless of your end user's geographic location. this is all because the CDN has nodes scattered throughout the world.
Customer: That looks suspiciously like a rather inappropriate standard text block.
Thank you for contacting the Rackspace Cloud.
Your session has ended. You may now close this window.
Yes, this chat sesson was not closed by me but by John M.
Data Durability
Welcome to Rackspace. How can I assist you?
You have been connected to Monica.
Monica: Welcome to Rackspace, may I help you with a hosting solution today?
Customer: Hi
Customer: I have a question about Cloud Files
Customer: What happens if one of my storage object gets lost by the server?
Monica: Sure :)
Customer: Will I get notified? Will the object name still be returned by list requests?
Customer: What will be the error code when I try to retrieve the object?
Customer: I find the Cloud Files API Developer guide a bit lacking in detail...
Monica: We offer persistent storage which is actually stored on a physical hard drive.. so you will not lose your information if it happens to go down
Customer: Yeah, but what if the hard drive crashes?
Customer: S3 offers two classes of durability guarantees, do you have something similar?
Monica: It is all RAID 10
Monica: so it is replicated 3 times
Customer: So what average durability do you guarantee?
Customer: And what happens when an object is lost?
The agent is sending you to http://www.rackspace.com/cloud/legal/sla/.
Monica: This is our Cloud files SLA
Customer: Yes, but it does not say anything about durability, just availability
Monica: Unfortunately we do not have anything on our site about the durability
Customer: So you do not make any guarantees?
Monica: Because we offer persistent storage on RAID 10 servers, objects should not be lost
Customer: Can you nevertheless tell me how the system will handle requests for a lost or damaged object?
Customer: [ I'm pretty sure that Amazon knows about RAID as well, yet they guarantee either 99.99% or 99.999999999% durability ]
Monica: The only way you were to lose an object is if you were to delete it
Monica: Our Cloud files is our most secure data storage medium
Customer: There is no such thing as absolute security.
Monica: We also guarantee 99.999999% on the durability
Customer: Could you point me to the paragraph where that is specified?
Monica: What is it that you are looking to store in our Cloud files?
Customer: I don't store data myself, I'm providing an online storage file system, and I'm interested in adding a CloudFiles backend.
The agent is sending you to http://www.rackspace.com/cloud/cloud_hosting_products/files/support/.
The agent is sending you to http://www.rackspace.com/cloud/cloud_hosting_products/files/compare/.
Customer: I don't think this conversation is going anywhere. Nevertheless, I appreciate your efforts. I know you have to work with the information you've got.
What bothers me here is not that Rackspace doesn't offer durability guarantees, and Monica does not to know how their system is going to react to a missing object. The problem is that she completely ignores a question, maintains that data is absolute secure because it's stored on a RAID-10 array, and points me to several unrelated webpages, before finally pulling a number out of thin-air that can not be found anywhere else.
Either Rackspace is dangerously overconfident in the reliability of its systems, and surprisingly unfamiliar with the behaviour of its own software, or they are deliberately refusing to give all the information needed to build reliable software on top of their storage service and misleading their customers about the security und durability of their data.
In either case, it's not a pretty picture. Their customer support obviously goes as far as giving wrong pieces of information without even flinching. If they now come up with some information on how the system would respond to a missing object after all, how do you know this is accurate and not just made up as well? You can't test it.
HTTP Headers
11:19:58 PM [Tim R] Hello! Thank you for contacting Rackspace Chat support! My name is Tim R, How can I help you today?
11:20:03 PM [Nikolaus] Hi Tim
11:20:12 PM [Nikolaus] I'm using the REST API to access CloudFiles
11:20:38 PM [Nikolaus] I was wondering if it's possible that CloudFiles is changing the case of metadata keys
11:20:57 PM [Nikolaus] I have stored an object with a metadata key of "foo"
11:21:17 PM [Nikolaus] but when I retrieve it, the HTTP header says "X-Object-Meta-Foo: value"
11:21:19 PM [Nikolaus] Note the capital F
11:22:11 PM [Tim R] Give me a moment
11:24:21 PM [Tim R] What is the call that you are running?
11:24:33 PM [Nikolaus] What do you mean with "call"?
11:24:39 PM [Nikolaus] I'm sending a HTTP GET
11:24:46 PM [Tim R] The full command
11:25:07 PM [Nikolaus] one moment
11:28:44 PM [Nikolaus] GET /v1/MossoCloudFS_e0f1f364-b0c0-4062-ad38-ca87466ee99f/nikratio-test/?marker=&limit=5000&format=json&prefix=s3ql_test%2F-s3ql_test_1370143692 ({'content-length': '0', 'connection': 'keep-alive', 'X-Auth-Token': '93a4bdd7d83f4404862308bf407ca959'})
11:29:04 PM [Nikolaus] wops, wrong line
11:29:10 PM [Nikolaus] GET /v1/MossoCloudFS_e0f1f364-b0c0-4062-ad38-ca87466ee99f/nikratio-test/s3ql_test/-s3ql_test_1370143692s3ql_%3D/1 ({'content-length': '0', 'connection': 'keep-alive', 'X-Auth-Token': '93a4bdd7d83f4404862308bf407ca959'})
11:29:36 PM [Nikolaus] the stuff in {...} are the request headers
11:30:30 PM [Tim R] What is the query you are running to set the metadata keys?
11:30:46 PM [Nikolaus] PUT /v1/MossoCloudFS_e0f1f364-b0c0-4062-ad38-ca87466ee99f/nikratio-test/s3ql_test/-s3ql_test_1370143692s3ql%3D/_1 ({'Content-Type': 'application/octet-stream', 'Content-Length': 9, 'connection': 'keep-alive', 'X-Object-Meta-jimmy': 'jups@42', 'X-Auth-Token': '93a4bdd7d83f4404862308bf407ca959'})
11:31:00 PM [Nikolaus] in this example, jimmy becomes Jimmy
11:33:08 PM [Tim R] the keys all have to have Caps when separating naming schemes by a delimiter (-)
11:33:48 PM [Nikolaus] Could you point me to a spec with details?
11:34:02 PM [Nikolaus] I don't think that's from RFC 2616
11:35:59 PM [Tim R] Give me a few moments
11:36:15 PM [Nikolaus] http://docs.rackspace.com/files/api/v1/cf-devguide/content/Create_Update_Object-d1e1965.html just says "The object can be created with custom metadata through HTTP headers identified with the X-Object-Meta- prefix."
11:41:29 PM [Tim R] Basically, think of the "X-Object-Meta-Jimmy" as the key, and the value is "jups@42" The data is maintained the same way as arrays, but all you need to know: the thing to the left of the ":" is the key
11:41:48 PM [Nikolaus] aeh, yes, that much is pretty obvious :-)
11:42:05 PM [Tim R] I'm trying to find some documentation on this.
11:42:06 PM [Nikolaus] My question is: why is RackSpace changing the case of the keys?
11:42:12 PM [Nikolaus] and in what way?
11:42:54 PM [Nikolaus] This silent mangling that appears to be done is making me a bit nervous
11:46:00 PM [Tim R] HTTP 1/1 header standars say that headers that have naming schemes separated by a delimiter will have each segment capatalized
11:47:06 PM [Nikolaus] Could you be more precise? I can't find that in http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2
11:49:12 PM [Tim R] Give me a few minutes, I'm trying to find the specific documentation that states this
11:52:11 PM [Tim R] http://tools.ietf.org/html/rfc822
11:58:29 PM [Nikolaus] I don't see any specific prescriptions for the field name in there either...
12:01:16 AM [Tim R] So RFC822 is about messages, but it sets the standards for headers, only 2 headers are non-caps after the delimination, Reply-cc and Reply-bcc to be exact. and they have since made "X-" headers obsolete in draft http://tools.ietf.org/html/draft-saintandre-xdash-00, the draft was submitted in 2011, and has not been converted into RFC yet, though
12:02:42 AM [Nikolaus] Could you be more precise? I don't see anything about delimination of field names in RFC822
12:03:08 AM [Nikolaus] http://tools.ietf.org/html/rfc822#section-3.2
12:03:40 AM [Nikolaus] All it says is that a field name is 1*
12:05:31 AM [Tim R] I'm deeply sorry, but the RFC is less than clear, but it did set the standard, and that only cc and bcc are allowed to be non-caps after a delimination
12:05:32 AM [Nikolaus] RFC 2616 adds that "Field names are case-insensitive", but there is nothing about delimiters, or specific capitalization rules
12:06:05 AM [Nikolaus] According to the RFCs, there is no such thing as a delimination of field headers.
12:06:10 AM [Nikolaus] Where did you get this concept from?
12:06:20 AM [Nikolaus] This isn't unclear, it simply does not exist.
12:13:27 AM [Nikolaus] Tim?
12:14:01 AM [Tim R] Give me a moment
12:20:55 AM [Tim R] "X-Special-action: This is a sample of user-defined field-names. There could also be a field-name "Special-action", but its name might later be preempted" basically, whenever you have a lower-case after the "-" it can be preempted, due to RFC822 A.3.3.
12:22:32 AM [Nikolaus] Any X- header may be preempted, this has nothing to do with the existance of a dash.
12:23:26 AM [Nikolaus] And preemption of a header is unrelated to its case anyway
12:23:35 AM [Nikolaus] I don't think we're getting anywhere here
12:23:50 AM [Tim R] Alright, I would recommend submitting a ticket about this
12:23:54 AM [Nikolaus] Could you maybe just escalate this issue?
12:24:29 AM [Nikolaus] Ok, thank you for your efforts.
12:24:32 AM [Tim R] I was actually getting this information from my escalation admin, so he said it is best to submit a ticket for this
12:24:34 AM [Tim R] You're welcome
(I am aware that RFC2616 says that HTTP headers are case-insensitive. However, I believe it is nevertheless reasonable to ask why Rackspace is not preserving the case, especially since we are taking about user-defined metadata keys).
Obviously Tim has no idea what he is talking about, but instead of just admitting that, he quotes random standards and adds a couple of extra "facts" to make them seem relevant to the question.