HTTP limitations? (DEF-3410) (SOLVED)

jonnyleeharris · August 16, 2018, 6:22pm

I am working on integrating Firebase into my game, which means creating a library for the REST API. I’ll gladly share it in the future when its done.

I’m having an issue with a request that I can get to work in Postman, but the identical request in Defold doesn’t seem to work.

I am making a PUT request to a URL, the payload seems to pass Firebase’s checks (good), but when it comes to authentication… not so much. To authenticate with the Firebase REST API you must append ?auth=KEY to your requests, where KEY is an authentication token generated via another REST authentication method.

The key is fine, yet Firebase complains that it cannot parse the key, yet when I make the identical PUT in Postman - it works just fine.

My initial thought was that Defold is doing some magic to the URL which is corrupting the key, because if I pass an invalid or malformed key to the REST path (in Postman) it gives me the same error.

https://FIREBASE/user_saves/X4Rj7Jht8EfLVJWgOLOW6Lg5hbs1/save_file.json?auth=A_VERY_VERY_VERY_LONG_KEY

Hopefully (edit: somebody!) has some clues on this one

jonnyleeharris · August 16, 2018, 6:37pm

I can happily share some keys/urls if anybody would like to verify I am not going out of my mind, only thing though is that the tokens expire after an hour. I am in London time zone and will probably be signing off for the day now

britzl · August 17, 2018, 5:48am

I can assist. I’m thinking that it has something to do with URL length. We have increased max length once in the past when I did some work on the AWS SDK where Gamelift URLs became really long.

jonnyleeharris · August 17, 2018, 9:24am

OK I’ll get some keys generated and URL to use, do you want to take it into DMs? Let me know.

Edit: Really messed up that reply…!

jonnyleeharris · August 17, 2018, 11:52am

In case we miss each other in real time @britzl - is there anything I can do on my end if the problem is related to URL length? I’m trying to see if I can send the data I need in headers or some other workaround - but this is very much dependant on the available API Firebase provide.

britzl · August 17, 2018, 3:09pm

What’s the length of a complete URL with all parameters added?

jonnyleeharris · August 17, 2018, 3:11pm

@britzl Over 1000 characters, anywhere between 1000-1100.

britzl · August 17, 2018, 4:47pm

Hmm, ok, from what I can tell the max URI length we support is 2048, so that should be enough then.

Could it be some encoding problem? Do you need to URL encode any of the arguments?

jonnyleeharris · August 17, 2018, 4:52pm

The URL looks like this:

https://triggered-d1327.firebaseio.com/test/anything/some_value.json?auth=eyJhbGciOiJSUzI1NiIsImtpZCI6IjY3NmY5MmU1MGQ5ZmUxNzdiM2I5NTJjNGM4ZWU2YjY1ZDk3ZWIwZmMifQ.eyJpc3MiOiJodHRwczovL3NlY3VyZXRva2VuLmdvb2dsZS5jb20vdHJpZ2dlcmVkLWQxMzI3IiwibmFtZSI6IkpvbmF0aGFuIEhhcnJpcyIsInBpY3R1cmUiOiJodHRwczovL2dyYXBoLmZhY2Vib29rLmNvbS8xMDE1NjM2ODg0NTY3NTQzNS9waWN0dXJlIiwiYXVkIjoidHJpZ2dlcmVkLWQxMzI3IiwiYXV0aF90aW1lIjoxNTM0NTE3MzEyLCJ1c2VyX2lkIjoiWDRSajdKaHQ4RWZMVkpXZ09MT1c2TGc1aGJzMSIsInN1YiI6Ilg0Umo3Smh0OEVmTFZKV2dPTE9XNkxnNWhiczEiLCJpYXQiOjE1MzQ1MTczMTIsImV4cCI6MTUzNDUyMDkxMiwiZmlyZWJhc2UiOnsiaWRlbnRpdGllcyI6eyJmYWNlYm9vay5jb20iOlsiMTAxNTYzNjg4NDU2NzU0MzUiXX0sInNpZ25faW5fcHJvdmlkZXIiOiJmYWNlYm9vay5jb20ifX0.LlCJP8yYHYV7P6cZRGs1iIipRNrXZIJ0qM2Lpxf7sC6FLnQsT1kSlBw7UqT32aoNe0Q4ckjet8YMAT1ndsprGM8KaZq7vYHvx2tQRo5ej81LWc1U4fHHWbFChPOMbD-QPTDrQUilKyiADX21Vhpynvq0LhS-zizCDKfC2cWwCB3rzgCivojek4zf4sunlFXC7zY9OFrivH4f3uFdB97uUy2i-lnAb2sY4_vRTvXg-pdymqUP6mL6fMlaQAzI5exZ-Dqc62jZfT846cF6ks1dnEy4s9m6LsLiz3qtHrP5RafhBd-MkA6-9TXR01nzVu_bjls6v-FUvayxAtqBkxHoxA

If you make this as a GET request, you’ll get a response

{
  "error" : "Auth token is expired"
}

If you try the same in Defold (I hope) you will get the error I get about not being able to parse the auth key.

jonnyleeharris · August 17, 2018, 4:56pm

For clarity:

The code:

local url = "https://triggered-d1327.firebaseio.com/test/anything/some_value.json?auth=eyJhbGciOiJSUzI1NiIsImtpZCI6IjY3NmY5MmU1MGQ5ZmUxNzdiM2I5NTJjNGM4ZWU2YjY1ZDk3ZWIwZmMifQ.eyJpc3MiOiJodHRwczovL3NlY3VyZXRva2VuLmdvb2dsZS5jb20vdHJpZ2dlcmVkLWQxMzI3IiwibmFtZSI6IkpvbmF0aGFuIEhhcnJpcyIsInBpY3R1cmUiOiJodHRwczovL2dyYXBoLmZhY2Vib29rLmNvbS8xMDE1NjM2ODg0NTY3NTQzNS9waWN0dXJlIiwiYXVkIjoidHJpZ2dlcmVkLWQxMzI3IiwiYXV0aF90aW1lIjoxNTM0NTE3MzEyLCJ1c2VyX2lkIjoiWDRSajdKaHQ4RWZMVkpXZ09MT1c2TGc1aGJzMSIsInN1YiI6Ilg0Umo3Smh0OEVmTFZKV2dPTE9XNkxnNWhiczEiLCJpYXQiOjE1MzQ1MTczMTIsImV4cCI6MTUzNDUyMDkxMiwiZmlyZWJhc2UiOnsiaWRlbnRpdGllcyI6eyJmYWNlYm9vay5jb20iOlsiMTAxNTYzNjg4NDU2NzU0MzUiXX0sInNpZ25faW5fcHJvdmlkZXIiOiJmYWNlYm9vay5jb20ifX0.LlCJP8yYHYV7P6cZRGs1iIipRNrXZIJ0qM2Lpxf7sC6FLnQsT1kSlBw7UqT32aoNe0Q4ckjet8YMAT1ndsprGM8KaZq7vYHvx2tQRo5ej81LWc1U4fHHWbFChPOMbD-QPTDrQUilKyiADX21Vhpynvq0LhS-zizCDKfC2cWwCB3rzgCivojek4zf4sunlFXC7zY9OFrivH4f3uFdB97uUy2i-lnAb2sY4_vRTvXg-pdymqUP6mL6fMlaQAzI5exZ-Dqc62jZfT846cF6ks1dnEy4s9m6LsLiz3qtHrP5RafhBd-MkA6-9TXR01nzVu_bjls6v-FUvayxAtqBkxHoxA"
	http.request(url, "GET", function(_,_,response)
		pprint(response)
	end)

Gives the following output:

{
  status = 401,
  response = {
  "error" : "Could not parse auth token."
}
,
  headers = {
    server = nginx,
    strict-transport-security = max-age=31556926; includeSubDomains; preload,
    content-type = application/json; charset=utf-8,
    connection = keep-alive,
    content-length = 46,
    cache-control = no-cache,
    date = Fri, 17 Aug 2018 16:55:17 GMT,
    access-control-allow-origin = *,
  }
}

But paste that into a browser, or into Postman, and you’ll get:

{
  "error" : "Auth token is expired"
}

britzl · August 17, 2018, 5:44pm

Interesting. If I build and launch in HTML5 it works and on my Mac it doesn’t. We do use a different implementation of http.request() on HTML5 though.

Have you tried to capture the data that is being sent to Firebase?

jonnyleeharris · August 17, 2018, 5:50pm

No, I haven’t. Interesting you say that though because I am using a Mac, as well as streaming the build to an iPhone (same result).

Been a while since I’ve used a data sniffer, I wonder if Charles is still up to the task. I’ll take a look.

britzl · August 17, 2018, 5:58pm

Charles will not work. You need to use a transparent proxy working on a low level.

jonnyleeharris · August 17, 2018, 6:00pm

Yes I am noticing that

jonnyleeharris · August 17, 2018, 6:48pm

OK, I used postman echo.

For the request:

local url = "https://postman-echo.com/get?auth=eyJhbGciOiJSUzI1NiIsImtpZCI6IjY3NmY5MmU1MGQ5ZmUxNzdiM2I5NTJjNGM4ZWU2YjY1ZDk3ZWIwZmMifQ.eyJpc3MiOiJodHRwczovL3NlY3VyZXRva2VuLmdvb2dsZS5jb20vdHJpZ2dlcmVkLWQxMzI3IiwibmFtZSI6IkpvbmF0aGFuIEhhcnJpcyIsInBpY3R1cmUiOiJodHRwczovL2dyYXBoLmZhY2Vib29rLmNvbS8xMDE1NjM2ODg0NTY3NTQzNS9waWN0dXJlIiwiYXVkIjoidHJpZ2dlcmVkLWQxMzI3IiwiYXV0aF90aW1lIjoxNTM0NTE3MzEyLCJ1c2VyX2lkIjoiWDRSajdKaHQ4RWZMVkpXZ09MT1c2TGc1aGJzMSIsInN1YiI6Ilg0Umo3Smh0OEVmTFZKV2dPTE9XNkxnNWhiczEiLCJpYXQiOjE1MzQ1MTczMTIsImV4cCI6MTUzNDUyMDkxMiwiZmlyZWJhc2UiOnsiaWRlbnRpdGllcyI6eyJmYWNlYm9vay5jb20iOlsiMTAxNTYzNjg4NDU2NzU0MzUiXX0sInNpZ25faW5fcHJvdmlkZXIiOiJmYWNlYm9vay5jb20ifX0.LlCJP8yYHYV7P6cZRGs1iIipRNrXZIJ0qM2Lpxf7sC6FLnQsT1kSlBw7UqT32aoNe0Q4ckjet8YMAT1ndsprGM8KaZq7vYHvx2tQRo5ej81LWc1U4fHHWbFChPOMbD-QPTDrQUilKyiADX21Vhpynvq0LhS-zizCDKfC2cWwCB3rzgCivojek4zf4sunlFXC7zY9OFrivH4f3uFdB97uUy2i-lnAb2sY4_vRTvXg-pdymqUP6mL6fMlaQAzI5exZ-Dqc62jZfT846cF6ks1dnEy4s9m6LsLiz3qtHrP5RafhBd-MkA6-9TXR01nzVu_bjls6v-FUvayxAtqBkxHoxA"

The response was:

{
  status = 200,
  response = {"args":{"auth":"eyJhbGciOiJSUzI1NiIsImtpZCI6IjY3NmY5MmU1MGQ5ZmUxNzdiM2I5NTJjNGM4ZWU2YjY1ZDk3ZWIwZmMifQ.eyJpc3MiOiJodHRwczovL3NlY3VyZXRva2VuLmdvb2dsZS5jb20vdHJpZ2dlcmVkLWQxMzI3IiwibmFtZSI6IkpvbmF0aGFuIEhhcnJpcyIsInBpY3R1cmUiOiJodHRwczovL2dyYXBoLmZhY2Vib29rLmNvbS8xMDE1NjM2ODg0NTY3NTQzNS9waWN0dXJlIiwiYXVkIjoidHJpZ2dlcmVkLWQxMzI3IiwiYXV0aF90aW1lIjoxNTM0NTE3MzEyLCJ1c2VyX2lkIjoiWDRSajdKaHQ4RWZMVkpXZ09MT1c2TGc1aGJzMSIsInN1YiI6Ilg0Umo3Smh0OEVmTFZKV2dPTE9XNkxnNWhiczEiLCJpYXQiOjE1MzQ1MTczMTIsImV4cCI6MTUzNDUyMDkxMiwiZmlyZW"},"headers":{"host":"postman-echo.com","x-forwarded-port":"443","x-forwarded-proto":"https"},"url":"https://postman-echo.com/get?auth=eyJhbGciOiJSUzI1NiIsImtpZCI6IjY3NmY5MmU1MGQ5ZmUxNzdiM2I5NTJjNGM4ZWU2YjY1ZDk3ZWIwZmMifQ.eyJpc3MiOiJodHRwczovL3NlY3VyZXRva2VuLmdvb2dsZS5jb20vdHJpZ2dlcmVkLWQxMzI3IiwibmFtZSI6IkpvbmF0aGFuIEhhcnJpcyIsInBpY3R1cmUiOiJodHRwczovL2dyYXBoLmZhY2Vib29rLmNvbS8xMDE1NjM2ODg0NTY3NTQzNS9waWN0dXJlIiwiYXVkIjoidHJpZ2dlcmVkLWQxMzI3IiwiYXV0aF90aW1lIjoxNTM0NTE3MzEyLCJ1c2VyX2lkIjoiWDRSajdKaHQ4RWZMVkpXZ09MT1c2TGc1aGJzMSIsInN1YiI6Ilg0Umo3Smh0OEVmTFZKV2dPTE9XNkxnNWhiczEiLCJpYXQiOjE1MzQ1MTczMTIsImV4cCI6MTUzNDUyMDkxMiwiZmlyZW"},
  headers = {
    server = nginx,
    content-length = 1156,
    content-type = application/json; charset=utf-8,
    connection = keep-alive,
    date = Fri, 17 Aug 2018 18:41:36 GMT,
    vary = Accept-Encoding,
    set-cookie = sails.sid=s%3AbRAlWCL1aNWCW5UO35xTDWuxlvEUK6Ty.yB%2FSUkicRkENPzWC5spZftydEZW5Teu1hiNArNkLrkU; Path=/; HttpOnly,
    etag = W/"484-NYDoATF7ogx14SCEpR3XSNw4c7A",
  }
}

It seems that postman echo is not receiving anything after:

https://postman-echo.com/get?auth=eyJhbGciOiJSUzI1NiIsImtpZCI6IjY3NmY5MmU1MGQ5ZmUxNzdiM2I5NTJjNGM4ZWU2YjY1ZDk3ZWIwZmMifQ.eyJpc3MiOiJodHRwczovL3NlY3VyZXRva2VuLmdvb2dsZS5jb20vdHJpZ2dlcmVkLWQxMzI3IiwibmFtZSI6IkpvbmF0aGFuIEhhcnJpcyIsInBpY3R1cmUiOiJodHRwczovL2dyYXBoLmZhY2Vib29rLmNvbS8xMDE1NjM2ODg0NTY3NTQzNS9waWN0dXJlIiwiYXVkIjoidHJpZ2dlcmVkLWQxMzI3IiwiYXV0aF90aW1lIjoxNTM0NTE3MzEyLCJ1c2VyX2lkIjoiWDRSajdKaHQ4RWZMVkpXZ09MT1c2TGc1aGJzMSIsInN1YiI6Ilg0Umo3Smh0OEVmTFZKV2dPTE9XNkxnNWhiczEiLCJpYXQiOjE1MzQ1MTczMTIsImV4cCI6MTUzNDUyMDkxMiwiZmly

which leaves out a good portion of the URL:

ZWJhc2UiOnsiaWRlbnRpdGllcyI6eyJmYWNlYm9vay5jb20iOlsiMTAxNTYzNjg4NDU2NzU0MzUiXX0sInNpZ25faW5fcHJvdmlkZXIiOiJmYWNlYm9vay5jb20ifX0.LlCJP8yYHYV7P6cZRGs1iIipRNrXZIJ0qM2Lpxf7sC6FLnQsT1kSlBw7UqT32aoNe0Q4ckjet8YMAT1ndsprGM8KaZq7vYHvx2tQRo5ej81LWc1U4fHHWbFChPOMbD-QPTDrQUilKyiADX21Vhpynvq0LhS-zizCDKfC2cWwCB3rzgCivojek4zf4sunlFXC7zY9OFrivH4f3uFdB97uUy2i-lnAb2sY4_vRTvXg-pdymqUP6mL6fMlaQAzI5exZ-Dqc62jZfT846cF6ks1dnEy4s9m6LsLiz3qtHrP5RafhBd-MkA6-9TXR01nzVu_bjls6v-FUvayxAtqBkxHoxA

britzl · August 17, 2018, 9:03pm

Yes, I did a similar test and found that we limit the path to 512 characters. This obviously needs to be fixed. I created a ticket to track this: DEF-3410. It should be an easy fix but I can’t make any promises as to when we will release it.

Does Firebase support an auth token header perhaps?

britzl · August 17, 2018, 9:10pm

Yes, it’s basically this:

char m_Path[512];

Should only be a matter of increasing to something a bit larger, say 2048. 2048 seems like a reasonable value based on this: http - What is the maximum length of a URL in different browsers? - Stack Overflow

jonnyleeharris · August 17, 2018, 9:38pm

I’ll take a look into workarounds - but unfortunately I don’t think they do.

At least I know what’s causing the problem and if you have a fix in the works I can sleep easy!

Thanks

britzl · August 18, 2018, 9:37am

This page seems to indicate support for an auth header, but I couldn’t get it to work:

https://firebase.google.com/docs/database/rest/auth#authenticate_with_an_access_token

Am I looking at the wrong docs?

jonnyleeharris · August 18, 2018, 10:30am

That’s the right page, but there are two ways to authenticate.

You can use a header to authenticate with Google OAuth2 access tokens, typically used on a server - which gives you full read and write access to the database.

If you want to authenticate with Firebase ID tokens, which are used for authenticating users - you must append the auth parameter to the request.