These functions are created by the cli tool when building
and follow the naming format: asset_serve_<name>_<ext>().
Those serving functions can be used directly in handlers and
callthrough to a http_serveable() function that uses the SHA1
of the asset as its ETag and automatically checks for if-none-match.
This commit is a flag day, your old modules will almost certainly
need to be updated in order to build properly with these changes.
Summary of changes:
- Offload HTTP bodies to disk if they are large (inspired by #100).
(disabled by default)
- The http_argument_get* macros now takes an explicit http_request parameter.
- Kore will now throw 404 errors almost immediately after an HTTP request
has come in instead of waiting until all data has arrived.
API changes:
- http_argument_get* macros now require an explicit http_request parameter.
(no more magic invokations).
- http_generic_404() is gone
- http_populate_arguments() is gone
- http_body_bytes() is gone
- http_body_text() is gone
- http_body_read() has been added
- http_populate_post() has been added
- http_populate_get() has been added
- http_file_read() has been added
- http_file_rewind() has been added
- http_file_lookup() no longer takes name, fname, data and len parameters.
- http_file_lookup() now returns a struct http_file pointer.
- http_populate_multipart_form() no longer takes an secondary parameter.
New configuration options:
- http_body_disk_offload:
Number of bytes after which Kore will offload the HTTP body to
disk instead of retaining it in memory. If 0 this feature is
disabled. (Default: 0)
- http_body_disk_path:
The path where Kore will store temporary HTTP body files.
(this directory does not get created if http_body_disk_offload is 0).
New example:
The upload example has been added, demonstrating how to deal with file
uploads from a multipart form.
Kore pre-allocates a kore_buf for the full size of the
incoming HTTP body ... but also was passing the full
size to the net_recv_reset() function.
Instead of this, properly read smaller chunks from the
network and append them to the body buffer as they roll in.
Allow setting it to 0 which will disable HTTP requests
that have a body (POST/PUT).
Reduce default http_body_max to 1MB by default, 10MB seems large.
Revisit to this code inspired by #100.
This basically turns off the HTTP layer for Kore. It does not
compile in anything for HTTP.
This allows Kore to be used as a network application platform as well.
Added an example for this called nohttp.
Other changes that sneaked in while hacking on this:
* Use calloc(), kill pendantic malloc option.
* Killed off SPDY/3.1 support completely, will be superseded by http2
Note that comes with massive changes to a lot of the core API
functions provided by Kore, these might break your application.
Add HTTP_REQUEST_NO_CONTENT_LENGTH which can be set by
a handler before calling http_response() to avoid Kore
from setting the content-length altogether.
If we are on a SPDY connection do not close the stream
if we do not pass data to http_response().
Introduces two new configuration knobs:
* socket_backlog (backlog for listen(2))
* http_request_limit
The second one is the most interesting one.
Before, kore would iterate over all received HTTP requests
in its queue before returning out of http_process().
Under heavy load this queue can cause Kore to spend a considerable
amount of time iterating over said queue. With the http_request_limit,
kore will process at MOST http_request_limit requests before returning
back to the event loop.
This means responses to processed requests are sent out much quicker
and allows kore to handle any other incoming requests more gracefully.
Signals Kore to not free any pointer set in req->hdlr_extra.
Useful in certain scenarios where you have data per request
bound to something in memory but do not want to lose it when
the request is freed by Kore.
Set this flag before your handler returns.
- The net code no longer has a recv_queue, instead reuse same recv buffer.
- Introduce net_recv_reset() to reset the recv buffer when needed.
- Have the workers spread the load better between them by slightly
delaying their next accept lock and giving them an accept treshold
so they don't go ahead and keep accepting connections if they end
up winning the race constantly between the workers.
- The kore_worker_acceptlock_release() is no longer available.
- Prepopulate the HTTP server response header that is added to each
response in both normal HTTP and SPDY modes.
- The path and host members of http_request are now allocated on the heap.
These changes overall result better performance on a multicore machine,
especially the worker load changes shine through.
This commit renames certain POST centric variable and configuration
naming to the correct HTTP body stuff.
API changes include http_postbody_text() and http_postbody_bytes() to
have become http_body_text() and http_body_bytes().
The developer is still responsible for validating the method their
page handler is called with. Hopefully this becomes a configuration
option soon enough.
Instead of letting http_requests spin, if we cannot allocate
a connection for the request we will queue them up put them to sleep.
When a connection becomes available, we'll wake up a request that
was waiting for a connection and let it continue.
This completely avoids consuming massive amounts of cpu time
when dealing with thousands of requests waiting for a pgsql
worker to become ready.
Gone is the ugly KORE_PGSQL macro that hid an overly complex
state machine for the pgsql api.
Gone is the pgsql array that was attached to http_requests.
Gone are the callback hacks inside the pgsql api.
Instead, I strongly encourage people to use the new state machine
api Kore offers to properly deal with asynchronous queries.
The pgsql example in examples/pgsql has been updated to reflect
these changes.
In order to use this, define states for your page handler:
struct http_state mystates[] = {
{ "PAGE_STATE_INIT", page_init },
{ "PAGE_STATE_RESULT", page_result },
};
In your page handler you can then simply call http_state_run() with
your states and http_request. This will cause Kore to start calling
your state callbacks beginning at index 0.
State callbacks have the same prototype as page handlers:
int func(struct http_request *);
However, unlike page handlers they MUST return one of the following:
- HTTP_STATE_OK: All good, just continue the fsm.
- HTTP_STATE_ERROR: Abort fsm and return KORE_RESULT_OK to Kore
(This will cancel the http request).
- HTTP_STATE_RETRY: Return KORE_RESULT_RETRY to Kore.
(Kore will retry your page handler next event loop).
- HTTP_STATE_COMPLETE: The fsm completed, break out cleanly.
Note that using this is completely optional and you can still
use the traditional way of writing page handlers.
The fsm is designed to get rid of the clutter that exists today
in Kore when dealing with non blocking tasks or pgsql calls.
This way we can get our code called whenever a stream is
completed. This cb handler does stand alone from an http_request
and is passed a netbuf data structure.
These 2 functions can be used to move an HTTP request
from/to the active http_requests list. Effectively
putting them to "sleep" or "waking them up".
Sprinkle this through the pgsql and task code.
If used correctly greatly reduces overhead for
managing sleeping tasks.
Tasks are now assigned to available threads instead
of a global task list.
You can now pass messages between your page handler
and the created task using the kore_task_channel_*
functions.
Only one task per time can be assigned to a request
but I feel this is probably a bad design choice.
Preferably we'd want to be able to start tasks
regardless of being in a page handler or not,
this not only ads flexibility but seems like
a better choice overall as it opens a lot more
possibilities about how tasks can be used.
Don't wait for a full event loop until we call the page handler
for a received pgsql result. This speeds up page loads using
KORE_PGSQL by quite a lot, especially on a non busy server.
Including but not limited to:
- Correctly use PQerrorMessage() in case we cleanup with PQfinish
- If we get a network error, cleanup the connection
- No longer call the page handler from inside kore_pgsql_handle()
but instead just put it to sleep in case we don't need it.
This does grow the http_requests list quite a bit with sleeping
connections and can perhaps be improved later on.
- Allow us to on error return OK from a page handler from inside
the completetion block for KORE_PGSQL().
- Count the cummulative time for a request to finish instead
of the latest run time for the handler.
Has support for full async pgsql queries. Most of the logic
is hidden behind a KORE_PGSQL() macro allowing you to insert
these pgsql calls in your page handlers without blocking the
kore worker while the query is going off.
There is place for improvement here, and perhaps KORE_PGSQL won't
stay as I feel this might overcomplicate things instead of making
them simpler as I thought it would.
Using authentication blocks one can define "authentication" mechanisms
in Kore for page handlers.
This can be used to require a session cookie (validated by your own validator)
for certain page handlers, and hopefully in the future provide a framework
for adding more authentication things (like HTTP Auth).
Right now only cookie checking is available.
- Parameter validation is now done only when http_process_*()
is called and upon http_argument_add().
- You MUST have defined your params in a param block or they will
be filtered out.
- http_argument_lookup() is dead, welcome http_argument_get() and
its brothers and sisters:
http_argument_get_string()
http_argument_get_uint16()
http_argument_get_int16()
http_argument_get_uint32()
http_argument_get_int32()
They will automatically do bounds checking on integers for you
and return proper integers or a NUL-terminated string.
- The http_argument_get* functions no longer create an additional
copy of the string which you need to free. Easier going.
- http_multiple_args() is dead, byebye
- Make some stuff we don't want to share with the modules static.
