1806 Commits

Author SHA1 Message Date
Joris Vink
17ceb32e23 Cleanup single/dso files properly with kodev clean.
Obey the out_dir too, in case its set differently.
2021-09-21 20:17:44 +02:00
Joris Vink
46b7a962bf be verbose about filemap resolving. 2021-09-21 16:17:46 +02:00
Joel Arbring
31aaf128a1 Check for .so file where we create it 2021-09-17 20:04:46 +02:00
Joris Vink
63bbc1fa0f Be sure content_length is 0, just in case. 2021-09-17 19:56:35 +02:00
Joris Vink
6b1f02e6b0 Remove another dead assignment. 2021-09-17 19:53:31 +02:00
Joris Vink
9f6043bbde Remove dead assignment. 2021-09-17 19:52:52 +02:00
Joris Vink
bcfb79a389 Remove dead assignment and unused vars. 2021-09-17 19:49:32 +02:00
Joris Vink
351eec7eb4 Add the on_body_chunk handler for routes.
If set, will call a given handler with the prototype of

`void body_chunk(struct http_request *req, const void *data, size_t len);`

for each chunk of the received HTTP body, allowing a developer to handle
it in their own way.

The incoming body is still being handled and retained in the same way
as before (in a kore_buf or temporary file).

While here, allow HTTP_STATUS_CONTINUE to work via http_response() and
make the handling of incoming HTTP header data a bit better.
2021-09-17 19:30:22 +02:00
Joris Vink
f5a58368b7 HTTP improvements.
Introduce an on_headers callback for routes, allowing one to inspect
the headers before the request is processed further.

Additionall,

Add a new way of obtaining HTTP headers. Much like http_argument_get_*()
functions, these new APIs allow you to fetch the data of an HTTP header
as a specified C type.

The new APIs are:

* http_request_header_int16()
* http_request_header_uint16()
* http_request_header_int32()
* http_request_header_uint32()
* http_request_header_int64()
* http_request_header_uint64()
* http_request_header_float()
* http_request_header_double()

Should make it easier to operate in HTTP header data in a safe way.
No need to always roll your own string to int conversion functions.
2021-09-15 22:16:22 +02:00
Joris Vink
e98a4ddab5 Change how routes are configured in Kore.
Routes are now configured in a context per route:

route /path {
	handler handler_name
	methods get post head
	validate qs:get id v_id
}

All route related configurations are per-route, allowing multiple
routes for the same path (for different methods).

The param context is removed and merged into the route context now
so that you use the validate keyword to specify what needs validating.
2021-09-15 11:09:52 +02:00
Joris Vink
2576427dc0 Simplify if a worker needs to be added to msg.
Just look if the kore_worker data structure its ps member is non NULL.
If it is, it was started by the parent process at some point.
2021-09-14 15:45:34 +02:00
Joris Vink
a2d48feeb7 Lets make sure milliseconds are formatted nicely. 2021-09-14 09:36:33 +02:00
Joris Vink
41511c1683 Log timestamps in UTC for, add milliseconds.
This is when using the normal foreground logs or a specified logfile.
2021-09-14 09:30:17 +02:00
Joris Vink
b6ec4081d5 Oops, remove debug. 2021-09-13 22:55:09 +02:00
Joris Vink
824d6421d5 Use correct format for strftime(). 2021-09-13 22:54:05 +02:00
Joris Vink
983f5a03f5 Initial mem and log earlier.
Kill the kodev mention.
2021-09-13 15:33:42 +02:00
Joris Vink
450aabbea1 Add timestamp prefix to log when not using syslog. 2021-09-13 15:07:43 +02:00
Joris Vink
77848e0708 Always use logfile when set, even if !foreground. 2021-09-12 15:19:37 +02:00
Joris Vink
4b4767ac39 Missed committing http.h from previous commit. 2021-09-12 15:12:48 +02:00
Joris Vink
862bf1a5f6 Add http_response_json().
Since its an HTTP response function it functions like http_response() but
takes a kore_json_item pointer that it will automatically convert to a kore_buf
and send/free using http_response_stream().

While here fix a problem with http_response_stream() which could end up
not calling the cb() in case of HTTP_METHOD_HEAD. Since the behaviour is
that it should call cb() when done it should do so immediately.
2021-09-12 15:10:06 +02:00
Joris Vink
cdd681d602 Let http_response_header() handle duplicates.
If a response header was previously by an application for an HTTP request
http_response_header() will now overwrite the previous value.
2021-09-12 14:30:33 +02:00
Joris Vink
eb0b8f21e3 Add http_response_close() to the C API.
This is the same as http_response() except it will automatically
close the connection after the response is sent.

This is a bit easier than setting CONN_CLOSE_EMPTY yourself manually.
2021-09-12 14:13:24 +02:00
Joris Vink
b77d727f72 Add a logfile configuration option.
This will log all output from Kore processes to the specified file.
2021-09-10 13:34:57 +02:00
Joris Vink
ff6bae6513 move startup log back into kore_server_start(). 2021-09-07 23:26:36 +02:00
Joris Vink
a6677b873f On linux, keep track of seccomp tracing properly.
With the new process startup code we must handle the SIGSTOP
from the processes if seccomp_tracing is enabled. Otherwise
they just hang indefinitely and we assume they failed to start,
which is somewhat true.
2021-09-07 23:05:25 +02:00
Joris Vink
1af0772ebe Don't add acme to msg framework if not needed. 2021-09-07 22:27:02 +02:00
Joris Vink
116f935e10 use the correct name for acme. 2021-09-07 22:19:21 +02:00
Joris Vink
9fd30db598 Change timeout for worker startup a bit.
Also give some feedback we are waiting for process startup.
2021-09-07 22:14:28 +02:00
Joris Vink
bbae4be6f1 remove unused kore_progname. 2021-09-07 22:06:18 +02:00
Joris Vink
3b20cda11c Rework worker startup/privsep config.
Starting with the privsep config, this commit changes the following:

- Removes the root, runas, keymgr_root, keymgr_runas, acme_root and
  acme_runas configuration options.

  Instead these are now configured via a privsep configuration context:

  privsep worker {
      root /tmp
      runas nobody
  }

  This is also configurable via Python using the new kore.privsep() method:

      kore.privsep("worker", root="/tmp", runas="nobody", skip=["chroot"])

Tied into this we also better handle worker startup:

- Per worker process, wait until it signalled it is ready.
- If a worker fails at startup, display its last log lines more clearly.
- Don't start acme process if no domain requires acme.
- Remove each process its individual startup log message in favour
  of a generalized one that displays its PID, root and user.
- At startup, log the kore version and built-ins in a nicer way.
- The worker processes now check things they need to start running
  before signaling they are ready (such as access to CA certs for
  TLS client authentication).
2021-09-07 21:59:22 +02:00
Joris Vink
1c33ce01d0 Add kore_build_date to version.c 2021-09-07 21:58:53 +02:00
Joris Vink
c8c9a24d99 Only set initial python deployment if needed.
We should only be setting this if an actual module was specified
on the command-line that will be loaded.
2021-09-07 21:15:17 +02:00
Joris Vink
599835e7fd Python: Only use parameters if needed.
We always called kore_pgsql_query_param_fields() regardless if the
params keyword was specified or not, instead only use it if actual
parameters have been given.

Otherwise use the kore_pgsql_query() function directly to execute the query.
2021-09-06 15:39:38 +02:00
Joris Vink
0ac54eb48d Add a kore.config.skipchroot to the Python API.
If set to True, will skip the chroot() of the Kore workers.

This can be handy in case you want to set your deployment target to
production or docker so you get user changes but you don't want
to chroot the processes.
2021-09-06 14:35:04 +02:00
Joris Vink
00ef837d62 call explicit fflush() on stdout 2021-09-06 14:16:09 +02:00
Joris Vink
06991d22d5 remove norwegian debug 2021-09-06 13:40:33 +02:00
Joris Vink
7f56c7dbf2 Change how worker processes do logging.
Before each worker process would either directly print to stdout if
Kore was running in foreground mode, or syslog otherwise.

With this commit the workers will submit their log messages to the
parent process who will either put it onto stdout or syslog.

This change in completely under the hood and users shouldn't care about it.
2021-09-06 13:28:38 +02:00
Joris Vink
8661aee2f4 only log key info if !kore_quiet 2021-09-06 13:26:54 +02:00
Joris Vink
ebe090833d add Matthew to BEERS 2021-09-05 18:04:54 +02:00
Joris Vink
5f11f796a8 Allow configuration to pickup values from environment.
Eg:

certfile $CERTFILE

will pickup the value from the set $CERTFILE environment variable.

This works for _any_ Kore configuration option.
2021-09-05 17:53:09 +02:00
Joris Vink
3c4acd9ac3 Allow curlopt keyword at httpclient request level.
Now you can set curlopt on kore.httpclient at both the
global httpclient object level and individual requests.

Eg:

client = kore.httpclient("https://kore.io",
    curlopt={
        kore.CURLOPT_VERBOSE: 1
    }
)

status, body = await client.get(
    curlopt={
        kore.CURLOPT_VERBOSE: 0
    }
)
2021-08-27 10:42:40 +02:00
Joris Vink
3eff4b9790 whitespace fixes 2021-08-27 10:12:11 +02:00
Joris Vink
355cf87b93 use correct format specifier. 2021-08-27 10:06:45 +02:00
Joris Vink
55aaef875d Add support for setting curlopts in kore.httpclient.
Much of the work done by Matthew Norström with minor cleanup by me.
2021-08-27 10:05:30 +02:00
Joris Vink
89085246e5 style nits 2021-07-10 10:03:01 +02:00
Joris Vink
6d7c774ba2 remove support for older openssl apis in keymgr. 2021-07-10 10:02:53 +02:00
Joris Vink
90056dbdcb make python_module_init() non static 2021-07-10 10:02:46 +02:00
Joris Vink
95139925ec Add query string support to the Python validator API.
Now you can specify the qs keyword in a route which can contain
validators for the query string.

Eg:

@kore.route("/", methods=["post"], qs={"id": "^[0-9]+$"})
def index:
    ...
2021-06-18 13:00:57 +02:00
Joris Vink
f1a65ef236 Small improvement to the Python kore.timer() api.
Do not allow kore.timer() to be called from the parent process
as it shouldn't be run there.

This makes Kore fail more gracefully.
2021-06-03 14:03:45 +02:00
Joris Vink
f39919e98c Add readlinkat to seccomp whitelist. 2021-05-10 23:05:58 +02:00