1518 Commits

Author SHA1 Message Date
Joris Vink
cd9971247c Add seccomp syscall filtering to kore.
With this commit all Kore processes (minus the parent) are running
under seccomp.

The worker processes get the bare minimum allowed syscalls while each module
like curl, pgsql, etc will add their own filters to allow what they require.

New API functions:
    int kore_seccomp_filter(const char *name, void *filter, size_t len);

    Adds a filter into the seccomp system (must be called before
    seccomp is enabled).

New helpful macro:
    define KORE_SYSCALL_ALLOW(name)

    Allow the syscall with a given name, should be used in
    a sock_filter data structure.

New hooks:
    void kore_seccomp_hook(void);

    Called before seccomp is enabled, allows developers to add their
    own BPF filters into seccomp.
2019-09-25 14:31:20 +02:00
Joris Vink
d99d0b2d77 add lock.trylock() and make lock.release() sync. 2019-09-20 10:03:43 +02:00
Joris Vink
06fd5ca2f2 Add aqcuire() and release() to pylock.
This allows it to be used from code without requiring the async with
context manager approach.
2019-09-20 09:56:21 +02:00
Joris Vink
c3b2a8b2a2 fix NOHTTP builds 2019-09-20 09:37:02 +02:00
Joris Vink
868b0a71a2 make coro tracing available at runtime, always.
call kore.corotrace(True) to enable it and kore.corotrace(False) to disable.
2019-09-18 10:55:13 +02:00
Joris Vink
95506204e8 Add kore.task_kill() to the python api.
Allows killing of coroutines, given their task id.

The kore.task_create() method now returns the task id for a newly
created task to the caller.

While here, change the coroutine task id to a uint32 from uint64.
There is no need for it to be 64bit. (famous last words)
2019-09-17 20:04:35 +02:00
Joris Vink
b107485ea4 if built with python+debug enable coro tracing 2019-09-16 20:36:23 +02:00
Joris Vink
c1653a5254 hide kore.coroname() behind PYTHON_CORO_TRACE. 2019-09-16 20:33:14 +02:00
Joris Vink
019006620d allow coroutines to set friendly names. 2019-09-16 20:31:25 +02:00
Joris Vink
960fabe44c s/PYTHON_CORO_DEBUG/PYTHON_CORO_TRACE 2019-09-16 20:25:34 +02:00
Joris Vink
1ee1021aba trace created/deleted. 2019-09-16 20:21:04 +02:00
Joris Vink
9bcccb9483 Add coroutine debugging.
If built with PYTHON_CORO_DEBUG in CFLAGS Kore will spew out coroutine
traces while running. These traces include the filename, function and line
number where the coroutines are waking up, running and suspended.
2019-09-16 20:11:42 +02:00
Joris Vink
dbb99a56d1 move new/wokenup/suspended coros to end of lists. 2019-09-16 19:34:47 +02:00
Joris Vink
58a6b4e331 trailing space fix 2019-09-13 23:22:38 +02:00
Joris Vink
3207ce8526 pgsql improvements round 2
- kill PQsetnonblocking(), it's not what one thinks.
- keep doing the PQconsumeInput()/PQisBusy() dance until
  the former clears all read data.
2019-09-13 23:20:51 +02:00
Frederic Cambus
a8c6ecbb89 Set default MAN_DIR value only if it's not already set. 2019-09-05 09:42:54 +02:00
Joris Vink
f2472ba485 allow python modules to set progname. 2019-09-04 20:37:33 +02:00
Joris Vink
88bd3ce045 tiny comment on koreapp 2019-09-04 20:10:48 +02:00
Joris Vink
c10813dc44 turn python-pgsql into a real kore python app 2019-09-04 20:07:04 +02:00
Joris Vink
8e858983bf python pgsql changes.
- decouple pgsql from the HTTP request allowing it to be used in other
  contexts as well (such as a task, etc).

- change names to dbsetup() and dbquery().

eg:

result = kore.dbquery("db", "select foo from bar")
2019-09-04 19:57:28 +02:00
Joris Vink
f3b7cba58c Call PQConsumeInput() again after PQisBusy().
Prevents a stall in case there is still data in the read end of the socket
but PQisBusy() told us to not fetch a result yet. In that case we end up
stalling due to epoll not giving us another EPOLLIN event due to EPOLLET.
2019-09-04 19:19:52 +02:00
Joris Vink
b54b93536d Use strdup() when munging environment pointers. 2019-08-27 13:12:44 +02:00
Joris Vink
c32880e12b http_argument_decode() can fail. 2019-08-02 11:34:45 +02:00
Joris Vink
0b9e174af4 http_argument_urldecode() could fail. 2019-07-20 20:35:58 +02:00
Joris Vink
0f58de36f4 Be a little less strict on url decoding. 2019-07-16 21:31:32 +02:00
Joris Vink
f2e17c1384 return NULL if PyUnicode_FromString fails. 2019-07-16 21:31:19 +02:00
Joris Vink
5055f3c872 add kore.worker() to the python api.
returns the current worker number or None if on parent.
2019-06-16 19:57:48 +02:00
Joris Vink
4a64b4f07b Improve curl timeout handling.
In case libcurl instructs us to call the timeout function as soon
as possible (timeout == 0 in curl_timeout), don't try to be clever
with a timeout value of 10ms.

Instead call the timeout function once we get back in the worker
event loop. This makes things a lot snappier as we don't depend
on epoll/kqueue waiting for io for 10ms (which actually isn't 10ms...).
2019-06-13 12:59:17 +02:00
Joris Vink
574c9a7084 make sure kodev run works in python modules. 2019-06-13 11:18:10 +02:00
Joris Vink
3114f8d8d0 Improve python experience.
- If Kore is built with PYTHON=1 you can now specify the module that
  should be loaded on the command-line.

     eg: $ kore -frn myapp

- Add skeleton generation for python applications to kodev.

     eg: $ kodev create -p myapp

This should make it a whole lot easier to get started with kore python.
2019-06-12 23:35:43 +02:00
Joris Vink
a46447b1f9 set hdlr_extra to NULL in pyhttp_iterobj_disconnect. 2019-06-12 21:09:57 +02:00
Joris Vink
6c18856a3e get rid of mtime in modules.
kore should always reload things when it was told to reload things.
there are use cases were a module reload is because the code itself changed.
2019-06-09 23:24:53 +02:00
Frederic Cambus
78ff364c8d Fix a couple of typos in the bundled examples. 2019-06-09 21:37:10 +02:00
Frederic Cambus
bae117b68c Set req->agent to "-" if it is NULL, as do other major HTTP servers.
This avoids passing a NULL value to a format string when writing a log
entry.
2019-06-09 21:36:12 +02:00
Joris Vink
d4f4e007e0 make sure prereqs are in order 2019-06-07 21:33:00 +02:00
Joris Vink
6ed3ac4658 kill debug 2019-06-07 21:25:18 +02:00
Joris Vink
3cc7d6e238 Allow kore.prerequests to be async. 2019-06-07 21:06:54 +02:00
Joris Vink
9cc58d45c1 Python HTTP improvements.
1) Add @kore.prerequest python decorator.

Using this decorator on a function will cause that function
to always be executed *before* any page handler is run.

eg:

@kore.prerequest
def _check(req):
    if req.method == kore.HTTP_METHOD_POST:
        req.populate_post()

2) Allow attributes to be set on the pyhttp object.
2019-06-05 23:45:45 +02:00
Joris Vink
89e58fa474 Improve iterator support for Python req.response().
If the connection on which we are about to send the response was
marked as disconnecting, do not go ahead and hook into the disconnect
callback (it will never be called, it is already disconnecting).

Instead just return, the connection will be removed anyway.
2019-06-05 10:35:47 +02:00
Joris Vink
93b1d621d7 Remove memleak from Python httpclient.
We grab a reference to the pyhttp_client for the client_op data structure
but never removed it. This caused the pyhttp_client object to never
be released when out of scope.
2019-06-05 10:27:20 +02:00
Joris Vink
a4d18ca276 Add HTTP runlocks.
A way to serialize access to HTTP page handlers in case you are
using some asynchronous api such as pgsql or libcurl stuff.
2019-06-02 16:29:54 +02:00
Joris Vink
1686ec22e6 Some C pgsql api improvements. 2019-06-01 23:14:50 +02:00
Joris Vink
ff7c85460c Wake up HTTP requests if python coros finish immediately.
python_coro_create() puts the HTTP request to sleep, but if they
finish immediately they will be removed from the list and should
properly be woken up or they are removed from the wrong list.
2019-05-30 17:15:08 +02:00
Joris Vink
a8aff8b737 Improve curl error string handling.
Introduce kore_curl_strerror(), use this in kore_curl_logerror()
instead of assuming our errbuf has been populated.

Also use it in the python httpclient when throwing an exception rather
then looking at the errbuf member which may or may not be empty.
2019-05-30 14:25:04 +02:00
Joris Vink
88553cd2dd Immediately remove completed HTTP requests.
No need to wait until the next time http_process() is called, which
could result in HTTP requests backing up even though we are processing
them at a fast pace.
2019-05-29 20:30:43 +02:00
Joris Vink
07fc7a9097 Improve HTTP processing.
If netwait is INFINITE but there are requests pending reduce the
netwait back down to 100ms so we keep processing them.
2019-05-29 15:27:44 +02:00
Joris Vink
c2d9f1413c Fix two concurrency issues with kore.lock().
1) If a coroutine was waken up by another releasing a lock it was waiting
on we would incorrectly remove the pylock_op when the newly awoken coroutine
hits the iternext for pylock.

2) If a coroutine attempts to grab a lock it was woken up for only to fail
we did not properly rearm the coroutine to be woken up later to try again.
2019-05-29 15:25:31 +02:00
Joris Vink
5f0153ba0e Fix unix binds on BSD families. 2019-05-28 21:44:46 +02:00
Joris Vink
b4aba48fad small python improvements.
- Check for an exception when in a timer context and log it if uncaught.
- Make sure tlsverify is settable regardless of client certs being specified.
2019-05-16 21:07:42 +02:00
Joris Vink
41366ba583 avoid using pkg-config on FreeBSD for CURL=1. 2019-05-14 20:53:27 +02:00