blackbeard420/kore
1
0
Fork 0
mirror of https://github.com/jorisvink/kore synced 2025-03-13 22:39:01 -04:00
Code Issues Projects Releases Wiki Activity
kore/src/pgsql.c

809 lines
18 KiB
C
Raw Normal View History

Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
/*
bump copyright year.
2021-01-11 23:46:08 +01:00
* Copyright (c) 2014-2021 Joris Vink <joris@coders.se>
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
*
* Permission to use, copy, modify, and distribute this software for any
* purpose with or without fee is hereby granted, provided that the above
* copyright notice and this permission notice appear in all copies.
*
* THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
* WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
* ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
* WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
* ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
* OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*/
#include <sys/param.h>
#include <sys/queue.h>
#include <libpq-fe.h>
Pull in pg_config.h correctly, fixes compilation for pgsql under OpenBSD
2014-07-18 15:59:07 +02:00
#include <pg_config.h>
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
#include "kore.h"
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
#if !defined(KORE_NO_HTTP)
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
#include "http.h"
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
#endif
Normalize kore_*.h headers
2014-07-03 22:14:46 +02:00
#include "pgsql.h"
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
Add seccomp syscall filtering to kore. With this commit all Kore processes (minus the parent) are running under seccomp. The worker processes get the bare minimum allowed syscalls while each module like curl, pgsql, etc will add their own filters to allow what they require. New API functions: int kore_seccomp_filter(const char *name, void *filter, size_t len); Adds a filter into the seccomp system (must be called before seccomp is enabled). New helpful macro: define KORE_SYSCALL_ALLOW(name) Allow the syscall with a given name, should be used in a sock_filter data structure. New hooks: void kore_seccomp_hook(void); Called before seccomp is enabled, allows developers to add their own BPF filters into seccomp.
2019-09-25 12:25:49 +00:00
#if defined(__linux__)
#include "seccomp.h"
static struct sock_filter filter_pgsql[] = {
seccomp improvements. More BPF helper macros, more helper for granular syscall checking. Use these throughout kore where it makes sense. The new helpers: - KORE_SYSCALL_DENY_ARG(name, arg, value, errno): Deny the system call with errno if the argument matches value. - KORE_SYSCALL_DENY_MASK(name, arg, mask, errno): Deny the system call with errno if the mask argument does not match the exact mask given. - KORE_SYSCALL_DENY_WITH_FLAG(name, arg, flag, errno): Deny the system call with errno if the argument contains the given flag. The reverse also exists: - KORE_SYSCALL_ALLOW_ARG() - KORE_SYSCALL_ALLOW_MASK() - KORE_SYSCALL_ALLOW_WITH_FLAG()
2019-09-26 13:51:53 +02:00
/* Allow us to create sockets and call connect. */
Add seccomp syscall filtering to kore. With this commit all Kore processes (minus the parent) are running under seccomp. The worker processes get the bare minimum allowed syscalls while each module like curl, pgsql, etc will add their own filters to allow what they require. New API functions: int kore_seccomp_filter(const char *name, void *filter, size_t len); Adds a filter into the seccomp system (must be called before seccomp is enabled). New helpful macro: define KORE_SYSCALL_ALLOW(name) Allow the syscall with a given name, should be used in a sock_filter data structure. New hooks: void kore_seccomp_hook(void); Called before seccomp is enabled, allows developers to add their own BPF filters into seccomp.
2019-09-25 12:25:49 +00:00
KORE_SYSCALL_ALLOW(connect),
seccomp improvements. More BPF helper macros, more helper for granular syscall checking. Use these throughout kore where it makes sense. The new helpers: - KORE_SYSCALL_DENY_ARG(name, arg, value, errno): Deny the system call with errno if the argument matches value. - KORE_SYSCALL_DENY_MASK(name, arg, mask, errno): Deny the system call with errno if the mask argument does not match the exact mask given. - KORE_SYSCALL_DENY_WITH_FLAG(name, arg, flag, errno): Deny the system call with errno if the argument contains the given flag. The reverse also exists: - KORE_SYSCALL_ALLOW_ARG() - KORE_SYSCALL_ALLOW_MASK() - KORE_SYSCALL_ALLOW_WITH_FLAG()
2019-09-26 13:51:53 +02:00
KORE_SYSCALL_ALLOW_ARG(socket, 0, AF_INET),
KORE_SYSCALL_ALLOW_ARG(socket, 0, AF_INET6),
KORE_SYSCALL_ALLOW_ARG(socket, 0, AF_UNIX),
/* Requires these calls. */
Add seccomp syscall filtering to kore. With this commit all Kore processes (minus the parent) are running under seccomp. The worker processes get the bare minimum allowed syscalls while each module like curl, pgsql, etc will add their own filters to allow what they require. New API functions: int kore_seccomp_filter(const char *name, void *filter, size_t len); Adds a filter into the seccomp system (must be called before seccomp is enabled). New helpful macro: define KORE_SYSCALL_ALLOW(name) Allow the syscall with a given name, should be used in a sock_filter data structure. New hooks: void kore_seccomp_hook(void); Called before seccomp is enabled, allows developers to add their own BPF filters into seccomp.
2019-09-25 12:25:49 +00:00
KORE_SYSCALL_ALLOW(getsockopt),
KORE_SYSCALL_ALLOW(getsockname),
};
#endif
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
struct pgsql_wait {
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
struct kore_pgsql *pgsql;
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
TAILQ_ENTRY(pgsql_wait) list;
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
};
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
struct pgsql_job {
struct kore_pgsql *pgsql;
TAILQ_ENTRY(pgsql_job) list;
Introduce a wait queue for our pgsql code. Instead of letting http_requests spin, if we cannot allocate a connection for the request we will queue them up put them to sleep. When a connection becomes available, we'll wake up a request that was waiting for a connection and let it continue. This completely avoids consuming massive amounts of cpu time when dealing with thousands of requests waiting for a pgsql worker to become ready.
2014-08-21 16:36:12 +02:00
};
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
#define PGSQL_CONN_MAX 2
#define PGSQL_CONN_FREE 0x01
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
#define PGSQL_LIST_INSERTED 0x0100
Add pgsql_queue_limit configuration option. Limits the number of queued asynchronous queries kore will allow before starting to return errors for KORE_PGSQL_ASYNC setups. By default set to 1000. Avoids uncontrolled growth of the pgsql_queue_wait pool.
2018-02-13 13:21:27 +01:00
#define PGSQL_QUEUE_LIMIT 1000
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
Add kore_pgsql_query_params(). This function uses PQsendQueryParams() instead of the normal PQsendQuery() allowing you to pass binary data in a cleaner fashion. A basic call would look something like: char *mydata = "Hello"; size_t mydata_len = strlen(mydata); kore_pgsql_query_params(&pgsql, req, "INSERT INTO foo VALUES($1::text)", KORE_PGSQL_FORMAT_TEXT, 1 mydata, mydata_len, KORE_PGSQL_FORMAT_TEXT); kore_pgsql_query_params() is variadic, allowing you to pass any count of parameters where each parameter has the following: data pointer, data length, type of parameter.
2014-09-28 21:39:16 +02:00
static void pgsql_queue_wakeup(void);
pgsql: don't do a PQcancel() if it's not needed.
2017-02-06 20:01:16 +01:00
static void pgsql_cancel(struct kore_pgsql *);
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
static void pgsql_set_error(struct kore_pgsql *, const char *);
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
static void pgsql_queue_add(struct kore_pgsql *);
static void pgsql_queue_remove(struct kore_pgsql *);
Properly release pgsql connection if needed
2014-08-14 14:46:21 +02:00
static void pgsql_conn_release(struct kore_pgsql *);
Layout that stuff like a 80 column terminal nerd. And mark a request as DONE when we get PGRES_COMMAND_OK.
2014-03-31 01:04:22 +02:00
static void pgsql_conn_cleanup(struct pgsql_conn *);
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
static void pgsql_read_result(struct kore_pgsql *);
static void pgsql_schedule(struct kore_pgsql *);
static struct pgsql_conn *pgsql_conn_create(struct kore_pgsql *,
struct pgsql_db *);
static struct pgsql_conn *pgsql_conn_next(struct kore_pgsql *,
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
struct pgsql_db *);
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
Introduce a wait queue for our pgsql code. Instead of letting http_requests spin, if we cannot allocate a connection for the request we will queue them up put them to sleep. When a connection becomes available, we'll wake up a request that was waiting for a connection and let it continue. This completely avoids consuming massive amounts of cpu time when dealing with thousands of requests waiting for a pgsql worker to become ready.
2014-08-21 16:36:12 +02:00
static struct kore_pool pgsql_job_pool;
static struct kore_pool pgsql_wait_pool;
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
static TAILQ_HEAD(, pgsql_conn) pgsql_conn_free;
Introduce a wait queue for our pgsql code. Instead of letting http_requests spin, if we cannot allocate a connection for the request we will queue them up put them to sleep. When a connection becomes available, we'll wake up a request that was waiting for a connection and let it continue. This completely avoids consuming massive amounts of cpu time when dealing with thousands of requests waiting for a pgsql worker to become ready.
2014-08-21 16:36:12 +02:00
static TAILQ_HEAD(, pgsql_wait) pgsql_wait_queue;
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
static LIST_HEAD(, pgsql_db) pgsql_db_conn_strings;
Several postgresql improvements. - Make pgsql_conn_count count per database rather then globally. This means you now define the number of clients *per* database registered rather then the number of clients in total of all databases. - In case a connection is in failed transaction state Kore will now automatically rollback the transaction before placing that connection back in the connection pool.
2017-07-11 15:11:13 +02:00
Add pgsql_queue_limit configuration option. Limits the number of queued asynchronous queries kore will allow before starting to return errors for KORE_PGSQL_ASYNC setups. By default set to 1000. Avoids uncontrolled growth of the pgsql_queue_wait pool.
2018-02-13 13:21:27 +01:00
u_int32_t pgsql_queue_count = 0;
u_int16_t pgsql_conn_max = PGSQL_CONN_MAX;
u_int32_t pgsql_queue_limit = PGSQL_QUEUE_LIMIT;
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
void
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
kore_pgsql_sys_init(void)
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
{
TAILQ_INIT(&pgsql_conn_free);
Introduce a wait queue for our pgsql code. Instead of letting http_requests spin, if we cannot allocate a connection for the request we will queue them up put them to sleep. When a connection becomes available, we'll wake up a request that was waiting for a connection and let it continue. This completely avoids consuming massive amounts of cpu time when dealing with thousands of requests waiting for a pgsql worker to become ready.
2014-08-21 16:36:12 +02:00
TAILQ_INIT(&pgsql_wait_queue);
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
LIST_INIT(&pgsql_db_conn_strings);
Introduce a wait queue for our pgsql code. Instead of letting http_requests spin, if we cannot allocate a connection for the request we will queue them up put them to sleep. When a connection becomes available, we'll wake up a request that was waiting for a connection and let it continue. This completely avoids consuming massive amounts of cpu time when dealing with thousands of requests waiting for a pgsql worker to become ready.
2014-08-21 16:36:12 +02:00
kore_pool_init(&pgsql_job_pool, "pgsql_job_pool",
sizeof(struct pgsql_job), 100);
kore_pool_init(&pgsql_wait_pool, "pgsql_wait_pool",
Add pgsql_queue_limit configuration option. Limits the number of queued asynchronous queries kore will allow before starting to return errors for KORE_PGSQL_ASYNC setups. By default set to 1000. Avoids uncontrolled growth of the pgsql_queue_wait pool.
2018-02-13 13:21:27 +01:00
sizeof(struct pgsql_wait), pgsql_queue_limit);
Add seccomp syscall filtering to kore. With this commit all Kore processes (minus the parent) are running under seccomp. The worker processes get the bare minimum allowed syscalls while each module like curl, pgsql, etc will add their own filters to allow what they require. New API functions: int kore_seccomp_filter(const char *name, void *filter, size_t len); Adds a filter into the seccomp system (must be called before seccomp is enabled). New helpful macro: define KORE_SYSCALL_ALLOW(name) Allow the syscall with a given name, should be used in a sock_filter data structure. New hooks: void kore_seccomp_hook(void); Called before seccomp is enabled, allows developers to add their own BPF filters into seccomp.
2019-09-25 12:25:49 +00:00
#if defined(__linux__)
kore_seccomp_filter("pgsql", filter_pgsql,
KORE_FILTER_LEN(filter_pgsql));
#endif
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
}
pgsql improvements. - adds new cleanup function that workers will call. - adds kore_pgsql_nfields() to return number of fields in result. - add kore_pgsql_fieldname() to return name of a given field. This commit also changes the behaviour of pgsql_conn_release() in that it will now cancel the active query before releasing the connection. This makes sure that if long running queries are active they are hopefully cancelled if an http request is removed while such queries are still running.
2017-02-06 11:40:33 +01:00
void
kore_pgsql_sys_cleanup(void)
{
struct pgsql_conn *conn, *next;
kore_pool_cleanup(&pgsql_job_pool);
kore_pool_cleanup(&pgsql_wait_pool);
Revert "TAILQ_FOREACH_SAFE() exists so use it." Because some asshole distributions claim to have a sane queue.h implementation while they do not.
2017-02-07 22:44:20 +01:00
for (conn = TAILQ_FIRST(&pgsql_conn_free); conn != NULL; conn = next) {
next = TAILQ_NEXT(conn, list);
pgsql improvements. - adds new cleanup function that workers will call. - adds kore_pgsql_nfields() to return number of fields in result. - add kore_pgsql_fieldname() to return name of a given field. This commit also changes the behaviour of pgsql_conn_release() in that it will now cancel the active query before releasing the connection. This makes sure that if long running queries are active they are hopefully cancelled if an http request is removed while such queries are still running.
2017-02-06 11:40:33 +01:00
pgsql_conn_cleanup(conn);
}
}
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
void
kore_pgsql_init(struct kore_pgsql *pgsql)
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
{
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
memset(pgsql, 0, sizeof(*pgsql));
pgsql->state = KORE_PGSQL_STATE_INIT;
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
}
int
kore_pgsql_setup(struct kore_pgsql *pgsql, const char *dbname, int flags)
{
struct pgsql_db *db;
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
if ((flags & KORE_PGSQL_ASYNC) && (flags & KORE_PGSQL_SYNC)) {
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
pgsql_set_error(pgsql, "invalid query init parameters");
Add kore_pgsql_query_params(). This function uses PQsendQueryParams() instead of the normal PQsendQuery() allowing you to pass binary data in a cleaner fashion. A basic call would look something like: char *mydata = "Hello"; size_t mydata_len = strlen(mydata); kore_pgsql_query_params(&pgsql, req, "INSERT INTO foo VALUES($1::text)", KORE_PGSQL_FORMAT_TEXT, 1 mydata, mydata_len, KORE_PGSQL_FORMAT_TEXT); kore_pgsql_query_params() is variadic, allowing you to pass any count of parameters where each parameter has the following: data pointer, data length, type of parameter.
2014-09-28 21:39:16 +02:00
return (KORE_RESULT_ERROR);
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
}
Convert pgsql-sync example after pgsql changes. Only check if we bound something if we're asynchronous.
2017-03-24 13:00:05 +01:00
if (flags & KORE_PGSQL_ASYNC) {
if (pgsql->req == NULL && pgsql->cb == NULL) {
pgsql_set_error(pgsql, "nothing was bound");
return (KORE_RESULT_ERROR);
}
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
}
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
db = NULL;
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
pgsql->flags |= flags;
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
LIST_FOREACH(db, &pgsql_db_conn_strings, rlist) {
if (!strcmp(db->name, dbname))
break;
}
if (db == NULL) {
pgsql_set_error(pgsql, "no database found");
return (KORE_RESULT_ERROR);
}
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
if ((pgsql->conn = pgsql_conn_next(pgsql, db)) == NULL)
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
return (KORE_RESULT_ERROR);
if (pgsql->flags & KORE_PGSQL_ASYNC) {
pgsql->conn->job = kore_pool_get(&pgsql_job_pool);
pgsql->conn->job->pgsql = pgsql;
}
return (KORE_RESULT_OK);
}
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
#if !defined(KORE_NO_HTTP)
void
kore_pgsql_bind_request(struct kore_pgsql *pgsql, struct http_request *req)
{
if (pgsql->req != NULL || pgsql->cb != NULL)
fatal("kore_pgsql_bind_request: already bound");
pgsql->req = req;
pgsql->flags |= PGSQL_LIST_INSERTED;
LIST_INSERT_HEAD(&(req->pgsqls), pgsql, rlist);
}
#endif
void
kore_pgsql_bind_callback(struct kore_pgsql *pgsql,
void (*cb)(struct kore_pgsql *, void *), void *arg)
{
if (pgsql->req != NULL)
fatal("kore_pgsql_bind_callback: already bound");
if (pgsql->cb != NULL)
fatal("kore_pgsql_bind_callback: already bound");
pgsql->cb = cb;
pgsql->arg = arg;
}
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
int
Some C pgsql api improvements.
2019-06-01 23:14:50 +02:00
kore_pgsql_query(struct kore_pgsql *pgsql, const void *query)
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
{
if (pgsql->conn == NULL) {
pgsql_set_error(pgsql, "no connection was set before query");
Add kore_pgsql_query_params(). This function uses PQsendQueryParams() instead of the normal PQsendQuery() allowing you to pass binary data in a cleaner fashion. A basic call would look something like: char *mydata = "Hello"; size_t mydata_len = strlen(mydata); kore_pgsql_query_params(&pgsql, req, "INSERT INTO foo VALUES($1::text)", KORE_PGSQL_FORMAT_TEXT, 1 mydata, mydata_len, KORE_PGSQL_FORMAT_TEXT); kore_pgsql_query_params() is variadic, allowing you to pass any count of parameters where each parameter has the following: data pointer, data length, type of parameter.
2014-09-28 21:39:16 +02:00
return (KORE_RESULT_ERROR);
}
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
if (pgsql->flags & KORE_PGSQL_SYNC) {
pgsql->result = PQexec(pgsql->conn->db, query);
if ((PQresultStatus(pgsql->result) != PGRES_TUPLES_OK) &&
(PQresultStatus(pgsql->result) != PGRES_COMMAND_OK)) {
pgsql_set_error(pgsql, PQerrorMessage(pgsql->conn->db));
return (KORE_RESULT_ERROR);
}
pgsql->state = KORE_PGSQL_STATE_DONE;
} else {
if (!PQsendQuery(pgsql->conn->db, query)) {
pgsql_set_error(pgsql, PQerrorMessage(pgsql->conn->db));
return (KORE_RESULT_ERROR);
}
pgsql_schedule(pgsql);
}
Add kore_pgsql_query_params(). This function uses PQsendQueryParams() instead of the normal PQsendQuery() allowing you to pass binary data in a cleaner fashion. A basic call would look something like: char *mydata = "Hello"; size_t mydata_len = strlen(mydata); kore_pgsql_query_params(&pgsql, req, "INSERT INTO foo VALUES($1::text)", KORE_PGSQL_FORMAT_TEXT, 1 mydata, mydata_len, KORE_PGSQL_FORMAT_TEXT); kore_pgsql_query_params() is variadic, allowing you to pass any count of parameters where each parameter has the following: data pointer, data length, type of parameter.
2014-09-28 21:39:16 +02:00
return (KORE_RESULT_OK);
}
Introduce a wait queue for our pgsql code. Instead of letting http_requests spin, if we cannot allocate a connection for the request we will queue them up put them to sleep. When a connection becomes available, we'll wake up a request that was waiting for a connection and let it continue. This completely avoids consuming massive amounts of cpu time when dealing with thousands of requests waiting for a pgsql worker to become ready.
2014-08-21 16:36:12 +02:00
Add kore_pgsql_query_params(). This function uses PQsendQueryParams() instead of the normal PQsendQuery() allowing you to pass binary data in a cleaner fashion. A basic call would look something like: char *mydata = "Hello"; size_t mydata_len = strlen(mydata); kore_pgsql_query_params(&pgsql, req, "INSERT INTO foo VALUES($1::text)", KORE_PGSQL_FORMAT_TEXT, 1 mydata, mydata_len, KORE_PGSQL_FORMAT_TEXT); kore_pgsql_query_params() is variadic, allowing you to pass any count of parameters where each parameter has the following: data pointer, data length, type of parameter.
2014-09-28 21:39:16 +02:00
int
Added new function kore_pgsql_v_query_params(). Same as kore_pgsql_query_params but takes a va_list as last parameter (non-v version takes a variable list of parameters). Lets people write easier to call wrappers around the query calls. I use it in a wrapper that takes next states (error, current, continue) as arguments in a handler with multiple async queries.
2016-06-07 13:12:31 +02:00
kore_pgsql_v_query_params(struct kore_pgsql *pgsql,
Some C pgsql api improvements.
2019-06-01 23:14:50 +02:00
const void *query, int binary, int count, va_list args)
Add kore_pgsql_query_params(). This function uses PQsendQueryParams() instead of the normal PQsendQuery() allowing you to pass binary data in a cleaner fashion. A basic call would look something like: char *mydata = "Hello"; size_t mydata_len = strlen(mydata); kore_pgsql_query_params(&pgsql, req, "INSERT INTO foo VALUES($1::text)", KORE_PGSQL_FORMAT_TEXT, 1 mydata, mydata_len, KORE_PGSQL_FORMAT_TEXT); kore_pgsql_query_params() is variadic, allowing you to pass any count of parameters where each parameter has the following: data pointer, data length, type of parameter.
2014-09-28 21:39:16 +02:00
{
Improve pgsql support. - Add kore_pgsql_query_param_fields() which allows you to pass in the arrays for values, lengths and formats yourself. - Add kore_pgsql_column_binary() which will return 1 if the given column index contains a binary result or 0 if it contains a text result. - Change the query call in req.pgsql() for Python to always use the parameterized queries. This adds the 'params' and 'binary' keywords to the req.pgsql method. Eg: result = await req.pgsql("db", "INSERT INTO foo (field) VALUES($1"), params=["this is my value"])
2019-04-25 23:13:13 +02:00
int i;
const char **values;
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
int *lengths, *formats, ret;
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
if (pgsql->conn == NULL) {
pgsql_set_error(pgsql, "no connection was set before query");
Add kore_pgsql_query_params(). This function uses PQsendQueryParams() instead of the normal PQsendQuery() allowing you to pass binary data in a cleaner fashion. A basic call would look something like: char *mydata = "Hello"; size_t mydata_len = strlen(mydata); kore_pgsql_query_params(&pgsql, req, "INSERT INTO foo VALUES($1::text)", KORE_PGSQL_FORMAT_TEXT, 1 mydata, mydata_len, KORE_PGSQL_FORMAT_TEXT); kore_pgsql_query_params() is variadic, allowing you to pass any count of parameters where each parameter has the following: data pointer, data length, type of parameter.
2014-09-28 21:39:16 +02:00
return (KORE_RESULT_ERROR);
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
}
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
Allow 0 parameters for kore_pgsql_query_params()
2014-10-12 01:17:35 +02:00
if (count > 0) {
lengths = kore_calloc(count, sizeof(int));
formats = kore_calloc(count, sizeof(int));
values = kore_calloc(count, sizeof(char *));
for (i = 0; i < count; i++) {
values[i] = va_arg(args, void *);
fix count in kore_pgsql_query_params(). should fix #159.
2016-12-27 07:42:42 +01:00
lengths[i] = va_arg(args, int);
Allow 0 parameters for kore_pgsql_query_params()
2014-10-12 01:17:35 +02:00
formats[i] = va_arg(args, int);
}
} else {
lengths = NULL;
formats = NULL;
values = NULL;
Add kore_pgsql_query_params(). This function uses PQsendQueryParams() instead of the normal PQsendQuery() allowing you to pass binary data in a cleaner fashion. A basic call would look something like: char *mydata = "Hello"; size_t mydata_len = strlen(mydata); kore_pgsql_query_params(&pgsql, req, "INSERT INTO foo VALUES($1::text)", KORE_PGSQL_FORMAT_TEXT, 1 mydata, mydata_len, KORE_PGSQL_FORMAT_TEXT); kore_pgsql_query_params() is variadic, allowing you to pass any count of parameters where each parameter has the following: data pointer, data length, type of parameter.
2014-09-28 21:39:16 +02:00
}
Attach tasks/pgsqls to http_requests once more. This way if an http request is removed while tasks or pgsqls are still active they are free'd out and cancelled properly.
2014-08-14 22:05:34 +02:00
Improve pgsql support. - Add kore_pgsql_query_param_fields() which allows you to pass in the arrays for values, lengths and formats yourself. - Add kore_pgsql_column_binary() which will return 1 if the given column index contains a binary result or 0 if it contains a text result. - Change the query call in req.pgsql() for Python to always use the parameterized queries. This adds the 'params' and 'binary' keywords to the req.pgsql method. Eg: result = await req.pgsql("db", "INSERT INTO foo (field) VALUES($1"), params=["this is my value"])
2019-04-25 23:13:13 +02:00
ret = kore_pgsql_query_param_fields(pgsql, query, binary, count,
values, lengths, formats);
kore_free(values);
kore_free(lengths);
kore_free(formats);
return (ret);
}
int
Some C pgsql api improvements.
2019-06-01 23:14:50 +02:00
kore_pgsql_query_param_fields(struct kore_pgsql *pgsql, const void *query,
Improve pgsql support. - Add kore_pgsql_query_param_fields() which allows you to pass in the arrays for values, lengths and formats yourself. - Add kore_pgsql_column_binary() which will return 1 if the given column index contains a binary result or 0 if it contains a text result. - Change the query call in req.pgsql() for Python to always use the parameterized queries. This adds the 'params' and 'binary' keywords to the req.pgsql method. Eg: result = await req.pgsql("db", "INSERT INTO foo (field) VALUES($1"), params=["this is my value"])
2019-04-25 23:13:13 +02:00
int binary, int count, const char **values, int *lengths, int *formats)
{
if (pgsql->conn == NULL) {
pgsql_set_error(pgsql, "no connection was set before query");
return (KORE_RESULT_ERROR);
}
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
if (pgsql->flags & KORE_PGSQL_SYNC) {
pgsql->result = PQexecParams(pgsql->conn->db, query, count,
NULL, (const char * const *)values, lengths, formats,
Improve pgsql support. - Add kore_pgsql_query_param_fields() which allows you to pass in the arrays for values, lengths and formats yourself. - Add kore_pgsql_column_binary() which will return 1 if the given column index contains a binary result or 0 if it contains a text result. - Change the query call in req.pgsql() for Python to always use the parameterized queries. This adds the 'params' and 'binary' keywords to the req.pgsql method. Eg: result = await req.pgsql("db", "INSERT INTO foo (field) VALUES($1"), params=["this is my value"])
2019-04-25 23:13:13 +02:00
binary);
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
if ((PQresultStatus(pgsql->result) != PGRES_TUPLES_OK) &&
(PQresultStatus(pgsql->result) != PGRES_COMMAND_OK)) {
pgsql_set_error(pgsql, PQerrorMessage(pgsql->conn->db));
Improve pgsql support. - Add kore_pgsql_query_param_fields() which allows you to pass in the arrays for values, lengths and formats yourself. - Add kore_pgsql_column_binary() which will return 1 if the given column index contains a binary result or 0 if it contains a text result. - Change the query call in req.pgsql() for Python to always use the parameterized queries. This adds the 'params' and 'binary' keywords to the req.pgsql method. Eg: result = await req.pgsql("db", "INSERT INTO foo (field) VALUES($1"), params=["this is my value"])
2019-04-25 23:13:13 +02:00
return (KORE_RESULT_ERROR);
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
}
pgsql->state = KORE_PGSQL_STATE_DONE;
} else {
if (!PQsendQueryParams(pgsql->conn->db, query, count, NULL,
Improve pgsql support. - Add kore_pgsql_query_param_fields() which allows you to pass in the arrays for values, lengths and formats yourself. - Add kore_pgsql_column_binary() which will return 1 if the given column index contains a binary result or 0 if it contains a text result. - Change the query call in req.pgsql() for Python to always use the parameterized queries. This adds the 'params' and 'binary' keywords to the req.pgsql method. Eg: result = await req.pgsql("db", "INSERT INTO foo (field) VALUES($1"), params=["this is my value"])
2019-04-25 23:13:13 +02:00
(const char * const *)values, lengths, formats, binary)) {
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
pgsql_set_error(pgsql, PQerrorMessage(pgsql->conn->db));
Improve pgsql support. - Add kore_pgsql_query_param_fields() which allows you to pass in the arrays for values, lengths and formats yourself. - Add kore_pgsql_column_binary() which will return 1 if the given column index contains a binary result or 0 if it contains a text result. - Change the query call in req.pgsql() for Python to always use the parameterized queries. This adds the 'params' and 'binary' keywords to the req.pgsql method. Eg: result = await req.pgsql("db", "INSERT INTO foo (field) VALUES($1"), params=["this is my value"])
2019-04-25 23:13:13 +02:00
return (KORE_RESULT_ERROR);
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
}
pgsql_schedule(pgsql);
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
}
Improve pgsql support. - Add kore_pgsql_query_param_fields() which allows you to pass in the arrays for values, lengths and formats yourself. - Add kore_pgsql_column_binary() which will return 1 if the given column index contains a binary result or 0 if it contains a text result. - Change the query call in req.pgsql() for Python to always use the parameterized queries. This adds the 'params' and 'binary' keywords to the req.pgsql method. Eg: result = await req.pgsql("db", "INSERT INTO foo (field) VALUES($1"), params=["this is my value"])
2019-04-25 23:13:13 +02:00
return (KORE_RESULT_OK);
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
}
Added new function kore_pgsql_v_query_params(). Same as kore_pgsql_query_params but takes a va_list as last parameter (non-v version takes a variable list of parameters). Lets people write easier to call wrappers around the query calls. I use it in a wrapper that takes next states (error, current, continue) as arguments in a handler with multiple async queries.
2016-06-07 13:12:31 +02:00
int
kore_pgsql_query_params(struct kore_pgsql *pgsql,
Some C pgsql api improvements.
2019-06-01 23:14:50 +02:00
const void *query, int binary, int count, ...)
Added new function kore_pgsql_v_query_params(). Same as kore_pgsql_query_params but takes a va_list as last parameter (non-v version takes a variable list of parameters). Lets people write easier to call wrappers around the query calls. I use it in a wrapper that takes next states (error, current, continue) as arguments in a handler with multiple async queries.
2016-06-07 13:12:31 +02:00
{
int ret;
va_list args;
Fix #144. Always initialize va_list in kore_pgsql_query_params. (#146)
2016-09-05 16:22:46 +02:00
va_start(args, count);
Improve pgsql support. - Add kore_pgsql_query_param_fields() which allows you to pass in the arrays for values, lengths and formats yourself. - Add kore_pgsql_column_binary() which will return 1 if the given column index contains a binary result or 0 if it contains a text result. - Change the query call in req.pgsql() for Python to always use the parameterized queries. This adds the 'params' and 'binary' keywords to the req.pgsql method. Eg: result = await req.pgsql("db", "INSERT INTO foo (field) VALUES($1"), params=["this is my value"])
2019-04-25 23:13:13 +02:00
ret = kore_pgsql_v_query_params(pgsql, query, binary, count, args);
Fix #144. Always initialize va_list in kore_pgsql_query_params. (#146)
2016-09-05 16:22:46 +02:00
va_end(args);
Added new function kore_pgsql_v_query_params(). Same as kore_pgsql_query_params but takes a va_list as last parameter (non-v version takes a variable list of parameters). Lets people write easier to call wrappers around the query calls. I use it in a wrapper that takes next states (error, current, continue) as arguments in a handler with multiple async queries.
2016-06-07 13:12:31 +02:00
return (ret);
}
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
int
kore_pgsql_register(const char *dbname, const char *connstring)
{
struct pgsql_db *pgsqldb;
LIST_FOREACH(pgsqldb, &pgsql_db_conn_strings, rlist) {
if (!strcmp(pgsqldb->name, dbname))
return (KORE_RESULT_ERROR);
}
pgsqldb = kore_malloc(sizeof(*pgsqldb));
pgsqldb->name = kore_strdup(dbname);
small pgsql fixes. - make sure conn_count per pgsqldb structure is initialized to 0. - allow pgsql_conn_max to be 0, meaning just create a new connection if none was free.
2017-08-21 14:25:09 +02:00
pgsqldb->conn_count = 0;
pgsqldb->conn_max = pgsql_conn_max;
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
pgsqldb->conn_string = kore_strdup(connstring);
LIST_INSERT_HEAD(&pgsql_db_conn_strings, pgsqldb, rlist);
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
return (KORE_RESULT_OK);
}
void
kore_pgsql_handle(void *c, int err)
{
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
struct kore_pgsql *pgsql;
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
struct pgsql_conn *conn = (struct pgsql_conn *)c;
Improvements to pgsql contrib code. Including but not limited to: - Correctly use PQerrorMessage() in case we cleanup with PQfinish - If we get a network error, cleanup the connection - No longer call the page handler from inside kore_pgsql_handle() but instead just put it to sleep in case we don't need it. This does grow the http_requests list quite a bit with sleeping connections and can perhaps be improved later on. - Allow us to on error return OK from a page handler from inside the completetion block for KORE_PGSQL(). - Count the cummulative time for a request to finish instead of the latest run time for the handler.
2014-03-31 00:57:00 +02:00
if (err) {
pgsql_conn_cleanup(conn);
return;
}
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
Changes to the event loop inside of Kore. Now anyone can schedule events and get a callback to work as long as the user data structure that is added for the event begins with a kore_event data structure. All event state is now kept in that kore_event structure and renamed CONN_[READ|WRITE]_POSSIBLE to KORE_EVENT_[READ|WRITE].
2018-10-09 19:34:40 +02:00
if (!(conn->evt.flags & KORE_EVENT_READ))
fatal("%s: read event not set", __func__);
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
pgsql = conn->job->pgsql;
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
pgsql improvements round 2 - kill PQsetnonblocking(), it's not what one thinks. - keep doing the PQconsumeInput()/PQisBusy() dance until the former clears all read data.
2019-09-13 23:20:51 +02:00
pgsql_read_result(pgsql);
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
if (pgsql->state == KORE_PGSQL_STATE_WAIT) {
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
#if !defined(KORE_NO_HTTP)
if (pgsql->req != NULL)
http_request_sleep(pgsql->req);
#endif
if (pgsql->cb != NULL)
pgsql->cb(pgsql, pgsql->arg);
Kore pgsql improvements. Don't wait for a full event loop until we call the page handler for a received pgsql result. This speeds up page loads using KORE_PGSQL by quite a lot, especially on a non busy server.
2014-04-17 10:49:48 +02:00
} else {
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
#if !defined(KORE_NO_HTTP)
if (pgsql->req != NULL)
http_request_wakeup(pgsql->req);
#endif
if (pgsql->cb != NULL)
pgsql->cb(pgsql, pgsql->arg);
Kore pgsql improvements. Don't wait for a full event loop until we call the page handler for a received pgsql result. This speeds up page loads using KORE_PGSQL by quite a lot, especially on a non busy server.
2014-04-17 10:49:48 +02:00
}
Improvements to pgsql contrib code. Including but not limited to: - Correctly use PQerrorMessage() in case we cleanup with PQfinish - If we get a network error, cleanup the connection - No longer call the page handler from inside kore_pgsql_handle() but instead just put it to sleep in case we don't need it. This does grow the http_requests list quite a bit with sleeping connections and can perhaps be improved later on. - Allow us to on error return OK from a page handler from inside the completetion block for KORE_PGSQL(). - Count the cummulative time for a request to finish instead of the latest run time for the handler.
2014-03-31 00:57:00 +02:00
}
void
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
kore_pgsql_continue(struct kore_pgsql *pgsql)
Improvements to pgsql contrib code. Including but not limited to: - Correctly use PQerrorMessage() in case we cleanup with PQfinish - If we get a network error, cleanup the connection - No longer call the page handler from inside kore_pgsql_handle() but instead just put it to sleep in case we don't need it. This does grow the http_requests list quite a bit with sleeping connections and can perhaps be improved later on. - Allow us to on error return OK from a page handler from inside the completetion block for KORE_PGSQL(). - Count the cummulative time for a request to finish instead of the latest run time for the handler.
2014-03-31 00:57:00 +02:00
{
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
if (pgsql->error) {
Large changes to the memory subsystem in kore. - Change pools to use mmap() for allocating regions. - Change kore_malloc() to use pools for commonly sized objects. (split into multiple of 2 buckets, starting at 8 bytes up to 8192). - Rename kore_mem_free() to kore_free(). The preallocated pools will hold up to 128K of elements per block size. In case a larger object is to be allocated kore_malloc() will use malloc() instead.
2016-07-12 13:54:14 +02:00
kore_free(pgsql->error);
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
pgsql->error = NULL;
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
}
pgsql improvements * set pgsql->result to NULL when we freed it * drain the connection pipe just in case in pgsql_conn_release()
2014-08-14 16:43:40 +02:00
if (pgsql->result) {
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
PQclear(pgsql->result);
pgsql improvements * set pgsql->result to NULL when we freed it * drain the connection pipe just in case in pgsql_conn_release()
2014-08-14 16:43:40 +02:00
pgsql->result = NULL;
}
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
switch (pgsql->state) {
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
case KORE_PGSQL_STATE_INIT:
case KORE_PGSQL_STATE_WAIT:
break;
case KORE_PGSQL_STATE_DONE:
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
#if !defined(KORE_NO_HTTP)
if (pgsql->req != NULL)
http_request_wakeup(pgsql->req);
#endif
Properly release pgsql connection if needed
2014-08-14 14:46:21 +02:00
pgsql_conn_release(pgsql);
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
break;
case KORE_PGSQL_STATE_ERROR:
case KORE_PGSQL_STATE_RESULT:
Add KORE_PGSQL_STATE_NOTIFY. Issue a LISTEN channel on a kore_pgsql, bind a callback to it and you will get called with pgsql->state being KORE_PGSQL_STATE_NOTIFY.
2018-07-18 11:38:17 +02:00
case KORE_PGSQL_STATE_NOTIFY:
Properly release pgsql connection if needed
2014-08-14 14:46:21 +02:00
kore_pgsql_handle(pgsql->conn, 0);
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
break;
default:
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
fatal("unknown pgsql state %d", pgsql->state);
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
}
}
void
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
kore_pgsql_cleanup(struct kore_pgsql *pgsql)
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
{
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
pgsql_queue_remove(pgsql);
Properly release pgsql connection if needed
2014-08-14 14:46:21 +02:00
if (pgsql->result != NULL)
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
PQclear(pgsql->result);
Improvements to pgsql contrib code. Including but not limited to: - Correctly use PQerrorMessage() in case we cleanup with PQfinish - If we get a network error, cleanup the connection - No longer call the page handler from inside kore_pgsql_handle() but instead just put it to sleep in case we don't need it. This does grow the http_requests list quite a bit with sleeping connections and can perhaps be improved later on. - Allow us to on error return OK from a page handler from inside the completetion block for KORE_PGSQL(). - Count the cummulative time for a request to finish instead of the latest run time for the handler.
2014-03-31 00:57:00 +02:00
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
if (pgsql->error != NULL)
Large changes to the memory subsystem in kore. - Change pools to use mmap() for allocating regions. - Change kore_malloc() to use pools for commonly sized objects. (split into multiple of 2 buckets, starting at 8 bytes up to 8192). - Rename kore_mem_free() to kore_free(). The preallocated pools will hold up to 128K of elements per block size. In case a larger object is to be allocated kore_malloc() will use malloc() instead.
2016-07-12 13:54:14 +02:00
kore_free(pgsql->error);
pgsql_conn improvements. set conn inside kore_pgsql to NULL when we've completed. if at kore_pgsql_cleanup() time the conn is still there we'll drain it as per documentation.
2014-03-31 01:09:08 +02:00
Introduce a wait queue for our pgsql code. Instead of letting http_requests spin, if we cannot allocate a connection for the request we will queue them up put them to sleep. When a connection becomes available, we'll wake up a request that was waiting for a connection and let it continue. This completely avoids consuming massive amounts of cpu time when dealing with thousands of requests waiting for a pgsql worker to become ready.
2014-08-21 16:36:12 +02:00
if (pgsql->conn != NULL)
Properly release pgsql connection if needed
2014-08-14 14:46:21 +02:00
pgsql_conn_release(pgsql);
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
pgsql->result = NULL;
pgsql->error = NULL;
pgsql->conn = NULL;
Attach tasks/pgsqls to http_requests once more. This way if an http request is removed while tasks or pgsqls are still active they are free'd out and cancelled properly.
2014-08-14 22:05:34 +02:00
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
if (pgsql->flags & PGSQL_LIST_INSERTED) {
LIST_REMOVE(pgsql, rlist);
pgsql->flags &= ~PGSQL_LIST_INSERTED;
}
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
}
amend example with additional query + kore_pgsql_logerror()
2014-04-02 23:01:47 +02:00
void
kore_pgsql_logerror(struct kore_pgsql *pgsql)
{
kore_log(LOG_NOTICE, "pgsql error: %s",
(pgsql->error) ? pgsql->error : "unknown");
}
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
int
pgsql fixes + new wrappers
2014-04-01 21:43:32 +02:00
kore_pgsql_ntuples(struct kore_pgsql *pgsql)
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
{
pgsql fixes + new wrappers
2014-04-01 21:43:32 +02:00
return (PQntuples(pgsql->result));
}
pgsql improvements. - adds new cleanup function that workers will call. - adds kore_pgsql_nfields() to return number of fields in result. - add kore_pgsql_fieldname() to return name of a given field. This commit also changes the behaviour of pgsql_conn_release() in that it will now cancel the active query before releasing the connection. This makes sure that if long running queries are active they are hopefully cancelled if an http request is removed while such queries are still running.
2017-02-06 11:40:33 +01:00
int
kore_pgsql_nfields(struct kore_pgsql *pgsql)
{
return (PQnfields(pgsql->result));
}
Add kore_pgsql_getlength() which will return length of a column.
2014-09-28 23:03:49 +02:00
int
kore_pgsql_getlength(struct kore_pgsql *pgsql, int row, int col)
{
return (PQgetlength(pgsql->result, row, col));
}
pgsql improvements. - adds new cleanup function that workers will call. - adds kore_pgsql_nfields() to return number of fields in result. - add kore_pgsql_fieldname() to return name of a given field. This commit also changes the behaviour of pgsql_conn_release() in that it will now cancel the active query before releasing the connection. This makes sure that if long running queries are active they are hopefully cancelled if an http request is removed while such queries are still running.
2017-02-06 11:40:33 +01:00
char *
kore_pgsql_fieldname(struct kore_pgsql *pgsql, int field)
{
return (PQfname(pgsql->result, field));
}
pgsql fixes + new wrappers
2014-04-01 21:43:32 +02:00
char *
kore_pgsql_getvalue(struct kore_pgsql *pgsql, int row, int col)
{
return (PQgetvalue(pgsql->result, row, col));
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
}
Improve pgsql support. - Add kore_pgsql_query_param_fields() which allows you to pass in the arrays for values, lengths and formats yourself. - Add kore_pgsql_column_binary() which will return 1 if the given column index contains a binary result or 0 if it contains a text result. - Change the query call in req.pgsql() for Python to always use the parameterized queries. This adds the 'params' and 'binary' keywords to the req.pgsql method. Eg: result = await req.pgsql("db", "INSERT INTO foo (field) VALUES($1"), params=["this is my value"])
2019-04-25 23:13:13 +02:00
int
kore_pgsql_column_binary(struct kore_pgsql *pgsql, int col)
{
return (PQfformat(pgsql->result, col));
}
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
static struct pgsql_conn *
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
pgsql_conn_next(struct kore_pgsql *pgsql, struct pgsql_db *db)
Add kore_pgsql_query_params(). This function uses PQsendQueryParams() instead of the normal PQsendQuery() allowing you to pass binary data in a cleaner fashion. A basic call would look something like: char *mydata = "Hello"; size_t mydata_len = strlen(mydata); kore_pgsql_query_params(&pgsql, req, "INSERT INTO foo VALUES($1::text)", KORE_PGSQL_FORMAT_TEXT, 1 mydata, mydata_len, KORE_PGSQL_FORMAT_TEXT); kore_pgsql_query_params() is variadic, allowing you to pass any count of parameters where each parameter has the following: data pointer, data length, type of parameter.
2014-09-28 21:39:16 +02:00
{
Several postgresql improvements. - Make pgsql_conn_count count per database rather then globally. This means you now define the number of clients *per* database registered rather then the number of clients in total of all databases. - In case a connection is in failed transaction state Kore will now automatically rollback the transaction before placing that connection back in the connection pool.
2017-07-11 15:11:13 +02:00
PGTransactionStatusType state;
struct pgsql_conn *conn;
Use a synchronous query to issue a ROLLBACK.
2018-07-28 22:28:19 +02:00
struct kore_pgsql rollback;
Add kore_pgsql_query_params(). This function uses PQsendQueryParams() instead of the normal PQsendQuery() allowing you to pass binary data in a cleaner fashion. A basic call would look something like: char *mydata = "Hello"; size_t mydata_len = strlen(mydata); kore_pgsql_query_params(&pgsql, req, "INSERT INTO foo VALUES($1::text)", KORE_PGSQL_FORMAT_TEXT, 1 mydata, mydata_len, KORE_PGSQL_FORMAT_TEXT); kore_pgsql_query_params() is variadic, allowing you to pass any count of parameters where each parameter has the following: data pointer, data length, type of parameter.
2014-09-28 21:39:16 +02:00
Several postgresql improvements. - Make pgsql_conn_count count per database rather then globally. This means you now define the number of clients *per* database registered rather then the number of clients in total of all databases. - In case a connection is in failed transaction state Kore will now automatically rollback the transaction before placing that connection back in the connection pool.
2017-07-11 15:11:13 +02:00
rescan:
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
conn = NULL;
Add kore_pgsql_query_params(). This function uses PQsendQueryParams() instead of the normal PQsendQuery() allowing you to pass binary data in a cleaner fashion. A basic call would look something like: char *mydata = "Hello"; size_t mydata_len = strlen(mydata); kore_pgsql_query_params(&pgsql, req, "INSERT INTO foo VALUES($1::text)", KORE_PGSQL_FORMAT_TEXT, 1 mydata, mydata_len, KORE_PGSQL_FORMAT_TEXT); kore_pgsql_query_params() is variadic, allowing you to pass any count of parameters where each parameter has the following: data pointer, data length, type of parameter.
2014-09-28 21:39:16 +02:00
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
TAILQ_FOREACH(conn, &pgsql_conn_free, list) {
if (!(conn->flags & PGSQL_CONN_FREE))
fatal("got a pgsql connection that was not free?");
if (!strcmp(conn->name, db->name))
break;
}
Several postgresql improvements. - Make pgsql_conn_count count per database rather then globally. This means you now define the number of clients *per* database registered rather then the number of clients in total of all databases. - In case a connection is in failed transaction state Kore will now automatically rollback the transaction before placing that connection back in the connection pool.
2017-07-11 15:11:13 +02:00
if (conn != NULL) {
state = PQtransactionStatus(conn->db);
if (state == PQTRANS_INERROR) {
conn->flags &= ~PGSQL_CONN_FREE;
TAILQ_REMOVE(&pgsql_conn_free, conn, list);
Use a synchronous query to issue a ROLLBACK.
2018-07-28 22:28:19 +02:00
kore_pgsql_init(&rollback);
rollback.conn = conn;
rollback.flags = KORE_PGSQL_SYNC;
Several postgresql improvements. - Make pgsql_conn_count count per database rather then globally. This means you now define the number of clients *per* database registered rather then the number of clients in total of all databases. - In case a connection is in failed transaction state Kore will now automatically rollback the transaction before placing that connection back in the connection pool.
2017-07-11 15:11:13 +02:00
Use a synchronous query to issue a ROLLBACK.
2018-07-28 22:28:19 +02:00
if (!kore_pgsql_query(&rollback, "ROLLBACK")) {
kore_pgsql_logerror(&rollback);
kore_pgsql_cleanup(&rollback);
Several postgresql improvements. - Make pgsql_conn_count count per database rather then globally. This means you now define the number of clients *per* database registered rather then the number of clients in total of all databases. - In case a connection is in failed transaction state Kore will now automatically rollback the transaction before placing that connection back in the connection pool.
2017-07-11 15:11:13 +02:00
pgsql_conn_cleanup(conn);
Use a synchronous query to issue a ROLLBACK.
2018-07-28 22:28:19 +02:00
} else {
kore_pgsql_cleanup(&rollback);
Several postgresql improvements. - Make pgsql_conn_count count per database rather then globally. This means you now define the number of clients *per* database registered rather then the number of clients in total of all databases. - In case a connection is in failed transaction state Kore will now automatically rollback the transaction before placing that connection back in the connection pool.
2017-07-11 15:11:13 +02:00
}
goto rescan;
}
}
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
if (conn == NULL) {
small pgsql fixes. - make sure conn_count per pgsqldb structure is initialized to 0. - allow pgsql_conn_max to be 0, meaning just create a new connection if none was free.
2017-08-21 14:25:09 +02:00
if (db->conn_max != 0 &&
db->conn_count >= db->conn_max) {
Add pgsql_queue_limit configuration option. Limits the number of queued asynchronous queries kore will allow before starting to return errors for KORE_PGSQL_ASYNC setups. By default set to 1000. Avoids uncontrolled growth of the pgsql_queue_wait pool.
2018-02-13 13:21:27 +01:00
if ((pgsql->flags & KORE_PGSQL_ASYNC) &&
pgsql_queue_count < pgsql_queue_limit) {
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
pgsql_queue_add(pgsql);
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
} else {
pgsql_set_error(pgsql,
"no available connection");
}
return (NULL);
Add kore_pgsql_query_params(). This function uses PQsendQueryParams() instead of the normal PQsendQuery() allowing you to pass binary data in a cleaner fashion. A basic call would look something like: char *mydata = "Hello"; size_t mydata_len = strlen(mydata); kore_pgsql_query_params(&pgsql, req, "INSERT INTO foo VALUES($1::text)", KORE_PGSQL_FORMAT_TEXT, 1 mydata, mydata_len, KORE_PGSQL_FORMAT_TEXT); kore_pgsql_query_params() is variadic, allowing you to pass any count of parameters where each parameter has the following: data pointer, data length, type of parameter.
2014-09-28 21:39:16 +02:00
}
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
if ((conn = pgsql_conn_create(pgsql, db)) == NULL)
return (NULL);
Add kore_pgsql_query_params(). This function uses PQsendQueryParams() instead of the normal PQsendQuery() allowing you to pass binary data in a cleaner fashion. A basic call would look something like: char *mydata = "Hello"; size_t mydata_len = strlen(mydata); kore_pgsql_query_params(&pgsql, req, "INSERT INTO foo VALUES($1::text)", KORE_PGSQL_FORMAT_TEXT, 1 mydata, mydata_len, KORE_PGSQL_FORMAT_TEXT); kore_pgsql_query_params() is variadic, allowing you to pass any count of parameters where each parameter has the following: data pointer, data length, type of parameter.
2014-09-28 21:39:16 +02:00
}
conn->flags &= ~PGSQL_CONN_FREE;
TAILQ_REMOVE(&pgsql_conn_free, conn, list);
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
return (conn);
}
Add kore_pgsql_query_params(). This function uses PQsendQueryParams() instead of the normal PQsendQuery() allowing you to pass binary data in a cleaner fashion. A basic call would look something like: char *mydata = "Hello"; size_t mydata_len = strlen(mydata); kore_pgsql_query_params(&pgsql, req, "INSERT INTO foo VALUES($1::text)", KORE_PGSQL_FORMAT_TEXT, 1 mydata, mydata_len, KORE_PGSQL_FORMAT_TEXT); kore_pgsql_query_params() is variadic, allowing you to pass any count of parameters where each parameter has the following: data pointer, data length, type of parameter.
2014-09-28 21:39:16 +02:00
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
static void
pgsql_set_error(struct kore_pgsql *pgsql, const char *msg)
{
if (pgsql->error != NULL)
Large changes to the memory subsystem in kore. - Change pools to use mmap() for allocating regions. - Change kore_malloc() to use pools for commonly sized objects. (split into multiple of 2 buckets, starting at 8 bytes up to 8192). - Rename kore_mem_free() to kore_free(). The preallocated pools will hold up to 128K of elements per block size. In case a larger object is to be allocated kore_malloc() will use malloc() instead.
2016-07-12 13:54:14 +02:00
kore_free(pgsql->error);
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
pgsql->error = kore_strdup(msg);
pgsql->state = KORE_PGSQL_STATE_ERROR;
Add kore_pgsql_query_params(). This function uses PQsendQueryParams() instead of the normal PQsendQuery() allowing you to pass binary data in a cleaner fashion. A basic call would look something like: char *mydata = "Hello"; size_t mydata_len = strlen(mydata); kore_pgsql_query_params(&pgsql, req, "INSERT INTO foo VALUES($1::text)", KORE_PGSQL_FORMAT_TEXT, 1 mydata, mydata_len, KORE_PGSQL_FORMAT_TEXT); kore_pgsql_query_params() is variadic, allowing you to pass any count of parameters where each parameter has the following: data pointer, data length, type of parameter.
2014-09-28 21:39:16 +02:00
}
static void
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
pgsql_schedule(struct kore_pgsql *pgsql)
Add kore_pgsql_query_params(). This function uses PQsendQueryParams() instead of the normal PQsendQuery() allowing you to pass binary data in a cleaner fashion. A basic call would look something like: char *mydata = "Hello"; size_t mydata_len = strlen(mydata); kore_pgsql_query_params(&pgsql, req, "INSERT INTO foo VALUES($1::text)", KORE_PGSQL_FORMAT_TEXT, 1 mydata, mydata_len, KORE_PGSQL_FORMAT_TEXT); kore_pgsql_query_params() is variadic, allowing you to pass any count of parameters where each parameter has the following: data pointer, data length, type of parameter.
2014-09-28 21:39:16 +02:00
{
int fd;
fd = PQsocket(pgsql->conn->db);
if (fd < 0)
fatal("PQsocket returned < 0 fd on open connection");
kore_platform_schedule_read(fd, pgsql->conn);
pgsql->state = KORE_PGSQL_STATE_WAIT;
only remove the pgsql if it was scheduled. fixes #172.
2017-01-29 10:34:53 +01:00
pgsql->flags |= KORE_PGSQL_SCHEDULED;
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
#if !defined(KORE_NO_HTTP)
if (pgsql->req != NULL)
http_request_sleep(pgsql->req);
#endif
if (pgsql->cb != NULL)
pgsql->cb(pgsql, pgsql->arg);
Add kore_pgsql_query_params(). This function uses PQsendQueryParams() instead of the normal PQsendQuery() allowing you to pass binary data in a cleaner fashion. A basic call would look something like: char *mydata = "Hello"; size_t mydata_len = strlen(mydata); kore_pgsql_query_params(&pgsql, req, "INSERT INTO foo VALUES($1::text)", KORE_PGSQL_FORMAT_TEXT, 1 mydata, mydata_len, KORE_PGSQL_FORMAT_TEXT); kore_pgsql_query_params() is variadic, allowing you to pass any count of parameters where each parameter has the following: data pointer, data length, type of parameter.
2014-09-28 21:39:16 +02:00
}
Introduce a wait queue for our pgsql code. Instead of letting http_requests spin, if we cannot allocate a connection for the request we will queue them up put them to sleep. When a connection becomes available, we'll wake up a request that was waiting for a connection and let it continue. This completely avoids consuming massive amounts of cpu time when dealing with thousands of requests waiting for a pgsql worker to become ready.
2014-08-21 16:36:12 +02:00
static void
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
pgsql_queue_add(struct kore_pgsql *pgsql)
Introduce a wait queue for our pgsql code. Instead of letting http_requests spin, if we cannot allocate a connection for the request we will queue them up put them to sleep. When a connection becomes available, we'll wake up a request that was waiting for a connection and let it continue. This completely avoids consuming massive amounts of cpu time when dealing with thousands of requests waiting for a pgsql worker to become ready.
2014-08-21 16:36:12 +02:00
{
struct pgsql_wait *pgw;
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
#if !defined(KORE_NO_HTTP)
if (pgsql->req != NULL)
http_request_sleep(pgsql->req);
#endif
Introduce a wait queue for our pgsql code. Instead of letting http_requests spin, if we cannot allocate a connection for the request we will queue them up put them to sleep. When a connection becomes available, we'll wake up a request that was waiting for a connection and let it continue. This completely avoids consuming massive amounts of cpu time when dealing with thousands of requests waiting for a pgsql worker to become ready.
2014-08-21 16:36:12 +02:00
pgw = kore_pool_get(&pgsql_wait_pool);
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
pgw->pgsql = pgsql;
Add pgsql_queue_limit configuration option. Limits the number of queued asynchronous queries kore will allow before starting to return errors for KORE_PGSQL_ASYNC setups. By default set to 1000. Avoids uncontrolled growth of the pgsql_queue_wait pool.
2018-02-13 13:21:27 +01:00
pgsql_queue_count++;
Introduce a wait queue for our pgsql code. Instead of letting http_requests spin, if we cannot allocate a connection for the request we will queue them up put them to sleep. When a connection becomes available, we'll wake up a request that was waiting for a connection and let it continue. This completely avoids consuming massive amounts of cpu time when dealing with thousands of requests waiting for a pgsql worker to become ready.
2014-08-21 16:36:12 +02:00
TAILQ_INSERT_TAIL(&pgsql_wait_queue, pgw, list);
}
static void
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
pgsql_queue_remove(struct kore_pgsql *pgsql)
Introduce a wait queue for our pgsql code. Instead of letting http_requests spin, if we cannot allocate a connection for the request we will queue them up put them to sleep. When a connection becomes available, we'll wake up a request that was waiting for a connection and let it continue. This completely avoids consuming massive amounts of cpu time when dealing with thousands of requests waiting for a pgsql worker to become ready.
2014-08-21 16:36:12 +02:00
{
struct pgsql_wait *pgw, *next;
Revert "TAILQ_FOREACH_SAFE() exists so use it." Because some asshole distributions claim to have a sane queue.h implementation while they do not.
2017-02-07 22:44:20 +01:00
for (pgw = TAILQ_FIRST(&pgsql_wait_queue); pgw != NULL; pgw = next) {
next = TAILQ_NEXT(pgw, list);
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
if (pgw->pgsql != pgsql)
Introduce a wait queue for our pgsql code. Instead of letting http_requests spin, if we cannot allocate a connection for the request we will queue them up put them to sleep. When a connection becomes available, we'll wake up a request that was waiting for a connection and let it continue. This completely avoids consuming massive amounts of cpu time when dealing with thousands of requests waiting for a pgsql worker to become ready.
2014-08-21 16:36:12 +02:00
continue;
Add pgsql_queue_limit configuration option. Limits the number of queued asynchronous queries kore will allow before starting to return errors for KORE_PGSQL_ASYNC setups. By default set to 1000. Avoids uncontrolled growth of the pgsql_queue_wait pool.
2018-02-13 13:21:27 +01:00
pgsql_queue_count--;
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
TAILQ_REMOVE(&pgsql_wait_queue, pgw, list);
kore_pool_put(&pgsql_wait_pool, pgw);
return;
}
}
static void
pgsql_queue_wakeup(void)
{
struct pgsql_wait *pgw, *next;
for (pgw = TAILQ_FIRST(&pgsql_wait_queue); pgw != NULL; pgw = next) {
next = TAILQ_NEXT(pgw, list);
#if !defined(KORE_NO_HTTP)
if (pgw->pgsql->req != NULL) {
if (pgw->pgsql->req->flags & HTTP_REQUEST_DELETE) {
shuffle around pgsql_queue_count.
2018-02-14 13:59:23 +01:00
pgsql_queue_count--;
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
TAILQ_REMOVE(&pgsql_wait_queue, pgw, list);
kore_pool_put(&pgsql_wait_pool, pgw);
continue;
}
http_request_wakeup(pgw->pgsql->req);
}
#endif
if (pgw->pgsql->cb != NULL)
pgw->pgsql->cb(pgw->pgsql, pgw->pgsql->arg);
Introduce a wait queue for our pgsql code. Instead of letting http_requests spin, if we cannot allocate a connection for the request we will queue them up put them to sleep. When a connection becomes available, we'll wake up a request that was waiting for a connection and let it continue. This completely avoids consuming massive amounts of cpu time when dealing with thousands of requests waiting for a pgsql worker to become ready.
2014-08-21 16:36:12 +02:00
shuffle around pgsql_queue_count.
2018-02-14 13:59:23 +01:00
pgsql_queue_count--;
Introduce a wait queue for our pgsql code. Instead of letting http_requests spin, if we cannot allocate a connection for the request we will queue them up put them to sleep. When a connection becomes available, we'll wake up a request that was waiting for a connection and let it continue. This completely avoids consuming massive amounts of cpu time when dealing with thousands of requests waiting for a pgsql worker to become ready.
2014-08-21 16:36:12 +02:00
TAILQ_REMOVE(&pgsql_wait_queue, pgw, list);
kore_pool_put(&pgsql_wait_pool, pgw);
return;
}
}
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
static struct pgsql_conn *
pgsql_conn_create(struct kore_pgsql *pgsql, struct pgsql_db *db)
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
{
struct pgsql_conn *conn;
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
if (db == NULL || db->conn_string == NULL)
Check connection string right before we connect
2014-04-14 08:45:31 +02:00
fatal("pgsql_conn_create: no connection string");
small pgsql fixes. - make sure conn_count per pgsqldb structure is initialized to 0. - allow pgsql_conn_max to be 0, meaning just create a new connection if none was free.
2017-08-21 14:25:09 +02:00
db->conn_count++;
make sure we can call pgsql_conn_cleanup() properly.
2017-01-10 15:29:03 +01:00
Changes to the event loop inside of Kore. Now anyone can schedule events and get a callback to work as long as the user data structure that is added for the event begins with a kore_event data structure. All event state is now kept in that kore_event structure and renamed CONN_[READ|WRITE]_POSSIBLE to KORE_EVENT_[READ|WRITE].
2018-10-09 19:34:40 +02:00
conn = kore_calloc(1, sizeof(*conn));
make sure we can call pgsql_conn_cleanup() properly.
2017-01-10 15:29:03 +01:00
conn->job = NULL;
conn->flags = PGSQL_CONN_FREE;
conn->name = kore_strdup(db->name);
TAILQ_INSERT_TAIL(&pgsql_conn_free, conn, list);
Changes to the event loop inside of Kore. Now anyone can schedule events and get a callback to work as long as the user data structure that is added for the event begins with a kore_event data structure. All event state is now kept in that kore_event structure and renamed CONN_[READ|WRITE]_POSSIBLE to KORE_EVENT_[READ|WRITE].
2018-10-09 19:34:40 +02:00
conn->evt.type = KORE_TYPE_PGSQL_CONN;
conn->evt.handle = kore_pgsql_handle;
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
conn->db = PQconnectdb(db->conn_string);
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
if (conn->db == NULL || (PQstatus(conn->db) != CONNECTION_OK)) {
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
pgsql_set_error(pgsql, PQerrorMessage(conn->db));
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
pgsql_conn_cleanup(conn);
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
return (NULL);
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
}
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
return (conn);
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
}
Properly release pgsql connection if needed
2014-08-14 14:46:21 +02:00
static void
pgsql_conn_release(struct kore_pgsql *pgsql)
{
int fd;
Several postgresql improvements. - Make pgsql_conn_count count per database rather then globally. This means you now define the number of clients *per* database registered rather then the number of clients in total of all databases. - In case a connection is in failed transaction state Kore will now automatically rollback the transaction before placing that connection back in the connection pool.
2017-07-11 15:11:13 +02:00
PGresult *result;
Properly release pgsql connection if needed
2014-08-14 14:46:21 +02:00
if (pgsql->conn == NULL)
return;
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
/* Async query cleanup */
if (pgsql->flags & KORE_PGSQL_ASYNC) {
pgsql improvements. - adds new cleanup function that workers will call. - adds kore_pgsql_nfields() to return number of fields in result. - add kore_pgsql_fieldname() to return name of a given field. This commit also changes the behaviour of pgsql_conn_release() in that it will now cancel the active query before releasing the connection. This makes sure that if long running queries are active they are hopefully cancelled if an http request is removed while such queries are still running.
2017-02-06 11:40:33 +01:00
if (pgsql->flags & KORE_PGSQL_SCHEDULED) {
fd = PQsocket(pgsql->conn->db);
kore_platform_disable_read(fd);
pgsql: don't do a PQcancel() if it's not needed.
2017-02-06 20:01:16 +01:00
if (pgsql->state != KORE_PGSQL_STATE_DONE)
pgsql_cancel(pgsql);
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
}
pgsql improvements. - adds new cleanup function that workers will call. - adds kore_pgsql_nfields() to return number of fields in result. - add kore_pgsql_fieldname() to return name of a given field. This commit also changes the behaviour of pgsql_conn_release() in that it will now cancel the active query before releasing the connection. This makes sure that if long running queries are active they are hopefully cancelled if an http request is removed while such queries are still running.
2017-02-06 11:40:33 +01:00
kore_pool_put(&pgsql_job_pool, pgsql->conn->job);
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
}
Properly release pgsql connection if needed
2014-08-14 14:46:21 +02:00
pgsql improvements * set pgsql->result to NULL when we freed it * drain the connection pipe just in case in pgsql_conn_release()
2014-08-14 16:43:40 +02:00
/* Drain just in case. */
Several postgresql improvements. - Make pgsql_conn_count count per database rather then globally. This means you now define the number of clients *per* database registered rather then the number of clients in total of all databases. - In case a connection is in failed transaction state Kore will now automatically rollback the transaction before placing that connection back in the connection pool.
2017-07-11 15:11:13 +02:00
while ((result = PQgetResult(pgsql->conn->db)) != NULL)
PQclear(result);
pgsql improvements * set pgsql->result to NULL when we freed it * drain the connection pipe just in case in pgsql_conn_release()
2014-08-14 16:43:40 +02:00
Properly release pgsql connection if needed
2014-08-14 14:46:21 +02:00
pgsql->conn->job = NULL;
pgsql->conn->flags |= PGSQL_CONN_FREE;
TAILQ_INSERT_TAIL(&pgsql_conn_free, pgsql->conn, list);
pgsql->conn = NULL;
Introduce a wait queue for our pgsql code. Instead of letting http_requests spin, if we cannot allocate a connection for the request we will queue them up put them to sleep. When a connection becomes available, we'll wake up a request that was waiting for a connection and let it continue. This completely avoids consuming massive amounts of cpu time when dealing with thousands of requests waiting for a pgsql worker to become ready.
2014-08-21 16:36:12 +02:00
pgsql->state = KORE_PGSQL_STATE_COMPLETE;
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
if (pgsql->cb != NULL)
pgsql->cb(pgsql, pgsql->arg);
Introduce a wait queue for our pgsql code. Instead of letting http_requests spin, if we cannot allocate a connection for the request we will queue them up put them to sleep. When a connection becomes available, we'll wake up a request that was waiting for a connection and let it continue. This completely avoids consuming massive amounts of cpu time when dealing with thousands of requests waiting for a pgsql worker to become ready.
2014-08-21 16:36:12 +02:00
pgsql_queue_wakeup();
Properly release pgsql connection if needed
2014-08-14 14:46:21 +02:00
}
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
static void
pgsql_conn_cleanup(struct pgsql_conn *conn)
{
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
struct kore_pgsql *pgsql;
Several postgresql improvements. - Make pgsql_conn_count count per database rather then globally. This means you now define the number of clients *per* database registered rather then the number of clients in total of all databases. - In case a connection is in failed transaction state Kore will now automatically rollback the transaction before placing that connection back in the connection pool.
2017-07-11 15:11:13 +02:00
struct pgsql_db *pgsqldb;
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
if (conn->flags & PGSQL_CONN_FREE)
TAILQ_REMOVE(&pgsql_conn_free, conn, list);
if (conn->job) {
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
pgsql = conn->job->pgsql;
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
#if !defined(KORE_NO_HTTP)
if (pgsql->req != NULL)
http_request_wakeup(pgsql->req);
#endif
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
pgsql->conn = NULL;
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
pgsql_set_error(pgsql, PQerrorMessage(conn->db));
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
Introduce a wait queue for our pgsql code. Instead of letting http_requests spin, if we cannot allocate a connection for the request we will queue them up put them to sleep. When a connection becomes available, we'll wake up a request that was waiting for a connection and let it continue. This completely avoids consuming massive amounts of cpu time when dealing with thousands of requests waiting for a pgsql worker to become ready.
2014-08-21 16:36:12 +02:00
kore_pool_put(&pgsql_job_pool, conn->job);
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
conn->job = NULL;
}
if (conn->db != NULL)
PQfinish(conn->db);
Several postgresql improvements. - Make pgsql_conn_count count per database rather then globally. This means you now define the number of clients *per* database registered rather then the number of clients in total of all databases. - In case a connection is in failed transaction state Kore will now automatically rollback the transaction before placing that connection back in the connection pool.
2017-07-11 15:11:13 +02:00
LIST_FOREACH(pgsqldb, &pgsql_db_conn_strings, rlist) {
fix compare match on db name. from Kevin Lam via patches@
2018-09-19 07:18:46 +02:00
if (!strcmp(pgsqldb->name, conn->name)) {
small pgsql fixes. - make sure conn_count per pgsqldb structure is initialized to 0. - allow pgsql_conn_max to be 0, meaning just create a new connection if none was free.
2017-08-21 14:25:09 +02:00
pgsqldb->conn_count--;
Several postgresql improvements. - Make pgsql_conn_count count per database rather then globally. This means you now define the number of clients *per* database registered rather then the number of clients in total of all databases. - In case a connection is in failed transaction state Kore will now automatically rollback the transaction before placing that connection back in the connection pool.
2017-07-11 15:11:13 +02:00
break;
}
}
Large changes to the memory subsystem in kore. - Change pools to use mmap() for allocating regions. - Change kore_malloc() to use pools for commonly sized objects. (split into multiple of 2 buckets, starting at 8 bytes up to 8192). - Rename kore_mem_free() to kore_free(). The preallocated pools will hold up to 128K of elements per block size. In case a larger object is to be allocated kore_malloc() will use malloc() instead.
2016-07-12 13:54:14 +02:00
kore_free(conn->name);
kore_free(conn);
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
}
Layout that stuff like a 80 column terminal nerd. And mark a request as DONE when we get PGRES_COMMAND_OK.
2014-03-31 01:04:22 +02:00
static void
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
pgsql_read_result(struct kore_pgsql *pgsql)
Layout that stuff like a 80 column terminal nerd. And mark a request as DONE when we get PGRES_COMMAND_OK.
2014-03-31 01:04:22 +02:00
{
pgsql improvements round 2 - kill PQsetnonblocking(), it's not what one thinks. - keep doing the PQconsumeInput()/PQisBusy() dance until the former clears all read data.
2019-09-13 23:20:51 +02:00
struct pgsql_conn *conn;
PGnotify *notify;
int saved_errno;
Add KORE_PGSQL_STATE_NOTIFY. Issue a LISTEN channel on a kore_pgsql, bind a callback to it and you will get called with pgsql->state being KORE_PGSQL_STATE_NOTIFY.
2018-07-18 11:38:17 +02:00
pgsql improvements round 2 - kill PQsetnonblocking(), it's not what one thinks. - keep doing the PQconsumeInput()/PQisBusy() dance until the former clears all read data.
2019-09-13 23:20:51 +02:00
conn = pgsql->conn;
for (;;) {
if (!PQconsumeInput(conn->db)) {
Call PQConsumeInput() again after PQisBusy(). Prevents a stall in case there is still data in the read end of the socket but PQisBusy() told us to not fetch a result yet. In that case we end up stalling due to epoll not giving us another EPOLLIN event due to EPOLLET.
2019-09-04 19:19:52 +02:00
pgsql->state = KORE_PGSQL_STATE_ERROR;
pgsql improvements round 2 - kill PQsetnonblocking(), it's not what one thinks. - keep doing the PQconsumeInput()/PQisBusy() dance until the former clears all read data.
2019-09-13 23:20:51 +02:00
pgsql->error = kore_strdup(PQerrorMessage(conn->db));
Call PQConsumeInput() again after PQisBusy(). Prevents a stall in case there is still data in the read end of the socket but PQisBusy() told us to not fetch a result yet. In that case we end up stalling due to epoll not giving us another EPOLLIN event due to EPOLLET.
2019-09-04 19:19:52 +02:00
return;
pgsql improvements round 2 - kill PQsetnonblocking(), it's not what one thinks. - keep doing the PQconsumeInput()/PQisBusy() dance until the former clears all read data.
2019-09-13 23:20:51 +02:00
}
saved_errno = errno;
if (PQisBusy(conn->db)) {
if (saved_errno != EAGAIN && saved_errno != EWOULDBLOCK)
continue;
Call PQConsumeInput() again after PQisBusy(). Prevents a stall in case there is still data in the read end of the socket but PQisBusy() told us to not fetch a result yet. In that case we end up stalling due to epoll not giving us another EPOLLIN event due to EPOLLET.
2019-09-04 19:19:52 +02:00
pgsql->state = KORE_PGSQL_STATE_WAIT;
trailing space fix
2019-09-13 23:22:38 +02:00
conn->evt.flags &= ~KORE_EVENT_READ;
Call PQConsumeInput() again after PQisBusy(). Prevents a stall in case there is still data in the read end of the socket but PQisBusy() told us to not fetch a result yet. In that case we end up stalling due to epoll not giving us another EPOLLIN event due to EPOLLET.
2019-09-04 19:19:52 +02:00
return;
}
pgsql improvements round 2 - kill PQsetnonblocking(), it's not what one thinks. - keep doing the PQconsumeInput()/PQisBusy() dance until the former clears all read data.
2019-09-13 23:20:51 +02:00
break;
Layout that stuff like a 80 column terminal nerd. And mark a request as DONE when we get PGRES_COMMAND_OK.
2014-03-31 01:04:22 +02:00
}
pgsql improvements round 2 - kill PQsetnonblocking(), it's not what one thinks. - keep doing the PQconsumeInput()/PQisBusy() dance until the former clears all read data.
2019-09-13 23:20:51 +02:00
while ((notify = PQnotifies(conn->db)) != NULL) {
Add KORE_PGSQL_STATE_NOTIFY. Issue a LISTEN channel on a kore_pgsql, bind a callback to it and you will get called with pgsql->state being KORE_PGSQL_STATE_NOTIFY.
2018-07-18 11:38:17 +02:00
pgsql->state = KORE_PGSQL_STATE_NOTIFY;
Consume all notifications on a connection.
2018-07-18 11:40:59 +02:00
pgsql->notify.extra = notify->extra;
pgsql->notify.channel = notify->relname;
Add KORE_PGSQL_STATE_NOTIFY. Issue a LISTEN channel on a kore_pgsql, bind a callback to it and you will get called with pgsql->state being KORE_PGSQL_STATE_NOTIFY.
2018-07-18 11:38:17 +02:00
Consume all notifications on a connection.
2018-07-18 11:40:59 +02:00
if (pgsql->cb != NULL)
pgsql->cb(pgsql, pgsql->arg);
Add KORE_PGSQL_STATE_NOTIFY. Issue a LISTEN channel on a kore_pgsql, bind a callback to it and you will get called with pgsql->state being KORE_PGSQL_STATE_NOTIFY.
2018-07-18 11:38:17 +02:00
PQfreemem(notify);
}
pgsql improvements round 2 - kill PQsetnonblocking(), it's not what one thinks. - keep doing the PQconsumeInput()/PQisBusy() dance until the former clears all read data.
2019-09-13 23:20:51 +02:00
pgsql->result = PQgetResult(conn->db);
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
if (pgsql->result == NULL) {
pgsql->state = KORE_PGSQL_STATE_DONE;
Layout that stuff like a 80 column terminal nerd. And mark a request as DONE when we get PGRES_COMMAND_OK.
2014-03-31 01:04:22 +02:00
return;
}
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
switch (PQresultStatus(pgsql->result)) {
Layout that stuff like a 80 column terminal nerd. And mark a request as DONE when we get PGRES_COMMAND_OK.
2014-03-31 01:04:22 +02:00
case PGRES_COPY_OUT:
case PGRES_COPY_IN:
case PGRES_NONFATAL_ERROR:
case PGRES_COPY_BOTH:
break;
case PGRES_COMMAND_OK:
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
pgsql->state = KORE_PGSQL_STATE_DONE;
Layout that stuff like a 80 column terminal nerd. And mark a request as DONE when we get PGRES_COMMAND_OK.
2014-03-31 01:04:22 +02:00
break;
case PGRES_TUPLES_OK:
Check for pg version >= 90200 before using PGRES_SINGLE_TUPLE
2014-07-03 22:58:04 +02:00
#if PG_VERSION_NUM >= 90200
Layout that stuff like a 80 column terminal nerd. And mark a request as DONE when we get PGRES_COMMAND_OK.
2014-03-31 01:04:22 +02:00
case PGRES_SINGLE_TUPLE:
Check for pg version >= 90200 before using PGRES_SINGLE_TUPLE
2014-07-03 22:58:04 +02:00
#endif
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
pgsql->state = KORE_PGSQL_STATE_RESULT;
Layout that stuff like a 80 column terminal nerd. And mark a request as DONE when we get PGRES_COMMAND_OK.
2014-03-31 01:04:22 +02:00
break;
case PGRES_EMPTY_QUERY:
case PGRES_BAD_RESPONSE:
case PGRES_FATAL_ERROR:
Introduce synchronous pgsql queries. Semantics for using pgsql API have changed quite heavily with this commit. See the examples for more information. Based on Github issue #95 by PauloMelo (paulo.melo@vintageform.pt) with several modifications by me.
2016-01-04 11:12:43 +01:00
pgsql_set_error(pgsql, PQresultErrorMessage(pgsql->result));
Layout that stuff like a 80 column terminal nerd. And mark a request as DONE when we get PGRES_COMMAND_OK.
2014-03-31 01:04:22 +02:00
break;
}
}
pgsql: don't do a PQcancel() if it's not needed.
2017-02-06 20:01:16 +01:00
static void
pgsql_cancel(struct kore_pgsql *pgsql)
{
PGcancel *cancel;
char buf[256];
if ((cancel = PQgetCancel(pgsql->conn->db)) != NULL) {
if (!PQcancel(cancel, buf, sizeof(buf)))
kore_log(LOG_ERR, "failed to cancel: %s", buf);
PQfreeCancel(cancel);
}
}
Reference in New Issue Copy Permalink